Decode Base64 as unicode string

[cymplecy]

I'm playing around with Base64 encoding/decoding but I'm having a problem decoding a base64 encoded string with non 7bit char back into it's original unicode appearance

[{"id":"8285d46540f7a8ce","type":"inject","z":"6c0906b6354c0863","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"A £ B","payloadType":"str","x":170,"y":480,"wires":[["1daf2f2b67d5610d"]]},{"id":"1daf2f2b67d5610d","type":"base64","z":"6c0906b6354c0863","name":"","action":"","property":"payload","x":320,"y":480,"wires":[["f62b1b976c7ac977","cda376d17ff5dfd6"]]},{"id":"f62b1b976c7ac977","type":"debug","z":"6c0906b6354c0863","name":"debug 479","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"payload","targetType":"msg","statusVal":"payload","statusType":"auto","x":330,"y":530,"wires":[]},{"id":"cda376d17ff5dfd6","type":"base64","z":"6c0906b6354c0863","name":"","action":"b64","property":"payload","x":560,"y":480,"wires":[["fb6f76747517a14a"]]},{"id":"fb6f76747517a14a","type":"debug","z":"6c0906b6354c0863","name":"debug 480","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"payload","targetType":"msg","statusVal":"payload","statusType":"auto","x":570,"y":530,"wires":[]}]

Any ideas?
[edit] And by ideas - I mean solutions ]

[zenofmud]

After doing a google search using 'valid base64 characters' It looks like £ is not a valid base64 character

Base64 is a scheme for converting binary data to printable ASCII characters, namely the upper- and lower-case Roman alphabet characters (A–Z, a–z), the numerals (0–9), and the "+" and "/" symbols, with the "=" symbol as a special suffix code.

Interestingly, the tool validator (in the link above) encode A £ B using 'Text to Base64' to QSDCoyBC and then if you use the 'Base64 to Text' option, it does convert it back to A £ B

It looks like the node just doesn't do a 'Base64 to text'.

[cymplecy]

Just for background - I've written several encoders/decoders in other languages so I have a reasonable grasp of the protocol

[zenofmud]

I just took a look at the code (70-base64.js in node-red-node-base64) and one of the first lines is this:
var regexp = new RegExp('^[A-Za-z0-9+\/=]*$'); // check it only contains valid characters
So £ is an invalid character in this node

[cymplecy]

I think you are mixing up the difference between the input string and the base64 representation of it.

The base64 encoding "QSDCoyBC" doesn't contain a £

The way an encoder works (99% of time) is to convert a string into the UTF-8 byte representation so that all the data is a buffer/list of 0-255 byte values.

So a "£" (which has a unicode value of 163) gets converted to its UTF-8 representation [194,163] before being encoded to Base64

[edit] I've realised that "£" is not a good example
A better one is "𐀀" which has a unicode value of 65536 - it's UTF-8 representation is [240, 144, 128, 128]

[zenofmud]

Maybe @dceejay will take a look at this.

[cymplecy]

I've come up with a solution

[{"id":"8285d46540f7a8ce","type":"inject","z":"6c0906b6354c0863","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"A £ B","payloadType":"str","x":120,"y":480,"wires":[["1daf2f2b67d5610d"]]},{"id":"1daf2f2b67d5610d","type":"base64","z":"6c0906b6354c0863","name":"","action":"","property":"payload","x":330,"y":480,"wires":[["f62b1b976c7ac977","cda376d17ff5dfd6","7a462bc3a020418d"]]},{"id":"f62b1b976c7ac977","type":"debug","z":"6c0906b6354c0863","name":"debug 479","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"payload","targetType":"msg","statusVal":"payload","statusType":"auto","x":320,"y":530,"wires":[]},{"id":"cda376d17ff5dfd6","type":"base64","z":"6c0906b6354c0863","name":"","action":"b64","property":"payload","x":560,"y":480,"wires":[["fb6f76747517a14a"]]},{"id":"fb6f76747517a14a","type":"debug","z":"6c0906b6354c0863","name":"debug 480","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"payload","targetType":"msg","statusVal":"payload","statusType":"auto","x":570,"y":530,"wires":[]},{"id":"692f795809e9cf96","type":"buffer-parser","z":"6c0906b6354c0863","name":"","data":"payload","dataType":"msg","specification":"spec","specificationType":"ui","items":[{"type":"utf8","name":"item1","offset":0,"length":-1,"offsetbit":0,"scale":"1","mask":""}],"swap1":"","swap2":"","swap3":"","swap1Type":"swap","swap2Type":"swap","swap3Type":"swap","msgProperty":"payload","msgPropertyType":"str","resultType":"value","resultTypeType":"return","multipleResult":true,"fanOutMultipleResult":false,"setTopic":true,"outputs":1,"x":780,"y":590,"wires":[["9826d91e501fb02d"]]},{"id":"b509edea0dd49467","type":"debug","z":"6c0906b6354c0863","name":"debug 481","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"payload","targetType":"msg","statusVal":"payload","statusType":"auto","x":570,"y":640,"wires":[]},{"id":"6fe53ce00e711715","type":"base64","z":"6c0906b6354c0863","name":"","action":"","property":"payload","x":560,"y":590,"wires":[["692f795809e9cf96","b509edea0dd49467"]]},{"id":"9826d91e501fb02d","type":"debug","z":"6c0906b6354c0863","name":"debug 482","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"payload","statusType":"msg","x":1000,"y":590,"wires":[]},{"id":"7a462bc3a020418d","type":"junction","z":"6c0906b6354c0863","x":440,"y":590,"wires":[["6fe53ce00e711715"]]}]

The buffer parser is configured like this

[dceejay]

Hi - just back from a trip and catching up.... so hmmmm yes - the behaviour is not quite ideal is it...
What is actually wrong is that the description of the option says "Convert Base64 to String" and then it proceeds to converts it toString("binary") rather than just toString() - which defaults to utf8...

Of course the whole point of base64 encoding is to be able to get transmit a binary payload so is really meant for true binary and if you use the default Convert Buffer <-> Base64 option - then you do get a binary buffer back that you can then toString yourself correctly...

So -I think I should just correct the behaviour of that option to just toString('utf8') it to a string as the label suggests and implies - As this would potentially be a breaking change for some people I would have to bump the node to v1.0

Thoughts please

[{"id":"8285d46540f7a8ce","type":"inject","z":"4adb98c6.b11238","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"A £ B","payloadType":"str","x":110,"y":640,"wires":[["1daf2f2b67d5610d"]]},{"id":"1daf2f2b67d5610d","type":"base64","z":"4adb98c6.b11238","name":"","action":"","property":"payload","x":260,"y":640,"wires":[["f62b1b976c7ac977","cda376d17ff5dfd6"]]},{"id":"f62b1b976c7ac977","type":"debug","z":"4adb98c6.b11238","name":"debug 479","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"payload","targetType":"msg","statusVal":"payload","statusType":"auto","x":270,"y":690,"wires":[]},{"id":"cda376d17ff5dfd6","type":"base64","z":"4adb98c6.b11238","name":"","action":"","property":"payload","x":500,"y":640,"wires":[["fb6f76747517a14a","bede4c28c2bf6249"]]},{"id":"fb6f76747517a14a","type":"debug","z":"4adb98c6.b11238","name":"debug 480","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"payload","targetType":"msg","statusVal":"payload","statusType":"auto","x":510,"y":690,"wires":[]},{"id":"c7a6bf1eeb447772","type":"inject","z":"4adb98c6.b11238","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"A £ B","payloadType":"str","x":110,"y":510,"wires":[["8cae002f0395ed64"]]},{"id":"8cae002f0395ed64","type":"base64","z":"4adb98c6.b11238","name":"","action":"str","property":"payload","x":260,"y":510,"wires":[["b9fb9006e6440745","fb9bc53d81ac7bd8"]]},{"id":"b9fb9006e6440745","type":"debug","z":"4adb98c6.b11238","name":"debug 481","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"payload","targetType":"msg","statusVal":"payload","statusType":"auto","x":270,"y":560,"wires":[]},{"id":"fb9bc53d81ac7bd8","type":"base64","z":"4adb98c6.b11238","name":"","action":"b64","property":"payload","x":500,"y":510,"wires":[["573f50d928f5b6c4"]]},{"id":"573f50d928f5b6c4","type":"debug","z":"4adb98c6.b11238","name":"debug 482","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"payload","targetType":"msg","statusVal":"payload","statusType":"auto","x":510,"y":560,"wires":[]},{"id":"bede4c28c2bf6249","type":"function","z":"4adb98c6.b11238","name":"function 2","func":"msg.payload = msg.payload.toString()\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":680,"y":640,"wires":[["3ecfea4c43cfb1ec"]]},{"id":"3ecfea4c43cfb1ec","type":"debug","z":"4adb98c6.b11238","name":"debug 483","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"payload","targetType":"msg","statusVal":"payload","statusType":"auto","x":730,"y":700,"wires":[]}]

[cymplecy]

@dceejay

So - I think I should just correct the behaviour of that option to just toString('utf8') it to a string as the label suggests and implies

That would be great

[dceejay]

Published as version 1.0

[cymplecy]

Looking good