Store messages in files and load them again

General
1

Hi folks,

I have ended up in a situation where I should store messages in files and load them afterwards back from those files. That works fine as long as the msg.payload contains a Buffer OR text data, since you can select both options in the FileIn node:

image

However in the following example the msg.payload contains both a Buffer AND text data:

  • msg.payload.value contains a Buffer (i.e. an image)
  • msg.payload.timestamp contains a timestamp

An image is loaded with the httpRequest node, then the message is stored in a file and loaded back from that file. But at the end the image-output node cannot display the image anymore:

image
image.png895Ă—182 40.7 KB 

[{"id":"ee154571.147958","type":"http request","z":"279b8956.27dfe6","name":"","method":"GET","ret":"bin","url":"https://accelerator-origin.kkomando.com/wp-content/uploads/2016/08/shutterstock_330700175-970x546.jpg","tls":"","x":2530,"y":1480,"wires":[["dd960318.2c13","15dc7bdf.af3554"]]},{"id":"dd960318.2c13","type":"image","z":"279b8956.27dfe6","name":"","width":200,"x":2750,"y":1480,"wires":[]},{"id":"42c88be5.b80494","type":"inject","z":"279b8956.27dfe6","name":"Start","topic":"SomeTopic","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":2370,"y":1480,"wires":[["ee154571.147958"]]},{"id":"92fb64a6.633828","type":"file in","z":"279b8956.27dfe6","name":"Load from /var/log/temp.msg","filename":"/var/log/temp.msg","format":"lines","chunk":false,"sendError":false,"x":3260,"y":1420,"wires":[["debad75f.485d58","58f6c87.e124c38"]]},{"id":"838f8d6c.5ad81","type":"file","z":"279b8956.27dfe6","name":"Save to /var/log/temp.msg","filename":"/var/log/temp.msg","appendNewline":false,"createDir":false,"overwriteFile":"true","x":2990,"y":1420,"wires":[["92fb64a6.633828"]]},{"id":"f841c9f6.267c88","type":"image","z":"279b8956.27dfe6","name":"","width":200,"x":3690,"y":1420,"wires":[]},{"id":"15dc7bdf.af3554","type":"function","z":"279b8956.27dfe6","name":"Create payload","func":"var outputMsg = {};\noutputMsg.payload = {};\noutputMsg.payload.timestamp = new Date();\noutputMsg.payload.value = msg.payload;\nreturn outputMsg;","outputs":1,"noerr":0,"x":2760,"y":1420,"wires":[["838f8d6c.5ad81","3bf5aea8.7ac602"]]},{"id":"debad75f.485d58","type":"change","z":"279b8956.27dfe6","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.value","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":3500,"y":1420,"wires":[["f841c9f6.267c88"]]},{"id":"3bf5aea8.7ac602","type":"debug","z":"279b8956.27dfe6","name":"Original message","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":2970,"y":1360,"wires":[]},{"id":"58f6c87.e124c38","type":"debug","z":"279b8956.27dfe6","name":"Loaded message","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":3510,"y":1360,"wires":[]}]

I need to treat the image as a Buffer, but now the 'entire' payload has become a single Buffer:
image

P.S. I also tried to load the message with this option:
image
But that doesn't help either:
image

Does anybody know how I can make sure that the loaded message is identical to the original message?

Thanks !!
Bart

1 Like
2

Buffer is not a JSON type, so it cannot be encoded in JSON and retrieved seamlessly.

As youve seen, the Buffer type does have a custom toJSON function which causes it to be encoded as an object with the type and data properties.

The best option you have is to add a function that recreates the Buffer object from that array.

3

Hey Nick,

Thanks !
OMG I now realize that this was one of the most stupid questions I have ever asked on this forum :woozy_face:

Indeed when I add this simple function node behind the FileIn node:

var outputMsg = {};
outputMsg.payload = JSON.parse(msg.payload);
return outputMsg;

Then the loaded message is identical to the original message:
image

So far so good...
Except from the fact that I still don't see the image appearing in the last image-output node...
In most of the cases that is caused when using UTF8 string instead of Buffer, which resuls in a corrupt image.

I assume that this is caused by the fact that the Buffer is converted to JSON and back (so some of the 47196 bytes in the buffer are not identical) ???
Any advise on this one?

4

In case anybody is wondering why on earth storing messages in files could be useful:
Well I'm currently developing streaming audio in the dashboard for another discussion.

I have now arrived at the point that it seems to be working fine with my own raspberry and usb microphone. However when @GChapo uses my dashboard fix to test with his raspberry and microphone, the sound is very bad. I have no clue what is going on over there, so I need to have access to his audio stream on my own Raspberry (to be able to debug it).

So I'm trying to use the node-red-contrib-sequencer nodes to copy his audio stream:
image

  • Glenn would record all his audio related messages during e.g. 10 seconds, and store those messages (inclusive timestamps) into one big file.
  • Glenn delivers me that file manually via Dropbox.
  • I replay all the messages from that file, with the same delays in between (based on the timestamps) in my dashboard. So all audio samples (hopefully) arrive at the same sample rate in my own dashboard.

This way I hope to be able to have an exact copy of Glenn's audio stream available at my own Raspberry, so I could debug the dashboard to see what is going wrong ...

5

Bart - you may want to try the base64 node - you can use it to convert the property that is a buffer into a base64 string which can then be saved... and then likewise regenerated at the other side.

1 Like
6

@BartButenaers as Dave says, using the base64 node is probably the best way to encode a buffer.

But just to repeat what I said in my original reply, you cannot just use JSON.parse and expect to get a Buffer object back - JSON.parse doesn't know anything about Buffer objects; they are a Node specific object type, not a standard JavaScript object.

So where I said you'd need to use a function to recreate the Buffer object from that array, I meant something like:

var parsedMessage = JSON.parse(msg.payload);
parsedMessage.value = Buffer.from(parsedMessage.value.data);
return parsedMessage;

The fact the Debug sidebar claims you have a Buffer might be a bug - or at least, an edge case. For all of the same reasons I've described here, when we send a message over the websocket to display in the sidebar, we don't simply JSON.stringify it - because that loses information about all the types JSON doesn't natively support. It's possible its misinterpreting what you're giving it as a real Buffer object type when displaying it.

7

That indeed did the job. The burglar has now been spotted in the last image-output node:

image
image935Ă—184 59.8 KB

Thanks for the explanation! I have always though that you just did JSON.stringify. Have never looked at that part of your code ...