# Store messages in files and load them again

**URL:** <https://discourse.nodered.org/t/store-messages-in-files-and-load-them-again/4346>\
**Category:** General\
**Created:** [27 October 2018 21:08 UTC](https://discourse.nodered.org/t/store-messages-in-files-and-load-them-again/4346 "2018-10-27T21:08:58Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [27 October 2018 21:08 UTC](https://discourse.nodered.org/t/store-messages-in-files-and-load-them-again/4346/1 "2018-10-27T21:08:58Z")

</div>

Hi folks,

I have ended up in a situation where I should store messages in files and load them afterwards back from those files. That works fine as long as the `msg.payload` contains a Buffer **OR** text data, since you can select both options in the FileIn node:

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/b/b81a7114dd5da84e973258a515bad5fbc1df5bda.png)

However in the following example the `msg.payload` contains both a Buffer **AND** text data:

- `msg.payload.value` contains a Buffer (i.e. an image)
- `msg.payload.timestamp` contains a timestamp

An image is loaded with the httpRequest node, then the message is stored in a file and loaded back from that file. But at the end the image-output node cannot display the image anymore:

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/f/fd1cc7d2963edc10320b4dcd1cb2650396303a0e.png)

```auto
[{"id":"ee154571.147958","type":"http request","z":"279b8956.27dfe6","name":"","method":"GET","ret":"bin","url":"https://accelerator-origin.kkomando.com/wp-content/uploads/2016/08/shutterstock_330700175-970x546.jpg","tls":"","x":2530,"y":1480,"wires":[["dd960318.2c13","15dc7bdf.af3554"]]},{"id":"dd960318.2c13","type":"image","z":"279b8956.27dfe6","name":"","width":200,"x":2750,"y":1480,"wires":[]},{"id":"42c88be5.b80494","type":"inject","z":"279b8956.27dfe6","name":"Start","topic":"SomeTopic","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":2370,"y":1480,"wires":[["ee154571.147958"]]},{"id":"92fb64a6.633828","type":"file in","z":"279b8956.27dfe6","name":"Load from /var/log/temp.msg","filename":"/var/log/temp.msg","format":"lines","chunk":false,"sendError":false,"x":3260,"y":1420,"wires":[["debad75f.485d58","58f6c87.e124c38"]]},{"id":"838f8d6c.5ad81","type":"file","z":"279b8956.27dfe6","name":"Save to /var/log/temp.msg","filename":"/var/log/temp.msg","appendNewline":false,"createDir":false,"overwriteFile":"true","x":2990,"y":1420,"wires":[["92fb64a6.633828"]]},{"id":"f841c9f6.267c88","type":"image","z":"279b8956.27dfe6","name":"","width":200,"x":3690,"y":1420,"wires":[]},{"id":"15dc7bdf.af3554","type":"function","z":"279b8956.27dfe6","name":"Create payload","func":"var outputMsg = {};\noutputMsg.payload = {};\noutputMsg.payload.timestamp = new Date();\noutputMsg.payload.value = msg.payload;\nreturn outputMsg;","outputs":1,"noerr":0,"x":2760,"y":1420,"wires":[["838f8d6c.5ad81","3bf5aea8.7ac602"]]},{"id":"debad75f.485d58","type":"change","z":"279b8956.27dfe6","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.value","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":3500,"y":1420,"wires":[["f841c9f6.267c88"]]},{"id":"3bf5aea8.7ac602","type":"debug","z":"279b8956.27dfe6","name":"Original message","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":2970,"y":1360,"wires":[]},{"id":"58f6c87.e124c38","type":"debug","z":"279b8956.27dfe6","name":"Loaded message","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":3510,"y":1360,"wires":[]}]

```

I need to treat the image as a Buffer, but now the '_entire_' payload has become a **single Buffer** :  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/9/92dabcb0d4dbbc494d1e5a9807d17512519f0e2e.png)

P.S. I also tried to load the message with this option:  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/5/5681aa1961f0c3f54918d4e375a39709eca4cee2.png)  
But that doesn't help either:  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/c/c7b8a5508fa7820aac46cff383e5a99638f092ae.png)

Does anybody know how I can make sure that the loaded message is identical to the original message?

Thanks !!  
Bart

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [27 October 2018 22:00 UTC](https://discourse.nodered.org/t/store-messages-in-files-and-load-them-again/4346/2 "2018-10-27T22:00:45Z")

</div>

Buffer is not a JSON type, so it cannot be encoded in JSON and retrieved seamlessly.

As youve seen, the Buffer type does have a custom toJSON function which causes it to be encoded as an object with the type and data properties.

The best option you have is to add a function that recreates the Buffer object from that array.

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [27 October 2018 22:38 UTC](https://discourse.nodered.org/t/store-messages-in-files-and-load-them-again/4346/3 "2018-10-27T22:38:17Z")

</div>

Hey Nick,

Thanks !  
OMG I now realize that this was one of the most stupid questions I have ever asked on this forum 🥴

Indeed when I add this simple **function** node behind the FileIn node:

```auto
var outputMsg = {};
outputMsg.payload = JSON.parse(msg.payload);
return outputMsg;

```

Then the loaded message is **identical** to the original message:  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/5/53b8ca977d9a07eade1cec2aa896ff116546deb3.png)

So far so good...  
Except from the fact that I still don't see the image appearing in the last image-output node...  
In most of the cases that is caused when using UTF8 string instead of Buffer, which resuls in a corrupt image.

I assume that this is caused by the fact that the Buffer is converted to JSON and back (so some of the 47196 bytes in the buffer are not identical) ???  
Any advise on this one?

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [27 October 2018 22:56 UTC](https://discourse.nodered.org/t/store-messages-in-files-and-load-them-again/4346/4 "2018-10-27T22:56:52Z")

</div>

In case anybody is wondering why on earth storing messages in files could be useful:  
Well I'm currently developing streaming audio in the dashboard for another [discussion](https://discourse.nodered.org/t/play-sound-from-usb-microphone/2551/28).

I have now arrived at the point that it seems to be working fine with my own raspberry and usb microphone. However when @GChapo uses my dashboard fix to test with his raspberry and microphone, the sound is very bad. I have no clue what is going on over there, so I need to have access to his audio stream on my own Raspberry (to be able to debug it).

So I'm trying to use the [node-red-contrib-sequencer](https://github.com/tedstriker/node-red-contrib-sequencer) nodes to copy his audio stream:  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/3/3c148e13738d77ae9f64839fe9b81f06ac1a5b96.png)

- Glenn would record all his audio related messages during e.g. 10 seconds, and store those messages (inclusive timestamps) into one big file.
- Glenn delivers me that file manually via Dropbox.
- I replay all the messages from that file, with the same delays in between (based on the timestamps) in my dashboard. So all audio samples (hopefully) arrive at the **same sample rate** in my own dashboard.

This way I hope to be able to have an exact copy of Glenn's audio stream available at my own Raspberry, so I could debug the dashboard to see what is going wrong ...

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [28 October 2018 08:40 UTC](https://discourse.nodered.org/t/store-messages-in-files-and-load-them-again/4346/5 "2018-10-28T08:40:34Z")

</div>

Bart - you may want to try the base64 node - you can use it to convert the property that is a buffer into a base64 string which can then be saved... and then likewise regenerated at the other side.

---

<div class="post-metadata">

**Author:** ![knolleary](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/knolleary/32/3_2.png) [@knolleary](https://discourse.nodered.org/u/knolleary)\
**Post date:** [28 October 2018 10:21 UTC](https://discourse.nodered.org/t/store-messages-in-files-and-load-them-again/4346/6 "2018-10-28T10:21:45Z")

</div>

@BartButenaers as Dave says, using the base64 node is probably the best way to encode a buffer.

But just to repeat what I said in my original reply, you cannot just use JSON.parse and expect to get a Buffer object back - JSON.parse doesn't know anything about Buffer objects; they are a Node specific object type, not a standard JavaScript object.

So where I said you'd need to use a function to recreate the Buffer object from that array, I meant something like:

```auto
var parsedMessage = JSON.parse(msg.payload);
parsedMessage.value = Buffer.from(parsedMessage.value.data);
return parsedMessage;

```

The fact the Debug sidebar claims you have a Buffer _might_ be a bug - or at least, an edge case. For all of the same reasons I've described here, when we send a message over the websocket to display in the sidebar, we don't simply JSON.stringify it - because that loses information about all the types JSON doesn't natively support. It's possible its misinterpreting what you're giving it as a real Buffer object type when displaying it.

---

<div class="post-metadata">

**Author:** ![BartButenaers](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bartbutenaers/32/10476_2.png) [@BartButenaers](https://discourse.nodered.org/u/BartButenaers)\
**Post date:** [28 October 2018 11:02 UTC](https://discourse.nodered.org/t/store-messages-in-files-and-load-them-again/4346/7 "2018-10-28T11:02:34Z")

</div>

> [@dceejay](#):
>
> you may want to try the base64 node

That indeed did the job. The burglar has now been spotted in the last image-output node:

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/7/7cfd36e0819830e404b18406595c272b528be3f2.jpeg)

> [@knolleary](#):
>
> we don't simply JSON.stringify it

Thanks for the explanation! I have always though that you just did JSON.stringify. Have never looked at that part of your code ...