# Exec and daemon nodes break up a data stream

**URL:** <https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695>\
**Category:** General\
**Created:** [30 April 2019 21:10 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695 "2019-04-30T21:10:33Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![drmibell](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/drmibell/32/8424_2.png) [@drmibell](https://discourse.nodered.org/u/drmibell)\
**Post date:** [30 April 2019 21:10 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/1 "2019-04-30T21:10:33Z")

</div>

I am trying to use a [software-defined radio receiver](https://www.rtl-sdr.com/about-rtl-sdr/) to get data from a wireless thermometer. The [program](https://github.com/merbanan/rtl_433) that controls the radio sends a stream of JSON strings separated by newline characters to its standard output. (All of this is running on a Pi with Raspbian.) If I start the program with the `exec` node (in spawn mode) or use the `daemon` node, I almost always get two messages for each JSON string. The place where the string is broken is unpredictable. I have managed to use a `join` node to put the strings back together so that a `json` node can handle them, but I still have to trap occasional errors (from the `json` node) and restart the program if they persist. Is there something I should know about how `exec` and `daemon` work (specifically what triggers an output message) that would help me get a more reliable flow?

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [1 May 2019 06:33 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/2 "2019-05-01T06:33:53Z")

</div>

Show us the config of the exec node and an example of split messages

---

<div class="post-metadata">

**Author:** ![drmibell](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/drmibell/32/8424_2.png) [@drmibell](https://discourse.nodered.org/u/drmibell)\
**Post date:** [1 May 2019 16:45 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/3 "2019-05-01T16:45:03Z")

</div>

The `exec` node configuration is:

 ![exec](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/f/fdbef37ebaac20555bee545b3c9661731cdc8634.jpeg)  
and typical output looks like this:

 ![output](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/5/53256b8e85cd3a73a6617755a8f738d0fcc5ded8.jpeg)

The transmitter sends a pair of messages every 30 seconds, and the program sends output like this to stdout (captured in a terminal session).

```auto
{"time" : "2019-04-30 14:02:53", "brand" : "OS", "model" : "OSv1 Temperature Sensor", "sid" : 11, "channel" : 1, "battery" : "OK", "temperature_C" : 23.100}
{"time" : "2019-04-30 14:02:53", "brand" : "OS", "model" : "OSv1 Temperature Sensor", "sid" : 11, "channel" : 1, "battery" : "OK", "temperature_C" : 23.100}

```

These strings parse properly when injected into a `json` node. I also tried the "Use old style output" checkbox, although I couldn't find documentation of what it does, and got similar results.

Any thoughts?

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [1 May 2019 17:18 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/4 "2019-05-01T17:18:01Z")

</div>

Show us the debug output for the final part of a split message

---

<div class="post-metadata">

**Author:** ![drmibell](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/drmibell/32/8424_2.png) [@drmibell](https://discourse.nodered.org/u/drmibell)\
**Post date:** [1 May 2019 17:50 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/5 "2019-05-01T17:50:43Z")

</div>

Sorry if my explanation wasn't as clear as it should be. You may have to click on or download the second image to see that it shows the output from two JSON strings split into four messages. Is this what you want?

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [1 May 2019 17:55 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/6 "2019-05-01T17:55:15Z")

</div>

Post your flow and use the inject node with the json/strings as input. (to emulate your device)

---

<div class="post-metadata">

**Author:** ![drmibell](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/drmibell/32/8424_2.png) [@drmibell](https://discourse.nodered.org/u/drmibell)\
**Post date:** [1 May 2019 18:22 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/7 "2019-05-01T18:22:50Z")

</div>

This is the guts of my flow, dashboard and error trapping nodes removed and an `inject` node added.

```auto
[{"id":"7db2a4ac.a0d8a4","type":"inject","z":"3705653e.c16b9a","name":"start","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":230,"y":220,"wires":[["c95a681c.9782e"]]},{"id":"c95a681c.9782e","type":"exec","z":"3705653e.c16b9a","command":"rtl_433 -F json","addpay":false,"append":"","useSpawn":"true","timer":"","oldrc":false,"name":"","x":400,"y":240,"wires":[["2f93e801.45b19","7973400d.e3bb08"],["7866c46e.e44bcc"],["6e907ec5.8cacd8"]]},{"id":"2f93e801.45b19","type":"debug","z":"3705653e.c16b9a","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":570,"y":220,"wires":[]},{"id":"7866c46e.e44bcc","type":"debug","z":"3705653e.c16b9a","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":570,"y":260,"wires":[]},{"id":"6e907ec5.8cacd8","type":"debug","z":"3705653e.c16b9a","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":570,"y":300,"wires":[]},{"id":"4ee12036.6d8bd8","type":"change","z":"3705653e.c16b9a","name":"","rules":[{"t":"set","p":"kill","pt":"msg","to":"","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":230,"y":260,"wires":[["c95a681c.9782e"]]},{"id":"384d2671.a3a73a","type":"inject","z":"3705653e.c16b9a","name":"stop","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":90,"y":260,"wires":[["4ee12036.6d8bd8"]]},{"id":"7973400d.e3bb08","type":"join","z":"3705653e.c16b9a","name":"","mode":"custom","build":"string","property":"payload","propertyType":"msg","key":"topic","joiner":"","joinerType":"str","accumulate":false,"timeout":"","count":"2","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":570,"y":180,"wires":[["5574a127.bbcbc","178e7d67.e33b8b"]]},{"id":"5574a127.bbcbc","type":"json","z":"3705653e.c16b9a","name":"","property":"payload","action":"","pretty":false,"x":730,"y":160,"wires":[["2fa0cd01.d029ca"]]},{"id":"2fa0cd01.d029ca","type":"debug","z":"3705653e.c16b9a","name":"object","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":850,"y":160,"wires":[]},{"id":"178e7d67.e33b8b","type":"debug","z":"3705653e.c16b9a","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":730,"y":200,"wires":[]},{"id":"ddb76cd6.bd8138","type":"inject","z":"3705653e.c16b9a","name":"string","topic":"","payload":"{\"time\" : \"2019-04-29 23:56:52\", \"brand\" : \"OS\", \"model\" : \"OSv1 Temperature Sensor\", \"sid\" : 11, \"channel\" : 1, \"battery\" : \"OK\", \"temperature_C\" : 12.800}","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":570,"y":140,"wires":[["5574a127.bbcbc"]]}]

```

As I said in my original post, the `json` node is happy to parse a string from the `rtl_433` program.

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [1 May 2019 21:21 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/8 "2019-05-01T21:21:59Z")

</div>

With rtl-sdr you have to check the output data on the stream as it can be flaky, so before it goes into the json node, first check if it is valid json (the json node won't validate, but will generate an error that can be captured with the 'catch' node)

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [1 May 2019 21:45 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/9 "2019-05-01T21:45:22Z")

</div>

The exec node just outputs "chunks" of data as given to it by the operating system. We don't have any control over it - it depends how the app and OS decide to flush their buffers... often it is on a line break, or after a certain time, or number of characters, or if the app calls a flush.  
In this case it does look like a deliberate flush as the break seems to be in the same place each time.  
Are there any command line parameters you could give (eg to remove the timestamp) to see if it can be manipulated ? (you could re-add a timestamp locally if required).  
Otherwise it may need some digging in the c source of the rtl\_433 code.  
Or you could maybe wrap it (ideally as a companion node to the rtl\_power node - PR please ! 🙂

---

<div class="post-metadata">

**Author:** ![drmibell](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/drmibell/32/8424_2.png) [@drmibell](https://discourse.nodered.org/u/drmibell)\
**Post date:** [2 May 2019 00:05 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/10 "2019-05-02T00:05:13Z")

</div>

Thanks, Dave. I suspected something of the sort. The break seems to come at almost but not exactly the same place each time -- I don't have the patience to count characters. I can redirect the output of `rtl_433` to a file, close the file, and read hundreds of messages from it into a `json` node with no errors, so I think the hardware and software are pretty solid. I'm scratching my head for a way to pipe the data into NR without having to involve stdout in the `exec` node. There is also an option in `rtl_433` to use mqtt, but I haven't been able to get it to work. The last thing I want to do is a contributed node, but... 🙂

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [2 May 2019 07:55 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/11 "2019-05-02T07:55:24Z")

</div>

The solution may be a function node that buffers and concatenates payloads till it sees a CR then sends the message on.

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [2 May 2019 09:04 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/12 "2019-05-02T09:04:35Z")

</div>

Your topic sparked my interest (had a rtl-sdr dongle laying around), I can confirm the same behaviour. Especially when the reception is bad, when I move the antenna closer to my devices i get more consistent data.

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [2 May 2019 09:19 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/13 "2019-05-02T09:19:25Z")

</div>

Another option may be to use another output style that isn’t json. Then parse it yourself.

---

<div class="post-metadata">

**Author:** ![drmibell](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/drmibell/32/8424_2.png) [@drmibell](https://discourse.nodered.org/u/drmibell)\
**Post date:** [2 May 2019 14:19 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/14 "2019-05-02T14:19:56Z")

</div>

Thanks, all, for the helpful comments.

> [@bakman2](#):
>
> I can confirm the same behaviour. Especially when the reception is bad

Thanks for confirming. I'm actually surprised at how well this stuff works, at least for me. I think in radio you get at most what you pay for, and these are really just toys.

> [@dceejay](#):
>
> use another output style that isn’t json

I started with the default, plain text output, and it also has the "chunking" issue. Using JSON made it easier to see what is happening.

> [@Colin](#):
>
> function node that buffers and concatenates payloads till it sees a CR

Doh! 😬 (why isn't there a head-slap emoji?) I was messing with finding the { } delimiters and sort of parsing JSON myself. This is dead easy and should work well enough. Thanks.

---

<div class="post-metadata">

**Author:** ![dceejay](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/dceejay/32/38_2.png) [@dceejay](https://discourse.nodered.org/u/dceejay)\
**Post date:** [2 May 2019 16:37 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/15 "2019-05-02T16:37:29Z")

</div>

Another head slap. Would the split node do this for you ? (Not near computer to check)  
Maybe not but would make a good PR to add it. 🤦‍♂️

---

<div class="post-metadata">

**Author:** ![drmibell](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/drmibell/32/8424_2.png) [@drmibell](https://discourse.nodered.org/u/drmibell)\
**Post date:** [2 May 2019 18:20 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/16 "2019-05-02T18:20:14Z")

</div>

> [@dceejay](#):
>
> Would the split node do this for you ?

Possibly, though at first glance I doubt it, because the payload I want is already split between two or more messages. I'll check. And thanks for the emoji -- I know it will come in handy.

---

<div class="post-metadata">

**Author:** ![drmibell](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/drmibell/32/8424_2.png) [@drmibell](https://discourse.nodered.org/u/drmibell)\
**Post date:** [3 May 2019 15:26 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/17 "2019-05-03T15:26:32Z")

</div>

@Colin's suggestion solves my immediate problem. I expect to play a bit more with sdr, so if I find a need to wrap `rtl_sdr` in a contributed node or modify the `rtl_power` node, I will start a new thread. Thanks again for all the help.

---

<div class="post-metadata">

**Author:** ![wb666greene](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/wb666greene/32/6534_2.png) [@wb666greene](https://discourse.nodered.org/u/wb666greene)\
**Post date:** [25 May 2019 15:07 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/18 "2019-05-25T15:07:35Z")

</div>

I'm having a similar issue with ffmpeg in an exec node only sending partial frames when converting rtsp streams into jpeg imags. Looking for CR or any specific character won't work for binary data buffers.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [25 May 2019 18:21 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/19 "2019-05-25T18:21:15Z")

</div>

Do you mean you receive the data as multiple partial frames? So it is all there but split up?

---

<div class="post-metadata">

**Author:** ![wb666greene](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/wb666greene/32/6534_2.png) [@wb666greene](https://discourse.nodered.org/u/wb666greene)\
**Post date:** [25 May 2019 19:50 UTC](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695/20 "2019-05-25T19:50:58Z")

</div>

I'm trying to figure out is this is the case.

[Next page](https://discourse.nodered.org/t/exec-and-daemon-nodes-break-up-a-data-stream/10695.md?page=2)