Split messages arriving out of order

General
1

As part of my Advent of Code efforts, I'm having to be creative in splitting messages

In Day 3, I need to split a message into two equal parts so I'm using a switch to add msg.complete after half the number of messages have passed

However, I'm finding that I have to stick a superfluous change node in as well otherwise the messages arrive out of order.

Can someone explain why this is happening without the extra change node - is it just timiing? And if so - is there a better way of ensuring correct order of messages arriving at the join?

image
image1191Ă—209 15.5 KB 

[{"id":"273b95be5f0a745f","type":"tab","label":"Flow 1","disabled":false,"info":"","env":[]},{"id":"b41a0bea9b144b3a","type":"junction","z":"273b95be5f0a745f","x":840,"y":180,"wires":[["86e060fbfa932ea5","0dc5f6c75011bcbf","7901ee4977a08814"]]},{"id":"1dd4fa914ea28353","type":"split","z":"273b95be5f0a745f","name":"","splt":" 1","spltType":"len","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"topic","x":270,"y":160,"wires":[["8cfe1f4b84945ba0"]]},{"id":"8cfe1f4b84945ba0","type":"switch","z":"273b95be5f0a745f","name":"","property":"parts.index","propertyType":"msg","rules":[{"t":"eq","v":"index1","vt":"msg"},{"t":"eq","v":"index2","vt":"msg"},{"t":"else"}],"checkall":"false","repair":false,"outputs":3,"x":410,"y":160,"wires":[["8194868092e2cd12"],["8194868092e2cd12"],["dfec8efbf4635efc"]]},{"id":"8194868092e2cd12","type":"change","z":"273b95be5f0a745f","name":"","rules":[{"t":"set","p":"complete","pt":"msg","to":"true","tot":"bool","dc":true}],"action":"","property":"","from":"","to":"","reg":false,"x":650,"y":140,"wires":[["b41a0bea9b144b3a"]]},{"id":"86e060fbfa932ea5","type":"debug","z":"273b95be5f0a745f","name":"complete msg object","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1080,"y":120,"wires":[]},{"id":"0dc5f6c75011bcbf","type":"debug","z":"273b95be5f0a745f","name":"msg.parts.index","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"parts.index","targetType":"msg","statusVal":"","statusType":"auto","x":1060,"y":240,"wires":[]},{"id":"8fe9703b7e8a1271","type":"inject","z":"273b95be5f0a745f","name":"","props":[{"p":"payload"},{"p":"index1","v":"2","vt":"num"},{"p":"index2","v":"5","vt":"num"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"abcdef","payloadType":"str","x":110,"y":160,"wires":[["1dd4fa914ea28353"]]},{"id":"dfec8efbf4635efc","type":"change","z":"273b95be5f0a745f","name":"","rules":[{"t":"delete","p":"complete","pt":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":660,"y":200,"wires":[["b41a0bea9b144b3a"]]},{"id":"7901ee4977a08814","type":"join","z":"273b95be5f0a745f","name":"","mode":"custom","build":"array","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":1030,"y":180,"wires":[["0eef0a3a2ae1e8f9"]]},{"id":"0eef0a3a2ae1e8f9","type":"debug","z":"273b95be5f0a745f","name":"split messages","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1220,"y":180,"wires":[]}]
2

I'm confused. To me it looks like your split doesn't do anything. Since there is an index1 and index2 in every split msg.

3

The split splits abcdef to be joined up into two arrays abc and def

This is a simple cut-down example of Advent of Code solution showing the issue

The index1 and index2 are variable in real life as the messages are not the same length - I've just hard coded them to 2 and 5 for this example

4

can you show us the output you see (and the matching flow) when you get the out of order condition?

The flow you posted gives me consistent results...

image

5

Sorry :slight_smile:
The flow give the correct result with the superfluous delete msg.complete - I would like to understand why its needed :slight_smile: And maybe a better way of doing it as I feel I'm relying on timing rather than logic :slight_smile:

image
image1686Ă—458 33.1 KB

6

But that isn't what your split node is actually doing. Since every message contains both properties, it cannot be split switched correctly.

Do you have a link for that?

There is a much simpler switch that will do exactly what you want with no need for a complete message. Let me know if you want a hint.

7

image
image776Ă—204 15.7 KB

Watch the animations in this blog post...

8

Your concentrating on the wrong part of the flow - that's just test data - don't worry about it :slight_smile:

I'm all ears (as long as not using JS or JSONata :slight_smile: )

9

I knew it was something like that but didn't know exactly what the sequences were :slight_smile:
So, as long as I've an equal number of nodes in the paths, then they will arrive at the join in order:)

Happy bunny now - thanks :slight_smile:

10

Relying on that is a bit dodgy, even if in practice it does work. It would be better to find a way that works without relying on subtleties of the underlying system.

11

Well that is true but if NR always works that way (or can be always set to work that way in future) then I can live with it - certainly for now :slight_smile:

I was more worried that it was a timing issue and the time taken for a node to do stuff that was important - glad its not :slight_smile:

12

I knew you wouldn't want that. I thought at first you wanted to split every other element of the string - that is slightly more complex. To simply split the first x entries in to 1 output and the rest into another, you need to look further into the msg properties that are added by the split node. One of them contains a number and that is all you need. :wink:

Happy to give more clues but I'm guessing you would prefer to work it out yourself?

13

I can't find a method of telling a split node dynamically how to split - maybe I can't see it in the docs????

14

Just for info you can switch and reconstruct sequence, while setting msg.parts.type to array, and join automatically
e.g.

[{"id":"8fe9703b7e8a1271","type":"inject","z":"273b95be5f0a745f","name":"","props":[{"p":"payload"},{"p":"index1","v":"2","vt":"num"},{"p":"index2","v":"5","vt":"num"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"abcdef","payloadType":"str","x":290,"y":120,"wires":[["1dd4fa914ea28353"]]},{"id":"1dd4fa914ea28353","type":"split","z":"273b95be5f0a745f","name":"","splt":" 1","spltType":"len","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"topic","x":350,"y":160,"wires":[["db41a23d5301370d"]]},{"id":"db41a23d5301370d","type":"change","z":"273b95be5f0a745f","name":"","rules":[{"t":"set","p":"parts.type","pt":"msg","to":"array","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":510,"y":160,"wires":[["8cfe1f4b84945ba0"]]},{"id":"8cfe1f4b84945ba0","type":"switch","z":"273b95be5f0a745f","name":"","property":"parts.index","propertyType":"msg","rules":[{"t":"lte","v":"index1","vt":"msg"},{"t":"gt","v":"index1","vt":"msg"}],"checkall":"true","repair":true,"outputs":2,"x":670,"y":160,"wires":[["b41a0bea9b144b3a"],["b41a0bea9b144b3a"]]},{"id":"b41a0bea9b144b3a","type":"junction","z":"273b95be5f0a745f","x":820,"y":160,"wires":[["86e060fbfa932ea5","7901ee4977a08814"]]},{"id":"86e060fbfa932ea5","type":"debug","z":"273b95be5f0a745f","name":"complete msg object","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1080,"y":120,"wires":[]},{"id":"7901ee4977a08814","type":"join","z":"273b95be5f0a745f","name":"","mode":"auto","build":"array","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":970,"y":160,"wires":[["0eef0a3a2ae1e8f9"]]},{"id":"0eef0a3a2ae1e8f9","type":"debug","z":"273b95be5f0a745f","name":"split messages","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1140,"y":160,"wires":[]}]

chrome_screenshot_1670156427876
chrome_screenshot_16701564278762048Ă—293 97.2 KB

1 Like
15

ooh - I see what you did there - very nice

image
image841Ă—240 12.4 KB

coupled with the slight "cheating" with setting it to array type :slight_smile:

16

image
image727Ă—492 16.1 KB

Example Flow 
[{"id":"1dd4fa914ea28353","type":"split","z":"273b95be5f0a745f","name":"","splt":" 1","spltType":"len","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"topic","x":290,"y":400,"wires":[["8a030eb6ba662cbb"]]},{"id":"8fe9703b7e8a1271","type":"inject","z":"273b95be5f0a745f","name":"","props":[{"p":"payload"},{"p":"index1","v":"2","vt":"num"},{"p":"index2","v":"5","vt":"num"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"abcdef","payloadType":"str","x":170,"y":400,"wires":[["1dd4fa914ea28353"]]},{"id":"8a030eb6ba662cbb","type":"switch","z":"273b95be5f0a745f","name":"","property":"parts.index","propertyType":"msg","rules":[{"t":"lte","v":"3","vt":"num"},{"t":"else"}],"checkall":"false","repair":false,"outputs":2,"x":410,"y":400,"wires":[["4ad6de13a9c84c5e"],["0806269d72e09fa7"]]},{"id":"4ad6de13a9c84c5e","type":"debug","z":"273b95be5f0a745f","name":"debug 29","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":560,"y":380,"wires":[]},{"id":"0806269d72e09fa7","type":"debug","z":"273b95be5f0a745f","name":"debug 30","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":560,"y":420,"wires":[]}]
17

I don't think me and you are on the same sheet of paper with this one :slight_smile:

18

You said you wanted abc in one output and def in the other. To do that, you need to split as normal and then switch on the msg.parts.index value so that any index <4 is output to the top output and everything else goes to the bottom output. Then you can join each one and job's done.

1 Like
19

Incidentally, the answer to yesterday's challange is much easier in code than it is in nodes. :slight_smile:

20

I agree (can be done with a handful of lines) but I have renewed appreciation for Simon's galient efforts to avoid code. It certainly does highlight areas of improvement for low code nodes.

For example, today's challenge would have been super simple with a good "string" node.

3 Likes