Split should send complete message once all has been split

General
21

That's not the answer :wink: Question is, what is a good practical workaround - for example doing the final join after the computation (as the flow solution above).

That won't work if messages are dropped so that your idea of

Which would fix that problem too.

So basically moving the join that generates the final complete message as close as possible to the joins is one way of reducing the risk of failure.

But yes, there is no guarantee that some message aren't slowed down by external dark matter radiation.

22

I don't think there is a perfect general solution for all cases - but I think there can be correct solutions for most individual cases. We do have mitigations in place to try to help - eg the array ordering as per I mentioned previously. Also the switch node has an option to recreate sequences - which can be used to "patch up" the parts as they are switched/filtered - so that they can be rejoined later. eg see this example using your scenario above.

[{"id":"bc5f86db20063fba","type":"function","z":"8f7d7811a9253b8f","name":"drop even numbers","func":"if (msg.payload % 2 === 0) { msg.payload = null }\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":452.5,"y":512.5,"wires":[["d934cee75db61d51"]]},{"id":"a32fc602b0ecc67e","type":"delay","z":"8f7d7811a9253b8f","name":"","pauseType":"random","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":390,"y":412.5,"wires":[["4d3b2998800ee242","bc5f86db20063fba"]]},{"id":"d934cee75db61d51","type":"join","z":"8f7d7811a9253b8f","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","useparts":false,"accumulate":"false","timeout":"","count":"","reduceRight":false,"x":532.5,"y":562.5,"wires":[["12347b7057460c89"]]},{"id":"0984ac1bb6342f05","type":"switch","z":"8f7d7811a9253b8f","name":"swicth > 4","property":"payload","propertyType":"msg","rules":[{"t":"gt","v":"4","vt":"num"},{"t":"lte","v":"4","vt":"num"}],"checkall":"true","repair":true,"outputs":2,"x":225,"y":437.5,"wires":[["a32fc602b0ecc67e"],["502f5bf797b3c220"]]},{"id":"4d3b2998800ee242","type":"debug","z":"8f7d7811a9253b8f","name":"debug 1","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":565,"y":400,"wires":[]},{"id":"12347b7057460c89","type":"debug","z":"8f7d7811a9253b8f","name":"debug 2","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":692.5,"y":562.5,"wires":[]},{"id":"94461abdba5c5882","type":"split","z":"8f7d7811a9253b8f","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","property":"payload","x":417.5,"y":325,"wires":[["0984ac1bb6342f05"]]},{"id":"502f5bf797b3c220","type":"delay","z":"8f7d7811a9253b8f","name":"","pauseType":"random","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":397.5,"y":462.5,"wires":[["5af83a479eae5acb"]]},{"id":"034e58bbf2a5d185","type":"inject","z":"8f7d7811a9253b8f","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[0,1,2,3,4,5,6,7,8,9]","payloadType":"json","x":232.5,"y":325,"wires":[["94461abdba5c5882"]]},{"id":"5af83a479eae5acb","type":"join","z":"8f7d7811a9253b8f","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","useparts":false,"accumulate":"false","timeout":"","count":"","reduceRight":false,"x":557.5,"y":462.5,"wires":[["30a8fab04feeaa50"]]},{"id":"30a8fab04feeaa50","type":"debug","z":"8f7d7811a9253b8f","name":"debug 3","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":722.5,"y":462.5,"wires":[]}]

But yes - can be very easy to break - which is why I don't think there is a general answer for all situations.

23

Wow, ok that's nice feature -

Screenshot 2026-01-03 at 10.54.40

Never noticed/understood before what that does.

Ok, so it uses the message count and index values on the split to know when a sequence comes to an end and only then does it send out it's own messages. In your example, both branches after the filter(switch node) have correct counts on the parts attribute on the message.

That then solves the split-filter-collect-without-dropped-messages use-case. Because each branch after the switch node has it's own collection of messages upon which it can join.

Nice!

If I was still writing my blog, I would do a write up on that! --> FlowHub: [Layout Learnings] Split Filter Collect flow

1 Like
24

If you add in Colin's suggestion - and rather than drop messages completely, replace them with nulls if some condition occurs - then you can easily re-filter after re-joining them. eg as below - which is one way to work around the split-filter-collect-WITH-dropped-messages use-case

[{"id":"c295fef495b72c5e","type":"function","z":"8f7d7811a9253b8f","name":"finally filter out null returns","func":"msg.payload = msg.payload.filter((val) => val !== null);\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":380,"y":600,"wires":[["12347b7057460c89"]]},{"id":"d934cee75db61d51","type":"join","z":"8f7d7811a9253b8f","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","useparts":false,"accumulate":"false","timeout":"","count":"","reduceRight":false,"x":562.5,"y":525,"wires":[["c295fef495b72c5e"]]},{"id":"12347b7057460c89","type":"debug","z":"8f7d7811a9253b8f","name":"debug 2","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":615,"y":600,"wires":[]},{"id":"bc5f86db20063fba","type":"function","z":"8f7d7811a9253b8f","name":"drop even numbers","func":"if (msg.payload % 2 === 0) { msg.payload = null }\nreturn msg;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":360,"y":525,"wires":[["d934cee75db61d51"]]},{"id":"a32fc602b0ecc67e","type":"delay","z":"8f7d7811a9253b8f","name":"","pauseType":"random","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":335,"y":462.5,"wires":[["4d3b2998800ee242","bc5f86db20063fba"]]},{"id":"0984ac1bb6342f05","type":"switch","z":"8f7d7811a9253b8f","name":"swicth > 4","property":"payload","propertyType":"msg","rules":[{"t":"lte","v":"4","vt":"num"},{"t":"gt","v":"4","vt":"num"}],"checkall":"true","repair":true,"outputs":2,"x":162,"y":445.5,"wires":[["502f5bf797b3c220"],["a32fc602b0ecc67e"]]},{"id":"4d3b2998800ee242","type":"debug","z":"8f7d7811a9253b8f","name":"debug 1","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":510,"y":462.5,"wires":[]},{"id":"94461abdba5c5882","type":"split","z":"8f7d7811a9253b8f","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","property":"payload","x":354.5,"y":333,"wires":[["0984ac1bb6342f05"]]},{"id":"502f5bf797b3c220","type":"delay","z":"8f7d7811a9253b8f","name":"","pauseType":"random","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":335,"y":412.5,"wires":[["5af83a479eae5acb"]]},{"id":"034e58bbf2a5d185","type":"inject","z":"8f7d7811a9253b8f","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[0,1,2,3,4,5,6,7,8,9]","payloadType":"json","x":169.5,"y":333,"wires":[["94461abdba5c5882"]]},{"id":"5af83a479eae5acb","type":"join","z":"8f7d7811a9253b8f","name":"","mode":"auto","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","useparts":false,"accumulate":"false","timeout":"","count":"","reduceRight":false,"x":500,"y":412.5,"wires":[["30a8fab04feeaa50"]]},{"id":"30a8fab04feeaa50","type":"debug","z":"8f7d7811a9253b8f","name":"debug 3","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":660,"y":412.5,"wires":[]}]
1 Like
25

If there is nothing asynchronous in the intervening flows then in fact the flow is deterministic, so external dark matter radiation will not slow them down. The strategy used is documented somewhere but I can't find it. If I remember correctly, at its simplest, when a node is given the processor it services one message from its input queue and then gives up the processor. Every other node will be given the processor before the first node gets it again.

26

Do you mean this page --> Messaging Hooks : Node-RED that has a diagram describing the message passing?

I want to avoid making assumption about the processing model underlying the flow diagrams. I.e. Erlang-Red being multi-processor/thread so that flows like this are truly depended on message arrival and not when they get processor time.

And also FBP makes no assumptions other than everything is independent of anything else!

27

In that case then the effects of dark matter radiation, not to mention the improbabilities induced by a nice hot cup of tea, may indeed be factors that you must allow for.

28

Probability factor of one to one. We have normality. I repeat, we have normality. Anything you still can't cope with is therefore your own problem.