Handling split/join node logic when "losing" split msgs is expected

General
1

Use-case: I am split'ing an array of recordIds to make sequential API requests (YouTube endpoint) and then evaluating the JSON responses, specifically determining if an array has any items in it. If the array is empty, then I "drop" that msg containing the response with the empty array (there's no further action my flow can perform on it

Problem: as you can begin to intuit, dropping msgs becomes a problem when attempting to aggregate them together again using the join node which in this case is not propagating the msgs as expected because the node is seeing that not all parts for that sequence/runtime are present

Questions:

  1. I am considering using the flow.get() state to effectively track how many items are in the 'macro' array of channels that I am querying, and then subtract from that array for any msgs that are dropped to determine if the msg.complete should be assigned to a specific msg - am I overthinking this?
  2. I'm aggregating these values so I can iterate thru them and perform batch updates to my datastore. I wonder if I should just scrap the join node and shoot them all off one by one but I see this becoming a massive scaling/volume problem over time - are there any other patterns I should be considering?

Edit: I read this post from earlier this year and tried to follow through with @knolleary's suggestion of using a switch node with the preserve sequence checkbox checked and unchecked, but neither approach had the expected outcome I'm looking for.

2

Is msg.parts making it to the join node from the split node?

3

yep! But keep in mind, msg.parts is a property on each msg, so each msg's part property would have to be evaluated against some kind of 'other' state -- at least that's how I currently understand it

edit: Clarifying my point above: if one or more msgs doesn't make it to the join node, then the way I understand it is that there is an incomplete number of parts and so all the msgs are "stuck" in the join node until I manually send a msg.complete (true bool) msg to release them all and propagate to the next node in the flow

4

So-o...

I would put a node in there - somewhere, sorry haven't done too much home work on this exact scenario - and create a msg.something that holds the total number.

Then sends it to the join node setting how many messages are needed.
(Sorry, this is q quick reply.)

5

Yes, but how are you removing the unwanted messages.
Can we see an example flow?
If you are using a switch node with "recreate sequence" msg.parts is adjusted for missing parts and the join node then receives the adjusted msg.parts payload.
e.g.

[{"id":"d8cf3f78.d1f4c","type":"join","z":"b779de97.b1b46","name":"","mode":"auto","build":"string","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":610,"y":3060,"wires":[["71ce21dc.3f43a"]]},{"id":"20bc118e.c906f6","type":"switch","z":"b779de97.b1b46","name":"","property":"payload","propertyType":"msg","rules":[{"t":"eq","v":"1","vt":"num"}],"checkall":"true","repair":true,"outputs":1,"x":460,"y":3060,"wires":[["d8cf3f78.d1f4c"]]},{"id":"466b9d1c.13d084","type":"split","z":"b779de97.b1b46","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":270,"y":3000,"wires":[["20bc118e.c906f6"]]},{"id":"a312c696.a5b76","type":"inject","z":"b779de97.b1b46","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[1,2,1,1,1,2,1,1,2,1]","payloadType":"json","x":170,"y":3080,"wires":[["466b9d1c.13d084"]]},{"id":"71ce21dc.3f43a","type":"debug","z":"b779de97.b1b46","name":"","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"true","targetType":"full","statusVal":"responseUrl","statusType":"msg","x":620,"y":3300,"wires":[]}]
1 Like
6

That's interesting, I tried the switch node with the create sequence option enabled and it did not impact the outcome, where I still had a bunch of msgs 'stuck' in the join node.

I might try this again on a separate mock flow, but I solved the problem by just sending all msgs to the join node, and handling which data I wrote to the msg properties

7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.