Join node multiple problems / questions

[bebR]

Hi,

I'm trying to use the join node without having previously split a message into parts.
I first tried to perform a manual combination to a key/value object, with a msg.uid key, which should be unique.
My condition to finish the join is to have exactly 2 messages.
And this does not work.

I simplified my flows to only use some inject nodes to test the join node.
What I get from these tests is:

• only the msg.complete property triggers the join, not the count, which is a problem for me
• if I use a merged object for my result, the count condition works, but only for the first time:
  • the first msg is buffered in the join node
  • the second one trigger the join
  • the third one trigger the join, even if I didn't the 'subsequent message option'

I didn't find any issue on the github repo about these probems.
Should I report a bug?

I use Node-RED v1.0.5, running on a Rpi 3b, with ArchlinuxARM. NodeJS v10.
Even on my WSL (Ubuntu) install, the behavior is the same.

[dceejay]

Can you share your inject based example thanks

[bebR]

Hi @dceejay,

Thanks for your answer.
Here is my test flow:

[{"id":"465bf6d5.382638","type":"join","z":"295d70f2.01c4b","name":"join to key/value obj on 2 msgs","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"2","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":570,"y":600,"wires":[["16b0329b.f87d25"]]},{"id":"16b0329b.f87d25","type":"debug","z":"295d70f2.01c4b","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":770,"y":600,"wires":[]},{"id":"eb31ed8c.becb88","type":"inject","z":"295d70f2.01c4b","name":"","topic":"topic1","payload":"1","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":480,"wires":[["465bf6d5.382638"]]},{"id":"5711a091.217af8","type":"join","z":"295d70f2.01c4b","name":"join to merged obj on 2 msgs","mode":"custom","build":"merged","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"2","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":740,"y":660,"wires":[["5a4d49cb.07e19"]]},{"id":"5a4d49cb.07e19","type":"debug","z":"295d70f2.01c4b","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":950,"y":660,"wires":[]},{"id":"96f344a4.ddd35","type":"inject","z":"295d70f2.01c4b","name":"","topic":"topic1","payload":"2","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":520,"wires":[["465bf6d5.382638"]]},{"id":"4f3f57d5.96d96","type":"inject","z":"295d70f2.01c4b","name":"","topic":"topic1","payload":"3","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":560,"wires":[["31f6a669.031942"]]},{"id":"e42b4639.237c18","type":"inject","z":"295d70f2.01c4b","name":"","topic":"topic2","payload":"4","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":600,"wires":[["465bf6d5.382638"]]},{"id":"640cb8fe.5f5608","type":"inject","z":"295d70f2.01c4b","name":"","topic":"topic1","payload":"{\"first_content\" : 1}","payloadType":"json","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":170,"y":660,"wires":[["5711a091.217af8"]]},{"id":"31f6a669.031942","type":"change","z":"295d70f2.01c4b","name":"","rules":[{"t":"set","p":"complete","pt":"msg","to":"","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":270,"y":560,"wires":[["465bf6d5.382638"]]},{"id":"898a53c0.94985","type":"change","z":"295d70f2.01c4b","name":"","rules":[{"t":"set","p":"complete","pt":"msg","to":"","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":410,"y":740,"wires":[["5711a091.217af8"]]},{"id":"3542d6ea.6c0e3a","type":"inject","z":"295d70f2.01c4b","name":"","topic":"topic1","payload":"{\"second_content\" : 2}","payloadType":"json","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":180,"y":700,"wires":[["5711a091.217af8"]]},{"id":"76c89b43.62d2dc","type":"inject","z":"295d70f2.01c4b","name":"","topic":"topic2","payload":"{\"fourth_content\" : 1}","payloadType":"json","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":180,"y":780,"wires":[["5711a091.217af8"]]},{"id":"ac5f5189.ba88f","type":"inject","z":"295d70f2.01c4b","name":"","topic":"topic1","payload":"{\"third_content\" : 3}","payloadType":"json","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":170,"y":740,"wires":[["898a53c0.94985"]]}]

I actually tested it again this morning. And the join output is triggeres when 2 messages with 2 different topics are received. Is this the way a join node should work?
If yes, how can I join messages based on a unique key?

My use case is to generate a totp code for a device and store the secret key to be able to check the code later.
Workflow is:

1. Ask for a device registration, with a UUID
2. Generate a secret key (msg with secret is hold in a join node, using the UUID for correlating messages)
3. Generate the QRcode
4. The device flashes the QRcode to store it
5. The device generates and sends a TOTP to NodeRED, giving its UUID
6. Join is triggered (2 messages with the same UUID) -> this is my problem
7. The TOTP is checked by generating a TOTP on the server with the stored secret key
8. If codes matches, store the secret and uuid in a database

Thanks again for any hint.

[bakman2]

If you click the join node once and read the information panel: the join node expects certain input:

To automatically join a sequence of messages, they should all have this property set. The split node generates this property but it can be manually created. It has the following properties:

• id - an identifier for the group of messages
• index - the position within the group
• count - the total number of messages in the group
• type - the type of message - string/array/object/buffer
• ch - for a string or buffer, the data used to the split the message as either the string or an array of bytes
• key - for an object, the key of the property this message was created from
• len - the length of each message when split using a fixed length value

To get this information, use the batch node. The batch and split node will provide all the information the join node requires. (thus used in conjunction; batch+join or split+join)

Example flow

[{"id":"d955a21.d0779e","type":"inject","z":"491d28d8.4552f","name":"","topic":"topic1","payload":"1","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":156,"y":240,"wires":[["641565e5.dcbdcc"]]},{"id":"ffea8058.3ee34","type":"inject","z":"491d28d8.4552f","name":"","topic":"topic1","payload":"2","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":156,"y":280,"wires":[["641565e5.dcbdcc"]]},{"id":"42dce0eb.76814","type":"inject","z":"491d28d8.4552f","name":"","topic":"topic1","payload":"3","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":156,"y":320,"wires":[["641565e5.dcbdcc"]]},{"id":"74cdebde.6e4ecc","type":"inject","z":"491d28d8.4552f","name":"","topic":"topic2","payload":"4","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":156,"y":360,"wires":[["641565e5.dcbdcc"]]},{"id":"c867fd2e.d40868","type":"inject","z":"491d28d8.4552f","name":"","topic":"topic1","payload":"{\"first_content\" : 1}","payloadType":"json","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":206,"y":432,"wires":[["4c118f71.2a3608"]]},{"id":"12a7b576.f194bb","type":"inject","z":"491d28d8.4552f","name":"","topic":"topic1","payload":"{\"second_content\" : 2}","payloadType":"json","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":216,"y":480,"wires":[["4c118f71.2a3608"]]},{"id":"dad5e33f.3c9598","type":"inject","z":"491d28d8.4552f","name":"","topic":"topic2","payload":"{\"fourth_content\" : 1}","payloadType":"json","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":216,"y":576,"wires":[["4c118f71.2a3608"]]},{"id":"870a4831.4855e8","type":"inject","z":"491d28d8.4552f","name":"","topic":"topic1","payload":"{\"third_content\" : 3}","payloadType":"json","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":206,"y":528,"wires":[["4c118f71.2a3608"]]},{"id":"ac759afd.2b1068","type":"join","z":"491d28d8.4552f","name":"","mode":"custom","build":"array","property":"","propertyType":"full","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"2","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":602,"y":288,"wires":[["e5ba804d.2b5ab"]]},{"id":"641565e5.dcbdcc","type":"batch","z":"491d28d8.4552f","name":"","mode":"count","count":"2","overlap":0,"interval":10,"allowEmptySequence":false,"topics":[],"x":482,"y":288,"wires":[["ac759afd.2b1068"]]},{"id":"6fcc788e.cb314","type":"join","z":"491d28d8.4552f","name":"","mode":"custom","build":"array","property":"","propertyType":"full","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"2","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":602,"y":480,"wires":[["e5ba804d.2b5ab"]]},{"id":"4c118f71.2a3608","type":"batch","z":"491d28d8.4552f","name":"","mode":"count","count":"2","overlap":0,"interval":10,"allowEmptySequence":false,"topics":[],"x":482,"y":480,"wires":[["6fcc788e.cb314"]]},{"id":"e5ba804d.2b5ab","type":"debug","z":"491d28d8.4552f","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":794,"y":360,"wires":[]}]

[dceejay]

So yes the node is working as designed... to join/merge things that have n different "keys".

for your scenario I would create my own function...
To start I would just save the key in flow context (via change node) - using the uuid as the key.
Then after qr generation a small function to read the flow context using then relevant key, do whatever combining I needed to do and send on to the database.

[bebR]

Ok, thanks for your explanation @dceejay.
Now I understand why it wasn't working.
I'm used to use EIP patterns at work, and I thought the Node-RED's join node would work like the aggregate pattern: https://camel.apache.org/components/latest/eips/aggregate-eip.html
Here you can see that the correlationExpression is required to aggregate messages together.
I think this would be nice to have such a node in Node-RED, this would avoid storing data in the flow context. I may propose this feature on github. What do you think?

EDIT: I found this library which might do what I want (https://flows.nodered.org/node/node-red-contrib-combine), I'll check it out.

[dceejay]

Well the join node does have a reduce mode

which can probably do it... but you need to be up on JSONata skills as well to work it out...
(I'm not - though the info side bar gives some examples.

[bebR]

This is a good idea, I'll try it a try.
But I'm also a not very skilled in JSONata
I'll report here if I find a solution (I may change the topic to match my use case more precisely)

Thanks again for your time and your anwers!