The emerge node, built via Node-Maker

[Steveorevo]

Created a simple but essential node using my Node-Maker side project. Often times I need to submit a msg to a node with various properties present and filled out; these properties may come from various asynchronous and parallel flow segments. Counting messages and waiting to gather all these values into a single msg can be a burden. The emerge node can simplify that burden by combining msgs until all the needed properties are present and/or meet their pre-defined requirements before a single, new, msg "emerges" (is sent) along its merry way.

Built with Node-Maker, this has been a test to see how viable both projects will be in the near future (I've got a lot of flows and nodes to build).

Included in the repo you'll find node-maker.json, a flow file that created the emerge node without having to venture far from Node-RED itself.

@steveorevo/node-red-emerge

[Colin]

Did you consider submitting, initially, a feature request to enhance the join node for this? I can see that something like this could be a useful enhancement. If it met with approval then you could implement that, rather than a new node.

[Steveorevo]

I did consider this. However, there are key reasons why a plugin node was a better route, even if it becomes apart of the join node or added to core itself:

• first off, I needed it (now)
• isolates changes
• gauges community necessity
• provides beta testing route
• opportunity for user feedback (before core)

I contribute to a number of open source projects; both large and small. For a project like Node-RED, with a proper plugin architecture, it’s ideal to implement features as plugins and then evaluate if they should become apart of core. Some commercial products do this (unfortunately at the expense of app developers; see “getting flux’d” in iOS, which “stole” night mode). But that’s no matter here because this is open source and I’m all for taking and modifying (attribution credit is welcome)!

I’m apart of the WordPress community and it’s become popular to rollout features as a separate add-on first that we call it Canonical Plugins.

This gives the opportunity for, say Nick and Dave to see how folks are using the project before making major changes to it. They can also evaluate user feedback, issues, and comments on the open source repo for it. Actually, very early in Node-RED development I created a credentials node and Nick bounced the idea of making such a feature apart of core; I think that’s how we have a credentials type input in subflows, a very cool feature.

So yes, the thought crossed my mind but it was faster to introduce via share-your-work a, “canonical node”. If it proves popular, Node-RED is MORE THAN WELCOME to take it, use it or modify it, or include it in the core palette.

[Colin]

I was thinking more with a view to discussing the spec and coming up with a proposal that would be of most benefit to most people and that would be most compatible with the core node.

@Steveorevo

Node-RED is MORE THAN WELCOME to take it,

I was thinking more of a contribution to the core.

[Steveorevo]

I think this is the process you’re referring to. I’ve opened discussion in core to facilitate feature requests before, but I’m 50/50 on success; the failures don’t get any replies. But I think it’s probably because the slack channel is a little stale? Seems like the forums is where it’s at now.

Always open to feedback. Thank you for your input!

[dceejay]

Yes, discussions much easier ( and long lasting) here on the forum. Having had a quick look at the node I can see that while it is an obvious candidate for a new mode in the join node, it is also somewhat different, such that it isn’t obvious ( to me) how to just drop it in, so yes happy to see it exist, and at same time discuss how it could be joined into join.

[cymplecy]

I just tried this node out and it didn't seem to work for me
I want to wait until 3 messages with different topics have arrived and then output but its not working for me
What have I got wrong?

[{"id":"c999bddccf728078","type":"emerge","z":"a5b78a370366734b","name":"","grpRules":[{"ruleProperty":"topic","rulePropertyType":"msg","ruleOperation":"==","ruleValue":"red","ruleValueType":"str"},{"ruleProperty":"topic","rulePropertyType":"msg","ruleOperation":"==","ruleValue":"green","ruleValueType":"str"},{"ruleProperty":"topic","rulePropertyType":"msg","ruleOperation":"==","ruleValue":"blue","ruleValueType":"str"}],"ruleProperty":"payload","rulePropertyType":"msg","ruleOperation":"==","ruleValue":"","ruleValueType":"str","rulesTimeout":30,"timeoutAction":"0","x":310,"y":290,"wires":[["041a3ab2449d6513"]]},{"id":"041a3ab2449d6513","type":"debug","z":"a5b78a370366734b","name":"debug 23","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":500,"y":290,"wires":[]},{"id":"9d5e796efacbc1b6","type":"inject","z":"a5b78a370366734b","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"red","payload":"","payloadType":"date","x":100,"y":270,"wires":[["c999bddccf728078"]]},{"id":"988004d53c37c84c","type":"inject","z":"a5b78a370366734b","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"green","payload":"","payloadType":"date","x":110,"y":320,"wires":[["c999bddccf728078"]]},{"id":"541d960381417cb1","type":"inject","z":"a5b78a370366734b","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"blue","payload":"","payloadType":"date","x":110,"y":380,"wires":[["c999bddccf728078"]]}]

[Colin]

I can't get it to do anything either.

[Steveorevo]

Just updated to version 1.1.2, included is an example flow. Please post your minimal flow if you have encountered an issue and I can cross-check it with the current version.

[Steveorevo]

Hi cymplecy, there seems to be some confusion as you're setting the topic three times. It's not possible for all those conditions to be true in order for the new message to emerge. In the example, the emerge node waits until three messages arrive:

1. msg.first_name
2. msg.last_name
3. msg.occupation

Only until all properties are present (because the msgs are merged) does a new message output containing:

{
    first_name: "Jack",
    last_name: "Burton",
    occupation: "truck driver"
}

Each of the properties have a different name and therefore, can eventually manifest as "not empty" in the rules.

Please let me know if that clarifies the issue or if you need further assistance.

[cymplecy]

There is confusion
I don't think this node does what I thought it did

Never mind

[Colin]

I was also confused about what it did. I thought that @cymplecy's example would merge messages until it had received topic values equal to the values specified, and would provide a merge of all the other properties of the message. Perhaps you need to add to the help text to clarify that this is not correct. Also perhaps it should not allow the same property to be specified twice in that way.

The following also doesn't work for me, I don't know why.

[{"id":"0a5d142fe493b32a","type":"emerge","z":"c4b3f62035982a12","name":"wait for all user info","grpRules":[{"ruleProperty":"payload.first_name","rulePropertyType":"msg","ruleOperation":"!empty","ruleValue":"","ruleValueType":"str"},{"ruleProperty":"payload.last_name","rulePropertyType":"msg","ruleOperation":"!empty","ruleValue":"","ruleValueType":"str"},{"ruleProperty":"payload.occupation","rulePropertyType":"msg","ruleOperation":"!empty","ruleValue":"","ruleValueType":"str"}],"ruleProperty":"payload","rulePropertyType":"msg","ruleOperation":"==","ruleValue":"","ruleValueType":"str","rulesTimeout":30,"timeoutAction":"1","x":310,"y":100,"wires":[["4c5d7bca520c0054"]]},{"id":"621efb1c34f39daa","type":"inject","z":"c4b3f62035982a12","name":"first_name","props":[{"p":"payload.first_name","v":"Jack","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":100,"y":40,"wires":[["0a5d142fe493b32a"]]},{"id":"cb8c62f1de3d970e","type":"inject","z":"c4b3f62035982a12","name":"last_name","props":[{"p":"payload.last_name","v":"Burton","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":100,"y":100,"wires":[["0a5d142fe493b32a"]]},{"id":"5683ab4dc09a100f","type":"inject","z":"c4b3f62035982a12","name":"occupation","props":[{"p":"payload.occupation","v":"truck driver","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":100,"y":160,"wires":[["0a5d142fe493b32a"]]},{"id":"4c5d7bca520c0054","type":"debug","z":"c4b3f62035982a12","name":"debug 1","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":500,"y":100,"wires":[]}]

[Steveorevo]

Your example is a perfect bug of what it's failing to do! Thank you for your feedback. The problem there is that it's doing a shallow merge; that is to say msg.payload is getting overwritten with a single property each time. Therefore the internal buffer is only containing msg.payload.first_name or msg.payload.last_name but not (Schrödinger's cat aside) both at the same time.

Had it been (sans payload) msg.first_name and msg.last_name that would have worked.

I've now implemented a deep merge routine that should provide the expected results and will now work with your example. Please see node-red-emerge version 1.2.0.

Thank you for your feedback!