Use the value of a msg or a variable as the number of messages "parts" - join node in manual mode

[aguida79]

Hi people,

I have a simple question, but didn't find the answer searching alongside Internet:
Is there a way to use a the value of a msg or a variable as the messages number of "parts" on join node in manual mode?

I tried to use a number value from the previous msg, or set that number as a flow variable and use it, but didn't work.
The idea is to use the maxColumns value as the message parts number:

Does anybody knows how to do it?

Thanks in advance.
Regards,
Alejandro

[Steve-Mcl]

Are you using the join node in connection with the spilt node?

If yes,
the you simply set the join node to automatic mode.

If no,
simply set msg.complete to true on the last message.

Alternatively,
if (for example) this flow runs once per minute, then set timeout to a suitable value e.g. 30 (sec)

[aguida79]

Thanks @Steve-Mcl for your answer.

I know what you are telling, but what I want to know is if there is any way to use the number value of a previous node msg or a flow variable, as the messages number of "parts" on join node in manual mode.
I need that because the number of parts is variable, but the information is part of the message itself that I'm receiving.

Thanks in advance.
Regards,
Alejandro

[jbudd]

Hi Alejandro.

1. Do you have control over the structure of your input messages?
   If so, get rid of the dots in the property names "qvs.getMatrixDataResponse" and "qvs.return". These names only complicate your problem.

2. In the case where maxColumns is 13:
   2a Do you get 13 completely seperate messages, each containing a single property to add to your output array?
   2b Does every one of these messages include the property maxColumns = 13?

2c Or are the 13 properties to add to your array contained in the "rows" array?

3. How often do this group of (perhaps) 13 messages arrive?
   What interval is there between two groups of messages?
   Can these timings be exploited to get the join node to timeout?

4. Why is @Steve-Mcl's suggestion of sending msg.complete after sending maxColumns messages not right for you?

[Steve-Mcl]

This would be far simpler to advise on if you provide the right level of information.

Do you split before join?
How do you get this stream of messages?

Can you share a minimal flow (use inject nodes & change/function nodes to simulate this data - remember, we do not have access to your data/database)?

Lastly, regarding msg.complete, you can simply add .complete to the msg in a node BEFORE the join, based on any dynamic property you desire (including your "number of parts is variable")

[aguida79]

Hi @jbudd, thanks for your answer.

My answers for your questions:

1. Do you have control over the structure of your input messages?

No, it is the HORRIBLE output of the ArcSight ESM API, and what I did with NodeRED is translate that output from this:

To this:

2a. Do you get 13 completely seperate messages, each containing a single property to add to your output array?

Yes

2b. Does every one of these messages include the property maxColumns = 13?

No, but I can add it if it is needed, modifying the function that I did

2c. Or are the 13 properties to add to your array contained in the "rows" array?

No

3. How often do this group of (perhaps) 13 messages arrive? What interval is there between two groups of messages? Can these timings be exploited to get the join node to timeout?

I'm receiving 13 messages or a multiple of 13 messages, one after the other. The idea is that in this case are 13, but in other runnings of the workflow can be a different amount, and I want to use the join node to join that fixed number of messages every time, obtaining that number from maxColumns msg that I got before I used a function to transform the output of the API in something better. There is no interval between two groups of messages, is one message after the other when the workflow run, so I cannot exploit these timings to get the join.

4. Why is @Steve-Mcl's suggestion of sending msg.complete after sending maxColumns messages not right for you?

Maybe can be an option, but in that case I need to modify my function that is transforming the output, and what I want to know is if there is a simple way to use the value of maxColumns in the "parts" of a join node, without modifying my function, like for example put the value of maxColumns in a flow variable when I get it, and use it in the "parts" of a join node in manual mode.

That was my original question, if it is possible to do that
I was not asking how to do it if I cannot use the variable in "parts" of a join node, I don't want to bother anyone, I'm a beginner with NodeRED and Javascript in general.

Really thanks for your answer.
Regards,
Alejandro

[aguida79]

Hi @Steve-Mcl .

Sorry that I didn't provide the right level of information: I'm a beginner with NodeRED and Javascript in general, and it was not clear to me what information you need to help me.

I'm not using the split node to split before join, I'm using a function that I created to format the new output, getting one entry at a time as a message, like "Device Address": "10.10.2.212"

This stream of messages is the result of my function, that process the original output of the ArcSight ESM API, which is horrible, like I said to @jbudd.

I can share a minimal flow and a sample JSON data if you want to help me to figure out how to solve my problem, but at the beginning, I only want to know if it is possible to use the numeric value of a flow variable to fill the "parts" value of a join node in manual mode, that is all.

I know that I can use the msg.complete to close the join, and I'm already using it after, in other function in the same flow, using the method that @jbudd suggested to me (a msg with the total number of rows in all the rows), because the function that created add this information, but in that case is a calc that I'm doing in the function, I don't have it in other place.

In the case of the first function, I'm trying to avoid to add calc to the function, and use the maxColumns number, that is all.

Thanks again for all your help, and sorry If when I tried to explain something it was not clear, english is not my native language.

Regards,
Alejandro

[jbudd]

Thanks for your replies.

SInce maxColumns is not a property of every message, you have to save it somehow. Either in a context variable or added to every message.

I don't think that the "After a number of message parts" field can accept a message property eg msg.maxColumns.
Therefore you need to devise a work-around.

It seems from your reply that time based work-arounds are not possible.

You could count messages as your function passes them to join, and when the count reaches your stored maxColumns value set msg.complete. This is Steve's approach and seems the simplest.

Alternatively, the join node says for automatic mode.

This mode assumes this node is either paired with a split node or the received messages will have a properly configured msg.parts property.

You could experiment with adding a "properly configured" msg.parts to every message.
To find the required structure, send an array through a split node and view the complete output message with a debug.