Store and merge data through join nodes?

I have several flows, with subflows and nest subflows. Any of these flows can have a join (not always with a split). All these flows do lots of different things, mostly in parallel. All the flows eventually join up and when all are done, provides a single response back to the user.

What I need to do is track any errors that occur along the way, consolidate them in the end, and return them to the user.

The problem is when flows split (using the split node or not), the global, flow and node contexts are duplicated, as is the msg object. If I don't do anything to consolidate data, the last flow to enter the join node provides the contexts and msg object. So there is no place to store errors in the flows, that I can consolidate, since msg is being used for actual data, and its very awkward to put errors in there too.

I thought of storing errors in global, or flow but as those are duplicate like msg, and can't merge those, they don't help.

I don't know if this is clear enough. Has anyone needed to do something similar and found a solution?

Code is my attempt to use global context but see that it was duplicated as well.

[
    {
        "id": "c300468d.039678",
        "type": "inject",
        "z": "57ffe4f8.972bcc",
        "name": "",
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "x": 200,
        "y": 200,
        "wires": [
            [
                "fd2a4a0e.fcdba8"
            ]
        ]
    },
    {
        "id": "3957735c.17df5c",
        "type": "change",
        "z": "57ffe4f8.972bcc",
        "name": "",
        "rules": [
            {
                "t": "set",
                "p": "warnings",
                "pt": "global",
                "to": "$append(\t   $globalContext('warnings'),\t   'warning A'\t)",
                "tot": "jsonata"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 630,
        "y": 180,
        "wires": [
            [
                "dedd415a.76faf"
            ]
        ]
    },
    {
        "id": "fd2a4a0e.fcdba8",
        "type": "change",
        "z": "57ffe4f8.972bcc",
        "name": "",
        "rules": [
            {
                "t": "set",
                "p": "warnings",
                "pt": "global",
                "to": "[]",
                "tot": "jsonata"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 390,
        "y": 200,
        "wires": [
            [
                "3957735c.17df5c",
                "8eddde95.b22da",
                "b394f589.28f4f8"
            ]
        ]
    },
    {
        "id": "8eddde95.b22da",
        "type": "change",
        "z": "57ffe4f8.972bcc",
        "name": "",
        "rules": [
            {
                "t": "set",
                "p": "warnings",
                "pt": "global",
                "to": "$append(\t   $globalContext('warnings'),\t   'warning B'\t)",
                "tot": "jsonata"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 630,
        "y": 240,
        "wires": [
            [
                "dedd415a.76faf"
            ]
        ]
    },
    {
        "id": "dedd415a.76faf",
        "type": "join",
        "z": "57ffe4f8.972bcc",
        "name": "",
        "mode": "custom",
        "build": "string",
        "property": "payload",
        "propertyType": "msg",
        "key": "topic",
        "joiner": "\\n",
        "joinerType": "str",
        "accumulate": false,
        "timeout": "",
        "count": "3",
        "reduceRight": false,
        "reduceExp": "",
        "reduceInit": "",
        "reduceInitType": "",
        "reduceFixup": "",
        "x": 840,
        "y": 220,
        "wires": [
            [
                "62e82f73.122ad"
            ]
        ]
    },
    {
        "id": "b394f589.28f4f8",
        "type": "change",
        "z": "57ffe4f8.972bcc",
        "name": "",
        "rules": [
            {
                "t": "set",
                "p": "warnings",
                "pt": "global",
                "to": "$append(\t   $globalContext('warnings'),\t   'warning C'\t)",
                "tot": "jsonata"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 630,
        "y": 300,
        "wires": [
            [
                "dedd415a.76faf"
            ]
        ]
    },
    {
        "id": "62e82f73.122ad",
        "type": "debug",
        "z": "57ffe4f8.972bcc",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "x": 990,
        "y": 220,
        "wires": []
    }
]

After the join, why not add a change node and use it to move global.warnings to msg.warnings?
(in yout test flow change the last debug node to display 'complete msg object')

that doesn't change anything really. The problem is that global.warnings only reflects the last flow values, so the first two flows warnings are lost.

So this is not what you want?
Screen Shot 2020-03-21 at 4.01.46 AM

Yes, that is what I'm aiming at, but the sample code I provided doesn't do that, and even when adding a change node, the global data behaves the same as payload (without a merge). So if I merge payload, which I need to do, I can't merge global data as well, unless I'm misunderstanding what you are proposing? Can you provide the flow that produced the output you posted?

(using Node Red 0.20.7; looking at upgrading to latest version)

Thanks.

the only thing I did to your flow was to add a 'change' node after the join node. In th join node I set msg.warnings to global.warnings and don't forget to change the debug node t display the 'Complete msg object'

If I do what you say, I get this. What am I missing.
image

[
    {
        "id": "aa4fcd3e.de079",
        "type": "inject",
        "z": "57ffe4f8.972bcc",
        "name": "",
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "x": 100,
        "y": 500,
        "wires": [
            [
                "dd2aac06.f544b"
            ]
        ]
    },
    {
        "id": "740c39b1.8a62f8",
        "type": "change",
        "z": "57ffe4f8.972bcc",
        "name": "",
        "rules": [
            {
                "t": "set",
                "p": "warnings",
                "pt": "global",
                "to": "$append(\t   $globalContext('warnings'),\t   'warning A'\t)",
                "tot": "jsonata"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 530,
        "y": 480,
        "wires": [
            [
                "dee59b6e.9bc188"
            ]
        ]
    },
    {
        "id": "dd2aac06.f544b",
        "type": "change",
        "z": "57ffe4f8.972bcc",
        "name": "",
        "rules": [
            {
                "t": "set",
                "p": "warnings",
                "pt": "global",
                "to": "[]",
                "tot": "jsonata"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 290,
        "y": 500,
        "wires": [
            [
                "740c39b1.8a62f8",
                "b4f2503b.f43",
                "877b55db.51dc08"
            ]
        ]
    },
    {
        "id": "b4f2503b.f43",
        "type": "change",
        "z": "57ffe4f8.972bcc",
        "name": "",
        "rules": [
            {
                "t": "set",
                "p": "warnings",
                "pt": "global",
                "to": "$append(\t   $globalContext('warnings'),\t   'warning B'\t)",
                "tot": "jsonata"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 530,
        "y": 540,
        "wires": [
            [
                "dee59b6e.9bc188"
            ]
        ]
    },
    {
        "id": "dee59b6e.9bc188",
        "type": "join",
        "z": "57ffe4f8.972bcc",
        "name": "",
        "mode": "custom",
        "build": "string",
        "property": "payload",
        "propertyType": "msg",
        "key": "topic",
        "joiner": "\\n",
        "joinerType": "str",
        "accumulate": false,
        "timeout": "",
        "count": "3",
        "reduceRight": false,
        "reduceExp": "",
        "reduceInit": "",
        "reduceInitType": "",
        "reduceFixup": "",
        "x": 740,
        "y": 520,
        "wires": [
            [
                "f33f48fb.b5f078"
            ]
        ]
    },
    {
        "id": "877b55db.51dc08",
        "type": "change",
        "z": "57ffe4f8.972bcc",
        "name": "",
        "rules": [
            {
                "t": "set",
                "p": "warnings",
                "pt": "global",
                "to": "$append(\t   $globalContext('warnings'),\t   'warning C'\t)",
                "tot": "jsonata"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 530,
        "y": 600,
        "wires": [
            [
                "dee59b6e.9bc188"
            ]
        ]
    },
    {
        "id": "783ea643.c84988",
        "type": "debug",
        "z": "57ffe4f8.972bcc",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "x": 1110,
        "y": 520,
        "wires": []
    },
    {
        "id": "f33f48fb.b5f078",
        "type": "change",
        "z": "57ffe4f8.972bcc",
        "name": "",
        "rules": [
            {
                "t": "set",
                "p": "warnings",
                "pt": "msg",
                "to": "$globalContext('warnings')",
                "tot": "jsonata"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 910,
        "y": 520,
        "wires": [
            [
                "783ea643.c84988"
            ]
        ]
    }
]

Your last flow works fine for me, however, I am on v1.0.4 so I would try upgrading.

Yah I suspected maybe it's a version difference. I may accelerate my upgrade path. Thanks for your help.

I upgraded to 1.0.4 and get the same result as before, with only 1 entry in the array msg.warnings. Could there be something else accounting for the difference other than the Node Red version?
I checked and now running Node 10.
I'm running Node Red in a docker container.

check to see if all your nodes are up to date. Go to the palette manager and scroll down and see if any nodes need updating:
Screen Shot 2020-03-25 at 10.24.09 AM

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.