Analyzing JSON with dynamic number of nodes and updating external systems

Sorry, if I'm asking some basic stuff. I'm just starting with Node-Red and here's my use case:

I take an JSON, which can have a dynamic number of nodes (>= 0), and I'd like to analyse the JSON and update external systems (e.g. via MQTT). The functional background: the JSON has warnings for my city (which can range from severe weather to covid warnings to earth quake or whatever).
all JSON nodes are of the same structure, to give you a feeling, here's a shortened JSON:

{
    "0": {
        "identifier": "DE-BY-A-W083-20200828-000",
        "sender": "DE-BY-A-W083",
        "sent": "2020-08-28T11:00:08+02:00",
        "status": "Actual",
        "msgType": "Update",
        "scope": "Public",
        "code": [
            "1.0",
            "nina"
        ],
		"info": [ "here's much more"]
    },
    "1": {
        "identifier": "DE-NW-BN-SE030-20201014-30-000",
        "sender": "DE-NW-BN-SE030",
        "sent": "2020-10-14T16:35:21+02:00",
        "status": "Actual",
        "msgType": "Update",
        "scope": "Public",
        "code": [
            "1.0",
            "medien_ueberregional",
            "nina"
        ],
		"info": [ "here's much more"]
    },
    "2": {
        "identifier": "DE-BY-M-W060-20201209-000",
        "sender": "DE-BY-M-W060",
        "sent": "2020-12-09T14:35:17+01:00",
        "status": "Actual",
        "msgType": "Alert",
        "scope": "Public",
        "code": [
            "1.0"
        ],
		"info": [ "here's much more"]
    }
}

I'd like to export a bunch of variables (let's say "identifier", "status" and "sent") in dynamic MQTT-topics:

  • my/mqtt/topic/identifier0 => DE-BY-A-W083-20200828-000
  • my/mqtt/topic/identifier1 => DE-NW-BN-SE030-20201014-30-000
  • my/mqtt/topic/identifier2 => DE-BY-M-W060-20201209-000
  • my/mqtt/topic/sent0 => 2020-08-28T11:00:08+02:00
  • ...

I could setup all my variables for "0", then copy them for "1", then copy them for "2" ... up until I am at "10" (which I assume would be the maximum of nodes for my city).
But I guess, there's a more elegant solution on this, perhaps one, which automatically sents "NULL" messages, if the JSON changed, and a node got purged or a new one got added, ...

So, again, please forgive, if I'm asking too basic stuff and just lead me to the right documentation, perhaps I'm just lacking the correct search terms for doing somethin like that.

Thanks!

here is an example. it splits the json, identifies 0,1,2,3 etc.
Then cretes a topic for each pay load and sends to topic
my/mqtt/topic/identifier2 where identifier2 dynamic . the properties you wish to send are filtered in the change node and are currently set to "identifier","sender","sent","status". Just ad and remove properties there.
the mqtt in node is currently sent to receive all topics.

[{"id":"9a4335c6.75c7e8","type":"inject","z":"8d22ae29.7df6d","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"0\":{\"identifier\":\"DE-BY-A-W083-20200828-000\",\"sender\":\"DE-BY-A-W083\",\"sent\":\"2020-08-28T11:00:08+02:00\",\"status\":\"Actual\",\"msgType\":\"Update\",\"scope\":\"Public\",\"code\":[\"1.0\",\"nina\"],\"info\":[\"here's much more\"]},\"1\":{\"identifier\":\"DE-NW-BN-SE030-20201014-30-000\",\"sender\":\"DE-NW-BN-SE030\",\"sent\":\"2020-10-14T16:35:21+02:00\",\"status\":\"Actual\",\"msgType\":\"Update\",\"scope\":\"Public\",\"code\":[\"1.0\",\"medien_ueberregional\",\"nina\"],\"info\":[\"here's much more\"]},\"2\":{\"identifier\":\"DE-BY-M-W060-20201209-000\",\"sender\":\"DE-BY-M-W060\",\"sent\":\"2020-12-09T14:35:17+01:00\",\"status\":\"Actual\",\"msgType\":\"Alert\",\"scope\":\"Public\",\"code\":[\"1.0\"],\"info\":[\"here's much more\"]}}","payloadType":"json","x":110,"y":3520,"wires":[["46610142.50118"]]},{"id":"46610142.50118","type":"function","z":"8d22ae29.7df6d","name":"","func":"for (let key in msg.payload){\n    let msg1 = {payload:{}};\n    msg1.payload[key] = Object.assign(msg.payload[key],{\"key\":key});\n    node.send(msg1)\n}\nreturn;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":260,"y":3580,"wires":[["10738584.23f9a2"]]},{"id":"10738584.23f9a2","type":"change","z":"8d22ae29.7df6d","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"$sift(payload.*, function($v,$k){$k in [\"key\",\"identifier\",\"sender\",\"sent\",\"status\"]})","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":440,"y":3520,"wires":[["b7341d97.470e78"]]},{"id":"b7341d97.470e78","type":"function","z":"8d22ae29.7df6d","name":"","func":"for (let key in msg.payload){\n    if (key !== \"key\"){\n        let msg1 = {payload:{}};\n        msg1.payload = msg.payload[key];\n        msg1.topic = \"my/mqtt/topic/\" + key + msg.payload.key;\n        node.send(msg1)\n    }\n}\nreturn","outputs":1,"noerr":0,"initialize":"","finalize":"","x":570,"y":3580,"wires":[["ec76a518.e86c88"]]},{"id":"ec76a518.e86c88","type":"mqtt out","z":"8d22ae29.7df6d","name":"","topic":"","qos":"2","retain":"false","broker":"35ccc936.fc2256","x":760,"y":3580,"wires":[]},{"id":"92cdcc2d.366b08","type":"mqtt in","z":"8d22ae29.7df6d","name":"","topic":"my/mqtt/topic/#","qos":"2","datatype":"auto","broker":"35ccc936.fc2256","x":260,"y":3660,"wires":[["a2bea5e6.17f7c8"]]},{"id":"a2bea5e6.17f7c8","type":"debug","z":"8d22ae29.7df6d","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":510,"y":3660,"wires":[]},{"id":"35ccc936.fc2256","type":"mqtt-broker","z":"","name":"free","broker":"broker.hivemq.com","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]
1 Like

Wow! Great!
It works and now I have to analyze and wrap my head around the steps you did there!

excuse my lack of JavaScript, is there some docs or could you explain what happens here?

for (let key in msg.payload){ // the JSON object is looped on the let-variable "key" (in the test data [0], [1], [3])
    let msg1 = {payload:{}};  // variable msg1 is initialized as JSON object with an empty "payload" 
    msg1.payload[key] = Object.assign(msg.payload[key],{"key":key}); // msg1.payload is filled with the "key" node and subsequent nodes
    node.send(msg1) // JSON object msg1 is send
}
return;

So now, the "change" node in node-red gets three msg1.payloads containg the subsequent nodes, correct?

$sift(payload.*, function($v,$k){$k in ["key","identifier","sender","sent","status"]})

sift now returns all nodes matching value/key pairs, the function finds - and fills "msg.payload" with it. So now we have a JSON containing only the found keys of the function, correct?

for (let key in msg.payload){ // the JSON object is looped on the let-variable "key" (now only the defined attributes)
    if (key !== "key"){       // do nothing for the key-identifier
        let msg1 = {payload:{}}; // initialize msg1 as empty JSON object
        msg1.payload = msg.payload[key]; // fill the msg1.payload with the current "key"-value
        msg1.topic = "my/mqtt/topic/" + key + msg.payload.key; // create the mqtt topic with the "key"-name
        node.send(msg1); // send the message - using the mqtt node I suppose
    }
}
return

so finally, the function iterates the sent msg.payload for the JSON and sends mqtt-messages to the created topics.
I think I got this right... :wink:

What if I want to straighten the JSON even more, like the "real" information come subsequent of the node "info" (which I shortened initially):

{
    "0": {
        "identifier": "DE-BY-A-W083-20200828-000",
        "sender": "DE-BY-A-W083",
        "sent": "2020-08-28T11:00:08+02:00",
        "status": "Actual",
        "msgType": "Update",
        "scope": "Public",
        "code": [
            "1.0",
            "nina"
        ],
        "references": "DE-BY-A-W083-20200317-000 DE-BY-A-W083-20191213-000 DE-BY-A-W083,DE-BY-A-W083-20200317-000 DE-BY-A-W083-20191213-000 DE-BY-A-W083-20190821-000,2019-08-21T00:00:00+00:00",
        "info": [
            {
                "language": "DE",
                "category": [
                    "Health",
                    "Other"
                ],
                "event": "Gefahreninformation",
                "urgency": "Immediate",
                "severity": "Minor",
                "certainty": "Observed",
                "eventCode": [

Would I search for the subsequent keys in the first function to create a "flat" JSON with only my information (like [0].info.[0].certainty for example => $certainty), or would I do that in the last function just before I send the mqtt messages?
...and how would this be done ideally? This time I guess it would be easiest to pick directly the "deep linked" attributes, which are of concern and not just straight out "flatten" the whole JSON. I'm a bit stuck in how to start to do this.

I did incluce "info" in the change-node, and then I added another function directly afterwards:

msg.payload["language"] = msg.payload["info"][0]["language"]
delete msg.payload["info"];
return msg;

Is this a efficient way to deal with that? like this:

[{"id":"47dc9260.3223cc","type":"tab","label":"Flow 6","disabled":false,"info":""},{"id":"9a4335c6.75c7e8","type":"inject","z":"47dc9260.3223cc","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"0\":{\"identifier\":\"DE-BY-A-W083-20200828-000\",\"sender\":\"DE-BY-A-W083\",\"sent\":\"2020-08-28T11:00:08+02:00\",\"status\":\"Actual\",\"msgType\":\"Update\",\"scope\":\"Public\",\"code\":[\"1.0\",\"nina\"],\"info\":[{\"language\":\"DE\",\"category\":[\"Health\",\"Other\"],\"event\":\"Gefahreninformation\",\"urgency\":\"Immediate\",\"severity\":\"Minor\",\"certainty\":\"Observed\"}]},\"1\":{\"identifier\":\"DE-NW-BN-SE030-20201014-30-000\",\"sender\":\"DE-NW-BN-SE030\",\"sent\":\"2020-10-14T16:35:21+02:00\",\"status\":\"Actual\",\"msgType\":\"Update\",\"scope\":\"Public\",\"code\":[\"1.0\",\"medien_ueberregional\",\"nina\"],\"info\":[{\"language\":\"DE\",\"category\":[\"Health\",\"Other\"],\"event\":\"Gefahreninformation\",\"urgency\":\"Immediate\",\"severity\":\"Minor\",\"certainty\":\"Observed\"}]},\"2\":{\"identifier\":\"DE-BY-M-W060-20201209-000\",\"sender\":\"DE-BY-M-W060\",\"sent\":\"2020-12-09T14:35:17+01:00\",\"status\":\"Actual\",\"msgType\":\"Alert\",\"scope\":\"Public\",\"code\":[\"1.0\"],\"info\":[{\"language\":\"DE\",\"category\":[\"Health\",\"Other\"],\"event\":\"Gefahreninformation\",\"urgency\":\"Immediate\",\"severity\":\"Minor\",\"certainty\":\"Observed\"}]}}","payloadType":"json","x":140,"y":140,"wires":[["46610142.50118"]]},{"id":"46610142.50118","type":"function","z":"47dc9260.3223cc","name":"","func":"for (let key in msg.payload){\n    let msg1 = {payload:{}};\n    msg1.payload[key] = Object.assign(msg.payload[key],{\"key\":key});\n    node.send(msg1)\n}\nreturn;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":290,"y":200,"wires":[["10738584.23f9a2"]]},{"id":"10738584.23f9a2","type":"change","z":"47dc9260.3223cc","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"$sift(payload.*, function($v,$k){$k in [\"key\",\"identifier\",\"sender\",\"sent\",\"status\", \"info\"]})","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":470,"y":140,"wires":[["8bb93548.e088e","d70f99b4.107078"]]},{"id":"b7341d97.470e78","type":"function","z":"47dc9260.3223cc","name":"","func":"for (let key in msg.payload){\n    if (key !== \"key\"){\n        let msg1 = {payload:{}};\n        msg1.payload = msg.payload[key];\n        msg1.topic = \"my/mqtt/topic/\" + key + msg.payload.key;\n        node.send(msg1)\n    }\n}\nreturn","outputs":1,"noerr":0,"initialize":"","finalize":"","x":820,"y":240,"wires":[["ec76a518.e86c88","2e7fd233.af7bae"]]},{"id":"ec76a518.e86c88","type":"mqtt out","z":"47dc9260.3223cc","name":"","topic":"","qos":"2","retain":"false","broker":"35ccc936.fc2256","x":1150,"y":220,"wires":[]},{"id":"92cdcc2d.366b08","type":"mqtt in","z":"47dc9260.3223cc","name":"","topic":"my/mqtt/topic/#","qos":"2","datatype":"auto","broker":"35ccc936.fc2256","x":260,"y":480,"wires":[["a2bea5e6.17f7c8"]]},{"id":"a2bea5e6.17f7c8","type":"debug","z":"47dc9260.3223cc","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":510,"y":480,"wires":[]},{"id":"2e7fd233.af7bae","type":"debug","z":"47dc9260.3223cc","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1070,"y":60,"wires":[]},{"id":"8bb93548.e088e","type":"debug","z":"47dc9260.3223cc","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":640,"y":60,"wires":[]},{"id":"d70f99b4.107078","type":"function","z":"47dc9260.3223cc","name":"","func":"msg.payload[\"language\"] = msg.payload[\"info\"][0][\"language\"]\ndelete msg.payload[\"info\"]\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":640,"y":240,"wires":[["b7341d97.470e78"]]},{"id":"35ccc936.fc2256","type":"mqtt-broker","z":null,"name":"free","broker":"broker.hivemq.com","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]
[{"id":"b470c929.b6afc8","type":"change","z":"86635056.c9893","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"\t$sift(payload[*], function($v,$k){$k in [\"language\",\"category\"]})\t","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":600,"y":260,"wires":[["23941637.d1ee72"]]},{"id":"473d8b6e.455aec","type":"function","z":"86635056.c9893","name":"","func":"let msg1 = {};\nfor (let key in msg.payload){\n        if (typeof(msg.payload[key])===\"string\"){\n            msg1 = {};\n            msg1.payload = msg.payload[key];\n            msg1.topic = \"my/mqtt/topic/\" + key + msg.key;\n            node.send(msg1);\n        }else{\n            msg1 = {};\n            msg1.payload = msg.payload[key]\n            msg1.key = msg.key\n            msg1.topic = \"\";\n            node.send([null,msg1]);\n        }\n}\nreturn","outputs":2,"noerr":0,"initialize":"","finalize":"","x":530,"y":200,"wires":[["ba63fc15.667a6","72d17cd5.800464"],["b470c929.b6afc8"]]},{"id":"23941637.d1ee72","type":"function","z":"86635056.c9893","name":"","func":"let msg1 = {};\nfor (let key in msg.payload){\n        if (typeof(msg.payload[key])===\"string\"){\n            msg1 = {};\n            msg1.payload = msg.payload[key];\n            msg1.topic = \"my/mqtt/topic/\" + key + msg.key;\n            node.send(msg1);\n        }else if (Array.isArray(msg.payload[key])){\n            msg1 = {};\n            msg1.payload = msg.payload[key].join(\",\");\n            msg1.key = msg.key\n            msg1.topic = \"my/mqtt/topic/\" + key + msg.key;\n            node.send(msg1);\n        }\n}\nreturn","outputs":1,"noerr":0,"initialize":"","finalize":"","x":690,"y":320,"wires":[["ba63fc15.667a6","72d17cd5.800464"]]},{"id":"e414c7d4.bd66f8","type":"change","z":"86635056.c9893","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"$sift(payload.*, function($v,$k){$k in [\"identifier\",\"info\", \"status\"]})","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":470,"y":140,"wires":[["473d8b6e.455aec"]]},{"id":"ba63fc15.667a6","type":"debug","z":"86635056.c9893","name":"","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"payload","targetType":"msg","statusVal":"key","statusType":"msg","x":930,"y":200,"wires":[]},{"id":"72d17cd5.800464","type":"mqtt out","z":"86635056.c9893","name":"","topic":"","qos":"","retain":"","broker":"35ccc936.fc2256","x":890,"y":260,"wires":[]},{"id":"29177694.a8e582","type":"function","z":"86635056.c9893","name":"","func":"for (let key in msg.payload){\n    let msg1 = {payload:{}};\n    msg1.payload[key] = msg.payload[key];\n    msg1.key = key;\n    node.send(msg1)\n}\nreturn;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":290,"y":200,"wires":[["e414c7d4.bd66f8"]]},{"id":"23e10532.33e1da","type":"inject","z":"86635056.c9893","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"0\":{\"identifier\":\"DE-BY-A-W083-20200828-000\",\"sender\":\"DE-BY-A-W083\",\"sent\":\"2020-08-28T11:00:08+02:00\",\"status\":\"Actual\",\"msgType\":\"Update\",\"scope\":\"Public\",\"code\":[\"1.0\",\"nina\"],\"info\":[{\"language\":\"DE\",\"category\":[\"Health\",\"Other\"],\"event\":\"Gefahreninformation\",\"urgency\":\"Immediate\",\"severity\":\"Minor\",\"certainty\":\"Observed\"}]},\"1\":{\"identifier\":\"DE-NW-BN-SE030-20201014-30-000\",\"sender\":\"DE-NW-BN-SE030\",\"sent\":\"2020-10-14T16:35:21+02:00\",\"status\":\"Actual\",\"msgType\":\"Update\",\"scope\":\"Public\",\"code\":[\"1.0\",\"medien_ueberregional\",\"nina\"],\"info\":[{\"language\":\"DE\",\"category\":[\"Health\",\"Other\"],\"event\":\"Gefahreninformation\",\"urgency\":\"Immediate\",\"severity\":\"Minor\",\"certainty\":\"Observed\"}]},\"2\":{\"identifier\":\"DE-BY-M-W060-20201209-000\",\"sender\":\"DE-BY-M-W060\",\"sent\":\"2020-12-09T14:35:17+01:00\",\"status\":\"Actual\",\"msgType\":\"Alert\",\"scope\":\"Public\",\"code\":[\"1.0\"],\"info\":[{\"language\":\"DE\",\"category\":[\"Health\",\"Other\"],\"event\":\"Gefahreninformation\",\"urgency\":\"Immediate\",\"severity\":\"Minor\",\"certainty\":\"Observed\"}]}}","payloadType":"json","x":230,"y":160,"wires":[["29177694.a8e582"]]},{"id":"7396b171.77ee68","type":"debug","z":"86635056.c9893","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":510,"y":480,"wires":[]},{"id":"b27ccfd0.4b4648","type":"mqtt in","z":"86635056.c9893","name":"","topic":"my/mqtt/topic/#","qos":"2","datatype":"auto","broker":"35ccc936.fc2256","x":260,"y":480,"wires":[["7396b171.77ee68"]]},{"id":"35ccc936.fc2256","type":"mqtt-broker","z":"","name":"free","broker":"broker.hivemq.com","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]

Have fun breaking this down.

1 Like

Thanks! I had fun! :wink:
If I understand correctly, you can make kinda "if"-conditions by simply "send"ing the right variables, so we got this
grafik
Thanks for the heads-up! learned a lot.

yes depending on the payload type you can send to different outputs, I tried to cover all types in your json the only one i did not test was "code".

[edit] The key property has been moved so the functions are not the same as the original.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.