Problem in updating a document in MongoDB

Hi

I'm using "node-red-node-mongodb" to write to a MongoDB cluster.

All fine as long as I'm using "save" to write a document with a set of objects.

When trying to "update" a certain document by adding additional objects or modifying existing ones I'm running into problems in addressing the right document and to update it.

Description of the node says " The query to select objects to update uses msg.query" but after hours of trying I'm not able to find the right way to do it. I admit that I'm not really fit in functions so I tried to use change nodes etc. to set the query but all I get is error

"MongoError: multi update only works with $ operators"

... and I can't find out how to handle this " $ operators" thingie.

Would really appreciate if anyone could point me into the right direction of even help me with a sample flow.

BR Matthias

hello Matthias .. it would be better if you send us a Flow example of what you have tried

here's an example flow of how to update one or update many documents

[{"id":"2b6ad9f594be555e","type":"mongodb out","z":"c7a877f7.e1d698","mongodb":"39adf9d0.a8c876","name":"","collection":"temperatures","payonly":false,"upsert":false,"multi":true,"operation":"update","x":640,"y":940,"wires":[]},{"id":"893da818d73cc952","type":"function","z":"c7a877f7.e1d698","name":"updateMany","func":"let ObjectId = mongodb.ObjectId\n\nlet newMsg = {};\n\nnewMsg.query = { \"_id\": { \"$in\": [ObjectId(\"60000f1879d9790bac657a94\"), ObjectId(\"6000162079d9790bac657a95\")] } };\n\nnewMsg.payload = { \"$set\": { \"testField\": 888 } };\n    \nreturn newMsg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[{"var":"mongodb","module":"mongodb"}],"x":350,"y":940,"wires":[["2b6ad9f594be555e"]]},{"id":"1c3c63331289ba7e","type":"inject","z":"c7a877f7.e1d698","name":"Inject","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payloadType":"date","x":150,"y":940,"wires":[["893da818d73cc952"]]},{"id":"7bfecb05a81cdb07","type":"function","z":"c7a877f7.e1d698","name":"updateOne","func":"var newMsg = {};\n\nnewMsg.query = { \"_id\": \"60000f1879d9790bac657a94\" } ;\n\nnewMsg.payload = { \"$set\": { \"testField\": 777 } };\n\nreturn newMsg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":350,"y":1020,"wires":[["46adb34e1bb086ec"]]},{"id":"7c8e8f3780a3ea6d","type":"inject","z":"c7a877f7.e1d698","name":"Inject","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payloadType":"date","x":150,"y":1020,"wires":[["7bfecb05a81cdb07"]]},{"id":"46adb34e1bb086ec","type":"mongodb out","z":"c7a877f7.e1d698","mongodb":"39adf9d0.a8c876","name":"","collection":"temperatures","payonly":false,"upsert":false,"multi":true,"operation":"update","x":640,"y":1020,"wires":[]},{"id":"39adf9d0.a8c876","type":"mongodb","hostname":"localhost","topology":"direct","connectOptions":"useUnifiedTopology=true","port":"27017","db":"nodered","name":""}]

Notice that in my query to update many, because im using the _id as a query field, we had to import the Mongo Driver library in the Function node in order to convert the ids into the special ObjectId mongo needs. Hope it makes sense .. try it with your db and if you have some problem let us know.

ps. always make a backup or use a test db in case you are unsure of some commands.

Hi Andy,

thanks a lot for your feedback ... as I expected handling of function is beyond my (low) knowledge level.

What I was successfully doing so far is retrieving a data set from LoRaWAN, generating a static _id for a certain device and writing it to a MongoDB through "save". As soon as the next data set arrives the same _id is being generated, the previous data set in MongoDB overwritten and I could present always the latest data set.

Now data sets from same device arrive but each set contains different infos and I want to present the combination of such infos by updating a document with the additional objects ....

I tried to put together the important parts of my process within a flow - so now trying first time to add a flow to a forum post ... cross fingers ...

BR, Matthias

[
    {
        "id": "66bf3280.eeb04c",
        "type": "inject",
        "z": "a6a7e3da.fa9b6",
        "name": "Data Set 1",
        "props": [
            {
                "p": "payload.mmi_DevEUI",
                "v": "A81758FFFE04FCE5",
                "vt": "str"
            },
            {
                "p": "payload.ts",
                "v": "",
                "vt": "date"
            },
            {
                "p": "payload.item1",
                "v": "blabla",
                "vt": "str"
            },
            {
                "p": "payload.item2",
                "v": "12345",
                "vt": "num"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "x": 140,
        "y": 220,
        "wires": [
            [
                "53da6288.39be74",
                "ce4f669f.f95f"
            ]
        ]
    },
    {
        "id": "481f2246.b8667c",
        "type": "inject",
        "z": "a6a7e3da.fa9b6",
        "name": "Data Set 2",
        "props": [
            {
                "p": "payload.mmi_DevEUI",
                "v": "A81758FFFE04FCE5",
                "vt": "str"
            },
            {
                "p": "payload.ts",
                "v": "",
                "vt": "date"
            },
            {
                "p": "payload.item3",
                "v": "testtest",
                "vt": "str"
            },
            {
                "p": "payload.item4",
                "v": "67890",
                "vt": "num"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "x": 140,
        "y": 260,
        "wires": [
            [
                "53da6288.39be74",
                "82c502b4.6d8e48"
            ]
        ]
    },
    {
        "id": "53da6288.39be74",
        "type": "function",
        "z": "a6a7e3da.fa9b6",
        "name": "shorten deveui to 12 char",
        "func": "var str = msg.payload.mmi_DevEUI;\n\nvar str_short_begin = str.slice(0, 6);\nvar str_short_end = str.slice(10, 16);\nvar str_short = str_short_begin + str_short_end;\n\nmsg.payload._id = str_short;\n\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "x": 350,
        "y": 240,
        "wires": [
            [
                "393a2e11.9e075a",
                "fd507e0a.82d6a8"
            ]
        ]
    },
    {
        "id": "393a2e11.9e075a",
        "type": "objectid",
        "z": "a6a7e3da.fa9b6",
        "name": "generate _id from deveui",
        "selectedProperty": "_id",
        "x": 610,
        "y": 240,
        "wires": [
            [
                "a7c2887b.991c3"
            ]
        ]
    },
    {
        "id": "ce4f669f.f95f",
        "type": "debug",
        "z": "a6a7e3da.fa9b6",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 290,
        "y": 180,
        "wires": []
    },
    {
        "id": "82c502b4.6d8e48",
        "type": "debug",
        "z": "a6a7e3da.fa9b6",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 290,
        "y": 300,
        "wires": []
    },
    {
        "id": "fd507e0a.82d6a8",
        "type": "debug",
        "z": "a6a7e3da.fa9b6",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 550,
        "y": 180,
        "wires": []
    },
    {
        "id": "a7c2887b.991c3",
        "type": "debug",
        "z": "a6a7e3da.fa9b6",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 810,
        "y": 180,
        "wires": []
    },
    {
        "id": "53472c44.bb002c",
        "type": "function",
        "z": "a6a7e3da.fa9b6",
        "name": "updateMany",
        "func": "let ObjectId = mongodb.ObjectId\n\nlet newMsg = {};\n\nnewMsg.query = { \"_id\": { \"$in\": [ObjectId(\"60000f1879d9790bac657a94\"), ObjectId(\"6000162079d9790bac657a95\")] } };\n\nnewMsg.payload = { \"$set\": { \"testField\": 888 } };\n    \nreturn newMsg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "x": 950,
        "y": 240,
        "wires": [
            []
        ]
    },
    {
        "id": "d503d41c.0cb168",
        "type": "mongodb out",
        "z": "a6a7e3da.fa9b6",
        "mongodb": "d5beac98.b8fba8",
        "name": "mmiDevice ",
        "collection": "test",
        "payonly": true,
        "upsert": true,
        "multi": true,
        "operation": "update",
        "x": 1230,
        "y": 240,
        "wires": []
    },
    {
        "id": "d5beac98.b8fba8",
        "type": "mongodb",
        "z": "",
        "hostname": "localhost",
        "topology": "direct",
        "connectOptions": "useUnifiedTopology=true",
        "port": "27017",
        "db": "nodered",
        "name": ""
    }
]

just a though .. you could combine the two different msgs coming from the Lorawan device using a Join node, before saving the data in Mongo .. this way you avoid the more complicated mongo update ?

Example

[{"id":"66bf3280.eeb04c","type":"inject","z":"54efb553244c241f","name":"Data Set 1","props":[{"p":"payload.mmi_DevEUI","v":"A81758FFFE04FCE5","vt":"str"},{"p":"payload.ts","v":"","vt":"date"},{"p":"payload.item1","v":"blabla","vt":"str"},{"p":"payload.item2","v":"12345","vt":"num"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":120,"y":2100,"wires":[["ce4f669f.f95f","53da6288.39be74"]]},{"id":"481f2246.b8667c","type":"inject","z":"54efb553244c241f","name":"Data Set 2","props":[{"p":"payload.mmi_DevEUI","v":"A81758FFFE04FCE5","vt":"str"},{"p":"payload.ts","v":"","vt":"date"},{"p":"payload.item3","v":"testtest","vt":"str"},{"p":"payload.item4","v":"67890","vt":"num"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":120,"y":2140,"wires":[["82c502b4.6d8e48","53da6288.39be74"]]},{"id":"53da6288.39be74","type":"function","z":"54efb553244c241f","name":"shorten deveui to 12 char","func":"var str = msg.payload.mmi_DevEUI;\n\nvar str_short_begin = str.slice(0, 6);\nvar str_short_end = str.slice(10, 16);\nvar str_short = str_short_begin + str_short_end;\n\nmsg.payload._id = str_short;\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":350,"y":2120,"wires":[["77f7e9a9de3b7a90"]]},{"id":"ce4f669f.f95f","type":"debug","z":"54efb553244c241f","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":230,"y":2040,"wires":[]},{"id":"82c502b4.6d8e48","type":"debug","z":"54efb553244c241f","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":230,"y":2200,"wires":[]},{"id":"a7c2887b.991c3","type":"debug","z":"54efb553244c241f","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":630,"y":2060,"wires":[]},{"id":"d503d41c.0cb168","type":"mongodb out","z":"54efb553244c241f","mongodb":"d5beac98.b8fba8","name":"mmiDevice ","collection":"test","payonly":true,"upsert":true,"multi":true,"operation":"store","x":870,"y":2120,"wires":[]},{"id":"77f7e9a9de3b7a90","type":"join","z":"54efb553244c241f","name":"","mode":"custom","build":"merged","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"6","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":550,"y":2120,"wires":[["a7c2887b.991c3","d503d41c.0cb168"]]},{"id":"d5beac98.b8fba8","type":"mongodb","hostname":"localhost","topology":"direct","connectOptions":"useUnifiedTopology=true","port":"27017","db":"nodered","name":""}]

The Join node waits for the 6 msg properties (mmi_DevEUI, ts, _id, item1,item2, item3, item4 ) before producing a merged object, that you can later send to the Mongo save node.

ps. i wanted to ask .. its only the last record that you want to have in Mongo db ? you dont need historical data / many records. If yes then maybe you dont even need Mongo and use Nodered's file system Context ?

First my apologies for late reply ... busy times :wink: ....

A few more details on the case:

I'm working on a LoRaWAN IoT solution. Historical data are are stored anyway in a separate DB which contains hundreds of millions of records ... what I want is a DB per device, containing the most current (combined) data set without having to collect that from the historical DB.

The problem is: more and more manufacturers start to split their data sets into several datagrams (in former times all was within one datagram) which may arrive hours or even days apart. So join nodes are not helpful.

Joining on a frontend by selecting records from the historical DB is also not possible, simply too many records involved.

I'm a little disappointed, that NR/MongoDB setup being so difficult for simply updating a given record with additional content. Not quite clear what an "update" function is good for, if that's not (easily) possible ....

So I think, I have to go the ugly way on reading the set, selecting what's new, replacing that and writing it back to the DB.

i see .. thats ok .. we'll have another go with the Mongodb "update" command.

The example below uses nodered db and test collection.

On first entry it adds a document and what is unique of this document, except from _id that is added by mongo, is the device_id: "A8175804FCE5" .. originally named _id in your flow but i renamed it to device_id because it makes things easier to update.

image

On subsequent msgs we select the document based on device_id and $set (update) existing fields and add new ones if they are missing.

image

Example Flow :

[{"id":"66bf3280.eeb04c","type":"inject","z":"54efb553244c241f","name":"Data Set 1","props":[{"p":"payload.mmi_DevEUI","v":"A81758FFFE04FCE5","vt":"str"},{"p":"payload.ts","v":"","vt":"date"},{"p":"payload.item1","v":"blabla","vt":"str"},{"p":"payload.item2","v":"12345","vt":"num"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":160,"y":2080,"wires":[["53da6288.39be74"]]},{"id":"481f2246.b8667c","type":"inject","z":"54efb553244c241f","name":"Data Set 2","props":[{"p":"payload.mmi_DevEUI","v":"A81758FFFE04FCE5","vt":"str"},{"p":"payload.ts","v":"","vt":"date"},{"p":"payload.item3","v":"testtest","vt":"str"},{"p":"payload.item4","v":"67890","vt":"num"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":160,"y":2120,"wires":[["53da6288.39be74"]]},{"id":"53da6288.39be74","type":"function","z":"54efb553244c241f","name":"shorten deveui to 12 char","func":"var str = msg.payload.mmi_DevEUI;\n\nvar str_short_begin = str.slice(0, 6);\nvar str_short_end = str.slice(10, 16);\nvar str_short = str_short_begin + str_short_end;\n\nmsg.payload.device_id = str_short;\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":370,"y":2100,"wires":[["fd507e0a.82d6a8","53472c44.bb002c"]]},{"id":"fd507e0a.82d6a8","type":"debug","z":"54efb553244c241f","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":590,"y":2040,"wires":[]},{"id":"d503d41c.0cb168","type":"mongodb out","z":"54efb553244c241f","mongodb":"d5beac98.b8fba8","name":"mmiDevice ","collection":"test","payonly":true,"upsert":true,"multi":false,"operation":"update","x":850,"y":2100,"wires":[]},{"id":"53472c44.bb002c","type":"function","z":"54efb553244c241f","name":"update","func":"msg.query = { \"device_id\": msg.payload.device_id };\nmsg.payload = { $set: msg.payload  }\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":620,"y":2100,"wires":[["d503d41c.0cb168"]]},{"id":"d5beac98.b8fba8","type":"mongodb","hostname":"localhost","topology":"direct","connectOptions":"useUnifiedTopology=true","port":"27017","db":"nodered","name":"","credentials":{}}]
1 Like

IT'S WORKING!!!

So incredibly easy, if you know how to do it!!

I ran into additional problems with manually using always the same _id for MongoDB to always hit the same record, but now it's much easier in updating based on a query with $set ...

Thanks a lot Andy! Saved my life :wink: ...

BR

Matthias

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.