Hello Everybody,
I am doing one project, in which I am accessing some data from the device and i need to send that data on the server or any other machine through mqtt protocol, but if server or target machine is not available at that time so i want to store that data and when server or machine is available then it should send.
So what type of storage i can use for this, please help me out.
Thank you.
You can store data in all kinds of ways. Choose an option that fits your data.
It could be in a file, sqlite, influxdb, mysql etc.
Maybe this node does what you want
(I've not tested it)
I tried with this but it is not storing data after loosing the connection.
Well I think that is the purpose of the node - so maybe contact the author direct to see what is wrong ?
Which connection are you talking about? Between which two devices/machines?
I am sending data on the server via mqtt. And I am accessing the server through VPN, when I am disconnecting the VPN my system is disconnecting from the server so it should store the data, but it is not storing. Please resolve my issue or guide me if I am missing anything.
You say you have a 'system' and a 'server' which are connected through a VPN. Is nod-red running on the 'system' and the mqtt broker running on the server? If so then the node suggested should do the job. It should save the data on the 'system' and send it to the mqtt broker when the connection re-appears. Note though that it says the data is saved in RAM so I think it may not survive a node-red restart. I don't know though.
Try using a stack buffer to store in memory.
The buffer size is configurable in this subflow, first in first out (FIFO):
https://flows-new.nodered.org/flow/d7eea525606ea45a7c850976a85b349e
Or if you prefer last in first out (LIFO)
https://flows-new.nodered.org/flow/d0719fa955611bbb073723a23a204ce8
You need to set this up so the data flows to the buffer in parallel to your normal flow. When the link is down "trigger" the push onto the stack and then pop off the stack when it comes back up prior to going back to the "normal" data flow. (A bit of setup needed here). The subflows in the link have controls inbuilt to do this but there is the need to link to these triggers.
I've used the ones in the link on another project and they work well for something similar to your requirements. It maybe worth looking at to see if it suits your needs?
Can we see the stored data in FIFO.
Not sure about the version posted but I made something similar where you passed in a buffer name & it stored the FIFO data in global storage - which is visible - so it is possible.
Yes, Node-red running on my system and mqtt broker on the server, then also it is not storing data. Only it is creating incoming and outgoing text file and the file remains blank after disconnecting to the server or mqtt broker.
Can you shear your flow. Or where i can see the stored data.
I'm not sure what you mean by "see" but you can see the buffer filling with the counter on the screen, you can also connect a message node in to see the array.
yaa, I want to see the array.
I'm pretty sure he means being able to see - at any point in time - what is in the buffer.
The subflow version i wrote allowed you to enter an ENV VAR string for the name (so you could have multiple buffers) & internally, the data STACK was stored in global context Array meaning at any point in time I could inspect the data in the buffer (without having to look through debug messages) by looking in the Globals. Very useful for understanding whats going on & debugging.
In particular, i used it for collecting data from a PLC & stacking it before putting into DB. If the DB went offline, the buffer would fill up (I could even deliberately pause the buffer - so I could inspect data before passing to DB). Once the DB connection was restored, the buffer would pop its data off the stack into the DB (no loss data collection).
@akhtar321h It will take me an hour or so dig it out & prepare it for you.
@akhtar321h this is the subflow node I made for buffering data.
Can be persisted to file by specifying a different "Store Name" (needs setup in node-red settings.js - see context-stores)
[{"id":"83c74899.222798","type":"subflow","name":"fifo buffer","info":"# About\nThis node acts like a FIFO. \nMessages added first are returned first.\nThis node performs 5 distinct operations, depending on the value of `topic`\n\n# node properties\n* **Buffer Name** - The name of the `flow` variable to store the buffer in.\n* **Store Name** - The name of store (e.g. \"file\" or \"default\"). NOTE: The storage must be setup in settings.js for this installation of node-red is using this option.\n* **Auto Status** - This flag sets the node to auto update the buffer cound status. This is needed to update all instances of the same buffer. Additionally, nodes with this option set `true` will output status info on output2 automatically when a change occurs.\n* **Topic** - The operation to perform (see Operations below).\n\n# Operations\nDepending on the value of node property `topic` (or `msg.topic` if `topic` is blank)\nThis node performs the following as below...\n* **put**\n * adds `msg.payload` to the buffer\n * returns the added value in `msg.added`\n* **get**\n * simply returns the next item in the buffer \n * `msg.payload` will be null if buffer is empty\n* **shift**\n * removes the last item from the buffer\n * `msg.payload` will be the value of the next item or null if buffer is empty\n* **clear**\n * empties the buffer\n* **status**\n * returns the count of items in `msg.payload`\n\n# Output\n1. All successful operations will return a `msg`\n * When an item is successfully added, a copy is returned in `msg.added`\n2. Status info will be sent from this output. useful to get the status at any time.\n","category":"","in":[{"x":60,"y":120,"wires":[{"id":"33c0ee20.33ae32"}]}],"out":[{"x":900,"y":100,"wires":[{"id":"f0ff1b50.004c18","port":0}]},{"x":900,"y":260,"wires":[{"id":"fa449f35.b95a2","port":0}]}],"env":[{"name":"bufferName","type":"str","value":"buffer","ui":{"label":{"en-US":"Buffer Name"},"type":"input","opts":{"types":["str","env"]}}},{"name":"storeName","type":"str","value":"file","ui":{"label":{"en-US":"Store Name"}}},{"name":"autoUpdateStatus","type":"bool","value":"false","ui":{"label":{"en-US":"Auto Status"}}},{"name":"topic","type":"str","value":"","ui":{"label":{"en-US":"Topic"}}}],"color":"#C0DEED","outputLabels":["msg","status"],"icon":"font-awesome/fa-database","status":{"x":900,"y":340,"wires":[{"id":"6572aae8.238064","port":0}]}},{"id":"f0ff1b50.004c18","type":"function","z":"83c74899.222798","name":"get next in buffer","func":"var getBuffer = msg._util.getBuffer;\nvar setBuffer = msg._util.setBuffer;\n\nvar buffer = getBuffer();\n\nvar entry = buffer.length ? buffer[buffer.length-1] : null;\nnode.status({text:`Buffer: ${buffer.length}`});\nmsg.payload = entry;\nreturn msg;","outputs":1,"noerr":0,"x":750,"y":100,"wires":[[]]},{"id":"bb7ccfd5.61128","type":"function","z":"83c74899.222798","name":"add to buffer","func":"var getBuffer = msg._util.getBuffer;\nvar setBuffer = msg._util.setBuffer;\n\nvar buffer = getBuffer();\nbuffer.unshift(msg.payload);\nsetBuffer(buffer);\nmsg.buffer = buffer;\nmsg.added = msg.payload;\nnode.status({text:`Buffer: ${buffer.length}`});\nreturn msg;","outputs":1,"noerr":0,"x":550,"y":220,"wires":[["f0ff1b50.004c18"]]},{"id":"adca17cc.e60358","type":"switch","z":"83c74899.222798","name":"","property":"topic","propertyType":"msg","rules":[{"t":"eq","v":"get","vt":"str"},{"t":"eq","v":"shift","vt":"str"},{"t":"eq","v":"clear","vt":"str"},{"t":"eq","v":"put","vt":"str"},{"t":"eq","v":"status","vt":"str"},{"t":"eq","v":"autostatus","vt":"str"}],"checkall":"true","repair":false,"outputs":6,"x":330,"y":220,"wires":[["f0ff1b50.004c18"],["8f13088f.c890b8"],["8f13088f.c890b8"],["bb7ccfd5.61128"],["88c020b4.c5b49"],["88c020b4.c5b49"]]},{"id":"6572aae8.238064","type":"status","z":"83c74899.222798","name":"","scope":null,"x":100,"y":340,"wires":[[]]},{"id":"8f13088f.c890b8","type":"function","z":"83c74899.222798","name":"shift / clear","func":"var getBuffer = msg._util.getBuffer;\nvar setBuffer = msg._util.setBuffer;\n\nvar buffer = getBuffer();\n\nif(msg.topic == \"clear\"){\n buffer = [];\n} else if(buffer.length){\n msg.item = buffer.pop();\n}\n\nsetBuffer(buffer);\n\n\nnode.status({text:`Buffer: ${buffer.length}`});\nmsg.payload = buffer.length;\n\nreturn msg;","outputs":1,"noerr":0,"x":550,"y":180,"wires":[["f0ff1b50.004c18"]]},{"id":"33c0ee20.33ae32","type":"function","z":"83c74899.222798","name":"init msg.utils","func":"\nif(msg.topic !== \"autostatus\"){\n msg.topic = env.get(\"topic\") || msg.topic;\n}\n\nvar _util = {\n bufferName: env.get(\"bufferName\"),\n storeName : env.get(\"storeName\")\n};\n\n_util.getBuffer = function(){\n return flow.get(\"$parent.\" + _util.bufferName, _util.storeName) || [];\n}\n\n_util.setBuffer = function(buffer){\n return flow.set(\"$parent.\" + _util.bufferName, buffer, _util.storeName);\n}\n\n_util.showError = function(errText, msg){\n let e = new Error(errText);\n node.error(e,msg);\n node.status({text:e})\n}\n\nmsg._util = _util;\nreturn msg;","outputs":1,"noerr":0,"x":210,"y":120,"wires":[["adca17cc.e60358"]]},{"id":"b9e97c41.4dc98","type":"inject","z":"83c74899.222798","name":"status updater","topic":"autostatus","payload":"autoUpdateStatus","payloadType":"env","repeat":"1","crontab":"","once":false,"onceDelay":0.1,"x":120,"y":40,"wires":[["6b0e0fc2.e4a33"]]},{"id":"6b0e0fc2.e4a33","type":"switch","z":"83c74899.222798","name":"is autoUpdateStatus set?","property":"payload","propertyType":"msg","rules":[{"t":"true"}],"checkall":"true","repair":false,"outputs":1,"x":370,"y":40,"wires":[["33c0ee20.33ae32"]]},{"id":"88c020b4.c5b49","type":"function","z":"83c74899.222798","name":"update status","func":"var buffer = msg._util.getBuffer();\nnode.status({text:`Buffer: ${buffer.length}`});\nmsg.payload = buffer.length;\nreturn msg;\n","outputs":1,"noerr":0,"x":560,"y":260,"wires":[["fa449f35.b95a2"]]},{"id":"fa449f35.b95a2","type":"rbe","z":"83c74899.222798","name":"","func":"rbe","gap":"","start":"","inout":"out","property":"payload","x":730,"y":260,"wires":[[]]},{"id":"aeac549b.1567f8","type":"subflow:83c74899.222798","z":"66a6a259.d08c0c","name":"FIFO1","env":[{"name":"bufferName","value":"FIFO1","type":"str"},{"name":"storeName","value":"default","type":"str"},{"name":"autoUpdateStatus","value":"true","type":"bool"}],"x":730,"y":660,"wires":[["9d9fde76.c3531"],["1c948287.5c03cd"]]},{"id":"1599929c.288ccd","type":"inject","z":"66a6a259.d08c0c","name":"","topic":"put","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":570,"y":580,"wires":[["aeac549b.1567f8"]]},{"id":"331c7cc7.065234","type":"inject","z":"66a6a259.d08c0c","name":"","topic":"get","payload":"true","payloadType":"bool","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":550,"y":620,"wires":[["aeac549b.1567f8"]]},{"id":"8706e3f1.5552a","type":"inject","z":"66a6a259.d08c0c","name":"","topic":"shift","payload":"true","payloadType":"bool","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":560,"y":660,"wires":[["aeac549b.1567f8"]]},{"id":"4bb4d489.e97c9c","type":"inject","z":"66a6a259.d08c0c","name":"","topic":"clear","payload":"true","payloadType":"bool","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":560,"y":700,"wires":[["aeac549b.1567f8"]]},{"id":"141c35f5.8d55ea","type":"debug","z":"66a6a259.d08c0c","name":"get only","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1020,"y":580,"wires":[]},{"id":"1c948287.5c03cd","type":"debug","z":"66a6a259.d08c0c","name":"status","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":870,"y":680,"wires":[]},{"id":"9d9fde76.c3531","type":"switch","z":"66a6a259.d08c0c","name":"","property":"topic","propertyType":"msg","rules":[{"t":"eq","v":"get","vt":"str"},{"t":"else"}],"checkall":"true","repair":false,"outputs":2,"x":870,"y":600,"wires":[["141c35f5.8d55ea"],["d113bfaa.152ca"]]},{"id":"d113bfaa.152ca","type":"debug","z":"66a6a259.d08c0c","name":"others","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1010,"y":620,"wires":[]},{"id":"831bd60d.beee58","type":"subflow:83c74899.222798","z":"66a6a259.d08c0c","name":"FIFO2","env":[{"name":"bufferName","value":"FIFO2","type":"str"},{"name":"storeName","value":"default","type":"str"}],"x":730,"y":880,"wires":[["bac89208.68dfd"],["c69f318a.68f69"]]},{"id":"86e29396.b5c16","type":"inject","z":"66a6a259.d08c0c","name":"","topic":"put","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":570,"y":800,"wires":[["831bd60d.beee58"]]},{"id":"21f65048.fc37c","type":"inject","z":"66a6a259.d08c0c","name":"","topic":"get","payload":"true","payloadType":"bool","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":550,"y":840,"wires":[["831bd60d.beee58"]]},{"id":"76a7cba3.f01024","type":"inject","z":"66a6a259.d08c0c","name":"","topic":"shift","payload":"true","payloadType":"bool","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":560,"y":880,"wires":[["831bd60d.beee58"]]},{"id":"7bced259.c2f52c","type":"inject","z":"66a6a259.d08c0c","name":"","topic":"clear","payload":"true","payloadType":"bool","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":560,"y":920,"wires":[["831bd60d.beee58"]]},{"id":"784628ea.e62948","type":"debug","z":"66a6a259.d08c0c","name":"get only","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1020,"y":800,"wires":[]},{"id":"c69f318a.68f69","type":"debug","z":"66a6a259.d08c0c","name":"status","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":870,"y":900,"wires":[]},{"id":"bac89208.68dfd","type":"switch","z":"66a6a259.d08c0c","name":"","property":"topic","propertyType":"msg","rules":[{"t":"eq","v":"get","vt":"str"},{"t":"else"}],"checkall":"true","repair":false,"outputs":2,"x":870,"y":820,"wires":[["784628ea.e62948"],["1a4c70cd.1d8e1f"]]},{"id":"1a4c70cd.1d8e1f","type":"debug","z":"66a6a259.d08c0c","name":"others","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1010,"y":840,"wires":[]},{"id":"b2f11c36.64f7e","type":"inject","z":"66a6a259.d08c0c","name":"","topic":"status","payload":"true","payloadType":"bool","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":560,"y":960,"wires":[["831bd60d.beee58"]]}]
@Steve-Mcl Thank you for your help, my problem is solved.
One more thing I want to know that the maximum length of the buffer. How much data we can store in the buffer.
In theory, As much as you have memory (including virtual memory) but I would not recommend going anywhere near that.
Copyright OpenJS Foundation and Node-RED contributors. All rights reserved. The OpenJS Foundation has registered trademarks and uses trademarks. For a list of trademarks of the OpenJS Foundation, please see our Trademark Policy and Trademark List. Trademarks and logos not indicated on the list of OpenJS Foundation trademarks are trademarks™ or registered® trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.
The OpenJS Foundation | Terms of Use | Privacy Policy | OpenJS Foundation Bylaws | Trademark Policy | Trademark List | Cookie Policy