How to queue / hold the sensor values ,when there is a internet failure occurs

[Energycharges]

Hi All,

Am trying to send my sensor data from raspberry node red to mongo atlas cloud db , am losing the datas when internet failure occurs . am using the 4g data for the internet .
can anyone give a solution to hold / queue the data during the down time and send it to cloud when internet come back

[craigcurtin]

What sort of data and how much ?

Wy not build into your system temporary local storage ? (Depending on the platform - Rpi SD card might be an issue)

In your node to write the data to the remote host - instead write it to a local data file (csv whatever) - appending to the end as you go - set a Global context variable to say file write is in use (depending on frequency of writing), once you have finished writing to the local file, clear the context variable.

In another routine, set a Global context variable, reading from the file (which blocks writes), establish the DB connection across the internet (if it fails, then clear the lock, back off and try again in 10 seconds)

Read the first line from the file, write it to the database, once OK, delete the line, close the file and clear the read lock

Craig

[Colin]

You can do this using node-red-contrib-queue-gate which can buffer messages until they have been successfully processed, which in your case means successfully sent. Look at the example titled " Retain Until Processed" for how it can be used to do this.
It can even be configured to save them over a node-red restart if that is required.

[Energycharges]

Im running a q gate node in node red . i need to get data to be queued one by one and send to cloud db, if there any network failure.

Tried many ways to get a solution for the issue. My program is running only when the q gate case is open ,if it is in Queueing that data is not passing over through q gate node. either it shows error or no flow output.
Please any1 can help me with solution.

Regards,

Energy charges

[craigcurtin]

DId you try the solution i posted above ?

Craig

[krambriw]

@Energycharges

Please any1 can help me with solution

Share the flow, in the recommended manner & format, you have made so far. And specify in detail what nodes you have used

[Colin]

Are you using the gate node the same way as the particular example I suggested does? Where it tries to process the messages one at a time and leaves then in the queue till they have been processed?

[Energycharges]

This is the flow i created .

when internet goes down , msg getting queue and when internet comes back queue is not getting cleared

Thanks

[Colin]

Does it work when the internet is not down?
If so, once there are some queued up and the internet is up again, if you click the start processing button does it send them?

[Energycharges]

Thanks for your response Colin.

Queueing problem returning back, working on it .

I will check the flow , and will update you .

Regards,
Energy Charges

[Colin]

I have realised that the sample flow is not sufficient to cope with what you want. I will try and come up with an enhanced one that automatically starts up when the internet comes back as well as starting automatically on node-red start

[Energycharges]

Thank you so much for the kind update Colin.

Please let me know once your done.
Meanwhile you can send me any reference , so that i can try on my side

Regards,
EnergyCharges

[Colin]

The issue is to make it automatically send in a Start Processing message when the first message is queued, and to also make that happen if there is a failure actioning the message (internet is down for example) and then the internet comes back up. I should work it out a bit later today.

I am not sure I understood your reply to my earlier question, does what you have work ok if the internet is up, you queue some messages, and then click the start Inject button?

[Energycharges]

Thanks for your reply Colin

Yes , your right. Looking for the same issue.

When internet is down , i need the data to be queued , once the internet is up i need the q-gate to allow data Automatically one by one .

Regards,
Energycharges

[Colin]

Try this. The top section is the queuing logic. Feed your messages in the top left hand corner. When the internet is down it will retry at the interval specified in the Retry inject node, it is currently set to 15 seconds, you probably want much longer than that.

Replace the bottom section with your processing nodes. Connect the output of the link at the top left of the bottom section to your process (instead of the delay node labelled 'process' and if the processing succeeds then send a message to the OK link at the bottom right, and if it fails then send a message to the Failed link there. So you can delete all the nodes in the bottom section except the three link nodes and put your process in there instead.

When you send in the first message there may be a delay of up to the Retry interval before it actions that message. If that is a problem for you then that can be handled fairly easily.

[{"id":"f82aed15.9f1a98","type":"q-gate","z":"bdd7be38.d3b55","name":"q-gate","controlTopic":"control","defaultState":"queueing","openCmd":"open","closeCmd":"close","toggleCmd":"toggle","queueCmd":"queue","defaultCmd":"default","triggerCmd":"trigger","flushCmd":"flush","resetCmd":"reset","peekCmd":"peek","dropCmd":"drop","statusCmd":"status","maxQueueLength":"100","keepNewest":false,"qToggle":false,"persist":false,"x":410,"y":1440,"wires":[["ae4b0558.378d8","b6aba10c.b691f"]]},{"id":"6ec96b3b.ec5c14","type":"debug","z":"bdd7be38.d3b55","name":"OK","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":830,"y":2080,"wires":[]},{"id":"fed5956.ed73668","type":"inject","z":"bdd7be38.d3b55","name":"inject messages","props":[{"p":"payload","v":"","vt":"date"},{"p":"topic","v":"","vt":"string"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":160,"y":1440,"wires":[["f82aed15.9f1a98"]]},{"id":"4ef6d24.5459eac","type":"inject","z":"bdd7be38.d3b55","name":"Retry","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"15","crontab":"","once":false,"onceDelay":0.1,"topic":"control","payload":"peek","payloadType":"str","x":170,"y":1620,"wires":[["a20949f9.306a48"]]},{"id":"4270c3f6.19012c","type":"delay","z":"bdd7be38.d3b55","name":"process","pauseType":"delay","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":340,"y":1920,"wires":[["b0c78702.b01988"]]},{"id":"a8381ab7.d1949","type":"link out","z":"bdd7be38.d3b55","name":"","links":["5a2f45b9.929cc4"],"x":795,"y":2040,"wires":[]},{"id":"5a2f45b9.929cc4","type":"link in","z":"bdd7be38.d3b55","name":"","links":["a8381ab7.d1949"],"x":95,"y":1500,"wires":[["1ec5d8f7.0e6ae7"]]},{"id":"1ec5d8f7.0e6ae7","type":"function","z":"bdd7be38.d3b55","name":"get next","func":"node.send({topic: \"control\", payload: \"drop\"})\nmsg.topic = \"control\"\nmsg.payload = \"peek\"\nreturn msg;","outputs":1,"noerr":0,"x":200,"y":1500,"wires":[["f82aed15.9f1a98"]]},{"id":"672111d5.4f229","type":"inject","z":"bdd7be38.d3b55","name":"Simulate internet up","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"internet","payload":"up","payloadType":"str","x":230,"y":1980,"wires":[["edb0af2.fd1645"]]},{"id":"bc5fa483.adc8f","type":"inject","z":"bdd7be38.d3b55","name":"Simulate internet down","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"internet","payload":"down","payloadType":"str","x":230,"y":2020,"wires":[["edb0af2.fd1645"]]},{"id":"edb0af2.fd1645","type":"join","z":"bdd7be38.d3b55","name":"","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":true,"timeout":"","count":"2","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":470,"y":2000,"wires":[["b7feed8e.7caf68"]]},{"id":"b0c78702.b01988","type":"change","z":"bdd7be38.d3b55","name":"topic:data","rules":[{"t":"set","p":"topic","pt":"msg","to":"data","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":480,"y":1920,"wires":[["edb0af2.fd1645"]]},{"id":"b7feed8e.7caf68","type":"switch","z":"bdd7be38.d3b55","name":"Data message?","property":"topic","propertyType":"msg","rules":[{"t":"eq","v":"data","vt":"str"}],"checkall":"true","repair":false,"outputs":1,"x":380,"y":2080,"wires":[["d96e80be.41aff"]]},{"id":"d96e80be.41aff","type":"switch","z":"bdd7be38.d3b55","name":"OK/Fail?","property":"payload.internet","propertyType":"msg","rules":[{"t":"eq","v":"up","vt":"str"},{"t":"else"}],"checkall":"true","repair":false,"outputs":2,"x":560,"y":2080,"wires":[["6ec96b3b.ec5c14","a8381ab7.d1949"],["ea9c5a96.2dcfe8","37735513.24d77a"]]},{"id":"ea9c5a96.2dcfe8","type":"debug","z":"bdd7be38.d3b55","name":"FAIL","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":830,"y":2120,"wires":[]},{"id":"a20949f9.306a48","type":"gate","z":"bdd7be38.d3b55","name":"","controlTopic":"gate_control","defaultState":"open","openCmd":"open","closeCmd":"close","toggleCmd":"toggle","defaultCmd":"default","persist":false,"x":530,"y":1620,"wires":[["f82aed15.9f1a98"]]},{"id":"ae4b0558.378d8","type":"change","z":"bdd7be38.d3b55","name":"Close gate","rules":[{"t":"set","p":"topic","pt":"msg","to":"gate_control","tot":"str"},{"t":"set","p":"payload","pt":"msg","to":"close","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":390,"y":1580,"wires":[["a20949f9.306a48"]]},{"id":"a78ce065.5c9b68","type":"change","z":"bdd7be38.d3b55","name":"Open gate","rules":[{"t":"set","p":"topic","pt":"msg","to":"gate_control","tot":"str"},{"t":"set","p":"payload","pt":"msg","to":"open","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":400,"y":1660,"wires":[["a20949f9.306a48"]]},{"id":"e5978b76.2fb2b","type":"link in","z":"bdd7be38.d3b55","name":"","links":["b6aba10c.b691f"],"x":135,"y":1920,"wires":[["4270c3f6.19012c"]]},{"id":"b6aba10c.b691f","type":"link out","z":"bdd7be38.d3b55","name":"","links":["e5978b76.2fb2b"],"x":615,"y":1440,"wires":[]},{"id":"37735513.24d77a","type":"link out","z":"bdd7be38.d3b55","name":"","links":["7f6b726d.0cab2c"],"x":795,"y":2160,"wires":[]},{"id":"7f6b726d.0cab2c","type":"link in","z":"bdd7be38.d3b55","name":"","links":["37735513.24d77a"],"x":95,"y":1660,"wires":[["a78ce065.5c9b68"]]},{"id":"5d647435.11ea14","type":"comment","z":"bdd7be38.d3b55","name":"Processed ok","info":"","x":690,"y":2020,"wires":[]},{"id":"cda552d1.59b25","type":"comment","z":"bdd7be38.d3b55","name":"Failed","info":"","x":670,"y":2160,"wires":[]},{"id":"6777d4.5738d02c","type":"comment","z":"bdd7be38.d3b55","name":"Replace this with your message processing, sending a message to the Fail or Success outputs","info":"","x":350,"y":1820,"wires":[]},{"id":"d1158219.184d38","type":"comment","z":"bdd7be38.d3b55","name":"Messages to be processed come in here","info":"","x":200,"y":1880,"wires":[]},{"id":"4681668e.9d5cd8","type":"comment","z":"bdd7be38.d3b55","name":"Feed messages to be queued in here and remove Inject node","info":"","x":270,"y":1400,"wires":[]},{"id":"177f73a0.c29e2c","type":"status","z":"bdd7be38.d3b55","name":"","scope":["f82aed15.9f1a98"],"x":160,"y":1720,"wires":[["b95b8054.e3339"]]},{"id":"b95b8054.e3339","type":"function","z":"bdd7be38.d3b55","name":"Queue empty?","func":"// expects msg.status.text in form \"queuing: <number>\"\n// sends a message if queue is empty\nif (parseInt(msg.status.text.split(\":\")[1]) !== 0) {\n    msg = null\n}\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":340,"y":1720,"wires":[["a78ce065.5c9b68"]]},{"id":"e72ba10d.c92638","type":"comment","z":"bdd7be38.d3b55","name":"Set this to appropriate retry interval","info":"","x":150,"y":1580,"wires":[]}]

[Energycharges]

Thanks Colin for you kind response and effort .

I got the output for queueing through Q-gate. with help of your logics , finally my data was queued automatically and stored in my cloud db.

I have a new issue hope you can sort it out.

while connecting modbus , i got 1 part of the output , other part need to convert and store harmonics data into csv file.

Can you please help me out?

Running separately im getting output in debug , while connecting along with function node its showing NaN error , or sometimes no output.

I guess i need to do some modifications in function node

Can u please guide me , to get my data stored into csv file .

Regards,
Energycharges

[Colin]

Glad that is working. For an unrelated question please start a new thread so that someone with the relevant experience can reply.