Delay logic request for suggestions

General
1

Background
I am storing static information about IOT items in a google sheet. I have a process to read that information into a global variable and then use that variable as a local data source. As I add new IOT devices, I need to add them into the google sheet. In my original, design, I assumed that I would enter all of the info into the sheet before needing the data. I now realize that it would be easier to build stub entries when the new device doesn't exist. I have been able to build a process to do this. The challenge I have run into is that when I have a new device, it typically does the lookup repeatedly because it does it once per IOT attribute (ie. temperature, humidity). I want to prevent this as I have managed to run up against my quota for sheet access.

Partial Solution
I have come up with this partial solution. First I take the stream of 1-N input msgs and send to 2 paths. The first is a delay node that holds (quasi-temporarily (5 minutes)) all of the msg's on the other path, I use an rbe node to turn the stream into a single msg per device name (the key for my lookup). I then, look to see if the device is already in the spreadsheet. When it is not, I use information to append my stub entry (3 of the desired 10 fields populated) to the spreadsheet. I then reload the global variable with the updated data. Finally, I send msg.flush to the delay node to get the delayed msgs on their way.

Use Case That Needs a Suggestion
The process above works very well when I add a single device. I foresee a time when I would like to add mutiple devices at the same time. When I do this, within my current design, completion of the process for device 1 will flush to held msgs for both device 1 and devices 2-N. Anyone have suggestions on how to handle this for an arbitrary number of devices with arbitrary names?

2

Shooting from the hip:

  • First thought: to prevent sending each attribute separately you could collect the data and use the rbe node filter based on the message topic. However, you'd need to know when the final attribute has been read.If you know that, this may work.

  • Second thought: instead of setting a delay you may want to use the Node-Red-Contrib-Timeout. Here you can set a specific time waiting for the last data to come in and send the data as soon as the timeout fires.

3

Also the delay node can be put into rate limit mode where it can save any topics that arrive (overriding any repeats so only the newest value per topic is held) and then output them either singly or all at the same time. I don't think this handles the whole use case but may make some parts easier.

1 Like
4

My challenge is that I have msgs 1.1, 1.2, 2.1, 2.2 and I want to process all of them eventually, but first I need to do a process for 1 before processing 1.1 and 1.2 and process 2 before 2.1 and 2.2. The delay node works well for holding them while doing the processing, but I can't figure out a way to release 1.1 and 1.2 while still holding 2.1 and 2.2.

I can imagine using a switch node with a 1 delay node for each of 1 and 2, but this requires knowing ahead of time what differentiates 1 and 2, but not sure about arbitrary inputs.

5

This looks to me that you will need to do some programing. Pushing the messages into an array (stored as a context variable) and shifting the messages out one by one eventually.

6

Perhaps node-red-contrib-queue-gate would be useful. You can feed a sequence of messages in and release one, service it, and when you have finished tell the node to release the next one. That is done by sending the Trigger command to the node.

7

@Colin
Thanks for the suggestion. With the addition of the queue-gate I was able to build what I wanted. It was still a little complex so I would welcome feedback from anyone that wants to look at this flow that simulates the scenario.

I no one offers simplifying suggestions, then it is just here for reference of anyone in the future that looks to do this sort of thing. The flow uses only base nodes plus the node-red-contrib-queue-gate.

image
image1083Ă—227 48.4 KB

[{"id":"d2affe59.39464","type":"tab","label":"QGate Hold during Prep","disabled":false,"info":""},{"id":"eb605d01.2844","type":"link out","z":"d2affe59.39464","name":"Send to q gate","links":["1c21548b.7b897b"],"x":620,"y":100,"wires":[],"l":true},{"id":"1c21548b.7b897b","type":"link in","z":"d2affe59.39464","name":"Send to q gate","links":["eb605d01.2844"],"x":160,"y":220,"wires":[["f5ef2908.a04478"]],"l":true},{"id":"b236836b.2f5b5","type":"q-gate","z":"d2affe59.39464","name":"q-gate","controlTopic":"control","defaultState":"queueing","openCmd":"open","closeCmd":"close","toggleCmd":"toggle","queueCmd":"queue","defaultCmd":"default","triggerCmd":"trigger","flushCmd":"flush","resetCmd":"reset","peekCmd":"peek","dropCmd":"drop","statusCmd":"status","maxQueueLength":"100","keepNewest":false,"qToggle":false,"persist":false,"x":710,"y":220,"wires":[["f99106b2.a5d188"]]},{"id":"7b182802.c8dbe8","type":"debug","z":"d2affe59.39464","name":"simulate rest of the process via debug output","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":1590,"y":100,"wires":[]},{"id":"e9ee8041.96b0c","type":"inject","z":"d2affe59.39464","name":"simulate inputs IOT1","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":190,"y":100,"wires":[["72c3f63d.72db08"]]},{"id":"9a380412.039218","type":"delay","z":"d2affe59.39464","name":"simulate a process via 1 second delay ","pauseType":"delay","timeout":"1","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":930,"y":280,"wires":[["80859744.f41e08","8d3467fb.b86258"]]},{"id":"1c71b0e0.31a8cf","type":"link out","z":"d2affe59.39464","name":"Send Control Message to QGate","links":["f34552ea.2604b"],"x":1850,"y":320,"wires":[],"l":true},{"id":"f34552ea.2604b","type":"link in","z":"d2affe59.39464","name":"Send Control Message to QGate","links":["1c71b0e0.31a8cf"],"x":210,"y":260,"wires":[["f5ef2908.a04478"]],"l":true},{"id":"e5651cb2.1ad19","type":"function","z":"d2affe59.39464","name":"drop processed and get next","func":"//send a drop to drop the \nnode.send({topic: \"control\", payload: \"peek\"})\nnode.send({topic: \"control\", payload: \"drop\"})\nreturn","outputs":1,"noerr":0,"x":1540,"y":300,"wires":[["1c71b0e0.31a8cf"]]},{"id":"ea5a4f86.f1cd1","type":"inject","z":"d2affe59.39464","name":"simulate inputs IOT2","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":190,"y":160,"wires":[["d389f7f8.94e568"]]},{"id":"7fb5d273.f7082c","type":"rbe","z":"d2affe59.39464","name":"RBE based on Device","func":"rbe","gap":"","start":"","inout":"out","property":"Device","x":400,"y":340,"wires":[["ab48dae3.b17418"]]},{"id":"bb8c920d.503d5","type":"link out","z":"d2affe59.39464","name":"Send to prep process","links":["46633fa1.ac5e3"],"x":640,"y":160,"wires":[],"l":true},{"id":"46633fa1.ac5e3","type":"link in","z":"d2affe59.39464","name":"Send to prep process","links":["bb8c920d.503d5","2e13f9a4.b01146"],"x":180,"y":340,"wires":[["7fb5d273.f7082c"]],"l":true},{"id":"d99ce47b.dbbbb8","type":"change","z":"d2affe59.39464","name":"Increment flow.Counter","rules":[{"t":"set","p":"Counter","pt":"flow","to":"$flowContext(\"Counter\") + 1","tot":"jsonata"},{"t":"set","p":"payload","pt":"msg","to":"Counter","tot":"flow"}],"action":"","property":"","from":"","to":"","reg":false,"x":290,"y":400,"wires":[["bedcefa3.f0b04"]]},{"id":"72c3f63d.72db08","type":"change","z":"d2affe59.39464","name":"Set Device = IOT1","rules":[{"t":"set","p":"Device","pt":"msg","to":"IOT1","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":390,"y":100,"wires":[["eb605d01.2844","bb8c920d.503d5"]]},{"id":"d389f7f8.94e568","type":"change","z":"d2affe59.39464","name":"Set Device = IOT2","rules":[{"t":"set","p":"Device","pt":"msg","to":"IOT2","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":390,"y":160,"wires":[["eb605d01.2844","bb8c920d.503d5"]]},{"id":"bedcefa3.f0b04","type":"delay","z":"d2affe59.39464","name":"simulate prep process via 3 second delay ","pauseType":"delay","timeout":"3","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":580,"y":400,"wires":[["fd298c26.e29db"]]},{"id":"fd298c26.e29db","type":"change","z":"d2affe59.39464","name":"Decrement flow.Counter","rules":[{"t":"set","p":"Counter","pt":"flow","to":"$flowContext(\"Counter\") - 1","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":870,"y":400,"wires":[["80859744.f41e08"]]},{"id":"80859744.f41e08","type":"switch","z":"d2affe59.39464","name":"Any Prep Processes Pending","property":"Counter","propertyType":"flow","rules":[{"t":"eq","v":"0","vt":"num"},{"t":"else"}],"checkall":"true","repair":false,"outputs":2,"x":1260,"y":320,"wires":[["e5651cb2.1ad19"],["cd35bafa.3199b8"]],"outputLabels":["No","Yes"]},{"id":"cd35bafa.3199b8","type":"function","z":"d2affe59.39464","name":"drop processed only","func":"//send a drop to drop the \nnode.send({topic: \"control\", payload: \"drop\"})\nreturn","outputs":1,"noerr":0,"x":1520,"y":340,"wires":[["1c71b0e0.31a8cf"]]},{"id":"f5ef2908.a04478","type":"change","z":"d2affe59.39464","name":"Increment flow.QueueCounter","rules":[{"t":"set","p":"QueueCounter","pt":"flow","to":"$flowContext(\"QueueCounter\") + 1","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":490,"y":220,"wires":[["b236836b.2f5b5"]]},{"id":"f99106b2.a5d188","type":"change","z":"d2affe59.39464","name":"Decrement flow.QueueCounter","rules":[{"t":"set","p":"QueueCounter","pt":"flow","to":"$flowContext(\"QueueCounter\") - 1","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":910,"y":220,"wires":[["9a380412.039218"]]},{"id":"8d3467fb.b86258","type":"switch","z":"d2affe59.39464","name":"Check if Queue is Empty","property":"QueueCounter","propertyType":"flow","rules":[{"t":"eq","v":"0","vt":"num"},{"t":"else"}],"checkall":"true","repair":false,"outputs":2,"x":1270,"y":140,"wires":[["7b182802.c8dbe8"],["f72c6696.4e8b78","7b182802.c8dbe8"]],"outputLabels":["Yes","No"]},{"id":"f72c6696.4e8b78","type":"change","z":"d2affe59.39464","name":"Set Device = Reset","rules":[{"t":"set","p":"Device","pt":"msg","to":"Reset","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":1510,"y":160,"wires":[["2e13f9a4.b01146"]]},{"id":"33ee8dc2.b27532","type":"comment","z":"d2affe59.39464","name":"Device = Reset is sent to allow RBE to pass the same IOT Device if an input comes in after the queue has been emptied.","info":"","x":1540,"y":200,"wires":[]},{"id":"2e13f9a4.b01146","type":"link out","z":"d2affe59.39464","name":"Send to prep process","links":["46633fa1.ac5e3"],"x":1720,"y":160,"wires":[],"l":true},{"id":"ab48dae3.b17418","type":"switch","z":"d2affe59.39464","name":"Check if Device is a reset and thus no processing is needed","property":"Device","propertyType":"msg","rules":[{"t":"eq","v":"Reset","vt":"str"},{"t":"else"}],"checkall":"true","repair":false,"outputs":2,"x":740,"y":340,"wires":[[],["d99ce47b.dbbbb8"]],"outputLabels":["Yes","No"]},{"id":"a47b39ee.35afe8","type":"config","z":"d2affe59.39464","name":"","properties":[{"p":"Counter","pt":"flow","to":"0","tot":"num"}],"active":true,"x":150,"y":40,"wires":[]}]
8

It should not need to be that complex, there is no need to keep track of the number in the queue (the node knows when it is empty so no need for you to keep track of it too). Have a look at the last example in the node's readme ( Retain Until Processed)

9

Colin...I agree in many circumstances the keeping track of the number of items in the queue is unnecessary, but I think in this case I need to. The reason is that I am using the calculated empty queue to send a reset to the RBE node. Without it if you have this sequence, there would be 1 or more messages stuck in the queue:

  1. 1-N messages come into the queue. All as type IOT1.
  2. The prep process completes starting the draining of the queue
  3. After the queue is empty, another message comes in with type IOT1. This message is queued
  4. The RBE node, seeing yet another IOT1 message does not pass it to the prep process
  5. Without the message going through the prep process that IOT1 message would remain in the queue until a new message comes in that passes through the RBE node.

Thus I use the queue count reaching 0 to trigger the reset message going to the RBE node. This then allows the next IOT1 through even it that was the last type sent to prep process.

Do you see an alternative to doing it in this way?

10

I haven't got time for an in-depth look at the moment I am afraid, but did you try the example I pointed to and see how that works, and convinced yourself that technique would not work for you?

11

Yes, I actually started with that example to build what I uploaded here. The example you pointed to has an inject node to start the processing of the items in the queue. All of my complexity is built to send the inputs to both the queue gate and a prep process that can in turn extract items from the queue gate once the prep is done.

12

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.