Context storage for delay module

Hi All,

I'm using a flow where I receive a signal via a webhook that I send to the delay module to add a random delay before sending it to discord for example.
Unfortunately I cannot make the delay module cache the signal in storage instead of memory, so it will be lost during a reboot. How can I fix this? :slight_smile:

Flow:

settings.js:

contextStorage: {
    default: {
        module:"localfilesystem"
    },
},

Thanks in advance!!

Kind regards,
Roy

use a change node before the delay node to save the value. and put the value in msg.delay to pass into the delay node

1 Like

I don't think that is possible just using the delay node.
You may need to save the webhook msg to file context using a change node, then after a random delay, retrieve the file context and send it.

Your context storage settings currently will cache the msg in memory first, and then flush it to the file store after 30 seconds. Within that 30 seconds, if you do a reboot, the shutdown signal will cause the memory cache to be flushed to file storage before shutdown completes.

If however, you want it to immediately write the data to file store instead of caching it first, will need to disable the cache;

contextStorage: {
   default: {
       module:"localfilesystem",
       config: {
           cache: 'false'
       }
   }
}

Edit - Just seen @zenofmud's reply (great minds think alike :laughing:)

1 Like

Thanks @Paul-Reed and @zenofmud .
I made some change but is still does not work, I have the feeling that I'm overlooking something crucial :slight_smile:

    contextStorage: {
        default: {
            module:"localfilesystem",
            config: {
                cache: "false"
                },
        },
    },
[{"id":"584d7a9c.b099d4","type":"tab","label":"test","disabled":false,"info":""},{"id":"9ed75af6.17c3c8","type":"delay","z":"584d7a9c.b099d4","name":"","pauseType":"random","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"2","randomLast":"5","randomUnits":"minutes","drop":false,"x":1000,"y":360,"wires":[["33d9788d.83b238"]]},{"id":"558bb2b2.b2374c","type":"http in","z":"584d7a9c.b099d4","name":"","url":"/webhook","method":"post","upload":false,"swaggerDoc":"","x":620,"y":360,"wires":[["b9ad0ab1.bab5d8"]]},{"id":"b9ad0ab1.bab5d8","type":"change","z":"584d7a9c.b099d4","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"delay","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":360,"wires":[["9ed75af6.17c3c8"]]},{"id":"33d9788d.83b238","type":"debug","z":"584d7a9c.b099d4","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1170,"y":360,"wires":[]},{"id":"1c4363f6.117cec","type":"inject","z":"584d7a9c.b099d4","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"test","payloadType":"str","x":650,"y":320,"wires":[["b9ad0ab1.bab5d8"]]}]

No, not quite correct.
You need to save the incoming msg to file context, then after a delay, retrieve it from file context again.
Try;

[{"id":"9ed75af6.17c3c8","type":"delay","z":"584d7a9c.b099d4","name":"","pauseType":"random","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"5","randomLast":"10","randomUnits":"seconds","drop":false,"x":970,"y":360,"wires":[["b14aec99.dc98a"]]},{"id":"b9ad0ab1.bab5d8","type":"change","z":"584d7a9c.b099d4","name":"","rules":[{"t":"set","p":"test","pt":"flow","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":810,"y":360,"wires":[["9ed75af6.17c3c8"]]},{"id":"33d9788d.83b238","type":"debug","z":"584d7a9c.b099d4","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1300,"y":360,"wires":[]},{"id":"1c4363f6.117cec","type":"inject","z":"584d7a9c.b099d4","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"test","payloadType":"str","x":650,"y":360,"wires":[["b9ad0ab1.bab5d8"]]},{"id":"b14aec99.dc98a","type":"change","z":"584d7a9c.b099d4","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"test","tot":"flow"}],"action":"","property":"","from":"","to":"","reg":false,"x":1130,"y":360,"wires":[["33d9788d.83b238"]]}]

But of course, although your data will be safely saved to file context, your delay node will have forgotten it's delay period once you reboot...

There is however a contrib node - node-red-contrib-stoptimer-varidelay which will continue the delay countdown if node-RED is stopped & restarted, a full deploy or system reboot is made.
It unfortunately has a fixed deploy period, but by injecting a random time setting (msg.delay), it should fulfil what you are asking.

Before trying this flow, don't forget to install node-red-contrib-stoptimer-varidelay and node-red-node-random (if it's not already in your palette) first.

[{"id":"b9ad0ab1.bab5d8","type":"change","z":"a4801683.1c0448","name":"","rules":[{"t":"set","p":"test","pt":"flow","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":220,"y":3160,"wires":[["f81cd210.1a62d"]]},{"id":"33d9788d.83b238","type":"debug","z":"a4801683.1c0448","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":820,"y":3160,"wires":[]},{"id":"1c4363f6.117cec","type":"inject","z":"a4801683.1c0448","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"test","payloadType":"str","x":80,"y":3160,"wires":[["b9ad0ab1.bab5d8"]]},{"id":"b14aec99.dc98a","type":"change","z":"a4801683.1c0448","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"test","tot":"flow"}],"action":"","property":"","from":"","to":"","reg":false,"x":650,"y":3160,"wires":[["33d9788d.83b238"]]},{"id":"f81cd210.1a62d","type":"random","z":"a4801683.1c0448","name":"","low":"60","high":"300","inte":"true","property":"delay","x":370,"y":3160,"wires":[["cd32e888.20eeb8"]]},{"id":"cd32e888.20eeb8","type":"stoptimer-varidelay","z":"a4801683.1c0448","duration":"5","durationType":"num","units":"Second","payloadtype":"num","payloadval":"0","name":"Delay","reporting":"none","persist":true,"x":500,"y":3160,"wires":[["b14aec99.dc98a"],[],[]]}]

This could be done with standard nodes and two injects, It you want to store a random delay, then you could use trigger node instead of delay, and generate the random using Jsonata in a change node. If delay fails to finish the auto inject will pick up failure.
e.g.

[{"id":"9d5d05ca.780f48","type":"change","z":"5a245aa1.510164","name":"","rules":[{"t":"set","p":"send","pt":"flow","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":290,"y":460,"wires":[["8d59c1de.6f28e"]]},{"id":"1c4363f6.117cec","type":"inject","z":"5a245aa1.510164","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"test","payloadType":"str","x":130,"y":460,"wires":[["9d5d05ca.780f48"]]},{"id":"8d59c1de.6f28e","type":"delay","z":"5a245aa1.510164","name":"","pauseType":"random","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"2","randomLast":"5","randomUnits":"minutes","drop":false,"x":460,"y":460,"wires":[["e07d0d66.0aaf8"]]},{"id":"e07d0d66.0aaf8","type":"change","z":"5a245aa1.510164","name":"","rules":[{"t":"set","p":"send","pt":"flow","to":"finished","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":630,"y":460,"wires":[["33d9788d.83b238"]]},{"id":"98d10fdc.7758d","type":"switch","z":"5a245aa1.510164","name":"","property":"send","propertyType":"flow","rules":[{"t":"neq","v":"finished","vt":"str"}],"checkall":"true","repair":false,"outputs":1,"x":480,"y":400,"wires":[["8d59c1de.6f28e"]]},{"id":"33d9788d.83b238","type":"debug","z":"5a245aa1.510164","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":670,"y":400,"wires":[]},{"id":"80cb0429.cbac4","type":"inject","z":"5a245aa1.510164","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":true,"onceDelay":"0.1","topic":"","payload":"send","payloadType":"flow","x":330,"y":400,"wires":[["98d10fdc.7758d"]]}]
1 Like

Thanks @Paul-Reed and @E1cid.
I tried both options and they seem to work, except for one small thing. If there are multiple messages in the delay queue it seems that only one is being forwarded after the reboot. So probably only 1 message is being saved to disk. Is that something that can be fixed as well?
Thanks!!

This may work untested

[{"id":"8d59c1de.6f28e","type":"delay","z":"5a245aa1.510164","name":"","pauseType":"random","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"2","randomLast":"5","randomUnits":"seconds","drop":false,"x":460,"y":460,"wires":[["e07d0d66.0aaf8"]]},{"id":"9d5d05ca.780f48","type":"change","z":"5a245aa1.510164","name":"","rules":[{"t":"set","p":"send","pt":"flow","to":"$append($flowContext(\"send\"),[{\"id\":_msg.id,\"message\":payload}])","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":290,"y":460,"wires":[["8d59c1de.6f28e"]]},{"id":"165ecc71.8db40c","type":"delay","z":"5a245aa1.510164","name":"","pauseType":"rate","timeout":"6","timeoutUnits":"seconds","rate":"1","nbRateUnits":"10","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":740,"y":400,"wires":[["8d59c1de.6f28e"]]},{"id":"e07d0d66.0aaf8","type":"change","z":"5a245aa1.510164","name":"","rules":[{"t":"set","p":"send","pt":"flow","to":"$flowContext(\"send\")[*][id!=_msgid]","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":650,"y":460,"wires":[["8f89d57c.db132"]]},{"id":"1c4363f6.117cec","type":"inject","z":"5a245aa1.510164","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"test","payloadType":"str","x":130,"y":460,"wires":[["9d5d05ca.780f48"]]},{"id":"6b95f972.78532","type":"change","z":"5a245aa1.510164","name":"","rules":[{"t":"move","p":"payload.id","pt":"msg","to":"_msgid","tot":"msg"},{"t":"move","p":"payload.message","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":530,"y":400,"wires":[["165ecc71.8db40c"]]},{"id":"8f89d57c.db132","type":"debug","z":"5a245aa1.510164","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":740,"y":500,"wires":[]},{"id":"fef21770.fe9138","type":"split","z":"5a245aa1.510164","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":370,"y":400,"wires":[["6b95f972.78532"]]},{"id":"80cb0429.cbac4","type":"inject","z":"5a245aa1.510164","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":true,"onceDelay":"0.1","topic":"","payload":"send","payloadType":"flow","x":230,"y":400,"wires":[["fef21770.fe9138"]]}]

[edit] added a delay/rate to missed messages.

This is a different approach, using node-red-contrib-queue-gate.

The q-gate node has persistence built in, and will queue the incoming messages and survive a reboot.
The top inject node injects a control msg telling the q-gate to release one message at a time.
You will just need to ensure that the msg's are released more frequently than they arrive, otherwise you will get a backlog of msg's queued in the q-gate, so play around with the timing.

queue

[{"id":"d4f8e272.1b5ef","type":"inject","z":"c9c4eeb8.bad8e","name":"Example data","props":[{"p":"payload"}],"repeat":"30","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"example data","payloadType":"str","x":210,"y":1960,"wires":[["2e41a519.54152a"]]},{"id":"44ebb53c.95638c","type":"debug","z":"c9c4eeb8.bad8e","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":520,"y":1960,"wires":[]},{"id":"f88216ee.027e48","type":"delay","z":"c9c4eeb8.bad8e","name":"","pauseType":"random","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"30","randomLast":"60","randomUnits":"seconds","drop":false,"x":420,"y":1900,"wires":[["2e41a519.54152a"]]},{"id":"2e41a519.54152a","type":"q-gate","z":"c9c4eeb8.bad8e","name":"","controlTopic":"control","defaultState":"queueing","openCmd":"open","closeCmd":"close","toggleCmd":"toggle","queueCmd":"queue","defaultCmd":"default","triggerCmd":"trigger","flushCmd":"flush","resetCmd":"reset","peekCmd":"peek","dropCmd":"drop","statusCmd":"status","maxQueueLength":"100","keepNewest":false,"qToggle":false,"persist":true,"storeName":"default","x":380,"y":1960,"wires":[["44ebb53c.95638c"]]},{"id":"5bde86a6.96f5c8","type":"inject","z":"c9c4eeb8.bad8e","name":"Release next msg","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"25","crontab":"","once":true,"onceDelay":0.1,"topic":"control","payload":"trigger","payloadType":"str","x":220,"y":1900,"wires":[["f88216ee.027e48"]]}]

This worked! I only had to make a small modification because the output was [object Object] when sent from disk.

Thanks @E1cid @Paul-Reed and @zenofmud

Yes i already picked up that error and corrected, i also removed the switch node as unneeded. i updated the post with the updated corrected flow.