Issue with Persistent Context when using q-gate

General
1

Hi @zenofmud. I'm little confused over cache. If I set it to false. wouldn't it should store my values instantaneously?

Case: 1
but here is my error on this code. and it is not storing.

[{"id":"2ad176d6.2631aa","type":"function","z":"8872dfa.15cfd2","name":"","func":"let msgCount = context.get(\"msgCount\") || 0;//get remembered count\nmsgCount++; //increment count\n\ncontext.set('msgCount', msgCount);\n\nmsg.payload = msgCount;\nreturn msg;\n\n","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":460,"y":300,"wires":[["abccbeba8b1946e7","d7d6820b747c07e8"]]},{"id":"d7d6820b747c07e8","type":"q-gate","z":"8872dfa.15cfd2","name":"","controlTopic":"control","defaultState":"queueing","openCmd":"open","closeCmd":"close","toggleCmd":"toggle","queueCmd":"queue","defaultCmd":"default","triggerCmd":"trigger","flushCmd":"flush","resetCmd":"reset","peekCmd":"peek","dropCmd":"drop","statusCmd":"status","maxQueueLength":"","keepNewest":true,"qToggle":false,"persist":true,"storeName":"file","x":550,"y":460,"wires":[["73417c9d5d868f88"]]},{"id":"3a69d2dbe6a2c80a","type":"inject","z":"8872dfa.15cfd2","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payloadType":"date","x":260,"y":280,"wires":[["2ad176d6.2631aa"]]},{"id":"73417c9d5d868f88","type":"debug","z":"8872dfa.15cfd2","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":790,"y":460,"wires":[]},{"id":"08eef9e2a6c18ea2","type":"inject","z":"8872dfa.15cfd2","name":"flush","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"control","payload":"flush","payloadType":"str","x":250,"y":460,"wires":[["d7d6820b747c07e8"]]},{"id":"abccbeba8b1946e7","type":"debug","z":"8872dfa.15cfd2","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":850,"y":300,"wires":[]}]

Error: File Store cache disabled - only asynchronous access supported

and
Case 2:
If I set cache to default and flush interval to 5 or 10 seconds. It is not flushing my all queue values. Suppose I push values at 1 sec rate. the after 20 counts I stop nodered and restart it. but It is never gonna give me all 20 values.

in my setting.js

image
image725Ă—93 10.3 KB

code is same above.

Is there any limitation on minimum time?

Using ubuntu on Virtualbox.

2

The error means what it says. If you disable the cache, the store can only be accessed in async mode as described here: Writing Functions : Node-RED

You will have to change your code to match.

How are you stopping Node-RED? Is it a clean stop or are you abruptly killing the process or killing the VM? If it isn't a clean stop, then Node-RED won't be able to flush out anything still in the cache.

3

Yes It is node-red-stop. and then node-red-restart.

4

I am seeing the same issue with the test flow. However, it appears to be a problem only with the q-gate node as if I modify the function to use global context in the 'file' store for the message count, then that aspect works ok. It is only the q-gate that does not keep its queue.

I suggest that you raise an issue on the node's github page. Or perhaps @drmibell is listening.

5

I'm listening, but I haven't been following. Will the issue be obvious if I run the test flow, or should I be asking for more description?

@pranavthakkar, what is in your settings.js in Case 1?

6

It was for me, yes. With the cache timeout set to 5 click the inject about 20 times in quick succession then immediately restart node red. On restart, the queued message count shown in the status drops from 20 back to zero or a small number. With the timeout set to 600 it works ok.

I haven't experimented enough to know whether the timing is critical but it seems to fail reliably.

[Edit] This is for case 2. With settings as posted in post #86

7

Hi @drmibell . In my case: 1 I just provided cache:false
in config of file.

In that It is never queuing the values.

8

[NOTE] - I've move this to a seperate topic since it seems to be related to the q-gate node.

9

Thanks, Paul. Is it worth fixing the typo in the title? I worry that it might break links elsewhere in the forum.

10

[admin] spelling in title corrected

11

As @knolleary says, for the moment the solution to Case 1 is "don't." Neither your function nor the queue-gate node is able to deal with asynchronous access (cache = false). The reason you cannot queue messages is that the error prevents the node from registering its listener for input events. Once this happens, it will not respond to any incoming messages, including control messages. The only fix I have found is to delete the node and replace it with a new instance.

12

@drmibell are you able to replicate case 2 or do you want me to attempt to narrow it down if possible?

13

Please have a go at it and let me know what you find. I have not been able to reproduce Case 2 on my development machine (mac laptop), so I plan to switch to a RPi. I can think of at least a couple of ways that this could depend on the OS, so I'll keep trying.

14

This may be a clue. I have a 5 second flushInterval. I have modified the function in the flow to save its msgCount in persistent node context also (and this works correctly). I have set the log level in node-red to Trace.

If I click the inject button a few times then in the trace log I see it flushing the function node context a few seconds later, but the q-gate node does not get flushed (apart from the fact that it sometimes gets flushed at the first interval after startup). However, if I click the Flush inject, which flushes the queue in the gate then I do see the gate node context getting flushed, and looking at the file itself I can see it has correctly flushed it. So presumably that means there is a difference in the way the context is handled in the node for adding to the queue vs flushing it.

15

Aha! It is more subtle than I thought. I think it is not directly related to the size of the flushInterval, but only accidentally.

I think if node-red is started and the queue is added to and then node-red is restarted before the first flushInterval has elapsed then it works correctly. However if node-red is started and then it is left till after the first interval has elapsed and then the queue is added to then that is the situation where the queue is not flushed on restart (or at all I think). I am not absoulutely certain I have fully analysed it but it is something like that anyway, and I am out of time at the moment.

With a short flushInterval this is a much more likely occurrence which is why it is easy to reproduce with a 5 second interval.

16

Thanks! I think you've solved it. The published version of the node is not saving the queue properly. Unfortunately, I have been working with a new branch that "accidentally" eliminates the bug as part of a major refactoring. It was foolish to assume that nothing had changed in that regard. As soon as I figure out how to use git properly to coordinate work on two branches, I'll publish a fix.

17

I have just published node-red-contrib-queue-gate version 1.5.4, which should resolve the issue described in @pranavthakkar's Case 2 and the behavior found by @Colin . Thanks both for helping fix this bug.

I still have questions about Case 1. The node could catch the error caused by cache = false, but what should it do about it? Also, does that mean that contributed nodes should always use the asynchronous forms of context.set and context.get? Any thoughts @knolleary?

18

Working now for me (Case 2).

1 Like
19

Thanks Mike. It is working on Case 2.

1 Like
20

I have just published node-red-contrib-queue-gate version 1.5.5, which should resolve the issue in Case 1. Since the node has no way of knowing whether the context store selected by the user supports synchronous operations, I have used the form for asynchronous access everywhere. Please let me know if you run into any further issues. I would be very interested in knowing whether you get any changes or improvements by running with the cache disabled.

2 Likes