[ANNOUNCE] node-red-contrib-multiple-queue

[drmibell]

This node provides multiple, independently controlled message queues. Each queue has the capabilities of the queueing mode of the q-gate node, with some additional features. The design goals were:

Flexible queue structure
The user does not need to define in advance how many queues are required or how they are identified.

Economy of memory
Queues are created as needed on a "per topic" basis and destroyed when empty. (Messages are routed using msg.topic by default, but any other message property can be used.) The size of each queue can be limited, and its behavior when full can be defined.

Complete command set
Each queue is controlled using the command set of the queueing mode of the q-gate node: trigger, status,flush, peek, drop, and reset. In addition, commands are provided to allow each individual queue to be paused or to resume queueing and to modify the settings that control its size limit and behavior when full or empty. Any of these commands can also be addressed to all queues simultaneously.

Flexible command set
None of the control message properties or payloads is "hard-wired" in the code. The edit dialog allows the user to define the message property and payload used for each command.

Enhanced status reporting
An option is provided to create a second output that delivers a status object containing the requested message queue and all the queue-specific settings.

Persistence
An option allows the queues and node settings to be restored from a non-volatile context store, if available, after a system or Node-RED restart.

Thanks to @nileio for suggesting this project. Please report bugs or leave suggestions here or on GitHub.

[Paul-Reed]

I don't want to sound unappreciative, but couldn't this of been an update to the existing q-gate node?
We seem to have an increasing number of gate nodes with similar functionality, and users may find it difficult choosing!

[drmibell]

Please see the discussion (issue) linked in my post. We thought about that approach, and others, but decided that a dedicated node was best in terms of design and useabiliy. For users who need only a gate or a single queue, learning the interface of the "one node to rule them all" could be more intimidating than deciding up front what functionality they need.

[Paul-Reed]

Thanks Mike, I did read the discussion earlier, but wondered if those obstacles could have been overcome.

I guess I'm also looking at it from the angle that I already use your q-gate node in a number of flows, but if I now want any of the new features I have to install yet another gate node (or update my existing flows to use this node).

[drmibell]

@Paul-Reed

wondered if those obstacles could have been overcome

Frankly, I tried and failed. I'm always open to suggestions or PR's, though.

@Paul-Reed

install yet another gate node (or update my existing flows... )

The new features all relate to queueing, not gating, and really solve a different problem than the gate nodes. Think in terms of mqtt, where you have an arbitrary set of message streams (how many not known in advance), identified by msg.topic values, also not fully known in advance. If these things were known, you could use some fixed number of q-gate and perhaps switch nodes and have multiple, independent queues. If not, use m-queue. Either way, I don't see the need to replace existing nodes.

The main technical issue I encountered is that gate and and q-gate use msg.topic to distinguish between ordinary message traffic and control messages. For better or worse, that choice was made early on. To free up msg.topic to select among queues would break all existing flows that use the gate nodes. Requiring a different property for queue selection would make life difficult for users of mqtt or other pub/sub services, who are the ones most likely to be interested in m-queue. Ultimately, the value of simplicity and doing one thing well seemed to outweigh the cost of adding a node to the pallete.

[PS] I do sympathize with the "cluttered palette" problem. I've complained about it myself. So when I considered packaging all three nodes together so the user wouldn't need to decide which one to download, I decided to do no harm.

[Paul-Reed]

Regardless of my feedback Mike, your nodes are very much appreciated, and there's always space for them in my palette

[drmibell]

You are too kind...

[Colin]

I think this will be a useful node for a particular set of problems. One issue I have is that I am trying to startup with a queue paused, but if I send a pause command to a queue that has not yet been created then the node errors out. I am wondering whether, if Protect empty queues is enabled, then pause and resume should create an empty queue if necessary. I have had to use a workaround of sending the node a dummy message, resetting it and then pausing it.
I also see that other commands such as maximum cannot be applied before the queue is created, which could be a problem in some cases. One can imagine wanting setting that up in a startup flow.

[drmibell]

I think I understand what you want to accomplish. The original idea was to let the creation and destruction of queues be determined entirely by the flow of messages being queued. Clearly, the protect and delete commands break the rule, so it's just a question of what other violations might be useful. How about a create command? It would create an empty queue with its name taken from msg.topic or whatever other property was defined as the Queue Selector. It would be logical to have its protect property set, since the user probably wants his new empty queue to exist until he deliberately deletes it. Then maxSize and keepNewest can be set to the node's defaults, leaving only the value of paused as an issue. In your use case, paused = true would be convenient, but I would vote for paused = false, since it would allow the queue to begin queueing messages as soon as it is created. Any thoughts?