Storing latest messages by topic for a possible request

General
1

Hi everyone,

I was recently searching for a solution to store latest messages by topic (or possibly any other configurable unique key) to be able to request these messages on demand. At first I was pretty confident that somehow this could be done with a standard node.

I played around with the rbe node which obviously stores messages by topic if the property "Apply mode for each msg.topic separately" is set. However, I could not find a method to get these stored messages.

In the join node's manual mode it is possible to trigger stored messages by sending a message with the msg.complete property. However the join node just concatenates incoming messages and doesn't overwrite messages by topic.

The trigger node seemed also promising with the following properties:
Send -> nothing
then -> wait for
then send -> the latest msg object
Handling -> each msg.topic
However, the trigger node only supports cyclic time triggers. So it cannot be triggerd by an incoming message similarly to the join node's manual mode.

Did I miss something?
What I ended up doing was to write a custom function node that uses context. Whenever it receives a message with a msg.trigger property, it returns all stored messages in an array to msg.payload.

P.S. I am aware that Node-RED is event driven and I am trying to avoid the request response pattern whenever possible. However, I already had a few scenarios where I could not avoid this pattern.

2

It is very easy to store a message or payload by topic by using a function node...

Store msg (or payload) by topic...

flow.set(msg.topic, msg.payload)

Get by topic

msg.payload = flow.get(msg.topic);

Here is a demo (copy + import using ctrl+i)...

[{"id":"7c4135cd.adba1c","type":"inject","z":"553814a2.1248ec","name":"data/shifts","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"data/shifts","payload":"{\"monday\":{\"start\":\"09:00\",\"end\":\"17:00\"},\"tuesday\":{\"start\":\"09:00\",\"end\":\"17:00\"},\"wednesday\":{\"start\":\"09:00\",\"end\":\"17:00\"},\"thursday\":{\"start\":\"09:00\",\"end\":\"17:00\"},\"friday\":{\"start\":\"09:00\",\"end\":\"13:00\"}}","payloadType":"json","x":660,"y":1160,"wires":[["ccb038a8.d93e38"]]},{"id":"a1ae23f1.513c1","type":"inject","z":"553814a2.1248ec","name":"data/setup","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"data/setup","payload":"{\"parameters\":[1,2,3,4,5,6,7,8.9],\"refer\":true,\"a_number\":123}","payloadType":"json","x":660,"y":1200,"wires":[["ccb038a8.d93e38"]]},{"id":"ccb038a8.d93e38","type":"function","z":"553814a2.1248ec","name":"store by topic","func":"flow.set(msg.topic, msg.payload)\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":850,"y":1180,"wires":[[]]},{"id":"d5f41689.f058f8","type":"function","z":"553814a2.1248ec","name":"get by topic","func":"msg.payload = flow.get(msg.topic)\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":850,"y":1260,"wires":[["393650b7.5db25"]]},{"id":"393650b7.5db25","type":"debug","z":"553814a2.1248ec","name":"","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"payload","targetType":"msg","statusVal":"payload","statusType":"auto","x":1020,"y":1260,"wires":[]},{"id":"bf18ce4f.22b63","type":"inject","z":"553814a2.1248ec","name":"data/shifts","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"data/shifts","payload":"","payloadType":"str","x":660,"y":1260,"wires":[["d5f41689.f058f8"]]},{"id":"25187cf7.07cfe4","type":"inject","z":"553814a2.1248ec","name":"data/setup","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"data/setup","payload":"","payloadType":"str","x":660,"y":1300,"wires":[["d5f41689.f058f8"]]}]
3

@Steve-Mcl Thanks, I came up with a similar solution that uses context in a single function node. However, I try to avoid custom function nodes whenever possible. That is why I asked if I missed something.

4

You could use a change node with JSONata but i dont understand your aversion to the function node. It is often the right solution.

PS, there is a new feature in latest node-red where you can use nested properties in the change node

e.g. Set flow.myStore[msg.topic] to msg.payload

Demo...

[{"id":"7c4135cd.adba1c","type":"inject","z":"553814a2.1248ec","name":"data/shifts","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"data/shifts","payload":"{\"monday\":{\"start\":\"09:00\",\"end\":\"17:00\"},\"tuesday\":{\"start\":\"09:00\",\"end\":\"17:00\"},\"wednesday\":{\"start\":\"09:00\",\"end\":\"17:00\"},\"thursday\":{\"start\":\"09:00\",\"end\":\"17:00\"},\"friday\":{\"start\":\"09:00\",\"end\":\"13:00\"}}","payloadType":"json","x":620,"y":1380,"wires":[["355a2efa.ed0ae2"]]},{"id":"a1ae23f1.513c1","type":"inject","z":"553814a2.1248ec","name":"data/setup","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"data/setup","payload":"{\"parameters\":[1,2,3,4,5,6,7,8.9],\"refer\":true,\"a_number\":123}","payloadType":"json","x":620,"y":1420,"wires":[["355a2efa.ed0ae2"]]},{"id":"393650b7.5db25","type":"debug","z":"553814a2.1248ec","name":"","active":true,"tosidebar":true,"console":false,"tostatus":true,"complete":"payload","targetType":"msg","statusVal":"payload","statusType":"auto","x":1010,"y":1500,"wires":[]},{"id":"bf18ce4f.22b63","type":"inject","z":"553814a2.1248ec","name":"data/shifts","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"data/shifts","payload":"","payloadType":"str","x":620,"y":1480,"wires":[["fe7a1f7b.24d1"]]},{"id":"25187cf7.07cfe4","type":"inject","z":"553814a2.1248ec","name":"data/setup","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"data/setup","payload":"","payloadType":"str","x":620,"y":1520,"wires":[["fe7a1f7b.24d1"]]},{"id":"355a2efa.ed0ae2","type":"change","z":"553814a2.1248ec","name":"","rules":[{"t":"set","p":"myStore[msg.topic]","pt":"flow","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":860,"y":1400,"wires":[[]]},{"id":"226ff1ef.a980fe","type":"inject","z":"553814a2.1248ec","name":"init","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":true,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":610,"y":1320,"wires":[["40ee91de.e767a"]]},{"id":"40ee91de.e767a","type":"change","z":"553814a2.1248ec","name":"init myStore","rules":[{"t":"set","p":"myStore","pt":"flow","to":"{}","tot":"json"}],"action":"","property":"","from":"","to":"","reg":false,"x":810,"y":1320,"wires":[[]]},{"id":"fe7a1f7b.24d1","type":"change","z":"553814a2.1248ec","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"myStore[msg.topic]","tot":"flow"}],"action":"","property":"","from":"","to":"","reg":false,"x":820,"y":1500,"wires":[["393650b7.5db25"]]}]
5

If you check out the uibuilder WIKI (or the examples library if you have uibuilder installed), you will find a caching example which does what you want. It is a simple function node.

6

@Steve-Mcl Thanks you for the hint to the new change node possibilities. That looks like a pretty elegant solution to me.

I think my aversion to function nodes comes from my early Node-RED days. I wrote a lot of function nodes with a lot of testing involved. Later I realized that I could have done many tasks much easier by using standard nodes. Some standard nodes are very powerful, e.g. join or trigger. But it takes a while to get to know all the different operating modes. Currently my strategy is to explore the standard nodes before starting to write a function node.

2 Likes
7

Ah yes, I understand. I had a pretty much identical experience. Even down to which nodes made you re-evaluate your approach. I thought maybe you had some notion about function nodes being evil or slow :slight_smile:

The only potential tripping point about the nested properties in change node is that you can't have a top level item (e.g. flow[msg.topic] or msg[some_prop])

1 Like
8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.