Does an equivalent node exist / what should I call it

Developing Nodes
1

Hi All

I am looking for a node that has an input node and an output node and caches any objects recieved by the input node.
The output node would emit these objects (keyed on an object property) each time a new value was set. I have done a bit of searching but not sure exactly what to search for!

Basically I have a load of lights that all call a rest api when their state changes. I want one node that will cache all the updates from the rest endoint and then separately I want an output node that will push a message for each state change.

Nodes:

  • Config
    creates a payload store. Configures which property name on the object is used to lookup values (for example lightId)
  • Input
    Adds or updates a value in the lookup
  • Output
    emits a message for the provided lookup value each time the value is updated
  • Lookup
    returns a messages with the currently stored payload for a given lookup value

I think that I probably need to create this node. I was thinking of calling it:

node-red-contrib-payload-store

suggestions / comments for a better name gratefully accepted.

2

I started work on one quite a long time ago to support uibuilder. However, I stopped when I realised that, for a genericly usable node, there are a LOT of different processing cases.

For example, how many cached items will you hold? Will you replay them all each time? Will you off caching by topic? Or by time? Or by some other property on the msg?

If you have a look at the uibuilder wiki or in the library uibuilder examples, you will find some examples of caching and how uibuilder issues a replay request when a new client connects (or someone reloads a uibuilder page).

I usually do caching by topic so that just the last msg of each topic type is retained. However, even that is not a given, you may have some topics that you don't want to retain.

In the end, I pretty much gave up on the idea of a custom node since it was easy enough to do in a function node anyway and that gives you total control over things.

So if you are going to go ahead with your node, I would suggest making sure that the name is specific to the function. You also need to make sure it has limits on size and complexity - or if not, that you warn flow authors that the data could get dangerously large if not treated carefully.

3

Well... the RBE/Filter node will do caching by topic - i.e. only sending on changes - but won't let you query it on demand. And the built in context will let you store anything and then query it - but won't trigger events on change. So maybe a combination of RBE node then writing to context would get you close ?

4

Yes, that is really something that would be extremely useful to be added to context storage :wink:

2 Likes
5

Sure you have mentioned this before Julian :wink:

6

Haha, one day, one of us will have the time to do something about it!

7

Do you mean you want to emit the complete set of inputs whenever an input is received. The Join node will do that for you, configured in key/value mode.

8

sorry, I possibly wasn't very clear.
I will only cache the last payload for each key (i.e. the cache will always be current with the current state of the light). The config node will basically hold a map of string => payload. Each time a new message arrives the value for that key (or lightId) will be updated.
When the value is updated any output node will emit a message with the new value for the specific key that the output node is watching.
It seems that some people in here might find it useful. I'll find it useful so I'll go ahead and create it.

9

so - why won't a simple RBE /filter node plus change node (to save to context) work ?

10

well, I don't understand these nodes well enough to understand your proposal. From what I understand of your proposal I would need several nodes to satisfy the different requirements and would store data in the context which I would then query in function nodes. I am sure this is all possible but for my needs it'll be a lot simpler to have one set of nodes that does all this and does not require function nodes to access the context.
Thanks

11

Well no need for functions as far as i can tell - just one filter and two change nodes...
which I would argue is simpler than writing your own.

image
image1490Ă—832 76.3 KB 

[{"id":"f162589ba24889b7","type":"tab","label":"Flow 1","disabled":false,"info":""},{"id":"8629dac32f0ac352","type":"inject","z":"f162589ba24889b7","name":"","props":[{"p":"payload"},{"p":"lightid","v":"L1","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"On","payloadType":"str","x":140,"y":150,"wires":[["2a0d5979cdee5fcf"]]},{"id":"2a0d5979cdee5fcf","type":"rbe","z":"f162589ba24889b7","name":"","func":"rbe","gap":"","start":"","inout":"out","septopics":true,"property":"payload","topi":"lightid","x":330,"y":240,"wires":[["21faae49b0c71370","85d240898546e208","b9e55b78a4b53459"]]},{"id":"21faae49b0c71370","type":"change","z":"f162589ba24889b7","name":"","rules":[{"t":"set","p":"lights[msg.lightid]","pt":"flow","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":575,"y":180,"wires":[[]]},{"id":"149ca931fd4b9099","type":"inject","z":"f162589ba24889b7","name":"","props":[{"p":"payload"},{"p":"lightid","v":"L1","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"Off","payloadType":"str","x":140,"y":195,"wires":[["2a0d5979cdee5fcf"]]},{"id":"53e4dd580ce39f06","type":"inject","z":"f162589ba24889b7","name":"","props":[{"p":"payload"},{"p":"lightid","v":"L2","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"On","payloadType":"str","x":140,"y":285,"wires":[["2a0d5979cdee5fcf"]]},{"id":"646c457519eabb09","type":"inject","z":"f162589ba24889b7","name":"","props":[{"p":"payload"},{"p":"lightid","v":"L2","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"Off","payloadType":"str","x":140,"y":330,"wires":[["2a0d5979cdee5fcf"]]},{"id":"85d240898546e208","type":"link out","z":"f162589ba24889b7","name":"","mode":"link","links":[],"x":510,"y":270,"wires":[]},{"id":"040ab24a97263c25","type":"inject","z":"f162589ba24889b7","name":"Query L1","props":[{"p":"lightid","v":"L1","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":160,"y":420,"wires":[["5c8ed49550eed77c"]]},{"id":"5c8ed49550eed77c","type":"change","z":"f162589ba24889b7","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"lights[msg.lightid]","tot":"flow"}],"action":"","property":"","from":"","to":"","reg":false,"x":365,"y":420,"wires":[["f4940ad0951f7a19","5d13aff6ca09ccae"]]},{"id":"f4940ad0951f7a19","type":"link out","z":"f162589ba24889b7","name":"","mode":"link","links":[],"x":530,"y":420,"wires":[]},{"id":"f40b6b0269dcf9c0","type":"inject","z":"f162589ba24889b7","name":"Query L2","props":[{"p":"lightid","v":"L2","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":165,"y":465,"wires":[["5c8ed49550eed77c"]]},{"id":"5d13aff6ca09ccae","type":"debug","z":"f162589ba24889b7","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":585,"y":465,"wires":[]},{"id":"b03c5e2504e1cc94","type":"comment","z":"f162589ba24889b7","name":"Use this for immediate updates","info":"","x":655,"y":225,"wires":[]},{"id":"1aaddd6c3624e495","type":"comment","z":"f162589ba24889b7","name":"Use this for queries","info":"","x":635,"y":375,"wires":[]},{"id":"b9e55b78a4b53459","type":"debug","z":"f162589ba24889b7","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":565,"y":315,"wires":[]}]
1 Like
12

Thanks very much for the example. I'll take a look.

13

ok, I can see how that's working. That's introduced me to a few nodes that I hadn't used before so thanks for that.
I think I'll still go ahead and create my node. It will require less code duplication (ok, it's only the change node that I would probably have to vopy around the place), will work globally across all flows (yup, I could use global scope instead) and will not require in and out node between different flows.
Thanks for the help anyway.

14

A node that watches a context variable and outputs when it changes sounds useful.
An equivalent to the RPi GPIO In node but for context variables?

I guess the variable would be an object and the node could be set to trigger on change of a specific value, or any.

Setting the context seems too trivial to need a special node.

15

The aversion we have to this is that it introduces invisible wiring - so data no longer flows down wires and can just pop up "anywhere" - this makes the flow less readable and understandable as you don't know why data is changing in some place that is not wired up and makes it hard to debug. It's an anti pattern for a tool that is all about wiring up flows of events.

2 Likes
16

ok but I'll need them anyway as I have one http node that the requests are made to and all flows need access to the data that is passed to the http node. In your example you have an out node that would attach to in nodes. How is that any different?

17

the link nodes visibly shows where they are connected to.

But no matter - you are free to create any node you wish.

I would suggest not using a generic name like payload-store as the existing context does that already... (without the name of course) - and you seem to be linking it to some http functionality. Or maybe just publish it privately without the Node-RED keyword ?

18

Yes, it's already sometimes hard to work out the origin of a message that causes problems.
An example I have had is a relay can sometimes start hunting (is that the right word? continuously switching) when a physical button and dashboard button get out of sync.
Actually that's the same situation where I see a "watch variable" node being helpful.

For easier debug, could the Node-Red core detect that a message is newly created and set a message property msg.sourcenodeid?

19

I have to mention, as I always do when this question arises, that using retained MQTT topics instead of context is often a good alternative to context, having the advantage that you get notified of both changes and initial values.

20

And that is a reasonable solution for anyone using MQTT. Might not be so good in some enterprise environments though.

While I absolutely understand the desire to avoid hidden "wiring". I've never really been able to see the justification here. I would expect that there would simply be another "in" node that acts exactly the same way as the MQTT-in or any of the other event-in nodes. When a change happens to a subscribed variable, the variable-change-in node (note the snappy name!) would issue a msg with the value as the payload same as any other source node. Nothing hidden about it.