How Node-red's memory recycling mechanism works

General
23

If your MQTT broker is capable of using MQTT v5, then you have additional options for message receipts which you might be able to use, not sure, I've never used them with MQTT or Node-RED I'm afraid.

Other message queue tools also have message receipts but that would be a big change of course.

24

Thank you very much for your prompt. I will study it~

25

I enjoyed reading this thread. As I have done several industrial control applications using MQTT. But so far, I have never done any in Javascript, Node-RED or node. A few of them were "near real-time" (or "soft" real time) so the inherent pause when the GC ran was not a concern for me.

I don't have any specific help for you, but your post triggered a few design tenets that I've tried to stick to in RT or industrial control apps:

  • think events, events, events, events
  • messages, messages, messages
  • see if you can make things work with a QoS of zero. If your design depends on knowing if the message was received, try to come up with a new design.
  • messages, messages, messages - Before I would do a QoS of 1 or 2,, I'd think about the receiver sending back an "I got your message" event

And what others have said in this thread. An "inversion of control, event driven" approach has gotten me out of many messy designs and given me a reliable solution.

The wonderful thing about messages and MQTT is testing and montoring becomes almost trivial. I debug by injecting messages into the broker, I write monitoring apps by listening to the bus.

Again, I've been able to get thru with QoS zero in all of my apps. My thought has been "if I need to know if an event was received then I should rethink my approach". [ And for local networks, my MQTT broker has been extremely reliable. I've never detected a lost message using QoS zero. ]

Good luck!

26

Or use MQTT v5 which has receipts.

Maybe not relevant here but if you genuinely need close to real-time or actual real-time, use a microprocessor platform like an ESP32, not a Linux platform which has too many other things that could happen at any moment in time.

27

If the receiver does not return a reply message to the sender, the sender's receiving event will not be triggered, which will cause the sender to stop operating.

May I ask if you have encountered such a situation before? How did you avoid such a situation at the application layer by developing interactive processes?

28

Because IoT architecture typically consists of cloud, edge, and terminal devices,

Each device is independent and may shut down separately at any time, resulting in the inability to receive or respond to messages.

I often encounter this problem in practical projects.

29

Suggestion for original poster to see how other handle the issue.

I use esp32's to send temps for tempature/gas pressure/etc for alerts and reports.
To solve the sender not receiving receipt of msg from server I use an array to store failed sent msg's
Then on the next timed send of data I send the current temp and the previous failed temp. If it keeps failing I store up to 2056 bytes of previous temp reports. Then when sensor gets connection again it sends all previous reports.

If days go by with out connection receipt of msg the 2056 bytes may become full. At this point my code just deletes the oldest data.

......
if rtc_read_memory_len < 1920:
# convert rtc_memory_list to a MessagePack serialization and write it to rtc memory
rtc_memory_list_bytes = umsgpack.dumps(rtc_memory_list)
......
else:
del rtc_memory_list[0] # <--- NOTE! oldest data is in first index of array 
rtc_memory_list_bytes = umsgpack.dumps(rtc_memory_list)
......
30

Doing queue management and caching is indeed a good method. Thank you for recommending it

31

I would like to ask if I have used too much flow context in Node red, such as:

const flow_name = "abc" + "xxx" + "....";
flow.set(flow_name, 1);

Alternatively:

const flow_name = "abc" + "xxx" + "....";
const flow_data = flow.get(flow_name ) || 0;

When my code become massive, I forgot where I had used flow_name before.
Is there any way to list the call locations for the target flow context？

32

Try not to use flow or global context, you get exactly this sort of issue. The node-red way is to pass values via messages. There are very few cases where using context is essential.

To find references you should be able to use the Search feature in the editor and search for the name.

34

My program has been refactored to reduce the use of flow context,

But in some places, it is difficult to avoid using flow context, such as:

In the IOT system, if the upper computer needs to perform timed over limit detection on the collected data, on the one hand, it needs to collect data from sensors on a regular basis, and on the other hand, it needs to receive artificially modified alarm thresholds at any time,

Obviously, these two behaviors are independent. My idea is to store the received threshold in the flow context, so that when the detection time comes, the threshold in the flow context can be used in the collection thread, and then logical judgment can be made.

Is there a better implementation method if the threshold is not stored in the flow context?

35

You can use a Join node to combine relevant data into one message, see this article in the cookbook for an example of how to all relevant messages into one object. Then you have all data when you need it.

36

1705400580399
1705400580399868×192 16.2 KB

Regarding the join node, if "receive 3 messages and pass" is set, but the actual temperature branch has been injected continuously 3 times, and the other two branches have not received data. Is the temperature value injected in the first two times lost or will it be retained?

37

Use the topic to wait for 3 different topics before passing

Manual mode - using the value of msg.topic as the key

38

Thank you. Let me think about how to redesign the program based on the join node

39

Personally, I just ignore failed sensor readings because they are mostly only useful at a point in time anyway and for historic data, I'd rather know that something caused a break in the data.

I do track whether the device is online or offline though and that is done by sending an online msg from the device to MQTT on a particular topic and setting an LWT that the broker automatically triggers on the same topic if it hasn't had a message on that topic for a set period - I usually use about 90sec since my sensors generally report once a minute.

As Colin says, generally best not to track such things.

However, I often have a similar problem where I don't know what updated some MQTT topics. To counter it, I have 2 additional MQTT topics for whatever I'm recording. One is the latest change timestamp and the other is what process updated it. If I'm also recording the data in context variables, the variable object mirrors the structure of the MQTT topics and so the data is captured there as well.

I don't have a fixed naming for this but tend to use the flow name and something that identifies the group or node instance - often a summary of the name again. The idea being that it will be a person that needs to know so the data should be human recognisable.

40

I tend to only use context for static things to keep usage down.
examples:
Track a logged in user (only gets big based on users logged in)
Static Array/Object like unique sensor ID's that must be verified before posting data.
Short lived things that there is no other way or speed issue if called.

If I can use a database i do.

41

Others have already given you better suggestions. And I really do not know your design, your environment, your circumstances, your goals. But I'll share a few things I do with my IoT projects and MQTT.

I use watchdog timers on end devices.
I have heartbeat MQTT messages flowing back to me so I can get a sense if an end-device is having trouble.

Like I said, I try very hard for a design where I can use QoS 0, not care too much if an end device gets one, several or sometimes no message(s)

The "controller" app in my industrial systems is an FSM. That makes it easier for me to design a message based flow.

I am fortunate enough to be able to trust the network in my applications. Like I said, my messages get to their endpoint, unless the endpoint is offline. I don't have to worry about an unreliable network.

Finally, another wonderful thing about message/event based design is that I've leveraged a CEP (Complex Event Processing) engine to keep tabs on the whole kaboodle. I've been using Esper. And I have Esper subscribe to the MQTT bus for all events. Then it's super easy to write rules to have Esper watch for a complex sequence of events and do something.

If an Esper error rule triggers, I get an alarm on my dashboard. I can have Esper calculate uptime/productivity metrics, etc all from just sniffing the bus.

Again, I think you're getting better suggestions from this group.
I just wanted to follow up with a response to you

42

I had the opposite issue myself, only redeclaring the message object all the time (making the garbage collector work alot) and not using the node-red context. I wrote a little tweet about it some week ago and some cool garbage collector plots of the runtime:

43

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.