Node-RED aborts with code 6 while under CAN-BUS load

General
1

greetings ,

i'm tryingbto convert CAN-BUS messages to a JSON based data format.Everything works just fine if don't send messages too quickly but if i replay a trace of a CAN-BUS at full speed Node Red just aborts with the following error from systemctl status node-red.service . it seems like amemory issue but i don't have a clue where to begin

Jun 19 20:05:10 iot2050 node-red[354]: 5: 0xffffb8dabcf8 [/lib/aarch64-linux-gnu/libnode.so.108]
Jun 19 20:05:10 iot2050 node-red[354]: 6: 0xffffb8dbd67c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/lib/aarch64-linux-gnu/libnode.so.108]
Jun 19 20:05:10 iot2050 node-red[354]: 7: 0xffffb8d9e12c v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/lib/aarch64>
Jun 19 20:05:11 iot2050 node-red[354]: 8: 0xffffb8d9f160 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/lib/aarch6>
Jun 19 20:05:11 iot2050 node-red[354]: 9: 0xffffb8d81b28 v8::internal::factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/lib/aarch64-linux-gnu/libnode.so.10>
Jun 19 20:05:11 iot2050 node-red[354]: 10: 0xffffb911f740 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/lib/aarch64-linux-gnu/libnode.so.108]
Jun 19 20:05:11 iot2050 node-red[354]: 11

2

I ujust tested and it doesnt seem to be the rate of messages but rather a certain situation that eventually. triggers it. i am using

red-contrib-socketcan 1.0.0

3

apparently i'm blind as a bat because i just saw this

2024-06-19T21:40:40.987555+02:00 iot2050 node-red[355]: [355:0xaaaae00ffc60] 660598 ms: Scavenge (reduce) 462.3 (472.7) -> 461.6 (473.0) MB, 30.2 / 0.0 ms (average mu = 0.185, current mu = 0.023) allocation failure;
2024-06-19T21:40:40.987876+02:00 iot2050 node-red[355]: [355:0xaaaae00ffc60] 666889 ms: Mark-sweep (reduce) 462.4 (473.0) -> 461.5 (473.2) MB, 6208.5 / 0.0 ms (average mu = 0.131, current mu = 0.076) allocation failure; scavenge might not succeed
2024-06-19T21:40:40.988230+02:00 iot2050 node-red[355]: <--- JS stacktrace --->
2024-06-19T21:40:40.988436+02:00 iot2050 node-red[355]: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2024-06-19T21:40:41.010483+02:00 iot2050 node-re

can i increase the javascript heap size or would i still slowly run out?

4

How many messages are we talking?

If you put a delay node (set to rate limit 1msg/100ms or even 1msg/10ms) BEFORE the can node, does it still happen?

You might simply be causing backpressure due to the high rate vs the can network speed/latency and exhausting available memory in the runtime.

5

i allready tried setting the heap size larger but i'll try it slowing down the can message trace a bit and see if that helps

6

a quick count tells me it's about 19000 but the flows are intended to handle a continuous stream so the amount shouldn't matter

7

If the rate you are asking for data is greater than the rate at which you are able to process the data then you may get data backed up in node red waiting to be serviced. That can lead to running out of memory.
Are you able to monitor the rate that the data are coming out of the end of your flow to see if it is keeping up?

8

Think of this as a data processing (like ETL) task. Now think about the "thing" between your data stream and the CAN hardware as being a funnel. If you poor too much too fast into the funnel, it will overflow.

To put it another way, if (on average) it takes 5 milliseconds to perform a transmission from Node-RED to the physical device, there is no point in hammering it with a burst stream of 19000 (or 2000 or 50000) continuous records, they will ultimately be backed up somewhere and will still take 5ms with or without the delay node. Even using rate limiting may not solve this entirely (since the delay node will hold the messages in a queue) but it might be better as you dont know what is happening in the CAN BUS nodes (are they building complex classes?, parsed packets? duplicating data?, storing on file? etc etc) whereas the delay node will simply queue exactly what you send to it in memory. So, if all 19000 records are approx 125 bits (~8bytes) then 19000*8 == 152KB (not huge)

Ultimately, you may need to look at scaling this horizontally and reducing the load on your node-red server.

9

i tried to rate limit to 1 message every 5 seconds but that also doesn't work. the bottleneck is probably the massive amount of json i'm trying to insert the values in.mmh i will need to get creative .

10

after trimming a lot of fat from the json structure and slowing down the input quite a bit it seems to finally work without crashing.

11

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.