@Colin Thanks for the reply, sorry for delayed response from my end, recently we were trying to figure out this with multiple approaches, so I have few updates related to this.
In one approach I tried to take the memory readings by keeping only one contrib package in the build, in this way we got that the build that was using node-red-contrib-opcua package increased gradually, rest of the packages were stable for longer duration. But I am not sure what might be the exact issue behind it, please suggest me some ways if you know of, how to track or find the possible leak block. I tried out the chrome inspector as many articles online suggest to compare the heap snapshots, but this too don't provide the possible js filename/function name.
Another update is, one of my colleague who is haveing some good experience C++, he followed the nodejs build from source documentation (link here), and tried out the nodejs asan build technique to test with leak sanitizer. In this case we had kept only node-red running without any contrib packages, this also reported a memory error as follows.
Direct leak of 24 byte(s) in 1 object(s) allocated from:
#0 0x7f3f9f098947 in operator new(unsigned long) (/lib/x86_64-linux-gnu/libasan.so.5+0x10f947)
#1 0x555a16e26da0 in node::native_module::NativeModuleLoader::LoadBuiltinModuleSource(v8::Isolate*, char const*) (memory-leak/node-js-asan-debug-out/bin/node+0x14bbda0)
#2 0x555a16e28d1a in node::native_module::NativeModuleLoader::LookupAndCompile(v8::Localv8::Context, char const*, std::vector<v8::Localv8::String, std::allocator<v8::Localv8::String > >, node::native_module::NativeModuleLoader::Result) (memory-leak/node-js-asan-debug-out/bin/node+0x14bdd1a)
#3 0x555a16e2b2e3 in node::native_module::NativeModuleLoader::CompileAsModule(v8::Localv8::Context, char const*, node::native_module::NativeModuleLoader::Result*) (memory-leak/node-js-asan-debug-out/bin/node+0x14c02e3)
#4 0x555a16e38106 in node::native_module::NativeModuleEnv::CompileFunction(v8::FunctionCallbackInfov8::Value const&) (memory-leak/node-js-asan-debug-out/bin/node+0x14cd106)
#5 0x555a178c13a5 in v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) (memory-leak/node-js-asan-debug-out/bin/node+0x1f563a5)
#6 0x555a178c7350 in v8::internal::MaybeHandlev8::internal::Object v8::internal::(anonymous namespace)::HandleApiCallHelper(v8::internal::Isolate*, v8::internal::Handlev8::internal::HeapObject, v8::internal::Handlev8::internal::HeapObject, v8::internal::Handlev8::internal::FunctionTemplateInfo, v8::internal::Handlev8::internal::Object, v8::internal::BuiltinArguments) (memory-leak/node-js-asan-debug-out/bin/node+0x1f5c350)
#7 0x555a178dcfe7 in v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) (memory-leak/node-js-asan-debug-out/bin/node+0x1f71fe7)
#8 0x555a178e0a95 in v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) (memory-leak/node-js-asan-debug-out/bin/node+0x1f75a95)
#9 0x555a1af49b3e in Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit (memory-leak/node-js-asan-debug-out/bin/node+0x55deb3e)
#10 0x555a1ad2e074 in Builtins_InterpreterEntryTrampoline (memory-leak/node-js-asan-debug-out/bin/node+0x53c3074)
#11 0x555a1ad2e074 in Builtins_InterpreterEntryTrampoline (memory-leak/node-js-asan-debug-out/bin/node+0x53c3074)
#12 0x555a1ad2e074 in Builtins_InterpreterEntryTrampoline (memory-leak/node-js-asan-debug-out/bin/node+0x53c3074)
#13 0x555a1ad2e074 in Builtins_InterpreterEntryTrampoline (memory-leak/node-js-asan-debug-out/bin/node+0x53c3074)
#14 0x555a1ad1657e in Builtins_ArgumentsAdaptorTrampoline (memory-leak/node-js-asan-debug-out/bin/node+0x53ab57e)
#15 0x555a1ad2e074 in Builtins_InterpreterEntryTrampoline (memory-leak/node-js-asan-debug-out/bin/node+0x53c3074)
#16 0x555a1ad2e074 in Builtins_InterpreterEntryTrampoline (memory-leak/node-js-asan-debug-out/bin/node+0x53c3074)
#17 0x555a1ad2e074 in Builtins_InterpreterEntryTrampoline (memory-leak/node-js-asan-debug-out/bin/node+0x53c3074)
#18 0x555a1ad1657e in Builtins_ArgumentsAdaptorTrampoline (memory-leak/node-js-asan-debug-out/bin/node+0x53ab57e)
#19 0x555a1ad2e074 in Builtins_InterpreterEntryTrampoline (memory-leak/node-js-asan-debug-out/bin/node+0x53c3074)
#20 0x555a1ad2e074 in Builtins_InterpreterEntryTrampoline (memory-leak/node-js-asan-debug-out/bin/node+0x53c3074)
#21 0x555a1ad2e074 in Builtins_InterpreterEntryTrampoline (memory-leak/node-js-asan-debug-out/bin/node+0x53c3074)
#22 0x555a1ad2e074 in Builtins_InterpreterEntryTrampoline (memory-leak/node-js-asan-debug-out/bin/node+0x53c3074)
#23 0x555a1ad2e074 in Builtins_InterpreterEntryTrampoline (memory-leak/node-js-asan-debug-out/bin/node+0x53c3074)
#24 0x555a1ad2e074 in Builtins_InterpreterEntryTrampoline (memory-leak/node-js-asan-debug-out/bin/node+0x53c3074)
#25 0x555a1ad2e074 in Builtins_InterpreterEntryTrampoline (memory-leak/node-js-asan-debug-out/bin/node+0x53c3074)
#26 0x555a1ad24959 in Builtins_JSEntryTrampoline (memory-leak/node-js-asan-debug-out/bin/node+0x53b9959)
#27 0x555a1ad24737 in Builtins_JSEntry (memory-leak/node-js-asan-debug-out/bin/node+0x53b9737)
#28 0x555a18066359 in v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) (memory-leak/node-js-asan-debug-out/bin/node+0x26fb359)
#29 0x555a1806aac0 in v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object, int, v8::internal::Handlev8::internal::Object*) (memory-leak/node-js-asan-debug-out/bin/node+0x26ffac0)
Direct leak of 24 byte(s) in 1 object(s) allocated from:
#0 0x7f3f9f098947 in operator new(unsigned long) (/lib/x86_64-linux-gnu/libasan.so.5+0x10f947)
#1 0x555a16e26da0 in node::native_module::NativeModuleLoader::LoadBuiltinModuleSource(v8::Isolate*, char const*) (memory-leak/node-js-asan-debug-out/bin/node+0x14bbda0)
#2 0x555a16e28d1a in node::native_module::NativeModuleLoader::LookupAndCompile(v8::Localv8::Context, char const*, std::vector<v8::Localv8::String, std::allocator<v8::Localv8::String > >, node::native_module::NativeModuleLoader::Result) (memory-leak/node-js-asan-debug-out/bin/node+0x14bdd1a)
#3 0x555a16e385e2 in node::native_module::NativeModuleEnv::LookupAndCompile(v8::Localv8::Context, char const*, std::vector<v8::Localv8::String, std::allocator<v8::Localv8::String > >, node::Environment) (memory-leak/node-js-asan-debug-out/bin/node+0x14cd5e2)
#4 0x555a16b1c4d1 in node::ExecuteBootstrapper(node::Environment*, char const*, std::vector<v8::Localv8::String, std::allocator<v8::Localv8::String > >, std::vector<v8::Localv8::Value, std::allocator<v8::Localv8::Value > >) (memory-leak/node-js-asan-debug-out/bin/node+0x11b14d1)
#5 0x555a16b1fdc9 in node::Environment::BootstrapInternalLoaders() (memory-leak/node-js-asan-debug-out/bin/node+0x11b4dc9)
#6 0x555a16b22693 in node::Environment::RunBootstrapping() (memory-leak/node-js-asan-debug-out/bin/node+0x11b7693)
#7 0x555a168e6403 in node::CreateEnvironment(node::IsolateData*, v8::Localv8::Context, std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&, node::EnvironmentFlags::Flags, node::ThreadId, std::unique_ptr<node::InspectorParentHandle, std::default_deletenode::InspectorParentHandle >) (memory-leak/node-js-asan-debug-out/bin/node+0xf7b403)
#8 0x555a16ddf2dd in node::NodeMainInstance::CreateMainEnvironment(int*) (memory-leak/node-js-asan-debug-out/bin/node+0x14742dd)
#9 0x555a16ddfa61 in node::NodeMainInstance::Run() (memory-leak/node-js-asan-debug-out/bin/node+0x1474a61)
#10 0x555a16b2d0ed in node::Start(int, char**) (memory-leak/node-js-asan-debug-out/bin/node+0x11c20ed)
#11 0x7f3f9ea480b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)
SUMMARY: AddressSanitizer: 1656 byte(s) leaked in 69 allocation(s).
This output may give you some idea, the command we used was
./node ASAN_OPTIONS=new_delete_type_mismatch=0:detect_leaks=1 ../lib/node_modules/node-red/node_modules/node-red/red.js --userDir=/home/ubuntu/nodeRedTemp2/.node-red/
I am not exactly sure about the second approach as I am not much familiar with it, you guys can have a look at it if there is by any chance a possibility of leak in node-red.
Lastly have you guys faced any issue with the exec node in node-red, just recently we got that when we run a 4k string generation flow the swap memory drains out really soon. I have attached a sample flow for the same, this flow when used sooner or later will give spawn ENOMEM error. Could it be the root cause of the memory leak issue ?
[{"id":"d2cce0a1.c9155","type":"tab","label":"4k Block","disabled":false,"info":""},{"id":"4399d6bc.8bfd98","type":"inject","z":"d2cce0a1.c9155","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"1","crontab":"","once":true,"onceDelay":".5","topic":"","payload":"","payloadType":"str","x":110,"y":180,"wires":[["4eb351c9.e5284"]]},{"id":"52c73fd4.13fd3","type":"debug","z":"d2cce0a1.c9155","name":"Output to cloud","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":560,"y":180,"wires":[]},{"id":"4eb351c9.e5284","type":"exec","z":"d2cce0a1.c9155","command":"base64 /dev/urandom | head -c 4096","addpay":false,"append":"","useSpawn":"false","timer":"","oldrc":false,"name":"4K Random Data","x":330,"y":180,"wires":[["52c73fd4.13fd3"],[],[]]}]