I am currently working on a PoC device-service for the EdgeX project, in essence the node-red service is a RESTful glue between the EdgeX IoT platform and lora-app-server.
It consists of 2 flows that will run concurrently (?). The first flow is a series of HTTP requests to both the EdgeX API and the lora-app-server API in order to register the node-red service. The second flow is a set of HTTP endpoints that will be used by EdgeX and lora-app-server in order to communicate.
The problem is that I am running this service in a raspberry pi 3 (using balena) that already hosts multiple containerised services. I need my node-red container service to be as lightweight as possible, ideally using 10-20mb of memory, with a max of 64. What can I do in order to optimise my flows as much as possible?
The flow is semi-finished, meaning that it needs some polishing, fixing a couple of URLs, etc. But neither the idea, nor a considerable number of new nodes will be added.
What do you think folks?
I am so humbled for this community, thank you so much for your time!
What I understand, both LoRa App Server and EdgeX supports MQTT. You are not going to use that??
Alternatively, instead of using NR as the "glue", have you considered writing a Python program instead? Might give you the small footprint you are targeting
I have searched the forums, setting memory was easy, disabling the web interface hasn't produced and concrete results. I was hoping that people will give little tips and hints so to optimise further, like avoiding certain nodes, etc.
Python script was my go-to solution but I figured that I could slash the dev time considerably using node-red as I would only need to focus on the http requests structure and not any actual program structure. Moreover, I needed concurrency, so it increased complexity (either threading or async). After spending a couple of days on node-red, not so sure anymore.
MQTT would be just the communication protocol, I use HTTP because I need to talk to certain apis, bootstrap the node-red service and then start operation .
But I would never use http for polling of events, it's not efficient, besides http has a lot of overhead compared to mqtt. Besides, by subscribing, no need for polling, you get a notification when something happens, it handles reconnections etc etc, you know most likely this already, I'm kicking in open doors...
use fewer node types - only install /load ones you need - but that is only a small saving (unless you had loaded masses of extra nodes)
don't use context (in memory) to store objects
don't process massive objects (like images). Creating then destroying them takes time and memory and the garbage collector has to kick in.
don't use http - ok - so not really - but those req/res objects are big compared to others that have been mentioned... Also you can turn off the http interface / admin / editor then node doesn't need to load up the express server.
don't let queues build up, as long as you stay ahead of the data rate required then memory shouldn't balloon up.
I have just checked the memory footprint of one of my node-red instances and it is 92 mb. So it will be a challenge to get this down to 10-20 mb.
So I would start with a blank flow and the suggestions of @dceejay above and measure the node-red memory footprint. If it is already above the 20 mb (which I fear) then creating any flow will not make it any better.
I am crossing my fingers for you and happy to hear you report back.
Jan.
Hi @odys,
The question for me is always 'which nodes are consuming most of my memory'. Because like Dave already explained, some nodes will queue messages or consume lots of memory in all kind of ways ...
I tried to build a contribution some time ago to analyze this kind of stuff, but there were so much limitations that it became undoable ... So you will to figure out manually which nodes you should remove/reconfigure/replace.
Bart
Thank you so much for your time and effort you put into replying, it is super helpful for me!
From what I see, Node-red was indeed a mistake as it did not decrease dev time considerably while crippling the service efficiency. At any rate, I will be trying to reduce the footprint as much as possible.
In essence, the service has 2 modules. One that is a series of http requests and registers the service and the second which is a set of endpoints and implements the actual functionality.
Regarding messages queues, could you elaborate please?
If I want data to be available for an entire path of nodes, is it more efficient to save it and load it as flow context or it is better to have it as msg.variable and just pass it from one node to the next one. Mind that in the end the msg could have 5-6 of those custom attributes, other than the usual ones (payload,etc).
Much better to pass them as part of the msg as they then stay as one unit, otherwise you need to tag each thing in context with something to correlate it with the correct msg. Don't forget JavaScript passes objects by reference.
I think that you are right, i am still trying to wrap my head around the architecture of this platform. One issue I have though is persistence, since it is very possible that the device service is restarted mid-bootstrap.
I will be implementing new logic tomorrow and will try to minimise context. Although node-red is not very efficient and I think that a couple of variables will not change considerably the footprint, thus I will try to weight the optimisation vs development complexity.
Thank you so much for your time and effort guys you put into the replies, it is much appreciated!
