HI,
I would like to share with you some use case and i hope to advice if Node-RED can be used for this use case or not.
I’m working on Street Lighting System so we have 5-10 senors on each pole sending different data. The number of poles may be 1 million and the rate of sending the data 6000 to 1000 event per second as each pole send the data in 15 minute period.
So if i plan to use Node-Red in a back-end server not edge to receive this number of events and to do some real-time processing like doing some validation on the readings and fire some event according to the validation then store the data in a repository may be elasticsearch.
First is Node-RED suitable for such use case?
In case Yes,
How many instances of Node-Red needed to handle such number of events?
In case of more than one, Do they orchestrate with each other or another tool needed, may be load balance?
What the hardware needed?
How to scale in case number of events increased?
In case No,
Would you please suggest some tool like Node_Red for such use cases?
I would never select a tool to handle that magnitude without a test implementation. This would allow you to test the scale and number of instances you need based on your flows and the hardware you are running it on. Without knowledge of these your questions are almost unanswerable
Thank you so much for answer,
Let me clarify my question taking GeoEvent developed by ESRI as an example. GeoEvent is very similar to Node-Red, it has inputs, processing and output.
And it’s clear from the documentation that it can handle 6000 event per second using specific hardware also they state recommendations for the hardware and number of instances to handle specific number of events and how to scale.
There is clear information about the capability of the product using let’s say recommend HW so you can size for your project. Of course there Will be test but having such information about the product is very important and save a lot of time.
So this is what i wanted to know about Node_red.
Well as you know Node-RED is an event based application programming environment. Users are capable of writing well performing applications and badly performing applications. It can run on hardware from Raspberry Pi to servers to cloud. Somewhere in there will be a combination capable of 6000 messages a second.
If you do run more than one, then how you load balance will be entirely dependent on how the events arrive.
If it’s HTTP based, then load balancing is well understood.
If its MQTT, then it will depend on the broker you are using as to whether it supports shared subscriptions or not.
If it’s something else, well, the answer will be something else.
If you want to auto-scale, you’ll need to decide how to monitor the through-put of each instance and what rules you want to apply. Node-RED doesn’t provide anything for auto-scaling instances - you’ll have to find an appropriate solution for that yourself.
You should also consider what time frames you need to scale to 1 million; I doubt you’ll have that on day one.
Thank you so much for your answers
I just want to make sure from the following points
1- Node-Red is designed to be used in such projects in production , I state this because there are articles on the web which restrict Node-Red for edge computing or for prototyping not production?.
2- Currently I’m running test for Node-Red on my local machine(16 GB Ram, CPU 4 Cores 2.60 GHZ, 8 Logical processors).
There is http node which receive the message and two Switch nodes check for the value of some field on the message then the results from both nodes are written into text file.
I test different scenarios for example send 1000 event per second(one each millisecond) or send 6000 request each 5 second and so on trying to define the capability of Node-Red in this environment.
But some time i found the number of requests received in Node-Red is less than the sent requests .Does this mean with high load Node-Red ignore some requests? I mean there is no queuing component in Node-Red and need to have one beside Node-Red for example Kafka?
