Hi everyone,
in a small team, we are working on a big industrial project to automate production in a factory, to give you an idea of the scope of the project ( we have 40+ RPI that sends input linked to a central Linux server that processes the input from the rpi's / all this monitored by 40+ android app for each RPI and a central dashboard on the server, both the RPI's and the central server have Node-red installed and communicate using open TCP ports / HTTP requests and WebSockets .)
I know it's kind of late to ask these questions but I want to know :
1.Can a single instance of node-red in the server handle the communication between the RPI's and the mobile apps (total of 80 devices) and also insert input data in a database? can
Lately I have been experiencing a bad connection between paired devices (RPI <-Server-> Android device), after debugging I found that WebSockets are losing connection after a while, like entering sleep mode, how can I avoid this?
Hi @nodekaiser,
Just a thought, I don't have practical experience, but recently I have read about Apache kafka and it's resources. Seems applicable to your case because could distribute messages and retain and reprocess if one rpis go down/sleep (by example). There is a lot of benefits, my knowledge about kafka still little.
I would have to review what you have, but at a guess, you are using dashboard and graphs and gauges? these can be very heavy if you dont take care to limit data and frequency of updates.
Again, as said above.
TBH, your architecture may need a revisit.
For example, i dont see MQTT mentioned? MQTT is very lightweight and could help with regards the number of TCP connections. Additionally, there are quite a few "dashboards" on Android that talk directly to MQTT. Since an MQTT Broker is stand-alone & can be on a beefy server, you can have thousands of clients without affecting the performance of node-red one single bit.
However, if you insist on building the dashboard in node-red, then a strong recommendation would be to move to the most excellent ui-builder.
From experience, collecting data from 80 machines to database is certainly possible so long as the architecture and protocols are right & you are not "hammering" things (e.g. taking results at the end of a cycle rather than constant on-line ms processing.)
That would be expected behaviour. Websocket clients have a regular ping/pong message exchange and will timeout if they don't keep receiving.
You need a client that is good at reconnecting but on a mobile/battery-powered device, that is a compromise. Most websocket clients have a stepped retry, so the retry gap gets longer over time.
What you really need is a push client of some kind in order to push notifications to the mobile devices, If you can live with a small delay (usually no more than a second but can sometimes be 15-20 sec) then something like a Telegram bot. I'm sure there are plenty of other options though, especially if the Android devices are on the same WiFi network as the central server.
I thought about the issue your having by running it back to a server when I started my giant temperature / nitrogen / ammonia monitoring project at the company.
38 RPI's - all of them have a influx database running on them. That way I have no single point of failure at the server. If a RPI goes down I do loose the data until I can bring it back online and flash its backup that's taken every night at 1am. X fingers I have units been running for over 2 years and not a 1 has gone down on its own. Had a fork lift driver hit one with a fork blade on the beam it was mounted to.
Think about using the RPI's to store the data. Then push it upon request. you could make a api for all units if you dont want a database like I did.
What is your write speed to InfluxDB? I have a pretty high writing speed (up to 60k bytes per second) and RPI 3B is struggling at such speed. So I try to keep the writing period short
lol - I only push 8k to 16k every 30sec. I don't know the max as I have no need to push more than that.
But pull from the database .... sometimes a few MB a sec when gathering 90days of data showing every value taken every 30 seconds for those 90 days. only takes 1-2 seconds to load.
oh and thats 1-2 to load the graph you see above from a old RPI-2
thats running grafana ---> https://grafana.com/
I'm sure if I threw up an instance of grafana on the dell r620 I have sitting in the server rack it would load stupid fast. But the PI does the job for hosting and its a easy backup of the SD card once every 6months. As all it does is access all the databases on the RPI's I'm not too concerned about it taking a dump as I can just flash a new card plug it in and it will run like nothings happened. Sometimes something just works so why make it better.
Cool. It is good to know. Since I was thinking about trying out Sqlite to see if there is any improvement on the write/read speed compared to InfluxDB.
My other option (easier option) is to use RPI 3B+ or RPI 4
That's what I thought. All right I will stick with InfluxDB
My another solution is to offer the option to batch processing the data. Data come in batches and then dump to InfluxDB in batch. It saves lots of system resources this way.
One thing I've noticed with InfluxDB is that things can get slow (at least on a Pi) if you let a measurement table get too large. Try keeping it trimmed down to the smallest you can. If needed, using a continuous query to aggregate long-term data.
For example, my home automation data is mainly per-minute readings from many sensors. I only keep that data for 7 days. But I also aggregate it to hourly data (each value is turned into 3 values per hour: average, max, min) and I keep that for 5 years. It is a good way to optimise the data and its handling.
I'm handling a system where I have about the same number of machines working with Siemens PLCs, plus 6 AGVs controlled by two separate fleet managers.
One Node-Red instance in a virtual server gets some data from each machine (for material tracing/order creation), pushes some of the data into MySQL databases, and shows some on the dashboard.
The data gathering and database access happens normally and without a hitch. In fact, if I disable the dashboard, node-red is only using 0,4% of a single server core to run normally. The performance issues come when I activate the dashboard, specially if I need fast updates.
I tried both accessing the dashboard from a RPi4 and from a smart TV, and their browsers can't cope with the updates, specially when gauges and graphs are involved. I did some optimizations:
I eliminated the LED nodes and used text nodes where I use a full circle icon that changes color with CSS to represent an LED. That alone helped a lot (specially with the 2Hz blink, sending the blink status as a CSS animation instead of updating every half a second)
Using text nodes from the normal dashboard and sending preformatted HTML strings with tables to present multiple data with a single node. That might not have been helped so much in regards to performance, but it made deployment much easier.
Still, the main problem is that the standard Dashboard is loading ALL the tabs ALL the time, so if you have several tabs with quick updates, it will overtax your browser.
As others have suggested, if you need to display a lot of information, go for UI-Builder or just send all the data to a DB and then make a separate frontend to show the data, loading just what you need. The first case might be preferrable if you need fast updates, but the second would help if the updates are not so fast but you need to display a lot of data that can be compartimentalized (only showing part of it at a time).
we spent the last two day's optimizing and fixing the project , to answer few remarks in the answers :
*the Dashboard on the server don't rely on node-red directly, it's a simple web app that show some data from a mySql db , inserting the data in the db is done by node-red .
the frequency of communication / sending data to/ from server is considered low (for each device 1 msg per second or few seconds , in some cases every few minutes ) , the question is how can a single threaded node-red handle two input's that arrive at the same time to the server and insert it in the DB or foreward it to mobile App , isn't there any data loss risk ?
*we didn't use MQTT because the web team working on the Dashboard and the mobile app (by the way a web App compiled in cordova) already started implementing Websockets / HTTP web services so to keep it simple we didn't want to add another method of communication .(just added a few open TCP ports on the server that are connected in Rpi Node-red) .
*yes all the devices (RPI's -server -mobile devices) are on the same network and are connected to internet via wifi , for now in the case of an internet disconnection (the app is disabled with a loader and the rpi continue collecting data without sending it to server , when internet is back the data is sent to the server ) is there a better way to manage this ? , maybe keep the system running without internet ?
Being single threaded, the never ever arrive at the same time
But I understand your query.
What I do is to locally (in node-red) queue the incoming data in flow context (file backed) and de-queue it to the database. There are several nodes and a couple of threads on the forum that discuss this technique.
MQTT would serve well you here. The loading would be reduced for a start.
There are definitely benefits from switching the node-red to node-red comms to MQTT. I wont go into this right now.
I've used that package myself in the past when I just wanted a simple ws: connection that reconnected sensibly. Of course, if you are using Dashboard (or uibuilder), you are also using Socket.IO which is a lot more sophisticated and does reconnections properly. It also has fallback to polling interfaces in case websockets are not available (used to be a real problem with enterprise routers and firewalls, largely gone now).
Hi @nodekaiser, Given that it is too late, I'm still curious about the choices that were made for the software and hardware architecture. I'm a retired embedded software developer for Rockwell Automation/Allen-Bradley so I'm biased; but I know there a a lot of small controllers out there, not only from RA, but several others. Most of those vendors would provide a complete factory automation solution and the small team could concentrate on the factory's application. Was it a matter of cost? Performance?
I can't speak for @nodekaiser but if I had to implement something at this moment, I'd try to get a controller that can be programmed in node-red instead of your usual PLC. I've worked with Siemens and Mitsubishi, and sure, they offer reasonable solutions, but:
They are proprietary. In my case, one of our suppliers included 30+ Siemens S7 PLCs that need the Step 7 PRO license (we only had the basic), and that is already several thousand euros in licenses, just to be able to open and monitor the program to see why a machine is not running. Mitsubishi is about the same (although a bit cheaper).
They use their own communication protocols, and if you have to interface different systems, you require expensive adapters to bridge between them. Once you select a PLC maker, you're married with them. Anything outside that, you're out of luck.
On the other hand, node-red is open, free to license and use, easy to get started with, can be developed and tested in a simple raspberry pi or in a computer and used from a browser on any device in the network, and can communicate with many different PLCs from different vendors, with computers, and read/write data from a database without any additional hardware modules.
If you want to prototype something, you can get a Raspberry Pi for less than 50€ and get started. later, you can get one of those fancy raspberry pi-based PLCs that have already I/O interfaces, and just dump the program in there. Still much cheaper than a propietary-based PLCs and system-agnostic.
As a bonus, and since node-red runs in basically a small computer, you can use any touchscreen as an HMI, running a browser in full screen.
Hi @accribbs , the choice was based mainly on previous experiences, we did a few projects with RPIs and Node-REd: park management and automation, an energy-saving solution with Z-Wave components for caravans .., i honestly don't know much about other controllers, but for us (working in African and European Market) we found RPI's very accessible and offer total freedom to customize the solution, we used Node-Red because we needed fast implementation, and our App was Event-driven. the customer demanded to keep thing's centralized in one central server on-site, so we went with (star topology) as architecture, if you have better suggestions u'r welcome
