Hi to all. In this forum several times has been touched the aspect that node-red is single user and not multi-tenant, sharing and suggesting several approach in order to reach the wanted goals.
In a more structured way I'm trying to make a conenction between Grafana, which borns as multitenant, and node-red itself. I already have the project avanced developed in node-red and I'm starting this connection with Grafana, moving some information in its dashboard, a platform that I'm starting and learning to know.
For this reason I was looking for some support on Grafana side. Since obviously it's not correct to use this forum in this step that implies more Grafana aspects that node-red ones, I would appreciate if someone interested in it would contact me even privately, sharing later on the results that could interest other users.
Thanks.
Just to give you a slight overview, the idea is to locally install node-red and remotely, on a server, to install node-red, grafana and a server mqtt. In this way the local node-red instance will communicate with the server via mqtt, and the user will connect with grafana directly on the server. Node-red is necessary server side and local side in order to perform several IoT action, other than collect and share data on database
If this provides a route to closer integration between node-red and Grafana I think that many here would be interested in it. Also a number here may have the knowledge to help, though it isn't clear what sort of knowledge you are looking for. I suggest you ask here and see what transpires. You can always move off forum later if it becomes too far off topic.
Ok, as the first step I need help for this.
As database I thought to go ahead with MySQL since I know there is the possibility to automate the synchronization between the local and the server installation.
It should work like drop-box/one-drive..., simplifying a lot data transfer and synchronization. But I'm still not able to properly set the configuration. This because Grafana needs in anyway a database to read from.
In the meanwhile I need to transfer some graph and table from node-red to Grafana. The main problem I'm encountering is that while in node-red I'm able to use several ways for manipulating data, choosing if, where and when use them, temporarily or globally store and use intermediate results, etc., I find a bit hard transfer this in Grafana, which borns for different scopes and with different peculiarities. Keeping in mind the use of environment variables in order to load data bounded to the logged user. But of course I'm sure this is because of my lack of adequate knowledge.
So at this step, as I said, the many help I need is for sure more related to Grafana aspect and the way it works.
For example, just to start, at the moment as single node-red project (let's call it system), each one needs at least 2 tables:
one with 200 columns and 500.000 rows updated each second;
one with 15 columns and up to 200.000 rows update every10 minutes.
In this case would be better organize a unique database with several tables, two for each system, or maybe just a unique big table with one user ID column and one system ID column?
Regarding the users, looking at Grafana's organization, team and user structure, there must be at least 2 type of users:
- who can only view data for one or more systems related to him
- who can even make IoT action for one or more systems, always related to him, of course.
That's 100,000,000 cell contents per second! What hardware will this be running on?
Also what sort of data is it in the tables and where is it coming from?
No, I explained myself in a wrong way. Every second I store one row with 200 items and every hour I delete all the rows related the oldest hour. At the moment this is running on a Pi 3B, and the cpu load is most than affordable
and it's even streaming an IP camera and recording it
data are mainly numbers (float, int, smallint and tinyint), coming from sensors and photovoltaic system (production, solar radiation, alarms (1/0), and so on).
At the moment for the IoT action I don't need to save the status on a database, and this is another point I'm questioning how to face with Grafana, for example
Do you then summarise that into a downsampled record somewhere?
Exactly, not all but some are summarized into the second table every 10 minutes. The values of the first table must still be available to be filtered and displayed, as well
In that case is there a reason you are not using InfluxDB, which has built in the ability to delete old data and to downsample data into a new table?
No, there is no real reason, actually. When I started this project I knew MySQL so I used it, knowing it could handle this amount of data without problems. I don't see a big problem in changing it. Just, as I said, I would semplify the data synchronization and i know MySQL can do it.
Anyway, when I summarize the data in the second table I need to add some other colums
InfluxDB is ideal for this situation and already has excellent integration with Grafana of course.
However, Grafana is not so good with non-timeseries data. Though it is certainly possible to build extensions, especially with the newer versions (not tried it myself but there are some excellent extensions such as the SNMP ones that I need to get round to trying).
I second the recommendation of using InfluxDB. It works very well with Node-RED and Grafana. This video is an excellent jumping off point.
And of course https://community.grafana.com/ is a good place to ask Grafana questions, while https://play.grafana.org/ is a good place to kick the tires.
Back on the original question, I am not clear about what you are looking for help with.
Simplifying, at first just transfer some panel from node-red dashboard to Grafana.
For example it's a while I'm facing with these easy stuff on node-red but apparently not so intuitive to overcome on Grafana
Do you mean that you want to implement in grafana a dashboard similar to that in node-red? If so then I still am not sure how that relates to your original post where you said
Maybe I'm not that good to explain it better in a short way
I imagined there could be this risk opening the topic, if I was able to use Grafana in a great way probably I wouldn't have askd for help.
Ok going step by step (it was just an easy example more related to Grafana that node-red as I underlined), but every step could have a number of different solution and without understand the overall plan and put toghether the issues and how they're related each others, every single solution just aimed at itself coulb be not the right approach. Indeed, it could open up other problems. This why I would have avoided it.
And, again, as I said I know it's much more related to Grafana, since in node-red the project is working perfectly and in a great way. It doesn't need at the moment to be changed in any way.
My contribution to this forum on this stuff would like to be to show at the end all the problems, how they have been solved amd how Grafana and Node-red can work together.
I decided to firstly post this help request in node-red Forum just because I think node-red is the starting step and will continue to maintain an important role in all the project.
That's it.
So is there any specific issue you are looking for help with (whether node-red or grafana related)?
I presume that you saw this thread, though you have not commented there. Using Grafana as Node-Red Dashboard?
