Logging multiple MQTT sources to CSV

General
1

Greetings all. I've read a few threads on logging to CSV and sort of have the gist.

What I'm trying to determine is the appropriate methodology to meet my goal.

Data source:

  • Cogeneration plant with many possible sensors and parameters (over 500)
  • Multiple MODBUS RTU devices
  • All feed into a single protocol converter sending MQTT
  • 5 data sets at differing send intervals of .2sec, 1 sec, 5 sec, 30 sec, etc
  • MQTT sends to a single broker

Ubuntu server

  • MQTT broker
  • Node Red w/ UI

Goal

  • Log all incoming data into a single CSV with timestamps sent from source protocol converter
  • populate fields in CSV with previous value if no updated value comes across
  • as log files will likely become large, split logs in fixed time intervals

Currently I have a flow to use a dashboard and provide remote monitoring, but we are having sporadic trips of the plant offline and I'm looking to graph data from CSV with better tools than built in to the plant control system.

Any thoughts, concerns, or advice appreciated.

node red flow
node red flow1302Ă—837 186 KB

2

First question: why CSV? Why not store the data in a database?

I would break the process in two. Part one would be recording the data to a database. Part two would be processing the data. You could read from the DB to populate the dashboard at will and have an option to create a CSV file on demand.

That’s my two cents

3

+1 to using a db. I would log them all to influx as they come in and do whatever post processing and data extraction you want later. You would end up with a much simpler solution.

4

CSV is desired to be able to import into external programs such as MegaLogViewer HD or Motec's I2 Pro data analysis programs for generation of histograms, etc.

Going into the database first may be more efficient in the implementation, but I would have to be able to export it back out to .csv for analysis.

Appreciate the thoughts.

5

You can export directly from influxdb to csv, performing transformation of the data on the way, such as min max mean sum, etc. So you can leave till later the decision on exactly what you need in the csv files.
Alternatively of course you could use something like Grafana for visualising the data.
I am surprised that MegaLogViewer appears to be not able to read from Influx, or other databases as far as I can see, which rather limits its usefulness.
.

1 Like
6

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.