Using Node-RED to import & reformat InfluxDB 1.8 data

General
1

I am working on a project whereby we will start Jan 1, 2022 with a clean Influx 2.0 database. All the fields & tags in the new database have been thought through and configured, new Flux queries written, etc.

From roughly mid-2020 thru Dec 31, 2021, we have been populating an Influx 1.8 database, but with very poorly constructed data.

To correctly organize the 1.8 data and store it in the 2.0 data, my plan was to use Node-RED to retrieve the 1.8 data, pass it through some functions to parse out the correct fields & tags, and then import (with the original timestamp) into the 2.0 database.

Here is an example:

1.8 data measurement:

  • fieldname: HTFN6Z1TSP
  • value: 354

2.0 data measurement:

  • temperature: 354 <-- this is the field
  • EquipType: Furnace <--- these are the 4 tags
  • EquipNumber: 6
  • EquipElement: Zone 1
  • MeasType: Actual

I am pretty sure that I can construct a function that will do the above parsing, but before I head down that path, does anyone foresee any problems with this approach?

We have a lot of data points. Probably 25 or 30 every minute just like the above (but with different equipment numbers, zones, etc.) going back about 1.5 years. Not sure how processor intensive a conversion like this would be or how long it would take.

2

How are the tag values to be determined?

I can't see any problem in principle with this. I would not try and read in too many at a time though, read it in maybe one hours worth at a time, process those, then read in the next hour.

It may take a long time, but does it matter?

Presumably you can do a series of short test runs to check it is working, and delete the new database and start again if necessary, or is there live data already going into the new one?

3

Sometime over the next few weeks, I would do a small demo to ensure it works as planned. I would probably do the actual conversion & import between Christmas and New Year's when the equipment is not operating, so hopefully ~6 days is enough time. Even if we ran into early 2022 (when live data will start flowing into the new DB), we are just populating old data points, so even that is not a deal breaker.

Here is how the tag values would be determined. Again, I think I could create a function node to do this (with some help of course!).

HTFN6Z1TSP

  • HTFN = EquipType tag: furnace (possible EquipType include Furnace, Generator, etc.)
  • 6 = EquipNumber tag: 6 (possible EquipNumber values include 6, 22, 301, etc.)
  • Z1 = EquipElement tag: zone 1 (possible EquipElement values include Zone 1, Zone 2, etc.)
  • SP = MeasTyp tag: setpoint (possible MeasTypes include Setpoint, Actual)

GNRT301Z2PV

  • GNRT = EquipType tag: generator (possible EquipType include Furnace, Generator, etc.)
  • 301 = EquipNumber tag: 301 (possible EquipNumber values include 6, 22, 301, etc.)
  • Z2 = EquipElement tag: zone 2 (possible EquipElement values include Zone 1, Zone 2, etc.)
  • PV = MeasTyp tag: actual (possible MeasTypes include Setpoint, Actual)
4

Yes, that looks eminently do-able.
I have to do a something rather similar myself, I have 10 years of historical weather data for here in an sql database, but have now moved to Influx. I have to transfer the historic stuff across the influx so I have the history.

5

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.