Influxdb Missing Data

[JCLCN]

Hi, I am new to node red. I'm using it to send some data to influxdb at a sampling rate of 1000Hz. The data is coming from a tcp in node (connected to OpenSignals (r)evolution). The data is in an array form. I've placed a debug node after the function node to check if all the wanted elements are passed through the function. The debug shows that all wanted elements are passed however when I connect it to Influxdb, some of the data are missing.

I've attached the picture from node red below.

I have tried adjusting the time precision in Influxdb to millisecond, microsecond and nanosecond but there are still data missing.
Another trial was using a batch node and to send the message once 1000 messages was received but it still doesn't work.
(OpenSignals only can be connected to Node red via TCP ports)

Thanks in advance ya

[bakman2]

On what hardware are you running node-red?

[Colin]

In what way are some of the data missing? Missing fields? Missing rows?
Have a look at the influx log and see if anything showing there.

[JCLCN]

@bakman2 my desktop windows 10

@Colin yep I've checked the influx log. Some of the data from node red are stored successfully but quite a number of rows are missing.

[Colin]

If you slow it down to 1/10 the speed is all the data stored?

[JCLCN]

@Colin slow it down as in change the sampling rate to 100Hz? If so I have tried and not all the data are stored

[JCLCN]

I tried with a simple example where I insert an array [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20] and not all the number are inserted into the influxdb.

Influxdb log: (Only 2 out of 20 numbers are stored)

My functions:

• F1:
  for (var i = 0; i <20; i++){
  var value1 = msg.payload[i];
  node.send({number:value1});
  }
  return;

• F2:
  var value1 = msg.number;
  msg.payload = {value:value1};
  return msg;

[bakman2]

You are not actually sending an array, but separate inserts, but in this case it does not matter. If you send 100 or more, you should prepare an actual array and use the batch node.

Your issue is caused by the lack of uniqueness:

InfluxDB identifies unique data points by their measurement, tag set, and timestamp

The timestamp could sometimes help, but if you insert many at the same time, the timestamp will remain the same, even in nanosecond precision.

Note that the 'value' is not taken into account, it simply overwrites the previous 'value'.
You can solve this by adding a tag with the value to make it unique. example; 'uniq:1','uniq:2'

function node example:

arr = []

for(x=0;x<1000;x++){
    arr.push(
        {
            measurement: "value_test",
            fields: {
               value:x
            },
            tags:{
                uniq:x
            }
          
        }) 
}

return {payload:arr}

Attach it to a influx batch node

[Steve-Mcl]

Hi. Firstly, please format code by putting ``` above and below.

Secondly,, not sure why you need F2 (just send {payload: value1} in the F1 loop).

Lastly, it might be the design of the influx node to discard incoming msgs if it is busy performing a task. If so, you might need some form of queue controlled by the completion of the influx node.

[Colin]

@Steve-Mcl

it might be the design of the influx node to discard incoming msg s if it is busy performing a task

I hope that is not the case, I would consider it a bug if so.

[JCLCN]

@bakman2 this works! All my data are stored in Influxdb now. Thank you so much!