Time Stamp and Influx DB problem

eng56 · 26 July 2019 12:23

Hello, I'm injecting in my database an array of 2 elements: the first one is the fields and the second one is the tags as sugested here : [Influx DB contrib ][(https://flows.nodered.org/node/node-red-contrib-influxdb)]

In field I have put my timestamp because I don't want Influx DB to set the time_stamp, I wan't to set it.

This is my debug message (msg.payload) before the input of InfluxDB node (the classic one not the Batch node):

[{"time":1564134384,"Concentration_C02":4080,"Temperature":40.8,"Relative_Humidity":102,"Frame_Index":2},{"Device":"test_datalog","Type_Produit":"Insafe_Carbon_Sigfox","Type_Message":"Datalog"}]

This my error message coming from InfluxDB node: Error: A 400 Bad Request error occurred: {"error":"partial write: points beyond retention policy dropped=1"}

This my DB retention configuration: ( I have deleted autogen retention policy)

name  duration  shardGroupDuration replicaN default
----  --------  ------------------ -------- -------
un_an 8760h0m0s 168h0m0s           1        true

Thank you for your help

Colin · 26 July 2019 14:08

Try adding three zeros to the end of the timestamp. I believe it needs to be a javascript timestamp, which is in milliseconds.

eng56 · 26 July 2019 14:52

Thank you for your answer now node-red recognize that it's a timestamp and display it as a date (without any other node as moment) but I still have the same problem :Error: A 400 Bad Request error occurred: {"error":"partial write: points beyond retention policy dropped=1"}

Colin · 26 July 2019 14:57

Is it ok if you don't give it the timestamp, so that influx adds the timestamp? I know you don't want to do this, just checking whether it is ok otherwise.
Also I notice that the influxdb out node does not say that you can include a timestamp. Have you tried the same thing using the batch node, which does say a timestamp can be included.

eng56 · 26 July 2019 15:13

Yes it's working very well when I don't change my timestamp but I'm receiving one payload every hour but each payload contains 3 messages (one measurement every 20 minutes).

I have just checked the InfluxDB contrib info and the time can be injected I think:

msg.payload = [
    [{
        numValue: 10,
        randomValue: Math.random()*10,
        strValue: "message1",
        time: new Date("2015-12-28T19:41:13Z").getTime()
    },
    {
        tag1:"sensor1",
        tag2:"device2"
    }],
    [{
        numValue: 20,
        randomValue: Math.random()*10,
        strValue: "message2",
        time: new Date("2015-12-28T19:41:14Z").getTime()
    },
    {
        tag1:"sensor1",
        tag2:"device2"
    }]
];
return msg;

Colin · 26 July 2019 16:33

Where have you found that example? Also the example shows an array of arrays whereas you have just shown an array in the data you showed.

eng56 · 26 July 2019 20:33

You are right it was a mistake, finally I have choosen the batch node and I'm injecting a message according to the influx node doc(the last exampke) but I still have the same error message

Error: A 400 Bad Request error occurred: {"error":"partial write: points beyond retention policy dropped=3"}

[{"measurement":"real_time","fields":{"Concentration_C02":3400,"Temperature":34,"Relative_Humidity":85,"Frame_Index":2},"tags":{"Device":"test_datalog_now","Type_Produit":"Insafe_Carbon_Sigfox","Type_Message":"Datalog"},"timestamp":1564128384000},{"measurement":"real_time","fields":{"Concentration_C02":3740,"Temperature":37.4,"Relative_Humidity":93.5,"Frame_Index":2},"tags":{"Device":"test_datalog_now","Type_Produit":"Insafe_Carbon_Sigfox","Type_Message":"Datalog"},"timestamp":1564131384000},{"measurement":"real_time","fields":{"Concentration_C02":4080,"Temperature":40.8,"Relative_Humidity":102,"Frame_Index":2},"tags":{"Device":"test_datalog_now","Type_Produit":"Insafe_Carbon_Sigfox","Type_Message":"Datalog"},"timestamp":1564134384000}]

Thank you for your support

mike · 27 July 2019 08:04

From the error message, it looks like the timestamps you've provided to the Batch output node are numbers, so these will be passed in to InfluxDb for parsing. Since InfluxDb uses nanosecond precision by default, these millisecond dates may be outside of your retention policy, causing this error to be thrown.

https://node-influx.github.io/class/src/index.js~InfluxDB.html#instance-method-writePoints

Have you tried using Javascript Date objects for your timestamps as in the example? E.g. "timestamp": new Date(1564128384000) rather than "timestamp":1564128384000? This should cause the library to convert to the right precision. I think another option might be to specify the precision (e.g. 'ms') with the writes as in the writePoints documentation.

eng56 · 27 July 2019 08:03

If I let my time-stamp in miliseconds Node Red recongnize automatically that it's a timestamp and in the debug I can display it as a date.
If I multiply by 1000 (to have a nanoseconds) node red don't recognize that it's a date.

Solution: To use a batchnode and using the Date object using a miliseconds precision

Thank you for your help !

Colin · 28 July 2019 08:13

You only need to multiply it by 1000 when sending it to the influx node. Leave it as a javascript date when using it elsewhere. That would be an alternative solution.

Paul-Reed · 23 November 2019 21:42

Multiplying an epoch in milliseconds (1564128384000) by 1000 results in microseconds.
Should be multiply by 1000000 which gives (1564128384000000000)