Colin, I have done that. Sometimes the data is visible in Data Explorer and sometimes not but when it is visible, it does not seem to recognise the timestamp. I'm struggling to work out why the data simply does not appear in the bucket (with or without timestamp) with no error messages. Changing to a simple table view in Influxdb 2.3 UI does make it a little easier.
I see from @grant1 that you helped him with the suggestion of using the batch node, which I've never used before. That could work well as I have what Grant had also done, which is to split the payload of 280ish objects for snapshot values every 5 minutes for 24 hours but I throttled the delivery of individual messages to 10/sec. I'll squirt the payload at a batch node and see what happens.
What you have should work, there is no need to complicate it. Show us what you get with the time field in the payload.
You still have not said what you have set the time precision to in the node, unless I missed it. If you don't answer questions it is difficult to help.
Colin, thanks for your forebearance. The Time Precision is set to mS which is what the timestamp resolution is downloaded from the cloud and the same as the Influxdb 1.8 production system, (albeit it with different data).
{"measurement":"InverteData","fields":{"time":"23:58:19","status":"NORMAL","vpv1":28.1,"vpv2":27.3,"ipv1":0,"ipv2":0,"ppv1":0,"ppv2":0,"ppv":0,"pac":32,"vacr":246,"iacr":1,"fac":50.01,"loadPower":0,"meter1ActivePower":6,"peps":0,"invVoltage":244.5,"invFrequency":50.01,"temperature":40.7,"batVoltage":52.94,"batPower":22,"batTemperature":27,"batPercent":93,"pvEnergyToday":22.7,"pvEnergyTotal":3029.5,"consumptionEnergyToday":0,"consumptionEnergyTotal":584.2,"exportedEnergyToday":29.6,"gridExportEnergyTotal":2618.6,"etoday":22.4,"etotal":2957.5,"invImportEnergyToday":0,"invImportEnergyTotal":17.4,"importedEnergyToday":0.1,"gridImportEnergyTotal":262.7,"chargeEnergyToday":2.8,"charge_energy_total":688.45,"dischargeEnergyToday":2.5,"discharge_energy_total":688.45,"timestamp":1658188699,"id":2730020,"import":0,"export":32},"tags":{"device":"SD2137G164"},"timestamp":1658188699}
This is the data object I've built by splitting the original payload, manipulating it and then joining again (I struggled to understand the Array.Map() function). Squirting this at the Influxdb instance using the batch node produces nothing, not even any error messages.
Since I also struggle to remove data when I am able to load something, the simplest way to clean the bucket is to delete it and recreate it. I've also reapplied the API token in Node Red to see if there was an issue there, but doesn't seem to be.
You have time in the fields and separately. That isn't going to work. I don't know why you have switched to the batch node before doing what I suggested with the normal output node.
Are you sure that is a millisecond timestamp. I think the one you shared earlier was seconds.
When you interrogate the database are you asking for all the data in that measurement?
Yes, I know. I can pull out the timestamp from the fields object but that's where it sits when it comes from the cloud.
I did do what you suggested, in fact I have been doing that for several days now, to no avail. I have built a message with minimal fields in; that didn't work; I even tried manually loading data from the Influxdb UI and that didn't work. It's why I said that maybe I needed to ask the question on an Influxdb forum.
Yes you're right. I'll change the Time Precision in the batch node. I have tried sending the data without a timestamp, relying on the ingestion time to set a timestamp, but data doesn't show up.
The Influxdb 2.3 UI shows buckets on the left, the a filter for measurements, then a filter for fields.
You can see the measurements for the sample_data in this screen shot. There are (usually) no meaurements for my "energy" bucket and when they do appear, they don't have timestamps. Sometimes measurements appear, but usually only after a period of time has passed. That's the reason why I keep deleting the bucket and recreating it, so I can work out what it is that I'm doing right but it feels quite random.
I've removed the timestamp from the fields object
{"measurement":"InverterData","fields":{"time":"23:58:19","status":"NORMAL","vpv1":28.1,"vpv2":27.3,"ipv1":0,"ipv2":0,"ppv1":0,"ppv2":0,"ppv":0,"pac":32,"vacr":246,"iacr":1,"fac":50.01,"loadPower":0,"meter1ActivePower":6,"peps":0,"invVoltage":244.5,"invFrequency":50.01,"temperature":40.7,"batVoltage":52.94,"batPower":22,"batTemperature":27,"batPercent":93,"pvEnergyToday":22.7,"pvEnergyTotal":3029.5,"consumptionEnergyToday":0,"consumptionEnergyTotal":584.2,"exportedEnergyToday":29.6,"gridExportEnergyTotal":2618.6,"etoday":22.4,"etotal":2957.5,"invImportEnergyToday":0,"invImportEnergyTotal":17.4,"importedEnergyToday":0.1,"gridImportEnergyTotal":262.7,"chargeEnergyToday":2.8,"charge_energy_total":688.45,"dischargeEnergyToday":2.5,"discharge_energy_total":688.45,"timestamp":1658188699,"id":2730020,"import":0,"export":32},"tags":{"device":"SD2137G164"},"timestamp":1658188699}
and corrected a typo in the measurement name. The format looks OK but it's still not getting to Influxdb.
I have tried pulling some coindesk measurments from the sample_data bucket and that worked OK with a payload of 1764 objects all looking good. From this I'm taking it that the configuration settings for the server are OK.
Show me the data you are sending now. When you say the data doesn't appear, does it create the measurement and are the fields available in the filter?
Colin, the data being sent is in the code snippet in my previous post but I've just realised that my attempt to remove the timestamp as a key value has failed.
{"measurement":"InverterData","fields":{"time":"23:58:21","status":"NORMAL","vpv1":31.5,"vpv2":28.4,"ipv1":0,"ipv2":0,"ppv1":0,"ppv2":0,"ppv":0,"pac":20,"vacr":248,"iacr":2,"fac":50.14,"loadPower":333,"meter1ActivePower":354,"peps":0,"invVoltage":247.7,"invFrequency":50.15,"temperature":41.3,"batVoltage":53.12,"batPower":428,"batTemperature":28,"batPercent":91,"pvEnergyToday":20.9,"pvEnergyTotal":2988.3,"consumptionEnergyToday":0,"consumptionEnergyTotal":475.7,"exportedEnergyToday":26.6,"gridExportEnergyTotal":2637.4,"etoday":20,"etotal":2929.7,"invImportEnergyToday":0,"invImportEnergyTotal":25.6,"importedEnergyToday":0.1,"gridImportEnergyTotal":209,"chargeEnergyToday":4.1,"charge_energy_total":731,"dischargeEnergyToday":3.2,"discharge_energy_total":731,"id":2750315,"import":0,"export":20},"tags":{"device":"SDxxxxxxxx"},"timestamp":1658275101}
This should be OK now.
When clicking on the energy bucket, nothing appeared in the next selection (Tags I think) and perhaps that's the root of the problem? However, if I click on the next selection to the right, I can see _field and then _measurement.
I was expecting all the key values to be entered into one datapoint of the database as separate columns, ie an object with fifty key values and one timestamp would be one row of data but it looks like each key value is being entered as a separate datapoint. It seems that the data is now being loaded and it may just be my lack of understanding that means that what I'm seeing is not what I would expect. Certainly using the CLI with 1.8 and doing a SELECT statement brings back all the key values for one timestamp along with the tags. But I am making some progress now.
Isn't the field selection just asking you what you want to see? So the equivalent of * would be to select them all. What do you see if you select some of them?
Colin, You may well be right, it's just not what I was expecting. When I select several fields, I still get one row of data for each key value although it does include the measurement and device as well. Annoyingly, the timestamp does appear to be associated with these data but I can't view it and when I export to csv, I get a spreadsheet that shows the start and stop times for the query selection (identical for every row) but no timestamp. The only way I can get timestamp is to select a field and group by time but this shows the field value but not the field name.
I'm sure it'll come good, I just need to reprogram myself to think like a time series database developer, then I'll get exactly what I'm looking for. The next step for me is to query the bucket and pull back the latest timestamp for each device and then pull the next days' data for that device. It wouldn't be a problem to increment each day of data and just handball it manually but I want to use Node-RED to automate the download of yesterdays data for days going forward.
If you are having problems with the gui then you might be better using an influx node in node-red and then you can easily tweak the query and see what you get out.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.
