Just checked, and if the all the data and timestamp is the same then "effectively" nothing will change in the database, and so no harm writing the same data over and over to update with the latest Ovo records, but of course it still has to be processed.
If any of the values do change (which they shouldn't as they come from the same Ovo database) then the last value overwrites the previous)
Thats good to hear influx works like that.
Iāve finally got round to looking at and trying the code out. Iāve found a few issues so far.
For some reason the date function on my rasp pi returns the previous day? So even if I edit the day node to subtract 0, I still get the previous days date. Iām running this at 00:20 if that has any impact. Iāve logged in to the pi and date returns the correct date.
I think some of the checking for no data need amending. If no data is available a msg with null electricity & gas objects is returned. These need checking for before the ālet gas = msg.payload.gas.dataā is executed
Use BASH to check the Pi's date/time is correct and then also make sure it is set to the correct timezone.
From bash the date time looks fine:
pi@raspberrypi-400:~ $ date
Sun 05 May 2024 12:11:26 AM BST
pi@raspberrypi-400:~ $
pi@raspberrypi-400:~ $ date
Sun 05 May 2024 01:04:38 AM BST
pi@raspberrypi-400:~ $
However, now itās after 1:00am, the date returned in nodered is correct, ie 5/5/2024. So I think itās something todo with BST. Are there different time settings for nodered?
If you mean in my code, that is deliberate. The Ovo data is only available for the previous day, and even then it's usually after 8am before you can grab it.
Not sure where you are seeing that ? - the function displays no data and returns null to the first output, the second output is for the trigger node, so that the request is repeated after an hour.
Nope, it should pick up the servers date and tz. Except that most nodes will use UTC.
Going to bed now - catch you tomorrow. 
I just squashed the data collection into one function, looks neater, but works the same.
Send the debug to an influxdb batch node, I have set the retention to "month" for the 1/2 hour data, so create a retention for that, or remove from the code.
EDIT: added more checks for partial data download.
[{"id":"5fa27756ada8a4f4","type":"function","z":"712086ff.3e3668","name":"Process Data","func":"// Check for status code errors\nif (msg.statusCode !== 200) {\n const text = `error - statusCode: ${msg.statusCode} - ${msg.payload.message}`;\n node.error(text);\n node.status({ fill: \"red\", shape: \"dot\", text: text });\n return;\n}\n\n// Function to process data for different topics\nfunction processData(mode) {\n let msgData = [];\n\n // Initialize date and index based on mode\n let date = new Date();\n let index;\n\n if (mode === \"monthly\") {\n index = date.getMonth() > 0 ? date.getMonth() : 12;\n } else {\n index = date.getDate() - 1; // array starts at 0\n }\n\n // Filter data based on mode \n const gas = mode === \"monthly\"\n ? (msg.payload?.gas?.data?.filter(el => el.month === index) || [])\n : (msg.payload?.gas?.data || []);\n\n const elec = mode === \"monthly\"\n ? (msg.payload?.electricity?.data?.filter(el => el.month === index) || [])\n : (msg.payload?.electricity?.data || []);\n\n // Function to check data availability\n function checkDataAvailability(mode, data, index) {\n if (mode === \"daily\" || mode === \"monthly\") { return data && data[index] !== undefined; } // Daily & Monthly mode check\n if (mode === \"half-hourly\") { return data && data.length === 48; } // Half-hourly mode check\n return false; // If mode is not recognized, return false\n }\n\n // Check data availability for both gas and elec data\n const isGasAvailable = checkDataAvailability(mode, gas, index);\n const isElecAvailable = checkDataAvailability(mode, elec, index);\n\n // If either gas or elec data is not available, handle the error\n if (!isGasAvailable || !isElecAvailable) {\n node.warn(`Missing ${mode} data for ${msg.targetDate}`);\n node.status({ fill: \"yellow\", shape: \"dot\", text: `Missing ${mode} data for ${msg.targetDate}` });\n return;\n } else {\n node.send([null, { reset: true }]);\n node.status({ fill: \"green\", shape: \"dot\", text: `Success ${mode} data for ${msg.targetDate}` });\n }\n\n // Process and create `msgData` based on the mode\n if (mode === \"monthly\") {\n msgData.push({\n measurement: \"meters_monthly\",\n fields: {\n kWh: gas[0].consumption,\n cost: parseFloat(gas[0].cost.amount)\n },\n tags: { type: \"gas\" },\n timestamp: new Date(`${gas[0].year}-${gas[0].month}-28`).getTime()\n });\n\n msgData.push({\n measurement: \"meters_monthly\",\n fields: {\n kWh: elec[0].consumption,\n cost: parseFloat(elec[0].cost.amount)\n },\n tags: { type: \"electric\" },\n timestamp: new Date(`${elec[0].year}-${elec[0].month}-28`).getTime()\n });\n } else if (mode === \"daily\") {\n msgData.push({\n measurement: \"meters_daily\",\n fields: {\n kWh: gas[index].consumption,\n cost: parseFloat(gas[index].cost.amount),\n rate: gas[index].rates.anytime,\n standing: gas[index].rates.standing\n },\n tags: { type: \"gas\" },\n timestamp: new Date(gas[index].interval.end).getTime()\n });\n\n msgData.push({\n measurement: \"meters_daily\",\n fields: {\n kWh: elec[index].consumption,\n cost: parseFloat(elec[index].cost.amount),\n rate: elec[index].rates.anytime,\n standing: elec[index].rates.standing\n },\n tags: { type: \"electric\" },\n timestamp: new Date(elec[index].interval.end).getTime()\n });\n } else if (mode === \"half-hourly\") {\n for (let index = 0; index < gas.length; index++) {\n\n msgData.push({\n measurement: \"meters\",\n fields: {\n kWh: gas[index].consumption\n },\n tags: { type: \"gas\" },\n timestamp: new Date(gas[index].interval.end).getTime()\n })\n\n msgData.push({\n measurement: \"meters\",\n fields: {\n kWh: elec[index].consumption\n },\n tags: { type: \"electric\" },\n timestamp: new Date(elec[index].interval.end).getTime()\n })\n }\n msg.retentionPolicy = \"month\"\n }\n\n return msgData;\n\n}\n\nmsg.payload = processData(msg.topic);\n\nif (!msg.payload) { return ([null, msg]); } // start trigger node, to re-request data\n\nreturn msg\n\n\n","outputs":2,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1935,"y":1425,"wires":[["fddc7b6a3a060484"],["bcc293757e5627df"]],"info":"// Check for status code errors\r\nif (msg.statusCode !== 200) {\r\n const text = `error - statusCode: ${msg.statusCode} - ${msg.payload.message}`;\r\n node.error(text);\r\n node.status({ fill: \"red\", shape: \"dot\", text: text });\r\n return;\r\n}\r\n\r\n// Function to process data for different topics\r\nfunction processData(mode) {\r\n let gas, elec, msgData = [];\r\n\r\n // Initialize date and index based on mode\r\n let date = new Date();\r\n let index;\r\n\r\n if (mode === \"monthly\") {\r\n index = date.getMonth() > 0 ? date.getMonth() : 12;\r\n } else {\r\n index = date.getDate() - 3; // start at 0\r\n }\r\n\r\n // Filter data based on mode\r\n gas = msg.payload.gas.data.filter(el => {\r\n if (mode === \"monthly\") return el.month === index;\r\n if (mode === \"daily\") return el.date === index;\r\n if (mode === \"half-hourly\") return el.interval.end === index; \r\n });\r\n\r\n elec = msg.payload.electricity.data.filter(el => {\r\n if (mode === \"monthly\") return el.month === index;\r\n if (mode === \"daily\") return el.date === index;\r\n if (mode === \"half-hourly\") return el.interval.end === index; \r\n });\r\n\r\n // Check if data is available\r\n if (gas.length === 0 || elec.length === 0) {\r\n node.warn(\"No Data\");\r\n node.status({ fill: \"yellow\", shape: \"dot\", text: \"No Data \" + msg.date });\r\n return [null, msg];\r\n } else {\r\n node.send([null, { reset: true }]);\r\n node.status({ fill: \"green\", shape: \"dot\", text: \"Success \" + msg.date });\r\n }\r\n\r\n // Process and create `msgData` based on the mode\r\n if (mode === \"monthly\") {\r\n msgData.push({\r\n measurement: \"meters_monthly\",\r\n fields: {\r\n kWh: gas[0].consumption,\r\n cost: parseFloat(gas[0].cost.amount)\r\n },\r\n tags: { type: \"gas\" },\r\n timestamp: new Date(`${gas[0].year}-${gas[0].month}-28`).getTime()\r\n });\r\n\r\n msgData.push({\r\n measurement: \"meters_monthly\",\r\n fields: {\r\n kWh: elec[0].consumption,\r\n cost: parseFloat(elec[0].cost.amount)\r\n },\r\n tags: { type: \"electric\" },\r\n timestamp: new Date(`${elec[0].year}-${elec[0].month}-28`).getTime()\r\n });\r\n } else if (mode === \"daily\") {\r\n msgData.push({\r\n measurement: \"meters_daily\",\r\n fields: {\r\n kWh: gas[index].consumption,\r\n cost: parseFloat(gas[index].cost.amount),\r\n rate: gas[index].rates.anytime,\r\n standing: gas[index].rates.standing\r\n },\r\n tags: { type: \"gas\" },\r\n timestamp: new Date(gas[index].interval.end).getTime()\r\n });\r\n\r\n msgData.push({\r\n measurement: \"meters_daily\",\r\n fields: {\r\n kWh: elec[index].consumption,\r\n cost: parseFloat(elec[index].cost.amount),\r\n rate: elec[index].rates.anytime,\r\n standing: elec[index].rates.standing\r\n },\r\n tags: { type: \"electric\" },\r\n timestamp: new Date(elec[index].interval.end).getTime()\r\n });\r\n } else if (mode === \"half-hourly\") {\r\n for (let index = 0; index < gas.length; index++) {\r\n\r\n msgData.push({\r\n measurement: \"meters\",\r\n retentionPolicy: \"month\",\r\n fields: {\r\n kWh: gas[index].consumption\r\n },\r\n tags: { type: \"gas\" },\r\n timestamp: new Date(gas[index].interval.end).getTime()\r\n })\r\n\r\n msgData.push({\r\n measurement: \"meters\",\r\n retentionPolicy: \"month\",\r\n fields: {\r\n kWh: elec[index].consumption\r\n },\r\n tags: { type: \"electric\" },\r\n timestamp: new Date(elec[index].interval.end).getTime()\r\n })\r\n }\r\n\r\n msg.retentionPolicy = \"month\"\r\n }\r\n\r\n msg.payload = msgData;\r\n return msg;\r\n}\r\n\r\n// Execute function based on msg.topic\r\nif (msg.topic === \"monthly\") {\r\n return processData(\"monthly\");\r\n} else if (msg.topic === \"daily\") {\r\n return processData(\"daily\");\r\n} else if (msg.topic === \"half-hourly\") {\r\n return processData(\"half-hourly\");\r\n} else {\r\n node.warn(\"Unknown topic: \" + msg.topic);\r\n node.status({ fill: \"red\", shape: \"dot\", text: \"Unknown topic: \" + msg.topic });\r\n return;\r\n}\r\n\r\n"},{"id":"f9e7bba66a8679d4","type":"inject","z":"712086ff.3e3668","name":"08:00 half-hourly","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"00 08 * * *","once":false,"onceDelay":0.1,"topic":"half-hourly","payload":"","payloadType":"date","x":1095,"y":1380,"wires":[["e377d8cd85113afb"]]},{"id":"a32901e8082ed7f3","type":"inject","z":"712086ff.3e3668","name":"08:01 daily","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"01 08 * * *","once":false,"onceDelay":0.1,"topic":"daily","payload":"","payloadType":"date","x":1075,"y":1425,"wires":[["e377d8cd85113afb"]]},{"id":"fddc7b6a3a060484","type":"debug","z":"712086ff.3e3668","name":"database","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":2145,"y":1395,"wires":[]},{"id":"f7b10d0530b5aa34","type":"function","z":"712086ff.3e3668","name":"set date","func":"// Check for valid topic\nif (![\"monthly\", \"daily\", \"half-hourly\"].includes(msg.topic) || !msg.topic) {\n node.error(\"Unknown topic: \" + msg.topic);\n node.status({ fill: \"red\", shape: \"dot\", text: \"Unknown topic: \" + msg.topic });\n return;\n}\n\nlet date = new Date();\ndate.setDate(date.getDate() - 1); // Set date to yesterday\n\nswitch (msg.topic) {\n\n case \"daily\":\n msg.date = date.toISOString().slice(0, 7); // eg 2024-02\n msg.targetDate = date.toISOString().split('T')[0]; // eg 2024-02-29\n break;\n\n case \"monthly\":\n msg.date = date.getFullYear(); // eg 2024\n msg.targetDate = date.getMonth()\n break;\n\n case \"half-hourly\":\n msg.date = date.toISOString().split('T')[0]; // eg 2024-02-29\n msg.targetDate = msg.date\n break;\n\n}\n\nmsg.cookies = { \"refresh_token\": msg.responseCookies?.refresh_token };\n\nnode.status({ fill: \"green\", shape: \"dot\", text: `${msg.topic} ${msg.date}` });\n\nreturn msg;\n","outputs":1,"timeout":"","noerr":0,"initialize":"","finalize":"","libs":[],"x":1650,"y":1425,"wires":[["28a3c26921bd6172"]],"info":"let date = new Date()\r\n\r\ndate.setDate(date.getDate() - 1); //yesterday\r\n\r\nvar year = date.toLocaleString(\"default\", { year: \"numeric\" });\r\nvar month = date.toLocaleString(\"default\", { month: \"2-digit\" });\r\nvar day = date.toLocaleString(\"default\", { day: \"2-digit\" });\r\n\r\nmsg.date = `${year}-${month}-${day}`;\r\n\r\nmsg.cookies = { \"refresh_token\": msg.responseCookies.refresh_token }\r\n\r\nreturn msg;"},{"id":"503a23925ef301f4","type":"http request","z":"712086ff.3e3668","name":"get token","method":"POST","ret":"obj","paytoqs":"ignore","url":"https://my.ovoenergy.com/api/v2/auth/login","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"x":1515,"y":1425,"wires":[["f7b10d0530b5aa34"]]},{"id":"e377d8cd85113afb","type":"change","z":"712086ff.3e3668","name":"get login","rules":[{"t":"set","p":"payload","pt":"msg","to":"login","tot":"flow"},{"t":"set","p":"account","pt":"msg","to":"account","tot":"flow"}],"action":"","property":"","from":"","to":"","reg":false,"x":1380,"y":1425,"wires":[["503a23925ef301f4"]]},{"id":"28a3c26921bd6172","type":"http request","z":"712086ff.3e3668","name":"get data ","method":"GET","ret":"obj","paytoqs":"ignore","url":"https://smartpaymapi.ovoenergy.com/usage/api/{{topic}}/{{account}}?date={{date}}","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"x":1785,"y":1425,"wires":[["5fa27756ada8a4f4"]]},{"id":"bcc293757e5627df","type":"trigger","z":"712086ff.3e3668","name":"request again","op1":"","op2":"0","op1type":"nul","op2type":"str","duration":"1","extend":false,"overrideDelay":false,"units":"hr","reset":"","bytopic":"all","topic":"topic","outputs":1,"x":2170,"y":1425,"wires":[["e377d8cd85113afb"]]},{"id":"41724b812e4d13ca","type":"function","z":"712086ff.3e3668","name":"1st of Month","func":"if (msg.payload && typeof msg.payload === 'number') {\n let date = new Date(msg.payload);\n if (date.getDate() === 1) { return msg };\n}\n","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1250,"y":1470,"wires":[["e377d8cd85113afb"]]},{"id":"9e9b76ac22fb9c94","type":"inject","z":"712086ff.3e3668","name":"08:03 monthly","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"03 08 * * *","once":false,"onceDelay":0.1,"topic":"monthly","payload":"","payloadType":"date","x":1085,"y":1470,"wires":[["41724b812e4d13ca"]]}]
I tried the test function below, and as you say, the node uses UTC. So this needs considering in the code. Or perhaps not test things between midnight and 1am and get some sleep instead!
msg.date1 = new Date().toLocaleString();
msg.date2 = new Date();
return msg;
{"_msgid":"85bb45ec1a5917e9","payload":1714905469821,"topic":"",
"date1":"5/5/2024, 11:37:49 AM","
date2":"2024-05-05T10:37:49.823Z"}
Thanks for the new code, Iāll have a proper look later.
The date issue I had was the UTC date being used by the node (and testing between 12 & 1am!)
I did find the test for null data in the half hourly data function failed when I ran it for the current day and there were no values. (both gas & electricity were null). This shouldnāt be an issue if run at the correct time, and ovo have updated the data,
Iām not sure if the data from the api and the ovo website are in step, but quite often the usage data on the web site for the previous day is not available at 8am (I check every day!). I also noticed last night (after 1am) that the half hour usage for the previous day had one value for gas usage for the first half hour. It maybe a good idea to check all data has been populated before reporting data available?
Yes, which is why you should always store and process in UTC. Date/time utilities were one of the first things I worked on back when I was a humble business systems programmer early in my career, I learned this lesson the hard way too.
Haha, that too!
I have added some more checks for missing data.
I found that the time the data becomes available varies quite a lot, I settled on 8am as that seemed to work most of the time, however the trigger node will re-request an hour later if the data is not there yet 
Iāve finally found some time to have a look at the new code!
The half hourly data seems to work, 96 data points returned. However, the daily data is failing with missing daily data for 2025-05-07. Checking the OVO portal, the data appears to be available.
Just had a play with the codeā¦ Iām no expert in thisā¦ But is the problem that the code uses the current system date (line 20) you subtract 1 as the array is indexed from 0. but the api call returns data upto the previous day. I changed this to line to minus 2, and it seems to work.
Rather than using the system date, should the date in the msg as set in the āset dateā node be used?
It looks like I missed out a line in the version I posted.
After line 14 insert this - date.setDate(date.getDate() - 1); //yesterday
I haven't really had much chance to test the code, since it's usually after midnight when I get some time and then there IS no data
Thanks, Iāve added that.
Same problem as you, I cant test it until tomorrow now!
Well in about 7 hours anyway
The bad weather finally gave me time to install and setup influxdb & grafana, and Iāve been testing over the last few days!
There seems to be a problem with the retry code. While the retry for the half hourly works, the daily data doesnāt seem to retry. Iāve added some debug nodes, and found there is no retry message being sent for daily, there is one for half hourly.
I did wonder if the check for missing data was correct, as Iāve noticed the gas data seems to be available for electricity (on the web anyway) but dont think thatās the case as no gas data was sent to influxdb.
A few thoughtsā¦
As an interesting aside, while testing, Iāve noticed the kwh figures from the api are slightly different to the values on the website.
Yes it should be each and msg.topic, not sure how that got unticked 
Not entirely clear What do you see on the debug from the top output of process data ?
PS reply to this message rather than the topic, so I get a notification, otherwise I may not see it.
I changed it to each message, and I think itās working, but not helped by ovo data issues!
It now shows a 2 on the retry node, I suspect saying there are 2 messages waiting to be retried, so that looks good. Also today, the retry for the daily stats retried and worked successfully, which has not happened before.
However, there seem to be some data issues with ovo
Iām thinking about options to load data between specified dates to get more historical data into influxdb. Then I need to learn flus and grafana, Iād like to compare data from different dates (week/month/year) and merge in data from other sensors. Iāve already included data from a number of temp sensors, and smart plugs.
I'm not actually using this flow at the moment, as I just posted it to get you started.
The retry was just a quick fix for late arriving data, but it will not work if the date rolls over to the next day. This is something you could look at improving, perhaps keep track of missing data and request specific days until you get it.
It worked perfectly today. Retrying until about midday for the daily and 3pm for the half hourly.
I also made a temporary change to the code to get the missing data from 3 days ago. I might change the check for 48 data items to just 1, but will see how it goes for a while first.
Many thanks for the code, Itās an impressive piece of software. Iāve learnt quite a lot from it, particularly the data handling.
Now its time to learn flux/grafana!
Copyright OpenJS Foundation and Node-RED contributors. All rights reserved. The OpenJS Foundation has registered trademarks and uses trademarks. For a list of trademarks of the OpenJS Foundation, please see our Trademark Policy and Trademark List. Trademarks and logos not indicated on the list of OpenJS Foundation trademarks are trademarksā¢ or registeredĀ® trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.
The OpenJS Foundation | Terms of Use | Privacy Policy | OpenJS Foundation Bylaws | Trademark Policy | Trademark List | Cookie Policy