Set array to global once a day, then get specific array by the hour

P-solver · 22 October 2023 19:27

I'm trying to set an global array that is fetched once a day, and then I want to fetch the global array in another flow and print a specific object from the global array.

This is my code right now, but I get the error "SyntaxError: Unexpected token N in JSON at position 0"

Code to set the global array:

global.set('tomorrowsprice', JSON.stringify(msg.payload));

Code to get the global array (run by timestamp each hour):

// get global array with tomorrows price
global_tomorrowsprice[] = JSON.parse(global.get('tomorrowsprice'));

// get next hours price from index "price"
const date = new Date();
date.setHours(date.getHours()+3); // uncomment for next hour
const index = date.getHours();
msg.payload = global_tomorrowsprice[index].price

return msg;

TotallyInformation · 22 October 2023 19:35

JSON.parse and JSON.stringify are never safe to use without wrapping in a try/catch block.

Also, you shouldn't need to parse the input if it is an in-memory global.

And, you haven't written back to the store.

// get global array with tomorrows price
global_tomorrowsprice = global.get('tomorrowsprice') || [];  
// Note the "or" to cope with the case where the global does not exist

// get next hours price from index "price"
const date = new Date();
date.setHours(date.getHours()+3); // uncomment for next hour
const index = date.getHours();
msg.payload = global_tomorrowsprice[index].price

global.set('tomorrowsprice', global_tomorrowsprice)

return msg;

P-solver · 22 October 2023 20:16

Ok, thanks for quick answer. (Although I added a couple of ";" on the lines at the end, if anyone needs to use this code)

I now get error "TypeError: Cannot read properties of undefined (reading 'price')" when running the hourly node.

The array looks like this from the first code (executed once a day):

array[24]
[0 … 9]
0: object
timestamp: "2023-10-21T22:00:00.000Z"
price: 122.94
currency: "SEK"
area: "SE2"
1: object
2: object
3: object
4: object
5: object
6: object
7: object
8: object
9: object
[10 … 19]
[20 … 23]

Any ideá of what I've missed out?

E1cid · 22 October 2023 20:17

That is a syntax error also. i.e [ ]

You used my last code example from Getting next hours electric spot-price from Nordpool via nordpool-api-plus - #5 by E1cid
But you didn't even acknowledge me in the thread.

P-solver · 22 October 2023 20:23

Sorry, wasn't my intention to not acknowledge you, but I thought I had the solution from the first answer.

TotallyInformation · 22 October 2023 20:28

These are mostly optional in JavaScript and I use "Javascript Standard" layout which specifies that they should not be used unless necessary.

This tells you that whichever hour is being processed does not contain a price property.

Wrap that line in a try/catch and in the catch, put

node.warn(`Index: ${index}`)

Or something like that so that you can see which hour has the problem.

E1cid · 22 October 2023 20:29

What is the value of index as it looks like you are getting a undefined from
msg.payload = global_tomorrowsprice[index].price
add node.warn({index}) prior to that line and show us the value.
Also what time zone is your machine in, what is the time there now.

P-solver · 22 October 2023 20:34

I get "1" if adding the warn-line.

Time now is 22:34

Objects of "1" in the array:

array[24]
[0 … 9]
0: object
1: object
timestamp: "2023-10-21T23:00:00.000Z"
price: 122.36
currency: "SEK"
area: "SE2"
2: object
3: object
4: object
5: object
6: object
7: object
8: object
9: object
[10 … 19]
[20 … 23]

E1cid · 22 October 2023 20:39

Can you copy your global context array value

There’s a great page in the docs (Working with messages : Node-RED) that will explain how to use the debug panel to find the right path/value to any data item.

Pay particular attention to the part about the buttons that appear under your mouse pointer when you over hover a debug message property in the sidebar.

BX00Cy7yHi

You will find a copy value in the context sidebar also.

[edit]
What do you want when the time is the next day, as 10.35 pm + 3 hours is 1.35am the next day?

P-solver · 22 October 2023 20:49

Is this what you want to see?

About the "+3";

Each hour I want to fetch the upcoming (next) hours price.
So if the clock is 22.00, I want to fetch 23:00's price.

[{"timestamp":"2023-10-22T22:00:00.000Z","price":345.22,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-22T23:00:00.000Z","price":286.95,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T00:00:00.000Z","price":284.39,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T01:00:00.000Z","price":282.53,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T02:00:00.000Z","price":325.91,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T03:00:00.000Z","price":549.82,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T04:00:00.000Z","price":354.41,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T05:00:00.000Z","price":399.19,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T06:00:00.000Z","price":439.55,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T07:00:00.000Z","price":429.43,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T08:00:00.000Z","price":416.76,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T09:00:00.000Z","price":553.66,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T10:00:00.000Z","price":556.22,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T11:00:00.000Z","price":549.94,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T12:00:00.000Z","price":551.8,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T13:00:00.000Z","price":556.22,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T14:00:00.000Z","price":450.26,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T15:00:00.000Z","price":456.3,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T16:00:00.000Z","price":450.37,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T17:00:00.000Z","price":451.19,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T18:00:00.000Z","price":444.67,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T19:00:00.000Z","price":421.18,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T20:00:00.000Z","price":371.51,"currency":"SEK","area":"SE2"},{"timestamp":"2023-10-23T21:00:00.000Z","price":279.39,"currency":"SEK","area":"SE2"}]

E1cid · 22 October 2023 20:55

Yes

Running this code

// get global array with tomorrows price
let global_tomorrowsprice = global.get('tomorrowsprice') || [];
node.warn(global_tomorrowsprice)
// get next hours price from index "price"
const date = new Date();
date.setHours(date.getHours()+3); // uncomment for next hour
const index = date.getHours();
msg.payload = global_tomorrowsprice[index].price

return msg;

with your data in global context returns no error for me.
Are you sure your global is set and not been deleted or changed?

P-solver · 22 October 2023 21:00

I've set the global_tomorrowprice with this code, but it might be wrong?

Is there a better way?

global.set('tomorrowsprice', JSON.stringify(msg.payload));

E1cid · 22 October 2023 21:05

@TotallyInformation said to not use JSON.stringify(), there is no reson to create a JSON. Just save the array as is.
use

global.set('tomorrowsprice', msg.payload);

Then try again with the code i tested with.

[edit]
Does your array of price objects always start a 10pm the day before and finish at 9pm the following day?
What time do you fetch the array daily?

P-solver · 23 October 2023 17:26

Got the global to work now, the strange thing is that I copied the nodes to do some testing, just passing by a string vith the value 'test' from one flow to the other, and it worked just fine.

Changed the 'test' value to the 'payload' value and suddently it was working, I now get the array from the line:

let global_tomorrowsprice = global.get('tomorrowsprice') || [];

To answer your questions:

Yes, the array always starts at 10pm and ends at 9pm next day.
This gets me to question the "+3" value I'm adding to fetch the correct array (e.g. next hours price)

So now I run the daily fetch of the array at 8.10pm thought about only adding "+1" to the index-value, but I guess this will not be correct when passing midnight?

Simplest should be to fetch the "active" arrays number, due to date + hour, and add "+1" to this I guess?

I appriciate all the help, but I can't really think clear now regarding the "index" issue.

E1cid · 23 October 2023 18:12

I would sort the array so the times are in order.
Then just add 1 hour.
At 8.15 the data would update. so you get the right times.
e.g.

[{"id":"2891406fda7184f6","type":"inject","z":"2493d0f1b4d72d1b","name":"fetch data at 8:15","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[{\"timestamp\":\"2023-10-22T22:00:00.000Z\",\"price\":345.22,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-22T23:00:00.000Z\",\"price\":286.95,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T00:00:00.000Z\",\"price\":284.39,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T01:00:00.000Z\",\"price\":282.53,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T02:00:00.000Z\",\"price\":325.91,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T03:00:00.000Z\",\"price\":549.82,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T04:00:00.000Z\",\"price\":354.41,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T05:00:00.000Z\",\"price\":399.19,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T06:00:00.000Z\",\"price\":439.55,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T07:00:00.000Z\",\"price\":429.43,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T08:00:00.000Z\",\"price\":416.76,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T09:00:00.000Z\",\"price\":553.66,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T10:00:00.000Z\",\"price\":556.22,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T11:00:00.000Z\",\"price\":549.94,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T12:00:00.000Z\",\"price\":551.8,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T13:00:00.000Z\",\"price\":556.22,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T14:00:00.000Z\",\"price\":450.26,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T15:00:00.000Z\",\"price\":456.3,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T16:00:00.000Z\",\"price\":450.37,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T17:00:00.000Z\",\"price\":451.19,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T18:00:00.000Z\",\"price\":444.67,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T19:00:00.000Z\",\"price\":421.18,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T20:00:00.000Z\",\"price\":371.51,\"currency\":\"SEK\",\"area\":\"SE2\"},{\"timestamp\":\"2023-10-23T21:00:00.000Z\",\"price\":279.39,\"currency\":\"SEK\",\"area\":\"SE2\"}]","payloadType":"json","x":460,"y":1020,"wires":[["20ab870bf9f6f6c5"]]},{"id":"20ab870bf9f6f6c5","type":"change","z":"2493d0f1b4d72d1b","name":"","rules":[{"t":"set","p":"tomorrowsprice","pt":"global","to":"$append($$.payload[[2..23]],$$.payload[[0,1]])","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":670,"y":1020,"wires":[["15eb7370f45911df"]]},{"id":"15eb7370f45911df","type":"debug","z":"2493d0f1b4d72d1b","name":"debug 346","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":850,"y":980,"wires":[]},{"id":"3315988dc15a1aec","type":"inject","z":"2493d0f1b4d72d1b","name":"get next hour","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":430,"y":1080,"wires":[["6e8ec59c9be2b188"]]}]

P-solver · 24 October 2023 04:54

I got everything to work ok, BUT then I found out that the node (Nordpool API plus) I was using to get the prices for tomorrow is not getting the accurate prices.

So now I have another node (Nordpool API) to fetch todays prices in the same type of array, but always from "00:00" to "00:00" (24h):

[{"Area":"SE2","Timestamp":"2023-10-24 00:00","StartTime":"2023-10-24 00:00","EndTime":"2023-10-24 01:00","Price":302.9,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 01:00","StartTime":"2023-10-24 01:00","EndTime":"2023-10-24 02:00","Price":295.52,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 02:00","StartTime":"2023-10-24 02:00","EndTime":"2023-10-24 03:00","Price":288.49,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 03:00","StartTime":"2023-10-24 03:00","EndTime":"2023-10-24 04:00","Price":293.06,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 04:00","StartTime":"2023-10-24 04:00","EndTime":"2023-10-24 05:00","Price":324.56,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 05:00","StartTime":"2023-10-24 05:00","EndTime":"2023-10-24 06:00","Price":441.11,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 06:00","StartTime":"2023-10-24 06:00","EndTime":"2023-10-24 07:00","Price":713.67,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 07:00","StartTime":"2023-10-24 07:00","EndTime":"2023-10-24 08:00","Price":681.34,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 08:00","StartTime":"2023-10-24 08:00","EndTime":"2023-10-24 09:00","Price":759.35,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 09:00","StartTime":"2023-10-24 09:00","EndTime":"2023-10-24 10:00","Price":675.02,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 10:00","StartTime":"2023-10-24 10:00","EndTime":"2023-10-24 11:00","Price":726.55,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 11:00","StartTime":"2023-10-24 11:00","EndTime":"2023-10-24 12:00","Price":747.75,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 12:00","StartTime":"2023-10-24 12:00","EndTime":"2023-10-24 13:00","Price":717.65,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 13:00","StartTime":"2023-10-24 13:00","EndTime":"2023-10-24 14:00","Price":706.41,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 14:00","StartTime":"2023-10-24 14:00","EndTime":"2023-10-24 15:00","Price":712.61,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 15:00","StartTime":"2023-10-24 15:00","EndTime":"2023-10-24 16:00","Price":705.7,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 16:00","StartTime":"2023-10-24 16:00","EndTime":"2023-10-24 17:00","Price":688.02,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 17:00","StartTime":"2023-10-24 17:00","EndTime":"2023-10-24 18:00","Price":656.04,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 18:00","StartTime":"2023-10-24 18:00","EndTime":"2023-10-24 19:00","Price":690.83,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 19:00","StartTime":"2023-10-24 19:00","EndTime":"2023-10-24 20:00","Price":623.71,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 20:00","StartTime":"2023-10-24 20:00","EndTime":"2023-10-24 21:00","Price":358.42,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 21:00","StartTime":"2023-10-24 21:00","EndTime":"2023-10-24 22:00","Price":288.37,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 22:00","StartTime":"2023-10-24 22:00","EndTime":"2023-10-24 23:00","Price":262.02,"Valuta":"SEK/MWh"},{"Area":"SE2","Timestamp":"2023-10-24 23:00","StartTime":"2023-10-24 23:00","EndTime":"2023-10-25 00:00","Price":241.4,"Valuta":"SEK/MWh"}]

So now my question is:

If I run this at 00:15 to get the "actual days prices (+next days price at 00:00)" into the array above, how is the easiest way to set the correct "index" at the time(s) the time is "00:00", as index willl be showing "1" ("00:00" + 1 hour) when the time is "00:00"?

hotNipi · 24 October 2023 05:32

Nordpool publishes day-ahead prices 12:45 CET or a bit later.
To have complete array for full day you'll need to query day-ahead (for tomorrow) data every day at some point after 12:45 CET. Then combine new data with old so you have 48 hours of data. (you can hold more historic data if you need to). That array then contains full set of today's prices (not for first ever day when you develop your flows of course, but you can also query data from past if you like to make array complete immediately)

I keep the data for 3 days so I can look back for yesterday if I need to. Data for tomorrow is blank until midday query. (Actually not blank but has 1 item)

Also when receiving new data, I reformat it in the way that timestamps are actually timestamps not the dates. It is way easier to deal with timestamps in code than with Date's in whatever format

Here's how I manage data storage.

let nordpool = global.get('nordpool') || []
//node.warn(msg.payload.length)
if(msg.payload && msg.payload.lenght != 0){
    msg.payload.forEach(function(element) {
        //get rid of duplicates if any
        let duplicate = nordpool.find(el => el.x == new Date(element.timestamp).getTime())
        if(duplicate == undefined){
            // x is timestamp
            // y is price
            // easy to use with charts.
            nordpool.push({
                x:new Date(element.timestamp).getTime(),
                y:element.price
            })
        }        
    });
}
//keep data in order time wise
nordpool.sort((a, b) => {
    return a.x - b.x;
});

//filter out too old data
let old = new Date()
old.setHours(0,0,0,0)
old.setDate(old.getDate()-3)
nordpool = nordpool.filter(el => el.x > old.getTime())

//store prices
global.set('nordpool',nordpool)
return msg;

P-solver · 24 October 2023 06:25

@hotNipi this looks interresting and useful.
Unfortunately, in the node I use now, that shows the correct prices, I can't get tomorrows price.

How are you getting the prices from Nordpool?

hotNipi · 24 October 2023 06:29

I use this nordpool-api node (i must to use version 4.0 cos current version does not have 'EE' area)

And to get day-ahead:

const today = new Date()
const tomorrow = new Date(today)
tomorrow.setDate(tomorrow.getDate() + 1)
msg.date = tomorrow.toISOString()
msg.payload = today.getTime()
return msg;

P-solver · 24 October 2023 06:35

So this code goes into a "function node" and then through the "nordpool-api" node?

Topic		Replies	Views
Global numeric array trouble using global get and set General	51	8752	24 July 2018
Problem with array persistence with global.get / global.set General	10	738	8 November 2020
Getting next hours electric spot-price from Nordpool via nordpool-api-plus General	7	252	2 November 2023
Nordpool data on graph General	6	549	1 May 2022
Change Node - access Array Object General	4	431	12 August 2021

Set array to global once a day, then get specific array by the hour

Related Topics