Best way to collect solar power forecast charts in influxDB (fields or tags or else)

[hominidae]

Hi folks,

at home, I have a freshly installed solar power plant and now wanted to do some information gathering and analysis with its data.
As I can already collect various energy based information in my home to mqtt and influx, including the current solar power, I wanted to compare these to the solar power forecast for my site.
I found a nice service at solcast (https://toolkit.solcast.com.au/) where you can define your power plant (as a hobbyist for free in a so called rooftop-site) and it comes with a well documented API (https://docs.solcast.com.au/#rooftop-sites).

I am already able to collect my own measurements and feed them back to my rooftop-site at solcast (for tuning their estimates) , as well as do the queries to their API and collect forecast information.

My challenge is, that I am debating on how to structure the data to be stored in influxDB.
For each query, you will receive a chart (like the next seven days, in 30min fixed intervals, as JSON response), containing an array of objects, basically consisting of timestamp and power estimates.
The estimates are based on the time of the query, so for each query I will get a fresh set of estimated power values. But the timestamps are on a fixed interval and will stay the same for each iteration/new query of the chart (besides the fact that for a new query, new timestamp values will appear as time passes).
This is what it looks like in their own UI:

What I want to achieve is that the charts will remain stored in influxdb and data at the same timestamp will nor be overwritten.
The idea is that I would be able to collect or compare individual charts later.
My current thinking is around whether to use a field or a tag (or both) for storing a "chart-ID" (in my mind simply the number/counter of the query from each invocation of the API throughout the day - the rooftop site will allow for up to 20 queries per day with the hobbyist license/account).

This is my first attempt and I am curious what you people think I am doing wrong or what I should do in order to improve (shows a seven day forecast array/chart with 30min intervals from now() ):

Many thanks in advance for your input!

regards,
Fred

[ozdw]

Fred, I'm also using Solcast and was about to create some flows to provide tuning data. Would you be able to share your tuning flow here?

thanks
dave

[hominidae]

Hi Dave,

I basically have two flows, one for feeding back measurements and the second for collecting forecasts.

I was about to create a new thread under the "share your projects" category here, as this thread is not intended to carry this information. I hopefully can manage to do that today.

Edit: see my example here: Power Forecasts and Data-Tuning for Solar Installations (Node-Red / solcast-API / InfluxDB / Grafana)

It takes about 10 days, after starting to feed measurements back to solcast until pv-tuning kicks in, but it definitely works well.

regards,
Fred

[ozdw]

Fred, thanks for sharing this. Very helpful. I have the flow working now but am struggling with passing the API key to http-request via msg.headers. I see your example you put the key in the username field in the http-request node properties dialog. I would like to avoid hard-coding parameters like that in nodes so am trying to use a config node and then place the auth info in the headers before the http-request node.

Have you by chance tried that?

dave

[hominidae]

Hi Dave,

well, you could construct the URL in the msg.payload or msg.headers earlier in the flow, instead of applying it fixed in the config of the HTTP-node.
But you will need two parameters, the rooftop-ID and the API-Key in the URL.
You can try this node and construct/store the payload in there https://flows.nodered.org/node/node-red-contrib-config
...or use a change-node with same JS/JSONATA to construct the message payload or headers.

I consider both parameters as stable/constant, so I don't honestly see the benefit putting these in a separate config node.
You will have to store them fixed in any case somewhere in one place only (so maybe put these in an external security vault or database, but a config node is not more secure than any other node).
It is more of an architectural decision.