Power Forecasts and Data-Tuning for Solar Installations (Node-Red / solcast-API / InfluxDB / Grafana)

Hi folks,

at home, I have a freshly installed solar power plant and now wanted to do some information gathering and analysis with its data.

As I can already collect various energy based information in my home to mqtt and influx, including the current solar power, I wanted to compare these to the solar power forecast for my site.

Part I - Introduction

I found a nice service at solcast (https://solcast.com/rooftop-solar/) where you can define your solar power plant (as a hobbyist for free in a so called rooftop-site) and it comes with a well documented API (https://docs.solcast.com.au/#rooftop-sites).

There is also a so called "PV-Tuning" feature, which will allow you to feed back real power measurements from your installation to solcast and these will be used to adjust/tune the forecasts specifically for your installation.
This does include compensation of azimuth/angle of your setup, in case your real installation is different from a single rooftop with a fixed azimuth and angle, which is the only way to define your setup in the solcast rooftop account.

For example, my site has actually two orientations, where the first batch of the solar modules is facing East and the second is facing West, with a different number of modules in East and West orientation.
So I defined a single rooftop, facing South and tried their algorithm.

This is the solcast UI, before tuning:

...and after approx. 10 days of feeding measurements to solcast, it looks like this:

The forecasts have become incredibly accurate, as you can see from my chart yesterday:

Part II - using the solcast API with Node-Red (flows and how its done)

...see next post

1 Like

Part II - using the solcast API with Node-Red (flows and how its done)

....using the solcast API is faily uncomplicated.

I've created two flows, one for posting measurements and one for collecting forecast data.

As data source for measurements, as well as data destination for solcast estimates, I've used an influxDB, which I already populate with data from my infrastructure.
If you want to use other means to manage your data, your mileage may vary and you will have to adopt to your specific setup, of course.

flow A) sending measurements from influxDB to your solcast rooftop acount

Originally the data for measurements is coming from my solar inverter.
As I have a quite sophisticated EV-Charging station that is capable of charging the vehicle optimized to the current flow of solar power available, the data is already collected there, including some more information.
Therefore I refrained from directly accessing my inverter (via modbus-tcp) but integrated with my EV charging station to collect the data and transfer this to influxDB (via mqqt-telegraf-influxDB toolchain).

The destination is described in the solcast API, here: https://docs.solcast.com.au/#measurements-rooftop-site
This is a simple HTTP(S) request to the API, sending the data as an array in JSON format.
You can invoke the API-service as often as you want (it is not counting against the number of 20 allowed API calls per day with your free hobbyist rooftop account). Also re-sending measurements already sent earlier is not a problems, as these will simply be overridden.
I finally resorted to collect and send data for from the last 25hours in an interval once every 24hours.
The flow offers some predefined injection-nodes with dataseries from other time-spans, that can be invoked manually.

My flow for posting measurements to solcast looks like this:


The different influxDB queries are defined manually inside the change-nodes.
The minimum intervall that can be given for a measurement is 5minutes, which is what I choose in the queries.

This is the influxDb query for the 25hrs recurring collection interval:

SELECT mean("value") / -1000 FROM "autogen"."mqtt_consumer" WHERE ("topic" = 'openWB/pv/W') AND (time >= now()-25h AND time <= now()-7m) GROUP BY time(5m) fill(0)

As my data is stored in W, but measurements are expected in kW units, notice the "* 1000" math operation inside the query. Also non existing data in that interval (when its dark, normally no solar data will be captured from the inverter) will be filled with "0" (number zero).
My data/influx measurements is stored via telegraf from mqtt topics, so in my example the measurement from influxDB is "openWB/pv/W". You will need to select the correct measurement in your own query.

I noticed, that the solcast API-Service can be sometimes "flaky", so an API invocation might fail occasionally . I catched the HTTP-response and if not a status 200, the message gets recreated after 5mins....works every time so far.

Inside the HTTP-Request node, you will have to specify the URL and credentials.
The URL is based on/comprised of your rooftop account ID and you can use basic identification with the use of your solcast API-Token as Username. Both information you can easily get from your solcast account info.

This is what the setup of the node looks like:

...and here's the flow:

solcast-measurements-flow.json (7.1 KB)

flow B) retrieving solcast estimates/forecasts and storing in influxDB

The flow for retrieving the forecasts is similar.
This is what the flow looks like:

I am using a special inject node from here: [ANNOUNCE] Ultimate (Swiss-Army-Knife) Node-Red Timer based flow control where I can run a number of events between dates (sunrise/sundown).
The hobbyist rooftop account will allow for 20 queries per day, so a counter-node is introduced to match/route this requirement, while events are created (using https://flows.nodered.org/node/node-red-contrib-counter).

Also other inject nodes are there to allow for manual injection and/or resetting the counter, which is nice for testing (during the first three month of your rooftop account, the limit will not be enforced until he first 1000 API-calls are used up).
Again, like in the measurement flow, of the API-call fails occasionally, a retry will be scheduled after 3mins. I found, that even when the call fails, the attempt will be counted against the number of calls, so the retry will increase the counter...just in case.

The API has two separate services, estimated actuals and forecasts.
The HTTP(S) node is configured similar, but with a GET Method and waiting/parsing a JSON response. using rooftop-id in the URL and API-Token as Username with basic authentication.

After a successful retrieval, the data (JSON array) gets re-formatted to be fed into the influxDB-node.
Here I opted to send each individual data point/message into the node. My current influxDB and node-Red instance would easily cope with the "stress", but generating a single bulk-message for influxDB is certainly possible. A single API-Call will create 336 data points.

Here's the flow:

solcast-estimates-flow.json (23.9 KB)

..and here are some pics from my grafana dashboards.

PV-Tuning kicks in only after some days (my site took 10 days) of feeding measurements, but the results are pretty impressive, I think.

...forecasts this week:

...values from last week:

...hope you enjoy it...feel free to ask questions or details I missed.

regards,
Fred

1 Like

Fred

I was reinventing the wheel I think
As far as I understood your setup: you're reading out your inverters with another system and maybe also the in and out of the grid (use of or injection into) and put these information in openWB. With openWB you're commanding your wallbox and can choose to charge only with the excess of solar power.
The same setup for me: a PV installation, a go-echarger, for a long time vzlogger to read out my digital meter (in/out of the grid) that runs on a rpi0W. Lately a second rpi0W (to test first) with openWB and it workes fine. No readout at this moment of the SMA bluetooth inverters (2).
Previously the rpi0W to readout the meter was an old notebook with BT and SBFspot was running, Now I've to checkout why the raspbian doesn't want to find the BT hardware.
But instead of doing that I was looking for more flexibility of what I'd like to do with the follwing upcoming rules in mind (2021 or 2022)
The electricity distribution compagny want to change the billing and they will split the bill in 2 parts, one fix part and one part depending on the consumption.
And it's that fix part that bothers me. They want to bill it depending the max amount of grid use in a quarter of an hour. The max of each month will be used as an yearly average to calculate that fee with a minimum of 2,5kWh.
So with what I've now vzlogger and openWB I can't do a lot to reduce that bill especially in winter.
I should need a system where I can forecast the amount of sun a day and the next day will produce (solcast and your flows). Depending that forecast I've to use only sunpower or I can use sun and grid power but I want to limit the grid power (in) probabely 2,5kW, the minimum fee for the fix bill component.
Therefore I was looking to put everything in node-red, to be more flexible.
I already saw, not tested, a node for the digital meter, saw that SBFspot has the ability to send its info to MQTT.
Why are you still using openWB and did not put everything in node-red yet?

A small home battery will be needed in the future to help out the mentionned limit, but commandable battery inverters are rare at this moment.

BR

Hi,
that is an interesting setup and UseCase.

Yes, I am also gathering the Power flow at grid level with an MPM3PM meter (openWB EVU-Kit).
The reason I decided not to put everything in node-red is at the end risk based.

In order for EV charging to be based on available access power at grid level, openWB needs the data in any case.
However I am uncertain how many simultaneous connections the modbus devices would allow (and at what reading frequency - ie. SMA is very conservative in their docs)
So using both (node-red or openWB) simultaneously over modbus has a risk of overlaoding the inverter/grid-meter modbus readouts.

I have a test flow, based on integration via mqtt from openWB and with modbus-tcp for connecting my SMA inverter and grid meter directly and show these in parallel on the dashboard.
The results are showing that openWB is doing a good job, although only reading out the data in 10sec intervals (I can share it, if you want).
If using modbus with node-red alone, then feeding this back to openWB via mqtt is certainly possible, but I decided to not do that until I need better resolution, over the 10sec interval openWB provided.

For integration with node-red, I would actually prefer using mqtt over anything else. This is the safest option, IMHO.
Therefore another integration option with node-red for you could be mbmd from volkszaehler.og.
Its a modbus deamon that can read many meter and inverter types, then pushing to mqtt (for influxdb, also telegraf has modbus plugin now). Also openhab and iobroker, FHEM have many integration option for meters and offer mqtt-support.
As for your SMA inverter running bluetooth only, I have no clue. Maybe upgrade with the webconnect module?

The challenge with solcast in your case, I see is pv-tuning.
You will need the PV measurements from your setup in order to do that and get a good result.

Edit: BTW, this is how my dashbord looks like while charging from PV: