Storing energy data

https://pastebin.com/LyVvR1qH

Something like this?

Please note that you don't have to use pastebin for posting flows, in fact it is easier and preferred if you post them here by pasting them in between 2 blocks of 3 backticks (so 3 backticks on the top line, then the pasted flow on the middle, followed by another 3 backticks on the third line.
```
code or flow here
```
This character, also known as the grave accent can be found on QWERTY and QWERTZ keyboard on the same key as the tilde (~), on most of those keyboards it is located left of the row of numbers.

Please see this topic for more information on how to post these:

I see. Sorry.

[{"id":"def1cf3e.911b1","type":"function","z":"12458ddf.1454e2","name":"","func":"var erg = parseInt(msg.payload) / 60000;\nerg = erg.toFixed(2);\nmsg.payload = erg.toString();\nreturn msg;","outputs":1,"noerr":0,"x":670,"y":260,"wires":[["9fc629b5.ba4668","36378296.b2343e"]]},{"id":"dc04ec31.f182b","type":"mqtt in","z":"12458ddf.1454e2","name":"","topic":"shellies/shellyplug-s-040BAD/relay/0/energy","qos":"2","datatype":"auto","broker":"51c9edc2.fc7ea4","x":270,"y":260,"wires":[["def1cf3e.911b1","325258ba.93dc7"]]},{"id":"9fc629b5.ba4668","type":"debug","z":"12458ddf.1454e2","name":"Energie","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":860,"y":220,"wires":[]},{"id":"36378296.b2343e","type":"function","z":"12458ddf.1454e2","name":"","func":"var globaleVariable = global.get(\"variable\"||2);\nvar globaleVariable = msg.payload;\nglobal.set(\"shellyEnergyTest\",globaleVariable);\n\nreturn {payload: globaleVariable};","outputs":1,"noerr":0,"x":850,"y":260,"wires":[[]]},{"id":"325258ba.93dc7","type":"function","z":"12458ddf.1454e2","name":"","func":"var shellyEnergyTest = global.get(\"shellyEnergyTest\");\n\nif (msg.payload > shellyEnergyTest)\n{\n    shellyEnergyTest = shellyEnergyTest + msg.payload;\n}\n\n\n\n\n\nmsg.payload = shellyEnergyTest.toString();\n\nreturn msg;","outputs":1,"noerr":0,"x":670,"y":320,"wires":[["615c8159.a4a8"]]},{"id":"16125ce4.6a1063","type":"inject","z":"12458ddf.1454e2","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":380,"y":320,"wires":[["325258ba.93dc7"]]},{"id":"615c8159.a4a8","type":"debug","z":"12458ddf.1454e2","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":870,"y":400,"wires":[]},{"id":"51c9edc2.fc7ea4","type":"mqtt-broker","z":"","broker":"192.168.2.173","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","willTopic":"","willQos":"0","willPayload":""}]

Well at least you have made a start. A couple of points. When posting flows just post the nodes that we need to see, so mark then then use the export feature. For example you posted the MQTT in node and the associated config node. That is of no use to us and in fact means that when we delete it we have to manually go and delete the mqtt server config node associated with the mqtt in node. In addition you have publicly given away your local subnet address and also the address of your mqtt server. It is best not to post such things on public sites.
Secondly I suggest that you put all this in one function node, I can't see any point splitting the algorithm into mulltiple nodes, and I can't make much sense out of what some of them are doing.
I see you are converting numbers to strings for some reason. I advise against that. Most times it is more useful (and certainly more efficient) to keep them as numbers.
On a detail point give your variables meaningful name. You have called the running total shellyEnergyTest. Can I suggest it would be much better to call it something like energyRunningTotal or something similar that will make it much easier to understand. When you fetch it from global context you need to think about what will happen the first time, before it has been written the first time. So the line that fetches it could be something like
var energyRunningTotal = global.get("energyRunningTotal") || 0
The || 0 on the end (spoken as Or Zero) says that if the value is not yet defined then set it to 0.
So that is a good start, subject to a few tweaks, now look back at the suggested algorithm and carry on. It will be going in no time.

Well,

im to unexperied to code something like this. I think i have to live with that problem in future.

I cant solve it

hey,

in your settings.json, add the following:

contextStorage: {
  storeInFile: { module: "localfilesystem"}
},

This will make the global.context persistent.
Description of persistent storage can be found here.

Then the total amount of power will be stored between node-red's restarts.

Then you just have to modify your code to detect when Shelly is reset:
[EDIT]

let storedTotalAmount = global.get("totalAmount", "storeInFile") || 0;
const previousShellyValue = global.get("previousShellyValue", "storeInFile") || 0

// first run: init value in persistent storage
if (storedTotalAmount <= 0) global.set("totalAmount", 0, "storeInFile");
if (previousShellyValue <= 0) global.set("previousShellyValue", 0, "storeInFile");

if (msg.payload < previousShellyValue) {
  // here we detect a reset of Shelly system
  storedTotalAmount = storedTotalAmount + msg.payload;
}else{
  // note that if there's no change in Shelly's payload, the difference will be 0
  storedTotalAmount = storedTotalAmount + msg.payload - previousShellyValue;
}
global.set("totalAmount", storedTotalAmount, "storeInFile");
global.set("previousShellyValue", msg.payload, "storeInFile");
msg.payload = storedTotalAmount;

return msg;

Correct me if I'm wrong (and of course, I was, at first edit :pleading_face:)

Hope it helps :wink:

You have to check the new input value less than the previous input value and use that to indicate that the shelly has reset.
In addition the amount you have to add to the previous value is the difference between the new input value and previous one (unless the shelly has reset).

@jledun thank you for that nice example but:

  1. if i add
contextStorage: {
  storeInFile: { module: "localfilesystem"}
},

to the settings.json, at the bottom my node-red wont start anymore.

For the rest: if i put the code in a function-node, how should i return it to a debug window or mqtt-message?

Just put a "return msg.payload" at the end?

because im getting some errormessages then

You have to put the contextStorage specification inside the module.exports section of the settings file. If you look you can see that it is just a javascript object declaration so make sure you get the commas in the right places.

A function must always return a complete message so you should have return msg at the end.
I advise against converting the values to strings as in the flow posted, keep them as numbers so you can send them to MQTT for example.

[Edit] See these for help with writing functions and handling messages. It is worth looking through the rest of the docs too.
https://nodered.org/docs/user-guide/writing-functions
https://nodered.org/docs/user-guide/messages

Thank you. It works.

I tried to restart my Raspberry and Turned off and on my Shelly device. Now it says

"ReferenceError: storeTotalAmount is not defined (line 35, col 17)"

for the total amount. Before restarting my raspberry it worked and showed me the total amount.

Well i saw there is a "d" missing in the variables name.

Edit:

If i use it in this way:

let storedTotalAmount = global.get("totalAmount", "storeInFile") || 0;

let lastValueProvidedByShelly = global.get("lastValueProvidedByShelly", "storeInFile") || 0;



// first run: init value in persistent storage

if (storedTotalAmount <= 0) global.set("totalAmount", 0, "storeInFile");

if (lastValueProvidedByShelly <= 0) global.set("lastValueProvidedByShelly", 0, "storeInFile");



// check if Shelly provides a new value

if (msg.payload > 0) {

  msg.payload = (storedTotalAmount + msg.payload).toString();



}else{

  // here, we detect a Shelly power off: the last recorded value was > 0 and the new value is <= 0

  if (lastValueProvidedByShelly > 0) {

    storedTotalAmout = storedTotalAmount + lastValueProvidedByShelly;

    global.set("totalAmount", storedTotalAmount, "storeInFile");

  } 

  msg.payload = storedTotalAmount.toString();

}



// then we have to store last value provided by Shelly

global.set("lastValueProvidedByShelly", msg.payload, "storeInFile");

return msg;

It is overwriting the Data given out. I was at a value of 27. Then i restart the device, and it startet to count from zero

As I said that algorithm does not work. In particular it does not work if node-red is reset as it is not storing the total each time. Also it relies on the first sample received from the shelly after it is reset being zero. If there is a bit of a delay in node-red getting going so the first value from the shelly is, for example, 1, then again it will fail. The algorithm I posted earlier should work.

You're right, I've edited the code to respect your requirements. Again, correct me if I'm wrong or if a typo appears.
I've also added the return msg; that I forgot at first edit.

There's still a default if Shelly is reset before sending the highest value but I don't know how you can handle this.

I think this copes with all the conditions.

// Given current total energy from Shelly in msg.payload
let runningTotalEnergy = global.get("runningTotalEnergy", "storeInFile") || 0
let previousShellyValue = global.get("previousShellyValue", "storeInFile") || 0
// test whether the Shelly has reset since last input
if (msg.payload < previousShellyValue) {
    // yes it has, just add new value to the total
    runningTotalEnergy += msg.payload
} else {
    // no so add the difference between this and last to the running total
    runningTotalEnergy += msg.payload - previousShellyValue
}
global.set("previousShellyValue", msg.payload, "storeInFile")
global.set("runningTotalEnergy",runningTotalEnergy, "storeInFile")
msg.payload = runningTotalEnergy
return msg;

Here is a test flow to check it out

[{"id":"565dd42b.e2948c","type":"debug","z":"514a90a5.c7bae8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":560,"y":1119,"wires":[]},{"id":"d383a419.4cdff8","type":"inject","z":"514a90a5.c7bae8","name":"","topic":"","payload":"0","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":133,"y":1021,"wires":[["f3e295a2.fbe39"]]},{"id":"62fc919c.21a678","type":"inject","z":"514a90a5.c7bae8","name":"","topic":"","payload":"1","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":127,"y":1080,"wires":[["f3e295a2.fbe39"]]},{"id":"eb957dc3.1c2c5","type":"inject","z":"514a90a5.c7bae8","name":"","topic":"","payload":"10","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":129,"y":1145,"wires":[["f3e295a2.fbe39"]]},{"id":"444dffc.dac8a8","type":"inject","z":"514a90a5.c7bae8","name":"","topic":"","payload":"20","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":133,"y":1211,"wires":[["f3e295a2.fbe39"]]},{"id":"f3e295a2.fbe39","type":"function","z":"514a90a5.c7bae8","name":"Total Energy","func":"// Given current total energy from Shelly in msg.payload\nlet runningTotalEnergy = global.get(\"runningTotalEnergy\", \"storeInFile\") || 0\nlet previousShellyValue = global.get(\"previousShellyValue\", \"storeInFile\") || 0\n// test whether the Shelly has reset since last input\nif (msg.payload < previousShellyValue) {\n    // yes it has, just add new value to the total\n    runningTotalEnergy += msg.payload\n} else {\n    // no so add the difference between this and last to the running total\n    runningTotalEnergy += msg.payload - previousShellyValue\n}\nglobal.set(\"previousShellyValue\", msg.payload, \"storeInFile\")\nglobal.set(\"runningTotalEnergy\",runningTotalEnergy, \"storeInFile\")\nmsg.payload = runningTotalEnergy\nreturn msg;","outputs":1,"noerr":0,"x":362,"y":1119,"wires":[["565dd42b.e2948c"]]}]
1 Like

Well, yes kind of.

If i attach my mqtt node to your function node, its having a strange behaivor:

Its reporting numbers like: "8000210270260101", then "80002102702601010" then "8000210270260101026" and so on. Every 30 seconds.

But every 30 seconds there are numbers like "101", then "127" then "154" and so on.

Thats the increasing energyconsumption in "Wattseconds"

Do you mean the big numbers are on the output and the small ones are on the input? If so then you have started with a large number in the total in the global context (which is persistent remember, so will have whatever it had following earlier experiments).
You can clear them from the context tab I think, or here is a little flow you can use.

[{"id":"975f0758.d6e53","type":"function","z":"514a90a5.c7bae8","name":"Clear history","func":"// Clears the energy context variables\nglobal.set(\"previousShellyValue\", 0, \"storeInFile\")\nglobal.set(\"runningTotalEnergy\",0, \"storeInFile\")\n","outputs":1,"noerr":0,"x":384,"y":1276,"wires":[[]]},{"id":"116da10.2a8eedf","type":"inject","z":"514a90a5.c7bae8","name":"Clear","topic":"","payload":"0","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":193,"y":1276,"wires":[["975f0758.d6e53"]]}]

Its looking like its adding the reportet energy to the total energy at the end of the existing number.

Yes. Small is input whats reportet over mqtt. and bis is the one i get out of your function

Ok i tested it out.

I cleared the context variable. Then i startet.. i took it up tp round about 790. When i powerd off and on my device, and it startet to report from zero again, the total looked like this: After the first 0 came -> 7900 then again a 0 came -> 79000 -> then a 23 came -> 7900023 -> and so on. do you understand?

Edit:

Ok the problem was:

The MQTT Node is giving out strings.

I replaced msg.payload to a local variable called "payload" and did a parseInt() at begin, and before returning it, i did .toString(). Now its working perfect!

1 Like

Thank you very much for all the reports and much help! I would never get this by myself.

I have learned alot about the context programming now with this!

Best not to convert it to a string on output. Keep it as a number. The MQTT Out node will convert it if necessary.
If you are after converting the input to a number with parseInt you should check that the result is a number. If something weird comes through mqtt then you don't want it messing up your total. So maybe something like

// Given current total energy from Shelly in msg.payload as a string
payload = parseInt(msg.payload)
// check that it has converted ok (so it is not not-a-number)
if (!isNaN(payload)) {
    let runningTotalEnergy = global.get("runningTotalEnergy", "storeInFile") || 0
    let previousShellyValue = global.get("previousShellyValue", "storeInFile") || 0
    // test whether the Shelly has reset since last input
    if (payload < previousShellyValue) {
        // yes it has, just add new value to the total
        runningTotalEnergy += payload
    } else {
        // no so add the difference between this and last to the running total
        runningTotalEnergy += payload - previousShellyValue
    }
    global.set("previousShellyValue", payload, "storeInFile")
    global.set("runningTotalEnergy",runningTotalEnergy, "storeInFile")
    msg.payload = runningTotalEnergy
} else {
    // an illegal string passed in so ignore it, don't pass on anything
    msg = null
}
return msg;
1 Like

Thanks. But i cant give it out with an Debugnode or on a Text Output at Dashboard as a number