Storing energy data

[Colin]

@MeisterQ

var globaleVariable = global.get("variable"||2);

I presume you meant
var globaleVariable = global.get("variable") || 2
though there is no point since the next line overwrites the variable anyway.

You have not got an else clause on the test in the second function so if the if the test fails it doesn't return a message at all.
As for whether this is complicated or not, then this is certainly not complicated.

As I said, I don't think you can rely on the LWT state as I think it will only trip after a certain time.

How about this for an idea. Keep in persistent memory the current running total and also the last received value from the device. The reason I asked what will happen to the received value following a reset is because it will go to zero (or a small number presumably) so you can use that to tell you the device has reset. So if you keep the previous value in persistent storage then each time a new value comes in you can compare it with the previous one. If the received value is higher than the last then add the difference onto the running total. If it is lower then set the saved previous value to the new one just received, and also add that to the running total.

[MeisterQ]

Sounds good.

Can you give me a bit of code-help?

But if i do it in this way:

Wouldnt the Total-Value stay at the same numbers, till the new value is higher? Means.. if the total is around 10kWh, the device connected to the shelly has to consume 10kWh first, till the total number will increase?

[Colin]

As I said "If the received value is higher than the last then add the difference onto the running total. If it is lower then set the saved previous value to the new one just received, and also add that to the running total." So the running total gets bigger every time there is a new value.

[Colin]

Ah, I see your possible confusion. Expanding a bit on my words to make it clearer:
If the new received value is higher than the last received value then add the difference onto the running total. If it is lower then set the saved previous value to the new one just received, and also add that to the running total.

[MeisterQ]

And in this point i need your help.

Iam really not good in programming, can you help me abit maybe?

[afelix]

When I have to work out programming problems, I usually work it out in steps. What do I have to do, what data do I need, where does it come in, what operations do I need to do on it. Then for every small step, work it out. First in logic, then later translating that logic to code. Can you work it out in just plain logic, like English words describing each step or perhaps pseudocode already?

[Colin]

@MeisterQ

can you help me abit maybe?

Assuming that you are writing a function node that is fed from the mqtt node, then the first thing it will need to do is to pick up the running total and previous value from context. Then compare the new value with the previous value. See if you can get that bit going.

[MeisterQ]

https://pastebin.com/LyVvR1qH

Something like this?

[afelix]

Please note that you don't have to use pastebin for posting flows, in fact it is easier and preferred if you post them here by pasting them in between 2 blocks of 3 backticks (so 3 backticks on the top line, then the pasted flow on the middle, followed by another 3 backticks on the third line.
```
code or flow here
```
This character, also known as the grave accent can be found on QWERTY and QWERTZ keyboard on the same key as the tilde (~), on most of those keyboards it is located left of the row of numbers.

Please see this topic for more information on how to post these:

How to share code or flow json

To share code samples or flow json in this forum, you need to take care to format it properly so that it is displayed correctly. The easiest way to do that is to click the 'Preformatted Text' button in the toolbar: [image] Then paste in your code or flow json. You should end up with three back-tick characters - ``` - on their own line before and after the code. On standard US English keyboards, this is on the same key as the ~ character. [IMG_20181111_155651] Using the backticks will ens…

[MeisterQ]

I see. Sorry.

[{"id":"def1cf3e.911b1","type":"function","z":"12458ddf.1454e2","name":"","func":"var erg = parseInt(msg.payload) / 60000;\nerg = erg.toFixed(2);\nmsg.payload = erg.toString();\nreturn msg;","outputs":1,"noerr":0,"x":670,"y":260,"wires":[["9fc629b5.ba4668","36378296.b2343e"]]},{"id":"dc04ec31.f182b","type":"mqtt in","z":"12458ddf.1454e2","name":"","topic":"shellies/shellyplug-s-040BAD/relay/0/energy","qos":"2","datatype":"auto","broker":"51c9edc2.fc7ea4","x":270,"y":260,"wires":[["def1cf3e.911b1","325258ba.93dc7"]]},{"id":"9fc629b5.ba4668","type":"debug","z":"12458ddf.1454e2","name":"Energie","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":860,"y":220,"wires":[]},{"id":"36378296.b2343e","type":"function","z":"12458ddf.1454e2","name":"","func":"var globaleVariable = global.get(\"variable\"||2);\nvar globaleVariable = msg.payload;\nglobal.set(\"shellyEnergyTest\",globaleVariable);\n\nreturn {payload: globaleVariable};","outputs":1,"noerr":0,"x":850,"y":260,"wires":[[]]},{"id":"325258ba.93dc7","type":"function","z":"12458ddf.1454e2","name":"","func":"var shellyEnergyTest = global.get(\"shellyEnergyTest\");\n\nif (msg.payload > shellyEnergyTest)\n{\n    shellyEnergyTest = shellyEnergyTest + msg.payload;\n}\n\n\n\n\n\nmsg.payload = shellyEnergyTest.toString();\n\nreturn msg;","outputs":1,"noerr":0,"x":670,"y":320,"wires":[["615c8159.a4a8"]]},{"id":"16125ce4.6a1063","type":"inject","z":"12458ddf.1454e2","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":380,"y":320,"wires":[["325258ba.93dc7"]]},{"id":"615c8159.a4a8","type":"debug","z":"12458ddf.1454e2","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":870,"y":400,"wires":[]},{"id":"51c9edc2.fc7ea4","type":"mqtt-broker","z":"","broker":"192.168.2.173","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","willTopic":"","willQos":"0","willPayload":""}]

[Colin]

Well at least you have made a start. A couple of points. When posting flows just post the nodes that we need to see, so mark then then use the export feature. For example you posted the MQTT in node and the associated config node. That is of no use to us and in fact means that when we delete it we have to manually go and delete the mqtt server config node associated with the mqtt in node. In addition you have publicly given away your local subnet address and also the address of your mqtt server. It is best not to post such things on public sites.
Secondly I suggest that you put all this in one function node, I can't see any point splitting the algorithm into mulltiple nodes, and I can't make much sense out of what some of them are doing.
I see you are converting numbers to strings for some reason. I advise against that. Most times it is more useful (and certainly more efficient) to keep them as numbers.
On a detail point give your variables meaningful name. You have called the running total shellyEnergyTest. Can I suggest it would be much better to call it something like energyRunningTotal or something similar that will make it much easier to understand. When you fetch it from global context you need to think about what will happen the first time, before it has been written the first time. So the line that fetches it could be something like
var energyRunningTotal = global.get("energyRunningTotal") || 0
The || 0 on the end (spoken as Or Zero) says that if the value is not yet defined then set it to 0.
So that is a good start, subject to a few tweaks, now look back at the suggested algorithm and carry on. It will be going in no time.

[MeisterQ]

Well,

im to unexperied to code something like this. I think i have to live with that problem in future.

I cant solve it

[jledun]

hey,

in your settings.json, add the following:

contextStorage: {
  storeInFile: { module: "localfilesystem"}
},

This will make the global.context persistent.
Description of persistent storage can be found here.

Then the total amount of power will be stored between node-red's restarts.

Then you just have to modify your code to detect when Shelly is reset:
[EDIT]

let storedTotalAmount = global.get("totalAmount", "storeInFile") || 0;
const previousShellyValue = global.get("previousShellyValue", "storeInFile") || 0

// first run: init value in persistent storage
if (storedTotalAmount <= 0) global.set("totalAmount", 0, "storeInFile");
if (previousShellyValue <= 0) global.set("previousShellyValue", 0, "storeInFile");

if (msg.payload < previousShellyValue) {
  // here we detect a reset of Shelly system
  storedTotalAmount = storedTotalAmount + msg.payload;
}else{
  // note that if there's no change in Shelly's payload, the difference will be 0
  storedTotalAmount = storedTotalAmount + msg.payload - previousShellyValue;
}
global.set("totalAmount", storedTotalAmount, "storeInFile");
global.set("previousShellyValue", msg.payload, "storeInFile");
msg.payload = storedTotalAmount;

return msg;

Correct me if I'm wrong (and of course, I was, at first edit )

Hope it helps

[Colin]

You have to check the new input value less than the previous input value and use that to indicate that the shelly has reset.
In addition the amount you have to add to the previous value is the difference between the new input value and previous one (unless the shelly has reset).

[MeisterQ]

@jledun thank you for that nice example but:

1. if i add

contextStorage: {
  storeInFile: { module: "localfilesystem"}
},

to the settings.json, at the bottom my node-red wont start anymore.

For the rest: if i put the code in a function-node, how should i return it to a debug window or mqtt-message?

Just put a "return msg.payload" at the end?

because im getting some errormessages then

[Colin]

You have to put the contextStorage specification inside the module.exports section of the settings file. If you look you can see that it is just a javascript object declaration so make sure you get the commas in the right places.

A function must always return a complete message so you should have return msg at the end.
I advise against converting the values to strings as in the flow posted, keep them as numbers so you can send them to MQTT for example.

[Edit] See these for help with writing functions and handling messages. It is worth looking through the rest of the docs too.
https://nodered.org/docs/user-guide/writing-functions
https://nodered.org/docs/user-guide/messages

[MeisterQ]

Thank you. It works.

I tried to restart my Raspberry and Turned off and on my Shelly device. Now it says

"ReferenceError: storeTotalAmount is not defined (line 35, col 17)"

for the total amount. Before restarting my raspberry it worked and showed me the total amount.

Well i saw there is a "d" missing in the variables name.

Edit:

If i use it in this way:

let storedTotalAmount = global.get("totalAmount", "storeInFile") || 0;

let lastValueProvidedByShelly = global.get("lastValueProvidedByShelly", "storeInFile") || 0;



// first run: init value in persistent storage

if (storedTotalAmount <= 0) global.set("totalAmount", 0, "storeInFile");

if (lastValueProvidedByShelly <= 0) global.set("lastValueProvidedByShelly", 0, "storeInFile");



// check if Shelly provides a new value

if (msg.payload > 0) {

  msg.payload = (storedTotalAmount + msg.payload).toString();



}else{

  // here, we detect a Shelly power off: the last recorded value was > 0 and the new value is <= 0

  if (lastValueProvidedByShelly > 0) {

    storedTotalAmout = storedTotalAmount + lastValueProvidedByShelly;

    global.set("totalAmount", storedTotalAmount, "storeInFile");

  } 

  msg.payload = storedTotalAmount.toString();

}



// then we have to store last value provided by Shelly

global.set("lastValueProvidedByShelly", msg.payload, "storeInFile");

return msg;

It is overwriting the Data given out. I was at a value of 27. Then i restart the device, and it startet to count from zero

[Colin]

As I said that algorithm does not work. In particular it does not work if node-red is reset as it is not storing the total each time. Also it relies on the first sample received from the shelly after it is reset being zero. If there is a bit of a delay in node-red getting going so the first value from the shelly is, for example, 1, then again it will fail. The algorithm I posted earlier should work.

[jledun]

You're right, I've edited the code to respect your requirements. Again, correct me if I'm wrong or if a typo appears.
I've also added the return msg; that I forgot at first edit.

There's still a default if Shelly is reset before sending the highest value but I don't know how you can handle this.

[Colin]

I think this copes with all the conditions.

// Given current total energy from Shelly in msg.payload
let runningTotalEnergy = global.get("runningTotalEnergy", "storeInFile") || 0
let previousShellyValue = global.get("previousShellyValue", "storeInFile") || 0
// test whether the Shelly has reset since last input
if (msg.payload < previousShellyValue) {
    // yes it has, just add new value to the total
    runningTotalEnergy += msg.payload
} else {
    // no so add the difference between this and last to the running total
    runningTotalEnergy += msg.payload - previousShellyValue
}
global.set("previousShellyValue", msg.payload, "storeInFile")
global.set("runningTotalEnergy",runningTotalEnergy, "storeInFile")
msg.payload = runningTotalEnergy
return msg;

Here is a test flow to check it out

[{"id":"565dd42b.e2948c","type":"debug","z":"514a90a5.c7bae8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":560,"y":1119,"wires":[]},{"id":"d383a419.4cdff8","type":"inject","z":"514a90a5.c7bae8","name":"","topic":"","payload":"0","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":133,"y":1021,"wires":[["f3e295a2.fbe39"]]},{"id":"62fc919c.21a678","type":"inject","z":"514a90a5.c7bae8","name":"","topic":"","payload":"1","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":127,"y":1080,"wires":[["f3e295a2.fbe39"]]},{"id":"eb957dc3.1c2c5","type":"inject","z":"514a90a5.c7bae8","name":"","topic":"","payload":"10","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":129,"y":1145,"wires":[["f3e295a2.fbe39"]]},{"id":"444dffc.dac8a8","type":"inject","z":"514a90a5.c7bae8","name":"","topic":"","payload":"20","payloadType":"num","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":133,"y":1211,"wires":[["f3e295a2.fbe39"]]},{"id":"f3e295a2.fbe39","type":"function","z":"514a90a5.c7bae8","name":"Total Energy","func":"// Given current total energy from Shelly in msg.payload\nlet runningTotalEnergy = global.get(\"runningTotalEnergy\", \"storeInFile\") || 0\nlet previousShellyValue = global.get(\"previousShellyValue\", \"storeInFile\") || 0\n// test whether the Shelly has reset since last input\nif (msg.payload < previousShellyValue) {\n    // yes it has, just add new value to the total\n    runningTotalEnergy += msg.payload\n} else {\n    // no so add the difference between this and last to the running total\n    runningTotalEnergy += msg.payload - previousShellyValue\n}\nglobal.set(\"previousShellyValue\", msg.payload, \"storeInFile\")\nglobal.set(\"runningTotalEnergy\",runningTotalEnergy, \"storeInFile\")\nmsg.payload = runningTotalEnergy\nreturn msg;","outputs":1,"noerr":0,"x":362,"y":1119,"wires":[["565dd42b.e2948c"]]}]