Flooding - early warning system

[hazymat]

Hello, this is an urgent and real world problem which I'd like to try to alleviate with Node-RED!

note: I realised after posting this, that I'm very lucky compared to some people in mainland europe right now who have been far more badly affected by flooding than we are. I know at least one person here on the forum is from Hungary which has been very badly affected. I hope everyone is ok. We are safe in the UK, although I will need to see a doctor this morning for reasons that might become apparent

The urgency
At 2.56am tonight I had a call from my neighbour... "our basement is flooding again". Hasn't happened for a few years. By 3.05am I found myself physically holding a pump hose onto a high pressure pump, only to have it pop off and geyser up about 4 metres, hitting my face before I was able to move out the way. Unfortunately we aren't talking rainwater here. The sewage system back flows into our properties. No, we don't live in some rural area, we live in a very busy and urban part of London (Chiswick) on a main road with bus stops, restaurants, and residential homes. It hits loads of us here, and I've seen first-hand the damage caused to some of our local restaurants. Yes, the UK is totally screwed! I'll spare you too many grim details, suffice to say it was an extremely powerful pump and foul water went straight into my face / up the nose etc. and it wasn't possible to shower straight away afterwards as sewage blocked...

There's a lot of info I could tell you here, a lot of background / history, but I'm going to cut to the data, as that's going to help me right now.

Here's a trace of the sump water level in our rear basement:

This shows the rear pump working normally. Water fills the sump, the float activates, and the pump dumps the water out. I mean - a cycle of 30 minutes - that's a lot of water in this case, as we had major floods here - but all is working okay.

Now let me overlay the situation with the front pump on top of that:

The blue trace is the front sump water level. This pump is connected to the sewage pipe much closer to where it exits the building, and near to the non-return valve on the main sewage line.

When there's some kind of blockage in the street, and sewage flows back towards the house, the sump pump attempts to pump water out towards the street and fails, so it fills up again immediately.

Let's consider this an "error state". During normal rainfall, that blue line looks far more "normal", i.e. it matches the frequency of the green trace. (The level creeps up slowly then eventually ejects quickly.)

Anyway, let's abstract all these mechanics out of the discussion - I had a phone call from my neighbour at 2.56am but this was too late. It would have been far less stressful if I had known 30 minutes earlier. We can do that, right?

Now you have a feel for the problem - please can someone help me code this thing up? I have done signal analysis stuff before but I could do with a few pointers to get me started.

Here's the data I get over MQTT from my Arduino Sump Pump system. This is an example message in Node-RED, I get these every 6 seconds.

{"Fmm":429,"Rmm":395,"Pmm":430,"Fpc":46,"Rpc":48,"Ppc":36,"Fal":69,"Ral":55,"Pal":83,"Farm":true,"Rarm":true,"Parm":true,"Fa":false,"Ra":false,"Pa":false,"Fh":0,"Rh":99.9,"Ph":99.9,"Ft":0,"Rt":16.2,"Pt":17.7}

That blue line on the graph corresponds to Fpc (front percent) and the green line corresponds to Rpc (rear percent).

Any practical help would be greatly appreciated at this stage!

[krambriw]

Oh my,,,,

Just to understand better, is it possible to break down, what conditions are we looking for? What are normal good conditions requiring no actions and when is the first indication that something bad is going to happen?

Is an ideal situation that the blue should follow the green and not oscillate that much? Could frequent changes of the blue be used as early warnings?

What does it mean that the blue is oscillating that much? Is that an indication that the front pump is working heavily, not capable of emtying, too much water to handle?

What are normal idle levels that should be expected for each? Could exceptions from those be used as early warning signals?

[hazymat]

@krambriw

Is an ideal situation that the blue should follow the green and not oscillate that much?

Yes, I included the green one to show how the blue one normally looks. Sorry that wasn't so clear!

So on a normal rainy day, the blue sump should fill slowly then empty quickly. When I say "fill slowly", I expect it to take at least 30 minutes, if not longer. Of course, if it fills but not quite to the point the pump kicks in, then doesn't rain for 2 months, then that oscillation could be 2 months!

But when the sewage system in the street fails, our non-return valve closes, which basically means our house sewage is trapped inside and can't get out. So, the pump operates, then almost immediately the water filters back in.

Here's a closeup of the trace, showing timings, so you can see how quickly this happens (i.e. in "error state"):

[jbudd]

Presumably each peak on the blue trace shows the time when the front pump turned on, roughly 5 times between 02:00 and 02:30

If Node-red is aware of these pump switched on events you could simply take two such events within 10 minutes (say) to flag an error. In this case that would be about 02:10.

Of course you can get the same directly from the fluid level data with a bit more work.

[hazymat]

@jbudd

each peak on the blue trace shows the time when the front pump turned on

Yes exactly that. (My post above has a better zoomed in version of the blue line.) Unfortunately I've no immediately obvious way of sensing when the pump goes on, other than using the water levels from this graph. I guess maybe I could install some kind of power sensing device but would rather get this job done more quickly using what I've got set up already...

[krambriw]

Yeah, a neat warning sent at 2:10 would have given a 46 min pre-warning to the neighbours phone call. This should work, worth coding & testing

[krambriw]

@hazymat

using the water levels

Saving them and measuring the time in between the peaks?

Or just measure the time between the latest?

[hazymat]

@krambriw

Saving them and measuring the time in between the peaks

Yes although the more you think about it, the more difficult this problem becomes. How can I know something was a peak unless I measure the stuff around it and analyse that as well?

Let's say the pump kicks in around 58% at normal times. What if there's drift and it never reaches 58% but this oscillation still happens, e.g. hitting 57%? So I would need to dynamically update the peak.

Also what if I measure it to be 57%, then 2 minutes later it's 57%, that could mean there's a problem, or it could mean there was no rain at all.

I guess I'm trying to say that I realise this needs some kind of waveform analysis, I'm just not sure where to start

[krambriw]

I think you could use the data and smooth it. Then you set fixed trigger levels. If the smoothed curve then breaks up through the trigger levels, you can possible use that as early warnings

When smoothing you need to take into account how frequent you get new values and their values. Getting a kind of time weighted resulting curve of the peak values. Is that called exponential moving average or something?

node-red-contrib-prib-functions

[krambriw]

Maybe it is not the actual peak values that should be monitored, instead the time between them??

[jbudd]

At every data point the level is higher, static or lower then the previous level. You can define a turning point something like:
The level is currently falling (lower than the last value)
The level was falling at the last data point too
The level was rising 3 data points ago (allowing a flat spot at the top of the trace either side of the pump switching on)
The level was rising 4 data points ago too

The red arrow shows when this would be detected as a turning point

So you have to retain (an array in a context variable?) the trend for 5 or 6 data points.

When you detect a turning point, how long is it since the previous one?

No doubt there are established methods to spot such turning points in a stream of data, but I think that's a fairly simple basis for doing it yourself.

[krambriw]

Such a beatiful example of quickly applying Node-RED to help with a real life problem, except for handling the muddy water of course, but maybe preventing it to happen or minimize damage in future

[hotNipi]

With this it is possible to determine the cycle and store the time into an array.

To test and adjust, use dashboard slider node and try to mimic actual conditions and see if it stores cycles as you need. Adjust threshold values as needed.

// cycle is considered complete when it has been over the hi threshold and fell under the lo threshold

let cycleStore = global.get('cycleStore') || { cycles: [], max: Number.MIN_SAFE_INTEGER, threshold: { lo: 44, hi: 65 }, limit: 20 }
let incoming = msg.payload
let state = ""
const between = (x, min, max) => { return x >= min && x <= max; };

if (between(incoming, cycleStore.threshold.lo, cycleStore.threshold.hi)) {
    state = "between"
}
else {
    if (incoming > cycleStore.threshold.hi) {
        state = "high"
        if (cycleStore.max < incoming) {
            cycleStore.max = incoming
            global.set("cycleStore", cycleStore);
        }
    }
    else {
        state = "low"
        if (cycleStore.max > cycleStore.threshold.lo) {
            cycleStore.cycles.push(Date.now());
            cycleStore.max = Number.MIN_SAFE_INTEGER
            while (cycleStore.cycles.length > cycleStore.limit) {
                cycleStore.cycles.shift()
            }
            global.set("cycleStore", cycleStore);
        }
    }
}
node.warn(cycleStore.cycles)
node.status({ fill: "green", shape: "dot", text: "state: " + state });

Then just analyze the frequency of cycles and rise alert if it happens too often.

[smanjunath211]

Is this is what 'machine learning' means..?

[krambriw]

Building on @hotNipi suggestion, a little simulator

I'm setting hi, lo, limit to 50, 20 and 5

Assuming that:

• in a normal situation the pump starts at a level above 50% and stops below 20%
• normally there is just one cycle every 30 min or so

In the flow I have added a timer that resets the cycles every 15 minutes. If I collect five cycles within 15 minutes I assume there is incoming trouble and I do create an alert

This can of course be further developed, with all the Node-RED possibilities, sending notifications to mobile devices etc etc

Do you have some emergency pump?? Then starting additional evacuation....

[
    {
        "id": "f4509715bb838737",
        "type": "function",
        "z": "84b7dba35cbfb04c",
        "name": "function 17",
        "func": "// cycle is considered complete when it has been over the hi threshold and fell under the lo threshold\n\nlet cycleStore = global.get('cycleStore') || { cycles: [], max: Number.MIN_SAFE_INTEGER, threshold: { lo: 20, hi: 50 }, limit: 5 }\nlet incoming = msg.payload\nlet state = \"\"\nconst between = (x, min, max) => { return x >= min && x <= max; };\n\nif (between(incoming, cycleStore.threshold.lo, cycleStore.threshold.hi)) {\n    state = \"between\"\n}\nelse {\n    if (incoming > cycleStore.threshold.hi) {\n        state = \"high\"\n        if (cycleStore.max < incoming) {\n            cycleStore.max = incoming\n            global.set(\"cycleStore\", cycleStore);\n        }\n    }\n    else {\n        state = \"low\"\n        if (cycleStore.max > cycleStore.threshold.lo) {\n            cycleStore.cycles.push(Date.now());\n            cycleStore.max = Number.MIN_SAFE_INTEGER\n            while (cycleStore.cycles.length > cycleStore.limit) {\n                cycleStore.cycles.shift()\n            }\n            global.set(\"cycleStore\", cycleStore);\n        }\n    }\n}\nnode.warn(cycleStore.cycles)\nnode.status({ fill: \"green\", shape: \"dot\", text: \"state: \" + state });\n\nif (cycleStore.cycles.length == cycleStore.limit){\n    node.warn(\"Alarm, muddy water flooding pre-warning\");\n}\n",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 520,
        "y": 230,
        "wires": [
            []
        ]
    },
    {
        "id": "8d24cbc80913efe8",
        "type": "inject",
        "z": "84b7dba35cbfb04c",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "58",
        "payloadType": "num",
        "x": 290,
        "y": 170,
        "wires": [
            [
                "f4509715bb838737"
            ]
        ]
    },
    {
        "id": "4382b418bc05d860",
        "type": "inject",
        "z": "84b7dba35cbfb04c",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "15",
        "payloadType": "num",
        "x": 290,
        "y": 280,
        "wires": [
            [
                "f4509715bb838737"
            ]
        ]
    },
    {
        "id": "7ac4e2d65524e554",
        "type": "inject",
        "z": "84b7dba35cbfb04c",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "900",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "true",
        "payloadType": "bool",
        "x": 280,
        "y": 400,
        "wires": [
            [
                "3f6f6a15dec434ab"
            ]
        ]
    },
    {
        "id": "3f6f6a15dec434ab",
        "type": "change",
        "z": "84b7dba35cbfb04c",
        "name": "",
        "rules": [
            {
                "t": "set",
                "p": "cycleStore.cycles",
                "pt": "global",
                "to": "[]",
                "tot": "jsonata"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 520,
        "y": 400,
        "wires": [
            []
        ]
    }
]

[krambriw]

Another example of simulator

Changed the limit to 20 cycles.

A cycle is made and counted once a minute. Every 15 minute the cycle array is cleared so it will normally not reach it's limit

If you open the gate, cycles are coming in at a 5 sec rate and the limit is soon reached. An alert is then issued and repeated until the cycle array is cleared and cycles are back to normal

In the flow I use the node-red-contrib-simple-gate

[{"id":"f4509715bb838737","type":"function","z":"84b7dba35cbfb04c","name":"function 17","func":"// cycle is considered complete when it has been over the hi threshold and fell under the lo threshold\n\nlet cycleStore = global.get('cycleStore') || { cycles: [], max: Number.MIN_SAFE_INTEGER, threshold: { lo: 20, hi: 50 }, limit: 20 }\nlet incoming = msg.payload\nlet state = \"\"\nconst between = (x, min, max) => { return x >= min && x <= max; };\n\nif (between(incoming, cycleStore.threshold.lo, cycleStore.threshold.hi)) {\n    state = \"between\"\n}\nelse {\n    if (incoming > cycleStore.threshold.hi) {\n        state = \"high\"\n        if (cycleStore.max < incoming) {\n            cycleStore.max = incoming\n            global.set(\"cycleStore\", cycleStore);\n        }\n    }\n    else {\n        state = \"low\"\n        if (cycleStore.max > cycleStore.threshold.lo) {\n            cycleStore.cycles.push(Date.now());\n            cycleStore.max = Number.MIN_SAFE_INTEGER\n            while (cycleStore.cycles.length > cycleStore.limit) {\n                cycleStore.cycles.shift()\n            }\n            global.set(\"cycleStore\", cycleStore);\n        }\n    }\n}\n//node.warn(cycleStore.cycles)\nnode.status({ fill: \"green\", shape: \"dot\", text: \"state: \" + state });\n\n//krambriw added\nif (cycleStore.cycles.length == cycleStore.limit){\n    if(state == \"low\"){\n        let alert = \"Alarm, muddy water flooding pre-warning\";\n        node.warn(alert);\n        node.status({ fill: \"red\", shape: \"dot\", text: alert });\n    }\n}\n","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1290,"y":190,"wires":[[]]},{"id":"8d24cbc80913efe8","type":"inject","z":"84b7dba35cbfb04c","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"60","crontab":"","once":true,"onceDelay":0.1,"topic":"","payload":"58","payloadType":"num","x":920,"y":190,"wires":[["f4509715bb838737","8375646acf8bfb0a"]]},{"id":"7ac4e2d65524e554","type":"inject","z":"84b7dba35cbfb04c","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"900","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"true","payloadType":"bool","x":920,"y":640,"wires":[["3f6f6a15dec434ab"]]},{"id":"3f6f6a15dec434ab","type":"change","z":"84b7dba35cbfb04c","name":"","rules":[{"t":"set","p":"cycleStore.cycles","pt":"global","to":"[]","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":1160,"y":640,"wires":[[]]},{"id":"8375646acf8bfb0a","type":"trigger","z":"84b7dba35cbfb04c","name":"","op1":"","op2":"15","op1type":"nul","op2type":"num","duration":"2","extend":false,"overrideDelay":false,"units":"s","reset":"","bytopic":"all","topic":"topic","outputs":1,"x":920,"y":250,"wires":[["f4509715bb838737"]]},{"id":"6ba13312cd141c86","type":"inject","z":"84b7dba35cbfb04c","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"5","crontab":"","once":true,"onceDelay":0.1,"topic":"","payload":"58","payloadType":"num","x":920,"y":340,"wires":[["dbb463e8c67ad7ed","e66ce211ade40662"]]},{"id":"dbb463e8c67ad7ed","type":"trigger","z":"84b7dba35cbfb04c","name":"","op1":"","op2":"15","op1type":"nul","op2type":"num","duration":"2","extend":false,"overrideDelay":false,"units":"s","reset":"","bytopic":"all","topic":"topic","outputs":1,"x":930,"y":400,"wires":[["e66ce211ade40662"]]},{"id":"e66ce211ade40662","type":"gate","z":"84b7dba35cbfb04c","name":"","controlTopic":"control","defaultState":"closed","openCmd":"open","closeCmd":"close","toggleCmd":"toggle","defaultCmd":"default","statusCmd":"status","persist":false,"storeName":"memory","x":1120,"y":340,"wires":[["f4509715bb838737"]]},{"id":"ada45b27ecf3f31a","type":"inject","z":"84b7dba35cbfb04c","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"control","payload":"open","payloadType":"str","x":940,"y":470,"wires":[["e66ce211ade40662"]]},{"id":"b9d8540c6e3f7b5e","type":"inject","z":"84b7dba35cbfb04c","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"control","payload":"close","payloadType":"str","x":940,"y":530,"wires":[["e66ce211ade40662"]]}]

[hazymat]

Thanks for these responses, I'm going to get them going ASAP and test!

[craigcurtin]

Wouldnt a Shelly EM (Flashed to Tasmota) be a better way as it could also send out warnings if the pump HAS NOT turned on (or OFF) for a defined period ?

Presumably you would already have something in place if you do not receive any data from the Arduino - but why not put in place a 2nd failsafe and code around that ?

Pretty easy to measure elapsed time between pump on/off using trigger nodes and take appropriate actions if they fall outside of normal parameters

Craig

[Trying_to_learn]

I think @BartButenaers may be someone you need to ask help on this.
He's been in your situation recently.

I did help in the problem too.

My thoughts.

1 - the maximum inflow rate. Water coming INTO your building
2 - the outflow rate. Water going OUT of your building.
Alas the first is hard to know as a flood can just happen and a whole wall of water can just flood you.
The outflow rate though is something you can know by how big your pumps are.

Understand if the outflow is LESS than the inflow: you have a big problem.

Suggestions:

You need a sump (hole in the floor/ground below the floor where the pumps can live.
You need a level sensor to detect the water's level.
From that you need to get TWO ..... values.
1 - when to turn the pump/s ON
2 - when to turn the pump/s OFF
3 - (of you want/can get it) We are not getting the water out fast enough!

So:
level 1 is when the water is.... say half filling the sump.
level 2 is when the water is near the bottom of the sump.
(Depending on they type of pump, it may not be a good idea to let the sump empty.)
level 3 - again, if you can get it - would be if the water gets to the top of the sump.

When the sensor sends a level 1 message: the pumps are turned on.
When the sensor sends a level 2 message: the pumps are turned off.
And for level 3: You need a bigger pump.

Alot of this can be simply done if the sensor sends analogue values for the water leve.

The higher the number, the deeper the water.

So then all you need is a switch node so:

if (value > start)  output 1
if (value < stop)   output 2
if (value > the_third one)  output 3

output 3 then warns you that things are not going good.

[krambriw]

@craigcurtin

it could also send out warnings if the pump HAS NOT turned on (or OFF) for a defined period ?

Easy to add as you say. You already have data that is expected to come in. Just add a trigger node with decent timer setting and it will fire when conditions are met

@Trying_to_learn

you have a big problem

I have understood that the problem is not about when starting/stopping the pumps. It's about that there is a blockage in the street. So this is about a pre-warning system to give you additional time, bring in all the buckets you can find or whatever other actions to take to avoid flooding damage as much as possible