# A bigger problem than the ones I have been mentioning to now

**URL:** <https://discourse.nodered.org/t/a-bigger-problem-than-the-ones-i-have-been-mentioning-to-now/12277>\
**Category:** General\
**Created:** [15 June 2019 23:33 UTC](https://discourse.nodered.org/t/a-bigger-problem-than-the-ones-i-have-been-mentioning-to-now/12277 "2019-06-15T23:33:10Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [15 June 2019 23:33 UTC](https://discourse.nodered.org/t/a-bigger-problem-than-the-ones-i-have-been-mentioning-to-now/12277/1 "2019-06-15T23:33:10Z")

</div>

Recently I have been busy with NR and flows.

I've mentioned weird errors, and a few other things which have been brought about because of strange happenings on my flow/s / machines.

I have a couple of RPIs. One is a RPI 2a and acts as a WAP for **local** stuff.  
I have 2 other WAPs. 1 is my _"modem"_ and one is another RPI which is mostly turned off just now for reasons beyond this post.

The RPI WAP is DHCP. (And different IP to the other WAPs)

(and I now realise this could be the cause of the problem, but: here goes.)

This RPI (as stated) is a WAP. It is on 24/7. It is my "main" machine for collecting information about _what's going on in my world_ stuff.

Saying that, it also scans for available WAPs.  
It looks for my main WAP being up/down and the other (third) WAP which is down 99% of the time.

The reason for this is so it can tell me if the main WAP is working or not, and reduce the annoyance when I am wondering why things aren't connecting to it. I can easily see it is down.  
(That fact in itself is a problem, but it only slightly but let's say that it is mostly up and hasn't _really_ been a problem.)  
I did that bit when I was starting off with NR and it also checks if my **internet** link is working.  
Just an exercise in checking things.  
Check the WAP, check the modem, check the internet..... ( a few other checks/ping there too, but...)

The local WAP is a bit _smarter_ in that when a device connects to it, a signal is sent out asking the device to identify itself.  
The device responds and all is sweet in the world.

Getting more adventurous I then started adding logs of what was happening and that is when things got bad.

I was/am seeing unexpected "identify yourself" message replies. These should only happen if a device is asked. Why are they happening?

(I did find some "contamination" of messages and quickly sorted that out.) (IQ +1)

I then started logging the requests. They too are happening for no real reason.

Then I got to this point/stage:

I am getting spruatic (?spelling?) messages saying that the main WAP is down.  
To double check myself, I got the second RPI (which is a RPZ(w)) to also check the state of things.

It is saying that **both** the main and RPI (local) WAP are going down/up at about the same time.

I stuck a lot of extra stuff on the flows to log what is going on, but it is still happening and even the logs aren't making sense.

Here is a breakdown of the flow monitoring the WAPs:  
(pulse trigger at given interval. Every 40 seconds) --\> Scan for WAP. --\> create list --\> split on name.  
If name == modem WAP, #1, if name == RPI WAP, #2

#1 and #2  
--\> timeout trigger (set to 80 seconds) to send `false` if it times out --\> (branch) indicator set to show condition.  
(branch) --\> switch (if `false`) --\> `function` node to timestamp message --\> (branch) log message.  
(branch) --\> another indicator.

Yeah, sorry. It will be hard to get your head around if you aren't me. I'll post the flow/screen shots later if you want. (I'm torn between _too much_ and _not enough_ information.)  
But I now notice that even this is detecting failures, now and then.  
Not as often as I remember, but enough to be of interest.  
Mostly because it is usually **both** which go down/up at about the same time.

Probably timing, but I am just not getting something.

I may extend the timout time to .... 120 seconds rather than 80 (twice the scan time) and see what happens.

 ![Screenshot%20from%202019-06-16%2009-34-14](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/2X/e/eda8ef28a6d84ae383f00a28eb6c181df5dcf137.png)

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [16 June 2019 00:01 UTC](https://discourse.nodered.org/t/a-bigger-problem-than-the-ones-i-have-been-mentioning-to-now/12277/2 "2019-06-16T00:01:20Z")

</div>

That whole flow is really hard to follow.

In my own system, I get my devices, including my Pi's to send out an MQTT message about every 50secs or so to something like `DEVICES/Pi2` with a payload of "Online". I use an MQTT connection that specifies a "Last Will and Testament" message with a timeout of 90 seconds and a payload of "Offine".

This means that I have a very simple set of flows that give a "heartbeat" to the MQTT broker (every 50s) and if the broker doesn't receive such a heartbeat message in 90s, it sets the topic to "Offline".

Now I can have as many instances of Node-RED (on any number of devices) that can listen to those `DEVICES/#` topics and react if anything goes offline.

Not only do I not have to worry about timings within Node-RED flows, I can keep the flows simple by separating out the processing.

I can get any instance of Node-RED to send out its own heartbeat but other devices can do the same thing directly. So all of my ESP8266 sensor platforms do that same heartbeat processing directly to the MQTT broker.

---

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [16 June 2019 00:13 UTC](https://discourse.nodered.org/t/a-bigger-problem-than-the-ones-i-have-been-mentioning-to-now/12277/3 "2019-06-16T00:13:44Z")

</div>

That is a nice idea.

I may think about that soon.

I also like how you are using the _LWT_ on MQTT to detect link failures.

What I do is very similar, but I `PING` the devices. I know that is limiting in some ways, but it is also useful in other ways.

On my devices - as most are RIP - they send back their CPU load and Temperature once connected.

But I don't think I can use what you suggest to detect a WAP.

To try and make it as easy to understand:  
Every 40 seconds I scan the WAP list and sort the names accordingly.  
A timeout of 80 seconds (now changed to 120) is used that if a message is not received it "times out" and sends a `false` signal indicating the WAP is not seen.

I am getting these _bursts_ of both WAPs being Offline/Down but I am not seeing any MQTT activity to support this in that the MQTT link from the _main_ RPI to the RPZ(w) isn't broken.  
I don't see a LWT sent when the RPI (main WAP) (allegedly) dies.

So it _doesn't make sense_ what I am seeing.

To further complicate it: I am getting those weird errors pointing to nodes saying they are _wrong_ but when I test them with the inputs they expect they don't complain.

So I am still at the stage of chasing shadows, rather than actual things.

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [16 June 2019 10:25 UTC](https://discourse.nodered.org/t/a-bigger-problem-than-the-ones-i-have-been-mentioning-to-now/12277/4 "2019-06-16T10:25:21Z")

</div>

> [@Trying\_to\_learn](#):
>
> What I do is very similar, but I `PING` the devices

Ping is a very different test. The MQTT/LWT approach ensures that a devices service is really working. All ping does is find whether the devices network card hasn't died. Because Ping works on a very low level in the network stack, it doesn't tell you very much.

I have something similar to keep track of network devices on my network but I use an ARP scan with a customised lookup file. ARP scan walks the network and tracks MAC addresses returning the IP addresses, if any, for each device found.

I do use ping to track whether my Node-RED instance can still detect a couple of devices. Namely my router and my NAS. Those flows also use MQTT/LWT to indicate whether the device appears to be contactable. But of course, this only tells me if Node-RED can ping them, it doesn't really tell me if they are actually working. For that, I'd need SNMP or I would need to try and connect to a service such as TELNET or a web interface.

Which ever way you look at it, I would say that it is likely that you've created something too complex to be easily debugged. You need to separate out the parts of the flow so that you can track down the actual error, all those timing delays _could_ be causing an issue but I can't work that out from here.

My advice is to simplify the flow, split it up. You may well find that simplifying it gets rid of the issue anyway.

---

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [16 June 2019 11:20 UTC](https://discourse.nodered.org/t/a-bigger-problem-than-the-ones-i-have-been-mentioning-to-now/12277/5 "2019-06-16T11:20:59Z")

</div>

> [@TotallyInformation](#):
>
> Ping is a very different test. The MQTT/LWT approach ensures that a devices service is really working. All ping does is find whether the devices network card hasn't died. Because Ping works on a very low level in the network stack, it doesn't tell you very much.

Yes, and I use `PING` as a low level detection system.

> [@TotallyInformation](#):
>
> but I use an ARP scan

I've heard of ARP, but I think it is tagged differently in my world at this point.  
And, that limits your scanning to fixed devices pretty much the same way my PING limits my scanning too.  
(ARP, that's a network thing about Address Resolution Protocol - guessing.)

> [@TotallyInformation](#):
>
> For that, I'd need SNMP

Simple Network Mail Protocol?  
(It's late.... Sorry)

I have got the flows _split up_ into sections which was/is the idea to help track problems.

What is really helping me now is **subflows** as I am finding a lot of repeat stuff and replacing them with subflows.

Just on how my system works - which isn't really that difficult:

Timed PING to all addresses.  
If a machine replies, it is tagged as "alive" and marked as such.  
Then, it is given a fixed time to start sending telemetry data (using NR) of CPU temp, processor load, etc.  
If that doesn't arrive in a specified time - and format - the machine is fagged as "hung".  
However, if the data is received, the machine is shown as working.

Yes there is a bit of code to do that and a lot of timers and conditions.  
But I think this would be true with your way also. Not that either is better.  
My way is just the way I did it with the knowledge I had at the time.

The WAP time outs are maybe just outside the scope of what I originally did, and I did it _that way_ again only because I knew no other way.

I've tweaked the time out from 80 to 120 seconds - another 40 seconds - and that is 3 periods of scanning for the PING to not work.

That may fix it. I guess these things need to be done/tried before anything can be learned.

I have to do a bit more on the LWT/BC of MQTT. I have the "keep alive time" set to 60 seconds, which is more than the ping time of 40 seconds. But again, if it was failing with 80 second time outs on one level, 60 may fail on this.

I have set all three messages for the LWT/BC to again give as much information as possible to what happened/is going on.

As all the LWT/BC are sent on separate channels (EOM and SOM respectively) I then parse any SOM EOM messages.

I am not seeing any of these when I am getting the "identify yourself" message replies.  
So that still has me stumped.

But then there is the WAP time out I just adjusted today - from memory. Or yesterday at worst.

Another thing which I am using is the `fan` node. Though I think it should be called a `bus` node, as it kind of represents a bus of sorts.

I am using them to help segregate flows into sections and having as few wires running all over the place and rather they be segmented by the `fan` nodes.

But that's just me.

Again: Thanks for the time and suggestions.  
Appreciated.

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [16 June 2019 13:39 UTC](https://discourse.nodered.org/t/a-bigger-problem-than-the-ones-i-have-been-mentioning-to-now/12277/6 "2019-06-16T13:39:48Z")

</div>

> [@Trying\_to\_learn](#):
>
> Address Resolution Protocol

Yes. This is the command, the macfile contains identifiers for common mac address types and known mac addresses.

```auto
sudo arp-scan --localnet --macfile=/home/pi/node/nrlive/mac-vendor-jk.txt

```

You can also do `ip neighbor` to see whether addresses in the current ARP table are reachable.

> **[Address Resolution Protocol](https://en.wikipedia.org/wiki/Address_Resolution_Protocol)**
>
> The Address Resolution Protocol (ARP) is a communication protocol used for discovering the link layer address, such as a MAC address, associated with a given internet layer address, typically an IPv4 address. This mapping is a critical function in the Internet protocol suite. ARP was defined in 1982 by .mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\\"""\\"""'""'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)...

> [@Trying\_to\_learn](#):
>
> Simple Network Mail Protocol

No, you are thinking of SMTP (Simple Mail Transport Protocol).

> **[Simple Network Management Protocol](https://en.wikipedia.org/wiki/Simple_Network_Management_Protocol)**
>
> Simple Network Management Protocol (SNMP) is an Internet Standard protocol for collecting and organizing information about managed devices on IP networks and for modifying that information to change device behavior. Devices that typically support SNMP include cable modems, routers, switches, servers, workstations, printers, and more.
> SNMP is widely used in network management for network monitoring. SNMP exposes management data in the form of variables on the managed systems organized in a man...

Most routers and NAS's can be configured to give useful operational data over SMTP.

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [16 June 2019 14:05 UTC](https://discourse.nodered.org/t/a-bigger-problem-than-the-ones-i-have-been-mentioning-to-now/12277/7 "2019-06-16T14:05:40Z")

</div>

No, based on the context of this thread, he will definitely have meant SNMP.

It's an object based protocol often used in managed switches & servers etc for monitoring.

SMTP is just email transport.

---

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [16 June 2019 21:12 UTC](https://discourse.nodered.org/t/a-bigger-problem-than-the-ones-i-have-been-mentioning-to-now/12277/8 "2019-06-16T21:12:38Z")

</div>

> [@TotallyInformation](#):
>
> Most routers and NAS's can be configured to give useful operational data over SMTP.

Was that a typo?

And thanks for the other stuff too.

I have touched ARP when I was doing routers/CISCO stuff.  
Then there is also RIP but I never got too far down that path.

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [16 June 2019 21:29 UTC](https://discourse.nodered.org/t/a-bigger-problem-than-the-ones-i-have-been-mentioning-to-now/12277/9 "2019-06-16T21:29:47Z")

</div>

Doh! 🤦‍♂️ Yes, SNMP of course as I'd already explained. Sorry for the confusion.

> [@Trying\_to\_learn](#):
>
> RIP

RIP is something different, it lets routers collaborate with each other. Largely discredited now I think due to vulnerabilities.

---

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [16 June 2019 21:33 UTC](https://discourse.nodered.org/t/a-bigger-problem-than-the-ones-i-have-been-mentioning-to-now/12277/10 "2019-06-16T21:33:03Z")

</div>

Moving on from that, here is a list of outputs I got overnight from my _scanners_:

This is the "Who are you" requests. (RAW format)

```auto
>> -- Mark -- 2019-6-14 20:35:27------------
{"WIFI_DEVICE":"GPS","IP_Address":"192.168.1.4"}
{"WIFI_DEVICE":"GPS","IP_Address":"192.168.1.4"}
>> -- Mark -- 2019-6-15 15:59:52------------
{"WIFI_DEVICE":"GPS","IP_Address":"192.168.1.4"}
{"WIFI_DEVICE":"GPS","IP_Address":"192.168.1.4"}

```

Newest at the bottom.

Now the ones I see are:

```auto
>> -- Mark -- 2019-6-16 21:02:49------------

```

So a whole day has gone by and all seems good there.

This is my MQTT stuff:

```auto
>> -- Mark -- 2019-6-16 21:02:49------------
2019-6-16 21:25:21 TimePi shutting down

```

Good.

A very brief look at visible WiFi networks.

```auto
2019-6-17 07:03:02["Telstra6C2C0D", "Telstra Air", "Fon WiFi", "TelstraCA5E31", "Telstra Air", "Fon WiFi", "PiNet", "BigPond121C", "TPG-VZBS", "Tango2", "Chilli", "TelstraAC4579", "Slowest Internet in the World", "Interweb Thingy", "TP-LINK_40B6"]

```

And any WAP changes of interest: (Last)

```auto
>> -- Mark -- 2019-6-16 21:02:49------------

```

That is a lot better than it used to be with a few "Who are you?" being transmitted and a lot of WAP up / downs.

That extra 40 seconds may have been the trick.

---

<div class="post-metadata">

**Author:** ![Trying\_to\_learn](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/trying_to_learn/32/28400_2.png) [@Trying\_to\_learn](https://discourse.nodered.org/u/Trying_to_learn)\
**Post date:** [16 June 2019 21:34 UTC](https://discourse.nodered.org/t/a-bigger-problem-than-the-ones-i-have-been-mentioning-to-now/12277/11 "2019-06-16T21:34:38Z")

</div>

> [@TotallyInformation](#):
>
> RIP is something different, it lets routers collaborate with each other. Largely discredited now I think due to vulnerabilities.

Thanks.

RIP was only covered and I have not used it on my ..... _lab_ as I haven't seen the benefits of it.  
But, anyway, thanks again.

No prob's on the typo. We all make mistakes. And I should know. 😉

Gotta go now. Monday morning and a new week of fun to be had.  
Back in about 7 hours maybe.