How long does my IP stay in the router's ARP table?

[SteveMann]

I asked this morning about finding my MAC address in the output of the ARP node:
Find an object in an array of objects
I got a lot of really good suggestions that led me to my solution.

My goal is to detect when my phone or my wife's phone is at home or not. My solution in the above referenced thread is to use the ARP node to get all of the IP and MAC addresses in my router's ARP table, then look for our phones.

I now have a function that does a msg.payload.find() to search the output of the ARP node. Works great. Thanks guys.

Now, how long does my phone's IP and MAC stay in the ARP table? My router manual doesn't say what the ARP timeout is or how to change it.

[edje11]

You can get the default arp cache timeout with this, but for that you need to have ssh access to your router.

cat /proc/sys/net/ipv4/neigh/default/gc_stale_time

Source: https://askubuntu.com/questions/516567/how-long-is-an-arp-entry-cached-for

What brand/model router do you have?

[TotallyInformation]

It isn't just the router that has an arp cache. I've been doing arp checks on my Pi for a few years now to compile a list of recently active devices. However, I'm switching over to nmap which gives some interesting options

That is a uibuilder bootstrap-vue table that uses a combined devices list that nmap updates periodically. It also has manual inputs for descriptive entries and this replaces my old Excel spreadsheet

[SteveMann]

The router is an Actiontec M1424WR GigE Router from Verizon FIOS.

I can't SSH into the router, but I can telnet and open a shell.
cat /proc/sys/net/ipv4/neigh/default/gc_stale_time
returns 60.
So, if I do nothing, in one hour away, the ARP table should forget the device?

I can even cd cat /proc/sys/net/ipv4/neigh/default/, but I can't find an editor. I tried nano, vi and emacs; all not found. Any ideas?
I also tried touch temp to see if I had write privileges, but it looks like I don't. I could try to echo 30 > gc_stale_time - any thoughts? (It's my only router, I don't want to brick it).

[SteveMann]

@TotallyInformation

However, I'm switching over to nmap which gives some interesting options

Sounds safer than messing with my Verizon router.. But I am looking for a solution for Node-Red to be able to track my phones (home or away).

[TotallyInformation]

This question comes up regularly. You need to understand that phones are designed to minimise power consumption and in doing so, they do clever things with their radios. Like turning them off when they can.

So typically, phones actually make quite poor "presence" indicators at the kind of resolution that you might want at home. They will generally appear and disappear quite regularly both on Bluetooth and WiFi. Though that may gradually improve over time as more modern chipsets learn how to stay active while minimising power use (like the new ESP32-S2 for example).

It may be enough for you though, you will need to experiment. Using nmap is actually quite similar to using ARP, both can be set to periodically scan for "alive" devices. My screenshot wasn't the best but if I showed you more, you would see that there are plenty of entries where the updated timestamp is quite current. My NAS has 2 network interfaces and 1 of them isn't currently active. You can also see that nmap has given me a round-trip-time in ms as well which indicates the latency of the devices network card to the nmap scanner (my "new" home server that runs on an old laptop).

I feed that data out to MQTT so I have a periodically updated table - I actually can't remember how fast it is running right now, probably every 5min but you could run it every minute. Maybe more.

[SteveMann]

@TotallyInformation

It may be enough for you though, you will need to experiment. Using nmap is actually quite similar to using ARP, both can be set to periodically scan for "alive" devices.

But, how do you access nmap? There's no nmap node that I can find.

[TotallyInformation]

Well you could run it from an exec node but I prefer to run it from a script that runs from a CRON timer.

#! /usr/bin/env bash
# Fast scan the local network for live devices and record
# to /tmp/nmap.xml which can be used in Node-RED
#
# To run manually:
#   sudo /home/home/nrmain/system/nmap_scan.sh
#
# To run via cron:
#   sudo crontab -e
#       01,16,31,46 * * * * /home/home/nrmain/system/nmap_scan.sh

# Run the scan
nmap -sn --oX /tmp/nmap.xml --privileged -R --system-dns --webxml 192.168.1.0/24
# Make sure ownership & ACLs on the output are secure
chown root:home /tmp/nmap.xml
chmod --silent 640 /tmp/nmap.xml
# Trigger the Node-RED update
#curl  --silent --output /dev/null 'http://localhost:1880/localnetscan' > /dev/null
curl 'http://localhost:1880/localnetscan'

This produces an XML file from the nmap command and then it triggers a flow in Node-RED using curl to an http-in node.

The Node-RED flow reads the XML file, references a global retained variable that is my master table & it updates that via a function node which also sends data out to MQTT (I've not included the mqtt-out node) and also sends to a uibuilder node (not included) on port #1. Port #2 of the function node outputs a copy of the master data to a log file (rather overkill).

[{"id":"e97f9cf0.f86dd","type":"inject","z":"d0860be6.7951b8","name":"","topic":"","payload":"","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":75,"y":320,"wires":[["e16eeebb.792f8"]],"l":false},{"id":"e16eeebb.792f8","type":"file in","z":"d0860be6.7951b8","name":"","filename":"/tmp/nmap.xml","format":"utf8","chunk":false,"sendError":false,"encoding":"none","x":320,"y":320,"wires":[["75a84cfb.fcccb4","d102c6.9932fd38"]]},{"id":"75a84cfb.fcccb4","type":"xml","z":"d0860be6.7951b8","name":"","property":"payload","attr":"","chr":"","x":470,"y":320,"wires":[["35baaad9.7a1f06","9d2dadf4.f2eb5"]]},{"id":"35baaad9.7a1f06","type":"function","z":"d0860be6.7951b8","name":"Update networks global var","func":"const network = global.get('network','file')\n\nconst newnetwork = {}\n\n// arr.forEach(callback(currentValue [, index [, array]])[, thisArg])\nmsg.payload.nmaprun.host.forEach( (host) => {\n    let mac = ''; vendor = ''\n    \n    // Should only be for localhost because nmap doesn't report the mac address for that\n    if (! host.address[1]) {\n        if ( host.address[0].$.addr !== '192.168.1.191' ) {\n            node.warn(host)\n            return\n        } else {\n            mac = '28:D2:44:69:FD:D9'\n            vendor = 'Lenovo'\n        }\n    } else {\n        mac = host.address[1].$.addr.toUpperCase()\n        vendor = host.address[1].$.vendor\n    }\n    \n    // Create the entry if it doesn't exist\n    if ( ! network.hasOwnProperty(mac) ) {\n        // ID = mac for anything coming from nmap\n        network[mac] = { id: mac, 'mac': mac }\n        newnetwork[mac] = host\n    }\n    \n    // Record latest IP address\n    network[mac].ipaddr = host.address[0].$.addr\n    // and vendor\n    network[mac].vendor = vendor\n    // Add a JS timestamp for when this was run \n    network[mac].updated = new Date(Number.parseInt(msg.payload.nmaprun.$.start, 10)*1000).toLocaleString()\n    network[mac].updatedBy = 'nmap'\n    // Add host name if available\n    if ( host.hostnames[0].hostname ) network[mac].host = host.hostnames[0].hostname[0].$.name\n    // Add latency to host if available\n    // srtt is in text in microseconds so convert to milliseconds. /1000 again to get to seconds \n    if ( host.times ) network[mac].srtt = Number.parseInt(host.times[0].$.srtt, 10)/1000\n\n})\n\n// Vuetify Table\n// network.schema = [\n//     {\"value\": \"mac\", text: 'MAC', dataType: 'string', eg: \"D8:6C:63:5E:41:5F\"},\n//     {\"value\": \"ipaddr\", text: 'IP', dataType: 'string', eg: \"192.168.1.1\"},\n//     {\"value\": \"vendor\", text: 'Vendor', dataType: 'string', eg: \"Expressif\"},\n//     {\"value\": \"updated\", text: 'Updated', dataType: 'Date', eg: 1581869007000},\n//     {\"value\": \"updatedBy\", text: 'By', dataType: 'string', eg: 'nmap'},\n//     {\"value\": \"srtt\", text: 'Latency', dataType: 'integer', eg: 97.269},\n//     {\"value\": \"host\", text: 'Host Name', dataType: 'string', eg: 'pi2.knightnet.co.uk'},\n//     {\"value\": \"hostname\", text: 'Host Descr', dataType: 'string', eg: 'D1M04 - Access Point'},\n//     {\"value\": \"dhcp\", text: 'DHCP', dataType: 'string', eg: 'Yes-Fixed'},\n//     {\"value\": \"description\", text: 'Description', dataType: 'string', eg: \"Some Text\"}\n// ]\n\n/** bootstrap-vue table format. https://bootstrap-vue.js.org/docs/components/table#field-definition-reference\n * BV native: key, label, headerTitle\n * uibuilder: primary, dataType, format, editable, eg\n * SEE COMMENT FOR DETAILS\n **/\n//network.schema = [\nglobal.set('schemas.network', {\n    id         : {\"key\": \"id\", primary: true, label: 'ID', dataType: 'string', \n                    thClass: 'd-none', tdClass: 'd-none', // Don't display this field in tables\n                    eg: \"D8:6C:63:5E:41:5F\"},\n    mac        : {\"key\": \"mac\", label: 'MAC', dataType: 'string', \n                    eg: \"D8:6C:63:5E:41:5F\"},\n    ipaddr     : {\"key\": \"ipaddr\", label: 'IP', dataType: 'text', eg: \"192.168.1.1\"},\n    vendor     : {\"key\": \"vendor\", label: 'Vendor', dataType: 'text', eg: \"Expressif\"},\n    updated    : {\"key\": \"updated\", label: 'Updated', dataType: 'date', eg: 1581869007000},\n    updatedBy  : {\"key\": \"updatedBy\", label: 'By', dataType: 'text', eg: 'nmap'},\n    srtt       : {\"key\": \"srtt\", label: 'Latency', dataType: 'number', eg: 97.269},\n    host       : {\"key\": \"host\", label: 'Host Name', dataType: 'text', eg: 'pi2.knightnet.co.uk'},\n    hostname   : {\"key\": \"hostname\", label: 'Host Descr', dataType: 'text', editable: true,\n                    description: 'Device Description', eg: 'D1M04 - Access Point'},\n    dhcp       : {\"key\": \"dhcp\", label: 'DHCP', dataType: 'text', editable: true, \n                    description: 'Is IP address set from DHCP?', eg: 'Yes-Fixed'},\n    description: {\"key\": \"description\", label: 'Description', dataType: 'text', editable: true, \n                    eg: \"Some Text\"},\n}, 'file')\n\nglobal.set('network', network, 'file')\n\nreturn [{topic: 'network', payload: network}, {topic: 'new-network', payload: newnetwork}];\n\n/*\n(\n    // Look for a mac address in reference array, \n     * return the array object that matches \n    $a := function($mac) {\n        $filter($.other, function($val, $i, $array) {\n            // Filter function to match mac addresses \n            $val.mac.$lowercase() = $mac\n        })\n    };\n   \n    // Flatten the input data\n     * Data comes from nmap: sudo nmap -sn --oX nmap.xml --privileged -R --system-dns --webxml 192.168.1.0/24\n     \n    payload.nmaprun.host.[(\n        // Grab matching mac address data from lookup array \n        $x := $a(address[1].\"$\".addr.$lowercase());\n        /* Build the output \n        {\n           \"IP\": address[0].\"$\".addr,\n           \"MAC\": address[1].\"$\".addr.$uppercase(),\n           \"vendor\": address[1].\"$\".vendor,\n           \"hostname\": hostnames.hostname.\"$\".name,\n           // srtt is in microseconds so convert to milliseconds. /1000 again to get to seconds \n           \"rtt\": $parseInteger(times.\"$\".srtt, '#0')/1000,\n           \"description\": $x.description,\n           \"name\": $x.hostname,\n           \"dhcp\": $x.dhcp,\n           \"refIP\": address[0].\"$\".addr = $x.ipaddr ? null : $x.ipaddr,\n            // Add a JS timestamp for when this was run \n            \"timestamp\": $parseInteger($$.payload.nmaprun.\"$\".start, '#0')*1000,\n           \"y\": $x\n        }\n    )][0]\n)\n*/","outputs":2,"noerr":0,"x":680,"y":320,"wires":[["99ae73de.4727a","965955f1.964818","36544259.7c139e"],["5d34242a.d5290c"]],"inputLabels":["nmap XML converted to JSON"],"outputLabels":["Device master","Newly added devices"],"info":"## Data Schema\n\nObject of objects. Each object's property is a row column.\n\nIf passed as an object of object, MUST be turned into an array for sending to bootstrap-vue's b-table component.\n\nThe column/field schema is written to the global variable: `schema.devices`.\n\n## Column Schema\n\nObject of objects, 1 row for each column in the data array\n\n### bootstrap-vue related\n\n* `key` {string}: column ID\n* `label` {string}: Used in table and edit dialog \n* `headerTitle` {string}: Added to each columns title attrib in the col headers\n\n### uibuilder specific\n\n* `id` {string}: Unique identifier for the row\n* `primary` {boolean}: If true, this column is the primary key that uniquely identifies a data table row\n* `dataType` {string}: HTML 5 `<input>` tag type attribute.\n* `editable` {boolean}: Is this column editable?\n* `eg` {string}: Gives an example of the data in this column\n* `format`\n* `description`\n"},{"id":"99ae73de.4727a","type":"debug","z":"d0860be6.7951b8","name":"Network Device Count","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"$count($keys(payload))","targetType":"jsonata","x":1111,"y":340,"wires":[]},{"id":"a4b7b52c.4cc688","type":"debug","z":"d0860be6.7951b8","name":"New Network Devices","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":1120,"y":440,"wires":[]},{"id":"965955f1.964818","type":"link out","z":"d0860be6.7951b8","name":"network-out","links":["54ffbc11.b95ab4"],"x":930,"y":340,"wires":[],"inputLabels":["devices object"],"l":true},{"id":"36544259.7c139e","type":"split","z":"d0860be6.7951b8","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":910,"y":280,"wires":[["23cf5dd4.ee91f2"]],"info":"Split the devices object into a msg for each device for sending to MQTT."},{"id":"bf65f0bf.66113","type":"comment","z":"d0860be6.7951b8","name":"Update Current Network List","info":"Updated by a cron job to run:\n\n    sudo /home/home/nrmain/system/nmap_scan.sh\n    \nSee `sudo crontab -l` for details","x":160,"y":240,"wires":[]},{"id":"185ba1a6.6563de","type":"http in","z":"d0860be6.7951b8","name":"","url":"/localnetscan","method":"get","upload":false,"swaggerDoc":"","x":130,"y":280,"wires":[["e16eeebb.792f8","740868b4.37e748"]],"info":"API to receive data from executions of imapfilter running on the same device"},{"id":"740868b4.37e748","type":"http response","z":"d0860be6.7951b8","name":"","statusCode":"","headers":{},"x":310,"y":280,"wires":[]},{"id":"23cf5dd4.ee91f2","type":"change","z":"d0860be6.7951b8","name":"topic & retain","rules":[{"t":"set","p":"topic","pt":"msg","to":"'network/'&payload.id","tot":"jsonata"},{"t":"set","p":"retain","pt":"msg","to":"true","tot":"bool"}],"action":"","property":"","from":"","to":"","reg":false,"x":1050,"y":280,"wires":[["5d9cdc1.68ae724"]]},{"id":"5d34242a.d5290c","type":"file","z":"d0860be6.7951b8","name":"","filename":"/home/home/nrmain/data/newnetworkdevices.log","appendNewline":true,"createDir":false,"overwriteFile":"false","encoding":"none","x":1050,"y":400,"wires":[["a4b7b52c.4cc688"]]}]

This is a slightly complex way of doing things since you could just call the nmap command from an exec node driven by a repeating trigger node. However, I have a couple of scripts that I prefer to run outside Node-RED for performance and reliability & like the fact that you can then interface back into Node-RED.

In a similar way, I also run some IMAPFILTER scripts against my email accounts that automatically categorise/file emails and get rid of spam. IMAPFILTER uses a LUA script and this has the ability to output the stats data to a curl command - in this case, the temporary file is avoided altogether:

    -- ...
    -- Send to Node-RED via simple http-in/-out nodes
    output, tblResult = osExecute('curl "http://localhost:1880/imapfilter?' ..
        'account=' .. acct ..
        '&removedFromInbox=' .. (sIbTotal - eIbTotal) ..
        '&unreadInbox=' .. eIbUnseen ..
        '" --silent --connect-timeout 0.1 --max-time 0.5')

    print('OUTPUT: ', output, 'RESULT: ', dump(tblResult))

I'm sure that others can think of more ways to achieve the same ends - Node-RED is nothing if not flexible.

[SteveMann]

Thanks for introducing me to nmap. I hadn't heard of this before.
It will take me a while to fully grok your cron job and what nmap can do for me.

[cinhcet]

I recently experimented with a different way of presence detection via phone using an automation app like tasker or automate (free tasker alternative for android)
As soon as my phone gets connected to a specific wireless network, it will make an http request to an endpoint in Node-RED.
Something similar happens if the wireless gets disconnected and you are connected to the mobile network, the wifi gets deactivated (saves battery) and a message is sent to Node-RED as well.
The last part is the "not so nice" part, since you need VPN or another way of sending a message securely over the mobile network to your local Node-RED.
At least absence detection works pretty reliable that way and does not suffer from battery saving mode.

[Colin]

If you do it the other way round and use ping in node-red to ping the phone then you don't need to do anything special on the phone, except give it a fixed ip address.

[ristomatti]

Wouldn't this approach have the same issues with phone power saving mode as the ARP approach? I doubt the phone will answer to pings either if its WiFi is sleeping.

[Colin]

@ristomatti

Wouldn't this approach have the same issues with phone power saving mode as the ARP approach

Yes, I said that in an early post, but so, presumably does the method proposed by @cinhcet. Better than ARP I think, though, as ARP itself has issues with reliability, in my experience, particularly with it still showing presence when it should not.

[cinhcet]

The power saving is not an issue with my approach (at least to detect if you move away from your home)

Edit: overcoming the power saving issue was the reason for experimenting with that... Because after updating to a new phone, the arp solution I was using before was basically broken completely

[Colin]

What about on arrival? That is the one I had most difficulty with. Seeing when I arrived.

[cinhcet]

that sometimes can take a few minutes

However, for me that is not a big deal, since I disable wifi anyway when I am not at home on my phone, so manual intervention is necessary to turn it on. Then it is 100% reliable.

For me absence detection was more important, which due to battery saving had false positives quite often with arp

[SteveMann]

In summary....

Thanks for all the education. ARP, exec node, nmap, etc. I had not used any of those before and now I know just enough to be dangerous.

Conclusion- IP is reliable for "home" or "away", IF the WiFi on the phone is live. Until this experience, I was unaware that either of our phones turned off WiFi to save battery. Turns out that both do.

SO, I am designing a simple device with an ESP-8266-01 that I can wire into our cars. If they are in the garage, they connect to the WiFi. I.E., Home. I'll probably put it into deep-sleep and wake every minute to update the status via MQTT.

[TotallyInformation]

@SteveMann

SO, I am designing a simple device with an ESP-8266-01 that I can wire into our cars. If they are in the garage, they connect to the WiFi. I.E., Home. I'll probably put it into deep-sleep and wake every minute to update the status via MQTT.

A dedicated device is certainly the most reliable way to go. There are also options for other radio types if Wi-Fi doesn't have the range you need. 433MHz is generally very reliable and, with care, can have better range and lower power needs. But the ESP's are ubiquitous and easy to program.

Check out the new ESP32-S2's if you can source them at a reasonable price (and don't mind using the Espressive tooling rather than Arduino which I think has limited support right now). Their Wi-Fi is new and supports always-on even in light power-save mode. It also supports "high accuracy" distance detection from the access point I believe. All rather new though so probably only for more experienced users right now.

[SteveMann]

I'll probably do something in the next week using the ESP8266-01 (I have a bunch of them). I do have to worry about running the battery down in my Jeep because I only drive it a couple of times a month.