Hi,
I am new to node-red. I am also a non programmer. I am an engineer and feel relatively comfortable modifying code from others.
I have a number of computing devices at my home
Six PCs (whole family plus personal)
OBI200 for telephony
Two printers
SmartThings hub
HomeServer PC always on
Sonos
Ecobee
8 Google home
Etc
I recently run into an issue. My obi200 was not connecting to one voip server and my wife was unable to dial long distance. God know for how long this was going on
I came across a post about monitor obi200 with nodered. This basically scraps the info from the obi200 status page. It works as designed however it only gave me an overall (obi200 up) but not the status for the individual voip lines listed
I thought it would be good if I can use node-red to send me an email:
If any of the voip servers is down
If my printers are down
If ecobee or sonos is down #2/3 can be achieved from the active status page of my router dd-wrt
However, I find a lot of conflicting and confusing information on how to do this. There are a lot of warnings of how hard it is to parse an html page. The node red script I came across basically does a full search of a string ("Normal (User Mode)") in the whole page to determine whether the obi200 is up (a bit primitive in my opinion)
Please let me know what you think
a. I should continue to try to parse the html into json or something I can refer to
b. I should try text and search (maybe with some clever way to extract portions of the page)
c. I should abandon this, likely lots of sweat and tears for somebody inexperienced as me
there are naturally a few different ways you could tackle this.
Grabbing the status page using an HTTP Request node will get you the content of the page as text. You could then use a Switch node with a rule set to 'msg.payloadcontainsNormal' and a second rule set to Otherwise. That would give you two outputs on the Switch node so you can handle those two cases as you want.
If it needs to do some smarter examination of the page, the built-in HTML node can help. It can use CSS selectors to target and pull-out specific parts of the page based on its structure. That is a pretty robust way of doing it if the page structure is predictable. There are guides online for how to use CSS selectors, so if that isn't something you're familiar with, I don't think its too hard to get to grips with.
The one caveat to all of this is that some web pages use javascript to dynamically generate their content after the browser has done the initial load. They use further HTTP requests under the covers to get more information and insert it into the page. That can make it trickier to grab the page and extract that content; the HTTP Request node would get the initial page - any javascript embedded on the page doesn't get run. In those cases, it is sometimes possible to dig in and identify what additional calls the page is making and to update the HTTP Request node to use those calls instead. One trick to see if this applies to you is, once the page has loaded, right click on it a select 'View Source'. That will give the page as it was when it was loaded - not any generated content since applied. If what you want is there, then you're good to go.
a) You know some HTML
b) The web page is well constructed
That would be my first approach, yes. Unless the device has a usable API.
That is quite fragile and not really recommended. If you can't parse the HTML there's a good chance you won't be able to get this to work well either. However, it is a last resort.
Ah, well, only you can decide that! Actually, with a little work, you might be surprised how easy it is (or you might end up cursing me for the suggestion!).
If you can post up an example of the HTML of the page, I'm sure that some of us would be willing to take a crack at decoding it for you as well.
Thanks, for your patience and help.
This is the page below (I try to carefully delete some personally identifiable information including a whole subsection)
I am able to pull it on red node using this script https://flows.nodered.org/flow/2417af9101b096f64662348750f122eb
As mentioned, this script appears to do a whole page search to determine status for the whole box. I want to determine status for each SP1 to SP4.
I thought I could at least pull out the section for "SP1" below? I tried using the html function with css selector but it doesnt work for me (entered "object", "SP1", and the whole string)
Appreciate assistance. Thanks!
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="default.xsl"?>
System Status
Current WAN Status
__NUMB_ENTRIES__("DHCPOptionNumberOfEntries",0)
Current method to assign an address to WAN interface.
Current IP address assigned to WAN interface
Current subnet mask for WAN interface
Current IP address of the default gateway for WAN interface
IP address of the DNS server for WAN interface
IP address of the alternative DNS servers for WAN interface
Physical address of this interface.
LLDP-MED discovery status
OBiBT Dongle Device Status
OBiBT Dongle Device State
Time duration the dongle will remain discoverable
Current Call State
The OBiBlueTooth Service this dongle binds to
Status of this service
Call state on this line
Status of this service
Call state on this line
Status of this service
Call state on this line
Status of this service
Call state on this line
OBiTALK Service Status
Service State
Current Call State
Thanks, for your patience and help.
This is the page below (I try to carefully delete some personally identifiable information including a whole subsection)
I am able to pull it on red node using this script https://flows.nodered.org/flow/2417af9101b096f64662348750f122eb
As mentioned, this script appears to do a whole page search to determine status for the whole box. I want to determine status for each SP1 to SP4.
I thought I could at least pull out the section for "SP1" below? I tried using the html function with css selector but it doesnt work for me (entered "object", "SP1", and the whole string)
Appreciate assistance. Thanks!
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="default.xsl"?>
<model reboot_req="false" >
<object name="WAN Status" status="true" provision="false" access="readOnly" minEntries="1" maxEntries="1" tr="0" >
<pagetype default="no" ></pagetype>
<pagetitle >
System Status
</pagetitle>
<description >
Current WAN Status
</description>
__NUMB_ENTRIES__("DHCPOptionNumberOfEntries",0)
<parameter name="AddressingType" access="readOnly" >
<description >
Current method to assign an address to WAN interface.
</description>
<syntax >
<string >
<enumeration value="DHCP" ></enumeration>
<enumeration value="Static" ></enumeration>
<enumeration value="PPPoE" ></enumeration>
</string>
</syntax>
<value hash="498fd365" type="list" default="DHCP" current="DHCP" ></value>
</parameter>
<parameter name="IPAddress" access="readOnly" >
<description >
Current IP address assigned to WAN interface
</description>
<syntax >
<string ></string>
</syntax>
<value hash="18c414be" type="input" default=" " current="192.168.1.31" ></value>
</parameter>
<parameter name="SubnetMask" access="readOnly" >
<description >
Current subnet mask for WAN interface
</description>
<syntax >
<string ></string>
</syntax>
<value hash="a85b61dc" type="input" default=" " current="255.255.255.0" ></value>
</parameter>
<parameter name="DefaultGateway" access="readOnly" >
<description >
Current IP address of the default gateway for WAN interface
</description>
<syntax >
<string ></string>
</syntax>
<value hash="78576ff6" type="input" default="" current="192.168.1.1" ></value>
</parameter>
<parameter name="DNSServer1" access="readOnly" >
<description >
IP address of the DNS server for WAN interface
</description>
<syntax >
<string >
<size maxLength="256" ></size>
</string>
</syntax>
<value hash="b67a236c" type="input" default="" current="192.168.1.1" ></value>
</parameter>
<parameter name="DNSServer2" access="readOnly" >
<description >
IP address of the alternative DNS servers for WAN interface
</description>
<syntax >
<string >
<size maxLength="256" ></size>
</string>
</syntax>
<value hash="b67a236d" type="input" default="" current="" ></value>
</parameter>
<parameter name="MACAddress" access="readOnly" >
<description >
Physical address of this interface.
</description>
<syntax >
<string ></string>
</syntax>
<value hash="bf7d3c16" type="input" default="" current="AAAAEF60BBBB" ></value>
</parameter>
<parameter name="LLDP-MEDStatus" access="readOnly" >
<description >
LLDP-MED discovery status
</description>
<syntax >
<string ></string>
</syntax>
<value hash="90dd1eb2" type="input" default="Disabled" ></value>
</parameter>
</object>
<object status="true" name="OBiBT Dongle 1 Status" provision="false" access="readOnly" minEntries="1" maxEntries="1" >
<description >
OBiBT Dongle Device Status
</description>
<parameter name="Status" access="readOnly" >
<description >
OBiBT Dongle Device State
</description>
<syntax >
<string ></string>
</syntax>
<value hash="a3ce8974" type="input" default=" " current="No Dongle" ></value>
</parameter>
<parameter name="Discoverable" access="readOnly" >
<description >
Time duration the dongle will remain discoverable
</description>
<syntax >
<string ></string>
</syntax>
<value hash="d6b3cb23" type="input" default=" " ></value>
</parameter>
<parameter name="CallState" access="readOnly" >
<description >
Current Call State
</description>
<syntax >
<string ></string>
</syntax>
<value hash="6d75f6ad" type="input" default=" " ></value>
</parameter>
<parameter name="BindingService" access="readOnly" >
<description >
The OBiBlueTooth Service this dongle binds to
</description>
<syntax >
<string ></string>
</syntax>
<value hash="417db31c" type="input" default=" " ></value>
</parameter>
</object>
<object status="true" name="OBiBT Dongle 2 Status" provision="false" access="readOnly" minEntries="1" maxEntries="1" >
<parameter name="Status" access="readOnly" >
<syntax >
<string ></string>
</syntax>
<value hash="df0ec315" type="input" default=" " current="No Dongle" ></value>
</parameter>
<parameter name="Discoverable" access="readOnly" >
<syntax >
<string ></string>
</syntax>
<value hash="a2daf984" type="input" default=" " ></value>
</parameter>
<parameter name="CallState" access="readOnly" >
<syntax >
<string ></string>
</syntax>
<value hash="4fd8ae" type="input" default=" " ></value>
</parameter>
<parameter name="BindingService" access="readOnly" >
<syntax >
<string ></string>
</syntax>
<value hash="b429fdbd" type="input" default=" " ></value>
</parameter>
</object>
<object name="SP1 Service Status" access="readOnly" minEntries="0" maxEntries="4" numEntriesParameter="NumberOfLines" enableParameter="Enable" subtitle="SP1 Service Status" status="true" notify="true" >
<parameter name="Status" provision="false" access="readOnly" >
<description >
Status of this service
</description>
<syntax >
<string >
<size maxLength="128" ></size>
</string>
</syntax>
<value hash="f84a7e6" type="input" default="" current="Registered (server=208.65.240.44:5060; expire in 2843s)" ></value>
</parameter>
<parameter name="PrimaryProxyServer" provision="false" access="readOnly" >
<syntax >
<string >
<size maxLength="128" ></size>
</string>
</syntax>
<value hash="d74170df" type="input" default="" ></value>
</parameter>
<parameter name="SecondaryProxyServer" provision="false" access="readOnly" >
<syntax >
<string >
<size maxLength="128" ></size>
</string>
</syntax>
<value hash="c0393e83" type="input" default="" ></value>
</parameter>
<parameter name="CallState" provision="false" access="readOnly" >
<description >
Call state on this line
</description>
<syntax >
<string >
<size maxLength="128" ></size>
</string>
</syntax>
<value hash="d61dd7df" type="input" default="" current="0 Active Calls" ></value>
</parameter>
</object>
<object name="SP2 Service Status" access="readOnly" minEntries="0" maxEntries="4" numEntriesParameter="NumberOfLines" enableParameter="Enable" subtitle="SP2 Service Status" status="true" notify="true" >
<parameter name="Status" provision="false" access="readOnly" >
<description >
Status of this service
</description>
<syntax >
<string >
<size maxLength="128" ></size>
</string>
</syntax>
<value hash="fbc67cc7" type="input" default="" current="Registered (server=77.72.174.128:5060; expire in 34s)" ></value>
</parameter>
<parameter name="PrimaryProxyServer" provision="false" access="readOnly" >
<syntax >
<string >
<size maxLength="128" ></size>
</string>
</syntax>
<value hash="a3689f40" type="input" default="" ></value>
</parameter>
<parameter name="SecondaryProxyServer" provision="false" access="readOnly" >
<syntax >
<string >
<size maxLength="128" ></size>
</string>
</syntax>
<value hash="32e58924" type="input" default="" ></value>
</parameter>
<parameter name="CallState" provision="false" access="readOnly" >
<description >
Call state on this line
</description>
<syntax >
<string >
<size maxLength="128" ></size>
</string>
</syntax>
<value hash="5b7a8d20" type="input" default="" current="0 Active Calls" ></value>
</parameter>
</object>
<object name="SP3 Service Status" access="readOnly" minEntries="0" maxEntries="4" numEntriesParameter="NumberOfLines" enableParameter="Enable" subtitle="SP3 Service Status" status="true" notify="true" >
<parameter name="Status" provision="false" access="readOnly" >
<description >
Status of this service
</description>
<syntax >
<string >
<size maxLength="128" ></size>
</string>
</syntax>
<value hash="e80851a8" type="input" default="" current="Registered (server=94.75.247.45:5060; expire in 53s)" ></value>
</parameter>
<parameter name="PrimaryProxyServer" provision="false" access="readOnly" >
<syntax >
<string >
<size maxLength="128" ></size>
</string>
</syntax>
<value hash="6f8fcda1" type="input" default="" ></value>
</parameter>
<parameter name="SecondaryProxyServer" provision="false" access="readOnly" >
<syntax >
<string >
<size maxLength="128" ></size>
</string>
</syntax>
<value hash="a591d3c5" type="input" default="" ></value>
</parameter>
<parameter name="CallState" provision="false" access="readOnly" >
<description >
Call state on this line
</description>
<syntax >
<string >
<size maxLength="128" ></size>
</string>
</syntax>
<value hash="e0d74261" type="input" default="" current="0 Active Calls" ></value>
</parameter>
</object>
<object name="SP4 Service Status" access="readOnly" minEntries="0" maxEntries="4" numEntriesParameter="NumberOfLines" enableParameter="Enable" subtitle="SP4 Service Status" status="true" notify="true" >
<parameter name="Status" provision="false" access="readOnly" >
<description >
Status of this service
</description>
<syntax >
<string >
<size maxLength="128" ></size>
</string>
</syntax>
<value hash="d44a2689" type="input" default="" current="Connected" ></value>
</parameter>
<parameter name="PrimaryProxyServer" provision="false" access="readOnly" >
<syntax >
<string >
<size maxLength="128" ></size>
</string>
</syntax>
<value hash="3bb6fc02" type="input" default="" ></value>
</parameter>
<parameter name="SecondaryProxyServer" provision="false" access="readOnly" >
<syntax >
<string >
<size maxLength="128" ></size>
</string>
</syntax>
<value hash="183e1e66" type="input" default="" ></value>
</parameter>
<parameter name="CallState" provision="false" access="readOnly" >
<description >
Call state on this line
</description>
<syntax >
<string >
<size maxLength="128" ></size>
</string>
</syntax>
<value hash="6633f7a2" type="input" default="" current="0 Active Calls" ></value>
</parameter>
</object>
<object status="true" name="OBiTALK Service Status" provision="false" access="readOnly" minEntries="1" maxEntries="1" >
<description >
OBiTALK Service Status
</description>
<parameter name="Status" access="readOnly" >
<description >
Service State
</description>
<syntax >
<string ></string>
</syntax>
<value hash="903f7230" type="input" default=" " current="Normal (User Mode) (10);ex-addr=198.66.111.65:10066(10066)" ></value>
</parameter>
<parameter name="CallState" access="readOnly" >
<description >
Current Call State
</description>
<syntax >
<string ></string>
</syntax>
<value hash="c384f5e9" type="input" default=" " current="0 Active Calls" ></value>
</parameter>
</object>
</model>
I basically copied the flow I posted earlier from somebody else. It does have an xml and json nodes. I have to admit, I have no idea what they are used for in this case. In the end it does a search for a string on the whole thing.
Thanks for your help again
So now we know that the output is XML and not HTML. That makes a significant difference and we can just pass the data into the XML node to get a JavaScript object that we can play with.
Thanks. This is great. I was a little confused but I can see now.
I assume I can point to a particular element systematically. I will fiddle with it.
Awesome!
Yes. Sorry, I ran out of time to try and dig down to the correct part of the resulting JSON. I see some JavaScript (function node) or JSONata (change node) in your future! Good hunting.
Thanks
I struggle quite a bit trying to figure out json but I guess still not understanding json pointing very well. I figured (remember reading somewhere) you can copy the path straight from the debugger view. Copy/paste worked well and now I have all four needed items out.
Next I need to modify the logic but likely wont have any problem
I really appreciate your help so far. I will work on this and post my results for others to use. I know enough to be dangerous
I was able to make it work
I have my script to check the obi200 status of individual service providers
I also created a similar one to check my printer status (mostly reuse by accessing and scrapping the device web interface)
I am now trying to do a monitor using ping for devices without an interface
I have mostly copied and modified previous scripts posted. How do I give credit to original script author?
Ok, I got the ping monitor working now
I will clean up over the weekend and post (although I feel some of stuff was straightforward - just challenging for a newbie).
Actually, with a little work, you might be surprised how easy it is (or you might end up cursing me for the suggestion!).
