Reading webserver data

Hi there Node-Red fans,

I have been doing pretty amazing things with node-red the last two years. I am no programmer but i have a fair understanding of the basics.
This challenge is a bit out of my league therefore I need a little guidance.

I am trying to read energy measurements from a logging server:

If I enter the following url into a web browser :

http://192.168.1.10:8080/user1?16&1=GetLogHour&19&100&20&10&

I get the following data:

{"header":["HourLog","","CH1 verbruik 01E00002","Year : 2019","Day Start : 100","Day's Qty : 20","Interval : 10 min","Units : TYPE 0"],
"footer":["Import","Export"],
"body":[["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],
["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],
["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],
["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],
["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],
["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],
["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],
["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],
["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],
["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],
["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],

how to I go about reading this into node-red so I can actually use the data for something useful, like display on a dashboard?
your help will be much appreciated.

Duncan

Try an inject node followed by a http request node and then a debug node

What do you want to display on the dashboard specifically ?

I have tried that and I get this error?

4/28/2019, 10:02:37 PMnode: 3e768908.1db9b6

msg : error

"Error: Parse Error"

4/28/2019, 10:02:37 PMnode: 91fb943f.54cffmsg : Object

object

_msgid: "b1d4eec3.68f8e"

topic: ""

payload: "Error: Parse Error : http://192.168.1.10:8080/user1?16&1=GetLogHour&19&100&20&10&"

statusCode: "HPE_INVALID_CONSTANT"

how have you configured the http request node?

This indicates a different content-type header than expected.
Open the url in the browser, use the web inspector to verify the content-header and match it in node-red with the msg.header option.

Hi bakman2, I don't know exactly what you mean, but this is the reply in the browser?

  1. Request URL:

http://192.168.1.10:8080/user1?64&1=GetLogHour&19&100&20&10&

  1. Request Method:

GET

  1. Status Code:

200 OK

  1. Remote Address:

192.168.1.10:8080

  1. Referrer Policy:

no-referrer-when-downgrade

  1. Request Headersview parsed

  2. GET /user1?64&1=GetLogHour&19&100&20&10& HTTP/1.1 Host: 192.168.1.10:8080 Connection: keep-alive Cache-Control: max-age=0 Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3 Accept-Encoding: gzip, deflate Accept-Language: nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7 Cookie: username=adminIP

  3. Query String Parametersview parsed

  4. 64&1=GetLogHour&19&100&20&10&

What is the application you are contacting?

It may be that it's an old application with a HTTP/0.9 interface. That wouldn't send back HTTP headers, only the raw body content and therefore the Header Parse Error.

Can you run curl -v to that URL in a command line to see what comes back?

Hi Steve, can you be a bit more specific? Where should I try the curl command?

This part is essential. Use a different browser to show these headers, or as @Steve-Mcl says, use

curl -v "http://192.168.1.10:8080/user1?64&1=GetLogHour&19&100&20&10&"

on the commandline (if you are on linux, windows does not have curl installed i believe).

Great, at least we know that, that works. Now how to get it working in node-red?
I appreciate your help.

HTTP/1.1 200 OK

{"header":["HourLog","","CH1 verbruik 01E00002","Year : 2019","Day Start : 100","Day's Qty : 20","Interval : 10 min","Units : TYPE 0"],
"footer":["Import","Export"],
"body":[["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],
["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],
["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],
["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],
["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],
["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],
["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],["3","0"],

I dont think you can use the HTTP Request node as the repsonse from that web server is not valid & therefore you get an HPE (Header Parse Error).

You could use the TCP node to & generate a GET command to IP 192.168.1.10, PORT 8080 & the response you get back would be the JSON as expected.

See this for an idea of how to generate a HTTP GET using TCP (or in this case telnet - same thing)...
https://www.the-art-of-web.com/system/telnet-http11/

Try this flow:

[{"id":"a92cf1.8c434b1","type":"inject","z":"cf07e2a5.1d217","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":264,"y":418,"wires":[["df1b46c2.dc0748"]]},{"id":"df1b46c2.dc0748","type":"exec","z":"cf07e2a5.1d217","command":"curl -v \"http://192.168.1.10:8080/user1?64&1=GetLogHour&19&100&20&10&\"","addpay":false,"append":"","useSpawn":"false","timer":"","oldrc":false,"name":"curl","x":424,"y":418,"wires":[["1f5ac981.6ccdde"],["f17d89c8.de8bf"],["cf10d274.0242e"]]},{"id":"cf10d274.0242e","type":"debug","z":"cf07e2a5.1d217","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":622,"y":462,"wires":[]},{"id":"1f5ac981.6ccdde","type":"debug","z":"cf07e2a5.1d217","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":622,"y":374,"wires":[]},{"id":"f17d89c8.de8bf","type":"debug","z":"cf07e2a5.1d217","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":622,"y":418,"wires":[]}]

note that your output is not a valid header, nor is it valid json as there the ending } are missing.

Hi Bakman2,

Great stuff that works!!! Now I will try to figure out how to get the usable data.
Thanks!!