Based on those troubleshooting steps, was nudged here. Has anyone else had this problem with calling the IoTaWatt query interface using http request node?
Can you try going to that URL manually with curl. Eg curl -i http... With the -i to print the header of the response and show us what that contains please.
Yes, here it is from the pi on which node red is running on (latest raspberry os on rpi2b, as of last week). What is interesting is that both the rpi, and my Mac, both imply a problem with the same part of the http: request:
ok - get a query yo know works and paste it into your browser - if that then works copy the address from the browser address bar and do curl -v {pasted url from browser}
(Note not what you type into the browser - -often the browser will re-encode it and add the %5B etc etc - so what it ends up being)
I'm the person who maintains the API that is in question. I've been trying to figure this out and have discovered that this is actually a continuation of this:
That thread starts out looking like something else, but morphs into the real issue. I think the analysis and resolution is flawed, but have the nagging feeling I'm missing something obvious.
Just want to bring in @unixneo, @knolleary, and a couple of others but I hit the new user limit (at Andrei, at Waynedejager if they are still interested.
This thread on the IoTaWatt forum shows how (at)samos has tried using some other APIs on IoTaWatt and they worked fine. He also used a much simpler (no non-URL characters) query API and it failed. The only perceived difference is that the failing requests elicit CHUNKED) responses.
To sum up the previous thread from March on this forum, the Chunk headers and delimiter were interpreted as a malformed response. A "solution" was proposed that essentially used a php script to remove a chunked header and trailer and send the response as if it wasn't chunked. That seemed to work on the 086x length response in question, but the query API can and does regularly send up to 100Kb and is capable of sending megabytes. When it starts, the ultimate length is unknown and so the HTTP/1.1 chunked encoding (sorry, at my newbie link limit so google chunked transfer encoding wiki)
What I can't understand is why node red doesn't seem to recognize chunked responses, and seems to have no hits when I google it. It's been around quite awhile.
I have recreated the node-red issue reported here and tested these various URLs myself. I've changed the chunked response in several ways with no luck. In all cases the responses still work with browsers and the HTTPrequest in various javascript apps.
What I'd like to find out is if there is support for chunked that I'm just missing. It looks as if I could drop down to the tcp in/out level and decode and aggregate the chunks into a string, but that seems foolish when it's such a ubiquitous function.
@overeasy and @knolleary, please see here where a 'very' simple get request response is successfully returned by the IoTaWatt using the http request node, and its node config. does this help?
Chromium on the RPi does exactly the same as FireFox on the Mac (does not re-write the address, and keeps them as [ and ], and correctly returns the csv from the IoTaWatt too
Something odd going on then.. if I go to https://www.bbc.co.uk/news/
using one of our http-request nodes I get a complete page back no problem.
If I do curl -I https://www.bbc.co.uk/news/ I can see the headers are set to chunked.
And that not Content-Length is specified
So the node can handle chunks just fine... that is why I am trying to get you to see the headers returned from that server and in particular both the Transfer-Encoding and the Content-Length (which should not be set if transfer-encoding is set to chunked)
I just tried adding keep-alive. That didn't help. But I have tried something else and have got it to work.
The older post from March had a clue with the curl response that said there was
This is a problem with newer ESP8266WebServer's chunking interacting with IoTaWatt's older implementation. Never bothered browsers but I'll get it fixed in a day or so. After reading through the various releases of ESP8266WebServer, it looks like they have changed it yet again but that version is not yet published to PlatformIO, so I will need to understand the implications of that.
Thanks for your efforts. I'll post back when I have resolved on my end.
It's been interesting to explore Node-Red and I think you'll be seeing more of me in the future. Nice project.
