Http request (open connection)

Hi I really need some help with this one. I have a security system I want to do a http request to. It's some kind of a connection that stays open and update the events if I use for example chrome. It's an xml document. I have tried the regular http request node but it does not connect at all. I tried the multipart decoder and it succseeded to get the latest event and then the the node says (no multipart url) and it don't get any more updates if I don't inject it again. Hope you understand what I'm trying to accomplish.

/Niclas

I guess websockets or ajax is used to update the client web page but it not possible to know without more info. Otherwise how would the server be able to send events back just with a http/https connection?

Hi Niclas,
My multipart decoder node sees that the response content-type is not multipart, so it will stop. My node-red-contrib-sse-client will most probably also not work, since your response content type doesn't contain "event-stream".
Do you have any extra info about the type of connection?
Bart

Hi Bart. I attach som images from the api documentation and a snippet from Wireshark.
The request url is http://192.168.80.121/arx/eventexport?end_date=keep.

Best regards Niclas

Niclas,
Just guessing based on this:

image

This is chunked transfer encoding which is the default encoding. The http-request node will (see code) wait until all chunks have arrived (via the "request" npm library), and then send the entire response body as an output message. But could it perhaps be that your system keeps sending chunks, so there is no end of the response. Which means that the http request node never sends an output message... Could it work like that? I mean that every chunk is in your case in fact a complete piece of data.

If so, the second example here could perhaps be of any help to you. Suppose you rewrite it a bit like this (out of my head, so not tested!):

const request = require('request')
request(
    { method: 'GET'
    , uri: 'http://192.168.80.121/arx/eventexport?end_date=keep'
    , gzip: true
}
, function (error, response, body) {
      console.log('Most probably the stream has been ended ...')
    }
  )
  .on('data', function(data) {
    // decompressed data as it is received
    console.log('decompressed chunk arrived: ' + data)
    node.send({payload: data});
})

The .on('data ...' function should be called every time a chunk arrives.

When you use this in a function node, don't forget to add the request library to the settings.js file (as described here). See also this discussion...

Thank you Bart. I think I need to spend some time reading now and try to understand how to add the library.

Many Thanks

Niclas,
Since I have no clue whether this will work, I will try to minimize the effort you spend to it...

  1. You need to add one line to your settings.js file (and restart Node-RED):

    image

    Note: this allows you to use the 'request' NPM package inside a function node using var request = global.get('request');.

  2. Import this test flow:

    image

    [{"id":"95326d57.32d04","type":"function","z":"97e84354.4bf71","name":"Get chunks","func":"var request = global.get('request');\n\nrequest({ \n    method: 'GET',\n    uri: 'https://jigsaw.w3.org/HTTP/ChunkedScript',\n    //uri: 'http://192.168.80.121/arx/eventexport?end_date=keep'\n    gzip: true\n},\nfunction (error, response, body) {\n      console.log('Most probably the stream has been ended ...')\n    }\n  )\n  .on('data', function(data) {\n    // decompressed data as it is received\n    console.log('decompressed chunk arrived: ' + data)\n    node.send({payload: data});\n})","outputs":1,"noerr":0,"initialize":"","finalize":"","x":430,"y":1080,"wires":[["4bb80b80.0ce0b4"]]},{"id":"f43d55de.a74158","type":"inject","z":"97e84354.4bf71","name":"Start streaming","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":210,"y":1080,"wires":[["95326d57.32d04"]]},{"id":"4bb80b80.0ce0b4","type":"debug","z":"97e84354.4bf71","name":"Display chunks","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":650,"y":1080,"wires":[]}]
    
  3. Inside the function node I call the first http1.1 test url from this page:

    image

    That url returns a chuncked encode response:

    image

    P.S. Afterwards you can uncomment your url inside the function node ...

  4. When you trigger the function node, you will see now multiple entries in the debug sidebar:

    So instead of waiting for the entire response to arrive, the function node will send an output message every time a chunk is received. That is why you get multiple output messages...

Hopefully this way you can get some data from your server...

2 Likes

Hi Bart.

Works great!! Can't thank you enough.
I added a function to convert the payload to string "msg.payload = msg.payload.toString('utf8');"
The output is now just plain text. I Do you think there is a way to get this as an xml or something that's easier to work with or do I need to use regex or jonata?

Best regard Niclas

Most probably others in this community will be able to give you a better answer.

But personally I wouldn't use xml, and convert it instead to json. Since json and jsonata is the standard in Node-RED, you will be able to use your data in lots of other nodes...

For example here you can find an example of the xml2js npm library.
Don't forget to add that also to your settings.js file:

    functionGlobalContext: {
        // os:require('os'),
        // jfive:require("johnny-five"),
        // j5board:require("johnny-five").Board({repl:false})
        request:require('request'),
        xml2js:require('xml2js')
    },

And add this line at the top of your function node:

var xml2js = global.get('xml2js');

But there might be better or easier ways to do this, I assume ...

.... like using the xml node (that uses xml2js)

1 Like

Or simply follow this: https://cookbook.nodered.org/basic/convert-xml

Damn, Dave was first :wink:
Don't add it to your function node, if there is already another node that does the job very well ...

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.