Announce node-red-contrib-http-logger

Share Your Nodes
1

[EDIT 22/10/2018: the title has been changed to node-red-contrib-http-logger, since we agreed in the discussion below that node-red-contrib-http-listener would be confusing ...

Hi folks,

From time to time we get questions on this forum similar to this one:

I can connect with my browser to my IP camera without problems. However when I use the same URL / username / password in Node-RED, this doesn't work.

Since Node-RED gets a difference response from the camera, it means that Node-RED sends a different request to the camera. But what is the difference between the request from Node-RED and the request from the browser, which causes the failure ...

The user can easily see the browser request in his browser developer tools, but the Node-RED requests will be created by NodeJS behind the scenes (invisible for the user).

Last week somebody asked me such a question, so I had only a limited set of options:

  1. Start guessing what Node-RED is requesting. Not my cup of tea ...
  2. Ask the user to install some third party tool (e.g. Wireshark ...), but less-technical users would interpret this as 'Bugger off and leave me alone ...'.
  3. Ask the user to make the IP camera public on his router, and debug remotely to determine what is going on.
  4. Find a way to let the user intercept the http(s) requests and responses via Node-RED (without the need of using extra third-party tools).

I prefer option 4, which has resulted in the new node-red-contrib-http-listener node. Until it is published on NPM, you can install it directly from Github:

npm install bartbutenaers/node-red-contrib-http-listener

Here is a short demo to give a first impression:

http_listener
http_listener781Ă—408 386 KB

I have some questions, so all 'constructive' feedback is very welcome!

  1. The underlying request emits a 'success' or an 'error' event. Currently I generate output messages with two different topics ('success' or 'error'). Or should I use two outputs on my node, one for success requests and one for error requests. Or should I use another solution ...

  2. The user can limit the number of output messages by filtering on URL (see readme for example). Currently if you pass as filter 'abc' all requests whose URL contains this text will be passed. For example _´http://www.abcdef.com:1234/somepath´_ will be passed. Did it this way because I assume that regular expressions are a bit overwhelming for some users. Any other suggestions?

    PS. I want to do the filtering INSIDE this node (instead of adding an extra filter node BEHIND it) to avoid overloading Node-RED with too much messages...

  3. Currently I don't see a way to get the raw request, as required in some other posts (e.g. this one). If somebody has an idea how to accomplish that, please let me know so I could add an extra option to this node.

    P.S. I use under the hood the global-request-logger module, which replaces the standard NodeJs request method by its own wrapper:

    http.request = attachLoggersToRequest.bind(http, 'http');

    And that function calls in turn NodeJs's original request function:

    function attachLoggersToRequest(protocol, options, callback) {
   let self = this;
   let req = ORIGINALS[protocol].request.call(self, options, callback);

    So perhaps we could intercept somewhere in this area the raw request. Had no time to investigate this in more detail ...

  4. I noticed that sometimes not all requests are displayed. E.g. when setImmediately=false in the HttpRequest node (see explanation from Nick), then only the first request is displayed (which is responded by 'unauthorised'). However the second request is not displayed. Will need to investigate that once I'm retired...

  5. Other ideas or suggestions??

Thanks !
Bart

2 Likes
2

Sorry Bart, but how is this different to the existing HTTP In node? That lets you see the full details of an http request sent to the node. You can filter what requests it receives by configuring the path.

I'm must be missing what this node is adding. Are there additional debugging capabilities that could be incorporated into the regular node?

Nick

3

Ah, does it intercept the requests being made by the http request node and let you log them? I guess I got confused by the name ... Http listener to me sounds like something listening for inbound requests.

4

Yes Nick,
It logs all outgoing http requests and also the corresponding response (it gets for those requests).
So it logs all http requests: those from your HttpRequest node, from my node-red-contrib-mulitpart-stream-decoder node, ...

I think this new contribution can be useful for troubleshooting:
Last week the user could connect to his IP camera from his browser, but not from the HttpRequest node.
By using this node I saw quickly that the browser used digest authentication, while the HttpRequest node used basic authentication. That was the reason for my 'digest authentication' feature request yesterday ...

Have been thinking about a name for my node, and thought 'listener' was the best one.
But indeed it might be confusing ... Some other names I have been thinking about:

  • node-red-contrib-http-intercept
  • node-red-contrib-http-mitm (man in the middle): but don't think this is good because you cannot alter the request with my node ...
  • node-red-contrib-http-traffic
  • others ???
1 Like
5

Or perhaps better if 'outbound' is included in the name...

6

Once you add this node to your flows, does it then always intercept? Or only if you turn it on? If the former, you probably need to make sure that the help text warns people of this so that they remove it when finished with.

Also, I assume it intercepts HTTPS traffic as well as HTTP?

In this case, I would think not. As the purpose of this node is to intercept traffic, it seems sensible to me to keep both error and success output on a single channel.

I think we decided this wasn't currently available in Node-RED though there was some debate whether the underlying ExpressJS still exposed the raw data.

Yes, I think you are right. Quite a limited use case but really useful when you need it.

Would make the name rather long :slight_smile:

"http-intercept" sounds OK to me?

7

Interesting node.

Names are important.
I would vote for something like node-red-contribute-http-logger (note also that the main Node.js module you are using is also called global-request-logger)

1 Like
8

Hi guys,

Thanks again for the feedback. Summarized:

  • I also prefer the name node-red-contrib-http-logger since we are only logging the requests.
  • This node intercepts both http and https. In this context the http in the node name might be a bit confusing. I will emphasize this both in the readme and in the info panel.
  • This node will start intercepting as soon as at least one node is started (i.e. the standard NodeJs function will be replaced by an intercepting function). And as soon as all nodes have stopped, the intercepting will be stopped also (i.e. the intercepting function will be replaced again by the standard NodeJs function). This way we have the least impact on e.g. performance, when we aren't interested in listening.

If no arguments against the new node name, I will rename the repository on Github and publish it on NPM ...
Bart

2 Likes
9

You can also rename this thread

10

Hi guys,

  • The node has been renamed to node-red-contrib-http-logger

  • The title of this thread has been updated

  • Version 1.0.1 has been published on NPM. @dceejay: do you need to do something manually to let it appear in the palette (or is that npm bug fixed already) ?

  • Besides the readme page, I have also updated the node label and info tab (to reflect the fact that both HTTP and HTTPS are supported):

    image

Have fun with it !!
Bart

3 Likes
11

Hello @BartButenaers.

Did you find any issue using http request with api-key headers against http-logger? Do you have a sample project/flow with http-logger and http request fully integrated?

I am having issues making the http-logger work and want to see a working project.

Thanks in advance.
Olivia

12

Hi @osalonga,

Not really. There are probably a million of things you can send in a http request. Didn't have time to test them all :wink:

Did a quick search, and found a sample request for an online api-key tester.

Here is some example flow to test that sample request:

image

[{"id":"f57450ea.ac8b","type":"http-logger","z":"5a89baed.89e9c4","name":"","filter":"api.classmarker.com","x":400,"y":980,"wires":[["f164a8b7.74cde8"]]},{"id":"1f25cbf8.71f354","type":"http request","z":"5a89baed.89e9c4","name":"","method":"GET","ret":"txt","paytoqs":false,"url":"https://api.classmarker.com/v1.json?api_key=d4tsE7SvEgzAKlJPFrlvAz3oe9uFQnxy&signature=4495a14efc483aa5ee2f6d4cd480f968&timestamp=1335783600","tls":"","proxy":"","authType":"basic","x":390,"y":900,"wires":[["9f433261.35598"]]},{"id":"cedfb31a.2f4ee","type":"inject","z":"5a89baed.89e9c4","name":"Send request","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":210,"y":900,"wires":[["1f25cbf8.71f354"]]},{"id":"9f433261.35598","type":"debug","z":"5a89baed.89e9c4","name":"Http response","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":600,"y":900,"wires":[]},{"id":"85b98128.01148","type":"inject","z":"5a89baed.89e9c4","name":"Start logging","topic":"","payload":"true","payloadType":"bool","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":210,"y":980,"wires":[["f57450ea.ac8b"]]},{"id":"8b0fc961.02b618","type":"inject","z":"5a89baed.89e9c4","name":"Stop logging","topic":"","payload":"false","payloadType":"bool","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":210,"y":1040,"wires":[["f57450ea.ac8b"]]},{"id":"f164a8b7.74cde8","type":"debug","z":"5a89baed.89e9c4","name":"Intercepted http request","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":630,"y":980,"wires":[]}]

  1. In the http-request node (or any other node that sends a http request) I add the sample request URL.

  2. In the http-logger node I specify as filter a part of that same URL (e.g. the hostname part), to make sure that only http requests containing that part are being logged. Otherwise you will be floaded with all kind of http requests that are being processed in your NodeJs instance ...

  3. Start the http-logger, so it's status will change to "listening".

  4. Then send the http request (which you want to analyze).

  5. On the output of the http-request node you will see the http response:

    image

  6. On the output of the http-logger node you will see both the original http request and the corresponding http response:

    image

    You can see here the real request that will be send by NodeJs. This hopefully gives you a clue what is going wrong in the http-request node part of your flow.

When this explanation is clear enough for you, please let me know and then I will add it to the readme page of the http-logger node.

Bart

13

@BartButenaers Thank you very much for replying.

I copied and executed the flow you posted above. 'Intercepted http request' node did not produce any debug log. However, 'Http Response' node produced an output debug log.

Regards,
Olivia

14

@osalonga,
Could you please have a look at your Node-RED startup log, and send me the Node-RED and NodeJs version numbers you are using. Perhaps the global-request-logger library (that I'm using behind the scenes) doesn't function well anymore on the newer versions. Unfortunately that library doesn't seems to be maintained anymore ...

15

As requested:
Node-RED version: v0.20.3
Node.js version: v10.15.3

16

I have the same problem.

8 Apr 18:38:39 - [info] Node-RED version: v0.20.0
8 Apr 18:38:39 - [info] Node.js version: v10.15.1
8 Apr 18:38:39 - [info] Windows_NT 10.0.17763 x64 LE

17

Hi guys (@TotallyInformation, @osalonga),

Had indeed the same problem on a recent NodeJs/Node-RED version...

  • Works fine when I do a simple http call.
  • But when I use the httprequest-node (which switched recently to the request library) then I never arrive in the handler of the global-request-logger library. So no output message is send ...

Since that library isn't maintained anymore, I tried to find an alternative. The only one that seems usable and well maintained is mitm (Man In The Middle).

So I have rewritten my http-logger node to use mitm, but I'm having also troubles with:

  • The handler is called only at the request, but not at the response. So I have no response data in my output message. I think I can get around that by adding my own handlers to the response (similar to the global-request-logger).
  • I need to get only the relevant data from the request and response (similar to the global-request-logger), because I don't want the entire request and response objects to become available via my output message. Don't want people to start messing around with those objects.
  • When I use the httprequest-node, I get a timeout for every request as soon as mitm is enabled. I have created an issue for this, and hopefully one of the mitm contributors can guide me through it.

As long as the last issue isn't solved, unfortunately I cannot offer a solution ...
Unless anybody knows another library that I could use ...

18

@TotallyInformation, @osalonga,

I have put a new version (2.0.0) on Github which is build on top of the mitm library. You can install it like this:

npm install bartbutenaers/node-red-contrib-http-logger

P.S. The readme file now contains an example flow to demonstrate how multiple http-logger nodes can be working simultaneously in the same Node-RED flow...

Was not an easy task to get it up and running, but got very usefull feedback from Andri Möll (the author of that library). I would appreciate if you guys could test whether it works now, before I publish the new version on NPM...

Thanks !!!

19

Hi Bart, the basics seem to work fine. Not tried anything complex such as setting a proxy.

1 Like
20

I found some issues:
(1) api key or auth headers got lost or over written before the endpoint is fired up.
(2) the request payload in json format is transformed to string data type.

@Bart, Thanks on working on this.