Announce node-red-contrib-http-logger


#1

[EDIT 22/10/2018: the title has been changed to node-red-contrib-http-logger, since we agreed in the discussion below that node-red-contrib-http-listener would be confusing ...

Hi folks,

From time to time we get questions on this forum similar to this one:

I can connect with my browser to my IP camera without problems. However when I use the same URL / username / password in Node-RED, this doesn't work.

Since Node-RED gets a difference response from the camera, it means that Node-RED sends a different request to the camera. But what is the difference between the request from Node-RED and the request from the browser, which causes the failure ...

The user can easily see the browser request in his browser developer tools, but the Node-RED requests will be created by NodeJS behind the scenes (invisible for the user).

Last week somebody asked me such a question, so I had only a limited set of options:

  1. Start guessing what Node-RED is requesting. Not my cup of tea ...
  2. Ask the user to install some third party tool (e.g. Wireshark ...), but less-technical users would interpret this as 'Bugger off and leave me alone ...'.
  3. Ask the user to make the IP camera public on his router, and debug remotely to determine what is going on.
  4. Find a way to let the user intercept the http(s) requests and responses via Node-RED (without the need of using extra third-party tools).

I prefer option 4, which has resulted in the new node-red-contrib-http-listener node. Until it is published on NPM, you can install it directly from Github:

npm install bartbutenaers/node-red-contrib-http-listener

Here is a short demo to give a first impression:

I have some questions, so all 'constructive' feedback is very welcome!

  1. The underlying request emits a 'success' or an 'error' event. Currently I generate output messages with two different topics ('success' or 'error'). Or should I use two outputs on my node, one for success requests and one for error requests. Or should I use another solution ...

  2. The user can limit the number of output messages by filtering on URL (see readme for example). Currently if you pass as filter 'abc' all requests whose URL contains this text will be passed. For example ´http://www.abcdef.com:1234/somepath´ will be passed. Did it this way because I assume that regular expressions are a bit overwhelming for some users. Any other suggestions?

    PS. I want to do the filtering INSIDE this node (instead of adding an extra filter node BEHIND it) to avoid overloading Node-RED with too much messages...

  3. Currently I don't see a way to get the raw request, as required in some other posts (e.g. this one). If somebody has an idea how to accomplish that, please let me know so I could add an extra option to this node.

    P.S. I use under the hood the global-request-logger module, which replaces the standard NodeJs request method by its own wrapper:

    http.request = attachLoggersToRequest.bind(http, 'http');
    

    And that function calls in turn NodeJs's original request function:

    function attachLoggersToRequest(protocol, options, callback) {
       let self = this;
       let req = ORIGINALS[protocol].request.call(self, options, callback);
    

    So perhaps we could intercept somewhere in this area the raw request. Had no time to investigate this in more detail ...

  4. I noticed that sometimes not all requests are displayed. E.g. when setImmediately=false in the HttpRequest node (see explanation from Nick), then only the first request is displayed (which is responded by 'unauthorised'). However the second request is not displayed. Will need to investigate that once I'm retired...

  5. Other ideas or suggestions??

Thanks !
Bart


#2

Sorry Bart, but how is this different to the existing HTTP In node? That lets you see the full details of an http request sent to the node. You can filter what requests it receives by configuring the path.

I'm must be missing what this node is adding. Are there additional debugging capabilities that could be incorporated into the regular node?

Nick


#3

Ah, does it intercept the requests being made by the http request node and let you log them? I guess I got confused by the name ... Http listener to me sounds like something listening for inbound requests.


#4

Yes Nick,
It logs all outgoing http requests and also the corresponding response (it gets for those requests).
So it logs all http requests: those from your HttpRequest node, from my node-red-contrib-mulitpart-stream-decoder node, ...

I think this new contribution can be useful for troubleshooting:
Last week the user could connect to his IP camera from his browser, but not from the HttpRequest node.
By using this node I saw quickly that the browser used digest authentication, while the HttpRequest node used basic authentication. That was the reason for my 'digest authentication' feature request yesterday ...

Have been thinking about a name for my node, and thought 'listener' was the best one.
But indeed it might be confusing ... Some other names I have been thinking about:

  • node-red-contrib-http-intercept
  • node-red-contrib-http-mitm (man in the middle): but don't think this is good because you cannot alter the request with my node ...
  • node-red-contrib-http-traffic
  • others ???

#5

Or perhaps better if 'outbound' is included in the name...


#6

Once you add this node to your flows, does it then always intercept? Or only if you turn it on? If the former, you probably need to make sure that the help text warns people of this so that they remove it when finished with.

Also, I assume it intercepts HTTPS traffic as well as HTTP?

In this case, I would think not. As the purpose of this node is to intercept traffic, it seems sensible to me to keep both error and success output on a single channel.

I think we decided this wasn't currently available in Node-RED though there was some debate whether the underlying ExpressJS still exposed the raw data.

Yes, I think you are right. Quite a limited use case but really useful when you need it.

Would make the name rather long :slight_smile:

"http-intercept" sounds OK to me?


#7

Interesting node.

Names are important.
I would vote for something like node-red-contribute-http-logger (note also that the main Node.js module you are using is also called global-request-logger)


#8

Hi guys,

Thanks again for the feedback. Summarized:

  • I also prefer the name node-red-contrib-http-logger since we are only logging the requests.
  • This node intercepts both http and https. In this context the http in the node name might be a bit confusing. I will emphasize this both in the readme and in the info panel.
  • This node will start intercepting as soon as at least one node is started (i.e. the standard NodeJs function will be replaced by an intercepting function). And as soon as all nodes have stopped, the intercepting will be stopped also (i.e. the intercepting function will be replaced again by the standard NodeJs function). This way we have the least impact on e.g. performance, when we aren't interested in listening.

If no arguments against the new node name, I will rename the repository on Github and publish it on NPM ...
Bart


#9

You can also rename this thread


#10

Hi guys,

  • The node has been renamed to node-red-contrib-http-logger

  • The title of this thread has been updated

  • Version 1.0.1 has been published on NPM. @dceejay: do you need to do something manually to let it appear in the palette (or is that npm bug fixed already) ?

  • Besides the readme page, I have also updated the node label and info tab (to reflect the fact that both HTTP and HTTPS are supported):

    image

Have fun with it !!
Bart