I need to get the RAW body from a listener node


#1

Hello

I am developping a node as part of a package.
The node is a WebHook (much like what you can imagine exist in SLACK... so, the user subscribes to events and the target app will deliver those events as POST to the Webhook endpoint).

To create the endpoint I use the following in my code:

RED.httpNode.post('/thisIsMyEndpoint', bodyParser.json(), mySuccessCallback, myErrorCallback);

NOW, I need to catch the UNPARSED BODY coming from the target application, before it is parsed.
Experimenting blindly (I'm not an expert) it seems that the presence or absence of the bodyParser.json(), parameter actually does not change the results: "mySuccessCallback" ALWAYS receives a parsed req.body object... WHich is not what I need

I tried to use something like this:
RED.httpNode.post('/thisIsMyEndpoint', myOwnFunction, mySuccessCallback, myErrorCallback);

But my own function also only receives a req, res, next and *req is already parsed.

Do you have any idea how to

  • either intercept the raw body, process it and, eventually PREVENT further processing
  • provide a req.raw_body attribute to the "mySuccessCallback"

Perhaps I am not using the right tools

Thanks
/Stefano


#2

alternatively : is there the possibility to use directly the Express app.post ?


#3

Although I've never tried it, there is an optional middleware section (commented out) in the node-red settings.js file:

    // The following property can be used to add a custom middleware function
    // in front of all http in nodes. This allows custom authentication to be
    // applied to all http in nodes, or any other sort of common request processing.
    httpNodeMiddleware: function(req,res,next) {
        // Handle/reject the request, or pass it on to the http in node by calling next();
        // Optionally skip our rawBodyParser by setting this to true;
        req.skipRawBodyParser = true;
        next();
    },

So it appears that you can set up some http in flows in node-red, which will receive the raw data from the calling application. Of course, this setting will apply to ALL of your exposed endpoints...


#4

Here is some info on how to get round this issue:

However, I would question exactly why you need the raw body? I suspect that there are better ways to do whatever it is that you want.


#5

A bit more context on the request - Stefan is trying to handle an http request that includes a checksum of its body in a header. In order to verify the checksum, Stefan needs the raw body.

Unfortunately we don't currently provide any way to prevent the JSON body parser from running if the content-type properly identifies it as JSON.


#6

Ah, I see. And it looks like Express doesn't provide it either any more if the stackoverflow answer is to be believed.

Any way it can be reconstructed?


#7

i haven't understood what you mean by saying that "express doesn't provide it either any more".
Do you mean that using Express it is no more possible to add Middleware that can be set to intercept the raw body? I think that some of my colleagues were actually able to do it.

Isn't there the possibility (and, if yes, HOW) in the code of my web hook to program directly an Express endpoint instead of using the RED.httpNode..post ?
I mean, if I could use the app.use and app.post directly....

Thanks so much


#8

See the quoted article. Express used to have the raw body returned but according to that post, doesn't any more.

No, middleware is a really important concept in Express and I use it extensively in my uibuilder node.

Check here:

This gives you access to Express directly & immediately after that, it shows the use of Express middleware.

Note that Node-RED has two different Express instances to work with. One for the admin UI and one for everything else. In truth, I think that my code is serious overkill but I took it from core code. When the admin UI gets split from the runtime, that kind of defensive coding may be more useful.

As you can tell from my var name, this gives you an Express app reference to work with. The middleware can be "use", "get", "post", etc. "use" is the most generic.

uibuilder is very heavily commented (the only way I can ever follow my own code later!) so hopefully everything is pretty well explained. I also have an example repo on GitHub that may be easier to follow. Not all of the example node's code is necessarily correct now as I've not updated it but it might be easier to start with it as it works through and shows each stage of the node's process.


#9

@TotallyInformation : i do not see in the code where you handler is getting to the req.on in order to get the raw data....


#10

With all due respect to @TotallyInformation, his reply misses a key point.

By the time a node gets access to RED.httpNode, the default set of body parsers have already been applied.

As I've said a few times, as the person who wrote the code, there is no way for you to get ahold of the raw request in the current code.

If I wasn't snowed under with high priority work right now, I'd be able to spend some time devising a way to expose it in a future release of Node-RED. But right now, I don't have time to do that. If someone wants to help on that - please do.


#11

pls contact me offline @knolleary . If it will not take too much i may be willing to give a try


#12

Sorry, I probably confused things there. I was only answering the question about getting the reference to the Express app. Of course, most of the middleware has already been applied.

I also mentioned previously that Express no longer provides access to the raw return as indicated in the StackOverflow question that I referenced.

@stefanopog, if you want to confirm whether Express provides some form of access to the raw return. I suggest creating a simple Express app, you will be able to easily examine the data directly without the complexity of Node-RED wrapped around it.