Dynamic inputs/outputs for httprequest node

When you have multiple http request nodes chained together and want to store what was sent/returned by each call it gets quite cumbersome to use change nodes before and after the node to move things where they need to be to prevent collisions with other httprequest nodes.

What I suggest is we make the inputs and outputs dynamic so we can reduce how many nodes are needed in flows utilizing the http request node.

So like for example with the msg.payload input it would be nice if that was a TypedInput that could be set using msg, flow, global, number, string, JSON, JSONATA, etc. Having JSONATA here would be huge as it would remove the need for tons of my nodes that come before httprequest that I use to format the payload. Same with the cookies, headers, followRedirects, requestTimeout, etc (except with an additional option of "none" that disables reading that input at all). Headers may be a bit more complicated because there is already a section to define them but technically the msg.headers input still applies and should be made dynamic as well.

Now for the outputs we could add some tabs and move it under an Output tab like so:

(this was a real quick rough draft, so not exactly like this but similar)

We can set it up to return stuff in the same format as before to prevent breaking change (which is what I have it currently showing).

It would also be nice as a little bonus to return the milliseconds spent on the request as an option on the output tab.

Although I somewhat agree that dealing with multiple http nodes in a chained flow can be problematic, but your suggestion makes it much more complex for the user to set it up, plus i suspect, would break backwards compatibility.

To deal with these types of flows, I use link-call nodes instead, which will greatly simplify the flows, where you need to have reusability.

eg

We cannot always be backwards compatible. Future compatibility is more important. If we stuck to this rule we would never have environment variables in tabs or groups or junctions or module support in functions etc etc so that is not a great arguement in my opinion.

As for the request itself, I to get frustrated with having to add change nodes after to move payloads around.
Not so much with the HTTP request node because when that node is used correctly, you should probably always have a switch node or something after it to check the status code before you even begin the process the payload so this use case is not as big as some others.

It bothers me to the point I even raised a PR to add input and output properties to the majority of the core nodes (where it made sense). But this has not been merged unfortunately. It would have greatly simplified many flows.

1 Like

If done correctly (using the original properties as the defaults) then this should have no backwards compatibility issues. In the example I gave the outputs would be configured to return the same way it does now but with the ability to change it to however you want. Many of the modules I have published have been updated to use dynamic inputs/outputs without any BC issues.

This doesn't really work well for me. It's just moving the problem elsewhere.

Yeah this has bothered me as well. I manage node-red-contrib-matrix-chat and I have made quite an effort to make most of the nodes have dynamic inputs/outputs for this reason. It has actually drastically reduced the number of nodes in many of my flows.

I've been playing with using Node-RED to seed data into a QA environment at work which is when I started stumbling into this problem with the httprequest node. If I could configure the incoming payload using a TypedInput I could set it to JSONATA and can get rid of all of the change nodes I have before my httprequest node that do this same thing.

Yeah this is another thing I am trying to fix. One solution I am trying out is having a Validation tab (you can actually see this in my original post picture). This tab has a function code editor that allows you to write k6 style validation that is ran against the response. This way you can do something like this:

check(response, {
    'status is 200': (r) => r.status === 200,
    'is authenticated': (r) => r.json().authenticated === true,
    'is correct user': (r) => r.json().user === msg.username,
});

The idea being if any of the checks fail it throws an exception so it prevents the current flow from continuing but still lets you capture it with a catch node (or having this be configured to either throw or output to a second output). Toying with the idea of also allowing you to modify the context directly here as well so you could extract something from the response and set it into the msg, flow, or global context. This is just an idea and still a WIP but I would love it if we had some sort of solution to that as well.

I REALLY hope this gets merged. That would be a huge quality of life improvement for me. Thanks for putting in the effort to do that.

While I agree that the use of Typed Inputs and controllable outputs is good and should certainly be acceptable and backwards compatible. Indeed, I've moved most of the uibuilder nodees settings to use them - though they are a pain to code to be honest - too much boiler-plate code required for each one.

However, I'm not convinced that also adding a function tab would be really in keeping with Node-RED's UNIX like approach to nodes. And yes, I realise that my statement is slightly ironic given the complexity of the main uibuilder node! :slight_smile:

If anything, it would be more sensible to have a simpler setting so that - optionally - you could set a list of allow and deny response codes where the deny codes would raise an exception. This would remove the need for code in that node which would be more in keeping with Node-RED's design philosophy I think?

What do you mean by that?

@GogoVega I was disappointed to see your response in FR: core nodes to permit input and output to properties other than the hard coded payload · Issue #4651 · node-red/node-red · GitHub. I feel you have misunderstood the benefits it offered.

I have recently had lot a rather large set of flows to develop. The amount of change nodes moving payloads around was unbearable. In fact, quite often they muddied the waters, making the flows more visually difficult to follow.

I suspect the PR is a bit like the link-call & config nodes in subflows, until they are added folk won't realise the benefits, simplifications & improvements.

Denying the flexibility and user choice just forces workarounds and they often make the wrong choice like using context. That then leads to concurrency issues etc. I've lost count of the number of times I've had to say "move the payload into another property then move it back". "Don't store it in context". Etc etc.

2 Likes

I understood the potential behind it. My reluctance is related to the concept of a message property common to all nodes as well as the concept that each node has one role.

I readily admit that it can avoid using a change node.

I allow it in my nodes for some properties but to be honest I don't know if giving this flexibility to the payload which is supposed to be the main property is a good concept.

This also means that each node will have at least two fields to define message properties, which takes up space on the UI and gives an additional load of code. If we want to go in this direction, we might have to rethink the node UI.

I am not against the suggestion but we must not forget that the chosen concept will be the standard for third-party nodes.

For the Editor, you need 2 HTML elements (if you want to save the value). Then you may need a validation function:

    /** Validate a typed input as a string
     * Must not be JSON. Can be a number only if allowNum=true. Can be an empty string only if allowBlank=true
     * Sets typedInput border to red if not valid since custom validation not available on std typedInput types
     * @param {string} value Input value
     * @param {string} inpName Input field name
     * @param {boolean} allowBlank true=allow blank string. Default=true
     * @param {boolean} allowNum true=allow numeric input. Default=false
     * @returns {boolean} True if valid
     */
    function tiValidateOptString(value, inpName, allowBlank = true, allowNum = false) {
        let isValid = true
        let f
        try {
            f = value.slice(0, 1)
        } catch (e) {}

        if (allowBlank === false && value === '') {
            isValid = false
            // console.log({ name: inpName, why: 'Blank failed', value: value, allowBlank: allowBlank })
        }

        if ( allowNum === false && (value !== '' && !isNaN(Number(value))) ) {
            isValid = false
            // console.log({ name: inpName, why: 'Num failed', value: value, allowNum: allowNum })
        }

        if ( f === '{' || f === '[' ) isValid = false

        $(`#node-input-${inpName} + .red-ui-typedInput-container`).css('border-color', isValid ? 'var(--red-ui-form-input-border-color)' : 'red')
        return isValid
    }

You also need to prep eaach typed input in oneditprepare:

        // @ts-ignore core data typed input
        $('#node-input-data').typedInput({
            // types: stdStrTypes,
            default: 'msg',
            typeField: $('#node-input-dataSourceType'),
        }).on('change', function(event, type, value) {
            // console.log('change')
            if ( type === 'msg' && !this.value ) {
                $('#node-input-data').typedInput('value', 'payload')
                // console.log('change 2')
            }
        } ).typedInput('width', tiWidth)

Then you also have to handle things in the runtime as well - with added complexity since you need to be able to handle things as async functions.

// ... 
// This now has to be async
async function inputMsgHandler(msg, send, done) {
    // ....

    // Get all of the typed input values (in parallel)
    await Promise.all([
        getSource('parent', this, msg, RED),
        getSource('elementId', this, msg, RED),
        getSource('heading', this, msg, RED),
        getSource('data', this, msg, RED), // contains core data
        getSource('position', this, msg, RED),
    ])

    // ....
}
// ...

And a processing utility with all the error checking, etc.

    /** Get an individual value for a typed input field and save to supplied node object - REQUIRES standardised node property names
     * Use propnameSource and propnameSourceType as standard input field names
     * @param {string} propName Name of the node property to check
     * @param {runtimeNode} node reference to node instance
     * @param {*} msg incoming msg
     * @param {runtimeRED} RED Reference to runtime RED library
     * @param {string} [src] Name of the typed input field (defaults to `${propName}Source`)
     * @param {string} [srcType] Name of the field holding the typed input field type (defaults to `${propName}SourceType`)
     */
    getSource: async function getSource(propName, node, msg, RED, src, srcType) {
        if (!propName) throw new Error('[uiblib:getSource] No propName provided, cannot continue')
        if (!node) throw new Error('[uiblib:getSource] No node object provided, cannot continue')
        if (!RED) throw new Error('[uiblib:getSource] No RED reference provided, cannot continue')

        if (!src) src = `${propName}Source`
        if (!srcType) srcType = `${propName}SourceType`

        if (!msg && srcType === 'msg') throw new Error('[uiblib:getSource] Type is msg but no msg object provided, cannot continue')

        if (!(src in node)) throw Error(`[uiblib:getSource] Property ${src} does not exist in supplied node object`)
        if (!(srcType in node)) throw Error(`[uiblib:getSource] Property ${srcType} does not exist in supplied node object`)

        const evaluateNodeProperty = promisify(RED.util.evaluateNodeProperty)

        if (node[src] !== '') {
            try {
                if (msg) node[propName] = await evaluateNodeProperty(node[src], node[srcType], node, msg)
                else node[propName] = await evaluateNodeProperty(node[src], node[srcType], node)
            } catch (e) {
                node.warn(`Cannot evaluate source for ${propName}. ${e.message} (${srcType})`)
            }
        }
    },

You will see that I've created some standard utility functions to help reduce my boilerplate otherwise a node with lots of typed inputs gets very bloated. In reality, the getSource function is in a separate shared library.

Thats quite a lot of work that has to be done for each typed input.

There are two possible validators: typedInput and the one defined in the node property.

You can retrieve the type without using typeField.

You bypass its use.

For the runtime part I create a Promise for RED.util.evaluateNodeProperty but it comes to the same thing.

Even without the custom validator which I needed for some of the inputs in that particular node I seem to remember, it is a lot of boilerplate (repeated) code, much of which could possibly be dealt with by the typed input itself. If I remember rightly, I had to add the width on to every one I've ever used because the default width was slightly off.

It is just a grumble. I'm using them pretty much everywhere now so I wish they could be simpler to code. They might be low-code in use but they certainly are not when writing custom nodes. :smile:

I admit that a big part of my code is the validation :sweat_smile: :shushing_face:

Besides, I should look again at the core code, but that will be for later :roll_eyes:

I agree :100:

I wish it were more declarative (as in simply add the necessary html attributes and "it just works")

I even wrote a helper that permits zero code declaration of typedInputs

E.g.

JS Code

oneditprepare: function () {
  RED.ui.utils.initTypedInputs(this) // init ALL elements with the attr `typed-input`
}

HTML

<!-- basic examples -->
<input type="text" id="node-input-property" typed-input data-types='["msg"]'>
<input type="text" id="node-input-property2" typed-input data-types='["msg","env"]'>
<!-- slightly more advanced example -->
<input type="text" id="node-input-property3" typed-input data-types='["msg","str","env"]' data-default-type="str"  data-type-field="#node-input-prop3Type">
<input type="hidden" id="node-input-prop3Type">

Before anyone says/asks...

  • RED.ui.utils.initTypedInputs(this) is pseudo / made up for this demonstration (but yes, I have the code that actually supports it)
  • Yes, there are cases where JS will be required (for custom validation, non built-in types etc) and I am not proposing we remove that.
1 Like

I do agree with the sentiment however having to 1) be aware your previous payload will be overwritten and 2) be aware of the best way to handle this muddies the waters to the point it creates unnecessary hurdles and hoops to jump through in the name of "concept".

1 Like

Yes please!!!! Can we have it... please! .... pretty please!!!! :smile:

Indeed, and of course I don't mind that. I just don't want to have to do it for everything.

If we could also have an improved runtime fn using promises, solid thrown errors (required for promises) and able to deal with a null msg argument (actually, I haven't tested that I realise - it possibly already works?)

1 Like

FR Raised here: FR: Declarative creation of typedinputs · Issue #5066 · node-red/node-red · GitHub

4 Likes

To further drive home the need for this with the httprequest node here is an example:

I created this flow to do various API commands to my modem. When I authenticate I get back the cookies needed to auth other requests in msg.responseCookies and have to use a change node to move this to msg.cookies so the other nodes use them. I also have to delete the payload that endpoint returned before calling the next httprequest node. All of these change nodes would just go away if we had dynamic inputs and outputs:

  • I could set the responseCookies to be returned on msg.cookies for my authenticate httprequest node
  • I have some static payloads that get set before doing some actions (reboot, renew IP, release IP, etc) which if we had a TypedInput for the incoming payload would allow me to just statically set this on the httprequest node.
  • No more deleting the payload so it doesn't get sent to the next httprequest node. I can now either configure the node to ignore the payload or set the previous httprequest node to not return on the payload (or return to another property to avoid this collision).

This makes the previous flow suddenly look like this:

And by setting the nodes to only return 2xx responses I can send the errors to a catch node so I don't even need to validate between the requests. Keeps these flows short and simple as they should be.