Automate updating nodes to the Flow Library

Developing Nodes
1

I am the author of the node-red-contrib-openhab3 Flow. I'm using a GitHub Action to build new releases of the npm package and publish it to the npm repository. I'd like to add an extra step to that release-pipeline so it automatically updates the Flows repository.
Currently I have to visit the Flows website with my browser, enter the npm module name for the flow (node-red-contrib-openhab3) and press Add node to update the Flows repository.

How do I go about this? Is there maybe a slick way to call the add node page with a parameter or something to automate this step?

2

Well, here is the code pulled from one of the flow pages (for uibuilder of course :wink: ):

        $('#refresh-button').click(function(e) {
            e.preventDefault();
            if (refreshSubmitted) {
                return;
            }
            refreshSubmitted = true;
            $("#refresh-node-error").hide();
            $("#refresh-node-label").hide();
            $("#refresh-button").addClass("submitted");
            $("#refresh-node-loader").show();
            var module = {
                _csrf: $("#refresh-csrf").val(),
                module: "node-red-contrib-uibuilder"
            };
            $.post("/add/node",module,function(data) {
                window.location = data;
            }).fail(function(err) {
                if (err.status == 403) {
                    window.location.reload();
                }
                refreshSubmitted = false;
                console.log("ERROR",err);
                $("#refresh-button").removeClass("submitted");
                $("#refresh-node-loader").hide();
                $("#refresh-node-label").show();
                $("#refresh-node-error").text(err.responseText).show();
            });
        });

So you can see that you need to do a POST to https://flows.nodered.org/add/node passing in some JSON with at least the module name. Not sure if it will work without the _csrf entry, if it doesn't you might be stuck.

3

csrf stands for cross-site request forgery and is there to prevent hackers from posting the form when you're browsing another site. E.g. you browse some innocent looking site and they make your browser do a hidden post to flows.nodered.org and since your browser has a cookie it goes through and does damage. So long story short... you're gonna have to first log in and GET the form to get the csrf token and then post with that token.

4

I don't think you need to be logged in, but you do need to get the page first to obtain the csrf token to include in the post to the /add/node endpoint.

We had to add the csrf protection after some individuals took it upon themselves to script mass refreshes of the library that undid all the work I had done to keep within various rate limits.

5

Well that explains it then and perfectly reasonable. Kind of puts a damper on @randomname's original ask :slight_smile:

In that case @randomname, if you still want to do this, I think the only realistic way is a browser automation. And obviously, make sure that you are only updating your own module and even then only doing it when you really need to.

6

I already tried to automate this using curl, which has support for cookies, but I was also required to provide the csrf token. Logging in was not required as far as I could determine.
Using a real browser that doesn't require a GUI and is thus suitable for automation (selenium comes to mind) is probably a solution that I can get to work, but goes too far for its purpose anf what it does IMHO.

I think I'll have to accept that every now and then, I'll have to visit the page manually from my browser and submit the node name there. I hope the need for this page will be rendered unnecessary in the future.

Thank you all for your responses!

7

I have found a solution for automation that only requires bash, curl and python3. It uses curl together with its cookie management and a python script for extracting the _csrf token from the page' html output. I can now verify that this page doesn't require you to login.

I'm sharing it here in case anyone else wants to automate it too. Save these files to disk, in the same folder:

update_node.py

#!/usr/bin/env bash

RESULT_FILE=/tmp/result.txt
FILE_WITH_CSRF=/tmp/output.html
COOKIES_FILE=/tmp/cookies.txt
MODULE_NAME=REPLACE_WITH_MODULE_NAME

# remove any previous cookies
if [ -f "${COOKIES_FILE}" ]; then
    rm "${COOKIES_FILE}"
fi

# get the _csrf token
curl -s \
    -c "${COOKIES_FILE}" \
    -b "${COOKIES_FILE}" \
    -XGET "https://flows.nodered.org/add/node" > ${FILE_WITH_CSRF}

input=$(cat ${FILE_WITH_CSRF})
CSRF_TOKEN=$(echo "${input}" | ./get_token.py)

if [[ ! -z "${CSRF_TOKEN}" ]]; then
    echo "_csrf token was found to be $TOKEN"
else
    echo "_csrf token not found in HTML!"
    exit 1
fi

# Send a node update request, with the _csrf token
curl -s \
    -XPOST "https://flows.nodered.org/add/node" \
    -c "${COOKIES_FILE}" \
    -b "${COOKIES_FILE}" \
    -H "referer: https://flows.nodered.org/add/node" \
    -H "Content-Type: application/x-www-form-urlencoded" \
    -H "x-requested-with: XMLHttpRequest" \
    -d "module=${MODULE_NAME}&_csrf=$CSRF_TOKEN" > ${RESULT_FILE}

# Show the add/node page' response
cat ${RESULT_FILE}

and get_token.py:

#!/usr/bin/env python3

import re
import sys
for i in sys.stdin:
  g = re.match('.*\<.*name="_csrf".*value="(.*)"\>.*', i);
  if g is not None:
    print (g.group(1))

Now set the MODULE_NAME variable in the update_node.py script, save it to disk and run the file. The module will now be updated, if an update is available, otherwise the script will print Module already at latest version, given you have used it on a valid node name.

8

Ah, I should have realised that the CSRF is in the headers, I foolishly made the assumption that you would need to run a page script. In that case, all you really need to do is use a request node to get a flow page, the output of that will give you the headers. You can extract the CSRF from those and pass it on to a 2nd request to do the PUT. No need for anything other than Node-RED. :grinning:

9

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.