Possibility of extending (core) nodes?

I was wondering if it was possible to extend existing nodes with new functionality, where forking a node and sending a pull request for added functionality is not always a preferred option, nor would forking and keeping your own updated version of the node be. While this is quite generic and might sound vague, I've use cases where currently I'm just writing custom nodes for every situation, every specific website... And it's not very efficient as most of the interaction I need I can do through existing nodes. For example, when parsing HTML or XML, most of the time I only need a specific part of it. An often given suggestion on the forums is to convert it to a js object through the XML node (internally through xml2js), then call a JSONata expression on it to get the information you need out again.

Coming from a (black box) test engineering background, I'm not exactly looking too forward to that solution. In my automations, I often deal(t) with sites under third party control where I don't have access to the source code as half of it is generated and every new iteration attributes or order of elements change around (including up and down the tree). For me when dealing with these situations, XPath is my preferred solution. I've (had) to deal with web applications so unstable that to create a stable test automation (that will survive a couple iterations of development/rolling out to production without having to change the code over and over), a text search on the page had to be run to find connected elements and go up and down through descendants/ascendants to find the correct elements/information. For one in particular, I wrote the automation 3 years ago, and with the client rolling out updates every 2 weeks, that particular test (a very detailed search form where running the test manually takes 2-3 hours) is still working without any changes to my code.

One of my use cases involves getting information from specific webpages of grocery stores in my area. Each of these stores use their own kind of website setup. Some use (internal) JSON based APIs that populate the page, and thus getting info out is as easy as connecting to that JSON API instead. Others add everything on the server level into the webpage, so parsing the entire page is needed. This use case is based on my worsening mobility, and the difficulty to get my needed groceries every day. Since most of the stores in the area support delivery above minimum amounts ordered, it's a viable solution. Because of my worsening health in general, I won't be able to fiddle with flows on the bad days, and that's exactly when things always go wrong. So a solution that is as-stable-as-possible is preferred.

Instead of writing custom nodes for all these specific cases, there are a couple options for me, including 1-liner function nodes connected to switch and change nodes, with the code from those function nodes calling libraries made available through settings.js, or executing local scripts that send the output back to Node-RED. If something needs to be changed to a flow like that, the result is quickly becoming maintainability hell. At this point the best solution feels like being able to add rule types/formatting to the core switch and change nodes.

This might not be the best thing to ask in general considering that Node-RED is of course coming from an (older) javascript background, and the newer class written format (ES6 and newer) is still far from the OOP based languages, but I was wondering if it would be possible to extend existing nodes, in a (similar-to-)OOP style. Since my own additions are aimed at use cases like the above, writing PRs for the core nodes would only make the core more convoluted. A different option would be to fork the core libraries and keep my own maintained version of these nodes and support them through every update, which I'm pretty sure I'm not able to do with my health. Yet another option would be to create custom nodes for the core switch/change functionality, where the code is just copied from the core library with a couple lines added, which again I'm very much not looking forward to. One of the reasons I'm hoping to port most of my home-grown solutions to NR flows is to ease maintainability, and have them be as stable as possible; not having to touch a finished (block of a) flow for months would be preferred.

Coming from an OOP background, the ability to have custom nodes being an extension of an existing node with specific functionality added/changed to it sounds, frankly, amazing. But looking at the core design of NR and nodes, I have not a clue if this can even be implemented at all, let alone without becoming a huge breaking change for existing nodes.

Can some of this be achieved by using subflows ? This allows you to pre-populate core nodes with certain parameters (like a request url), and some extra (parsing) functionality and encapsulate it into a single subflow.

Recently this has been extended to allow the use of "environment variables" but they should really be called instance variables that can be set on that instance of the subflow.

There is also ongoing work in this areas (coming soon to v 1 hopefully) - where you can create a custom config menu for a subflow (eg like other nodes have) - see https://github.com/node-red/node-red/pull/2196 - which will then eventually be followed by the ability to export the subflow as a custom node... (https://github.com/node-red/designs/blob/master/designs/subflow-node-modules.md)

2 Likes

Holy crap, this is going to be a big deal for handing flows off to other people in my organization for "care and feeding!"

1 Like

An approach like this could replace most of the one-liner function nodes, by creating a generic subflow for the functionality, with the custom aspects become configurable. Sounds perfect actually.

The environment variable approach should already be able to set up basic configuration through placeholder values and checks for the placeholder to be replaced.

I hadn’t been able to find this in the documentation so far, but with this knowledge I can. Couple improvements for the documentation? The subflow page in the editor guide is missing the 0.20.x changes regarding the new environment/instance variables. And with the current name for these instance variables being “environment variables”, I had wrongly assumed they would indeed be written on an environment level rather than a subflow level, and thus that the documentation page for those wouldn’t be relevant for my situation. The wrong assumption as proven.

As it is not explicitly described in the documentation for subflow instance properties, when a env var test is defined on the subflow, is this accessed in a function node the same way as well (env.get("test")) as well?

docs always lag code .... and given the more substantive changes just coming then the screen shots would need to change again... but yes, they can always be improved.

and yes env.get("test") is the way. (which is why they did get called environment variables)... I guess from the inside perspective they are env, and from outside they are instance... ah the joy of language.

1 Like

I’m going to see if I can implement as many of my needs through the subflow with instance properties. I’ll come back to this topic at a later stage.

I’ll see what I can do here to help now that I’ve puzzled the sources of information together. PR for the docs coming in a couple weeks I hope.

As I said it's a fairly volatile area... - check the github project PRs list around subflows - though how to use env vars isn't changing so that would be safe.