Node-RED Survey: Shaping the Future of Node-RED's User Experience

[TotallyInformation]

@gregorius

@vrilcode

Node-RED has awesome data transformation possibilities with JSONata (it’s a beast, but powerful). What I miss is data validation functionality to check, whether incoming data is appropriate. Something like Zod, but in a way that feels more “natural” for Node-RED.

There is json schema validation which partly perhaps provides what you're looking for. Admittedly JSON Schema is a horribly complex and is a real pain to get right.

I've used Zod a bit now with Astro and it is quite nice. If you are a coder.

But my view is that these should probably be contributed nodes rather than core?

@gregorius

@vrilcode

Mustache templating is fine, but it is very restrictive. Would be great to have more flexible templating language in the core.

Mustache is designed to be functional limited because the intention is to have no logic in your templates. Which is a good thing. IMO everything else is a road to nowhere - Talking Heads.

It is certainly a slippery slope. But I get that some may want the convenience. Again, such a thing could easily be a contributed node however. EJS might be a good candidate since it is also JavaScript based.

[Steve-Mcl]

@gregorius

There is json schema validation which partly perhaps provides what you're looking for

And just in case you were not aware, the built in JSON node can also do schema validation

@vrilcode

Mustache templating is fine, but it is very restrictive. Would be great to have more flexible templating language in the core.

I agree with @TotallyInformation, this is more suited to a plugin/contrib node. There are a few existing ones:

• node-red-contrib-file-nunjucks (node) - Node-RED
• node-red-contrib-eta (node) - Node-RED
• node-red-contrib-handlebarsjs (node) - Node-RED
• node-red-contrib-pug (node) - Node-RED
• node-red-contrib-velocity (node) - Node-RED
• There are likely more in the flows lib: Library - Node-RED

And if you want to use zod you can always import it in a function node. I did that in this demo here: Model Context Protocol (MCP) over HTTP - #4 by Steve-Mcl and you can import pretty much any lib see this article for details

[gregorius]

@Steve-Mcl

JSON node can also do schema validation

I recall having a discussion about that and JSON node converts and validates - I just want to validate an msg object. That can't be done with the JSON node.

[vrilcode]

@gregorius

Mustache is designed to be functional limited because the intention is to have no logic in your templates. Which is a good thing. IMO everything else is a road to nowhere - Talking Heads.

You can take this view, but then you should create better options for data formatting so that you don't have to use Function Nodes every time (which I always consider a secondary option in Node-RED). JSONata is very well suited for data formatting, but as far as I can see, there is no option for reusing complex formatting. In templating languages other than Mustache, you have (custom) filters or similar. The logical consequence for Node-RED would therefore be that you can easily extend JSONata with your own functions ( Embedding and Extending JSONata · JSONata ), similar to $moment(). I don't currently see this option in Node-RED. The JSONata code in the Change Node cannot be easily extended without hacks.

[vrilcode]

@TotallyInformation

I've used Zod a bit now with Astro and it is quite nice. If you are a coder.

But my view is that these should probably be contributed nodes rather than core?

If data transformation is one of Node-RED's core functionalities, why shouldn't data validation be one too? Node-RED is a kind of data hub. In my opinion, it is essential to check that no incorrect or dangerous data is being slipped in.

[gregorius]

I don't quite understand why I should provide a better option for data formatting when everything works just fine for me.

If node red doesn’t fit your needs why use it?

I don't even know what you mean with data formatting- normally data is interchanged between machines and made into pretty graphs for human consumption. For those graphs, most solutions use json and css - both of which are wonderfully supported by node red.

Btw it is not I that take that view, rather it's the design principles of mustache. So if anything, they should provide something that meets your needs.

[AllanOricil]

@vrilcode

@TotallyInformation

I've used Zod a bit now with Astro and it is quite nice. If you are a coder.

But my view is that these should probably be contributed nodes rather than core?

If data transformation is one of Node-RED's core functionalities, why shouldn't data validation be one too? Node-RED is a kind of data hub. In my opinion, it is essential to check that no incorrect or dangerous data is being slipped in.

You can add a json schema node to validate your node inputs and outputs. You can encapsulate these nodes in a subflow node.

Node-RED doesn’t perform data validation natively because wires DO NOT send data only, but objects as well. There is no serialization/deserialization in the flow. A node can output a class instance, for example. This allows people to create factories, or use dependency injection in their flows, as well as a lot more.

Nodes built with my [node template](Benchmark of Node.js validators | Codetain - end-to-end software development) perform data validation using json schemas and ajv. I didn’t use zod because it is way [slower than ajv](Benchmark of Node.js validators | Codetain - end-to-end software development). Zod would have tremendous performance impact during runtime.

[vrilcode]

@gregorius

If node red doesn’t fit your needs why use it?

It fits basically. This discussion is about extensions. When you don’t need formatting of values (for instance: human readable dates for reports etc), why do you join this discussion?

[vrilcode]

My suggestion was not to integrate Zod in particular. See my posts above.

[gregorius]

@vrilcode

why do you join this discussion?

Because you replied to one of my comments and claimed that I had to offer some alternative:

@vrilcode

You can take this view, but then you should create better options for data formatting so that you don't have to use Function Nodes every time

Why did you do that? As I point out in my reply:

@gregorius

Btw it is not I that take that view, rather it's the design principles of mustache. So if anything, they should provide something that meets your needs.

@vrilcode

I don't currently see this option in Node-RED. The JSONata code in the Change Node cannot be easily extended without hacks.

A more appropriate place to mention that would be over at my comment where I point out that NodeRED should be more extendable. Being able to extend JSONata via nodes/plugins is a good point - that seems, at least in your case, a desirable feature:

@gregorius

Open up the internal APIs so that node packages can do more, things like being able to gets a heads when Node-RED starts/stops/restarts/flows reload, clearly defining when the lifecycle of a plugin begins, accessing the events when messages get passed around.

Now you have already pointed out that there is a solution for your problem and that is to use a function node but you don't like that. So if it was simpler for you to create a custom node, you could put everything in that custom node. Now creating custom node isn't that simple but if one could do that in Node-RED - just as subflows are created in Node-RED - that would a good step in the right direction. So then you can like a second comment of mine where I point out exactly that - ironically also in this thread:

@gregorius

Which actually brings me to a third wish (after all there are three wishes with each good genie!) that creating nodes was easier and doable within Node-RED itself. Again, I created my own solution using NodeDev but that's perhaps not the solution rather an attempt at a solution.

The point about liking these comments is that then these desires that you have might actually get on the radar of the NodeRED developers - when more people than just me think this way then the NodeRED developers might actually think about it too.

[TotallyInformation]

@Steve-Mcl

I agree with @TotallyInformation, this is more suited to a plugin/contrib node. There are a few existing ones:

I should also have pointed out that you can integrate any template library with UIBUILDER simply by making the library available and changing setings.js

UIBUILDER Documentation v7

Thus allowing server-side templating. I mention EJS but I'd forgotten about ETA as well. But any templating engine that you can integrate with ExpressJS will work.

@vrilcode

@TotallyInformation

But my view is that these should probably be contributed nodes rather than core?

If data transformation is one of Node-RED's core functionalities, why shouldn't data validation be one too? Node-RED is a kind of data hub. In my opinion, it is essential to check that no incorrect or dangerous data is being slipped in.

Node-RED's core function is as a low-code compute platform. Data transformation is one aspect of it's use.

In some ways, Node-RED already has too many nodes in core. This can and does limit how fast they can evolve which is not always a good things. It also makes it harder for people who aren't core devs to contribute to changes/improvements.

Thankfully, contributed nodes are first-class citizens in Node-RED so this is not an issue.

You could - and people regularly do - argue that all sorts of things might be included in core but the more things that go in, the harder it is to both maintain and to change.

I am not doubting that Zod might be a really useful capability to use with Node-RED, only questioning whether it should be in core. Personally, I would rather that core devs focused on continuing the excellent work of evolving and improving Node-RED itself.

But that is merely my view - though based on ongoing experience from the very early days of Node-RED. - we are all part of this community and this community is generally very good at listening to and critiquing different viewpoints. This should not be taken personally, it is merely a sign that we are passionate about the platform and want to see it become its best self.

[AllanOricil]

I know. I was answering the following post. Updated my answer above with the quote. I answered your question with my response, and why it can’t be done.

@vrilcode

If data transformation is one of Node-RED's core functionalities, why shouldn't data validation be one too? Node-RED is a kind of data hub. In my opinion, it is essential to check that no incorrect or dangerous data is being slipped in.

[ralphwetzel]

@TotallyInformation

I mention EJS but I'd forgotten about ETA as well.

Off topic: There’s a node to use ETA in Node-RED: node-red-contrib-eta

[Semmu]

i already filled the form, but got some new ideas, so sharing them:

• there is a pattern i use very frequently in my flows: a (basically empty) inject node to trigger a flow manually, and function nodes to create custom message objects. (or sometimes just do a little transformation). function nodes themselves can be very powerful, but its also a trap, since they can used for too many things, instead of more specialized node types, which makes the flows visually very hard to parse, since too many nodes are simply function nodes. so i think it would be nice to have a dedicated message builder or transformer node, which could utilize e.g. jsonnet for more powerful transformations.
• also it would be nice to have more manual entry points for flows. instead of placing inject nodes “everywhere”, some nodes themselves (like functions nodes) could have an optional setting to display an inject button on themselves, eliminating the need to place inject nodes to be wired up to them.
• i also regularly need to place random debug nodes and wire them up just to debug a flow, especially when working with external API. it would be nice to flag certain connections as a breakpoint, so when messages pass through them, they are automatically logged to the debug sidebar. or maybe even stop the execution and ask for confirmation? maybe that’s too much, but automatic debug logging on certain wires would definitely be useful and would make debugging easier and faster.

sorry if any of these ideas are already implemented, that means i did not find them haha.

[TotallyInformation]

@Semmu

dedicated message builder or transformer node, which could utilize e.g. jsonnet for more powerful transformations.

Interestingly, the change node is already very close to this. While JSONata is not quite as versatile as jsonnet, it doesn't need to be in the context of Node-RED. The only thing that I really like from jsonnet that could be built into JSONata is the concept of self so that you can back-refer to parts of an object already defined, I really like that.

@Semmu

function nodes themselves can be very powerful, but its also a trap, since they can used for too many things, instead of more specialized node types, which makes the flows visually very hard to parse

I'm an old-school coder so I am more than happy with using function nodes to simplify the visual layout - this is the flip-side of the argument you are making I think. But all I do is make sure I have a sensible and descriptive name for the fn node. And I usually group related nodes and give the group a descriptive name as well.

@Semmu

also it would be nice to have more manual entry points for flows. instead of placing inject nodes “everywhere”, some nodes themselves (like functions nodes) could have an optional setting to display an inject button on themselves, eliminating the need to place inject nodes to be wired up to them.

This one I'm afraid I can't agree with. Having a single, clear inject node keeps things obvious. But, perhaps more importantly, keeps other nodes simpler. If I need a lot of manual injects somewhere, I will typically hide their name so they take up less visual space.

@Semmu

i also regularly need to place random debug nodes and wire them up just to debug a flow, especially when working with external API. it would be nice to flag certain connections as a breakpoint, so when messages pass through them, they are automatically logged to the debug sidebar. or maybe even stop the execution and ask for confirmation? maybe that’s too much, but automatic debug logging on certain wires would definitely be useful and would make debugging easier and faster.

There is absolutely nothing stopping any node from outputting to the debug panel, it is a simple enough API.

But one of the alternatives that has been discussed is using an inline debug node which should have an on/off configuration switch. Though, using your thoughts, perhaps it should also allow control via an input msg as well which would let you control debug outputs via your flow.

I've actually created a test node that is an inline debug, threw-out:

Sorry for the pun-ish naming. The icon indicates whether output is on or off.

@Semmu

sorry if any of these ideas are already implemented, that means i did not find them haha.

Really doesn't matter, useful discussion points anyway, thanks for sharing.

---

@dimitrieh, happy to discuss whether that test node should be fed back into the current debug node or become a contributed node.

[TotallyInformation]

Hi @dimitrieh, thought of another thing that would be really great. In fact, I just hit this.

With a large set of flows, it can be quite challenging to go back and forth between different parts/tabs.

It would therefore be great if Node-RED leveraged the browser's hash navigation history when at least changing tabs. This would need tab id's to be hash url names rather than the current form of course but it would let Node-RED update the browser navigation history and allow using the browser back and forward buttons to move back and forward between previously visited tabs.

[gregorius]

@Semmu

but automatic debug logging on certain wires would definitely be useful and would make debugging easier and faster.

My introspection package has message tracing (it's a bit of scroll down) might be what you're kind of looking for.

[Steve-Mcl]

@TotallyInformation

Hi @dimitrieh, thought of another thing that would be really great. In fact, I just hit this.

With a large set of flows, it can be quite challenging to go back and forth between different parts/tabs.

@dimitrieh This is very important (more important than people realise). Please read this related discussion: Suggestion: Add forward/backward tab navigation - #3 by Steve-Mcl

[tmichaeltx]

Post survey, I had another thought that might be worth consideration. I like have certain flows built and then left with the intention of using them indefinitely. The problem with this approach is that over time with upgrades to node-red and new version of node.js, it becomes possible that a node I have used has been abandoned by its creator and a functioning flow stops working. It would be ideal if there was some process there were at least 4 types of node support:

• Nodes that may or may not be maintained by the creator
• Nodes where the creator has agreed to be notified of upgrades that might break the node
  • sub-category 1 - Creator has notified
  • sub-category 2 - Creator has responded that they have tested the node in the current (newly updated) environment and should remain functional
• Nodes that the node-red team has designated as important enough to make into a native part of Node-red at such time as the original creator no longer wants to maintain them
• Native to Node-red

If we had this and a way to search through nodes in each category, it would make it easier to focus on using nodes that are more likely to be effective for the long run and find any that aren’t being supported and might be the source of issues.

I recognize when the node interfaces with a system outside of the node-red environment, changes there can make a node no longer functional, but at least it would care for one end of that connection being maintained.

I guess the other downside is that I would imagine the node-red team could only agree to take over ownership of a very limited number of nodes and that could feel insulting to developers who build great nodes that don’t get offered a chance to put their node into that category.

Just a thought from one who benefits massively both from Node-red and the many nodes created by others.

[dimitrieh]

@TotallyInformation

In some ways, Node-RED already has too many nodes in core. This can and does limit how fast they can evolve which is not always a good things. It also makes it harder for people who aren't core devs to contribute to changes/improvements.

Thankfully, contributed nodes are first-class citizens in Node-RED so this is not an issue.

Can you elaborate on this? Curious to thoughts on this.

A lot of more recently developed applications (HA/Raycast/Figma) have moved to a place where their plugins/integrations code has been internalized/taken up into the main code base and in that way needs to pass code/security standards and are certain to have been reviewed. Harder to contribute, but also increasing quality/trust potentially? Take my thoughts here with a grain of salt, but it is a relevant topic due to the fact that with Node-RED’s custom node system it is currently hard to know which ones are good + actively maintained. Survey results point in this direction as well as an opportunity to improve upon.