Taking this along for sure! Corresponds with ideas I already had based on auditing the application. All fall in flow debugging and flow management capabilities.
A question perhaps into this direction. What are everyone’s thoughts on node anatomy and potential changes/improvements there?
There is a lot of potential directions here. The nodes in Node-RED very much are like little boxes, with a name that can endlessly widen (though there is a bit of a hidden trick to have a line break in there) and not much more. There is a button for certain nodes like inject/debug, but that’s it. Hovering over entry/exit points reveals their name, and there is a little icon.
Easy reading of flows (without having to click into nodes), plus introducing by-pass/breakpoint actions on nodes makes me look into the direction of introducing small changes to the node anatomy.
Is node red closer to something like Linux or closer to something like n8n?
I.e., does NR provide a specific set of functionality or does it aim to be more generalistic, providing a platform for general problem solving?
I think answering those two questions - for the long term - might well provide guidance on how node red handles plugins/contributions.
One thing I realised early on was that node red was probably open sourced because of the need to provide support for all the various IoT devices out there. No one company/developer could cover them all. So what better way than to organise a community around the core and have the community develop plugins/contributions for unknown third party devices.
N8N definitely takes a more centralised approached providing guarantees on functionality of their nodes. But they aren't focused on IIoT as Node-RED is.
EDIT: looking at the node contributions as a spectrum: there is the linux approach where anyone can write and distribute code for Linux, there is no barrier. Things that go into the kernel are gatekeeped but not software that runs on Linux. Then there is the app store approach such as apple whereby developers can contribute but have to jump through hoops and must use Apple tools. Adding to the Apple kernel is impossible. N8N seems to be neither with contributions mainly coming from N8N the company.
FlowFuse as a company (or their investors) would probably prefer the app-store model since it guarantees a revenue stream and security/quality of contributions. With a win-win situation of expanding the applications of Node-RED while at the same time, preventing competition from entering the market. Node-RED as open source software with a non-paid community would prefer the Linux model of wild west contributions with no guarantee of long-term maintenance nor security nor scope of functionality.
Certainly an idea that I have discussed briefly already. If I remember my conversation right there are performance implications that come with it.
Taking this along and will get back to you when this become relevant!
yes! Going back and forth should be easier and has been mentioned as an improvement necessary especially when managing flows better. A question in regards to tab management. What are your thoughts regarding vertical tabs? So having tabs on the left side like https://zen-browser.app/. Information vs screen real estate wise this makes much more sense, but would at the same time be a deviation from the usual “tabs at the top”.
Actually spoke to Steve about this, so yes fully aware and cannot but agree that this is very necessary!
Thanks for the input. Can you look at one of my previous replies? (I think this one: Node-RED Survey: Shaping the Future of Node-RED's User Experience - #80 by dimitrieh ). I ask a question there about what people’s thoughts are in regards to management of custom code/custom integrations and how to go about that. Trade-offs with externally managed/taken up in the code base. I think its key here to think in a way where that code stays collaborative and linked to the original creator, and at the same time that it gets the required support irregardless of the personal situation of the one contributor.
Good questions and good considerations. Initial personal take on this is that ideally we want it to be useful and being used. Node-RED being owned by the OpenJS Foundation also brings a lot of guarantees (and has some parallels to the HA project). More activity brings more contributions brings more possibilities, and refinement. How would you answer your questions (also if looking how it is done for a project like HA)?
Being relevant and being used is a great end-goal, however how do you or your company, FlowFuse, plan to get there?
I would be more interested what FlowFuse as a company has to say. OpenJS and Node-RED Open Source offerings do not make money to finance development. So it's probably more interesting - for the long term - how FlowFuse is going to increase their revenue streams.
Why does FlowFuse concern me? Well because the (most) core (active) developers are employed by FlowFuse so there seems to be a 1-to-1 correlation between their financial well-being and the well-being of Node-RED!
Unfortunately you seem to have made my point. Let me try one more time to convince you.
Firstly, It is important. I have seen/lived this first hand in an highly automated automotive plant where the brain context switching due to lack of good, easy, seamless FWD/BACK navigation can mean the difference between a short stoppage and a major breakdown. I am happy to expand on this. Like I said, people don't know how important this is until they have had it. Even when they do have it they cannot readily quantify it or recognise it, they just know "they like Tool X better than Tool Y".
Vertical tabs would occlude even more of the desktop.
Now, if the palette and right hand sidebar could collapse into dropdowns alongside the hamburger menu, by all means put vertical tabs in a dropdown there too.
I do like the general sentiment here. Certainly I could see adding a button to function nodes would be an easy win - as you can set them to have no input to enable the button - thus eliminating the need for the inject node for simple cases - but you wouldn't’t want to add all the timer type capability the inject also has into the function node. (indeed I may have such a PR lying around somewhere )
Re parsing flows - giving your function nodes good names can really help… And I’m not sure we need yet another “transformer” node… why have to learn yet another syntax beyond javascript (and maybe jsonata and mustache)… I mean by all means go ahead… but not in core.
Re debug - yes passthrough is def something we need to look at - as it would save forking the flow with all the performance implications that duplication of messages has. Again a simple win would be to add an option to select whether to show the button or a pin. But that would need to be weighed against the UX of having to "edit” the node every time to enable/disable any output vs the click of the button.
But why stop at the inject node, I could replace a change node with a function node. Switch node? Pffff function node with multiple outputs can replace a switch node! Timer with function node? setTimeout(...) for the win!
Of course, this is said with much tongue-in-cheek - either Node-RED is RISC or its CISC! Most flows could probably could be replaced by one giant function node with a giant button on it - the BGBFN - Big Green Buttoned Function Node.
Keep the buttons to a minimum and to the nodes that have buttons - perhaps - are exculsively used in the flow editor. After all, a button isn't triggered in the backend. That the debug node (which also has a button) can produce output to the system console is - perhaps - a misnomer.
How would a button on a function node be used in the backend? Function nodes are backend nodes, not really affected by the frontend. Mixing frontend buttons with backend functionality - for me - is a dark pattern.
Well - of course it wouldn’t… it would only be used for debug/test while the editor is open… and is indeed part of the reason I haven’t actually raised it as a proper PR. Indeed part of the ethos of Node-RED is that nodes should really only do one thing and do it well - so yes potentially more nodes, but easier to work out / read the flow.
But yes BGBFN for the win !
Even with the experience I've built up over the last decade+, I find it incredibly hard to make sense of the core Node-RED code. And as someone who has not be a professional developer for decades, I struggle with the complexity of trying to maintain a dev environment and tooling that would allow me to contribute changes to the core, let alone work out how to include suitable test scripts.
I would be far more comfortable helping with the improvement and fixes for nodes but since core nodes are wrapped up with the Node-RED core, this is back to the same issue.
This is why I'm reluctant to encourage more nodes to be part of core. The tight coupling of the actual core to the nodes creates a massive barrier to people who might otherwise wish to contribute.
This also flows through into documentation as well. There is a wealth of knowledge in the Node-RED community who could contribute to improving documentation but there is significant friction to doing so. Especially if you are not a developer with experience in using GitHub, PR's, etc.
Microsoft had an excellent capability at one point - which I think they've stopped now for some reason - whereby anyone reading their documentation could add comments, suggest improvements or corrections. You didn't need a GitHub account but the comments were added to GitHub for review.
Thinking back to my PHP days as well, one of the things that, in my opinion, made PHP so popular was its accessible documentation. And there as well, if you had an account, you could make comments, suggestions and add examples.
This is all part of improving community engagement. Of course, it requires some checks and balances, but these are well trodden paths.
As for contributed nodes, I think we all agree that the flows catalogue requires some tweaks. @tmichaeltx has also articulated this. The catalogue too should have a comments ability so that people can highlight strengths and weaknesses. We know that time since last update is not always a good measure. We also know that the current scoring is very limited. Having flags for "Node-RED core supported" and "Node-RED endorsed" might be good. If there were any way to occasionally check nodes for insecure dependencies (such as GitHub can do for our own repo's), that might also be another indicator.
My belief is that the very best and most successful open and closed source platforms are those that have a thriving 3rd-party community where initial barriers are low but standards are set and monitored.
Of course, all comments systems should require authenticated users and some basic sanity checks but there are trusted members of the community who would doubtless lend a little time to help.
If you are seriously looking at drawing more things into core, I would suggest that before then, you look to implement some standards for nodes (though noting that some of the code in core nodes is now rather dated ). Then, if you continue to move things into core, that you find a way to reduce the friction for contributing because having to maintain a full Node-RED dev environment just to work on a node is not sustainable in my opinion.
This is absolutely an important point. But even core nodes might not be actively maintained! So finding the right balance is key. Low barriers to participation coupled with improvements to the nodes catalogue as mentioned might be an interesting way to go.
Ah, well .... ... while I know you are thinking about flow anatomy here, let me side-track onto the actual code for nodes for a sec.
The current anatomy, while it worked well to begin with, does not follow best practice and is VERY hard to understand.
Some years ago, I refactored my nodes to make them a lot easier to follow and comprehend. Take a look at my Testbed nodes to see how I structure things now.
In the runtime, I clearly delineate between the different contexts (code that runs when node-red starts even if a node is not added to a flow, code that runs for nodes added to a flow and code that runs when a node receives a msg). I've also started naming runtime script files as *.cjs rather than *.js to prepare for that inevitable time when we need to start the move to ESM.
For the Editor, I split the panel HTML, help HTML and JavaScript into separate files so that they can be properly worked on and linted in an IDE. Code common across several nodes is put into a plugin, common styling is put into a resource CSS file.
As for the visual anatomy in a flow, I think you forgot to mention that a node's internal settings are visible in the Information sidebar (though hidden away under "More...". I think it would be much better to show that info by default rather than hiding it by default.
The ability to add some labels to wires would be helpful. The ability to manipulate wires more would be super nice (but, I recognise, somewhat complex) - like with many diagram tools. Changing from spline to stepped or straight would be good, editing step breaks and spline curves also good. Changing colour, thickness and dash/solid styles would be welcome.
An ability to disable a node without breaking the flow would be welcome. The ability to disable a WIRE (which would break the flow) would also be nice, especially if that meant that Node-RED would no longer clone a msg if all-but-one wire was disabled).
Not forgetting extending the description icon to everywhere that can take a description.
Having a method to more easily get multi-line boxes would be tremendous. Maybe a key combo like many visual tools have (shift+enter for example).
Allowing the Editor to be used as a more natural diagram layout would, I personally think, really extend Node-RED's usefulness and allow people more creative ways to express their logic.
That is what convinced me to stick with Node-RED! While it started life in the world of IoT, it was crafted in a way that allowed it to spread its wings and be far more.
There really isn't anything like Node-RED anywhere in my opinion.
I am convinced that we can find a way to strike a balance though.
Yes, dumping too much to the the debug panel is a performance killer for sure.
That could be a viable option. However, it should not be the only option. This is because the Editor already is very wide. You have the 2 sets of panels and you need a decent width for the main panel, especially when you have a node's config panel open. Fine for people with super-wide monitors perhaps, but not really useful for anyone else.
It might be more useful if Node-RED's panels could be torn off to separate windows. That would be a MASSIVE boon to usability by the way since it would then be feasible to use it across multiple screens as well.
One thing that would also be a MASSIVE boon to productivity for complex flows would be tab groups. These are soooo very useful in the browser and would be super-useful for the Editor.
I think I could as well having recently looked at the UI's for electronic patient record systems used in intensive care units! Lets just say that the lead clinical consultant was considering moving back to paper because of the danger to patients posed by the UI.
I've already got the fix for that Dave. Allowing control from the input msg. I will add that to my testbed node, an easy fix. You could also, I think, have a custom action for the node which would allow enable/disable from the right-click menu.
As a developer, you can mitigate this doing the following:
You must write tests for your flows, and run them before every release.
You must test your flows in a production like environment before every release.
You must pin dependencies, or allow minors and patches updates only.
Create and maintain a private registry with all your dependencies.
Create and maintain a private artifact/container registry with all your production ready builds. This will help you to do rollbacks more reliably and faster.
Deploy your next release in stages (50/50) and keep monitoring it. If something unexpected happens try to fix immediately, and write new tests to cover something you didn’t think about. If you can’t fix it immediately, and it is a critical issue, just rollback to your previous build.
Then if the flow stops working it is probably your fault.
I once did do an analyse and in fact, many nodes aren't maintained or haven't been updated in while.
But the reasons aren't always that the maintainer stopped rather that these nodes are complete, there is nothing more to add. Since Node-RED and its nodes following the Unix principle of do one thing only and do it well, nodes can will been completed.
Also the NodeRED team does a lot to maintain backward compatibility including the internal APIs that nodes use. Thus even old nodes continue to function.
This is not always possible of course and the world of node.js microservices moves VERY fast so you are strongly recommended to keep up. Better to have a series of small upgrades than one almighty one that is almost guaranteed to have issues.
This is particularly true to major upgrades where, for example, the minimum version of node.js may change because we want/need to use newer features.
And at some point in the not too far distant future, we are likely to be forced to move from CommonJS style modules to modern ES Modules. We are already seeing the authors of some common libraries trying to force the issue by only supplying ESM's.
err - so you now have to add two inject nodes (one to enable and one to disable) - in front of the debug node ? so that I can enable/disable the output quickly from the editor ? Plus you now have to learn some new msg syntax of this control message… That all sounds worse to me. But yes the action is a possibility to have a keyboard shortcut to toggle it.
Taking this along and will get back to you when this become relevant!
... while I know you are thinking about flow anatomy here, let me side-track onto the actual code for nodes for a sec.
