Some of you might remember my post from last year about my open-source project, EdgeLinkd, a Rust-based Node-RED compatible run-time engine that can partially run Node-RED workflows.
Today, I'm excited to share the latest progress: EdgeLinkd now integrates the full Node-RED web UI and includes a built-in web server. This means you can open, design, and run Node-RED flows directly in your browser, all powered by a native Rust backend. No external Node-RED installation is required.
Currently, the following nodes have been fully implemented and pass all Node-RED ported unit tests:
My plan is to implement all of Node-RED's built-in nodes and port the complete Node-RED test suite. In my project, only the function node uses the QuickJS JavaScript interpreter to run user scripts inside the node, everything else is implemented purely in Rust.
There is no compatibility with third-party nodes, 3rd-party nodes need to be reimplemented in Rust too.
Sad. That rather undermines one of the key use-cases of Node-RED for me. The fact that it uses the same language as web pages - namely JavaScript (and no, not TypeScript which I intensely dislike).
Compatible with Node-RED that's how I know what is the "correct" behaviour. The function node isn't tested since I don't use Javascript code in Erlang-Red - it's all Erlang.
The testsuite relies on the unit test nodes to define tests. Those can installed in Node-RED and used there. All I did was reimplement them in Erlang and then I could run the test suite.
I test by turning low-level JS unit tests into Rust ones. For example, the object property expression parser includes all its tests directly from Node-RED:
For node "spec" tests, I'm using PyO3 to make my flow engine a Python module, allowing me to port Node-RED tests to Python. This way, I can use Pytest with its nested testing support to compare test results with Node-RED's tests.
Wow, really great success. I was thinking of doing the same but in Golang.
Is there a way to get rid of some of the current nuisances, i.e.
decouple the data storage, currently NR stores the data together with the working code. it would be much better if it give the option to store it in a real database (MySQL, Postgres, etc). There are many benefits, like you could potentially run multiple Node-RED instances connecting to the same data, enabling load balancing and higher availability
If you are refering to the context stores, you can already do it with a custom context store implementation. There is one for Redis. I think it is the one Flowfuse uses to enable HA
Bear in mind you wont be able to restore class objects stored in the context when using that plugin, only data objects, unless you do extra work. You will need to reconstruct the class object using the retrieved data. When storing the state of your class objects in the context, you will need to ensure you serialized it properly so that you can reconstruct it later. It is like working with DAOs
You could create a wrapper around that plugin to automatically serialize and deserialize your data into class objects.
No, context is typically ephemeral. It will largely be only important for the running server, and is usually irrelevant.
What I'm focusing on, and what's relevant here, is the persistent data – flows.json. This is the application's definition itself, independent of the dynamic data it might operate on
You may share global context between multiple instances of the same flow, each running on its own process and own container replica. Stick sessions is an example of sharing contexts using redis. Context being ephemeral or not is a choice and dependent on the use case, not a rule.
Another reason for persistent context stores is for when a container running the flow goes down and your server is configured to recreate it, and restart from where it stopped.
Flowfuse stores flows.json and other stuff in a db. It defaults to local file system but your can easily configure a postgres. Have you tried it?
My experience from creating Erlang-Red is that many problems are non-problems once one starts developing the code. For example, I've been thinking about global and flow context and how to implement that in Erlang-Red. In the end, I will drop both in favour for Erlangs statemachine behaviour or generic server behaviour. Both are good for storing data.
When I began out, I wanted to maintain 100% compatibility to Node-RED. The more I've worked on the project, the more I've realised that 100% won't fly. There are simply things that work better using an "Erlang paradigm" than a NodeJS paradigm. So my aim has become keeping the flow editor much the same but not being afraid to create divergences between Node-RED functionality and Erlang-Red functionality. After all, that's were the learnings are!
On the other hand, learning from ten years of Node-RED development makes implementing the backend simpler. I spent much time just copying functionality and not having to decide what to do. So following the functionality of the nodes means many decisions on what nodes should or should not be doing has already been made.
I've also moved my target audience from Node-RED users (i.e. non-developers, low-coders) to a more developer-centric visual more-coder approach, i.e., I actually encourage adding code to the flows (via a module node). Why? Because Erlang code is compactor than NodeJS code. Because without knowing what developers what to develop, there is no way to know what nodes to offer etc. There is the next problem, how to implement the package management of Node-RED? Answer: don't. Why? The core nodes offer enough work and provide enough functionality to implement simple solutions.
For me, the most important element is not altering the flows.json format, I still want to be able to import and export code from Erlang-Red and be able to import that to Node-RED. This leaves the door open for one day being able to create visually once and execute multiple places. "Write once, execute everywhere" or whatever the Java/JVM selling point was back in the day.
Absolutely! Building on the idea of detaching data to a database, a significant architectural leap for Node-RED would be to split it into two distinct executables, especially when considering a performant language like Rust:
Workflow Editor: This remains your comprehensive, interactive environment for designing, debugging, and managing flows – the UI you interact with. For instance adding it to your main app dashboard.
Workflow Runner: This would be a truly lean, optimized, standalone executable – the single-purpose engine for executing your deployed flows, reading definitions directly from the database.
Implementing this Runner in Rust would be transformative. We're talking about an incredibly small binary with a minimal memory footprint, perfectly suited for high-density, low-resource environments like edge devices, embedded systems, or massive IoT deployments.
Compare this to the current Node.js implementation which typically has a minimum footprint of 70-80MB (but easily scales into 100MB+ for anything serious)
Be aware that this runner has to be able to interact with the editor (design phase) for which it needs to be able to manage flows, allow for inject nodes to trigger messages and send debugging details to the flow editor via a websocket.
All these things this runner won't have to do at execution time.
A runner that you imagine would either need to be able to deactivate the editor communication (that's my chosen solution with Erlang-Red) or there has to be a go-between that facilitates the communication to the editor on the one hand and the start/stopping of the runner when flows are deployed/edited in the editor.
My point is that a runner can be made even slimmer if it doesn't need to have the overhead of editor communication.
Aiming for the flows to be designed in Node-RED and then deployed to a Rust backend isn't a good idea - IMHO. Handling the 100% compatibility is a pain and implementing a Javascript interpreted in Rust might well be non-trivial. You will end up wanting Rust code in the flow, so that the function node might well only speak Rust (as is the case in Erlang-Red). Or you add a Rust function node and then you'll need the editor-backend to be able to run Rust code ...
Also I don't know how Rust works, but having code evaluation makes life a lot simpler. Erlang allows for Erlang code to be evaluated at runtime but also allows code to be compiled at runtime. This is useful for the correct functionality of something like a function node.
P.s. replace 'Rust' with 'Go' or whatever language you like, the sentiments are the same.
You're right, a runner can be significantly slimmer by shedding the overhead of editor communication.
The runner's sole purpose should be to interact with the database, reading flow definitions and executing them. The editor, on the other hand, would have its own integrated "editor runner" for development – essentially how Node-RED works today for local testing.
Once a flow is finalized and deployed, it goes to dedicated "production runners." These production runners are blissfully unaware of the editor or how the flow was built. Their only concern is having access to the flow data from the database to execute.
In that step, you could also "optimise" the flow by removing editor-only nodes (e.g. inject without repeat, debug without system console logging, etc) and normalise away junction nodes so that connections are direct between nodes.
Adding a "compile" step to flow execution could do these "optimisation" to the flow before its sent to the execution engine.
I need to clarify: Edgelinkd was initially a runner without a WebUI, but I found this useless since 90% of the time you need to access hardware. Ideally, the development and deployment platforms should be the same device.
Using dynamic stuff like Msg kills "high performance," even with Rust. For true high performance and scalability, write code directly. My aim's to keep resource use super low so it can run on a 32-bit ARM with just 32MB of memory or something.
Rust is a fully static language for source code, you have to compile the world, so runtime user-written Rust is impossible.
Honestly, if a user can write Rust code, they probably wouldn't be using flow-based programming anyway.
