Use to develop distributed event driven applications

Core Development Feature Requests
1

Hi,

This is less of a feature request for node-red itself since IMO it does not really fit into the core concept of noder-ed. This is more of a thing I would like and would work on, but I would like some feedback on it to know:

How reasonably implementable is the Idea for a singular individual? (I would want to work on it as a side project)
Does the Idea extend the usefulness/ease of use for users / is there already something similar which I just didn't come across?
General feedback on the idea, would you think it's useful or might even use it?

The Problem: node-red only runs on a singular machine, so operating across multiple devices is unintuitive/harder than it has to be.

I've been thinking about modifying node-red to let it function as a distributed system. What I mean by that is probably best described in some sort of a User Story. (node-red+ refers to the modified node-red software)

The user has multiple devices (let's say a phone and a desktop pc), which he wants to automate and potentially work together. The user has the software installed on both pc and phone. Phone and Computer, therefore, form a collection of "clients" (devices that run node-red+ as part of the same distributed system), and Flows are shared between them. To edit the flows, the User can utilize the node-red(+) editor like usual, and changes get propagated to all clients. node-red+ also provides messaging, so the clients can communicate with each other/the backend.

The Main Idea: The User can specify a set of clients for each node so that the node will get executed on the specified clients. (some nodes could even be restricted to certain clients, depending on the device the client runs on)

The coordination needed to pass the messages from one client to the other will be automatically handled.

The "Backend"
The Backend provides the messaging for the clients as well as central storage for the flows and settings.

The "Client"
Each Client runs node-red+

  • provides access to the editor
  • changes to flows made in the editor get committed to the backend/propagated in the distributed system to all other clients

Building upon that Applications for major platforms could be made, that even give access to new device-specific nodes in node-red+.
(App on android could expose a node to change phone settings/wallpaper/run an app etc.)

The impact/use of the Idea:
use node-red to develop distributed event-driven applications with ease.
Flows that include processing in several clients are visible(and editable) in one UI, as a whole.
Would reduce the barrier of entry for people wanting to automate things across devices, control their smart home, etc.

Another Example: User wants to synchronize the clipboard of two devices. Say phone and laptop. So in the node-red+ editor, he takes an event node that triggers every time the clipboard of the device changes (let's say the message payload is the content of the clipboard). He routes that to a node that sets the clipboard. He specifies that the event trigger should run on the first device, while the set clipboard should run on the second. Then he does that again for the other direction and he has set up clipboard sync in a couple of minutes.

Notes:
The Backend should be adjustable in what kind of messaging it provides and how it stores the data.
Messaging should probably be able to work everywhere, and not only when connected to some Network.

2

While not technically in the way you described it, but nearly everything of that can already be done with Node-RED today. Building distributed event-driven applications is what it was conceived for and where it excels. In fact, one of the initial suggestions for "RED" was "Rapid Event Developer". :grinning:

For the "backend" there are ready-to-use solutions like MQTT which handle the messaging part between different devices (not necessarily running Node-RED). Broker systems are the glue in a heterogeneous environment. In fact, I am running dozens of different devices at home (PCs, Pis, micro controllers like the ESP8266) that communicate via an MQTT broker.

You can also create multiple different backends for different tasks, and do the orchestration of those in Node-RED. That's how we use it in our commercial applications at work.

Endless possibilities even today, without modifying a single line of core code. :wink:

Where did this idea come from? Don't get me wrong, but it sounds rather theoretical. :nerd_face: How much practical experience do you have with Node-RED?
Is there a specific use-case you are analyzing or targeting?

3

This project already exists https://github.com/namgk/dnr-editor

Since then various enhancements are already being added to the core on order to make this even more realisable.

2 Likes
4

Where did this idea come from?

I started to use node-red because I wanted to automate something on my phone, for that purpose I wanted a raspberry pi to make API calls to gather information and relay that information to my phone to be further utilized (namely get some things from Reddit API and send links to pictures to my phone to be set as wallpaper).
The API and data wrangling part was very easy to set up, however for relaying that data to my phone I used Tasker and Join (two apps aimed at automating phone behavior), which was more difficult that it had to be (and might not have been the best choice). Then I thought, what if I could have some node-red software running on my phone that was connected to the node-red on the pi so that I could just use a single flow. That flow would start on the pi making the API calls, wrangling the data, and then seamlessly transition to be run on the phone, where it would set the image as wallpaper.

In theory, I would just need to run node-red on the phone, use some messaging solution to get the info there and when messages get received they are processed accordingly. That would be possible today, but as stated, ideally I wanted it all to be in a single node-red flow and not split up over multiple instances.

How much practical experience do you have with Node-RED?

Only a couple of hours max.

Is there a specific use-case you are analyzing or targeting?

In general, an easier barrier of entry when using node-red for distributed flows to make it a valid option for people who do not necessarily want to setup/maintain their own messaging solution (be it MQTT or something else).

I would love to be able to use node-red to supersede tasker and join since they are pretty convoluted in my opinion. I also feel like it would be easier to have all the flows in one view/editor available and not being forced to use multiple editors on multiple devices, where the tight coupling between them could lead to problems when maintaining/altering the functionality.

This project already exists GitHub - namgk/dnr-editor: Distributed Data-Flow Coordination Platform Based on Node-RED

At first glance, it looks like the thing I described. I will look into that in detail now. Thank you for pointing that out!

5

I hope you have a LOT of time on your hands if you decide to start this project. It would be very complex indeed.

And I'm not sure that you would, in the end, solve the problem you actually articulated - the ability to run node-red on the phone with full fidelity and to replace tasker. Because now you've taken on two large projects. Not only to write a fully distributed Node-RED but also to write a Node.js version of Tasker that will run on an Android mobile phone.

Either of these would be major undertakings on their own.

Let's leave aside the viability of a Node.js version of Tasker and just focus on communications. How would this work? You would need to take on a 3rd major aspect - to make cross-WAN communications both safe and easy to set up. Frankly, if you cracked that problem, you could probably afford to pay for quite a few programmers to do the rest of the work from the royalties.

But there are other issues with cross-platform Node.js. There are quite a few Node.js modules that rely on C/C++ or other compiled libraries and these must be compiled for the target platform - either by the module author in advance or by the module installer at install time. On a smartphone, this is also a very difficult, slow and error prone job. You will likely be missing various libraries and the speed of installation would be glacial. And then, what hardware access does Android OS allow you to have? Many features you might take for granted on a PC/Mac you would need to test on a smartphone and either block or warn potential users about.

Don't get me wrong, I'd love to see even just a full-featured Node.js running on Android with libraries that would allow the replacement of Tasker - assuming that both security and performance were up to scratch.

6

Dave,
Do you mean things like the new pluggable message routing (which could be used to send messages to a remote router) and Groups of nodes (which could run remote in a far future)? Or do you mean something else?

I had a quick look at the paper of the dnr-editor, and that is a very nice piece of software too. I can only have deep respect for the capabilities of Mike and Nam. But what a pity to see that such projects get unmaintained and out of sync with the original Node-RED...

7

Correct. Both of those are steps on the way.

1 Like
8

As you've probably found out already, Node-RED can be run on Android by at least two means. Either by installing the (unofficial) REDMobile app or by installing Termux and then running NR on it's command line. REDMobile has some phone specific nodes but IIRC I had varying experiences with them. Termux itself comes with some CLI utils to interact with the phone which you could (at least in theory) run using an exec node.

Both of these options are very heavy on resource use and I only use Termux randomly to test things while on my tablet. For a phone app you'd definitely want something that's designed to be run on a phone. But you could try them out first if you haven't yet to get a feel of how feasible something like you described would be.

2 Likes
9

So, I have looked at the DNR Project, which looked promising, but since it is based on an older node-red version bringing this up to date would require a lot of work. Since the desired use-case is a long way out I think getting node-red support needed features as a foundation would be a better use of time.

Regarding that, I have found the "Design: Runtime Editor Split" proposal(?). However, it is a bit confusing to me what the current state of that is. In the new designs repo, it is not listed, but in the old wiki, it seems incomplete. Was this design scrapped, completed, integrated into another design, or just forgotten about?

Assuming that this feature makes it into node-red. The next step would be to support the distributed use-case (one could view this as integrating the work from DNR into node-red).

Then, packing a node-red runtime into an android app, that gets configured to be part of the distributed system managed by one editor could be done.

At that point, one could create a service that provides WAN Messaging with MQTT and Storage for the data, and maybe even host the editor on the internet (behind auth). The service would probably not be free (someone has to pay for the used resources), but if one would like he/she could set up their own service.

I would hope to let the project not deviate too far from standard node-red, to keep compatibility and reduce the amount of maintenance necessary. The power of a system like node-red heavily depends on the compatibility to communicate/access multiple devices/APIs. Therefore I would like to keep the community of node-red and not venture off into uncharted territories on my own.

This however all builds on the assumption that editor and runtime get split as well as proper distributed node-red support. If that doesn't make it to standard node-red I think this would not be a feasible undertaking for an individual.

So the main question now becomes:

Was this design scrapped, completed, integrated into another design, or just forgotten about?

10

The runtime/editor split work was done to a large degree in the 0.20 release - https://nodered.org/blog/2019/03/12/version-0-20-released#runtimeeditor-split

That did the work to split the runtime and editor into separate npm modules. There remains a close dependency between them - but that dependency is now a more well-defined runtime API.

But I'm also less convinced that a complete split is needed for a distributed model. We already support running with the editor disabled - which is what you'd want for the 'remote' node-red instances. The runtime still exposes its HTTP API for remote administration. The only benefit would be the disk space saved by not having the @node-red/editor-client module installed.

As @dceejay mentioned, there are already a number of pieces coming rapidly down the roadmap that help move towards enabling a distributed model.

The Grouping feature released in 1.1 will (eventually) include the ability to add custom meta-data to groups of nodes. That meta data could be used to identify where particular nodes should be run.

The pluggable message routing will allow for custom code to added to the message routing layer - enabling a flow to span multiple runtimes. A missing piece here is having the additional hooks in the runtime to be able to add custom code to the 'deploy' path, which could be used to distribute the flow to the different parts. That's something I'll be looking at once we get 1.2 done in the next week or two.

1 Like
11

That is great to hear!

I think the best course of action, for now, would be to follow along with the development and review this project once those features are in.

1 Like
12

I think that you need to spend some time looking at this aspect. Understanding the size and complexity of running a large Node.js app on Android.

13

Will do once the foundation is laid and I have time to start the project.
While the ultimate goal would be to replace Tasker, there probably will be an integration node, so that one could use Tasker for things not yet doable using the node-red approach. While it would be great to have all the functionality of Tasker directly translated to node-red, I think it will be a gradual process adding nodes/features as time goes on.

Maybe restricting the nodes runnable on certain runtimes might "solve" the problem. One would have to do some calculations elsewhere, but if some nodes just don't work on a runtime deployed in Android...

For now, I cannot further comment on the complexities of trying to use Node.js on Android, since I have no experience with using JS or Node.js. Once I get around to it I will keep your advice in mind.

14

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.