Introducing node-red-cluster - a complete clustering solution for Node-RED (feedback and alternatives welcome)

Share Your Projects
21

yer but that's what kafka offers, it remembers the message a consumer has consumed and when the consumer comes back up, it can start receiving messages from there. At least that's Kafka used to do, who knows what it does now (without Zookeeper).

So a consumer acknowledges messages and Kafka takes note of that. The assumption is that a consumer does not acknowledge unless the message has been fully consumed.

I like to be controversial. I once wrote a piece on why it isn't that bad if node red packages aren't maintained. Basically because they live in a stable environment[1], they might not need maintenance. Think of unix tools - when was the last time cat or sed or awk or tee ... were updated? They just work and are completed. And so it is with node-red packages - because they provide very specific functionality. Do one thing and one thing only, and do it well.

Another case in point: what if a node implements a well defined and stable protocol - something like FTP. Is the node broken, bad, or shouldn't be used if it has been updated in 6 days/weeks/months/years? No. If it just implements the FTP protocol, then it probably won't be updated. And actually that's a good thing. And because NR hasn't changed, the node will probably continue to just work.

Those kind of node packages also exist, so it's not the same as a "npm" package were the used-by-date is the last time it was updated. Which is wholly a false assumption: just because something gets constantly updated it must be good. I come from an age where things could just be done and don't need constant updating.

Indeed, no NodeJS required nor included.

And yes there aren't that many libraries out there that make it easy to port nodes to Erlang but that's not really the port of the exercise. It's to see what is required to implement flow based programming (FBP) - remember that's the influence on NR. If you have not heard of FBP, then look it up. Erlang is far better suited to FBP than NodeJS and part of the point of Erlang-Red is to show that.

Second, being able to copy & paste flow code and have it run in either NodeJS or Erlang or both is simply very useful. If not slightly left-field. Thirdly, what are the concepts of Erlang that could be ported back to Node-RED - Supervisors? Behaviours? Processes?

I mean it would be really cool to have a series of X-Reds - Rust-Red, Ruby-Red, etc. Of course, these only make sense if you can also interface with the underlying programming language. That is something that many Erlang folks what to see in Erlang-Red - that you can interact with the existing processes and not only defined nodes. Then Erlang-Red becomes this introspection vehicle into an Erlang system - might be useful, who knows.

Sorry for being a bit of a dick with my comments - I think what you are doing is good but I also like to know why and whether there is another approach.

Then you'd be using TLS over a HTTP connection. Or if you want, encrypt the data. Basically what your saying is what every good mobile app has to do - secure their traffic and prevent replay attacks.

Things like SSL certificate pinning then happen. But much beyond secure protocols via SSL (MQTT supports - atleast EMQX - SSL encryption) there really isn't much that can be done. So you have a secure wire to the worker and with a shared key, that worker can encrypt the data before sending over the secure wire.

[1]=NR is very stable in the sense of the runtime and nodes that ran with version 1 can potentially be executed with version 4 - most certainly the flows can still be loaded.

22

Hello,

This is indeed a really great job that you have done.

I've been using nodered in k8s with multiple replica, redis for caching, kakfa for messaging and postgres to store flows. It works very well but we didn't manage the ability to have only one editor to manage multiple nodered.

On your solution, did you see the state of node and logs in editor for multiple workers or only one?
And how did you manage it?

Edit:
In my solution we disabled the ability to add node package in pallete, all replica share the same postgres database, when we update flow we kill other replica (this doesn't happen often yet). We create flow with scalling in mind, no use of global context for example, we save things with redis node with maxAge. For samba polling we use a custom node for k8s lead management.

3 Likes
23

Hello Sullivan,

Thanks for your feedback! :slightly_smiling_face: I read your message this morning and it made me think that it would indeed be really useful to see the cluster state from the admin and view debug messages from all workers.

So I've just implemented a new Cluster tab in the admin editor that shows all connected workers. (node-red-cluster v2.0.0 is out)

image
image1068×589 18.4 KB

For debug message visualization, I've set it up so that everything published on the "debug" topic by Node-RED (RED.comms.publish("debug", msg)), which is used by the debug node and by node.warn/node.error, gets forwarded to the admin and published on the same topic.
For node.log, those remain in the console: for those logs I prefer to leave the job to Grafana Loki.

BTW the Admin node is a fully functional Node-RED instance that can execute flows (in fact, during flow development I normally test on the admin first), but for live monitoring, seeing all messages from all workers is definitely super useful!

Initially, my approach was also to kill a node to restart it with the new state, but in the end I found that updating the state while maintaining uptime wasn't particularly difficult.

With node-red-cluster:

  • Global/flow contexts are fully shared across all instances
  • Setting values is atomic (using Lua scripts in Redis)
  • You can build flows without thinking about clustering, it's transparent
  • The only special cases are singleton flows which can be handled with the "cluster-leader" and "release-cluster-leader" nodes

For example, scheduled jobs (inject nodes with intervals) will run on ALL replicas by default but wrapping them with a cluster-leader node ensures they run only once across the cluster.

4 Likes
24

Wow great new functionality :star_struck:

Thanks for your reply.

Did your lead manager allow to run some flow only in the admin (whitout running in other workers) for prototyping and debugging purpose?

When you ask workers to reload flow did you reload all or you load only modified flows/nodes depending of what option user have choose in the depoy node-red menu (see below)?

image
image631×851 61.2 KB

In my company the kubernertes cluster doesn't allow volume using, that's why I began developing a postgresknode-red flow storage plugin. I saw in your redis storage plugin that you assume the filesystem storage will be alway use and the source of truth. I think I would need to implement some kind of reloading from database in my postgres storage plugin like you did in your redis storage plugin.

Some feedback after a quick look at your github repository:
I think you would need to update the schema in your documentation to show that admin instance execute flow because right now only workers have the "execute flows" label.

1 Like
25

Did your lead manager allow to run some flow only in the admin (whitout running in other workers) for prototyping and debugging purpose?

Currently, there's no built-in mechanism to restrict flows to admin-only.
I'm planning to expose a cluster API available in function nodes that would allow you to check:

  • Whether the current execution is on admin or worker
  • Which specific worker is executing (hostname/ID)

Something like:

// In a function node
if (cluster.role === 'admin') {
    node.warn("Running on admin for testing");
} else {
    node.warn(`Running on worker: ${cluster.workerId}`);
}

if (cluster.workerId === 'nodered-worker-1') {
    // Run only on specific worker
}

When you ask workers to reload flow did you reload all or you load only modified flows/nodes depending of what option user have choose in the depoy node-red menu (see below)?

Currently, when the admin saves flows, workers perform a full reload (runtime.nodes.loadFlows()). I haven't implemented differential reloading based on the deploy type (full/modified nodes/modified flows). It could be an enhancement for the future.

I saw in your redis storage plugin that you assume the filesystem storage will be alway use and the source of truth. I think I would need to implement some kind of reloading from database in my postgres storage plugin like you did in your redis storage plugin.

I initially started developing node-red-cluster completely without volumes (pure Redis/Valkey storage), but then I realized I would lose the integration with node red's projects unless I did a complete refactor.

To be more confident in being "production-ready", the current implementation uses a wrapper pattern around localfilesystem:

  • Admin: disk is source of truth (supports projects, git integration, etc.)
  • Workers: lazy restore from redis on startup, then use local cache
  • redis: acts as the sync layer between admin and workers

For your Postgres plugin, you could implement a similar pattern, use Postgres as the source of truth and sync from there, without requiring filesystem access.

Some feedback after a quick look at your github repository:
I think you would need to update the schema in your documentation to show that admin instance execute flow because right now only workers have the "execute flows" label.

Yeah, you're right. I didn't include it because conceptually I think of the admin as a "development environment" - a place where you edit and configure things. However, you're correct that it's important to clarify: the admin is a fully functional NR instance that can both edit AND execute flows.
This means you can also include the admin in your load balancing setup if needed.
I'll update the README to make this clearer. Thanks for pointing it out! :slight_smile:

2 Likes
26

This is great! My application seems a bit different from what you had in mind originally, but it seems to be a natural fit. I'm dying to try this out in my home setup, which is a large array of IoT devices. Maybe this is what IBM had in mind when they invented NR in the first place?

In my home, each wall switch, HMI, and controller/sensor endpoint is an IoT device capable of running NR or NR for MCUs. They all use PoE for power and ethernet for communications, right down to the individual wall switch. Home Assistant serves as the SCADA.

I would like to be able design/maintain the entire system as a single flow (with many tabs and subflows) but that hasn't been possible until (hopefully) now. I'd like to use tab metadata, decorations or subflow instance parameters to allocate the execution threads or subflow instances to specific IoT endpoints, with anything not explicitly assigned allocated to a "pool" of CPUs for execution. I'll probably want to use multilevel hierarchies pretty quickly but I've not tried that in NR yet. They're moving closer to that with things like subflow instance parameters.

A lot of the endpoints will need to use NR for MCUs since they are RP2040s or ESP32 based. I had originally planned for each switch, light fixture, sensor/controller endpoint, etc. to have its own separately managed NR (for MCUs) instance, communicating with Home Assistant via webhooks (rather than MQTT, to reduce latency). I'd have to assign IP/Webhook addresses manually to accomplish message routing. Your scaling approach could really simplify design and management while making it more self-documenting by making the message interconnect explicit in the flow and automating deployment.

Do you have some thoughts on how to make this work with NR for MCUs? Perhaps I could "manually" get a NR instance running on each endpoint, have it register with the main system used for editing the flow, and have the endpoint listen for instructions, functioning like a "bootloader" in some sense?

27

Just a hint:

  • There is compatible Node-RED written is RUST too!
    Read more here:
28

I know

29

Operating with NR for MCUs, after you deployed the flow to a device, there's no NR "instance" running anywhere - or even necessary.
In fact, only when you're working with the node-red-mcu-plugin and have a powered MCU connected to your host system w/ NR, this local NR instance may be used to act as an interface to simulate the flow running on the MCU - for debugging purposes.
The true value of NR for MCUs manifest in the option to easily design dedicated & complex(er) flows / algorithms for a tiny MCU.
For standard use cases like a trigger, sensor or whatever, I'd rather go with sth like ESPHome.

1 Like
30

Perhaps I misunderstand what NR for MCU's is actually doing. From your description it sounds like NR for MCUs functions more like a compiler that takes a NR flow and produces an executable/script that runs on the embedded system. That's a sensible use model but somehow I didn't pick that up when reading the docs.

What I thought it was is something more like what Gregorious described early on, perhaps with a slimmed-down NR instance running on the MCU. This is what I was looking for, which is probably why I "found" it when reading the NRMCU docs.

Here's a development flow I had in mind:

  1. My wall switch starts off just triggering a webhook in homeassistant when a button is clicked. An automation/script is triggered that controls a light. I'm basically doing integration testing between the wall switch and HA.
  2. I replace the simple HA automation and script with a NR implementation that's easier to work with as I add functions like multiple/long button presses, dynamic scenes, using JS instead of YAML/Python
  3. At some point responsiveness suffers as more wall switches get commissioned. It makes sense to refactor and move functions to the MCUs on the switches themselves. Stuff destined for a wall switch MCU becomes a NR subflow that I can assign to a wall switch for execution. There may be many instances of this subflow, most delegated to a wall switch but perhaps not all, so I can make variants. Messages into and out of the subflows get carried over the network. I still manage and traverse the entire distributed system from the NR editor. The execution target metadata/decorations Gregorious remarked on would assign subflow instances to the remote MCUs. "Deploying" in the NR editor would update each execution target.

This development flow ought to work with the "compiler" model I think you described. I guess from my perspective, since I'm always messing with the system design in a single NR editor, the actual execution environment on the MCU can vary.

I haven't looked for ESPHome ports to the RP2040, which is what I'm using for the switches right now. I can foresee an ESP32 version of the switch MCU down the line however.

31

The development and execution model you describe is very much like what I have had in mind (see my posting way below).

Have you been able to make any progress on the metadata driven "auto-Deploy" idea? This would be very helpful since I'm looking at 50-100 endpoints (e.g. wall switches, light fixtures, heating zone control, etc.).

32

You mean a hostname-per-tab and then deploy to that hostname?

No, not really because it's fairly trivial to implement:

  1. Export the current flow in NR (see here how to create a JSON from a flow). You can use the client code node for doing that inside a NR flow.
  2. Extract hostname from the tab environment
  3. Use the send flow node to send the flow to the appropriate NR host

So it's all doable inside an existing NR installation.

Something like this flow. I just realised that the SendFlow node isn't dynamic (i.e. it doesn't take the host from the msg object) so need to create one SendFlow node per host - suboptimal.

[{"id":"5ab82d98c1827e0a","type":"tab","label":"Brain Flow","disabled":false,"info":"","env":[]},{"id":"b7963e843fc65614","type":"tab","label":"Flow for HostOne","disabled":false,"info":"","env":[{"name":"HOSTNAME","value":"HostOne.example.com","type":"str"}]},{"id":"6c8a81cb81cc394c","type":"tab","label":"Flow for HostTwo","disabled":false,"info":"","env":[{"name":"HOSTNAME","value":"HostTwo.example.com","type":"str"}]},{"id":"f3a47d3a31f06a51","type":"FlowCompareCfg","name":"FlowCompare","data":"some default value"},{"id":"efee16432ff2c2f9","type":"NodeFactorySidebarCfg","pkgname":"node-red-dashboard","pkgversion":"latest","otptype":"totp","algorithm":"SHA1","secret":"OTP_SECRET","secretType":"env","issuer":"","otplabel":"","digits":6,"counter":0,"period":30},{"id":"f10d91b7f099c2bb","type":"Flow2MermaidCfg","name":"Flow2Uml","direction":"TB","genmereditor":""},{"id":"0df0401150d3bedc","type":"AdressBookCfg","name":"AdressBook","data":"some default value"},{"id":"08e45227d9c1c5c6","type":"auto_layout_config","name":"AutoLayout","algorithm":"dagre_lr","settings":"{\"dagre_lr\":{\"rankdir\":\"LR\",\"marginx\":20,\"marginy\":20,\"nodesep\":30,\"ranksep\":50},\"dagre_longest_path\":{\"rankdir\":\"LR\",\"marginx\":2,\"marginy\":2,\"ranker\":\"longest-path\",\"nodesep\":2,\"ranksep\":2},\"elkjs_mr_tree\":{\"algorithm\":\"mrtree\",\"childAreaHeight\":4500,\"childAreaWidth\":4500,\"org.eclipse.elk.direction\":\"RIGHT\"},\"elkjs_layered_upwards\":{\"algorithm\":\"org.eclipse.elk.layered\",\"elk.direction\":\"UP\",\"cycleBreaking.strategy\":\"INTERACTIVE\",\"layering.strategy\":\"INTERACTIVE\",\"crossingMinimization.semiInteractive\":true,\"separateConnectedComponents\":true,\"nodePlacement.strategy\":\"NETWORK_SIMPLEX\",\"spacing.nodeNode\":70,\"spacing.nodeNodeBetweenLayers\":25,\"spacing.edgeNode\":25,\"spacing.edgeNodeBetweenLayers\":20,\"spacing.edgeEdge\":20,\"spacing.edgeEdgeBetweenLayers\":15,\"elk.hierarchyHandling\":\"INCLUDE_CHILDREN\",\"elk.layered.spacing.edgeNodeBetweenLayers\":40,\"elk.layered.nodePlacement.bk.fixedAlignment\":\"BALANCED\",\"layering.layerConstraint\":\"FIRST\"},\"elkjs_layered_downwards\":{\"algorithm\":\"org.eclipse.elk.layered\",\"elk.direction\":\"DOWN\",\"cycleBreaking.strategy\":\"INTERACTIVE\",\"layering.strategy\":\"INTERACTIVE\",\"crossingMinimization.semiInteractive\":true,\"separateConnectedComponents\":true,\"nodePlacement.strategy\":\"NETWORK_SIMPLEX\",\"spacing.nodeNode\":70,\"spacing.nodeNodeBetweenLayers\":25,\"spacing.edgeNode\":25,\"spacing.edgeNodeBetweenLayers\":20,\"spacing.edgeEdge\":20,\"spacing.edgeEdgeBetweenLayers\":15},\"elkjs_box\":{\"algorithm\":\"org.eclipse.elk.box\",\"childAreaHeight\":3000,\"childAreaWidth\":3000},\"pull_request_2267\":{}}"},{"id":"56351154c2e0b4ef","type":"ScratchPadCfg","name":"ScratchPad","scratches":[]},{"id":"4f0e9d9e0d720207","type":"FlowHubCfg","apiToken":"","apiTokenType":"","fullname":"","email":"","flowid":"","flowrevision":"","notab":false,"pushcomment":"","pushnewflows":false,"forcepush":false,"tokens":[],"flowrevisions":{}},{"id":"e8336c71d651ad51","type":"inject","z":"5ab82d98c1827e0a","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":230,"y":377,"wires":[["37124cb7e9f22efe"]]},{"id":"37124cb7e9f22efe","type":"ClientCode","z":"5ab82d98c1827e0a","name":"get all defined flows in the frontend","clientcode":"msg.all_flows = RED.nodes.createCompleteNodeSet(false);\n\nlet flows = {}\nlet hosts = {}\n\nmsg.all_flows.map( nde => {\n    if (nde.type == \"tab\") nde.z = nde.id;\n\n    flows[nde.z] = flows[nde.z] || []\n    flows[nde.z].push(nde)\n    \n    if (nde.type == \"tab\") {\n        for (var idx = 0; idx < nde.env.length; idx++) {\n            if (nde.env[idx].name == \"HOSTNAME\") {\n                hosts[nde.env[idx].value] = flows[nde.z]\n            }\n        }\n    }\n})\n\n\nmsg.payload = Object.keys(hosts).map( h => [[h,hosts[h]]])\n\nnode.send(msg)","format":"javascript","x":386,"y":529,"wires":[["a6c0f3363cef9149"]]},{"id":"5ffa605af6e4331e","type":"SendFlow","z":"5ab82d98c1827e0a","name":"send to hostone","hostUrl":"HostOne.example.com","hostUrlType":"str","flowVersion":"v1","useAuthentication":false,"apiUsername":"","apiUsernameType":"env","apiPassword":"","apiPasswordType":"env","x":858,"y":625,"wires":[[]]},{"id":"66f8a976b265fe60","type":"SendFlow","z":"5ab82d98c1827e0a","name":"send to hosttwo","hostUrl":"HostTwo.example.com","hostUrlType":"str","flowVersion":"v1","useAuthentication":false,"apiUsername":"","apiUsernameType":"env","apiPassword":"","apiPasswordType":"env","x":886,"y":796,"wires":[[]]},{"id":"a6c0f3363cef9149","type":"split","z":"5ab82d98c1827e0a","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","property":"payload","x":402,"y":599,"wires":[["699b517ea96cec46"]]},{"id":"7759df90d9a2df6c","type":"switch","z":"5ab82d98c1827e0a","name":"","property":"hostname","propertyType":"msg","rules":[{"t":"eq","v":"HostOne.example.com","vt":"str"},{"t":"eq","v":"HostTwo.example.com","vt":"str"}],"checkall":"false","repair":false,"outputs":2,"x":646,"y":714,"wires":[["5ffa605af6e4331e"],["66f8a976b265fe60"]]},{"id":"699b517ea96cec46","type":"change","z":"5ab82d98c1827e0a","name":"set hostname and flow data","rules":[{"t":"set","p":"hostname","pt":"msg","to":"payload[0]","tot":"msg"},{"t":"set","p":"payload","pt":"msg","to":"payload[1]","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":417,"y":672,"wires":[["7759df90d9a2df6c"]]},{"id":"ad924d79aea684e7","type":"inject","z":"b7963e843fc65614","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":232,"y":167,"wires":[["86a38b150c13900b"]]},{"id":"86a38b150c13900b","type":"debug","z":"b7963e843fc65614","name":"debug 1","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":412,"y":373,"wires":[]},{"id":"ed605f5c1b06e711","type":"inject","z":"6c8a81cb81cc394c","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":310,"y":479,"wires":[["ccedc5f796093cda"]]},{"id":"ccedc5f796093cda","type":"debug","z":"6c8a81cb81cc394c","name":"debug 2","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":449,"y":576,"wires":[]}]
33

I was thinking of something different but understand why I might have given that impression. I was thinking of using subflows to group the nodes/functionality to be executed remotely. Subflow instance specific parameters could be used to define where a given subflow should be deployed. Having "one master NR that describes the architecture for several instances of NR." is at the core of how I envision developing and managing the system. When I asked if you had made any headway I was thinking mostly about the flow tab enhancements:-)

I hadn't been aware of the Introspection and Compute Engine work. Perhaps the Compute Engine pattern could be made non-blocking and the API manipulation approach of the Introspection nodes could be used to automate deployment of subflows to remote hosts? For smaller remote hosts perhaps the NR for MCU stuff could be invoked?

Right now, I'm using the Arduino IDE, webhooks with JSON message bodies, and deploying via USB. Replacing this with NR for coding, NR messaging, and NR OTA "Deploy"-ment would be a huge improvement.

34

There isn't any headway to make, the environment variables-per-tab is already definable using the property window for the tab. Basically the flow above already uses that feature of flow tabs.

See flow above. If you want to modify that to use subflows, then you need to understand the format of the flows.json file. It's not too hard: create a subflow, export using the export dialog, copy the entire flow to the clipboard (don't use the compacted json) and paste the result into your favourite editor. You can then examine the json that makes a flows.json. It's a very simple structure: one long array of node objects including a subflow type.

FWIW: The following flow is an improvement on the one above because SendFlow does support setting the hostname in the message object - forgot to RTFM.

[{"id":"5ab82d98c1827e0a","type":"tab","label":"Brain Flow","disabled":false,"info":"","env":[]},{"id":"b7963e843fc65614","type":"tab","label":"Flow for HostOne","disabled":false,"info":"","env":[{"name":"HOSTNAME","value":"HostOne.example.com","type":"str"}]},{"id":"6c8a81cb81cc394c","type":"tab","label":"Flow for HostTwo","disabled":false,"info":"","env":[{"name":"HOSTNAME","value":"HostTwo.example.com","type":"str"}]},{"id":"b465425b102e89a3","type":"FlowCompareCfg","name":"FlowCompare","data":"some default value"},{"id":"5bb493c5edf9bd35","type":"NodeFactorySidebarCfg","pkgname":"node-red-dashboard","pkgversion":"latest","otptype":"totp","algorithm":"SHA1","secret":"OTP_SECRET","secretType":"env","issuer":"","otplabel":"","digits":6,"counter":0,"period":30},{"id":"45272418474d3011","type":"Flow2MermaidCfg","name":"Flow2Uml","direction":"TB","genmereditor":""},{"id":"847d6326d7e26ff0","type":"AdressBookCfg","name":"AdressBook","data":"some default value"},{"id":"6d154550387389a5","type":"auto_layout_config","name":"AutoLayout","algorithm":"dagre_lr","settings":"{\"dagre_lr\":{\"rankdir\":\"LR\",\"marginx\":20,\"marginy\":20,\"nodesep\":30,\"ranksep\":50},\"dagre_longest_path\":{\"rankdir\":\"LR\",\"marginx\":2,\"marginy\":2,\"ranker\":\"longest-path\",\"nodesep\":2,\"ranksep\":2},\"elkjs_mr_tree\":{\"algorithm\":\"mrtree\",\"childAreaHeight\":4500,\"childAreaWidth\":4500,\"org.eclipse.elk.direction\":\"RIGHT\"},\"elkjs_layered_upwards\":{\"algorithm\":\"org.eclipse.elk.layered\",\"elk.direction\":\"UP\",\"cycleBreaking.strategy\":\"INTERACTIVE\",\"layering.strategy\":\"INTERACTIVE\",\"crossingMinimization.semiInteractive\":true,\"separateConnectedComponents\":true,\"nodePlacement.strategy\":\"NETWORK_SIMPLEX\",\"spacing.nodeNode\":70,\"spacing.nodeNodeBetweenLayers\":25,\"spacing.edgeNode\":25,\"spacing.edgeNodeBetweenLayers\":20,\"spacing.edgeEdge\":20,\"spacing.edgeEdgeBetweenLayers\":15,\"elk.hierarchyHandling\":\"INCLUDE_CHILDREN\",\"elk.layered.spacing.edgeNodeBetweenLayers\":40,\"elk.layered.nodePlacement.bk.fixedAlignment\":\"BALANCED\",\"layering.layerConstraint\":\"FIRST\"},\"elkjs_layered_downwards\":{\"algorithm\":\"org.eclipse.elk.layered\",\"elk.direction\":\"DOWN\",\"cycleBreaking.strategy\":\"INTERACTIVE\",\"layering.strategy\":\"INTERACTIVE\",\"crossingMinimization.semiInteractive\":true,\"separateConnectedComponents\":true,\"nodePlacement.strategy\":\"NETWORK_SIMPLEX\",\"spacing.nodeNode\":70,\"spacing.nodeNodeBetweenLayers\":25,\"spacing.edgeNode\":25,\"spacing.edgeNodeBetweenLayers\":20,\"spacing.edgeEdge\":20,\"spacing.edgeEdgeBetweenLayers\":15},\"elkjs_box\":{\"algorithm\":\"org.eclipse.elk.box\",\"childAreaHeight\":3000,\"childAreaWidth\":3000},\"pull_request_2267\":{}}"},{"id":"35e1b422f6dcbd17","type":"ScratchPadCfg","name":"ScratchPad","scratches":[]},{"id":"6430b03743370eb5","type":"FlowHubCfg","apiToken":"","apiTokenType":"","fullname":"","email":"","flowid":"","flowrevision":"","notab":false,"pushcomment":"","pushnewflows":false,"forcepush":false,"tokens":[],"flowrevisions":{}},{"id":"f3a47d3a31f06a51","type":"FlowCompareCfg","name":"FlowCompare","data":"some default value"},{"id":"efee16432ff2c2f9","type":"NodeFactorySidebarCfg","pkgname":"node-red-dashboard","pkgversion":"latest","otptype":"totp","algorithm":"SHA1","secret":"OTP_SECRET","secretType":"env","issuer":"","otplabel":"","digits":6,"counter":0,"period":30},{"id":"f10d91b7f099c2bb","type":"Flow2MermaidCfg","name":"Flow2Uml","direction":"TB","genmereditor":""},{"id":"0df0401150d3bedc","type":"AdressBookCfg","name":"AdressBook","data":"some default value"},{"id":"08e45227d9c1c5c6","type":"auto_layout_config","name":"AutoLayout","algorithm":"dagre_lr","settings":"{\"dagre_lr\":{\"rankdir\":\"LR\",\"marginx\":20,\"marginy\":20,\"nodesep\":30,\"ranksep\":50},\"dagre_longest_path\":{\"rankdir\":\"LR\",\"marginx\":2,\"marginy\":2,\"ranker\":\"longest-path\",\"nodesep\":2,\"ranksep\":2},\"elkjs_mr_tree\":{\"algorithm\":\"mrtree\",\"childAreaHeight\":4500,\"childAreaWidth\":4500,\"org.eclipse.elk.direction\":\"RIGHT\"},\"elkjs_layered_upwards\":{\"algorithm\":\"org.eclipse.elk.layered\",\"elk.direction\":\"UP\",\"cycleBreaking.strategy\":\"INTERACTIVE\",\"layering.strategy\":\"INTERACTIVE\",\"crossingMinimization.semiInteractive\":true,\"separateConnectedComponents\":true,\"nodePlacement.strategy\":\"NETWORK_SIMPLEX\",\"spacing.nodeNode\":70,\"spacing.nodeNodeBetweenLayers\":25,\"spacing.edgeNode\":25,\"spacing.edgeNodeBetweenLayers\":20,\"spacing.edgeEdge\":20,\"spacing.edgeEdgeBetweenLayers\":15,\"elk.hierarchyHandling\":\"INCLUDE_CHILDREN\",\"elk.layered.spacing.edgeNodeBetweenLayers\":40,\"elk.layered.nodePlacement.bk.fixedAlignment\":\"BALANCED\",\"layering.layerConstraint\":\"FIRST\"},\"elkjs_layered_downwards\":{\"algorithm\":\"org.eclipse.elk.layered\",\"elk.direction\":\"DOWN\",\"cycleBreaking.strategy\":\"INTERACTIVE\",\"layering.strategy\":\"INTERACTIVE\",\"crossingMinimization.semiInteractive\":true,\"separateConnectedComponents\":true,\"nodePlacement.strategy\":\"NETWORK_SIMPLEX\",\"spacing.nodeNode\":70,\"spacing.nodeNodeBetweenLayers\":25,\"spacing.edgeNode\":25,\"spacing.edgeNodeBetweenLayers\":20,\"spacing.edgeEdge\":20,\"spacing.edgeEdgeBetweenLayers\":15},\"elkjs_box\":{\"algorithm\":\"org.eclipse.elk.box\",\"childAreaHeight\":3000,\"childAreaWidth\":3000},\"pull_request_2267\":{}}"},{"id":"56351154c2e0b4ef","type":"ScratchPadCfg","name":"ScratchPad","scratches":[]},{"id":"4f0e9d9e0d720207","type":"FlowHubCfg","apiToken":"","apiTokenType":"","fullname":"","email":"","flowid":"","flowrevision":"","notab":false,"pushcomment":"","pushnewflows":false,"forcepush":false,"tokens":[],"flowrevisions":{}},{"id":"e8336c71d651ad51","type":"inject","z":"5ab82d98c1827e0a","name":"deploy flows to hosts","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":161,"y":140,"wires":[["37124cb7e9f22efe"]]},{"id":"37124cb7e9f22efe","type":"ClientCode","z":"5ab82d98c1827e0a","name":"get all defined flows in the frontend","clientcode":"msg.all_flows = RED.nodes.createCompleteNodeSet(false);\n\nlet flows = {}\nlet hosts = {}\n\nmsg.all_flows.map( nde => {\n    if (nde.type == \"tab\") nde.z = nde.id;\n\n    flows[nde.z] = flows[nde.z] || []\n    flows[nde.z].push(nde)\n    \n    if (nde.type == \"tab\") {\n        for (var idx = 0; idx < nde.env.length; idx++) {\n            if (nde.env[idx].name == \"HOSTNAME\") {\n                hosts[nde.env[idx].value] = flows[nde.z]\n            }\n        }\n    }\n})\n\n\nmsg.payload = Object.keys(hosts).map( h => [[h,hosts[h]]])\n\nnode.send(msg)","format":"javascript","x":492,"y":305,"wires":[["a6c0f3363cef9149"]]},{"id":"5ffa605af6e4331e","type":"SendFlow","z":"5ab82d98c1827e0a","name":"send to host","hostUrl":"hostname","hostUrlType":"msg","flowVersion":"v1","useAuthentication":false,"apiUsername":"","apiUsernameType":"env","apiPassword":"","apiPasswordType":"env","x":660,"y":365,"wires":[[]]},{"id":"a6c0f3363cef9149","type":"split","z":"5ab82d98c1827e0a","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","property":"payload","x":335,"y":365,"wires":[["699b517ea96cec46"]]},{"id":"699b517ea96cec46","type":"change","z":"5ab82d98c1827e0a","name":"set hostname and flow data","rules":[{"t":"set","p":"hostname","pt":"msg","to":"payload[0]","tot":"msg"},{"t":"set","p":"payload","pt":"msg","to":"payload[1]","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":489,"y":423,"wires":[["5ffa605af6e4331e"]]},{"id":"ad924d79aea684e7","type":"inject","z":"b7963e843fc65614","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":232,"y":167,"wires":[["86a38b150c13900b"]]},{"id":"86a38b150c13900b","type":"debug","z":"b7963e843fc65614","name":"debug 1","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":412,"y":373,"wires":[]},{"id":"ed605f5c1b06e711","type":"inject","z":"6c8a81cb81cc394c","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":310,"y":479,"wires":[["ccedc5f796093cda"]]},{"id":"ccedc5f796093cda","type":"debug","z":"6c8a81cb81cc394c","name":"debug 2","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":449,"y":576,"wires":[]}]
1 Like
35

OK, I'm a ditz (I'm saying that, not you) and appreciate your patience. I said "flow tab" but was thinking "links" as used to connect flows between tabs or in/out of subflows. Or I could adopt a design pattern that uses existing nodes that talk to the outside world. Having the NR links do this however would be much cleaner .

I basically understand the method you're describing but need to study the Deploy-to-host and Send-to-host nodes. That would leave "extending" what the Deploy button does to automate the json flow/subflow gathering/processing.

<EDIT. My assertion about PicoWeb below squeezing a NR server into an RPi Pico are a hallucination (otherwise known as BS). I apologize! >
I did see some folks claim to have squeezed a complete NR server into an RPi Pico.

That would seem plug compatible with your approach. Until I try it I'll have no idea how far I can push the RP2040. Clearly, I'm trading off bloat for "elegance" and hoped for simplified development and maintenance. I think the RP2040 may have just enough juice to do this, but if not, NR for MCUs may do the trick, or I can try bigger iron in the same form factor that's already coming out.

I'm definitely going to give your approach a try. Fingers crossed:-)