Erlang-RED - Erlang backed Node-RED

[gregorius]

Hi There,

Inspired by @mdkrieg and his py-RED project, I have created Erlang-RED - what can be better than combining two extremely niche programming paradigms - visual flow based programming meets tail recursion and function matching

What is shown in the gif is the left terminal side starting Erlang and then Node-RED flow designer being loaded from Erlang. The inject button triggers flow execution on the Erlang side and the debug output comes from an Erlang debug node.

Wikipedia link for those not familiar with Erlang.

Why do this?

Because Erlang is truly concurrent and is ideal for executing flows in parallel. Each node in a flow becomes a process within Erlang. Erlang is designed to be flooded with processes, they are extremely lightweight. Message passing in Erlang is built in. Message cloning is built in. It is even possible to have multiple instances of Erlang on different hosts communicating transparently. So having flows spread across multiple servers is built in.

But one thing you don't want to be doing is coding in Erlang Unless of course Prolog and Lisp means something for you. What I've put together is very basic Erlang code (ddg'ed not chat-gpt'ed) but it does support (in minimal form) inject, switch, change and debug nodes. Basically what I've done is built a frame for spinning up flows, editing flows and deploying flows and some example nodes.

I find Erlang extremely elegant (sorry Javascript does not come close). Probably this line sums it up for me:

websocket_pid ! { debug, Data }

It sends a data packet to a process that holds the websocket connection to the browser. The websocket pid can be located anywhere, the sender does not care. Erlang ensures that message is sent, received and passed onto the browser. 'debug' is an atom that distinguishes the message from other messages - i.e. it's the message type.

I hope this inspires others to do something similar.

[gregorius]

For this project, I need unit tests to ensure that node functionality is correct and continues to be correct. So I started to create flows within Node-RED that represent mini unit tests. I then export those and have eunit (which is Erlangs testing framework) execute those flows. Each flow is designed to test a specific functionality of a particular node.

This became tedious because I had to continually switch between terminal (make eunit_tests) and NR (to create flows). Plus it was hard to connect errors to nodes or wires since errors where terminal based - it needed to become visual.

So I created some nodes for testing these flows within the Erlang backend but having the output streamed to Node-RED:

What the gif shows is my list of unit tests, at the press of a button, all tests are executed and a notification per test is shown for the result. In addition, the tree list shows which tests failed/succeed. I can execute tests individually or all of them.

But the best bit is that all errors are pushed to the debug panel and from there I get directly to the error node.

The idea is to create flows that represent needed functionality and then gradually implement the functionality in Erlang.

The mole hill is turning into a hill and that's mutating to a mountain - it's a lot of work! But the nice thing is that it made me think about implementing unit tests visually in NR.

[Buckskin]

Yowzah!!

[gregorius]

Yet another update, this time a description of my development process: flow driven development!

The core idea is to create flow tests within the Node-RED flow editor and then have those flows testing the Erlang implementation. What I describe how I create these tests and how I can fix them from within Node-RED.

It kind of describes a possible implementation of doing unit testing of flows, this might have broader usage beyond Erlang-RED.

Btw the animated gif above is outdated, the tests now run in parallel and are super fast - I made the mistake of triggering them sequential in Erlang by not spawning an extra process for each test.

[gregorius]

So I've managed to implement the file read, json and a single JSONata stanza to have this flow work:

[{"id":"c4690c0a085d6ef5","type":"tab","label":"[json] parse example json content","disabled":false,"info":"parse a json string into an object.\n\nread a file containing JSON, pass that off to a JSON node to convert that to list of maps (in this case).\n\nthen count the number of elements in the list and compare that number with 8!\n\na more complicated flow that is supported - yeah! :+1:\n\n## Sometime bug here\n\nDebug node can generate decode errors when deal with the content here and it has to do with format.\n\n--> [decodeObject](https://github.com/node-red/node-red/blob/1eb9aa0eedcbcb6fab78ec950523ea71aeaa2760/packages/node_modules/%40node-red/editor-client/src/js/ui/utils.js#L1277-L1300) causes errors if the format on the debug message isn't correct.","env":[{"name":"ERED_KEEPRUNNING","value":"false","type":"bool"}]},{"id":"453848d471cdd83d","type":"inject","z":"c4690c0a085d6ef5","name":"","props":[{"p":"filename","v":"priv/testdata/test.json","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":273,"y":231,"wires":[["962cb9fb663ba64d","57f58d3e81d73921"]]},{"id":"962cb9fb663ba64d","type":"file in","z":"c4690c0a085d6ef5","name":"","filename":"filename","filenameType":"msg","format":"utf8","chunk":false,"sendError":false,"encoding":"none","allProps":false,"x":427,"y":336,"wires":[["7778fb3372eda641","9e8394f480ba405a","f3d21b6c8e431f9f"]]},{"id":"7778fb3372eda641","type":"json","z":"c4690c0a085d6ef5","name":"","property":"payload","action":"obj","pretty":false,"x":591,"y":460,"wires":[["84fa8ce2a3953064","4ae89e9e828d9ebc","c1724fc6ab23c0c4"]]},{"id":"84fa8ce2a3953064","type":"debug","z":"c4690c0a085d6ef5","name":"debug 5","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":834,"y":459,"wires":[]},{"id":"9e8394f480ba405a","type":"debug","z":"c4690c0a085d6ef5","name":"debug 6","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":714,"y":335,"wires":[]},{"id":"f3d21b6c8e431f9f","type":"ut-assert-success","z":"c4690c0a085d6ef5","name":"","x":718,"y":289,"wires":[]},{"id":"13bdc3b87be1f394","type":"ut-assert-debug","z":"c4690c0a085d6ef5","name":"","nodeid":"84fa8ce2a3953064","msgtype":"normal","inverse":false,"x":920,"y":509.5,"wires":[]},{"id":"4ae89e9e828d9ebc","type":"ut-assert-values","z":"c4690c0a085d6ef5","name":"","rules":[{"t":"set","p":"payload","pt":"msg"}],"x":889,"y":609.5,"wires":[[]]},{"id":"c1724fc6ab23c0c4","type":"change","z":"c4690c0a085d6ef5","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"$count($$.payload)","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":973,"y":371,"wires":[["b1a7846c21cceac0","56d121e9029634cd"]]},{"id":"b1a7846c21cceac0","type":"ut-assert-values","z":"c4690c0a085d6ef5","name":"","rules":[{"t":"eql","p":"payload","pt":"msg","to":"8","tot":"num"}],"x":1325,"y":371,"wires":[[]]},{"id":"56d121e9029634cd","type":"debug","z":"c4690c0a085d6ef5","name":"debug 11","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1148,"y":195,"wires":[]},{"id":"57f58d3e81d73921","type":"debug","z":"c4690c0a085d6ef5","name":"debug 8","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":608,"y":231,"wires":[]}]

It reads the json file, converts that json to a list of maps (Erlang speak for array of objects) and then counts how many objects are in the array - using the only working JSONata stanza $count($$.payload). That number is then tested to be 8 and the test passes.

It also shows the usage of the testing framework with a assert node checking whether a debug message was generated, another checking the values of a property on the message and a third assert ensuring that it was reached, i.e, the msg reached it.

Interesting project since I've now come to realise (and respect) how complicated some of the nodes really are (switch & change are monsters in disguise) - junction nodes are a wonder of simplicity thankfully

[gregorius]

Another update, thanks to @mwmiller there is an online, live version of Erlang-RED available.

It's read-only and basically only good for going to the testing tab, pressing refresh flows and then pressing test all.

Flows can be loaded by double clicking on them in the testing panel. Deploying them does say that they got saved but that's a cunning lie.

This is very much beta and buggy, if it doesn't work then its intention was to work but it didn't quite fulfil those good intentions.

And yes: this is really Erlang, there is no NodeJS involved.

[gregorius]

Just as an update, I'm still working on ErlangRED, some highlights:

• JSONata has been ported (partly) so that JSONata stanza can be utilised. It made me realise how important JSONata is for having a good visual low code environment. In porting JSONata, I did create a yecc parser for the language (yecc being the same as yacc/bison but for Erlang).
• MQTT in/out nodes work - so now message can be easily exchanged between a Node-RED instance and ErlangRED. That also gave a good reason to ensure that config nodes are supported, which they are now.
• There are just over 100 visual flow tests to ensure that ErlangRED nodes are compatiable with the current Node-RED functionality. These "test" flows can be utilised by other projects (e.g. PythonRED) to also ensure compatibility or by NodeRED itself for regression testing.

Next step is getting a working exec node to start a mosquito MQTT broker so that the MQTT nodes can be tested with a broker (they work with a broker but a test has to start the broker first, hence the exec node).

[gregorius]

@gregorius

Next step is getting a working exec node to start a mosquito MQTT broker so that the MQTT nodes can be tested with a broker (they work with a broker but a test has to start the broker first, hence the exec node).

There is now a working Exec node and an working Function node that allows coding Erlang in flows, no Javascript inside the box.

CSV and Markdown nodes are now also supported.

As always, a test run is available over there.