Erlang-RED - Erlang backed Node-RED

[gregorius]

Hi There,

Inspired by @mdkrieg and his py-RED project, I have created Erlang-RED - what can be better than combining two extremely niche programming paradigms - visual flow based programming meets tail recursion and function matching

What is shown in the gif is the left terminal side starting Erlang and then Node-RED flow designer being loaded from Erlang. The inject button triggers flow execution on the Erlang side and the debug output comes from an Erlang debug node.

Wikipedia link for those not familiar with Erlang.

Why do this?

Because Erlang is truly concurrent and is ideal for executing flows in parallel. Each node in a flow becomes a process within Erlang. Erlang is designed to be flooded with processes, they are extremely lightweight. Message passing in Erlang is built in. Message cloning is built in. It is even possible to have multiple instances of Erlang on different hosts communicating transparently. So having flows spread across multiple servers is built in.

But one thing you don't want to be doing is coding in Erlang Unless of course Prolog and Lisp means something for you. What I've put together is very basic Erlang code (ddg'ed not chat-gpt'ed) but it does support (in minimal form) inject, switch, change and debug nodes. Basically what I've done is built a frame for spinning up flows, editing flows and deploying flows and some example nodes.

I find Erlang extremely elegant (sorry Javascript does not come close). Probably this line sums it up for me:

websocket_pid ! { debug, Data }

It sends a data packet to a process that holds the websocket connection to the browser. The websocket pid can be located anywhere, the sender does not care. Erlang ensures that message is sent, received and passed onto the browser. 'debug' is an atom that distinguishes the message from other messages - i.e. it's the message type.

I hope this inspires others to do something similar.

[gregorius]

For this project, I need unit tests to ensure that node functionality is correct and continues to be correct. So I started to create flows within Node-RED that represent mini unit tests. I then export those and have eunit (which is Erlangs testing framework) execute those flows. Each flow is designed to test a specific functionality of a particular node.

This became tedious because I had to continually switch between terminal (make eunit_tests) and NR (to create flows). Plus it was hard to connect errors to nodes or wires since errors where terminal based - it needed to become visual.

So I created some nodes for testing these flows within the Erlang backend but having the output streamed to Node-RED:

What the gif shows is my list of unit tests, at the press of a button, all tests are executed and a notification per test is shown for the result. In addition, the tree list shows which tests failed/succeed. I can execute tests individually or all of them.

But the best bit is that all errors are pushed to the debug panel and from there I get directly to the error node.

The idea is to create flows that represent needed functionality and then gradually implement the functionality in Erlang.

The mole hill is turning into a hill and that's mutating to a mountain - it's a lot of work! But the nice thing is that it made me think about implementing unit tests visually in NR.

[Buckskin]

Yowzah!!

[gregorius]

Yet another update, this time a description of my development process: flow driven development!

The core idea is to create flow tests within the Node-RED flow editor and then have those flows testing the Erlang implementation. What I describe how I create these tests and how I can fix them from within Node-RED.

It kind of describes a possible implementation of doing unit testing of flows, this might have broader usage beyond Erlang-RED.

Btw the animated gif above is outdated, the tests now run in parallel and are super fast - I made the mistake of triggering them sequential in Erlang by not spawning an extra process for each test.

[gregorius]

So I've managed to implement the file read, json and a single JSONata stanza to have this flow work:

[{"id":"c4690c0a085d6ef5","type":"tab","label":"[json] parse example json content","disabled":false,"info":"parse a json string into an object.\n\nread a file containing JSON, pass that off to a JSON node to convert that to list of maps (in this case).\n\nthen count the number of elements in the list and compare that number with 8!\n\na more complicated flow that is supported - yeah! :+1:\n\n## Sometime bug here\n\nDebug node can generate decode errors when deal with the content here and it has to do with format.\n\n--> [decodeObject](https://github.com/node-red/node-red/blob/1eb9aa0eedcbcb6fab78ec950523ea71aeaa2760/packages/node_modules/%40node-red/editor-client/src/js/ui/utils.js#L1277-L1300) causes errors if the format on the debug message isn't correct.","env":[{"name":"ERED_KEEPRUNNING","value":"false","type":"bool"}]},{"id":"453848d471cdd83d","type":"inject","z":"c4690c0a085d6ef5","name":"","props":[{"p":"filename","v":"priv/testdata/test.json","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":273,"y":231,"wires":[["962cb9fb663ba64d","57f58d3e81d73921"]]},{"id":"962cb9fb663ba64d","type":"file in","z":"c4690c0a085d6ef5","name":"","filename":"filename","filenameType":"msg","format":"utf8","chunk":false,"sendError":false,"encoding":"none","allProps":false,"x":427,"y":336,"wires":[["7778fb3372eda641","9e8394f480ba405a","f3d21b6c8e431f9f"]]},{"id":"7778fb3372eda641","type":"json","z":"c4690c0a085d6ef5","name":"","property":"payload","action":"obj","pretty":false,"x":591,"y":460,"wires":[["84fa8ce2a3953064","4ae89e9e828d9ebc","c1724fc6ab23c0c4"]]},{"id":"84fa8ce2a3953064","type":"debug","z":"c4690c0a085d6ef5","name":"debug 5","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":834,"y":459,"wires":[]},{"id":"9e8394f480ba405a","type":"debug","z":"c4690c0a085d6ef5","name":"debug 6","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":714,"y":335,"wires":[]},{"id":"f3d21b6c8e431f9f","type":"ut-assert-success","z":"c4690c0a085d6ef5","name":"","x":718,"y":289,"wires":[]},{"id":"13bdc3b87be1f394","type":"ut-assert-debug","z":"c4690c0a085d6ef5","name":"","nodeid":"84fa8ce2a3953064","msgtype":"normal","inverse":false,"x":920,"y":509.5,"wires":[]},{"id":"4ae89e9e828d9ebc","type":"ut-assert-values","z":"c4690c0a085d6ef5","name":"","rules":[{"t":"set","p":"payload","pt":"msg"}],"x":889,"y":609.5,"wires":[[]]},{"id":"c1724fc6ab23c0c4","type":"change","z":"c4690c0a085d6ef5","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"$count($$.payload)","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":973,"y":371,"wires":[["b1a7846c21cceac0","56d121e9029634cd"]]},{"id":"b1a7846c21cceac0","type":"ut-assert-values","z":"c4690c0a085d6ef5","name":"","rules":[{"t":"eql","p":"payload","pt":"msg","to":"8","tot":"num"}],"x":1325,"y":371,"wires":[[]]},{"id":"56d121e9029634cd","type":"debug","z":"c4690c0a085d6ef5","name":"debug 11","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1148,"y":195,"wires":[]},{"id":"57f58d3e81d73921","type":"debug","z":"c4690c0a085d6ef5","name":"debug 8","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":608,"y":231,"wires":[]}]

It reads the json file, converts that json to a list of maps (Erlang speak for array of objects) and then counts how many objects are in the array - using the only working JSONata stanza $count($$.payload). That number is then tested to be 8 and the test passes.

It also shows the usage of the testing framework with a assert node checking whether a debug message was generated, another checking the values of a property on the message and a third assert ensuring that it was reached, i.e, the msg reached it.

Interesting project since I've now come to realise (and respect) how complicated some of the nodes really are (switch & change are monsters in disguise) - junction nodes are a wonder of simplicity thankfully