Migrating from Python3 to Node-RED

General
1

I have a test system that is implemented largely in Python3 under Ubuntu Linux. It uses the Django web framework with Django data models for an MySQL database. I'm contemplating migrating this to Node-RED but have some questions about migrating and also executing bits of Python code during the various phases of the migration.

Does anyone have experience with this?

2

Yes, I have made that journey myself during a couple of years

I still have, maintain and use several of my older & newer Python scripts for special purposes & services and run them in various distributed computers in my network. Basically, I use MQTT to do full two-way communication with all of them. So my suggestion is; start very simple and create a python script with MQTT support included, learn how to start and run it from Node-RED using the exec node, try out how you can send/receive messages to/from it in Node-RED

3

Thanks for your reply. The test system already uses MQTT as its transport protocol along with JSON payloads for both commands and responses. I plan to keep the diagnostic sub-system running on the UUTs in Python and migrate just the test sequencer and data collection portions to Node-RED.

Right now, there is a test sequencer running as Linux process for each UUT and I need to decide if those should just be separate flows or if they should each be a separate NR instance. They run identical code and are passed configuration data from a Test Control Manager. This may be a use case for FlowFuse.

Both the test sequencers and test control manager access a MySQL database to pull configuration data and store and analyze test results.

I'm contemplating whether to start the migration with the test sequencer or the test control manager.

4

Hi,
Well I assume you then need to think what advantages it would be to migrate Python code to Javascript. If your modules are working and performing the expected way and delivering the results your need, I would really think twice; I personally use Node-RED only to manage my Python scripts, monitoring that they are working as expected, restarting them if that would be needed and capturing/collecting/using/presenting the results they are delivering

For sure it is likely possible you can migrate to a flow including some function nodes for the special needs but as I wrote, is it worth the effort? Myself I have some Python scripts doing object recognition using YOLO and that is not possible to do with the same performance in Javascript

5

Thank you again. That is exactly the kind of input I needed. I'm starting on a Pro/Con list to see if it is even worth it. My thoughts are to make it easier for a junior engineer to modify the testing process without having to dig too deep into the Python code. The "visual' flow of NR lends itself well to that goal. On the other hand, this also enables an uninformed person to modify the test process.

Time to do some thinking.

6

Just one more thing; if you have a lot of those distributed UUT's that needs regular updates/changes then it could be heavy if you do not have an "automatic" updating process for all the Python stuff. I assume, without having personal experience, that this is one of the strenghts of FlowFuse, to handle & manage many distributed end devices running Node-RED

7

Hi Walter,

Thank you for pointing this out. The UUTs are imaged using Intel PXE-boot along the tftp-boot and a firmware installation script. The Python-based diagnostic sub-system is then installed by using a Linux RPM which is SCP'd to the UUT.

Since no part of the UUT firmware is using Node-RED, FlowFuse won't help with that part.

Thanks again for your comments. Much appreciated.

8

Honestly the visual thing might help more than hurt though — if someone breaks the flow it's way more obvious than hunting through Python code. But definitely need to test changes before they hit the real UUTs.

9

A picture is worth a 1000 words.

Basically textual programming is dead, anyone who still does that is wasting their time. It's now the domain of AIs and any human doing it should realise that they are similar to a carpenter creating furniture while IKEA is grabbing the market. (With the only difference being that a human programming is probably turning out the same stuff that the AI is generating only 100x slower, while the carpenter tends to make far nicer things than IKEA does, also 100x slower.)

Visual two dimensional programming is the next human domain for programming. We should be investigating how to get it right so that out children will be programming in 3D (after all they are already coding in 2D using their phones - no one programmers the alarm clock on their phone using a terminal, it's all 2D buttons and scrollers).

How to get 2D programming right? Well let's start with transporting the hard-won ideas of textual programming to 2D - i.e. KISS, DRY, Version control, Testing (unit v. functional), TDD, and the rest. We probably also need to describe visual paradigms similar to programming paradigms, i.e. OOP, Functional, Declarative, List Oriented, Event driven, Message passing and the rest. To distinguish what the visual code actually means.

But most of this is beyond the imagination of the average non-AI until some large silicon valley tech company comes along and sells us this - for much money preferably. For only something that costs money can be good.

10

Unless it is a Jackson Pollock! :rofl:

Visual programming, like any other, is great - until it isn't. The difficulty with it (again, like other styles) is that the transition point is highly personal. One person's Pollock is perfectly clear to someone else.

Visual programming is great for experimentation and discovery. It is also highly useful for prototyping, where you get from initial thought process to something useable very quickly.

But it is much less useful - in my view anyway - for complex production "code". Where the level of complexity and the potential exposure of sensitive processes (sensitive to change, not just security/privacy) increases risk.

So, as usual, a balance is the wise choice.

One of the things that makes Node-RED so useful (again, in my view) is that you have the ability to collapse complex visual flows down to function nodes and also the ability to easily offload complex processes to external services (whether API's, DB engines, etc) thanks to its excellent interop nodes (http, tcp, udp, MQTT, etc).

Well, that is a very strong - and almost certainly indefensible - statement of course. Even more so in the open source world where the cost of the leading AI tools already makes them at least highly restricted if not totally unafordable.

And indeed, many organisations are already finding that their wholesale shift from people to AI for programming is turning out to be both more expensive and less viable.

I'm no AI luddite, I'm fairly sure that people will work out ways to make AI affordable - both in monetary cost and environmental - we are nowhere near that point and neither is AI coding anywhere near reliable enough, even with top-tier AI's, to fully replace human coders.

Sorry, but that is a rather obnoxious statement! Maybe beyond your imagination? The thing is, AI's don't have imagination. They synthesise from human inginuity.

The issue is time and space (and yes, money) to allow people to conjure up amazing things. Then passing those ideas to others to manufacture. If we don't have sufficient people with sufficient time, space and money, we won't get the imaginative, innovative ideas you are (rightly) calling for. AI now has a part to play, for sure but it cannot be the brain of the operation.

It isn't that it has to cost, it is that it costs to create it. Someone has to keep the innovators alive and interested and that cannot happen unless those people are at least well fed, housed and trained/educated. There are many approaches to doing that. Unfortunately, the tech powerhouses of the richest country (USA) are driven by unconstrained capitalism which results in the "Tech Bro's" with all the wealth and acting as the new slave owners. Other countries have a bit more balance but have mostly followed or are following similar patterns.

OK, I need to stop since I've gone well off-topic!

Anyway, what I was trying to get across is that Node-RED's visual programming harness coupled with its node.js and ExpressJS powered service, while fantastic for many things, is not the be-all and end-all of things. Find the balance that is right for each situation and don't get so blinkered that you fall into the trap of owning a hammer and believing everything is now a nail.

Coming back to your original question. Personally, I would think hard about what advantages I might get from using Node-RED. Where are the pain-points in your current approach? Would it be sensible to introduce a visual controller into your processes somewhere? Are there areas where you might want end-users or customers to have more freedom and control? Is Python coding a constraint on your business, would node.js/JavaScript be better?

11

and for other than OCD reasons, there is no column alignment required. In general, I agree with you but there's a lot that could go wrong in a function node that could still be hard to detect. For sure, changes go through a rigorous regression test cycle including fault injection testing.

12

Thank you all for the spirited discussion. I love visual programming since I still do flow charting for a lot of the processes I architect. Remember flow charting? I also hand code because, well, I can, and I find it rewarding.

I think that it is important to remember that we all don't work the same way and we all have opinions on how best to do things. With that, we still need to be respectful of the opinion of others, even if we don't agree with it.

For now, the UUT diagnostic code will remain in Python, since it seldom gets updated. The test case handler will also remain in Python while the UI and results analysis will be migrated to Node-RED. Node-RED will use the Django REST API to get and send data to the current system, which will continue to run in parallel as a backup/safety net.