Tutorial - Read a sensor by running a Node-RED flow on an ESP32 via node-red-mcu

[BartButenaers]

Hi folks,

With a lot of help from @phoddie and @ralphwetzel, I managed at last to measure distances from a HC-SR04 sensor by running node-red-mcu on my WT32-ETH01 board. So the least I could do is write down how we did it. And hopefully then more users start experimenting with this nice technology. Because these guys could use some help from this community...

Note that a lot of other boards are supported (e.g. Wemos D1 mini, ...) but the concept is always the same. The WT32-ETH01 board is a bit more difficult, so other boards need a few less steps as described below.

WT32-ETH01 board

I need this project to measure the water level in some underground tanks in my garden. Since there is no wifi, I have installed a waterproof CAT5 wire. At the tanks I will in install a WT32-ETH01 board, which has an ESP32 microcontroller and an onboard RJ45 connector:

Since the board has no build-in POE, I use a POE splitter as described in another discussion:

While the HC-SR04 is a well known ultrasonic distance sensor, I have used a JSN-SR04T which is a waterproof alternative.

Setup Moddable SDK & node-red-mcu(-plugin)

Before you can get started, you will need to install some tools on the host were Node-RED is running. In my case this is a Raspberry.

1. First you need to install the Moddable SDK, which contains all the development tools and runtime software to run Javascript applications on microcontrollers. To accomplish this, I followed the steps in the Linux section of their getting started guide.

   

   • Installing: steps you need to do only the first time that you setup the SDK
   • Updating: steps you need to do when you want to install a new version of the SDK. Because the SDK is frequently updated with bug fixes and new features (e.g. new sensor types).

2. At the end of their Getting Started guide, there is also a section " What's Next: Building and Running Apps on Development Boards and MCUs". For each device, there is a link to an installation procedure to install the development tools for your target device. Since I wanted to program an ESP32, I have clicked the "Using the Moddable SDK with ESP32" link and executed all the steps.

3. Now we need to install the MCU sidebar for our Node-RED flow editor. Of course you could also run all the required Moddable mcconfig commands manually, but the sidebar generates these commands automatically behind the scenes. Go to the "manage palette" menu to install the node-red-mcu-plugin:

   

   This sidebar node will also install node-red-mcu automatically as a dependency. The node-red-mcu is a Moddable SDK project to run a Node-RED flow on a microcontroller.

4. Install the Sensor node via this command:

   cd ~/.node-red
   npm install /home/pi/.node-red/node_modules/@ralphwetzel/node-red-mcu-plugin/node-red-mcu/nodes/mcu
   

   Now you will get a bunch of MCU related nodes in your palette:

   

Setup the serial interface

To be able to flash and debug the MCU, you need to connect your host (where Node-RED is running) to your MCU via a serial interface. For most boards (e.g. Wemos D1 mini) this will happen via a simple USB connection. However the WT32-ETH01 board has no USB connector, we need a CH340G Serial Converter USB to TTL converter:

And then the 4 wires need to be connected to these pins of the WT32-ETH01 board:

USB-TTL converter	WT32-ETH01
GND	GND
TX	RX0 (not RXD)
RX	TX0 (not TXD)
5V	5V

On the USB-TTL converter, set the jumper to S1 and 5V (not S1 and 3.3V).

Program the ESP32

These steps need to be repeated, every time you want to flash a Node-RED flow to your MCU:

1. Draw a flow that needs to be flashed to the MCU. For example:

   

   Some remarks:

   • On the node-red-mcu readme page you will see that a series of core nodes have an MCU implementation, which means they are fully supported on MCU's.
   • As you can see there, also some existing sensor related nodes have been modified, to make sure they can run on an MCU. But you can also control these same sensors via their Sensor node, and the Sensor node offers an implementation for much more sensor types...
   • Other nodes (e.g. function nodes) will also be converted automatically for your MCU, but they 'might' contain a bit too much code for a low-resource MCU. I haven't tried that, but perhaps @colinl can give some extra info.

2. Select in your MCU sidebar which flow needs to be flashed to the MCU:

   

3. Select your target platform and target board in the sidebar. In my case:

   

4. Select which serial port on the Raspberry you want to use. To do this, click on the serial port field and press the down array to show a list of all detected serial ports. If your TTL-USB converter is correctly inserted, it should show up in the list automatically:

   

5. If you need WIFI, then enter the SSID and password in the sidebar. At the moment of writing this tutorial, I realize that I have only tested my board via wifi. So I need to test it via CAT cable as soon as I have time...

6. Optionally it might be a good idea to cleanup all temporary files from the previous build. Select build target "clean" in the sidebar, and then press the blue build button:

   

   The cleanup is completed, when the following text appears in the Console Monitor tabsheet:

   

   Note that a build will take much longer after a clean, because the build cache will also be cleared. Which means everything needs to be build from scratch, even the stuff that hasn't changed. Of course you could skip this cleanup step, unless you have the impression that something did not went well (e.g. when you have the impression that the updates in your flow did not get deployed on your MCU....).

7. Build the binary image of the selected flow for your MCU. Select build target "build" in the sidebar, and then press the blue build button:

   

   You can follow the status of the build process in the Console Monitor tabsheet. When the build is completed, the following text appears:

   

8. Now put the WT32-ETH01 board in programming mode, by connecting its IO01 pin to GND. Then power off/on the board to activate the programming mode. This is not required on most other board types, since it will then happen all automatically via the serial USB interface.

9. Now flash the binary image to the MCU, via the build target "deploy" in the sidebar, and then press the blue build button:

   

   You can follow the status of the deploy process in the Console Monitor tabsheet. When the flashing is completed, the following text appears:

   

10. Now put the WT32-ETH01 board in normal mode, by deconnecting the IO01 pin again from GND. Then power off/on the board to activate the normal mode. This is not required on most other board types, since it will happen all automatically via the serial USB interface.

11. Finally activate the Moddable debugger, called xsbug. Because that debugger communicates with the MCU via the serial interface, to exchange messages for step 11. You can start it (to receive messages from the MCU) by selecting build target "xsbug" and pressing the blue build button.

    

    As you can see in the screenshot, there are two debugger types (i.e. debugger targets):

    • xsbug: xsbug is the standard debugger with a graphical user interface that can be started on your Raspberry where Node-RED runs, to debug the Javascript code running on the MCU.
    • console: to start xsbug-log, which is a console logger for headless Raspberry systems like mine. Since there is no graphical layer installed, I needed to use the console log...

    When the debugger has started listening to messages arriving from the MCU, the following text appears:

    

    Note that there is a shortcut (to do exactly the same) by clicking on the arrow of the blue button, and then "Reconnect to xsbug":

    

12. If everything goes well, the nodes should get their correct status in your flow editor (which is send by the MCU). If you have deployed Debug nodes, the corresponding messages should be displayed in the debug sidebar.

    As mentioned in the previous step, all this data arrives from the MCU via the serial interface and xsbug (or xsbug-log) will send this data to Node-RED. Which means that you will see only node status updates and messages in the Debug sidebar when you have a serial connection and xsbug (or xsbug-log) is active!!!

In case it doesn't work, you might have a look at my cheat-sheet. That might give you some pointers of where to start searching.

Have fun with it, and please support the Moddable team by start using their stuff!!!

Bart

[phoddie]

This is a fantastic write-up! Very clear and detailed. Thank you for taking the time to share your knowledge. After reading your tutorial, I think I finally understand your project -- measuring the level of water in a tank using the ultrasonic distance sensor.

I very much appreciate your pioneering effort here. Early adopters shake-out issues, making the project more robust for everyone who follows. You definitely choose a road with a some extra challenges by using the WT32-ETH01 board without auto-programming circuitry or USB bridge and a headless Raspberry Pi for development. But, eventually it all works! Many thanks to @ralphwetzel for his patient efforts on your journey.

Other nodes (e.g. function nodes) will also be converted automatically for your MCU, but they 'might' contain a bit too much code for a low-resource MCU.

On an ESP32 like you are using, you should not easily run into resource constraints with Function nodes. @Colin has been working on an ESP8266 which has much less free memory than an ESP32 (roughly 40 KB versus 250 KB) so he has bumped into memory limits much more quickly.

Have fun with it, and please support the Moddable team by start using their stuff!!!

The Node-RED MCU Edition project continues to evolve rapidly. Feedback from the Node-RED community helps set priorities. I've recently been working on an OTA Update node to let flows manage firmware updates. That was motivated by community input. And it should be convenient for those WT32-ETH01 boards in your garden to receive updated flows.

[Colin]

Is there a reason you are doing all this from a headless pi rather than from your PC? That would give you the advantages of using the GUI xsbug.

[BartButenaers]

I had Node-RED running on my Raspberry, so I installed Ralph's sidebar there. And Moddable is also supported on Linux, so I installed that also on my Raspberry. Only afterwards during experimenting it became clear that I could not use xsbug, since I had a headless Raspberry. So I needed to start using xsbug-log, which appeared afterwards to lack some features. Which meanwhile all has been fixed.

So yes if I had known this all from the start, most probably I would have installed this on my Windows portable and used xsbug with a graphical ui. But I didn't knew all of that at the time being.

But I am looking very forward to your step-by-step tutorial about debugging Node-RED on an mcu via xsbug on a Windows PC. Can't wait to read your article

[Colin]

@BartButenaers

tutorial about debugging Node-RED on an mcu via xsbug on a Windows PC

Luckily I don't use Windows, all my PCs run on Ubuntu, so installing it is exactly the same as in the superb tutorial you have already provided. The only difference is that you can just select the 'All' build option and (for a D1 Mini anyway) it builds, downloads, opens xsbug, and runs.

[BartButenaers]

Perhaps you can share a screenshot of xsbug, and some extra info about that. So that people (including me) know what to expect when using xsbug instead of xsbug-log...

[BartButenaers]

I have added a bunch of extra information to my tutorial above, and also some screenshots (e.g. of the xterm terminal window in the second tabsheet) to make it a bit more clear what you should expect to see if everything goes well.

I have to admit that - once you have setup everything once - it goes very smoothly to deploy a new flow to an MCU. It happens in a few clicks in the sidebar

[BartButenaers]

@BartButenaers

At the moment of writing this tutorial, I realize that I have only tested my board via wifi. So I need to test it via CAT cable as soon as I have time...

My WT32-ETH01 board only worked via WIFI, but not via wired ethernet connection (i.e. CAT cable). But tomorrow @phoddie already managed to resolve my issue. So now I am ready to install my board in my garden, and use it outside the range of my WIFI via a CAT cable

As I said before, what a service...

BTW. So this is why my bug fix procedure went:

1. I created a Github issue for the Moddable repo.

   Note that it is sometimes not very clear whether instead you should register an issue in their node-red-mcu repo, because it is sometimes not clear for us where the problem is located. Anyway, these guys will pick it up anyway...

2. After they have fixed the problem in Moddable, I had to reinstall their SDK:

   

   So you can use the shorter "updating" procedure, instead of the longer "installing" procedure.

3. When you press the blue "Build" button, the Moddable SDK version will appear in the top of the Console Monitor log:

   

   At the end (after "-g") you can also fit their Github commit number, in this example c43982f. You can search it in their commit history to determine what has been changed meanwhile.

4. Once the SDK is updated, I did a build and deploy again via my MCU sidebar in my flow editor.

5. And I tested the updated flow on my WT32-ETH01 and it worked.

P.S. I am not quite sure how it goes when the bug is in node-red-mcu. How I need to update node-red-mcu, because that library is automatically installed as a dependency of the sidebar from @ralphwetzel ?

[ralphwetzel]

@BartButenaers

ow I need to update node-red-mcu, because that library is automatically installed as a dependency of the sidebar from @ralphwetzel ?

If you re-run the installation of the plugin, it will pull the latest node-red-mcu version from the repo. You need to do this via the command line though...

[Colin]

In addition to
npm install ralphwetzel/node-red-mcu-plugin
if you have separately cloned node-red-mcu in order to have a fixed folder location for the sensor node, don't forget to go there and
git pull

[phoddie]

Note that it is sometimes not very clear whether instead you should register an issue in their node-red-mcu repo, because it is sometimes not clear for us where the problem is located. Anyway, these guys will pick it up anyway...

Good point. The source of the problem is not always obvious, not even for me. The most important thing is to report the issue -- whether to the Moddable SDK, Node-RED MCU, or MCU plug-ing repositories. Just make your best guess about the right place to report it.

In the case of this issue, you got it right! The fundamental issue was in the Moddable SDK. Interestingly, the problem was only triggered by an unusual behavior of the Node-RED MCU runtime, so we had never seen it before.

One note regarding Moddable SDK updates.... it is important to remember to rebuild the tools after updating. If you only update the git repository and don't rebuild tools, you may be running out-of-date tool binaries. The instructions explain all this and it is very quick to do. Because this is a common oversight, we are working on a solution to automatically rebuild tools when needed, but it isn't quite there yet.

[bakman2]

Although I don't really grasp the full scope what this could be used for, but thinking outside of the box:

let's say i create a flow with an inject node, every 2 hours - is there a way to trigger deepsleep after the flow has run (or a custom node at the end of the flow that initiates deepsleep or something).

or something like an interrupt (for example I am using a ball switch - underrated btw - to trigger a wakeup).