Improved hardware support

[ozpos]

Not dumb by any means.

@JGKK

If i grasp your idea for an approach correctly you would want code to run on something like an arduino or even a pi that offers a standardized way to act as a proxy/bridge to forward/bridge communication between sensor hardware and nodered directly via the arduino or esp? Is this right so far?

yes and yes. Although I have not done this but others say they have.

@JGKK

So from nodered at that point why not just use the inbuild mqtt nodes?

No reason.

@JGKK

And isn’t this the point where we are back to where there are already projects like tasmota, esp easy or the myriad of great libraries for arduino for all kinds of sensors you could think of out there which allow you to do this and if they dont do it put development effort there?

Yes leave the libraries as they are. The only additional effort on the part of the maker is to compose the serial response back to NR via MQ in the format described by the info page for that particular chips UI. A tiny effort when the payoff is drag n drop connection and integration with higher level services.

@JGKK

Maybe i am beeing daft here but me like others here are failing to see the exact structure you have in mind? Can you take me through it step by step?

Not at all daft. I have described it above somewhere but here is another go. In NR you have an MCP23017 front end (derived from existing front ends so not much effort for those in that game). In the info page for the device you simply state the format of the outgoing message. Say 'channel_n=y', the info page also defines what the response should look like. e.g. 'channel_1=x, channel_2=x...' where x is 1 or 0, or all channels are returned as a binary string '0101100110100110'. It is up to the developer of the NR front end as they will have written the code to unpack the returned message for use within the flow.

At the sharp end there is an incoming message (format also defined by NR in the info page). The maker talks to the hardware (they do this all the time) ad format a response in the way described above and bingo. End to end happiness. NR have static definitions for chips that never break with node and makers familiar tools and libraries remain in tact so they are cozy too.

@JGKK

what your are trying to achieve sounds like rebuilding a whole eco system that already exists

Not at all. Just a NR front end to a chip so anyone can wire it and integrate with other services.

Is that any clearer ?

[JGKK]

But thats just not how the libraries on an arduino or similar work So you would actually have to provide both. The integration in Nodered in form of a node and the abstraction layer running on the micro controller that provides mqtt and facilitates the communication between the node and the underlying libraries on the controller. Which is where we leave the scope of any nodered node and start supplying c code for people to flash on there esps for the node to work at all

Edit
Which is what the projects mentioned above by a few people like esp easy and RPieasy already do.

[ozpos]

@afelix

Correct me if I’m wrong, but wasn’t the whole idea to approach this on low level because of the time overhead and Node-red being less suitable for real-time I/O operations?

It has been accepted that js/node does not work well with hardware drivers and is definitely a brittle coupling of the two. The proposed front ends already exist (just glued to c/python drivers at the moment). Remove that bus binding within NR and leave it where it should be in the target device.

Critical timing like closed control loops etc can remain where they should be, in the target endpoint.

This low-level overhead of MQ command response to and from the endpoint is orders of magnitude less than the critical timing an I2C/SPI bus imposes.

[ozpos]

@knolleary

t doesn't change the fact that someone has to write code to talk to the hardware and expose it via an interface for others to use.

I do not think that is necessary. Remotely connected endpoints will not need code to talk to hardware (code from NR I mean). For a streamlined out of the box experience then stub out the remote hardware with a few lines of emulation code (read msg in decode write msg out) even I could do that no UI work there.

[knolleary]

@ozpos

I do not think that is necessary.

At this point I respectively withdraw from this conversation.

I don't understand how you propose this would work if no-one is going to write code to talk to the actual hardware.

I have tried endlessly to make sense of your proposal. I'm sorry that I wasn't able to get there.

[afelix]

For clarity, you’ve mentioned UI and frontend a couple times. Do you mean the user interface that users editing flows see when configuring the node, or does it reference something else?

[ozpos]

@JGKK

The integration in Nodered in form of a node

Yes

@JGKK

and the abstraction layer running on the micro controller

No I do not think so. Anyone flashing an arduino can read a command string from NR call the chip via I2c or whatever and format a string to send back to NR.

Out of the box, a simple stub could replace the hardware end at the NR end. I am definitely not suggesting that NR community get into supplying code for end devices.

Just as horrific as supplying code for makers endpoints is their view of writing a web front end for their sensor chips.

[ozpos]

Yes the user interface that users editing flows see when configuring the node.

[afelix]

Again, correct me if I'm wrong, but to summarise, you're looking for a system that would provide such a common user interface (in Node-RED terms, a node, and this is looking at the editor side of it), and have a common protocol (like the aforementioned MQTT) be set up that this node is interacting with, and on hardware level that there's a program running that connects to the sensors/chips and interacts with them, that can be controlled through that common protocol?

If so, what's stopping the current generation of makers/softies from already setting this up using proven underlying libraries, adding a couple lines to it for an MQTT connection, and handle it in the flow through the MQTT-in/out nodes? That exact setup is how I'm handling all my low level hardware. I use proven underlying libraries, that I know work fine, add a few lines to set up an MQTT connection and command listeners to it, then in the flow hook it up through the standard MQTT nodes. This also allows me to use my own structure of MQTT topics that have so far proven to work well, and by using the low level hardware libraries I don't run into any issues with updating versions either. It's the most stable solution I've yet found and integrated into any of my systems to this day. The most recent addition of this was through an e-ink screen, where I used the manufacturer's libraries, and put it together with MQTT and a command structure to change the text on display from my flow.

[krambriw]

I might be blind but let me describe a small esp32 project I just finalized

ds18b20 -> esp32 with mqtt <-> broker <-> NR w MQTT in node <-> GUI

In this simple case I just have a temp sensor. The code in the esp32 reads the sensor and publishes the temperature at regular interval to topics defined by the code in the esp32, NR takes care and passes this to my GUI

What I have understood is if you have more complex/competent sensors or other hardware devices connected to the esp32, there has to be

• more code in the esp32 to handle this, both for writing and reading data to and from the hardware
• a node in NR that will handle the specific "pattern" exposed from the esp32 so that configuration and presentation can and will work

This is actually the way the Shelly devices work. You configure them via https and you read & write data via the same if you use their cloud service, otherwise like I do it, with mqtt locally or via a cloud service

As long as the "pattern" is unchanged, you would not have to change anything in the node but if you like to make any changes to that, the code in both the esp32 and the node most likely has to be adopted. Similar if you introduce a new hardware, it will require other code in the esp32 and a new node in NR (eventual re-use might be possible at both ends but they will be new per definition)

With "intelligent" programming, I'm just speculating on this, it might even be possible to create a node that is self-configuring it's structure including setting up the html for configuration, automatically populating with data received typical for the specific attached device/hardware

Further more, if you select mqtt for communication, you could consider if the node should include the mqtt client code as well or if the end user should have to use the standard mqtt nodes in between. Not to forget, the code in the esp32 (fw) will certainly need to be updated from time to time. Would the node have the capability to upload new fw to the esp32 OTA? Eventually also for the attached hardware if it has fw to be flashed?

Following this path might be interesting, but before starting anything else, I see the need for an in-depth analyze and design documents

[krambriw]

@afelix
Funny, many of our thoughts...overlapping

[afelix]

It's a proven strategy after all, I'm waiting for my ESP32 LoRaWAN gateway to come in, so I can also roll it out to my LoRa-operated ESP32 devices, and have the gateway correspond with Node-RED over MQTT. The primary one of those the GPS tracker hanging on my wheelchair, to track my own activity. My plans for setting it up and using it haven't changed, and it follows the layout posted by both of us above

[ozpos]

@knolleary

We have pointed out we have a model where we use the python libraries as a stable API between the Node world and the underlying hardware. That model should never get broken by node version changes.

But it already has. Otherwise folk would not be clutching to node 8.

I am suggesting that a new class of remotely connected devices could be provided alongside your existing model. Very little effort required. Just take out the bus bindings from existing chip frontends.

@knolleary

Your proposal to insert MQ between the node-red node and the hardware driver is just a different approach to achieve the same decoupling - albeit with more moving parts needed (the MQTT Broker).

Yes it is a different approach but by replacing hardware bindings with a couple of MQ connections is not increasing complexity and maintenance burden, it is reducing it to near zero.

@knolleary

It doesn't change the fact that someone has to write code to talk to the hardware and expose it via an interface for others to use. And if you want nodes that 'just work' for new users, that code has to be maintained.

That someone will be the maker. Most makers when buying an arduino expect to write code and flash devices. The libraries they use to talk to chips are maintained by the hardware suppliers. If all they have to do is decode an incoming string and send back a reply string and in return they have a visual presence of their device and can connect it to other services then they will do it. If they have to consider writing their own NR UI then no chance.

The NR info page for the device defines and exposes the interface. Output msg format and input msg format.

[knolleary]

@ozpos

That someone will be the maker. Most makers when buying an arduino expect to write code and flash devices.

So you assume the "maker", just getting started with hardware would know how to do that.

Versus something that just works out of the box without having to write any code.

You must acknowledge you are just making a different set of trade-offs and you own set of assumptions about the audience for these nodes.

@ozpos

But it already has. Otherwise folk would not be clutching to node 8.

Some nodes have not followed the sort of abstraction we have used in other nodes. That is something that needs fixing for those nodes.

And, yet again, I ask: can you tell us what nodes are stuck on Node 8? If they are nodes maintained by the core project, we can address that. If they are nodes maintained (or maybe abandoned) by others, then that is something we can look at - there may be interest in the wider community to help update them.

[dceejay]

My Friday afternoon contribution

so from the isolation point of view it can be argued that we already have that with several of the models described above - most notably the sensor + ESP (or arduino) -> MQTT -> Node-RED . There are many libraries for aduino/esp that make that side "easy" - and several plug and play projects like Tasmota and ESPeasy that make it "really easy". The upside is that none of that involves Node-RED so no maintenance - the downside is that it is another processor/chip - plus the networking overheads/delays etc.

If we want to stick on the Pi (which as an overall project we don't but for now lets go with it) then one of the best performing interfaces is the PIGPio one - and that does have low level bindings for pins, pwm, spi, i2c, waveforms etc... so all good things you want (at one level). The problem is that it either needs to bind locally and run as root - or via the gpiod interface. (which on my pi0 seems to use about 5% cpu for some reason - but again no matter)... So question then is how do we expose that gpiod interface to node. Someone has to write (and maintain) some code...

Looking at what others have done there are two main projects that have done some of this work - one is js-pigpio https://github.com/miketrebilcock/js-pigpio/blob/master/js-pigpio/index.js and the other is @stheine/pigpiod - https://github.com/stheine/pigpiod - the former is what we use under the covers of our pi-gpiod node.

Looking at both - the stheine one does expose a lot more of the underlying pigpiod interface but in a way much more suited for use in a function node than through a ui. They also have some examples of slightly higher level functions that wrap a particular chip - in this case the MCP3204 - https://github.com/stheine/pigpiod/blob/master/example/mcp3204.js - so that can be made more accessible.... but again we note that a) that is about the only higher level abstraction exposed - which means either they got bored or it's not that easy - (certainly not for beginners) - or maybe just the raw is sufficient.

Extending either of these could make for a locally isolated solution (though gpiod does also potentially open up the network as well) - which could then allow more performant local integration of "chips".

However we do note that both have been inactive for several years... either they are perfect as-is (no they aren't)- or they will need development and maintenance - and once again we ask - who is volunteering to take this on ?

[dceejay]

@knolleary

And, yet again, I ask: can you tell us what nodes are stuck on Node 8?

Hear, Hear ! - I vote for this... please list them before proceeding any further in this debate.

[ozpos]

Sorry for splitting your paragraph.

@afelix

Again, correct me if I'm wrong, but to summarise, you're looking for a system that would provide such a common user interface (in Node-RED terms, a node, and this is looking at the editor side of it),

Not a common user interface for lots of different chips but specific GUI interface per chip (these have already been coded, they are just bound to hardware libraries at the moment).

@afelix

and have a common protocol (like the aforementioned MQTT) be set up that this node is interacting with,

Not suggesting that the GUI device has to use MQ just any way of getting a string to the target and receiving one back. The end user of NR can wire MQ if they wish.

@afelix

and on hardware level that there's a program running that connects to the sensors/chips and interacts with them, that can be controlled through that common protocol?

At the hardware level and on the physical endpoint there is the sensor/peripheral chip with its local bus access libraries. The program running there reads commands from serial in (the command may have come from a usb physical connection or MQ over wifi it does not matter as long as it can read the output from the front end talk to the chip and send back a serial response. The interconnecting protocol should be transparent.

@afelix

If so, what's stopping the current generation of makers/softies from already setting this up using proven underlying libraries, adding a couple lines to it for an MQTT connection, and handle it in the flow through the MQTT-in/out nodes?

Nothing apart from the complexity of the task in creating a GUI for the likes of a 23017 or a multichannel A/D.

Then you have the answer right there.

I am very surprised that someone in this community hasn't already done this in the time spent talking about it.

Unless it is the daunting task of programming and debugging real-time software on an arduino (softies perspective) or the daunting task of building and debugging a GUI in NR (a makers perspective).

[molesworth]

I've been following this thread, carefully reading the various discussions, and I'm still not sure what exactly you're asking about or looking for. Initially it seemed to be a request for nodes to interface to specific devices, but now seems to be a more general idea about hardware connectivity, user interfaces and communications methods.

@ozpos

Not a common user interface for lots of different chips but specific GUI interface per chip...

I'm not even sure what you mean by "GUI" in this context. Do you mean a dashboard widget, or a configurable node to be used within a flow to access the "chip"? (I use the word loosely, since lots of devices are more complex than single chips.)

@ozpos

Nothing apart from the complexity of the task in creating a GUI for the likes of a 23017 or a multichannel A/D.

Then you have the answer right there.

I am very surprised that someone in this community hasn't already done this in the time spent talking about it.

But there's nothing stopping anyone developing these nodes, if they can see a need for them. One of the great things about Node-RED, which I'm impressed with as a new user, is that it is easily extended, and that new nodes can be added and made available to others who might have a need for them.

For my own approach I wouldn't be interested in these, neither the ones you specifically mention or hardware interface nodes in general, since my main server is a micro-PC, and all my "devices" are built on ESP-xxxx or R-Pi hardware, and communicating via MQTT.

However, I can see that for a different approach it would be nice to have nodes to directly access hardware, and there are already quite a number of these available. What I'm struggling with is what you are looking for that is different from these types of node. Have a look at some of them, and maybe try to explain more clearly what you need that is new or different.

[ozpos]

@knolleary

So you assume the "maker", just getting started with hardware would know how to do that.

I would'nt ask my a grandmother or grandfather to do it even if I had one. But it depends. The arduino scene is pretty big and they help each other at that end and expect to roll their sleeves up a little. They have at their fingertips free IDE and loads of working source code to interface with chips. All they have to do is modify the source a little to read from serial in and write to serial out. If they are not web developers then creating a GUI front end for the chip is almost incomprehensible. No one IDE, or integrated tool set. No concept of stepping in and out of browsers just far far more fragmented and difficult than at the sharp end.

@knolleary

Versus something that just works out of the box without having to write any code.

This approach can also work out of the box as mention above with a few lines of stub code. It just has zero coupling to hardware drivers and the front end GUI design already exists.

@knolleary

You must acknowledge you are just making a different set of trade-offs and you own set of assumptions about the audience for these nodes.

I most definitely do not. Where is the trade off in removing a hardware library dependency that causes a continuous maintenance problem.

The audience is attracted by ease of use and if they can connect their end points to NR.

What I see is the introduction of remote hardware nodes along side the hard to maintain local hardware nodes.

Give it a go and see. Provide an arduino person or Pi Zero person with a GUI front end to 23017 that spits out a msg and updates the NR end with the msg reply. They only need to write a few lines of code at their end (familiar territory) and it is done.

@knolleary

And, yet again, I ask: can you tell us what nodes are stuck on Node 8?

I have listed the nodes earlier MCP23017 for starters all the ncd-red-xxxx devices, Si7021, max31855.

I have also said I am not looking for anything for myself I have already found ways around the supplied pigpiod implementation that does not expose I2c, SPI or PWM or scripting. Albeit in a very unsatisfactory way.

But hey give me an ncd-red-mcp23017 GUI that just spits out a string and updates itself with a reply, has no bindings to any hardware libraries and it will prove the concept or not. That has to be much easier than fixing the pigpiod interface. Besides which I have been convinced that I2C is probably too low level for a chip wth lots of I/O pins.

[knolleary]

@ozpos

Where is the trade off in removing a hardware library dependency that causes a continuous maintenance problem.

The trade off I was referring to was between providing a Node-RED node that works out of the box, and a Node-RED node that does nothing until the end user has also written a bunch of additional code.

You are assuming the audience of this node is capable of the latter. I cannot see how you can argue otherwise.