Improved hardware support

[ozpos]

Sorry if this is a long one, I have to unpick each point I am definitely not having a go, and sorry it is just terminology on my part. By core I meant distributed like the serial port that you sight. The out of the box experience is so important. Like an interview the first 5 minutes is all you need. First impressions are what it is all about if you want to attract more willing hands.

There is nothing very interesting or instructive about the serial UI interface. As for pigpiod unless it has been fixed recently I does not give access to SPI, I2C, or scripting (and never has) which renders it pretty useless if you want to read a temperature or whatever. Also the api has not changed for donkeys so if is not there now then it will never be with the current state of affairs. So get away from the hardware altogether and just supply UI serialised interfaces and forget it.

I and others have to go back to 8 lts to get a ui to a chip that works because the underlying hardware driver is easily broken with new node verstions. A ui for a remote connected MQ chip is completely static (no maintenance at all). All the team have to contend with is maintaining an interface with MQ which you do already.

@knolleary

The Raspberry Pi install script we provide gives you the core Node-RED and adds a few of these non-core nodes because it makes sense to install them on the Pi.

Only if they work with the latest node lts / maintenance release that the community have time to support.

@knolleary

It is exceedingly rare for a pure JS node to need any upkeep due to newer versions of Node.

Others have sited that attempting to write hardware drivers in JS is futile and I think they are correct otherwise they would be everywhere and JS security issues would be even more problematic than they are already.

Python (chosen by Pi Foundation for educational purposes) is far too heavyweight for a Pi Zero. And, if you have to resort to multiple exec nodes calling python libraries every second in order to drive your sensors it will drive a Pi4 into the ground. Been there and wasted precious time and resources.

@knolleary

The issues tend to be where nodes require C code that has to be built for newer versions of Node.

Yes I know. But a ui that serialises a chips i/o command structure is nothing more than passing strings back and forth via MQ. That would not break with node version changes.

Hardware people are good at C and I2C and SPI and Interrtupts and debounce and so on. Leave the fun stuff for them give them a UI to a chip and bingo they will definitely let the world know they have an arduino or whatever driver for your example chip front ends. No maintenance at all.

@knolleary

An ideal model is for the Node-RED node to make use of existing well-maintained low-level libraries - as those libraries will typically have a much larger audience than just Node-RED.

The low level C library apis have not changed for years. And as you say they have a very brittle interface with node. Just get rid of them all.

@knolleary

f the node has a maintainer who is interested in maintaining it, then it won't die. If they lose interest in it, the community can step in.

I got the impression that the NR team aim to reduce the current and future maintenance burden.

@knolleary

This is why some of our existing nodes lean on existing python libraries - its simply a pragmatic way of getting things working.

Great for education rubbish at anything fast and efficient.

@knolleary

The key point is, at this point in time, its fairly irrelevant as to whether the code ends up in the Node-RED organisation on github, or someone's personal account. The question is whether there are people willing and able to do the work and take on the longer-term effort.

The makers will look after their end with pride and conviction especially if given an example of a static UI definition that they can work with. The softies will look after their end with pride and conviction also. But for the life of me it seems like they will never meet and agree the terms of engagement. Simultaneous engineering was supposed to take care of this impasse.

@knolleary

I speak from bad experiences of nodes being contributed to the Node-RED project on github and then getting abandoned - leaving Dave and I to add it to the long list of things we have to maintain.

So don't do it. Give the makers a UI or two, piece of cake for a softy.

I am only 50 50 but well over sixty so I have seen it and experienced it all before.

Best of luck, I have enjoyed most of the interaction but can smell the dislike towards an alternative point of view from an outsider.

The UI design, learning a new IDE (not JetBrains), a whole bucket load of uncoordinated tools that look like they have been made up on the hoof is all too much for my 50%.

Best regards to all.

[TotallyInformation]

@ozpos

but can smell the dislike towards an alternative point of view from an outsider.

Sorry, but for the record - for other people who might come across this thread in the future, I can't let this go unchallenged. We are not disliking an alternative point of view because someone is an "outsider". We are disliking it because our experience tells us that it is non-optimal.

A group of hardware-related modules dedicated to Pi GPIO would, as several people have said, be a great idea and would really help a certain class of users, probably educational users most of all since they may not have the resources to manage the additional complexity of multiple devices (e.g. Arduino/ESP + Pi).

But this is not core to what Node-RED now is. To make a success of such a set of modules would need community just like any other open source initiative and would doubtless gain help from some folk on the forum.

[dceejay]

Ok - lets try this another way... it seems like your preference seems to be with SPI and i2C (or at least they have come up several times on both sides of the conversation) - so let's just focus on them (for now). You also mention you want a UI or two... What would they look like ? can you sketch out something - paper and pencil and photo would be fine - so at least we can try to get to grips with what seems to be missing.

The existing i2c node lets you scan the bus and send raw commands in and read things out - so is pretty low level in that regard - but often we find people want something slightly higher level than that - as they want to target a specific device and use known terms and commands - so they wrap the low level into a higher level package. This is often where the issues start in that they then abandon it and don't update the underlying code as and when required... - but hey - that's the "other" issue... Let's start with the basics and try to understand the design you are proposing properly.

[knolleary]

@ozpos

The low level C library apis have not changed for years. And as you say they have a very brittle interface with node. Just get rid of them all

You dont want JS because it's bad for low level hardware I/O - agreed.

You don't want python because it's slow.

You don't want C because it's brittle.

At this point I don't know what you want.

This isn't about you being new to this forum. Since the start of this thread I've tried to understand at a technical level what you are looking for.

I have agreed consistently that it would be great thing for good quality hardware nodes to exist. And that the community can make that happen if there are the people to do it.

But I continue to struggle to understand at a technical level what you are suggesting.

[drmibell]

@ozpos

a whole bucket load of uncoordinated tools that look like they have been made up on the hoof

Welcome to the world of open source. The NR project is an example of how to do it right, and this community consistently embraces new people and ideas. I just don't believe that you are the first to have discovered the fatal flaw in the project and have found the only cure for it. (Yes, your posts clearly give that impression.) I think there is merit in some of what you suggest, but you are clearly failing to generate much interest in your ideas.

[iiot2k]

Hi,
at moment I work on nodes for connect control/automation world with node-red. It includes very popular hardware. But there are lot of nodes to write and test. I hope, that the nodes are ready for use in short time. On beginning, the nodes runs on raspberry pi. Later on other hardware. My targets are most cheap hardware avaible on market. The mcp23017 or gpio for example. Which digital and analog modules are needed urgent from the node-red user ?

[ozpos]

Only to reiterate what I have already said before. What triggered my entire input here was that v1.1.1 looks really good, brilliant and very stable.
While users take pleasure in these new features, I saw an opportunity for the community to level things up a little between the hardware and software ease of use.

My experience with NR began when it was d (I believe it is the only way forward, open source that is). At that time it was limited as to what chips you could connect. Softies think about connecting apps/services, makes think about connecting chips. They both have to be done. Then there was an explosion of devices but their maintenance could not keep up with node version frequency. The chips have not changed, the I2c/spi protocols have not changed. It is simply the case that if left out there then they will decline and they have done so.

At the top I asked if some HW devices could be considered, not for me, I have already had to find unsatisfactory ways around the problem by replacing python with C and far to much use of exec nodes. The outcome of which drags down the great work that has taken place.

The answer was an emphatic NO. So I probed and pushed to find what was behind that NO and it was resources. I suggested that if a good example of at least one device was supported/distributed then the door would be left open for more hands to give it a go. Answer NO.

So I spent a few hours thinking about it and realized that you could provide several front ends to chips without any maintenance overhead at all. Answer is still NO.

How many softies here (with no clue of hardware) could comprehend a multitasking real time os with priority scheduling within a 2kB footprint, 2k. ?

Similarly how many makers (little knowledge of software) could implement a front end to a chip in NR ?

How can ui front ends to chips be brittle ?

Where is the maintenance overhead ?

Leave that to makers. They love driving chips but designing and testing a UI, forget it I have tried !

Makers want to see the chip and its I/O so it can be wired up. Just like softies need to configure Ip addresses, ports and protocols.

I keep mentioning the port expander 23017, mainly because it is a useful chip and has more fiddly bits to configure than a serial port or plain old GPIO. The front end would have address selection fields, check boxes for input or output, 16-bit or 8-bit, power on state etc. All way beyond most makers capabilities and no maintenance headache foisted on the community.

@dceejay

Ok - lets try this another way... it seems like your preference seems to be with SPI and i2C

Not really, they are of little use in isolation and wiring wise within NR they are just buses. Other buses similarly would not need to be implemented anyway. Besides they are brittle and everyone wants less maintenance. I mention them because they are very popular buses that can be found on all low level MCUs. It is a chip front end that is required not an I2c or SPI front end.

@dceejay

What would they look like ? can you sketch out something - paper and pencil and photo would be fine - so at least we can try to get to grips with what seems to be missing.

The front ends already exist. They just don't work because they are bound to brittle C/python I2c/SPI drivers. Instead of calling I2c/SPI just serialise the calls to an output. On the incoming side deserialise the response and send to outputs or status indicators. The user can then wire the device to mq and away they go. You could also replace absent physical hardware with a an emulation (serial in unpack, parse serialise a response and send it out). Even I could do that.

@dceejay

...so they wrap the low level into a higher level package. This is often where the issues start in that they then abandon...

I agree completely.

As short of a description as I can:
NR comes with a remote hardware group of nodes. Say one is an 8 channel A/D. It does not matter if it is I2C or SPI or anything for that matter. It has settings for maybe a conversion time and single or differential inputs.
If the setting is single ended then it presents itself with 8 inputs otherwise 4 etc.
The settings/commands are serialised and sent to the output (connection to MQ), the returned message (from MQ) is unpacked and data/status is written to the relevant outputs.
The actual format of the returned message is defined in the info page for the device.

So the maker chooses their favourite MCU and writes code to read the settings/commands from serial in (MQ) and write them to the device via the bus drivers supplied with the MCU (I2C/SPI/...). The maker could have some fun here with interrupts or polling or whatever and read data from the device and in line with your docs build a string of results to be sent back.

Job done. The maker is always going to maintain his/her end and the NR end has a maintenance free hardware interface and every one is happy.

[ozpos]

That is ok so be it. I did not think I had it in me in the first place. So I am not disappointed that I cannot make a chip front end for NR. I do not think this is the only solution at all.

However it is maintenance free which is what everyone says they want.

[zenofmud]

@ozpos

The maker is always going to maintain his/her end

I think this is the crux of the disagreement. While it would be great if makers always maintained their work, the sad truth is they don’t

If you would like I could provide you with a list of all the nodes that ‘makers’ have abandoned/not maintained in over three years. I created it a couple months ago.

History shows that makers do not ‘always’ maintain there code for what ever reason.

[Bobo]

@ozpos

At the top I asked if some HW devices could be considered The answer was an emphatic NO

I think the answer was an emphatic - it would be a good idea but it wasn't a priority for many based on other work, and other interests.

You haven't asked if anyone here might be willing to collaborate on getting one or two HW nodes built. To me that would seem to be your best marketing strategy.

[krambriw]

@ozpos

By all means say if it is a daft idea but please explain why ?

No, might not be a daft idea at all, it might just be me not understanding your use cases for NR. If it is industrial automation or for a laboratory I can understand but for a normal home user, I doubt it is or will be that common

I'm personally not using NR for such "low level" I/O, I just use "ready made" devices that have a higher level interface. Earlier did much more "low level", built and designed rf rx/tx devices, pic programming, c/c++ etc etc. With age comes convenience, nowadays I try to avoid such things (had however to do some c/c++ AGAIN, just to get my ESP32 doing the stuff I wanted). Also for I/O control, there are already many nodes available, there are nodes for OPC UA, BACnet, Modbus and you can purchase your own Siemens S7 controller and integrate it with NR. I mean if you really are going professional, it's just the cost

You mention Python several times. It might not be good for your use cases for various reasons but I have so many Python scripts running since years in my RPi's. It's working rock solid, very stable. If it consumes resources, I don't really care as long as it fits together with the rest within the limitations. Anyway, they are not time critical so then it's ok having not the best from performance point of view

This discussion is or should, as I read & understand it, also cover a discussion about interfacing and integrations in general. Is an important system architecture aspect. For many years I was the product owner and responsible for developing interfaces between many kind of systems and products to our professional management system used by customers in many countries. I had a team of 30+ developers and project managers, we used C#, C++, Python, Cython. Most of the effort was actually spent on "things" next to the coding; quality processes, system testing on all levels, development and end user documentation etc etc. What was an interesting take-away, or lesson learned, was that the maintenance workload increased dramatically the more interfaces we produced. New versions required due to new updates in the underlying systems and products, new features required...and please note, the interface to our management systems was still unchanged and the same, we used ActiveMQ, another high performance MQ system. After a certain time we spent more time and effort on maintaining existing interfaces than creating new ones

The difference was that we all had salary for doing this. Here with NR, we are expecting deliveries from an unpaid community. Expectations should therefore be tuned accordingly

[ozpos]

I was speaking from the position where node could not break their code, it was late and I got carried away a bit.

Ok modify always to some if you wish. The fact is there will be many many more if this decoupling is done.

Think about it this way. Lest say a maker fathoms out how to build and debug a front end interface. It would be a massive achievement and investment in time and effort to them, the bulk of which is well outside of their domain and comfort zone. Then it stops working because of a node bug fix. Their code is working fine.

Why should node break their C/python code?

Why would or should they go through all that again. This simply should not happen.

So they sit tight until support for their node version is no longer maintained and then they have a stark choice to make.

@zenofmud

If you would like I could provide you with a list of all the nodes that ‘makers’ have abandoned/not maintained in over three years. I created it a couple months ago.

Thats very kind and I believe you wholeheartedly. From the point at which it was clear to me that Hardware devices could not be maintained by the community I have been speaking from the decoupled point of view.

@zenofmud

History shows that makers do not ‘always’ maintain there code for what ever reason.

The reason is glaringly obvious to me. Why should makers maintain their code against node versions. There should be no coupling here at all.

[ozpos]

@Bobo

it would be a good idea but it wasn't a priority for many based on other work, and other interests .

That is exactly the view from a makers perspective but in the opposite direction. Why should they maintain their code against new versions of node, it has little to do with them.

@Bobo

You haven't asked if anyone here might be willing to collaborate on getting one or two HW nodes built. To me that would seem to be your best marketing strategy.

I have already been informed that there is little support for this decoupling idea and so that is the way things are.

I thought decoupling was a corner stone of solid software development and part of the coding mantra but clearly I am wrong here too.

I have at all times praised this project. It has achieved one of the hardest things around by abstracting away server side front end code so users can design without getting pulled down into the workings of http and html and css and on and on.

With lots of help (thanks to those) I have given it a go and dedicated 5 days to the problem fielding resistance from all sides.

What I cannot fathom is that this approach makes both ends of the spectrum happy. Your front ends remain stable with node changes and the makers code remains stable against node changes. No-brainer to me.

[knolleary]

@ozpos

What I cannot fathom is that this approach makes both ends of the spectrum happy. Your front ends remain stable with node changes and the makers code remains stable against node changes. No-brainer to me.

The resistance is because we have failed to grasp exactly what you are proposing and our attempts to clarify have so far been unsuccessful.

@ozpos

I thought decoupling was a corner stone of solid software development and part of the coding mantra but clearly I am wrong here too.

There are lots of ways to decouple things. Lots of the existing nodes are already decoupled - they use the python libraries to provide an abstraction over the hardware that can be called using a higher-level interface from the node.js code.

You are proposing a different sort of decoupling. But so far it has been light on concrete detail, which makes it hard to have any sort of technical discussion.

So lets talk about decoupling.

To have a Node-RED node that can interface with a piece of hardware you need, starting at the Node-RED end of things:

1. a Node-RED node. This will be some JavaScript code that runs in the node.js runtime environment. It will provide the HTML for the Node-RED editor that allows the user to configure what they want the node to do.

2. the JavaScript code of the node then needs someway of interfacing with the hardware. You have proposed a decoupled approach, using some messaging system. If you mean MQTT, then that means this node requires an MQTT Broker to be running locally.

3. Something is then subscribed to those MQTT topics to receive messages from the Node-RED node. That something is then responsible for driving the hardware as required. What we've been trying to understand is what that something is, who maintains it and what it does exactly.

To be honest, I am not convinced this will gain a lot over the python library approach some nodes already use. You are adding two TCP hops between the node-red node and the code that interacts directly with the hardware.

[ozpos]

I am not suggesting a change in focus just a way of getting rid of this problem once and for all. Node version changes should not break connected devices.

My use case may be more orientated towards the sharp end if Iot integration but the key word is integration.

@krambriw

I'm personally not using NR for such "low level" I/O, I just use "ready made" devices that have a higher level interface.

What could be a higher level interface than the likes of MQ.

I mention python only in the context of cramming as much data capture into a pi Zero. and keeping a little headroom. The language means nothing to me. Nor do any others for that matter. Some are nice to use some not but only because of familiarity and features that facilitate freedom of expression.

@krambriw

... so many Python scripts running since years in my RPi's

All with the latest node lts ?

@krambriw

consumes resources, I don't really care as long as it fits together with the rest

My sentiment also. If however you need ten Pi Zero W scattered about then things start to get expensive in comparison if you have to move up to a Pi4.

But there still remains the big problem of coupling to node.

@krambriw

What was an interesting take-away, or lesson learned, was that the maintenance workload increased dramatically the more interfaces we produced.

My proposed solution has MQ or similar as its interface and that is it.

@krambriw

...and please note, the interface to our management systems was still unchanged and the same

Which is exactly the position of the maker in relation to their embedded code (and the chips do not change). Node versions should not break low level drivers, end of.

@krambriw

The difference was that we all had salary for doing this. Here with NR, we are expecting deliveries from an unpaid community. Expectations should therefore be tuned accordingly

Of course. This way would get rid of this brittleness.

[JGKK]

If i grasp your idea for an approach correctly you would want code to run on something like an arduino or even a pi that offers a standardized way to act as a proxy/bridge to forward/bridge communication between sensor hardware and nodered directly via the arduino or esp? Is this right so far?
The problem i see that when you would want to develop something like this Most of the code and effort would actually have to live outside of the scope of nodered as a library or similar for arduino. It looks to me that it be the more efficient approach to develop that library for the esps and arduinos with a standardized interface over mqtt that is well documented but can be used by anything and that can talk over an mqtt broker. So from nodered at that point why not just use the inbuild mqtt nodes?
And isn’t this the point where we are back to where there are already projects like tasmota, esp easy or the myriad of great libraries for arduino for all kinds of sensors you could think of out there which allow you to do this and if they dont do it put development effort there?
Maybe i am beeing daft here but me like others here are failing to see the exact structure you have in mind? Can you take me through it step by step?
I use a whole lot of esps connected to nodered over mqtt that have a whole lot of different sensors connected to them and from my knowledge point of view what your are trying to achieve sounds like rebuilding a whole eco system that already exists in other projects like arduino and that can be easily worked with from nodered due to one of its biggest strengths which is its interoperability over different protocols.
Im sorry if this is beeing dumb from my side but i just dont seem to be able to quite fully grasp what you are proposing so please take me through it like a little child.

[ozpos]

1. yes.

2. yes see #16 @ristomatti. Or #37 @totallyinformation, #57 @krambriw, #62 + #66 @cinhcet

3. No

@knolleary

That something is then responsible for driving the hardware as required.

The maker (within their field of expertise) receives a msg from 1. unpacks it and uses the target hardware drivers to talk to the chip and read back a response (could be a null response) This response is sent out of the serial port to MQ (not done this but other say it works fine). Note any real-time processing is done on the target node and does not need to go back and forth to NR. For example a batch of analog readings can be maintained within the target and flushed to NR when required, the choice is down to the maker and different depending on the type of device.

I do not think there is a something. Surely the UI needs to subscribe to the returned message so it can display or react to the results of the outgoing message to the target.

@knolleary

To be honest, I am not convinced this will gain a lot over the python library approach some nodes already use. You are adding two TCP hops between the node-red node and the code that interacts directly with the hardware.

Those python end points can remain python end points, they are just not connected/compiled against node. They connect to NR via MQ.

Yes 2 hops. One piddly message once per second so what. If time is critical then batch and timestamp the results on the target. I am not proposing nitty gritty handshake comms between an I2c bus and NR but I suppose it is quite doable and pretty low overhead but I am not proposing that.

[ristomatti]

Would setting up RPiEasy with its GUI to interface with NR through MQTT be close to what you're after? Let's imagine its hardware interfaces were written in C/C++ and not Python.

If it's not, what would be the core differences?

[afelix]

@ozpos

Yes 2 hops. One piddly message once per second so what. If time is critical then batch results on the target. I am not proposing nitty gritty handshake comms between an I2c bus and NR but I suppose it is quite doable and pretty low overhead but I am not proposing that.

Correct me if I’m wrong, but wasn’t the whole idea to approach this on low level because of the time overhead and Node-red being less suitable for real-time I/O operations? What is the use of adding this low-level overhead to it?

[knolleary]

What motivated this whole thread from your perspective was the fact that us dropping Node 8 might break some nodes. And that you believed, rightly, that nodes should Just Work.

We have pointed out we have a model where we use the python libraries as a stable API between the Node world and the underlying hardware. That model should never get broken by node version changes.

Your proposal to insert MQ between the node-red node and the hardware driver is just a different approach to achieve the same decoupling - albeit with more moving parts needed (the MQTT Broker).

It doesn't change the fact that someone has to write code to talk to the hardware and expose it via an interface for others to use. And if you want nodes that 'just work' for new users, that code has to be maintained.