First a big thank you to all involved in 1.1.0. Looks fantastic and what a great choice of features.
My concern is with low level efficient modules that interface with real-time hardware on Pi4 and Pi Zero via GPIO, I2C and SPI using user-space privileged access.
Working backwards from the latest supported version of node we ended up with 8.17.0 after testing several modules against each release in turn.
The combination of hardware (max31855, Si7021 and mcp23017) meant we had to write our own in C using the very capable pigpiod_if2 library (python maxed-out the Pi Zero).
Is there a way to improve this situation ?
For instance could a single set of drivers for light level, temperature and humidity be maintained in step with the latest NR and node lts versions as templates to help lower the bar of entry a little ?
Hi, Thanks for the response. I expected that would be the case. My reply below is not arguing with the facts about resources or trying to pile more work onto the team but it is more about priorities and an attempt to gather support for a way of improving the situation.
Could I paint a plausible external perspective;
Interested pundits browse for HA or IOT and discover Node-Red as 'A flow-based programming tool for wiring together hardware devices, APIs, and online services' (IBM). So they look at all the fantastic stuff going on with MQ and the cloud and online services and loads of other high level stuff. However when they try to get simple low cost (end-user accessible) hardware to produce basic temperature/humidity data or whatever they are at a loss to find working examples and gain anything like a toe hold. For them to succeed they would have to turn into real-time embedded systems engineers. How many potential users hit this wall ?
I used to be a real-time embedded systems engineer with a degree in electronics and more than 30 years self employed real-time software experience. My interest began when NR was opensourced and the Raspberry Pi had taken off big time. Just to remind others of the volumes involved here 'as of December 2019, more than 30 million boards had been sold' (Wikipedia) and in Q1 2020 a further 1.75 million Pis were added to those mega sales volumes (electronicsweekly.com).
Back at the start it was my opinion (influenced by actual tests) that the Pi lacked enough resources to run NR and a slimmed down browser as an in situ GUI. Later things improved and lots of hardware nodes popped up over time. A couple of years ago I gave it another go and found that those promising hardware related nodes had not kept up with the rapid development of node and were only OK with v8.17.0. The chips I needed drivers for (max31855, Si7021, MCP20317) did have python libraries but these were too bloated for the Pi Zero W and took a big chunk of cpu bandwidth even from a Pi4 and none of them worked with more up to date versions of node. Getting one second readings into an influxDB was extremely tight. I managed to get the very capable and mature pigpiod_if2 based library (user space privileged access) to work using C. The end result is not ideal with exec nodes loading and unloading all over the place but it does work and I have systems with > 1yr up-time (a big thanks to NR devs. here, this code base is very stable indeed).
Have I been unlucky in my choice of chips (perhaps), are my embedded skills not what they used to be (for sure).
Has anyone estimated how much effort it would take to include a minimal set of hardware nodes into the automatic build, test and release machine ?
again - short answer is no -- no-one has estimated what it would take, other than more time than we have available.We used to have a couple of hardware specific nodes in the core (serialport and pi-gpio) and we moved both of those out as indeed they started causing issues with respect to the underlying library updates not supporting (and then dropping support for) various versions of node.js etc.
If someone wanted to setup and support a list as you propose then I'm sure we would all be delighted. Nick is the only one here actually paid to look after Node-RED full time, the rest of us help out as best we can. So Nick is focussed on the core naturally, and the rest is up to the community. As you no doubt appreciate there are many different operating systems and platforms and chips and APIs - all of which can be combined in myriad ways so trying to select one above another is a tricky task.
I fully appreciate the situation now however it is still a little absurd that the very purpose of NR is to put the interconnection of Iot devices in the hands of non programmers who then go on to discover that they cannot find any basic working examples of physical I/O.
As for OS and hardware just choose the most popular. I believe such a move would bring into the fold many more willing and capable hands. The flip side will likely be stagnation and eventually oblivion which would be a tragic waste of all that good will time and effort put in by all.
From a commercial perspective a lot of big business hate the Pi especially when it killed the very lucrative RS232/485/parallel IO to Ethernet adapter market that could ask ÂŁ350 1 off for a lump of DIN rail mounted plastic with a couple of chips on a 2" square pcb and you still need a controller to make it go. This puts a big damper on any financial support from the industrial control business sector. Instead I think one place to look would be with an existing manufacturer of Pi expansion boards or even a chip manufacturer where there is a clear financial interest in expanding their market share. No one can ignore the potential of 30+ million devices out there.
In my experience the Pi Zero W (good $10 Iot price tag) works OK if you are careful not to use unnecessary bloated js libraries and now 8Gb RAM on its big sister/brother ensures you have enough resources to run a touch interface GUI home automation control panel for less than the cost of a Nest thermostat !
Your latest version is fantastic and I really believe that will satisfy your current user base so now is the time to seize the moment and bring back in house some simple physical I/O.
When 8.17.0 is dropped it will not matter what features are in the next version it will be useless without Iot data input.
Well a good place to start would be to help create an inventory of nodes that
a) you think should be supported, and
b) which you think are broken after 8.17. (and/or 10.x)
Even just creating and curating a list of known good nodes would be a good resource to point people at - and give folk a target list of nodes they could maybe help maintain. Would you be up for that ?
It is true to say that is one purpose of Node-RED, but its scope of usage is much broader then that.
But the reality of the situation is Node 8.17 was dropped by Node themselves at the end of last year. It is no longer receiving fixes of any kind. There could be a zero-day security vulnerability discovered and we would be powerless to do anything about it. We can't wish that away. Our plan means the 1.x branch would still be supported, running on Node 8.x for over a year.
It would certainly be great to have a well maintained set of libraries for common peripherals. I've no doubt the demand for such libraries is much broader than just Node-RED. And if there are members of the community who have the skills and time, then it would be worth pursuing. But taking on long-term maintainer-ship of hardware libraries requires very particular sets of skills and resources to do. The core of the project doesn't have limitless resources. In fact, it barely has enough to deliver what it does already.
As the Iot attack surface balloons and people start to experience a lack of control or they find a hacker has switched their lights and dryer on then they will realise that the convenience of turning on/off devices remotely is not really worth the risk and there will be a move away from the cloud back to local control and better peace of mind and security. NR could be there already, established and trusted to pick up the pieces.
Does one think that an insurance company will pay out if their house burns down as a result of a smart home hack ?
I came to this forum with congratulations on the latest release and also to share my perception and experiences with Node-RED.
Collectively you have created a superb robust platform that abstracts away an enormous amount of complexity so many more users can gain access and build remarkable and useful systems.
I am sorry in advance if my response is a little brutal or rude and I hope my perception is completely wrong but I have just skimmed a minuscule fraction of discussions across the forum and it appears that this has been touched on before more than once but never gained any prominent collective opinion or momentum so why would one do the same again ?
I ask the community as a whole, what is the point of Node-RED without easy plug and play hardware input ?
Could the problem be that the majority of contributing members just have their own personal systems to maintain and of course as volunteers you have limited time to commit. That is perfectly ok I suppose as long as your systems can be built from source then things are just dandy ?
What about a consensus of opinion before more effort is just ignored or left in the long grass ?
There is plenty of "point" of Node-RED without any direct hardware integration.
Many people for example use esp32 devices as hardware input which communicate via MQTT with Node-RED. Or others use Z-Wave, or Modbus or KNX or just serial.
The reason why Node-RED is so successful in my opinion is that it is NOT opinionated about the hardware options.
As has been pointed out several times now, a constructive way of approaching your point would be to compile a list of what you think is essential hardware integration? And why it will break in future versions of Node?
Hi, Sorry I do not know how to thread by response to yours as you just did with mine.
Are you saying that Node-RED Iot hardware input is not important to the future of Node-RED ?
Could you please tell me what the main scope of usage is ?
Re 8.17: I do not suppose anyone wants to stay with an out of date and unsupported version of node, me included. So installations stuck on 8.x have a year to do what exactly ?
Libraries: The Pi has C/Python libraries already. The most efficient of which in my opinion is pigpio @ abyz.me.uk. Where microsecond sampling and I2C/SPI/PWM on any pin can be achieved fairly easily if you are a programmer. Adafruit libraries are more about learning than efficiency. A $10 Pi used for fast sampling does not work well on a huge slab of Python code. If there is to be no abstraction/visibility within the Node-RED echo system then I think many Pi users will simply see NR as irrelevant to their I/O related projects. We all know how far people can expand their knowledge and capability when presented with a way forward in the form of an example. That must be orders of magnitude easier than taking over the maintenance of an entire library of code (especially real-time embedded code).
Has the Node-RED user population use case been published ?
assumptions: Possibly but the hardware devices you sight have an abstract interface within Node-RED. Why is there to be no I2c, SPI or even Pi GPIO. Just look at the Pi volumes and the $10 price tag. These situations/opportunities do not come around very often Just how many chips do you think have abstracted their inner workings via I2C or SPI ?
Lists: Sorry but consensus must come before construction.
You said IoT Hardware input was the "very purpose" of Node-RED. I was making the point that it is more broadly applicable. That doesn't mean IoT is not important - of course it is. But we have to keep it in perspective in terms of where we prioritise the work we do. There are a long list of companies who embed Node-RED in their own commercial products and services - some IoT related, some not.
Find a route to get off 8.17. If there are components that are as critical as you say currently stuck on 8.17, then I would hope the wider community (wider than just Node-RED) would find a path to help move forward. But to do that, they need to be identified - which is why getting a list of things that are stuck on 8.x would be a useful first step to help understand the scope of the problem.
We are trying to understand what you are actually proposing and to do that, we're seeking a more concrete sense of what you think should be done. Without that understanding, you're not going to achieve a consensus.
Are you suggesting the Node-RED project should be responsible for the ongoing maintenance of low-level libraries for accessing some undefined set of hardware?
Or do you mean we should provide a set of Node-RED nodes that make use of libraries that other people have created? You mention the pigpio library - we already have nodes for that - node-red-node-pi-gpiod (node) - Node-RED
You originally asked:
Yes - that absolutely could happen if there were people in the community willing to take on that effort. We'd welcome it with open arms. That is what being an open source community is about.
All my comments are under pinned by the fact that the Pi has an unrivalled volume of sales, that the Pi Zero W can run NR and 8GB RAM is on its way. Node-RED is already in the game and right there when the box is opened. As the RAM increases there is the strong possibility that one of your many competitors could take your place.
This is the impression given by wikipedia.
I totally understand and agree but at the same time surprised that the volumes of people using these devices does not focus the team even more towards cementing your presence on the Pi.
With similar volumes ? Could there be a conflict of interest then if these companies are not using the Pi ? Which camp is influencing the direction of travel the most ?
User-space access to SPI for max31855 (thermocouple amp), I2C for Si7021(hunidity and temperature) and MCP23017(16 way port expander all pretty popular I think. The most recent node lts version that worked was 8.17. The MCP23017 was the most difficult part to drive. The wider community would not have any influence or inclination to fix a node in Node-RED.
From the survey:-
Most respondents use Node-RED in a personal capacity, for home automation and similar tasks.
However, Node-RED is being used in production and there is a growing amount of commercial usage.
Speaking from a hardware perspective Home Automation is more than just lights and switches. Diversity and flexibility shrink rapidly without I2C, SPI chip hookups plus you run out of ports very quickly without the likes of 23017.
If HA is the largest group then in order of preference include as core modules at least a port expander, multiplexed analog input, thermocouple amplifier, humidity sensor, zigbee, PWM and of course servo motor control.
No.
That node does not expose I2C or SPI or the pipe interface and scripting. Provide for instance MCP23017 and max31855 interfaces to the pigpiod library.
Here we are back to priorities again. I believe that the majority of hardware oriented Pi users will not use Node-RED if they cannot plug in some hardware and wire it up on screen.
I do not have sufficient js skills to do this otherwise I would have offered it up for consumption. I stayed well clear of this language throughout my career. The control and monitoring systems I used to design for the hospitality industry would not allow any js on their back office systems. Imagine how dull the GUI was !
I am not so sure that would be the case. A RPi is good for much more than plugging in hardware modules. Indeed I would suggest that may make up only a small part of the use cases out there. After all, there is better (and cheaper) hardware out there now just to provide basic micro-controller capability.
But I have NR running all sorts of "magical" stuff on RPi, with some quite important flows maintaining feeds from things like emergency services providing instant updates to mobile devices.
I have to say I disagree on the importance of this. These are common needs for a maker but as one myself and active home automaton user, my experience hasn't shown a Raspberry Pi to be the best hardware platform for these usages.
Yes it can definitely be used for that with drivers written in C or dedicated Python scripts (due to the fact Raspberry Pi foundation has focuses on Python and most existing libraries are for it). My first attempts of interaction with sensors using JavaScript was such a hit and miss, I decided to dig into Arduino's and everything just snapped together. Nowadays you get Arduino compatible ESP8266/ESP32 development boards for a couple of bucks and ready made firmware which you can flash to commnicate with your Node-RED instance using MQTT.
Node.js/JavaScript is not the ideal platform for real-time computation and Node-RED runs on top of it. I'm not saying it cannot be done but it's not Ideal. As a home automation focused Node-RED user I've instead found NR excellent for orchestrating the automation, communicating with both various home automation products and my own custom (Arduino) devices.
Yes I know, very capable devices. My experience began as soon as I could get my hands on one. I have used them for kodi, pihole and home automation. The capability of a Pi Zero W if a long way from a Pi 4. I spent a lot of time stripping out bloated js libraries left right and centre in an attempt to move an app from a Pi 4 to a Zero.
From memory, I think I put "home automation" as well. To me, if you are using it at home, for the benefit of those at home, it is all home automation to some extent.
I don't think that demonstrates much about how/what hardware devices are directly connected to a RPi, therefore I don't think you can make the conclusion you did, imo naturally.
I think @ozpos you will find that your perspective on home automation is a niche within the field that people mean when they clicked that fiel in the survey. Which itself is only a limited part of the applications that nodered is meant for.
I think very little people actually start connecting their own sensors to their pis or even use arduinos/esps that they program themselves when they get into home automation.
What i would say 90 % of people in home automation do is either use something like openhab, home assistant, io broker, fhem and so on and connect nodered to that as they have the interfaces/bindings to all the commercial home automation products that people will actually use before they ever wire a single sensor over gpio pins.
This is what nodered is good at as people pointed out above. Nodered is perfect for connecting things and orchestrating workflows/automation which also includes diy solutions that people solder together but it may not be the one stop shop for your personal needs. Thats why those nodes connecting to other systems that cover those niches are so popular and i would say for the use case of home automation very much more important for the mass of users than low level hardware support.
I think you have a certain angle on what home automation means to you and what you want to use nodered for which is build by your experience and needs which is of course very valid but this is just not what many people will do. You are trying to push hard in this thread that this is what nodered should be and where the focus should be but shouldn’t it give you thought that nobody has come to say yes this is what we have been waiting we had the focus wrong all along?
Your usecase is very valid which has been said by senior community members above but its not the one thing that nodered will focus on and it shouldn’t.
If you feel that nodered is lacking for your use case use its strength which is its open source nature, low entry point for development of nodes and flexibility.
Contribute to those areas lacking add to the whole instead of trying to push people that they should change it so it fits your perspective. Thats what open source is about after all less complaining because you have no influence like in commercial projects and more contributing so everybody has something for their use case in the end.
Hey @ozpos I would suggest now is a great time to get to grips with it & join in.
Its a lot of fun & surprisingly you can do SO MUCH with JS these days. You could even use typescript if you cant get over the "no types" issue of JS.
It took me a couple of month to stop complaining about no types & now - I really (suprisingly) enjoy writing JS / NodeJS stuff.
You seem to be quite knowledgeable on these matters & I bet your experience (should you step up and do some development for node-red) would benefit us all immensely.
Cheers, Steve.
EDIT: Forgot to say, I suspect there are many NON home automation users - perhaps more non HA users. I find professionals and systems integrators tend not to hang around the forum as much. I use node-red in an industrial setting for integrating old tech with new. I would bet there are more non HA users NOT active on the forum than there are active.
