Npm, versioning and palette manager confusions

I have confirmed that the basic issue here is a regression in node-red. I have updated another system that had a number of nodes showing the update option in the palette manager, from node-red 1.2.3 to 3.2.1 and now only one node is showing an update. Could it be to do with the fact on both systems I have a node installed direct from git, so in package.json I have
"node-red-contrib-owfs": "git://github.com/colinl/node-red-contrib-owfs.git#automatic_retry",

Your suggested command shows them all, and also npm modules I have installed

$ npm outdated --parseable | grep -v git | cut -d ":" -f 4 
match@1.2.10
moment@2.29.1
node-red-contrib-queue-gate@1.5.2
node-red-contrib-simple-gate@0.5.0
node-red-node-email@1.11.0

The palette manager has no knowledge of what your package.json says. None at all.

All it knows is the exact version of the module that has been loaded, and the version that is offered by the catalog and then makes a decision based on that and that alone.

We did merge a contribution that was intended to improve the comparison of versions when a version has additional labels (such as 1.2.3-someText). I wonder if that has gone awry in some cases.

I will see if I can reproduce using one of the modules you mention.

On the latest system, of the three nodes due an update queue-gate, simple-gate and email, only queue-gate is offered in palette manager.

On the original system fs-ops, influxdb, msg-resend, and ping are offered but not queue-gate, uibuilder, dashboard and email.

I note that queue gate is offered on the first (1.5.1 to 1.5.2) but not the second (1.5.0 to 1.5.2), so what can be made of that I don't know.

It does appear to be consistent. I installed node-red-contrib-queue-gate@1.5.0 on a third system and it did not offer the update, but if I install 1.5.1 it does offer it.

@Colin thanks - I have reproduced and pushed the fix. It'll be in 1.3.3, although not going to rush that out today.

As I mentioned, we merged a change to support better semver comparisons when a version has extra information in it. There was a bug in that code that meant if the change in version number was >1, then it was overlooked. This is why you were seeing it work with 1.5.1->1.5.2 (because 1->2 is a difference of 1) but not 1.5.0->1.5.2 (0->2 is not, hold on to your hat, a difference of 1).

1 Like

Excellent

It would be good to see this one fixed, assuming it is a bug, as this one does break things. Node-red 1.3.2 MQTT LWT regression

I don't think this should happen without some interaction with the user. Possibly being told what it is going to upgrade and being given the option to say yes or no. Also worth considering whether it should update across major version numbers as that could involve breaking changes. That all gets a bit tedious in a shell script though doesn't it.

As Requested...

npm outdated --parseable | grep -v git | cut -d ":" -f 4
pi@rpi418buster:.node-red:22:34[0]> npm outdated --parseable | grep -v git | cut -d ":" -f 4
@node-red-contrib-themes/midnight-red@1.5.0
dateformat@4.5.1
i2c-bus@5.2.1
moment@2.29.1
node-red-contrib-tasmota@0.9.9
node-red-contrib-ui-time-scheduler@1.10.1
node-red-contrib-virtual-smart-home@1.23.1
node-red-contrib-web-worldmap@2.13.2
node-red-node-email@1.11.0
node-red-node-rbe@0.5.0
node-red-node-sqlite@0.6.0
node-red-node-tail@0.3.1

Is that helpful? Don't assume too much knowledge in replying. I simply followed your instruction and sent you the output.

Its not a test machine but I have clone SDs (for reasons beyond me and nothing to do with this discussion my clone SSD will not boot even after a FULL clone - but the SD clones of my main SSD do work - hence the info above).

I didn't ENTIRELY understand your next paragraph... run npm i etc... so that would do some kind of npm install over the existing 10.x before running your upgrade script?

Happy to try that in daylight on a clone.... over to you Dave

Not sure I'm the best person to ask here as my view is that people should learn how to do things "properly" - by which I (very opinionatedly) mean using native tools.

A lack of understanding is always going to result in confusion for at least a few people when edge cases don't quite work as expected.

Still, I am known to be a bit weird :grinning:

I would say that, if you are going to try and update things, you need to tell people what you are going to do and give them a clue and a pointer as to why.

Again, my view is that keeping things up-to-date is generally much safer than "if it ain't broke don't fix it". I've seen far too many painful, late nights sweating over nigh-on impossible upgrades.

So I would say that, in my opinion (for whatever it is worth), yes try to update everything, but keep the user informed with at least links out to more documentation. But I realise that this might be more work than time might permit.

It should run npm install of all the listed modules to bring them up to the latest versions. So, for example, the sqlite node should be up to date so that when you run the script you shouldn't have the build problems that you encountered (the version you currently have installed is not compatible with the newer nodejs). At least that is the current thinking.

First question

npm i $(npm outdated --parseable | grep -v git | cut -d ":" -f 4 | xargs)

Do I do that in the PI folder or the .node-red folder?

Pete

Hi Pete, in the .node-red folder please

I heartily agree, but (once again) you've found the rub. There isn't much point in an intuitive, low-code programming environment if keeping it working requires deep knowledge of operating system, package management, and version control stuff. That's why Dave's installation script is such a big contribution and worth the effort to maintain. I could live without it now but not when I started using NR. Sure there are edge cases, but there are also a lot of questions on the forum from folks who have tried DIY or followed outdated or incomplete recipes from YouTube.

[EDIT]: None of this was meant as a knock on @scargill's script, blog, or YouTube channel!!! I have followed them for years and suspect that was where I first learned of NR.

1 Like

Something missing there I think, Dave. I ran

npm i $(npm outdated --parseable | grep -v git | cut -d ":" -f 4 | xargs)

in the .node-red folder
(which didn't throw any errors)
I then ran the NR install/upgrade script in the /home/pi folder .. It removed the "old" node-red but left the existing node v10.24.1 and npm 6.14.12

the operation took only a couple of minutes.

This can take 20-30 minutes on the slower Pi versions - please wait.
  Stop Node-RED                       ✔
  Remove old version of Node-RED      ✔
  Remove old version of Node.js       -
  Leave existing Node.js              -   Node v10.24.1   Npm 6.14.12
  Clean npm cache                     -
  Install Node-RED core               ✔   1.3.2
  Move global nodes to local          -
  Install extra Pi nodes              ✔
  Npm rebuild existing nodes          -
  Add shortcut commands               ✔
  Update systemd script               ✔
Any errors will be logged to   /var/log/nodered-install.log
All done.
  You can now start Node-RED with the command  node-red-start
  or using the icon under   Menu / Programming / Node-RED
  Then point your browser to localhost:1880 or http://{your_pi_ip-address}:1880
Started  Thu 15 Apr 09:50:58 CEST 2021  -  Finished  Thu 15 Apr 09:52:42 CEST 2021
pi@rpi418buster:~:09:52[0]>

So, what happened to updating nodejs etc?
Here's the output - and that Tasmota error is new - never had that before...

node-red-start

Start Node-RED

Once Node-RED has started, point a browser at http://192.168.1.19:1880
On Pi Node-RED works better with the Firefox or Chrome browser

Use   node-red-stop                          to stop Node-RED
Use   node-red-start                         to start Node-RED again
Use   node-red-log                           to view the recent log output
Use   sudo systemctl enable nodered.service  to autostart Node-RED at every boot
Use   sudo systemctl disable nodered.service to disable autostart on boot

To find more nodes and example flows - go to http://flows.nodered.org

Starting as a systemd service.
15 Apr 09:57:01 - [info]
Welcome to Node-RED
===================
15 Apr 09:57:01 - [info] Node-RED version: v1.3.2
15 Apr 09:57:01 - [info] Node.js  version: v10.24.1
15 Apr 09:57:01 - [info] Linux 5.10.17-v7l+ arm LE
15 Apr 09:57:02 - [info] Loading palette nodes
15 Apr 09:57:09 - [info] Worldmap version 2.13.2
15 Apr 09:57:09 - [info] Dashboard version 2.28.2 started at /ui
15 Apr 09:57:11 - [info] Settings file  : /home/pi/.node-red/settings.js
15 Apr 09:57:11 - [info] HTTP Static    : /home/pi/.node-red/public
15 Apr 09:57:11 - [info] Context store  : 'default' [module=localfilesystem]
15 Apr 09:57:11 - [info] User directory : /home/pi/.node-red
15 Apr 09:57:11 - [warn] Projects disabled : set editorTheme.projects.enabled=true to enable
15 Apr 09:57:11 - [info] Flows file     : /home/pi/.node-red/flows.json
15 Apr 09:57:11 - [info] Server now running at http://127.0.0.1:1880/
15 Apr 09:57:11 - [info] Starting flows
15 Apr 09:57:12 - [warn] [Tasmota Light:lt21 light] Broker configuration is wrong or missing, please review the node settings
15 Apr 09:57:12 - [error] [Tasmota Light:d6eb53f7.49dc7] TypeError: Cannot read property 'subscribe' of null
15 Apr 09:57:12 - [warn] [Tasmota Sensor:lt21 sense] Broker configuration is wrong or missing, please review the node settings
15 Apr 09:57:12 - [error] [Tasmota Sensor:85004139.b711a] TypeError: Cannot read property 'subscribe' of null
15 Apr 09:57:12 - [warn] [Tasmota Switch:lt21 switch] Broker configuration is wrong or missing, please review the node settings
15 Apr 09:57:12 - [error] [Tasmota Switch:1e77de9c.1b5321] TypeError: Cannot read property 'subscribe' of null
15 Apr 09:57:13 - [info] Started flows
15 Apr 09:57:13 - [info] [sqlitedb:6e9a0125.59bc4] opened /home/pi/dbs/iot.db ok
15 Apr 09:57:13 - [info] [sqlitedb:fdbe27fe.fb7848] opened /home/pi/dbs/iot.db ok
15 Apr 09:57:13 - [info] [sqlitedb:33875a87.57ade6] opened /home/pi/dbs/iot.db ok
15 Apr 09:57:13 - [info] [mqtt-broker:4c682b3a.2ab5c4] Connected to broker: mqtt://127.0.0.1:1883
15 Apr 09:57:20 - [info] [e-mail:peterscargill@gmail.com] Message sent: 250 2.0.0 OK  1618473440 h81sm1739176wmf.41 - gsmtp

Definitely still old node - node -v gives v10.24.1

The script won't upgrade nodejs v10 as v10 is ok. How did you get to v12 last time?

The tasmota error suggests there is a problem with the upgraded tasmota node. I don't think that is anything to do with the other problems you have been having.

The tasmota problem is probably this one, though the symptom is not quite the same. There is a recovery strategy there if you read through the thread. Personally I prefer to use MQTT directly rather than going through a contrib node for controlling tasmota devices. Problem after update to 0.9.8 · Issue #26 · DaveMDS/node-red-contrib-tasmota · GitHub

@dceejay, Pete's issue with the Tasmota node does bring up a further question about forcing nodes to be upgraded. Ideally the user should upgrade everything and then check that the system still runs ok before upgrading node-red and nodejs. Otherwise one may end up with issues where it isn't clear whether the contrib node upgrade is the problem or the nodejs/node-red upgrade.
Perhaps another option would be to check for outdated nodes, tell the user what they are, tell him/her he should upgrade them first and only carry with the nodejs/node-red update if the user says to do it anyway.

Hi Colin

The Tasmota node - ignore that - red herring - it seems the (Blitzwold BW-LT21) bulb is shot - won't show up on my nwtwork.. how did I get to NPM 12 last time - no idea.. I though the idea of dave's latest suggestion was to get the setup to move to v14.... I'm wondering if he made an assumption - I just used his latest instruction above and the NR upgrade script... so I'm no further forward :slight_smile: - do I need to do anything else now to get v14?

I think that was the nub of my personal approach and recommendation. If you don't know how to use apt and npm, how do you know in what order to do things?

You make a good point of course. However, I might counter by saying that if this is the case, Dave's script needs more work and needs to become more of a central feature. And then it raises issues about other platforms and we get into problems with 2-tier approaches and certain OS's being left behind.

And yes, this is by no means a dig at Dave's script which is excellent but we are highlighting the issues. The more a script attempts to take on, the more complex and potentially less robust it becomes. At the least, you start finding all the edge-cases - as indeed we are.

At what point do you say that actually showing people how to use your OS's package manager and npm is the best approach? Since you certainly need to know something about managing at least your OS just to use your computer. You also need to know something about managing npm packages to realistically work with Node-RED even if that is just using the nice palette manager.

If you are a Windows, MacOS or Fedora user, you already can't use the script so you are already on your own (actually not quite fair since the docs do tell you how to install things).

1 Like

If you uninstall nodejs using apt then the script will install nodejs 12. It would probably be worth doing that first to make sure the problems with sqlite node rebuild have gone.

Then if you want to get 14 (do you have a reason for wanting to do that?) then edit the file /etc/apt/sources.list.d/nodesource.list and change all the 10s to 14s. Then run
sudo apt update && sudo apt full-upgrade
which should install nodejs 14. Then go into your .node-red folder and run
npm rebuild
then run the script to rebuild node-red.