Thank heavens for frequent backups. This morning on my Raspberry Pi 3 B+ I noticed that node-red-contrib-blynk-ws was a version behind and after patiently doing a backup using the SD-backup program, and in the /home/pi/.node-red directory, having stopped Node-Red running, I did an npm uiinstall of the latest node-red-contrib-blynk-ws only to find, not for the first time, that 3 packages had been updated and 528 REMOVED. Running Node-Red revealed "cannot find module #i2c-bus'" which turned out to be the tip of the iceberg. My simple update had wiped most of my nodes. This happens a lot and is extremely frustrating. The current build is only a month old.
This is due to the version of npm you are using.
NPM now tries to ensure your node_modules directory reflects what you have in your package.json file. It will remove any modules that are not listed in your package.json file (or are not part of the dependency tree of those modules).
Any module installed via the Palette Manager in node-red will get added to your package.json file.
This mainly concerns modules you have manually installed.
You have two options:
-
Downgrade npm to an older version that doesn't do this (not advised)
-
Make sure you list the modules you want in the package.json file. The newer version of npm will automatically add things to your package file when you run an install - the older version required you to add an extra command line argument to get it to update the file.
Nick
The first failure of the day was while updating nodes in the palette manager. I tried that first.
The palette manager uses npm.
If you had nodes already installed that were not listed in your package.json then npm will have removed them - regardless of how you ran npm.
Have you installed node-red-contrib-npm?
If yes, this may cause the issue.
@cflurin why do you say that? Given everything I've explained, can you explain why installing that module will have had an impact here?
I had a similar issue on a rpi using node-red-contrib-npm
Removing the node has fixed the issue.
I didn't further investigate.
Edit:
The node-red-contrib-npm will also install npm v2.15.12 in .node-red/node_modules
Question: which npm does node-red use when installing/uninstalling with the manage-palette?
Hi
Note the package.json file below....
Any idea why some entries have a tilde before the version?
Pete
{
"name": "node-red-project",
"description": "A Node-RED Project",
"version": "0.0.1",
"dependencies": {
"i2c-bus": "^4.0.1",
"node-red-contrib-bigtimer": "^1.9.2",
"node-red-contrib-blynk-ws": "~0.7.1",
"node-red-contrib-chatbot": "~0.12.5",
"node-red-contrib-diode": "^1.0.1",
"node-red-contrib-esplogin": "^1.3.6",
"node-red-contrib-geofence": "^0.2.4",
"node-red-contrib-i2c": "~0.5.2",
"node-red-contrib-influxdb": "~0.2.1",
"node-red-contrib-owntracks": "0.0.2",
"node-red-contrib-sqldbs": "^1.1.8",
"node-red-contrib-timeout": "^1.0.7",
"node-red-contrib-web-worldmap": "~1.1.12",
"node-red-dashboard": "~2.9.6",
"node-red-node-arduino": "0.0.17",
"node-red-node-ping": "0.0.15",
"node-red-node-sqlite": "^0.3.0",
"sqlitedb": "^1.0.0"
}
}
See the semver section within the npm docs on their website for a detailed explanation
Sadly I think after checking that section that the carat might have something to do with breaking changes and sure enough now all is working BUT SQLITE is regularly reporting that my database (thankfully not critical) is locked.. msg : error
"Error: SQLITE_BUSY: database is locked" - deleting the journal file didn't help. I wasn't getting this issue until today's node updates.
Hi Nick - the nodes are now working (I;m getting a sqlite locked message but I only use sqlite for logging as it happens) and I did NOT manually update the package.json file which appears to have everything in it.
Is there a way to automatically bring package.json up to date so it has all installed (or, even better, all used) nodes there?
No there is not. Sorry.
OK, thanks.
Do you know which version this appeared in, to save us searching the release notes? If you don't remember then I will look it up.
As far as I remember, it was npm 5 that introduces this behaviour.
There is no automatic way of fixing it, but can you advise how to manually construct a package.json file to get my flows running again?
I do have npm logs, could I extract the nodes I used to have from there?
Note: the logs all end with an error
Error: EACCES: permission denied, access '/home/pi/.node-red/node_modules/bcryptjs'
Is this related or a different problem?
Edit : Node-Red v0.19.2
node.js 8.11.1
npm 6.4.0
If you want to do that, you can edit the JSON file manually. Though I'd advise using an editor that understands the format. Such as VScode.
Here is an example from my dev machine:
{
"name": "node-red-project",
"description": "A Node-RED Project",
"version": "0.0.1",
"scripts": {
"start": "node ../node_modules/node-red/red.js --userDir ../data",
"update": "cd ..; npm install --unsafe-perm node-red",
"check": "npm outdated",
"check-data": "npm outdated",
"update-data": "npm update",
"check-master": "cd ..; npm outdated",
"update-master": "cd ..; npm update",
"log": "sudo journalctl -u nrlive -f -n 0 -o cat"
},
"dependencies": {
"node-red-contrib-blockly": "github:bartbutenaers/node-red-contrib-blockly",
"node-red-contrib-moment": "*",
"node-red-contrib-uibuilder": "*",
"node-red-node-google": "*",
"node-red-node-sqlite": "*",
"vue": "*"
}
}
You need to understand the format for the versions. I've taken a shortcut here since I installed these manually (not via the admin ui). I used "*" which will always pick up the latest version. This is a little dangerous because you may pick up a major version change that breaks something. However, if that does happen, it is easily undone. You'll also notice that I am installing Bart's Blockly node direct from GitHub to get the latest development version. If you use *, then go back to using the NR admin ui, it will change the entries to something safer.
... Sorry, just coming into London, need to get off the train. I'll pick up the other questions later unless someone has already answered them ... Urr, OK, didn't get a chance to send so carrying on now I'm in the hotel.
If you can't remember what you installed. You could also open your existing flow, the NR log should show you everything that needs installing.
Different issue I think. Make sure you use --unsafe-perm when installing Node-RED. I was getting a similar issue with the serial library.
I guess that this happened because I installed stuff with whatever version of npm apt-get (Raspbian) gives, then later used the script from nodered.org documentation to update? If that's right I suggest the script is broken.
Anyway, It was in need of housekeeping. After various reboots, restarts and reinstallations I managed to get to the admin page so I have exported each flow.
I then deleted .node-red/ and reinstalled again.
I can import the flows one at a time and see the missing nodes. Presumably Import and Manage Palette will update the package.json file, otherwise I'll have to try and understand the format. Thanks for your example, mine was almost empty!
Unfortunately ui tabs etc have to be rebuilt.
I don't think that Dave's script is broken. It is pretty definitive actually. But I think some things have changed, possibly with the move to Node.JS v8, not sure. Certainly I suddenly started getting issues that I hadn't previously. Installing Node-RED with the unsafe-perm setting (which I think as long been recommended) seemed to fix it.
Yes, installing from the admin ui will set up the package.json file correctly.
Are these not kept by an export? I think they are.
If you were on Raspbian Jessie with the default Node-RED and manually installed npm using apt and didn't follow the instructions to update it (using sudo npm install -g npm) - then yes you would have been back on debians default of npm 1.4.2 - which would break things.
If you were on Stretch then the nodejs is at least version 8.x and the built in update-nodejs-and-nodered script would have installed the latest npm for you (after getting 1.4.2 it then upgrades it to latest). But again just running simple apt install npm would only give you npm 1.4.2 .
The bash script does try to undo any of this mess you may have got into - but I'm first to admit it may not be perfect as it's hard to know exactly how you got into that situation. If we can work out the steps we can try to add to it to try to catch these edge cases.
