Specify node_modules folder in settings.js

TotallyInformation · 26 April 2021 17:00

With a hard link, if you delete the original, the link becomes the file. A soft link will not open anything if you delete the original but if you put another file back with the original name, the soft link will open it fine whereas the hard link is still the original file (just in a different place in the FS). But in both cases, as long as the original still exists, you can amend it via either of the links or via the original and all will be in step because the links point to the original. Another thing you will note is that the file size of a soft-link is 0b but the file size of a hard link is the same as the original (because it IS the original).

I generally find it much safer and better to use soft-links not hard-links as I think I said further up. Also, hard links do not work across filing system boundaries (e.g. different drives in Windows).

But either way, if you are creating a link from a master install to a 2nd instance, the link HAS to show the original contents - unless, in the hard-link case, the original gets deleted and then replaced.

Bottom line, use a soft-link and I think you will find that it works.

Lupin_III · 26 April 2021 17:47

Have you made some test?
Nothing change to me (Windows 10, but event with debian buster is more or like the same). With all the test I have performed, even after your clarification I always get the same results:

my_folder/package.json is necessary but seems not be used (even if it doesn't contain any nodes, the nodes installed via the main instance are still present, if the link to the .node-red\node_modules dir exists, of course)
It's not possible to install/update a node by the second instance, I get a loop message error, the .node-red\node_modules link became broken and nothing works anymore. I have to recreate the link and restart it. (my_folder/node_modules became a simple folder inside my_folder)

This is what happen to me, at least. I'm not that expert to explain why.
A part for the command for creating the link to .node-red/package.json (the correct one is simply "mklink" as suggested) what I post here Specify node_modules folder in settings.js - #12 by Lupin_III is the only way I found to perform it, with its limitation, unfortunately!

Maybe a big help could be having a dedicated option in the settings.js to specify, if needed, the dir to use for "node_modules" dir and "package.json" file.

TotallyInformation · 26 April 2021 20:17

Yes. Not a full one with Node-RED but certainly tests with files and links. I have also used file and folder links with Node-RED in the past.

But it is expected and so there is no guarantee that you will get consistent results in the future.

That doesn't sound right. How can you get a loop simply from using a linked file/folder?

Have you done what I've said and replaced the hard link with a soft link? Until you have, we can't progress any further.

No, I don't think that is a good idea at all I'm afraid. Node-RED expects to have a userDir folder and that folder contains the package.json, the settings.js, the node_modules as well as the flows and other stuff.

I suggest that you build a clean environment with your master install, get some nodes installed and create a simple flow. Then create a SOFT LINK of the userDir folder to another folder and a script to start a second instance of Node-RED from that linked folder. Then add a new node to the master and restart the 2nd. Then report back.

Colin · 26 April 2021 20:56

I am not sure whether you realise, but that isn't actually what @Lupin_III is trying to do. He is trying to link just node_modules and package.json so that the additional instances have different settings and flows but use the same set of installed nodes.

TotallyInformation · 26 April 2021 21:04

Yes, I realise that Colin, I'm just trying to walk him through something simpler so that we can get to the bottom of his issue because something isn't adding up which indicates that we are missing some piece of the puzzle.

Colin · 26 April 2021 21:06

Good plan.

TotallyInformation · 26 April 2021 21:22

I have now done a full test. Using my existing dev environment

C:\src\nr2>npm run startc

> node-red-master@1.1.0 startc C:\src\nr2
> node node_modules/node-red/red.js --userDir ./data2

PID:  27676
V8 Total Heap Size: 4128.84 MB
/**********************
 * THIS IS USING DATA2
 **********************/
26 Apr 22:16:39 - [info]

Welcome to Node-RED
===================

26 Apr 22:16:39 - [info] Node-RED version: v1.3.2
26 Apr 22:16:39 - [info] Node.js  version: v14.15.4
26 Apr 22:16:39 - [info] Windows_NT 10.0.19042 x64 LE
26 Apr 22:16:40 - [info] Loading palette nodes
26 Apr 22:16:43 - [info] +-----------------------------------------------------
26 Apr 22:16:43 - [info] | uibuilder initialised:
26 Apr 22:16:43 - [info] |   Using Node-RED's webserver
26 Apr 22:16:43 - [info] |   root folder: C:\src\nr2\data2\uibuilder
26 Apr 22:16:43 - [info] |   version . .: 3.3.0
26 Apr 22:16:43 - [info] |   packages . : vue,bootstrap,bootstrap-vue,jquery,moonjs,socket.io
26 Apr 22:16:43 - [info] +-----------------------------------------------------
26 Apr 22:16:43 - [info] Dashboard version 2.28.2 started at /nr/ui
26 Apr 22:16:43 - [info] Settings file  : C:\src\nr2\data2\settings.js
26 Apr 22:16:43 - [info] HTTP Static    : C:\src\nr2\public
26 Apr 22:16:43 - [info] Context store  : 'default' [module=memory]
26 Apr 22:16:43 - [info] Context store  : 'file' [module=localfilesystem]
26 Apr 22:16:43 - [info] User directory : C:\src\nr2\data2
26 Apr 22:16:43 - [warn] Projects disabled : editorTheme.projects.enabled=false
26 Apr 22:16:43 - [info] Flows file     : C:\src\nr2\data2\flows_dev.json
26 Apr 22:16:43 - [info] Creating new flow file
26 Apr 22:16:43 - [info] Starting flows
26 Apr 22:16:43 - [info] Started flows
26 Apr 22:16:43 - [info] Server now running at http://127.0.0.1:1881/red/

This shows the 2nd instance starting up, the primary instance is also running.

I did the following:

Create a new userDir folder
Copied the setttings.js file from the master folder to the 2nd and changed the port and added a note
Created SOFT LINKS from the master package.json file and node_modules folder into the 2nd folder.
Started both instances to make sure they were working.
Added a new node to the master instance
Restarted the 2nd instance
Checked that the new node was picked up and available. It was.

So everything works exactly as we described.

Lupin_III · 27 April 2021 07:37

Right TotallyInformation, it's ok till this step, behavior as expected. Now try this:

Open only the second instance;
Try to install a new node

To me:
this is my_folder before running the second instance (and yes I made soft links)

This is the error I get when I try to install a new node:

And this is what happened to the dir "my_folder" (node-red instance is still open)

As you can see the link to .node-red/node_modules doesn't exist anymore.
Regarding the error I get, you can find more details in this post Unable to install/update nodes

Lupin_III · 27 April 2021 08:32

Yes, the correct command to create the links and that I've used are:

"mklink /J" --> for linking .node-red\node_modules
"mklink" --> for linking .node-red\package.json (even if you don't need it)

Colin · 27 April 2021 10:51

According to the docs I have found using /J makes a hard link to a folder. To make a soft link to a folder it should be /D. However that may be completely wrong as I don't use Windows so can't test it. https://www.howtogeek.com/howto/16226/complete-guide-to-symbolic-links-symlinks-on-windows-or-linux/

TotallyInformation · 27 April 2021 10:59

mklink /?
Creates a symbolic link.

MKLINK [[/D] | [/H] | [/J]] Link Target

        /D      Creates a directory symbolic link.  Default is a file
                symbolic link.
        /H      Creates a hard link instead of a symbolic link.
        /J      Creates a Directory Junction.
        Link    Specifies the new symbolic link name.
        Target  Specifies the path (relative or absolute) that the new link
                refers to.

mklink /j creates a JUNCTION. That is a Windows-only special type of hard-link that should generally be avoided for a very important reason. If you delete the link, it will delete the ORIGINAL as well!

To create a file link, you don't need a parameter. Use /D to create a soft link for a folder.

If you want to avoid the confusion over command line stuff, I recommend installing Link Shell Extension (schinagl.priv.at). Then you can do it from Windows File Explorer.

Colin · 27 April 2021 11:24

Though I would have expected a hard link to work perfectly well for node_modules, since it isn't recreating the folder each time. Best to stick to soft links though.

Lupin_III · 27 April 2021 14:32

Thanks for clarification even if I understood it different.
This is what I found but probably it's my fault since, as I said, for me it's the first time I came across with this staff.

Deleting my_folder\node_modules doesn't delete .node-red\node_modules (but maybe the junction still exists and I should use a dedicated command to delete it).

But it's not that important. To remain on the topic even using mklink /D nothing change. I created the soft link to .node-red\node_modules with mklink /D as suggested but I still get the same error!!!

If in your test you don't get the same error, at this point it's probably something to do with my OS and what I've installed. I don't know what to say and what to try anymore.

Colin · 27 April 2021 14:54

Of course not, why should it? if it is a symbolic link then it will delete the link, if it is a hard link then it will delete the local reference to the folder but not the actual folder. The folder will only be deleted when all references to it have been deleted.
Set up the links then show us a directory listing showing them as links. Then run (still in the .node-red directory)
npm install google-translate-tts
and if it doesn't work show us what you get.

Lupin_III · 27 April 2021 16:02

Well, we were talking about the "junction" in my_folder\node_modules to .node-red\node_modules. Deleting the junction even the target should be delete, as said.

It's exactly what I did in the last post:
Please don't misunderstand me but since I'm getting always the same error, are you performing this test too, or this is what you think is the way it's supposed to behavior?
This is .node-red folder

Now, I create the links in my_folder from command line running it as administrator:

mklink /D "C:\my_folder\node_modules" C:\Users\Io\.node-red\node_modules
mklink "C:\my_folder\package.json" C:\Users\Io\.node-red\package.json

this is my_folder after creating the links

Running

node-red -u "C:\my_folder" -p 1900

and trying to install a node from manage palette I got the same error

and the link to C:\Users\Io.node-red\node_modules breaks

For your last point

Again the problem is not the main node-red. From this instance I can install/update everything without problem. I can do it from manage palette, as well as by command line from within .node-red directory.
The problem is from the other instances

Colin · 27 April 2021 16:18

No, a Hard Link (which apparently is what a Junction is) makes a second reference to the folder. So now there are two references to it, the on in .node-red and the one in the second folder. Deleting one of those deletes one of the references but because the other one still exists the folder is not actually deleted. It will not be deleted until all references to it are deleted.

No, I don't use windows. I am trying to help you to analyse what is happening.

I want to see the command line install output when run in the second folder please (after setting up the link again obviously). Sometimes it is easier to see what is happening when run from the command line. Also it is possible that it works from the command line but not from node-red (though I would be surprised if it is that).

TotallyInformation · 27 April 2021 20:08

That used to happen on Windows for junctions, maybe it doesn't any more. I certainly know that I've been caught out by it in the past. Could well have been Windows XP/2000 though. The disadvantage of having been around a long time

What? I thought you wanted a master from which all other instances inherited? Why would you then let someone install into a child instance?

Well whatever, I can confirm that doesn't work, it screws up the node_modules folder somehow. No idea how or why.

Now we know what you are actually trying, I recommend that you take Node-RED out of the equation. Try with two starting folders. In the first, do npm init -y. Then install any random npm package. Next, create the links in the 2nd folder and try to install an npm package in there. See if you get the same problem. If you do, raise it as an issue against npm.

UPDATE: I can confirm that trying the above doesn't do the same as with Node-RED. However, it does not produce a consistent result.

I can also say categorically, if you do this, you MUST delete the package.lock.json files in both folders.

One other thing to try. With Node-RED, try stopping all instances of Node-RED before doing the install in the child. Also delete the package.lock.json files. See if that makes a difference.

UPDATE 2: I can also definitively confirm that doing this on Linux also does not work. However, you do get some more errors that give a clue as to what is going on. npm is using a lot of symbolic links of its own and it gets in a horrible mess if you try to use a node_modules folder that is, itself a linked folder.

So the final outcome. DO NOT TRY TO INSTALL A MODULE TO A LINKED node_modules FOLDER.

However, installing to the actual folder and using it as a linked folder elsewhere is perfectly fine.

Lupin_III · 28 April 2021 09:39

Yes, I have more or less come up to the same conclusion, and yes, as you can see from the imagines, the package.lock.json exists only in the .node-red folder.

It would have been useful to have the possibility of installing/upgrading nodes from whatever instance and have them available within all the others. But actually it doesn't seem to be possibile. (In my opinion it will be nice to have an option in settings.js to specify the path to use for it)

So, in conclusion at moment the best way to achieve something similar is create soft links to .node-red\node-modules and .node-red\package.json. At this point you install/update nodes only from the main instance in node-red, and they will be available in all the other instances (which have been linked to the main one and after their restarted, of course)

Thanks to all for the supports

Lupin_III · 28 April 2021 09:56

This is the result. As for the other test, the link to .node-red\node-modules is broken. There are some WARM and ERR, but from the main instance I'm able to install it without any problem

C:\Users\Io>cd C:\my_folder

C:\my_folder>npm install google-translate-tts
npm WARN deprecated debug@4.1.1: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm ERR! code ENOENT
npm ERR! syscall rename
npm ERR! path C:\my_folder\node_modules\coord-parser
npm ERR! dest C:\my_folder\node_modules\node-red-contrib-cron-plus\node_modules\coord-parser
npm ERR! errno -4058
npm ERR! enoent ENOENT: no such file or directory, rename 'C:\my_folder\node_modules\coord-parser' -> 'C:\my_folder\node_modules\node-red-contrib-cron-plus\node_modules\coord-parser'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\Io\AppData\Roaming\npm-cache\_logs\2021-04-28T09_50_38_739Z-debug.log

C:\my_folder>

This is the log
[2021-04-28T09_50_38_739Z-debug.txt|attachment]
(upload://cYFRvXu8S1V4PMxxZaynuVjtPDy.txt) (485.1 KB)

Lupin_III · 28 April 2021 09:57

2021-04-28T09_50_38_739Z-debug.txt (485.1 KB)

Topic		Replies	Views
Node-RED User Directory General	3	4103	30 June 2020
Windows - Settings.js General	3	685	26 April 2020
Settings.js overwritten General	7	457	18 November 2021
Shared system folders in multi user instances? General	34	1957	16 April 2022
Locations of node_modules - global vs. local General	2	218	25 March 2023

Specify node_modules folder in settings.js

Related topics