Admin Api: install node from local file, dependencies?

I want to install the contrib-node-red-grpc node to a device that has no internet access. Let's call this device the target device. The target device is connected to another device via local network. Let's call that device "the deployer". Both devices are running linux; the deployer has a somewhat modified Debian.
Presently, i have ssh access to the target for developing- and debug purposes but ssh to the target will be unavailable later when i want to put the solution for this task into use.

The steps i took:

-on the deployer i used npm pack to download contrib-node-red-grpc and the dependencies lodash.get, grpc-grpc-js, grpc-proto-loader. This resulted in four .tar.gz files.
-i copied the tar archives to the target device via scp into /var/lib/node-red/extra_nodes on the target.
-on the target, i ran tar xvf to unzip grpc-grpc-js-1.2.6.tgz. This resulted in a folder /var/lib/node-red/extra_nodes/package. That folder contains the expected files.
-i used curl on the deployer to POST to /red/nodes the JSON {"module": "/var/lib/node-red/extra_nodes/package"}
This, however, does not work because the package.json file of grpc-grpc-js has no node-red key - it is a node-node but not a node-red contribution node.
On the other hand, when i unpack node-red-contrib-grpc (my final goal :wink: ), node-red on the target tries to install it but fails because it cannot fetch the dependencies - the target device has no internet connection.

My question: how can i install node-red-contrib-grpc and its dependencies on the target only using the node red admin api and scp?

best regards and thanks for any hints in advance,
Jochen Menzel

One way is to repack the contrib-node-red-grpc file locally... on a dev machine npm install that package in a separate directory... go into that directory and edit the package.json file to add a bundleddependencies section - see https://docs.npmjs.com/cli/v6/configuring-npm/package-json#bundleddependencies - which will be an array of just the other package names... then run npm pack in that directory... that will create you a .tgz file with all the necessary dependencies in. You can now scp that to your target and npm i that file in the .node-red-directory.

1 Like

Hi, dceejay!

Thanks for your quick reply and the promising suggestion!
Today i spent some time trying to make it work, but unfortunately, it did not behave as expected:
I used npm install --prefix node-red-contrib-grpc to download and install node-red-contrib-grpc and all its dependencies. So far, so well.
Then i edited /node_modules/node-red-contrib-grpc/package.json and added the bundledDependencies section. However, this had no effect:
when i ran npm pack in /node_modules/node-red-contrib-grpc, it created a tar.gz that had the same size and content as when i run npm pack node-red-contrib-grpc (i.e. tell npm to download and tar just node-red-contrib-grpc, with no dependencies in it.)
This is quite unexpected behavior. What am i doing wrong?

In relation to my previous post, seems i was a little tired when i wrote it.
To give a more precise rendition of what i tried - i added a newly-made folder in the call to npm:
npm install --prefix (name of folder here) node-red-contrib-grpc and used said folder in subsequent comands.

Finally, i found the solution by myself.
openbase.com gave the crucial hint in point 3:

If you wish to include dependencies and use npm-pack you must do the following:

1. create bundledDependencies section in package.json
2. remember to update bundledDependencies before executing npm pack
3. remember to execute npm install before executing npm pack

Quite to my regret, the official npm documentation on bundledDependencies does not mention the neccessity to re-run npm install inside the folder of the node you want to bundle-package. This piece of information cost a noob like me ~14 hours of frustrated try-and-error. :frowning: So, to spare others the same confusion about the behavior of npm pack, i suggest that a suitable hint be added to the npm documentation. :wink:

Finally, to make things easier, i paste below a script i dug out that automates the entire process. The additional npm install step is yet missing in the script, but well,..

/*
 * This script will download a package (and all of its dependencies) from the
 * online NPM registry, then create a gzip'd tarball containing that package
 * and all of its dependencies. This archive can then be copied to a machine
 * without internet access and installed using npm.
 *
 * The idea is pretty simple:
 *  - npm install [package]
 *  - rewrite [package]/package.json to copy dependencies to bundleDependencies
 *  - npm pack [package]
 *
 *  It is necessary to do this (intead of using pac) because when npm installs
 *  a module, it will actually strip out the node_modules folder from the
 *  tarball unless bundleDependencies is set.
 *
 *  Author: Jack Gill (https://github.com/jackgill)
 *  Date: 11/27/2013
 *  License: MIT License (see end of file)
 */

// Original gist: https://gist.github.com/jackgill/7687308
// Slightly modified by @phillipj too enable specifying @version of packages
// updated by Dipl.-Ing. J. Menzel to fix broken regular expression and packet path resolution in 2021-2

'use strict';

var fs = require('fs');
var cp = require('child_process');
var path = require('path');

if (process.argv.length != 3) {
  console.log("Usage: %s <package name>[@version]", process.argv[1]);
  console.log("\nExamples:");
  console.log("\tnode pack.js react");
  console.log("\tnode pack.js angular@1.5.5");
  console.log("\tnode pack.js @angular/core@2.0.1");
  process.exit(0);
}

var fullPackageStr = process.argv[2]; // might include @version suffix
console.log("started with argv2: " + fullPackageStr);
// extracts package or @scope/package name, removing the possible @version suffix
var packageName = fullPackageStr.match(/^(@?[\w]+\/[\w-]+)|^([\w-]+)/)[0];
console.log("use package name: " + packageName);

// Copy dependencies to bundleDependencies
function rewritePackageJSON(fileName) {
  var contents = fs.readFileSync(fileName);
  var json = JSON.parse(contents);
  if (json.dependencies) {
    json.bundleDependencies = Object.keys(json.dependencies);
    fs.writeFileSync(fileName, JSON.stringify(json, null, 2));
  }
}

// Install the package from the online registry // npm install
var installProcess = cp.exec(' : ' + fullPackageStr, function(err, stdout) {
  if (err) {
    console.log("\n✘ Error executing npm install:", err);
    process.exit(0);
  }
  else {
	  console.log("\n✔ installed package: ", fullPackageStr);
  }
   console.log(stdout)

  // Set bundleDependencies for the package
  rewritePackageJSON(path.join('node_modules', packageName, 'package.json'));

process.exit(0);

  console.log("now npm pack: " + packageName);
  // Create a new .tgz file which bundles all dependencies
  var packProcess = cp.exec('npm pack ' + path.resolve(path.join('node_modules', packageName)), function(err) {
    if (err) {
      console.log("\n✘ Error executing npm pack:", err);
    }
    else {
      console.log("\n✔ Bundled package", fullPackageStr);
    }
  });

  pipeToProcessOutputs(packProcess);
});

pipeToProcessOutputs(installProcess);

function pipeToProcessOutputs(aChildProcess) {
  aChildProcess.stderr.pipe(process.stderr);
  aChildProcess.stdout.pipe(process.stdout);
}

/*

   The MIT License (MIT)

Copyright (c) 2013 Jack Gill

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

*/