Problem loading an external module

rozek · 24 November 2023 06:14

I just tried to install an external module and use it within a function node, following the docs:

within the function editor, I configured the external module (javascript-interface-library -> JIL)
verified that functionExternalModules was set to true in my settings.js file and
deployed without errors

The logs said

24 Nov 06:51:22 - [info] Modul wird installiert: javascript-interface-library, Version: latest
24 Nov 06:51:24 - [info] Installiertes Modul: javascript-interface-library
24 Nov 06:51:24 - [info] Flows werden gestartet
24 Nov 06:51:24 - [info] Flows sind gestartet

But, surprisingly, I could not find the folder ~/.node-red/externalModules/ mentioned in the docs - are they wrong here?

In the function node, I tried

const { ValueIsNonEmptyString } = JIL
return msg;

but that crashed with the message

TypeError: Cannot destructure property 'ValueIsNonEmptyString' of 'JIL' as it is undefined.

Does anybody know what is going wrong here?

Thanks in advance for any help!

Amendment:

I am using the latest Node-RED 3.1 (installed approx. 3 days ago)
node --version says v21.2.0
externalModules in my settings.js is empty (i.e., set to defaults)

TotallyInformation · 24 November 2023 09:01

As the error indicates, you can't deconstruct JIL because it has already been required. You have to do JIL.ValueIsNonEmptyString( ... ). Or possibly, you could do const ValueIsNonEmptyString = JIL.ValueIsNonEmptyString if you want to use the marginally shorter name.

rozek · 24 November 2023 13:11

Well, since const { ValueIsNonEmptyString } = JIL is just another way of writing const ValueIsNonEmptyString = JIL.ValueIsNonEmptyString this response should definitely be wrong.

Nevertheless, I tried it and got the error TypeError: Cannot read properties of undefined (reading 'ValueIsNonEmptyString') instead

Colin · 24 November 2023 13:46

Can you select and export the function node and paste it here please, so we can try it.

rozek · 24 November 2023 13:52

Sure, here it is

[
    {
        "id": "5248a8427f232978",
        "type": "inject",
        "z": "a38a9db5ce759685",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 80,
        "y": 560,
        "wires": [
            [
                "ee86e89fa5ccab7b"
            ]
        ]
    },
    {
        "id": "ee86e89fa5ccab7b",
        "type": "function",
        "z": "a38a9db5ce759685",
        "name": "function 1",
        "func": "const { ValueIsNonEmptyString } = JIL\n\nreturn msg;",
        "outputs": 1,
        "timeout": 0,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [
            {
                "var": "JIL",
                "module": "javascript-interface-library"
            }
        ],
        "x": 240,
        "y": 560,
        "wires": [
            [
                "2acc049673067f0e"
            ]
        ]
    },
    {
        "id": "2acc049673067f0e",
        "type": "debug",
        "z": "a38a9db5ce759685",
        "name": "debug 3",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 400,
        "y": 560,
        "wires": []
    }
]

Colin · 24 November 2023 14:15

I see the same problem, I don't know why.

rozek · 24 November 2023 14:17

thanks for testing...

rozek · 24 November 2023 20:15

does anybody have any idea? I'm surprised that such a basic feature fails and nobody ever ran into the same situation...

And, no, I do not want any kind of workaround as I plan to publish the affected flow and do not want my users to start complicated installations...

TotallyInformation · 24 November 2023 20:35

It really isn't surprising that, out of 100's of thousands of possible npm libraries, some don't actually work.

Your first task should be to try a short test node.js file to see if the module actually works in Node.js at all. And in fact it doesn't.

Error [ERR_REQUIRE_ESM]: require() of ES Module D:\src\nr\data\node_modules\javascript-interface-library\dist\javascript-interface-library.js from D:\src\nr\data\settings.js not supported.
javascript-interface-library.js is treated as an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which declares all .js files in that package scope as ES modules.
Instead rename javascript-interface-library.js to end in .cjs, change the requiring code to use dynamic import() which is available in all CommonJS modules, or change "type": "module" to "type": "commonjs" in D:\src\nr\data\node_modules\javascript-interface-library\package.json to treat all .js files as CommonJS (using .mjs for all ES modules instead).

Maybe it could be finagled into working but life is too short I'm afraid.

rozek · 24 November 2023 20:39

I just tried to create ~/.node-red/externalModules myself and got the following message

Node-RED 1.3 Verzeichnis externer Module erkannt:
    /Users/andreas/.node-red/externalModules
Dieses Verzeichnis wird nicht mehr verwendet. Die externen Module werden
in Ihrem Node-RED-Benutzerverzeichnis neu installiert:
   /Users/andreas/.node-red
Löschen Sie das alte externalModules-Verzeichnis, um diese Meldung abzustellen.

which basically says that the folder mentioned in the docs is not used any longer

Are there more mistakes in the docs in that area?

rozek · 24 November 2023 20:42

strange...

I can npm install javascript-interface-library without any problems.

I do not add it to the settings.js file but to the list of modules needed for a function node (functionExternalModules is true in my settings.js)

Additionally, I don't see any error messages in my server log

Amendment: I just tried to analyze the error message given by TotallyInformation to see if there is a solution:

"dynamic import" does not seem to be a solution as that is, by definition, asynchronous and could cause race conditions...
changing "package.json" is also not an option as that could cause side-effects for other users of that library

ralphwetzel · 24 November 2023 21:15

I could have an explanation for what you see:

github.com

node-red/node-red/blob/a55554193bac2ee4997948c6a15dd752844cace4/packages/node_modules/@node-red/nodes/core/function/10-function.js#L316-L323


      
          if (spec && (spec !== "")) {
              moduleLoadPromises.push(RED.import(module.module).then(lib => {
                  sandbox[vname] = lib.default;
              }).catch(err => {
                  node.error(RED._("function.error.moduleLoadError",{module:module.spec, error:err.toString()}))
                  throw err;
              }));
          }

After the function node imported an ExternalModule as lib, it references (line 318) lib.default to the variable name defined on the property editor.

lib.default yet is only present if the module has defined an export default or export { ..., ... as default, ...}.

javascript-interface-library doesn't define an export default, but only export { ... }.

Is this a bug of Node-RED? I'm not sure...

rozek · 24 November 2023 21:17

THAT's interesting! So Node-RED requires importable libraries to define a default export!? I never knew that - and it also does not seem to be documented.

Thanks a lot for this detail!

~~Unfortunately, this means a LOT of ugly work for me...~~

Ok, so my current plan is

create another module
import * as JIL from 'javascript-interface-library'
export default JIL

that should work, shouldn't it?

rozek · 24 November 2023 21:56

will it ever work?

Now I get the following server log message:

24 Nov 22:54:13 - [error] [function:function 1] Fehler beim Laden des Moduls : TypeError: Cannot add property noConflict, object is not extensible
24 Nov 22:54:13 - [error] [function:function 1] Function node failed to load external modules

I do not know who wants to add a property noConflict to any object - it's definitely neither me nor the library...

But, according to ~/.node-red/node_modules, both the auxiliary node-red-javascript-interface-library and its dependency, the actual javascript-interface-library were successfully installed...

rozek · 24 November 2023 22:08

By the way: dynamic imports do not work in Node-RED

The code

(async () => {
  node.warn('loading JIL')
  const JIL = await import('javascript-interface-library')
  node.warn('JIL = ' + JIL)
})()

crashes the server with the following log output:

24 Nov 23:05:56 - [error] TypeError [ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING]: A dynamic import callback was not specified.
    at importModuleDynamicallyCallback (node:internal/modules/esm/utils:192:9)
    at Function node:ee86e89fa5ccab7b [function 1]:4:15
    at Function node:ee86e89fa5ccab7b [function 1]:6:3
    at Function node:ee86e89fa5ccab7b [function 1]:8:3
    at Script.runInContext (node:vm:133:12)
    at processMessage (/Users/andreas/.nvm/versions/node/v21.2.0/lib/node_modules/node-red/node_modules/@node-red/nodes/core/function/10-function.js:419:33)
    at FunctionNode._inputCallback (/Users/andreas/.nvm/versions/node/v21.2.0/lib/node_modules/node-red/node_modules/@node-red/nodes/core/function/10-function.js:342:17)
    at /Users/andreas/.nvm/versions/node/v21.2.0/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:210:26
    at Object.trigger (/Users/andreas/.nvm/versions/node/v21.2.0/lib/node_modules/node-red/node_modules/@node-red/util/lib/hooks.js:166:13)
    at Node._emitInput (/Users/andreas/.nvm/versions/node/v21.2.0/lib/node_modules/node-red/node_modules/@node-red/runtime/lib/nodes/Node.js:202:11)

(and importing node-red-javascript-interface-library) doesn't work either

(giving up for today - I really did not expect that a simple module import could cause such a complete full-stop of my development...)

ralphwetzel · 24 November 2023 22:10

You could try - but I'm not sure:

The import() call to your module returns an empty object - which is significantly different to many other modules (randomly picked) I tested with. That's the true cause for this issue. Reason unclear...

Additionally, we should discuss (ping @Steve-Mcl) to change line 318 to sth like

sandbox[vname] = lib.default ? lib.default : lib;

This yet needs some investigation for side effects...

rozek · 24 November 2023 22:12

no, I simply do not provide a default export from the javascript-interface-library - that's why I hacked the ugly node-red-javascript-interface-library.

But - as written - now a different error occurs after the successful module installation.

ralphwetzel · 24 November 2023 22:12

rozek:

The code

(async () => {
  node.warn('loading JIL')
  const JIL = await import('javascript-interface-library')
  node.warn('JIL = ' + JIL)
})()

crashes the server with the following log output:

Where did you try to run this? In a function node? That's never going to work...

rozek · 24 November 2023 22:13

yes, in the function node - why shouldn't that work? it's standard JavaScript

Should I provide a complete URL?

Amendment: no, a complete URL will crash as well...

ralphwetzel · 24 November 2023 22:15

Might be. But the function node code runs in a sandbox.
If that was possible, why should there be the action with functionExternalModules ?

Topic		Replies	Views
Load external modules in the function node General function-node	13	246	30 December 2023
externalModules General function-node	7	221	9 February 2023
Calling an NPM module from a function within a function General	5	572	21 September 2021
functionGlobalContext on Raspberry PI General	20	2605	29 June 2021
Module Import to settings.js Error General function-node	4	420	10 May 2022

Problem loading an external module

Related Topics