Achieve a more stable flow

General
1

[info] Node-RED version: v3.1.5
[info] Node.js version: v16.20.2

I have a flow in Node-RED that utilizes the exec node to execute a command. The command is "node /opt/node-red/script/googlehtml.js". The flow tends to get stuck on this node at random times. Sometimes it can complete 4 runs before getting stuck, and other times it can complete 50.

What can I do to achieve a more stable flow?

[
    {
        "id": "16eb6e9cb91ab636",
        "type": "exec",
        "z": "2eae83f77d8ec666",
        "command": "node /opt/node-red/script/googlehtml.js",
        "addpay": false,
        "append": "",
        "useSpawn": "false",
        "timer": "",
        "winHide": false,
        "oldrc": false,
        "name": "Sparar ner Google HTML till /opt/node-red/temp/out.html",
        "x": 590,
        "y": 1380,
        "wires": [
            [
                "3c196409affb01d3",
                "189c500d19f16f40"
            ],
            [
                "4764325e4be52e15"
            ],
            [
                "d3da1bf50e397d43"
            ]
        ]
    }
]
2

What do you mean by getting stuck?

Have you checked all the outputs of the exec node in case there is information there?

3

Good point. The node prior to the exec node operates normally at all times. When the flow halts, there's no output whatsoever from the exec node. Another detail is that I have to restart Node-RED to prevent it from halting at the same point again.

4

Add a debug node showing what is going into the exec node and three debug nodes showing what is coming out. When it stops post a screenshot showing the exec node and the debug output.

When it stops does the editor keep working ok?

Edit: configure the debug nodes to Output complete message.

5

Curious. Why are you spawning node via an exec node instead of simply running this script in a function node?

What does googlehtml.js do? Cant you simply implement whatever it does in regular nodes?

6

Debug nodes with the 'msg complete' object are already attached to all outputs of the exec node, as well as to the node preceding it (and everything else in the flow). When the flow halts, there is no output at all from the exec node, which is one of the issues that complicates the problem.

Editing nodes and other components works perfectly. The 30 nodes before in the flow also function excellently up to this exec node.

2024-02-17 23:19:48node: Skriver över kod till googlehtml.js
msg : Object
object
topic: ""
_msgid: "c7f2bbfdac72abba"
files: array[1]
firstSubdirectory: "Catharina_Ingelman-Sundberg_-_Boken_Om_Vikingarna-AUDiOBOOK-WEB-SE-2009-OLDSWE_iNT"
transformedSubdirectory: "Catharina Ingelman-Sundberg - Boken Om Vikingarna- www.storytel.com/se/"
fullPath: "/mnt/unionfs/Media/preLjudbok/downloads/Catharina_Ingelman-Sundberg_-_Boken_Om_Vikingarna-AUDiOBOOK-WEB-SE-2009-OLDSWE_iNT"
rc: object
code: 0
filteredFiles: array[3]
filename: "/opt/node-red/script/googlehtml.js"
payload: string
const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
const AdblockerPlugin = require('puppeteer-extra-plugin-adblocker');
const fs = require('fs');
const fetch = require('node-fetch');

// Använd plugin-programmen
puppeteer.use(StealthPlugin());
puppeteer.use(AdblockerPlugin({ blockTrackers: true }));

(async () => {
    try {
        let browserWSEndpoint = '';
        await fetch("http://172.19.0.100:9222/json/version")
            .then(response => response.json())
            .then(data => {
                browserWSEndpoint = data.webSocketDebuggerUrl;
            })
            .catch(error => console.error('Fel vid hämtning av browserWSEndpoint:', error));

        const browser = await puppeteer.connect({ browserWSEndpoint, ignoreHTTPSErrors: true });
        const page = await browser.newPage();

        await page.goto('https://www.google.com/search?q=Catharina%20Ingelman-Sundberg%20-%20Boken%20Om%20Vikingarna-%20www.s...
url: "https://www.google.com/search?q=Catharina%20Ingelman-Sundberg%20-%20Boken%20Om%20Vikingarna-%20www.storytel.com%2Fse%2F"
7

I have been experiencing issues running scripts that need to be dynamically altered with the function node. It has been a while since I created the flow, but I believe that was my reasoning.
The script, in brief, searches Google with a query string and saves the HTML output. Here is an example from /opt/node-red/script/googlehtml.js

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
const AdblockerPlugin = require('puppeteer-extra-plugin-adblocker');
const fs = require('fs');
const fetch = require('node-fetch');

// Använd plugin-programmen
puppeteer.use(StealthPlugin());
puppeteer.use(AdblockerPlugin({ blockTrackers: true }));

(async () => {
    try {
        let browserWSEndpoint = '';
        await fetch("http://172.19.0.100:9222/json/version")
            .then(response => response.json())
            .then(data => {
                browserWSEndpoint = data.webSocketDebuggerUrl;
            })
            .catch(error => console.error('Fel vid hämtning av browserWSEndpoint:', error));

        const browser = await puppeteer.connect({ browserWSEndpoint, ignoreHTTPSErrors: true });
        const page = await browser.newPage();

        await page.goto('https://www.google.com/search?q=Catharina%20Ingelman-Sundberg%20-%20Foerfoeljd-%20www.storytel.com%2Fse%2F', { waitUntil: 'networkidle2' });

        const htmlContent = await page.content();
        fs.writeFileSync('/opt/node-red/temp/out.html', htmlContent);

        // Flytta stängningen av alla sidor hit, innan browser.close()
        const pages = await browser.pages();
        await Promise.all(pages.map(page => page.close()));

        await browser.close();
    } catch (error) {
        console.error('Ett oväntat fel uppstod:', error);
    }
})();
8

Does your exec node have a value for timeout?

When the flow "gets stuck", is the external script still running or has it crashed?

9

The script does not run at all; the node does not execute. It's as if the message just gets lost between the preceding node and the exec node.

10

A simplified section of your flow:

image
image994×202 19.3 KB

Are you saying that the debug node "The node prior to the exec node" shows an output but none of "Exec stdout", "Exec stderr" or "Exec exit value" do?

Set a timeout - 10 seconds maybe - on the exec node. Does a timeout message show up?

11

Yes, thats exactly what I'm saying!

Here's a real example of what that part of the flow looks like:

image
image1447×172 69.4 KB

[
    {
        "id": "8a0ef51a775db706",
        "type": "function",
        "z": "2eae83f77d8ec666",
        "name": "Förbereder googlehtml.js",
        "func": "let url = msg.url;\n\nmsg.payload = `const puppeteer = require('puppeteer-extra');\nconst StealthPlugin = require('puppeteer-extra-plugin-stealth');\nconst AdblockerPlugin = require('puppeteer-extra-plugin-adblocker');\nconst fs = require('fs');\nconst fetch = require('node-fetch');\n\n// Använd plugin-programmen\npuppeteer.use(StealthPlugin());\npuppeteer.use(AdblockerPlugin({ blockTrackers: true }));\n\n(async () => {\n    try {\n        let browserWSEndpoint = '';\n        await fetch(\"http://172.19.0.100:9222/json/version\")\n            .then(response => response.json())\n            .then(data => {\n                browserWSEndpoint = data.webSocketDebuggerUrl;\n            })\n            .catch(error => console.error('Fel vid hämtning av browserWSEndpoint:', error));\n\n        const browser = await puppeteer.connect({ browserWSEndpoint, ignoreHTTPSErrors: true });\n        const page = await browser.newPage();\n\n        await page.goto('${url}', { waitUntil: 'networkidle2' });\n\n        const htmlContent = await page.content();\n        fs.writeFileSync('/opt/node-red/temp/out.html', htmlContent);\n\n        // Flytta stängningen av alla sidor hit, innan browser.close()\n        const pages = await browser.pages();\n        await Promise.all(pages.map(page => page.close()));\n\n        await browser.close();\n    } catch (error) {\n        console.error('Ett oväntat fel uppstod:', error);\n    }\n})();`;\n\nreturn msg;\n",
        "outputs": 1,
        "timeout": "",
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 490,
        "y": 1300,
        "wires": [
            [
                "357edbfe0ad91ee0",
                "ee647288aed249f0"
            ]
        ]
    },
    {
        "id": "357edbfe0ad91ee0",
        "type": "file",
        "z": "2eae83f77d8ec666",
        "name": "Skriver över kod till googlehtml.js",
        "filename": "/opt/node-red/script/googlehtml.js",
        "filenameType": "str",
        "appendNewline": false,
        "createDir": false,
        "overwriteFile": "true",
        "encoding": "none",
        "x": 520,
        "y": 1340,
        "wires": [
            [
                "6c6246a25dd941de",
                "16eb6e9cb91ab636"
            ]
        ]
    },
    {
        "id": "ee647288aed249f0",
        "type": "debug",
        "z": "2eae83f77d8ec666",
        "name": "Förbereder googlehtml.js",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 1030,
        "y": 1300,
        "wires": []
    },
    {
        "id": "6c6246a25dd941de",
        "type": "debug",
        "z": "2eae83f77d8ec666",
        "name": "Skriver över kod till googlehtml.js",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 1060,
        "y": 1340,
        "wires": []
    },
    {
        "id": "3c196409affb01d3",
        "type": "debug",
        "z": "2eae83f77d8ec666",
        "name": "OUTPUT - Sparar ner Google HTML till /mnt/local/Media/test/temp/out.html",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 1190,
        "y": 1360,
        "wires": []
    },
    {
        "id": "4764325e4be52e15",
        "type": "debug",
        "z": "2eae83f77d8ec666",
        "name": "ERROR - Sparar ner Google HTML till /mnt/local/Media/test/temp/out.html",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 1180,
        "y": 1380,
        "wires": []
    },
    {
        "id": "d3da1bf50e397d43",
        "type": "debug",
        "z": "2eae83f77d8ec666",
        "name": "EXIT - Sparar ner Google HTML till /mnt/local/Media/test/temp/out.html",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 1170,
        "y": 1400,
        "wires": []
    },
    {
        "id": "16eb6e9cb91ab636",
        "type": "exec",
        "z": "2eae83f77d8ec666",
        "command": "node /opt/node-red/script/googlehtml.js",
        "addpay": false,
        "append": "",
        "useSpawn": "false",
        "timer": "",
        "winHide": false,
        "oldrc": false,
        "name": "Sparar ner Google HTML till /opt/node-red/temp/out.html",
        "x": 590,
        "y": 1380,
        "wires": [
            [
                "3c196409affb01d3",
                "189c500d19f16f40"
            ],
            [
                "4764325e4be52e15"
            ],
            [
                "d3da1bf50e397d43"
            ]
        ]
    }
]
12

I don't reject the possibility that your message flows along the "wire" from "Skriver över kod till googlehtml.js" to it's debug node but somehow fails to flow to the exec node.
It does seem a very remote possiblity though.

Can you insert progress messages in your external script - "started", "blah", "blah", "exiting"?
If you send the messages to stderr they hopefully won't interfere with processing the normal output.
I don't do stand-alone javascript so I don't know if that is possible.

Change the exec node to output while the command is running and (I repeat) set a timeout.

image
image699×654 31.6 KB

13

Thank you. I've adjusted the timeout and spawn mode. I will also add stderr logging to the external script and let the flow run overnight to see if I can gather any useful information from it.

14

Also space out your flow a bit so that there is room under the exec node for the status. It is vital to know whether, when it stops, there is a pid shown there.

15

The flow has been running smoothly throughout the night and into the morning without any issues. I'm keeping my fingers crossed that this is the solution. Why this works better, however, is question I don't have the answer to.

16

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.