Periodic 502 Errors with Node-RED Version 2.2 - Need Help for Troubleshooting

hnuzhoulin · 12 December 2023 10:56

I am currently using Node-RED version 2.2, deployed locally and managed with systemd.
Recently, I've been experiencing periodic 502 errors, but these occurrences are not on a fixed schedule. When attempting to access port 1880 during these times, the browser shows a 502 error. However, there are no apparent anomalies or error messages in the logs, even though the logging level is set to 'trace'. Restarting the service seems to resolve the issue temporarily.

I suspect it might be related to some issues, as the number of IoT devices connected through Node-RED has significantly increased compared to two months ago. Any insights or suggestions on how to further investigate and resolve this issue would be greatly appreciated.

hnuzhoulin · 12 December 2023 11:03

another info: when the dashboard is 502,systemd shows the service works fine.

Colin · 12 December 2023 11:14

Please tell us a lot more about your system and network.
What hardware/OS is node red running on?
Is node red running in a Docker container, as a Home Assistant Plugin or a normal install?
When you get the error are you running the browser in the machine running node red, another machine in the local network, across the internet, via some sort of proxy or VPN etc? The possibilities are endless.

Please stop node red and start it again in a command window, then copy/paste the startup log. That will tell us a lot about your system. Copy/paste please, not screenshot.

Steve-Mcl · 12 December 2023 13:07

This suggests you are running behind a gateway or proxy (NGinx? HAProxy?)

Which service? node-red?

If yes, then this suggests you are crashing node-red OR have a loop causing such heavy load that it cannot respond.

If you have added devices then you have modified flows? Then perhaps there is a bug in your flows that causes a crash or a loop under certain circumstances.

NOTE: a loop can be where a message is passed around and around hogging the node process OR it could be (for example) and MQTT loop where you are subscribing to a topic that results in the same topic being changed thus creating an endles loop of SUB -> PUB -> SUB -> and so on.

hnuzhoulin · 13 December 2023 01:22

Thanks .no gateway or nginx,I just access node-red direct.

we do not change any flow,it works fine much months.

node-red sub and pub data from/to kafka,I found some alert about kafka which host nerwork has packet drops. it maybe the problem? I will check kafka.

hnuzhoulin · 13 December 2023 01:42

Node-RED is running on CentOS 7.9, hosted on Alibaba Cloud. It is running directly inside a Docker container, not as a plugin or a component of any other platform. The primary function of this Node-RED setup is to process MQTT data and then push it to Kafka or another MQTT, as well as subscribe to data from Kafka, process it, and then send it to other Kafka or MQTT.

I have written a script to access port 1880, and if the response is not 200, it triggers an alarm. When the alarm goes off, I try to access port 1880 from another node using Chrome, but the page does not load. The browser occasionally shows a 502 error and sometimes just shows 'empty'. The logs following a systemd restart of Node-RED are as follows:

Dec 12 23:44:50 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:44:50 - [info] [mqtt-broker:tenant_middleware] Connected to broker: mqtt://172.18.76.173:1883
Dec 12 23:44:50 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:44:50 - [error] [kafka-producer:measure-xinhao] NestedError: refreshBrokerMetadata failed
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z systemd: Stopping Node-RED graphical event wiring tool...
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [info] Stopping flows
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [debug] red/nodes/flows.stop : stopping flow : 9315ae661c6a0622
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] [flow:9315ae661c6a0622] stop flow
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node mqtt in:89b1c97b20a38685
.....
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node mqtt in:c1e7c1b0779df06a
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node kafka-producer:aca344e5d682a81f
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node debug:5d2b32d115fc3a84
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [debug] red/nodes/flows.stop : stopping flow : c63aebfc83c8eb29
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] [flow:c63aebfc83c8eb29] stop flow
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node mqtt in:5b064c2e6730a360
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node debug:505aaaa162794450
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node json:f1877752c20fa730
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node kafka-consumer:a404befcba440bf0
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node kafka-producer:d3ea6068c7ca51c3
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node debug:0fd920bd5dcae5e3
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [debug] red/nodes/flows.stop : stopping flow : b3232cb9a4f7d15f
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] [flow:b3232cb9a4f7d15f] stop flow
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node mqtt in:04efd336fcbeb1c9
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node kafka-producer:c93fe2904aa073a7
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node mqtt out:776e936dce462cdb
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [debug] red/nodes/flows.stop : stopping flow : 6cf8186019735673
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] [flow:6cf8186019735673] stop flow
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node kafka-consumer:8e264839903c8efe
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node mqtt out:1586fdc77bc90ef8
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node debug:d431615c1ad5f52b
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [debug] red/nodes/flows.stop : stopping flow : tenant
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] [flow:tenant] stop flow
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node mqtt in:966f4e28c1a244e9a15aaddea66d6a34
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node kafka-producer:4a0cd5ca67ce418e
...
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node kafka-producer:eae5dbd79b1444fa889edac27d619b28
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node mqtt in:fd696c6f4c8a42c495e9e77c0e86d1ee
......
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [debug] red/nodes/flows.stop : stopping flow : 7d62d43ee85cba6d
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] [flow:7d62d43ee85cba6d] stop flow
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node mqtt in:16fcd91e60e013b6
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node kafka-producer:b45115178da4cb04
....
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node mqtt in:57e84e3abdc7662a
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node mqtt out:9287d999014f9d4f
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [debug] red/nodes/flows.stop : stopping flow : f050a24709b413b4
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] [flow:f050a24709b413b4] stop flow
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node mqtt in:9355180943a26ff0
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node kafka-producer:47addee7183c7c85
...
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node debug:9ba510691f10e051
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [debug] red/nodes/flows.stop : stopping flow : global
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] [flow:global] stop flow
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node mqtt-broker:b0dc117744115cae
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopping node kafka-broker:b948f7e9a16470c9
....
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopped node kafka-producer:979581302b9e411cafd31963ab2c5891 (10ms)
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopped node kafka-producer:c505263c7f501d47 (7ms)
......
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [trace] Stopped node mqtt-broker:5cef1ca393a365be (4ms)
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:47:59 - [info] Stopped flows
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z systemd: Stopped Node-RED graphical event wiring tool.
Dec 12 23:47:59 iZwz995gddyo8yur1ec5x3Z systemd: Started Node-RED graphical event wiring tool.
Dec 12 23:48:00 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:00 - [info]
Dec 12 23:48:00 iZwz995gddyo8yur1ec5x3Z Node-RED: Welcome to Node-RED
Dec 12 23:48:00 iZwz995gddyo8yur1ec5x3Z Node-RED: ===================
Dec 12 23:48:00 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:00 - [info] Node-RED version: v2.2.2
Dec 12 23:48:00 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:00 - [info] Node.js  version: v14.19.0
Dec 12 23:48:00 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:00 - [info] Linux 3.10.0-1160.45.1.el7.x86_64 x64 LE
Dec 12 23:48:00 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:00 - [info] Loading palette nodes
Dec 12 23:48:00 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:00 - [debug] Module: node-red-contrib-elastic 1.2.0 /root/.node-red/node_modules/node-red-contrib-elastic
Dec 12 23:48:00 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:00 - [debug] Module: node-red-contrib-elasticsearch-jupalcf 1.1.14 /root/.node-red/node_modules/node-red-contrib-elasticsearch-jupalcf
Dec 12 23:48:00 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:00 - [debug] Module: node-red-contrib-kafka-client 0.0.10 /root/.node-red/node_modules/node-red-contrib-kafka-client
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [info] Settings file  : /root/.node-red/settings.js
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [info] Context store  : 'default' [module=memory]
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [info] User directory : /root/.node-red
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [warn] Projects disabled : editorTheme.projects.enabled=false
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [info] Flows file     : /root/.node-red/flows.json
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [info] Server now running at http://127.0.0.1:1880/
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [debug] loaded flow revision: 920af607af503301876c6d53c2518b7f
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [debug] red/runtime/nodes/credentials.load : no user key present
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [debug] red/runtime/nodes/credentials.load : using default key
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [debug] red/runtime/nodes/credentials.load : keyType=system
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [warn]
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: ---------------------------------------------------------------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: Your flow credentials file is encrypted using a system-generated key.
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: If the system-generated key is lost for any reason, your credentials.....
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: You should set your own key using the 'credentialSecret' option in....
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: file using your chosen key the next time you deploy a change.
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: ---------------------------------------------------------------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] runtime event: {"id":"runtime-state","retain":true}
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] runtime event: {"id":"runtime-deploy","payload":{"revision":"920af607af503301876c6d53c2518b7f"},"retain":true}
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [info] Starting flows
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [debug] red/nodes/flows.start : starting flow : global
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [debug] red/nodes/flows.start : starting flow : 9315ae661c6a0622
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [debug] red/nodes/flows.start : starting flow : c63aebfc83c8eb29
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [debug] red/nodes/flows.start : starting flow : b3232cb9a4f7d15f
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [debug] red/nodes/flows.start : starting flow : 6cf8186019735673
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [debug] red/nodes/flows.start : starting flow : tenant
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [debug] red/nodes/flows.start : starting flow : 7d62d43ee85cba6d
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [debug] red/nodes/flows.start : starting flow : f050a24709b413b4
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:global] start flow [global]
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:global] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:global]  id               | type         | alias
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:global] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:global]  b0dc117744115cae | mqtt-broker  |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:global]  1fcebd3f7151ed91 | kafka-broker |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:global]  360bacfd3fe042cd9078a343b1854dfc | mqtt-broker  |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:global]  02fdcfd940cdf42c | mqtt-broker  |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:global]  b948f7e9a16470c9 | kafka-broker |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:global]  78d2b34634dc25b7 | mqtt-broker  |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:global]  5cef1ca393a365be | mqtt-broker  |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:global]  fb1c2dcd03b8a026 | kafka-broker |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:global]  c592453c0d0fe76c | kafka-broker |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:global] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622] start flow [9315ae661c6a0622]
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622]  id               | type         | alias
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622]  89b1c97b20a38685 | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622]  1fe0538d3e440919 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622]  39164416a07e4785 | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622]  97695a51fe2c1647 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622]  c098c4fb8b4d7a6b | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622]  c4bc11789606a22a | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622]  2c3f4b923f06b1a1 | debug        |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622]  1ba10c327be9f671 | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622]  f58b782595991934 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622]  8451d570a1fcbe55 | debug        |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622]  5864d35ab1cacfed | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622]  f879b33e44891595 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622]  87f95bbca2b39748 | debug        |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622]  80513992250390bf | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622]  8d8dce535ae3cc20 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622]  f46e88eaac328fa1 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622]  c1e7c1b0779df06a | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622]  aca344e5d682a81f | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622]  5d2b32d115fc3a84 | debug        |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:9315ae661c6a0622] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:c63aebfc83c8eb29] start flow [c63aebfc83c8eb29]
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:c63aebfc83c8eb29] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:c63aebfc83c8eb29]  id               | type         | alias
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:c63aebfc83c8eb29] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:c63aebfc83c8eb29]  5b064c2e6730a360 | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:c63aebfc83c8eb29]  505aaaa162794450 | debug        |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:c63aebfc83c8eb29]  f1877752c20fa730 | json         |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:c63aebfc83c8eb29]  a404befcba440bf0 | kafka-consumer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:c63aebfc83c8eb29]  d3ea6068c7ca51c3 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:c63aebfc83c8eb29]  0fd920bd5dcae5e3 | debug        |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:c63aebfc83c8eb29] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:b3232cb9a4f7d15f] start flow [b3232cb9a4f7d15f]
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:b3232cb9a4f7d15f] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:b3232cb9a4f7d15f]  id               | type         | alias
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:b3232cb9a4f7d15f] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:b3232cb9a4f7d15f]  04efd336fcbeb1c9 | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:b3232cb9a4f7d15f]  c93fe2904aa073a7 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:b3232cb9a4f7d15f]  776e936dce462cdb | mqtt out     |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:b3232cb9a4f7d15f] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:6cf8186019735673] start flow [6cf8186019735673]
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:6cf8186019735673] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:6cf8186019735673]  id               | type         | alias
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:6cf8186019735673] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:6cf8186019735673]  8e264839903c8efe | kafka-consumer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:6cf8186019735673]  1586fdc77bc90ef8 | mqtt out     |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:6cf8186019735673]  d431615c1ad5f52b | debug        |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:6cf8186019735673] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant] start flow [tenant]
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant]  id               | type         | alias
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant]  966f4e28c1a244e9a15aaddea66d6a34 | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant]  4a0cd5ca67ce418e | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant]  582d8f5421d44de9820464a52f555c50 | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant]  0861e9c6588c4ab4aa982f3f4ddf5b7f | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant]  c5f207b67a8941b69a128047cd096ab0 | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant]  399aa72ee64e44bc88042e0ecdbebeac | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant]  c88e95ecbfc94e8a8d2bf46d349c50bb | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant]  07bd0570964e4f779a8de021e652ab79 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant]  0ba88bcdeafa4f78af9346b4303b33b7 | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant]  210798c1d2664a94854e7bfae54c0ef2 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant]  34bb7090b84cafe4 | debug        |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant]  769cd2ea4dd74992af08d17ef7213bd0 | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant]  eae5dbd79b1444fa889edac27d619b28 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant]  fd696c6f4c8a42c495e9e77c0e86d1ee | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant]  d8ed4d50d6114d49972dc42316a4dcbd | kafka-producer |
...
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant]  d4ac770aa1ae4aa5837696c365730542 | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant]  2a2c45d85ed6446cbdd1460b43b7e123 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:tenant] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:7d62d43ee85cba6d] start flow [7d62d43ee85cba6d]
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:7d62d43ee85cba6d] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:7d62d43ee85cba6d]  id               | type         | alias
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:7d62d43ee85cba6d] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:7d62d43ee85cba6d]  16fcd91e60e013b6 | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:7d62d43ee85cba6d]  c505263c7f501d47 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:7d62d43ee85cba6d]  30e2524fccc9df0b | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:7d62d43ee85cba6d]  b45115178da4cb04 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:7d62d43ee85cba6d]  9e597d4294d98e47 | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:7d62d43ee85cba6d]  efe14819923aa344 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:7d62d43ee85cba6d]  9ba9c3e13312997c | mqtt in      |
....
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:7d62d43ee85cba6d]  80a0db337075d1ab | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:7d62d43ee85cba6d]  00457ec6045bee40 | debug        |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:7d62d43ee85cba6d]  bbbb644bf7231eaa | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:7d62d43ee85cba6d]  a9c780b55070d073 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:7d62d43ee85cba6d]  346d2d8fb0e68e7b | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:7d62d43ee85cba6d]  cf3b517a3934ede9 | mqtt out     |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:7d62d43ee85cba6d]  9287d999014f9d4f | mqtt out     |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:7d62d43ee85cba6d] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:f050a24709b413b4] start flow [f050a24709b413b4]
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:f050a24709b413b4] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:f050a24709b413b4]  id               | type         | alias
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:f050a24709b413b4] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:f050a24709b413b4]  9355180943a26ff0 | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:f050a24709b413b4]  47addee7183c7c85 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:f050a24709b413b4]  6a7107a71c641a68 | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:f050a24709b413b4]  29c5871a9080c3b4 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:f050a24709b413b4]  d87d55773ec88599 | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:f050a24709b413b4]  fb0d8a5447a382f8 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:f050a24709b413b4]  099eb206d02f45299da220af4b698c94 | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:f050a24709b413b4]  2b8cdeec278049078fa000c65260c89a | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:f050a24709b413b4]  803d12e45c9f4b5ca044e68373719cd6 | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:f050a24709b413b4]  85f7426e5e0a405580362e67539262f8 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:f050a24709b413b4]  ca2423479b2a5199 | mqtt out     |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:f050a24709b413b4]  91543e068ab3b4b5 | mqtt out     |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:f050a24709b413b4]  d4d305f16b4247bca7e1290080e7fd5b | mqtt in      |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:f050a24709b413b4]  c81170bc6c2949468791a57dd321dc55 | kafka-producer |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:f050a24709b413b4]  9ba510691f10e051 | debug        |
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] [flow:f050a24709b413b4] ------------------|--------------|-----------------
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] runtime event: {"id":"runtime-state","retain":true}
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [info] Started flows
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [info] [mqtt-broker:内网121] Connected to broker: mqtt://12x.2x4.x31.11:3883
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [info] [mqtt-broker:灰度环境的MQTT] Connected to broker: mqtt://12x.2x4.x31.11:12883
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [info] [mqtt-broker:b0dc117744115cae] Connected to broker: mqtt://localhost:1883
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [info] [mqtt-broker:tenant_middleware] Connected to broker: mqtt://172.18.76.173:1883
Dec 12 23:48:01 iZwz995gddyo8yur1ec5x3Z Node-RED: 12 Dec 23:48:01 - [trace] comms.open RkogZunr65kTw5U9iY3T/KUlQJImspwrK+3gKOTpBxE=
Dec 13 00:04:05 iZwz995gddyo8yur1ec5x3Z Node-RED: 13 Dec 00:04:05 - [trace] comms.open EZOQ1tuWRnjNO7+gD+tnkZQZ5as5gv2C7bw9t4dv0Zs=
Dec 13 00:04:16 iZwz995gddyo8yur1ec5x3Z Node-RED: 13 Dec 00:04:16 - [trace] comms.close EZOQ1tuWRnjNO7+gD+tnkZQZ5as5gv2C7bw9t4dv0Zs=

Colin · 13 December 2023 09:07

I think it would be worthwhile looking back at the node-red log for a time when the comms failed. That may help to identify whether it is a node-red problem or something else.
You probably don't need the log level set to Trace.

Can you still ping the node red url when it fails?

hnuzhoulin · 13 December 2023 09:53

It happened again, but this time I didn't rush to restart Node-RED. Instead, I verified if the data flow in the processes behind Node-RED was interrupted. Upon checking, I found that although the CPU and load on the server hosting Node-RED were high, it was continuously processing data and successfully sending it to the destination. The subsequent programs successfully received the data and wrote it into InfluxDB.

When I checked the monitoring again, I noticed that the number of TCP connections had risen from 400 to over 4,000 in the last four hours. All these new TCP connections were to Kafka. Here, Node-RED acts as a Kafka producer, so I couldn't find any useful information on the Kafka side. Looking back at the historical monitoring, I noticed that each time there was a failure, there was this phenomenon of connection counts skyrocketing to over 4,000.

During the failure, I was able to SSH into the server hosting Node-RED, and from other servers, I could telnet to port 1880. It seems that only the dashboard was unusable. The flows I have in Node-RED are quite simple, serving primarily a forwarding role. The exported files are as follows:

flows202312131652.json (60.1 KB)

hnuzhoulin · 13 December 2023 09:56

Yes, I can ping the node when it fails.
and it works fine as data forward,seems just 1880 dashboard do not work.

But the load increase.
more info click this

hnuzhoulin · 13 December 2023 09:57

BTW, if do not set to trace,no logs when nodered running.

Colin · 13 December 2023 10:07

level: "info"
should show enough.

You need to work out what is going on with kafka. It sounds as if it is clogging up node-red so it is not able to provide the dashboard successfully.

hnuzhoulin · 13 December 2023 10:27

one possibility is nodered cause some error,it connects more than normal connection with kafka?

and the other is kafka cause some problem,then nodered have to create new connection to kafka, if so,why not close old connection?

besause all state of these network connections are established,so I think the first one.

Colin · 13 December 2023 10:32

It might be worth asking on the kafka node's github page. Perhaps the author of the node can help.

hnuzhoulin · 14 December 2023 03:29

Thanks，these flow seems work well long time.

But the network connection number with kafka increase so fast and the flow is so simple

system · 12 February 2024 03:29

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
NodeRed 2.2 - Error 502 General	13	3731	7 February 2022
Get 502 Bad Gateway from HA while selecting Node-RED General	5	1752	24 April 2022
Connection lost - dashboard (and editor) General	16	287	23 December 2022
Have a troubles after update to Node-RED 2.1.2 General	7	282	24 December 2021
Node-RED frequent crashing - "Connection Lost X" General	6	523	24 June 2020

Periodic 502 Errors with Node-RED Version 2.2 - Need Help for Troubleshooting

Related Topics