Confusions regarding the level of effort needed to debug a flow

[dr94node]

I'm new to node red
I've been programming in node.js as my professional occupation for some years.

I used to have a small automation system build with Huginn which I'm running into some performance issues from time to time so I'm investigating other options.

I don't think Apache Airflow is simple enough for my use case so I pick node red as it seems light weight and I love JavaScript in general.

However I had a rough start, I'm running into a few issues already when migrating one of the simpler workflow I have.

This workflow will retrieve some JSON data from a certain API endpoint and parse it, based on the response it will decide whether it will go one of 2 optional routes, or stop processing the event further.

Sounds simple enough right?

The first problem I run into is that my API key for making the requests can not be properly stored, I cannot find a built in solution for it so I installed a third party package node-red-contrib-credentials .
I consulted Node credentials : Node-RED but it seems intended for node developers.

The second problem I'm facing is I'm unable to debug the output msg object as a whole, I can only look at one property at a time, by default it is msg.payload, make working with http request node a big hassle because it produces msg.statusCode, msg.headers etc. as well.

The third problem is I quickly realized that I need a whole bunch of debug nodes, pairing with each existing node in order to get a clear picture of what is coming in and out of each node, not to mention it cause filtering logs a heavy labor.

The fourth time I'm surprised to find out that in a function node the node.log will not log into debug panel, only node.warn and node.error log into debug panel... But why? Not to mention the node context is resetting itself if I modify the function node, as a result I have to use flow context instead.

The last problem I had is with the mustache template, it is unclear where it works outside of template node and http request node in the URL field, granted it should be very useful elsewhere.

When compared with Huginn I never had these problems, each agent maintain its own events queue, JavaScript agents can log to its own panel with this.log and liquid template works in every field, memorized values are preserved between modifications.

[Steve-Mcl]

@dr94node

The first problem I run into is that my API key for making the requests can not be properly stored

You can use context (memory, file or other)

@dr94node

The second problem I'm facing is I'm unable to debug the output msg object as a whole

Double click the debug node
& choose "show complete message"

@dr94node

The fourth time I'm surprised to find out that in a function node the node.log will not log into debug panel, only node.warn and node.error log into debug panel... But why

Historical reasons - there is a topic knocking around.

@dr94node

Not to mention the node context is resetting itself if I modify the function node

This is normal. When a node is deployed it is effectively destroyed and recreated. Use flow or global context. Or use file backed context (memory is the default)

Your issues are mostly a matter of familiarity but it is definitely interesting to hear your perspective coming from another platform.

I'm certain things can be improved & this is an open source project - I look forward to seeing you around.

Ps, welcome to the forum and node-red.

[Steve-Mcl]

Also, here are a couple of canned texts that may help you get a head start (hope you don't mind)...

There’s a great page in the docs (Working with messages : Node-RED) that will explain how to use the debug panel to find the right path to any data item.

Pay particular attention to the part about the buttons that appear under your mouse pointer when you over hover a debug message property in the sidebar.

I recommend watching this playlist: Node-RED Essentials. The videos are done by the developers of node-red. They're nice & short and to the point. You will understand a whole lot more in about 1 hour. A small investment for a lot of gain.

[dr94node]

@Steve-Mcl

@dr94node

The first problem I run into is that my API key for making the requests can not be properly stored

You can use context (memory, file or other)

I guess it might not be immediately obvious to me how can I manually put a value to context? If I use a node to modify the context doesn't that means the node itself can expose my credentials & secrets? Well I guess I can just remove the node after the values are set...

@dr94node

The second problem I'm facing is I'm unable to debug the output msg object as a whole

Double click the debug node
& choose "show complete message"

Can't seem to find the option, I'm using official docker image, the version seems to be is 3.1.1
NVM found it.

@dr94node

The fourth time I'm surprised to find out that in a function node the node.log will not log into debug panel, only node.warn and node.error log into debug panel... But why

Historical reasons - there is a topic knocking around.

Cool good to know.

@dr94node

Not to mention the node context is resetting itself if I modify the function node

This is normal. When a node is deployed it is effectively destroyed and recreated. Use flow or global context. Or use file backed context (memory is the default)

From my limited computer programming knowledge regarding this matter, "context" is the setting and surrounding of a certain function(in our case, a node) which seems a bit strange if itself is not kept around due to the code running in it has been modified. This also brings in another question regarding how would one node keep some values to itself without exposing them to other nodes if flow context is recommended by default. Practically I guess one can develop with flow context and switch to node context once the code is verified working but still can be a pain point once there's a need to change the node.

Your issues are mostly a matter of familiarity but it is definitely interesting to hear your perspective coming from another platform.

I'm certain things can be improved & this is an open source project - I look forward to seeing you around.

Ps, welcome to the forum and node-red.

Thank you for the quick response, looking forward to my first PR to node-red repo.

[dr94node]

I did a quick count, making a simple workflow to work I manually injected 100+ times😂
I think Huginn has this "re emit" event feature so I can simply click a random event from second-to-last agent(node in Node-Red) to test the newest agent(node) I've added.

What for more it can directly dry-run an agent with last few events from upstream, making testing a breeze.