Assistance with Creating a Custom Node for Real-Time Data Processing

[salac64365]

Hello Everyone ,

I’ve been working on a project where I’m using Node-RED to process real-time data streams from multiple IoT sensors. The flow involves receiving data, performing some transformations, and then pushing the results to a dashboard for visualization. So far, I’ve been able to manage this using standard nodes, but as the project grows more complex, I’m finding that I need a more custom solution to handle specific processing logic.

I’m thinking of developing a custom node to better manage my real-time data processing needs. However, this is my first time attempting to create a custom Node-RED node, and I’m a bit stuck on where to start.

Here are a few specific questions I have:

1. Initial Setup: What are the best practices for setting up the development environment for creating a new Node-RED node? Any recommended tools or workflows?
2. Logic Implementation: How can I integrate JavaScript functions for data processing within the custom node? Is there a way to reuse existing JavaScript code that I’ve already developed in Node-RED, or should everything be re-implemented inside the custom node?
3. Packaging and Deployment: Once the custom node is created, how do I package it properly for deployment within a Node-RED instance, especially if I intend to share it with others?

Issue with Real-Time Data Reading from MQTT Node - Unable to Display Graph

Hello, I'm facing a challenge while trying to display a real-time graph based on data retrieved from an MQTT node in Node-RED. My approach involves reading data from the MQTT node and transforming it into the appropriate format. However, despite my efforts, I'm unable to visualize a functional graph. It seems that I might be missing something in my approach. Here's a snippet of the code I'm currently using in the function node: [Bez tytułu]

5. Debugging Tips: Any suggestions for debugging custom nodes during development? I’ve used the built-in Node-RED debug node extensively but wonder if there are more advanced gen ai techniques I should explore.

I’d greatly appreciate any advice or resources that the community could provide. Thanks in advance for your help!

[TotallyInformation]

@salac64365

Initial Setup: What are the best practices for setting up the development environment for creating a new Node-RED node? Any recommended tools or workflows?

This can be fairly personal. You will be developing for a very complex Node.js app so you will want to be somewhat familiar with Node.js, npm and, of course, JavaScript. You will want a good IDE - VSCode is almost certainly the best of breed right now. And you would be well advised to make sure you have a decent configuration for ESLint and Git/GitHub.

There is a simple starter node in the Node-RED source. Personally, I don't like the approach many people take and so I use a somewhat deconstructed method that, I think, allows much clearer understanding and parsing of the complex timings and interactions that your nodes have to deal with. I use this repo to test out my ideas: GitHub - TotallyInformation/Node-RED-Testbed: A blank canvas to test out ideas for Node-RED custom nodes. One of the benefits of this approach that I've found is that it makes it really clear in the runtime code what happens when - e.g. at node-red startup, at the deployment of a node to a flow and at the time a node instance receives a message. These things are complex and interlocked. Being able to see them more clearly saves a lot of head-scratching (it does for me anyway).

But there are plenty of other ideas and approaches. Node.js is nothing if not flexible.

Here is another toolchain from a member of this forum: 🎉 nrg-cli v1.2.1 released - #2 by AllanOricil

How you approach things rather depends on your experience, worldview and whether you will be working with others.

@salac64365

Logic Implementation: How can I integrate JavaScript functions for data processing within the custom node?

Yes, node.js is great for this. If you need to pick up 3rd-party packages, most people will have published to npm and you can very easily include such packages either for runtime use or for develop-time. Check out a few package.json files to get a feel.

@salac64365

Is there a way to reuse existing JavaScript code that I’ve already developed in Node-RED

If it is JavaScript code, you will generally want to move that code into your own node. If it consists of Node-RED flows, you may need to re-engineer the logic into node.js modules.

@salac64365

should everything be re-implemented inside the custom node?

It should be accessible to your custom node. However, not everything has to be directly in your node's code. For runtime code, you can use node.js modules and indeed your should do that. Check out node-red-contrib-uibuilder for a fairly extreme example. The main uibuilder node in particular makes use of around 11 other library modules of my own writing and around 6 external 3rd-party modules. (and a LOT more external modules for developing, not for running).

Your nodes exist in 2 contexts though and you should remember that. The runtime code (*.js) does the heavy lifting during operation. The *.html Editor code defines how your node looks and behaves in the Node-RED Editor. Even the editor code does not have to all live in a single html file though (even though many people do write nodes that way). Check out the testbed nodes I shared above to see how you can easily minimise what is in your node's html file to make it much easier to manage common code (shared across multiple nodes) and move some of the different parts of the code (namely the javascript and the help info) into separate files.

In general, it is much easier - especially as nodes get more complex over time - to keep things split out into logical files.

@salac64365

Packaging and Deployment: Once the custom node is created, how do I package it properly for deployment within a Node-RED instance, especially if I intend to share it with others?

This one is easy. Because a node or collection of nodes exists in a package. Your package is defined by a package.json file in the root of your repo.

Typically, as you finish a release, you push any final changes to GitHub and create a GitHub release. Then you publish that release using the npm tool to the npm library.

If you include a tag in your published release called "node-red", it should turn up in the flows library and the Node-RED palette manager for people to install.

@salac64365

Debugging Tips: Any suggestions for debugging custom nodes during development? I’ve used the built-in Node-RED debug node extensively but wonder if there are more advanced gen ai techniques I should explore.

A gazillion console.log messages!

But for when you get really stuck/confused, the node-red-contrib-inspector nodes are invaluable along with your browser dev tools. Even just viewing the Node-red runtime log is easier using the inspector much easier than trying to use a terminal window. So well worth the effort of setting it up and working out how to use it.

[AllanOricil]

@salac64365 try building a node with the nrg cli and tell me how it went.

GitHub - AllanOricil/nrg-cli: A CLI tool to quickly generate, build, and manage...

It simplifies the workflow, while making your node future proof to browser. node, javascript and css spec changes over time, and enables debugging client and server-side code using chrome/vscode dev tools. It does a lot more. Besides it, you won't need to write boirlerplate code anymore.

Your whole dev environment is setup with just these commands.

npm install -g @allanoricil/nrg-cli
nrg create -n project-name
cd project-name
npm install
npm run start

if you want to see changes appear on the browser as soon as you save your code, run npm run watch instead of npm run start. The watcher rebuilds your code, and reloads node-red with the latest changes. You don't even need to refresh the browser, it does it for you.

If you need to debug the server-side code of your node using a debugger, instead of console.logs, add breakpoints to your source code, then run npm run start:debug. Then finally attach a debugger to it using chrome or vscode.

Every nrg project is created with at least one node. And if you need to create another one, just run nrg create node -n node-name from the root of an nrg project.

Once you are happy with your node, and are ready to publish it. Just run

npm run build
npm publish

To make this experiment more reliable, create the same node from scratch, without nrg, and tell me how much time you spent to get to a dev environment.