Hello!
Preface
In the beginning of the year I decided to broaden my knowledge in ELK stack, thus I set up a local ElasticSearch and Kibana and then I needed some data ... lots of it.
Instead of using numerous meaningless data sets I though about leveraging my Node-Red instance and all the running flows inside it that produce thousands of messages per day. Should be enough to get me going.
And as we know how changing the light bulb leads to fixing everything else, ELK will wait and my first custom Node-Red node logIO is almost ready for release.
I'll appreciate if you take a look, maybe even test it if fits in your flows and give me some pointers or ideas before I make 1.0.0 release.
I'd aso like to thank everyone here, this forum was super helpful during development and I took some inspirations not only from official nodes but also nodes you built and shared here.
logIO
As the name suggests, logIO node logs INPUT and OUTPUT messages from nodes. Contrary to other nodes out there, you don't have to wire each and every node you want to log directly to logIO, but you can use it similar to core catch or status components. Of course you can use it also as core debug component with additional improvement here too.
Log outputs:
logIO currently supports following log outputs (multiple can be selected):
- Node-Red debugger panel
- System console (supports: colored output based on log level, json output, utc/local timestamps, additional meta data)
- File (supports file rotation, compression and deletion of old logs, json output, utc/local timestamps, additional meta data)
- ElasticSearch (sends logs directly to elasticsearch - tested with ES version > 8, supports: utc/local timestamps, additional meta data, custom index definition)
Log modes
inline [default] mode
In this mode, logIO behaves similar to core debug node, where you can pipe other nodes into logIO and input messages will be logged.
BUT you will notice that in inline mode, logIO has also an output, thus you can include in in the middle of the flow. if so, logIO will automatically log every input and output message of all the components that come after it.
flow, select, all modes
In these modes, logIO behaves similar to core catch or status components, meaning you don't have to wire other components into it, but it will still log input and output messages of all, current flow or selected nodes.
Controlling the node logging
By default, logIO will log messages of all connected or selected nodes from deploy onward.
You can disable this behavior by deselecting the Automatically start logging at startup and activate the logIO node through incoming message.
To activate logging, incoming message should have a _logIO_ object with key activate set to true:
{
"payload" : "your payload",
"topic": "your topic",
"_logIO_" : { "activate": true }
}
To deactivate logging, incoming message should have a _logIO_ object with key activate set to false:
{
"payload" : "your payload",
"topic": "your topic",
"_logIO_" : { "activate": false }
}
You can also toggle the logging through components button same as in core debug component.
Have in mind that those two actions are not the same and for example if you activated the logIO through incoming message but logIO is deactivated in editor, logging will still be paused.
Logging status and mode are also displayed in status of the component for easier visualization.
If logging is deactivated and logIO is in inline mode, logIO will act as a pass through component. It will not produce any logs, just pass the message onward as is.
Log levels: [error, warn, info, debug]
Default log level is set to debug.
If message doesn't have its own logLevel, it will be assigned one as defined in logIO and always logged. In this case logIO doesn't act as a filter, but only attaches the selected logLevel to the log entry.
Of course more common case is that you want to log you messages based on severity, so if the message contains _logIO_ object with key logLevel set to one of allowed levels, that one will be used for:
- displaying that log level in log entry
- omitting the message from logs, if current message logLevel priority is lower than logIO log level priority
e.g: logIO log level is set to warn
This message will be logged.
{
"payload" : "your payload",
"topic": "your topic",
"_logIO_" : { "logLevel": "error" }
}
This message will be logged.
{
"payload" : "your payload",
"topic": "your topic",
"_logIO_" : { "logLevel": "warn" }
}
This message will be logged with warn logLevel (same as logIO defined log level)
{
"payload" : "your payload",
"topic": "your topic",
}
This message will NOT be logged.
{
"payload" : "your payload",
"topic": "your topic",
"_logIO_" : { "logLevel": "info" }
}
Here I have a question, would it make more sense to put logLevel directly on message and not inside _logLevel_ key? What are you currently using - is there any convention?
{
"payload" : "your payload",
"topic": "your topic",
"logLevel": "error"
}
To install and try it out, search for @zigasebenik/node-red-log-io or install manually (as I don't know how long in takes to appear in Node-Red after submitting it).
npm install @zigasebenik/node-red-log-io
Github repo is here.
I have to polish the editor a bit and properly document everything in the following days based on potential upcoming changes. Nevertheless it's ready to test it out.
