A Syslog Server for the Dashboard

[Christian-Me]

Hi perhaps somebody is interested.

Just finished a syslog server for the node-red dashboard. It makes use of ui-table. I hope somebody finds it useful. It is build around ui-table and a subflow I use for all kinds of tables in my projects.

https://raw.githubusercontent.com/Christian-Me/node-red-contrib-home/master/Node-RED/syslog/screenshots/demo.gif(image larger than 4096KB)

Features

• can handle quite huge amounts of data only limited by the clients browser. As every line is sent individually Node-RED is don`t have to hold the complete table (amount limited by maxDisplay) I Managed to get more than 100.000 rows before chrome spits out an out of memory error (maxDisplay=0 means unlimited)
• can store lines for reconnects for tab changes or new clients (amount limited by maxStore) (maxStore = 0 will eat all Node-Red memory!)
• store messages in background (when the table is invisible)
• filter messages by host right at the input
• can use the filter capabilities of ui-table for existing data.
• columns can be resized and ordered (layout will be saved)

I don't know if it useful or only a showing the features of ui-table. BTW: Is a log handler available to send Node-RED log as Syslog messages? I don`t want to reinvent the wheel. That could be a nice usecase too.

Important: Enter the name of your dashboard tab into the ui-table handler config.

The flow is too big to post here but you can find it on my github

Christian-Me/node-red-contrib-home

Technical things

• the reusable ui-table handler subflow take care of everything around ui-table. Sending Data line by line. Updating existing Data (as used in remote device table), sorting/hiding columns and rows. and many other things
• keeping control of the amount of rows in the font-end is a little bit crudely implemented but effective by simply deleting currentId-maxRows if currentId-maxRows>0. If the flow stumbles there could be old rows left. They will disappear after tab change or refresh. If maxStore is smaller than maxRows in the beginning non existing rows will be "deleted" after refresh. But ui-table seems fine with that.
• all data preparation is done in feed table so it could be adopted for other sources.
• I used the syslog input node (https://flows.nodered.org/node/node-red-contrib-syslog-input)
• the table is formatted by a json configured in ui-table handler (more info see tabulator docs)
• ui-table filters only apply to existing data. New rows will appear unfiltered. Don`t know right now to solve this right now.

[IoTPlay]

Hi @Christian-Me, I am trying this out. I can see the messages coming in from the setup rsyslogs of other RPi's, I can even see where in the Flow context that they are stored.

But... The UI does not update / show the messages, I did click the live slider - and TableConfig is showing ResponsiveLayout: true.

I left the UI to Syslog Server - the way you have it on the flow in the handler.
Anything else I should be doing?

[Christian-Me]

Hi @IoTPlay,
great that you give it a try.
The Date is stored in flow context

• tableConfig holds all fonfiguration
• tableData the cahed table content
• tableEdit the edits (if edit is enabled)

You can configure it as you like here

@IoTPlay

But... The UI does not update / show the messages, I did click the live slider - and TableConfig is showing ResponsiveLayout: true .

the ui-table handler is quite versatile and not all features are used in this case: the tableConfig sould look like this: (responsive layout is not used right now (see remote device table for an example). The columnWidths will pop up when you modify column width ...)

When you acivate the slider you should see the data. If you are using a diffent source please take a look at the demodata how it should look like:

{
    "facility": "daemon",
    "facilityCode": 3,
    "severity": "info",
    "severityCode": 6,
    "tag": "systemd[1]",
    "timestamp": "2018-12-26T17:53:57.000Z",
    "hostname": "localhost",
    "address": "127.0.0.1",
    "family": "IPv4",
    "port": 38514,
    "size": 80,
    "msg": "some demo data"
}

But you can feed in everything as long as the properties match the field in the tabulator json:

i.e. property hostname matches the field value

first I would place a debug node on the output of the feed table node and ui-table handler. Should look like this

and

[IoTPlay]

Got it working, thank you, will give feedback on its usability in my env.

Thank you !

[happytm]

@Christian-Me

syslog input node

Hi @Christian-Me

I wanted to use following MQTT messages instead of syslog input node. Is it possible to display data from these messages in your ui-table?

Published topic 'Livingroom/voltage with paylod 2.62
Published topic 'Livingroom/temperature with paylod 1
Published topic 'Livingroom/humidity with paylod 2
Published topic 'Livingroom/pressure with paylod 77
Published topic 'Livingroom/light with paylod 5
Published topic 'DeviceStatus with paylod {"Location":"Livingroom","rssi":-66,"mode":0,"ip":6,"channel":7,"sleeptime":1,"uptime":140}
Published topic 'Livingroom/rssi with paylod -66
Published topic 'Livingroom/mode with paylod 0
Published topic 'Livingroom/ip with paylod 6
Published topic 'Livingroom/channel with paylod 7
Published topic 'Livingroom/sleeptime with paylod 1
Published topic 'Livingroom/uptime with paylod 140
Published topic 'SensorValues with paylod {"Location":"Bedroom1","Voltage":2.62,"Sensor1":"level","SensorValue1":8,"Sensor2":"presence","SensorValue2":0,"Sensor3":"motion","SensorValue3":81,"Sensor4":"custom","SensorValue4":18}

Published topic 'Bedroom1/voltage with paylod 2.62
Published topic 'Bedroom1/level with paylod 8
Published topic 'Bedroom1/presence with paylod 0
Published topic 'Bedroom1/motion with paylod 81
Published topic 'Bedroom1/custom with paylod 18
Published topic 'DeviceStatus with paylod {"Location":"Bedroom1","rssi":-67,"mode":0,"ip":26,"channel":7,"sleeptime":1,"uptime":146}
Published topic 'Bedroom1/rssi with paylod -67
Published topic 'Bedroom1/mode with paylod 0
Published topic 'Bedroom1/ip with paylod 26
Published topic 'Bedroom1/channel with paylod 7
Published topic 'Bedroom1/sleeptime with paylod 1
Published topic 'Bedroom1/uptime with paylod 146
Published topic 'SensorValues with paylod {"Location":"Livingroom","Voltage":2.62,"Sensor1":"temperature","SensorValue1":1,"Sensor2":"humidity","SensorValue2":2,"Sensor3":"pressure","SensorValue3":224,"Sensor4":"light","SensorValue4":6}

Published topic 'Livingroom/voltage with paylod 2.62
Published topic 'Livingroom/temperature with paylod 1
Published topic 'Livingroom/humidity with paylod 2
Published topic 'Livingroom/pressure with paylod 224
Published topic 'Livingroom/light with paylod 6
Published topic 'DeviceStatus with paylod {"Location":"Livingroom","rssi":-67,"mode":0,"ip":6,"channel":7,"sleeptime":1,"uptime":140}
Published topic 'Livingroom/rssi with paylod -67
Published topic 'Livingroom/mode with paylod 0
Published topic 'Livingroom/ip with paylod 6
Published topic 'Livingroom/channel with paylod 7
Published topic 'Livingroom/sleeptime with paylod 1
Published topic 'Livingroom/uptime with paylod 140

Thanks.

[happytm]

@Christian-Me,

I liked your another flow at [Announce] remote device table (and collaboration wanted) very much. You have created very powerful and useful table there but unfortunately that thread is closed for some reason. This type of tables are very efficient way of displaying large amount of data in just one page. I think I can compare it with MQTT Explorer unfortunately it is not available on smartphone.

Please keep up good work.

Thanks.

[Christian-Me]

Hi,

certainly you can feed in and data you want.

First you have to prepare your data. I would do it in a function node right after your MQTT in node. You have simple topic : value messages. These you could split into Room / Datapoint / value.
Your message coming out of this node should look like

msg.payload={"room":"Livingroom","dataPoint":"temperature","value":1);

They you have to modify the tabulator json for your 3 fields but leave the id in (nessesary for counting and deleting messages):

leave the rowProperty to log if you do not want to change to much in the feed Table node
You should filter out the DeviceStatus mesages as they have a different format (JSON) (perhaps feed these into the remote device table.

If you like to filter your messages by room change this (not tested!)

var hosts = context.get("hosts") || ['[all]'];
if (!hosts.includes(msg.payload.hostname)) {
    hosts.push(msg.payload.hostname);
    hosts.sort();
    context.set("hosts",hosts);
    node.send([null,{topic:"filterOptions",options:hosts}]);
}

to (again leave the context name to hosts as it could have impact somewhere else - you can change this later) it assumes that the topic & message is already prepared in msg.payload.room, msg.payload.datapoint, msg.payload.value

var hosts = context.get("hosts") || ['[all]'];
if (!hosts.includes(msg.payload.room)) {
    hosts.push(msg.payload.room);
    hosts.sort();
    context.set("hosts",hosts);
    node.send([null,{topic:"filterOptions",options:hosts}]);
}

Then you should end with a table with 4 columns (id, room, datapoint and value)

[Christian-Me]

@happytm

unfortunately that thread is closed for some reason

Thank you ... All treads are closed automatically after a certain period. Feel free to open a new topic if you have "new" questions.

The problem with so "complicated" flows is that it takes a lot of afford to even test them. And sometimes the timing of posting these is critical as they disappear in the timeline very quickly.

May I ask: What firmware are you using? It seams like a ESP8266 or ESP32 firmware but not ESPEasy.

[zenofmud]

@happytm

Published topic 'Livingroom/uptime with paylod 140

is there a typo? You have paylod where is should be payload

[happytm]

I got my work cut out for me. It will be fun doing this.

@Christian-Me

May I ask: What firmware are you using? It seams like a ESP8266 or ESP32 firmware but not ESPEasy.

I am using my custom code for a local Standalone IOT network using ESP8266 devices. My goal was to create a network of battery powered devices (ESP8266) sending data using probe request to a gateway (ESP8266). The gateway is main powered and basically collect all these data from slave devices and publish it via MQTT. The gateway has built in MQTT broker. Code for Master and slave is at following links:

https://github.com/happytm/BatteryNode/blob/master/ProbeReceiver.ino (for gateway or master)

https://github.com/happytm/BatteryNode/blob/master/ProbeSender.ino (for all slave devices)

@zenofmud

is there a typo? You have paylod where is should be payload

Yes it is a typo in my code. Thanks for catching it.

Thanks.

[TotallyInformation]

@Christian-Me

BTW: Is a log handler available to send Node-RED log as Syslog messages? I don`t want to reinvent the wheel. That could be a nice usecase too.

Sorry, missed this when you originally created the thread.

If you have Node-RED running via systemd, the Node-RED log is automatically surfaced to syslog via systemd's journal service.

[happytm]

@Christian-Me

I am using your flow with minor modifications to table and feeding it from MQTT in node.

My flow file is here : https://github.com/happytm/BatteryNode/blob/master/test/ui-table.json

I have created 6 inject nodes to simulate published data from 3 of my ESP8266 devices.

I am using ESP8266 code from here for publishing MQTT messages : https://github.com/happytm/BatteryNode/blob/master/ProbeReceiver.ino

Now I have to solve following with above flow:

• My table update sometimes but not every time it receive MQTT message.

• I also wanted to add "time received" column to display time of the data received for
  each row.

• I also need to figure out Signal, Uptime in milliseconds and Last seen columns with
  your help.

-Can you look at my function node and see if it is correct or I need to do something
more to implement it.

Thanks.