I am willing to use filter nodes from logstash to do some changes in a payload received from a MQTT subscriber node in node-red.
I would like to check for a field with value greater than 1000 and then add a new field in a payload and finally save this modified payload to elasticsearch. So far, i have used a 'Ruby' filter node, which allows to write ruby codes in 'Code' section. For e.g;
'if event.get("rpm") > 1000 event.set("Idle", false) end'
Additionally, i have also entered {"Idle", false} in 'Add field' section.
This does not make any changes to the payload or may be i am doing something wrong.
I checked for a better documentation but it seems there is not a good documentation for most of the nodes from logstash.
I am looking for a guidance. Many thanks
Please show us what you see in a debug node showing the output of the MQTT node, and tell us exactly what field you want to add and what value it should contain.
You can add the property by feeding the message through a Switch node to split off those messages with msg.payload.Epm_nEng.Epm_nEng > 1000, and feed those through a Change node to set msg.payload.idle to the boolean false.
For the count, do you mean that you need to keep a count of the number of messages received for each engine, and increment that and add the property to each message? If so then does the count just keep incrementing over the months and years? Cannot logstash be queried to give you the count of the number of records for a particular engine at the time you need to report on it, rather than adding it into the database?
The payloads are received in milliseconds by the MQTT subscriber. If i use switch node and change node to accomplish the task, then will each and every payload will be successfully processed? Will not there be any payload drops?
I think this is where logstash comes to existence. Please correct me, if i am having any misunderstandings. Is there a possibility to accomplish my task by using logstash filters?
And for counting the number of messages, yes i can also use a 'Count' query in elasticsearch to get a total number of documents for each engine, but it might take a while for a query to process if we have millions of documents for each engine.
So setting a counter for each message for an engine with help to reduce the query time as we could only query for the latest documents in an elasticsearch.
I would be really glad if my misunderstanding could be cleared. Thank you
If you are worried about the overheads in node-red then you would be better to go straight to logstash without involving node-red at all. Certainly the overheads in a switch and change node will be much less than those maintaining a count of records for each engine.
Do you know that is correct? A very quick google appears to suggest that indexed fields are supported, in which case a count may actually be very efficient.
However, if do you want to maintain a count in node red then you will need to understand how to use persistent context in node red, so start by reading up on that in the docs. Then you can put an array of engines in context and increment the relevant count for each record.
Looking at the data I have to wonder why you are using logstash rather than something like MySQL or similar, or even influxdb. Then doing a count, even over millions of records, would take very little time.
