Check MQTT Message exist in "File"

[nbech]

Im looking to take a published mqtt message and compare with a reference file to see if message payload is listed.

Because my text conversions produces some false positives, Im looking to help that by taking the last however many published mqtt messages within 5 seconds of each other, to compare with a reference file to see if either of the message payload is listed. After the 5 seconds from the first message was sent, it will start over and wait for the next message payload.

Here is an example of how the messages are sent..
2/20/2022 10:00:13 - Payload: j3hh432j
2/20/2022 10:00:14 - Payload: l3hh432j
2/20/2022 10:00:17 - Payload: l3hh432j
The example above shows three messages sent within 4 seconds, which include 2 different payloads. Im hoping to check if any of these groupings exist in a reference file. Lets say j3hh432j exist but the l3hh432j doesnt, it will still count these three messages as "ok" to help avoid false positives.

To give some backstory on my project.. I am working on a project that detects images and converts to text, the text is gathered and sent to an mqtt topic. I am looking to have it be where if the message payload sent exist in a reference file, then it will trigger a "ok" event, if the message payload sent is not listed in the reference file, then it will trigger a "bad" event and send an email.

I am using a reference file because I will have 100-200 message payload stored for the message payload to be compared.

These will be exact matches.

Here is an example of the txt file.
j3hh432j
657654j
zzu3h12
5478372
Bh3j4 h3
1245 2213
BA Nu
...

With this said, do you have an pointers as to the best way?

Thank you!

[Colin]

Rather than storing in a file I suggest storing them in an array in persistent context. Then the lookup will be trivial.

[greengolfer]

Maybe an overkill for your situation... In my use case see Use of NodeRed for international weather, climate, ocean, hydrology data exchange I am using redis to do this kind of check. We use the payload j3hh432j as a key in redis. Through redis get/set it is fairly easy to do this. Obviously, you'll need a redis instance (typically a docker version of it will do) for this to work.

As Colin wrote, an array in a persistent context is simpler than "grepping" into a file...

[nbech]

I have a .txt file with names, the flow I have below, gets a message from mqtt and check if the message is found listed in the .txt file. If found, it will read "true".
The problem im having is its not an exact match. If I post an message of "12345" it returns "true". I only want it to read true if its an exact match (In this case of what I have in the .txt file, its "12345678" or "1234").

Here is the flow hotNipi was helping me with.
flows.json (5.3 KB)

names.txt (103 Bytes)
I have a .txt file with lines of names…
test;
12345678;
1234;
hello;

[Buckskin]

I think you need to make sure you are comparing strings and not numbers

[nbech]

I have done as you said but as you can see in the screenshot its saying "true" when I search "te" and "test". I only want it to say "true" if the payload is an exact match.

This also happeneds when I pass "1234" as a message it reads "true" because I have "12345678" listed in the .txt file. Is there anything I can add to the file or to the function that would only read as true if the message contains the full string? ie- "12345678"

I have added ; at the end of each name/number row in the .txt file hoping that I can split data that way and have the function search the different rows of data looking for exact match.

I have a .txt file with lines of names…
test;
12345678;
1234;
hello;

Do you have any pointers on what im missing?

[smcgann99]

Well this is the problem with the function -

//using string.includes() method cos those words contain nonprintable characters \n //but mostly those will not harm so assuming that exact match is not important

You will need to do an exact match to avoid the problem you are seeing.

[nbech]

That what I’m looking for! Exact match.
Is that done in a function? I’m very green with this. I’m not familiar with node

[smcgann99]

Yes you will see that comment in the function node. I don't know what is in your text file but as a start you could try changing if (word.includes(entryToLookFor) to if (word==(entryToLookFor)

[nbech]

That worked! Thank you!

I have one more question, the application that im using that sends the names as an mqtt message only when an image gets captured by the camera. The software send the image to google ai and returns the results. Sometimes it gets the letters confused.

One way I thought about fixing this is to have the application sends mutable messages within 5 a second block, hoping that with one of them will be a perfect match.
Is there a way queue these 2-3 messages and check if one of them matches. If one of the 2-3 mqtt messages matches the text file, it will consider it as true. After 5 seconds, the queue will clear and be ready for the next photo to be taken.

Basically im going to be getting alerts anytime a name doesn't match the text file that I have listed. Im trying to have it be close to 100%, so im having it send 2-3 messages with a 5 seconds block. I want to process these messages and check if anyone of them match. If they any, then it will be true, if none match then it will mark as false.

Thank you again! Im marking this as being solved.

[smcgann99]

Do you mean you will send multiple messages to the google ai ?
If so will these be "different" images or several copies of the same one ?

In other words if you send the same image you are likely to get the same result

[nbech]

These are different images. An image is taken every .5 seconds as an object passes. Depending on the speed as it passes the frame, it takes anywhere from 2-3 photos per object.

When the image is processed, I have a script that sends the returned result from google ai to the mqtt topic which is picked up with this function in node red.