Tcp in node loosing/stacking messages

SeRiusRod · 3 September 2019 09:29

I was doing some modifications to my system when I found that I was losing mqtt messages. I debugged the issue to the first node, a tcp input.
I use mysensors on my devices, and it has a network gateway that publishes the messages on the home network. I connect to the gateway with a tcp in node, and then transform and republish to mqtt.
Mysensors has a tool that allows me to debug the output from the gateway, and I could see the message following.

I've found that sometimes the tcp in node joins several messages separated by \n and possibly \r into the same payload. The following nodes don't expect this and messages are lost.
Also during series of messages, like when a device presents itself and it's child sensors, the tcp node drops some msgs.

My system runs in docker containers in a photonOS VM on a big esxi server.

Is this behavior normal?
Can someone help me debug or some this problem?

kuema · 3 September 2019 09:51

With TCP one should never rely on proper separation of the incoming bytes. Even if the message was sent as one chunk of data, it is not guaranteed it arrives as one chunk at a time due to segmentation, fragmentation, OS buffers... so you can receive only parts of your message, or even multiple message in one buffer, as you have observed.

However, the order of the bytes is guaranteed to be the same, of course.

dceejay · 3 September 2019 09:51

yes - that can happen - ... as @kuema just said

If you know they are strings then you can tell the node to expect strings and split them - but in general you probably can't

kuema · 3 September 2019 09:52

Exactly. Depending on your protocol, you would need to parse and split the bytes at the appropriate boundaries.

kuema · 3 September 2019 09:56

I do not know MySensors, but maybe it supports sending your data via MQTT directly?

SeRiusRod · 3 September 2019 12:01

Yes, it supports a MQTT gateway, but then I'll lose the possibility of using that tool, that only works with usb and lan gateways.
The tool allows to inspect and manage the topology, ask nodes for heartbeat, presentation, etc and allows OTA to them. And debug communications, of course.

I've modified the debug node to output to the console and this is what I've got now:

3 Sep 13:49:15 - [info] [debug:WIFI input] 
1;1;1;0;17;2818.8
3 Sep 13:49:15 - [info] [debug:WIFI input] 
1;1;1;0;18;1.804
3 Sep 13:49:15 - [info] [debug:WIFI input] 
1;1;1;0;24;1804
3 Sep 13:49:15 - [info] [debug:WIFI input] 
1;2;1;0;17;1.3
1;3;1;0;17;2429.2
3 Sep 13:50:19 - [info] [debug:WIFI input] 
1;1;1;0;17;1699.3
1;1;1;0;18;1.842
1;1;1;0;24;1842
1;2;1;0;17;0.7
1;3;1;0;17;1568.5
3 Sep 13:50:19 - [info] [debug:WIFI input] 
1;4;1;0;17;94.9

What we could name "topic" has always five elements, and the sixth is the payload. What I don't know is if between msgs and at the end it's adding \r\n or only \n and how many.
I've made the following function node, but I'm having problems with it. Is throwing "undefined" replace errors every other output.

try {
    const messages = msg.payload.replace(/(\\r\\n|\\n|\\r)/gm, "").split('\n');
    var topic = "", payload = "";
    
    messages.forEach( function(message, idx, array) {
        var parts = message.split(';');
        topic = 'mys-in/' + parts[0] + '/' + parts[1] + '/' + parts[2] + '/' + parts[3] + '/' + parts[4];
        payload = parts[5].replace(/(\\r\\n|\\n|\\r)/gm, "").trim();
        node.send({topic: topic, payload: payload});
    });
}
catch(err) {
    node.error(err, msg);
}
finally {
    return null;
}

SeRiusRod · 3 September 2019 13:55

This "stacking" of messages seems to occur and become worse depending on the workload of the nodes following it. I explain: If the following node is a function with only a node.debug(msg.payload);, it occurs now an then. But as the node does more things, it starts to stack, more and more.

That may be normal, but I don't find it right. I can imagine a big series manufacturing machine dropping bottle boxes or even crashing and making a pile of them because is driven with node-red and it dropped only one detection message.

... and I only have 3 devices in this testing network, only one is on ATM.

PS. Just one more thing: I've replaced "\n" and "\r" to "*" and "#", and then output to debug in a function. When debug outputs "1;1;1;0;18;2.319*1;1;1;0;24;2319↵" what does that latest char means? Should not have anything more.

ukmoose · 3 September 2019 12:38

So if you were using newline/carriage return as the end of line character and now you have decided not to. The string now includes any newline / carriage return characters ↵

If you are running a machine like the one in your suggestion I would suggest raw tcp isn’t the protocol to choose.

SeRiusRod · 3 September 2019 13:47

No, the tcp message sent by the gateway doesn't include that ↵ character and that is not configurable. It's the TCP node that it's adding them. If you see, there are payloads on that screenshot without them at the end. Apparently it's added when multiple messages are joined by the TCP node.

That's why I was asking what they could be, as if I'm removing \n and \r that should be something else. But I can't see the original char code through debug or console.

And that prevents me from making any processing function, because it's throwing me undefines.

SeRiusRod · 3 September 2019 14:41

Ok, I think I've got it under control. I had to make a function to get the code for that "return" character. The resulting function was sorter than I thought because a simple .trim() does the job also for this "return", and I think now I'm not loosing any messages. I'll be testing that.

About the recommendation for using a MQTT gateway on this server, I also think that your right, but still don't have a replacement for over the air updates to my devices.

drmibell · 4 September 2019 21:04

Glad to hear you've solved the problem. For the record, I think I had a similar issue with data coming in through the exec node. I was able to fix it using an optional configuration of the split node. You might give it a try.