Tcp in node loosing/stacking messages

[SeRiusRod]

I was doing some modifications to my system when I found that I was losing mqtt messages. I debugged the issue to the first node, a tcp input.
I use mysensors on my devices, and it has a network gateway that publishes the messages on the home network. I connect to the gateway with a tcp in node, and then transform and republish to mqtt.
Mysensors has a tool that allows me to debug the output from the gateway, and I could see the message following.

I've found that sometimes the tcp in node joins several messages separated by \n and possibly \r into the same payload. The following nodes don't expect this and messages are lost.
Also during series of messages, like when a device presents itself and it's child sensors, the tcp node drops some msgs.

My system runs in docker containers in a photonOS VM on a big esxi server.

Is this behavior normal?
Can someone help me debug or some this problem?

[kuema]

With TCP one should never rely on proper separation of the incoming bytes. Even if the message was sent as one chunk of data, it is not guaranteed it arrives as one chunk at a time due to segmentation, fragmentation, OS buffers... so you can receive only parts of your message, or even multiple message in one buffer, as you have observed.

However, the order of the bytes is guaranteed to be the same, of course.

[dceejay]

yes - that can happen - ... as @kuema just said

If you know they are strings then you can tell the node to expect strings and split them - but in general you probably can't

[kuema]

Exactly. Depending on your protocol, you would need to parse and split the bytes at the appropriate boundaries.

[kuema]

I do not know MySensors, but maybe it supports sending your data via MQTT directly?

[SeRiusRod]

Yes, it supports a MQTT gateway, but then I'll lose the possibility of using that tool, that only works with usb and lan gateways.
The tool allows to inspect and manage the topology, ask nodes for heartbeat, presentation, etc and allows OTA to them. And debug communications, of course.

I've modified the debug node to output to the console and this is what I've got now:

3 Sep 13:49:15 - [info] [debug:WIFI input] 
1;1;1;0;17;2818.8
3 Sep 13:49:15 - [info] [debug:WIFI input] 
1;1;1;0;18;1.804
3 Sep 13:49:15 - [info] [debug:WIFI input] 
1;1;1;0;24;1804
3 Sep 13:49:15 - [info] [debug:WIFI input] 
1;2;1;0;17;1.3
1;3;1;0;17;2429.2
3 Sep 13:50:19 - [info] [debug:WIFI input] 
1;1;1;0;17;1699.3
1;1;1;0;18;1.842
1;1;1;0;24;1842
1;2;1;0;17;0.7
1;3;1;0;17;1568.5
3 Sep 13:50:19 - [info] [debug:WIFI input] 
1;4;1;0;17;94.9

What we could name "topic" has always five elements, and the sixth is the payload. What I don't know is if between msgs and at the end it's adding \r\n or only \n and how many.
I've made the following function node, but I'm having problems with it. Is throwing "undefined" replace errors every other output.

try {
    const messages = msg.payload.replace(/(\\r\\n|\\n|\\r)/gm, "").split('\n');
    var topic = "", payload = "";
    
    messages.forEach( function(message, idx, array) {
        var parts = message.split(';');
        topic = 'mys-in/' + parts[0] + '/' + parts[1] + '/' + parts[2] + '/' + parts[3] + '/' + parts[4];
        payload = parts[5].replace(/(\\r\\n|\\n|\\r)/gm, "").trim();
        node.send({topic: topic, payload: payload});
    });
}
catch(err) {
    node.error(err, msg);
}
finally {
    return null;
}

[SeRiusRod]

This "stacking" of messages seems to occur and become worse depending on the workload of the nodes following it. I explain: If the following node is a function with only a node.debug(msg.payload);, it occurs now an then. But as the node does more things, it starts to stack, more and more.

That may be normal, but I don't find it right. I can imagine a big series manufacturing machine dropping bottle boxes or even crashing and making a pile of them because is driven with node-red and it dropped only one detection message.

... and I only have 3 devices in this testing network, only one is on ATM.

PS. Just one more thing: I've replaced "\n" and "\r" to "*" and "#", and then output to debug in a function. When debug outputs "1;1;1;0;18;2.319*1;1;1;0;24;2319↵" what does that latest char means? Should not have anything more.

[ukmoose]

Just one more thing: I've replaced "\n" and "\r" to " " and "#", and then output to debug in a function. When debug outputs "1;1;1;0;18;2.319 1;1;1;0;24;2319↵" what does that latest char means?

So if you were using newline/carriage return as the end of line character and now you have decided not to. The string now includes any newline / carriage return characters ↵

If you are running a machine like the one in your suggestion I would suggest raw tcp isn’t the protocol to choose.

[SeRiusRod]

So if you were using newline/carriage return as the end of line character and now you have decided not to. The string now includes any newline / carriage return characters ↵

No, the tcp message sent by the gateway doesn't include that ↵ character and that is not configurable. It's the TCP node that it's adding them. If you see, there are payloads on that screenshot without them at the end. Apparently it's added when multiple messages are joined by the TCP node.

That's why I was asking what they could be, as if I'm removing \n and \r that should be something else. But I can't see the original char code through debug or console.

And that prevents me from making any processing function, because it's throwing me undefines.

[SeRiusRod]

Ok, I think I've got it under control. I had to make a function to get the code for that "return" character. The resulting function was sorter than I thought because a simple .trim() does the job also for this "return", and I think now I'm not loosing any messages. I'll be testing that.

About the recommendation for using a MQTT gateway on this server, I also think that your right, but still don't have a replacement for over the air updates to my devices.

[drmibell]

Ok, I think I've got it under control.

Glad to hear you've solved the problem. For the record, I think I had a similar issue with data coming in through the exec node. I was able to fix it using an optional configuration of the split node. You might give it a try.

[garybia]

I disagree with what I think you said. This is cut from Wiki

The Transmission Control Protocol ( TCP ) is one of the main protocols of the Internet protocol suite. It originated in the initial network implementation in which it complemented the Internet Protocol (IP). Therefore, the entire suite is commonly referred to as TCP/IP . TCP provides reliable, ordered, and error-checked delivery of a stream of bytes between applications running on hosts communicating via an IP network.

UDP does not

[kuema]

It says exactly what I tried to explain. I think you misread my post. The order is guaranteed by TCP, thus reliable and error-checked, but the byte stream can reach the application in arbitrarily sized chunks. So you have to take care of that when implementing your application protocols.

UDP is fire-and-forget, it is not guaranteed that you get all datagrams, but when you do, you receive it as it was sent, not as a stream of chunks. However, you have to take care of order and duplicates in the application yourself.