Speed up serial communication

DirkB · 7 June 2023 13:11

I want to retrieve data from my sensor. Baud rate is fix at 9600.
My flow works so far, but the communication stops after 1 second because my sensor runs into a WatchDog and restarts.
Therefore, I would like to design the flow and especially the node function in such a way that the 112 bytes can be received in less than 1 second. Attempts with a for loop were unsuccessful, as I can only use a node.send from a function but do not have a node.receive function to receive a new msg object, for example.
Does anyone have any suggestions on how I can optimise the code to make it run faster - or maybe a completely new approach?

Here my flow:

[
    {
        "id": "dd740ba2282df4a6",
        "type": "tab",
        "label": "Messfühler auslesen",
        "disabled": false,
        "info": ""
    },
    {
        "id": "dde8b67b44289268",
        "type": "inject",
        "z": "dd740ba2282df4a6",
        "name": "sende 0xCB",
        "props": [
            {
                "p": "payload"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "[\"0xCB\"]",
        "payloadType": "bin",
        "x": 210,
        "y": 140,
        "wires": [
            [
                "d26ab8ac82582542"
            ]
        ]
    },
    {
        "id": "0b785fa2f97ab2ee",
        "type": "function",
        "z": "dd740ba2282df4a6",
        "name": "",
        "func": "var ACK = Buffer.from([0x13]);    // When sensor is ready to send data it sends ACK in response to 0xCB\nvar REQ = Buffer.from([0x11]);    // Request to sensor = REQ to send another byte\nvar datensatz = flow.get('datensatz') || new Array(111);\nvar counter = flow.get('counter');                              \n\nif (Buffer.compare(msg.payload, ACK) === 0 && counter == !0) {   // when the ACK arrives, send a REQ to output 1\n    node.warn(\"ACK received - send REQ\");\n\n    msg.counter = counter;\n    msg.payload = REQ;\n\n    return [msg, null];\n}\n\nelse {\n\n    if (counter < 111) {               // If not all data are there yet\n\n        datensatz[counter] = msg.payload;   // Save received data in data set at counter position\n        flow.set('datensatz', datensatz);\n\n        counter++;                         \n        flow.set('counter', counter);\n\n        msg.counter = counter;\n        msg.payload = REQ;\n\n        return [msg, null];             // send REQ to output 1\n\n    }\n\n    else {                                             // counter >= 112\n\n        node.warn(\"send Datensatz to output 2\");\n\n        flow.set('counter', 0);                       // reset counter\n        msg.counter = counter;\n        msg.payload = flow.get('datensatz');         // copy datensatz to msg\n\n        return [null, msg];                          // output 2 gets compete datensatz\n\n    }\n\n}",
        "outputs": 2,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 420,
        "y": 280,
        "wires": [
            [
                "d26ab8ac82582542",
                "34c82f95247b334e"
            ],
            [
                "8c0d6351e4d21748",
                "275df45533ca8e0a"
            ]
        ]
    },
    {
        "id": "8c0d6351e4d21748",
        "type": "debug",
        "z": "dd740ba2282df4a6",
        "name": "datensatz",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": true,
        "complete": "payload",
        "targetType": "msg",
        "statusVal": "payload",
        "statusType": "msg",
        "x": 660,
        "y": 280,
        "wires": []
    },
    {
        "id": "d26ab8ac82582542",
        "type": "serial request",
        "z": "dd740ba2282df4a6",
        "name": "",
        "serial": "dccc90be94647fa3",
        "x": 430,
        "y": 140,
        "wires": [
            [
                "6ad5dfb0e9991354",
                "0b785fa2f97ab2ee"
            ]
        ]
    },
    {
        "id": "6ad5dfb0e9991354",
        "type": "debug",
        "z": "dd740ba2282df4a6",
        "name": "debug 181",
        "active": false,
        "tosidebar": true,
        "console": false,
        "tostatus": true,
        "complete": "payload",
        "targetType": "msg",
        "statusVal": "payload",
        "statusType": "msg",
        "x": 670,
        "y": 120,
        "wires": []
    },
    {
        "id": "34c82f95247b334e",
        "type": "debug",
        "z": "dd740ba2282df4a6",
        "name": "",
        "active": false,
        "tosidebar": true,
        "console": false,
        "tostatus": true,
        "complete": "counter",
        "targetType": "msg",
        "statusVal": "counter",
        "statusType": "msg",
        "x": 670,
        "y": 200,
        "wires": []
    },
    {
        "id": "275df45533ca8e0a",
        "type": "debug",
        "z": "dd740ba2282df4a6",
        "name": "",
        "active": false,
        "tosidebar": true,
        "console": false,
        "tostatus": true,
        "complete": "counter",
        "targetType": "msg",
        "statusVal": "counter",
        "statusType": "msg",
        "x": 670,
        "y": 360,
        "wires": []
    },
    {
        "id": "4f1a9ec07a2ddef1",
        "type": "config",
        "z": "dd740ba2282df4a6",
        "name": "",
        "properties": [
            {
                "p": "datensatz",
                "pt": "flow",
                "to": "[ ]",
                "tot": "json"
            },
            {
                "p": "counter",
                "pt": "flow",
                "to": "0",
                "tot": "num"
            },
            {
                "p": "DatenErhalten",
                "pt": "flow",
                "to": "false",
                "tot": "bool"
            }
        ],
        "active": true,
        "x": 130,
        "y": 60,
        "wires": []
    },
    {
        "id": "dccc90be94647fa3",
        "type": "serial-port",
        "serialport": "COM4",
        "serialbaud": "9600",
        "databits": "8",
        "parity": "none",
        "stopbits": "1",
        "waitfor": "",
        "dtr": "none",
        "rts": "none",
        "cts": "none",
        "dsr": "none",
        "newline": "1",
        "bin": "bin",
        "out": "count",
        "addchar": "",
        "responsetimeout": "1000"
    }
]

TotallyInformation · 7 June 2023 16:01

112 bytes of data is 112*8 = 896 bits and at 9600 bits per second, that should complete in 0.0933333333 of a second! (if I got my high-school maths right!).

So if it is taking >1 second and if you are getting watchdog errors, I would suggest that your serial port either is FUBAR'd or the settings on it are incorrect.

DirkB · 12 June 2023 05:28

Hello Julien,

thank you for your answer.
However, I am absolutely sure that the Com interface is configured correctly and also works correctly as I receive 108 or 109 bytes from my device depending on the situation.
I rather assume that my function with which I request the bytes and then store them in an array has a too high runtime and therefore the watch dog limit of 1 sec is exceeded. Since I am not a JavaScript pro, it is possible that there is a more efficient way.
Is there possibly another approach to manage the communication? The start signal of 0xCB must be followed by an ACK of 0x13. When this comes, each byte can be requested with REQ = 0x11.
Just by the way, as a non-native speaker, the word FUBAR'd was completely unknown to me, but Google could help ;-).
Here is the code of the function node that handles the communication for your information.
I look forward to any suggestions.

var ACK = Buffer.from([0x13]);    // When sensor is ready to send data it sends ACK in response to 0xCB
var REQ = Buffer.from([0x11]);    // Request to sensor = REQ to send another byte
var datensatz = flow.get('datensatz') || new Array(111);
var counter = flow.get('counter');                              

if (Buffer.compare(msg.payload, ACK) === 0 && counter ==! 0) {   // when the ACK arrives, send a REQ to output 1
    node.warn("ACK received - send REQ");

    msg.counter = counter;
    msg.payload = REQ;

    return [msg, null];
}

else {

    if (counter < 111) {               // If not all data are there yet

        datensatz[counter] = msg.payload;   // Save received data in data set at counter position
        flow.set('datensatz', datensatz);

        counter++;                         
        flow.set('counter', counter);

        msg.counter = counter;
        msg.payload = REQ;

        return [msg, null];             // send REQ to output 1

    }

    else {                                             // counter >= 112

        node.warn("send Datensatz to output 2");

        flow.set('counter', 0);                       // reset counter
        msg.counter = counter;
        msg.payload = flow.get('datensatz');         // copy datensatz to msg

        return [null, msg];                          // output 2 gets compete datensatz

    }

}

Steve-Mcl · 12 June 2023 08:49

It seems strange that EVERY byte requires a REQ (0x11).

Normally, with serial comms, 1 request is made, all relevant data is returned then you send an ACK

Do you have a manual for the device?

If not, have a play! Try setting the serial config to return a buffer after a silence less than 1s e.g

DirkB · 12 June 2023 09:11

Hello Steve,

Yes, I am 100% sure that every byte must be requested with REQ. I am in close contact with the author of the firmware of the device I want to read.
Is it possible that my function takes too long to go through the loops?
My environment is:
Node-RED version: v3.0.2
Node.js version: v20.2.0
Windows_NT 10.0.19045 x64 LE

I am grateful for any advice to get the communication working.

Steve-Mcl · 12 June 2023 09:38

That is not great (IMHO) - never seen anything like that before.

Many protocols I have used have specific request commands and known terminators to help a user consume the data without having to request every character!

As for your code, it is pretty tight and should operate at "full speed" (as fast as it can go) but let me say this (some assumptions made but it should be close):

There are 112 REQ, 112 responses and some ACKs

one bit takes 1/9600 s= 0.00014 s
8+1+1=10 bits takes 10x1/9600=1.04 ms

then (1.04+1.04)*112 = 232.96ms - lets round that up to 250ms (for ACKs and easy numbers)

That leaves 750ms for processing time

750/112 = 6.6ms overhead (flows, nodes, UART, OS etc) per operation

This is gonna be tight.

Perhaps you could/should suggest to the firmware author to improve this? Perhaps chose a known, popular, open lightweight protocol instead of inventing a new one? Examples: Embedded Systems/Common Protocols - Wikibooks, open books for an open world

Disclaimer: My maths may be wrong

DirkB · 12 June 2023 10:44

For various reasons, the firmware of the unit cannot be changed.
I may have thought of another approach. Is it conceivable to write a small, crisp programme in Pyton, for example, which fetches the 112 bytes at the lower level, i.e. in the approx. 200ms, and then transfers them to node-red?
Would it be possible to call such a small subroutine from node-red?

mudwalker · 12 June 2023 10:51

Yes, using an Exec Node. Use the Search function to find < Python exec node >

Steve-Mcl · 12 June 2023 10:54

Yes, but I would not personally want to execute python over and over again.

If I were to take this route then I would have the python application run continuously and simply push the values to node-red using MQTT.

DirkB · 12 June 2023 13:11

Hi Steve,

Thank you very much for your valuable tips on this topic.
I was able to arrange with the firmware programmer to have the Watch Dog re-triggered. Now there is no longer this time lapse as long as the device is in the send loop.
For me, this is the best solution (instead of Pyton etc...) and my communication with the device now works as intended.

Thanks again.

TotallyInformation · 12 June 2023 18:27

Some last thoughts in case this crops up again.

If forced into this situation, I would take one of 2 approaches I think:

As Steve says, create a small daemon service that listens to MQTT on one side and collates the data from the device on the other.
Put a real-time processing device between the hardware and the server running Node-RED. Something like a cheap ESP32 which uses RTOS. It would be able to respond easily in time and would act as the intermediary. Effectively acting the same as your Python programme but with dedicated hardware that is still easy to programme and low cost.

system · 26 June 2023 18:28

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Setting the timer in node-red function node for few milliseconds General	17	1797	22 August 2020
Struggeling with function to devide serial node response General	4	141	8 March 2023
Serial data from arduino General	3	775	4 March 2019
Stuck with delay node General	2	551	2 August 2018
Sending Struct Function to Node-Red via serial General	21	1625	28 June 2020

Speed up serial communication

Related topics