Hey all,
I am trying to use the serial node to receive messages from our NEO-M8P GPS module. The messages are UBX (hex based messages) and seem to be coming in at a condensed rate. They repeat once a second, but the spacing between messages seem to be very short. Here is a snip from COM Port Toolkit (CPTK).
CPTK is splitting the messages, hence the separate lines for the packets.
This is a snip of my current serial port settings.
I have tried a silence timeout of down to 0.000001 ms. Of course this did not work but I just added a zero one at a time. I also tried stopping and restarting after each change to try and see if that worked.
We are also using the USB to receive another type of message (RTCM) which has a different set of start bytes. The RTCM are not present in the snip due to the module not having achieved its "survey-in" mode. After it reaches this point, It should start sending the RTCM messages in the same data feed. The different start bytes kills the possibility to use the on character setting as we will not know if the previous packet was RTCM or UBX. All of the messages types have a different length and can change length; no fixed length or start timeout splits. They all have CRC checksums as the end of the message and do not have a linefeed or carriage return to separate them; not "on character" splits. It is very frustrating...
If anyone has an idea of how to make this work or has a different idea, I am all ears.
Thank you for your time.
Looks like you first have to determine the protocol. Looking at the documentation, there is a lot to it. I read a note;
e.g. all UBX messages output on a UART port with a baud rate of 9600
Hey bakman2,
We are using USB at 115200. Also, the baud rate over UART can be changed in the modules settings in U-Center (U-Blox proprietary application).
On a side note, the modules output was sniffed using the USB interface. No serial -> USB converter.
Any other ideas?
1 Like
The latest version of serialport can wait for a start character - so you could tell it to wait for the B5 that seems to start every line, and also has a split on silence option which may help. It seems to be sending every 0.2 of a second not once per second judging by the timestamp differences.
Hey dceejay,
That would work great but the start character will change when RTCM messages are received as well. As I mentioned, they are not present in the snip because the module did not complete the survey in process at the time I took the snip.
You are right about the timestamps. I was using another module with different output frequency settings. Here is the correct snip. Still without the RTCM messages. I will see if I can get a snip of that shortly.
I have been using the "split on silence" feature but It does not seem to catch the "silence" (see image above). As mentioned in the original message, I tried reducing the timeout setting down to 0.000001 ms in steps to see when it would catch the silence. It does not catch the "burst" packet batches and CPTK is not clear if the messages have any pause in the messages at all.
Thanks for your suggestions.
Well worst case you should be able to say wait for 0 characters - and the node would then spit out one character at a time - and you would then have to re batch them in some manner... maybe looking for those trigger bytes in a loop then outputting the chunk once accumulated... not ideal but.
1 Like
Unfortunately I came to the same conclusion.
You have any links or previous flows that deal with this kind of problem?
I'll be working on this today and post when I have a working flow.
So this is some code I had about 3 years ago so may need some work...
context.buff = context.buff || "";
context.count = context.count || 0;
console.log(":",msg.payload,":");
// wait for an 'a' and reset the buffer and counter
if (msg.payload === "a") {
context.count = 0;
context.buff = "";
}
// add payload to the buffer and increase the count
context.buff = context.buff + msg.payload;
context.count += 1;
// when we reach 12 then output the whole message
if (context.count == 12) {
msg.topic = context.buff.slice(1,3); // return the first two chars as the topic
msg.payload = context.buff.slice(3); // return rest of the buffer as the payloadreturn msg;
return msg;
}
return null;
so looking at it it waits for a (so replace that with whatever you need) - then in this case just counts 12 - but you wait for another character...(or maybe the next B5 - which you then use to both send what you have and reset the accumulation for next time... so worst case you will always be 0.2 secs behind. (as you would be waiting for start of next to signal end of previous)...
Thank you for your example.
Your example really helped me develop a plan. Unfortunately, doing a byte by byte arrangement of the data with its complexities bogged down the CPU to 30-50% load. Usually without the byte by byte arrangement, it would hover around 1-2%; even with the same message rate/amount of messages.
For disclaimer, The previous code used Buffer.compare in the IF statements which I believe made the processing extremely difficult and sluggish.
Just for fun, here is the code I have come up with.
//Setup
//RTCM = 0xd300
//UBX = 0xb562
context.UBX1 = context.UBX1 || Buffer.from("b5", "hex");
context.RTCM1 = context.RTCM1 || Buffer.from("d3", "hex");
context.UBX2 = context.UBX2 || Buffer.from("62", "hex");
context.RTCM2 = context.RTCM2 || Buffer.from("00", "hex");
context.GPSMessageStart1 = context.GPSMessageStart1 || 0;
context.GPSMessageStart2 = context.GPSMessageStart2 || 0;
context.GPSMessageStart3 = context.GPSMessageStart3 || 0;
context.buff = context.buff || Buffer.from("", "hex");
var byte = Buffer.from(msg.payload,"hex");
var arr;
//Test messages
//console.log("context.GPSMessageStart1: " + context.GPSMessageStart1);
//console.log("context.GPSMessageStart2: " + context.GPSMessageStart2);
//console.log("context.GPSMessageStart3: " + context.GPSMessageStart3);
//node.log("context.buff[n]: " + context.buff[context.buff.length - 1]);
//console.log("byte: " + byte[0]);
//console.log("context.UBX1: " + context.UBX1[0]);
//console.log("context.RTCM1: " + context.RTCM1[0]);
// Check first Start Byte
if (byte[0] === context.UBX1[0] || byte[0] === context.RTCM1[0]) {
// If first time algorithum run with frst start byte
if (context.GPSMessageStart1 === 0 && context.GPSMessageStart2 === 0 && context.GPSMessageStart3 === 0) {
console.log("Start1.1");
context.GPSMessageStart1 = 1;
context.buff = byte;
}
// If second set of first start byte arrives
else if (context.GPSMessageStart1 === 1 && context.GPSMessageStart2 === 1 && context.GPSMessageStart3 === 0) {
console.log("Start2.1");
context.GPSMessageStart3 = 1;
arr = [context.buff, byte];
context.buff = Buffer.concat(arr);
}
}
// Check second start byte
else if ((byte[0] === context.UBX2[0] || byte[0] === context.RTCM2[0]) && (context.buff[context.buff.length - 1] === context.UBX1[0] || context.buff[context.buff.length - 1] === context.RTCM1[0])) {
// If first time algorithum run with second start byte
if (context.GPSMessageStart1 === 1 && context.GPSMessageStart2 === 0 && context.GPSMessageStart3 === 0) {
console.log("Start1.2");
context.GPSMessageStart2 = 1;
arr = [context.buff, byte];
context.buff = Buffer.concat(arr);
}
// If second set of second start byte arrives
else if (context.GPSMessageStart1 === 1 && context.GPSMessageStart2 === 1 && context.GPSMessageStart3 === 1) {
console.log("Start2.2");
context.GPSMessageStart1 = 1;
context.GPSMessageStart2 = 1;
context.GPSMessageStart3 = 0;
arr = [context.buff, byte];
context.buff = Buffer.concat(arr);
msg.payload = context.buff.slice(0, context.buff.length - 2)
context.buff = context.buff.slice(context.buff.length - 2, context.buff.length);
console.log("msg.payload: " + msg.payload.toString());
console.log("context.buff: " + context.buff.toString());
return msg;
}
}
// If middle of message arrives
else if (context.GPSMessageStart1 === 1 && context.GPSMessageStart2 === 1 && context.GPSMessageStart3 === 0) {
console.log("MID");
arr = [context.buff, byte];
context.buff = Buffer.concat(arr);
}
// Catch outliers
else {
console.log("????");
}
This code seems to work great on an I7-8700K. Running this on a RPI 3 B+ may have varying results. I will test later and run a TOP on it to see performance with byte access driven IF statements vs the Buffer.compare I was using previously.
Thank you for you help.
