Extracting my valuable data from a msg.payload?

General
1

Hi

I’m sending, with USB trough serial, data from an arduino to my computer where I have node-red running.
I can see the entire messages coming in the debug and with all the data to be directed to a file where they are recorded. This works good.
But I’d like to extract some of the data to be redirected to a node-red_dashboard graph in real time. And I have no idea how to extract the valuable part of the incoming message (msg.payload) to do what I want.
I can notice, from what I send from the arduino, each “\n” makes a new message to node-red.
Each messages from my arduino are like pair value with the “name of the variable” = x separated with “\t” , like so !
variable_1 = x \t variable_22 = y \t variable_3 = z
I guess, I can add some special character to “highlight” my valuable data and extract it !

Or maybe I could get the second or third data by counting ?
Which would be the best or at least a good practice to do ?
In case it’s not obvious, I’m a beginner !
Any advices ?

2

Send the data to a debug node and then post a screenshot of the data so we can see exactly what is there.

3

Hi @hary

From what you’ve shared, you have:

variable_1 = x \t variable_22 = y \t variable_3 = z

Separating the variables by a tab character is a good start. In a Function node, you can split that up using the String.split function:

var myVars = msg.payload.split("\t");

That will give you an array containing the variables:

[
   "variable_1 = x",
   "variable_2 = y",
   "variable_3 = z"
]

The next task is to split each of those up to the variable name and the value. Again we can use the split function:

var result = {};
myVars.forEach(function(v) {
   // The following split uses ' = ' - because your example output appears to
   // have spaces around the equals sign.
   var parts = v.split(" = ");
   result[parts[0]] = parts[1];
});

That will give you result which is an object that looks like:

{
   "variable_1"  : "x",
   "variable_2"  : "y",
   "variable_3"  : "z"
}

What you then do with result will depend on what else you want the flow to do. For example, you could just assign it to msg.payload and return:

msg.payload = result;
return msg;

Then, in a later node you can use msg.payload.variable_1 etc to access individual values.

All of this is based on the information you’ve provided - it makes a few assumptions on the format. So if this doesn’t work precisely as-is, then it may need some tweaking to match your exact data format. But I hope this gives you enough of a steer to get you going.

1 Like
4

Another idea:

My thinking is to change the string into something that will look like JSON and then parse it, like this:

// Convert the string: “a = 1 \t b = 2 \t c = 3”
// into the string: ‘{“a”: “1”,“b”: “2”,“c”: “3”}’
// so we can pass it to JSON.parse(’{“a”: “1”,“b”: “2”,“c”: “3”}’)
// and get a JSON object
t1 = “a = 1 \t b = 2 \t c = 3”; // this should come from msg.payload
t2 = t1.replace(/ /g, ‘"’); // replace the spaces with double quotes
t3 = t2.replace(/=/g, ‘: ‘); // replace the equal with a colon
t4 = t3.replace(/\t/g, ‘,’); / replace the tab (\t) with a comma and space
nJson = JSON.parse(’{"’ + t4 + ‘"}’); // Parse the string to make it a JSON object
// nJsin is now a JSON object and you can access the variables as such: nJson.a
// which will return 1

But as Colin said, it would be good to see the data and also an explanation of how you want to access it.

5

Arr !
Thanks everybody for your help, but I think I need to follow some lesson first.
Plus I’m a bit lost.
Have the function in node-red to be written in node.js or JavaScript ? and where to find tutorial for this ?
Is there an easy way to see my result ? are my function being compiled or are they just run as python script ? If so, is there a way to work and learn like with python ?
Or if they’re compiled, what should I use to learn ?

I’m most used of C/C++ or Python.

@knolleary and @ncherry
As I’m programming the Arduino, I can still format my data more less the way I want before sending them trying to simplify their use later .
Even maybe in JSON format as I understand, it’s kind of a standard these days !

6

@hary

The Function node in the palette lets you write JavaScript that gets run against the messages it receives. The docs describe what you can do there - https://nodered.org/docs/writing-functions - but it does assume some familiarity with JavaScript.

From what you’ve said, you have a Serial In node wired to a Debug node. The next step is to insert a Function node between those two. Edit that node and paste in the following code (based on what I described in my original reply):

// 'msg' is the message object the Function node receives
var myVars = msg.payload.split("\t");
var result = {};
myVars.forEach(function(v) {
   // The following split uses ' = ' - because your example output appears to
   // have spaces around the equals sign.
   var parts = v.split(" = ");
   result[parts[0]] = parts[1];
});
msg.payload = result;
// The Function node returns the message to be sent to the next node(s) in the flow
return msg;

As I said before, I believe that will work for the format you are currently sending - give or take. If you changed the format to be valid JSON then it would be much easier to handle - you would add a JSON node after the Serial In node to parse the data and convert it to an Object you can use to access the individual values of.

Some more reading on working with messages: https://nodered.org/docs/user-guide/messages

1 Like
7

First no problem, we all started somewhere. I’m still learning and I’m using a ton of languages (syntax gets me confused often while switching).

Hmm, Node-Red is built on top of Node.js, so it the v8 version of JavaScript. So learning Node.js should work like node-red. I’ll often pop into a shell, run node and experiment with the JavaScript. I did that for my code above.

I’m not an expert and I get things wrong, so take that into account when using my advice. I found the asynchronous nature of Node-Red/Node,js a new learning experience and it still bytes me today. So for and while loops may not work as expected. That’s where call back come in. Also note that often (always?) the msg.payload is a string. So to convert from a string that looks like JSON. you use JSON.parse(msg.payload). There should be Node-Red tutorials around.

I know others will chime in with advice. Good Luck.

8

mistake
See my next message please

9

Hi.

I think I’m getting closer.
To lighten the data transmitted other usb, I only send the numbers and string that are important in a same message as :
< \t 125489 \t 781 \t 24.50 \t 26.75 \t Tc_K_ON \t 2978 \t >
not any more like :
variable_1 = x \t variable_22 = y \t variable_3 = z
only :
< \t x \t y \t z\t >

There are others messages sent, but they don’t have the “<” at their beginning.

Each messages seams to be separated by “\n” I think, the arduino 's Serial.println("") function.

If the message has a “<”, it’s redirected (from a switch node) to a specific function node where a function splits it into pieces like so :

var outputMsgs =[];
var myVars = msg.payload.split("\t");
for (var w in myVars ){
      outputMsgs.push({payload:myVars[w]});
}
outputMsgs  = outputMsgs.shift();
return [outputMsgs ];

but by that time the first element of the array ("<") is disturbing the procedure and I would like to get rid of it.
As you can see, I’m using the var.shift() function but still, the “<” is displayed !
Does my code seem to be correct ?

10

you have to escape the backslash in the split like this:
var myVars = msg.payload.split("\\t");

11

The right code is :

var outputMsgs =[];
var myVars = msg.payload.split("\t");
for (var w in myVars ){
      outputMsgs.push({payload:myVars[w]});
}
outputMsgs.shift();
return [outputMsgs ];

Now it sends one message for each value but on the same output.
How can I send each message on its own output node ? (one message by output, 5 messages => 5 outputs, I know the number of my outputs which is constant)

12

Covered in the docs https://nodered.org/docs/writing-functions#multiple-outputs

13

Interesting because when I use this with your data, nothing comes out....

14

In order for his sample data to work, you will probably need to replace the strings \t with an actual tab character…

15

@ukmoose
Yes I manage to make it working now ! Thks

To others, the data are coming from an arduino with the Serial.print() function on USB port.

16

You know, this is why when I ask a question, I create a small version of my issue with a copy of the data in it so someone else can help me without doing much work.

17

Hi there, I just started with node-red a month ago and I see the same problem here that I have with my code. Can someone help with a very similar setup? I used what you have been talking about here replacing variable_1 = x \t with connect-id : x and sco-id : y and so on. I use ":" instead of "=" but I get:
3/10/2020, 6:19:29 PMnode: 959a0e80.8839aconnect-id, sco-id, asset-id : msg.payload : Object
{ empty }
I am attempting to read in a log file to extract the data from: connect-id, sco-id, asset-id. Here is the log file:
3 rows retrieved, max rows: 10; local time: Tue Mar 03 18:59:09 UTC 2020
execution start: 2020-03-03 18:30:22.153 UTC
{"compile-ms":"9","exec-ms":"94","server-rec'd":"1583260222134","statusDetail":"","status":"DEFAULT","host":"python-85fb5dbf54-t8tbs","total-ms":"107"}

2020-03-03 18:30:22.153 INFO 451: starting: api replaySession, level: TRACE
2020-03-03 18:30:22.165 DEBUG starting execution
2020-03-03 18:30:22.165 DEBUG replaySession
2020-03-03 18:30:22.166 DEBUG connect-id : r6GJIPvyVflqXpJd
2020-03-03 18:30:22.166 DEBUG sco-id : 11692
2020-03-03 18:30:22.166 DEBUG asset-id : 58754
2020-03-03 18:30:22.235 TRACE PS: SELECT date_start, date_end, attendees_list, transcripts, sco_info, meeting_permissions_list, server FROM session_log WHERE connect_id = ? AND sco_id = ? AND asset_id = ? /* ec81aa30-5d7c-11ea-8ee0-53b4772b249d replaySession */
  -> args: ('r6GJIPvyVflqXpJd', 11692, 58754)
  -> consistency: ONE
2020-03-03 18:30:22.258 DEBUG Failed to find meeting [r6GJIPvyVflqXpJd / 11692 / 58754] in session log
2020-03-03 18:30:22.259 DEBUG 1 datalake keyspace queries
2020-03-03 18:30:22.259 DEBUG executed without exceptions
18

I would feed the log into a split node to output one line at a time. Next would come a switch node set to stop after first match. It would have three ‘contains’ conditions looking for ‘connect-id’, ‘sco-id’ and ‘asset-id’ and each output going a new flow leg where I’d paste the line and get the value and use it however you would like.

19

Thank you for the input. Greatly appreciated. What I have is the following you could take a look. The result part section seems to be where I am having trouble with:
msg.filename="meeting.log";
var myVars = msg.payload.split("\t");

//That will give me an array containing the variables:
[
"connect-id : x",
"sco-id : y",
"asset-id : z"
]

//The next task is to split each of those up to the variable name and the value. Again we can use the split function:
var result = {};
myVars.forEach(function(v) {
// The following split uses ' : ' - because your example output
var parts = v.split(" : ");
result[parts[0]] = parts[1];
});
msg.payload = result;
return msg;

20

What does the input file look like?