Manipulate Vector for Dashboard Chart node

[Sasko]

Hey Community, this is my first post and I hope, that you can help me

For a task I want to take (battery-) data from a matlab/simulink simulation and sort and output it in the node-red dashboard via the chart node. From Matlab I get a app. 20000 x 1 Vector, the original plan was to safe the data in a database and process it from there, but one thing after another. So I safed the data in an excel .xlsx and used the alafile-in node with a timestamp (temporarily), to trigger the event. The Output is an array of objects. When I split the output 2 times, I get 20k simgle values.

Now I searched the dashboard docu for the chart node, which needs a special format for stored data:

[{
"series": ["A", "B", "C"],
"data": [
    [{ "x": 1504029632890, "y": 5 },
     { "x": 1504029636001, "y": 4 },
     { "x": 1504029638656, "y": 2 }
    ],
    [{ "x": 1504029633514, "y": 6 },
     { "x": 1504029636622, "y": 7 },
     { "x": 1504029639539, "y": 6 }
    ],
    [{ "x": 1504029634400, "y": 7 },
     { "x": 1504029637959, "y": 7 },
     { "x": 1504029640317, "y": 7 }
    ]
],
"labels": [""]
}]

Can someone help me with manipulating the output with a function node, so that I get the data in this format? I'm actually not that good in javascript, but I'm currently taking a course to close that gap. I've read the javascript array docu and the working with messages docu, but I couldn't figure out how to solve it.

I've tried a node in that way:

var array =
array = array.push(msg.payload)
return msg

Thank you.

[TotallyInformation]

Hi, to get the structure you've shown, you need an array containing an object containing series, data and labels properties.

With the data property being an array of arrays of objects.

Indeed, the data you are showing in the chart example is for, say, a line chart with date/time on the x-axis and the values on the y-axis. However, it sounds like you only have the values in your source data so you would have to use an incrementing number for your x-axis?

It might help to share an extract of the source data.

[Steve-Mcl]

Hi.

As you have provided no example data its impossible to demonstrate changing your data into the correct format.

All I can do is offer some advice in the form of an annotated screenshot and sample code...

FOREWORD - it looks far more complicated than it actually is. Also, the code is longhand to show you what is happening

//1. lets create a variable to hold the finished article...
let chartData = [];  //See #1 - create an empty array

//2. lets create a variable to hold the everything...
let element1 = {};  //create element1 as an empty object {}

//3, now add the .property "series" to the element1 {object}
element1.series = []; // add a .property called series of type [array]
element1.series.push("A"); //add 1st array element "A"
element1.series.push("B"); //add another array element "B"
element1.series.push("C"); //add another array element "C"

//4. now add the .property "data" to the element1 {object}
element1.data = []; // add a .property called data of type [array]



/*********** NEXT - ADDING DATA  ***************************
  NORMALLY, ADDING DATA WOULD BE DYNAMIC 
  PERHAPS LOOPING THROUGH AN ARRAY
  THIS EXAMPLE SHOWS ADDING HARD CODED VALUES
************************************************************/
//4b. now we add 3 arrays to the data [array]
var dataElement1 = [];
var dataElement2 = [];
var dataElement3 = [];

//4c. now add the 3 {objects} inside each of the dataElements
dataElement1.push( { "x": 1504029632890,  "y": 5 }); //add hard coded x & y values
dataElement1.push( { "x": 1504029636001,  "y": 4 }); //add hard coded x & y values
dataElement1.push( { "x": 1504029638656,  "y": 2 }); //add hard coded x & y values

dataElement2.push( { "x": 1504029633514,  "y": 6 }); //add hard coded x & y values
dataElement2.push( { "x": 1504029636622,  "y": 7 }); //add hard coded x & y values
dataElement2.push( { "x": 1504029639539,  "y": 6 }); //add hard coded x & y values

dataElement3.push( { "x": 1504029634400,  "y": 7 }); //add hard coded x & y values
dataElement3.push( { "x": 1504029637959,  "y": 7 }); //add hard coded x & y values
dataElement3.push( { "x": 1504029640317,  "y": 7 }); //add hard coded x & y values

//now we need to push the dataelements into the data [array]
element1.data.push(dataElement1);
element1.data.push(dataElement2);
element1.data.push(dataElement3);

/*******************************************/

//5. now add a labels [array] to element1
element1.labels = []; // add a .property called labels of type [array]
element1.labels.push(""); //add an empty string to the "labels" [array]

//Now add the element1 [array] into the chartData [array]
chartData.push(element1)

//Lastly, return the chartData as the payload 
msg.payload = chartData;
return msg;

Here you can see it in action...

[Sasko]

This is the array I get as the raw data. I cut out some data, actually it is all nearly the same and I believe, you don't need to see 201069 times approximately 50

[{"SoC":50},{"SoC":50},{"SoC":50},{"SoC":49.99999999131063},{"SoC":49.999999982621254},{"SoC":49.9999999739319},{"SoC":49.99999993267416},{"SoC":49.99999989141649},{"SoC":49.999999850158915},{"SoC":49.99999980890143},{"SoC":49.99999972202188},{"SoC":49.99999963514274},{"SoC":49.999999548264014},{"SoC":49.99999946138568},
...
,{"SoC":49.56632264709009},{"SoC":49.5654996665415},{"SoC":49.56406416665728},{"SoC":49.56263056634608},{"SoC":49.56227004079587},{"SoC":49.56191051530844},{"SoC":49.561552000869824},{"SoC":49.560321890900724},{"SoC":49.55910351391681},{"SoC":49.557896715628516},{"SoC":49.55752284144956},{"SoC":49.557148276397136},{"SoC":49.556772841485106},{"SoC":49.556396507202095},{"SoC":49.555273060265584},{"SoC":49.55414174966797},{"SoC":49.553002065664366},{"SoC":49.552075711173885}]

So this is an array filled with objects, indeed.

So with your code, I would do something like this, right:

let chartData =;
let element1 = {};

element1.series = ;
element1.series.push("SoC");

element1.data = ;

So far, so good. But how do I feed the data coming from the alafile-in node in my data object, maybe even dynamically ?

[Steve-Mcl]

Loop through the array, and since you don't have a timestamp set x to an incrementing value.

Something like this (untested as I'm on my phone)...

var x = 0;
var data = msg.payload
var len = data.length
for(x = 0; x < len; x++){
  element1.data.push({
    x : x,
    y : data[x].SoC
  })
}
//5. now add a labels [array] to element1
element1.labels = []; // add a .property called labels of type [array]
element1.labels.push(""); //add an empty string to the "labels" [array]

//Now add the element1 [array] into the chartData [array]
chartData.push(element1)

//Lastly, return the chartData as the payload 
msg.payload = chartData;
return msg;

Edit....
That is a LOT of data. Don't be surprised if it's as slow as a slow gin on a slow night up on mount slowden!

[Sasko]

Nice, it's working

Here is the Output:

[{"series":["SoC"],"data":[{"x":0,"y":50},{"x":1,"y":50},{"x":2,"y":50},{"x":3,"y":49.99999999131063},{"x":4,"y":49.999999982621254},{"x":5,"y":49.9999999739319},{"x":6,"y":49.99999993267416},{"x":7,"y":49.99999989141649},{"x":8,"y":49.999999850158915},{"x":9,"y":49.99999980890143},{"x":10,"y":49.99999972202188},{"x":11,"y":49.99999963514274},{"x":12,"y":49.999999548264014},{"x":13,"y":49.99999946138568},{"x":14,"y":49.999999374507745},{"x":15,"y":49.99999927512818},{"x":16,"y":49.999999175749146},{"x":17,"y":49.99999907637063},{"x":18,"y":49.99999897699263},
...
,{"x":989,"y":49.560321890900724},{"x":990,"y":49.55910351391681},{"x":991,"y":49.557896715628516},{"x":992,"y":49.55752284144956},{"x":993,"y":49.557148276397136},{"x":994,"y":49.556772841485106},{"x":995,"y":49.556396507202095},{"x":996,"y":49.555273060265584},{"x":997,"y":49.55414174966797},{"x":998,"y":49.553002065664366},{"x":999,"y":49.552075711173885}],"labels":[""]}]

But there is another Problem, like you have mentioned:

it's slow. not only slow, my browser dies. Well not actually, but it says a website is slowing down my browser and I can wait or stop it, but it doesnt matter how long I wait. I think that the dashboard nodes are not made to handle that much data.

But I think, I will recreate the whole process, because originally it was supposed to work with data from a database. I think live data will actually work much better with node-red, one of the reasons for choosing the framework.

But anyway: Thank you very much for solving the problem so quickly and uncomplicated

[Steve-Mcl]

You could limit it to say 500 items.

Alternatively, you could create a running average and peak per 1000 samples (or similar summarisation)

However, you are correct in that either a DB query or live is better.

[TotallyInformation]

Just note that there is little point in having a 20k point line chart unless you have a rather crazy monitor that will display 20k points

So in reality, you either need to reduce the data (best done in SQL in the db) or just take a more reasonable number of points. In principle, you should never chart more points than you have pixels to display them. In reality, any more than a fraction of your pixel number is likely to be unusable in any meaningful way.

[Sasko]

The Data is from a Simulink Simulation, so I cannot change or cut it.

Do you guys have any documentations I could read to get data from a database ?

[bakman2]

You can make the loop shorter, ie; last 50 elements.

[Steve-Mcl]

or he could ...

@Steve-Mcl

limit it to say 500 items.

or

@Steve-Mcl

create a running average and peak per 1000 samples (or similar summarisation)

@Sasko - do you know how you would achieve this?

[Sasko]

@Steve-Mcl the second option would be better I think, just not to manipulate the data.

I think I need another function node. Then pass the output from the first one. Then create a loop, something like

sample = [ ]
avg = 0
for x = 0 to 999 {

avg= avg+ y

}

avg= avg/1000

sample.push(avg)

this would be the first sample. well I think I need a second loop and I'm not allowed to hardcore 0 to 999, to scan the whole vector. You have a suggestion ?

[Steve-Mcl]

not a bad starting point.

PS please put your code inside triple backticks 9helps with formating, readability and more importantly, stops the forum screwing up the code.

```
code here
```

So, I think something like this would give you three lines on a chart (Average, Max, Min)...

// SET THE NEXT LINE AS REQUIRED
const sampleSize = 200; //how many data elements to down sample

//define some helper functions...
const arrAvg = arr => arr.reduce((max, p) => max + p.SoC, 0) / arr.length
const arrMax = arr => arr.reduce((max, p) => p.SoC > max ? p.SoC : max, arr[0].SoC);
const arrMin = arr => arr.reduce((min, p) => p.SoC < min ? p.SoC : min, arr[0].SoC);

//1. lets create a variable to hold the finished article...
let chartData = [];  //See #1 - create an empty array

//2. lets create a variable to hold the everything...
let element1 = {};  //create element1 as an empty object {}

//3, now add the .property "series" to the element1 {object}
element1.series = []; // add a .property called series of type [array]
element1.series.push("Average"); 
element1.series.push("Max"); 
element1.series.push("Min"); 

//4. now add the .property "data" to the element1 {object}
element1.data = []; // add a .property called data of type [array]


/*********** NEXT - ADDING DATA  ***************************/

var data = msg.payload; // << the incoming data
var len = data.length; // << get its length
//console.log("sampleSize", sampleSize);
var dataRemaining = len;
var position = 0;
//console.log("position ", position);
//console.log("dataRemaining ", dataRemaining)

while(dataRemaining > 0){
  let sampleSet = data.slice(position, position + sampleSize)
  if(!sampleSet || !sampleSet.length){
    //console.log("no data - breaking out of loop")
    break;
  }
  //console.log("length of this sample: ", sampleSet.length)
  //console.log("sampleSet: ", sampleSet)
  element1.data.push({
    average: arrAvg(sampleSet), 
    max: arrMax(sampleSet), 
    min: arrMin(sampleSet)
  })
  position += sampleSet.length;
  dataRemaining -= sampleSet.length;
  //console.log("position ", position);
  //console.log("dataRemaining ", dataRemaining);
}

/*******************************************/

//5. now add a labels [array] to element1
element1.labels = []; // add a .property called labels of type [array]
element1.labels.push(""); //add an empty string to the "labels" [array]

//Now add the element1 [array] into the chartData [array]
chartData.push(element1)

//Lastly, return the chartData as the payload 
msg.payload = chartData;
return msg;
//console.log(JSON.stringify(chartData, null, "  "))

Use it / adapt it to your requirements

I updated the online example - to use sample data - so you can see it in action...

[Sasko]

Ok nice, so the data organisation wors perfect, thanks to you @Steve-Mcl . Now I want to pass the data to the Chart node, but for some reason after I start the process, the node loses the connection and nothing is displayed. even if I rise the sample size.