Iterating through object from Firebase for Chart Historical Data

Firstly, to everyone for all your hard work and help :slight_smile: Secondly, I could really do with some help here! I am trying to build a function which will render the output needed to feed stored data into line charts as detailed here: https://github.com/node-red/node-red-dashboard/blob/master/Charts.md

Env:
I am calling data in from Firebase Realtime DB using the node-red-contrib-firebase module - which is working fine.

Msg payload format
Data objects are being pulled in from Firebase and the incoming msg.payload looks like this:

msg.payload = {
    "-LcgJ2ryhWC4d7tVOX1X": {
        "score": "1",
        "ts": "1555521617.281100",
    },
    "-LcgJ2sr0o9mu26pEKUn": {
        "score": "2",
        "ts": "1555521617.281200",
    }
}

What am I trying to do?
I am trying to build a function to feed historical data into a chart.

The idea being to insert data into the appropriate x and y arrays within the data property of the data property of the historicalData object as shown below.

...
    "data":[
        [{ "x": ts[key0], "y": score[key0] }],
        [{ "x": ts[key1], "y": score[key1] }
        ] //etc...iterating through msg object
    ],
...

I believe this then needs to be passed into the output array.

Current code attempt

var output = [];

var historicalData = {
    "series":["Sentiment"],
    "data":[[{}]], //Several hundred x,y items from DB added here
    "labels":[""]
};

for (i = 0; i < Object.keys(msg.payload).length; i++){
    var key = Object.keys(msg.payload)[i];
    
    var ts = msg.payload[key].ts;
    var score= msg.payload[key].score;
    
    if (score){
    historicalData.data.x = ts;
    historicalData.data.y = tx;
    }
    
}

Can anyone please help me figure this out! I have spent almost 20 hours today revising object constructions, iterations, spreads, and the rest trying to solve this without asking anyone else for their time...so believe me, I have tried to look this one up and I still can't for the life of me find a sensible way to do this object updating.

Any help would be much appreciated!

Best wishes

Dan

Ah...do I have to create separate variable as array, populate it using push in a for loop and then insert to the historicalData object?, e.g.

var newData = [];
for loop{
newData.push (x: ts, y:score);
}

historicalData.data = newData;

Is that right?
And will the {} be automatically added if I do that?

You are getting closer and closer... :grinning:

I tend to use a function node (named "Setup payload" in below pic) as a boilerplate when I need to build a line chart with stored data. It looks like:

series = [];
data = [];
labels = [];

series.push("Field1");
labels.push("Field1");
data.push(msg.data);

msg.payload = [{"series":series, "data":data,"labels": labels}];

return msg;

So the main concern is to deliver a well formatted msg.data property to this function node.

In you case something like this should work (need to adapt the date formatting though):

msg.payload = {
    "-LcgJ2ryhWC4d7tVOX1X": {
        "score": "1",
        "ts": "2018-10-08T14:25:17Z"
    },
    "-LcgJ2sr0o9mu26pEKUn": {
        "score": "2",
        "ts": "2018-10-08T14:34:58Z"
    },
    "-LcgJ2ryhWC4d7tVOX1A": {
        "score": "3",
        "ts": "2018-10-08T14:44:36Z"
    }
}

let data =[];
for (let prop in msg.payload) {
    data.push({"x":msg.payload[prop].ts, "y":msg.payload[prop].score});
}

msg.data = data;
return msg;

Testing flow:

[{"id":"70178ff4.d615f","type":"tab","label":"Chart - ","disabled":false,"info":""},{"id":"c1bea52.3e19458","type":"ui_chart","z":"70178ff4.d615f","name":"","group":"d4cc65a.8bf5b98","order":0,"width":0,"height":0,"label":"{{msg.label}}","chartType":"line","legend":"true","xformat":"HH:mm:ss","interpolate":"step","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":"1","removeOlderPoints":"100","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"outputs":1,"x":710,"y":280,"wires":[[]]},{"id":"57728020.28e44","type":"inject","z":"70178ff4.d615f","name":"","topic":"","payload":"[]","payloadType":"json","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":530,"y":280,"wires":[["c1bea52.3e19458"]]},{"id":"7543d309.0aa91c","type":"function","z":"70178ff4.d615f","name":"Setup payload","func":"series = [];\ndata = [];\nlabels = [];\n\nseries.push(\"Field1\");\nlabels.push(\"Field1\");\ndata.push(msg.data);\n\nmsg.payload = [{\"series\":series, \"data\":data,\"labels\": labels}];\n\nreturn msg;","outputs":1,"noerr":0,"x":480,"y":220,"wires":[["c1bea52.3e19458","ccda5a07.fabd58"]]},{"id":"5affa314.0a890c","type":"function","z":"70178ff4.d615f","name":"Dataset","func":"msg.payload = {\n    \"-LcgJ2ryhWC4d7tVOX1X\": {\n        \"score\": \"1\",\n        \"ts\": \"2018-10-08T14:25:17Z\"\n    },\n    \"-LcgJ2sr0o9mu26pEKUn\": {\n        \"score\": \"2\",\n        \"ts\": \"2018-10-08T14:34:58Z\"\n    },\n    \"-LcgJ2ryhWC4d7tVOX1A\": {\n        \"score\": \"3\",\n        \"ts\": \"2018-10-08T14:44:36Z\"\n    }\n}\n\nlet data =[];\nfor (let prop in msg.payload) {\n    data.push({\"x\":msg.payload[prop].ts, \"y\":msg.payload[prop].score});\n}\n\nmsg.data = data;\nreturn msg;","outputs":1,"noerr":0,"x":280,"y":220,"wires":[["7543d309.0aa91c"]]},{"id":"95c65575.5e9ee8","type":"inject","z":"70178ff4.d615f","name":"Trigger","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":130,"y":220,"wires":[["5affa314.0a890c"]]},{"id":"ccda5a07.fabd58","type":"debug","z":"70178ff4.d615f","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":690,"y":220,"wires":[]},{"id":"d4cc65a.8bf5b98","type":"ui_group","z":"","name":"Group 1","tab":"90c6a3a3.ea2ca","disp":true,"width":"10","collapse":false},{"id":"90c6a3a3.ea2ca","type":"ui_tab","z":"","name":"Tab1","icon":"dashboard","order":6}]

Dude! That is so helpful (-:

I really, really wish this info was part of the help files. I have been banging my head against this for so long.

Once I have it really nicely worked out, I shall write it up and make a Git Pull request to the repo!

Many thanks, will try it and report back (-:

3 Likes

Yes please do.

1 Like

Out of interest @Andrei, in your flow, there is a node at the bottom (see pic) which just seems to inject an empty array... what exactly is that there for please?

Injecting an empty array in the dashboard chart node will clear the chart. Sometimes is useful.

Aha! I literally just worked it out as I got your message saying that :joy:

Thank you!

Ha...what I have now realised, of course, is that I have to do something else first!

Some advice on how to do this would be really helpful!

I need to:

  1. Take the original object data.
  2. Loop through it using the sentiment node and calculate the resulting score
  3. Insert the resultant score: as a new property to the original object
  4. Write each new object thus created into a new data store...

Any advice on how to to that with the loop nodes would really help...I find them very confusing - but I am sure they are actually really simple to use once I get my head around them!

What I can then do is, as each new item comes in, is just do the same thing with all the new items and I will just read all the chart data from the db. Avoiding all the mess of trying to do two things at once...

For that purpose we would need to know what is the structure of your dataset. Can you possibly share a sample file ?

Same as the payload above :slight_smile:

OK! Back from seeing the Life of Brian at the cinema now! Will see if I can solve it first and then will share a nice crisp data structure example :slight_smile: ... the only difficulty being that I've now got Always Look On the Bright Side of Life running through my head on continuous loop...