How to chart/graph from stored text files to the chart

General
1

I have tried searching but the results are more for real time display on the chart rather than from an existing file.

So, I have a heap of files that have temperature readings from multiple devices that I want to chart.

As the length of the file/s is unknown, I don't know what mechanics are needed to read from the file and put the data on the chart.

2

I think if you search the old forum you will find a lot of examples.

1 Like
3

(just talking/thinking here)

So on one of said links, I found a bit of stuff:

To display a complete chart in one go - for example from a set of points retrieved from a database, the data must be supplied in the form of an array, that holds an object that has series,labels, and data arrays. This is broadly the same as the raw format used by the angular chart.js library.

You will need to process your data into this structure in order to render it correctly.

So as this is where I am at, I'm guessing I will need to do that last line in my flow.

So.... that means opening the file, reading the lines (one by one) into a "structure" until EOF().
All that in one Function() node - at a guess. Then send it all to the Chart node.

Can I do all that in a Function() node?
I can kind of get the "do until EOF()" part, and have a read_line(FILE) in the loop.

Just want to check this isn't another fool's errand I am chasing not knowing the how to part of what I am wanting to do.

4

Personally, I'd use the File In node to read the file. Then pass it to a Function node to get its contents into the format needed by the chart node.

5

That is what I am doing.

Now I am re-learning how to read, parse, split and format text.

(How to find " - " in a string, and split it there to the left and right of it to two variables.
I know I have done it before, but can't remember where/how I did it.)

That will "format" the string from what it is to something I think the chart wants.
x value, y value.

6

What the chart wants is documented here https://github.com/node-red/node-red-dashboard/blob/master/Charts.md

7

If you want to share an example couple lines of your data, I'm sure we could offer some advice.

8

DC, that is the link I am reading.

(About 8 new/more tabs open)

Nick,

This is the 4 nodes I have so far:
(Chart not included only because I am still working on getting the data right.)

[{"id":"800d634b.cca2f8","type":"inject","z":"6472f474.da7424","name":"","topic":"","payload":"/home/me/TEMP/random_data.db","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":130,"y":220,"wires":[["90ab8d56.8d9c28"]]},{"id":"90ab8d56.8d9c28","type":"file in","z":"6472f474.da7424","name":"Read file","filename":"/home/me/TEMP/random_data.db","format":"lines","chunk":false,"sendError":false,"x":260,"y":220,"wires":[["362b4d02.83adea"]]},{"id":"362b4d02.83adea","type":"function","z":"6472f474.da7424","name":"","func":"\nreturn msg;","outputs":1,"noerr":0,"x":400,"y":220,"wires":[["b1508aa4.5cd908"]]},{"id":"b1508aa4.5cd908","type":"debug","z":"6472f474.da7424","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":560,"y":280,"wires":[]}]

This is the "random data file" it is reading:

cat random_data.db 
1532518343818 - 25
1532518344819 - 24
1532518345825 - 26
1532518346826 - 18
1532518347827 - 27
1532518348828 - 8
1532518349830 - 29
1532518350831 - 10
1532518351833 - 3
1532518352836 - 10
1532518353840 - 23
1532518354842 - 16
1532518355844 - 29
1532518356849 - 23
1532518357854 - 22
1532518358857 - 17
1532518359875 - 20

Which is pretty well the structure of the files I am creating anyway.
Just the values will vary depending if they are measuring Temperature, Load, etc.

9

I'm posting here only to keep new things away from older things I posted.

I made a bit of progress with this:

[{"id":"3284c08b.60dbc8","type":"debug","z":"6472f474.da7424","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":410,"y":330,"wires":[]},{"id":"f6d7994d.ff9fb8","type":"inject","z":"6472f474.da7424","name":"test","topic":"","payload":"1532518356849 - 23","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":100,"y":330,"wires":[["fa800697.befa9"]]},{"id":"fa800697.befa9","type":"function","z":"6472f474.da7424","name":"","func":"//var str = \"Hello world!\";\n//var res = str.substring(1, 4);\n\nmsg.payload = msg.payload.split(\" \").map(function(v) { return parseInt(v)});\n\nreturn msg;","outputs":1,"noerr":0,"x":260,"y":330,"wires":[["3284c08b.60dbc8"]]}]

(Taken from a post you did way back a while ago.)

10

This is my next step/stage on the learning curve:

[{"id":"362b4d02.83adea","type":"function","z":"6472f474.da7424","name":"","func":"msg.payload = msg.payload.split(\" \").map(function(v) { return parseInt(v)});\n\nreturn msg;","outputs":1,"noerr":0,"x":380,"y":190,"wires":[["3655290f.e793ce"]]},{"id":"3655290f.e793ce","type":"change","z":"6472f474.da7424","name":"","rules":[{"t":"move","p":"payload.1","pt":"msg","to":"payload.2","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":360,"y":270,"wires":[["b1508aa4.5cd908","76acfb38.7d5ebc"]]},{"id":"76acfb38.7d5ebc","type":"function","z":"6472f474.da7424","name":"","func":"var t = msg.payload[0];\nt = t - 153251835;\n//msg.payload[0] = t;\n\n//  {label:\"indoor temperature\", payload:22}\n//  var msg2 = { payload:\"0\", topic: device};\n//  {topic:\"temperature\", payload:22, timestamp:1520527095000}\n\nmsg = {label:\"Time\", payload:msg.payload[0], \"Temp\":msg.payload[1]};\n\nreturn msg;","outputs":1,"noerr":0,"x":560,"y":270,"wires":[["6397865d.a94918"]]},{"id":"6397865d.a94918","type":"ui_chart","z":"6472f474.da7424","name":"","group":"f25bb428.e9272","order":0,"width":0,"height":0,"label":"chart","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"linear","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":1,"removeOlderPoints":"200","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"x":760,"y":270,"wires":[[],[]]},{"id":"90ab8d56.8d9c28","type":"file in","z":"6472f474.da7424","name":"Read file","filename":"/home/me/TEMP/random_data.db","format":"lines","chunk":false,"sendError":false,"x":240,"y":190,"wires":[["362b4d02.83adea"]]},{"id":"800d634b.cca2f8","type":"inject","z":"6472f474.da7424","name":"","topic":"","payload":"/home/me/TEMP/random_data.db","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":190,"wires":[["90ab8d56.8d9c28"]]},{"id":"f25bb428.e9272","type":"ui_group","z":"","name":"Graph","tab":"91f90dd2.4c30b8","order":1,"disp":true,"width":"10","collapse":false},{"id":"91f90dd2.4c30b8","type":"ui_tab","z":"","name":"Charts","icon":"dashboard","order":19}]

Seems I am not getting all the stuff needed to get the chart working.

11

I don't see any Debug nodes in your flow. That ought to be step 1 in trying to look at the messages you are passing around so you can see if the structure matches what you expect and what the docs say you need to provide.

12

Sorry.... I have debug nodes.... I thought I included them. They are the only "output" things I have/had working. The graph was still a couple of steps ahead.

There is the inject node to start it all. The READ FILE node, a FUNCTION node, a CHANGE node, another FUNCTION node and the output/debug node.

This is the flow with chart included.

[{"id":"800d634b.cca2f8","type":"inject","z":"6472f474.da7424","name":"","topic":"","payload":"/home/me/TEMP/random_data.db","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":190,"wires":[["90ab8d56.8d9c28"]]},{"id":"90ab8d56.8d9c28","type":"file in","z":"6472f474.da7424","name":"Read file","filename":"/home/me/TEMP/random_data.db","format":"lines","chunk":false,"sendError":false,"x":240,"y":190,"wires":[["362b4d02.83adea"]]},{"id":"362b4d02.83adea","type":"function","z":"6472f474.da7424","name":"","func":"msg.payload = msg.payload.split(\" \").map(function(v) { return parseInt(v)});\n\nreturn msg;","outputs":1,"noerr":0,"x":380,"y":190,"wires":[["3655290f.e793ce"]]},{"id":"3655290f.e793ce","type":"change","z":"6472f474.da7424","name":"","rules":[{"t":"move","p":"payload.1","pt":"msg","to":"payload.2","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":360,"y":270,"wires":[["76acfb38.7d5ebc"]]},{"id":"76acfb38.7d5ebc","type":"function","z":"6472f474.da7424","name":"","func":"var t = msg.payload[0];\nt = t - 153251835;\n//msg.payload[0] = t;\n\n//  {label:\"indoor temperature\", payload:22}\n//  var msg2 = { payload:\"0\", topic: device};\n//  {topic:\"temperature\", payload:22, timestamp:1520527095000}\n//  { \"x\": 1504029632890, \"y\": 5 }\n\nmsg = {payload:t, \"Temp\":msg.payload[1]};\n\nreturn msg;","outputs":1,"noerr":0,"x":560,"y":270,"wires":[["6397865d.a94918","b1508aa4.5cd908"]]},{"id":"b1508aa4.5cd908","type":"debug","z":"6472f474.da7424","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":680,"y":300,"wires":[]},{"id":"6397865d.a94918","type":"ui_chart","z":"6472f474.da7424","name":"","group":"f25bb428.e9272","order":0,"width":0,"height":0,"label":"chart","chartType":"line","legend":"false","xformat":"HH:mm:ss","interpolate":"linear","nodata":"","dot":false,"ymin":"","ymax":"","removeOlder":1,"removeOlderPoints":"200","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"colors":["#1f77b4","#aec7e8","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"useOldStyle":false,"x":760,"y":270,"wires":[[],[]]},{"id":"f25bb428.e9272","type":"ui_group","z":"","name":"Graph","tab":"91f90dd2.4c30b8","order":1,"disp":true,"width":"10","collapse":false},{"id":"91f90dd2.4c30b8","type":"ui_tab","z":"","name":"Charts","icon":"dashboard","order":19}]
13

Your goal is to create a single message that looks like the structure from the docs here: https://github.com/node-red/node-red-dashboard/blob/master/Charts.md#line-charts-1

Your flow currently splits that file into one message per line, and transforms it into a message that looks like:
{"payload":1532365105014,"_msgid":"fe461090.01b9f"}

Let's step through a bit more to get it working.

First up, your first function currently does:

msg.payload = msg.payload.split(" ").map(function(v) { return parseInt(v)});
return msg;

For an input of 1532518344819 - 24 that results in a payload of [1532518344819,null,24]

We can tidy that up a bit by change the pattern it splits on:

msg.payload = msg.payload.split(/ - /).map(function(v) { return parseInt(v)});
return msg;

An input of 1532518344819 - 24 now gives a result of [1532518344819,24]

From the docs, what we actually want is something that looks like: { "x": 1504029632890, "y": 5 }.

var parts = msg.payload.split(/ - /).map(function(v) { return parseInt(v)});
msg.payload = {x:parts[0], y:parts[1]};
return msg;

At this point we still have a stream of individual messages. To combine them back into a single message we can use the Join node. This will work because the original File In node has already included the metadata needed in each message to identify them as part of a single sequence of messages.

The Join node can be configured in manual mode to combine each msg.payload to create an Array.

The final step is another Function node to put this newly formed msg.payload into the structure expected by the Chart node:

msg.payload = [{
  "series": ["A"],
  "data": [msg.payload],
  "labels": [""]
}];
return msg;

And we're done.

Here's the flow - wire it to your Chart node and you should be on your way.

[{"id":"ac0bd66f.c23df8","type":"inject","z":"43d009f5.24a068","name":"","topic":"","payload":"/home/me/TEMP/random_data.db","payloadType":"str","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":680,"wires":[["9778be14.2a9f4"]]},{"id":"9778be14.2a9f4","type":"file in","z":"43d009f5.24a068","name":"Read file","filename":"/home/me/TEMP/random_data.db","format":"lines","chunk":false,"sendError":false,"x":180,"y":740,"wires":[["15397051.3f513"]]},{"id":"15397051.3f513","type":"function","z":"43d009f5.24a068","name":"","func":"var parts = msg.payload.split(/ - /).map(function(v) { return parseInt(v)});\nmsg.payload = {x:parts[0], y:parts[1]};\nreturn msg;\n","outputs":1,"noerr":0,"x":310,"y":740,"wires":[["b8953dea.692a5"]]},{"id":"b8953dea.692a5","type":"join","z":"43d009f5.24a068","name":"","mode":"custom","build":"array","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":450,"y":740,"wires":[["37a6c7e0.ee0df8"]]},{"id":"37a6c7e0.ee0df8","type":"function","z":"43d009f5.24a068","name":"","func":"msg.payload = [{\n  \"series\": [\"A\"],\n  \"data\": [msg.payload],\n  \"labels\": [\"\"]\n}];\nreturn msg;","outputs":1,"noerr":0,"x":590,"y":740,"wires":[["e0bb31c9.d2379"]]},{"id":"e0bb31c9.d2379","type":"debug","z":"43d009f5.24a068","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":770,"y":740,"wires":[]}]
2 Likes
14

Nick,

Thanks - again.

I had better explain how stupid I am.

You are right in what you said, and I get that the SPLIT part could have been done better.
I got around the value, null, value with the CHANGE node.

What is sort of still confusing me is the workings.
You may have seen the commented lines in the FUNCTION node:

//  {label:"indoor temperature", payload:22}
//  var msg2 = { payload:"0", topic: device};
//  {topic:"temperature", payload:22, timestamp:1520527095000}
//  { "x": 1504029632890, "y": 5 }
//msg = {payload:t, "Temp":msg.payload[1]};

They are cut/pastes from other sites showing the syntax of how to build the structure.
(Well, ok, maybe not, but that was how I was "building" the structure.)

Having them on the same screen I could look at them and then edit/construct the line with what I wanted.

So, in your code:

var parts = msg.payload.split(/ - /).map(function(v) { return parseInt(v)});

kind of throws me.

When I found your post to someone else back in Feb 15
(Forgive the many quotes, but to get the confusion clarified...... I can't see any other way)

No problem. Everyone has to start somewhere.

    msg.payload = msg.payload.split(" ");
    return msg;

This will give you back a single message whose payload is an array or the individual values.

So to get the first two values you would use msg.payload[0] and msg.payload[1].

They will still be String types. If you want them as Numbers (so you can use them in calculations etc) then you could do:

msg.payload = msg.payload.split(" ").map(function(v) { return parseInt(v)});
return msg;

Which ensures each value is parsed to its Number type.

I used that (or those two lines) to try and get it working.

But I am lost where/how it works. As shown there.

Your newer reply helps.

Rather than "leaving it as msg.payload" you now say it is PARTS = msg.payload.......
Where as the original left it as msg.payload. (I'll stop there and keep with the here and now.)

Ok, so PARTS is the new variable which is a split of msg.payload and splitting it with the / - / and mapping it to (function(v), and returning the Int(v).

I sort of get it NOW. But to me MAP is a whole other kind of command. But that is my fault/problem.

So, I put it through the node as shown, expecting to get integer values given to me after the SWITCH to put msg.payload[2] to msg.payload[1]. I also tried to shrink the number, but that met problems.

So then, I tried to construct the last part where msg.payload = {x:part[0], y:parts[1]}; but I think I got messed up with the TYPES of data at that point.

I guess I can only hope to get even partly as smart with this as you are.

I feel I have a lot more mistakes to make on this journey.

Again: Thanks.
Did what I said in this help explain how I am confused?
It isn't that I am trying to shift blame. I am simply showing where/how I got confused with what I was reading and so kept failing to understand what was going on.

15

(Obviously) It works.