Max data for a Chart on Rpi Zero W2

General
1

This isn't a problem, but more of a feasibility question if allowed...

I'm running an RPi Zero W 2 somewhere, collecting SNMP data to chart.
The chart is supposed to be 10 days of data.
I also have the chart output write to a file every few hours so it can restore the chart in a data loss, or share it to a central interface I use to view history's. The file is overwritten each time, so it only has data from the last chart output.
The chart collects 1 data point per minute each for 3 topics. I also have it filter out data that hasn't changed.

I'm 3 days into the chart, and the storage file is almost 3mb now, so I assume it will be about 10mb when filled. This means the chart pumps out a 10mb data message on every received message.

Is this going to kill the Pi Zero? Or drag down it's performance to near-bricked?

2

How large is the SD card you are using?

3

3 days at 1/minute is 4320 samples. With three lines that makes 12960 data points. If the file takes up 3MB (I assume that you mean Mega Bytes, not what you wrote, which is milli bits) then that means each data point is 700 bytes long in the file.

Have you looked in the file to see what is actually there to see why it is so long?

Edit, sorry, got that wrong, it is 230 bytes per sample, which still does not make sense.

4

The SD card is 32gb.

The file is the output of the chart node. It goes to a write-file node, set to overwrite with each new message so the data doesn't over-accumulate.

5

Have a look in the file to see why it is so long. Post a few lines from the start of it here if necessary.

6

Ok, will do when near it again.

7

BTW, I'm not really concerned that the file is large. It's collecting all the data you calculated above, so it seems reasonable for it to be a large file. I'm more concerned that generating these large data movements is going to overwhelm the Zero.

8

I can't see why each data point should take 700 bytes. I have just done some tests and get about 75 bytes/sample, which would make the files much smaller. Perhaps you could Export the chart node and up to the File node so we can see exactly what you have done. Plus post a short excerpt of the file.

9

Here's the truncated payload headed for the file node:

[
    {
        "series": [
            "KW out",
            "PA I (x10)",
            "PA V"
        ],
        "data": [
            [
                {
                    "x": 1736306311632,
                    "y": 22.8
                },
                {
                    "x": 1736306321640,
                    "y": 22.9
                },
                {
                    "x": 1736306361644,
                    "y": 22.8
                },
                {
                    "x": 1736306371643,
                    "y": 22.9
                },
                {
                    "x": 1736306401702,
                    "y": 22.8
                },
                {
                    "x": 1736306411652,
                    "y": 23
                },
                {
                    "x": 1736306421664,
                    "y": 22.9
                },
                {
                    "x": 1736306431649,
                    "y": 22.8
                },
        ...

Here's the chart through file node:

[
    {
        "id": "b94a1d4f804b3c31",
        "type": "ui_chart",
        "z": "535cc4d2ad04841c",
        "name": "",
        "group": "3ed3178aa2180877",
        "order": 0,
        "width": "12",
        "height": "7",
        "label": "Fax",
        "chartType": "line",
        "legend": "true",
        "xformat": "dd HH:mm",
        "interpolate": "linear",
        "nodata": "",
        "dot": false,
        "ymin": "0",
        "ymax": "100",
        "removeOlder": "5",
        "removeOlderPoints": "",
        "removeOlderUnit": "86400",
        "cutout": 0,
        "useOneColor": false,
        "useUTC": false,
        "colors": [
            "#1f77b4",
            "#aec7e8",
            "#ff7f0e",
            "#2ca02c",
            "#98df8a",
            "#d62728",
            "#ff9896",
            "#9467bd",
            "#c5b0d5"
        ],
        "outputs": 1,
        "useDifferentColor": false,
        "className": "",
        "x": 590,
        "y": 160,
        "wires": [
            [
                "2654b28b1aeb1964",
                "a61f072e31f6f5c3"
            ]
        ]
    },
    {
        "id": "c8360d20ca5270f1",
        "type": "file",
        "z": "535cc4d2ad04841c",
        "name": "",
        "filename": "NRData/10DayFaxChart.txt",
        "filenameType": "str",
        "appendNewline": false,
        "createDir": true,
        "overwriteFile": "true",
        "encoding": "none",
        "x": 1420,
        "y": 180,
        "wires": [
            []
        ]
    },
    {
        "id": "2654b28b1aeb1964",
        "type": "switch",
        "z": "535cc4d2ad04841c",
        "name": "",
        "property": "payload",
        "propertyType": "msg",
        "rules": [
            {
                "t": "empty"
            },
            {
                "t": "else"
            }
        ],
        "checkall": "true",
        "repair": false,
        "outputs": 2,
        "x": 730,
        "y": 160,
        "wires": [
            [
                "01d6fc2e62ac2f5a"
            ],
            [
                "0210f151fd2c7d43"
            ]
        ]
    },
    {
        "id": "4d9a9cd112bb469c",
        "type": "json",
        "z": "535cc4d2ad04841c",
        "name": "",
        "property": "payload",
        "action": "",
        "pretty": true,
        "x": 1210,
        "y": 180,
        "wires": [
            [
                "c8360d20ca5270f1",
                "834c0d089fe0a9a9"
            ]
        ]
    },
    {
        "id": "834c0d089fe0a9a9",
        "type": "debug",
        "z": "535cc4d2ad04841c",
        "name": "debug 75",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 1360,
        "y": 140,
        "wires": []
    },
    {
        "id": "01d6fc2e62ac2f5a",
        "type": "change",
        "z": "535cc4d2ad04841c",
        "name": "",
        "rules": [
            {
                "t": "set",
                "p": "payload",
                "pt": "msg",
                "to": "",
                "tot": "date"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 900,
        "y": 140,
        "wires": [
            [
                "4d12d85227107a32"
            ]
        ]
    },
    {
        "id": "599148b81e226459",
        "type": "delay",
        "z": "535cc4d2ad04841c",
        "name": "",
        "pauseType": "delay",
        "timeout": "5",
        "timeoutUnits": "seconds",
        "rate": "1",
        "nbRateUnits": "1",
        "rateUnits": "second",
        "randomFirst": "1",
        "randomLast": "5",
        "randomUnits": "seconds",
        "drop": false,
        "allowrate": false,
        "outputs": 1,
        "x": 1080,
        "y": 180,
        "wires": [
            [
                "4d9a9cd112bb469c"
            ]
        ]
    },
    {
        "id": "0210f151fd2c7d43",
        "type": "delay",
        "z": "535cc4d2ad04841c",
        "name": "",
        "pauseType": "queue",
        "timeout": "5",
        "timeoutUnits": "seconds",
        "rate": "1",
        "nbRateUnits": "3",
        "rateUnits": "hour",
        "randomFirst": "1",
        "randomLast": "5",
        "randomUnits": "seconds",
        "drop": true,
        "allowrate": false,
        "outputs": 2,
        "x": 910,
        "y": 180,
        "wires": [
            [
                "599148b81e226459"
            ],
            []
        ]
    },
    {
        "id": "a61f072e31f6f5c3",
        "type": "debug",
        "z": "535cc4d2ad04841c",
        "name": "debug 77",
        "active": false,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 740,
        "y": 80,
        "wires": []
    },
    {
        "id": "e99fe0bd55518073",
        "type": "inject",
        "z": "535cc4d2ad04841c",
        "name": "",
        "props": [
            {
                "p": "flush",
                "v": "true",
                "vt": "bool"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "01 00 * * *",
        "once": true,
        "onceDelay": 0.1,
        "topic": "",
        "x": 730,
        "y": 200,
        "wires": [
            [
                "0210f151fd2c7d43"
            ]
        ]
    },
    {
        "id": "3ed3178aa2180877",
        "type": "ui_group",
        "name": "FAX",
        "tab": "34e52e0ab57fe93c",
        "order": 1,
        "disp": true,
        "width": "12",
        "collapse": false,
        "className": ""
    },
    {
        "id": "34e52e0ab57fe93c",
        "type": "ui_tab",
        "name": "History",
        "icon": "dashboard",
        "disabled": false,
        "hidden": false
    }
]
10

For what it's worth, i have my answer...the Zero is operating at a crawl with a 6mb chart...

11

Have you looked at the file? SInce you have ticked Format JSON String in the JSON node I think you will find the file is mostly full of formatting spaces. I suspect that the file size will dramatically reduce if you untick that option.

On the rest of the flow, why have you got a Delay node delaying each message by 5 seconds? I don't see what that accomplishes. You have not configured it as rate limit, which could make sense, but just as a delay.

Also the Flush inject that you have will do nothing, as the Rate Limit node is not configured in a queuing mode, so there will never be anything to flush.

Is that with the 1 msg/3 hours rate limit included? If so then the problem is nothing to do with writing the file.

12

Yes, the file was formatted. Unchecking that box did reduce the file size significantly. Thanks.

Delay node: The delay is intended to allow the file to be read by another process before it's written with new data.

The flush inject works as intended during testing. It's solely for troubleshooting. The rate limiting as I've configured it does queue. Rate limiting has never really behaved as I've expected it to. It always looks like this:

image
image535×248 17.4 KB

1 msg/3 hours is indeed how it's configured. However, since reducing the chart to 1 day, performance has improved significantly. Perhaps the 4 messages in queue were too large and now they're manageable, or maybe the memory overhead of holding the chart data was too much. Either way, it looks like I'll need something more powerful to move the data around.

The problem has never been writing the file... The question was whether or not the data being pumped out of the chart node would be too much for a Pi Zero.