Why are the contents of files deleted?

nowomanman · 16 December 2023 19:58

Hi,

I have installed Node Red with Docker on a Raspberry Pi 4.
I have a chart in which I display temperatures as a graph.
Since this chart was often empty at the beginning, I was looking for a way to save the data. I have also found one.
But recently the content of the files has been deleted and I have not found a clue in any log as to why this is happening. So I don't know how I can prevent this or why it happens at all. But it is very annoying.

My installation:
Raspberry Pi OS (Bullseye)
Docker:
Client: Docker Engine - Community

Version: 25.0.0-beta.2
API version: 1.44
Go version: go1.21.5
Git commit: 7b3a60f
Built: Tue Dec 12 16:28:46 2023
OS/Arch: linux/arm64
Context: default

Server: Docker Engine - Community
Engine:
Version: 25.0.0-beta.2
API version: 1.44 (minimum version 1.24)
Go version: go1.21.5
Git commit: 92884c2
Built: Tue Dec 12 16:28:46 2023
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.6.26
GitCommit: 3dd1e886e55dd695541fdcd67420c2888645a495
runc:
Version: 1.1.10
GitCommit: v1.1.10-0-g18a0cb0
docker-init:
Version: 0.19.0
GitCommit: de40ad0

Docker Compose version v2.23.3

Directory:
/opt/nodered/

docker-compose.yml:

version: "3.7"

services:
node-red:
image: nodered/node-red:latest
container_name: node-red
Environment:
- TZ=Europe/Amsterdam
ports:
- "1880:1880"
volumes:
- node-red-data:/data
- ./temp-data:/home
Restart: if not stopped
networks:
- node-red-net

Volumes:
node-red-data:

networks:
node-red-net:
external: true
Name: node-red-net

The directory in which the data is located:
/opt/nodered/temp-data

Authorizations of the file:
-rw-rw-rw- 1 root root

Flows/Nodes:

[
    {
        "id": "67b87ba115798d5d",
        "type": "mqtt in",
        "z": "f6f2187d.f17ca8",
        "g": "b1758e321249d0e8",
        "name": "Küche",
        "topic": "temp/outside",
        "qos": "0",
        "datatype": "auto-detect",
        "broker": "17eb78de4afb253f",
        "nl": false,
        "rap": true,
        "rh": 0,
        "inputs": 0,
        "x": 90,
        "y": 140,
        "wires": [
            [
                "a7bca3f79db9a314",
                "454b28ab60627ba5",
                "b073864d775c966c",
                "b7aa17a737acda56",
                "29bd06c1b4c1fda7"
            ]
        ]
    },
    {
        "id": "a7bca3f79db9a314",
        "type": "ui_chart",
        "z": "f6f2187d.f17ca8",
        "g": "b1758e321249d0e8",
        "name": "",
        "group": "5b086e8c2eee96c1",
        "order": 3,
        "width": 6,
        "height": 6,
        "label": "Temperatur draußen letzten 8 Wochen",
        "chartType": "line",
        "legend": "true",
        "xformat": "dd HH:mm",
        "interpolate": "bezier",
        "nodata": "",
        "dot": false,
        "ymin": "-20",
        "ymax": "40",
        "removeOlder": "8",
        "removeOlderPoints": "",
        "removeOlderUnit": "604800",
        "cutout": 0,
        "useOneColor": false,
        "useUTC": false,
        "colors": [
            "#d62728",
            "#aec7e8",
            "#ff7f0e",
            "#2ca02c",
            "#98df8a",
            "#1f77b4",
            "#ff9896",
            "#9467bd",
            "#c5b0d5"
        ],
        "outputs": 1,
        "useDifferentColor": false,
        "className": "",
        "x": 470,
        "y": 180,
        "wires": [
            [
                "0ecf46a5cde31204"
            ]
        ]
    },
    {
        "id": "454b28ab60627ba5",
        "type": "debug",
        "z": "f6f2187d.f17ca8",
        "g": "b1758e321249d0e8",
        "name": "debug 1",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 300,
        "y": 60,
        "wires": []
    },
    {
        "id": "b073864d775c966c",
        "type": "ui_chart",
        "z": "f6f2187d.f17ca8",
        "g": "b1758e321249d0e8",
        "name": "",
        "group": "5b086e8c2eee96c1",
        "order": 2,
        "width": 6,
        "height": 6,
        "label": "Temperatur draußen letzten 12 Stunden",
        "chartType": "line",
        "legend": "true",
        "xformat": "HH:mm",
        "interpolate": "bezier",
        "nodata": "",
        "dot": false,
        "ymin": "-10",
        "ymax": "10",
        "removeOlder": "12",
        "removeOlderPoints": "",
        "removeOlderUnit": "3600",
        "cutout": 0,
        "useOneColor": false,
        "useUTC": false,
        "colors": [
            "#2ca02c",
            "#aec7e8",
            "#ff7f0e",
            "#d62728",
            "#98df8a",
            "#1f77b4",
            "#ff9896",
            "#9467bd",
            "#c5b0d5"
        ],
        "outputs": 1,
        "useDifferentColor": false,
        "className": "",
        "x": 480,
        "y": 120,
        "wires": [
            []
        ]
    },
    {
        "id": "9673589d7815b17d",
        "type": "ui_button",
        "z": "f6f2187d.f17ca8",
        "g": "b1758e321249d0e8",
        "name": "",
        "group": "5b086e8c2eee96c1",
        "order": 5,
        "width": 2,
        "height": 1,
        "passthru": false,
        "label": "restore",
        "tooltip": "",
        "color": "",
        "bgcolor": "",
        "className": "",
        "icon": "",
        "payload": "",
        "payloadType": "str",
        "topic": "",
        "topicType": "str",
        "x": 90,
        "y": 360,
        "wires": [
            [
                "365dafb703db805b"
            ]
        ]
    },
    {
        "id": "56ca315f23e1ceb1",
        "type": "file",
        "z": "f6f2187d.f17ca8",
        "g": "b1758e321249d0e8",
        "name": "",
        "filename": "/home/chart.log",
        "filenameType": "str",
        "appendNewline": true,
        "createDir": true,
        "overwriteFile": "true",
        "x": 660,
        "y": 300,
        "wires": [
            []
        ]
    },
    {
        "id": "365dafb703db805b",
        "type": "file in",
        "z": "f6f2187d.f17ca8",
        "g": "b1758e321249d0e8",
        "name": "",
        "filename": "/home/chart.log",
        "filenameType": "str",
        "format": "utf8",
        "allProps": false,
        "x": 300,
        "y": 360,
        "wires": [
            [
                "253884caecfa64bf"
            ]
        ]
    },
    {
        "id": "0ecf46a5cde31204",
        "type": "json",
        "z": "f6f2187d.f17ca8",
        "g": "b1758e321249d0e8",
        "name": "",
        "property": "payload",
        "action": "",
        "pretty": false,
        "x": 710,
        "y": 180,
        "wires": [
            [
                "6b10ac5dab4db1e3"
            ]
        ]
    },
    {
        "id": "253884caecfa64bf",
        "type": "json",
        "z": "f6f2187d.f17ca8",
        "g": "b1758e321249d0e8",
        "name": "",
        "property": "payload",
        "action": "",
        "pretty": false,
        "x": 490,
        "y": 360,
        "wires": [
            [
                "a7bca3f79db9a314"
            ]
        ]
    },
    {
        "id": "6b10ac5dab4db1e3",
        "type": "function",
        "z": "f6f2187d.f17ca8",
        "g": "b1758e321249d0e8",
        "name": "",
        "func": "if (msg.topic === \"save\") {\n    msg.payload = context.last;\n    return msg;\n}\nelse {\n    context.last = msg.payload;\n}\nreturn null;",
        "outputs": 1,
        "timeout": "",
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 340,
        "y": 300,
        "wires": [
            [
                "56ca315f23e1ceb1"
            ]
        ]
    },
    {
        "id": "b7aa17a737acda56",
        "type": "ui_button",
        "z": "f6f2187d.f17ca8",
        "g": "b1758e321249d0e8",
        "name": "",
        "group": "5b086e8c2eee96c1",
        "order": 4,
        "width": 1,
        "height": 1,
        "passthru": true,
        "label": "save",
        "tooltip": "",
        "color": "",
        "bgcolor": "",
        "className": "",
        "icon": "",
        "payload": "",
        "payloadType": "str",
        "topic": "save",
        "topicType": "str",
        "x": 130,
        "y": 300,
        "wires": [
            [
                "6b10ac5dab4db1e3"
            ]
        ]
    },
    {
        "id": "29bd06c1b4c1fda7",
        "type": "ui_text",
        "z": "f6f2187d.f17ca8",
        "g": "b1758e321249d0e8",
        "group": "5b086e8c2eee96c1",
        "order": 1,
        "width": "6",
        "height": "1",
        "name": "",
        "label": "Aktuelle Temperatur",
        "format": "{{msg.payload}}",
        "layout": "col-center",
        "className": "",
        "style": false,
        "font": "",
        "fontSize": 16,
        "color": "#000000",
        "x": 600,
        "y": 60,
        "wires": []
    },
    {
        "id": "17eb78de4afb253f",
        "type": "mqtt-broker",
        "name": "Broker",
        "broker": "mosquitto",
        "port": "1883",
        "clientid": "",
        "autoConnect": true,
        "usetls": false,
        "protocolVersion": "4",
        "keepalive": "60",
        "cleansession": true,
        "autoUnsubscribe": true,
        "birthTopic": "",
        "birthQos": "0",
        "birthRetain": "false",
        "birthPayload": "",
        "birthMsg": {},
        "closeTopic": "",
        "closeQos": "0",
        "closeRetain": "false",
        "closePayload": "",
        "closeMsg": {},
        "willTopic": "",
        "willQos": "0",
        "willRetain": "false",
        "willPayload": "",
        "willMsg": {},
        "userProps": "",
        "sessionExpiry": ""
    },
    {
        "id": "5b086e8c2eee96c1",
        "type": "ui_group",
        "name": "Temperatur Draußen",
        "tab": "be7464e7b0f7fc9e",
        "order": 1,
        "disp": true,
        "width": "6",
        "collapse": false,
        "className": ""
    },
    {
        "id": "be7464e7b0f7fc9e",
        "type": "ui_tab",
        "name": "Pi Zero",
        "icon": "dashboard",
        "order": 1,
        "disabled": false,
        "hidden": false
    }
]

Can someone help me so that the contents of the files are no longer deleted?

jbudd · 16 December 2023 20:13

I can't explain how your file got erased, except to mention that the write-file node has the option to append or overwrite the file contents.

It might be a good idea to store your data in a database, specifically designed for efficient storage and retrieval of data.
The one I use is Mariadb, it uses Structured Query Language (SQL) to store and recover data and it runs very well along with Node-red on a Raspberry Pi 4.
Alternatives include Influxdb and Sqlite.

As far as I know, using Arch Linux and Docker should not be a bar to using the databases above, though it will surely make setting them up more complicated!

nowomanman · 16 December 2023 20:54

Database as its own container?

Are there one or more examples, because I can learn better with examples than with the documentation, unfortunately.

No, my system is not Arch, it is based on Debian Bullseye, the Raspberry Pi OS.

Thanks for the help.

jbudd · 16 December 2023 21:17

Oh OK. I was misled by this

I have had a look at Docker but was put off by the idea of having to learn yet another mark-up language, so no idea if a db should be in it's own container.

There are a good many youtube tutorials for using a database with Node-red.

Colin · 16 December 2023 21:30

Is it when the container restarts that the data disappears?

nowomanman · 17 December 2023 09:53

Apparently yes, I've just had a look, my 8 containers have been up for 23 hours.
So there must have been a reboot, but not from the host, because I would have received an email.
Unfortunately I don't see anything in the individual Docker logs and I don't really see anything in the system logs either.
And I have not yet found whether Docker has its own log...
Well, next weekend at the latest I'll have to set up the whole system again due to a hard disk change.
Let's see if it still happens then.
But I'll have a look at the database.
But if someone has hints why Docker restarts without rebooting the system, please let me know.
Thanks for your help.

Colin · 17 December 2023 10:11

Have you mapped the folder you are saving them in to a folder outside the container?

If I were doing this then, rather than saving the data into files, I would use Persistent Context which is built into the core of node red. That will save the data in a subdirectory of the folder containing the flows file, so it should not get lost when the container restarts.

Do you have to use Docker? If you don't have a good reason then don't do it, it just adds complexity.

nowomanman · 17 December 2023 10:39

Yes, I have the directory /opt/nodered/temp-data so bind mount to /home in the container.
Nevertheless, the content of the files was deleted...
Could it be because the files end with .log?

As for the link, unfortunately my English is not the best and unfortunately I have had too little to do with Node Red so far. I don't really understand what it says and therefore don't know what to do.

Well, it all started with Tandoor, which is very difficult to install without Docker. Furthermore, I once had the problem that I needed php8.1, but this was not yet included in the repository. I installed it manually, then the system said something about unneeded packages during an update/upgrade. Since I was too inexperienced, I had them deleted and bang everything was broken. That's why I liked Docker, everyone has their own OS with their own packages and doesn't get in each other's way. That's why I use Docker.

Colin · 17 December 2023 10:47

No.

I don't know enough about Docker to know whether there is a problem with your docker compose file.

If you look at the file from outside the container while the container is running do you see a file containing the date?

If you then stop the container is the data still there?

nowomanman · 17 December 2023 11:05

Yes, they are there while it is running.

Yes, the date is 26.11.

And yet the contents of the file have been deleted.

Well, I have now looked again for the settings.js, the one with the localfilesystem was apparently still commented out. I didn't see it, because I have no experience with js and only know # for commenting out. I have now commented it in and will continue to monitor it. the new installation will not come until next weekend.

Colin · 17 December 2023 11:31

It is not clear what question you are replying to. Please try again, with numbers.

If you look at the file while the container is running is the data in the file?
If you stop the container is the data still in the file?
If you restart the container does the data disappear?

nowomanman · 17 December 2023 11:44

Yes
Yes
No

That's why I'm surprised...
but well, I just commented in the contextStorage in settings.js, which was commented out.

Wait, I'll try that on the original server where the contextStorage is not yet commented in, maybe that will make a difference.

nowomanman · 17 December 2023 11:49

original Server contextStorage also commented out in settings.js.

yes
yes
no

Colin · 17 December 2023 12:02

That was in answer to the question

Perhaps there is a translation error, because your answer "no" means that the data did not disappear, so it is still there.

I should have asked the same question as for 1 and 2
4. If you restart the container is the data still in the file?

nowomanman · 17 December 2023 12:16

The data is still there and does not disappear.

yes

Colin · 17 December 2023 12:22

So what is the problem?

nowomanman · 17 December 2023 12:24

That the data just disappeared yesterday and I can't figure out WHY!
Because as you can see now, nothing disappears now, but yesterday the charts were suddenly empty and I looked in the file and it was empty...

nowomanman · 17 December 2023 12:48

do I have to enter this in the settings.js?

contextStorage: {
default: {
module: "localfilesystem",
config: {
~/.node-red/context
}
}
}

and do I have to create the .node-red/context directory myself or does node red do this automatically?

and do I then have to specify /home/.node-red/context when "write file"?

E1cid · 17 December 2023 13:15

Looking at you flow.json you posted.

The function never returns any message, so never writes to the file write node, also your context read and write are in the wrong syntax

if (msg.topic === "save") {
 msg.payload = context.last;
 return msg;
}
else {
 context.last = msg.payload;
}
return null;

https://nodered.org/docs/user-guide/writing-functions#storing-data
e.g.
msg.payload = context.get("last");

nowomanman · 17 December 2023 13:26

Then why does it save the data I want in the file?

Maybe it helps to see the flows, because save and restore work.
But it is not clear why the data in the file is sporadically deleted...

Topic		Replies	Views
Recover lost flows General	5	3150	6 December 2019
Node Red in Docker loosing all flows every restart/reboot General	5	1560	13 July 2018
What is the function of file 【/usr/src/node-red/core】 in Node-Red? General docker	6	136	4 January 2024
No "file" node in palette (multi-tenant-node-red, docker) General	9	498	3 May 2021
Node-Red - Docker: Disappearing Objects General	5	285	2 August 2022

Why are the contents of files deleted?

Related Topics