Context store 'localfilesystem' not saved to file system

Hello, I have problems persisting data in file context storage. Usage in my function nodes works perfectly well. Data appears correctly in editor context side panel. Sharing across nodes and between flows works fine. All is perfect until I restart Node Red. It is then as if the data had never been saved. Inspecting .node-red/context/global/global.json shows

{
    "sensorDevices": {}
}

This implies that 'something' has happened, but no values are saved. The Editor always shows seemingly correct data.
IMG_A79575DE16F1-1
In particular, whenever I do a refresh, there is never an indication that sensorDevices could be empty. However, the data that is shown in the editor seems never to be flushed to disk.

What am I missing or overlooking? Where should I start to debug? Has anyone seen similar behaviour?

Thank you for any help or hint!

For completeness I append settings.js and context calls in my function nodes.

My settings.js entry for context store is:

// HMW 2023-02-07 Expliziten File System Store ermöglichen
    contextStorage: {
       default: "memoryOnly",
       memoryOnly: { module: 'memory' },
       file: { module: 'localfilesystem' }
    },

Usage examples for retrieving and setting data:

var sensorDevices = global.get("sensorDevices", "file");
global.set("sensorDevices", sensorDevices, "file");

Welcome to the forum @hmw

Which version of node-red are you using? If not the latest then I suggest upgrading. It may not help, but is worth trying.
Have you checked the file contents after saving something in context and then leaving node red 10 minutes to allow it to flush to file?
If it is still empty then note the timestamp on the file and stop node-red (don't start it again). Does the timestamp change?

Also start node-red in a terminal (after stopping it), write to the context and stop node red. Paste the full terminal output here.

Hi Colin, thank you for your very quick reply. Here is my feedback:

  • Version is v3.0.2. I even reinstalled NR again.
  • Here is the sequence of actions and results from your suggested procedure:
    Status at the beginning:
pi@rpi4sys1 ~> ls -al .node-red/context/global/
insgesamt 12
drwxr-xr-x 2 pi pi 4096  7. Feb 18:06 ./
drwxr-xr-x 4 pi pi 4096  3. Feb 10:58 ../
-rw-r--r-- 1 pi pi   48  7. Feb 18:06 global.json
pi@rpi4sys1 ~> ls -al .node-red/context/global/
insgesamt 12
drwxr-xr-x 2 pi pi 4096  7. Feb 18:15 ./
drwxr-xr-x 4 pi pi 4096  3. Feb 10:58 ../
-rw-r--r-- 1 pi pi   48  7. Feb 18:15 global.json

Action in NR including writes to global file context. Resulting content in global.json is identical to previous content.

pi@rpi4sys1 ~> cat .node-red//context/global/global.json 
{
    "sensorDevices": {},
    "sensCalib": {}
}⏎
pi@rpi4sys1 ~>

Stopped NR and resulting timestamp with ls -al

pi@rpi4sys1 ~> node-red-stop
Stop Node-RED
 
Use   node-red-start   to start Node-RED again
 
pi@rpi4sys1 ~> 
pi@rpi4sys1 ~> ls -al .node-red/context/global/
insgesamt 12
drwxr-xr-x 2 pi pi 4096  7. Feb 18:17 ./
drwxr-xr-x 4 pi pi 4096  3. Feb 10:58 ../
-rw-r--r-- 1 pi pi   48  7. Feb 18:17 global.json

Starting NR again after writing to global context and appending NR-Log output

pi@rpi4sys1 ~> node-red-start

Start Node-RED
 
Once Node-RED has started, point a browser at http://192.168.54.227:1880
On Pi Node-RED works better with the Firefox or Chrome browser
 
Use   node-red-stop                          to stop Node-RED
Use   node-red-start                         to start Node-RED again
Use   node-red-log                           to view the recent log output
Use   sudo systemctl enable nodered.service  to autostart Node-RED at every boot
Use   sudo systemctl disable nodered.service to disable autostart on boot
 
To find more nodes and example flows - go to http://flows.nodered.org
 
7 Feb 18:20:37 - [info]
Willkommen bei Node-RED
===================
7 Feb 18:20:37 - [info] Node-RED Version: v3.0.2
7 Feb 18:20:37 - [info] Node.js  Version: v16.19.0
7 Feb 18:20:37 - [info] Linux 5.15.84-v7l+ arm LE
7 Feb 18:20:38 - [info] Paletten-Nodes werden geladen
7 Feb 18:20:40 - [info] Dashboard version 3.3.1 started at /ui
7 Feb 18:20:40 - [info] Einstellungsdatei: /home/pi/.node-red/settings.js
7 Feb 18:20:40 - [info] Kontextspeicher: memoryOnly [module=memory]
7 Feb 18:20:40 - [info] Kontextspeicher: file [module=localfilesystem]
7 Feb 18:20:40 - [info] Benutzerverzeichnis: /home/pi/.node-red
7 Feb 18:20:40 - [info] Projektverzeichnis: /home/pi/.node-red/projects
7 Feb 18:20:40 - [info] Server wird jetzt auf http://127.0.0.1:1880/ ausgefĂĽhrt
7 Feb 18:20:40 - [info] Aktives Projekt: SensorManagement
7 Feb 18:20:40 - [info] Flow-Datei: /home/pi/.node-red/projects/SensorManagement/flows.json
7 Feb 18:20:40 - [warn] Verwende unverschlĂĽsselte Credentials
7 Feb 18:20:40 - [info] Flows werden gestartet
Tesla API: No token found. Getting new token now...
Tesla API: doing api call to fetch new access token
7 Feb 18:20:41 - [info] Flows sind gestartet
7 Feb 18:20:41 - [info] [sqlitedb:001dd50d78d305f7] opened *<--- removed --->* ok
7 Feb 18:20:41 - [warn] [function:InitContext] [InitContext] sensorDevices is undefined
7 Feb 18:20:41 - [warn] [function:InitContext] [InitContext] mBusMap.size is undefined
7 Feb 18:20:41 - [info] [mqtt-broker:MQTTHMWIOT] Verbindung zum Broker mqtt://localhost:1883 aufgebaut
*<--- removed some lines about sqlitedb actions --->*
^C⏎                                                                                                                                 pi@rpi4sys1 ~ [SIGINT]> node-red-stop
Stop Node-RED
 
Use   node-red-start   to start Node-RED again
 
pi@rpi4sys1 ~> 

I apologise for the German language, but this is a German installation. If you need any help with this, please let me know ...

I could also remove any SQL related actions, but I do not think that this could possibly be related?

Well the timestamp is changing so it is writing to the file.

I have no idea what might be the problem here. Hopefully someone else can suggest something.

Could you export your flow and attach it to a reply so I could take a look?

Or better yet, could you create a small flow that demonstrates the problem and export and share that one. If you can’t then send the current flow.

Also might I suggest you change your settings.js file to this

     	contextStorage: {
			default    : { module: "localfilesystem" },
 			memory     : { module: "memory" }
     	},

then you only need use

var sensorDevices = global.get("sensorDevices");
global.set("sensorDevices", sensorDevices);

unless you want to temporally save something in memory.

In order to make code readable and usable it is necessary to surround your code with three backticks (also known as a left quote or backquote ```)

``` 
   code goes here 
```

You can edit and correct your post by clicking the pencil :pencil2: icon.

See this post for more details - How to share code or flow json

I think you have not said exactly what sort of objects are in sensorDevics. If they are not serializable, they could be returned as undefined.

[Edit] Have you tried with some primitive data types?

Hi all. Thank you for assisting me. I did some further investigation by now and have more details now.

I followed your advice(s) and implemented a simple test flow including primitive data types. Voilá, I can see the primitive data type in the global.json file, but not the map data type!

The file looks like this:

{
    "sensorDevices": {},
    "simpleVar": "sVar=3",
    "mapVar": {}
}

Please ignore the sensorDevices for now. I just did not want to change anything in my reports.

You can see the test flow here:

[
    {
        "id": "55aa4010d6d3a95c",
        "type": "inject",
        "z": "fe0c0f2c10548c43",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 100,
        "y": 780,
        "wires": [
            [
                "e799324ef7b16231",
                "a477bc90d0e43edf"
            ]
        ]
    },
    {
        "id": "e799324ef7b16231",
        "type": "function",
        "z": "fe0c0f2c10548c43",
        "name": "SetIncrGFCtest",
        "func": "/*\nSet and increment global context variables\n*/\nconst me = \"[\" + node.name + \"] \";\n//node.warn(me);\n\n// Get global context simple var or create it\nvar simpleVar = global.get(\"simpleVar\", \"file\") || \"sVar=-1\";\n// Init or increment simpleVar\nlet aSV = simpleVar.split(\"=\");\nlet inx = Number(aSV[1]) + 1;\nsimpleVar = aSV[0] + \"=\" + inx;\n// Write back to global. context\nglobal.set(\"simpleVar\", simpleVar, \"file\");\nnode.warn(me + \"simpleVar = \" + simpleVar);\n\n// Get global context map var or create it\nvar mapVar = global.get(\"mapVar\", \"file\") || new Map;\nif (mapVar === undefined) {\n    node.warn(me + \"mapVar === undefined\");\n    global.set(\"mapVar\", null, \"file\");\n    mapVar = new Map()\n};\n// Increment entry\nlet elemName = \"firstElem\";\nlet elemVal = 0;\nnode.warn(me + \"mapVar.has(elemName) = \" + mapVar.has(elemName));\nif (mapVar.has(elemName)) {\n    elemVal = mapVar.get(elemName);\n    elemVal++\n}\nmapVar.set(elemName, elemVal);\n// Write back to global. context\nglobal.set(\"mapVar\", mapVar, \"file\");\nnode.warn(me + \"elemVal = \" + elemVal);\n\n//Save in msg too for inspection\nmsg.simpleVar = simpleVar;\nmsg.mapVar = mapVar;\n\nnode.status({ fill: \"green\", shape: \"dot\", text: mapVar.get(\"firstElem\") + \", \" + simpleVar});\n\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 280,
        "y": 780,
        "wires": [
            [
                "d66a2aa90b3e0aeb"
            ]
        ]
    },
    {
        "id": "d66a2aa90b3e0aeb",
        "type": "debug",
        "z": "fe0c0f2c10548c43",
        "name": "full",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 445,
        "y": 780,
        "wires": [],
        "l": false
    },
    {
        "id": "a477bc90d0e43edf",
        "type": "function",
        "z": "fe0c0f2c10548c43",
        "name": "ReadGCVtest",
        "func": "/*\nRead global context variables\n*/\nconst me = \"[\" + node.name + \"] \";\n//node.warn(me);\n\n// Get global context map var if available\nvar mapVar = global.get(\"mapVar\", \"file\");\nif (mapVar === undefined) {\n    node.warn(me + \"mapVar undefined\");\n};\nlet elemName = \"firstElem\";\nlet elemVal = mapVar.get(elemName);\nnode.warn(me + \"elemVal = \" + elemVal);\n\n// Get global context simple var if available\nvar simpleVar = global.get(\"simpleVar\", \"file\");\nif (simpleVar === undefined) {\n    node.warn(me + \"simpleVar undefined\");\n};\nnode.warn(me + \"simpleVar = \" + simpleVar);\n\nnode.status({ fill: \"green\", shape: \"dot\", text: mapVar.get(elemName) + \", \" + simpleVar });\n\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 280,
        "y": 840,
        "wires": [
            [
                "d30739a470e34f95"
            ]
        ]
    },
    {
        "id": "d30739a470e34f95",
        "type": "debug",
        "z": "fe0c0f2c10548c43",
        "name": "full",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 445,
        "y": 840,
        "wires": [],
        "l": false
    }
]

I would be interested to see if you reproduce the behaviour. If so, has anybody any idea how to achieve saving map data in the global context file? Are maps really not serialisable? I would not really like to reprogram everything without using maps. In addition I would not know now which other data types might possibly not be supported ...

@zenofmud I think I did use ``` in my posts properly. It looks OK for me in the browser?! Am I missing something? How about the others reading this?

As always, greeting and thanks to all of you!

map is not serializable by JSON.stringify and is therefore (currently) unsupported.

1 Like

Wow. Thank you for your quick reply. This is understandable, but really frustrating. Do you have any idea for plans to support this?

Have the programming experts an advice how to get around this with the least effort?

Not a Node-RED issue, this is a JavaScript thing. However, you can convert a map to an array.

I understand this is a general JS topic. I have played around with converting to and from array. In fact this was easier than expected. The following does the trick back and forth even for my relatively complex sensorDevices map.

var sensorDevices = global.get("sensorDevices", "file");
const sensArr = Array.from(sensorDevices).map(([name, val]) => ({ name, val }));
const reMap = new Map();
for (let inx=0; inx<sensArr.length; inx++) {
    reMap.set(sensArr[inx].name, sensArr[inx].val)        
}
msg.sensorDevices = sensorDevices;
msg.sensArr = sensArr;
msg.reMap = reMap;
return msg;

Thanks again.

If sensorDevices in context is an Array, then you don't need the Array.from(sensorDevices).

Understood. But the whole topic is about a map in context. Had I chosen an array in the first place, I would not even have run into the problem at all. I chose maps because they seemed to me more appropriate from a pure clean programming point of view.

I thought the problem was that it is not possible to store a map in context, one has to store an array instead, therefore global.get("sensorDevices", "file") will be an array.

Actually it is possible to store and retrieve a map in context store. The only thing is that it is not stored in global context file, since it is not serialisable. As long as you do not restart Node Red everything works perfectly well. This is why I was caught by surprise.

Yes, you are right. I am not clear what your solution is then.

It is possible to do that in the memory store since that doesn't have to be serialised. It would fail for the file store I am fairly sure. But best not to rely on it still working in the future since who knows how the in-memory store might change in the future. Or you might one day choose to move the store to REDIS and then wonder why it wouldn't work.

The solution is to use maps as usual, but convert them to an array for saving across restarts. When I try to load the map and it is not there, I reconstruct the map as shown from the file context array that survived and continue as usual. So I do not need to rewrite my code other than saving as an array and restoring after restart. In my case saving from time to time suffices, so the overhead is not really an issue. If I needed to update the context array after every map update, that would be something to consider.

Good point, but I have to choose between rewriting quite some code or hope serialisation will come any time later. I have opted for the latter solution and will rewrite if I really have to. May be I am lucky. If not, tough luck, but it only hits myself :wink:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.