Storing a value in text format_ using local pc as database

General
1

Hello,

I am creating a graph and want to store value for long period of time and for that I want to store the value in a database- doesn't want to loose it even if the PC got turned off which will most likely happen.

Since its only one number, I would like to store it in text formal using write node.

Everything goes well_ it stores a number and reads the same number from file until node -red is restarted. When it restarted it initializes with one.

Any suggestion how to overcome this??

[
    {
        "id": "d3e1163e876f1c6f",
        "type": "inject",
        "z": "146ab432f501375f",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": true,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "true",
        "payloadType": "bool",
        "x": 90,
        "y": 620,
        "wires": [
            [
                "efcea3e48f367448"
            ]
        ]
    },
    {
        "id": "efcea3e48f367448",
        "type": "function",
        "z": "146ab432f501375f",
        "name": "",
        "func": "\n\nif(msg.payload== true)\n{\n    var count=global.get('assembly')||0;\n    count = count+1;\n     msg.payload= count;\n    \nreturn msg;\n}\nelse\n{\n    return null;\n}",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 300,
        "y": 620,
        "wires": [
            [
                "161417918f469518"
            ]
        ]
    },
    {
        "id": "161417918f469518",
        "type": "file",
        "z": "146ab432f501375f",
        "name": "",
        "filename": "D:\\Node_red_write\\hello.txt",
        "appendNewline": true,
        "createDir": false,
        "overwriteFile": "true",
        "encoding": "none",
        "x": 520,
        "y": 620,
        "wires": [
            [
                "166fefa7d7af0e67"
            ]
        ]
    },
    {
        "id": "166fefa7d7af0e67",
        "type": "debug",
        "z": "146ab432f501375f",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 850,
        "y": 620,
        "wires": []
    },
    {
        "id": "db04ca1b12942cb4",
        "type": "file in",
        "z": "146ab432f501375f",
        "name": "",
        "filename": "D:\\Node_red_write\\hello.txt",
        "format": "utf8",
        "chunk": false,
        "sendError": false,
        "encoding": "none",
        "allProps": false,
        "x": 320,
        "y": 760,
        "wires": [
            [
                "853ca13cc3889c1f"
            ]
        ]
    },
    {
        "id": "0673468542d63908",
        "type": "inject",
        "z": "146ab432f501375f",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "1",
        "crontab": "",
        "once": true,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 110,
        "y": 760,
        "wires": [
            [
                "db04ca1b12942cb4"
            ]
        ]
    },
    {
        "id": "853ca13cc3889c1f",
        "type": "json",
        "z": "146ab432f501375f",
        "name": "",
        "property": "payload",
        "action": "",
        "pretty": false,
        "x": 510,
        "y": 760,
        "wires": [
            [
                "12d82746689c6c5c"
            ]
        ]
    },
    {
        "id": "12d82746689c6c5c",
        "type": "rbe",
        "z": "146ab432f501375f",
        "name": "",
        "func": "rbe",
        "gap": "",
        "start": "",
        "inout": "out",
        "septopics": true,
        "property": "payload",
        "topi": "topic",
        "x": 650,
        "y": 760,
        "wires": [
            [
                "83e5d9a9e3807bbe"
            ]
        ]
    },
    {
        "id": "83e5d9a9e3807bbe",
        "type": "function",
        "z": "146ab432f501375f",
        "name": "",
        "func": "\nglobal.set('assembly',msg.payload);\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 800,
        "y": 760,
        "wires": [
            [
                "09269eb007ac8fa1"
            ]
        ]
    },
    {
        "id": "09269eb007ac8fa1",
        "type": "debug",
        "z": "146ab432f501375f",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "payload",
        "targetType": "msg",
        "statusVal": "",
        "statusType": "auto",
        "x": 970,
        "y": 760,
        "wires": []
    },
    {
        "id": "8657f45eee6aad19",
        "type": "comment",
        "z": "146ab432f501375f",
        "name": "Simulating with boolean- true",
        "info": "",
        "x": 140,
        "y": 580,
        "wires": []
    },
    {
        "id": "8761750981b161ed",
        "type": "comment",
        "z": "146ab432f501375f",
        "name": "Extracting value from a pc and setting into global variable",
        "info": "",
        "x": 230,
        "y": 720,
        "wires": []
    }
]
2

Are the values timestamped?

If so, you could use InfluxDB and then recall the values between timeX and timeY, or even now. You can also use a Retention Policy for InfluxDB to store the values for certain period of time

If this is a route you could use, there are plenty of topics available on the Forum covering this.

1 Like
3

The pc will be a standalone and will not be communicating with external network. Do you know if its possible without influxdb, just with the local machine only?

4

Have you read up on persistent context?
https://nodered.org/docs/user-guide/context#context-stores

1 Like
5

Yes, its storing value into variables isnt it. I want to store restored even if the pc got restarted.

I believe context value will be erased when node red stopped.

6

may be write to a CSV file ?

[{"id":"e73ad588de171439","type":"inject","z":"7a196fc9029e392d","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":240,"y":360,"wires":[["dbf01a4280a355a6"]]},{"id":"a006353a01d13b7a","type":"file","z":"7a196fc9029e392d","name":"","filename":"c:\\temp\\test1.csv","filenameType":"str","appendNewline":true,"createDir":true,"overwriteFile":"false","encoding":"none","x":780,"y":360,"wires":[[]]},{"id":"5d9b99b9a9a195ec","type":"change","z":"7a196fc9029e392d","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"$string(payload, true)\t","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":560,"y":360,"wires":[["a006353a01d13b7a"]]},{"id":"dbf01a4280a355a6","type":"function","z":"7a196fc9029e392d","name":"function 1","func":"var one = Math.floor(Math.random() * 100);\nmsg.payload=one;\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":390,"y":360,"wires":[["5d9b99b9a9a195ec"]]}]
7

I would use the InfluxDB route if the dataset is over a period of time, InfluxDB should be able to be loaded onto the local PC.

Other than that, you could build an array and then when you have enough values, delete the first value, move the locations of the rest of the values up one, and add the new value at the end (FIFO - First In First Out). As to how to do this, I did use a flow at one time, and the relevant part for a FIFO is this:

[{"id":"a823c9cf.2a6178","type":"function","z":"b4973237.46ab6","name":"2 msg transport delay","func":"// stores messages in a fifo until the specified number have been received, \n// then releases them as new messages are received.\n// during the filling phase the earliest message is passed on each time \n// a message is received, but it is also left in the fifo\nvar fifoMaxLength = 2;\nvar fifo = context.get('fifo') || [];\n// push the new message onto the top of the array, messages are shifted down and\n// drop off the front\nvar length = fifo.push(msg);  // returns new length\nif (length > fifoMaxLength) {\n    newMsg = fifo.shift();\n} else {\n    // not full yet, make a copy of the msg and pass it on\n    var newMsg = JSON.parse(JSON.stringify(fifo[0]));\n}\ncontext.set('fifo', fifo);\nreturn newMsg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":258,"y":208,"wires":[["999a52c2.f465f"]]}]

BTW, I can't claim credit for this Node - iirc it was @Colin and it was built, in this case, for 2 values, but that could be increased.

1 Like
8

I am a fan of using databases for data storage. However, it would be ridiculous to install a database to store just one number.

No, you believe wrong.
This forum post might better explain persistent context, though it is 4 years old.

1 Like
9

Persistent context should certainly do what you want.
Alternatively, if you are using MQTT then put it in a Retained Topic in MQTT and when you restart the latest value will automatically be made available.

1 Like
10

again if you do this - you need to configure MQTT to save retained topics to disk, or it will also get lost if the PC reboots.

1 Like
11

Indeed. Though that is the default for mosquitto (on pi and ubuntu at least).

2 Likes
12

You simply need to configure a persistent store and use that for your value. No external services needed.

In settings.js:

    /** Context Storage
     * The following property can be used to enable context storage. The configuration
     * provided here will enable file-based context that flushes to disk every 30 seconds.
     * Refer to the documentation for further options: https://nodered.org/docs/api/context/
     *
     * Default is to use 'memory' only.
     * 'memory' and 'localfilesystem' are both synchronous.
     */
    contextStorage: {
        default: {
            module: 'memory',
        },
        file: {
            module: 'localfilesystem',
            // config: {
            //     flushInterval: '10', // default is 30s
            //     dir: '/home/user/.node-red/context',
            // },
        },
    },

You can define multiple file stores if you wish to have different flush intervals for different things. There are also alternative plug-in store types such as the REDIS one.

2 Likes
13

Hello,

Even though there are many solution, I found persistent context as the easiest solution for my issue.

Is there any space limitation for this feature?? for another usecase I would like to store 10 values in the same flow by using persistent context variable.

14

Thanks Colin. This is also a good solution.

15

Thanks a lot. I am not using influx DB yet, however thats my next topic to learn.

1 Like
16

Thanks for the solution.

17

You should have no problem.

1 Like
18

I have context variables with thousands of entries. you should remember, however, that it will not only be stored on-disk but also in memory. For reference, my current live server has 2.4Mb of content.

1 Like
19

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.