# CPU spikes while writting data to context

**URL:** <https://discourse.nodered.org/t/cpu-spikes-while-writting-data-to-context/62259>\
**Category:** General\
**Created:** [5 May 2022 22:12 UTC](https://discourse.nodered.org/t/cpu-spikes-while-writting-data-to-context/62259 "2022-05-05T22:12:37Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Aivaras1](https://avatars.discourse-cdn.com/v4/letter/a/8797f3/32.png) [@Aivaras1](https://discourse.nodered.org/u/Aivaras1)\
**Post date:** [5 May 2022 22:12 UTC](https://discourse.nodered.org/t/cpu-spikes-while-writting-data-to-context/62259/1 "2022-05-05T22:12:37Z")

</div>

Hello,

So I have a problem. I am making data accumulation for graph. Data is received from modbus communication. Problems occurred when I tried to accumulate data. Random CPU spikes (95-98%) Does someone has any suggestion how to solve a problem ?

Hardware - beaglebone green gateway

Also I removed comments from nano/var/lib/node-red/.node-red/settings.js

```
contextStorage: {
    default: {
        module:"localfilesystem"
    },
},

```

Code for data accumulation

```auto
[
    {
        "id": "e2eb84e67135a1d1",
        "type": "tab",
        "label": "Flow 1",
        "disabled": false,
        "info": "",
        "env": []
    },
    {
        "id": "36994b8b41eac99e",
        "type": "function",
        "z": "e2eb84e67135a1d1",
        "name": "Data",
        "func": "const data = global.get(\"ALLDATA\");\n\nmsg.payload = {\t\n\"Timestamp\": new Date().getTime(),\n\"R5004\": Number.parseFloat(data.R5004.toFixed(2)),\n\"R5007\": Number.parseFloat(data.R5007.toFixed(2)),\n\"R5012\": Number.parseFloat(data.R5012.toFixed(2)),\n\"R5014\": Number.parseFloat(data.R5014.toFixed(2)),\n\"R5020\": Number.parseFloat(data.R5020.toFixed(2)),\n\"R5021\": Number.parseFloat(data.R5021.toFixed(2)),\n\"R5022\": Number.parseFloat(data.R5022.toFixed(2)),\n\"R5023\": Number.parseFloat(data.R5023.toFixed(2)),\n\"R5024\": Number.parseFloat(data.R5024.toFixed(2)),\n\"R5025\": Number.parseFloat(data.R5025.toFixed(2)),\n\"R5026\": Number.parseFloat(data.R5026.toFixed(2)),\n\"R5028\": Number.parseFloat(data.R5028.toFixed(2)),\n\"R5029\": Number.parseFloat(data.R5029.toFixed(2)),\n\"R5030\": Number.parseFloat(data.R5030.toFixed(2)),\n\"R5031\": Number.parseFloat(data.R5031.toFixed(2)),\n\"R5032\": Number.parseFloat(data.R5032.toFixed(2)),\n\"R5033\": Number.parseFloat(data.R5033.toFixed(2)),\n\"R5034\": Number.parseFloat(data.R5034.toFixed(2)),\n\"R5035\": Number.parseFloat(data.R5035.toFixed(2)),\n\"R5037\": Number.parseFloat(data.R5037.toFixed(2)),\n\"R5038\": Number.parseFloat(data.R5038.toFixed(2)),\n\"R5039\": Number.parseFloat(data.R5039.toFixed(2)),\n\"R5044\": Number.parseFloat(data.R5044.toFixed(2)),\n\"R5045\": Number.parseFloat(data.R5045.toFixed(2)),\n};\n\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 590,
        "y": 540,
        "wires": [
            [
                "ca9d2154a7fcb14d"
            ]
        ]
    },
    {
        "id": "f17d37076ba69816",
        "type": "inject",
        "z": "e2eb84e67135a1d1",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "1",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 430,
        "y": 540,
        "wires": [
            [
                "36994b8b41eac99e"
            ]
        ]
    },
    {
        "id": "ca9d2154a7fcb14d",
        "type": "function",
        "z": "e2eb84e67135a1d1",
        "name": "",
        "func": "const measures = context.get(\"DATA\") || [];\n\n\n\nif(measures.length >= 86400) {\n\n measures.pop();\n\n}\n\n\n\nmeasures.unshift(msg.payload);\n\ncontext.set(\"DATA\", measures);\n\nmsg.payload = measures;\n\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 760,
        "y": 540,
        "wires": [
            []
        ]
    }
]

```

[flows.json](https://discourse.nodered.org/uploads/short-url/vgx9pMRouJal5gMRAmWsRSgETAz.json) (3.1 KB)

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [5 May 2022 22:25 UTC](https://discourse.nodered.org/t/cpu-spikes-while-writting-data-to-context/62259/2 "2022-05-05T22:25:54Z")

</div>

> [@Aivaras1](#):
>
> Hardware - beaglebone green gateway

Does this use an SD-Card for storage? Also, have you checked your filing system for errors? Final question, how large is the output data getting?

It isn't uncommon to see spikes on output. Especially with SD-Card interfaces which generally have quite limited capabilities. The cards can also be limited and cheap cards in particular can end up with large numbers of errors. Spikes are worse of course if the data is getting large.

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [5 May 2022 23:03 UTC](https://discourse.nodered.org/t/cpu-spikes-while-writting-data-to-context/62259/3 "2022-05-05T23:03:08Z")

</div>

From my mobile phone, it looks like you are setting up to 84000 elements of approx 40 float values. At a rough guess that's 16MB of data to convert to JSON when the context storage flushes to file (that's what happens under the hood)

Additionally, the JSON stringifier it uses is not the built in one because it has to check for circular references etc. In simple terms it's slower & more cpu intensive than you imagine.

I would not hold this much data in file backed context. I'd write it to database.

---

<div class="post-metadata">

**Author:** ![Aivaras1](https://avatars.discourse-cdn.com/v4/letter/a/8797f3/32.png) [@Aivaras1](https://discourse.nodered.org/u/Aivaras1)\
**Post date:** [6 May 2022 05:22 UTC](https://discourse.nodered.org/t/cpu-spikes-while-writting-data-to-context/62259/4 "2022-05-06T05:22:38Z")

</div>

Hi, I am using internal Emmc. File size is about 16MB. I am attaching what I got from tracing function.

 ![Screenshot_1](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/1/c/1cf160ade2751ca08e42ad71568bf7462e62d0c7.png)  
 ![Screenshot_2](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/0/9/09c0e0921dd2dac969d7187402c7e10768b0e0ad.png)

---

<div class="post-metadata">

**Author:** ![Aivaras1](https://avatars.discourse-cdn.com/v4/letter/a/8797f3/32.png) [@Aivaras1](https://discourse.nodered.org/u/Aivaras1)\
**Post date:** [6 May 2022 05:24 UTC](https://discourse.nodered.org/t/cpu-spikes-while-writting-data-to-context/62259/5 "2022-05-06T05:24:04Z")

</div>

So maybe I need to use SQLite ?  
Because I can't use external database for that.

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [6 May 2022 08:09 UTC](https://discourse.nodered.org/t/cpu-spikes-while-writting-data-to-context/62259/6 "2022-05-06T08:09:31Z")

</div>

I'd be more interested in memory and swap utilisation.

> [@Aivaras1](#):
>
> maybe I need to use SQLite

Perhaps. And maybe think whether you can restructure the data into a relational table. If you can, then you will be better off when using SQLite because otherwise you still have to stringify/parse the JSON.

If you were already running something like memcached, that might also work.

Also, depending on exactly what you are trying to achieve, even running InfluxDB and moving the charts to Grafana might be better.

---

<div class="post-metadata">

**Author:** ![davidz](https://avatars.discourse-cdn.com/v4/letter/d/a88e57/32.png) [@davidz](https://discourse.nodered.org/u/davidz)\
**Post date:** [6 May 2022 17:02 UTC](https://discourse.nodered.org/t/cpu-spikes-while-writting-data-to-context/62259/7 "2022-05-06T17:02:56Z")

</div>

Besides @TotallyInformation and @Steve-Mcl suggestions, you can use memory storage to optimize the performance. In your function node, you are writing 16MB data to EMMC every 30 seconds in default. This can affect the system performance and reduce EMMC life significantly.

For large-size temporary data, use memory storage. You can modify "setting.js" as follows:

```auto
   contextStorage: {
      default: "file",
      memoryOnly: { module: 'memory' },
      file: { module: 'localfilesystem' }
    },

```

Then in your function call, use the following to get temporary data from the memory:

```auto
var measures = context.get("DATA",'memoryOnly');
if(measures==undefined)
{
    measures=[];
    context.set('DATA',measures,'memoryOnly');
}

```

There is no need to write context using "context.set". Since in Node-RED, array is called by reference in context. When you update measures, "DATA" is updated automatically.

This way, you can avoid writing/reading from EMMC, boost your performance and prolong your EMMC life.

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [6 May 2022 17:10 UTC](https://discourse.nodered.org/t/cpu-spikes-while-writting-data-to-context/62259/8 "2022-05-06T17:10:08Z")

</div>

I also always forget to say that you can alter the writeback period for the file storage. That might be an option though obviously there is a danger that you might loose some data.

---

<div class="post-metadata">

**Author:** ![Aivaras1](https://avatars.discourse-cdn.com/v4/letter/a/8797f3/32.png) [@Aivaras1](https://discourse.nodered.org/u/Aivaras1)\
**Post date:** [9 May 2022 07:30 UTC](https://discourse.nodered.org/t/cpu-spikes-while-writting-data-to-context/62259/9 "2022-05-09T07:30:30Z")

</div>

So context function will look like this ? Because I need only 24hours of data

```auto
var measures = context.get("DATA",'memoryOnly');
if(measures==undefined)
{
    measures=[];
    context.set('DATA',measures,'memoryOnly');
}

if(measures.length >= 86400) {

  measures.pop();

}

measures.unshift(msg.payload);

//context.set("DATA", measures);

msg.payload = measures;

return msg;

```

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [9 May 2022 08:11 UTC](https://discourse.nodered.org/t/cpu-spikes-while-writting-data-to-context/62259/10 "2022-05-09T08:11:51Z")

</div>

> [@Aivaras1](#):
>
> ```auto
> var measures = context.get("DATA",'memoryOnly');
> if(measures==undefined)
> {
> measures=[];
> context.set('DATA',measures,'memoryOnly');
> }
> 
> ```

You can write that as:

```auto
const measures = context.get("DATA",'memoryOnly') || []

```

You only need to do the set once at the end.

---

<div class="post-metadata">

**Author:** ![Aivaras1](https://avatars.discourse-cdn.com/v4/letter/a/8797f3/32.png) [@Aivaras1](https://discourse.nodered.org/u/Aivaras1)\
**Post date:** [9 May 2022 08:15 UTC](https://discourse.nodered.org/t/cpu-spikes-while-writting-data-to-context/62259/11 "2022-05-09T08:15:21Z")

</div>

So something like this ?

```auto
const measures = context.get("DATA",'memoryOnly') || []

if(measures.length >= 86400) {

  measures.pop();

}

measures.unshift(msg.payload);

context.set('DATA',measures,'memoryOnly');

msg.payload = measures;

return msg;

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [8 July 2022 08:16 UTC](https://discourse.nodered.org/t/cpu-spikes-while-writting-data-to-context/62259/12 "2022-07-08T08:16:09Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.