# Using Node-RED to transfer data from InfluxDB 1.7 to 2.3, but receiving ENOBUFS error when dataset is too large

**URL:** <https://discourse.nodered.org/t/using-node-red-to-transfer-data-from-influxdb-1-7-to-2-3-but-receiving-enobufs-error-when-dataset-is-too-large/65532>\
**Category:** General\
**Created:** [24 July 2022 14:19 UTC](https://discourse.nodered.org/t/using-node-red-to-transfer-data-from-influxdb-1-7-to-2-3-but-receiving-enobufs-error-when-dataset-is-too-large/65532 "2022-07-24T14:19:48Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![grant1](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/grant1/32/49890_2.png) [@grant1](https://discourse.nodered.org/u/grant1)\
**Post date:** [24 July 2022 14:19 UTC](https://discourse.nodered.org/t/using-node-red-to-transfer-data-from-influxdb-1-7-to-2-3-but-receiving-enobufs-error-when-dataset-is-too-large/65532/1 "2022-07-24T14:19:48Z")

</div>

I am using Node-RED to convert my InfluxDB 1.7 data to populate InfluxDB 2.3 using the attached flow.  
[Influx1.7 to 2.3.json](https://discourse.nodered.org/uploads/short-url/eGTpGnFONYa8sYFEz5hX343HxWh.json) (2.7 KB)

In both databases, the frequency is 1 reading per minute. If I run the flow and limit the query (to retrieve the data from 1.7) for, say, a 2-hour window, it works perfectly:

Influx 1.7 (viewed in Grafana)

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/c/7/c79c18c8d1d64da3a083275a904bf6be8542fa03.png)

Influx 2.3 (viewed in Grafana)

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/e/0/e0fdd9503a05959f0200d3e9d8769902f9adb0fe.png)

However, when I widen up the query to a full month's worth of data, I get the following error after a minute or so (and only a portion of the data appearing in 2.3).

![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/6/c/6c1e6dd215ca871beec915fb6ac947a198e6972d.png)

I Googled the ENOBUFS error and found [some discussions & solutions](https://stackoverflow.com/questions/44679614/can-someone-explain-an-enobufs-error), notably to throttle the data flow to get the job done (apparently my system's memory can no longer handle the buffer's size). I put a 5-second delay node where the blue circle is below, but the ENOBUFS error still happens and only part of the data makes the jump to 2.3.

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/5/8/58965eff8163e039b0df7bec1249989968d947f9.png)

Does it make sense that the ENOBUFS error is happening when writing the data to InfluxDB 2.0? Maybe a delay node is not the right solution?

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [24 July 2022 14:47 UTC](https://discourse.nodered.org/t/using-node-red-to-transfer-data-from-influxdb-1-7-to-2-3-but-receiving-enobufs-error-when-dataset-is-too-large/65532/2 "2022-07-24T14:47:15Z")

</div>

For future reference, when posting a flow then paste it directly here, using the `</>` button at the top of the forum entry window when pasting it.

I don't see a delay node in the flow.

As I understand it, for a months worth of data, you are fetching 43200 values and then sending all of them flat out one after another to the influx out node. I guess that is overloading something somewhere. You would be much better to write them out in one go using the influx batch node. That is what it is for.

Does influx not provide tools or techniques for migrating to 2.0?

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [24 July 2022 14:50 UTC](https://discourse.nodered.org/t/using-node-red-to-transfer-data-from-influxdb-1-7-to-2-3-but-receiving-enobufs-error-when-dataset-is-too-large/65532/3 "2022-07-24T14:50:10Z")

</div>

> [@Colin](#):
>
> Does influx not provide tools or techniques for migrating to 2.0?

Answering my own question [https://docs.influxdata.com/influxdb/v2/install/upgrade/v1-to-v2/](https://docs.influxdata.com/influxdb/v2.3/upgrade/v1-to-v2/)

---

<div class="post-metadata">

**Author:** ![grant1](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/grant1/32/49890_2.png) [@grant1](https://discourse.nodered.org/u/grant1)\
**Post date:** [24 July 2022 15:00 UTC](https://discourse.nodered.org/t/using-node-red-to-transfer-data-from-influxdb-1-7-to-2-3-but-receiving-enobufs-error-when-dataset-is-too-large/65532/4 "2022-07-24T15:00:19Z")

</div>

Thanks @Colin. When I uploaded my flow as an attachment (instead of pasting directly), I thought "that's different", and of course now I realize I did not follow the correct protocol. Anyway, I elected to not use the tools that you linked to because I am actually massaging my data considerably (adding / removing tags, changing field names, etc.) and I find using function nodes for that is much easier for me to work with. The flow that I posted did not include the delay node (I tried that, but it did not make any difference). And you correct, 43300 values for about 1 month's worth of data.

I will try to InfluxDB batch node, which admittedly I have never used. Will report here with some results.

---

<div class="post-metadata">

**Author:** ![grant1](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/grant1/32/49890_2.png) [@grant1](https://discourse.nodered.org/u/grant1)\
**Post date:** [24 July 2022 15:22 UTC](https://discourse.nodered.org/t/using-node-red-to-transfer-data-from-influxdb-1-7-to-2-3-but-receiving-enobufs-error-when-dataset-is-too-large/65532/5 "2022-07-24T15:22:52Z")

</div>

OK, tried the batch node using this flow

```auto
[{"id":"262a3923.e7b216","type":"influxdb in","z":"e34b3a780748064f","influxdb":"eeb221fb.ab27f","name":"","query":"SELECT last(\"HTFN1Z1T\") FROM \"stations\" WHERE time >= 1609480800000ms and time <= 1610773199000ms GROUP BY time(1m) fill(null)","rawOutput":false,"precision":"ms","retentionPolicy":"","org":"my-org","x":340,"y":280,"wires":[["467fc95d165f998e"]]},{"id":"803d82f.ff80f8","type":"inject","z":"e34b3a780748064f","name":"","repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":240,"y":200,"wires":[["262a3923.e7b216"]]},{"id":"467fc95d165f998e","type":"split","z":"e34b3a780748064f","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":450,"y":360,"wires":[["14f2646498b1cb03"]]},{"id":"14f2646498b1cb03","type":"function","z":"e34b3a780748064f","name":"","func":"msg.payload = [\n {\n measurement:\"SingleRotaryFurnaceZoneData\",\n fields: {\n temperature: msg.payload.last\n },\n tags:{\n MeasType:\"actual\",\n EquipNumber:\"1\",\n EquipZone:\"zone1\"\n },\n timestamp: msg.payload.time\n }\n];\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":580,"y":480,"wires":[["31fda5cb865c09ab"]]},{"id":"31fda5cb865c09ab","type":"influxdb batch","z":"e34b3a780748064f","influxdb":"e55f0c8e.1659f","precision":"","retentionPolicy":"","name":"","database":"database","precisionV18FluxV20":"ms","retentionPolicyV18Flux":"","org":"heattreat","bucket":"junkbucket","x":830,"y":520,"wires":[]},{"id":"eeb221fb.ab27f","type":"influxdb","hostname":"172.31.19.130","port":"8086","protocol":"http","database":"DST","name":"test","usetls":false,"tls":"d50d0c9f.31e858","influxdbVersion":"1.x","url":"http://localhost:8086","rejectUnauthorized":true},{"id":"e55f0c8e.1659f","type":"influxdb","hostname":"3.17.31.71","port":"8086","protocol":"http","database":"DST","name":"","usetls":false,"tls":"","influxdbVersion":"2.0","url":"http://192.168.10.25:8086","rejectUnauthorized":false},{"id":"d50d0c9f.31e858","type":"tls-config","name":"","cert":"","key":"","ca":"","certname":"","keyname":"","caname":"","servername":"","verifyservercert":false}]

```

and got the same ENOBUFS error. I changed the timespan down to 15 days (21600 values) and got the same error. Moved it down to 5 days (7200 values) and it worked.

I think I understand the concept that I am sending too many values at a time. Perhaps a loop or something that sends 5 days at a time?

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [24 July 2022 16:05 UTC](https://discourse.nodered.org/t/using-node-red-to-transfer-data-from-influxdb-1-7-to-2-3-but-receiving-enobufs-error-when-dataset-is-too-large/65532/6 "2022-07-24T16:05:52Z")

</div>

> [@grant1](#):
>
> tried the batch node using this flow

You are still writing them one at a time. You can remove the split node and convert the whole input array into the form required by the batch node and output the whole set of data in one write operation. I don't know if there is a limit to how many you can write in one operation.

---

<div class="post-metadata">

**Author:** ![grant1](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/grant1/32/49890_2.png) [@grant1](https://discourse.nodered.org/u/grant1)\
**Post date:** [24 July 2022 18:23 UTC](https://discourse.nodered.org/t/using-node-red-to-transfer-data-from-influxdb-1-7-to-2-3-but-receiving-enobufs-error-when-dataset-is-too-large/65532/7 "2022-07-24T18:23:46Z")

</div>

I had not realized I was writing the values one at a time until you pointed that out. Thanks.

I removed the split node and started by first examining the output from the Influx 1.7 query. I can see it is an array with 21540 objects, with each object being:

`payload[*].time` is the timestamp of the value  
and  
`payload[*].last` is the value  
_(pardon the use of the `*`...it is meant to indicate all of the 21540 values in the array)_

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/b/9/b9fe113d0319fb5ca5470c117e19ae9b9f9611ee.png)

And the Influx batch node format requires me to simply insert these two values here:  
 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/4/b/4b8f746cf6391538f1b9c69a38cba96e10de5e05.png)

How does one convert this whole array so that the function node can take the values from payload.last and payload.time? I tried some JSONata expressions (in a change node, before the function node) but nothing worked.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [24 July 2022 20:39 UTC](https://discourse.nodered.org/t/using-node-red-to-transfer-data-from-influxdb-1-7-to-2-3-but-receiving-enobufs-error-when-dataset-is-too-large/65532/8 "2022-07-24T20:39:15Z")

</div>

Have a look at Array.map() which will transform a complete array, given a function that is applied to each element.

---

<div class="post-metadata">

**Author:** ![grant1](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/grant1/32/49890_2.png) [@grant1](https://discourse.nodered.org/u/grant1)\
**Post date:** [24 July 2022 23:52 UTC](https://discourse.nodered.org/t/using-node-red-to-transfer-data-from-influxdb-1-7-to-2-3-but-receiving-enobufs-error-when-dataset-is-too-large/65532/9 "2022-07-24T23:52:33Z")

</div>

Thank you @Colin. I needed some more time to think about this, and I got it all working well using a single function node containing this:

```auto
msg.payload = msg.payload.map(function(value) {
    return {
        measurement: "SingleRotaryFurnaceZoneData",
        fields: {
            temperature:value.last,
        },
        tags:{
            MeasType:"actual",
            EquipNumber:"1",
            EquipZone:"zone1"
        },
        timestamp: value.time
    };
});
return msg;

```

 ![image](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/6/0/600f5c73031413da1ee22392c254139e8a1b46f2.png)

An array with 378,000 objects (about 9.5 months of data, with 1 reading per minute) took \< 10 seconds to import into Influx 2.3 using the batch node.

---

<div class="post-metadata">

**Author:** ![Colin](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/colin/32/17040_2.png) [@Colin](https://discourse.nodered.org/u/Colin)\
**Post date:** [25 July 2022 08:30 UTC](https://discourse.nodered.org/t/using-node-red-to-transfer-data-from-influxdb-1-7-to-2-3-but-receiving-enobufs-error-when-dataset-is-too-large/65532/10 "2022-07-25T08:30:25Z")

</div>

Excellent.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [8 August 2022 08:30 UTC](https://discourse.nodered.org/t/using-node-red-to-transfer-data-from-influxdb-1-7-to-2-3-but-receiving-enobufs-error-when-dataset-is-too-large/65532/11 "2022-08-08T08:30:34Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.