# Sqlite large file import - best practice?

**URL:** <https://discourse.nodered.org/t/sqlite-large-file-import-best-practice/54753>\
**Category:** General\
**Tags:** database\
**Created:** [5 December 2021 17:11 UTC](https://discourse.nodered.org/t/sqlite-large-file-import-best-practice/54753 "2021-12-05T17:11:29Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [5 December 2021 17:11 UTC](https://discourse.nodered.org/t/sqlite-large-file-import-best-practice/54753/1 "2021-12-05T17:11:29Z")

</div>

I have a large (700MB+) tab separated file and want to insert this into a sqlite db.

On the commandline I could do a dot command and perform an import using mode tabs or read commands, which is extremely fast.

Let's assume I only have node-red + the sqlite node available.

What could be the best (performant) approach to get the data into the db ?

The sqlite documentation describes that it can perform 50K inserts per second using begin/commit transactions. Can node-red split the file into 50K lines/array elements (which in turn also need some sanitizing while looping over them). Are there bottlenecks I need to take into account somewhere ?

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [8 December 2021 14:11 UTC](https://discourse.nodered.org/t/sqlite-large-file-import-best-practice/54753/2 "2021-12-08T14:11:49Z")

</div>

Anyone has ideas ?

Currently I am running a bash script, but would like some more node-red control when processing.

---

<div class="post-metadata">

**Author:** ![zenofmud](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/zenofmud/32/316_2.png) [@zenofmud](https://discourse.nodered.org/u/zenofmud)\
**Post date:** [8 December 2021 18:33 UTC](https://discourse.nodered.org/t/sqlite-large-file-import-best-practice/54753/3 "2021-12-08T18:33:10Z")

</div>

> [@bakman2](#):
>
> On the commandline I could do a dot command and perform an import using mode tabs or read commands, which is extremely fast.

What about using an `exec` node?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [6 February 2022 18:33 UTC](https://discourse.nodered.org/t/sqlite-large-file-import-best-practice/54753/4 "2022-02-06T18:33:44Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.