Sqlite SELECT OK from terminal, empty result from NR

I have quite complex SQL query.
From the terminal on the server everything is OK with the command
sqlite3 power.sqlite ".read sql.txt" (my query is stored in the sql.txt file).

Unfortunatelly from the NR the result of the query is: msg.payload : array[0] [ empty ]

[
    {
        "id": "ce0dde258edf86ae",
        "type": "sqlite",
        "z": "3ec042cc2b88ead8",
        "mydb": "a33f828b.b04e2",
        "sqlquery": "msg.topic",
        "sql": "create index if not exists power_jd on power (julianday(timestamp), total_kwh); with pwr(timestamp, reading, ratetoprior) as ( select julianday(timestamp), total_kwh, (select(c.total_kwh - p.total_kwh) / (julianday(c.timestamp) - julianday(p.timestamp)) from power as p where julianday(p.timestamp) < julianday(c.timestamp) order by julianday(p.timestamp) desc limit 1) from power as c order by julianday(timestamp) ), periods(timestamp) as ( select julianday(strftime('%Y-%m-%d %H', (min(timestamp)), '-1 month') || ':00:00.000') from pwr union all select julianday(datetime(timestamp, '+1 month')) from periods where timestamp < (select max(timestamp) from pwr) ), readings(timestamp, reading) as ( select timestamp, (select reading - (b.timestamp - p.timestamp) * ratetoprior from pwr as b where b.timestamp >= p.timestamp limit 1) as reading from periods as p where timestamp between(select min(timestamp) from pwr) and(select max(timestamp) from pwr) ), used(timestamp, kwh) as ( select timestamp, reading - lag(reading) over() from readings ) select strftime('%m.%Y', timestamp) as \"mm-yyy\", cast(kwh as int) as \"kwh\" from used where kwh is not null",
        "name": "",
        "x": 580,
        "y": 1140,
        "wires": [
            [
                "f0ee7a20a2b3b935"
            ]
        ]
    },
    {
        "id": "247626382557a383",
        "type": "function",
        "z": "3ec042cc2b88ead8",
        "name": "function 4",
        "func": "msg.topic = String.raw`create index if not exists power_jd on power (julianday(timestamp), total_kwh);\n\nwith pwr(timestamp, reading, ratetoprior) as\n(\n    select julianday(timestamp),\n        total_kwh,\n        (select(c.total_kwh - p.total_kwh) / (julianday(c.timestamp) - julianday(p.timestamp))\n                  from power as p\n                 where julianday(p.timestamp) < julianday(c.timestamp)\n              order by julianday(p.timestamp) desc\n                 limit 1)\n          from power as c\n      order by julianday(timestamp)\n     ),\nperiods(timestamp) as\n    (\n        select julianday(strftime('%Y-%m-%d %H', (min(timestamp)), '-1 month') || ':00:00.000')\n           from pwr\n      union all\n         select julianday(datetime(timestamp, '+1 month'))\n           from periods\n          where timestamp < (select max(timestamp) from pwr)\n     ),\nreadings(timestamp, reading) as\n    (\n        select timestamp,\n            (select reading - (b.timestamp - p.timestamp) * ratetoprior\n               from pwr as b\n              where b.timestamp >= p.timestamp\n              limit 1) as reading\n        from periods as p\n       where timestamp between(select min(timestamp) from pwr)\nand(select max(timestamp) from pwr)\n    ),\nused(timestamp, kwh) as\n    (\n        select timestamp,\n            reading - lag(reading) over()\n        from readings\n    )\n  select strftime('%m.%Y', timestamp) as \"mm-yyy\",\n    cast(kwh as int) as \"kwh\"\n    from used\n   where kwh is not null`;\n\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 340,
        "y": 1140,
        "wires": [
            [
                "ce0dde258edf86ae"
            ]
        ]
    },
    {
        "id": "07691c24cdc0a11a",
        "type": "inject",
        "z": "3ec042cc2b88ead8",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 140,
        "y": 1140,
        "wires": [
            [
                "247626382557a383"
            ]
        ]
    },
    {
        "id": "f0ee7a20a2b3b935",
        "type": "debug",
        "z": "3ec042cc2b88ead8",
        "name": "debug 5",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 800,
        "y": 1140,
        "wires": []
    },
    {
        "id": "a33f828b.b04e2",
        "type": "sqlitedb",
        "db": "/root/sqlite-data/power.sqlite",
        "mode": "RWC"
    }
]

My setup:
Sqlite 3.34.1
NR 3.0.2
node-red-node-sqlite
link to download database for testing: power.sqlite

Add a catch node connected to a debug node (set to display the complete msg object) and see if that shows anything.

Also remove all the `\NR from the query.

Nothing at all

What do you mean?

I have forget to mention simple query

SELECT * FROM POWER

works without any problem

Instead of using a function node to build the query, try using the template node

I'm not certain that nodejs SQLite driver understands this syntax. If it does then this is probably a path issue.

You can avoid that altogether though. Instead, simply use a file node to read your text file, move the payload in the topic, then pass that through to the SQLite node.

Thks for the hint, but the result is the same.
I wander which sqlite NR is using.
If I uninstall sqlite manualy on my machine apt remove sqlite3 the simple query SELECT * FROM POWER still works from the NR.
Below are my attempts.

  1. simple SQL query
  2. complex SQL query via text template node
  3. complex SQL query via function template
  4. complex SQL query via fixed statement in the sqlite node

Only simple query works.
sqlite is not installed manually on my server (not accessible from the command line on my server)

If I install sqlite manually on my server, complex query does not work from NR, but from the terminal on the server everything works as expected.

the database file is here: power.sqlite

[
    {
        "id": "ce0dde258edf86ae",
        "type": "sqlite",
        "z": "3ec042cc2b88ead8",
        "mydb": "a33f828b.b04e2",
        "sqlquery": "msg.topic",
        "sql": "",
        "name": "",
        "x": 800,
        "y": 1100,
        "wires": [
            [
                "f0ee7a20a2b3b935"
            ]
        ]
    },
    {
        "id": "247626382557a383",
        "type": "function",
        "z": "3ec042cc2b88ead8",
        "name": "msg.topic query",
        "func": "msg.topic = String.raw`create index if not exists power_jd on power (julianday(timestamp), total_kwh);\n\nwith pwr(timestamp, reading, ratetoprior) as\n(\n    select julianday(timestamp),\n        total_kwh,\n        (select(c.total_kwh - p.total_kwh) / (julianday(c.timestamp) - julianday(p.timestamp))\n                  from power as p\n                 where julianday(p.timestamp) < julianday(c.timestamp)\n              order by julianday(p.timestamp) desc\n                 limit 1)\n          from power as c\n      order by julianday(timestamp)\n     ),\nperiods(timestamp) as\n    (\n        select julianday(strftime('%Y-%m-%d %H', (min(timestamp)), '-1 month') || ':00:00.000')\n           from pwr\n      union all\n         select julianday(datetime(timestamp, '+1 month'))\n           from periods\n          where timestamp < (select max(timestamp) from pwr)\n     ),\nreadings(timestamp, reading) as\n    (\n        select timestamp,\n            (select reading - (b.timestamp - p.timestamp) * ratetoprior\n               from pwr as b\n              where b.timestamp >= p.timestamp\n              limit 1) as reading\n        from periods as p\n       where timestamp between(select min(timestamp) from pwr)\nand(select max(timestamp) from pwr)\n    ),\nused(timestamp, kwh) as\n    (\n        select timestamp,\n            reading - lag(reading) over()\n        from readings\n    )\n  select strftime('%m.%Y', timestamp) as \"mm-yyy\",\n    cast(kwh as int) as \"kwh\"\n    from used\n   where kwh is not null`;\n\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 520,
        "y": 1160,
        "wires": [
            [
                "ce0dde258edf86ae"
            ]
        ]
    },
    {
        "id": "07691c24cdc0a11a",
        "type": "inject",
        "z": "3ec042cc2b88ead8",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 140,
        "y": 1160,
        "wires": [
            [
                "247626382557a383"
            ]
        ]
    },
    {
        "id": "f0ee7a20a2b3b935",
        "type": "debug",
        "z": "3ec042cc2b88ead8",
        "name": "debug 5",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 1020,
        "y": 1100,
        "wires": []
    },
    {
        "id": "db673915dc44f66c",
        "type": "function",
        "z": "3ec042cc2b88ead8",
        "name": "simple query SELECT * FROM POWER",
        "func": "msg.topic = String.raw`select \n* from power`;\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 440,
        "y": 1080,
        "wires": [
            [
                "ce0dde258edf86ae"
            ]
        ]
    },
    {
        "id": "829e92c901243fd4",
        "type": "inject",
        "z": "3ec042cc2b88ead8",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 140,
        "y": 1080,
        "wires": [
            [
                "db673915dc44f66c"
            ]
        ]
    },
    {
        "id": "ee7d773fe9a4b153",
        "type": "catch",
        "z": "3ec042cc2b88ead8",
        "name": "",
        "scope": null,
        "uncaught": false,
        "x": 900,
        "y": 1040,
        "wires": [
            [
                "f0ee7a20a2b3b935"
            ]
        ]
    },
    {
        "id": "94ffcfb746384ecf",
        "type": "template",
        "z": "3ec042cc2b88ead8",
        "name": "text template in msg.topic",
        "field": "topic",
        "fieldType": "msg",
        "format": "handlebars",
        "syntax": "plain",
        "template": "create index if not exists power_jd on power (julianday(timestamp), total_kwh); with pwr(timestamp, reading, ratetoprior) as ( select julianday(timestamp), total_kwh, (select(c.total_kwh - p.total_kwh) / (julianday(c.timestamp) - julianday(p.timestamp)) from power as p where julianday(p.timestamp) < julianday(c.timestamp) order by julianday(p.timestamp) desc limit 1) from power as c order by julianday(timestamp) ), periods(timestamp) as ( select julianday(strftime('%Y-%m-%d %H', (min(timestamp)), '-1 month') || ':00:00.000') from pwr union all select julianday(datetime(timestamp, '+1 month')) from periods where timestamp < (select max(timestamp) from pwr) ), readings(timestamp, reading) as ( select timestamp, (select reading - (b.timestamp - p.timestamp) * ratetoprior from pwr as b where b.timestamp >= p.timestamp limit 1) as reading from periods as p where timestamp between(select min(timestamp) from pwr) and(select max(timestamp) from pwr) ), used(timestamp, kwh) as ( select timestamp, reading - lag(reading) over() from readings ) select strftime('%m.%Y', timestamp) as \"mm-yyy\", cast(kwh as int) as \"kwh\" from used where kwh is not null",
        "output": "str",
        "x": 490,
        "y": 1120,
        "wires": [
            [
                "ce0dde258edf86ae"
            ]
        ]
    },
    {
        "id": "b690b9397c1e64fd",
        "type": "inject",
        "z": "3ec042cc2b88ead8",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 140,
        "y": 1120,
        "wires": [
            [
                "94ffcfb746384ecf"
            ]
        ]
    },
    {
        "id": "c3f14e145fd45e79",
        "type": "sqlite",
        "z": "3ec042cc2b88ead8",
        "mydb": "a33f828b.b04e2",
        "sqlquery": "fixed",
        "sql": "create index if not exists power_jd on power (julianday(timestamp), total_kwh); with pwr(timestamp, reading, ratetoprior) as ( select julianday(timestamp), total_kwh, (select(c.total_kwh - p.total_kwh) / (julianday(c.timestamp) - julianday(p.timestamp)) from power as p where julianday(p.timestamp) < julianday(c.timestamp) order by julianday(p.timestamp) desc limit 1) from power as c order by julianday(timestamp) ), periods(timestamp) as ( select julianday(strftime('%Y-%m-%d %H', (min(timestamp)), '-1 month') || ':00:00.000') from pwr union all select julianday(datetime(timestamp, '+1 month')) from periods where timestamp < (select max(timestamp) from pwr) ), readings(timestamp, reading) as ( select timestamp, (select reading - (b.timestamp - p.timestamp) * ratetoprior from pwr as b where b.timestamp >= p.timestamp limit 1) as reading from periods as p where timestamp between(select min(timestamp) from pwr) and(select max(timestamp) from pwr) ), used(timestamp, kwh) as ( select timestamp, reading - lag(reading) over() from readings ) select strftime('%m.%Y', timestamp) as \"mm-yyy\", cast(kwh as int) as \"kwh\" from used where kwh is not null",
        "name": "",
        "x": 800,
        "y": 1200,
        "wires": [
            [
                "f0ee7a20a2b3b935"
            ]
        ]
    },
    {
        "id": "0c8e5351cf581324",
        "type": "inject",
        "z": "3ec042cc2b88ead8",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 340,
        "y": 1200,
        "wires": [
            [
                "c3f14e145fd45e79"
            ]
        ]
    },
    {
        "id": "a33f828b.b04e2",
        "type": "sqlitedb",
        "db": "/root/sqlite-data/power.sqlite",
        "mode": "RWC"
    }
]

I am not using that format in the NR. sqlite3 power.sqlite ".read sql.txt"

it Is my test attempt from the command line to prove my SQL query is OK and it works properly.

Use debug nodes to thoroughly examine the message topic going into the SQLite node.

Have a look at the SQL syntax you generate from your functions and template nodes.

You can even copy the debug output and test it on the command line.

One thing I do notice is that the database is in a root directory. Are you running node-red as root? Ps, don't!

As mentioned in my first post, the result of the debug node is: msg.payload : array[0] [ empty ]

  1. From the terminal/command line everything works.
  2. From the NR simple query works (it means the path to the database is OK).
  3. From the NR complex query does not work and does not return any error message (even "catch" node does not catch anything.

That does not mean you have carried out the test I suggested.

Fine, but I dont know that what you produce in the function node is EXACTLY the same as the content of the sql.txt file. I will assume that it is the same.

I didnt suggest the path was wrong for the database, but rather the sql.txt file - however, I have now loaded your flow and can see you dont use that syntax in node-red (so moot point)

It may be the NodeJS SQLite driver does not support multiple queries (I havent looked) - so what happens if you remove the create index ... ; part and simply run the CTE query part?

However, if the NodeJS SQLite driver DOES support multiple queries, try adding a semicolon at the end of the CTE since you are attempting to run multiple query operations - it might be something as simple as that.

BINGO.

Thanks for that. If I remove the first part of my multiple query SQL, everything works as expected.

Thanks a lot

@Petr I took a look at your database and the index is already defined for the table. Since the index exists and your create has a create index if not exists, nothing will happen.

If you drop the index you can recreate it but there will be no result returned because you are just creating it.

You can add an inject with msg.topic being set to drop index power_jd; and connect it to the sqlite node. Then you can run it and then run the original command to add the index to the power table schema.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.