Node-RED freezes while sqlite query

no, it depends on what order the nodes were wired. This is not something to be concerned about usually but I can understand your curiosity having previously had a flow that was heavy and so you could visualise the effects.

yes. see previous response (its the same question and same answer essentially)

Yes. BUT. if function 16 is blocking (has no async / is a hard loop / no callbacks etc) then you might not see that debug until function 16 is done due to a combination of how the flow is ordered, how the NR core is optimised (i.e. it bundles debug messages to avoid excessive comms traffic), the state of the NodeJS event loop etc etc etc.

The bottleneck should be the CSV node. To confirm this, first delete the CSV node, and run the query without CSV conversion.

After finding the bottleneck, you can delete the CSV node, and write a simple CSV conversion function inside function 16 instead. This will speed up the process significantly. An example of the CSV conversion function is as the following.

function ConvertToCSV(objArray, str) {
    var array = typeof objArray != 'object' ? JSON.parse(objArray) : objArray;
    for (let i = 0; i < array.length; i++) {
        let line = '';
        for (var index in array[i]) {
            if (line != '') line += ','

            line += array[i][index];
        }
        str += line + '\r\n';
    }
    return str;
}

Or just remove the wire connecting the sql node to the following function node.

So what string conversion is taking place inside function 16? If you are trying to reformat the datetime information, that could be costly.

In general, output value formatting should take place inside the SQL statement, so you can skip that function code and send the query results directly to the CSV node in streaming mode.

Of course, you will have to learn some SQL syntax, but I've almost always found that databases are more efficient at processing the raw data than whatever code I write to loop over the results. And it doesn't tie up the nodejs task queue since it happens outside of node-red in the database itself.

2 Likes

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.