Bug in CSV node?

General
1

Hi all,
I read several sensors, and put the readings into an SQL file. With one measurement every 10 minutes, I should get some 52000 records per year.
I'm trying to output those readings via an SQL query to a CSV file using the CSV node.
I entered the column names in the 'Columns' property, and set 'Output' to 'Output once, until msg.reset'.
In my first tests it worked beautifully. However, when testing with more records I noticed the column headers are missing.
If I reduce the number of records with 'LIMIT ' I always get the headers back, even if 'LIMIT' is larger than the number of records.

Any thoughts?

2

Can you create a simple flow to demonstrate the problem please? Use an Inject node to inject a small dataset as produced by the database. You can get that by feeding it into a debug node and using the Copy Value button then paste it into a debug node in {} mode. Keep it down to just a few records obviously.

3

if you set ...until msg.reset then indeed the headers are only sent once.... until you send msg.reset...

so yes please to an example showing what you mean.

4

Unfortunately I can't reproduce the problem with a small dataset. First time I noticed I had approximately 19000 records.
The SQL query reads:
SELECT (field names) FROM (database) > WHERE date >= (one year ago);

When I send the output to a csv node the headers are missing.

If I change my query to:
SELECT (field names) FROM (database) > WHERE date >= (one year ago) LIMIT 20000;

I get my headers plus my 19000 records.

@dceejay I assumed msg.reset was activated when the node is ready.
I want my CSV starting with one header line, and I assumed I did it the right way. Was I wrong?

5

So yes the node is reset when you deploy
(or you could explicitly sent a msg.reset before you do your query to test it.)
And yes you did it the right way. You should get one set of headers only. But you would then need to reset before the next query.

6

Thanks, that did the trick.

Is there a maximum number of records that can be exported using the CSV node?

7

Depends how you feed it.. If it is one result object then the limit will be linked to the amount of system memory available.

8

Ok, thanks. Good to know, but it won't be a problem in this case.

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.