CSV Parser when triggering msg includes 'parts'

[RobC]

Hi everyone,
Thanks so much for a brilliant tool and community support.
I'm not sure if this is a bug or by design, I understand that when parsing a CSV to produce a message per line the CSV node will add msg.parts to the output.
What I found was that if the triggering message had a msg.parts it broke in some cases.
My use case was scan a folder for new files, split for each, fetch file data and send to CSV. If I selected 'message per row' from the CSV, I got what was expected. When 'a single message [array]' was selected I got only 1 object associated with the last line.
The only work around I could come up with was to recreate the input message in a function (var m={}; m.payload = msg.payload; return m; (setting msg.parts to null breaks CSV too)
This doesn't seem desirable but is this the way it should be handled?

Thanks, Rob

(not sure how to properly add flow o_0 but here is an example:) node-red v1.2.9

 [{"id":"728f95e1.0ce26c","type":"inject","z":"49c3e2f4.d02dcc","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[\"Col1,Col2\\n\\rV1,V2\\n\\rV3,V4\\n\\rV5,V6\"]","payloadType":"json","x":180,"y":540,"wires":[["ca800ba8.1cd258"]]},{"id":"ca800ba8.1cd258","type":"split","z":"49c3e2f4.d02dcc","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":330,"y":540,"wires":[["1da7046.b1972fc"]]},{"id":"1da7046.b1972fc","type":"csv","z":"49c3e2f4.d02dcc","name":"","sep":",","hdrin":true,"hdrout":"none","multi":"mult","ret":"\\r\\n","temp":"","skip":"0","strings":true,"include_empty_strings":false,"include_null_values":false,"x":490,"y":540,"wires":[["28f585be.71874a"]]},{"id":"28f585be.71874a","type":"debug","z":"49c3e2f4.d02dcc","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":670,"y":540,"wires":[]}]

[bakman2]

You can post code by using 3 backticks like this:

```
your code
```

The msg.parts is a special property that is added to keep track of the elements that belong together so that you can manipulate the individual objects and join them afterwards with a join node if required.

If you only get a single element in the array, it could be that the input csv is not completely valid csv (ie. like empty lines in between), when you select 'message per row' it might look ok, but there could still be invalid rows in the output.

so to test your theory, you could use the message per row and add a join node behind it in automatic mode and you should end up with array (ie the same as selecting 'single array' in the csv node)

[dceejay]

Doesn't seem quite right - if you want to raise an issue against it then I'm happy to take a look.
But first - what does your real data look like (when more than one item in the array before the split).

[RobC]

Thanks dceejay,
I was trying to create the simplest flow to demonstrate but the 'real-world' use case is where you would want to parse a number of CSV files that arrive in a folder.
The first part lists the files found in the folder and then a split node sends them through to the parser one by one.
It's this split node that inserts msg.parts (of course) but then that hangs around until we get to the load data and parse with CSV node. The msg.parts attribute seems to cause the behavior if we want an array out of the CSV (as in my case). You can't set msg.parts to null as CSV breaks (and that would harm the resultant join down the track anyway).

I can come up with a sample flow but it will be pointing to literal folders and expect there to be files, but I'll have a go at that today.

Thanks again for your quick reply. I'm an Aussie and its brilliant to see replies arrive over night
Cheers,
Rob

[dceejay]

Hi, you don't need to read actual files , just catch an example going into the split with a debug node and paste it. Thanks

[RobC]

ah ok. I'll mock one up directly. thanks.

[RobC]

Here is my sample flow.

it goes looking at C:\Temp for files that look like 'test.csv' (note fs-file-lister node) - this can be replaced by an inject, function or template that delivers ["C:\temp\test.csv","C:\temp\test2.csv"]
The two files are:
test.csv
col1,col2,col3
V1,V2,V3
Va,Vb,Vc
last,line,test

and
test2.csv
col1,col2,col3
1b,2b,3b
a,b,c
green,yellow,zombie
x,y,test2

When selected for 'Output a single message [array]' the CSV returns only the last line of both files:
[{"col1":"last","col2":"line","col3":"test"},{"col1":"x","col2":"y","col3":"test2"}]

[{"id":"22e1f11b.8536ae","type":"inject","z":"69a3eb1f.9d3404","name":"Periodic","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":220,"y":340,"wires":[["c5342392.4763b"]]},{"id":"c5342392.4763b","type":"fs-file-lister","z":"69a3eb1f.9d3404","name":"","start":"C:\\temp","pattern":"test*.csv","folders":"*","hidden":true,"lstype":"files","path":true,"single":true,"depth":0,"stat":false,"showWarnings":true,"x":400,"y":340,"wires":[["ea1699f3.1754c8","6e224270.cd0b3c"]]},{"id":"4338b90e.28bd28","type":"comment","z":"69a3eb1f.9d3404","name":"Fetch new files","info":"","x":320,"y":260,"wires":[]},{"id":"ea1699f3.1754c8","type":"debug","z":"69a3eb1f.9d3404","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":610,"y":260,"wires":[]},{"id":"e27a29aa.dbe898","type":"debug","z":"69a3eb1f.9d3404","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1250,"y":340,"wires":[]},{"id":"a55c956d.93b8a8","type":"file in","z":"69a3eb1f.9d3404","name":"Fetch File","filename":"","format":"utf8","chunk":false,"sendError":false,"encoding":"utf8","x":940,"y":340,"wires":[["fed36973.ed18a8"]]},{"id":"fed36973.ed18a8","type":"csv","z":"69a3eb1f.9d3404","name":"","sep":",","hdrin":true,"hdrout":"none","multi":"mult","ret":"\\r\\n","temp":"","skip":"0","strings":true,"include_empty_strings":false,"include_null_values":false,"x":1090,"y":340,"wires":[["e27a29aa.dbe898"]]},{"id":"6e224270.cd0b3c","type":"split","z":"69a3eb1f.9d3404","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":570,"y":340,"wires":[["9fc72b48.961548"]]},{"id":"9fc72b48.961548","type":"change","z":"69a3eb1f.9d3404","name":"","rules":[{"t":"set","p":"filename","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":750,"y":340,"wires":[["a55c956d.93b8a8"]]},{"id":"b48a4080.c7f46","type":"comment","z":"69a3eb1f.9d3404","name":"select array output - includes only last line","info":"","x":1100,"y":260,"wires":[]}]

if we clobber the msg and recreate with only 'payload'

it works (but seems an awful work around)

[{"id":"22e1f11b.8536ae","type":"inject","z":"69a3eb1f.9d3404","name":"Periodic","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":220,"y":340,"wires":[["c5342392.4763b"]]},{"id":"c5342392.4763b","type":"fs-file-lister","z":"69a3eb1f.9d3404","name":"","start":"C:\\temp","pattern":"test*.csv","folders":"*","hidden":true,"lstype":"files","path":true,"single":true,"depth":0,"stat":false,"showWarnings":true,"x":400,"y":340,"wires":[["ea1699f3.1754c8","6e224270.cd0b3c"]]},{"id":"4338b90e.28bd28","type":"comment","z":"69a3eb1f.9d3404","name":"Fetch new files","info":"","x":320,"y":260,"wires":[]},{"id":"ea1699f3.1754c8","type":"debug","z":"69a3eb1f.9d3404","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":610,"y":260,"wires":[]},{"id":"e27a29aa.dbe898","type":"debug","z":"69a3eb1f.9d3404","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":810,"y":460,"wires":[]},{"id":"a55c956d.93b8a8","type":"file in","z":"69a3eb1f.9d3404","name":"Fetch File","filename":"","format":"utf8","chunk":false,"sendError":false,"encoding":"utf8","x":320,"y":460,"wires":[["69f7b800.6a3728"]]},{"id":"fed36973.ed18a8","type":"csv","z":"69a3eb1f.9d3404","name":"","sep":",","hdrin":true,"hdrout":"none","multi":"mult","ret":"\\r\\n","temp":"","skip":"0","strings":true,"include_empty_strings":false,"include_null_values":false,"x":630,"y":460,"wires":[["e27a29aa.dbe898"]]},{"id":"6e224270.cd0b3c","type":"split","z":"69a3eb1f.9d3404","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":570,"y":340,"wires":[["9fc72b48.961548"]]},{"id":"9fc72b48.961548","type":"change","z":"69a3eb1f.9d3404","name":"","rules":[{"t":"set","p":"filename","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":750,"y":340,"wires":[["a55c956d.93b8a8"]]},{"id":"b48a4080.c7f46","type":"comment","z":"69a3eb1f.9d3404","name":"clobber msg.parts here","info":"","x":440,"y":520,"wires":[]},{"id":"69f7b800.6a3728","type":"function","z":"69a3eb1f.9d3404","name":"","func":"var m = {\"payload\":msg.payload};\nreturn m;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":480,"y":460,"wires":[["fed36973.ed18a8"]]}]

Thanks again for taking a look.

[zenofmud]

Try this, put a change node between the function' and csv` nodes with this setting:

[RobC]

Thanks zenofmud,
Yes, that's what we normally would do when cascading splits (so we can line up the joins).
Seems msg.parts coming in to CSV parser is possibly something to document (I think it would only care about that property if it was converting from array/object to CSV not the other way around?).

For my use, I'd rather a config switch (array/object -> CSV or visa versa or 2 nodes). Is there a great benefit to having the all-in-one 'auto detect' CSV node?

Cheers,
Rob

P.S. Node-red is brilliant, you guys do a fantastic job. My catch phrase is "you can do that in node"

[dceejay]

Think I have a handle on it... fix pushed to dev branch - should make it into 1.3.