CSV and limiting output

[Tinbum1]

I have a flow that accesses date on the web that is in CSV format. The CSV is updated with new information added to the first few rows. That is all I am interested in.

I'm not sure why my flow won't work and only allow the first 100 messages through.

[{"id":"8be23377afc3327e","type":"tab","label":"Flow 3","disabled":false,"info":"","env":[]},{"id":"6e75254b2a6dbc45","type":"inject","z":"8be23377afc3327e","name":"Test","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":310,"y":200,"wires":[["f5d13f178f205ce6"]]},{"id":"f5d13f178f205ce6","type":"http request","z":"8be23377afc3327e","name":"Test","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://data.nationalgrideso.com/backend/dataset/22bc9bdd-4919-439a-95b8-5213bb3bdf4c/resource/a2dd7992-32d5-4723-b0f5-273caaecbde1/download/service-update-industry-notifications-test150220231030.csv","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"x":430,"y":200,"wires":[["117a41275d1be480"]]},{"id":"117a41275d1be480","type":"csv","z":"8be23377afc3327e","name":"","sep":",","hdrin":true,"hdrout":"none","multi":"one","ret":"\\n","temp":"","skip":"0","strings":true,"include_empty_strings":false,"include_null_values":false,"x":550,"y":200,"wires":[["df8f0c10c749bf37"]]},{"id":"af176573fce1c9ad","type":"delay","z":"8be23377afc3327e","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"2","nbRateUnits":"2","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":900,"y":180,"wires":[["1f3c09a91373ab2d"]]},{"id":"df8f0c10c749bf37","type":"delay","z":"8be23377afc3327e","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"100","nbRateUnits":"10","rateUnits":"minute","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":true,"allowrate":false,"outputs":2,"x":710,"y":200,"wires":[["af176573fce1c9ad"],["d16dc28241588aaf"]]},{"id":"1f3c09a91373ab2d","type":"debug","z":"8be23377afc3327e","name":"debug 191","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1070,"y":180,"wires":[]},{"id":"d16dc28241588aaf","type":"debug","z":"8be23377afc3327e","name":"debug 192","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":910,"y":240,"wires":[]}]

[E1cid]

limited to first 10 lines, just for example

[{"id":"117a41275d1be480","type":"csv","z":"8be23377afc3327e","name":"","sep":",","hdrin":true,"hdrout":"none","multi":"one","ret":"\\n","temp":"","skip":"0","strings":true,"include_empty_strings":false,"include_null_values":false,"x":550,"y":200,"wires":[["23b1ee0e0ad9681e"]]},{"id":"f5d13f178f205ce6","type":"http request","z":"8be23377afc3327e","name":"Test","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://data.nationalgrideso.com/backend/dataset/22bc9bdd-4919-439a-95b8-5213bb3bdf4c/resource/a2dd7992-32d5-4723-b0f5-273caaecbde1/download/service-update-industry-notifications-test150220231030.csv","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"x":430,"y":200,"wires":[["117a41275d1be480"]]},{"id":"23b1ee0e0ad9681e","type":"switch","z":"8be23377afc3327e","name":"","property":"parts.index","propertyType":"msg","rules":[{"t":"lt","v":"10","vt":"num"}],"checkall":"true","repair":false,"outputs":1,"x":690,"y":200,"wires":[["d16dc28241588aaf"]]},{"id":"6e75254b2a6dbc45","type":"inject","z":"8be23377afc3327e","name":"Test","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":310,"y":200,"wires":[["f5d13f178f205ce6"]]},{"id":"d16dc28241588aaf","type":"debug","z":"8be23377afc3327e","name":"debug 192","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":910,"y":240,"wires":[]}]

but if the online service has ssh you should be able to do some thing like scp user@host:/some/file /dev/stdout | head > destfile using the exec node Depending on your set up.

[UnborN]

or you could set your CSV node's Output to create a "single msg array" instead of individual msgs and slice the amount of elements you need from it.

In the example we slice the top 5

[{"id":"6e75254b2a6dbc45","type":"inject","z":"8be23377afc3327e","name":"Test","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":190,"y":240,"wires":[["f5d13f178f205ce6"]]},{"id":"f5d13f178f205ce6","type":"http request","z":"8be23377afc3327e","name":"Test","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://data.nationalgrideso.com/backend/dataset/22bc9bdd-4919-439a-95b8-5213bb3bdf4c/resource/a2dd7992-32d5-4723-b0f5-273caaecbde1/download/service-update-industry-notifications-test150220231030.csv","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"x":310,"y":240,"wires":[["117a41275d1be480"]]},{"id":"117a41275d1be480","type":"csv","z":"8be23377afc3327e","name":"","sep":",","hdrin":true,"hdrout":"none","multi":"mult","ret":"\\n","temp":"","skip":"0","strings":true,"include_empty_strings":false,"include_null_values":false,"x":450,"y":240,"wires":[["42775e6038883b59","90106a345a021b75"]]},{"id":"b8520f1a17929ef0","type":"debug","z":"8be23377afc3327e","name":"debug 193","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":790,"y":240,"wires":[]},{"id":"42775e6038883b59","type":"function","z":"8be23377afc3327e","name":"slice","func":"msg.payload = msg.payload.slice(0,5)  // slice first 5\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":610,"y":240,"wires":[["b8520f1a17929ef0"]]},{"id":"90106a345a021b75","type":"debug","z":"8be23377afc3327e","name":"debug 192","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":590,"y":180,"wires":[]}]

[Tinbum1]

That's brilliant, thank you so much.