Help with extraction of data from csv file

#1

Hi I am trying to extract data extracted from a power monitor from an email attachment. All of the data is stored as shown in the csv file(better viewed at http://convertcsv.com/csv-viewer-editor.htm). It has many columns and I need to filter for some of them to be monitored. I don't fully know how to manage the Binary Buffer. I've tried somethings with the binary node but don't understand how to get string values in the pattern or how to manage the patterns fully. Or is there an easier way to extract each column? Any help is appreciated.

I can't upload the file so I may need to list the column headings, or atleast an example extract(better suggestions are welcome)...

TIMESTAMP_UTC [s];DATE_UTC;I_L1 [A];I_L2 [A];I_L3 [A];I_N [A];U_L1 [V];U_L2 [V];U_L3 [V];PF_L1 [-];PF_L2 [-];PF_L3 [-];P_L1 [W];P_L2 [W];P_L3 [W];Q_L1 [Var];Q_L2 [Var];Q_L3 [Var];S_L1 [VA];S_L2 [VA];S_L3 [VA];P3P [W];P3PP [Var];P3PA [VA];E_L1 [Wh];E_L2 [Wh];E_L3 [Wh];EP_L1 [Varh];EP_L2 [Varh];EP_L3 [Varh];EA_L1 [VAh];EA_L2 [VAh];EA_L3 [VAh];E3P [Wh];E3PP [Varh];E3PA [VAh];THD_I_L1 [%];THD_I_L2 [%];THD_I_L3 [%];THD_I_N [%];THD_U_L1 [%];THD_U_L2 [%];THD_U_L3 [%];E_ID1 [Wh];E_ID2 [Wh];E_ID3 [Wh];E_ID4 [Wh];_TRMS_ID1 [A];I_TRMS_ID2 [A];I_TRMS_ID3 [A];I_TRMS_ID4 [A]I_AC_ID1 [A];I_AC_ID2 [A];I_AC_ID3 [A];I_AC_ID4 [A]I_DC_ID1 [A];I_DC_ID2 [A];I_DC_ID3 [A];I_DC_ID4 [A]I_DC_ID128 [A];P_ID1 [W];P_ID2 [W];P_ID3 [W];P_ID4 [W]

Example data for first 10 rows:
1550757600;2019-02-21 14:00;0.00;0.00;0.00;0.00;230.63;228.07;229.88;1.00

Node red code:

[{"id":"76f36cd6.1c5014","type":"e-mail in","z":"15f7205.dcb07e","name":"","protocol":"IMAP","server":"imap.gmail.com","useSSL":true,"port":"993","box":"INBOX","disposition":"Read","repeat":"60","x":150,"y":260,"wires":[["c1cb971b.e18ed8","cb4986f9.1703a8"]]},{"id":"c1cb971b.e18ed8","type":"debug","z":"15f7205.dcb07e","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":610,"y":260,"wires":[]},{"id":"cb4986f9.1703a8","type":"function","z":"15f7205.dcb07e","name":"","func":"msg.payload=msg.attachments[0].content;\nmsg.payload=toString(msg.payload);\nreturn msg","outputs":1,"noerr":0,"x":290,"y":320,"wires":[["5be10971.2a43e8"]]},{"id":"5be10971.2a43e8","type":"binary","z":"15f7205.dcb07e","name":"","pattern":"b8=>TIMESTAMP_UTC ,b8=>DATE_UTC,b32f=>I_L1","x":450,"y":300,"wires":[["c1cb971b.e18ed8"]]}]

Like I said is there a way to extract the columns easier, or do I need to try something other than nodeRed to extract this Csv files data.

Much appreciated in advance,

Dillon

0 Likes

#2

Have you tried csv node?

0 Likes

#3

I tried to add it, but the data from the file comes out as binary like shown, which I don't know how to easily handle.

image

0 Likes

#4

Unfortunately your flow isn't currently importable.Please read the following post How to share code or flow json and then edit the above message.

So either your attachment is compressed ( you could try adding a node to unzip it) or the attachment isn't a csv file...

0 Likes

#5

I tried to redit the code and have succesfully imported what I have currently in my version. I will test the zip node, the attachment is definitely a csv file an image shown below
image

0 Likes

#6

Thanks for editing your flow. Now it is easier to se that you have a binary node in it.

Taken from the README of the binary node that is in your flow

If the input is a buffer it tries to parse it and creates a javascript object. If the input is a javascript object it tries to serialize it and creates a buffer.

If your Email has an attachment that is a csv file, why are you sending it through a binary node?

Your flow should be something like

Email-IN --- Function (or even better Change node) --- CSV node --- debug

0 Likes

#7

I changed the binary to csv node but still have problems with the buffer even though I tried to use the Node-contrib-zip to try unzipping. I got an error from the zip node.

New node red code:

[{"id":"ffa64d8.1ea97b","type":"e-mail in","z":"da76c366.cea06","name":"","protocol":"IMAP","server":"imap.gmail.com","useSSL":true,"port":"993","box":"INBOX","disposition":"Read","repeat":"60","x":170,"y":420,"wires":[["98c45a02.7a17a8","24ceea5.48e5616"]]},{"id":"e07fcdc.4aac63","type":"debug","z":"da76c366.cea06","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":630,"y":420,"wires":[]},{"id":"24ceea5.48e5616","type":"function","z":"da76c366.cea06","name":"","func":"msg.payload=msg.attachments[0].content;\nreturn msg","outputs":1,"noerr":0,"x":190,"y":520,"wires":[["966d523a.ba83a","b8c140c9.90352"]]},{"id":"966d523a.ba83a","type":"csv","z":"da76c366.cea06","name":"","sep":";","hdrin":"","hdrout":"","multi":"one","ret":"\\n","temp":"TIMESTAMP_UTC [s];DATE_UTC;I_L1 [A];I_L2 [A];I_L3 [A];I_N [A];U_L1 [V];U_L2 [V];U_L3 [V];PF_L1 [-];PF_L2 [-];PF_L3 [-];P_L1 [W];P_L2 [W];P_L3 [W];Q_L1 [Var];Q_L2 [Var];Q_L3 [Var];S_L1 [VA];S_L2 [VA];S_L3 [VA];P3P [W];P3PP [Var];P3PA [VA];E_L1 [Wh];E_L2 [Wh];E_L3 [Wh];EP_L1 [Varh];EP_L2 [Varh];EP_L3 [Varh];EA_L1 [VAh];EA_L2 [VAh];EA_L3 [VAh];E3P [Wh];E3PP [Varh];E3PA [VAh];THD_I_L1 [%];THD_I_L2 [%];THD_I_L3 [%];THD_I_N [%];THD_U_L1 [%];THD_U_L2 [%];THD_U_L3 [%];E_ID1 [Wh];E_ID2 [Wh];E_ID3 [Wh];E_ID4 [Wh];_TRMS_ID1 [A];I_TRMS_ID2 [A];I_TRMS_ID3 [A];I_TRMS_ID4 [A]I_AC_ID1 [A];I_AC_ID2 [A];I_AC_ID3 [A];I_AC_ID4 [A]I_DC_ID1 [A];I_DC_ID2 [A];I_DC_ID3 [A];I_DC_ID4 [A]I_DC_ID128 [A];P_ID1 [W];P_ID2 [W];P_ID3 [W];P_ID4 [W]","skip":"0","x":390,"y":620,"wires":[["e07fcdc.4aac63"]]},{"id":"b8c140c9.90352","type":"zip","z":"da76c366.cea06","name":"","mode":"decompress","filename":"","outasstring":false,"x":210,"y":620,"wires":[["e07fcdc.4aac63"]]}]

image

Picture of output. I've even highlighted to show it's a csv file

0 Likes

#8

if it helps the file is here. I needed to convert to txt file for here but it can be rechanged to csv
TDC-C2-ABB_measurements_pc_2019-02-21_15-16-01_STACK1H.txt (3.3 MB)

0 Likes

#9

e-mail node info says about attachment :

content: // The actual content of the data contained in a Node.js Buffer object
// We can turn this into a base64 data string with content.toString('base64')

Try this in your function node
msg.payload=msg.attachments[0].content.toString('base64')

0 Likes

#10

ok, it somewhat helped. I used "latin1" instead of base64 which was able to convert the data to string characters for me. Just now the seperation? I used a csv node but when I try to seperate it using the semicolon(; )/comma(,) it doesn't work. Instead the server will lose connection. So how could I better do the relevant data extraction. Is another program more relevant??

Current code:

[{"id":"aec7375b.582a88","type":"e-mail in","z":"a3693599.be25b8","name":"","protocol":"IMAP","server":"imap.gmail.com","useSSL":true,"port":"993","box":"INBOX","disposition":"Read","repeat":"60","x":170,"y":220,"wires":[["74ef9fed.0b6aa","44ff2d6d.f70a74"]]},{"id":"74ef9fed.0b6aa","type":"function","z":"a3693599.be25b8","name":"","func":"msg.payload=msg.attachments[0].content.toString('latin1');\nreturn msg;","outputs":1,"noerr":0,"x":290,"y":280,"wires":[["44ff2d6d.f70a74","e97fb838.ecde98"]]},{"id":"44ff2d6d.f70a74","type":"debug","z":"a3693599.be25b8","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":550,"y":220,"wires":[]},{"id":"e97fb838.ecde98","type":"csv","z":"a3693599.be25b8","name":"","sep":";","hdrin":"","hdrout":"","multi":"one","ret":"\\n","temp":"TIMESTAMP_UTC [s],DATE_UTC,I_L1 [A],I_L2 [A],I_L3 [A],I_N [A],U_L1 [V],U_L2 [V],U_L3 [V],PF_L1 [-],PF_L2 [-],PF_L3 [-],P_L1 [W],P_L2 [W],P_L3 [W],Q_L1 [Var],Q_L2 [Var],Q_L3 [Var],S_L1 [VA],S_L2 [VA],S_L3 [VA],P3P [W],P3PP [Var],P3PA [VA],E_L1 [Wh],E_L2 [Wh],E_L3 [Wh],EP_L1 [Varh],EP_L2 [Varh],EP_L3 [Varh],EA_L1 [VAh],EA_L2 [VAh],EA_L3 [VAh],E3P [Wh],E3PP [Varh],E3PA [VAh],THD_I_L1 [%],THD_I_L2 [%],THD_I_L3 [%],THD_I_N [%],THD_U_L1 [%],THD_U_L2 [%],THD_U_L3 [%],E_ID1 [Wh],E_ID2 [Wh],E_ID3 [Wh],E_ID4 [Wh],E_ID5 [Wh],E_ID6 [Wh],E_ID7 [Wh],E_ID8 [Wh],E_ID9 [Wh],E_ID10 [Wh],E_ID11 [Wh],E_ID12 [Wh],E_ID13 [Wh],E_ID14 [Wh],E_ID15 [Wh],E_ID16 [Wh],E_ID17 [Wh],E_ID18 [Wh],E_ID19 [Wh],E_ID20 [Wh],E_ID21 [Wh],E_ID22 [Wh],E_ID23 [Wh],E_ID24 [Wh],E_ID25 [Wh],E_ID26 [Wh],E_ID27 [Wh],E_ID28 [Wh],E_ID29 [Wh],E_ID30 [Wh],E_ID31 [Wh],E_ID32 [Wh],E_ID33 [Wh],E_ID34 [Wh],E_ID35 [Wh],E_ID36 [Wh],E_ID37 [Wh],E_ID38 [Wh],E_ID39 [Wh],E_ID40 [Wh],E_ID41 [Wh],E_ID42 [Wh],E_ID43 [Wh],E_ID44 [Wh],E_ID45 [Wh],E_ID46 [Wh],E_ID47 [Wh],E_ID48 [Wh],E_ID49 [Wh],E_ID50 [Wh],E_ID51 [Wh],E_ID52 [Wh],E_ID53 [Wh],E_ID54 [Wh],E_ID55 [Wh],E_ID56 [Wh],E_ID57 [Wh],E_ID58 [Wh],E_ID59 [Wh],E_ID60 [Wh],E_ID61 [Wh],E_ID62 [Wh],E_ID63 [Wh],E_ID64 [Wh],E_ID65 [Wh],E_ID66 [Wh],E_ID67 [Wh],E_ID68 [Wh],E_ID69 [Wh],E_ID70 [Wh],E_ID71 [Wh],E_ID72 [Wh],E_ID73 [Wh],E_ID74 [Wh],E_ID75 [Wh],E_ID76 [Wh],E_ID77 [Wh],E_ID78 [Wh],E_ID79 [Wh],E_ID80 [Wh],E_ID81 [Wh],E_ID82 [Wh],E_ID83 [Wh],E_ID84 [Wh],E_ID85 [Wh],E_ID86 [Wh],E_ID87 [Wh],E_ID88 [Wh],E_ID89 [Wh],E_ID90 [Wh],E_ID91 [Wh],E_ID92 [Wh],E_ID93 [Wh],E_ID94 [Wh],E_ID95 [Wh],E_ID96 [Wh],E_ID97 [Wh],E_ID98 [Wh],E_ID99 [Wh],E_ID100 [Wh],E_ID101 [Wh],E_ID102 [Wh],E_ID103 [Wh],E_ID104 [Wh],E_ID105 [Wh],E_ID106 [Wh],E_ID107 [Wh],E_ID108 [Wh],E_ID109 [Wh],E_ID110 [Wh],E_ID111 [Wh],E_ID112 [Wh],E_ID113 [Wh],E_ID114 [Wh],E_ID115 [Wh],E_ID116 [Wh],E_ID117 [Wh],E_ID118 [Wh],E_ID119 [Wh],E_ID120 [Wh],E_ID121 [Wh],E_ID122 [Wh],E_ID123 [Wh],E_ID124 [Wh],E_ID125 [Wh],E_ID126 [Wh],E_ID127 [Wh],E_ID128 [Wh],I_TRMS_ID1 [A],I_TRMS_ID2 [A],I_TRMS_ID3 [A],I_TRMS_ID4 [A],I_TRMS_ID5 [A],I_TRMS_ID6 [A],I_TRMS_ID7 [A],I_TRMS_ID8 [A],I_TRMS_ID9 [A],I_TRMS_ID10 [A],I_TRMS_ID11 [A],I_TRMS_ID12 [A],I_TRMS_ID13 [A],I_TRMS_ID14 [A],I_TRMS_ID15 [A],I_TRMS_ID16 [A],I_TRMS_ID17 [A],I_TRMS_ID18 [A],I_TRMS_ID19 [A],I_TRMS_ID20 [A],I_TRMS_ID21 [A],I_TRMS_ID22 [A],I_TRMS_ID23 [A],I_TRMS_ID24 [A],I_TRMS_ID25 [A],I_TRMS_ID26 [A],I_TRMS_ID27 [A],I_TRMS_ID28 [A],I_TRMS_ID29 [A],I_TRMS_ID30 [A],I_TRMS_ID31 [A],I_TRMS_ID32 [A],I_TRMS_ID33 [A],I_TRMS_ID34 [A],I_TRMS_ID35 [A],I_TRMS_ID36 [A],I_TRMS_ID37 [A],I_TRMS_ID38 [A],I_TRMS_ID39 [A],I_TRMS_ID40 [A],I_TRMS_ID41 [A],I_TRMS_ID42 [A],I_TRMS_ID43 [A],I_TRMS_ID44 [A],I_TRMS_ID45 [A],I_TRMS_ID46 [A],I_TRMS_ID47 [A],I_TRMS_ID48 [A],I_TRMS_ID49 [A],I_TRMS_ID50 [A],I_TRMS_ID51 [A],I_TRMS_ID52 [A],I_TRMS_ID53 [A],I_TRMS_ID54 [A],I_TRMS_ID55 [A],I_TRMS_ID56 [A],I_TRMS_ID57 [A],I_TRMS_ID58 [A],I_TRMS_ID59 [A],I_TRMS_ID60 [A],I_TRMS_ID61 [A],I_TRMS_ID62 [A],I_TRMS_ID63 [A],I_TRMS_ID64 [A],I_TRMS_ID65 [A],I_TRMS_ID66 [A],I_TRMS_ID67 [A],I_TRMS_ID68 [A],I_TRMS_ID69 [A],I_TRMS_ID70 [A],I_TRMS_ID71 [A],I_TRMS_ID72 [A],I_TRMS_ID73 [A],I_TRMS_ID74 [A],I_TRMS_ID75 [A],I_TRMS_ID76 [A],I_TRMS_ID77 [A],I_TRMS_ID78 [A],I_TRMS_ID79 [A],I_TRMS_ID80 [A],I_TRMS_ID81 [A],I_TRMS_ID82 [A],I_TRMS_ID83 [A],I_TRMS_ID84 [A],I_TRMS_ID85 [A],I_TRMS_ID86 [A],I_TRMS_ID87 [A],I_TRMS_ID88 [A],I_TRMS_ID89 [A],I_TRMS_ID90 [A],I_TRMS_ID91 [A],I_TRMS_ID92 [A],I_TRMS_ID93 [A],I_TRMS_ID94 [A],I_TRMS_ID95 [A],I_TRMS_ID96 [A],I_TRMS_ID97 [A],I_TRMS_ID98 [A],I_TRMS_ID99 [A],I_TRMS_ID100 [A],I_TRMS_ID101 [A],I_TRMS_ID102 [A],I_TRMS_ID103 [A],I_TRMS_ID104 [A],I_TRMS_ID105 [A],I_TRMS_ID106 [A],I_TRMS_ID107 [A],I_TRMS_ID108 [A],I_TRMS_ID109 [A],I_TRMS_ID110 [A],I_TRMS_ID111 [A],I_TRMS_ID112 [A],I_TRMS_ID113 [A],I_TRMS_ID114 [A],I_TRMS_ID115 [A],I_TRMS_ID116 [A],I_TRMS_ID117 [A],I_TRMS_ID118 [A],I_TRMS_ID119 [A],I_TRMS_ID120 [A],I_TRMS_ID121 [A],I_TRMS_ID122 [A],I_TRMS_ID123 [A],I_TRMS_ID124 [A],I_TRMS_ID125 [A],I_TRMS_ID126 [A],I_TRMS_ID127 [A],I_TRMS_ID128 [A],I_AC_ID1 [A],I_AC_ID2 [A],I_AC_ID3 [A],I_AC_ID4 [A],I_AC_ID5 [A],I_AC_ID6 [A],I_AC_ID7 [A],I_AC_ID8 [A],I_AC_ID9 [A],I_AC_ID10 [A],I_AC_ID11 [A],I_AC_ID12 [A],I_AC_ID13 [A],I_AC_ID14 [A],I_AC_ID15 [A],I_AC_ID16 [A],I_AC_ID17 [A],I_AC_ID18 [A],I_AC_ID19 [A],I_AC_ID20 [A],I_AC_ID21 [A],I_AC_ID22 [A],I_AC_ID23 [A],I_AC_ID24 [A],I_AC_ID25 [A],I_AC_ID26 [A],I_AC_ID27 [A],I_AC_ID28 [A],I_AC_ID29 [A],I_AC_ID30 [A],I_AC_ID31 [A],I_AC_ID32 [A],I_AC_ID33 [A],I_AC_ID34 [A],I_AC_ID35 [A],I_AC_ID36 [A],I_AC_ID37 [A],I_AC_ID38 [A],I_AC_ID39 [A],I_AC_ID40 [A],I_AC_ID41 [A],I_AC_ID42 [A],I_AC_ID43 [A],I_AC_ID44 [A],I_AC_ID45 [A],I_AC_ID46 [A],I_AC_ID47 [A],I_AC_ID48 [A],I_AC_ID49 [A],I_AC_ID50 [A],I_AC_ID51 [A],I_AC_ID52 [A],I_AC_ID53 [A],I_AC_ID54 [A],I_AC_ID55 [A],I_AC_ID56 [A],I_AC_ID57 [A],I_AC_ID58 [A],I_AC_ID59 [A],I_AC_ID60 [A],I_AC_ID61 [A],I_AC_ID62 [A],I_AC_ID63 [A],I_AC_ID64 [A],I_AC_ID65 [A],I_AC_ID66 [A],I_AC_ID67 [A],I_AC_ID68 [A],I_AC_ID69 [A],I_AC_ID70 [A],I_AC_ID71 [A],I_AC_ID72 [A],I_AC_ID73 [A],I_AC_ID74 [A],I_AC_ID75 [A],I_AC_ID76 [A],I_AC_ID77 [A],I_AC_ID78 [A],I_AC_ID79 [A],I_AC_ID80 [A],I_AC_ID81 [A],I_AC_ID82 [A],I_AC_ID83 [A],I_AC_ID84 [A],I_AC_ID85 [A],I_AC_ID86 [A],I_AC_ID87 [A],I_AC_ID88 [A],I_AC_ID89 [A],I_AC_ID90 [A],I_AC_ID91 [A],I_AC_ID92 [A],I_AC_ID93 [A],I_AC_ID94 [A],I_AC_ID95 [A],I_AC_ID96 [A],I_AC_ID97 [A],I_AC_ID98 [A],I_AC_ID99 [A],I_AC_ID100 [A],I_AC_ID101 [A],I_AC_ID102 [A],I_AC_ID103 [A],I_AC_ID104 [A],I_AC_ID105 [A],I_AC_ID106 [A],I_AC_ID107 [A],I_AC_ID108 [A],I_AC_ID109 [A],I_AC_ID110 [A],I_AC_ID111 [A],I_AC_ID112 [A],I_AC_ID113 [A],I_AC_ID114 [A],I_AC_ID115 [A],I_AC_ID116 [A],I_AC_ID117 [A],I_AC_ID118 [A],I_AC_ID119 [A],I_AC_ID120 [A],I_AC_ID121 [A],I_AC_ID122 [A],I_AC_ID123 [A],I_AC_ID124 [A],I_AC_ID125 [A],I_AC_ID126 [A],I_AC_ID127 [A],I_AC_ID128 [A],I_DC_ID1 [A],I_DC_ID2 [A],I_DC_ID3 [A],I_DC_ID4 [A],I_DC_ID5 [A],I_DC_ID6 [A],I_DC_ID7 [A],I_DC_ID8 [A],I_DC_ID9 [A],I_DC_ID10 [A],I_DC_ID11 [A],I_DC_ID12 [A],I_DC_ID13 [A],I_DC_ID14 [A],I_DC_ID15 [A],I_DC_ID16 [A],I_DC_ID17 [A],I_DC_ID18 [A],I_DC_ID19 [A],I_DC_ID20 [A],I_DC_ID21 [A],I_DC_ID22 [A],I_DC_ID23 [A],I_DC_ID24 [A],I_DC_ID25 [A],I_DC_ID26 [A],I_DC_ID27 [A],I_DC_ID28 [A],I_DC_ID29 [A],I_DC_ID30 [A],I_DC_ID31 [A],I_DC_ID32 [A],I_DC_ID33 [A],I_DC_ID34 [A],I_DC_ID35 [A],I_DC_ID36 [A],I_DC_ID37 [A],I_DC_ID38 [A],I_DC_ID39 [A],I_DC_ID40 [A],I_DC_ID41 [A],I_DC_ID42 [A],I_DC_ID43 [A],I_DC_ID44 [A],I_DC_ID45 [A],I_DC_ID46 [A],I_DC_ID47 [A],I_DC_ID48 [A],I_DC_ID49 [A],I_DC_ID50 [A],I_DC_ID51 [A],I_DC_ID52 [A],I_DC_ID53 [A],I_DC_ID54 [A],I_DC_ID55 [A],I_DC_ID56 [A],I_DC_ID57 [A],I_DC_ID58 [A],I_DC_ID59 [A],I_DC_ID60 [A],I_DC_ID61 [A],I_DC_ID62 [A],I_DC_ID63 [A],I_DC_ID64 [A],I_DC_ID65 [A],I_DC_ID66 [A],I_DC_ID67 [A],I_DC_ID68 [A],I_DC_ID69 [A],I_DC_ID70 [A],I_DC_ID71 [A],I_DC_ID72 [A],I_DC_ID73 [A],I_DC_ID74 [A],I_DC_ID75 [A],I_DC_ID76 [A],I_DC_ID77 [A],I_DC_ID78 [A],I_DC_ID79 [A],I_DC_ID80 [A],I_DC_ID81 [A],I_DC_ID82 [A],I_DC_ID83 [A],I_DC_ID84 [A],I_DC_ID85 [A],I_DC_ID86 [A],I_DC_ID87 [A],I_DC_ID88 [A],I_DC_ID89 [A],I_DC_ID90 [A],I_DC_ID91 [A],I_DC_ID92 [A],I_DC_ID93 [A],I_DC_ID94 [A],I_DC_ID95 [A],I_DC_ID96 [A],I_DC_ID97 [A],I_DC_ID98 [A],I_DC_ID99 [A],I_DC_ID100 [A],I_DC_ID101 [A],I_DC_ID102 [A],I_DC_ID103 [A],I_DC_ID104 [A],I_DC_ID105 [A],I_DC_ID106 [A],I_DC_ID107 [A],I_DC_ID108 [A],I_DC_ID109 [A],I_DC_ID110 [A],I_DC_ID111 [A],I_DC_ID112 [A],I_DC_ID113 [A],I_DC_ID114 [A],I_DC_ID115 [A],I_DC_ID116 [A],I_DC_ID117 [A],I_DC_ID118 [A],I_DC_ID119 [A],I_DC_ID120 [A],I_DC_ID121 [A],I_DC_ID122 [A],I_DC_ID123 [A],I_DC_ID124 [A],I_DC_ID125 [A],I_DC_ID126 [A],I_DC_ID127 [A],I_DC_ID128 [A],P_ID1 [W],P_ID2 [W],P_ID3 [W],P_ID4 [W],P_ID5 [W],P_ID6 [W],P_ID7 [W],P_ID8 [W],P_ID9 [W],P_ID10 [W],P_ID11 [W],P_ID12 [W],P_ID13 [W],P_ID14 [W],P_ID15 [W],P_ID16 [W],P_ID17 [W],P_ID18 [W],P_ID19 [W],P_ID20 [W],P_ID21 [W],P_ID22 [W],P_ID23 [W],P_ID24 [W],P_ID25 [W],P_ID26 [W],P_ID27 [W],P_ID28 [W],P_ID29 [W],P_ID30 [W],P_ID31 [W],P_ID32 [W],P_ID33 [W],P_ID34 [W],P_ID35 [W],P_ID36 [W],P_ID37 [W],P_ID38 [W],P_ID39 [W],P_ID40 [W],P_ID41 [W],P_ID42 [W],P_ID43 [W],P_ID44 [W],P_ID45 [W],P_ID46 [W],P_ID47 [W],P_ID48 [W],P_ID49 [W],P_ID50 [W],P_ID51 [W],P_ID52 [W],P_ID53 [W],P_ID54 [W],P_ID55 [W],P_ID56 [W],P_ID57 [W],P_ID58 [W],P_ID59 [W],P_ID60 [W],P_ID61 [W],P_ID62 [W],P_ID63 [W],P_ID64 [W],P_ID65 [W],P_ID66 [W],P_ID67 [W],P_ID68 [W],P_ID69 [W],P_ID70 [W],P_ID71 [W],P_ID72 [W],P_ID73 [W],P_ID74 [W],P_ID75 [W],P_ID76 [W],P_ID77 [W],P_ID78 [W],P_ID79 [W],P_ID80 [W],P_ID81 [W],P_ID82 [W],P_ID83 [W],P_ID84 [W],P_ID85 [W],P_ID86 [W],P_ID87 [W],P_ID88 [W],P_ID89 [W],P_ID90 [W],P_ID91 [W],P_ID92 [W],P_ID93 [W],P_ID94 [W],P_ID95 [W],P_ID96 [W],P_ID97 [W],P_ID98 [W],P_ID99 [W],P_ID100 [W],P_ID101 [W],P_ID102 [W],P_ID103 [W],P_ID104 [W],P_ID105 [W],P_ID106 [W],P_ID107 [W],P_ID108 [W],P_ID109 [W],P_ID110 [W],P_ID111 [W],P_ID112 [W],P_ID113 [W],P_ID114 [W],P_ID115 [W],P_ID116 [W],P_ID117 [W],P_ID118 [W],P_ID119 [W],P_ID120 [W],P_ID121 [W],P_ID122 [W],P_ID123 [W],P_ID124 [W],P_ID125 [W],P_ID126 [W],P_ID127 [W],P_ID128 [W],","skip":"0","x":370,"y":380,"wires":[["44ff2d6d.f70a74"]]}]
0 Likes

#11

If I download the file and use that directly - it parses fairly easily.
What happens if you feed your function into my csv ?

[{"id":"c7737c73.d0d6a","type":"inject","z":"82738787.0e0338","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":140,"y":1560,"wires":[["4091b149.dffc4"]]},{"id":"4091b149.dffc4","type":"file in","z":"82738787.0e0338","name":"","filename":"/tmp/tdc.txt","format":"utf8","chunk":false,"sendError":false,"x":290,"y":1560,"wires":[["9d0a80e1.fe061"]]},{"id":"9d0a80e1.fe061","type":"csv","z":"82738787.0e0338","name":"","sep":";","hdrin":true,"hdrout":"","multi":"one","ret":"\\n","temp":"","skip":"0","x":430,"y":1560,"wires":[["9e209f4b.f0e0d"]]},{"id":"9e209f4b.f0e0d","type":"debug","z":"82738787.0e0338","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":570,"y":1560,"wires":[]}]
0 Likes

#12

I tried it and with the csv file nodes it crashes the server once the email node fetches the file, not sure why?

also I'm curious to know how you have the "file In" node, which is not available for me?

0 Likes

#13

File in

0 Likes

#14

But I don't have this Storage and I searched for the file nodes but for me I can't find it??

0 Likes

#15

This is strange. Probably now Node-RED core guys want to know what is the OP system you are running and version of Node-RED and nodejs.

0 Likes

#16

Well I use the IBM Node-Red Starter kit, version 0.19.5. NodeJs (UnKnown)? I can't find where to find this info since it's held on their cloud system.

0 Likes

#17

It's definitely something out of regular user knowledge base and needs some input of leaders.

0 Likes

#18

So I wait??

0 Likes

#19

The file nodes are removed from the cloud version as there is no permanent file system. (when the app gets restarted it may or may not start on the same instance - so there the files may no longer be there on the instance you are running on).

I thought your email plus function was working ?

0 Likes

#20

Yes, it outputs all of the data, but how can I order it?? It comes out as a very big string, if I use csv with this my service disconnects.
I want to process certain columns eg. E_ID3 [Wh]
Output with email+function extract:

image

0 Likes