Parse CSV and Skip lines starting with #

[Meistro]

Hello all,
I want to use this data without editing the source file. The quantity of lines at the start of the file with # varies so I can't easily just skip the first so many lines.

# RaceRender Data: TrackAddict 4.4.2 on Android 8.0.0 [samsung SM-G930V / Qualcomm Technologies	Inc MSM8996] (Mode: 0)					
# Vehicle App ID: 50E0D39E63639A52						
# End Point: 42.408026	-86.140483  @ -1.00 deg					
# Split Point 1: 42.408731	-86.139226  @ -1.00 deg					
# Split Point 2: 42.407319	-86.135581  @ -1.00 deg					
# GPS: Android; Mode: Android						
# OBD Mode: BT (OBDII); ID: "ELM327 v1.5"						
# OBD Settings: AP1;AF1;RPR0						
# User Settings: SL1;U0;AS1;LT0/1;EC0;VC0;VQ3;VS0;VSOIS0;VIF0						
# Device Free Space: 1692 MB						
Time	UTC Time	Lap	Sector	Predicted Lap Time	Predicted vs Best Lap	GPS_Update
0	1593265304	0	0	0	0	1
0.033	1593265304	0	0	0	0	0
0.057	1593265304	0	0	0	0	1
0.069	1593265304	0	0	0	0	1

Plus, some # lines are scattered throughout the file...

825.592	1593266130	3	3	209.94
825.638	1593266130	3	3	209.94
# Sector 3: 00:01:24.397				
# Lap 3: 00:03:30.761				
825.646	1593266130	4	1	0

I don't mind using these Sector and Lap infos but I have no clue how to deal with them.

Anyone have a better idea how I can tackle this?

Thanks
Mark

[UnborN]

Hi Mark,

Maybe you can read the file line by line and use a function node after the read file node
that filters any line that starts with a #. (Assuming the hashtag is always at the very start of the line).

if (msg.payload[0] !== '#' ) {
return msg;
}

We dont know how you want to process the data afterwards.

[Meistro]

Hey thanks this is working
My end goal is to get this data into Mysql for querying. The issue I was having is importing those logs into mysql caused errors because of those additional lines. To create the database I stripped the file down to only a few lines and imported through PhpMyadmin

My idea is to drop the file to node-red server with FTP and Watch the folder. This will trigger the process to clean out the # lines from the file and INSERT INTO the database. Then after a delay, delete the file.

I have another long time running app that query mysql to show what I want. I was using Torque Pro app before and now I'm using TrackAddict to generate these logs. The logs are personalized to the racetrack I'm at now. They are configured different however which leads me here for help.

How can I add to your solution so that blanks are also not included? Will this matter if I then run it through the CSV function?

[Meistro]

Scratch that request for stripping the blanks. This is working very well with CSV node.

I was using a change node before to rename some of these. I don't know if there is a difference in speed doing it with change node or adding the headers I want in the csv node. I guess I can try both ways and time it using the same file.

Thanks again!!!

[UnborN]

I saw it also in my test that the csv was handling the empty lines but you can use

if (msg.payload && msg.payload[0] !== '#') {
return msg;
}

which also checks msg.payload is true .. that it has a value