Hi All. I'm a Node-RED noob. Trying to figure out if Node-RED is a feasible option to import multiple csv files, perform some transforms ie. merges, joins, field level changes and mappings across staging tables. Usual ETL stuff. I've read the documentation, tried to find relevant plugins and looked at different posts etc and not yet found an answer. Is anyone here actually using the platform for this sort of thing? I cant even figure out how to say perform a join across two tables to merge together. Any pointers would be appreciated!
What do you mean with ETL ?
What did you try ?
What part couldnt you find ?
There is a CSV node that can read csv files and parse them in to arrays/json objects.
Did you go through the documentation and the youtube playlist to get grasp the basics ?
It kind-of is. But it isn't as good as Python for ETL I'm afraid. Not really the fault of Node-RED itself but more a reflection of the libraries available to Python vs those available for JavaScript/Node.js.
I've long wished it were better but every time I look, I hit the same limitations. Meaning that, for me, I mostly end up falling back to Excel with PowerQuery since my Python foo isn't too strong in the data science area. 
Having said that, it rather depends what you are trying to do. Node-RED is quite good at some of the Extract and Load parts, not quite so good in the transformation. But if you can live with tools such as alasql, you can get a fair bit done with CSV. Otherwise, you probably need your data already in a DB engine wvwn if only SQLite.
Oh, and even PowerShell is better at CSV work than JavaScript!
Thanks - I have tried to find relevant content to answer my question for sure. I just wanted to verify if any more experienced Node Red users have been able to leverage the platform for the purposes of transforming source tables of data and loading into target systems, or am I trying to use it for something that it is not intended to be used for?
thanks so much for the feedback! Seems like perhaps the answer here is that I'd be trying to bend Red to do something that there may be better options available.
It entirely depends on what you are doing.
If you are simply taking CSV, filtering out or changing some data fields and then writing it to a database, node-red is absolutely fine for the job.
Why don't you just give it a go? It doesn't cost anything but time. And there are a lot of guys here who can help when you get stuck.
Yes, sorry I think my last post came across a bit negative.
The other reason for maybe still using Node-RED is as an orchestrator since it is excellent at that. You can use it to easily create flows that call out to Python, etc if you need to use their strengths.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.
