First got to share appreciation of node red for how easy it is to jump into and all the reasons why it's so popular. Now I would like to share some thoughts on a closely related use-case I got from a similar low-code system called Pentaho ETL (mostly dead and abandoned these days unfortunately). The…

[image] ThingsTinkerer: And if you accidentally make the wrong choice, like using the fs node which doesn't add parts, then it's damn near impossible. While not a complete answer, perhaps the fs node should be updated to include msg.parts as needed - I think it pre-dates the use of msg.parts. …

Try the exec node to list files and then split. e.g. ls <path_to_directory> -p | grep -v / -p adds / to directories grep -v / returns only lines without / -E .jpg will return all files containg .jpg Thx, this is probably the best solution for that example. The downsides is it removes the sim…

[image] ThingsTinkerer: Then it's a matter of adding a 'last' flag to the last item in the array, then split to individual messages No need to add an array item, as the last message is the one debug after the join. Or when msg.parts.index == msg.parts.count - 1 p.s. @TotallyInformation just…

While not a complete answer, perhaps the fs node should be updated to include msg.parts as needed - I think it pre-dates the use of msg.parts. And, of course, any flow can utilise the same metadata schema and so make use of the features. Like most high-level "languages", Node-RED does sometimes r…

[image] E1cid: No need to add an array item, as the last message is the one debug after the join. Or when msg.parts.index == msg.parts.count - 1 p.s. @TotallyInformation just mentioned that he may add msg.parts to the fs node, You can probably wait for an fs update, which willm solve an issue. …

I understand your position. However, I don't necessarily agree with it all. Node-RED is perfectly able to run a flow as a one-off - as long as you don't mind the flow hanging around afterwards. It is also possible to auto-remove a flow as well as it happens - not easy certainly but definitely possib…

[image] ThingsTinkerer: The batch processing tools available are cumbersome and fragile But isn't the elephant in the room that your incoming data/batch data is actually fragile instead ? Perhaps I am not following your issue completely, batch processing in node-red works quite well. But if …

[image] ThingsTinkerer: Still, even with that, it's kinda fragile in the sense that if a message fails in the delete node, it might stop and instead be passed on to an exception node? Not a big deal as it should be extremely unlikely. Just in general, for a more important task, you need to cover…

[image] ThingsTinkerer: Inject node (sends 1 message once per day) --> Log start --> Get files in folder (generates x amount of messages) --> Delete 1 file per message (x amount of messages) --> Log each file deleted (x messages) --> Log stop (for the last of x messages) That is possible, but…

A new type of flow

General

E1cid 16 September 2024 14:52 28

There is the catch node to catch errors from nodes, and there is an example of a flow that guarantees delivery, which you could adapt to check the file was deleted before continuing. [Announce] Improved Guaranteed Delivery flow

Topic		Replies	Views
ETL Pipelines with Node-RED? Share Your Projects	75	1847	13 January 2024
Use split and join General	6	1668	1 November 2020
Nodes That Erase Messages General	21	667	23 November 2022
Wait for payload to finish flow before passing another payload General	42	1452	29 June 2022
Atomic processing of multiple nodes General	40	4018	13 November 2021

A new type of flow

Related topics