Just an update on this, I've got the following going:
What this shows is the web2disk node streaming 145MB from web to disk. The disk2stream node then unzips that file, sees a 1.8GB JsonL file and begins to stream that line-by-line generating 2M messages, that end at the debug node.
What the point here is that memory is complete flat the entire time, it's all being streamed. The other thing is that the whole thing took 90s to do.
That's a promising start ... 
Copyright OpenJS Foundation and Node-RED contributors. All rights reserved. The OpenJS Foundation has registered trademarks and uses trademarks. For a list of trademarks of the OpenJS Foundation, please see our Trademark Policy and Trademark List. Trademarks and logos not indicated on the list of OpenJS Foundation trademarks are trademarksā¢ or registeredĀ® trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.
The OpenJS Foundation | Terms of Use | Privacy Policy | OpenJS Foundation Bylaws | Trademark Policy | Trademark List | Cookie Policy