Morning guys,
Dave's function node is a very good code snippet to get you started! By concatenating the chunks you can easily search for the magic bytes, even across chunk boundaries. I started my multipart-decoder node developments the same way, but it became very quickly visible that my raspberry became very heavily loaded (see screenshots) by this:
- For every chunk the ENTIRE buffer is scanned (from the start) to find the index of the magic bytes. However that can easily be solved by remembering until which index you have already searched, and pass that to the next 'indexOf' call.
- For every chunk the ENTIRE buffer (that you have collected until now) will be copied, due to the 'concatenate'. This involves a huge amount of bytes being copied over and over again. I solved that by pushing the chunks into an array, and do only a single concatenate when all chunks of a buffer had arrived. Disadvantage is that this makes it harder to find magic bytes across chunks in the array. Had solved that finally by using another npm library which offers a Buffer interface facade for N other buffers underneath. Caution: multiple npm libraries are available, but I found out that not all of them handled the bytes copying efficiently...
That is one of the reasons why I started last month writing a Buffer-node, to help users to accomplish things like this (e.g. for audio and video). But it is not completed yet...
Bart
