To be able to do image processing (on images from IP cams), you need raw image data i.e. the original pixel values. Because you need uncompressed data, i.e. real pixel values. However most of the time our inputs are (compressed) jpeg images. Which means you need the execute the following steps:
Decode the input jpeg images (to get a raw image).
Do some processing on the raw image.
Encode the (processed) raw image back to a jpeg image.
Currently some existing image processing nodes do ALL those 3 steps internally: you inject a jpeg image via an input message, and the output message will again contain a jpeg image. This is a very convenient way of working, but it is very bad for performance when you have multiple image processing nodes after each other in a chain. Because you will end up with repetitive decodings and encodings of the same images:
It would be much faster if we could do something like this:
A first node that decodes the jpeg to raw image, or get somehow the raw images from the camera directly (instead of jpegs)?
2. Pass the raw image through the image processing nodes in the flow
3. At the end (somehow) convert the raw image back to jpeg.
Summarized:
It would be nice to hear ideas about how to accomplish something like that. For example I don't know how I could use ffmpeg for decoding/encoding jpeg images? Or other fast (cross platform) ways to implement such nodes?
Some nodes already offer something similar, e.g. by passing Jimp image instances between the nodes. But of course there are other image processing libraries available, which have their own type of image instances (e.g. OpenCv offers matrix instances..). So not sure what is the best way to mix nodes from different image processing libraries into a single flow. I assume that raw images can be imported/exported quickly in every library, without too much of system resources required (since little or no conversion is required)?? If that would be the case, it could be useful if image processing nodes could support (next to their proprietary image instance objects) raw images anyway (which they internally convert to/from proprietary image instance objects).
So long as you don't branch the flow, it's fairly fast.
However, jump is 100% JavaScript so it's not the fastest image processor.
You could (potentially) use a wasm of opencv (to avoid compilation/c++ bindings) but if it is raw spread you require then something like sharp (or native opencv) is the way to go.
Which nodes do you think my "some nodes" were referring to
Just wondering if there would be an easy way to communicate with your nodes via raw image data, instead of jimp objects. Do you think raw data can be convert to/from Jimp objects without much overhead?
Yes I have been playing with a couple of wasm-based libraries last weekend, but it wasn't a big success yet. For example OpenCv.js (and its wasm build) don't support imDecode and imEncode. So I tried some other wasm libraries, but no luck so far. I have registered a couple of Github issues, but no solution yet.
Yes you are right that native would be much faster. But then you end up with platform limitations... Suppose I e.g. want to run Node-RED in a Docker container, with an Alpine linux without build tools. Then you will need very technical skills to get it running...
The separate encoding/decoding nodes would however offer us lots of flexibility:
You could install encoding/decoding nodes, that use native libraries if you want higher speeds.
You could install encoding/decoding nodes, that use wasm libraries if you are willing to trade a bit of speed, but run it it on all platforms without building headaches.
That would be more flexible compared to embedding the encoding/decoding inside other image processing nodes...
Yes indeed the message cloning is a performance killer. And of course with (uncompressed) raw images that will even become much worse.
Another disadvantage of raw image data is that you need to specify somewhere in the message/container how the raw image data is composed. And there are a lot of different raw image formats.
However it would have been nice if we had some kind of (unwritten) standard/guideline. I really like the fact that there are no constraints about messages, because that allows a lot of flexibility. But it is a pity that you need a lot of conversion nodes in a flow to convert images. Because that makes flows complex, and uses lots of system resources. Moreover in this case all those nodes handle the same data type (i.e. a simple image), but the same image is being send in various different formats...
