Snappy Compression Issues with Kafka Consumer

Use case: I am writing a Kafka consumer for my node-red project.
Libraries used: node-red-contrib-kafkajs, node-red-contrib-kafka-manager
Error: "Snappy compression not implemented" / "Snappy codec is not available"

I have tried most of the node-red Kafka libraries & but all libraries throw error or shutdowns application, saying snappy compression is not supported.
There are no configuration available to support kafka message compression.

Can anyone help me on this? Is anyone able to write a kafka consumer, which can consume messages generated from a non node-red producer?

Do you have snappy codec installed?

From Snappy codec related problem · Issue #460 · dpkp/kafka-python · GitHub ...

Do you have the snappy codec installed? It's not explicitly required by us in, but if you want to use that feature you'll need python-snappy installed

That would need to be raised as an issue or feature request on the node in question. You can find the repository by visiting the NPM page for each of the kafka nodes where you can raise issues.

Yes, I have snappy installed.
I don't require python-snappy, since my application is not python based, hoewever I have snappy installed. Still getting the same error

To resolve this, I had to write my own custom - library in node-red using kafkaJS.
Kafka JS does not support snappy compression by default hence all libraries using kafkaJS throw this error.
We need to use KafkaJS & add Snappy dependency manually & write a custom Consumer node to resolve this, no built-in support exist in any library/depedency as of now.

Great job.

Will you be publishing this node on NPM for the benefit of the community?

Please post a link to it in NPM for future readers.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.