# Snappy Compression Issues with Kafka Consumer

**URL:** <https://discourse.nodered.org/t/snappy-compression-issues-with-kafka-consumer/70078>\
**Category:** General\
**Created:** [4 November 2022 09:21 UTC](https://discourse.nodered.org/t/snappy-compression-issues-with-kafka-consumer/70078 "2022-11-04T09:21:39Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![elfinbird](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/elfinbird/32/70199_2.png) [@elfinbird](https://discourse.nodered.org/u/elfinbird)\
**Post date:** [4 November 2022 09:21 UTC](https://discourse.nodered.org/t/snappy-compression-issues-with-kafka-consumer/70078/1 "2022-11-04T09:21:39Z")

</div>

_Use case:_ I am writing a Kafka consumer for my node-red project.  
_Libraries used_: node-red-contrib-kafkajs, node-red-contrib-kafka-manager  
_Error_: "Snappy compression not implemented" / "Snappy codec is not available"

**Question**  
I have tried most of the node-red Kafka libraries & but all libraries throw error or shutdowns application, saying `snappy compression is not supported`.  
There are no configuration available to support kafka message compression.

Can anyone help me on this? Is anyone able to write a kafka consumer, which can consume messages generated from a non node-red producer?

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [4 November 2022 10:27 UTC](https://discourse.nodered.org/t/snappy-compression-issues-with-kafka-consumer/70078/2 "2022-11-04T10:27:33Z")

</div>

> [@elfinbird](#):
>
> "Snappy codec is not available"

Do you have snappy codec installed?

From [Snappy codec related problem · Issue #460 · dpkp/kafka-python · GitHub](https://github.com/dpkp/kafka-python/issues/460) ...

> Do you have the snappy codec installed? It's not explicitly required by us in setup.py, but if you want to use that feature you'll need `python-snappy` installed

> [@elfinbird](#):
>
> There are no configuration available to support kafka message compression

That would need to be raised as an issue or feature request on the node in question. You can find the repository by visiting the NPM page for each of the kafka nodes where you can raise issues.

---

<div class="post-metadata">

**Author:** ![elfinbird](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/elfinbird/32/70199_2.png) [@elfinbird](https://discourse.nodered.org/u/elfinbird)\
**Post date:** [7 November 2022 09:56 UTC](https://discourse.nodered.org/t/snappy-compression-issues-with-kafka-consumer/70078/3 "2022-11-07T09:56:37Z")

</div>

Yes, I have snappy installed.  
I don't require `python-snappy`, since my application is not python based, hoewever I have `snappy` installed. Still getting the same error

---

<div class="post-metadata">

**Author:** ![elfinbird](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/elfinbird/32/70199_2.png) [@elfinbird](https://discourse.nodered.org/u/elfinbird)\
**Post date:** [14 November 2022 08:09 UTC](https://discourse.nodered.org/t/snappy-compression-issues-with-kafka-consumer/70078/4 "2022-11-14T08:09:45Z")

</div>

To resolve this, I had to write my own custom - library in node-red using kafkaJS.  
Kafka JS does not support snappy compression by default hence all libraries using kafkaJS throw this error.  
We need to use KafkaJS & add Snappy dependency manually & write a custom Consumer node to resolve this, no built-in support exist in any library/depedency as of now.

---

<div class="post-metadata">

**Author:** ![Steve-Mcl](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/steve-mcl/32/4826_2.png) [@Steve-Mcl](https://discourse.nodered.org/u/Steve-Mcl)\
**Post date:** [14 November 2022 09:04 UTC](https://discourse.nodered.org/t/snappy-compression-issues-with-kafka-consumer/70078/5 "2022-11-14T09:04:51Z")

</div>

Great job.

Will you be publishing this node on NPM for the benefit of the community?

Please post a link to it in NPM for future readers.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/1X/d073cd938eafa2e558d7c2cd59003b3ef4963033.png) [@system](https://discourse.nodered.org/u/system)\
**Post date:** [28 November 2022 09:05 UTC](https://discourse.nodered.org/t/snappy-compression-issues-with-kafka-consumer/70078/6 "2022-11-28T09:05:50Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.