# Elasticsearch Alternative - Zinc search

**URL:** <https://discourse.nodered.org/t/elasticsearch-alternative-zinc-search/61742>\
**Category:** Share Your Projects\
**Created:** [25 April 2022 07:27 UTC](https://discourse.nodered.org/t/elasticsearch-alternative-zinc-search/61742 "2022-04-25T07:27:40Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [25 April 2022 07:27 UTC](https://discourse.nodered.org/t/elasticsearch-alternative-zinc-search/61742/1 "2022-04-25T07:27:40Z")

</div>

For the uninitiated, Elasticsearch is a full text search/analytics engine for bigdata lakes.

It can ingest loads of data, somewhat similar to Influxdb, but where influxdb is geared towards timeseries based data, elastic is geared towards "documents", which can be anything, including time.

Having used Elasticsearch on and off, I find it hard to deal within a personal environment (ingesting data/setting up indexes not straight forward).

I came across [Zinc search](https://github.com/zinclabs/zinc/) (still in alpha stage), which is an alternative and supports the elasticsearch DSL query language. Ingestion is also compatible with elastic, with the [ndjson](http://ndjson.org) format.

It has a rudimentary kibana-like search interface and has an API for ingesting/searching. It is schemaless and can create indexes on the fly.

This is pretty cool stuff to store/search IoT data as you can post your payloads directly to an index of your liking.

Example setup with docker-compose

```auto
services:
  zinc:
    container_name: zinc
    image: public.ecr.aws/h9e2j3o7/zinc:latest
    ports:
      - "4080:4080"
    environment:
      TZ: 'Europe/Amsterdam'
      ZINC_PROMETHEUS_ENABLE: 'true'
      ZINC_TELEMETRY: 'disabled'
      ZINC_FIRST_ADMIN_USER: 'admin'
      ZINC_FIRST_ADMIN_PASSWORD: 'yoursecretpassword'
    volumes:
      - '/<path>/data:/data'
    restart: unless-stopped

```

This exposes port 4080 with a webinterface for searching.  
It uses basic auth for its API.

To post data, setup a http request node with basic authentication, set the method to `PUT` and the url to:

`http://<ip>:4080/api/<index name>/document`

The index name can be anything you want, eg: weather  
it will create the index on the fly if it does not exist.

Inject an msg.payload with an object and see your data magically appear into the index (may need to refresh the page).

To search query, add a new http request node, set the method to `POST`, using basic auth:

```auto
const index = "<the index you used>"
msg.url = `http://<ip>:4080/api/${index}/_search`
msg.payload = {
    "search_type":"alldocuments",
    "max_results":100,
    "sort_fields":["-@timestamp"],
    "query":{"term":"","start_time":null,"end_time":null},
    "fields":["_all"]
    
}
return msg;

```

The output is not (fully?) compatible with grafana, but there are requests to support it.

Personally I find this easier to work with than Influx, as influx needs a specific input format with tags/metrics, whereas with zinc you can simply inject your complete data object, which becomes searchable and aggregatable (like influx).

I don't know how it will perform with huge data loads, but for personal IoT I doubt it becomes a problem.

[Documentation](https://docs.zinclabs.io/#features)

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [25 April 2022 11:45 UTC](https://discourse.nodered.org/t/elasticsearch-alternative-zinc-search/61742/2 "2022-04-25T11:45:07Z")

</div>

Thanks for sharing that, looks interesting and ElasticSearch is a _monster_!

Another thing to add to the ever-growing list of things to look at :sigh: 🙂

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [25 April 2022 11:51 UTC](https://discourse.nodered.org/t/elasticsearch-alternative-zinc-search/61742/3 "2022-04-25T11:51:17Z")

</div>

I would love to see someone integrate the systemd journal logs [as per this ElasticSearch example](https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-input-journald.htmlhttps://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-input-journald.html). That might be REALLY useful.

Another potentially really useful integration would be with Telegraf since that tool (from the same people as InfluxDB) already makes system monitoring really easy. [telegraf/plugins/outputs/elasticsearch at master · influxdata/telegraf (github.com)](https://github.com/influxdata/telegraf/tree/master/plugins/outputs/elasticsearch)

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [25 April 2022 13:11 UTC](https://discourse.nodered.org/t/elasticsearch-alternative-zinc-search/61742/4 "2022-04-25T13:11:01Z")

</div>

Indeed, monitoring comes in different flavors, elastic likes logs, influx likes metrics, especially with prometheus. I see that [zinc is compatible](https://docs.zinclabs.io/ingestion/filebeat/) with filebeat (which is a shipper used for elastic).

I will try to set it up on a linux container.

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [25 April 2022 15:14 UTC](https://discourse.nodered.org/t/elasticsearch-alternative-zinc-search/61742/5 "2022-04-25T15:14:00Z")

</div>

I cannot get Filebeat to work (i see there is an issue reported).

syslog-ng does work

 ![Screenshot 2022-04-25 at 17.12.06](https://us1.discourse-cdn.com/flex026/uploads/nodered/original/3X/6/c/6c252c15d9c3c54a4e3a8cb73ab15793f5fac73f.jpeg)

---

<div class="post-metadata">

**Author:** ![bakman2](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/bakman2/32/6207_2.png) [@bakman2](https://discourse.nodered.org/u/bakman2)\
**Post date:** [25 April 2022 18:22 UTC](https://discourse.nodered.org/t/elasticsearch-alternative-zinc-search/61742/6 "2022-04-25T18:22:22Z")

</div>

Ok got filebeat working now too. Had to switch to a more recent image.

---

<div class="post-metadata">

**Author:** ![TotallyInformation](https://sea2.discourse-cdn.com/flex026/user_avatar/discourse.nodered.org/totallyinformation/32/31_2.png) [@TotallyInformation](https://discourse.nodered.org/u/TotallyInformation)\
**Post date:** [25 April 2022 18:36 UTC](https://discourse.nodered.org/t/elasticsearch-alternative-zinc-search/61742/7 "2022-04-25T18:36:09Z")

</div>

Nice. I can certainly see some uses for it.