For the uninitiated, Elasticsearch is a full text search/analytics engine for bigdata lakes.
It can ingest loads of data, somewhat similar to Influxdb, but where influxdb is geared towards timeseries based data, elastic is geared towards "documents", which can be anything, including time.
Having used Elasticsearch on and off, I find it hard to deal within a personal environment (ingesting data/setting up indexes not straight forward).
I came across Zinc search (still in alpha stage), which is an alternative and supports the elasticsearch DSL query language. Ingestion is also compatible with elastic, with the ndjson format.
It has a rudimentary kibana-like search interface and has an API for ingesting/searching. It is schemaless and can create indexes on the fly.
This is pretty cool stuff to store/search IoT data as you can post your payloads directly to an index of your liking.
Example setup with docker-compose
services:
zinc:
container_name: zinc
image: public.ecr.aws/h9e2j3o7/zinc:latest
ports:
- "4080:4080"
environment:
TZ: 'Europe/Amsterdam'
ZINC_PROMETHEUS_ENABLE: 'true'
ZINC_TELEMETRY: 'disabled'
ZINC_FIRST_ADMIN_USER: 'admin'
ZINC_FIRST_ADMIN_PASSWORD: 'yoursecretpassword'
volumes:
- '/<path>/data:/data'
restart: unless-stopped
This exposes port 4080 with a webinterface for searching.
It uses basic auth for its API.
To post data, setup a http request node with basic authentication, set the method to PUT and the url to:
http://<ip>:4080/api/<index name>/document
The index name can be anything you want, eg: weather
it will create the index on the fly if it does not exist.
Inject an msg.payload with an object and see your data magically appear into the index (may need to refresh the page).
To search query, add a new http request node, set the method to POST, using basic auth:
const index = "<the index you used>"
msg.url = `http://<ip>:4080/api/${index}/_search`
msg.payload = {
"search_type":"alldocuments",
"max_results":100,
"sort_fields":["-@timestamp"],
"query":{"term":"","start_time":null,"end_time":null},
"fields":["_all"]
}
return msg;
The output is not (fully?) compatible with grafana, but there are requests to support it.
Personally I find this easier to work with than Influx, as influx needs a specific input format with tags/metrics, whereas with zinc you can simply inject your complete data object, which becomes searchable and aggregatable (like influx).
I don't know how it will perform with huge data loads, but for personal IoT I doubt it becomes a problem.
