Over the past week or so my grafana graphs have been displaying "NO DATA". I haven't been tweaking my system over that time.
There seems to be a problem with the port.
Node Red gives this error
"Error: connect ECONNREFUSED 127.0.0.1:8086"
Error: No host available
Grafana gives this error message
InfluxDB Error: Bad Gateway
Up until now this has been up, running and stable. Not sure what changed.
What's happened to my setup? How can I fix it?
Thx!
As we don't know your setup, difficult to say.
Gut feeling, influxdb stopped...
To check that influxdb is still running, then assuming you are running on a Debian based system, run
systemctl status influxdb
If that shows that it is still running and shows no errors then does
ping 127.0.0.1
work?
If not then what is in the file /etc/hosts?
yes, influx is running and I can ping the host with 0% packet loss. However I need to ping it several times before it will access.
pi@MicroPi:~ $ influx -host 192.168.1.154
Failed to connect to http://192.168.1.154:8086: Get http://192.168.1.154:8086/ping: dial tcp 192.168.1.154:8086: connect: connection refused
Please check your connection settings and ensure 'influxd' is running.
pi@MicroPi:~ $ influx -host 192.168.1.154
Failed to connect to http://192.168.1.154:8086: Get http://192.168.1.154:8086/ping: dial tcp 192.168.1.154:8086: connect: connection refused
Please check your connection settings and ensure 'influxd' is running.
pi@MicroPi:~ $ influx -host 192.168.1.154
Connected to http://192.168.1.154:8086 version 1.8.10
InfluxDB shell version: 1.8.10
>
/etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
127.0.1.1 MicroPi
More details. When I first go to the page some of the graphs may or may not display. Seems like, when it gets a new data point it will render the graph, when it gets the next data point it goes back to NO DATA.
Node red is complaining about 127.0.0.1, that is why I asked you to ping that.
Does influx -host 127.0.0.1 work reliably?
Show us the output from the systemd status command please.
systemctl status influxdb
● influxdb.service - InfluxDB is an open-source, distributed, time series databa
Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset:
Active: activating (start) since Mon 2025-01-13 08:33:52 EST; 17s ago
Docs: https://docs.influxdata.com/influxdb/
Cntrl PID: 1203 (influxd-systemd)
Tasks: 14 (limit: 4915)
CGroup: /system.slice/influxdb.service
├─1203 /bin/bash -e /usr/lib/influxdb/scripts/influxd-systemd-start.s
├─1205 /usr/bin/influxd -config /etc/influxdb/influxdb.conf
└─1579 sleep 1
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.8
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.8
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.8
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.8
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.8
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.8
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.9
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.9
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.9
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.9
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.9
lines 1-22/22 (END)...skipping...
● influxdb.service - InfluxDB is an open-source, distributed, time series database
Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset: enabled)
Active: activating (start) since Mon 2025-01-13 08:33:52 EST; 17s ago
Docs: https://docs.influxdata.com/influxdb/
Cntrl PID: 1203 (influxd-systemd)
Tasks: 14 (limit: 4915)
CGroup: /system.slice/influxdb.service
├─1203 /bin/bash -e /usr/lib/influxdb/scripts/influxd-systemd-start.sh
├─1205 /usr/bin/influxd -config /etc/influxdb/influxdb.conf
└─1579 sleep 1
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.838012Z lvl=info msg="Opened shard" log_id=0u3ymF3W000 service=stor
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.839054Z lvl=info msg="Opened file" log_id=0u3ymF3W000 engine=tsm1 s
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.848532Z lvl=info msg="Opened shard" log_id=0u3ymF3W000 service=stor
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.849588Z lvl=info msg="Opened file" log_id=0u3ymF3W000 engine=tsm1 s
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.896389Z lvl=info msg="Opened shard" log_id=0u3ymF3W000 service=stor
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.897922Z lvl=info msg="Opened file" log_id=0u3ymF3W000 engine=tsm1 s
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.917472Z lvl=info msg="Opened shard" log_id=0u3ymF3W000 service=stor
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.920527Z lvl=info msg="Opened file" log_id=0u3ymF3W000 engine=tsm1 s
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.957309Z lvl=info msg="Opened shard" log_id=0u3ymF3W000 service=stor
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.959354Z lvl=info msg="Opened file" log_id=0u3ymF3W000 engine=tsm1 s
Jan 13 08:34:09 MicroPi influxd-systemd-start.sh[1203]: ts=2025-01-13T13:34:09.977804Z lvl=info msg="Opened shard" log_id=0u3ymF3W000 service=stor
~
No, sometimes yes, sometimes no
Influx has only been running for 17 seconds. It is repeatedly stopping and restarting.
Are there any logs in /var/log/influxdb? If so look in there to get some clues.
No, the folder is empty
sudo journalctl -f -u influxdb.service
should show the service log
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.231121Z lvl=info msg="Opened file" log_id=0u42VM4W000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/outside/autogen/566/000000001-000000001.tsm id=0 duration=0.324ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.232607Z lvl=info msg="Opened shard" log_id=0u42VM4W000 service=store trace_id=0u42VMVl000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/outside/autogen/518 duration=48.011ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.234105Z lvl=info msg="Opened file" log_id=0u42VM4W000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/outside/autogen/59/000000001-000000001.tsm id=0 duration=0.287ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.238987Z lvl=info msg="Opened shard" log_id=0u42VM4W000 service=store trace_id=0u42VMVl000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/outside/autogen/542 duration=45.952ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.240462Z lvl=info msg="Opened file" log_id=0u42VM4W000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/outside/autogen/590/000000001-000000001.tsm id=0 duration=0.264ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.282645Z lvl=info msg="Opened shard" log_id=0u42VM4W000 service=store trace_id=0u42VMVl000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/outside/autogen/566 duration=52.885ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.282568Z lvl=info msg="Opened shard" log_id=0u42VM4W000 service=store trace_id=0u42VMVl000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/outside/autogen/590 duration=43.426ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.284712Z lvl=info msg="Opened file" log_id=0u42VM4W000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/outside/autogen/638/000000001-000000001.tsm id=0 duration=0.272ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.286840Z lvl=info msg="Opened file" log_id=0u42VM4W000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/outside/autogen/614/000000001-000000001.tsm id=0 duration=0.271ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.288641Z lvl=info msg="Opened shard" log_id=0u42VM4W000 service=store trace_id=0u42VMVl000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/outside/autogen/59 duration=55.834ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.290784Z lvl=info msg="Opened file" log_id=0u42VM4W000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/outside/autogen/662/000000001-000000001.tsm id=0 duration=0.230ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.327952Z lvl=info msg="Opened shard" log_id=0u42VM4W000 service=store trace_id=0u42VMVl000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/outside/autogen/614 duration=44.985ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.329823Z lvl=info msg="Opened file" log_id=0u42VM4W000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/outside/autogen/686/000000002-000000002.tsm id=0 duration=0.461ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.339679Z lvl=info msg="Opened shard" log_id=0u42VM4W000 service=store trace_id=0u42VMVl000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/outside/autogen/638 duration=56.352ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.341378Z lvl=info msg="Opened file" log_id=0u42VM4W000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/outside/autogen/710/000000002-000000002.tsm id=0 duration=0.256ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.363783Z lvl=info msg="Opened shard" log_id=0u42VM4W000 service=store trace_id=0u42VMVl000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/outside/autogen/662 duration=74.784ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.365210Z lvl=info msg="Reading file" log_id=0u42VM4W000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/pantry/autogen/1829/_00001.wal size=57640
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.404581Z lvl=info msg="Opened shard" log_id=0u42VM4W000 service=store trace_id=0u42VMVl000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/outside/autogen/686 duration=76.357ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.406120Z lvl=info msg="Opened file" log_id=0u42VM4W000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/outside/autogen/734/000000002-000000002.tsm id=0 duration=0.248ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.425947Z lvl=info msg="Opened shard" log_id=0u42VM4W000 service=store trace_id=0u42VMVl000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/outside/autogen/710 duration=86.084ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.427963Z lvl=info msg="Opened file" log_id=0u42VM4W000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/outside/autogen/74/000000001-000000001.tsm id=0 duration=0.298ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.452436Z lvl=info msg="Opened shard" log_id=0u42VM4W000 service=store trace_id=0u42VMVl000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/outside/autogen/734 duration=47.712ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.455663Z lvl=info msg="Opened file" log_id=0u42VM4W000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/outside/autogen/758/000000002-000000002.tsm id=0 duration=0.441ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.457559Z lvl=info msg="Opened shard" log_id=0u42VM4W000 service=store trace_id=0u42VMVl000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/pantry/autogen/1829 duration=93.572ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.459739Z lvl=info msg="Opened file" log_id=0u42VM4W000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/outside/autogen/782/000000002-000000002.tsm id=0 duration=0.359ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.466666Z lvl=info msg="Opened shard" log_id=0u42VM4W000 service=store trace_id=0u42VMVl000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/outside/autogen/74 duration=40.417ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.469027Z lvl=info msg="Opened file" log_id=0u42VM4W000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/outside/autogen/806/000000002-000000002.tsm id=0 duration=0.269ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.502860Z lvl=info msg="Opened shard" log_id=0u42VM4W000 service=store trace_id=0u42VMVl000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/outside/autogen/758 duration=50.272ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.505226Z lvl=info msg="Opened file" log_id=0u42VM4W000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/outside/autogen/830/000000002-000000002.tsm id=0 duration=0.253ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.506454Z lvl=info msg="Opened shard" log_id=0u42VM4W000 service=store trace_id=0u42VMVl000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/outside/autogen/782 duration=48.548ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.508594Z lvl=info msg="Opened file" log_id=0u42VM4W000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/outside/autogen/854/000000002-000000002.tsm id=0 duration=0.335ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.516803Z lvl=info msg="Opened shard" log_id=0u42VM4W000 service=store trace_id=0u42VMVl000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/outside/autogen/806 duration=49.991ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.518436Z lvl=info msg="Opened file" log_id=0u42VM4W000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/outside/autogen/878/000000002-000000002.tsm id=0 duration=0.275ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.707876Z lvl=info msg="Opened shard" log_id=0u42VM4W000 service=store trace_id=0u42VMVl000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/outside/autogen/878 duration=190.725ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.709249Z lvl=info msg="Opened file" log_id=0u42VM4W000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/outside/autogen/902/000000002-000000002.tsm id=0 duration=0.239ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.831281Z lvl=info msg="Opened shard" log_id=0u42VM4W000 service=store trace_id=0u42VMVl000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/outside/autogen/854 duration=324.612ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.833311Z lvl=info msg="Opened shard" log_id=0u42VM4W000 service=store trace_id=0u42VMVl000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/outside/autogen/830 duration=329.752ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.835044Z lvl=info msg="Opened file" log_id=0u42VM4W000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/outside/autogen/91/000000001-000000001.tsm id=0 duration=0.265ms
Jan 13 09:56:36 MicroPi influxd-systemd-start.sh[24873]: ts=2025-01-13T14:56:36.835317Z lvl=info msg="Opened file" log_id=0u42VM4W000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/outside/autogen/926/000000002-000000002.tsm id=0 duration=0.329ms
Then what?
Jan 13 13:09:49 MicroPi influxd-systemd-start.sh[4570]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1(0xfd18240, 0x76b8000)
Jan 13 13:09:49 MicroPi influxd-systemd-start.sh[4570]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:508 +0x48
Jan 13 13:09:49 MicroPi influxd-systemd-start.sh[4570]: created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions
Jan 13 13:09:49 MicroPi influxd-systemd-start.sh[4570]: /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:508 +0xfc
Jan 13 13:09:49 MicroPi systemd[1]: influxdb.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jan 13 13:09:49 MicroPi systemd[1]: influxdb.service: Failed with result 'exit-code'.
Jan 13 13:09:49 MicroPi systemd[1]: influxdb.service: Service RestartSec=100ms expired, scheduling restart.
Jan 13 13:09:49 MicroPi systemd[1]: influxdb.service: Scheduled restart job, restart counter is at 2673.
Jan 13 13:09:49 MicroPi systemd[1]: Stopped InfluxDB is an open-source, distributed, time series database.
Jan 13 13:09:49 MicroPi systemd[1]: Starting InfluxDB is an open-source, distributed, time series database...
Jan 13 13:09:49 MicroPi influxd-systemd-start.sh[5144]: Merging with configuration at: /etc/influxdb/influxdb.conf
Jan 13 13:09:49 MicroPi influxd-systemd-start.sh[5144]: ts=2025-01-13T18:09:49.807130Z lvl=info msg="InfluxDB starting" log_id=0u4DZpwl000 version=1.8.10 branch=1.8 commit=688e697c51fd
Jan 13 13:09:49 MicroPi influxd-systemd-start.sh[5144]: ts=2025-01-13T18:09:49.807938Z lvl=info msg="Go runtime" log_id=0u4DZpwl000 version=go1.13.8 maxprocs=4
Jan 13 13:09:49 MicroPi influxd-systemd-start.sh[5144]: Merging with configuration at: /etc/influxdb/influxdb.conf
Jan 13 13:09:49 MicroPi influxd-systemd-start.sh[5144]: InfluxDB API unavailable after 1 attempts...
Jan 13 13:09:49 MicroPi influxd-systemd-start.sh[5144]: ts=2025-01-13T18:09:49.917124Z lvl=info msg="Using data dir" log_id=0u4DZpwl000 service=store path=/var/lib/influxdb/data
Jan 13 13:09:49 MicroPi influxd-systemd-start.sh[5144]: ts=2025-01-13T18:09:49.917229Z lvl=info msg="Compaction settings" log_id=0u4DZpwl000 service=store max_concurrent_compactions=2 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
Jan 13 13:09:49 MicroPi influxd-systemd-start.sh[5144]: ts=2025-01-13T18:09:49.917286Z lvl=info msg="Open store (start)" log_id=0u4DZpwl000 service=store trace_id=0u4DZqNG000 op_name=tsdb_open op_event=start
Jan 13 13:09:49 MicroPi influxd-systemd-start.sh[5144]: ts=2025-01-13T18:09:49.992569Z lvl=info msg="Opened file" log_id=0u4DZpwl000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/_internal/monitor/1824/000000321-000000004.tsm id=0 duration=28.338ms
Jan 13 13:09:50 MicroPi influxd-systemd-start.sh[5144]: ts=2025-01-13T18:09:49.992497Z lvl=info msg="Opened file" log_id=0u4DZpwl000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/_internal/monitor/1800/000000321-000000004.tsm id=0 duration=22.397ms
That is from hours later. We need to see a complete log from influxdb starting to the next restart.
Does this tell us anything?
influxdb.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jan 14 11:03:06 MicroPi systemd[1]: influxdb.service: Failed with result 'exit-code'.
Post a complete sequence from influx startup to when it fails and restarts again.
The forum will only let me post so much of this. Anyway I found these messages that say it's running out of memory.
Jan 14 11:37:06 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:06.863534Z lvl=info msg="Opened shard" log_id=0u5QdS_G000 service=store trace_id=0u5QdT00000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/outsideLowHigh/autogen/1657 duration=71.271ms
Jan 14 11:37:06 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:06.882649Z lvl=info msg="Opened shard" log_id=0u5QdS_G000 service=store trace_id=0u5QdT00000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/outsideLowHigh/autogen/1684 duration=65.304ms
Jan 14 11:37:07 MicroPi influxd-systemd-start.sh[7826]: InfluxDB API unavailable after 21 attempts...
Jan 14 11:37:08 MicroPi influxd-systemd-start.sh[7826]: InfluxDB API unavailable after 22 attempts...
Jan 14 11:37:08 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:08.677743Z lvl=info msg="Reading file" log_id=0u5QdS_G000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/rmh/autogen/1816/_00002.wal size=8181246
Jan 14 11:37:09 MicroPi influxd-systemd-start.sh[7826]: InfluxDB API unavailable after 23 attempts...
Jan 14 11:37:10 MicroPi influxd-systemd-start.sh[7826]: InfluxDB API unavailable after 24 attempts...
Jan 14 11:37:10 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:10.477091Z lvl=info msg="Opened shard" log_id=0u5QdS_G000 service=store trace_id=0u5QdT00000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/rmh/autogen/1816 duration=4443.078ms
Jan 14 11:37:10 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:10.532327Z lvl=info msg="Open store (end)" log_id=0u5QdS_G000 service=store trace_id=0u5QdT00000 op_name=tsdb_open op_event=end op_elapsed=24579.465ms
Jan 14 11:37:10 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:10.532493Z lvl=info msg="Opened service" log_id=0u5QdS_G000 service=subscriber
Jan 14 11:37:10 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:10.532520Z lvl=info msg="Starting monitor service" log_id=0u5QdS_G000 service=monitor
Jan 14 11:37:10 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:10.532544Z lvl=info msg="Registered diagnostics client" log_id=0u5QdS_G000 service=monitor name=build
Jan 14 11:37:10 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:10.532563Z lvl=info msg="Registered diagnostics client" log_id=0u5QdS_G000 service=monitor name=runtime
Jan 14 11:37:10 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:10.532582Z lvl=info msg="Registered diagnostics client" log_id=0u5QdS_G000 service=monitor name=network
Jan 14 11:37:10 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:10.532614Z lvl=info msg="Registered diagnostics client" log_id=0u5QdS_G000 service=monitor name=system
Jan 14 11:37:10 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:10.532677Z lvl=info msg="Starting precreation service" log_id=0u5QdS_G000 service=shard-precreation check_interval=10m advance_period=30m
Jan 14 11:37:10 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:10.532937Z lvl=info msg="Starting snapshot service" log_id=0u5QdS_G000 service=snapshot
Jan 14 11:37:10 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:10.532960Z lvl=info msg="Starting continuous query service" log_id=0u5QdS_G000 service=continuous_querier
Jan 14 11:37:10 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:10.533053Z lvl=info msg="Storing statistics" log_id=0u5QdS_G000 service=monitor db_instance=_internal db_rp=monitor interval=10s
Jan 14 11:37:10 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:10.533189Z lvl=info msg="Starting HTTP service" log_id=0u5QdS_G000 service=httpd authentication=false
Jan 14 11:37:10 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:10.533342Z lvl=info msg="opened HTTP access log" log_id=0u5QdS_G000 service=httpd path=stderr
Jan 14 11:37:10 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:10.533890Z lvl=info msg="Listening on HTTP" log_id=0u5QdS_G000 service=httpd addr=[::]:8086 https=false
Jan 14 11:37:10 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:10.534079Z lvl=info msg="Starting retention policy enforcement service" log_id=0u5QdS_G000 service=retention check_interval=30m
Jan 14 11:37:10 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:10.534589Z lvl=info msg="Listening for signals" log_id=0u5QdS_G000
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: [httpd] ::1 - - [14/Jan/2025:11:37:11 -0500] "GET /health HTTP/1.1" 200 107 "-" "curl/7.64.0" cd92f995-d295-11ef-8001-e45f01115804 325
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: InfluxDB started
Jan 14 11:37:11 MicroPi systemd[1]: Started InfluxDB is an open-source, distributed, time series database.
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.486732Z lvl=info msg="TSM compaction (start)" log_id=0u5QdS_G000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0u5Qf0kW000 op_name=tsm1_compact_group db_shard_id=1826 op_event=start
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.486788Z lvl=info msg="Beginning compaction" log_id=0u5QdS_G000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0u5Qf0kW000 op_name=tsm1_compact_group db_shard_id=1826 tsm1_files_n=13
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.486809Z lvl=info msg="Compacting file" log_id=0u5QdS_G000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0u5Qf0kW000 op_name=tsm1_compact_group db_shard_id=1826 tsm1_index=0 tsm1_file=/var/lib/influxdb/data/_internal/monitor/1826/000000129-000000004.tsm
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.486830Z lvl=info msg="Compacting file" log_id=0u5QdS_G000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0u5Qf0kW000 op_name=tsm1_compact_group db_shard_id=1826 tsm1_index=1 tsm1_file=/var/lib/influxdb/data/_internal/monitor/1826/000000257-000000004.tsm
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.486849Z lvl=info msg="Compacting file" log_id=0u5QdS_G000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0u5Qf0kW000 op_name=tsm1_compact_group db_shard_id=1826 tsm1_index=2 tsm1_file=/var/lib/influxdb/data/_internal/monitor/1826/000000289-000000003.tsm
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.486868Z lvl=info msg="Compacting file" log_id=0u5QdS_G000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0u5Qf0kW000 op_name=tsm1_compact_group db_shard_id=1826 tsm1_index=3 tsm1_file=/var/lib/influxdb/data/_internal/monitor/1826/000000297-000000002.tsm
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.486886Z lvl=info msg="Compacting file" log_id=0u5QdS_G000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0u5Qf0kW000 op_name=tsm1_compact_group db_shard_id=1826 tsm1_index=4 tsm1_file=/var/lib/influxdb/data/_internal/monitor/1826/000000305-000000002.tsm
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.486904Z lvl=info msg="Compacting file" log_id=0u5QdS_G000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0u5Qf0kW000 op_name=tsm1_compact_group db_shard_id=1826 tsm1_index=5 tsm1_file=/var/lib/influxdb/data/_internal/monitor/1826/000000313-000000002.tsm
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.486923Z lvl=info msg="Compacting file" log_id=0u5QdS_G000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0u5Qf0kW000 op_name=tsm1_compact_group db_shard_id=1826 tsm1_index=6 tsm1_file=/var/lib/influxdb/data/_internal/monitor/1826/000000314-000000001.tsm
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.486941Z lvl=info msg="Compacting file" log_id=0u5QdS_G000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0u5Qf0kW000 op_name=tsm1_compact_group db_shard_id=1826 tsm1_index=7 tsm1_file=/var/lib/influxdb/data/_internal/monitor/1826/000000315-000000001.tsm
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.486962Z lvl=info msg="Compacting file" log_id=0u5QdS_G000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0u5Qf0kW000 op_name=tsm1_compact_group db_shard_id=1826 tsm1_index=8 tsm1_file=/var/lib/influxdb/data/_internal/monitor/1826/000000316-000000001.tsm
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.486980Z lvl=info msg="Compacting file" log_id=0u5QdS_G000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0u5Qf0kW000 op_name=tsm1_compact_group db_shard_id=1826 tsm1_index=9 tsm1_file=/var/lib/influxdb/data/_internal/monitor/1826/000000317-000000001.tsm
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.486998Z lvl=info msg="Compacting file" log_id=0u5QdS_G000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0u5Qf0kW000 op_name=tsm1_compact_group db_shard_id=1826 tsm1_index=10 tsm1_file=/var/lib/influxdb/data/_internal/monitor/1826/000000318-000000001.tsm
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.487018Z lvl=info msg="Compacting file" log_id=0u5QdS_G000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0u5Qf0kW000 op_name=tsm1_compact_group db_shard_id=1826 tsm1_index=11 tsm1_file=/var/lib/influxdb/data/_internal/monitor/1826/000000319-000000001.tsm
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.487036Z lvl=info msg="Compacting file" log_id=0u5QdS_G000 engine=tsm1 tsm1_strategy=full tsm1_optimize=false trace_id=0u5Qf0kW000 op_name=tsm1_compact_group db_shard_id=1826 tsm1_index=12 tsm1_file=/var/lib/influxdb/data/_internal/monitor/1826/000000320-000000001.tsm
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.490251Z lvl=info msg="TSM compaction (start)" log_id=0u5QdS_G000 engine=tsm1 tsm1_level=2 tsm1_strategy=level trace_id=0u5Qf0lW000 op_name=tsm1_compact_group db_shard_id=1827 op_event=start
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.490308Z lvl=info msg="Beginning compaction" log_id=0u5QdS_G000 engine=tsm1 tsm1_level=2 tsm1_strategy=level trace_id=0u5Qf0lW000 op_name=tsm1_compact_group db_shard_id=1827 tsm1_files_n=4
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.490328Z lvl=info msg="Compacting file" log_id=0u5QdS_G000 engine=tsm1 tsm1_level=2 tsm1_strategy=level trace_id=0u5Qf0lW000 op_name=tsm1_compact_group db_shard_id=1827 tsm1_index=0 tsm1_file=/var/lib/influxdb/data/_internal/monitor/1827/000000008-000000002.tsm
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.490349Z lvl=info msg="Compacting file" log_id=0u5QdS_G000 engine=tsm1 tsm1_level=2 tsm1_strategy=level trace_id=0u5Qf0lW000 op_name=tsm1_compact_group db_shard_id=1827 tsm1_index=1 tsm1_file=/var/lib/influxdb/data/_internal/monitor/1827/000000016-000000002.tsm
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.490368Z lvl=info msg="Compacting file" log_id=0u5QdS_G000 engine=tsm1 tsm1_level=2 tsm1_strategy=level trace_id=0u5Qf0lW000 op_name=tsm1_compact_group db_shard_id=1827 tsm1_index=2 tsm1_file=/var/lib/influxdb/data/_internal/monitor/1827/000000025-000000002.tsm
Jan 14 11:37:11 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:11.490387Z lvl=info msg="Compacting file" log_id=0u5QdS_G000 engine=tsm1 tsm1_level=2 tsm1_strategy=level trace_id=0u5Qf0lW000 op_name=tsm1_compact_group db_shard_id=1827 tsm1_index=3 tsm1_file=/var/lib/influxdb/data/_internal/monitor/1827/000000038-000000002.tsm
Jan 14 11:37:12 MicroPi influxd-systemd-start.sh[7826]: [httpd] 127.0.0.1 - root [14/Jan/2025:11:37:12 -0500] "POST /write?db=apartment&p=%5BREDACTED%5D&precision=n&rp=&u=root HTTP/1.1 " 204 0 "-" "-" ce39bf6f-d295-11ef-8002-e45f01115804 7978
Jan 14 11:37:12 MicroPi influxd-systemd-start.sh[7826]: [httpd] 127.0.0.1 - root [14/Jan/2025:11:37:12 -0500] "POST /write?db=apartment&p=%5BREDACTED%5D&precision=n&rp=&u=root HTTP/1.1 " 204 0 "-" "-" ce830de0-d295-11ef-8003-e45f01115804 11593
Jan 14 11:37:12 MicroPi influxd-systemd-start.sh[7826]: [httpd] 127.0.0.1 - root [14/Jan/2025:11:37:12 -0500] "POST /write?db=apartment&p=%5BREDACTED%5D&precision=n&rp=&u=root HTTP/1.1 " 204 0 "-" "-" ce831c8a-d295-11ef-8004-e45f01115804 22442
Jan 14 11:37:12 MicroPi influxd-systemd-start.sh[7826]: [httpd] 127.0.0.1 - root [14/Jan/2025:11:37:12 -0500] "POST /write?db=outside&p=%5BREDACTED%5D&precision=n&rp=&u=root HTTP/1.1 " 204 0 "-" "-" ce853b67-d295-11ef-8006-e45f01115804 29702
Jan 14 11:37:12 MicroPi influxd-systemd-start.sh[7826]: [httpd] 127.0.0.1 - root [14/Jan/2025:11:37:12 -0500] "POST /write?db=pantry&p=%5BREDACTED%5D&precision=n&rp=&u=root HTTP/1.1 " 204 0 "-" "-" ce83ab51-d295-11ef-8005-e45f01115804 45609
Jan 14 11:37:13 MicroPi influxd-systemd-start.sh[7826]: [httpd] 127.0.0.1 - root [14/Jan/2025:11:37:13 -0500] "POST /write?db=compost&p=%5BREDACTED%5D&precision=n&rp=&u=root HTTP/1.1 " 204 0 "-" "-" cea5d5e8-d295-11ef-8007-e45f01115804 10944
Jan 14 11:37:13 MicroPi influxd-systemd-start.sh[7826]: [httpd] 127.0.0.1 - root [14/Jan/2025:11:37:13 -0500] "POST /write?db=compost&p=%5BREDACTED%5D&precision=n&rp=&u=root HTTP/1.1 " 204 0 "-" "-" cea5e9d6-d295-11ef-8008-e45f01115804 16338
Jan 14 11:37:13 MicroPi influxd-systemd-start.sh[7826]: ts=2025-01-14T16:37:13.268878Z lvl=info msg="Sending usage statistics to usage.influxdata.com" log_id=0u5QdS_G000
Jan 14 11:37:15 MicroPi influxd-systemd-start.sh[7826]: [httpd] 127.0.0.1 - root [14/Jan/2025:11:37:15 -0500] "POST /write?db=apartment&p=%5BREDACTED%5D&precision=n&rp=&u=root HTTP/1.1 " 204 0 "-" "-" d0075007-d295-11ef-8009-e45f01115804 10853
Jan 14 11:37:15 MicroPi influxd-systemd-start.sh[7826]: [httpd] 127.0.0.1 - root [14/Jan/2025:11:37:15 -0500] "POST /write?db=apartment&p=%5BREDACTED%5D&precision=n&rp=&u=root HTTP/1.1 " 204 0 "-" "-" d009318d-d295-11ef-800a-e45f01115804 10537
Jan 14 11:37:16 MicroPi influxd-systemd-start.sh[7826]: runtime: out of memory: cannot allocate 24576-byte block (711196672 in use)
Jan 14 11:37:16 MicroPi influxd-systemd-start.sh[7826]: fatal error: out of memory
Jan 14 11:37:16 MicroPi influxd-systemd-start.sh[7826]: runtime: out of memory: cannot allocate 16384-byte block (711196672 in use)
Jan 14 11:37:16 MicroPi influxd-systemd-start.sh[7826]: fatal error: out of memory
Jan 14 11:37:16 MicroPi influxd-systemd-start.sh[7826]: goroutine 8684 [running]:
Jan 14 11:37:16 MicroPi influxd-systemd-start.sh[7826]: runtime.throw(0xfebdde, 0xd)
Jan 14 11:37:16 MicroPi influxd-systemd-start.sh[7826]: /usr/local/go/src/runtime/panic.go:774 +0x5c fp=0x7f5f364 sp=0x7f5f350 pc=0x41644
Jan 14 11:37:16 MicroPi influxd-systemd-start.sh[7826]: runtime.(*mcache).refill(0xb6ef9a34, 0x55)
Jan 14 11:37:16 MicroPi influxd-systemd-start.sh[7826]: /usr/local/go/src/runtime/mcache.go:140 +0xfc fp=0x7f5f378 sp=0x7f5f364 pc=0x262ec
Jan 14 11:37:16 MicroPi influxd-systemd-start.sh[7826]: runtime.(*mcache).nextFree(0xb6ef9a34, 0xd55, 0x0, 0x386c5a0, 0xb6ef9a34)
Jan 14 11:37:16 MicroPi influxd-systemd-start.sh[7826]: /usr/local/go/src/runtime/malloc.go:854 +0x7c fp=0x7f5f398 sp=0x7f5f378 pc=0x1b0f4
That would explain it then.
What does the top command say about memory usage?
Seems to be using more CPU than memory. The CPU reading is almost always above 100% The memory hovers around 10%
MiB Mem : 7467.1 total, 4858.7 free, 947.4 used, 1660.9 buff/cache
MiB Swap: 100.0 total, 0.0 free, 100.0 used. 5858.2 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
971 influxdb 20 0 2951004 786876 222600 R 307.9 10.3 0:49.42 influxd
Of course, as it is continually starting and crashing, most of the time it is just starting up, reading in the database config and so on, so you may not ever see the memory utilisation high.
This obviously isn't a node-red issue, I suspect that your database is just too big. Googling for
influxd-systemd-start.sh runtime: out of memory: cannot allocate block
may be helpful. I suggest stopping everything you can on the system in order to free up as much memory as possible. Are you running a browser or GUI on the device? If so shut those down and connect using SSH. Obviously shut down node-red so that it is not trying to add to the db. If you can get influxd to start successfully then you can delete some of the data. Otherwise I suspect you may not be able to recover the data, unless you can fit more RAM.
Possibly posting in an influxdb forum may get you somewhere.
If you do find a solution then post back here in case anyone else has the same problem.
A thought though, does influxd have access to the full memory in your system?
Copyright OpenJS Foundation and Node-RED contributors. All rights reserved. The OpenJS Foundation has registered trademarks and uses trademarks. For a list of trademarks of the OpenJS Foundation, please see our Trademark Policy and Trademark List. Trademarks and logos not indicated on the list of OpenJS Foundation trademarks are trademarks™ or registered® trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.
The OpenJS Foundation | Terms of Use | Privacy Policy | OpenJS Foundation Bylaws | Trademark Policy | Trademark List | Cookie Policy