Influx stopped writing data

Before going to the influx forum I thought I would try here in case anyone has seen a similar problem.
After working perfectly for months influx suddenly stopped writing data at exactly midnight UTC this morning. I was not doing anything at that time (other than snoring possibly) and the Pi did not not reboot. I have enabled all the logs in influx and can see it apparently accepting the write requests and not generating any errors. When I look at the files in the influx data directories I can see that it is not writing anything. When I use the influx command line to insert data it also looks ok from the logs but there is no data added.
I have run out of ideas. Has anyone any suggestions before I subscribe to the influx forum and ask there?

SD card OK? The midnight thing is weird!

I had one time similar issues with a wrong retention policy, but i could locate the problem with the log file...

It is going to a usb disc. I have checked on startup that it is opening the right folders and that they are writable.

Yes, I thought along those lines too, but using the command line client I have checked for retention policies and there is only the one.

If you enabled all the possible logging options, the problem should show itself somewhere in the logs...hopefully :slight_smile:

using SSL ? certificate expired ?

I believe I have enabled everything, I see this when it writes

May 20 14:15:33 owl2 influxd[18436]: [httpd] 2019/05/20 14:15:33 Write body received by handler: tydwr conservatory/power=0
May 20 14:15:33 owl2 influxd[18436]: [httpd] - root [20/May/2019:14:15:33 +0100] "POST /write?db=tydwr&p=%5BREDACTED%5D&precision=n&rp=&u=root HTTP/1.1" 204 0 "-" "-" 59584af6-7b01-11e9-827a-000000000000 4583

I don't know what the numbers in the POST mean, perhaps I need to find out.

Did you try to create a new database and to write there ?

Is that 204 an http error ?

looks the same here...

Mai 20 19:16:00 Big influxd[760]: [httpd] - - [20/May/2019:19:16:00 +0200] "POST /write?db=infra&precision=ms HTTP/1.1" 204 0 "-" "-" f06a1f92-7b22-11e9-806f-000000000000 234
Mai 20 19:16:00 Big influxd[760]: [httpd] - - [20/May/2019:19:16:00 +0200] "POST /write?db=infra&precision=ms HTTP/1.1" 204 0 "-" "-" f06b5fe0-7b22-11e9-8070-000000000000 328

I had ones the the same problem, writing data without error, but Infux Admin didn't show the new entry's.
Rebuilding the influx index files helped me.

# Shutdown InfluxDB
systemctl stop influxdb

# Remove all TSI index files
rm -r /var/lib/influxdb/data/*/*/*/index

# Rebuild TSI index files
su --command "influx_inspect buildtsi -datadir /var/lib/influxdb/data -waldir /var/lib/influxdb/wal" influxdb

# Start InfluxDB again
systemctl start influxdb

Thanks for the suggestion but I don't think that is valid any more. There aren't any TSI index files, it is TSM now apparently. There may be an equivalent or TSM but I didn't find it.

That's a good idea I thought so tried it. The create database seemed to work and SHOW DATABASES showed it, but writes to it still just seemed to be ignored.
There isn't anything vital in the history so I stopped influxdb, copied the data and wal directories in case I wanted to go back and removed the contents of data and wal, then restarted influxdb and it all seems to be going ok again.
I realise I am a bit out of data with updates, using 1.1.1 whereas I think 1.7 is current, so probably I should upgrade.
Thanks everyone for the suggestions.

Well that is pretty old. I try to keep mine up-to-date and haven't had any issues with updates.

Yes, I will upgrade it. Perhaps the problem was a bug that has been fixed.

tar xvfz influxdb-1.7.6_linux_armhf.tar.gz

I have it installed via an apt package so it gets updated via sudo apt-get update && sudo apt-get upgrade like everything else.

In fact on my Pi3, I have webmin installed and that is set to automatically check for updates every 24 hours and updates everything for me - I love automation!

1 Like

I should think so too! :slight_smile:

Is that from the repo used here?

This pi is still on jessie, it is not clear from the post whether there is a jessie version there. I will have to investigate.