Influxdb upgrade from v1.8.6 broken [FIX] v1.8.9 still broken

I know this is not a Node-Red issue per se, but I'm sure some are interested.

A few weeks ago I updated my pi4, which had a working v1.8.6 influxdb. Not until today, did I try to access it via Grafana only to be greeted by this error 'InfluxDB Error: Bad Gateway' after some RPi CLI I find Influx is not running, after an hour of reading online, trying the CHMOD fix (did not work), uninstalling directly installing later versions from v1.8.7 through to latest version; still I got the same error:

'influxdb_1.8.9 failed to connect to http://localhost:8086: get http://localhost:8086/ping: dial tcp [::1]:8086: connect: connection refused'

So I found this link with the old 'working version' of Influxdb v 1.8.6 which I used to re-install all worked perfect :smile:

Check your OS version (32-64bit) (armhf works for RPi4 32bit OS)

4 Likes

Yes 1.8.6 is the stable version of Influxdb.

1.8.7 - 1.8.9 did not solve the problem. It has been more than a month.
To make things worse, RPI Debian pushes the update to version 1.8.9 and break many systems. We have to look into alternative databases down the road.

Hopefully Influxdb developers can fix it at 1.8.10.

I had this problem yesterday and after a great deal of searching, I found that it was one of my databases kicking off, possibly on a particular shard ( displayed the error). This was after an update from InfluxDB 1.8.6 > 1.8.9 (iirc).

Rightly or wrongly I used this command to change permissions and all worked again!

sudo chown -R influxdb:influxdb /var/lib/influxdb/*

This was a command given in an InfluxDB thread/info page which I will endeavour to find over the next few hours, it doesn't seem to appear in the myriad of links I checked out yesterday!!

HTH

(RPi4 running Buster)

Found it -
Troubleshoot systemd errors

I followed some fault finding in another Influx Community thread (again seemingly lost!) which basically involved checking if the service was running and restarting making sure influxd was started as user not admin.

They should have been set to that already.

I guessed so, but I can only comment on what worked for me.

InfluxDB has been running for a long time on this RPi (12-18 months?) and all of a sudden (possibly after an update - not 100% sure) it stopped working. It seemed to concern a particular Shard and from all reading, it appears that :8086 is the last thing to be opened on a startup. I am afraid that I only know enough about Linux to be very dangerous! Hopefully, these events add to the experience I have and improve my knowledge!!

I spent several hours chasing this, entered the above command and it all worked again. Could have been a corruption, but there were no power glitches as far as I am aware and all other functions on the Pi seem to be working.

UPDATE: This was the thread whose diagnostics lead me to the solution
Run: open server: listen: listen tcp 127.0.0.1:8088: bind: address already in use

I tried several of the suggested 'permissions' fixes, one did work but did not persist after reboot, for now I have placed my working v1.8.6 package on hold: sudo apt-mark hold influxdb @mudwalker is your influxdb ok after reboot?

OK, just done a reboot and NO!

Now to work out how to downgrade!!

ADDED:
Just for the record:
Downloaded from the link above
sudo dpkg -i influxdb_1.8.6_armhf.deb
sudo apt-mark hold influxdb

(There learnt something!! :rofl:)

1 Like

To get 1.8.9 load, you also need to do the following work now:

  1. Edit the file /etc/systemd/system/influxd.service
  2. Find the line "Type=forking" and comment it out by putting a '#" in front:
    Restart=on-failure
    #Type=forking
    PIDFile=/var/lib/influxdb/influxd.pid
  3. sudo systemctl daemon-reload
  4. Now you should be able to start the database with command: "sudo systemctl start influxdb"
2 Likes

Thank you for this, it is much appreciated. Although the experimenter in me wants to do this, I am going to leave things as they are until the new release. At the end of the day, I am just not that good to debug problems with the software.

Word is that it is fixed in 1.8.10rc0 (near the end)

1 Like

It is a wise idea. It is the best to let the official release fix the bug, instead of spending lots of effort on customizing it.

Before the fix happens, it is the best to stay at 1.8.6 :grinning_face_with_smiling_eyes:

1 Like

Commenting out Type=forking worked perfectly for me!!!! Thank you!!!

1 Like

Just found this thread after my setup on a pi zero is now broken after updating.
For some reason influxd is now trying to run on port 8088.

The influx command still tries to connect on 8086.

Looks like a full system restore may be in order.

Check in the influxdb conf file, probably /etc/influxdb/influxdb.conf

I had exactly this problem, and downloading influxdb_1.8.6_armhf.deb from the link at the top of this thread, reinstalling as per mudwalker's instructions, cured it for me.
It's clearly a problem with 1.8.9

I am stuck with high CPU load by the influxd process. I was able to fix the problem that after a reboot I got the "connect: connection refused'" error due to some timing issues. I fixed this with giving it more time and with the 'no-forking' hint.
But the CPU load (about 150% on a Raspberry Pi 4 instead of about 15%) still remains. (no special retention policies, about 1-2 entries per second into the database). This makes the Pi unnecessary hot and wastes energy. I would like to solve that problem.

I found the hints quoted above.
What I'm not sure about is the 'downloaded from the link above'.
My knowledge on these topics is zero. I do know how to google and copy/paste step-by-step solutions.
What exact link and what is the exact download command. Into which directory? This is probably very obvious for many people, but not for me. I would very much appreciate if someone could post here a step by step recipe on how to downgrade from 1.8.9 to 1.8.6 and to keep it there. I'm doing apt-get update regularly since I learned that this is an important thing to do.
Thanks for any help here.
Urs.

Download from this link to your Download directory...

https://newreleases.io/project/github/influxdata/influxdb/release/v1.8.6

Change directory to Download directory
cd ./Downloads

and then input the commands above...
sudo dpkg -i influxdb_1.8.6_armhf.deb
sudo apt-mark hold influxdb

(Assuming Raspberry Pi 4)

1 Like

Thanks 'mudwalker'.

  • First I stopped all input from node-red to influxdb
  • sudo systemctl stop influxdb (to stop the influxdb, do not know if this is needed)
  • sudo dpkg -i influxdb_1.8.6_armhf.deb (from the directory where I downloaded the .deb file)
  • sudo systemctl start influxdb
  • sudo apt-mark hold influxdb
  • reconnected the influxdb node in node-red (speed node shows about 55 msgs/min)

The output of these commands, when there were any, looked reassuring.

As a result it seems the influxd process uses an average of about 110% CPU. This is down from the previous 150%. We are still not where we were initially, but still, this worked out well.

Now doing a step back and looking at it. The newer software uses 40% more CPU load for exactly the same task as the older one. Sadly in the IT world we are used to this. And since I did not pay anything for the influxdb SW, I'm not going to complain. But I would be willing to pay something to avoid this.

As mentioned previously, look in the logs. When something unexpected is happening that is usually the first thing to do.

Hello Colin
Yes, check the logs, I hear you.
In my frenzy to reduce the load I switched off logging wherever I could. The /var/log/influxdb directory is empty. There is absolutely nothing in the system logs which has any relation to influxdb. The most recent log entries are about me when I logged into the Pi4.
Do you have any hints about what logging I should enable (probably something in /etc/influxdb/influxdb.conf ?

I missed this thread previously - is this a Pi problem?

Because I've not had any problems with v1.8.9 on my Intel 64-bit Debian Buster system. I've not messed with or changed anything, InfluxDB is running on standard settings.

I install InfluxDB from the recommended repo: https://repos.influxdata.com/debian buster InRelease