Does anyone have any suggestions for how to operate Node Red on a couple of Pis in some sort of failover cluster? - my home automation pi died the day I went away for Christmas, with obviously irritating consequences...
1 Like
What is your Pi doing? If you are using the GPIO pins then is is more complicated. Are you using other devices that communicate with MQTT?
I currently run Node Red and Mosquitto on a 'main' server that is a normal PC running Ubuntu 18.04. Other Raspberry Pis on the network communicate through MQTT. If one goes down the server sends an email.
Another option I have used is an Arduino board that runs commands from Node Red through serial but has a default routine to run if connection is lost. Arduino boards are much more stable. I have used them in industrial environments with years of up-time.
1 Like
I'm also running 2 Pi's. As @willwelker says, you will have issues with physically connected hardware. In my case that's an Arduino that I've not got round to replacing with an ESP8266 and, more importantly, my RFXtrx433E that listens to and controls all of my 433MHz based remote switches and sensors. This is USB connected.
In my case, I make sure that everything that is physically connected is disambiguated via MQTT so that the majority of my flows can run from anywhere.
Each Pi reports, via both MQTT brokers, whether it is alive - or more accurately, whether Node-RED can talk to the network and each has a simple flow that reboots the device after a couple of minutes if it can't ping the local router.
Each Pi runs a broker and Node-RED of course and each has a Telegram bot defined so that I am kept informed. My core network devices (gateway, router and switch) along with both Pi's are all connected to a PC UPS.
With a couple of caveats, I've found this to be a VERY reliable solution capable of running for years at a time. Those caveats are:
- You need a really good SD card in the Pi - I always use Samsung Evo or Evo Pro cards. Also make them much bigger than you need - this gives more room for wear levelling.
- You need a good power supply for the Pi. Especially if you have additional hardware connected to it.
- My Pi2 which runs a much upgraded older version of Rasbian has a problem when the UPS switches to/from battery, the network connection is lost and doesn't come back without a reboot. Never managed to work out what causes that and I probably never will. It doesn't happen on my Pi 3.
I am slowly migrating the live Node-RED service to the Pi3 and when complete, I will rebuild the Pi2 to be the backup or possibly make it an internet facing server (DMZ) much locked down.
I also have a Synology NAS and I use one of the backup capabilities on that to reach out to the Pi and back up all of the key files so that it is easy to rebuild should I ever need to (I haven't in several years). When I do a major update of Node.JS or Node-RED. I also take a full copy of my live Node-RED so that it is trivial to roll back. Note that I don't install Node-RED globally (as per the default installation) as I find that too restrictive. I install to a folder and create my own systemd startup script - the userDir folder is a sub-folder of the main project. That way, I can easily take a full copy of the working version of Node-RED and restore it simply by renaming folders. I can also run multiple instances even with different versions of Node-RED if I need to.
1 Like
Hi... Everything I’m using is IP based, with the exception of a zwave usb stick.
I think I can add a second zwave controller on the failover pi. I think... I was also going to do a bit of research to see if there is such a thing as a usb->Ethernet bridge, I’m not holding out any hopes though.
Other than that it’s node Red and mosquito. I’m thinking something along the lines of a shared-nothing cluster, so a script to copy the userdir and mosquito settings/data from primary to secondary periodically to facilitate failover and startup of the services on the failover node.
Shared disk type cluster is out because it adds too much extra hardware and adds an additional single point of failure. (Yes, the network is a spof, but that’s pretty much unavoidable in a home nevironment.)
Very good point about wear levelling.
For GPIO redundancy.
I have Pi set up to control my sprinkler system with relay board. A redundant Pi is basically a clone that takes over if the primary stops responding to MQTT. A GPIO sends a LOW signal to a relay to switch it. The GPIO pins are set for PULLUP. Now if I use a diode in between each GPIO pin from both Pis to the relay board, either board can send a LOW signal to the relay. But an external PULLUP resistor is required since no HIGH signal can get through the diodes. This way, either Pi can control the relays without conflicting signal being possible.
2 Likes
Hey Will (@WillWelker), you have got my complete attention now! If it is not too much asked, could you please add a small drawing of the circuit and the materials used (resistor value, which diode). Thanks a lot!!!
I agree. While I don't need anything like this myself, I can see that many people would find it really useful.
In my project I included the diodes in the lead wires. In practice, failure of one Pi was so rare that I quit using a redundant Pi.
2 Likes
Hey Will (@WillWelker),
Happy Newyear!
I really appreciate the nice drawing you made!! But some more questions:
- Could you please share the type of diode that you have used.
- You use an external 3,3V power supply, instead of one of the Pi's power pins. I assume you do that since the power of a raspberry is gone if a Raspberry fails?
- Your diagram is for a GPIO output (to control a relay). But is something similar also possible for a GPIO input? I mean wire the same sensor to two raspberries (so both raspberries can read the same sensor signal).
You can wire one input up to GPIO pins on both boards and they should not interfere. Both should read it ok. However for connections such as a serial port or 1-Wire connections it gets more complicated.
1 Like
well.... mostly yes... but as they are GPIO - some may reset to being an output on failure so may drag the other input, so you would need to test/check before assuming this would work.
Good point. I presume that the solution is to not use as inputs, pins that default to outputs. Are there any such pins in fact?
I use something like the 1N4148 switching diode though you could get away with using most any one (even an LED).
The same concept could work in reverse for inputs. You could use a PULLDOWN resistor on each Pi GPIO input. then a diode inline from the signal source to each GPIO pin. So the diodes would let a HIGH signal go to each pin but would not let a HIGH or LOW go from one GPIO to the other. But as dceejay mentioned, you would want to test. Assuming your signal is 3.3v, a dead Pi could draw down the signal voltage. I have never tried that.
As a memory aid for using diodes, I consider the end of the diode that has the line, is the direction of 'positive electrical flow'. Since the schematic diagram looks like an arrow it makes it easy to see which direction the 'positive electricity' will flow even though actual electrons are doing the reverse.
**Edit to add:
just thought of a scenario where this could cause a problem. If you were sending a HIGH signal to the GPIO pins and for what ever reason one of the Pis set its GPIO to LOW output, this would conflict with the incoming HIGH signal. You would just need to test for you particualr application I guess.
1 Like
Ok thanks guys!!! Have just ordered a bunch of diodes to test later on ...
