"The flows on the server have been updated" with Pi gpiod nodes

General
1

My project is running on a Raspberry Pi 4, with Node Red running in Docker and I am deploying this to the Pi through Balena io. The project has two Docker images - one running the Pi GPIOD and the other running Node Red. I am using the Raspberry Pi GPIO node-red-node-pi-gpiod to control the Pi GPIO pins. This much you can find on my GitHub balena-node-red-raspberry-pi-gpio]. It works fine.

Unfortunately I am running into an issue very similar to this issue here. Every time the Pi is restarted the browser gives me the "The flows on the server have been updated" message. Reloading the window wipes the flow. Going through the dialogs and clicking 'Cancel' brings the flow back, albeit without the MQTT credentials which I have to re-enter.

The project is using only standard flows - MQTT in/out, CSV, switch and a conversion from string to int, and the pigpiod nodes. Following from the hint in the existing thread if I remove the GPIO control nodes then the reboot goes through as expected, if I put the nodes back to control the pins then I get the merge issue. flows.json is also in my GitHub.

Can anyone help with a workaround? This is really spoiling the fun of Node Red in an otherwise fun project.

2

Hi
I'm no docker expert but I believe it basically runs things in a sandbox environment so maybe its not saving your flows when it restarts?

Much easier to play with Node-RED natively on a Pi using the install script here

https://nodered.org/docs/getting-started/raspberrypi

3

Many thanks for the reply - I thought the same thing initially and have been adding a Docker volume to the container which is certainly permanent. But this is likely not the problem.

While reading around the issue I stumbled across the previous issue that is very similar - and discovered that when I remove the gpiod nodes the restart works fine and does not require merging. It seems there is a bug somewhere, perhaps in the node-red-node-pi-gpiod nodes. Some of the links from the previous issue are dead and I cannot find what the fix is.

4

To think that Pi devices are now BIG enough to run docker instances... how the world has changed! Coming from running docker, if not KVM instances, on servers with 32GB, 64GB, later on 128GB and even more, the Pi is not just a fun micro computer anymore!

5

The Pi is really a powerful computer in a small package. Most of my working life I have written bare-metal C on very small ARM devices (26MHz/128KB RAM) so a multi-core GHz Pi with GB RAM is not small at all. What is really nice to see is the increasingly small delta between Intel/AMD builds and ARM for almost everything Linux. We can expect this to accelerate as Apple goes ARM.

Anyway, back to the problem at hand... this is driving me a bit nuts. I want to use the Pi to control a real-world system (irrigation, as it happens) but if it cannot restart cleanly then it is not reliable enough to deploy.

6

Have you tried without docker? Do you get the same issue?

Also, what does the merge reveal? I know from updating some of my contrib nodes, in trying to maintain compatibility when adding new features or tweaking the UI layout, that I default or update some client side properties that cause the node to show as requiring a deploy but that should only be a one time operation (and not disruptive) but the nodes in question might need a bit TLC.

Can you locate the GitHub repository of the node in question? Perhaps raise an issue?

7

Right, after almost 20 years with virtualization... all types, definitely do a sanity build NOT in docker of the total solution. This is a divide and conquer thing. Make sure everything is legit before you add the complexity of virtualization, application virtualization in the case of docker.

As a side note, I have been trying to batch load packages via npm cli and I have seen some odd issues getting node red to see the packages consistently. I only mention this in case you might be tripping over something like I have been. If load the packages via the palette rarely have issues, if I load cli one package at a time, sometimes I see something odd. but batching them seems to be an issue, unless I am doing something wrong and just do not realize it... I opened a different topic on same.

8

Many thanks for the replies. First let's deal with the Docker thing here - Docker is likely not the problem here, at least not alone. The cause in the previous issue (read it here) was one of the nodes misbehaving and here too if I remove the Pi GPIO nodes then the whole problem goes away. I will build a test outside Docker which will be interesting anyway but Docker images are not so mysterious.

9

If the flows haven't actually changed at all then the only difference could be the perceived timestamp of the flow file. If for some reason it appears to be more recent then it may trigger this.

10

OK I can have a look at that. For 'more recent' what would the reference be? The flows haven't changed, the flows.json and other config files persist in the /root/.node-red directory which is where they should be.

11

Say for example the flows are on a volume mounted on a host set to a different time zone to that of the running container

1 Like
12

With respect, docker instances are magical and voodoo under the hood. Virtualization of all types, OS level such as KVM or VMware, or even application isolation, such as docker, have to do some evil things, to do what they do. When you have explicit hardware control in scope, this is where it can get nasty. VMware for example, after 20 years of improvement, still finds odd issues where abstracting physical hardware to a virtual hardware layer is problematic. I know this from experience, I digress, but PCIe bus issues drove me nuts in my past working with KVM and VMware OS isolation virtualization.

You might have an issue where the GPIO control, hardware driver, can't handle or is not flexible enough to understand the 'docker' environment. GPIO design was not originally, expected to be a shared resource across multiple processes concurrently once pin assignment is owned. If process A own pin 26, process B was limited as to what it could do with pin 26, type of thing. Various implementations of the GPIO control have abstracted the GPIO pin access, to offset this to a great degree, but there still are some issues with GPIO pins being shared across processes.

I actually wrote my own python library for GPIO control that ensured that pin ownership protected, when my various parallel scripts might step on each other. If I reserved a GPIO pin, other scripts were limited to what they could do.

Given the nature of docker, this could be at the root of your scenario. Docker was designed to support non-hardware-intrusive application isolation, not drive shared device control with in a docker instance. For example, docker can read and write to storage that is 'published' to the docker instance. But docker does not, and should not, have direct control to or of the disk drive control subsystem. For example, I would never try to run parted, or gparted in docker, on storage, that is shared across core OS and docker instances. Asking docker to drive GPIO pins, might be breaking the abstraction layers below your docker instance.

2 Likes
13

Thank you for the replies. @Nodi.Rubrum - I can appreciate the scenarios you describe but given the fact that the GPIO control works fine until the machine reboots and then only thing that is weird here is a Node Red message seemingly unconnected to reality leads me away from complex low-level sharing issues. I have used Docker a lot, and many flavours of VMWare, and can only agree about the low-level abstraction issues but hardware level PCIe abstraction is different from a handful of GPIO pins.

Given that when the containers I am using are started they work perfectly every time this seems like a problem that can be solved.

14

Ah missed the fact you said it was only at boot time... When it boots - when exactly does the clock get set correctly ? I have certainly seen that sometimes the ntp doesn't set the clock right away - most obvious when I try to do an apt upgrade on boot and the upgrade fails as certs are invalid time... once ntp kicks in all is good... So as I suggested above the same may be happening here - if the container thinks the flow is a different time that it was then it may well look to be "newer" ie updated... etc etc

15

I have a crazy idea...

If you run ntpdate from host, what do you get? If you run same from within the docker container what do you get? If you run the above again say after 5 minutes and then 15 minutes after that for 1st run now, 2nd run +5 and 3rd run +15, are you seeing any drift? I once had a virtual machine that would not start to drift until about 3 or 4 hours since last boot, but after a few days, would be more than 10 minutes out of sync, and then all kinds of things would start failing.

Most security (time bound protocols, say like Kerberos) only allow for a delta of a few minutes if that. Same for Microsoft AD authentications, less than 4 minutes if memory serves.

Now do the above after you have exercised your code several times, i.e. interfacing with the GPIO pins. I wonder if there is something odd causing the fake hardware clock to drift, is it just a time emulation, so anything that causes the docker instance to 'pause' could push the fake hw clock to drift from its perspective, I know this sounds nuts... but... stranger things. Given the issue only happens when the GPIO pins are leveraged, right?

16

So, I spent a bit of time and did some digging and guess what? Having ruled out a lot of crazy & complex ideas (thanks @Nodi.Rubrum for a few more :wink: ) I came back to the simple things in the getting started guide.

Key here is the use of hostname in the flows_{hostname}.json. It turns out that Balena changes the hostname with every deployment of a new Docker image, hence the need to specify the name of the flows.json file. Simple as that.

Thank you all for your help, we can consider this one solved.

1 Like
17

I didn't realise that ! - Useful tip.

18

That would have driven me nuts... the flows file changing per docker instance. Nuts. I set a common static name for the flows file, in settings.js file, so I can replicate the same image across multiple devices.

19

Indeed I thought we did fix it in the docker node-red container

20

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.