What you mention there is the possibility to use a docker-compose file, but I don't think you can add a group to node-red via a docker-compose file.
You need to create a simple image that sets the permission.
So you need to create a Dockerfile with following contents on the host machine where portainer is running.
FROM nodered/node-red
USER root
RUN addgroup node-red dialout
USER node-red
You can a build an docker image based on the above Dockerfile using below command (should be run in the same folder where you have created the above Dockerfile)
[root@nuc-jan test]# docker build -t node-red-with-dialout-permissions .
Sending build context to Docker daemon 2.048kB
Step 1/4 : FROM nodered/node-red:1.2.9-12
---> 9772fc53c59f
Step 2/4 : USER root
---> Using cache
---> f51e7cd902b0
Step 3/4 : RUN addgroup node-red dialout
---> Running in 15b6a066db67
Removing intermediate container 15b6a066db67
---> dfcc57c65d21
Step 4/4 : USER node-red
---> Running in 407d4e0100a9
Removing intermediate container 407d4e0100a9
---> 0e44d20e1653
Successfully built 0e44d20e1653
Successfully tagged node-red-with-dialout-permissions:latest
[root@nuc-jan test]#
Now you can deploy a container using image node-red-with-dialout-permissions:latest in portainer as you used to do.
I have tested this and it works for me as you can see in screenshot below:
