What you mention there is the possibility to use a docker-compose file, but I don't think you can add a group to node-red via a docker-compose file.
You need to create a simple image that sets the permission.
So you need to create a Dockerfile with following contents on the host machine where portainer is running.
RUN addgroup node-red dialout
You can a build an docker image based on the above Dockerfile using below command (should be run in the same folder where you have created the above Dockerfile)
[root@nuc-jan test]# docker build -t node-red-with-dialout-permissions .
Sending build context to Docker daemon 2.048kB
Step 1/4 : FROM nodered/node-red:1.2.9-12
Step 2/4 : USER root
---> Using cache
Step 3/4 : RUN addgroup node-red dialout
---> Running in 15b6a066db67
Removing intermediate container 15b6a066db67
Step 4/4 : USER node-red
---> Running in 407d4e0100a9
Removing intermediate container 407d4e0100a9
Successfully built 0e44d20e1653
Successfully tagged node-red-with-dialout-permissions:latest
Now you can deploy a container using image
node-red-with-dialout-permissions:latest in portainer as you used to do.
I have tested this and it works for me as you can see in screenshot below: