Running Docker Containers as Current Host User

Making local development less aggravating

Posted on Aug 5, 2018
Tags docker

ed: If you want to jump right to the solution, jump ahead to Ok so what actually works?.

Docker is an excellent tool for local web development. It allows creating non-trivial environments without polluting the local system with tools.

There are still some things that make working with it just a tad bit harder than necessary.

Today’s topic involves running Docker containers using the local host system’s current logged-in user.

The Problem

You are working on a project that requires Node NPM, PHP Composer or a similar tool that downloads or compiles outside dependencies or assets for you.

You do not want to install the language (PHP, Node) locally to run this tool, so you choose to run a Docker container. Here is how it would look like to spin up a container with PHP Composer support:

docker container run --rm \
    -v ${PWD}:/var/www \
    -w /var/www \
    jtreminio/php:7.2 composer require psr/log

This starts a temporary container using my PHP 7.2 image that has Composer pre-installed and runs composer require psr/log.

It generates the following:

$ ls -la
total 20K
drwxrwxr-x. 3 jtreminio jtreminio 4.0K Aug  4 19:34 ./
drwxr-xr-x. 7 jtreminio jtreminio 4.0K Aug  4 19:31 ../
drwxr-xr-x. 4 root      root      4.0K Aug  4 19:34 vendor/
-rw-r--r--. 1 root      root        53 Aug  4 19:34 composer.json
-rw-r--r--. 1 root      root      2.1K Aug  4 19:34 composer.lock

Although it works as intended, the problem is that the directories and files generated by Composer are root-owned.

Since I do not use root, attempting to do anything other than read the files is blocked:

$ echo '' > composer.json 
bash: composer.json: Permission denied

You have to sudo to edit or remove everything. This quickly gets old when your project uses Composer, Webpack, Yarn, etc because your system becomes littered with root-owned files and directories.

The Reason

Docker on Linux runs as a daemon. The official installation instructions recommend installing as root and selectively adding users to the docker group so they can run all Docker commands.

$ ps -fe | grep dockerd
255:root      2356     1  0 Aug03 ?        00:04:06 /usr/bin/dockerd

When you create a new container it does not get created as your current user, but as root, which the daemon is running under.

We can verify that the container runs as root with user/group ID 0:0:

$ docker container run --rm \
    -v ${PWD}:/var/www \
    -w /var/www \
    jtreminio/php:7.2 whoami
root
$ docker container run --rm \
    -v ${PWD}:/var/www \
    -w /var/www \
    jtreminio/php:7.2 \
        bash -c "echo \$(id -u \${USER}):\$(id -g \${USER})"
0:0

So run as your local user, right?

When you run Docker containers you can specify a user ID, plus a group ID. It is easy enough to do:

docker container run --rm \
    -v ${PWD}:/var/www \
    -w /var/www \
    -u $(id -u ${USER}):$(id -g ${USER}) \
    jtreminio/php:7.2 composer require psr/log

This generates the following:

$ ls -la
total 20K
drwxrwxr-x. 3 jtreminio jtreminio 4.0K Aug  4 20:09 ./
drwxr-xr-x. 7 jtreminio jtreminio 4.0K Aug  4 19:31 ../
drwxr-xr-x. 4 jtreminio jtreminio 4.0K Aug  4 20:09 vendor/
-rw-r--r--. 1 jtreminio jtreminio   53 Aug  4 20:09 composer.json
-rw-r--r--. 1 jtreminio jtreminio 2.1K Aug  4 20:09 composer.lock

In my system, my user jtreminio has user ID 1000 and group ID 1000, so the new line

-u $(id -u ${USER}):$(id -g ${USER})

gets interpreted as

-u 1000:1000

This does exactly what we want, but of course there is a catch: the container user is no longer root, or whatever the author decided to use. On Composer and NPM this simply means any internal cache directories cannot be written to since they are root-owned, but that really is not much of a problem because we are tearing the containers down as soon as they finish running what we told them to.

The Composer container above is deleted as soon as composer require psr/log finishes executing.

What if you want to run a web app with PHP-FPM? It must be able to create its PID file at /var/run/php-fpm.pid, if you are using file-based sessions it must write to /var/lib/php/sessions. Any number of things that require root or a predefined user will no longer work because the container is running using your user/group ID:

$ docker container run --rm \
    -v ${PWD}:/var/www \
    -w /var/www \
    -u $(id -u ${USER}):$(id -g ${USER}) \
    jtreminio/php:7.2 whoami
whoami: cannot find name for user ID 1000
$ docker container run --rm \
    -v ${PWD}:/var/www \
    -w /var/www \
    -u $(id -u ${USER}):$(id -g ${USER}) \
    jtreminio/php:7.2 \
        bash -c "echo \$(id -u \${USER}):\$(id -g \${USER})"
1000:1000

If you try to do anything that requires elevated permissions or a specific user, you will be denied:

$ docker container run --rm \
    -v ${PWD}:/var/www \
    -w /var/www \
    -u $(id -u ${USER}):$(id -g ${USER}) \
    jtreminio/php:7.2 ls -la /var/lib/php/sessions
total 8
drwxr-xr-x. 1 www-data www-data 4096 Jul  9 12:35 .
drwxr-xr-x. 1 root     root     4096 Jul 26 09:08 ..
$ docker container run --rm \
    -v ${PWD}:/var/www \
    -w /var/www \
    -u $(id -u ${USER}):$(id -g ${USER}) \
    jtreminio/php:7.2 touch /var/lib/php/sessions/foo
touch: cannot touch '/var/lib/php/sessions/foo': Permission denied

The above means running the PHP-FPM daemon as your local user will quickly encounter permissions issues. In this case it is because /var/lib/php/sessions is owned by www-data:www-data which most likely does not share your local user’s IDs:

$ docker container run --rm \
    -v ${PWD}:/var/www \
    -w /var/www \
    jtreminio/php:7.2 \
        bash -c "echo \$(id -u www-data):\$(id -g www-data)"
33:33

OK, run it as an non-root internal user?

So far we have found that

  • root works great inside the container but is annoying to work with on the host system, and
  • your local user works great on your host system, but will quickly run into permission problems inside the container.

What if we try to run the containers as a non-root user that has required permissions to write inside the container directories?

Try with the container’s internal www-data user:

$ docker container run --rm \
    -v ${PWD}:/var/www \
    -w /var/www \
    -u www-data \
    jtreminio/php:7.2 \
        bash -c "touch /var/lib/php/sessions/foo && echo \$?"
0

As expected this worked fine, but if you try running Composer:

$ docker container run --rm \
    -v ${PWD}:/var/www \
    -w /var/www \
    -u www-data \
    jtreminio/php:7.2 composer require psr/log

[ErrorException]                                                              
file_put_contents(./composer.json): failed to open stream: Permission denied  

The problem is that the internal container user www-data with user/group ID 33:33, does not have write permissions to my host’s current directory:

$ ls -la
total 8.0K
drwxrwxr-x. 2 jtreminio jtreminio 4.0K Aug  4 20:34 ./
drwxr-xr-x. 7 jtreminio jtreminio 4.0K Aug  4 19:31 ../

We might as well add another grievance to our list:

  • non-root internal user with required permissions works great inside the container but completely falters on host system with volumes.

Why is this happening?

Host and containers do not share users and groups.

On my local system, there is no www-data user:

$ groups www-data
groups: ‘www-data’: no such user

Similarly, if I try using my direct username to run the container I see errors:

$ docker container run --rm \
    -v ${PWD}:/var/www \
    -w /var/www \
    -u jtreminio \
    jtreminio/php:7.2 whoami
docker: Error response from daemon: linux spec user: unable to find user jtreminio: no matching entries in passwd file.

As far as the container is aware, it is its own separate system with its own list of users at /etc/passwd and list of groups at /etc/group. It has no idea about any users or groups that exist on the host system. Likewise, the host does not know about users in the container. By convention root is 0:0 which will match any Linux system, but anything else is up to each distro and each image author.

Try sharing /etc/passwd!

Now we know that the

  • host system does not know about container’s /etc/passwd, and
  • container does not know about the host system’s /etc/passwd

What happens if we bind the host’s /etc/passwd to the container’s?

Not much as you would think, actually. My host system still does not have a user matching 33:33, so even if I bind the /etc/passwd into the container, attempting to write to www-data-owned directories still will not work.

This is because Linux permissions are not name-based, but ID-based.

For example, on my local system:

$ ls -lan
total 8.0K
drwxrwxr-x. 2 1000 1000 4.0K Aug  4 20:34 ./
drwxr-xr-x. 7 1000 1000 4.0K Aug  4 19:31 ../

and in the container:

$ docker container run --rm \
    -v ${PWD}:/var/www \
    -w /var/www \
    -u $(id -u ${USER}):$(id -g ${USER}) \
    jtreminio/php:7.2 ls -lan /var/lib/php/sessions
total 8
drwxr-xr-x. 1 33 33 4096 Jul  9 12:35 .
drwxr-xr-x. 1  0  0 4096 Jul 26 09:08 ..

In the container www-data:www-data is simply an alias to 33:33.

User namespaces

The container does not care what user and group name is used, it simply wants the IDs to match.

There is a concept in Docker Engine called User Namespaces. Here is a great introduction to them.

The concept boils down to mapping the internal container user/group IDs to reflect different values.

For example, you can tell Docker to use your current user/group ID as the “floor” for container IDs. In my example, my jtreminio account with 1000:1000 would map directly to 0:0 in a container.

In other words, we tell Docker to consider our current user on the host as root in containers! Local system user ID 1000 maps directly to container user ID 0.

User namespaces would allow us to run all containers as root internally which would completely eliminate any permission issues, and any generated files and directories on shared volumes would be owned by the host user/group so we would no longer need to sudo to edit or delete them.

This sounds exactly what we need, right? There is a catch (because of course there is)… in fact, multiple catches!

You are running the container as root

Normally this would be a security concern. Countless articles and guides strongly recommend not running as root in containers.

In this case, this is not the problem, because the container’s internal root maps to our current, non-root user on host.

The problem actually comes down to root being able to do anything and everything in the container, which opens the door to unforeseen permissions issues when you deploy to production, where you would not be using user namespaces.

In production, most containers will run as non-root, with permissions specifically tailored for that user/group ID. Attempting to do anything the user is not permitted will rightfully result in permissions denied errors. This is a safety feature you want, and running as root completely ignores this.

You might run as root on your local system and everything works perfectly fine. You are happy, files are generated as expected, all actions are permitted and you are a happy developer. Once you deploy to production you may suddenly realize that you were developing on a system very much unlike production. Something that worked flawlessly on your development system can easily break on production due to insufficient permissions for whatever user is running the container service.

Your user/group ID is the floor, it goes up!

Your user/group ID maps directly to 0:0 in the container, but there are more accounts than just the one root and none of them are mapped to your host user.

This means that while 0:0 maps to my host system’s 1000:1000, the container’s www-data user with IDs 33:33 would map to my host system’s 1033:1033, which does not exist.

Anything done as non-root in the container will run against the same issues we saw earlier: what might be considered sufficient permissions inside the container will probably not work the same on your host.

My host’s directories are still owned by 1000:1000 and a user with 1033:1033 will be denied.

All containers on your system are affected

If the previous two reasons are not enough, the fact that Docker does not come with User Namespaces support enabled by default, to enable it requires editing a JSON config file and restarting the Docker daemon, and now all your existing and future containers will implement namespaces whether you want them to or not should be enough to make you question if it is worth the hassle.

Ok so what actually works?

The only sure-fire way of resolving all previous issues and getting on with Making the World a Better Place is completely replacing the internal user/group IDs with known, good values.

In the container, www-data maps to 33:33. Why not change it so it maps to my host system’s jtreminio with 1000:1000?

In other words, make the container’s www-data user/group ID be 1000:1000.

You cannot change an existing account’s user/group IDs. Remember, the user and group names are simply aliases to the IDs. You can rename www-data to anything you want but the IDs will not change.

The only thing you can do is delete and recreate the user/group completely.

For this we will need to create a Dockerfile. We need to recreate the user before any potential entrypoint scripts are run. Unknown issues can and will occur if you attempt to delete a user/group while it is in use by another process.

First, delete the user and group:

FROM jtreminio/php:7.2

RUN userdel -f www-data &&\
    if getent group www-data ; then groupdel www-data; fi

We test if the group actually exists before trying to delete, to avoid possible errors when the group does not exist.

Next, we can generate the user and group with our IDs:

FROM jtreminio/php:7.2

ARG USER_ID=1000
ARG GROUP_ID=1000

RUN userdel -f www-data &&\
    if getent group www-data ; then groupdel www-data; fi &&\
    groupadd -g ${GROUP_ID} www-data &&\
    useradd -l -u ${USER_ID} -g www-data www-data &&\
    install -d -m 0755 -o www-data -g www-data /home/www-data

After we delete the user and group, we recreate it with the defined values.

1000 is my user and group ID, and now the container’s www-data has the same.

Remember to use the -l flag with useradd! There is an amazingly fun issue where a high UID value will generate huge log files and freeze your system! Click here for all the juicy details.

We also take the time to generate a home directory for our new user. This makes it much easier to have your container perform SSH actions using your host’s SSH keys as long as you bind a volume.

Even though we are done with recreating the user and group, there is a problem: Any files and directories that were owned by the www-data user will not automatically point to the new ID values. Since the names are just labels, /var/lib/php/sessions is still owned by 33:33, not matching our spiffy new 1000:1000!

This is the part where you need to know just enough about the container you want to run that you can pinpoint which files and directories you need to update the owner for.

Thankfully, the rallying cry of containers is to have them do one single thing, which limits the number of places we need to update. In this image, it is not very many at all:

FROM jtreminio/php:7.2

ARG USER_ID=1000
ARG GROUP_ID=1000

RUN userdel -f www-data &&\
    if getent group www-data ; then groupdel www-data; fi &&\
    groupadd -g ${GROUP_ID} www-data &&\
    useradd -l -u ${USER_ID} -g www-data www-data &&\
    install -d -m 0755 -o www-data -g www-data /home/www-data &&\
    chown --changes --silent --no-dereference --recursive \
          --from=33:33 ${USER_ID}:${GROUP_ID} \
        /home/www-data \
        /.composer \
        /var/run/php-fpm \
        /var/lib/php/sessions

You can figure out what you need by looking at the Dockerfile used to generate the image. For the jtreminio/php:7.2 image we can see that /.composer, /var/run/php-fpm and /var/lib/php/sessions directories need updated.

Finally, it does not hurt to be explicit about the user we want to run the container:

FROM jtreminio/php:7.2

ARG USER_ID=1000
ARG GROUP_ID=1000

RUN userdel -f www-data &&\
    if getent group www-data ; then groupdel www-data; fi &&\
    groupadd -g ${GROUP_ID} www-data &&\
    useradd -l -u ${USER_ID} -g www-data www-data &&\
    install -d -m 0755 -o www-data -g www-data /home/www-data &&\
    chown --changes --silent --no-dereference --recursive \
          --from=33:33 ${USER_ID}:${GROUP_ID} \
        /home/www-data \
        /.composer \
        /var/run/php-fpm \
        /var/lib/php/sessions
        
USER www-data

If you run the container now, you will automatically become the www-data user which now has user/group ID 1000:1000. Any files generated by this container will be owned by my host system’s user.

What should we do about the hard-coded IDs, though? Your coworkers may have different IDs on their machines.

Make it dynamic

You can pass arguments when building an image using --build-arg.

Update the Dockerfile first:

FROM jtreminio/php:7.2

ARG USER_ID
ARG GROUP_ID

RUN if [ ${USER_ID:-0} -ne 0 ] && [ ${GROUP_ID:-0} -ne 0 ]; then \
    userdel -f www-data &&\
    if getent group www-data ; then groupdel www-data; fi &&\
    groupadd -g ${GROUP_ID} www-data &&\
    useradd -l -u ${USER_ID} -g www-data www-data &&\
    install -d -m 0755 -o www-data -g www-data /home/www-data &&\
    chown --changes --silent --no-dereference --recursive \
          --from=33:33 ${USER_ID}:${GROUP_ID} \
        /home/www-data \
        /.composer \
        /var/run/php-fpm \
        /var/lib/php/sessions \
;fi
        
USER www-data

We make both USER_ID and GROUP_ID optional. If both are not defined, we skip the whole user delete/recreate process completely. This Dockerfile can now be used for both dev and production purposes.

To build the above image you would do:

$ docker image build \
    --build-arg USER_ID=$(id -u ${USER}) \
    --build-arg GROUP_ID=$(id -g ${USER}) \
    -t php_test \
    .

Sending build context to Docker daemon   68.1kB
Step 1/5 : FROM jtreminio/php:7.2
 ---> 88efaa8cad72
Step 2/5 : ARG USER_ID
 ---> Running in 51d424a6d618
Removing intermediate container 51d424a6d618
 ---> 3feace11551d
Step 3/5 : ARG GROUP_ID
 ---> Running in aaaceff5ee4e
Removing intermediate container aaaceff5ee4e
 ---> 9f110b0f92c2
Step 4/5 : RUN if [ ${USER_ID:-0} -ne 0 ] && [ ${GROUP_ID:-0} -ne 0 ]; then     userdel -f www-data &&    if getent group www-data ; then groupdel www-data; fi &&    groupadd -g ${GROUP_ID} www-data &&    useradd -l -u ${USER_ID} -g www-data www-data &&    install -d -m 0755 -o www-data -g www-data /home/www-data &&    chown --changes --silent --no-dereference --recursive           --from=33:33 ${USER_ID}:${GROUP_ID}         /home/www-data         /.composer         /var/run/php-fpm         /var/lib/php/sessions ;fi
 ---> Running in b516a3ab4600
changed ownership of '/.composer/keys.tags.pub' from 33:33 to 1000:1000
changed ownership of '/.composer/keys.dev.pub' from 33:33 to 1000:1000
changed ownership of '/.composer' from 33:33 to 1000:1000
changed ownership of '/var/run/php-fpm' from 33:33 to 1000:1000
changed ownership of '/var/lib/php/sessions' from 33:33 to 1000:1000
Removing intermediate container b516a3ab4600
 ---> c126c9f07252
Step 5/5 : USER www-data
 ---> Running in 76831280f239
Removing intermediate container 76831280f239
 ---> 5c237c8e46cd
Successfully built 5c237c8e46cd
Successfully tagged php_test:latest

Try running Composer with the new image:

$ docker container run --rm \
    -v ${PWD}:/var/www \
    -w /var/www \
    php_test:latest composer require psr/log

and check out the result:

$ ls -lan
total 24K
drwxrwxr-x. 3 1000 1000 4.0K Aug  4 22:50 ./
drwxr-xr-x. 7 1000 1000 4.0K Aug  4 19:31 ../
drwxr-xr-x. 4 1000 1000 4.0K Aug  4 22:50 vendor/
-rw-rw-r--. 1 1000 1000  545 Aug  4 22:48 Dockerfile
-rw-r--r--. 1 1000 1000   53 Aug  4 22:50 composer.json
-rw-r--r--. 1 1000 1000 2.1K Aug  4 22:50 composer.lock

Try creating a file in a protected directory:

$ docker container run --rm \
    -v ${PWD}:/var/www \
    -w /var/www \
    php_test:latest \
        bash -c "touch /var/lib/php/sessions/foo && echo \$?"
0

User Docker Compose

You can take the docker image build command and save it to a bash file for easy execution.

However, most folks would rather use Docker Compose for local development.

Here is what your docker-compose.yml might look like:

version: '3.2'
services:
  php:
    build:
      context: .
      dockerfile: Dockerfile
      args:
        USER_ID: $(id -u ${USER})
        GROUP_ID: $(id -g ${USER})
    volumes:
      - ${HOME}/.composer:/.composer
      - ${PWD}:/var/www
    ports:
      - 9000:9000

Unfortunately, Docker Compose cannot parse commands, so $(id -u ${USER}) would not work in the YAML file. Thankfully we can use .env file to pass values:

# docker-compose.yml
version: '3.2'
services:
  php:
    build:
      context: .
      dockerfile: Dockerfile
      args:
        USER_ID: ${USER_ID:-0}
        GROUP_ID: ${GROUP_ID:-0}
    volumes:
      - ${HOME}/.composer:/.composer
      - ${PWD}:/var/www
    ports:
      - 9000:9000
; .env
USER_ID=1000
GROUP_ID=1000

With a single docker-compose up -d --build we are up and running:

$ docker-compose up -d --build
Creating network "temp_default" with the default driver
Building php
Step 1/5 : FROM jtreminio/php:7.2
 ---> 88efaa8cad72
Step 2/5 : ARG USER_ID
 ---> Running in 1b406c5797aa
Removing intermediate container 1b406c5797aa
 ---> c61d4e7b14b9
Step 3/5 : ARG GROUP_ID
 ---> Running in 14ac56de9dc8
Removing intermediate container 14ac56de9dc8
 ---> 576ada9d6dbe
Step 4/5 : RUN if [ ${USER_ID:-0} -ne 0 ] && [ ${GROUP_ID:-0} -ne 0 ]; then     userdel -f www-data &&    if getent group www-data ; then groupdel www-data; fi &&    groupadd -g ${GROUP_ID} www-data &&    useradd -l -u ${USER_ID} -g www-data www-data &&    install -d -m 0755 -o www-data -g www-data /home/www-data &&    chown --changes --silent --no-dereference --recursive           --from=33:33 ${USER_ID}:${GROUP_ID}         /home/www-data         /.composer         /var/run/php-fpm         /var/lib/php/sessions ;fi
 ---> Running in 632e67cdc854
changed ownership of '/.composer/keys.tags.pub' from 33:33 to 1000:1000
changed ownership of '/.composer/keys.dev.pub' from 33:33 to 1000:1000
changed ownership of '/.composer' from 33:33 to 1000:1000
changed ownership of '/var/run/php-fpm' from 33:33 to 1000:1000
changed ownership of '/var/lib/php/sessions' from 33:33 to 1000:1000
Removing intermediate container 632e67cdc854
 ---> 1bb2e0bbff3c
Step 5/5 : USER www-data
 ---> Running in 8efe4779c454
Removing intermediate container 8efe4779c454
 ---> b3bd40096b94
Successfully built b3bd40096b94
Successfully tagged temp_php:latest
Creating temp_php_1 ... done

We now have a PHP-FPM container running with my system user/group ID 1000:1000 waiting for connections from Nginx or Apache.

What containers need this?

Not all containers actually require these steps. For simpler images like the official Nginx or Apache that do not generate files you would be interested in editing while in development, you can skip this and let the container run as the normal internal user.

For containers that only fetch dependencies to local disk, like PHP Composer or Node NPM, you can get by with simply passing your user/group ID directly. The container will complain about a user not existing for those values, but will otherwise work fine. Cache directories may need a little bit of work.

Containers with long-running daemons that generate a large amount of files, like parsing SCSS to CSS, will benefit you greatly to implement this concept.

MacOS and Windows users…

You do not actually need to do any of this. Like, this whole article, you can skip it. Docker does not actually run natively on either of these operating systems, but transparently in a virtual machine.

This virtual machine is run under whatever user installed Docker, which is usually not root on MacOS or administrator on Windows.

While you do not need this, it is still a good idea to implement it. Your coworkers that run Linux as their main OS will benefit greatly, and if you ever make the switch you will already have your Docker stuff up to date.

You will also only require a single Dockerfile between all your coworkers.

Wrapping it up

While you can run all containers as the default internal user, dedicating just a few minutes per image to switch to your user/group ID will help eliminate one fairly big point of frustration.

Overall Docker is an amazing technology but little things like this can mar an otherwise enjoyable experience. Knowing how to smooth the bumps will help make Docker’s promise of Nirvana come true.

Until next time, this is Señor PHP Developer Juan Treminio wishing you adios!