Building Docker Images Smaller, Rootless and Non-Shell for Kubernetes

After building a Docker image faster, I wanted to build it for the K8s cluster. Running the container on the local machine isn’t the same as running it on a cluster. I’m packaging a Go application in my example. But the same principles apply to any other language.

Starting Dockerfile

I’m starting with the following Dockerfile(Dockerfile_1):

ARG GO_VERSION=1.20.3
FROM golang:${GO_VERSION}-buster as builder

WORKDIR /app

COPY go.mod go.sum /app/
RUN go mod download -x

COPY . /app/
RUN go build -o app

FROM debian:buster

WORKDIR /app

COPY --from=builder /app/app /app/

ENTRYPOINT [ "/app/app" ]

And I build it with the following command:

docker buildx build -t rnemet/echo:0.0.1 . -f Dockerfile_1 --cache-to type=registry,ref=rnemet/echo:test --cache-from type=registry,ref=rnemet/echo:test --cache-from type=registry,ref=rnemet/echo:main --load

I’m using the --cache-to and --cache-from flags to cache the build process and the --load to load the image into the local Docker daemon. Also, I’m using BuildKit to build an image.

Let’s see what I got:

❯ docker images rnemet/echo
REPOSITORY    TAG       IMAGE ID       CREATED         SIZE
rnemet/echo   0.0.1     6fc43a3d85eb   4 minutes ago   133MB

If I run the container as a Pod in a K8s cluster or a Docker container on my local machine:

❯ docker exec -it  echo bash
root@30fa7aa78401:/app# whoami
root

I managed:

• to open a shell in the container
• found out that I’m running as root
• I run the whoami command, which is not a command I need when I run the container

The whoami is not the problem, but the fact that it is present. I don’t need it, and it’s not part of my application. There must be others as well. I need to clean up the image.

Then I learned that my app is run as root. This isn’t good. My app needs to run as a non-root user. Why? There are many reasons. For example, the app can access the whole system with root privileges. Do you remember? The container runtime isolates the process from the host system using namespaces. But the process can still access the host system.

As the cherry on top, I can open a shell in the container and run commands as root. I do not even want to discuss why this is not good.

Smaller, Rootless, and Non-Shell Image

The first thing I have to do is to change the target image. I’m using debian:buster as a target image. That needs to be changed. I decided to use scratch as a target image. It’s a minimal image. It’s not a Linux distribution. It’s not a shell. It’s not a root user. It’s a blank slate. So my next Dockerfile(Dockerfile_2):

ARG GO_VERSION=1.20.3
FROM golang:${GO_VERSION}-buster as builder

WORKDIR /app

COPY go.mod go.sum /app/
RUN go mod download -x

COPY . /app/
RUN go build -o app

FROM scratch

WORKDIR /app

COPY --from=builder /app/app /app/

ENTRYPOINT [ "/app/app" ]

And build it:

docker buildx build -t rnemet/echo:0.0.2 . -f Dockerfile_2 --cache-to type=registry,ref=rnemet/echo:test --cache-from type=registry,ref=rnemet/echo:test --cache-from type=registry,ref=rnemet/echo:main --load

And the result:

❯ docker images rnemet/echo
REPOSITORY    TAG       IMAGE ID       CREATED          SIZE
rnemet/echo   0.0.2     188f310d88e2   42 seconds ago   19.1MB
rnemet/echo   0.0.1     6fc43a3d85eb   59 minutes ago   133MB

Wow, the image is smaller. But it fails when I try to run it as a Pod in a K8s cluster or a Docker container on my local machine. Argh…

The problem is how I build the application. I’m building it on a particular Linux image with Go installed. That means all the libs and tools that Go needs to build and run the application are part of the image. So, I must build the app with all its dependencies packed together. When it comes to Go, it’s easy(Dockerfile_3):

...
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o app
...

I’m using the CGO_ENABLED=0 flag to disable using Cgo. I’m using the -ldflags="-s -w" flag to strip the debug information from the binary. And results are:

❯ docker images rnemet/echo
REPOSITORY    TAG       IMAGE ID       CREATED             SIZE
rnemet/echo   0.0.3     81e4098f0e76   15 seconds ago      13MB
rnemet/echo   0.0.2     188f310d88e2   25 minutes ago      19.1MB
rnemet/echo   0.0.1     6fc43a3d85eb   About an hour ago   133MB

The image is smaller, and it works. But I’m still running it as root. I need to change that, too. But when working with scratch image, I need to create the user in the builder image and then copy the /etc/password file to the scratch image(Dockerfile_3):

ARG GO_VERSION=1.20.3
FROM golang:${GO_VERSION}-buster as builder

WORKDIR /app

COPY go.mod go.sum /app/
RUN go mod download -x

COPY . /app/
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o app

RUN useradd -u 10001 appuser

FROM scratch

WORKDIR /app
COPY --from=builder /etc/passwd /etc/passwd

USER appuser
COPY --chown=appuser:appuser --from=builder /app/app /app/

ENTRYPOINT [ "/app/app" ]

I’m running as a non-root user with no shell, and the image is smaller. And I should be happy, but no. Why? Let’s go a few steps back.

Why Is Image Smaller?

In this case, the image is smaller because I’m using a scratch image as a target image, which produces an image size of around 19MB. Plus, I’m using stripping debugging information from the binary. That’s why the image is so small. That is nice, right?

Cgo Or No Cgo?

I’m using the CGO_ENABLED=0 flag to disable using Cgo. Why? Because I’m using a scratch image as a target image.

If I use debian:buster as a target image, I don’t need to disable Cgo. Why? Because debian:buster image has all the libs and tools that Go needs to build and run the application. So I don’t need to pack all the libs and tools with the application. I can use the libs and tools that are part of the image.

The cgo tool is a tool that allows Go programs to call C code. It enables your Go app to call OS’s native libraries dynamically. Using it leads to smaller and faster builds. But, as you can see, it is not always possible to use it. So, in this case, I have to disable it and create a static binary. The static binary will be bigger but will have all the libs the app needs to run. It will be more portable. It is common to disable cgo when creating multi-architecture images.

Unless your app depends on C libraries, you can generally disable cgo. Then you have to leave cgo enabled. It’s always some trade-off.

Do I Always Need To Scratch When It Itches?

While the scratch image is minimal, it uses the root user, and I can not use the USER instruction to change it. I had to do a workaround. See Dockerfile_3. I had to add the user to the builder image and then copy /etc/password file to the scratch image. Not a big deal, but it’s a workaround. And I’m not too fond of workarounds. I wonder if it is the only workaround I will do when using the scratch image.

Small detour. You need a base image that you can trust and use easily. When choosing a base image, you should trust the creator of the base image, and you don’t want to make any changes or workarounds. In most cases, you’ll see Alpine Linux because it is small, and people trust it. But it has its own problems. My personal choice is Distroless images. They are small, secure, and easy to use. It is not as small as a scratch image, but still small. And they are secure. They are based on Debian, and they use non-root user. So you don’t need to do any workarounds.

Let’s check it out. I’m using distroless/static-debian11:nonroot as a target image(Dockerfile_4):

ARG GO_VERSION=1.20.3
FROM golang:${GO_VERSION}-buster as builder

WORKDIR /app

COPY go.mod go.sum /app/
RUN go mod download -x

COPY . /app/
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o app

FROM gcr.io/distroless/static-debian11:nonroot

WORKDIR /app

COPY --from=builder /app/app /app/

ENTRYPOINT [ "/app/app" ]

And results are:

❯ docker images rnemet/echo
REPOSITORY    TAG       IMAGE ID       CREATED              SIZE
rnemet/echo   0.0.4     f04110959f3c   About a minute ago   15.5MB
rnemet/echo   0.0.3     0280e6cfb546   5 hours ago          13MB
rnemet/echo   0.0.2     188f310d88e2   6 hours ago          19.1MB
rnemet/echo   0.0.1     6fc43a3d85eb   7 hours ago          133MB

Using distroless/static-debian11:nonroot as a target image, I’m getting an image size of around 15.5MB. And I’m running as a non-root user. I do not have any workarounds. As well, I’m having USER instruction. My Dockerfile is clean and easy to read. I like it.

One last try to use cgo in Dockerfile_5:

ARG GO_VERSION=1.20.3
FROM golang:${GO_VERSION}-buster as builder

WORKDIR /app

COPY go.mod go.sum /app/
RUN go mod download -x

COPY . /app/
RUN go build -ldflags="-s -w" -o app

FROM gcr.io/distroless/base-debian11:nonroot

WORKDIR /app

COPY --from=builder /app/app /app/

ENTRYPOINT [ "/app/app" ]

After building it, the results are:

❯ docker images rnemet/echo
REPOSITORY    TAG       IMAGE ID       CREATED          SIZE
rnemet/echo   0.0.5     030cf87dd36e   3 minutes ago    33.5MB
rnemet/echo   0.0.4     f04110959f3c   15 minutes ago   15.5MB
rnemet/echo   0.0.3     0280e6cfb546   5 hours ago      13MB
rnemet/echo   0.0.2     188f310d88e2   6 hours ago      19.1MB
rnemet/echo   0.0.1     6fc43a3d85eb   7 hours ago      133MB

So, the smallest image is when we use a scratch image as a target image. But we have to do some workarounds. The second smallest image is when we use distroless/static-debian11:nonroot as a target image. And we don’t have to do any workarounds.

When you compare Distrolees images with a scratch image, you can see that the second is smaller. But, I trust Google and do not have to do any workarounds if I use Distrolees images. In the case of the Go app, I had only one workaround. If I need to pack another app based on Java or NodeJS, I could do more work using a scratch image. Distrolees images already come prepackaged with some tools. So I don’t have to do any workarounds.

Base Image Conclusion

Let’s summarize the results:

In the end, I got what I wanted, a fast build, a small image, a non-root user, no shell access, and no workarounds. I’m happy with the results.

I’m not telling you to use Distroless images without any doubts. Maybe for your case, Alpine Linux](https://hub.docker.com/_/alpine) is better as it is tiny. Or you prefer Wolfi. Or start from scratch. As you can see, there will be some trade-offs. You need to consider your apps needs, the effort willing to put into your Dockerfile and your trust in the creator of the base image.

Ah, one more thing please try to avoid the latest tags. Use versioned base images whenever possible. You’ll have much more control and less headache.

Nice to Have: Labels

Add some metadata to your images. They can be handy later on. Like:

LABEL version="0.0.1-beta"
LABEL vendor="rnemet.dev"
LABEL release-date="2024-02-12"

You could also use ARG to set labels. Like:

ARG VERSION=0.0.1-beta
ARG VENDOR=rnemet.dev
ARG RELEASE_DATE=2024-02-12
LABEL version="0.0.1-beta"
LABEL vendor="rnemet.dev"
LABEL release-date="2024-02-12"

Conclusion

I hope you enjoyed this post and learned something new. If you found this helpful post, please share it with your friends. You can subscribe to my newsletter at the bottom of this post and/or share this post. I would appreciate it.

References

• BuildKit
• Linux namespaces
• Docker user namespaces
• Scratch image
• Building image from scratch
• Distroless images
• Wolfi
• Alpine Linux