Earthly, Podman And Docker Compose

Podman is a daemon-less container engine for developing, managing, and running OCI containers on your Linux System. With podman, containers can either be run as root or in rootless mode, which improves security as an attacker will not have root privileges over your system. It has a CLI that serves as a drop-in replacement for Docker to make migration easier, so most users can alias Docker to podman without any issues. You can find out more in the project’s documentation.

Podman recently had the fourth major release in February, which is one of their most significant releases ever, featuring over 60 new features, and a completely rewritten network stack. Being so fresh, most OSes don’t have podman 4 release yet, however you can already try it out on Arch Linux or one of its derivatives (e.g Manjaro); or build from source (however that’s admittedly a bit complicated given the number of dependencies). This article shows how to install podman 4 on Arch Linux for rootless and use docker-compose and Earthly.

Mục Lục

Install Core Components

Podman 4 has a couple of required dependencies. First of all, it needs a container networking solution. In previous versions, podman relied on the CNI, for the new version however, the team decided to write a dedicated networking stack from scratch, replacing CNI (which is still supported for compatibility). The new stack consists of two components: netavark, a container networking tool built specifically for podman; and aardvark-dns which provides DNS for the containers. The reasons behind this change is explained in RedHat’s blog:

Podman aims to deliver a dedicated single-node container management tool, and the CNI was created to serve Kubernetes, so it is inherently based on clusters. Podman requires new functionality, such as support for container names and aliases in Domain Name System (DNS) lookups, that’s not very useful to the CNI. Meanwhile, the CNI project is considering deprecating functionality that Podman relies on because it is not needed to support Kubernetes. Given the inherent tension between Podman’s goals and the CNI’s, our team evaluated the options and decided that our best course of action was to create Netavark and Aardvark and tailor them to the needs of Podman’s users. – Podman 4.0’s new network stack: What you need to know

For rootless podman we also need a way to connect our user-mode container networks to the Internet in an unprivileged way, which can be done with slirp4netns.

For rootless OverlayFS support, fuse-overlayfs is required. Unfortunately, as a user-space file system, it offers significantly lower performance than native OverlayFS, the team warns.

On Arch Linux, all required packages are hosted in the Community repository.

$ sudo
 pacman -S netavark aardvark-dns slirp4netns fuse-overlayfs podmanpacman -S netavark aardvark-dns slirp4netns fuse-overlayfs podman

You can verify your installation by starting a basic container. Note that rootless won’t work just yet, so we’re using sudo for the time being.

$ sudo
 podman run busybox echo "Hello World"
podman run busybox echo  Resolved
 "busybox"
 as an alias (/etc/containers/registries.conf.d/00-shortnames.conf)as an alias (/etc/containers/registries.conf.d/00-shortnames.conf)  Trying
 to pull docker.io/library/busybox:latest...to pull docker.io/library/busybox:latest...  Getting
 image source signaturesimage source signatures  Copying
 blob 50e8d59317eb done  blob 50e8d59317eb done  Copying
 config 1a80408de7 done  config 1a80408de7 done  Writing
 manifest to image destinationmanifest to image destination  Storing
 signaturessignatures  Hello
 WorldWorld

Installing on Other Distros

You can start by looking at podman’s installation instructions, however bear in mind that it’s very likely that the latest podman package for your (not rolling release) OS is for an older version (e.g. at the time of writing this article, the latest release for Ubuntu 22.04 is 3.4), so you will likely have to build all components from scratch. Also, note that the guide on the above page for building from source is for version 3 too, so it might not be 100% applicable.

System Configuration for Rootless

For rootless podman, unprivileged users must be able to create namespaces. Check the value of kernel.unprivileged_userns_clone by running:

$ sysctl
 kernel.unprivileged_userns_clonekernel.unprivileged_userns_clone

If it is currently set to 0, enable it by setting 1 via sysctl or kernel parameter.

Furthermore, subuid and subgid must be set for each user that wants to run rootless podman. /etc/subuid and /etc/subgid do not exist by default. If they do not exist yet in your system, create them and add the subuids and subgids with usermod.

$ sudo
 touch /etc/subuid /etc/subgidtouch /etc/subuid /etc/subgid$ sudo
 usermod --add-subuids 100000-165535 --add-subgids 100000-165535 $USER
usermod --add-subuids 100000-165535 --add-subgids 100000-165535

See more about Configuration on the Arch wiki.

Setting Up Image Registries

You might have noticed an interesting line in the output of podman run earlier:

Resolved
 "busybox"
 as an alias (/etc/containers/registries.conf.d/00-shortnames.conf)as an alias (/etc/containers/registries.conf.d/00-shortnames.conf)

podman discourages using unqualified image names, as it always entails an inherent risk that the image being pulled is spoofed. Instead, podman’s default distribution includes a predefined set of aliases in /etc/containers/registries.conf.d/00-shortnames.conf to resolve the fully qualified names of some widely used images. However, the files under /etc/containers is only considered when running podman as root. So when running rootless, you will receive the following error:

$ podman
 run busybox echo "Hello World"
run busybox echo  Error
: short-name "busybox"
 did not resolve to an alias and no unqualified-search registries are defined in "/home/user/.config/containers/registries.conf"
: short-namedid not resolve to an alias and no unqualified-search registries are defined in

This issue can be resolved by using the fully qualified image name, i.e docker.io/library/busybox. However, this would require modifying all existing scripts that already use short image names. Instead you can copy 00-shortnames.conf to your user config directory and assign the necessary permissions.

$ mkdir
 -p ${XDG_CONFIG_HOME}
/containers/registries.conf.d-p/containers/registries.conf.d$ sudo
 cp /etc/containers/registries.conf.d/00-shortnames.conf ${XDG_CONFIG_HOME}
/containers/registries.conf.d/00-shortnames.confcp /etc/containers/registries.conf.d/00-shortnames.conf/containers/registries.conf.d/00-shortnames.conf$ sudo
 chown $UID
:$GID
 ${XDG_CONFIG_HOME}
/containers/registries.conf.d/00-shortnames.confchown/containers/registries.conf.d/00-shortnames.conf

Afterwards, you should be able to refer to the image via its short name:

$ podman
 run busybox echo "Hello World"
run busybox echo  Resolved
 "busybox"
 as an alias (/home/user/.config/containers/registries.conf.d/00-shortnames.conf)as an alias (/home/user/.config/containers/registries.conf.d/00-shortnames.conf)  Trying
 to pull docker.io/library/busybox:latest...to pull docker.io/library/busybox:latest...  Getting
 image source signaturesimage source signatures  Copying
 blob 50e8d59317eb done  blob 50e8d59317eb done  Copying
 config 1a80408de7 done  config 1a80408de7 done  Writing
 manifest to image destinationmanifest to image destination  Storing
 signaturessignatures  Hello
 WorldWorld

However, if you run an image that doesn’t have an alias in this list, you will still get an error:

$ podman
 run curlimages/curl earthly.devrun curlimages/curl earthly.dev  Error
: short-name "curlimages/curl"
 did not resolve to an alias and no unqualified-search registries are defined in "/home/user/.config/containers/registries.conf"
: short-namedid not resolve to an alias and no unqualified-search registries are defined in

This means you have to extend this list with each alias you want to allow.

The second approach is (if you aren’t worried about spoofing) to add docker.io as an unqualified search registry in ${XDG_CONFIG_HOME}/containers/registries.conf.

This file can also be used for setting up mirrors, e.g an internal company mirror for DockerHub or Quay, or the public GCR mirror of DockerHub, as in the following snippet. Note that I eventually left it commented as it doesn’t contain the Earthly images I’ll be using later in this tutorial.

$ cat
 >>
$HOME
/.config/containers/registries.conf <<EOF
/.config/containers/registries.conf unqualified-search-registries = ['docker.io'] # [[registry]] # # Ref: https://cloud.google.com/container-registry/docs/pulling-cached-images # prefix="docker.io" # location="mirror.gcr.io"EOF

Running curlimages/curl should output something like this now:

$ podman
 run curlimages/curl earthly.devrun curlimages/curl earthly.dev  Resolving
 "curlimages/curl"
 using unqualified-search registries (/home/user/.config/containers/registries.conf)using unqualified-search registries (/home/user/.config/containers/registries.conf)  Trying
 to pull docker.io/curlimages/curl:latest...to pull docker.io/curlimages/curl:latest...  Getting
 image source signaturesimage source signatures  Copying
 blob 28867d2f810e done  blob 28867d2f810e done  Copying
 blob 3aa4d0bbde19 done  blob 3aa4d0bbde19 done  Copying
 blob 968ce6b2fb58 done  blob 968ce6b2fb58 done  Copying
 blob 701f5fe5d595 done  blob 701f5fe5d595 done  Copying
 blob 9c3e0e9fd2ff done  blob 9c3e0e9fd2ff done  Copying
 blob 1082a46b0d76 done  blob 1082a46b0d76 done  Copying
 blob 610724250ccf done  blob 610724250ccf done  Copying
 blob a8b5e80ef070 done  blob a8b5e80ef070 done  Copying
 blob b518d4c718b9 done  blob b518d4c718b9 done  Copying
 config 375c62ad36 done  config 375c62ad36 done  Writing
 manifest to image destinationmanifest to image destination  Storing
 signaturessignatures    %
 Total % Received % Xferd  Average Speed   Time Time  Time  CurrentTotal % Received % Xferd Average Speed Time Time Time Current                                Dload
  Upload   Total   Spent Left  SpeedUpload Total Spent Left Speed  100
 35  100 35 0  0  73   0 --:--:-- --:--:-- --:--:-- 7335 100 35 0 0 73 0 --:--:-- --:--:-- --:--:-- 73  Redirecting
 to https://earthly.dev/%to https://earthly.dev/%

Authenticating to Private Registries

Podman’s authentication mechanisms are compatible with Docker including support for credential helpers. Basic authentication works exactly the same, e.g. podman login $MY_REGISTRY -u $MY_DOCKER_USER -p $MY_DOCKER_PASSWORD can be used to login through the CLI. Podman uses a credential file in JSON format, located at ${XDG_CONFIG_HOME}/containers/auth.json, and falls back to $HOME/.docker/config.json if the former can’t be found, so that Docker’s authentication configuration can be directly reused. If you are using credential helpers with Docker, you can continue to use them with podman too. For instance, I am using docker-credential-acr-env for unattended login to a private Azure Container Registry. I have placed the docker-credential-acr-env executable on my $PATH, and the following in my auth.json:

This continues to work with podman without any change.

You can find out more about authentication configuration on the manual page of containers-auth.json.

Using an Init

When you run a container with the --init flag it will fail with the following error message:

Error: container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory

This happens because podman doesn’t provide an init executable out of the box. Installing the catatonit package will provide it as the default init binary for podman and resolve the issue.

$ sudo
 pacman -S catatonitpacman -S catatonit

If you would like to use a different init, e.g tini (which is the default for docker), you can provide it with --init-path. Remember that it should be built as a static binary, as it is executed in the container.

$ podman
 run --init --init-path="/usr/bin/tini"
 --rm php:cli bash -c "ls -al /"
run --init --init-path=--rm php:cli bash -c

Docker Compose

There are two ways to use the lightweight orchestration framework. podman can serve as the backend for docker-compose. In a rootless setup, this requires you to start the podman.service user unit, and set the DOCKER_HOST variable to point to the userland podman socket:

$ systemctl
 --user enable podman.service--user enable podman.service$ systemctl
 --user start podman.service--user start podman.service$ export
 DOCKER_HOST=
unix://${XDG_RUNTIME_DIR}
/podman/podman.sockunix:///podman/podman.sock

You most likely want to export the variable in your .bashrc or equivalent.

Let’s try it out with a basic WordPress application:

version
:
 
"3.9"    services
:  
db
: 
image
:
 mysql:5.7 
volumes
:   
-
 ./db_data:/var/lib/mysql 
restart
:
 always 
environment
:   
MYSQL_ROOT_PASSWORD
:
 somewordpress   
MYSQL_DATABASE
:
 wordpress   
MYSQL_USER
:
 wordpress   
MYSQL_PASSWORD
:
 wordpress      
wordpress
: 
depends_on
:   
-
 db 
image
:
 wordpress:latest 
volumes
:   
-
 ./wordpress_data:/var/www/html 
ports
:   
-
 
"8000:80" 
restart
:
 always 
environment
:   
WORDPRESS_DB_HOST
:
 db   
WORDPRESS_DB_USER
:
 wordpress   
WORDPRESS_DB_PASSWORD
:
 wordpress   
WORDPRESS_DB_NAME
:
 wordpress

Start it up:

$ docker-compose
 upup

You can verify that it’s running by visiting localhost:8000:

Note that at the time of writing there’s an unresolved issue causing containers to fail on named mounts with the following message:

Error response from daemon: fill out specgen: /var/lib/mysql: duplicate mount destination

Hopefully it gets fixed soon.

Alternatively to running docker-compose, there’s a podman-compose utility that uses a daemon-less process model that directly executes podman.

$ sudo
 pacman -S podman-composepacman -S podman-compose

The CLI is similar to docker-compose, e.g podman-compose up will bring up the services. However, I noticed that sending the keyboard interrupt signal will not necessarily shut them down, so make sure you run podman-compose down if you don’t want the processes lingering around in the background.

Using Earthly

Note: Make sure you are on v0.6.15 or later for cgroups v2 support.

Earthly runs BuildKit in a container which requires the cgroups CPU controller to set CPU limits. On some systemd-based systems using cgroups v2 (including Arch), non-root users do not have CPU delegation permissions, which causes enabling the CPU controller to fail. As a consequence, when running Earthly you might get the following error right in the beginning:

sh
: write error: No such file or directory: write error: No such file or directory

buildkitd
: operation not permitted: operation not permittedError
: buildkit process has exited: buildkit process has exited

If rootless, check that your user has permissions to delegate at least cpu and pids:

$ cat
 "/sys/fs/cgroup/user.slice/user-
$(
id
 -u)
.slice/user@
$(
id
 -u)
.service/cgroup.controllers"
-u-u

If the permissions are missing, you can add these for all users by creating or modifying the file at /etc/systemd/system/[email protected]/delegate.conf

[Service]Delegate
=
memory pids cpu io

After a reboot, you should see these permissions and successfully run Earthly.

 VERSION 0.6FROM
 python:3python:3 build:  RUN
 mkdir -p /src && echo "print('Hello World')"
 >> /src/hello.pymkdir -p /src && echo>> /src/hello.py SAVE ARTIFACT src /src docker:  COPY
 +build/src src+build/src src  ENTRYPOINT
 ["python3"
, "./src/hello.py"
] SAVE IMAGE python-example:latest

When running Earthly, you should see in the logs that it uses podman

$ earthly
 +docker+docker

 1. Init 🚀————————————————————————————————————————————————————————————————————————————————        buildkitd | Found buildkit daemon as podman container (earthly-buildkitd) 2. Build 🔧————————————————————————————————————————————————————————————————————————————————         python:3 | --> Load metadata linux/amd64            +base | --> FROM python:3            +base | 
[       ]
   0% resolve docker.io/library/python:3@sha256:48d2ed838ff2f27066f550cdb2887f9f601
[██████████]
 100% resolve docker.io/library/python:3@sha256:48d2ed838ff2f27066f550cdb2887f9f601af8921a72e1f0366c37a0ee4e5d3a           +build | --> RUN mkdir -p /src && echo "print('Hello World')" >> /src/hello.py

You should be able to run the created image with podman:

$ podman
 run python-example:latestrun python-example:latestHello
 WorldWorld

Conclusion

As you can see there’s much we can do today using the latest version of podman in rootless mode. Additionally to being a substitute for Docker’s core functionality, we can to run docker-compose services and use the containerized build tool, Earthly, to build images. There are definitely some rough edges which hopefully get smoothened out as podman gradually gains more traction; just make sure to expect some bumps along the way when you give it a try :).

Earthly makes CI/CD super simple
Fast, repeatable CI/CD with an instantly familiar syntax – like Dockerfile and Makefile had a baby.

Go to our homepage to learn more