Larvitz Blog
Mastodon

Fedora Hummingbird: Distroless, Container-Native Linux and a Slim mastogreet

Header image

Table of Contents

A few weeks ago I wrote up how I deploy mastogreet onto my RHEL hosts with containers.podman and Quadlet generation. The Ansible side is genuinely tidy. The base image, on the other hand, is python:3.12-slim from Docker Hub: a Debian-flavored Python with a full shell, apt, GNU coreutils, and a long tail of binaries the bot will never touch. “Slim” is generous.

This article is the rebuild. I took the same mastogreet and put it on top of registry.access.redhat.com/hi/python:3.13, a distroless Python from Project Hummingbird. The runtime image came out roughly 10% smaller (126 MB versus 140 MB on amd64), which turns out to be the least interesting result. The more meaningful changes are qualitative: no shell, no package manager, a signed SBOM, and one UID-mapping footgun I didn’t see coming until the first smoke-test run. The Ansible role needed a two-line variable diff and zero task changes.

In practice, “distroless” here means the runtime image contains only the Python interpreter and the libraries the application links against. No /bin/bash, no dnf, no coreutils, no incidental tooling. Everything that would let an attacker spelunk after they land a code-execution primitive is simply not there to spelunk with.

The rest of this article walks through what the Hummingbird ecosystem is, the rebuild itself, the smoke-test footgun, and where Red Hat Hardened Images (the commercial downstream) fits in.

What “Fedora Hummingbird” Actually Refers To

There are two related things that share the Hummingbird name. It’s worth keeping them separate before going any further; the rest of the article uses both terms with intent.

Project Hummingbird: the image catalog

Project Hummingbird is the upstream community building the image catalog. As of the Summit announcement it had roughly 49 unique minimal, hardened, distroless container images, multiplied to about 157 variants once you count FIPS builds and multi-arch (x86_64 and aarch64). Coverage spans Python, Go, Node.js, Rust, Ruby, OpenJDK, .NET, PostgreSQL, MariaDB, nginx, Caddy, a core-runtime base for compiled binaries, and a few dozen others.

The build pipeline is Konflux, with hermetic builds, pinned package lists, continuous scanning via Syft and Grype, SBOM generation, and image signatures. Roughly 95% of the content is Fedora Rawhide packages used unmodified, so the security story isn’t “we forked everything,” it’s “we built a tight pipeline around upstream.”

Fedora Hummingbird Linux: the operating system

Fedora Hummingbird Linux is the new image-based rolling Linux distribution that takes the same model up a level. The root filesystem is read-only, writable state is confined to /var and /etc, updates are atomic with rollback, and the whole OS is delivered as an OCI image you can pull and boot via bootc. The kernel is the Always Ready Kernel (ARK) from the CKI project, tracking Linus’ mainline.

The OS images currently live at quay.io/hummingbird-community/bootc-os. The Fedora Council has granted the trademark, and integration into Fedora proper is happening through a SIG.

The agentic framing

The marketing layer on top of both, in Red Hat’s press release, frames Hummingbird as the OS for “agent-first builders” in the “agentic era.” I suspect that framing is mostly aimed at making the project legible to current industry conversations, and I’d argue the same properties (small attack surface, hermetic builds, fast CVE turnover, registration-free deployment, machine-readable SBOMs) are also exactly what a non-agentic ops team has wanted from container base images for a decade. So: useful regardless of whether the workload calling podman run happens to be a person, a CI job, or an LLM. The distribution itself doesn’t care.

Why the Model Is Useful

The Hummingbird containers are distroless in a stronger sense than python:3.12-slim. No package manager, no /bin/sh, no /bin/bash, no shell utilities. The runtime image carries the language interpreter, its supporting libraries, the RPM metadata for the packages that are present (so vulnerability scanners can still introspect), and almost nothing else. The often-cited nginx comparison is illustrative: the Docker Hub nginx image has roughly 270 binaries in /bin/; the Hummingbird nginx image has 31.

A handful of properties fall out of that:

  • Smaller attack surface. No shell means an attacker who lands a code execution primitive in your Python process can’t reach for /bin/sh -c, can’t enumerate the filesystem with familiar tooling, and can’t curl a second-stage payload. None of that is a substitute for not having a code execution primitive, but it raises the floor.
  • Near-zero CVE goal. Because the package list is pinned and the pipeline scans continuously with Syft and Grype, the catalog’s stated objective is “zero CVE reports in every image” at any given point in time. In practice that means CVE fixes flow through within hours to days, not the weeks-or-months you sometimes see on community images.
  • SBOMs and signatures by default. Each image ships an SBOM you can pull with cosign download sbom, and images are signed. For supply-chain-conscious environments this is the difference between “trust me” and “verify.”
  • Hermetic, reproducible builds. Builds happen in isolation from arbitrary network access, against pinned package lists. The same input produces the same output.
  • Chunkah re-layering. Traditional OCI images bundle many packages into a handful of large filesystem layers, which means changing one package invalidates the whole layer’s cache. Hummingbird’s chunkah tool re-cuts those layers into smaller content-addressable chunks grouped by package origin, so updates and related images share more cached content. The visible side of this is the 33-layer pull you’ll see in the build output below; the practical side is that a CVE refresh only re-downloads the chunks that actually changed.

The trade-off is real: you can’t podman exec -it into a running container and poke around with a shell, because there is no shell. You also can’t RUN apt-get install ... your way to a dependency in production. Both of those are intentional, and both of them push you toward multi-stage builds and proper observability. We’ll get to that.

The Image Catalog and the Registry Layout

The catalog is at registry.access.redhat.com/hi/. Python is registry.access.redhat.com/hi/python. Tags work the way you’d hope:

  • :latest follows the newest supported Python (currently 3.14)
  • :3.13, :3.12, :3.11 stay pinned to those branches
  • :3.13-builder is the same Python but with dnf, bash, and friends, intended for multi-stage build stages
  • :3.13-fips is the FIPS-validated variant for compliance-bound environments

Equivalents exist for Node.js (hi/nodejs), Go (hi/go), Java/OpenJDK (hi/openjdk), and so on. There’s also hi/core-runtime for compiled binaries that don’t need a language runtime at all.

A few alternative registries exist for the same content with different trust properties:

  • quay.io/hummingbird/ mirrors the catalog without Red Hat’s signing
  • quay.io/hummingbird-community/ is the community-supported entry point
  • quay.io/hummingbird-rawhide/ follows Fedora Rawhide directly

For anything I run on real hosts I want the signed copies, so registry.access.redhat.com/hi/ it is.

Rebuilding mastogreet on Hummingbird Python

Time for a concrete example. The original mastogreet Containerfile is straightforward:

FROM python:3.12-slim

ARG UID=10001
ARG GID=10001

RUN groupadd --system --gid ${GID} mastogreet \
 && useradd  --system --uid ${UID} --gid ${GID} --home-dir /app --no-create-home mastogreet

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY mastowelcome.py .

RUN install -d -o mastogreet -g mastogreet /data
VOLUME ["/data"]

USER mastogreet

ENV PYTHONUNBUFFERED=1 \
    MASTOGREET_CONFIG=/data/config.yaml \
    MASTOGREET_DB=/data/state.db

It works. It builds in seconds. The image weighs about 140 MB on disk because python:3.12-slim carries Debian’s userland, pip‘s build tooling, libraries the bot never imports, and a full shell so the RUN groupadd && useradd line above can even exist.

Here is the same bot on top of hi/python:

# Build stage: includes dnf, bash, build tools. Never ships to production.
FROM registry.access.redhat.com/hi/python:3.13-builder AS builder

USER 0
RUN ["python3", "-m", "venv", "/opt/venv"]
ENV PATH="/opt/venv/bin:$PATH"

COPY requirements.txt /tmp/requirements.txt
RUN ["pip3", "install", "--no-cache-dir", "-r", "/tmp/requirements.txt"]

# Pre-create the /data skeleton with the right ownership so the
# runtime stage doesn't need a shell or coreutils to do it itself.
RUN ["install", "-d", "-o", "65532", "-g", "65532", "/data-skel"]

# Runtime stage: distroless. No shell, no dnf, no surprises.
FROM registry.access.redhat.com/hi/python:3.13

COPY --from=builder /opt/venv /opt/venv
COPY --from=builder --chown=65532:65532 /data-skel /data
COPY --chown=65532:65532 mastowelcome.py /app/mastowelcome.py

ENV PATH="/opt/venv/bin:$PATH" \
    PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    MASTOGREET_CONFIG=/data/config.yaml \
    MASTOGREET_DB=/data/state.db

WORKDIR /app
USER 65532
VOLUME ["/data"]

CMD ["python3", "mastowelcome.py"]

A few details are worth pulling apart.

Array Syntax Everywhere

Every RUN instruction uses the JSON array form, not the shell-string form. The shell form (RUN pip install -r requirements.txt) implicitly invokes /bin/sh -c. The Hummingbird runtime image doesn’t have a shell, and the builder image has one but it’s good practice to write the Containerfile so that the same instructions would work if you moved them to the runtime stage. Same logic for CMD ["python3", "mastowelcome.py"]: array form, no shell needed.

This is the most common porting mistake people hit: a perfectly reasonable RUN echo "PATH=/opt/venv/bin" >> /etc/profile falls over in the runtime image, and the error message is the cheerful executable file not found in $PATH: bash.

No useradd, Because There’s a User Already

The original Containerfile created a custom mastogreet:10001 user. The Hummingbird images already ship a non-root user at UID 65532 by default, which is the convention Google’s distroless project popularised and which Hummingbird adopted.

Adding a second non-root user is only possible in the builder stage (since the runtime has no useradd), and would force me to either copy /etc/passwd and /etc/group across stages or set USER numerically in the runtime image.

Easier and cleaner: drop the custom UID, use the one the image gives me, and chown the /data skeleton to 65532 in the builder stage. On the Ansible side, mastogreet_container_uid: 65532 falls out as a single variable change and everything else still composes.

Virtual Environment as the Transport Mechanism

The builder stage installs dependencies into /opt/venv. The runtime stage receives /opt/venv whole, via COPY --from=builder. This is the idiomatic Hummingbird Python pattern, and it’s there for a reason: pip is technically retained in the runtime image (the catalog kept it for compatibility with how interpreted-language workflows expect to behave), but you really don’t want to invoke it there, because you have no shell to wrap it in and no compilers to build native extensions. The venv keeps all the work in the builder stage where the tools exist, and ships a self-contained directory the runtime can just point PATH at.

If a dependency needs C compilation, the builder stage is also where dnf install -y gcc python3-devel libffi-devel happens; none of that ships to production.

The State Directory, Pre-Owned

In the original Containerfile, RUN install -d -o mastogreet -g mastogreet /data ran in the final image because Debian has install(1) everywhere. In Hummingbird, that command runs in the builder stage and writes to a /data-skel path, which then gets copied to /data in the runtime stage with COPY --from=builder --chown=65532:65532. Same effect, but the runtime image never had to host coreutils.

SELinux and Quadlet Compatibility

Nothing about the bind-mount story changes. The host still mounts /opt/mastogreet/ to /data with the :Z flag, and the Ansible role still ensures the host-side directory is owned by 65532 instead of 10001. The Quadlet unit generated by containers.podman.podman_container is identical in shape to the one in the previous article; only the variables move.

What You Actually Get

Building it for real:

$ podman build -t git.linuxserver.pro/chris/mastogreet:hummingbird -f Containerfile-hi .
[1/2] STEP 1/7: FROM registry.access.redhat.com/hi/python:3.13-builder AS builder
Trying to pull registry.access.redhat.com/hi/python:3.13-builder...
... (33 chunkah layers stream in)
[1/2] STEP 6/7: RUN ["pip3", "install", "--no-cache-dir", "-r", "/tmp/requirements.txt"]
Collecting Mastodon.py<2,>=1.8 (from -r /tmp/requirements.txt (line 1))
...
Successfully installed Mastodon.py-1.8.1 PyYAML-6.0.3 blurhash-1.1.5
certifi-2026.4.22 charset_normalizer-3.4.7 decorator-5.2.1 idna-3.15
python-dateutil-2.9.0.post0 python-magic-0.4.27 requests-2.34.2
six-1.17.0 urllib3-2.7.0
[1/2] STEP 7/7: RUN ["install", "-d", "-o", "65532", "-g", "65532", "/data-skel"]
[2/2] STEP 1/9: FROM registry.access.redhat.com/hi/python:3.13
...
[2/2] COMMIT git.linuxserver.pro/chris/mastogreet:hummingbird
Successfully tagged git.linuxserver.pro/chris/mastogreet:hummingbird

The build is uneventful. The interesting bit is what comes out the other end:

$ podman images | grep mastogreet
git.linuxserver.pro/chris/mastogreet  hummingbird  e97c9dafdcc9  21 seconds ago  126 MB

A head-to-head against the same bot built on python:3.12-slim:

$ podman images | grep mastogreet
git.linuxserver.pro/chris/mastogreet  latest       e835eccb3d62  140 MB
git.linuxserver.pro/chris/mastogreet  hummingbird  e97c9dafdcc9  126 MB

126 MB versus 140 MB. That’s a roughly 10% shrink, which is worth saying clearly: if you came here expecting Google-style 30 MB distroless images, you won’t find them. hi/python is a full Python 3.13 runtime with the Fedora-derived support libraries it depends on, just without the shell, package manager, and incidental tooling. The size win is real but it isn’t the headline, and anyone selling Hummingbird on file-size alone is selling it wrong.

The headline is everything else:

  • The image has no bash, no dash, no sh. podman exec -it mastogreet sh exits with executable file 'sh' not found in $PATH.
  • It has no apt, no dnf, no rpm binary. The RPM database is preserved as data for scanners, but the executable that would manipulate it isn’t there.
  • The chunkah re-layering is visible in the pull: 33 small layers rather than half a dozen fat ones. Repeated pulls of related images deduplicate heavily, and CVE refreshes only re-pull the chunks that changed.
  • The image is signed, and ships with an SBOM you can pull:

sh cosign download sbom --platform linux/amd64 \ registry.access.redhat.com/hi/python:3.13 > python.spdx.json

Rootless Smoke Test, with the Footgun

Before deploying the image to a host, I always smoke-test the container locally. With Hummingbird this surfaces a UID gotcha worth covering, because it’s the one I expect most readers will hit on their first run.

The naive command, copied over from the previous bot:

$ podman run --rm \
    --userns=keep-id:uid=10001,gid=10001 \
    -e MASTOGREET_ACCESS_TOKEN=... \
    -e MASTOGREET_INSTANCE_URL=https://burningboard.net \
    -v "$PWD/smoke:/data:Z" \
    git.linuxserver.pro/chris/mastogreet:hummingbird

Traceback (most recent call last):
  File "/app/mastowelcome.py", line 28, in init_db
    conn = sqlite3.connect(path)
sqlite3.OperationalError: unable to open database file

What happened: the image’s USER 65532 would normally run the process as UID 65532 inside the container, and /data inside the image is owned by 65532 from the builder skeleton copy. But --userns=keep-id:uid=10001,gid=10001 overrode that, mapping the host user to UID 10001 inside instead. The bot process therefore runs as 10001, /data is owned by 65532, and the open-for-write on state.db fails with EACCES.

The fix is to match the userns mapping to the image’s actual UID:

$ podman run --rm \
    --userns=keep-id:uid=65532,gid=65532 \
    -e MASTOGREET_ACCESS_TOKEN=... \
    -e MASTOGREET_INSTANCE_URL=https://burningboard.net \
    -v "$PWD/smoke:/data:Z" \
    git.linuxserver.pro/chris/mastogreet:hummingbird

Traceback (most recent call last):
  File "/app/mastowelcome.py", line 257, in main
    me = mastodon.me()
  ...
mastodon.errors.MastodonUnauthorizedError: ('Mastodon API returned error',
    401, 'Unauthorized', 'Der Zugriffstoken ist ungültig')

That looks like a failure, but it’s actually the smoke test passing: the bot started, sqlite opened /data/state.db cleanly, the Mastodon client built and made an authenticated HTTPS call to burningboard.net, and the server replied 401 because I supplied a placeholder token. Everything up to “real API conversation” works.

This wrinkle is specifically a rootless smoke-test thing. The production path through the Ansible role is unaffected: the host-side /opt/mastogreet/ is chowned to 65532 as root by the file task, the Quadlet runs the container without rootless userns remapping, and the in-container UID matches the host directory ownership without any keep-id gymnastics. Worth being explicit about, because someone will copy the smoke-test snippet from the previous article verbatim, hit this error, and spend twenty minutes wondering whether Hummingbird broke their bot. It didn’t; the UID just moved.

Day-to-Day Differences

The bot behaves identically once running. podman logs mastogreet shows the same authentication line, the same backfill cutoff, the same poll loop. The only operational difference is that podman exec -it mastogreet sh no longer drops me into a shell, which is fine because I don’t actually need to do that. When I do want to inspect the running container, I either use podman exec mastogreet python3 -c '...' or run the :3.13-builder image side-by-side for poking purposes.

Updating the Ansible Role

The role from the previous article needs two trivial changes:

# group_vars/all.yml
mastogreet_image: git.linuxserver.pro/chris/mastogreet:hi-20260516
mastogreet_container_uid: 65532
mastogreet_container_gid: 65532

That’s it. The Ensure state directory exists with container UID ownership task picks up the new UID, the templated config.yaml lands with the right ownership, and the Quadlet unit gets re-rendered because mastogreet_image changed. Handler chain reloads systemd, restarts the service, and the next container creation pulls the Hummingbird-based image. Zero changes to the role’s tasks or handlers.

This is also a decent illustration of why the role-plus-variables approach pays off: the entire base-image migration is a two-line variable diff in group_vars/all.yml. No host-side commands, no manual daemon-reload, no remembered steps.

Red Hat Hardened Images: The Commercial Downstream

The article up to this point has been about the Fedora-community side: free, vendor-neutral, available to anyone who can podman pull. The images at registry.access.redhat.com/hi/ are themselves free to consume. The commercial wrapper around the same artefacts is Red Hat Hardened Images, and the separation is fairly clean.

What you get when you buy in:

  • Support under your existing RHEL or OpenShift subscription. Hardened-image consumption is covered; you can open a case if something breaks. The community images you pull anonymously are best-effort.
  • The Calunga trusted-libraries index. A vetted Python package index at packages.redhat.com/trusted-libraries/python/, accessed via a mounted pip.conf secret in the builder stage. The point isn’t faster downloads; the point is that somebody has signed and stands behind the packages. For regulated environments this matters more than for a Mastodon bot.
  • FIPS variants and validated cryptography. The :3.13-fips tag points at images linked against FIPS-validated OpenSSL, with the validation paperwork attached.
  • Contractual CVE response times. Same near-zero CVE design goal as the community catalog, with SLAs around it.

The underlying images are built by the same Konflux pipeline. The commercial offer adds support and contractual guarantees; it does not gatekeep the artefacts themselves. If you want to evaluate and contribute, you do that entirely from the community side. If you need someone to call at 2am, you upgrade.

A Brief Note on Fedora Hummingbird the OS

I’ve focused on the image catalog because that’s the part I’m using in anger today. The OS side is interesting independently. A bootable container that becomes a full Linux host, with atomic updates, rollback, and a read-only root, is a meaningful shift for the kind of small-fleet operator this blog gets read by. I’m running it in a VM on my workstation to see how it behaves; the experience so far is “Fedora, but the upgrade story is bootc upgrade && systemctl reboot, and there’s no dnf upgrade to forget about.” I’ll write that up properly once I have more than a few weeks of bootc time on it.

The relevant point for this article is that the OS and the image catalog share machinery: the same Konflux pipeline produces both, the same hermetic-build guarantees apply, and the same SBOMs ship for both. If you’re already running Hummingbird containers, the OS is a natural extension. If you’re not, the OS is also a perfectly good place to start.

When This Pays Off and When It Doesn’t

Hummingbird images are worth reaching for when:

  • The workload runs in production and survives reinstalls. The cost of a multi-stage Containerfile is paid once.
  • Security review will ask for an SBOM, signatures, or a CVE-response timeline. Showing up with cosign verify and a Konflux attestation is a much shorter conversation than promising to “look into it.”
  • The container is part of a fleet, not a one-off. The compounding benefit of small, fast-updating images is real once you have more than a handful of hosts.
  • You already accept the trade-offs of distroless (no shell access, multi-stage build mandatory, debugging shifts to logs and structured tooling).

I’d skip them for:

  • Throw-away development containers where I want to podman exec sh and iterate. The builder variant is closer to that experience than the runtime one, but python:3.12-slim is still faster to spin up.
  • Workloads that genuinely depend on tools the distroless image doesn’t ship (custom shell-script entrypoints, tini-style PID-1 wrappers that aren’t already in the base, things that expect getent at runtime). You can usually work around this, but if the workaround dominates the Containerfile, the gain is gone.
  • Anything where the operator workflow assumes a shell is one podman exec away. Worth adopting eventually, but not as a same-day forklift.

For the mastogreet case, the rebuild took about half an hour: drafting the multi-stage Containerfile, fixing one RUN line that I left in shell form, bumping the UID in group_vars/all.yml, and untangling the rootless smoke-test footgun once. Since cutting over to the Hummingbird-based image the journal output has been indistinguishable from the previous deployment. The next rebuild will move from :3.13 to :3.14 so I stay on a refreshed CVE baseline rather than pin too tightly.

References

Comments

You can use your Mastodon or other ActivityPub account to comment on this article by replying to the associated post.

Search for the copied link on your Mastodon instance to reply.

Loading comments...