Table of Contents
I’ve written about Podman Quadlets, rootless production Podman, and Ansible-driven Quadlet deployments before. Those articles cover the steady state: containers running, systemd managing them, configuration under version control. What they don’t cover is the moment between git push and systemctl restart: the pipeline that builds the image, proves where it came from, and hands it off to the host.
MastoSum is a FastAPI app I run that watches Mastodon hashtag streams, stores matching posts, and generates news-style summaries through LLMs or a deterministic fallback. The application topology is intentionally boring: one container image, four roles selected by an environment variable (web, worker, beat, flower), Postgres and Valkey as backing services. What’s worth writing about is how it ships.
The deployment story is fully automated, tag-driven, and runs entirely rootless inside a single Podman user context. Every piece, the CI runner, the registry, the build pipeline, the signature verification, the deploy trigger, the service restarts, lives inside the mastosum@ user on one RHEL host. This article walks through each link in that chain.
The Stack at Rest
Before getting into the pipeline, it helps to see what the running stack looks like. On the production host, everything runs as the mastosum user:
$ podman ps
CONTAINER ID IMAGE COMMAND STATUS PORTS NAMES
dda7c5013624 registry.redhat.io/rhel10/valkey-8:latest run-valkey Up 4 days (healthy) 6379/tcp mastosum-valkey
13a829bea824 registry.redhat.io/rhel10/postgresql-16:latest run-postgresql Up 4 days (healthy) 5432/tcp mastosum-postgres
bedb39c79a9d code.forgejo.org/forgejo/runner:12 forgejo-runner da... Up 2 hours mastosum-runner
7055f528d292 git.linuxserver.pro/chris/mastosum:stable python3 -m mastos... Up 25 minutes (healthy) 127.0.0.1:8000->8000/tcp, 5555/tcp, 8080/tcp mastosum-web
9d3c7666cc6a git.linuxserver.pro/chris/mastosum:stable python3 -m mastos... Up 25 minutes (healthy) 5555/tcp, 8000/tcp, 8080/tcp mastosum-worker
b4d69e12ba35 git.linuxserver.pro/chris/mastosum:stable python3 -m mastos... Up 25 minutes (healthy) 5555/tcp, 8000/tcp, 8080/tcp mastosum-beat
51e77dea5ffb git.linuxserver.pro/chris/mastosum:stable celery -A mastosu... Up 25 minutes (healthy) 127.0.0.1:5555->5555/tcp, 8000/tcp, 8080/tcp mastosum-flower
Seven containers. Three images: the app image (mastosum:stable) running four roles, plus Postgres and Valkey from Red Hat’s container catalog, plus the Forgejo Actions runner. All of them managed by systemd via rootless Podman Quadlets. The runner is a first-class member of the stack, not a separate daemon on a separate host. This setup, the runner living inside the same user context it deploys into is what makes the pipeline clean. We’ll get to why.
The app image uses Red Hat’s UBI10 Python 3.12 minimal base:
FROM registry.access.redhat.com/ubi10/python-312-minimal AS builder
# ... venv, pip install, etc.
FROM registry.access.redhat.com/ubi10/python-312-minimal
# ... copy venv, set up /data, USER 1001
CMD ["python3", "-m", "mastosum.container_entrypoint"]
UBI minimal weighs roughly 40 MB compressed and ships with a pre-created non-root user at UID 1001. No extra useradd step, no shadow-utils dependency, and far less base-system surface area than a full distribution image. The image is signed by Red Hat and distributed through registry.redhat.io, which means the trust chain starts at a verified source before any of my code enters the picture.
The four app roles reference the moving tag git.linuxserver.pro/chris/mastosum:stable in their Quadlet files. I never edit Quadlets to deploy a new version. A release pushes both an immutable :v<version> tag (audit trail, rollback) and re-points :stable to the same digest. The Quadlets don’t know or care which version is current; they just pull whatever :stable resolves to.
The Pipeline: Tag, Build, Push, Sign
I create releases as signed git tags. The workflow is triggered by any tag matching v*; whether the Forgejo instance or the workflow itself verifies the tag signature before building is a separate configuration choice. The important property is that the release tag carries a cryptographically verifiable identity, while the workflow that follows from it is deterministic.
git --git-dir=.repo.git --work-tree=. tag -s v0.2.9 -m "Release 0.2.9"
git --git-dir=.repo.git --work-tree=. push origin v0.2.9
That push triggers .forgejo/workflows/release.yml, which runs on the mastosum-prod runner. The workflow has four steps, each building on the last:
1. Build
- name: Build image
run: |
docker build \
--build-arg GIT_HASH="$(echo "$GITHUB_SHA" | cut -c1-7)" \
-f Containerfile \
-t "${IMAGE}:${{ steps.tag.outputs.tag }}" \
-t "${IMAGE}:stable" \
.
The build uses docker (the CLI binary from the docker.io package, which talks to Podman’s compatible socket). The GIT_HASH build arg bakes the commit SHA into the image so the /health endpoint can report exactly which revision is running. Both tags (:v0.2.9 and :stable) are applied at build time to the same image. They will have the same digest.
2. Push
- name: Push image
env:
REGISTRY_USER: ${{ secrets.REGISTRY_USER }}
REGISTRY_TOKEN: ${{ secrets.REGISTRY_TOKEN }}
run: |
echo "${REGISTRY_TOKEN}" | docker login -u "${REGISTRY_USER}" --password-stdin git.linuxserver.pro
docker push "${IMAGE}:${{ steps.tag.outputs.tag }}"
docker push "${IMAGE}:stable"
Two tags, same digest. :stable moves forward; :v0.2.9 stays pinned forever. The registry credentials are Forgejo Actions secrets scoped to the repository, using a robot account with write-only push access.
3. Sign
- name: Sign image with cosign
env:
COSIGN_KEY: ${{ secrets.COSIGN_KEY }}
COSIGN_PASSWORD: ${{ secrets.COSIGN_PASSWORD }}
run: |
umask 077
KEY=$(mktemp)
printf '%s' "${COSIGN_KEY}" > "${KEY}"
DIGEST=$(docker inspect --format '{{ index .RepoDigests 0 }}' "${IMAGE}:${{ steps.tag.outputs.tag }}")
cosign sign --yes --key "${KEY}" "${DIGEST}"
shred -u "${KEY}" 2>/dev/null || rm -f "${KEY}"
The private key material is written to a temporary file with umask 077, used exactly once, and shredded. The signature lands in the registry as a <digest>.sig tag plus a Rekor transparency-log entry. The private key itself lives only in the Forgejo Actions secret store and in an offline backup; the CI runner never sees it beyond the duration of this step.
A note on the docker inspect line: RepoDigests[0] picks the first digest the local daemon knows about, which is usually correct after a fresh push but can be fragile if multiple digests are present. A safer alternative that queries the registry directly:
DIGEST="$(skopeo inspect --format '{{.Digest}}' "docker://${IMAGE}:${TAG}")"
cosign sign --yes --key "${KEY}" "${IMAGE}@${DIGEST}"
This signs the digest as observed from the registry, independent of local daemon state.
4. Deploy Handoff
- name: Deploy to host
run: |
rm -f /data/deploy-status
printf '%s' "${{ steps.tag.outputs.tag }}" > /data/deploy-trigger
for _ in $(seq 1 60); do
if [ -f /data/deploy-status ]; then
cat /data/deploy-status
grep -q '^ok ' /data/deploy-status && exit 0 || exit 1
fi
sleep 5
done
echo "Timed out waiting for deploy completion" >&2
exit 1
This is the handoff. The runner writes the tag name to /data/deploy-trigger and polls for /data/deploy-status to appear. The file I/O is the bridge between CI and host: the runner doesn’t SSH anywhere, doesn’t call systemctl, doesn’t touch the Podman socket for deployment purposes. It drops a file and waits.
Signature Verification: Design and Current Reality
The trust story has three layers, and it’s worth separating the parts that are fully operational from the part that’s waiting on registry support.
Registry authentication (operational)
Forgejo requires authentication for push access. The credentials are a robot account with write-only scope on the package. The runner injects them from repository secrets. Pull access from the production host uses a read-only robot account stored in ~/.config/containers/auth.json.
Host-side cosign verification in the deploy script (operational)
The deploy script (detailed in the next section) calls cosign verify against the root-installed public key before pulling the image. This is the active enforcement path: if verification fails, the deploy stops before any service is touched.
Container policy enforcement via policy.json (target state)
The intended long-term mechanism is for podman pull itself to reject unsigned images via /etc/containers/policy.json, without needing a separate cosign verify call:
{
"transports": {
"docker": {
"git.linuxserver.pro/chris/mastosum": [
{
"type": "sigstoreSigned",
"keyPath": "/etc/containers/cosign-mastosum.pub",
"signedIdentity": {
"type": "matchRepository"
}
}
]
}
}
}
The trust anchor (cosign-mastosum.pub) is installed by root at /etc/containers/cosign-mastosum.pub. The mastosum user cannot modify it, which means even if the runner or any container in the stack is compromised, the attacker cannot swap the public key to accept a malicious image. The policy only applies to git.linuxserver.pro/chris/mastosum; the Red Hat images (Postgres, Valkey) are verified through Red Hat’s own signing infrastructure via the default policy on RHEL hosts.
The policy is documented in the repo (deploy/etc/policy.json.example, deploy/etc/registries.d-git.linuxserver.pro.yaml.example) but the actual trust material lives only on the production host, outside of version control.
Why this isn’t active yet: Forgejo’s container registry does not yet support the OCI 1.1 Referrers API end-to-end. cosign’s OCI 1.1 referrer push fails against it, and use-sigstore-attachments: true in registries.d — the mechanism Podman and Skopeo use to locate sigstore signatures alongside images, cannot locate the signature through the referrers API. In my current Forgejo setup, containers/image with use-sigstore-attachments does not discover the signature reliably, even though cosign verify against the same registry succeeds. I’m treating policy.json enforcement as the target state and keeping explicit cosign verify in the deploy script until the registry/referrers path works end-to-end.
In the meantime, the explicit cosign verify call in the deploy script provides equivalent enforcement. Once Forgejo’s registry supports the Referrers API, the deploy script can drop that step and rely on policy.json alone. The verification setup documented above is the target state; the deploy script carries the active enforcement today.
The Runner in the Same User Context
This is the architectural decision that makes everything else simple. The Forgejo runner runs as a Quadlet container inside the mastosum@ user:
# deploy/quadlet/mastosum-runner.container
[Container]
ContainerName=mastosum-runner
Image=code.forgejo.org/forgejo/runner:12
User=root
SecurityLabelDisable=true
Network=mastosum.network
Volume=%h/runner-data:/data:Z
Volume=%h/runner-data/config.yml:/data/config.yml:ro
Volume=%t/podman/podman.sock:%t/podman/podman.sock:rw
Environment=CONFIG_FILE=/data/config.yml
Environment=DOCKER_HOST=unix:///run/user/1000/podman/podman.sock
Exec=forgejo-runner daemon --config /data/config.yml
The User=root here is container root inside the rootless user namespace, not host root. The container still belongs to the mastosum user’s Podman context and has no privileges beyond what that user already holds.
The runner mounts the host’s rootless Podman socket at the same path it exists on the host. The workflow’s volumes declaration maps /home/mastosum/runner-data:/data, and the runner’s valid_volumes config permits exactly that path. The trigger file and the status file live in ~/runner-data/, which the host’s systemd path unit watches.
Why put the runner in the same user context?
- No cross-user permission problems. The socket is the user’s own. The data directory is the user’s own. The Quadlet files are the user’s own. No
sudo, noroot, nopodman --connection. - The trigger file is local I/O. The workflow doesn’t need an SSH key, an API token, or a webhook receiver. It writes a file to a bind-mounted directory, and systemd picks it up. No network, no auth, no moving parts.
- SELinux stays on. The runner container has
SecurityLabelDisable=trueso it can talk to the Podman socket (container_ttocontainer_runtime_tis denied regardless of file labelling). The option is scoped to this one container; the host policy is untouched. Per-job containers get the same carve-out viacontainer.optionsin the runner config. - The runner is just another service.
systemctl --user restart mastosum-runner.serviceif it hangs.journalctl --user -u mastosum-runner.serviceif you need logs. No separate snowflake infrastructure.
The runner does not cross into a different Unix account or host-root context, but access to the user’s Podman socket is still highly privileged within the mastosum boundary: it can control every container, mount accessible user files, and affect the entire app stack. I treat the runner as part of the production trust domain, not an external agent.
The Deploy Bridge: systemd Path Unit
On the host side, a systemd path unit watches for the trigger file:
# ~/.config/systemd/user/mastosum-deploy.path
[Path]
PathExists=%h/runner-data/deploy-trigger
Unit=mastosum-deploy.service
When the trigger file appears, systemd starts mastosum-deploy.service:
# ~/.config/systemd/user/mastosum-deploy.service
[Service]
Type=oneshot
ExecStart=/bin/bash -c '\
set -u; \
TAG="$(cat %h/runner-data/deploy-trigger 2>/dev/null || echo stable)"; \
rm -f %h/runner-data/deploy-trigger %h/runner-data/deploy-status; \
if %h/bin/deploy "$TAG"; then \
printf "ok %%s\\n" "$TAG" > %h/runner-data/deploy-status; \
else \
rc=$?; \
printf "fail %%s\\n" "$TAG" > %h/runner-data/deploy-status; \
exit "$rc"; \
fi'
The service reads the tag from the trigger file, removes it (so the path unit doesn’t re-fire), runs the deploy script, and writes back a status file the CI workflow is polling for. If the deploy succeeds, the workflow exits 0 and the pipeline goes green. If it fails, the status file carries fail and the workflow exits 1.
The deploy script itself:
#!/usr/bin/env bash
set -euo pipefail
TAG="${1:-stable}"
IMAGE="git.linuxserver.pro/chris/mastosum"
SERVICES=(mastosum-web mastosum-worker mastosum-beat mastosum-flower)
HEALTH_TIMEOUT="${MASTOSUM_DEPLOY_HEALTH_TIMEOUT:-120}"
log() { logger -t mastosum-deploy -- "$*"; printf '%s\n' "$*" >&2; }
log "Starting deploy of ${TAG}"
# Resolve the pushed digest and verify the cosign signature against the
# root-installed public key. If verification fails, the deploy stops
# before any service is touched.
DIGEST="$(skopeo inspect --format '{{.Digest}}' "docker://${IMAGE}:stable")"
REF="${IMAGE}@${DIGEST}"
cosign verify --key /etc/containers/cosign-mastosum.pub "${REF}"
# Pull by digest so we deploy exactly the verified reference, then tag
# it locally as :stable for the Quadlets (which reference the moving tag).
podman pull "${REF}"
podman tag "${REF}" "${IMAGE}:stable"
for svc in "${SERVICES[@]}"; do
log "Restarting ${svc}"
systemctl --user restart "${svc}.service"
done
log "Waiting up to ${HEALTH_TIMEOUT}s per service for healthy status"
for svc in "${SERVICES[@]}"; do
deadline=$(( $(date +%s) + HEALTH_TIMEOUT ))
while :; do
status=$(podman inspect --format '{{ .State.Health.Status }}' "${svc}" 2>/dev/null || echo "missing")
case "${status}" in
healthy)
log "${svc}: healthy"
break
;;
unhealthy)
log "ERROR: ${svc} reported unhealthy"
journalctl --user -u "${svc}.service" -n 100 --no-pager | logger -t mastosum-deploy
exit 1
;;
*)
if [ "$(date +%s)" -ge "${deadline}" ]; then
log "ERROR: ${svc} did not become healthy within ${HEALTH_TIMEOUT}s"
exit 1
fi
sleep 2
;;
esac
done
done
log "Deploy of ${TAG} succeeded"
A few things to note about this script:
The deploy trigger carries the release tag for logging and status reporting. The script intentionally pulls :stable, not the version tag. The immutable :v0.2.9 tag is the audit trail and rollback anchor; :stable is the deployment pointer. The workflow already pushed both tags to the same digest, so pulling :stable gets the same image. The trigger value (v0.2.9) appears in the deploy logs and the status file the workflow polls, but the pull target is always the moving tag.
cosign verification happens before the pull, and the pull targets a digest. The script resolves the current digest of :stable from the registry with skopeo inspect, verifies it against the root-installed public key, pulls by that immutable digest, and then tags it locally as :stable so the Quadlets find it. This closes the race window between verification and pull: the exact verified reference is what gets deployed. If verification fails, the script exits before any service is restarted. This is the active enforcement path today; once Forgejo’s registry supports OCI 1.1 referrers, the cosign verify call can be dropped in favour of policy.json enforcement at the podman pull level.
After a successful pull, services restart in sequence, and the script polls each container’s health check until it reports healthy or the timeout expires.
The whole bridge — workflow writes file, path unit fires, service runs script, workflow reads result — is a pattern I’ve come to appreciate for its simplicity. There’s no webhook receiver to secure, no API to authenticate, no network port to expose. The filesystem is the API.
Rollback
The immutable per-version tags make rollback straightforward:
# From any machine with push access:
podman pull git.linuxserver.pro/chris/mastosum:v0.2.8
podman tag git.linuxserver.pro/chris/mastosum:v0.2.8 git.linuxserver.pro/chris/mastosum:stable
podman push git.linuxserver.pro/chris/mastosum:stable
# Then trigger a deploy from the host:
ssh mastosum@prod 'echo v0.2.8 > ~/runner-data/deploy-trigger'
Move :stable back to the last known good version, write the trigger file, and the same path unit + deploy script takes over. The deploy script’s cosign verify step still works because the old version tag was also signed. No special rollback procedure, no database migration reversal (the app’s startup migrations are additive and forward-only by design). The Quadlets don’t change. The environment file doesn’t change. Only the image digest that :stable points to changes.
The Full Picture
Putting it all together, the flow from git push to healthy containers:
- You push a version tag.
git push origin v0.2.9. The tag is signed locally; the workflow is triggered by its presence. - Forgejo fires the workflow. The
mastosum-prodrunner picks it up. - The workflow builds. UBI10 base, Python venv, GIT_HASH baked in.
- The workflow pushes
:v0.2.9and:stableto the Forgejo registry. Both tags point to the same digest. - The workflow signs the digest with cosign. Private key is shredded after use. The signature lands as a
<digest>.sigtag and a Rekor transparency-log entry. - The workflow writes
v0.2.9to/home/mastosum/runner-data/deploy-triggerand polls for the status file. - systemd sees the trigger file.
mastosum-deploy.pathfiresmastosum-deploy.service. - The deploy script runs. It resolves the
:stabledigest, verifies the cosign signature against the root-installed public key, pulls the image by verified digest, tags it locally as:stable, restarts the four app services, and waits for each to report healthy. - The deploy script writes
ok v0.2.9to/home/mastosum/runner-data/deploy-status. - The workflow reads the status file, sees
ok, and exits 0.
No manual steps. No SSH from CI to host. No webhook receiver. No additional daemon. The CI runner, the deploy machinery, and the application itself all run as the same user, in the same Podman context, managed by the same systemd.
The pieces are simple individually. The value is in how they compose: a tag push triggers a signed build, which writes a file, which systemd reacts to, which verifies the signature and pulls and restarts. Each link does one thing, and the filesystem is the interface between them.
References
- Podman in Production: Quadlets, Secrets, Auto-Updates — the Podman architecture this deployment rests on
- Ansible-Native Quadlets: Deploying a Mastodon Greeter Bot — Quadlet generation and Podman secrets in Ansible
- Container Forgejo Runner with Podman Quadlet — the Quadlet pattern the runner uses
- Speeding Up Forgejo CI with a Custom OCI Image — Forgejo’s built-in OCI registry
- cosign documentation
- containers-policy.json(5)
Comments
You can use your Mastodon or other ActivityPub account to comment on this article by replying to the associated post.
Search for the copied link on your Mastodon instance to reply.
Loading comments...