project-charlie/docs/runners.md

Runners

Design for Charlie/runners — the Gitea Actions runner stack. Pairs with the runner image at amacocian/gitea-runner and the reusable workflow + composite actions in this repo's .gitea/ (see automation.md).

Status: deployed. One runner each on morgott + melina, registered instance-wide. Idle and accepting jobs.

Goal

Self-hosted Gitea Actions runners that can deploy any Charlie/<stack> to any host in the fleet. The runner itself is stateless — it holds only its own registration credential. All real work (git, sops, docker compose) happens on the target host via SSH using a short-lived step-ca-issued user cert.

Why two

• Capacity / availability. With a single runner, a long-running job blocks all other deploys. Two runners can pick up two jobs in parallel.
• No more "this runner can only deploy that host". Earlier designs required one runner per target host because each runner mounted that host's docker socket + NFS + age key. The SSH-into-host model removes that coupling — any runner deploys any host. We add more runners only for capacity, not for new hosts.

morgott + melina is what we've stood up. Adding mohg/miquella runners later is purely a capacity decision (no design changes needed).

Image

amacocian/gitea-runner — upstream gitea/act_runner plus:

Layered on top	Why
git	git fetch && git reset --hard on the target host (we ship git in the image so wrappers can run remote git via ssh too)
sops	SOPS decrypt on the target host
docker-cli + docker-cli-compose	docker compose on the target host
bash, sudo, jq	wrapper scripts
step-cli	mints SSH user certs from step-ca's runners JWK provisioner at job time
openssh-client	the charlie-ssh-* wrappers in /usr/local/bin/
charlie-entrypoint.sh	at container start: bootstraps step-ca trust + materialises git credentials from $GITEA_PAT for act cross-repo uses:

Built locally, pushed manually to the gitea registry. Bump recipe.

Stack — Charlie/runners

Charlie/runners/
├── docker-compose.yml             # network_mode: host; pulls amacocian/gitea-runner
├── envs/
│   └── .gitkeep                   # decrypted at deploy time, gitignored
├── secrets/
│   ├── morgott.env.sops.yaml      # encrypted (morgott + admin recipients)
│   └── melina.env.sops.yaml       # encrypted (melina + admin recipients)
├── .sops.yaml                     # path-regex creation rules per host
├── .env.example                   # documents the env shape
├── .gitignore
└── README.md

The encrypted <host>.env.sops.yaml files contain:

GITEA_INSTANCE_URL=https://gitea.alexandru.macocian.me
GITEA_RUNNER_NAME=<host>
GITEA_RUNNER_LABELS=self-hosted,<host>
GITEA_RUNNER_REGISTRATION_TOKEN=...   # one-time, from Gitea UI
GITEA_PAT=...                         # sys-gitea-runner read-only PAT
STEPCA_URL=https://morgott.lan:9000
STEPCA_FINGERPRINT=...                # captured at step-ca init
STEPCA_PROVISIONER=runners
STEPCA_PROVISIONER_PASSWORD=...       # from vaultwarden

State lives host-local at /var/lib/charlie/runner/ (just the .runner registration credential). NFS + many small files = bad time, and .runner is per-runner identity so it must not be shared across hosts.

Bootstrap (one-time, per host)

The chicken-and-egg situation we worried about earlier turned out smaller than expected. The runner image is in the registry; the deploy workflow exists; runners can deploy themselves through other runners (any-runner- to-any-host). The first-ever runner does need a manual bring-up though because there's no runner to deploy it.

Per host, one-time:

1. Mint a registration token in Gitea: site admin → Actions → Runners. Same token works for both runners (it's instance-wide and single-use semantics don't apply here as long as you reset only when you mean to).

2. Encrypt the host's env on the admin device, in the runners repo:

   cd ~/Dev/runners
   HOST=morgott   # or melina
   
   tmp=$(mktemp)
   cat > "$tmp" <<EOF
   GITEA_INSTANCE_URL=https://gitea.alexandru.macocian.me
   GITEA_RUNNER_NAME=$HOST
   GITEA_RUNNER_LABELS=self-hosted,$HOST
   GITEA_RUNNER_REGISTRATION_TOKEN=<paste-from-gitea-ui>
   GITEA_PAT=<sys-gitea-runner Gitea Repos PAT from vaultwarden>
   STEPCA_URL=https://morgott.lan:9000
   STEPCA_FINGERPRINT=<captured-at-step-ca-init>
   STEPCA_PROVISIONER=runners
   STEPCA_PROVISIONER_PASSWORD=<from vaultwarden: Charlie cert-issuer runners JWK provisioner password>
   EOF
   
   sops --encrypt --input-type dotenv --output-type yaml \
        --filename-override "secrets/$HOST.env.sops.yaml" "$tmp" > "secrets/$HOST.env.sops.yaml"
   shred -u "$tmp"
   
   git add secrets/$HOST.env.sops.yaml
   git commit -m "Add encrypted env for $HOST runner"
   git push
   

3. On the host, decrypt + bring up. Same shape as automation.md → Manual fallback, but for the runner stack itself:

   cd /mnt/nas/stacks/runners
   sudo git fetch --prune origin
   sudo git reset --hard origin/main
   
   HOST=$(hostname -s | tr '[:upper:]' '[:lower:]')
   sudo install -d -m 0700 /var/lib/charlie/runner
   
   sudo SOPS_AGE_KEY_FILE=/etc/charlie/age.key \
     sops --decrypt --output-type dotenv \
          --output "envs/$HOST.env" "secrets/$HOST.env.sops.yaml"
   sudo chmod 600 "envs/$HOST.env"
   
   sudo docker compose --env-file "envs/$HOST.env" pull
   sudo docker compose --env-file "envs/$HOST.env" up -d
   sudo docker compose --env-file "envs/$HOST.env" logs runner | tail -10
   #  ...look for all three [charlie-entrypoint] lines:
   #     git auth configured for sys-gitea-runner@gitea.alexandru.macocian.me
   #     step-ca trust bootstrapped: https://morgott.lan:9000
   #     step-ca provisioner password stashed
   sudo rm -f "envs/$HOST.env"
   

4. Verify in Gitea: Site Administration → Actions → Runners. The new entry should show Idle with the labels you set.

5. Repeat steps 1–4 on the next host.

Updates (image bump or env change)

Once both runners exist, runner updates flow through the deploy workflow like any other stack. Push to Charlie/runners, the workflow runs on one of the runners, SSHes into the target host, recreates that host's runner container.

The "runner deploys itself" issue is real: when the workflow is recreating the runner container on a host, that host's runner is briefly down. But:

• Compose's --force-recreate brings the new container up before the old one fully stops.
• If the runner that picked up the job IS the one being recreated, the job dies mid-way (the container hosting act_runner goes away). To avoid this, manually trigger updates with workflow_dispatch and pick the runner explicitly via the dispatch UI's runner selector — or just do the update manually with the recipe above.

When the runner image itself bumps (amacocian/gitea-runner), order:

1. Build + push the new image tag.
2. Commit + push Charlie/runners with the new image: in docker-compose.yml.
3. Wait for both runners to update (or do steps 3 of bootstrap by hand on each host).

Permissions model

Identity	Where used	Purpose
sys-gitea-runner (LDAP)	~sys-gitea-runner/.git-credentials (host); SSH user via cert (target host); RUNNER_REPO_PAT org secret	THE deployment identity. Member of charlie-deploy. Read-only on Charlie org.
sys-host-<host> (LDAP)	/root/.git-credentials + /root/.docker/config.json (host)	Root-level operations on each host: sudo git ..., sudo docker pull. Read-only on Charlie org.

The runner's act_runner registration credential is its own identity in Gitea (issued by the Gitea API at registration). It can pick up jobs and report results; it cannot read other repos. When it needs to clone something for act's uses: resolution, it uses the RUNNER_REPO_PAT materialised by the entrypoint shim.

Open / future

• step ca bootstrap re-pin on cert rotation. If step-ca's root fingerprint rotates, every runner needs STEPCA_FINGERPRINT updated. The entrypoint runs step ca bootstrap --force so an updated fingerprint just needs the new value in the encrypted env file.
• SSH host certs. sshd on each host should present a step-ca-issued host cert; the runner should pin @cert-authority *.lan ... rather than TOFU each host. Tracked in cert-issuer.md → Open question 3.
• More runners. Currently 2. Add a third only when current capacity is the bottleneck (deploy queue grows). Adding to mohg/miquella is three commands: encrypt env, push, bootstrap on the new host.
• Tag pinning. Caller workflows pin to @main. Switch to @v1 semver when the contract stabilises. Until then, after every push to this repo or the runner image, busting act cache on existing runners is necessary (sudo docker exec runner rm -rf /root/.cache/act).