Runners

Design for Charlie/runners — the Gitea Actions runner stack. Pairs with the runner image at amacocian/gitea-runner and the reusable workflow + composite actions in .gitea/ (see automation.md).

Status: deployed. One runner each on morgott + melina, registered instance-wide.

Goal

Self-hosted Gitea Actions runners that can deploy any Charlie/<stack> to any host. The runner is stateless — it holds only its own registration credential. All real work (git, sops, docker compose) happens on the target host via SSH using a short-lived step-ca-issued user cert.

Two runners gives capacity + cross-deploy capability. Adding mohg/miquella runners later is a capacity decision; no design change needed.

Image

amacocian/gitea-runner — upstream gitea/act_runner plus:

Layered on top	Why
`git`, `sops`, `docker-cli`, `docker-cli-compose`	Workflow steps run inside this container; need these tools available locally for `act` and remotely over ssh
`bash`, `sudo`, `jq`	wrapper scripts
`step-cli`	mints SSH user certs from step-ca's `runners` JWK provisioner at job time
`openssh-client`	`charlie-ssh-*` wrappers in `/usr/local/bin/`
`charlie-entrypoint.sh`	at container start: bootstraps step-ca trust + materialises git credentials from `$GITEA_PAT`

Built locally, pushed manually. Bump recipe.

Stack layout

Charlie/runners/
├── docker-compose.yml             # network_mode: host
├── envs/.gitkeep                  # decrypted at deploy, gitignored
├── secrets/
│   ├── morgott.env.sops.yaml
│   └── melina.env.sops.yaml
├── .sops.yaml
├── .env.example
└── .gitea/workflows/deploy.yaml   # cross-deploy; see "Updates"

State lives host-local at /var/lib/charlie/runner/ (just the .runner registration credential). NFS + per-runner identity = bad fit for shared storage.

Encrypted env contents are documented in Charlie/runners/.env.example.

Bootstrap (per host, one-time)

Recipe lives with the stack: Charlie/runners/README.md → Bootstrap.

The first-ever runner needs manual bring-up because there's no runner to deploy it. Subsequent runners can be bootstrapped manually too (simpler) or you can land the encrypted env first and let the existing runner deploy the new one.

Updates (image bump or env change)

Automatic on push to Charlie/runners. Runner updates are the one stack that does not call the shared deploy-stack.yml — instead it ships its own .gitea/workflows/deploy.yaml with two cross-deploy jobs:

Job	`runs-on`	Target
`deploy-morgott`	`[self-hosted, melina]`	morgott
`deploy-melina`	`[self-hosted, morgott]` (after `deploy-morgott`)	melina

No runner ever recreates itself mid-job, so the deploy is a normal synchronous compose pull && up -d with real exit codes.

Why the duplication

Gitea bug #32348: runs-on: label routing is broken inside reusable workflows but works from a top-level workflow. Cross-deploy needs precise routing, so the reusable workflow can't be used here. Full bug write-up: automation.md → Gitea quirks.

When the upstream fix lands (PRs #36388 / #37478), collapse the runners workflow to a matrix: { host: [morgott, melina] } calling deploy-stack.yml.

Image bumps

Build + push the new image tag.
Commit + push Charlie/runners with the new image: in docker-compose.yml.
Cross-deploy workflow runs automatically.

Manual fallback

If both runners are down, neither cross-deploy job has anywhere to land. Use the Bootstrap recipe (it's the same recipe, just done by hand).

Permissions model

Identity	Where used	Purpose
`sys-gitea-runner` (LDAP)	`~sys-gitea-runner/.git-credentials` (host); SSH user via cert (target host); `RUNNER_REPO_PAT` org secret	THE deployment identity. Member of `charlie-deploy`. Read-only on `Charlie` org.
`sys-host-<host>` (LDAP)	`/root/.git-credentials` + `/root/.docker/config.json` (host)	Root-level operations on each host: `sudo git ...`, `sudo docker pull`. Read-only on `Charlie` org.

The act_runner registration credential is its own identity in Gitea. It can pick up jobs and report results; it cannot read other repos. When it needs to clone for act's uses: resolution, it uses the RUNNER_REPO_PAT materialised by the entrypoint shim.

Open / future

step ca bootstrap re-pin on cert rotation. If step-ca's root fingerprint rotates, every runner needs STEPCA_FINGERPRINT updated. The entrypoint runs step ca bootstrap --force so an updated fingerprint just needs the new value in the encrypted env.
SSH host certs. sshd should present a step-ca-issued host cert; the runner should pin @cert-authority *.lan ... rather than TOFU. Tracked in cert-issuer.md.
3+ runners. Currently 2. Adding a third (mohg/miquella) breaks the "deploy from the other host" trick because there's no longer exactly one other host. Likely answer: designate two "deployer" runners. Defer until #32348 is fixed; the matrix-on-shared-workflow shape may make the question moot.
Tag pinning. Caller workflows pin to @main. After every push to this repo or the runner image, busting act cache is necessary (sudo docker exec runner rm -rf /root/.cache/act). Switch to @v1 when the contract stabilises.

6.1 KiB Raw Blame History