Files

T

Alexandru Macocian d615872325 Cleanup docs after migration has finished

2026-05-06 18:10:33 +02:00

19 KiB

Raw Blame History

Automation

Reusable Gitea Actions that every Charlie/<stack> repo consumes. Lives at .gitea/ in this repo. Replaces the per-stack deploy.sh shell scripts that existed during the reorg.

Design context: strategy.md, secrets.md, runners.md.

Architecture

The runner is stateless. It registers with Gitea, picks up jobs, and SSHes into the target host using a short-lived step-ca-issued user cert to do the actual work as sys-gitea-runner (LDAP, member of charlie-deploy).

push ──► Gitea Actions ──► any runner picks up the job
                                │
                                │ step ssh certificate sys-gitea-runner ...   (5 min validity)
                                │ ssh -M sys-gitea-runner@<host>             (multiplex master)
                                ▼
                            on the target host:
                              cd /mnt/nas/stacks/<stack>
                              git fetch && git reset --hard
                              sops --decrypt …
                              docker compose pull && up -d
                              rm decrypted plaintext
                                ▲
                                │ ssh -O exit  (close multiplex)
                                │ wipe /tmp/charlie-ssh.*

This sidesteps the Gitea reusable-workflow label-routing bug (#32348) — any runner can deploy any stack to any host. The runner image (amacocian/gitea-runner) ships step-cli + openssh-client and a charlie-entrypoint.sh that runs step ca bootstrap at container start.

What's here

Kind	Path	Purpose
Reusable workflow	`.gitea/workflows/deploy-stack.yml`	The deployment task. Opens SSH session → per-host compose resolution + `git fetch && reset` + SOPS decrypt + `docker compose up -d` + cleanup, all on the target host.
Composite action	`.gitea/actions/charlie-ssh-open`	Mints a 5-min SSH user cert for `sys-gitea-runner`, opens an SSH ControlMaster session to the target. Exports `CHARLIE_SSH_*` env to subsequent steps.
Composite action	`.gitea/actions/charlie-ssh-target`	Runs a command on the target through the multiplex socket. Optional `workdir:` cd's into a directory first.
Composite action	`.gitea/actions/charlie-ssh-close`	Tears down the multiplex + wipes the ephemeral cert. Idempotent; intended for `if: always()`.

The three charlie-ssh-* actions are also usable directly for ad-hoc SSH-from-CI patterns (e.g. one-off maintenance tasks, status checks). For deploys, use the higher-level deploy-stack.yml workflow.

How Gitea handles cross-repo references

Gitea Actions implements GitHub's reusable-workflow + composite-action contract:

Reusable workflows (on: workflow_call) are referenced from a caller's jobs.<id>.uses: as <owner>/<repo>/.gitea/workflows/<file>.yml@<ref>.
Composite actions (runs.using: composite) are referenced from a step's uses: as <owner>/<repo>/<path-to-action-dir>@<ref>.

In our setup both forms must be absolute URLs with auth because of Gitea bugs and instance-wide REQUIRE_SIGNIN_VIEW. See Gitea quirks.

Caller patterns

Every uses: URL has to be the absolute form with the sys-gitea-runner PAT prefix. The relative form makes act default to github.com; the unauthenticated absolute form fails on auth. See Gitea quirks below for the full story.

Single-host stack, with secrets

# Charlie/<stack>/.gitea/workflows/deploy.yaml
name: Deploy
on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  deploy:
    uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/workflows/deploy-stack.yml@main
    with:
      host: morgott
      decrypt: |
        secrets/env.sops.yaml -> .env
    secrets: inherit

RUNNER_REPO_PAT is set as a Charlie org-level secret (Settings → Actions → Secrets); every stack repo inherits it via secrets: inherit. The PAT belongs to the LDAP-backed sys-gitea-runner service account with read on the Charlie org.

Single-host stack, no secrets

Drop the decrypt: input. secrets: inherit is still required so the called workflow can clone its sibling actions.

jobs:
  deploy:
    uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/workflows/deploy-stack.yml@main
    with:
      host: morgott
    secrets: inherit

Multiple secrets / non-`.env` formats

with:
  host: morgott
  decrypt: |
    secrets/env.sops.yaml         -> .env
    secrets/config.sops.yaml      -> config/app.yaml
    secrets/credentials.sops.json -> credentials.json

Output format is inferred from the destination extension (.env → dotenv, .json → json, .yaml/.yml → yaml, .ini → ini, anything else treated as raw bytes).

Shared (multi-host) stack

jobs:
  deploy:
    strategy:
      matrix:
        host: [morgott, melina, mohg, miquella]
    uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/workflows/deploy-stack.yml@main
    with:
      host: ${{ matrix.host }}
      decrypt: |
        secrets/${{ matrix.host }}.env.sops.yaml -> envs/${{ matrix.host }}.env
    secrets: inherit

deploy-stack.yml resolves compose files per strategy.md → Resolution order:

docker-compose.<host>.yml if it exists, only that.
Otherwise docker-compose.yml + (if present) overrides/<host>.yml.
If envs/<host>.env exists, passed as --env-file.

Custom stack path

By default the workflow operates on /mnt/nas/stacks/<repo-name>/ on the target host. Override for repos that don't follow the convention:

with:
  host: morgott
  stack-path: /mnt/nas/stacks/cert-issuer

Direct SSH for non-deploy jobs

If you need to run something on a host outside the deploy lifecycle (a status check, a one-off migration, etc.), use the charlie-ssh-* actions directly:

jobs:
  smoke-test:
    runs-on: self-hosted
    steps:
      - uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/actions/charlie-ssh-open@main
        with:
          host: morgott
      - uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/actions/charlie-ssh-target@main
        with:
          run: |
            curl -fsS https://homarr.alexandru.macocian.me/api/health
      - if: always()
        uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/actions/charlie-ssh-close@main

Inputs reference — `deploy-stack.yml`

Input	Required	Default	Description
`host`	yes	—	Hostname the workflow SSHes into. Resolves via the host's normal LAN DNS (Marika).
`stack-path`	no	`/mnt/nas/stacks/<repo-name>`	Absolute working-tree path on the target host. Must already be a `git` repo.
`decrypt`	no	`""`	Multi-line `src -> dst` SOPS decrypt list. Skipped if empty. Decryption runs on the target host using its `/etc/charlie/age.key`.
`compose-args`	no	`""`	Extra args appended to `docker compose ... up -d`. Use sparingly.
`pull`	no	`true`	Run `docker compose ... pull` before `up -d`.
`ref`	no	The branch the workflow ran on	Git ref to check out into the working tree on the target host.

Requires the RUNNER_REPO_PAT secret to be in scope (set at org level; callers pass via secrets: inherit).

Inputs reference — `charlie-ssh-open`

Input	Required	Default	Description
`host`	yes	—	Hostname to SSH into. LAN-resolvable.
`user`	no	`sys-gitea-runner`	Remote user.
`duration`	no	`5m`	Cert validity. Capped by the `runners` provisioner config in step-ca (currently 10m max).

Exports to $GITHUB_ENV so subsequent steps can use them:

Variable	Meaning
`CHARLIE_SSH_HOST`	The target host.
`CHARLIE_SSH_USER`	The remote user.
`CHARLIE_SSH_DIR`	Tmpdir holding the ephemeral key + cert.
`CHARLIE_SSH_SOCKET`	Unix socket for the SSH multiplex master.

Inputs reference — `charlie-ssh-target`

Input	Required	Default	Description
`run`	yes	—	Shell command(s) to run on the target. Multi-line allowed; runs as `bash -lc <command>`.
`workdir`	no	`""`	Optional directory to cd into before running.

Reads CHARLIE_SSH_* from job env (must come after charlie-ssh-open).

Inputs reference — `charlie-ssh-close`

No inputs. Reads CHARLIE_SSH_* from job env. Idempotent. Always end your job with this in an if: always() step.

Host prerequisites

Every host that wants to be a deploy target needs:

Requirement	Why
`sys-gitea-runner` resolves via LDAP/SSSD; `charlie-deploy` group exists with `sys-gitea-runner` as a member	Runner SSHes in as this user; group grants docker + stacks access. See onboarding.md → Charlie deploy group.
sshd trusts the step-ca SSH user CA (`/etc/ssh/trusted_user_ca.pub` populated by `sync-ca.sh`)	Runner-issued user certs are accepted by sshd. See onboarding.md → Trusted CA.
`/etc/charlie/age.key` (`0640`, `root:charlie-deploy`)	SOPS decrypt by `sys-gitea-runner`. See onboarding.md → Host age key (SOPS).
`/var/run/docker.sock` is `root:charlie-deploy`	`docker compose` works without sudo. See onboarding.md → Charlie deploy group.
`/mnt/nas/stacks/<stack>/.git` exists, owned `sys-gitea-runner:charlie-deploy 0775+s`	The deploy step does `git fetch && git reset --hard`, not a fresh clone.

The runner host itself only needs the runner stack (and step-ca trust if you want to run step ssh login interactively as a human). The runner container reaches step-ca via network_mode: host and the host's resolver.

Manual fallback (no runner available)

The deploy workflow is the happy path. When you need to push by hand — runner outage, gitea-on-melina (which can't be deployed by a runner that depends on it), first-time bring-up before runners exist, or debugging — SSH into the target host yourself and run the equivalent commands. SOPS is offline crypto; nothing else in the fleet has to be up.

Stack with secrets

# SSH in via gssh as your normal LDAP user (alex/amacocian).
ssh morgott

# Become sys-gitea-runner. -H sets HOME so credential helpers work.
sudo -u sys-gitea-runner -H bash

cd /mnt/nas/stacks/<stack>
git fetch --prune origin
git reset --hard origin/main

HOST=$(hostname -s | tr '[:upper:]' '[:lower:]')

# Decrypt every secret the workflow would have decrypted. Mirror the
# `decrypt:` block in .gitea/workflows/deploy.yaml.
SOPS_AGE_KEY_FILE=/etc/charlie/age.key \
  sops --decrypt --output-type dotenv --output .env secrets/env.sops.yaml
chmod 600 .env

# Pick the right compose files (mirrors deploy-stack.yml resolution).
if [ -f "docker-compose.$HOST.yml" ]; then
  COMPOSE_FILES=(-f "docker-compose.$HOST.yml")
else
  COMPOSE_FILES=(-f docker-compose.yml)
  [ -f "overrides/$HOST.yml" ] && COMPOSE_FILES+=(-f "overrides/$HOST.yml")
fi
ENVFILE=()
[ -f "envs/$HOST.env" ] && ENVFILE=(--env-file "envs/$HOST.env")

docker compose "${ENVFILE[@]}" "${COMPOSE_FILES[@]}" pull
docker compose "${ENVFILE[@]}" "${COMPOSE_FILES[@]}" up -d

# Clean up plaintext.
rm -f .env
exit  # leave sys-gitea-runner shell

Stack without secrets

Same as above, minus the decrypt and the rm -f .env lines.

Gitea quirks (leave them here for the next time this bites)

Hard-won lessons from getting the first end-to-end pipeline working. Worth re-reading before debugging anything in this area.

Cross-repo `uses:` needs absolute URL with PAT

The relative form Charlie/project-charlie/...@main makes act default to GitHub. The absolute form without auth fails because Gitea has REQUIRE_SIGNIN_VIEW=true instance-wide. So every uses: referencing a sibling repo has to be:

uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/<repo>/...@<ref>

This applies transitively — inside deploy-stack.yml, the uses: for the charlie-ssh-* composite actions also have to be absolute-URL form, otherwise act tries github.com from inside the workflow.

The PAT belongs to the LDAP-backed sys-gitea-runner service account (LDAP → synced into Gitea via Authentik OAuth → placed in a runners team in the Charlie org with read on all repos). PAT itself is set as a Charlie org-level secret so every stack repo inherits it. Callers must include secrets: inherit so the called workflow sees the secret.

References:

https://github.com/go-gitea/gitea/issues/25929 — the canonical issue
https://github.com/go-gitea/gitea/pull/36173 — the fix that landed in Gitea 1.27 (we pre-date it; the PAT-in-URL workaround is needed)

Reusable-workflow `runs-on` label matching is broken

runs-on: [self-hosted, morgott] inside a reusable workflow does not correctly route to a runner that advertises both labels. Gitea's matching either picks any runner with self-hosted (loose match) or hangs. This is #32348 — open, type/bug. PR #36388 was attempted, closed; replaced by #37478, still WIP at time of writing.

We sidestep it entirely by using runs-on: self-hosted and SSHing to the target host inside the job. Any runner can deploy any stack.

Documented exception: Charlie/runners. The runners stack genuinely needs precise routing (cross-deploy: each host's runner deploys the other host so no runner ever recreates itself mid-job). Top-level runs-on: in a regular workflow does respect labels — the bug is reusable-workflow-specific — so Charlie/runners ships a hand-rolled .gitea/workflows/deploy.yaml with two pinned jobs instead of calling deploy-stack.yml. When #32348 is fixed, collapse it into a matrix: { host: [morgott, melina] } calling the shared workflow. Background: runners.md → Updates.

`act` aggressively caches reusable workflows / composite actions

After pushing a fix to Charlie/project-charlie, the runner still uses the old version because act caches at /root/.cache/act/Charlie-project-charlie@main. The cache key is the ref string, not the commit. To force a refresh after pushing a fix:

sudo docker exec runner rm -rf /root/.cache/act

Long-term fix: stop using @main for cross-repo refs. Pin to specific tags (@v1, @<sha>). New ref string → new cache entry → automatic refresh. Left for later (until the contract stabilises).

LAN DNS inside the runner container

Default Docker DNS (127.0.0.11 embedded resolver) doesn't know about .lan hostnames. The runner stack uses network_mode: host so the container shares the host's resolver and can reach morgott.lan, cert-issuer.lan (well, that one's docker-internal — morgott.lan is the right name), etc.

Note about cert-issuer.lan

cert-issuer.lan is the docker-network alias for the step-ca container, only resolvable from inside the cert-issuer's docker network. From anywhere else (including the runner container with network_mode: host), use morgott.lan:9000 (cert-issuer is pinned to morgott — see cert-issuer.md).

Workflow logs collapse

The Gitea Actions UI shows everything happening in Set up job rather than separate cards per step. Cosmetic; doesn't reflect the actual step structure. The full chronological log inside Set up job is what to scroll through. Suspected to be a Gitea / act_runner UI bug; not impacting correctness.

`insufficient permission for adding an object to repository database .git/objects`

Symptom: deploy step "Sync working tree" fails with the above git error on git fetch against /mnt/nas/stacks/<stack>/.

Cause: the stack's .git/ is owned by root because the working tree was bootstrapped manually with sudo git clone ... instead of going through the deploy workflow (or onboarding step 5 in onboarding.md → Charlie deploy group). The deploy workflow runs as sys-gitea-runner over SSH and can't write into root-owned objects.

Fix once on the affected host:

sudo chown -R sys-gitea-runner:charlie-deploy /mnt/nas/stacks/<stack>
sudo find /mnt/nas/stacks/<stack> -type d -exec chmod 2775 {} \;
sudo find /mnt/nas/stacks/<stack> -type f -exec chmod g+rw {} \;

Then re-trigger the workflow. To avoid re-introducing this for new stacks, do the manual bootstrap clone as sys-gitea-runner:

sudo -u sys-gitea-runner -H git clone <url> /mnt/nas/stacks/<stack>

`strategy.max-parallel: 1` doesn't actually serialize

Symptom: shared-stack deploy lands on host A and host B in parallel; host A's if: always() cleanup rms the decrypted .env while host B's compose up is still running, producing couldn't read env file. Or: two parallel git fetch && reset --hard against the same NFS-shared .git race on lock files.

Cause: Gitea Actions accepts strategy.max-parallel: syntactically but does not reliably honor it for matrix jobs that call a reusable workflow. Both jobs get queued and pick up runners simultaneously.

Fix: don't use the matrix shape for shared stacks. Spell each host out as a separate job, chained with needs: for true sequential execution. Same shape we use in Charlie/runners and Charlie/otelcollector.

jobs:
  deploy-morgott:
    uses: ... deploy-stack.yml@main
    with: { host: morgott, ... }
    secrets: inherit
  deploy-melina:
    needs: deploy-morgott
    uses: ... deploy-stack.yml@main
    with: { host: melina, ... }
    secrets: inherit

Long-term cleanup: have deploy-stack.yml decrypt to a per-host filename (e.g. .env.<host> or /run/charlie-secrets/<stack>/.env) and pass it via --env-file. Then matrix-with-true-parallelism becomes safe, since neither the working tree nor the env file collide. Until then, needs: is the working pattern.

Adding new reusable parts

If you find yourself copying logic across stack workflows, pull it in here. Two flavors:

Composite action at .gitea/actions/<name>/action.yml — small, single step's worth of logic. Cheap to call from inside any workflow.
Reusable workflow at .gitea/workflows/<name>.yml — owns the whole job (runner selection, sequencing, multiple steps).

Pin callers to @main for now; cut @v1 etc. once the contract is stable. Breaking changes to inputs need a major bump.

19 KiB Raw Blame History