project-charlie/docs/automation.md

Automation

Reusable Gitea Actions that every Charlie/<stack> repo consumes. Lives at .gitea/ in this repo. Replaces the per-stack deploy.sh shell scripts that existed during the reorg.

Design context: strategy.md, secrets.md, runners.md.

Architecture

The runner is stateless. It registers with Gitea, picks up jobs, and SSHes into the target host using a short-lived step-ca-issued user cert to do the actual work as sys-gitea-runner (LDAP, member of charlie-deploy).

push ──► Gitea Actions ──► any runner picks up the job
                                │
                                │ step ssh certificate sys-gitea-runner ...   (5 min validity)
                                │ ssh -M sys-gitea-runner@<host>             (multiplex master)
                                ▼
                            on the target host:
                              cd /mnt/nas/stacks/<stack>
                              git fetch && git reset --hard
                              sops --decrypt …
                              docker compose pull && up -d
                              rm decrypted plaintext
                                ▲
                                │ ssh -O exit  (close multiplex)
                                │ wipe /tmp/charlie-ssh.*

This sidesteps the Gitea reusable-workflow label-routing bug (#32348) — any runner can deploy any stack to any host. The runner image (amacocian/gitea-runner) ships step-cli + openssh-client and a charlie-entrypoint.sh that runs step ca bootstrap at container start.

What's here

Kind	Path	Purpose
Reusable workflow	.gitea/workflows/deploy-stack.yml	The deployment task. Opens SSH session → per-host compose resolution + git fetch && reset + SOPS decrypt + docker compose up -d + cleanup, all on the target host.
Reusable workflow	.gitea/workflows/build-image.yml	Build + push a Docker image from a source repo (amacocian/<name>) to the Gitea container registry. Tags with latest and a UTC timestamp (YYYY-MM-DDTHHMMSSZ); deployment repos pin the timestamp tag to roll out.
Composite action	.gitea/actions/charlie-ssh-open	Mints a 5-min SSH user cert for sys-gitea-runner, opens an SSH ControlMaster session to the target. Exports CHARLIE_SSH_* env to subsequent steps.
Composite action	.gitea/actions/charlie-ssh-target	Runs a command on the target through the multiplex socket. Optional workdir: cd's into a directory first.
Composite action	.gitea/actions/charlie-ssh-close	Tears down the multiplex + wipes the ephemeral cert. Idempotent; intended for if: always().

The three charlie-ssh-* actions are also usable directly for ad-hoc SSH-from-CI patterns (e.g. one-off maintenance tasks, status checks). For deploys, use the higher-level deploy-stack.yml workflow.

How Gitea handles cross-repo references

Gitea Actions implements GitHub's reusable-workflow + composite-action contract:

• Reusable workflows (on: workflow_call) are referenced from a caller's jobs.<id>.uses: as <owner>/<repo>/.gitea/workflows/<file>.yml@<ref>.
• Composite actions (runs.using: composite) are referenced from a step's uses: as <owner>/<repo>/<path-to-action-dir>@<ref>.

In our setup both forms must be absolute URLs with auth because of Gitea bugs and instance-wide REQUIRE_SIGNIN_VIEW. See Gitea quirks.

Caller patterns

Every uses: URL has to be the absolute form with the sys-gitea-runner PAT prefix. The relative form makes act default to github.com; the unauthenticated absolute form fails on auth. See Gitea quirks below for the full story.

Single-host stack, with secrets

# Charlie/<stack>/.gitea/workflows/deploy.yaml
name: Deploy
on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  deploy:
    uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/workflows/deploy-stack.yml@main
    with:
      host: morgott
      decrypt: |
        secrets/env.sops.yaml -> .env
    secrets: inherit

RUNNER_REPO_PAT is set as a Charlie org-level secret (Settings → Actions → Secrets); every stack repo inherits it via secrets: inherit. The PAT belongs to the LDAP-backed sys-gitea-runner service account with read on the Charlie org.

Single-host stack, no secrets

Drop the decrypt: input. secrets: inherit is still required so the called workflow can clone its sibling actions.

jobs:
  deploy:
    uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/workflows/deploy-stack.yml@main
    with:
      host: morgott
    secrets: inherit

Multiple secrets / non-.env formats

with:
  host: morgott
  decrypt: |
    secrets/env.sops.yaml         -> .env
    secrets/config.sops.yaml      -> config/app.yaml
    secrets/credentials.sops.json -> credentials.json

Output format is inferred from the destination extension (.env → dotenv, .json → json, .yaml/.yml → yaml, .ini → ini, anything else treated as raw bytes).

Shared (multi-host) stack

jobs:
  deploy:
    strategy:
      matrix:
        host: [morgott, melina, mohg, miquella]
    uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/workflows/deploy-stack.yml@main
    with:
      host: ${{ matrix.host }}
      decrypt: |
        secrets/${{ matrix.host }}.env.sops.yaml -> envs/${{ matrix.host }}.env
    secrets: inherit

deploy-stack.yml resolves compose files per strategy.md → Resolution order:

1. docker-compose.<host>.yml if it exists, only that.
2. Otherwise docker-compose.yml + (if present) overrides/<host>.yml.
3. If envs/<host>.env exists, passed as --env-file.

Custom stack path

By default the workflow operates on /mnt/nas/stacks/<repo-name>/ on the target host. Override for repos that don't follow the convention:

with:
  host: morgott
  stack-path: /mnt/nas/stacks/cert-issuer

Direct SSH for non-deploy jobs

If you need to run something on a host outside the deploy lifecycle (a status check, a one-off migration, etc.), use the charlie-ssh-* actions directly:

jobs:
  smoke-test:
    runs-on: self-hosted
    steps:
      - uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/actions/charlie-ssh-open@main
        with:
          host: morgott
      - uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/actions/charlie-ssh-target@main
        with:
          run: |
            curl -fsS https://homarr.alexandru.macocian.me/api/health
      - if: always()
        uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/actions/charlie-ssh-close@main

Inputs reference — deploy-stack.yml

Input	Required	Default	Description
host	yes	—	Hostname the workflow SSHes into. Resolves via the host's normal LAN DNS (Marika).
stack-path	no	/mnt/nas/stacks/<repo-name>	Absolute working-tree path on the target host. Must already be a git repo.
decrypt	no	""	Multi-line src -> dst SOPS decrypt list. Skipped if empty. Decryption runs on the target host using its /etc/charlie/age.key.
compose-args	no	""	Extra args appended to docker compose ... up -d. Use sparingly.
pull	no	true	Run docker compose ... pull before up -d.
ref	no	The branch the workflow ran on	Git ref to check out into the working tree on the target host.

Requires the RUNNER_REPO_PAT secret to be in scope (set at org level; callers pass via secrets: inherit).

Registry login (built-in)

When pull == true (the default), deploy-stack.yml runs a fresh docker login gitea.alexandru.macocian.me on the target host immediately before docker compose pull, using a dedicated read-only PAT for sys-gitea-runner (scope: read:package). The PAT lives as SOPS ciphertext inlined in deploy-stack.yml, encrypted to every host's age key; decryption happens on the target with /etc/charlie/age.key.

This is independent of ~/.docker/config.json — necessary because build-image.yml's cleanup wipes the RW caller PAT after each build, which would otherwise leave the next deploy 401ing on pull. The cleanup step at the end of every deploy-stack run also docker logouts unconditionally (if: always()) and scrubs any leftover ciphertext under /tmp/charlie-registry-creds.*.sops.yaml, so the PAT never persists on disk regardless of where the workflow failed.

Operator-facing details (encrypt recipe, paste location, rotation, host add): secrets.md → Registry credentials (read-only PAT).

Callers don't pass anything for this — no input, no secret. It's an internal implementation detail of deploy-stack.yml.

Inputs reference — build-image.yml

Source repos use this workflow to build and push their container image to the Gitea registry. Deployment repos (Charlie/<stack>) then pin the timestamp tag in their compose file and trigger deploy-stack.yml to roll out.

Same SSH-into-host architecture as deploy-stack.yml: the runner is stateless; docker build / docker push execute on the build host's daemon as sys-gitea-runner over a short-lived step-ca-issued cert. The runner container deliberately does not have a docker socket bind mount.

Input	Required	Default	Description
host	yes	—	Build host the runner SSHes into. Must have docker + the standard charlie-deploy setup. Any deploy-target host qualifies.
image	no	gitea.alexandru.macocian.me/<github.repository> (lowercased)	Fully-qualified image name without the tag. Override only if the image should not live at its repo's natural path.
context	no	.	Docker build context, relative to the caller repo root.
dockerfile	no	Dockerfile	Dockerfile path, relative to context.
build-args	no	""	Multi-line KEY=VALUE list. Empty lines and # comments ignored.
extra-tags	no	""	Additional tags (one per line) pushed alongside latest + the timestamp. Just the tag, not the full image:tag form.
push	no	true	If false, build only — useful for PR validation.
ref	no	The branch the workflow ran on	Git ref to check out and build.
registry-user	no	${{ github.repository_owner }}	Username for docker login. Must match the user that owns CALLER_REGISTRY_PAT and have write:package on the target image namespace.

Every successful build pushes two tags:

Tag	Mutability	Purpose
latest	mutable	Convenience for local pulls and one-off docker run. Never reference from a stack compose file — there's no way for deploy-stack.yml to know a new digest landed.
YYYY-MM-DDTHHMMSSZ (UTC)	immutable	The tag deployment repos pin. Bumping it in docker-compose.yml and pushing the stack triggers a redeploy. Sortable lexicographically.

Outputs (consumable by downstream jobs in the caller workflow):

Output	Example	Use
image	gitea.alexandru.macocian.me/<owner>/<repo>	The image ref without the tag.
timestamp-tag	2026-05-22T130927Z	The tag the caller can write into the consumer stack's compose file (e.g. via a follow-up job that opens a PR against Charlie/<stack>).

Requires two caller-owned PATs — see Secret placement below.

Caller — minimal

# <owner>/<repo>/.gitea/workflows/build.yaml
name: Build image
on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  build:
    uses: https://${{ github.repository_owner }}:${{ secrets.CALLER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/workflows/build-image.yml@main
    with:
      host: melina
    secrets: inherit

Caller — with build args and an extra tag

jobs:
  build:
    uses: https://${{ github.repository_owner }}:${{ secrets.CALLER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/workflows/build-image.yml@main
    with:
      host: melina
      dockerfile: docker/Dockerfile
      build-args: |
        VERSION=${{ github.ref_name }}
        COMMIT=${{ github.sha }}
      extra-tags: |
        ${{ github.ref_name }}
    secrets: inherit

Build-host prerequisites

The host named in host: runs the actual docker build / docker push. Requirements are a strict subset of the regular deploy-target prerequisites (Host prerequisites):

Requirement	Why
sys-gitea-runner resolvable via LDAP + member of charlie-deploy	SSH session + docker socket access.
sshd trusts the step-ca SSH user CA	Runner-issued user certs are accepted.
/var/run/docker.sock is root:charlie-deploy	docker build / push work without sudo.
docker buildx plugin installed (docker-buildx-plugin / docker-cli-buildx)	The workflow exports DOCKER_BUILDKIT=1 so Dockerfiles can use RUN --mount=type=cache etc. Without buildx, docker build errors with "BuildKit is enabled but the buildx component is missing". Verify with docker buildx version.
git available on the host	Cloning the caller source into the scratch workdir.
Outbound HTTPS to gitea.alexandru.macocian.me	docker push + git clone.

No SOPS / age key / /mnt/nas/stacks is touched by a build — those are deploy concerns only.

Builds accumulate base-image layers in the build host's docker storage. The workflow drops the timestamp-tagged image after pushing, but base layers stay cached (intentionally — speeds up subsequent builds). Run docker system prune on the build host periodically.

Secret placement (image-source repos)

Unlike deploy-stack.yml (which relies on the sys-gitea-runner system account for everything), build-image.yml uses caller-owned PATs. Each user/org that wants to build images provides their own credentials — least privilege, no central PAT to share across namespaces, and the package-write PAT only ever has access to that user's own namespace.

sys-gitea-runner is still the SSH identity into the build host (via step-ca cert) — that's unchanged and unrelated to these secrets.

Set both at the user-level Settings → Actions → Secrets on the account that owns the source repos. All repos under that personal namespace then inherit both via secrets: inherit:

Secret	Gitea PAT scope	Owner	Used for
CALLER_REPO_PAT	read:repository (All)	The calling user/org	Resolving the absolute-URL uses: reference to this workflow + its sibling composite actions in Charlie/project-charlie, and cloning the caller source on the build host.
CALLER_REGISTRY_PAT	write:package (All, or scoped to the target namespace)	The calling user/org	docker login + push to the Gitea container registry.

Two distinct tokens so the package-write PAT is never used over HTTPS git (and the repo-read PAT can never be used to push images); revoking one doesn't take the other down.

Recipe — onboarding a new image-building namespace (one-time)

1. Grant the calling user read access to Charlie/project-charlie. Either add them as a Read collaborator on the repo (Charlie/project-charlie → Settings → Collaborators) or as a Read member of the Charlie org. Required so their PAT can resolve the uses: URL.
2. Log in to Gitea as the calling user. Settings → Applications → "Generate New Token" twice:
   • build-repo-ro: read:repository, Repository and Organization Access: All (or at minimum: Charlie/project-charlie + the user's own source repos).
   • build-package-rw: write:package, Repository and Organization Access: All (or scoped to the namespace receiving the images).
3. Still logged in as the calling user → Settings → Actions → Secrets → "Add Secret":
   • CALLER_REPO_PAT = the build-repo-ro value.
   • CALLER_REGISTRY_PAT = the build-package-rw value.
4. Drop the Caller — minimal snippet into each source repo at .gitea/workflows/build.yaml. Push.

Inputs reference — charlie-ssh-open

Input	Required	Default	Description
host	yes	—	Hostname to SSH into. LAN-resolvable.
user	no	sys-gitea-runner	Remote user.
duration	no	5m	Cert validity. Capped by the runners provisioner config in step-ca (currently 10m max).

Exports to $GITHUB_ENV so subsequent steps can use them:

Variable	Meaning
CHARLIE_SSH_HOST	The target host.
CHARLIE_SSH_USER	The remote user.
CHARLIE_SSH_DIR	Tmpdir holding the ephemeral key + cert.
CHARLIE_SSH_SOCKET	Unix socket for the SSH multiplex master.

Inputs reference — charlie-ssh-target

Input	Required	Default	Description
run	yes	—	Shell command(s) to run on the target. Multi-line allowed; runs as bash -lc <command>.
workdir	no	""	Optional directory to cd into before running.

Reads CHARLIE_SSH_* from job env (must come after charlie-ssh-open).

Inputs reference — charlie-ssh-close

No inputs. Reads CHARLIE_SSH_* from job env. Idempotent. Always end your job with this in an if: always() step.

Host prerequisites

Every host that wants to be a deploy target needs:

Requirement	Why
sys-gitea-runner resolves via LDAP/SSSD; charlie-deploy group exists with sys-gitea-runner as a member	Runner SSHes in as this user; group grants docker + stacks access. See onboarding.md → Charlie deploy group.
sshd trusts the step-ca SSH user CA (/etc/ssh/trusted_user_ca.pub populated by sync-ca.sh)	Runner-issued user certs are accepted by sshd. See onboarding.md → Trusted CA.
/etc/charlie/age.key (0640, root:charlie-deploy)	SOPS decrypt by sys-gitea-runner. See onboarding.md → Host age key (SOPS).
/var/run/docker.sock is root:charlie-deploy	docker compose works without sudo. See onboarding.md → Charlie deploy group.
/mnt/nas/stacks/<stack>/.git exists, owned sys-gitea-runner:charlie-deploy 0775+s	The deploy step does git fetch && git reset --hard after the first run. If the working tree is missing, the workflow auto-clones it on first deploy (see Auto-provisioning new stacks below).

The runner host itself only needs the runner stack (and step-ca trust if you want to run step ssh login interactively as a human). The runner container reaches step-ca via network_mode: host and the host's resolver.

Auto-provisioning new stacks

The reusable workflow's "Provision or verify stack working tree" step auto-clones a stack on its first deploy. Behaviour:

• <path>/.git exists → no-op, fall through to Sync.
• <path> missing or empty → sudo install -d the parent (one-shot, ownership = sys-gitea-runner), then git clone --branch <ref> the stack repo using the runner's RUNNER_REPO_PAT as a one-shot http.extraHeader. The PAT is not persisted to .git/config.
• <path> exists, non-empty, no .git → fail loud. Refuses to clobber a hand-created tree; operator must clear it manually first.

Subsequent git fetch calls in the Sync step use the host's existing git credentials for sys-gitea-runner (same as manually-cloned stacks). A host with no configured git identity will fail at Sync — surfacing the gap rather than silently embedding a long-lived PAT on disk. Tracked as roadmap.md → Identity & access → Per-host git identity.

Manual fallback (no runner available)

The deploy workflow is the happy path. When you need to push by hand — runner outage, gitea-on-melina (which can't be deployed by a runner that depends on it), first-time bring-up before runners exist, or debugging — SSH into the target host yourself and run the equivalent commands. SOPS is offline crypto; nothing else in the fleet has to be up.

Stack with secrets

# SSH in via gssh as your normal LDAP user (alex/amacocian).
ssh morgott

# Become sys-gitea-runner. -H sets HOME so credential helpers work.
sudo -u sys-gitea-runner -H bash

cd /mnt/nas/stacks/<stack>
git fetch --prune origin
git reset --hard origin/main

HOST=$(hostname -s | tr '[:upper:]' '[:lower:]')

# Decrypt every secret the workflow would have decrypted. Mirror the
# `decrypt:` block in .gitea/workflows/deploy.yaml.
SOPS_AGE_KEY_FILE=/etc/charlie/age.key \
  sops --decrypt --output-type dotenv --output .env secrets/env.sops.yaml
chmod 600 .env

# Pick the right compose files (mirrors deploy-stack.yml resolution).
if [ -f "docker-compose.$HOST.yml" ]; then
  COMPOSE_FILES=(-f "docker-compose.$HOST.yml")
else
  COMPOSE_FILES=(-f docker-compose.yml)
  [ -f "overrides/$HOST.yml" ] && COMPOSE_FILES+=(-f "overrides/$HOST.yml")
fi
ENVFILE=()
[ -f "envs/$HOST.env" ] && ENVFILE=(--env-file "envs/$HOST.env")

docker compose "${ENVFILE[@]}" "${COMPOSE_FILES[@]}" pull
docker compose "${ENVFILE[@]}" "${COMPOSE_FILES[@]}" up -d

# Clean up plaintext.
rm -f .env
exit  # leave sys-gitea-runner shell

Stack without secrets

Same as above, minus the decrypt and the rm -f .env lines.

Gitea quirks (leave them here for the next time this bites)

Hard-won lessons from getting the first end-to-end pipeline working. Worth re-reading before debugging anything in this area.

Cross-repo uses: needs absolute URL with PAT

The relative form Charlie/project-charlie/...@main makes act default to GitHub. The absolute form without auth fails because Gitea has REQUIRE_SIGNIN_VIEW=true instance-wide. So every uses: referencing a sibling repo has to be:

uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/<repo>/...@<ref>

This applies transitively — inside deploy-stack.yml, the uses: for the charlie-ssh-* composite actions also have to be absolute-URL form, otherwise act tries github.com from inside the workflow.

The PAT belongs to the LDAP-backed sys-gitea-runner service account (LDAP → synced into Gitea via Authentik OAuth → placed in a runners team in the Charlie org with read on all repos). PAT itself is set as a Charlie org-level secret so every stack repo inherits it. Callers must include secrets: inherit so the called workflow sees the secret.

References:

• https://github.com/go-gitea/gitea/issues/25929 — the canonical issue
• https://github.com/go-gitea/gitea/pull/36173 — the fix that landed in Gitea 1.27 (we pre-date it; the PAT-in-URL workaround is needed)

Reusable-workflow runs-on label matching is broken

runs-on: [self-hosted, morgott] inside a reusable workflow does not correctly route to a runner that advertises both labels. Gitea's matching either picks any runner with self-hosted (loose match) or hangs. This is #32348 — open, type/bug. PR #36388 was attempted, closed; replaced by #37478, still WIP at time of writing.

We sidestep it entirely by using runs-on: self-hosted and SSHing to the target host inside the job. Any runner can deploy any stack.

Documented exception: Charlie/runners. The runners stack genuinely needs precise routing (cross-deploy: each host's runner deploys the other host so no runner ever recreates itself mid-job). Top-level runs-on: in a regular workflow does respect labels — the bug is reusable-workflow-specific — so Charlie/runners ships a hand-rolled .gitea/workflows/deploy.yaml with two pinned jobs instead of calling deploy-stack.yml. When #32348 is fixed, collapse it into a matrix: { host: [morgott, melina] } calling the shared workflow. Background: runners.md → Updates.

act aggressively caches reusable workflows / composite actions

After pushing a fix to Charlie/project-charlie, the runner still uses the old version because act caches at /root/.cache/act/Charlie-project-charlie@main. The cache key is the ref string, not the commit. To force a refresh after pushing a fix:

sudo docker exec runner rm -rf /root/.cache/act

Long-term fix: stop using @main for cross-repo refs. Pin to specific tags (@v1, @<sha>). New ref string → new cache entry → automatic refresh. Left for later (until the contract stabilises).

LAN DNS inside the runner container

Default Docker DNS (127.0.0.11 embedded resolver) doesn't know about .lan hostnames. The runner stack uses network_mode: host so the container shares the host's resolver and can reach morgott.lan, cert-issuer.lan (well, that one's docker-internal — morgott.lan is the right name), etc.

Note about cert-issuer.lan

cert-issuer.lan is the docker-network alias for the step-ca container, only resolvable from inside the cert-issuer's docker network. From anywhere else (including the runner container with network_mode: host), use morgott.lan:9000 (cert-issuer is pinned to morgott — see cert-issuer.md).

Workflow logs collapse

The Gitea Actions UI shows everything happening in Set up job rather than separate cards per step. Cosmetic; doesn't reflect the actual step structure. The full chronological log inside Set up job is what to scroll through. Suspected to be a Gitea / act_runner UI bug; not impacting correctness.

insufficient permission for adding an object to repository database .git/objects

Symptom: deploy step "Sync working tree" fails with the above git error on git fetch against /mnt/nas/stacks/<stack>/.

Cause: the stack's .git/ is owned by root because the working tree was bootstrapped manually with sudo git clone ... instead of going through the deploy workflow (or onboarding step 5 in onboarding.md → Charlie deploy group). The deploy workflow runs as sys-gitea-runner over SSH and can't write into root-owned objects.

Fix once on the affected host:

sudo chown -R sys-gitea-runner:charlie-deploy /mnt/nas/stacks/<stack>
sudo find /mnt/nas/stacks/<stack> -type d -exec chmod 2775 {} \;
sudo find /mnt/nas/stacks/<stack> -type f -exec chmod g+rw {} \;

Then re-trigger the workflow. To avoid re-introducing this for new stacks, do the manual bootstrap clone as sys-gitea-runner:

sudo -u sys-gitea-runner -H git clone <url> /mnt/nas/stacks/<stack>

strategy.max-parallel: 1 doesn't actually serialize

Symptom: shared-stack deploy lands on host A and host B in parallel; host A's if: always() cleanup rms the decrypted .env while host B's compose up is still running, producing couldn't read env file. Or: two parallel git fetch && reset --hard against the same NFS-shared .git race on lock files.

Cause: Gitea Actions accepts strategy.max-parallel: syntactically but does not reliably honor it for matrix jobs that call a reusable workflow. Both jobs get queued and pick up runners simultaneously.

Fix: don't use the matrix shape for shared stacks. Spell each host out as a separate job, chained with needs: for true sequential execution. Same shape we use in Charlie/runners and Charlie/otelcollector.

jobs:
  deploy-morgott:
    uses: ... deploy-stack.yml@main
    with: { host: morgott, ... }
    secrets: inherit
  deploy-melina:
    needs: deploy-morgott
    uses: ... deploy-stack.yml@main
    with: { host: melina, ... }
    secrets: inherit

Long-term cleanup: have deploy-stack.yml decrypt to a per-host filename (e.g. .env.<host> or /run/charlie-secrets/<stack>/.env) and pass it via --env-file. Then matrix-with-true-parallelism becomes safe, since neither the working tree nor the env file collide. Until then, needs: is the working pattern.

Adding new reusable parts

If you find yourself copying logic across stack workflows, pull it in here. Two flavors:

• Composite action at .gitea/actions/<name>/action.yml — small, single step's worth of logic. Cheap to call from inside any workflow.
• Reusable workflow at .gitea/workflows/<name>.yml — owns the whole job (runner selection, sequencing, multiple steps).

Pin callers to @main for now; cut @v1 etc. once the contract is stable. Breaking changes to inputs need a major bump.