project-charlie/docs/strategy.md

# Strategy

## Pattern

GitOps via Gitea Actions runners. The runner is stateless and SSHes into the target host using a short-lived step-ca-issued cert. The job is a thin caller of the [shared `deploy-stack.yml` reusable workflow](../.gitea/workflows/deploy-stack.yml) — see [automation.md](automation.md).

```
dev box ── git push ──► Gitea (Charlie/<stack>) ── webhook ──► any runner
                                                                   │
                                                                   │ ssh sys-gitea-runner@<host>
                                                                   │ (step-ca cert, 5min)
                                                                   ▼
                                          on the target host:
                                            git fetch && git reset --hard origin/main
                                            sops --decrypt …  (if `decrypt:` set)
                                            docker compose pull && docker compose up -d
                                            rm decrypted plaintext
```

The git working tree at `/mnt/nas/stacks/<stack>/` **is** the deployment. Runtime state lives separately under `/mnt/nas/data/<stack>/` (see [layout.md](layout.md)).

## Rules

- One owner host per stack (singleton stacks). The host named in the workflow's `host:` input is where the runner SSHes.
- Shared stacks (run on multiple hosts — see [Shared stacks](#shared-stacks)) follow the same NFS-shared-tree model but each host runs its own `docker compose up -d` against host-specific overrides.
- No hand-edits on the host once a stack is repo-managed. The deploy workflow does `git fetch && git reset --hard`, which clobbers any drift on purpose.
- Runtime state never enters git. Compose files bind into `/mnt/nas/data/<stack>/`, not into the working tree.
- `.env` ignored, `.env.example` tracked. Plaintext secrets stay out — encrypted form lives under `secrets/`.
- Image tags pinned. No `:latest`.
- Any runner can deploy any stack to any host. Routing-by-runner-label is intentionally avoided (Gitea bug; see [automation.md → Gitea quirks](automation.md#gitea-quirks-leave-them-here-for-the-next-time-this-bites)). The runner SSHes to wherever `host:` points. Exception: [`Charlie/runners`](runners.md) needs precise routing for cross-deploy.

## Repo types in use

- **Owned stacks** — the working tree under `Charlie/<stack>/` is the source of truth. `docker-compose.yml`, `secrets/`, etc.
- **Pull-mirrors** — read-only copy of an upstream we want offline (or as a reference for forks we maintain). Configured via Gitea's pull-mirror feature. Naming: `Charlie/<name>-mirror`.

## Shared stacks

Some stacks run on **multiple hosts** with the same image but slightly different configuration (e.g. `otelcollector`, `portainer-agent`). Pattern borrowed from Ansible inventories + Kustomize overlays + Kubernetes DaemonSets:

```
Charlie/<stack>/
├── docker-compose.yml             # the "what" — default
├── docker-compose.<host>.yml      # full per-host compose (when delta is structural)
├── envs/
│   ├── morgott.env                # per-host env (Ansible host_vars equivalent)
│   └── melina.env
├── overrides/
│   └── <host>.yml                 # additive overlay (when env can't express it)
├── secrets/                       # SOPS-encrypted; decrypted at deploy
│   └── env.sops.yaml              # (or per-host: <host>.env.sops.yaml)
├── .sops.yaml
├── .gitea/workflows/deploy.yaml
└── README.md
```

### Per-host resolution order

Applied identically by [`deploy-stack.yml`](../.gitea/workflows/deploy-stack.yml) and the [manual fallback recipe](automation.md#manual-fallback-no-runner-available):

1. If `docker-compose.<host>.yml` exists → use **only** that.
2. Else `docker-compose.yml` + (if present) `overrides/<host>.yml` layered on top.
3. If `envs/<host>.env` exists → loaded with `--env-file`.

Hostname is lowercased (`hostname -s | tr '[:upper:]' '[:lower:]'`).

### When to use which

| Delta type | Use |
|---|---|
| Different scalar value (port, env var, URL) | `envs/<host>.env` |
| Add an extra service or scalar field | `overrides/<host>.yml` (compose merges additively) |
| **Replace** a list (volumes, command, networks) | `docker-compose.<host>.yml` (full file) |
| Different image | `docker-compose.<host>.yml` |

Compose's `!override` and `!reset` YAML tags exist for list-replacement, but support is uneven across compose versions (notably broken on Synology DSM's bundled compose). Prefer the full-file approach when replacing lists.

### Multi-host workflow shape

Despite [strategy.md → shared-stack pattern](#shared-stacks) supporting matrix-style fan-out in principle, **don't use `strategy.matrix`** for shared stacks today. Gitea Actions doesn't reliably honor `strategy.max-parallel: 1`, so two matrix jobs end up racing on the NFS-shared `.git` and (for SOPS stacks) the `.env` plaintext. Use `needs:` for true serialization instead:

```yaml
jobs:
  deploy-morgott:
    uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/workflows/deploy-stack.yml@main
    with:
      host: morgott
      decrypt: |
        secrets/env.sops.yaml -> .env
    secrets: inherit

  deploy-melina:
    needs: deploy-morgott
    uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/workflows/deploy-stack.yml@main
    with:
      host: melina
      decrypt: |
        secrets/env.sops.yaml -> .env
    secrets: inherit
```

Background: [automation.md → `strategy.max-parallel: 1` doesn't actually serialize](automation.md#strategymax-parallel-1-doesnt-actually-serialize). Long-term fix is a per-host decrypt path in `deploy-stack.yml`, after which matrix becomes safe again.

### Mental model

- "What" = `docker-compose.yml` (or per-host file).
- "Where" = the host named in the workflow's `host:` input.
- Per-host vars = `envs/<host>.env`.
- Per-host structural deltas = `overrides/<host>.yml` for additive, `docker-compose.<host>.yml` for replacement.
- Group defaults = compose env-var defaults (`${VAR:-default}`).

## Secrets

SOPS-in-repo with one age key per host. The reusable workflow SSHes into the target host as `sys-gitea-runner` and runs `sops --decrypt` there using the host's `/etc/charlie/age.key`. Plaintext stays on the target host (mode 0600), gets cleaned up by an `if: always()` step. See [secrets.md](secrets.md).

## Why not Portainer/Komodo/Dockge

- **Portainer Stacks-from-Git**: works, but no submodule support, opinionated about working-tree location, adds a control plane on top of git.
- **Komodo**: nice, also pull-based, also relocates working trees. Reconsider if we ever want a UI.
- **Dockge**: file mgmt only, no git wiring.

The runner-driven pattern keeps us in plain `git` + `docker compose` and adds nothing the team doesn't already understand.