project-charlie/docs/strategy.md

Strategy

Pattern

GitOps via Gitea Actions runners. The runner is stateless and SSHes into the target host using a short-lived step-ca-issued cert. The job is a thin caller of the shared deploy-stack.yml reusable workflow — see automation.md.

dev box ── git push ──► Gitea (Charlie/<stack>) ── webhook ──► any runner
                                                                   │
                                                                   │ ssh sys-gitea-runner@<host>
                                                                   │ (step-ca cert, 5min)
                                                                   ▼
                                          on the target host:
                                            git fetch && git reset --hard origin/main
                                            sops --decrypt …  (if `decrypt:` set)
                                            docker compose pull && docker compose up -d
                                            rm decrypted plaintext

The git working tree at /mnt/nas/stacks/<stack>/ is the deployment. Runtime state lives separately under /mnt/nas/data/<stack>/ (see layout.md).

Rules

• One owner host per stack (singleton stacks). The host named in the workflow's host: input is where the runner SSHes.
• Shared stacks (run on multiple hosts — see Shared stacks) follow the same NFS-shared-tree model but each host runs its own docker compose up -d against host-specific overrides.
• No hand-edits on the host once a stack is repo-managed. The deploy workflow does git fetch && git reset --hard, which clobbers any drift on purpose.
• Runtime state never enters git. Compose files bind into /mnt/nas/data/<stack>/, not into the working tree.
• .env ignored, .env.example tracked. Plaintext secrets stay out — encrypted form lives under secrets/.
• Image tags pinned. No :latest.
• Any runner can deploy any stack to any host. Routing-by-runner-label is intentionally avoided (Gitea bug; see automation.md → Gitea quirks). The runner SSHes to wherever host: points. Exception: Charlie/runners needs precise routing for cross-deploy.

Repo types in use

• Owned stacks — the working tree under Charlie/<stack>/ is the source of truth. docker-compose.yml, secrets/, etc.
• Pull-mirrors — read-only copy of an upstream we want offline (or as a reference for forks we maintain). Configured via Gitea's pull-mirror feature. Naming: Charlie/<name>-mirror.

Shared stacks

Some stacks run on multiple hosts with the same image but slightly different configuration (e.g. otelcollector, portainer-agent). Pattern borrowed from Ansible inventories + Kustomize overlays + Kubernetes DaemonSets:

Charlie/<stack>/
├── docker-compose.yml             # the "what" — default
├── docker-compose.<host>.yml      # full per-host compose (when delta is structural)
├── envs/
│   ├── morgott.env                # per-host env (Ansible host_vars equivalent)
│   └── melina.env
├── overrides/
│   └── <host>.yml                 # additive overlay (when env can't express it)
├── secrets/                       # SOPS-encrypted; decrypted at deploy
│   └── env.sops.yaml              # (or per-host: <host>.env.sops.yaml)
├── .sops.yaml
├── .gitea/workflows/deploy.yaml
└── README.md

Per-host resolution order

Applied identically by deploy-stack.yml and the manual fallback recipe:

1. If docker-compose.<host>.yml exists → use only that.
2. Else docker-compose.yml + (if present) overrides/<host>.yml layered on top.
3. If envs/<host>.env exists → loaded with --env-file.

Hostname is lowercased (hostname -s | tr '[:upper:]' '[:lower:]').

When to use which

Delta type	Use
Different scalar value (port, env var, URL)	envs/<host>.env
Add an extra service or scalar field	overrides/<host>.yml (compose merges additively)
Replace a list (volumes, command, networks)	docker-compose.<host>.yml (full file)
Different image	docker-compose.<host>.yml

Compose's !override and !reset YAML tags exist for list-replacement, but support is uneven across compose versions (notably broken on Synology DSM's bundled compose). Prefer the full-file approach when replacing lists.

Multi-host workflow shape

Despite strategy.md → shared-stack pattern supporting matrix-style fan-out in principle, don't use strategy.matrix for shared stacks today. Gitea Actions doesn't reliably honor strategy.max-parallel: 1, so two matrix jobs end up racing on the NFS-shared .git and (for SOPS stacks) the .env plaintext. Use needs: for true serialization instead:

jobs:
  deploy-morgott:
    uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/workflows/deploy-stack.yml@main
    with:
      host: morgott
      decrypt: |
        secrets/env.sops.yaml -> .env
    secrets: inherit

  deploy-melina:
    needs: deploy-morgott
    uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/workflows/deploy-stack.yml@main
    with:
      host: melina
      decrypt: |
        secrets/env.sops.yaml -> .env
    secrets: inherit

Background: automation.md → strategy.max-parallel: 1 doesn't actually serialize. Long-term fix is a per-host decrypt path in deploy-stack.yml, after which matrix becomes safe again.

Mental model

• "What" = docker-compose.yml (or per-host file).
• "Where" = the host named in the workflow's host: input.
• Per-host vars = envs/<host>.env.
• Per-host structural deltas = overrides/<host>.yml for additive, docker-compose.<host>.yml for replacement.
• Group defaults = compose env-var defaults (${VAR:-default}).

Secrets

SOPS-in-repo with one age key per host. The reusable workflow SSHes into the target host as sys-gitea-runner and runs sops --decrypt there using the host's /etc/charlie/age.key. Plaintext stays on the target host (mode 0600), gets cleaned up by an if: always() step. See secrets.md.

Stack-emitted metrics (textfile pattern)

Stacks that produce their own metrics — anything not already covered by the otelcol receivers (hostmetrics, docker_stats, prometheus) — publish them as Prometheus textfiles in a well-known host-local directory, and let Charlie/otelcollector's node-exporter pick them up via --collector.textfile.

Convention:

/var/lib/charlie/<stack>/metrics/<name>.prom

• The producing stack bind-mounts the dir read-write and writes atomically (build to a temp file in the same dir, then mv). Atomic mv prevents node-exporter ever scraping a half-written file.
• Charlie/otelcollector bind-mounts the parent (/var/lib/charlie/<stack>/metrics) read-only into node-exporter and runs with --collector.textfile.directory=….
• The metric name(s) must match Charlie/otelcollector's prometheus receiver metric_relabel_configs keep-regex (currently node_hwmon_.*|node_rapl_.*|node_textfile_.*|trivy_.*). Extend the regex when adding a new family.
• Host attribution comes for free from otelcol's resourcedetection processor (host.name); textfiles do not bake host into label sets.

First user of this pattern: Charlie/trivy-server + Charlie/trivy-runner emit trivy_*.prom. The pattern generalises — future small exporters (cert-expiry, backup-age, drift-count, etc.) should reuse it instead of standing up their own scrape endpoints.

Why not Portainer/Komodo/Dockge

• Portainer Stacks-from-Git: works, but no submodule support, opinionated about working-tree location, adds a control plane on top of git.
• Komodo: nice, also pull-based, also relocates working trees. Reconsider if we ever want a UI.
• Dockge: file mgmt only, no git wiring.

The runner-driven pattern keeps us in plain git + docker compose and adds nothing the team doesn't already understand.