Auto-provision stacks

2026-05-13 15:14:13 +02:00
parent e70cbd3473
commit 54886ff237
5 changed files with 91 additions and 6 deletions
@@ -103,14 +103,58 @@ jobs:
        with:
          host: ${{ inputs.host }}

-      - name: Verify stack working tree exists
+      - name: Provision or verify stack working tree
        uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/actions/charlie-ssh-target@main
        with:
+          # First-deploy auto-provisioning. If $path/.git is missing we
+          # clone the stack repo into place using the runner's PAT, then
+          # fall through to the existing Sync step.
+          #
+          # Auth model: the PAT is passed as a one-shot
+          # `http.extraHeader` for the clone *only*. It is NOT written to
+          # `.git/config`, so subsequent `git fetch` calls in the Sync
+          # step use the host's existing git credentials (same as
+          # manually-cloned stacks). On a host with no sys-gitea-runner
+          # git credentials configured, Sync will fail loudly — surfacing
+          # the missing per-host identity instead of silently planting a
+          # long-lived PAT on disk. Tracked as roadmap.md → Identity &
+          # access → "Per-host git identity".
+          #
+          # Gitea Actions masks `${{ secrets.* }}` in workflow logs; the
+          # base64-encoded value is briefly visible in the target host's
+          # process list while git runs. Acceptable on the intranet-only
+          # homelab.
          run: |
-            test -d "${{ steps.resolve.outputs.path }}/.git" || {
-              echo "::error::stack path '${{ steps.resolve.outputs.path }}' is not a git working tree"
-              exit 1
-            }
+            set -euo pipefail
+            path="${{ steps.resolve.outputs.path }}"
+            ref="${{ steps.resolve.outputs.ref }}"
+            repo="${{ github.repository }}"
+            if [ -d "$path/.git" ]; then
+              exit 0
+            fi
+            if [ -e "$path" ]; then
+              # Refuse to clone over a non-git directory — almost
+              # certainly a hand-created working tree we shouldn't
+              # clobber. Operator decides: empty it manually, then re-run.
+              if [ -n "$(ls -A "$path" 2>/dev/null)" ]; then
+                echo "::error::stack path '$path' exists, is not a git working tree, and is non-empty — refusing to provision"
+                exit 1
+              fi
+              # Empty pre-existing dir: rmdir so `git clone` can create.
+              rmdir "$path"
+            fi
+            parent="$(dirname "$path")"
+            # /mnt/nas/stacks itself should already exist on every host
+            # per onboarding.md; this just ensures any deeper parent (rare)
+            # is in place. Owned by the deploy user so subsequent
+            # `git fetch/reset` don't need sudo.
+            sudo install -d -o "$(id -u)" -g "$(id -g)" -m 0755 "$parent"
+            echo "::notice::provisioning stack working tree at $path from $repo (ref: $ref)"
+            auth=$(printf '%s' 'sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}' | base64 -w0)
+            git -c "http.extraheader=Authorization: Basic ${auth}" \
+                clone --branch "$ref" \
+                "https://gitea.alexandru.macocian.me/${repo}.git" \
+                "$path"

      - name: Sync working tree
        uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/actions/charlie-ssh-target@main
@@ -238,12 +238,31 @@ Every host that wants to be a deploy target needs:
 | sshd trusts the step-ca SSH user CA (`/etc/ssh/trusted_user_ca.pub` populated by `sync-ca.sh`) | Runner-issued user certs are accepted by sshd. See [onboarding.md → Trusted CA](onboarding.md#trusted-ca). |
 | `/etc/charlie/age.key` (`0640`, `root:charlie-deploy`) | SOPS decrypt by `sys-gitea-runner`. See [onboarding.md → Host age key (SOPS)](onboarding.md#host-age-key-sops). |
 | `/var/run/docker.sock` is `root:charlie-deploy` | `docker compose` works without sudo. See [onboarding.md → Charlie deploy group](onboarding.md#charlie-deploy-group). |
-| `/mnt/nas/stacks/<stack>/.git` exists, owned `sys-gitea-runner:charlie-deploy 0775+s` | The deploy step does `git fetch && git reset --hard`, not a fresh clone. |
+| `/mnt/nas/stacks/<stack>/.git` exists, owned `sys-gitea-runner:charlie-deploy 0775+s` | The deploy step does `git fetch && git reset --hard` after the first run. If the working tree is missing, the workflow auto-clones it on first deploy (see [Auto-provisioning new stacks](#auto-provisioning-new-stacks) below). |

 The runner host itself only needs the runner stack (and step-ca trust if
 you want to run `step ssh login` interactively as a human). The runner
 container reaches step-ca via `network_mode: host` and the host's resolver.

+### Auto-provisioning new stacks
+
+The reusable workflow's "Provision or verify stack working tree" step
+auto-clones a stack on its first deploy. Behaviour:
+
+- **`<path>/.git` exists** → no-op, fall through to Sync.
+- **`<path>` missing or empty** → `sudo install -d` the parent (one-shot,
+  ownership = `sys-gitea-runner`), then `git clone --branch <ref>` the
+  stack repo using the runner's `RUNNER_REPO_PAT` as a one-shot
+  `http.extraHeader`. The PAT is **not** persisted to `.git/config`.
+- **`<path>` exists, non-empty, no `.git`** → fail loud. Refuses to
+  clobber a hand-created tree; operator must clear it manually first.
+
+Subsequent `git fetch` calls in the Sync step use the host's existing
+git credentials for `sys-gitea-runner` (same as manually-cloned stacks).
+A host with no configured git identity will fail at Sync — surfacing the
+gap rather than silently embedding a long-lived PAT on disk. Tracked as
+[roadmap.md → Identity & access → Per-host git identity](roadmap.md#identity--access).
+
 ## Manual fallback (no runner available)

 The deploy workflow is the happy path. When you need to push by hand —
@@ -24,6 +24,7 @@ Canonical list of `Charlie/<stack>` repos. Working trees live at `/mnt/nas/stack
 | [`portainer`](https://gitea.alexandru.macocian.me/Charlie/portainer) | `portainer/` | upstream | container UI; CSRF trusted-origins set via CLI |
 | [`cert-issuer`](https://gitea.alexandru.macocian.me/Charlie/cert-issuer) | `cert-issuer/` | upstream `smallstep/step-ca:0.30.2` | SSH user-cert CA. Design: [cert-issuer.md](cert-issuer.md). |
 | [`cert-syncer`](https://gitea.alexandru.macocian.me/Charlie/cert-syncer) | `cert-syncer/` | upstream `smallstep/step-cli:0.30.2-trixie` | Active bridge from cert-issuer to `/mnt/nas/ca/`. Renews managed server certs. Design: [cert-issuer.md](cert-issuer.md). |
+| [`trivy-server`](https://gitea.alexandru.macocian.me/Charlie/trivy-server) | `trivy-server/` | upstream `aquasec/trivy:0.56.2` (+ `mcuadros/ofelia`, `mikefarah/yq`) | Central trivy vuln-DB cache on `morgott.lan:4954` (LAN-only, no auth). Daily 03:00 sweep of compose-declared image refs across `/mnt/nas/stacks/*/`. Pairs with `trivy-runner`. |

 ## Melina (owner host)

@@ -50,6 +51,7 @@ Canonical list of `Charlie/<stack>` repos. Working trees live at `/mnt/nas/stack
 | [`runners`](https://gitea.alexandru.macocian.me/Charlie/runners) | morgott, melina | per-host `secrets/<host>.env.sops.yaml`, host-local `/var/lib/charlie/runner/` | morgott, melina (cross-deploy; see [runners.md](runners.md)) | Gitea Actions runners. Image: [`amacocian/gitea-runner`](https://gitea.alexandru.macocian.me/amacocian/gitea-runner). |
 | [`portainer-agent`](https://gitea.alexandru.macocian.me/Charlie/portainer-agent) | morgott, melina, mohg, miquella | `docker-compose.yml` + `docker-compose.miquella.yml` | morgott, melina | DSM gets a per-host compose (no `/var/lib/docker/volumes` bind). mohg + miquella deploy by hand until SSSD-onboarded. |
 | [`otelcollector`](https://gitea.alexandru.macocian.me/Charlie/otelcollector) | morgott, melina, mohg, malenia | `docker-compose.yml` (shared); `secrets/env.sops.yaml` for the vmauth ingest header | morgott, melina | mohg deploys by hand until reonboarding; malenia (`launchd` + `install-mac.sh`) still ships the credential in `otelcol.mac.yaml` until reonboarded — see roadmap. |
+| [`trivy-runner`](https://gitea.alexandru.macocian.me/Charlie/trivy-runner) | morgott, melina | `docker-compose.yml` + `overrides/<host>.yml` (per-host trivy-server URL + cron schedule) | morgott, melina (sequential, mirrors otelcollector) | Daily `docker ps` sweep; staggered 03:00 morgott / 03:15 melina. Writes to `/var/lib/charlie/trivy/metrics/` consumed by otelcollector's node-exporter. |

 ## Mirrors (Gitea pull-mirror, no host action)

@@ -18,6 +18,9 @@ Open work, grouped by theme. Done items are dropped — git history has them.
 - [ ] **Build pipelines** on `amacocian/*` source repos to push images to gitea registry (replaces "build on laptop").
 - [ ] **Grafana dashboards as code.** Grafana 12.2's UI Git Sync only supports github.com — Gitea isn't an option. Pattern: dashboards live as JSON files inside `Charlie/victoriametrics/dashboards/`, mounted into grafana's provisioning path with `foldersFromFilesStructure: true` so subdirs become Grafana folders. Edits flow via a "Cannot save provisioned dashboard" → "Copy JSON" → paste into repo loop. Sidecar that auto-syncs UI edits → git deferred until manual loop becomes painful.
 - [x] **Initial dashboard set.** `hosts/` (overview, host-detail, thermal-power, filesystem-network), `processes/` (overview + CPU + memory broken down per process), `containers/` (overview, per-container resources, per-stack roll-up). Container metrics added by enabling the otelcol `docker_stats` receiver (mounts `/var/run/docker.sock`, promotes `com.docker.compose.project|service` to metric labels).
+- [x] **Container CVE sweep.** [`Charlie/trivy-server`](https://gitea.alexandru.macocian.me/Charlie/trivy-server) (morgott, central vuln-DB cache + daily compose-declared image sweep) + [`Charlie/trivy-runner`](https://gitea.alexandru.macocian.me/Charlie/trivy-runner) (per-host, daily `docker ps` sweep). Results land as Prometheus textfiles → node-exporter (textfile collector) → otelcol → VM. Dashboards under `containers/security/`. LAN-only between the two stacks; no public DNS, no auth — morgott + melina sit on the secure intranet.
+- [ ] **Trivy CVE alerting.** vmalert rule on `trivy_image_vulnerabilities{severity="CRITICAL"} > 0` → alertmanager via the existing route. Deferred until the sweep has produced a stable baseline (otherwise the first run is one giant page).
+- [ ] **CI build-time CVE scan.** Run `trivy image` as a step inside the eventual `amacocian/*` build pipelines (see "Build pipelines" above) so we catch HIGH/CRITICAL at build time, not just at the next nightly sweep.
 - [ ] **Runner observability (phase 2).** Gitea Actions runner dashboards in `runners/`. Likely a small exporter polling Gitea's Actions API → otelcol → VM. Tracked in [`Charlie/victoriametrics/dashboards/runners/README.md`](https://gitea.alexandru.macocian.me/Charlie/victoriametrics/src/branch/main/dashboards/runners/README.md).

 ## Deploy workflow
@@ -113,6 +113,23 @@ Background: [automation.md → `strategy.max-parallel: 1` doesn't actually seria

 SOPS-in-repo with one age key per host. The reusable workflow SSHes into the target host as `sys-gitea-runner` and runs `sops --decrypt` there using the host's `/etc/charlie/age.key`. Plaintext stays on the target host (mode 0600), gets cleaned up by an `if: always()` step. See [secrets.md](secrets.md).

+## Stack-emitted metrics (textfile pattern)
+
+Stacks that produce their own metrics — anything not already covered by the otelcol receivers (`hostmetrics`, `docker_stats`, `prometheus`) — publish them as **Prometheus textfiles** in a well-known host-local directory, and let [`Charlie/otelcollector`](https://gitea.alexandru.macocian.me/Charlie/otelcollector)'s node-exporter pick them up via `--collector.textfile`.
+
+Convention:
+
+```
+/var/lib/charlie/<stack>/metrics/<name>.prom
+```
+
+- The producing stack bind-mounts the dir **read-write** and writes atomically (build to a temp file in the same dir, then `mv`). Atomic mv prevents node-exporter ever scraping a half-written file.
+- `Charlie/otelcollector` bind-mounts the parent (`/var/lib/charlie/<stack>/metrics`) **read-only** into node-exporter and runs with `--collector.textfile.directory=…`.
+- The metric name(s) must match `Charlie/otelcollector`'s `prometheus` receiver `metric_relabel_configs` keep-regex (currently `node_hwmon_.*|node_rapl_.*|node_textfile_.*|trivy_.*`). Extend the regex when adding a new family.
+- Host attribution comes for free from otelcol's `resourcedetection` processor (`host.name`); textfiles **do not bake `host` into label sets**.
+
+First user of this pattern: [`Charlie/trivy-server`](https://gitea.alexandru.macocian.me/Charlie/trivy-server) + [`Charlie/trivy-runner`](https://gitea.alexandru.macocian.me/Charlie/trivy-runner) emit `trivy_*.prom`. The pattern generalises — future small exporters (cert-expiry, backup-age, drift-count, etc.) should reuse it instead of standing up their own scrape endpoints.
+
 ## Why not Portainer/Komodo/Dockge

 - **Portainer Stacks-from-Git**: works, but no submodule support, opinionated about working-tree location, adds a control plane on top of git.