diff --git a/docs/provisioning.md b/docs/provisioning.md index 35a7ec3..2ed57a6 100644 --- a/docs/provisioning.md +++ b/docs/provisioning.md @@ -11,7 +11,8 @@ one provisioning role. | ---- | -------- | | Tool | Ansible (declarative, idempotent, cross-platform incl. macOS). | | Phases | Phase 0 trust bootstrap is operator-run; Phase 1+ convergence is CI-driven. | -| Phase 1 invocation | Gitea Actions, push from runner over step-ca SSH cert, mirroring `deploy-stack.yml`. | +| Phase 1 invocation | Gitea Actions on push to `main`: per-host jobs serialized via `needs:`, each runs `ansible-playbook --check` (dry-run-only for now; apply gated/disabled). Mirrors `otelcollector`. | +| Phase 1 auth | SSH as `sys-gitea-runner` (step-ca cert); `become` root via the LDAP `admins` group's NOPASSWD sudo rule. | | Repo | New `Charlie/provisioning` (playbooks, roles, inventory, Phase-1 workflow). | | Inventory | Ansible inventory is authoritative; [hosts](hosts.md) links to it instead of duplicating host facts. | | Secrets | SOPS/age, matching the rest of Charlie. | @@ -26,45 +27,118 @@ CI/SSH-cert target. | 1 | Hostname + Marika DNS/DHCP entry | — | off-host | | 2 | CA trust: X.509 root + SSH user CA | 1 | 0 | | 3 | `/etc/charlie/age.key` | — | 0 | -| 4 | NFS `/mnt/nas` autofs mount | 1 | 1 | -| 5 | LDAP/SSSD users, groups, sudo | 2, 3 | 1 | -| 6 | Docker / Compose | — | 1 | -| 7 | `charlie-deploy` group + Docker socket access | 6 | 1 | -| 8 | `sys-gitea-runner` Git + registry creds | 2, 3 | 1 | -| 9 | otelcollector | 4, 7 | 1 | -| 10 | autofs `/Users` homes ([shared-homes](shared-homes.md)) | 4, 5 | 1 | +| 4 | SSSD/LDAP client config (resolves `sys-gitea-runner` + `admins`) | 2, 3 | 0 | +| 5 | `admins` NOPASSWD sudoers (`/etc/sudoers.d/90-ldap-admins`) | 4 | 0 | +| 6 | NFS `/mnt/nas` autofs mount | 1 | 1 | +| 7 | Docker / Compose | — | 1 | +| 8 | `charlie-deploy` group + Docker socket access | 7 | 1 | +| 9 | `sys-gitea-runner` Git + registry creds | 2, 3 | 1 | +| 10 | otelcollector | 6, 8 | 1 | +| 11 | autofs `/Users` homes ([shared-homes](shared-homes.md)) | 4, 6 | 1 | ## Two-phase model ```mermaid flowchart LR Op[Operator + Sherlock] -->|Phase 0: out-of-band SSH| Bare[Bare host] - Bare -->|trust: CA roots, SSH CA, age key| Trusted[Trusted host] + Bare -->|"Phase 0: CA + SSH CA, age key, SSSD, admins sudoers"| Trusted[Trusted host] Runner[Gitea runner] -->|Phase 1: step-ca SSH cert| Trusted Trusted -->|converged| Target[Deploy target] ``` -- **Phase 0 — trust bootstrap (operator-run).** Resolves the chicken-and-egg: - a bare node has no age key, no CA trust, no LDAP identity, so it cannot be a - CI target yet. Operator connects out-of-band (console / initial key) and lays - down CA roots, the SSH user CA, and `/etc/charlie/age.key`. Minimal, run once. +- **Phase 0 — trust + identity bootstrap (operator-run).** Resolves the + chicken-and-egg: a bare node has no age key, no CA trust, and no LDAP + identity, so the CI runner cannot SSH in as `sys-gitea-runner` or `become` + root yet. Operator connects out-of-band (console / initial key) and lays down, + in order: CA roots + SSH user CA, `/etc/charlie/age.key`, the SSSD/LDAP client + config (so `sys-gitea-runner` and `admins` resolve), and the `%admins` + NOPASSWD sudoers file (`/etc/sudoers.d/90-ldap-admins`). Minimal, run once. + NFS, Docker, registry creds, otelcollector, and `/Users` homes all converge in + Phase 1. - **Phase 1+ — convergence (CI-driven).** Once trust and LDAP land, the host is a normal step-ca SSH target. A Gitea workflow runs the playbook idempotently; re-runnable on every host forever. +### Phase 0 — minimal manual bootstrap + +Only what the CI runner needs to *log in and escalate* must exist before the +first converge. Everything else is Phase 1 (the tool converges it as root): + +| Must be manual (Phase 0) | Why it can't wait for Phase 1 | +| ------------------------ | ----------------------------- | +| CA trust (X.509 root + SSH user CA) | sshd must accept the step-ca cert to let the runner in. | +| `/etc/charlie/age.key` | needed to decrypt the LDAP bind secret for SSSD. | +| SSSD/LDAP client config | so `sys-gitea-runner` + `admins` resolve at login. | +| `%admins` NOPASSWD sudoers | so `become` works on the very first converge. | + +- **NFS and Docker are *not* Phase 0.** They are installed/configured as root by + Phase 1, after the runner connects and escalates. Keeping them manual would be + redundant. +- **Home-dir wrinkle.** `sys-gitea-runner`'s home is `/Users/sys-gitea-runner` + (NFS-backed). On a truly bare node, before the `nfs`/`homes` roles run, that + path won't resolve. Login still works; point Ansible's `remote_tmp` at `/tmp` + for the first converge. + +`bootstrap.yml` does all of the above (`trust` then `identity`). It is operator-run +out-of-band, e.g. `ansible-playbook playbooks/bootstrap.yml -l -u +--ask-become-pass -e trust_src_dir=`. The public CA material (X.509 root + +SSH user CA pubkey) comes from an operator-provided bundle — a mounted NAS share or +a USB stick on the control node (`trust_src_dir`); `/mnt/nas` isn't available yet. +The host age key is copied from the bundle if present, otherwise generated on the +host (register its printed recipient in [hosts](hosts.md) + the repo `.sops.yaml`). + +### Phase 1 connection & auth + +The converge workflow reuses the [`charlie-ssh-open`][ssh-open] action: it mints +a short-lived step-ca SSH user cert and opens a multiplex master to the target. +Ansible attaches to that same socket (`ControlPath`), so there is no second cert +or auth handshake. + +| Layer | Mechanism | Credential | +| ----- | --------- | ---------- | +| SSH login | step-ca SSH user cert, principal `sys-gitea-runner` ([cert-issuer](cert-issuer.md) `runners` provisioner) | short-lived cert | +| `become` root | `sys-gitea-runner` ∈ LDAP `admins` → `%admins ALL=(ALL) NOPASSWD:ALL` | none (NOPASSWD) | + +`sys-gitea-runner` is already in `charlie-deploy` (Docker socket = root-equivalent) +on every deploy target, so a separate sudo identity would add no real boundary; +reusing the runner identity keeps Phase 1 simple and auditable. NOPASSWD is the +linchpin — `become` works over the non-interactive socket with no sudo password +to vault or inject. + +The converge workflow runs automatically on push to `main` and fans out one job +per host, serialized via `needs:` — Gitea Actions does not reliably honour +`strategy.max-parallel`, and host jobs share the runner's SSH multiplex socket +and Ansible venv, so they must not overlap (same shape as +[`otelcollector`][otel-deploy]'s deploy). Each job runs the reusable +`converge-host.yml`. **It is currently dry-run-only**: it runs +`ansible-playbook --check` to prove the converge does not break, with the real +apply commented out until the dry runs are proven green across the fleet. When +re-enabled it becomes a dry-run → apply gate (apply only if `--check` is green). +Active order: `morgott` → `melina`; `mohg` stays manual until its Phase 0 +bootstrap lands, and macOS/NAS are out of scope for `site.yml` (`linux` group +only). + +[ssh-open]: https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/.gitea/actions/charlie-ssh-open +[otel-deploy]: https://gitea.alexandru.macocian.me/Charlie/otelcollector/src/branch/main/.gitea/workflows/deploy.yaml + ## Proposed roles | Role | Provides | Phase | | ---- | -------- | ----- | | `trust` | CA roots, SSH user CA, `/etc/charlie/age.key` | 0 | +| `identity` | SSSD/LDAP client config; `%admins` NOPASSWD sudoers | 0 + 1 | | `nfs` | autofs `/mnt/nas` mount | 1 | -| `identity` | SSSD + LDAP client config; sudo via groups | 1 | | `docker` | Docker/Compose engine | 1 | -| `deploy-group` | `charlie-deploy`, Docker socket access | 1 | -| `gitea-access` | `sys-gitea-runner` Git + registry creds | 1 | +| `deploy_group` | `charlie-deploy`, Docker socket access | 1 | +| `gitea_access` | `sys-gitea-runner` Git + registry creds | 1 | | `observability` | otelcollector | 1 | | `homes` | autofs `/Users` + LDAP automount map | 1 | +`bootstrap.yml` (Phase 0) runs `trust` then `identity`; `site.yml` (Phase 1) +re-asserts `identity` idempotently alongside the rest. The `identity` role must +therefore manage the `%admins` sudoers drop-in so a bare node becomes sudo-able +before its first CI converge. + ## Inventory & secrets - `inventory/` holds the authoritative host list; `host_vars/` carries @@ -78,7 +152,6 @@ flowchart LR | Item | Note | | ---- | ---- | | macOS (Malenia) | Ansible supports it, but roles branch on OS (launchd vs systemd, no apt, native `/Users`). Decide per-role branching vs a separate playbook. | -| Existing hosts | Converge current hosts in place, or apply only to new nodes first? | | First-run-only steps | Hostname/DNS and Phase-0 trust are not idempotent CI work; keep them operator-gated. | | Home creation owner | Per-user home provisioning may belong to the `homes` role or DirectoryAdmin at account creation. See [shared-homes](shared-homes.md). | | Break-glass | Keep a local sudoer with a local home so LDAP/NFS outages don't lock out provisioning. |