Charlie/project-charlie
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(provisioning): automated dry-run converge model

Browse Source 
Update the provisioning design/tracking doc for the Phase 1 converge
automation: push-to-main fanout serialized via `needs:` (mirrors
otelcollector), currently dry-run-only with the apply gated/disabled
until the dry runs are proven green. Resolve the "Existing hosts"
open question.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Alexandru Macocian
2026-06-24 17:08:21 +02:00
parent a00711d460
commit 13dd92c95b
1 changed files with 90 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/provisioning.md
+90 -17
View File
@@ -11,7 +11,8 @@ one provisioning role.
| ---- | -------- |
| Tool | Ansible (declarative, idempotent, cross-platform incl. macOS). |
| Phases | Phase 0 trust bootstrap is operator-run; Phase 1+ convergence is CI-driven. |
| Phase 1 invocation | Gitea Actions, push from runner over step-ca SSH cert, mirroring `deploy-stack.yml`. |
| Phase 1 invocation | Gitea Actions on push to `main`: per-host jobs serialized via `needs:`, each runs `ansible-playbook --check` (dry-run-only for now; apply gated/disabled). Mirrors `otelcollector`. |
| Phase 1 auth | SSH as `sys-gitea-runner` (step-ca cert); `become` root via the LDAP `admins` group's NOPASSWD sudo rule. |
| Repo | New `Charlie/provisioning` (playbooks, roles, inventory, Phase-1 workflow). |
| Inventory | Ansible inventory is authoritative; [hosts](hosts.md) links to it instead of duplicating host facts. |
| Secrets | SOPS/age, matching the rest of Charlie. |
@@ -26,45 +27,118 @@ CI/SSH-cert target.
| 1 | Hostname + Marika DNS/DHCP entry | — | off-host |
| 2 | CA trust: X.509 root + SSH user CA | 1 | 0 |
| 3 | `/etc/charlie/age.key` | — | 0 |
| 4 | NFS `/mnt/nas` autofs mount | 1 | 1 |
| 5 | LDAP/SSSD users, groups, sudo | 2, 3 | 1 |
| 6 | Docker / Compose | — | 1 |
| 7 | `charlie-deploy` group + Docker socket access | 6 | 1 |
| 8 | `sys-gitea-runner` Git + registry creds | 2, 3 | 1 |
| 9 | otelcollector | 4, 7 | 1 |
| 10 | autofs `/Users` homes ([shared-homes](shared-homes.md)) | 4, 5 | 1 |
| 4 | SSSD/LDAP client config (resolves `sys-gitea-runner` + `admins`) | 2, 3 | 0 |
| 5 | `admins` NOPASSWD sudoers (`/etc/sudoers.d/90-ldap-admins`) | 4 | 0 |
| 6 | NFS `/mnt/nas` autofs mount | 1 | 1 |
| 7 | Docker / Compose | — | 1 |
| 8 | `charlie-deploy` group + Docker socket access | 7 | 1 |
| 9 | `sys-gitea-runner` Git + registry creds | 2, 3 | 1 |
| 10 | otelcollector | 6, 8 | 1 |
| 11 | autofs `/Users` homes ([shared-homes](shared-homes.md)) | 4, 6 | 1 |
## Two-phase model
```mermaid
flowchart LR
Op[Operator + Sherlock] -->|Phase 0: out-of-band SSH| Bare[Bare host]
Bare -->|trust: CA roots, SSH CA, age key| Trusted[Trusted host]
Bare -->|"Phase 0: CA + SSH CA, age key, SSSD, admins sudoers"| Trusted[Trusted host]
Runner[Gitea runner] -->|Phase 1: step-ca SSH cert| Trusted
Trusted -->|converged| Target[Deploy target]
```
- **Phase 0 — trust bootstrap (operator-run).** Resolves the chicken-and-egg:
a bare node has no age key, no CA trust, no LDAP identity, so it cannot be a
CI target yet. Operator connects out-of-band (console / initial key) and lays
down CA roots, the SSH user CA, and `/etc/charlie/age.key`. Minimal, run once.
- **Phase 0 — trust + identity bootstrap (operator-run).** Resolves the
chicken-and-egg: a bare node has no age key, no CA trust, and no LDAP
identity, so the CI runner cannot SSH in as `sys-gitea-runner` or `become`
root yet. Operator connects out-of-band (console / initial key) and lays down,
in order: CA roots + SSH user CA, `/etc/charlie/age.key`, the SSSD/LDAP client
config (so `sys-gitea-runner` and `admins` resolve), and the `%admins`
NOPASSWD sudoers file (`/etc/sudoers.d/90-ldap-admins`). Minimal, run once.
NFS, Docker, registry creds, otelcollector, and `/Users` homes all converge in
Phase 1.
- **Phase 1+ — convergence (CI-driven).** Once trust and LDAP land, the host is
a normal step-ca SSH target. A Gitea workflow runs the playbook idempotently;
re-runnable on every host forever.
### Phase 0 — minimal manual bootstrap
Only what the CI runner needs to *log in and escalate* must exist before the
first converge. Everything else is Phase 1 (the tool converges it as root):
| Must be manual (Phase 0) | Why it can't wait for Phase 1 |
| ------------------------ | ----------------------------- |
| CA trust (X.509 root + SSH user CA) | sshd must accept the step-ca cert to let the runner in. |
| `/etc/charlie/age.key` | needed to decrypt the LDAP bind secret for SSSD. |
| SSSD/LDAP client config | so `sys-gitea-runner` + `admins` resolve at login. |
| `%admins` NOPASSWD sudoers | so `become` works on the very first converge. |
- **NFS and Docker are *not* Phase 0.** They are installed/configured as root by
Phase 1, after the runner connects and escalates. Keeping them manual would be
redundant.
- **Home-dir wrinkle.** `sys-gitea-runner`'s home is `/Users/sys-gitea-runner`
(NFS-backed). On a truly bare node, before the `nfs`/`homes` roles run, that
path won't resolve. Login still works; point Ansible's `remote_tmp` at `/tmp`
for the first converge.
`bootstrap.yml` does all of the above (`trust` then `identity`). It is operator-run
out-of-band, e.g. `ansible-playbook playbooks/bootstrap.yml -l <host> -u <initial-admin>
--ask-become-pass -e trust_src_dir=<bundle>`. The public CA material (X.509 root +
SSH user CA pubkey) comes from an operator-provided bundle — a mounted NAS share or
a USB stick on the control node (`trust_src_dir`); `/mnt/nas` isn't available yet.
The host age key is copied from the bundle if present, otherwise generated on the
host (register its printed recipient in [hosts](hosts.md) + the repo `.sops.yaml`).
### Phase 1 connection & auth
The converge workflow reuses the [`charlie-ssh-open`][ssh-open] action: it mints
a short-lived step-ca SSH user cert and opens a multiplex master to the target.
Ansible attaches to that same socket (`ControlPath`), so there is no second cert
or auth handshake.
| Layer | Mechanism | Credential |
| ----- | --------- | ---------- |
| SSH login | step-ca SSH user cert, principal `sys-gitea-runner` ([cert-issuer](cert-issuer.md) `runners` provisioner) | short-lived cert |
| `become` root | `sys-gitea-runner` ∈ LDAP `admins` → `%admins ALL=(ALL) NOPASSWD:ALL` | none (NOPASSWD) |
`sys-gitea-runner` is already in `charlie-deploy` (Docker socket = root-equivalent)
on every deploy target, so a separate sudo identity would add no real boundary;
reusing the runner identity keeps Phase 1 simple and auditable. NOPASSWD is the
linchpin — `become` works over the non-interactive socket with no sudo password
to vault or inject.
The converge workflow runs automatically on push to `main` and fans out one job
per host, serialized via `needs:` — Gitea Actions does not reliably honour
`strategy.max-parallel`, and host jobs share the runner's SSH multiplex socket
and Ansible venv, so they must not overlap (same shape as
[`otelcollector`][otel-deploy]'s deploy). Each job runs the reusable
`converge-host.yml`. **It is currently dry-run-only**: it runs
`ansible-playbook --check` to prove the converge does not break, with the real
apply commented out until the dry runs are proven green across the fleet. When
re-enabled it becomes a dry-run → apply gate (apply only if `--check` is green).
Active order: `morgott` → `melina`; `mohg` stays manual until its Phase 0
bootstrap lands, and macOS/NAS are out of scope for `site.yml` (`linux` group
only).
[ssh-open]: https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/.gitea/actions/charlie-ssh-open
[otel-deploy]: https://gitea.alexandru.macocian.me/Charlie/otelcollector/src/branch/main/.gitea/workflows/deploy.yaml
## Proposed roles
| Role | Provides | Phase |
| ---- | -------- | ----- |
| `trust` | CA roots, SSH user CA, `/etc/charlie/age.key` | 0 |
| `identity` | SSSD/LDAP client config; `%admins` NOPASSWD sudoers | 0 + 1 |
| `nfs` | autofs `/mnt/nas` mount | 1 |
| `identity` | SSSD + LDAP client config; sudo via groups | 1 |
| `docker` | Docker/Compose engine | 1 |
| `deploy-group` | `charlie-deploy`, Docker socket access | 1 |
| `gitea-access` | `sys-gitea-runner` Git + registry creds | 1 |
| `deploy_group` | `charlie-deploy`, Docker socket access | 1 |
| `gitea_access` | `sys-gitea-runner` Git + registry creds | 1 |
| `observability` | otelcollector | 1 |
| `homes` | autofs `/Users` + LDAP automount map | 1 |
`bootstrap.yml` (Phase 0) runs `trust` then `identity`; `site.yml` (Phase 1)
re-asserts `identity` idempotently alongside the rest. The `identity` role must
therefore manage the `%admins` sudoers drop-in so a bare node becomes sudo-able
before its first CI converge.
## Inventory & secrets
- `inventory/` holds the authoritative host list; `host_vars/<host>` carries
@@ -78,7 +152,6 @@ flowchart LR
| Item | Note |
| ---- | ---- |
| macOS (Malenia) | Ansible supports it, but roles branch on OS (launchd vs systemd, no apt, native `/Users`). Decide per-role branching vs a separate playbook. |
| Existing hosts | Converge current hosts in place, or apply only to new nodes first? |
| First-run-only steps | Hostname/DNS and Phase-0 trust are not idempotent CI work; keep them operator-gated. |
| Home creation owner | Per-user home provisioning may belong to the `homes` role or DirectoryAdmin at account creation. See [shared-homes](shared-homes.md). |
| Break-glass | Keep a local sudoer with a local home so LDAP/NFS outages don't lock out provisioning. |