project-charlie/docs/provisioning.md

# Host provisioning

Status: **planned**. Tracking doc for the host auto-provisioning tool that takes
a node from bare to a fully working Charlie deploy target, step by step. Pairs
with [shared home directories](shared-homes.md), whose autofs `/Users` work is
one provisioning role.

## Decisions

| Area | Decision |
| ---- | -------- |
| Tool | Ansible (declarative, idempotent, cross-platform incl. macOS). |
| Phases | Phase 0 trust bootstrap is operator-run; Phase 1+ convergence is CI-driven. |
| Phase 1 invocation | Gitea Actions on push to `main`: per-host jobs serialized via `needs:`, each runs `ansible-playbook --check` (dry-run-only for now; apply gated/disabled). Mirrors `otelcollector`. |
| Phase 1 auth | SSH as `sys-gitea-runner` (step-ca cert); `become` root via the LDAP `admins` group's NOPASSWD sudo rule. |
| Repo | New `Charlie/provisioning` (playbooks, roles, inventory, Phase-1 workflow). |
| Inventory | Ansible inventory is authoritative; [hosts](hosts.md) links to it instead of duplicating host facts. |
| Secrets | SOPS/age, matching the rest of Charlie. |

## Provisioning steps

Dependency-ordered. "Bootstrap" steps must exist before the node can be a normal
CI/SSH-cert target.

| # | Step | Depends on | Phase |
| - | ---- | ---------- | ----- |
| 1 | Hostname + Marika DNS/DHCP entry | — | off-host |
| 2 | CA trust: X.509 root + SSH user CA | 1 | 0 |
| 3 | `/etc/charlie/age.key` | — | 0 |
| 4 | SSSD/LDAP client config (resolves `sys-gitea-runner` + `admins`) | 2, 3 | 0 |
| 5 | `admins` NOPASSWD sudoers (`/etc/sudoers.d/90-ldap-admins`) | 4 | 0 |
| 6 | NFS `/mnt/nas` autofs mount | 1 | 1 |
| 7 | Docker / Compose | — | 1 |
| 8 | `charlie-deploy` group + Docker socket access | 7 | 1 |
| 9 | `sys-gitea-runner` Git + registry creds | 2, 3 | 1 |
| 10 | otelcollector | 6, 8 | 1 |
| 11 | autofs `/Users` homes ([shared-homes](shared-homes.md)) | 4, 6 | 1 |

## Two-phase model

```mermaid
flowchart LR
  Op[Operator + Sherlock] -->|Phase 0: out-of-band SSH| Bare[Bare host]
  Bare -->|"Phase 0: CA + SSH CA, age key, SSSD, admins sudoers"| Trusted[Trusted host]
  Runner[Gitea runner] -->|Phase 1: step-ca SSH cert| Trusted
  Trusted -->|converged| Target[Deploy target]
```

- **Phase 0 — trust + identity bootstrap (operator-run).** Resolves the
  chicken-and-egg: a bare node has no age key, no CA trust, and no LDAP
  identity, so the CI runner cannot SSH in as `sys-gitea-runner` or `become`
  root yet. Operator connects out-of-band (console / initial key) and lays down,
  in order: CA roots + SSH user CA, `/etc/charlie/age.key`, the SSSD/LDAP client
  config (so `sys-gitea-runner` and `admins` resolve), and the `%admins`
  NOPASSWD sudoers file (`/etc/sudoers.d/90-ldap-admins`). Minimal, run once.
  NFS, Docker, registry creds, otelcollector, and `/Users` homes all converge in
  Phase 1.
- **Phase 1+ — convergence (CI-driven).** Once trust and LDAP land, the host is
  a normal step-ca SSH target. A Gitea workflow runs the playbook idempotently;
  re-runnable on every host forever.

### Phase 0 — minimal manual bootstrap

Only what the CI runner needs to *log in and escalate* must exist before the
first converge. Everything else is Phase 1 (the tool converges it as root):

| Must be manual (Phase 0) | Why it can't wait for Phase 1 |
| ------------------------ | ----------------------------- |
| CA trust (X.509 root + SSH user CA) | sshd must accept the step-ca cert to let the runner in. |
| `/etc/charlie/age.key` | needed to decrypt the LDAP bind secret for SSSD. |
| SSSD/LDAP client config | so `sys-gitea-runner` + `admins` resolve at login. |
| `%admins` NOPASSWD sudoers | so `become` works on the very first converge. |

- **NFS and Docker are *not* Phase 0.** They are installed/configured as root by
  Phase 1, after the runner connects and escalates. Keeping them manual would be
  redundant.
- **Home-dir wrinkle.** `sys-gitea-runner`'s home is `/Users/sys-gitea-runner`
  (NFS-backed). On a truly bare node, before the `nfs`/`homes` roles run, that
  path won't resolve. Login still works; point Ansible's `remote_tmp` at `/tmp`
  for the first converge.

`bootstrap.yml` does all of the above (`trust` then `identity`). It is operator-run
out-of-band, e.g. `ansible-playbook playbooks/bootstrap.yml -l <host> -u <initial-admin>
--ask-become-pass -e trust_src_dir=<bundle>`. The public CA material (X.509 root +
SSH user CA pubkey) comes from an operator-provided bundle — a mounted NAS share or
a USB stick on the control node (`trust_src_dir`); `/mnt/nas` isn't available yet.
The host age key is copied from the bundle if present, otherwise generated on the
host (register its printed recipient in [hosts](hosts.md) + the repo `.sops.yaml`).

### Phase 1 connection & auth

The converge workflow reuses the [`charlie-ssh-open`][ssh-open] action: it mints
a short-lived step-ca SSH user cert and opens a multiplex master to the target.
Ansible attaches to that same socket (`ControlPath`), so there is no second cert
or auth handshake.

| Layer | Mechanism | Credential |
| ----- | --------- | ---------- |
| SSH login | step-ca SSH user cert, principal `sys-gitea-runner` ([cert-issuer](cert-issuer.md) `runners` provisioner) | short-lived cert |
| `become` root | `sys-gitea-runner` ∈ LDAP `admins` → `%admins ALL=(ALL) NOPASSWD:ALL` | none (NOPASSWD) |

`sys-gitea-runner` is already in `charlie-deploy` (Docker socket = root-equivalent)
on every deploy target, so a separate sudo identity would add no real boundary;
reusing the runner identity keeps Phase 1 simple and auditable. NOPASSWD is the
linchpin — `become` works over the non-interactive socket with no sudo password
to vault or inject.

The converge workflow runs automatically on push to `main` and fans out one job
per host, serialized via `needs:` — Gitea Actions does not reliably honour
`strategy.max-parallel`, and host jobs share the runner's SSH multiplex socket
and Ansible venv, so they must not overlap (same shape as
[`otelcollector`][otel-deploy]'s deploy). Each job runs the reusable
`converge-host.yml`. **It is currently dry-run-only**: it runs
`ansible-playbook --check` to prove the converge does not break, with the real
apply commented out until the dry runs are proven green across the fleet. When
re-enabled it becomes a dry-run → apply gate (apply only if `--check` is green).
Active order: `morgott` → `melina`; `mohg` stays manual until its Phase 0
bootstrap lands, and macOS/NAS are out of scope for `site.yml` (`linux` group
only).

[ssh-open]: https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/.gitea/actions/charlie-ssh-open
[otel-deploy]: https://gitea.alexandru.macocian.me/Charlie/otelcollector/src/branch/main/.gitea/workflows/deploy.yaml

## Proposed roles

| Role | Provides | Phase |
| ---- | -------- | ----- |
| `trust` | CA roots, SSH user CA, `/etc/charlie/age.key` | 0 |
| `identity` | SSSD/LDAP client config; `%admins` NOPASSWD sudoers | 0 + 1 |
| `nfs` | autofs `/mnt/nas` mount | 1 |
| `docker` | Docker/Compose engine | 1 |
| `deploy_group` | `charlie-deploy`, Docker socket access | 1 |
| `gitea_access` | `sys-gitea-runner` Git + registry creds | 1 |
| `observability` | otelcollector | 1 |
| `homes` | autofs `/Users` + LDAP automount map | 1 |

`bootstrap.yml` (Phase 0) runs `trust` then `identity`; `site.yml` (Phase 1)
re-asserts `identity` idempotently alongside the rest. The `identity` role must
therefore manage the `%admins` sudoers drop-in so a bare node becomes sudo-able
before its first CI converge.

## Inventory & secrets

- `inventory/` holds the authoritative host list; `host_vars/<host>` carries
  role, OS family, SOPS recipient pubkey, and whether the host runs stacks.
- Per-host/group secrets are SOPS-encrypted under the repo's `.sops.yaml`,
  decrypted at run time with the host age key (Phase 1) or the operator key
  (Phase 0). Same pattern as stack repos.

## Open questions

| Item | Note |
| ---- | ---- |
| macOS (Malenia) | Ansible supports it, but roles branch on OS (launchd vs systemd, no apt, native `/Users`). Decide per-role branching vs a separate playbook. |
| First-run-only steps | Hostname/DNS and Phase-0 trust are not idempotent CI work; keep them operator-gated. |
| Home creation owner | Per-user home provisioning may belong to the `homes` role or DirectoryAdmin at account creation. See [shared-homes](shared-homes.md). |
| Break-glass | Keep a local sudoer with a local home so LDAP/NFS outages don't lock out provisioning. |