project-charlie/docs/provisioning.md

Host provisioning

Status: planned. Tracking doc for the host auto-provisioning tool that takes a node from bare to a fully working Charlie deploy target, step by step. Pairs with shared home directories, whose autofs /Users work is one provisioning role.

Decisions

Area	Decision
Tool	Ansible (declarative, idempotent, cross-platform incl. macOS).
Phases	Phase 0 trust bootstrap is operator-run; Phase 1+ convergence is CI-driven.
Phase 1 invocation	Gitea Actions on push to main: per-host jobs serialized via needs:, each runs ansible-playbook --check (dry-run-only for now; apply gated/disabled). Mirrors otelcollector.
Phase 1 auth	SSH as sys-gitea-runner (step-ca cert); become root via the LDAP admins group's NOPASSWD sudo rule.
Repo	New Charlie/provisioning (playbooks, roles, inventory, Phase-1 workflow).
Inventory	Ansible inventory is authoritative; hosts links to it instead of duplicating host facts.
Secrets	SOPS/age, matching the rest of Charlie.

Provisioning steps

Dependency-ordered. "Bootstrap" steps must exist before the node can be a normal CI/SSH-cert target.

#	Step	Depends on	Phase
1	Hostname + Marika DNS/DHCP entry	—	off-host
2	CA trust: X.509 root + SSH user CA	1	0
3	/etc/charlie/age.key	—	0
4	SSSD/LDAP client config (resolves sys-gitea-runner + admins)	2, 3	0
5	admins NOPASSWD sudoers (/etc/sudoers.d/90-ldap-admins)	4	0
6	NFS /mnt/nas autofs mount	1	1
7	Docker / Compose	—	1
8	charlie-deploy group + Docker socket access	7	1
9	sys-gitea-runner Git + registry creds	2, 3	1
10	otelcollector	6, 8	1
11	autofs /Users homes (shared-homes)	4, 6	1

Two-phase model

flowchart LR
  Op[Operator + Sherlock] -->|Phase 0: out-of-band SSH| Bare[Bare host]
  Bare -->|"Phase 0: CA + SSH CA, age key, SSSD, admins sudoers"| Trusted[Trusted host]
  Runner[Gitea runner] -->|Phase 1: step-ca SSH cert| Trusted
  Trusted -->|converged| Target[Deploy target]

• Phase 0 — trust + identity bootstrap (operator-run). Resolves the chicken-and-egg: a bare node has no age key, no CA trust, and no LDAP identity, so the CI runner cannot SSH in as sys-gitea-runner or become root yet. Operator connects out-of-band (console / initial key) and lays down, in order: CA roots + SSH user CA, /etc/charlie/age.key, the SSSD/LDAP client config (so sys-gitea-runner and admins resolve), and the %admins NOPASSWD sudoers file (/etc/sudoers.d/90-ldap-admins). Minimal, run once. NFS, Docker, registry creds, otelcollector, and /Users homes all converge in Phase 1.
• Phase 1+ — convergence (CI-driven). Once trust and LDAP land, the host is a normal step-ca SSH target. A Gitea workflow runs the playbook idempotently; re-runnable on every host forever.

Phase 0 — minimal manual bootstrap

Only what the CI runner needs to log in and escalate must exist before the first converge. Everything else is Phase 1 (the tool converges it as root):

Must be manual (Phase 0)	Why it can't wait for Phase 1
CA trust (X.509 root + SSH user CA)	sshd must accept the step-ca cert to let the runner in.
/etc/charlie/age.key	needed to decrypt the LDAP bind secret for SSSD.
SSSD/LDAP client config	so sys-gitea-runner + admins resolve at login.
%admins NOPASSWD sudoers	so become works on the very first converge.

• NFS and Docker are not Phase 0. They are installed/configured as root by Phase 1, after the runner connects and escalates. Keeping them manual would be redundant.
• Home-dir wrinkle. sys-gitea-runner's home is /Users/sys-gitea-runner (NFS-backed). On a truly bare node, before the nfs/homes roles run, that path won't resolve. Login still works; point Ansible's remote_tmp at /tmp for the first converge.

bootstrap.yml does all of the above (trust then identity). It is operator-run out-of-band, e.g. ansible-playbook playbooks/bootstrap.yml -l <host> -u <initial-admin> --ask-become-pass -e trust_src_dir=<bundle>. The public CA material (X.509 root + SSH user CA pubkey) comes from an operator-provided bundle — a mounted NAS share or a USB stick on the control node (trust_src_dir); /mnt/nas isn't available yet. The host age key is copied from the bundle if present, otherwise generated on the host (register its printed recipient in hosts + the repo .sops.yaml).

Phase 1 connection & auth

The converge workflow reuses the charlie-ssh-open action: it mints a short-lived step-ca SSH user cert and opens a multiplex master to the target. Ansible attaches to that same socket (ControlPath), so there is no second cert or auth handshake.

Layer	Mechanism	Credential
SSH login	step-ca SSH user cert, principal sys-gitea-runner (cert-issuer runners provisioner)	short-lived cert
become root	sys-gitea-runner ∈ LDAP admins → %admins ALL=(ALL) NOPASSWD:ALL	none (NOPASSWD)

sys-gitea-runner is already in charlie-deploy (Docker socket = root-equivalent) on every deploy target, so a separate sudo identity would add no real boundary; reusing the runner identity keeps Phase 1 simple and auditable. NOPASSWD is the linchpin — become works over the non-interactive socket with no sudo password to vault or inject.

The converge workflow runs automatically on push to main and fans out one job per host, serialized via needs: — Gitea Actions does not reliably honour strategy.max-parallel, and host jobs share the runner's SSH multiplex socket and Ansible venv, so they must not overlap (same shape as otelcollector's deploy). Each job runs the reusable converge-host.yml. It is currently dry-run-only: it runs ansible-playbook --check to prove the converge does not break, with the real apply commented out until the dry runs are proven green across the fleet. When re-enabled it becomes a dry-run → apply gate (apply only if --check is green). Active order: morgott → melina; mohg stays manual until its Phase 0 bootstrap lands, and macOS/NAS are out of scope for site.yml (linux group only).

Proposed roles

Role	Provides	Phase
trust	CA roots, SSH user CA, /etc/charlie/age.key	0
identity	SSSD/LDAP client config; %admins NOPASSWD sudoers	0 + 1
nfs	autofs /mnt/nas mount	1
docker	Docker/Compose engine	1
deploy_group	charlie-deploy, Docker socket access	1
gitea_access	sys-gitea-runner Git + registry creds	1
observability	otelcollector	1
homes	autofs /Users + LDAP automount map	1

bootstrap.yml (Phase 0) runs trust then identity; site.yml (Phase 1) re-asserts identity idempotently alongside the rest. The identity role must therefore manage the %admins sudoers drop-in so a bare node becomes sudo-able before its first CI converge.

Inventory & secrets

• inventory/ holds the authoritative host list; host_vars/<host> carries role, OS family, SOPS recipient pubkey, and whether the host runs stacks.
• Per-host/group secrets are SOPS-encrypted under the repo's .sops.yaml, decrypted at run time with the host age key (Phase 1) or the operator key (Phase 0). Same pattern as stack repos.

Open questions

Item	Note
macOS (Malenia)	Ansible supports it, but roles branch on OS (launchd vs systemd, no apt, native /Users). Decide per-role branching vs a separate playbook.
First-run-only steps	Hostname/DNS and Phase-0 trust are not idempotent CI work; keep them operator-gated.
Home creation owner	Per-user home provisioning may belong to the homes role or DirectoryAdmin at account creation. See shared-homes.
Break-glass	Keep a local sudoer with a local home so LDAP/NFS outages don't lock out provisioning.