From d9408e96ecb64b0dcd8d3019065bc9c5f4998af8 Mon Sep 17 00:00:00 2001 From: Alexandru Macocian Date: Mon, 4 May 2026 19:02:54 +0200 Subject: [PATCH] Setup base for SOPS Co-authored-by: Copilot --- docs/cert-issuer.md | 6 +-- docs/hosts.md | 17 ++++++- docs/onboarding.md | 101 +++++++++++++++++++++++++++++++++++++-- docs/secrets.md | 112 +++++++++++++++++++++++++++++++++++--------- 4 files changed, 206 insertions(+), 30 deletions(-) diff --git a/docs/cert-issuer.md b/docs/cert-issuer.md index 9fbee70..d5515a9 100644 --- a/docs/cert-issuer.md +++ b/docs/cert-issuer.md @@ -137,16 +137,16 @@ Today [`onboarding.md → Trusted CA`](onboarding.md#trusted-ca) runs `sync-ca.s ### 1. Where does the CA key live -Today: `/mnt/nas/ca/ca.key`, world-readable to anyone with NFS / shell on a host that mounts the share. +Today: lives in step-ca's data dir at `/mnt/nas/data/cert-issuer/step/secrets/`, encrypted at rest with `DOCKER_STEPCA_INIT_PASSWORD` (in `.env`, currently plaintext on the NAS). Options: - **TPM-sealed on the issuer host.** Strongest for an on-prem fleet. Requires the issuer to run on a host with TPM 2.0 (morgott, melina, mohg — yes; mac mini, DSM — no, but the issuer doesn't need to run there). `step-ca` supports `tpmkms` directly. CA key never exists in cleartext on disk. - **YubiHSM 2 / YubiKey.** Even stronger; physical device. Adds hardware to budget. - **PKCS#11 with a software token** ([SoftHSM](https://www.opendnssec.org/softhsm/)). Same API as a real HSM, no hardware. Lets us prototype the path before buying a YubiHSM. Not actually more secure than file-on-disk; useful as a stepping stone. -- **Sealed file at rest, decrypted into memory at issuer start.** SOPS-encrypt `ca.key` to the issuer host's age key (same key store as [secrets.md](secrets.md)), `step-ca` reads it once, plaintext never touches disk. Simplest, ties cleanly to the secrets design. +- **SOPS-sealed `.env` (next step, blocked on nothing).** SOPS-encrypt the `cert-issuer/.env` (which contains `DOCKER_STEPCA_INIT_PASSWORD`) to morgott's host age key + admin's key. Now the unseal password isn't on the NAS in cleartext; cert-issuer's `deploy.sh` decrypts it at start time. Same pattern we're rolling out for every other stack via [secrets.md](secrets.md). The CA private key on disk is still encrypted-at-rest by step-ca itself with that password — moving the password into SOPS shrinks the attack surface from "anyone with NFS read on `/mnt/nas/stacks/cert-issuer/.env`" to "root on morgott." -Recommend: start with the **SOPS-sealed** option (rides on the [secrets.md](secrets.md) infrastructure we're building anyway). Move to **TPM-sealed** as a hardening step once the rest of the system is stable. +Recommended sequence: **SOPS-seal first** (small change, rides on the secrets pilot landing). Move to **TPM-sealed** as a later hardening step. ### 2. Where the issuer runs diff --git a/docs/hosts.md b/docs/hosts.md index be8887b..e2d7411 100644 --- a/docs/hosts.md +++ b/docs/hosts.md @@ -11,4 +11,19 @@ | Malenia | apps | Mac Mini 2012 | — | | Miquella | NAS, exports `/volume1/ubuntu` → `/mnt/nas` (NFS) | Synology DSM | source | -Bootstrap procedure: wiki → **First Time Setup**. +Bootstrap procedure: [onboarding.md](onboarding.md). + +## Age public keys (SOPS) + +Public keys are non-secret. Used as recipients in `Charlie//.sops.yaml` files. Generated per [onboarding.md → Host age key (SOPS)](onboarding.md#host-age-key-sops). Private keys live at `/etc/charlie/age.key` on each host (`0600`, `root:root`). + +| Owner | Public key | +|---|---| +| **admin** | `age19d6mc8arznnwnvtp0xrphlseuwf94p0f3pznszrtmd5wgmjpzqdq9mltvs` | +| morgott | `age1m5ha3fd8psek5pym4spk7encjk4vmsml46eya3eymp7q9d9jre5s7xe84r` | +| melina | `age1xfp878l4ul5zulqqul0u60c44c5wj94cmladgc6jt5jrlmrdrezsd8twh2` | +| mohg | _(generated when first stack on mohg needs SOPS)_ | +| miquella | _(generated when first stack on miquella needs SOPS)_ | +| malenia | _(generated when first stack on malenia needs SOPS)_ | + +Backup of admin private key: vaultwarden Secure Note `Charlie SOPS admin age key`. diff --git a/docs/onboarding.md b/docs/onboarding.md index d1f31c0..a4eead1 100644 --- a/docs/onboarding.md +++ b/docs/onboarding.md @@ -4,6 +4,38 @@ End-to-end recipe for bringing a new host into the fleet. Migrated from the wiki > **Note on step-ca migration.** [`Charlie/cert-issuer`](cert-issuer.md) is deployed; its SSH user CA pubkey now lives at `/mnt/nas/ca/ssh/ssh_user_ca.pub` and is distributed by `sync-ca.sh` exactly as before. The X.509 root at `/mnt/nas/ca/ca.crt` (LDAP TLS) is still the original — not yet migrated. The pending rewrite of `sync-ca.sh` to fetch directly from step-ca's HTTPS endpoint is captured in [Step-ca migration impact](#step-ca-migration-impact). Until then, this section stays accurate. +## Admin age key (one-time) + +Before onboarding any host, generate the **admin age key** on the admin laptop. This key is listed as a recipient on every `.sops.yaml` in `Charlie/*` so secrets stay decryptable / re-encryptable from off-host. Loss = re-key every repo, so keep a backup. Design: [secrets.md → Admin key custody](secrets.md#2-admin-key-custody-decided-option-b). + +```bash +# On the admin laptop (one-time per fleet, NOT per host). +sudo pacman -S age sops # arch; or apt install age sops on debian/ubuntu; + # or brew install age sops on macOS. + +# Generate the keypair. Default location is what sops looks at. +mkdir -p ~/.config/sops/age +test ! -f ~/.config/sops/age/keys.txt || { echo "admin key already exists"; exit 1; } +age-keygen -o ~/.config/sops/age/keys.txt +chmod 600 ~/.config/sops/age/keys.txt + +# Print the pubkey — goes into docs/hosts.md and into every .sops.yaml. +grep '^# public key:' ~/.config/sops/age/keys.txt +``` + +Example output: `# public key: age1adminadminadminadminadminadminadminadminadminadminad` + +### Back up to vaultwarden + +The full file is small (~190 bytes). Store the **entire contents** as a Secure Note: + +- **Name:** `Charlie SOPS admin age key` +- **Notes:** paste the file content (one `# created:` line, one `# public key:` line, one `AGE-SECRET-KEY-1...` line). + +Recover by reversing: paste the note contents back into `~/.config/sops/age/keys.txt`, `chmod 600`. + +--- + ## Order of operations 1. [Hostname](#hostname) @@ -13,10 +45,15 @@ End-to-end recipe for bringing a new host into the fleet. Migrated from the wiki 5. [Metrics (otelcollector)](#metrics-otelcollector) 6. [DNS](#dns) 7. [Trusted CA](#trusted-ca) -8. [LDAP / SSSD](#ldap--sssd) -9. [SSH defaults](#ssh-defaults) +8. [Host age key (SOPS)](#host-age-key-sops) +9. [LDAP / SSSD](#ldap--sssd) +10. [SSH defaults](#ssh-defaults) -Every step except (4) and (8) is required. (4) is required if the host runs containers. (8) is required if humans will SSH into it. +Every step except (4) and (9) is required. (4) is required if the host runs containers. (9) is required if humans will SSH into it. + +One-time tasks not tied to a specific host: + +- [Admin age key (one-time)](#admin-age-key-one-time) — done once, on the admin laptop, before any host gets onboarded. --- @@ -507,6 +544,64 @@ sudo launchctl kickstart -k system/com.openssh.sshd --- +## Host age key (SOPS) + +Each host gets one [age](https://github.com/FiloSottile/age) keypair, used to decrypt SOPS-encrypted secrets at deploy time. Public key is non-secret and gets registered in [`docs/hosts.md`](hosts.md#age-public-keys-sops). Private key never leaves the host. Design: [secrets.md](secrets.md). + +### Ubuntu / Mac + +```bash +# Install age. (Apt on Ubuntu; brew on macOS.) +sudo apt install age # or: brew install age +age --version + +# Generate the key. Refuse to clobber an existing one. +sudo install -d -m 0700 -o root -g root /etc/charlie +test ! -f /etc/charlie/age.key || { echo "age.key already exists; refusing to overwrite"; exit 1; } +sudo sh -c 'age-keygen -o /etc/charlie/age.key 2>&1' +# That prints the public key on stderr. Capture it for the next step. + +sudo chmod 0600 /etc/charlie/age.key +sudo chown root:root /etc/charlie/age.key + +# Print the pubkey one more time for safety (the comment line at the top of +# the key file). This goes into docs/hosts.md. +sudo grep '^# public key:' /etc/charlie/age.key +``` + +Example output: + +``` +# public key: age1xyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyz +``` + +### Register the public key + +Add a row to the [Age public keys (SOPS)](hosts.md#age-public-keys-sops) table in `docs/hosts.md`. Pubkeys are public; commit + push is fine. + +### Make the host actually able to decrypt + +SOPS looks for the age key at `$SOPS_AGE_KEY_FILE`. The `deploy.sh` of stacks that consume SOPS sets this when invoking `sops`. No system-wide config needed. + +```bash +# Sanity-check that root can read the key (sops always runs as root via deploy.sh / sudo). +sudo test -r /etc/charlie/age.key && echo OK +``` + +### Adding this host to a stack's recipients + +When this host needs to decrypt a SOPS-encrypted file in a `Charlie/` repo, add its pubkey to that repo's `.sops.yaml` and re-encrypt: + +```bash +# On the admin laptop, in the stack repo: +$EDITOR .sops.yaml # add the new host's pubkey to the recipients list +sops updatekeys secrets/env.sops.yaml +git commit -am "Grant SOPS access" +git push +``` + +--- + ## LDAP / SSSD Identity: SSSD-backed against `ldaps://ldap.lan` (the [`ldap`](https://gitea.alexandru.macocian.me/Charlie/ldap) stack on morgott). diff --git a/docs/secrets.md b/docs/secrets.md index 0b523ff..3755cde 100644 --- a/docs/secrets.md +++ b/docs/secrets.md @@ -1,6 +1,6 @@ # Secrets sourcing -Status: **draft / decision pending**. Captures the design space so [Phase 2 (runners)](roadmap.md#phase-2--ci) and [Phase 5 → Secrets](roadmap.md#secrets) can land on the same answer. +Status: **decided.** SOPS-in-repo with one `age` key per host. Pilot pending. This doc is now the spec; [`docs/onboarding.md`](onboarding.md#host-age-key-sops) is the operational recipe. ## Goal @@ -77,41 +77,105 @@ SOPS supports several recipient backends. Picking which one(s) we use is the loa | Backend | Fit | Notes | |---|---|---| -| **age key file** | **Leading candidate.** | One private key per host, file at `/etc/charlie/age.key` (root, `0600`). Public key listed in each repo's `.sops.yaml` that the host needs to decrypt. Bootstrap = generate key once, paste pub into the right `.sops.yaml`(s), commit. Heterogeneous hosts (DSM, mac mini) all just need a file. | -| **age via SSH host key** | Promising. | `age` can use an existing SSH ed25519 key as a recipient. Reuses something every host already has. Caveat: SOPS support for SSH-as-age recipients goes through the `ssh-to-age` conversion — workable, slightly more moving parts than a dedicated age key. | +| **age key file** | **Chosen.** | One private key per host, file at `/etc/charlie/age.key` (root, `0600`). Public key listed in each repo's `.sops.yaml` that the host needs to decrypt. Bootstrap = generate key once, paste pub into the right `.sops.yaml`(s), commit. Heterogeneous hosts (DSM, mac mini) all just need a file. | +| **age via SSH host key** | Rejected. | `age` can use an existing SSH ed25519 key as a recipient. Reuses something every host already has. Caveat: SOPS support for SSH-as-age recipients goes through the `ssh-to-age` conversion — workable, slightly more moving parts than a dedicated age key. Per-host key file is simpler and the convenience of "reusing" SSH keys is mostly cosmetic. | | **GPG** | No. | Heavier, key-management UX is worse, no advantage over age. | | **HashiCorp Vault Transit** | No. | Reintroduces the central control plane we're trying to avoid. | | **Cloud KMS** (AWS / GCP / Azure) | No. | Off-prem dependency for an on-prem homelab. | | **TPM-sealed age key** | Maybe later. | Strongest. Most x86 hosts have TPM 2.0; mac mini and DSM do not. Heterogeneity makes it a bad default; reasonable to add per-host as a hardening step after v1. | -| **Vaultwarden-stored age key, fetched at bootstrap** | Hybrid. | Age key lives encrypted in vaultwarden; fetched once during host bootstrap (same flow as CA trust / SSSD setup) and written to `/etc/charlie/age.key`. Rotation = re-fetch. Doesn't solve "how do you authenticate to vaultwarden during bootstrap" — but that's a one-time, human-present operation, not a per-deploy one. | +| **Vaultwarden-stored age key, fetched at bootstrap** | Hybrid; see Open Q #1. | Age key lives encrypted in vaultwarden; fetched once during host bootstrap (same flow as CA trust / SSSD setup) and written to `/etc/charlie/age.key`. Rotation = re-fetch. Doesn't solve "how do you authenticate to vaultwarden during bootstrap" — but that's a one-time, human-present operation, not a per-deploy one. | -### Recommended starting point +### Chosen shape -- **age key per host**, generated at first-time-setup, stored at `/etc/charlie/age.key` (root, `0600`). -- Public key is non-secret; published in [`docs/hosts.md`](hosts.md) and added to the `.sops.yaml` of each stack repo the host owns. -- A second "admin" age key (held by alex, on the admin laptop) is added to *every* repo's `.sops.yaml` so secrets stay decryptable / re-encryptable from off-host. -- **Bootstrap mechanism for the host's age key**: TBD — see [Open question #1](#1-host-age-key-bootstrap). Strong default: generate during first-time-setup, paste pub key into `.sops.yaml`s, commit. +- **One `age` key per host**, generated at first-time-setup, stored at `/etc/charlie/age.key` (root-owned, `0600`). +- Public key is non-secret; published in [`docs/hosts.md`](hosts.md#age-public-keys-sops) and added to the `.sops.yaml` of each stack repo the host owns. +- One **admin ("skeleton") `age` key** held by alex, on the admin laptop, listed in *every* repo's `.sops.yaml` so secrets stay decryptable / re-encryptable from off-host. +- **Per-host keys, not per-stack.** Reasoning: if root on the host is compromised, the per-stack keys would all be readable anyway; if a *container* is compromised, it can't reach `/etc/charlie/age.key` because that file isn't bind-mounted into containers. Container isolation already provides the per-stack property. + +### Pilot: `Charlie/ddclient` + +[`ddclient`](https://gitea.alexandru.macocian.me/Charlie/ddclient) is the Tier 1 pilot — single host (morgott), tiny `.env` (one Cloudflare token-ish thing), no downstream dependencies. The recipe: + +```bash +# On the admin laptop. Assumes age + sops installed (see onboarding.md). + +cd ~/Dev/ddclient + +# 1. Drop the .sops.yaml. Recipients = morgott's pubkey + admin's pubkey. +# Pubkeys come from docs/hosts.md. +cat > .sops.yaml <<'EOF' +creation_rules: + - path_regex: secrets/.*\.sops\.(ya?ml|env|json)$ + age: >- + , + +EOF + +# 2. Take the existing .env, encrypt it. +mkdir -p secrets +sops --encrypt --input-type dotenv --output-type yaml \ + /mnt/nas/stacks/ddclient/.env > secrets/env.sops.yaml +# Verify (decrypt back, diff): +sops --decrypt secrets/env.sops.yaml | diff - /mnt/nas/stacks/ddclient/.env + +# 3. Wire deploy.sh to materialize plaintext at deploy time. +cat > deploy.sh <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" +trap 'rm -f .env' EXIT +sops --decrypt secrets/env.sops.yaml > .env +chmod 600 .env +sudo docker compose --env-file .env up -d +EOF +chmod +x deploy.sh + +# 4. Commit + push. +git add .sops.yaml secrets/env.sops.yaml deploy.sh +git commit -m "Adopt SOPS for secrets" +git push + +# 5. On morgott: pull, run the new deploy.sh, verify ddclient still works. +ssh morgott.lan 'cd /mnt/nas/stacks/ddclient && git pull && ./deploy.sh' +# Last resort if it breaks: original .env is still on morgott; falls back to +# `sudo docker compose up -d` as before. + +# 6. Once verified, remove the plaintext .env from morgott. +ssh morgott.lan 'rm /mnt/nas/stacks/ddclient/.env' +``` + +Success = ddclient is up after `./deploy.sh`, no plaintext `.env` on morgott. Then we roll the same shape across the rest of the migrated stacks, one at a time. ## Open questions -### 1. Host age key bootstrap +### 1. Host age key bootstrap (decided: option a) How does the key get onto a freshly-installed host? -- **Generate in place during first-time-setup**, capture the pub key, commit it into the relevant `.sops.yaml`s. Private key never leaves the host. Simple, recommended. -- **Mint centrally** (admin laptop), copy to the host over SSH during bootstrap. Easier to pre-stage `.sops.yaml`s before the host exists, but the key crosses a wire once. -- **Store in vaultwarden**, fetch at bootstrap. Lets you re-image a host without rotating its key. Buys: re-image without `sops updatekeys` churn. Costs: vaultwarden has to be reachable (and authenticatable-to) during host bootstrap. +- **✅ Generate in place during first-time-setup.** Capture the pub key, commit it into the relevant `.sops.yaml`s. Private key never leaves the host. Simple. **Chosen.** +- Mint centrally (admin laptop), copy to the host over SSH during bootstrap. Easier to pre-stage `.sops.yaml`s before the host exists, but the key crosses a wire once. +- Store in vaultwarden, fetch at bootstrap. Lets you re-image a host without rotating its key. Buys: re-image without `sops updatekeys` churn. Costs: vaultwarden has to be reachable (and authenticatable-to) during host bootstrap. -Recommend option 1 unless re-imaging frequency makes the rotation cost felt. +Rationale: re-imaging is rare; `sops updatekeys` for the (small) set of stacks owned by a re-imaged host is cheap. In-place generation has the smallest moving parts. -### 2. Admin key custody +### 2. Admin key custody (decided: option b) The "decrypt anywhere" admin key is the master skeleton key for the fleet. Where does it live? - Admin laptop only, file at `~/.config/sops/age/keys.txt` (`0600`). Loss = re-key every repo. -- Admin laptop **and** vaultwarden as backup. Loss recoverable. +- **✅ Admin laptop and vaultwarden as backup.** Loss recoverable. **Chosen.** - Hardware-backed (YubiKey + age-plugin-yubikey). Strongest, adds a hardware dependency for routine ops. +Vaultwarden item shape (Secure Note): + +``` +Name: Charlie SOPS admin age key +Notes: + (includes the # public key: ... comment line and the AGE-SECRET-KEY-1... line) +``` + +Generate the admin key once (recipe in [`docs/onboarding.md → Admin age key (one-time)`](onboarding.md#admin-age-key-one-time)). + ### 3. Per-environment / per-host secrets in the same repo Shared stacks ([`otelcollector`](https://gitea.alexandru.macocian.me/Charlie/otelcollector), [`portainer-agent`](https://gitea.alexandru.macocian.me/Charlie/portainer-agent)) need different secrets per host. Two shapes: @@ -130,8 +194,8 @@ Recommend the per-file shape — it's the only one consistent with the constrain ### 5. Vaultwarden's role going forward - Pure human use case (browser autofill, secure notes). No change. -- Optional: store the age private key as a backup item per host. Useful for re-imaging. -- Optional: store the admin age key (if not on YubiKey). +- Holds the **admin age key** as a Secure Note (see Open Q #2). +- Optional: store each host's age private key as a Secure Note for re-image convenience. Trade-off: convenience now vs. one more place to leak. Not done by default. Vaultwarden becomes a backup / human convenience, not a runtime dependency. @@ -152,8 +216,10 @@ Both currently in vaultwarden as the source of truth; `.env` on morgott is the w ## Next steps -1. Settle [open questions 1 + 2](#open-questions) here, in-place. -2. Generate age keys for morgott + melina + mohg + miquella + malenia. Capture pub keys in [`docs/hosts.md`](hosts.md). -3. Spike on [`Charlie/ddclient`](https://gitea.alexandru.macocian.me/Charlie/ddclient) (Tier 1, single-host, tiny `.env`): add `.sops.yaml`, commit `secrets/env.sops.yaml`, wire into `deploy.sh`. -4. Roll across migrated stacks one tier at a time. -5. Fold the result into the [`Charlie/runners` design](roadmap.md#phase-2--ci) before any pipeline depends on it. +1. ✅ Settle [open questions 1 + 2](#open-questions). Done. +2. Generate the **admin age key** on the laptop (one-time). Store in vaultwarden. Recipe: [`onboarding.md → Admin age key (one-time)`](onboarding.md#admin-age-key-one-time). +3. Generate **morgott's host age key**. Capture pubkey in [`docs/hosts.md`](hosts.md#age-public-keys-sops). Recipe: [`onboarding.md → Host age key (SOPS)`](onboarding.md#host-age-key-sops). +4. Spike on [`Charlie/ddclient`](https://gitea.alexandru.macocian.me/Charlie/ddclient) per [Pilot recipe](#pilot-charlieddclient). +5. Roll across migrated stacks one tier at a time. +6. Generate age keys for melina + mohg + miquella + malenia as their stacks need SOPS. +7. Fold the result into the [`Charlie/runners` design](roadmap.md#phase-2--ci) before any pipeline depends on it.