Setup base for SOPS

Co-authored-by: Copilot <copilot@github.com>
This commit is contained in:
Alexandru Macocian
2026-05-04 19:02:54 +02:00
parent 3d742a5024
commit d9408e96ec
4 changed files with 206 additions and 30 deletions
+3 -3
View File
@@ -137,16 +137,16 @@ Today [`onboarding.md → Trusted CA`](onboarding.md#trusted-ca) runs `sync-ca.s
### 1. Where does the CA key live
Today: `/mnt/nas/ca/ca.key`, world-readable to anyone with NFS / shell on a host that mounts the share.
Today: lives in step-ca's data dir at `/mnt/nas/data/cert-issuer/step/secrets/`, encrypted at rest with `DOCKER_STEPCA_INIT_PASSWORD` (in `.env`, currently plaintext on the NAS).
Options:
- **TPM-sealed on the issuer host.** Strongest for an on-prem fleet. Requires the issuer to run on a host with TPM 2.0 (morgott, melina, mohg — yes; mac mini, DSM — no, but the issuer doesn't need to run there). `step-ca` supports `tpmkms` directly. CA key never exists in cleartext on disk.
- **YubiHSM 2 / YubiKey.** Even stronger; physical device. Adds hardware to budget.
- **PKCS#11 with a software token** ([SoftHSM](https://www.opendnssec.org/softhsm/)). Same API as a real HSM, no hardware. Lets us prototype the path before buying a YubiHSM. Not actually more secure than file-on-disk; useful as a stepping stone.
- **Sealed file at rest, decrypted into memory at issuer start.** SOPS-encrypt `ca.key` to the issuer host's age key (same key store as [secrets.md](secrets.md)), `step-ca` reads it once, plaintext never touches disk. Simplest, ties cleanly to the secrets design.
- **SOPS-sealed `.env` (next step, blocked on nothing).** SOPS-encrypt the `cert-issuer/.env` (which contains `DOCKER_STEPCA_INIT_PASSWORD`) to morgott's host age key + admin's key. Now the unseal password isn't on the NAS in cleartext; cert-issuer's `deploy.sh` decrypts it at start time. Same pattern we're rolling out for every other stack via [secrets.md](secrets.md). The CA private key on disk is still encrypted-at-rest by step-ca itself with that password — moving the password into SOPS shrinks the attack surface from "anyone with NFS read on `/mnt/nas/stacks/cert-issuer/.env`" to "root on morgott."
Recommend: start with the **SOPS-sealed** option (rides on the [secrets.md](secrets.md) infrastructure we're building anyway). Move to **TPM-sealed** as a hardening step once the rest of the system is stable.
Recommended sequence: **SOPS-seal first** (small change, rides on the secrets pilot landing). Move to **TPM-sealed** as a later hardening step.
### 2. Where the issuer runs
+16 -1
View File
@@ -11,4 +11,19 @@
| Malenia | apps | Mac Mini 2012 | — |
| Miquella | NAS, exports `/volume1/ubuntu``/mnt/nas` (NFS) | Synology DSM | source |
Bootstrap procedure: wiki → **First Time Setup**.
Bootstrap procedure: [onboarding.md](onboarding.md).
## Age public keys (SOPS)
Public keys are non-secret. Used as recipients in `Charlie/<stack>/.sops.yaml` files. Generated per [onboarding.md → Host age key (SOPS)](onboarding.md#host-age-key-sops). Private keys live at `/etc/charlie/age.key` on each host (`0600`, `root:root`).
| Owner | Public key |
|---|---|
| **admin** | `age19d6mc8arznnwnvtp0xrphlseuwf94p0f3pznszrtmd5wgmjpzqdq9mltvs` |
| morgott | `age1m5ha3fd8psek5pym4spk7encjk4vmsml46eya3eymp7q9d9jre5s7xe84r` |
| melina | `age1xfp878l4ul5zulqqul0u60c44c5wj94cmladgc6jt5jrlmrdrezsd8twh2` |
| mohg | _(generated when first stack on mohg needs SOPS)_ |
| miquella | _(generated when first stack on miquella needs SOPS)_ |
| malenia | _(generated when first stack on malenia needs SOPS)_ |
Backup of admin private key: vaultwarden Secure Note `Charlie SOPS admin age key`.
+98 -3
View File
@@ -4,6 +4,38 @@ End-to-end recipe for bringing a new host into the fleet. Migrated from the wiki
> **Note on step-ca migration.** [`Charlie/cert-issuer`](cert-issuer.md) is deployed; its SSH user CA pubkey now lives at `/mnt/nas/ca/ssh/ssh_user_ca.pub` and is distributed by `sync-ca.sh` exactly as before. The X.509 root at `/mnt/nas/ca/ca.crt` (LDAP TLS) is still the original — not yet migrated. The pending rewrite of `sync-ca.sh` to fetch directly from step-ca's HTTPS endpoint is captured in [Step-ca migration impact](#step-ca-migration-impact). Until then, this section stays accurate.
## Admin age key (one-time)
Before onboarding any host, generate the **admin age key** on the admin laptop. This key is listed as a recipient on every `.sops.yaml` in `Charlie/*` so secrets stay decryptable / re-encryptable from off-host. Loss = re-key every repo, so keep a backup. Design: [secrets.md → Admin key custody](secrets.md#2-admin-key-custody-decided-option-b).
```bash
# On the admin laptop (one-time per fleet, NOT per host).
sudo pacman -S age sops # arch; or apt install age sops on debian/ubuntu;
# or brew install age sops on macOS.
# Generate the keypair. Default location is what sops looks at.
mkdir -p ~/.config/sops/age
test ! -f ~/.config/sops/age/keys.txt || { echo "admin key already exists"; exit 1; }
age-keygen -o ~/.config/sops/age/keys.txt
chmod 600 ~/.config/sops/age/keys.txt
# Print the pubkey — goes into docs/hosts.md and into every .sops.yaml.
grep '^# public key:' ~/.config/sops/age/keys.txt
```
Example output: `# public key: age1adminadminadminadminadminadminadminadminadminadminad`
### Back up to vaultwarden
The full file is small (~190 bytes). Store the **entire contents** as a Secure Note:
- **Name:** `Charlie SOPS admin age key`
- **Notes:** paste the file content (one `# created:` line, one `# public key:` line, one `AGE-SECRET-KEY-1...` line).
Recover by reversing: paste the note contents back into `~/.config/sops/age/keys.txt`, `chmod 600`.
---
## Order of operations
1. [Hostname](#hostname)
@@ -13,10 +45,15 @@ End-to-end recipe for bringing a new host into the fleet. Migrated from the wiki
5. [Metrics (otelcollector)](#metrics-otelcollector)
6. [DNS](#dns)
7. [Trusted CA](#trusted-ca)
8. [LDAP / SSSD](#ldap--sssd)
9. [SSH defaults](#ssh-defaults)
8. [Host age key (SOPS)](#host-age-key-sops)
9. [LDAP / SSSD](#ldap--sssd)
10. [SSH defaults](#ssh-defaults)
Every step except (4) and (8) is required. (4) is required if the host runs containers. (8) is required if humans will SSH into it.
Every step except (4) and (9) is required. (4) is required if the host runs containers. (9) is required if humans will SSH into it.
One-time tasks not tied to a specific host:
- [Admin age key (one-time)](#admin-age-key-one-time) — done once, on the admin laptop, before any host gets onboarded.
---
@@ -507,6 +544,64 @@ sudo launchctl kickstart -k system/com.openssh.sshd
---
## Host age key (SOPS)
Each host gets one [age](https://github.com/FiloSottile/age) keypair, used to decrypt SOPS-encrypted secrets at deploy time. Public key is non-secret and gets registered in [`docs/hosts.md`](hosts.md#age-public-keys-sops). Private key never leaves the host. Design: [secrets.md](secrets.md).
### Ubuntu / Mac
```bash
# Install age. (Apt on Ubuntu; brew on macOS.)
sudo apt install age # or: brew install age
age --version
# Generate the key. Refuse to clobber an existing one.
sudo install -d -m 0700 -o root -g root /etc/charlie
test ! -f /etc/charlie/age.key || { echo "age.key already exists; refusing to overwrite"; exit 1; }
sudo sh -c 'age-keygen -o /etc/charlie/age.key 2>&1'
# That prints the public key on stderr. Capture it for the next step.
sudo chmod 0600 /etc/charlie/age.key
sudo chown root:root /etc/charlie/age.key
# Print the pubkey one more time for safety (the comment line at the top of
# the key file). This goes into docs/hosts.md.
sudo grep '^# public key:' /etc/charlie/age.key
```
Example output:
```
# public key: age1xyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyz
```
### Register the public key
Add a row to the [Age public keys (SOPS)](hosts.md#age-public-keys-sops) table in `docs/hosts.md`. Pubkeys are public; commit + push is fine.
### Make the host actually able to decrypt
SOPS looks for the age key at `$SOPS_AGE_KEY_FILE`. The `deploy.sh` of stacks that consume SOPS sets this when invoking `sops`. No system-wide config needed.
```bash
# Sanity-check that root can read the key (sops always runs as root via deploy.sh / sudo).
sudo test -r /etc/charlie/age.key && echo OK
```
### Adding this host to a stack's recipients
When this host needs to decrypt a SOPS-encrypted file in a `Charlie/<stack>` repo, add its pubkey to that repo's `.sops.yaml` and re-encrypt:
```bash
# On the admin laptop, in the stack repo:
$EDITOR .sops.yaml # add the new host's pubkey to the recipients list
sops updatekeys secrets/env.sops.yaml
git commit -am "Grant <host> SOPS access"
git push
```
---
## LDAP / SSSD
Identity: SSSD-backed against `ldaps://ldap.lan` (the [`ldap`](https://gitea.alexandru.macocian.me/Charlie/ldap) stack on morgott).
+89 -23
View File
@@ -1,6 +1,6 @@
# Secrets sourcing
Status: **draft / decision pending**. Captures the design space so [Phase 2 (runners)](roadmap.md#phase-2--ci) and [Phase 5 → Secrets](roadmap.md#secrets) can land on the same answer.
Status: **decided.** SOPS-in-repo with one `age` key per host. Pilot pending. This doc is now the spec; [`docs/onboarding.md`](onboarding.md#host-age-key-sops) is the operational recipe.
## Goal
@@ -77,41 +77,105 @@ SOPS supports several recipient backends. Picking which one(s) we use is the loa
| Backend | Fit | Notes |
|---|---|---|
| **age key file** | **Leading candidate.** | One private key per host, file at `/etc/charlie/age.key` (root, `0600`). Public key listed in each repo's `.sops.yaml` that the host needs to decrypt. Bootstrap = generate key once, paste pub into the right `.sops.yaml`(s), commit. Heterogeneous hosts (DSM, mac mini) all just need a file. |
| **age via SSH host key** | Promising. | `age` can use an existing SSH ed25519 key as a recipient. Reuses something every host already has. Caveat: SOPS support for SSH-as-age recipients goes through the `ssh-to-age` conversion — workable, slightly more moving parts than a dedicated age key. |
| **age key file** | **Chosen.** | One private key per host, file at `/etc/charlie/age.key` (root, `0600`). Public key listed in each repo's `.sops.yaml` that the host needs to decrypt. Bootstrap = generate key once, paste pub into the right `.sops.yaml`(s), commit. Heterogeneous hosts (DSM, mac mini) all just need a file. |
| **age via SSH host key** | Rejected. | `age` can use an existing SSH ed25519 key as a recipient. Reuses something every host already has. Caveat: SOPS support for SSH-as-age recipients goes through the `ssh-to-age` conversion — workable, slightly more moving parts than a dedicated age key. Per-host key file is simpler and the convenience of "reusing" SSH keys is mostly cosmetic. |
| **GPG** | No. | Heavier, key-management UX is worse, no advantage over age. |
| **HashiCorp Vault Transit** | No. | Reintroduces the central control plane we're trying to avoid. |
| **Cloud KMS** (AWS / GCP / Azure) | No. | Off-prem dependency for an on-prem homelab. |
| **TPM-sealed age key** | Maybe later. | Strongest. Most x86 hosts have TPM 2.0; mac mini and DSM do not. Heterogeneity makes it a bad default; reasonable to add per-host as a hardening step after v1. |
| **Vaultwarden-stored age key, fetched at bootstrap** | Hybrid. | Age key lives encrypted in vaultwarden; fetched once during host bootstrap (same flow as CA trust / SSSD setup) and written to `/etc/charlie/age.key`. Rotation = re-fetch. Doesn't solve "how do you authenticate to vaultwarden during bootstrap" — but that's a one-time, human-present operation, not a per-deploy one. |
| **Vaultwarden-stored age key, fetched at bootstrap** | Hybrid; see Open Q #1. | Age key lives encrypted in vaultwarden; fetched once during host bootstrap (same flow as CA trust / SSSD setup) and written to `/etc/charlie/age.key`. Rotation = re-fetch. Doesn't solve "how do you authenticate to vaultwarden during bootstrap" — but that's a one-time, human-present operation, not a per-deploy one. |
### Recommended starting point
### Chosen shape
- **age key per host**, generated at first-time-setup, stored at `/etc/charlie/age.key` (root, `0600`).
- Public key is non-secret; published in [`docs/hosts.md`](hosts.md) and added to the `.sops.yaml` of each stack repo the host owns.
- A second "admin" age key (held by alex, on the admin laptop) is added to *every* repo's `.sops.yaml` so secrets stay decryptable / re-encryptable from off-host.
- **Bootstrap mechanism for the host's age key**: TBD — see [Open question #1](#1-host-age-key-bootstrap). Strong default: generate during first-time-setup, paste pub key into `.sops.yaml`s, commit.
- **One `age` key per host**, generated at first-time-setup, stored at `/etc/charlie/age.key` (root-owned, `0600`).
- Public key is non-secret; published in [`docs/hosts.md`](hosts.md#age-public-keys-sops) and added to the `.sops.yaml` of each stack repo the host owns.
- One **admin ("skeleton") `age` key** held by alex, on the admin laptop, listed in *every* repo's `.sops.yaml` so secrets stay decryptable / re-encryptable from off-host.
- **Per-host keys, not per-stack.** Reasoning: if root on the host is compromised, the per-stack keys would all be readable anyway; if a *container* is compromised, it can't reach `/etc/charlie/age.key` because that file isn't bind-mounted into containers. Container isolation already provides the per-stack property.
### Pilot: `Charlie/ddclient`
[`ddclient`](https://gitea.alexandru.macocian.me/Charlie/ddclient) is the Tier 1 pilot — single host (morgott), tiny `.env` (one Cloudflare token-ish thing), no downstream dependencies. The recipe:
```bash
# On the admin laptop. Assumes age + sops installed (see onboarding.md).
cd ~/Dev/ddclient
# 1. Drop the .sops.yaml. Recipients = morgott's pubkey + admin's pubkey.
# Pubkeys come from docs/hosts.md.
cat > .sops.yaml <<'EOF'
creation_rules:
- path_regex: secrets/.*\.sops\.(ya?ml|env|json)$
age: >-
<morgott pubkey from hosts.md>,
<admin pubkey from hosts.md>
EOF
# 2. Take the existing .env, encrypt it.
mkdir -p secrets
sops --encrypt --input-type dotenv --output-type yaml \
/mnt/nas/stacks/ddclient/.env > secrets/env.sops.yaml
# Verify (decrypt back, diff):
sops --decrypt secrets/env.sops.yaml | diff - /mnt/nas/stacks/ddclient/.env
# 3. Wire deploy.sh to materialize plaintext at deploy time.
cat > deploy.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")"
trap 'rm -f .env' EXIT
sops --decrypt secrets/env.sops.yaml > .env
chmod 600 .env
sudo docker compose --env-file .env up -d
EOF
chmod +x deploy.sh
# 4. Commit + push.
git add .sops.yaml secrets/env.sops.yaml deploy.sh
git commit -m "Adopt SOPS for secrets"
git push
# 5. On morgott: pull, run the new deploy.sh, verify ddclient still works.
ssh morgott.lan 'cd /mnt/nas/stacks/ddclient && git pull && ./deploy.sh'
# Last resort if it breaks: original .env is still on morgott; falls back to
# `sudo docker compose up -d` as before.
# 6. Once verified, remove the plaintext .env from morgott.
ssh morgott.lan 'rm /mnt/nas/stacks/ddclient/.env'
```
Success = ddclient is up after `./deploy.sh`, no plaintext `.env` on morgott. Then we roll the same shape across the rest of the migrated stacks, one at a time.
## Open questions
### 1. Host age key bootstrap
### 1. Host age key bootstrap (decided: option a)
How does the key get onto a freshly-installed host?
- **Generate in place during first-time-setup**, capture the pub key, commit it into the relevant `.sops.yaml`s. Private key never leaves the host. Simple, recommended.
- **Mint centrally** (admin laptop), copy to the host over SSH during bootstrap. Easier to pre-stage `.sops.yaml`s before the host exists, but the key crosses a wire once.
- **Store in vaultwarden**, fetch at bootstrap. Lets you re-image a host without rotating its key. Buys: re-image without `sops updatekeys` churn. Costs: vaultwarden has to be reachable (and authenticatable-to) during host bootstrap.
- **Generate in place during first-time-setup.** Capture the pub key, commit it into the relevant `.sops.yaml`s. Private key never leaves the host. Simple. **Chosen.**
- Mint centrally (admin laptop), copy to the host over SSH during bootstrap. Easier to pre-stage `.sops.yaml`s before the host exists, but the key crosses a wire once.
- Store in vaultwarden, fetch at bootstrap. Lets you re-image a host without rotating its key. Buys: re-image without `sops updatekeys` churn. Costs: vaultwarden has to be reachable (and authenticatable-to) during host bootstrap.
Recommend option 1 unless re-imaging frequency makes the rotation cost felt.
Rationale: re-imaging is rare; `sops updatekeys` for the (small) set of stacks owned by a re-imaged host is cheap. In-place generation has the smallest moving parts.
### 2. Admin key custody
### 2. Admin key custody (decided: option b)
The "decrypt anywhere" admin key is the master skeleton key for the fleet. Where does it live?
- Admin laptop only, file at `~/.config/sops/age/keys.txt` (`0600`). Loss = re-key every repo.
- Admin laptop **and** vaultwarden as backup. Loss recoverable.
- **✅ Admin laptop and vaultwarden as backup.** Loss recoverable. **Chosen.**
- Hardware-backed (YubiKey + age-plugin-yubikey). Strongest, adds a hardware dependency for routine ops.
Vaultwarden item shape (Secure Note):
```
Name: Charlie SOPS admin age key
Notes: <full contents of ~/.config/sops/age/keys.txt>
(includes the # public key: ... comment line and the AGE-SECRET-KEY-1... line)
```
Generate the admin key once (recipe in [`docs/onboarding.md → Admin age key (one-time)`](onboarding.md#admin-age-key-one-time)).
### 3. Per-environment / per-host secrets in the same repo
Shared stacks ([`otelcollector`](https://gitea.alexandru.macocian.me/Charlie/otelcollector), [`portainer-agent`](https://gitea.alexandru.macocian.me/Charlie/portainer-agent)) need different secrets per host. Two shapes:
@@ -130,8 +194,8 @@ Recommend the per-file shape — it's the only one consistent with the constrain
### 5. Vaultwarden's role going forward
- Pure human use case (browser autofill, secure notes). No change.
- Optional: store the age private key as a backup item per host. Useful for re-imaging.
- Optional: store the admin age key (if not on YubiKey).
- Holds the **admin age key** as a Secure Note (see Open Q #2).
- Optional: store each host's age private key as a Secure Note for re-image convenience. Trade-off: convenience now vs. one more place to leak. Not done by default.
Vaultwarden becomes a backup / human convenience, not a runtime dependency.
@@ -152,8 +216,10 @@ Both currently in vaultwarden as the source of truth; `.env` on morgott is the w
## Next steps
1. Settle [open questions 1 + 2](#open-questions) here, in-place.
2. Generate age keys for morgott + melina + mohg + miquella + malenia. Capture pub keys in [`docs/hosts.md`](hosts.md).
3. Spike on [`Charlie/ddclient`](https://gitea.alexandru.macocian.me/Charlie/ddclient) (Tier 1, single-host, tiny `.env`): add `.sops.yaml`, commit `secrets/env.sops.yaml`, wire into `deploy.sh`.
4. Roll across migrated stacks one tier at a time.
5. Fold the result into the [`Charlie/runners` design](roadmap.md#phase-2--ci) before any pipeline depends on it.
1. Settle [open questions 1 + 2](#open-questions). Done.
2. Generate the **admin age key** on the laptop (one-time). Store in vaultwarden. Recipe: [`onboarding.md → Admin age key (one-time)`](onboarding.md#admin-age-key-one-time).
3. Generate **morgott's host age key**. Capture pubkey in [`docs/hosts.md`](hosts.md#age-public-keys-sops). Recipe: [`onboarding.md → Host age key (SOPS)`](onboarding.md#host-age-key-sops).
4. Spike on [`Charlie/ddclient`](https://gitea.alexandru.macocian.me/Charlie/ddclient) per [Pilot recipe](#pilot-charlieddclient).
5. Roll across migrated stacks one tier at a time.
6. Generate age keys for melina + mohg + miquella + malenia as their stacks need SOPS.
7. Fold the result into the [`Charlie/runners` design](roadmap.md#phase-2--ci) before any pipeline depends on it.