project-charlie/docs/secrets.md

Secrets sourcing

Status: decided. SOPS-in-repo with one age key per host. The reusable deploy-stack.yml workflow SSHes into the target host as sys-gitea-runner and runs sops --decrypt there using /etc/charlie/age.key. See automation.md. Pilot complete (Charlie/ddclient, Charlie/homarr). This doc is the spec; docs/onboarding.md is the operational recipe; docs/automation.md is the caller-facing reference for the workflow that consumes the encrypted files.

Goal

Every host can, on demand:

1. Obtain only the secrets it's allowed to decrypt (and nothing else).
2. Materialize them where compose / runner jobs can consume them (typically .env files).
3. Do all of the above non-interactively, suitable for being driven by a Gitea Actions runner job.

Out of scope: per-user secrets, browser-side autofill, secret editing UX. Vaultwarden continues to serve those for humans.

Constraints

• Source of truth lives with the stack. A secret for mediacompose belongs in Charlie/mediacompose, not in some separate vault we have to keep in sync. The blast radius of a leak is then naturally bounded by repo access.
• Per-machine decryption. A compromised host should leak only the secrets it's already cleared to read. Concretely: only the secrets in repos whose .sops.yaml lists that host's key.
• Self-identifying host. Same convention used by the reusable deploy workflow: hostname -s | tr '[:upper:]' '[:lower:]'.
• No :latest problem for secrets. Versioning lives in git like everything else; rotation is a commit + sops updatekeys.

Direction: SOPS in-repo

Encrypted files live next to docker-compose.yml in each Charlie/<stack> repo. They're decrypted on the target host at deploy time. The reusable deploy-stack.yml workflow owns the lifecycle: SSH into target as sys-gitea-runner → git fetch && reset --hard → inline sops --decrypt (target's /etc/charlie/age.key) → docker compose up -d → cleanup of plaintext via if: always().

Why this over a central vault:

• No "who controls the secrets" question. Whoever can write to Charlie/mediacompose controls mediacompose's secrets. We already have to answer that question for code; we don't need to answer it twice.
• No central control plane to keep available. Vault outage doesn't block deploys. A vault that has to be online for every deploy is itself a stack with secrets, and the bootstrap problem becomes recursive.
• Per-repo access control falls out for free. Per-stack access is .sops.yaml's list of recipients. Per-host access is the same list, viewed from the host side.
• Reads as code in PRs. Diffs show that a secret changed (and which one), not the value. No second review surface.

What we give up:

• A "where is this password" lookup for humans. Solved by: humans keep using vaultwarden for that. The two systems don't need to share storage; they only need to not contradict each other (rotation hygiene).
• Online rotation. Rotating means commit + push + redeploy, not "click a button in the UI."

Layout in a stack repo

Charlie/<stack>/
├── docker-compose.yml
├── .env.example                # tracked, redacted, as today
├── secrets/
│   └── env.sops.yaml           # encrypted; decrypts to .env
├── .sops.yaml                  # creation rules: which recipients can decrypt this dir
└── .gitea/workflows/deploy.yaml  # calls Charlie/project-charlie/.gitea/workflows/deploy-stack.yml@main

.sops.yaml (sketch):

creation_rules:
  - path_regex: secrets/.*\.sops\.(ya?ml|env|json)$
    age: >-
      age1morgott...,
      age1admin-alex...

For shared stacks (otelcollector, portainer-agent), the rule lists every host that runs the stack — same multi-host pattern as the compose layout already does.

Inside the deploy workflow

The stack repo's .gitea/workflows/deploy.yaml is a thin caller of the reusable workflow:

name: Deploy
on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  deploy:
    uses: Charlie/project-charlie/.gitea/workflows/deploy-stack.yml@main
    with:
      host: morgott
      decrypt: |
        secrets/env.sops.yaml -> .env

The reusable workflow decrypts to .env (mode 0600), runs docker compose pull && up -d, and removes the plaintext in an if: always() cleanup step. Manual fallback recipe (when the runner is unavailable): automation.md → Manual fallback.

The actual open question: decryption

SOPS supports several recipient backends. Picking which one(s) we use is the load-bearing decision in this doc.

Backend	Fit	Notes
age key file	Chosen.	One private key per host, file at /etc/charlie/age.key (root, 0600). Public key listed in each repo's .sops.yaml that the host needs to decrypt. Bootstrap = generate key once, paste pub into the right .sops.yaml(s), commit. Heterogeneous hosts (DSM, mac mini) all just need a file.
age via SSH host key	Rejected.	age can use an existing SSH ed25519 key as a recipient. Reuses something every host already has. Caveat: SOPS support for SSH-as-age recipients goes through the ssh-to-age conversion — workable, slightly more moving parts than a dedicated age key. Per-host key file is simpler and the convenience of "reusing" SSH keys is mostly cosmetic.
GPG	No.	Heavier, key-management UX is worse, no advantage over age.
HashiCorp Vault Transit	No.	Reintroduces the central control plane we're trying to avoid.
Cloud KMS (AWS / GCP / Azure)	No.	Off-prem dependency for an on-prem homelab.
TPM-sealed age key	Maybe later.	Strongest. Most x86 hosts have TPM 2.0; mac mini and DSM do not. Heterogeneity makes it a bad default; reasonable to add per-host as a hardening step after v1.
Vaultwarden-stored age key, fetched at bootstrap	Hybrid; see Open Q #1.	Age key lives encrypted in vaultwarden; fetched once during host bootstrap (same flow as CA trust / SSSD setup) and written to /etc/charlie/age.key. Rotation = re-fetch. Doesn't solve "how do you authenticate to vaultwarden during bootstrap" — but that's a one-time, human-present operation, not a per-deploy one.

Chosen shape

• One age key per host, generated at first-time-setup, stored at /etc/charlie/age.key (root-owned, 0600).
• Public key is non-secret; published in docs/hosts.md and added to the .sops.yaml of each stack repo the host owns.
• One admin ("skeleton") age key held by alex, on the admin laptop, listed in every repo's .sops.yaml so secrets stay decryptable / re-encryptable from off-host.
• Per-host keys, not per-stack. Reasoning: if root on the host is compromised, the per-stack keys would all be readable anyway; if a container is compromised, it can't reach /etc/charlie/age.key because that file isn't bind-mounted into containers. Container isolation already provides the per-stack property.

Pilot: Charlie/ddclient

ddclient is the Tier 1 pilot — single host (morgott), tiny .env (one Cloudflare token-ish thing), no downstream dependencies. Prereq: morgott's runner is up (see runners.md). Recipe:

# On the admin laptop. Assumes age + sops installed (see onboarding.md).

cd ~/Dev/ddclient

# 1. Drop the .sops.yaml. Recipients = morgott's pubkey + admin's pubkey.
#    Pubkeys come from docs/hosts.md.
cat > .sops.yaml <<'EOF'
creation_rules:
  - path_regex: secrets/.*\.sops\.(ya?ml|env|json)$
    age: >-
      <morgott pubkey from hosts.md>,
      <admin pubkey from hosts.md>
EOF

# 2. Take the existing .env, encrypt it.
mkdir -p secrets
sops --encrypt --input-type dotenv --output-type yaml \
     /mnt/nas/stacks/ddclient/.env > secrets/env.sops.yaml
# Verify (decrypt back, diff):
sops --decrypt secrets/env.sops.yaml | diff - /mnt/nas/stacks/ddclient/.env

# 3. Wire the deploy workflow to call the shared reusable workflow.
mkdir -p .gitea/workflows
cat > .gitea/workflows/deploy.yaml <<'EOF'
name: Deploy
on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  deploy:
    uses: Charlie/project-charlie/.gitea/workflows/deploy-stack.yml@main
    with:
      host: morgott
      decrypt: |
        secrets/env.sops.yaml -> .env
EOF

# 4. Remove the per-stack deploy.sh — the reusable workflow replaces it.
git rm -f deploy.sh 2>/dev/null || true

# 5. Commit + push. Pushing to main triggers the workflow on morgott's runner.
git add .sops.yaml secrets/env.sops.yaml .gitea/workflows/deploy.yaml
git commit -m "Adopt SOPS + reusable deploy workflow"
git push

# 6. Watch the deploy in Gitea Actions. Verify ddclient is up.
#    Last resort if it breaks: the original .env is still on morgott;
#    fall back to the manual recipe in docs/automation.md.

# 7. Once verified, remove the leftover plaintext .env from morgott.
ssh morgott.lan 'rm /mnt/nas/stacks/ddclient/.env'

Success = ddclient is up after a git push, no plaintext .env on morgott. Then we roll the same shape across the rest of the migrated stacks, one at a time.

Open questions

1. Host age key bootstrap (decided: option a)

How does the key get onto a freshly-installed host?

• ✅ Generate in place during first-time-setup. Capture the pub key, commit it into the relevant .sops.yamls. Private key never leaves the host. Simple. Chosen.
• Mint centrally (admin laptop), copy to the host over SSH during bootstrap. Easier to pre-stage .sops.yamls before the host exists, but the key crosses a wire once.
• Store in vaultwarden, fetch at bootstrap. Lets you re-image a host without rotating its key. Buys: re-image without sops updatekeys churn. Costs: vaultwarden has to be reachable (and authenticatable-to) during host bootstrap.

Rationale: re-imaging is rare; sops updatekeys for the (small) set of stacks owned by a re-imaged host is cheap. In-place generation has the smallest moving parts.

2. Admin key custody (decided: option b)

The "decrypt anywhere" admin key is the master skeleton key for the fleet. Where does it live?

• Admin laptop only, file at ~/.config/sops/age/keys.txt (0600). Loss = re-key every repo.
• ✅ Admin laptop and vaultwarden as backup. Loss recoverable. Chosen.
• Hardware-backed (YubiKey + age-plugin-yubikey). Strongest, adds a hardware dependency for routine ops.

Vaultwarden item shape (Secure Note):

Name:  Charlie SOPS admin age key
Notes: <full contents of ~/.config/sops/age/keys.txt>
       (includes the # public key: ... comment line and the AGE-SECRET-KEY-1... line)

Generate the admin key once (recipe in docs/onboarding.md → Admin age key (one-time)).

3. Per-environment / per-host secrets in the same repo

Shared stacks (otelcollector, portainer-agent) need different secrets per host. Two shapes:

• One file per host: secrets/morgott.env.sops.yaml, secrets/melina.env.sops.yaml, …. The deploy workflow's decrypt: input picks by ${{ matrix.host }} (mirrors the existing compose-per-host pattern).
• One file with all of them, every host's key on the recipient list. Cleaner, but means morgott's key can decrypt melina's secrets in that one file. Loses the per-host blast-radius property.

Recommend the per-file shape — it's the only one consistent with the constraint above.

4. Materialization target

• Plaintext .env (or other destination per the workflow's decrypt: mapping) in the working tree, removed by the workflow's if: always() cleanup step. Current default in deploy-stack.yml.
• /run/charlie-secrets/<stack>/.env (tmpfs). Survives the deploy job but not reboots; nicer if multiple jobs need to read it. Reconsider if a use case shows up.
• Process substitution / fd, no file ever on disk. Cleanest, but breaks docker compose config debugging and not all compose versions handle it.

5. Vaultwarden's role going forward

• Pure human use case (browser autofill, secure notes). No change.
• Holds the admin age key as a Secure Note (see Open Q #2).
• Optional: store each host's age private key as a Secure Note for re-image convenience. Trade-off: convenience now vs. one more place to leak. Not done by default.

Vaultwarden becomes a backup / human convenience, not a runtime dependency.

Things this design intentionally does not do

• Manage secrets for stacks that aren't runner-deployed. Those keep their hand-edited .env until they migrate.
• Replace vaultwarden's human-facing UX.
• Solve secrets for Charlie/gitea itself — gitea-on-melina is deployed manually (see conventions) and bootstraps the rest of the system; it can't depend on a runner that depends on it. But: SOPS is offline crypto, so gitea's secrets can live in Charlie/gitea encrypted to melina's age key, and a manual sops -d | docker compose up -d works without any other Charlie service being up.

Stacks already on plain .env waiting for SOPS

These migrated before secrets work landed and currently keep credentials in hand-edited .env files on morgott. They'll be the first to fold into the SOPS pattern once Charlie/runners is up:

• cert-issuer — DOCKER_STEPCA_INIT_PASSWORD (vault unseal). Losing it = re-init the CA.
• gssh — Issuer__ProvisionerPassword (the JWK provisioner password Gssh uses to authenticate to step-ca). Losing it = remove + re-add the gssh provisioner.

Both currently in vaultwarden as the source of truth; .env on morgott is the working copy. The SOPS migration replaces the working copy, not the source of truth.

Next steps

1. ✅ Settle open questions 1 + 2. Done.
2. ✅ Generate the admin age key on the laptop. Stored in vaultwarden. Recipe: onboarding.md → Admin age key (one-time).
3. ✅ Generate morgott + melina host age keys. Pubkeys in docs/hosts.md. Recipe: onboarding.md → Host age key (SOPS).
4. ✅ Reusable deploy-stack.yml workflow + charlie-ssh-* composite actions shipped in Charlie/project-charlie. SSH-into-host model with step-ca-issued certs; SOPS decrypt runs on the target host. See automation.md.
5. Stand up Charlie/runners on morgott + melina. Blocks everything below.
6. Spike on Charlie/ddclient per Pilot recipe.
7. Roll across migrated stacks one tier at a time.
8. Generate age keys for mohg + miquella + malenia as their stacks need SOPS.