project-charlie/docs/cert-issuer.md

Cert issuer

Status: deployed. Charlie/cert-issuer runs on morgott (smallstep/step-ca:0.30.2); Charlie/gssh mints user certs against it via a JWK provisioner. The CA key is no longer mounted into Gssh.

This doc captures the design space and the open questions. Operational details (provisioners table, add/restart procedure, init recipe, gotchas) live in Charlie/cert-issuer's README. Onboarding-side work (sync-ca, host trust) lives in onboarding.md → Trusted CA.

Context

The SSH user-cert signing CA used to live as a flat file at /mnt/nas/ca/ssh/ssh_user_ca and ssh_user_ca.pub, mounted into Charlie/gssh (and readable by anyone with shell on morgott). Gssh shelled out to ssh-keygen -s against the mounted key after each OIDC login against Authentik.

Two problems that drove the extraction:

1. The CA key was replicated to every host that ran Gssh. cat ca.key exfiltrated the master key for the whole identity story.
2. The cert-minting logic was locked inside Gssh. Anything else that wanted short-lived SSH access — host bootstrap, runners, future services — had to either re-implement the same ssh-keygen dance or go through Gssh's user-facing flow (wrong shape).

Both resolved by extracting the issuer.

What we shipped

smallstep step-ca running in Charlie/cert-issuer. Single Go binary, container-friendly. Gives us:

• JWK provisioners (machine callers — today: gssh, admin).
• OIDC provisioner support (Authentik-compatible) for direct human auth — not wired yet, see Open question #3.
• Configurable validity windows + principals templates.
• Backends for the CA key: file (current), PKCS#11, YubiHSM, AWS/GCP/Azure KMS, tpmkms (TPM 2.0).
• Audit logging.
• SSH host certs as well as user certs (planned, see Open question #3 and Host onboarding step 3).

Architecture

        ┌──────────┐     OAuth/OIDC      ┌────────────┐
 user ─►│  Gssh    │────────────────────►│ Authentik  │
        └────┬─────┘                     └────────────┘
             │ step ssh certificate  (auth: JWK provisioner password)
             ▼
        ┌──────────────────┐    ┌────────┐
        │ step-ca          │───►│ CA key │  (today: file; planned: TPM-sealed)
        │ (Charlie/        │    └────────┘
        │  cert-issuer)    │
        └───────┬─────────┘
                ▲
                │ step ca root, /ssh/roots, step ca certificate
                │
        ┌───────┴─────────┐
        │ cert-syncer      │ hourly loop. Pure producer; never restarts anyone.
        │ (Charlie/        │ Writes:
        │  cert-syncer)    │   /mnt/nas/ca/ca.crt
        └───────┬─────────┘   /mnt/nas/ca/ssh/ssh_user_ca.pub
                │           /mnt/nas/ca/<host>/<host>.{crt,key}
                ▼
        /mnt/nas/ca/  ─── read by sync-ca.sh on every host ─── /etc/{sssd,ssh}/...
              │
              │ read by stack consumers + sidecar
              ▼
        ┌───────────────────────┐
        │ consumer stack       │ e.g. Charlie/ldap = openldap + ldap-ui + cert-watch
        │   openldap          │ cert-watch polls the cert dir, restarts openldap on
        │   ldap-ui           │ change. Privilege (docker.sock) is per-stack, not
        │   cert-watch (side) │ in cert-syncer.
        └───────────────────────┘

Three separable concerns, three repos:

• cert-issuer — passive responder. Mints certs on demand. Holds the CA private key.
• cert-syncer — active puller. Refreshes trust files and renews managed server certs. Has rw on /mnt/nas/ca/. Doesn't know who consumes anything.
• cert-watch — sidecar image. Lives in each consumer stack. Watches the cert files for that stack and restarts the consumer on change. Has docker.sock for its own stack only. Travels with the stack across hosts.

Hosts read from /mnt/nas/ca/ exactly as before — they don't know step-ca exists.

Other consumers (planned)

• Host bootstrap. First-time-setup gets a host cert from the issuer instead of being trusted blindly via TOFU.
• Runners. Short-lived deploy cert per job, expires after the job. No long-lived host keys floating around with shell access.
• Operator scripts. charlie-ssh <host> wraps "fetch a 5-minute user cert, ssh in" — useful when Gssh's web flow is overkill.

Host onboarding

Full onboarding recipe (Ubuntu + Mac, all sections — hostname, NAS, docker, portainer agent, otelcollector, DNS, CA, LDAP/SSSD, SSH defaults) lives in onboarding.md. Below is just the cert-issuer-relevant slice and what it replaces.

A new host joining the fleet has to be told: "trust this CA, and present an identity signed by it." None of this uses step ca bootstrap directly — that's a per-user trust pin for the step CLI, not host-wide trust. The pieces below are what onboarding.md → Trusted CA will look like once cert-issuer takes over from /mnt/nas/ca/.

What every host does, as root

1. Pin the X.509 root into the system trust store (so anything on the host — not just step — can validate certs the CA issues, including the issuer's own serving cert):

   curl --resolve cert-issuer.lan:9000:<issuer-ip> \
        -o /usr/local/share/ca-certificates/charlie-root.crt \
        https://cert-issuer.lan:9000/roots.pem
   # Verify the fingerprint against the one captured at init.
   step certificate fingerprint /usr/local/share/ca-certificates/charlie-root.crt
   sudo update-ca-certificates
   

2. Trust the SSH user CA (so sshd accepts user certs minted by step-ca):

   # Pubkey is non-secret. Captured from the issuer init log,
   # also fetchable via `step ssh config --roots`.
   sudo tee /etc/ssh/charlie_user_ca.pub <<'EOF'
   ecdsa-sha2-nistp256 AAAAE2Vj... <user CA pubkey>
   EOF
   sudo chmod 644 /etc/ssh/charlie_user_ca.pub
   
   # In /etc/ssh/sshd_config:
   #   TrustedUserCAKeys /etc/ssh/charlie_user_ca.pub
   sudo systemctl restart ssh
   

3. (Later, once host certs are in scope.) Get a host cert so SSH clients can verify this host without TOFU. Out of scope for v1; sketched here so we don't forget:

   • Provisioner type for non-human callers: TBD (Open question #3). Likely a JWK provisioner whose token is fetched once during install.
   • step ssh certificate --host <hostname>.lan /etc/ssh/ssh_host_ed25519_key.pub writes …-cert.pub.
   • Add HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub to sshd_config.
   • Distribute the SSH host CA pubkey to clients as @cert-authority *.lan ... in ~/.ssh/known_hosts (or system-wide).

What the admin user does, optionally

4. Per-user step bootstrap (so alex on this host can run step commands without re-typing --ca-url and --fingerprint):

   step ca bootstrap \
     --ca-url https://cert-issuer.lan:9000 \
     --fingerprint <root-fingerprint>
   step ca health
   

   This writes ~/.step/ for that user only. Skippable on hosts where no human ever runs step.

What this replaces

Today onboarding.md → Trusted CA runs sync-ca.sh from a systemd timer (or LaunchDaemon on Mac), which copies /mnt/nas/ca/ca.crt and /mnt/nas/ca/ssh/ssh_user_ca.pub into local trust stores. After migration, the same script (or its successor in Charlie/first-time-setup) does steps 1–2 above against step-ca's HTTPS endpoint instead, and the /mnt/nas/ca/ dependency goes away. Full delta: onboarding.md → Step-ca migration impact.

Open questions

1. Where does the CA key live

Today: lives in step-ca's data dir at /mnt/nas/data/cert-issuer/step/secrets/, encrypted at rest with DOCKER_STEPCA_INIT_PASSWORD (in .env, currently plaintext on the NAS).

Options:

• TPM-sealed on the issuer host. Strongest for an on-prem fleet. Requires the issuer to run on a host with TPM 2.0 (morgott, melina, mohg — yes; mac mini, DSM — no, but the issuer doesn't need to run there). step-ca supports tpmkms directly. CA key never exists in cleartext on disk.
• YubiHSM 2 / YubiKey. Even stronger; physical device. Adds hardware to budget.
• PKCS#11 with a software token (SoftHSM). Same API as a real HSM, no hardware. Lets us prototype the path before buying a YubiHSM. Not actually more secure than file-on-disk; useful as a stepping stone.
• SOPS-sealed .env (next step, blocked on nothing). SOPS-encrypt the cert-issuer/.env (which contains DOCKER_STEPCA_INIT_PASSWORD) to morgott's host age key + admin's key. Now the unseal password isn't on the NAS in cleartext; the deploy workflow decrypts it on the runner at deploy time. Same pattern we're rolling out for every other stack via secrets.md + automation.md. The CA private key on disk is still encrypted-at-rest by step-ca itself with that password — moving the password into SOPS shrinks the attack surface from "anyone with NFS read on /mnt/nas/stacks/cert-issuer/.env" to "root on morgott."

Recommended sequence: SOPS-seal first (small change, rides on the secrets pilot landing). Move to TPM-sealed as a later hardening step.

2. Where the issuer runs

Landed on morgott for v1 (cohabitation with the old Gssh, no net increase in blast radius). Pinning the issuer to a specific host matters more once we adopt TPM-sealed CA keys (Open question #1) — at that point the choice of host is binding because the key is sealed to that TPM.

3. Caller authn + authz

Two distinct caller classes with different authn:

• Humans, via Gssh → Gssh authenticates the user via OIDC (Authentik), then asks the issuer for a cert with principals = the user's LDAP groups (or username). The issuer trusts Gssh as a service identity (mTLS or signed token); Gssh is responsible for having authenticated the human.
  • Alternative: have the issuer do the OIDC dance itself (step-ca OIDC provisioner). Gssh just redirects. Cleaner separation; Gssh stops handling tokens.
• Machines (runners, host bootstrap) → service identity (mTLS cert, JWT signed by an internal key, or step-ca's JWK/X5C provisioner). Should not go through the OIDC path.

Authz rules to define:

• Which callers can request which principals? (A runner shouldn't be able to mint a cert with principal=alex.)
• Maximum validity per caller class? (Humans: 8h; runners: minutes; bootstrap: hours.)
• Which callers can request host certs vs user certs?

Sketch as a config table the issuer enforces — won't try to fit it in the API.

4. Audit + revocation

• Audit: issuer logs every cert (principals, fingerprint, requester, validity). Cheap, do it from day one.
• Revocation: SSH certs don't have OCSP/CRL the way X.509 does. Two mechanisms:
  • Short validity windows (the current Gssh approach). 5–60 minutes for users, minutes for runners. Compromise window = validity window.
  • RevokedKeys file distributed to every host's sshd_config. Hard revoke, but requires host-side push.

Recommend short validity + audit log for v1; add RevokedKeys distribution if and when an incident demands it.

5. Migration path

Done. Sequence that worked:

1. Stand up Charlie/cert-issuer on morgott. New SSH CA generated by step-ca (didn't reuse the old /mnt/nas/ca/ssh/ssh_user_ca key — it had been live on disk on every gssh host).
2. Replace /mnt/nas/ca/ssh/ssh_user_ca.pub with step-ca's SSH user CA pubkey. The existing sync-ca.timer on every host picked it up on next tick.
3. Add a gssh JWK provisioner to step-ca with 5-min user cert validity.
4. Rewrite Charlie/gssh's Services/Issuer/* as a step-ca client. Container ships step CLI; CertificateIssuerInitializationService runs step ca bootstrap; CertificateIssuerService.CreateEphemeralSignedCredentials shells out to step ssh certificate. Public surface unchanged.
5. Build new image + push, update Gssh's .env with step-ca config, drop the CA bind mount from its compose, redeploy.

Safety net during cutover: WireGuard tunnel from operator laptop → home network with direct SSH access. Took Gssh out of the critical path so a broken Gssh image didn't lock anyone out. Decommissioned post-cutover; server-side WG kept around for next time.

Not done in this migration:

• The old /mnt/nas/ca/ssh/ssh_user_ca (private key) is still on the NAS but unread. Archive + delete after a stability window.

• Rotate the X.509 root from /mnt/nas/ca/ca.crt. Step-ca minted a fresh one; the old root is still in trust stores via sync-ca.sh. Decommissioning is a separate later step (see onboarding.md → Step-ca migration impact).

• Replace Authentik. The issuer trusts Authentik (directly via OIDC provisioner, or transitively via Gssh) for human identity.

• Replace SSSD/LDAP for the authorization side on the host. The cert says "this user is alex"; the host's SSSD-backed PAM still decides what alex can do (sudo, group membership, etc.).

• Issue X.509 certs (Caddy edge certs, internal mTLS, etc.). That's a separate hat — step-ca can wear it, but mixing the two on day one bloats the decision space.

Next steps

Done:

• ✅ Bring up step-ca on morgott. PKI initialised, root fingerprint in vaultwarden.
• ✅ Add gssh (5-min user certs), admin (16h default / 24h max user certs), and cert-syncer (1y default / 2y max X.509 server certs) JWK provisioners.
• ✅ Rewrite Gssh as a step-ca client. CA key no longer mounted into Gssh.
• ✅ Replace /mnt/nas/ca/ssh/ssh_user_ca.pub with step-ca's SSH user CA pubkey; rolled to morgott via sync-ca.timer.
• ✅ Stand up Charlie/cert-syncer on morgott. Manages X.509 root and SSH user CA in /mnt/nas/ca/, plus renews managed server certs.
• ✅ Stand up Charlie/cert-watch sidecar pattern. Restart-on-cert-change is the consumer stack's responsibility, not cert-syncer's.
• ✅ Cut openldap over to a step-ca-issued ldap.lan cert. SSSD on morgott + melina kicked to pick up the new X.509 root.

Pending (none blocking):

1. Roll the new X.509 root to remaining hosts (mohg, miquella, malenia) — they auto-pick up at the next hourly sync-ca.timer tick; cached SSSD entries mask any latency until then.
2. Add Authentik as an OIDC provisioner on step-ca. Lets Gssh stop carrying the JWK password — step-ca authenticates the human directly via the OIDC token Gssh forwards.
3. Issue SSH host certs. Replace TOFU with @cert-authority *.lan ... trust client-side and HostCertificate server-side. cert-syncer is the natural place to manage these (would extend MANAGED_CERTS with a host-cert variant).
4. Settle Open question #1: SOPS-sealed first (rides on secrets.md), then TPM-sealed once stable.
5. After runners are alive (Phase 2): add runners as a third caller class for short-lived per-job certs.