Charlie/project-charlie
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update docs with certificate issuance and One trust

Browse Source
This commit is contained in:
Alexandru Macocian
2026-05-04 18:14:42 +02:00
parent 9a8c3c79a5
commit 3d742a5024
9 changed files with 1124 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+3
View File
@@ -14,6 +14,9 @@ Centralized source of truth for the homelab. Per-app stacks live as repos under
- [Repo plan (`Charlie/*`)](docs/repos.md)
- [Migration recipe](docs/migration.md)
- [Reorg runbook](docs/reorg.md)
- [Host onboarding](docs/onboarding.md)
- [Secrets sourcing](docs/secrets.md)
- [Cert issuer](docs/cert-issuer.md)
- [Roadmap](docs/roadmap.md)
## Assumptions
docs/cert-issuer.md
+219
View File
@@ -0,0 +1,219 @@
# Cert issuer
Status: **deployed.** [`Charlie/cert-issuer`](https://gitea.alexandru.macocian.me/Charlie/cert-issuer) runs on morgott (`smallstep/step-ca:0.30.2`); [`Charlie/gssh`](https://gitea.alexandru.macocian.me/Charlie/gssh) mints user certs against it via a JWK provisioner. The CA key is no longer mounted into Gssh.
This doc captures the design space and the open questions. Operational details (provisioners table, add/restart procedure, init recipe, gotchas) live in [Charlie/cert-issuer's README](https://gitea.alexandru.macocian.me/Charlie/cert-issuer/src/branch/main/README.md). Onboarding-side work (sync-ca, host trust) lives in [onboarding.md → Trusted CA](onboarding.md#trusted-ca).
## Context
The SSH user-cert signing CA used to live as a flat file at `/mnt/nas/ca/ssh/ssh_user_ca` and `ssh_user_ca.pub`, mounted into [`Charlie/gssh`](https://gitea.alexandru.macocian.me/Charlie/gssh) (and readable by anyone with shell on morgott). Gssh shelled out to `ssh-keygen -s` against the mounted key after each OIDC login against [Authentik](identity.md).
Two problems that drove the extraction:
1. **The CA key was replicated to every host that ran Gssh.** `cat ca.key` exfiltrated the master key for the whole identity story.
2. **The cert-minting logic was locked inside Gssh.** Anything else that wanted short-lived SSH access — host bootstrap, runners, future services — had to either re-implement the same `ssh-keygen` dance or go through Gssh's user-facing flow (wrong shape).
Both resolved by extracting the issuer.
## What we shipped
[smallstep `step-ca`](https://smallstep.com/docs/step-ca/) running in [`Charlie/cert-issuer`](https://gitea.alexandru.macocian.me/Charlie/cert-issuer). Single Go binary, container-friendly. Gives us:
- JWK provisioners (machine callers — today: `gssh`, `admin`).
- OIDC provisioner support ([Authentik](identity.md)-compatible) for direct human auth — not wired yet, see [Open question #3](#3-caller-authn--authz).
- Configurable validity windows + principals templates.
- Backends for the CA key: file (current), PKCS#11, YubiHSM, AWS/GCP/Azure KMS, **`tpmkms`** (TPM 2.0).
- Audit logging.
- SSH host certs as well as user certs (planned, see [Open question #3](#3-caller-authn--authz) and [Host onboarding](#host-onboarding) step 3).
### Architecture
```
┌──────────┐ OAuth/OIDC ┌────────────┐
user ─►│ Gssh │────────────────────►│ Authentik │
└────┬─────┘ └────────────┘
│ step ssh certificate (auth: JWK provisioner password)
▼
┌──────────────────┐ ┌────────┐
│ step-ca │───►│ CA key │ (today: file; planned: TPM-sealed)
│ (Charlie/ │ └────────┘
│ cert-issuer) │
└───────┬─────────┘
▲
│ step ca root, /ssh/roots, step ca certificate
│
┌───────┴─────────┐
│ cert-syncer │ hourly loop. Pure producer; never restarts anyone.
│ (Charlie/ │ Writes:
│ cert-syncer) │ /mnt/nas/ca/ca.crt
└───────┬─────────┘ /mnt/nas/ca/ssh/ssh_user_ca.pub
│ /mnt/nas/ca/<host>/<host>.{crt,key}
▼
/mnt/nas/ca/ ─── read by sync-ca.sh on every host ─── /etc/{sssd,ssh}/...
│
│ read by stack consumers + sidecar
▼
┌───────────────────────┐
│ consumer stack │ e.g. Charlie/ldap = openldap + ldap-ui + cert-watch
│ openldap │ cert-watch polls the cert dir, restarts openldap on
│ ldap-ui │ change. Privilege (docker.sock) is per-stack, not
│ cert-watch (side) │ in cert-syncer.
└───────────────────────┘
```
Three separable concerns, three repos:
- **cert-issuer** — passive responder. Mints certs on demand. Holds the CA private key.
- **cert-syncer** — active puller. Refreshes trust files and renews managed server certs. Has rw on `/mnt/nas/ca/`. Doesn't know who consumes anything.
- **cert-watch** — sidecar image. Lives in each consumer stack. Watches the cert files for that stack and restarts the consumer on change. Has docker.sock for *its own stack only*. Travels with the stack across hosts.
Hosts read from `/mnt/nas/ca/` exactly as before — they don't know step-ca exists.
### Other consumers (planned)
- **Host bootstrap.** First-time-setup gets a host cert from the issuer instead of being trusted blindly via TOFU.
- **Runners.** Short-lived deploy cert per job, expires after the job. No long-lived host keys floating around with shell access.
- **Operator scripts.** `charlie-ssh <host>` wraps "fetch a 5-minute user cert, ssh in" — useful when Gssh's web flow is overkill.
## Host onboarding
Full onboarding recipe (Ubuntu + Mac, all sections — hostname, NAS, docker, portainer agent, otelcollector, DNS, CA, LDAP/SSSD, SSH defaults) lives in [onboarding.md](onboarding.md). Below is just the cert-issuer-relevant slice and what it replaces.
A new host joining the fleet has to be told: "trust this CA, and present an identity signed by it." None of this uses `step ca bootstrap` directly — that's a per-user trust pin for the `step` CLI, not host-wide trust. The pieces below are what [`onboarding.md → Trusted CA`](onboarding.md#trusted-ca) will look like once cert-issuer takes over from `/mnt/nas/ca/`.
### What every host does, as root
1. **Pin the X.509 root into the system trust store** (so anything on the host — not just `step` — can validate certs the CA issues, including the issuer's own serving cert):
```bash
curl --resolve cert-issuer.lan:9000:<issuer-ip> \
-o /usr/local/share/ca-certificates/charlie-root.crt \
https://cert-issuer.lan:9000/roots.pem
# Verify the fingerprint against the one captured at init.
step certificate fingerprint /usr/local/share/ca-certificates/charlie-root.crt
sudo update-ca-certificates
```
2. **Trust the SSH user CA** (so `sshd` accepts user certs minted by step-ca):
```bash
# Pubkey is non-secret. Captured from the issuer init log,
# also fetchable via `step ssh config --roots`.
sudo tee /etc/ssh/charlie_user_ca.pub <<'EOF'
ecdsa-sha2-nistp256 AAAAE2Vj... <user CA pubkey>
EOF
sudo chmod 644 /etc/ssh/charlie_user_ca.pub
# In /etc/ssh/sshd_config:
# TrustedUserCAKeys /etc/ssh/charlie_user_ca.pub
sudo systemctl restart ssh
```
3. **(Later, once host certs are in scope.) Get a host cert** so SSH clients can verify *this* host without TOFU. Out of scope for v1; sketched here so we don't forget:
- Provisioner type for non-human callers: TBD ([Open question #3](#3-caller-authn--authz)). Likely a JWK provisioner whose token is fetched once during install.
- `step ssh certificate --host <hostname>.lan /etc/ssh/ssh_host_ed25519_key.pub` writes `…-cert.pub`.
- Add `HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub` to `sshd_config`.
- Distribute the SSH **host** CA pubkey to clients as `@cert-authority *.lan ...` in `~/.ssh/known_hosts` (or system-wide).
### What the admin user does, optionally
4. **Per-user `step` bootstrap** (so `alex` on this host can run `step` commands without re-typing `--ca-url` and `--fingerprint`):
```bash
step ca bootstrap \
--ca-url https://cert-issuer.lan:9000 \
--fingerprint <root-fingerprint>
step ca health
```
This writes `~/.step/` for that user only. Skippable on hosts where no human ever runs `step`.
### What this replaces
Today [`onboarding.md → Trusted CA`](onboarding.md#trusted-ca) runs `sync-ca.sh` from a systemd timer (or LaunchDaemon on Mac), which copies `/mnt/nas/ca/ca.crt` and `/mnt/nas/ca/ssh/ssh_user_ca.pub` into local trust stores. After migration, the same script (or its successor in [`Charlie/first-time-setup`](repos.md#pending--tier-5-long-tail)) does steps 1–2 above against step-ca's HTTPS endpoint instead, and the `/mnt/nas/ca/` dependency goes away. Full delta: [onboarding.md → Step-ca migration impact](onboarding.md#step-ca-migration-impact).
## Open questions
### 1. Where does the CA key live
Today: `/mnt/nas/ca/ca.key`, world-readable to anyone with NFS / shell on a host that mounts the share.
Options:
- **TPM-sealed on the issuer host.** Strongest for an on-prem fleet. Requires the issuer to run on a host with TPM 2.0 (morgott, melina, mohg — yes; mac mini, DSM — no, but the issuer doesn't need to run there). `step-ca` supports `tpmkms` directly. CA key never exists in cleartext on disk.
- **YubiHSM 2 / YubiKey.** Even stronger; physical device. Adds hardware to budget.
- **PKCS#11 with a software token** ([SoftHSM](https://www.opendnssec.org/softhsm/)). Same API as a real HSM, no hardware. Lets us prototype the path before buying a YubiHSM. Not actually more secure than file-on-disk; useful as a stepping stone.
- **Sealed file at rest, decrypted into memory at issuer start.** SOPS-encrypt `ca.key` to the issuer host's age key (same key store as [secrets.md](secrets.md)), `step-ca` reads it once, plaintext never touches disk. Simplest, ties cleanly to the secrets design.
Recommend: start with the **SOPS-sealed** option (rides on the [secrets.md](secrets.md) infrastructure we're building anyway). Move to **TPM-sealed** as a hardening step once the rest of the system is stable.
### 2. Where the issuer runs
Landed on **morgott** for v1 (cohabitation with the old Gssh, no net increase in blast radius). Pinning the issuer to a specific host matters more once we adopt TPM-sealed CA keys ([Open question #1](#1-where-does-the-ca-key-live)) — at that point the choice of host is binding because the key is sealed to that TPM.
### 3. Caller authn + authz
Two distinct caller classes with different authn:
- **Humans, via Gssh** → Gssh authenticates the user via OIDC (Authentik), then asks the issuer for a cert with principals = the user's LDAP groups (or username). The issuer trusts Gssh as a service identity (mTLS or signed token); Gssh is responsible for having authenticated the human.
- Alternative: have the issuer do the OIDC dance itself (`step-ca` OIDC provisioner). Gssh just redirects. Cleaner separation; Gssh stops handling tokens.
- **Machines (runners, host bootstrap)** → service identity (mTLS cert, JWT signed by an internal key, or `step-ca`'s JWK/X5C provisioner). Should *not* go through the OIDC path.
Authz rules to define:
- Which callers can request which **principals**? (A runner shouldn't be able to mint a cert with `principal=alex`.)
- Maximum **validity** per caller class? (Humans: 8h; runners: minutes; bootstrap: hours.)
- Which callers can request **host certs** vs **user certs**?
Sketch as a config table the issuer enforces — won't try to fit it in the API.
### 4. Audit + revocation
- **Audit:** issuer logs every cert (principals, fingerprint, requester, validity). Cheap, do it from day one.
- **Revocation:** SSH certs don't have OCSP/CRL the way X.509 does. Two mechanisms:
- Short validity windows (the current Gssh approach). 5–60 minutes for users, minutes for runners. Compromise window = validity window.
- `RevokedKeys` file distributed to every host's `sshd_config`. Hard revoke, but requires host-side push.
Recommend short validity + audit log for v1; add `RevokedKeys` distribution if and when an incident demands it.
### 5. Migration path
Done. Sequence that worked:
1. Stand up [`Charlie/cert-issuer`](https://gitea.alexandru.macocian.me/Charlie/cert-issuer) on morgott. New SSH CA generated by step-ca (didn't reuse the old `/mnt/nas/ca/ssh/ssh_user_ca` key — it had been live on disk on every gssh host).
2. Replace `/mnt/nas/ca/ssh/ssh_user_ca.pub` with step-ca's SSH user CA pubkey. The existing `sync-ca.timer` on every host picked it up on next tick.
3. Add a `gssh` JWK provisioner to step-ca with 5-min user cert validity.
4. Rewrite [`Charlie/gssh`](https://gitea.alexandru.macocian.me/Charlie/gssh)'s `Services/Issuer/*` as a step-ca client. Container ships `step` CLI; `CertificateIssuerInitializationService` runs `step ca bootstrap`; `CertificateIssuerService.CreateEphemeralSignedCredentials` shells out to `step ssh certificate`. Public surface unchanged.
5. Build new image + push, update Gssh's `.env` with step-ca config, drop the CA bind mount from its compose, redeploy.
**Safety net during cutover:** WireGuard tunnel from operator laptop → home network with direct SSH access. Took Gssh out of the critical path so a broken Gssh image didn't lock anyone out. Decommissioned post-cutover; server-side WG kept around for next time.
Not done in this migration:
- The old `/mnt/nas/ca/ssh/ssh_user_ca` (private key) is still on the NAS but unread. Archive + delete after a stability window.
- Rotate the X.509 root from `/mnt/nas/ca/ca.crt`. Step-ca minted a fresh one; the old root is still in trust stores via `sync-ca.sh`. Decommissioning is a separate later step (see [onboarding.md → Step-ca migration impact](onboarding.md#step-ca-migration-impact)).
- Replace Authentik. The issuer trusts Authentik (directly via OIDC provisioner, or transitively via Gssh) for human identity.
- Replace SSSD/LDAP for the *authorization* side on the host. The cert says "this user is `alex`"; the host's SSSD-backed PAM still decides what `alex` can do (sudo, group membership, etc.).
- Issue X.509 certs (Caddy edge certs, internal mTLS, etc.). That's a separate hat — `step-ca` *can* wear it, but mixing the two on day one bloats the decision space.
## Next steps
Done:
- ✅ Bring up `step-ca` on morgott. PKI initialised, root fingerprint in vaultwarden.
- ✅ Add `gssh` (5-min user certs), `admin` (16h default / 24h max user certs), and `cert-syncer` (1y default / 2y max X.509 server certs) JWK provisioners.
- ✅ Rewrite Gssh as a step-ca client. CA key no longer mounted into Gssh.
- ✅ Replace `/mnt/nas/ca/ssh/ssh_user_ca.pub` with step-ca's SSH user CA pubkey; rolled to morgott via `sync-ca.timer`.
- ✅ Stand up [`Charlie/cert-syncer`](https://gitea.alexandru.macocian.me/Charlie/cert-syncer) on morgott. Manages X.509 root and SSH user CA in `/mnt/nas/ca/`, plus renews managed server certs.
- ✅ Stand up [`Charlie/cert-watch`](https://gitea.alexandru.macocian.me/Charlie/cert-watch) sidecar pattern. Restart-on-cert-change is the consumer stack's responsibility, not cert-syncer's.
- ✅ Cut openldap over to a step-ca-issued `ldap.lan` cert. SSSD on morgott + melina kicked to pick up the new X.509 root.
Pending (none blocking):
1. **Roll the new X.509 root to remaining hosts** (mohg, miquella, malenia) — they auto-pick up at the next hourly `sync-ca.timer` tick; cached SSSD entries mask any latency until then.
2. **Add Authentik as an OIDC provisioner** on step-ca. Lets Gssh stop carrying the JWK password — step-ca authenticates the human directly via the OIDC token Gssh forwards.
3. **Issue SSH host certs.** Replace TOFU with `@cert-authority *.lan ...` trust client-side and `HostCertificate` server-side. cert-syncer is the natural place to manage these (would extend `MANAGED_CERTS` with a host-cert variant).
4. **Settle [Open question #1](#1-where-does-the-ca-key-live):** SOPS-sealed first (rides on [secrets.md](secrets.md)), then TPM-sealed once stable.
5. **After runners are alive ([Phase 2](roadmap.md#phase-2--ci)):** add runners as a third caller class for short-lived per-job certs.
docs/conventions.md
+1 -1
View File
@@ -7,4 +7,4 @@
- Workflows pin host: `runs-on: [self-hosted, morgott]` or `[self-hosted, melina]`. Never bare `self-hosted`.
- Image tags pinned in compose. No `:latest`.
- A runner never redeploys itself mid-job — cross-host: morgott deploys melina's runner, and vice versa.
- Secrets sourcing: TBD (candidate: vaultwarden).
- Secrets sourcing: SOPS-encrypted in each `Charlie/<stack>` repo, decrypted on the owner host at deploy time. Design + open decisions in [secrets.md](secrets.md).
docs/identity.md
+1 -1
View File
@@ -10,6 +10,6 @@
OAuth → Authentik → backend mints ephemeral SSH cert (signed by CA trusted on every host) → SSH proxied over WebSocket to JS frontend.
CA material lives under `/mnt/nas/ca/` and is synced to hosts by `sync-ca.sh`.
CA material lives under `/mnt/nas/ca/` and is synced to hosts by `sync-ca.sh`. Extracting cert issuance into a standalone service (so the CA key stops living on every host that needs to mint) is captured in [cert-issuer.md](cert-issuer.md).
Host logins / sudo are SSSD-backed against `open-ldap`; LDAP groups drive permissions.
docs/inventory.md
+37 -23
View File
@@ -11,44 +11,58 @@ State: May 3, 2026 — end of Tier 4.
| Charlie repo | Stack path | Image source | Notes |
|---|---|---|---|
| `caddy` | `caddy/` | local build | edge reverse proxy + caddy-security |
| `authentik` | `authentik/` | upstream | IdP; postgres + redis + server + worker |
| `ldap` | `ldap/` | upstream | openldap + ldap-ui |
| `ddclient` | `ddclient/` | upstream | dynamic DNS |
| `vaultwarden` | `vaultwarden/` | upstream | password vault |
| `bookstack` | `bookstack/` | upstream | wiki (+ mariadb) |
| `monica` | `monica/` | gitea-registry `monica:4.x` | CRM (custom 4.x fork; + mariadb) |
| `miniflux` | `miniflux/` | upstream | RSS (+ postgres) |
| `mealie` | `mealie/` | upstream | recipes (sqlite) |
| `homarr` | `homarr/` | upstream | dashboard |
| `paperless-ngx` | `paperless-ngx/` | upstream | docs (+ redis) |
| `victoriametrics` | `victoriametrics/` | upstream | vm + vmauth + vmalert + alertmanager + grafana |
| `gssh` | `gssh/` | gitea-registry `gssh:latest` | web SSH terminal |
| [`caddy`](https://gitea.alexandru.macocian.me/Charlie/caddy) | `caddy/` | local build | edge reverse proxy + caddy-security |
| [`authentik`](https://gitea.alexandru.macocian.me/Charlie/authentik) | `authentik/` | upstream | IdP; postgres + redis + server + worker |
| [`ldap`](https://gitea.alexandru.macocian.me/Charlie/ldap) | `ldap/` | upstream | openldap + ldap-ui |
| [`ddclient`](https://gitea.alexandru.macocian.me/Charlie/ddclient) | `ddclient/` | upstream | dynamic DNS |
| [`vaultwarden`](https://gitea.alexandru.macocian.me/Charlie/vaultwarden) | `vaultwarden/` | upstream | password vault |
| [`bookstack`](https://gitea.alexandru.macocian.me/Charlie/bookstack) | `bookstack/` | upstream | wiki (+ mariadb) |
| [`monica`](https://gitea.alexandru.macocian.me/Charlie/monica) | `monica/` | gitea-registry `monica:4.x` | CRM (custom 4.x fork; + mariadb) |
| [`miniflux`](https://gitea.alexandru.macocian.me/Charlie/miniflux) | `miniflux/` | upstream | RSS (+ postgres) |
| [`mealie`](https://gitea.alexandru.macocian.me/Charlie/mealie) | `mealie/` | upstream | recipes (sqlite) |
| [`homarr`](https://gitea.alexandru.macocian.me/Charlie/homarr) | `homarr/` | upstream | dashboard |
| [`paperless-ngx`](https://gitea.alexandru.macocian.me/Charlie/paperless-ngx) | `paperless-ngx/` | upstream | docs (+ redis) |
| [`victoriametrics`](https://gitea.alexandru.macocian.me/Charlie/victoriametrics) | `victoriametrics/` | upstream | vm + vmauth + vmalert + alertmanager + grafana |
| [`gssh`](https://gitea.alexandru.macocian.me/Charlie/gssh) | `gssh/` | gitea-registry `gssh:latest` | web SSH terminal |
### Melina (owner host)
| Charlie repo | Stack path | Image source | Notes |
|---|---|---|---|
| `gitea` | `gitea/` | upstream | the git server itself (deployed manually, never via runner-on-self) |
| `mediacompose` | `mediacompose/` | upstream | jellyfin + arr + qbt-via-gluetun |
| `searxng` | `searxng/` | upstream | metasearch (+ valkey) |
| `invidious` | `invidious/` | gitea-registry `invidious:oidc-support` | YouTube frontend (+ companion + db + gluetun) |
| `planka` | `planka/` | upstream | kanban (+ postgres) |
| `audiobookshelf` | `audiobookshelf/` | upstream | audiobooks |
| `directoryadmin` | `directoryadmin/` | gitea-registry `directoryadmin:latest` | LDAP admin UI |
| [`gitea`](https://gitea.alexandru.macocian.me/Charlie/gitea) | `gitea/` | upstream | the git server itself (deployed manually, never via runner-on-self) |
| [`mediacompose`](https://gitea.alexandru.macocian.me/Charlie/mediacompose) | `mediacompose/` | upstream | jellyfin + arr + qbt-via-gluetun |
| [`searxng`](https://gitea.alexandru.macocian.me/Charlie/searxng) | `searxng/` | upstream | metasearch (+ valkey) |
| [`invidious`](https://gitea.alexandru.macocian.me/Charlie/invidious) | `invidious/` | gitea-registry `invidious:oidc-support` | YouTube frontend (+ companion + db + gluetun) |
| [`planka`](https://gitea.alexandru.macocian.me/Charlie/planka) | `planka/` | upstream | kanban (+ postgres) |
| [`audiobookshelf`](https://gitea.alexandru.macocian.me/Charlie/audiobookshelf) | `audiobookshelf/` | upstream | audiobooks |
| [`directoryadmin`](https://gitea.alexandru.macocian.me/Charlie/directoryadmin) | `directoryadmin/` | gitea-registry `directoryadmin:latest` | LDAP admin UI |
### Singleton — morgott
| Charlie repo | Stack path | Image source | Notes |
|---|---|---|---|
| `portainer` | `portainer/` | upstream | container UI; CSRF trusted-origins set via CLI |
| [`portainer`](https://gitea.alexandru.macocian.me/Charlie/portainer) | `portainer/` | upstream | container UI; CSRF trusted-origins set via CLI |
| [`cert-issuer`](https://gitea.alexandru.macocian.me/Charlie/cert-issuer) | `cert-issuer/` | upstream `smallstep/step-ca:0.30.2` | SSH user-cert CA. `gssh` mints via JWK provisioner. Design: [cert-issuer.md](cert-issuer.md). |
| [`cert-syncer`](https://gitea.alexandru.macocian.me/Charlie/cert-syncer) | `cert-syncer/` | upstream `smallstep/step-cli:0.30.2-trixie` | Active bridge from cert-issuer to `/mnt/nas/ca/`. Renews managed server certs (today: `ldap.lan`), keeps trust artefacts fresh. Doesn't restart consumers — see cert-watch. Design: [cert-issuer.md](cert-issuer.md). |
### Sidecar image (consumed by other stacks)
| Charlie repo | Image | Used by | Notes |
|---|---|---|---|
| [`cert-watch`](https://gitea.alexandru.macocian.me/Charlie/cert-watch) | gitea-registry `cert-watch:latest` | [`ldap`](https://gitea.alexandru.macocian.me/Charlie/ldap) (today) | Tiny alpine sidecar. Polls cert files, restarts sibling containers on change via docker.sock. Add to any stack consuming a cert-syncer-managed cert. |
### Shared — multi-host
| Charlie repo | Hosts | Layout | Notes |
|---|---|---|---|
| `portainer-agent` | morgott, melina, mohg, miquella | `docker-compose.yml` + `docker-compose.miquella.yml` | DSM gets a per-host compose (no `/var/lib/docker/volumes` bind) |
| `otelcollector` | morgott, melina, miquella | `docker-compose.yml` (shared) | malenia (`launchd` + `install-mac.sh`) deferred — see roadmap |
| [`portainer-agent`](https://gitea.alexandru.macocian.me/Charlie/portainer-agent) | morgott, melina, mohg, miquella | `docker-compose.yml` + `docker-compose.miquella.yml` | DSM gets a per-host compose (no `/var/lib/docker/volumes` bind) |
| [`otelcollector`](https://gitea.alexandru.macocian.me/Charlie/otelcollector) | morgott, melina, miquella | `docker-compose.yml` (shared) | malenia (`launchd` + `install-mac.sh`) deferred — see roadmap |
## Planned (new infra)
| Charlie repo | Owner | Status | Notes |
|---|---|---|---|
| `runners` | morgott + melina | not started | Gitea Actions runners. Design: roadmap [Phase 2](roadmap.md#phase-2--ci). |
## Pending (Tier 5 — long tail)
docs/onboarding.md
+684
View File
@@ -0,0 +1,684 @@
# Host onboarding
End-to-end recipe for bringing a new host into the fleet. Migrated from the wiki so changes go through PRs.
> **Note on step-ca migration.** [`Charlie/cert-issuer`](cert-issuer.md) is deployed; its SSH user CA pubkey now lives at `/mnt/nas/ca/ssh/ssh_user_ca.pub` and is distributed by `sync-ca.sh` exactly as before. The X.509 root at `/mnt/nas/ca/ca.crt` (LDAP TLS) is still the original — not yet migrated. The pending rewrite of `sync-ca.sh` to fetch directly from step-ca's HTTPS endpoint is captured in [Step-ca migration impact](#step-ca-migration-impact). Until then, this section stays accurate.
## Order of operations
1. [Hostname](#hostname)
2. [NAS mount](#nas-mount)
3. [Docker](#docker)
4. [Portainer agent](#portainer-agent)
5. [Metrics (otelcollector)](#metrics-otelcollector)
6. [DNS](#dns)
7. [Trusted CA](#trusted-ca)
8. [LDAP / SSSD](#ldap--sssd)
9. [SSH defaults](#ssh-defaults)
Every step except (4) and (8) is required. (4) is required if the host runs containers. (8) is required if humans will SSH into it.
---
## Hostname
### Ubuntu
```sh
sudo hostnamectl set-hostname <HOSTNAME>
```
### Mac
```sh
sudo scutil --set HostName <HOSTNAME>
sudo scutil --set ComputerName <HOSTNAME>
sudo scutil --set LocalHostName <HOSTNAME>
# Add the new hostname to /etc/hosts.
sudo nano /etc/hosts
```
---
## NAS mount
NAS is `miquella.lan:/volume1/ubuntu` (NFSv4), mounted at `/mnt/nas` on every host.
### Ubuntu
Make sure mDNS works (Bonjour ships on macOS):
```sh
sudo apt install avahi-daemon avahi-utils
sudo systemctl enable --now avahi-daemon
avahi-browse -at # sanity check
```
Install NFS client utils:
```sh
sudo apt install nfs-common
```
Whitelist the host on the NAS: DSM → `Control Panel` → `Shared Folder` → `ubuntu` → `NFS Permissions`.
Add to `/etc/fstab`:
```
miquella.lan:/volume1/ubuntu /mnt/nas nfs vers=4.1,_netdev,nofail,x-systemd.automount,x-systemd.requires=network-online.target 0 0
```
### Mac
Redirect `/mnt/` to `/mnt/System/Volumes/Data/mnt/` via `synthetic.conf` (System Integrity Protection wants it that way):
```sh
sudo bash -c 'printf "mnt\tSystem/Volumes/Data/mnt\n" > /etc/synthetic.conf'
sudo chown root:wheel /etc/synthetic.conf
sudo chmod 644 /etc/synthetic.conf
sudo reboot
```
Edit `/etc/auto_master`, append at the bottom:
```
/- auto_nfs -nobrowse,nosuid
```
Create `/etc/auto_nfs`:
```
/System/Volumes/Data/mnt/nas -fstype=nfs,vers=4,resvport miquella.lan:/volume1/ubuntu
```
Reload:
```sh
sudo automount -vc
```
---
## Docker
### Ubuntu
If docker is installed via snap, **uninstall it** and reinstall via `apt`. Snap docker doesn't play nicely with `systemd` ordering against the NAS automount.
Wire docker to wait for the NAS:
```sh
sudo systemctl edit docker
```
Add:
```ini
[Unit]
Wants=network-online.target
After=network-online.target
RequiresMountsFor=/mnt/nas/docker
After=mnt-nas.automount
Requires=mnt-nas.automount
```
### Mac
```sh
brew install docker docker-compose docker-machine colima
```
Set colima to start at boot:
```sh
sudo cp /mnt/nas/home/plist/dev.colima.start.plist /Library/LaunchDaemons/dev.colima.start.plist
sudo chown root:wheel /Library/LaunchDaemons/dev.colima.start.plist
sudo chmod 644 /Library/LaunchDaemons/dev.colima.start.plist
sudo launchctl load -w /Library/LaunchDaemons/dev.colima.start.plist
```
> `home/plist/` is Tier 5 backlog — see [`plist`](repos.md#pending--tier-5-long-tail). Path will move once that repo is migrated.
---
## Portainer agent
Bring the agent up locally so the host shows up in portainer:
```sh
cd /mnt/nas/stacks/portainer-agent
sudo docker compose up -d
```
If the host needs a different layout (e.g. DSM), `deploy.sh` picks the right per-host compose. See [strategy.md → Shared stacks](strategy.md#shared-stacks).
Then go to https://portainer.alexandru.macocian.me and add a new environment for this host.
---
## Metrics (otelcollector)
### Ubuntu
```sh
cd /mnt/nas/stacks/otelcollector
sudo docker compose up -d
```
### Mac
The mac path doesn't run otelcollector in docker (NFS / colima friction). Install the binary directly:
```sh
cd ~
# Check latest release, swap the version below if newer.
curl -s https://api.github.com/repos/open-telemetry/opentelemetry-collector-releases/releases/latest | grep tag_name
curl -LO https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.144.0/otelcol-contrib_0.144.0_darwin_amd64.tar.gz
tar -xzf otelcol-contrib_0.144.0_darwin_amd64.tar.gz
sudo mv otelcol-contrib /usr/local/bin/otelcol-contrib
sudo chmod +x /usr/local/bin/otelcol-contrib
rm otelcol-contrib_0.144.0_darwin_amd64.tar.gz
otelcol-contrib --version
```
Config + LaunchDaemon:
```sh
sudo mkdir -p /usr/local/etc/otelcol-contrib
sudo cp /mnt/nas/stacks/otelcollector/otelcol.mac.yaml /usr/local/etc/otelcol-contrib/config.yaml
sudo cp /mnt/nas/stacks/otelcollector/com.otelcol.contrib.plist /Library/LaunchDaemons/com.otelcol.contrib.plist
sudo chown root:wheel /Library/LaunchDaemons/com.otelcol.contrib.plist
sudo chmod 644 /Library/LaunchDaemons/com.otelcol.contrib.plist
sudo launchctl load /Library/LaunchDaemons/com.otelcol.contrib.plist
```
Verify:
```sh
ps aux | grep otelcol
tail -f /var/log/otelcol-contrib.log
tail -f /var/log/otelcol-contrib.err
```
Service management:
```sh
# Stop
sudo launchctl unload /Library/LaunchDaemons/com.otelcol.contrib.plist
sudo pkill -9 otelcol-contrib
# Start
sudo launchctl load /Library/LaunchDaemons/com.otelcol.contrib.plist
# Restart
sudo launchctl unload /Library/LaunchDaemons/com.otelcol.contrib.plist
sudo launchctl load /Library/LaunchDaemons/com.otelcol.contrib.plist
```
> Tracked: re-onboard malenia (mac) for otelcollector — currently deferred (NFS unreliable from malenia). See [roadmap.md → Phase 1](roadmap.md#phase-1--reorg--charlie-repos).
---
## DNS
Intranet DNS is served by **Marika** (OpenWrt router). DHCP from Marika hands out Marika as the resolver, so a new host doesn't need any client-side DNS config — just a working DHCP lease.
### On Marika (one-time per new host)
Add a static lease + hostname so the host gets a stable IP and an `<hostname>.lan` A record:
- LuCI → **Network → DHCP and DNS → Static Leases** → add MAC + hostname + IP.
- Or `/etc/config/dhcp`:
```
config host
option name '<hostname>'
option mac '<mac>'
option ip '<ip>'
```
then `/etc/init.d/dnsmasq restart` on the router.
### On the new host
```sh
# Force a lease refresh so the host picks up Marika as the resolver
# (only needed if the host is already up and currently using a stale config).
sudo dhclient -r && sudo dhclient
```
Verify:
```sh
resolvectl status | grep -A1 'Current DNS Server' # expect: Marika's IP
getent hosts <some-other-host>.lan # should resolve
```
> Historically there was a `docker-dnsmasq` stack on miquella; that's gone. Marika's built-in dnsmasq is the only resolver now.
---
## Trusted CA
> **Partially migrated.** The SSH user CA pubkey at `ssh/ssh_user_ca.pub` is now step-ca's (rolled May 4, 2026). The X.509 root at `ca.crt` is still the original. Both are still distributed by `sync-ca.sh` from `/mnt/nas/ca/`. Pending rewrite of the script to fetch directly from step-ca: see [Step-ca migration impact](#step-ca-migration-impact).
Source of truth: `/mnt/nas/ca/`
- `ca.crt` — internal X.509 root (LDAP TLS, internal services). Original.
- `ssh/ssh_user_ca.pub` — SSH user-cert CA pubkey (sshd `TrustedUserCAKeys`). Now contains step-ca's user CA pubkey from [`Charlie/cert-issuer`](cert-issuer.md).
Each host runs `sync-ca.sh` periodically to copy these into local trust stores and bounce sssd / sshd if anything changed.
### Ubuntu
`/usr/local/sbin/sync-ca.sh` (mode `0755`, `root:root`):
```bash
#!/usr/bin/env bash
set -euo pipefail
SRC="/mnt/nas/ca/ca.crt"
SSSD_DST="/etc/sssd/certs/ca.crt"
SYS_DST="/usr/local/share/ca-certificates/macocian-ca.crt"
SSH_USER_CA_SRC="/mnt/nas/ca/ssh/ssh_user_ca.pub"
SSH_USER_CA_DST="/etc/ssh/trusted_user_ca.pub"
install -d -m 0755 /etc/sssd/certs
install -d -m 0755 /usr/local/share/ca-certificates
install -d -m 0755 /etc/ssh
if [[ ! -f "$SRC" ]]; then
echo "sync-ca: source $SRC not found; skipping"
exit 0
fi
changed_trust=0
changed_sshd=0
copy_if_changed () {
local src="$1" dst="$2" flagvar="$3"
if [[ ! -f "$src" ]]; then
echo "sync-ca: source $src not found; skipping its target $dst"
return 0
fi
local tmp
tmp="$(mktemp "${dst}.XXXX")"
cp -f "$src" "$tmp"
chmod 0644 "$tmp"
if [[ ! -f "$dst" ]] || ! cmp -s "$tmp" "$dst"; then
mv -f "$tmp" "$dst"
chown root:root "$dst" || true
echo "sync-ca: updated $dst"
if [[ "$flagvar" == "changed_trust" ]]; then
changed_trust=1
elif [[ "$flagvar" == "changed_sshd" ]]; then
changed_sshd=1
fi
else
rm -f "$tmp"
fi
}
copy_if_changed "$SRC" "$SSSD_DST" changed_trust
copy_if_changed "$SRC" "$SYS_DST" changed_trust
copy_if_changed "$SSH_USER_CA_SRC" "$SSH_USER_CA_DST" changed_sshd
if [[ "$changed_trust" -eq 1 ]]; then
update-ca-certificates >/dev/null 2>&1 || true
systemctl try-reload-or-restart sssd || true
fi
if [[ "$changed_sshd" -eq 1 ]]; then
systemctl try-reload-or-restart sshd || systemctl try-reload-or-restart ssh || true
fi
```
Systemd units:
`/etc/systemd/system/sync-ca.service`:
```ini
[Unit]
Description=Sync private CA from NAS into local trust
Wants=network-online.target
After=network-online.target
RequiresMountsFor=/mnt/nas/ca
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/sync-ca.sh
```
`/etc/systemd/system/sync-ca.timer`:
```ini
[Unit]
Description=Run sync-ca hourly and at boot
[Timer]
OnBootSec=2min
OnUnitActiveSec=1h
Unit=sync-ca.service
Persistent=true
[Install]
WantedBy=timers.target
```
Permissions + enable:
```bash
sudo chmod 755 /usr/local/sbin/sync-ca.sh
sudo chown root:root /usr/local/sbin/sync-ca.sh
sudo systemctl daemon-reload
sudo systemctl enable --now sync-ca.timer
sudo systemctl start sync-ca.service
```
### Mac
`/usr/local/sbin/sync-ca.sh` (mode `0755`):
```bash
#!/usr/bin/env bash
set -euo pipefail
SRC="/mnt/nas/ca/ca.crt"
# macOS equivalent for LDAP CLI trust:
SSSD_DST="/etc/openldap/cacerts/ca.crt"
SYS_DST="/usr/local/share/ca-certificates/macocian-ca.crt"
SSH_USER_CA_SRC="/mnt/nas/ca/ssh/ssh_user_ca.pub"
SSH_USER_CA_DST="/etc/ssh/trusted_user_ca.pub"
install -d -m 0755 /etc/openldap/cacerts
install -d -m 0755 /usr/local/share/ca-certificates
install -d -m 0755 /etc/ssh
if [[ ! -f "$SRC" ]]; then
echo "sync-ca: source $SRC not found; skipping"
exit 0
fi
changed_trust=0
changed_sshd=0
copy_if_changed () {
local src="$1" dst="$2" flagvar="$3"
if [[ ! -f "$src" ]]; then
echo "sync-ca: source $src not found; skipping its target $dst"
return 0
fi
local tmp
tmp="$(mktemp "${dst}.XXXX")"
cp -f "$src" "$tmp"
chmod 0644 "$tmp"
if [[ ! -f "$dst" ]] || ! cmp -s "$tmp" "$dst"; then
mv -f "$tmp" "$dst"
chown root:wheel "$dst" 2>/dev/null || chown root:root "$dst" || true
echo "sync-ca: updated $dst"
if [[ "$flagvar" == "changed_trust" ]]; then
changed_trust=1
elif [[ "$flagvar" == "changed_sshd" ]]; then
changed_sshd=1
fi
else
rm -f "$tmp"
fi
}
copy_if_changed "$SRC" "$SSSD_DST" changed_trust
copy_if_changed "$SRC" "$SYS_DST" changed_trust
copy_if_changed "$SSH_USER_CA_SRC" "$SSH_USER_CA_DST" changed_sshd
if [[ "$changed_trust" -eq 1 ]]; then
/usr/bin/security add-trusted-cert -d -r trustRoot \
-k /Library/Keychains/System.keychain "$SSSD_DST" || true
if [[ -f /etc/openldap/ldap.conf ]]; then
if ! grep -qE '^TLS_CACERT[[:space:]]+/etc/openldap/cacerts/ca\.crt$' /etc/openldap/ldap.conf; then
sudo sed -i '' -e '/^TLS_CACERT/d' /etc/openldap/ldap.conf 2>/dev/null || true
echo "TLS_CACERT /etc/openldap/cacerts/ca.crt" | sudo tee -a /etc/openldap/ldap.conf >/dev/null
echo "sync-ca: updated /etc/openldap/ldap.conf"
fi
else
echo "TLS_CACERT /etc/openldap/cacerts/ca.crt" | sudo tee /etc/openldap/ldap.conf >/dev/null
echo "sync-ca: created /etc/openldap/ldap.conf"
fi
fi
if [[ "$changed_sshd" -eq 1 ]]; then
/bin/launchctl kickstart -k system/com.openssh.sshd || true
fi
```
LaunchDaemon at `/Library/LaunchDaemons/com.macocian.sync-ca.plist`:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.macocian.sync-ca</string>
<key>ProgramArguments</key>
<array>
<string>/usr/local/sbin/sync-ca.sh</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>StartInterval</key>
<integer>3600</integer>
<key>StandardOutPath</key>
<string>/var/log/sync-ca-macos.log</string>
<key>StandardErrorPath</key>
<string>/var/log/sync-ca-macos.log</string>
</dict>
</plist>
```
Permissions + load:
```sh
sudo chown root:wheel /Library/LaunchDaemons/com.macocian.sync-ca.plist
sudo chmod 0644 /Library/LaunchDaemons/com.macocian.sync-ca.plist
sudo launchctl load -w /Library/LaunchDaemons/com.macocian.sync-ca.plist
```
Append to `/etc/ssh/sshd_config`:
```
TrustedUserCAKeys /etc/ssh/trusted_user_ca.pub
PubkeyAuthentication yes
```
```sh
sudo launchctl kickstart -k system/com.openssh.sshd
```
---
## LDAP / SSSD
Identity: SSSD-backed against `ldaps://ldap.lan` (the [`ldap`](https://gitea.alexandru.macocian.me/Charlie/ldap) stack on morgott).
### Ubuntu
Install SSSD:
```bash
sudo apt update
sudo apt install sssd-ldap libpam-sss libnss-sss oddjob-mkhomedir oddjob sssd-tools
```
`/etc/sssd/sssd.conf`:
```ini
[sssd]
services = nss, pam
config_file_version = 2
domains = LDAP
[domain/LDAP]
debug_level = 9
id_provider = ldap
auth_provider = ldap
ldap_uri = ldaps://ldap.lan
ldap_search_base = dc=alexandru,dc=macocian,dc=me
ldap_default_bind_dn = cn=admin,dc=alexandru,dc=macocian,dc=me
ldap_default_authtok = __PASSWORD__
ldap_id_use_start_tls = false
ldap_tls_reqcert = demand
ldap_tls_cacert = /etc/sssd/certs/ca.crt
# posixAccount + posixGroup/memberUid
ldap_schema = rfc2307
ldap_user_object_class = posixAccount
ldap_user_name = uid
ldap_user_uid_number = uidNumber
ldap_user_gid_number = gidNumber
ldap_user_home_directory = homeDirectory
ldap_user_shell = loginShell
ldap_group_object_class = posixGroup
ldap_group_name = cn
ldap_group_gid_number = gidNumber
ldap_group_member = memberUid
```
Permissions + restart:
```bash
sudo chmod 600 /etc/sssd/sssd.conf
sudo systemctl restart sssd
```
`/etc/ssh/sshd_config`:
```
PubkeyAuthentication yes
TrustedUserCAKeys /etc/ssh/trusted_user_ca.pub
```
Append to **both** `/etc/pam.d/common-session` and `/etc/pam.d/common-session-noninteractive`:
```
session required pam_mkhomedir.so skel=/etc/skel/ umask=0077
```
Grant `admins` sudo:
```bash
echo '%admins ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/90-ldap-admins
sudo visudo -cf /etc/sudoers.d/90-ldap-admins
```
### Mac
Copy the LDAP plist:
```sh
sudo cp /mnt/nas/home/plist/ldap.lan.plist /Library/Preferences/OpenDirectory/Configurations/LDAPv3/ldap.lan.plist
```
> Bind password may need refreshing in Directory Utility after copy.
Grant SSH to `users`:
```sh
sudo dseditgroup -o create com.apple.access_ssh
sudo dscl . -append /Groups/com.apple.access_ssh NestedGroups "users@/LDAPv3/ldap.lan"
sudo dseditgroup -o create -n /Local/Default com.apple.access_loginwindow
sudo dseditgroup -o edit -a users -t group -n /Local/Default com.apple.access_loginwindow
sudo dscl . -append /Groups/com.apple.access_loginwindow NestedGroups "users@/LDAPv3/ldap.lan"
sudo dseditgroup -o edit -a "users@/LDAPv3/ldap.lan" -t group com.apple.access_loginwindow
```
Grant sudo to `admins`:
```sh
sudo dseditgroup -o create -n /Local/Default admins
sudo dscl . -append /Groups/admins NestedGroups "admins@/LDAPv3/ldap.lan"
echo '%admins ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/90-ldap-admins >/dev/null
sudo chmod 440 /etc/sudoers.d/90-ldap-admins
sudo visudo -cf /etc/sudoers.d/90-ldap-admins
```
---
## SSH defaults
Global readline config (advanced command-line editing):
```sh
sudo cp /mnt/nas/home/shell/inputrc /etc/inputrc
```
> `home/shell/` is Tier 5 backlog. Path will move once [`shell`](repos.md#pending--tier-5-long-tail) is migrated.
Color + sane defaults in `/etc/profile`:
```sh
sudo tee -a /etc/profile >/dev/null <<'EOF'
# === Global color and terminal setup ===
export TERM=${TERM:-xterm-256color}
export CLICOLOR=1
export LSCOLORS=ExFxBxDxCxegedabagacad
# Make sure /etc/inputrc is used
export INPUTRC=/etc/inputrc
# Colored prompt for interactive shells
if [ "$PS1" ]; then
PS1='\[\e[1;32m\]\u@\h\[\e[0m\]:\[\e[1;34m\]\w\[\e[0m\]\$ '
fi
EOF
```
---
## Step-ca migration impact
[`Charlie/cert-issuer`](cert-issuer.md) is deployed. Migration so far:
- ✅ SSH user CA at `/mnt/nas/ca/ssh/ssh_user_ca.pub` swapped to step-ca's pubkey. `sync-ca.sh` distributes it unchanged.
- ✅ [`Charlie/gssh`](https://gitea.alexandru.macocian.me/Charlie/gssh) rewritten as a step-ca client. Old CA key no longer mounted into Gssh.
Pending. When [`Charlie/cert-issuer`](cert-issuer.md) takes over from `/mnt/nas/ca/` entirely, this is what changes for onboarding:
| Today | After step-ca takes over |
|---|---|
| `sync-ca.sh` reads `/mnt/nas/ca/ca.crt` | Pulls from `https://cert-issuer.lan:9000/roots.pem` (or the bootstrapped `~/.step/certs/root_ca.crt`). |
| `sync-ca.sh` reads `/mnt/nas/ca/ssh/ssh_user_ca.pub` | Pulls via `step ssh config --roots`. |
| Trust source = NFS share readable by every host | Trust source = HTTPS endpoint pinned by root fingerprint at first contact. |
| SSH host trust = TOFU (`known_hosts` accumulates) | sshd presents an SSH host cert from step-ca (`HostCertificate`); clients trust via `@cert-authority *.lan ...`. |
| Adding a host = NFS whitelist + `sync-ca.timer` | Adding a host = NFS whitelist + one-time fetch of root + register host with step-ca for its host cert. |
What does **not** change:
- LDAP / SSSD section. SSSD still wants `ldap_tls_cacert = /etc/sssd/certs/ca.crt`. step-ca populates that file via the same `sync-ca.sh` rewrite.
- The hourly refresh model. Just changes its *source*, not its *behaviour*.
- Everything outside the [Trusted CA](#trusted-ca) section.
What goes away once the rewrite ships:
- `/mnt/nas/ca/` as a NAS dependency. After migration nothing reads it; archive then delete.
- The `sync-ca.timer` hard dependency on `RequiresMountsFor=/mnt/nas/ca` (since it'll fetch over HTTPS, not NFS).
The actual rewrite of `sync-ca.sh` (and its Mac equivalent) lives with [`Charlie/cert-issuer`](cert-issuer.md). Until then this section stays accurate and authoritative.
docs/repos.md
+17 -10
View File
@@ -5,36 +5,43 @@ See [inventory.md](inventory.md) for the canonical state. Migration recipe: [mig
## Done
### Tier 1 — pilot (Day 1)
`ddclient`, `homarr`, `vaultwarden`
[`ddclient`](https://gitea.alexandru.macocian.me/Charlie/ddclient), [`homarr`](https://gitea.alexandru.macocian.me/Charlie/homarr), [`vaultwarden`](https://gitea.alexandru.macocian.me/Charlie/vaultwarden)
### Tier 2 — our code (Day 1)
`caddy`, `gssh`, `directoryadmin`, `mediacompose`
[`caddy`](https://gitea.alexandru.macocian.me/Charlie/caddy), [`gssh`](https://gitea.alexandru.macocian.me/Charlie/gssh), [`directoryadmin`](https://gitea.alexandru.macocian.me/Charlie/directoryadmin), [`mediacompose`](https://gitea.alexandru.macocian.me/Charlie/mediacompose)
### Tier 3 — pinned upstream + stateful (Day 1)
`ldap`, `authentik`, `bookstack`, `monica`, `miniflux`, `planka`, `audiobookshelf`, `gitea`
[`ldap`](https://gitea.alexandru.macocian.me/Charlie/ldap), [`authentik`](https://gitea.alexandru.macocian.me/Charlie/authentik), [`bookstack`](https://gitea.alexandru.macocian.me/Charlie/bookstack), [`monica`](https://gitea.alexandru.macocian.me/Charlie/monica), [`miniflux`](https://gitea.alexandru.macocian.me/Charlie/miniflux), [`planka`](https://gitea.alexandru.macocian.me/Charlie/planka), [`audiobookshelf`](https://gitea.alexandru.macocian.me/Charlie/audiobookshelf), [`gitea`](https://gitea.alexandru.macocian.me/Charlie/gitea)
### Tier 4 — was upstream-cloned, collapsed to canonical (Day 1)
`mealie`, `paperless-ngx`, `victoriametrics`, `searxng`, `invidious`
[`mealie`](https://gitea.alexandru.macocian.me/Charlie/mealie), [`paperless-ngx`](https://gitea.alexandru.macocian.me/Charlie/paperless-ngx), [`victoriametrics`](https://gitea.alexandru.macocian.me/Charlie/victoriametrics), [`searxng`](https://gitea.alexandru.macocian.me/Charlie/searxng), [`invidious`](https://gitea.alexandru.macocian.me/Charlie/invidious)
### Tier 5 — long tail (Day 1–2)
`portainer`, `portainer-agent` (shared, 4 hosts), `otelcollector` (shared, 3 hosts — malenia deferred)
[`portainer`](https://gitea.alexandru.macocian.me/Charlie/portainer), [`portainer-agent`](https://gitea.alexandru.macocian.me/Charlie/portainer-agent) (shared, 4 hosts), [`otelcollector`](https://gitea.alexandru.macocian.me/Charlie/otelcollector) (shared, 3 hosts — malenia deferred)
### Post-migration infra (May 4, 2026)
[`cert-issuer`](https://gitea.alexandru.macocian.me/Charlie/cert-issuer) — step-ca on morgott, replaces the per-host SSH CA mount that `gssh` used to need. [`cert-syncer`](https://gitea.alexandru.macocian.me/Charlie/cert-syncer) — active bridge from step-ca to `/mnt/nas/ca/`; renews managed server certs (today: `ldap.lan`) and keeps trust artefacts fresh. [`cert-watch`](https://gitea.alexandru.macocian.me/Charlie/cert-watch) — sidecar image that lives next to a consumer of a cert-syncer-managed cert and restarts it on change. Design: [cert-issuer.md](cert-issuer.md).
### Decommissioned
`watchtower` — retired without replacement (May 3, 2026). Diun-based image-update flow planned (see roadmap Phase 3).
## Planned (new infra)
| # | Repo | Owner | Notes |
|---|---|---|---|
| 1 | `runners` | morgott + melina | Gitea Actions runners. Design: roadmap [Phase 2](roadmap.md#phase-2--ci). |
## Pending — Tier 5 long tail
| # | Repo | Owner | Notes |
|---|---|---|---|
| 1 | `shell` | — (no docker) | clone target: `/usr/local/share/charlie-shell/` |
| 2 | `first-time-setup` | — (docs) | bootstrap docs / scripts |
| 2 | `first-time-setup` | — (docs) | bootstrap docs / scripts. Live recipe (preserved from wiki): [onboarding.md](onboarding.md). Will become a standalone repo once the [step-ca migration](onboarding.md#step-ca-migration-impact) settles. |
| 3 | `server-mgmt-utils` | morgott | mostly dead. Extract temp/storage scripts to `otelcollector/scripts/` or new `host-scripts` repo, then archive the original. |
| ? | `plist` | TBD | classify before deciding |
## Build pipelines (deferred)
Once runners are up, replace "build on laptop" with workflows on:
- `amacocian/Gssh` → push `gssh:latest`
- `amacocian/DirectoryAdmin` → push `directoryadmin:latest`
- `amacocian/monica` (`4.x`) → push `monica:4.x`
- `amacocian/invidious` (`oidc-support`) → push `invidious:oidc-support`
- [`amacocian/Gssh`](https://gitea.alexandru.macocian.me/amacocian/Gssh) → push `gssh:latest`
- [`amacocian/DirectoryAdmin`](https://gitea.alexandru.macocian.me/amacocian/DirectoryAdmin) → push `directoryadmin:latest`
- [`amacocian/monica`](https://gitea.alexandru.macocian.me/amacocian/monica) (`4.x`) → push `monica:4.x`
- [`amacocian/invidious`](https://gitea.alexandru.macocian.me/amacocian/invidious) (`oidc-support`) → push `invidious:oidc-support`
docs/roadmap.md
+3 -2
View File
@@ -17,7 +17,7 @@
## Phase 2 — CI
- [ ] Design runner stack — `Charlie/runners`. Open question: secrets sourcing (vaultwarden? Gitea Actions secrets? SOPS?).
- [ ] Design runner stack — `Charlie/runners`. Secrets sourcing design + open questions in [secrets.md](secrets.md).
- [ ] Stand up runners on morgott + melina, register org-scope to `Charlie`.
- [ ] First end-to-end pipeline: push to `Charlie/ddclient` → webhook → redeploy on morgott.
- [ ] Build pipelines on `amacocian/*` source repos to push images to gitea registry (replaces "build on laptop").
@@ -55,10 +55,11 @@ These don't block earlier phases, but capture them now so they're not forgotten.
### Secrets
- [ ] **Decide on secrets sourcing for runners + stacks.** Candidates: vaultwarden as source-of-truth + a runner-side fetch step that materialises `.env` per job, Gitea Actions secrets (per-repo), SOPS-encrypted-in-repo. Investigate before any deploy pipeline that needs to write a `.env`.
- [ ] **Decide on secrets sourcing for runners + stacks.** Design + survey of existing tools in [secrets.md](secrets.md). Open: bootstrap auth model, per-host vs per-collection scoping. Settle before any deploy pipeline that needs to write a `.env`.
### Identity & access
- [x] **Extract SSH cert issuance into a standalone service.** [`Charlie/cert-issuer`](https://gitea.alexandru.macocian.me/Charlie/cert-issuer) (step-ca) deployed on morgott May 4, 2026. Gssh rewritten as a step-ca client; CA key no longer mounted into Gssh. Follow-up work (sync-ca rewrite, OIDC provisioner, host certs, TPM-sealed CA key) tracked in [cert-issuer.md → Next steps](cert-issuer.md#next-steps).
- [ ] **Auto-mount user home folders on SSH connect.** Currently each user's `$HOME` is local to the host they SSH into. Want each user's home to be a NAS-backed share automounted on connect (NFS or autofs + LDAP user attribute). Should pair with the LDAP-backed SSSD setup (`Charlie/ldap`). Design before more users land.
- [ ] **Miquella into the LDAP/SSH fold.** DSM is currently outside the SSSD-backed identity. Configure DSM's LDAP client to point at `Charlie/ldap`, trust the SSH signing CA (`/volume1/ubuntu/ca/ssh/ssh_user_ca.pub`), and accept ephemeral certs from gssh. Goal: malenia + miquella both onboarded into the gssh terminal flow with LDAP-driven access.
- [ ] **Per-host git identity.** Each host needs its own credential to talk to Gitea — currently we paste a PAT inline. Decide: per-host PAT stored in the host's keyring, host-keyed SSH key registered as a deploy key on each `Charlie/<stack>` (rotation pain), or one "host-fleet" service account in Gitea with org-scope read + per-stack write.
docs/secrets.md
+159
View File
@@ -0,0 +1,159 @@
# Secrets sourcing
Status: **draft / decision pending**. Captures the design space so [Phase 2 (runners)](roadmap.md#phase-2--ci) and [Phase 5 → Secrets](roadmap.md#secrets) can land on the same answer.
## Goal
Every host can, on demand:
1. Obtain only the secrets it's allowed to decrypt (and nothing else).
2. Materialize them where compose / runner jobs can consume them (typically `.env` files).
3. Do all of the above non-interactively, suitable for being driven by a Gitea Actions runner job.
Out of scope: per-user secrets, browser-side autofill, secret editing UX. Vaultwarden continues to serve those for humans.
## Constraints
- **Source of truth lives with the stack.** A secret for `mediacompose` belongs in [`Charlie/mediacompose`](https://gitea.alexandru.macocian.me/Charlie/mediacompose), not in some separate vault we have to keep in sync. The blast radius of a leak is then naturally bounded by repo access.
- **Per-machine decryption.** A compromised host should leak only the secrets it's already cleared to read. Concretely: only the secrets in repos whose `.sops.yaml` lists that host's key.
- **Self-identifying host.** Same convention used by [`deploy.sh`](strategy.md#canonical-deploysh): `hostname -s | tr '[:upper:]' '[:lower:]'`.
- **No `:latest` problem for secrets.** Versioning lives in git like everything else; rotation is a commit + `sops updatekeys`.
## Direction: SOPS in-repo
Encrypted files live next to `docker-compose.yml` in each `Charlie/<stack>` repo. They're decrypted on the owner host at deploy time. The runner pulls the repo, decrypts, runs compose, deletes the plaintext.
Why this over a central vault:
- **No "who controls the secrets" question.** Whoever can write to `Charlie/mediacompose` controls mediacompose's secrets. We already have to answer that question for code; we don't need to answer it twice.
- **No central control plane to keep available.** Vault outage doesn't block deploys. A vault that has to be online for every deploy is itself a stack with secrets, and the bootstrap problem becomes recursive.
- **Per-repo access control falls out for free.** Per-stack access is `.sops.yaml`'s list of recipients. Per-host access is the same list, viewed from the host side.
- **Reads as code in PRs.** Diffs show *that* a secret changed (and which one), not the value. No second review surface.
What we give up:
- A "where is this password" lookup for humans. Solved by: humans keep using vaultwarden for that. The two systems don't need to share storage; they only need to not contradict each other (rotation hygiene).
- Online rotation. Rotating means commit + push + redeploy, not "click a button in the UI."
### Layout in a stack repo
```
Charlie/<stack>/
├── docker-compose.yml
├── .env.example # tracked, redacted, as today
├── secrets/
│ └── env.sops.yaml # encrypted; decrypts to .env
├── .sops.yaml # creation rules: which recipients can decrypt this dir
└── deploy.sh
```
`.sops.yaml` (sketch):
```yaml
creation_rules:
- path_regex: secrets/.*\.sops\.(ya?ml|env|json)$
age: >-
age1morgott...,
age1admin-alex...
```
For shared stacks ([`otelcollector`](https://gitea.alexandru.macocian.me/Charlie/otelcollector), [`portainer-agent`](https://gitea.alexandru.macocian.me/Charlie/portainer-agent)), the rule lists every host that runs the stack — same multi-host pattern as the compose layout already does.
### Inside `deploy.sh`
```bash
# Materialize plaintext for this deploy, then drop it after up -d.
trap 'rm -f .env' EXIT
sops --decrypt secrets/env.sops.yaml > .env
chmod 600 .env
sudo docker compose --env-file .env "${args[@]}" up -d
```
Tmpfs-backed working dir would be cleaner; keeping plaintext in CWD is fine for a v1 because the trap covers it and the file is `0600`.
## The actual open question: decryption
SOPS supports several recipient backends. Picking which one(s) we use is the load-bearing decision in this doc.
| Backend | Fit | Notes |
|---|---|---|
| **age key file** | **Leading candidate.** | One private key per host, file at `/etc/charlie/age.key` (root, `0600`). Public key listed in each repo's `.sops.yaml` that the host needs to decrypt. Bootstrap = generate key once, paste pub into the right `.sops.yaml`(s), commit. Heterogeneous hosts (DSM, mac mini) all just need a file. |
| **age via SSH host key** | Promising. | `age` can use an existing SSH ed25519 key as a recipient. Reuses something every host already has. Caveat: SOPS support for SSH-as-age recipients goes through the `ssh-to-age` conversion — workable, slightly more moving parts than a dedicated age key. |
| **GPG** | No. | Heavier, key-management UX is worse, no advantage over age. |
| **HashiCorp Vault Transit** | No. | Reintroduces the central control plane we're trying to avoid. |
| **Cloud KMS** (AWS / GCP / Azure) | No. | Off-prem dependency for an on-prem homelab. |
| **TPM-sealed age key** | Maybe later. | Strongest. Most x86 hosts have TPM 2.0; mac mini and DSM do not. Heterogeneity makes it a bad default; reasonable to add per-host as a hardening step after v1. |
| **Vaultwarden-stored age key, fetched at bootstrap** | Hybrid. | Age key lives encrypted in vaultwarden; fetched once during host bootstrap (same flow as CA trust / SSSD setup) and written to `/etc/charlie/age.key`. Rotation = re-fetch. Doesn't solve "how do you authenticate to vaultwarden during bootstrap" — but that's a one-time, human-present operation, not a per-deploy one. |
### Recommended starting point
- **age key per host**, generated at first-time-setup, stored at `/etc/charlie/age.key` (root, `0600`).
- Public key is non-secret; published in [`docs/hosts.md`](hosts.md) and added to the `.sops.yaml` of each stack repo the host owns.
- A second "admin" age key (held by alex, on the admin laptop) is added to *every* repo's `.sops.yaml` so secrets stay decryptable / re-encryptable from off-host.
- **Bootstrap mechanism for the host's age key**: TBD — see [Open question #1](#1-host-age-key-bootstrap). Strong default: generate during first-time-setup, paste pub key into `.sops.yaml`s, commit.
## Open questions
### 1. Host age key bootstrap
How does the key get onto a freshly-installed host?
- **Generate in place during first-time-setup**, capture the pub key, commit it into the relevant `.sops.yaml`s. Private key never leaves the host. Simple, recommended.
- **Mint centrally** (admin laptop), copy to the host over SSH during bootstrap. Easier to pre-stage `.sops.yaml`s before the host exists, but the key crosses a wire once.
- **Store in vaultwarden**, fetch at bootstrap. Lets you re-image a host without rotating its key. Buys: re-image without `sops updatekeys` churn. Costs: vaultwarden has to be reachable (and authenticatable-to) during host bootstrap.
Recommend option 1 unless re-imaging frequency makes the rotation cost felt.
### 2. Admin key custody
The "decrypt anywhere" admin key is the master skeleton key for the fleet. Where does it live?
- Admin laptop only, file at `~/.config/sops/age/keys.txt` (`0600`). Loss = re-key every repo.
- Admin laptop **and** vaultwarden as backup. Loss recoverable.
- Hardware-backed (YubiKey + age-plugin-yubikey). Strongest, adds a hardware dependency for routine ops.
### 3. Per-environment / per-host secrets in the same repo
Shared stacks ([`otelcollector`](https://gitea.alexandru.macocian.me/Charlie/otelcollector), [`portainer-agent`](https://gitea.alexandru.macocian.me/Charlie/portainer-agent)) need different secrets per host. Two shapes:
- One file per host: `secrets/morgott.env.sops.yaml`, `secrets/melina.env.sops.yaml`, …. `deploy.sh` picks by hostname (mirrors the existing compose-per-host pattern).
- One file with all of them, every host's key on the recipient list. Cleaner, but means morgott's key can decrypt melina's secrets in that one file. Loses the per-host blast-radius property.
Recommend the per-file shape — it's the only one consistent with the constraint above.
### 4. Materialization target
- Plaintext `.env` in the working tree, deleted on EXIT trap (proposed default).
- `/run/charlie-secrets/<stack>/.env` (tmpfs). Survives the deploy script but not reboots; nicer if multiple commands need to read it.
- Process substitution / fd, no file ever on disk. Cleanest, but breaks `docker compose config` debugging and not all compose versions handle it.
### 5. Vaultwarden's role going forward
- Pure human use case (browser autofill, secure notes). No change.
- Optional: store the age private key as a backup item per host. Useful for re-imaging.
- Optional: store the admin age key (if not on YubiKey).
Vaultwarden becomes a backup / human convenience, not a runtime dependency.
## Things this design intentionally does not do
- Manage secrets for stacks that aren't runner-deployed. Those keep their hand-edited `.env` until they migrate.
- Replace vaultwarden's human-facing UX.
- Solve secrets for `Charlie/gitea` itself — gitea-on-melina is deployed manually (see [conventions](conventions.md)) and bootstraps the rest of the system; it can't depend on a runner that depends on it. But: SOPS is offline crypto, so gitea's secrets *can* live in `Charlie/gitea` encrypted to melina's age key, and a manual `sops -d | docker compose up -d` works without any other Charlie service being up.
## Stacks already on plain `.env` waiting for SOPS
These migrated before secrets work landed and currently keep credentials in hand-edited `.env` files on morgott. They'll be the first to fold into the SOPS pattern once it ships:
- [`cert-issuer`](https://gitea.alexandru.macocian.me/Charlie/cert-issuer) — `DOCKER_STEPCA_INIT_PASSWORD` (vault unseal). Losing it = re-init the CA.
- [`gssh`](https://gitea.alexandru.macocian.me/Charlie/gssh) — `Issuer__ProvisionerPassword` (the JWK provisioner password Gssh uses to authenticate to step-ca). Losing it = remove + re-add the `gssh` provisioner.
Both currently in vaultwarden as the source of truth; `.env` on morgott is the working copy. The SOPS migration replaces the working copy, not the source of truth.
## Next steps
1. Settle [open questions 1 + 2](#open-questions) here, in-place.
2. Generate age keys for morgott + melina + mohg + miquella + malenia. Capture pub keys in [`docs/hosts.md`](hosts.md).
3. Spike on [`Charlie/ddclient`](https://gitea.alexandru.macocian.me/Charlie/ddclient) (Tier 1, single-host, tiny `.env`): add `.sops.yaml`, commit `secrets/env.sops.yaml`, wire into `deploy.sh`.
4. Roll across migrated stacks one tier at a time.
5. Fold the result into the [`Charlie/runners` design](roadmap.md#phase-2--ci) before any pipeline depends on it.