commit 615575b087bcf6f3a7f931f9a397a0d4130b3f83 Author: Alexandru Macocian Date: Sun May 3 13:31:05 2026 +0200 Initial scaffold: docs, inventory, strategy, migration recipes diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..85f3364 --- /dev/null +++ b/.gitignore @@ -0,0 +1,15 @@ +# Catch-all for env files. Per-stack .gitignores also list these explicitly +# so they're protected even if a stack is copied out of the repo. +.env +**/.env +**/.env.local +**/.env.*.local + +# Runner state +**/data/ + +# Editor / OS junk +.DS_Store +.idea/ +.vscode/ +*.swp diff --git a/README.md b/README.md new file mode 100644 index 0000000..17e106d --- /dev/null +++ b/README.md @@ -0,0 +1,21 @@ +# Project Charlie + +Centralized source of truth for the homelab. Per-app stacks live as repos under the `Charlie` org in Gitea; this repo orchestrates them. + +## Docs + +- [Topology](docs/topology.md) +- [Hosts](docs/hosts.md) +- [Identity & access](docs/identity.md) +- [Strategy](docs/strategy.md) +- [Conventions](docs/conventions.md) +- [Stack inventory](docs/inventory.md) +- [Repo plan (`Charlie/*`)](docs/repos.md) +- [Migration recipe](docs/migration.md) +- [Roadmap](docs/roadmap.md) + +## Assumptions + +- `/mnt/nas` from miquella is mounted on every app host (NFS). +- Hosts are bootstrapped per the wiki **First Time Setup** (CA trust, SSSD/LDAP, docker, otel, etc.). +- Long-form prose lives in the wiki (`https://wiki.alexandru.macocian.me`). This repo stays brief. diff --git a/docs/conventions.md b/docs/conventions.md new file mode 100644 index 0000000..e8c1d3d --- /dev/null +++ b/docs/conventions.md @@ -0,0 +1,10 @@ +# Conventions + +- One stack = one repo under `Charlie/`. Repo contains `compose.yaml`, `.env.example`, `.gitea/workflows/deploy.yaml`. +- Secrets never in Git. Use `.env` (host-local, gitignored) or Gitea Actions secrets. +- Persistent state stays under `/mnt/nas//` on miquella. Repos define bind paths, never ship data. +- Deploys are pull-based: runner on the target host runs `docker compose pull && up -d`. +- Workflows pin host: `runs-on: [self-hosted, morgott]` or `[self-hosted, melina]`. Never bare `self-hosted`. +- Image tags pinned in compose. No `:latest`. +- A runner never redeploys itself mid-job — cross-host: morgott deploys melina's runner, and vice versa. +- Secrets sourcing: TBD (candidate: vaultwarden). diff --git a/docs/hosts.md b/docs/hosts.md new file mode 100644 index 0000000..be8887b --- /dev/null +++ b/docs/hosts.md @@ -0,0 +1,14 @@ +# Hosts + +| Name | Role | Hardware | NAS | +|------|------|----------|-----| +| Radagon | TP-Link router (WAN) | — | — | +| Marika | OpenWrt router (intranet gateway) | — | — | +| Godfrey | Network switch | — | — | +| Morgott | Caddy + control-plane apps | Intel NUC | yes | +| Melina | Gitea + media stack + apps | Intel NUC, 4c/31G | yes | +| Mohg | apps | Fujitsu Esprimo | yes | +| Malenia | apps | Mac Mini 2012 | — | +| Miquella | NAS, exports `/volume1/ubuntu` → `/mnt/nas` (NFS) | Synology DSM | source | + +Bootstrap procedure: wiki → **First Time Setup**. diff --git a/docs/identity.md b/docs/identity.md new file mode 100644 index 0000000..b1da04c --- /dev/null +++ b/docs/identity.md @@ -0,0 +1,15 @@ +# Identity & access + +| Service | URL | Purpose | +|---------|-----|---------| +| Authentik | https://id.alexandru.macocian.me/ | IdP (OIDC / OAuth2 / SAML) | +| Caddy auth | https://auth.alexandru.macocian.me/ | `caddy-security`, fronts protected apps | +| Web terminal | https://terminal.alexandru.macocian.me/ | Browser SSH via ephemeral signed certs | + +## Terminal flow + +OAuth → Authentik → backend mints ephemeral SSH cert (signed by CA trusted on every host) → SSH proxied over WebSocket to JS frontend. + +CA material lives under `/mnt/nas/ca/` and is synced to hosts by `sync-ca.sh`. + +Host logins / sudo are SSSD-backed against `open-ldap`; LDAP groups drive permissions. diff --git a/docs/inventory.md b/docs/inventory.md new file mode 100644 index 0000000..2e4ad1d --- /dev/null +++ b/docs/inventory.md @@ -0,0 +1,72 @@ +# Stack inventory + +Source: `docker compose ls -a` on each host + `find /mnt/nas -name 'docker-compose.y*ml'`. Compose paths under `/mnt/nas/home/` unless noted. + +**Repo type:** +- `init` — no git history yet; `git init` in place, push to `Charlie/`. +- `rewrite` — existing git repo (typically on github.com); change `origin` to `Charlie/`. +- `upstream+overrides` — upstream repo we don't author; track `upstream`, add `origin` = `Charlie/` containing only overrides. +- `mirror` — pull-mirror of an upstream repo we use unmodified, kept under `Charlie/` for backup/availability. + +## Morgott (owner host) + +| Stack | Path | Status | Repo type | Charlie repo | +|-------|------|--------|-----------|--------------| +| caddy | `caddy/` | running | init | `Charlie/caddy` | +| authentik | `authentik/authentik/` | running | init (overrides only) | `Charlie/authentik` | +| open-ldap | `open-ldap/` | running | init | `Charlie/ldap` | +| ddclient | `ddclient/` | running | init | `Charlie/ddclient` | +| vaultwarden | `vaultwarden/` | running | init | `Charlie/vaultwarden` | +| bookstack | `bookstack/` | running | init | `Charlie/bookstack` | +| monica | `monica/monica/` | running | init | `Charlie/monica` | +| miniflux | `miniflux/` | running | init | `Charlie/miniflux` | +| mealie | `mealie/mealie/docker/` | running | upstream+overrides | `Charlie/mealie` | +| homarr | `homarr/` | running | init | `Charlie/homarr` | +| paperless-ngx | `paperless-ngx/` | running | upstream+overrides | `Charlie/paperless-ngx` | +| portainer | `portainer/` | running | init | `Charlie/portainer` | +| portainer-agent | `portainer-agent/` | running | init | `Charlie/portainer-agent` | +| otelcollector | `otelcollector/` | running | init | `Charlie/otelcollector` | +| VictoriaMetrics | `VictoriaMetrics/deployment/docker/` | running (`docker`) | upstream+overrides | `Charlie/victoriametrics` | +| watchtower | `watchtower/` | running | init | `Charlie/watchtower` (retire after pipelines) | +| Gssh | `Gssh/` | running | rewrite | `Charlie/gssh` | + +## Melina (owner host) + +| Stack | Path | Status | Repo type | Charlie repo | +|-------|------|--------|-----------|--------------| +| gitea | `gitea/` | running | init | `Charlie/gitea` | +| MediaServerCompose | `MediaServerCompose/` | running | rewrite | `Charlie/mediacompose` | +| invidious | `invidious/` | running | upstream+overrides | `Charlie/invidious` | +| searxng-docker | `searxng-docker/` | running | upstream+overrides | `Charlie/searxng` | +| planka | `planka/` | running | init | `Charlie/planka` | +| audiobookshelf | `audiobookshelf/` | running | init | `Charlie/audiobookshelf` | +| DirectoryAdmin | `DirectoryAdmin/` | running | rewrite | `Charlie/directoryadmin` | +| portainer-agent | `portainer-agent/` | running | shared with morgott | `Charlie/portainer-agent` | +| otelcollector | `otelcollector/` | running | shared with morgott | `Charlie/otelcollector` | + +## Mohg + +TBD — re-run inventory. + +## Non-stack dirs (kept on NAS, not stack repos) + +| Path | Purpose | Action | +|------|---------|--------| +| `/mnt/nas/ca/` | host CA material (ssh, ldap.lan) | leave; consider `Charlie/ca-tooling` for `sync-ca.sh` only | +| `/mnt/nas/ldap/certs/` | LDAP server certs | leave | +| `/mnt/nas/media-server/` | bulk media (downloads, movies, tv) | leave; bind target | +| `/mnt/nas/portainer/data/` | portainer state | leave | +| `/mnt/nas/home/shell/` | shared shell config (`profile.d`, `inputrc`) | `Charlie/shell` (init) | +| `/mnt/nas/home/plist/` | unclear | classify before action | +| `/mnt/nas/home/ServerManagementUtils/` | filebeat / metricbeat / logstash / orchestrator | rewrite → `Charlie/server-mgmt-utils` | + +## Mirror-only (Charlie hosts a backup mirror, no overrides) + +Useful for upstreams we may patch later or want offline copies of: +- `invidious`, `searxng-docker`, `VictoriaMetrics`, `mealie`, `paperless-ngx`, `nitter`. + +(See [strategy.md](strategy.md) on Gitea pull-mirror config.) + +## Removed since last scan + +`apache-guacamole`, `docker-dnsmasq`, `Docker-Minecraft-Paper`, `elasticsearch/`, `elk/`, `ferdium-server`, `LibreChat`, `localai`, `nitter` (still on disk, not running), `openobserve`, `reranker`, `scraper`, `step-ca`, `vrising` (game data only), `xstarbound`. `nitter` and `vrising` directories remain on disk but no longer have running stacks. diff --git a/docs/migration.md b/docs/migration.md new file mode 100644 index 0000000..9e0e5ce --- /dev/null +++ b/docs/migration.md @@ -0,0 +1,85 @@ +# Migration — NAS folder → `Charlie/` repo + +For `init`-type stacks. Run on the owner host. Container keeps running throughout. + +```bash +# 0. In Gitea UI: create empty Charlie/, private, uninitialized. + +cd /mnt/nas/home/ +ls -la && du -sh */ 2>/dev/null # spot the heavy state dirs + +git init -b main + +# .gitignore BEFORE first add — never stage runtime state +$EDITOR .gitignore # see template below + +cp .env .env.example +$EDITOR .env.example # redact every secret + +git add . +git status +git diff --cached --stat +# If state dirs slipped through: fix .gitignore, `git rm --cached -r `, repeat. + +git commit -m "Initial import from /mnt/nas/home/" +git remote add origin https://gitea.alexandru.macocian.me/Charlie/.git +git push -u origin main +``` + +## `.gitignore` template + +```gitignore +# runtime state +data/ +db-data/ +appdata/ +*-data/ +cache/ +logs/ +*.log +*.sqlite* + +# secrets +.env +.env.* +!.env.example +``` + +Tune per stack — e.g. `paperless-ngx` also has `consume/`, `export/`, `media/`, `media-bk/`. + +## `rewrite`-type + +```bash +cd /mnt/nas/home/ +git remote rename origin github 2>/dev/null || true +git remote add origin https://gitea.alexandru.macocian.me/Charlie/.git +git push -u origin main +``` + +## `upstream+overrides` + +```bash +cd /mnt/nas/home/ +git remote rename origin upstream +git remote add origin https://gitea.alexandru.macocian.me/Charlie/.git +# create our overlay branch +git checkout -b charlie +# add our files: docker-compose.override.yml, .env.example, custom Dockerfile… +git add docker-compose.override.yml .env.example +git commit -m "Charlie overrides" +git push -u origin charlie +``` + +The pipeline tracks `origin/charlie`, not upstream's default. + +## `mirror` + +No host-side action. In Gitea: New Migration → **Pull mirror** → upstream URL → schedule (e.g. daily). Repo lives at `Charlie/` and is read-only. + +## Post-migration check + +```bash +git -C /mnt/nas/home/ status # should be clean +git -C /mnt/nas/home/ remote -v # origin → gitea +sudo docker compose ps # still up +``` diff --git a/docs/repos.md b/docs/repos.md new file mode 100644 index 0000000..4b22408 --- /dev/null +++ b/docs/repos.md @@ -0,0 +1,73 @@ +# Repo plan — `Charlie/*` + +Order of carving. See [inventory.md](inventory.md) for the full list with repo types. + +## Tier 0 — bootstrap + +| Repo | Owner host | Repo type | Why first | +|------|------------|-----------|-----------| +| `project-charlie` | — | new | meta-repo (this) | + +## Tier 1 — pilot the loop (small, low blast radius) + +Goal: prove `git push → webhook → runner → docker compose up -d` end-to-end before touching anything important. + +| Repo | Owner | Source | Repo type | +|------|-------|--------|-----------| +| `ddclient` | morgott | `home/ddclient/` | init | +| `homarr` | morgott | `home/homarr/` | init | +| `vaultwarden` | morgott | `home/vaultwarden/` | init | + +## Tier 2 — own image, our code + +| Repo | Owner | Source | Repo type | +|------|-------|--------|-----------| +| `gssh` | morgott | `home/Gssh/` | rewrite | +| `directoryadmin` | melina | `home/DirectoryAdmin/` | rewrite | +| `mediacompose` | melina | `home/MediaServerCompose/` | rewrite | +| `caddy` | morgott | `home/caddy/` (custom Dockerfile) | init | + +## Tier 3 — pinned upstream images, plain init + +| Repo | Owner | Source | Repo type | +|------|-------|--------|-----------| +| `ldap` | morgott | `home/open-ldap/` | init | +| `authentik` | morgott | `home/authentik/authentik/` | init | +| `bookstack` | morgott | `home/bookstack/` | init | +| `monica` | morgott | `home/monica/monica/` | init | +| `miniflux` | morgott | `home/miniflux/` | init | +| `planka` | melina | `home/planka/` | init | +| `audiobookshelf` | melina | `home/audiobookshelf/` | init | +| `gitea` | melina | `home/gitea/` | init (deployed by *morgott* runner — never self) | + +## Tier 4 — upstream + overrides + +Two-remote model. See [strategy.md](strategy.md#upstreamoverrides). + +| Repo | Owner | Source | Upstream | +|------|-------|--------|----------| +| `invidious` | melina | `home/invidious/` | iv-org/invidious | +| `searxng` | melina | `home/searxng-docker/` | searxng/searxng-docker | +| `victoriametrics` | morgott | `home/VictoriaMetrics/deployment/docker/` | VictoriaMetrics/VictoriaMetrics | +| `mealie` | morgott | `home/mealie/mealie/docker/` | mealie-recipes/mealie | +| `paperless-ngx` | morgott | `home/paperless-ngx/` | paperless-ngx/paperless-ngx | + +## Tier 5 — long tail / utilities + +| Repo | Owner | Source | Repo type | +|------|-------|--------|-----------| +| `otelcollector` | shared | `home/otelcollector/` | init | +| `portainer` | morgott | `home/portainer/` | init | +| `portainer-agent` | shared | `home/portainer-agent/` | init | +| `server-mgmt-utils` | morgott | `home/ServerManagementUtils/` | rewrite | +| `shell` | — | `home/shell/` | init (config-only, not deployed) | +| `watchtower` | morgott | `home/watchtower/` | init then **archive after Diun + pipelines land** | + +## Mirrors (no overrides, backup only) + +Pull-mirrors under `Charlie/mirrors/` (or as plain `Charlie/-mirror`): +- `nitter`, plus any of the Tier 4 upstreams we want a guaranteed offline copy of. + +## Deferred / dormant + +`plist/` — classify before deciding. diff --git a/docs/roadmap.md b/docs/roadmap.md new file mode 100644 index 0000000..72772d8 --- /dev/null +++ b/docs/roadmap.md @@ -0,0 +1,22 @@ +# Roadmap + +## Phase 1 — CI plumbing + +- [x] Inventory `/mnt/nas/` stacks. +- [ ] Push `Charlie/project-charlie` (this repo). +- [ ] Set up Gitea pull-mirrors for relevant upstreams (see [strategy.md](strategy.md#mirror)). +- [ ] Design runner stack (host docker socket; secrets sourcing TBD — candidate: vaultwarden). +- [ ] Stand up runners on morgott + melina, register org-scope to `Charlie`. +- [ ] Pilot end-to-end on `Charlie/ddclient`: push → webhook → redeploy. + +## Phase 2 — Per-app repos + +Walk Tier 1 → Tier 5 in [repos.md](repos.md), one stack at a time. Each migration follows [migration.md](migration.md). + +## Phase 3 — Automation + +- Diun watching registries → webhook → `redeploy` workflow with `force-pull`. +- Retire watchtower. +- Drift detection: running compose vs repo compose. +- Backup verification against miquella. +- Cert/secret expiry checks (authentik, caddy, signing CA). diff --git a/docs/strategy.md b/docs/strategy.md new file mode 100644 index 0000000..55496f9 --- /dev/null +++ b/docs/strategy.md @@ -0,0 +1,63 @@ +# Strategy + +## Pattern + +Pull-based GitOps via Gitea Actions runners on the owner host. + +``` +dev box ── git push ──► Gitea (Charlie/) ── webhook ──► runner on owner host + │ + ▼ + git -C /mnt/nas/home/ pull --ff-only + docker compose pull && docker compose up -d +``` + +The working tree on the owner host **is** the deployment. `git status` on the host is the source-of-truth diff against `Charlie/`. + +## Rules + +- One owner host per stack. Only that host's runner ever touches the working tree. +- No hand-edits on the host once a stack is repo-managed. `git pull --ff-only` will refuse to clobber. +- `.gitignore` first, `git add` second. Runtime state (`data/`, `db-data/`, `cache/`, `*-data/`) never enters git. +- `.env` ignored, `.env.example` tracked. Secrets stay out. +- Image tags pinned. No `:latest`. +- Cross-host self-update: the runner that deploys `Charlie/gitea` lives on morgott; the runner that updates the morgott runner lives on melina. A runner never redeploys itself mid-job. + +## Repo types + +### `init` + +Stack is currently just a NAS folder. `git init` in place on the owner host, push to `Charlie/`. Procedure: see [migration.md](migration.md). + +### `rewrite` + +Stack is already a clone (typically of a github.com repo we own). `git remote set-url origin https://gitea.alexandru.macocian.me/Charlie/` (and optionally keep github as `github` remote). Push. + +### `upstream+overrides` + +Stack is a clone of an upstream we don't author (e.g. `invidious`, `searxng-docker`, `VictoriaMetrics`). Two remotes: + +- `upstream` — read-only tracking remote. +- `origin` — `Charlie/`, contains *only* our overrides as commits on top: `docker-compose.override.yml`, `.env.example`, custom `Dockerfile`s, optional patches. + +Updating the upstream is `git fetch upstream && git rebase upstream/` (or `merge`), then push to `origin`. Pipeline runs on push to `origin/main`. + +### `mirror` + +Pure backup. Configured via Gitea's **pull mirror** feature (Repo → Settings → Mirror). Gitea polls the upstream on a schedule and pushes nothing back. Useful for upstreams we want guaranteed offline access to even when we're not customising them. + +## Image-update flow (Phase 3) + +Diun watches registries; on a tag/digest change it POSTs to a Gitea webhook that triggers a `redeploy` workflow with `force-pull: true` on the owner host. Replaces watchtower. + +## Secrets + +Open question. Candidate: vaultwarden as the source, with a runner-side fetch step that materialises `.env` per job. Decided when we design the runner stack. + +## Why not Portainer/Komodo/Dockge + +- **Portainer Stacks-from-Git**: works, but no submodule support, opinionated about working-tree location, adds a control plane on top of git. +- **Komodo**: nice, also pull-based, also relocates working trees. Reconsider if we ever want a UI. +- **Dockge**: file mgmt only, no git wiring. + +The runner-driven pattern keeps the existing `/mnt/nas/home//` layout and adds nothing the team doesn't already understand. diff --git a/docs/topology.md b/docs/topology.md new file mode 100644 index 0000000..9c79e49 --- /dev/null +++ b/docs/topology.md @@ -0,0 +1,42 @@ +# Topology + +```mermaid +graph TD + Internet((Internet)) + Radagon{Radagon} + Devices{{Local Devices}} + Marika{Marika} + Godfrey[[Godfrey]] + Morgott[Morgott] + Melina[Melina] + Mohg[Mohg] + Malenia[Malenia] + Miquella[Miquella] + + Internet <--> Radagon + Radagon <-->|80/443| Marika + subgraph Extranet + Radagon <--> Devices + end + subgraph Intranet + Marika <--> Godfrey + Godfrey <-->|80/443| Morgott + Melina --> Godfrey + Mohg --> Godfrey + Malenia --> Godfrey + Miquella --> Godfrey + Morgott --> Melina + Morgott --> Mohg + Morgott --> Malenia + Morgott --> Miquella + Melina --> Miquella + Malenia --> Miquella + Mohg --> Miquella + end +``` + +## Ingress path + +`Internet → Radagon → Marika → Morgott:caddy → container (local or sibling host over Godfrey)`. + +All `*.alexandru.macocian.me` traffic terminates at Caddy on Morgott.