diff --git a/README.md b/README.md index 17e106d..59965ff 100644 --- a/README.md +++ b/README.md @@ -9,9 +9,11 @@ Centralized source of truth for the homelab. Per-app stacks live as repos under - [Identity & access](docs/identity.md) - [Strategy](docs/strategy.md) - [Conventions](docs/conventions.md) +- [Layout](docs/layout.md) - [Stack inventory](docs/inventory.md) - [Repo plan (`Charlie/*`)](docs/repos.md) - [Migration recipe](docs/migration.md) +- [Reorg runbook](docs/reorg.md) - [Roadmap](docs/roadmap.md) ## Assumptions diff --git a/docs/inventory.md b/docs/inventory.md index 2e4ad1d..91ebb0f 100644 --- a/docs/inventory.md +++ b/docs/inventory.md @@ -1,72 +1,79 @@ # Stack inventory -Source: `docker compose ls -a` on each host + `find /mnt/nas -name 'docker-compose.y*ml'`. Compose paths under `/mnt/nas/home/` unless noted. +Source: `docker compose ls -a` on each host + `find /mnt/nas/home -maxdepth 2 -type d` (May 3, 2026). -**Repo type:** -- `init` — no git history yet; `git init` in place, push to `Charlie/`. -- `rewrite` — existing git repo (typically on github.com); change `origin` to `Charlie/`. -- `upstream+overrides` — upstream repo we don't author; track `upstream`, add `origin` = `Charlie/` containing only overrides. -- `mirror` — pull-mirror of an upstream repo we use unmodified, kept under `Charlie/` for backup/availability. +**Repo type** — see [strategy.md](strategy.md#repo-types). ## Morgott (owner host) -| Stack | Path | Status | Repo type | Charlie repo | -|-------|------|--------|-----------|--------------| -| caddy | `caddy/` | running | init | `Charlie/caddy` | -| authentik | `authentik/authentik/` | running | init (overrides only) | `Charlie/authentik` | -| open-ldap | `open-ldap/` | running | init | `Charlie/ldap` | -| ddclient | `ddclient/` | running | init | `Charlie/ddclient` | -| vaultwarden | `vaultwarden/` | running | init | `Charlie/vaultwarden` | -| bookstack | `bookstack/` | running | init | `Charlie/bookstack` | -| monica | `monica/monica/` | running | init | `Charlie/monica` | -| miniflux | `miniflux/` | running | init | `Charlie/miniflux` | -| mealie | `mealie/mealie/docker/` | running | upstream+overrides | `Charlie/mealie` | -| homarr | `homarr/` | running | init | `Charlie/homarr` | -| paperless-ngx | `paperless-ngx/` | running | upstream+overrides | `Charlie/paperless-ngx` | -| portainer | `portainer/` | running | init | `Charlie/portainer` | -| portainer-agent | `portainer-agent/` | running | init | `Charlie/portainer-agent` | -| otelcollector | `otelcollector/` | running | init | `Charlie/otelcollector` | -| VictoriaMetrics | `VictoriaMetrics/deployment/docker/` | running (`docker`) | upstream+overrides | `Charlie/victoriametrics` | -| watchtower | `watchtower/` | running | init | `Charlie/watchtower` (retire after pipelines) | -| Gssh | `Gssh/` | running | rewrite | `Charlie/gssh` | +| Old path (`home/`) | New path (`stacks/`) | Charlie repo | Repo type | +|---|---|---|---| +| `caddy/` | `caddy/` | `caddy` | init | +| `authentik/authentik/` | `authentik/` | `authentik` | init | +| `open-ldap/` | `ldap/` | `ldap` | init | +| `ddclient/` | `ddclient/` | `ddclient` | init | +| `vaultwarden/` | `vaultwarden/` | `vaultwarden` | init | +| `bookstack/` | `bookstack/` | `bookstack` | init | +| `monica/monica/` | `monica/` | `monica` | init | +| `miniflux/` | `miniflux/` | `miniflux` | init | +| `mealie/mealie/docker/` | `mealie/` | `mealie` | init | +| `homarr/` | `homarr/` | `homarr` | init | +| `paperless-ngx/` | `paperless-ngx/` | `paperless-ngx` | init | +| `portainer/` | `portainer/` | `portainer` | init | +| `portainer-agent/` | `portainer-agent/` | `portainer-agent` | init | +| `otelcollector/` | `otelcollector/` | `otelcollector` | init | +| `VictoriaMetrics/deployment/docker/` | `victoriametrics/` | `victoriametrics` | init | +| `watchtower/` | `watchtower/` | `watchtower` | init (retire after Diun) | +| `Gssh/` | `gssh/` | `gssh` | rewrite | +| `ServerManagementUtils/` | `server-mgmt-utils/` | `server-mgmt-utils` | rewrite | ## Melina (owner host) -| Stack | Path | Status | Repo type | Charlie repo | -|-------|------|--------|-----------|--------------| -| gitea | `gitea/` | running | init | `Charlie/gitea` | -| MediaServerCompose | `MediaServerCompose/` | running | rewrite | `Charlie/mediacompose` | -| invidious | `invidious/` | running | upstream+overrides | `Charlie/invidious` | -| searxng-docker | `searxng-docker/` | running | upstream+overrides | `Charlie/searxng` | -| planka | `planka/` | running | init | `Charlie/planka` | -| audiobookshelf | `audiobookshelf/` | running | init | `Charlie/audiobookshelf` | -| DirectoryAdmin | `DirectoryAdmin/` | running | rewrite | `Charlie/directoryadmin` | -| portainer-agent | `portainer-agent/` | running | shared with morgott | `Charlie/portainer-agent` | -| otelcollector | `otelcollector/` | running | shared with morgott | `Charlie/otelcollector` | +| Old path (`home/`) | New path (`stacks/`) | Charlie repo | Repo type | +|---|---|---|---| +| `gitea/` | `gitea/` | `gitea` | init | +| `MediaServerCompose/` | `mediacompose/` | `mediacompose` | rewrite | +| `invidious/` | `invidious/` | `invidious` | init (was upstream-cloned; redo as our compose) | +| `searxng-docker/` | `searxng/` | `searxng` | init (same) | +| `planka/` | `planka/` | `planka` | init | +| `audiobookshelf/` | `audiobookshelf/` | `audiobookshelf` | init | +| `DirectoryAdmin/` | `directoryadmin/` | `directoryadmin` | rewrite | ## Mohg TBD — re-run inventory. -## Non-stack dirs (kept on NAS, not stack repos) +## Config-only (not deployed, but version-controlled) -| Path | Purpose | Action | -|------|---------|--------| -| `/mnt/nas/ca/` | host CA material (ssh, ldap.lan) | leave; consider `Charlie/ca-tooling` for `sync-ca.sh` only | -| `/mnt/nas/ldap/certs/` | LDAP server certs | leave | -| `/mnt/nas/media-server/` | bulk media (downloads, movies, tv) | leave; bind target | -| `/mnt/nas/portainer/data/` | portainer state | leave | -| `/mnt/nas/home/shell/` | shared shell config (`profile.d`, `inputrc`) | `Charlie/shell` (init) | -| `/mnt/nas/home/plist/` | unclear | classify before action | -| `/mnt/nas/home/ServerManagementUtils/` | filebeat / metricbeat / logstash / orchestrator | rewrite → `Charlie/server-mgmt-utils` | +| Old path | New path | Charlie repo | +|---|---|---| +| `home/shell/` | `stacks/shell/` | `shell` | +| `home/first-time-setup.md` | `stacks/first-time-setup/` | `first-time-setup` | +| `home/plist/` | TBD — classify | — | -## Mirror-only (Charlie hosts a backup mirror, no overrides) +## Mirrors -Useful for upstreams we may patch later or want offline copies of: -- `invidious`, `searxng-docker`, `VictoriaMetrics`, `mealie`, `paperless-ngx`, `nitter`. +Pull-mirror only (Gitea config; no host action). Naming: `Charlie/-mirror`. -(See [strategy.md](strategy.md) on Gitea pull-mirror config.) +| Mirror | Upstream | +|---|---| +| `invidious-mirror` | https://github.com/iv-org/invidious | +| `searxng-docker-mirror` | https://github.com/searxng/searxng-docker | +| `victoriametrics-mirror` | https://github.com/VictoriaMetrics/VictoriaMetrics | +| `mealie-mirror` | https://github.com/mealie-recipes/mealie | +| `paperless-ngx-mirror` | https://github.com/paperless-ngx/paperless-ngx | +| `nitter-mirror` | https://github.com/zedeus/nitter | +| `caddy-mirror` | https://github.com/caddyserver/caddy | +| `authentik-mirror` | https://github.com/goauthentik/authentik | +| `gitea-mirror` | https://github.com/go-gitea/gitea | +| `act-runner-mirror` | https://gitea.com/gitea/act_runner | -## Removed since last scan +## Non-stack dirs -`apache-guacamole`, `docker-dnsmasq`, `Docker-Minecraft-Paper`, `elasticsearch/`, `elk/`, `ferdium-server`, `LibreChat`, `localai`, `nitter` (still on disk, not running), `openobserve`, `reranker`, `scraper`, `step-ca`, `vrising` (game data only), `xstarbound`. `nitter` and `vrising` directories remain on disk but no longer have running stacks. +| Path | Action | +|---|---| +| `/mnt/nas/ca/` | leave; out of scope | +| `/mnt/nas/ldap/` → `/mnt/nas/ldap-certs/` | rename during reorg | +| `/mnt/nas/media-server/` → `/mnt/nas/media/` | rename when `mediacompose` migrates | +| `/mnt/nas/portainer/data/` → `/mnt/nas/data/portainer/` | move when `portainer` migrates | +| `/mnt/nas/home/` | delete after every stack migrated | diff --git a/docs/layout.md b/docs/layout.md new file mode 100644 index 0000000..5b1d824 --- /dev/null +++ b/docs/layout.md @@ -0,0 +1,80 @@ +# Layout + +Canonical `/mnt/nas/` shape after the reorg. + +``` +/mnt/nas/ +├── stacks/ ← per-stack git working trees. Small, all-in-git. +│ ├── caddy/ +│ ├── ldap/ +│ ├── ... +├── data/ ← per-stack runtime state. Bind-mounted into containers. +│ ├── caddy/ +│ ├── ldap/ +│ ├── mediacompose/ +│ │ ├── jellyfin/ +│ │ ├── sonarr/ +│ │ └── ... +│ ├── ... +├── media/ ← bulk media (renamed from media-server/) +│ ├── downloads/ +│ ├── movies/ +│ └── tv/ +├── ca/ ← host CA material; out of scope for the reorg +├── ldap-certs/ ← LDAP server certs (renamed from ldap/); out of scope +└── home/ ← legacy; deleted at end of reorg +``` + +## Naming + +- All directories under `stacks/` and `data/` are **kebab-case, lowercase**. +- Repo names under `Charlie/` match the `stacks/` directory name 1:1. + +| Old | New | +|-----|-----| +| `home/Gssh/` | `stacks/gssh/` | +| `home/MediaServerCompose/` | `stacks/mediacompose/` | +| `home/DirectoryAdmin/` | `stacks/directoryadmin/` | +| `home/open-ldap/` | `stacks/ldap/` | +| `home/VictoriaMetrics/deployment/docker/` | `stacks/victoriametrics/` | +| `home/mealie/mealie/docker/` | `stacks/mealie/` | +| `home/authentik/authentik/` | `stacks/authentik/` | +| `home/searxng-docker/` | `stacks/searxng/` | +| `home/paperless-ngx/` | `stacks/paperless-ngx/` | +| `home/ServerManagementUtils/` | `stacks/server-mgmt-utils/` | +| `home/portainer-agent/` | `stacks/portainer-agent/` | + +## Bind-mount convention + +Every compose file binds **only** `/mnt/nas/data//...` for runtime state. No bind into `./` (which is the git working tree). + +```yaml +# stacks/ldap/docker-compose.yml +services: + openldap: + volumes: + - /mnt/nas/data/ldap/db:/var/lib/ldap + - /mnt/nas/data/ldap/config:/etc/ldap/slapd.d +``` + +Small read-only configs that *should* be version-controlled (Caddyfile, openldap bootstrap LDIFs, etc.) live inside the `stacks//` working tree and are bind-mounted as files: + +```yaml +volumes: + - ./Caddyfile:/etc/caddy/Caddyfile:ro +``` + +## Out-of-scope dirs (leave alone for now) + +- `/mnt/nas/ca/` — host CA material. Considered separately later (`Charlie/ca-tooling`). +- `/mnt/nas/ldap-certs/` — LDAP server certs (after rename from `ldap/`). Same. +- `/mnt/nas/#recycle` — Synology recycle bin. + +## Stacks-only-as-config (no container) + +Some `stacks//` may not deploy a container — they're just version-controlled config that belongs to the homelab: + +- `stacks/shell/` — shared `profile.d`, `inputrc`. Cloned by hosts to `/usr/local/share/charlie-shell/`, sourced by `/etc/profile`. +- `stacks/first-time-setup/` — bootstrap docs / scripts (currently `home/first-time-setup.md`). + +These have no `docker-compose.yml` and no entry in the runner deploy list. diff --git a/docs/migration.md b/docs/migration.md index 9e0e5ce..ca71bb2 100644 --- a/docs/migration.md +++ b/docs/migration.md @@ -1,85 +1,119 @@ -# Migration — NAS folder → `Charlie/` repo +# Migration -For `init`-type stacks. Run on the owner host. Container keeps running throughout. +Per-stack recipe to move `home//` → `stacks//` + `data//` and create `Charlie/`. + +## Pre-conditions + +- Owner host SSH session as admin. +- Container is **stopped**: `cd /mnt/nas/home/ && sudo docker compose down`. +- Empty Gitea repo created (Charlie org, private, uninitialized). + +## `init` — no existing git history ```bash -# 0. In Gitea UI: create empty Charlie/, private, uninitialized. +OLD=/mnt/nas/home/ +NEW=/mnt/nas/stacks/ +DATA=/mnt/nas/data/ -cd /mnt/nas/home/ -ls -la && du -sh */ 2>/dev/null # spot the heavy state dirs +# 1. Make destinations +sudo mkdir -p "$NEW" "$DATA" +sudo chown -R alex:alex "$NEW" -git init -b main +# 2. Move runtime state OUT of the old dir, into data/ +# (Identify state dirs first: data/, db-data/, appdata/, cache/, …) +sudo mv "$OLD"/data "$DATA"/ 2>/dev/null || true +sudo mv "$OLD"/db-data "$DATA"/ 2>/dev/null || true +sudo mv "$OLD"/ "$DATA"/ 2>/dev/null || true -# .gitignore BEFORE first add — never stage runtime state -$EDITOR .gitignore # see template below +# 3. Copy the deployment files into the new working tree +cp "$OLD"/docker-compose.yml "$NEW"/ +cp "$OLD"/docker-compose.override.yml "$NEW"/ 2>/dev/null || true +cp "$OLD"/.env "$NEW"/ 2>/dev/null || true +cp -r "$OLD"/ "$NEW"/ 2>/dev/null || true -cp .env .env.example -$EDITOR .env.example # redact every secret +cd "$NEW" -git add . -git status -git diff --cached --stat -# If state dirs slipped through: fix .gitignore, `git rm --cached -r `, repeat. +# 4. Rewrite the compose so binds reference /mnt/nas/data//... +$EDITOR docker-compose.yml docker-compose.override.yml -git commit -m "Initial import from /mnt/nas/home/" -git remote add origin https://gitea.alexandru.macocian.me/Charlie/.git -git push -u origin main -``` - -## `.gitignore` template - -```gitignore -# runtime state -data/ -db-data/ -appdata/ -*-data/ -cache/ -logs/ -*.log -*.sqlite* - -# secrets +# 5. .gitignore + .env.example +cat > .gitignore <<'EOF' .env .env.* !.env.example -``` +EOF +cp .env .env.example +$EDITOR .env.example # redact secrets -Tune per stack — e.g. `paperless-ngx` also has `consume/`, `export/`, `media/`, `media-bk/`. - -## `rewrite`-type - -```bash -cd /mnt/nas/home/ -git remote rename origin github 2>/dev/null || true -git remote add origin https://gitea.alexandru.macocian.me/Charlie/.git +# 6. Init + first commit +git init -b main +git add . +git status +git diff --cached --stat +git commit -m "Initial import" +git remote add origin https://gitea.alexandru.macocian.me/Charlie/.git git push -u origin main + +# 7. Bring it back up FROM THE NEW LOCATION +sudo docker compose up -d +sudo docker compose ps +sudo docker compose logs -f --tail=50 + +# 8. Once verified, retire the old dir (do NOT delete data inside it — already moved) +sudo mv "$OLD" "$OLD".retired +# Final removal happens at end of full reorg. ``` -## `upstream+overrides` +## `rewrite` — already a git repo + +For stacks that are clones of github.com repos we own (e.g. `Gssh`, `MediaServerCompose`, `DirectoryAdmin`). ```bash -cd /mnt/nas/home/ -git remote rename origin upstream -git remote add origin https://gitea.alexandru.macocian.me/Charlie/.git -# create our overlay branch -git checkout -b charlie -# add our files: docker-compose.override.yml, .env.example, custom Dockerfile… -git add docker-compose.override.yml .env.example -git commit -m "Charlie overrides" -git push -u origin charlie +OLD=/mnt/nas/home/ +NEW=/mnt/nas/stacks/ +DATA=/mnt/nas/data/ + +sudo mkdir -p "$DATA" +sudo chown -R alex:alex "$DATA" + +# 1. Move state out +sudo mv "$OLD"/ "$DATA"/ 2>/dev/null || true + +# 2. Move the whole git repo to the new location (preserves .git) +sudo mv "$OLD" "$NEW" +sudo chown -R alex:alex "$NEW" + +cd "$NEW" + +# 3. Repoint origin to Charlie (keep github as a fallback remote) +git remote rename origin github +git remote add origin https://gitea.alexandru.macocian.me/Charlie/.git + +# 4. Update compose binds to /mnt/nas/data//... +$EDITOR docker-compose.yml docker-compose.override.yml + +# 5. .gitignore additions for the data dirs we just moved out +$EDITOR .gitignore + +# 6. Commit, push to Charlie +git add . +git commit -m "Move bind paths to /mnt/nas/data/" +git push -u origin main + +# 7. Bring back up +sudo docker compose up -d ``` -The pipeline tracks `origin/charlie`, not upstream's default. +## `mirror` — pull-mirror only, no host action -## `mirror` +In Gitea: New Migration → Pull mirror → upstream URL → name `Charlie/-mirror` → schedule `24h0m0s`. -No host-side action. In Gitea: New Migration → **Pull mirror** → upstream URL → schedule (e.g. daily). Repo lives at `Charlie/` and is read-only. +No filesystem changes on any host. -## Post-migration check +## Post-stack checklist -```bash -git -C /mnt/nas/home/ status # should be clean -git -C /mnt/nas/home/ remote -v # origin → gitea -sudo docker compose ps # still up -``` +- [ ] Container is up at the new location. +- [ ] `docker inspect --format '{{ range .Mounts }}{{ .Source }}{{"\n"}}{{ end }}'` confirms binds point at `/mnt/nas/data//...`. +- [ ] `git -C /mnt/nas/stacks/ status` is clean. +- [ ] Old `home/.retired/` exists and is empty of state (only deployment leftovers). +- [ ] Service reachable via Caddy as before. diff --git a/docs/reorg.md b/docs/reorg.md new file mode 100644 index 0000000..273500a --- /dev/null +++ b/docs/reorg.md @@ -0,0 +1,78 @@ +# Reorg runbook + +The one-time migration from `/mnt/nas/home/` to `/mnt/nas/stacks/` + `/mnt/nas/data/`. + +## Phase 0 — preconditions + +- [ ] Admin SSH on morgott + melina + mohg. +- [ ] `Charlie/project-charlie` pushed (done). +- [ ] Decided owner host per stack (see [repos.md](repos.md)). +- [ ] Snapshot taken on miquella (Synology snapshot of `volume1/ubuntu`). + +## Phase 1 — scaffolding (no downtime) + +```bash +# On any one host (NAS is shared). +sudo mkdir -p /mnt/nas/stacks /mnt/nas/data +sudo chown alex:alex /mnt/nas/stacks /mnt/nas/data +``` + +Optional rename now (still no downtime — these are bulk dirs nothing reads except by full path): + +```bash +# Only if you commit to the new names. Update bind paths in compose files BEFORE renaming. +sudo mv /mnt/nas/media-server /mnt/nas/media # if no compose binds it yet at the old path +sudo mv /mnt/nas/ldap /mnt/nas/ldap-certs +``` + +(Actually: don't rename `media-server` until `mediacompose` is migrated — its compose binds the old path. Leave both renames for the per-stack pass.) + +## Phase 2 — create empty Charlie repos in Gitea + +Per [repos.md](repos.md), create empty repos: + +- All `init` and `rewrite` rows. +- All `mirror` rows (configured as pull-mirrors, see [migration.md](migration.md#mirror)). + +No host changes yet. + +## Phase 3 — per-stack migration loop + +Order: Tier 1 → 5 from [repos.md](repos.md). For each stack on its owner host: + +1. `cd /mnt/nas/home/ && sudo docker compose down` +2. Run [migration.md](migration.md) recipe (`init` or `rewrite`). +3. Verify checklist at the bottom of [migration.md](migration.md). +4. ✅ on to next. + +Pilot: do `ddclient` first end-to-end before touching anything else. If it works, the pattern works. + +## Phase 4 — sweep + +Once **every** stack is migrated and verified: + +```bash +# On miquella or any NFS client: +ls /mnt/nas/home/ # expect: only *.retired/ +sudo rm -rf /mnt/nas/home/*.retired +sudo rmdir /mnt/nas/home + +# Final layout check +ls /mnt/nas/ # expect: ca, data, ldap-certs, media, stacks, #recycle +``` + +## Phase 5 — runner stack + +Now that everything is git-tracked, design + deploy `Charlie/runners` (separate effort). Pilot redeploy via webhook on `Charlie/ddclient`. + +## Rollback strategy + +Any stack failing to come up at the new location: + +1. `sudo docker compose down` in `/mnt/nas/stacks//`. +2. `sudo mv /mnt/nas/home/.retired /mnt/nas/home/` (state was moved, not copied — restore is just a path rename of the directory shell). +3. Move state back: `sudo mv /mnt/nas/data//* /mnt/nas/home//`. +4. `cd /mnt/nas/home/ && sudo docker compose up -d`. +5. Investigate, retry. + +Because state was **moved** (not copied), there's only ever one source of truth. No risk of two diverging copies. diff --git a/docs/repos.md b/docs/repos.md index 4b22408..7eaf241 100644 --- a/docs/repos.md +++ b/docs/repos.md @@ -1,73 +1,74 @@ # Repo plan — `Charlie/*` -Order of carving. See [inventory.md](inventory.md) for the full list with repo types. +Carving order. See [inventory.md](inventory.md) for the full list. Migration recipe: [migration.md](migration.md). Sequenced runbook: [reorg.md](reorg.md). ## Tier 0 — bootstrap -| Repo | Owner host | Repo type | Why first | -|------|------------|-----------|-----------| -| `project-charlie` | — | new | meta-repo (this) | +| Repo | Owner | Status | +|---|---|---| +| `project-charlie` | — | done | -## Tier 1 — pilot the loop (small, low blast radius) +## Tier 1 — pilot -Goal: prove `git push → webhook → runner → docker compose up -d` end-to-end before touching anything important. +Prove the loop end-to-end. Smallest blast radius. -| Repo | Owner | Source | Repo type | -|------|-------|--------|-----------| +| Repo | Owner | Source | Type | +|---|---|---|---| | `ddclient` | morgott | `home/ddclient/` | init | | `homarr` | morgott | `home/homarr/` | init | | `vaultwarden` | morgott | `home/vaultwarden/` | init | -## Tier 2 — own image, our code +## Tier 2 — our code -| Repo | Owner | Source | Repo type | -|------|-------|--------|-----------| +| Repo | Owner | Source | Type | +|---|---|---|---| | `gssh` | morgott | `home/Gssh/` | rewrite | | `directoryadmin` | melina | `home/DirectoryAdmin/` | rewrite | | `mediacompose` | melina | `home/MediaServerCompose/` | rewrite | +| `server-mgmt-utils` | morgott | `home/ServerManagementUtils/` | rewrite | | `caddy` | morgott | `home/caddy/` (custom Dockerfile) | init | ## Tier 3 — pinned upstream images, plain init -| Repo | Owner | Source | Repo type | -|------|-------|--------|-----------| -| `ldap` | morgott | `home/open-ldap/` | init | -| `authentik` | morgott | `home/authentik/authentik/` | init | -| `bookstack` | morgott | `home/bookstack/` | init | -| `monica` | morgott | `home/monica/monica/` | init | -| `miniflux` | morgott | `home/miniflux/` | init | -| `planka` | melina | `home/planka/` | init | -| `audiobookshelf` | melina | `home/audiobookshelf/` | init | -| `gitea` | melina | `home/gitea/` | init (deployed by *morgott* runner — never self) | +| Repo | Owner | Source | +|---|---|---| +| `ldap` | morgott | `home/open-ldap/` | +| `authentik` | morgott | `home/authentik/authentik/` | +| `bookstack` | morgott | `home/bookstack/` | +| `monica` | morgott | `home/monica/monica/` | +| `miniflux` | morgott | `home/miniflux/` | +| `planka` | melina | `home/planka/` | +| `audiobookshelf` | melina | `home/audiobookshelf/` | +| `gitea` | melina | `home/gitea/` (deployed by morgott runner — never self) | -## Tier 4 — upstream + overrides +## Tier 4 — was upstream-cloned, now plain init -Two-remote model. See [strategy.md](strategy.md#upstreamoverrides). +These previously lived as upstream-clones with `docker-compose.override.yml`. We're collapsing override → canonical compose. **Hand-merge step required** in [migration.md](migration.md). -| Repo | Owner | Source | Upstream | -|------|-------|--------|----------| -| `invidious` | melina | `home/invidious/` | iv-org/invidious | -| `searxng` | melina | `home/searxng-docker/` | searxng/searxng-docker | -| `victoriametrics` | morgott | `home/VictoriaMetrics/deployment/docker/` | VictoriaMetrics/VictoriaMetrics | -| `mealie` | morgott | `home/mealie/mealie/docker/` | mealie-recipes/mealie | -| `paperless-ngx` | morgott | `home/paperless-ngx/` | paperless-ngx/paperless-ngx | +| Repo | Owner | Source | Image source | +|---|---|---|---| +| `invidious` | melina | `home/invidious/` | `quay.io/invidious/invidious` | +| `searxng` | melina | `home/searxng-docker/` | `searxng/searxng` | +| `victoriametrics` | morgott | `home/VictoriaMetrics/deployment/docker/` | `victoriametrics/victoria-metrics` | +| `mealie` | morgott | `home/mealie/mealie/docker/` | `ghcr.io/mealie-recipes/mealie` | +| `paperless-ngx` | morgott | `home/paperless-ngx/` | `ghcr.io/paperless-ngx/paperless-ngx` | ## Tier 5 — long tail / utilities -| Repo | Owner | Source | Repo type | -|------|-------|--------|-----------| +| Repo | Owner | Source | Type | +|---|---|---|---| | `otelcollector` | shared | `home/otelcollector/` | init | | `portainer` | morgott | `home/portainer/` | init | | `portainer-agent` | shared | `home/portainer-agent/` | init | -| `server-mgmt-utils` | morgott | `home/ServerManagementUtils/` | rewrite | -| `shell` | — | `home/shell/` | init (config-only, not deployed) | -| `watchtower` | morgott | `home/watchtower/` | init then **archive after Diun + pipelines land** | +| `shell` | — (config-only) | `home/shell/` | init | +| `first-time-setup` | — (docs-only) | `home/first-time-setup.md` | init | +| `watchtower` | morgott | `home/watchtower/` | init then archive | -## Mirrors (no overrides, backup only) +## Mirrors -Pull-mirrors under `Charlie/mirrors/` (or as plain `Charlie/-mirror`): -- `nitter`, plus any of the Tier 4 upstreams we want a guaranteed offline copy of. +See [inventory.md](inventory.md#mirrors). Configured in Gitea UI; no host action. -## Deferred / dormant +## Deferred `plist/` — classify before deciding. +`nitter/`, `vrising/` — on disk, not running. Leave as-is or archive. diff --git a/docs/roadmap.md b/docs/roadmap.md index 72772d8..3c51e88 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1,19 +1,19 @@ # Roadmap -## Phase 1 — CI plumbing +## Phase 1 — reorg + CI plumbing - [x] Inventory `/mnt/nas/` stacks. -- [ ] Push `Charlie/project-charlie` (this repo). -- [ ] Set up Gitea pull-mirrors for relevant upstreams (see [strategy.md](strategy.md#mirror)). -- [ ] Design runner stack (host docker socket; secrets sourcing TBD — candidate: vaultwarden). -- [ ] Stand up runners on morgott + melina, register org-scope to `Charlie`. -- [ ] Pilot end-to-end on `Charlie/ddclient`: push → webhook → redeploy. +- [x] Push `Charlie/project-charlie`. +- [ ] Snapshot `volume1/ubuntu` on miquella. +- [ ] Create empty `Charlie/` repos per [repos.md](repos.md). +- [ ] Create Gitea pull-mirrors per [inventory.md](inventory.md#mirrors). +- [ ] Walk [reorg.md](reorg.md) Tier 1 (pilot: `ddclient`). +- [ ] Walk Tiers 2–5. +- [ ] Delete `/mnt/nas/home/`. +- [ ] Design + deploy runner stack (`Charlie/runners`). +- [ ] First end-to-end pipeline: push to `Charlie/ddclient` → webhook → redeploy on morgott. -## Phase 2 — Per-app repos - -Walk Tier 1 → Tier 5 in [repos.md](repos.md), one stack at a time. Each migration follows [migration.md](migration.md). - -## Phase 3 — Automation +## Phase 2 — Automation - Diun watching registries → webhook → `redeploy` workflow with `force-pull`. - Retire watchtower. diff --git a/docs/strategy.md b/docs/strategy.md index 55496f9..f368f54 100644 --- a/docs/strategy.md +++ b/docs/strategy.md @@ -8,43 +8,46 @@ Pull-based GitOps via Gitea Actions runners on the owner host. dev box ── git push ──► Gitea (Charlie/) ── webhook ──► runner on owner host │ ▼ - git -C /mnt/nas/home/ pull --ff-only + git -C /mnt/nas/stacks/ pull --ff-only docker compose pull && docker compose up -d ``` -The working tree on the owner host **is** the deployment. `git status` on the host is the source-of-truth diff against `Charlie/`. +The git working tree at `/mnt/nas/stacks//` **is** the deployment. Runtime state lives separately under `/mnt/nas/data//` (see [layout.md](layout.md)). ## Rules - One owner host per stack. Only that host's runner ever touches the working tree. - No hand-edits on the host once a stack is repo-managed. `git pull --ff-only` will refuse to clobber. -- `.gitignore` first, `git add` second. Runtime state (`data/`, `db-data/`, `cache/`, `*-data/`) never enters git. +- Runtime state never enters git. Compose files bind into `/mnt/nas/data//`, not into the working tree. - `.env` ignored, `.env.example` tracked. Secrets stay out. - Image tags pinned. No `:latest`. -- Cross-host self-update: the runner that deploys `Charlie/gitea` lives on morgott; the runner that updates the morgott runner lives on melina. A runner never redeploys itself mid-job. +- Cross-host self-update: the runner that deploys `Charlie/gitea` lives on morgott; the runner that updates morgott's runner lives on melina. A runner never redeploys itself mid-job. ## Repo types +Three. (We dropped `upstream+overrides` — see [Why no upstream cloning](#why-no-upstream-cloning).) + ### `init` -Stack is currently just a NAS folder. `git init` in place on the owner host, push to `Charlie/`. Procedure: see [migration.md](migration.md). +Stack is currently just a NAS folder. `git init` in the new `/mnt/nas/stacks//`, push to `Charlie/`. Recipe: [migration.md](migration.md#init). ### `rewrite` -Stack is already a clone (typically of a github.com repo we own). `git remote set-url origin https://gitea.alexandru.macocian.me/Charlie/` (and optionally keep github as `github` remote). Push. - -### `upstream+overrides` - -Stack is a clone of an upstream we don't author (e.g. `invidious`, `searxng-docker`, `VictoriaMetrics`). Two remotes: - -- `upstream` — read-only tracking remote. -- `origin` — `Charlie/`, contains *only* our overrides as commits on top: `docker-compose.override.yml`, `.env.example`, custom `Dockerfile`s, optional patches. - -Updating the upstream is `git fetch upstream && git rebase upstream/` (or `merge`), then push to `origin`. Pipeline runs on push to `origin/main`. +Code we author, currently a clone of a github.com repo. Move to `/mnt/nas/stacks//`, `git remote set-url origin` to Charlie. Recipe: [migration.md](migration.md#rewrite). ### `mirror` -Pure backup. Configured via Gitea's **pull mirror** feature (Repo → Settings → Mirror). Gitea polls the upstream on a schedule and pushes nothing back. Useful for upstreams we want guaranteed offline access to even when we're not customising them. +Pull-mirror of an upstream we don't customise (or want as offline backup). Configured via Gitea's pull-mirror feature. Read-only, never deployed from. Naming: `Charlie/-mirror`. + +## Why no upstream cloning + +Previously we considered cloning upstream's source on the host and layering a `docker-compose.override.yml`. Dropped because: + +- For every "upstream image" stack (`invidious`, `searxng`, `victoriametrics`, `mealie`, `paperless-ngx`), we only ever ran the published image. Their source tree is dead weight on the host. +- Override-on-template means the upstream's compose can change under us silently. +- Two-remote / rebase / force-push dance was the price for dubious value. + +Instead: our `docker-compose.yml` references the published image directly. It is the only compose. Upstream's compose is a *reference document* we read in their mirror repo when we want to see what changed. ## Image-update flow (Phase 3) @@ -60,4 +63,4 @@ Open question. Candidate: vaultwarden as the source, with a runner-side fetch st - **Komodo**: nice, also pull-based, also relocates working trees. Reconsider if we ever want a UI. - **Dockge**: file mgmt only, no git wiring. -The runner-driven pattern keeps the existing `/mnt/nas/home//` layout and adds nothing the team doesn't already understand. +The runner-driven pattern keeps us in plain `git` + `docker compose` and adds nothing the team doesn't already understand.