Charlie/project-charlie
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rework documentation

Browse Source
This commit is contained in:
Alexandru Victor Macocian
2026-06-13 17:26:49 +02:00
parent 71913c5bc9
commit 17b42e0f73
16 changed files with 439 additions and 2619 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitea/README.md
+18 -59
View File
@@ -1,64 +1,23 @@
# Reusable Gitea Actions
Shared automation consumed by every `Charlie/<stack>` repo. Full caller guide,
manual-fallback recipes, and host prerequisites: [docs/automation.md](../docs/automation.md).
Shared automation consumed by Charlie stacks and image-source repos.
## Contracts
| Item | Online source | Used by |
| --------------------- | --------------------------------------------------------------------------------------------------------------- | -------------------------- |
| Deploy stack workflow | <https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/.gitea/workflows/deploy-stack.yml> | `Charlie/<stack>` repos |
| Build image workflow | <https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/.gitea/workflows/build-image.yml> | `amacocian/<source>` repos |
| SSH open action | <https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/.gitea/actions/charlie-ssh-open> | reusable workflows |
| SSH target action | <https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/.gitea/actions/charlie-ssh-target> | reusable workflows |
| SSH close action | <https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/.gitea/actions/charlie-ssh-close> | reusable workflows |
### Reusable workflows
| Contract | State |
| --------------------- | -------------------------------------------------------------------- |
| Cross-repo references | Absolute Gitea URLs with PAT auth. |
| Caller pin | `@main` until the workflow/action API is tagged. |
| Runner model | Any runner can pick up jobs; target host is selected by SSH. |
| Runner exception | `Charlie/runners` cross-deploys itself with a hand-written workflow. |
| Path | Purpose |
|---|---|
| [`workflows/deploy-stack.yml`](workflows/deploy-stack.yml) | `git pull` + (optional) SOPS decrypt + per-host compose resolution + `docker compose up -d`. Replaces the old per-repo `deploy.sh`. |
| [`workflows/build-image.yml`](workflows/build-image.yml) | Build + push a Docker image from a source repo to the Gitea container registry. Tags `latest` + UTC timestamp (`YYYY-MM-DDTHHMMSSZ`). Called from `amacocian/<name>` repos. |
### Composite actions
| Path | Purpose |
|---|---|
| [`actions/sops-decrypt`](actions/sops-decrypt/action.yml) | Decrypt one or more SOPS files using the host's age key at `/etc/charlie/age.key`. |
## Caller skeleton
Drop this into `Charlie/<stack>/.gitea/workflows/deploy.yaml`:
```yaml
name: Deploy
on:
push:
branches: [main]
workflow_dispatch:
jobs:
deploy:
uses: Charlie/project-charlie/.gitea/workflows/deploy-stack.yml@main
with:
host: morgott
decrypt: |
secrets/env.sops.yaml -> .env
```
For a multi-host stack, add a matrix:
```yaml
jobs:
deploy:
strategy:
matrix:
host: [morgott, melina, mohg, miquella]
uses: Charlie/project-charlie/.gitea/workflows/deploy-stack.yml@main
with:
host: ${{ matrix.host }}
decrypt: |
secrets/${{ matrix.host }}.env.sops.yaml -> envs/${{ matrix.host }}.env
```
## Versioning
Pin to `@main` for now. We'll cut tags (`@v1`, …) once the contract stops moving.
Breaking changes to inputs require a major bump.
## See also
- [docs/automation.md](../docs/automation.md) — full caller guide, manual-fallback recipes, host prerequisites.
- [docs/vulnerability-scanning.md](../docs/vulnerability-scanning.md) — daily CVE sweep (trivy-server + trivy-runner) feeding Grafana via the same otelcol pipeline.
See
[automation](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/automation.md)
and
[runners](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/runners.md).
README.md
+27 -20
View File
@@ -1,27 +1,34 @@
# Project Charlie
Centralized source of truth for the homelab. Per-app stacks live as repos under the `Charlie` org in Gitea; this repo orchestrates them.
Current source of truth for the Charlie homelab deployment model, fleet
inventory, and agent-facing next work.
| Area | Location |
| ------------------- | ------------------------------------------------------------- |
| Project repo | <https://gitea.alexandru.macocian.me/Charlie/project-charlie> |
| Stack repos | `https://gitea.alexandru.macocian.me/Charlie/<stack>` |
| Source/image repos | `https://gitea.alexandru.macocian.me/amacocian/<repo>` |
| Stack working trees | `/mnt/nas/stacks/<stack>` |
| Runtime data | `/mnt/nas/data/<stack>` |
| Operator tooling | <https://gitea.alexandru.macocian.me/amacocian/sherlock> |
## Docs
- [Topology](docs/topology.md)
- [Hosts](docs/hosts.md)
- [Identity & access](docs/identity.md)
- [Strategy](docs/strategy.md)
- [Conventions](docs/conventions.md)
- [Layout](docs/layout.md)
- [Stack inventory](docs/inventory.md)
- [Host onboarding](docs/onboarding.md)
- [Secrets](docs/secrets.md)
- [Automation — reusable workflows + actions](docs/automation.md)
- [Runners (`Charlie/runners` design)](docs/runners.md)
- [Cert issuer](docs/cert-issuer.md)
- [Vulnerability scanning](docs/vulnerability-scanning.md)
- [Operator tooling — `Charlie/sherlock`](docs/inventory.md#operator-tooling)
- [Roadmap](docs/roadmap.md)
| Topic | Doc |
| ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Architecture | [strategy](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/strategy.md), [topology](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/topology.md), [layout](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/layout.md) |
| Fleet | [hosts](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/hosts.md), [inventory](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/inventory.md), [onboarding](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/onboarding.md) |
| Access | [identity](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/identity.md), [secrets](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/secrets.md), [cert issuer](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/cert-issuer.md) |
| Automation | [automation](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/automation.md), [runners](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/runners.md), [vulnerability scanning](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/vulnerability-scanning.md) |
| Standards | [conventions](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/conventions.md), [next work](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/roadmap.md) |
## Assumptions
## Current state
- `/mnt/nas` from miquella is mounted on every app host (NFS).
- Hosts are bootstrapped per [onboarding.md](docs/onboarding.md) (CA trust, SSSD/LDAP, docker, otel, etc.).
- Long-form prose lives in the wiki (`https://wiki.alexandru.macocian.me`). This repo stays brief.
- Services live in stack repos and deploy through the shared workflow in this
repo.
- Hosts are onboarded for common deployment, SSH cert auth, SOPS, and
observability.
- Sherlock is the operator MCP/agent wrapper and now lives in
`amacocian/sherlock`.
- This repo should stay brief: current state, links, tables, and agent-facing
context only.
docs/automation.md
+49 -645
View File
@@ -1,650 +1,54 @@
# Automation
Reusable Gitea Actions that every `Charlie/<stack>` repo consumes. Lives at
[`.gitea/`](../.gitea/) in this repo. Replaces the per-stack `deploy.sh`
shell scripts that existed during the reorg.
Reusable Gitea Actions live in this repo and are consumed by stack/source repos.
Design context: [strategy.md](strategy.md), [secrets.md](secrets.md),
[runners.md](runners.md).
## Architecture
The runner is **stateless**. It registers with Gitea, picks up jobs, and
SSHes into the target host using a short-lived `step-ca`-issued user cert
to do the actual work as `sys-gitea-runner` (LDAP, member of
`charlie-deploy`).
```
push ──► Gitea Actions ──► any runner picks up the job
│
│ step ssh certificate sys-gitea-runner ... (5 min validity)
│ ssh -M sys-gitea-runner@<host> (multiplex master)
▼
on the target host:
cd /mnt/nas/stacks/<stack>
git fetch && git reset --hard
sops --decrypt …
docker compose pull && up -d
rm decrypted plaintext
▲
│ ssh -O exit (close multiplex)
│ wipe /tmp/charlie-ssh.*
```mermaid
flowchart LR
Push[Stack push] --> Gitea
Gitea --> Runner[Any runner]
Runner -->|step-ca SSH cert| Host[Target host]
Host --> Sync[Git sync]
Sync --> Secrets[SOPS decrypt]
Secrets --> Compose[Compose pull/up]
```
This sidesteps the Gitea reusable-workflow label-routing bug
([#32348](https://github.com/go-gitea/gitea/issues/32348)) — any runner
can deploy any stack to any host. The runner image
([`amacocian/gitea-runner`](https://gitea.alexandru.macocian.me/amacocian/gitea-runner))
ships `step-cli` + `openssh-client` and a `charlie-entrypoint.sh` that
runs `step ca bootstrap` at container start.
## What's here
| Kind | Path | Purpose |
|---|---|---|
| Reusable workflow | [`.gitea/workflows/deploy-stack.yml`](../.gitea/workflows/deploy-stack.yml) | The deployment task. Opens SSH session → per-host compose resolution + `git fetch && reset` + SOPS decrypt + `docker compose up -d` + cleanup, all on the target host. |
| Reusable workflow | [`.gitea/workflows/build-image.yml`](../.gitea/workflows/build-image.yml) | Build + push a Docker image from a source repo (`amacocian/<name>`) to the Gitea container registry. Tags with `latest` and a UTC timestamp (`YYYY-MM-DDTHHMMSSZ`); deployment repos pin the timestamp tag to roll out. |
| Composite action | [`.gitea/actions/charlie-ssh-open`](../.gitea/actions/charlie-ssh-open/action.yml) | Mints a 5-min SSH user cert for `sys-gitea-runner`, opens an SSH ControlMaster session to the target. Exports `CHARLIE_SSH_*` env to subsequent steps. |
| Composite action | [`.gitea/actions/charlie-ssh-target`](../.gitea/actions/charlie-ssh-target/action.yml) | Runs a command on the target through the multiplex socket. Optional `workdir:` cd's into a directory first. |
| Composite action | [`.gitea/actions/charlie-ssh-close`](../.gitea/actions/charlie-ssh-close/action.yml) | Tears down the multiplex + wipes the ephemeral cert. Idempotent; intended for `if: always()`. |
The three `charlie-ssh-*` actions are also usable directly for ad-hoc
SSH-from-CI patterns (e.g. one-off maintenance tasks, status checks). For
deploys, use the higher-level `deploy-stack.yml` workflow.
## How Gitea handles cross-repo references
Gitea Actions implements GitHub's reusable-workflow + composite-action
contract:
- **Reusable workflows** (`on: workflow_call`) are referenced from a caller's
`jobs.<id>.uses:` as `<owner>/<repo>/.gitea/workflows/<file>.yml@<ref>`.
- **Composite actions** (`runs.using: composite`) are referenced from a step's
`uses:` as `<owner>/<repo>/<path-to-action-dir>@<ref>`.
In our setup both forms must be **absolute URLs with auth** because of
Gitea bugs and instance-wide `REQUIRE_SIGNIN_VIEW`. See
[Gitea quirks](#gitea-quirks-leave-them-here-for-the-next-time-this-bites).
## Caller patterns
> Every `uses:` URL has to be the **absolute** form with the
> `sys-gitea-runner` PAT prefix. The relative form makes `act` default to
> github.com; the unauthenticated absolute form fails on auth. See
> [Gitea quirks](#gitea-quirks-leave-them-here-for-the-next-time-this-bites)
> below for the full story.
### Single-host stack, with secrets
```yaml
# Charlie/<stack>/.gitea/workflows/deploy.yaml
name: Deploy
on:
push:
branches: [main]
workflow_dispatch:
jobs:
deploy:
uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/workflows/deploy-stack.yml@main
with:
host: morgott
decrypt: |
secrets/env.sops.yaml -> .env
secrets: inherit
```
`RUNNER_REPO_PAT` is set as a `Charlie` org-level secret (Settings →
Actions → Secrets); every stack repo inherits it via `secrets: inherit`.
The PAT belongs to the LDAP-backed `sys-gitea-runner` service account
with read on the `Charlie` org.
### Single-host stack, no secrets
Drop the `decrypt:` input. `secrets: inherit` is still required so the
called workflow can clone its sibling actions.
```yaml
jobs:
deploy:
uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/workflows/deploy-stack.yml@main
with:
host: morgott
secrets: inherit
```
### Multiple secrets / non-`.env` formats
```yaml
with:
host: morgott
decrypt: |
secrets/env.sops.yaml -> .env
secrets/config.sops.yaml -> config/app.yaml
secrets/credentials.sops.json -> credentials.json
```
Output format is inferred from the **destination** extension (`.env` →
dotenv, `.json` → json, `.yaml`/`.yml` → yaml, `.ini` → ini, anything
else treated as raw bytes).
### Shared (multi-host) stack
```yaml
jobs:
deploy:
strategy:
matrix:
host: [morgott, melina, mohg, miquella]
uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/workflows/deploy-stack.yml@main
with:
host: ${{ matrix.host }}
decrypt: |
secrets/${{ matrix.host }}.env.sops.yaml -> envs/${{ matrix.host }}.env
secrets: inherit
```
`deploy-stack.yml` resolves compose files per
[strategy.md → Resolution order](strategy.md#per-host-resolution-order):
1. `docker-compose.<host>.yml` if it exists, **only** that.
2. Otherwise `docker-compose.yml` + (if present) `overrides/<host>.yml`.
3. If `envs/<host>.env` exists, passed as `--env-file`.
### Custom stack path
By default the workflow operates on `/mnt/nas/stacks/<repo-name>/` on the
target host. Override for repos that don't follow the convention:
```yaml
with:
host: morgott
stack-path: /mnt/nas/stacks/cert-issuer
```
### Direct SSH for non-deploy jobs
If you need to run something on a host outside the deploy lifecycle (a
status check, a one-off migration, etc.), use the `charlie-ssh-*` actions
directly:
```yaml
jobs:
smoke-test:
runs-on: self-hosted
steps:
- uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/actions/charlie-ssh-open@main
with:
host: morgott
- uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/actions/charlie-ssh-target@main
with:
run: |
curl -fsS https://homarr.alexandru.macocian.me/api/health
- if: always()
uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/actions/charlie-ssh-close@main
```
## Inputs reference — `deploy-stack.yml`
| Input | Required | Default | Description |
|---|---|---|---|
| `host` | yes | — | Hostname the workflow SSHes into. Resolves via the host's normal LAN DNS (Marika). |
| `stack-path` | no | `/mnt/nas/stacks/<repo-name>` | Absolute working-tree path on the target host. Must already be a `git` repo. |
| `decrypt` | no | `""` | Multi-line `src -> dst` SOPS decrypt list. Skipped if empty. Decryption runs on the target host using its `/etc/charlie/age.key`. |
| `compose-args` | no | `""` | Extra args appended to `docker compose ... up -d`. Use sparingly. |
| `pull` | no | `true` | Run `docker compose ... pull` before `up -d`. |
| `ref` | no | The branch the workflow ran on | Git ref to check out into the working tree on the target host. |
Requires the `RUNNER_REPO_PAT` secret to be in scope (set at org level;
callers pass via `secrets: inherit`).
### Registry login (built-in)
When `pull == true` (the default), `deploy-stack.yml` runs a fresh `docker
login gitea.alexandru.macocian.me` on the target host immediately before
`docker compose pull`, using a dedicated **read-only** PAT for
`sys-gitea-runner` (scope: `read:package`). The PAT lives as SOPS
ciphertext inlined in `deploy-stack.yml`, encrypted to every host's age
key; decryption happens on the target with `/etc/charlie/age.key`.
This is independent of `~/.docker/config.json` — necessary because
`build-image.yml`'s cleanup wipes the RW caller PAT after each build,
which would otherwise leave the next deploy 401ing on pull. The cleanup
step at the end of every deploy-stack run also `docker logout`s
unconditionally (`if: always()`) and scrubs any leftover ciphertext under
`/tmp/charlie-registry-creds.*.sops.yaml`, so the PAT never persists on
disk regardless of where the workflow failed.
Operator-facing details (encrypt recipe, paste location, rotation, host
add): [`secrets.md → Registry credentials (read-only PAT)`](secrets.md#registry-credentials-read-only-pat).
Callers don't pass anything for this — no input, no secret. It's an
internal implementation detail of `deploy-stack.yml`.
## Inputs reference — `build-image.yml`
Source repos use this workflow to build and push their container image
to the Gitea registry. Deployment repos (`Charlie/<stack>`) then pin
the timestamp tag in their compose file and trigger `deploy-stack.yml`
to roll out.
Same SSH-into-host architecture as `deploy-stack.yml`: the runner is
stateless; `docker build` / `docker push` execute on the build host's
daemon as `sys-gitea-runner` over a short-lived step-ca-issued cert.
The runner container deliberately does not have a docker socket bind
mount.
| Input | Required | Default | Description |
|---|---|---|---|
| `host` | yes | — | Build host the runner SSHes into. Must have docker + the standard `charlie-deploy` setup. Any deploy-target host qualifies. |
| `image` | no | `gitea.alexandru.macocian.me/<github.repository>` (lowercased) | Fully-qualified image name without the tag. Override only if the image should not live at its repo's natural path. |
| `context` | no | `.` | Docker build context, relative to the caller repo root. |
| `dockerfile` | no | `Dockerfile` | Dockerfile path, relative to `context`. |
| `build-args` | no | `""` | Multi-line `KEY=VALUE` list. Empty lines and `#` comments ignored. |
| `extra-tags` | no | `""` | Additional tags (one per line) pushed alongside `latest` + the timestamp. Just the tag, not the full `image:tag` form. |
| `push` | no | `true` | If `false`, build only — useful for PR validation. |
| `ref` | no | The branch the workflow ran on | Git ref to check out and build. |
| `registry-user` | no | `${{ github.repository_owner }}` | Username for `docker login`. Must match the user that owns `CALLER_REGISTRY_PAT` and have `write:package` on the target image namespace. |
Every successful build pushes **two** tags:
| Tag | Mutability | Purpose |
|---|---|---|
| `latest` | mutable | Convenience for local pulls and one-off `docker run`. **Never reference from a stack compose file** — there's no way for `deploy-stack.yml` to know a new digest landed. |
| `YYYY-MM-DDTHHMMSSZ` (UTC) | immutable | The tag deployment repos pin. Bumping it in `docker-compose.yml` and pushing the stack triggers a redeploy. Sortable lexicographically. |
Outputs (consumable by downstream jobs in the caller workflow):
| Output | Example | Use |
|---|---|---|
| `image` | `gitea.alexandru.macocian.me/<owner>/<repo>` | The image ref without the tag. |
| `timestamp-tag` | `2026-05-22T130927Z` | The tag the caller can write into the consumer stack's compose file (e.g. via a follow-up job that opens a PR against `Charlie/<stack>`). |
Requires two caller-owned PATs — see [Secret placement](#secret-placement-image-source-repos) below.
### Caller — minimal
```yaml
# <owner>/<repo>/.gitea/workflows/build.yaml
name: Build image
on:
push:
branches: [main]
workflow_dispatch:
jobs:
build:
uses: https://${{ github.repository_owner }}:${{ secrets.CALLER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/workflows/build-image.yml@main
with:
host: melina
secrets: inherit
```
### Caller — with build args and an extra tag
```yaml
jobs:
build:
uses: https://${{ github.repository_owner }}:${{ secrets.CALLER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/workflows/build-image.yml@main
with:
host: melina
dockerfile: docker/Dockerfile
build-args: |
VERSION=${{ github.ref_name }}
COMMIT=${{ github.sha }}
extra-tags: |
${{ github.ref_name }}
secrets: inherit
```
### Build-host prerequisites
The host named in `host:` runs the actual `docker build` / `docker push`.
Requirements are a strict subset of the regular deploy-target prerequisites
([Host prerequisites](#host-prerequisites)):
| Requirement | Why |
|---|---|
| `sys-gitea-runner` resolvable via LDAP + member of `charlie-deploy` | SSH session + docker socket access. |
| sshd trusts the step-ca SSH user CA | Runner-issued user certs are accepted. |
| `/var/run/docker.sock` is `root:charlie-deploy` | `docker build` / `push` work without sudo. |
| `docker buildx` plugin installed (`docker-buildx-plugin` / `docker-cli-buildx`) | The workflow exports `DOCKER_BUILDKIT=1` so Dockerfiles can use `RUN --mount=type=cache` etc. Without buildx, `docker build` errors with "BuildKit is enabled but the buildx component is missing". Verify with `docker buildx version`. |
| `git` available on the host | Cloning the caller source into the scratch workdir. |
| Outbound HTTPS to `gitea.alexandru.macocian.me` | `docker push` + `git clone`. |
No SOPS / age key / `/mnt/nas/stacks` is touched by a build — those are
deploy concerns only.
Builds accumulate base-image layers in the build host's docker storage.
The workflow drops the timestamp-tagged image after pushing, but base
layers stay cached (intentionally — speeds up subsequent builds). Run
`docker system prune` on the build host periodically.
### Secret placement (image-source repos)
Unlike `deploy-stack.yml` (which relies on the `sys-gitea-runner` system
account for everything), `build-image.yml` uses **caller-owned** PATs.
Each user/org that wants to build images provides their own credentials
— least privilege, no central PAT to share across namespaces, and the
package-write PAT only ever has access to that user's own namespace.
`sys-gitea-runner` is still the SSH identity into the build host (via
step-ca cert) — that's unchanged and unrelated to these secrets.
Set both at the **user-level** Settings → Actions → Secrets on the
account that owns the source repos. All repos under that personal
namespace then inherit both via `secrets: inherit`:
| Secret | Gitea PAT scope | Owner | Used for |
|---|---|---|---|
| `CALLER_REPO_PAT` | `read:repository` (All) | The calling user/org | Resolving the absolute-URL `uses:` reference to this workflow + its sibling composite actions in `Charlie/project-charlie`, and cloning the caller source on the build host. |
| `CALLER_REGISTRY_PAT` | `write:package` (All, or scoped to the target namespace) | The calling user/org | `docker login` + push to the Gitea container registry. |
Two distinct tokens so the package-write PAT is never used over HTTPS
git (and the repo-read PAT can never be used to push images); revoking
one doesn't take the other down.
#### Recipe — onboarding a new image-building namespace (one-time)
1. **Grant the calling user read access to `Charlie/project-charlie`.**
Either add them as a Read collaborator on the repo (Charlie/project-charlie
→ Settings → Collaborators) or as a Read member of the `Charlie` org.
Required so their PAT can resolve the `uses:` URL.
2. Log in to Gitea as the calling user. Settings → Applications →
"Generate New Token" twice:
- `build-repo-ro`: `read:repository`, Repository and Organization
Access: All (or at minimum: `Charlie/project-charlie` + the user's
own source repos).
- `build-package-rw`: `write:package`, Repository and Organization
Access: All (or scoped to the namespace receiving the images).
3. Still logged in as the calling user → Settings → Actions → Secrets →
"Add Secret":
- `CALLER_REPO_PAT` = the `build-repo-ro` value.
- `CALLER_REGISTRY_PAT` = the `build-package-rw` value.
4. Drop the [Caller — minimal](#caller--minimal) snippet into each
source repo at `.gitea/workflows/build.yaml`. Push.
## Inputs reference — `charlie-ssh-open`
| Input | Required | Default | Description |
|---|---|---|---|
| `host` | yes | — | Hostname to SSH into. LAN-resolvable. |
| `user` | no | `sys-gitea-runner` | Remote user. |
| `duration` | no | `5m` | Cert validity. Capped by the `runners` provisioner config in step-ca (currently 10m max). |
Exports to `$GITHUB_ENV` so subsequent steps can use them:
| Variable | Meaning |
|---|---|
| `CHARLIE_SSH_HOST` | The target host. |
| `CHARLIE_SSH_USER` | The remote user. |
| `CHARLIE_SSH_DIR` | Tmpdir holding the ephemeral key + cert. |
| `CHARLIE_SSH_SOCKET` | Unix socket for the SSH multiplex master. |
## Inputs reference — `charlie-ssh-target`
| Input | Required | Default | Description |
|---|---|---|---|
| `run` | yes | — | Shell command(s) to run on the target. Multi-line allowed; runs as `bash -lc <command>`. |
| `workdir` | no | `""` | Optional directory to cd into before running. |
Reads `CHARLIE_SSH_*` from job env (must come after `charlie-ssh-open`).
## Inputs reference — `charlie-ssh-close`
No inputs. Reads `CHARLIE_SSH_*` from job env. Idempotent. Always end your
job with this in an `if: always()` step.
## Host prerequisites
Every host that wants to be a deploy target needs:
| Requirement | Why |
|---|---|
| `sys-gitea-runner` resolves via LDAP/SSSD; `charlie-deploy` group exists with `sys-gitea-runner` as a member | Runner SSHes in as this user; group grants docker + stacks access. See [onboarding.md → Charlie deploy group](onboarding.md#charlie-deploy-group). |
| sshd trusts the step-ca SSH user CA (`/etc/ssh/trusted_user_ca.pub` populated by `sync-ca.sh`) | Runner-issued user certs are accepted by sshd. See [onboarding.md → Trusted CA](onboarding.md#trusted-ca). |
| `/etc/charlie/age.key` (`0640`, `root:charlie-deploy`) | SOPS decrypt by `sys-gitea-runner`. See [onboarding.md → Host age key (SOPS)](onboarding.md#host-age-key-sops). |
| `/var/run/docker.sock` is `root:charlie-deploy` | `docker compose` works without sudo. See [onboarding.md → Charlie deploy group](onboarding.md#charlie-deploy-group). |
| `/mnt/nas/stacks/<stack>/.git` exists, owned `sys-gitea-runner:charlie-deploy 0775+s` | The deploy step does `git fetch && git reset --hard` after the first run. If the working tree is missing, the workflow auto-clones it on first deploy (see [Auto-provisioning new stacks](#auto-provisioning-new-stacks) below). |
The runner host itself only needs the runner stack (and step-ca trust if
you want to run `step ssh login` interactively as a human). The runner
container reaches step-ca via `network_mode: host` and the host's resolver.
### Auto-provisioning new stacks
The reusable workflow's "Provision or verify stack working tree" step
auto-clones a stack on its first deploy. Behaviour:
- **`<path>/.git` exists** → no-op, fall through to Sync.
- **`<path>` missing or empty** → `sudo install -d` the parent (one-shot,
ownership = `sys-gitea-runner`), then `git clone --branch <ref>` the
stack repo using the runner's `RUNNER_REPO_PAT` as a one-shot
`http.extraHeader`. The PAT is **not** persisted to `.git/config`.
- **`<path>` exists, non-empty, no `.git`** → fail loud. Refuses to
clobber a hand-created tree; operator must clear it manually first.
Subsequent `git fetch` calls in the Sync step use the host's existing
git credentials for `sys-gitea-runner` (same as manually-cloned stacks).
A host with no configured git identity will fail at Sync — surfacing the
gap rather than silently embedding a long-lived PAT on disk. Tracked as
[roadmap.md → Identity & access → Per-host git identity](roadmap.md#identity--access).
## Manual fallback (no runner available)
The deploy workflow is the happy path. When you need to push by hand —
runner outage, gitea-on-melina (which can't be deployed by a runner that
depends on it), first-time bring-up before runners exist, or debugging —
SSH into the target host yourself and run the equivalent commands. SOPS
is offline crypto; nothing else in the fleet has to be up.
### Stack with secrets
```bash
# SSH in via gssh as your normal LDAP user (alex/amacocian).
ssh morgott
# Become sys-gitea-runner. -H sets HOME so credential helpers work.
sudo -u sys-gitea-runner -H bash
cd /mnt/nas/stacks/<stack>
git fetch --prune origin
git reset --hard origin/main
HOST=$(hostname -s | tr '[:upper:]' '[:lower:]')
# Decrypt every secret the workflow would have decrypted. Mirror the
# `decrypt:` block in .gitea/workflows/deploy.yaml.
SOPS_AGE_KEY_FILE=/etc/charlie/age.key \
sops --decrypt --output-type dotenv --output .env secrets/env.sops.yaml
chmod 600 .env
# Pick the right compose files (mirrors deploy-stack.yml resolution).
if [ -f "docker-compose.$HOST.yml" ]; then
COMPOSE_FILES=(-f "docker-compose.$HOST.yml")
else
COMPOSE_FILES=(-f docker-compose.yml)
[ -f "overrides/$HOST.yml" ] && COMPOSE_FILES+=(-f "overrides/$HOST.yml")
fi
ENVFILE=()
[ -f "envs/$HOST.env" ] && ENVFILE=(--env-file "envs/$HOST.env")
docker compose "${ENVFILE[@]}" "${COMPOSE_FILES[@]}" pull
docker compose "${ENVFILE[@]}" "${COMPOSE_FILES[@]}" up -d
# Clean up plaintext.
rm -f .env
exit # leave sys-gitea-runner shell
```
### Stack without secrets
Same as above, minus the decrypt and the `rm -f .env` lines.
## Gitea quirks (leave them here for the next time this bites)
Hard-won lessons from getting the first end-to-end pipeline working.
Worth re-reading before debugging anything in this area.
### Cross-repo `uses:` needs absolute URL with PAT
The relative form `Charlie/project-charlie/...@main` makes `act` default
to GitHub. The absolute form without auth fails because Gitea has
`REQUIRE_SIGNIN_VIEW=true` instance-wide. So every `uses:` referencing a
sibling repo has to be:
```
uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/<repo>/...@<ref>
```
This applies **transitively** — inside `deploy-stack.yml`, the `uses:`
for the `charlie-ssh-*` composite actions also have to be absolute-URL
form, otherwise act tries github.com from inside the workflow.
The PAT belongs to the LDAP-backed `sys-gitea-runner` service account
(LDAP → synced into Gitea via Authentik OAuth → placed in a `runners`
team in the `Charlie` org with read on all repos). PAT itself is set as
a `Charlie` org-level secret so every stack repo inherits it. Callers
must include `secrets: inherit` so the called workflow sees the secret.
References:
- https://github.com/go-gitea/gitea/issues/25929 — the canonical issue
- https://github.com/go-gitea/gitea/pull/36173 — the fix that landed in Gitea 1.27 (we pre-date it; the PAT-in-URL workaround is needed)
### Reusable-workflow `runs-on` label matching is broken
`runs-on: [self-hosted, morgott]` inside a reusable workflow does **not**
correctly route to a runner that advertises both labels. Gitea's matching
either picks any runner with `self-hosted` (loose match) or hangs. This
is [#32348](https://github.com/go-gitea/gitea/issues/32348) — open,
type/bug. PR #36388 was attempted, closed; replaced by #37478, still
WIP at time of writing.
We sidestep it entirely by using `runs-on: self-hosted` and SSHing to
the target host inside the job. Any runner can deploy any stack.
**Documented exception: [`Charlie/runners`](runners.md).** The runners
stack genuinely needs precise routing (cross-deploy: each host's runner
deploys the *other* host so no runner ever recreates itself mid-job).
Top-level `runs-on:` in a regular workflow *does* respect labels — the
bug is reusable-workflow-specific — so `Charlie/runners` ships a
hand-rolled
[`.gitea/workflows/deploy.yaml`](https://gitea.alexandru.macocian.me/Charlie/runners/src/branch/main/.gitea/workflows/deploy.yaml)
with two pinned jobs instead of calling `deploy-stack.yml`. When #32348
is fixed, collapse it into a `matrix: { host: [morgott, melina] }`
calling the shared workflow. Background: [runners.md → Updates](runners.md#updates-image-bump-or-env-change).
### `act` aggressively caches reusable workflows / composite actions
After pushing a fix to `Charlie/project-charlie`, the runner still uses
the old version because act caches at
`/root/.cache/act/Charlie-project-charlie@main`. The cache key is the
ref string, not the commit. To force a refresh after pushing a fix:
```bash
sudo docker exec runner rm -rf /root/.cache/act
```
Long-term fix: stop using `@main` for cross-repo refs. Pin to specific
tags (`@v1`, `@<sha>`). New ref string → new cache entry → automatic
refresh. Left for later (until the contract stabilises).
### LAN DNS inside the runner container
Default Docker DNS (`127.0.0.11` embedded resolver) doesn't know about
`.lan` hostnames. The runner stack uses `network_mode: host` so the
container shares the host's resolver and can reach `morgott.lan`,
`cert-issuer.lan` (well, that one's docker-internal — `morgott.lan` is
the right name), etc.
### Note about cert-issuer.lan
`cert-issuer.lan` is the docker-network alias for the step-ca container,
only resolvable from inside the cert-issuer's docker network. From
anywhere else (including the runner container with `network_mode: host`),
use `morgott.lan:9000` (cert-issuer is pinned to morgott — see
[cert-issuer.md](cert-issuer.md)).
### Workflow logs collapse
The Gitea Actions UI shows everything happening in `Set up job` rather
than separate cards per step. Cosmetic; doesn't reflect the actual step
structure. The full chronological log inside `Set up job` is what to
scroll through. Suspected to be a Gitea / act_runner UI bug; not
impacting correctness.
### `insufficient permission for adding an object to repository database .git/objects`
Symptom: deploy step "Sync working tree" fails with the above git error
on `git fetch` against `/mnt/nas/stacks/<stack>/`.
Cause: the stack's `.git/` is owned by `root` because the working tree
was bootstrapped manually with `sudo git clone ...` instead of going
through the deploy workflow (or onboarding step 5 in
[onboarding.md → Charlie deploy group](onboarding.md#charlie-deploy-group)).
The deploy workflow runs as `sys-gitea-runner` over SSH and can't write
into root-owned objects.
Fix once on the affected host:
```bash
sudo chown -R sys-gitea-runner:charlie-deploy /mnt/nas/stacks/<stack>
sudo find /mnt/nas/stacks/<stack> -type d -exec chmod 2775 {} \;
sudo find /mnt/nas/stacks/<stack> -type f -exec chmod g+rw {} \;
```
Then re-trigger the workflow. To avoid re-introducing this for new
stacks, do the manual bootstrap clone *as* `sys-gitea-runner`:
```bash
sudo -u sys-gitea-runner -H git clone <url> /mnt/nas/stacks/<stack>
```
### `strategy.max-parallel: 1` doesn't actually serialize
Symptom: shared-stack deploy lands on host A and host B in parallel;
host A's `if: always()` cleanup `rm`s the decrypted `.env` while host B's
`compose up` is still running, producing `couldn't read env file`. Or:
two parallel `git fetch && reset --hard` against the same NFS-shared
`.git` race on lock files.
Cause: Gitea Actions accepts `strategy.max-parallel:` syntactically but
does not reliably honor it for matrix jobs that call a reusable
workflow. Both jobs get queued and pick up runners simultaneously.
Fix: don't use the matrix shape for shared stacks. Spell each host out as
a separate job, chained with `needs:` for true sequential execution.
Same shape we use in [`Charlie/runners`](https://gitea.alexandru.macocian.me/Charlie/runners/src/branch/main/.gitea/workflows/deploy.yaml)
and [`Charlie/otelcollector`](https://gitea.alexandru.macocian.me/Charlie/otelcollector/src/branch/main/.gitea/workflows/deploy.yaml).
```yaml
jobs:
deploy-morgott:
uses: ... deploy-stack.yml@main
with: { host: morgott, ... }
secrets: inherit
deploy-melina:
needs: deploy-morgott
uses: ... deploy-stack.yml@main
with: { host: melina, ... }
secrets: inherit
```
Long-term cleanup: have `deploy-stack.yml` decrypt to a per-host
filename (e.g. `.env.<host>` or `/run/charlie-secrets/<stack>/.env`)
and pass it via `--env-file`. Then matrix-with-true-parallelism becomes
safe, since neither the working tree nor the env file collide. Until
then, `needs:` is the working pattern.
## Adding new reusable parts
If you find yourself copying logic across stack workflows, pull it in here.
Two flavors:
- **Composite action** at `.gitea/actions/<name>/action.yml` — small,
single step's worth of logic. Cheap to call from inside any workflow.
- **Reusable workflow** at `.gitea/workflows/<name>.yml` — owns the whole
job (runner selection, sequencing, multiple steps).
Pin callers to `@main` for now; cut `@v1` etc. once the contract is
stable. Breaking changes to inputs need a major bump.
| Workflow/action | Online source | Purpose |
| -------------------- | --------------------------------------------------------------------------------------------------------------- | ----------------------------------- |
| `deploy-stack.yml` | <https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/.gitea/workflows/deploy-stack.yml> | Deploy a stack on a target host. |
| `build-image.yml` | <https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/.gitea/workflows/build-image.yml> | Build and push a source repo image. |
| `charlie-ssh-open` | <https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/.gitea/actions/charlie-ssh-open> | Open a cert-backed SSH session. |
| `charlie-ssh-target` | <https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/.gitea/actions/charlie-ssh-target> | Run a command through that session. |
| `charlie-ssh-close` | <https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/.gitea/actions/charlie-ssh-close> | Close and clean up the session. |
## Deploy path
| Stage | State |
| ------------- | ---------------------------------------------------------- |
| Trigger | Push or manual dispatch in `Charlie/<stack>`. |
| Runner | Any self-hosted runner can pick up the job. |
| Auth | step-ca SSH user cert for `sys-gitea-runner`. |
| Target | Host from the workflow `host` input. |
| Work | Git sync, optional SOPS decrypt, compose pull/up, cleanup. |
| Registry pull | Built into `deploy-stack.yml` through an encrypted RO PAT. |
## Build path
| Stage | State |
| ---------- | -------------------------------------- |
| Trigger | Source repo push or manual dispatch. |
| Runner | Any self-hosted runner. |
| Build host | Host from workflow `host` input. |
| Tags | `latest` plus UTC timestamp. |
| Deploy | Stack repos pin the desired image tag. |
## Gitea constraints kept in mind
| Constraint | Current handling |
| --------------------------------------------- | ------------------------------------------------------ |
| Cross-repo `uses` needs auth | Absolute Gitea URLs with PAT auth. |
| Reusable-workflow runner labels unreliable | Use `runs-on: self-hosted`; route by SSH target. |
| `act` caches `@main` aggressively | Tag workflow/action contracts when stable. |
| Shared-stack matrix can race on NFS/plaintext | Serialize jobs with `needs` where required. |
| Runner stack must not recreate itself mid-job | `Charlie/runners` cross-deploys with its own workflow. |
Workflow inputs are documented in the workflow files themselves.
docs/cert-issuer.md
+27 -88
View File
@@ -1,97 +1,36 @@
# Cert issuer
[`Charlie/cert-issuer`](https://gitea.alexandru.macocian.me/Charlie/cert-issuer) — [smallstep `step-ca`](https://smallstep.com/docs/step-ca/) on morgott. Issues short-lived SSH user certs (Gssh, runners, admin) and X.509 server certs (cert-syncer-managed).
Operational details (provisioners table, add/restart procedure, init recipe, gotchas) live in [`Charlie/cert-issuer`'s README](https://gitea.alexandru.macocian.me/Charlie/cert-issuer/src/branch/main/README.md). Onboarding-side work (`sync-ca.sh`, host trust) lives in [onboarding.md → Trusted CA](onboarding.md#trusted-ca).
## Architecture
```
┌──────────┐ OAuth/OIDC ┌────────────┐
user ─►│ Gssh │────────────────────►│ Authentik │
└────┬─────┘ └────────────┘
│ step ssh certificate (auth: JWK provisioner password)
▼
┌──────────────────┐ ┌────────┐
│ step-ca │───►│ CA key │ (today: file; planned: TPM-sealed)
│ (Charlie/ │ └────────┘
│ cert-issuer) │
└───────┬─────────┘
▲
│ step ca root, /ssh/roots, step ca certificate
│
┌───────┴─────────┐
│ cert-syncer │ hourly loop. Pure producer; never restarts anyone.
│ (Charlie/ │ Writes:
│ cert-syncer) │ /mnt/nas/ca/ca.crt
└───────┬─────────┘ /mnt/nas/ca/ssh/ssh_user_ca.pub
│ /mnt/nas/ca/<host>/<host>.{crt,key}
▼
/mnt/nas/ca/ ─── read by sync-ca.sh on every host ─── /etc/{sssd,ssh}/...
│
│ read by stack consumers + sidecar
▼
┌───────────────────────┐
│ consumer stack │ e.g. Charlie/ldap = openldap + ldap-ui + cert-watch
│ openldap │ cert-watch polls cert files, restarts openldap on
│ ldap-ui │ change. docker.sock privilege is per-stack, not
│ cert-watch (side) │ in cert-syncer.
└───────────────────────┘
```mermaid
flowchart LR
CertIssuer[cert-issuer / step-ca] --> SSHCerts[SSH user certs]
CertIssuer --> X509[X.509 certs]
CertSyncer[cert-syncer] --> NASCA["/mnt/nas/ca"]
NASCA --> Hosts[Hosts]
NASCA --> Consumers[Cert-consuming stacks]
```
Three separable concerns, three repos:
| Repo | Role |
| --------------------- | ---------------------------------------------- |
| `Charlie/cert-issuer` | step-ca on morgott. |
| `Charlie/cert-syncer` | Refreshes CA/cert material into `/mnt/nas/ca`. |
| `Charlie/cert-watch` | Consumer-side cert restart sidecar. |
- **cert-issuer** — passive responder. Mints certs on demand. Holds the CA private key.
- **cert-syncer** — active puller. Refreshes trust files and renews managed server certs. Has rw on `/mnt/nas/ca/`. Doesn't know who consumes anything.
- **cert-watch** — sidecar image. Lives in each consumer stack. Watches cert files for that stack and restarts the consumer on change. Has docker.sock for *its own stack only*. Travels with the stack across hosts.
## Issued material
Hosts read from `/mnt/nas/ca/` exactly as before — they don't know step-ca exists.
| Material | Used by |
| ------------------ | --------------------------------------------- |
| SSH user certs | Gssh, runners, operator/admin SSH. |
| SSH user CA pubkey | Host `sshd` trust. |
| X.509 server certs | Internal services that consume managed certs. |
| X.509 root | Host trust stores and service clients. |
## Open questions
## Boundaries
### 1. Where does the CA key live
| Boundary | State |
| ------------------ | --------------------------------------------- |
| Human identity | Authentik remains the IdP. |
| Host authorization | LDAP/SSSD groups remain authoritative. |
| Cert storage | `/mnt/nas/ca` remains the distribution point. |
| CA key | Held by cert-issuer. |
Today: step-ca's data dir at `/mnt/nas/data/cert-issuer/step/secrets/`, encrypted at rest with `DOCKER_STEPCA_INIT_PASSWORD` (now SOPS-sealed, decrypted at deploy time).
Future hardening:
- **TPM-sealed on the issuer host.** Strongest for an on-prem fleet. `step-ca` supports `tpmkms` directly. CA key never exists in cleartext on disk. Pins the issuer to a specific host.
- **YubiHSM 2 / YubiKey.** Even stronger; physical device. Adds hardware to budget.
- **PKCS#11 with SoftHSM.** Same API as a real HSM, no hardware. Useful as a stepping stone to YubiHSM.
### 2. Caller authn + authz
Two distinct caller classes:
- **Humans, via Gssh** — Gssh authenticates the user via OIDC (Authentik), then asks the issuer for a cert with principals = LDAP groups/username. Future: have step-ca do the OIDC dance itself (cleaner separation).
- **Machines (runners, host bootstrap)** — service identity (JWK/X5C provisioner). Should *not* go through OIDC.
Authz rules to define:
- Which callers can request which **principals**? (A runner shouldn't mint a cert with `principal=alex`.)
- Maximum **validity** per caller class. (Humans: 8h; runners: minutes; bootstrap: hours. Today: gssh 5m, admin 16h/24h, runners 5m/10m, cert-syncer 1y/2y X.509.)
- Which callers can request **host certs** vs **user certs**?
### 3. Audit + revocation
- **Audit:** issuer logs every cert (principals, fingerprint, requester, validity). Cheap, do it from day one.
- **Revocation:** SSH certs lack OCSP/CRL. Two mechanisms:
- Short validity windows (current). Compromise window = validity window.
- `RevokedKeys` file distributed to every host's `sshd_config`. Hard revoke, requires host-side push.
Short validity + audit log for v1; add `RevokedKeys` distribution if/when an incident demands it.
## Pending
- **Roll the new X.509 root to remaining hosts** (mohg, miquella, malenia) — they auto-pick up at the next hourly `sync-ca.timer` tick.
- **Add Authentik as an OIDC provisioner** on step-ca. Lets Gssh stop carrying the JWK password — step-ca authenticates the human directly via the OIDC token Gssh forwards.
- **Issue SSH host certs.** Replace TOFU with `@cert-authority *.lan ...` trust client-side and `HostCertificate` server-side. cert-syncer is the natural place to manage these (extend `MANAGED_CERTS` with a host-cert variant).
- **Move CA key to TPM-seal** ([open question #1](#1-where-does-the-ca-key-live)).
- **Decommission the old X.509 root** at `/mnt/nas/ca/ca.crt`. step-ca minted a fresh one; the old root is still in trust stores. Decommissioning is a separate later step.
- **Rewrite `sync-ca.sh`** to fetch directly from step-ca's HTTPS endpoint instead of `/mnt/nas/ca/`. Removes the NAS dependency.
## Scope (what step-ca does NOT replace)
- Authentik. The issuer trusts Authentik (directly via OIDC provisioner, or transitively via Gssh) for human identity.
- SSSD/LDAP for authorization on the host. The cert says "this user is `alex`"; SSSD-backed PAM still decides what `alex` can do (sudo, group membership).
- X.509 certs for Caddy edge / internal mTLS. step-ca *can* do this; mixing on day one bloats the decision space.
Operational details live in `Charlie/cert-issuer`.
docs/conventions.md
+10 -6
View File
@@ -1,8 +1,12 @@
# Conventions
- One stack = one repo under `Charlie/`. Repo contains `docker-compose.yml`, `.env.example`, `.gitea/workflows/deploy.yaml`. **No `deploy.sh`** — the workflow calls the shared [`deploy-stack.yml`](../.gitea/workflows/deploy-stack.yml) reusable workflow that lives here in [`project-charlie`](../README.md). See [automation.md](automation.md).
- Secrets never in Git in plaintext. SOPS-encrypted under `secrets/` in the stack repo, decrypted on the target host at deploy time by the reusable workflow (which SSHes into the host as `sys-gitea-runner` and runs `sops --decrypt` there using `/etc/charlie/age.key`). Design + open decisions in [secrets.md](secrets.md).
- Persistent state stays under `/mnt/nas/data/<stack>/` on miquella. Repos define bind paths, never ship data.
- Deploys are SSH-driven: any runner picks up the job, then opens a short-lived step-ca-issued SSH session to the target host (as `sys-gitea-runner`, member of `charlie-deploy`) and runs all commands there. The runner itself is stateless. Manual fallback recipe: [automation.md → Manual fallback](automation.md#manual-fallback-no-runner-available).
- Workflows pin the target host via the reusable workflow's `host:` input. Routing-by-runner-label is intentionally avoided (Gitea bug; see [automation.md → Gitea quirks](automation.md#gitea-quirks-leave-them-here-for-the-next-time-this-bites)).
- Image tags pinned in compose. No `:latest`.
| Area | Convention |
| ----------------- | ----------------------------------------------------------------------------- |
| Stack ownership | One stack repo under `Charlie/` per deployed service group. |
| Deploy entrypoint | `.gitea/workflows/deploy.yaml` calls this repo's shared deploy workflow. |
| Host selection | Workflow `host` input; not runner labels. |
| Working tree | `/mnt/nas/stacks/<stack>` is reset to the repo state on deploy. |
| Runtime data | `/mnt/nas/data/<stack>` only; never inside the git working tree. |
| Secrets | SOPS files under `secrets/`; plaintext ignored and deploy-time only. |
| Images | Compose pins concrete tags or digests. No `:latest` in stack repos. |
| Docs | State, references, tables, lists. No copied workflow bodies or shell recipes. |
docs/hosts.md
+25 -21
View File
@@ -1,29 +1,33 @@
# Hosts
| Name | Role | Hardware | NAS |
|------|------|----------|-----|
| Radagon | TP-Link router (WAN) | — | — |
| Marika | OpenWrt router (intranet gateway) | — | — |
| Godfrey | Network switch | — | — |
| Morgott | Caddy + control-plane apps | Intel NUC | yes |
| Melina | Gitea + media stack + apps | Intel NUC, 4c/31G | yes |
| Mohg | apps | Fujitsu Esprimo | yes |
| Malenia | apps | Mac Mini 2012 | — |
| Miquella | NAS, exports `/volume1/ubuntu` → `/mnt/nas` (NFS) | Synology DSM | source |
| Host | Role | State |
| -------- | ----------------------------------- | --------- |
| Radagon | WAN router | active |
| Marika | OpenWrt intranet gateway + DNS/DHCP | active |
| Godfrey | Network switch | active |
| Morgott | Edge ingress + control-plane stacks | onboarded |
| Melina | Gitea + media/apps | onboarded |
| Mohg | App host | onboarded |
| Malenia | Mac app/utility host | onboarded |
| Miquella | NAS; exports `/mnt/nas` | onboarded |
Bootstrap procedure: [onboarding.md](onboarding.md).
| Host | Runs stacks | Notes |
| -------- | ----------: | ----------------------------------------------- |
| Morgott | yes | Caddy, identity, observability, certs, runners. |
| Melina | yes | Gitea, media, app stacks, runners. |
| Mohg | yes | Capacity host. |
| Malenia | yes | macOS-specific host. |
| Miquella | limited | NAS plus selected host services. |
## Age public keys (SOPS)
## SOPS recipients
Public keys are non-secret. Used as recipients in `Charlie/<stack>/.sops.yaml` files. Generated per [onboarding.md → Host age key (SOPS)](onboarding.md#host-age-key-sops). Private keys live at `/etc/charlie/age.key` on each host (`0600`, `root:root`).
Public age keys are non-secret. Private keys stay on each host at
`/etc/charlie/age.key`.
| Owner | Public key |
|---|---|
| **admin** | `age19d6mc8arznnwnvtp0xrphlseuwf94p0f3pznszrtmd5wgmjpzqdq9mltvs` |
| Owner | Public key |
| ------- | ---------------------------------------------------------------- |
| admin | `age19d6mc8arznnwnvtp0xrphlseuwf94p0f3pznszrtmd5wgmjpzqdq9mltvs` |
| morgott | `age1m5ha3fd8psek5pym4spk7encjk4vmsml46eya3eymp7q9d9jre5s7xe84r` |
| melina | `age1xfp878l4ul5zulqqul0u60c44c5wj94cmladgc6jt5jrlmrdrezsd8twh2` |
| mohg | _(generated when first stack on mohg needs SOPS)_ |
| miquella | _(generated when first stack on miquella needs SOPS)_ |
| malenia | _(generated when first stack on malenia needs SOPS)_ |
| melina | `age1xfp878l4ul5zulqqul0u60c44c5wj94cmladgc6jt5jrlmrdrezsd8twh2` |
Backup of admin private key: vaultwarden Secure Note `Charlie SOPS admin age key`.
Admin key backup: vaultwarden Secure Note `Charlie SOPS admin age key`.
docs/identity.md
+24 -10
View File
@@ -1,15 +1,29 @@
# Identity & access
| Service | URL | Purpose |
|---------|-----|---------|
| Authentik | https://id.alexandru.macocian.me/ | IdP (OIDC / OAuth2 / SAML) |
| Caddy auth | https://auth.alexandru.macocian.me/ | `caddy-security`, fronts protected apps |
| Web terminal | https://terminal.alexandru.macocian.me/ | Browser SSH via ephemeral signed certs |
```mermaid
flowchart LR
User --> Authentik
Authentik --> Gssh
Gssh --> StepCA[step-ca]
StepCA --> Cert[SSH user cert]
Cert --> Host[Target host]
LDAP[LDAP/SSSD] --> Host
```
## Terminal flow
| System | Role | Current owner |
| -------------- | -------------------------------------- | --------------------- |
| Authentik | IdP for web apps and human auth | `Charlie/authentik` |
| Caddy security | Edge auth for protected routes | `Charlie/caddy` |
| LDAP / SSSD | Host users, groups, sudo, deploy group | `Charlie/ldap` |
| step-ca | SSH user certs and managed X.509 certs | `Charlie/cert-issuer` |
| Gssh | Web terminal + cert-backed SSH | `Charlie/gssh` |
| Sherlock | Operator agent/MCP credential wrapper | `amacocian/sherlock` |
OAuth → Authentik → backend mints ephemeral SSH cert (signed by CA trusted on every host) → SSH proxied over WebSocket to JS frontend.
| Principal | Used for |
| ------------------- | ------------------------------------------------------------- |
| `sys-gitea-runner` | CI deploy/build SSH identity; member of `charlie-deploy`. |
| `charlie-deploy` | Docker socket, stack working trees, SOPS age key read access. |
| `sys-host-<host>` | Host-local root Git/registry operations. |
| `sys-trivy-scanner` | Read-only registry pulls for CVE scanning. |
CA material lives under `/mnt/nas/ca/` and is synced to hosts by `sync-ca.sh`. Extracting cert issuance into a standalone service (so the CA key stops living on every host that needs to mint) is captured in [cert-issuer.md](cert-issuer.md).
Host logins / sudo are SSSD-backed against `open-ldap`; LDAP groups drive permissions.
Terminal flow: Authentik login -> Gssh -> step-ca SSH cert -> target host SSH.
docs/inventory.md
+46 -72
View File
@@ -1,80 +1,54 @@
# Stack inventory
# Inventory
Canonical list of `Charlie/<stack>` repos. Working trees live at `/mnt/nas/stacks/<name>/`.
Stack repos live under `https://gitea.alexandru.macocian.me/Charlie/<stack>`.
**Image source** — `upstream` = published image used directly; `gitea-registry` = built off-host, pushed to `gitea.alexandru.macocian.me/amacocian/<name>:<tag>`.
## Stacks
## Morgott (owner host)
| Stack | Host(s) | Purpose |
| ----------------- | ------------------------------- | ------------------------------------- |
| `caddy` | morgott | Edge reverse proxy + auth. |
| `authentik` | morgott | IdP. |
| `ldap` | morgott | LDAP directory + UI. |
| `ddclient` | morgott | Dynamic DNS. |
| `vaultwarden` | morgott | Password vault. |
| `bookstack` | morgott | Wiki. |
| `monica` | morgott | CRM. |
| `miniflux` | morgott | RSS. |
| `mealie` | morgott | Recipes. |
| `homarr` | morgott | Dashboard. |
| `paperless-ngx` | morgott | Document management. |
| `victoriametrics` | morgott | Metrics, alerts, Grafana. |
| `gssh` | morgott | Web SSH terminal. |
| `portainer` | morgott | Container UI. |
| `cert-issuer` | morgott | step-ca. |
| `cert-syncer` | morgott | Cert material sync/renewal. |
| `calendarsync` | morgott | Google/Outlook calendar sync. |
| `trivy-server` | morgott | Shared Trivy DB + compose image scan. |
| `gitea` | melina | Git server. |
| `mediacompose` | melina | Media stack. |
| `searxng` | melina | Metasearch. |
| `invidious` | melina | YouTube frontend. |
| `nitter` | melina | X/Twitter frontend. |
| `redlib` | melina | Reddit frontend. |
| `planka` | melina | Kanban. |
| `audiobookshelf` | melina | Audiobooks. |
| `directoryadmin` | melina | LDAP admin UI. |
| `newsletter-rss` | melina | Email newsletter to Atom feeds. |
| `runners` | morgott, melina | Gitea Actions runners. |
| `portainer-agent` | morgott, melina, mohg, miquella | Portainer agents. |
| `otelcollector` | morgott, melina, mohg, malenia | Host/container telemetry. |
| `trivy-runner` | morgott, melina | Running-container CVE scan. |
| Charlie repo | Stack path | Image source | Notes |
|---|---|---|---|
| [`caddy`](https://gitea.alexandru.macocian.me/Charlie/caddy) | `caddy/` | local build | edge reverse proxy + caddy-security |
| [`authentik`](https://gitea.alexandru.macocian.me/Charlie/authentik) | `authentik/` | upstream | IdP; postgres + redis + server + worker |
| [`ldap`](https://gitea.alexandru.macocian.me/Charlie/ldap) | `ldap/` | upstream | openldap + ldap-ui |
| [`ddclient`](https://gitea.alexandru.macocian.me/Charlie/ddclient) | `ddclient/` | upstream | dynamic DNS |
| [`vaultwarden`](https://gitea.alexandru.macocian.me/Charlie/vaultwarden) | `vaultwarden/` | upstream | password vault |
| [`bookstack`](https://gitea.alexandru.macocian.me/Charlie/bookstack) | `bookstack/` | upstream | wiki (+ mariadb) |
| [`monica`](https://gitea.alexandru.macocian.me/Charlie/monica) | `monica/` | gitea-registry `monica:4.x` | CRM (custom 4.x fork; + mariadb) |
| [`miniflux`](https://gitea.alexandru.macocian.me/Charlie/miniflux) | `miniflux/` | upstream | RSS (+ postgres) |
| [`mealie`](https://gitea.alexandru.macocian.me/Charlie/mealie) | `mealie/` | upstream | recipes (sqlite) |
| [`homarr`](https://gitea.alexandru.macocian.me/Charlie/homarr) | `homarr/` | upstream | dashboard |
| [`paperless-ngx`](https://gitea.alexandru.macocian.me/Charlie/paperless-ngx) | `paperless-ngx/` | upstream | docs (+ redis) |
| [`victoriametrics`](https://gitea.alexandru.macocian.me/Charlie/victoriametrics) | `victoriametrics/` | upstream | vm + vmauth + vmalert + alertmanager + grafana |
| [`gssh`](https://gitea.alexandru.macocian.me/Charlie/gssh) | `gssh/` | gitea-registry `gssh:latest` | web SSH terminal |
| [`portainer`](https://gitea.alexandru.macocian.me/Charlie/portainer) | `portainer/` | upstream | container UI; CSRF trusted-origins set via CLI |
| [`cert-issuer`](https://gitea.alexandru.macocian.me/Charlie/cert-issuer) | `cert-issuer/` | upstream `smallstep/step-ca:0.30.2` | SSH user-cert CA. Design: [cert-issuer.md](cert-issuer.md). |
| [`cert-syncer`](https://gitea.alexandru.macocian.me/Charlie/cert-syncer) | `cert-syncer/` | upstream `smallstep/step-cli:0.30.2-trixie` | Active bridge from cert-issuer to `/mnt/nas/ca/`. Renews managed server certs. Design: [cert-issuer.md](cert-issuer.md). |
| [`calendarsync`](https://gitea.alexandru.macocian.me/Charlie/calendarsync) | `calendarsync/` | upstream `ghcr.io/inovex/calendarsync:0.11.0` (+ `mcuadros/ofelia:0.3.22`) | Bidirectional Google ↔ Outlook calendar sync via inovex/CalendarSync. Two one-way pipelines on a 15-min cron, staggered 7 min apart. No UI, no port in steady state. OAuth tokens persisted (AES) under `/mnt/nas/data/calendarsync/{g2o,o2g}/state/`. |
| [`trivy-server`](https://gitea.alexandru.macocian.me/Charlie/trivy-server) | `trivy-server/` | upstream `aquasec/trivy:0.70.0` (+ `mcuadros/ofelia:0.3.22`, `mikefarah/yq:4.53.2`) | Central trivy vuln-DB cache on `morgott.lan:4954` (LAN-only, no auth). Daily 03:00 sweep of compose-declared image refs across `/mnt/nas/stacks/*/`. Pairs with `trivy-runner`. |
## Source/image repos
## Melina (owner host)
| Charlie repo | Stack path | Image source | Notes |
|---|---|---|---|
| [`gitea`](https://gitea.alexandru.macocian.me/Charlie/gitea) | `gitea/` | upstream | the git server itself (deployed manually, never via runner-on-self) |
| [`mediacompose`](https://gitea.alexandru.macocian.me/Charlie/mediacompose) | `mediacompose/` | upstream | jellyfin + arr + qbt-via-gluetun |
| [`searxng`](https://gitea.alexandru.macocian.me/Charlie/searxng) | `searxng/` | upstream | metasearch (+ valkey) |
| [`invidious`](https://gitea.alexandru.macocian.me/Charlie/invidious) | `invidious/` | gitea-registry `invidious:oidc-support` | YouTube frontend (+ companion + db + gluetun) |
| [`nitter`](https://gitea.alexandru.macocian.me/Charlie/nitter) | `nitter/` | upstream `zedeus/nitter:<commit>` (no versioning) | Twitter/X frontend (+ redis). Requires operator-seeded `sessions.jsonl` on host. |
| [`redlib`](https://gitea.alexandru.macocian.me/Charlie/redlib) | `redlib/` | gitea-registry `redlib:sha-<commit>` (no versioning) | Reddit frontend. Stateless, no secrets. Built off-host from `Dockerfile.ubuntu` because upstream quay image is stale ([#551](https://github.com/redlib-org/redlib/issues/551)). |
| [`planka`](https://gitea.alexandru.macocian.me/Charlie/planka) | `planka/` | upstream | kanban (+ postgres) |
| [`audiobookshelf`](https://gitea.alexandru.macocian.me/Charlie/audiobookshelf) | `audiobookshelf/` | upstream | audiobooks |
| [`directoryadmin`](https://gitea.alexandru.macocian.me/Charlie/directoryadmin) | `directoryadmin/` | gitea-registry `directoryadmin:latest` | LDAP admin UI |
## Sidecar image (consumed by other stacks)
| Charlie repo | Image | Used by | Notes |
|---|---|---|---|
| [`cert-watch`](https://gitea.alexandru.macocian.me/Charlie/cert-watch) | gitea-registry `cert-watch:latest` | [`ldap`](https://gitea.alexandru.macocian.me/Charlie/ldap) | Tiny alpine sidecar. Polls cert files, restarts sibling containers on change via docker.sock. Add to any stack consuming a cert-syncer-managed cert. |
## Shared — multi-host
| Charlie repo | Hosts | Layout | Auto-deploy | Notes |
|---|---|---|---|---|
| [`runners`](https://gitea.alexandru.macocian.me/Charlie/runners) | morgott, melina | per-host `secrets/<host>.env.sops.yaml`, host-local `/var/lib/charlie/runner/` | morgott, melina (cross-deploy; see [runners.md](runners.md)) | Gitea Actions runners. Image: [`amacocian/gitea-runner`](https://gitea.alexandru.macocian.me/amacocian/gitea-runner). |
| [`portainer-agent`](https://gitea.alexandru.macocian.me/Charlie/portainer-agent) | morgott, melina, mohg, miquella | `docker-compose.yml` + `docker-compose.miquella.yml` | morgott, melina | DSM gets a per-host compose (no `/var/lib/docker/volumes` bind). mohg + miquella deploy by hand until SSSD-onboarded. |
| [`otelcollector`](https://gitea.alexandru.macocian.me/Charlie/otelcollector) | morgott, melina, mohg, malenia | `docker-compose.yml` (shared); `secrets/env.sops.yaml` for the vmauth ingest header | morgott, melina | mohg deploys by hand until reonboarding; malenia (`launchd` + `install-mac.sh`) still ships the credential in `otelcol.mac.yaml` until reonboarded — see roadmap. |
| [`trivy-runner`](https://gitea.alexandru.macocian.me/Charlie/trivy-runner) | morgott, melina | `docker-compose.yml` + `overrides/<host>.yml` (per-host trivy-server URL + cron schedule) | morgott, melina (sequential, mirrors otelcollector) | Daily `docker ps` sweep; staggered 03:00 morgott / 03:15 melina. Writes to `/var/lib/charlie/trivy/metrics/` consumed by otelcollector's node-exporter. |
| Repo | Purpose |
| ------------------------ | --------------------------------------------------------------- |
| `amacocian/gitea-runner` | Runner image consumed by `Charlie/runners`. |
| `amacocian/sherlock` | Public operator credential wallet, MCP launcher, agent wrapper. |
| `amacocian/<app>` | Custom app images consumed by selected stacks. |
## Operator tooling
Runs on the operator's workstation, not on a Charlie host. Installed per-operator (e.g. `go install`), no Docker, no `deploy-stack.yml` caller.
| Charlie repo | Purpose | Status |
|---|---|---|
| [`sherlock`](https://gitea.alexandru.macocian.me/Charlie/sherlock) | Per-operator credential broker + agent-CLI wrapper. Holds the Authentik session once, mints per-service tokens, injects them into stdio MCPs, and execs the agent CLI of choice (Copilot, Claude Code, …). Design: [`Charlie/sherlock/plan.md`](https://gitea.alexandru.macocian.me/Charlie/sherlock/src/branch/main/plan.md). | phase 0 — scaffold |
## Mirrors (Gitea pull-mirror, no host action)
Naming: `Charlie/<name>-mirror`.
| Mirror | Upstream | Status |
|---|---|---|
| `gssh` source | `https://github.com/AlexMacocian/Gssh` | pushed to `amacocian/Gssh` (manual) |
| `monica` source | `https://github.com/AlexMacocian/monica` (4.x) | mirror under `amacocian/monica` |
| `directoryadmin` source | `https://github.com/AlexMacocian/DirectoryAdmin` | mirror under `amacocian/DirectoryAdmin` |
| `invidious` source | `https://github.com/AlexMacocian/invidious` (oidc-support) | mirror under `amacocian/invidious` |
| `redlib` source | `https://github.com/redlib-org/redlib` | mirror under `amacocian/redlib`; image built locally from `Dockerfile.ubuntu`, pushed to `gitea.alexandru.macocian.me/amacocian/redlib:<tag>` |
| `gitea-runner` source | none (laptop only) | image built locally, pushed to `gitea.alexandru.macocian.me/amacocian/gitea-runner:<tag>`. Source: [`amacocian/gitea-runner`](https://gitea.alexandru.macocian.me/amacocian/gitea-runner). |
Optional later (upstreams we don't customise but want offline): `caddy`, `authentik`, `paperless-ngx`, `mealie`, `victoriametrics`, `searxng-docker`, `nitter`, `gitea`, `act_runner`.
| Tool | Repo | State |
| -------- | -------------------------------------------------------- | ------------------------------------------------------- |
| Sherlock | <https://gitea.alexandru.macocian.me/amacocian/sherlock> | Up and running; provides Gitea, Grafana, and Gssh MCPs. |
docs/layout.md
+17 -76
View File
@@ -1,80 +1,21 @@
# Layout
Canonical `/mnt/nas/` shape after the reorg.
| Path | Contents |
| -------------------------- | ----------------------------------------------- |
| `/mnt/nas/stacks/<stack>` | Git working tree for a stack repo. |
| `/mnt/nas/data/<stack>` | Runtime state and bind-mounted data. |
| `/mnt/nas/media` | Bulk media. |
| `/mnt/nas/ca` | CA roots, SSH CA pubkey, managed cert material. |
| `/mnt/nas/ldap-certs` | LDAP cert material. |
| `/var/lib/charlie/<stack>` | Host-local state or textfile metrics. |
| `/etc/charlie/age.key` | Host SOPS age private key. |
```
/mnt/nas/
├── stacks/ ← per-stack git working trees. Small, all-in-git.
│ ├── caddy/
│ ├── ldap/
│ ├── ...
├── data/ ← per-stack runtime state. Bind-mounted into containers.
│ ├── caddy/
│ ├── ldap/
│ ├── mediacompose/
│ │ ├── jellyfin/
│ │ ├── sonarr/
│ │ └── ...
│ ├── ...
├── media/ ← bulk media (renamed from media-server/)
│ ├── downloads/
│ ├── movies/
│ └── tv/
├── ca/ ← host CA material; out of scope for the reorg
├── ldap-certs/ ← LDAP server certs (renamed from ldap/); out of scope
└── home/ ← legacy; deleted at end of reorg
```
| Naming | Rule |
| --------------- | ------------------------- |
| Stack repo | `Charlie/<stack>` |
| Stack path | `/mnt/nas/stacks/<stack>` |
| Data path | `/mnt/nas/data/<stack>` |
| Directory style | lowercase kebab-case |
## Naming
- All directories under `stacks/` and `data/` are **kebab-case, lowercase**.
- Repo names under `Charlie/` match the `stacks/` directory name 1:1.
| Old | New |
|-----|-----|
| `home/Gssh/` | `stacks/gssh/` |
| `home/MediaServerCompose/` | `stacks/mediacompose/` |
| `home/DirectoryAdmin/` | `stacks/directoryadmin/` |
| `home/open-ldap/` | `stacks/ldap/` |
| `home/VictoriaMetrics/deployment/docker/` | `stacks/victoriametrics/` |
| `home/mealie/mealie/docker/` | `stacks/mealie/` |
| `home/authentik/authentik/` | `stacks/authentik/` |
| `home/searxng-docker/` | `stacks/searxng/` |
| `home/paperless-ngx/` | `stacks/paperless-ngx/` |
| `home/ServerManagementUtils/` | `stacks/server-mgmt-utils/` |
| `home/portainer-agent/` | `stacks/portainer-agent/` |
## Bind-mount convention
Every compose file binds **only** `/mnt/nas/data/<stack>/...` for runtime state. No bind into `./` (which is the git working tree).
```yaml
# stacks/ldap/docker-compose.yml
services:
openldap:
volumes:
- /mnt/nas/data/ldap/db:/var/lib/ldap
- /mnt/nas/data/ldap/config:/etc/ldap/slapd.d
```
Small read-only configs that *should* be version-controlled (Caddyfile, openldap bootstrap LDIFs, etc.) live inside the `stacks/<stack>/` working tree and are bind-mounted as files:
```yaml
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile:ro
```
## Out-of-scope dirs (leave alone for now)
- `/mnt/nas/ca/` — host CA material. Considered separately later (`Charlie/ca-tooling`).
- `/mnt/nas/ldap-certs/` — LDAP server certs (after rename from `ldap/`). Same.
- `/mnt/nas/#recycle` — Synology recycle bin.
## Stacks-only-as-config (no container)
Some `stacks/<name>/` may not deploy a container — they're just version-controlled config that belongs to the homelab:
- `stacks/shell/` — shared `profile.d`, `inputrc`. Cloned by hosts to `/usr/local/share/charlie-shell/`, sourced by `/etc/profile`.
- `stacks/first-time-setup/` — bootstrap docs / scripts (currently `home/first-time-setup.md`).
These have no `docker-compose.yml` and no entry in the runner deploy list.
Runtime state belongs in `/mnt/nas/data` or host-local `/var/lib/charlie`, not
in stack repos.
docs/onboarding.md
+28 -1064
View File
File diff suppressed because it is too large Load Diff
docs/roadmap.md
+23 -46
View File
@@ -1,52 +1,29 @@
# Roadmap
# Next work
Open work, grouped by theme. Done items are dropped — git history has them.
Project Charlie is mostly in steady state. Next work is agent-facing: prompts,
skills, and MCP context.
## Hosts
## Prompting and skills
- [ ] **Mohg reonboarding.** Wipe + reinstall mohg, run a `charlie-host-bootstrap` (see [Centralization](#centralization)) end-to-end. Folds mohg into auto-deploy for [`portainer-agent`](https://gitea.alexandru.macocian.me/Charlie/portainer-agent) and [`otelcollector`](https://gitea.alexandru.macocian.me/Charlie/otelcollector).
- [ ] **Malenia (mac) re-onboard.** NFS mount unreliable from malenia → otelcollector mac install deferred. Investigate NFS/SMB/AFP options or switch sync mechanism. Once reonboarded, refactor `otelcol.mac.yaml` to read `${env:VM_INGEST_AUTH}` and drop the credential from git (collapses rotation from 3 places to 1).
- [ ] **Break the LDAP-on-morgott boot cycle.** `ldap` runs as a docker stack on morgott, so sssd can't be a hard prereq for docker (sssd → ldap container → docker → circular). We currently work around it with a local `/etc/group` shadow entry for `charlie-deploy` (see [onboarding.md → Charlie deploy group](onboarding.md#charlie-deploy-group), step 3). Architecturally cleaner: move `Charlie/ldap` onto a host whose docker socket doesn't depend on an LDAP-resolved group, or run slapd natively somewhere. Removes the GID-duplication and the "if LDAP renumbers the group, update each host's /etc/group" footgun.
- [ ] **Miquella into the LDAP/SSH fold.** DSM is currently outside SSSD-backed identity. Configure DSM's LDAP client to point at `Charlie/ldap`, trust the SSH signing CA, accept ephemeral certs from gssh. Goal: malenia + miquella both onboarded into the gssh terminal flow with LDAP-driven access.
| Work item | Target |
| ----------------------- | ----------------------------------------------------------------------------------------------------- |
| Repository instructions | Add concise agent rules for Charlie docs, stack repos, and deployment safety. |
| Skill pack | Create Charlie skills for stack deploys, Sherlock/MCP use, observability triage, and secret rotation. |
| MCP context | Document Gitea, Grafana, and Gssh tool boundaries through Sherlock. |
| Prompt library | Store reusable prompts for stack changes, incidents, CVE triage, and host work. |
| Validation checklist | Define the minimal checks for docs-only, stack, workflow, and secret changes. |
| Link policy | Link online resources only; no local workstation paths in docs. |
## Automation / observability
## Platform follow-ups
- [ ] **Diun watching registries** → webhook → `redeploy` workflow with `force-pull` (replaces what watchtower used to do).
- [ ] **Drift detection.** Running compose vs repo compose.
- [ ] **Backup verification** against miquella.
- [ ] **Cert/secret expiry checks** (authentik, caddy, signing CA).
- [ ] **Build pipelines** on `amacocian/*` source repos to push images to gitea registry (replaces "build on laptop").
- [ ] **Grafana dashboards as code.** Grafana 12.2's UI Git Sync only supports github.com — Gitea isn't an option. Pattern: dashboards live as JSON files inside `Charlie/victoriametrics/dashboards/`, mounted into grafana's provisioning path with `foldersFromFilesStructure: true` so subdirs become Grafana folders. Edits flow via a "Cannot save provisioned dashboard" → "Copy JSON" → paste into repo loop. Sidecar that auto-syncs UI edits → git deferred until manual loop becomes painful.
- [x] **Initial dashboard set.** `hosts/` (overview, host-detail, thermal-power, filesystem-network), `processes/` (overview + CPU + memory broken down per process), `containers/` (overview, per-container resources, per-stack roll-up). Container metrics added by enabling the otelcol `docker_stats` receiver (mounts `/var/run/docker.sock`, promotes `com.docker.compose.project|service` to metric labels).
- [x] **Container CVE sweep.** [`Charlie/trivy-server`](https://gitea.alexandru.macocian.me/Charlie/trivy-server) (morgott, central vuln-DB cache + daily compose-declared image sweep) + [`Charlie/trivy-runner`](https://gitea.alexandru.macocian.me/Charlie/trivy-runner) (per-host, daily `docker ps` sweep). Results land as Prometheus textfiles → node-exporter (textfile collector) → otelcol → VM. Dashboards under `containers/security/`. LAN-only between the two stacks; no public DNS, no auth — morgott + melina sit on the secure intranet.
- [ ] **Trivy CVE alerting.** vmalert rule on `trivy_image_vulnerabilities{severity="CRITICAL"} > 0` → alertmanager via the existing route. Deferred until the sweep has produced a stable baseline (otherwise the first run is one giant page).
- [ ] **CI build-time CVE scan.** Run `trivy image` as a step inside the eventual `amacocian/*` build pipelines (see "Build pipelines" above) so we catch HIGH/CRITICAL at build time, not just at the next nightly sweep.
- [ ] **Runner observability (phase 2).** Gitea Actions runner dashboards in `runners/`. Likely a small exporter polling Gitea's Actions API → otelcol → VM. Tracked in [`Charlie/victoriametrics/dashboards/runners/README.md`](https://gitea.alexandru.macocian.me/Charlie/victoriametrics/src/branch/main/dashboards/runners/README.md).
| Work item | Owner area |
| ------------------------------------ | ------------------------- |
| Tag workflow/action contracts | Automation |
| Per-host secret materialization path | Deploy workflow |
| CVE alerting baseline | Observability |
| Build-time image scanning | Source/image repos |
| Dashboards as code | `Charlie/victoriametrics` |
| Cert issuer hardening | `Charlie/cert-issuer` |
| Bootstrap tooling | Host onboarding |
## Deploy workflow
- [ ] **Per-host decrypt path in `deploy-stack.yml`.** Today both halves of a shared-stack deploy decrypt to the same `/mnt/nas/stacks/<stack>/.env` over NFS, so we serialize via `needs:` instead of `strategy.matrix`. If decrypt landed at e.g. `/run/charlie-secrets/<stack>-<host>/.env` (host-local tmpfs, per-host filename), matrix-with-true-parallelism becomes safe and the workflow files for shared stacks collapse back to a single matrix job. Also helps once Gitea's `runs-on:` label routing in reusable workflows is fixed (see [#32348](https://github.com/go-gitea/gitea/issues/32348)).
- [ ] **Tag pinning** for cross-repo `uses:`. Caller workflows reference `@main`, which means after every push to this repo or the runner image, runners need their `act` cache busted (`sudo docker exec runner rm -rf /root/.cache/act`). Switch to `@v1` once contract stabilises.
## Centralization
- [ ] **`Charlie/shell`.** `profile.d/`, `inputrc`, etc., cloned to `/usr/local/share/charlie-shell/` (or `/etc/profile.d/charlie.sh` sourcing a NAS path).
- [ ] **`Charlie/charlie-tools`** (or similar). Host-level scripts: `sync-ca.sh`, future temp/storage metric collectors (after `server-mgmt-utils` is dismantled).
- [ ] **`charlie-host-bootstrap`.** Script (or Ansible playbook) that runs every numbered section of [onboarding.md](onboarding.md) from a fresh OS install. Validation plan: wipe mohg, run the bootstrapper end-to-end, confirm it joins the fleet without manual fixups. Pairs with [Mohg reonboarding](#hosts).
## Identity & access
- [ ] **Auto-mount user home folders on SSH connect.** Currently each user's `$HOME` is local to the host they SSH into. Want each user's home to be a NAS-backed share automounted on connect (NFS or autofs + LDAP user attribute). Should pair with the LDAP-backed SSSD setup (`Charlie/ldap`).
- [ ] **Per-host git identity.** Each host needs its own credential to talk to Gitea — currently we paste a PAT inline. Decide: per-host PAT in the host's keyring, host-keyed SSH key registered as a deploy key on each `Charlie/<stack>` (rotation pain), or one "host-fleet" service account with org-scope read + per-stack write.
- [ ] **Per-repo permissions via LDAP-backed service accounts.** Use `Charlie/ldap` to provision service accounts (e.g. `svc-runner-morgott`, `svc-deploy-mediacompose`) that Gitea recognises through OIDC. Per-repo `read` / `write` / `admin` roles map to LDAP groups. Replaces the "everything pushes as `amacocian`" pattern. Reduces blast radius of a compromised runner.
- [ ] **Cert-issuer follow-ups** — see [cert-issuer.md → Pending](cert-issuer.md#pending) (X.509 root rollout, Authentik OIDC provisioner, SSH host certs, TPM-sealed CA key, `sync-ca.sh` rewrite).
## Cleanup
- [ ] **NAS layout polish.**
- Rename `/mnt/nas/media-server/` → `/mnt/nas/media/` and update mediacompose `.env`.
- Rename `/mnt/nas/ldap/` → `/mnt/nas/ldap-certs/`.
- Move `/mnt/nas/portainer/data/` → `/mnt/nas/data/portainer/`.
- [ ] **`server-mgmt-utils`.** Mostly dead. Extract the temp/storage metric scripts into either `Charlie/otelcollector/scripts/` or new `Charlie/host-scripts`, then archive.
- [ ] **`plist/`.** Classify and decide.
- [ ] **Secret rotation pass.** Every stack with secrets exposed during the original NAS-to-git reorg.
Done work stays in Git history, not this doc.
docs/runners.md
+23 -87
View File
@@ -1,93 +1,29 @@
# Runners
Design for [`Charlie/runners`](https://gitea.alexandru.macocian.me/Charlie/runners) — the Gitea Actions runner stack. Pairs with the runner image at [`amacocian/gitea-runner`](https://gitea.alexandru.macocian.me/amacocian/gitea-runner) and the reusable workflow + composite actions in [`.gitea/`](../.gitea/) (see [automation.md](automation.md)).
| Area | Current state |
| ------------ | ------------------------------------------------------ |
| Stack repo | `Charlie/runners` |
| Image repo | `amacocian/gitea-runner` |
| Hosts | morgott, melina |
| Registration | instance-wide Gitea Actions runners |
| Storage | host-local `/var/lib/charlie/runner` |
| Deploy model | cross-deploy; each runner updates the other host first |
Status: **deployed.** One runner each on morgott + melina, registered instance-wide.
## Runtime model
## Goal
| Item | State |
| ---------------- | ------------------------------------------- |
| Runner container | Stateless except registration credential. |
| Job routing | Any runner for normal stacks. |
| Target work | SSH to host as `sys-gitea-runner`. |
| SSH auth | short-lived step-ca cert. |
| Docker access | target host socket via `charlie-deploy`. |
| Repo access | `RUNNER_REPO_PAT` and host Git credentials. |
Self-hosted Gitea Actions runners that can deploy any `Charlie/<stack>` to any host. The runner is **stateless** — it holds only its own registration credential. All real work (git, sops, docker compose) happens on the target host via SSH using a short-lived step-ca-issued user cert.
## Exceptions
Two runners gives capacity + cross-deploy capability. Adding mohg/miquella runners later is a capacity decision; no design change needed.
## Image
[`amacocian/gitea-runner`](https://gitea.alexandru.macocian.me/amacocian/gitea-runner) — upstream `gitea/act_runner` plus:
| Layered on top | Why |
|---|---|
| `git`, `sops`, `docker-cli`, `docker-cli-compose` | Workflow steps run inside this container; need these tools available locally for `act` and remotely over ssh |
| `bash`, `sudo`, `jq` | wrapper scripts |
| `step-cli` | mints SSH user certs from step-ca's `runners` JWK provisioner at job time |
| `openssh-client` | `charlie-ssh-*` wrappers in `/usr/local/bin/` |
| `charlie-entrypoint.sh` | at container start: bootstraps step-ca trust + materialises git credentials from `$GITEA_PAT` |
Built locally, pushed manually. [Bump recipe](https://gitea.alexandru.macocian.me/amacocian/gitea-runner#build--push).
## Stack layout
```
Charlie/runners/
├── docker-compose.yml # network_mode: host
├── envs/.gitkeep # decrypted at deploy, gitignored
├── secrets/
│ ├── morgott.env.sops.yaml
│ └── melina.env.sops.yaml
├── .sops.yaml
├── .env.example
└── .gitea/workflows/deploy.yaml # cross-deploy; see "Updates"
```
State lives **host-local** at `/var/lib/charlie/runner/` (just the `.runner` registration credential). NFS + per-runner identity = bad fit for shared storage.
Encrypted env contents are documented in [`Charlie/runners/.env.example`](https://gitea.alexandru.macocian.me/Charlie/runners/src/branch/main/.env.example).
## Bootstrap (per host, one-time)
Recipe lives with the stack: [`Charlie/runners/README.md → Bootstrap`](https://gitea.alexandru.macocian.me/Charlie/runners/src/branch/main/README.md#bootstrap-per-host-one-time).
The first-ever runner needs manual bring-up because there's no runner to deploy it. Subsequent runners can be bootstrapped manually too (simpler) or you can land the encrypted env first and let the existing runner deploy the new one.
## Updates (image bump or env change)
Automatic on push to `Charlie/runners`. Runner updates are the **one stack that does not call the shared [`deploy-stack.yml`](../.gitea/workflows/deploy-stack.yml)** — instead it ships its own [`.gitea/workflows/deploy.yaml`](https://gitea.alexandru.macocian.me/Charlie/runners/src/branch/main/.gitea/workflows/deploy.yaml) with two cross-deploy jobs:
| Job | `runs-on` | Target |
|---|---|---|
| `deploy-morgott` | `[self-hosted, melina]` | morgott |
| `deploy-melina` | `[self-hosted, morgott]` (after `deploy-morgott`) | melina |
No runner ever recreates itself mid-job, so the deploy is a normal synchronous `compose pull && up -d` with real exit codes.
### Why the duplication
Gitea bug [#32348](https://github.com/go-gitea/gitea/issues/32348): `runs-on:` label routing is broken inside reusable workflows but works from a top-level workflow. Cross-deploy needs precise routing, so the reusable workflow can't be used here. Full bug write-up: [automation.md → Gitea quirks](automation.md#gitea-quirks-leave-them-here-for-the-next-time-this-bites).
When the upstream fix lands (PRs #36388 / #37478), collapse the runners workflow to a `matrix: { host: [morgott, melina] }` calling `deploy-stack.yml`.
### Image bumps
1. Build + push the new image tag.
2. Commit + push `Charlie/runners` with the new `image:` in `docker-compose.yml`.
3. Cross-deploy workflow runs automatically.
### Manual fallback
If both runners are down, neither cross-deploy job has anywhere to land. Use the [Bootstrap](#bootstrap-per-host-one-time) recipe (it's the same recipe, just done by hand).
## Permissions model
| Identity | Where used | Purpose |
|---|---|---|
| `sys-gitea-runner` (LDAP) | `~sys-gitea-runner/.git-credentials` (host); SSH user via cert (target host); `RUNNER_REPO_PAT` org secret | THE deployment identity. Member of `charlie-deploy`. Read-only on `Charlie` org. |
| Image-source caller (Gitea user) | `CALLER_REPO_PAT` (`read:repository`) + `CALLER_REGISTRY_PAT` (`write:package`), set per Gitea namespace | Image builds run from non-`Charlie` namespaces under that namespace's own identity; both PATs are set there so [`build-image.yml`](../.gitea/workflows/build-image.yml) can fetch + clone + push without loaning out `sys-gitea-runner` credentials. The caller user must be a Read collaborator on `Charlie/project-charlie`. See [automation.md → Secret placement](automation.md#secret-placement-image-source-repos). |
| `sys-host-<host>` (LDAP) | `/root/.git-credentials` + `/root/.docker/config.json` (host) | Root-level operations on each host: `sudo git ...`, `sudo docker pull`. Read-only on `Charlie` org. |
The act_runner registration credential is its **own** identity in Gitea. It can pick up jobs and report results; it cannot read other repos. When it needs to clone for `act`'s `uses:` resolution, it uses the `RUNNER_REPO_PAT` materialised by the entrypoint shim.
## Open / future
- **`step ca bootstrap` re-pin on cert rotation.** If step-ca's root fingerprint rotates, every runner needs `STEPCA_FINGERPRINT` updated. The entrypoint runs `step ca bootstrap --force` so an updated fingerprint just needs the new value in the encrypted env.
- **SSH host certs.** sshd should present a step-ca-issued host cert; the runner should pin `@cert-authority *.lan ...` rather than TOFU. Tracked in [cert-issuer.md](cert-issuer.md).
- **3+ runners.** Currently 2. Adding a third (mohg/miquella) breaks the "deploy from the *other* host" trick because there's no longer exactly one other host. Likely answer: designate two "deployer" runners. Defer until [#32348](https://github.com/go-gitea/gitea/issues/32348) is fixed; the matrix-on-shared-workflow shape may make the question moot.
- **Tag pinning.** Caller workflows pin to `@main`. After every push to this repo or the runner image, busting `act` cache is necessary (`sudo docker exec runner rm -rf /root/.cache/act`). Switch to `@v1` when the contract stabilises.
| Exception | State |
| -------------------------- | -------------------------------------------------- |
| `Charlie/runners` workflow | Does not use `deploy-stack.yml`. |
| Precise runner labels | Used only for runner cross-deploy. |
| `@main` cache | Bust or tag when workflow/action contract changes. |
docs/secrets.md
+31 -86
View File
@@ -1,95 +1,40 @@
# Secrets
SOPS-in-repo with one [age](https://github.com/FiloSottile/age) key per host. Encrypted files live next to `docker-compose.yml` in each `Charlie/<stack>` repo. The reusable [`deploy-stack.yml`](../.gitea/workflows/deploy-stack.yml) workflow SSHes into the target host as `sys-gitea-runner` and runs `sops --decrypt` there using `/etc/charlie/age.key`.
Operational recipe: [`onboarding.md → Host age key (SOPS)`](onboarding.md#host-age-key-sops).
Caller-facing reference: [`automation.md`](automation.md).
## Goal
Every host can:
1. Obtain only the secrets it's allowed to decrypt.
2. Materialize them where compose can consume them (typically `.env`).
3. Do this non-interactively from a Gitea Actions runner.
## Constraints
- **Source of truth lives with the stack.** A secret for `mediacompose` belongs in [`Charlie/mediacompose`](https://gitea.alexandru.macocian.me/Charlie/mediacompose), not in a separate vault. Blast radius is bounded by repo access.
- **Per-machine decryption.** A compromised host leaks only the secrets in repos whose `.sops.yaml` lists that host's key.
- **Self-identifying host.** `hostname -s | tr '[:upper:]' '[:lower:]'` everywhere.
- **Versioning is git.** Rotation = commit + (when needed) `sops updatekeys` + push.
## Layout in a stack repo
```
Charlie/<stack>/
├── docker-compose.yml
├── .env.example # tracked, redacted
├── secrets/
│ └── env.sops.yaml # encrypted; decrypts to .env
├── .sops.yaml # creation rules: which recipients can decrypt
└── .gitea/workflows/deploy.yaml # calls deploy-stack.yml
```mermaid
flowchart LR
StackRepo[Stack repo secrets/] --> SOPS[SOPS ciphertext]
SOPS -->|age recipient| HostKey["/etc/charlie/age.key"]
HostKey --> Deploy[Deploy on target host]
Deploy --> Plaintext["Plaintext env/config"]
Plaintext --> Compose[docker compose]
```
For shared stacks, `.sops.yaml` lists every host that runs the stack.
| Area | Current state |
| -------------- | -------------------------------------------------------- |
| Stack secrets | SOPS-encrypted files under each stack repo's `secrets/`. |
| Decryption | Target host decrypts with `/etc/charlie/age.key`. |
| Admin recovery | Admin age key listed in every relevant `.sops.yaml`. |
| Human vault | Vaultwarden stores human-facing secrets and PAT notes. |
| Plaintext | Deploy-time only; ignored by Git. |
## Keys
## Key material
- **One age key per host**, at `/etc/charlie/age.key` (`0640 root:charlie-deploy`). Public keys in [`hosts.md`](hosts.md#age-public-keys-sops).
- **One admin key** held by alex on the admin laptop, listed in *every* repo's `.sops.yaml` so secrets stay decryptable from off-host. Backup as a vaultwarden Secure Note (`Charlie SOPS admin age key`).
- **Per-host, not per-stack.** Container isolation already provides the per-stack property — `/etc/charlie/age.key` isn't bind-mounted into containers.
| Secret/key | Owner | Location |
| -------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------ |
| Host age private key | each host | `/etc/charlie/age.key` |
| Host age public key | docs | [hosts](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/hosts.md#sops-recipients) |
| Admin age private key | operator | vaultwarden Secure Note `Charlie SOPS admin age key` |
| Registry RO PAT ciphertext | project-charlie | inline in `deploy-stack.yml` |
## Rotation
## PATs
```bash
# In the stack repo on the admin laptop.
tmp=$(mktemp) && chmod 600 "$tmp"
printf 'KEY=new-value\n' > "$tmp"
sops --encrypt --input-type dotenv --output-type yaml \
--filename-override secrets/env.sops.yaml "$tmp" > secrets/env.sops.yaml
shred -u "$tmp"
git commit -am 'Rotate <KEY>'
git push # triggers redeploy
```
| Identity | Scope | Used for |
| ------------------ | ----------------- | ------------------------------------------ |
| `sys-gitea-runner` | `read:repository` | workflow/action fetches and host Git sync. |
| `sys-gitea-runner` | `read:package` | registry pulls during deploy. |
| `sys-host-<host>` | `read:repository` | host-local root Git operations. |
| `sys-host-<host>` | `read:package` | host-local root registry pulls. |
| caller user/org | `read:repository` | source-repo build workflow fetch/clone. |
| caller user/org | `write:package` | source-repo image push. |
For per-host secret files, repeat per host.
Adding/removing recipients (e.g. onboarding mohg): edit `.sops.yaml`, then `sops updatekeys secrets/env.sops.yaml`.
## Registry credentials (read-only PAT)
`deploy-stack.yml` runs `docker login gitea.alexandru.macocian.me` on the target host immediately before `docker compose pull`. This is independent of any creds cached in `~/.docker/config.json` — necessary because `build-image.yml`'s cleanup wipes the RW caller PAT after each build, which otherwise leaves the next deploy stranded.
The login uses a dedicated **read-only** PAT (scope: `read:package`) belonging to `sys-gitea-runner`. A host compromise leaks pull access only; pushes still require the per-namespace RW PATs that callers inject into `build-image.yml`.
The ciphertext lives **inlined** in [`deploy-stack.yml`](../.gitea/workflows/deploy-stack.yml) between `SOPS_BLOB` markers in the `Registry login` step. Recipients = every host age key + the admin key (see [`/.sops.yaml`](../.sops.yaml)). Decryption happens on the target host using its `/etc/charlie/age.key`.
### Encrypt / paste recipe
On the admin laptop, in this repo:
```bash
mkdir -p secrets
tmp=$(mktemp) && chmod 600 "$tmp"
printf 'REGISTRY_USER=sys-gitea-runner\nREGISTRY_PAT=<ro-pat>\n' > "$tmp"
sops --encrypt --input-type dotenv --output-type yaml \
--filename-override secrets/registry-creds.sops.yaml \
"$tmp" > secrets/registry-creds.sops.yaml
shred -u "$tmp"
```
Open the resulting `secrets/registry-creds.sops.yaml`, copy its full contents, and paste it into `.gitea/workflows/deploy-stack.yml` replacing the `REPLACE_ME_WITH_ENCRYPTED_YAML` line (preserve the surrounding comment markers). The `secrets/registry-creds.sops.yaml` file itself is not committed (it's a scratch artifact) — only the inlined paste is the source of truth. `git commit -am 'Rotate registry RO PAT' && git push`.
### Rotation
Regenerate the PAT in Gitea (`sys-gitea-runner` → Settings → Applications → tokens, `read:package` only), repeat the recipe, repaste, push. Next deploy on every host picks it up automatically.
### Adding a host
When a new host gets an age key during onboarding, add its pubkey to the `creation_rules` block in [`/.sops.yaml`](../.sops.yaml), then `sops updatekeys secrets/registry-creds.sops.yaml` (regenerate locally first via the recipe above), repaste, push.
## What this design intentionally doesn't do
- Manage secrets for stacks that aren't runner-deployed.
- Replace vaultwarden's human-facing UX. Vaultwarden stays for browser autofill + secure notes (including the admin age key backup).
- Solve secrets for [`Charlie/gitea`](https://gitea.alexandru.macocian.me/Charlie/gitea) auto-deploy — gitea-on-melina is deployed manually (it bootstraps the runners that would deploy it). SOPS is offline crypto though, so its secrets *can* live encrypted in the repo and a manual `sops -d | docker compose up -d` works.
Registry and repo credentials are separate on purpose.
docs/strategy.md
+31 -130
View File
@@ -1,139 +1,40 @@
# Strategy
## Pattern
GitOps via Gitea Actions runners. The runner is stateless and SSHes into the target host using a short-lived step-ca-issued cert. The job is a thin caller of the [shared `deploy-stack.yml` reusable workflow](../.gitea/workflows/deploy-stack.yml) — see [automation.md](automation.md).
```
dev box ── git push ──► Gitea (Charlie/<stack>) ── webhook ──► any runner
│
│ ssh sys-gitea-runner@<host>
│ (step-ca cert, 5min)
▼
on the target host:
git fetch && git reset --hard origin/main
sops --decrypt … (if `decrypt:` set)
docker compose pull && docker compose up -d
rm decrypted plaintext
```mermaid
flowchart LR
Dev[Operator push] --> Gitea
Gitea --> Runner[Any runner]
Runner -->|SSH cert| Host[Target host]
Host --> Stack["/mnt/nas/stacks"]
Host --> Data["/mnt/nas/data"]
```
The git working tree at `/mnt/nas/stacks/<stack>/` **is** the deployment. Runtime state lives separately under `/mnt/nas/data/<stack>/` (see [layout.md](layout.md)).
| Area | Current state |
| ------------------- | -------------------------------------------------------------------- |
| Deployment model | GitOps through Gitea Actions and shared reusable workflows. |
| Runner model | Stateless runners; target work happens over SSH on the host. |
| Deployment identity | `sys-gitea-runner` with short-lived step-ca SSH certs. |
| Stack sync | Target working tree is fetched, reset, cleaned, decrypted, composed. |
| Drift stance | Host edits are overwritten by repo state. |
| Data stance | Persistent state lives outside Git. |
## Rules
## Stack types
- One owner host per stack (singleton stacks). The host named in the workflow's `host:` input is where the runner SSHes.
- Shared stacks (run on multiple hosts — see [Shared stacks](#shared-stacks)) follow the same NFS-shared-tree model but each host runs its own `docker compose up -d` against host-specific overrides.
- No hand-edits on the host once a stack is repo-managed. The deploy workflow does `git fetch && git reset --hard`, which clobbers any drift on purpose.
- Runtime state never enters git. Compose files bind into `/mnt/nas/data/<stack>/`, not into the working tree.
- `.env` ignored, `.env.example` tracked. Plaintext secrets stay out — encrypted form lives under `secrets/`.
- Image tags pinned. No `:latest`.
- Any runner can deploy any stack to any host. Routing-by-runner-label is intentionally avoided (Gitea bug; see [automation.md → Gitea quirks](automation.md#gitea-quirks-leave-them-here-for-the-next-time-this-bites)). The runner SSHes to wherever `host:` points. Exception: [`Charlie/runners`](runners.md) needs precise routing for cross-deploy.
| Type | Shape |
| ----------------- | -------------------------------------------------------------------- |
| Singleton stack | One owner host named in workflow `host`. |
| Shared stack | Same repo deployed to multiple hosts. |
| Source/image repo | Builds images through `build-image.yml`; stack repo pins the result. |
| Operator tool | Runs on the operator workstation; not deployed as a stack. |
## Repo types in use
## Shared-stack resolution
- **Owned stacks** — the working tree under `Charlie/<stack>/` is the source of truth. `docker-compose.yml`, `secrets/`, etc.
- **Pull-mirrors** — read-only copy of an upstream we want offline (or as a reference for forks we maintain). Configured via Gitea's pull-mirror feature. Naming: `Charlie/<name>-mirror`.
| File | Meaning |
| --------------------------- | ------------------------------- |
| `docker-compose.yml` | Default stack definition. |
| `docker-compose.<host>.yml` | Full host-specific replacement. |
| `overrides/<host>.yml` | Host-specific additive overlay. |
| `envs/<host>.env` | Host-specific environment file. |
## Shared stacks
Some stacks run on **multiple hosts** with the same image but slightly different configuration (e.g. `otelcollector`, `portainer-agent`). Pattern borrowed from Ansible inventories + Kustomize overlays + Kubernetes DaemonSets:
```
Charlie/<stack>/
├── docker-compose.yml # the "what" — default
├── docker-compose.<host>.yml # full per-host compose (when delta is structural)
├── envs/
│ ├── morgott.env # per-host env (Ansible host_vars equivalent)
│ └── melina.env
├── overrides/
│ └── <host>.yml # additive overlay (when env can't express it)
├── secrets/ # SOPS-encrypted; decrypted at deploy
│ └── env.sops.yaml # (or per-host: <host>.env.sops.yaml)
├── .sops.yaml
├── .gitea/workflows/deploy.yaml
└── README.md
```
### Per-host resolution order
Applied identically by [`deploy-stack.yml`](../.gitea/workflows/deploy-stack.yml) and the [manual fallback recipe](automation.md#manual-fallback-no-runner-available):
1. If `docker-compose.<host>.yml` exists → use **only** that.
2. Else `docker-compose.yml` + (if present) `overrides/<host>.yml` layered on top.
3. If `envs/<host>.env` exists → loaded with `--env-file`.
Hostname is lowercased (`hostname -s | tr '[:upper:]' '[:lower:]'`).
### When to use which
| Delta type | Use |
|---|---|
| Different scalar value (port, env var, URL) | `envs/<host>.env` |
| Add an extra service or scalar field | `overrides/<host>.yml` (compose merges additively) |
| **Replace** a list (volumes, command, networks) | `docker-compose.<host>.yml` (full file) |
| Different image | `docker-compose.<host>.yml` |
Compose's `!override` and `!reset` YAML tags exist for list-replacement, but support is uneven across compose versions (notably broken on Synology DSM's bundled compose). Prefer the full-file approach when replacing lists.
### Multi-host workflow shape
Despite [strategy.md → shared-stack pattern](#shared-stacks) supporting matrix-style fan-out in principle, **don't use `strategy.matrix`** for shared stacks today. Gitea Actions doesn't reliably honor `strategy.max-parallel: 1`, so two matrix jobs end up racing on the NFS-shared `.git` and (for SOPS stacks) the `.env` plaintext. Use `needs:` for true serialization instead:
```yaml
jobs:
deploy-morgott:
uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/workflows/deploy-stack.yml@main
with:
host: morgott
decrypt: |
secrets/env.sops.yaml -> .env
secrets: inherit
deploy-melina:
needs: deploy-morgott
uses: https://sys-gitea-runner:${{ secrets.RUNNER_REPO_PAT }}@gitea.alexandru.macocian.me/Charlie/project-charlie/.gitea/workflows/deploy-stack.yml@main
with:
host: melina
decrypt: |
secrets/env.sops.yaml -> .env
secrets: inherit
```
Background: [automation.md → `strategy.max-parallel: 1` doesn't actually serialize](automation.md#strategymax-parallel-1-doesnt-actually-serialize). Long-term fix is a per-host decrypt path in `deploy-stack.yml`, after which matrix becomes safe again.
### Mental model
- "What" = `docker-compose.yml` (or per-host file).
- "Where" = the host named in the workflow's `host:` input.
- Per-host vars = `envs/<host>.env`.
- Per-host structural deltas = `overrides/<host>.yml` for additive, `docker-compose.<host>.yml` for replacement.
- Group defaults = compose env-var defaults (`${VAR:-default}`).
## Secrets
SOPS-in-repo with one age key per host. The reusable workflow SSHes into the target host as `sys-gitea-runner` and runs `sops --decrypt` there using the host's `/etc/charlie/age.key`. Plaintext stays on the target host (mode 0600), gets cleaned up by an `if: always()` step. See [secrets.md](secrets.md).
## Stack-emitted metrics (textfile pattern)
Stacks that produce their own metrics — anything not already covered by the otelcol receivers (`hostmetrics`, `docker_stats`, `prometheus`) — publish them as **Prometheus textfiles** in a well-known host-local directory, and let [`Charlie/otelcollector`](https://gitea.alexandru.macocian.me/Charlie/otelcollector)'s node-exporter pick them up via `--collector.textfile`.
Convention:
```
/var/lib/charlie/<stack>/metrics/<name>.prom
```
- The producing stack bind-mounts the dir **read-write** and writes atomically (build to a temp file in the same dir, then `mv`). Atomic mv prevents node-exporter ever scraping a half-written file.
- `Charlie/otelcollector` bind-mounts the parent (`/var/lib/charlie/<stack>/metrics`) **read-only** into node-exporter and runs with `--collector.textfile.directory=…`.
- The metric name(s) must match `Charlie/otelcollector`'s `prometheus` receiver `metric_relabel_configs` keep-regex (currently `node_hwmon_.*|node_rapl_.*|node_textfile_.*|trivy_.*`). Extend the regex when adding a new family.
- Host attribution comes for free from otelcol's `resourcedetection` processor (`host.name`); textfiles **do not bake `host` into label sets**.
First user of this pattern: [`Charlie/trivy-server`](https://gitea.alexandru.macocian.me/Charlie/trivy-server) + [`Charlie/trivy-runner`](https://gitea.alexandru.macocian.me/Charlie/trivy-runner) emit `trivy_*.prom`. The pattern generalises — future small exporters (cert-expiry, backup-age, drift-count, etc.) should reuse it instead of standing up their own scrape endpoints.
## Why not Portainer/Komodo/Dockge
- **Portainer Stacks-from-Git**: works, but no submodule support, opinionated about working-tree location, adds a control plane on top of git.
- **Komodo**: nice, also pull-based, also relocates working trees. Reconsider if we ever want a UI.
- **Dockge**: file mgmt only, no git wiring.
The runner-driven pattern keeps us in plain `git` + `docker compose` and adds nothing the team doesn't already understand.
Shared stacks are serialized when they touch the same NFS working tree or
plaintext secret path.
docs/topology.md
+25 -35
View File
@@ -1,42 +1,32 @@
# Topology
```mermaid
graph TD
Internet((Internet))
Radagon{Radagon}
Devices{{Local Devices}}
Marika{Marika}
Godfrey[[Godfrey]]
Morgott[Morgott]
Melina[Melina]
Mohg[Mohg]
Malenia[Malenia]
Miquella[Miquella]
Internet <--> Radagon
Radagon <-->|80/443| Marika
subgraph Extranet
Radagon <--> Devices
end
subgraph Intranet
Marika <--> Godfrey
Godfrey <-->|80/443| Morgott
Melina --> Godfrey
Mohg --> Godfrey
Malenia --> Godfrey
Miquella --> Godfrey
Morgott --> Melina
Morgott --> Mohg
Morgott --> Malenia
Morgott --> Miquella
Melina --> Miquella
Malenia --> Miquella
Mohg --> Miquella
end
flowchart LR
Internet --> Radagon --> Marika --> Godfrey
Godfrey --> Morgott
Godfrey --> Melina
Godfrey --> Mohg
Godfrey --> Malenia
Godfrey --> Miquella
Morgott -->|Caddy| PublicApps[Public apps]
Miquella -->|NFS /mnt/nas| AppHosts[App hosts]
```
## Ingress path
| Layer | Components |
| ---------------- | ------------------------------ |
| WAN | Internet -> Radagon |
| Intranet gateway | Marika |
| Switching | Godfrey |
| Edge ingress | Morgott / Caddy |
| App hosts | Morgott, Melina, Mohg, Malenia |
| Storage | Miquella / `/mnt/nas` |
`Internet → Radagon → Marika → Morgott:caddy → container (local or sibling host over Godfrey)`.
| Path | State |
| -------------- | ----------------------------------------------- |
| Public HTTP(S) | Internet -> Radagon -> Marika -> Morgott:Caddy |
| Internal DNS | Marika serves `.lan` names |
| Stack data | Hosts mount Miquella at `/mnt/nas` |
| Metrics | otelcollector -> VictoriaMetrics -> Grafana |
| CI deploys | Gitea runner -> step-ca SSH cert -> target host |
All `*.alexandru.macocian.me` traffic terminates at Caddy on Morgott.
All `*.alexandru.macocian.me` public traffic terminates at Caddy on Morgott.
docs/vulnerability-scanning.md
+35 -174
View File
@@ -1,183 +1,44 @@
# Vulnerability scanning
Daily CVE sweep of every container image in the fleet. Two stacks
emit Prometheus metrics; node-exporter ships them through the existing
otelcollector → VictoriaMetrics → Grafana pipeline.
## Stacks
| Stack | Host | Scans |
|---|---|---|
| [`Charlie/trivy-server`](https://gitea.alexandru.macocian.me/Charlie/trivy-server) | morgott | Long-running `trivy server` (shared vuln-DB cache on `morgott.lan:4954`, LAN-only) + daily 03:00 sweep of every `image:` ref declared in any `/mnt/nas/stacks/*/docker-compose*.yml` (incl. per-host overrides + `x-*` anchors). |
| [`Charlie/trivy-runner`](https://gitea.alexandru.macocian.me/Charlie/trivy-runner) | morgott + melina | Daily sweep of currently-running images (`docker ps`). Stagger: morgott 03:00, melina 03:15. |
Both stacks talk to the same trivy server on `morgott.lan:4954`, so
the ~1GB vuln DB is downloaded once, not per-host. No public DNS, no
auth — morgott + melina are on the secure intranet.
## Architecture
```
┌─────────────────────────────────────────┐
│ trivy-server (morgott) │
│ │
│ trivy-server ◄── shared vuln DB │
│ ▲ cache (1GB) │
│ │ │
│ compose-scanner ──► trivy_compose.prom
│ ▲ │
│ yq-extractor ──── walks /mnt/nas/stacks/*/
│ ▲ │
│ ofelia (cron, 03:00) │
└───────┼─────────────────────────────────┘
│
│ http://morgott.lan:4954 (LAN, no auth)
│
┌────────────┼────────────────┐
│ │ │
▼ ▼ ▼
┌──────────┐ ┌──────────┐ morgott + melina
│ trivy- │ │ trivy- │ (per host)
│ runner │ │ runner │
│ morgott │ │ melina │ each:
│ │ │ │ docker-helper ──► images.txt
│ │ │ │ scanner ──► trivy_running.prom
│ │ │ │ ofelia (03:00 / 03:15)
└──────────┘ └──────────┘
│ │
└─────┬──────┘
▼
/var/lib/charlie/trivy/metrics/
trivy_compose.prom (morgott only)
trivy_running.prom (per host)
│
▼
node-exporter --collector.textfile (in Charlie/otelcollector)
│
▼
otelcol prometheus receiver (keep regex includes `trivy_.*`)
│
▼
OTLP → VictoriaMetrics → Grafana
```mermaid
flowchart LR
Compose[Compose image refs] --> Server[trivy-server]
Running[Running containers] --> Runner[trivy-runner]
Server --> Textfiles[Prometheus textfiles]
Runner --> Textfiles
Textfiles --> OTel[otelcollector]
OTel --> VM[VictoriaMetrics]
VM --> Grafana
```
| Stack | Host(s) | Scans |
| ---------------------- | --------------- | ----------------------------------------------- |
| `Charlie/trivy-server` | morgott | Compose-declared image refs across stack repos. |
| `Charlie/trivy-runner` | morgott, melina | Currently running container images. |
## Pipeline
| Stage | State |
| --------- | --------------------------------------------------------------------- |
| Trivy DB | Shared server on morgott. |
| Schedule | Daily sweeps. |
| Metrics | Prometheus textfiles under `/var/lib/charlie/trivy/metrics`. |
| Shipping | node-exporter textfile collector -> otelcollector -> VictoriaMetrics. |
| Dashboard | Grafana `Container CVEs`. |
## Metrics
```
trivy_image_vulnerabilities{image="…", severity="CRITICAL|HIGH|MEDIUM|LOW|UNKNOWN", scanner="compose|running"} <count>
trivy_image_scan_timestamp_seconds{image="…", scanner="…"} <unix_time>
trivy_image_scan_errors_total{image="…", scanner="…"} 0|1
```
| Metric | Meaning |
| ------------------------------------ | ----------------------------------------------- |
| `trivy_image_vulnerabilities` | Vulnerability counts by image/severity/scanner. |
| `trivy_image_scan_timestamp_seconds` | Last scan timestamp. |
| `trivy_image_scan_errors_total` | Scan error state. |
Notes:
- `host_name` is added by otelcol's `resourcedetection` processor — not
baked into the textfile.
- `scanner="compose"` = images **declared** in compose files (canonical,
catches stuff pulled but not running). `scanner="running"` = images
returned by `docker ps`. Their difference is interesting:
- `compose - running` = pulled or declared but not running.
- `running - compose` = drift / hand-started containers.
- The vuln count is **package occurrences**, not unique CVE IDs. A CVE
affecting N packages in the same image counts N times. Decision was
to keep it that way; trivy's per-package output reflects the real
patch surface.
## Access
## Pipeline behaviour
| Identity | Purpose |
| ------------------- | --------------------------------------------- |
| `sys-trivy-scanner` | Read-only registry access for private images. |
- **Cadence**: daily at 03:00 (server compose-sweep), 03:00 (morgott
running-sweep), 03:15 (melina running-sweep). Schedules live in each
stack's `overrides/<host>.yml` as docker labels (ofelia
label-discovery mode — `--docker`).
- **DB refresh**: trivy itself refreshes its vuln DB every 6h. Counts
legitimately drift between sweeps as new CVEs are disclosed against
unchanged images.
- **Atomic writes**: each scanner writes to a temp file in the same
directory then `mv`s — node-exporter never sees a half-written file.
## Dashboard
`Container CVEs` (uid `charlie-containers-security-cves`) under
`containers/security/`. Selectors: host, scanner (`running` default),
severity (`CRITICAL+HIGH` default).
Panels: top-N vulnerable images, severity breakdown table, CVE count
over time, scan-error count, stalest-scan stat (catches stuck sweeps).
## Auth
Trivy needs creds to pull private images from the Gitea registry
(Charlie/* + amacocian/* are private). Both stacks ship a
`secrets/env.sops.yaml` with:
```
TRIVY_USERNAME=sys-trivy-scanner
TRIVY_PASSWORD=<gitea-PAT-with-read:package>
```
`sys-trivy-scanner` is an LDAP-managed service account
([Charlie/ldap](https://gitea.alexandru.macocian.me/Charlie/ldap))
with the **least** privilege needed:
- Member of the Charlie `Runners` team (registry read access for org
packages).
- Collaborator on each `amacocian/*` repo that emits a private
package.
- PAT scope: `read:package` only.
Rotate by issuing a new PAT in Gitea and re-encrypting both
`secrets/env.sops.yaml` files.
## Manual operations
Trigger a sweep ad-hoc (skips ofelia, runs the script directly):
```bash
# Compose-sweep (morgott only):
docker exec trivy-server-yq sh /srv/scripts/extract-images.sh
docker exec trivy-server-scanner sh /srv/scripts/scan-compose.sh
# Running-sweep (per host):
docker exec trivy-runner-helper sh /srv/scripts/list-running-images.sh
docker exec trivy-runner-scanner sh /srv/scripts/scan-running.sh
# Inspect output (host-side):
cat /var/lib/charlie/trivy/metrics/trivy_running.prom
cat /var/lib/charlie/trivy/metrics/trivy_compose.prom
# In Grafana Explore:
trivy_image_vulnerabilities
{__name__=~"trivy.*"}
```
`ofelia` 0.3.x has no `run-job` subcommand (added in 0.4); always use
the direct `docker exec` path above.
## Open follow-ups
- **Alerting**: vmalert rule on `trivy_image_vulnerabilities{severity="CRITICAL"} > 0`
routed via the existing alertmanager. Deferred until a stable
baseline.
- **CI build-time scan**: run `trivy image` inside the eventual
`amacocian/*` build pipelines (see [roadmap.md → Build pipelines](roadmap.md))
so HIGH/CRITICAL issues fail the build instead of waiting for the
next nightly sweep.
- **`.trivyignore`** / accepted-CVE story for noisy false positives.
## Triage workflow
When a Grafana alert (or a manual dashboard check) flags an image:
1. Note the image ref and the scanner label (`compose` vs `running`).
2. Run a one-off scan inside the appropriate stack's scanner container
with `--severity HIGH,CRITICAL` to see the per-CVE detail:
```bash
docker exec trivy-runner-scanner trivy image \
--server "$TRIVY_SERVER" \
--severity HIGH,CRITICAL \
<image-ref>
```
3. Decide per CVE: bump base image / wait for upstream fix / suppress
via `.trivyignore` (justified). The stack repo (`Charlie/<stack>`)
is the canonical place to pin the fix.
4. Re-run the sweep manually (commands above) to confirm green before
waiting for cron.
Alerting and build-time scan policy belong in
[next work](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/roadmap.md).