sherlock/plan.md

# Charlie Copilot "Sherlock" — research + plan

## Problem restatement

Make Copilot CLI a first-class operator on the Charlie homelab:

- Repos on **Gitea** (`gitea.alexandru.macocian.me`).
- Hosts on Ubuntu, **no direct SSH** — only via `~/Dev/gssh` (OIDC-authenticated WebSocket SSH proxy, ephemeral certs signed by the in-house CA).
- Identity is **Authentik** (`id.alexandru.macocian.me`) — OIDC/OAuth2/SAML.
- Want Copilot to call into Gitea, Grafana, ssh (via gssh), and "anything else" without me pasting tokens.
- Some downstream services use their own OAuth (not federated via Authentik) → need a way to do a one-off browser dance and cache that token too.

The desired "sherlock" sits between Copilot CLI and N service-specific MCPs, handling auth once and injecting per-service credentials.

---

## Research findings

### 1. What Copilot CLI already supports (verified locally)

Looking at `~/.copilot/`:

- `/mcp` slash command manages MCP servers.
- `~/.copilot/mcp-oauth-config/` shows Copilot CLI **already implements the full MCP 2025-06-18 authorization spec for HTTP servers**: OAuth 2.1 + PKCE, RFC 9728 protected-resource metadata discovery, RFC 7591 dynamic client registration. Sample entry:

  ```json
  {
    "serverUrl": "https://icm-mcp-prod.azure-api.net/v1/",
    "authorizationServerUrl": "https://login.microsoftonline.com/.../v2.0",
    "clientId": "aebc6443-...",
    "redirectUri": "http://127.0.0.1:34573/",
    ...
  }
  ```

  → Copilot opens a loopback redirect, runs the browser flow itself, caches tokens.

- STDIO MCP servers — per spec — **read credentials from environment variables** (no in-band OAuth). This is the natural injection point for the sherlock.

**Implication:** we don't need to reimplement OAuth inside the MCPs we build. We have two clean options per MCP:

| Transport | How auth happens |
|---|---|
| HTTP MCP | Copilot does OAuth 2.1 against the MCP itself (Authentik = AS, the MCP = resource server). |
| STDIO MCP | The sherlock spawns it and injects a per-service token via `env`. |

### 2. MCP authorization spec (2025-06-18) — the relevant rules

- HTTP MCPs **MUST** implement RFC 9728 Protected Resource Metadata; clients **MUST** discover the AS via `WWW-Authenticate` on a 401 or `/.well-known/oauth-protected-resource`.
- Authorization servers **MUST** implement OAuth 2.1 + PKCE; **SHOULD** support RFC 7591 dynamic client registration.
- STDIO transports **SHOULD NOT** do in-band auth — read from env.
- Tokens **MUST** be audience-bound (the MCP cannot reuse a Copilot-issued token to call downstream APIs blindly — that's the "confused deputy" the spec calls out).

### 3. Industry / prior art for "credential brokers for AI agents"

| Source | Pattern | Relevance |
|---|---|---|
| **IETF draft "Credential Broker for Agents" (CB4A)** | A broker issues short-lived, narrowly-scoped proxy credentials to agents; agents never hold long-lived OAuth. Policy decision is separated from credential delivery. Builds on SPIFFE concepts. | Exact match for what we want. |
| **TrueFoundry "OAuth 2.0 for MCP" architecture** | Splits concerns into (a) inbound auth ("who is calling the MCP?"), (b) access control ("what tools can they use?"), (c) outbound auth ("which downstream OAuth token to inject?"). Per-(user, server) grants in a vault with atomic refresh. | Concrete blueprint for the sherlock daemon. |
| **HashiCorp Vault** | Long-standing pattern: brokers short-lived dynamic creds for cloud IAM, DBs, SSH (Vault SSH CA). | Vault SSH CA matches gssh's ephemeral-cert approach. We could lean on Vault or roll an Authentik-native equivalent. |
| **1Password `op run` / Connect** | Wraps a child process and injects secrets as env vars from a vault — the simplest possible "broker". | Mental model for the `sherlock copilot` wrapper. |
| **`sops` + `age`** | Already used in project-charlie for secrets at rest. | Good for bootstrapping the broker's own client-credentials, not for live user tokens. |
| **RFC 8693 OAuth Token Exchange** | The standardized way to swap one token for another (e.g. Authentik ID token → service-scoped access token). | This is the kernel of the sherlock's "exchange" tool. Authentik supports token exchange via its OAuth2 provider. |

### 4. What we already have in Charlie that maps onto this

- **Authentik** = the AS for the whole sherlock. Already federates Gitea, Grafana, Caddy-protected apps (per `docs/identity.md`).
- **gssh** = a working example of "OIDC → ephemeral cert → proxy to host". This is the model for the SSH MCP: the MCP doesn't hold SSH keys, it asks gssh for a short-lived cert each call.
- **sops/age per host** = how the broker daemon could bootstrap its own client_secret.
- **Caddy + caddy-security** = ready-made reverse proxy with OIDC if we want the sherlock to be an HTTP MCP behind `sherlock.alexandru.macocian.me`.

---

## Architectural options

### Option A — "Wrapper only" (simplest)

A `sherlock` shell command that:

1. Ensures the user has a fresh Authentik token (browser flow, cached in `~/.config/sherlock/token`).
2. For each configured service, does a token exchange to mint a service-scoped token.
3. Writes a per-session MCP config to `$XDG_RUNTIME_DIR/sherlock/mcp.json` listing STDIO MCPs with the right `env` block.
4. `exec`s `copilot --mcp-config …`.

Pros: ~200 lines of code, no daemon, no new long-running service.
Cons: every Copilot launch re-exchanges tokens; no shared cache across terminals; one-off non-Authentik OAuth flows are awkward (no place to host the callback persistently).

### Option B — "Sherlock MCP" (single HTTP MCP)

One HTTP MCP server `sherlock-mcp`, hosted behind Caddy. Copilot does OAuth 2.1 against it natively (using the machinery we already saw in `~/.copilot/mcp-oauth-config/`). The sherlock exposes tools like `gitea.list_repos`, `grafana.query`, `ssh.run(host, cmd)`, `oauth.consent(service)`.

Pros: single integration point in Copilot; works from any machine; reuses Copilot's OAuth client.
Cons: every new tool is a code change inside the sherlock; tool surface gets fat; we lose the modularity of having separate MCPs per service.

### Option C — Hybrid: broker daemon + thin stdio MCPs + tiny wrapper *(recommended)*

```
┌────────────────────────────────────────────────────────────┐
│  user                                                      │
│    │                                                       │
│    │ `sherlock copilot`                                    │
│    ▼                                                       │
│  ┌─────────────┐  spawns, injects env  ┌──────────────┐    │
│  │ copilot CLI │ ────────────────────▶ │ gitea-mcp    │    │
│  │             │                       │ grafana-mcp  │    │
│  │             │                       │ gssh-mcp     │    │
│  └─────┬───────┘                       └──────┬───────┘    │
│        │ unix socket                          │            │
│        ▼                                      ▼            │
│  ┌──────────────────────────────────────────────────┐      │
│  │ sherlock-broker (systemd --user daemon)          │      │
│  │  - owns the Authentik OAuth state                │      │
│  │  - per-service token cache + refresh             │      │
│  │  - RFC 8693 token exchange                       │      │
│  │  - hosts loopback HTTP for browser callbacks     │      │
│  │  - exposes sherlock-mcp (stdio) for consent flow │      │
│  └────────────────────┬─────────────────────────────┘      │
│                       │ OIDC / token exchange              │
│                       ▼                                    │
│                 Authentik (id.alexandru.macocian.me)       │
└────────────────────────────────────────────────────────────┘
```

Pieces:

1. **`sherlock-broker`** — long-running per-user daemon (systemd `--user` unit).
   - Listens on a Unix socket (`$XDG_RUNTIME_DIR/sherlock.sock`).
   - Owns the Authentik OAuth client (PKCE; loopback redirect on a stable port).
   - For each registered service, knows: AS, scopes, exchange mode (federated-via-Authentik | own-oauth | static-pat).
   - Performs RFC 8693 token exchange against Authentik for downstream service tokens.
   - For non-Authentik services, runs the standard auth-code+PKCE dance in a browser and caches the result.
   - Atomic refresh with single-flight to avoid 401 storms.
   - Exposes a tiny RPC: `get_token(service, scopes) → { token, expires_at }`.

2. **Service MCPs** — one stdio binary per service (`gitea-mcp`, `grafana-mcp`, `gssh-mcp`, …).
   - Each is *thin*: it just wraps the service's API and reads `XXX_TOKEN` from env on startup *and* refreshes via the broker socket when a 401 comes back.
   - `gssh-mcp` shells out to `~/Dev/gssh` with a short-lived OIDC token from the broker — mirrors how `gssh` already works.

3. **`sherlock-mcp`** — a *stdio* MCP exposed to Copilot just for the human-in-the-loop stuff:
   - `oauth.consent(service)` → opens a browser, waits for the broker to receive the callback, returns success/failure.
   - `auth.whoami()`, `auth.list_services()`, `auth.revoke(service)`.
   - This is the only MCP that needs UI interaction; the others stay fully non-interactive.

4. **`sherlock` CLI** — the wrapper.
   - `sherlock login` — kick off the Authentik flow up front.
   - `sherlock status` — show cached tokens / expiry / registered services.
   - `sherlock copilot [...]` — ensure the broker is up and logged in, render a per-session MCP config, `exec copilot "$@"`.
   - `sherlock add-service <name>` — register a new service (writes a small TOML/YAML entry that the broker hot-reloads).

Pros: each MCP stays small and replaceable; tokens are cached *once* and shared across terminals; non-Authentik OAuth flows have a stable home (the broker's loopback listener); matches CB4A / TrueFoundry guidance; mirrors the gssh pattern already proven in the homelab.
Cons: more moving parts than A; needs a systemd unit and a stable socket path.

---

## Recommendation

Go with **Option C**, but build it in slices so we have something usable after each slice.

## Phased plan

Sliced finer than the original draft so the first end-to-end demo (login + wrapper) lands before any service MCP work.

### Phase 0 — bootstrap

- Create `Charlie/sherlock` repo on Gitea; clone to `~/Dev/charlie/sherlock/`.
- Sherlock-flavoured scaffold (NOT the per-app stack scaffold — sherlock is operator-side tooling, no `docker-compose.yml`, no `deploy-stack.yml` caller):
  - `README.md` (TOC only), `.gitignore` (Go), `go.mod` (`gitea.alexandru.macocian.me/Charlie/sherlock`, Go 1.23).
  - Directory skeleton `cmd/{sherlock,sherlock-broker}/`, `internal/{broker,authn,agent,agent/profiles,config,socket,mcp}/`.
  - Minimal `cmd/sherlock/main.go` with `version` subcommand so CI has something to build.
  - `.gitea/workflows/ci.yaml` — `go vet` + `go test -race` + `go build`.
- `docs/` with one file per topic: `architecture.md`, `auth-model.md`, `service-registry.md`, `agent-profiles.md`, `gssh-integration.md`, `conventions.md`. Service-registry and agent-profile TOML schemas are documented only — loaders land in Phase 1.
- Built-in Copilot agent profile under `internal/agent/profiles/copilot.toml` (embed-only; not yet read by code).
- Add a row to `project-charlie/docs/inventory.md` (new "Operator tooling" section) and link from `project-charlie/README.md`.

### Phase 1 — login broker + agent-agnostic wrapper (no MCPs yet)

Smallest slice that proves the auth path and the wrapper indirection end-to-end. No per-service MCP work in this phase.

- `sherlock-broker`:
  - Authentik PKCE flow with loopback redirect on a stable port.
  - Persisted token cache (mechanism — `age`-encrypted blob vs OS keyring — decided at the top of this phase).
  - Unix-socket RPC at `$XDG_RUNTIME_DIR/sherlock.sock`: `get_id_token()`, `whoami()`, `status()`.
  - Single-flight refresh.
- `sherlock` CLI:
  - `sherlock login` / `sherlock logout` / `sherlock status`.
  - `sherlock run <agent> [args…]` — loads the agent profile, renders the per-session MCP config (empty in Phase 1), `exec`s the agent's binary with the right argv + env.
  - `sherlock copilot [args…]` — sugar for `sherlock run copilot`.
- Agent-profile loader in `internal/agent/`. Built-in `copilot.toml` is the only profile shipped; user overrides under `~/.config/sherlock/agents.d/` are picked up. A worked Claude Code profile in `docs/agent-profiles.md` proves the design is extensible.

**Exit criterion:** `sherlock login` then `sherlock copilot` execs copilot with a (possibly empty) MCP config. `sherlock status` shows a valid JWT. Dropping a `claude.toml` under `agents.d/` makes `sherlock claude` start working with no rebuild.

### Phase 2 — Gitea MCP (first real service)

- `gitea-mcp` (stdio): `list_repos`, `get_file`, `list_issues`, `create_issue`, `search_code`.
- Broker performs RFC 8693 token exchange against Authentik (or passthrough of the ID token if audiences align).
- The wrapper renders an MCP config listing `gitea-mcp` with the right `env`.

**Exit criterion:** `sherlock copilot` then `> list my charlie repos` works, no PAT pasted anywhere. Per-user audit trail in Gitea logs.

### Phase 3 — gssh MCP

Reuses the existing gssh server (`gssh.alexandru.macocian.me`) — sherlock does NOT shell out to a local client and does NOT mint SSH certs. Full design in [docs/gssh-integration.md](docs/gssh-integration.md).

- `gssh-mcp`: thin Go HTTP+WebSocket client to gssh's `/api/v1/session/*` endpoints, authenticated with an Authentik JWT from the broker.
- Tools: `ssh.run(host, command, timeout)`, `ssh.list_hosts()`, later `ssh.put_file(host, path, content)`.
- Per-host policy is already enforced server-side by gssh's `SessionService` against JWT claims — sherlock does not duplicate it.
- Completion sentinel strategy: `command; echo __SHERLOCK_DONE__$?` in the WebSocket shell stream, parsed client-side.

**Exit criterion:** `> on melina, show me docker ps` works and the SSH session is logged centrally by gssh.

### Phase 4 — Grafana MCP + `sherlock-mcp` consent server

- `grafana-mcp` with `query`, `list_dashboards`, `get_dashboard`. Auth via Authentik OIDC (Grafana already federates).
- `sherlock-mcp` (stdio) with `oauth.consent`, `auth.whoami`, `auth.list_services`, `auth.revoke`.
- First "non-Authentik" service (something with its own OAuth — e.g. GitHub.com itself, or a SaaS) registered to exercise the consent path end-to-end.

### Phase 5 — polish

- `systemd --user` units for the broker.
- `sherlock add-service` scaffolder.
- Audit log of every tool call + which token was used (ship to the existing OTEL collector at `Charlie/otelcollector`).
- Per-service scope minimisation; refuse to issue write-scoped tokens unless the wrapper was launched with `--write`.
- Docs page `docs/sherlock.md` in project-charlie, linked from the README.

---

## Decided

- **Name:** `sherlock`. (Replaces "sherlock" everywhere — `sherlock-broker`, `sherlock-mcp`, `sherlock copilot …`, etc.)
- **Repo location:** new repo `Charlie/sherlock`, cloned to `~/Dev/charlie/sherlock/` (sibling of the existing per-app stack repos like `gitea/`, `runners/`, etc.). Added to `docs/inventory.md` under a new "Operator tooling" section and linked from `project-charlie/README.md`.
- **Language:** Go. Single static binaries per `cmd/<binary>/`; best-in-class OAuth/OIDC libs (`golang.org/x/oauth2`, `coreos/go-oidc`); official MCP Go SDK; native UDS + systemd integration.
- **Phase 2 Gitea auth model:** broker mints an OIDC token *as the operator* via Authentik. Every MCP call shows up under the operator in Gitea's audit log. No dedicated `sherlock` service account.
- **Service-registry format:** TOML, under `~/.config/sherlock/services.d/*.toml`. Hot-reloaded by the broker.
- **SSH backend:** reuse the existing gssh server. `gssh-mcp` is a thin JWT-authenticated HTTP+WS client; no local SSH-cert work in sherlock. See [docs/gssh-integration.md](docs/gssh-integration.md).
- **Wrapper extensibility:** agent-agnostic. Each supported agent CLI (Copilot, Claude Code, Aider, …) is described by a TOML profile under `~/.config/sherlock/agents.d/`, with built-in defaults embedded in `internal/agent/profiles/`. `sherlock copilot` and `sherlock claude` are sugar for `sherlock run copilot` / `sherlock run claude`. Adding a new agent MUST be a TOML drop-in, never a code change.
- **Docs convention:** `README.md` is TOC only; every topic lives as its own file under `docs/`. New concerns get new files, never appended sections.
- **Token persistence (Phase 1):** OS keyring via `zalando/go-keyring`. Strict pre-flight (`keyring.Probe()`) at startup of both `sherlock` and `sherlock-broker`; fail fast with exit code 3 and a per-OS hint if Secret Service / Keychain / Credential Manager is unavailable. See [docs/storage.md](docs/storage.md).
- **RPC framing (Phase 1):** JSON-over-newline on the UDS at `$XDG_RUNTIME_DIR/sherlock.sock`. One JSON object per line, both directions. Debuggable with `socat`/`nc`. See [docs/rpc.md](docs/rpc.md).
- **Broker lifecycle (Phase 1):** forked child process. The wrapper CLI dials the socket; if no one is listening it `fork+exec`s `sherlock-broker daemon` with `setsid`, polls the socket, then proceeds. Per-user PID-file flock prevents double-start; broker auto-exits after `SHERLOCK_BROKER_IDLE` (default 1h). No systemd in Phase 1.
- **Loopback redirect port:** `127.0.0.1:6990` (unassigned per IANA). The actual Authentik OAuth2 provider config is deferred — the broker ships with empty defaults and `login_start` returns `not_configured` until `SHERLOCK_AUTHENTIK_ISSUER` / `SHERLOCK_AUTHENTIK_CLIENT_ID` are set. End-to-end flow is exercised by an integration test against a stub Authentik (`internal/authn/flow_test.go`).

## Still open

Tracked here and revisited at the top of the relevant phase.

1. **Phase 2 — Authentik provider creation:** stand up the actual `sherlock` OAuth2 provider in Authentik and document the env vars in `docs/storage.md` (carry-over from the Phase 1 deferral above).
2. **Phase 3 — gssh completion semantics:** keep the client-side `__SHERLOCK_DONE__` sentinel, or land a server-side one-shot run-and-close mode in gssh.