Jarvis/docs/spec.md

# jarvis — design spec

Status: draft. This document describes the target design for jarvis as an
incident-enrichment service. The [README] documents what exists today; this spec
documents where it is going and why the pieces fit together.

## 1. Vision

When something breaks in the **Charlie** homelab, the first job is not fixing —
it is *understanding*. jarvis is the on-call teammate that does the boring first
pass for you.

An alert fires. Before you have opened a laptop, jarvis has already:

- read the alert and restated it in plain language,
- checked the relevant dashboards, metrics, and logs in Grafana,
- looked at the offending stack's repo, recent deploys, and failed jobs in
  Gitea,
- optionally poked the affected host read-only over the SSH gateway,
- grounded all of it in the Charlie docs, skills, and troubleshooting guides,

and written a short, skimmable triage that lands in your RSS reader as one more
item to read over coffee. You skim it and decide: **act, watch, or ignore**.

jarvis never pages you awake to go spelunking. It hands you a conclusion and its
evidence, and lets *you* stay the decision-maker. It is an enrichment and triage
tool, not an auto-remediation system: it reads, it reasons, it reports. It does
not change anything.

The end state feels like a competent junior SRE who reads every alert, does the
homework, and leaves you a tidy note — except it is instant, tireless, and
already knows the whole estate.

## 2. Product behaviour

### 2.1 Lifecycle

```mermaid
flowchart LR
  VM[vmalert / Grafana] -->|webhook| WH[POST /webhook/grafana]
  WH --> Q[(queue)]
  Q --> AG[copilot agent<br/>headless, RO tools]
  AG --> DB[(SQLite /data)]
  DB --> FE[GET /feed.atom]
  FE -->|Basic auth| MF[miniflux]
```

An alert batch arrives on the authenticated webhook, is queued, and processed
**serially** (one heavyweight agent run at a time). Each alert becomes one feed
entry: the AI triage plus the raw alert for audit. miniflux polls the
authenticated Atom feed and surfaces new entries in the reader.

Only **firing** alerts are enriched. Resolved notifications are acknowledged and
dropped — the value is in the first-pass triage of an active problem, and the
reader already carries the firing entry as history.

The feed is a **rolling daily window**: it serves entries from the current day
and resets each day. miniflux is the durable store — it caches every entry it
has fetched — so jarvis keeps only what a fresh reader needs to catch up, not a
long archive.

### 2.2 Endpoints

| Method | Path               | Auth         | Consumer            |
| ------ | ------------------ | ------------ | ------------------- |
| POST   | `/webhook/grafana` | Bearer token | Grafana contact pt. |
| GET    | `/feed.atom`       | HTTP Basic   | miniflux            |
| GET    | `/healthz`         | none         | healthcheck         |

### 2.3 Feed entry contract

Every entry is Markdown and skimmable in a reader in under a minute. It always
carries enough for a triage decision without opening anything else:

| Field          | Purpose                                                   |
| -------------- | --------------------------------------------------------- |
| Title          | `[status/severity] <summary>` for at-a-glance scanning.   |
| Verdict        | `act` / `watch` / `ignore` — the triage call, up top.     |
| What fired     | One plain-language sentence.                              |
| Findings       | What jarvis actually observed (live metrics, logs, jobs). |
| Likely cause   | Most probable first, tied to concrete config or evidence. |
| Suggested next | The single most useful next step, if any.                 |
| Confidence     | How much to trust the above; what jarvis could not check. |
| Links          | Dashboard, generator, silence URLs from the alert.        |

On diagnostic failure the entry still publishes with the raw alert and the
error, so an outage in jarvis never silently swallows an alert.

## 3. Architecture

jarvis is a single Go service plus a headless agent, in one container.

```mermaid
flowchart TB
  subgraph C[jarvis container]
    S[jarvis server<br/>Go: webhook, queue, store, feed]
    subgraph agent[sherlock copilot -p ...]
      CO[copilot CLI<br/>--yolo, RO]
      G[grafana-mcp]
      H[gssh-mcp]
      GT[gitea-mcp]
    end
    S -->|spawns per alert| agent
    CTX[/context: cloned Charlie repos<br/>skills + TSG + docs/]
    CO -.grounds in.-> CTX
  end
  CO --> G --> GR[(Grafana)]
  CO --> H --> GS[(gssh gateway)]
  CO --> GT --> GI[(Gitea API)]
```

| Component        | Repo                     | Role                                    |
| ---------------- | ------------------------ | --------------------------------------- |
| jarvis service   | [amacocian/jarvis]       | Webhook → queue → store → Atom feed.    |
| diagnostic agent | Copilot CLI + [sherlock] | Headless reasoning with RO tool access. |
| MCP servers      | [sherlock]               | `grafana-mcp`, `gssh-mcp`, `gitea-mcp`. |
| deploy stack     | [Charlie/jarvis]         | Compose, secrets, host pin, config.     |
| context repos    | [Charlie] org            | Cloned into `/context` for grounding.   |

The build change is small: the diagnoser invokes **`sherlock copilot`** instead
of `copilot` directly, so sherlock renders the `.mcp.json` and injects the
service MCPs. jarvis never touches a service token itself.

## 4. Identity and access

### 4.1 One identity, LDAP as source of truth

jarvis is **a single LDAP account** — nothing more. LDAP is the source of truth;
Authentik syncs from it. Every service resolves that one identity through its
own front door:

- **Gitea** logs the account in through Authentik.
- **Grafana** authenticates the account through Authentik.
- **gssh / SSH** resolves the account through LDAP directly.

So one account, defined once in LDAP, is the whole identity surface. Its
permissions are its LDAP group memberships, scoped **read-only**, mirroring what
a human on-call operator may see. jarvis is not a bag of static per-service
tokens; it is a directory principal. Static tokens appear only where a service
cannot speak machine SSO (see §4.4).

### 4.2 No blast radius

On the hosts, passwordless sudo is granted solely by the LDAP `admins` group
(`/etc/sudoers.d/90-ldap-admins`: `%admins ALL=(ALL) NOPASSWD:ALL`). The jarvis
account is **deliberately not in `admins`**, so it can never `sudo` and cannot
mutate a deployment, restart a service, or read a root-owned secret. Combined
with read-only tool sets and the gssh host allow-list, the worst jarvis can do
is look. This is what makes `--yolo` acceptable (§6.4).

### 4.3 The headless problem

sherlock's only login path today is an interactive **PKCE browser flow** on
`127.0.0.1:6990` (see [sherlock auth-model]). A headless container has no
browser and no operator to click. That flow cannot run in jarvis.

The SSO-native fix is the **OAuth2 client-credentials grant**
(machine-to-machine): sherlock presents the jarvis account's credentials
directly to Authentik's token endpoint and receives a JWT — no redirect, no
browser. So the JWT can represent the jarvis LDAP account (and carry its group
claims) rather than an anonymous Authentik service account, the grant uses the
account's app-password form (`client_id` + `username` + app password) rather
than a bare `client_id` + `client_secret`. Grafana's `[auth.jwt]` integration
and the gssh gateway's JWT bearer scheme already validate exactly this token, so
the permission model is unchanged; only the grant differs.

### 4.4 Per-service model

| Service | Grant                      | SSO?    | Identity                                                       |
| ------- | -------------------------- | ------- | -------------------------------------------------------------- |
| Grafana | `client_credentials` (JWT) | full    | jarvis LDAP account → Authentik → `[auth.jwt]`.                |
| gssh    | unchanged (JWT bearer)     | full    | jarvis LDAP account → Authentik → JWT bearer + host allow-list + ephemeral SSH cert. |
| Gitea   | scoped read-only PAT       | partial | jarvis Gitea user (Authentik login); PAT is the API credential. |

**gssh needs no change.** It already validates an Authentik JWT, mints an
ephemeral 5-minute SSH certificate for the principal, and enforces a per-account
host allow-list; the host `sshd` trusts that CA (`TrustedUserCAKeys`). Once
sherlock can mint the JWT headlessly (§4.3), `gssh-mcp` works as-is.

**Gitea is the unavoidable exception.** Its OAuth2 server is
authorization-code-only; it has no client-credentials or other headless grant,
so machine access to the Gitea *API* must use a Personal Access Token. The
identity is still the same account (its Gitea login resolves through Authentik);
only the API *token* is static. It is stored as SOPS ciphertext, scoped
read-only, and rotated like any other Charlie secret. The same PAT already backs
the read-only repo clone at container start.

### 4.5 Least privilege

- No service registers write tools; `grafana-mcp` and `gssh-mcp` run their
  read-only tool subsets as today.
- gssh access is bounded by the account's **host allow-list**. gssh command
  execution runs arbitrary single commands, but with no sudo (§4.2) and
  read-only group membership the reachable surface is inspection only; the
  prompt reinforces read-only intent, and enforcement lives in gssh, not jarvis.
- The Gitea PAT is scoped to read on repositories, organizations, issues, and
  actions — enough for config, deploy status, and failed-job triage.
- Read access to a stack's runtime data (where needed) is a matter of adding the
  jarvis account to that stack's `<stack>-rw` LDAP group, never `admins`.

### 4.6 Token lifecycle

client-credentials access tokens are short-lived and not refreshable in the
usual sense; renewal is simply re-minting from the token endpoint. sherlock's
token source keeps the bearer fresh transparently, the same way it does for the
refresh-token path today — the MCP request path never sees expiry.

## 5. Required changes in sherlock

The headless grant is the one true blocker and lives entirely in [sherlock].
gssh needs no change (§4.4).

| Area               | Change                                                                                                                                                    |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `internal/authn/`  | Non-interactive `client_credentials` acquisition (app-password form, representing the jarvis account); token-source renewal re-mints instead of using `refresh_token`. No loopback, no browser. |
| `internal/config/` | Per-service `grant = "client_credentials"` plus the account `username` + app password (from secret). Default grant stays the browser flow.                 |
| `cmd/grafana-mcp/` | Select the grant from config; skip the `OnAuthURL` browser path when the grant is non-interactive.                                                         |
| `cmd/gitea-mcp/`   | Accept a static read-only PAT (`token = "…"`) and send Gitea's `Authorization: token <PAT>` scheme, bypassing OAuth for this service only.                 |
| `docs/`            | Document the machine grant in `auth-model.md` and `configuration.md`.                                                                                      |

Config shape (contract, not implementation):

```toml
[services.grafana]
provider = "sherlock-cli"      # existing Authentik provider (client id)
grant    = "client_credentials"
username = "jarvis"            # the LDAP account; app password injected from secret
base_url = "https://grafana.example"

[services.gitea]
base_url = "https://gitea.example"
token    = "…"                # scoped read-only PAT, injected from SOPS
```

## 6. Diagnostic agent

### 6.1 Grounding

The agent runs with its working directory set to the cloned **project-charlie**
checkout so the Copilot CLI auto-loads Charlie's custom instructions and skills
(`charlie-deployment-status`, `charlie-grafana-logs`, `charlie-stack-gitea`) and
can consult the [TSG TOC][tsg-toc] before probing live systems. Sibling repos
cloned under `/context` are made readable via `--add-dir` so the agent can grep
any stack's compose files, scrape configs, and alert rules.

### 6.2 Tool access

Live read access comes from the sherlock MCPs, giving the agent the same
surfaces an operator would open:

- **Grafana** — dashboards, Prometheus/Loki queries, alert rules, annotations.
- **gssh** — host allow-list and read-only command execution on affected hosts.
- **Gitea** — repo contents, recent commits, workflow runs, failed job logs,
  releases.

### 6.3 Prompt contract

The prompt keeps jarvis' role (on-call SRE producing a short triage) but defers
process to the loaded Charlie instructions and skills rather than restating
them. It requires the section structure from §2.3, an explicit
`act`/`watch`/`ignore` verdict, and read-only behaviour. Each run is bounded by
`JARVIS_DIAG_TIMEOUT`.

### 6.4 Safety posture

The container is a throwaway diagnostic sandbox with read-only credentials.
`--yolo` auto-approves tools, which is acceptable precisely because every tool
and credential is read-only and the blast radius is bounded by SSO scope. jarvis
publishes conclusions; it never remediates.

## 7. Deployment

Unchanged shape from today (see [Charlie/jarvis]): built by project-charlie's
`build-image` scoped workflow into [amacocian/jarvis], deployed by the
`Charlie/jarvis` stack via `deploy-stack.yml`, pinned to host `morgott`, state
under `/mnt/nas/data/jarvis`.

New deploy inputs, all SOPS-encrypted:

| Secret / config        | Purpose                                                                          |
| ---------------------- | -------------------------------------------------------------------------------- |
| jarvis app password    | The LDAP account's Authentik app password, for the Grafana `client_credentials` grant. |
| Gitea read-only PAT    | `gitea-mcp` and repo clone.                                                       |
| sherlock `config.toml` | Service targets and grants (non-secret targets; secret credentials injected).    |

The image additionally ships the sherlock CLI and its MCP binaries
(`grafana-mcp`, `gssh-mcp`, `gitea-mcp`) so `sherlock copilot` works headless.

### 7.1 Manual provisioning (outside the repos)

Two identity artifacts are configured by hand, matching how Charlie manages its
IdP (Authentik providers and accounts are edited in the UI, not GitOps
blueprints; the [authentik] stack is on 2025.8, which supports the app-password
`client_credentials` flow):

- the **LDAP `jarvis` account** (synced into Authentik; not a member of
  `admins`), plus the `<stack>-rw` groups it needs to read;
- its **Authentik app password** and the Grafana provider's scope mappings, so
  the minted JWT carries the claims `[auth.jwt]` maps onto a Grafana identity.

The account's Gitea read-only PAT is minted once from its Gitea login (which
itself resolves through Authentik).

## 8. Phased delivery

1. **sherlock client-credentials grant** — §5. Unblocks headless auth for
   Grafana; add the Gitea PAT mode. gssh is unchanged. Validate each MCP with
   `--probe`.
2. **jarvis build** — ship sherlock + MCP binaries in the image, invoke
   `sherlock copilot`, set the workdir/`--add-dir` grounding, drop resolved
   alerts, roll the feed daily, and update the prompt and feed-entry contract
   (verdict + findings + confidence).
3. **jarvis deploy** — one LDAP `jarvis` account (synced to Authentik), new
   secrets and sherlock config, Grafana `[auth.jwt]` wiring, docs. No `admins`
   membership.

## 9. Open questions

- Which `<stack>-rw` LDAP groups (if any) does the jarvis account need for
  reading runtime data, versus API/log access being enough?
- Is a single Authentik provider fine for both Grafana and gssh, or is a
  separate audit trail per service worth the extra config?
- Prompt tuning: how much live probing is worthwhile before the triage's value
  per run stops justifying the agent runtime and token cost?

[amacocian/jarvis]: https://gitea.alexandru.macocian.me/amacocian/jarvis
[authentik]: https://gitea.alexandru.macocian.me/Charlie/authentik
[charlie]: https://gitea.alexandru.macocian.me/Charlie
[charlie/jarvis]: https://gitea.alexandru.macocian.me/Charlie/jarvis
[readme]: ../README.md
[sherlock]: https://gitea.alexandru.macocian.me/amacocian/sherlock
[sherlock auth-model]: https://gitea.alexandru.macocian.me/amacocian/sherlock/src/branch/main/docs/auth-model.md
[tsg-toc]: https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/tsg/toc.md