Jarvis/docs/spec.md

jarvis — design spec

Status: draft. This document describes the target design for jarvis as an incident-enrichment service. The README documents what exists today; this spec documents where it is going and why the pieces fit together.

1. Vision

When something breaks in the Charlie homelab, the first job is not fixing — it is understanding. jarvis is the on-call teammate that does the boring first pass for you.

An alert fires. Before you have opened a laptop, jarvis has already:

• read the alert and restated it in plain language,
• checked the relevant dashboards, metrics, and logs in Grafana,
• looked at the offending stack's repo, recent deploys, and failed jobs in Gitea,
• optionally poked the affected host read-only over the SSH gateway,
• grounded all of it in the Charlie docs, skills, and troubleshooting guides,

and written a short, skimmable triage that lands in your RSS reader as one more item to read over coffee. You skim it and decide: act, watch, or ignore.

jarvis never pages you awake to go spelunking. It hands you a conclusion and its evidence, and lets you stay the decision-maker. It is an enrichment and triage tool, not an auto-remediation system: it reads, it reasons, it reports. It does not change anything.

The end state feels like a competent junior SRE who reads every alert, does the homework, and leaves you a tidy note — except it is instant, tireless, and already knows the whole estate.

2. Product behaviour

2.1 Lifecycle

flowchart LR
  VM[vmalert / Grafana] -->|webhook| WH[POST /webhook/grafana]
  WH --> Q[(queue)]
  Q --> AG[copilot agent<br/>headless, RO tools]
  AG --> DB[(SQLite /data)]
  DB --> FE[GET /feed.atom]
  FE -->|Basic auth| MF[miniflux]

An alert batch arrives on the authenticated webhook, is queued, and processed serially (one heavyweight agent run at a time). Each alert becomes one feed entry: the AI triage plus the raw alert for audit. miniflux polls the authenticated Atom feed and surfaces new entries in the reader.

Only firing alerts are enriched. Resolved notifications are acknowledged and dropped — the value is in the first-pass triage of an active problem, and the reader already carries the firing entry as history.

The feed is a rolling daily window: it serves entries from the current day and resets each day. miniflux is the durable store — it caches every entry it has fetched — so jarvis keeps only what a fresh reader needs to catch up, not a long archive.

2.2 Endpoints

Method	Path	Auth	Consumer
POST	/webhook/grafana	Bearer token	Grafana contact pt.
GET	/feed.atom	HTTP Basic	miniflux
GET	/healthz	none	healthcheck

2.3 Feed entry contract

Every entry is Markdown and skimmable in a reader in under a minute. It always carries enough for a triage decision without opening anything else:

Field	Purpose
Title	[status/severity] <summary> for at-a-glance scanning.
Verdict	act / watch / ignore — the triage call, up top.
What fired	One plain-language sentence.
Findings	What jarvis actually observed (live metrics, logs, jobs).
Likely cause	Most probable first, tied to concrete config or evidence.
Suggested next	The single most useful next step, if any.
Confidence	How much to trust the above; what jarvis could not check.
Links	Dashboard, generator, silence URLs from the alert.

On diagnostic failure the entry still publishes with the raw alert and the error, so an outage in jarvis never silently swallows an alert.

3. Architecture

jarvis is a single Go service plus a headless agent, in one container.

flowchart TB
  subgraph C[jarvis container]
    S[jarvis server<br/>Go: webhook, queue, store, feed]
    subgraph agent[sherlock copilot -p ...]
      CO[copilot CLI<br/>--yolo, RO]
      G[grafana-mcp]
      H[gssh-mcp]
      GT[gitea-mcp]
    end
    S -->|spawns per alert| agent
    CTX[/context: cloned Charlie repos<br/>skills + TSG + docs/]
    CO -.grounds in.-> CTX
  end
  CO --> G --> GR[(Grafana)]
  CO --> H --> GS[(gssh gateway)]
  CO --> GT --> GI[(Gitea API)]

Component	Repo	Role
jarvis service	amacocian/jarvis	Webhook → queue → store → Atom feed.
diagnostic agent	Copilot CLI + sherlock	Headless reasoning with RO tool access.
MCP servers	sherlock	grafana-mcp, gssh-mcp, gitea-mcp.
deploy stack	Charlie/jarvis	Compose, secrets, host pin, config.
context repos	Charlie org	Cloned into /context for grounding.

The build change is small: the diagnoser invokes sherlock copilot instead of copilot directly, so sherlock renders the .mcp.json and injects the service MCPs. jarvis never touches a service token itself.

4. Identity and access

4.1 One identity, LDAP as source of truth

jarvis is a single LDAP account — nothing more. LDAP is the source of truth; Authentik syncs from it. Every service resolves that one identity through its own front door:

• Gitea logs the account in through Authentik.
• Grafana authenticates the account through Authentik.
• gssh / SSH resolves the account through LDAP directly.

So one account, defined once in LDAP, is the whole identity surface. Its permissions are its LDAP group memberships, scoped read-only, mirroring what a human on-call operator may see. jarvis is not a bag of static per-service tokens; it is a directory principal. Static tokens appear only where a service cannot speak machine SSO (see §4.4).

4.2 No blast radius

On the hosts, passwordless sudo is granted solely by the LDAP admins group (/etc/sudoers.d/90-ldap-admins: %admins ALL=(ALL) NOPASSWD:ALL). The jarvis account is deliberately not in admins, so it can never sudo and cannot mutate a deployment, restart a service, or read a root-owned secret. Combined with read-only tool sets and the gssh host allow-list, the worst jarvis can do is look. This is what makes --yolo acceptable (§6.4).

4.3 The headless problem

sherlock's only login path today is an interactive PKCE browser flow on 127.0.0.1:6990 (see sherlock auth-model). A headless container has no browser and no operator to click. That flow cannot run in jarvis.

The SSO-native fix is the OAuth2 client-credentials grant (machine-to-machine): sherlock presents the jarvis account's credentials directly to Authentik's token endpoint and receives a JWT — no redirect, no browser. So the JWT can represent the jarvis LDAP account (and carry its group claims) rather than an anonymous Authentik service account, the grant uses the account's app-password form (client_id + username + app password) rather than a bare client_id + client_secret. Grafana's [auth.jwt] integration and the gssh gateway's JWT bearer scheme already validate exactly this token, so the permission model is unchanged; only the grant differs.

4.4 Per-service model

Service	Grant	SSO?	Identity
Grafana	client_credentials (JWT)	full	jarvis LDAP account → Authentik → [auth.jwt].
gssh	unchanged (JWT bearer)	full	jarvis LDAP account → Authentik → JWT bearer + host allow-list + ephemeral SSH cert.
Gitea	scoped read-only PAT	partial	jarvis Gitea user (Authentik login); PAT is the API credential.

gssh needs no change. It already validates an Authentik JWT, mints an ephemeral 5-minute SSH certificate for the principal, and enforces a per-account host allow-list; the host sshd trusts that CA (TrustedUserCAKeys). Once sherlock can mint the JWT headlessly (§4.3), gssh-mcp works as-is.

Gitea is the unavoidable exception. Its OAuth2 server is authorization-code-only; it has no client-credentials or other headless grant, so machine access to the Gitea API must use a Personal Access Token. The identity is still the same account (its Gitea login resolves through Authentik); only the API token is static. It is stored as SOPS ciphertext, scoped read-only, and rotated like any other Charlie secret. The same PAT already backs the read-only repo clone at container start.

4.5 Least privilege

• No service registers write tools; grafana-mcp and gssh-mcp run their read-only tool subsets as today.
• gssh access is bounded by the account's host allow-list. gssh command execution runs arbitrary single commands, but with no sudo (§4.2) and read-only group membership the reachable surface is inspection only; the prompt reinforces read-only intent, and enforcement lives in gssh, not jarvis.
• The Gitea PAT is scoped to read on repositories, organizations, issues, and actions — enough for config, deploy status, and failed-job triage.
• Read access to a stack's runtime data (where needed) is a matter of adding the jarvis account to that stack's <stack>-rw LDAP group, never admins.

4.6 Token lifecycle

client-credentials access tokens are short-lived and not refreshable in the usual sense; renewal is simply re-minting from the token endpoint. sherlock's token source keeps the bearer fresh transparently, the same way it does for the refresh-token path today — the MCP request path never sees expiry.

5. Required changes in sherlock

The headless grant is the one true blocker and lives entirely in sherlock. gssh needs no change (§4.4).

Area	Change
internal/authn/	Non-interactive client_credentials acquisition (app-password form, representing the jarvis account); token-source renewal re-mints instead of using refresh_token. No loopback, no browser.
internal/config/	Per-service grant = "client_credentials" plus the account username + app password (from secret). Default grant stays the browser flow.
cmd/grafana-mcp/	Select the grant from config; skip the OnAuthURL browser path when the grant is non-interactive.
cmd/gitea-mcp/	Accept a static read-only PAT (token = "…") and send Gitea's Authorization: token <PAT> scheme, bypassing OAuth for this service only.
docs/	Document the machine grant in auth-model.md and configuration.md.

Config shape (contract, not implementation):

[services.grafana]
provider = "sherlock-cli"      # existing Authentik provider (client id)
grant    = "client_credentials"
username = "jarvis"            # the LDAP account; app password injected from secret
base_url = "https://grafana.example"

[services.gitea]
base_url = "https://gitea.example"
token    = "…"                # scoped read-only PAT, injected from SOPS

6. Diagnostic agent

6.1 Grounding

The agent runs with its working directory set to the cloned project-charlie checkout so the Copilot CLI auto-loads Charlie's custom instructions and skills (charlie-deployment-status, charlie-grafana-logs, charlie-stack-gitea) and can consult the TSG TOC before probing live systems. Sibling repos cloned under /context are made readable via --add-dir so the agent can grep any stack's compose files, scrape configs, and alert rules.

6.2 Tool access

Live read access comes from the sherlock MCPs, giving the agent the same surfaces an operator would open:

• Grafana — dashboards, Prometheus/Loki queries, alert rules, annotations.
• gssh — host allow-list and read-only command execution on affected hosts.
• Gitea — repo contents, recent commits, workflow runs, failed job logs, releases.

6.3 Prompt contract

The prompt keeps jarvis' role (on-call SRE producing a short triage) but defers process to the loaded Charlie instructions and skills rather than restating them. It requires the section structure from §2.3, an explicit act/watch/ignore verdict, and read-only behaviour. Each run is bounded by JARVIS_DIAG_TIMEOUT.

6.4 Safety posture

The container is a throwaway diagnostic sandbox with read-only credentials. --yolo auto-approves tools, which is acceptable precisely because every tool and credential is read-only and the blast radius is bounded by SSO scope. jarvis publishes conclusions; it never remediates.

7. Deployment

Unchanged shape from today (see Charlie/jarvis): built by project-charlie's build-image scoped workflow into amacocian/jarvis, deployed by the Charlie/jarvis stack via deploy-stack.yml, pinned to host morgott, state under /mnt/nas/data/jarvis.

New deploy inputs, all SOPS-encrypted:

Secret / config	Purpose
jarvis app password	The LDAP account's Authentik app password, for the Grafana client_credentials grant.
Gitea read-only PAT	gitea-mcp and repo clone.
sherlock config.toml	Service targets and grants (non-secret targets; secret credentials injected).

The image additionally ships the sherlock CLI and its MCP binaries (grafana-mcp, gssh-mcp, gitea-mcp) so sherlock copilot works headless.

7.1 Manual provisioning (outside the repos)

Two identity artifacts are configured by hand, matching how Charlie manages its IdP (Authentik providers and accounts are edited in the UI, not GitOps blueprints; the authentik stack is on 2025.8, which supports the app-password client_credentials flow):

• the LDAP jarvis account (synced into Authentik; not a member of admins), plus the <stack>-rw groups it needs to read;
• its Authentik app password and the Grafana provider's scope mappings, so the minted JWT carries the claims [auth.jwt] maps onto a Grafana identity.

The account's Gitea read-only PAT is minted once from its Gitea login (which itself resolves through Authentik).

8. Phased delivery

1. sherlock client-credentials grant — §5. Unblocks headless auth for Grafana; add the Gitea PAT mode. gssh is unchanged. Validate each MCP with --probe.
2. jarvis build — ship sherlock + MCP binaries in the image, invoke sherlock copilot, set the workdir/--add-dir grounding, drop resolved alerts, roll the feed daily, and update the prompt and feed-entry contract (verdict + findings + confidence).
3. jarvis deploy — one LDAP jarvis account (synced to Authentik), new secrets and sherlock config, Grafana [auth.jwt] wiring, docs. No admins membership.

9. Open questions

• Which <stack>-rw LDAP groups (if any) does the jarvis account need for reading runtime data, versus API/log access being enough?
• Is a single Authentik provider fine for both Grafana and gssh, or is a separate audit trail per service worth the extra config?
• Prompt tuning: how much live probing is worthwhile before the triage's value per run stops justifying the agent runtime and token cost?