diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 288c2c7..01a1857 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -14,7 +14,10 @@ ## Project docs - Start with the [docs map][toc] for stable project context. -- Troubleshooting guides will live under `tsg/` with their own TOC. +- Troubleshooting guides will live under `tsg/` with their own [TOC][tsg-toc]. +- When asked for help with a problem, issue, outage, symptom, broken service, + failure, or error, check the [TSG TOC][tsg-toc] before probing live systems or + stack repos. If a matching TSG exists, follow it first. | Need | Doc | | --------------------------------- | ------------------------------------------------------------------------- | @@ -40,6 +43,10 @@ details into docs; Mermaid diagrams are fine. - Do not preserve history or reasoning in docs; Git history has that. - Explain what exists and where it lives, not how every piece is implemented. +- In TSG TOCs, keep issue titles and symptom keywords at 50 characters or less. + Symptoms must be keywords only. Keep tables narrow. Do not use inline HTML. + Keep skills in the table using aligned continuation rows with blank ID, issue, + and symptoms cells; use reference-style skill links. ## Architecture guardrails @@ -83,6 +90,7 @@ [runners]: https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/runners.md [secrets]: https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/secrets.md [strategy]: https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/strategy.md -[toc]: https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/toc.yaml +[toc]: https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/toc.md +[tsg-toc]: https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/tsg/toc.md [topology]: https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/topology.md [vulnerability-scanning]: https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/vulnerability-scanning.md diff --git a/.github/skills/charlie-grafana-logs/SKILL.md b/.github/skills/charlie-grafana-logs/SKILL.md new file mode 100644 index 0000000..1688ba1 --- /dev/null +++ b/.github/skills/charlie-grafana-logs/SKILL.md @@ -0,0 +1,33 @@ +--- +name: charlie-grafana-logs +description: Fast path for checking Grafana/VictoriaLogs logs for Charlie stacks and containers. Use when asked to inspect logs, errors, container output, or symptoms for a Charlie service. +--- + +# Charlie Grafana logs + +Use the Loki-compatible Grafana datasource first: + +- Datasource UID: `victorialogs-loki` +- Datasource name: `VictoriaLogs Loki` +- Common labels: `compose_project`, `service_name`, `service`, `container`, + `host`, `job`, `stream` + +Preferred flow: + +1. If the stack is known, query by `compose_project=""`. +2. If the service/container is known, add `service_name=""`. +3. Use `query_loki_stats` on the selector before pulling logs. +4. Pull a small recent sample with `query_loki_logs`. +5. For errors, use case-insensitive filters for terms like `error`, `fail`, + `fatal`, `panic`, `denied`, `permission`, `warning`, `unable`, and `cannot`. +6. Use `count_over_time` for exact counts when the frequency matters. + +For Docker Compose stacks, the most precise selector is usually +`{compose_project="", service_name=""}`. + +If labels are missing or the selector returns no streams, list label names and +values for the same time range before changing the query shape. + +Capture the datasource, selector, time range, count, and representative log +lines in troubleshooting notes so the same evidence can become a TSG or skill +update. diff --git a/.github/skills/charlie-stack-gitea/SKILL.md b/.github/skills/charlie-stack-gitea/SKILL.md new file mode 100644 index 0000000..2a2c17a --- /dev/null +++ b/.github/skills/charlie-stack-gitea/SKILL.md @@ -0,0 +1,36 @@ +--- +name: charlie-stack-gitea +description: Fast path for using Project Charlie inventory to find stack ownership, Gitea repositories, compose files, workflows, and runtime paths. Use when troubleshooting or documenting any Charlie stack. +--- + +# Charlie stack Gitea + +Start from `docs/inventory.md` instead of guessing repository names or hosts. + +Inventory gives: + +- Stack name and owner host. +- Stack repository pattern: `Charlie/`. +- Source/image repository pattern: `amacocian/`. +- Sherlock/Gitea/Grafana/Gssh MCP context. + +Preferred flow for stack troubleshooting: + +1. Read the stack row in `docs/inventory.md`. +2. Open the stack repository as `Charlie/` with Gitea MCP tools. +3. Read the stack repo `README.md` for state, ownership, deploy notes, and known + prep. +4. Read `docker-compose.yml` for service names, mounted paths, users, + healthchecks, and image pins. +5. If deployment behavior matters, inspect the stack repo workflow and the + shared `Charlie/project-charlie` reusable workflow it calls. +6. Map the inventory host to Gssh as `.lan` for read-only live checks when + needed. + +Runtime paths follow the project convention: + +- Stack working tree: `/mnt/nas/stacks/` +- Runtime data: `/mnt/nas/data/` + +When pairing with log triage, use the inventory stack name as the Grafana +`compose_project` value unless the logs show a different label. diff --git a/README.md b/README.md index eae574f..a1ce7fb 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ inventory, and next work. | Topic | Doc | | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Docs index | [toc.yaml](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/toc.yaml) | +| Docs index | [toc.md](docs/toc.md) | | Architecture | [strategy](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/strategy.md), [topology](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/topology.md), [layout](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/layout.md) | | Fleet | [hosts](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/hosts.md), [inventory](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/inventory.md), [onboarding](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/onboarding.md) | | Access | [identity](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/identity.md), [secrets](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/secrets.md), [cert issuer](https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/cert-issuer.md) | diff --git a/docs/roadmap.md b/docs/roadmap.md index e192925..3e241ef 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1,16 +1,16 @@ # Next work -Project Charlie is mostly in steady state. Next work is prompts, skills, and -MCP context. +Project Charlie is mostly in steady state. Next work is prompts, skills, and MCP +context. ## Prompting and skills -| Work item | Target | -| ----------------------- | ----------------------------------------------------------------------------------------------------- | -| Skill pack | Create Charlie skills for stack deploys, Sherlock/MCP use, observability triage, and secret rotation. | -| MCP context | Document Gitea, Grafana, and Gssh tool boundaries through Sherlock. | -| Prompt library | Store reusable prompts for stack changes, incidents, CVE triage, and host work. | -| Validation checklist | Define the minimal checks for docs-only, stack, workflow, and secret changes. | +| Work item | Target | +| -------------------- | ----------------------------------------------------------------------------------------------------- | +| Skill pack | Create Charlie skills for stack deploys, Sherlock/MCP use, observability triage, and secret rotation. | +| MCP context | Document Gitea, Grafana, and Gssh tool boundaries through Sherlock. | +| Prompt library | Store reusable prompts for stack changes, incidents, CVE triage, and host work. | +| Validation checklist | Define the minimal checks for docs-only, stack, workflow, and secret changes. | ## Platform follow-ups diff --git a/docs/toc.md b/docs/toc.md new file mode 100644 index 0000000..e43e441 --- /dev/null +++ b/docs/toc.md @@ -0,0 +1,21 @@ +# Documentation map + +Repository: [Charlie/project-charlie](https://gitea.alexandru.macocian.me/Charlie/project-charlie) + +| ID | Doc | Use for | +| --- | --- | --- | +| overview | [Project Charlie](../README.md) | Repo purpose, canonical locations, current state. | +| topology | [Topology](topology.md) | Network, ingress, host/storage/metrics flow. | +| strategy | [Strategy](strategy.md) | Deployment model, stack types, shared-stack rules. | +| layout | [Layout](layout.md) | NAS paths, stack/data separation, naming. | +| hosts | [Hosts](hosts.md) | Fleet roles and SOPS recipients. | +| inventory | [Inventory](inventory.md) | Stack ownership, hosts, source/image repos, operator tooling. | +| onboarding | [Host onboarding](onboarding.md) | Required host state and deploy-target prerequisites. | +| identity | [Identity & access](identity.md) | Authentik, LDAP, step-ca, Gssh, Sherlock, service identities. | +| secrets | [Secrets](secrets.md) | SOPS, age keys, PAT identities, secret placement. | +| cert-issuer | [Cert issuer](cert-issuer.md) | step-ca, cert-syncer, cert-watch, trust boundaries. | +| automation | [Automation](automation.md) | Reusable workflows/actions, deploy path, build path, Gitea constraints. | +| runners | [Runners](runners.md) | Runner hosts, runtime model, runner-stack exceptions. | +| vulnerability-scanning | [Vulnerability scanning](vulnerability-scanning.md) | Trivy stacks, CVE metrics, Grafana dashboard. | +| conventions | [Conventions](conventions.md) | Stack and deployment conventions. | +| next-work | [Next work](roadmap.md) | Prompting, skills, MCP context, platform follow-ups. | diff --git a/docs/toc.yaml b/docs/toc.yaml deleted file mode 100644 index 15ecc16..0000000 --- a/docs/toc.yaml +++ /dev/null @@ -1,95 +0,0 @@ -version: 1 -base_url: "https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main" -repo: "Charlie/project-charlie" -purpose: "Documentation-only map for Project Charlie docs." - -docs: - - id: overview - title: "Project Charlie" - path: "README.md" - url: "https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/README.md" - use_for: "Repo purpose, canonical locations, current state." - - - id: topology - title: "Topology" - path: "docs/topology.md" - url: "https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/topology.md" - use_for: "Network, ingress, host/storage/metrics flow." - - - id: strategy - title: "Strategy" - path: "docs/strategy.md" - url: "https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/strategy.md" - use_for: "Deployment model, stack types, shared-stack rules." - - - id: layout - title: "Layout" - path: "docs/layout.md" - url: "https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/layout.md" - use_for: "NAS paths, stack/data separation, naming." - - - id: hosts - title: "Hosts" - path: "docs/hosts.md" - url: "https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/hosts.md" - use_for: "Fleet roles and SOPS recipients." - - - id: inventory - title: "Inventory" - path: "docs/inventory.md" - url: "https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/inventory.md" - use_for: "Stack ownership, hosts, source/image repos, operator tooling." - - - id: onboarding - title: "Host onboarding" - path: "docs/onboarding.md" - url: "https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/onboarding.md" - use_for: "Required host state and deploy-target prerequisites." - - - id: identity - title: "Identity & access" - path: "docs/identity.md" - url: "https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/identity.md" - use_for: "Authentik, LDAP, step-ca, Gssh, Sherlock, service identities." - - - id: secrets - title: "Secrets" - path: "docs/secrets.md" - url: "https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/secrets.md" - use_for: "SOPS, age keys, PAT identities, secret placement." - - - id: cert-issuer - title: "Cert issuer" - path: "docs/cert-issuer.md" - url: "https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/cert-issuer.md" - use_for: "step-ca, cert-syncer, cert-watch, trust boundaries." - - - id: automation - title: "Automation" - path: "docs/automation.md" - url: "https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/automation.md" - use_for: "Reusable workflows/actions, deploy path, build path, Gitea constraints." - - - id: runners - title: "Runners" - path: "docs/runners.md" - url: "https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/runners.md" - use_for: "Runner hosts, runtime model, runner-stack exceptions." - - - id: vulnerability-scanning - title: "Vulnerability scanning" - path: "docs/vulnerability-scanning.md" - url: "https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/vulnerability-scanning.md" - use_for: "Trivy stacks, CVE metrics, Grafana dashboard." - - - id: conventions - title: "Conventions" - path: "docs/conventions.md" - url: "https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/conventions.md" - use_for: "Stack and deployment conventions." - - - id: next-work - title: "Next work" - path: "docs/roadmap.md" - url: "https://gitea.alexandru.macocian.me/Charlie/project-charlie/src/branch/main/docs/roadmap.md" - use_for: "Prompting, skills, MCP context, platform follow-ups." diff --git a/tsg/permissions.md b/tsg/permissions.md new file mode 100644 index 0000000..18d8fd2 --- /dev/null +++ b/tsg/permissions.md @@ -0,0 +1,45 @@ +# Permissions troubleshooting + +## PRM0001 + +### Description + +A container runs as a non-root UID/GID, but its bind-mounted runtime data +directory is owned by another host identity. The service may start and read the +mount, but writes fail when it creates state, cache, database, lock, or snapshot +files. + +Use `/charlie-stack-gitea` to find the stack repo and runtime paths, then +`/charlie-grafana-logs` to confirm the log signature. + +### Symptoms + +- Repeated `permission denied` write failures in container logs. +- Application-specific save, cache, database, lock, or snapshot errors. +- The mounted host path is present, but ownership does not match the compose + service `user`. + +For `nitter-redis`, Redis reported temp RDB writes failing under `/data`, then +`Background saving error`. + +### How to confirm + +Check the stack inventory row, then inspect the stack repo `README.md` and +`docker-compose.yml` for the service user, bind mount, and expected first-time +host prep. + +In Grafana logs, start with the stack as `compose_project` and narrow to the +service with `service_name`. Count the relevant permission errors and keep a few +representative lines. + +On the owner host, compare the bind-mounted host path ownership and mode with +the UID/GID used by the compose service. + +### Ways to fix + +- Change ownership of the runtime data path to the UID/GID used by the service. +- If only one file is affected, change that file rather than the whole runtime + directory. +- If the runtime path was created by the wrong process, recreate it with the + ownership documented by the stack repo. +- Redeploy or restart the service only after the host path ownership is correct. diff --git a/tsg/toc.md b/tsg/toc.md new file mode 100644 index 0000000..e910a48 --- /dev/null +++ b/tsg/toc.md @@ -0,0 +1,12 @@ +# Troubleshooting guide map + +Repository: +[Charlie/project-charlie](https://gitea.alexandru.macocian.me/Charlie/project-charlie) + +| ID | Issue | Symptoms | Skills | +| ------- | ------------------------------------------------------- | ----------------------------- | ------------------------------------------ | +| PRM0001 | [Runtime bind mount unwritable](permissions.md#prm0001) | permission denied, bind mount | [charlie-stack-gitea][skill-stack-gitea] | +| | | | [charlie-grafana-logs][skill-grafana-logs] | + +[skill-grafana-logs]: ../.github/skills/charlie-grafana-logs/SKILL.md +[skill-stack-gitea]: ../.github/skills/charlie-stack-gitea/SKILL.md