amacocianandCopilot
4f1f27c9b6
Document the recurring melina outage: an AI crawler (Anthropic/ClaudeBot, 216.73.216.0/22) floods Gitea's expensive commit-history endpoints on the OpenTTD mirror nightly, pinning CPU and overheating the host (temp 40->80C+, RAPL 2->3W) until it drops off the network. Add confirm/fix steps and wire it into the TSG TOC. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5331417-9af3-45d7-98a6-dff9293ef651
82 lines
3.8 KiB
Markdown
82 lines
3.8 KiB
Markdown
# Gitea troubleshooting
|
||
|
||
## GIT0001
|
||
|
||
### GIT0001 Description
|
||
|
||
A host running the `gitea` stack (melina) becomes unreachable roughly daily. The
|
||
root cause is an automated crawler (typically an AI bot) that walks Gitea's most
|
||
expensive endpoints — per-file commit history (`repo.RefCommits`,
|
||
`/<owner>/<repo>/commits/commit/<hash>/<file>`) and diffs — across a large mirror
|
||
repository. Each request spawns git subprocesses and takes several seconds, so a
|
||
sustained flood pins the CPU.
|
||
|
||
The sustained CPU load drives the host's temperature and RAPL power up until it
|
||
overheats and drops off the network. Only the affected host is impacted; sibling
|
||
hosts (e.g. morgott) keep serving and shipping logs.
|
||
|
||
Use `charlie-stack-gitea` to locate the stack and runtime paths, and
|
||
`charlie-grafana-logs` to confirm the log signature.
|
||
|
||
### GIT0001 Symptoms
|
||
|
||
- The host is unreachable over SSH (`ssh: connect ... Network is unreachable`),
|
||
usually starting after local midnight and lasting into the morning.
|
||
- Thermal/power dashboard `charlie-hosts-thermal-power` for the host shows the
|
||
hottest sensor jump from ~40 °C to 80 °C+ and RAPL power rise (e.g. 2 W → 3 W),
|
||
sustained rather than spiky.
|
||
- Gitea log volume explodes (e.g. ~100 lines/10 min → 4000–6000 lines/10 min)
|
||
and stays high.
|
||
- Gitea logs are dominated by `HTTPRequest [I] router: completed GET ...` and
|
||
`[W] router: slow GET /<owner>/<repo>/commits/commit/... @ repo/commit.go`.
|
||
- The client IPs belong to a small crawler range (observed:
|
||
`216.73.216.0/22`, Anthropic / ClaudeBot).
|
||
|
||
### GIT0001 How to confirm
|
||
|
||
1. Check reachability. The host is down while sibling hosts stay up, which points
|
||
to a host-level (thermal) event, not a network outage.
|
||
|
||
2. Open the thermal/power dashboard and confirm the temperature/power spike lines
|
||
up with the outage window:
|
||
|
||
- Dashboard UID: `charlie-hosts-thermal-power`
|
||
- Metrics: `node_hwmon_temp_celsius`, `node_rapl_*_joules_total`
|
||
(label `host_name`)
|
||
- Note: the metrics datasource may not be queryable through the Grafana MCP
|
||
(HTTP 401). Read the panels in the UI and cross-check with logs.
|
||
|
||
3. In `charlie-grafana-logs` (datasource `victorialogs-loki`), confirm the flood:
|
||
|
||
- Volume: `count_over_time({compose_project="gitea"}[10m])` — look for the
|
||
sustained jump.
|
||
- Shape: run `query_loki_patterns` on `{compose_project="gitea"}` — the top
|
||
patterns will be `router: completed GET` and `router: slow GET`.
|
||
- Source: `{compose_project="gitea"} |= "slow"` — the `for <ip>` field and the
|
||
`repo/commit.go` handler expose the crawler IPs and the hammered repo.
|
||
- Quantify: `sum(count_over_time({compose_project="gitea"} |= "<ip-prefix>" [30m]))`.
|
||
|
||
### GIT0001 Ways to fix
|
||
|
||
Fix in the `Charlie/caddy` edge (owner host morgott), which is reachable even
|
||
while melina is down. In order of priority:
|
||
|
||
1. **Block the abusive crawler IP ranges** (immediate, no image change). Add a
|
||
`remote_ip` matcher + `abort` for the AI-bot ranges. Keep the list current
|
||
from an upstream source such as
|
||
[`rxerium/ai-bot-ip-ranges`](https://github.com/rxerium/ai-bot-ip-ranges)
|
||
(weekly-updated, per-provider CIDR lists).
|
||
|
||
2. **Rate-limit per client IP** (defense-in-depth). Reuse the
|
||
`defs/fragments/rate-limit.caddy` fragment in the site. This requires the
|
||
`github.com/mholt/caddy-ratelimit` module compiled into the image
|
||
(`amacocian/Caddy`), so rebuild and bump the image tag **before** deploying
|
||
the Caddyfile change — otherwise Caddy fails to parse `rate_limit`.
|
||
|
||
3. **Reduce anonymous exposure of expensive endpoints.** Consider Gitea
|
||
`[service] REQUIRE_SIGNIN_VIEW`, or hiding large mirror repos from anonymous
|
||
browsing, so crawlers cannot reach `repo.RefCommits`/diff pages unauthenticated.
|
||
|
||
After mitigation, confirm the host recovers (reachable again, temperature back to
|
||
~40 °C) and Gitea log volume returns to baseline.
|