# Gitea troubleshooting

## GIT0001

### GIT0001 Description

A host running the `gitea` stack (melina) becomes unreachable roughly daily. The
root cause is an automated crawler (typically an AI bot) that walks Gitea's most
expensive endpoints — per-file commit history (`repo.RefCommits`,
`/<owner>/<repo>/commits/commit/<hash>/<file>`) and diffs — across a large mirror
repository. Each request spawns git subprocesses and takes several seconds, so a
sustained flood pins the CPU.

The sustained CPU load drives the host's temperature and RAPL power up until it
overheats and drops off the network. Only the affected host is impacted; sibling
hosts (e.g. morgott) keep serving and shipping logs.

Use `charlie-stack-gitea` to locate the stack and runtime paths, and
`charlie-grafana-logs` to confirm the log signature.

### GIT0001 Symptoms

- The host is unreachable over SSH (`ssh: connect ... Network is unreachable`),
  usually starting after local midnight and lasting into the morning.
- Thermal/power dashboard `charlie-hosts-thermal-power` for the host shows the
  hottest sensor jump from ~40 °C to 80 °C+ and RAPL power rise (e.g. 2 W → 3 W),
  sustained rather than spiky.
- Gitea log volume explodes (e.g. ~100 lines/10 min → 4000–6000 lines/10 min)
  and stays high.
- Gitea logs are dominated by `HTTPRequest [I] router: completed GET ...` and
  `[W] router: slow GET /<owner>/<repo>/commits/commit/... @ repo/commit.go`.
- The client IPs belong to a small crawler range (observed:
  `216.73.216.0/22`, Anthropic / ClaudeBot).

### GIT0001 How to confirm

1. Check reachability. The host is down while sibling hosts stay up, which points
   to a host-level (thermal) event, not a network outage.

2. Open the thermal/power dashboard and confirm the temperature/power spike lines
   up with the outage window:

   - Dashboard UID: `charlie-hosts-thermal-power`
   - Metrics: `node_hwmon_temp_celsius`, `node_rapl_*_joules_total`
     (label `host_name`)
   - Note: the metrics datasource may not be queryable through the Grafana MCP
     (HTTP 401). Read the panels in the UI and cross-check with logs.

3. In `charlie-grafana-logs` (datasource `victorialogs-loki`), confirm the flood:

   - Volume: `count_over_time({compose_project="gitea"}[10m])` — look for the
     sustained jump.
   - Shape: run `query_loki_patterns` on `{compose_project="gitea"}` — the top
     patterns will be `router: completed GET` and `router: slow GET`.
   - Source: `{compose_project="gitea"} |= "slow"` — the `for <ip>` field and the
     `repo/commit.go` handler expose the crawler IPs and the hammered repo.
   - Quantify: `sum(count_over_time({compose_project="gitea"} |= "<ip-prefix>" [30m]))`.

### GIT0001 Ways to fix

Fix in the `Charlie/caddy` edge (owner host morgott), which is reachable even
while melina is down. In order of priority:

1. **Block the abusive crawler IP ranges** (immediate, no image change). Add a
   `remote_ip` matcher + `abort` for the AI-bot ranges. Keep the list current
   from an upstream source such as
   [`rxerium/ai-bot-ip-ranges`](https://github.com/rxerium/ai-bot-ip-ranges)
   (weekly-updated, per-provider CIDR lists).

2. **Rate-limit per client IP** (defense-in-depth). Reuse the
   `defs/fragments/rate-limit.caddy` fragment in the site. This requires the
   `github.com/mholt/caddy-ratelimit` module compiled into the image
   (`amacocian/Caddy`), so rebuild and bump the image tag **before** deploying
   the Caddyfile change — otherwise Caddy fails to parse `rate_limit`.

3. **Reduce anonymous exposure of expensive endpoints.** Consider Gitea
   `[service] REQUIRE_SIGNIN_VIEW`, or hiding large mirror repos from anonymous
   browsing, so crawlers cannot reach `repo.RefCommits`/diff pages unauthenticated.

After mitigation, confirm the host recovers (reachable again, temperature back to
~40 °C) and Gitea log volume returns to baseline.