project-charlie/tsg/gitea.md

Gitea troubleshooting

GIT0001

GIT0001 Description

A host running the gitea stack (melina) becomes unreachable roughly daily. The root cause is an automated crawler (typically an AI bot) that walks Gitea's most expensive endpoints — per-file commit history (repo.RefCommits, /<owner>/<repo>/commits/commit/<hash>/<file>) and diffs — across a large mirror repository. Each request spawns git subprocesses and takes several seconds, so a sustained flood pins the CPU.

The sustained CPU load drives the host's temperature and RAPL power up until it overheats and drops off the network. Only the affected host is impacted; sibling hosts (e.g. morgott) keep serving and shipping logs.

Use charlie-stack-gitea to locate the stack and runtime paths, and charlie-grafana-logs to confirm the log signature.

GIT0001 Symptoms

• The host is unreachable over SSH (ssh: connect ... Network is unreachable), usually starting after local midnight and lasting into the morning.
• Thermal/power dashboard charlie-hosts-thermal-power for the host shows the hottest sensor jump from ~40 °C to 80 °C+ and RAPL power rise (e.g. 2 W → 3 W), sustained rather than spiky.
• Gitea log volume explodes (e.g. ~100 lines/10 min → 4000–6000 lines/10 min) and stays high.
• Gitea logs are dominated by HTTPRequest [I] router: completed GET ... and [W] router: slow GET /<owner>/<repo>/commits/commit/... @ repo/commit.go.
• The client IPs belong to a small crawler range (observed: 216.73.216.0/22, Anthropic / ClaudeBot).

GIT0001 How to confirm

1. Check reachability. The host is down while sibling hosts stay up, which points to a host-level (thermal) event, not a network outage.

2. Open the thermal/power dashboard and confirm the temperature/power spike lines up with the outage window:

   • Dashboard UID: charlie-hosts-thermal-power
   • Metrics: node_hwmon_temp_celsius, node_rapl_*_joules_total (label host_name)
   • Note: the metrics datasource may not be queryable through the Grafana MCP (HTTP 401). Read the panels in the UI and cross-check with logs.

3. In charlie-grafana-logs (datasource victorialogs-loki), confirm the flood:

   • Volume: count_over_time({compose_project="gitea"}[10m]) — look for the sustained jump.
   • Shape: run query_loki_patterns on {compose_project="gitea"} — the top patterns will be router: completed GET and router: slow GET.
   • Source: {compose_project="gitea"} |= "slow" — the for <ip> field and the repo/commit.go handler expose the crawler IPs and the hammered repo.
   • Quantify: sum(count_over_time({compose_project="gitea"} |= "<ip-prefix>" [30m])).

GIT0001 Ways to fix

Fix in the Charlie/caddy edge (owner host morgott), which is reachable even while melina is down. In order of priority:

1. Block the abusive crawler IP ranges (immediate, no image change). Add a remote_ip matcher + abort for the AI-bot ranges. Keep the list current from an upstream source such as rxerium/ai-bot-ip-ranges (weekly-updated, per-provider CIDR lists).

2. Rate-limit per client IP (defense-in-depth). Reuse the defs/fragments/rate-limit.caddy fragment in the site. This requires the github.com/mholt/caddy-ratelimit module compiled into the image (amacocian/Caddy), so rebuild and bump the image tag before deploying the Caddyfile change — otherwise Caddy fails to parse rate_limit.

3. Reduce anonymous exposure of expensive endpoints. Consider Gitea [service] REQUIRE_SIGNIN_VIEW, or hiding large mirror repos from anonymous browsing, so crawlers cannot reach repo.RefCommits/diff pages unauthenticated.

After mitigation, confirm the host recovers (reachable again, temperature back to ~40 °C) and Gitea log volume returns to baseline.