Vulnerability scanning

Daily CVE sweep of every container image in the fleet. Two stacks emit Prometheus metrics; node-exporter ships them through the existing otelcollector → VictoriaMetrics → Grafana pipeline.

Stacks

Stack	Host	Scans
`Charlie/trivy-server`	morgott	Long-running `trivy server` (shared vuln-DB cache on `morgott.lan:4954`, LAN-only) + daily 03:00 sweep of every `image:` ref declared in any `/mnt/nas/stacks//docker-compose.yml` (incl. per-host overrides + `x-*` anchors).
`Charlie/trivy-runner`	morgott + melina	Daily sweep of currently-running images (`docker ps`). Stagger: morgott 03:00, melina 03:15.

Both stacks talk to the same trivy server on morgott.lan:4954, so the ~1GB vuln DB is downloaded once, not per-host. No public DNS, no auth — morgott + melina are on the secure intranet.

Architecture

        ┌─────────────────────────────────────────┐
        │ trivy-server (morgott)                  │
        │                                         │
        │   trivy-server  ◄── shared vuln DB     │
        │       ▲           cache (1GB)           │
        │       │                                 │
        │   compose-scanner  ──► trivy_compose.prom
        │       ▲                                 │
        │   yq-extractor ──── walks /mnt/nas/stacks/*/
        │       ▲                                 │
        │   ofelia (cron, 03:00)                  │
        └───────┼─────────────────────────────────┘
                │
                │ http://morgott.lan:4954 (LAN, no auth)
                │
   ┌────────────┼────────────────┐
   │            │                │
   ▼            ▼                ▼
┌──────────┐ ┌──────────┐    morgott + melina
│ trivy-   │ │ trivy-   │    (per host)
│ runner   │ │ runner   │
│ morgott  │ │ melina   │    each:
│          │ │          │      docker-helper ──► images.txt
│          │ │          │      scanner       ──► trivy_running.prom
│          │ │          │      ofelia (03:00 / 03:15)
└──────────┘ └──────────┘
       │            │
       └─────┬──────┘
             ▼
   /var/lib/charlie/trivy/metrics/
       trivy_compose.prom    (morgott only)
       trivy_running.prom    (per host)
             │
             ▼
   node-exporter --collector.textfile        (in Charlie/otelcollector)
             │
             ▼
   otelcol prometheus receiver (keep regex includes `trivy_.*`)
             │
             ▼
   OTLP → VictoriaMetrics → Grafana

Metrics

trivy_image_vulnerabilities{image="…", severity="CRITICAL|HIGH|MEDIUM|LOW|UNKNOWN", scanner="compose|running"} <count>
trivy_image_scan_timestamp_seconds{image="…", scanner="…"}                                                    <unix_time>
trivy_image_scan_errors_total{image="…", scanner="…"}                                                          0|1

Notes:

host_name is added by otelcol's resourcedetection processor — not baked into the textfile.
scanner="compose" = images declared in compose files (canonical, catches stuff pulled but not running). scanner="running" = images returned by docker ps. Their difference is interesting:
- compose - running = pulled or declared but not running.
- running - compose = drift / hand-started containers.
The vuln count is package occurrences, not unique CVE IDs. A CVE affecting N packages in the same image counts N times. Decision was to keep it that way; trivy's per-package output reflects the real patch surface.

Pipeline behaviour

Cadence: daily at 03:00 (server compose-sweep), 03:00 (morgott running-sweep), 03:15 (melina running-sweep). Schedules live in each stack's overrides/<host>.yml as docker labels (ofelia label-discovery mode — --docker).
DB refresh: trivy itself refreshes its vuln DB every 6h. Counts legitimately drift between sweeps as new CVEs are disclosed against unchanged images.
Atomic writes: each scanner writes to a temp file in the same directory then mvs — node-exporter never sees a half-written file.

Dashboard

Container CVEs (uid charlie-containers-security-cves) under containers/security/. Selectors: host, scanner (running default), severity (CRITICAL+HIGH default).

Panels: top-N vulnerable images, severity breakdown table, CVE count over time, scan-error count, stalest-scan stat (catches stuck sweeps).

Auth

Trivy needs creds to pull private images from the Gitea registry (Charlie/* + amacocian/* are private). Both stacks ship a secrets/env.sops.yaml with:

TRIVY_USERNAME=sys-trivy-scanner
TRIVY_PASSWORD=<gitea-PAT-with-read:package>

sys-trivy-scanner is an LDAP-managed service account (Charlie/ldap) with the least privilege needed:

Member of the Charlie Runners team (registry read access for org packages).
Collaborator on each amacocian/* repo that emits a private package.
PAT scope: read:package only.

Rotate by issuing a new PAT in Gitea and re-encrypting both secrets/env.sops.yaml files.

Manual operations

Trigger a sweep ad-hoc (skips ofelia, runs the script directly):

# Compose-sweep (morgott only):
docker exec trivy-server-yq      sh /srv/scripts/extract-images.sh
docker exec trivy-server-scanner sh /srv/scripts/scan-compose.sh

# Running-sweep (per host):
docker exec trivy-runner-helper  sh /srv/scripts/list-running-images.sh
docker exec trivy-runner-scanner sh /srv/scripts/scan-running.sh

# Inspect output (host-side):
cat /var/lib/charlie/trivy/metrics/trivy_running.prom
cat /var/lib/charlie/trivy/metrics/trivy_compose.prom

# In Grafana Explore:
trivy_image_vulnerabilities
{__name__=~"trivy.*"}

ofelia 0.3.x has no run-job subcommand (added in 0.4); always use the direct docker exec path above.

Open follow-ups

Alerting: vmalert rule on trivy_image_vulnerabilities{severity="CRITICAL"} > 0 routed via the existing alertmanager. Deferred until a stable baseline.
CI build-time scan: run trivy image inside the eventual amacocian/* build pipelines (see roadmap.md → Build pipelines) so HIGH/CRITICAL issues fail the build instead of waiting for the next nightly sweep.
.trivyignore / accepted-CVE story for noisy false positives.

Triage workflow

When a Grafana alert (or a manual dashboard check) flags an image:

Note the image ref and the scanner label (compose vs running).

Run a one-off scan inside the appropriate stack's scanner container with --severity HIGH,CRITICAL to see the per-CVE detail:

docker exec trivy-runner-scanner trivy image \
  --server "$TRIVY_SERVER" \
  --severity HIGH,CRITICAL \
  <image-ref>

Decide per CVE: bump base image / wait for upstream fix / suppress via .trivyignore (justified). The stack repo (Charlie/<stack>) is the canonical place to pin the fix.
Re-run the sweep manually (commands above) to confirm green before waiting for cron.

7.6 KiB Raw Blame History