fix(monitoring): drop Qwen3.6 scrape job — causes CUDA OOM on each scrape (t_02c15dae)

/metrics?model=Qwen3.6 forces the router to attempt loading Qwen3.6 every
90s scrape cycle, triggering a CUDA OOM error since VRAM is already consumed
by the resident Llama3+Phi3.5 models. This produces real GPU power spikes
(~110W load-attempt), not a benign counter read like the Llama3/Phi3.5 jobs.

nvidia_gpu_exporter (:9835) already provides GPU power/VRAM/utilization at
zero wake cost. No Grafana dashboard panel references Qwen3.6 model-specific
llama-server metrics. Ryan approved full removal.

Job is commented out (not deleted) for easy revert if Qwen3.6 is ever
re-added as a resident model. Llama3/Phi3.5 scrape jobs untouched.
This commit is contained in:
Hermes Agent service account
2026-08-13 18:46:54 -05:00
parent a2ddb65425
commit a04435ee9b

View File

@@ -242,18 +242,29 @@ prometheus:
# wakes the GPU sub-server to P2 (~110W); at 15s, continuous spikes kept
# the GPU drawing ~110W despite zero inference load. At 90s, the GPU gets
# ~80s of genuine P8 idle (~20W) between wake-ups. (t_e7d547ea, 2026-08-13)
- job_name: llama-server-astro-orbiter-qwen3
scrape_interval: 90s
metrics_path: /metrics
params:
model: ["Qwen3.6-35B-A3B-UD-Q4_K_S"]
static_configs:
- targets:
- 10.1.71.130:8002
labels:
hostname: astro-orbiter
endpoint: astro-orbiter-router
model: Qwen3.6-35B-A3B-UD-Q4_K_S
# llama-server-astro-orbiter-qwen3 REMOVED (t_02c15dae, 2026-08-13):
# /metrics?model=Qwen3.6 forces the router to attempt loading Qwen3.6 each
# scrape cycle, which fails with CUDA OOM (VRAM already consumed by resident
# Llama3+Phi3.5). This generated real GPU power spikes, not just a benign
# counter read. nvidia_gpu_exporter (:9835) already covers GPU power/VRAM/
# utilization at zero wake cost. No Grafana dashboard panel depends on
# Qwen3.6-specific llama-server metrics. Ryan approved full removal.
# Commented out rather than deleted for easy revert if Qwen3.6 is ever
# re-added as a resident model.
#
# - job_name: llama-server-astro-orbiter-qwen3
# scrape_interval: 90s
# metrics_path: /metrics
# params:
# model: ["Qwen3.6-35B-A3B-UD-Q4_K_S"]
# static_configs:
# - targets:
# - 10.1.71.130:8002
# labels:
# hostname: astro-orbiter
# endpoint: astro-orbiter-router
# model: Qwen3.6-35B-A3B-UD-Q4_K_S
- job_name: llama-server-astro-orbiter-llama3
scrape_interval: 90s