fix(monitoring): drop Qwen3.6 scrape job — causes CUDA OOM on each scrape (t_02c15dae)
/metrics?model=Qwen3.6 forces the router to attempt loading Qwen3.6 every 90s scrape cycle, triggering a CUDA OOM error since VRAM is already consumed by the resident Llama3+Phi3.5 models. This produces real GPU power spikes (~110W load-attempt), not a benign counter read like the Llama3/Phi3.5 jobs. nvidia_gpu_exporter (:9835) already provides GPU power/VRAM/utilization at zero wake cost. No Grafana dashboard panel references Qwen3.6 model-specific llama-server metrics. Ryan approved full removal. Job is commented out (not deleted) for easy revert if Qwen3.6 is ever re-added as a resident model. Llama3/Phi3.5 scrape jobs untouched.
This commit is contained in:
@@ -242,18 +242,29 @@ prometheus:
|
|||||||
# wakes the GPU sub-server to P2 (~110W); at 15s, continuous spikes kept
|
# wakes the GPU sub-server to P2 (~110W); at 15s, continuous spikes kept
|
||||||
# the GPU drawing ~110W despite zero inference load. At 90s, the GPU gets
|
# the GPU drawing ~110W despite zero inference load. At 90s, the GPU gets
|
||||||
# ~80s of genuine P8 idle (~20W) between wake-ups. (t_e7d547ea, 2026-08-13)
|
# ~80s of genuine P8 idle (~20W) between wake-ups. (t_e7d547ea, 2026-08-13)
|
||||||
- job_name: llama-server-astro-orbiter-qwen3
|
|
||||||
scrape_interval: 90s
|
# llama-server-astro-orbiter-qwen3 REMOVED (t_02c15dae, 2026-08-13):
|
||||||
metrics_path: /metrics
|
# /metrics?model=Qwen3.6 forces the router to attempt loading Qwen3.6 each
|
||||||
params:
|
# scrape cycle, which fails with CUDA OOM (VRAM already consumed by resident
|
||||||
model: ["Qwen3.6-35B-A3B-UD-Q4_K_S"]
|
# Llama3+Phi3.5). This generated real GPU power spikes, not just a benign
|
||||||
static_configs:
|
# counter read. nvidia_gpu_exporter (:9835) already covers GPU power/VRAM/
|
||||||
- targets:
|
# utilization at zero wake cost. No Grafana dashboard panel depends on
|
||||||
- 10.1.71.130:8002
|
# Qwen3.6-specific llama-server metrics. Ryan approved full removal.
|
||||||
labels:
|
# Commented out rather than deleted for easy revert if Qwen3.6 is ever
|
||||||
hostname: astro-orbiter
|
# re-added as a resident model.
|
||||||
endpoint: astro-orbiter-router
|
#
|
||||||
model: Qwen3.6-35B-A3B-UD-Q4_K_S
|
# - job_name: llama-server-astro-orbiter-qwen3
|
||||||
|
# scrape_interval: 90s
|
||||||
|
# metrics_path: /metrics
|
||||||
|
# params:
|
||||||
|
# model: ["Qwen3.6-35B-A3B-UD-Q4_K_S"]
|
||||||
|
# static_configs:
|
||||||
|
# - targets:
|
||||||
|
# - 10.1.71.130:8002
|
||||||
|
# labels:
|
||||||
|
# hostname: astro-orbiter
|
||||||
|
# endpoint: astro-orbiter-router
|
||||||
|
# model: Qwen3.6-35B-A3B-UD-Q4_K_S
|
||||||
|
|
||||||
- job_name: llama-server-astro-orbiter-llama3
|
- job_name: llama-server-astro-orbiter-llama3
|
||||||
scrape_interval: 90s
|
scrape_interval: 90s
|
||||||
|
|||||||
Reference in New Issue
Block a user