rblundon/homelab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(hindsight): cut over LLM model to DeepSeek-R1-Distill-Qwen-32B-AWQ (t_r1d32b_swap)

Browse Source 
astro-orbiter's vLLM primary model changed 2026-09-01 from
Qwen2.5-32B-Instruct-AWQ to DeepSeek-R1-Distill-Qwen-32B-AWQ
(single-model deployment, see ansible/roles/deploy-vllm/README.md).
Same endpoint (http://astro-orbiter:8000/v1) and API key — only the
served model name changed.

Flagged for follow-up, not fixed here: DeepSeek-R1 emits a
<think>...</think> reasoning trace before its final answer; Hindsight's
fact-extraction prompt/parsing has not been specifically verified
against this reasoning-model output shape. Watch extraction quality
after this rolls out; HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS left
at 4096 pending confirmation that's enough headroom for a real retain
to complete past the reasoning trace.
This commit is contained in:
Hermes Agent service account
2026-08-31 20:23:38 -05:00
parent 53a55e7317
commit f907acde95
1 changed files with 31 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

45
cluster/applications/hindsight/values.yaml
View File

@@ -63,20 +63,35 @@ api:
# CUT OVER to vLLM (t_5508360a, 2026-08-31): dashboard explicitly
# approved "stop and disable llama-swap and start vLLM" as a breaking
# change. llama-swap is now stopped+disabled on astro-orbiter; vLLM
# (Qwen2.5-32B-Instruct-AWQ, :8000) is the permanent, boot-persistent
# replacement. This exact config (base URL, model name, retry-safe low
# concurrency) was validated end-to-end in t_e6facb19's shadow window
# (health, /v1/chat/completions, live hindsight_retain+recall
# round-trip) before that task reverted it pending this decision --
# now re-applied for real. See roles/deploy-vllm/README.md "Critical
# architectural finding" + "Consumer cutover status" for the full history.
# was the permanent, boot-persistent replacement (originally
# Qwen2.5-32B-Instruct-AWQ). This exact config (base URL, model name,
# retry-safe low concurrency) was validated end-to-end in t_e6facb19's
# shadow window (health, /v1/chat/completions, live hindsight_retain+
# recall round-trip) before that task reverted it pending this
# decision -- now re-applied for real. See roles/deploy-vllm/README.md
# "Critical architectural finding" + "Consumer cutover status" for the
# full history.
#
# MODEL SWAP (t_r1d32b_swap, 2026-09-01): astro-orbiter's vLLM primary
# model changed from Qwen2.5-32B-Instruct-AWQ to
# DeepSeek-R1-Distill-Qwen-32B-AWQ (single-model deployment; nomic-embed
# and Qwen3-8B-AWQ both disabled on that host). Same endpoint
# (http://astro-orbiter:8000/v1), same API key (vLLM instance/port
# unchanged) -- only the served model name changed. DeepSeek-R1 emits
# a <think>...</think> reasoning trace before its final answer;
# Hindsight's fact-extraction prompt parsing has not been specifically
# audited against this reasoning-model output shape -- flag for
# follow-up if extraction quality regresses (see README note below).
HINDSIGHT_API_LLM_BASE_URL: "http://astro-orbiter:8000/v1"
HINDSIGHT_API_LLM_PROVIDER: "openai"
HINDSIGHT_API_LLM_MODEL: "Qwen2.5-32B-Instruct-AWQ"
# vLLM's max_model_len=8192 (role default, see deploy-vllm/defaults) is
# far below the 64000 default Hindsight's retain path assumes. Cap
# explicitly or long retains 400 with a context-length error (found +
# fixed in t_e6facb19's validation window).
HINDSIGHT_API_LLM_MODEL: "DeepSeek-R1-Distill-Qwen-32B-AWQ"
# vLLM's max_model_len is now 32768 (up from the prior 8192 role
# default) -- more headroom than before, but DeepSeek-R1 reasoning
# traces consume completion tokens before the final answer even
# appears, so the completion cap is left unchanged pending live
# verification that 4096 is enough for a real retain to complete
# (rather than truncating mid-<think>). Raise if extraction responses
# come back truncated/empty.
HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS: "4096"
# DO NOT set HINDSIGHT_API_EMBEDDINGS_* here (t_e6facb19, 2026-08-31
# attempted this, reverted after a production incident — see below).
@@ -113,8 +128,10 @@ api:
# 1 and pin retain to 1 as well. The upstream chart exposes these as native
# semaphore config (HINDSIGHT_API_*_MAX_CONCURRENT); no code change needed.
# Kept at 1 post-cutover: vLLM's single-process-per-model design is also
# effectively serial for a single Qwen2.5-32B instance under this GPU's
# VRAM budget (KV cache sized for low concurrency at max_model_len=8192).
# effectively serial for a single generative model instance under this
# GPU's VRAM budget (KV cache sized tight against the 24GB card at
# max_model_len=32768 for DeepSeek-R1-Distill-Qwen-32B-AWQ, t_r1d32b_swap
# 2026-09-01 — previously 8192 for Qwen2.5-32B-Instruct-AWQ).
HINDSIGHT_API_LLM_MAX_CONCURRENT: "1"
HINDSIGHT_API_RETAIN_LLM_MAX_CONCURRENT: "1"
# Client + per-request timeout. Default is 120s; a 29K-token retain runs