rblundon/homelab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

hindsight: cap RETAIN_MAX_COMPLETION_TOKENS for vLLM's 8192 ctx

Browse Source 
Default 64000 exceeded vLLM Qwen2.5-32B-Instruct-AWQ's --max-model-len
8192, causing every retain call to 500 with
'max_tokens=64000 cannot be greater than max_model_len=8192'.
llama-swap's Qwen3.8-27B ran at ctx=65536 so this never surfaced
before the vLLM cutover. Lowered to 4096.
This commit is contained in:
Hermes Agent service account
2026-08-31 18:29:22 -05:00
parent 9d6869ad9d
commit f3a5687adf
1 changed files with 10 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
cluster/applications/hindsight/values.yaml
View File

@@ -73,6 +73,16 @@ api:
HINDSIGHT_API_LLM_BASE_URL: "http://astro-orbiter:8000/v1" HINDSIGHT_API_LLM_BASE_URL: "http://astro-orbiter:8000/v1"
HINDSIGHT_API_LLM_PROVIDER: "openai" HINDSIGHT_API_LLM_PROVIDER: "openai"
HINDSIGHT_API_LLM_MODEL: "Qwen2.5-32B-Instruct-AWQ" HINDSIGHT_API_LLM_MODEL: "Qwen2.5-32B-Instruct-AWQ"
# vLLM's Qwen2.5-32B-Instruct-AWQ is deployed with --max-model-len 8192
# (roles/deploy-vllm/defaults/main.yml vllm_max_model_len; llama-swap's
# Qwen3.8-27B ran at ctx=65536, matching Hindsight's default
# RETAIN_MAX_COMPLETION_TOKENS=64000). Discovered via a live retain 500:
# "max_tokens=64000 cannot be greater than max_model_len=8192" — every
# retain call was requesting a 64K-token completion budget the new
# backend physically cannot serve. Lowered to 4096, comfortably under
# 8192 with room for the ~3000-char (RETAIN_CHUNK_SIZE) input chunk plus
# prompt overhead. Verified with a live hindsight_retain call post-fix.
HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS: "4096"
# DO NOT set HINDSIGHT_API_EMBEDDINGS_* here (t_e6facb19, 2026-08-31 # DO NOT set HINDSIGHT_API_EMBEDDINGS_* here (t_e6facb19, 2026-08-31
# attempted this, reverted after a production incident — see below). # attempted this, reverted after a production incident — see below).
# #