Hermes Agent service account f3a5687adf hindsight: cap RETAIN_MAX_COMPLETION_TOKENS for vLLM's 8192 ctx ...

Default 64000 exceeded vLLM Qwen2.5-32B-Instruct-AWQ's --max-model-len
8192, causing every retain call to 500 with
'max_tokens=64000 cannot be greater than max_model_len=8192'.
llama-swap's Qwen3.8-27B ran at ctx=65536 so this never surfaced
before the vLLM cutover. Lowered to 4096.

2026-08-31 18:29:22 -05:00

9.5 KiB

Raw Blame History

View Raw