feat(deploy-vllm): swap Qwen2.5-32B for DeepSeek-R1-Distill-Qwen-32B-AWQ (t_r1d32b_swap)
Single-model deployment per Ryan's direction: - Primary model: casperhansen/deepseek-r1-distill-qwen-32b-awq, max_model_len=32768 - nomic-embed-text-v1.5 and Qwen3-8B-AWQ both disabled (single-model requirement) - kv_cache_dtype: int4_per_token_head required to fit 32768 ctx on 24GB RTX 3090 (fp16 KV: 26GB needed, doesn't fit at any utilization; fp8 KV: OOM'd during FlashInfer warmup with ~50-150MB margin; int4 KV: clean single-attempt start) - Added kv_cache_dtype / kv_cache_memory_bytes as new optional per-model template fields in vllm.service.j2 (guarded, no effect on other models) Verified live: /health 200, /v1/models confirms max_model_len=32768, live /v1/completions smoke test + manual chat completion both passed (genuine <think> reasoning trace, correct arithmetic). NRestarts=0, steady-state VRAM 23.2GB/24.576GB. Ansible idempotent re-run confirmed changed=0. Known follow-up (not done here): Hindsight's HINDSIGHT_API_LLM_MODEL cluster config still references the retired Qwen2.5-32B-Instruct-AWQ — needs separate GitOps update to point at the new model.
This commit is contained in:
@@ -46,6 +46,12 @@ ExecStart={{ vllm_venv_path }}/bin/python -m vllm.entrypoints.openai.api_server
|
||||
{% endif %}
|
||||
{% if item.quantization is defined and item.quantization != 'none' %}
|
||||
--quantization {{ item.quantization }} \
|
||||
{% endif %}
|
||||
{% if item.kv_cache_dtype is defined %}
|
||||
--kv-cache-dtype {{ item.kv_cache_dtype }} \
|
||||
{% endif %}
|
||||
{% if item.kv_cache_memory_bytes is defined %}
|
||||
--kv-cache-memory-bytes {{ item.kv_cache_memory_bytes }} \
|
||||
{% endif %}
|
||||
--gpu-memory-utilization {{ item.gpu_memory_utilization }} \
|
||||
--max-model-len {{ item.max_model_len }} \
|
||||
|
||||
Reference in New Issue
Block a user