fix(llm-inference): bitsandbytes int4 OOM — pending switch to llama.cpp+GGUF

bitsandbytes quantizes on-the-fly: loads full bf16 weights (~54GB RAM peak)
before compressing to int4. Kills the 40GB OptiPlex on torch.compile warmup.

Fix in next commit: switch serve phase to llama.cpp + GGUF Q4_K_M.
Pre-quantized weights load directly — peak RAM ~16GB, no compile overhead.
This commit is contained in:
Hermes Agent service account
2026-08-03 12:35:46 -05:00
parent e879cf73d3
commit 22a020e4c7
4 changed files with 17 additions and 4 deletions

View File

@@ -14,6 +14,7 @@ ExecStart={{ llm_venv_path }}/bin/python -m vllm.entrypoints.openai.api_server \
--model {{ llm_hf_model }} \
--host {{ llm_serve_host }} \
--port {{ llm_serve_port }} \
--quantization {{ llm_quantization }} \
--gpu-memory-utilization {{ llm_gpu_memory_utilization }} \
--max-model-len {{ llm_max_model_len }} \
--enable-prefix-caching