fix(llm-inference): bitsandbytes int4 OOM — pending switch to llama.cpp+GGUF
bitsandbytes quantizes on-the-fly: loads full bf16 weights (~54GB RAM peak) before compressing to int4. Kills the 40GB OptiPlex on torch.compile warmup. Fix in next commit: switch serve phase to llama.cpp + GGUF Q4_K_M. Pre-quantized weights load directly — peak RAM ~16GB, no compile overhead.
This commit is contained in:
@@ -14,6 +14,7 @@ ExecStart={{ llm_venv_path }}/bin/python -m vllm.entrypoints.openai.api_server \
|
||||
--model {{ llm_hf_model }} \
|
||||
--host {{ llm_serve_host }} \
|
||||
--port {{ llm_serve_port }} \
|
||||
--quantization {{ llm_quantization }} \
|
||||
--gpu-memory-utilization {{ llm_gpu_memory_utilization }} \
|
||||
--max-model-len {{ llm_max_model_len }} \
|
||||
--enable-prefix-caching
|
||||
|
||||
Reference in New Issue
Block a user