fix(llm-inference): bitsandbytes int4 OOM — pending switch to llama.cpp+GGUF

bitsandbytes quantizes on-the-fly: loads full bf16 weights (~54GB RAM peak)
before compressing to int4. Kills the 40GB OptiPlex on torch.compile warmup.

Fix in next commit: switch serve phase to llama.cpp + GGUF Q4_K_M.
Pre-quantized weights load directly — peak RAM ~16GB, no compile overhead.
This commit is contained in:
Hermes Agent service account
2026-08-03 12:35:46 -05:00
parent e879cf73d3
commit 22a020e4c7
4 changed files with 17 additions and 4 deletions

View File

@@ -22,9 +22,11 @@
become: true
become_user: "{{ llm_venv_owner }}"
- name: Install vLLM
- name: Install vLLM and bitsandbytes
ansible.builtin.pip:
name: vllm
name:
- vllm
- bitsandbytes
state: present
virtualenv: "{{ llm_venv_path }}"
become: true