fix(llm-inference): bitsandbytes int4 OOM — pending switch to llama.cpp+GGUF

bitsandbytes quantizes on-the-fly: loads full bf16 weights (~54GB RAM peak)
before compressing to int4. Kills the 40GB OptiPlex on torch.compile warmup.

Fix in next commit: switch serve phase to llama.cpp + GGUF Q4_K_M.
Pre-quantized weights load directly — peak RAM ~16GB, no compile overhead.
This commit is contained in:
Hermes Agent service account committed 2026-08-03 12:35:46 -05:00
1 parent e879cf73d3
commit 22a020e4c7
4 files changed
+17 -4

No files matched your search

+4 -2
View File
@@ -22,9 +22,11 @@
become: true
become_user: "{{ llm_venv_owner }}"
- name: Install vLLM
- name: Install vLLM and bitsandbytes
ansible.builtin.pip:
name: vllm
name:
- vllm
- bitsandbytes
state: present
virtualenv: "{{ llm_venv_path }}"
become: true