Hermes Agent service account 22a020e4c7 fix(llm-inference): bitsandbytes int4 OOM — pending switch to llama.cpp+GGUF ...

bitsandbytes quantizes on-the-fly: loads full bf16 weights (~54GB RAM peak)
before compressing to int4. Kills the 40GB OptiPlex on torch.compile warmup.

Fix in next commit: switch serve phase to llama.cpp + GGUF Q4_K_M.
Pre-quantized weights load directly — peak RAM ~16GB, no compile overhead.

2026-08-03 12:35:46 -05:00

1.3 KiB

Raw Blame History

View Raw