feat(llm): add nomic-embed-text-v1.5-Q4_K_M to astro-orbiter router
OpenViking Phase 1b (t_34b96e83) — Ryan-approved implementation. Changes: - roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2: Add [nomic-embed-text-v1.5] section with embedding=true, n-gpu-layers=99, ctx-size=8192, load-on-startup=true, sleep-idle-seconds=-1. No flash-attn or KV cache params (embedding models use bidirectional forward pass, not autoregressive KV cache). Var: llm_router_nomic_ctx_size. - roles/llm-inference-multimodel/defaults/main.yml: Add llm_router_nomic_ctx_size: 8192. - host_vars/astro-orbiter/vars.yml: Add nomic-embed-text-v1.5-Q4_K_M.gguf to llm_staged_models list (size_bytes: 84106624, source: nomic-ai/nomic-embed-text-v1.5-GGUF). Update VRAM note to reflect 5 registered models (nomic adds ~84MB, negligible given sleep-idle-seconds=-1 / load-on-startup=true pinning). - playbooks/day2_add_nomic_embed.yml: New day2 playbook following the coder-alias pattern: Phase 1: idempotent GGUF download (exact size check) Phase 2: redeploy preset INI Phase 3: redeploy + restart systemd unit Phase 4: /v1/models gate (all 5 models present) Phase 5: /v1/embeddings smoke test (vector returned, not empty) VRAM: ~84MB, always pinned. No impact on generative model LRU behavior. peter-parker Helm values already point at :8002 for the embedding endpoint.
This commit is contained in:
@@ -34,21 +34,23 @@ common_root_lv: ubuntu-lv
|
||||
# (t_33acbb2e) so the router can keep more than one GGUF resident on-demand
|
||||
# and LRU-evict when needed.
|
||||
#
|
||||
# VRAM NOTE (t_33acbb2e, updated t_55c164f5): With models-max=4 and all 4 GGUFs
|
||||
# registered, worst case is all 4 loaded simultaneously:
|
||||
# VRAM NOTE (t_33acbb2e, updated t_55c164f5, updated t_34b96e83): With models-max=4 and all 5 GGUFs
|
||||
# registered, worst case is all 5 loaded simultaneously:
|
||||
# Qwen3.6-35B-A3B Q4_K_S: ~21.5GB (weights ~19.5GB + KV ~2GB @ 64K ctx, q4_0)
|
||||
# Phi-3.5-mini-instruct Q8_0: ~4.3GB (weights ~3.8GB + KV ~0.5GB @ 32K ctx)
|
||||
# Meta-Llama-3.1-8B Q4_K_M: ~5.6GB (weights ~4.6GB + KV ~0.2GB @ 8K ctx)
|
||||
# Qwen2.5-Coder-14B Q4_K_M: ~9.0GB (weights ~8.4GB + KV ~0.6GB @ 16K ctx)
|
||||
# Total worst-case: ~40.4GB >> 24GB RTX 3090
|
||||
# nomic-embed-text-v1.5 Q4_K_M: ~0.09GB (~84MB, embedding only — no KV cache)
|
||||
# Total worst-case: ~40.5GB >> 24GB RTX 3090
|
||||
#
|
||||
# OOM RISK: Full co-residency is impossible on 24GB. LRU eviction prevents this
|
||||
# in practice: models-max=4 means the router can REGISTER 4 models but only keeps
|
||||
# in practice: models-max=4 means the router can REGISTER 5 models but only keeps
|
||||
# up to 4 LOADED simultaneously — the router will evict the LRU model when a new
|
||||
# one is needed. In single-user homelab operation, only one model is active at a
|
||||
# time. The realistic maximum co-residency is 2 models (whichever was last used).
|
||||
# one is needed. nomic-embed-text-v1.5 is pinned via sleep-idle-seconds=-1 and
|
||||
# load-on-startup=true but it uses only ~84MB, so it never meaningfully changes
|
||||
# the budget. In single-user homelab operation, only one generative model is active
|
||||
# at a time alongside the always-resident embedding model.
|
||||
# Qwen3.6-35B alone uses ~21.5GB; co-residency with Coder (~9GB) = ~30.5GB > 24GB.
|
||||
# So effectively: Qwen3.6-35B + any second model will OOM IF both are held concurrently.
|
||||
# LRU eviction handles this automatically — the router evicts the idle model before
|
||||
# loading the new one. Ryan should be aware this means model-switching always incurs
|
||||
# a ~30-60s cold-load latency when switching between Qwen3.6-35B and any other model.
|
||||
@@ -68,4 +70,8 @@ llm_staged_models:
|
||||
url: "https://huggingface.co/bartowski/Qwen2.5-Coder-14B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
|
||||
size_bytes: 8988111072
|
||||
source_repo: "bartowski/Qwen2.5-Coder-14B-Instruct-GGUF"
|
||||
- filename: "nomic-embed-text-v1.5-Q4_K_M.gguf"
|
||||
url: "https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF/resolve/main/nomic-embed-text-v1.5.Q4_K_M.gguf"
|
||||
size_bytes: 84106624
|
||||
source_repo: "nomic-ai/nomic-embed-text-v1.5-GGUF"
|
||||
|
||||
|
||||
Reference in New Issue
Block a user