feat(llm): add nomic-embed-text-v1.5-Q4_K_M to astro-orbiter router
OpenViking Phase 1b (t_34b96e83) — Ryan-approved implementation. Changes: - roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2: Add [nomic-embed-text-v1.5] section with embedding=true, n-gpu-layers=99, ctx-size=8192, load-on-startup=true, sleep-idle-seconds=-1. No flash-attn or KV cache params (embedding models use bidirectional forward pass, not autoregressive KV cache). Var: llm_router_nomic_ctx_size. - roles/llm-inference-multimodel/defaults/main.yml: Add llm_router_nomic_ctx_size: 8192. - host_vars/astro-orbiter/vars.yml: Add nomic-embed-text-v1.5-Q4_K_M.gguf to llm_staged_models list (size_bytes: 84106624, source: nomic-ai/nomic-embed-text-v1.5-GGUF). Update VRAM note to reflect 5 registered models (nomic adds ~84MB, negligible given sleep-idle-seconds=-1 / load-on-startup=true pinning). - playbooks/day2_add_nomic_embed.yml: New day2 playbook following the coder-alias pattern: Phase 1: idempotent GGUF download (exact size check) Phase 2: redeploy preset INI Phase 3: redeploy + restart systemd unit Phase 4: /v1/models gate (all 5 models present) Phase 5: /v1/embeddings smoke test (vector returned, not empty) VRAM: ~84MB, always pinned. No impact on generative model LRU behavior. peter-parker Helm values already point at :8002 for the embedding endpoint.
This commit is contained in:
@@ -160,3 +160,8 @@ llm_router_phi_flash_attn: "{{ llm_router_flash_attn }}"
|
||||
llm_router_coder_ctx_size: 16384
|
||||
llm_router_coder_flash_attn: "true"
|
||||
llm_router_preset_path: /opt/llama-server-router-preset.ini
|
||||
# nomic-embed-text-v1.5: embedding model, ctx-size=8192 per task t_34b96e83
|
||||
# No flash_attn or KV cache params — embedding models use bidirectional forward pass,
|
||||
# not autoregressive KV cache. load-on-startup=true / sleep-idle-seconds=-1 keep it
|
||||
# always warm at negligible VRAM cost (~84MB).
|
||||
llm_router_nomic_ctx_size: 8192
|
||||
|
||||
@@ -40,6 +40,14 @@
|
||||
; Hermes custom_providers routing — see role README / deployment report for
|
||||
; the alias-naming ambiguity flag (Ryan's pasted TOML used different alias
|
||||
; strings: "llama-3.1-8b" / "phi-3.5-mini").
|
||||
;
|
||||
; UPDATED (t_34b96e83, 2026-08-13, per Ryan approval): Added nomic-embed-text-v1.5
|
||||
; embedding model. Embedding models fold cleanly into the router preset via
|
||||
; embedding=true. No alias needed — clients call it by section name.
|
||||
; VRAM estimate ~90MB (negligible). sleep-idle-seconds=-1 keeps it always loaded
|
||||
; since embedding calls are latency-sensitive and it costs near-nothing to hold.
|
||||
; load-on-startup=true ensures the embedding endpoint is warm at boot without
|
||||
; waiting for the first request. — War Machine.
|
||||
; ------------------------------------------------------------------------------
|
||||
|
||||
; --- Production model: Qwen3.6-35B-A3B-UD-Q4_K_S ----------------------------
|
||||
@@ -124,3 +132,25 @@ cache-type-v = {{ llm_router_cache_type_v }}
|
||||
batch-size = {{ llm_router_batch_size }}
|
||||
ubatch-size = {{ llm_router_ubatch_size }}
|
||||
parallel = {{ llm_router_parallel }}
|
||||
|
||||
; --- Embedding model: nomic-embed-text-v1.5 ----------------------------------
|
||||
; Primary model ID: nomic-embed-text-v1.5 (section name / client-visible ID)
|
||||
; ~84MB GGUF — negligible VRAM, always-loaded. Embedding endpoint: /v1/embeddings.
|
||||
; embedding=true: required to expose /v1/embeddings and embed the model (not chat).
|
||||
; n-gpu-layers=99: GPU offload all layers (tiny model, no reason to leave on CPU).
|
||||
; ctx-size=8192: per task spec (OpenViking Phase 1b, t_34b96e83).
|
||||
; load-on-startup=true: warm at boot — embedding callers (peter-parker) are
|
||||
; latency-sensitive; no cold-load wait on first request.
|
||||
; sleep-idle-seconds=-1: never evict — ~84MB is negligible, always keep hot.
|
||||
; NO flash-attn, NO KV cache params: embedding models use a different forward
|
||||
; pass (bidirectional, no autoregressive KV cache). These keys are irrelevant
|
||||
; for embedding inference and may be silently ignored or cause warnings; omit.
|
||||
; Source: nomic-ai/nomic-embed-text-v1.5-GGUF (public, no auth needed)
|
||||
; Added 2026-08-13 (t_34b96e83) — War Machine.
|
||||
[nomic-embed-text-v1.5]
|
||||
model = {{ llm_models_dir }}/nomic-embed-text-v1.5-Q4_K_M.gguf
|
||||
embedding = true
|
||||
n-gpu-layers = {{ llm_router_gpu_layers }}
|
||||
ctx-size = {{ llm_router_nomic_ctx_size }}
|
||||
load-on-startup = true
|
||||
sleep-idle-seconds = -1
|
||||
|
||||
Reference in New Issue
Block a user