- Change emptyDir to persistent volume (open-webui-data PVC)
Fixes data loss on pod restart (admin user disappears)
- Add ENABLE_OLLAMA_API=false
Disable broken Ollama endpoint probing
vLLM uses OpenAI-compatible API, not Ollama
- Create pvc.yaml for persistent data storage
10Gi NFS-backed storage via nfs-emporium StorageClass
The 'open-webui' item doesn't exist in 1Password yet.
webui-secret-key is manually patched in the secret for now.
Re-enable ExternalSecret pull once the 1Password item is created.
- Add ExternalName Service for astro-orbiter (10.1.71.130:8000)
- Remove Ollama-specific configuration (OLLAMA_BASE_URLS)
- Keep only OpenAI-compatible configuration (OPENAI_BASE_URL)
Fixes: Open WebUI pod was attempting to use Ollama endpoints (/api/tags)
against vLLM which only supports OpenAI-compatible API (/v1/models).
Pod now reaches astro-orbiter via Kubernetes Service DNS name.
Resolves: HTTP 404 errors from Upstream ollama request failed
- ExternalSecret: use key/property format (not path) for 1Password reference
- Deployment: remove admin password env var (use single vllm-api-key)
- Syncs vLLM API key from op://mk-labs/vllm/api-key
astro-orbiter's vLLM primary model changed 2026-09-01 from
DeepSeek-R1-Distill-Qwen-32B-AWQ to Gemma-4-26B-A4B-it-AWQ (Google,
Apache 2.0, US-origin). Same endpoint/API key — only the served model
name changed. max_model_len also bumped to 65536 (was 32768).
Gemma 4 does not emit an always-on <think> reasoning trace like
DeepSeek-R1 did, so HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS=4096
should have more effective headroom for real extraction output than
before, not less.
Retired DeepSeek-R1-Distill-Qwen-32B after confirming its auto tool-choice
reliability is a known, documented DeepSeek-R1-distillation limitation
(trained on pure reasoning traces, no function-calling data — upstream
GitHub-confirmed, not a config gap). Model choice moved to Gemma 4 26B A4B
(Google, Apache 2.0, US-origin, matches Ryan's model-origin preference):
- cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit — MoE (25.2B total / 3.8B active),
chosen over the dense 31B variant for smaller on-disk footprint
(~17.2GB vs ~20.9GB), buying more KV-cache headroom on this 24GB card
- max_model_len=65536 (comfortably over Hermes's 64K floor; native
context is 256K, no extension trick needed)
- Native gemma4 tool-call-parser + gemma4 reasoning-parser (both
registered in this host's vLLM 0.28.0) — purpose-built for this
model's actual output format, not a same-family approximation
- kv_cache_dtype: int4_per_token_head from the outset (learned from the
DeepSeek swap's fp16->fp8->int4 trial-and-error escalation)
Bug found and fixed during deployment: the repo's config.json declares
quant_method 'compressed-tensors' (llm-compressor output) despite the
repo name saying 'AWQ-4bit'. Passing --quantization awq explicitly
caused a hard pydantic ValidationError on every startup attempt.
Fix: omit the quantization field entirely and let vLLM auto-detect from
the model's own config.json — confirmed clean single-attempt start,
NRestarts=0, once removed.
Verified live:
- /health 200, /v1/models confirms max_model_len=65536
- Live completion: correct answer, no unwanted reasoning trace by default
- tool_choice=auto with a clear trigger prompt: correct tool_calls
response with valid JSON args — the exact test DeepSeek-R1-Distill
failed (it either answered in plain text or burned tokens reasoning
about how to call the tool instead of calling it)
- tool_choice=auto with an irrelevant tool present: correctly answered
in plain text, did not over-trigger the tool
- Ansible idempotent re-run confirmed: changed=0, NRestarts=0, clean
journalctl (zero error/traceback lines) after a fresh restart
Known follow-up (not done here): Hindsight's HINDSIGHT_API_LLM_MODEL
cluster config still references the retired DeepSeek-R1-Distill-Qwen-32B
(itself a follow-up from the prior Qwen2.5-32B swap) — needs another
GitOps update to point at Gemma-4-26B-A4B-it-AWQ.
astro-orbiter's vLLM primary model changed 2026-09-01 from
Qwen2.5-32B-Instruct-AWQ to DeepSeek-R1-Distill-Qwen-32B-AWQ
(single-model deployment, see ansible/roles/deploy-vllm/README.md).
Same endpoint (http://astro-orbiter:8000/v1) and API key — only the
served model name changed.
Flagged for follow-up, not fixed here: DeepSeek-R1 emits a
<think>...</think> reasoning trace before its final answer; Hindsight's
fact-extraction prompt/parsing has not been specifically verified
against this reasoning-model output shape. Watch extraction quality
after this rolls out; HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS left
at 4096 pending confirmation that's enough headroom for a real retain
to complete past the reasoning trace.
Single-model deployment per Ryan's direction:
- Primary model: casperhansen/deepseek-r1-distill-qwen-32b-awq, max_model_len=32768
- nomic-embed-text-v1.5 and Qwen3-8B-AWQ both disabled (single-model requirement)
- kv_cache_dtype: int4_per_token_head required to fit 32768 ctx on 24GB RTX 3090
(fp16 KV: 26GB needed, doesn't fit at any utilization; fp8 KV: OOM'd during
FlashInfer warmup with ~50-150MB margin; int4 KV: clean single-attempt start)
- Added kv_cache_dtype / kv_cache_memory_bytes as new optional per-model
template fields in vllm.service.j2 (guarded, no effect on other models)
Verified live: /health 200, /v1/models confirms max_model_len=32768,
live /v1/completions smoke test + manual chat completion both passed
(genuine <think> reasoning trace, correct arithmetic). NRestarts=0,
steady-state VRAM 23.2GB/24.576GB. Ansible idempotent re-run confirmed
changed=0.
Known follow-up (not done here): Hindsight's HINDSIGHT_API_LLM_MODEL
cluster config still references the retired Qwen2.5-32B-Instruct-AWQ —
needs separate GitOps update to point at the new model.
Companion commit to 6bfcc76 (vllm permanent residency cutover). Now that
llama-swap is stopped+disabled and vLLM is the permanent serving layer:
- HINDSIGHT_API_LLM_BASE_URL: llama-swap :8001 -> vLLM :8000
- HINDSIGHT_API_LLM_MODEL: Qwen3.8-27B-Q4_K_M -> Qwen2.5-32B-Instruct-AWQ
- Added HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS=4096 (vLLM's
max_model_len=8192 is far below the 64000 Hindsight retain default;
uncapped this 400s on long retains -- found in t_e6facb19's validation)
- ExternalSecret HINDSIGHT_API_LLM_API_KEY source: 1Password 'nous' item
(unused fallback) -> 1Password 'vllm' item's api-key (vLLM validates
its bearer token, unlike llama-swap which accepted anything)
Embeddings intentionally UNCHANGED -- Hindsight stays on its bundled
local 384-dim embedder (HINDSIGHT_API_EMBEDDINGS_PROVIDER still unset).
Switching to vLLM's nomic-embed (768-dim) is a separate, destructive,
explicitly-approved-only migration (~1300 rows across ~20 banks) --
out of scope here, not part of the dashboard's 'stop llama-swap /
start vLLM' instruction.
Dashboard decision (t_5508360a): 'stop and disable llama-swap and start
vLLM and its 3 models' -- explicit approval of a breaking change.
Applied:
- llama-swap stopped + disabled on astro-orbiter (systemd unit removed
from multi-user.target.wants, files left in place -- full teardown is
t_6dff1ecc, separate task)
- vllm_service_enabled/state flipped to true/started -- vLLM is now the
permanent, boot-persistent serving layer (was shadow-only/staged)
- Attempted enabling Qwen3-8B-AWQ (the 3rd model) -- does NOT fit.
Qwen2.5-32B-Instruct-AWQ (~18.6GB) + nomic-embed (~0.8GB) leaves only
~1.25GiB free on the 23.55GiB usable budget, below the 3.53GiB floor
Qwen3-8B-AWQ needs even at gpu_memory_utilization=0.15. Confirmed via
journalctl: identical ValueError on 7/7 consecutive restart attempts,
not a transient crash-loop. Reverted Qwen3-8B-AWQ to enabled: false.
2 of the 3 requested models fit permanently, not 3.
Verified live: /health 200 on both :8000 and :8020, live completion and
live embedding both returned correct real output, NRestarts=0 on both
services after a clean idempotent re-run (changed=0).
Critical finding: flipping vllm_service_enabled/state=true/started and
restarting llama-swap alongside it broke llama-swap's ability to load
ANY of its own generative models -- every /v1/chat/completions request
against Qwen3.8-27B-Q4_K_M or Qwen3-8B aux models failed with
'upstream command exited prematurely' (llama-server OOM at spawn,
~1.8GB free on this 24GB card once vLLM's ~22.8GB was claimed).
Confirmed by direct A/B: same request 500s with vLLM running, 200s
seconds after stopping it.
This breaks 21 Hermes agent profiles' aux-model tasks (skills_hub,
approval, mcp, title_generation, profile_describer, compression) plus
OpenViking's VLM -- a far larger blast radius than Hindsight's single
LLM endpoint. Reverted:
- vllm_service_enabled/state back to role defaults (false/stopped) --
vLLM stays staged, startable for a brief validated shadow window,
NOT safe to leave resident in production.
- Hindsight's HINDSIGHT_API_LLM_BASE_URL back to llama-swap
(astro-orbiter:8001, Qwen3.8-27B-Q4_K_M) and the API key secret
source back to the Nous fallback item (pre-task state) --
the vLLM cutover, while functionally validated in isolation
(health, /v1/chat/completions, and a live hindsight_retain+recall
round-trip all succeeded), requires continuous vLLM availability
which is now known to be unsafe on this card.
Comment posted on t_6dff1ecc: teardown remains correctly blocked --
full cutover is not achievable within this card's VRAM budget as
currently scoped. Needs a human decision on aux-model migration
strategy (see roles/deploy-vllm README's 'Critical architectural
finding' section) before any further progress.
Default 64000 exceeded vLLM Qwen2.5-32B-Instruct-AWQ's --max-model-len
8192, causing every retain call to 500 with
'max_tokens=64000 cannot be greater than max_model_len=8192'.
llama-swap's Qwen3.8-27B ran at ctx=65536 so this never surfaced
before the vLLM cutover. Lowered to 4096.
HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai pointed at vLLM's
nomic-embed-text-v1.5 (768-dim) crash-looped hindsight-api:
'Cannot change embedding dimension from 384 to 768: memory_units
table contains 1289 rows with embeddings.' Hindsight was never
actually using astro-orbiter for embeddings (defaults to a bundled
local 384-dim sentence-transformers model when the env var is unset)
-- this was a genuine architecture assumption error, not a config
typo. Re-embedding all existing memory data across ~20 agent banks is
a destructive, irreversible operation requiring its own explicit,
approved task -- not something to trigger as a side effect of an
infra migration. Keeps the LLM cutover (astro-orbiter:8000, vLLM
Qwen2.5-32B-Instruct-AWQ) which is safe and already validated.
- vllm.service.j2: branch on role==embedding for --runner pooling
--convert embed, --no-enable-prefix-caching, per-model
trust_remote_code toggle (needed for nomic-embed-text-v1.5's custom
NomicBertModel code), and enforce_eager toggle (needed to avoid CUDA
graph capture OOM when co-resident with another vLLM process on this
24GB card).
- tasks/verify.yml: split completions vs embedding smoke tests --
embedding-mode instances don't serve /v1/completions. Assert a
non-empty embedding vector, not just HTTP 200.
- host_vars/astro-orbiter: enable nomic-embed-text-v1.5 (port 8020),
lower primary model's gpu_memory_utilization 0.95->0.90 + add
enforce_eager after finding 0.95 crash-looped 6-7x before stabilizing
with co-resident nomic-embed (real fix, confirmed via NRestarts=0
after clean stop/start, not luck).
- Hindsight (values.yaml + externalsecret.yaml): cut LLM + embeddings
over to vLLM (:8000, :8020), wire the previously-unset
HINDSIGHT_API_EMBEDDINGS_* env vars for the first time, and swap the
API key secret source from the Nous fallback item to vllm/api-key
(vLLM enforces real auth, llama-swap did not).
- README: document the embedding-mode branch, VRAM findings, and a
genuine architecture gap -- vLLM's one-model-per-process design
cannot replace llama-swap's 5-model LRU roster on this 24GB card, so
21 Hermes profiles' aux-model consumers (Qwen3-8B-no_think,
Phi-3.5-mini, Meta-Llama-3.1-8B, Qwen2.5-Coder-14B) and OpenViking's
VLM stay on llama-swap. Full teardown (t_6dff1ecc) needs a human
decision on the aux-model strategy before it can proceed.
Idempotent vLLM OpenAI-compatible serving role, staged-first (does not
start/enable the systemd unit or touch production traffic by default).
Validated end-to-end against astro-orbiter in a brief shadow window
(llama-swap stopped ~5 min, per homelab-llm-inference skill's documented
shadow-validation pattern):
- /health 200, /v1/models returns Qwen2.5-32B-Instruct-AWQ,
/v1/completions live smoke test passes, clean journalctl
- 3 consecutive full-role runs confirmed changed=0 (idempotent)
- production restored: llama-swap active, /v1/embeddings against
nomic-embed-text-v1.5 confirmed still working (Hindsight retain path)
Deviates from the original spec's model choices (Qwen2.5-32B-Instruct /
Qwen3-8B-Instruct bf16) to use the official Qwen AWQ pre-quantized variants
instead -- vLLM does not do safe on-the-fly quantization on this host
(bitsandbytes OOM history) and unquantized bf16 32B does not fit 24GB VRAM.
Two real bugs found+fixed during first-start validation (systemd-only
repro, not visible via interactive SSH testing):
1. ninja not on systemd's minimal PATH -> vLLM torch.compile
FileNotFoundError. Fixed via explicit PATH env in the unit.
2. FlashInfer sampler JIT fails to compile on RTX 3090 (SM86) --
known upstream issue class (vLLM GH #23023, #44305). Fixed via
VLLM_USE_FLASHINFER_SAMPLER=0 (falls back to native sampler).
Also fixed a real idempotency bug: force-upgrading setuptools to latest
fought with vLLM's own setuptools<81.0.0 pin, causing an install/downgrade
flip-flop (changed:true) on every run.
vllm_service_enabled defaults to false -- a host reboot must not
auto-start vLLM and VRAM-collide with the still-live llama-swap production
service. Cutover (enabling + starting + migrating consumers) is an
explicit, separate step outside this role, gated on adding embedding-mode
support (--task embed) for nomic-embed-text-v1.5, which this role does
not yet implement (Hindsight retain still depends on llama-swap's
nomic-embed until that follow-up lands).
Role: roles/deploy-vllm/ (defaults/handlers/meta/tasks/templates/README)
Playbook: playbooks/day1_deploy_vllm.yml
t_d7f8cd65 — astro-orbiter is a single serial llama-swap process; Hindsight's
default LLM_MAX_CONCURRENT=32 burst N parallel retains -> node rejects/times
out extras -> 502. Cap global + retain concurrency to 1 (native semaphore
config, no code change). Raise client timeout 120->600s and ingress
proxy-read/send 300->600s so a single long retain (~90s under load) survives.
stepfun rejects Hindsight tagged fact-extraction (400 'missing tags');
solar-pro4:free is the previously-verified-good Nous model for retain/recall
(t_e3375410 / t_d0dffc3d). base_url + provider unchanged.
Scoped to the hindsight ingress (cosmic-rewind.local.mk-labs.cloud) only.
A slow agentic reflect (~132s on solar-pro4) exceeded the 60s default
proxy-read-timeout, causing nginx to return 504 before the app delivered 200.
Raising read+send to 300s lets the response land. No global configmap change.
Refs t_e3375410.
astro-orbiter llama-server-router retired :8002 on 2026-08-18
(llama-swap now serves the OpenAI-compatible API on :8001).
OpenViking still pointed at the dead :8002, so every search/find
call failed: dense embedding -> Connection error -> 500 INTERNAL
server error on /api/v1/search/search and /api/v1/search/find.
Verified from the pod: :8002 = connection refused,
:8001/v1/embeddings (nomic-embed-text-v1.5) = 200 with vectors.
Repoints embedding.dense.api_base and vlm.api_base to :8001.
t_abc05db5
Resolved conflicts between local bafd76a (t_c5cef2b2, Qwen3-8B single-variant)
and origin/main 5cf4468 (t_664289a0, Qwen3-8B dual-thinking deployment).
Conflict resolution strategy: took origin/main version throughout — it is the
authoritative result from t_664289a0 which deployed the live no_think config to
astro-orbiter and already reflects the correct production state.
Changes incorporated from origin/main:
- defaults/main.yml: Qwen3-8B-Q4_K_M-no_think model entry (port 8107, n_gpu_layers=99,
chat_template_file), row6 matrix entry, dual-thinking comment block.
- templates/llama-server-router-preset.ini.j2: [Qwen3-8B-Q4_K_M] with sleep-idle-seconds=60
plus new [Qwen3-8B-Q4_K_M-no_think] section with chat-template-file directive.
- templates/qwen3-no-think.jinja.j2: new file — Qwen3 template with enable_thinking=false.
- tasks/models.yml: template deploy task for qwen3-no-think.jinja (chat_templates tag).
- tasks/swapmode.yml: GATE 2 assert updated to 7 models.
- templates/llama-swap-config.yaml.j2: chat_template_file flag support.
Also pulled in monitoring defaults (llm_monitoring_enabled, VRAM exporter settings,
Grafana dashboard vars, Prometheus scrape config) from origin/main monitoring branch.
- Remove global --n-gpu-layers from router unit ExecStart in preset mode
(llama.cpp CLI arg outranked per-model INI n-gpu-layers=0; root cause from
War Machine's run 1). Flag now emitted only in --models-dir mode.
- All 5 preset INI sections carry explicit n-gpu-layers:
Qwen3.8=99, Phi=99, nomic=99, Coder=0, Llama=0.
- host_vars/astro-orbiter: llm_router_models_max 2 -> 4 so CPU-offloaded
models count as loaded without LRU-evicting Qwen3.8.
- defaults: llm_router_coder_gpu_layers / llm_router_llama_gpu_layers = 0.
- verify.yml: fix pre-existing .meta attribute crash in router mode.
- New playbook day2_cpu_offload_aux_models.yml.
Deployed + verified on astro-orbiter (gates A-E PASS): concurrent residency
achieved, Qwen3.8 stays GPU-resident. Measured CPU throughput Llama 9.0 /
Coder 4.7 tok/s. VRAM note: llama.cpp 6ea215d allocates ~1.4-1.7GB CUDA-context
per CPU model even at n-gpu-layers=0 -> ~24,004 MiB steady-state, below the
24,576 MiB physical limit. Comments corrected to match the measurement.
Report: friday/inbox/ryan/2026-08-17-llm-cpu-offload-coder-llama-deployed.md
Ryan-directed model swap (kanban t_f5f7e9ad, 2026-08-16).
Changes:
- Replace [Qwen3.6-35B-A3B-UD-Q4_K_S] with [Qwen3.8-27B-Q4_K_M] in
llama-server-router-preset.ini.j2 (production model slot).
- Qwen3.8-27B: dense 27B VLM, Apache-2.0, Alibaba Aug 2026.
Unsloth Dynamic V3.0 GGUF quantization.
Q4_K_M chosen: 17,106,775,008 bytes, 17.1GB. Measured VRAM: 17,068 MiB
at ctx=32768 (q4_0 KV cache).
- ctx-size set to 32768 (32K) via new variable llm_router_qwen38_ctx_size.
Native context is 262K; 32K chosen to maintain eviction headroom on 24GB RTX 3090.
- models-max reduced 4 -> 2 in host_vars. Qwen3.8 (17.6GB) + nomic-embed
(558MB) exhaust the 24GB card; no auxiliary model can co-reside with Qwen3.8.
LRU eviction handles model switching with ~30-60s cold-load latency.
- llm_router_expected_model_id updated to Qwen3.8-27B-Q4_K_M.
- Qwen3.6 GGUF retained at /opt/models/Qwen3.6-35B-A3B-UD-Q4_K_S.gguf
(not deleted — pending stable period and explicit cleanup task).
- day2_swap_qwen38.yml playbook added for Ansible idempotent redeployment.
Architecture note: Qwen3.8 uses Gated DeltaNet; llama.cpp 6ea215d logs
'fused Gated Delta Net (chunked) not supported, set to disabled'. Inference
works correctly on the non-fused fallback. A llama.cpp update may improve
throughput on the GDN layers.
Smoke test passed: model responded via router endpoint (http://10.1.71.130:8002).
VRAM: 17,630 MiB (Qwen3.8) + 5,928 MiB (Llama-8B concurrent) = 23,558 MiB.
Also commits accumulated but unpushed changes:
- nomic-embed batch-size/rope-scaling fix (t_openviking_embed_batch)
- per-model ctx-size day2 playbook (day2_per_model_ctx_size.yml)
- llama-server-router.service.j2 minor update
- vlm.model was 'llama3.1-8b' which doesn't exist on astro-orbiter's
/v1/models, causing every summarization call to 400 and endless
circuit-breaker retries. Correct id: Meta-Llama-3.1-8B-Instruct-Q4_K_M.
- embedding.max_input_tokens=1536 still let chunks through that actually
tokenized to 2000-2860 real tokens (estimator undercounts vs llama.cpp's
tokenizer by 1.35x-1.86x on this corpus). Lowered to 1024 for real margin
under the 2048 n_ctx ceiling.
astro-orbiter's llama.cpp router hard-caps nomic-embed-text-v1.5 effective
context at 2048 tokens regardless of ctx-size (known nomic-bert/RoPE limitation
in llama.cpp, not fixable server-side). OpenViking chunks observed at
2000-3400 tokens were tripping 400 exceed_context_size_error and endless
circuit-breaker re-enqueue for viking://temp/default/08140552_5f1c9e/homelab.tar/*.
Set embedding.max_input_tokens: 1536 (well under 2048) since OpenViking's
chunk-time token estimator uses a different tokenizer than llama.cpp's context
counter, so token counts won't match 1:1 - 1536 leaves ~25% headroom.
Approved by Ryan as lowest-risk mitigation (does not touch astro-orbiter/
llama.cpp serving config, which is War Machine's domain and already fixed
separately for the ubatch-size issue).
- Build/push image: the-seas.local.mk-labs.cloud/library/maelstrom-ui:v0.3.17-1
(upstream volcengine/openviking web-studio/, pinned to commit 3cd1d4e9)
- Deployment + Service serving the static SPA via nginx (reverse-proxies
/api, /health, /ready to openviking backend; /bot deliberately NOT proxied)
- Ingress at maelstrom.local.mk-labs.cloud (TLS via letsencrypt-internal)
- ExternalSecret wiring scoped maelstrom-ui-key from
op://mk-labs/openviking/maelstrom-ui-key into the pod env (MAELSTROM_UI_KEY)
Per approved plan: inbox/ryan/2026-08-14-maelstrom-ui-deployment-plan.md
Key mint + approval: system/inbox/agents/nick-fury/2026-08-14-maelstrom-ui-key-mint-complete.md
Ryan approval: inbox/ryan/2026-08-14-maelstrom-key-approval.md
