Idempotent vLLM OpenAI-compatible serving role, staged-first (does not
start/enable the systemd unit or touch production traffic by default).
Validated end-to-end against astro-orbiter in a brief shadow window
(llama-swap stopped ~5 min, per homelab-llm-inference skill's documented
shadow-validation pattern):
- /health 200, /v1/models returns Qwen2.5-32B-Instruct-AWQ,
/v1/completions live smoke test passes, clean journalctl
- 3 consecutive full-role runs confirmed changed=0 (idempotent)
- production restored: llama-swap active, /v1/embeddings against
nomic-embed-text-v1.5 confirmed still working (Hindsight retain path)
Deviates from the original spec's model choices (Qwen2.5-32B-Instruct /
Qwen3-8B-Instruct bf16) to use the official Qwen AWQ pre-quantized variants
instead -- vLLM does not do safe on-the-fly quantization on this host
(bitsandbytes OOM history) and unquantized bf16 32B does not fit 24GB VRAM.
Two real bugs found+fixed during first-start validation (systemd-only
repro, not visible via interactive SSH testing):
1. ninja not on systemd's minimal PATH -> vLLM torch.compile
FileNotFoundError. Fixed via explicit PATH env in the unit.
2. FlashInfer sampler JIT fails to compile on RTX 3090 (SM86) --
known upstream issue class (vLLM GH #23023, #44305). Fixed via
VLLM_USE_FLASHINFER_SAMPLER=0 (falls back to native sampler).
Also fixed a real idempotency bug: force-upgrading setuptools to latest
fought with vLLM's own setuptools<81.0.0 pin, causing an install/downgrade
flip-flop (changed:true) on every run.
vllm_service_enabled defaults to false -- a host reboot must not
auto-start vLLM and VRAM-collide with the still-live llama-swap production
service. Cutover (enabling + starting + migrating consumers) is an
explicit, separate step outside this role, gated on adding embedding-mode
support (--task embed) for nomic-embed-text-v1.5, which this role does
not yet implement (Hindsight retain still depends on llama-swap's
nomic-embed until that follow-up lands).
Role: roles/deploy-vllm/ (defaults/handlers/meta/tasks/templates/README)
Playbook: playbooks/day1_deploy_vllm.yml
t_d7f8cd65 — astro-orbiter is a single serial llama-swap process; Hindsight's
default LLM_MAX_CONCURRENT=32 burst N parallel retains -> node rejects/times
out extras -> 502. Cap global + retain concurrency to 1 (native semaphore
config, no code change). Raise client timeout 120->600s and ingress
proxy-read/send 300->600s so a single long retain (~90s under load) survives.
stepfun rejects Hindsight tagged fact-extraction (400 'missing tags');
solar-pro4:free is the previously-verified-good Nous model for retain/recall
(t_e3375410 / t_d0dffc3d). base_url + provider unchanged.
Scoped to the hindsight ingress (cosmic-rewind.local.mk-labs.cloud) only.
A slow agentic reflect (~132s on solar-pro4) exceeded the 60s default
proxy-read-timeout, causing nginx to return 504 before the app delivered 200.
Raising read+send to 300s lets the response land. No global configmap change.
Refs t_e3375410.
astro-orbiter llama-server-router retired :8002 on 2026-08-18
(llama-swap now serves the OpenAI-compatible API on :8001).
OpenViking still pointed at the dead :8002, so every search/find
call failed: dense embedding -> Connection error -> 500 INTERNAL
server error on /api/v1/search/search and /api/v1/search/find.
Verified from the pod: :8002 = connection refused,
:8001/v1/embeddings (nomic-embed-text-v1.5) = 200 with vectors.
Repoints embedding.dense.api_base and vlm.api_base to :8001.
t_abc05db5
Resolved conflicts between local bafd76a (t_c5cef2b2, Qwen3-8B single-variant)
and origin/main 5cf4468 (t_664289a0, Qwen3-8B dual-thinking deployment).
Conflict resolution strategy: took origin/main version throughout — it is the
authoritative result from t_664289a0 which deployed the live no_think config to
astro-orbiter and already reflects the correct production state.
Changes incorporated from origin/main:
- defaults/main.yml: Qwen3-8B-Q4_K_M-no_think model entry (port 8107, n_gpu_layers=99,
chat_template_file), row6 matrix entry, dual-thinking comment block.
- templates/llama-server-router-preset.ini.j2: [Qwen3-8B-Q4_K_M] with sleep-idle-seconds=60
plus new [Qwen3-8B-Q4_K_M-no_think] section with chat-template-file directive.
- templates/qwen3-no-think.jinja.j2: new file — Qwen3 template with enable_thinking=false.
- tasks/models.yml: template deploy task for qwen3-no-think.jinja (chat_templates tag).
- tasks/swapmode.yml: GATE 2 assert updated to 7 models.
- templates/llama-swap-config.yaml.j2: chat_template_file flag support.
Also pulled in monitoring defaults (llm_monitoring_enabled, VRAM exporter settings,
Grafana dashboard vars, Prometheus scrape config) from origin/main monitoring branch.
- Remove global --n-gpu-layers from router unit ExecStart in preset mode
(llama.cpp CLI arg outranked per-model INI n-gpu-layers=0; root cause from
War Machine's run 1). Flag now emitted only in --models-dir mode.
- All 5 preset INI sections carry explicit n-gpu-layers:
Qwen3.8=99, Phi=99, nomic=99, Coder=0, Llama=0.
- host_vars/astro-orbiter: llm_router_models_max 2 -> 4 so CPU-offloaded
models count as loaded without LRU-evicting Qwen3.8.
- defaults: llm_router_coder_gpu_layers / llm_router_llama_gpu_layers = 0.
- verify.yml: fix pre-existing .meta attribute crash in router mode.
- New playbook day2_cpu_offload_aux_models.yml.
Deployed + verified on astro-orbiter (gates A-E PASS): concurrent residency
achieved, Qwen3.8 stays GPU-resident. Measured CPU throughput Llama 9.0 /
Coder 4.7 tok/s. VRAM note: llama.cpp 6ea215d allocates ~1.4-1.7GB CUDA-context
per CPU model even at n-gpu-layers=0 -> ~24,004 MiB steady-state, below the
24,576 MiB physical limit. Comments corrected to match the measurement.
Report: friday/inbox/ryan/2026-08-17-llm-cpu-offload-coder-llama-deployed.md
Ryan-directed model swap (kanban t_f5f7e9ad, 2026-08-16).
Changes:
- Replace [Qwen3.6-35B-A3B-UD-Q4_K_S] with [Qwen3.8-27B-Q4_K_M] in
llama-server-router-preset.ini.j2 (production model slot).
- Qwen3.8-27B: dense 27B VLM, Apache-2.0, Alibaba Aug 2026.
Unsloth Dynamic V3.0 GGUF quantization.
Q4_K_M chosen: 17,106,775,008 bytes, 17.1GB. Measured VRAM: 17,068 MiB
at ctx=32768 (q4_0 KV cache).
- ctx-size set to 32768 (32K) via new variable llm_router_qwen38_ctx_size.
Native context is 262K; 32K chosen to maintain eviction headroom on 24GB RTX 3090.
- models-max reduced 4 -> 2 in host_vars. Qwen3.8 (17.6GB) + nomic-embed
(558MB) exhaust the 24GB card; no auxiliary model can co-reside with Qwen3.8.
LRU eviction handles model switching with ~30-60s cold-load latency.
- llm_router_expected_model_id updated to Qwen3.8-27B-Q4_K_M.
- Qwen3.6 GGUF retained at /opt/models/Qwen3.6-35B-A3B-UD-Q4_K_S.gguf
(not deleted — pending stable period and explicit cleanup task).
- day2_swap_qwen38.yml playbook added for Ansible idempotent redeployment.
Architecture note: Qwen3.8 uses Gated DeltaNet; llama.cpp 6ea215d logs
'fused Gated Delta Net (chunked) not supported, set to disabled'. Inference
works correctly on the non-fused fallback. A llama.cpp update may improve
throughput on the GDN layers.
Smoke test passed: model responded via router endpoint (http://10.1.71.130:8002).
VRAM: 17,630 MiB (Qwen3.8) + 5,928 MiB (Llama-8B concurrent) = 23,558 MiB.
Also commits accumulated but unpushed changes:
- nomic-embed batch-size/rope-scaling fix (t_openviking_embed_batch)
- per-model ctx-size day2 playbook (day2_per_model_ctx_size.yml)
- llama-server-router.service.j2 minor update
- vlm.model was 'llama3.1-8b' which doesn't exist on astro-orbiter's
/v1/models, causing every summarization call to 400 and endless
circuit-breaker retries. Correct id: Meta-Llama-3.1-8B-Instruct-Q4_K_M.
- embedding.max_input_tokens=1536 still let chunks through that actually
tokenized to 2000-2860 real tokens (estimator undercounts vs llama.cpp's
tokenizer by 1.35x-1.86x on this corpus). Lowered to 1024 for real margin
under the 2048 n_ctx ceiling.
astro-orbiter's llama.cpp router hard-caps nomic-embed-text-v1.5 effective
context at 2048 tokens regardless of ctx-size (known nomic-bert/RoPE limitation
in llama.cpp, not fixable server-side). OpenViking chunks observed at
2000-3400 tokens were tripping 400 exceed_context_size_error and endless
circuit-breaker re-enqueue for viking://temp/default/08140552_5f1c9e/homelab.tar/*.
Set embedding.max_input_tokens: 1536 (well under 2048) since OpenViking's
chunk-time token estimator uses a different tokenizer than llama.cpp's context
counter, so token counts won't match 1:1 - 1536 leaves ~25% headroom.
Approved by Ryan as lowest-risk mitigation (does not touch astro-orbiter/
llama.cpp serving config, which is War Machine's domain and already fixed
separately for the ubatch-size issue).
- Build/push image: the-seas.local.mk-labs.cloud/library/maelstrom-ui:v0.3.17-1
(upstream volcengine/openviking web-studio/, pinned to commit 3cd1d4e9)
- Deployment + Service serving the static SPA via nginx (reverse-proxies
/api, /health, /ready to openviking backend; /bot deliberately NOT proxied)
- Ingress at maelstrom.local.mk-labs.cloud (TLS via letsencrypt-internal)
- ExternalSecret wiring scoped maelstrom-ui-key from
op://mk-labs/openviking/maelstrom-ui-key into the pod env (MAELSTROM_UI_KEY)
Per approved plan: inbox/ryan/2026-08-14-maelstrom-ui-deployment-plan.md
Key mint + approval: system/inbox/agents/nick-fury/2026-08-14-maelstrom-ui-key-mint-complete.md
Ryan approval: inbox/ryan/2026-08-14-maelstrom-key-approval.md
The letsencrypt-internal ClusterIssuer does not exist on the cluster.
Only letsencrypt-prod and letsencrypt-staging are available.
Both use DNS-01 via Cloudflare for the mk-labs.cloud zone,
so they work for internal-only hosts with no public HTTP reachability.
Fixes: https://github.com/volcengine/openviking/issues/...
Closes: kanban task t_1c2cc2db
Bug 1: ExternalSecret referenced three separate 1Password items
(openviking-root-api-key, openviking-embedding-api-key, openviking-vlm-api-key)
but Ryan created ONE item 'openviking' with three fields inside.
Changed all remoteRef.key values to 'openviking' and corrected field property names.
Bug 2: values.yaml had two invalid embedding config fields:
- encoding_format: 'float' (not in upstream schema, removed)
- max_concurrent under embedding.dense (wrong nesting, moved to embedding level)
Verified against upstream chart schema at github.com/volcengine/openviking
Remove resource-level sync-wave annotations that caused ArgoCD deadlock.
The wave 8 annotation was meant for platform-level ordering (apps-of-apps)
but was incorrectly applied to individual resources within the Application.
This caused ArgoCD to apply Deployment (wave 0) before ExternalSecret (wave 8),
resulting in CreateContainerConfigError since the pod needed the secret first.
Changes:
- namespace.yaml: Remove sync-wave annotation, add clarifying comment
- externalsecret.yaml: Change sync-wave from 8 to -1 (must sync before Deployment at wave 0)
- application.yaml: Remove sync-wave annotation, document as platform-level only in comments
This ensures:
1. ExternalSecret syncs first (wave -1)
2. Deployment uses it immediately (implicit wave 0)
3. No deadlock
Task: t_3906c41a
- Changed source 1 from 'chart: deploy/helm/openviking' to 'path: deploy/helm/openviking'
- ArgoCD multi-source now correctly resolves the Helm chart from the git repo
- targetRevision: main now correctly refers to a git branch, not a chart version
- Fixes: invalid revision 'main': improper constraint error
Platform Knowledge Infrastructure pilot - context database for large
file trees, shared skills, and long-term logs to reduce agent token
consumption. Pilot scope: two corpora (hermes/ skills library,
personal/homelab/), two consumer profiles (Wong, Shuri) for before/after
token comparison.
- namespace.yaml: openviking namespace, sync-wave 8 (after Harbor wave 7)
- externalsecret.yaml: credentials from 1Password via onepassword-connect
ClusterSecretStore (Wong, t_32766900)
- values.yaml: Helm overrides - px-fa-direct-access storage (30Gi),
embedding (nomic-embed-text-v1.5) + VLM (Llama-3.1-8B) via astro-orbiter
router (:8002), internal-only ingress
- application.yaml: multi-source ArgoCD Application, Harbor pattern
(Peter Parker, t_eefdcc17 + reconciled in t_3e54efa8)
Prerequisites verified complete before this commit:
- nomic-embed-text-v1.5-Q4_K_M live on astro-orbiter router (War Machine,
t_34b96e83, commit ad70b34)
- All 3 1Password items provisioned (root/embedding/vlm api keys)
- Storage class corrected to px-fa-direct-access after live PV audit
showed pure-block/pure-file have zero provisioned volumes (t_77b3ff79)
- Dry-run validated against live cluster prior to commit
Constraint: vault (~/friday) remains canonical source of truth; OpenViking
index is a derived cache, rebuilt from vault source files.
Honcho/lincoln explicitly out of scope for this work.
/metrics?model=Qwen3.6 forces the router to attempt loading Qwen3.6 every
90s scrape cycle, triggering a CUDA OOM error since VRAM is already consumed
by the resident Llama3+Phi3.5 models. This produces real GPU power spikes
(~110W load-attempt), not a benign counter read like the Llama3/Phi3.5 jobs.
nvidia_gpu_exporter (:9835) already provides GPU power/VRAM/utilization at
zero wake cost. No Grafana dashboard panel references Qwen3.6 model-specific
llama-server metrics. Ryan approved full removal.
Job is commented out (not deleted) for easy revert if Qwen3.6 is ever
re-added as a resident model. Llama3/Phi3.5 scrape jobs untouched.
Each /metrics?model=<id> request on the llama.cpp router wakes the GPU
sub-server to P2 (~110W). At 15s with two active jobs (llama3 + phi35),
combined scrape frequency (~7-8s effective) keeps the GPU continuously at
P2 despite zero real inference requests.
At 90s: each scrape wakes GPU for ~5-10s then it drops to P8 (~20W) for
~80s. Verified via iptables block test on 2026-08-13 (t_e7d547ea):
Before block: 110-115W P2 continuously
After block: 19-21W P8 consistently
After unblock: returned to 110W P2 within seconds
qwen3 interval also set to 90s (model not loaded so moot, but consistent).
Ref: t_e7d547ea
Port 8000 (gemma-2-27b-it-GGUF) is dead after the day2 router cutover on
2026-08-12. Production inference now runs through llama-server-router on
port 8002.
The router exposes per-model Prometheus metrics via /metrics?model=<id>.
Since a single /metrics request without ?model returns HTTP 400, replaced
the single stale job with three per-model jobs — one per model registered
in the router per /v1/models:
- Qwen3.6-35B-A3B-UD-Q4_K_S (currently unloaded but registered)
- Meta-Llama-3.1-8B-Instruct-Q4_K_M (loaded)
- Phi-3.5-mini-instruct-Q8_0 (loaded)
Static 'model' label carries the canonical llama.cpp model id (not alias).
Added 'endpoint: astro-orbiter-router' to identify the scrape origin.
Removed dead :8000 target entirely.
- host_vars/astro-orbiter/vars.yml: add Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf
to llm_staged_models (size_bytes=8988111072, bartowski GGUF public repo).
Updated VRAM note to reflect 4-model roster and LRU eviction semantics.
- defaults/main.yml: add llm_router_coder_ctx_size=16384 and
llm_router_coder_flash_attn=true variables for per-model ctx tuning.
- templates/llama-server-router-preset.ini.j2: add [Qwen2.5-Coder-14B-Instruct-Q4_K_M]
section with alias=Qwen2.5-Coder-14B-Instruct-4bit, ctx-size=16384, flash-attn=true.
- playbooks/day2_add_coder_alias.yml: new playbook that downloads the GGUF (if
absent/mismatched), deploys updated preset INI and systemd unit, restarts
llama-server-router, and verifies all 4 models in /v1/models.
VRAM: Coder ~9GB. Full 4-model co-residency impossible on 24GB — LRU eviction
handles this automatically. Qwen3.6-35B <-> Coder switches incur ~30-60s cold load.
- Add Meta-Llama-3.1-8B-Instruct-4bit alias on Meta-Llama-3.1-8B-Instruct-Q4_K_M
entry in the preset INI (per Ryan dashboard note).
- Document known issue: Phi-3.5-mini json_schema grammar sampler incompatibility
in router mode (GH #23460 variant — chat template token format mismatch).
Meta-Llama works with json_schema response_format; confirmed via live test.
- Phi-3.5-mini-instruct-8bit alias remains working for model routing;
structured output (json_schema) fails due to the model's token format.
Changes:
- host_vars/astro-orbiter/vars.yml: add llm_router_models_max: 4 (overrides
conservative default of 1). Detailed VRAM OOM risk note included inline:
worst-case 3-model co-residency ~31GB > 24GB RTX 3090. LRU eviction
mitigates in single-user operation; flagged for Ryan's review.
- playbooks/day2_bump_router_models_max.yml: new targeted playbook; deploys
updated router unit, restarts the live service, verifies /health 200 and
/v1/models lists all 3 GGUFs post-restart.
- group_vars/all/semaphore.yml: add llm_router_update_unit template pointing
at the new playbook.
- roles/llm-inference-multimodel/defaults/main.yml: update comment to reflect
the var is now overridden in host_vars rather than 'hardcoded to 1'.
- roles/llm-inference-multimodel/templates/llama-server-router.service.j2:
correct stale 'HARDCODED TO 1' comment — value is variable-driven.
Constraints honored:
- --parallel 1 left untouched (not in scope, not modified anywhere)
- No ad-hoc SSH/systemctl/curl state mutation; all execution via Semaphore
- No installed/vendored code patched
Adds idempotent, data-driven GGUF staging for the two new router models on
astro-orbiter alongside the production Qwen3.6-35B-A3B-UD-Q4_K_S. Both files
were already staged live (byte-exact); this commit codifies them in Ansible so
future re-runs and any new model adds are version-controlled and audit-friendly.
Changes:
- roles/llm-inference-multimodel/tasks/stage_model.yml (NEW)
Idempotent per-model task: stat -> exact byte-size guard -> conditional
get_url -> ownership/mode ensure -> notify router restart handler only on
actual download. Loops from models.yml; nothing hardcoded.
- roles/llm-inference-multimodel/tasks/models.yml
Appends the stage_model.yml loop (tagged: models) after the existing Qwen3.6
download tasks. Data driven from host_vars/astro-orbiter/vars.yml.
- roles/llm-inference-multimodel/defaults/main.yml
Adds llm_staged_models: [] default (empty = safe no-op for hosts with no
staged model list defined).
- roles/llm-inference-multimodel/handlers/main.yml
Adds 'restart llama-server-router on new GGUF' handler. Only fires when
stage_model.yml performs an actual download or corrects ownership/mode.
Normal idempotent re-runs (files already correct) do NOT fire this handler.
- host_vars/astro-orbiter/vars.yml
Adds llm_staged_models list with the two new models:
* Phi-3.5-mini-instruct-Q8_0.gguf (4,061,222,688 bytes,
bartowski/Phi-3.5-mini-instruct-GGUF)
* Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf (4,920,739,232 bytes,
bartowski/Meta-Llama-3.1-8B-Instruct-GGUF)
- playbooks/day1_deploy_llm_inference_multimodel.yml
Updates header comment: removes stale 'Semaphore broken' note, documents
the correct execution channel (Semaphore template
llm_inference_multimodel_stage_models, --tags models).
- group_vars/all/semaphore.yml
Adds llm_inference_multimodel_stage_models template entry (config-as-code).
Template is scoped to --tags models explicitly. Phase 4 (verify) is
EXCLUDED: verify.yml starts llama-server-qwen on :8002, which would collide
with the production llama-server-router.service already running on :8002.
Semaphore template created via API: project 1 / template id 19.
Execution: triggered immediately after this commit via Semaphore REST API.
Promote llama-server-router to production on port 8002.
- Stops + disables llama-server-qwen (unit preserved as rollback target)
- Redeploys llama-server-router unit with --port 8002 (not 8003)
- PORT DECISION: rebind router to :8002 — zero Hermes profile config changes needed
- Re-runs validation gates 1-3 against production endpoint (hard gates)
- Gate 4: bundled SvelteKit UI check (HTTP 415 from ansible.builtin.uri is
a false negative — llama.cpp requires Accept-Encoding: gzip; real browsers work)
- Rollback via --tags cutover_rollback (uses 'never' Ansible tag to prevent
accidental execution during normal cutover run)
All 4 gates PASSED on cutover run 2026-08-12:
Gate 1: n_ctx=65536 >= 64000 PASS
Gate 2: finish_reason=tool_calls PASS
Gate 2b: finish_reason=stop (no spurious tool_calls) PASS
Gate 3: 20410 MiB / 23000 MiB ceiling PASS
Gate 4: SvelteKit HTML confirmed via curl + gunzip PASS
Production endpoint: llama-server-router on :8002 (enabled + running)
Rollback target: llama-server-qwen unit at /etc/systemd/system/ (stopped, disabled)
- Retire llama-server-aux (Phi-4, 8000) and llama-server-toolcall (Mistral-Small-24B, 8001): stopped, disabled, unit files removed from host and Ansible role
- Promote llama-server-qwen (Qwen2.5-14B-Instruct-1M, port 8002) to sole production model, serving both friday and war-machine Hermes profiles
- Verified live: n_ctx=65536/n_ctx_train=1010000, and tool_calls response via /v1/chat/completions probe (no hallucination)
- Deleted superseded GGUF weights (phi-4, mistral-small, orphaned base-Qwen, gemma-2-27b) from astro-orbiter, ~45GB reclaimed
- Updated friday and war-machine Hermes profile configs (model + compression + skills_hub aux) to point at 10.1.71.130:8002
- Ryan explicitly accepted single-model tradeoffs for both profiles
Mistral (llama-server-toolcall, 8001) stopped temporarily on astro-orbiter to
free ~6.2GB VRAM headroom for this test window per Ryan/JARVIS approval.
Not a permanent decommission of Mistral.
- New llama-server-qwen systemd unit template, gated by llm_qwen_service_enabled (default false)
- Idempotent GGUF download task (bartowski Qwen2.5-14B-Instruct-Q5_K_M, stat-guarded)
- Launch flags per local-llm-64k-context-recommendation.md: ctx-size 65536, flash-attn, q8_0 KV cache, batch 2048/ubatch 512, jinja, parallel 1
- verify.yml only starts/verifies the qwen unit when llm_qwen_service_enabled=true
- README: documents live VRAM gate finding (nvidia-smi 2026-08-06: Phi-4+Mistral already ~16.6/24GB, ~7.5GB free -- insufficient for Qwen weights concurrently) and options
- Does NOT touch llama-server-aux (8000) or llama-server-toolcall (8001) service state
