Set OPENAI_API_BASE_URL (config.py default) to our local vLLM endpoint.
Open WebUI was using hardcoded 'https://api.openai.com/v1' for model discovery.
ExternalSecret only provides vllm-api-key from 1Password.
Using same key for both purposes is acceptable for now.
Avoids need for manual secret patching.
The open-webui item doesn't exist in 1Password yet.
webui-secret-key is manually managed via Kubernetes secret patch.
Template now only pulls vllm-api-key from 1Password.
- Change emptyDir to persistent volume (open-webui-data PVC)
Fixes data loss on pod restart (admin user disappears)
- Add ENABLE_OLLAMA_API=false
Disable broken Ollama endpoint probing
vLLM uses OpenAI-compatible API, not Ollama
- Create pvc.yaml for persistent data storage
10Gi NFS-backed storage via nfs-emporium StorageClass
The 'open-webui' item doesn't exist in 1Password yet.
webui-secret-key is manually patched in the secret for now.
Re-enable ExternalSecret pull once the 1Password item is created.
- Add ExternalName Service for astro-orbiter (10.1.71.130:8000)
- Remove Ollama-specific configuration (OLLAMA_BASE_URLS)
- Keep only OpenAI-compatible configuration (OPENAI_BASE_URL)
Fixes: Open WebUI pod was attempting to use Ollama endpoints (/api/tags)
against vLLM which only supports OpenAI-compatible API (/v1/models).
Pod now reaches astro-orbiter via Kubernetes Service DNS name.
Resolves: HTTP 404 errors from Upstream ollama request failed
- ExternalSecret: use key/property format (not path) for 1Password reference
- Deployment: remove admin password env var (use single vllm-api-key)
- Syncs vLLM API key from op://mk-labs/vllm/api-key
astro-orbiter's vLLM primary model changed 2026-09-01 from
DeepSeek-R1-Distill-Qwen-32B-AWQ to Gemma-4-26B-A4B-it-AWQ (Google,
Apache 2.0, US-origin). Same endpoint/API key — only the served model
name changed. max_model_len also bumped to 65536 (was 32768).
Gemma 4 does not emit an always-on <think> reasoning trace like
DeepSeek-R1 did, so HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS=4096
should have more effective headroom for real extraction output than
before, not less.
astro-orbiter's vLLM primary model changed 2026-09-01 from
Qwen2.5-32B-Instruct-AWQ to DeepSeek-R1-Distill-Qwen-32B-AWQ
(single-model deployment, see ansible/roles/deploy-vllm/README.md).
Same endpoint (http://astro-orbiter:8000/v1) and API key — only the
served model name changed.
Flagged for follow-up, not fixed here: DeepSeek-R1 emits a
<think>...</think> reasoning trace before its final answer; Hindsight's
fact-extraction prompt/parsing has not been specifically verified
against this reasoning-model output shape. Watch extraction quality
after this rolls out; HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS left
at 4096 pending confirmation that's enough headroom for a real retain
to complete past the reasoning trace.
Companion commit to 6bfcc76 (vllm permanent residency cutover). Now that
llama-swap is stopped+disabled and vLLM is the permanent serving layer:
- HINDSIGHT_API_LLM_BASE_URL: llama-swap :8001 -> vLLM :8000
- HINDSIGHT_API_LLM_MODEL: Qwen3.8-27B-Q4_K_M -> Qwen2.5-32B-Instruct-AWQ
- Added HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS=4096 (vLLM's
max_model_len=8192 is far below the 64000 Hindsight retain default;
uncapped this 400s on long retains -- found in t_e6facb19's validation)
- ExternalSecret HINDSIGHT_API_LLM_API_KEY source: 1Password 'nous' item
(unused fallback) -> 1Password 'vllm' item's api-key (vLLM validates
its bearer token, unlike llama-swap which accepted anything)
Embeddings intentionally UNCHANGED -- Hindsight stays on its bundled
local 384-dim embedder (HINDSIGHT_API_EMBEDDINGS_PROVIDER still unset).
Switching to vLLM's nomic-embed (768-dim) is a separate, destructive,
explicitly-approved-only migration (~1300 rows across ~20 banks) --
out of scope here, not part of the dashboard's 'stop llama-swap /
start vLLM' instruction.
Critical finding: flipping vllm_service_enabled/state=true/started and
restarting llama-swap alongside it broke llama-swap's ability to load
ANY of its own generative models -- every /v1/chat/completions request
against Qwen3.8-27B-Q4_K_M or Qwen3-8B aux models failed with
'upstream command exited prematurely' (llama-server OOM at spawn,
~1.8GB free on this 24GB card once vLLM's ~22.8GB was claimed).
Confirmed by direct A/B: same request 500s with vLLM running, 200s
seconds after stopping it.
This breaks 21 Hermes agent profiles' aux-model tasks (skills_hub,
approval, mcp, title_generation, profile_describer, compression) plus
OpenViking's VLM -- a far larger blast radius than Hindsight's single
LLM endpoint. Reverted:
- vllm_service_enabled/state back to role defaults (false/stopped) --
vLLM stays staged, startable for a brief validated shadow window,
NOT safe to leave resident in production.
- Hindsight's HINDSIGHT_API_LLM_BASE_URL back to llama-swap
(astro-orbiter:8001, Qwen3.8-27B-Q4_K_M) and the API key secret
source back to the Nous fallback item (pre-task state) --
the vLLM cutover, while functionally validated in isolation
(health, /v1/chat/completions, and a live hindsight_retain+recall
round-trip all succeeded), requires continuous vLLM availability
which is now known to be unsafe on this card.
Comment posted on t_6dff1ecc: teardown remains correctly blocked --
full cutover is not achievable within this card's VRAM budget as
currently scoped. Needs a human decision on aux-model migration
strategy (see roles/deploy-vllm README's 'Critical architectural
finding' section) before any further progress.
Default 64000 exceeded vLLM Qwen2.5-32B-Instruct-AWQ's --max-model-len
8192, causing every retain call to 500 with
'max_tokens=64000 cannot be greater than max_model_len=8192'.
llama-swap's Qwen3.8-27B ran at ctx=65536 so this never surfaced
before the vLLM cutover. Lowered to 4096.
HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai pointed at vLLM's
nomic-embed-text-v1.5 (768-dim) crash-looped hindsight-api:
'Cannot change embedding dimension from 384 to 768: memory_units
table contains 1289 rows with embeddings.' Hindsight was never
actually using astro-orbiter for embeddings (defaults to a bundled
local 384-dim sentence-transformers model when the env var is unset)
-- this was a genuine architecture assumption error, not a config
typo. Re-embedding all existing memory data across ~20 agent banks is
a destructive, irreversible operation requiring its own explicit,
approved task -- not something to trigger as a side effect of an
infra migration. Keeps the LLM cutover (astro-orbiter:8000, vLLM
Qwen2.5-32B-Instruct-AWQ) which is safe and already validated.
- vllm.service.j2: branch on role==embedding for --runner pooling
--convert embed, --no-enable-prefix-caching, per-model
trust_remote_code toggle (needed for nomic-embed-text-v1.5's custom
NomicBertModel code), and enforce_eager toggle (needed to avoid CUDA
graph capture OOM when co-resident with another vLLM process on this
24GB card).
- tasks/verify.yml: split completions vs embedding smoke tests --
embedding-mode instances don't serve /v1/completions. Assert a
non-empty embedding vector, not just HTTP 200.
- host_vars/astro-orbiter: enable nomic-embed-text-v1.5 (port 8020),
lower primary model's gpu_memory_utilization 0.95->0.90 + add
enforce_eager after finding 0.95 crash-looped 6-7x before stabilizing
with co-resident nomic-embed (real fix, confirmed via NRestarts=0
after clean stop/start, not luck).
- Hindsight (values.yaml + externalsecret.yaml): cut LLM + embeddings
over to vLLM (:8000, :8020), wire the previously-unset
HINDSIGHT_API_EMBEDDINGS_* env vars for the first time, and swap the
API key secret source from the Nous fallback item to vllm/api-key
(vLLM enforces real auth, llama-swap did not).
- README: document the embedding-mode branch, VRAM findings, and a
genuine architecture gap -- vLLM's one-model-per-process design
cannot replace llama-swap's 5-model LRU roster on this 24GB card, so
21 Hermes profiles' aux-model consumers (Qwen3-8B-no_think,
Phi-3.5-mini, Meta-Llama-3.1-8B, Qwen2.5-Coder-14B) and OpenViking's
VLM stay on llama-swap. Full teardown (t_6dff1ecc) needs a human
decision on the aux-model strategy before it can proceed.
t_d7f8cd65 — astro-orbiter is a single serial llama-swap process; Hindsight's
default LLM_MAX_CONCURRENT=32 burst N parallel retains -> node rejects/times
out extras -> 502. Cap global + retain concurrency to 1 (native semaphore
config, no code change). Raise client timeout 120->600s and ingress
proxy-read/send 300->600s so a single long retain (~90s under load) survives.
stepfun rejects Hindsight tagged fact-extraction (400 'missing tags');
solar-pro4:free is the previously-verified-good Nous model for retain/recall
(t_e3375410 / t_d0dffc3d). base_url + provider unchanged.
Scoped to the hindsight ingress (cosmic-rewind.local.mk-labs.cloud) only.
A slow agentic reflect (~132s on solar-pro4) exceeded the 60s default
proxy-read-timeout, causing nginx to return 504 before the app delivered 200.
Raising read+send to 300s lets the response land. No global configmap change.
Refs t_e3375410.
astro-orbiter llama-server-router retired :8002 on 2026-08-18
(llama-swap now serves the OpenAI-compatible API on :8001).
OpenViking still pointed at the dead :8002, so every search/find
call failed: dense embedding -> Connection error -> 500 INTERNAL
server error on /api/v1/search/search and /api/v1/search/find.
Verified from the pod: :8002 = connection refused,
:8001/v1/embeddings (nomic-embed-text-v1.5) = 200 with vectors.
Repoints embedding.dense.api_base and vlm.api_base to :8001.
t_abc05db5
- vlm.model was 'llama3.1-8b' which doesn't exist on astro-orbiter's
/v1/models, causing every summarization call to 400 and endless
circuit-breaker retries. Correct id: Meta-Llama-3.1-8B-Instruct-Q4_K_M.
- embedding.max_input_tokens=1536 still let chunks through that actually
tokenized to 2000-2860 real tokens (estimator undercounts vs llama.cpp's
tokenizer by 1.35x-1.86x on this corpus). Lowered to 1024 for real margin
under the 2048 n_ctx ceiling.
astro-orbiter's llama.cpp router hard-caps nomic-embed-text-v1.5 effective
context at 2048 tokens regardless of ctx-size (known nomic-bert/RoPE limitation
in llama.cpp, not fixable server-side). OpenViking chunks observed at
2000-3400 tokens were tripping 400 exceed_context_size_error and endless
circuit-breaker re-enqueue for viking://temp/default/08140552_5f1c9e/homelab.tar/*.
Set embedding.max_input_tokens: 1536 (well under 2048) since OpenViking's
chunk-time token estimator uses a different tokenizer than llama.cpp's context
counter, so token counts won't match 1:1 - 1536 leaves ~25% headroom.
Approved by Ryan as lowest-risk mitigation (does not touch astro-orbiter/
llama.cpp serving config, which is War Machine's domain and already fixed
separately for the ubatch-size issue).
