nginx-proxy (new role):
- Shared nginx + certbot under /opt/nginx, replacing the copies that
lived inside the Paperclip compose project
- Apps publish via tasks_from: add_vhost (cert marker -> issue cert ->
write vhost -> nginx -t -> reload); a vhost that fails nginx -t is
rolled back
- Certs issued one per domain via Cloudflare DNS challenge; domains
registered as marker files in cert-domains/; optional LE staging
- Stock nginx.conf plus conf.d (directory mounts, not single-file
mounts); catch-all server drops unknown Host/SNI
- Containers renamed to nginx and certbot
hermes_docker (new role):
- Official Hermes image, host networking, gateway and dashboard bound
to loopback and published through nginx-proxy
- Dashboard login via basic-auth provider, secrets from the vault
- Main model set from hermes_model (fantasmic endpoint, key from vault);
config.yaml overrides are merged from a 0600 staged file that is
removed afterwards, so the key never appears in a command line
- Data dir owner-only (HERMES_HOME_MODE=0700)
paperclip-docker:
- Remove nginx, certbot and docker-compose download tasks; deploy with
community.docker.docker_compose_v2 (idempotent, remove_orphans)
- Stop passing the Cloudflare token and certbot email to the Paperclip
container via env_file
- Bind Postgres to 127.0.0.1 (host networking exposed it on all
interfaces); stop chowning pgdata, which the postgres image owns
- Handler recreates the paperclip service so .env changes apply
- Domain m-v.wdw-labs.cloud -> cranium-command.wdw-labs.cloud
Playbooks:
- Add day1_deploy_nginx_proxy.yml and day1_deploy_hermes_docker.yml
- deploy_paperclip_stack.yml: run nginx-proxy before paperclip-docker,
CLI-based container checks (no Docker SDK dependency)
- Remove deploy-hermes.yml
Host vars: add nginx_proxy_certbot_email and nginx_proxy_certbot_staging
(staging stays on until the switch to real certs).
- Create playbooks/build_paperclip_image.yml to build image locally from repo
- Clone/update paperclipai/paperclip main branch on deploy host
- Build with --target production for lean, current, self-managed image
- Update docker-compose to use paperclip-local:latest instead of reeoss image
- Gives full control over Paperclip version and dependency updates
- OpenCode v1.2.24 local adapter pre-installed in source build
- Enables rapid iteration and custom patches on Paperclip platform
Deployment flow:
1. ansible-playbook playbooks/build_paperclip_image.yml # Build from source
2. ansible-playbook playbooks/deploy_paperclip_stack.yml # Deploy with local image
- Create roles/paperclip-docker with full Docker Compose stack deployment
- Template OpenCode v1.2.24 config (provider structure) with Gemma-4-26B model
- Integrate vLLM endpoint (localhost:8000) with vault-injected API key auth
- Wire secrets from ansible vault: vllm_api_key, cloudflare_token, paperclip auth secrets
- Add pre-deployment checks (PostgreSQL, nginx, certbot cert management)
- Implement idempotent stack lifecycle (containers, volumes, networking)
- Deploy to RTX 3090 via muppet-vision.wdw-labs.cloud with nginx SSL termination
Paperclip now serves as CEO reasoning engine with local Gemma inference,
accessible to Hermes platform agents via OpenCode adapter on shared vLLM.
Set OPENAI_API_BASE_URL (config.py default) to our local vLLM endpoint.
Open WebUI was using hardcoded 'https://api.openai.com/v1' for model discovery.
ExternalSecret only provides vllm-api-key from 1Password.
Using same key for both purposes is acceptable for now.
Avoids need for manual secret patching.
The open-webui item doesn't exist in 1Password yet.
webui-secret-key is manually managed via Kubernetes secret patch.
Template now only pulls vllm-api-key from 1Password.
- Change emptyDir to persistent volume (open-webui-data PVC)
Fixes data loss on pod restart (admin user disappears)
- Add ENABLE_OLLAMA_API=false
Disable broken Ollama endpoint probing
vLLM uses OpenAI-compatible API, not Ollama
- Create pvc.yaml for persistent data storage
10Gi NFS-backed storage via nfs-emporium StorageClass
The 'open-webui' item doesn't exist in 1Password yet.
webui-secret-key is manually patched in the secret for now.
Re-enable ExternalSecret pull once the 1Password item is created.
- Add ExternalName Service for astro-orbiter (10.1.71.130:8000)
- Remove Ollama-specific configuration (OLLAMA_BASE_URLS)
- Keep only OpenAI-compatible configuration (OPENAI_BASE_URL)
Fixes: Open WebUI pod was attempting to use Ollama endpoints (/api/tags)
against vLLM which only supports OpenAI-compatible API (/v1/models).
Pod now reaches astro-orbiter via Kubernetes Service DNS name.
Resolves: HTTP 404 errors from Upstream ollama request failed
- ExternalSecret: use key/property format (not path) for 1Password reference
- Deployment: remove admin password env var (use single vllm-api-key)
- Syncs vLLM API key from op://mk-labs/vllm/api-key
astro-orbiter's vLLM primary model changed 2026-09-01 from
DeepSeek-R1-Distill-Qwen-32B-AWQ to Gemma-4-26B-A4B-it-AWQ (Google,
Apache 2.0, US-origin). Same endpoint/API key — only the served model
name changed. max_model_len also bumped to 65536 (was 32768).
Gemma 4 does not emit an always-on <think> reasoning trace like
DeepSeek-R1 did, so HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS=4096
should have more effective headroom for real extraction output than
before, not less.
Retired DeepSeek-R1-Distill-Qwen-32B after confirming its auto tool-choice
reliability is a known, documented DeepSeek-R1-distillation limitation
(trained on pure reasoning traces, no function-calling data — upstream
GitHub-confirmed, not a config gap). Model choice moved to Gemma 4 26B A4B
(Google, Apache 2.0, US-origin, matches Ryan's model-origin preference):
- cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit — MoE (25.2B total / 3.8B active),
chosen over the dense 31B variant for smaller on-disk footprint
(~17.2GB vs ~20.9GB), buying more KV-cache headroom on this 24GB card
- max_model_len=65536 (comfortably over Hermes's 64K floor; native
context is 256K, no extension trick needed)
- Native gemma4 tool-call-parser + gemma4 reasoning-parser (both
registered in this host's vLLM 0.28.0) — purpose-built for this
model's actual output format, not a same-family approximation
- kv_cache_dtype: int4_per_token_head from the outset (learned from the
DeepSeek swap's fp16->fp8->int4 trial-and-error escalation)
Bug found and fixed during deployment: the repo's config.json declares
quant_method 'compressed-tensors' (llm-compressor output) despite the
repo name saying 'AWQ-4bit'. Passing --quantization awq explicitly
caused a hard pydantic ValidationError on every startup attempt.
Fix: omit the quantization field entirely and let vLLM auto-detect from
the model's own config.json — confirmed clean single-attempt start,
NRestarts=0, once removed.
Verified live:
- /health 200, /v1/models confirms max_model_len=65536
- Live completion: correct answer, no unwanted reasoning trace by default
- tool_choice=auto with a clear trigger prompt: correct tool_calls
response with valid JSON args — the exact test DeepSeek-R1-Distill
failed (it either answered in plain text or burned tokens reasoning
about how to call the tool instead of calling it)
- tool_choice=auto with an irrelevant tool present: correctly answered
in plain text, did not over-trigger the tool
- Ansible idempotent re-run confirmed: changed=0, NRestarts=0, clean
journalctl (zero error/traceback lines) after a fresh restart
Known follow-up (not done here): Hindsight's HINDSIGHT_API_LLM_MODEL
cluster config still references the retired DeepSeek-R1-Distill-Qwen-32B
(itself a follow-up from the prior Qwen2.5-32B swap) — needs another
GitOps update to point at Gemma-4-26B-A4B-it-AWQ.
astro-orbiter's vLLM primary model changed 2026-09-01 from
Qwen2.5-32B-Instruct-AWQ to DeepSeek-R1-Distill-Qwen-32B-AWQ
(single-model deployment, see ansible/roles/deploy-vllm/README.md).
Same endpoint (http://astro-orbiter:8000/v1) and API key — only the
served model name changed.
Flagged for follow-up, not fixed here: DeepSeek-R1 emits a
<think>...</think> reasoning trace before its final answer; Hindsight's
fact-extraction prompt/parsing has not been specifically verified
against this reasoning-model output shape. Watch extraction quality
after this rolls out; HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS left
at 4096 pending confirmation that's enough headroom for a real retain
to complete past the reasoning trace.
Single-model deployment per Ryan's direction:
- Primary model: casperhansen/deepseek-r1-distill-qwen-32b-awq, max_model_len=32768
- nomic-embed-text-v1.5 and Qwen3-8B-AWQ both disabled (single-model requirement)
- kv_cache_dtype: int4_per_token_head required to fit 32768 ctx on 24GB RTX 3090
(fp16 KV: 26GB needed, doesn't fit at any utilization; fp8 KV: OOM'd during
FlashInfer warmup with ~50-150MB margin; int4 KV: clean single-attempt start)
- Added kv_cache_dtype / kv_cache_memory_bytes as new optional per-model
template fields in vllm.service.j2 (guarded, no effect on other models)
Verified live: /health 200, /v1/models confirms max_model_len=32768,
live /v1/completions smoke test + manual chat completion both passed
(genuine <think> reasoning trace, correct arithmetic). NRestarts=0,
steady-state VRAM 23.2GB/24.576GB. Ansible idempotent re-run confirmed
changed=0.
Known follow-up (not done here): Hindsight's HINDSIGHT_API_LLM_MODEL
cluster config still references the retired Qwen2.5-32B-Instruct-AWQ —
needs separate GitOps update to point at the new model.
Companion commit to 6bfcc76 (vllm permanent residency cutover). Now that
llama-swap is stopped+disabled and vLLM is the permanent serving layer:
- HINDSIGHT_API_LLM_BASE_URL: llama-swap :8001 -> vLLM :8000
- HINDSIGHT_API_LLM_MODEL: Qwen3.8-27B-Q4_K_M -> Qwen2.5-32B-Instruct-AWQ
- Added HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS=4096 (vLLM's
max_model_len=8192 is far below the 64000 Hindsight retain default;
uncapped this 400s on long retains -- found in t_e6facb19's validation)
- ExternalSecret HINDSIGHT_API_LLM_API_KEY source: 1Password 'nous' item
(unused fallback) -> 1Password 'vllm' item's api-key (vLLM validates
its bearer token, unlike llama-swap which accepted anything)
Embeddings intentionally UNCHANGED -- Hindsight stays on its bundled
local 384-dim embedder (HINDSIGHT_API_EMBEDDINGS_PROVIDER still unset).
Switching to vLLM's nomic-embed (768-dim) is a separate, destructive,
explicitly-approved-only migration (~1300 rows across ~20 banks) --
out of scope here, not part of the dashboard's 'stop llama-swap /
start vLLM' instruction.
rblundon
