---
# ------------------------------------------------------------------------------
# Playbook: day2_qwen38_ctx128k_rollback.yml
# Purpose:  Roll back Qwen3.8-27B-Q4_K_M ctx-size from 131072 back to 65536
#           on astro-orbiter's production router (port 8002).
#
# What this playbook does:
#   1. Renders the updated llama-server-router-preset.ini.j2 (now with
#      llm_router_qwen38_ctx_size: 65536) to
#      /opt/llama-server-router-preset.ini.
#   2. Restarts llama-server-router.service.
#   3. Verifies the router loads Qwen3.8-27B at ctx=65536 in status.args.
#
# Context:
#   - t_441470b9 (2026-08-16): ctx-size bumped 32768 -> 131072. Verified VRAM
#     at 131072 ctx with only Qwen3.8 + nomic-embed co-resident: ~20,282 MiB
#     + 558 MiB = ~20.8 GB on 24 GB RTX 3090. Comfortably safe.
#   - t_72646029 (2026-08-17): Phi-3.5mini moved to GPU (n-gpu-layers=99)
#     to enable concurrent residency with CPU-offloaded Coder-14B and
#     Llama-3.1-8B. This added ~2GB CUDA context buffers for Phi + shifted
#     Phi's model weights onto the GPU (~3.8GB).
#   - NEW steady-state VRAM: Qwen3.8 @ 131072 ctx (~20,282 MiB) + nomic-embed
#     (~558 MiB) + Llama CUDA ctx (~1,706 MiB) + Coder CUDA ctx (~1,390 MiB)
#     = ~24,004 MiB. Adding Phi-3.5 (~3,800 MiB weights + ~1.4 GB CUDA ctx)
#     pushes total to ~29,000+ MiB — exceeding the 24,576 MiB RTX 3090 limit.
#     Qwen3.8-27B-131072 now fails to load (HTTP 500, OOM before llama.cpp
#     reaches the model-loading phase).
#   - FIX: reduce Qwen3.8 ctx-size 131072 -> 65536. This reduces KV cache
#     from ~6GB to ~3GB, freeing ~3GB of VRAM. New estimated steady-state:
#     Qwen3.8 @ 65536 ctx (~17,068 MiB) + nomic (~558) + Llama ctx (~1,706)
#     + Coder ctx (~1,390) + Phi-3.5 (~3,800 + ~1,400 CUDA ctx) = ~25,922 MiB.
#     Still over 24,576 — see "Phase 2" below for the secondary fix.
#
#   IMPORTANT: Rolling back ctx-size alone may NOT be sufficient. The
#   hardware reference (astro-orbiter-hardware.md line 166, t_72646029)
#   states steady-state ~24,004 MiB WITHOUT Phi on GPU. Adding Phi-3.5 back
#   to GPU tips it over. This playbook handles the context rollback; if Qwen3.8
#   still fails to load after Phase R, Wong should escalate to Ryan for a
#   decision on either (a) offloading Phi-3.5mini to CPU (n-gpu-layers=0),
#   or (b) adding a second GPU. Document the Phase 2 finding as a separate
#   follow-up task if needed.
#
#   The 64K floor from the 2026-08-12 cutover validation (t_cd0d5388, Gate 1)
#   still applies — ctx-size=65536 satisfies it.
#
# Run:
#   cd /home/hermes/git/homelab/ansible
#   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook \
#     -i inventory.yml \
#     playbooks/day2_qwen38_ctx128k_rollback.yml
#
# Task reference: t_c9fed26c — War Machine benchmark, 2026-08-18
# Root cause: t_72646029 CPU-offload deployment added Phi-3.5 to GPU,
#   shifting total VRAM past the 24,576 MiB ceiling when Qwen3.8 runs at 128K.
# ------------------------------------------------------------------------------
- name: Roll back Qwen3.8-27B ctx-size to 65536 on astro-orbiter
  hosts: astro-orbiter
  become: true
  vars:
    llm_router_preset_enabled: true
    llm_router_qwen38_ctx_size: 65536

  roles:
    - role: llm-inference-multimodel
      tags: [preset, systemd, verify]