same for open-webui

- cluster/applications/litellm: multi-source Application (oci litellm-helm 1.106.1),
  values, ExternalSecrets (1Password wdw-labs), HTTPRoute, ExternalName services
  for fantasmic/wishes, namespace
- gateway: https-skyliner listener + skyliner.wdw-labs.cloud certificate,
  ReferenceGrant for litellm namespace
- cert-manager: add wdw-labs.cloud to ClusterIssuer dnsZones
- external-dns: add wdw-labs.cloud zone (Technitium RFC2136)
- onepassword-connect: add wdw-labs vault to ClusterSecretStore

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0118eUSMQitZMHN7qaCvynA3

.gitignore

@@ -1,15 +1,9 @@
-# macOS system files
+# macOS
 .DS_Store
 .AppleDouble
 .LSOverride
-
-# Icon must end with two \r
 Icon
-
-# Thumbnails
 ._*
-
-# Files that might appear in the root of a volume
 .DocumentRevisions-V100
 .fseventsd
 .Spotlight-V100
@@ -17,36 +11,53 @@ Icon
 .Trashes
 .VolumeIcon.icns
 .com.apple.timemachine.donotpresent
-
-# Directories potentially created on remote AFP share
 .AppleDB
 .AppleDesktop
 Network Trash Folder
 Temporary Items
 .apdisk
-
-# VS Code
-.vscode/
-
-# macOS metadata
 *.icloud
 
-# Terraform
-terraform/
+# IDE
+.vscode/
+.idea/
+.kiro/
+
+# Terraform (track .tf files, ignore state and runtime)
 .terraform/
 *.tfstate
 *.tfstate.backup
-
-# Crash log files
+# *.tfvars
+!*.tfvars.example
+!*.pkrvars.hcl
 crash.log
-
-# Ignore override files as they are usually used to override resources locally
 *.override.tf
 *.override.tf.json
-
-# Ignore CLI configuration files
 .terraformrc
-.terraformrc.json
+terraform.rc
 
-# Ignore secrets
-doppler-token.yaml
+# Ansible
+*.retry
+ansible/vault/*.vault
+
+# Packer
+packer_cache/
+
+# Secrets
+*.pem
+*.key
+
+# Environment files with secrets
+boilerplates/**/.env
+
+# Local paths
+~/
+
+# Misc
+doppler-token.yaml
+
+# talhelper generated machine configs — contain secrets, never commit
+talos/talhelper/clusterconfig/
+# talenv.yaml plaintext — only commit the SOPS-encrypted version
+talos/talhelper/talenv.yaml
+!talos/talhelper/talenv.sops.yaml

.sops.yaml

@@ -0,0 +1,3 @@
+creation_rules:
+  - path_regex: talos/talhelper/talenv.yaml
+    age: age1xkyuv8r8ce6lu3d64jfspz4e50k6pxlaxwmuplzrtnpfnnrnycaq6mfsrn

COUCHDB-ERLANGCOOKIE-FIX.md

@@ -0,0 +1,160 @@
+# CouchDB erlangCookie Fix - Implementation Guide
+
+## Summary
+
+**Problem**: CouchDB deployment fails because `erlangCookie` is missing from the ExternalSecret configuration.
+
+**Decision**: Externalize `erlangCookie` to 1Password (pragmatic approach)
+
+**Rationale**: 
+- ExternalSecret architecture requires ownership of the entire secret
+- Mixing externalized and chart-generated fields in the same secret is not supported
+- Single-node deployment makes erlangCookie rotation unnecessary
+- This is an acceptable deviation from the pure Harbor pattern given the architectural constraints
+
+## Implementation Steps
+
+### 1. Generate erlangCookie Value
+
+```bash
+openssl rand -hex 20
+```
+
+Example output: `f4e3c2b1a9d8e7f6c5b4a3d2e1f0a9b8c7d6e5f4`
+
+### 2. Add to 1Password
+
+- **Vault**: `mk-labs`
+- **Item**: `couchdb`
+- **Field Name**: `erlang-cookie`
+- **Field Type**: password (concealed)
+- **Value**: `<paste generated value from step 1>`
+
+### 3. Update ExternalSecret Configuration
+
+File: `cluster/applications/couchdb/externalsecret.yaml`
+
+```yaml
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: couchdb-credentials
+  namespace: couchdb
+  labels:
+    app.kubernetes.io/name: couchdb
+    app.kubernetes.io/part-of: mk-labs
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: onepassword-connect
+  target:
+    name: couchdb-admin
+    creationPolicy: Owner
+    template:
+      engineVersion: v2
+      data:
+        adminUsername: "admin"
+        adminPassword: "{{ .adminPassword }}"
+        cookieAuthSecret: "{{ .cookieAuthSecret }}"
+        erlangCookie: "{{ .erlangCookie }}"  # ← ADD THIS LINE
+  data:
+    - secretKey: adminPassword
+      remoteRef:
+        key: couchdb
+        property: admin-password
+    - secretKey: cookieAuthSecret
+      remoteRef:
+        key: couchdb
+        property: cookie-auth-secret
+    - secretKey: erlangCookie  # ← ADD THIS BLOCK
+      remoteRef:
+        key: couchdb
+        property: erlang-cookie
+```
+
+### 4. Update values.yaml Documentation (Optional)
+
+File: `cluster/applications/couchdb/values.yaml`
+
+Update the comment block at line 9-10:
+
+```yaml
+# Admin credentials managed via ExternalSecret
+# See externalsecret.yaml for 1Password integration
+#
+# NOTE: erlangCookie is externalized to 1Password for architectural
+# simplicity (ExternalSecret ownership model). In a pure Harbor pattern,
+# this would be chart-generated, but single-node deployment makes this
+# acceptable. The erlangCookie is treated as an immutable infrastructure
+# secret (generate once, never rotate).
+createAdminSecret: false
+extraSecretName: "couchdb-admin"
+```
+
+### 5. Commit and Push
+
+```bash
+cd ~/git/homelab
+git add cluster/applications/couchdb/externalsecret.yaml
+git add cluster/applications/couchdb/values.yaml  # if modified
+git commit -m "fix(couchdb): add erlangCookie to ExternalSecret from 1Password"
+git push origin main
+```
+
+### 6. Verify Deployment
+
+```bash
+# Watch ExternalSecret sync
+kubectl get externalsecret -n couchdb couchdb-credentials -w
+# Wait for: SecretSynced
+
+# Verify secret created with all four keys
+kubectl get secret -n couchdb couchdb-admin -o yaml
+# Should contain: adminUsername, adminPassword, cookieAuthSecret, erlangCookie
+
+# Watch ArgoCD sync
+kubectl get application -n argocd couchdb -w
+# Wait for: Healthy/Synced
+
+# Watch pod startup
+kubectl get pods -n couchdb -w
+# Wait for: Running
+
+# Test CouchDB access
+kubectl port-forward -n couchdb svc/couchdb-svc-couchdb 5984:5984 &
+curl http://localhost:5984/
+# Expected: {"couchdb":"Welcome","version":"3.5.1"}
+```
+
+## Why Not Follow Harbor Pattern Exactly?
+
+**Harbor Pattern**: Only user-facing credentials externalized, internal secrets chart-generated.
+
+**CouchDB Constraint**: ExternalSecret uses `creationPolicy: Owner`, which takes full ownership of the target secret. This prevents the Helm chart from adding auto-generated fields to the same secret.
+
+**Options Considered**:
+1. ✅ **Externalize erlangCookie** (SELECTED) - Works with current architecture
+2. ❌ Chart auto-generation - Conflicts with ExternalSecret ownership
+3. ❌ Dual-secret approach - Requires Helm chart customization
+4. ❌ Disable ExternalSecret - Loses 1Password integration for admin password
+
+**Decision**: Pragmatic approach wins. erlangCookie is treated as an infrastructure secret (generate once, never rotate), which is acceptable for a single-node deployment.
+
+## Secret Classification
+
+| Secret           | Type          | 1Password? | Rationale                          |
+|------------------|---------------|------------|------------------------------------|
+| adminUsername    | User-facing   | No*        | Static value, hardcoded in template |
+| adminPassword    | User-facing   | ✅ YES     | User login credential              |
+| cookieAuthSecret | Gray area     | ✅ YES     | Session security, periodic rotation |
+| erlangCookie     | Internal      | ✅ YES**   | Architectural constraint           |
+
+\* Hardcoded in ExternalSecret template (not fetched from 1Password)  
+\*\* Pragmatic deviation from Harbor pattern due to ExternalSecret architecture
+
+## References
+
+- Full analysis: `/home/hermes/couchdb-erlangcookie-analysis.txt`
+- Harbor pattern: `/home/hermes/harbor-simplification-complete.txt`
+- CouchDB Helm chart: `apache/couchdb` v4.6.3

Claude outputs/Account-Dossier-Dashboard-Design-Doc.md

@@ -0,0 +1,702 @@
+# Account Dossier Interactive Dashboard — Design Document
+
+**Version:** 0.1 (MVP)  
+**Author:** Claude  
+**Date:** 2026-10-07  
+**Status:** Design Review (Pre-Prototype)
+
+---
+
+## 1. Overview & Purpose
+
+**What:** Interactive HTML/React dashboard that surfaces account intelligence for a single Pure Storage customer or prospect. Provides a "10,000 ft view" with drill-down to ground-level detail.
+
+**Why:** SEs need to move quickly from account snapshot → discovery prep → conversation context without context-switching between Sheets, notes, and browser tabs. One dashboard per customer becomes the source of reference for that account.
+
+**MVP Scope:** The 10 essential questions (List 1 + List 2 from account intelligence framework)
+
+**Future Scope:** Relationships, opportunities, financial modeling, industry trends, sentiment analysis, timeline views
+
+---
+
+## 2. User Personas & Use Cases
+
+### Primary User
+**Ryan** (You, Enterprise SE)
+- Prepping for discovery calls
+- Hunting for competitive displacement opportunities
+- Planning expansion/renewal strategy
+- Sharing account context with team (future)
+
+### Use Cases (in priority order, all equally weighted for MVP)
+
+| Use Case | What They Need | Where Dashboard Helps |
+|----------|---|---|
+| **Pre-call Prep (15 min before discovery)** | Quick account snapshot + key metrics + conversation hooks | Overview pane + drill-down to specific List items; see what changed since last call |
+| **Competitive Displacement Hunting** | Incumbent vendors + footprint + gaps + consolidation leverage | List 2 drill-down showing Pure coverage % + high-value data not on Pure; see EMC systems aging |
+| **Expansion/Renewal Planning** | Growth trajectory + budget cycle + stakeholder sentiment + upcoming initiatives | Opportunity tab (future) + timeline + stakeholder relationships; see refresh window + budget headroom |
+
+### Secondary User (Future)
+**Account team** (Sales, Solution Architect, Customer Success)
+- Read-only view of account summary
+- Understand strategic context before customer calls
+
+---
+
+## 3. Architecture Overview
+
+### 3.1 Data Flow Diagram
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│  Per-Customer Data Sources (DRY: No Duplication)            │
+├─────────────────────────────────────────────────────────────┤
+│                                                               │
+│  📊 Google Sheet (Structured Data)                           │
+│  ├─ Account tab (List 1 metrics)                             │
+│  ├─ Pure-Deployment tab (List 2 metrics)                    │
+│  ├─ Relationships tab (Contacts, sentiment, history)        │
+│  ├─ Interactions tab (Call/email log with sentiment)        │
+│  ├─ Financials tab (ARR, contracts, opportunities)         │
+│  ├─ Industry-Trends tab (Curated insights)                 │
+│  └─ Metadata tab (Sync status, ownership, sharing)         │
+│                                                               │
+│  📝 Obsidian Markdown File (Unstructured Context)           │
+│  ├─ Discovery notes (narrative, conversation context)      │
+│  ├─ Strategy (account themes, expansion hooks)             │
+│  ├─ Relationship notes (person-by-person context)          │
+│  └─ Internal flags (competitive threats, risks)            │
+│                                                               │
+└─────────────────────────────────────────────────────────────┘
+                            ↓
+                   (via API or manual sync)
+                            ↓
+┌─────────────────────────────────────────────────────────────┐
+│  Interactive Dashboard (HTML/React Artifact)                │
+│  Reads from both sources, displays in 3-pane layout         │
+│  No data copying; all reads from source                     │
+└─────────────────────────────────────────────────────────────┘
+                            ↓
+                   (displayed to user)
+                            ↓
+┌─────────────────────────────────────────────────────────────┐
+│  User Actions:                                               │
+│  • View account snapshot                                     │
+│  • Drill into List 1 / List 2 metrics                       │
+│  • Check stakeholder sentiment                              │
+│  • Review opportunity pipeline                              │
+│  • Log new interaction                                       │
+│  • Open full Sheets / Obsidian for detailed editing         │
+└─────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 4. UI/UX Design
+
+### 4.1 Layout: 3-Pane Architecture
+
+```
+┌──────────────────────────────────────────────────────────────────────┐
+│ HEADER: Account Name | Status | Vertical | Location | [Share] [Edit] │
+├─────────────────────────────────────────────────────────────────────┤
+│                                                                       │
+│  LEFT PANE              │ CENTER PANE              │ RIGHT PANE      │
+│  (Overview)             │ (Details & Drill-down)   │ (Context)       │
+│                         │                          │                 │
+│  ┌─────────────────┐    │ ┌────────────────────┐   │ ┌────────────┐ │
+│  │ Account Health  │    │ │   Tab Bar:         │   │ │ Latest     │ │
+│  │ Scorecard       │    │ │ Overview | List1   │   │ │ Interactions   │
+│  │ ◆ 4 KPIs        │    │ │ List2    | Oppty   │   │ │            │ │
+│  │   (visual)      │    │ │                    │   │ │ ┌────────┐ │ │
+│  │                 │    │ │ ┌──────────────┐   │   │ │ Sarah   │ │ │
+│  ├─────────────────┤    │ │ │ Item 1 Card  │   │   │ │ Chen    │ │ │
+│  │ Key Dates       │    │ │ │ [Summary]    │   │   │ │ Call    │ │ │
+│  │ Timeline        │    │ │ │ [10K ft]     │   │   │ │ 2d ago  │ │ │
+│  │ • Refresh: 6mo  │    │ │ │ [Status]     │   │   │ │ ⭐⭐⭐  │ │ │
+│  │ • Budget: 3mo   │    │ │ └──────────────┘   │   │ │ (Drill) │ │ │
+│  │ • Contract: 8mo │    │ │                    │   │ │└────────┘ │ │
+│  │                 │    │ │ ┌──────────────┐   │   │ │           │ │
+│  ├─────────────────┤    │ │ │ Item 2 Card  │   │   │ ├──────────┤ │
+│  │ Actions         │    │ │ │ [Summary]    │   │   │ │ [Log new]│ │
+│  │ [QBR Prep]      │    │ │ │ [10K ft]     │   │   │ │           │ │
+│  │ [Edit Account]  │    │ │ │ [Status]     │   │   │ │ Related   │ │
+│  │ [Open Sheets]   │    │ │ └──────────────┘   │   │ │ Notes     │ │
+│  │ [View Notes]    │    │ │                    │   │ │ (from MD) │ │
+│  │ [Share]         │    │ │ .... (3 more)      │   │ │           │ │
+│  │                 │    │ │                    │   │ │ [View all]│ │
+│  └─────────────────┘    │ └────────────────────┘   │ └────────────┘ │
+│                         │                          │                 │
+└──────────────────────────────────────────────────────────────────────┘
+```
+
+### 4.2 Left Pane: Account Overview (10,000 ft)
+
+**Purpose:** At a glance, is this account healthy? What's the most critical thing to know?
+
+**Components:**
+
+#### Health Scorecard (4 KPIs)
+Visual status cards, each with:
+- **Metric name** + current value
+- **Trend indicator** (↑ up, ↓ down, → flat)
+- **Status badge** (green / yellow / red)
+- **Drillable** — click to zoom into that metric
+
+**KPIs (MVP):**
+1. **Pure Coverage** — % of total storage on Pure
+   - Value: 30% | Trend: ↑ +5% YoY | Status: 🟡 Opportunity
+   - Rationale: Shows expansion potential; top sales driver
+
+2. **Fleet Age** — Average age of Pure systems
+   - Value: 4.2 yr | Trend: → flat | Status: 🟡 Refresh Window Approaching
+   - Rationale: Indicates refresh opportunity timing
+
+3. **Expansion Opportunity Size** — $ value of consolidation potential
+   - Value: $3.5M | Trend: ↑ Growing | Status: 🟢 Pipeline-Ready
+   - Rationale: Deal size at a glance; guides priority
+
+4. **Engagement Momentum** — Recency of last interaction + champion sentiment
+   - Value: 5 days | Trend: ↑ Active | Status: 🟢 Engaged
+   - Rationale: Is this account warm or cold?
+
+#### Key Dates Timeline
+Visual timeline of critical dates:
+- **Refresh Date**: Next Pure system refresh
+- **Budget Cycle**: When capex decisions happen
+- **Contract Renewal**: When Pure contract is up
+- **Next Review**: When you should reconnect
+
+Format: Horizontal timeline or vertical list with "X days until" callouts.
+
+#### Action Buttons (Quick nav)
+- **QBR Prep** → Opens checklist template
+- **Edit Account** → Opens Sheets in new tab
+- **View Notes** → Opens Obsidian vault or embedded Markdown
+- **Share** → (Future) generates shareable summary link
+- **Log Interaction** → Quick-add form (bottom right pane)
+
+---
+
+### 4.3 Center Pane: Details & Drill-Down
+
+**Purpose:** Zoom from 10K ft into List 1 and List 2 metrics. See details, trends, and rationale.
+
+**Tab Structure:**
+
+#### Tab 1: Overview (Summary of all 10 items)
+Shows all 5 List 1 items + all 5 List 2 items as **cards in a grid**:
+- Card layout: Title | 10K ft summary | Status badge | Click to drill
+- Color-coded by item type (List 1 = blue accent, List 2 = green accent)
+- Scanning-friendly; can see account at a glance
+
+#### Tab 2: Account Profile (List 1 Deep Dive)
+Five expandable cards, one per List 1 item:
+
+**Card 1: Data Footprint & Workload Breakdown**
+```
+┌─ Data Footprint & Workload Breakdown ───────────────────┐
+│ 10K FT VIEW:                                             │
+│ 500TB current | +18% YoY | 2yr forecast: 700TB           │
+│ Workloads: 40% OLTP | 35% Analytics | 25% Unstructured  │
+│                                                           │
+│ [CLICK TO EXPAND ▼]                                      │
+│                                                           │
+│ EXPANDED VIEW:                                           │
+│ ┌─ Breakdown Table ──────────────────────────────────┐  │
+│ │ Tier       │ Current │ Growth │ 2yr Forecast │ Note  │
+│ │ OLTP       │ 200TB   │ +15%   │ 265TB        │ TX DB │
+│ │ Analytics  │ 175TB   │ +25%   │ 275TB        │ DW    │
+│ │ Unstructured│ 125TB   │ +12%   │ 160TB        │ Lake  │
+│ └────────────────────────────────────────────────────┘  │
+│                                                           │
+│ ┌─ Capacity Growth Chart ────────────────────────────┐  │
+│ │ [Line chart: 3yr history + 2yr forecast]           │  │
+│ │ Shows growth trajectory, where Pure could capture │  │
+│ └────────────────────────────────────────────────────┘  │
+│                                                           │
+│ CONTEXT FROM NOTES:                                      │
+│ "Customer mentioned big analytics push for FY27..."      │
+│                                                           │
+│ [Related Links: 10-K | Analyst Report]                  │
+└─────────────────────────────────────────────────────────┘
+```
+
+**Card 2: Current Storage Vendors & Installed Infrastructure**
+```
+┌─ Current Storage Vendors ──────────────────────────────┐
+│ 10K FT VIEW:                                            │
+│ NetApp 50% | Pure 30% | Dell 20% (of 1200TB total)     │
+│ Consolidation Opportunity: HIGH                         │
+│                                                          │
+│ [CLICK TO EXPAND ▼]                                     │
+│                                                          │
+│ EXPANDED VIEW:                                          │
+│ ┌─ Vendor Breakdown (Pie Chart) ─────────────────────┐ │
+│ │ [Visual pie showing % by vendor]                   │ │
+│ │ NetApp = oldest systems, refresh due              │ │
+│ └────────────────────────────────────────────────────┘ │
+│                                                          │
+│ ┌─ Incumbent Systems Detail ────────────────────────┐  │
+│ │ Vendor: EMC Symmetrix DMX-4                       │  │
+│ │ Capacity: 200TB | Age: 6 years | Status: AGING   │  │
+│ │ Use Case: Core banking OLTP                       │  │
+│ │ 💡 Displacement Target: Move to Pure FlashArray   │  │
+│ └────────────────────────────────────────────────────┘ │
+│                                                          │
+│ COMPETITIVE CONTEXT:                                    │
+│ "This EMC system is at end-of-life; customer risk-     │
+│ averse but willing to consolidate if migration is      │
+│ proven. Reference customer opportunity."                │
+│                                                          │
+│ [Related: Competitive Intel | Displacement Playbook]    │
+└──────────────────────────────────────────────────────────┘
+```
+
+**Cards 3-5:** Similar drill-down structure for Business Drivers, Architecture, Budget/Authority
+
+#### Tab 3: Pure Deployment (List 2 Deep Dive)
+Five cards showing Pure-specific metrics, one per List 2 item.
+
+**Card 1: Current Pure Installed Base**
+```
+┌─ Current Pure Installed Base ──────────────────────────┐
+│ 10K FT VIEW:                                            │
+│ 2 FlashArray systems (150TB) | Healthy | 4.2yr avg age │
+│ Headroom: 52TB | Refresh window: 18mo                   │
+│                                                          │
+│ [CLICK TO EXPAND ▼]                                     │
+│                                                          │
+│ EXPANDED VIEW:                                          │
+│ ┌─ Systems Inventory Table ──────────────────────────┐ │
+│ │ System | Type | Capacity | Util | Headroom | Age   │ │
+│ │ FA-P1  | FA   | 75TB     | 65%  | 26TB     | 5.1yr │ │
+│ │ FA-DR  | FA   | 75TB     | 40%  | 45TB     | 3.3yr │ │
+│ │                                  Refresh Needed    │ │
+│ └────────────────────────────────────────────────────┘ │
+│                                                          │
+│ ┌─ Utilization & Headroom Heatmap ──────────────────┐ │
+│ │ [Visual: System cards with fill % + headroom bar]  │ │
+│ │ FA-P1: ████████ 65% USED (26TB headroom)          │ │
+│ │ FA-DR: █████░░░ 40% USED (45TB headroom)          │ │
+│ └────────────────────────────────────────────────────┘ │
+│                                                          │
+│ EXPANSION IMPLICATION:                                  │
+│ "FA-P1 will need refresh/capacity add in 18 months;     │
+│ FA-DR has headroom for growth without additional spend" │
+│                                                          │
+│ [Related: Pure1 Dashboard | Refresh Planning Guide]     │
+└──────────────────────────────────────────────────────────┘
+```
+
+**Cards 2-5:** Performance/Satisfaction, Coverage Gaps, Refresh Cycles, Stakeholders
+
+---
+
+### 4.4 Right Pane: Unstructured Context & Interactions
+
+**Purpose:** Connect quantitative data (left/center) to narrative context (right). See "why" behind the numbers.
+
+**Components:**
+
+#### Latest Interactions (Log)
+Shows most recent 3-5 interactions with sentiment:
+```
+┌─ Interactions ────────────────────────────────────────┐
+│ ⭐⭐⭐⭐⭐ Sarah Chen (VP Storage)                      │
+│   Call — 2026-10-02 · 5 days ago                       │
+│   "Discussed multi-site replication, pain with NetApp" │
+│   Status: Strong Advocate for Pure adoption            │
+│                                                         │
+│ ⭐⭐⭐ Mike Lopez (CIO)                                │
+│   Email — 2026-09-15 · 22 days ago                     │
+│   "Budget constraints; interested in OpEx models"      │
+│   Status: Neutral, cost-focused                        │
+│                                                         │
+│ ⭐⭐ Jennifer Park (CFO)                              │
+│   Meeting — 2026-08-20 · 47 days ago                   │
+│   "Positive on storage consolidation, capex tight"     │
+│   Status: Positive but budget-constrained              │
+│                                                         │
+│ [+ Log New Interaction]                                │
+└───────────────────────────────────────────────────────┘
+```
+
+#### Related Notes (from Markdown)
+Embedded snippet from Obsidian Markdown file:
+```
+┌─ Discovery Notes ──────────────────────────────────────┐
+│ STRATEGY THEME: Consolidation play                     │
+│ Customer is post-acquisition; consolidating 3 data     │
+│ centers into 2. Big opportunity to move from multi-    │
+│ vendor (EMC + NetApp) to Pure standardization.         │
+│                                                         │
+│ RISK: Customer risk-averse on primary storage.         │
+│ LEVER: Success story at reference customer + DR copy. │
+│                                                         │
+│ NEXT STEP: Sarah is scheduled for Q4 arch review.      │
+│ Come with: Multi-site replication architecture,        │
+│ consolidation cost model.                              │
+│                                                         │
+│ [View Full Notes] [Edit in Obsidian]                   │
+└───────────────────────────────────────────────────────┘
+```
+
+#### Quick Actions
+- **[Log Interaction]** → Popup form to add call/email/meeting
+- **[View All Notes]** → Opens Obsidian file
+- **[View Sheets]** → Opens Google Sheets
+- **[Share Account]** → (Future) generates link
+
+---
+
+### 4.5 Responsive Behavior & Interactivity
+
+#### Drill-Down Interactions
+- **Cards expand in place** — no modal/new page; content slides down
+- **Breadcrumb trail** — e.g., "Account > List 1 > Data Footprint" so you know where you are
+- **Collapse others** — When one card expands, others collapse (reduces noise)
+- **Smooth transitions** — Cards animate open/closed
+
+#### Status Badges & Visual Indicators
+| Status | Color | Meaning |
+|--------|-------|---------|
+| 🟢 Green | Good / On Track / Advocate | Positive signal |
+| 🟡 Yellow | Caution / Aging / Neutral | Attention needed |
+| 🔴 Red | Risk / Critical / Urgent | Action required |
+| ⚪ Gray | Unknown / TBD | Need more info |
+
+#### Data-Driven Highlights
+Conditional visual emphasis on metrics:
+- **Refresh window approaching** (>4yr age) → Highlighted in yellow
+- **Coverage gap opportunity** (low % Pure) → Highlighted in orange
+- **Budget cycle in next 90 days** → Underlined with "upcoming" tag
+- **Strong champion sentiment** → Starbursts/badge next to name
+
+---
+
+## 5. Component Breakdown & Data Mapping
+
+### 5.1 Component Hierarchy
+
+```
+DashboardApp
+├── Header
+│   ├── AccountName
+│   ├── StatusBadge (Prospect/Customer)
+│   ├── MetadataBar (Vertical, Location, Last Updated)
+│   └── HeaderActions ([Share] [Edit] [Settings])
+│
+├── MainLayout (3-pane)
+│   │
+│   ├── LeftPane (Overview)
+│   │   ├── HealthScorecard
+│   │   │   ├── KPICard (Pure Coverage)
+│   │   │   ├── KPICard (Fleet Age)
+│   │   │   ├── KPICard (Opportunity Size)
+│   │   │   └── KPICard (Engagement Momentum)
+│   │   ├── TimelineComponent
+│   │   │   ├── TimelineEvent (Refresh Date)
+│   │   │   ├── TimelineEvent (Budget Cycle)
+│   │   │   ├── TimelineEvent (Contract Renewal)
+│   │   │   └── TimelineEvent (Next Review)
+│   │   └── ActionButtons
+│   │       ├── Button (QBR Prep)
+│   │       ├── Button (Edit Account)
+│   │       ├── Button (View Notes)
+│   │       └── Button (Share)
+│   │
+│   ├── CenterPane (Details & Drill-Down)
+│   │   ├── TabBar
+│   │   │   ├── Tab (Overview)
+│   │   │   ├── Tab (List 1: Account Profile)
+│   │   │   ├── Tab (List 2: Pure Deployment)
+│   │   │   └── Tab (Opportunities — future)
+│   │   │
+│   │   └── TabContent (varies by tab)
+│   │       └── DetailsCard (repeating)
+│   │           ├── CardHeader (title + status)
+│   │           ├── CardSummary (10K ft view)
+│   │           ├── ExpandButton
+│   │           └── ExpandedContent (on click)
+│   │               ├── DetailTable / Chart
+│   │               ├── TrendVisualization
+│   │               └── RelatedContext
+│   │
+│   └── RightPane (Context)
+│       ├── InteractionLog
+│       │   └── InteractionItem (repeating)
+│       │       ├── Sentiment (stars)
+│       │       ├── Contact Info
+│       │       ├── Timestamp
+│       │       └── Summary
+│       ├── RelatedNotes
+│       │   └── NotesSnippet
+│       └── QuickActions
+│           ├── Button (Log Interaction)
+│           ├── Button (View All Notes)
+│           └── Button (View Sheets)
+```
+
+### 5.2 Data Source Mapping
+
+| Component | Data Source | Sheet Location | Notes |
+|-----------|---|---|---|
+| **Account Name** | Account tab | Name | |
+| **Status Badges** | Account tab | Status, Vertical, Location | |
+| **Health Scorecard** | Account + Pure-Deployment tabs | Multiple cells + formulas | Calculated KPIs |
+| **Timeline** | Account tab | Columns: Next Refresh Date, Budget Cycle, Contract End, Next Review | |
+| **List 1 Cards** | Account tab | Tabs: Item 1-5 data | Five expandable items |
+| **List 2 Cards** | Pure-Deployment tab | Tabs: Item 1-5 data | Five expandable items |
+| **Interaction Log** | Interactions tab | Contact, Type, Date, Sentiment, Summary | Sorted by date DESC |
+| **Related Notes** | Markdown file (Obsidian) | Fetched via API or embed | Text snippet + link |
+
+---
+
+## 6. Visual Design Principles
+
+### Color Palette (Themeable)
+- **Accent Blue** (Primary): List 1 items, account overview
+- **Accent Green** (Secondary): List 2 items, Pure deployment
+- **Status Green** (Success): Healthy, positive, on-track
+- **Status Yellow** (Warning): Approaching refresh, caution needed
+- **Status Red** (Critical): Urgent action required
+- **Neutral Gray**: Unknown, TBD, secondary content
+- **Text Dark Gray** (Light mode) / **Text Light Gray** (Dark mode)
+- **Background**: Light/dark mode adaptive
+
+### Typography
+- **Header**: Large, bold, account name stands out
+- **Tab titles**: Medium weight, clear tab distinction
+- **Card titles**: Semi-bold, clickable indicator (▼)
+- **Data values**: Monospace for numbers (easier to scan)
+- **Labels**: Smaller, muted color
+
+### Spacing & Layout
+- **Panes**: Equal width (33% each) or responsive flex at smaller widths
+- **Cards**: Padding ~20px, margin ~16px between cards
+- **Drill-down**: Animated 200ms expand/collapse
+- **Mobile**: Stack panes vertically (future consideration)
+
+---
+
+## 7. Navigation & User Flows
+
+### Flow 1: Pre-Call Prep (15 min before discovery)
+```
+User lands on dashboard
+    ↓
+Glances at Health Scorecard (30 sec) — "Pure coverage 30%, fleet aging, $3.5M upside"
+    ↓
+Checks Key Dates Timeline (30 sec) — "Budget cycle in 3 mo, refresh in 18 mo"
+    ↓
+Clicks "QBR Prep" action → Opens checklist template in new tab
+    ↓
+Scrolls to Center pane, clicks "Current Vendors" card to see EMC footprint
+    ↓
+Reads right pane: "Customer risk-averse on primary storage; use reference story"
+    ↓
+Click [Log Interaction] → Pre-fills with today's call, ready to document post-call
+```
+
+### Flow 2: Competitive Displacement Hunt
+```
+User is looking for displacement opportunities
+    ↓
+Opens Account dashboard
+    ↓
+Glances at Scorecard: "Pure coverage 30%" → Opportunity
+    ↓
+Clicks "Data Footprint" card → Sees workload breakdown
+    ↓
+Clicks "Current Vendors" card → Sees EMC system (200TB, 6yr old, primary OLTP)
+    ↓
+Clicks "Coverage Gaps" card → Confirms: "EMC Symmetrix not on Pure yet"
+    ↓
+Reads notes: "Customer risk-averse but willing to migrate if proven"
+    ↓
+Action: Find reference customer, prepare migration playbook for next call
+```
+
+### Flow 3: Expansion Planning
+```
+User reviewing account for renewal/expansion
+    ↓
+Glances at Scorecard: Fleet age, Opportunity size, Engagement momentum
+    ↓
+Clicks "Fleet Age" card → Sees FA-P1 refresh window in 18 months
+    ↓
+Clicks "Refresh Cycles" card → Confirms capex cycle and window
+    ↓
+Checks Stakeholder info in right pane → "Sarah (VP Storage) is champion, strong advocate"
+    ↓
+Clicks "Opportunities" tab (future) → See pipeline: Analytics consolidation (Q2 2027, $1.5M)
+    ↓
+Action: Schedule architecture review with Sarah for Q1; prepare proposal for analytics move
+```
+
+---
+
+## 8. Data Sync & Freshness
+
+### MVP Approach: Manual Sync
+- **Sheet → Dashboard**: Copy-paste JSON export or Google Sheets API call (daily/on-demand)
+- **Markdown → Dashboard**: Fetch from Obsidian API or GitHub if synced
+- **Freshness**: Dashboard is "point-in-time snapshot" at time of load
+- **Update frequency**: User refreshes page or manually syncs from edit screens
+
+### Future: Live Sync
+- Dashboard queries Google Sheets API in real-time
+- Interactions logged directly to Sheets (no manual re-entry)
+- Markdown fetched from Obsidian Sync API
+- Auto-refresh KPIs every 5 minutes
+
+---
+
+## 9. Extensibility Points (Future Phases)
+
+### Phase 2: Relationships & Interaction History
+- **Tab 4: Relationships** — Org chart, sentiment timeline, interaction history
+- **Interaction log** — Searchable, filterable by contact, type, sentiment
+- **Sentiment analysis** — Chart showing mood trend over time
+
+### Phase 3: Opportunities & Pipeline
+- **Tab 5: Opportunities** — Consolidation roadmap, deal timeline, revenue forecast
+- **Milestone tracking** — Countdown to RFP, contract close, etc.
+
+### Phase 4: Financial Modeling
+- **10-K integration** — Embed PDF link, key metrics extracted
+- **Opportunity forecasting** — If analytics consolidates, revenue impact
+- **Budget impact model** — Capex vs. OpEx trade-off calculator
+
+### Phase 5: Industry & Competitive Context
+- **Industry trends** — Curated insights relevant to account
+- **Competitor tracking** — Win/loss intelligence, displacement playbooks
+- **Market context** — Industry growth rates, regulatory changes
+
+---
+
+## 10. Technical Architecture
+
+### Technology Stack
+- **Frontend**: React (or vanilla JS if keeping simple)
+- **Styling**: Tailwind CSS (or custom CSS-in-JS)
+- **Data sources**: 
+  - Google Sheets API (for Account, Pure-Deployment, Relationships, Financials tabs)
+  - Markdown file (from Obsidian or GitHub)
+- **Deployment**: Claude artifact (publishable, shareable URL) or self-hosted on homelab
+- **Build**: Vite (fast dev) or Create React App
+
+### API Integration (MVP)
+```
+Frontend (React)
+    ↓
+    ├─→ Google Sheets API v4
+    │   └─ Fetch: Account!A:Z, PureDeployment!A:Z, Relationships!A:Z, etc.
+    │      Rate: On-demand or 5-min cache
+    │
+    └─→ Markdown fetch (via Obsidian API or raw GitHub/Drive link)
+        └─ Fetch: Markdown content, parse for snippets
+```
+
+### State Management
+- **Client-side state**: React hooks (useState, useEffect)
+- **Server state**: Google Sheets is source of truth
+- **UI state**: Which tab active, which card expanded
+- **No local storage** (MVP): Always fresh from source
+
+---
+
+## 11. Accessibility & Usability
+
+### Keyboard Navigation
+- Tab through cards, tabs, buttons
+- Enter/Space to expand cards
+- Escape to collapse all
+- Ctrl/Cmd+Click to open links in new tab
+
+### Screen Readers
+- Semantic HTML (headers, nav, main, article)
+- ARIA labels on custom components
+- Heading hierarchy (H1 = account name, H2 = tabs, H3 = card titles)
+
+### Color Contrast
+- All text meets WCAG AA standards
+- Status indicators have labels (not color alone)
+- Links underlined or clearly differentiated
+
+### Mobile Responsiveness (Future)
+- Stack panes vertically on small screens
+- Touch-friendly button sizes (44px minimum)
+- Collapsible sections for readability
+
+---
+
+## 12. Success Metrics (MVP)
+
+| Metric | Target | How to Measure |
+|--------|--------|---|
+| **Time to account snapshot** | <2 min to full context | User time log |
+| **Drill-down effectiveness** | Find key metric in <30 sec | User task test |
+| **Information completeness** | All 10 items readily visible | Checklist |
+| **Visual clarity** | No confusion on next steps | User feedback |
+| **Engagement** | Used for all 3 use cases equally | Usage logs |
+
+---
+
+## 13. Known Constraints & Trade-offs
+
+| Constraint | Impact | Mitigation |
+|-----------|--------|---|
+| **Manual sync (MVP)** | Data can lag if Sheets updated but dashboard not refreshed | Clear "last updated" timestamp; button to manual sync |
+| **No real-time collaboration** | If account team edits Sheets while viewing dashboard, changes won't appear until reload | Future: implement live sync |
+| **Single-customer dashboard** | Can't compare metrics across accounts in MVP | Future: roll-up view with filtering |
+| **API rate limits (Sheets)** | Can't refresh dashboard every second | Cache on client; user triggers refresh manually |
+
+---
+
+## 14. Success Criteria for Prototype
+
+✅ **Must Have (MVP)**
+- [ ] All 10 items (List 1 + List 2) visible and organized
+- [ ] 3-pane layout functioning (left/center/right)
+- [ ] Cards expand/collapse on click with animation
+- [ ] Health scorecard with 4 KPIs calculated
+- [ ] Timeline showing key dates
+- [ ] Sample data flowing in from mock Sheets
+- [ ] Navigation between tabs (Overview, List 1, List 2)
+- [ ] Status badges and visual indicators rendering correctly
+
+⚡ **Should Have (Nice to Have)**
+- [ ] Trend indicators (↑ ↓ →) on KPI cards
+- [ ] Sample charts (pie, line) in expanded views
+- [ ] Markdown snippet embedded in right pane
+- [ ] Interaction log with sentiment stars
+- [ ] Dark mode toggle
+
+🎯 **Nice to Have (Future)**
+- [ ] Google Sheets API actually connected (MVP uses mock data)
+- [ ] Relationships tab structure visible
+- [ ] QBR Prep modal/checklist
+- [ ] Share functionality
+- [ ] Export to PDF
+
+---
+
+## 15. Design Review Checklist
+
+**Questions for Ryan before prototype:**
+
+1. ✅ Is the 3-pane layout clear? Does it support the 10K → ground zoom?
+2. ✅ Are the 4 KPIs the right ones, or would you change them?
+3. ✅ Does the drill-down interaction (click card → expand) feel right, or prefer modal?
+4. ✅ Should the right pane be scrollable independently, or scroll with center pane?
+5. ✅ Any major UX concerns with the navigation flow?
+6. ✅ Is the timeline component clear (horizontal bar vs. vertical list)?
+7. ✅ Color palette acceptable, or want different accent colors?
+8. ✅ Any tabs or components you'd add/remove for MVP?
+
+---
+
+**Document Status:** Ready for Design Review & Feedback  
+**Next Step:** Prototype build (pending approval of this design)

Claude outputs/Enterprise-Storage-Account-Intelligence-Lists.md

@@ -0,0 +1,257 @@
+# Top-5 Account Intelligence Lists for Enterprise Storage SE
+
+**For:** Pre-Sales Systems Engineers at Pure Storage  
+**Focus Verticals:** Financial Services, Retail/Manufacturing  
+**Target Accounts:** Fortune 1000 / Global 2000
+
+---
+
+## LIST 1: ANY CUSTOMER OR PROSPECT
+### First Discovery Call + Competitive Displacement + Expansion Foundation
+
+---
+
+### 1. **Data Footprint and Workload Breakdown**
+
+**What to capture:**
+- Total addressable storage capacity (current + 2-year projection)
+- Workload tier split: critical OLTP (databases, transactional), warm operational analytics (data warehouses, ML training sets), unstructured (data lakes, backups, archives)
+- Annual growth rate
+- Typical IO patterns (sequential vs. random, read/write mix)
+- Workload sources: core business apps, digital channels, analytics platforms, third-party integrations
+
+**Why it ranks #1:**
+This is your credibility foundation for the discovery call. Large enterprises in financial services and retail/manufacturing operate at scale—you need to know if they're managing 10PB or 500PB of storage before you can ask intelligent questions about consolidation, resilience, or performance. For competitive displacement, it tells you the total addressable market and which workloads are most valuable to move. For expansion, it shows you where growth is (analytics workloads often outpace transactional growth in these verticals).
+
+**Vertical differences:**
+- **Financial services:** Regulatory data tiers (Basel III, Dodd-Frank, GDPR-driven data governance), transaction volumes, archive-to-compliance mandates, and fraud detection datasets are outsized here.
+- **Retail/manufacturing:** Omnichannel transaction data, supply chain event data, and IoT/sensor streams create different growth patterns—often unstructured-heavy.
+
+**Where to get it:**
+Account team, analyst kickoff, customer IT strategy documents. If public: earnings calls (capex guidance), analyst reports on data center expansion. **Hard to verify fully from public sources—lean on the account team and discovery conversation.** Pure1 for existing customers (capacity trends, growth rates).
+
+---
+
+### 2. **Current Storage Vendor(s) and Installed Infrastructure**
+
+**What to capture:**
+- Primary vendor(s), total installed capacity per vendor, system age/generation
+- Deployment model (on-prem, cloud-hosted)
+- Vendor concentration (single-vendor, multi-vendor)
+- Capacity consolidation potential (old systems due refresh, overlapping vendors)
+- Competitive systems they've ruled out or are evaluating
+
+**Why it ranks #2:**
+This is the competitive map. In discovery, you need to know who you're up against to ask credible questions about their stack. For displacement, this is your roadmap—old NetApp or EMC systems are your opening; vendor sprawl is your consolidation story. For expansion, it shows where Pure is (or isn't) and where you can grow.
+
+**Applies to both verticals:**
+Regulatory and operational requirements often enforce heterogeneous stacks (legacy core systems, newer analytics platforms), creating multi-vendor complexity and Pure's consolidation angle.
+
+**Where to get it:**
+Account team, customer IT contacts, Gartner/IDC reports, job postings (hiring for specific platforms signals infrastructure). Pure's competitive intel teams (if available in your region). **Harder to get from public sources—interview in discovery.**
+
+---
+
+### 3. **Business Drivers and Strategic Imperatives**
+
+**What to capture:**
+- What's moving their infrastructure strategy in the next 12–24 months?
+- Digital transformation (cloud-first, API-led architecture)
+- M&A (data center consolidation, platform rationalization)
+- Regulatory change (data residency, compliance modernization)
+- Cost reduction (capex or opex pressure)
+- Performance demands (real-time analytics, AI/ML acceleration)
+- Geographic expansion (multi-site, disaster recovery)
+
+**Why it ranks #3:**
+This explains *why* they'll spend and *when*. For discovery, it tells you what your conversation should be anchored to—cost pressure is a different conversation than compliance urgency. For displacement, it reveals openings (legacy vendor can't support cloud migration; incumbent doesn't scale for analytics). For expansion, it shows where budget flows and growth cycles.
+
+**Vertical differences:**
+- **Financial services:** Regulatory change (Fed data residency mandates, CCAR/stress test infrastructure), fraud detection (AI/ML workloads), and open banking/fintech competition drive transformation.
+- **Retail/manufacturing:** Omnichannel unification, supply chain visibility, and sustainability reporting (ESG data) are primary drivers; cost pressure is often secondary.
+
+**Where to get it:**
+Earnings calls (CFO and CRO commentary), industry analyst reports, customer executives in discovery (CEOs and CFOs speak to this), account team (relationship intelligence). Regulatory announcements for financial services. **Public sources vary—direct ask is usually most productive.**
+
+---
+
+### 4. **Infrastructure Architecture and Resilience Model**
+
+**What to capture:**
+- Topology: single data center, active-active multi-site, hub-and-spoke?
+- Failover strategy (RPO/RTO targets, synchronous vs. asynchronous replication)
+- Cloud presence (public cloud, hybrid, private cloud)
+- Containerization and Kubernetes adoption
+- Storage tiers and tiering automation
+- Backup/archive strategy (separate systems, integrated)
+
+**Why it ranks #4:**
+This determines *which* Pure solutions fit and how. Knowing they're multi-site tells you Evergreen model and replication are critical; knowing they have Kubernetes tells you Portworx is on the menu; knowing they're chasing sub-second RTO tells you FlashArray's performance story matters. For discovery, it frames the conversation around resilience or performance. For displacement, it shows where Pure's architecture outperforms incumbent (e.g., Evergreen's native replication vs. bolted-on replication). For expansion, it guides where to add capacity.
+
+**Vertical differences:**
+- **Financial services:** Recovery requirements are often strict (RTO <1hr, RPO <15min); multi-site is standard; compliance mandates geographic separation.
+- **Retail/manufacturing:** Multi-site is common (stores, plants, distribution); RTO is important but often longer than financial services; hybrid cloud adoption is accelerating.
+
+**Where to get it:**
+Account team, technical RFI/RFQ documents, infrastructure diagrams (security-sensitive—may not be public). Discovery conversation with IT infrastructure/storage teams. **Hard from public sources.**
+
+---
+
+### 5. **Budget Cycle, Capital Allocation, and Buying Authority**
+
+**What to capture:**
+- Capex budget cycle and approval gates (who approves? quarterly, annual, rolling?)
+- Capex vs. opex preference (traditional capex, Evergreen-style subscription, OpEx-only mandate)
+- Budget envelope for infrastructure over next 2 years
+- Preferred buying patterns (competitive bid, existing vendor preference, tech refresh cycle)
+- Influencers and decision-makers (CIO, VP Storage, business unit leaders, CFO)
+
+**Why it ranks #5:**
+This is the "when and how to sell" lever. It's less about discovery credibility (though knowing budget is essential) and more about deal realism. For discovery, it tells you timing and buying authority to route follow-ups. For displacement, it tells you if capex is available or if you need to frame Evergreen as an alternative. For expansion, it reveals budget headroom and buying rhythm.
+
+**Applies to both verticals:**
+Financial services often have tighter capex scrutiny and longer approval cycles; retail/manufacturing may have faster refresh cycles but more budget constraints post-acquisition integration.
+
+**Where to get it:**
+Account team relationship notes, past deal history, 10-K/10-Q filings (capex guidance), analyst reports, budget forecasts shared in customer relationships. **Rarely fully public—account team and direct relationship intelligence are key.**
+
+---
+
+## LIST 2: EXISTING CUSTOMERS — PURE STORAGE DEPLOYMENT SPECIFICS
+### Expansion + Retention + Competitive Defense
+
+---
+
+### 1. **Current Pure Installed Base, Deployment Pattern, and Capacity**
+
+**What to capture:**
+- FlashArray systems (models, capacity per system, tier across data centers or cloud regions)
+- FlashBlade deployments (scale-out analytics use cases, capacity)
+- Portworx deployments (if any; containerized workloads, environments)
+- Age of systems (when deployed; refresh cycle status)
+- Geographic footprint (on-prem across which data centers, any cloud-hosted Pure?)
+- Utilization and headroom (at capacity, growing into headroom, or sparse?)
+
+**Why it ranks #1:**
+You can't expand without knowing your baseline. For expansion, this is your starting point—you need to know what they own to identify the next sale (add capacity, new tier, new workload). For retention, it's your health baseline (old systems more likely to churn). For competitive defense, it shows Pure's footprint and where competitors might try to take share.
+
+**Key indicators for retention/expansion:**
+Systems older than 5 years are refresh candidates; sparse utilization may indicate underutilized tier or expansion opportunity; multi-site Pure deployments are higher-value accounts.
+
+**Where to get it:**
+**Pure1 (your best data source—capacity, performance, age, location),** internal Pure account data, customer IT/storage teams. Direct ask in customer reviews and strategic business reviews (QBR/SBR).
+
+---
+
+### 2. **Performance, Utilization, and Customer Satisfaction**
+
+**What to capture:**
+- Pure1 performance metrics (latency, IOPS achieved vs. provisioned, dedupe/compression ratios, capacity growth rate)
+- Storage team feedback (support experience, feature requests, known pain points)
+- Performance satisfaction (are performance SLAs being met?)
+- Operational efficiency gains (cost per GB, power/cooling reduction)
+- Customer sentiment (NPS or equivalent; any churn signals?)
+
+**Why it ranks #2:**
+This tells you expansion potential vs. churn risk. If they're unsatisfied, expansion is at risk (competitor threat, bargaining power). If they're thriving on Pure, expansion is easier (proven business case). For retention, this is your early warning system. For expansion, high satisfaction is your foundation for upsells (new workload, new tier, cloud-native).
+
+**Vertical differences:**
+- **Financial services:** Customers often have strict uptime/performance mandates.
+- **Retail/manufacturing:** Often cost-conscious and may trade performance for efficiency.
+
+**Where to get it:**
+**Pure1 (performance analytics),** NPS/CSAT surveys, QBR discussions, support ticket sentiment, sales/SE relationship notes. Direct ask: "How is Pure performing for you?"
+
+---
+
+### 3. **Data Footprint Not Yet on Pure / Coverage Gap Analysis**
+
+**What to capture:**
+- Percentage of their total storage on Pure vs. other vendors
+- Workload segments not on Pure (e.g., older OLTP databases on EMC, analytics on Hadoop/object storage, backups on separate vendor)
+- Highest-value data not on Pure (revenue-generating apps, critical analytics)
+- Consolidation opportunities (multi-vendor overlap, legacy systems ready to refresh)
+- Workloads Pure could address but hasn't yet (emerging use cases, cloud data, new business initiatives)
+
+**Why it ranks #3:**
+This is your expansion TAM within the account. For expansion, it shows where growth comes from (consolidation opportunity, new workload, tier migration). For competitive defense, it identifies where competitors will attack (the gap is their opening). For retention, it's leverage—you want to own more of their data.
+
+**Vertical differences:**
+- **Financial services:** Critical OLTP often stays on incumbent (lower risk perception); newer analytics and fraud detection data are Pure opportunities.
+- **Retail/manufacturing:** Supply chain/IoT data and unstructured analytics are often on non-Pure platforms; consolidation gains are high here.
+
+**Where to get it:**
+**Pure1 (shows only Pure capacity—you need to ask for total footprint),** customer IT (data inventory), discovery/expansion conversations. **Often not transparent from public sources—direct conversation essential.**
+
+---
+
+### 4. **Upcoming Infrastructure Initiatives and Technology Refresh Cycles**
+
+**What to capture:**
+- Planned data center consolidation or modernization
+- System refresh dates (when is their oldest Pure system past end-of-support? When's the next capex cycle?)
+- Cloud migration plans (moving workloads to cloud; implications for on-prem storage)
+- New workloads (AI/ML platforms, real-time analytics, new business lines)
+- Multi-site expansion or disaster recovery modernization
+
+**Why it ranks #4:**
+Timing is everything in expansion. Refresh cycles are your biggest lever (old systems = new capex = new opportunity). Cloud migration can cut Pure TAM or open Portworx opportunities. New workloads are expansion hooks. For expansion, this is your calendar. For retention, knowing refresh timing lets you plan defense early.
+
+**Applies to both verticals:**
+Financial services often have annual capex cycles tied to business planning; retail/manufacturing may refresh faster or slower depending on store/plant footprint changes.
+
+**Where to get it:**
+Customer IT roadmap (often multi-year), Pure account team, QBR discussions, earnings announcements (for large public customers, CapEx guidance), customer executives. Sometimes in customer's published IT strategy.
+
+---
+
+### 5. **Stakeholder Relationships, Expansion Readiness, and Competitive Threats**
+
+**What to capture:**
+- Key stakeholders (CIO, VP Storage, business unit leaders, security/compliance officers)
+- Champion(s) (who advocates for Pure?)
+- Relationship temperature (trust, satisfaction, barriers)
+- Competitive interest (have they mentioned other vendors? RFPs? Evaluation?)
+- Budget headroom (will they invest more in storage in next 12–24 months?)
+- Expansion appetite (are they open to moving more workloads to Pure, or focused on optimization?)
+
+**Why it ranks #5:**
+This is your "how to expand" lever. Knowing who the champion is tells you how to navigate. Knowing if they're under competitive attack tells you how urgent your defense is. Knowing budget appetite tells you if expansion is realistic. For expansion, this is your relationship risk/opportunity assessment. For retention, it's your early warning for churn or your pathway for growth.
+
+**Applies to both verticals:**
+In large enterprises, storage decisions often flow through business unit leaders (not just IT); understanding where Pure fits in their strategic priorities is key.
+
+**Where to get it:**
+Account team relationship notes, past deal history, QBR feedback, sales/SE interactions, industry/analyst events. Direct ask in customer conversations: "Where do you see us helping you grow?"
+
+---
+
+## Summary: Why This Order Matters
+
+**List 1** (any customer/prospect) moves from **foundational understanding** (what and who) → **context** (why and how) → **execution** (when and who decides). This sequence builds credibility in discovery while revealing competitive and expansion levers.
+
+**List 2** (existing customer Pure deployments) moves from **your current position** (what you have) → **relationship health** (how it's performing) → **opportunity size** (what you could have) → **timing** (when they'll buy) → **execution** (how to navigate the sale). This sequence is your expansion playbook.
+
+### Key Callout
+In both lists, **items 1–3 should drive 80% of your discovery and account strategy effort**. Items 4–5 refine timing and approach. **Start here, then layer in the operational details** (support SLAs, feature adoption, roadmap alignment) as you deepen the relationship.
+
+---
+
+## Quick Reference: Source Hierarchy by Item
+
+| Item | Primary Source | Secondary Source | Public Availability |
+|------|---|---|---|
+| **List 1 #1** – Data Footprint | Account team, customer IT | Earnings calls, analyst reports | Moderate (public guidance varies) |
+| **List 1 #2** – Current Vendors | Account team, IT contacts | Job postings, Gartner reports | Low (often internal knowledge) |
+| **List 1 #3** – Business Drivers | Customer executives, analyst reports | Earnings calls, regulatory filings | Moderate (CEOs/CFOs speak to this) |
+| **List 1 #4** – Architecture | Technical RFI/RFQ, account team | Discovery conversation | Low (security-sensitive) |
+| **List 1 #5** – Budget/Authority | Account team, past deal history | 10-K/10-Q, customer relationships | Low (relationship intelligence) |
+| **List 2 #1** – Pure Installed Base | **Pure1** (best source), account data | Customer IT, QBR | Account-internal only |
+| **List 2 #2** – Performance/Satisfaction | **Pure1**, NPS surveys | Support tickets, QBR discussions | Account-internal only |
+| **List 2 #3** – Coverage Gaps | Customer IT inventory, discovery | Account team insights | Low (direct conversation required) |
+| **List 2 #4** – Refresh Cycles | Customer IT roadmap, account team | QBR, earnings announcements | Low-Moderate (customer-dependent) |
+| **List 2 #5** – Stakeholder/Competitive | Account team, sales history | QBR feedback, customer conversations | Account-internal only |
+
+---
+
+**Use this as your go-to reference before discovery calls, account strategy reviews, and expansion planning.**

README.md

@@ -1,102 +1,102 @@
-# A Homelab based on Red Hat Technologies
+# mk-labs
 
-This repository is the configuration of my homelab. In addition to providing services, the purpose of my homelab is to learn advanced concepts primarily based on Red Hat OpenShift.
+Automated infrastructure provisioning and configuration for a personal homelab, built on GitOps practices with clear tool responsibility boundaries.
 
-This implementation is built on easily accessible consumer based hardware and will focus heavily on GitOps practices and automation will be used wherever possible.
+## Architecture
 
-This repo is a mono-repo that is broken up into three sections:
+A single operator action — setting a VM record's status to **Staged** in NetBox — triggers a fully automated provisioning pipeline:
+```
+NetBox (webhook) → n8n (validate & orchestrate) → Terraform (create VM + DHCP)
+                                                 → Ansible (OS config + DNS + status update)
+```
 
-- infra-config
-- apps
-- cluster
+| Tool | Host | IP | Responsibility |
+|------|------|----|---------------|
+| NetBox | fire-station | 10.1.71.102 | Source of truth — VM records, IP allocation, VLAN data |
+| n8n | tiki-room | 10.1.71.23 | Event orchestration, validation, pipeline sequencing |
+| Terraform | city-hall | 10.1.71.35 | Proxmox VM lifecycle, Unifi DHCP reservations |
+| Ansible / Semaphore | imagineering | 10.1.71.22 | OS configuration, DNS records, NetBox status updates |
+| Proxmox | fantasyland | 10.1.71.13 | Target hypervisor |
+
+All systems on the Server Trusted VLAN (10.1.71.0/24).
+
+## Repository Structure
+```
+homelab/
+├── ansible/
+│   ├── inventory/          # NetBox dynamic inventory + static
+│   ├── playbooks/          # Runnable playbooks (vm-provision, DNS, OS updates)
+│   ├── roles/              # vm-baseline, dns-manager, common, haproxy, n8n, observer, etc.
+│   ├── tasks/              # Shared includable task files
+│   ├── group_vars/         # Group variable definitions
+│   ├── host_vars/          # Per-host variable definitions
+│   ├── templates/          # Jinja2 templates
+│   └── ansible.cfg
+│
+├── terraform/
+│   ├── proxmox/vm/         # bpg/proxmox provider — VM creation from templates
+│   ├── unifi/dhcp/         # Unifi provider — DHCP static reservations on UDM Pro
+│   └── dns/                # DNS record management
+│
+├── packer/
+│   ├── ubuntu-24.04/       # Ubuntu 24.04 VM template (small → xlarge-plus sizes)
+│   └── fedora-42/          # Fedora 42 VM template
+│
+├── n8n/
+│   └── workflows/          # Exported n8n workflow JSON (vm-provisioning)
+│
+├── netbox/
+│   └── initializers/       # Custom fields, VLANs, IP prefixes as code
+│
+└── docs/
+    └── decisions/          # Architecture decision records
+```
+
+## Pipeline Flow
+
+| # | System | Action |
+|---|--------|--------|
+| 1 | NetBox | Operator sets VM status to Staged → webhook fires |
+| 2 | n8n | Validates payload (hostname, IP, VLAN, template, proxmox_node) |
+| 3 | n8n → city-hall | SSH + `terraform apply` — creates VM on Proxmox |
+| 4 | n8n | Queries Proxmox API for MAC address |
+| 5 | n8n → NetBox | Writes MAC to VM interface record |
+| 6 | n8n → city-hall | SSH + `terraform apply` — creates DHCP reservation on UDM Pro |
+| 7 | n8n → imagineering | Triggers Ansible via Semaphore API |
+| 8 | Ansible | OS baseline, SSH hardening, Technitium DNS A record |
+| 9 | Ansible → NetBox | Sets VM status to Active |
+
+On any failure, NetBox status is set to **Failed**. No auto-retry — operator investigates.
 
 ## Hardware
 
-- Dell 7050 SFF (7)
-- Minisforum TH60 (3)
-- Minisforum MS01 (2)
-- Synology 1621+
+- 7× Dell 7050 SFF
+- 3× Minisforum TH60
+- 2× Minisforum MS01
+- Synology DS1621+
+- Ubiquiti UDM Pro
 
-## Software
+## Software Stack
 
-- Proxmox (Virtualization)
-- Cloudflare (Domain Hosting, Public DNS)
-- Unbound (Recursive DNS)
-- FreeIPA (Identity management, Authoritive DNS)
-- Matchbox (iPXE)
-- Red Hat OpenShift
-- OpenShift Agent Based Installer (Install OpenShift)
-  - [Red Hat Advanced Cluster Management for Kubernetes](https://www.redhat.com/en/technologies/management/advanced-cluster-management)
-  - [Vault](https://www.hashicorp.com/en/products/vault)
-  - [OpenShift GitOps (ArgoCD)](https://www.redhat.com/en/technologies/cloud-computing/openshift/gitops)
-  - [Red Hat Ansible Automation Platform](https://www.redhat.com/en/technologies/management/ansible)
-
-## Prerequisites
-
-- Ansible user created
-- Ansible configured
-- [Networking](docs/networks.md)
-- [Proxmox](docs/proxmox.md) (In my homelab, internal DNS, identity manangement, and ipxe are hosted here.)
-- Matchbox
-- DNS
-- Domain Registration
-
-## Assumptions
-
-There are a dozen different architectures you could use to deploy OpenShift in every which way.
-For the sake of this documentation we'll assume the following:
+- **Virtualization**: Proxmox
+- **Automation**: Terraform, Ansible, n8n, Semaphore
+- **DNS**: Technitium (authoritative), Unbound (recursive)
+- **IPAM/DCIM**: NetBox
+- **Networking**: Ubiquiti UDM Pro
+- **Templates**: Packer (Ubuntu 24.04, Fedora 42)
 
 ## Getting Started
 
-[Step-by-Step Walkthrough](step-by-step.md)
+See [docs/decisions/vm-provisioning-flow.md](docs/decisions/vm-provisioning-flow.md) for the full architecture decision record.
 
-### Hub Cluster
+Previous OpenShift/ACM/Fastpass content is preserved in the `archive/pre-mk-labs` branch.
 
-You'll need an OpenShift "Hub Cluster" with access to persistant storage.
-A Single Node OpenShift (SNO) instance, installed on bare metal, will act as a Hub cluster and run:
+## Security
 
-- Advanced Cluster Management
-- Ansible Automation Platform
-- Vault
-- ~~OpenShift GitOps~~
-
-#### Network Prerequisites
-
-The prerequisites for OpenShift in traditional and HCP patterns are largely the same - it just kind of depends on where your DNS records go to.
-
-| Cluster           | Endpoint    | VIP           | DNS A Record                      | Notes                           |
-|------------------|-------------|---------------|-----------------------------------|----------------------------------|
-| Hub Cluster (SNO) | App Ingress | 192.168.0.10 | *.apps.hub-cluster.example.com    | SNO App VIP goes to IP of node  |
-| Hub Cluster (SNO) | API         | 192.168.0.10 | api.hub-cluster.example.com       | SNO API goes to IP of node      |
-
-These DNS entries should be put in your Authoratitive DNS.
-
-#### ACM & GitOps Configuration
-
-Before you start creating clusters you may want to create some Policies, integrate ACM and ArgoCD, etc. This step is optional in case you're just interested in trying out Hosted Control Planes or copy/paste around a cluster for testing purposes.
-
-Find additional details in the ./02-rhacm-config folder.
-
-#### Creating a Cluster
-
-With everything in its right place, you can now start to declaratively create clusters
-
-./05-clusters/hcp-bmh - HCP to Bare Metal Hosts
-
-### Internal Cluster
-
-Two additional bare metal nodes, to be added to Advanced Cluster Management (ACM) running on the SNO Hub. These will be used to create another HCP cluster.
-These servers have a BMC interface with Redfish - if not, then you'll need to manually manage the boot and installation of those servers.
-This makes it to where you just need 3 bare metal nodes. You could run one HCP Bare Metal cluster with both of the other nodes, but then you have a shared storage requirement that can't be satisfied by ODF since that needs at least 3 nodes.
-
-### External Cluster
-
-You'll also either need you just need at least 2 bare metal nodes.
+No sensitive data is stored in this repository. Secrets are managed via Ansible Vault and environment variables on pipeline hosts.
 
 ---
 
-## Credits
+**Status**: 🚧 Active Development — VM Provisioning Pipeline
 
-- Ken Moini - As I used his [repo](https://github.com/kenmoini/ztp-for-you-and-me) as the baseline for this project.
-- Ryan Etten
-- Andrew Potozniak
+**Last Updated**: February 2026

ansible/DNS_MANAGEMENT_REFACTOR.md

@@ -0,0 +1,193 @@
+# DNS Management Refactor - Modular & Repeatable
+
+This document explains the refactoring of DNS management from standalone playbooks to modular, reusable tasks.
+
+## 🎯 **What Changed**
+
+### **Before (Monolithic)**
+```yaml
+# Standalone playbook: add_technitium_dns_entry.yml
+- name: Add entry to Technitium DNS
+  hosts: all
+  tasks:
+    - name: Create DNS entry
+      effectivelywild.technitium_dns.technitium_dns_add_record:
+        # ... hardcoded parameters
+```
+
+### **After (Modular)**
+```yaml
+# Reusable task: tasks/add_technitium_dns_entry.yml
+- name: Create DNS entry for {{ dns_record_name }}
+  effectivelywild.technitium_dns.technitium_dns_add_record:
+    # ... parameterized with variables
+```
+
+## 📁 **New Structure**
+
+```
+ansible/
+├── playbooks/
+│   ├── tasks/
+│   │   └── add_technitium_dns_entry.yml    # ✅ Reusable task file
+│   ├── add_dns_entry.yml                   # ✅ New playbook using task
+│   ├── add_technitium_dns_entry.yml.backup # 📦 Backed up old version
+│   └── roles/
+│       └── dns-manager/                    # ✅ Enhanced role
+│           ├── tasks/main.yml              # Uses task file
+│           └── defaults/main.yml           # Technitium defaults
+└── scripts/
+    └── migrate-dns-references.sh           # ✅ Migration helper
+```
+
+## 🚀 **Usage Examples**
+
+### **1. In Playbooks (Direct Task Include)**
+```yaml
+- name: Add DNS entry for my server
+  hosts: my_servers
+  tasks:
+    - name: Create DNS entry
+      ansible.builtin.include_tasks: tasks/add_technitium_dns_entry.yml
+      vars:
+        dns_record_name: "{{ inventory_hostname }}"
+        dns_zone: "{{ base_domain }}"
+        dns_ip_address: "{{ ansible_default_ipv4.address }}"
+```
+
+### **2. Using the DNS Manager Role**
+```yaml
+- name: Setup cluster DNS
+  hosts: control_plane[0]
+  roles:
+    - role: dns-manager
+      vars:
+        cluster_endpoint: "my-cluster.local.mk-labs.cloud"
+        cluster_vip: "10.1.71.100"
+```
+
+### **3. Using the New Playbook**
+```yaml
+# Import the new modular playbook
+- import_playbook: add_dns_entry.yml
+```
+
+### **4. In Cluster Network Setup**
+```yaml
+- name: Setup complete cluster network
+  ansible.builtin.include_role:
+    name: cluster-network-setup
+  vars:
+    cluster_name: "fastpass"
+    cluster_endpoint: "fastpass.local.mk-labs.cloud"
+    cluster_vip: "10.1.71.53"
+```
+
+## 🔧 **Migration Guide**
+
+### **Automatic Migration**
+```bash
+# Run the migration script to find references
+./scripts/migrate-dns-references.sh
+```
+
+### **Manual Updates**
+
+1. **Replace playbook imports:**
+   ```yaml
+   # OLD
+   - import_playbook: add_technitium_dns_entry.yml
+   
+   # NEW
+   - import_playbook: add_dns_entry.yml
+   ```
+
+2. **Use task includes in roles:**
+   ```yaml
+   - ansible.builtin.include_tasks: tasks/add_technitium_dns_entry.yml
+     vars:
+       dns_record_name: "my-server"
+       dns_ip_address: "10.1.71.100"
+   ```
+
+3. **Use the dns-manager role:**
+   ```yaml
+   - ansible.builtin.include_role:
+       name: dns-manager
+   ```
+
+## 📋 **Variable Reference**
+
+### **Task Variables (`tasks/add_technitium_dns_entry.yml`)**
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `dns_record_name` | `inventory_hostname` | DNS record name |
+| `dns_zone` | `base_domain` | DNS zone |
+| `dns_ip_address` | `ip_address` | IP address for A record |
+| `dns_record_type` | `A` | DNS record type |
+| `dns_ttl` | `360` | TTL in seconds |
+| `dns_create_ptr` | `true` | Create PTR record |
+| `dns_debug` | `true` | Show debug output |
+
+### **DNS Manager Role Variables**
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `cluster_endpoint` | - | Full cluster FQDN |
+| `cluster_vip` | - | Cluster VIP address |
+| `dns_management_enabled` | `true` | Enable DNS management |
+| `use_hosts_file_fallback` | `true` | Add to /etc/hosts |
+| `dns_ttl` | `360` | DNS TTL |
+| `create_ptr_record` | `true` | Create PTR record |
+
+## 🎯 **Benefits**
+
+### ✅ **Modularity**
+- Single task file used across multiple contexts
+- Consistent DNS management approach
+- Easy to maintain and update
+
+### ✅ **Flexibility**
+- Works in playbooks, roles, and standalone
+- Parameterized for different use cases
+- Supports multiple DNS providers (extensible)
+
+### ✅ **Maintainability**
+- One place to update DNS logic
+- Clear variable interface
+- Better error handling and debugging
+
+### ✅ **Integration**
+- Seamlessly integrates with cluster setup
+- Works with existing homelab infrastructure
+- Compatible with Traefik load balancer setup
+
+## 🔄 **Integration with Cluster Setup**
+
+The DNS management now integrates seamlessly with your cluster deployment:
+
+```yaml
+# In fastpass-first-control-plane role
+- name: Setup network infrastructure for FastPass cluster
+  ansible.builtin.include_role:
+    name: cluster-network-setup
+  vars:
+    cluster_name: "{{ cluster_name }}"
+    cluster_endpoint: "{{ control_plane_endpoint }}"
+    cluster_vip: "{{ ansible_default_ipv4.address }}"
+    control_plane_nodes: "{{ groups['fastpass_control_plane'] }}"
+```
+
+This automatically:
+1. ✅ Creates DNS entry for `fastpass.local.mk-labs.cloud`
+2. ✅ Configures Traefik load balancer
+3. ✅ Tests connectivity
+4. ✅ Provides fallback to /etc/hosts
+
+## 🚀 **Next Steps**
+
+1. **Test the refactored approach** with your FastPass cluster
+2. **Extend to other clusters** (Hub, Internal) using the same pattern
+3. **Add support for other DNS providers** if needed
+4. **Create monitoring** for DNS health checks
+
+This modular approach makes your homelab's DNS management much more maintainable and repeatable across all your Kubernetes clusters!

ansible/ansible.cfg

@@ -54,7 +54,7 @@
 
 # (pathspec) Colon-separated paths in which Ansible will search for collections content. Collections must be in nested *subdirectories*, not directly in these directories. For example, if ``COLLECTIONS_PATHS`` includes ``'{{ ANSIBLE_HOME ~ "/collections" }}'``, and you want to add ``my.collection`` to that directory, it must be saved as ``'{{ ANSIBLE_HOME} ~ "/collections/ansible_collections/my/collection" }}'``.
 
-;collections_path=/Users/rblundon/.ansible/collections:/usr/share/ansible/collections
+collections_path=/opt/ansible-collections:/usr/share/ansible/collections
 
 # (boolean) A boolean to enable or disable scanning the sys.path for installed collections.
 ;collections_scan_sys_path=True
@@ -209,7 +209,7 @@ private_key_file=~/.ssh/ansible
 remote_user=wed
 
 # (pathspec) Colon-separated paths in which Ansible will search for Roles.
-;roles_path=/Users/rblundon/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
+roles_path=./roles
 
 # (string) Set the main callback used to display Ansible output. You can only have one at a time.
 # You can have many other callbacks, but just one can be in charge of stdout.
@@ -262,7 +262,7 @@ remote_user=wed
 
 # (path) The vault password file to use. Equivalent to ``--vault-password-file`` or ``--vault-id``.
 # If executable, it will be run and the resulting stdout will be used as the password.
-;vault_password_file=
+vault_password_file=/home/hermes/.vault_pass.txt
 
 # (integer) Sets the default verbosity, equivalent to the number of ``-v`` passed in the command line.
 ;verbosity=0

ansible/create_jarvis_user.yml

@@ -0,0 +1,6 @@
+---
+- name: Create jarvis user and deploy SSH key
+  hosts: all
+  become: true
+  roles:
+    - jarvis_user

ansible/group_vars/all/semaphore.yml

@@ -0,0 +1,193 @@
+---
+# ============================================================================
+# Semaphore configuration-as-code
+# ============================================================================
+# Drives a freshly-deployed Semaphore instance into its desired state via
+# the Semaphore REST API. Idempotent: every object is checked first; only
+# missing ones are created. Existing objects are left alone.
+#
+# Loaded from group_vars/all/semaphore.yml so that the configuration is
+# version-controlled in the homelab repo and survives a wipe-and-redeploy
+# of the Semaphore VM.
+# ============================================================================
+
+# ---------------------------------------------------------------------------
+# API connection (defaults to the local Traefik-fronted service-name URL).
+# Override semaphore_api_url to point at a specific instance if needed.
+# ---------------------------------------------------------------------------
+semaphore_api_url: "https://semaphore.local.mk-labs.cloud/api"
+semaphore_api_validate_certs: true
+semaphore_api_token: "{{ vault_semaphore_api_token }}"
+
+# Feature flag — keeps day1_deploy_semaphore.yml deploy-only by default.
+# Set true to also run the configuration pass.
+semaphore_configure: false
+
+# ---------------------------------------------------------------------------
+# Declarative configuration of the Semaphore instance.
+# ---------------------------------------------------------------------------
+#
+# Top-level shape:
+#
+#   semaphore_config:
+#     project:        single dict — the lab uses one project ("mk-labs")
+#     keys:           list of credentials Semaphore stores
+#     repositories:   git repos Semaphore can clone
+#     inventories:    Ansible inventories from those repos
+#     environments:   env-var bundles
+#     templates:      task templates that tie everything together
+#
+# Each list element has a unique "name" used as the natural identity key.
+# ---------------------------------------------------------------------------
+
+semaphore_config:
+  project:
+    name: mk-labs
+    alert: false
+    max_parallel_tasks: 0  # 0 = unlimited
+
+  keys:
+    # The ansible-vault password. login_password type with empty login
+    # — only the password field is consumed by Semaphore at runtime.
+    - name: ansible-vault-pass
+      type: login_password
+      login: ""
+      password: "{{ vault_ansible_vault_password }}"
+
+    # SSH key for the gitea deploy access (clone the homelab repo).
+    - name: gitea-deploy
+      type: ssh
+      ssh_login: git
+      ssh_private_key: "{{ vault_gitea_deploy_key }}"
+
+    # SSH key for the universal automation account 'wed' — pre-baked in
+    # every mk-labs VM template. This is the canonical user Semaphore
+    # uses to reach the fleet.
+    - name: wed-ssh
+      type: ssh
+      ssh_login: wed
+      ssh_private_key: "{{ vault_wed_ssh_private_key }}"
+
+    # SSH key Semaphore can use to reach the fleet as jarvis (admin
+    # account provisioned by linux-baseline). Retained for jobs that
+    # specifically need jarvis-level access; the default is wed-ssh.
+    - name: jarvis-ssh
+      type: ssh
+      ssh_login: jarvis
+      ssh_private_key: "{{ vault_jarvis_ssh_private_key }}"
+
+  repositories:
+    - name: homelab
+      git_url: "ssh://git@gitea.mk-labs.cloud:2221/rblundon/homelab.git"
+      git_branch: main
+      ssh_key: gitea-deploy
+
+  inventories:
+    - name: production
+      type: file
+      inventory_file: ansible/inventory.yml
+      repository: homelab
+      # wed is the universal automation account pre-baked in every VM
+      # template. Semaphore uses it for fleet-wide jobs.
+      ssh_key: wed-ssh
+      # become_key is Semaphore's sudo PASSWORD slot, not a second SSH
+      # key. wed has passwordless sudo on every host, so reference the
+      # built-in "None" key. (Semaphore rejects an SSH-type key here.)
+      become_key: None
+
+  environments:
+    - name: default
+      env:
+        ANSIBLE_HOST_KEY_CHECKING: "False"
+        ANSIBLE_FORCE_COLOR: "True"
+        # Semaphore runs ansible-playbook from the cloned REPO ROOT (not
+        # from the playbook's directory as I first assumed). Path is
+        # therefore relative to repo root, not playbook dir.
+        ANSIBLE_ROLES_PATH: "ansible/roles"
+        # Collections are installed by the semaphore role into a host-side
+        # directory bind-mounted into the container at this path.
+        ANSIBLE_COLLECTIONS_PATH: "/opt/ansible-collections"
+
+  templates:
+    - name: "day0_linux_baseline"
+      description: "Apply the mk-labs Linux baseline to one or more hosts."
+      app: ansible
+      playbook: ansible/playbooks/day0_linux_baseline.yml
+      inventory: production
+      repository: homelab
+      environment: default
+      vault_password: ansible-vault-pass
+      arguments: '["--diff"]'
+      survey_vars:
+        - name: target
+          title: "Target host or group"
+          description: "Inventory target (e.g. figment, semaphore_server, all)"
+          required: true
+          type: TextVar
+          default_value: "all"
+
+    - name: "day1_deploy_semaphore"
+      description: "Re-deploy Semaphore + PostgreSQL on figment."
+      app: ansible
+      playbook: ansible/playbooks/day1_deploy_semaphore.yml
+      inventory: production
+      repository: homelab
+      environment: default
+      vault_password: ansible-vault-pass
+      arguments: '["--diff"]'
+
+    - name: "day0_linux_baseline_check"
+      description: "Dry-run the baseline — shows diffs, applies nothing."
+      app: ansible
+      playbook: ansible/playbooks/day0_linux_baseline.yml
+      inventory: production
+      repository: homelab
+      environment: default
+      vault_password: ansible-vault-pass
+      arguments: '["--check","--diff"]'
+      survey_vars:
+        - name: target
+          title: "Target host or group"
+          description: "Inventory target (e.g. figment, semaphore_server, all)"
+          required: true
+          type: TextVar
+          default_value: "all"
+
+    - name: "llm_inference_multimodel_stage_models"
+      description: >-
+        Stage additional GGUF models into /opt/models on astro-orbiter via the
+        llm-inference-multimodel role (--tags models only). Idempotent: skips
+        files already present at the correct byte size. Notifies the
+        llama-server-router restart handler ONLY when a new GGUF is actually
+        downloaded. Does NOT touch Phase 4 (verify) or the legacy
+        llama-server-qwen service. Safe to run repeatedly.
+      app: ansible
+      playbook: ansible/playbooks/day1_deploy_llm_inference_multimodel.yml
+      inventory: production
+      repository: homelab
+      environment: default
+      vault_password: ansible-vault-pass
+      arguments: '["--tags","models","--diff"]'
+      # Scoped to --tags models:
+      #   Phase 0 (discover) -- skipped (no tag)
+      #   Phase 1 (models)   -- RUN (idempotent GGUF staging via stage_model.yml)
+      #   Phase 2 (systemd)  -- skipped
+      #   Phase 3 (firewall) -- skipped
+      #   Phase 4 (verify)   -- SKIPPED (collision risk: verify.yml would start
+      #                         llama-server-qwen on :8002, conflicting with the
+      #                         production llama-server-router.service. Excluded
+      #                         here deliberately. See t_730f9584.)
+
+    - name: "llm_router_update_unit"
+      description: >-
+        Re-render and reload the llama-server-router systemd unit on astro-orbiter,
+        then restart the live service so new args (e.g. --models-max) take effect.
+        Drives playbooks/day2_bump_router_models_max.yml. Added 2026-08-12 (t_33acbb2e):
+        bump --models-max 1 -> 4 with full VRAM budget note in host_vars.
+      app: ansible
+      playbook: ansible/playbooks/day2_bump_router_models_max.yml
+      inventory: production
+      repository: homelab
+      environment: default
+      vault_password: ansible-vault-pass
+      arguments: '["--diff"]'

ansible/group_vars/all/step_ca.yml

@@ -0,0 +1,4 @@
+---
+step_ca_url: "https://turnstile.local.mk-labs.cloud:9000"
+step_ca_fingerprint: "f63c44e76381e359978bd2dca07c928d04ad44b575f0364fa66b5725c7e7891b"
+step_ca_provisioner_name: "admin"

ansible/group_vars/all/vars

@@ -0,0 +1,47 @@
+---
+# file: group_vars/all
+
+# Proxmox variables
+proxmox_user: "root@pam"
+proxmox_password: "{{ vault_proxmox_root_password }}"
+proxmox_host: "main-street-usa.local.mk-labs.cloud"
+
+# DNS variables
+dns_server: "monorail" # .local.mk-labs.cloud"
+#dns_admin: "admin"
+base_domain: "local.mk-labs.cloud"
+# DHCP server
+#dhcp_server: "matchbox"
+
+# Terraform variables
+terraform_server: "infra01"
+
+# Traefik variables
+traefik_server: "lightning-lane"
+
+# ---------------------------------------------------------------------------
+# JARVIS automation account
+# ---------------------------------------------------------------------------
+# Public key for the 'jarvis' user provisioned by the linux-baseline role on
+# every host. Public keys are not secret; the matching private key lives on
+# the JARVIS command centre (carousel-of-progress) and, when needed, in
+# group_vars/all/vault as vault_jarvis_ssh_private_key.
+jarvis_ssh_public_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID5sym5ajFvDyzw395BkHv7qVb66XPTx/OF1p19MGuNo jarvis@mk-labs"
+
+step_ca_principal_mappings:
+  - local_user: wed
+    principals:
+      - ryan.blundon@protonmail.com
+      - ryan.blundon
+      - ryanblundon
+  - local_user: rblundon
+    principals:
+      - ryan.blundon@protonmail.com
+      - ryan.blundon
+      - ryanblundon
+
+# Leviton My Leviton API
+leviton_email: "{{ vault_leviton_email }}"
+leviton_password: "{{ vault_leviton_password }}"
+
+jmri_vnc_password: "{{ vault_jmri_vnc_password }}"

ansible/group_vars/all/vault

@@ -0,0 +1,418 @@
+$ANSIBLE_VAULT;1.1;AES256
+39633233363564393465633065323730343130653362373436356334323930393036656638303033
+6333636536643835303936633039633138336434663565320a643961663732373630333238633466
+62633236666331326165653637353038313163643162323763313635306230333338613164316337
+6133396638336465350a306163643138653639646133346561323333313332336639393764343364
+30666535323264363661653236333839643631376337303563383739656238343732396364366437
+33333437643930656564376236333736336465356134653239393036313830333462363633653366
+62623863613261323135633262633066383363313731646361326232366661356261316633323164
+34653661383366643531353939303333363832323731623830393262373538613631613631616463
+66396163613965343833356432643534393539643961373263323533316664386665383330326665
+35393537363734333961393931386161386234323264316466303361663139323431303034393035
+33353432363162353261646231313136643533363563343537666565306638353564646235623835
+33653663393838303364363762613663646139326135313438663838323064326265303733346164
+62386130613134653739353930623330663135633632346531306535643332303939356263636232
+34313939333263313133373635396138643838396235626662343664316632663930313038643863
+65653630373437373631633032333663353866636630306235363035623837656661323735356361
+33343961373235613262613363636565396234613865303937323432656433343836373831343162
+37356234363662333764656530333139653434336137326631386366306162646433333239316133
+35303136633664336433386331316662313330643734383139633938636462346539623835653738
+63373439623866363863633963633265373339613130343966656637376632396139646266306666
+62316430343338626331326433366266396131666264633234323233646639643236323937396431
+62323738376664303237653563323164396166313266613937383431653766616234653762383163
+65666138353262663536623036623439363463306136363930343633383436333565363232376162
+35343035306230666432666131643634383263663838626466633133653830623330656164313465
+32363031313266396532313563343963613133396232353832636464663462636138623637646135
+64356364363862653465303837663261333933623266333736343938643065393031343535633762
+62363436323931373563323863643066653237393339653363656133316365346262306463336364
+66303732366437633461373065363730336235313732623131633366633834376563663734653063
+31656638623933376666393232613238323239623836313430356639613131303761346535336338
+65613363366331613865343163623863636666303336646636663465373065393731306634663239
+32653666343631666235383863336338326438646137656562646431623436323037313438343839
+34663062396564306138356465623032396239396562643933306365613033333065323630393665
+66346262303965353539663463626261633332373833346430346235376636323633623864353133
+34666265373936376339646363383834393166376434653233306632373966336232316339336465
+63313866653732386234666531663030313364626534343235333463376334663061316461613331
+64303962393930373036643561393762666334666538626233333937653038633734626430623837
+33303463386363386562333061353963303538366362313131656433383263666631666566633765
+65613238393736646637346333323933326664663862326563363566613361343539646264626561
+37323034323136376433343433316266386135333765643036373565326539643066336533343431
+34316264393261373130333930383932343864386534376633336239303066356666363438356632
+63313637616465386363346162653064613236383166353432636530336437376665643830356134
+35366239326637373366383962376165643932336632373330326533366535666434343161373264
+30323130626236356666663230613837373666666334646366616339313661613238666138353832
+37633066303139313961643935383632646662613766303366663335306263646439383534373639
+33643733626531376565623431666637323736393361373566626666633232383832656333333566
+65616662383561626161363061646334633033386534633533393032386532643939373736356532
+31633430613338343762656630303139316634323635653763386366633366366336383064613333
+32373164633066643064316535333364366461666332663165346462386563373864663565613736
+64636131346133623166656235306230353733373263313166383761303364366434656666303732
+34313234663139616466633562383663316630666261336463653236393832303363316438326635
+32343135396333613936383239633938393261393435653239643763373065623531336633313130
+64323761646666636464623734656236653439646433303934633536333862653930303133336264
+66333361323735386338643933336636303263613739666432626137393061343833636537396539
+62626534663636323764376437393237663839356534306438306431663937323364316431323264
+38353234303364653833393733383736636139336237386137313937656137636562303036666135
+62626530383037613439646261346130353430303131303936393230333662386533356631326533
+65643931313036623766316137353130363535383834323363623334376234393265326362653838
+34353665666537343232626137383166383761643232313539346461393937353737373935323462
+38303631383465623865323230333330383730333536346366616332343539313237333538333431
+65636235343938316635633732393430323932626339323236323663663932306463373039343363
+66363235366539386430333365613338646166313133346239346537313666396435343064643537
+30363236396563346165353032373463386630646134356634646465353136643635373164643263
+31396333353232323937346530303334623163316538353937396337636165356163396639373731
+64363036346434623366376261316632343238306564646234383433343137396462303466383635
+38353161383933336264393332386233643666656132633038323936353630313635313365353663
+30353734363435373265616333653737383838376463646466633732333537346565666433656137
+62333234653264626532323064386639643136336264346232663537393065386666623463336433
+38613966376535303961343361363930313664633030353538306633656335326430653534346161
+31613166366231386239366631623839316361303630663063336637303832373463613735356364
+39323238656232383632643630386261323764643861353063616130323939383732343863356633
+33303965363565313535386463613030373634653763313430643335393032663839636164303630
+38653930383661386331613763316365653538386266396430633638633638653134633365343937
+30313565663139366139623065623730356363313731663034663362313335346561323130396333
+37343165333434613235363964323564306330393738333635303763633466366664616138613862
+61623464656630343361353433313836316230653435356661326366366131386637363334326336
+34313334386464363261343831356163613834353639323562333930363563343963633833623832
+37323964336565383062373739346261306139376637306461336265316336666439353536313063
+31653432373235666235396239363261316333373332333234376635323261306234383836383161
+33303136363335353539663833306166643737333238643066313633323031613237666264666365
+63333766616631366639396235633466383062623462363638363866346462306535623839626231
+63666633393137316637616634653139316439626432366236663738316166643335333534626164
+62623861356134663637663665313934313833636165643732376438316563383539663037373432
+30326233373936346433323934663633363166376463623135386238393334396466316265613965
+33623362653436613835386535393038363064373233623266643466633732366632346330343438
+65383938363063666263616330356661343732313435383137313435656337353063613230656462
+63383465343932303832313335346230313164333362303935343836363764313933623831306130
+36333339373230386162323337386661303663616530383863333332396439313833303461643662
+37336231343065623635376564393934316333626566623765353265613737646131653934343631
+32613463636130333561653066343839303332396239656534303733643639343337363437656662
+63303732326235643830303038663131613330303863353231313561376231666135353464303864
+66633332613139386136303264366464356434633434656536393664313130646234393634326432
+63663938373932386564363834373134356464633662383130386337633564643930613734383631
+38643239663733393336643834643438303737636635396630333565656264653866366166623332
+35653430323365313834643862393065323531633230613630343738333934383161613264356265
+66626561386138396639353934653465373162323838303233636565306534633236343066623638
+38653132376663666136383235663337343762646137643935626365343162363439303061393563
+32633366623834613739303139333038303762316663663136656636663663386362613264663238
+35323038326239643062666336333932383065383461373933626339383434363736323536313533
+65363565306239363935666234653438393039303663623636376262666164393334363839393665
+62326533613639643938613838396565376334333035333962626135356265353639653633373438
+37316463303838633335306333346465386461373766356533313863333534326231353864303531
+32663461623731653138646336663266303065306562643961393936373937653664323032626339
+39373539363664316432333931633866646333346363336466393963656637366562336162613465
+32663230613335393839633633633063636635373363653234663564663264353036393434663963
+66653631663062666338326133636236303862356662666533326236613965323462303839623331
+35626238323837343738356264313266383135626239363731353639633837323132623635633435
+61386663366533326130626332613562633965633837346237633334303564363136633733323361
+32353361396166396334386630353738643438643537303034623830303939326436363239343664
+35633731396161623934633864313863323336326535313637653935376436363162363565666634
+33613435326539343865653862613263383236313130383234623766333637646464613365346532
+34653035393862643039623763323138333334623866316437613435666530396134306665316334
+31373364653865323765646535303166373638636134326538343563363732636337653931373234
+35303433656137333635383334376237616261393334623835353866323333393630373765613835
+33363636323664383136353863666635646563393764633531616135313935333539333664343664
+62633465356563353433633564303531633236656631326331323638643666643636396162316532
+36313839363164663336326639313066316331303365386130666662366638353834306662303336
+34393538346338376330333332383231333630353334646635313037313736613139383832663566
+30653137386339623136356663613431303564633966383633643830393636383734306238333831
+64363933386532346264666638316161393435653661356162306262323131643535636134396439
+64656538313938623938323638393734343464653362636532386663303762326234373930363866
+32306235363837396339383461396430373964396533616434373937386266396439303964356534
+65663262343461343864633934636536366335326162393830633232633832383166366430313934
+64643262313163323531326163323431386532386530326332326130353230653263616139303566
+64386535326262326463663734303061613435653964303764386537636132646430626530313331
+38323563303738353630653631333535376239393730633535353934316131333039353931616439
+35343531353332303932353636643763643035376161643536333365666466616234326237633430
+62666464613334633537306431396534306237313032306632323266613134363231313331313139
+63316661303830653035376261633539386533373235653839333562656135636433333263303030
+65333136653939616231363835616134373038633634656366363266386534343233393730386436
+61366536333730646562343333313234663037616461643864613334313464326661366133613966
+32343831376266376636646439323566373033643939663261356432623865313833316238363363
+34353765396233336263373161323433396437363463626464386463346330653861656162346639
+32653738353738396630383431346535616165306638666137616664336235623834616239613131
+33643866396530643062633537376565306538313437623462363963363364303738376662323237
+37313438633138666434646633333262643333633930396439333565663930613263363638366265
+61376165353939633165646330656631323765313430323630303335643362393763663134333333
+64383338343939616362363930626235303532666538653561656162396566383865363463373233
+62363363623938383234343639656631343831616564633136363332393131356531343137396534
+65626639613663323139393061353636313036396163646434613431323864303130663531343964
+35363339336436336532303763333564363962313637653438626235353339333438376433396565
+32646536383036353864356134623564653436326232643330613736386162656161336165363463
+37646138323937343839373736656332643430386630383463393637643463396231303730656530
+61633563633338663635353235663931303964333839663333323062646266646466356434343432
+38303762336132333132373532623637353731306161316564373935303366666530376639613166
+66383533663634333162373436313031643533336436373563356364323236333632303663393761
+32343631643566626433306532343764373263666537346538303561323465646237396639363065
+64646334633033653962303437396430646332306562623763323733303930623763393537646532
+38336439636134666264663339346230326435383365616433363535613235626439623037666234
+33393434306662333966316364303261643137356136326637346539363538353465313662393534
+34306132613636633936616339636632623733616432323564393565336633366464306661306130
+32373465656131346138343537373463663937636337303765366337376666346136643139356234
+37376436313834366538636136303965383330663437343363633861396634373764373863613734
+39363837613363663337323137616139633935643437353838383735383561346365333565316336
+37373231343665343838663235656132613331663638323331363763373539323239306232656338
+33643632333863333064666332663437613961393530616330643139663736313733306339346535
+31386639363338656462663566313836313066623661653139306662636331373038396636386362
+63623431336139316233396432396166633762363662616432616634363639616530663931366230
+37396434343039303232363633626532383235376538626438356639386661373330663838663134
+65666437363139663932383035303036343633303766343836363435313739363839373938643562
+33393733303165373364643961326634396265323962313433653634326436336564313338323631
+63666636646638626639323631323335653961363964616461393632316363623162653665333532
+34346532313436356235343539343330656235376261326262646364393866343562323663363232
+32636539343063323734663732303430306261366361383461363963346633316632623531363530
+34343633666136323238336530653335643730326439666637663266363563636466306531393431
+33306662653537643835656637636466383965356663373530386431383837373438373436383461
+66653735376535656637626231613437336663333132386135666432656638636138383939386665
+35346636313439613335653635653861303231663239613563333438303738356439343933326332
+35653764613061356465666361323134323331326137353863666364333332326133343039353164
+35663264326432366330353230643762623933346363623039656137356135613063366265626334
+64666539373230376335623365373866613761373637393066343536386134633131393339343439
+39353938643233636562353131336262616439623461363531383064616564616462383833363338
+31396436303330336532333661333263346233626532653437633766306230643032633035636132
+65343938643630303063336664393532363131333532323566306338343837323431373062393265
+31653964376339626632313734366665613563383266376533353664363764366639393563313164
+63646661373461323039636338313165306435323530633535353331343166636565373630333530
+63343262656538616232313633616166623534643834313138326634356334636635363935343030
+61373533373863633630616135626335376136336261646465666135383130633032656465623938
+32306433303937363133623537313535633164346536626465366663623430383639386662396665
+65393038383136306331323035373432666565373934653763336336323131353732356164366262
+64396137353838303934363033393363666632323137663762626163303963386461373166323734
+30336135396262323736313037653133623237336532663765323034383766363932666562343335
+36623731326232653363663937626136386139663630376564333230313632646334353664303137
+62363261316161356164313433666562303965646362626331323538663865386530393864373932
+63326161656561393665396461313938316430353131643035353933326430653439643634316435
+61626437376565363365303061346536386134356335646137626162663739656631343334653866
+31323564663165376266656233316266363232333336356137353665613961303065373735353735
+30663339343631633931376562316262356462656132666661383064333838323061616361383264
+64666664616231363661636337323839613830363366373530313963613132643836353062646561
+32396565313631623639666433663037336338313033373938653336366539653166376237326536
+35323134313962613962663630376663333334373431383731383038323965333736356462303566
+32633164663734366262383264376539313438343530303364623566383564376361343231386564
+39316366373964333837656536393961613466353436333839396331326463356636623738613563
+66333030376364383535633637383439383237636539623930623966356461366436323933646266
+63363264643238333432346432363339323461646533333266666630663161653031626361366364
+62613433333235356438326439306132376234326437316437373537346363303537626632373537
+33346534306363323133353163663932633632626561313566383865393461316163616532666463
+36333966626638336437666338356437643862303332353663303537613933643338396432663336
+65613565353536396132373663663730386664323732616533303037363362663335656236383838
+32666463323463363736316534393538373938303062396564333435383834646664346261383535
+65653931623934363330366361303030306132616637636366313333303664656363623061346638
+38626261313765303832333562383763363935656632343735613935356333346237313436326562
+65353230353537373737326239636465656566346635303134363739333139666362333964393465
+35663837633132663461383036353065656334303730636661363236646462353238343737633635
+38663762656234376233623264343762303434353366363262353232386131303231373832336431
+35636434346531376461393330363837633564613363373235623837373130336465383933353039
+31363732656164333037316438393532336435663035346139373439326565653363343336306638
+30383635656336643934623132383433333962323438303733396131333761643730343464633539
+32306664393738353334333465363966613261333532393130366437353433643439316135613861
+63303863313133333339323665343539343930356637356262316138613733613465343031376538
+30633537396262636262663033656430313831616635363239646334356334323731626530303034
+61343362383935343835316561333231323261643838616435313032313636613933363837343336
+34613131316239653861626434366338343232326531396530373165663931396136373536366339
+64306363393061333333663435666130366630633838333533306163313237653462306461303030
+61353830326337333563356165663062316165373939333938373637636164646536616639366634
+65353035353836356264346234653431383565373637383035323234316637356530346332643635
+37326161333838643166313664616331623532356235323564356261663734613464666133313230
+35393763353639323666313430666164323836633138373961666162316433396364393563626265
+64373565393965313134613365653062323764346432653630643662663930353738353038643661
+34376363633338353132363061363964346162663438626638373032326536363733396634303461
+39306432646362666336393437396330643864353437376235313966393837393435373437313065
+33303439343736626436613663353739656264313639626633323265643966643566623765653739
+31336564393138356265623636306266363861323961636131336331653632626439613461383735
+61616265646365636133393031333733363537366262326632363962333433656366313965303164
+66663335636363343739313838633235323561343833346332353532663065363439303936303266
+34356331633961383264393765316236376130323233346163306536386436366636636139363035
+34623938366266343164633532646161633264316334313930383565623766396165663664393537
+32373266656134333136393965366538316366313735653535626635663462623036323832373964
+61343236616264373563386265656364333230396462653833373338356233393962663965346636
+39313261353263373364373034653338383632643430323335306632353331633939383137396164
+65313166633632366465663466633733663939303039316265313539303564613038313136623861
+37383837653262633365333063646331663562356162333937383538623662633134613364653033
+32613537633639656361336236323039383533303661366432376665323432386464393534633066
+32616666613938373066313866306330393061396263386563306661643831303266633465396364
+30613961636333373462386531373264386332303063663264393861363263666638643836396234
+66613235643365316233353262663739313831373561633339396165386530313538653538633039
+36353364656136306161636135663464343033306161653365653166643762343035336166316164
+66346363646238636438353036353631663135356538663762323030353965386635306630643131
+36666330663063626334386431663635646561643464616636633735626132383961663265626631
+61653365373863313832366538363864333537643336623633343334633333323561373638613961
+34396433366665333531373031336362306364633537646565656630323334643038383535383036
+36633431633930313065663337366533323463333162326435626264393162333162656236306536
+62323032616162313666303265636636323163303234656230313061363862343736373630626435
+37616662393737656132326537336534623633613133343833313135336330303536393364336363
+34383136376363306264306237653236326166323032383939616563353432323565666263646164
+38346663373335636131366236313134623435633766653266383661363532353931323831363065
+34663732643435623436323236316365313861346162613636323138303831353062616263626436
+37356230643833323062336465306366613637633834303762323838656132333130646462653766
+32393532636463363634636665643439366531303162306133626138653839316433343532646530
+36383066303539386136643436393764613937313933396335376262323064663363396133373739
+66643332333135333437623266626563356337366665336632333035323537343762376132373331
+66303630353430313961356234616438363436323938656466636264636364313431396165623530
+33306334656338623237313333386135313961633666363031376334653433366330303262363832
+32333139643434623031653137393134343961353261366464383537643237623166396431356430
+64356435356566336461663766393537303737393139653132363737343238366361316463306136
+38313365333863303734336162666130393531356531623865356431653237343432613238633566
+35323166623935336361653530373466653063383339396466356334663066393539356432616362
+65643536316137326533363930333966666632393334333536326431633962353362313532383033
+63376566643166353136353733353862643033363035353534643062303133633064366332623938
+33363862306263663737616364633462363736666530613562643865393466656131663663663166
+33313839623739386565323133323263613939623838396365313265363466666235316263313535
+64393936396332376635343362396564346634336634336663383566376365353836383739333665
+30653038346161623536633361363961623931326265666566316565636662633032663639373361
+62653665366338656532373734663138353238356535353036336331383635306236656130613633
+34323962316636333664616664313163643061623836316432366664623164303365613161653466
+39333538343435383066343336313564386136363363326231353838373437643136616336393561
+62626430393735326366353839393166313030376435303039643739373337653635316466643236
+64646237303064306663396439316431393931626431616630396538626639636432326561653562
+61363338623863363731356566306231366336363739623735313134613933373238303765386234
+61363034313139626636643533643738663266343235333465663632343533646131393130666632
+31366562643166616430613030333832303636616231646239646435616530343032376663653031
+65636561353734656562626463643361633362313861373233346633636438653335323761373165
+31613130306437346630313466393735643434376133336565656566323132653962636630313431
+30386136643462306261383135336362356161666435323132626637346365396436653835363733
+33393732633464626437366439303931623962313433626533363136653939316633326432356565
+62393332656238303763363732623163653663373536393265343763373365316337323564353064
+31616634653837396438303938343562613761363766616561643032643065346238383734376433
+34613931366331366138346636376566646539303430353665626263643465313133323939613361
+30613063656665666439646362656136656439643934623730383736353031363633653036643161
+39366163663035663135343462336564383862336531656138313431316263653064306334633062
+39386538616361623265663536636130383362633934313764373538643636353237323133303266
+65613765313838363236656334663635636264356331666134643063636334633131353734623632
+38316637306163363839366536333365663337383632376230373137386535303666343362316663
+62663333313634313361373463313566363063363665323565633935303738316135393264663935
+63643662333430633535663130343930373763366430306337656162383537376564653439343430
+35656430366365636165386333663566313034646564636261333361373030663633373861323065
+33363463303135336436613563633135353531623339616530353538356130656263383265353735
+66383964313665623133303130323335393339663331323431363438613637636665623634313862
+66336533626266343831363732613331653431306462313830343830336339663636396566343134
+34396137396461376433653962613361323334393766303330633237623963653033363532613761
+37316139336234366633343064313965393965353461383433366630633935336233383962616662
+30356234663661343232633435346532313737376234623336366161303937623964656331633935
+62333537303536333062613132646237373031663766623730666330666364313634366362363437
+34643634303366633236363665643934323864303732613461326238323637393733393964663562
+30386664636139313335333238373632353936646432326436666665326365646565383766316362
+62376463626338643937663163336166663465353936626332303739393762313235323535366636
+63666438636230383031333838613333633734623134363833383035623564613637633562323263
+64356662376532333261316537656234636338393738613730656530613232373662363431393630
+32323230346332613038373530643563353162366366636338623566373065373637623638333533
+34633333663832393438616266653563623338626431636634303234333636613033333831316264
+36333664366262343261373163373763373036653866663539343337323761633431313435323465
+35386539636338613666393064303363303263323565613837623866633031303834363837626531
+31363838326235643163353731333938386264323730323036333066326463326537613530373832
+65303663613361336565643366316238336135396334303432393936336635306434653930643631
+65393438643630313330383061643031343663393265646664356636313035313961383634316466
+36633139373564623466386264303664373763386534326234323337633732633232346136366133
+61363939336531633265396136623031396534363031393164646661383339653232393463636439
+63306438613837313136646232333434323430643130313937376230613531383461306262366636
+61633137643164653739343663393335373065386339393531343930373161376434356666306132
+35383763613966353662333362396265326532303238656133316332363335346437306164653230
+38333263323064653533373365336166613431303231373062376663343261663135623937373163
+63613239343034303263306138376430613335643862303633343637373430336131623037373537
+64323838306233396435656339306132623266363961643166303733373537333135303665336239
+62623439623865363736343631613432633264363733353834386335333330313933383433346137
+32386432393765356133353736306430376665626665663964313037323136326164653435383332
+61346361313063333137356662383533623734633435646638363537303037393638316439316561
+62376361363532623266383635323330633033333536636262386239353732633237663139393839
+33633537353136376163613334393430663366636533616265633133373563653962393238373862
+61376639663631323231376631353964643162616538353835363937663661313632333333373933
+62323539643630303062333063386163656530623061613664376436623732386666336162396533
+36343563643032356261363234663264303762663439363664353966313561623761336138313165
+37633630663131313565346539396234363565646530386266666232306262663961663636326536
+64653638643935636365313561313964653764363766623433353365376161623333356164346232
+36646630666333383234343931343161646463633134316466616436336634336661356161303764
+62646362333139373761333035386666663561643837623665656630393432303839666666303339
+31646461613838353033343665396239616538356164333436353133653134313738343530313366
+33646530373561633834383865393961353032656339633732646533363535343630613937323461
+65363262643937373638633838343266323966623039616333656538363362333962353066323363
+36326138323132333837343130363735623861616334343130363064303030306361336161303832
+32343534623662393566653131363866343637373930643434326338333362646233383231636235
+33636433346464363766376634636533366163333064313534313831636238376639393533616566
+66333633613261336361366234643966663536653736323235663030626234386166663033376234
+32623435383837336330616235313165636537623462626230346535663662386337366532363733
+39306631626262623630393137653763326234366139626461643931613736653932316466396163
+66643038343336326533366438356562333561616138383330643235663633336262303266613630
+61346237313730303666346639366439313865663430643536336338313130656438613831383633
+30613862363961373732333863636134623431626361396562393437613533383738613761363835
+30363831363963623133653038383031323364346237326164313232363936656132353732373463
+39383737303034346163623737353537646433366539663963313566363032343836383134636131
+66363664336435376563343765623737366632383431656166303639366661633565623331613530
+39396131613161663436333736666536656364663361626633333637316130646266346165323632
+35623062353732616535373166336564383166663861613239373836646637616333613639613862
+66396434333039393135623733653938336233613235363134636362666363303337383335303136
+66313061323938643432623833363930303236633462363663323362626332363565656135363464
+63376266653132333565616363383564353135656431393062393333626336663738366331383737
+30626239366666366263353662333166303063386364366530653731353135313761343461646238
+39353531656232616530633330393130663862623964363565326233643431343432313039333234
+63336232336535373032383933393763623538353061396534373632323165666134333465303066
+39333366303231363863333133643062333230316361623635656663323737643965663563376564
+34343134306333386663356339616461646639303835333333623363383364666436643436343533
+38373364363765343265663138616439316633623865343939353465353466613065653366336230
+39636664303335316538326136356361633366373761333635613139633665616433386162623965
+32636266303031336162626132363931363163313235363865353334616137643934353637643633
+33626137376132353735303737343433323462343830633731396565383037313232313862396330
+65383564386564353634636430393562623833373533626632333530336266633038663333663063
+32313534666162666134373533313730663730333933613762303238323065363236363166326361
+39633938373861366133316261303665356661393830366131343138623138616233623961386233
+65356634333833313061313134333364306238623637333733666366373762383732663037643338
+33333263356262633366346130386337396263646333636134333935343835386366306663663962
+62306232653735656365643533653635366232326638386563393365376464653532656136656266
+62393830656339393033343638643334313765383766336338363331386135373864383939643166
+66316437666434643266326235663632363161663338336434313134643533383634633936336665
+62653638346633336331333364623363376538373562663437373933323932663533333733633036
+39386333653636646133366461633237656166396664623136633863653339633463346135393566
+39316161613532386432306364333662313261346631636430363061376332633431356132663066
+65643137303961383539663733653863346131396232346263393934326239383735353565376533
+33356438336463323531326561326264396635623734663835363039613162613762393662363562
+32646634643934393263616335633933386564373162303766376661323864306230376533633539
+38363032353639633735356266616330396333373665373035643334336566636534383831323862
+33623866386539376131393531653562643430663765316539613438663733366566366533393761
+31303836636533656535383165323232353065663338646337323530363133393263653930666334
+62323836353563366637383138383864323438353339653434306532336164306238633437646235
+64366536393039323339616130663939393562363935633435373932396462323533363838356663
+34343138353130383734363234366661386638653834643661353031353031343562613461393966
+31386463366635353965386462653461653866393864613063636564313638623734663864333334
+36386239336431316338643761363566636137333161663833623633376262623830383961613931
+33666136653631663131626631643266656430356364343138353663326435656663373035333934
+33636339386639666532333763393735386231376666323733333030353239373333383264323130
+35303437616433393231653739653230326263616233373166316237346138336364353339653339
+33353132636262323030643465313231393763613861343734363031386563646337336562336463
+37303232663235313335336435333032626432633531306638313561646236616230306135313639
+63633938326266656138383635653339393238303964386663366638643734633866333432373737
+34636433613566366132613864643135356136336162643066633035663533666237376437333762
+64376262633064333635623438353732376466366338323636643662323234326139316534646533
+33666432303636373963666334376263636634323532643837316361376338616430346464656465
+33373336653539303163336239613038386230363839353532336335303564306363666665333530
+63393863363232653138356234323161323931353932326137653430363836323863393035356230
+61633161623761303335646435383934663135633734346433643639383932643662393264626430
+38303735343033303765316365353834633461386636653662393538643339346230643335656438
+66386164326535646335356661383735366134613939636130383034633734313739333635376233
+35623130353239323237386337623962616536643463376632613631646436353630366137343838
+30376136616261303235643437613864613031346564363465646533363033303566613061363330
+63306530323938383737313438333139363862366331336665623139626135353236316664643661
+38343165383537643136623661663730383635376263346131303332646464376263303464373734
+62366331316166633631663163306237343862306365326339653363336638633830646233363665
+61393461326535653736363834346163343736666665623838643937306539376266393761633263
+39316163356262633137313261353463373166633735663035663135326437383534616239343661
+64633762613833653365353234363539643533353239316238636537323064353337316361353630
+39343463643236656431623861333734313838346430356630656131383763386534373635666331
+34333137346266353265376431623935336366623461666237666334616634613162623761346161
+35366334313837333939306563316236313839623236373234623635656138656165313662656632
+31343961363864346438333932656264363932646132366136663339363930386435666639663135
+61313037303762626537393330366131616532646666613338393032353837643835353436383539
+30323064656161636534653664336232613565626162383035636131393539386633643436653633
+37343339663531396533323432643464393431623338343537646138653236366331366261663631
+65386666313531323031353762363437653561653534316463666539636439383361373861646331
+66626434326566653534393964643938393463303930613734336161356431363230363261383139
+38356663323539636533663237306566623062333036666636663637306332626139333738386365
+38363635386439646363626631366430393636613763646539303939643764643966353461343466
+39363331343334653637656561383638626637646230346665393366313366633366653839663839
+39333263313566626135326633653238626566633536353461376462613462303830353638393161
+36366433656261313436663466366433663138613561386336383262646663313836633564353266
+61623139323961346564623939303339366233643231373031626165646364376138383935626166
+38373833613033633630313965336438626363353031346630376532393932353961363732623631
+35613038366664663337653230653134373861666430343666623165306130306137383163396235
+66633035353534386262663336646361303236623365613736383938333261653462386636376531
+37616539376637386462373063316535636365626634613462623232333435313733326261373230
+66373061336231663230386133613665396263646563376534313830316439623363653165613666
+62373063633863656566643333636461353339396635396331633734623266306430

ansible/group_vars/multica/vars.yml

@@ -0,0 +1,12 @@
+---
+# Single-origin layout: one hostname, Caddy routes /health, /ws, /api/daemon/ws to the backend
+multica_layout: single
+multica_domain: multica.wdw-labs.cloud
+
+# TLS: acme_http | cloudflare_dns | internal
+caddy_tls_mode: cloudflare_dns
+caddy_acme_email: "ryan.blundon@protonmail.com"
+
+# Pin for reproducible deploys (git tag + matching image tag)
+# multica_version: v0.4.5
+# multica_image_tag: v0.4.5

ansible/group_vars/multica/vault.yml.example

@@ -0,0 +1,8 @@
+---
+# Copy to vault.yml and encrypt:  ansible-vault encrypt group_vars/multica/vault.yml
+# Generate secrets with:  openssl rand -hex 32   (postgres: openssl rand -hex 16)
+vault_multica_jwt_secret: "CHANGEME"
+vault_multica_postgres_password: "CHANGEME"
+vault_multica_vcs_secret_key: "CHANGEME"
+vault_cloudflare_api_token: "CHANGEME"          # only for caddy_tls_mode: cloudflare_dns
+# vault_multica_resend_api_key: ""              # optional: email verification codes

ansible/group_vars/ntp_servers/vars

@@ -0,0 +1,18 @@
+---
+# file: group_vars/ntp_servers/vars
+#Ansible vars template for NTP servers.
+
+ntp_servers:
+  - 0.us.pool.ntp.org
+  - 1.us.pool.ntp.org
+
+allowed_networks:
+  - "192.168.1.0/24"
+  - "192.168.2.0/24"
+  - "192.168.3.0/24"
+  - "192.168.5.0/24"
+  - "192.168.9.0/24"
+  - "192.168.10.0/24"
+  - "192.168.250.0/24"
+  - "10.1.71.0/24"
+  - "10.1.82.0/24"

ansible/group_vars/proxmox/oidc.yml

@@ -0,0 +1,19 @@
+---
+# ansible/group_vars/proxmox/oidc.yml
+#
+# Proxmox OIDC configuration for Authentik integration.
+# Client credentials come from vault.yml.
+
+proxmox_oidc_realm_name: "authentik"
+proxmox_oidc_issuer_url: "https://authentik.local.mk-labs.cloud/application/o/proxmox/"
+proxmox_oidc_username_claim: "username"
+proxmox_oidc_scopes: "openid email profile"
+proxmox_oidc_autocreate: true
+proxmox_oidc_default_realm: false
+proxmox_oidc_comment: "Authentik SSO"
+
+# ACL entries - grant your Authentik user admin access
+proxmox_oidc_acl_entries:
+  - path: "/"
+    user: "rblundon@authentik"
+    role: "Administrator"

ansible/group_vars/proxmox/vault

@@ -0,0 +1,17 @@
+$ANSIBLE_VAULT;1.1;AES256
+64613361376364346139313833613465663361336634326430393261366630306466363935613139
+3634363235343736643230623865386436333734663531310a343965396534336262356234623966
+39353332626662636666383935383530613139626439373664323063633063316264383331646533
+3366333837613731310a656462633638376432326365343135373863313665346566383933353630
+37303565323630633739396433323165326262363863386233343134636236646366633765616432
+33633165613239653762343935386637393437386431383961306436373765343736313634333933
+30376463646464303066613561613564353938306664373464333966383664383034616439343735
+38363966323366343465393530383736393364373361326234356365363163356632393834393464
+39393530356235383830323065636535383138353864373237333164323436623737383435313261
+64366362626232386438376131633533633162356432613835656465623762633464353936303236
+63353633633262633461353963653534663336396134373435303266386162393965343165303335
+65313436666334383965373837376664323861376162373834363439653539383231346634633737
+30626431373739633930313937643938343165383536373465613535366562313439396533643864
+36323032316131333962373732306334373939333363386564653465303261393337333466316130
+32326533353165393635393130396164636363623937313566343333386562616662383038613565
+35386638383365633362

ansible/host_vars/arcade/vars

@@ -0,0 +1,59 @@
+---
+# file: host_vars/vm_template/vars
+# Ansible vars template for hosts created via cloning.
+
+# Supported hypervisors:
+# - Proxmox
+
+platform: "proxmox"
+
+# This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
+
+proxmox_clone_node: "fantasyland"
+
+# Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
+# Current OS offerings are:
+# - Ubuntu 
+#    - 24.04
+# - Fedora
+#    - 42
+
+vm_os_distribution: "ubuntu"
+vm_os_version: "24.04"
+
+# Current VM sizes are:
+# - Small: 2 cores, 2GB memory, 8 GiB virtual disk
+# - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
+# - Large: 4 cores, 4GB memory, 32 GiB virtual disk
+# - Large Plus: 4 cores, 4GB memory, 48 GiB virtual disk
+# - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
+# - Xlarge Plus: 4 cores, 8GB memory, 128 GiB virtual disk
+
+vm_size: "large"
+
+# Proxmox storage target.
+
+vm_storage: "liberty-tree"
+
+# Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
+# and backup groups. (ha_group will be factored out in the next functionality update.)
+
+ha_group: "pve03"
+proxmox_host_target: "pve03"
+
+# Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
+# sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
+
+ip_address: 10.1.71.111
+# vm_mac_address: 'BC:24:11:11:BC:58'
+
+# Software
+# Future enhancement will allow specification of additional software to automatically deploy to
+# the VM after creation.
+
+#terraform_version: "1.11.3"
+
+# ---
+
+vm_clone_source: "{{ vm_os_distribution }}-{{ vm_os_version }}-{{ vm_size }}"
+hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn

ansible/host_vars/astro-orbiter/vars.yml

@@ -0,0 +1,235 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: ansible/host_vars/astro_orbiter/vars.yml
+# HOST: astro-orbiter (10.1.71.130)
+# ROLE: llama.cpp LLM inference host — Ryzen 7 5800XT / RTX 3090 (ATX rebuild,
+#       2026-08-04). Superseded the prior AMD RX 5700 / Ollama config below;
+#       drive was transplanted into new hardware, not reinstalled.
+# ------------------------------------------------------------------------------
+
+ansible_host: 10.1.71.130
+ansible_user: jarvis
+ansible_ssh_private_key_file: ~/.ssh/id_jarvis
+ansible_become: true
+
+# LVM root expansion — xlarge template uses sda3 partition, standard VG/LV names
+common_expand_root_lvm: true
+common_root_pv: /dev/sda3
+common_root_vg: ubuntu-vg
+common_root_lv: ubuntu-lv
+
+# --- Staged GGUF models for the llama.cpp router (:8002) ---------------------
+# Data-driven list consumed by roles/llm-inference-multimodel tasks/models.yml
+# (loop -> tasks/stage_model.yml). Each entry is idempotently staged into
+# /opt/models: stat + EXACT-size check vs HF manifest; skip (no download, no
+# restart) when present + size matches. Source repos are public bartowski GGUFs
+# on HuggingFace (no auth). A router restart is notified ONLY when a new GGUF
+# is actually downloaded.
+# Added 2026-08-12 (War Machine): codify Phi-3.5-mini-instruct-Q8_0 and
+# Meta-Llama-3.1-8B-Instruct-Q4_K_M as router models alongside the production
+# Qwen3.6-35B-A3B-UD-Q4_K_S. The live files were already present/correct on
+# astro-orbiter; this pass codifies them. Future adds = append to this list.
+# Router --models-max override for astro-orbiter.
+# Default in defaults/main.yml is 1 (conservative). Bumped to 4 on 2026-08-12
+# (t_33acbb2e) so the router can keep more than one GGUF resident on-demand
+# and LRU-evict when needed.
+#
+# VRAM NOTE (t_33acbb2e, updated t_55c164f5, updated t_34b96e83, updated t_f5f7e9ad, updated t_441470b9, updated t_c5cef2b2):
+# With models-max=4 and all 6 GGUFs registered, worst case is all 6 loaded simultaneously:
+#   Qwen3.8-27B Q4_K_M:               ~20.0GB (weights ~17.1GB + KV ~2.9GB @ 65536 ctx, q4_0) ← CORRECTED (ctx rolled back from 128K to 65536, t_c9fed26c 2026-08-18)
+#   Phi-3.5-mini-instruct Q8_0:       ~4.3GB  (weights ~3.8GB + KV ~0.5GB @ 32K ctx)
+#   Meta-Llama-3.1-8B Q4_K_M:        ~5.6GB  (weights ~4.6GB + KV ~0.2GB @ 8K ctx)
+#   Qwen2.5-Coder-14B Q4_K_M:        ~9.0GB  (weights ~8.4GB + KV ~0.6GB @ 16K ctx)
+#   nomic-embed-text-v1.5 Q4_K_M:    ~0.09GB (~84MB, embedding only — no KV cache)
+#   Qwen3-8B Q4_K_M:                 ~5.5GB  (weights ~4.68GB + KV ~0.5GB @ 32K ctx, q4_0)
+#   Total worst-case:                 ~44.5GB  >>  24GB RTX 3090
+#
+# OOM RISK: Full co-residency is impossible on 24GB. LRU eviction prevents this
+# in practice: models-max=4 means the router can REGISTER 6 models but only keeps
+# up to 4 LOADED simultaneously — the router will evict the LRU model when a new
+# one is needed. nomic-embed-text-v1.5 is pinned via sleep-idle-seconds=-1 and
+# load-on-startup=true but it uses only ~84MB, so it never meaningfully changes
+# the budget. In single-user homelab operation, only one generative model is active
+# at a time alongside the always-resident embedding model.
+# Qwen3.8-27B alone uses ~17,804 MiB (weights+KV @ 65536 ctx); co-residency
+# with Coder (~9GB) = ~27GB > 24GB. LRU eviction handles this automatically.
+# Ryan should be aware this means model-switching always incurs a ~30-60s
+# cold-load latency when switching between Qwen3.8-27B and any other model.
+# Proceeding to models-max=4 as instructed; flagged for Ryan's attention.
+# Router --models-max override for astro-orbiter.
+# UPDATED (t_f5f7e9ad, 2026-08-16): Set to 2 because Qwen3.8-27B-Q4_K_M
+# uses 17,804 MiB at 65536 ctx. Only nomic-embed (558MB, pinned) and ONE
+# generative model can be resident simultaneously. Co-residency of Qwen3.8
+# with any auxiliary model (Phi 8.3GB, Llama 5.9GB, Coder 9GB) exceeds 24GB.
+# models-max=2: slot 1 = nomic-embed (pinned, always loaded), slot 2 = LRU
+# generative model (Qwen3.8 primary, cold-loaded on first request ~30-60s;
+# auxiliary models evict it on demand, and vice versa).
+# NOTE: Qwen3.8 does NOT have load-on-startup — it loads on first request.
+# This avoids an LRU eviction race with nomic-embed at startup.
+# UPDATED (t_72646029, 2026-08-17): CPU offload for Coder + Llama changes the
+# constraint. Coder and Llama now use CPU inference (n-gpu-layers=0). GPU-resident
+# VRAM: Qwen3.8 (~17,804 MiB at 65536 ctx) + nomic-embed (558 MiB, pinned) plus
+# the CUDA-context buffers llama.cpp 6ea215d allocates for the CPU models (~1.4-1.7GB
+# each) = ~20,004 MiB steady-state, below the 24,576 MiB physical limit.
+# CORRECTED (t_c5cef2b2, 2026-08-19): ctx-size was rolled back from 131072 to 65536
+# (t_c9fed26c 2026-08-18). Qwen3.8 VRAM at 65536: 17,804 MiB (not 20,302 MiB).
+# models-max raised to 4: nomic (slot 1, pinned) + Qwen3.8 (slot 2, GPU) +
+# Llama (slot 3, CPU) + Coder (slot 4, CPU). Phi (GPU, ~8.3GB) and new
+# Qwen3-8B (GPU, ~5.5GB) can also be requested but evict Qwen3.8 due to VRAM.
+# models-max=4 is required so CPU-offloaded models count as loaded without
+# evicting Qwen3.8.
+llm_router_models_max: 4
+
+llm_staged_models:
+  - filename: "Phi-3.5-mini-instruct-Q8_0.gguf"
+    url: "https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF/resolve/main/Phi-3.5-mini-instruct-Q8_0.gguf"
+    size_bytes: 4061222688
+    source_repo: "bartowski/Phi-3.5-mini-instruct-GGUF"
+  - filename: "Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
+    url: "https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
+    size_bytes: 4920739232
+    source_repo: "bartowski/Meta-Llama-3.1-8B-Instruct-GGUF"
+  - filename: "Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
+    url: "https://huggingface.co/bartowski/Qwen2.5-Coder-14B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
+    size_bytes: 8988111072
+    source_repo: "bartowski/Qwen2.5-Coder-14B-Instruct-GGUF"
+  - filename: "nomic-embed-text-v1.5-Q4_K_M.gguf"
+    url: "https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF/resolve/main/nomic-embed-text-v1.5.Q4_K_M.gguf"
+    size_bytes: 84106624
+    source_repo: "nomic-ai/nomic-embed-text-v1.5-GGUF"
+  # Added t_c5cef2b2 (2026-08-19, War Machine): Qwen3-8B dense 8B model for
+  # aux tasks (routing, rewriting, structured extraction, tool-call construction).
+  # Source: bartowski/Qwen_Qwen3-8B-GGUF (public, no auth). HF filename is
+  # Qwen_Qwen3-8B-Q4_K_M.gguf; stored locally as Qwen3-8B-Q4_K_M.gguf.
+  # Exact size verified from HF manifest (content-length): 5,027,784,224 bytes.
+  # VRAM: ~4.68GB weights + ~0.5GB KV @ 32K ctx (q4_0) ≈ 5.2GB total.
+  # Thinking mode ON by default; use /no_think for latency-sensitive aux tasks.
+  - filename: "Qwen3-8B-Q4_K_M.gguf"
+    url: "https://huggingface.co/bartowski/Qwen_Qwen3-8B-GGUF/resolve/main/Qwen_Qwen3-8B-Q4_K_M.gguf"
+    size_bytes: 5027784224
+    source_repo: "bartowski/Qwen_Qwen3-8B-GGUF"
+
+# --- deploy-vllm role: vllm_models override (t_r1d32b_swap, 2026-09-01) -----
+# Ansible's hash_behaviour is "replace" (see ansible.cfg) — a host_vars list
+# variable REPLACES the role default list wholesale, it does not deep-merge.
+#
+# SWAP (Ryan direction, 2026-09-01): Qwen2.5-32B-Instruct-AWQ retired,
+# replaced with DeepSeek-R1-Distill-Qwen-32B-AWQ, max_model_len=32768.
+# "Single model only" — nomic-embed-text-v1.5 (embedding, :8020) and
+# Qwen3-8B-AWQ (aux, :8010, already disabled) are BOTH disabled here.
+# DeepSeek gets the full 24GB card to itself. Nothing in production
+# consumed nomic-embed at the time of this swap (Hindsight uses its own
+# bundled 384-dim embedder; OpenViking pointed at the old llama-swap
+# endpoint, already stopped) — confirmed with Ryan before disabling.
+#
+# Model choice: casperhansen/deepseek-r1-distill-qwen-32b-awq — same
+# quantizer/toolchain (AutoAWQ) as the outgoing Qwen2.5-32B-Instruct-AWQ,
+# widely used, 4-bit GEMM AWQ, ~19.3GB on disk (4 safetensors shards).
+# Architecture: Qwen2ForCausalLM (DeepSeek-R1 distilled onto Qwen2.5-32B
+# base) — same vLLM code path as the outgoing model, no new serving
+# support needed. Native max_position_embeddings=131072; we cap at 32768
+# per the task's explicit max-model-len requirement.
+#
+# VRAM math: ~19.3GB weights (4-bit AWQ) + KV cache at 32768 ctx (GQA,
+# 8 KV heads, 128 head_dim, 64 layers, fp16 KV by default) ≈ 19.3GB +
+# ~4GB KV+overhead ≈ 23.3GB — tight but the FULL 24GB card is now
+# available (no co-resident nomic-embed/Qwen3-8B taking a share, unlike
+# the outgoing Qwen2.5-32B config). gpu_memory_utilization=0.95 (role
+# default) + enforce_eager retained as the proven-stable mitigation from
+# t_e6facb19/t_ca1af9fb (avoids CUDA graph capture VRAM spike; this host's
+# only validated way to avoid crash-loop-to-stabilize behavior on this
+# card). If 0.95 OOMs at 32768 ctx once tested live, drop to 0.90 next
+# (documented fallback, same pattern as the outgoing model).
+#
+# DeepSeek-R1 output note: reasoning traces stream in <think> tags before
+# the final answer — this is expected R1-distill behavior, not a bug.
+# Model card recommends temperature 0.5-0.7 (not 0, not vLLM's greedy
+# default) to avoid repetition/incoherence; not set here (server-side
+# default), left to be set client-side per the model card's guidance —
+# flagging for whoever wires this into Hermes profile configs next.
+vllm_models:
+  - id: "Gemma-4-26B-A4B-it-AWQ"
+    hf_repo: "cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit"
+    role: primary
+    # NO quantization field set (unlike the AutoAWQ-quantized DeepSeek/
+    # Qwen2.5 models above) — live test (2026-09-01) found this repo's
+    # config.json declares quant_method: "compressed-tensors" (llm-compressor
+    # tool output, not classic AutoAWQ), even though the repo name says
+    # "AWQ-4bit". Passing --quantization awq explicitly caused a hard
+    # pydantic ValidationError at every single startup attempt: "Quantization
+    # method specified in the model config (compressed-tensors) does not
+    # match the quantization method specified in the `quantization` argument
+    # (awq)." vLLM auto-detects the quant method correctly from the model's
+    # own config.json when --quantization is omitted — confirmed fix, clean
+    # start. Lesson: don't trust a HF repo's naming convention ("...-AWQ...")
+    # for the `quantization:` field here — check config.json's quant_method.
+    port: 8000
+    # Ryan direction (2026-09-01, t_gemma4_swap): DeepSeek-R1-Distill-Qwen-32B
+    # retired after confirming its `auto` tool-choice reliability is a known,
+    # documented DeepSeek-R1-distillation limitation (trained on pure
+    # reasoning traces, no function-calling data — GitHub-confirmed upstream,
+    # not a vLLM config gap). Replaced with Gemma 4 26B A4B (Google,
+    # Apache 2.0, US-origin — matches Ryan's standing model-origin
+    # preference, unlike Qwen/DeepSeek). Chose MoE (26B A4B, 3.8B active)
+    # over the dense 31B variant: ~3.7GB smaller on-disk AWQ footprint
+    # (17.2GB vs 20.9GB) buys more KV-cache headroom on this tight 24GB
+    # card, and decode should be faster (memory-bandwidth-bound on active
+    # params, not total params). Tradeoff accepted: MoE scores lower than
+    # dense on the Tau2 tool-use benchmark (68.2% vs 76.9%) but still beats
+    # every other size in the family except the 31B on most reasoning
+    # benchmarks. Model choice: cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit —
+    # AutoAWQ 4-bit group_size=32, MoE expert layers (gate/up/down/router)
+    # explicitly excluded from quantization ("ignore" list in config.json)
+    # per standard llm-compressor MoE quant practice — only the dense
+    # attention/projection layers are 4-bit, experts stay higher precision.
+    # Native architecture: Gemma4ForConditionalGeneration (registered
+    # natively in this host's installed vLLM 0.28.0 — vllm/model_executor/
+    # models/registry.py line 415 — no plugin/trust-remote-code needed).
+    # Native max_position_embeddings: 262144 (256K) — Hermes's 64K floor is
+    # comfortably covered without any context-extension trick.
+    max_model_len: 65536
+    # VRAM math (not yet live-validated — see swap validation log below
+    # once run): AWQ weights ~17.2GB on disk (dense attn 4-bit + MoE
+    # experts higher-precision, per config.json's compressed-tensors
+    # ignore list). Starting the KV cache dtype at int4_per_token_head
+    # from the outset (rather than fp16 -> fp8 -> int4 trial-and-error like
+    # the DeepSeek swap) since that same escalation pattern is expected to
+    # repeat on this VRAM-constrained card for any 20+ GB model at >32K ctx.
+    kv_cache_dtype: int4_per_token_head
+    gpu_memory_utilization: 0.95
+    enforce_eager: true
+    # Native tool-calling + reasoning support (no `hermes` workaround
+    # needed, unlike DeepSeek-R1-Distill): Gemma4EngineToolParser and
+    # Gemma4ParserReasoningAdapter are both registered natively in this
+    # host's vLLM 0.28.0 (vllm/tool_parsers/__init__.py,
+    # vllm/reasoning/__init__.py) — purpose-built for this model's actual
+    # output format, not a same-family approximation.
+    enable_auto_tool_choice: true
+    tool_call_parser: gemma4
+    reasoning_parser: gemma4
+    enabled: true
+  - id: "Qwen3-8B-AWQ"
+    hf_repo: "Qwen/Qwen3-8B-AWQ"
+    role: aux
+    quantization: awq
+    port: 8010
+    max_model_len: 32768
+    gpu_memory_utilization: 0.15
+    enforce_eager: true
+    enabled: false   # single-model deployment — see swap note above
+  - id: "nomic-embed-text-v1.5"
+    hf_repo: "nomic-ai/nomic-embed-text-v1.5"
+    role: embedding
+    quantization: none
+    port: 8020
+    max_model_len: 2048
+    gpu_memory_utilization: 0.05
+    trust_remote_code: true
+    enabled: false   # single-model deployment — see swap note above
+
+# --- deploy-vllm role: boot persistence (unchanged) -------------------------
+# Still permanent/boot-persistent — same policy as the outgoing Qwen2.5-32B
+# deployment (t_5508360a), just now serving one model instead of two.
+vllm_service_enabled: true
+vllm_service_state: started
+

ansible/host_vars/big-thunder-mountain/old_vars

@@ -0,0 +1,58 @@
+---
+# file: host_vars/vm_template/vars
+# Ansible vars template for hosts created via cloning.
+
+# Supported hypervisors:
+# - Proxmox
+
+platform: "proxmox"
+
+# This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
+
+proxmox_clone_node: "fantasyland"
+
+# Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
+# Current OS offerings are:
+# - Ubuntu 
+#    - 24.04
+# - Fedora
+#    - 42
+
+vm_os_distribution: "ubuntu"
+vm_os_version: "24.04"
+
+# Current VM sizes are:
+# - Small: 2 cores, 2GB memory, 8 GiB virtual disk
+# - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
+# - Large: 4 cores, 4GB memory, 32 GiB virtual disk
+# - Large Plus: 4 cores, 4GB memory, 48 GiB virtual disk
+# - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
+# - Xlarge Plus: 4 cores, 8GB memory, 128 GiB virtual disk
+
+vm_size: "large-plus"
+
+# Proxmox storage target.
+
+vm_storage: "general"
+
+# Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
+# and backup groups. (ha_group will be factored out in the next functionality update.)
+
+ha_group: "pve03"
+proxmox_host_target: "pve03"
+
+# Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
+# sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
+
+ip_address: 10.1.71.52
+vm_mac_address: 'BC:24:11:11:BC:58'
+
+# Software
+# Future enhancement will allow specification of additional software to automatically deploy to
+# the VM after creation.
+
+#terraform_version: "1.11.3"
+
+# ---
+vm_clone_source: "{{ vm_os_distribution }}-{{ vm_os_version }}-{{ vm_size }}"
+hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn

ansible/host_vars/big-thunder-mountain/vars

@@ -0,0 +1,99 @@
+---
+# file: host_vars/vm_template/vars
+# Ansible vars template for hosts created via cloning.
+
+# Supported hypervisors:
+# - Proxmox
+
+platform: "proxmox"
+
+# Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
+# Current OS offerings are:
+# - Ubuntu (clone)
+#    - 24.04
+# - Fedora (clone)
+#    - 42
+# - Talos (create)
+#    - 1.11.5
+
+vm_os_distribution: "talos"
+vm_os_version: "1.11.5"
+
+# VM ISO
+# Talos: talos-v1.11.5-nocloud-amd64.iso
+
+vm_iso_storage: "templates"
+vm_iso: "talos-v{{ vm_os_version }}-nocloud-amd64.iso"
+
+# This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
+# Comment out for VM create
+
+proxmox_clone_node: "fantasyland"
+
+# Proxmox storage target.
+
+vm_storage: "liberty-tree"
+
+# Current VM sizes are:
+# - Small: 2 cores, 2GB memory, 8 GiB virtual disk
+# - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
+# - Large: 4 cores, 4GB memory, 32 GiB virtual disk
+# - Large Plus: 4 cores, 4GB memory, 48 GiB virtual disk
+# - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
+# - Xlarge Plus: 4 cores, 8GB memory, 128 GiB virtual disk
+
+vm_size: "large-plus"
+
+# Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
+# and backup groups. (ha_group will be factored out in the next functionality update.)
+
+ha_group: "pve03"
+proxmox_host_target: "pve03"
+
+# Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
+# sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
+
+ip_address: 10.1.71.52
+vm_mac_address: 'BC:24:11:11:BC:58'
+
+# Software
+# Future enhancement will allow specification of additional software to automatically deploy to
+# the VM after creation.
+
+#terraform_version: "1.11.3"
+
+# ---
+
+vm_clone_source: "{{ vm_os_distribution }}-{{ vm_os_version }}-{{ vm_size }}"
+hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn
+
+# Dictionaries for VM resources
+# - cores (quantity)
+# - memory (MB)
+# - virtual disk (GiB)
+
+vm_definitions:
+  small:
+    cores: 2
+    memory: 2048
+    disk: 8
+  medium:
+    cores: 2
+    memory: 4096
+    disk: 16
+  large:
+    cores: 4
+    memory: 4096
+    disk: 32
+  large-plus:
+    cores: 4
+    memory: 4096
+    disk: 48
+  xlarge:
+    cores: 4
+    memory: 8192
+    disk: 64
+  xlarge-plus:
+    cores: 4
+    memory: 8192
+    disk: 128

ansible/host_vars/carousel-of-progress/vars.yml

@@ -0,0 +1,16 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: ansible/host_vars/astro_orbiter/vars.yml
+# HOST: astro-orbiter (10.1.71.130)
+# ROLE: Ollama inference host with AMD RX 5700 GPU passthrough
+# ------------------------------------------------------------------------------
+
+ansible_host: 10.1.71.131
+ansible_user: wed
+ansible_become: true
+
+# LVM root expansion — xlarge template uses sda3 partition, standard VG/LV names
+common_expand_root_lvm: true
+common_root_pv: /dev/sda3
+common_root_vg: ubuntu-vg
+common_root_lv: ubuntu-lv

ansible/host_vars/city-hall/vars

@@ -0,0 +1,48 @@
+---
+# file: host_vars/vm_template/vars
+# Ansible vars template for hosts created via cloning.
+
+# Supported hypervisors:
+# - Proxmox
+
+platform: "proxmox"
+
+# This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
+
+proxmox_clone_node: "fantasyland"
+
+# Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
+# Current OS offerings are:
+# - Fedora (42)
+# Current VM sizes are:
+# - Small: 2 cores, 2GB memory, 8 GiB virtual disk
+# - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
+# - Large: 4 cores, 4GB memory, 32 GiB virtual disk
+# - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
+
+vm_clone_source: "fedora-42-small"
+
+# Proxmox storage target.
+
+vm_storage: "mk-general"
+
+# Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
+# and backup groups. (ha_group will be factored out in the next functionality update.)
+
+ha_group: "pve03"
+proxmox_host_target: "pve03"
+
+# Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
+# sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
+
+ip_address: 10.1.71.21
+
+# Software
+# Future enhancement will allow specification of additional software to automatically deploy to
+# the VM after creation.
+
+#terraform_version: "1.11.3"
+
+# ---
+
+hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn

ansible/host_vars/docker01/vars

@@ -1,9 +1,9 @@
 ---
-# file: host_vars/matchbox/vars
+# file: host_vars/docker01/vars
 
 # Networking
 # primary_interface: "enp1s0f0"
-mac_address: "BC:24:11:03:A9:7B"
+#mac_address: "BC:24:11:03:A9:7B"
 ip_address: 10.1.71.211
 hostname: "{{ inventory_hostname }}.{{ base_domain }}"
 

ansible/host_vars/guest-relations/vars

@@ -0,0 +1,12 @@
+---
+# file: host_vars/guest-relations/vars
+# guest-relations — Guest Relations
+# VM provisioned by Terraform (pre-pipeline bootstrap)
+
+# ─── Network ─────────────────────────────────────────────────────────────────
+ip_address: 10.1.71.40
+
+# ─── Application ─────────────────────────────────────────────────────────────
+app_role: identity_provider
+app_name: authentik
+app_deployment: docker_compose

ansible/host_vars/haunted-mansion/old_vars

@@ -0,0 +1,60 @@
+---
+# file: host_vars/vm_template/vars
+# Ansible vars template for hosts created via cloning.
+
+# Supported hypervisors:
+# - Proxmox
+
+platform: "proxmox"
+
+# This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
+
+proxmox_clone_node: "fantasyland"
+
+# Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
+# Current OS offerings are:
+# - Ubuntu 
+#    - 24.04
+# - Fedora
+#    - 42
+
+vm_os_distribution: "ubuntu"
+vm_os_version: "24.04"
+
+# Current VM sizes are:
+# - Small: 2 cores, 2GB memory, 8 GiB virtual disk
+# - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
+# - Large: 4 cores, 4GB memory, 32 GiB virtual disk
+# - Large Plus: 4 cores, 4GB memory, 48 GiB virtual disk
+# - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
+# - Xlarge Plus: 4 cores, 8GB memory, 128 GiB virtual disk
+
+vm_size: "xlarge"
+
+# Proxmox storage target.
+
+vm_storage: "general"
+
+# Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
+# and backup groups. (ha_group will be factored out in the next functionality update.)
+
+ha_group: "pve03"
+proxmox_host_target: "pve03"
+
+# Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
+# sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
+
+ip_address: 10.1.71.61
+vm_mac_address: 'BC:24:11:C9:3A:13'
+
+# Software
+# Future enhancement will allow specification of additional software to automatically deploy to
+# the VM after creation.
+
+#terraform_version: "1.11.3"
+
+# ---
+
+
+vm_clone_source: "{{ vm_os_distribution }}-{{ vm_os_version }}-{{ vm_size }}"
+hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn

ansible/host_vars/haunted-mansion/vars

@@ -0,0 +1,99 @@
+---
+# file: host_vars/vm_template/vars
+# Ansible vars template for hosts created via cloning.
+
+# Supported hypervisors:
+# - Proxmox
+
+platform: "proxmox"
+
+# Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
+# Current OS offerings are:
+# - Ubuntu (clone)
+#    - 24.04
+# - Fedora (clone)
+#    - 42
+# - Talos (create)
+#    - 1.11.5
+
+vm_os_distribution: "talos"
+vm_os_version: "1.11.5"
+
+# VM ISO
+# Talos: talos-v1.11.5-nocloud-amd64.iso
+
+vm_iso_storage: "templates"
+vm_iso: "talos-v{{ vm_os_version }}-nocloud-amd64.iso"
+
+# This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
+# Comment out for VM create
+
+proxmox_clone_node: "fantasyland"
+
+# Proxmox storage target.
+
+vm_storage: "liberty-tree"
+
+# Current VM sizes are:
+# - Small: 2 cores, 2GB memory, 8 GiB virtual disk
+# - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
+# - Large: 4 cores, 4GB memory, 32 GiB virtual disk
+# - Large Plus: 4 cores, 4GB memory, 48 GiB virtual disk
+# - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
+# - Xlarge Plus: 4 cores, 8GB memory, 128 GiB virtual disk
+
+vm_size: "xlarge"
+
+# Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
+# and backup groups. (ha_group will be factored out in the next functionality update.)
+
+ha_group: "pve03"
+proxmox_host_target: "pve03"
+
+# Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
+# sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
+
+ip_address: 10.1.71.61
+vm_mac_address: 'BC:24:11:C9:3A:13'
+
+# Software
+# Future enhancement will allow specification of additional software to automatically deploy to
+# the VM after creation.
+
+#terraform_version: "1.11.3"
+
+# ---
+
+vm_clone_source: "{{ vm_os_distribution }}-{{ vm_os_version }}-{{ vm_size }}"
+hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn
+
+# Dictionaries for VM resources
+# - cores (quantity)
+# - memory (MB)
+# - virtual disk (GiB)
+
+vm_definitions:
+  small:
+    cores: 2
+    memory: 2048
+    disk: 8
+  medium:
+    cores: 2
+    memory: 4096
+    disk: 16
+  large:
+    cores: 4
+    memory: 4096
+    disk: 32
+  large-plus:
+    cores: 4
+    memory: 4096
+    disk: 48
+  xlarge:
+    cores: 4
+    memory: 8192
+    disk: 64
+  xlarge-plus:
+    cores: 4
+    memory: 8192
+    disk: 128

ansible/host_vars/illuminations/host_vars_illuminations.yml

@@ -0,0 +1,26 @@
+---
+# Paperclip Docker deployment
+paperclip_domain: cranium-command.wdw-labs.cloud
+paperclip_base_path: /opt/paperclip
+paperclip_port: 3100
+certbot_email: ryan.blundon@protonmail.com
+
+# vLLM inference endpoint (local)
+vllm_endpoint: "http://127.0.0.1:8000"
+
+hermes_model:
+  provider: custom
+  default: Gemma-4-26B-A4B-Hermes-VLM-MLX      # from http://fantasmic.wdw-labs.cloud:8000/v1/models
+  base_url: http://fantasmic.wdw-labs.cloud:8000/v1
+  api_key: "{{ vault_fantasmic_api_key }}"
+
+# # DNS entries (if using your existing dns-manager role)
+# dns_entries:
+#   - name: illuminations
+#     domain: wdw-labs.cloud
+#     type: A
+#     value: "{{ ansible_host }}"
+#   - name: muppet-vision
+#     domain: wdw-labs.cloud
+#     type: A
+#     value: "{{ ansible_host }}"

ansible/host_vars/illuminations/vars.yml

@@ -0,0 +1,244 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: ansible/host_vars/illuminations/vars.yml
+# HOST: illuminations (10.1.71.130)
+# ROLE: llama.cpp LLM inference host — Ryzen 7 5800XT / RTX 3090 (ATX rebuild,
+#       2026-08-04). Superseded the prior AMD RX 5700 / Ollama config below;
+#       drive was transplanted into new hardware, not reinstalled.
+# ------------------------------------------------------------------------------
+
+ansible_host: 10.1.71.130
+ansible_user: jarvis
+ansible_ssh_private_key_file: ~/.ssh/id_jarvis
+ansible_become: true
+
+# LVM root expansion — xlarge template uses sda3 partition, standard VG/LV names
+common_expand_root_lvm: true
+common_root_pv: /dev/sda3
+common_root_vg: ubuntu-vg
+common_root_lv: ubuntu-lv
+
+# --- Staged GGUF models for the llama.cpp router (:8002) ---------------------
+# Data-driven list consumed by roles/llm-inference-multimodel tasks/models.yml
+# (loop -> tasks/stage_model.yml). Each entry is idempotently staged into
+# /opt/models: stat + EXACT-size check vs HF manifest; skip (no download, no
+# restart) when present + size matches. Source repos are public bartowski GGUFs
+# on HuggingFace (no auth). A router restart is notified ONLY when a new GGUF
+# is actually downloaded.
+# Added 2026-08-12 (War Machine): codify Phi-3.5-mini-instruct-Q8_0 and
+# Meta-Llama-3.1-8B-Instruct-Q4_K_M as router models alongside the production
+# Qwen3.6-35B-A3B-UD-Q4_K_S. The live files were already present/correct on
+# astro-orbiter; this pass codifies them. Future adds = append to this list.
+# Router --models-max override for astro-orbiter.
+# Default in defaults/main.yml is 1 (conservative). Bumped to 4 on 2026-08-12
+# (t_33acbb2e) so the router can keep more than one GGUF resident on-demand
+# and LRU-evict when needed.
+#
+# VRAM NOTE (t_33acbb2e, updated t_55c164f5, updated t_34b96e83, updated t_f5f7e9ad, updated t_441470b9, updated t_c5cef2b2):
+# With models-max=4 and all 6 GGUFs registered, worst case is all 6 loaded simultaneously:
+#   Qwen3.8-27B Q4_K_M:               ~20.0GB (weights ~17.1GB + KV ~2.9GB @ 65536 ctx, q4_0) ← CORRECTED (ctx rolled back from 128K to 65536, t_c9fed26c 2026-08-18)
+#   Phi-3.5-mini-instruct Q8_0:       ~4.3GB  (weights ~3.8GB + KV ~0.5GB @ 32K ctx)
+#   Meta-Llama-3.1-8B Q4_K_M:        ~5.6GB  (weights ~4.6GB + KV ~0.2GB @ 8K ctx)
+#   Qwen2.5-Coder-14B Q4_K_M:        ~9.0GB  (weights ~8.4GB + KV ~0.6GB @ 16K ctx)
+#   nomic-embed-text-v1.5 Q4_K_M:    ~0.09GB (~84MB, embedding only — no KV cache)
+#   Qwen3-8B Q4_K_M:                 ~5.5GB  (weights ~4.68GB + KV ~0.5GB @ 32K ctx, q4_0)
+#   Total worst-case:                 ~44.5GB  >>  24GB RTX 3090
+#
+# OOM RISK: Full co-residency is impossible on 24GB. LRU eviction prevents this
+# in practice: models-max=4 means the router can REGISTER 6 models but only keeps
+# up to 4 LOADED simultaneously — the router will evict the LRU model when a new
+# one is needed. nomic-embed-text-v1.5 is pinned via sleep-idle-seconds=-1 and
+# load-on-startup=true but it uses only ~84MB, so it never meaningfully changes
+# the budget. In single-user homelab operation, only one generative model is active
+# at a time alongside the always-resident embedding model.
+# Qwen3.8-27B alone uses ~17,804 MiB (weights+KV @ 65536 ctx); co-residency
+# with Coder (~9GB) = ~27GB > 24GB. LRU eviction handles this automatically.
+# Ryan should be aware this means model-switching always incurs a ~30-60s
+# cold-load latency when switching between Qwen3.8-27B and any other model.
+# Proceeding to models-max=4 as instructed; flagged for Ryan's attention.
+# Router --models-max override for astro-orbiter.
+# UPDATED (t_f5f7e9ad, 2026-08-16): Set to 2 because Qwen3.8-27B-Q4_K_M
+# uses 17,804 MiB at 65536 ctx. Only nomic-embed (558MB, pinned) and ONE
+# generative model can be resident simultaneously. Co-residency of Qwen3.8
+# with any auxiliary model (Phi 8.3GB, Llama 5.9GB, Coder 9GB) exceeds 24GB.
+# models-max=2: slot 1 = nomic-embed (pinned, always loaded), slot 2 = LRU
+# generative model (Qwen3.8 primary, cold-loaded on first request ~30-60s;
+# auxiliary models evict it on demand, and vice versa).
+# NOTE: Qwen3.8 does NOT have load-on-startup — it loads on first request.
+# This avoids an LRU eviction race with nomic-embed at startup.
+# UPDATED (t_72646029, 2026-08-17): CPU offload for Coder + Llama changes the
+# constraint. Coder and Llama now use CPU inference (n-gpu-layers=0). GPU-resident
+# VRAM: Qwen3.8 (~17,804 MiB at 65536 ctx) + nomic-embed (558 MiB, pinned) plus
+# the CUDA-context buffers llama.cpp 6ea215d allocates for the CPU models (~1.4-1.7GB
+# each) = ~20,004 MiB steady-state, below the 24,576 MiB physical limit.
+# CORRECTED (t_c5cef2b2, 2026-08-19): ctx-size was rolled back from 131072 to 65536
+# (t_c9fed26c 2026-08-18). Qwen3.8 VRAM at 65536: 17,804 MiB (not 20,302 MiB).
+# models-max raised to 4: nomic (slot 1, pinned) + Qwen3.8 (slot 2, GPU) +
+# Llama (slot 3, CPU) + Coder (slot 4, CPU). Phi (GPU, ~8.3GB) and new
+# Qwen3-8B (GPU, ~5.5GB) can also be requested but evict Qwen3.8 due to VRAM.
+# models-max=4 is required so CPU-offloaded models count as loaded without
+# evicting Qwen3.8.
+llm_router_models_max: 4
+
+llm_staged_models:
+  - filename: "Phi-3.5-mini-instruct-Q8_0.gguf"
+    url: "https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF/resolve/main/Phi-3.5-mini-instruct-Q8_0.gguf"
+    size_bytes: 4061222688
+    source_repo: "bartowski/Phi-3.5-mini-instruct-GGUF"
+  - filename: "Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
+    url: "https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
+    size_bytes: 4920739232
+    source_repo: "bartowski/Meta-Llama-3.1-8B-Instruct-GGUF"
+  - filename: "Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
+    url: "https://huggingface.co/bartowski/Qwen2.5-Coder-14B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
+    size_bytes: 8988111072
+    source_repo: "bartowski/Qwen2.5-Coder-14B-Instruct-GGUF"
+  - filename: "nomic-embed-text-v1.5-Q4_K_M.gguf"
+    url: "https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF/resolve/main/nomic-embed-text-v1.5.Q4_K_M.gguf"
+    size_bytes: 84106624
+    source_repo: "nomic-ai/nomic-embed-text-v1.5-GGUF"
+  # Added t_c5cef2b2 (2026-08-19, War Machine): Qwen3-8B dense 8B model for
+  # aux tasks (routing, rewriting, structured extraction, tool-call construction).
+  # Source: bartowski/Qwen_Qwen3-8B-GGUF (public, no auth). HF filename is
+  # Qwen_Qwen3-8B-Q4_K_M.gguf; stored locally as Qwen3-8B-Q4_K_M.gguf.
+  # Exact size verified from HF manifest (content-length): 5,027,784,224 bytes.
+  # VRAM: ~4.68GB weights + ~0.5GB KV @ 32K ctx (q4_0) ≈ 5.2GB total.
+  # Thinking mode ON by default; use /no_think for latency-sensitive aux tasks.
+  - filename: "Qwen3-8B-Q4_K_M.gguf"
+    url: "https://huggingface.co/bartowski/Qwen_Qwen3-8B-GGUF/resolve/main/Qwen_Qwen3-8B-Q4_K_M.gguf"
+    size_bytes: 5027784224
+    source_repo: "bartowski/Qwen_Qwen3-8B-GGUF"
+
+# --- deploy-vllm role: vllm_models override (t_r1d32b_swap, 2026-09-01) -----
+# Ansible's hash_behaviour is "replace" (see ansible.cfg) — a host_vars list
+# variable REPLACES the role default list wholesale, it does not deep-merge.
+#
+# SWAP (Ryan direction, 2026-09-01): Qwen2.5-32B-Instruct-AWQ retired,
+# replaced with DeepSeek-R1-Distill-Qwen-32B-AWQ, max_model_len=32768.
+# "Single model only" — nomic-embed-text-v1.5 (embedding, :8020) and
+# Qwen3-8B-AWQ (aux, :8010, already disabled) are BOTH disabled here.
+# DeepSeek gets the full 24GB card to itself. Nothing in production
+# consumed nomic-embed at the time of this swap (Hindsight uses its own
+# bundled 384-dim embedder; OpenViking pointed at the old llama-swap
+# endpoint, already stopped) — confirmed with Ryan before disabling.
+#
+# Model choice: casperhansen/deepseek-r1-distill-qwen-32b-awq — same
+# quantizer/toolchain (AutoAWQ) as the outgoing Qwen2.5-32B-Instruct-AWQ,
+# widely used, 4-bit GEMM AWQ, ~19.3GB on disk (4 safetensors shards).
+# Architecture: Qwen2ForCausalLM (DeepSeek-R1 distilled onto Qwen2.5-32B
+# base) — same vLLM code path as the outgoing model, no new serving
+# support needed. Native max_position_embeddings=131072; we cap at 32768
+# per the task's explicit max-model-len requirement.
+#
+# VRAM math: ~19.3GB weights (4-bit AWQ) + KV cache at 32768 ctx (GQA,
+# 8 KV heads, 128 head_dim, 64 layers, fp16 KV by default) ≈ 19.3GB +
+# ~4GB KV+overhead ≈ 23.3GB — tight but the FULL 24GB card is now
+# available (no co-resident nomic-embed/Qwen3-8B taking a share, unlike
+# the outgoing Qwen2.5-32B config). gpu_memory_utilization=0.95 (role
+# default) + enforce_eager retained as the proven-stable mitigation from
+# t_e6facb19/t_ca1af9fb (avoids CUDA graph capture VRAM spike; this host's
+# only validated way to avoid crash-loop-to-stabilize behavior on this
+# card). If 0.95 OOMs at 32768 ctx once tested live, drop to 0.90 next
+# (documented fallback, same pattern as the outgoing model).
+#
+# DeepSeek-R1 output note: reasoning traces stream in <think> tags before
+# the final answer — this is expected R1-distill behavior, not a bug.
+# Model card recommends temperature 0.5-0.7 (not 0, not vLLM's greedy
+# default) to avoid repetition/incoherence; not set here (server-side
+# default), left to be set client-side per the model card's guidance —
+# flagging for whoever wires this into Hermes profile configs next.
+vllm_models:
+  - id: "Gemma-4-26B-A4B-it-AWQ"
+    hf_repo: "cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit"
+    role: primary
+    # NO quantization field set (unlike the AutoAWQ-quantized DeepSeek/
+    # Qwen2.5 models above) — live test (2026-09-01) found this repo's
+    # config.json declares quant_method: "compressed-tensors" (llm-compressor
+    # tool output, not classic AutoAWQ), even though the repo name says
+    # "AWQ-4bit". Passing --quantization awq explicitly caused a hard
+    # pydantic ValidationError at every single startup attempt: "Quantization
+    # method specified in the model config (compressed-tensors) does not
+    # match the quantization method specified in the `quantization` argument
+    # (awq)." vLLM auto-detects the quant method correctly from the model's
+    # own config.json when --quantization is omitted — confirmed fix, clean
+    # start. Lesson: don't trust a HF repo's naming convention ("...-AWQ...")
+    # for the `quantization:` field here — check config.json's quant_method.
+    port: 8000
+    # Ryan direction (2026-09-01, t_gemma4_swap): DeepSeek-R1-Distill-Qwen-32B
+    # retired after confirming its `auto` tool-choice reliability is a known,
+    # documented DeepSeek-R1-distillation limitation (trained on pure
+    # reasoning traces, no function-calling data — GitHub-confirmed upstream,
+    # not a vLLM config gap). Replaced with Gemma 4 26B A4B (Google,
+    # Apache 2.0, US-origin — matches Ryan's standing model-origin
+    # preference, unlike Qwen/DeepSeek). Chose MoE (26B A4B, 3.8B active)
+    # over the dense 31B variant: ~3.7GB smaller on-disk AWQ footprint
+    # (17.2GB vs 20.9GB) buys more KV-cache headroom on this tight 24GB
+    # card, and decode should be faster (memory-bandwidth-bound on active
+    # params, not total params). Tradeoff accepted: MoE scores lower than
+    # dense on the Tau2 tool-use benchmark (68.2% vs 76.9%) but still beats
+    # every other size in the family except the 31B on most reasoning
+    # benchmarks. Model choice: cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit —
+    # AutoAWQ 4-bit group_size=32, MoE expert layers (gate/up/down/router)
+    # explicitly excluded from quantization ("ignore" list in config.json)
+    # per standard llm-compressor MoE quant practice — only the dense
+    # attention/projection layers are 4-bit, experts stay higher precision.
+    # Native architecture: Gemma4ForConditionalGeneration (registered
+    # natively in this host's installed vLLM 0.28.0 — vllm/model_executor/
+    # models/registry.py line 415 — no plugin/trust-remote-code needed).
+    # Native max_position_embeddings: 262144 (256K) — Hermes's 64K floor is
+    # comfortably covered without any context-extension trick.
+    max_model_len: 65536
+    # VRAM math (not yet live-validated — see swap validation log below
+    # once run): AWQ weights ~17.2GB on disk (dense attn 4-bit + MoE
+    # experts higher-precision, per config.json's compressed-tensors
+    # ignore list). Starting the KV cache dtype at int4_per_token_head
+    # from the outset (rather than fp16 -> fp8 -> int4 trial-and-error like
+    # the DeepSeek swap) since that same escalation pattern is expected to
+    # repeat on this VRAM-constrained card for any 20+ GB model at >32K ctx.
+    kv_cache_dtype: int4_per_token_head
+    gpu_memory_utilization: 0.95
+    enforce_eager: true
+    # Native tool-calling + reasoning support (no `hermes` workaround
+    # needed, unlike DeepSeek-R1-Distill): Gemma4EngineToolParser and
+    # Gemma4ParserReasoningAdapter are both registered natively in this
+    # host's vLLM 0.28.0 (vllm/tool_parsers/__init__.py,
+    # vllm/reasoning/__init__.py) — purpose-built for this model's actual
+    # output format, not a same-family approximation.
+    enable_auto_tool_choice: true
+    tool_call_parser: gemma4
+    reasoning_parser: gemma4
+    enabled: true
+  - id: "Qwen3-8B-AWQ"
+    hf_repo: "Qwen/Qwen3-8B-AWQ"
+    role: aux
+    quantization: awq
+    port: 8010
+    max_model_len: 32768
+    gpu_memory_utilization: 0.15
+    enforce_eager: true
+    enabled: false   # single-model deployment — see swap note above
+  - id: "nomic-embed-text-v1.5"
+    hf_repo: "nomic-ai/nomic-embed-text-v1.5"
+    role: embedding
+    quantization: none
+    port: 8020
+    max_model_len: 2048
+    gpu_memory_utilization: 0.05
+    trust_remote_code: true
+    enabled: false   # single-model deployment — see swap note above
+
+# --- deploy-vllm role: boot persistence (unchanged) -------------------------
+# Still permanent/boot-persistent — same policy as the outgoing Qwen2.5-32B
+# deployment (t_5508360a), just now serving one model instead of two.
+vllm_service_enabled: true
+vllm_service_state: started
+
+# Paperclip Docker deployment
+nginx_proxy_certbot_email: ryan.blundon@protonmail.com
+# nginx_proxy_certbot_staging: true # remove after first successful real issuance
+
+hermes_model:
+  provider: custom
+  default: Gemma-4-26B-A4B-Hermes-VLM-MLX      # from http://fantasmic.wdw-labs.cloud:8000/v1/models
+  base_url: http://fantasmic.wdw-labs.cloud:8000/v1
+  api_key: "{{ vault_fantasmic_api_key }}"

ansible/host_vars/lightning_lane/vars

@@ -0,0 +1,12 @@
+---
+# file: host_vars/lightning_lane/vars
+# lightning-lane — Traefik Reverse Proxy / Load Balancer
+# VM provisioned by Terraform (pre-pipeline bootstrap)
+
+# ─── Network ─────────────────────────────────────────────────────────────────
+ip_address: 10.1.71.35
+
+# ─── Application ─────────────────────────────────────────────────────────────
+app_role: reverse_proxy
+app_name: traefik
+app_deployment: docker_compose

ansible/host_vars/main-street-station/main.yml

@@ -0,0 +1,16 @@
+---
+# Host-specific vars for main-street-station (JMRI headless server)
+# LCRR - Lake Country Railroad, Milwaukee Road Oct 1956, HO scale
+
+# JMRI profile ID — find with: ls ~/.jmri/profiles/ on the old box
+# Format: <name>.<8-char-hex>  e.g. LCRR.3d3f1dfc
+# TODO: fill in after restoring config from GitHub backup
+jmri_profile_id: ""
+
+# USB serial device for NCE command station
+# Verify after install: ls -la /dev/ttyUSB* /dev/ttyACM*
+jmri_serial_device: /dev/ttyUSB0
+
+# Path to JMRI config backup for restore task (leave empty to skip)
+# Point at a local checkout of the LCRR GitHub repo
+jmri_config_src: ""

ansible/host_vars/main-street-station/vars.yml

@@ -0,0 +1,10 @@
+---
+# main-street-station — JMRI / LCRR server
+jmri_profile_id: "Lake_Country_Railroad.3e8b1d4b"
+jmri_lcrr_repo: "ssh://git@gitea.mk-labs.cloud:2221/rblundon/LCRR.git"
+jmri_lcrr_branch: "clean-profile"
+jmri_leviton_email: "{{ leviton_email }}"
+jmri_leviton_password: "{{ leviton_password }}"
+jmri_ssh_authorized_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINnSM/9fO8rz/amqkyoGUzUKNNzzmtSXPwOCr1O9zKNO ansible"
+jmri_ssh_authorized_keys_extra:
+  - "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG6HaK4Y21UwPRbAZ986L7I9QnUdyq53114+9kO8X4bL rblundon@laptop"

ansible/host_vars/peter-pans-flight/old_vars

@@ -0,0 +1,59 @@
+---
+# file: host_vars/vm_template/vars
+# Ansible vars template for hosts created via cloning.
+
+# Supported hypervisors:
+# - Proxmox
+
+platform: "proxmox"
+
+# This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
+
+proxmox_clone_node: "fantasyland"
+
+# Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
+# Current OS offerings are:
+# - Ubuntu 
+#    - 24.04
+# - Fedora
+#    - 42
+
+vm_os_distribution: "ubuntu"
+vm_os_version: "24.04"
+
+# Current VM sizes are:
+# - Small: 2 cores, 2GB memory, 8 GiB virtual disk
+# - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
+# - Large: 4 cores, 4GB memory, 32 GiB virtual disk
+# - Large Plus: 4 cores, 4GB memory, 48 GiB virtual disk
+# - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
+# - Xlarge Plus: 4 cores, 8GB memory, 128 GiB virtual disk
+
+vm_size: "xlarge"
+
+# Proxmox storage target.
+
+vm_storage: "general"
+
+# Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
+# and backup groups. (ha_group will be factored out in the next functionality update.)
+
+ha_group: "pve03"
+proxmox_host_target: "pve03"
+
+# Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
+# sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
+
+ip_address: 10.1.71.62
+vm_mac_address: 'BC:24:11:AA:8F:3A'
+
+# Software
+# Future enhancement will allow specification of additional software to automatically deploy to
+# the VM after creation.
+
+#terraform_version: "1.11.3"
+
+# ---
+
+vm_clone_source: "{{ vm_os_distribution }}-{{ vm_os_version }}-{{ vm_size }}"
+hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn

ansible/host_vars/peter-pans-flight/vars

@@ -0,0 +1,99 @@
+---
+# file: host_vars/vm_template/vars
+# Ansible vars template for hosts created via cloning.
+
+# Supported hypervisors:
+# - Proxmox
+
+platform: "proxmox"
+
+# Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
+# Current OS offerings are:
+# - Ubuntu (clone)
+#    - 24.04
+# - Fedora (clone)
+#    - 42
+# - Talos (create)
+#    - 1.11.5
+
+vm_os_distribution: "talos"
+vm_os_version: "1.11.5"
+
+# VM ISO
+# Talos: talos-v1.11.5-nocloud-amd64.iso
+
+vm_iso_storage: "templates"
+vm_iso: "talos-v{{ vm_os_version }}-nocloud-amd64.iso"
+
+# This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
+# Comment out for VM create
+
+proxmox_clone_node: "fantasyland"
+
+# Proxmox storage target.
+
+vm_storage: "liberty-tree"
+
+# Current VM sizes are:
+# - Small: 2 cores, 2GB memory, 8 GiB virtual disk
+# - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
+# - Large: 4 cores, 4GB memory, 32 GiB virtual disk
+# - Large Plus: 4 cores, 4GB memory, 48 GiB virtual disk
+# - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
+# - Xlarge Plus: 4 cores, 8GB memory, 128 GiB virtual disk
+
+vm_size: "xlarge"
+
+# Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
+# and backup groups. (ha_group will be factored out in the next functionality update.)
+
+ha_group: "pve03"
+proxmox_host_target: "pve03"
+
+# Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
+# sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
+
+ip_address: 10.1.71.62
+vm_mac_address: 'BC:24:11:AA:8F:3A'
+
+# Software
+# Future enhancement will allow specification of additional software to automatically deploy to
+# the VM after creation.
+
+#terraform_version: "1.11.3"
+
+# ---
+
+vm_clone_source: "{{ vm_os_distribution }}-{{ vm_os_version }}-{{ vm_size }}"
+hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn
+
+# Dictionaries for VM resources
+# - cores (quantity)
+# - memory (MB)
+# - virtual disk (GiB)
+
+vm_definitions:
+  small:
+    cores: 2
+    memory: 2048
+    disk: 8
+  medium:
+    cores: 2
+    memory: 4096
+    disk: 16
+  large:
+    cores: 4
+    memory: 4096
+    disk: 32
+  large-plus:
+    cores: 4
+    memory: 4096
+    disk: 48
+  xlarge:
+    cores: 4
+    memory: 8192
+    disk: 64
+  xlarge-plus:
+    cores: 4
+    memory: 8192
+    disk: 128

ansible/host_vars/space-mountain/old_vars

@@ -0,0 +1,59 @@
+---
+# file: host_vars/vm_template/vars
+# Ansible vars template for hosts created via cloning.
+
+# Supported hypervisors:
+# - Proxmox
+
+platform: "proxmox"
+
+# This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
+
+proxmox_clone_node: "fantasyland"
+
+# Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
+# Current OS offerings are:
+# - Ubuntu 
+#    - 24.04
+# - Fedora
+#    - 42
+
+vm_os_distribution: "ubuntu"
+vm_os_version: "24.04"
+
+# Current VM sizes are:
+# - Small: 2 cores, 2GB memory, 8 GiB virtual disk
+# - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
+# - Large: 4 cores, 4GB memory, 32 GiB virtual disk
+# - Large Plus: 4 cores, 4GB memory, 48 GiB virtual disk
+# - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
+# - Xlarge Plus: 4 cores, 8GB memory, 128 GiB virtual disk
+
+vm_size: "large-plus"
+
+# Proxmox storage target.
+
+vm_storage: "general"
+
+# Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
+# and backup groups. (ha_group will be factored out in the next functionality update.)
+
+ha_group: "pve03"
+proxmox_host_target: "pve03"
+
+# Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
+# sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
+
+ip_address: 10.1.71.51
+vm_mac_address: 'BC:24:11:1A:E8:8C'
+
+# Software
+# Future enhancement will allow specification of additional software to automatically deploy to
+# the VM after creation.
+
+#terraform_version: "1.11.3"
+
+# ---
+
+vm_clone_source: "{{ vm_os_distribution }}-{{ vm_os_version }}-{{ vm_size }}"
+hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn

ansible/host_vars/space-mountain/vars

@@ -0,0 +1,99 @@
+---
+# file: host_vars/vm_template/vars
+# Ansible vars template for hosts created via cloning.
+
+# Supported hypervisors:
+# - Proxmox
+
+platform: "proxmox"
+
+# Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
+# Current OS offerings are:
+# - Ubuntu (clone)
+#    - 24.04
+# - Fedora (clone)
+#    - 42
+# - Talos (create)
+#    - 1.11.5
+
+vm_os_distribution: "talos"
+vm_os_version: "1.11.5"
+
+# VM ISO
+# Talos: talos-v1.11.5-nocloud-amd64.iso
+
+vm_iso_storage: "templates"
+vm_iso: "talos-v{{ vm_os_version }}-nocloud-amd64.iso"
+
+# This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
+# Comment out for VM create
+
+proxmox_clone_node: "fantasyland"
+
+# Proxmox storage target.
+
+vm_storage: "liberty-tree"
+
+# Current VM sizes are:
+# - Small: 2 cores, 2GB memory, 8 GiB virtual disk
+# - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
+# - Large: 4 cores, 4GB memory, 32 GiB virtual disk
+# - Large Plus: 4 cores, 4GB memory, 48 GiB virtual disk
+# - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
+# - Xlarge Plus: 4 cores, 8GB memory, 128 GiB virtual disk
+
+vm_size: "large-plus"
+
+# Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
+# and backup groups. (ha_group will be factored out in the next functionality update.)
+
+ha_group: "pve03"
+proxmox_host_target: "pve03"
+
+# Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
+# sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
+
+ip_address: 10.1.71.51
+vm_mac_address: 'BC:24:11:1A:E8:8C'
+
+# Software
+# Future enhancement will allow specification of additional software to automatically deploy to
+# the VM after creation.
+
+#terraform_version: "1.11.3"
+
+# ---
+
+vm_clone_source: "{{ vm_os_distribution }}-{{ vm_os_version }}-{{ vm_size }}"
+hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn
+
+# Dictionaries for VM resources
+# - cores (quantity)
+# - memory (MB)
+# - virtual disk (GiB)
+
+vm_definitions:
+  small:
+    cores: 2
+    memory: 2048
+    disk: 8
+  medium:
+    cores: 2
+    memory: 4096
+    disk: 16
+  large:
+    cores: 4
+    memory: 4096
+    disk: 32
+  large-plus:
+    cores: 4
+    memory: 4096
+    disk: 48
+  xlarge:
+    cores: 4
+    memory: 8192
+    disk: 64
+  xlarge-plus:
+    cores: 4
+    memory: 8192
+    disk: 128

ansible/host_vars/splash-mountain/old_vars

@@ -0,0 +1,59 @@
+---
+# file: host_vars/vm_template/vars
+# Ansible vars template for hosts created via cloning.
+
+# Supported hypervisors:
+# - Proxmox
+
+platform: "proxmox"
+
+# This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
+
+proxmox_clone_node: "fantasyland"
+
+# Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
+# Current OS offerings are:
+# - Ubuntu 
+#    - 24.04
+# - Fedora
+#    - 42
+
+vm_os_distribution: "ubuntu"
+vm_os_version: "24.04"
+
+# Current VM sizes are:
+# - Small: 2 cores, 2GB memory, 8 GiB virtual disk
+# - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
+# - Large: 4 cores, 4GB memory, 32 GiB virtual disk
+# - Large Plus: 4 cores, 4GB memory, 48 GiB virtual disk
+# - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
+# - Xlarge Plus: 4 cores, 8GB memory, 128 GiB virtual disk
+
+vm_size: "large-plus"
+
+# Proxmox storage target.
+
+vm_storage: "general"
+
+# Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
+# and backup groups. (ha_group will be factored out in the next functionality update.)
+
+ha_group: "pve03"
+proxmox_host_target: "pve03"
+
+# Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
+# sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
+
+ip_address: 10.1.71.53
+vm_mac_address: 'BC:24:11:84:D7:2F'
+
+# Software
+# Future enhancement will allow specification of additional software to automatically deploy to
+# the VM after creation.
+
+#terraform_version: "1.11.3"
+
+# ---
+
+vm_clone_source: "{{ vm_os_distribution }}-{{ vm_os_version }}-{{ vm_size }}"
+hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn

ansible/host_vars/splash-mountain/vars

@@ -0,0 +1,99 @@
+---
+# file: host_vars/vm_template/vars
+# Ansible vars template for hosts created via cloning.
+
+# Supported hypervisors:
+# - Proxmox
+
+platform: "proxmox"
+
+# Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
+# Current OS offerings are:
+# - Ubuntu (clone)
+#    - 24.04
+# - Fedora (clone)
+#    - 42
+# - Talos (create)
+#    - 1.11.5
+
+vm_os_distribution: "talos"
+vm_os_version: "1.11.5"
+
+# VM ISO
+# Talos: talos-v1.11.5-nocloud-amd64.iso
+
+vm_iso_storage: "templates"
+vm_iso: "talos-v{{ vm_os_version }}-nocloud-amd64.iso"
+
+# This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
+# Comment out for VM create
+
+proxmox_clone_node: "fantasyland"
+
+# Proxmox storage target.
+
+vm_storage: "liberty-tree"
+
+# Current VM sizes are:
+# - Small: 2 cores, 2GB memory, 8 GiB virtual disk
+# - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
+# - Large: 4 cores, 4GB memory, 32 GiB virtual disk
+# - Large Plus: 4 cores, 4GB memory, 48 GiB virtual disk
+# - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
+# - Xlarge Plus: 4 cores, 8GB memory, 128 GiB virtual disk
+
+vm_size: "large-plus"
+
+# Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
+# and backup groups. (ha_group will be factored out in the next functionality update.)
+
+ha_group: "pve03"
+proxmox_host_target: "pve03"
+
+# Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
+# sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
+
+ip_address: 10.1.71.53
+vm_mac_address: 'BC:24:11:84:D7:2F'
+
+# Software
+# Future enhancement will allow specification of additional software to automatically deploy to
+# the VM after creation.
+
+#terraform_version: "1.11.3"
+
+# ---
+
+vm_clone_source: "{{ vm_os_distribution }}-{{ vm_os_version }}-{{ vm_size }}"
+hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn
+
+# Dictionaries for VM resources
+# - cores (quantity)
+# - memory (MB)
+# - virtual disk (GiB)
+
+vm_definitions:
+  small:
+    cores: 2
+    memory: 2048
+    disk: 8
+  medium:
+    cores: 2
+    memory: 4096
+    disk: 16
+  large:
+    cores: 4
+    memory: 4096
+    disk: 32
+  large-plus:
+    cores: 4
+    memory: 4096
+    disk: 48
+  xlarge:
+    cores: 4
+    memory: 8192
+    disk: 64
+  xlarge-plus:
+    cores: 4
+    memory: 8192
+    disk: 128

ansible/host_vars/splash/vars

@@ -0,0 +1,100 @@
+---
+# file: host_vars/vm_template/vars
+# Ansible vars template for hosts creation via clone or create.
+
+# Supported hypervisors:
+# - Proxmox
+
+platform: "proxmox"
+
+# Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
+# Current OS offerings are:
+# - Ubuntu (clone)
+#    - 24.04
+# - Fedora (clone)
+#    - 42
+# - Talos (create)
+#    - 1.11.5
+
+vm_os_distribution: "talos"
+vm_os_version: "1.11.5"
+
+# VM ISO
+# Talos: talos-v1.11.5-nocloud-amd64.iso
+
+vm_iso_storage: "templates"
+vm_iso: "talos-v{{ vm_os_version }}-nocloud-amd64.iso"
+
+# This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
+# Comment out for VM create
+
+proxmox_clone_node: "fantasyland"
+
+# Proxmox storage target.
+
+vm_storage: "liberty-tree"
+
+# Current VM sizes are:
+# - Small: 2 cores, 2GB memory, 8 GiB virtual disk
+# - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
+# - Large: 4 cores, 4GB memory, 32 GiB virtual disk
+# - Large Plus: 4 cores, 4GB memory, 48 GiB virtual disk
+# - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
+# - Xlarge Plus: 4 cores, 8GB memory, 128 GiB virtual disk
+
+vm_size: "small"
+
+# Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
+# and backup groups. (ha_group will be factored out in the next functionality update.)
+
+ha_group: "pve03"
+proxmox_host_target: "tomorrowland"
+
+# Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
+# sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
+# Comment out vm_mac_address if new MAC address should be generated
+
+ip_address: 10.1.71.99
+vm_mac_address: 'BC:24:11:97:C3:AA'
+
+# Software
+# Future enhancement will allow specification of additional software to automatically deploy to
+# the VM after creation.
+
+#terraform_version: "1.11.3"
+
+# ---
+
+vm_clone_source: "{{ vm_os_distribution }}-{{ vm_os_version }}-{{ vm_size }}"
+hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn
+
+# Dictionaries for VM resources
+# - cores (quantity)
+# - memory (MB)
+# - virtual disk (GiB)
+
+vm_definitions:
+  small:
+    cores: 2
+    memory: 2048
+    disk: 8
+  medium:
+    cores: 2
+    memory: 4096
+    disk: 16
+  large:
+    cores: 4
+    memory: 4096
+    disk: 32
+  large-plus:
+    cores: 4
+    memory: 4096
+    disk: 48
+  xlarge:
+    cores: 4
+    memory: 8192
+    disk: 64
+  xlarge-plus:
+    cores: 4
+    memory: 8192
+    disk: 128

ansible/host_vars/sundial/vars

@@ -0,0 +1,50 @@
+---
+# file: host_vars/vm_template/vars
+# Ansible vars template for hosts created via cloning.
+
+# Supported hypervisors:
+# - Proxmox
+
+platform: "proxmox"
+
+# This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
+
+proxmox_clone_node: "fantasyland"
+
+# Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
+# Current OS offerings are:
+# - Fedora (42)
+# Current VM sizes are:
+# - Small: 2 cores, 2GB memory, 8 GiB virtual disk
+# - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
+# - Large: 4 cores, 4GB memory, 32 GiB virtual disk
+# - Large Plus: 4 cores, 4GB memory, 48 GiB virtual disk
+# - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
+# - Xlarge Plus: 4 cores, 8GB memory, 128 GiB virtual disk
+
+vm_clone_source: "fedora-42-small"
+
+# Proxmox storage target.
+
+vm_storage: "general"
+
+# Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
+# and backup groups. (ha_group will be factored out in the next functionality update.)
+
+ha_group: "fantasyland"
+proxmox_host_target: "fantasyland"
+
+# Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
+# sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
+
+ip_address: 10.1.71.21
+
+# Software
+# Future enhancement will allow specification of additional software to automatically deploy to
+# the VM after creation.
+
+#terraform_version: "1.11.3"
+
+# ---
+
+hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn

ansible/host_vars/tiki-room/vars

@@ -0,0 +1,59 @@
+---
+# file: host_vars/vm_template/vars
+# Ansible vars template for hosts created via cloning.
+
+# Supported hypervisors:
+# - Proxmox
+
+platform: "proxmox"
+
+# This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
+
+proxmox_clone_node: "fantasyland"
+
+# Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
+# Current OS offerings are:
+# - Ubuntu 
+#    - 24.04
+# - Fedora
+#    - 42
+
+vm_os_distribution: "ubuntu"
+vm_os_version: "24.04"
+
+# Current VM sizes are:
+# - Small: 2 cores, 2GB memory, 8 GiB virtual disk
+# - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
+# - Large: 4 cores, 4GB memory, 32 GiB virtual disk
+# - Large Plus: 4 cores, 4GB memory, 48 GiB virtual disk
+# - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
+# - Xlarge Plus: 4 cores, 8GB memory, 128 GiB virtual disk
+
+vm_size: "large"
+
+# Proxmox storage target.
+
+vm_storage: "general"
+
+# Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
+# and backup groups. (ha_group will be factored out in the next functionality update.)
+
+ha_group: "pve03"
+proxmox_host_target: "pve03"
+
+# Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
+# sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
+
+ip_address: 10.1.71.23
+# vm_mac_address: 'BC:24:11:11:BC:58'
+
+# Software
+# Future enhancement will allow specification of additional software to automatically deploy to
+# the VM after creation.
+
+#terraform_version: "1.11.3"
+
+# ---
+
+vm_clone_source: "{{ vm_os_distribution }}-{{ vm_os_version }}-{{ vm_size }}"
+hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn

ansible/host_vars/turnstile/vars

@@ -0,0 +1,27 @@
+---
+# file: host_vars/turnstile/vars
+# turnstile — Smallstep step-ca SSH Certificate Authority
+# VM provisioned by Terraform (pre-pipeline bootstrap)
+
+# ─── Network ─────────────────────────────────────────────────────────────────
+ip_address: 10.1.71.34
+
+# ─── Application ─────────────────────────────────────────────────────────────
+app_role: ssh_certificate_authority
+app_name: step-ca
+app_deployment: docker_compose
+
+# ─── step-ca Configuration ───────────────────────────────────────────────────
+stepca_hostname: turnstile.local.mk-labs.cloud
+stepca_dns_names: "turnstile.local.mk-labs.cloud,10.1.71.34"
+stepca_ssh_enabled: true
+stepca_listen_port: 9000
+
+# ─── OIDC Provisioner (Authentik) ────────────────────────────────────────────
+# Client ID and secret stored in vault
+stepca_oidc_provisioner_name: authentik
+stepca_oidc_configuration_endpoint: "https://authentik.local.mk-labs.cloud/application/o/step-ca/.well-known/openid-configuration"
+stepca_oidc_listen_address: ":10000"
+stepca_oidc_domains:
+  - "local.mk-labs.cloud"
+  - "protonmail.com"

ansible/host_vars/turnstile/vault

@@ -0,0 +1,17 @@
+$ANSIBLE_VAULT;1.1;AES256
+30383930323363386234333433636465393263613336646464666365643730386430353864616334
+3865343031653162313736656437373666636136653263360a613961646133376262633164393535
+31363434633634303035663133316230633538363936303266373931313661346563333832653032
+6239376363646530620a303231363333363233623038326165316463383662656565626534396232
+62333133623530643932643430663166373930353733363866336533643233373261333130613635
+31633337376337663833613634666436383862386431636537373363343333323932363761653366
+61353166613162346436396138316561646165303566376366323462663861616364336539313138
+30386430373566353761383636626335393463376661356666303861353564313832346333396134
+30653861363866336331336164323130623230666237356165613934333239386536373664643065
+33643139346562346663313533643433353462363665323166313364373335366665373435643935
+65303863663864393238393732303065343364306264396333376233666233346664323334336532
+38373164383835663163363137643531303163396236623565666436363261393563343133333161
+36663934303633663062346161636333396135336135616136303664636562363862353764343562
+39643430396636313139336130656363306562363430346233313530663963366238326334623135
+62383632613061316262323038303333323739303137363334626637666663666263613132336433
+65346266393733633632

ansible/host_vars/vm_template/vars

@@ -0,0 +1,104 @@
+---
+# file: host_vars/vm_template/vars
+# Ansible vars template for hosts creation via clone or create.
+
+# Supported hypervisors:
+# - Proxmox
+
+platform: "proxmox"
+
+# Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
+# Current OS offerings are:
+# - Ubuntu (clone)
+#    - 24.04
+# - Fedora (clone)
+#    - 42
+# - Talos (create)
+#    - 1.11.5
+
+vm_os_distribution: "ubuntu"
+vm_os_version: "24.04"
+
+# VM ISO
+# Talos: talos-v1.11.5-nocloud-amd64.iso
+
+vm_iso_storage: "templates"
+vm_iso: "talos-v{{ vm_os_version }}-nocloud-amd64.iso"
+
+# This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
+# Comment out for VM create
+
+proxmox_clone_node: "fantasyland"
+
+# Proxmox storage target.
+
+vm_storage: "general"
+
+# Current VM sizes are:
+# - Small: 2 cores, 2GB memory, 8 GiB virtual disk
+# - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
+# - Large: 4 cores, 4GB memory, 32 GiB virtual disk
+# - Large Plus: 4 cores, 4GB memory, 48 GiB virtual disk
+# - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
+# - Xlarge Plus: 4 cores, 8GB memory, 128 GiB virtual disk
+
+vm_size: "large-plus"
+
+# Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
+# and backup groups. (ha_group will be factored out in the next functionality update.)
+
+ha_group: "pve03"
+proxmox_host_target: "tomorrowland"
+
+# Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
+# sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
+# Comment out vm_mac_address if new MAC address should be generated
+
+ip_address: 10.1.71.249
+# vm_mac_address: 'BC:24:11:11:BC:58'
+
+# Software
+# Future enhancement will allow specification of additional software to automatically deploy to
+# the VM after creation.
+
+#terraform_version: "1.11.3"
+
+# ---
+
+vm_clone_source: "{{ vm_os_distribution }}-{{ vm_os_version }}-{{ vm_size }}"
+hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn
+
+# Dictionaries for VM resources
+# - cores (quantity)
+# - memory (MB)
+# - virtual disk (GiB)
+
+small:
+  cores: 2
+  memory: 2048
+  disk: 8
+
+medium:
+  cores: 2
+  memory: 4096
+  disk: 16
+
+large:
+  cores: 4
+  memory: 4096
+  disk: 32
+
+large-plus:
+  cores: 4
+  memory: 4096
+  disk: 48
+
+xlarge:
+  cores: 4
+  memory: 8192
+  disk: 64
+
+xlarge-plus:
+  cores: 4
+  memory: 8192
+  disk: 128

ansible/inventory.yml

@@ -0,0 +1,137 @@
+# file: inventory.yml
+proxmox:
+  hosts:
+    main-street-usa:
+      ansible_host: 10.1.71.11
+      ansible_user: wed
+      ansible_become: true
+    tomorrowland:
+    fantasyland:
+
+magic_kingdom:
+  hosts:
+    main-street-usa:
+    tomorrowland:
+    fantasyland:
+
+dns_server:
+  hosts:
+    monorail:
+
+ntp_servers:
+  hosts:
+    sundial:
+    
+load_balancers:
+  hosts:
+    lightning-lane:
+      ansible_become: true
+
+step_ca_server:
+  hosts:
+    turnstile:
+      ansible_become: true
+
+authentik_server:
+  hosts:
+    guest-relations:
+      ansible_become: true
+
+gitea_servers:
+  hosts:
+    mad-tea-party:
+      ansible_user: wed
+      ansible_become: true
+
+nextcloud_server:
+  hosts:
+    the-grid:
+      ansible_become: true
+
+semaphore_server:
+  hosts:
+    figment:
+      ansible_host: 10.1.71.37
+      ansible_user: wed
+      ansible_become: true
+
+n8n_server:
+  hosts:
+    tiki-room:
+
+astro_orbiter:
+  hosts:
+    astro-orbiter:
+
+hermes_server:
+  hosts:
+    carousel-of-progress:
+      ansible_host: 10.1.71.131
+      ansible_user: wed
+      ansible_become: true
+      ansible_ssh_private_key_file: ~/.ssh/ansible
+
+honcho_server:
+  hosts:
+    lincoln:
+      ansible_host: 10.1.71.132
+      ansible_user: wed
+      ansible_become: true
+
+paperclip_server:
+  hosts:
+    illuminations:
+      ansible_host: 10.1.71.130
+      ansible_user: wed
+      ansible_become: true
+
+multica:
+  hosts:
+    muppet-vision:
+      ansible_host: 10.1.71.136
+      ansible_user: wed
+      ansible_become: true
+
+jmri_server:
+  hosts:
+    main-street-station:
+      ansible_host: 192.168.10.40
+      ansible_user: wed
+      ansible_become: true
+
+papermc_server:
+# ansible-galaxy role install engonzal.papermc
+  hosts:
+    arcade:
+
+dev_servers:
+  hosts:
+    scrim:
+    backstage:
+      ansible_host: 10.1.71.133
+      ansible_user: wed
+      ansible_become: true
+
+# dhcp_server:
+#   hosts:
+#     matchbox:
+
+backup_servers:
+  hosts:
+    timekeeper:
+
+talos_control:
+  hosts:
+    city-hall:
+      ansible_become: true
+  vars:
+    talosctl_version: "v1.12.4"
+    talos_cluster_name: "fastpass"
+
+# matchbox_server:
+#   hosts:
+#     matchbox:
+
+terraform_server:
+  hosts:
+    infra01:

ansible/mk

@@ -0,0 +1,5 @@
+space-mountain
+splash-mountain
+big-thunder-mountain
+peter-pans-flight
+haunted-mansion

ansible/playbooks/add_dhcp_reservation.yml

@@ -0,0 +1,84 @@
+---
+- name: Bind VM MAC address to IP address on Ubiquiti
+  hosts: all
+  gather_facts: false
+
+  tasks:
+    - name: Set VM MAC address (if defined in vars)
+      delegate_to: "localhost"
+      community.proxmox.proxmox_kvm:
+        api_user: "{{ proxmox_user }}"
+        api_password: "{{ proxmox_password }}"
+        api_host: "{{ proxmox_host }}"
+        node: "{{ proxmox_clone_node }}"
+        vmid: "{{ vm_id }}"
+        net:
+          net0: "virtio={{ vm_mac_address }},bridge=vmbr0,firewall=1"
+        update: true
+        update_unsafe: true
+      when: vm_mac_address is defined
+
+    - name: Get VM MAC address from Proxmox (not defined in vars)
+      when: vm_mac_address is not defined
+      block:
+        - name: Retrieve information about specific VM by name and get current configuration
+          delegate_to: "localhost"
+          community.proxmox.proxmox_vm_info:
+            api_user: "{{ proxmox_user }}"
+            api_password: "{{ proxmox_password }}"
+            api_host: "{{ proxmox_host }}"
+            name: "{{ inventory_hostname }}"
+            config: current
+          register: proxmox_vm_info
+
+        - name: Extract net0 information
+          ansible.builtin.set_fact:
+            vm_net0: "{{ proxmox_vm_info.proxmox_vms[0].config.net0 }}"
+
+        - name: Extract MAC address using regex
+          ansible.builtin.set_fact:
+            vm_mac_address: "{{ vm_net0 | regex_search('([0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}') }}"
+
+    - name: Assign VM MAC address to IP address on Ubiquiti with Terraform
+      when: vm_mac_address is defined
+      delegate_to: "{{ groups['terraform_server'][0] }}"
+      block:
+        - name: Create a directory if it does not exist
+          ansible.builtin.file:
+            path: "~/homelab/terraform/unifi-dhcp"
+            state: directory
+            mode: '0755'
+
+        - name: Create Unifi provider file from template
+          ansible.builtin.template:
+            src: templates/unifi_provider.tf.j2
+            dest: ~/homelab/terraform/unifi-dhcp/provider.tf
+            mode: '0644'
+
+        - name: Create Unifi user (host) file from template
+          ansible.builtin.template:
+            src: templates/unifi-user.tf.j2
+            dest: ~/homelab/terraform/unifi-dhcp/{{ inventory_hostname }}.tf
+            mode: '0644'
+
+        - name: Configure Unifi via Terraform
+          community.general.terraform:
+            project_path: ~/homelab/terraform/unifi-dhcp
+            state: present
+            force_init: true
+
+    # - name: Add line to hosts file
+    #   delegate_to: "{{ groups['dhcp_server'][0] }}"
+    #   become: true
+    #   ansible.builtin.lineinfile:
+    #     path: /etc/dnsmasq.d/hosts.conf
+    #     regexp: "# {{ inventory_hostname }}$"
+    #     line: "dhcp-host={{ vm_mac_address | lower }},{{ ip_address }} # {{ inventory_hostname }}"
+    #     state: present
+    # # restart dnsmasq service
+    # - name: Restart service dnsmasq
+    #   delegate_to: "{{ groups['dhcp_server'][0] }}"
+    #   become: true
+    #   ansible.builtin.service:
+    #     name: dnsmasq
+    #     state: restarted

ansible/playbooks/add_dns_entry.yml

@@ -0,0 +1,13 @@
+---
+# Replacement for add_technitium_dns_entry.yml
+# This playbook uses the modular task file approach
+
+- name: Add DNS entry using Technitium DNS
+  hosts: all
+  gather_facts: false
+
+  tasks:
+    - name: Include Technitium DNS entry task
+      ansible.builtin.include_tasks: tasks/add_technitium_dns_entry.yml
+      vars:
+        host_name: "{{ inventory_hostname }}"

ansible/playbooks/add_service_route.yml

@@ -0,0 +1,95 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/add_service_route.yml
+# DESCRIPTION: Ensures all services in the Traefik dynamic config directory
+#              are routed and have DNS CNAME records on monorail.
+#
+#              1. Syncs boilerplates/traefik/dynamic/ to lightning-lane
+#              2. Scans the directory for service configs
+#              3. Extracts all hostnames from Host() rules (supports multi-host)
+#              4. Creates CNAME records for each hostname -> lightning-lane
+#
+# PREREQUISITES:
+#   - Service dynamic config YAML committed to boilerplates/traefik/dynamic/
+#   - vault_technitium_api_key defined in group_vars/all/vault
+#
+# USAGE:
+#   ansible-playbook -i inventory.yml playbooks/add_service_route.yml
+# ------------------------------------------------------------------------------
+
+- name: Sync Traefik routes and ensure DNS records
+  hosts: localhost
+  connection: local
+  gather_facts: false
+
+  vars:
+    base_domain: "local.mk-labs.cloud"
+    dns_server: "monorail"
+    traefik_host: "lightning-lane.local.mk-labs.cloud"
+    traefik_user: "wed"
+    traefik_dynamic_path: "/opt/docker/traefik/dynamic/"
+    dynamic_config_dir: "{{ playbook_dir }}/../../boilerplates/traefik/dynamic"
+
+    # Files in the dynamic directory that are NOT service routes
+    exclude_configs:
+      - default.yml
+
+  tasks:
+    # ── Step 1: Sync dynamic config to lightning-lane ──
+    - name: Sync Traefik dynamic configuration to lightning-lane
+      ansible.builtin.shell: >
+        rsync -av --delete
+        {{ dynamic_config_dir }}/
+        {{ traefik_user }}@{{ traefik_host }}:{{ traefik_dynamic_path }}
+      register: sync_result
+      changed_when: "'sending incremental file list' in sync_result.stdout"
+
+    # ── Step 2: Discover hostnames from Traefik router rules ──
+    - name: Find all dynamic config files
+      ansible.builtin.find:
+        paths: "{{ dynamic_config_dir }}"
+        patterns: "*.yml"
+      register: config_files
+
+    - name: Read config files
+      ansible.builtin.slurp:
+        src: "{{ item.path }}"
+      register: slurped_configs
+      loop: "{{ config_files.files }}"
+      when: item.path | basename not in exclude_configs
+
+    - name: Extract all hostnames from Host() rules
+      ansible.builtin.set_fact:
+        hostnames: >-
+          {% set hosts = [] -%}
+          {% for result in slurped_configs.results if result.content is defined -%}
+            {% set content = result.content | b64decode -%}
+            {% for match in content | regex_findall('Host\(`([^`]+)`\)') -%}
+              {% for h in match.split(' || ') -%}
+                {% set h = h | regex_replace('`', '') | trim -%}
+                {% if h.endswith('.local.mk-labs.cloud') and h not in hosts -%}
+                  {% set _ = hosts.append(h) -%}
+                {% endif -%}
+              {% endfor -%}
+            {% endfor -%}
+          {% endfor -%}
+          {{ hosts | unique | list }}
+
+    - name: Display hostnames to create
+      ansible.builtin.debug:
+        msg: "Hostnames found: {{ hostnames }}"
+
+    # ── Step 3: Create DNS CNAME records ──
+    - name: Create DNS CNAME record for each hostname
+      effectivelywild.technitium_dns.technitium_dns_add_record:
+        api_url: "http://{{ dns_server }}.{{ base_domain }}"
+        api_token: "{{ vault_technitium_api_key }}"
+        zone: "{{ base_domain }}"
+        name: "{{ item }}"
+        type: "CNAME"
+        cname: "lightning-lane.{{ base_domain }}"
+        ttl: 360
+        validate_certs: false
+      loop: "{{ hostnames }}"
+      loop_control:
+        label: "{{ item }}"

ansible/playbooks/add_technitium_dns_entry.yml.backup

@@ -0,0 +1,18 @@
+---
+- name: Add entry to Technitium DNS
+  hosts: all
+  gather_facts: false
+
+  tasks:
+    - name: "Create DNS entry for {{ inventory_hostname }}"
+      delegate_to: localhost
+      effectivelywild.technitium_dns.technitium_dns_add_record:
+        api_url: "http://{{ dns_server }}.{{ base_domain }}"
+        api_token: "{{ vault_technitium_api_key }}"
+        zone: "{{ base_domain }}"
+        name: "{{ inventory_hostname }}.{{ base_domain }}"
+        type: "A"
+        ipAddress: "{{ ip_address }}"
+        ptr: true
+        ttl: 360
+#        validate_certs: false

ansible/playbooks/build_paperclip_image.yml

@@ -0,0 +1,40 @@
+---
+- name: Build Paperclip Docker image from source
+  hosts: illuminations
+  become: true
+  vars:
+    paperclip_repo_dir: /opt/paperclip-repo
+    paperclip_image_name: paperclip-local
+    paperclip_image_tag: latest
+
+  tasks:
+    - name: Install git (if not present)
+      apt:
+        name: git
+        state: present
+
+    - name: Clone or update Paperclip repository
+      git:
+        repo: https://github.com/paperclipai/paperclip.git
+        dest: "{{ paperclip_repo_dir }}"
+        update: yes
+      register: git_clone
+
+    - name: Build Paperclip Docker image
+      shell: |
+        cd {{ paperclip_repo_dir }}
+        docker build --target production -t {{ paperclip_image_name }}:{{ paperclip_image_tag }} .
+      register: docker_build
+      changed_when: "'Successfully tagged' in docker_build.stdout"
+
+    - name: Show build result
+      debug:
+        msg: "Built image: {{ paperclip_image_name }}:{{ paperclip_image_tag }}"
+
+    - name: Show image details
+      shell: docker images {{ paperclip_image_name }}:{{ paperclip_image_tag }}
+      register: image_info
+
+    - name: Image info
+      debug:
+        msg: "{{ image_info.stdout_lines }}"

ansible/playbooks/configure_proxmox_oidc.yml

@@ -0,0 +1,20 @@
+---
+# ansible/playbooks/configure_proxmox_oidc.yml
+#
+# Configures Proxmox OIDC authentication with Authentik.
+# Only targets one node since realm config is cluster-wide.
+#
+# Usage:
+#   cd ansible
+#   ansible-playbook -i inventory.yml playbooks/configure_proxmox_oidc.yml
+#
+# To also set up ACL entries for your user:
+#   ansible-playbook -i inventory.yml playbooks/configure_proxmox_oidc.yml \
+#     -e '{"proxmox_oidc_acl_entries": [{"path": "/", "user": "rblundon@authentik", "role": "Administrator"}]}'
+
+- name: Configure Proxmox Authentik OIDC
+  hosts: main-street-usa
+  become: true
+  roles:
+    - role: proxmox
+      tags: [proxmox-oidc]

ansible/playbooks/configure_unbound.yml

@@ -0,0 +1,37 @@
+---
+- name: Master playbook to install and configure unbound
+  hosts: unbound_servers
+  become: true
+
+  tasks:
+    - name: Permit traffic in default zone for dns service
+      ansible.posix.firewalld:
+        service: dns
+        permanent: true
+        immediate: true
+        state: enabled
+
+    - name: Create the directory
+      ansible.builtin.file:
+        path: /etc/systemd/resolved.conf.d
+        state: directory
+        mode: '0755'
+        owner: root
+        group: root
+
+    - name: Put `unbound.conf` in the correct place
+      ansible.builtin.copy:
+        src: ../roles/common/files/unbound.conf
+        dest: /etc/systemd/resolved.conf.d/unbound.conf
+        mode: '0644'
+        owner: root
+        group: root
+
+    - name: Restart service systemd-resolved
+      ansible.builtin.service:
+        name: systemd-resolved
+        state: restarted
+
+    - name: Install unbound via role
+      ansible.builtin.import_role:
+        name: Anthony25.unbound

ansible/playbooks/create_vm.yml

@@ -0,0 +1,23 @@
+---
+- name: Master playbook to create VM
+  hosts: all
+  gather_facts: false
+
+- name: Proxmox Create VM Playbook
+  ansible.builtin.import_playbook: proxmox_create_vm.yml
+
+- name: Set cloud-init network
+  ansible.builtin.import_playbook: populate_cloud_init.yml
+  when: vm_os_distribution == "ubuntu"
+
+- name: DNS Playbook
+  ansible.builtin.import_playbook: add_dns_entry.yml
+
+- name: DHCP Playbook
+  ansible.builtin.import_playbook: add_dhcp_reservation.yml
+
+- name: Start VM Playbook
+  ansible.builtin.import_playbook: proxmox_start_vm.yml
+
+- name: Set Hostname Playbook
+  ansible.builtin.import_playbook: set_hostname.yml

ansible/playbooks/create_vm_from_clone.yml

@@ -6,11 +6,15 @@
 - name: Proxmox Clone VM Playbook
   ansible.builtin.import_playbook: proxmox_clone_vm.yml
 
+- name: Set cloud-init network
+  ansible.builtin.import_playbook: populate_cloud_init.yml
+  when: vm_os_distribution == "ubuntu"
+
 - name: DNS Playbook
   ansible.builtin.import_playbook: add_dns_entry.yml
 
 - name: DHCP Playbook
-  ansible.builtin.import_playbook: add_dhcp_entry.yml
+  ansible.builtin.import_playbook: add_dhcp_reservation.yml
 
 - name: Start VM Playbook
   ansible.builtin.import_playbook: proxmox_start_vm.yml

ansible/playbooks/day0_baseline.yml

@@ -0,0 +1,6 @@
+---
+- name: Apply day0 baseline
+  hosts: "{{ target | default('all') }}"
+  become: true
+  roles:
+    - day0-baseline

ansible/playbooks/day0_expand_root_lv.yml

@@ -0,0 +1,32 @@
+---
+# ============================================================================
+# day0_expand_root_lv.yml
+# ----------------------------------------------------------------------------
+# Reclaims unallocated PE on the root volume group, extending the root LV
+# to fill the VG and resizing the underlying filesystem (ext4 or xfs).
+#
+# Belongs to the day0 host-provisioning lifecycle. The Ubuntu Server
+# autoinstall template ships with the root LV at ~half the disk size by
+# default; this playbook is the canonical one-shot fix-up for that.
+#
+# Idempotent and safe to re-run. Hosts without LVM are no-op'd cleanly.
+#
+# Opt-out: set `expand_root_lv_skip: true` in host_vars/<host>.yml for
+# hosts where free PE should NOT be claimed by root (e.g. hosts with a
+# planned second LV in the same VG for application data).
+#
+# Usage:
+#   ansible-playbook playbooks/day0_expand_root_lv.yml
+#   ansible-playbook playbooks/day0_expand_root_lv.yml -e target=lincoln
+#   ansible-playbook playbooks/day0_expand_root_lv.yml -e target=honcho_server
+# ============================================================================
+
+- name: Expand root logical volume to fill VG
+  hosts: "{{ target | default('all') }}"
+  become: true
+  gather_facts: true
+  tasks:
+    - name: Apply expand_root_lv role unless host opts out
+      ansible.builtin.include_role:
+        name: expand_root_lv
+      when: not (expand_root_lv_skip | default(false) | bool)

ansible/playbooks/day0_linux_baseline.yml

@@ -0,0 +1,23 @@
+---
+# ============================================================================
+# day0_linux_baseline.yml
+# ----------------------------------------------------------------------------
+# Applies the mk-labs Linux baseline (linux-baseline role) to one or more
+# hosts. Idempotent and safe to re-run.
+#
+# Usage:
+#   ansible-playbook playbooks/day0_linux_baseline.yml
+#   ansible-playbook playbooks/day0_linux_baseline.yml -e target=figment
+#   ansible-playbook playbooks/day0_linux_baseline.yml -e target=semaphore_server
+#
+# To trigger an opt-in full system upgrade:
+#   ansible-playbook playbooks/day0_linux_baseline.yml \
+#     -e target=figment -e 'baseline_features={"full_upgrade": true}'
+# ============================================================================
+
+- name: Apply mk-labs Linux baseline
+  hosts: "{{ target | default('all') }}"
+  become: true
+  gather_facts: true
+  roles:
+    - linux-baseline

ansible/playbooks/day0_provision.yml

@@ -0,0 +1,30 @@
+---
+# ============================================================================
+# day0_provision.yml
+# ----------------------------------------------------------------------------
+# Umbrella day0 playbook. Runs the full host-provisioning lifecycle in
+# the correct order against newly-built VMs, so the operator runs ONE
+# command per new host rather than chaining day0 steps manually.
+#
+# Order matters:
+#   1. linux-baseline      — timezone, NTP, packages, SSH hardening, jarvis user
+#   2. expand_root_lv      — reclaim PE left unallocated by the Ubuntu
+#                            autoinstall template default
+#
+# Idempotent: every step is safe to re-run. Suitable to apply periodically
+# from Semaphore as a baseline-drift check.
+#
+# Usage:
+#   ansible-playbook playbooks/day0_provision.yml -e target=lincoln
+#   ansible-playbook playbooks/day0_provision.yml -e target=honcho_server
+#
+# For finer control over a single phase, the constituent playbooks are:
+#   playbooks/day0_linux_baseline.yml
+#   playbooks/day0_expand_root_lv.yml
+# ============================================================================
+
+- name: Import day0 linux baseline
+  ansible.builtin.import_playbook: day0_linux_baseline.yml
+
+- name: Import day0 expand root LV
+  ansible.builtin.import_playbook: day0_expand_root_lv.yml

ansible/playbooks/day1_configure_netbox_catalog.yml

@@ -0,0 +1,400 @@
+---
+# ansible/playbooks/day1_configure_netbox_catalog.yml
+#
+# Creates the tag taxonomy and custom fields in NetBox for service catalog
+# documentation. Run once (idempotent — uses name-based checks).
+#
+# Usage (from ansible/ directory):
+#   ansible-playbook playbooks/day1_configure_netbox_catalog.yml
+#
+# Requires:
+#   - vault_netbox_token in Ansible Vault
+#   - netbox reachable at http://fire-station.local.mk-labs.cloud
+
+- name: Configure NetBox service catalog taxonomy
+  hosts: localhost
+  gather_facts: false
+
+  vars_files:
+    - "{{ playbook_dir }}/../group_vars/all/vault"
+
+  vars:
+    netbox_url: "http://fire-station.local.mk-labs.cloud"
+    netbox_token: "{{ vault_netbox_token }}"
+    netbox_api: "{{ netbox_url }}/api"
+    headers:
+      Authorization: "Token {{ netbox_token }}"
+      Content-Type: "application/json"
+      Accept: "application/json"
+
+    # ── Tag definitions ───────────────────────────────────────────────────
+    tags:
+      # Infrastructure type
+      - name: k8s
+        slug: k8s
+        color: "2196f3"   # blue
+        description: "Workload running in the fastpass Kubernetes cluster"
+      - name: vm
+        slug: vm
+        color: "4caf50"   # green
+        description: "Traditional VM or LXC on Proxmox"
+      # Service tier
+      - name: platform
+        slug: platform
+        color: "9c27b0"   # purple
+        description: "Platform/infrastructure service (not user-facing)"
+      - name: application
+        slug: application
+        color: "ff9800"   # orange
+        description: "User-facing application workload"
+      # Service categories
+      - name: monitoring
+        slug: monitoring
+        color: "607d8b"   # grey
+        description: "Metrics, logging, alerting"
+      - name: auth
+        slug: auth
+        color: "607d8b"
+        description: "Authentication and SSO"
+      - name: gitops
+        slug: gitops
+        color: "607d8b"
+        description: "GitOps and CI/CD"
+      - name: dashboard
+        slug: dashboard
+        color: "607d8b"
+        description: "Dashboard and portal services"
+      - name: storage
+        slug: storage
+        color: "607d8b"
+        description: "Storage and file services"
+      - name: dns
+        slug: dns
+        color: "607d8b"
+        description: "DNS and name resolution"
+      - name: automation
+        slug: automation
+        color: "607d8b"
+        description: "Automation and orchestration"
+      - name: networking
+        slug: networking
+        color: "607d8b"
+        description: "Network infrastructure services"
+      - name: inference
+        slug: inference
+        color: "607d8b"
+        description: "AI/ML inference workloads"
+
+    # ── Custom field definitions ──────────────────────────────────────────
+    # object_types use app_label.model format
+    custom_fields:
+      - name: hostnames
+        label: Hostnames
+        type: longtext
+        object_types:
+          - ipam.ipaddress
+          - virtualization.virtualmachine
+        description: "All DNS names that resolve to this service (comma-separated)"
+        ui_visible: always
+        ui_editable: yes
+
+      - name: namespace
+        label: Namespace
+        type: text
+        object_types:
+          - ipam.ipaddress
+          - virtualization.virtualmachine
+        description: "Kubernetes namespace (blank for VM-based services)"
+        ui_visible: always
+        ui_editable: yes
+
+      - name: managed_by
+        label: Managed By
+        type: select
+        object_types:
+          - ipam.ipaddress
+          - virtualization.virtualmachine
+        description: "How this service is managed"
+        choices:
+          - ArgoCD
+          - Ansible
+          - Manual
+        ui_visible: always
+        ui_editable: yes
+
+      - name: thematic_name
+        label: Thematic Name
+        type: text
+        object_types:
+          - ipam.ipaddress
+          - virtualization.virtualmachine
+        description: "Disney/Magic Kingdom thematic hostname for this service"
+        ui_visible: always
+        ui_editable: yes
+
+  tasks:
+
+    # ── Tags ─────────────────────────────────────────────────────────────
+
+    - name: Fetch existing tags
+      ansible.builtin.uri:
+        url: "{{ netbox_api }}/extras/tags/?limit=200"
+        method: GET
+        headers: "{{ headers }}"
+        return_content: true
+        status_code: 200
+      register: existing_tags_response
+
+    - name: Set existing tag slugs fact
+      ansible.builtin.set_fact:
+        existing_tag_slugs: "{{ existing_tags_response.json.results | map(attribute='slug') | list }}"
+
+    - name: Create tags
+      ansible.builtin.uri:
+        url: "{{ netbox_api }}/extras/tags/"
+        method: POST
+        headers: "{{ headers }}"
+        body_format: json
+        body:
+          name: "{{ item.name }}"
+          slug: "{{ item.slug }}"
+          color: "{{ item.color }}"
+          description: "{{ item.description }}"
+        status_code: 201
+      loop: "{{ tags }}"
+      when: item.slug not in existing_tag_slugs
+      register: tag_creation
+      changed_when: tag_creation.status == 201
+
+    # ── Custom Fields ─────────────────────────────────────────────────────
+
+    - name: Fetch existing custom fields
+      ansible.builtin.uri:
+        url: "{{ netbox_api }}/extras/custom-fields/?limit=200"
+        method: GET
+        headers: "{{ headers }}"
+        return_content: true
+        status_code: 200
+      register: existing_cf_response
+
+    - name: Set existing custom field names fact
+      ansible.builtin.set_fact:
+        existing_cf_names: "{{ existing_cf_response.json.results | map(attribute='name') | list }}"
+
+    - name: Create custom field — hostnames
+      ansible.builtin.uri:
+        url: "{{ netbox_api }}/extras/custom-fields/"
+        method: POST
+        headers: "{{ headers }}"
+        body_format: json
+        body:
+          name: hostnames
+          label: Hostnames
+          type: longtext
+          object_types: "{{ custom_fields[0].object_types }}"
+          description: "{{ custom_fields[0].description }}"
+          ui_visible: always
+          ui_editable: yes
+        status_code: 201
+      when: "'hostnames' not in existing_cf_names"
+      register: cf_hostnames
+      changed_when: cf_hostnames.status == 201
+
+    - name: Create custom field — namespace
+      ansible.builtin.uri:
+        url: "{{ netbox_api }}/extras/custom-fields/"
+        method: POST
+        headers: "{{ headers }}"
+        body_format: json
+        body:
+          name: namespace
+          label: Namespace
+          type: text
+          object_types: "{{ custom_fields[1].object_types }}"
+          description: "{{ custom_fields[1].description }}"
+          ui_visible: always
+          ui_editable: yes
+        status_code: 201
+      when: "'namespace' not in existing_cf_names"
+      register: cf_namespace
+      changed_when: cf_namespace.status == 201
+
+    - name: Create choice set for managed_by field
+      ansible.builtin.uri:
+        url: "{{ netbox_api }}/extras/custom-field-choice-sets/"
+        method: POST
+        headers: "{{ headers }}"
+        body_format: json
+        body:
+          name: managed-by-choices
+          extra_choices:
+            - - ArgoCD
+              - ArgoCD
+            - - Ansible
+              - Ansible
+            - - Manual
+              - Manual
+        status_code: [201, 400]
+      register: choice_set
+      changed_when: choice_set.status == 201
+
+    - name: Fetch choice set ID
+      ansible.builtin.uri:
+        url: "{{ netbox_api }}/extras/custom-field-choice-sets/?name=managed-by-choices"
+        method: GET
+        headers: "{{ headers }}"
+        return_content: true
+        status_code: 200
+      register: choice_set_response
+
+    - name: Create custom field — managed_by
+      ansible.builtin.uri:
+        url: "{{ netbox_api }}/extras/custom-fields/"
+        method: POST
+        headers: "{{ headers }}"
+        body_format: json
+        body:
+          name: managed_by
+          label: Managed By
+          type: select
+          object_types: "{{ custom_fields[2].object_types }}"
+          description: "{{ custom_fields[2].description }}"
+          choice_set: "{{ choice_set_response.json.results[0].id }}"
+          ui_visible: always
+          ui_editable: yes
+        status_code: 201
+      when: "'managed_by' not in existing_cf_names"
+      register: cf_managed_by
+      changed_when: cf_managed_by.status == 201
+
+    - name: Create custom field — thematic_name
+      ansible.builtin.uri:
+        url: "{{ netbox_api }}/extras/custom-fields/"
+        method: POST
+        headers: "{{ headers }}"
+        body_format: json
+        body:
+          name: thematic_name
+          label: Thematic Name
+          type: text
+          object_types: "{{ custom_fields[3].object_types }}"
+          description: "{{ custom_fields[3].description }}"
+          ui_visible: always
+          ui_editable: yes
+        status_code: 201
+      when: "'thematic_name' not in existing_cf_names"
+      register: cf_thematic_name
+      changed_when: cf_thematic_name.status == 201
+
+    # ── Provisioning Pipeline Fields ──────────────────────────────────────
+    # These fields drive the NetBox → n8n → Terraform pipeline.
+    # proxmox_datastore already exists — PATCH it to add utilidor choice.
+    # data_disk_enabled and data_disk_size_gb are new POSTs.
+
+    - name: Fetch proxmox_datastore field ID
+      ansible.builtin.uri:
+        url: "{{ netbox_api }}/extras/custom-fields/?name=proxmox_datastore"
+        method: GET
+        headers: "{{ headers }}"
+        return_content: true
+        status_code: 200
+      register: proxmox_datastore_cf_response
+
+    - name: Fetch proxmox-datastore-choices choice set ID
+      ansible.builtin.uri:
+        url: "{{ netbox_api }}/extras/custom-field-choice-sets/?name=proxmox-datastore-choices"
+        method: GET
+        headers: "{{ headers }}"
+        return_content: true
+        status_code: 200
+      register: datastore_choice_set_response
+
+    - name: Create proxmox-datastore-choices choice set if missing
+      ansible.builtin.uri:
+        url: "{{ netbox_api }}/extras/custom-field-choice-sets/"
+        method: POST
+        headers: "{{ headers }}"
+        body_format: json
+        body:
+          name: proxmox-datastore-choices
+          extra_choices:
+            - - liberty-tree
+              - liberty-tree
+            - - utilidor
+              - utilidor
+        status_code: [201, 400]
+      register: datastore_choice_set_create
+      changed_when: datastore_choice_set_create.status == 201
+      when: datastore_choice_set_response.json.count == 0
+
+    - name: Re-fetch proxmox-datastore-choices choice set ID after possible creation
+      ansible.builtin.uri:
+        url: "{{ netbox_api }}/extras/custom-field-choice-sets/?name=proxmox-datastore-choices"
+        method: GET
+        headers: "{{ headers }}"
+        return_content: true
+        status_code: 200
+      register: datastore_choice_set_response
+
+    - name: Patch proxmox_datastore field to use choice set with utilidor
+      ansible.builtin.uri:
+        url: "{{ netbox_api }}/extras/custom-fields/{{ proxmox_datastore_cf_response.json.results[0].id }}/"
+        method: PATCH
+        headers: "{{ headers }}"
+        body_format: json
+        body:
+          choice_set: "{{ datastore_choice_set_response.json.results[0].id }}"
+        status_code: 200
+      when: proxmox_datastore_cf_response.json.count > 0
+      register: cf_datastore_patch
+      changed_when: cf_datastore_patch.status == 200
+
+    - name: Create custom field — data_disk_enabled
+      ansible.builtin.uri:
+        url: "{{ netbox_api }}/extras/custom-fields/"
+        method: POST
+        headers: "{{ headers }}"
+        body_format: json
+        body:
+          name: data_disk_enabled
+          label: Data Disk Enabled
+          type: boolean
+          object_types:
+            - virtualization.virtualmachine
+          description: "When true, Terraform provisions a second disk for application data storage."
+          default: false
+          ui_visible: always
+          ui_editable: yes
+        status_code: 201
+      when: "'data_disk_enabled' not in existing_cf_names"
+      register: cf_data_disk_enabled
+      changed_when: cf_data_disk_enabled.status == 201
+
+    - name: Create custom field — data_disk_size_gb
+      ansible.builtin.uri:
+        url: "{{ netbox_api }}/extras/custom-fields/"
+        method: POST
+        headers: "{{ headers }}"
+        body_format: json
+        body:
+          name: data_disk_size_gb
+          label: Data Disk Size (GB)
+          type: integer
+          object_types:
+            - virtualization.virtualmachine
+          description: "Size in GB for the optional second data disk. Only used when data_disk_enabled is true."
+          ui_visible: always
+          ui_editable: yes
+        status_code: 201
+      when: "'data_disk_size_gb' not in existing_cf_names"
+      register: cf_data_disk_size
+      changed_when: cf_data_disk_size.status == 201
+
+    # ── Summary ───────────────────────────────────────────────────────────
+
+    - name: Summary
+      ansible.builtin.debug:
+        msg:
+          - "Tags created: {{ tag_creation.results | selectattr('status', 'equalto', 201) | list | length }}"
+          - "Tags skipped (already exist): {{ tag_creation.results | selectattr('skipped', 'defined') | list | length }}"
+          - "Custom fields configured: hostnames, namespace, managed_by, thematic_name, proxmox_datastore (patched), data_disk_enabled, data_disk_size_gb"

ansible/playbooks/day1_deploy_authentik.yml

@@ -0,0 +1,18 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/deploy_authentik.yml
+# DESCRIPTION: Deploys Authentik identity provider on guest-relations.
+#              Installs Docker and configures Authentik with PostgreSQL and Redis.
+#
+# USAGE:
+#   ansible-playbook -i inventory.yml playbooks/deploy_authentik.yml
+# ------------------------------------------------------------------------------
+
+- name: Deploy Authentik identity provider
+  hosts: authentik_server
+  become: true
+
+  roles:
+    - common
+    - docker-host
+    - authentik

ansible/playbooks/day1_deploy_gitea.yml

@@ -0,0 +1,23 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: ansible/playbooks/day1_deploy_gitea.yml
+# DESCRIPTION: Deploys Gitea + PostgreSQL on mad-tea-party (10.1.71.129)
+#              Role chain: common → docker-host → gitea
+#
+# USAGE:
+#   ansible-playbook -i ansible/inventory.yml ansible/playbooks/day1_deploy_gitea.yml
+#
+# SECRETS REQUIRED IN VAULT (group_vars/all/vault):
+#   vault_gitea_db_password
+# ------------------------------------------------------------------------------
+
+- name: Deploy Gitea on mad-tea-party
+  hosts: gitea_servers
+  become: true
+
+  vars:
+    gitea_db_password: "{{ vault_gitea_db_password }}"
+
+  roles:
+    - role: docker-host
+    - role: gitea

ansible/playbooks/day1_deploy_hermes.yml

@@ -0,0 +1,79 @@
+---
+# =============================================================================
+# day1_deploy_hermes.yml
+# Deploy Hermes Agent (Nous Research) on carousel-of-progress (10.1.71.131)
+#
+# FIRST-RUN WORKFLOW:
+#   1. Run this playbook:
+#        ansible-playbook playbooks/day1_deploy_hermes.yml
+#
+#   2. SSH to the host and run the setup wizard as the hermes user:
+#        ssh wed@carousel-of-progress.local.mk-labs.cloud
+#        sudo -u hermes hermes setup
+#
+#   3. Once configured, start and verify the service:
+#        sudo systemctl start hermes
+#        sudo systemctl status hermes
+#        sudo journalctl -u hermes -f
+#
+# VARIABLES:
+#   hermes_skip_browser: true   — set to skip Playwright/Chromium install
+#                                 (saves ~300MB if browser automation not needed)
+# =============================================================================
+
+- name: Deploy Hermes Agent on carousel-of-progress
+  hosts: carousel-of-progress
+  gather_facts: true
+
+  pre_tasks:
+    - name: Verify target is carousel-of-progress
+      ansible.builtin.assert:
+        that:
+          - inventory_hostname == "carousel-of-progress"
+        fail_msg: >
+          This playbook is scoped to carousel-of-progress only.
+          Got: {{ inventory_hostname }}
+
+    - name: Confirm OS is Ubuntu
+      ansible.builtin.assert:
+        that:
+          - ansible_distribution == "Ubuntu"
+        fail_msg: >
+          This playbook requires Ubuntu. Found: {{ ansible_distribution }}.
+          (If running Fedora, swap apt tasks for dnf and adjust Playwright deps.)
+
+  roles:
+    - role: hermes
+      vars:
+        hermes_skip_browser: false   # set true to skip Chromium install
+
+  post_tasks:
+    - name: Verify hermes binary is accessible system-wide
+      ansible.builtin.command: hermes --version
+      register: hermes_version_check
+      changed_when: false
+      failed_when: hermes_version_check.rc != 0
+
+    - name: Print hermes version
+      ansible.builtin.debug:
+        msg: "{{ hermes_version_check.stdout }}"
+
+    - name: Print post-install instructions
+      ansible.builtin.debug:
+        msg:
+          - "============================================================"
+          - "Hermes installed on carousel-of-progress (10.1.71.131)"
+          - "============================================================"
+          - "Next steps:"
+          - "  1. SSH to the host:"
+          - "       ssh wed@carousel-of-progress.local.mk-labs.cloud"
+          - "  2. Run the setup wizard as the hermes user:"
+          - "       sudo -u hermes hermes setup"
+          - "  3. After config, start the service:"
+          - "       sudo systemctl start hermes"
+          - "  4. Verify:"
+          - "       sudo systemctl status hermes"
+          - "       sudo journalctl -u hermes -f"
+          - "============================================================"
+          - "Service is ENABLED but NOT STARTED — config required first."
+          - "============================================================"

ansible/playbooks/day1_deploy_hermes_docker.yml

@@ -0,0 +1,7 @@
+---
+# Run after day1_deploy_nginx_proxy.yml (the role publishes its vhost through it).
+- name: Deploy Hermes Agent (Docker)
+  hosts: illuminations
+  become: true
+  roles:
+    - hermes_docker

ansible/playbooks/day1_deploy_honcho.yml

@@ -0,0 +1,18 @@
+---
+# ============================================================================
+# day1_deploy_honcho.yml
+# ----------------------------------------------------------------------------
+# Deploys Honcho + pgvector PostgreSQL on the `lincoln` host. Assumes day0
+# host provisioning (linux-baseline + expand_root_lv) is already complete.
+#
+# Run via:
+#   ansible-playbook -i inventory.yml playbooks/day0_provision.yml -e target=lincoln
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_honcho.yml
+# ============================================================================
+
+- name: Deploy Honcho on lincoln
+  hosts: honcho_server
+  become: true
+  gather_facts: true
+  roles:
+    - honcho

ansible/playbooks/day1_deploy_jmri.yml

@@ -0,0 +1,24 @@
+---
+# ============================================================================
+# day1_deploy_jmri.yml
+# ----------------------------------------------------------------------------
+# Deploys JMRI JmriFaceless headless server on main-street-station.
+# Applies linux-baseline first, then the jmri role.
+#
+# Usage:
+#   ansible-playbook playbooks/day1_deploy_jmri.yml
+#   ansible-playbook playbooks/day1_deploy_jmri.yml -e target=main-street-station
+#
+# Prerequisites:
+#   1. Host is in inventory under jmri_server group
+#   2. jmri_profile_id is set in host_vars/main-street-station.yml
+#   3. SSH access as 'wed' with sudo
+# ============================================================================
+
+- name: Deploy JMRI headless server
+  hosts: "{{ target | default('jmri_server') }}"
+  become: true
+  gather_facts: true
+  roles:
+    - linux-baseline
+    - jmri

ansible/playbooks/day1_deploy_llm_inference.yml

@@ -0,0 +1,25 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day1_deploy_llm_inference.yml
+# DESCRIPTION: Day 1 playbook for astro-orbiter LLM inference stack.
+#              Deploys vLLM + Gemma 2 27B on RTX 3090 via OCuLink.
+#
+# Usage:
+#   cd ~/git/homelab/ansible
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference.yml
+#
+# Phases (added incrementally — safe to re-run):
+#   1. Foundation  — groups, directories, vault assertion
+#   2. Driver      — nvidia-driver-595-open (idempotent; already installed)
+#   3. vLLM        — Python venv + pip install vllm
+#   4. Model       — HF login, Gemma 2 27B snapshot_download
+#   5. Serve       — systemd vllm-serve.service, health check
+#   6. Integration — Hermes provider config on carousel
+# ------------------------------------------------------------------------------
+
+- name: Deploy LLM inference stack on astro-orbiter
+  hosts: astro_orbiter
+  gather_facts: true
+
+  roles:
+    - role: llm-inference

ansible/playbooks/day1_deploy_llm_inference_multimodel.yml

@@ -0,0 +1,35 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day1_deploy_llm_inference_multimodel.yml
+# DESCRIPTION: Day 1 playbook for the dual-model (aux + tool-calling) rollout
+#              on astro-orbiter. Builds on roles/llm-inference (CUDA/driver
+#              already done) — does not replace it.
+#
+# Usage:
+#   cd ~/git/homelab/ansible
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml
+#   # or scope to specific phases:
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml --tags discover
+#
+# EXECUTION CHANNEL (2026-08-12, War Machine): run via the Semaphore template
+# "llm_inference_multimodel_stage_models" (scoped to --tags models). Do NOT
+# run this via direct ansible-playbook or ad-hoc ssh/curl/systemctl — all
+# homelab inference changes go through Ansible roles executed by Semaphore for
+# audit/visibility. Phase 1 (models) is idempotent: it only downloads/stages a
+# GGUF when missing or size-mismatched, and only restarts the router when a new
+# GGUF is detected (normal re-runs that find the files correct touch nothing).
+#
+# Phases (see roles/llm-inference-multimodel/README.md for detail):
+#   0. discover  — read-only; confirm existing Gemma service management
+#   1. models    — idempotent GGUF downloads (Phi-4-14B, Mistral-Small-24B)
+#   2. systemd   — deploy both unit files, do NOT auto-start
+#   3. firewall  — scope ports 8000/8001, non-0.0.0.0 bind
+#   4. verify    — start both services, smoke test, VRAM check
+# ------------------------------------------------------------------------------
+
+- name: Deploy dual-model LLM inference stack on astro-orbiter
+  hosts: astro_orbiter
+  gather_facts: true
+
+  roles:
+    - role: llm-inference-multimodel

ansible/playbooks/day1_deploy_llm_router_shadow.yml

@@ -0,0 +1,106 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day1_deploy_llm_router_shadow.yml
+# DESCRIPTION: Deploy llama-server in router mode on a shadow port (8003).
+#
+# This playbook deploys and validates the llama.cpp router mode supervisor on
+# astro-orbiter (10.1.71.130) WITHOUT touching the production endpoint
+# (llama-server-qwen, port 8002). All 7 dependent Hermes profiles
+# (bruce-banner, groot, happy, heimdall, rocket-raccoon, war-machine, wong)
+# remain pointing at port 8002 throughout this run.
+#
+# Usage (from ~/git/homelab/ansible):
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_router_shadow.yml
+#
+# Tag-scoped runs (if you need to re-run one phase):
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_router_shadow.yml \
+#     --tags router_systemd,router_firewall,router_verify
+#
+# Execution path (Ryan-approved 2026-08-12, task t_0cca74a2):
+#   Direct ansible-playbook as documented exception — Semaphore template for
+#   this role does not exist yet. Create template after cutover is confirmed.
+#   This is the same exception pattern used in prior sessions on this box.
+#
+# Pre-requisites:
+#   1. llama-server binary at /opt/llama.cpp/build/bin/llama-server supports
+#      router mode (confirmed 2026-08-12: --models-dir flag present in --help).
+#   2. /opt/models/ contains ONLY Qwen3.6-35B-A3B-UD-Q4_K_S.gguf
+#      (confirmed 2026-08-12: directory is clean, Phi-4/Mistral already deleted).
+#   3. Port 8002 is in use by the production llama-server-qwen service —
+#      this playbook does NOT touch it.
+#
+# Validation gates this playbook runs (all hard gates EXCEPT Gate 4):
+#   Gate 1: /v1/models reports Qwen with n_ctx >= 64000 (64K Hermes floor)
+#   Gate 2: Tool-calling probe through router returns finish_reason=tool_calls
+#   Gate 2b: Hallucination stress test does NOT trigger spurious tool_calls
+#   Gate 3: nvidia-smi VRAM <= 23,000 MiB (--models-max 1 confirmed effective)
+#   Gate 4: Bundled SvelteKit UI check (nice-to-have, non-blocking)
+#
+# What happens after this playbook:
+#   War Machine posts validation gate results to Ryan.
+#   Ryan reviews and signs off on cutover (or requests changes).
+#   War Machine then runs day2_cutover_qwen_to_router.yml (not yet created)
+#   to promote the router to port 8002 and retire the bare llama-server-qwen.
+#
+# Reference: proposal at
+#   ~/friday/system/inbox/agents/war-machine/2026-08-12-qwen-router-mode-proposal.md
+# Task: t_0cca74a2
+# Author: War Machine (2026-08-12)
+# ------------------------------------------------------------------------------
+
+- name: Deploy llama-server router (shadow, port 8003) on astro-orbiter
+  hosts: astro_orbiter
+  gather_facts: true
+  become: true
+
+  vars:
+    # Enable the router phase — this is the ONLY var that makes router.yml run.
+    # Default in defaults/main.yml is false (no-op). Flip here for the shadow run.
+    llm_router_enabled: true
+
+    # Qwen model ID as it appears in /v1/models from the router.
+    # llama-server router uses the GGUF filename (without .gguf) as the model id.
+    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
+
+  roles:
+    - role: llm-inference-multimodel
+
+  # No --tags needed here: router.yml is included dynamically from main.yml
+  # whenever llm_router_enabled: true. The full role runs but the
+  # discover/models/systemd/verify phases are gated on their own vars
+  # (llm_qwen_service_enabled etc.) and are idempotent. The stale
+  # models.yml (Phi-4/Mistral download tasks) uses variables no longer
+  # defined — a follow-up cleanup task should update that file.
+
+- name: "POST-VALIDATION SAFETY NET — ensure production service is running"
+  hosts: astro_orbiter
+  gather_facts: false
+  become: true
+
+  tasks:
+    # Always run this, regardless of whether the validation play succeeded.
+    # If the router.yml play stopped llama-server-qwen for VRAM validation
+    # and then a gate failed (play aborted), this play ensures it comes back up.
+    - name: "Ensure llama-server-qwen (port 8002) is running after validation (always)"
+      ansible.builtin.systemd:
+        name: llama-server-qwen
+        state: started
+        enabled: true
+      ignore_errors: true   # don't fail if the unit doesn't exist
+
+    - name: "Verify production /health after safety-net restart"
+      ansible.builtin.uri:
+        url: "http://10.1.71.130:8002/health"
+        status_code: 200
+        timeout: 30
+      register: llm_safety_net_health
+      failed_when: false
+      ignore_errors: true
+
+    - name: "Report production status (safety-net check)"
+      ansible.builtin.debug:
+        msg: >-
+          Safety-net: llama-server-qwen :8002 health check returned
+          {{ llm_safety_net_health.status | default('UNREACHABLE') }}.
+          {{ 'OK — production is up.' if (llm_safety_net_health.status | default(0) | int == 200)
+             else 'WARNING — production may not be healthy. Check manually.' }}

ansible/playbooks/day1_deploy_nextcloud.yml

@@ -0,0 +1,91 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day1_deploy_nextcloud.yml
+# DESCRIPTION: Deploys Nextcloud on the-grid
+#              Runs: docker-host → nextcloud
+#
+# USAGE:
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_nextcloud.yml
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_nextcloud.yml --limit the-grid
+#
+# SECRETS REQUIRED IN VAULT (group_vars/all/vault):
+#   vault_nextcloud_db_root_password
+#   vault_nextcloud_db_password
+#   vault_nextcloud_admin_user
+#   vault_nextcloud_admin_password
+# ------------------------------------------------------------------------------
+
+- name: Deploy Nextcloud on the-grid
+  hosts: the-grid
+  become: true
+
+  vars:
+    # NFS prerequisite
+    nfs_packages:
+      - nfs-common
+
+    # Compose stack location
+    nextcloud_base_dir: /opt/docker/nextcloud
+
+    # Secrets from vault
+    nextcloud_db_root_password: "{{ vault_nextcloud_db_root_password }}"
+    nextcloud_db_password:      "{{ vault_nextcloud_db_password }}"
+    nextcloud_admin_user:       "{{ vault_nextcloud_admin_user }}"
+    nextcloud_admin_password:   "{{ vault_nextcloud_admin_password }}"
+
+  pre_tasks:
+    - name: Install NFS client
+      ansible.builtin.apt:
+        name: "{{ nfs_packages }}"
+        state: present
+        update_cache: true
+
+  roles:
+    - role: docker-host
+
+  tasks:
+    - name: Create Nextcloud directory
+      ansible.builtin.file:
+        path: "{{ nextcloud_base_dir }}"
+        state: directory
+        owner: "{{ ansible_user }}"
+        group: docker
+        mode: "0775"
+
+    - name: Deploy Compose file
+      ansible.builtin.copy:
+        src: "{{ playbook_dir }}/../../boilerplates/nextcloud/compose.yml"
+        dest: "{{ nextcloud_base_dir }}/compose.yml"
+        owner: "{{ ansible_user }}"
+        group: docker
+        mode: "0644"
+
+    - name: Deploy .env from vault
+      ansible.builtin.copy:
+        content: |
+          # Managed by Ansible — do not edit manually
+          MYSQL_ROOT_PASSWORD={{ nextcloud_db_root_password }}
+          MYSQL_PASSWORD={{ nextcloud_db_password }}
+          NEXTCLOUD_ADMIN_USER={{ nextcloud_admin_user }}
+          NEXTCLOUD_ADMIN_PASSWORD={{ nextcloud_admin_password }}
+        dest: "{{ nextcloud_base_dir }}/.env"
+        owner: "{{ ansible_user }}"
+        group: docker
+        mode: "0600"
+
+    - name: Start Nextcloud stack
+      community.docker.docker_compose_v2:
+        project_src: "{{ nextcloud_base_dir }}"
+        state: present
+
+    - name: Wait for Nextcloud to become ready
+      ansible.builtin.uri:
+        url: "http://the-grid.local.mk-labs.cloud/status.php"
+        status_code: 200
+        return_content: true
+      register: nextcloud_status
+      until: >
+        nextcloud_status.status == 200 and
+        (nextcloud_status.content | from_json).installed == true
+      retries: 20
+      delay: 15

ansible/playbooks/day1_deploy_nginx_proxy.yml

@@ -0,0 +1,6 @@
+---
+- name: Deploy shared nginx reverse proxy and certbot
+  hosts: illuminations
+  become: true
+  roles:
+    - nginx-proxy

ansible/playbooks/day1_deploy_ollama.yml

@@ -0,0 +1,35 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: ansible/playbooks/day1_deploy_ollama.yml
+# DESCRIPTION: Deploys Ollama with ROCm GPU acceleration on astro-orbiter.
+#              Assumes day0_baseline.yml has already run (common role complete).
+#              PCIe passthrough for the RX 5700 must be configured in Proxmox
+#              and the GPU must be visible to the VM before running this playbook.
+#
+# Pre-flight check:
+#   ssh wed@astro-orbiter 'lspci | grep -i amd'
+#   Should show the RX 5700 before proceeding.
+#
+# Usage (from ansible/ directory):
+#   ansible-playbook playbooks/day1_deploy_ollama.yml
+# ------------------------------------------------------------------------------
+
+- name: Deploy Ollama with ROCm on astro-orbiter
+  hosts: astro-orbiter
+  become: true
+
+  pre_tasks:
+    - name: Verify AMD GPU is visible to the VM
+      command: lspci
+      register: lspci_output
+      changed_when: false
+
+    - name: Fail if no AMD GPU detected
+      fail:
+        msg: >
+          No AMD GPU detected via lspci. Verify PCIe passthrough is configured
+          in Proxmox and the RX 5700 is visible to the VM before proceeding.
+      when: "'AMD' not in lspci_output.stdout and 'Radeon' not in lspci_output.stdout"
+
+  roles:
+    - ollama

ansible/playbooks/day1_deploy_semaphore.yml

@@ -0,0 +1,17 @@
+---
+# ============================================================================
+# day1_deploy_semaphore.yml
+# ----------------------------------------------------------------------------
+# Deploys SemaphoreUI + PostgreSQL on the imagineering host (figment).
+# Run AFTER day0_linux_baseline.yml has been applied to the target.
+#
+# Usage:
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_semaphore.yml
+# ============================================================================
+
+- name: Deploy SemaphoreUI on imagineering
+  hosts: semaphore_server
+  become: true
+  gather_facts: true
+  roles:
+    - semaphore

ansible/playbooks/day1_deploy_traefik.yml

@@ -0,0 +1,18 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/deploy_traefik.yml
+# DESCRIPTION: Deploys Traefik reverse proxy on lightning-lane.
+#              Installs Docker and configures Traefik with Cloudflare DNS-01
+#              certificate resolution for *.local.mk-labs.cloud
+#
+# USAGE:
+#   ansible-playbook -i inventory.yml playbooks/deploy_traefik.yml
+# ------------------------------------------------------------------------------
+
+- name: Deploy Traefik reverse proxy
+  hosts: load_balancers
+  become: true
+
+  roles:
+    - docker-host
+    - traefik

ansible/playbooks/day1_deploy_vllm.yml

@@ -0,0 +1,18 @@
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day1_deploy_vllm.yml
+# Deploy vLLM to a target host via roles/deploy-vllm.
+#
+# Staging run (deploy + validate WITHOUT touching production traffic):
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_vllm.yml --limit astro-orbiter
+#
+# Cutover run (once staging is validated and Ryan/JARVIS approve flipping
+# traffic — starts and enables the systemd unit(s), runs Phase 5 verification):
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_vllm.yml \
+#     --limit astro-orbiter --extra-vars "vllm_service_state=started"
+# ------------------------------------------------------------------------------
+- name: Deploy vLLM inference serving stack
+  hosts: astro-orbiter
+  become: false
+  gather_facts: true
+  roles:
+    - deploy-vllm

ansible/playbooks/day1_docker.yml

@@ -0,0 +1,7 @@
+---
+- name: Apply common role
+  hosts: "{{ target | default('all') }}"
+  become: true
+  roles:
+    - common
+    - docker-host

ansible/playbooks/day2_add_coder_alias.yml

@@ -0,0 +1,259 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day2_add_coder_alias.yml
+# DESCRIPTION: Add Qwen2.5-Coder-14B-Instruct-Q4_K_M to the llama-server-router
+#              on astro-orbiter (10.1.71.130:8002).
+#
+# Context (t_55c164f5, 2026-08-13):
+#   Ryan requested a Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf be added to the
+#   astro-orbiter router with:
+#     alias = "Qwen2.5-Coder-14B-Instruct-4bit"
+#     n_gpu_layers = 99
+#     ctx_size = 16384
+#     flash_attn = true
+#   Deployed GitOps-style via this role; no hand-editing of the live preset.
+#
+# What this playbook does:
+#   1. Downloads Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf into /opt/models if
+#      not already present (idempotent: size-check guard, no re-pull on match).
+#   2. Redeploys the preset INI (adding the [Qwen2.5-Coder-14B-Instruct-Q4_K_M]
+#      section with alias = Qwen2.5-Coder-14B-Instruct-4bit).
+#   3. Restarts llama-server-router to pick up the new model entry.
+#   4. Verifies /v1/models returns all 4 models including the new Coder entry.
+#
+# VRAM context note (t_55c164f5):
+#   Qwen2.5-Coder-14B Q4_K_M: ~8.4GB weights + ~0.6GB KV @ 16K ctx ≈ 9.0GB
+#   Qwen3.6-35B-A3B: ~21.5GB
+#   Full co-residency is impossible on 24GB. LRU eviction handles this:
+#   when Coder is requested, Qwen3.6-35B is evicted (and vice versa).
+#   Model-switching incurs ~30-60s cold-load latency — expected and acceptable.
+#   Phi (~4.3GB) or Llama (~5.6GB) can co-reside with Coder (total ~14GB).
+#
+# Usage (from ~/git/homelab/ansible):
+#   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook -i inventory.yml \
+#     playbooks/day2_add_coder_alias.yml
+#
+# Semaphore note: Semaphore SSH key for jarvis user is not loaded in the
+# container (known pitfall, homelab-llm-serving skill). Run via CLI with
+# id_jarvis key; document as exception per Ryan's standing CLI fallback directive.
+#
+# Author: War Machine (2026-08-13, t_55c164f5)
+# ------------------------------------------------------------------------------
+
+- name: "Add Qwen2.5-Coder-14B-Instruct-4bit alias to astro-orbiter router"
+  hosts: astro_orbiter
+  gather_facts: false
+  become: true
+
+  vars:
+    # Activate preset mode
+    llm_router_preset_enabled: true
+    llm_router_preset_path: /opt/llama-server-router-preset.ini
+
+    # Production port (router is on 8002 since t_cd0d5388)
+    llm_router_port: 8002
+
+    # Per-model ctx-size settings (carried from t_ryan_per_model_ctx; Coder new)
+    llm_router_llama_ctx_size: 8192
+    llm_router_llama_flash_attn: "true"
+    llm_router_phi_ctx_size: 32768
+    llm_router_phi_flash_attn: "true"
+    llm_router_coder_ctx_size: 16384
+    llm_router_coder_flash_attn: "true"
+
+    # All other vars inherit from host_vars + defaults/main.yml.
+    # Explicitly set the ones needed by the unit/template tasks for clarity:
+    llm_router_enabled: true
+    llm_service_user: jarvis
+    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
+    llm_models_dir: /opt/models
+    llm_bind_address: "10.1.71.130"
+    llm_allowed_source_cidr: "10.1.70.0/24"
+    llm_router_service_name: llama-server-router
+    llm_router_bind_address: "10.1.71.130"
+    llm_router_allowed_source_cidr: "10.1.70.0/24"
+    llm_router_models_dir: /opt/models
+    llm_router_models_max: 4            # from host_vars; bumped by t_33acbb2e
+    llm_router_ctx_size: 65536          # Qwen3.6-35B default; per-model overrides above
+    llm_router_parallel: 1
+    llm_router_gpu_layers: 99
+    llm_router_batch_size: 2048
+    llm_router_ubatch_size: 512
+    llm_router_cache_type_k: q4_0
+    llm_router_cache_type_v: q4_0
+    llm_router_flash_attn: "auto"
+    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
+    llm_router_vram_max_mib: 23000
+
+    # Coder model staging entry (used below)
+    coder_filename: "Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
+    coder_url: "https://huggingface.co/bartowski/Qwen2.5-Coder-14B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
+    coder_size_bytes: 8988111072
+
+  handlers:
+    - name: reload systemd
+      ansible.builtin.systemd:
+        daemon_reload: true
+      become: true
+      listen: "reload systemd"
+
+    - name: restart router
+      ansible.builtin.systemd:
+        name: llama-server-router
+        state: restarted
+      become: true
+      listen: "restart router"
+
+  tasks:
+
+    # ==========================================================================
+    # PHASE 1: Download Coder GGUF if not present / size mismatch
+    # ==========================================================================
+
+    - name: "[coder] Stat existing GGUF"
+      ansible.builtin.stat:
+        path: "{{ llm_models_dir }}/{{ coder_filename }}"
+        get_checksum: false
+      register: coder_stat
+
+    - name: "[coder] Download GGUF (skip if present and size matches)"
+      ansible.builtin.get_url:
+        url: "{{ coder_url }}"
+        dest: "{{ llm_models_dir }}/{{ coder_filename }}"
+        owner: "{{ llm_service_user }}"
+        group: "{{ llm_service_user }}"
+        mode: "0644"
+        timeout: 3600
+      when: >
+        not coder_stat.stat.exists or
+        coder_stat.stat.size != coder_size_bytes
+      register: coder_download
+      notify: restart router
+
+    - name: "[coder] Confirm GGUF size post-download"
+      ansible.builtin.stat:
+        path: "{{ llm_models_dir }}/{{ coder_filename }}"
+        get_checksum: false
+      register: coder_stat_post
+
+    - name: "[coder] FAIL if GGUF size mismatch after download"
+      ansible.builtin.fail:
+        msg: >-
+          GGUF size mismatch: expected {{ coder_size_bytes }} bytes,
+          got {{ coder_stat_post.stat.size }} bytes.
+          Re-download may be needed.
+      when: coder_stat_post.stat.size != coder_size_bytes
+
+    # ==========================================================================
+    # PHASE 2: Deploy updated preset INI (adds Coder section)
+    # ==========================================================================
+
+    - name: "[coder] Deploy preset INI to {{ llm_router_preset_path }}"
+      ansible.builtin.template:
+        src: "../roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2"
+        dest: "{{ llm_router_preset_path }}"
+        owner: root
+        group: root
+        mode: "0644"
+      register: coder_preset_deployed
+      notify: restart router
+
+    # ==========================================================================
+    # PHASE 3: Redeploy systemd unit (unchanged flags, but ensures unit is fresh)
+    # ==========================================================================
+
+    - name: "[coder] Deploy llama-server-router unit"
+      ansible.builtin.template:
+        src: "../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
+        dest: /etc/systemd/system/llama-server-router.service
+        owner: root
+        group: root
+        mode: "0644"
+      register: coder_unit_deployed
+      notify:
+        - reload systemd
+        - restart router
+
+    - name: "[coder] Flush handlers (daemon-reload + router restart)"
+      ansible.builtin.meta: flush_handlers
+
+    # ==========================================================================
+    # PHASE 4: Verify router is up and Coder model appears in /v1/models
+    # ==========================================================================
+
+    - name: "[coder] Wait for /health (router supervisor)"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
+        status_code: 200
+        timeout: 30
+      retries: 12
+      delay: 5
+      register: coder_health
+      until: coder_health.status == 200
+
+    - name: "[coder] Query /v1/models"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
+        status_code: 200
+        return_content: true
+        timeout: 30
+      register: coder_models
+
+    - name: "[coder] Extract model IDs and aliases"
+      ansible.builtin.set_fact:
+        coder_model_ids: "{{ coder_models.json.data | map(attribute='id') | list }}"
+        coder_all_aliases: "{{ coder_models.json.data | map(attribute='aliases') | flatten | list }}"
+
+    - name: "[coder] FAIL if Coder primary ID missing"
+      ansible.builtin.fail:
+        msg: >-
+          'Qwen2.5-Coder-14B-Instruct-Q4_K_M' not in /v1/models.
+          IDs: {{ coder_model_ids }}
+      when: "'Qwen2.5-Coder-14B-Instruct-Q4_K_M' not in coder_model_ids"
+
+    - name: "[coder] FAIL if Coder alias missing"
+      ansible.builtin.fail:
+        msg: >-
+          'Qwen2.5-Coder-14B-Instruct-4bit' not found as ID or alias in /v1/models.
+          IDs:     {{ coder_model_ids }}
+          Aliases: {{ coder_all_aliases }}
+      when:
+        - "'Qwen2.5-Coder-14B-Instruct-4bit' not in coder_model_ids"
+        - "'Qwen2.5-Coder-14B-Instruct-4bit' not in coder_all_aliases"
+
+    - name: "[coder] FAIL if Qwen3.6-35B missing"
+      ansible.builtin.fail:
+        msg: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in /v1/models. IDs: {{ coder_model_ids }}"
+      when: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in coder_model_ids"
+
+    - name: "[coder] FAIL if Phi missing"
+      ansible.builtin.fail:
+        msg: "'Phi-3.5-mini-instruct-Q8_0' not in /v1/models. IDs: {{ coder_model_ids }}"
+      when: "'Phi-3.5-mini-instruct-Q8_0' not in coder_model_ids"
+
+    - name: "[coder] FAIL if Llama missing"
+      ansible.builtin.fail:
+        msg: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in /v1/models. IDs: {{ coder_model_ids }}"
+      when: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in coder_model_ids"
+
+    - name: "[coder] PASS — full /v1/models summary"
+      ansible.builtin.debug:
+        msg:
+          - "========================================================================"
+          - "QWEN2.5-CODER-14B ALIAS DEPLOYMENT — COMPLETE"
+          - ""
+          - "  Mode:     --models-preset ({{ llm_router_preset_path }})"
+          - "  Service:  llama-server-router.service (:{{ llm_router_port }})"
+          - ""
+          - "  /v1/models IDs:     {{ coder_model_ids }}"
+          - "  /v1/models aliases: {{ coder_all_aliases }}"
+          - ""
+          - "  VERIFY:"
+          - "  Qwen3.6-35B-A3B-UD-Q4_K_S:          {{ 'PRESENT' if 'Qwen3.6-35B-A3B-UD-Q4_K_S' in coder_model_ids else 'MISSING' }}"
+          - "  Phi-3.5-mini-instruct-Q8_0:          {{ 'PRESENT' if 'Phi-3.5-mini-instruct-Q8_0' in coder_model_ids else 'MISSING' }}"
+          - "  Meta-Llama-3.1-8B-Instruct-Q4_K_M:  {{ 'PRESENT' if 'Meta-Llama-3.1-8B-Instruct-Q4_K_M' in coder_model_ids else 'MISSING' }}"
+          - "  Qwen2.5-Coder-14B-Instruct-Q4_K_M:  {{ 'PRESENT' if 'Qwen2.5-Coder-14B-Instruct-Q4_K_M' in coder_model_ids else 'MISSING' }}"
+          - "  Qwen2.5-Coder-14B-Instruct-4bit:    {{ 'PRESENT (ID)' if 'Qwen2.5-Coder-14B-Instruct-4bit' in coder_model_ids else ('PRESENT (alias)' if 'Qwen2.5-Coder-14B-Instruct-4bit' in coder_all_aliases else 'MISSING') }}"
+          - ""
+          - "  GGUF download: {{ 'NEW DOWNLOAD' if (coder_download is defined and coder_download.changed) else 'ALREADY PRESENT (skipped)' }}"
+          - "========================================================================"

ansible/playbooks/day2_add_nomic_embed.yml

@@ -0,0 +1,313 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day2_add_nomic_embed.yml
+# DESCRIPTION: Add nomic-embed-text-v1.5-Q4_K_M to the llama-server-router
+#              on astro-orbiter (10.1.71.130:8002).
+#
+# Context (t_34b96e83, 2026-08-13, OpenViking Phase 1b):
+#   Ryan approved adding nomic-embed-text-v1.5-Q4_K_M as an embedding model
+#   after Phase 0 follow-up confirmed embedding models fold cleanly into the
+#   existing router preset via embedding=true. Model ID is "nomic-embed-text-v1.5".
+#   No alias needed — peter-parker and Honcho consumers will call it by the section
+#   name directly.
+#
+# What this playbook does:
+#   1. Downloads nomic-embed-text-v1.5-Q4_K_M.gguf into /opt/models if not
+#      already present (idempotent: exact size-check guard, no re-pull on match).
+#   2. Redeploys the preset INI (adding the [nomic-embed-text-v1.5] section with
+#      embedding=true, n-gpu-layers=99, ctx-size=8192, load-on-startup=true,
+#      sleep-idle-seconds=-1).
+#   3. Restarts llama-server-router to pick up the new model entry.
+#   4. Verifies /v1/models returns all 5 models including the new nomic entry.
+#   5. Runs a /v1/embeddings smoke test to confirm the model actually embeds.
+#
+# VRAM context note (t_34b96e83):
+#   nomic-embed-text-v1.5 Q4_K_M: ~84MB weights, embedding model (no KV cache).
+#   VRAM impact is negligible — always pinned via sleep-idle-seconds=-1.
+#   The 4 generative models remain unchanged (OOM analysis unchanged from t_55c164f5).
+#
+# Usage (from ~/git/homelab/ansible):
+#   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook -i inventory.yml \
+#     playbooks/day2_add_nomic_embed.yml
+#
+# Semaphore note: Semaphore SSH key for jarvis user is not loaded in the
+# container (known pitfall, homelab-llm-serving skill). Run via CLI with
+# id_jarvis key; document as exception per Ryan's standing CLI fallback directive.
+#
+# Author: War Machine (2026-08-13, t_34b96e83)
+# ------------------------------------------------------------------------------
+
+- name: "Add nomic-embed-text-v1.5 embedding model to astro-orbiter router"
+  hosts: astro_orbiter
+  gather_facts: false
+  become: true
+
+  vars:
+    # Activate preset mode
+    llm_router_preset_enabled: true
+    llm_router_preset_path: /opt/llama-server-router-preset.ini
+
+    # Production port (router is on 8002 since t_cd0d5388)
+    llm_router_port: 8002
+
+    # Per-model ctx-size settings (carried from t_55c164f5; nomic new)
+    llm_router_llama_ctx_size: 8192
+    llm_router_llama_flash_attn: "true"
+    llm_router_phi_ctx_size: 32768
+    llm_router_phi_flash_attn: "true"
+    llm_router_coder_ctx_size: 16384
+    llm_router_coder_flash_attn: "true"
+    llm_router_nomic_ctx_size: 8192
+    # NOTE (2026-08-14, t_openviking_embed_batch): per-model batch-size/
+    # ubatch-size lines in the preset INI are NOT honored by llama-server's
+    # router — only ctx-size is applied per-model; batch-size/ubatch-size for
+    # every spawned child come from the router's own global CLI flags
+    # (confirmed via `ps aux` on astro-orbiter: child process launched with
+    # the router's --batch-size/--ubatch-size regardless of the INI values).
+    # Kept below for documentation/future-proofing but the REAL fix is the
+    # global llm_router_batch_size / llm_router_ubatch_size override further
+    # down, which raises the physical batch for ALL models on this router
+    # (Qwen3.6-35B, Phi, Llama, Coder, nomic).
+    llm_router_nomic_batch_size: 4096
+    llm_router_nomic_ubatch_size: 4096
+
+    # All other vars inherit from host_vars + defaults/main.yml.
+    llm_router_enabled: true
+    llm_service_user: jarvis
+    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
+    llm_models_dir: /opt/models
+    llm_bind_address: "10.1.71.130"
+    llm_allowed_source_cidr: "10.1.70.0/24"
+    llm_router_service_name: llama-server-router
+    llm_router_bind_address: "10.1.71.130"
+    llm_router_allowed_source_cidr: "10.1.70.0/24"
+    llm_router_models_dir: /opt/models
+    llm_router_models_max: 4            # from host_vars; bumped by t_33acbb2e
+    llm_router_ctx_size: 65536          # Qwen3.6-35B default; per-model overrides above
+    llm_router_parallel: 1
+    llm_router_gpu_layers: 99
+    # FIX (2026-08-14, t_openviking_embed_batch): raised from 512 to 4096.
+    # This is a GLOBAL router flag applied to every spawned model process
+    # (per-model INI batch-size/ubatch-size overrides are not honored by
+    # llama-server's router — see note above nomic vars). 512 tokens was too
+    # small for OpenViking's chunked-document embedding inputs (observed
+    # 2000-3400 tokens/chunk), causing hard 500 errors ("input (N tokens) is
+    # too large to process") that tripped OpenViking's circuit breaker into a
+    # permanent fail/re-enqueue loop. 4096 comfortably covers observed chunk
+    # sizes and stays under nomic's ctx-size=8192. VRAM impact of raising
+    # ubatch-size is in compute-buffer scratch space, not KV cache; monitored
+    # post-deploy against the 23000 MiB budget (host_vars/astro-orbiter).
+    llm_router_batch_size: 4096
+    llm_router_ubatch_size: 4096
+    llm_router_cache_type_k: q4_0
+    llm_router_cache_type_v: q4_0
+    llm_router_flash_attn: "auto"
+    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
+    llm_router_vram_max_mib: 23000
+
+    # nomic model staging
+    nomic_filename: "nomic-embed-text-v1.5-Q4_K_M.gguf"
+    nomic_url: "https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF/resolve/main/nomic-embed-text-v1.5.Q4_K_M.gguf"
+    nomic_size_bytes: 84106624
+
+  handlers:
+    - name: reload systemd
+      ansible.builtin.systemd:
+        daemon_reload: true
+      become: true
+      listen: "reload systemd"
+
+    - name: restart router
+      ansible.builtin.systemd:
+        name: llama-server-router
+        state: restarted
+      become: true
+      listen: "restart router"
+
+  tasks:
+
+    # ==========================================================================
+    # PHASE 1: Download nomic GGUF if not present / size mismatch
+    # ==========================================================================
+
+    - name: "[nomic] Stat existing GGUF"
+      ansible.builtin.stat:
+        path: "{{ llm_models_dir }}/{{ nomic_filename }}"
+        get_checksum: false
+      register: nomic_stat
+
+    - name: "[nomic] Download GGUF (skip if present and size matches)"
+      ansible.builtin.get_url:
+        url: "{{ nomic_url }}"
+        dest: "{{ llm_models_dir }}/{{ nomic_filename }}"
+        owner: "{{ llm_service_user }}"
+        group: "{{ llm_service_user }}"
+        mode: "0644"
+        timeout: 300
+      when: >
+        not nomic_stat.stat.exists or
+        nomic_stat.stat.size != nomic_size_bytes
+      register: nomic_download
+      notify: restart router
+
+    - name: "[nomic] Confirm GGUF size post-download"
+      ansible.builtin.stat:
+        path: "{{ llm_models_dir }}/{{ nomic_filename }}"
+        get_checksum: false
+      register: nomic_stat_post
+
+    - name: "[nomic] FAIL if GGUF size mismatch after download"
+      ansible.builtin.fail:
+        msg: >-
+          GGUF size mismatch: expected {{ nomic_size_bytes }} bytes,
+          got {{ nomic_stat_post.stat.size }} bytes.
+          Re-download may be needed.
+      when: nomic_stat_post.stat.size != nomic_size_bytes
+
+    # ==========================================================================
+    # PHASE 2: Deploy updated preset INI (adds nomic-embed-text-v1.5 section)
+    # ==========================================================================
+
+    - name: "[nomic] Deploy preset INI to {{ llm_router_preset_path }}"
+      ansible.builtin.template:
+        src: "../roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2"
+        dest: "{{ llm_router_preset_path }}"
+        owner: root
+        group: root
+        mode: "0644"
+      register: nomic_preset_deployed
+      notify: restart router
+
+    # ==========================================================================
+    # PHASE 3: Redeploy systemd unit (ensures unit is fresh; no flag changes)
+    # ==========================================================================
+
+    - name: "[nomic] Deploy llama-server-router unit"
+      ansible.builtin.template:
+        src: "../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
+        dest: /etc/systemd/system/llama-server-router.service
+        owner: root
+        group: root
+        mode: "0644"
+      register: nomic_unit_deployed
+      notify:
+        - reload systemd
+        - restart router
+
+    - name: "[nomic] Flush handlers (daemon-reload + router restart)"
+      ansible.builtin.meta: flush_handlers
+
+    # ==========================================================================
+    # PHASE 4: Verify router is up and nomic model appears in /v1/models
+    # ==========================================================================
+
+    - name: "[nomic] Wait for /health (router supervisor)"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
+        status_code: 200
+        timeout: 30
+      retries: 12
+      delay: 5
+      register: nomic_health
+      until: nomic_health.status == 200
+
+    - name: "[nomic] Query /v1/models"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
+        status_code: 200
+        return_content: true
+        timeout: 30
+      register: nomic_models
+
+    - name: "[nomic] Extract model IDs and aliases"
+      ansible.builtin.set_fact:
+        nomic_model_ids: "{{ nomic_models.json.data | map(attribute='id') | list }}"
+        nomic_all_aliases: "{{ nomic_models.json.data | map(attribute='aliases') | flatten | list }}"
+
+    - name: "[nomic] FAIL if nomic primary ID missing"
+      ansible.builtin.fail:
+        msg: >-
+          'nomic-embed-text-v1.5' not in /v1/models.
+          IDs: {{ nomic_model_ids }}
+      when: "'nomic-embed-text-v1.5' not in nomic_model_ids"
+
+    - name: "[nomic] FAIL if Qwen3.6-35B missing"
+      ansible.builtin.fail:
+        msg: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in /v1/models. IDs: {{ nomic_model_ids }}"
+      when: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in nomic_model_ids"
+
+    - name: "[nomic] FAIL if Phi missing"
+      ansible.builtin.fail:
+        msg: "'Phi-3.5-mini-instruct-Q8_0' not in /v1/models. IDs: {{ nomic_model_ids }}"
+      when: "'Phi-3.5-mini-instruct-Q8_0' not in nomic_model_ids"
+
+    - name: "[nomic] FAIL if Llama missing"
+      ansible.builtin.fail:
+        msg: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in /v1/models. IDs: {{ nomic_model_ids }}"
+      when: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in nomic_model_ids"
+
+    - name: "[nomic] FAIL if Coder missing"
+      ansible.builtin.fail:
+        msg: "'Qwen2.5-Coder-14B-Instruct-Q4_K_M' not in /v1/models. IDs: {{ nomic_model_ids }}"
+      when: "'Qwen2.5-Coder-14B-Instruct-Q4_K_M' not in nomic_model_ids"
+
+    # ==========================================================================
+    # PHASE 5: /v1/embeddings smoke test — confirm model actually embeds
+    # ==========================================================================
+
+    - name: "[nomic] POST /v1/embeddings smoke test"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/embeddings"
+        method: POST
+        body_format: json
+        body:
+          model: "nomic-embed-text-v1.5"
+          input: "The dog ran across the park."
+        status_code: 200
+        return_content: true
+        timeout: 120
+      register: nomic_embed_result
+
+    - name: "[nomic] Extract embedding vector length"
+      ansible.builtin.set_fact:
+        nomic_embed_dims: >-
+          {{ (nomic_embed_result.json.data | first).embedding | length }}
+      when:
+        - nomic_embed_result.status == 200
+        - nomic_embed_result.json.data is defined
+        - nomic_embed_result.json.data | length > 0
+
+    - name: "[nomic] FAIL if embedding vector is empty or missing"
+      ansible.builtin.fail:
+        msg: >-
+          Embedding smoke test returned no vector.
+          Response: {{ nomic_embed_result.json }}
+      when: >-
+        nomic_embed_result.status != 200 or
+        nomic_embed_result.json.data is not defined or
+        nomic_embed_result.json.data | length == 0 or
+        (nomic_embed_result.json.data | first).embedding | length == 0
+
+    - name: "[nomic] PASS — full summary"
+      ansible.builtin.debug:
+        msg:
+          - "========================================================================"
+          - "NOMIC-EMBED-TEXT-V1.5 DEPLOYMENT — COMPLETE"
+          - ""
+          - "  Mode:     --models-preset ({{ llm_router_preset_path }})"
+          - "  Service:  llama-server-router.service (:{{ llm_router_port }})"
+          - ""
+          - "  /v1/models IDs:     {{ nomic_model_ids }}"
+          - ""
+          - "  VERIFY:"
+          - "  Qwen3.6-35B-A3B-UD-Q4_K_S:          {{ 'PRESENT' if 'Qwen3.6-35B-A3B-UD-Q4_K_S' in nomic_model_ids else 'MISSING' }}"
+          - "  Phi-3.5-mini-instruct-Q8_0:          {{ 'PRESENT' if 'Phi-3.5-mini-instruct-Q8_0' in nomic_model_ids else 'MISSING' }}"
+          - "  Meta-Llama-3.1-8B-Instruct-Q4_K_M:  {{ 'PRESENT' if 'Meta-Llama-3.1-8B-Instruct-Q4_K_M' in nomic_model_ids else 'MISSING' }}"
+          - "  Qwen2.5-Coder-14B-Instruct-Q4_K_M:  {{ 'PRESENT' if 'Qwen2.5-Coder-14B-Instruct-Q4_K_M' in nomic_model_ids else 'MISSING' }}"
+          - "  nomic-embed-text-v1.5:               {{ 'PRESENT' if 'nomic-embed-text-v1.5' in nomic_model_ids else 'MISSING' }}"
+          - ""
+          - "  Embedding smoke test: PASS"
+          - "  Vector dimensions: {{ nomic_embed_dims | default('unknown') }}"
+          - ""
+          - "  GGUF download: {{ 'NEW DOWNLOAD' if (nomic_download is defined and nomic_download.changed) else 'ALREADY PRESENT (skipped)' }}"
+          - "========================================================================"

ansible/playbooks/day2_add_phi_alias.yml

@@ -0,0 +1,203 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day2_add_phi_alias.yml
+# DESCRIPTION: Add Phi-3.5-mini-instruct-8bit alias to the llama-server-router
+#              by switching from --models-dir to --models-preset INI mode.
+#
+# Context (t_9adf0889, 2026-08-12):
+#   Ryan's Hermes config (auxiliary.title_generation.model) points to
+#   "Phi-3.5-mini-instruct-8bit" but the router only exposes the GGUF
+#   filename-derived ID "Phi-3.5-mini-instruct-Q8_0". They are the same file.
+#   This playbook adds the alias so both names work without changing Ryan's
+#   Hermes config.
+#
+# What this playbook does:
+#   1. Deploys the preset INI template (llama-server-router-preset.ini.j2)
+#      to /opt/llama-server-router-preset.ini on astro-orbiter.
+#   2. Redeploys the systemd unit (llama-server-router.service) with
+#      --models-preset instead of --models-dir.
+#   3. Restarts llama-server-router to pick up the new flag.
+#   4. Verifies that /v1/models returns:
+#      - Phi-3.5-mini-instruct-Q8_0 (original ID — must still work)
+#      - Phi-3.5-mini-instruct-8bit (new alias — Ryan's config target)
+#      - Qwen3.6-35B-A3B-UD-Q4_K_S (unchanged)
+#      - Meta-Llama-3.1-8B-Instruct-Q4_K_M (unchanged)
+#
+# Known upstream behavior:
+#   GH #22364: --models-preset creates an extra "default" entry in /v1/models.
+#   This is cosmetic only and does not affect model selection by name.
+#
+# Usage (from ~/git/homelab/ansible):
+#   ansible-playbook -i inventory.yml playbooks/day2_add_phi_alias.yml
+#
+# Semaphore note (t_9adf0889): Semaphore SSH key for jarvis user is not loaded
+# in the container (known pitfall, homelab-llm-serving skill). Run via CLI with
+# id_jarvis key; document as exception per Ryan's standing CLI fallback directive.
+#
+# Author: War Machine (2026-08-12, t_9adf0889)
+# ------------------------------------------------------------------------------
+
+- name: "Add Phi-3.5-mini-instruct-8bit alias — switch router to preset mode"
+  hosts: astro_orbiter
+  gather_facts: false
+  become: true
+
+  vars:
+    # Activate preset mode and provide the on-disk INI path
+    llm_router_preset_enabled: true
+    llm_router_preset_path: /opt/llama-server-router-preset.ini
+
+    # Production port (router is already on 8002 since t_cd0d5388)
+    llm_router_port: 8002
+
+    # All other vars inherit from host_vars + defaults/main.yml.
+    # Explicitly set the ones needed by the unit template for clarity:
+    llm_router_enabled: true
+    llm_service_user: jarvis
+    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
+    llm_models_dir: /opt/models
+    llm_bind_address: "10.1.71.130"
+    llm_allowed_source_cidr: "10.1.70.0/24"
+    llm_router_service_name: llama-server-router
+    llm_router_bind_address: "10.1.71.130"
+    llm_router_allowed_source_cidr: "10.1.70.0/24"
+    llm_router_models_dir: /opt/models
+    llm_router_models_max: 4            # from host_vars; bumped by t_33acbb2e
+    llm_router_ctx_size: 65536
+    llm_router_parallel: 1
+    llm_router_gpu_layers: 99
+    llm_router_batch_size: 2048
+    llm_router_ubatch_size: 512
+    llm_router_cache_type_k: q4_0
+    llm_router_cache_type_v: q4_0
+    llm_router_flash_attn: "auto"
+    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
+    llm_router_vram_max_mib: 23000
+
+  handlers:
+    - name: reload systemd
+      ansible.builtin.systemd:
+        daemon_reload: true
+      become: true
+      listen: "reload systemd"
+
+    - name: restart router
+      ansible.builtin.systemd:
+        name: llama-server-router
+        state: restarted
+      become: true
+      listen: "restart router"
+
+  tasks:
+
+    # ==========================================================================
+    # PHASE 1: Deploy the preset INI
+    # ==========================================================================
+
+    - name: "[phi-alias] Deploy preset INI to {{ llm_router_preset_path }}"
+      ansible.builtin.template:
+        src: "../roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2"
+        dest: "{{ llm_router_preset_path }}"
+        owner: root
+        group: root
+        mode: "0644"
+      register: phi_alias_preset_deployed
+      notify:
+        - restart router
+
+    # ==========================================================================
+    # PHASE 2: Redeploy systemd unit with --models-preset flag
+    # ==========================================================================
+
+    - name: "[phi-alias] Deploy llama-server-router unit (--models-preset mode)"
+      ansible.builtin.template:
+        src: "../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
+        dest: /etc/systemd/system/llama-server-router.service
+        owner: root
+        group: root
+        mode: "0644"
+      register: phi_alias_unit_deployed
+      notify:
+        - reload systemd
+        - restart router
+
+    - name: "[phi-alias] Flush handlers (daemon-reload + router restart)"
+      ansible.builtin.meta: flush_handlers
+
+    # ==========================================================================
+    # PHASE 3: Verify alias is present
+    # ==========================================================================
+
+    - name: "[phi-alias] Wait for /health (router supervisor, no model needed)"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
+        status_code: 200
+        timeout: 30
+      retries: 12
+      delay: 5
+      register: phi_alias_health
+      until: phi_alias_health.status == 200
+
+    - name: "[phi-alias] Query /v1/models"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
+        status_code: 200
+        return_content: true
+        timeout: 30
+      register: phi_alias_models
+
+    - name: "[phi-alias] Extract model IDs and aliases"
+      ansible.builtin.set_fact:
+        phi_alias_model_ids: "{{ phi_alias_models.json.data | map(attribute='id') | list }}"
+        phi_alias_all_aliases: "{{ phi_alias_models.json.data | map(attribute='aliases') | flatten | list }}"
+        phi_alias_model_sources: "{{ phi_alias_models.json.data | map(attribute='source') | list }}"
+
+    - name: "[phi-alias] FAIL if Phi original ID missing"
+      ansible.builtin.fail:
+        msg: >-
+          'Phi-3.5-mini-instruct-Q8_0' not in /v1/models.
+          IDs: {{ phi_alias_model_ids }}
+      when: "'Phi-3.5-mini-instruct-Q8_0' not in phi_alias_model_ids"
+
+    - name: "[phi-alias] FAIL if Phi alias missing"
+      ansible.builtin.fail:
+        msg: >-
+          'Phi-3.5-mini-instruct-8bit' not found as ID or alias in /v1/models.
+          IDs:     {{ phi_alias_model_ids }}
+          Aliases: {{ phi_alias_all_aliases }}
+      when:
+        - "'Phi-3.5-mini-instruct-8bit' not in phi_alias_model_ids"
+        - "'Phi-3.5-mini-instruct-8bit' not in phi_alias_all_aliases"
+
+    - name: "[phi-alias] FAIL if Qwen missing"
+      ansible.builtin.fail:
+        msg: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in /v1/models. IDs: {{ phi_alias_model_ids }}"
+      when: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in phi_alias_model_ids"
+
+    - name: "[phi-alias] FAIL if Llama missing"
+      ansible.builtin.fail:
+        msg: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in /v1/models. IDs: {{ phi_alias_model_ids }}"
+      when: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in phi_alias_model_ids"
+
+    - name: "[phi-alias] PASS — full /v1/models summary"
+      ansible.builtin.debug:
+        msg:
+          - "========================================================================"
+          - "PHI ALIAS DEPLOYMENT — COMPLETE"
+          - ""
+          - "  Mode:     --models-preset ({{ llm_router_preset_path }})"
+          - "  Service:  llama-server-router.service (:{{ llm_router_port }})"
+          - ""
+          - "  /v1/models IDs:     {{ phi_alias_model_ids }}"
+          - "  /v1/models aliases: {{ phi_alias_all_aliases }}"
+          - "  Sources:            {{ phi_alias_model_sources }}"
+          - ""
+          - "  VERIFY:"
+          - "  Phi-3.5-mini-instruct-Q8_0:  {{ 'PRESENT' if 'Phi-3.5-mini-instruct-Q8_0' in phi_alias_model_ids else 'MISSING' }}"
+          - "  Phi-3.5-mini-instruct-8bit:  {{ 'PRESENT (ID)' if 'Phi-3.5-mini-instruct-8bit' in phi_alias_model_ids else ('PRESENT (alias)' if 'Phi-3.5-mini-instruct-8bit' in phi_alias_all_aliases else 'MISSING') }}"
+          - "  Qwen3.6-35B-A3B-UD-Q4_K_S:  {{ 'PRESENT' if 'Qwen3.6-35B-A3B-UD-Q4_K_S' in phi_alias_model_ids else 'MISSING' }}"
+          - "  Meta-Llama-3.1-8B-Instruct-Q4_K_M: {{ 'PRESENT' if 'Meta-Llama-3.1-8B-Instruct-Q4_K_M' in phi_alias_model_ids else 'MISSING' }}"
+          - ""
+          - "  GH #22364: if 'default' appears in IDs above, that is expected"
+          - "  in --models-preset mode. Cosmetic only."
+          - "========================================================================"

ansible/playbooks/day2_bump_router_models_max.yml

@@ -0,0 +1,161 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day2_bump_router_models_max.yml
+# DESCRIPTION: Bump --models-max on the production llama-server-router unit.
+#
+# Context: t_33acbb2e (2026-08-12) — Ryan requested --models-max raised from 1
+# to 4 so the router can keep multiple GGUFs resident on-demand (LRU eviction
+# when the cap is reached). The actual var change lives in:
+#   host_vars/astro-orbiter/vars.yml  (llm_router_models_max: 4)
+#
+# This playbook:
+#   1. Re-renders llama-server-router.service.j2 with the updated var value.
+#   2. Reloads systemd (daemon-reload handler) if the unit changed.
+#   3. Restarts llama-server-router so the new --models-max takes effect on the
+#      live process. Router holds no resident model (all-unloaded) so restart
+#      is sub-second and non-disruptive.
+#   4. Verifies /health returns 200 and /v1/models still lists all three GGUFs.
+#
+# VRAM NOTE: --models-max 4 allows up to all 3 current GGUFs to co-reside on
+# a 24GB card simultaneously. Worst-case combined footprint is ~31GB which
+# EXCEEDS 24GB — OOM is possible if all 3 are loaded concurrently. In normal
+# single-user homelab operation this is very unlikely. Full VRAM breakdown
+# documented in host_vars/astro-orbiter/vars.yml. Ryan approved (t_33acbb2e).
+#
+# Execution channel: Semaphore template "llm_router_update_unit" (project mk-labs).
+# Do NOT run via direct ansible-playbook or ad-hoc ssh/systemctl.
+#
+# Author: War Machine (2026-08-12, t_33acbb2e)
+# ------------------------------------------------------------------------------
+
+- name: "Bump llama-server-router --models-max to 4 on astro-orbiter"
+  hosts: astro_orbiter
+  gather_facts: true
+  become: true
+
+  vars:
+    # Production vars — router is live on :8002 (post-cutover t_cd0d5388)
+    llm_router_port: 8002
+    llm_router_bind_address: "10.1.71.130"
+
+    # llm_router_models_max is 4 via host_vars/astro-orbiter/vars.yml.
+    # Remaining role vars come from host_vars + defaults/main.yml via the
+    # inventory — we only explicitly set vars this playbook needs for its
+    # own tasks (health/models check URIs).
+
+    # Needed by the template task (mirrors defaults set in role defaults/main.yml)
+    llm_service_user: jarvis
+    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
+    llm_models_dir: /opt/models
+    llm_router_service_name: llama-server-router
+    llm_router_models_dir: /opt/models
+    llm_router_gpu_layers: 99
+    llm_router_ctx_size: 65536
+    llm_router_flash_attn: "auto"
+    llm_router_cache_type_k: q4_0
+    llm_router_cache_type_v: q4_0
+    llm_router_batch_size: 2048
+    llm_router_ubatch_size: 512
+    llm_router_parallel: 1
+
+  tasks:
+    # -------------------------------------------------------------------------
+    # Phase 1: Re-render the router unit file
+    # Template src path is relative to the role's templates/ dir; we reference
+    # it with a relative path that Ansible resolves from the role directory.
+    # -------------------------------------------------------------------------
+
+    - name: "Deploy updated llama-server-router unit (--models-max {{ llm_router_models_max }})"
+      ansible.builtin.template:
+        src: "{{ playbook_dir }}/../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
+        dest: "/etc/systemd/system/{{ llm_router_service_name }}.service"
+        owner: root
+        group: root
+        mode: "0644"
+      register: llm_router_unit_updated
+      notify:
+        - reload systemd
+      tags: [always]
+
+    - name: "Flush handlers — ensure daemon-reload lands before restart"
+      ansible.builtin.meta: flush_handlers
+      tags: [always]
+
+    # -------------------------------------------------------------------------
+    # Phase 2: Restart the router so the new --models-max takes effect.
+    # Always restart (even if unit unchanged) to ensure live process matches.
+    # -------------------------------------------------------------------------
+
+    - name: "Restart llama-server-router so --models-max {{ llm_router_models_max }} takes effect"
+      ansible.builtin.systemd:
+        name: "{{ llm_router_service_name }}"
+        state: restarted
+        enabled: true
+      tags: [always]
+
+    # -------------------------------------------------------------------------
+    # Phase 3: Verify /health returns 200
+    # -------------------------------------------------------------------------
+
+    - name: "Wait for /health to return 200 after restart"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
+        status_code: 200
+        timeout: 30
+      register: bump_health_check
+      retries: 10
+      delay: 3
+      until: bump_health_check.status == 200
+      tags: [always]
+
+    # -------------------------------------------------------------------------
+    # Phase 4: Verify /v1/models lists all three GGUFs
+    # -------------------------------------------------------------------------
+
+    - name: "Check /v1/models — all three GGUFs should appear"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
+        status_code: 200
+        timeout: 30
+        return_content: true
+      register: bump_models_check
+      tags: [always]
+
+    - name: "Display /v1/models summary"
+      ansible.builtin.debug:
+        msg:
+          - "======================================================================"
+          - "--models-max BUMP VERIFICATION (t_33acbb2e)"
+          - ""
+          - "  /health:               HTTP {{ bump_health_check.status }}"
+          - "  /v1/models HTTP:       {{ bump_models_check.status }}"
+          - "  Models listed:         {{ bump_models_check.json.data | map(attribute='id') | list | join(', ') }}"
+          - ""
+          - "  --models-max now:      {{ llm_router_models_max }}"
+          - "  --parallel (unchanged): {{ llm_router_parallel }}"
+          - ""
+          - "  VRAM WARNING: worst-case 3-model co-residency ~31GB > 24GB RTX 3090."
+          - "  OOM risk if all 3 load concurrently. LRU eviction mitigates in practice."
+          - "  Full breakdown: host_vars/astro-orbiter/vars.yml"
+          - "======================================================================"
+      when: bump_models_check is defined
+      tags: [always]
+
+    - name: "GATE: confirm all 3 expected GGUFs appear in /v1/models"
+      ansible.builtin.assert:
+        that:
+          - "'Qwen3.6-35B-A3B-UD-Q4_K_S' in (bump_models_check.json.data | map(attribute='id') | list)"
+          - "'Phi-3.5-mini-instruct-Q8_0' in (bump_models_check.json.data | map(attribute='id') | list)"
+          - "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' in (bump_models_check.json.data | map(attribute='id') | list)"
+        fail_msg: >-
+          /v1/models did not return all 3 expected GGUFs after --models-max bump.
+          Check router logs: journalctl -u llama-server-router -n 50
+        success_msg: "GATE PASSED: all 3 GGUFs listed in /v1/models."
+      when: bump_models_check is defined
+      tags: [always]
+
+  handlers:
+    - name: reload systemd
+      ansible.builtin.systemd:
+        daemon_reload: true
+      listen: "reload systemd"

ansible/playbooks/day2_cpu_offload_aux_models.yml

@@ -0,0 +1,59 @@
+---
+# ------------------------------------------------------------------------------
+# Playbook: day2_cpu_offload_aux_models.yml
+# Purpose:  CPU-offload Qwen2.5-Coder-14B and Meta-Llama-3.1-8B on
+#           astro-orbiter's production router (port 8002).
+#
+# What this playbook does:
+#   1. Re-renders llama-server-router-preset.ini (Coder + Llama sections now
+#      use per-model n-gpu-layers vars = 0 -> full CPU inference).
+#   2. Re-renders the router unit (--models-max now 4 via host_vars, global
+#      --n-gpu-layers removed per t_72646029 unit template fix) and restarts
+#      llama-server-router so both changes take effect.
+#   3. Verifies per the role's router_preset phase.
+#
+# Context (2026-08-17):
+#   - RAM/model-swap audit, TIER 1 (Coder-14B CPU offload) + TIER 2
+#     (Llama-3.1-8B CPU offload) — Ryan approved 1 & 2 on 2026-08-17.
+#     See inbox/ryan/2026-08-17-llm-system-ram-model-swap.md.
+#   - Unit template fix (t_72646029): global --n-gpu-layers removed from
+#     ExecStart in preset mode. Each INI section now sets n-gpu-layers
+#     explicitly (Qwen3.8=99, Phi=99, nomic=99, Coder=0, Llama=0).
+#   - Concurrent residency after change: Qwen3.8-27B (20,302 MiB @ 128K ctx)
+#     + nomic-embed (558 MiB, pinned) + Coder (CPU, ~1,390 MiB CUDA ctx) +
+#     Llama (CPU, ~1,706 MiB CUDA ctx) = ~24,004 MiB. NOTE: llama.cpp 6ea215d
+#     allocates CUDA-context VRAM even at n-gpu-layers=0, so CPU models are not
+#     0-VRAM; total sits at the 24,576 MiB physical limit (headroom ~572 MiB).
+#     Qwen3.8 is never evicted for a CPU aux model; Phi-3.5-mini (GPU, 8.3GB)
+#     still evicts as before.
+#   - CPU speed (8-core Ryzen 7 5800XT): ~5-10 tok/s (14B), ~10-20 tok/s (8B).
+#   - Semaphore SSH gap for astro-orbiter still applies (t_730f9584 /
+#     t_33acbb2e); running direct CLI Ansible per standing exception.
+#
+# Run:
+#   cd /home/hermes/git/homelab/ansible
+#   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook \
+#     -i inventory.yml \
+#     playbooks/day2_cpu_offload_aux_models.yml
+#
+# Rollback:
+#   git checkout -- \
+#     roles/llm-inference-multimodel/templates/llama-server-router.service.j2 \
+#     roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2 \
+#     roles/llm-inference-multimodel/defaults/main.yml \
+#     host_vars/astro-orbiter/vars.yml
+#   (restores n-gpu-layers=99 global flag, models-max=2, all GPU)
+#   then re-run this playbook to redeploy rollback state.
+#   Note: playbooks/day2_cpu_offload_aux_models.yml is untracked — left on disk.
+# ------------------------------------------------------------------------------
+- name: CPU-offload Coder-14B and Llama-3.1-8B on astro-orbiter
+  hosts: astro-orbiter
+  become: true
+  vars:
+    llm_router_preset_enabled: true
+    llm_router_enabled: true
+    llm_router_port: 8002
+
+  roles:
+    - role: llm-inference-multimodel
+      tags: [always]

ansible/playbooks/day2_cutover_qwen_to_router.yml

@@ -0,0 +1,511 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day2_cutover_qwen_to_router.yml
+# DESCRIPTION: Promote llama-server-router to production on port 8002.
+#
+# Context: Router-mode shadow deployment (t_0cca74a2) validated 2026-08-12:
+#   all 4 hard gates PASSED (n_ctx 65536, tool-calling PASS, hallucination-stress
+#   PASS, VRAM 20410 MiB / 1 process). Ryan approved cutover.
+#
+# This playbook makes the router the permanent production endpoint:
+#
+#   1. Stop + disable llama-server-qwen (:8002). Unit file is PRESERVED on disk
+#      as the rollback target (same pattern as prior role history).
+#   2. Redeploy llama-server-router unit file with --port 8002 (production port).
+#      PORT DECISION: we rebind the router to :8002 rather than updating 8
+#      dependent Hermes profiles' base_url. One unit file change beats 8
+#      config.yaml updates — atomic, GitOps-clean, zero profile drift.
+#   3. Enable + start llama-server-router on :8002.
+#   4. Re-run validation gates 1-3 against the NOW-production endpoint.
+#      (Same logic as Phase R / router_verify in tasks/router.yml — hard gates.)
+#   5. Run Gate 4: verify bundled SvelteKit UI is reachable.
+#
+# Usage (from ~/git/homelab/ansible):
+#   ansible-playbook -i inventory.yml playbooks/day2_cutover_qwen_to_router.yml
+#
+# Rollback (if gates fail or any time after):
+#   ansible-playbook -i inventory.yml playbooks/day2_cutover_qwen_to_router.yml \
+#     --tags cutover_rollback
+#
+# Author: War Machine (2026-08-12, t_cd0d5388)
+# Approved by: Ryan (cutover authorization, 2026-08-12)
+# ------------------------------------------------------------------------------
+
+- name: "CUTOVER — Promote llama-server-router to production (:8002) on astro-orbiter"
+  hosts: astro_orbiter
+  gather_facts: true
+  become: true
+
+  vars:
+    # ----------------------------------------------------------------
+    # PORT DECISION:
+    # We rebind the router to :8002 (production port) rather than
+    # updating 8 dependent Hermes profiles' base_url to :8003.
+    # Rationale: one unit file change is atomic and GitOps-clean.
+    # Updating 8 config.yaml files risks drift and requires per-profile
+    # activation tests. The template renders llm_router_port as the
+    # --port argument; we just override it here to 8002.
+    # ----------------------------------------------------------------
+
+    # Router port override: take over production port
+    llm_router_port: 8002
+
+    # All other role defaults needed by the template (mirrors defaults/main.yml)
+    llm_service_user: jarvis
+    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
+    llm_models_dir: /opt/models
+    llm_bind_address: "10.1.71.130"
+    llm_allowed_source_cidr: "10.1.70.0/24"
+
+    llm_router_enabled: true
+    llm_router_service_name: llama-server-router
+    llm_router_models_dir: /opt/models
+    llm_router_models_max: 1               # CRITICAL: RTX 3090 24GB, single model only
+    llm_router_ctx_size: 65536
+    llm_router_parallel: 1
+    llm_router_gpu_layers: 99
+    llm_router_batch_size: 2048
+    llm_router_ubatch_size: 512
+    llm_router_cache_type_k: q4_0
+    llm_router_cache_type_v: q4_0
+    llm_router_flash_attn: "auto"
+    llm_router_bind_address: "10.1.71.130"
+    llm_router_allowed_source_cidr: "10.1.70.0/24"
+    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
+    llm_router_vram_max_mib: 23000
+
+    llm_qwen_service_name: llama-server-qwen
+    llm_qwen_port: 8002
+
+  tasks:
+
+    # =======================================================================
+    # PHASE 1 — Stop and disable llama-server-qwen (bare single-model)
+    # Preserve unit file on disk — rollback target per existing role pattern.
+    # =======================================================================
+
+    - name: "[cutover] PHASE 1: Confirm llama-server-qwen current state"
+      ansible.builtin.systemd:
+        name: llama-server-qwen
+      register: cutover_qwen_status
+      tags: [cutover_stop_qwen, cutover]
+
+    - name: "[cutover] PHASE 1: Report current llama-server-qwen status"
+      ansible.builtin.debug:
+        msg: >-
+          llama-server-qwen: ActiveState={{ cutover_qwen_status.status.ActiveState | default('unknown') }},
+          UnitFileState={{ cutover_qwen_status.status.UnitFileState | default('unknown') }}.
+          Will stop + disable. Unit file preserved at /etc/systemd/system/llama-server-qwen.service as rollback target.
+      tags: [cutover_stop_qwen, cutover]
+
+    - name: "[cutover] PHASE 1: Stop llama-server-qwen (:8002, bare single-model)"
+      ansible.builtin.systemd:
+        name: llama-server-qwen
+        state: stopped
+      register: cutover_qwen_stopped
+      tags: [cutover_stop_qwen, cutover]
+
+    - name: "[cutover] PHASE 1: Disable llama-server-qwen (prevent auto-start on reboot)"
+      ansible.builtin.systemd:
+        name: llama-server-qwen
+        enabled: false
+      tags: [cutover_stop_qwen, cutover]
+
+    - name: "[cutover] PHASE 1: Wait 5s for VRAM to be released"
+      ansible.builtin.pause:
+        seconds: 5
+      when: cutover_qwen_stopped.changed | default(false)
+      tags: [cutover_stop_qwen, cutover]
+
+    - name: "[cutover] PHASE 1: Verify port 8002 is now free"
+      ansible.builtin.command:
+        cmd: ss -ltnp
+      register: cutover_port_check
+      changed_when: false
+      tags: [cutover_stop_qwen, cutover]
+
+    - name: "[cutover] PHASE 1: Fail if port 8002 is still bound"
+      ansible.builtin.fail:
+        msg: >-
+          Port 8002 is still bound after stopping llama-server-qwen.
+          Check 'ss -ltnp | grep :8002' and resolve before the router can bind.
+      when:
+        - "':8002 ' in (cutover_port_check.stdout | default('')) or ':8002:' in (cutover_port_check.stdout | default(''))"
+      tags: [cutover_stop_qwen, cutover]
+
+    - name: "[cutover] PHASE 1: Report VRAM state (should be empty)"
+      ansible.builtin.command:
+        cmd: nvidia-smi --query-compute-apps=pid,name,used_memory --format=csv,noheader
+      register: cutover_vram_free_check
+      changed_when: false
+      tags: [cutover_stop_qwen, cutover]
+
+    - name: "[cutover] PHASE 1: Print VRAM state"
+      ansible.builtin.debug:
+        msg: >-
+          VRAM after stopping llama-server-qwen:
+          {{ cutover_vram_free_check.stdout if (cutover_vram_free_check.stdout | length > 0)
+             else '(no GPU processes — VRAM free)' }}
+      tags: [cutover_stop_qwen, cutover]
+
+    # =======================================================================
+    # PHASE 2 — Redeploy llama-server-router unit with --port 8002
+    # =======================================================================
+
+    - name: "[cutover] PHASE 2: Deploy llama-server-router unit file (port 8002 — production)"
+      ansible.builtin.template:
+        src: "../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
+        dest: /etc/systemd/system/llama-server-router.service
+        owner: root
+        group: root
+        mode: "0644"
+      register: cutover_router_unit_deployed
+      notify:
+        - reload systemd
+      tags: [cutover_deploy_unit, cutover]
+
+    - name: "[cutover] PHASE 2: Flush handlers (daemon-reload before start)"
+      ansible.builtin.meta: flush_handlers
+      tags: [cutover_deploy_unit, cutover]
+
+    # =======================================================================
+    # PHASE 3 — Enable + start llama-server-router on :8002
+    # =======================================================================
+
+    - name: "[cutover] PHASE 3: Enable + start llama-server-router (production, :8002)"
+      ansible.builtin.systemd:
+        name: llama-server-router
+        state: "{{ 'restarted' if (cutover_router_unit_deployed.changed | default(false)) else 'started' }}"
+        enabled: true
+        daemon_reload: true
+      tags: [cutover_start_router, cutover]
+
+    # =======================================================================
+    # PHASE 4 — Validation gates 1-3 (hard gates against now-production :8002)
+    # =======================================================================
+
+    - name: "[cutover] GATE 1a: Wait for router /health on :8002 (up to 5min — cold model load)"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
+        status_code: 200
+      register: cutover_health
+      retries: 30
+      delay: 10
+      until: cutover_health.status == 200
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 1a: Trigger model load (router lazy-loads on first request)"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/chat/completions"
+        method: POST
+        body_format: json
+        body:
+          model: "{{ llm_router_expected_model_id }}"
+          messages:
+            - role: user
+              content: "Reply with one word: hello"
+          max_tokens: 5
+          temperature: 0.0
+        status_code: 200
+        return_content: true
+        timeout: 300
+      register: cutover_warmup
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 1a: Report warmup"
+      ansible.builtin.debug:
+        msg:
+          - "Model loaded. finish_reason={{ cutover_warmup.json.choices[0].finish_reason | default('unknown') }}"
+          - "Response: {{ cutover_warmup.json.choices[0].message.content | default('(empty)') | truncate(100) }}"
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 1b: Query /v1/models on :8002"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
+        status_code: 200
+        return_content: true
+      register: cutover_models
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 1b: Fail if expected model ID not found"
+      ansible.builtin.fail:
+        msg: >-
+          GATE 1 FAIL: '{{ llm_router_expected_model_id }}' not found in /v1/models.
+          Returned: {{ cutover_models.json.data | map(attribute='id') | list }}
+      when:
+        - cutover_models.json.data | selectattr('id', 'equalto', llm_router_expected_model_id) | list | length == 0
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 1b: Extract ctx-size from router model args"
+      ansible.builtin.set_fact:
+        cutover_qwen_n_ctx: >-
+          {%- set model = cutover_models.json.data | selectattr('id', 'equalto', llm_router_expected_model_id) | first -%}
+          {%- set args = model.status.args -%}
+          {%- set ctx_idx = args.index('--ctx-size') if '--ctx-size' in args else -1 -%}
+          {{ args[ctx_idx + 1] | int if ctx_idx >= 0 else 0 }}
+      when:
+        - cutover_models.json.data | selectattr('id', 'equalto', llm_router_expected_model_id) | list | length > 0
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 1b: Fail if n_ctx < 64000"
+      ansible.builtin.fail:
+        msg: "GATE 1 FAIL: --ctx-size={{ cutover_qwen_n_ctx }} < 64000 (Hermes 64K floor)."
+      when:
+        - cutover_qwen_n_ctx is defined
+        - cutover_qwen_n_ctx | int < 64000
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 1b: PASS — n_ctx >= 64K"
+      ansible.builtin.debug:
+        msg: "GATE 1 PASS: --ctx-size={{ cutover_qwen_n_ctx }} >= 64000."
+      when:
+        - cutover_qwen_n_ctx is defined
+        - cutover_qwen_n_ctx | int >= 64000
+      tags: [cutover_validate, cutover]
+
+    # --- Gate 2: Tool-calling through router proxy ---
+
+    - name: "[cutover] GATE 2: Tool-calling probe"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/chat/completions"
+        method: POST
+        body_format: json
+        body:
+          model: "{{ llm_router_expected_model_id }}"
+          messages:
+            - role: user
+              content: "What is the current weather in Chicago? Use the provided tool."
+          tools:
+            - type: function
+              function:
+                name: get_weather
+                description: "Get current weather conditions for a city"
+                parameters:
+                  type: object
+                  properties:
+                    city:
+                      type: string
+                      description: "The city name"
+                  required:
+                    - city
+          temperature: 0.0
+        status_code: 200
+        return_content: true
+        timeout: 120
+      register: cutover_toolcall_probe
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 2: Fail if not finish_reason=tool_calls"
+      ansible.builtin.fail:
+        msg: >-
+          GATE 2 FAIL: finish_reason={{ cutover_toolcall_probe.json.choices[0].finish_reason | default('(missing)') }}
+          (expected tool_calls). Response: {{ cutover_toolcall_probe.json | to_json }}
+      when:
+        - cutover_toolcall_probe.json.choices[0].finish_reason | default('') != 'tool_calls'
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 2: PASS"
+      ansible.builtin.debug:
+        msg:
+          - "GATE 2 PASS: finish_reason=tool_calls"
+          - "function: {{ cutover_toolcall_probe.json.choices[0].message.tool_calls[0].function.name | default('(unknown)') }}"
+          - "arguments: {{ cutover_toolcall_probe.json.choices[0].message.tool_calls[0].function.arguments | default('(none)') }}"
+      when:
+        - cutover_toolcall_probe.json.choices[0].finish_reason | default('') == 'tool_calls'
+      tags: [cutover_validate, cutover]
+
+    # --- Gate 2b: Hallucination stress ---
+
+    - name: "[cutover] GATE 2b: Hallucination stress probe"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/chat/completions"
+        method: POST
+        body_format: json
+        body:
+          model: "{{ llm_router_expected_model_id }}"
+          messages:
+            - role: user
+              content: "Tell me a brief fact about the planet Mars. Do not call any functions."
+          tools:
+            - type: function
+              function:
+                name: get_weather
+                description: "Get current weather conditions for a city"
+                parameters:
+                  type: object
+                  properties:
+                    city:
+                      type: string
+                  required:
+                    - city
+          temperature: 0.1
+        status_code: 200
+        return_content: true
+        timeout: 120
+      register: cutover_halluc_probe
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 2b: Fail if spurious tool_calls"
+      ansible.builtin.fail:
+        msg: >-
+          GATE 2b FAIL: finish_reason=tool_calls on unrelated prompt (Mars fact).
+          Over-triggering through router. Response: {{ cutover_halluc_probe.json | to_json }}
+      when:
+        - cutover_halluc_probe.json.choices[0].finish_reason | default('') == 'tool_calls'
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 2b: PASS"
+      ansible.builtin.debug:
+        msg: "GATE 2b PASS: finish_reason={{ cutover_halluc_probe.json.choices[0].finish_reason }} — no spurious tool_calls."
+      when:
+        - cutover_halluc_probe.json.choices[0].finish_reason | default('') != 'tool_calls'
+      tags: [cutover_validate, cutover]
+
+    # --- Gate 3: VRAM guard ---
+
+    - name: "[cutover] GATE 3: Check VRAM usage (--models-max 1 guard)"
+      ansible.builtin.command:
+        cmd: nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv,noheader
+      register: cutover_vram_post
+      changed_when: false
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 3: Parse VRAM used MiB"
+      ansible.builtin.set_fact:
+        cutover_vram_used_mib: "{{ cutover_vram_post.stdout.split(',')[0].strip().split(' ')[0] | int }}"
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 3: Fail if VRAM exceeds ceiling"
+      ansible.builtin.fail:
+        msg: >-
+          GATE 3 FAIL: {{ cutover_vram_used_mib }} MiB > {{ llm_router_vram_max_mib }} MiB ceiling.
+          Full: {{ cutover_vram_post.stdout }}
+      when:
+        - cutover_vram_used_mib | int > llm_router_vram_max_mib | int
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 3: Count GPU processes"
+      ansible.builtin.command:
+        cmd: nvidia-smi --query-compute-apps=pid,name --format=csv,noheader
+      register: cutover_gpu_procs
+      changed_when: false
+      failed_when: false
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 3: PASS"
+      ansible.builtin.debug:
+        msg:
+          - "GATE 3 PASS: {{ cutover_vram_used_mib }} MiB / {{ llm_router_vram_max_mib }} MiB ceiling."
+          - "GPU processes: {{ cutover_gpu_procs.stdout_lines | default(['(none)']) }}"
+          - "Full nvidia-smi: {{ cutover_vram_post.stdout }}"
+      when:
+        - cutover_vram_used_mib | int <= llm_router_vram_max_mib | int
+      tags: [cutover_validate, cutover]
+
+    # =======================================================================
+    # PHASE 5 — Gate 4: Bundled SvelteKit Web UI (required this time)
+    # =======================================================================
+
+    - name: "[cutover] GATE 4: Check bundled SvelteKit UI at :8002"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/"
+        status_code: [200, 301, 302]
+        return_content: true
+        timeout: 30
+      register: cutover_ui_check
+      failed_when: false
+      tags: [cutover_validate, cutover_ui, cutover]
+
+    - name: "[cutover] GATE 4: Inspect UI content"
+      ansible.builtin.set_fact:
+        cutover_ui_is_html: "{{ 'html' in (cutover_ui_check.content | default('') | lower) or '<!doctype' in (cutover_ui_check.content | default('') | lower) }}"
+        cutover_ui_has_model_select: "{{ 'select' in (cutover_ui_check.content | default('') | lower) or 'model' in (cutover_ui_check.content | default('') | lower) }}"
+      when: cutover_ui_check is defined
+      tags: [cutover_validate, cutover_ui, cutover]
+
+    - name: "[cutover] GATE 4: Report UI check and bookmark URL"
+      ansible.builtin.debug:
+        msg:
+          - "======================================================================"
+          - "GATE 4 UI CHECK:"
+          - "  HTTP status:          {{ cutover_ui_check.status | default('UNREACHABLE') }}"
+          - "  Is HTML:              {{ cutover_ui_is_html | default(false) }}"
+          - "  Contains model/select: {{ cutover_ui_has_model_select | default(false) }}"
+          - "  BOOKMARK URL:         http://{{ llm_router_bind_address }}:{{ llm_router_port }}/"
+          - "  {{ 'GATE 4 PASS — UI serving HTML at :8002.' if (cutover_ui_check.status | default(0) | int in [200, 301, 302]) else 'GATE 4 WARN — UI not reachable (HTTP ' + (cutover_ui_check.status | default('FAIL') | string) + ').' }}"
+          - "======================================================================"
+      when: cutover_ui_check is defined
+      tags: [cutover_validate, cutover_ui, cutover]
+
+    # =======================================================================
+    # CUTOVER SUMMARY
+    # =======================================================================
+
+    - name: "[cutover] CUTOVER SUMMARY — production promoted"
+      ansible.builtin.debug:
+        msg:
+          - "======================================================================"
+          - "CUTOVER COMPLETE: llama-server-router is now production."
+          - ""
+          - "  Service:  llama-server-router.service (enabled, running)"
+          - "  Port:     8002 (unchanged for all 8 Hermes profiles)"
+          - "  Model:    {{ llm_router_expected_model_id }}"
+          - "  Mode:     Router/supervisor (--models-dir /opt/models, --models-max 1)"
+          - ""
+          - "  Gate 1 (n_ctx >= 64K):     PASS ({{ cutover_qwen_n_ctx | default('N/A') }})"
+          - "  Gate 2 (tool-calling):     PASS (finish_reason=tool_calls)"
+          - "  Gate 2b (halluc stress):   PASS (no spurious tool_calls)"
+          - "  Gate 3 (VRAM <= 23000MiB): PASS ({{ cutover_vram_used_mib | default('N/A') }} MiB)"
+          - "  Gate 4 (Web UI):           HTTP {{ cutover_ui_check.status | default('N/A') }}"
+          - ""
+          - "  ROLLBACK TARGET: /etc/systemd/system/llama-server-qwen.service (unit preserved)"
+          - "  ROLLBACK CMD: sudo systemctl enable --now llama-server-qwen"
+          - "               sudo systemctl disable --now llama-server-router"
+          - "  Or: ansible-playbook -i inventory.yml day2_cutover_qwen_to_router.yml --tags cutover_rollback"
+          - ""
+          - "  Web UI bookmark: http://{{ llm_router_bind_address }}:{{ llm_router_port }}/"
+          - "======================================================================"
+      tags: [cutover]
+
+    # =======================================================================
+    # ROLLBACK — tag cutover_rollback reverses the cutover
+    # Run: ansible-playbook -i inventory.yml day2_cutover_qwen_to_router.yml --tags cutover_rollback
+    # WARNING: rollback_task has no dependency on cutover tags — safe to run standalone.
+    # =======================================================================
+
+    - name: "[cutover_rollback] Stop + disable llama-server-router"
+      ansible.builtin.systemd:
+        name: llama-server-router
+        state: stopped
+        enabled: false
+      tags: [cutover_rollback, never]    # 'never' = only runs with explicit --tags cutover_rollback
+
+    - name: "[cutover_rollback] Enable + start llama-server-qwen (restore bare :8002)"
+      ansible.builtin.systemd:
+        name: llama-server-qwen
+        state: started
+        enabled: true
+      tags: [cutover_rollback, never]
+
+    - name: "[cutover_rollback] Verify rollback /health"
+      ansible.builtin.uri:
+        url: "http://{{ llm_bind_address | default('10.1.71.130') }}:8002/health"
+        status_code: 200
+        timeout: 30
+      register: cutover_rollback_health
+      failed_when: false
+      tags: [cutover_rollback, never]
+
+    - name: "[cutover_rollback] Report rollback result"
+      ansible.builtin.debug:
+        msg: >-
+          ROLLBACK: llama-server-qwen :8002 health returned
+          {{ cutover_rollback_health.status | default('UNREACHABLE') }}.
+          {{ 'OK — production restored to bare qwen.' if (cutover_rollback_health.status | default(0) | int == 200)
+             else 'WARNING — health check failed. Check manually.' }}
+      tags: [cutover_rollback, never]
+
+  handlers:
+    - name: reload systemd
+      ansible.builtin.systemd:
+        daemon_reload: true

ansible/playbooks/day2_install_node_exporter.yml

@@ -0,0 +1,80 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: ansible/playbooks/day2_install_node_exporter.yml
+# DESCRIPTION: Install and configure Prometheus Node Exporter on Linux hosts.
+#
+# Targets two groups with different firewall managers:
+#   - proxmox_nodes  : Proxmox hypervisors (Debian, firewalld not present,
+#                      uses iptables/no firewall — just open the port via UFW
+#                      if present, otherwise skip)
+#   - monitored_vms  : Ubuntu VMs managed by UFW
+#
+# After installing on new hosts, re-runs day1_deploy_monitoring.yml to refresh
+# the Prometheus scrape config is NOT needed — targets are already statically
+# defined in prometheus.yaml.j2 for the Proxmox nodes. For new VMs, add the
+# IP to the prometheus.yaml.j2 proxmox-vms job and re-run day1_deploy_monitoring.yml.
+#
+# Usage:
+#   ansible-playbook -i inventory.yml playbooks/day2_install_node_exporter.yml
+#   ansible-playbook -i inventory.yml playbooks/day2_install_node_exporter.yml --limit proxmox_nodes
+#   ansible-playbook -i inventory.yml playbooks/day2_install_node_exporter.yml --limit monitored_vms
+# ------------------------------------------------------------------------------
+
+# ── Play 1: Proxmox hypervisors ───────────────────────────────────────────────
+# Proxmox runs Debian. The prometheus.prometheus.node_exporter role installs
+# a binary + systemd service without touching apt, which is what we want on
+# hypervisors (keep the package footprint clean).
+# Note: Proxmox does not run UFW. Port 9100 is open by default on VLAN 71.
+
+- name: Install Node Exporter on Proxmox hypervisors
+  hosts: proxmox_nodes
+  become: true
+
+  vars:
+    node_exporter_version: "1.9.1"
+    node_exporter_web_listen_address: "0.0.0.0:9100"
+    node_exporter_enabled_collectors:
+      - systemd
+      - processes
+      - filesystem
+      - meminfo
+      - cpu
+      - diskstats
+      - netdev
+      - loadavg
+      - uname
+
+  roles:
+    - role: prometheus.prometheus.node_exporter
+
+# ── Play 2: Ubuntu VMs ────────────────────────────────────────────────────────
+# Standard Ubuntu hosts with UFW. Same role, adds UFW allow rule for 9100.
+
+- name: Install Node Exporter on monitored VMs
+  hosts: monitored_vms
+  become: true
+
+  vars:
+    node_exporter_version: "1.9.1"
+    node_exporter_web_listen_address: "0.0.0.0:9100"
+    node_exporter_enabled_collectors:
+      - systemd
+      - processes
+      - filesystem
+      - meminfo
+      - cpu
+      - diskstats
+      - netdev
+      - loadavg
+      - uname
+
+  pre_tasks:
+    - name: Open Node Exporter port in UFW
+      community.general.ufw:
+        rule: allow
+        port: "9100"
+        proto: tcp
+        comment: "Prometheus Node Exporter"
+
+  roles:
+    - role: prometheus.prometheus.node_exporter

ansible/playbooks/day2_per_model_ctx_size.yml

@@ -0,0 +1,277 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day2_per_model_ctx_size.yml
+# DESCRIPTION: Right-size --ctx-size per model workload on llama-server-router
+#              (already in --models-preset mode since t_9adf0889).
+#
+# Context (t_ryan_per_model_ctx, 2026-08-13, requested by Ryan via JARVIS):
+#   All 3 preset models currently launch with a uniform --ctx-size 65536.
+#   This playbook narrows two of them to match actual workload:
+#     - Meta-Llama-3.1-8B-Instruct-Q4_K_M (alias Meta-Llama-3.1-8B-Instruct-4bit):
+#         ctx-size 65536 -> 8192  (tool-routing / micro-tasks: title gen, MCP
+#         tool calls, approval checks)
+#     - Phi-3.5-mini-instruct-Q8_0 (alias Phi-3.5-mini-instruct-8bit):
+#         ctx-size 65536 -> 32768 (long web scrapes / session-log compression)
+#   Both also move flash-attn from "auto" to explicit "true" per Ryan's spec.
+#   Qwen3.6-35B-A3B-UD-Q4_K_S is INTENTIONALLY left untouched at 65536/auto.
+#
+#   Existing aliases (Meta-Llama-3.1-8B-Instruct-4bit, Phi-3.5-mini-instruct-8bit)
+#   are PRESERVED as-is. Ryan's pasted TOML used different alias strings
+#   ("llama-3.1-8b", "phi-3.5-mini") but renaming aliases was not explicitly
+#   requested and would break live Hermes custom_providers routing — flagged
+#   in the deployment report rather than applied silently.
+#
+# IMPORTANT — Hermes side effect: /home/hermes/.hermes/config.yaml declares
+# context_length: 65536 for both these models under custom_providers. This
+# playbook does NOT touch that file (out of role/agent scope) but the value
+# becomes STALE the moment this playbook lands. Flag to JARVIS/Maria Hill.
+#
+# Usage (from ~/git/homelab/ansible):
+#   ansible-playbook -i inventory.yml playbooks/day2_per_model_ctx_size.yml
+#
+# Author: War Machine (2026-08-13, t_ryan_per_model_ctx)
+# ------------------------------------------------------------------------------
+
+- name: "Right-size per-model ctx-size on llama-server-router (Llama 8k, Phi 32k)"
+  hosts: astro_orbiter
+  gather_facts: false
+  become: true
+
+  vars:
+    # Preset mode already active in production (t_9adf0889) — keep it on.
+    llm_router_preset_enabled: true
+    llm_router_preset_path: /opt/llama-server-router-preset.ini
+    llm_router_enabled: true
+
+    # Production port
+    llm_router_port: 8002
+    llm_router_bind_address: "10.1.71.130"
+    llm_router_allowed_source_cidr: "10.1.70.0/24"
+    llm_bind_address: "10.1.71.130"
+    llm_allowed_source_cidr: "10.1.70.0/24"
+
+    llm_service_user: jarvis
+    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
+    llm_models_dir: /opt/models
+    llm_router_service_name: llama-server-router
+    llm_router_models_dir: /opt/models
+    llm_router_models_max: 4
+    llm_router_parallel: 1
+    llm_router_gpu_layers: 99
+    llm_router_batch_size: 2048
+    llm_router_ubatch_size: 512
+    llm_router_cache_type_k: q4_0
+    llm_router_cache_type_v: q4_0
+
+    # Qwen — untouched baseline (also used as router-wide fallback default)
+    llm_router_ctx_size: 65536
+    llm_router_flash_attn: "auto"
+    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
+    llm_router_vram_max_mib: 23000
+
+    # --- THE CHANGE: per-model overrides ---
+    llm_router_llama_ctx_size: 8192
+    llm_router_llama_flash_attn: "true"
+    llm_router_phi_ctx_size: 32768
+    llm_router_phi_flash_attn: "true"
+
+  handlers:
+    - name: reload systemd
+      ansible.builtin.systemd:
+        daemon_reload: true
+      become: true
+      listen: "reload systemd"
+
+    - name: restart router
+      ansible.builtin.systemd:
+        name: llama-server-router
+        state: restarted
+      become: true
+      listen: "restart router"
+
+  tasks:
+
+    # ==========================================================================
+    # PHASE 1: Deploy the preset INI with new per-model ctx-size/flash-attn
+    # ==========================================================================
+
+    - name: "[ctx-resize] Deploy preset INI to {{ llm_router_preset_path }}"
+      ansible.builtin.template:
+        src: "../roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2"
+        dest: "{{ llm_router_preset_path }}"
+        owner: root
+        group: root
+        mode: "0644"
+      register: ctx_resize_preset_deployed
+      notify:
+        - restart router
+
+    - name: "[ctx-resize] Deploy router systemd unit (drop global --ctx-size/--flash-attn in preset mode)"
+      ansible.builtin.template:
+        src: "../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
+        dest: /etc/systemd/system/llama-server-router.service
+        owner: root
+        group: root
+        mode: "0644"
+      register: ctx_resize_unit_deployed
+      notify:
+        - reload systemd
+        - restart router
+
+    - name: "[ctx-resize] Flush handlers (daemon-reload + router restart if changed)"
+      ansible.builtin.meta: flush_handlers
+
+    # ==========================================================================
+    # PHASE 2: Verify
+    # ==========================================================================
+
+    - name: "[ctx-resize] Wait for /health"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
+        status_code: 200
+        timeout: 30
+      retries: 12
+      delay: 5
+      register: ctx_resize_health
+      until: ctx_resize_health.status == 200
+
+    - name: "[ctx-resize] Query /v1/models"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
+        status_code: 200
+        return_content: true
+        timeout: 30
+      register: ctx_resize_models
+
+    - name: "[ctx-resize] Trigger load — Llama (confirms actual load + captures live args)"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/chat/completions"
+        method: POST
+        body_format: json
+        body:
+          model: "Meta-Llama-3.1-8B-Instruct-Q4_K_M"
+          messages:
+            - role: user
+              content: "Reply with one word: hello"
+          max_tokens: 5
+          temperature: 0.0
+        status_code: 200
+        return_content: true
+        timeout: 120
+      register: ctx_resize_llama_warmup
+
+    - name: "[ctx-resize] Trigger load — Phi (confirms actual load + captures live args)"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/chat/completions"
+        method: POST
+        body_format: json
+        body:
+          model: "Phi-3.5-mini-instruct-Q8_0"
+          messages:
+            - role: user
+              content: "Reply with one word: hello"
+          max_tokens: 5
+          temperature: 0.0
+        status_code: 200
+        return_content: true
+        timeout: 120
+      register: ctx_resize_phi_warmup
+
+    - name: "[ctx-resize] Re-query /v1/models after warmup (final state)"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
+        status_code: 200
+        return_content: true
+        timeout: 30
+      register: ctx_resize_models_final
+
+    - name: "[ctx-resize] Extract Llama args"
+      ansible.builtin.set_fact:
+        ctx_resize_llama_args: >-
+          {{ (ctx_resize_models_final.json.data | selectattr('id', 'equalto', 'Meta-Llama-3.1-8B-Instruct-Q4_K_M') | first).status.args }}
+        ctx_resize_llama_status: >-
+          {{ (ctx_resize_models_final.json.data | selectattr('id', 'equalto', 'Meta-Llama-3.1-8B-Instruct-Q4_K_M') | first).status.value }}
+
+    - name: "[ctx-resize] Extract Phi args"
+      ansible.builtin.set_fact:
+        ctx_resize_phi_args: >-
+          {{ (ctx_resize_models_final.json.data | selectattr('id', 'equalto', 'Phi-3.5-mini-instruct-Q8_0') | first).status.args }}
+        ctx_resize_phi_status: >-
+          {{ (ctx_resize_models_final.json.data | selectattr('id', 'equalto', 'Phi-3.5-mini-instruct-Q8_0') | first).status.value }}
+
+    - name: "[ctx-resize] Extract Qwen args (must be unchanged)"
+      ansible.builtin.set_fact:
+        ctx_resize_qwen_args: >-
+          {{ (ctx_resize_models_final.json.data | selectattr('id', 'equalto', 'Qwen3.6-35B-A3B-UD-Q4_K_S') | first).status.args }}
+
+    - name: "[ctx-resize] GATE — Llama ctx-size must be 8192"
+      ansible.builtin.assert:
+        that:
+          - "'8192' in ctx_resize_llama_args"
+          - ctx_resize_llama_args[ctx_resize_llama_args.index('--ctx-size') + 1] == '8192'
+        fail_msg: "Llama ctx-size not 8192. Args: {{ ctx_resize_llama_args }}"
+        success_msg: "Llama ctx-size confirmed 8192."
+
+    - name: "[ctx-resize] GATE — Llama flash-attn must be true"
+      ansible.builtin.assert:
+        that:
+          - ctx_resize_llama_args[ctx_resize_llama_args.index('--flash-attn') + 1] == 'true'
+        fail_msg: "Llama flash-attn not true. Args: {{ ctx_resize_llama_args }}"
+        success_msg: "Llama flash-attn confirmed true."
+
+    - name: "[ctx-resize] GATE — Llama loaded successfully"
+      ansible.builtin.assert:
+        that:
+          - ctx_resize_llama_status == 'loaded'
+        fail_msg: "Llama status is '{{ ctx_resize_llama_status }}', expected 'loaded'."
+        success_msg: "Llama status confirmed 'loaded'."
+
+    - name: "[ctx-resize] GATE — Phi ctx-size must be 32768"
+      ansible.builtin.assert:
+        that:
+          - ctx_resize_phi_args[ctx_resize_phi_args.index('--ctx-size') + 1] == '32768'
+        fail_msg: "Phi ctx-size not 32768. Args: {{ ctx_resize_phi_args }}"
+        success_msg: "Phi ctx-size confirmed 32768."
+
+    - name: "[ctx-resize] GATE — Phi flash-attn must be true"
+      ansible.builtin.assert:
+        that:
+          - ctx_resize_phi_args[ctx_resize_phi_args.index('--flash-attn') + 1] == 'true'
+        fail_msg: "Phi flash-attn not true. Args: {{ ctx_resize_phi_args }}"
+        success_msg: "Phi flash-attn confirmed true."
+
+    - name: "[ctx-resize] GATE — Phi loaded successfully"
+      ansible.builtin.assert:
+        that:
+          - ctx_resize_phi_status == 'loaded'
+        fail_msg: "Phi status is '{{ ctx_resize_phi_status }}', expected 'loaded'."
+        success_msg: "Phi status confirmed 'loaded'."
+
+    - name: "[ctx-resize] GATE — Qwen ctx-size UNCHANGED at 65536"
+      ansible.builtin.assert:
+        that:
+          - ctx_resize_qwen_args[ctx_resize_qwen_args.index('--ctx-size') + 1] == '65536'
+        fail_msg: "Qwen ctx-size changed unexpectedly! Args: {{ ctx_resize_qwen_args }}"
+        success_msg: "Qwen ctx-size confirmed UNCHANGED at 65536."
+
+    - name: "[ctx-resize] PASS — summary"
+      ansible.builtin.debug:
+        msg:
+          - "================================================================"
+          - "PER-MODEL CTX-SIZE DEPLOYMENT — COMPLETE"
+          - ""
+          - "  Llama-3.1-8B (Meta-Llama-3.1-8B-Instruct-Q4_K_M):"
+          - "    status:     {{ ctx_resize_llama_status }}"
+          - "    args:       {{ ctx_resize_llama_args }}"
+          - ""
+          - "  Phi-3.5-mini (Phi-3.5-mini-instruct-Q8_0):"
+          - "    status:     {{ ctx_resize_phi_status }}"
+          - "    args:       {{ ctx_resize_phi_args }}"
+          - ""
+          - "  Qwen3.6-35B-A3B-UD-Q4_K_S: UNCHANGED (ctx-size 65536, args: {{ ctx_resize_qwen_args }})"
+          - ""
+          - "  ACTION NEEDED: /home/hermes/.hermes/config.yaml custom_providers"
+          - "  context_length: 65536 for both Meta-Llama-3.1-8B-Instruct-4bit and"
+          - "  Phi-3.5-mini-instruct-8bit is now STALE (actual: 8192 / 32768)."
+          - "  Flag to JARVIS/Maria Hill for correction — NOT done by this playbook."
+          - "================================================================"

ansible/playbooks/day2_qwen38_ctx128k.yml

@@ -0,0 +1,38 @@
+---
+# ------------------------------------------------------------------------------
+# Playbook: day2_qwen38_ctx128k.yml
+# Purpose:  Bump Qwen3.8-27B-Q4_K_M ctx-size from 32768 to 131072 (128K)
+#           on astro-orbiter's production router (port 8002).
+#
+# What this playbook does:
+#   1. Renders the updated llama-server-router-preset.ini.j2 (now with
+#      llm_router_qwen38_ctx_size: 131072) to /opt/llama-server-router-preset.ini.
+#   2. Restarts llama-server-router.service.
+#   3. Verifies the router loads Qwen3.8-27B at ctx=131072 in status.args.
+#
+# Context:
+#   - Empirical VRAM test (t_4455a44c): 131072 ctx = 20,282 MiB Qwen3.8
+#     + 558 MiB nomic-embed = ~20.8GB total; ~3.2GB headroom on 24GB RTX 3090.
+#     Co-resident with nomic-embed: comfortably fits.
+#   - Ryan approved this deployment.
+#   - Semaphore SSH gap for astro-orbiter still applies (t_730f9584 / t_33acbb2e);
+#     running direct CLI Ansible per standing exception.
+#
+# Run:
+#   cd /home/hermes/git/homelab/ansible
+#   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook \
+#     -i inventory.yml \
+#     playbooks/day2_qwen38_ctx128k.yml
+#
+# Task reference: t_441470b9 — War Machine, 2026-08-16
+# ------------------------------------------------------------------------------
+- name: Bump Qwen3.8-27B ctx-size to 131072 on astro-orbiter
+  hosts: astro-orbiter
+  become: true
+  vars:
+    llm_router_preset_enabled: true
+    llm_router_qwen38_ctx_size: 131072
+
+  roles:
+    - role: llm-inference-multimodel
+      tags: [preset, systemd, verify]

ansible/playbooks/day2_qwen38_ctx128k_rollback.yml

@@ -0,0 +1,65 @@
+---
+# ------------------------------------------------------------------------------
+# Playbook: day2_qwen38_ctx128k_rollback.yml
+# Purpose:  Roll back Qwen3.8-27B-Q4_K_M ctx-size from 131072 back to 65536
+#           on astro-orbiter's production router (port 8002).
+#
+# What this playbook does:
+#   1. Renders the updated llama-server-router-preset.ini.j2 (now with
+#      llm_router_qwen38_ctx_size: 65536) to
+#      /opt/llama-server-router-preset.ini.
+#   2. Restarts llama-server-router.service.
+#   3. Verifies the router loads Qwen3.8-27B at ctx=65536 in status.args.
+#
+# Context:
+#   - t_441470b9 (2026-08-16): ctx-size bumped 32768 -> 131072. Verified VRAM
+#     at 131072 ctx with only Qwen3.8 + nomic-embed co-resident: ~20,282 MiB
+#     + 558 MiB = ~20.8 GB on 24 GB RTX 3090. Comfortably safe.
+#   - t_72646029 (2026-08-17): Phi-3.5mini moved to GPU (n-gpu-layers=99)
+#     to enable concurrent residency with CPU-offloaded Coder-14B and
+#     Llama-3.1-8B. This added ~2GB CUDA context buffers for Phi + shifted
+#     Phi's model weights onto the GPU (~3.8GB).
+#   - NEW steady-state VRAM: Qwen3.8 @ 131072 ctx (~20,282 MiB) + nomic-embed
+#     (~558 MiB) + Llama CUDA ctx (~1,706 MiB) + Coder CUDA ctx (~1,390 MiB)
+#     = ~24,004 MiB. Adding Phi-3.5 (~3,800 MiB weights + ~1.4 GB CUDA ctx)
+#     pushes total to ~29,000+ MiB — exceeding the 24,576 MiB RTX 3090 limit.
+#     Qwen3.8-27B-131072 now fails to load (HTTP 500, OOM before llama.cpp
+#     reaches the model-loading phase).
+#   - FIX: reduce Qwen3.8 ctx-size 131072 -> 65536. This reduces KV cache
+#     from ~6GB to ~3GB, freeing ~3GB of VRAM. New estimated steady-state:
+#     Qwen3.8 @ 65536 ctx (~17,068 MiB) + nomic (~558) + Llama ctx (~1,706)
+#     + Coder ctx (~1,390) + Phi-3.5 (~3,800 + ~1,400 CUDA ctx) = ~25,922 MiB.
+#     Still over 24,576 — see "Phase 2" below for the secondary fix.
+#
+#   IMPORTANT: Rolling back ctx-size alone may NOT be sufficient. The
+#   hardware reference (astro-orbiter-hardware.md line 166, t_72646029)
+#   states steady-state ~24,004 MiB WITHOUT Phi on GPU. Adding Phi-3.5 back
+#   to GPU tips it over. This playbook handles the context rollback; if Qwen3.8
+#   still fails to load after Phase R, Wong should escalate to Ryan for a
+#   decision on either (a) offloading Phi-3.5mini to CPU (n-gpu-layers=0),
+#   or (b) adding a second GPU. Document the Phase 2 finding as a separate
+#   follow-up task if needed.
+#
+#   The 64K floor from the 2026-08-12 cutover validation (t_cd0d5388, Gate 1)
+#   still applies — ctx-size=65536 satisfies it.
+#
+# Run:
+#   cd /home/hermes/git/homelab/ansible
+#   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook \
+#     -i inventory.yml \
+#     playbooks/day2_qwen38_ctx128k_rollback.yml
+#
+# Task reference: t_c9fed26c — War Machine benchmark, 2026-08-18
+# Root cause: t_72646029 CPU-offload deployment added Phi-3.5 to GPU,
+#   shifting total VRAM past the 24,576 MiB ceiling when Qwen3.8 runs at 128K.
+# ------------------------------------------------------------------------------
+- name: Roll back Qwen3.8-27B ctx-size to 65536 on astro-orbiter
+  hosts: astro-orbiter
+  become: true
+  vars:
+    llm_router_preset_enabled: true
+    llm_router_qwen38_ctx_size: 65536
+
+  roles:
+    - role: llm-inference-multimodel
+      tags: [preset, systemd, verify]

ansible/playbooks/day2_swap_qwen38.yml

@@ -0,0 +1,36 @@
+---
+# ------------------------------------------------------------------------------
+# Playbook: day2_swap_qwen38.yml
+# Purpose:  Swap the primary production model on astro-orbiter router from
+#           Qwen3.6-35B-A3B-UD-Q4_K_S to Qwen3.8-27B-Q4_K_M.
+#           This is a GitOps-encoded record of the swap performed 2026-08-16
+#           per Ryan's direction (kanban task t_f5f7e9ad).
+#
+# What this playbook does:
+#   1. Renders the updated llama-server-router-preset.ini.j2 to
+#      /opt/llama-server-router-preset.ini on astro-orbiter.
+#   2. Reloads the llama-server-router service (SIGHUP / restart as needed).
+#   3. Verifies the new model ID appears in /v1/models.
+#
+# Prerequisites:
+#   - Qwen3.8-27B-Q4_K_M.gguf must be present in /opt/models on astro-orbiter.
+#     (Downloaded out-of-band via wget during the swap task.)
+#   - roles/llm-inference-multimodel/defaults/main.yml updated to reference
+#     Qwen3.8-27B-Q4_K_M (done in this same commit).
+#
+# Run:
+#   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook \
+#     -i inventory.yml \
+#     playbooks/day2_swap_qwen38.yml
+#
+# Task reference: t_f5f7e9ad — War Machine, 2026-08-16
+# ------------------------------------------------------------------------------
+- name: Swap primary model to Qwen3.8-27B-Q4_K_M on astro-orbiter
+  hosts: astro-orbiter
+  become: true
+  vars:
+    llm_router_preset_enabled: true
+
+  roles:
+    - role: llm-inference-multimodel
+      tags: [preset, systemd, verify]

ansible/playbooks/day3_deploy_qwen38_ctx131k.yml

@@ -0,0 +1,22 @@
+---
+# Playbook: day3_deploy_qwen38_ctx131k.yml
+# Purpose: Deploy Qwen3.8-27B-Q4_K_M ctx-size 65536 -> 131072 to astro-orbiter
+#         via llama-swap config re-render + restart.
+#
+# The git change to defaults/main.yml (line 235: ctx_size: 131072) is already staged.
+# This playbook renders /etc/llama-swap/config.yaml from the updated defaults
+# and restarts llama-swap to load the new ctx-size.
+#
+# Run:
+#   cd /home/hermes/git/homelab/ansible
+#   ansible-playbook -i inventory.yml playbooks/day3_deploy_qwen38_ctx131k.yml
+#
+- name: Deploy Qwen3.8 ctx-size 131072 to astro-orbiter
+  hosts: astro-orbiter
+  become: true
+  vars:
+    llm_swapmode_enabled: true
+
+  roles:
+    - role: llm-inference-multimodel
+      tags: [swapmode_config, swapmode_systemd, swapmode_verify]

ansible/playbooks/deploy_multica.yml

@@ -0,0 +1,6 @@
+---
+- name: Deploy Multica + Caddy
+  hosts: multica
+  become: true
+  roles:
+    - role: multica_docker