fix: correct metric names in llama-swap monitoring (llamacpp_* -> llamaswap_*), update alerts + dashboard + scrape config

- Remove global --n-gpu-layers from router unit ExecStart in preset mode
  (llama.cpp CLI arg outranked per-model INI n-gpu-layers=0; root cause from
  War Machine's run 1). Flag now emitted only in --models-dir mode.
- All 5 preset INI sections carry explicit n-gpu-layers:
  Qwen3.8=99, Phi=99, nomic=99, Coder=0, Llama=0.
- host_vars/astro-orbiter: llm_router_models_max 2 -> 4 so CPU-offloaded
  models count as loaded without LRU-evicting Qwen3.8.
- defaults: llm_router_coder_gpu_layers / llm_router_llama_gpu_layers = 0.
- verify.yml: fix pre-existing .meta attribute crash in router mode.
- New playbook day2_cpu_offload_aux_models.yml.

Deployed + verified on astro-orbiter (gates A-E PASS): concurrent residency
achieved, Qwen3.8 stays GPU-resident. Measured CPU throughput Llama 9.0 /
Coder 4.7 tok/s. VRAM note: llama.cpp 6ea215d allocates ~1.4-1.7GB CUDA-context
per CPU model even at n-gpu-layers=0 -> ~24,004 MiB steady-state, below the
24,576 MiB physical limit. Comments corrected to match the measurement.
Report: friday/inbox/ryan/2026-08-17-llm-cpu-offload-coder-llama-deployed.md

COUCHDB-ERLANGCOOKIE-FIX.md

@@ -0,0 +1,160 @@
+# CouchDB erlangCookie Fix - Implementation Guide
+
+## Summary
+
+**Problem**: CouchDB deployment fails because `erlangCookie` is missing from the ExternalSecret configuration.
+
+**Decision**: Externalize `erlangCookie` to 1Password (pragmatic approach)
+
+**Rationale**: 
+- ExternalSecret architecture requires ownership of the entire secret
+- Mixing externalized and chart-generated fields in the same secret is not supported
+- Single-node deployment makes erlangCookie rotation unnecessary
+- This is an acceptable deviation from the pure Harbor pattern given the architectural constraints
+
+## Implementation Steps
+
+### 1. Generate erlangCookie Value
+
+```bash
+openssl rand -hex 20
+```
+
+Example output: `f4e3c2b1a9d8e7f6c5b4a3d2e1f0a9b8c7d6e5f4`
+
+### 2. Add to 1Password
+
+- **Vault**: `mk-labs`
+- **Item**: `couchdb`
+- **Field Name**: `erlang-cookie`
+- **Field Type**: password (concealed)
+- **Value**: `<paste generated value from step 1>`
+
+### 3. Update ExternalSecret Configuration
+
+File: `cluster/applications/couchdb/externalsecret.yaml`
+
+```yaml
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: couchdb-credentials
+  namespace: couchdb
+  labels:
+    app.kubernetes.io/name: couchdb
+    app.kubernetes.io/part-of: mk-labs
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: onepassword-connect
+  target:
+    name: couchdb-admin
+    creationPolicy: Owner
+    template:
+      engineVersion: v2
+      data:
+        adminUsername: "admin"
+        adminPassword: "{{ .adminPassword }}"
+        cookieAuthSecret: "{{ .cookieAuthSecret }}"
+        erlangCookie: "{{ .erlangCookie }}"  # ← ADD THIS LINE
+  data:
+    - secretKey: adminPassword
+      remoteRef:
+        key: couchdb
+        property: admin-password
+    - secretKey: cookieAuthSecret
+      remoteRef:
+        key: couchdb
+        property: cookie-auth-secret
+    - secretKey: erlangCookie  # ← ADD THIS BLOCK
+      remoteRef:
+        key: couchdb
+        property: erlang-cookie
+```
+
+### 4. Update values.yaml Documentation (Optional)
+
+File: `cluster/applications/couchdb/values.yaml`
+
+Update the comment block at line 9-10:
+
+```yaml
+# Admin credentials managed via ExternalSecret
+# See externalsecret.yaml for 1Password integration
+#
+# NOTE: erlangCookie is externalized to 1Password for architectural
+# simplicity (ExternalSecret ownership model). In a pure Harbor pattern,
+# this would be chart-generated, but single-node deployment makes this
+# acceptable. The erlangCookie is treated as an immutable infrastructure
+# secret (generate once, never rotate).
+createAdminSecret: false
+extraSecretName: "couchdb-admin"
+```
+
+### 5. Commit and Push
+
+```bash
+cd ~/git/homelab
+git add cluster/applications/couchdb/externalsecret.yaml
+git add cluster/applications/couchdb/values.yaml  # if modified
+git commit -m "fix(couchdb): add erlangCookie to ExternalSecret from 1Password"
+git push origin main
+```
+
+### 6. Verify Deployment
+
+```bash
+# Watch ExternalSecret sync
+kubectl get externalsecret -n couchdb couchdb-credentials -w
+# Wait for: SecretSynced
+
+# Verify secret created with all four keys
+kubectl get secret -n couchdb couchdb-admin -o yaml
+# Should contain: adminUsername, adminPassword, cookieAuthSecret, erlangCookie
+
+# Watch ArgoCD sync
+kubectl get application -n argocd couchdb -w
+# Wait for: Healthy/Synced
+
+# Watch pod startup
+kubectl get pods -n couchdb -w
+# Wait for: Running
+
+# Test CouchDB access
+kubectl port-forward -n couchdb svc/couchdb-svc-couchdb 5984:5984 &
+curl http://localhost:5984/
+# Expected: {"couchdb":"Welcome","version":"3.5.1"}
+```
+
+## Why Not Follow Harbor Pattern Exactly?
+
+**Harbor Pattern**: Only user-facing credentials externalized, internal secrets chart-generated.
+
+**CouchDB Constraint**: ExternalSecret uses `creationPolicy: Owner`, which takes full ownership of the target secret. This prevents the Helm chart from adding auto-generated fields to the same secret.
+
+**Options Considered**:
+1. ✅ **Externalize erlangCookie** (SELECTED) - Works with current architecture
+2. ❌ Chart auto-generation - Conflicts with ExternalSecret ownership
+3. ❌ Dual-secret approach - Requires Helm chart customization
+4. ❌ Disable ExternalSecret - Loses 1Password integration for admin password
+
+**Decision**: Pragmatic approach wins. erlangCookie is treated as an infrastructure secret (generate once, never rotate), which is acceptable for a single-node deployment.
+
+## Secret Classification
+
+| Secret           | Type          | 1Password? | Rationale                          |
+|------------------|---------------|------------|------------------------------------|
+| adminUsername    | User-facing   | No*        | Static value, hardcoded in template |
+| adminPassword    | User-facing   | ✅ YES     | User login credential              |
+| cookieAuthSecret | Gray area     | ✅ YES     | Session security, periodic rotation |
+| erlangCookie     | Internal      | ✅ YES**   | Architectural constraint           |
+
+\* Hardcoded in ExternalSecret template (not fetched from 1Password)  
+\*\* Pragmatic deviation from Harbor pattern due to ExternalSecret architecture
+
+## References
+
+- Full analysis: `/home/hermes/couchdb-erlangcookie-analysis.txt`
+- Harbor pattern: `/home/hermes/harbor-simplification-complete.txt`
+- CouchDB Helm chart: `apache/couchdb` v4.6.3

ansible/ansible.cfg

@@ -54,7 +54,7 @@
 
 # (pathspec) Colon-separated paths in which Ansible will search for collections content. Collections must be in nested *subdirectories*, not directly in these directories. For example, if ``COLLECTIONS_PATHS`` includes ``'{{ ANSIBLE_HOME ~ "/collections" }}'``, and you want to add ``my.collection`` to that directory, it must be saved as ``'{{ ANSIBLE_HOME} ~ "/collections/ansible_collections/my/collection" }}'``.
 
-;collections_path=/Users/rblundon/.ansible/collections:/usr/share/ansible/collections
+collections_path=/opt/ansible-collections:/usr/share/ansible/collections
 
 # (boolean) A boolean to enable or disable scanning the sys.path for installed collections.
 ;collections_scan_sys_path=True
@@ -209,7 +209,7 @@ private_key_file=~/.ssh/ansible
 remote_user=wed
 
 # (pathspec) Colon-separated paths in which Ansible will search for Roles.
-roles_path=/opt/git/homelab/ansible/playbooks/roles:/Users/rblundon/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
+roles_path=./roles
 
 # (string) Set the main callback used to display Ansible output. You can only have one at a time.
 # You can have many other callbacks, but just one can be in charge of stdout.
@@ -262,7 +262,7 @@ roles_path=/opt/git/homelab/ansible/playbooks/roles:/Users/rblundon/.ansible/rol
 
 # (path) The vault password file to use. Equivalent to ``--vault-password-file`` or ``--vault-id``.
 # If executable, it will be run and the resulting stdout will be used as the password.
-;vault_password_file=
+vault_password_file=/home/hermes/.vault_pass.txt
 
 # (integer) Sets the default verbosity, equivalent to the number of ``-v`` passed in the command line.
 ;verbosity=0

ansible/create_jarvis_user.yml

@@ -0,0 +1,6 @@
+---
+- name: Create jarvis user and deploy SSH key
+  hosts: all
+  become: true
+  roles:
+    - jarvis_user

ansible/group_vars/all/semaphore.yml

@@ -0,0 +1,193 @@
+---
+# ============================================================================
+# Semaphore configuration-as-code
+# ============================================================================
+# Drives a freshly-deployed Semaphore instance into its desired state via
+# the Semaphore REST API. Idempotent: every object is checked first; only
+# missing ones are created. Existing objects are left alone.
+#
+# Loaded from group_vars/all/semaphore.yml so that the configuration is
+# version-controlled in the homelab repo and survives a wipe-and-redeploy
+# of the Semaphore VM.
+# ============================================================================
+
+# ---------------------------------------------------------------------------
+# API connection (defaults to the local Traefik-fronted service-name URL).
+# Override semaphore_api_url to point at a specific instance if needed.
+# ---------------------------------------------------------------------------
+semaphore_api_url: "https://semaphore.local.mk-labs.cloud/api"
+semaphore_api_validate_certs: true
+semaphore_api_token: "{{ vault_semaphore_api_token }}"
+
+# Feature flag — keeps day1_deploy_semaphore.yml deploy-only by default.
+# Set true to also run the configuration pass.
+semaphore_configure: false
+
+# ---------------------------------------------------------------------------
+# Declarative configuration of the Semaphore instance.
+# ---------------------------------------------------------------------------
+#
+# Top-level shape:
+#
+#   semaphore_config:
+#     project:        single dict — the lab uses one project ("mk-labs")
+#     keys:           list of credentials Semaphore stores
+#     repositories:   git repos Semaphore can clone
+#     inventories:    Ansible inventories from those repos
+#     environments:   env-var bundles
+#     templates:      task templates that tie everything together
+#
+# Each list element has a unique "name" used as the natural identity key.
+# ---------------------------------------------------------------------------
+
+semaphore_config:
+  project:
+    name: mk-labs
+    alert: false
+    max_parallel_tasks: 0  # 0 = unlimited
+
+  keys:
+    # The ansible-vault password. login_password type with empty login
+    # — only the password field is consumed by Semaphore at runtime.
+    - name: ansible-vault-pass
+      type: login_password
+      login: ""
+      password: "{{ vault_ansible_vault_password }}"
+
+    # SSH key for the gitea deploy access (clone the homelab repo).
+    - name: gitea-deploy
+      type: ssh
+      ssh_login: git
+      ssh_private_key: "{{ vault_gitea_deploy_key }}"
+
+    # SSH key for the universal automation account 'wed' — pre-baked in
+    # every mk-labs VM template. This is the canonical user Semaphore
+    # uses to reach the fleet.
+    - name: wed-ssh
+      type: ssh
+      ssh_login: wed
+      ssh_private_key: "{{ vault_wed_ssh_private_key }}"
+
+    # SSH key Semaphore can use to reach the fleet as jarvis (admin
+    # account provisioned by linux-baseline). Retained for jobs that
+    # specifically need jarvis-level access; the default is wed-ssh.
+    - name: jarvis-ssh
+      type: ssh
+      ssh_login: jarvis
+      ssh_private_key: "{{ vault_jarvis_ssh_private_key }}"
+
+  repositories:
+    - name: homelab
+      git_url: "ssh://git@gitea.mk-labs.cloud:2221/rblundon/homelab.git"
+      git_branch: main
+      ssh_key: gitea-deploy
+
+  inventories:
+    - name: production
+      type: file
+      inventory_file: ansible/inventory.yml
+      repository: homelab
+      # wed is the universal automation account pre-baked in every VM
+      # template. Semaphore uses it for fleet-wide jobs.
+      ssh_key: wed-ssh
+      # become_key is Semaphore's sudo PASSWORD slot, not a second SSH
+      # key. wed has passwordless sudo on every host, so reference the
+      # built-in "None" key. (Semaphore rejects an SSH-type key here.)
+      become_key: None
+
+  environments:
+    - name: default
+      env:
+        ANSIBLE_HOST_KEY_CHECKING: "False"
+        ANSIBLE_FORCE_COLOR: "True"
+        # Semaphore runs ansible-playbook from the cloned REPO ROOT (not
+        # from the playbook's directory as I first assumed). Path is
+        # therefore relative to repo root, not playbook dir.
+        ANSIBLE_ROLES_PATH: "ansible/roles"
+        # Collections are installed by the semaphore role into a host-side
+        # directory bind-mounted into the container at this path.
+        ANSIBLE_COLLECTIONS_PATH: "/opt/ansible-collections"
+
+  templates:
+    - name: "day0_linux_baseline"
+      description: "Apply the mk-labs Linux baseline to one or more hosts."
+      app: ansible
+      playbook: ansible/playbooks/day0_linux_baseline.yml
+      inventory: production
+      repository: homelab
+      environment: default
+      vault_password: ansible-vault-pass
+      arguments: '["--diff"]'
+      survey_vars:
+        - name: target
+          title: "Target host or group"
+          description: "Inventory target (e.g. figment, semaphore_server, all)"
+          required: true
+          type: TextVar
+          default_value: "all"
+
+    - name: "day1_deploy_semaphore"
+      description: "Re-deploy Semaphore + PostgreSQL on figment."
+      app: ansible
+      playbook: ansible/playbooks/day1_deploy_semaphore.yml
+      inventory: production
+      repository: homelab
+      environment: default
+      vault_password: ansible-vault-pass
+      arguments: '["--diff"]'
+
+    - name: "day0_linux_baseline_check"
+      description: "Dry-run the baseline — shows diffs, applies nothing."
+      app: ansible
+      playbook: ansible/playbooks/day0_linux_baseline.yml
+      inventory: production
+      repository: homelab
+      environment: default
+      vault_password: ansible-vault-pass
+      arguments: '["--check","--diff"]'
+      survey_vars:
+        - name: target
+          title: "Target host or group"
+          description: "Inventory target (e.g. figment, semaphore_server, all)"
+          required: true
+          type: TextVar
+          default_value: "all"
+
+    - name: "llm_inference_multimodel_stage_models"
+      description: >-
+        Stage additional GGUF models into /opt/models on astro-orbiter via the
+        llm-inference-multimodel role (--tags models only). Idempotent: skips
+        files already present at the correct byte size. Notifies the
+        llama-server-router restart handler ONLY when a new GGUF is actually
+        downloaded. Does NOT touch Phase 4 (verify) or the legacy
+        llama-server-qwen service. Safe to run repeatedly.
+      app: ansible
+      playbook: ansible/playbooks/day1_deploy_llm_inference_multimodel.yml
+      inventory: production
+      repository: homelab
+      environment: default
+      vault_password: ansible-vault-pass
+      arguments: '["--tags","models","--diff"]'
+      # Scoped to --tags models:
+      #   Phase 0 (discover) -- skipped (no tag)
+      #   Phase 1 (models)   -- RUN (idempotent GGUF staging via stage_model.yml)
+      #   Phase 2 (systemd)  -- skipped
+      #   Phase 3 (firewall) -- skipped
+      #   Phase 4 (verify)   -- SKIPPED (collision risk: verify.yml would start
+      #                         llama-server-qwen on :8002, conflicting with the
+      #                         production llama-server-router.service. Excluded
+      #                         here deliberately. See t_730f9584.)
+
+    - name: "llm_router_update_unit"
+      description: >-
+        Re-render and reload the llama-server-router systemd unit on astro-orbiter,
+        then restart the live service so new args (e.g. --models-max) take effect.
+        Drives playbooks/day2_bump_router_models_max.yml. Added 2026-08-12 (t_33acbb2e):
+        bump --models-max 1 -> 4 with full VRAM budget note in host_vars.
+      app: ansible
+      playbook: ansible/playbooks/day2_bump_router_models_max.yml
+      inventory: production
+      repository: homelab
+      environment: default
+      vault_password: ansible-vault-pass
+      arguments: '["--diff"]'

ansible/group_vars/all/vars

@@ -19,6 +19,15 @@ terraform_server: "infra01"
 # Traefik variables
 traefik_server: "lightning-lane"
 
+# ---------------------------------------------------------------------------
+# JARVIS automation account
+# ---------------------------------------------------------------------------
+# Public key for the 'jarvis' user provisioned by the linux-baseline role on
+# every host. Public keys are not secret; the matching private key lives on
+# the JARVIS command centre (carousel-of-progress) and, when needed, in
+# group_vars/all/vault as vault_jarvis_ssh_private_key.
+jarvis_ssh_public_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID5sym5ajFvDyzw395BkHv7qVb66XPTx/OF1p19MGuNo jarvis@mk-labs"
+
 step_ca_principal_mappings:
   - local_user: wed
     principals:
@@ -30,3 +39,9 @@ step_ca_principal_mappings:
       - ryan.blundon@protonmail.com
       - ryan.blundon
       - ryanblundon
+
+# Leviton My Leviton API
+leviton_email: "{{ vault_leviton_email }}"
+leviton_password: "{{ vault_leviton_password }}"
+
+jmri_vnc_password: "{{ vault_jmri_vnc_password }}"

ansible/group_vars/all/vault

@@ -1,306 +1,365 @@
 $ANSIBLE_VAULT;1.1;AES256
-33333438373661633337383161343862383963623736636136653339363064386538386437323135
+66393233316132396639356564316439343234383066633231646134313361666463656536323732
-6436666330363032616365656434356161396530363363610a333864313064633232366133366232
+6630616536646439613533363430306466306233643730350a343364633233333335643833326163
-37316636366334346536663337663065303635626638666264666435393933343832653061323237
+64393933613963313533623733316339396236363663343635346663323366663166363839663837
-6663333736386636350a653931626665346234323832393334373563303130626262646565653231
+3331363062653239380a326566623264623837326636383939346430666537613361333638366630
-30326263623965356639386438636531306430343239313162383065386135653534636134376333
+63353062393335316663633739313532366363623739653631366539323435336361353331386230
-64616432303732646535643530353963363734663538393539633034326535643539356561656632
+39366634643964336233353961316630616462663166316266613037623363346335373638656365
-36363330346131393262326561356631343631356434666264616134643136646238376336373665
+38353733396636386133373836346336383231663661346137373164386338623733393566373563
-65346361636435316437353235393263313766623438336662353532306463353134366435326364
+35653933343036633365643535303934326537356136666539316137363433643266346630386439
-36613865343264616364333631353238663238313338333232626262313637633662633163366638
+38613332646238366536333536343031356532656336613530663830613264346339353034323362
-61323166393739656536656435336538303439636561363961353832313431666432393463636537
+66613530626361323535653232313730373463373332313561616631393461353730653464343063
-62323939376265353330356334303334303532643136376531343765613738656562386665336132
+38616338363938346161616636316232313838616463326432353639613837343162646363343232
-36346232313432396431666535663362333332613037623531636135376439393936353261653863
+33323064363139376566343866626364373662393138353666646234373461666163363139313631
-65373061316461303866363338376131343736323933383636626338616431656533363833663266
+64343261326566363265323463663538343034306136326234386664333837333937333136653563
-62363935313262393733333566386337633630666332353263666462373164346362313034636239
+61333531353434633339383661636363363535316366353330313566323133616438373161303135
-33656462396238343065613965613232623132343562323364653366313231323531326364376631
+35353630613037316466353832333033393030636331386438393133366333653832393731366363
-65643961303862653030343364326533356336393031636437393465636263626236303633326139
+62373638303737393162303461646239653865653834613662666636373364633165383062643831
-65643432663037333332346261323663613031386563383533336538383133373332343363326530
+34386232376361323638353361666530366432356331353963303930326535663536373339333062
-66643231666636366435623639643166333863376533633537363364396663363732333932663333
+32396266373430343339636635366434313635313766363863336464633961666332353834626163
-62333239613264353233303064333931326236396538323130353061366139623362393734323434
+61653637316163636465343630353431313863653033643237356434313564366361373435376662
-36343537393363343861633131346561353637646336313233376162656561623362653565623339
+63643737353830663236643862613533623237373531646136383763303766336139303632666235
-63386337613461343534643964323932356661383436303966346339386131336133643132376330
+64623834323966363363663730626437323432623966663537346162656265363562643836633731
-38656535353335303966636235643666306336396435636434363733313665366234393032333065
+65663631633462663764393132326165346639353033633035636432613039336164303538396632
-37633231376134636435343233363032666231666134623365653238373836326237373831326262
+66396264393865306666643636353638613661313230313337383634663839363439656533333932
-66303234303164303961316166623062616139353264343864363938313733653563663661633535
+36633432306131396539386539633063653230363932376264323537396434353364643432653661
-37326636346435626362336437666364623264363565653935336438386262376663303230313265
+32643162363066636432336363323534316436613838646562313538326566666239633234646236
-62663665666132313436663330616230383235346333623563313262376530356535613263316436
+30346534636533623365326564613561363362333364363037646561656635623935653466613565
-35313766363763386234333733636464326134633136346532346632623534646365623136343363
+65646361313436356261643762313339333864356338386136306162386262636464393130303963
-30316435613031333036383835656230353430636135343961356363613139353763343233633834
+38646464316432326431326661343632396235626234366133353461623862316662326432356234
-38656431363139396535653565633631393462316334666464333839343035616531643466643531
+37626366383861373831633639616465663564643866356664623066386535646163336134356534
-35613733373331643961353830376266376433363938373563653833343464633465663766333066
+33366664616232353863626465626364313530353335306565336665663866303736323162393362
-34633364313762633034633461623232656537653762623532613332386132333365383062653136
+63626261653161663664363833313461653034326330653835393737616135646462366665383935
-63323234626266336462333534636236653230343530336532653831313339613531343731313237
+30363639306330636634386433646231363530633061336364313338653632323831393630383934
-39653737663665373361326433643364656434393838326339636238303964366336363961653366
+37316362326338313733646332336263386239626539383330353362616132333161613464313066
-36643938633635313432663765316130326238306163346566313062626439626263633033623731
+34663434326662326233363432306433363666356132383866346336336261636435366332666135
-63323532346366653238663931653334613734633963363632643462316130393138363436666532
+34616231613638363339356333616536643266636363643131653330396162306264303566396461
-32373833323330356164626332646232623031326365653765336261323661616137373732356164
+64363763376365356533636430643866333361363062376237653237663731663934306265646630
-63383633353764643233323430333233306337623834353934643864636339646239636430316166
+33393637656335643366383564373966343265393630333835303731316339373133633462383364
-33643030333261323461643031613730343131343830353463313766353035326436306566343161
+33383435383331303264313334393532373932333334343862326635346135613932356337373034
-35643534323763316165343832346433356364343061383036306434386631356537363331303035
+62316262326331313135376465343336373266663338396533666431616462613932663861646238
-34303430393034393763343964613935383739656662643066363633333839396465386231383837
+62653563623535633738383033326235383666646333653731316233376231623661306462303732
-30336331393365346636363732616661633635326364366435366466383639353839343731616365
+32643064373236613336396233323435393939386530323331336138353364663762356538316562
-65326633636263653438343561633632646234316461616666363630333966326230356331373335
+33376530623664623733386133333433303031373337313366386236376539613964316135343865
-31353665626233663433633563623765343431376639316232346666663065656462616438663333
+33363963366165333238356663663435386439336366646138313034343636653463323938633136
-62663662336662346563333935333261626635343764356631356363356362346231306535376466
+39363966376238306662303265643034306136663661393738633436393432303139313132616534
-36393932666662653934626134613065623461303834393231616332363461306534303666353632
+66323432313635386162333838323136623634653264643438303264636430633232323434666532
-35646363333065356631393863396536633838343365343238636332303862306631346639656166
+32616664663063653735316237643539633133356661333132323238376333356464313262653836
-30373735336361653863326331373834663164393863666631623866353338366561326635323132
+39303566316332663737323437633031353330333365383837636336643763313433313937396531
-35306535626564376666623837666435663337613164623966306138353161313239373436346239
+38363536343438663966663436613132663661613134383431633765383164373762343435316161
-36656266323233636233326139326266346464353665323465663666646264613466636132626464
+62303631646235343063383230343232383336356562303563373933346530393333316634316437
-37353762306538643233663338613062353134323832653139663036666337643131613062643936
+32316330306163396434663031393965663163666537353031613365353437666466333464626238
-37633363376365356333353433613839653130323036643737633163336531323032393937333432
+35313739646535356665323734393965303064306132626261363062363438383164346261393463
-39343238396534353330346664346135326333333638666261396463656663323935643337313462
+30643438623363323161323230306230386332363635386234666639623566643536626637616533
-32396137613432623664323134356361323861306230353165663162663732326563626162346630
+37396136643930633262333331656363376433333234343630306535313262306235663263663362
-34626536393962393764393764393234373138656432346332633963393135386238313563353761
+32653434363035613732363136303363393939323337613661333439393637646262383039386661
-34366438353337396264373032336166663937343931633635666166343638386237396333626133
+62326163323562333339323636363565623664396164383332633666386130613766393138346134
-38383766616566336163363931393438366566386565663938356630366430393830653037363738
+33343338393536316431353439353062663164643634396363353131303038353965393466383030
-30666465326266363834653934643434666466326632366463303965303366326364316430366561
+36656465383938353936346361393963356630666630373236626237303064303062383638373730
-32636631663232373163636535366333623261366466623262393631626334626263656534386536
+35633866646535313432353338623462323235346433653431313031363163393666626432363238
-66323833353264623538663232376138346361646362313565333534363535663031643833373064
+39623361316132626230633336636163623466313666346631656134343762656566353432353264
-38313062346339636335633962323933383230623634303431383236626166636535326465316439
+30353436356237653231363564626134633039363035313232616333336436393638396233626638
-38646636336130373763646635386265373065326263616438646565313439613830316135616435
+32663230396539323761313838313466376165646430346634383332346134653662393161363337
-62326430336337643265306261343036653938633634626561333664313035393933646533326661
+62646161343665383364306665333164666231386531626465373366623761643161656462303733
-62626437306432353938356338616462663130646530623636393233373837336138383963323165
+37653438616233353432626466623163316565353764323762613635333832343634323665356336
-35643231613565313434386163643637633130626435376561376235346438303234386637383835
+35353162326233333836396337356466636131383838313436626336663132346339623261366465
-37356238616263646665336435306233396436666664396532373063626162346237376662393639
+30623261303933396562353331636638376135663330643638643536346261626632626139386535
-35373665373638396434353062666430633362653163633639623836346438323632643164653638
+66653332366361336636666437643165656239613031303638333232303836383132616636633938
-62623463343465303236346562386338646536656438633966643430343266313935323630616633
+31343034643037623731643931316463303639656266323231313666356336333133323135363330
-62313531343764343966333439343866613965303036616634363162303434643533366465313562
+63373365303131353161303630633738353536393631353034666139383435303461316131646138
-62393666313838313938356265633531326431333730626632343139666338656465306431636162
+61333731356538366366613831303565613365633965323235366166313534653965366433656533
-66343532346238393464366134656334363964626264623263353537633638346239396539653538
+62666136313662366638356237343734336333313034396465346632336262306531633535643238
-37303339613036306533353132326534306433353364613536316263326363323134333639346236
+35333831366532386235316565303936616264373337356134643066396531383533353336303131
-39376263313431666462383863336638306631663563313364333235333338323364613361633532
+31393837623564386535323532653733393734393164373235396566333565356237356438313762
-38313263353539633662353533646333353262356436353938303036663130306362303935356136
+32623765326639386262393639376461326163333237313232386138643130643231626466643663
-63313234363138383864613664666635613464653765636461383738356135623662313866383036
+39643061393566353434333136366335393536376234366266376265333234643536633035653933
-64626437343465333563346434323762313232643230323336366539326631396463303961666461
+36316132663539306465343039323935356361373439346437386234386464623962643464643562
-61656338643532616337653362636362373837373836666535633762343335623033316264366261
+61336434613834336161633237383361303930313464613666313834356330343138633735386530
-62633137356132306430623537616135613965393737323263343463633839393165646630383331
+36616233323366323961653965613438346136373738366266316134356266623664313539636235
-36616339346534346636396532333836633436323232323364303963313030656330386336633538
+37313033373466383134346361646562366531333338386330653736626530396238356639303131
-34383534313236396461653462306130376462396138303561636331663062303864326238303335
+38363738396236386461316433316261326435646130383336316234363461393237623633633336
-65663832333631613963326639386665643135366632393438666361376334636233336239633837
+30386630376565646337383738663939663462623232316635346635653830306664653336343033
-36383761616364363262306535653065303536363066316431303464373230326436336661396231
+64316430396664393532313766326437636636626232613036666666656430323136356436333564
-66333438376664333731333630383031366237396638326534636532343535303964653436633438
+36303334303562393832336433343438396430373833623137363736386665343866313064353063
-64616263366561653334346138393866353437393037343930623237326231303261383733336232
+32303634393131326464656535633734386462646339663533666430336265653965333538633866
-37646533393439653965386563333232656565663531396663303334313839323361363864396332
+61623666643839653239373335633735373738363736313665323365613635313766656635613832
-61656638303130303433666561646663323139643337326561333832373538306236326538363637
+33616461643539636165383233636533626230343138663630323731626139393230383464313430
-38666534633330653564653066643335623938373331376339393338343533366436333330313165
+61333438393337316239376435313337313437333931623238616133666138363235386533633437
-65366562386366666632653064663637343536313462326562626436356363306134306237613265
+35356163363231656536353934643539643562343732626630383565623730626533313230656164
-32366338643237333739626331393763323934396265383138633035646533353536633866313432
+35333735666135343364663233626163363930383262363266303265303638396239636361366534
-38336137383036373434363530353833653938663035306337643638316463373762613931373038
+31333863333565356135613232393165353266343632633532343061663331633337343538376265
-32373361326136616331663733616133623936663530356566383461336334663937383230386462
+39316630613439356262396634316361356436336634396337353339616536356336653930613966
-38656332613763653339363565613762383735333163633831356364333563363630623666616430
+34656439653366363562636639346430623561303463356337363830373966366632303337663564
-30353237653730613361643233386139303639666137653733353334326530306431376666663664
+35663632313265323365636238303364366230353039353561616636633664643233343430336237
-34633531393261343165663330373961646665396139323762313663376639366464306533623333
+63373264643935616331616632633065366638363833306337633563653065363464343137623533
-62326432623334336166656565656230313063656163636561326632666438633966323930316264
+36373231363739373335346464623533393336613634333636613937366136326464336332346166
-66373866663461616336336264356561373232313737353836646632383438333162383832616662
+61376263623835646163353134643963663964373732313833346163323138633230393537636664
-30636437613163613631623165366631306438643765383263626133653231333361383633363136
+30366234303334656130336630346130656237306161376566336534653630616439323764373665
-36653831623562653838633133646633393038666164346461333531373034356435363336636338
+61383338326163336164353265326163646165623235626137623237306666333832306461613630
-62623464376264363537306166613461396630303039373239656664396564333435303164613235
+66373331356465346261643466323662393661623433383265376666623932343861323139383531
-61346238386233653462616632303337663036626465663361636130333566303337313237626237
+64633536373362643935633734366235396433333237306166646164363930613862613365303663
-65636633316366343064663636383362316665633263633136323563363935346138356635653036
+38343833336137353634313362666665306666393635663633353934363832343739616331386130
-36383835323838366363363335323731333066393334373265653563336533616362633163343661
+66336561633039326434313833303465366638303961626138333165623331386230616130626639
-66326630353537653633316631343332303231373630363635623136386537666434383032323165
+34613962366230333065633761333335613636363533656461626632343631666563383738623330
-33333633643064376239346139633664633330323966366430353135616633633465363030386665
+30333834346233653938633330663166616331376436356533366461336264643264336139343262
-36353164623564343934323063313763316566373830643163616633643937323732333561353832
+35393665656230663232366133393037643536366234343537326631623332373131323739363638
-66623364666335316230346665353631663465633034323265666663653866623037353935393262
+39653162646366316639313631393631666261623230313538613666393732626438393763646330
-61623065656637636666383239343661616233396663333038393963366537326136653863333235
+36346661313131313630343432616365666633353762623261613039623331396330623939626132
-65633137626134383763663366396438366634313465363965343830333966373930333134353339
+34626333386538326434356432623965666662663437646237373537326534653634346239653634
-35396538303537316636333339613731623636646230323662613663306537326566656666323434
+31373038303639333037613637393862356263323066666630313262366633313932396465633337
-65346531623533336535393530373963343864313963666335636334396663393837396665373565
+66653930303934616236323064613761353935613835356561313334323762633064306661346666
-39646138356134313732666633663061646330643438393736653433313238373135656636643337
+37653262343865386236343634316336386630393739626437333065323433613531393738313432
-39396362383363333434376336613933613438653636663334336337653633613433623336623461
+31376233353463373237653164386363633334366332356538343966663939656165323465333030
-32373565306435646438306365343465653863656434663937653938393839333739363639326536
+39656532363363333432626638626438396539336461326338353732376235316133616666316261
-64303534383961333333386237366239386330386533343939613664363439373135333731393739
+37353063343366376433653961333233306461303133376661303332386230346231383837396133
-39363166306636333766333332393265613133356262356666343039666237663664636266626664
+37323137343066383966343535633363643233663530613566313330336232366638396165373631
-65316339323638353333653631363365356562623535666235313234613236326130616261663830
+30626531363033313833303836366434613736396339643032663066333865306535323739666162
-34663063626539363734313562656233656131326265626637353239636638396564313561383761
+64626133616433653864376662623464343131303938303237316264393765303035663833376464
-62663638623135343333316464346236353364616239633566393039633339306334623234616331
+32663264383236303766323935306463643138396237373338653238633464616238306132633735
-34366435663938346335323932633330366639653636336636373562306561656536653363366233
+31626538653262326533326266336633623532623935383266373533363466313033393235663538
-39383064623435356338636335343662343435336564363965646439393964323162363736346637
+66653038646233303665343634383666343363383238326533366136363838303332323230316662
-65323765333364353831396433666561376437333834313334316165386631616332633533666336
+35383235646638653539633961663036663933306463626335356631646662636230356261363261
-37313533623535313934393230353439363231623063663961376537393638636237623134323337
+65633261353830373865636630353932323937666331353635373736376436333361613330366633
-30393032333530316330333736616463633262616631353238323933303766646137386465643366
+30663939356165393132636131663966373433623063356265353131306532643066306630656363
-33356665396437623732373437373537303034353431353031313162323730316566396238653038
+66636636353262633437663264613266613663656137386231306231646264363661613035343538
-30366234663037653736353137376264363934646366353761323364663538346662323934343266
+37353633643065643236376537336238663137623735613038623766393231643131653436333262
-30373662613461663561623466633939396439653663353365346133623933613462623231616562
+31626463646432613563393665346532386161366435396364663239386236616233356131323536
-66643738643661643766323266626266313330633936313964366536323263373666303130623738
+33633936623762666534633862363466353736386137636363633733623366346337613365636439
-32323230356665323439646532343930333936373536626466656566363161646663616266326133
+66663035313430386464623833646135333062313830396637323961386135363461326539623432
-32393137663562613531323832313565346632303131646432363961613331346636653830323136
+32653865623530313637393561343465636430373162333162646631643235653931333830326266
-36323436373964313737313131346131353630636561636639316464323666636336656265343565
+36376631316165343631326165623838306239623764363262376634663236393933343838376663
-65303936353165326137613638366532303265356537343337656165366663383961643134656435
+39663834306165313330393739363133396436376437643232346336386531356638343063376465
-61383565306265333939613561613035616663336361396536343939343761663833346134646365
+61613037623137306666383231376539656361326132396662613061376134376266633764336266
-34323233666439623436323261616432363838616437386138393735383364393164646638373630
+64336334313335643635303632666431383637306334376462643630646339396435313830313363
-33386632346239383037633132636431366264383633333365306338333661303662363533376339
+33383162316261663035393962306234613865613366353465373035656434366261383133653331
-62633663366362303038616666336335346332316366353136323666346531393938666236343163
+63643235616362663663343330303765363263366130393837613939323264373937333162636639
-31663665643839313439663839353431343061386465366331663637373532613266383065393639
+31643438666338646135663538343231643235646364623761653064633566656663383465626133
-62613565626135363133623261333061373738633333623337396631356332643039306638303434
+31373935646266303565303539376162623132316438623565316537306337636630313861623937
-65663762373630323263333239613939346133626534613632396433376234663636363062656537
+34313832636533623033616139373965303839356530353935643363613464356364343162336466
-62386664343462346334663662663865653766373738306438316533643962356136653565373761
+66376130653162666661313139613530306666633432346639656466653364376435636461626362
-62393437326138323065396630653838326138653865643937353431656137393732346361353734
+32653633303561346233643463373534653434323134353434373839373937626663336464303866
-31326634653331353232633061303230393462363065613832303630383237366463363666613864
+33363264623038313835396231373132396163363662626264346461333539326365326165323066
-37643135303034326335326563353064343835323863313134616262386564313235653762376332
+62333139383334333334353031616430323339623066363232313937323465356266323934313761
-36636533646235343138396537623266333566633466393764353731393935306432313266636133
+66623835653961303830383030643537393130653935313265333062393034336562633535323263
-63326362653033613264333139633638643638326532313239646239386332323330306538366362
+65616237373232336534393834653162363461336262653862666637326266663966356665363036
-37646432363535396161336465376363343430333261396536346464356135343462663462613136
+35383437326465663635303664643236633435303862633965346133376536316233386333313634
-36666238633663326230373834303431336566366666666534343537313635366537396262656236
+33643565326633386565653961646463383866646636303537643436623734393234633938333933
-36646639643730393134613662383238623835626230313833653932303832323366343765343434
+65366164633165623333623362393639656661326332306538663738356364373734316563653038
-34363033323836663035363166313630313062663162313165636138373261386636623831666632
+34646531616662386232613034366332656262343164333531353037363036646262623663666236
-33373962626237666130373031633437653966386561363232363332333633303961323864333263
+38613238666136363431623664633863636365396236666532383930336636353031396232656435
-39666230383332376537313137643635623336316336646537333137633564376432356432326639
+61346238643431653231623861373964383931336535363262373437353532393165316562386134
-61383565313233376435636366366531363462306535636131636163636130663732383161616364
+36363263666135646237383666373833373737396330616163376439663736663937666161313831
-37343064333264663533343739356139353365386565383833363165326536306431393764303632
+63663531656635663339306365656663636633343733636165386230376332616331313638386538
-30653865623566363965306636366530303261616663306433393135656366383138356337383365
+32386466323232363533613334333333346161376430373436373961316564343061326164306138
-36323736643035383231663034626564643235646237623632383164633062326132363331353634
+33616263666262323430303730626266396535626439623364376239346564323730323534323938
-61653761346439326665366565383534393635616365353361353863363763383664303166383434
+33346364393033353865393864326361643734353234613563393138363334383536396535393166
-33393732313837373336616564343864633330663566653933356237613538653366663838383132
+34623163616336653436393639313965353237633566313039303137326234383230323235363234
-33303437323438313837616134616466626562313262653866313034383764656566373331343738
+37626161356166356365366164363863636563316332393638616535376466343537373966643839
-30613062303630663365636230396231313065356662376163353632613564363066316239636434
+32613930643533336264626136626465303339376632323034386161663661376466616233633065
-66356535386263353737383838653538316366343935323337386561353533353161636538646363
+66313739346162363838346663623266383130383736656334323430623463666439386532643630
-63643739376366326536333261306263653563353331393133323964653764373932663136316335
+33666639613830386136363535363830333234653961663739343537306634616531616263623762
-36333535626231373864316634613366353535393932373039646161646466363663626561373033
+64666230373830636238353062666330623061613663376638343763626264363130313464383661
-64333133353931303361363466346666376431363336323166356162363537633536336638653838
+38326530333362616163363735323861376366333665623536383566653837306131623732373639
-35323562653138393833383130353232616564393738326638366565616233633634336438303632
+31316661353332633630326162663738636562336666326637353764323431613666303038373532
-30353765623565336330386335306162336662643466663335623836383238316161633339373137
+62343661336338306561356235396636343130633365303466613637633363613862663233633731
-32393165323735643034633464313337386362363331353432363233396538393835383933313637
+66623530353132666261316637303763363830623734346262333633646238613131346564303734
-65656330383732633537646361666439303266653165313233636464313964383639346635376538
+62336434353432326239333232383833633962313537626430663130393733623162626131656366
-64633861623439626331643431326336646135656539313836323135373862316438393237663564
+64333535623138326239336165666562376663663334323036323539653734333835386331653438
-31373135356135356662336265633862343435356530393835376430666566653036373338636332
+63353861666239396437346361306634613462386335376137333963333838616138633730393865
-64663230323361356262356537656262396665663538386664363735646234306231613339636338
+62353539376136316564666136646639363635663736636439393462633165646632623664383663
-34366433326535353965376236643661646339366566663531653864316533356466653864373633
+31613137306461616361323832393036323933626531363536336261356636303531633239333362
-33363330333765373662646634326236313434383433653563383436316635353461386662646165
+32663134363263383039646162643539663737333861386437326337616362343963373532346238
-31626566616163373137326165323066646535663766636265386435633965326165303831323633
+37346137363933623839373838353939386630303461346438666534616434333031373730393537
-39613139623835316165326138386431383530383035653931616334353235636138303761383037
+30313134643963623564356266656430613430626238613266316335336265613132616562626261
-39336163373332346365333935656563343939623335313461363138656239666263366236663464
+35386435313933626634616463616166646466363939313639646264346464363337656339323366
-32303732663937363964346639323236623166373334386239613564643834316536363634326461
+36366665363739356564363232313762323565323134616134666337336534353464373637373130
-64623939363635613863613065656363303734643766306564326437663037623839303734393036
+63613265366436313131356332316531633732356461383064383031613337343363646432373936
-64326363626562363934313930383766383462663766336666663338653765656137393362643366
+33633339386632653032663837346130623636356464326637303338376132623734333932396232
-36326336323638393161353464396637633566383262333234656233653334623131323532376637
+61333033386265356630316134383066343164613130666664643732643362666561346132656266
-62613064626532616631646232386631623839326665346431333232333463373031316237363462
+31623633333039633837383264363937623435643061393935393762346430396335373864633634
-30643731653666643331643839343865646437613135346363646131383262396134376363303335
+33336136353332663366313334353739303539633364663231636539333132303966383432376262
-34303164303236383139326264623462363062313139653465346435336163653863633764613631
+61356563323232613433653262623663336634626532653465306638316633663564633862666666
-33623666316530353163396539323238663935643764643939333966336137353034663164643733
+30366133616336326661626238653933383164336366333438626235636631336165386664343736
-34393132323339626331626532643266336435653636353930393738363134353661653636643263
+62663961346664656333306435323833366632346366356238653731653937626333653630623334
-30313138393263383837333438356463636162383938396137363035646433663932613066386138
+64326662346138386433333232643262333835326263343239353264373038613634356436396630
-37316535373939326131643930353435383232623366393630663463633233393232373730333631
+38323931643361663238623766323930666130356339363564366661663033303831363138343737
-37623837373533383462323431323335316231356436303239313535633837353931613730303363
+66633535326131396236653261303836613364306537633637323031663166316338323533323731
-31613765626466636636613366346166393766373834643337353764666532376335376139316138
+30326235323066396663613531653061643661336631613835626266626436386662353465383065
-38646133643736636363363434306334373933633264366530373534666637393835366266306633
+64396562343966303362636136616438353661626466636635323961613438646634336563636534
-66653966613136653936313732396366323830376236626530306139333030323966383035643733
+38343566396530643961356434643933636235643561353232643062303232323437666261363061
-36353833323539393136303035393866366563323430636336356639623734656135356566373062
+62636530396466303466653333633930376465366561376363316137323263333561343334383364
-34666433316139316664643234623530363732373338623162363636303463346463323131333065
+39313863643062643766396564363137386231373136346138396162376264653538303464633161
-64656566353837336266386233323439623237366364343635306137663038326434366531623062
+61663363623937356138356430666461623130323466623162653863393736326264393836336637
-64613365323339613465326464653965613439653766623762363139366264663765363862323839
+34663931646566333535666664653237643732316663323230383239393763376266356135326438
-63346539323961343532626465363266626234363964313531353662636635623165376130616536
+35656266353865623663373366373830613361373664346632363031356265313364623866643438
-64363361383766636333626366616531343833633033643965626435313130653938316430313532
+61363866353934636337616239633330623734666138396166313864333939663563636138653930
-64363731633139383236326231653434663132313463326563393739393132663765396337643035
+35653137363033326432373661613434313137623163356134613265393238346438306165313639
-32373963393830356362366130376135353935396631393533333430363531636234333832373939
+36666662393165353565633531663536613037623230373063316639663632643139353235303462
-36656233343663623262346337323034653036373432303266353333626434353137353939346665
+31393263656265656131613164363035343233626433656135353331613532363236616439363731
-37633432303162323538343339643635636464353333313063363662663033396630303162333639
+62666432356363323937666435326638323437346136366131613636653430306131623966356263
-31633537373364666234653938633461343738333464653736316331343038393333643936646462
+39303739306233303862636535633431363630393432613663633836396566653039383735303336
-36373232373962653031333730363166393463313232616631383233386533373464366465623335
+62623331623034636363636661653236386337326666656532343737336262336462613762326531
-65393462353638633931353331313233356230303565313030663739626230323766666466326339
+34303066303834386366636430343161653665343362363038396562626133636135656538306435
-35303361383065353333666161306433383139366439633730666364383130316637633464646336
+36393364333066346238396362663664643236373532336263656233386663323663623137343462
-66643065303934663765623233333261313939373236396136366566306438626439393163636430
+64386337333130316434663564613665666238623132343437656637653035373738313735366630
-30656334663361396230303239663338373664346362623934313130343064353435626439356562
+66383639616166393265616434623463326437313530326130376339313662303836636664366232
-35653638613037656133313765313165333066353732363361386234353635626563323932356237
+32366634633030333130316435616233396231663937343732313066373834326464623139363663
-31363333326536353536323931643739333764323637356164663566343563633761333462383735
+31323931626364303230666162316436653065366137663631376265383063316534343736373261
-66383134646139373130386366316532656139336231303931656265636130643130616531366530
+31613637316235386539343766323439653062633137663730343236343661346162653366656332
-62383132643236623835323362383866613765343762333536333365336338373264633130656438
+32663932313063383561636266373766633535656131386133386135663863396261306530326632
-36343331666431326530626137333665623836636136303031333965356366396239396534663263
+63653936626236316539613262386231616433393064323461626536363831666461316131383837
-35343730303534313830623430666566386564653036356635623038303133623231616635396463
+30646266646266393666396362326238613231303335336532303836363264323233343534636635
-65626137633033663631373366373238353766323533643462396562633164633730633135396266
+66313538643033343262373463363866346566353263303966323933383963363463393761383865
-36316362353964306463336237383232306530613539653462326137383938333638353038396132
+64663932343830643531643466303438343161396133666463353762393737613036646166333265
-66343265643162363735663235353161393732326530373735363662633165316265323065396163
+66376231613232666164663964636134653061633330383863373836306366393838393235656331
-31663761303637616566396262346238366631663562323566323364663464373930373831323630
+35613231306263373230326634623262326333356263353961633836396531633431383163633361
-39333165336264336539353634633936396561626363353466303931343436306636356663346438
+30356534666466653734333437383964346564346165326664633738653338313263633837316531
-64373561666638656439353964626436613164643663336336333337393332346132313830316430
+36613034323433643839333264323864613033313137663131623265643364333664646235666232
-37336139333539316639646337383737373530643533336133633135333936343165303764623366
+33393039313666323266643362323337316465306564303230646561303434666630616137633831
-35346432656263396439663635653066303861643938633033393334336336396133626237336432
+34346439616634343337306636643733316464376631616266376437636439396337306637333432
-31393061653839653964613239353336653666366163616163616230396661313233613934326339
+39306333363035393436316434656436353738303861633933376531383862316466373736323639
-36333733663032613633613037343337646237666565316239303231313163383264376636336338
+35336137373866336631386436646231653366366435363932376434303063613961353261343661
-37656532373634383132346163316337336534316465383531316465353861633062336235376631
+33383638393165336438376662306431333837356435626137356130323836396335636166306662
-33333963313730653164313866323461353334653231333762336232366434333631396135333466
+36386161353739353637353861306666383966323339303262616239633930373633323937356632
-36643362653137323934323966303734316334306133656139613763336431323236643264663530
+65383032613031666665623631613430666662656336663931646533636230303261646530623765
-38633730346531396435666530316563386566323737343766653135386537393165356264356534
+64643939326435643539373564336531623236653731636636346361363064333963376566616530
-62323962306332393063313062313230333334353833393362656166353638323366353934656263
+62326639663632666634326233363635383830643163373938646165656163643864336436373466
-38633531306438663134616262323332323338653361356336353962376331323537623734316532
+36353832306632386230373832333234643638313238626333303963383962343265366137656136
-35306434383336633436656437663765626530626535383464356532613536356533646332646135
+37333261343161633562346638323632616566646162633133663466346535656463393932386135
-39613030323432643161366363313262633261366566613931303663636262353466313933303162
+62653332313066363965386335356430326539316366633537356364666230326237306236393563
-66373535353934626135326439653630303437666535396266303532626636613563616633623237
+35323363633936353034323232353366373566666332323737653237323135646665626139393436
-38623938323137393065616136376663363234633735363264356438353534616338393832386337
+65333762653536656161386532363765336538653763666236343166653933626666633130393033
-66663339313062613766666435626364353935643932633233643630323464336135336339376437
+31643531646633623663313237353333313136663863663430306131316165663765653732663164
-30323536373639343738613463613466396335626339343135306666663931656564393664643761
+36326531626365326330643064336230313466343731376437316563303339336333326636633066
-65396336613465313035396134636438636134316236636566333031646666383939383264303437
+61386135626430616661313236623030316362373338643233326365646531633265626238383830
-38656337326233363866336363363161333236326137393465633935626165663136316661316131
+65346132643537366537626132666165616138656139626132396639376230333262643766386363
-66393764313130373766386235663134376239323037373365333730643336656532316434663265
+39383034663034326165643636613237623234613666333532383733623462303331326238636461
-35646664383338326366386536616339393938336539373762643839613763323434363037326330
+34383139383466356139333934353837613964326538336463643832623062633034613762363061
-66353738643735656165396138343262653563353934376237366564636638623630393238373530
+61346361656363366136353336326433326266336564316366393565626262303637316564356566
-39336335303564616232393165653639326166333536343966353933633365326539383632646639
+39376661383763373436306238393666653561306538333638306233356139346634653363346164
-37636238376137646239353731343933346562613630656130653865633834626363656231373162
+31303835383062376638626266303237323832623735653066353936376339633637333562333561
-65333662653962373139633466383834626566336234316161633038653137393266393564376439
+36646462653834316131323166366661386161646538346464386232306239363030366363633663
-62343365666637396639326438326631613664666662653935346230316632396361663038663239
+65306439616339626635326531636435356134376561303235393337373564373937623636643432
-63336335643665353162393330623635663839666162333561376265363330383530653733333239
+61393535643038626562366331353831663338333838383066323632383633346564396566653330
-37373665646562343562313764386534663239636130646564333433313661363338326439363865
+34386563393832313537623061666466366661333934613766366165366330353835323637643635
-63353064613161366461383862623363333637633430376361366566383332636634646630363139
+38613363396663356564646132613536653033616337386566623662333832383938303138316662
-65303139396539323336313666336362626339326666343061303431356361343438313365353431
+34303339323166383863363831636233323335313565393933636435396666313337663037323432
-30303266363664373233303231346166643839343564306363613236316665306437333636343131
+63333139333165646262666339343736383966346133356138326437386334626461636530336432
-61303234666664323138376335623639393939633836353664343232316630393263323233343366
+61356631366163366561303636333230643732316261376365386463333565623533663966336365
-32393736336166323166616139323066613062646139656236343233623630616262343034383631
+39333365393766306536366231366435363030353263393534653534373064636361333532323735
-61653662383537646364363764353335333630353431633534313731633265626263303863326565
+61313163323831653362356333386638343566356261353534303738613730373632363534666337
-37616534303164316162353165396236656565303532326235393237366434323739633662663261
+62313339613138646361356431616236613435393233343732626263653332663265393934616134
-33383635653734303233356438636132343731383439623463633732373833303339353662386435
+35313165613766643938393839373261633439396661623961353934373130623865353639633038
-63316138353766303962616435383138373739383835353662613666356565366237353932393263
+34353631346433663131653965326337663561613330323562666336656237633163356562323931
-34326561613234356537343234303166613534356134643766383332646263623430653632386365
+62393833663233333538383063303937386365306135343962623333663435663431396438666362
-61376165656366366532346465343932313032643163313362373633373066653036346238343938
+61386266353838653532393233353939363738306634666537313761313835333864633764666262
-66626133306163353966386662363039323166353330633239386264306337386361653430633838
+30333032623766633334383031636633636539336237613235376466356430663938653565626235
-66366636616661313939316562373330313564383033643035393738353939613066316138643238
+64373537656566306136633630366130363630633462656330623633393735386630343437336436
-35376336633238636463376234376364386464396461636336313562616361396435323639666433
+65343635366531646130616534623136636666323139326462306533653532643962656530336633
-34653139613962623235613035363634383735326564333332333632656237333238356566646536
+35616537303932343539336638333730663639396330653761346136363431346536666138336462
-62346435303932363231306634643037623935653235613330666666656433396463656231643364
+66396565613166623934316532383835316137303134363466306163356233356530323231666464
-34313235613064646432633935656231643162363530313761303339323830653564303963643532
+30353933306530323734306564626234343864373964333264353366326265316333343330356532
-39326163396461613161363031616336666564326630323465663831653431353566643661656435
+38363837646635633461653562303264353633343461633339376665616331613733666663353130
-35636238633663346139343732353233663930353664623636303134303633356232313063346566
+65396363613731366234326466323738663563646166653237613364323734616465643764633537
-30633762653239356131613539396435396235326431633062616436613266353266636133656531
+35373865353532383566363632366564353536643739663761303565333138383638653665663664
-63346533383764393733396436306136383830313265656539376136343536353737313365333865
+65643366316461613630366437623736353739356538336237613431306363663234373265623962
-61653665393037346231656663363432666332383837393264656231386435323333376633333265
+34653565373335653563356135313835643266356261623037336536613733323733363933376538
-39633563653161663531303330393237316132653339653531316538316534353334383161623833
+36626134396563623733656534363331626262643339633932373035626134343531623634666463
-63656534643833656366623031326132343732663536393636393537383338613462336638626334
+61623036313334616639633930393562663631653565656136666537393731333430663062643362
-39326264623032393661303335356235343736613964636261633038333236623232613633343263
+63633663396562343965313261373965356163393538666466303661363531393266316462626166
-34363865306233346433393664363038303464363834313463636363366333313433333930336634
+38616536653665366462383064373766396438616665346666376232653031323566313164383164
-65396232663031323965343336366339623239386566323362396564333932336134336231646430
+36643231646439663637333165376439333432383532316661333766363136636236326338386537
-34376236636536626264663532323530656634333137363431313339643164333236303337373639
+33306130353634346136356234363438383865313136393839663066333935623565333730613538
-31326131636161386665623838303939616337636666346363616465323263373061633832363839
+63323831663866363831383930303434333936646564316435303931396362303534386335343330
-39343131386333666231643931353330343136363732396465383336303866623733316235646234
+65383635346662626363626365666166636361633365643735303762393832316436646139303835
-65613531613333653231666234353532313738363266353336303564613061393639323933646339
+63633733656361643233323332613632653837663262306661626438316262653931333061336366
-64396334396431313563326136656539613535663334323330326566613264393965623065616634
+35353939353835333361623261613738383734656132613139393264393038373765343131333330
-32313333663535313531343363656230656334346430643365363838656430616366653766333136
+37663962313566366463623437323965326365623437363038633661313461383634626661666236
-32346137656136666432383934313865613535303962333464316465313464333233646162376165
+32633335323861643037383261393164393933353531636134323765353962633732396230636331
-37363265316139643332393761663164346537663064346133323661393030393533396633646134
+32613865373739366530303538313566346434633933393330346637346136373036306666336164
-39383539333364626263323364353764316539323162346434656234393562326133613632633137
+35373732346334353432616561623031663331346431383235306537386466623339356366663335
-34323537393838393534643861356365376461636235386638376661623439303832643231616431
+36333733626433653465336431666530626665373564336339626163633131353330656437643638
-32663264306166366336396130326435346432343732323764346232356439633361663937313233
+31616563616665343635356231633135663665326131636664373338323736393364353636613762
-62636130663238333433396335356331653166366130626632663162653736656238343134353134
+66636135633766323866376235633535613735613465303239343036663438333331626431623435
-39373663633437626661393734363264313137663138643665353633366663366562386337396561
+61323537386434323638666537643236623632626430666263376534643336613635663762643736
-38383464333531306138316266626461326531616364303732343337386464366464633834383439
+30323237353265613062373265643562373637383337326264653639306263373865333262376665
-66303334333234306166396235666164313833333164356561663063326563303934373933313334
+62646331373931373762303461366163393839633135393964313937616437323865653735383630
-32326466343761333666373936373332663939643662363665323534333638316637666665326564
+32653936336534666565373437666130396265363561333635316461663766346336623865376133
-63616134623530333739633265343637626561383831663836626538313539653536373165636133
+35373665303433326265623531613038636166643130616637653165376263643634376439613765
-62396432376332356133336461633664336264616566376664643861363838313766393833343366
+35363031373630333966656466616235616337306335363132386335613462363664653634633864
-31386636343835663062396232646663363935396366343361633462643734373437666331383035
+61303635653932353730663666386263633662633736313461643932386161313762663761313336
-30373963323834343138653063313635643062396130373861643134363062363135623730663335
+62653665313033656537643936373465633932626166366430643763313030393838393039323230
-34373935623239363866393932363966366233643732646461343964383563393064396331353331
+62633334383938343433306262393536653930653030393033306661666264313630643564333166
-31646662323364376639316562666465356438386637393130616330663563633839326661643239
+35383237633932656331653030363434313534613637373465663264643061303538653666653861
-62346633353963336239633738336431326638626232313965336361636430323739653734636436
+35373936643037333866636131373338363062663035323531626431633362663364396365353139
-39316363316137366233616431346639663335393737653562356532363634346237336266333132
+66383737666437353764333231303662393630643933376161366430376530613365363830373534
-31343836326538653164653163653738656238326366313532613433333337383263646439326661
+38376633333936626430393163323830346166643537326430616236393733653761363235356363
-65306135363138646665363463636436303864316432663734393130313932643833356161373139
+35333131663032383861336262653936376565646662313965303265623763613330653461333835
-65376334376136313237393336636630643933373165336432643233306366633663333930396364
+63613563323135633438383931343731656333303362316533376339376636623037376431336366
-31393739333730326434313639376365356432376338386430363133666436643837633533616333
+61393236383364356162633062666265653534326363363862666539623761623065386537616563
-36363464633233343130656163353137663165343064373039363439346631393436393430383361
+62666561316437303763376635346536666437373361386666643139643737663333323933613661
-61353563346531353561303935363531333235313731303237326334363763646131653961613334
+38326566663932333930616435626133616531306461356466326437623235613233393434626563
-33306534663238343433363237393234643236393931343230656438353866313638636633396464
+34373966633834373430386132353163366465626262353863353335323830393266393562393133
-61636561333562383964663261316135373235313234613564333333336438646636303561616634
+65383632653438646435343333386261653066613663623232373564666465613136353039313036
-63363033613238323463646165373361613834353230653138623132303731393436386139663533
+61646462626433646330396664363938376530376438646262343231393262383733636233636333
-38393263306139303231626363353931646163323364666161653861643739353262313561393165
+35633862336566636439653464613564333162613836343636316334316665383164353131373431
-6262
+63623030306564346562346237333934616134346536303365396533626262333937396432393830
+38313763393463646437666137353835373735646365373934363936346564326362376565353133
+36653362333432326133393837316331666663663263396461363239306239363733633137396633
+38393865613431653337313665313762653635656531353465623436343132303064303564393066
+63616639353962366666616261393766643364333634346630616436376565313236316539633537
+66653239636561393433383639646462616433653166613130373134376535633937353366383230
+61613335663434333835653236343633633038346335333861356637353965396632393833646635
+36653034356234663831333764303338663464316362646339376338393236336161616263363538
+63356631613239326161343031643936623366643432663732346438333265666535623664396333
+62303239363339626566613439396234303536333333653433393666383635643235376165666234
+30336236393962666335353233666463346530323531316438373130303933383465316638646461
+62643066376666363236333231386237376466633932313836323163363061313333633434663763
+64323365353336333736363436376232653436633739613437343538633632356665656364616637
+62313666346436656564663335636635393632353430666236313863613464626434323939383538
+65386265343434303632313739353239323565333734656566356164643430613538333234383566
+37646431316363316139646435333732313339623666613738663039613239613738393565333330
+33376432373933656435303737653762666464363865633831373330393435633332636261336139
+34336236373535636165353262323966363164633135613534353661316364616637663465363864
+39306163346365643339643937396165666366663339336438373031613937636464383531613962
+34373433663533626665656364623634373335313033646165303764396563356235343033383138
+62326361353630393938643764616636313461633734646661386536356235656665393864386465
+37666262346561656436343036646330363664306135333464663265306165353039396665336664
+62346631366330653762646538323565613864383534636532633033643533323736373931643130
+32343435333333613734626234363132373734353035326232366264336161383631353133663230
+32636533333866343763336439373336356237303636376334333433376630353338333261333037
+31666537363666653238383939663464346662636133346561326335346163363061393830616237
+36643430626534653331653665316535303139343763663965363164636238366533303038653935
+32356432316633373137316237663331336463363431393033323635646564366639346230353363
+38346663363039363962323137383366303862356530353238656563306131643236626536656334
+38303462306562366532346163323061393437353063326539393466616439346564383036303235
+35633731393661323962633631373061303930323638326565636162316436646337383266626561
+32326430363031396530396238353862333133363731623736376239626561626165663337373261
+39353461343461643238646635633562653865323336366634613264616662323232653861663038
+64396330626633303031333334343335393039623135353266383561313231643433393963326637
+39313530636361373831306234383166346266656261663830636631333564356536323565336266
+38306561376366626236306633613564386166616630613032633163613837313462343662653261
+63353437663436303634633336636532646439636465663362346138313665336334313039613631
+65326135383831613531323265353831313562346161663265366434623236636635333038366536
+33646631663662393331323162343438626666366636613438383665633136326439376166373462
+33363864613136643461663436396362643066633437376631623031613366656238396165313832
+31313131653263666334393664343239306235373862313339373563643137393633343663613936
+61356564636238363136623031336638333566633766636362303938653531306131396665303033
+63353362636463636236643464343562383161343432383766396330623764393837613435396162
+31303162356437663932663964663239623764666366663061313535346438373334636263653531
+34643133356638653031373036343162653135663734623035353033633561366266623566383233
+36356434643538643430383532393762333535636639353361353763333363313131646264336332
+34373331343930633962623963666365306132356334646636626461316236343839383266363635
+65623434336239313330343437646333353362303232346638623161616133636636626236643465
+36303636363965363765656533386534633839346363363738386532386531326538643134363132
+66613235373362633166343565323766306335336365333439323764623964393263623236623832
+34346132383136303038363764333039626234616132386464666633663536656230666133363533
+38306665643361666539636666316432623430623939663636343164386438313765633031313534
+65393633323837326166343936326263343833646331326464376138633461613532303135393036
+65353830373065393038343039323937303634346665393135383639303162396565646232663736
+39376434646634353330383933303164653431373433346335666131386165343035303964626665
+35383035653631346638326637326235393833623264323030373238646335346332353362393230
+61616664613562383639306564376661306665396138613066326631616531623132633966633832
+65363637376264336635633132633332373634383864626564623966356464373864393832323738
+37616338646633326636323461376137663632376262363738303336616463326238333465343533
+31316132386530326539

ansible/host_vars/astro-orbiter/vars.yml

@@ -2,11 +2,14 @@
 # ------------------------------------------------------------------------------
 # FILE: ansible/host_vars/astro_orbiter/vars.yml
 # HOST: astro-orbiter (10.1.71.130)
-# ROLE: Ollama inference host with AMD RX 5700 GPU passthrough
+# ROLE: llama.cpp LLM inference host — Ryzen 7 5800XT / RTX 3090 (ATX rebuild,
+#       2026-08-04). Superseded the prior AMD RX 5700 / Ollama config below;
+#       drive was transplanted into new hardware, not reinstalled.
 # ------------------------------------------------------------------------------
 
 ansible_host: 10.1.71.130
-ansible_user: wed
+ansible_user: jarvis
+ansible_ssh_private_key_file: ~/.ssh/id_jarvis
 ansible_become: true
 
 # LVM root expansion — xlarge template uses sda3 partition, standard VG/LV names
@@ -15,11 +18,79 @@ common_root_pv: /dev/sda3
 common_root_vg: ubuntu-vg
 common_root_lv: ubuntu-lv
 
-# Ollama — all defaults apply; explicitly documented here for visibility
+# --- Staged GGUF models for the llama.cpp router (:8002) ---------------------
-ollama_rocm_version: "6.2"
+# Data-driven list consumed by roles/llm-inference-multimodel tasks/models.yml
-ollama_default_model: "qwen3:8b"
+# (loop -> tasks/stage_model.yml). Each entry is idempotently staged into
-ollama_hsa_override_gfx_version: "10.1.0"
+# /opt/models: stat + EXACT-size check vs HF manifest; skip (no download, no
-ollama_data_disk: /dev/sdb
+# restart) when present + size matches. Source repos are public bartowski GGUFs
-ollama_data_vg: ollama-vg
+# on HuggingFace (no auth). A router restart is notified ONLY when a new GGUF
-ollama_data_lv: ollama-lv
+# is actually downloaded.
-ollama_data_dir: /var/lib/ollama
+# Added 2026-08-12 (War Machine): codify Phi-3.5-mini-instruct-Q8_0 and
+# Meta-Llama-3.1-8B-Instruct-Q4_K_M as router models alongside the production
+# Qwen3.6-35B-A3B-UD-Q4_K_S. The live files were already present/correct on
+# astro-orbiter; this pass codifies them. Future adds = append to this list.
+# Router --models-max override for astro-orbiter.
+# Default in defaults/main.yml is 1 (conservative). Bumped to 4 on 2026-08-12
+# (t_33acbb2e) so the router can keep more than one GGUF resident on-demand
+# and LRU-evict when needed.
+#
+# VRAM NOTE (t_33acbb2e, updated t_55c164f5, updated t_34b96e83, updated t_f5f7e9ad, updated t_441470b9):
+# With models-max=4 and all 5 GGUFs registered, worst case is all 5 loaded simultaneously:
+#   Qwen3.8-27B Q4_K_M:               ~23.3GB (weights ~17.1GB + KV ~6.2GB @ 128K ctx, q4_0) ← UPDATED
+#   Phi-3.5-mini-instruct Q8_0:       ~4.3GB  (weights ~3.8GB + KV ~0.5GB @ 32K ctx)
+#   Meta-Llama-3.1-8B Q4_K_M:        ~5.6GB  (weights ~4.6GB + KV ~0.2GB @ 8K ctx)
+#   Qwen2.5-Coder-14B Q4_K_M:        ~9.0GB  (weights ~8.4GB + KV ~0.6GB @ 16K ctx)
+#   nomic-embed-text-v1.5 Q4_K_M:    ~0.09GB (~84MB, embedding only — no KV cache)
+#   Total worst-case:                 ~42.3GB  >>  24GB RTX 3090
+#
+# OOM RISK: Full co-residency is impossible on 24GB. LRU eviction prevents this
+# in practice: models-max=4 means the router can REGISTER 5 models but only keeps
+# up to 4 LOADED simultaneously — the router will evict the LRU model when a new
+# one is needed. nomic-embed-text-v1.5 is pinned via sleep-idle-seconds=-1 and
+# load-on-startup=true but it uses only ~84MB, so it never meaningfully changes
+# the budget. In single-user homelab operation, only one generative model is active
+# at a time alongside the always-resident embedding model.
+# Qwen3.8-27B alone uses ~23.1GB (weights+KV); co-residency with Coder (~9GB) = ~32GB > 24GB.
+# LRU eviction handles this automatically — the router evicts the idle model before
+# loading the new one. Ryan should be aware this means model-switching always incurs
+# a ~30-60s cold-load latency when switching between Qwen3.8-27B and any other model.
+# Proceeding to models-max=4 as instructed; flagged for Ryan's attention.
+# Router --models-max override for astro-orbiter.
+# UPDATED (t_f5f7e9ad, 2026-08-16): Set to 2 because Qwen3.8-27B-Q4_K_M
+# uses 17,804 MiB at 65536 ctx. Only nomic-embed (558MB, pinned) and ONE
+# generative model can be resident simultaneously. Co-residency of Qwen3.8
+# with any auxiliary model (Phi 8.3GB, Llama 5.9GB, Coder 9GB) exceeds 24GB.
+# models-max=2: slot 1 = nomic-embed (pinned, always loaded), slot 2 = LRU
+# generative model (Qwen3.8 primary, cold-loaded on first request ~30-60s;
+# auxiliary models evict it on demand, and vice versa).
+# NOTE: Qwen3.8 does NOT have load-on-startup — it loads on first request.
+# This avoids an LRU eviction race with nomic-embed at startup.
+# UPDATED (t_72646029, 2026-08-17): CPU offload for Coder + Llama changes the
+# constraint. Coder and Llama now use CPU inference (n-gpu-layers=0). GPU-resident
+# VRAM: Qwen3.8 (~20,302 MiB at 128K ctx) + nomic-embed (558 MiB, pinned) plus the
+# CUDA-context buffers llama.cpp 6ea215d allocates for the CPU models (~1.4-1.7GB
+# each) = ~24,004 MiB steady-state, below the 24,576 MiB physical limit.
+# models-max raised to 4: nomic (slot 1, pinned) + Qwen3.8 (slot 2, GPU) +
+# Llama (slot 3, CPU) + Coder (slot 4, CPU). Phi (GPU, ~8.3GB) can still be
+# requested but evicts Qwen3.8 due to VRAM constraint. models-max=4
+# is required so CPU-offloaded models count as loaded without evicting Qwen3.8.
+llm_router_models_max: 4
+
+llm_staged_models:
+  - filename: "Phi-3.5-mini-instruct-Q8_0.gguf"
+    url: "https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF/resolve/main/Phi-3.5-mini-instruct-Q8_0.gguf"
+    size_bytes: 4061222688
+    source_repo: "bartowski/Phi-3.5-mini-instruct-GGUF"
+  - filename: "Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
+    url: "https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
+    size_bytes: 4920739232
+    source_repo: "bartowski/Meta-Llama-3.1-8B-Instruct-GGUF"
+  - filename: "Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
+    url: "https://huggingface.co/bartowski/Qwen2.5-Coder-14B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
+    size_bytes: 8988111072
+    source_repo: "bartowski/Qwen2.5-Coder-14B-Instruct-GGUF"
+  - filename: "nomic-embed-text-v1.5-Q4_K_M.gguf"
+    url: "https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF/resolve/main/nomic-embed-text-v1.5.Q4_K_M.gguf"
+    size_bytes: 84106624
+    source_repo: "nomic-ai/nomic-embed-text-v1.5-GGUF"
+

ansible/host_vars/carousel-of-progress/vars.yml

@@ -0,0 +1,16 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: ansible/host_vars/astro_orbiter/vars.yml
+# HOST: astro-orbiter (10.1.71.130)
+# ROLE: Ollama inference host with AMD RX 5700 GPU passthrough
+# ------------------------------------------------------------------------------
+
+ansible_host: 10.1.71.131
+ansible_user: wed
+ansible_become: true
+
+# LVM root expansion — xlarge template uses sda3 partition, standard VG/LV names
+common_expand_root_lvm: true
+common_root_pv: /dev/sda3
+common_root_vg: ubuntu-vg
+common_root_lv: ubuntu-lv

ansible/host_vars/main-street-station/main.yml

@@ -0,0 +1,16 @@
+---
+# Host-specific vars for main-street-station (JMRI headless server)
+# LCRR - Lake Country Railroad, Milwaukee Road Oct 1956, HO scale
+
+# JMRI profile ID — find with: ls ~/.jmri/profiles/ on the old box
+# Format: <name>.<8-char-hex>  e.g. LCRR.3d3f1dfc
+# TODO: fill in after restoring config from GitHub backup
+jmri_profile_id: ""
+
+# USB serial device for NCE command station
+# Verify after install: ls -la /dev/ttyUSB* /dev/ttyACM*
+jmri_serial_device: /dev/ttyUSB0
+
+# Path to JMRI config backup for restore task (leave empty to skip)
+# Point at a local checkout of the LCRR GitHub repo
+jmri_config_src: ""

ansible/host_vars/main-street-station/vars.yml

@@ -0,0 +1,10 @@
+---
+# main-street-station — JMRI / LCRR server
+jmri_profile_id: "Lake_Country_Railroad.3e8b1d4b"
+jmri_lcrr_repo: "ssh://git@gitea.mk-labs.cloud:2221/rblundon/LCRR.git"
+jmri_lcrr_branch: "clean-profile"
+jmri_leviton_email: "{{ leviton_email }}"
+jmri_leviton_password: "{{ leviton_password }}"
+jmri_ssh_authorized_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINnSM/9fO8rz/amqkyoGUzUKNNzzmtSXPwOCr1O9zKNO ansible"
+jmri_ssh_authorized_keys_extra:
+  - "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG6HaK4Y21UwPRbAZ986L7I9QnUdyq53114+9kO8X4bL rblundon@laptop"

ansible/inventory.yml

@@ -50,18 +50,38 @@ nextcloud_server:
 
 semaphore_server:
   hosts:
-    imagineering:
+    figment:
       ansible_host: 10.1.71.37
+      ansible_user: wed
       ansible_become: true
 
 n8n_server:
   hosts:
     tiki-room:
 
-ollama_server:
+astro_orbiter:
   hosts:
     astro-orbiter:
-      ansible_host: 10.1.71.130
+
+hermes_server:
+  hosts:
+    carousel-of-progress:
+      ansible_host: 10.1.71.131
+      ansible_user: wed
+      ansible_become: true
+      ansible_ssh_private_key_file: ~/.ssh/ansible
+
+honcho_server:
+  hosts:
+    lincoln:
+      ansible_host: 10.1.71.132
+      ansible_user: wed
+      ansible_become: true
+
+jmri_server:
+  hosts:
+    main-street-station:
+      ansible_host: 192.168.10.40
       ansible_user: wed
       ansible_become: true
 
@@ -73,6 +93,10 @@ papermc_server:
 dev_servers:
   hosts:
     scrim:
+    backstage:
+      ansible_host: 10.1.71.133
+      ansible_user: wed
+      ansible_become: true
 
 # dhcp_server:
 #   hosts:

ansible/playbooks/add_service_route.yml

@@ -6,7 +6,8 @@
 #
 #              1. Syncs boilerplates/traefik/dynamic/ to lightning-lane
 #              2. Scans the directory for service configs
-#              3. Creates CNAME records for each service -> lightning-lane
+#              3. Extracts all hostnames from Host() rules (supports multi-host)
+#              4. Creates CNAME records for each hostname -> lightning-lane
 #
 # PREREQUISITES:
 #   - Service dynamic config YAML committed to boilerplates/traefik/dynamic/
@@ -14,15 +15,6 @@
 #
 # USAGE:
 #   ansible-playbook -i inventory.yml playbooks/add_service_route.yml
-#
-# ADDING A NEW SERVICE:
-#   1. Create boilerplates/traefik/dynamic/<service>.yml
-#   2. Commit and push
-#   3. Run this playbook
-#
-# EXCLUDING FILES:
-#   Files that are not service routes (e.g., default.yml for middleware
-#   definitions) should be added to the exclude_configs list below.
 # ------------------------------------------------------------------------------
 
 - name: Sync Traefik routes and ensure DNS records
@@ -33,13 +25,12 @@
   vars:
     base_domain: "local.mk-labs.cloud"
     dns_server: "monorail"
-    traefik_host: "10.1.71.35"
+    traefik_host: "lightning-lane.local.mk-labs.cloud"
     traefik_user: "wed"
     traefik_dynamic_path: "/opt/docker/traefik/dynamic/"
     dynamic_config_dir: "{{ playbook_dir }}/../../boilerplates/traefik/dynamic"
 
     # Files in the dynamic directory that are NOT service routes
-    # (middleware definitions, TLS options, etc.)
     exclude_configs:
       - default.yml
 
@@ -53,38 +44,52 @@
       register: sync_result
       changed_when: "'sending incremental file list' in sync_result.stdout"
 
-    # ── Step 2: Discover service configs ──
+    # ── Step 2: Discover hostnames from Traefik router rules ──
     - name: Find all dynamic config files
       ansible.builtin.find:
         paths: "{{ dynamic_config_dir }}"
         patterns: "*.yml"
       register: config_files
 
-    - name: Build service list from config filenames
+    - name: Read config files
-      ansible.builtin.set_fact:
+      ansible.builtin.slurp:
-        service_names: >-
+        src: "{{ item.path }}"
-          {{ config_files.files
+      register: slurped_configs
-             | map(attribute='path')
+      loop: "{{ config_files.files }}"
-             | map('basename')
+      when: item.path | basename not in exclude_configs
-             | reject('in', exclude_configs)
-             | map('regex_replace', '\.yml$', '')
-             | list }}
 
-    - name: Display services to route
+    - name: Extract all hostnames from Host() rules
+      ansible.builtin.set_fact:
+        hostnames: >-
+          {% set hosts = [] -%}
+          {% for result in slurped_configs.results if result.content is defined -%}
+            {% set content = result.content | b64decode -%}
+            {% for match in content | regex_findall('Host\(`([^`]+)`\)') -%}
+              {% for h in match.split(' || ') -%}
+                {% set h = h | regex_replace('`', '') | trim -%}
+                {% if h.endswith('.local.mk-labs.cloud') and h not in hosts -%}
+                  {% set _ = hosts.append(h) -%}
+                {% endif -%}
+              {% endfor -%}
+            {% endfor -%}
+          {% endfor -%}
+          {{ hosts | unique | list }}
+
+    - name: Display hostnames to create
       ansible.builtin.debug:
-        msg: "Services found: {{ service_names }}"
+        msg: "Hostnames found: {{ hostnames }}"
 
     # ── Step 3: Create DNS CNAME records ──
-    - name: Create DNS CNAME record for each service
+    - name: Create DNS CNAME record for each hostname
       effectivelywild.technitium_dns.technitium_dns_add_record:
         api_url: "http://{{ dns_server }}.{{ base_domain }}"
         api_token: "{{ vault_technitium_api_key }}"
         zone: "{{ base_domain }}"
-        name: "{{ item }}.{{ base_domain }}"
+        name: "{{ item }}"
         type: "CNAME"
         cname: "lightning-lane.{{ base_domain }}"
         ttl: 360
         validate_certs: false
-      loop: "{{ service_names }}"
+      loop: "{{ hostnames }}"
       loop_control:
-        label: "{{ item }}.{{ base_domain }}"
+        label: "{{ item }}"

ansible/playbooks/day0_baseline.yml

@@ -1,6 +1,6 @@
 ---
-- name: Apply common role
+- name: Apply day0 baseline
   hosts: "{{ target | default('all') }}"
   become: true
   roles:
-    - common
+    - day0-baseline

ansible/playbooks/day0_expand_root_lv.yml

@@ -0,0 +1,32 @@
+---
+# ============================================================================
+# day0_expand_root_lv.yml
+# ----------------------------------------------------------------------------
+# Reclaims unallocated PE on the root volume group, extending the root LV
+# to fill the VG and resizing the underlying filesystem (ext4 or xfs).
+#
+# Belongs to the day0 host-provisioning lifecycle. The Ubuntu Server
+# autoinstall template ships with the root LV at ~half the disk size by
+# default; this playbook is the canonical one-shot fix-up for that.
+#
+# Idempotent and safe to re-run. Hosts without LVM are no-op'd cleanly.
+#
+# Opt-out: set `expand_root_lv_skip: true` in host_vars/<host>.yml for
+# hosts where free PE should NOT be claimed by root (e.g. hosts with a
+# planned second LV in the same VG for application data).
+#
+# Usage:
+#   ansible-playbook playbooks/day0_expand_root_lv.yml
+#   ansible-playbook playbooks/day0_expand_root_lv.yml -e target=lincoln
+#   ansible-playbook playbooks/day0_expand_root_lv.yml -e target=honcho_server
+# ============================================================================
+
+- name: Expand root logical volume to fill VG
+  hosts: "{{ target | default('all') }}"
+  become: true
+  gather_facts: true
+  tasks:
+    - name: Apply expand_root_lv role unless host opts out
+      ansible.builtin.include_role:
+        name: expand_root_lv
+      when: not (expand_root_lv_skip | default(false) | bool)

ansible/playbooks/day0_linux_baseline.yml

@@ -0,0 +1,23 @@
+---
+# ============================================================================
+# day0_linux_baseline.yml
+# ----------------------------------------------------------------------------
+# Applies the mk-labs Linux baseline (linux-baseline role) to one or more
+# hosts. Idempotent and safe to re-run.
+#
+# Usage:
+#   ansible-playbook playbooks/day0_linux_baseline.yml
+#   ansible-playbook playbooks/day0_linux_baseline.yml -e target=figment
+#   ansible-playbook playbooks/day0_linux_baseline.yml -e target=semaphore_server
+#
+# To trigger an opt-in full system upgrade:
+#   ansible-playbook playbooks/day0_linux_baseline.yml \
+#     -e target=figment -e 'baseline_features={"full_upgrade": true}'
+# ============================================================================
+
+- name: Apply mk-labs Linux baseline
+  hosts: "{{ target | default('all') }}"
+  become: true
+  gather_facts: true
+  roles:
+    - linux-baseline

ansible/playbooks/day0_provision.yml

@@ -0,0 +1,30 @@
+---
+# ============================================================================
+# day0_provision.yml
+# ----------------------------------------------------------------------------
+# Umbrella day0 playbook. Runs the full host-provisioning lifecycle in
+# the correct order against newly-built VMs, so the operator runs ONE
+# command per new host rather than chaining day0 steps manually.
+#
+# Order matters:
+#   1. linux-baseline      — timezone, NTP, packages, SSH hardening, jarvis user
+#   2. expand_root_lv      — reclaim PE left unallocated by the Ubuntu
+#                            autoinstall template default
+#
+# Idempotent: every step is safe to re-run. Suitable to apply periodically
+# from Semaphore as a baseline-drift check.
+#
+# Usage:
+#   ansible-playbook playbooks/day0_provision.yml -e target=lincoln
+#   ansible-playbook playbooks/day0_provision.yml -e target=honcho_server
+#
+# For finer control over a single phase, the constituent playbooks are:
+#   playbooks/day0_linux_baseline.yml
+#   playbooks/day0_expand_root_lv.yml
+# ============================================================================
+
+- name: Import day0 linux baseline
+  ansible.builtin.import_playbook: day0_linux_baseline.yml
+
+- name: Import day0 expand root LV
+  ansible.builtin.import_playbook: day0_expand_root_lv.yml

ansible/playbooks/day1_deploy_hermes.yml

@@ -0,0 +1,79 @@
+---
+# =============================================================================
+# day1_deploy_hermes.yml
+# Deploy Hermes Agent (Nous Research) on carousel-of-progress (10.1.71.131)
+#
+# FIRST-RUN WORKFLOW:
+#   1. Run this playbook:
+#        ansible-playbook playbooks/day1_deploy_hermes.yml
+#
+#   2. SSH to the host and run the setup wizard as the hermes user:
+#        ssh wed@carousel-of-progress.local.mk-labs.cloud
+#        sudo -u hermes hermes setup
+#
+#   3. Once configured, start and verify the service:
+#        sudo systemctl start hermes
+#        sudo systemctl status hermes
+#        sudo journalctl -u hermes -f
+#
+# VARIABLES:
+#   hermes_skip_browser: true   — set to skip Playwright/Chromium install
+#                                 (saves ~300MB if browser automation not needed)
+# =============================================================================
+
+- name: Deploy Hermes Agent on carousel-of-progress
+  hosts: carousel-of-progress
+  gather_facts: true
+
+  pre_tasks:
+    - name: Verify target is carousel-of-progress
+      ansible.builtin.assert:
+        that:
+          - inventory_hostname == "carousel-of-progress"
+        fail_msg: >
+          This playbook is scoped to carousel-of-progress only.
+          Got: {{ inventory_hostname }}
+
+    - name: Confirm OS is Ubuntu
+      ansible.builtin.assert:
+        that:
+          - ansible_distribution == "Ubuntu"
+        fail_msg: >
+          This playbook requires Ubuntu. Found: {{ ansible_distribution }}.
+          (If running Fedora, swap apt tasks for dnf and adjust Playwright deps.)
+
+  roles:
+    - role: hermes
+      vars:
+        hermes_skip_browser: false   # set true to skip Chromium install
+
+  post_tasks:
+    - name: Verify hermes binary is accessible system-wide
+      ansible.builtin.command: hermes --version
+      register: hermes_version_check
+      changed_when: false
+      failed_when: hermes_version_check.rc != 0
+
+    - name: Print hermes version
+      ansible.builtin.debug:
+        msg: "{{ hermes_version_check.stdout }}"
+
+    - name: Print post-install instructions
+      ansible.builtin.debug:
+        msg:
+          - "============================================================"
+          - "Hermes installed on carousel-of-progress (10.1.71.131)"
+          - "============================================================"
+          - "Next steps:"
+          - "  1. SSH to the host:"
+          - "       ssh wed@carousel-of-progress.local.mk-labs.cloud"
+          - "  2. Run the setup wizard as the hermes user:"
+          - "       sudo -u hermes hermes setup"
+          - "  3. After config, start the service:"
+          - "       sudo systemctl start hermes"
+          - "  4. Verify:"
+          - "       sudo systemctl status hermes"
+          - "       sudo journalctl -u hermes -f"
+          - "============================================================"
+          - "Service is ENABLED but NOT STARTED — config required first."
+          - "============================================================"

ansible/playbooks/day1_deploy_honcho.yml

@@ -0,0 +1,18 @@
+---
+# ============================================================================
+# day1_deploy_honcho.yml
+# ----------------------------------------------------------------------------
+# Deploys Honcho + pgvector PostgreSQL on the `lincoln` host. Assumes day0
+# host provisioning (linux-baseline + expand_root_lv) is already complete.
+#
+# Run via:
+#   ansible-playbook -i inventory.yml playbooks/day0_provision.yml -e target=lincoln
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_honcho.yml
+# ============================================================================
+
+- name: Deploy Honcho on lincoln
+  hosts: honcho_server
+  become: true
+  gather_facts: true
+  roles:
+    - honcho

ansible/playbooks/day1_deploy_jmri.yml

@@ -0,0 +1,24 @@
+---
+# ============================================================================
+# day1_deploy_jmri.yml
+# ----------------------------------------------------------------------------
+# Deploys JMRI JmriFaceless headless server on main-street-station.
+# Applies linux-baseline first, then the jmri role.
+#
+# Usage:
+#   ansible-playbook playbooks/day1_deploy_jmri.yml
+#   ansible-playbook playbooks/day1_deploy_jmri.yml -e target=main-street-station
+#
+# Prerequisites:
+#   1. Host is in inventory under jmri_server group
+#   2. jmri_profile_id is set in host_vars/main-street-station.yml
+#   3. SSH access as 'wed' with sudo
+# ============================================================================
+
+- name: Deploy JMRI headless server
+  hosts: "{{ target | default('jmri_server') }}"
+  become: true
+  gather_facts: true
+  roles:
+    - linux-baseline
+    - jmri

ansible/playbooks/day1_deploy_llm_inference.yml

@@ -0,0 +1,25 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day1_deploy_llm_inference.yml
+# DESCRIPTION: Day 1 playbook for astro-orbiter LLM inference stack.
+#              Deploys vLLM + Gemma 2 27B on RTX 3090 via OCuLink.
+#
+# Usage:
+#   cd ~/git/homelab/ansible
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference.yml
+#
+# Phases (added incrementally — safe to re-run):
+#   1. Foundation  — groups, directories, vault assertion
+#   2. Driver      — nvidia-driver-595-open (idempotent; already installed)
+#   3. vLLM        — Python venv + pip install vllm
+#   4. Model       — HF login, Gemma 2 27B snapshot_download
+#   5. Serve       — systemd vllm-serve.service, health check
+#   6. Integration — Hermes provider config on carousel
+# ------------------------------------------------------------------------------
+
+- name: Deploy LLM inference stack on astro-orbiter
+  hosts: astro_orbiter
+  gather_facts: true
+
+  roles:
+    - role: llm-inference

ansible/playbooks/day1_deploy_llm_inference_multimodel.yml

@@ -0,0 +1,35 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day1_deploy_llm_inference_multimodel.yml
+# DESCRIPTION: Day 1 playbook for the dual-model (aux + tool-calling) rollout
+#              on astro-orbiter. Builds on roles/llm-inference (CUDA/driver
+#              already done) — does not replace it.
+#
+# Usage:
+#   cd ~/git/homelab/ansible
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml
+#   # or scope to specific phases:
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml --tags discover
+#
+# EXECUTION CHANNEL (2026-08-12, War Machine): run via the Semaphore template
+# "llm_inference_multimodel_stage_models" (scoped to --tags models). Do NOT
+# run this via direct ansible-playbook or ad-hoc ssh/curl/systemctl — all
+# homelab inference changes go through Ansible roles executed by Semaphore for
+# audit/visibility. Phase 1 (models) is idempotent: it only downloads/stages a
+# GGUF when missing or size-mismatched, and only restarts the router when a new
+# GGUF is detected (normal re-runs that find the files correct touch nothing).
+#
+# Phases (see roles/llm-inference-multimodel/README.md for detail):
+#   0. discover  — read-only; confirm existing Gemma service management
+#   1. models    — idempotent GGUF downloads (Phi-4-14B, Mistral-Small-24B)
+#   2. systemd   — deploy both unit files, do NOT auto-start
+#   3. firewall  — scope ports 8000/8001, non-0.0.0.0 bind
+#   4. verify    — start both services, smoke test, VRAM check
+# ------------------------------------------------------------------------------
+
+- name: Deploy dual-model LLM inference stack on astro-orbiter
+  hosts: astro_orbiter
+  gather_facts: true
+
+  roles:
+    - role: llm-inference-multimodel

ansible/playbooks/day1_deploy_llm_router_shadow.yml

@@ -0,0 +1,106 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day1_deploy_llm_router_shadow.yml
+# DESCRIPTION: Deploy llama-server in router mode on a shadow port (8003).
+#
+# This playbook deploys and validates the llama.cpp router mode supervisor on
+# astro-orbiter (10.1.71.130) WITHOUT touching the production endpoint
+# (llama-server-qwen, port 8002). All 7 dependent Hermes profiles
+# (bruce-banner, groot, happy, heimdall, rocket-raccoon, war-machine, wong)
+# remain pointing at port 8002 throughout this run.
+#
+# Usage (from ~/git/homelab/ansible):
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_router_shadow.yml
+#
+# Tag-scoped runs (if you need to re-run one phase):
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_router_shadow.yml \
+#     --tags router_systemd,router_firewall,router_verify
+#
+# Execution path (Ryan-approved 2026-08-12, task t_0cca74a2):
+#   Direct ansible-playbook as documented exception — Semaphore template for
+#   this role does not exist yet. Create template after cutover is confirmed.
+#   This is the same exception pattern used in prior sessions on this box.
+#
+# Pre-requisites:
+#   1. llama-server binary at /opt/llama.cpp/build/bin/llama-server supports
+#      router mode (confirmed 2026-08-12: --models-dir flag present in --help).
+#   2. /opt/models/ contains ONLY Qwen3.6-35B-A3B-UD-Q4_K_S.gguf
+#      (confirmed 2026-08-12: directory is clean, Phi-4/Mistral already deleted).
+#   3. Port 8002 is in use by the production llama-server-qwen service —
+#      this playbook does NOT touch it.
+#
+# Validation gates this playbook runs (all hard gates EXCEPT Gate 4):
+#   Gate 1: /v1/models reports Qwen with n_ctx >= 64000 (64K Hermes floor)
+#   Gate 2: Tool-calling probe through router returns finish_reason=tool_calls
+#   Gate 2b: Hallucination stress test does NOT trigger spurious tool_calls
+#   Gate 3: nvidia-smi VRAM <= 23,000 MiB (--models-max 1 confirmed effective)
+#   Gate 4: Bundled SvelteKit UI check (nice-to-have, non-blocking)
+#
+# What happens after this playbook:
+#   War Machine posts validation gate results to Ryan.
+#   Ryan reviews and signs off on cutover (or requests changes).
+#   War Machine then runs day2_cutover_qwen_to_router.yml (not yet created)
+#   to promote the router to port 8002 and retire the bare llama-server-qwen.
+#
+# Reference: proposal at
+#   ~/friday/system/inbox/agents/war-machine/2026-08-12-qwen-router-mode-proposal.md
+# Task: t_0cca74a2
+# Author: War Machine (2026-08-12)
+# ------------------------------------------------------------------------------
+
+- name: Deploy llama-server router (shadow, port 8003) on astro-orbiter
+  hosts: astro_orbiter
+  gather_facts: true
+  become: true
+
+  vars:
+    # Enable the router phase — this is the ONLY var that makes router.yml run.
+    # Default in defaults/main.yml is false (no-op). Flip here for the shadow run.
+    llm_router_enabled: true
+
+    # Qwen model ID as it appears in /v1/models from the router.
+    # llama-server router uses the GGUF filename (without .gguf) as the model id.
+    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
+
+  roles:
+    - role: llm-inference-multimodel
+
+  # No --tags needed here: router.yml is included dynamically from main.yml
+  # whenever llm_router_enabled: true. The full role runs but the
+  # discover/models/systemd/verify phases are gated on their own vars
+  # (llm_qwen_service_enabled etc.) and are idempotent. The stale
+  # models.yml (Phi-4/Mistral download tasks) uses variables no longer
+  # defined — a follow-up cleanup task should update that file.
+
+- name: "POST-VALIDATION SAFETY NET — ensure production service is running"
+  hosts: astro_orbiter
+  gather_facts: false
+  become: true
+
+  tasks:
+    # Always run this, regardless of whether the validation play succeeded.
+    # If the router.yml play stopped llama-server-qwen for VRAM validation
+    # and then a gate failed (play aborted), this play ensures it comes back up.
+    - name: "Ensure llama-server-qwen (port 8002) is running after validation (always)"
+      ansible.builtin.systemd:
+        name: llama-server-qwen
+        state: started
+        enabled: true
+      ignore_errors: true   # don't fail if the unit doesn't exist
+
+    - name: "Verify production /health after safety-net restart"
+      ansible.builtin.uri:
+        url: "http://10.1.71.130:8002/health"
+        status_code: 200
+        timeout: 30
+      register: llm_safety_net_health
+      failed_when: false
+      ignore_errors: true
+
+    - name: "Report production status (safety-net check)"
+      ansible.builtin.debug:
+        msg: >-
+          Safety-net: llama-server-qwen :8002 health check returned
+          {{ llm_safety_net_health.status | default('UNREACHABLE') }}.
+          {{ 'OK — production is up.' if (llm_safety_net_health.status | default(0) | int == 200)
+             else 'WARNING — production may not be healthy. Check manually.' }}

ansible/playbooks/day1_deploy_semaphore.yml

@@ -1,22 +1,17 @@
 ---
-# ------------------------------------------------------------------------------
+# ============================================================================
-# FILE: playbooks/day1_deploy_semaphore.yml
+# day1_deploy_semaphore.yml
-# DESCRIPTION: Deploys Semaphore on imagineering
+# ----------------------------------------------------------------------------
-#              Runs: common → docker-host → semaphore
+# Deploys SemaphoreUI + PostgreSQL on the imagineering host (figment).
+# Run AFTER day0_linux_baseline.yml has been applied to the target.
 #
-# USAGE:
+# Usage:
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_semaphore.yml
-#
+# ============================================================================
-# SECRETS REQUIRED IN VAULT (group_vars/all/vault):
-#   vault_semaphore_database_password
-#   vault_semaphore_admin_password
-#   vault_semaphore_access_key_encryption
-# ------------------------------------------------------------------------------
 
-- name: Deploy Semaphore
+- name: Deploy SemaphoreUI on imagineering
   hosts: semaphore_server
   become: true
-
+  gather_facts: true
   roles:
-    - docker-host
     - semaphore

ansible/playbooks/day2_add_coder_alias.yml

@@ -0,0 +1,259 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day2_add_coder_alias.yml
+# DESCRIPTION: Add Qwen2.5-Coder-14B-Instruct-Q4_K_M to the llama-server-router
+#              on astro-orbiter (10.1.71.130:8002).
+#
+# Context (t_55c164f5, 2026-08-13):
+#   Ryan requested a Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf be added to the
+#   astro-orbiter router with:
+#     alias = "Qwen2.5-Coder-14B-Instruct-4bit"
+#     n_gpu_layers = 99
+#     ctx_size = 16384
+#     flash_attn = true
+#   Deployed GitOps-style via this role; no hand-editing of the live preset.
+#
+# What this playbook does:
+#   1. Downloads Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf into /opt/models if
+#      not already present (idempotent: size-check guard, no re-pull on match).
+#   2. Redeploys the preset INI (adding the [Qwen2.5-Coder-14B-Instruct-Q4_K_M]
+#      section with alias = Qwen2.5-Coder-14B-Instruct-4bit).
+#   3. Restarts llama-server-router to pick up the new model entry.
+#   4. Verifies /v1/models returns all 4 models including the new Coder entry.
+#
+# VRAM context note (t_55c164f5):
+#   Qwen2.5-Coder-14B Q4_K_M: ~8.4GB weights + ~0.6GB KV @ 16K ctx ≈ 9.0GB
+#   Qwen3.6-35B-A3B: ~21.5GB
+#   Full co-residency is impossible on 24GB. LRU eviction handles this:
+#   when Coder is requested, Qwen3.6-35B is evicted (and vice versa).
+#   Model-switching incurs ~30-60s cold-load latency — expected and acceptable.
+#   Phi (~4.3GB) or Llama (~5.6GB) can co-reside with Coder (total ~14GB).
+#
+# Usage (from ~/git/homelab/ansible):
+#   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook -i inventory.yml \
+#     playbooks/day2_add_coder_alias.yml
+#
+# Semaphore note: Semaphore SSH key for jarvis user is not loaded in the
+# container (known pitfall, homelab-llm-serving skill). Run via CLI with
+# id_jarvis key; document as exception per Ryan's standing CLI fallback directive.
+#
+# Author: War Machine (2026-08-13, t_55c164f5)
+# ------------------------------------------------------------------------------
+
+- name: "Add Qwen2.5-Coder-14B-Instruct-4bit alias to astro-orbiter router"
+  hosts: astro_orbiter
+  gather_facts: false
+  become: true
+
+  vars:
+    # Activate preset mode
+    llm_router_preset_enabled: true
+    llm_router_preset_path: /opt/llama-server-router-preset.ini
+
+    # Production port (router is on 8002 since t_cd0d5388)
+    llm_router_port: 8002
+
+    # Per-model ctx-size settings (carried from t_ryan_per_model_ctx; Coder new)
+    llm_router_llama_ctx_size: 8192
+    llm_router_llama_flash_attn: "true"
+    llm_router_phi_ctx_size: 32768
+    llm_router_phi_flash_attn: "true"
+    llm_router_coder_ctx_size: 16384
+    llm_router_coder_flash_attn: "true"
+
+    # All other vars inherit from host_vars + defaults/main.yml.
+    # Explicitly set the ones needed by the unit/template tasks for clarity:
+    llm_router_enabled: true
+    llm_service_user: jarvis
+    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
+    llm_models_dir: /opt/models
+    llm_bind_address: "10.1.71.130"
+    llm_allowed_source_cidr: "10.1.70.0/24"
+    llm_router_service_name: llama-server-router
+    llm_router_bind_address: "10.1.71.130"
+    llm_router_allowed_source_cidr: "10.1.70.0/24"
+    llm_router_models_dir: /opt/models
+    llm_router_models_max: 4            # from host_vars; bumped by t_33acbb2e
+    llm_router_ctx_size: 65536          # Qwen3.6-35B default; per-model overrides above
+    llm_router_parallel: 1
+    llm_router_gpu_layers: 99
+    llm_router_batch_size: 2048
+    llm_router_ubatch_size: 512
+    llm_router_cache_type_k: q4_0
+    llm_router_cache_type_v: q4_0
+    llm_router_flash_attn: "auto"
+    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
+    llm_router_vram_max_mib: 23000
+
+    # Coder model staging entry (used below)
+    coder_filename: "Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
+    coder_url: "https://huggingface.co/bartowski/Qwen2.5-Coder-14B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
+    coder_size_bytes: 8988111072
+
+  handlers:
+    - name: reload systemd
+      ansible.builtin.systemd:
+        daemon_reload: true
+      become: true
+      listen: "reload systemd"
+
+    - name: restart router
+      ansible.builtin.systemd:
+        name: llama-server-router
+        state: restarted
+      become: true
+      listen: "restart router"
+
+  tasks:
+
+    # ==========================================================================
+    # PHASE 1: Download Coder GGUF if not present / size mismatch
+    # ==========================================================================
+
+    - name: "[coder] Stat existing GGUF"
+      ansible.builtin.stat:
+        path: "{{ llm_models_dir }}/{{ coder_filename }}"
+        get_checksum: false
+      register: coder_stat
+
+    - name: "[coder] Download GGUF (skip if present and size matches)"
+      ansible.builtin.get_url:
+        url: "{{ coder_url }}"
+        dest: "{{ llm_models_dir }}/{{ coder_filename }}"
+        owner: "{{ llm_service_user }}"
+        group: "{{ llm_service_user }}"
+        mode: "0644"
+        timeout: 3600
+      when: >
+        not coder_stat.stat.exists or
+        coder_stat.stat.size != coder_size_bytes
+      register: coder_download
+      notify: restart router
+
+    - name: "[coder] Confirm GGUF size post-download"
+      ansible.builtin.stat:
+        path: "{{ llm_models_dir }}/{{ coder_filename }}"
+        get_checksum: false
+      register: coder_stat_post
+
+    - name: "[coder] FAIL if GGUF size mismatch after download"
+      ansible.builtin.fail:
+        msg: >-
+          GGUF size mismatch: expected {{ coder_size_bytes }} bytes,
+          got {{ coder_stat_post.stat.size }} bytes.
+          Re-download may be needed.
+      when: coder_stat_post.stat.size != coder_size_bytes
+
+    # ==========================================================================
+    # PHASE 2: Deploy updated preset INI (adds Coder section)
+    # ==========================================================================
+
+    - name: "[coder] Deploy preset INI to {{ llm_router_preset_path }}"
+      ansible.builtin.template:
+        src: "../roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2"
+        dest: "{{ llm_router_preset_path }}"
+        owner: root
+        group: root
+        mode: "0644"
+      register: coder_preset_deployed
+      notify: restart router
+
+    # ==========================================================================
+    # PHASE 3: Redeploy systemd unit (unchanged flags, but ensures unit is fresh)
+    # ==========================================================================
+
+    - name: "[coder] Deploy llama-server-router unit"
+      ansible.builtin.template:
+        src: "../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
+        dest: /etc/systemd/system/llama-server-router.service
+        owner: root
+        group: root
+        mode: "0644"
+      register: coder_unit_deployed
+      notify:
+        - reload systemd
+        - restart router
+
+    - name: "[coder] Flush handlers (daemon-reload + router restart)"
+      ansible.builtin.meta: flush_handlers
+
+    # ==========================================================================
+    # PHASE 4: Verify router is up and Coder model appears in /v1/models
+    # ==========================================================================
+
+    - name: "[coder] Wait for /health (router supervisor)"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
+        status_code: 200
+        timeout: 30
+      retries: 12
+      delay: 5
+      register: coder_health
+      until: coder_health.status == 200
+
+    - name: "[coder] Query /v1/models"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
+        status_code: 200
+        return_content: true
+        timeout: 30
+      register: coder_models
+
+    - name: "[coder] Extract model IDs and aliases"
+      ansible.builtin.set_fact:
+        coder_model_ids: "{{ coder_models.json.data | map(attribute='id') | list }}"
+        coder_all_aliases: "{{ coder_models.json.data | map(attribute='aliases') | flatten | list }}"
+
+    - name: "[coder] FAIL if Coder primary ID missing"
+      ansible.builtin.fail:
+        msg: >-
+          'Qwen2.5-Coder-14B-Instruct-Q4_K_M' not in /v1/models.
+          IDs: {{ coder_model_ids }}
+      when: "'Qwen2.5-Coder-14B-Instruct-Q4_K_M' not in coder_model_ids"
+
+    - name: "[coder] FAIL if Coder alias missing"
+      ansible.builtin.fail:
+        msg: >-
+          'Qwen2.5-Coder-14B-Instruct-4bit' not found as ID or alias in /v1/models.
+          IDs:     {{ coder_model_ids }}
+          Aliases: {{ coder_all_aliases }}
+      when:
+        - "'Qwen2.5-Coder-14B-Instruct-4bit' not in coder_model_ids"
+        - "'Qwen2.5-Coder-14B-Instruct-4bit' not in coder_all_aliases"
+
+    - name: "[coder] FAIL if Qwen3.6-35B missing"
+      ansible.builtin.fail:
+        msg: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in /v1/models. IDs: {{ coder_model_ids }}"
+      when: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in coder_model_ids"
+
+    - name: "[coder] FAIL if Phi missing"
+      ansible.builtin.fail:
+        msg: "'Phi-3.5-mini-instruct-Q8_0' not in /v1/models. IDs: {{ coder_model_ids }}"
+      when: "'Phi-3.5-mini-instruct-Q8_0' not in coder_model_ids"
+
+    - name: "[coder] FAIL if Llama missing"
+      ansible.builtin.fail:
+        msg: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in /v1/models. IDs: {{ coder_model_ids }}"
+      when: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in coder_model_ids"
+
+    - name: "[coder] PASS — full /v1/models summary"
+      ansible.builtin.debug:
+        msg:
+          - "========================================================================"
+          - "QWEN2.5-CODER-14B ALIAS DEPLOYMENT — COMPLETE"
+          - ""
+          - "  Mode:     --models-preset ({{ llm_router_preset_path }})"
+          - "  Service:  llama-server-router.service (:{{ llm_router_port }})"
+          - ""
+          - "  /v1/models IDs:     {{ coder_model_ids }}"
+          - "  /v1/models aliases: {{ coder_all_aliases }}"
+          - ""
+          - "  VERIFY:"
+          - "  Qwen3.6-35B-A3B-UD-Q4_K_S:          {{ 'PRESENT' if 'Qwen3.6-35B-A3B-UD-Q4_K_S' in coder_model_ids else 'MISSING' }}"
+          - "  Phi-3.5-mini-instruct-Q8_0:          {{ 'PRESENT' if 'Phi-3.5-mini-instruct-Q8_0' in coder_model_ids else 'MISSING' }}"
+          - "  Meta-Llama-3.1-8B-Instruct-Q4_K_M:  {{ 'PRESENT' if 'Meta-Llama-3.1-8B-Instruct-Q4_K_M' in coder_model_ids else 'MISSING' }}"
+          - "  Qwen2.5-Coder-14B-Instruct-Q4_K_M:  {{ 'PRESENT' if 'Qwen2.5-Coder-14B-Instruct-Q4_K_M' in coder_model_ids else 'MISSING' }}"
+          - "  Qwen2.5-Coder-14B-Instruct-4bit:    {{ 'PRESENT (ID)' if 'Qwen2.5-Coder-14B-Instruct-4bit' in coder_model_ids else ('PRESENT (alias)' if 'Qwen2.5-Coder-14B-Instruct-4bit' in coder_all_aliases else 'MISSING') }}"
+          - ""
+          - "  GGUF download: {{ 'NEW DOWNLOAD' if (coder_download is defined and coder_download.changed) else 'ALREADY PRESENT (skipped)' }}"
+          - "========================================================================"

ansible/playbooks/day2_add_nomic_embed.yml

@@ -0,0 +1,313 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day2_add_nomic_embed.yml
+# DESCRIPTION: Add nomic-embed-text-v1.5-Q4_K_M to the llama-server-router
+#              on astro-orbiter (10.1.71.130:8002).
+#
+# Context (t_34b96e83, 2026-08-13, OpenViking Phase 1b):
+#   Ryan approved adding nomic-embed-text-v1.5-Q4_K_M as an embedding model
+#   after Phase 0 follow-up confirmed embedding models fold cleanly into the
+#   existing router preset via embedding=true. Model ID is "nomic-embed-text-v1.5".
+#   No alias needed — peter-parker and Honcho consumers will call it by the section
+#   name directly.
+#
+# What this playbook does:
+#   1. Downloads nomic-embed-text-v1.5-Q4_K_M.gguf into /opt/models if not
+#      already present (idempotent: exact size-check guard, no re-pull on match).
+#   2. Redeploys the preset INI (adding the [nomic-embed-text-v1.5] section with
+#      embedding=true, n-gpu-layers=99, ctx-size=8192, load-on-startup=true,
+#      sleep-idle-seconds=-1).
+#   3. Restarts llama-server-router to pick up the new model entry.
+#   4. Verifies /v1/models returns all 5 models including the new nomic entry.
+#   5. Runs a /v1/embeddings smoke test to confirm the model actually embeds.
+#
+# VRAM context note (t_34b96e83):
+#   nomic-embed-text-v1.5 Q4_K_M: ~84MB weights, embedding model (no KV cache).
+#   VRAM impact is negligible — always pinned via sleep-idle-seconds=-1.
+#   The 4 generative models remain unchanged (OOM analysis unchanged from t_55c164f5).
+#
+# Usage (from ~/git/homelab/ansible):
+#   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook -i inventory.yml \
+#     playbooks/day2_add_nomic_embed.yml
+#
+# Semaphore note: Semaphore SSH key for jarvis user is not loaded in the
+# container (known pitfall, homelab-llm-serving skill). Run via CLI with
+# id_jarvis key; document as exception per Ryan's standing CLI fallback directive.
+#
+# Author: War Machine (2026-08-13, t_34b96e83)
+# ------------------------------------------------------------------------------
+
+- name: "Add nomic-embed-text-v1.5 embedding model to astro-orbiter router"
+  hosts: astro_orbiter
+  gather_facts: false
+  become: true
+
+  vars:
+    # Activate preset mode
+    llm_router_preset_enabled: true
+    llm_router_preset_path: /opt/llama-server-router-preset.ini
+
+    # Production port (router is on 8002 since t_cd0d5388)
+    llm_router_port: 8002
+
+    # Per-model ctx-size settings (carried from t_55c164f5; nomic new)
+    llm_router_llama_ctx_size: 8192
+    llm_router_llama_flash_attn: "true"
+    llm_router_phi_ctx_size: 32768
+    llm_router_phi_flash_attn: "true"
+    llm_router_coder_ctx_size: 16384
+    llm_router_coder_flash_attn: "true"
+    llm_router_nomic_ctx_size: 8192
+    # NOTE (2026-08-14, t_openviking_embed_batch): per-model batch-size/
+    # ubatch-size lines in the preset INI are NOT honored by llama-server's
+    # router — only ctx-size is applied per-model; batch-size/ubatch-size for
+    # every spawned child come from the router's own global CLI flags
+    # (confirmed via `ps aux` on astro-orbiter: child process launched with
+    # the router's --batch-size/--ubatch-size regardless of the INI values).
+    # Kept below for documentation/future-proofing but the REAL fix is the
+    # global llm_router_batch_size / llm_router_ubatch_size override further
+    # down, which raises the physical batch for ALL models on this router
+    # (Qwen3.6-35B, Phi, Llama, Coder, nomic).
+    llm_router_nomic_batch_size: 4096
+    llm_router_nomic_ubatch_size: 4096
+
+    # All other vars inherit from host_vars + defaults/main.yml.
+    llm_router_enabled: true
+    llm_service_user: jarvis
+    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
+    llm_models_dir: /opt/models
+    llm_bind_address: "10.1.71.130"
+    llm_allowed_source_cidr: "10.1.70.0/24"
+    llm_router_service_name: llama-server-router
+    llm_router_bind_address: "10.1.71.130"
+    llm_router_allowed_source_cidr: "10.1.70.0/24"
+    llm_router_models_dir: /opt/models
+    llm_router_models_max: 4            # from host_vars; bumped by t_33acbb2e
+    llm_router_ctx_size: 65536          # Qwen3.6-35B default; per-model overrides above
+    llm_router_parallel: 1
+    llm_router_gpu_layers: 99
+    # FIX (2026-08-14, t_openviking_embed_batch): raised from 512 to 4096.
+    # This is a GLOBAL router flag applied to every spawned model process
+    # (per-model INI batch-size/ubatch-size overrides are not honored by
+    # llama-server's router — see note above nomic vars). 512 tokens was too
+    # small for OpenViking's chunked-document embedding inputs (observed
+    # 2000-3400 tokens/chunk), causing hard 500 errors ("input (N tokens) is
+    # too large to process") that tripped OpenViking's circuit breaker into a
+    # permanent fail/re-enqueue loop. 4096 comfortably covers observed chunk
+    # sizes and stays under nomic's ctx-size=8192. VRAM impact of raising
+    # ubatch-size is in compute-buffer scratch space, not KV cache; monitored
+    # post-deploy against the 23000 MiB budget (host_vars/astro-orbiter).
+    llm_router_batch_size: 4096
+    llm_router_ubatch_size: 4096
+    llm_router_cache_type_k: q4_0
+    llm_router_cache_type_v: q4_0
+    llm_router_flash_attn: "auto"
+    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
+    llm_router_vram_max_mib: 23000
+
+    # nomic model staging
+    nomic_filename: "nomic-embed-text-v1.5-Q4_K_M.gguf"
+    nomic_url: "https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF/resolve/main/nomic-embed-text-v1.5.Q4_K_M.gguf"
+    nomic_size_bytes: 84106624
+
+  handlers:
+    - name: reload systemd
+      ansible.builtin.systemd:
+        daemon_reload: true
+      become: true
+      listen: "reload systemd"
+
+    - name: restart router
+      ansible.builtin.systemd:
+        name: llama-server-router
+        state: restarted
+      become: true
+      listen: "restart router"
+
+  tasks:
+
+    # ==========================================================================
+    # PHASE 1: Download nomic GGUF if not present / size mismatch
+    # ==========================================================================
+
+    - name: "[nomic] Stat existing GGUF"
+      ansible.builtin.stat:
+        path: "{{ llm_models_dir }}/{{ nomic_filename }}"
+        get_checksum: false
+      register: nomic_stat
+
+    - name: "[nomic] Download GGUF (skip if present and size matches)"
+      ansible.builtin.get_url:
+        url: "{{ nomic_url }}"
+        dest: "{{ llm_models_dir }}/{{ nomic_filename }}"
+        owner: "{{ llm_service_user }}"
+        group: "{{ llm_service_user }}"
+        mode: "0644"
+        timeout: 300
+      when: >
+        not nomic_stat.stat.exists or
+        nomic_stat.stat.size != nomic_size_bytes
+      register: nomic_download
+      notify: restart router
+
+    - name: "[nomic] Confirm GGUF size post-download"
+      ansible.builtin.stat:
+        path: "{{ llm_models_dir }}/{{ nomic_filename }}"
+        get_checksum: false
+      register: nomic_stat_post
+
+    - name: "[nomic] FAIL if GGUF size mismatch after download"
+      ansible.builtin.fail:
+        msg: >-
+          GGUF size mismatch: expected {{ nomic_size_bytes }} bytes,
+          got {{ nomic_stat_post.stat.size }} bytes.
+          Re-download may be needed.
+      when: nomic_stat_post.stat.size != nomic_size_bytes
+
+    # ==========================================================================
+    # PHASE 2: Deploy updated preset INI (adds nomic-embed-text-v1.5 section)
+    # ==========================================================================
+
+    - name: "[nomic] Deploy preset INI to {{ llm_router_preset_path }}"
+      ansible.builtin.template:
+        src: "../roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2"
+        dest: "{{ llm_router_preset_path }}"
+        owner: root
+        group: root
+        mode: "0644"
+      register: nomic_preset_deployed
+      notify: restart router
+
+    # ==========================================================================
+    # PHASE 3: Redeploy systemd unit (ensures unit is fresh; no flag changes)
+    # ==========================================================================
+
+    - name: "[nomic] Deploy llama-server-router unit"
+      ansible.builtin.template:
+        src: "../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
+        dest: /etc/systemd/system/llama-server-router.service
+        owner: root
+        group: root
+        mode: "0644"
+      register: nomic_unit_deployed
+      notify:
+        - reload systemd
+        - restart router
+
+    - name: "[nomic] Flush handlers (daemon-reload + router restart)"
+      ansible.builtin.meta: flush_handlers
+
+    # ==========================================================================
+    # PHASE 4: Verify router is up and nomic model appears in /v1/models
+    # ==========================================================================
+
+    - name: "[nomic] Wait for /health (router supervisor)"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
+        status_code: 200
+        timeout: 30
+      retries: 12
+      delay: 5
+      register: nomic_health
+      until: nomic_health.status == 200
+
+    - name: "[nomic] Query /v1/models"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
+        status_code: 200
+        return_content: true
+        timeout: 30
+      register: nomic_models
+
+    - name: "[nomic] Extract model IDs and aliases"
+      ansible.builtin.set_fact:
+        nomic_model_ids: "{{ nomic_models.json.data | map(attribute='id') | list }}"
+        nomic_all_aliases: "{{ nomic_models.json.data | map(attribute='aliases') | flatten | list }}"
+
+    - name: "[nomic] FAIL if nomic primary ID missing"
+      ansible.builtin.fail:
+        msg: >-
+          'nomic-embed-text-v1.5' not in /v1/models.
+          IDs: {{ nomic_model_ids }}
+      when: "'nomic-embed-text-v1.5' not in nomic_model_ids"
+
+    - name: "[nomic] FAIL if Qwen3.6-35B missing"
+      ansible.builtin.fail:
+        msg: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in /v1/models. IDs: {{ nomic_model_ids }}"
+      when: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in nomic_model_ids"
+
+    - name: "[nomic] FAIL if Phi missing"
+      ansible.builtin.fail:
+        msg: "'Phi-3.5-mini-instruct-Q8_0' not in /v1/models. IDs: {{ nomic_model_ids }}"
+      when: "'Phi-3.5-mini-instruct-Q8_0' not in nomic_model_ids"
+
+    - name: "[nomic] FAIL if Llama missing"
+      ansible.builtin.fail:
+        msg: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in /v1/models. IDs: {{ nomic_model_ids }}"
+      when: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in nomic_model_ids"
+
+    - name: "[nomic] FAIL if Coder missing"
+      ansible.builtin.fail:
+        msg: "'Qwen2.5-Coder-14B-Instruct-Q4_K_M' not in /v1/models. IDs: {{ nomic_model_ids }}"
+      when: "'Qwen2.5-Coder-14B-Instruct-Q4_K_M' not in nomic_model_ids"
+
+    # ==========================================================================
+    # PHASE 5: /v1/embeddings smoke test — confirm model actually embeds
+    # ==========================================================================
+
+    - name: "[nomic] POST /v1/embeddings smoke test"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/embeddings"
+        method: POST
+        body_format: json
+        body:
+          model: "nomic-embed-text-v1.5"
+          input: "The dog ran across the park."
+        status_code: 200
+        return_content: true
+        timeout: 120
+      register: nomic_embed_result
+
+    - name: "[nomic] Extract embedding vector length"
+      ansible.builtin.set_fact:
+        nomic_embed_dims: >-
+          {{ (nomic_embed_result.json.data | first).embedding | length }}
+      when:
+        - nomic_embed_result.status == 200
+        - nomic_embed_result.json.data is defined
+        - nomic_embed_result.json.data | length > 0
+
+    - name: "[nomic] FAIL if embedding vector is empty or missing"
+      ansible.builtin.fail:
+        msg: >-
+          Embedding smoke test returned no vector.
+          Response: {{ nomic_embed_result.json }}
+      when: >-
+        nomic_embed_result.status != 200 or
+        nomic_embed_result.json.data is not defined or
+        nomic_embed_result.json.data | length == 0 or
+        (nomic_embed_result.json.data | first).embedding | length == 0
+
+    - name: "[nomic] PASS — full summary"
+      ansible.builtin.debug:
+        msg:
+          - "========================================================================"
+          - "NOMIC-EMBED-TEXT-V1.5 DEPLOYMENT — COMPLETE"
+          - ""
+          - "  Mode:     --models-preset ({{ llm_router_preset_path }})"
+          - "  Service:  llama-server-router.service (:{{ llm_router_port }})"
+          - ""
+          - "  /v1/models IDs:     {{ nomic_model_ids }}"
+          - ""
+          - "  VERIFY:"
+          - "  Qwen3.6-35B-A3B-UD-Q4_K_S:          {{ 'PRESENT' if 'Qwen3.6-35B-A3B-UD-Q4_K_S' in nomic_model_ids else 'MISSING' }}"
+          - "  Phi-3.5-mini-instruct-Q8_0:          {{ 'PRESENT' if 'Phi-3.5-mini-instruct-Q8_0' in nomic_model_ids else 'MISSING' }}"
+          - "  Meta-Llama-3.1-8B-Instruct-Q4_K_M:  {{ 'PRESENT' if 'Meta-Llama-3.1-8B-Instruct-Q4_K_M' in nomic_model_ids else 'MISSING' }}"
+          - "  Qwen2.5-Coder-14B-Instruct-Q4_K_M:  {{ 'PRESENT' if 'Qwen2.5-Coder-14B-Instruct-Q4_K_M' in nomic_model_ids else 'MISSING' }}"
+          - "  nomic-embed-text-v1.5:               {{ 'PRESENT' if 'nomic-embed-text-v1.5' in nomic_model_ids else 'MISSING' }}"
+          - ""
+          - "  Embedding smoke test: PASS"
+          - "  Vector dimensions: {{ nomic_embed_dims | default('unknown') }}"
+          - ""
+          - "  GGUF download: {{ 'NEW DOWNLOAD' if (nomic_download is defined and nomic_download.changed) else 'ALREADY PRESENT (skipped)' }}"
+          - "========================================================================"

ansible/playbooks/day2_add_phi_alias.yml

@@ -0,0 +1,203 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day2_add_phi_alias.yml
+# DESCRIPTION: Add Phi-3.5-mini-instruct-8bit alias to the llama-server-router
+#              by switching from --models-dir to --models-preset INI mode.
+#
+# Context (t_9adf0889, 2026-08-12):
+#   Ryan's Hermes config (auxiliary.title_generation.model) points to
+#   "Phi-3.5-mini-instruct-8bit" but the router only exposes the GGUF
+#   filename-derived ID "Phi-3.5-mini-instruct-Q8_0". They are the same file.
+#   This playbook adds the alias so both names work without changing Ryan's
+#   Hermes config.
+#
+# What this playbook does:
+#   1. Deploys the preset INI template (llama-server-router-preset.ini.j2)
+#      to /opt/llama-server-router-preset.ini on astro-orbiter.
+#   2. Redeploys the systemd unit (llama-server-router.service) with
+#      --models-preset instead of --models-dir.
+#   3. Restarts llama-server-router to pick up the new flag.
+#   4. Verifies that /v1/models returns:
+#      - Phi-3.5-mini-instruct-Q8_0 (original ID — must still work)
+#      - Phi-3.5-mini-instruct-8bit (new alias — Ryan's config target)
+#      - Qwen3.6-35B-A3B-UD-Q4_K_S (unchanged)
+#      - Meta-Llama-3.1-8B-Instruct-Q4_K_M (unchanged)
+#
+# Known upstream behavior:
+#   GH #22364: --models-preset creates an extra "default" entry in /v1/models.
+#   This is cosmetic only and does not affect model selection by name.
+#
+# Usage (from ~/git/homelab/ansible):
+#   ansible-playbook -i inventory.yml playbooks/day2_add_phi_alias.yml
+#
+# Semaphore note (t_9adf0889): Semaphore SSH key for jarvis user is not loaded
+# in the container (known pitfall, homelab-llm-serving skill). Run via CLI with
+# id_jarvis key; document as exception per Ryan's standing CLI fallback directive.
+#
+# Author: War Machine (2026-08-12, t_9adf0889)
+# ------------------------------------------------------------------------------
+
+- name: "Add Phi-3.5-mini-instruct-8bit alias — switch router to preset mode"
+  hosts: astro_orbiter
+  gather_facts: false
+  become: true
+
+  vars:
+    # Activate preset mode and provide the on-disk INI path
+    llm_router_preset_enabled: true
+    llm_router_preset_path: /opt/llama-server-router-preset.ini
+
+    # Production port (router is already on 8002 since t_cd0d5388)
+    llm_router_port: 8002
+
+    # All other vars inherit from host_vars + defaults/main.yml.
+    # Explicitly set the ones needed by the unit template for clarity:
+    llm_router_enabled: true
+    llm_service_user: jarvis
+    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
+    llm_models_dir: /opt/models
+    llm_bind_address: "10.1.71.130"
+    llm_allowed_source_cidr: "10.1.70.0/24"
+    llm_router_service_name: llama-server-router
+    llm_router_bind_address: "10.1.71.130"
+    llm_router_allowed_source_cidr: "10.1.70.0/24"
+    llm_router_models_dir: /opt/models
+    llm_router_models_max: 4            # from host_vars; bumped by t_33acbb2e
+    llm_router_ctx_size: 65536
+    llm_router_parallel: 1
+    llm_router_gpu_layers: 99
+    llm_router_batch_size: 2048
+    llm_router_ubatch_size: 512
+    llm_router_cache_type_k: q4_0
+    llm_router_cache_type_v: q4_0
+    llm_router_flash_attn: "auto"
+    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
+    llm_router_vram_max_mib: 23000
+
+  handlers:
+    - name: reload systemd
+      ansible.builtin.systemd:
+        daemon_reload: true
+      become: true
+      listen: "reload systemd"
+
+    - name: restart router
+      ansible.builtin.systemd:
+        name: llama-server-router
+        state: restarted
+      become: true
+      listen: "restart router"
+
+  tasks:
+
+    # ==========================================================================
+    # PHASE 1: Deploy the preset INI
+    # ==========================================================================
+
+    - name: "[phi-alias] Deploy preset INI to {{ llm_router_preset_path }}"
+      ansible.builtin.template:
+        src: "../roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2"
+        dest: "{{ llm_router_preset_path }}"
+        owner: root
+        group: root
+        mode: "0644"
+      register: phi_alias_preset_deployed
+      notify:
+        - restart router
+
+    # ==========================================================================
+    # PHASE 2: Redeploy systemd unit with --models-preset flag
+    # ==========================================================================
+
+    - name: "[phi-alias] Deploy llama-server-router unit (--models-preset mode)"
+      ansible.builtin.template:
+        src: "../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
+        dest: /etc/systemd/system/llama-server-router.service
+        owner: root
+        group: root
+        mode: "0644"
+      register: phi_alias_unit_deployed
+      notify:
+        - reload systemd
+        - restart router
+
+    - name: "[phi-alias] Flush handlers (daemon-reload + router restart)"
+      ansible.builtin.meta: flush_handlers
+
+    # ==========================================================================
+    # PHASE 3: Verify alias is present
+    # ==========================================================================
+
+    - name: "[phi-alias] Wait for /health (router supervisor, no model needed)"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
+        status_code: 200
+        timeout: 30
+      retries: 12
+      delay: 5
+      register: phi_alias_health
+      until: phi_alias_health.status == 200
+
+    - name: "[phi-alias] Query /v1/models"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
+        status_code: 200
+        return_content: true
+        timeout: 30
+      register: phi_alias_models
+
+    - name: "[phi-alias] Extract model IDs and aliases"
+      ansible.builtin.set_fact:
+        phi_alias_model_ids: "{{ phi_alias_models.json.data | map(attribute='id') | list }}"
+        phi_alias_all_aliases: "{{ phi_alias_models.json.data | map(attribute='aliases') | flatten | list }}"
+        phi_alias_model_sources: "{{ phi_alias_models.json.data | map(attribute='source') | list }}"
+
+    - name: "[phi-alias] FAIL if Phi original ID missing"
+      ansible.builtin.fail:
+        msg: >-
+          'Phi-3.5-mini-instruct-Q8_0' not in /v1/models.
+          IDs: {{ phi_alias_model_ids }}
+      when: "'Phi-3.5-mini-instruct-Q8_0' not in phi_alias_model_ids"
+
+    - name: "[phi-alias] FAIL if Phi alias missing"
+      ansible.builtin.fail:
+        msg: >-
+          'Phi-3.5-mini-instruct-8bit' not found as ID or alias in /v1/models.
+          IDs:     {{ phi_alias_model_ids }}
+          Aliases: {{ phi_alias_all_aliases }}
+      when:
+        - "'Phi-3.5-mini-instruct-8bit' not in phi_alias_model_ids"
+        - "'Phi-3.5-mini-instruct-8bit' not in phi_alias_all_aliases"
+
+    - name: "[phi-alias] FAIL if Qwen missing"
+      ansible.builtin.fail:
+        msg: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in /v1/models. IDs: {{ phi_alias_model_ids }}"
+      when: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in phi_alias_model_ids"
+
+    - name: "[phi-alias] FAIL if Llama missing"
+      ansible.builtin.fail:
+        msg: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in /v1/models. IDs: {{ phi_alias_model_ids }}"
+      when: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in phi_alias_model_ids"
+
+    - name: "[phi-alias] PASS — full /v1/models summary"
+      ansible.builtin.debug:
+        msg:
+          - "========================================================================"
+          - "PHI ALIAS DEPLOYMENT — COMPLETE"
+          - ""
+          - "  Mode:     --models-preset ({{ llm_router_preset_path }})"
+          - "  Service:  llama-server-router.service (:{{ llm_router_port }})"
+          - ""
+          - "  /v1/models IDs:     {{ phi_alias_model_ids }}"
+          - "  /v1/models aliases: {{ phi_alias_all_aliases }}"
+          - "  Sources:            {{ phi_alias_model_sources }}"
+          - ""
+          - "  VERIFY:"
+          - "  Phi-3.5-mini-instruct-Q8_0:  {{ 'PRESENT' if 'Phi-3.5-mini-instruct-Q8_0' in phi_alias_model_ids else 'MISSING' }}"
+          - "  Phi-3.5-mini-instruct-8bit:  {{ 'PRESENT (ID)' if 'Phi-3.5-mini-instruct-8bit' in phi_alias_model_ids else ('PRESENT (alias)' if 'Phi-3.5-mini-instruct-8bit' in phi_alias_all_aliases else 'MISSING') }}"
+          - "  Qwen3.6-35B-A3B-UD-Q4_K_S:  {{ 'PRESENT' if 'Qwen3.6-35B-A3B-UD-Q4_K_S' in phi_alias_model_ids else 'MISSING' }}"
+          - "  Meta-Llama-3.1-8B-Instruct-Q4_K_M: {{ 'PRESENT' if 'Meta-Llama-3.1-8B-Instruct-Q4_K_M' in phi_alias_model_ids else 'MISSING' }}"
+          - ""
+          - "  GH #22364: if 'default' appears in IDs above, that is expected"
+          - "  in --models-preset mode. Cosmetic only."
+          - "========================================================================"

ansible/playbooks/day2_bump_router_models_max.yml

@@ -0,0 +1,161 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day2_bump_router_models_max.yml
+# DESCRIPTION: Bump --models-max on the production llama-server-router unit.
+#
+# Context: t_33acbb2e (2026-08-12) — Ryan requested --models-max raised from 1
+# to 4 so the router can keep multiple GGUFs resident on-demand (LRU eviction
+# when the cap is reached). The actual var change lives in:
+#   host_vars/astro-orbiter/vars.yml  (llm_router_models_max: 4)
+#
+# This playbook:
+#   1. Re-renders llama-server-router.service.j2 with the updated var value.
+#   2. Reloads systemd (daemon-reload handler) if the unit changed.
+#   3. Restarts llama-server-router so the new --models-max takes effect on the
+#      live process. Router holds no resident model (all-unloaded) so restart
+#      is sub-second and non-disruptive.
+#   4. Verifies /health returns 200 and /v1/models still lists all three GGUFs.
+#
+# VRAM NOTE: --models-max 4 allows up to all 3 current GGUFs to co-reside on
+# a 24GB card simultaneously. Worst-case combined footprint is ~31GB which
+# EXCEEDS 24GB — OOM is possible if all 3 are loaded concurrently. In normal
+# single-user homelab operation this is very unlikely. Full VRAM breakdown
+# documented in host_vars/astro-orbiter/vars.yml. Ryan approved (t_33acbb2e).
+#
+# Execution channel: Semaphore template "llm_router_update_unit" (project mk-labs).
+# Do NOT run via direct ansible-playbook or ad-hoc ssh/systemctl.
+#
+# Author: War Machine (2026-08-12, t_33acbb2e)
+# ------------------------------------------------------------------------------
+
+- name: "Bump llama-server-router --models-max to 4 on astro-orbiter"
+  hosts: astro_orbiter
+  gather_facts: true
+  become: true
+
+  vars:
+    # Production vars — router is live on :8002 (post-cutover t_cd0d5388)
+    llm_router_port: 8002
+    llm_router_bind_address: "10.1.71.130"
+
+    # llm_router_models_max is 4 via host_vars/astro-orbiter/vars.yml.
+    # Remaining role vars come from host_vars + defaults/main.yml via the
+    # inventory — we only explicitly set vars this playbook needs for its
+    # own tasks (health/models check URIs).
+
+    # Needed by the template task (mirrors defaults set in role defaults/main.yml)
+    llm_service_user: jarvis
+    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
+    llm_models_dir: /opt/models
+    llm_router_service_name: llama-server-router
+    llm_router_models_dir: /opt/models
+    llm_router_gpu_layers: 99
+    llm_router_ctx_size: 65536
+    llm_router_flash_attn: "auto"
+    llm_router_cache_type_k: q4_0
+    llm_router_cache_type_v: q4_0
+    llm_router_batch_size: 2048
+    llm_router_ubatch_size: 512
+    llm_router_parallel: 1
+
+  tasks:
+    # -------------------------------------------------------------------------
+    # Phase 1: Re-render the router unit file
+    # Template src path is relative to the role's templates/ dir; we reference
+    # it with a relative path that Ansible resolves from the role directory.
+    # -------------------------------------------------------------------------
+
+    - name: "Deploy updated llama-server-router unit (--models-max {{ llm_router_models_max }})"
+      ansible.builtin.template:
+        src: "{{ playbook_dir }}/../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
+        dest: "/etc/systemd/system/{{ llm_router_service_name }}.service"
+        owner: root
+        group: root
+        mode: "0644"
+      register: llm_router_unit_updated
+      notify:
+        - reload systemd
+      tags: [always]
+
+    - name: "Flush handlers — ensure daemon-reload lands before restart"
+      ansible.builtin.meta: flush_handlers
+      tags: [always]
+
+    # -------------------------------------------------------------------------
+    # Phase 2: Restart the router so the new --models-max takes effect.
+    # Always restart (even if unit unchanged) to ensure live process matches.
+    # -------------------------------------------------------------------------
+
+    - name: "Restart llama-server-router so --models-max {{ llm_router_models_max }} takes effect"
+      ansible.builtin.systemd:
+        name: "{{ llm_router_service_name }}"
+        state: restarted
+        enabled: true
+      tags: [always]
+
+    # -------------------------------------------------------------------------
+    # Phase 3: Verify /health returns 200
+    # -------------------------------------------------------------------------
+
+    - name: "Wait for /health to return 200 after restart"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
+        status_code: 200
+        timeout: 30
+      register: bump_health_check
+      retries: 10
+      delay: 3
+      until: bump_health_check.status == 200
+      tags: [always]
+
+    # -------------------------------------------------------------------------
+    # Phase 4: Verify /v1/models lists all three GGUFs
+    # -------------------------------------------------------------------------
+
+    - name: "Check /v1/models — all three GGUFs should appear"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
+        status_code: 200
+        timeout: 30
+        return_content: true
+      register: bump_models_check
+      tags: [always]
+
+    - name: "Display /v1/models summary"
+      ansible.builtin.debug:
+        msg:
+          - "======================================================================"
+          - "--models-max BUMP VERIFICATION (t_33acbb2e)"
+          - ""
+          - "  /health:               HTTP {{ bump_health_check.status }}"
+          - "  /v1/models HTTP:       {{ bump_models_check.status }}"
+          - "  Models listed:         {{ bump_models_check.json.data | map(attribute='id') | list | join(', ') }}"
+          - ""
+          - "  --models-max now:      {{ llm_router_models_max }}"
+          - "  --parallel (unchanged): {{ llm_router_parallel }}"
+          - ""
+          - "  VRAM WARNING: worst-case 3-model co-residency ~31GB > 24GB RTX 3090."
+          - "  OOM risk if all 3 load concurrently. LRU eviction mitigates in practice."
+          - "  Full breakdown: host_vars/astro-orbiter/vars.yml"
+          - "======================================================================"
+      when: bump_models_check is defined
+      tags: [always]
+
+    - name: "GATE: confirm all 3 expected GGUFs appear in /v1/models"
+      ansible.builtin.assert:
+        that:
+          - "'Qwen3.6-35B-A3B-UD-Q4_K_S' in (bump_models_check.json.data | map(attribute='id') | list)"
+          - "'Phi-3.5-mini-instruct-Q8_0' in (bump_models_check.json.data | map(attribute='id') | list)"
+          - "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' in (bump_models_check.json.data | map(attribute='id') | list)"
+        fail_msg: >-
+          /v1/models did not return all 3 expected GGUFs after --models-max bump.
+          Check router logs: journalctl -u llama-server-router -n 50
+        success_msg: "GATE PASSED: all 3 GGUFs listed in /v1/models."
+      when: bump_models_check is defined
+      tags: [always]
+
+  handlers:
+    - name: reload systemd
+      ansible.builtin.systemd:
+        daemon_reload: true
+      listen: "reload systemd"

ansible/playbooks/day2_cpu_offload_aux_models.yml

@@ -0,0 +1,59 @@
+---
+# ------------------------------------------------------------------------------
+# Playbook: day2_cpu_offload_aux_models.yml
+# Purpose:  CPU-offload Qwen2.5-Coder-14B and Meta-Llama-3.1-8B on
+#           astro-orbiter's production router (port 8002).
+#
+# What this playbook does:
+#   1. Re-renders llama-server-router-preset.ini (Coder + Llama sections now
+#      use per-model n-gpu-layers vars = 0 -> full CPU inference).
+#   2. Re-renders the router unit (--models-max now 4 via host_vars, global
+#      --n-gpu-layers removed per t_72646029 unit template fix) and restarts
+#      llama-server-router so both changes take effect.
+#   3. Verifies per the role's router_preset phase.
+#
+# Context (2026-08-17):
+#   - RAM/model-swap audit, TIER 1 (Coder-14B CPU offload) + TIER 2
+#     (Llama-3.1-8B CPU offload) — Ryan approved 1 & 2 on 2026-08-17.
+#     See inbox/ryan/2026-08-17-llm-system-ram-model-swap.md.
+#   - Unit template fix (t_72646029): global --n-gpu-layers removed from
+#     ExecStart in preset mode. Each INI section now sets n-gpu-layers
+#     explicitly (Qwen3.8=99, Phi=99, nomic=99, Coder=0, Llama=0).
+#   - Concurrent residency after change: Qwen3.8-27B (20,302 MiB @ 128K ctx)
+#     + nomic-embed (558 MiB, pinned) + Coder (CPU, ~1,390 MiB CUDA ctx) +
+#     Llama (CPU, ~1,706 MiB CUDA ctx) = ~24,004 MiB. NOTE: llama.cpp 6ea215d
+#     allocates CUDA-context VRAM even at n-gpu-layers=0, so CPU models are not
+#     0-VRAM; total sits at the 24,576 MiB physical limit (headroom ~572 MiB).
+#     Qwen3.8 is never evicted for a CPU aux model; Phi-3.5-mini (GPU, 8.3GB)
+#     still evicts as before.
+#   - CPU speed (8-core Ryzen 7 5800XT): ~5-10 tok/s (14B), ~10-20 tok/s (8B).
+#   - Semaphore SSH gap for astro-orbiter still applies (t_730f9584 /
+#     t_33acbb2e); running direct CLI Ansible per standing exception.
+#
+# Run:
+#   cd /home/hermes/git/homelab/ansible
+#   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook \
+#     -i inventory.yml \
+#     playbooks/day2_cpu_offload_aux_models.yml
+#
+# Rollback:
+#   git checkout -- \
+#     roles/llm-inference-multimodel/templates/llama-server-router.service.j2 \
+#     roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2 \
+#     roles/llm-inference-multimodel/defaults/main.yml \
+#     host_vars/astro-orbiter/vars.yml
+#   (restores n-gpu-layers=99 global flag, models-max=2, all GPU)
+#   then re-run this playbook to redeploy rollback state.
+#   Note: playbooks/day2_cpu_offload_aux_models.yml is untracked — left on disk.
+# ------------------------------------------------------------------------------
+- name: CPU-offload Coder-14B and Llama-3.1-8B on astro-orbiter
+  hosts: astro-orbiter
+  become: true
+  vars:
+    llm_router_preset_enabled: true
+    llm_router_enabled: true
+    llm_router_port: 8002
+
+  roles:
+    - role: llm-inference-multimodel
+      tags: [always]

ansible/playbooks/day2_cutover_qwen_to_router.yml

@@ -0,0 +1,511 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day2_cutover_qwen_to_router.yml
+# DESCRIPTION: Promote llama-server-router to production on port 8002.
+#
+# Context: Router-mode shadow deployment (t_0cca74a2) validated 2026-08-12:
+#   all 4 hard gates PASSED (n_ctx 65536, tool-calling PASS, hallucination-stress
+#   PASS, VRAM 20410 MiB / 1 process). Ryan approved cutover.
+#
+# This playbook makes the router the permanent production endpoint:
+#
+#   1. Stop + disable llama-server-qwen (:8002). Unit file is PRESERVED on disk
+#      as the rollback target (same pattern as prior role history).
+#   2. Redeploy llama-server-router unit file with --port 8002 (production port).
+#      PORT DECISION: we rebind the router to :8002 rather than updating 8
+#      dependent Hermes profiles' base_url. One unit file change beats 8
+#      config.yaml updates — atomic, GitOps-clean, zero profile drift.
+#   3. Enable + start llama-server-router on :8002.
+#   4. Re-run validation gates 1-3 against the NOW-production endpoint.
+#      (Same logic as Phase R / router_verify in tasks/router.yml — hard gates.)
+#   5. Run Gate 4: verify bundled SvelteKit UI is reachable.
+#
+# Usage (from ~/git/homelab/ansible):
+#   ansible-playbook -i inventory.yml playbooks/day2_cutover_qwen_to_router.yml
+#
+# Rollback (if gates fail or any time after):
+#   ansible-playbook -i inventory.yml playbooks/day2_cutover_qwen_to_router.yml \
+#     --tags cutover_rollback
+#
+# Author: War Machine (2026-08-12, t_cd0d5388)
+# Approved by: Ryan (cutover authorization, 2026-08-12)
+# ------------------------------------------------------------------------------
+
+- name: "CUTOVER — Promote llama-server-router to production (:8002) on astro-orbiter"
+  hosts: astro_orbiter
+  gather_facts: true
+  become: true
+
+  vars:
+    # ----------------------------------------------------------------
+    # PORT DECISION:
+    # We rebind the router to :8002 (production port) rather than
+    # updating 8 dependent Hermes profiles' base_url to :8003.
+    # Rationale: one unit file change is atomic and GitOps-clean.
+    # Updating 8 config.yaml files risks drift and requires per-profile
+    # activation tests. The template renders llm_router_port as the
+    # --port argument; we just override it here to 8002.
+    # ----------------------------------------------------------------
+
+    # Router port override: take over production port
+    llm_router_port: 8002
+
+    # All other role defaults needed by the template (mirrors defaults/main.yml)
+    llm_service_user: jarvis
+    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
+    llm_models_dir: /opt/models
+    llm_bind_address: "10.1.71.130"
+    llm_allowed_source_cidr: "10.1.70.0/24"
+
+    llm_router_enabled: true
+    llm_router_service_name: llama-server-router
+    llm_router_models_dir: /opt/models
+    llm_router_models_max: 1               # CRITICAL: RTX 3090 24GB, single model only
+    llm_router_ctx_size: 65536
+    llm_router_parallel: 1
+    llm_router_gpu_layers: 99
+    llm_router_batch_size: 2048
+    llm_router_ubatch_size: 512
+    llm_router_cache_type_k: q4_0
+    llm_router_cache_type_v: q4_0
+    llm_router_flash_attn: "auto"
+    llm_router_bind_address: "10.1.71.130"
+    llm_router_allowed_source_cidr: "10.1.70.0/24"
+    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
+    llm_router_vram_max_mib: 23000
+
+    llm_qwen_service_name: llama-server-qwen
+    llm_qwen_port: 8002
+
+  tasks:
+
+    # =======================================================================
+    # PHASE 1 — Stop and disable llama-server-qwen (bare single-model)
+    # Preserve unit file on disk — rollback target per existing role pattern.
+    # =======================================================================
+
+    - name: "[cutover] PHASE 1: Confirm llama-server-qwen current state"
+      ansible.builtin.systemd:
+        name: llama-server-qwen
+      register: cutover_qwen_status
+      tags: [cutover_stop_qwen, cutover]
+
+    - name: "[cutover] PHASE 1: Report current llama-server-qwen status"
+      ansible.builtin.debug:
+        msg: >-
+          llama-server-qwen: ActiveState={{ cutover_qwen_status.status.ActiveState | default('unknown') }},
+          UnitFileState={{ cutover_qwen_status.status.UnitFileState | default('unknown') }}.
+          Will stop + disable. Unit file preserved at /etc/systemd/system/llama-server-qwen.service as rollback target.
+      tags: [cutover_stop_qwen, cutover]
+
+    - name: "[cutover] PHASE 1: Stop llama-server-qwen (:8002, bare single-model)"
+      ansible.builtin.systemd:
+        name: llama-server-qwen
+        state: stopped
+      register: cutover_qwen_stopped
+      tags: [cutover_stop_qwen, cutover]
+
+    - name: "[cutover] PHASE 1: Disable llama-server-qwen (prevent auto-start on reboot)"
+      ansible.builtin.systemd:
+        name: llama-server-qwen
+        enabled: false
+      tags: [cutover_stop_qwen, cutover]
+
+    - name: "[cutover] PHASE 1: Wait 5s for VRAM to be released"
+      ansible.builtin.pause:
+        seconds: 5
+      when: cutover_qwen_stopped.changed | default(false)
+      tags: [cutover_stop_qwen, cutover]
+
+    - name: "[cutover] PHASE 1: Verify port 8002 is now free"
+      ansible.builtin.command:
+        cmd: ss -ltnp
+      register: cutover_port_check
+      changed_when: false
+      tags: [cutover_stop_qwen, cutover]
+
+    - name: "[cutover] PHASE 1: Fail if port 8002 is still bound"
+      ansible.builtin.fail:
+        msg: >-
+          Port 8002 is still bound after stopping llama-server-qwen.
+          Check 'ss -ltnp | grep :8002' and resolve before the router can bind.
+      when:
+        - "':8002 ' in (cutover_port_check.stdout | default('')) or ':8002:' in (cutover_port_check.stdout | default(''))"
+      tags: [cutover_stop_qwen, cutover]
+
+    - name: "[cutover] PHASE 1: Report VRAM state (should be empty)"
+      ansible.builtin.command:
+        cmd: nvidia-smi --query-compute-apps=pid,name,used_memory --format=csv,noheader
+      register: cutover_vram_free_check
+      changed_when: false
+      tags: [cutover_stop_qwen, cutover]
+
+    - name: "[cutover] PHASE 1: Print VRAM state"
+      ansible.builtin.debug:
+        msg: >-
+          VRAM after stopping llama-server-qwen:
+          {{ cutover_vram_free_check.stdout if (cutover_vram_free_check.stdout | length > 0)
+             else '(no GPU processes — VRAM free)' }}
+      tags: [cutover_stop_qwen, cutover]
+
+    # =======================================================================
+    # PHASE 2 — Redeploy llama-server-router unit with --port 8002
+    # =======================================================================
+
+    - name: "[cutover] PHASE 2: Deploy llama-server-router unit file (port 8002 — production)"
+      ansible.builtin.template:
+        src: "../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
+        dest: /etc/systemd/system/llama-server-router.service
+        owner: root
+        group: root
+        mode: "0644"
+      register: cutover_router_unit_deployed
+      notify:
+        - reload systemd
+      tags: [cutover_deploy_unit, cutover]
+
+    - name: "[cutover] PHASE 2: Flush handlers (daemon-reload before start)"
+      ansible.builtin.meta: flush_handlers
+      tags: [cutover_deploy_unit, cutover]
+
+    # =======================================================================
+    # PHASE 3 — Enable + start llama-server-router on :8002
+    # =======================================================================
+
+    - name: "[cutover] PHASE 3: Enable + start llama-server-router (production, :8002)"
+      ansible.builtin.systemd:
+        name: llama-server-router
+        state: "{{ 'restarted' if (cutover_router_unit_deployed.changed | default(false)) else 'started' }}"
+        enabled: true
+        daemon_reload: true
+      tags: [cutover_start_router, cutover]
+
+    # =======================================================================
+    # PHASE 4 — Validation gates 1-3 (hard gates against now-production :8002)
+    # =======================================================================
+
+    - name: "[cutover] GATE 1a: Wait for router /health on :8002 (up to 5min — cold model load)"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
+        status_code: 200
+      register: cutover_health
+      retries: 30
+      delay: 10
+      until: cutover_health.status == 200
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 1a: Trigger model load (router lazy-loads on first request)"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/chat/completions"
+        method: POST
+        body_format: json
+        body:
+          model: "{{ llm_router_expected_model_id }}"
+          messages:
+            - role: user
+              content: "Reply with one word: hello"
+          max_tokens: 5
+          temperature: 0.0
+        status_code: 200
+        return_content: true
+        timeout: 300
+      register: cutover_warmup
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 1a: Report warmup"
+      ansible.builtin.debug:
+        msg:
+          - "Model loaded. finish_reason={{ cutover_warmup.json.choices[0].finish_reason | default('unknown') }}"
+          - "Response: {{ cutover_warmup.json.choices[0].message.content | default('(empty)') | truncate(100) }}"
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 1b: Query /v1/models on :8002"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
+        status_code: 200
+        return_content: true
+      register: cutover_models
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 1b: Fail if expected model ID not found"
+      ansible.builtin.fail:
+        msg: >-
+          GATE 1 FAIL: '{{ llm_router_expected_model_id }}' not found in /v1/models.
+          Returned: {{ cutover_models.json.data | map(attribute='id') | list }}
+      when:
+        - cutover_models.json.data | selectattr('id', 'equalto', llm_router_expected_model_id) | list | length == 0
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 1b: Extract ctx-size from router model args"
+      ansible.builtin.set_fact:
+        cutover_qwen_n_ctx: >-
+          {%- set model = cutover_models.json.data | selectattr('id', 'equalto', llm_router_expected_model_id) | first -%}
+          {%- set args = model.status.args -%}
+          {%- set ctx_idx = args.index('--ctx-size') if '--ctx-size' in args else -1 -%}
+          {{ args[ctx_idx + 1] | int if ctx_idx >= 0 else 0 }}
+      when:
+        - cutover_models.json.data | selectattr('id', 'equalto', llm_router_expected_model_id) | list | length > 0
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 1b: Fail if n_ctx < 64000"
+      ansible.builtin.fail:
+        msg: "GATE 1 FAIL: --ctx-size={{ cutover_qwen_n_ctx }} < 64000 (Hermes 64K floor)."
+      when:
+        - cutover_qwen_n_ctx is defined
+        - cutover_qwen_n_ctx | int < 64000
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 1b: PASS — n_ctx >= 64K"
+      ansible.builtin.debug:
+        msg: "GATE 1 PASS: --ctx-size={{ cutover_qwen_n_ctx }} >= 64000."
+      when:
+        - cutover_qwen_n_ctx is defined
+        - cutover_qwen_n_ctx | int >= 64000
+      tags: [cutover_validate, cutover]
+
+    # --- Gate 2: Tool-calling through router proxy ---
+
+    - name: "[cutover] GATE 2: Tool-calling probe"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/chat/completions"
+        method: POST
+        body_format: json
+        body:
+          model: "{{ llm_router_expected_model_id }}"
+          messages:
+            - role: user
+              content: "What is the current weather in Chicago? Use the provided tool."
+          tools:
+            - type: function
+              function:
+                name: get_weather
+                description: "Get current weather conditions for a city"
+                parameters:
+                  type: object
+                  properties:
+                    city:
+                      type: string
+                      description: "The city name"
+                  required:
+                    - city
+          temperature: 0.0
+        status_code: 200
+        return_content: true
+        timeout: 120
+      register: cutover_toolcall_probe
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 2: Fail if not finish_reason=tool_calls"
+      ansible.builtin.fail:
+        msg: >-
+          GATE 2 FAIL: finish_reason={{ cutover_toolcall_probe.json.choices[0].finish_reason | default('(missing)') }}
+          (expected tool_calls). Response: {{ cutover_toolcall_probe.json | to_json }}
+      when:
+        - cutover_toolcall_probe.json.choices[0].finish_reason | default('') != 'tool_calls'
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 2: PASS"
+      ansible.builtin.debug:
+        msg:
+          - "GATE 2 PASS: finish_reason=tool_calls"
+          - "function: {{ cutover_toolcall_probe.json.choices[0].message.tool_calls[0].function.name | default('(unknown)') }}"
+          - "arguments: {{ cutover_toolcall_probe.json.choices[0].message.tool_calls[0].function.arguments | default('(none)') }}"
+      when:
+        - cutover_toolcall_probe.json.choices[0].finish_reason | default('') == 'tool_calls'
+      tags: [cutover_validate, cutover]
+
+    # --- Gate 2b: Hallucination stress ---
+
+    - name: "[cutover] GATE 2b: Hallucination stress probe"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/chat/completions"
+        method: POST
+        body_format: json
+        body:
+          model: "{{ llm_router_expected_model_id }}"
+          messages:
+            - role: user
+              content: "Tell me a brief fact about the planet Mars. Do not call any functions."
+          tools:
+            - type: function
+              function:
+                name: get_weather
+                description: "Get current weather conditions for a city"
+                parameters:
+                  type: object
+                  properties:
+                    city:
+                      type: string
+                  required:
+                    - city
+          temperature: 0.1
+        status_code: 200
+        return_content: true
+        timeout: 120
+      register: cutover_halluc_probe
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 2b: Fail if spurious tool_calls"
+      ansible.builtin.fail:
+        msg: >-
+          GATE 2b FAIL: finish_reason=tool_calls on unrelated prompt (Mars fact).
+          Over-triggering through router. Response: {{ cutover_halluc_probe.json | to_json }}
+      when:
+        - cutover_halluc_probe.json.choices[0].finish_reason | default('') == 'tool_calls'
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 2b: PASS"
+      ansible.builtin.debug:
+        msg: "GATE 2b PASS: finish_reason={{ cutover_halluc_probe.json.choices[0].finish_reason }} — no spurious tool_calls."
+      when:
+        - cutover_halluc_probe.json.choices[0].finish_reason | default('') != 'tool_calls'
+      tags: [cutover_validate, cutover]
+
+    # --- Gate 3: VRAM guard ---
+
+    - name: "[cutover] GATE 3: Check VRAM usage (--models-max 1 guard)"
+      ansible.builtin.command:
+        cmd: nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv,noheader
+      register: cutover_vram_post
+      changed_when: false
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 3: Parse VRAM used MiB"
+      ansible.builtin.set_fact:
+        cutover_vram_used_mib: "{{ cutover_vram_post.stdout.split(',')[0].strip().split(' ')[0] | int }}"
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 3: Fail if VRAM exceeds ceiling"
+      ansible.builtin.fail:
+        msg: >-
+          GATE 3 FAIL: {{ cutover_vram_used_mib }} MiB > {{ llm_router_vram_max_mib }} MiB ceiling.
+          Full: {{ cutover_vram_post.stdout }}
+      when:
+        - cutover_vram_used_mib | int > llm_router_vram_max_mib | int
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 3: Count GPU processes"
+      ansible.builtin.command:
+        cmd: nvidia-smi --query-compute-apps=pid,name --format=csv,noheader
+      register: cutover_gpu_procs
+      changed_when: false
+      failed_when: false
+      tags: [cutover_validate, cutover]
+
+    - name: "[cutover] GATE 3: PASS"
+      ansible.builtin.debug:
+        msg:
+          - "GATE 3 PASS: {{ cutover_vram_used_mib }} MiB / {{ llm_router_vram_max_mib }} MiB ceiling."
+          - "GPU processes: {{ cutover_gpu_procs.stdout_lines | default(['(none)']) }}"
+          - "Full nvidia-smi: {{ cutover_vram_post.stdout }}"
+      when:
+        - cutover_vram_used_mib | int <= llm_router_vram_max_mib | int
+      tags: [cutover_validate, cutover]
+
+    # =======================================================================
+    # PHASE 5 — Gate 4: Bundled SvelteKit Web UI (required this time)
+    # =======================================================================
+
+    - name: "[cutover] GATE 4: Check bundled SvelteKit UI at :8002"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/"
+        status_code: [200, 301, 302]
+        return_content: true
+        timeout: 30
+      register: cutover_ui_check
+      failed_when: false
+      tags: [cutover_validate, cutover_ui, cutover]
+
+    - name: "[cutover] GATE 4: Inspect UI content"
+      ansible.builtin.set_fact:
+        cutover_ui_is_html: "{{ 'html' in (cutover_ui_check.content | default('') | lower) or '<!doctype' in (cutover_ui_check.content | default('') | lower) }}"
+        cutover_ui_has_model_select: "{{ 'select' in (cutover_ui_check.content | default('') | lower) or 'model' in (cutover_ui_check.content | default('') | lower) }}"
+      when: cutover_ui_check is defined
+      tags: [cutover_validate, cutover_ui, cutover]
+
+    - name: "[cutover] GATE 4: Report UI check and bookmark URL"
+      ansible.builtin.debug:
+        msg:
+          - "======================================================================"
+          - "GATE 4 UI CHECK:"
+          - "  HTTP status:          {{ cutover_ui_check.status | default('UNREACHABLE') }}"
+          - "  Is HTML:              {{ cutover_ui_is_html | default(false) }}"
+          - "  Contains model/select: {{ cutover_ui_has_model_select | default(false) }}"
+          - "  BOOKMARK URL:         http://{{ llm_router_bind_address }}:{{ llm_router_port }}/"
+          - "  {{ 'GATE 4 PASS — UI serving HTML at :8002.' if (cutover_ui_check.status | default(0) | int in [200, 301, 302]) else 'GATE 4 WARN — UI not reachable (HTTP ' + (cutover_ui_check.status | default('FAIL') | string) + ').' }}"
+          - "======================================================================"
+      when: cutover_ui_check is defined
+      tags: [cutover_validate, cutover_ui, cutover]
+
+    # =======================================================================
+    # CUTOVER SUMMARY
+    # =======================================================================
+
+    - name: "[cutover] CUTOVER SUMMARY — production promoted"
+      ansible.builtin.debug:
+        msg:
+          - "======================================================================"
+          - "CUTOVER COMPLETE: llama-server-router is now production."
+          - ""
+          - "  Service:  llama-server-router.service (enabled, running)"
+          - "  Port:     8002 (unchanged for all 8 Hermes profiles)"
+          - "  Model:    {{ llm_router_expected_model_id }}"
+          - "  Mode:     Router/supervisor (--models-dir /opt/models, --models-max 1)"
+          - ""
+          - "  Gate 1 (n_ctx >= 64K):     PASS ({{ cutover_qwen_n_ctx | default('N/A') }})"
+          - "  Gate 2 (tool-calling):     PASS (finish_reason=tool_calls)"
+          - "  Gate 2b (halluc stress):   PASS (no spurious tool_calls)"
+          - "  Gate 3 (VRAM <= 23000MiB): PASS ({{ cutover_vram_used_mib | default('N/A') }} MiB)"
+          - "  Gate 4 (Web UI):           HTTP {{ cutover_ui_check.status | default('N/A') }}"
+          - ""
+          - "  ROLLBACK TARGET: /etc/systemd/system/llama-server-qwen.service (unit preserved)"
+          - "  ROLLBACK CMD: sudo systemctl enable --now llama-server-qwen"
+          - "               sudo systemctl disable --now llama-server-router"
+          - "  Or: ansible-playbook -i inventory.yml day2_cutover_qwen_to_router.yml --tags cutover_rollback"
+          - ""
+          - "  Web UI bookmark: http://{{ llm_router_bind_address }}:{{ llm_router_port }}/"
+          - "======================================================================"
+      tags: [cutover]
+
+    # =======================================================================
+    # ROLLBACK — tag cutover_rollback reverses the cutover
+    # Run: ansible-playbook -i inventory.yml day2_cutover_qwen_to_router.yml --tags cutover_rollback
+    # WARNING: rollback_task has no dependency on cutover tags — safe to run standalone.
+    # =======================================================================
+
+    - name: "[cutover_rollback] Stop + disable llama-server-router"
+      ansible.builtin.systemd:
+        name: llama-server-router
+        state: stopped
+        enabled: false
+      tags: [cutover_rollback, never]    # 'never' = only runs with explicit --tags cutover_rollback
+
+    - name: "[cutover_rollback] Enable + start llama-server-qwen (restore bare :8002)"
+      ansible.builtin.systemd:
+        name: llama-server-qwen
+        state: started
+        enabled: true
+      tags: [cutover_rollback, never]
+
+    - name: "[cutover_rollback] Verify rollback /health"
+      ansible.builtin.uri:
+        url: "http://{{ llm_bind_address | default('10.1.71.130') }}:8002/health"
+        status_code: 200
+        timeout: 30
+      register: cutover_rollback_health
+      failed_when: false
+      tags: [cutover_rollback, never]
+
+    - name: "[cutover_rollback] Report rollback result"
+      ansible.builtin.debug:
+        msg: >-
+          ROLLBACK: llama-server-qwen :8002 health returned
+          {{ cutover_rollback_health.status | default('UNREACHABLE') }}.
+          {{ 'OK — production restored to bare qwen.' if (cutover_rollback_health.status | default(0) | int == 200)
+             else 'WARNING — health check failed. Check manually.' }}
+      tags: [cutover_rollback, never]
+
+  handlers:
+    - name: reload systemd
+      ansible.builtin.systemd:
+        daemon_reload: true

ansible/playbooks/day2_per_model_ctx_size.yml

@@ -0,0 +1,277 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day2_per_model_ctx_size.yml
+# DESCRIPTION: Right-size --ctx-size per model workload on llama-server-router
+#              (already in --models-preset mode since t_9adf0889).
+#
+# Context (t_ryan_per_model_ctx, 2026-08-13, requested by Ryan via JARVIS):
+#   All 3 preset models currently launch with a uniform --ctx-size 65536.
+#   This playbook narrows two of them to match actual workload:
+#     - Meta-Llama-3.1-8B-Instruct-Q4_K_M (alias Meta-Llama-3.1-8B-Instruct-4bit):
+#         ctx-size 65536 -> 8192  (tool-routing / micro-tasks: title gen, MCP
+#         tool calls, approval checks)
+#     - Phi-3.5-mini-instruct-Q8_0 (alias Phi-3.5-mini-instruct-8bit):
+#         ctx-size 65536 -> 32768 (long web scrapes / session-log compression)
+#   Both also move flash-attn from "auto" to explicit "true" per Ryan's spec.
+#   Qwen3.6-35B-A3B-UD-Q4_K_S is INTENTIONALLY left untouched at 65536/auto.
+#
+#   Existing aliases (Meta-Llama-3.1-8B-Instruct-4bit, Phi-3.5-mini-instruct-8bit)
+#   are PRESERVED as-is. Ryan's pasted TOML used different alias strings
+#   ("llama-3.1-8b", "phi-3.5-mini") but renaming aliases was not explicitly
+#   requested and would break live Hermes custom_providers routing — flagged
+#   in the deployment report rather than applied silently.
+#
+# IMPORTANT — Hermes side effect: /home/hermes/.hermes/config.yaml declares
+# context_length: 65536 for both these models under custom_providers. This
+# playbook does NOT touch that file (out of role/agent scope) but the value
+# becomes STALE the moment this playbook lands. Flag to JARVIS/Maria Hill.
+#
+# Usage (from ~/git/homelab/ansible):
+#   ansible-playbook -i inventory.yml playbooks/day2_per_model_ctx_size.yml
+#
+# Author: War Machine (2026-08-13, t_ryan_per_model_ctx)
+# ------------------------------------------------------------------------------
+
+- name: "Right-size per-model ctx-size on llama-server-router (Llama 8k, Phi 32k)"
+  hosts: astro_orbiter
+  gather_facts: false
+  become: true
+
+  vars:
+    # Preset mode already active in production (t_9adf0889) — keep it on.
+    llm_router_preset_enabled: true
+    llm_router_preset_path: /opt/llama-server-router-preset.ini
+    llm_router_enabled: true
+
+    # Production port
+    llm_router_port: 8002
+    llm_router_bind_address: "10.1.71.130"
+    llm_router_allowed_source_cidr: "10.1.70.0/24"
+    llm_bind_address: "10.1.71.130"
+    llm_allowed_source_cidr: "10.1.70.0/24"
+
+    llm_service_user: jarvis
+    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
+    llm_models_dir: /opt/models
+    llm_router_service_name: llama-server-router
+    llm_router_models_dir: /opt/models
+    llm_router_models_max: 4
+    llm_router_parallel: 1
+    llm_router_gpu_layers: 99
+    llm_router_batch_size: 2048
+    llm_router_ubatch_size: 512
+    llm_router_cache_type_k: q4_0
+    llm_router_cache_type_v: q4_0
+
+    # Qwen — untouched baseline (also used as router-wide fallback default)
+    llm_router_ctx_size: 65536
+    llm_router_flash_attn: "auto"
+    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
+    llm_router_vram_max_mib: 23000
+
+    # --- THE CHANGE: per-model overrides ---
+    llm_router_llama_ctx_size: 8192
+    llm_router_llama_flash_attn: "true"
+    llm_router_phi_ctx_size: 32768
+    llm_router_phi_flash_attn: "true"
+
+  handlers:
+    - name: reload systemd
+      ansible.builtin.systemd:
+        daemon_reload: true
+      become: true
+      listen: "reload systemd"
+
+    - name: restart router
+      ansible.builtin.systemd:
+        name: llama-server-router
+        state: restarted
+      become: true
+      listen: "restart router"
+
+  tasks:
+
+    # ==========================================================================
+    # PHASE 1: Deploy the preset INI with new per-model ctx-size/flash-attn
+    # ==========================================================================
+
+    - name: "[ctx-resize] Deploy preset INI to {{ llm_router_preset_path }}"
+      ansible.builtin.template:
+        src: "../roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2"
+        dest: "{{ llm_router_preset_path }}"
+        owner: root
+        group: root
+        mode: "0644"
+      register: ctx_resize_preset_deployed
+      notify:
+        - restart router
+
+    - name: "[ctx-resize] Deploy router systemd unit (drop global --ctx-size/--flash-attn in preset mode)"
+      ansible.builtin.template:
+        src: "../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
+        dest: /etc/systemd/system/llama-server-router.service
+        owner: root
+        group: root
+        mode: "0644"
+      register: ctx_resize_unit_deployed
+      notify:
+        - reload systemd
+        - restart router
+
+    - name: "[ctx-resize] Flush handlers (daemon-reload + router restart if changed)"
+      ansible.builtin.meta: flush_handlers
+
+    # ==========================================================================
+    # PHASE 2: Verify
+    # ==========================================================================
+
+    - name: "[ctx-resize] Wait for /health"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
+        status_code: 200
+        timeout: 30
+      retries: 12
+      delay: 5
+      register: ctx_resize_health
+      until: ctx_resize_health.status == 200
+
+    - name: "[ctx-resize] Query /v1/models"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
+        status_code: 200
+        return_content: true
+        timeout: 30
+      register: ctx_resize_models
+
+    - name: "[ctx-resize] Trigger load — Llama (confirms actual load + captures live args)"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/chat/completions"
+        method: POST
+        body_format: json
+        body:
+          model: "Meta-Llama-3.1-8B-Instruct-Q4_K_M"
+          messages:
+            - role: user
+              content: "Reply with one word: hello"
+          max_tokens: 5
+          temperature: 0.0
+        status_code: 200
+        return_content: true
+        timeout: 120
+      register: ctx_resize_llama_warmup
+
+    - name: "[ctx-resize] Trigger load — Phi (confirms actual load + captures live args)"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/chat/completions"
+        method: POST
+        body_format: json
+        body:
+          model: "Phi-3.5-mini-instruct-Q8_0"
+          messages:
+            - role: user
+              content: "Reply with one word: hello"
+          max_tokens: 5
+          temperature: 0.0
+        status_code: 200
+        return_content: true
+        timeout: 120
+      register: ctx_resize_phi_warmup
+
+    - name: "[ctx-resize] Re-query /v1/models after warmup (final state)"
+      ansible.builtin.uri:
+        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
+        status_code: 200
+        return_content: true
+        timeout: 30
+      register: ctx_resize_models_final
+
+    - name: "[ctx-resize] Extract Llama args"
+      ansible.builtin.set_fact:
+        ctx_resize_llama_args: >-
+          {{ (ctx_resize_models_final.json.data | selectattr('id', 'equalto', 'Meta-Llama-3.1-8B-Instruct-Q4_K_M') | first).status.args }}
+        ctx_resize_llama_status: >-
+          {{ (ctx_resize_models_final.json.data | selectattr('id', 'equalto', 'Meta-Llama-3.1-8B-Instruct-Q4_K_M') | first).status.value }}
+
+    - name: "[ctx-resize] Extract Phi args"
+      ansible.builtin.set_fact:
+        ctx_resize_phi_args: >-
+          {{ (ctx_resize_models_final.json.data | selectattr('id', 'equalto', 'Phi-3.5-mini-instruct-Q8_0') | first).status.args }}
+        ctx_resize_phi_status: >-
+          {{ (ctx_resize_models_final.json.data | selectattr('id', 'equalto', 'Phi-3.5-mini-instruct-Q8_0') | first).status.value }}
+
+    - name: "[ctx-resize] Extract Qwen args (must be unchanged)"
+      ansible.builtin.set_fact:
+        ctx_resize_qwen_args: >-
+          {{ (ctx_resize_models_final.json.data | selectattr('id', 'equalto', 'Qwen3.6-35B-A3B-UD-Q4_K_S') | first).status.args }}
+
+    - name: "[ctx-resize] GATE — Llama ctx-size must be 8192"
+      ansible.builtin.assert:
+        that:
+          - "'8192' in ctx_resize_llama_args"
+          - ctx_resize_llama_args[ctx_resize_llama_args.index('--ctx-size') + 1] == '8192'
+        fail_msg: "Llama ctx-size not 8192. Args: {{ ctx_resize_llama_args }}"
+        success_msg: "Llama ctx-size confirmed 8192."
+
+    - name: "[ctx-resize] GATE — Llama flash-attn must be true"
+      ansible.builtin.assert:
+        that:
+          - ctx_resize_llama_args[ctx_resize_llama_args.index('--flash-attn') + 1] == 'true'
+        fail_msg: "Llama flash-attn not true. Args: {{ ctx_resize_llama_args }}"
+        success_msg: "Llama flash-attn confirmed true."
+
+    - name: "[ctx-resize] GATE — Llama loaded successfully"
+      ansible.builtin.assert:
+        that:
+          - ctx_resize_llama_status == 'loaded'
+        fail_msg: "Llama status is '{{ ctx_resize_llama_status }}', expected 'loaded'."
+        success_msg: "Llama status confirmed 'loaded'."
+
+    - name: "[ctx-resize] GATE — Phi ctx-size must be 32768"
+      ansible.builtin.assert:
+        that:
+          - ctx_resize_phi_args[ctx_resize_phi_args.index('--ctx-size') + 1] == '32768'
+        fail_msg: "Phi ctx-size not 32768. Args: {{ ctx_resize_phi_args }}"
+        success_msg: "Phi ctx-size confirmed 32768."
+
+    - name: "[ctx-resize] GATE — Phi flash-attn must be true"
+      ansible.builtin.assert:
+        that:
+          - ctx_resize_phi_args[ctx_resize_phi_args.index('--flash-attn') + 1] == 'true'
+        fail_msg: "Phi flash-attn not true. Args: {{ ctx_resize_phi_args }}"
+        success_msg: "Phi flash-attn confirmed true."
+
+    - name: "[ctx-resize] GATE — Phi loaded successfully"
+      ansible.builtin.assert:
+        that:
+          - ctx_resize_phi_status == 'loaded'
+        fail_msg: "Phi status is '{{ ctx_resize_phi_status }}', expected 'loaded'."
+        success_msg: "Phi status confirmed 'loaded'."
+
+    - name: "[ctx-resize] GATE — Qwen ctx-size UNCHANGED at 65536"
+      ansible.builtin.assert:
+        that:
+          - ctx_resize_qwen_args[ctx_resize_qwen_args.index('--ctx-size') + 1] == '65536'
+        fail_msg: "Qwen ctx-size changed unexpectedly! Args: {{ ctx_resize_qwen_args }}"
+        success_msg: "Qwen ctx-size confirmed UNCHANGED at 65536."
+
+    - name: "[ctx-resize] PASS — summary"
+      ansible.builtin.debug:
+        msg:
+          - "================================================================"
+          - "PER-MODEL CTX-SIZE DEPLOYMENT — COMPLETE"
+          - ""
+          - "  Llama-3.1-8B (Meta-Llama-3.1-8B-Instruct-Q4_K_M):"
+          - "    status:     {{ ctx_resize_llama_status }}"
+          - "    args:       {{ ctx_resize_llama_args }}"
+          - ""
+          - "  Phi-3.5-mini (Phi-3.5-mini-instruct-Q8_0):"
+          - "    status:     {{ ctx_resize_phi_status }}"
+          - "    args:       {{ ctx_resize_phi_args }}"
+          - ""
+          - "  Qwen3.6-35B-A3B-UD-Q4_K_S: UNCHANGED (ctx-size 65536, args: {{ ctx_resize_qwen_args }})"
+          - ""
+          - "  ACTION NEEDED: /home/hermes/.hermes/config.yaml custom_providers"
+          - "  context_length: 65536 for both Meta-Llama-3.1-8B-Instruct-4bit and"
+          - "  Phi-3.5-mini-instruct-8bit is now STALE (actual: 8192 / 32768)."
+          - "  Flag to JARVIS/Maria Hill for correction — NOT done by this playbook."
+          - "================================================================"

ansible/playbooks/day2_qwen38_ctx128k.yml

@@ -0,0 +1,38 @@
+---
+# ------------------------------------------------------------------------------
+# Playbook: day2_qwen38_ctx128k.yml
+# Purpose:  Bump Qwen3.8-27B-Q4_K_M ctx-size from 32768 to 131072 (128K)
+#           on astro-orbiter's production router (port 8002).
+#
+# What this playbook does:
+#   1. Renders the updated llama-server-router-preset.ini.j2 (now with
+#      llm_router_qwen38_ctx_size: 131072) to /opt/llama-server-router-preset.ini.
+#   2. Restarts llama-server-router.service.
+#   3. Verifies the router loads Qwen3.8-27B at ctx=131072 in status.args.
+#
+# Context:
+#   - Empirical VRAM test (t_4455a44c): 131072 ctx = 20,282 MiB Qwen3.8
+#     + 558 MiB nomic-embed = ~20.8GB total; ~3.2GB headroom on 24GB RTX 3090.
+#     Co-resident with nomic-embed: comfortably fits.
+#   - Ryan approved this deployment.
+#   - Semaphore SSH gap for astro-orbiter still applies (t_730f9584 / t_33acbb2e);
+#     running direct CLI Ansible per standing exception.
+#
+# Run:
+#   cd /home/hermes/git/homelab/ansible
+#   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook \
+#     -i inventory.yml \
+#     playbooks/day2_qwen38_ctx128k.yml
+#
+# Task reference: t_441470b9 — War Machine, 2026-08-16
+# ------------------------------------------------------------------------------
+- name: Bump Qwen3.8-27B ctx-size to 131072 on astro-orbiter
+  hosts: astro-orbiter
+  become: true
+  vars:
+    llm_router_preset_enabled: true
+    llm_router_qwen38_ctx_size: 131072
+
+  roles:
+    - role: llm-inference-multimodel
+      tags: [preset, systemd, verify]

ansible/playbooks/day2_qwen38_ctx128k_rollback.yml

@@ -0,0 +1,65 @@
+---
+# ------------------------------------------------------------------------------
+# Playbook: day2_qwen38_ctx128k_rollback.yml
+# Purpose:  Roll back Qwen3.8-27B-Q4_K_M ctx-size from 131072 back to 65536
+#           on astro-orbiter's production router (port 8002).
+#
+# What this playbook does:
+#   1. Renders the updated llama-server-router-preset.ini.j2 (now with
+#      llm_router_qwen38_ctx_size: 65536) to
+#      /opt/llama-server-router-preset.ini.
+#   2. Restarts llama-server-router.service.
+#   3. Verifies the router loads Qwen3.8-27B at ctx=65536 in status.args.
+#
+# Context:
+#   - t_441470b9 (2026-08-16): ctx-size bumped 32768 -> 131072. Verified VRAM
+#     at 131072 ctx with only Qwen3.8 + nomic-embed co-resident: ~20,282 MiB
+#     + 558 MiB = ~20.8 GB on 24 GB RTX 3090. Comfortably safe.
+#   - t_72646029 (2026-08-17): Phi-3.5mini moved to GPU (n-gpu-layers=99)
+#     to enable concurrent residency with CPU-offloaded Coder-14B and
+#     Llama-3.1-8B. This added ~2GB CUDA context buffers for Phi + shifted
+#     Phi's model weights onto the GPU (~3.8GB).
+#   - NEW steady-state VRAM: Qwen3.8 @ 131072 ctx (~20,282 MiB) + nomic-embed
+#     (~558 MiB) + Llama CUDA ctx (~1,706 MiB) + Coder CUDA ctx (~1,390 MiB)
+#     = ~24,004 MiB. Adding Phi-3.5 (~3,800 MiB weights + ~1.4 GB CUDA ctx)
+#     pushes total to ~29,000+ MiB — exceeding the 24,576 MiB RTX 3090 limit.
+#     Qwen3.8-27B-131072 now fails to load (HTTP 500, OOM before llama.cpp
+#     reaches the model-loading phase).
+#   - FIX: reduce Qwen3.8 ctx-size 131072 -> 65536. This reduces KV cache
+#     from ~6GB to ~3GB, freeing ~3GB of VRAM. New estimated steady-state:
+#     Qwen3.8 @ 65536 ctx (~17,068 MiB) + nomic (~558) + Llama ctx (~1,706)
+#     + Coder ctx (~1,390) + Phi-3.5 (~3,800 + ~1,400 CUDA ctx) = ~25,922 MiB.
+#     Still over 24,576 — see "Phase 2" below for the secondary fix.
+#
+#   IMPORTANT: Rolling back ctx-size alone may NOT be sufficient. The
+#   hardware reference (astro-orbiter-hardware.md line 166, t_72646029)
+#   states steady-state ~24,004 MiB WITHOUT Phi on GPU. Adding Phi-3.5 back
+#   to GPU tips it over. This playbook handles the context rollback; if Qwen3.8
+#   still fails to load after Phase R, Wong should escalate to Ryan for a
+#   decision on either (a) offloading Phi-3.5mini to CPU (n-gpu-layers=0),
+#   or (b) adding a second GPU. Document the Phase 2 finding as a separate
+#   follow-up task if needed.
+#
+#   The 64K floor from the 2026-08-12 cutover validation (t_cd0d5388, Gate 1)
+#   still applies — ctx-size=65536 satisfies it.
+#
+# Run:
+#   cd /home/hermes/git/homelab/ansible
+#   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook \
+#     -i inventory.yml \
+#     playbooks/day2_qwen38_ctx128k_rollback.yml
+#
+# Task reference: t_c9fed26c — War Machine benchmark, 2026-08-18
+# Root cause: t_72646029 CPU-offload deployment added Phi-3.5 to GPU,
+#   shifting total VRAM past the 24,576 MiB ceiling when Qwen3.8 runs at 128K.
+# ------------------------------------------------------------------------------
+- name: Roll back Qwen3.8-27B ctx-size to 65536 on astro-orbiter
+  hosts: astro-orbiter
+  become: true
+  vars:
+    llm_router_preset_enabled: true
+    llm_router_qwen38_ctx_size: 65536
+
+  roles:
+    - role: llm-inference-multimodel
+      tags: [preset, systemd, verify]

ansible/playbooks/day2_swap_qwen38.yml

@@ -0,0 +1,36 @@
+---
+# ------------------------------------------------------------------------------
+# Playbook: day2_swap_qwen38.yml
+# Purpose:  Swap the primary production model on astro-orbiter router from
+#           Qwen3.6-35B-A3B-UD-Q4_K_S to Qwen3.8-27B-Q4_K_M.
+#           This is a GitOps-encoded record of the swap performed 2026-08-16
+#           per Ryan's direction (kanban task t_f5f7e9ad).
+#
+# What this playbook does:
+#   1. Renders the updated llama-server-router-preset.ini.j2 to
+#      /opt/llama-server-router-preset.ini on astro-orbiter.
+#   2. Reloads the llama-server-router service (SIGHUP / restart as needed).
+#   3. Verifies the new model ID appears in /v1/models.
+#
+# Prerequisites:
+#   - Qwen3.8-27B-Q4_K_M.gguf must be present in /opt/models on astro-orbiter.
+#     (Downloaded out-of-band via wget during the swap task.)
+#   - roles/llm-inference-multimodel/defaults/main.yml updated to reference
+#     Qwen3.8-27B-Q4_K_M (done in this same commit).
+#
+# Run:
+#   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook \
+#     -i inventory.yml \
+#     playbooks/day2_swap_qwen38.yml
+#
+# Task reference: t_f5f7e9ad — War Machine, 2026-08-16
+# ------------------------------------------------------------------------------
+- name: Swap primary model to Qwen3.8-27B-Q4_K_M on astro-orbiter
+  hosts: astro-orbiter
+  become: true
+  vars:
+    llm_router_preset_enabled: true
+
+  roles:
+    - role: llm-inference-multimodel
+      tags: [preset, systemd, verify]

ansible/playbooks/roles/common/tasks/main.yml

@@ -1,77 +0,0 @@
----
-# ------------------------------------------------------------------------------
-# FILE: roles/common/tasks/main.yml
-# DESCRIPTION: Baseline configuration applied to all managed Ubuntu hosts.
-#              Handles hostname, timezone, core packages, NTP, ansible user,
-#              and optional LVM root volume expansion.
-# ------------------------------------------------------------------------------
-
-- name: Set hostname
-  hostname:
-    name: "{{ inventory_hostname | replace('_', '-') }}"
-
-- name: Set timezone
-  timezone:
-    name: "{{ common_timezone }}"
-
-- name: Update apt cache
-  apt:
-    update_cache: yes
-    cache_valid_time: 3600
-
-- name: Install base utility packages
-  apt:
-    name: "{{ common_packages }}"
-    state: present
-
-- name: Install chrony
-  apt:
-    name: chrony
-    state: present
-
-- name: Configure chrony to use sundial
-  template:
-    src: chrony.conf.j2
-    dest: /etc/chrony/chrony.conf
-    mode: '0644'
-  notify: restart chrony
-
-- name: Ensure chrony is enabled and running
-  systemd:
-    name: chrony
-    state: started
-    enabled: yes
-
-- name: Ensure ansible user has passwordless sudo
-  lineinfile:
-    path: /etc/sudoers.d/{{ ansible_user }}
-    line: "{{ ansible_user }} ALL=(ALL) NOPASSWD:ALL"
-    create: yes
-    mode: '0440'
-    validate: 'visudo -cf %s'
-
-# ------------------------------------------------------------------------------
-# LVM root volume expansion
-# Extends the root PV to the full disk size and grows the LV + filesystem.
-# This codifies what was previously done manually after provisioning.
-# Runs only when common_expand_root_lvm is true (default: true).
-# Safe to re-run — pvresize and lvextend are idempotent when already at max.
-# ------------------------------------------------------------------------------
-
-- name: Expand root PV to full disk size
-  command: pvresize {{ common_root_pv }}
-  register: pvresize_result
-  changed_when: "'changed' in pvresize_result.stdout or pvresize_result.rc == 0"
-  when: common_expand_root_lvm | bool
-
-- name: Extend root LV to 100% of free VG space
-  lvol:
-    vg: "{{ common_root_vg }}"
-    lv: "{{ common_root_lv }}"
-    size: +100%FREE
-    resizefs: yes
-  when:
-    - common_expand_root_lvm | bool
-  ignore_errors: yes
-  # ignore_errors because lvextend returns non-zero when already at max size.
-  # resizefs: yes handles the resize2fs call inline — no separate task needed.

ansible/playbooks/roles/semaphore/defaults/main.yml

@@ -1,5 +0,0 @@
----
-# file: roles/semaphore/defaults/main.yml
-
-semaphore_compose_dir: /opt/docker/semaphore
-semaphore_ssh_key_file: "~/.ssh/ansible"

ansible/playbooks/roles/semaphore/handlers/main.yml

@@ -1,7 +0,0 @@
----
-# file: roles/semaphore/handlers/main.yml
-
-- name: Restart Semaphore
-  community.docker.docker_compose_v2:
-    project_src: "{{ semaphore_compose_dir }}"
-    state: restarted

ansible/playbooks/roles/semaphore/tasks/main.yml

@@ -1,50 +0,0 @@
----
-# file: roles/semaphore/tasks/main.yml
-
-- name: Create Semaphore directory
-  ansible.builtin.file:
-    path: "{{ semaphore_compose_dir }}"
-    state: directory
-    owner: wed
-    group: docker
-    mode: '0755'
-
-- name: Copy Compose file from boilerplate
-  ansible.builtin.copy:
-    src: "{{ playbook_dir }}/../../boilerplates/semaphore/compose.yaml"
-    dest: "{{ semaphore_compose_dir }}/compose.yaml"
-    owner: wed
-    group: docker
-    mode: '0644'
-
-- name: Deploy Semaphore .env file
-  ansible.builtin.template:
-    src: env.j2
-    dest: "{{ semaphore_compose_dir }}/.env"
-    owner: wed
-    group: docker
-    mode: '0600'
-
-- name: Copy SSH key for Ansible authentication
-  ansible.builtin.copy:
-    src: "{{ semaphore_ssh_key_file }}"
-    dest: "{{ semaphore_compose_dir }}/ansible_key"
-    owner: "1001"
-    group: "1001"
-    mode: '0600'
-
-- name: Start Semaphore containers
-  community.docker.docker_compose_v2:
-    project_src: "{{ semaphore_compose_dir }}"
-    state: present
-  register: semaphore_compose
-
-- name: Wait for Semaphore to be ready
-  ansible.builtin.uri:
-    url: "http://localhost:3000/api/ping"
-    method: GET
-    status_code: 200
-  register: semaphore_health
-  retries: 48
-  delay: 5
-  until: semaphore_health.status == 200

ansible/playbooks/roles/semaphore/templates/env.j2

@@ -1,3 +0,0 @@
-DATABASE_PASSWORD={{ vault_semaphore_database_password }}
-SEMAPHORE_ADMIN_PASSWORD={{ vault_semaphore_admin_password }}
-SEMAPHORE_ACCESS_KEY_ENCRYPTION={{ vault_semaphore_access_key_encryption }}

ansible/requirements.yml

@@ -1,21 +1,6 @@
-# requirements.yml
 ---
 collections:
-  - name: community.general
+  - name: containers.podman
-    version: 11.4.1
+    version: ">=1.10.0"
-
+  - name: effectivelywild.technitium_dns
-  # - name: nccurry.openshift
+    version: ">=1.1.0"
-  #   version: 1.4.0
-
-  - name: somaz94.ansible_k8s_iac_tool
-    version: 1.1.6
-
-  # - name: prometheus.prometheus
-  #   version: 0.27.0
-
-  - name: kubernetes.core
-
-  - name: community.proxmox
-    version: 1.4.0
-
-  - name: effectivelywild.technitium_dns

ansible/roles/common/defaults/main.yml

@@ -8,6 +8,7 @@ common_timezone: America/Chicago
 common_ntp_server: "sundial.local.mk-labs.cloud"
 
 common_packages:
+  - acl
   - curl
   - wget
   - vim

ansible/roles/common/tasks/main.yml

@@ -0,0 +1,21 @@
+- name: Create cast group
+  ansible.builtin.group:
+    name: cast
+    state: present
+    system: true
+
+- name: Create cast user
+  ansible.builtin.user:
+    name: cast
+    group: cast
+    shell: /bin/bash
+    create_home: true
+    state: present
+
+- name: Ensure cast user has passwordless sudo
+  ansible.builtin.lineinfile:
+    path: /etc/sudoers.d/cast
+    line: "cast ALL=(ALL) NOPASSWD:ALL"
+    create: yes
+    mode: '0440'
+    validate: 'visudo -cf %s'

ansible/roles/day0-baseline/defaults/main.yml

@@ -0,0 +1,12 @@
+---
+# Timezone
+timezone: "UTC"
+
+# Packages to ensure are present
+baseline_packages:
+  - chrony
+  - vim
+  - htop
+  - curl
+  - wget
+  - rsync

ansible/roles/day0-baseline/handlers/main.yml

@@ -0,0 +1,6 @@
+---
+- name: Restart sshd
+  ansible.builtin.systemd:
+    name: sshd
+    state: restarted
+  become: true

ansible/roles/day0-baseline/tasks/main.yml

@@ -0,0 +1,32 @@
+---
+- name: Set timezone
+  community.general.timezone:
+    name: "{{ timezone }}"
+  become: true
+
+- name: Enable and start chronyd
+  ansible.builtin.systemd:
+    name: chronyd
+    state: started
+    enabled: true
+  become: true
+
+- name: Update all packages
+  ansible.builtin.package:
+    name: "*"
+    state: latest
+  become: true
+
+- name: Install baseline packages
+  ansible.builtin.package:
+    name: "{{ baseline_packages }}"
+    state: present
+  become: true
+
+- name: Disable root SSH login
+  ansible.builtin.lineinfile:
+    path: /etc/ssh/sshd_config
+    regexp: '^PermitRootLogin'
+    line: "PermitRootLogin no"
+  become: true
+  notify: Restart sshd

ansible/roles/expand_root_lv/README.md

@@ -0,0 +1,70 @@
+# expand-root-lv role
+
+Idempotent role that extends the root LVM logical volume to fill its
+volume group and grows the underlying filesystem (ext4 or xfs).
+
+## Where this runs in the lifecycle
+
+Part of the **day0** host-provisioning lifecycle. The canonical entry
+points are:
+
+```
+playbooks/day0_expand_root_lv.yml   # standalone
+playbooks/day0_provision.yml        # umbrella (baseline + expand_root_lv)
+```
+
+Day1 application-deploy playbooks should NOT include this role —
+day0 is assumed complete before day1 begins.
+
+## Why this role exists
+
+The Ubuntu Server autoinstall template (used by the mk-labs `wed`-baked
+VM templates) provisions the root LV at roughly half the available disk
+size — a longstanding installer default that surprises every operator
+who hasn't been bitten by it before. ~90% of mk-labs VMs need this
+fix-up before they're fully useful.
+
+## Idempotency
+
+- If `vg_free_count == 0`, the `lvextend` step is skipped and the
+  filesystem-grow step is also skipped (nothing to resize against).
+- If the target volume group doesn't exist on the host (e.g. a non-LVM
+  layout), the role exits cleanly via `meta: end_play`.
+- Safe to leave in a recurring playbook so future disk expansions
+  (Proxmox-side disk grow → reboot → run role) are picked up
+  automatically.
+
+## Opt-out for multi-LV hosts
+
+If a host will have a **second logical volume in the same VG** (e.g. a
+dedicated `/var/lib/postgresql` LV for a database server), this role's
+"grow root to fill VG" behavior is wrong — it will consume the free PE
+that was being reserved for the second LV.
+
+Set in `host_vars/<host>.yml`:
+
+```yaml
+expand_root_lv_skip: true
+```
+
+The day0 playbook checks this flag and skips the role cleanly.
+
+## Defaults
+
+| Variable                      | Default       | Purpose                                             |
+|-------------------------------|---------------|-----------------------------------------------------|
+| `expand_root_lv_vg_name`      | `ubuntu-vg`   | LVM volume group name (Ubuntu installer default).   |
+| `expand_root_lv_lv_name`      | `ubuntu-lv`   | LVM logical volume name (Ubuntu installer default). |
+| `expand_root_lv_mountpoint`   | `/`           | Mountpoint of the filesystem to grow.               |
+
+Override the VG/LV names in `host_vars/<host>.yml` for hosts that use a
+different LVM layout.
+
+## Limitations
+
+- Does not extend the underlying partition. If the operator grows the
+  Proxmox disk and the partition itself needs to grow before lvextend
+  can claim the new space, run `growpart /dev/sda 3` (or equivalent)
+  first. A future enhancement could automate this via `cloud-utils`'
+  `growpart` package, but it's out of scope for the initial template
+  fix-up case where the partition already covers the whole disk.

ansible/roles/expand_root_lv/defaults/main.yml

@@ -0,0 +1,26 @@
+---
+# ============================================================================
+# expand-root-lv role defaults
+# ============================================================================
+# Extends the root LVM logical volume to fill its volume group, then grows
+# the underlying filesystem to match. Idempotent: when there's no free PE
+# in the VG (i.e. the LV already fills the VG), the lvextend step is a
+# no-op and resize2fs/xfs_growfs simply confirms the filesystem is at
+# capacity.
+#
+# Designed for Ubuntu cloud-image-style installations where the autoinstall
+# template provisions an LV at half the disk size (the Ubuntu Server
+# installer's longstanding default). Run once after VM provisioning to
+# reclaim the unallocated PE; safe to leave in a day1 playbook so future
+# disk expansions are picked up automatically.
+# ============================================================================
+
+# The LV and VG names follow the Ubuntu Server installer's convention.
+# Override per-host if your template differs.
+expand_root_lv_vg_name: ubuntu-vg
+expand_root_lv_lv_name: ubuntu-lv
+
+# Mount point we expect to be backed by the target LV. Used purely for
+# the resize2fs / xfs_growfs decision — the role inspects this path's
+# filesystem type and dispatches to the correct grow command.
+expand_root_lv_mountpoint: /

ansible/roles/expand_root_lv/meta/main.yml

@@ -0,0 +1,23 @@
+---
+galaxy_info:
+  role_name: expand_root_lv
+  author: JARVIS
+  description: >-
+    Idempotent role that extends the root LVM logical volume to fill its
+    volume group and grows the underlying filesystem (ext4 or xfs). Fixes
+    the half-disk LV that the Ubuntu Server autoinstall template ships
+    with by default.
+  license: MIT
+  min_ansible_version: "2.14"
+  platforms:
+    - name: Ubuntu
+      versions:
+        - noble
+        - jammy
+  galaxy_tags:
+    - lvm
+    - cloud-init
+    - homelab
+    - storage
+
+dependencies: []

ansible/roles/expand_root_lv/tasks/main.yml

@@ -0,0 +1,70 @@
+---
+# ============================================================================
+# expand-root-lv / main
+# ----------------------------------------------------------------------------
+# 1. Confirm the target VG exists (skip role cleanly on non-LVM hosts).
+# 2. Read free physical-extent count for the VG.
+# 3. Extend the LV to +100%FREE only when free_pe > 0.
+# 4. Grow the filesystem on the mountpoint (ext4 -> resize2fs, xfs -> xfs_growfs).
+# Each step is idempotent and skips when there's nothing to do.
+# ============================================================================
+
+- name: Gather LVM facts
+  ansible.builtin.command:
+    cmd: "vgs --noheadings --nosuffix --units b -o vg_name,vg_free_count {{ expand_root_lv_vg_name }}"
+  register: vg_info
+  changed_when: false
+  failed_when: false
+
+- name: Skip role when target VG is absent
+  ansible.builtin.meta: end_play
+  when: vg_info.rc != 0
+
+- name: Parse free PE count
+  ansible.builtin.set_fact:
+    expand_root_lv_free_pe: "{{ (vg_info.stdout.split() | last | int) if vg_info.stdout | length > 0 else 0 }}"
+
+- name: Extend LV to fill VG (only if free PE > 0)
+  ansible.builtin.command:
+    cmd: "lvextend -l +100%FREE /dev/{{ expand_root_lv_vg_name }}/{{ expand_root_lv_lv_name }}"
+  register: lvextend_result
+  when: expand_root_lv_free_pe | int > 0
+  changed_when: lvextend_result.rc == 0
+
+- name: Detect filesystem type at mountpoint
+  ansible.builtin.command:
+    cmd: "findmnt {{ expand_root_lv_mountpoint }} -no FSTYPE"
+  register: fstype_result
+  changed_when: false
+
+- name: Set filesystem type fact
+  ansible.builtin.set_fact:
+    expand_root_lv_fstype: "{{ fstype_result.stdout | trim }}"
+
+- name: Grow ext4 filesystem
+  ansible.builtin.command:
+    cmd: "resize2fs /dev/{{ expand_root_lv_vg_name }}/{{ expand_root_lv_lv_name }}"
+  register: resize_result
+  when:
+    - expand_root_lv_fstype == "ext4"
+    - lvextend_result.changed | default(false)
+  changed_when: resize_result.rc == 0
+
+- name: Grow xfs filesystem
+  ansible.builtin.command:
+    cmd: "xfs_growfs {{ expand_root_lv_mountpoint }}"
+  register: xfs_result
+  when:
+    - expand_root_lv_fstype == "xfs"
+    - lvextend_result.changed | default(false)
+  changed_when: xfs_result.rc == 0
+
+- name: Report current root size
+  ansible.builtin.command:
+    cmd: "df -h {{ expand_root_lv_mountpoint }}"
+  register: df_result
+  changed_when: false
+
+- name: Show post-resize disk usage
+  ansible.builtin.debug:
+    msg: "{{ df_result.stdout_lines }}"

ansible/roles/hermes/defaults/main.yml

@@ -0,0 +1,26 @@
+---
+# Hermes service user
+hermes_user: hermes
+hermes_group: hermes
+hermes_home: /home/hermes
+
+# Install flags
+# Set to true if you don't need browser automation (skips Playwright/Chromium)
+hermes_skip_browser: false
+
+# systemd service name (gateway)
+hermes_service_name: hermes
+
+# Path where hermes binary will be accessible system-wide
+hermes_bin_symlink: /usr/local/bin/hermes
+
+# -----------------------------------------------------------------------------
+# Hermes Web UI (dashboard) — fronted by Traefik on lightning-lane via the
+# 'jarvis' service. Binds to 0.0.0.0 so Traefik can reach it from upstream;
+# --insecure is acceptable because TLS + auth are terminated at Traefik.
+# -----------------------------------------------------------------------------
+hermes_dashboard_enabled: true
+hermes_dashboard_service_name: hermes-dashboard
+hermes_dashboard_host: 0.0.0.0
+hermes_dashboard_port: 9119
+hermes_dashboard_insecure: true

ansible/roles/hermes/handlers/main.yml

@@ -0,0 +1,9 @@
+---
+- name: reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true
+
+- name: restart hermes
+  ansible.builtin.systemd:
+    name: "{{ hermes_service_name }}"
+    state: restarted

ansible/roles/hermes/tasks/main.yml

@@ -0,0 +1,199 @@
+---
+# ---------------------------------------------------------------------------
+# 1. System prerequisites (run as root via become)
+# ---------------------------------------------------------------------------
+
+- name: Install system packages required by Hermes installer
+  ansible.builtin.apt:
+    name:
+      - git
+      - curl
+      - ffmpeg
+      - ripgrep
+    state: present
+    update_cache: true
+  become: true
+
+- name: Install Node.js 22 (required for browser automation and WhatsApp bridge)
+  block:
+    - name: Download NodeSource setup script
+      ansible.builtin.get_url:
+        url: https://deb.nodesource.com/setup_22.x
+        dest: /tmp/nodesource_setup.sh
+        mode: "0755"
+
+    - name: Run NodeSource setup script
+      ansible.builtin.command: bash /tmp/nodesource_setup.sh
+      args:
+        creates: /etc/apt/sources.list.d/nodesource.list
+
+    - name: Install nodejs
+      ansible.builtin.apt:
+        name: nodejs
+        state: present
+        update_cache: true
+  become: true
+
+# ---------------------------------------------------------------------------
+# 2. Install Playwright system deps for Chromium (root-only step)
+#    Per docs: this is the one thing that genuinely needs root.
+#    Skipped entirely if hermes_skip_browser is true.
+# ---------------------------------------------------------------------------
+
+- name: Install Playwright Chromium system dependencies
+  ansible.builtin.command: npx --yes playwright install-deps chromium
+  become: true
+  when: not hermes_skip_browser
+  changed_when: true
+  environment:
+    DEBIAN_FRONTEND: noninteractive
+
+# ---------------------------------------------------------------------------
+# 3. Create dedicated hermes service user
+# ---------------------------------------------------------------------------
+
+- name: Create hermes group
+  ansible.builtin.group:
+    name: "{{ hermes_group }}"
+    state: present
+    system: true
+  become: true
+
+- name: Create hermes user
+  ansible.builtin.user:
+    name: "{{ hermes_user }}"
+    group: "{{ hermes_group }}"
+    home: "{{ hermes_home }}"
+    shell: /bin/bash
+    system: true
+    create_home: true
+    comment: "Hermes Agent service account"
+  become: true
+
+- name: Grant hermes user passwordless sudo
+  ansible.builtin.copy:
+    content: "hermes ALL=(ALL) NOPASSWD:ALL\n"
+    dest: /etc/sudoers.d/hermes
+    owner: root
+    group: root
+    mode: "0440"
+    validate: /usr/sbin/visudo -cf %s
+  become: true
+
+- name: Ensure hermes home directory has correct permissions
+  ansible.builtin.file:
+    path: "{{ hermes_home }}"
+    owner: "{{ hermes_user }}"
+    group: "{{ hermes_group }}"
+    mode: "0750"
+    state: directory
+  become: true
+
+# ---------------------------------------------------------------------------
+# 4. Run the Hermes installer as the hermes user
+# ---------------------------------------------------------------------------
+
+- name: Check if hermes is already installed
+  ansible.builtin.stat:
+    path: "{{ hermes_home }}/.hermes/hermes-agent/venv/bin/hermes"
+  register: hermes_binary
+
+- name: Run Hermes installer as hermes user
+  ansible.builtin.shell: |
+    curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh \
+      | bash -s -- --skip-setup{% if hermes_skip_browser %} --skip-browser{% endif %}
+  args:
+    executable: /bin/bash
+  become: true
+  become_user: "{{ hermes_user }}"
+  environment:
+    HOME: "{{ hermes_home }}"
+    PATH: "{{ hermes_home }}/.local/bin:/usr/local/bin:/usr/bin:/bin"
+  when: not hermes_binary.stat.exists
+  changed_when: true
+
+# ---------------------------------------------------------------------------
+# 5. Symlink hermes binary into system PATH
+#    Per docs: service accounts often lack ~/.local/bin in PATH; symlink
+#    into /usr/local/bin so hermes is always accessible.
+# ---------------------------------------------------------------------------
+
+- name: Symlink hermes binary to system PATH
+  ansible.builtin.file:
+    src: "{{ hermes_home }}/.hermes/hermes-agent/venv/bin/hermes"
+    dest: "{{ hermes_bin_symlink }}"
+    state: link
+    force: true
+  become: true
+
+# ---------------------------------------------------------------------------
+# 6. Install systemd service unit
+#    NOTE: The service starts in 'gateway' mode for persistent operation.
+#    You must run 'sudo -u hermes hermes setup' interactively on first boot
+#    to configure your LLM provider and any messaging gateways before
+#    enabling the service.
+# ---------------------------------------------------------------------------
+
+- name: Deploy hermes systemd service unit
+  ansible.builtin.template:
+    src: hermes.service.j2
+    dest: /etc/systemd/system/{{ hermes_service_name }}.service
+    owner: root
+    group: root
+    mode: "0644"
+  become: true
+  notify:
+    - reload systemd
+
+- name: Flush handlers to reload systemd now
+  ansible.builtin.meta: flush_handlers
+
+- name: Enable hermes service (but do not start — config required first)
+  ansible.builtin.systemd:
+    name: "{{ hermes_service_name }}"
+    enabled: true
+    state: stopped
+  become: true
+
+# ---------------------------------------------------------------------------
+# 7. Install systemd service unit for the Hermes Web UI (dashboard)
+#    Separate unit from the gateway so the UI can be restarted / disabled
+#    independently. Fronted upstream by Traefik (jarvis service) on
+#    lightning-lane, so binding 0.0.0.0 with --insecure is intentional.
+# ---------------------------------------------------------------------------
+
+- name: Deploy hermes-dashboard systemd service unit
+  ansible.builtin.template:
+    src: hermes-dashboard.service.j2
+    dest: /etc/systemd/system/{{ hermes_dashboard_service_name }}.service
+    owner: root
+    group: root
+    mode: "0644"
+  become: true
+  register: hermes_dashboard_unit
+  when: hermes_dashboard_enabled
+
+- name: Reload systemd to pick up hermes-dashboard unit changes
+  ansible.builtin.systemd:
+    daemon_reload: true
+  become: true
+  when:
+    - hermes_dashboard_enabled
+    - hermes_dashboard_unit is changed
+
+- name: Enable and start hermes-dashboard service
+  ansible.builtin.systemd:
+    name: "{{ hermes_dashboard_service_name }}"
+    enabled: true
+    state: started
+  become: true
+  when: hermes_dashboard_enabled
+
+- name: Restart hermes-dashboard on unit change
+  ansible.builtin.systemd:
+    name: "{{ hermes_dashboard_service_name }}"
+    state: restarted
+  become: true
+  when:
+    - hermes_dashboard_enabled
+    - hermes_dashboard_unit is changed

ansible/roles/hermes/templates/hermes-dashboard.service.j2

@@ -0,0 +1,24 @@
+[Unit]
+Description=Hermes Dashboard (Web UI)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User={{ hermes_user }}
+Group={{ hermes_group }}
+WorkingDirectory={{ hermes_home }}
+Environment="HOME={{ hermes_home }}"
+Environment="PATH={{ hermes_home }}/.local/bin:/usr/local/bin:/usr/bin:/bin"
+Environment="HERMES_HOME={{ hermes_home }}/.hermes"
+ExecStart={{ hermes_home }}/.hermes/hermes-agent/venv/bin/hermes dashboard --host {{ hermes_dashboard_host }} --port {{ hermes_dashboard_port }}{% if hermes_dashboard_insecure %} --insecure{% endif %}
+
+TimeoutStopSec=30
+Restart=always
+RestartSec=5
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=hermes-dashboard
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/hermes/templates/hermes.service.j2

@@ -0,0 +1,23 @@
+[Unit]
+Description=Hermes Agent (Nous Research)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User={{ hermes_user }}
+Group={{ hermes_group }}
+WorkingDirectory={{ hermes_home }}
+Environment="HOME={{ hermes_home }}"
+Environment="PATH={{ hermes_home }}/.local/bin:/usr/local/bin:/usr/bin:/bin"
+Environment="HERMES_HOME={{ hermes_home }}/.hermes"
+ExecStart={{ hermes_home }}/.hermes/hermes-agent/venv/bin/hermes gateway
+TimeoutStopSec=200
+Restart=on-failure
+RestartSec=10
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=hermes
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/honcho/README.md

@@ -0,0 +1,118 @@
+# Honcho role
+
+Deploys [Honcho](https://honcho.dev) — Plastic Labs' memory + theory-of-mind
+layer for stateful agents — on a single mk-labs VM (`lincoln`) using
+rootful Podman + Quadlet.
+
+## Topology
+
+```
+                                 ┌──────────────────────────────┐
+                                 │  lightning-lane (Traefik)    │
+                                 │  hall-of-presidents.l...     │
+                                 └─────────────┬────────────────┘
+                                               │
+                                               │  HTTP :8000
+                                               ▼
+   ┌──────────────────────────────── lincoln ───────────────────────────────┐
+   │                                                                        │
+   │   ┌───────────────┐    ┌───────────────┐    ┌──────────────────────┐   │
+   │   │ honcho-api    │    │ honcho-deriver│    │ honcho-postgres      │   │
+   │   │ (FastAPI:8000)│◄──►│ (worker loop) │◄──►│ pgvector/pgvector:pg16│  │
+   │   └──────┬────────┘    └──────┬────────┘    └──────────────────────┘   │
+   │          │                    │                                        │
+   │          └────────┬───────────┘                                        │
+   │                   ▼                                                    │
+   │            Anthropic Claude API (outbound, east-west to Internet)      │
+   │                                                                        │
+   └────────────────────────────────────────────────────────────────────────┘
+```
+
+## What this role does
+
+1. Installs Podman + ensures the rootful Quadlet drop-in dir exists.
+2. Creates a user-defined Podman network (`honcho-net`).
+3. Creates a named volume for Postgres data (`honcho_postgres_data`).
+4. Deploys a pgvector-enabled Postgres 16 container and waits for it to be
+   ready.
+5. Deploys the Honcho API container (`honcho-api`) — FastAPI on :8000.
+6. Deploys the Honcho deriver worker container (`honcho-deriver`) — **stopped
+   and disabled by default** (see below).
+7. Verifies the API answers on `/docs`.
+
+## Deriver service (disabled by default)
+
+The `honcho-deriver` service is **created but not started** by default
+(`honcho_deriver_autostart: false`). This prevents autonomous token burn.
+
+**Why disabled:**  
+The deriver runs background reasoning loops that consume LLM tokens
+continuously, independent of the Hermes client-side `contextCadence` and
+`dialecticCadence` settings. Even with both cadences set to zero in
+`~/.hermes/honcho.json`, the deriver polls Postgres every second and fires
+Claude API calls autonomously.
+
+In production, this discovered behavior cost $10 USD overnight before the
+service was stopped.
+
+**Manual start (if needed for testing):**
+
+```bash
+# On lincoln.local.mk-labs.cloud as jarvis or with sudo:
+sudo systemctl start honcho-deriver.service
+
+# Check status:
+sudo systemctl status honcho-deriver.service
+
+# Stop again:
+sudo systemctl stop honcho-deriver.service
+```
+
+**To enable permanently:**  
+Override `honcho_deriver_autostart: true` in `group_vars` or inventory, then
+re-run the playbook.
+
+## Required vault entries
+
+Add to `group_vars/all/vault` (ansible-vault encrypted):
+
+```yaml
+vault_honcho_database_password: "<strong random>"
+vault_honcho_jwt_secret: "<32+ byte random>"
+vault_honcho_anthropic_api_key: "sk-ant-..."
+```
+
+## LLM provider switching
+
+This role starts with Anthropic Claude. To swap to a local
+OpenAI-compatible endpoint later (e.g. astro-orbiter once we have the
+GPU running llama.cpp/Ollama in OpenAI-compatible mode), override these
+in `group_vars/honcho_server` or via a playbook variable:
+
+```yaml
+honcho_llm_transport: "openai"
+honcho_deriver_model:   "qwen3:8b"
+honcho_summary_model:   "qwen3:8b"
+honcho_dialectic_model: "qwen3:8b"
+# and add LLM_OPENAI_API_KEY (or set the base URL override in the template)
+```
+
+## Traefik route
+
+The Traefik file-provider YAML for `hall-of-presidents.local.mk-labs.cloud`
+lives at `boilerplates/traefik/dynamic/honcho.yml`. It points at
+`http://lincoln.local.mk-labs.cloud:8000`.
+
+## Known Quadlet pitfall (carried from the semaphore role)
+
+Quadlet regenerates the systemd unit on `.container` file change but does
+NOT restart the running container. This role explicitly handles that by
+gating a `state: restarted` task on the template's `changed` status —
+the same pattern documented in `homelab-application-deployment`'s
+quadlet-patterns reference.
+
+## Honcho version pinning
+
+The `honcho_image` default is `ghcr.io/plastic-labs/honcho:latest`.
+Move to a digest-pinned tag once we've validated the deployment works
+against a known-good build.

ansible/roles/honcho/defaults/main.yml

@@ -0,0 +1,159 @@
+---
+# ============================================================================
+# honcho role defaults
+# ============================================================================
+# Deploys Honcho (https://honcho.dev — plastic-labs/honcho) as a rootful
+# Podman + Quadlet service on a single VM. Three containers:
+#
+#   honcho-postgres   pgvector-enabled PostgreSQL backing store
+#   honcho-api        FastAPI server on :8000  (theory-of-mind read/write)
+#   honcho-deriver    background worker that consumes the queue and calls
+#                     out to the configured LLM provider for derivations
+#
+# All three share a user-defined podman network. State persists to named
+# volumes. Traefik on lightning-lane terminates TLS and routes the
+# `hall-of-presidents.local.mk-labs.cloud` host to honcho-api:8000.
+#
+# LLM provider for the first deployment is Anthropic Claude. Switching
+# providers is a single env-var change (see DERIVER_MODEL_CONFIG__TRANSPORT
+# / SUMMARY_MODEL_CONFIG__TRANSPORT and the LLM_*_API_KEY variables).
+# ============================================================================
+
+# ---------------------------------------------------------------------------
+# Image pinning
+# ---------------------------------------------------------------------------
+# Honcho does not yet publish a Docker Hub image we trust; we use the
+# GitHub Container Registry build. Pin to a digest-stable tag.
+honcho_image: "ghcr.io/plastic-labs/honcho:latest"
+# pgvector/pgvector image follows the upstream postgres version channel.
+honcho_postgres_image: "docker.io/pgvector/pgvector:pg16"
+
+# ---------------------------------------------------------------------------
+# Container & network identifiers
+# ---------------------------------------------------------------------------
+honcho_container_name: honcho-api
+honcho_deriver_container_name: honcho-deriver
+honcho_postgres_container_name: honcho-postgres
+honcho_network_name: honcho-net
+
+# Named podman volumes
+honcho_postgres_volume: honcho_postgres_data
+
+# ---------------------------------------------------------------------------
+# Networking
+# ---------------------------------------------------------------------------
+# Honcho's FastAPI default port is 8000. Bind to 0.0.0.0 so Traefik on
+# lightning-lane can reach it; auth is enabled (JWT) so the open port is
+# not an open door.
+honcho_listen_address: "0.0.0.0"
+honcho_listen_port: 8000
+
+# Public-facing URL used for absolute links / OIDC callbacks.
+# Leave EMPTY to make Honcho host-agnostic — FastAPI emits relative URLs.
+# Set to a fully-qualified URL only if a specific feature requires it.
+honcho_web_url: ""
+
+# ---------------------------------------------------------------------------
+# PostgreSQL configuration
+# ---------------------------------------------------------------------------
+honcho_db_user: honcho
+honcho_db_name: honcho
+# honcho_db_password sourced from vault below
+
+# pgvector-enabled Postgres uses the same env vars as the stock image
+honcho_postgres_initdb_args: "--encoding=UTF8 --locale=C"
+
+# ---------------------------------------------------------------------------
+# LLM provider configuration
+# ---------------------------------------------------------------------------
+# Supported transports: openai, anthropic, gemini. We start with anthropic
+# and can flip to a local OpenAI-compatible endpoint later by changing
+# these knobs plus the API-key env-var name.
+#
+# Honcho has THREE distinct LLM subsystems, each with its own settings
+# class and env-var prefix:
+#
+#   DERIVER  — generates observations from messages (background worker)
+#   SUMMARY  — periodic session summaries
+#   DIALECTIC— answers theory-of-mind queries (the `peers/{peer}/chat` API)
+#
+# Honcho's dialectic subsystem uses PER-LEVEL config (minimal/low/medium/
+# high/max) under DIALECTIC_LEVELS__<level>__MODEL_CONFIG__*. Defaults
+# point at OpenAI, so any honcho.dev install without an OpenAI key will
+# fail dialectic queries with "Missing API key for openai model config".
+# We override all five levels to Anthropic below in the API template.
+honcho_llm_transport: "anthropic"
+honcho_deriver_model: "claude-sonnet-4-5"
+honcho_summary_model: "claude-sonnet-4-5"
+honcho_dialectic_model: "claude-sonnet-4-5"
+
+# Deriver tuning. Honcho batches representation tasks until the per-batch
+# token threshold is reached. For homelab use with one chatty operator,
+# that means short bursts of conversation can sit unprocessed forever.
+# DERIVER_FLUSH_ENABLED=true bypasses the batching threshold so each
+# message is processed promptly.
+#
+# honcho_deriver_enabled controls whether the deriver Quadlet is created.
+# honcho_deriver_autostart controls whether the service starts on boot.
+# Set autostart=false to create the service but leave it disabled — useful
+# after discovering the deriver burns tokens autonomously despite client-side
+# cadence=0 settings.
+honcho_deriver_enabled: true
+honcho_deriver_autostart: false
+honcho_deriver_workers: 1
+honcho_deriver_polling_seconds: "1.0"
+honcho_deriver_flush_enabled: true
+
+# ---------------------------------------------------------------------------
+# Embeddings
+# ---------------------------------------------------------------------------
+# Anthropic does not provide an embedding API. Honcho defaults to
+# OpenAI text-embedding-3-small, which fails without an OpenAI key.
+# Three options going forward:
+#   1. Set EMBED_MESSAGES=false — disables semantic search/vector recall
+#      but theory-of-mind derivation still works. Default for now.
+#   2. Provide an OpenAI API key purely for embeddings (cheap; embeddings
+#      are ~$0.02 per million tokens). Set honcho_embed_messages=true and
+#      add vault_honcho_openai_api_key.
+#   3. Stand up a local BGE/nomic embedding endpoint (e.g. on astro-orbiter
+#      once GPU is ready) and point Honcho at it via
+#      EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL.
+honcho_embed_messages: true
+
+# Embedding provider (transport must be openai-compatible). The default
+# OpenAI endpoint requires honcho_openai_api_key. To swap to a local
+# OpenAI-compatible embedder (e.g. Ollama), set honcho_embedding_base_url
+# to the upstream /v1 URL and the API key can be any non-empty string.
+honcho_embedding_transport: "openai"
+honcho_embedding_model: "text-embedding-3-small"
+honcho_embedding_base_url: ""
+
+# ---------------------------------------------------------------------------
+# Auth
+# ---------------------------------------------------------------------------
+# AUTH_USE_AUTH=true means clients must present a JWT signed with
+# AUTH_JWT_SECRET. The secret lives in vault.
+honcho_auth_enabled: true
+
+# ---------------------------------------------------------------------------
+# Vault inputs (defined in group_vars/all/vault, encrypted with ansible-vault)
+# ---------------------------------------------------------------------------
+# vault_honcho_database_password      - postgres role password
+# vault_honcho_jwt_secret             - 32+ byte random string for JWT signing
+# vault_honcho_anthropic_api_key      - Anthropic API key for Claude calls
+# vault_honcho_openai_api_key         - OpenAI API key, embeddings-only
+honcho_db_password: "{{ vault_honcho_database_password }}"
+honcho_jwt_secret: "{{ vault_honcho_jwt_secret }}"
+honcho_anthropic_api_key: "{{ vault_honcho_anthropic_api_key }}"
+honcho_openai_api_key: "{{ vault_honcho_openai_api_key }}"
+
+# ---------------------------------------------------------------------------
+# Health check
+# ---------------------------------------------------------------------------
+honcho_health_check_retries: 30
+honcho_health_check_delay: 2
+
+# ---------------------------------------------------------------------------
+# Quadlet location (rootful)
+# ---------------------------------------------------------------------------
+honcho_quadlet_dir: /etc/containers/systemd

ansible/roles/honcho/meta/main.yml

@@ -0,0 +1,27 @@
+---
+galaxy_info:
+  role_name: honcho
+  author: JARVIS
+  description: >-
+    Deploys Honcho (plastic-labs/honcho) — a memory + theory-of-mind layer
+    for stateful agents — with a pgvector-enabled PostgreSQL backing store,
+    via rootful Podman Quadlet on Ubuntu. Designed for the mk-labs
+    `lincoln` host. Fronted by Traefik at hall-of-presidents.local.mk-labs.cloud.
+  license: MIT
+  min_ansible_version: "2.14"
+  platforms:
+    - name: Ubuntu
+      versions:
+        - noble
+        - jammy
+  galaxy_tags:
+    - honcho
+    - memory
+    - llm
+    - podman
+    - quadlet
+    - homelab
+
+# linux-baseline is applied separately as a day0 playbook. We don't depend
+# on it here so this role stays composable.
+dependencies: []

ansible/roles/honcho/tasks/api.yml

@@ -0,0 +1,34 @@
+---
+# ============================================================================
+# Honcho API container (Quadlet)
+# ----------------------------------------------------------------------------
+# Runs `fastapi run src/main.py` via the image's default CMD. The
+# entrypoint.sh in the image runs the DB migration first; safe to do on
+# every restart (idempotent).
+# ============================================================================
+
+- name: Deploy Honcho API Quadlet
+  ansible.builtin.template:
+    src: honcho-api.container.j2
+    dest: "{{ honcho_quadlet_dir }}/{{ honcho_container_name }}.container"
+    owner: root
+    group: root
+    mode: "0644"
+  register: honcho_api_quadlet
+
+- name: Reload systemd to pick up Quadlet changes
+  ansible.builtin.systemd:
+    daemon_reload: true
+  when: honcho_api_quadlet.changed
+
+- name: Ensure Honcho API container is started and enabled
+  ansible.builtin.systemd:
+    name: "{{ honcho_container_name }}.service"
+    state: started
+    enabled: true
+
+- name: Restart Honcho API on Quadlet change
+  ansible.builtin.systemd:
+    name: "{{ honcho_container_name }}.service"
+    state: restarted
+  when: honcho_api_quadlet.changed

ansible/roles/honcho/tasks/deriver.yml

@@ -0,0 +1,36 @@
+---
+# ============================================================================
+# Honcho deriver worker container (Quadlet)
+# ----------------------------------------------------------------------------
+# Same image as the API container, but runs the deriver script instead of
+# the FastAPI server. Pulls jobs off the in-database queue and calls the
+# configured LLM provider for theory-of-mind derivations.
+# ============================================================================
+
+- name: Deploy Honcho deriver Quadlet
+  ansible.builtin.template:
+    src: honcho-deriver.container.j2
+    dest: "{{ honcho_quadlet_dir }}/{{ honcho_deriver_container_name }}.container"
+    owner: root
+    group: root
+    mode: "0644"
+  register: honcho_deriver_quadlet
+
+- name: Reload systemd to pick up Quadlet changes
+  ansible.builtin.systemd:
+    daemon_reload: true
+  when: honcho_deriver_quadlet.changed
+
+- name: Ensure Honcho deriver service exists but is disabled by default
+  ansible.builtin.systemd:
+    name: "{{ honcho_deriver_container_name }}.service"
+    state: "{{ 'started' if honcho_deriver_autostart | bool else 'stopped' }}"
+    enabled: "{{ honcho_deriver_autostart | bool }}"
+
+- name: Restart Honcho deriver on Quadlet change (only if autostart enabled)
+  ansible.builtin.systemd:
+    name: "{{ honcho_deriver_container_name }}.service"
+    state: restarted
+  when:
+    - honcho_deriver_quadlet.changed
+    - honcho_deriver_autostart | bool

ansible/roles/honcho/tasks/main.yml

@@ -0,0 +1,42 @@
+---
+# ============================================================================
+# honcho / main entrypoint
+# ----------------------------------------------------------------------------
+# Order matters:
+#   1. Podman + Quadlet support installed and ready
+#   2. Network created (containers reference it by name)
+#   3. Volumes created (postgres data)
+#   4. Postgres started (Honcho API + deriver depend on it being ready)
+#   5. Honcho API started
+#   6. Honcho deriver worker started
+#   7. Verify reachable on listen port
+# ============================================================================
+
+- name: Install Podman and dependencies
+  ansible.builtin.import_tasks: podman.yml
+  tags: [honcho, podman]
+
+- name: Ensure podman network exists
+  ansible.builtin.import_tasks: network.yml
+  tags: [honcho, network]
+
+- name: Ensure podman volumes exist
+  ansible.builtin.import_tasks: volumes.yml
+  tags: [honcho, volumes]
+
+- name: Deploy PostgreSQL (pgvector) container
+  ansible.builtin.import_tasks: postgres.yml
+  tags: [honcho, postgres]
+
+- name: Deploy Honcho API container
+  ansible.builtin.import_tasks: api.yml
+  tags: [honcho, api]
+
+- name: Deploy Honcho deriver worker
+  ansible.builtin.import_tasks: deriver.yml
+  when: honcho_deriver_enabled | bool
+  tags: [honcho, deriver]
+
+- name: Verify Honcho is reachable
+  ansible.builtin.import_tasks: verify.yml
+  tags: [honcho, verify]

ansible/roles/honcho/tasks/network.yml

@@ -0,0 +1,17 @@
+---
+# ============================================================================
+# Podman network — single user-defined network shared by api/deriver/postgres
+# ============================================================================
+
+- name: Check if honcho network exists
+  ansible.builtin.command:
+    cmd: "podman network exists {{ honcho_network_name }}"
+  register: honcho_network_check
+  failed_when: false
+  changed_when: false
+
+- name: Create honcho podman network
+  ansible.builtin.command:
+    cmd: "podman network create {{ honcho_network_name }}"
+  when: honcho_network_check.rc != 0
+  changed_when: true

ansible/roles/honcho/tasks/podman.yml

@@ -0,0 +1,24 @@
+---
+# ============================================================================
+# Podman + Quadlet prerequisites
+# ----------------------------------------------------------------------------
+# Mirrors the pattern from the semaphore role. Ubuntu 24.04's podman is new
+# enough that Quadlet ships out of the box (>= 4.4).
+# ============================================================================
+
+- name: Ensure Podman is installed
+  ansible.builtin.package:
+    name:
+      - podman
+      - podman-compose
+    state: present
+  become: true
+
+- name: Ensure Quadlet drop-in directory exists
+  ansible.builtin.file:
+    path: "{{ honcho_quadlet_dir }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+  become: true

ansible/roles/honcho/tasks/postgres.yml

@@ -0,0 +1,48 @@
+---
+# ============================================================================
+# pgvector PostgreSQL container (Quadlet)
+# ----------------------------------------------------------------------------
+# Honcho stores embeddings in pgvector, so we use the pgvector-enabled
+# image rather than stock postgres. Same env-var surface as the upstream
+# image; the pgvector extension is created by Honcho's migration script
+# on first start (scripts/provision_db.py).
+# ============================================================================
+
+- name: Deploy Honcho PostgreSQL Quadlet
+  ansible.builtin.template:
+    src: honcho-postgres.container.j2
+    dest: "{{ honcho_quadlet_dir }}/{{ honcho_postgres_container_name }}.container"
+    owner: root
+    group: root
+    mode: "0644"
+  register: postgres_quadlet
+
+- name: Reload systemd to pick up Quadlet changes
+  ansible.builtin.systemd:
+    daemon_reload: true
+  when: postgres_quadlet.changed
+
+- name: Ensure Honcho PostgreSQL container is started and enabled
+  ansible.builtin.systemd:
+    name: "{{ honcho_postgres_container_name }}.service"
+    state: started
+    enabled: true
+
+- name: Wait for PostgreSQL to accept connections
+  ansible.builtin.command:
+    cmd: >-
+      podman exec {{ honcho_postgres_container_name }}
+      pg_isready -U {{ honcho_db_user }} -d {{ honcho_db_name }}
+  register: pg_ready
+  until: pg_ready.rc == 0
+  retries: 30
+  delay: 2
+  changed_when: false
+
+# Quadlet does not auto-restart on .container changes — same pitfall as
+# semaphore role. Force restart only when the template was modified.
+- name: Restart Honcho PostgreSQL on Quadlet change
+  ansible.builtin.systemd:
+    name: "{{ honcho_postgres_container_name }}.service"
+    state: restarted
+  when: postgres_quadlet.changed

ansible/roles/honcho/tasks/verify.yml

@@ -0,0 +1,27 @@
+---
+# ============================================================================
+# Post-deploy verification
+# ----------------------------------------------------------------------------
+# Polls the local Honcho API port until it responds. Fails loudly if the
+# service doesn't come up — the operator should not get an "all green"
+# playbook result while Honcho is silently broken.
+# ============================================================================
+
+- name: Wait for Honcho HTTP endpoint
+  ansible.builtin.uri:
+    url: "http://127.0.0.1:{{ honcho_listen_port }}/docs"
+    status_code: [200, 307]
+    return_content: false
+  register: honcho_ping
+  until: honcho_ping.status in [200, 307]
+  retries: "{{ honcho_health_check_retries }}"
+  delay: "{{ honcho_health_check_delay }}"
+
+- name: Report Honcho status
+  ansible.builtin.debug:
+    msg: >-
+      Honcho API reachable on http://127.0.0.1:{{ honcho_listen_port }}/docs.
+      Traefik should now route hall-of-presidents.local.mk-labs.cloud to
+      this backend. Deriver enabled: {{ honcho_deriver_enabled }};
+      LLM transport: {{ honcho_llm_transport }};
+      deriver model: {{ honcho_deriver_model }}.

ansible/roles/honcho/tasks/volumes.yml

@@ -0,0 +1,17 @@
+---
+# ============================================================================
+# Named podman volumes
+# ----------------------------------------------------------------------------
+# Honcho itself is stateless; only postgres needs durable storage.
+# ============================================================================
+
+- name: Ensure named volumes exist
+  ansible.builtin.command:
+    cmd: "podman volume create {{ item }}"
+  register: volume_create
+  changed_when: "'already exists' not in (volume_create.stderr | default(''))"
+  failed_when:
+    - volume_create.rc != 0
+    - "'already exists' not in (volume_create.stderr | default(''))"
+  loop:
+    - "{{ honcho_postgres_volume }}"

ansible/roles/honcho/templates/honcho-api.container.j2

@@ -0,0 +1,85 @@
+# mk-labs Honcho API service (FastAPI on :8000)
+# Managed by Ansible - honcho role
+
+[Unit]
+Description=Honcho API
+After=network-online.target {{ honcho_postgres_container_name }}.service
+Wants=network-online.target
+Requires={{ honcho_postgres_container_name }}.service
+
+[Container]
+Image={{ honcho_image }}
+ContainerName={{ honcho_container_name }}
+Network={{ honcho_network_name }}
+PublishPort={{ honcho_listen_address }}:{{ honcho_listen_port }}:8000
+
+# Run the entrypoint that performs DB migration before starting FastAPI.
+Exec=/app/docker/entrypoint.sh
+
+# -- Database ---------------------------------------------------------------
+Environment=DB_CONNECTION_URI=postgresql+psycopg://{{ honcho_db_user }}:{{ honcho_db_password }}@{{ honcho_postgres_container_name }}:5432/{{ honcho_db_name }}
+
+# -- Auth (USE_AUTH + JWT secret) -------------------------------------------
+Environment=AUTH_USE_AUTH={{ honcho_auth_enabled | string | lower }}
+Environment=AUTH_JWT_SECRET={{ honcho_jwt_secret }}
+
+# -- LLM provider keys ------------------------------------------------------
+{% if honcho_llm_transport == 'anthropic' %}
+Environment=LLM_ANTHROPIC_API_KEY={{ honcho_anthropic_api_key }}
+{% endif %}
+
+# OpenAI key — used only by the embedding subsystem. Always present so
+# that conclusion vectorisation and any later semantic-search path works
+# regardless of which provider serves LLM completions.
+{% if honcho_openai_api_key | length > 0 %}
+Environment=LLM_OPENAI_API_KEY={{ honcho_openai_api_key }}
+Environment=OPENAI_API_KEY={{ honcho_openai_api_key }}
+{% endif %}
+
+# -- Deriver subsystem ------------------------------------------------------
+Environment=DERIVER_FLUSH_ENABLED={{ honcho_deriver_flush_enabled | string | lower }}
+Environment=DERIVER_MODEL_CONFIG__TRANSPORT={{ honcho_llm_transport }}
+Environment=DERIVER_MODEL_CONFIG__MODEL={{ honcho_deriver_model }}
+
+# -- Summary subsystem ------------------------------------------------------
+Environment=SUMMARY_MODEL_CONFIG__TRANSPORT={{ honcho_llm_transport }}
+Environment=SUMMARY_MODEL_CONFIG__MODEL={{ honcho_summary_model }}
+
+# -- Dialectic subsystem (per-level overrides; defaults are openai) ---------
+Environment=DIALECTIC_LEVELS__minimal__MODEL_CONFIG__TRANSPORT={{ honcho_llm_transport }}
+Environment=DIALECTIC_LEVELS__minimal__MODEL_CONFIG__MODEL={{ honcho_dialectic_model }}
+Environment=DIALECTIC_LEVELS__low__MODEL_CONFIG__TRANSPORT={{ honcho_llm_transport }}
+Environment=DIALECTIC_LEVELS__low__MODEL_CONFIG__MODEL={{ honcho_dialectic_model }}
+Environment=DIALECTIC_LEVELS__medium__MODEL_CONFIG__TRANSPORT={{ honcho_llm_transport }}
+Environment=DIALECTIC_LEVELS__medium__MODEL_CONFIG__MODEL={{ honcho_dialectic_model }}
+Environment=DIALECTIC_LEVELS__high__MODEL_CONFIG__TRANSPORT={{ honcho_llm_transport }}
+Environment=DIALECTIC_LEVELS__high__MODEL_CONFIG__MODEL={{ honcho_dialectic_model }}
+Environment=DIALECTIC_LEVELS__max__MODEL_CONFIG__TRANSPORT={{ honcho_llm_transport }}
+Environment=DIALECTIC_LEVELS__max__MODEL_CONFIG__MODEL={{ honcho_dialectic_model }}
+
+# -- Embeddings (text-embedding-3-small via OpenAI by default) --------------
+Environment=EMBED_MESSAGES={{ honcho_embed_messages | string | lower }}
+Environment=EMBEDDING__MODEL_CONFIG__TRANSPORT={{ honcho_embedding_transport }}
+Environment=EMBEDDING__MODEL_CONFIG__MODEL={{ honcho_embedding_model }}
+{% if honcho_embedding_base_url | length > 0 %}
+Environment=EMBEDDING__MODEL_CONFIG__BASE_URL={{ honcho_embedding_base_url }}
+{% endif %}
+
+# -- Vector store -----------------------------------------------------------
+Environment=VECTOR_STORE_TYPE=pgvector
+
+# -- Public-facing URL (empty -> relative URLs) -----------------------------
+{% if honcho_web_url | length > 0 %}
+Environment=HONCHO_PUBLIC_URL={{ honcho_web_url }}
+{% endif %}
+
+# Timezone matches host baseline
+Environment=TZ=America/Chicago
+Environment=LOG_LEVEL=INFO
+
+[Service]
+Restart=always
+TimeoutStartSec=180
+
+[Install]
+WantedBy=multi-user.target default.target