llm-inference-multimodel: role + day1 playbook (phase 0 discover approved)

Previous ctx-size=8192/parallel=4 gave 2048 tokens/slot, too small for
context compression inputs (observed live rejection at 3826 tokens).

Measured VRAM on astro-orbiter (RTX 3090 24GB): weights ~17GB resident,
~294KiB/token pool-wide for KV cache+buffers at prior sizing.

New: ctx-size=16384, parallel=2 -> 8192 tokens/slot (matches model's
native n_ctx_train max). Projected VRAM ~21.8GB, ~2.7GB headroom.

Applied directly via ansible-playbook (Semaphore currently broken --
fix tracked separately).

COUCHDB-ERLANGCOOKIE-FIX.md

@@ -0,0 +1,160 @@
+# CouchDB erlangCookie Fix - Implementation Guide
+
+## Summary
+
+**Problem**: CouchDB deployment fails because `erlangCookie` is missing from the ExternalSecret configuration.
+
+**Decision**: Externalize `erlangCookie` to 1Password (pragmatic approach)
+
+**Rationale**: 
+- ExternalSecret architecture requires ownership of the entire secret
+- Mixing externalized and chart-generated fields in the same secret is not supported
+- Single-node deployment makes erlangCookie rotation unnecessary
+- This is an acceptable deviation from the pure Harbor pattern given the architectural constraints
+
+## Implementation Steps
+
+### 1. Generate erlangCookie Value
+
+```bash
+openssl rand -hex 20
+```
+
+Example output: `f4e3c2b1a9d8e7f6c5b4a3d2e1f0a9b8c7d6e5f4`
+
+### 2. Add to 1Password
+
+- **Vault**: `mk-labs`
+- **Item**: `couchdb`
+- **Field Name**: `erlang-cookie`
+- **Field Type**: password (concealed)
+- **Value**: `<paste generated value from step 1>`
+
+### 3. Update ExternalSecret Configuration
+
+File: `cluster/applications/couchdb/externalsecret.yaml`
+
+```yaml
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: couchdb-credentials
+  namespace: couchdb
+  labels:
+    app.kubernetes.io/name: couchdb
+    app.kubernetes.io/part-of: mk-labs
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: onepassword-connect
+  target:
+    name: couchdb-admin
+    creationPolicy: Owner
+    template:
+      engineVersion: v2
+      data:
+        adminUsername: "admin"
+        adminPassword: "{{ .adminPassword }}"
+        cookieAuthSecret: "{{ .cookieAuthSecret }}"
+        erlangCookie: "{{ .erlangCookie }}"  # ← ADD THIS LINE
+  data:
+    - secretKey: adminPassword
+      remoteRef:
+        key: couchdb
+        property: admin-password
+    - secretKey: cookieAuthSecret
+      remoteRef:
+        key: couchdb
+        property: cookie-auth-secret
+    - secretKey: erlangCookie  # ← ADD THIS BLOCK
+      remoteRef:
+        key: couchdb
+        property: erlang-cookie
+```
+
+### 4. Update values.yaml Documentation (Optional)
+
+File: `cluster/applications/couchdb/values.yaml`
+
+Update the comment block at line 9-10:
+
+```yaml
+# Admin credentials managed via ExternalSecret
+# See externalsecret.yaml for 1Password integration
+#
+# NOTE: erlangCookie is externalized to 1Password for architectural
+# simplicity (ExternalSecret ownership model). In a pure Harbor pattern,
+# this would be chart-generated, but single-node deployment makes this
+# acceptable. The erlangCookie is treated as an immutable infrastructure
+# secret (generate once, never rotate).
+createAdminSecret: false
+extraSecretName: "couchdb-admin"
+```
+
+### 5. Commit and Push
+
+```bash
+cd ~/git/homelab
+git add cluster/applications/couchdb/externalsecret.yaml
+git add cluster/applications/couchdb/values.yaml  # if modified
+git commit -m "fix(couchdb): add erlangCookie to ExternalSecret from 1Password"
+git push origin main
+```
+
+### 6. Verify Deployment
+
+```bash
+# Watch ExternalSecret sync
+kubectl get externalsecret -n couchdb couchdb-credentials -w
+# Wait for: SecretSynced
+
+# Verify secret created with all four keys
+kubectl get secret -n couchdb couchdb-admin -o yaml
+# Should contain: adminUsername, adminPassword, cookieAuthSecret, erlangCookie
+
+# Watch ArgoCD sync
+kubectl get application -n argocd couchdb -w
+# Wait for: Healthy/Synced
+
+# Watch pod startup
+kubectl get pods -n couchdb -w
+# Wait for: Running
+
+# Test CouchDB access
+kubectl port-forward -n couchdb svc/couchdb-svc-couchdb 5984:5984 &
+curl http://localhost:5984/
+# Expected: {"couchdb":"Welcome","version":"3.5.1"}
+```
+
+## Why Not Follow Harbor Pattern Exactly?
+
+**Harbor Pattern**: Only user-facing credentials externalized, internal secrets chart-generated.
+
+**CouchDB Constraint**: ExternalSecret uses `creationPolicy: Owner`, which takes full ownership of the target secret. This prevents the Helm chart from adding auto-generated fields to the same secret.
+
+**Options Considered**:
+1. ✅ **Externalize erlangCookie** (SELECTED) - Works with current architecture
+2. ❌ Chart auto-generation - Conflicts with ExternalSecret ownership
+3. ❌ Dual-secret approach - Requires Helm chart customization
+4. ❌ Disable ExternalSecret - Loses 1Password integration for admin password
+
+**Decision**: Pragmatic approach wins. erlangCookie is treated as an infrastructure secret (generate once, never rotate), which is acceptable for a single-node deployment.
+
+## Secret Classification
+
+| Secret           | Type          | 1Password? | Rationale                          |
+|------------------|---------------|------------|------------------------------------|
+| adminUsername    | User-facing   | No*        | Static value, hardcoded in template |
+| adminPassword    | User-facing   | ✅ YES     | User login credential              |
+| cookieAuthSecret | Gray area     | ✅ YES     | Session security, periodic rotation |
+| erlangCookie     | Internal      | ✅ YES**   | Architectural constraint           |
+
+\* Hardcoded in ExternalSecret template (not fetched from 1Password)  
+\*\* Pragmatic deviation from Harbor pattern due to ExternalSecret architecture
+
+## References
+
+- Full analysis: `/home/hermes/couchdb-erlangcookie-analysis.txt`
+- Harbor pattern: `/home/hermes/harbor-simplification-complete.txt`
+- CouchDB Helm chart: `apache/couchdb` v4.6.3

ansible/ansible.cfg

@@ -54,7 +54,7 @@
 
 # (pathspec) Colon-separated paths in which Ansible will search for collections content. Collections must be in nested *subdirectories*, not directly in these directories. For example, if ``COLLECTIONS_PATHS`` includes ``'{{ ANSIBLE_HOME ~ "/collections" }}'``, and you want to add ``my.collection`` to that directory, it must be saved as ``'{{ ANSIBLE_HOME} ~ "/collections/ansible_collections/my/collection" }}'``.
 
-;collections_path=/Users/rblundon/.ansible/collections:/usr/share/ansible/collections
+collections_path=/opt/ansible-collections:/usr/share/ansible/collections
 
 # (boolean) A boolean to enable or disable scanning the sys.path for installed collections.
 ;collections_scan_sys_path=True
@@ -262,7 +262,7 @@ roles_path=./roles
 
 # (path) The vault password file to use. Equivalent to ``--vault-password-file`` or ``--vault-id``.
 # If executable, it will be run and the resulting stdout will be used as the password.
-;vault_password_file=
+vault_password_file=/home/hermes/.vault_pass.txt
 
 # (integer) Sets the default verbosity, equivalent to the number of ``-v`` passed in the command line.
 ;verbosity=0

ansible/group_vars/all/semaphore.yml

@@ -104,6 +104,9 @@ semaphore_config:
         # from the playbook's directory as I first assumed). Path is
         # therefore relative to repo root, not playbook dir.
         ANSIBLE_ROLES_PATH: "ansible/roles"
+        # Collections are installed by the semaphore role into a host-side
+        # directory bind-mounted into the container at this path.
+        ANSIBLE_COLLECTIONS_PATH: "/opt/ansible-collections"
 
   templates:
     - name: "day0_linux_baseline"

ansible/group_vars/all/vars

@@ -39,3 +39,9 @@ step_ca_principal_mappings:
       - ryan.blundon@protonmail.com
       - ryan.blundon
       - ryanblundon
+
+# Leviton My Leviton API
+leviton_email: "{{ vault_leviton_email }}"
+leviton_password: "{{ vault_leviton_password }}"
+
+jmri_vnc_password: "{{ vault_jmri_vnc_password }}"

ansible/group_vars/all/vault

@@ -1,436 +1,365 @@
 $ANSIBLE_VAULT;1.1;AES256
-65653162653536373535643935613236366335356333336264346637323164633739633965656664
+66393233316132396639356564316439343234383066633231646134313361666463656536323732
-3434656433323239366665356134386564643135663135360a653232636463656332343565313763
+6630616536646439613533363430306466306233643730350a343364633233333335643833326163
-61326635636333393730356131356466363063613734653565643766383431363166656532393934
+64393933613963313533623733316339396236363663343635346663323366663166363839663837
-3631373264353563650a616362633165666239323563623161656238633737316236303133313034
+3331363062653239380a326566623264623837326636383939346430666537613361333638366630
-66353461643535373638616364663365643961656635643566613336633466343739386234313532
+63353062393335316663633739313532366363623739653631366539323435336361353331386230
-38353066313763643266623739366261646633363962336239623830613265383335383763376534
+39366634643964336233353961316630616462663166316266613037623363346335373638656365
-62653865626662383330656530336534366136313965373764353133326139623132646364396337
+38353733396636386133373836346336383231663661346137373164386338623733393566373563
-33333366366434393037373361376538376334336265663162336239646237306332623039343032
+35653933343036633365643535303934326537356136666539316137363433643266346630386439
-37333832383762663534643638666234623062373734626335323537616334303638376164656634
+38613332646238366536333536343031356532656336613530663830613264346339353034323362
-64623962313064363166383930636136383935613237613434373733383836666532363265643739
+66613530626361323535653232313730373463373332313561616631393461353730653464343063
-37376333363736326334383864633734393263623536393239393564643131616132373230356633
+38616338363938346161616636316232313838616463326432353639613837343162646363343232
-61636165333537356632303034346431353139383065393832323236626331353236666263376666
+33323064363139376566343866626364373662393138353666646234373461666163363139313631
-38306631653862343064656135396638636363613139373062653735366565376165353330636162
+64343261326566363265323463663538343034306136326234386664333837333937333136653563
-38653530366263356437323433653437373362356166633235386330623333326264333961643434
+61333531353434633339383661636363363535316366353330313566323133616438373161303135
-30323130613431353064393138303465316630383462653062326264393035393735396134396665
+35353630613037316466353832333033393030636331386438393133366333653832393731366363
-36383963633938363662633035626538663366323461656136323934353731626264396364643337
+62373638303737393162303461646239653865653834613662666636373364633165383062643831
-63356337343161366237353135313966616630316362396362343962356564643365626664633165
+34386232376361323638353361666530366432356331353963303930326535663536373339333062
-62383765303436303239376365316632353463666664306164313933346162343833303963313835
+32396266373430343339636635366434313635313766363863336464633961666332353834626163
-62353565333231646462383664356230623165363333653335343438346233316534333132366634
+61653637316163636465343630353431313863653033643237356434313564366361373435376662
-31326138663433663132393761343335656636366237656337666331633062303363633938303536
+63643737353830663236643862613533623237373531646136383763303766336139303632666235
-31303238323932336330363064323963306132633961313634616232636163353439363666623230
+64623834323966363363663730626437323432623966663537346162656265363562643836633731
-65666632653730303134343032616331363932616532313365633038623135396463316666373637
+65663631633462663764393132326165346639353033633035636432613039336164303538396632
-34343736383731363239303330306561356162623962396564323239376661363761336434376264
+66396264393865306666643636353638613661313230313337383634663839363439656533333932
-34373037623331643666366165306161356365326261366566323233656538363061636637343164
+36633432306131396539386539633063653230363932376264323537396434353364643432653661
-39323130613530386661343839623361303862323763373864626536376562643766643263376665
+32643162363066636432336363323534316436613838646562313538326566666239633234646236
-64373334333436653061653962373437366437643034306633343830626230346532653565353135
+30346534636533623365326564613561363362333364363037646561656635623935653466613565
-33326339656266616237643362333062346564333563326463393535356465666166626463643132
+65646361313436356261643762313339333864356338386136306162386262636464393130303963
-36316263323761653333363632386261633432383431386435363362663734626634353761663035
+38646464316432326431326661343632396235626234366133353461623862316662326432356234
-32383464396632396361623531303937383366633131613861646364303662343761323530303139
+37626366383861373831633639616465663564643866356664623066386535646163336134356534
-39333866663164363431656233663064383633386261313063646435333666376465626532343636
+33366664616232353863626465626364313530353335306565336665663866303736323162393362
-62386166306635353061623365313738306237616430386461343262613039346261353662346462
+63626261653161663664363833313461653034326330653835393737616135646462366665383935
-65616261656534336631646434316336646337333661336530633834633037363834346539646466
+30363639306330636634386433646231363530633061336364313338653632323831393630383934
-33346466643238343133326339393264613763626532633938613765373262393236643533383661
+37316362326338313733646332336263386239626539383330353362616132333161613464313066
-33306233623130666161326365366164333434373463323662316432633263323761316338623065
+34663434326662326233363432306433363666356132383866346336336261636435366332666135
-66306332656438303938393430633563303761303239346266386664333837613336316539643736
+34616231613638363339356333616536643266636363643131653330396162306264303566396461
-31353234613664313061306433656165373737336166343930653837303532346163613564376661
+64363763376365356533636430643866333361363062376237653237663731663934306265646630
-61376237383562353639393565303332633366376632386532333736303738333362353334353366
+33393637656335643366383564373966343265393630333835303731316339373133633462383364
-36343166633533303834346265666536353633343564623062613030323934653863343062323136
+33383435383331303264313334393532373932333334343862326635346135613932356337373034
-34323635363238623336653062666636646263343731316537346635346235613862623530326633
+62316262326331313135376465343336373266663338396533666431616462613932663861646238
-30396338386233663335613661646132386230356631326331653962393336653937376264313931
+62653563623535633738383033326235383666646333653731316233376231623661306462303732
-65363465343265643034363963656262666131393632653561626330396166386566396363663538
+32643064373236613336396233323435393939386530323331336138353364663762356538316562
-66323935383166623038326437323730363266316530313938373261366363613731656638386634
+33376530623664623733386133333433303031373337313366386236376539613964316135343865
-31316261616630306131633936326635333632643731313730303438656563343133373732376537
+33363963366165333238356663663435386439336366646138313034343636653463323938633136
-38623330633539653034343364656261316336353466653663646333653636653266373363333732
+39363966376238306662303265643034306136663661393738633436393432303139313132616534
-61333461346131366233336162363033623636386634613731303032393738333537393131316539
+66323432313635386162333838323136623634653264643438303264636430633232323434666532
-65643238613432306436376432373364656530323331633762393566396531396466653463353832
+32616664663063653735316237643539633133356661333132323238376333356464313262653836
-34393662616435343137373364663839613464346564663631343837366530313061326561656664
+39303566316332663737323437633031353330333365383837636336643763313433313937396531
-33623163333864346661393634333830336133383337373833346132353531656236363661323338
+38363536343438663966663436613132663661613134383431633765383164373762343435316161
-64313663356432373137343461333539386665643130316463336134396436623730666165623939
+62303631646235343063383230343232383336356562303563373933346530393333316634316437
-33373832353932386238353866336130356530613766626235316332623132343735636335633437
+32316330306163396434663031393965663163666537353031613365353437666466333464626238
-37303766386434373930383663386565383732636635646133393637383963353134633433616431
+35313739646535356665323734393965303064306132626261363062363438383164346261393463
-33616464633431666361393931623631313731343863636234646130313136346639323662373236
+30643438623363323161323230306230386332363635386234666639623566643536626637616533
-62623830633838326432326339383533373430346635643739646639633739343639636665376333
+37396136643930633262333331656363376433333234343630306535313262306235663263663362
-30396238356161373734633562643466643036363239396538613465346238356366363730623965
+32653434363035613732363136303363393939323337613661333439393637646262383039386661
-32336533303331363336666238363863313066376230353932323466316432653264346334373931
+62326163323562333339323636363565623664396164383332633666386130613766393138346134
-61303066313831376233313261633336643265396139663037333031636666383738313763313366
+33343338393536316431353439353062663164643634396363353131303038353965393466383030
-38386138363738353265613363353339323930343934393036343234626162353037326638636236
+36656465383938353936346361393963356630666630373236626237303064303062383638373730
-66343634323136663134373737366433616632653737316534633232323237343566623361666131
+35633866646535313432353338623462323235346433653431313031363163393666626432363238
-34393331373835613934316235336635666633646465373866356566656261373066653937316339
+39623361316132626230633336636163623466313666346631656134343762656566353432353264
-37396366643564333130383237613933613736376161616231666139643030613338333133356336
+30353436356237653231363564626134633039363035313232616333336436393638396233626638
-37343362363433653638373335363130313362306236393231646637383736663236376537303533
+32663230396539323761313838313466376165646430346634383332346134653662393161363337
-31633230333933666131613764336532373633353132373839326562356438386236643537386231
+62646161343665383364306665333164666231386531626465373366623761643161656462303733
-35616662643431383661343937613738393963373865383465306532336137373730653832633930
+37653438616233353432626466623163316565353764323762613635333832343634323665356336
-37656661326434633730346364326661653937313561333437383064313933363231393164616464
+35353162326233333836396337356466636131383838313436626336663132346339623261366465
-31333664386134376264376339303838656431633966663263303465376161633866376161313531
+30623261303933396562353331636638376135663330643638643536346261626632626139386535
-62346266316533326639303661396164376666383864363262316630636561326366373363633063
+66653332366361336636666437643165656239613031303638333232303836383132616636633938
-33666263333132323232346235633837363138663539366131303633313662326664373332353664
+31343034643037623731643931316463303639656266323231313666356336333133323135363330
-39306235313732376164306164386534346666633037316334366431363063663137356464656234
+63373365303131353161303630633738353536393631353034666139383435303461316131646138
-35656164303933306661373932663831346531636530386139373837633263396632643235383361
+61333731356538366366613831303565613365633965323235366166313534653965366433656533
-65393939303338656537353130316361366139363933363031393735666336323931373632303663
+62666136313662366638356237343734336333313034396465346632336262306531633535643238
-63333137663265353962653861316365643239396562333933383964663730663230353331386638
+35333831366532386235316565303936616264373337356134643066396531383533353336303131
-62396435323162663036303334323936633436626136663537376562363430376239356136343261
+31393837623564386535323532653733393734393164373235396566333565356237356438313762
-36323062363935643562326161643065333437343331613135383932333835396565646631646437
+32623765326639386262393639376461326163333237313232386138643130643231626466643663
-39633035326662653334616366323736323032616264623166326563326361363263623736623739
+39643061393566353434333136366335393536376234366266376265333234643536633035653933
-62633762346135626666636463343132303433346330303631656634646537616537363764626564
+36316132663539306465343039323935356361373439346437386234386464623962643464643562
-30316539303964626635613338623261613430656432376265386636613566313732386561653564
+61336434613834336161633237383361303930313464613666313834356330343138633735386530
-66363538643463653233373433353965623765303038333931623036666435303330633136663966
+36616233323366323961653965613438346136373738366266316134356266623664313539636235
-31313964323439376637613530343635376166333765306466353337623633343539313330373437
+37313033373466383134346361646562366531333338386330653736626530396238356639303131
-64343930636465636265383261633664313762303533326234626461623563636466643039343336
+38363738396236386461316433316261326435646130383336316234363461393237623633633336
-62346462343639323862353036326335383735323337396465613737336661633465363330363161
+30386630376565646337383738663939663462623232316635346635653830306664653336343033
-38323561613935303334303261323032616332333231363762303831373961323033626663383637
+64316430396664393532313766326437636636626232613036666666656430323136356436333564
-35323134666636353766313330316231323938326532333030383638646230643735653663346634
+36303334303562393832336433343438396430373833623137363736386665343866313064353063
-38363261626663393139313031633032343038396433353563363531306232386539303734323463
+32303634393131326464656535633734386462646339663533666430336265653965333538633866
-63313938373735383631373665613333303530383335363262306230326665666535333564346163
+61623666643839653239373335633735373738363736313665323365613635313766656635613832
-39336132643464666538356461393330616264383632623037623634336530376164336336323336
+33616461643539636165383233636533626230343138663630323731626139393230383464313430
-61646165333835333961323439663864386562326131653564623861386336303934346263643036
+61333438393337316239376435313337313437333931623238616133666138363235386533633437
-32666632623537376537646634313839363038666336386238343531323664396461633030373334
+35356163363231656536353934643539643562343732626630383565623730626533313230656164
-35396463373339633931383636353062396562383763373939633336346463623733303039663733
+35333735666135343364663233626163363930383262363266303265303638396239636361366534
-31663834326661623064323132353431366362636466623332363532316435386333636562633562
+31333863333565356135613232393165353266343632633532343061663331633337343538376265
-36613465363936623462316136643664336261353938653362393938666266663330323163653338
+39316630613439356262396634316361356436336634396337353339616536356336653930613966
-63333732616330323438633038366465353865353965663333376466613834633166303761633361
+34656439653366363562636639346430623561303463356337363830373966366632303337663564
-37613863313230653235653034613137303231343961616330316664653333303436356561353562
+35663632313265323365636238303364366230353039353561616636633664643233343430336237
-37623839646536363036613964336230373537653130366330633137356661366561386566323165
+63373264643935616331616632633065366638363833306337633563653065363464343137623533
-38653633373538633762643935346661363961663265616433383832666635666331366635636637
+36373231363739373335346464623533393336613634333636613937366136326464336332346166
-32336532373164373135363533333539623739383330646431356363636430326433376364393265
+61376263623835646163353134643963663964373732313833346163323138633230393537636664
-35643561613033623231366362393737663964396266313232666362326436303331653764366436
+30366234303334656130336630346130656237306161376566336534653630616439323764373665
-30363135663234356363613065643763353836626133656334653138393562393032623631343130
+61383338326163336164353265326163646165623235626137623237306666333832306461613630
-30303936656636366465633163323061333863393532366563316233353466613566316563336539
+66373331356465346261643466323662393661623433383265376666623932343861323139383531
-33326137383263393266393166336361656538353963313366353434333864623238333435653836
+64633536373362643935633734366235396433333237306166646164363930613862613365303663
-39376138356461663262356665363835653638623031396539353132623737333736353937623533
+38343833336137353634313362666665306666393635663633353934363832343739616331386130
-65386333396666633365313262323739633461383464386663313163646632646335646433663834
+66336561633039326434313833303465366638303961626138333165623331386230616130626639
-34616663386661363135653765386534633339653232326133396332646136653230393431626637
+34613962366230333065633761333335613636363533656461626632343631666563383738623330
-30366433336633643837363434383439343562366164353235653833306138643431316537343931
+30333834346233653938633330663166616331376436356533366461336264643264336139343262
-34396535393432383163653231623837633961366437653434623164333333323563666330613866
+35393665656230663232366133393037643536366234343537326631623332373131323739363638
-36393561656135303731666638393737653565303836326264373137303264656334653163366433
+39653162646366316639313631393631666261623230313538613666393732626438393763646330
-35376362326133666336303732656561326164663832623732613830393063343231323363353366
+36346661313131313630343432616365666633353762623261613039623331396330623939626132
-66636233366436363565336231663964303532343962333732313832323762343638356231343461
+34626333386538326434356432623965666662663437646237373537326534653634346239653634
-65373062383564346266323362336666613735316161366136303032313039373464383532343932
+31373038303639333037613637393862356263323066666630313262366633313932396465633337
-30396530373030303933353264363330353132376164303062623538613632323932373230363838
+66653930303934616236323064613761353935613835356561313334323762633064306661346666
-32633766316533316665393033326161373361333166643436356430623262663032646262616633
+37653262343865386236343634316336386630393739626437333065323433613531393738313432
-32643436646264343366343862356339626561346561353463616534306139383133656433636331
+31376233353463373237653164386363633334366332356538343966663939656165323465333030
-32333735346163353434616530623235373037376664326266343933326439663965653865333766
+39656532363363333432626638626438396539336461326338353732376235316133616666316261
-31653163643134346331313331633561643336663130353936323439636331383134646665656630
+37353063343366376433653961333233306461303133376661303332386230346231383837396133
-63663866663938663632646664323936613434333731306264306633643335393566343564346537
+37323137343066383966343535633363643233663530613566313330336232366638396165373631
-35326132303632373731373532623662373861356534343536613731366666373439343563663737
+30626531363033313833303836366434613736396339643032663066333865306535323739666162
-64326339313533323936393362613564333065623731653930363264373061636665313135623062
+64626133616433653864376662623464343131303938303237316264393765303035663833376464
-63373565306535373763356664333465343666626439626666393734303962303762393066643139
+32663264383236303766323935306463643138396237373338653238633464616238306132633735
-62663334313736663132386232613061326636313664326462353361633362323464653038396662
+31626538653262326533326266336633623532623935383266373533363466313033393235663538
-64363862356439633564313065363361313161643962316166643936346539626339396230333633
+66653038646233303665343634383666343363383238326533366136363838303332323230316662
-39333132393135613264653363323233643032376266313630313735373966373966386239336635
+35383235646638653539633961663036663933306463626335356631646662636230356261363261
-34643739363935353463373434343430633438346136306337393061646139346230636531376537
+65633261353830373865636630353932323937666331353635373736376436333361613330366633
-36633861303264633662373365623261303635363836333163623864323338613739383134636237
+30663939356165393132636131663966373433623063356265353131306532643066306630656363
-62326561356535626564633038643230646431613866643239646563323035353434313837663863
+66636636353262633437663264613266613663656137386231306231646264363661613035343538
-66326530353265626361643738333266346339326532663166656432613631313137396463643864
+37353633643065643236376537336238663137623735613038623766393231643131653436333262
-64343339373131363066303437633436306462316166356131366134626331343635323837643136
+31626463646432613563393665346532386161366435396364663239386236616233356131323536
-39306461316230663134346563313662616532616239333735303061373138383363386232373062
+33633936623762666534633862363466353736386137636363633733623366346337613365636439
-61643838366266613637336433323062336434313836376434636433306135303734376661376361
+66663035313430386464623833646135333062313830396637323961386135363461326539623432
-38373438623331653165643837303733393436626166623762623531316164666163316661643761
+32653865623530313637393561343465636430373162333162646631643235653931333830326266
-39616539653831336261333335643834663533363233333732643431363836386633663539386638
+36376631316165343631326165623838306239623764363262376634663236393933343838376663
-39356336653438663061306230316464616533363036363434663335383530326432376632396338
+39663834306165313330393739363133396436376437643232346336386531356638343063376465
-38386130386532333861353362313838383964663861663962353039613565636339346162383331
+61613037623137306666383231376539656361326132396662613061376134376266633764336266
-32313337353430343130326466623666326263343438663463376134336131323735663261356263
+64336334313335643635303632666431383637306334376462643630646339396435313830313363
-34613638303262366666643036353766653039393739303738366330643033316632616232393032
+33383162316261663035393962306234613865613366353465373035656434366261383133653331
-36336161646434373332646663313834623064336437316666623663613861333838363731336564
+63643235616362663663343330303765363263366130393837613939323264373937333162636639
-63636633623664353432323836306564616239343731316335333234336465363232326430643231
+31643438666338646135663538343231643235646364623761653064633566656663383465626133
-30623063316130396531376237363363386465393838303463336439616333353338366436323634
+31373935646266303565303539376162623132316438623565316537306337636630313861623937
-66626463326136343533343263303033333766306632303333376136643137353831323135666430
+34313832636533623033616139373965303839356530353935643363613464356364343162336466
-61313663336363356531633866663366333433343433643664663665643335633035303034666334
+66376130653162666661313139613530306666633432346639656466653364376435636461626362
-62653233616136656462343930636534636432353237363538356365636635373865353532643764
+32653633303561346233643463373534653434323134353434373839373937626663336464303866
-32646431376165616235396133323162373330386664636336336266363961316462613263303131
+33363264623038313835396231373132396163363662626264346461333539326365326165323066
-33303635306136303366353331653239306130616464303835363437313566363436663032353061
+62333139383334333334353031616430323339623066363232313937323465356266323934313761
-32356639333061636536383238396665393232663562363662366563653831343037626535666432
+66623835653961303830383030643537393130653935313265333062393034336562633535323263
-33353261623936313563323564643134666232333839386162663331326461393239623135383530
+65616237373232336534393834653162363461336262653862666637326266663966356665363036
-31323861666631663933636338323239386336356339623738646430333136336635643562353263
+35383437326465663635303664643236633435303862633965346133376536316233386333313634
-64313931656435383134353063383665646538306661396163653434373031326134336235646461
+33643565326633386565653961646463383866646636303537643436623734393234633938333933
-39343734313965313365643135643166366133313465323366633038663333366535633532623966
+65366164633165623333623362393639656661326332306538663738356364373734316563653038
-36666134316165613830323933333339336663363964323764313430613130653236336664393863
+34646531616662386232613034366332656262343164333531353037363036646262623663666236
-64306437316235306565333238666164353738336539353064343161343834656262653666636663
+38613238666136363431623664633863636365396236666532383930336636353031396232656435
-36353063326262653466303534616530616139636166343137666439336335363539393439633436
+61346238643431653231623861373964383931336535363262373437353532393165316562386134
-36623963623839623834656431396162396562373330316438383739363637613366613163336138
+36363263666135646237383666373833373737396330616163376439663736663937666161313831
-35373730383165316330366332373939643839336634613934653538303866343738656534653263
+63663531656635663339306365656663636633343733636165386230376332616331313638386538
-33303361343237356430373266653062313037663230366531366363393937653439363732333036
+32386466323232363533613334333333346161376430373436373961316564343061326164306138
-31343365633862623038316536363031356336383461396562626236313038313239623665316439
+33616263666262323430303730626266396535626439623364376239346564323730323534323938
-30336137356438343362363536303234346637373833313231663333616263303233336161383130
+33346364393033353865393864326361643734353234613563393138363334383536396535393166
-34316265613736383830363639366139313537313661633736303634326237643330663735356561
+34623163616336653436393639313965353237633566313039303137326234383230323235363234
-65626136613831376432383363383731663763356432663264346561646535633966643462326232
+37626161356166356365366164363863636563316332393638616535376466343537373966643839
-31623930303262616535356633663938323635643462663731366433623961623435303062616263
+32613930643533336264626136626465303339376632323034386161663661376466616233633065
-63353836333235336362353039343431313262646137303364613838306339623930666466636132
+66313739346162363838346663623266383130383736656334323430623463666439386532643630
-32656434623738333565636164303535393734346462353436313030396264623332393932653861
+33666639613830386136363535363830333234653961663739343537306634616531616263623762
-36383533363030356664326332376662616139343065343932656262623334633334633462383635
+64666230373830636238353062666330623061613663376638343763626264363130313464383661
-63666439343130623936376636633635613335363264363136643366636535613938373866356336
+38326530333362616163363735323861376366333665623536383566653837306131623732373639
-35666632313565633630373039316262316339333035366332356138393261613836633437363136
+31316661353332633630326162663738636562336666326637353764323431613666303038373532
-33653033343536623264366466356561323063313031633464353963383661316131643166663566
+62343661336338306561356235396636343130633365303466613637633363613862663233633731
-36373937326333393164353234336433656336626134363236303032396531353862333535336466
+66623530353132666261316637303763363830623734346262333633646238613131346564303734
-65393865333565386538386665373166383235633739383934643334653763366161396565616166
+62336434353432326239333232383833633962313537626430663130393733623162626131656366
-61333666353666393230326134636661376237353937646236343034626135366535383631333035
+64333535623138326239336165666562376663663334323036323539653734333835386331653438
-61346231386233326564656636363066396636393966353539666231646464323636303038656562
+63353861666239396437346361306634613462386335376137333963333838616138633730393865
-31656238373331303031393034346630643263613365643232663861373336363662356265326164
+62353539376136316564666136646639363635663736636439393462633165646632623664383663
-61323866663033636534313365376466303538666463663539623231313632303262326638633539
+31613137306461616361323832393036323933626531363536336261356636303531633239333362
-62313239356365313932653735363662666266643765633138393037653662656566656130663133
+32663134363263383039646162643539663737333861386437326337616362343963373532346238
-61373531316261646234666361313766626139613762306563343561363961656565666163313532
+37346137363933623839373838353939386630303461346438666534616434333031373730393537
-35323837333737366138316164376537613834313135306461633735613362376233306134373537
+30313134643963623564356266656430613430626238613266316335336265613132616562626261
-65323339393039336230373237386434333639623762643133363339303265623065326438623531
+35386435313933626634616463616166646466363939313639646264346464363337656339323366
-65356632363237613937633132326231333938336639663438653861353761396434643136643238
+36366665363739356564363232313762323565323134616134666337336534353464373637373130
-62363162666366653831626531353236623532656437313134343633626666383437363761626331
+63613265366436313131356332316531633732356461383064383031613337343363646432373936
-35386431646564653233613632373435623665613335316164636562356264663033366639636439
+33633339386632653032663837346130623636356464326637303338376132623734333932396232
-63356231393362663736366431616161633065303537366538393935363033303636653836346132
+61333033386265356630316134383066343164613130666664643732643362666561346132656266
-30633036653836656433636631313863303132666531326563303266316566623934666361653932
+31623633333039633837383264363937623435643061393935393762346430396335373864633634
-35623265613635343435373361353635343261656233356535303037383433353731313539613463
+33336136353332663366313334353739303539633364663231636539333132303966383432376262
-64363334383666663639623735393934646366386437633638613034316366336337316561333731
+61356563323232613433653262623663336634626532653465306638316633663564633862666666
-38363339386561373834333037626565623335613563323366656365643032373834616339663838
+30366133616336326661626238653933383164336366333438626235636631336165386664343736
-64646534623432323363303032323666643636323536666239616238323932353165663535633465
+62663961346664656333306435323833366632346366356238653731653937626333653630623334
-64633934303565396533313165393438373231376664616237373562306130363531393833353365
+64326662346138386433333232643262333835326263343239353264373038613634356436396630
-63623033343762326437306536663734386365653131346235303837336236366665346234316163
+38323931643361663238623766323930666130356339363564366661663033303831363138343737
-32616539666634653839633739303837646563313364333864343231306663383530313963386434
+66633535326131396236653261303836613364306537633637323031663166316338323533323731
-34313965393731653963643336336134343764343065646436303739356530333761386465333333
+30326235323066396663613531653061643661336631613835626266626436386662353465383065
-35653363653361366462636162663134333431323866306235343764333035373534656138643933
+64396562343966303362636136616438353661626466636635323961613438646634336563636534
-30653962613339313833343532613939636332633236303733313135313932356136383439326635
+38343566396530643961356434643933636235643561353232643062303232323437666261363061
-31386565343337333662663762356361623835353830333865613437336532623936343365343231
+62636530396466303466653333633930376465366561376363316137323263333561343334383364
-33336137616266303835366662643737666333346433366663666462396531643463373562623237
+39313863643062643766396564363137386231373136346138396162376264653538303464633161
-38383537666639363839653538303266303965613762373561643433376664313966326561653764
+61663363623937356138356430666461623130323466623162653863393736326264393836336637
-66396566366265343962393633653765643563666634646637623161323538623961393039333036
+34663931646566333535666664653237643732316663323230383239393763376266356135326438
-33363530653935386331396533666432626634616463663763396663353133393461623866383061
+35656266353865623663373366373830613361373664346632363031356265313364623866643438
-64366132646132663263636131333031643736663031663336623065386266653638376639336334
+61363866353934636337616239633330623734666138396166313864333939663563636138653930
-36653932643831303931613532333036323766376235303433633564303062623336646166303533
+35653137363033326432373661613434313137623163356134613265393238346438306165313639
-63353063303033623032643333653733393332303466366538353062373666363438366432316131
+36666662393165353565633531663536613037623230373063316639663632643139353235303462
-36613335396531663238343139313061613363393463363035366132353439353433646237646265
+31393263656265656131613164363035343233626433656135353331613532363236616439363731
-37633964316466636337363331383565323866653036386435376530343661323565666539663230
+62666432356363323937666435326638323437346136366131613636653430306131623966356263
-38386232376436623064613232636637313931316535643661306536343661653135313335616534
+39303739306233303862636535633431363630393432613663633836396566653039383735303336
-34633761663066316430313361393063623033373636633133653536383764306433346235343761
+62623331623034636363636661653236386337326666656532343737336262336462613762326531
-37393764633561653632653565383937383436666662336633633261653861323533303732613738
+34303066303834386366636430343161653665343362363038396562626133636135656538306435
-64653739323636613639643465316437356163356238656139663034313738363532373861393333
+36393364333066346238396362663664643236373532336263656233386663323663623137343462
-64613038386535396565656538663163313161353831306237313064666566336534646233393137
+64386337333130316434663564613665666238623132343437656637653035373738313735366630
-37336265316361613430663961356361653636636235323439306537666363353361643664363631
+66383639616166393265616434623463326437313530326130376339313662303836636664366232
-63643631336134323032636463393639613636643130313136326538376461363937643436633561
+32366634633030333130316435616233396231663937343732313066373834326464623139363663
-39663433623663346462356437666230313937396132633834386661613436653839616136343234
+31323931626364303230666162316436653065366137663631376265383063316534343736373261
-35333437386663353863306166393634363734323636333137343938353461386366326239396236
+31613637316235386539343766323439653062633137663730343236343661346162653366656332
-65303939623437376535633463626166653935343033303066383064306332356230623566393062
+32663932313063383561636266373766633535656131386133386135663863396261306530326632
-34666566353238653730613561383434343332333732356334623836616334393464373965633334
+63653936626236316539613262386231616433393064323461626536363831666461316131383837
-37313564393463663862386535626338336235613132376530666635343832386231366533363562
+30646266646266393666396362326238613231303335336532303836363264323233343534636635
-32626162626434366261623431373938646237643462623337653466653538653732393634366639
+66313538643033343262373463363866346566353263303966323933383963363463393761383865
-62666231376239393066366336386166646631396632613631313663633435646337313465643231
+64663932343830643531643466303438343161396133666463353762393737613036646166333265
-63356566333764333238326165613832643161346366396635663032623133386132666333343337
+66376231613232666164663964636134653061633330383863373836306366393838393235656331
-63323637383035363833653162326231616461653062613765363165313638313234343266333334
+35613231306263373230326634623262326333356263353961633836396531633431383163633361
-62383362343232363431613364383930376338303839633261656236636264353335333938333731
+30356534666466653734333437383964346564346165326664633738653338313263633837316531
-63616666626437643432393438376661616262643939363239353836323036303139666338336261
+36613034323433643839333264323864613033313137663131623265643364333664646235666232
-31306437353536623561333531346630643761336235373230373135326361393739613934653361
+33393039313666323266643362323337316465306564303230646561303434666630616137633831
-38346532313364633638306665306563646433353038346238313530363163396166353535343537
+34346439616634343337306636643733316464376631616266376437636439396337306637333432
-34343330373063383566333335386162326339383835373665363464616139383331646436343365
+39306333363035393436316434656436353738303861633933376531383862316466373736323639
-30386564323733393461653662323334656637363566343737356239306462303762383332656239
+35336137373866336631386436646231653366366435363932376434303063613961353261343661
-30643439343161636434656338636533396364643632383030646639356534343963666466613566
+33383638393165336438376662306431333837356435626137356130323836396335636166306662
-39373833633265623563396363653830393661633636646265643362633731626462636531313362
+36386161353739353637353861306666383966323339303262616239633930373633323937356632
-35366331383635643564316461633431646565393863373635323834363934336335653831613330
+65383032613031666665623631613430666662656336663931646533636230303261646530623765
-39646262633636343832626362393730643163616565363765346266316636343065393630333866
+64643939326435643539373564336531623236653731636636346361363064333963376566616530
-62333931336161636230326132393739616665646462653666346532356265643235333131343635
+62326639663632666634326233363635383830643163373938646165656163643864336436373466
-31326638626162356333356261623531646464633139666539373261376230336661313937363036
+36353832306632386230373832333234643638313238626333303963383962343265366137656136
-32313464316364623838343231346538303237393137363233366234373031346238303932383461
+37333261343161633562346638323632616566646162633133663466346535656463393932386135
-66613537346136313262313337636565383639383863346535393361363930616535626264376565
+62653332313066363965386335356430326539316366633537356364666230326237306236393563
-30303236663065363161663836356130363130363339643335653366623065373566353136353933
+35323363633936353034323232353366373566666332323737653237323135646665626139393436
-39363761636130646537393166623335353431616462343364383535393661313738343265313138
+65333762653536656161386532363765336538653763666236343166653933626666633130393033
-38323066333239616137623233323664616232323262666530373064366138316431643430643462
+31643531646633623663313237353333313136663863663430306131316165663765653732663164
-32653262386565396335646431633534663031613536656561643434366464353335313239393464
+36326531626365326330643064336230313466343731376437316563303339336333326636633066
-36333866343063656533623432306231343065343530343638653739343731336131626532376566
+61386135626430616661313236623030316362373338643233326365646531633265626238383830
-37653833313233386538666464386232353466303035316232646339653533663831323261636330
+65346132643537366537626132666165616138656139626132396639376230333262643766386363
-32646166366633656631393762626236356633393130386165643264663437666433336138353938
+39383034663034326165643636613237623234613666333532383733623462303331326238636461
-37653866646364333134313366316366636132623764366562623932383239643265646461646434
+34383139383466356139333934353837613964326538336463643832623062633034613762363061
-63656233316639616134666635373434323030643136626238646637386634333439616231336136
+61346361656363366136353336326433326266336564316366393565626262303637316564356566
-38326431363062326232626238613334356633353362653138643237646339333762616330383934
+39376661383763373436306238393666653561306538333638306233356139346634653363346164
-63613235376530373834333834316539343330313862666262633439333062633261646138626434
+31303835383062376638626266303237323832623735653066353936376339633637333562333561
-33616162316632366537393939343163333530393939613936323164663034326536666133303365
+36646462653834316131323166366661386161646538346464386232306239363030366363633663
-63643337343037363363663235316330316131353064383933323131643036623862303835336633
+65306439616339626635326531636435356134376561303235393337373564373937623636643432
-39376134313264333834306330366136643434386566393466313835396466306363663565303065
+61393535643038626562366331353831663338333838383066323632383633346564396566653330
-33393431636163376365396139346162346239343838616561373162643931623365336436626636
+34386563393832313537623061666466366661333934613766366165366330353835323637643635
-39613030356162363931316166376231303033306366616162366239396333323838363465353261
+38613363396663356564646132613536653033616337386566623662333832383938303138316662
-64303561386563383834386166323739626237376164326533616533343338316430366365333966
+34303339323166383863363831636233323335313565393933636435396666313337663037323432
-31343233666365663830323030336161623466633261303637373537336333663262383238393833
+63333139333165646262666339343736383966346133356138326437386334626461636530336432
-33383033343630663237653338636436346565356238313434346666336436653835326133313038
+61356631366163366561303636333230643732316261376365386463333565623533663966336365
-65353530333839373136326132326439306431333334633933663065316531616263633533346431
+39333365393766306536366231366435363030353263393534653534373064636361333532323735
-62383862653162613532383136363830313964366161306166343934363865316533643736373431
+61313163323831653362356333386638343566356261353534303738613730373632363534666337
-61323135343030346336356232326138376564343131636263336332333239303733323338663830
+62313339613138646361356431616236613435393233343732626263653332663265393934616134
-32373639343063353261623831623237663133646535636338353333653237653539656438666436
+35313165613766643938393839373261633439396661623961353934373130623865353639633038
-63636537336636313762336531346531313637653834386535313666306636303338306465303764
+34353631346433663131653965326337663561613330323562666336656237633163356562323931
-63366133396263636164386339343131653361363831363932303734333634343734636163383539
+62393833663233333538383063303937386365306135343962623333663435663431396438666362
-33613430353236616132636431656636633561316161623661373663396462633930343865336236
+61386266353838653532393233353939363738306634666537313761313835333864633764666262
-32373534666135303730303663333337643033346231646661393137643161383833626335336561
+30333032623766633334383031636633636539336237613235376466356430663938653565626235
-66366433303436313132636230313531316261613564323963626438623530633634363265613939
+64373537656566306136633630366130363630633462656330623633393735386630343437336436
-37666636313632353831343238646237666464333039306630343531626339323562346162663130
+65343635366531646130616534623136636666323139326462306533653532643962656530336633
-65633133366564306230666436663630316434616634656661306461353866396662376331666630
+35616537303932343539336638333730663639396330653761346136363431346536666138336462
-30323166386535633361316633303830623031393531343532633930616439326465613631613838
+66396565613166623934316532383835316137303134363466306163356233356530323231666464
-38326630366534363036666433333237363636643366333161343336363936323730313339343132
+30353933306530323734306564626234343864373964333264353366326265316333343330356532
-35356231666633386537366236303136663261336532386338316565343239333261613838623861
+38363837646635633461653562303264353633343461633339376665616331613733666663353130
-31366531343232653431343436343766386331303133363534616536633337656433326265656161
+65396363613731366234326466323738663563646166653237613364323734616465643764633537
-66356235616135363634313835666134346232356663326361396537303339373334323861383161
+35373865353532383566363632366564353536643739663761303565333138383638653665663664
-33393438313061613465666536666636643166613463363165393338643066613231636138343537
+65643366316461613630366437623736353739356538336237613431306363663234373265623962
-64333731376139353934316631393561356163366237386337393061343235633131613962383731
+34653565373335653563356135313835643266356261623037336536613733323733363933376538
-66376261633832323864356564326532356363343762313563393331393933316336373434333237
+36626134396563623733656534363331626262643339633932373035626134343531623634666463
-35396531303330396363333163373361643837383766333730303031613665313237383532666264
+61623036313334616639633930393562663631653565656136666537393731333430663062643362
-65613237623136336239653566646366376538356362643062386436363466616331336665616432
+63633663396562343965313261373965356163393538666466303661363531393266316462626166
-31383365363630663530643764633335336332386332663661393733343038626265383432306630
+38616536653665366462383064373766396438616665346666376232653031323566313164383164
-38653861356636313634326266633637653764396233313463393964653739306234666662336432
+36643231646439663637333165376439333432383532316661333766363136636236326338386537
-33393139343864346233653763343762616230383131353166396265653031326232656333626264
+33306130353634346136356234363438383865313136393839663066333935623565333730613538
-63616339663564363665626338316661623230646534373231313662343736316462663763333364
+63323831663866363831383930303434333936646564316435303931396362303534386335343330
-30333137386161336664386536366538313934313039333832323763313264373861666239643466
+65383635346662626363626365666166636361633365643735303762393832316436646139303835
-33383334633736333634306262316265323634386635313764376532646364356565396538656137
+63633733656361643233323332613632653837663262306661626438316262653931333061336366
-63316232643832383865303934636364313036666237636632393137373933656263646534623230
+35353939353835333361623261613738383734656132613139393264393038373765343131333330
-65616231323035643531353430326630353761633234636234343834656664323161383335343235
+37663962313566366463623437323965326365623437363038633661313461383634626661666236
-62616533366465303961306134386265303933616236346332623536393236633366333634316264
+32633335323861643037383261393164393933353531636134323765353962633732396230636331
-31396361343833623131306266336238343866303361303336346566656639616134383063396235
+32613865373739366530303538313566346434633933393330346637346136373036306666336164
-31626633333665373934333961646263663730653331326137383031383736646633356536333431
+35373732346334353432616561623031663331346431383235306537386466623339356366663335
-39303632383030613465656236333763393737353865386235333830613665653363393536646630
+36333733626433653465336431666530626665373564336339626163633131353330656437643638
-65353733613536653963393831636530636231366266343262376631393062623163313031623230
+31616563616665343635356231633135663665326131636664373338323736393364353636613762
-62326561653165306339633439376461346330393231313366666339346536313765363163616630
+66636135633766323866376235633535613735613465303239343036663438333331626431623435
-35373564366563656161376336323031346339313734633139383563393966373637616266666265
+61323537386434323638666537643236623632626430666263376534643336613635663762643736
-34623330313265373935396130643231353131313163393333396337396666636439633035326635
+30323237353265613062373265643562373637383337326264653639306263373865333262376665
-32663762333537663839346531663132613636366639643364373333343262326264366136363331
+62646331373931373762303461366163393839633135393964313937616437323865653735383630
-61613833643639656632353931663830613634383334653036323462323262386265313637656535
+32653936336534666565373437666130396265363561333635316461663766346336623865376133
-36333834306438633532616335663562383730306337656336663035643136386261326130336363
+35373665303433326265623531613038636166643130616637653165376263643634376439613765
-32623330333764616565333162383234656463363432313039666265653563323232613930303765
+35363031373630333966656466616235616337306335363132386335613462363664653634633864
-30623834666665636332303862663832303537656664643362643636616265613739356664646462
+61303635653932353730663666386263633662633736313461643932386161313762663761313336
-61326466303266626166333730343330396439356663376362666536623539353666636230373533
+62653665313033656537643936373465633932626166366430643763313030393838393039323230
-35363461326263366137313037323166396133646430616664343165623533393963623535626237
+62633334383938343433306262393536653930653030393033306661666264313630643564333166
-39633362393636373131626364636437343361303435396138363735326235383237633964336138
+35383237633932656331653030363434313534613637373465663264643061303538653666653861
-31643935646431386537643733353862666138333238323461623638356535316365653530376464
+35373936643037333866636131373338363062663035323531626431633362663364396365353139
-33343465653134613536343563376564663133336532343330666131346663363434623238303862
+66383737666437353764333231303662393630643933376161366430376530613365363830373534
-31306331633238623836346466616230356532316232323734356638343532643032633937653061
+38376633333936626430393163323830346166643537326430616236393733653761363235356363
-37393265333233626563353963363637653338393964363832663734303566356633623733393164
+35333131663032383861336262653936376565646662313965303265623763613330653461333835
-64383536343862393134336638353733373262396563303266346535346266306238616531336162
+63613563323135633438383931343731656333303362316533376339376636623037376431336366
-35363638333431396163303530353461363338656363336366376430316464353131623661346366
+61393236383364356162633062666265653534326363363862666539623761623065386537616563
-35313930353538343830333337356163613765356361343663613933366464353935336361636436
+62666561316437303763376635346536666437373361386666643139643737663333323933613661
-65356465346535613231363932303339646162303238376236323461303062313336656366306563
+38326566663932333930616435626133616531306461356466326437623235613233393434626563
-31353464333539343839353764393665393235653537313939373563303436626635663766303336
+34373966633834373430386132353163366465626262353863353335323830393266393562393133
-35333838323734386537623334366235653534623664396664656264633735353634663332353364
+65383632653438646435343333386261653066613663623232373564666465613136353039313036
-65306132346464363966313963346633353538633936363164393566376630376365316139376132
+61646462626433646330396664363938376530376438646262343231393262383733636233636333
-38323039653465346462666439656134393964623563393664643634313638323832356164323863
+35633862336566636439653464613564333162613836343636316334316665383164353131373431
-31366535336238663236666162336166313933376531643137343665653765623961333464633332
+63623030306564346562346237333934616134346536303365396533626262333937396432393830
-30306265306364643738386561303833326363303836333032393462643764663664613536363835
+38313763393463646437666137353835373735646365373934363936346564326362376565353133
-30663937333038663435326637663636383165363632326337653932336130333932333861366235
+36653362333432326133393837316331666663663263396461363239306239363733633137396633
-66313566613033383631376132333634653639336666386164613934353230613239383239313038
+38393865613431653337313665313762653635656531353465623436343132303064303564393066
-30653162643233636638393036353032353632316166333938313966626133643237376461313065
+63616639353962366666616261393766643364333634346630616436376565313236316539633537
-34323762376133393535613864636461616261343031393266666165316236323231323534376465
+66653239636561393433383639646462616433653166613130373134376535633937353366383230
-63386363343933316139356163363132343666366132313038643938653865663934383335336364
+61613335663434333835653236343633633038346335333861356637353965396632393833646635
-31646563346339323633643366653861323030636138663861636434306238623738636331393538
+36653034356234663831333764303338663464316362646339376338393236336161616263363538
-66333765626132346533313261646434633236633432333430613266333161643566326434363633
+63356631613239326161343031643936623366643432663732346438333265666535623664396333
-62373033356337393232353566666263353539326564333766303335666135343461386530626562
+62303239363339626566613439396234303536333333653433393666383635643235376165666234
-63633137623361366362616432613237353231616465656232373835376463386132663331633737
+30336236393962666335353233666463346530323531316438373130303933383465316638646461
-63306136326264643365356634336233303163663135366531643362303666653630363962666637
+62643066376666363236333231386237376466633932313836323163363061313333633434663763
-61626539613333306263323564373435613961633439633261373530363137376132326462633035
+64323365353336333736363436376232653436633739613437343538633632356665656364616637
-30353838336433313135623530643663643232653364316263356164633661383937393238363262
+62313666346436656564663335636635393632353430666236313863613464626434323939383538
-34393666663361613735623435626238646266376533333963653630663462653733373130393338
+65386265343434303632313739353239323565333734656566356164643430613538333234383566
-32626361366632633630323363376238343534626230373835353036333065396435653964326534
+37646431316363316139646435333732313339623666613738663039613239613738393565333330
-33356466643839626162316438356437623732656664613434613165643666656230656434633638
+33376432373933656435303737653762666464363865633831373330393435633332636261336139
-61316134333631326365313532313335633634356466663834666531653365333333616137643835
+34336236373535636165353262323966363164633135613534353661316364616637663465363864
-30666664353134313236653337366466623966366131313962346564663432633535363938333261
+39306163346365643339643937396165666366663339336438373031613937636464383531613962
-37353464383832373366613531666161323632616236613631653433643562326134623838393135
+34373433663533626665656364623634373335313033646165303764396563356235343033383138
-62323533373336663434386166333862366639626562343830396533646138636261326161393339
+62326361353630393938643764616636313461633734646661386536356235656665393864386465
-33613961353431653537373931396531623061653163376230663338376636613136353433646231
+37666262346561656436343036646330363664306135333464663265306165353039396665336664
-33633236306238336432323134393463653161303662336266313562316337343430373532663930
+62346631366330653762646538323565613864383534636532633033643533323736373931643130
-39613133646432343238393762386266303531663032306130653532376262353238636231393539
+32343435333333613734626234363132373734353035326232366264336161383631353133663230
-36376332633563633661353835386364633461363437333038363865356237643739323133653235
+32636533333866343763336439373336356237303636376334333433376630353338333261333037
-66373931336330626663383136363265363133393239393131623465383433363336333237653131
+31666537363666653238383939663464346662636133346561326335346163363061393830616237
-64656662316665356665303765343932363733666362353031656234373363663364666537663030
+36643430626534653331653665316535303139343763663965363164636238366533303038653935
-34646233386630633438313166396663383732366233373139343431373463346133316663643036
+32356432316633373137316237663331336463363431393033323635646564366639346230353363
-31633536663266383132393032313563653035303034336161356139396636353365636163383031
+38346663363039363962323137383366303862356530353238656563306131643236626536656334
-30666637363933643262393065633463313836653530663232303736306537636533343431333966
+38303462306562366532346163323061393437353063326539393466616439346564383036303235
-32386330356564613932306466353931393839316562353362303765663263383738363765333136
+35633731393661323962633631373061303930323638326565636162316436646337383266626561
-36383036376130306133393166333734316231376165393832383735623838353466353664353834
+32326430363031396530396238353862333133363731623736376239626561626165663337373261
-39326136656239626434313930306435333533623533643236393437313038393235386364323766
+39353461343461643238646635633562653865323336366634613264616662323232653861663038
-62323537666333613066356236313361376138656232343430363438346261646165623565313435
+64396330626633303031333334343335393039623135353266383561313231643433393963326637
-65326532623761343239666664636637613034623736386665303634613737613964333630613533
+39313530636361373831306234383166346266656261663830636631333564356536323565336266
-31653731353533613866663166653237663162623236353838646465373734613466616362363833
+38306561376366626236306633613564386166616630613032633163613837313462343662653261
-39346335373539376439643937363233373362616334323231623232353061343830383336666466
+63353437663436303634633336636532646439636465663362346138313665336334313039613631
-31303663643061653866326632333336626462623835366564356231316263326130393138306330
+65326135383831613531323265353831313562346161663265366434623236636635333038366536
-66333532336164633931306136373531666131356130313262313038626138653864633734636132
+33646631663662393331323162343438626666366636613438383665633136326439376166373462
-61353638333133383035333937626333376564316465333539653138323739356265613664653536
+33363864613136643461663436396362643066633437376631623031613366656238396165313832
-35656136343036363532613335336230373364336432343731326132666662636664356364303335
+31313131653263666334393664343239306235373862313339373563643137393633343663613936
-66613736633231393534626539333536626565663231396536363062353037346563643934313236
+61356564636238363136623031336638333566633766636362303938653531306131396665303033
-32363138386430633761383065343834376631643539626330313638333534333734333239383632
+63353362636463636236643464343562383161343432383766396330623764393837613435396162
-39383833656134393165666533353739303834633330653936643637316639663033363865653631
+31303162356437663932663964663239623764666366663061313535346438373334636263653531
-36313735613962393264623938336432336538393937316634626165383934356562353035316533
+34643133356638653031373036343162653135663734623035353033633561366266623566383233
-33353735653133636439393763356430303863326235623932623464356636666265653065383736
+36356434643538643430383532393762333535636639353361353763333363313131646264336332
-31613666303065653035303131646364386636393035663865336164376536323462666239356333
+34373331343930633962623963666365306132356334646636626461316236343839383266363635
-33373562383863323635613634643165336465613734613034366236633835363733306662613462
+65623434336239313330343437646333353362303232346638623161616133636636626236643465
-30326565346365633131363331643165346466366666633231643538303430376530393139393063
+36303636363965363765656533386534633839346363363738386532386531326538643134363132
-32666664313162303065616261326261353130623564343761633861373034646161643163373533
+66613235373362633166343565323766306335336365333439323764623964393263623236623832
-36386665626530313030343032376531656333356332666437383530613834356336386465353539
+34346132383136303038363764333039626234616132386464666633663536656230666133363533
-66623062646236333465316137316564363938303966613561643830353332333537373736626438
+38306665643361666539636666316432623430623939663636343164386438313765633031313534
-64653239613063646430386631313435306462303566333734663462626530356530663235303166
+65393633323837326166343936326263343833646331326464376138633461613532303135393036
-31633366613264316137303334613366366537646261303962646364643238383062643732386436
+65353830373065393038343039323937303634346665393135383639303162396565646232663736
-39643430613562663136633130666563396538306633633034323064386136303034306531323564
+39376434646634353330383933303164653431373433346335666131386165343035303964626665
-63646633303035353831653438366635646562613639313230396462373161363030386264623033
+35383035653631346638326637326235393833623264323030373238646335346332353362393230
-62343031616135346238303337313931643461386363396663333231626661613938626366653837
+61616664613562383639306564376661306665396138613066326631616531623132633966633832
-33303265656561313332353463363534666232363561353532643532386133306332333930373161
+65363637376264336635633132633332373634383864626564623966356464373864393832323738
-66626135303036633566303961646461383236663737643265346566393963343634623164633966
+37616338646633326636323461376137663632376262363738303336616463326238333465343533
-61313031333863383237313633663633333866366230313936396334383361393338656465346437
+31316132386530326539
-65613763643061386463386431386539656635393435633032343030633738373936613833623938
-39316239363835633339343161623237656230633763313031643663623466323764663366376165
-36366530363535376636383561343161643037626639323830396665303238663534633531613735
-39316639633730376533643932633432353835353439646666653766666338646662643162373263
-65356563303235633963633232393736636537346637376462626637303535383063373134613732
-32663763663364633463633738343535316436303365343665356133313836616164343434613133
-34626434643531343461346332636539636463313539393830613434333933643632313737356564
-39386137383165616139626363613732626336366461663339346134656530396464323533313265
-32346535356466383135636638616166313663613565353765346264376535653135356638663538
-30623834393935636562656639643737373462643137363063663737306361336339653334633665
-65316237633436303761393766393234326538653633363936303534646566373338623935666538
-38313761643437313565663762613838636163633066313630353631353934396338353665653330
-35393661356630313436373761383462396434643535316364383338343433613061353637333563
-38376562373462316334313632343965326235323231653030666666616163346438626437386637
-65656163653935656530366239663237633937336166613132356333623964666630386331383331
-32303166386431643739303363646633376331623166313135653265656636663236363936336462
-63303135613539663233366362323565396137623264353431373836356233666332633731666364
-62376565633037373832393366306431613863353137306535393664313866616235633638306365
-32633761386462386339343066316263663438623330373733333634363736623637636661636331
-38353835333962633537633864616132643563616437336334326633323636393833363666353261
-36363732383565623233343439666430303961306263363032383238653265323637313430613133
-63633338323536386139613233343636326564356564353065386664646230666561336338366436
-35353162633864633764373535366634303738346236633362626165393632383762343036313930
-62363864626338393736373938343264356235393763653432306330363363313762333137303362
-35393061356165666230343135663438303834363533323064626532663662323063386333623166
-39353037363363373166316238393565353266353665666465333134303234373263663135346364
-36646463343032326339663130393963663861323130356365396365643265313833353234323330
-34336661363733613362343739366636336265356464613033343132353031663965656634613831
-33656466376132656465383664363231653235623036346634666166623532323635313130303836
-36393064356637623139653333633164666332303539623863383538386262363064356130353337
-62393063633637383965346139626265666463356362633163363537303166336264346332636363
-39613735626132366137356365303331633037623132623637383164636565653963626632626338
-65383663633831613666396534333061643966313366303937333261303135353762656331336638
-35643539326161646433633862623762626539396137303863616139383832613639653033653933
-39373734393066613132323739383036313830346239306132656464646462626661633931623932
-36626163353563633764323466303966636161646532316134383034393733653363363933353738
-34383563666166323865656430346661633663363635623566336632353633303838303436616266
-34326131623331616533306465373365396437303764383231356631313335393364643664663864
-33313033303463376139376466643434383461643264316534306166303163373661643962343030
-36653665353232363630363437376266306531633934626230653338383664343638313334306636
-38666162623761366237383236346564333333633162623832656462616662313031316166383465
-31663362336663353564636335393738333565356432306139363661656663313234396664623832
-38393630666338663031323464383535303539643931393732616666366661316163626339643437
-34316532613063353264303462366534613035653834356562646637633137643839653237353937
-34326230396137383531393962346137643035333834376537363865643366623932303662303839
-64373431306464363762356563663038343737356165313832613965653065326532356133646137
-30326232643334656363343533383163643130343836366430333836396463386666623465366336
-61343236303336623863303630336437613932353832623866633661653566623431653938306238
-37326630616131646634383539353234323766363633333030613937616632333432366565343537
-35613237343838333430656336313232653530373863663031386433356337643334303363666532
-65633033333463366636616138646632636534333963643864383033343463666338373630313232
-63326365666666663262383664353664623963343366626364363965626435666363306237346130
-30306661386334653137336464393465646135353436336138616563366163366664346331646666
-61613334323330663835366163623836363534666531353737656464663935333765326235306566
-61353962363034663563386137373432656166663661653861396361653930653539363333613435
-64373731616330396130363464653865303931333637393333386531303365646130646161366263
-61643234333563386338643331303164323638346639353765656632386262366666376665356164
-34393464656634303130323738353161643537613337383661343232363961613931613234643361
-34663938626364636562613332373161323932386532356261353137393533633731653034633966
-62316666613738383665303433376438303266646264353133303566636436373966343662363561
-63333662613930323666656166373763303961333332363231396532376564393966333135373966
-65623664303537376362383861663238663463396537353866666333656664323562373165646633
-35396661353639356363613834653432346539633135663565376234383866343433383339666362
-63373330336562323138343934646466373738333039346361623836653231633566316435383636
-61326535393339616639366563383662636136353162393961303362393866363436663630303762
-65343632396238623137333462633633653761336635663265326332623631393566323530623562
-36666431396532303939316662663138356631336434373732393463306336303435626232316638
-36363733383831363930666132396661633733616464393264343339333166633739346236383833
-33373962313762323430343161616361636363366334613032323637316261656333633639333066
-34653635393031323262353037396130383964363037633463653331653965363562326532623037
-383331306234353238633435663136623634

ansible/host_vars/astro-orbiter/vars.yml

@@ -2,11 +2,14 @@
 # ------------------------------------------------------------------------------
 # FILE: ansible/host_vars/astro_orbiter/vars.yml
 # HOST: astro-orbiter (10.1.71.130)
-# ROLE: Ollama inference host with AMD RX 5700 GPU passthrough
+# ROLE: llama.cpp LLM inference host — Ryzen 7 5800XT / RTX 3090 (ATX rebuild,
+#       2026-08-04). Superseded the prior AMD RX 5700 / Ollama config below;
+#       drive was transplanted into new hardware, not reinstalled.
 # ------------------------------------------------------------------------------
 
 ansible_host: 10.1.71.130
-ansible_user: wed
+ansible_user: jarvis
+ansible_ssh_private_key_file: ~/.ssh/id_jarvis
 ansible_become: true
 
 # LVM root expansion — xlarge template uses sda3 partition, standard VG/LV names
@@ -15,11 +18,3 @@ common_root_pv: /dev/sda3
 common_root_vg: ubuntu-vg
 common_root_lv: ubuntu-lv
 
-# Ollama — all defaults apply; explicitly documented here for visibility
-ollama_rocm_version: "6.2"
-ollama_default_model: "qwen3:8b"
-ollama_hsa_override_gfx_version: "10.1.0"
-ollama_data_disk: /dev/sdb
-ollama_data_vg: ollama-vg
-ollama_data_lv: ollama-lv
-ollama_data_dir: /var/lib/ollama

ansible/host_vars/main-street-station/main.yml

@@ -0,0 +1,16 @@
+---
+# Host-specific vars for main-street-station (JMRI headless server)
+# LCRR - Lake Country Railroad, Milwaukee Road Oct 1956, HO scale
+
+# JMRI profile ID — find with: ls ~/.jmri/profiles/ on the old box
+# Format: <name>.<8-char-hex>  e.g. LCRR.3d3f1dfc
+# TODO: fill in after restoring config from GitHub backup
+jmri_profile_id: ""
+
+# USB serial device for NCE command station
+# Verify after install: ls -la /dev/ttyUSB* /dev/ttyACM*
+jmri_serial_device: /dev/ttyUSB0
+
+# Path to JMRI config backup for restore task (leave empty to skip)
+# Point at a local checkout of the LCRR GitHub repo
+jmri_config_src: ""

ansible/host_vars/main-street-station/vars.yml

@@ -0,0 +1,10 @@
+---
+# main-street-station — JMRI / LCRR server
+jmri_profile_id: "Lake_Country_Railroad.3e8b1d4b"
+jmri_lcrr_repo: "ssh://git@gitea.mk-labs.cloud:2221/rblundon/LCRR.git"
+jmri_lcrr_branch: "clean-profile"
+jmri_leviton_email: "{{ leviton_email }}"
+jmri_leviton_password: "{{ leviton_password }}"
+jmri_ssh_authorized_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINnSM/9fO8rz/amqkyoGUzUKNNzzmtSXPwOCr1O9zKNO ansible"
+jmri_ssh_authorized_keys_extra:
+  - "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG6HaK4Y21UwPRbAZ986L7I9QnUdyq53114+9kO8X4bL rblundon@laptop"

ansible/inventory.yml

@@ -59,12 +59,9 @@ n8n_server:
   hosts:
     tiki-room:
 
-ollama_server:
+astro_orbiter:
   hosts:
     astro-orbiter:
-      ansible_host: 10.1.71.130
-      ansible_user: wed
-      ansible_become: true
 
 hermes_server:
   hosts:
@@ -72,6 +69,7 @@ hermes_server:
       ansible_host: 10.1.71.131
       ansible_user: wed
       ansible_become: true
+      ansible_ssh_private_key_file: ~/.ssh/ansible
 
 honcho_server:
   hosts:
@@ -80,6 +78,13 @@ honcho_server:
       ansible_user: wed
       ansible_become: true
 
+jmri_server:
+  hosts:
+    main-street-station:
+      ansible_host: 192.168.10.40
+      ansible_user: wed
+      ansible_become: true
+
 papermc_server:
 # ansible-galaxy role install engonzal.papermc
   hosts:
@@ -88,6 +93,10 @@ papermc_server:
 dev_servers:
   hosts:
     scrim:
+    backstage:
+      ansible_host: 10.1.71.133
+      ansible_user: wed
+      ansible_become: true
 
 # dhcp_server:
 #   hosts:

ansible/playbooks/day1_deploy_jmri.yml

@@ -0,0 +1,24 @@
+---
+# ============================================================================
+# day1_deploy_jmri.yml
+# ----------------------------------------------------------------------------
+# Deploys JMRI JmriFaceless headless server on main-street-station.
+# Applies linux-baseline first, then the jmri role.
+#
+# Usage:
+#   ansible-playbook playbooks/day1_deploy_jmri.yml
+#   ansible-playbook playbooks/day1_deploy_jmri.yml -e target=main-street-station
+#
+# Prerequisites:
+#   1. Host is in inventory under jmri_server group
+#   2. jmri_profile_id is set in host_vars/main-street-station.yml
+#   3. SSH access as 'wed' with sudo
+# ============================================================================
+
+- name: Deploy JMRI headless server
+  hosts: "{{ target | default('jmri_server') }}"
+  become: true
+  gather_facts: true
+  roles:
+    - linux-baseline
+    - jmri

ansible/playbooks/day1_deploy_llm_inference.yml

@@ -0,0 +1,25 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day1_deploy_llm_inference.yml
+# DESCRIPTION: Day 1 playbook for astro-orbiter LLM inference stack.
+#              Deploys vLLM + Gemma 2 27B on RTX 3090 via OCuLink.
+#
+# Usage:
+#   cd ~/git/homelab/ansible
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference.yml
+#
+# Phases (added incrementally — safe to re-run):
+#   1. Foundation  — groups, directories, vault assertion
+#   2. Driver      — nvidia-driver-595-open (idempotent; already installed)
+#   3. vLLM        — Python venv + pip install vllm
+#   4. Model       — HF login, Gemma 2 27B snapshot_download
+#   5. Serve       — systemd vllm-serve.service, health check
+#   6. Integration — Hermes provider config on carousel
+# ------------------------------------------------------------------------------
+
+- name: Deploy LLM inference stack on astro-orbiter
+  hosts: astro_orbiter
+  gather_facts: true
+
+  roles:
+    - role: llm-inference

ansible/playbooks/day1_deploy_llm_inference_multimodel.yml

@@ -0,0 +1,31 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day1_deploy_llm_inference_multimodel.yml
+# DESCRIPTION: Day 1 playbook for the dual-model (aux + tool-calling) rollout
+#              on astro-orbiter. Builds on roles/llm-inference (CUDA/driver
+#              already done) — does not replace it.
+#
+# Usage:
+#   cd ~/git/homelab/ansible
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml
+#   # or scope to specific phases:
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml --tags discover
+#
+# KNOWN GAP (2026-08-05): Semaphore is currently broken; this is being run
+# via direct ansible-playbook as an accepted interim stopgap. Retarget
+# through Semaphore once it's repaired.
+#
+# Phases (see roles/llm-inference-multimodel/README.md for detail):
+#   0. discover  — read-only; confirm existing Gemma service management
+#   1. models    — idempotent GGUF downloads (Phi-4-14B, Mistral-Small-24B)
+#   2. systemd   — deploy both unit files, do NOT auto-start
+#   3. firewall  — scope ports 8000/8001, non-0.0.0.0 bind
+#   4. verify    — start both services, smoke test, VRAM check
+# ------------------------------------------------------------------------------
+
+- name: Deploy dual-model LLM inference stack on astro-orbiter
+  hosts: astro_orbiter
+  gather_facts: true
+
+  roles:
+    - role: llm-inference-multimodel

ansible/roles/jmri/defaults/main.yml

@@ -0,0 +1,57 @@
+---
+# JMRI version to install
+# Update both jmri_version AND jmri_build_hash together when upgrading.
+# Find the build hash in the release asset filename on:
+# https://github.com/JMRI/JMRI/releases
+jmri_version: "5.16"
+jmri_build_hash: "909e15189e"
+jmri_install_dir: /opt/JMRI
+jmri_download_url: "https://github.com/JMRI/JMRI/releases/download/v{{ jmri_version }}/JMRI.{{ jmri_version }}+R{{ jmri_build_hash }}.tgz"
+
+# Service user
+jmri_user: jmri
+jmri_group: jmri
+jmri_home: /home/jmri
+
+# Profile — set per-host in host_vars
+jmri_profile_id: ""
+
+# LCRR git repo (set per-host in host_vars; leave blank to skip clone)
+jmri_lcrr_repo: ""
+jmri_lcrr_branch: "main"
+
+# SSH key for jmri user (for jmri-gui X11 access — set per-host in host_vars)
+jmri_ssh_authorized_key: ""
+jmri_ssh_authorized_keys_extra: []  # additional keys (e.g. operator laptops)
+
+# SSH key for jmri user → Gitea
+jmri_gitea_key: /home/jmri/.ssh/id_ed25519_gitea
+
+# Config restore source (legacy tar-based restore — leave blank to skip)
+jmri_config_src: ""
+
+# Ports (for documentation / firewall rules)
+jmri_json_port: 12080
+jmri_withrottle_port: 12090
+
+# ---------------------------------------------------------------------------
+# Phase 4 — Xpra virtual display
+# Replaces TigerVNC with rootless Xpra — proper window management,
+# persistent sessions, SSH-native attach (no VNC client needed).
+# Connect from macOS/Linux: xpra attach ssh://jmri@main-street-station/100
+# ---------------------------------------------------------------------------
+jmri_xpra_display: "100"
+
+# ---------------------------------------------------------------------------
+# Phase 2 — Layout power monitor (Leviton Decora Smart Wi-Fi)
+# ---------------------------------------------------------------------------
+jmri_leviton_email: ""          # set via group_vars (vault-backed)
+jmri_leviton_password: ""       # set via group_vars (vault-backed)
+jmri_leviton_switch_name: "Layout"
+jmri_monitor_poll_interval: 30  # seconds between polls (active hours)
+jmri_monitor_quiet_start: 1     # hour (24h) to stop polling
+jmri_monitor_quiet_end: 10      # hour (24h) to resume polling
+jmri_monitor_stop_delay: 30     # seconds of OFF state before stopping JMRI
+
+
+

ansible/roles/jmri/handlers/main.yml

@@ -0,0 +1,32 @@
+---
+- name: Reload udev
+  ansible.builtin.command: udevadm control --reload-rules
+  changed_when: false
+
+- name: Trigger udev
+  ansible.builtin.command: udevadm trigger --subsystem-match=tty
+  changed_when: false
+
+- name: Reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true
+
+- name: Restart jmri-monitor
+  ansible.builtin.systemd:
+    name: jmri-monitor
+    state: restarted
+
+- name: Restart jmri
+  ansible.builtin.systemd:
+    name: jmri
+    state: restarted
+
+- name: Restart jmri-xpra
+  ansible.builtin.systemd:
+    name: jmri-xpra
+    state: restarted
+
+- name: Restart sshd
+  ansible.builtin.systemd:
+    name: ssh
+    state: restarted

ansible/roles/jmri/meta/main.yml

@@ -0,0 +1,13 @@
+---
+galaxy_info:
+  role_name: jmri
+  author: JARVIS
+  description: Deploy JMRI JmriFaceless headless server as a systemd service
+  license: MIT
+  min_ansible_version: "2.12"
+  platforms:
+    - name: Ubuntu
+      versions:
+        - jammy
+        - noble
+dependencies: []

ansible/roles/jmri/tasks/main.yml

@@ -0,0 +1,388 @@
+---
+- name: Install Java runtime (full — required for GUI mode)
+  ansible.builtin.apt:
+    name:
+      - openjdk-21-jre
+      - openjdk-21-jdk
+    state: present
+    update_cache: true
+
+- name: Create JMRI system group
+  ansible.builtin.group:
+    name: "{{ jmri_group }}"
+    state: present
+    system: true
+
+- name: Create JMRI service user
+  ansible.builtin.user:
+    name: "{{ jmri_user }}"
+    group: "{{ jmri_group }}"
+    home: "{{ jmri_home }}"
+    shell: /bin/bash
+    system: true
+    create_home: true
+    state: present
+
+- name: Deploy SSH authorized key for jmri user
+  ansible.posix.authorized_key:
+    user: "{{ jmri_user }}"
+    key: "{{ jmri_ssh_authorized_key }}"
+    state: present
+  when: jmri_ssh_authorized_key | length > 0
+
+- name: Deploy extra SSH authorized keys for jmri user
+  ansible.posix.authorized_key:
+    user: "{{ jmri_user }}"
+    key: "{{ item }}"
+    state: present
+  loop: "{{ jmri_ssh_authorized_keys_extra }}"
+
+- name: Add JMRI user to dialout group (serial device access)
+  ansible.builtin.user:
+    name: "{{ jmri_user }}"
+    groups: dialout
+    append: true
+
+# ---------------------------------------------------------------------------
+# Phase 1 — Stable USB device symlinks
+# Creates /dev/jmri/loconet and /dev/jmri/nce via udev ID_SERIAL matching.
+# ---------------------------------------------------------------------------
+- name: Deploy udev rules for JMRI USB devices
+  ansible.builtin.template:
+    src: 99-jmri-devices.rules.j2
+    dest: /etc/udev/rules.d/99-jmri-devices.rules
+    owner: root
+    group: root
+    mode: '0644'
+  notify:
+    - Reload udev
+    - Trigger udev
+
+
+# ---------------------------------------------------------------------------
+# Phase 2 — LocoNet traffic monitor
+# Installs jmri-monitor: watches /dev/jmri/loconet, starts/stops jmri.service
+# ---------------------------------------------------------------------------
+- name: Install python3-venv (monitor virtualenv support)
+  ansible.builtin.apt:
+    name: python3-venv
+    state: present
+
+- name: Create virtualenv for jmri-monitor
+  ansible.builtin.command:
+    cmd: python3 -m venv /opt/jmri-monitor
+    creates: /opt/jmri-monitor/bin/python3
+
+- name: Install decora_wifi into jmri-monitor virtualenv
+  ansible.builtin.pip:
+    name: decora_wifi
+    state: present
+    virtualenv: /opt/jmri-monitor
+
+- name: Deploy jmri-monitor script
+  ansible.builtin.template:
+    src: jmri-monitor.py.j2
+    dest: /usr/local/bin/jmri-monitor
+    owner: root
+    group: root
+    mode: '0755'
+  notify: Restart jmri-monitor
+
+- name: Deploy jmri-monitor systemd unit
+  ansible.builtin.template:
+    src: jmri-monitor.service.j2
+    dest: /etc/systemd/system/jmri-monitor.service
+    owner: root
+    group: root
+    mode: '0644'
+  notify:
+    - Reload systemd
+    - Restart jmri-monitor
+
+- name: Enable jmri-monitor service
+  ansible.builtin.systemd:
+    name: jmri-monitor
+    enabled: true
+    state: started
+    daemon_reload: true
+
+# ---------------------------------------------------------------------------
+# Phase 2b — LCRR config repo (clone/pull .jmri from Gitea)
+# ---------------------------------------------------------------------------
+- name: Ensure jmri .ssh directory exists
+  ansible.builtin.file:
+    path: /home/jmri/.ssh
+    state: directory
+    owner: "{{ jmri_user }}"
+    group: "{{ jmri_group }}"
+    mode: '0700'
+
+- name: Deploy jmri SSH config for Gitea
+  ansible.builtin.copy:
+    dest: /home/jmri/.ssh/config
+    owner: "{{ jmri_user }}"
+    group: "{{ jmri_group }}"
+    mode: '0600'
+    content: |
+      Host gitea.mk-labs.cloud
+          HostName gitea.mk-labs.cloud
+          User git
+          Port 2221
+          IdentityFile {{ jmri_gitea_key }}
+          StrictHostKeyChecking accept-new
+
+- name: Clone LCRR config repo if not present
+  ansible.builtin.git:
+    repo: "{{ jmri_lcrr_repo }}"
+    dest: "{{ jmri_home }}/LCRR"
+    version: "{{ jmri_lcrr_branch }}"
+    accept_hostkey: true
+    key_file: "{{ jmri_gitea_key }}"
+    update: false
+  become_user: "{{ jmri_user }}"
+  when: jmri_lcrr_repo | length > 0
+  notify: Restart jmri
+
+- name: Remove auto-generated .jmri dir if it exists (will be replaced by symlink)
+  ansible.builtin.file:
+    path: "{{ jmri_home }}/.jmri"
+    state: absent
+  when:
+    - jmri_lcrr_repo | length > 0
+
+- name: Link .jmri config from LCRR repo
+  ansible.builtin.file:
+    src: "{{ jmri_home }}/LCRR/.jmri"
+    dest: "{{ jmri_home }}/.jmri"
+    state: link
+    owner: "{{ jmri_user }}"
+    group: "{{ jmri_group }}"
+    force: true
+  when: jmri_lcrr_repo | length > 0
+  notify: Restart jmri
+
+# ---------------------------------------------------------------------------
+# JMRI install / upgrade — version-marker pattern
+# Writes {{ jmri_install_dir }}/.jmri_installed_version after each install.
+# On subsequent runs: read the marker, skip everything if it matches
+# jmri_version. If it differs (or is absent), stop JMRI cleanly, wipe the
+# old install, download the new archive, extract, and write the new marker.
+# To upgrade: bump jmri_version + jmri_build_hash in defaults/main.yml (or
+# host_vars) and re-run the playbook.
+# ---------------------------------------------------------------------------
+
+- name: Read installed JMRI version marker (if present)
+  ansible.builtin.slurp:
+    src: "{{ jmri_install_dir }}/.jmri_installed_version"
+  register: jmri_version_marker
+  ignore_errors: true
+
+- name: Determine whether JMRI install/upgrade is needed
+  ansible.builtin.set_fact:
+    jmri_needs_install: >-
+      {{
+        jmri_version_marker is failed or
+        (jmri_version_marker.content | b64decode | trim) != jmri_version
+      }}
+
+- name: Stop JMRI service before upgrade (if running)
+  ansible.builtin.systemd:
+    name: jmri
+    state: stopped
+  failed_when: false   # service may not exist yet on first install
+  when: jmri_needs_install
+
+- name: Remove existing JMRI install directory (upgrade path)
+  ansible.builtin.file:
+    path: "{{ jmri_install_dir }}"
+    state: absent
+  when: jmri_needs_install
+
+- name: Remove stale JMRI archive from /tmp (if version changed)
+  ansible.builtin.file:
+    path: "/tmp/JMRI-{{ jmri_version }}.tgz"
+    state: absent
+  when: jmri_needs_install
+
+- name: Download JMRI release archive
+  ansible.builtin.get_url:
+    url: "{{ jmri_download_url }}"
+    dest: /tmp/JMRI-{{ jmri_version }}.tgz
+    mode: '0644'
+  when: jmri_needs_install
+
+- name: Create JMRI install directory
+  ansible.builtin.file:
+    path: "{{ jmri_install_dir }}"
+    state: directory
+    owner: "{{ jmri_user }}"
+    group: "{{ jmri_group }}"
+    mode: '0755'
+
+- name: Extract JMRI archive
+  ansible.builtin.unarchive:
+    src: /tmp/JMRI-{{ jmri_version }}.tgz
+    dest: "{{ jmri_install_dir }}"
+    remote_src: true
+    owner: "{{ jmri_user }}"
+    group: "{{ jmri_group }}"
+    extra_opts: ['--strip-components=1']
+  when: jmri_needs_install
+
+- name: Write installed version marker
+  ansible.builtin.copy:
+    content: "{{ jmri_version }}\n"
+    dest: "{{ jmri_install_dir }}/.jmri_installed_version"
+    owner: "{{ jmri_user }}"
+    group: "{{ jmri_group }}"
+    mode: '0644'
+  when: jmri_needs_install
+
+- name: Restore JMRI config from backup
+  ansible.builtin.copy:
+    src: "{{ jmri_config_src }}/"
+    dest: "{{ jmri_home }}/.jmri/"
+    owner: "{{ jmri_user }}"
+    group: "{{ jmri_group }}"
+    mode: '0644'
+    directory_mode: '0755'
+  when: jmri_config_src | length > 0
+  notify: Restart jmri
+
+- name: Deploy JmriFaceless systemd unit
+  ansible.builtin.template:
+    src: jmri.service.j2
+    dest: /etc/systemd/system/jmri.service
+    owner: root
+    group: root
+    mode: '0644'
+  notify:
+    - Reload systemd
+    - Restart jmri
+
+- name: Enable jmri service (monitor manages start/stop — do not start directly)
+  ansible.builtin.systemd:
+    name: jmri
+    enabled: false
+    daemon_reload: true
+
+# ---------------------------------------------------------------------------
+# Phase 3 — X11 remote GUI access (retained for fallback; VNC preferred)
+# ---------------------------------------------------------------------------
+- name: Install xauth (required for SSH X11 forwarding fallback)
+  ansible.builtin.apt:
+    name: xauth
+    state: present
+
+- name: Remove old jmri X11 sshd drop-in if present (renamed)
+  ansible.builtin.file:
+    path: /etc/ssh/sshd_config.d/20-jmri-x11.conf
+    state: absent
+  notify: Restart sshd
+
+- name: Deploy sshd drop-in to enable X11 forwarding (must load before hardening)
+  ansible.builtin.copy:
+    dest: /etc/ssh/sshd_config.d/09-jmri-x11.conf
+    owner: root
+    group: root
+    mode: '0644'
+    content: |
+      # Allow X11 forwarding for JMRI GUI sessions (jmri role)
+      # Must be numbered below 10-mk-labs-hardening.conf — first match wins.
+      X11Forwarding yes
+      X11UseLocalhost yes
+  notify: Restart sshd
+
+# ---------------------------------------------------------------------------
+# Phase 4 — Xpra virtual display
+# Replaces TigerVNC. Rootless mode: each JMRI window appears as a native
+# window on the client. Sessions are persistent across disconnects.
+# Connect: xpra attach ssh://jmri@main-street-station/{{ jmri_xpra_display }}
+# ---------------------------------------------------------------------------
+- name: Remove TigerVNC (replaced by Xpra)
+  ansible.builtin.apt:
+    name: tigervnc-standalone-server
+    state: absent
+  notify: Reload systemd
+
+- name: Disable and stop jmri-vnc service if present
+  ansible.builtin.systemd:
+    name: jmri-vnc
+    enabled: false
+    state: stopped
+  failed_when: false
+
+- name: Remove jmri-vnc systemd unit if present
+  ansible.builtin.file:
+    path: /etc/systemd/system/jmri-vnc.service
+    state: absent
+  notify: Reload systemd
+
+- name: Remove jmri VNC password directory if present
+  ansible.builtin.file:
+    path: "{{ jmri_home }}/.vnc"
+    state: absent
+
+- name: Install xpra.org apt signing key
+  ansible.builtin.get_url:
+    url: https://xpra.org/gpg.asc
+    dest: /usr/share/keyrings/xpra.asc
+    mode: '0644'
+
+- name: Add xpra.org upstream apt repository
+  ansible.builtin.copy:
+    dest: /etc/apt/sources.list.d/xpra.list
+    owner: root
+    group: root
+    mode: '0644'
+    content: |
+      deb [arch=amd64 signed-by=/usr/share/keyrings/xpra.asc] https://xpra.org/ noble main
+
+- name: Install Xpra from upstream repo (v6.x)
+  ansible.builtin.apt:
+    name: xpra
+    state: latest
+    update_cache: true
+
+- name: Deploy jmri-xpra systemd unit
+  ansible.builtin.template:
+    src: jmri-xpra.service.j2
+    dest: /etc/systemd/system/jmri-xpra.service
+    owner: root
+    group: root
+    mode: '0644'
+  notify:
+    - Reload systemd
+    - Restart jmri-xpra
+
+- name: Enable and start jmri-xpra service
+  ansible.builtin.systemd:
+    name: jmri-xpra
+    enabled: true
+    state: started
+    daemon_reload: true
+
+- name: Deploy jmri-gui script
+  ansible.builtin.template:
+    src: jmri-gui.j2
+    dest: /usr/local/bin/jmri-gui
+    owner: root
+    group: root
+    mode: '0755'
+
+- name: Deploy sudoers drop-in for jmri-gui (wed can manage jmri service)
+  ansible.builtin.copy:
+    dest: /etc/sudoers.d/jmri-gui
+    owner: root
+    group: root
+    mode: '0440'
+    validate: 'visudo -cf %s'
+    content: |
+      # jmri user can manage its own service (for jmri-gui interactive sessions)
+      jmri ALL=(root) NOPASSWD: /usr/bin/systemctl start jmri.service
+      jmri ALL=(root) NOPASSWD: /usr/bin/systemctl stop jmri.service
+      jmri ALL=(root) NOPASSWD: /opt/jmri-monitor/bin/python3
+      # wed retains service control for automation/admin use
+      wed ALL=(root) NOPASSWD: /usr/bin/systemctl start jmri.service
+      wed ALL=(root) NOPASSWD: /usr/bin/systemctl stop jmri.service
+      wed ALL=(root) NOPASSWD: /opt/jmri-monitor/bin/python3

ansible/roles/jmri/templates/99-jmri-devices.rules.j2

@@ -0,0 +1,15 @@
+# JMRI USB device symlinks — managed by Ansible, do not edit manually.
+# Creates stable /dev/jmri-* symlinks at the top level of /dev so JMRI
+# can enumerate them alongside real tty devices.
+
+# LocoNet interface — RR-CirKits LocoBuffer-NG (Microchip CDC)
+SUBSYSTEM=="tty", ENV{ID_SERIAL}=="RR-CirKits_LocoBuffer-NG_CDC_ACM_SERIAL_DEVICE_AA5700218A", \
+    SYMLINK+="jmri-loconet", MODE="0666"
+
+# NCE Power Pro command station (FTDI FT232)
+SUBSYSTEM=="tty", ENV{ID_SERIAL}=="ftdi_usb_serial_converter_ftDYQHZX", \
+    SYMLINK+="jmri-nce", MODE="0666"
+
+# LCC buffer (Microchip CDC)
+SUBSYSTEM=="tty", ENV{ID_SERIAL}=="Microchip_Technology_Inc._Simple_CDC_Device_Demo", \
+    SYMLINK+="jmri-lcc", MODE="0666"

ansible/roles/jmri/templates/jmri-gui.j2

@@ -0,0 +1,110 @@
+#!/bin/bash
+# jmri-gui — launch JMRI GUI on Xpra virtual display
+# Managed by Ansible — do not edit manually.
+#
+# Usage (connect as jmri user):
+#   jmri-gui panelpro       Launch PanelPro on Xpra display
+#   jmri-gui decoderpro     Launch DecoderPro on Xpra display
+#   jmri-gui status         Show service status and attach command
+#
+# Then attach from macOS/Linux:
+#   xpra attach ssh://jmri@main-street-station/{{ jmri_xpra_display }}
+
+set -euo pipefail
+
+JMRI_DIR="{{ jmri_install_dir }}"
+JMRI_SERVICE="jmri.service"
+MONITOR_SERVICE="jmri-monitor.service"
+XPRA_SERVICE="jmri-xpra.service"
+DISPLAY=":{{ jmri_xpra_display }}"
+export DISPLAY
+
+usage() {
+    echo "Usage: jmri-gui <panelpro|decoderpro|status>"
+    exit 1
+}
+
+status() {
+    echo "=== JMRI daemon ==="
+    systemctl status "$JMRI_SERVICE" --no-pager -l 2>&1 | head -8
+    echo ""
+    echo "=== Xpra display ==="
+    systemctl status "$XPRA_SERVICE" --no-pager -l 2>&1 | head -5
+    echo ""
+    echo "=== Layout monitor ==="
+    systemctl status "$MONITOR_SERVICE" --no-pager -l 2>&1 | head -5
+    echo ""
+    echo "To attach: xpra attach ssh://jmri@$(hostname -f)/{{ jmri_xpra_display }}"
+}
+
+launch() {
+    local app="$1"
+    local binary
+
+    case "$app" in
+        panelpro)   binary="PanelPro" ;;
+        decoderpro) binary="DecoderPro" ;;
+        *)          usage ;;
+    esac
+
+    # Ensure Xpra display is running
+    if ! systemctl is-active --quiet "$XPRA_SERVICE" 2>/dev/null; then
+        echo "Starting Xpra display..."
+        sudo systemctl start "$XPRA_SERVICE"
+        sleep 2
+    fi
+
+    # Stop the JMRI daemon if running (we're taking over the hardware connections)
+    if systemctl is-active --quiet "$JMRI_SERVICE" 2>/dev/null; then
+        echo "Stopping JMRI daemon..."
+        sudo systemctl stop "$JMRI_SERVICE"
+    fi
+
+    # Force AWT out of headless mode
+    export JMRI_OPTIONS="-Djava.awt.headless=false"
+
+    echo "Launching $binary on Xpra display $DISPLAY..."
+    echo ""
+    echo "Attach from your workstation:"
+    echo "  xpra attach ssh://jmri@$(hostname -f)/{{ jmri_xpra_display }}"
+    echo ""
+
+    "$JMRI_DIR/$binary" &
+
+    echo "$binary launched. Attach with xpra to see windows."
+    echo ""
+
+    # Restart daemon if layout switch is still on
+    if systemctl is-active --quiet "$MONITOR_SERVICE" 2>/dev/null; then
+        if sudo /opt/jmri-monitor/bin/python3 - <<'EOF'
+from decora_wifi import DecoraWiFiSession
+from decora_wifi.models.residential_account import ResidentialAccount
+import sys
+session = DecoraWiFiSession()
+person = session.login("{{ jmri_leviton_email }}", "{{ jmri_leviton_password }}")
+perms = person.get_residential_permissions()
+for perm in perms:
+    acct_id = perm.data.get('residentialAccountId')
+    if not acct_id:
+        continue
+    acct = ResidentialAccount(session, acct_id)
+    acct.refresh()
+    for r in acct.get_residences():
+        for s in r.get_iot_switches():
+            if s.data.get('name') == '{{ jmri_leviton_switch_name }}':
+                sys.exit(0 if s.data.get('power') == 'ON' else 1)
+sys.exit(1)
+EOF
+        then
+            echo "Layout is ON — JMRI daemon will restart when GUI is closed."
+        else
+            echo "Layout is OFF — JMRI daemon will not restart."
+        fi
+    fi
+}
+
+case "${1:-}" in
+    panelpro|decoderpro) launch "$1" ;;
+    status)              status ;;
+    *)                   usage ;;
+esac

ansible/roles/jmri/templates/jmri-monitor.py.j2

@@ -0,0 +1,156 @@
+#!/opt/jmri-monitor/bin/python3
+"""
+jmri-monitor — Leviton Decora Smart Wi-Fi layout power monitor
+Managed by Ansible — do not edit manually.
+
+Polls the Leviton cloud API for the "{{ jmri_leviton_switch_name }}" switch state.
+  - Switch ON  after being OFF → systemctl start jmri.service
+  - Switch OFF for {{ jmri_monitor_stop_delay }}s  → systemctl stop jmri.service
+
+Quiet hours {{ jmri_monitor_quiet_start }}:00–{{ jmri_monitor_quiet_end }}:00: no polling (layout assumed off).
+"""
+
+import subprocess
+import time
+import logging
+import sys
+from datetime import datetime
+
+LEVITON_EMAIL    = "{{ jmri_leviton_email }}"
+LEVITON_PASSWORD = "{{ jmri_leviton_password }}"
+SWITCH_NAME      = "{{ jmri_leviton_switch_name }}"
+POLL_INTERVAL    = {{ jmri_monitor_poll_interval }}
+QUIET_START      = {{ jmri_monitor_quiet_start }}
+QUIET_END        = {{ jmri_monitor_quiet_end }}
+STOP_DELAY       = {{ jmri_monitor_stop_delay }}
+JMRI_SERVICE     = "jmri.service"
+
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s [jmri-monitor] %(levelname)s: %(message)s",
+    datefmt="%Y-%m-%d %H:%M:%S",
+    stream=sys.stdout,
+)
+log = logging.getLogger(__name__)
+
+
+def systemctl(action):
+    try:
+        result = subprocess.run(
+            ["systemctl", action, JMRI_SERVICE],
+            capture_output=True, text=True, timeout=30,
+        )
+        if result.returncode == 0:
+            log.info("systemctl %s %s: OK", action, JMRI_SERVICE)
+        else:
+            log.warning("systemctl %s %s: %s", action, JMRI_SERVICE, result.stderr.strip())
+    except Exception as e:
+        log.error("systemctl %s failed: %s", action, e)
+
+
+def is_quiet_hours():
+    hour = datetime.now().hour
+    if QUIET_START < QUIET_END:
+        return QUIET_START <= hour < QUIET_END
+    else:
+        # wraps midnight e.g. 23–6
+        return hour >= QUIET_START or hour < QUIET_END
+
+
+def get_switch_state():
+    """Returns True if switch is ON, False if OFF, None on error."""
+    try:
+        from decora_wifi import DecoraWiFiSession
+        from decora_wifi.models.residential_account import ResidentialAccount
+
+        session = DecoraWiFiSession()
+        person = session.login(LEVITON_EMAIL, LEVITON_PASSWORD)
+        if not person:
+            log.error("Leviton login failed")
+            return None
+
+        perms = person.get_residential_permissions()
+        for perm in perms:
+            acct_id = perm.data.get('residentialAccountId')
+            if not acct_id:
+                continue
+            acct = ResidentialAccount(session, acct_id)
+            acct.refresh()
+            for residence in acct.get_residences():
+                for switch in residence.get_iot_switches():
+                    if switch.data.get('name') == SWITCH_NAME:
+                        state = switch.data.get('power', 'OFF')
+                        session.call_api('/Person/logout', {}, 'post')
+                        return state == 'ON'
+
+        all_names = []
+        for perm in perms:
+            acct_id = perm.data.get('residentialAccountId')
+            if acct_id:
+                acct = ResidentialAccount(session, acct_id)
+                acct.refresh()
+                for r in acct.get_residences():
+                    all_names += [s.data.get('name') for s in r.get_iot_switches()]
+        log.warning("Switch '%s' not found — available: %s", SWITCH_NAME, all_names)
+        session.call_api('/Person/logout', {}, 'post')
+        return None
+
+    except ImportError:
+        log.error("decora_wifi not installed")
+        return None
+    except Exception as e:
+        log.error("Error querying Leviton API: %s", e)
+        return None
+
+
+def main():
+    log.info("Layout power monitor starting")
+    log.info("Switch: '%s'  Poll: %ds  Quiet: %02d:00–%02d:00  Stop delay: %ds",
+             SWITCH_NAME, POLL_INTERVAL, QUIET_START, QUIET_END, STOP_DELAY)
+
+    layout_on = False
+    off_since = None
+
+    while True:
+        if is_quiet_hours():
+            log.debug("Quiet hours — sleeping 60s")
+            # If layout was on when quiet hours started, stop JMRI
+            if layout_on:
+                log.info("Quiet hours began — stopping JMRI")
+                layout_on = False
+                off_since = None
+                systemctl("stop")
+            time.sleep(60)
+            continue
+
+        state = get_switch_state()
+
+        if state is True:
+            off_since = None
+            if not layout_on:
+                log.info("Layout switch ON — starting JMRI")
+                layout_on = True
+                systemctl("start")
+
+        elif state is False:
+            if layout_on:
+                if off_since is None:
+                    off_since = time.monotonic()
+                    log.info("Layout switch OFF — waiting %ds before stopping JMRI", STOP_DELAY)
+                elif time.monotonic() - off_since >= STOP_DELAY:
+                    log.info("Layout switch OFF for %ds — stopping JMRI", STOP_DELAY)
+                    layout_on = False
+                    off_since = None
+                    systemctl("stop")
+            else:
+                off_since = None
+
+        else:
+            # API error — don't change state, try again next poll
+            log.warning("Could not determine switch state — retrying in %ds", POLL_INTERVAL)
+
+        time.sleep(POLL_INTERVAL)
+
+
+if __name__ == "__main__":
+    main()

ansible/roles/jmri/templates/jmri-monitor.service.j2

@@ -0,0 +1,20 @@
+[Unit]
+Description=JMRI LocoNet Traffic Monitor
+Documentation=https://www.jmri.org/
+# Must start after udev has processed devices
+After=systemd-udev-settle.service
+
+[Service]
+Type=simple
+# Runs as root — needs to call systemctl start/stop jmri.service
+User=root
+ExecStart=/usr/local/bin/jmri-monitor
+Restart=always
+RestartSec=10
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=jmri-monitor
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/jmri/templates/jmri-xpra.service.j2

@@ -0,0 +1,26 @@
+[Unit]
+Description=Xpra virtual display for JMRI (:{{ jmri_xpra_display }})
+After=network.target
+# Start before jmri.service so the display is ready when JMRI launches
+Before=jmri.service
+
+[Service]
+Type=simple
+User={{ jmri_user }}
+Group={{ jmri_group }}
+ExecStart=/usr/bin/xpra start \
+    :{{ jmri_xpra_display }} \
+    --daemon=no \
+    --mdns=no \
+    --notifications=no \
+    --systemd-run=no \
+    --pulseaudio=no \
+    --speaker=off \
+    --microphone=off \
+    --video-encoders=none \
+    --start-via-proxy=no
+Restart=on-failure
+RestartSec=5
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/jmri/templates/jmri.service.j2

@@ -0,0 +1,31 @@
+[Unit]
+Description=JMRI Server (JmriFaceless)
+Documentation=https://www.jmri.org/
+# jmri-monitor starts and stops this service based on LocoNet traffic.
+# Do NOT enable this unit directly — it is managed by jmri-monitor.service.
+After=network.target systemd-udev-settle.service
+
+[Service]
+Type=simple
+User={{ jmri_user }}
+Group={{ jmri_group }}
+WorkingDirectory={{ jmri_install_dir }}
+ExecStart={{ jmri_install_dir }}/JmriFaceless --profile={{ jmri_profile_id }}
+
+# Monitor owns lifecycle — do not auto-restart
+Restart=no
+
+# Give hardware time to settle on start
+TimeoutStartSec=30
+
+# Logging — view with: journalctl -u jmri -f
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=jmri
+
+# Serial device access
+SupplementaryGroups=dialout
+
+[Install]
+# Intentionally no WantedBy — started only by jmri-monitor
+WantedBy=

ansible/roles/llm-inference-multimodel/README.md

@@ -0,0 +1,132 @@
+# llm-inference-multimodel
+
+Deploys **two independent llama-server systemd services** on astro-orbiter's
+RTX 3090 (24GB), alongside — not replacing — the existing `llm-inference` role:
+
+| Instance | Port | Model | Quant | ctx | parallel | ~VRAM |
+|---|---|---|---|---|---|---|
+| `llama-server-aux` | 8000 | Phi-4-14B-Instruct | Q4_K_M | 8192 | 2 | ~10.0GB |
+| `llama-server-toolcall` | 8001 | Mistral-Small-24B-Instruct-2501 | Q3_K_M | 4096 | 1 | ~13.2GB |
+
+Combined estimate: **~23.2GB / 24GB** (~0.8GB headroom). See
+`/home/hermes/astro-orbiter-multi-model-plan.md` for the full approved design
+(VRAM math, model selection rationale, rollback plan, validation harness).
+
+## Relationship to `roles/llm-inference`
+
+This role does **not** replace `llm-inference`. It assumes that role's
+prerequisites are already satisfied on the host:
+
+- NVIDIA driver installed
+- `/opt/llama.cpp` cloned and built with CUDA (`/opt/llama.cpp/build/bin/llama-server` exists)
+- `jarvis` service user + `/home/jarvis` present
+
+The pre-existing single-model Gemma llama-server (however it is currently
+run) is **never modified, restarted, or deleted** by this role. It is the
+rollback target.
+
+## Phases
+
+Run the whole role, or scope with `--tags`:
+
+```
+ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml
+# or, once merged into a single play:
+ansible-playbook -i inventory.yml <playbook>.yml --tags discover,models,systemd,firewall,verify
+```
+
+0. **discover** (`tasks/discover.yml`) — READ-ONLY. Confirms via
+   `service_facts` + `pgrep` whether the existing Gemma llama-server actually
+   runs as a systemd unit today, or some ad hoc way (nohup/screen/tmux). Does
+   **not** assume a unit exists — this was an open unknown in the plan and is
+   resolved here as a fact-gathering step, not an assumption. Also records
+   baseline VRAM and current port 8000/8001 listeners for comparison later.
+
+   **If this reports no unit found**, stop and read the debug message —
+   it means plan §6's rollback story ("systemctl start the old unit to
+   revert") isn't actually available yet, and that should be fixed (codify
+   the existing process as a systemd unit) before proceeding to Phase 2.
+
+1. **models** (`tasks/models.yml`) — Idempotent GGUF download to
+   `/opt/models/` with a stat + minimum-size guard (mirrors the pattern in
+   `roles/llm-inference/tasks/serve.yml` and the `llm-inference-homelab`
+   skill), so reruns don't re-pull 8.5GB/11.7GB files or mistake a truncated
+   partial download for complete.
+
+2. **systemd** (`tasks/systemd.yml`) — Templates and deploys both unit files
+   to `/etc/systemd/system/`. **Deliberately does not start or enable either
+   service** — units land on disk as a separately reviewable checkpoint.
+   Two fully independent units (not one unit with two ExecStarts) so either
+   instance can be restarted/stopped without affecting the other.
+
+3. **firewall** (`tasks/firewall.yml`) — Scopes ports 8000 and 8001 via `ufw`
+   to `llm_allowed_source_cidr` (default the Hermes LAN subnet), rather than
+   leaving them open. Both unit templates also bind to
+   `llm_bind_address` (default `10.1.71.130`, the host's private LAN IP) —
+   **not `0.0.0.0`** — which is a deliberate change from the pre-existing
+   Gemma pattern flagged as insecure in the plan.
+
+4. **verify** (`tasks/verify.yml`) — The only phase that actually starts +
+   enables both services. Waits for `/health` on both ports, smoke-tests
+   `/v1/models` and a trivial `/v1/chat/completions` call on each, checks
+   `nvidia-smi` VRAM usage against the plan's design estimate, and greps
+   `dmesg` for OOM-kill events.
+
+   **This smoke test is not the tool-calling validation harness.** See
+   below.
+
+## Key variables
+
+Defined in `defaults/main.yml` (all overridable via `host_vars`/`group_vars`
+or `-e`):
+
+- `llm_service_user` (jarvis), `llm_binary_path`, `llm_models_dir`, `llm_bind_address`, `llm_allowed_source_cidr`
+- Aux: `llm_aux_port`, `llm_aux_model_path`, `llm_aux_model_url`, `llm_aux_ctx_size`, `llm_aux_parallel`, `llm_aux_gpu_layers`
+- Tool-calling: `llm_toolcall_port`, `llm_toolcall_model_path`, `llm_toolcall_model_url`, `llm_toolcall_ctx_size`, `llm_toolcall_parallel`, `llm_toolcall_gpu_layers`
+
+`vars/main.yml` holds constants not meant to be overridden per-host (HF token
+reference, expected-VRAM figures used only for the verify.yml report).
+
+## ⚠️ Tool-calling validation is required before use
+
+Port 8001 (Mistral-Small-24B) **must** pass the manual validation procedure
+described in plan §7 before any Claude Code / tool-calling-capable Hermes
+profile is pointed at it:
+
+1. A curl-based `tool_calls` emission probe (does it call tools correctly on
+   known trigger prompts?)
+2. A hallucination stress test (does it fabricate `tool_calls` on prompts
+   that shouldn't trigger any?)
+3. A shadow-mode period (run parallel to the existing tool-calling path,
+   compare outputs, before a hard cutover)
+
+This is **intentionally not automated into this role** — it is a
+correctness/safety judgment call, not a repeatable infra check. See
+`docs/validation-log.md` in this role directory for the procedure reference
+and a place to log results once Ryan runs it.
+
+## Known gap: Semaphore is broken (as of 2026-08-05)
+
+The normal execution/audit path (Semaphore) is currently non-functional.
+This role was authored to be run via direct `ansible-playbook` as an accepted
+interim stopgap, executed personally by Ryan. **This is a known gap, not the
+intended long-term operational path** — once Semaphore is repaired, retarget
+execution of this role (and future changes to it) through Semaphore so runs
+are audited/logged there again. Flag this in any future work that touches
+this role.
+
+## Rollback
+
+The existing Gemma llama-server and its GGUF are untouched by every phase of
+this role. To roll back:
+
+1. `systemctl stop llama-server-aux llama-server-toolcall`
+2. `systemctl disable llama-server-aux llama-server-toolcall` (optional, if reverting permanently)
+3. Confirm the original Gemma service (name determined by `discover.yml`,
+   commonly `llama-server.service`) is (still) running: `systemctl status llama-server`
+4. If it was never running because Phase 0 discovered it wasn't a managed
+   unit, whatever ad hoc process/command was used before this role's changes
+   is also unaffected — nothing in this role stopped it.
+
+No files belonging to the existing Gemma deployment (GGUF, unit file, or
+otherwise) are ever written to or deleted by this role.

ansible/roles/llm-inference-multimodel/defaults/main.yml

@@ -0,0 +1,59 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference-multimodel/defaults/main.yml
+# DESCRIPTION: Overridable defaults for the llm-inference-multimodel role.
+#              Deploy target: astro-orbiter (10.1.71.130, RTX 3090 24GB).
+#              Built ALONGSIDE roles/llm-inference (not a replacement) — that
+#              role's CUDA/build/driver phases are the prerequisite; this role
+#              assumes /opt/llama.cpp/build/bin/llama-server already exists.
+#
+#              See /home/hermes/astro-orbiter-multi-model-plan.md for the full
+#              approved design (VRAM math, rationale, rollback story).
+# ------------------------------------------------------------------------------
+
+# Shared
+llm_service_user: jarvis
+llm_binary_path: /opt/llama.cpp/build/bin/llama-server
+llm_models_dir: /opt/models
+
+# Bind address — deliberately NOT 0.0.0.0 (see plan §5). Default to the private
+# LAN interface so both instances are reachable from Hermes but not the world.
+# Override to 127.0.0.1 if even LAN-wide reachability is unwanted and a reverse
+# proxy/localhost-only tunnel is used instead.
+llm_bind_address: "10.1.71.130"
+
+# Firewall scoping (Phase 3) — subnet/hosts allowed to reach the ports above.
+# Override per-environment; default assumes Hermes runs somewhere on this /24.
+llm_allowed_source_cidr: "10.1.70.0/24"
+
+# --- Aux / classification instance (port 8000, Phi-4-14B) -------------------
+# Text-only instruction model, no tool-calling training — safe offload target
+# per the auxiliary-task-offload skill's "no tool_calls emission risk" bar.
+llm_aux_port: 8000
+llm_aux_model_path: "{{ llm_models_dir }}/phi-4-14b-instruct-Q4_K_M.gguf"
+llm_aux_model_url: "https://huggingface.co/bartowski/phi-4-GGUF/resolve/main/phi-4-Q4_K_M.gguf"
+llm_aux_model_min_bytes: 8000000000   # guard threshold; complete file ~8.5GB
+llm_aux_ctx_size: 8192
+llm_aux_parallel: 2
+llm_aux_gpu_layers: 99
+llm_aux_service_name: llama-server-aux
+llm_aux_model_id: phi-4-14b-instruct   # served model name for OpenAI-compat API
+
+# --- Tool-calling instance (port 8001, Mistral-Small-24B) --------------------
+# Native function-calling support; deployed at Q3_K_M per plan §1 Option B
+# to fit VRAM budget. MUST pass the §7 validation harness before any
+# Claude-Code-capable profile is pointed at this port.
+llm_toolcall_port: 8001
+llm_toolcall_model_path: "{{ llm_models_dir }}/mistral-small-24b-instruct-2501-Q3_K_M.gguf"
+llm_toolcall_model_url: "https://huggingface.co/bartowski/Mistral-Small-24B-Instruct-2501-GGUF/resolve/main/Mistral-Small-24B-Instruct-2501-Q3_K_M.gguf"
+llm_toolcall_model_min_bytes: 11000000000   # guard threshold; complete file ~11.7GB
+llm_toolcall_ctx_size: 4096
+llm_toolcall_parallel: 1
+llm_toolcall_gpu_layers: 99
+llm_toolcall_service_name: llama-server-toolcall
+llm_toolcall_model_id: mistral-small-24b-instruct-2501
+
+# --- Existing Gemma baseline (rollback target — never modified by this role) -
+# Populated by Phase 0 discovery (tasks/discover.yml) if not already known.
+# Set here only as a fallback name to search for; discovery is authoritative.
+llm_existing_gemma_service_name_guess: llama-server

ansible/roles/llm-inference-multimodel/docs/validation-log.md

@@ -0,0 +1,46 @@
+# Tool-Calling Model Validation Log
+
+This file tracks the manual validation procedure required by
+`astro-orbiter-multi-model-plan.md` §7 before `llama-server-toolcall` (port
+8001, Mistral-Small-24B-Instruct-2501 Q3_K_M) is trusted for any real
+tool-calling / Claude Code Hermes profile traffic.
+
+This is **not automated by the role** — `tasks/verify.yml` only confirms the
+endpoint is up and can produce a basic completion. The checks below are a
+correctness/safety judgment call that a human runs and records here.
+
+## Procedure (plan §7 summary)
+
+1. **`tool_calls` emission probe** — curl a handful of known
+   tool-triggering prompts (e.g. "what's the weather in Austin right now")
+   against `POST http://10.1.71.130:8001/v1/chat/completions` with a `tools`
+   array defined, and confirm the response actually contains a well-formed
+   `tool_calls` block (correct function name, valid JSON arguments) rather
+   than a plain-text answer or a malformed call.
+
+2. **Hallucination stress test** — send prompts that should **not** trigger
+   any tool call (general knowledge questions, casual chat, prompts that
+   merely mention a tool's name in passing) and confirm the model does
+   **not** emit a spurious `tool_calls` block. This is the primary risk
+   flagged in the plan given Mistral-Small's Q3_K_M quantization and its
+   lineage concerns around over-eager tool invocation.
+
+3. **Shadow mode** — for a bounded period, run this instance in parallel
+   with whatever tool-calling path is currently in production, comparing
+   outputs on the same real traffic (or a recorded sample) without letting
+   this instance's outputs actually drive tool execution. Only cut over
+   once outputs are consistently correct.
+
+See the `llm-inference-homelab` skill's `scripts/tool-calling-validation.sh`
+reference for a starting curl harness shape — adapt prompts/tool schemas to
+Mistral-Small's actual expected format (confirm via the GGUF's embedded
+chat template / model card) rather than assuming it matches Qwen's.
+
+## Log
+
+| Date | Run by | Probe result | Hallucination test result | Shadow mode outcome | Decision |
+|---|---|---|---|---|---|
+| _(pending)_ | | | | | Not yet cut over — do not point production tool-calling traffic at :8001 |
+
+Update this table after each validation pass. Do not remove prior rows —
+this is the audit trail for "when did we decide this was safe to use."

ansible/roles/llm-inference-multimodel/handlers/main.yml

@@ -0,0 +1,27 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference-multimodel/handlers/main.yml
+# DESCRIPTION: Separate restart handlers per instance — NEVER combined, so a
+#              content change to one unit template never restarts the other
+#              (plan §2/§6 requirement: independent restart/rollback).
+# ------------------------------------------------------------------------------
+
+- name: Reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true
+  become: true
+  listen: "reload systemd"
+
+- name: Restart llama-server-aux
+  ansible.builtin.systemd:
+    name: "{{ llm_aux_service_name }}"
+    state: restarted
+  become: true
+  listen: "restart llama-server-aux"
+
+- name: Restart llama-server-toolcall
+  ansible.builtin.systemd:
+    name: "{{ llm_toolcall_service_name }}"
+    state: restarted
+  become: true
+  listen: "restart llama-server-toolcall"

ansible/roles/llm-inference-multimodel/meta/main.yml

@@ -0,0 +1,17 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference-multimodel/meta/main.yml
+# ------------------------------------------------------------------------------
+galaxy_info:
+  role_name: llm_inference_multimodel
+  author: rblundon
+  license: MIT
+  description: >
+    Deploys two independent llama-server instances on astro-orbiter's RTX 3090:
+    an aux/classification instance (Phi-4-14B Q4_K_M, port 8000) and a
+    tool-calling instance (Mistral-Small-24B-Instruct-2501 Q3_K_M, port 8001).
+    Built alongside roles/llm-inference (not a replacement); assumes that
+    role's CUDA build/driver work is already done. See
+    /home/hermes/astro-orbiter-multi-model-plan.md for the full design.
+  min_ansible_version: "2.15"
+dependencies: []

ansible/roles/llm-inference-multimodel/tasks/discover.yml

@@ -0,0 +1,87 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference-multimodel/tasks/discover.yml
+# DESCRIPTION: Phase 0 — READ-ONLY fact gathering on how the existing Gemma
+#              llama-server is actually managed on astro-orbiter TODAY.
+#
+#              Per plan §0/§4: "Service management: unverified — plan requires
+#              confirming systemd unit exists before touching anything.
+#              Do not assume." This file performs that confirmation. It makes
+#              NO changes to the host — no `state: present/started/stopped`,
+#              no file writes, no service actions. Every task here is either
+#              a `_facts` module, a `command`/`shell` in check-safe read mode,
+#              or a `stat`.
+#
+#              Outcomes recorded as facts for later phases/for a human to read
+#              in the play recap — this file does not branch role behavior
+#              on the result (that would be over-engineering a role meant to
+#              run once); it surfaces what's true so a human confirms before
+#              Phase 2 proceeds.
+# ------------------------------------------------------------------------------
+
+- name: Gather service facts (systemd unit inventory)
+  ansible.builtin.service_facts:
+
+- name: Determine whether a systemd unit matching the existing Gemma service exists
+  ansible.builtin.set_fact:
+    llm_existing_gemma_unit_found: "{{ (llm_existing_gemma_service_name_guess + '.service') in ansible_facts.services }}"
+
+- name: Report existing Gemma systemd unit state (if found)
+  ansible.builtin.debug:
+    msg: >-
+      Existing unit '{{ llm_existing_gemma_service_name_guess }}.service' found:
+      state={{ ansible_facts.services[llm_existing_gemma_service_name_guess + '.service'].state | default('unknown') }},
+      status={{ ansible_facts.services[llm_existing_gemma_service_name_guess + '.service'].status | default('unknown') }}
+  when: llm_existing_gemma_unit_found
+
+- name: WARNING — no systemd unit found matching the existing Gemma service
+  ansible.builtin.debug:
+    msg: >-
+      No systemd unit named '{{ llm_existing_gemma_service_name_guess }}.service'
+      was found via service_facts. This means the current single-model
+      llama-server is likely run some other way (manual nohup, screen/tmux,
+      or a differently-named unit). DO NOT PROCEED to Phase 2 assuming a
+      clean rollback target exists. Before continuing: (1) check for any
+      running llama-server process via `ansible -m command -a "pgrep -fa
+      llama-server"`, (2) if found running ad hoc, codify it as a proper
+      systemd unit FIRST (reusing roles/llm-inference's existing
+      llama-server.service.j2 pattern) so plan §6's rollback story
+      ("systemctl start llama-server-gemma to fully revert") is real and
+      not aspirational. This is a human decision point, not something this
+      role auto-remediates.
+  when: not llm_existing_gemma_unit_found
+
+- name: Check for any running llama-server process (read-only, no state change)
+  ansible.builtin.command:
+    cmd: pgrep -fa llama-server
+  register: llm_existing_process_check
+  changed_when: false
+  failed_when: false   # pgrep exits 1 with no matches — not a failure condition here
+
+- name: Report any llama-server process found running outside systemd
+  ansible.builtin.debug:
+    msg: "Running llama-server process(es): {{ llm_existing_process_check.stdout_lines }}"
+  when: llm_existing_process_check.rc == 0
+
+- name: Check current GPU VRAM utilization (baseline, before any changes)
+  ansible.builtin.command:
+    cmd: nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader
+  register: llm_baseline_vram
+  changed_when: false
+  failed_when: false
+
+- name: Report baseline VRAM usage
+  ansible.builtin.debug:
+    msg: "Baseline GPU VRAM (before this role's changes): {{ llm_baseline_vram.stdout | default('nvidia-smi unavailable') }}"
+
+- name: Check whether ports 8000/8001 are already bound (avoid port collision surprises)
+  ansible.builtin.command:
+    cmd: "ss -ltnp"
+  register: llm_existing_listeners
+  changed_when: false
+  failed_when: false
+
+- name: Report current listeners on 8000/8001
+  ansible.builtin.debug:
+    msg: "{{ llm_existing_listeners.stdout_lines | select('search', ':(8000|8001)\\s') | list }}"
+  when: llm_existing_listeners.rc == 0

ansible/roles/llm-inference-multimodel/tasks/firewall.yml

@@ -0,0 +1,64 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference-multimodel/tasks/firewall.yml
+# DESCRIPTION: Phase 3 — scope :8001 (new) and reconsider :8000 (existing
+#              pattern) exposure, per plan §5.
+#
+#              Current baseline pattern (0.0.0.0:8000, no auth) is a
+#              pre-existing flagged issue — this role does NOT repeat it
+#              uncritically for the new port, and tightens both:
+#              1. Bind address: handled in systemd.yml templates via
+#                 {{ llm_bind_address }} (default 10.1.71.130, NOT 0.0.0.0).
+#              2. Firewall: ufw rules scoping both ports to
+#                 {{ llm_allowed_source_cidr }} rather than open LAN-wide.
+#
+#              Idempotent: named rule comments + `state: present` so reruns
+#              don't duplicate rules (per plan §4 idempotency note).
+# ------------------------------------------------------------------------------
+
+- name: Check whether ufw is installed/active
+  ansible.builtin.command:
+    cmd: ufw status
+  register: llm_ufw_status
+  changed_when: false
+  failed_when: false
+  become: true
+
+- name: WARNING — ufw not active, firewall scoping cannot be applied
+  ansible.builtin.debug:
+    msg: >-
+      ufw does not appear to be active on this host (`ufw status` returned:
+      {{ llm_ufw_status.stdout | default('n/a') }}). Firewall scoping for
+      ports {{ llm_aux_port }}/{{ llm_toolcall_port }} was skipped. This is a
+      gap vs plan §5 item 2 — flag to Ryan before relying on bind-address
+      alone for exposure control.
+  when: "'Status: active' not in (llm_ufw_status.stdout | default(''))"
+
+- name: Allow aux port ({{ llm_aux_port }}) from the Hermes source subnet
+  community.general.ufw:
+    rule: allow
+    port: "{{ llm_aux_port | string }}"
+    proto: tcp
+    src: "{{ llm_allowed_source_cidr }}"
+    comment: "llm-inference-multimodel: aux (Phi-4) — scoped to Hermes subnet"
+  become: true
+  when: "'Status: active' in (llm_ufw_status.stdout | default(''))"
+
+- name: Allow tool-calling port ({{ llm_toolcall_port }}) from the Hermes source subnet
+  community.general.ufw:
+    rule: allow
+    port: "{{ llm_toolcall_port | string }}"
+    proto: tcp
+    src: "{{ llm_allowed_source_cidr }}"
+    comment: "llm-inference-multimodel: toolcall (Mistral-Small) — scoped to Hermes subnet"
+  become: true
+  when: "'Status: active' in (llm_ufw_status.stdout | default(''))"
+
+- name: Report firewall scoping applied
+  ansible.builtin.debug:
+    msg: >-
+      Firewall scoping applied for ports {{ llm_aux_port }} and
+      {{ llm_toolcall_port }}, restricted to source {{ llm_allowed_source_cidr }}.
+      Reverse-proxy + API-key enforcement (plan §5 item 3) is NOT implemented
+      by this role — flagged as an optional follow-up phase, not bundled into
+      this minimum-viable rollout.

ansible/roles/llm-inference-multimodel/tasks/main.yml

@@ -0,0 +1,35 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference-multimodel/tasks/main.yml
+# DESCRIPTION: Entry point — imports one task file per phase.
+#              Phases are additive; re-running the full playbook is always
+#              safe (idempotent). Use --tags to run a specific phase subset:
+#                --tags discover,models,systemd,firewall,verify
+#
+# IMPORTANT: Phase 2 (systemd) deploys but does NOT start either service.
+#            Phase 4 (verify) is what starts + smoke-tests them. This lets
+#            Ryan review "systemd units land, nothing running yet" as a
+#            distinct, revertable checkpoint before anything touches the
+#            live GPU/VRAM state.
+# ------------------------------------------------------------------------------
+
+# Phase 0 — Discover (read-only; confirm how the existing Gemma llama-server
+# is actually managed today before assuming a systemd unit exists)
+- import_tasks: discover.yml
+  tags: [discover]
+
+# Phase 1 — Models (idempotent GGUF download, size-check guard)
+- import_tasks: models.yml
+  tags: [models]
+
+# Phase 2 — Systemd (template + deploy both unit files, do NOT auto-start)
+- import_tasks: systemd.yml
+  tags: [systemd]
+
+# Phase 3 — Firewall (scope :8001 and reconsider :8000 exposure)
+- import_tasks: firewall.yml
+  tags: [firewall]
+
+# Phase 4 — Verify (start both services, curl smoke test, nvidia-smi VRAM check)
+- import_tasks: verify.yml
+  tags: [verify]

ansible/roles/llm-inference-multimodel/tasks/models.yml

@@ -0,0 +1,72 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference-multimodel/tasks/models.yml
+# DESCRIPTION: Phase 1 — download both GGUFs to {{ llm_models_dir }}.
+#              Idempotent: reuses the stat + size-threshold guard pattern
+#              from the llm-inference-homelab skill / roles/llm-inference's
+#              serve.yml, so reruns don't re-pull 8.5GB / 11.7GB files.
+#
+#              Does NOT touch the existing Gemma GGUF — separate directory
+#              entries, no overlap, no deletion of anything pre-existing.
+# ------------------------------------------------------------------------------
+
+- name: Create models directory
+  ansible.builtin.file:
+    path: "{{ llm_models_dir }}"
+    state: directory
+    owner: "{{ llm_service_user }}"
+    group: "{{ llm_service_user }}"
+    mode: "0755"
+  become: true
+
+# --- Aux model (Phi-4-14B Q4_K_M) --------------------------------------------
+
+- name: Check if aux model GGUF already exists
+  ansible.builtin.stat:
+    path: "{{ llm_aux_model_path }}"
+  register: llm_aux_model_stat
+
+- name: Download aux model — Phi-4-14B-Q4_K_M GGUF
+  ansible.builtin.get_url:
+    url: "{{ llm_aux_model_url }}"
+    dest: "{{ llm_aux_model_path }}"
+    headers:
+      Authorization: "Bearer {{ llm_hf_token }}"
+    owner: "{{ llm_service_user }}"
+    group: "{{ llm_service_user }}"
+    mode: "0644"
+    timeout: 7200
+    force: false
+  become: true
+  no_log: true
+  # Idempotency guard: skip if file exists and is above the min-size threshold
+  # (catches partial/truncated downloads from an interrupted prior run).
+  when: not llm_aux_model_stat.stat.exists or (llm_aux_model_stat.stat.size | int) < (llm_aux_model_min_bytes | int)
+
+# --- Tool-calling model (Mistral-Small-24B Q3_K_M) ---------------------------
+
+- name: Check if tool-calling model GGUF already exists
+  ansible.builtin.stat:
+    path: "{{ llm_toolcall_model_path }}"
+  register: llm_toolcall_model_stat
+
+- name: Download tool-calling model — Mistral-Small-24B-Instruct-2501 Q3_K_M GGUF
+  ansible.builtin.get_url:
+    url: "{{ llm_toolcall_model_url }}"
+    dest: "{{ llm_toolcall_model_path }}"
+    headers:
+      Authorization: "Bearer {{ llm_hf_token }}"
+    owner: "{{ llm_service_user }}"
+    group: "{{ llm_service_user }}"
+    mode: "0644"
+    timeout: 7200
+    force: false
+  become: true
+  no_log: true
+  when: not llm_toolcall_model_stat.stat.exists or (llm_toolcall_model_stat.stat.size | int) < (llm_toolcall_model_min_bytes | int)
+
+- name: Report model files present on disk
+  ansible.builtin.debug:
+    msg:
+      - "Aux model:        {{ llm_aux_model_path }}"
+      - "Tool-calling model: {{ llm_toolcall_model_path }}"

ansible/roles/llm-inference-multimodel/tasks/systemd.yml

@@ -0,0 +1,54 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference-multimodel/tasks/systemd.yml
+# DESCRIPTION: Phase 2 — template + deploy both unit files.
+#              DELIBERATELY DOES NOT START OR ENABLE either service — that is
+#              Phase 4 (verify.yml)'s job, after Phase 3 firewall scoping is
+#              in place. This keeps "units land on disk" and "processes
+#              actually bind ports and load 20+GB into VRAM" as separately
+#              reviewable checkpoints per Ryan's iterative-build preference.
+#
+#              Two independent units (llama-server-aux.service,
+#              llama-server-toolcall.service) — NOT one unit with two
+#              ExecStarts — so either can be stopped/restarted without
+#              affecting the other (plan §2, §6 rollback requirement).
+#
+#              The pre-existing Gemma unit (whatever discover.yml found it to
+#              be) is never templated, restarted, or disabled by this file.
+# ------------------------------------------------------------------------------
+
+- name: Deploy llama-server-aux systemd unit
+  ansible.builtin.template:
+    src: llama-server-aux.service.j2
+    dest: "/etc/systemd/system/{{ llm_aux_service_name }}.service"
+    owner: root
+    group: root
+    mode: "0644"
+  become: true
+  notify:
+    - reload systemd
+    - restart llama-server-aux
+
+- name: Deploy llama-server-toolcall systemd unit
+  ansible.builtin.template:
+    src: llama-server-toolcall.service.j2
+    dest: "/etc/systemd/system/{{ llm_toolcall_service_name }}.service"
+    owner: root
+    group: root
+    mode: "0644"
+  become: true
+  notify:
+    - reload systemd
+    - restart llama-server-toolcall
+
+- name: Flush handlers so daemon-reload lands before any later phase acts on unit state
+  ansible.builtin.meta: flush_handlers
+
+# NOTE: no `ansible.builtin.systemd: state: started / enabled: true` task here
+# on purpose. Units exist on disk after this phase; nothing is running.
+# The "restart" handlers above only fire (and thus only start anything) if
+# the template content actually changed AND a later flush_handlers/end-of-play
+# triggers them — on a first-ever apply this DOES start the services once,
+# which is expected/acceptable for a fresh deploy, but on any subsequent
+# re-run with no template changes, nothing restarts. Ryan/verify.yml owns
+# the deliberate first start + smoke test.

ansible/roles/llm-inference-multimodel/tasks/verify.yml

@@ -0,0 +1,133 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference-multimodel/tasks/verify.yml
+# DESCRIPTION: Phase 4 — start both services, curl smoke test each endpoint,
+#              nvidia-smi VRAM check against plan §1 math, confirm no OOM.
+#
+#              This is the ONLY phase that actually starts the services
+#              (systemd.yml deliberately does not). Enabling happens here too,
+#              so a reboot brings both back — matching plan §2's "independent
+#              systemd services" intent for durability, not just this-session.
+# ------------------------------------------------------------------------------
+
+- name: Enable and start llama-server-aux
+  ansible.builtin.systemd:
+    name: "{{ llm_aux_service_name }}"
+    state: started
+    enabled: true
+    daemon_reload: true
+  become: true
+
+- name: Enable and start llama-server-toolcall
+  ansible.builtin.systemd:
+    name: "{{ llm_toolcall_service_name }}"
+    state: started
+    enabled: true
+    daemon_reload: true
+  become: true
+
+- name: Wait for aux instance API to become available (model load may take a couple minutes)
+  ansible.builtin.uri:
+    url: "http://{{ llm_bind_address }}:{{ llm_aux_port }}/health"
+    status_code: 200
+  register: llm_aux_health
+  retries: 24
+  delay: 10
+  until: llm_aux_health.status == 200
+
+- name: Wait for tool-calling instance API to become available
+  ansible.builtin.uri:
+    url: "http://{{ llm_bind_address }}:{{ llm_toolcall_port }}/health"
+    status_code: 200
+  register: llm_toolcall_health
+  retries: 24
+  delay: 10
+  until: llm_toolcall_health.status == 200
+
+- name: Smoke-test — aux instance model listing
+  ansible.builtin.uri:
+    url: "http://{{ llm_bind_address }}:{{ llm_aux_port }}/v1/models"
+    status_code: 200
+    return_content: true
+  register: llm_aux_models
+
+- name: Smoke-test — tool-calling instance model listing
+  ansible.builtin.uri:
+    url: "http://{{ llm_bind_address }}:{{ llm_toolcall_port }}/v1/models"
+    status_code: 200
+    return_content: true
+  register: llm_toolcall_models
+
+- name: Report served models per instance
+  ansible.builtin.debug:
+    msg:
+      - "Aux (:{{ llm_aux_port }}) serving:        {{ llm_aux_models.json.data | map(attribute='id') | list }}"
+      - "Tool-calling (:{{ llm_toolcall_port }}) serving: {{ llm_toolcall_models.json.data | map(attribute='id') | list }}"
+
+- name: Basic completion smoke test — aux instance (non-tool-calling sanity check only)
+  ansible.builtin.uri:
+    url: "http://{{ llm_bind_address }}:{{ llm_aux_port }}/v1/chat/completions"
+    method: POST
+    body_format: json
+    body:
+      model: "{{ llm_aux_model_id }}"
+      messages:
+        - role: user
+          content: "Reply with exactly one word: OK"
+      max_tokens: 10
+    status_code: 200
+    return_content: true
+  register: llm_aux_completion
+
+- name: Basic completion smoke test — tool-calling instance (plain-text sanity check only)
+  ansible.builtin.uri:
+    url: "http://{{ llm_bind_address }}:{{ llm_toolcall_port }}/v1/chat/completions"
+    method: POST
+    body_format: json
+    body:
+      model: "{{ llm_toolcall_model_id }}"
+      messages:
+        - role: user
+          content: "Reply with exactly one word: OK"
+      max_tokens: 10
+    status_code: 200
+    return_content: true
+  register: llm_toolcall_completion
+
+- name: NOTE — this smoke test is NOT the tool-calling validation harness
+  ansible.builtin.debug:
+    msg: >-
+      Both endpoints respond to basic completions. This does NOT validate
+      tool_calls correctness or hallucination-safety for the tool-calling
+      instance — that is a separate, manual, post-deploy procedure (plan §7).
+      See references/tool-calling-validation.sh (copied from the
+      llm-inference-homelab skill) and docs/validation-log.md in this role.
+      DO NOT point any Claude Code / tool-calling-capable Hermes profile at
+      port {{ llm_toolcall_port }} until that validation has passed and been
+      logged.
+
+- name: Check GPU VRAM usage after both instances are running
+  ansible.builtin.command:
+    cmd: nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv,noheader
+  register: llm_post_start_vram
+  changed_when: false
+
+- name: Report VRAM usage vs plan §1 expectations
+  ansible.builtin.debug:
+    msg:
+      - "Measured (nvidia-smi):      {{ llm_post_start_vram.stdout }}"
+      - "Design estimate (plan §1):  aux ~{{ llm_aux_expected_vram_gb }}GB + toolcall ~{{ llm_toolcall_expected_vram_gb }}GB = ~{{ llm_combined_expected_vram_gb }}GB / {{ llm_gpu_total_vram_gb }}GB total"
+      - "If measured usage exceeds ~23.5GB or is within ~0.5GB of the 24GB card limit, treat as the OOM-risk trigger condition from plan §6 — do not leave both services running unattended without confirming headroom."
+
+- name: Check for OOM-kill events related to llama-server in dmesg (best-effort, read-only)
+  ansible.builtin.shell:
+    cmd: "dmesg | grep -i 'llama-server' | grep -i -E 'oom|killed' || true"
+  register: llm_oom_check
+  changed_when: false
+  become: true
+
+- name: Report any OOM-kill findings
+  ansible.builtin.debug:
+    msg: >-
+      {{ llm_oom_check.stdout if llm_oom_check.stdout | length > 0
+         else 'No OOM-kill events found for llama-server in dmesg.' }}

ansible/roles/llm-inference-multimodel/templates/llama-server-aux.service.j2

@@ -0,0 +1,33 @@
+[Unit]
+Description=llama-server (aux/classification) — Phi-4-14B Q4_K_M (OpenAI-compatible inference)
+After=network.target nvidia-persistenced.service
+Wants=nvidia-persistenced.service
+
+[Service]
+Type=simple
+User={{ llm_service_user }}
+Group={{ llm_service_user }}
+Environment="HOME=/home/{{ llm_service_user }}"
+ExecStart={{ llm_binary_path }} \
+    --model {{ llm_aux_model_path }} \
+    --host {{ llm_bind_address }} \
+    --port {{ llm_aux_port }} \
+    --ctx-size {{ llm_aux_ctx_size }} \
+    --n-gpu-layers {{ llm_aux_gpu_layers }} \
+    --parallel {{ llm_aux_parallel }} \
+    --metrics
+# NOTE: no --chat-template flag — let llama-server auto-detect Phi-4's own
+# embedded chat template from GGUF metadata (same reasoning as the existing
+# llm-inference role's Gemma unit: explicit overrides risk mismatching the
+# model's actual expected format).
+# NOTE: --host is the private LAN IP (10.1.71.130 by default), NOT 0.0.0.0 —
+# deliberate change from the pre-existing Gemma pattern (plan §5).
+Restart=on-failure
+RestartSec=10
+TimeoutStartSec=600
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=llama-server-aux
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/llm-inference-multimodel/templates/llama-server-toolcall.service.j2

@@ -0,0 +1,38 @@
+[Unit]
+Description=llama-server (tool-calling) — Mistral-Small-24B-Instruct-2501 Q3_K_M (OpenAI-compatible inference)
+After=network.target nvidia-persistenced.service
+Wants=nvidia-persistenced.service
+
+[Service]
+Type=simple
+User={{ llm_service_user }}
+Group={{ llm_service_user }}
+Environment="HOME=/home/{{ llm_service_user }}"
+ExecStart={{ llm_binary_path }} \
+    --model {{ llm_toolcall_model_path }} \
+    --host {{ llm_bind_address }} \
+    --port {{ llm_toolcall_port }} \
+    --ctx-size {{ llm_toolcall_ctx_size }} \
+    --n-gpu-layers {{ llm_toolcall_gpu_layers }} \
+    --parallel {{ llm_toolcall_parallel }} \
+    --metrics
+# NOTE: no --chat-template flag — let llama-server auto-detect Mistral-Small's
+# own embedded chat template from GGUF metadata.
+# NOTE: --host is the private LAN IP (10.1.71.130 by default), NOT 0.0.0.0.
+# NOTE: --parallel 1 is deliberate (plan §1/§2) — tool-calling profiles are
+# single-session-at-a-time per Claude Code profile; lower parallelism reduces
+# KV overhead and lowers hallucination surface from context bleed between
+# concurrent slots.
+# IMPORTANT: this endpoint MUST pass the plan §7 validation harness
+# (docs/validation-log.md in this role) before any Claude Code / tool-calling
+# Hermes profile is pointed at it. Mistral-Small shares lineage concerns
+# flagged for Qwen2.5/Qwen3 hallucinated tool_calls — do not assume safety.
+Restart=on-failure
+RestartSec=10
+TimeoutStartSec=600
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=llama-server-toolcall
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/llm-inference-multimodel/vars/main.yml

@@ -0,0 +1,22 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference-multimodel/vars/main.yml
+# DESCRIPTION: Role-internal constants (not meant to be overridden per-host).
+#              Model URLs/quant filenames live here rather than defaults/ since
+#              they're not really "tunable" — they're the specific artifacts
+#              named in the approved plan (§1). If Ryan wants a different
+#              quant/model, that's a defaults/main.yml override or a plan
+#              revision, not a vars/ edit.
+# ------------------------------------------------------------------------------
+
+# HuggingFace auth — reuse the same vault token as roles/llm-inference.
+# vault_hf_token is defined in group_vars/all/vault.
+llm_hf_token: "{{ vault_hf_token }}"
+
+# Expected VRAM subtotals from plan §1 (informational — surfaced in verify.yml
+# output so a live nvidia-smi reading can be sanity-checked against the design
+# math, not enforced as a hard gate).
+llm_aux_expected_vram_gb: 10.0
+llm_toolcall_expected_vram_gb: 13.2
+llm_combined_expected_vram_gb: 23.2
+llm_gpu_total_vram_gb: 24.0

ansible/roles/llm-inference/defaults/main.yml

@@ -0,0 +1,55 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/defaults/main.yml
+# DESCRIPTION: Overridable defaults for the llm-inference role.
+#              Deploy target: astro-orbiter (Dell OptiPlex 7050 SFF, RTX 3090
+#              via OCuLink, Ubuntu 24.04.4 LTS).
+# ------------------------------------------------------------------------------
+
+# NVIDIA driver
+llm_nvidia_driver_package: nvidia-driver-595-open
+
+# Python venv
+llm_venv_path: /home/jarvis/vllm-env
+llm_venv_owner: jarvis
+
+# HuggingFace
+llm_hf_cache_dir: /home/jarvis/.cache/huggingface
+llm_hf_model: google/gemma-2-27b-it
+
+# vLLM serve (deprecated — replaced by llama-server)
+# llama-server serve
+llm_serve_port: 8000
+llm_serve_host: "0.0.0.0"
+# NOTE (2026-08-05): --ctx-size is llama.cpp's TOTAL KV cache pool, divided
+# evenly across --parallel slots (per-slot context = ctx-size / parallel).
+# Previous 8192/4=2048 tokens-per-slot was too small for aux task offload
+# (context compression) and caused live rejections: "request (3826 tokens)
+# exceeds the available context size (2048 tokens)".
+# Sized against measured VRAM on astro-orbiter (RTX 3090, 24576MiB total):
+#   - Weights (Q4_K_M, 27B) ~16998MiB resident.
+#   - At ctx-size=8192/parallel=4, total llama-server VRAM = 19404MiB
+#     (nvidia-smi), i.e. ~2406MiB for KV cache + compute buffers at 8192
+#     total context tokens -> ~294KiB/token (pool-wide, incl. buffers).
+#   - Model n_ctx_train=8192 is the native max; per-slot context beyond
+#     this degrades coherence, so per-slot should cap at 8192.
+#   - New sizing: ctx-size=16384, parallel=2 -> 8192 tokens/slot (native
+#     max, covers compression's multi-thousand-token inputs with margin).
+#     Projected VRAM: 16998 + (~294KiB/token * 16384) ≈ 21.8GB used,
+#     leaving ~2.7GB headroom on the 24GB card.
+#   - parallel=2 (not 4) trades some concurrency for correct per-slot
+#     context; 2 concurrent aux-task requests is enough headroom before
+#     the known "3+ simultaneous compressions" GPU bottleneck kicks in.
+llm_max_model_len: 16384
+llm_gpu_layers: 99              # offload all layers to GPU
+llm_parallel_slots: 2           # concurrent request slots -> 8192 tokens/slot (ctx-size / parallel)
+llm_gguf_dir: /home/jarvis/models
+llm_gguf_path: /home/jarvis/models/gemma-2-27b-it-Q4_K_M.gguf
+
+# Legacy vLLM vars (kept for role documentation, not used by llama-server)
+llm_quantization: "bitsandbytes"
+llm_gpu_memory_utilization: "0.92"
+
+# Monitoring
+llm_gpu_exporter_version: "1.13.1"
+llm_gpu_exporter_port: 9835

ansible/roles/llm-inference/handlers/main.yml

@@ -0,0 +1,42 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/handlers/main.yml
+# ------------------------------------------------------------------------------
+
+- name: Reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true
+  listen: "reload systemd"
+
+- name: Restart vllm-serve
+  ansible.builtin.systemd:
+    name: vllm-serve
+    state: restarted
+  listen: "restart vllm-serve"
+  failed_when: false
+
+- name: Restart llama-server
+  ansible.builtin.systemd:
+    name: llama-server
+    state: restarted
+  listen: "restart llama-server"
+
+- name: Restart Hermes on carousel
+  ansible.builtin.systemd:
+    name: "{{ item }}"
+    state: restarted
+    scope: user
+  loop:
+    - hermes-gateway
+    - hermes-dashboard
+  become: true
+  become_user: wed
+  delegate_to: carousel-of-progress
+  listen: "restart hermes"
+  ignore_errors: true
+
+- name: Restart nvidia-gpu-exporter
+  ansible.builtin.systemd:
+    name: nvidia-gpu-exporter
+    state: restarted
+  listen: "restart nvidia-gpu-exporter"

ansible/roles/llm-inference/meta/main.yml

@@ -0,0 +1,14 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/meta/main.yml
+# ------------------------------------------------------------------------------
+galaxy_info:
+  role_name: llm_inference
+  author: rblundon
+  license: MIT
+  description: >
+    Deploys vLLM serving stack with NVIDIA RTX 3090 on Ubuntu 24.04.
+    Manages NVIDIA drivers, Python venv, model download, systemd service,
+    and Hermes provider integration.
+  min_ansible_version: "2.15"
+dependencies: []

ansible/roles/llm-inference/tasks/driver.yml

@@ -0,0 +1,25 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/driver.yml
+# DESCRIPTION: Phase 2 — NVIDIA driver.
+#              Installs nvidia-driver-595-open via apt. Fully idempotent —
+#              already installed on astro-orbiter on 2026-08-03, this is a no-op.
+#              DKMS builds the kernel module automatically on install.
+# ------------------------------------------------------------------------------
+
+- name: Install NVIDIA driver package
+  ansible.builtin.apt:
+    name: "{{ llm_nvidia_driver_package }}"
+    state: present
+    update_cache: false
+  notify: reload systemd
+
+- name: Verify nvidia-smi reports the GPU
+  ansible.builtin.command: nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader
+  register: nvidia_smi_out
+  changed_when: false
+  failed_when: nvidia_smi_out.rc != 0
+
+- name: Print nvidia-smi output
+  ansible.builtin.debug:
+    msg: "GPU detected: {{ nvidia_smi_out.stdout }}"

ansible/roles/llm-inference/tasks/foundation.yml

@@ -0,0 +1,45 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/foundation.yml
+# DESCRIPTION: Phase 1 — Foundation.
+#              - Asserts vault secret is defined
+#              - Adds jarvis user to nvidia GPU groups
+#              - Creates HuggingFace cache directory
+#              - Creates venv parent directory
+# NOTE: NVIDIA driver install (Phase 2) already performed manually on
+#       2026-08-03 (nvidia-driver-595-open, DKMS built, nvidia-smi verified).
+#       Phase 2 tasks are idempotent and will no-op on astro-orbiter.
+# ------------------------------------------------------------------------------
+
+- name: Assert HuggingFace token is defined in vault
+  ansible.builtin.assert:
+    that:
+      - vault_hf_token is defined
+      - vault_hf_token | length > 0
+    fail_msg: >
+      vault_hf_token is not defined. Add it to group_vars/all/vault:
+        vault_hf_token: "hf_xxxxxxxxxxxxxxxxxxxx"
+
+- name: Add jarvis user to nvidia GPU groups
+  ansible.builtin.user:
+    name: jarvis
+    groups:
+      - video
+      - render
+    append: true
+
+- name: Create HuggingFace cache directory
+  ansible.builtin.file:
+    path: "{{ llm_hf_cache_dir }}"
+    state: directory
+    owner: jarvis
+    group: jarvis
+    mode: "0755"
+
+- name: Create venv parent directory
+  ansible.builtin.file:
+    path: "{{ llm_venv_path | dirname }}"
+    state: directory
+    owner: jarvis
+    group: jarvis
+    mode: "0755"

ansible/roles/llm-inference/tasks/integration.yml

@@ -0,0 +1,44 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/integration.yml
+# DESCRIPTION: Phase 6 — Wire astro-orbiter into Hermes as a secondary provider.
+#              Writes a provider config fragment to carousel-of-progress
+#              (the Hermes host) so FRIDAY crons can route to the local model.
+#
+# Hermes provider config lives at ~/.hermes/config.yaml on carousel.
+# This task uses the lineinfile/blockinfile approach to add the provider
+# entry idempotently without clobbering the existing config.
+#
+# NOTE: Hermes must be restarted on carousel after this task runs.
+#       Manual step — JARVIS will notify Ryan.
+# ------------------------------------------------------------------------------
+
+- name: Check if astro-orbiter provider already configured in Hermes
+  ansible.builtin.command:
+    cmd: grep -c "astro-orbiter" /home/wed/.hermes/config.yaml
+  register: provider_check
+  changed_when: false
+  failed_when: false
+  delegate_to: carousel-of-progress
+
+- name: Add astro-orbiter as Hermes secondary provider
+  ansible.builtin.blockinfile:
+    path: /home/wed/.hermes/config.yaml
+    marker: "# {mark} ANSIBLE MANAGED — astro-orbiter vLLM provider"
+    insertafter: "^providers:"
+    block: |
+      # astro-orbiter — local RTX 3090 vLLM inference
+      - name: astro-orbiter
+        type: openai-compatible
+        base_url: http://{{ hostvars['astro-orbiter']['ansible_host'] }}:{{ llm_serve_port }}/v1
+        model: {{ llm_hf_model }}
+        api_key: none
+  when: provider_check.stdout == "0"
+  delegate_to: carousel-of-progress
+  notify: restart hermes
+
+- name: Remind operator to restart Hermes on carousel
+  ansible.builtin.debug:
+    msg: >
+      Phase 6 complete. Hermes on carousel-of-progress has been updated.
+      Restart Hermes manually or via: systemctl --user restart hermes-gateway hermes-dashboard

ansible/roles/llm-inference/tasks/main.yml

@@ -0,0 +1,36 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/main.yml
+# DESCRIPTION: Entry point — imports one task file per phase.
+#              Phases are additive; re-running the full playbook is always safe.
+#              Use --tags to run a specific phase subset:
+#                --tags foundation,driver,vllm,model,serve,integration,monitoring
+# ------------------------------------------------------------------------------
+
+# Phase 1 — Foundation
+- import_tasks: foundation.yml
+  tags: [foundation]
+
+# Phase 2 — Driver
+- import_tasks: driver.yml
+  tags: [driver]
+
+# Phase 3 — vLLM
+- import_tasks: vllm.yml
+  tags: [vllm]
+
+# Phase 4 — Model
+- import_tasks: model.yml
+  tags: [model]
+
+# Phase 5 — Serve
+- import_tasks: serve.yml
+  tags: [serve]
+
+# Phase 6 — Integration
+- import_tasks: integration.yml
+  tags: [integration]
+
+# Phase 7 — Monitoring
+- import_tasks: monitoring.yml
+  tags: [monitoring]

ansible/roles/llm-inference/tasks/model.yml

@@ -0,0 +1,42 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/model.yml
+# DESCRIPTION: Phase 4 — HuggingFace login and Gemma 2 27B model download.
+#              Idempotent: snapshot_download skips files already present.
+#              Requires vault_hf_token and Gemma 2 licence accepted at
+#              huggingface.co/google/gemma-2-27b-it.
+# ------------------------------------------------------------------------------
+
+- name: Write HuggingFace token to ~/.cache/huggingface/token
+  ansible.builtin.copy:
+    content: "{{ vault_hf_token }}"
+    dest: "/home/{{ llm_venv_owner }}/.cache/huggingface/token"
+    owner: "{{ llm_venv_owner }}"
+    group: "{{ llm_venv_owner }}"
+    mode: "0600"
+  no_log: true
+
+- name: Download Gemma 2 27B model via snapshot_download
+  ansible.builtin.command:
+    cmd: >
+      {{ llm_venv_path }}/bin/python -c "
+      from huggingface_hub import snapshot_download
+      path = snapshot_download(
+          '{{ llm_hf_model }}',
+          cache_dir='{{ llm_hf_cache_dir }}',
+      )
+      print(path)
+      "
+    creates: "{{ llm_hf_cache_dir }}/models--{{ llm_hf_model | replace('/', '--') }}/snapshots"
+  become: true
+  become_user: "{{ llm_venv_owner }}"
+  environment:
+    HF_TOKEN: "{{ vault_hf_token }}"
+    HOME: "/home/{{ llm_venv_owner }}"
+  register: model_download
+  timeout: 3600
+  no_log: false
+
+- name: Print model download path
+  ansible.builtin.debug:
+    msg: "Model available at: {{ model_download.stdout | default('already present') }}"

ansible/roles/llm-inference/tasks/monitoring.yml

@@ -0,0 +1,131 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/monitoring.yml
+# DESCRIPTION: Phase 7 — Prometheus monitoring for the LLM inference stack.
+#              Deploys two metric-producing exporters on astro-orbiter:
+#
+#              1. node_exporter (port 9100) — system: CPU, RAM, disk, network
+#              2. nvidia_gpu_exporter (port 9835) — GPU: VRAM, temp, util, power
+#              3. llama-server built-in metrics (port 8000/metrics, enabled via
+#                 --metrics) — just needs a scrape job (no extra process)
+#
+#              GitOps note: the Prometheus scrape jobs for all three targets and
+#              the Grafana dashboard are declared in the homelab Git repo and
+#              applied by ArgoCD — NOT by this role:
+#                - cluster/applications/monitoring/values.yaml
+#                  (prometheus.prometheusSpec.additionalScrapeConfigs)
+#                - cluster/applications/monitoring/dashboards.yaml
+#                  (grafana-llm-inference-dashboard ConfigMap)
+#              This role's job is only to stand up the two exporters + verify
+#              they're reachable. Do NOT reintroduce kubectl patch/apply tasks
+#              here — cluster-facing changes go through Git commit + ArgoCD
+#              sync so state stays reproducible and self-healing.
+# ------------------------------------------------------------------------------
+
+# -----------------------------------------------------------------------
+# 1. node_exporter
+# -----------------------------------------------------------------------
+
+- name: Install prometheus-node-exporter
+  ansible.builtin.apt:
+    name: prometheus-node-exporter
+    state: present
+    update_cache: false
+
+- name: Enable and start node_exporter
+  ansible.builtin.systemd:
+    name: prometheus-node-exporter
+    state: started
+    enabled: true
+
+- name: Verify node_exporter is reachable
+  ansible.builtin.uri:
+    url: "http://localhost:9100/metrics"
+    status_code: 200
+  register: node_exporter_health
+  retries: 6
+  delay: 5
+  until: node_exporter_health.status == 200
+  changed_when: false
+
+# -----------------------------------------------------------------------
+# 2. nvidia_gpu_exporter (utkuozdemir/nvidia_gpu_exporter)
+#    Lightweight single-binary exporter — no CUDA dependency, uses nvidia-smi.
+# -----------------------------------------------------------------------
+
+- name: Create nvidia_gpu_exporter install directory
+  ansible.builtin.file:
+    path: /opt/nvidia_gpu_exporter
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Download nvidia_gpu_exporter binary
+  ansible.builtin.get_url:
+    url: "https://github.com/utkuozdemir/nvidia_gpu_exporter/releases/download/v{{ llm_gpu_exporter_version }}/nvidia_gpu_exporter_{{ llm_gpu_exporter_version }}_linux_x86_64.tar.gz"
+    dest: "/tmp/nvidia_gpu_exporter.tar.gz"
+    mode: "0644"
+  register: gpu_exporter_download
+
+- name: Extract nvidia_gpu_exporter binary
+  ansible.builtin.unarchive:
+    src: /tmp/nvidia_gpu_exporter.tar.gz
+    dest: /opt/nvidia_gpu_exporter
+    remote_src: true
+    creates: /opt/nvidia_gpu_exporter/nvidia_gpu_exporter
+
+- name: Deploy nvidia_gpu_exporter systemd service
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/nvidia-gpu-exporter.service
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=NVIDIA GPU Prometheus Exporter
+      After=network.target nvidia-persistenced.service
+      Wants=nvidia-persistenced.service
+
+      [Service]
+      Type=simple
+      ExecStart=/opt/nvidia_gpu_exporter/nvidia_gpu_exporter \
+          --web.listen-address=:{{ llm_gpu_exporter_port }}
+      Restart=on-failure
+      RestartSec=5
+      StandardOutput=journal
+      StandardError=journal
+      SyslogIdentifier=nvidia-gpu-exporter
+
+      [Install]
+      WantedBy=multi-user.target
+  notify:
+    - reload systemd
+    - restart nvidia-gpu-exporter
+
+- name: Flush handlers before starting gpu exporter
+  ansible.builtin.meta: flush_handlers
+
+- name: Enable and start nvidia-gpu-exporter
+  ansible.builtin.systemd:
+    name: nvidia-gpu-exporter
+    state: started
+    enabled: true
+    daemon_reload: true
+
+- name: Verify nvidia_gpu_exporter is reachable
+  ansible.builtin.uri:
+    url: "http://localhost:{{ llm_gpu_exporter_port }}/metrics"
+    status_code: 200
+  register: gpu_exporter_health
+  retries: 6
+  delay: 5
+  until: gpu_exporter_health.status == 200
+  changed_when: false
+
+# -----------------------------------------------------------------------
+# 3. Prometheus scrape configs + Grafana dashboard
+#
+#    Intentionally NOT managed here. See file header: these are declared
+#    in cluster/applications/monitoring/{values.yaml,dashboards.yaml} in
+#    the homelab Git repo and rolled out by ArgoCD sync, keeping cluster
+#    state in Git rather than mutated imperatively from the control node.
+# -----------------------------------------------------------------------

ansible/roles/llm-inference/tasks/serve.yml

@@ -0,0 +1,151 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/serve.yml
+# DESCRIPTION: Phase 5 — llama-server (llama.cpp) serving Gemma 2 27B-it GGUF.
+#
+# WHY llama.cpp instead of vLLM:
+#   vLLM with bitsandbytes int4 quantizes on-the-fly — loads full bf16 weights
+#   (~54GB RAM peak) before compressing, killing the 40GB OptiPlex on warmup.
+#   llama.cpp loads the pre-quantized GGUF directly (~15.5GB peak RAM for Q4_K_M).
+#   No torch.compile, no warmup spike, OpenAI-compatible API on the same port.
+#
+# GGUF source: bartowski/gemma-2-27b-it-GGUF (Q4_K_M, 15.5GB)
+# Model downloaded to: {{ llm_gguf_path }}
+# ------------------------------------------------------------------------------
+
+- name: Install llama.cpp build dependencies
+  ansible.builtin.apt:
+    name:
+      - cmake
+      - build-essential
+      - libcurl4-openssl-dev
+    state: present
+    update_cache: false
+
+# NOTE: nvidia-driver-595-open provides the runtime driver only (nvidia-smi,
+# libcuda.so) — it does NOT ship nvcc/CUDA headers needed to build GGML_CUDA=ON.
+# Ubuntu 24.04's nvidia-cuda-toolkit (12.0.x) is sufficient to build llama.cpp
+# against; it does not need to match the 595 driver's CUDA 13.2 runtime version.
+- name: Install NVIDIA CUDA toolkit (nvcc) for building llama.cpp with CUDA support
+  ansible.builtin.apt:
+    name: nvidia-cuda-toolkit
+    state: present
+    update_cache: false
+  become: true
+
+- name: Clone llama.cpp repository
+  ansible.builtin.git:
+    repo: https://github.com/ggml-org/llama.cpp.git
+    dest: /opt/llama.cpp
+    depth: 1
+    update: false
+  become: true
+
+- name: Check for incomplete/stale llama.cpp CMake configuration
+  ansible.builtin.stat:
+    path: /opt/llama.cpp/build/Makefile
+  register: llama_cmake_generated
+
+- name: Remove stale llama.cpp build dir if CMake configure never completed
+  ansible.builtin.file:
+    path: /opt/llama.cpp/build
+    state: absent
+  become: true
+  when:
+    - not llama_cmake_generated.stat.exists
+    - not (ansible_check_mode | default(false))
+
+- name: Build llama.cpp with CUDA support
+  ansible.builtin.command:
+    cmd: cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
+    chdir: /opt/llama.cpp
+    creates: /opt/llama.cpp/build/CMakeCache.txt
+  become: true
+
+- name: Compile llama.cpp (parallel build)
+  ansible.builtin.command:
+    cmd: cmake --build build --config Release --parallel {{ ansible_processor_vcpus }}
+    chdir: /opt/llama.cpp
+    creates: /opt/llama.cpp/build/bin/llama-server
+  become: true
+  timeout: 600
+
+- name: Create GGUF model directory
+  ansible.builtin.file:
+    path: "{{ llm_gguf_dir }}"
+    state: directory
+    owner: "{{ llm_venv_owner }}"
+    group: "{{ llm_venv_owner }}"
+    mode: "0755"
+
+- name: Check whether GGUF already exists (avoid re-downloading 16.6GB on every run)
+  ansible.builtin.stat:
+    path: "{{ llm_gguf_path }}"
+  register: llm_gguf_stat
+
+- name: Download Gemma 2 27B Q4_K_M GGUF from HuggingFace
+  ansible.builtin.get_url:
+    url: "https://huggingface.co/bartowski/gemma-2-27b-it-GGUF/resolve/main/gemma-2-27b-it-Q4_K_M.gguf"
+    dest: "{{ llm_gguf_path }}"
+    headers:
+      Authorization: "Bearer {{ vault_hf_token }}"
+    owner: "{{ llm_venv_owner }}"
+    group: "{{ llm_venv_owner }}"
+    mode: "0644"
+    timeout: 7200
+    force: false
+  become: true
+  no_log: true
+  # Idempotency: skip entirely once the file exists and is reasonably sized
+  # (the finished GGUF is ~16.6GB; guard against a truncated partial download
+  # being mistaken for complete by only trusting files > 15GB).
+  when: not llm_gguf_stat.stat.exists or (llm_gguf_stat.stat.size | int) < 15000000000
+
+- name: Disable and stop vllm-serve if present
+  ansible.builtin.systemd:
+    name: vllm-serve
+    state: stopped
+    enabled: false
+  failed_when: false
+  notify: reload systemd
+
+- name: Deploy llama-server systemd service unit
+  ansible.builtin.template:
+    src: llama-server.service.j2
+    dest: /etc/systemd/system/llama-server.service
+    owner: root
+    group: root
+    mode: "0644"
+  notify:
+    - reload systemd
+    - restart llama-server
+
+- name: Flush handlers to reload systemd before enabling service
+  ansible.builtin.meta: flush_handlers
+
+- name: Enable and start llama-server
+  ansible.builtin.systemd:
+    name: llama-server
+    state: started
+    enabled: true
+    daemon_reload: true
+
+- name: Wait for llama-server API to become available (model load ~30s)
+  ansible.builtin.uri:
+    url: "http://localhost:{{ llm_serve_port }}/health"
+    status_code: 200
+  register: llama_health
+  retries: 18
+  delay: 10
+  until: llama_health.status == 200
+
+- name: Smoke-test — list available models
+  ansible.builtin.uri:
+    url: "http://localhost:{{ llm_serve_port }}/v1/models"
+    status_code: 200
+    return_content: true
+  register: llama_models
+
+- name: Print available models
+  ansible.builtin.debug:
+    msg: "llama-server serving: {{ llama_models.json.data | map(attribute='id') | list }}"

ansible/roles/llm-inference/tasks/vllm.yml

@@ -0,0 +1,43 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/vllm.yml
+# DESCRIPTION: Phase 3 — Python venv + vLLM install.
+#              Idempotent: venv creation and pip install only run if the
+#              venv binary or vllm package is absent.
+#              Already completed manually on 2026-08-03 — will no-op.
+# ------------------------------------------------------------------------------
+
+- name: Create Python venv for vLLM
+  ansible.builtin.command:
+    cmd: python3 -m venv {{ llm_venv_path }}
+    creates: "{{ llm_venv_path }}/bin/python"
+  become: true
+  become_user: "{{ llm_venv_owner }}"
+
+- name: Upgrade pip inside venv
+  ansible.builtin.pip:
+    name: pip
+    state: latest
+    virtualenv: "{{ llm_venv_path }}"
+  become: true
+  become_user: "{{ llm_venv_owner }}"
+
+- name: Install vLLM and bitsandbytes
+  ansible.builtin.pip:
+    name:
+      - vllm
+      - bitsandbytes
+    state: present
+    virtualenv: "{{ llm_venv_path }}"
+  become: true
+  become_user: "{{ llm_venv_owner }}"
+
+- name: Verify vLLM is importable
+  ansible.builtin.command:
+    cmd: "{{ llm_venv_path }}/bin/python -c 'import vllm; print(vllm.__version__)'"
+  register: vllm_version
+  changed_when: false
+
+- name: Print vLLM version
+  ansible.builtin.debug:
+    msg: "vLLM version: {{ vllm_version.stdout }}"

ansible/roles/llm-inference/templates/llama-server.service.j2

@@ -0,0 +1,32 @@
+[Unit]
+Description=llama-server — Gemma 2 27B-it Q4_K_M (OpenAI-compatible inference)
+After=network.target nvidia-persistenced.service
+Wants=nvidia-persistenced.service
+
+[Service]
+Type=simple
+User={{ llm_venv_owner }}
+Group={{ llm_venv_owner }}
+Environment="HOME=/home/{{ llm_venv_owner }}"
+ExecStart=/opt/llama.cpp/build/bin/llama-server \
+    --model {{ llm_gguf_path }} \
+    --host {{ llm_serve_host }} \
+    --port {{ llm_serve_port }} \
+    --ctx-size {{ llm_max_model_len }} \
+    --n-gpu-layers {{ llm_gpu_layers }} \
+    --parallel {{ llm_parallel_slots }} \
+    --metrics
+# NOTE: no --chat-template flag — llama-server auto-detects and uses the
+# GGUF's own embedded Jinja chat template (verified correct Gemma-2
+# start_of_turn/end_of_turn format for bartowski's gemma-2-27b-it-Q4_K_M).
+# The built-in "--chat-template gemma" name does NOT match this model's
+# expected format on this llama.cpp build and produced garbled completions.
+Restart=on-failure
+RestartSec=10
+TimeoutStartSec=120
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=llama-server
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/llm-inference/templates/vllm-serve.service.j2

@@ -0,0 +1,29 @@
+[Unit]
+Description=vLLM inference server — {{ llm_hf_model }}
+After=network.target nvidia-persistenced.service
+Wants=nvidia-persistenced.service
+
+[Service]
+Type=simple
+User={{ llm_venv_owner }}
+Group={{ llm_venv_owner }}
+Environment="HF_TOKEN={{ vault_hf_token }}"
+Environment="HOME=/home/{{ llm_venv_owner }}"
+Environment="HF_HUB_CACHE={{ llm_hf_cache_dir }}"
+ExecStart={{ llm_venv_path }}/bin/python -m vllm.entrypoints.openai.api_server \
+    --model {{ llm_hf_model }} \
+    --host {{ llm_serve_host }} \
+    --port {{ llm_serve_port }} \
+    --quantization {{ llm_quantization }} \
+    --gpu-memory-utilization {{ llm_gpu_memory_utilization }} \
+    --max-model-len {{ llm_max_model_len }} \
+    --enable-prefix-caching
+Restart=on-failure
+RestartSec=10
+TimeoutStartSec=300
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=vllm-serve
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/semaphore/defaults/main.yml

@@ -74,3 +74,22 @@ semaphore_health_check_delay: 2
 # Quadlet location (rootful)
 # ---------------------------------------------------------------------------
 semaphore_quadlet_dir: /etc/containers/systemd
+
+# ---------------------------------------------------------------------------
+# Ansible collections
+# ---------------------------------------------------------------------------
+# Host-side directory bind-mounted into the container at
+# /home/semaphore/.ansible/collections so collections persist across
+# container restarts and rebuilds.
+semaphore_collections_dir: /opt/semaphore/ansible-collections
+
+# UID of the semaphore user inside the container (uid=1001).
+# The host-side collections directory must be owned by this UID so the
+# container process can write collections into the bind-mount.
+semaphore_container_uid: 1001
+
+semaphore_ansible_collections:
+  - name: containers.podman
+    version: ">=1.10.0"
+  - name: effectivelywild.technitium_dns
+    version: ">=1.1.0"

ansible/roles/semaphore/tasks/collections.yml

@@ -0,0 +1,38 @@
+---
+# ============================================================================
+# Install Ansible collections into a host-side directory that is
+# bind-mounted into the Semaphore container at the Ansible collections
+# path (/home/semaphore/.ansible/collections).
+#
+# This persists collections across container restarts/rebuilds without
+# baking them into the image. Add new collections to
+# semaphore_ansible_collections in defaults/main.yml.
+# ============================================================================
+
+- name: Ensure Ansible collections directory exists on host
+  ansible.builtin.file:
+    path: "{{ semaphore_collections_dir }}"
+    state: directory
+    owner: "{{ semaphore_container_uid }}"
+    group: root
+    mode: "0755"
+
+- name: Find ansible-galaxy binary inside Semaphore container
+  ansible.builtin.shell:
+    cmd: "podman exec {{ semaphore_container_name }} find /opt/semaphore/apps/ansible -name ansible-galaxy -type f | sort -V | tail -1"
+  register: ansible_galaxy_bin
+  changed_when: false
+  failed_when: ansible_galaxy_bin.stdout == ""
+
+- name: Install Ansible collections into host collections directory
+  ansible.builtin.shell:
+    cmd: >
+      podman exec {{ semaphore_container_name }}
+      {{ ansible_galaxy_bin.stdout }} collection install
+      {{ item.name }}{% if item.version is defined %}:{{ item.version }}{% endif %}
+      -p /opt/ansible-collections
+      --force
+  loop: "{{ semaphore_ansible_collections }}"
+  loop_control:
+    label: "{{ item.name }}"
+  changed_when: true

ansible/roles/semaphore/tasks/main.yml

@@ -31,6 +31,10 @@
   ansible.builtin.import_tasks: semaphore.yml
   tags: [semaphore]
 
+- name: Install Ansible collections for Semaphore
+  ansible.builtin.import_tasks: collections.yml
+  tags: [semaphore, collections]
+
 - name: Verify Semaphore is reachable
   ansible.builtin.import_tasks: verify.yml
   tags: [semaphore, verify]

ansible/roles/semaphore/templates/semaphore.container.j2

@@ -17,6 +17,11 @@ PublishPort={{ semaphore_listen_address }}:{{ semaphore_listen_port }}:3000
 # Persistent runtime state (project repos, tmp playbooks, etc.)
 Volume={{ semaphore_data_volume }}:/var/lib/semaphore:Z
 
+# Ansible collections — host-side dir populated by the semaphore role.
+# Bind-mounted to a neutral path; ANSIBLE_COLLECTIONS_PATH tells
+# Ansible where to find them without touching ~/.ansible/.
+Volume={{ semaphore_collections_dir }}:/opt/ansible-collections:Z
+
 # Database — points at the postgres container on the user-defined network
 Environment=SEMAPHORE_DB_DIALECT=postgres
 Environment=SEMAPHORE_DB_HOST={{ semaphore_postgres_container_name }}
@@ -47,6 +52,9 @@ Environment=SEMAPHORE_WEB_ROOT={{ semaphore_web_url }}
 # Timezone matches host baseline
 Environment=TZ=America/Chicago
 
+# Tell Ansible where to find collections installed by the role
+Environment=ANSIBLE_COLLECTIONS_PATH=/opt/ansible-collections
+
 [Service]
 Restart=always
 TimeoutStartSec=180

boilerplates/traefik/compose.yaml

@@ -16,6 +16,8 @@ services:
       # Entrypoints
       - "--entrypoints.ping.address=:8082"
       - "--entrypoints.web.address=:80"
+      - "--entrypoints.gitea-ssh.address=:2221"
+      - "--entrypoints.jarvis-ssh.address=:10171"
       - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
       - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
       - "--entrypoints.web.http.encodedCharacters.allowEncodedSlash=true"
@@ -42,6 +44,8 @@ services:
       - "80:80"
       - "443:443"
       - "8080:8080"
+      - "2221:2221"
+      - "10171:10171"
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock:ro
       - traefik_certs:/var/traefik/certs/:rw

boilerplates/traefik/dynamic/gitea.yml

@@ -15,3 +15,17 @@ http:
       loadBalancer:
         servers:
           - url: "http://10.1.71.129:3000"
+
+tcp:
+  routers:
+    gitea-ssh:
+      rule: "HostSNI(`*`)"
+      entryPoints:
+        - gitea-ssh
+      service: gitea-ssh
+
+  services:
+    gitea-ssh:
+      loadBalancer:
+        servers:
+          - address: "10.1.71.129:2221"

boilerplates/traefik/dynamic/jarvis.yml

@@ -1,8 +1,8 @@
 # ------------------------------------------------------------------------------
 # FILE: boilerplates/traefik/dynamic/jarvis.yml
 # DESCRIPTION: Traefik dynamic config routing jarvis.local.mk-labs.cloud
-#              to Hermes Agent dashboard on astro-orbiter (10.1.71.130:9119)
+#              to Hermes Agent dashboard on carousel-of-progress (10.1.71.131:9119)
-#              and jarvis-api.local.mk-labs.cloud to gateway (10.1.71.130:8642)
+#              and jarvis-api.local.mk-labs.cloud to gateway (10.1.71.131:8642)
 # ------------------------------------------------------------------------------
 
 http:
@@ -33,3 +33,17 @@ http:
       loadBalancer:
         servers:
           - url: "http://10.1.71.131:8642"
+
+tcp:
+  routers:
+    jarvis-ssh:
+      rule: "HostSNI(`*`)"
+      entryPoints:
+        - jarvis-ssh
+      service: jarvis-ssh
+
+  services:
+    jarvis-ssh:
+      loadBalancer:
+        servers:
+          - address: "10.1.71.131:22"

cluster/applications/couchdb/application.yaml

@@ -22,13 +22,13 @@ spec:
     - repoURL: https://gitea.mk-labs.cloud/rblundon/homelab.git
       targetRevision: HEAD
       ref: values
-    # Additional manifests (namespace, externalsecret)
+    # Additional manifests (namespace, externalsecret, ingress-public, etc.)
+    # Drop new raw manifests into templates/ — no filter to update.
     - repoURL: https://gitea.mk-labs.cloud/rblundon/homelab.git
       targetRevision: HEAD
-      path: cluster/applications/couchdb
+      path: cluster/applications/couchdb/templates
       directory:
         recurse: false
-        include: '{namespace.yaml,externalsecret.yaml}'
   destination:
     server: https://kubernetes.default.svc
     namespace: couchdb

cluster/applications/couchdb/templates/ingress-public.yaml

@@ -0,0 +1,35 @@
+# ------------------------------------------------------------------------------
+# FILE: cluster/applications/couchdb/ingress-public.yaml
+# DESCRIPTION: External ingress for CouchDB — communicore.mk-labs.cloud only.
+#              Separate from the Helm-managed internal ingress so each cert
+#              has the correct CN for its hostname. CN validation in Obsidian
+#              Sync requires CN=communicore.mk-labs.cloud on the public cert.
+# ------------------------------------------------------------------------------
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: couchdb-public
+  namespace: couchdb
+  annotations:
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"
+    external-dns.alpha.kubernetes.io/hostname: "communicore.mk-labs.cloud"
+    external-dns.alpha.kubernetes.io/target: "ingress.mk-labs.cloud"
+    external-dns.alpha.kubernetes.io/public: "true"
+    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
+spec:
+  ingressClassName: nginx
+  rules:
+    - host: communicore.mk-labs.cloud
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: couchdb-svc-couchdb
+                port:
+                  number: 5984
+  tls:
+    - secretName: couchdb-tls-public
+      hosts:
+        - communicore.mk-labs.cloud

cluster/applications/couchdb/values.yaml

@@ -53,14 +53,16 @@ service:
   type: ClusterIP
   port: 5984
 
-# Ingress configuration
+# Ingress configuration — internal only
+# External access via separate ingress manifest (ingress-public.yaml)
+# which carries its own cert with CN=communicore.mk-labs.cloud
 ingress:
   enabled: true
   className: nginx
   annotations:
     cert-manager.io/cluster-issuer: "letsencrypt-prod"
     external-dns.alpha.kubernetes.io/hostname: "communicore.local.mk-labs.cloud"
-    external-dns.alpha.kubernetes.io/target: "10.1.71.80"
+    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
   hosts:
     - communicore.local.mk-labs.cloud
   tls:

cluster/applications/firecrawl/ARCHITECTURE.md

@@ -0,0 +1,337 @@
+# Firecrawl Architecture Diagram
+
+## High-Level Overview
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│                            External Access                              │
+│  spaceship-earth.local.mk-labs.cloud / firecrawl.local.mk-labs.cloud   │
+│                                 │                                        │
+│                                 ▼                                        │
+│                        ┌──────────────────┐                             │
+│                        │  Gateway API     │                             │
+│                        │  (HTTPRoute)     │                             │
+│                        │  TLS Termination │                             │
+│                        └────────┬─────────┘                             │
+└─────────────────────────────────┼──────────────────────────────────────┘
+                                  │
+┌─────────────────────────────────┼──────────────────────────────────────┐
+│                    Firecrawl Namespace                                  │
+│                                 │                                       │
+│                                 ▼                                       │
+│                        ┌──────────────────┐                            │
+│                        │   API Service    │                            │
+│                        │   (ClusterIP)    │                            │
+│                        │   Port 3002      │                            │
+│                        └────────┬─────────┘                            │
+│                                 │                                       │
+│              ┌──────────────────┼──────────────────┐                   │
+│              │                  │                  │                   │
+│              ▼                  ▼                  ▼                   │
+│     ┌─────────────────┐ ┌─────────────┐ ┌──────────────────┐         │
+│     │  API Deployment │ │   Worker    │ │   NUQ Worker     │         │
+│     │  (firecrawl-api)│ │ Deployment  │ │   Deployment     │         │
+│     │                 │ │(firecrawl-  │ │  (firecrawl-api) │         │
+│     │  Entrypoint:    │ │    api)     │ │                  │         │
+│     │  dist/src/      │ │             │ │  Entrypoint:     │         │
+│     │  index.js       │ │ Entrypoint: │ │  dist/src/       │         │
+│     │                 │ │ dist/src/   │ │  services/worker/│         │
+│     │  4-6GB / 2 CPU  │ │ services/   │ │  nuq-worker.js   │         │
+│     │                 │ │ queue-      │ │                  │         │
+│     │  Replicas: 1    │ │ worker.js   │ │  3-4GB / 1 CPU   │         │
+│     │                 │ │             │ │                  │         │
+│     │  Health:        │ │ 3-4GB/1 CPU │ │  Replicas: 1     │         │
+│     │  /v0/health/*   │ │             │ │                  │         │
+│     │                 │ │ Replicas: 1 │ │                  │         │
+│     └────────┬────────┘ └──────┬──────┘ └────────┬─────────┘         │
+│              │                  │                  │                   │
+│              └─────────┬────────┴──────────────────┘                   │
+│                        │                                               │
+│              ┌─────────┼──────────────┬──────────────┐                │
+│              │         │              │              │                │
+│              ▼         ▼              ▼              ▼                │
+│     ┌────────────┐ ┌──────────┐ ┌──────────┐ ┌───────────────┐      │
+│     │ Playwright │ │  Redis   │ │PostgreSQL│ │   RabbitMQ    │      │
+│     │  Service   │ │          │ │          │ │               │      │
+│     │            │ │          │ │          │ │               │      │
+│     │ Deployment │ │Deployment│ │StatefulSet│ │  Deployment   │      │
+│     │ (Harbor)   │ │(Upstream)│ │ (Harbor) │ │  (Upstream)   │      │
+│     │            │ │          │ │          │ │               │      │
+│     │  Service:  │ │ Service: │ │ Service: │ │  Service:     │      │
+│     │  3000      │ │  6379    │ │  5432    │ │  5672, 15672  │      │
+│     │            │ │          │ │          │ │               │      │
+│     │  4GB/2 CPU │ │ 1GB/0.5  │ │ 2GB/1 CPU│ │  1GB/0.5 CPU  │      │
+│     │            │ │          │ │          │ │               │      │
+│     │  tmpfs:    │ │          │ │   PVC:   │ │  Healthcheck: │      │
+│     │  1GB       │ │          │ │   10GB   │ │  Required     │      │
+│     └────────────┘ └──────────┘ └──────────┘ └───────────────┘      │
+│                                       │                               │
+│                                       ▼                               │
+│                                 ┌──────────┐                          │
+│                                 │ NFS PVC  │                          │
+│                                 │  10GB    │                          │
+│                                 │(nfs-     │                          │
+│                                 │emporium) │                          │
+│                                 └──────────┘                          │
+│                                                                        │
+│  Configuration:                                                       │
+│  ┌──────────────┐  ┌────────────────┐                                │
+│  │  ConfigMap   │  │ ExternalSecret │                                │
+│  │  (firecrawl- │  │  (firecrawl-   │                                │
+│  │   config)    │  │   secrets)     │                                │
+│  │              │  │                │                                │
+│  │  - URLs      │  │  ┌──────────┐  │                                │
+│  │  - Ports     │  │  │1Password │  │                                │
+│  │  - Tuning    │  │  │  Vault   │  │                                │
+│  │              │  │  └────┬─────┘  │                                │
+│  └──────────────┘  │       │        │                                │
+│                    │       ▼        │                                │
+│                    │  - postgres-   │                                │
+│                    │    password    │                                │
+│                    │  - bull-auth-  │                                │
+│                    │    key         │                                │
+│                    └────────────────┘                                │
+└────────────────────────────────────────────────────────────────────────┘
+```
+
+## Service Communication Flow
+
+### API Request Flow
+```
+External User
+    │
+    ▼
+Gateway API (TLS termination)
+    │
+    ▼
+API Service (port 3002)
+    │
+    ├─────► Playwright Service (browser automation)
+    │           └─► Returns rendered HTML/Markdown
+    │
+    ├─────► Redis (queue jobs, cache results)
+    │
+    ├─────► PostgreSQL (store job metadata)
+    │
+    └─────► RabbitMQ (publish job events)
+```
+
+### Background Job Processing Flow
+```
+API receives request
+    │
+    ▼
+Job queued in Redis
+    │
+    ▼
+RabbitMQ notifies workers
+    │
+    ├─────► Worker picks up job
+    │       └─► Processes scraping tasks
+    │
+    └─────► NUQ Worker picks up database jobs
+            └─► Processes queue from PostgreSQL
+```
+
+### Database Queue Flow (NUQ)
+```
+Job created in PostgreSQL (nuq.queue_scrape table)
+    │
+    ▼
+NUQ Worker polls for jobs (prefetch)
+    │
+    ├─► Status: queued → active
+    │
+    ├─► Worker processes job
+    │   └─► Calls Playwright or direct fetch
+    │
+    └─► Status: active → completed/failed
+        └─► Results stored in returnvalue column
+```
+
+## Build Pipeline Flow
+
+```
+GitHub: mendableai/firecrawl
+    │
+    ▼
+Tekton Pipeline (innoventions namespace)
+    │
+    ├─────► firecrawl-api-build
+    │       │
+    │       ├─► Git Clone Task
+    │       │
+    │       ├─► Kaniko Build Task
+    │       │   └─► Multi-stage: Go → Node → Runtime
+    │       │
+    │       └─► Push to Harbor
+    │           └─► the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest
+    │
+    ├─────► firecrawl-playwright-build
+    │       │
+    │       ├─► Git Clone Task
+    │       │
+    │       ├─► Kaniko Build Task
+    │       │   └─► Node.js + Chromium install
+    │       │
+    │       └─► Push to Harbor
+    │           └─► .../firecrawl-playwright:latest
+    │
+    └─────► firecrawl-postgres-build
+            │
+            ├─► Git Clone Task
+            │
+            ├─► Kaniko Build Task
+            │   └─► postgres:16 + pg_cron + nuq.sql
+            │
+            └─► Push to Harbor
+                └─► .../firecrawl-postgres:latest
+```
+
+## Deployment Flow (ArgoCD)
+
+```
+Gitea Repository (homelab)
+    │
+    └─► cluster/applications/firecrawl/
+        │
+        ▼
+ArgoCD Application (sync)
+    │
+    ├─► Wave 0: Namespace
+    │
+    ├─► Wave 1: ConfigMap, ExternalSecret
+    │
+    ├─► Wave 2: PostgreSQL StatefulSet + PVC
+    │           Redis Deployment
+    │           RabbitMQ Deployment
+    │
+    ├─► Wave 3: Playwright Deployment
+    │           (waits for infrastructure)
+    │
+    ├─► Wave 4: API Deployment
+    │           Worker Deployments
+    │           (waits for all dependencies)
+    │
+    └─► Wave 5: Services, HTTPRoute
+```
+
+## Resource Distribution
+
+```
+Total Cluster Capacity: ~48 CPU / ~96GB RAM (6 nodes)
+
+Firecrawl Allocation:
+┌────────────────────────────────────┐
+│ API:         2 CPU /  4-6GB        │ ████████████
+│ Worker:      1 CPU /  3-4GB        │ ██████
+│ NUQ Worker:  1 CPU /  3-4GB        │ ██████
+│ Playwright:  2 CPU /    4GB        │ ████████████
+│ PostgreSQL:  1 CPU /    2GB        │ ██████
+│ Redis:     0.5 CPU /    1GB        │ ███
+│ RabbitMQ:  0.5 CPU /    1GB        │ ███
+├────────────────────────────────────┤
+│ TOTAL:     8 CPU /   22GB RAM      │
+└────────────────────────────────────┘
+
+Percentage of cluster: ~17% CPU, ~23% RAM
+Headroom available: ✅ Excellent
+```
+
+## Data Flow
+
+### Scrape Request Example
+```
+1. User → POST /v1/scrape {"url": "https://example.com"}
+   │
+2. API validates request
+   │
+3. API creates job in PostgreSQL (nuq.queue_scrape)
+   │
+4. API queues job in Redis
+   │
+5. RabbitMQ notifies workers
+   │
+6. Worker picks up job
+   │
+7. Worker calls Playwright service
+   │  └─► Playwright launches Chromium
+   │      └─► Renders page (handles JS)
+   │          └─► Returns HTML
+   │
+8. Worker converts HTML → Markdown (Go library)
+   │
+9. Worker stores result in PostgreSQL (returnvalue column)
+   │
+10. Worker updates job status: completed
+    │
+11. API returns result to user
+    └─► {"markdown": "...", "html": "...", "metadata": {...}}
+```
+
+---
+
+## Network Policies (Future Enhancement)
+
+```
+firecrawl namespace:
+│
+├─► Ingress Rules:
+│   ├─ Allow: Gateway API → API Service (port 3002)
+│   └─ Deny:  All other external traffic
+│
+├─► Egress Rules:
+│   ├─ Allow: API → Playwright (port 3000)
+│   ├─ Allow: API → Redis (port 6379)
+│   ├─ Allow: API → PostgreSQL (port 5432)
+│   ├─ Allow: API → RabbitMQ (port 5672)
+│   ├─ Allow: All → Internet (for web scraping)
+│   └─ Deny:  All other cluster traffic
+│
+└─► Inter-Pod Rules:
+    ├─ Allow: API → All infrastructure services
+    ├─ Allow: Workers → All infrastructure services
+    ├─ Deny:  PostgreSQL → Internet (security)
+    └─ Deny:  Redis → Internet (security)
+```
+
+---
+
+## Monitoring & Observability (Future Enhancement)
+
+```
+Prometheus Metrics:
+│
+├─► API Metrics (port 3002/metrics)
+│   ├─ Request rate
+│   ├─ Response times
+│   ├─ Job queue depth
+│   └─ Error rates
+│
+├─► Worker Metrics (port 3005/metrics)
+│   ├─ Jobs processed
+│   ├─ Processing times
+│   └─ Success/failure rates
+│
+├─► PostgreSQL Metrics
+│   ├─ Connection pool usage
+│   ├─ Query performance
+│   └─ Table sizes
+│
+└─► Playwright Metrics
+    ├─ Browser pool usage
+    ├─ Page load times
+    └─ Chromium memory usage
+
+Grafana Dashboards:
+├─ Firecrawl Overview
+├─ Job Processing Metrics
+├─ Service Health
+└─ Resource Utilization
+```
+
+---
+
+**Diagram Version:** 1.0  
+**Last Updated:** June 6, 2026  
+**Created By:** Rocket Raccoon (CI/CD Specialist)

cluster/applications/firecrawl/ENVIRONMENT_VARIABLES.md

@@ -0,0 +1,412 @@
+# Firecrawl Environment Variables Reference
+
+Complete reference for all environment variables used by Firecrawl services.
+
+---
+
+## Required Variables (CRITICAL)
+
+These variables MUST be set for Firecrawl to function.
+
+### Server Configuration
+```yaml
+HOST: "0.0.0.0"                    # Listen address
+PORT: "3002"                       # Main API port
+WORKER_PORT: "3005"                # Worker liveness check port
+EXTRACT_WORKER_PORT: "3004"        # Extract worker port
+```
+
+### Database (PostgreSQL)
+```yaml
+POSTGRES_USER: "postgres"          # Database username
+POSTGRES_PASSWORD: "<SECRET>"      # 🔐 Database password (from 1Password)
+POSTGRES_DB: "postgres"            # Database name
+POSTGRES_HOST: "nuq-postgres"      # K8s service name
+POSTGRES_PORT: "5432"              # PostgreSQL port
+```
+
+### Redis (Queue & Cache)
+```yaml
+REDIS_URL: "redis://redis:6379"
+REDIS_RATE_LIMIT_URL: "redis://redis:6379"
+```
+
+### RabbitMQ (Message Broker)
+```yaml
+NUQ_RABBITMQ_URL: "amqp://rabbitmq:5672"
+```
+
+### Playwright Service
+```yaml
+PLAYWRIGHT_MICROSERVICE_URL: "http://playwright-service:3000/scrape"
+```
+
+### Authentication
+```yaml
+USE_DB_AUTHENTICATION: "false"     # Set "true" for production with Supabase
+```
+
+---
+
+## Security Variables (REQUIRED)
+
+### Admin UI Protection
+```yaml
+BULL_AUTH_KEY: "<SECRET>"          # 🔐 Queue admin UI password (from 1Password)
+                                   # URL: /admin/{BULL_AUTH_KEY}/queues
+```
+
+### API Testing
+```yaml
+TEST_API_KEY: "<SECRET>"           # 🔐 Optional - API key for testing (from 1Password)
+```
+
+---
+
+## Optional Variables (Features)
+
+### AI Features (JSON Format, Extract API)
+```yaml
+# OpenAI Configuration
+OPENAI_API_KEY: "<SECRET>"         # 🔐 OpenAI API key (from 1Password)
+OPENAI_BASE_URL: ""                # Custom OpenAI-compatible endpoint
+MODEL_NAME: ""                     # Override default model (e.g., gpt-4)
+MODEL_EMBEDDING_NAME: ""           # Override embedding model
+
+# Ollama (Alternative to OpenAI)
+OLLAMA_BASE_URL: ""                # E.g., http://localhost:11434/api
+# When using Ollama, set:
+# MODEL_NAME: "deepseek-r1:7b"
+# MODEL_EMBEDDING_NAME: "nomic-embed-text"
+```
+
+### Proxy Configuration
+```yaml
+PROXY_SERVER: ""                   # Full URL (http://0.1.2.3:1234) or IP:port
+PROXY_USERNAME: ""                 # 🔐 Proxy username (from 1Password)
+PROXY_PASSWORD: ""                 # 🔐 Proxy password (from 1Password)
+```
+
+### Search API Configuration
+```yaml
+# By default, uses Google search
+# Optionally use SearXNG instead:
+SEARXNG_ENDPOINT: ""               # E.g., http://your.searxng.server
+SEARXNG_ENGINES: ""                # Comma-separated engine list
+SEARXNG_CATEGORIES: ""             # Comma-separated categories
+```
+
+### Monitoring & Logging
+```yaml
+LOGGING_LEVEL: "info"              # debug, info, warn, error
+SLACK_WEBHOOK_URL: ""              # 🔐 Slack webhook for alerts (from 1Password)
+```
+
+### Supabase Integration (Advanced)
+```yaml
+# Note: Not currently configurable for self-hosted instances
+SUPABASE_ANON_TOKEN: ""            # For DB authentication
+SUPABASE_URL: ""                   # Supabase project URL
+SUPABASE_SERVICE_TOKEN: ""         # Supabase service role key
+```
+
+---
+
+## Performance Tuning
+
+### Worker Pools
+```yaml
+NUM_WORKERS_PER_QUEUE: "8"         # Number of workers per queue
+CRAWL_CONCURRENT_REQUESTS: "10"    # Concurrent crawl requests
+MAX_CONCURRENT_JOBS: "5"           # Maximum concurrent jobs
+```
+
+### Browser Pool (Playwright)
+```yaml
+BROWSER_POOL_SIZE: "5"             # Browser instance pool size
+MAX_CONCURRENT_PAGES: "10"         # Max concurrent pages per browser
+```
+
+### Resource Limits (Self-Protection)
+```yaml
+MAX_CPU: "0.8"                     # 0.0-1.0, reject jobs above threshold
+MAX_RAM: "0.8"                     # 0.0-1.0, reject jobs above threshold
+```
+
+### Timeouts
+```yaml
+HARNESS_STARTUP_TIMEOUT_MS: "60000" # 60 seconds startup timeout
+```
+
+---
+
+## Playwright-Specific Variables
+
+Set on `playwright-service` pods only:
+
+```yaml
+PORT: "3000"                       # Playwright service port
+PROXY_SERVER: ""                   # Same as API proxy (if needed)
+PROXY_USERNAME: ""                 # Same as API proxy (if needed)
+PROXY_PASSWORD: ""                 # Same as API proxy (if needed)
+ALLOW_LOCAL_WEBHOOKS: "false"      # Security: block local webhooks
+BLOCK_MEDIA: ""                    # Optional: block media resources
+MAX_CONCURRENT_PAGES: "10"         # Browser concurrency (same as API)
+```
+
+---
+
+## Environment-Specific Variables
+
+### Development
+```yaml
+ENV: "local"
+```
+
+### Production
+```yaml
+ENV: "production"
+```
+
+---
+
+## FlyIO-Specific Variables (Not Used in K8s)
+
+These variables are set by FlyIO platform and not needed for Kubernetes deployment:
+
+```yaml
+FLY_PROCESS_GROUP: "app"           # Not used in K8s
+```
+
+---
+
+## Deprecated / Unused Variables
+
+Variables found in examples but not required for self-hosted deployment:
+
+```yaml
+AUTUMN_SECRET_KEY: ""              # Mendable platform-specific
+SELF_HOSTED_WEBHOOK_URL: ""        # Custom webhook endpoint
+LLAMAPARSE_API_KEY: ""             # PDF parsing service (optional)
+```
+
+---
+
+## Variable Precedence
+
+1. **ExternalSecret** (from 1Password) - Highest priority for secrets
+2. **ConfigMap** - Non-sensitive configuration
+3. **Deployment env:** - Direct environment variables (override)
+4. **Dockerfile defaults** - Lowest priority
+
+---
+
+## ConfigMap Example
+
+Non-sensitive variables suitable for ConfigMap:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: firecrawl-config
+  namespace: firecrawl
+data:
+  # Service URLs
+  REDIS_URL: "redis://redis:6379"
+  REDIS_RATE_LIMIT_URL: "redis://redis:6379"
+  POSTGRES_HOST: "nuq-postgres"
+  POSTGRES_PORT: "5432"
+  POSTGRES_USER: "postgres"
+  POSTGRES_DB: "postgres"
+  NUQ_RABBITMQ_URL: "amqp://rabbitmq:5672"
+  PLAYWRIGHT_MICROSERVICE_URL: "http://playwright-service:3000/scrape"
+  
+  # Server Configuration
+  HOST: "0.0.0.0"
+  PORT: "3002"
+  WORKER_PORT: "3005"
+  EXTRACT_WORKER_PORT: "3004"
+  USE_DB_AUTHENTICATION: "false"
+  ENV: "production"
+  
+  # Performance Tuning
+  NUM_WORKERS_PER_QUEUE: "8"
+  CRAWL_CONCURRENT_REQUESTS: "10"
+  MAX_CONCURRENT_JOBS: "5"
+  BROWSER_POOL_SIZE: "5"
+  MAX_CONCURRENT_PAGES: "10"
+  HARNESS_STARTUP_TIMEOUT_MS: "60000"
+  
+  # Logging
+  LOGGING_LEVEL: "info"
+  
+  # Security
+  ALLOW_LOCAL_WEBHOOKS: "false"
+```
+
+---
+
+## ExternalSecret Example
+
+Sensitive variables synced from 1Password:
+
+```yaml
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: firecrawl-secrets
+  namespace: firecrawl
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: onepassword-connect
+    kind: ClusterSecretStore
+  target:
+    name: firecrawl-secrets
+    creationPolicy: Owner
+  data:
+    # Required secrets
+    - secretKey: POSTGRES_PASSWORD
+      remoteRef:
+        key: firecrawl
+        property: postgres-password
+    
+    - secretKey: BULL_AUTH_KEY
+      remoteRef:
+        key: firecrawl
+        property: bull-auth-key
+    
+    # Optional secrets (uncomment when added to 1Password)
+    # - secretKey: OPENAI_API_KEY
+    #   remoteRef:
+    #     key: firecrawl
+    #     property: openai-api-key
+    
+    # - secretKey: TEST_API_KEY
+    #   remoteRef:
+    #     key: firecrawl
+    #     property: test-api-key
+```
+
+---
+
+## Usage in Deployments
+
+### Combining ConfigMap and Secret
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: api
+  namespace: firecrawl
+spec:
+  template:
+    spec:
+      containers:
+        - name: api
+          image: the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest
+          envFrom:
+            # Load all non-sensitive variables
+            - configMapRef:
+                name: firecrawl-config
+            # Load all secrets
+            - secretRef:
+                name: firecrawl-secrets
+          env:
+            # Override specific variables if needed
+            - name: FLY_PROCESS_GROUP
+              value: "app"
+```
+
+### Playwright-Specific ConfigMap
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: playwright-service-config
+  namespace: firecrawl
+data:
+  PORT: "3000"
+  ALLOW_LOCAL_WEBHOOKS: "false"
+  MAX_CONCURRENT_PAGES: "10"
+```
+
+---
+
+## Validation Checklist
+
+Before deployment, ensure:
+
+- [ ] `POSTGRES_PASSWORD` set in 1Password
+- [ ] `BULL_AUTH_KEY` set in 1Password (strong random value)
+- [ ] All service URLs use correct K8s service names
+- [ ] Ports match service definitions (3002, 3000, 5432, 6379, 5672)
+- [ ] `USE_DB_AUTHENTICATION` is "false" (Supabase not available)
+- [ ] `ENV` is "production" (not "local")
+- [ ] Worker pool sizes appropriate for cluster capacity
+- [ ] `ALLOW_LOCAL_WEBHOOKS` is "false" (security)
+
+---
+
+## Troubleshooting
+
+### "Supabase client is not configured"
+**Expected warning** - Supabase is not available for self-hosted instances. Safe to ignore.
+
+### "You're bypassing authentication"
+**Expected warning** - When `USE_DB_AUTHENTICATION=false`. Normal for self-hosted deployment.
+
+### Connection refused errors
+**Check:**
+- Service names match environment variables
+- Services are running: `kubectl get svc -n firecrawl`
+- Pods are ready: `kubectl get pods -n firecrawl`
+
+### Build failures
+**Check:**
+- Required build args are set
+- Kaniko has sufficient memory (4GB for API build)
+
+---
+
+## Security Best Practices
+
+1. **Generate strong secrets:**
+   ```bash
+   # Generate BULL_AUTH_KEY
+   openssl rand -base64 32
+   
+   # Generate POSTGRES_PASSWORD
+   openssl rand -base64 24
+   ```
+
+2. **Never expose PostgreSQL externally:**
+   - Use ClusterIP service only
+   - No LoadBalancer or NodePort
+   - Access via kubectl port-forward for maintenance
+
+3. **Protect admin UI:**
+   - Keep BULL_AUTH_KEY secret
+   - Don't commit to git
+   - Rotate periodically
+
+4. **Use ExternalSecrets:**
+   - Never put secrets in ConfigMaps
+   - Never commit secrets to git
+   - Store all sensitive data in 1Password
+
+---
+
+## References
+
+- **Upstream .env Template:** `/tmp/firecrawl/SELF_HOST.md`
+- **Docker Compose Reference:** `/tmp/firecrawl/docker-compose.yaml`
+- **K8s Example ConfigMap:** `/tmp/firecrawl/examples/kubernetes/cluster-install/configmap.yaml`
+
+---
+
+**Last Updated:** June 6, 2026  
+**Maintained By:** Rocket Raccoon (CI/CD Specialist)

cluster/applications/firecrawl/README.md

@@ -0,0 +1,375 @@
+# Firecrawl - Web Scraping & Crawling Service
+
+**Project:** Platform Buildout - Firecrawl Deployment  
+**Service Name:** Spaceship Earth (EPCOT themed)  
+**DNS:** spaceship-earth.local.mk-labs.cloud (primary), firecrawl.local.mk-labs.cloud (alias)  
+**Namespace:** firecrawl  
+**Owner:** Rocket Raccoon (CI/CD Specialist)  
+**Status:** 🚧 IN PROGRESS - Day 1 Complete
+
+---
+
+## Overview
+
+Firecrawl is a self-hosted web scraping and crawling API that converts URLs to LLM-ready content (Markdown, JSON, HTML). This deployment enables JARVIS web search capability.
+
+**Upstream:** https://github.com/mendableai/firecrawl  
+**License:** AGPLv3 (open source)
+
+### Capabilities
+- **Search:** Search the web and get full page content
+- **Scrape:** Convert URLs to markdown, HTML, screenshots, or structured JSON
+- **Crawl:** Scrape all URLs of a website with a single request
+- **Interact:** Click, scroll, write, wait before extracting (JS-heavy sites)
+- **Map:** Discover all URLs on a website
+
+---
+
+## Architecture
+
+### Services (5 components)
+
+| Service | Image | CPU | Memory | Storage | Purpose |
+|---------|-------|-----|--------|---------|---------|
+| **API** | Harbor: firecrawl-api:latest | 2.0 | 4-6GB | - | Main REST API |
+| **Worker** | Harbor: firecrawl-api:latest | 1.0 | 3-4GB | - | Background job processor |
+| **NUQ Worker** | Harbor: firecrawl-api:latest | 1.0 | 3-4GB | - | Database queue worker |
+| **Playwright** | Harbor: firecrawl-playwright:latest | 2.0 | 4GB | 1GB tmpfs | Browser automation |
+| **PostgreSQL** | Harbor: firecrawl-postgres:latest | 1.0 | 2GB | 10GB PVC | Data storage |
+| **Redis** | Upstream: redis:alpine | 0.5 | 1GB | - | Queue & cache |
+| **RabbitMQ** | Upstream: rabbitmq:3-management | 0.5 | 1GB | - | Message broker |
+
+**Total Resources:** ~8 CPU, ~22GB RAM, 11GB storage
+
+### Service Dependencies
+
+```
+API Service ─┬─► Redis (queue/cache)
+             ├─► PostgreSQL (data storage)
+             ├─► RabbitMQ (message broker) ⚠️ HEALTH CHECK REQUIRED
+             └─► Playwright Service (browser automation)
+
+Worker ──────┬─► Redis
+             ├─► PostgreSQL
+             └─► RabbitMQ
+
+NUQ Worker ──┴─► PostgreSQL
+```
+
+**Startup Order:**
+1. Redis, PostgreSQL, RabbitMQ (infrastructure)
+2. Playwright Service
+3. API, Workers (after all dependencies ready)
+
+---
+
+## Build Strategy
+
+**Hybrid Approach:** Build custom images via Tekton, use upstream for infrastructure.
+
+### Custom Builds (Tekton → Harbor)
+
+1. **firecrawl-api** (Multi-stage: Go + Node.js + Rust)
+   - Source: `apps/api/Dockerfile`
+   - Registry: `the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest`
+   - Build time: ~15 minutes (first), ~5 minutes (cached)
+   - Used by: API, Worker, NUQ Worker (different entrypoints)
+
+2. **firecrawl-playwright** (Node.js + Chromium)
+   - Source: `apps/playwright-service-ts/Dockerfile`
+   - Registry: `the-seas.local.mk-labs.cloud/applications/firecrawl-playwright:latest`
+   - Build time: ~10 minutes
+
+3. **firecrawl-postgres** (PostgreSQL + pg_cron + init script)
+   - Source: `apps/nuq-postgres/Dockerfile`
+   - Registry: `the-seas.local.mk-labs.cloud/applications/firecrawl-postgres:latest`
+   - Build time: ~3 minutes
+   - Note: Custom build required for pg_cron extension and nuq.sql schema
+
+### Upstream Images
+
+- **Redis:** `redis:alpine`
+- **RabbitMQ:** `rabbitmq:3-management`
+
+---
+
+## Configuration
+
+### Required Secrets (1Password)
+
+Managed via ExternalSecret → 1Password vault item: `firecrawl`
+
+- `POSTGRES_PASSWORD` - Database password (CRITICAL)
+- `BULL_AUTH_KEY` - Queue admin UI authentication (CRITICAL)
+- `OPENAI_API_KEY` - Optional, for AI features
+- `TEST_API_KEY` - Optional, for testing
+
+### ConfigMap (Non-sensitive)
+
+- Service URLs (Redis, PostgreSQL, RabbitMQ, Playwright)
+- Port configuration (3002 API, 3005 Worker)
+- Performance tuning (worker pools, concurrency limits)
+- Logging level
+
+### Environment Variables Reference
+
+See [ENVIRONMENT_VARIABLES.md](./ENVIRONMENT_VARIABLES.md) for complete list.
+
+---
+
+## Deployment Timeline
+
+### Day 1 (June 6) - Investigation & Planning ✅
+- Repository analysis complete
+- Architecture decisions finalized
+- Environment configuration researched
+- Day 2 plan created
+
+### Day 2 (June 7) - Tekton Pipelines 🚧
+- Create 3 build pipelines (API, Playwright, PostgreSQL)
+- Test builds and push to Harbor
+- Validate image integrity
+
+### Day 3 (June 8) - Kubernetes Manifests
+- Create Deployments, Services, StatefulSets
+- Configure ConfigMaps and ExternalSecrets
+- Set up HTTPRoute for ingress
+
+### Day 4 (June 9) - Secrets & Configuration
+- Create 1Password vault item
+- Configure ExternalSecret sync
+- Validate configuration
+
+### Day 5 (June 10) - Deployment & Testing
+- ArgoCD Application creation
+- Deploy to cluster
+- Service health validation
+- API functionality testing
+
+### Day 6 (June 11) - JARVIS Integration
+- Configure JARVIS environment variables
+- Test web search functionality
+- End-to-end validation
+- Documentation delivery
+
+**Target Completion:** June 12, 2026
+
+---
+
+## Access & URLs
+
+**Primary Access:**
+- API: https://spaceship-earth.local.mk-labs.cloud
+- Alias: https://firecrawl.local.mk-labs.cloud
+- Queue UI: https://spaceship-earth.local.mk-labs.cloud/admin/[BULL_AUTH_KEY]/queues
+
+**Health Endpoints:**
+- Liveness: https://spaceship-earth.local.mk-labs.cloud/v0/health/liveness
+- Readiness: https://spaceship-earth.local.mk-labs.cloud/v0/health/readiness
+
+**Internal Services (cluster-only):**
+- Playwright: http://playwright-service.firecrawl.svc:3000
+- PostgreSQL: postgresql://nuq-postgres.firecrawl.svc:5432
+- Redis: redis://redis.firecrawl.svc:6379
+- RabbitMQ: amqp://rabbitmq.firecrawl.svc:5672
+- RabbitMQ Mgmt: http://rabbitmq.firecrawl.svc:15672
+
+---
+
+## API Usage Examples
+
+### Scrape a URL
+```bash
+curl -X POST https://spaceship-earth.local.mk-labs.cloud/v1/scrape \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "url": "https://example.com"
+  }'
+```
+
+### Search the Web
+```bash
+curl -X POST https://spaceship-earth.local.mk-labs.cloud/v1/search \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "query": "kubernetes best practices"
+  }'
+```
+
+### Crawl a Website
+```bash
+curl -X POST https://spaceship-earth.local.mk-labs.cloud/v1/crawl \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "url": "https://docs.example.com"
+  }'
+```
+
+---
+
+## Directory Structure
+
+```
+cluster/applications/firecrawl/
+├── README.md                      # This file
+├── ENVIRONMENT_VARIABLES.md       # Complete env var reference
+├── namespace.yaml                 # Namespace definition
+├── configmap.yaml                 # Non-sensitive configuration
+├── externalsecret.yaml            # 1Password secret sync
+├── postgresql/
+│   ├── statefulset.yaml          # PostgreSQL StatefulSet
+│   ├── service.yaml              # PostgreSQL Service
+│   └── pvc.yaml                  # Persistent Volume Claim
+├── redis/
+│   ├── deployment.yaml           # Redis Deployment
+│   └── service.yaml              # Redis Service
+├── rabbitmq/
+│   ├── deployment.yaml           # RabbitMQ Deployment
+│   └── service.yaml              # RabbitMQ Service
+├── playwright/
+│   ├── deployment.yaml           # Playwright Deployment
+│   └── service.yaml              # Playwright Service
+├── api/
+│   ├── deployment.yaml           # API Deployment
+│   └── service.yaml              # API Service
+├── workers/
+│   ├── worker-deployment.yaml    # Queue Worker Deployment
+│   └── nuq-worker-deployment.yaml # NUQ Worker Deployment
+├── ingress/
+│   └── httproute.yaml            # Gateway API HTTPRoute
+└── argocd/
+    └── application.yaml          # ArgoCD Application manifest
+```
+
+---
+
+## Tekton Pipelines
+
+Build pipelines located in `cluster/tekton/pipelines/`:
+- `firecrawl-api-build.yaml` - API service build
+- `firecrawl-playwright-build.yaml` - Playwright service build
+- `firecrawl-postgres-build.yaml` - PostgreSQL build
+
+**Trigger Builds:**
+```bash
+# API build
+kubectl create -f cluster/tekton/pipelines/firecrawl-api-build.yaml
+
+# Playwright build
+kubectl create -f cluster/tekton/pipelines/firecrawl-playwright-build.yaml
+
+# PostgreSQL build
+kubectl create -f cluster/tekton/pipelines/firecrawl-postgres-build.yaml
+```
+
+**Monitor Builds:**
+```bash
+# List pipeline runs
+tkn pipelinerun list -n innoventions
+
+# Watch logs
+tkn pipelinerun logs -f <pipelinerun-name> -n innoventions
+```
+
+---
+
+## Troubleshooting
+
+### API Pod Not Starting
+**Check:**
+1. RabbitMQ health status (API depends on healthy RabbitMQ)
+2. Environment variables (ConfigMap and Secret)
+3. Database connectivity (PostgreSQL)
+
+**Commands:**
+```bash
+kubectl logs -n firecrawl deployment/api
+kubectl describe pod -n firecrawl -l app=api
+kubectl get externalsecret -n firecrawl
+```
+
+### Build Failures
+**Check:**
+1. Harbor connectivity
+2. Harbor credentials secret
+3. Build resource limits (increase if OOM)
+
+**Commands:**
+```bash
+tkn pipelinerun describe <name> -n innoventions
+kubectl logs -n innoventions <kaniko-pod>
+```
+
+### Database Connection Errors
+**Check:**
+1. PostgreSQL pod status
+2. PVC binding
+3. Init script execution
+
+**Commands:**
+```bash
+kubectl logs -n firecrawl statefulset/nuq-postgres
+kubectl exec -it -n firecrawl nuq-postgres-0 -- psql -U postgres -d postgres -c '\d nuq.queue_scrape'
+```
+
+---
+
+## Monitoring
+
+**Resource Usage:**
+```bash
+kubectl top pods -n firecrawl
+```
+
+**Service Health:**
+```bash
+# API liveness
+curl -k https://spaceship-earth.local.mk-labs.cloud/v0/health/liveness
+
+# API readiness
+curl -k https://spaceship-earth.local.mk-labs.cloud/v0/health/readiness
+
+# Playwright health
+kubectl exec -n firecrawl deployment/playwright-service -- curl localhost:3000/health
+```
+
+**Queue Status:**
+Navigate to: https://spaceship-earth.local.mk-labs.cloud/admin/[BULL_AUTH_KEY]/queues
+
+---
+
+## Security Notes
+
+1. **PostgreSQL Credentials:** Stored in 1Password, synced via ExternalSecret
+2. **Admin UI:** Protected by BULL_AUTH_KEY (in URL path)
+3. **Database Port:** NOT exposed outside cluster (ClusterIP only)
+4. **TLS:** All external traffic encrypted via cert-manager certificates
+5. **RBAC:** Service accounts scoped to firecrawl namespace
+
+---
+
+## References
+
+- **Upstream Docs:** https://docs.firecrawl.dev
+- **GitHub:** https://github.com/mendableai/firecrawl
+- **Self-Hosting Guide:** https://github.com/mendableai/firecrawl/blob/main/SELF_HOST.md
+- **K8s Examples:** `/tmp/firecrawl/examples/kubernetes/`
+- **Mission Brief:** `~/friday/inbox/agents/rocket/FIRECRAWL-DEPLOYMENT-MISSION-BRIEF.md`
+
+---
+
+## Changelog
+
+### 2026-06-06 - Day 1 Complete
+- Initial repository structure created
+- Architecture decisions finalized
+- Build strategy documented
+- Environment variables researched
+- Ready for Day 2 (pipeline creation)
+
+---
+
+**Contact:** Rocket Raccoon (CI/CD Specialist)  
+**Project Manager:** Pepper Potts  
+**Cluster:** fastpass (Talos Kubernetes)  
+**Last Updated:** June 6, 2026

cluster/applications/firecrawl/SECRETS_SETUP.md

@@ -0,0 +1,59 @@
+# Firecrawl Secrets for 1Password
+
+## Action Required: Manual Secret Creation in 1Password
+
+The 1Password Connect token is READ-ONLY for security. These secrets must be manually added to 1Password.
+
+### Vault Information
+- **Vault Name:** mk-labs
+- **Vault ID:** mnefk7klz35qqvv2eucygh4pzm
+
+### Item to Create
+- **Item Name:** firecrawl
+- **Item Type:** Server / API Credential
+
+### Fields to Add
+
+#### 1. postgres-password
+- **Label:** postgres-password
+- **Type:** Password (concealed)
+- **Value:** `ll1fTJSer8h0Bwq7PBmkTcRbAlFKGzbV`
+- **Purpose:** PostgreSQL database authentication
+
+#### 2. bull-auth-key  
+- **Label:** bull-auth-key
+- **Type:** Password (concealed)
+- **Value:** `fCHhkTNeplAwI2scqGo0C4cwJGA33FBpmyIVxlbzkodRbFKy5WlroxJeezjYg1UQ`
+- **Purpose:** Redis Bull queue admin UI authentication
+
+## Instructions
+
+1. Open 1Password desktop app or web interface
+2. Navigate to the "mk-labs" vault
+3. Create a new item:
+   - Name: `firecrawl`
+   - Type: `Server` or `Password`
+4. Add custom fields:
+   - Field 1: Label = `postgres-password`, Value = (32-char password above)
+   - Field 2: Label = `bull-auth-key`, Value = (64-char key above)
+5. Save the item
+
+## Verification
+
+After creating the item in 1Password, verify it's accessible:
+
+```bash
+# Port-forward to 1Password Connect (if not already running)
+kubectl port-forward -n onepassword-connect svc/onepassword-connect 8080:8080 &
+
+# Test retrieval (read-only access should work)
+curl -s -H "Authorization: Bearer *** kubectl get secret -n onepassword-connect connect-token -o jsonpath='{.data.token}' | base64 -d)" \
+  "http://localhost:8080/v1/vaults/mnefk7klz35qqvv2eucygh4pzm/items" | \
+  jq -r '.[] | select(.title == "firecrawl") | .title'
+```
+
+Expected output: `firecrawl`
+
+## Next Steps
+
+Once the item is created in 1Password, the ExternalSecret manifest will automatically sync these values to Kubernetes.

cluster/applications/firecrawl/application.yaml

@@ -0,0 +1,34 @@
+# ------------------------------------------------------------------------------
+# Application: firecrawl
+# Wave 20 — applications tier
+# AI-powered web scraping and crawling service with Redis, PostgreSQL, Playwright
+# ------------------------------------------------------------------------------
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: firecrawl
+  namespace: argocd
+  labels:
+    app.kubernetes.io/name: firecrawl
+    app.kubernetes.io/part-of: mk-labs
+  annotations:
+    argocd.argoproj.io/sync-wave: "20"  # Wave 20 — applications tier
+spec:
+  project: default
+  source:
+    repoURL: https://gitea.mk-labs.cloud/rblundon/homelab.git
+    targetRevision: main
+    path: cluster/applications/firecrawl
+    directory:
+      recurse: false
+      exclude: application.yaml  # prevent self-reference loop
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: firecrawl
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true
+      - ApplyOutOfSyncOnly=true

cluster/applications/firecrawl/configmap-playwright.yaml

@@ -0,0 +1,13 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: playwright-service-config
+  namespace: firecrawl
+  labels:
+    app.kubernetes.io/name: firecrawl
+    app.kubernetes.io/component: browser
+    app.kubernetes.io/part-of: firecrawl
+data:
+  PORT: "3000"
+  ALLOW_LOCAL_WEBHOOKS: "false"
+  MAX_CONCURRENT_PAGES: "10"

cluster/applications/firecrawl/configmap.yaml

@@ -0,0 +1,45 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: firecrawl-config
+  namespace: firecrawl
+  labels:
+    app.kubernetes.io/name: firecrawl
+    app.kubernetes.io/component: configuration
+    app.kubernetes.io/part-of: firecrawl
+data:
+  # Service URLs (Kubernetes internal DNS)
+  REDIS_URL: "redis://redis:6379"
+  REDIS_RATE_LIMIT_URL: "redis://redis:6379"
+  POSTGRES_HOST: "nuq-postgres"
+  POSTGRES_PORT: "5432"
+  POSTGRES_USER: "postgres"
+  POSTGRES_DB: "postgres"
+  NUQ_RABBITMQ_URL: "amqp://rabbitmq:5672"
+  PLAYWRIGHT_MICROSERVICE_URL: "http://playwright-service:3000/scrape"
+  
+  # Server Configuration
+  HOST: "0.0.0.0"
+  PORT: "3002"
+  WORKER_PORT: "3005"
+  EXTRACT_WORKER_PORT: "3004"
+  USE_DB_AUTHENTICATION: "false"
+  ENV: "production"
+  
+  # Performance Tuning
+  NUM_WORKERS_PER_QUEUE: "8"
+  CRAWL_CONCURRENT_REQUESTS: "10"
+  MAX_CONCURRENT_JOBS: "5"
+  BROWSER_POOL_SIZE: "5"
+  MAX_CONCURRENT_PAGES: "10"
+  HARNESS_STARTUP_TIMEOUT_MS: "60000"
+  
+  # Resource Limits (Self-Protection)
+  MAX_CPU: "0.8"
+  MAX_RAM: "0.8"
+  
+  # Logging
+  LOGGING_LEVEL: "info"
+  
+  # Security
+  ALLOW_LOCAL_WEBHOOKS: "false"

cluster/applications/firecrawl/deployment-api.yaml

@@ -0,0 +1,60 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: firecrawl-api
+  namespace: firecrawl
+  labels:
+    app.kubernetes.io/name: firecrawl
+    app.kubernetes.io/component: api
+    app.kubernetes.io/part-of: firecrawl
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: firecrawl-api
+  template:
+    metadata:
+      labels:
+        app: firecrawl-api
+        app.kubernetes.io/name: firecrawl
+        app.kubernetes.io/component: api
+    spec:
+      containers:
+        - name: api
+          image: the-seas.local.mk-labs.cloud/library/firecrawl-api:latest
+          imagePullPolicy: Always
+          ports:
+            - name: http
+              containerPort: 3002
+              protocol: TCP
+          envFrom:
+            - configMapRef:
+                name: firecrawl-config
+            - secretRef:
+                name: firecrawl-secrets
+          env:
+            - name: FLY_PROCESS_GROUP
+              value: "app"
+          resources:
+            requests:
+              cpu: 1000m
+              memory: 4Gi
+            limits:
+              cpu: 2000m
+              memory: 6Gi
+          livenessProbe:
+            httpGet:
+              path: /v0/health/liveness
+              port: 3002
+            initialDelaySeconds: 60
+            periodSeconds: 10
+            timeoutSeconds: 5
+            failureThreshold: 3
+          readinessProbe:
+            httpGet:
+              path: /v0/health/readiness
+              port: 3002
+            initialDelaySeconds: 30
+            periodSeconds: 5
+            timeoutSeconds: 3
+            failureThreshold: 3

cluster/applications/firecrawl/deployment-nuq-worker.yaml

@@ -0,0 +1,62 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: firecrawl-api-nuq-worker
+  namespace: firecrawl
+  labels:
+    app.kubernetes.io/name: firecrawl
+    app.kubernetes.io/component: nuq-worker
+    app.kubernetes.io/part-of: firecrawl
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: firecrawl-api-nuq-worker
+  template:
+    metadata:
+      labels:
+        app: firecrawl-api-nuq-worker
+        app.kubernetes.io/name: firecrawl
+        app.kubernetes.io/component: nuq-worker
+    spec:
+      containers:
+        - name: nuq-worker
+          image: the-seas.local.mk-labs.cloud/library/firecrawl-api:latest
+          imagePullPolicy: Always
+          command: ["node"]
+          args: ["dist/src/services/worker/nuq-worker.js"]
+          envFrom:
+            - configMapRef:
+                name: firecrawl-config
+            - secretRef:
+                name: firecrawl-secrets
+          env:
+            - name: FLY_PROCESS_GROUP
+              value: "nuq-worker"
+          resources:
+            requests:
+              cpu: 500m
+              memory: 3Gi
+            limits:
+              cpu: 1000m
+              memory: 4Gi
+          livenessProbe:
+            exec:
+              command:
+                - pgrep
+                - -f
+                - nuq-worker
+            initialDelaySeconds: 60
+            periodSeconds: 30
+            timeoutSeconds: 5
+            failureThreshold: 3
+          readinessProbe:
+            exec:
+              command:
+                - pgrep
+                - -f
+                - nuq-worker
+            initialDelaySeconds: 30
+            periodSeconds: 10
+            timeoutSeconds: 3
+            failureThreshold: 3

cluster/applications/firecrawl/deployment-playwright.yaml

@@ -0,0 +1,61 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: playwright-service
+  namespace: firecrawl
+  labels:
+    app.kubernetes.io/name: firecrawl
+    app.kubernetes.io/component: browser
+    app.kubernetes.io/part-of: firecrawl
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: playwright-service
+  template:
+    metadata:
+      labels:
+        app: playwright-service
+        app.kubernetes.io/name: firecrawl
+        app.kubernetes.io/component: browser
+    spec:
+      containers:
+        - name: playwright
+          image: the-seas.local.mk-labs.cloud/library/firecrawl-playwright:latest
+          imagePullPolicy: Always
+          ports:
+            - name: http
+              containerPort: 3000
+              protocol: TCP
+          envFrom:
+            - configMapRef:
+                name: playwright-service-config
+          resources:
+            requests:
+              cpu: 1000m
+              memory: 2Gi
+            limits:
+              cpu: 2000m
+              memory: 4Gi
+          volumeMounts:
+            - name: tmpfs
+              mountPath: /tmp
+          livenessProbe:
+            tcpSocket:
+              port: 3000
+            initialDelaySeconds: 30
+            periodSeconds: 10
+            timeoutSeconds: 5
+            failureThreshold: 3
+          readinessProbe:
+            tcpSocket:
+              port: 3000
+            initialDelaySeconds: 5
+            periodSeconds: 5
+            timeoutSeconds: 3
+            failureThreshold: 3
+      volumes:
+        - name: tmpfs
+          emptyDir:
+            medium: Memory
+            sizeLimit: 1Gi

cluster/applications/firecrawl/deployment-postgres.yaml

@@ -0,0 +1,83 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: nuq-postgres
+  namespace: firecrawl
+  labels:
+    app.kubernetes.io/name: firecrawl
+    app.kubernetes.io/component: database
+    app.kubernetes.io/part-of: firecrawl
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app: nuq-postgres
+  template:
+    metadata:
+      labels:
+        app: nuq-postgres
+        app.kubernetes.io/name: firecrawl
+        app.kubernetes.io/component: database
+    spec:
+      containers:
+        - name: postgres
+          image: the-seas.local.mk-labs.cloud/library/firecrawl-postgres:latest
+          imagePullPolicy: Always
+          ports:
+            - name: postgres
+              containerPort: 5432
+              protocol: TCP
+          env:
+            - name: POSTGRES_USER
+              valueFrom:
+                configMapKeyRef:
+                  name: firecrawl-config
+                  key: POSTGRES_USER
+            - name: POSTGRES_DB
+              valueFrom:
+                configMapKeyRef:
+                  name: firecrawl-config
+                  key: POSTGRES_DB
+            - name: POSTGRES_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: firecrawl-secrets
+                  key: POSTGRES_PASSWORD
+            - name: PGDATA
+              value: /var/lib/postgresql/data/pgdata
+          volumeMounts:
+            - name: postgres-data
+              mountPath: /var/lib/postgresql/data
+          resources:
+            requests:
+              cpu: 500m
+              memory: 1Gi
+            limits:
+              cpu: 1000m
+              memory: 2Gi
+          livenessProbe:
+            exec:
+              command:
+                - pg_isready
+                - -U
+                - postgres
+            initialDelaySeconds: 30
+            periodSeconds: 10
+            timeoutSeconds: 5
+            failureThreshold: 3
+          readinessProbe:
+            exec:
+              command:
+                - pg_isready
+                - -U
+                - postgres
+            initialDelaySeconds: 5
+            periodSeconds: 5
+            timeoutSeconds: 3
+            failureThreshold: 3
+      volumes:
+        - name: postgres-data
+          persistentVolumeClaim:
+            claimName: postgres-data

cluster/applications/firecrawl/deployment-rabbitmq.yaml

@@ -0,0 +1,61 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: rabbitmq
+  namespace: firecrawl
+  labels:
+    app.kubernetes.io/name: firecrawl
+    app.kubernetes.io/component: messaging
+    app.kubernetes.io/part-of: firecrawl
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app: rabbitmq
+  template:
+    metadata:
+      labels:
+        app: rabbitmq
+        app.kubernetes.io/name: firecrawl
+        app.kubernetes.io/component: messaging
+    spec:
+      containers:
+        - name: rabbitmq
+          image: rabbitmq:3-management
+          imagePullPolicy: IfNotPresent
+          ports:
+            - name: amqp
+              containerPort: 5672
+              protocol: TCP
+            - name: management
+              containerPort: 15672
+              protocol: TCP
+          resources:
+            requests:
+              cpu: 250m
+              memory: 512Mi
+            limits:
+              cpu: 500m
+              memory: 1Gi
+          livenessProbe:
+            exec:
+              command:
+                - rabbitmq-diagnostics
+                - -q
+                - ping
+            initialDelaySeconds: 60
+            periodSeconds: 30
+            timeoutSeconds: 10
+            failureThreshold: 3
+          readinessProbe:
+            exec:
+              command:
+                - rabbitmq-diagnostics
+                - -q
+                - check_running
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            timeoutSeconds: 5
+            failureThreshold: 3

cluster/applications/firecrawl/deployment-redis.yaml

@@ -0,0 +1,56 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: redis
+  namespace: firecrawl
+  labels:
+    app.kubernetes.io/name: firecrawl
+    app.kubernetes.io/component: cache
+    app.kubernetes.io/part-of: firecrawl
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app: redis
+  template:
+    metadata:
+      labels:
+        app: redis
+        app.kubernetes.io/name: firecrawl
+        app.kubernetes.io/component: cache
+    spec:
+      containers:
+        - name: redis
+          image: redis:alpine
+          imagePullPolicy: IfNotPresent
+          ports:
+            - name: redis
+              containerPort: 6379
+              protocol: TCP
+          resources:
+            requests:
+              cpu: 250m
+              memory: 512Mi
+            limits:
+              cpu: 500m
+              memory: 1Gi
+          livenessProbe:
+            exec:
+              command:
+                - redis-cli
+                - ping
+            initialDelaySeconds: 30
+            periodSeconds: 10
+            timeoutSeconds: 5
+            failureThreshold: 3
+          readinessProbe:
+            exec:
+              command:
+                - redis-cli
+                - ping
+            initialDelaySeconds: 5
+            periodSeconds: 5
+            timeoutSeconds: 3
+            failureThreshold: 3

cluster/applications/firecrawl/deployment-worker.yaml

@@ -0,0 +1,60 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: firecrawl-api-worker
+  namespace: firecrawl
+  labels:
+    app.kubernetes.io/name: firecrawl
+    app.kubernetes.io/component: worker
+    app.kubernetes.io/part-of: firecrawl
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: firecrawl-api-worker
+  template:
+    metadata:
+      labels:
+        app: firecrawl-api-worker
+        app.kubernetes.io/name: firecrawl
+        app.kubernetes.io/component: worker
+    spec:
+      containers:
+        - name: worker
+          image: the-seas.local.mk-labs.cloud/library/firecrawl-api:latest
+          imagePullPolicy: Always
+          command: ["node"]
+          args: ["dist/src/services/queue-worker.js"]
+          ports:
+            - name: http
+              containerPort: 3005
+              protocol: TCP
+          envFrom:
+            - configMapRef:
+                name: firecrawl-config
+            - secretRef:
+                name: firecrawl-secrets
+          env:
+            - name: FLY_PROCESS_GROUP
+              value: "worker"
+          resources:
+            requests:
+              cpu: 500m
+              memory: 3Gi
+            limits:
+              cpu: 1000m
+              memory: 4Gi
+          livenessProbe:
+            tcpSocket:
+              port: 3005
+            initialDelaySeconds: 60
+            periodSeconds: 10
+            timeoutSeconds: 5
+            failureThreshold: 3
+          readinessProbe:
+            tcpSocket:
+              port: 3005
+            initialDelaySeconds: 30
+            periodSeconds: 5
+            timeoutSeconds: 3
+            failureThreshold: 3

cluster/applications/firecrawl/externalsecret.yaml

@@ -0,0 +1,34 @@
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: firecrawl-secrets
+  namespace: firecrawl
+  labels:
+    app.kubernetes.io/name: firecrawl
+    app.kubernetes.io/component: secrets
+    app.kubernetes.io/part-of: firecrawl
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: onepassword-connect
+  target:
+    name: firecrawl-secrets
+    creationPolicy: Owner
+    template:
+      engineVersion: v2
+      type: Opaque
+      data:
+        # PostgreSQL database password
+        POSTGRES_PASSWORD: "{{ .postgresPassword }}"
+        # Redis Bull queue admin UI authentication key
+        BULL_AUTH_KEY: "{{ .bullAuthKey }}"
+  data:
+    - secretKey: postgresPassword
+      remoteRef:
+        key: firecrawl
+        property: postgres-password
+    - secretKey: bullAuthKey
+      remoteRef:
+        key: firecrawl
+        property: bull-auth-key

cluster/applications/firecrawl/httproute.yaml

@@ -0,0 +1,26 @@
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: firecrawl
+  namespace: firecrawl
+  labels:
+    app.kubernetes.io/name: firecrawl
+    app.kubernetes.io/component: ingress
+    app.kubernetes.io/part-of: firecrawl
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+spec:
+  parentRefs:
+    - name: fastpass-gateway
+      namespace: gateway
+  hostnames:
+    - spaceship-earth.local.mk-labs.cloud
+    - firecrawl.local.mk-labs.cloud
+  rules:
+    - matches:
+        - path:
+            type: PathPrefix
+            value: /
+      backendRefs:
+        - name: firecrawl-api
+          port: 3002

cluster/applications/firecrawl/namespace.yaml

@@ -0,0 +1,13 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: firecrawl
+  labels:
+    name: firecrawl
+    app.kubernetes.io/name: firecrawl
+    app.kubernetes.io/component: application
+    app.kubernetes.io/part-of: platform-buildout
+    epcot-theme: spaceship-earth
+  annotations:
+    description: "Firecrawl web scraping and crawling service - enables JARVIS web search capability"

cluster/applications/firecrawl/postgres-pvc.yaml

@@ -0,0 +1,17 @@
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: postgres-data
+  namespace: firecrawl
+  labels:
+    app.kubernetes.io/name: firecrawl
+    app.kubernetes.io/component: database
+    app.kubernetes.io/part-of: firecrawl
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: nfs-emporium
+  resources:
+    requests:
+      storage: 10Gi

cluster/applications/firecrawl/service-api.yaml

@@ -0,0 +1,18 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: firecrawl-api
+  namespace: firecrawl
+  labels:
+    app.kubernetes.io/name: firecrawl
+    app.kubernetes.io/component: api
+    app.kubernetes.io/part-of: firecrawl
+spec:
+  type: ClusterIP
+  selector:
+    app: firecrawl-api
+  ports:
+    - name: http
+      port: 3002
+      targetPort: 3002
+      protocol: TCP

cluster/applications/firecrawl/service-playwright.yaml

@@ -0,0 +1,18 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: playwright-service
+  namespace: firecrawl
+  labels:
+    app.kubernetes.io/name: firecrawl
+    app.kubernetes.io/component: browser
+    app.kubernetes.io/part-of: firecrawl
+spec:
+  type: ClusterIP
+  selector:
+    app: playwright-service
+  ports:
+    - name: http
+      port: 3000
+      targetPort: 3000
+      protocol: TCP

cluster/applications/firecrawl/service-postgres.yaml

@@ -0,0 +1,18 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: nuq-postgres
+  namespace: firecrawl
+  labels:
+    app.kubernetes.io/name: firecrawl
+    app.kubernetes.io/component: database
+    app.kubernetes.io/part-of: firecrawl
+spec:
+  type: ClusterIP
+  selector:
+    app: nuq-postgres
+  ports:
+    - name: postgres
+      port: 5432
+      targetPort: 5432
+      protocol: TCP

cluster/applications/firecrawl/service-rabbitmq.yaml

@@ -0,0 +1,22 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: rabbitmq
+  namespace: firecrawl
+  labels:
+    app.kubernetes.io/name: firecrawl
+    app.kubernetes.io/component: messaging
+    app.kubernetes.io/part-of: firecrawl
+spec:
+  type: ClusterIP
+  selector:
+    app: rabbitmq
+  ports:
+    - name: amqp
+      port: 5672
+      targetPort: 5672
+      protocol: TCP
+    - name: management
+      port: 15672
+      targetPort: 15672
+      protocol: TCP

cluster/applications/firecrawl/service-redis.yaml

@@ -0,0 +1,18 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: redis
+  namespace: firecrawl
+  labels:
+    app.kubernetes.io/name: firecrawl
+    app.kubernetes.io/component: cache
+    app.kubernetes.io/part-of: firecrawl
+spec:
+  type: ClusterIP
+  selector:
+    app: redis
+  ports:
+    - name: redis
+      port: 6379
+      targetPort: 6379
+      protocol: TCP

cluster/applications/minecraft/DEPLOYMENT-PLAN.md

@@ -0,0 +1,304 @@
+# Journey Into Imagination — Minecraft Server Deployment Plan
+
+**Application:** PaperMC Minecraft Server  
+**Namespace:** `minecraft`  
+**Deployment name:** `papermc`  
+**Service / DNS name:** `journey-into-imagination`  
+**ArgoCD path:** `cluster/applications/minecraft/`  
+**Reviewed:** _Pending Ryan approval_
+
+---
+
+## Version Pinning
+
+| Component | Version |
+|-----------|---------|
+| itzg/minecraft-server image | `2026.7.0` |
+| PaperMC (Minecraft version) | `1.21.4` (build 232 — last stable 1.21.x) |
+| SleepMost plugin | `5.6.2` (auto-downloaded at server start) |
+
+> **Note on PaperMC versioning:** PaperMC migrated to a new version scheme (`26.x`) as of mid-2026. The task requested 1.21.x, so we pin `VERSION=1.21.4`. When you're ready to upgrade to the latest stable, change `VERSION` in `deployment.yaml` and `configmap.yaml` to `26.2` and bump the image tag to the latest `itzg/minecraft-server` release. Check plugin compatibility (SleepMost) before upgrading.
+
+---
+
+## Storage
+
+- **StorageClass:** `pure-block` (confirmed live — `kubectl get storageclass`)
+- **PVC size:** 50Gi — expandable via Pure Storage CSI if the world grows
+- **Mount:** `/data` inside the container (worlds, plugins, configs all live here)
+
+---
+
+## Networking Architecture
+
+```
+Internet (players)
+    |
+    | DNS: journey-into-imagination.mk-labs.cloud → WAN IP (Cloudflare A record)
+    |
+UniFi UDM Pro
+    | Port Forward: WAN:25565 → 10.1.71.80:25565 (TCP)
+    |
+ingress-nginx LoadBalancer (10.1.71.80)
+    | tcp-services ConfigMap: 25565 → minecraft/journey-into-imagination:25565
+    |
+journey-into-imagination Service (ClusterIP, minecraft namespace)
+    |
+papermc Pod (port 25565)
+```
+
+---
+
+## Prerequisites — Manual Steps Before Deploying
+
+### 1. Generate a real RCON password
+
+```bash
+# Generate a secure password
+openssl rand -base64 24
+# Output example: abc123... (save this — you'll need it)
+
+# Base64-encode it for the Secret
+echo -n 'YOUR_GENERATED_PASSWORD' | base64
+```
+
+Edit `cluster/applications/minecraft/secret.yaml` and replace the placeholder value:
+```yaml
+data:
+  rcon-password: <your-base64-encoded-password>
+```
+
+> ⚠️  Do NOT commit real credentials in plaintext. If this is a concern long-term,
+> migrate to ExternalSecrets + 1Password (Day 2 operation).
+
+### 2. UniFi Port Forward (Ryan — manual step)
+
+In UniFi Network → Firewall & Security → Port Forwarding:
+- **Name:** `minecraft-papermc`
+- **Protocol:** TCP
+- **External Port:** 25565
+- **Internal IP:** 10.1.71.80 (ingress-nginx LoadBalancer)
+- **Internal Port:** 25565
+- **Enabled:** Yes
+
+### 3. Cloudflare DNS Records (Ryan — manual step)
+
+In Cloudflare Dashboard → mk-labs.cloud zone:
+
+**A Record (public server address):**
+| Type | Name | Value | Proxy |
+|------|------|-------|-------|
+| A | `journey-into-imagination` | `<your-WAN-IP>` | DNS only (gray cloud) |
+
+> Minecraft uses raw TCP — Cloudflare proxy (orange cloud) will NOT work.
+> Use DNS-only mode (gray cloud).
+
+**SRV Record (allows clients to connect without specifying port):**
+| Type | Name | Service | Proto | Priority | Weight | Port | Target |
+|------|------|---------|-------|----------|--------|------|--------|
+| SRV | `_minecraft._tcp.journey-into-imagination` | `_minecraft` | `_tcp` | 0 | 5 | 25565 | `journey-into-imagination.mk-labs.cloud` |
+
+> The SRV record allows players to connect using `journey-into-imagination.mk-labs.cloud`
+> without specifying `:25565`. Most modern Minecraft clients resolve SRV records.
+
+### 4. Internal DNS (Technitium — automated via ExternalDNS)
+
+The Service in `service.yaml` has this annotation:
+```yaml
+external-dns.alpha.kubernetes.io/hostname: journey-into-imagination.local.mk-labs.cloud
+```
+
+ExternalDNS (Technitium provider) will create the internal A record automatically when ArgoCD syncs.
+No manual action required for internal DNS.
+
+---
+
+## Deployment Steps
+
+### Step 1: Update the RCON Secret
+
+Complete prerequisite #1 above, then commit the updated secret.
+
+### Step 2: Commit and Push to Gitea
+
+```bash
+cd ~/git/homelab
+git status   # review what changed
+git add cluster/applications/minecraft/ cluster/platform/ingress-nginx/values.yaml
+git commit -m "feat(minecraft): add Journey Into Imagination PaperMC server
+
+- Namespace, Deployment, Service (journey-into-imagination), PVC, Secret, ConfigMap
+- ArgoCD Application at cluster/applications/minecraft/
+- ingress-nginx TCP forwarding: 25565 -> minecraft/journey-into-imagination:25565
+- PaperMC 1.21.4, itzg/minecraft-server:2026.7.0, SleepMost 5.6.2
+- StorageClass: pure-block, 50Gi PVC for world data"
+
+git push
+```
+
+### Step 3: Verify ArgoCD Discovers and Syncs
+
+ArgoCD app-of-apps auto-discovers `cluster/applications/minecraft/application.yaml`.
+
+```bash
+# Watch ArgoCD pick it up (from carousel-of-progress)
+kubectl get application minecraft -n argocd -w
+
+# Or check ArgoCD UI at argocd.local.mk-labs.cloud
+```
+
+Wait for status: `Synced` / `Healthy`
+
+### Step 4: Watch the Pod Start Up
+
+First startup will download the PaperMC jar — this can take 2-3 minutes.
+
+```bash
+kubectl get pods -n minecraft -w
+
+# Tail the logs to watch PaperMC boot
+kubectl logs -n minecraft -l app.kubernetes.io/component=papermc -f
+```
+
+Look for:
+```
+[Server thread/INFO]: Done (X.XXXs)! For help, type "help"
+```
+
+### Step 5: Install SleepMost Plugin (auto via PLUGINS env var)
+
+The `PLUGINS` env var in the ConfigMap points to the SleepMost JAR download URL.
+itzg/minecraft-server downloads and installs it automatically on startup.
+
+Verify it loaded:
+```bash
+kubectl exec -n minecraft -it deployment/papermc -- rcon-cli
+# In rcon console:
+plugins
+# Should list: SleepMost
+```
+
+### Step 6: Verify Connectivity
+
+**Internal test (from carousel-of-progress):**
+```bash
+# Port probe — should succeed
+nc -zv journey-into-imagination.local.mk-labs.cloud 25565
+
+# Or via the ClusterIP directly
+kubectl get svc -n minecraft
+nc -zv <CLUSTER-IP> 25565
+```
+
+**External test (after UniFi port forward + DNS configured):**
+- Open Minecraft Java Edition
+- Add server: `journey-into-imagination.mk-labs.cloud`
+- Should connect and show MOTD: "Journey Into Imagination"
+
+---
+
+## Plugin Details
+
+### SleepMost v5.6.2
+
+- **Source:** https://github.com/mrgeneralq/sleep-most
+- **Download:** https://github.com/mrgeneralq/sleep-most/releases/download/v5.6.2/SleepMost-5.6.2.jar
+- **Compatibility:** PaperMC 1.8 through 1.21.x (use 5.6.2 for 1.21.x; 5.7.0+ drops backward compat)
+- **Configuration:** Lives at `/data/plugins/SleepMost/config.yml` after first boot
+
+Default behavior: configurable percentage of online players must sleep to skip night.
+To tune (exec into the pod after first start):
+```bash
+kubectl exec -n minecraft -it deployment/papermc -- bash
+cat /data/plugins/SleepMost/config.yml
+# Edit as needed, then /sleepmost reload in rcon
+```
+
+> **Geyser compatibility note:** SleepMost 5.x does NOT require Geyser to function.
+> If you add Geyser later for Bedrock crossplay, SleepMost is compatible.
+
+---
+
+## Day 2 Operations
+
+### Enabling the Whitelist
+
+When ready to lock the server to approved players:
+
+1. Edit `cluster/applications/minecraft/configmap.yaml`:
+   ```yaml
+   WHITE_LIST: "true"
+   ENFORCE_WHITELIST: "true"
+   ```
+
+2. Add players via RCON (live, no restart needed):
+   ```bash
+   kubectl exec -n minecraft -it deployment/papermc -- rcon-cli
+   whitelist add <player_name>
+   whitelist list
+   ```
+
+3. Commit the ConfigMap change for GitOps consistency.
+   Note: the Deployment will restart when the ConfigMap changes (env var reload).
+   Players will be briefly disconnected — plan accordingly.
+
+### Console / RCON Access
+
+```bash
+# Interactive RCON (from inside the pod)
+kubectl exec -n minecraft -it deployment/papermc -- rcon-cli
+
+# One-shot command
+kubectl exec -n minecraft -it deployment/papermc -- rcon-cli "list"
+kubectl exec -n minecraft -it deployment/papermc -- rcon-cli "say Server restarting in 5 minutes"
+```
+
+### Upgrading PaperMC Version
+
+1. Check PaperMC API: `curl -sA 'Mozilla/5.0' https://fill.papermc.io/v3/projects/paper/versions | head`
+2. Update `VERSION` in `deployment.yaml` env vars and the comment header
+3. Update image tag if a new `itzg/minecraft-server` release is out
+4. Verify SleepMost compatibility with the new version
+5. Commit, push — ArgoCD handles the rolling restart (Recreate strategy)
+
+### Expanding PVC Storage
+
+Pure Storage CSI supports online volume expansion:
+```bash
+kubectl patch pvc papermc-world-data -n minecraft \
+  -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'
+```
+No pod restart required.
+
+### Backups
+
+No automated backup is configured in this skeleton.
+Recommended Day 2 addition: CronJob that runs `rcon-cli save-all` + `rcon-cli save-off`,
+copies `/data/world*` to a separate PVC or object store, then `rcon-cli save-on`.
+
+---
+
+## Troubleshooting
+
+| Symptom | Check |
+|---------|-------|
+| Pod stuck in `Init` / slow start | `kubectl logs -n minecraft -l app.kubernetes.io/component=papermc` — jar download may be slow |
+| Port 25565 connection refused externally | Verify UniFi port forward is active; `kubectl get svc -n minecraft` shows correct ClusterIP |
+| Players can't authenticate | `ONLINE_MODE=true` requires Mojang auth; check player has a valid Java account |
+| World data lost after pod restart | Verify PVC is `Bound`; check `pure-block` StorageClass is healthy |
+| EULA error in logs | `EULA=TRUE` is set in deployment.yaml — this should not occur; check env var injection |
+
+---
+
+## File Manifest Summary
+
+| File | Purpose |
+|------|---------|
+| `namespace.yaml` | `minecraft` namespace |
+| `application.yaml` | ArgoCD Application (sync-wave 20) |
+| `pvc.yaml` | 50Gi pure-block PVC for world data |
+| `secret.yaml` | RCON password (⚠️ placeholder — update before deploy) |
+| `configmap.yaml` | server.properties overrides + SleepMost plugin download |
+| `deployment.yaml` | PaperMC Deployment (Recreate strategy, resource limits set) |
+| `service.yaml` | ClusterIP Service named `journey-into-imagination` |
+| `../../../platform/ingress-nginx/values.yaml` | Added `tcp: 25565` forwarding entry |

cluster/applications/minecraft/application.yaml

@@ -0,0 +1,26 @@
+---
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: minecraft
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "20"
+spec:
+  project: default
+  source:
+    repoURL: https://gitea.mk-labs.cloud/rblundon/homelab.git
+    targetRevision: main
+    path: cluster/applications/minecraft
+    directory:
+      recurse: false
+      exclude: application.yaml  # prevent self-reference loop with apps-of-apps
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: minecraft
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true

cluster/applications/minecraft/backup-cronjob.yaml

@@ -0,0 +1,124 @@
+---
+# Minecraft world backup — hourly CronJob, 3-day local retention
+#
+# Strategy (two-tier):
+#   Short-term : this CronJob — hourly tarballs on papermc-backups PVC, 72-hour retention
+#   Long-term  : Pure FlashArray protection group snapshots on utilidor (managed separately)
+#
+# Backup sequence:
+#   1. RCON save-all  — flush all dirty chunks to disk
+#   2. RCON save-off  — pause auto-save to keep the world consistent during tar
+#   3. tar world directories to /backups/world-YYYY-MM-DDTHH-MM.tar.gz
+#   4. RCON save-on   — re-enable auto-save
+#   5. Prune backups older than 3 days
+#
+# RCON password sourced from the papermc-rcon Secret (ESO-managed, same as server).
+# The backup pod mounts both PVCs read-write; world-data is safe because the
+# server has already quiesced saves via RCON before the tar runs.
+#
+# Note: both PVCs are RWO. The backup job runs only while the main server pod is
+# running (RCON is reachable), so there is no volume attach conflict — they are
+# mounted on the same node by the scheduler. If the server pod is down, the backup
+# job will fail at the RCON step, which is correct behavior (nothing to back up).
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: minecraft-backup
+  namespace: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: backup
+spec:
+  schedule: "0 * * * *"     # every hour on the hour
+  concurrencyPolicy: Forbid  # skip if a previous backup is still running
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      backoffLimit: 0        # don't retry — a partial backup is worse than no backup
+      template:
+        metadata:
+          labels:
+            app.kubernetes.io/name: minecraft
+            app.kubernetes.io/component: backup
+        spec:
+          restartPolicy: Never
+          securityContext:
+            runAsNonRoot: false
+            fsGroup: 1000
+          containers:
+            - name: backup
+              # Same image as the server — has both /bin/sh and rcon-cli built in.
+              # itzg/rcon-cli is distroless (no shell); don't use it for scripted jobs.
+              image: itzg/minecraft-server:2026.7.0
+              imagePullPolicy: IfNotPresent
+              env:
+                - name: RCON_HOST
+                  value: "journey-into-imagination.minecraft.svc.cluster.local"
+                - name: RCON_PORT
+                  value: "25575"
+                - name: RCON_PASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: papermc-rcon
+                      key: rcon-password
+              command:
+                - /bin/sh
+                - -c
+                - |
+                  set -e
+
+                  TIMESTAMP=$(date -u +%Y-%m-%dT%H-%M)
+                  BACKUP_FILE="/backups/world-${TIMESTAMP}.tar.gz"
+                  RETAIN_DAYS=3
+
+                  echo "[backup] Starting backup at ${TIMESTAMP}"
+
+                  # Step 1: quiesce the world
+                  echo "[backup] Flushing chunks (save-all)..."
+                  rcon-cli save-all
+
+                  echo "[backup] Pausing auto-save (save-off)..."
+                  rcon-cli save-off
+
+                  # Step 2: archive world directories (nether/end may not exist yet)
+                  echo "[backup] Archiving world to ${BACKUP_FILE}..."
+                  DIRS_TO_BACKUP="world"
+                  [ -d /data/world_nether ] && DIRS_TO_BACKUP="$DIRS_TO_BACKUP world_nether"
+                  [ -d /data/world_the_end ] && DIRS_TO_BACKUP="$DIRS_TO_BACKUP world_the_end"
+                  FILES_TO_BACKUP=""
+                  [ -f /data/whitelist.json ] && FILES_TO_BACKUP="$FILES_TO_BACKUP whitelist.json"
+                  [ -f /data/ops.json ] && FILES_TO_BACKUP="$FILES_TO_BACKUP ops.json"
+                  [ -f /data/banned-players.json ] && FILES_TO_BACKUP="$FILES_TO_BACKUP banned-players.json"
+                  [ -f /data/banned-ips.json ] && FILES_TO_BACKUP="$FILES_TO_BACKUP banned-ips.json"
+
+                  tar -czf "${BACKUP_FILE}" -C /data $DIRS_TO_BACKUP $FILES_TO_BACKUP || {
+                    echo "[backup] ERROR: tar failed — re-enabling saves and exiting"
+                    rcon-cli save-on
+                    exit 1
+                  }
+
+                  echo "[backup] Archive complete: $(du -sh ${BACKUP_FILE} | cut -f1)"
+
+                  # Step 3: resume auto-save
+                  echo "[backup] Resuming auto-save (save-on)..."
+                  rcon-cli save-on
+
+                  # Step 4: prune old backups
+                  echo "[backup] Pruning backups older than ${RETAIN_DAYS} days..."
+                  find /backups -name "world-*.tar.gz" -mtime +${RETAIN_DAYS} -delete
+                  REMAINING=$(find /backups -name "world-*.tar.gz" | wc -l)
+                  echo "[backup] Done. ${REMAINING} backup(s) retained."
+              volumeMounts:
+                - name: world-data
+                  mountPath: /data
+                  readOnly: true
+                - name: backups
+                  mountPath: /backups
+          volumes:
+            - name: world-data
+              persistentVolumeClaim:
+                claimName: papermc-world-data
+            - name: backups
+              persistentVolumeClaim:
+                claimName: papermc-backups

cluster/applications/minecraft/backup-pvc.yaml

@@ -0,0 +1,19 @@
+---
+# Minecraft backup storage — local hourly backups, 3-day retention
+# Sized for ~72 backup tarballs; world data compresses well (~90% reduction typical)
+# Long-term retention handled by Pure FlashArray protection group snapshots (utilidor)
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: papermc-backups
+  namespace: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: backup
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: px-fa-direct-access
+  resources:
+    requests:
+      storage: 10Gi

cluster/applications/minecraft/configmap.yaml

@@ -0,0 +1,58 @@
+---
+# server.properties overrides for Journey Into Imagination
+# itzg/minecraft-server reads these as environment variables and
+# writes them into server.properties on startup.
+# Full list: https://docker-minecraft-server.readthedocs.io/en/latest/configuration/server-properties/
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: papermc-config
+  namespace: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: papermc
+data:
+  # Server identity
+  MOTD: "Journey Into Imagination"
+  SERVER_NAME: "Journey Into Imagination"
+
+  # Game settings
+  GAMEMODE: "survival"
+  DIFFICULTY: "normal"
+  PVP: "true"
+  MAX_PLAYERS: "20"
+  ALLOW_FLIGHT: "true"
+  SPAWN_PROTECTION: "0"
+  SEED: "-2786778798491440147"
+
+  # Auth
+  ONLINE_MODE: "false"
+  # Mojang's sessionserver blocks some datacenter/homelab IPs via Cloudflare WAF.
+  # enforce-secure-profile=false allows players to join without mandatory profile key validation.
+  # ONLINE_MODE=true still enforces Mojang account auth — this only relaxes the key signing requirement.
+  ENFORCE_SECURE_PROFILE: "false"
+
+  # Whitelist — set to true when ready to lock down the server
+  WHITE_LIST: "false"
+  ENFORCE_WHITELIST: "false"
+
+  # Performance / tick safety
+  MAX_TICK_TIME: "180000"
+
+  # RCON (enabled so plugins/admin tools can connect)
+  ENABLE_RCON: "true"
+  RCON_PORT: "25575"
+
+  # Plugin auto-download — minecraft-prometheus-exporter for Grafana metrics
+  # v3.1.2: https://github.com/sladkoff/minecraft-prometheus-exporter
+  # Exposes /metrics on port 9225 (HTTP, Prometheus scrape target)
+  #
+  # SkinsRestorer v15.12.4 — restores player skins in offline-mode
+  # https://github.com/SkinsRestorer/SkinsRestorer
+  # Players set skins via /skin <username>; fetches from Mojang even in offline-mode
+  PLUGINS: |
+    https://github.com/sladkoff/minecraft-prometheus-exporter/releases/download/v3.1.2/minecraft-prometheus-exporter-3.1.2.jar
+    https://github.com/SkinsRestorer/SkinsRestorer/releases/download/15.12.4/SkinsRestorer.jar
+    https://github.com/mrgeneralq/sleep-most/releases/download/v5.5.3/sleep-most-5.5.3.jar
+    https://github.com/852DuartePls/Bukkit-AntiSilverFish/releases/download/v0.0.4/AntiSilverFish-0.0.4.jar
+

cluster/applications/minecraft/deployment.yaml

@@ -0,0 +1,202 @@
+---
+# PaperMC Minecraft Server — Journey Into Imagination
+#
+# Image: itzg/minecraft-server:2026.7.0
+# PaperMC version: 26.2 (build 62, latest stable as of 2026-07-19; requires Java 25)
+# NOTE: 26.x series uses PaperMC's independent versioning (not Minecraft version-based).
+#       Plugin ecosystem is still catching up — verify plugin compat before adding new ones.
+#
+# itzg image documentation: https://docker-minecraft-server.readthedocs.io
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: papermc
+  namespace: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: papermc
+    app.kubernetes.io/version: "26.2"
+spec:
+  replicas: 1
+  # Recreate strategy — Minecraft server requires exclusive access to world data.
+  # RollingUpdate WILL cause world corruption if both pods mount the PVC simultaneously.
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: minecraft
+      app.kubernetes.io/component: papermc
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: minecraft
+        app.kubernetes.io/component: papermc
+        app.kubernetes.io/version: "26.2"
+    spec:
+      securityContext:
+        runAsNonRoot: false  # itzg image requires root for some setup steps
+        fsGroup: 1000
+      containers:
+        - name: papermc
+          # Pinned tag — never use :latest in production
+          image: itzg/minecraft-server:2026.7.0
+          imagePullPolicy: IfNotPresent
+          ports:
+            - name: minecraft
+              containerPort: 25565
+              protocol: TCP
+            - name: rcon
+              containerPort: 25575
+              protocol: TCP
+            - name: metrics
+              containerPort: 9225
+              protocol: TCP
+          env:
+            - name: EULA
+              value: "TRUE"
+            - name: TYPE
+              value: "PAPER"
+            - name: VERSION
+              value: "26.2"
+            # Server settings from ConfigMap
+            - name: MOTD
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: MOTD
+            - name: SERVER_NAME
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: SERVER_NAME
+            - name: GAMEMODE
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: GAMEMODE
+            - name: DIFFICULTY
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: DIFFICULTY
+            - name: PVP
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: PVP
+            - name: MAX_PLAYERS
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: MAX_PLAYERS
+            - name: ALLOW_FLIGHT
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: ALLOW_FLIGHT
+            - name: SPAWN_PROTECTION
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: SPAWN_PROTECTION
+            - name: SEED
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: SEED
+            - name: ONLINE_MODE
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: ONLINE_MODE
+            - name: ENFORCE_SECURE_PROFILE
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: ENFORCE_SECURE_PROFILE
+            - name: WHITE_LIST
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: WHITE_LIST
+            - name: ENFORCE_WHITELIST
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: ENFORCE_WHITELIST
+            - name: MAX_TICK_TIME
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: MAX_TICK_TIME
+            - name: ENABLE_RCON
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: ENABLE_RCON
+            - name: RCON_PORT
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: RCON_PORT
+            # Plugin auto-download — itzg image fetches these URLs on startup
+            - name: PLUGINS
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: PLUGINS
+            # RCON password from Secret
+            - name: RCON_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: papermc-rcon
+                  key: rcon-password
+          volumeMounts:
+            - name: world-data
+              mountPath: /data
+            - name: sleep-most-config
+              mountPath: /data/plugins/sleep-most/config.yml
+              subPath: config.yml
+              readOnly: true
+            - name: prometheus-exporter-config
+              mountPath: /data/plugins/PrometheusExporter/config.yml
+              subPath: config.yml
+              readOnly: true
+          resources:
+            requests:
+              cpu: "1"
+              memory: "2Gi"
+            limits:
+              cpu: "4"
+              memory: "6Gi"
+          # Startup probe — give the server time to download PaperMC jar on first boot
+          startupProbe:
+            tcpSocket:
+              port: minecraft
+            initialDelaySeconds: 30
+            periodSeconds: 10
+            failureThreshold: 30  # 5 minutes total
+          # Liveness probe — restart if the TCP port goes away
+          livenessProbe:
+            tcpSocket:
+              port: minecraft
+            initialDelaySeconds: 0
+            periodSeconds: 30
+            failureThreshold: 3
+          # Readiness probe — only route traffic when server is accepting connections
+          readinessProbe:
+            tcpSocket:
+              port: minecraft
+            initialDelaySeconds: 0
+            periodSeconds: 10
+            failureThreshold: 3
+      volumes:
+        - name: world-data
+          persistentVolumeClaim:
+            claimName: papermc-world-data
+        - name: sleep-most-config
+          configMap:
+            name: sleep-most-config
+        - name: prometheus-exporter-config
+          configMap:
+            name: prometheus-exporter-config

cluster/applications/minecraft/externalsecret.yaml

@@ -0,0 +1,28 @@
+---
+# ExternalSecret — pulls RCON password from 1Password Connect
+# Prereq: create a "minecraft" item in the mk-labs 1Password vault
+#   with a field named "rcon_password" set to a strong random password.
+#   Generate one: openssl rand -base64 24
+#
+# ClusterSecretStore: onepassword-connect (platform/onepassword-connect/)
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: papermc-rcon
+  namespace: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: papermc
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: onepassword-connect
+    kind: ClusterSecretStore
+  target:
+    name: papermc-rcon
+    creationPolicy: Owner
+  data:
+    - secretKey: rcon-password
+      remoteRef:
+        key: minecraft
+        property: rcon_password

cluster/applications/minecraft/grafana-dashboard.yaml

@@ -0,0 +1,704 @@
+---
+# Grafana dashboard — Minecraft server stats (ID 20659)
+# Source: https://grafana.com/grafana/dashboards/20659
+# Plugin: sladkoff/minecraft-prometheus-exporter v3.1.2
+#
+# NOTE: Dashboard 20659 was written for exporter v1/v2 (no mc_ prefix).
+# Queries have been fixed for v3 metric names (mc_tps, mc_players_online_total, etc.)
+# Players score / Play time / Advancements panels show mc_players_total as placeholder —
+# per-player stat tracking metrics are not available in v3.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: dashboard-minecraft
+  namespace: monitoring
+  labels:
+    grafana_dashboard: "1"
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: grafana-dashboard
+data:
+  minecraft-server-stats.json: |
+    {
+      "annotations": {
+        "list": [
+          {
+            "builtIn": 1,
+            "datasource": {
+              "type": "datasource",
+              "uid": "grafana"
+            },
+            "enable": true,
+            "hide": true,
+            "iconColor": "rgba(0, 211, 255, 1)",
+            "name": "Annotations & Alerts",
+            "type": "dashboard"
+          }
+        ]
+      },
+      "description": "Modern dashboard for minecraft-prometheus-exporter\r\nhttps://github.com/Joshi425/minecraft-exporter",
+      "editable": true,
+      "fiscalYearStartMonth": 0,
+      "gnetId": 20659,
+      "graphTooltip": 0,
+      "id": null,
+      "links": [],
+      "panels": [
+        {
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 0
+          },
+          "id": 16,
+          "panels": [],
+          "title": "Server stats",
+          "type": "row"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "thresholds"
+              },
+              "custom": {
+                "align": "auto",
+                "cellOptions": {
+                  "type": "auto"
+                },
+                "inspect": false
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              }
+            },
+            "overrides": [
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "Time"
+                },
+                "properties": [
+                  {
+                    "id": "displayName",
+                    "value": "Time"
+                  },
+                  {
+                    "id": "custom.align"
+                  }
+                ]
+              },
+              {
+                "matcher": {
+                  "id": "byRegexp",
+                  "options": "/Value/"
+                },
+                "properties": [
+                  {
+                    "id": "unit",
+                    "value": "short"
+                  },
+                  {
+                    "id": "decimals",
+                    "value": 2
+                  },
+                  {
+                    "id": "custom.align"
+                  }
+                ]
+              }
+            ]
+          },
+          "gridPos": {
+            "h": 9,
+            "w": 4,
+            "x": 0,
+            "y": 1
+          },
+          "id": 9,
+          "options": {
+            "cellHeight": "sm",
+            "footer": {
+              "countRows": false,
+              "fields": "",
+              "reducer": [
+                "sum"
+              ],
+              "show": false
+            },
+            "showHeader": true
+          },
+          "pluginVersion": "10.4.0",
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "prometheus"
+              },
+              "editorMode": "code",
+              "expr": "mc_players_online_total",
+              "instant": true,
+              "legendFormat": "{{player}}",
+              "refId": "A"
+            }
+          ],
+          "title": "Players",
+          "transformations": [
+            {
+              "id": "merge",
+              "options": {
+                "reducers": []
+              }
+            }
+          ],
+          "type": "table"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "description": "",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "continuous-GrYlRd"
+              },
+              "custom": {
+                "axisBorderShow": false,
+                "axisCenteredZero": false,
+                "axisColorMode": "text",
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 25,
+                "gradientMode": "scheme",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "insertNulls": false,
+                "lineInterpolation": "linear",
+                "lineStyle": {
+                  "fill": "solid"
+                },
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "auto",
+                "spanNulls": 3600000,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "min": 0,
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "ms"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 9,
+            "w": 10,
+            "x": 4,
+            "y": 1
+          },
+          "id": 13,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom",
+              "showLegend": true
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "none"
+            }
+          },
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "prometheus"
+              },
+              "disableTextWrap": false,
+              "editorMode": "builder",
+              "exemplar": false,
+              "expr": "mc_tick_duration_median",
+              "fullMetaSearch": false,
+              "includeNullMetadata": true,
+              "instant": false,
+              "legendFormat": "{{dimension_id}}",
+              "range": true,
+              "refId": "A",
+              "useBackend": false
+            }
+          ],
+          "title": "Time per tick",
+          "type": "timeseries"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "description": "",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "continuous-RdYlGr"
+              },
+              "custom": {
+                "axisBorderShow": false,
+                "axisCenteredZero": false,
+                "axisColorMode": "text",
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 27,
+                "gradientMode": "opacity",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "insertNulls": false,
+                "lineInterpolation": "linear",
+                "lineStyle": {
+                  "fill": "solid"
+                },
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "auto",
+                "spanNulls": 3600000,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "max": 20,
+              "min": 0,
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              }
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 9,
+            "w": 10,
+            "x": 14,
+            "y": 1
+          },
+          "id": 12,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom",
+              "showLegend": true
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "none"
+            }
+          },
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "prometheus"
+              },
+              "disableTextWrap": false,
+              "editorMode": "builder",
+              "exemplar": false,
+              "expr": "mc_tps",
+              "fullMetaSearch": false,
+              "includeNullMetadata": false,
+              "instant": false,
+              "legendFormat": "{{ dimension_id }}",
+              "range": true,
+              "refId": "A",
+              "useBackend": false
+            }
+          ],
+          "title": "Ticks per second",
+          "type": "timeseries"
+        },
+        {
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 10
+          },
+          "id": 15,
+          "panels": [],
+          "title": "Player & entity metrics",
+          "type": "row"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                }
+              },
+              "mappings": []
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 13,
+            "x": 0,
+            "y": 11
+          },
+          "id": 14,
+          "options": {
+            "legend": {
+              "displayMode": "table",
+              "placement": "right",
+              "showLegend": true,
+              "values": [
+                "value"
+              ]
+            },
+            "pieType": "donut",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "tooltip": {
+              "mode": "single",
+              "sort": "none"
+            }
+          },
+          "pluginVersion": "10.4.0",
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "prometheus"
+              },
+              "disableTextWrap": false,
+              "editorMode": "builder",
+              "expr": "sum(mc_loaded_chunks_total)",
+              "fullMetaSearch": false,
+              "includeNullMetadata": true,
+              "instant": false,
+              "legendFormat": "{{entity}}",
+              "range": true,
+              "refId": "A",
+              "useBackend": false
+            }
+          ],
+          "title": "Entities loaded",
+          "type": "piechart"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                }
+              },
+              "mappings": []
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 11,
+            "x": 13,
+            "y": 11
+          },
+          "id": 18,
+          "options": {
+            "legend": {
+              "displayMode": "table",
+              "placement": "right",
+              "showLegend": true,
+              "values": [
+                "value",
+                "percent"
+              ]
+            },
+            "pieType": "pie",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "none"
+            }
+          },
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "prometheus"
+              },
+              "disableTextWrap": false,
+              "editorMode": "builder",
+              "expr": "mc_players_total",
+              "fullMetaSearch": false,
+              "includeNullMetadata": true,
+              "instant": false,
+              "legendFormat": "{{player}}",
+              "range": true,
+              "refId": "A",
+              "useBackend": false
+            }
+          ],
+          "title": "Players score",
+          "type": "piechart"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "continuous-BlPu"
+              },
+              "links": [],
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "s"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 7,
+            "w": 13,
+            "x": 0,
+            "y": 19
+          },
+          "id": 3,
+          "options": {
+            "displayMode": "gradient",
+            "maxVizHeight": 300,
+            "minVizHeight": 16,
+            "minVizWidth": 8,
+            "namePlacement": "top",
+            "orientation": "horizontal",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "showUnfilled": true,
+            "sizing": "auto",
+            "text": {},
+            "valueMode": "color"
+          },
+          "pluginVersion": "10.4.0",
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "prometheus"
+              },
+              "editorMode": "code",
+              "expr": "mc_players_total",
+              "legendFormat": "{{ player }}",
+              "range": true,
+              "refId": "A"
+            }
+          ],
+          "title": "Play time",
+          "type": "bargauge"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "continuous-BlPu"
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              }
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 7,
+            "w": 11,
+            "x": 13,
+            "y": 19
+          },
+          "id": 17,
+          "options": {
+            "displayMode": "gradient",
+            "maxVizHeight": 300,
+            "minVizHeight": 16,
+            "minVizWidth": 8,
+            "namePlacement": "auto",
+            "orientation": "horizontal",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "showUnfilled": true,
+            "sizing": "auto",
+            "valueMode": "color"
+          },
+          "pluginVersion": "10.4.0",
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "prometheus"
+              },
+              "disableTextWrap": false,
+              "editorMode": "code",
+              "expr": "mc_players_total",
+              "fullMetaSearch": false,
+              "includeNullMetadata": true,
+              "instant": false,
+              "legendFormat": "{{player}}",
+              "range": true,
+              "refId": "A",
+              "useBackend": false
+            }
+          ],
+          "title": "Advancements made",
+          "type": "bargauge"
+        }
+      ],
+      "refresh": "10s",
+      "schemaVersion": 39,
+      "tags": [],
+      "templating": {
+        "list": []
+      },
+      "time": {
+        "from": "now-30m",
+        "to": "now"
+      },
+      "timepicker": {},
+      "timezone": "",
+      "title": "Minecraft server stats [Prometheus]",
+      "uid": "prometheus",
+      "version": 15,
+      "weekStart": ""
+    }

cluster/applications/minecraft/namespace.yaml

@@ -0,0 +1,8 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/managed-by: argocd

cluster/applications/minecraft/prometheus-exporter-config.yaml

@@ -0,0 +1,35 @@
+---
+# minecraft-prometheus-exporter plugin configuration
+# Port set to 9225 to match Service/ServiceMonitor definitions.
+# Default is 9940 — must be explicitly set here or the plugin won't be scraped.
+# Mounted read-only at /data/plugins/PrometheusExporter/config.yml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: prometheus-exporter-config
+  namespace: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: papermc
+data:
+  config.yml: |
+    host: 0.0.0.0
+    port: 9225
+    enable_metrics:
+      entities_total: true
+      villagers_total: true
+      loaded_chunks_total: true
+      jvm_memory: true
+      players_online_total: true
+      players_total: true
+      whitelisted_players: false
+      tps: true
+      world_size: true
+      jvm_threads: true
+      jvm_gc: true
+      tick_duration_median: true
+      tick_duration_average: true
+      tick_duration_min: false
+      tick_duration_max: true
+      player_online: false
+      player_statistic: false

cluster/applications/minecraft/pvc.yaml

@@ -0,0 +1,18 @@
+---
+# World data persistent volume — Pure Storage FlashArray direct access CSI
+# StorageClass: px-fa-direct-access (direct FlashArray block, bypasses Portworx data plane)
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: papermc-world-data
+  namespace: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: papermc
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: px-fa-direct-access
+  resources:
+    requests:
+      storage: 50Gi

cluster/applications/minecraft/service.yaml

@@ -0,0 +1,39 @@
+---
+# Service named after the thematic identity: journey-into-imagination
+# This is the DNS name used in ingress-nginx TCP forwarding config.
+# Internal DNS: journey-into-imagination.local.mk-labs.cloud
+apiVersion: v1
+kind: Service
+metadata:
+  name: journey-into-imagination
+  namespace: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: papermc
+  annotations:
+    # Internal DNS via ExternalDNS + Technitium
+    external-dns.alpha.kubernetes.io/hostname: "journey-into-imagination.local.mk-labs.cloud,journey-into-imagination.mk-labs.cloud"
+    # Non-standard external port 10182 → internal 25565 (via ingress-nginx tcp forwarding)
+    # Public DNS via ExternalDNS + Cloudflare
+    # Cloudflare proxy MUST be off — TCP (non-HTTP) traffic cannot be proxied
+    external-dns.alpha.kubernetes.io/public: "true"
+    external-dns.alpha.kubernetes.io/target: "ingress.mk-labs.cloud"
+    external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
+spec:
+  type: ClusterIP
+  selector:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: papermc
+  ports:
+    - name: minecraft
+      port: 25565
+      targetPort: 25565
+      protocol: TCP
+    - name: rcon
+      port: 25575
+      targetPort: 25575
+      protocol: TCP
+    - name: metrics
+      port: 9225
+      targetPort: 9225
+      protocol: TCP

cluster/applications/minecraft/servicemonitor.yaml

@@ -0,0 +1,23 @@
+---
+# ServiceMonitor — tells Prometheus to scrape minecraft-prometheus-exporter
+# Plugin exposes /metrics on port 9225 (HTTP)
+# Dashboard: https://grafana.com/grafana/dashboards/20659
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: minecraft
+  namespace: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: papermc
+    # Label required for kube-prometheus-stack to discover this ServiceMonitor
+    release: monitoring
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: minecraft
+      app.kubernetes.io/component: papermc
+  endpoints:
+    - port: metrics
+      path: /metrics
+      interval: 15s

cluster/applications/minecraft/sleep-most-config.yaml

@@ -0,0 +1,121 @@
+---
+# sleep-most plugin configuration
+# Single-player sleep: calculation-method: players, players-required: 1
+# Mounted read-only at /data/plugins/sleep-most/config.yml
+# To change settings: edit this ConfigMap, commit, push — ArgoCD will sync,
+# then run: kubectl exec -n minecraft deployment/papermc -- rcon-cli "sleepmost reload"
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: sleep-most-config
+  namespace: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: papermc
+data:
+  config.yml: |
+    # WELCOME TO THE OFFICIAL SLEEP-MOST PLUGIN
+    # AUTHORS: MrGeneralQ
+    # CONTRIBUTORS: Malin, Nozemi, HorrendousEntity
+    # VERSION: 5.5.3
+    # SUPPORT NEEDED? Join our discord at --> https://discord.pseudonova.com/ WE ARE HAPPY TO HELP YOU FURTHER!
+    # YOU CAN FIND THE DOCS FOR THIS PLUGIN HERE --> https://mrgeneralq.gitbook.io/sleepmost/
+
+
+    # WARNING: SUPPORT IS ONLY GIVEN TO THE LATEST VERSION
+    update-checker-enabled: true
+
+    # specify the speed/ticks for the animation
+    # DEFAULT: 85 (recommended)
+    nightcycle-animation-speed: 85
+    nightcycle-animation-speed-max: 170
+
+
+    sleep:
+      # for each world, you can create different configurations
+      world:
+        enabled: true
+        calculation-method: players
+        percentage-required: 0.5
+        players-required: 1
+        mob-no-target: true
+        use-exempt: false
+        use-afk: false
+        use-bossbar: false
+        use-sound-night-skipped: false
+        use-sound-storm-skipped: false
+        use-title-night-skipped: false
+        use-title-storm-skipped: false
+        exempt-creative: false
+        exempt-spectator: false
+        prevent-sleep: false
+        prevent-phantom: false
+        nightcycle-animation: false
+        storm-sleep: true
+        skip-delay: 0
+        heal: false
+        feed: false
+        skip-night-sound: ui.toast.challenge_complete
+        skip-storm-sound: entity.wither.spawn
+        reset-time-since-rest: true
+        allow-kick: false
+        gsit-hook: false
+        non-sleeping-clock-animation: false
+        skip-msg-audience: all
+        gsit-sleep: true
+        insomnia-milk: false
+        gsit-sleep-cmd: false
+        force-nightcycle-animation: true
+        non-sleeping-sound: false
+        disable-daylight-cycle-gamerule: true
+        dynamic-animation-speed: false
+        phantom-reset-audience: all
+        clock-animation: true
+        skip-storm: true
+        allow-sleep-cmd: true
+        insomnia-chance: 0.0
+        exempt-below-y: -1
+        non-sleeping-title: false
+        exempt-flying: false
+
+      world2:
+        enabled: false
+        calculation-method: players
+        percentage-required: 0.5
+        players-required: 1
+        mob-no-target: true
+        use-exempt: false
+        use-afk: false
+        use-bossbar: false
+        use-sound-night-skipped: false
+        use-sound-storm-skipped: false
+        use-title-night-skipped: false
+        use-title-storm-skipped: false
+        exempt-creative: false
+        exempt-spectator: false
+        prevent-sleep: false
+        prevent-phantom: false
+        nightcycle-animation: false
+        storm-sleep: true
+        skip-delay: 0
+        heal: false
+        feed: false
+        skip-night-sound: ui.toast.challenge_complete
+        skip-storm-sound: entity.wither.spawn
+        reset-time-since-rest: true
+
+    # specify the time which the world will be set after reset
+    time-after-reset: 0
+
+    # configure when players can and cannot sleep
+    # this also controls when the animation will stop
+    # please note that this does not
+    time:
+      night-start: 12542
+      night-end: 23850
+
+    messages:
+      cooldown: 10
+
+    # debug mode
+    debug-mode: false

cluster/applications/monitoring/values.yaml

@@ -212,6 +212,36 @@ prometheus:
       #         array: utilidor
       #         array_type: physical
 
+      # astro-orbiter — LLM inference host (Ryzen 7 5800XT / RTX 3090)
+      # Managed by roles/llm-inference (Phase monitoring). Three targets:
+      # node (system), gpu (nvidia_gpu_exporter), llama-server (inference metrics)
+      - job_name: node-astro-orbiter
+        scrape_interval: 30s
+        static_configs:
+          - targets:
+              - 10.1.71.130:9100
+            labels:
+              hostname: astro-orbiter
+
+      - job_name: gpu-astro-orbiter
+        scrape_interval: 15s
+        static_configs:
+          - targets:
+              - 10.1.71.130:9835
+            labels:
+              hostname: astro-orbiter
+              gpu: rtx3090
+
+      - job_name: llama-server-astro-orbiter
+        scrape_interval: 15s
+        metrics_path: /metrics
+        static_configs:
+          - targets:
+              - 10.1.71.130:8000
+            labels:
+              hostname: astro-orbiter
+              model: bartowski/gemma-2-27b-it-GGUF
+
 # ─── Grafana ──────────────────────────────────────────────────────────────────
 grafana:
   enabled: true