llm-inference-multimodel: role + day1 playbook (phase 0 discover approved)

Previous ctx-size=8192/parallel=4 gave 2048 tokens/slot, too small for
context compression inputs (observed live rejection at 3826 tokens).

Measured VRAM on astro-orbiter (RTX 3090 24GB): weights ~17GB resident,
~294KiB/token pool-wide for KV cache+buffers at prior sizing.

New: ctx-size=16384, parallel=2 -> 8192 tokens/slot (matches model's
native n_ctx_train max). Projected VRAM ~21.8GB, ~2.7GB headroom.

Applied directly via ansible-playbook (Semaphore currently broken --
fix tracked separately).

COUCHDB-ERLANGCOOKIE-FIX.md

@@ -0,0 +1,160 @@
+# CouchDB erlangCookie Fix - Implementation Guide
+
+## Summary
+
+**Problem**: CouchDB deployment fails because `erlangCookie` is missing from the ExternalSecret configuration.
+
+**Decision**: Externalize `erlangCookie` to 1Password (pragmatic approach)
+
+**Rationale**: 
+- ExternalSecret architecture requires ownership of the entire secret
+- Mixing externalized and chart-generated fields in the same secret is not supported
+- Single-node deployment makes erlangCookie rotation unnecessary
+- This is an acceptable deviation from the pure Harbor pattern given the architectural constraints
+
+## Implementation Steps
+
+### 1. Generate erlangCookie Value
+
+```bash
+openssl rand -hex 20
+```
+
+Example output: `f4e3c2b1a9d8e7f6c5b4a3d2e1f0a9b8c7d6e5f4`
+
+### 2. Add to 1Password
+
+- **Vault**: `mk-labs`
+- **Item**: `couchdb`
+- **Field Name**: `erlang-cookie`
+- **Field Type**: password (concealed)
+- **Value**: `<paste generated value from step 1>`
+
+### 3. Update ExternalSecret Configuration
+
+File: `cluster/applications/couchdb/externalsecret.yaml`
+
+```yaml
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: couchdb-credentials
+  namespace: couchdb
+  labels:
+    app.kubernetes.io/name: couchdb
+    app.kubernetes.io/part-of: mk-labs
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: onepassword-connect
+  target:
+    name: couchdb-admin
+    creationPolicy: Owner
+    template:
+      engineVersion: v2
+      data:
+        adminUsername: "admin"
+        adminPassword: "{{ .adminPassword }}"
+        cookieAuthSecret: "{{ .cookieAuthSecret }}"
+        erlangCookie: "{{ .erlangCookie }}"  # ← ADD THIS LINE
+  data:
+    - secretKey: adminPassword
+      remoteRef:
+        key: couchdb
+        property: admin-password
+    - secretKey: cookieAuthSecret
+      remoteRef:
+        key: couchdb
+        property: cookie-auth-secret
+    - secretKey: erlangCookie  # ← ADD THIS BLOCK
+      remoteRef:
+        key: couchdb
+        property: erlang-cookie
+```
+
+### 4. Update values.yaml Documentation (Optional)
+
+File: `cluster/applications/couchdb/values.yaml`
+
+Update the comment block at line 9-10:
+
+```yaml
+# Admin credentials managed via ExternalSecret
+# See externalsecret.yaml for 1Password integration
+#
+# NOTE: erlangCookie is externalized to 1Password for architectural
+# simplicity (ExternalSecret ownership model). In a pure Harbor pattern,
+# this would be chart-generated, but single-node deployment makes this
+# acceptable. The erlangCookie is treated as an immutable infrastructure
+# secret (generate once, never rotate).
+createAdminSecret: false
+extraSecretName: "couchdb-admin"
+```
+
+### 5. Commit and Push
+
+```bash
+cd ~/git/homelab
+git add cluster/applications/couchdb/externalsecret.yaml
+git add cluster/applications/couchdb/values.yaml  # if modified
+git commit -m "fix(couchdb): add erlangCookie to ExternalSecret from 1Password"
+git push origin main
+```
+
+### 6. Verify Deployment
+
+```bash
+# Watch ExternalSecret sync
+kubectl get externalsecret -n couchdb couchdb-credentials -w
+# Wait for: SecretSynced
+
+# Verify secret created with all four keys
+kubectl get secret -n couchdb couchdb-admin -o yaml
+# Should contain: adminUsername, adminPassword, cookieAuthSecret, erlangCookie
+
+# Watch ArgoCD sync
+kubectl get application -n argocd couchdb -w
+# Wait for: Healthy/Synced
+
+# Watch pod startup
+kubectl get pods -n couchdb -w
+# Wait for: Running
+
+# Test CouchDB access
+kubectl port-forward -n couchdb svc/couchdb-svc-couchdb 5984:5984 &
+curl http://localhost:5984/
+# Expected: {"couchdb":"Welcome","version":"3.5.1"}
+```
+
+## Why Not Follow Harbor Pattern Exactly?
+
+**Harbor Pattern**: Only user-facing credentials externalized, internal secrets chart-generated.
+
+**CouchDB Constraint**: ExternalSecret uses `creationPolicy: Owner`, which takes full ownership of the target secret. This prevents the Helm chart from adding auto-generated fields to the same secret.
+
+**Options Considered**:
+1. ✅ **Externalize erlangCookie** (SELECTED) - Works with current architecture
+2. ❌ Chart auto-generation - Conflicts with ExternalSecret ownership
+3. ❌ Dual-secret approach - Requires Helm chart customization
+4. ❌ Disable ExternalSecret - Loses 1Password integration for admin password
+
+**Decision**: Pragmatic approach wins. erlangCookie is treated as an infrastructure secret (generate once, never rotate), which is acceptable for a single-node deployment.
+
+## Secret Classification
+
+| Secret           | Type          | 1Password? | Rationale                          |
+|------------------|---------------|------------|------------------------------------|
+| adminUsername    | User-facing   | No*        | Static value, hardcoded in template |
+| adminPassword    | User-facing   | ✅ YES     | User login credential              |
+| cookieAuthSecret | Gray area     | ✅ YES     | Session security, periodic rotation |
+| erlangCookie     | Internal      | ✅ YES**   | Architectural constraint           |
+
+\* Hardcoded in ExternalSecret template (not fetched from 1Password)  
+\*\* Pragmatic deviation from Harbor pattern due to ExternalSecret architecture
+
+## References
+
+- Full analysis: `/home/hermes/couchdb-erlangcookie-analysis.txt`
+- Harbor pattern: `/home/hermes/harbor-simplification-complete.txt`
+- CouchDB Helm chart: `apache/couchdb` v4.6.3

ansible/ansible.cfg

@@ -262,7 +262,7 @@ roles_path=./roles
 
 # (path) The vault password file to use. Equivalent to ``--vault-password-file`` or ``--vault-id``.
 # If executable, it will be run and the resulting stdout will be used as the password.
-;vault_password_file=
+vault_password_file=/home/hermes/.vault_pass.txt
 
 # (integer) Sets the default verbosity, equivalent to the number of ``-v`` passed in the command line.
 ;verbosity=0

ansible/group_vars/all/vars

@@ -39,3 +39,9 @@ step_ca_principal_mappings:
       - ryan.blundon@protonmail.com
       - ryan.blundon
       - ryanblundon
+
+# Leviton My Leviton API
+leviton_email: "{{ vault_leviton_email }}"
+leviton_password: "{{ vault_leviton_password }}"
+
+jmri_vnc_password: "{{ vault_jmri_vnc_password }}"

ansible/group_vars/all/vault

@@ -1,436 +1,365 @@
 $ANSIBLE_VAULT;1.1;AES256
-65653162653536373535643935613236366335356333336264346637323164633739633965656664
+66393233316132396639356564316439343234383066633231646134313361666463656536323732
-3434656433323239366665356134386564643135663135360a653232636463656332343565313763
+6630616536646439613533363430306466306233643730350a343364633233333335643833326163
-61326635636333393730356131356466363063613734653565643766383431363166656532393934
+64393933613963313533623733316339396236363663343635346663323366663166363839663837
-3631373264353563650a616362633165666239323563623161656238633737316236303133313034
+3331363062653239380a326566623264623837326636383939346430666537613361333638366630
-66353461643535373638616364663365643961656635643566613336633466343739386234313532
+63353062393335316663633739313532366363623739653631366539323435336361353331386230
-38353066313763643266623739366261646633363962336239623830613265383335383763376534
+39366634643964336233353961316630616462663166316266613037623363346335373638656365
-62653865626662383330656530336534366136313965373764353133326139623132646364396337
+38353733396636386133373836346336383231663661346137373164386338623733393566373563
-33333366366434393037373361376538376334336265663162336239646237306332623039343032
+35653933343036633365643535303934326537356136666539316137363433643266346630386439
-37333832383762663534643638666234623062373734626335323537616334303638376164656634
+38613332646238366536333536343031356532656336613530663830613264346339353034323362
-64623962313064363166383930636136383935613237613434373733383836666532363265643739
+66613530626361323535653232313730373463373332313561616631393461353730653464343063
-37376333363736326334383864633734393263623536393239393564643131616132373230356633
+38616338363938346161616636316232313838616463326432353639613837343162646363343232
-61636165333537356632303034346431353139383065393832323236626331353236666263376666
+33323064363139376566343866626364373662393138353666646234373461666163363139313631
-38306631653862343064656135396638636363613139373062653735366565376165353330636162
+64343261326566363265323463663538343034306136326234386664333837333937333136653563
-38653530366263356437323433653437373362356166633235386330623333326264333961643434
+61333531353434633339383661636363363535316366353330313566323133616438373161303135
-30323130613431353064393138303465316630383462653062326264393035393735396134396665
+35353630613037316466353832333033393030636331386438393133366333653832393731366363
-36383963633938363662633035626538663366323461656136323934353731626264396364643337
+62373638303737393162303461646239653865653834613662666636373364633165383062643831
-63356337343161366237353135313966616630316362396362343962356564643365626664633165
+34386232376361323638353361666530366432356331353963303930326535663536373339333062
-62383765303436303239376365316632353463666664306164313933346162343833303963313835
+32396266373430343339636635366434313635313766363863336464633961666332353834626163
-62353565333231646462383664356230623165363333653335343438346233316534333132366634
+61653637316163636465343630353431313863653033643237356434313564366361373435376662
-31326138663433663132393761343335656636366237656337666331633062303363633938303536
+63643737353830663236643862613533623237373531646136383763303766336139303632666235
-31303238323932336330363064323963306132633961313634616232636163353439363666623230
+64623834323966363363663730626437323432623966663537346162656265363562643836633731
-65666632653730303134343032616331363932616532313365633038623135396463316666373637
+65663631633462663764393132326165346639353033633035636432613039336164303538396632
-34343736383731363239303330306561356162623962396564323239376661363761336434376264
+66396264393865306666643636353638613661313230313337383634663839363439656533333932
-34373037623331643666366165306161356365326261366566323233656538363061636637343164
+36633432306131396539386539633063653230363932376264323537396434353364643432653661
-39323130613530386661343839623361303862323763373864626536376562643766643263376665
+32643162363066636432336363323534316436613838646562313538326566666239633234646236
-64373334333436653061653962373437366437643034306633343830626230346532653565353135
+30346534636533623365326564613561363362333364363037646561656635623935653466613565
-33326339656266616237643362333062346564333563326463393535356465666166626463643132
+65646361313436356261643762313339333864356338386136306162386262636464393130303963
-36316263323761653333363632386261633432383431386435363362663734626634353761663035
+38646464316432326431326661343632396235626234366133353461623862316662326432356234
-32383464396632396361623531303937383366633131613861646364303662343761323530303139
+37626366383861373831633639616465663564643866356664623066386535646163336134356534
-39333866663164363431656233663064383633386261313063646435333666376465626532343636
+33366664616232353863626465626364313530353335306565336665663866303736323162393362
-62386166306635353061623365313738306237616430386461343262613039346261353662346462
+63626261653161663664363833313461653034326330653835393737616135646462366665383935
-65616261656534336631646434316336646337333661336530633834633037363834346539646466
+30363639306330636634386433646231363530633061336364313338653632323831393630383934
-33346466643238343133326339393264613763626532633938613765373262393236643533383661
+37316362326338313733646332336263386239626539383330353362616132333161613464313066
-33306233623130666161326365366164333434373463323662316432633263323761316338623065
+34663434326662326233363432306433363666356132383866346336336261636435366332666135
-66306332656438303938393430633563303761303239346266386664333837613336316539643736
+34616231613638363339356333616536643266636363643131653330396162306264303566396461
-31353234613664313061306433656165373737336166343930653837303532346163613564376661
+64363763376365356533636430643866333361363062376237653237663731663934306265646630
-61376237383562353639393565303332633366376632386532333736303738333362353334353366
+33393637656335643366383564373966343265393630333835303731316339373133633462383364
-36343166633533303834346265666536353633343564623062613030323934653863343062323136
+33383435383331303264313334393532373932333334343862326635346135613932356337373034
-34323635363238623336653062666636646263343731316537346635346235613862623530326633
+62316262326331313135376465343336373266663338396533666431616462613932663861646238
-30396338386233663335613661646132386230356631326331653962393336653937376264313931
+62653563623535633738383033326235383666646333653731316233376231623661306462303732
-65363465343265643034363963656262666131393632653561626330396166386566396363663538
+32643064373236613336396233323435393939386530323331336138353364663762356538316562
-66323935383166623038326437323730363266316530313938373261366363613731656638386634
+33376530623664623733386133333433303031373337313366386236376539613964316135343865
-31316261616630306131633936326635333632643731313730303438656563343133373732376537
+33363963366165333238356663663435386439336366646138313034343636653463323938633136
-38623330633539653034343364656261316336353466653663646333653636653266373363333732
+39363966376238306662303265643034306136663661393738633436393432303139313132616534
-61333461346131366233336162363033623636386634613731303032393738333537393131316539
+66323432313635386162333838323136623634653264643438303264636430633232323434666532
-65643238613432306436376432373364656530323331633762393566396531396466653463353832
+32616664663063653735316237643539633133356661333132323238376333356464313262653836
-34393662616435343137373364663839613464346564663631343837366530313061326561656664
+39303566316332663737323437633031353330333365383837636336643763313433313937396531
-33623163333864346661393634333830336133383337373833346132353531656236363661323338
+38363536343438663966663436613132663661613134383431633765383164373762343435316161
-64313663356432373137343461333539386665643130316463336134396436623730666165623939
+62303631646235343063383230343232383336356562303563373933346530393333316634316437
-33373832353932386238353866336130356530613766626235316332623132343735636335633437
+32316330306163396434663031393965663163666537353031613365353437666466333464626238
-37303766386434373930383663386565383732636635646133393637383963353134633433616431
+35313739646535356665323734393965303064306132626261363062363438383164346261393463
-33616464633431666361393931623631313731343863636234646130313136346639323662373236
+30643438623363323161323230306230386332363635386234666639623566643536626637616533
-62623830633838326432326339383533373430346635643739646639633739343639636665376333
+37396136643930633262333331656363376433333234343630306535313262306235663263663362
-30396238356161373734633562643466643036363239396538613465346238356366363730623965
+32653434363035613732363136303363393939323337613661333439393637646262383039386661
-32336533303331363336666238363863313066376230353932323466316432653264346334373931
+62326163323562333339323636363565623664396164383332633666386130613766393138346134
-61303066313831376233313261633336643265396139663037333031636666383738313763313366
+33343338393536316431353439353062663164643634396363353131303038353965393466383030
-38386138363738353265613363353339323930343934393036343234626162353037326638636236
+36656465383938353936346361393963356630666630373236626237303064303062383638373730
-66343634323136663134373737366433616632653737316534633232323237343566623361666131
+35633866646535313432353338623462323235346433653431313031363163393666626432363238
-34393331373835613934316235336635666633646465373866356566656261373066653937316339
+39623361316132626230633336636163623466313666346631656134343762656566353432353264
-37396366643564333130383237613933613736376161616231666139643030613338333133356336
+30353436356237653231363564626134633039363035313232616333336436393638396233626638
-37343362363433653638373335363130313362306236393231646637383736663236376537303533
+32663230396539323761313838313466376165646430346634383332346134653662393161363337
-31633230333933666131613764336532373633353132373839326562356438386236643537386231
+62646161343665383364306665333164666231386531626465373366623761643161656462303733
-35616662643431383661343937613738393963373865383465306532336137373730653832633930
+37653438616233353432626466623163316565353764323762613635333832343634323665356336
-37656661326434633730346364326661653937313561333437383064313933363231393164616464
+35353162326233333836396337356466636131383838313436626336663132346339623261366465
-31333664386134376264376339303838656431633966663263303465376161633866376161313531
+30623261303933396562353331636638376135663330643638643536346261626632626139386535
-62346266316533326639303661396164376666383864363262316630636561326366373363633063
+66653332366361336636666437643165656239613031303638333232303836383132616636633938
-33666263333132323232346235633837363138663539366131303633313662326664373332353664
+31343034643037623731643931316463303639656266323231313666356336333133323135363330
-39306235313732376164306164386534346666633037316334366431363063663137356464656234
+63373365303131353161303630633738353536393631353034666139383435303461316131646138
-35656164303933306661373932663831346531636530386139373837633263396632643235383361
+61333731356538366366613831303565613365633965323235366166313534653965366433656533
-65393939303338656537353130316361366139363933363031393735666336323931373632303663
+62666136313662366638356237343734336333313034396465346632336262306531633535643238
-63333137663265353962653861316365643239396562333933383964663730663230353331386638
+35333831366532386235316565303936616264373337356134643066396531383533353336303131
-62396435323162663036303334323936633436626136663537376562363430376239356136343261
+31393837623564386535323532653733393734393164373235396566333565356237356438313762
-36323062363935643562326161643065333437343331613135383932333835396565646631646437
+32623765326639386262393639376461326163333237313232386138643130643231626466643663
-39633035326662653334616366323736323032616264623166326563326361363263623736623739
+39643061393566353434333136366335393536376234366266376265333234643536633035653933
-62633762346135626666636463343132303433346330303631656634646537616537363764626564
+36316132663539306465343039323935356361373439346437386234386464623962643464643562
-30316539303964626635613338623261613430656432376265386636613566313732386561653564
+61336434613834336161633237383361303930313464613666313834356330343138633735386530
-66363538643463653233373433353965623765303038333931623036666435303330633136663966
+36616233323366323961653965613438346136373738366266316134356266623664313539636235
-31313964323439376637613530343635376166333765306466353337623633343539313330373437
+37313033373466383134346361646562366531333338386330653736626530396238356639303131
-64343930636465636265383261633664313762303533326234626461623563636466643039343336
+38363738396236386461316433316261326435646130383336316234363461393237623633633336
-62346462343639323862353036326335383735323337396465613737336661633465363330363161
+30386630376565646337383738663939663462623232316635346635653830306664653336343033
-38323561613935303334303261323032616332333231363762303831373961323033626663383637
+64316430396664393532313766326437636636626232613036666666656430323136356436333564
-35323134666636353766313330316231323938326532333030383638646230643735653663346634
+36303334303562393832336433343438396430373833623137363736386665343866313064353063
-38363261626663393139313031633032343038396433353563363531306232386539303734323463
+32303634393131326464656535633734386462646339663533666430336265653965333538633866
-63313938373735383631373665613333303530383335363262306230326665666535333564346163
+61623666643839653239373335633735373738363736313665323365613635313766656635613832
-39336132643464666538356461393330616264383632623037623634336530376164336336323336
+33616461643539636165383233636533626230343138663630323731626139393230383464313430
-61646165333835333961323439663864386562326131653564623861386336303934346263643036
+61333438393337316239376435313337313437333931623238616133666138363235386533633437
-32666632623537376537646634313839363038666336386238343531323664396461633030373334
+35356163363231656536353934643539643562343732626630383565623730626533313230656164
-35396463373339633931383636353062396562383763373939633336346463623733303039663733
+35333735666135343364663233626163363930383262363266303265303638396239636361366534
-31663834326661623064323132353431366362636466623332363532316435386333636562633562
+31333863333565356135613232393165353266343632633532343061663331633337343538376265
-36613465363936623462316136643664336261353938653362393938666266663330323163653338
+39316630613439356262396634316361356436336634396337353339616536356336653930613966
-63333732616330323438633038366465353865353965663333376466613834633166303761633361
+34656439653366363562636639346430623561303463356337363830373966366632303337663564
-37613863313230653235653034613137303231343961616330316664653333303436356561353562
+35663632313265323365636238303364366230353039353561616636633664643233343430336237
-37623839646536363036613964336230373537653130366330633137356661366561386566323165
+63373264643935616331616632633065366638363833306337633563653065363464343137623533
-38653633373538633762643935346661363961663265616433383832666635666331366635636637
+36373231363739373335346464623533393336613634333636613937366136326464336332346166
-32336532373164373135363533333539623739383330646431356363636430326433376364393265
+61376263623835646163353134643963663964373732313833346163323138633230393537636664
-35643561613033623231366362393737663964396266313232666362326436303331653764366436
+30366234303334656130336630346130656237306161376566336534653630616439323764373665
-30363135663234356363613065643763353836626133656334653138393562393032623631343130
+61383338326163336164353265326163646165623235626137623237306666333832306461613630
-30303936656636366465633163323061333863393532366563316233353466613566316563336539
+66373331356465346261643466323662393661623433383265376666623932343861323139383531
-33326137383263393266393166336361656538353963313366353434333864623238333435653836
+64633536373362643935633734366235396433333237306166646164363930613862613365303663
-39376138356461663262356665363835653638623031396539353132623737333736353937623533
+38343833336137353634313362666665306666393635663633353934363832343739616331386130
-65386333396666633365313262323739633461383464386663313163646632646335646433663834
+66336561633039326434313833303465366638303961626138333165623331386230616130626639
-34616663386661363135653765386534633339653232326133396332646136653230393431626637
+34613962366230333065633761333335613636363533656461626632343631666563383738623330
-30366433336633643837363434383439343562366164353235653833306138643431316537343931
+30333834346233653938633330663166616331376436356533366461336264643264336139343262
-34396535393432383163653231623837633961366437653434623164333333323563666330613866
+35393665656230663232366133393037643536366234343537326631623332373131323739363638
-36393561656135303731666638393737653565303836326264373137303264656334653163366433
+39653162646366316639313631393631666261623230313538613666393732626438393763646330
-35376362326133666336303732656561326164663832623732613830393063343231323363353366
+36346661313131313630343432616365666633353762623261613039623331396330623939626132
-66636233366436363565336231663964303532343962333732313832323762343638356231343461
+34626333386538326434356432623965666662663437646237373537326534653634346239653634
-65373062383564346266323362336666613735316161366136303032313039373464383532343932
+31373038303639333037613637393862356263323066666630313262366633313932396465633337
-30396530373030303933353264363330353132376164303062623538613632323932373230363838
+66653930303934616236323064613761353935613835356561313334323762633064306661346666
-32633766316533316665393033326161373361333166643436356430623262663032646262616633
+37653262343865386236343634316336386630393739626437333065323433613531393738313432
-32643436646264343366343862356339626561346561353463616534306139383133656433636331
+31376233353463373237653164386363633334366332356538343966663939656165323465333030
-32333735346163353434616530623235373037376664326266343933326439663965653865333766
+39656532363363333432626638626438396539336461326338353732376235316133616666316261
-31653163643134346331313331633561643336663130353936323439636331383134646665656630
+37353063343366376433653961333233306461303133376661303332386230346231383837396133
-63663866663938663632646664323936613434333731306264306633643335393566343564346537
+37323137343066383966343535633363643233663530613566313330336232366638396165373631
-35326132303632373731373532623662373861356534343536613731366666373439343563663737
+30626531363033313833303836366434613736396339643032663066333865306535323739666162
-64326339313533323936393362613564333065623731653930363264373061636665313135623062
+64626133616433653864376662623464343131303938303237316264393765303035663833376464
-63373565306535373763356664333465343666626439626666393734303962303762393066643139
+32663264383236303766323935306463643138396237373338653238633464616238306132633735
-62663334313736663132386232613061326636313664326462353361633362323464653038396662
+31626538653262326533326266336633623532623935383266373533363466313033393235663538
-64363862356439633564313065363361313161643962316166643936346539626339396230333633
+66653038646233303665343634383666343363383238326533366136363838303332323230316662
-39333132393135613264653363323233643032376266313630313735373966373966386239336635
+35383235646638653539633961663036663933306463626335356631646662636230356261363261
-34643739363935353463373434343430633438346136306337393061646139346230636531376537
+65633261353830373865636630353932323937666331353635373736376436333361613330366633
-36633861303264633662373365623261303635363836333163623864323338613739383134636237
+30663939356165393132636131663966373433623063356265353131306532643066306630656363
-62326561356535626564633038643230646431613866643239646563323035353434313837663863
+66636636353262633437663264613266613663656137386231306231646264363661613035343538
-66326530353265626361643738333266346339326532663166656432613631313137396463643864
+37353633643065643236376537336238663137623735613038623766393231643131653436333262
-64343339373131363066303437633436306462316166356131366134626331343635323837643136
+31626463646432613563393665346532386161366435396364663239386236616233356131323536
-39306461316230663134346563313662616532616239333735303061373138383363386232373062
+33633936623762666534633862363466353736386137636363633733623366346337613365636439
-61643838366266613637336433323062336434313836376434636433306135303734376661376361
+66663035313430386464623833646135333062313830396637323961386135363461326539623432
-38373438623331653165643837303733393436626166623762623531316164666163316661643761
+32653865623530313637393561343465636430373162333162646631643235653931333830326266
-39616539653831336261333335643834663533363233333732643431363836386633663539386638
+36376631316165343631326165623838306239623764363262376634663236393933343838376663
-39356336653438663061306230316464616533363036363434663335383530326432376632396338
+39663834306165313330393739363133396436376437643232346336386531356638343063376465
-38386130386532333861353362313838383964663861663962353039613565636339346162383331
+61613037623137306666383231376539656361326132396662613061376134376266633764336266
-32313337353430343130326466623666326263343438663463376134336131323735663261356263
+64336334313335643635303632666431383637306334376462643630646339396435313830313363
-34613638303262366666643036353766653039393739303738366330643033316632616232393032
+33383162316261663035393962306234613865613366353465373035656434366261383133653331
-36336161646434373332646663313834623064336437316666623663613861333838363731336564
+63643235616362663663343330303765363263366130393837613939323264373937333162636639
-63636633623664353432323836306564616239343731316335333234336465363232326430643231
+31643438666338646135663538343231643235646364623761653064633566656663383465626133
-30623063316130396531376237363363386465393838303463336439616333353338366436323634
+31373935646266303565303539376162623132316438623565316537306337636630313861623937
-66626463326136343533343263303033333766306632303333376136643137353831323135666430
+34313832636533623033616139373965303839356530353935643363613464356364343162336466
-61313663336363356531633866663366333433343433643664663665643335633035303034666334
+66376130653162666661313139613530306666633432346639656466653364376435636461626362
-62653233616136656462343930636534636432353237363538356365636635373865353532643764
+32653633303561346233643463373534653434323134353434373839373937626663336464303866
-32646431376165616235396133323162373330386664636336336266363961316462613263303131
+33363264623038313835396231373132396163363662626264346461333539326365326165323066
-33303635306136303366353331653239306130616464303835363437313566363436663032353061
+62333139383334333334353031616430323339623066363232313937323465356266323934313761
-32356639333061636536383238396665393232663562363662366563653831343037626535666432
+66623835653961303830383030643537393130653935313265333062393034336562633535323263
-33353261623936313563323564643134666232333839386162663331326461393239623135383530
+65616237373232336534393834653162363461336262653862666637326266663966356665363036
-31323861666631663933636338323239386336356339623738646430333136336635643562353263
+35383437326465663635303664643236633435303862633965346133376536316233386333313634
-64313931656435383134353063383665646538306661396163653434373031326134336235646461
+33643565326633386565653961646463383866646636303537643436623734393234633938333933
-39343734313965313365643135643166366133313465323366633038663333366535633532623966
+65366164633165623333623362393639656661326332306538663738356364373734316563653038
-36666134316165613830323933333339336663363964323764313430613130653236336664393863
+34646531616662386232613034366332656262343164333531353037363036646262623663666236
-64306437316235306565333238666164353738336539353064343161343834656262653666636663
+38613238666136363431623664633863636365396236666532383930336636353031396232656435
-36353063326262653466303534616530616139636166343137666439336335363539393439633436
+61346238643431653231623861373964383931336535363262373437353532393165316562386134
-36623963623839623834656431396162396562373330316438383739363637613366613163336138
+36363263666135646237383666373833373737396330616163376439663736663937666161313831
-35373730383165316330366332373939643839336634613934653538303866343738656534653263
+63663531656635663339306365656663636633343733636165386230376332616331313638386538
-33303361343237356430373266653062313037663230366531366363393937653439363732333036
+32386466323232363533613334333333346161376430373436373961316564343061326164306138
-31343365633862623038316536363031356336383461396562626236313038313239623665316439
+33616263666262323430303730626266396535626439623364376239346564323730323534323938
-30336137356438343362363536303234346637373833313231663333616263303233336161383130
+33346364393033353865393864326361643734353234613563393138363334383536396535393166
-34316265613736383830363639366139313537313661633736303634326237643330663735356561
+34623163616336653436393639313965353237633566313039303137326234383230323235363234
-65626136613831376432383363383731663763356432663264346561646535633966643462326232
+37626161356166356365366164363863636563316332393638616535376466343537373966643839
-31623930303262616535356633663938323635643462663731366433623961623435303062616263
+32613930643533336264626136626465303339376632323034386161663661376466616233633065
-63353836333235336362353039343431313262646137303364613838306339623930666466636132
+66313739346162363838346663623266383130383736656334323430623463666439386532643630
-32656434623738333565636164303535393734346462353436313030396264623332393932653861
+33666639613830386136363535363830333234653961663739343537306634616531616263623762
-36383533363030356664326332376662616139343065343932656262623334633334633462383635
+64666230373830636238353062666330623061613663376638343763626264363130313464383661
-63666439343130623936376636633635613335363264363136643366636535613938373866356336
+38326530333362616163363735323861376366333665623536383566653837306131623732373639
-35666632313565633630373039316262316339333035366332356138393261613836633437363136
+31316661353332633630326162663738636562336666326637353764323431613666303038373532
-33653033343536623264366466356561323063313031633464353963383661316131643166663566
+62343661336338306561356235396636343130633365303466613637633363613862663233633731
-36373937326333393164353234336433656336626134363236303032396531353862333535336466
+66623530353132666261316637303763363830623734346262333633646238613131346564303734
-65393865333565386538386665373166383235633739383934643334653763366161396565616166
+62336434353432326239333232383833633962313537626430663130393733623162626131656366
-61333666353666393230326134636661376237353937646236343034626135366535383631333035
+64333535623138326239336165666562376663663334323036323539653734333835386331653438
-61346231386233326564656636363066396636393966353539666231646464323636303038656562
+63353861666239396437346361306634613462386335376137333963333838616138633730393865
-31656238373331303031393034346630643263613365643232663861373336363662356265326164
+62353539376136316564666136646639363635663736636439393462633165646632623664383663
-61323866663033636534313365376466303538666463663539623231313632303262326638633539
+31613137306461616361323832393036323933626531363536336261356636303531633239333362
-62313239356365313932653735363662666266643765633138393037653662656566656130663133
+32663134363263383039646162643539663737333861386437326337616362343963373532346238
-61373531316261646234666361313766626139613762306563343561363961656565666163313532
+37346137363933623839373838353939386630303461346438666534616434333031373730393537
-35323837333737366138316164376537613834313135306461633735613362376233306134373537
+30313134643963623564356266656430613430626238613266316335336265613132616562626261
-65323339393039336230373237386434333639623762643133363339303265623065326438623531
+35386435313933626634616463616166646466363939313639646264346464363337656339323366
-65356632363237613937633132326231333938336639663438653861353761396434643136643238
+36366665363739356564363232313762323565323134616134666337336534353464373637373130
-62363162666366653831626531353236623532656437313134343633626666383437363761626331
+63613265366436313131356332316531633732356461383064383031613337343363646432373936
-35386431646564653233613632373435623665613335316164636562356264663033366639636439
+33633339386632653032663837346130623636356464326637303338376132623734333932396232
-63356231393362663736366431616161633065303537366538393935363033303636653836346132
+61333033386265356630316134383066343164613130666664643732643362666561346132656266
-30633036653836656433636631313863303132666531326563303266316566623934666361653932
+31623633333039633837383264363937623435643061393935393762346430396335373864633634
-35623265613635343435373361353635343261656233356535303037383433353731313539613463
+33336136353332663366313334353739303539633364663231636539333132303966383432376262
-64363334383666663639623735393934646366386437633638613034316366336337316561333731
+61356563323232613433653262623663336634626532653465306638316633663564633862666666
-38363339386561373834333037626565623335613563323366656365643032373834616339663838
+30366133616336326661626238653933383164336366333438626235636631336165386664343736
-64646534623432323363303032323666643636323536666239616238323932353165663535633465
+62663961346664656333306435323833366632346366356238653731653937626333653630623334
-64633934303565396533313165393438373231376664616237373562306130363531393833353365
+64326662346138386433333232643262333835326263343239353264373038613634356436396630
-63623033343762326437306536663734386365653131346235303837336236366665346234316163
+38323931643361663238623766323930666130356339363564366661663033303831363138343737
-32616539666634653839633739303837646563313364333864343231306663383530313963386434
+66633535326131396236653261303836613364306537633637323031663166316338323533323731
-34313965393731653963643336336134343764343065646436303739356530333761386465333333
+30326235323066396663613531653061643661336631613835626266626436386662353465383065
-35653363653361366462636162663134333431323866306235343764333035373534656138643933
+64396562343966303362636136616438353661626466636635323961613438646634336563636534
-30653962613339313833343532613939636332633236303733313135313932356136383439326635
+38343566396530643961356434643933636235643561353232643062303232323437666261363061
-31386565343337333662663762356361623835353830333865613437336532623936343365343231
+62636530396466303466653333633930376465366561376363316137323263333561343334383364
-33336137616266303835366662643737666333346433366663666462396531643463373562623237
+39313863643062643766396564363137386231373136346138396162376264653538303464633161
-38383537666639363839653538303266303965613762373561643433376664313966326561653764
+61663363623937356138356430666461623130323466623162653863393736326264393836336637
-66396566366265343962393633653765643563666634646637623161323538623961393039333036
+34663931646566333535666664653237643732316663323230383239393763376266356135326438
-33363530653935386331396533666432626634616463663763396663353133393461623866383061
+35656266353865623663373366373830613361373664346632363031356265313364623866643438
-64366132646132663263636131333031643736663031663336623065386266653638376639336334
+61363866353934636337616239633330623734666138396166313864333939663563636138653930
-36653932643831303931613532333036323766376235303433633564303062623336646166303533
+35653137363033326432373661613434313137623163356134613265393238346438306165313639
-63353063303033623032643333653733393332303466366538353062373666363438366432316131
+36666662393165353565633531663536613037623230373063316639663632643139353235303462
-36613335396531663238343139313061613363393463363035366132353439353433646237646265
+31393263656265656131613164363035343233626433656135353331613532363236616439363731
-37633964316466636337363331383565323866653036386435376530343661323565666539663230
+62666432356363323937666435326638323437346136366131613636653430306131623966356263
-38386232376436623064613232636637313931316535643661306536343661653135313335616534
+39303739306233303862636535633431363630393432613663633836396566653039383735303336
-34633761663066316430313361393063623033373636633133653536383764306433346235343761
+62623331623034636363636661653236386337326666656532343737336262336462613762326531
-37393764633561653632653565383937383436666662336633633261653861323533303732613738
+34303066303834386366636430343161653665343362363038396562626133636135656538306435
-64653739323636613639643465316437356163356238656139663034313738363532373861393333
+36393364333066346238396362663664643236373532336263656233386663323663623137343462
-64613038386535396565656538663163313161353831306237313064666566336534646233393137
+64386337333130316434663564613665666238623132343437656637653035373738313735366630
-37336265316361613430663961356361653636636235323439306537666363353361643664363631
+66383639616166393265616434623463326437313530326130376339313662303836636664366232
-63643631336134323032636463393639613636643130313136326538376461363937643436633561
+32366634633030333130316435616233396231663937343732313066373834326464623139363663
-39663433623663346462356437666230313937396132633834386661613436653839616136343234
+31323931626364303230666162316436653065366137663631376265383063316534343736373261
-35333437386663353863306166393634363734323636333137343938353461386366326239396236
+31613637316235386539343766323439653062633137663730343236343661346162653366656332
-65303939623437376535633463626166653935343033303066383064306332356230623566393062
+32663932313063383561636266373766633535656131386133386135663863396261306530326632
-34666566353238653730613561383434343332333732356334623836616334393464373965633334
+63653936626236316539613262386231616433393064323461626536363831666461316131383837
-37313564393463663862386535626338336235613132376530666635343832386231366533363562
+30646266646266393666396362326238613231303335336532303836363264323233343534636635
-32626162626434366261623431373938646237643462623337653466653538653732393634366639
+66313538643033343262373463363866346566353263303966323933383963363463393761383865
-62666231376239393066366336386166646631396632613631313663633435646337313465643231
+64663932343830643531643466303438343161396133666463353762393737613036646166333265
-63356566333764333238326165613832643161346366396635663032623133386132666333343337
+66376231613232666164663964636134653061633330383863373836306366393838393235656331
-63323637383035363833653162326231616461653062613765363165313638313234343266333334
+35613231306263373230326634623262326333356263353961633836396531633431383163633361
-62383362343232363431613364383930376338303839633261656236636264353335333938333731
+30356534666466653734333437383964346564346165326664633738653338313263633837316531
-63616666626437643432393438376661616262643939363239353836323036303139666338336261
+36613034323433643839333264323864613033313137663131623265643364333664646235666232
-31306437353536623561333531346630643761336235373230373135326361393739613934653361
+33393039313666323266643362323337316465306564303230646561303434666630616137633831
-38346532313364633638306665306563646433353038346238313530363163396166353535343537
+34346439616634343337306636643733316464376631616266376437636439396337306637333432
-34343330373063383566333335386162326339383835373665363464616139383331646436343365
+39306333363035393436316434656436353738303861633933376531383862316466373736323639
-30386564323733393461653662323334656637363566343737356239306462303762383332656239
+35336137373866336631386436646231653366366435363932376434303063613961353261343661
-30643439343161636434656338636533396364643632383030646639356534343963666466613566
+33383638393165336438376662306431333837356435626137356130323836396335636166306662
-39373833633265623563396363653830393661633636646265643362633731626462636531313362
+36386161353739353637353861306666383966323339303262616239633930373633323937356632
-35366331383635643564316461633431646565393863373635323834363934336335653831613330
+65383032613031666665623631613430666662656336663931646533636230303261646530623765
-39646262633636343832626362393730643163616565363765346266316636343065393630333866
+64643939326435643539373564336531623236653731636636346361363064333963376566616530
-62333931336161636230326132393739616665646462653666346532356265643235333131343635
+62326639663632666634326233363635383830643163373938646165656163643864336436373466
-31326638626162356333356261623531646464633139666539373261376230336661313937363036
+36353832306632386230373832333234643638313238626333303963383962343265366137656136
-32313464316364623838343231346538303237393137363233366234373031346238303932383461
+37333261343161633562346638323632616566646162633133663466346535656463393932386135
-66613537346136313262313337636565383639383863346535393361363930616535626264376565
+62653332313066363965386335356430326539316366633537356364666230326237306236393563
-30303236663065363161663836356130363130363339643335653366623065373566353136353933
+35323363633936353034323232353366373566666332323737653237323135646665626139393436
-39363761636130646537393166623335353431616462343364383535393661313738343265313138
+65333762653536656161386532363765336538653763666236343166653933626666633130393033
-38323066333239616137623233323664616232323262666530373064366138316431643430643462
+31643531646633623663313237353333313136663863663430306131316165663765653732663164
-32653262386565396335646431633534663031613536656561643434366464353335313239393464
+36326531626365326330643064336230313466343731376437316563303339336333326636633066
-36333866343063656533623432306231343065343530343638653739343731336131626532376566
+61386135626430616661313236623030316362373338643233326365646531633265626238383830
-37653833313233386538666464386232353466303035316232646339653533663831323261636330
+65346132643537366537626132666165616138656139626132396639376230333262643766386363
-32646166366633656631393762626236356633393130386165643264663437666433336138353938
+39383034663034326165643636613237623234613666333532383733623462303331326238636461
-37653866646364333134313366316366636132623764366562623932383239643265646461646434
+34383139383466356139333934353837613964326538336463643832623062633034613762363061
-63656233316639616134666635373434323030643136626238646637386634333439616231336136
+61346361656363366136353336326433326266336564316366393565626262303637316564356566
-38326431363062326232626238613334356633353362653138643237646339333762616330383934
+39376661383763373436306238393666653561306538333638306233356139346634653363346164
-63613235376530373834333834316539343330313862666262633439333062633261646138626434
+31303835383062376638626266303237323832623735653066353936376339633637333562333561
-33616162316632366537393939343163333530393939613936323164663034326536666133303365
+36646462653834316131323166366661386161646538346464386232306239363030366363633663
-63643337343037363363663235316330316131353064383933323131643036623862303835336633
+65306439616339626635326531636435356134376561303235393337373564373937623636643432
-39376134313264333834306330366136643434386566393466313835396466306363663565303065
+61393535643038626562366331353831663338333838383066323632383633346564396566653330
-33393431636163376365396139346162346239343838616561373162643931623365336436626636
+34386563393832313537623061666466366661333934613766366165366330353835323637643635
-39613030356162363931316166376231303033306366616162366239396333323838363465353261
+38613363396663356564646132613536653033616337386566623662333832383938303138316662
-64303561386563383834386166323739626237376164326533616533343338316430366365333966
+34303339323166383863363831636233323335313565393933636435396666313337663037323432
-31343233666365663830323030336161623466633261303637373537336333663262383238393833
+63333139333165646262666339343736383966346133356138326437386334626461636530336432
-33383033343630663237653338636436346565356238313434346666336436653835326133313038
+61356631366163366561303636333230643732316261376365386463333565623533663966336365
-65353530333839373136326132326439306431333334633933663065316531616263633533346431
+39333365393766306536366231366435363030353263393534653534373064636361333532323735
-62383862653162613532383136363830313964366161306166343934363865316533643736373431
+61313163323831653362356333386638343566356261353534303738613730373632363534666337
-61323135343030346336356232326138376564343131636263336332333239303733323338663830
+62313339613138646361356431616236613435393233343732626263653332663265393934616134
-32373639343063353261623831623237663133646535636338353333653237653539656438666436
+35313165613766643938393839373261633439396661623961353934373130623865353639633038
-63636537336636313762336531346531313637653834386535313666306636303338306465303764
+34353631346433663131653965326337663561613330323562666336656237633163356562323931
-63366133396263636164386339343131653361363831363932303734333634343734636163383539
+62393833663233333538383063303937386365306135343962623333663435663431396438666362
-33613430353236616132636431656636633561316161623661373663396462633930343865336236
+61386266353838653532393233353939363738306634666537313761313835333864633764666262
-32373534666135303730303663333337643033346231646661393137643161383833626335336561
+30333032623766633334383031636633636539336237613235376466356430663938653565626235
-66366433303436313132636230313531316261613564323963626438623530633634363265613939
+64373537656566306136633630366130363630633462656330623633393735386630343437336436
-37666636313632353831343238646237666464333039306630343531626339323562346162663130
+65343635366531646130616534623136636666323139326462306533653532643962656530336633
-65633133366564306230666436663630316434616634656661306461353866396662376331666630
+35616537303932343539336638333730663639396330653761346136363431346536666138336462
-30323166386535633361316633303830623031393531343532633930616439326465613631613838
+66396565613166623934316532383835316137303134363466306163356233356530323231666464
-38326630366534363036666433333237363636643366333161343336363936323730313339343132
+30353933306530323734306564626234343864373964333264353366326265316333343330356532
-35356231666633386537366236303136663261336532386338316565343239333261613838623861
+38363837646635633461653562303264353633343461633339376665616331613733666663353130
-31366531343232653431343436343766386331303133363534616536633337656433326265656161
+65396363613731366234326466323738663563646166653237613364323734616465643764633537
-66356235616135363634313835666134346232356663326361396537303339373334323861383161
+35373865353532383566363632366564353536643739663761303565333138383638653665663664
-33393438313061613465666536666636643166613463363165393338643066613231636138343537
+65643366316461613630366437623736353739356538336237613431306363663234373265623962
-64333731376139353934316631393561356163366237386337393061343235633131613962383731
+34653565373335653563356135313835643266356261623037336536613733323733363933376538
-66376261633832323864356564326532356363343762313563393331393933316336373434333237
+36626134396563623733656534363331626262643339633932373035626134343531623634666463
-35396531303330396363333163373361643837383766333730303031613665313237383532666264
+61623036313334616639633930393562663631653565656136666537393731333430663062643362
-65613237623136336239653566646366376538356362643062386436363466616331336665616432
+63633663396562343965313261373965356163393538666466303661363531393266316462626166
-31383365363630663530643764633335336332386332663661393733343038626265383432306630
+38616536653665366462383064373766396438616665346666376232653031323566313164383164
-38653861356636313634326266633637653764396233313463393964653739306234666662336432
+36643231646439663637333165376439333432383532316661333766363136636236326338386537
-33393139343864346233653763343762616230383131353166396265653031326232656333626264
+33306130353634346136356234363438383865313136393839663066333935623565333730613538
-63616339663564363665626338316661623230646534373231313662343736316462663763333364
+63323831663866363831383930303434333936646564316435303931396362303534386335343330
-30333137386161336664386536366538313934313039333832323763313264373861666239643466
+65383635346662626363626365666166636361633365643735303762393832316436646139303835
-33383334633736333634306262316265323634386635313764376532646364356565396538656137
+63633733656361643233323332613632653837663262306661626438316262653931333061336366
-63316232643832383865303934636364313036666237636632393137373933656263646534623230
+35353939353835333361623261613738383734656132613139393264393038373765343131333330
-65616231323035643531353430326630353761633234636234343834656664323161383335343235
+37663962313566366463623437323965326365623437363038633661313461383634626661666236
-62616533366465303961306134386265303933616236346332623536393236633366333634316264
+32633335323861643037383261393164393933353531636134323765353962633732396230636331
-31396361343833623131306266336238343866303361303336346566656639616134383063396235
+32613865373739366530303538313566346434633933393330346637346136373036306666336164
-31626633333665373934333961646263663730653331326137383031383736646633356536333431
+35373732346334353432616561623031663331346431383235306537386466623339356366663335
-39303632383030613465656236333763393737353865386235333830613665653363393536646630
+36333733626433653465336431666530626665373564336339626163633131353330656437643638
-65353733613536653963393831636530636231366266343262376631393062623163313031623230
+31616563616665343635356231633135663665326131636664373338323736393364353636613762
-62326561653165306339633439376461346330393231313366666339346536313765363163616630
+66636135633766323866376235633535613735613465303239343036663438333331626431623435
-35373564366563656161376336323031346339313734633139383563393966373637616266666265
+61323537386434323638666537643236623632626430666263376534643336613635663762643736
-34623330313265373935396130643231353131313163393333396337396666636439633035326635
+30323237353265613062373265643562373637383337326264653639306263373865333262376665
-32663762333537663839346531663132613636366639643364373333343262326264366136363331
+62646331373931373762303461366163393839633135393964313937616437323865653735383630
-61613833643639656632353931663830613634383334653036323462323262386265313637656535
+32653936336534666565373437666130396265363561333635316461663766346336623865376133
-36333834306438633532616335663562383730306337656336663035643136386261326130336363
+35373665303433326265623531613038636166643130616637653165376263643634376439613765
-32623330333764616565333162383234656463363432313039666265653563323232613930303765
+35363031373630333966656466616235616337306335363132386335613462363664653634633864
-30623834666665636332303862663832303537656664643362643636616265613739356664646462
+61303635653932353730663666386263633662633736313461643932386161313762663761313336
-61326466303266626166333730343330396439356663376362666536623539353666636230373533
+62653665313033656537643936373465633932626166366430643763313030393838393039323230
-35363461326263366137313037323166396133646430616664343165623533393963623535626237
+62633334383938343433306262393536653930653030393033306661666264313630643564333166
-39633362393636373131626364636437343361303435396138363735326235383237633964336138
+35383237633932656331653030363434313534613637373465663264643061303538653666653861
-31643935646431386537643733353862666138333238323461623638356535316365653530376464
+35373936643037333866636131373338363062663035323531626431633362663364396365353139
-33343465653134613536343563376564663133336532343330666131346663363434623238303862
+66383737666437353764333231303662393630643933376161366430376530613365363830373534
-31306331633238623836346466616230356532316232323734356638343532643032633937653061
+38376633333936626430393163323830346166643537326430616236393733653761363235356363
-37393265333233626563353963363637653338393964363832663734303566356633623733393164
+35333131663032383861336262653936376565646662313965303265623763613330653461333835
-64383536343862393134336638353733373262396563303266346535346266306238616531336162
+63613563323135633438383931343731656333303362316533376339376636623037376431336366
-35363638333431396163303530353461363338656363336366376430316464353131623661346366
+61393236383364356162633062666265653534326363363862666539623761623065386537616563
-35313930353538343830333337356163613765356361343663613933366464353935336361636436
+62666561316437303763376635346536666437373361386666643139643737663333323933613661
-65356465346535613231363932303339646162303238376236323461303062313336656366306563
+38326566663932333930616435626133616531306461356466326437623235613233393434626563
-31353464333539343839353764393665393235653537313939373563303436626635663766303336
+34373966633834373430386132353163366465626262353863353335323830393266393562393133
-35333838323734386537623334366235653534623664396664656264633735353634663332353364
+65383632653438646435343333386261653066613663623232373564666465613136353039313036
-65306132346464363966313963346633353538633936363164393566376630376365316139376132
+61646462626433646330396664363938376530376438646262343231393262383733636233636333
-38323039653465346462666439656134393964623563393664643634313638323832356164323863
+35633862336566636439653464613564333162613836343636316334316665383164353131373431
-31366535336238663236666162336166313933376531643137343665653765623961333464633332
+63623030306564346562346237333934616134346536303365396533626262333937396432393830
-30306265306364643738386561303833326363303836333032393462643764663664613536363835
+38313763393463646437666137353835373735646365373934363936346564326362376565353133
-30663937333038663435326637663636383165363632326337653932336130333932333861366235
+36653362333432326133393837316331666663663263396461363239306239363733633137396633
-66313566613033383631376132333634653639336666386164613934353230613239383239313038
+38393865613431653337313665313762653635656531353465623436343132303064303564393066
-30653162643233636638393036353032353632316166333938313966626133643237376461313065
+63616639353962366666616261393766643364333634346630616436376565313236316539633537
-34323762376133393535613864636461616261343031393266666165316236323231323534376465
+66653239636561393433383639646462616433653166613130373134376535633937353366383230
-63386363343933316139356163363132343666366132313038643938653865663934383335336364
+61613335663434333835653236343633633038346335333861356637353965396632393833646635
-31646563346339323633643366653861323030636138663861636434306238623738636331393538
+36653034356234663831333764303338663464316362646339376338393236336161616263363538
-66333765626132346533313261646434633236633432333430613266333161643566326434363633
+63356631613239326161343031643936623366643432663732346438333265666535623664396333
-62373033356337393232353566666263353539326564333766303335666135343461386530626562
+62303239363339626566613439396234303536333333653433393666383635643235376165666234
-63633137623361366362616432613237353231616465656232373835376463386132663331633737
+30336236393962666335353233666463346530323531316438373130303933383465316638646461
-63306136326264643365356634336233303163663135366531643362303666653630363962666637
+62643066376666363236333231386237376466633932313836323163363061313333633434663763
-61626539613333306263323564373435613961633439633261373530363137376132326462633035
+64323365353336333736363436376232653436633739613437343538633632356665656364616637
-30353838336433313135623530643663643232653364316263356164633661383937393238363262
+62313666346436656564663335636635393632353430666236313863613464626434323939383538
-34393666663361613735623435626238646266376533333963653630663462653733373130393338
+65386265343434303632313739353239323565333734656566356164643430613538333234383566
-32626361366632633630323363376238343534626230373835353036333065396435653964326534
+37646431316363316139646435333732313339623666613738663039613239613738393565333330
-33356466643839626162316438356437623732656664613434613165643666656230656434633638
+33376432373933656435303737653762666464363865633831373330393435633332636261336139
-61316134333631326365313532313335633634356466663834666531653365333333616137643835
+34336236373535636165353262323966363164633135613534353661316364616637663465363864
-30666664353134313236653337366466623966366131313962346564663432633535363938333261
+39306163346365643339643937396165666366663339336438373031613937636464383531613962
-37353464383832373366613531666161323632616236613631653433643562326134623838393135
+34373433663533626665656364623634373335313033646165303764396563356235343033383138
-62323533373336663434386166333862366639626562343830396533646138636261326161393339
+62326361353630393938643764616636313461633734646661386536356235656665393864386465
-33613961353431653537373931396531623061653163376230663338376636613136353433646231
+37666262346561656436343036646330363664306135333464663265306165353039396665336664
-33633236306238336432323134393463653161303662336266313562316337343430373532663930
+62346631366330653762646538323565613864383534636532633033643533323736373931643130
-39613133646432343238393762386266303531663032306130653532376262353238636231393539
+32343435333333613734626234363132373734353035326232366264336161383631353133663230
-36376332633563633661353835386364633461363437333038363865356237643739323133653235
+32636533333866343763336439373336356237303636376334333433376630353338333261333037
-66373931336330626663383136363265363133393239393131623465383433363336333237653131
+31666537363666653238383939663464346662636133346561326335346163363061393830616237
-64656662316665356665303765343932363733666362353031656234373363663364666537663030
+36643430626534653331653665316535303139343763663965363164636238366533303038653935
-34646233386630633438313166396663383732366233373139343431373463346133316663643036
+32356432316633373137316237663331336463363431393033323635646564366639346230353363
-31633536663266383132393032313563653035303034336161356139396636353365636163383031
+38346663363039363962323137383366303862356530353238656563306131643236626536656334
-30666637363933643262393065633463313836653530663232303736306537636533343431333966
+38303462306562366532346163323061393437353063326539393466616439346564383036303235
-32386330356564613932306466353931393839316562353362303765663263383738363765333136
+35633731393661323962633631373061303930323638326565636162316436646337383266626561
-36383036376130306133393166333734316231376165393832383735623838353466353664353834
+32326430363031396530396238353862333133363731623736376239626561626165663337373261
-39326136656239626434313930306435333533623533643236393437313038393235386364323766
+39353461343461643238646635633562653865323336366634613264616662323232653861663038
-62323537666333613066356236313361376138656232343430363438346261646165623565313435
+64396330626633303031333334343335393039623135353266383561313231643433393963326637
-65326532623761343239666664636637613034623736386665303634613737613964333630613533
+39313530636361373831306234383166346266656261663830636631333564356536323565336266
-31653731353533613866663166653237663162623236353838646465373734613466616362363833
+38306561376366626236306633613564386166616630613032633163613837313462343662653261
-39346335373539376439643937363233373362616334323231623232353061343830383336666466
+63353437663436303634633336636532646439636465663362346138313665336334313039613631
-31303663643061653866326632333336626462623835366564356231316263326130393138306330
+65326135383831613531323265353831313562346161663265366434623236636635333038366536
-66333532336164633931306136373531666131356130313262313038626138653864633734636132
+33646631663662393331323162343438626666366636613438383665633136326439376166373462
-61353638333133383035333937626333376564316465333539653138323739356265613664653536
+33363864613136643461663436396362643066633437376631623031613366656238396165313832
-35656136343036363532613335336230373364336432343731326132666662636664356364303335
+31313131653263666334393664343239306235373862313339373563643137393633343663613936
-66613736633231393534626539333536626565663231396536363062353037346563643934313236
+61356564636238363136623031336638333566633766636362303938653531306131396665303033
-32363138386430633761383065343834376631643539626330313638333534333734333239383632
+63353362636463636236643464343562383161343432383766396330623764393837613435396162
-39383833656134393165666533353739303834633330653936643637316639663033363865653631
+31303162356437663932663964663239623764666366663061313535346438373334636263653531
-36313735613962393264623938336432336538393937316634626165383934356562353035316533
+34643133356638653031373036343162653135663734623035353033633561366266623566383233
-33353735653133636439393763356430303863326235623932623464356636666265653065383736
+36356434643538643430383532393762333535636639353361353763333363313131646264336332
-31613666303065653035303131646364386636393035663865336164376536323462666239356333
+34373331343930633962623963666365306132356334646636626461316236343839383266363635
-33373562383863323635613634643165336465613734613034366236633835363733306662613462
+65623434336239313330343437646333353362303232346638623161616133636636626236643465
-30326565346365633131363331643165346466366666633231643538303430376530393139393063
+36303636363965363765656533386534633839346363363738386532386531326538643134363132
-32666664313162303065616261326261353130623564343761633861373034646161643163373533
+66613235373362633166343565323766306335336365333439323764623964393263623236623832
-36386665626530313030343032376531656333356332666437383530613834356336386465353539
+34346132383136303038363764333039626234616132386464666633663536656230666133363533
-66623062646236333465316137316564363938303966613561643830353332333537373736626438
+38306665643361666539636666316432623430623939663636343164386438313765633031313534
-64653239613063646430386631313435306462303566333734663462626530356530663235303166
+65393633323837326166343936326263343833646331326464376138633461613532303135393036
-31633366613264316137303334613366366537646261303962646364643238383062643732386436
+65353830373065393038343039323937303634346665393135383639303162396565646232663736
-39643430613562663136633130666563396538306633633034323064386136303034306531323564
+39376434646634353330383933303164653431373433346335666131386165343035303964626665
-63646633303035353831653438366635646562613639313230396462373161363030386264623033
+35383035653631346638326637326235393833623264323030373238646335346332353362393230
-62343031616135346238303337313931643461386363396663333231626661613938626366653837
+61616664613562383639306564376661306665396138613066326631616531623132633966633832
-33303265656561313332353463363534666232363561353532643532386133306332333930373161
+65363637376264336635633132633332373634383864626564623966356464373864393832323738
-66626135303036633566303961646461383236663737643265346566393963343634623164633966
+37616338646633326636323461376137663632376262363738303336616463326238333465343533
-61313031333863383237313633663633333866366230313936396334383361393338656465346437
+31316132386530326539
-65613763643061386463386431386539656635393435633032343030633738373936613833623938
-39316239363835633339343161623237656230633763313031643663623466323764663366376165
-36366530363535376636383561343161643037626639323830396665303238663534633531613735
-39316639633730376533643932633432353835353439646666653766666338646662643162373263
-65356563303235633963633232393736636537346637376462626637303535383063373134613732
-32663763663364633463633738343535316436303365343665356133313836616164343434613133
-34626434643531343461346332636539636463313539393830613434333933643632313737356564
-39386137383165616139626363613732626336366461663339346134656530396464323533313265
-32346535356466383135636638616166313663613565353765346264376535653135356638663538
-30623834393935636562656639643737373462643137363063663737306361336339653334633665
-65316237633436303761393766393234326538653633363936303534646566373338623935666538
-38313761643437313565663762613838636163633066313630353631353934396338353665653330
-35393661356630313436373761383462396434643535316364383338343433613061353637333563
-38376562373462316334313632343965326235323231653030666666616163346438626437386637
-65656163653935656530366239663237633937336166613132356333623964666630386331383331
-32303166386431643739303363646633376331623166313135653265656636663236363936336462
-63303135613539663233366362323565396137623264353431373836356233666332633731666364
-62376565633037373832393366306431613863353137306535393664313866616235633638306365
-32633761386462386339343066316263663438623330373733333634363736623637636661636331
-38353835333962633537633864616132643563616437336334326633323636393833363666353261
-36363732383565623233343439666430303961306263363032383238653265323637313430613133
-63633338323536386139613233343636326564356564353065386664646230666561336338366436
-35353162633864633764373535366634303738346236633362626165393632383762343036313930
-62363864626338393736373938343264356235393763653432306330363363313762333137303362
-35393061356165666230343135663438303834363533323064626532663662323063386333623166
-39353037363363373166316238393565353266353665666465333134303234373263663135346364
-36646463343032326339663130393963663861323130356365396365643265313833353234323330
-34336661363733613362343739366636336265356464613033343132353031663965656634613831
-33656466376132656465383664363231653235623036346634666166623532323635313130303836
-36393064356637623139653333633164666332303539623863383538386262363064356130353337
-62393063633637383965346139626265666463356362633163363537303166336264346332636363
-39613735626132366137356365303331633037623132623637383164636565653963626632626338
-65383663633831613666396534333061643966313366303937333261303135353762656331336638
-35643539326161646433633862623762626539396137303863616139383832613639653033653933
-39373734393066613132323739383036313830346239306132656464646462626661633931623932
-36626163353563633764323466303966636161646532316134383034393733653363363933353738
-34383563666166323865656430346661633663363635623566336632353633303838303436616266
-34326131623331616533306465373365396437303764383231356631313335393364643664663864
-33313033303463376139376466643434383461643264316534306166303163373661643962343030
-36653665353232363630363437376266306531633934626230653338383664343638313334306636
-38666162623761366237383236346564333333633162623832656462616662313031316166383465
-31663362336663353564636335393738333565356432306139363661656663313234396664623832
-38393630666338663031323464383535303539643931393732616666366661316163626339643437
-34316532613063353264303462366534613035653834356562646637633137643839653237353937
-34326230396137383531393962346137643035333834376537363865643366623932303662303839
-64373431306464363762356563663038343737356165313832613965653065326532356133646137
-30326232643334656363343533383163643130343836366430333836396463386666623465366336
-61343236303336623863303630336437613932353832623866633661653566623431653938306238
-37326630616131646634383539353234323766363633333030613937616632333432366565343537
-35613237343838333430656336313232653530373863663031386433356337643334303363666532
-65633033333463366636616138646632636534333963643864383033343463666338373630313232
-63326365666666663262383664353664623963343366626364363965626435666363306237346130
-30306661386334653137336464393465646135353436336138616563366163366664346331646666
-61613334323330663835366163623836363534666531353737656464663935333765326235306566
-61353962363034663563386137373432656166663661653861396361653930653539363333613435
-64373731616330396130363464653865303931333637393333386531303365646130646161366263
-61643234333563386338643331303164323638346639353765656632386262366666376665356164
-34393464656634303130323738353161643537613337383661343232363961613931613234643361
-34663938626364636562613332373161323932386532356261353137393533633731653034633966
-62316666613738383665303433376438303266646264353133303566636436373966343662363561
-63333662613930323666656166373763303961333332363231396532376564393966333135373966
-65623664303537376362383861663238663463396537353866666333656664323562373165646633
-35396661353639356363613834653432346539633135663565376234383866343433383339666362
-63373330336562323138343934646466373738333039346361623836653231633566316435383636
-61326535393339616639366563383662636136353162393961303362393866363436663630303762
-65343632396238623137333462633633653761336635663265326332623631393566323530623562
-36666431396532303939316662663138356631336434373732393463306336303435626232316638
-36363733383831363930666132396661633733616464393264343339333166633739346236383833
-33373962313762323430343161616361636363366334613032323637316261656333633639333066
-34653635393031323262353037396130383964363037633463653331653965363562326532623037
-383331306234353238633435663136623634

ansible/host_vars/astro-orbiter/vars.yml

@@ -2,11 +2,14 @@
 # ------------------------------------------------------------------------------
 # FILE: ansible/host_vars/astro_orbiter/vars.yml
 # HOST: astro-orbiter (10.1.71.130)
-# ROLE: Ollama inference host with AMD RX 5700 GPU passthrough
+# ROLE: llama.cpp LLM inference host — Ryzen 7 5800XT / RTX 3090 (ATX rebuild,
+#       2026-08-04). Superseded the prior AMD RX 5700 / Ollama config below;
+#       drive was transplanted into new hardware, not reinstalled.
 # ------------------------------------------------------------------------------
 
 ansible_host: 10.1.71.130
-ansible_user: wed
+ansible_user: jarvis
+ansible_ssh_private_key_file: ~/.ssh/id_jarvis
 ansible_become: true
 
 # LVM root expansion — xlarge template uses sda3 partition, standard VG/LV names
@@ -15,11 +18,3 @@ common_root_pv: /dev/sda3
 common_root_vg: ubuntu-vg
 common_root_lv: ubuntu-lv
 
-# Ollama — all defaults apply; explicitly documented here for visibility
-ollama_rocm_version: "6.2"
-ollama_default_model: "qwen3:8b"
-ollama_hsa_override_gfx_version: "10.1.0"
-ollama_data_disk: /dev/sdb
-ollama_data_vg: ollama-vg
-ollama_data_lv: ollama-lv
-ollama_data_dir: /var/lib/ollama

ansible/host_vars/main-street-station/main.yml

@@ -0,0 +1,16 @@
+---
+# Host-specific vars for main-street-station (JMRI headless server)
+# LCRR - Lake Country Railroad, Milwaukee Road Oct 1956, HO scale
+
+# JMRI profile ID — find with: ls ~/.jmri/profiles/ on the old box
+# Format: <name>.<8-char-hex>  e.g. LCRR.3d3f1dfc
+# TODO: fill in after restoring config from GitHub backup
+jmri_profile_id: ""
+
+# USB serial device for NCE command station
+# Verify after install: ls -la /dev/ttyUSB* /dev/ttyACM*
+jmri_serial_device: /dev/ttyUSB0
+
+# Path to JMRI config backup for restore task (leave empty to skip)
+# Point at a local checkout of the LCRR GitHub repo
+jmri_config_src: ""

ansible/host_vars/main-street-station/vars.yml

@@ -0,0 +1,10 @@
+---
+# main-street-station — JMRI / LCRR server
+jmri_profile_id: "Lake_Country_Railroad.3e8b1d4b"
+jmri_lcrr_repo: "ssh://git@gitea.mk-labs.cloud:2221/rblundon/LCRR.git"
+jmri_lcrr_branch: "clean-profile"
+jmri_leviton_email: "{{ leviton_email }}"
+jmri_leviton_password: "{{ leviton_password }}"
+jmri_ssh_authorized_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINnSM/9fO8rz/amqkyoGUzUKNNzzmtSXPwOCr1O9zKNO ansible"
+jmri_ssh_authorized_keys_extra:
+  - "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG6HaK4Y21UwPRbAZ986L7I9QnUdyq53114+9kO8X4bL rblundon@laptop"

ansible/inventory.yml

@@ -59,12 +59,9 @@ n8n_server:
   hosts:
     tiki-room:
 
-ollama_server:
+astro_orbiter:
   hosts:
     astro-orbiter:
-      ansible_host: 10.1.71.130
-      ansible_user: wed
-      ansible_become: true
 
 hermes_server:
   hosts:
@@ -72,6 +69,7 @@ hermes_server:
       ansible_host: 10.1.71.131
       ansible_user: wed
       ansible_become: true
+      ansible_ssh_private_key_file: ~/.ssh/ansible
 
 honcho_server:
   hosts:
@@ -80,6 +78,13 @@ honcho_server:
       ansible_user: wed
       ansible_become: true
 
+jmri_server:
+  hosts:
+    main-street-station:
+      ansible_host: 192.168.10.40
+      ansible_user: wed
+      ansible_become: true
+
 papermc_server:
 # ansible-galaxy role install engonzal.papermc
   hosts:
@@ -88,6 +93,10 @@ papermc_server:
 dev_servers:
   hosts:
     scrim:
+    backstage:
+      ansible_host: 10.1.71.133
+      ansible_user: wed
+      ansible_become: true
 
 # dhcp_server:
 #   hosts:

ansible/playbooks/day1_deploy_jmri.yml

@@ -0,0 +1,24 @@
+---
+# ============================================================================
+# day1_deploy_jmri.yml
+# ----------------------------------------------------------------------------
+# Deploys JMRI JmriFaceless headless server on main-street-station.
+# Applies linux-baseline first, then the jmri role.
+#
+# Usage:
+#   ansible-playbook playbooks/day1_deploy_jmri.yml
+#   ansible-playbook playbooks/day1_deploy_jmri.yml -e target=main-street-station
+#
+# Prerequisites:
+#   1. Host is in inventory under jmri_server group
+#   2. jmri_profile_id is set in host_vars/main-street-station.yml
+#   3. SSH access as 'wed' with sudo
+# ============================================================================
+
+- name: Deploy JMRI headless server
+  hosts: "{{ target | default('jmri_server') }}"
+  become: true
+  gather_facts: true
+  roles:
+    - linux-baseline
+    - jmri

ansible/playbooks/day1_deploy_llm_inference.yml

@@ -0,0 +1,25 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day1_deploy_llm_inference.yml
+# DESCRIPTION: Day 1 playbook for astro-orbiter LLM inference stack.
+#              Deploys vLLM + Gemma 2 27B on RTX 3090 via OCuLink.
+#
+# Usage:
+#   cd ~/git/homelab/ansible
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference.yml
+#
+# Phases (added incrementally — safe to re-run):
+#   1. Foundation  — groups, directories, vault assertion
+#   2. Driver      — nvidia-driver-595-open (idempotent; already installed)
+#   3. vLLM        — Python venv + pip install vllm
+#   4. Model       — HF login, Gemma 2 27B snapshot_download
+#   5. Serve       — systemd vllm-serve.service, health check
+#   6. Integration — Hermes provider config on carousel
+# ------------------------------------------------------------------------------
+
+- name: Deploy LLM inference stack on astro-orbiter
+  hosts: astro_orbiter
+  gather_facts: true
+
+  roles:
+    - role: llm-inference

ansible/playbooks/day1_deploy_llm_inference_multimodel.yml

@@ -0,0 +1,31 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day1_deploy_llm_inference_multimodel.yml
+# DESCRIPTION: Day 1 playbook for the dual-model (aux + tool-calling) rollout
+#              on astro-orbiter. Builds on roles/llm-inference (CUDA/driver
+#              already done) — does not replace it.
+#
+# Usage:
+#   cd ~/git/homelab/ansible
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml
+#   # or scope to specific phases:
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml --tags discover
+#
+# KNOWN GAP (2026-08-05): Semaphore is currently broken; this is being run
+# via direct ansible-playbook as an accepted interim stopgap. Retarget
+# through Semaphore once it's repaired.
+#
+# Phases (see roles/llm-inference-multimodel/README.md for detail):
+#   0. discover  — read-only; confirm existing Gemma service management
+#   1. models    — idempotent GGUF downloads (Phi-4-14B, Mistral-Small-24B)
+#   2. systemd   — deploy both unit files, do NOT auto-start
+#   3. firewall  — scope ports 8000/8001, non-0.0.0.0 bind
+#   4. verify    — start both services, smoke test, VRAM check
+# ------------------------------------------------------------------------------
+
+- name: Deploy dual-model LLM inference stack on astro-orbiter
+  hosts: astro_orbiter
+  gather_facts: true
+
+  roles:
+    - role: llm-inference-multimodel

ansible/roles/jmri/defaults/main.yml

@@ -0,0 +1,57 @@
+---
+# JMRI version to install
+# Update both jmri_version AND jmri_build_hash together when upgrading.
+# Find the build hash in the release asset filename on:
+# https://github.com/JMRI/JMRI/releases
+jmri_version: "5.16"
+jmri_build_hash: "909e15189e"
+jmri_install_dir: /opt/JMRI
+jmri_download_url: "https://github.com/JMRI/JMRI/releases/download/v{{ jmri_version }}/JMRI.{{ jmri_version }}+R{{ jmri_build_hash }}.tgz"
+
+# Service user
+jmri_user: jmri
+jmri_group: jmri
+jmri_home: /home/jmri
+
+# Profile — set per-host in host_vars
+jmri_profile_id: ""
+
+# LCRR git repo (set per-host in host_vars; leave blank to skip clone)
+jmri_lcrr_repo: ""
+jmri_lcrr_branch: "main"
+
+# SSH key for jmri user (for jmri-gui X11 access — set per-host in host_vars)
+jmri_ssh_authorized_key: ""
+jmri_ssh_authorized_keys_extra: []  # additional keys (e.g. operator laptops)
+
+# SSH key for jmri user → Gitea
+jmri_gitea_key: /home/jmri/.ssh/id_ed25519_gitea
+
+# Config restore source (legacy tar-based restore — leave blank to skip)
+jmri_config_src: ""
+
+# Ports (for documentation / firewall rules)
+jmri_json_port: 12080
+jmri_withrottle_port: 12090
+
+# ---------------------------------------------------------------------------
+# Phase 4 — Xpra virtual display
+# Replaces TigerVNC with rootless Xpra — proper window management,
+# persistent sessions, SSH-native attach (no VNC client needed).
+# Connect from macOS/Linux: xpra attach ssh://jmri@main-street-station/100
+# ---------------------------------------------------------------------------
+jmri_xpra_display: "100"
+
+# ---------------------------------------------------------------------------
+# Phase 2 — Layout power monitor (Leviton Decora Smart Wi-Fi)
+# ---------------------------------------------------------------------------
+jmri_leviton_email: ""          # set via group_vars (vault-backed)
+jmri_leviton_password: ""       # set via group_vars (vault-backed)
+jmri_leviton_switch_name: "Layout"
+jmri_monitor_poll_interval: 30  # seconds between polls (active hours)
+jmri_monitor_quiet_start: 1     # hour (24h) to stop polling
+jmri_monitor_quiet_end: 10      # hour (24h) to resume polling
+jmri_monitor_stop_delay: 30     # seconds of OFF state before stopping JMRI
+
+
+

ansible/roles/jmri/handlers/main.yml

@@ -0,0 +1,32 @@
+---
+- name: Reload udev
+  ansible.builtin.command: udevadm control --reload-rules
+  changed_when: false
+
+- name: Trigger udev
+  ansible.builtin.command: udevadm trigger --subsystem-match=tty
+  changed_when: false
+
+- name: Reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true
+
+- name: Restart jmri-monitor
+  ansible.builtin.systemd:
+    name: jmri-monitor
+    state: restarted
+
+- name: Restart jmri
+  ansible.builtin.systemd:
+    name: jmri
+    state: restarted
+
+- name: Restart jmri-xpra
+  ansible.builtin.systemd:
+    name: jmri-xpra
+    state: restarted
+
+- name: Restart sshd
+  ansible.builtin.systemd:
+    name: ssh
+    state: restarted

ansible/roles/jmri/meta/main.yml

@@ -0,0 +1,13 @@
+---
+galaxy_info:
+  role_name: jmri
+  author: JARVIS
+  description: Deploy JMRI JmriFaceless headless server as a systemd service
+  license: MIT
+  min_ansible_version: "2.12"
+  platforms:
+    - name: Ubuntu
+      versions:
+        - jammy
+        - noble
+dependencies: []

ansible/roles/jmri/tasks/main.yml

@@ -0,0 +1,388 @@
+---
+- name: Install Java runtime (full — required for GUI mode)
+  ansible.builtin.apt:
+    name:
+      - openjdk-21-jre
+      - openjdk-21-jdk
+    state: present
+    update_cache: true
+
+- name: Create JMRI system group
+  ansible.builtin.group:
+    name: "{{ jmri_group }}"
+    state: present
+    system: true
+
+- name: Create JMRI service user
+  ansible.builtin.user:
+    name: "{{ jmri_user }}"
+    group: "{{ jmri_group }}"
+    home: "{{ jmri_home }}"
+    shell: /bin/bash
+    system: true
+    create_home: true
+    state: present
+
+- name: Deploy SSH authorized key for jmri user
+  ansible.posix.authorized_key:
+    user: "{{ jmri_user }}"
+    key: "{{ jmri_ssh_authorized_key }}"
+    state: present
+  when: jmri_ssh_authorized_key | length > 0
+
+- name: Deploy extra SSH authorized keys for jmri user
+  ansible.posix.authorized_key:
+    user: "{{ jmri_user }}"
+    key: "{{ item }}"
+    state: present
+  loop: "{{ jmri_ssh_authorized_keys_extra }}"
+
+- name: Add JMRI user to dialout group (serial device access)
+  ansible.builtin.user:
+    name: "{{ jmri_user }}"
+    groups: dialout
+    append: true
+
+# ---------------------------------------------------------------------------
+# Phase 1 — Stable USB device symlinks
+# Creates /dev/jmri/loconet and /dev/jmri/nce via udev ID_SERIAL matching.
+# ---------------------------------------------------------------------------
+- name: Deploy udev rules for JMRI USB devices
+  ansible.builtin.template:
+    src: 99-jmri-devices.rules.j2
+    dest: /etc/udev/rules.d/99-jmri-devices.rules
+    owner: root
+    group: root
+    mode: '0644'
+  notify:
+    - Reload udev
+    - Trigger udev
+
+
+# ---------------------------------------------------------------------------
+# Phase 2 — LocoNet traffic monitor
+# Installs jmri-monitor: watches /dev/jmri/loconet, starts/stops jmri.service
+# ---------------------------------------------------------------------------
+- name: Install python3-venv (monitor virtualenv support)
+  ansible.builtin.apt:
+    name: python3-venv
+    state: present
+
+- name: Create virtualenv for jmri-monitor
+  ansible.builtin.command:
+    cmd: python3 -m venv /opt/jmri-monitor
+    creates: /opt/jmri-monitor/bin/python3
+
+- name: Install decora_wifi into jmri-monitor virtualenv
+  ansible.builtin.pip:
+    name: decora_wifi
+    state: present
+    virtualenv: /opt/jmri-monitor
+
+- name: Deploy jmri-monitor script
+  ansible.builtin.template:
+    src: jmri-monitor.py.j2
+    dest: /usr/local/bin/jmri-monitor
+    owner: root
+    group: root
+    mode: '0755'
+  notify: Restart jmri-monitor
+
+- name: Deploy jmri-monitor systemd unit
+  ansible.builtin.template:
+    src: jmri-monitor.service.j2
+    dest: /etc/systemd/system/jmri-monitor.service
+    owner: root
+    group: root
+    mode: '0644'
+  notify:
+    - Reload systemd
+    - Restart jmri-monitor
+
+- name: Enable jmri-monitor service
+  ansible.builtin.systemd:
+    name: jmri-monitor
+    enabled: true
+    state: started
+    daemon_reload: true
+
+# ---------------------------------------------------------------------------
+# Phase 2b — LCRR config repo (clone/pull .jmri from Gitea)
+# ---------------------------------------------------------------------------
+- name: Ensure jmri .ssh directory exists
+  ansible.builtin.file:
+    path: /home/jmri/.ssh
+    state: directory
+    owner: "{{ jmri_user }}"
+    group: "{{ jmri_group }}"
+    mode: '0700'
+
+- name: Deploy jmri SSH config for Gitea
+  ansible.builtin.copy:
+    dest: /home/jmri/.ssh/config
+    owner: "{{ jmri_user }}"
+    group: "{{ jmri_group }}"
+    mode: '0600'
+    content: |
+      Host gitea.mk-labs.cloud
+          HostName gitea.mk-labs.cloud
+          User git
+          Port 2221
+          IdentityFile {{ jmri_gitea_key }}
+          StrictHostKeyChecking accept-new
+
+- name: Clone LCRR config repo if not present
+  ansible.builtin.git:
+    repo: "{{ jmri_lcrr_repo }}"
+    dest: "{{ jmri_home }}/LCRR"
+    version: "{{ jmri_lcrr_branch }}"
+    accept_hostkey: true
+    key_file: "{{ jmri_gitea_key }}"
+    update: false
+  become_user: "{{ jmri_user }}"
+  when: jmri_lcrr_repo | length > 0
+  notify: Restart jmri
+
+- name: Remove auto-generated .jmri dir if it exists (will be replaced by symlink)
+  ansible.builtin.file:
+    path: "{{ jmri_home }}/.jmri"
+    state: absent
+  when:
+    - jmri_lcrr_repo | length > 0
+
+- name: Link .jmri config from LCRR repo
+  ansible.builtin.file:
+    src: "{{ jmri_home }}/LCRR/.jmri"
+    dest: "{{ jmri_home }}/.jmri"
+    state: link
+    owner: "{{ jmri_user }}"
+    group: "{{ jmri_group }}"
+    force: true
+  when: jmri_lcrr_repo | length > 0
+  notify: Restart jmri
+
+# ---------------------------------------------------------------------------
+# JMRI install / upgrade — version-marker pattern
+# Writes {{ jmri_install_dir }}/.jmri_installed_version after each install.
+# On subsequent runs: read the marker, skip everything if it matches
+# jmri_version. If it differs (or is absent), stop JMRI cleanly, wipe the
+# old install, download the new archive, extract, and write the new marker.
+# To upgrade: bump jmri_version + jmri_build_hash in defaults/main.yml (or
+# host_vars) and re-run the playbook.
+# ---------------------------------------------------------------------------
+
+- name: Read installed JMRI version marker (if present)
+  ansible.builtin.slurp:
+    src: "{{ jmri_install_dir }}/.jmri_installed_version"
+  register: jmri_version_marker
+  ignore_errors: true
+
+- name: Determine whether JMRI install/upgrade is needed
+  ansible.builtin.set_fact:
+    jmri_needs_install: >-
+      {{
+        jmri_version_marker is failed or
+        (jmri_version_marker.content | b64decode | trim) != jmri_version
+      }}
+
+- name: Stop JMRI service before upgrade (if running)
+  ansible.builtin.systemd:
+    name: jmri
+    state: stopped
+  failed_when: false   # service may not exist yet on first install
+  when: jmri_needs_install
+
+- name: Remove existing JMRI install directory (upgrade path)
+  ansible.builtin.file:
+    path: "{{ jmri_install_dir }}"
+    state: absent
+  when: jmri_needs_install
+
+- name: Remove stale JMRI archive from /tmp (if version changed)
+  ansible.builtin.file:
+    path: "/tmp/JMRI-{{ jmri_version }}.tgz"
+    state: absent
+  when: jmri_needs_install
+
+- name: Download JMRI release archive
+  ansible.builtin.get_url:
+    url: "{{ jmri_download_url }}"
+    dest: /tmp/JMRI-{{ jmri_version }}.tgz
+    mode: '0644'
+  when: jmri_needs_install
+
+- name: Create JMRI install directory
+  ansible.builtin.file:
+    path: "{{ jmri_install_dir }}"
+    state: directory
+    owner: "{{ jmri_user }}"
+    group: "{{ jmri_group }}"
+    mode: '0755'
+
+- name: Extract JMRI archive
+  ansible.builtin.unarchive:
+    src: /tmp/JMRI-{{ jmri_version }}.tgz
+    dest: "{{ jmri_install_dir }}"
+    remote_src: true
+    owner: "{{ jmri_user }}"
+    group: "{{ jmri_group }}"
+    extra_opts: ['--strip-components=1']
+  when: jmri_needs_install
+
+- name: Write installed version marker
+  ansible.builtin.copy:
+    content: "{{ jmri_version }}\n"
+    dest: "{{ jmri_install_dir }}/.jmri_installed_version"
+    owner: "{{ jmri_user }}"
+    group: "{{ jmri_group }}"
+    mode: '0644'
+  when: jmri_needs_install
+
+- name: Restore JMRI config from backup
+  ansible.builtin.copy:
+    src: "{{ jmri_config_src }}/"
+    dest: "{{ jmri_home }}/.jmri/"
+    owner: "{{ jmri_user }}"
+    group: "{{ jmri_group }}"
+    mode: '0644'
+    directory_mode: '0755'
+  when: jmri_config_src | length > 0
+  notify: Restart jmri
+
+- name: Deploy JmriFaceless systemd unit
+  ansible.builtin.template:
+    src: jmri.service.j2
+    dest: /etc/systemd/system/jmri.service
+    owner: root
+    group: root
+    mode: '0644'
+  notify:
+    - Reload systemd
+    - Restart jmri
+
+- name: Enable jmri service (monitor manages start/stop — do not start directly)
+  ansible.builtin.systemd:
+    name: jmri
+    enabled: false
+    daemon_reload: true
+
+# ---------------------------------------------------------------------------
+# Phase 3 — X11 remote GUI access (retained for fallback; VNC preferred)
+# ---------------------------------------------------------------------------
+- name: Install xauth (required for SSH X11 forwarding fallback)
+  ansible.builtin.apt:
+    name: xauth
+    state: present
+
+- name: Remove old jmri X11 sshd drop-in if present (renamed)
+  ansible.builtin.file:
+    path: /etc/ssh/sshd_config.d/20-jmri-x11.conf
+    state: absent
+  notify: Restart sshd
+
+- name: Deploy sshd drop-in to enable X11 forwarding (must load before hardening)
+  ansible.builtin.copy:
+    dest: /etc/ssh/sshd_config.d/09-jmri-x11.conf
+    owner: root
+    group: root
+    mode: '0644'
+    content: |
+      # Allow X11 forwarding for JMRI GUI sessions (jmri role)
+      # Must be numbered below 10-mk-labs-hardening.conf — first match wins.
+      X11Forwarding yes
+      X11UseLocalhost yes
+  notify: Restart sshd
+
+# ---------------------------------------------------------------------------
+# Phase 4 — Xpra virtual display
+# Replaces TigerVNC. Rootless mode: each JMRI window appears as a native
+# window on the client. Sessions are persistent across disconnects.
+# Connect: xpra attach ssh://jmri@main-street-station/{{ jmri_xpra_display }}
+# ---------------------------------------------------------------------------
+- name: Remove TigerVNC (replaced by Xpra)
+  ansible.builtin.apt:
+    name: tigervnc-standalone-server
+    state: absent
+  notify: Reload systemd
+
+- name: Disable and stop jmri-vnc service if present
+  ansible.builtin.systemd:
+    name: jmri-vnc
+    enabled: false
+    state: stopped
+  failed_when: false
+
+- name: Remove jmri-vnc systemd unit if present
+  ansible.builtin.file:
+    path: /etc/systemd/system/jmri-vnc.service
+    state: absent
+  notify: Reload systemd
+
+- name: Remove jmri VNC password directory if present
+  ansible.builtin.file:
+    path: "{{ jmri_home }}/.vnc"
+    state: absent
+
+- name: Install xpra.org apt signing key
+  ansible.builtin.get_url:
+    url: https://xpra.org/gpg.asc
+    dest: /usr/share/keyrings/xpra.asc
+    mode: '0644'
+
+- name: Add xpra.org upstream apt repository
+  ansible.builtin.copy:
+    dest: /etc/apt/sources.list.d/xpra.list
+    owner: root
+    group: root
+    mode: '0644'
+    content: |
+      deb [arch=amd64 signed-by=/usr/share/keyrings/xpra.asc] https://xpra.org/ noble main
+
+- name: Install Xpra from upstream repo (v6.x)
+  ansible.builtin.apt:
+    name: xpra
+    state: latest
+    update_cache: true
+
+- name: Deploy jmri-xpra systemd unit
+  ansible.builtin.template:
+    src: jmri-xpra.service.j2
+    dest: /etc/systemd/system/jmri-xpra.service
+    owner: root
+    group: root
+    mode: '0644'
+  notify:
+    - Reload systemd
+    - Restart jmri-xpra
+
+- name: Enable and start jmri-xpra service
+  ansible.builtin.systemd:
+    name: jmri-xpra
+    enabled: true
+    state: started
+    daemon_reload: true
+
+- name: Deploy jmri-gui script
+  ansible.builtin.template:
+    src: jmri-gui.j2
+    dest: /usr/local/bin/jmri-gui
+    owner: root
+    group: root
+    mode: '0755'
+
+- name: Deploy sudoers drop-in for jmri-gui (wed can manage jmri service)
+  ansible.builtin.copy:
+    dest: /etc/sudoers.d/jmri-gui
+    owner: root
+    group: root
+    mode: '0440'
+    validate: 'visudo -cf %s'
+    content: |
+      # jmri user can manage its own service (for jmri-gui interactive sessions)
+      jmri ALL=(root) NOPASSWD: /usr/bin/systemctl start jmri.service
+      jmri ALL=(root) NOPASSWD: /usr/bin/systemctl stop jmri.service
+      jmri ALL=(root) NOPASSWD: /opt/jmri-monitor/bin/python3
+      # wed retains service control for automation/admin use
+      wed ALL=(root) NOPASSWD: /usr/bin/systemctl start jmri.service
+      wed ALL=(root) NOPASSWD: /usr/bin/systemctl stop jmri.service
+      wed ALL=(root) NOPASSWD: /opt/jmri-monitor/bin/python3

ansible/roles/jmri/templates/99-jmri-devices.rules.j2

@@ -0,0 +1,15 @@
+# JMRI USB device symlinks — managed by Ansible, do not edit manually.
+# Creates stable /dev/jmri-* symlinks at the top level of /dev so JMRI
+# can enumerate them alongside real tty devices.
+
+# LocoNet interface — RR-CirKits LocoBuffer-NG (Microchip CDC)
+SUBSYSTEM=="tty", ENV{ID_SERIAL}=="RR-CirKits_LocoBuffer-NG_CDC_ACM_SERIAL_DEVICE_AA5700218A", \
+    SYMLINK+="jmri-loconet", MODE="0666"
+
+# NCE Power Pro command station (FTDI FT232)
+SUBSYSTEM=="tty", ENV{ID_SERIAL}=="ftdi_usb_serial_converter_ftDYQHZX", \
+    SYMLINK+="jmri-nce", MODE="0666"
+
+# LCC buffer (Microchip CDC)
+SUBSYSTEM=="tty", ENV{ID_SERIAL}=="Microchip_Technology_Inc._Simple_CDC_Device_Demo", \
+    SYMLINK+="jmri-lcc", MODE="0666"

ansible/roles/jmri/templates/jmri-gui.j2

@@ -0,0 +1,110 @@
+#!/bin/bash
+# jmri-gui — launch JMRI GUI on Xpra virtual display
+# Managed by Ansible — do not edit manually.
+#
+# Usage (connect as jmri user):
+#   jmri-gui panelpro       Launch PanelPro on Xpra display
+#   jmri-gui decoderpro     Launch DecoderPro on Xpra display
+#   jmri-gui status         Show service status and attach command
+#
+# Then attach from macOS/Linux:
+#   xpra attach ssh://jmri@main-street-station/{{ jmri_xpra_display }}
+
+set -euo pipefail
+
+JMRI_DIR="{{ jmri_install_dir }}"
+JMRI_SERVICE="jmri.service"
+MONITOR_SERVICE="jmri-monitor.service"
+XPRA_SERVICE="jmri-xpra.service"
+DISPLAY=":{{ jmri_xpra_display }}"
+export DISPLAY
+
+usage() {
+    echo "Usage: jmri-gui <panelpro|decoderpro|status>"
+    exit 1
+}
+
+status() {
+    echo "=== JMRI daemon ==="
+    systemctl status "$JMRI_SERVICE" --no-pager -l 2>&1 | head -8
+    echo ""
+    echo "=== Xpra display ==="
+    systemctl status "$XPRA_SERVICE" --no-pager -l 2>&1 | head -5
+    echo ""
+    echo "=== Layout monitor ==="
+    systemctl status "$MONITOR_SERVICE" --no-pager -l 2>&1 | head -5
+    echo ""
+    echo "To attach: xpra attach ssh://jmri@$(hostname -f)/{{ jmri_xpra_display }}"
+}
+
+launch() {
+    local app="$1"
+    local binary
+
+    case "$app" in
+        panelpro)   binary="PanelPro" ;;
+        decoderpro) binary="DecoderPro" ;;
+        *)          usage ;;
+    esac
+
+    # Ensure Xpra display is running
+    if ! systemctl is-active --quiet "$XPRA_SERVICE" 2>/dev/null; then
+        echo "Starting Xpra display..."
+        sudo systemctl start "$XPRA_SERVICE"
+        sleep 2
+    fi
+
+    # Stop the JMRI daemon if running (we're taking over the hardware connections)
+    if systemctl is-active --quiet "$JMRI_SERVICE" 2>/dev/null; then
+        echo "Stopping JMRI daemon..."
+        sudo systemctl stop "$JMRI_SERVICE"
+    fi
+
+    # Force AWT out of headless mode
+    export JMRI_OPTIONS="-Djava.awt.headless=false"
+
+    echo "Launching $binary on Xpra display $DISPLAY..."
+    echo ""
+    echo "Attach from your workstation:"
+    echo "  xpra attach ssh://jmri@$(hostname -f)/{{ jmri_xpra_display }}"
+    echo ""
+
+    "$JMRI_DIR/$binary" &
+
+    echo "$binary launched. Attach with xpra to see windows."
+    echo ""
+
+    # Restart daemon if layout switch is still on
+    if systemctl is-active --quiet "$MONITOR_SERVICE" 2>/dev/null; then
+        if sudo /opt/jmri-monitor/bin/python3 - <<'EOF'
+from decora_wifi import DecoraWiFiSession
+from decora_wifi.models.residential_account import ResidentialAccount
+import sys
+session = DecoraWiFiSession()
+person = session.login("{{ jmri_leviton_email }}", "{{ jmri_leviton_password }}")
+perms = person.get_residential_permissions()
+for perm in perms:
+    acct_id = perm.data.get('residentialAccountId')
+    if not acct_id:
+        continue
+    acct = ResidentialAccount(session, acct_id)
+    acct.refresh()
+    for r in acct.get_residences():
+        for s in r.get_iot_switches():
+            if s.data.get('name') == '{{ jmri_leviton_switch_name }}':
+                sys.exit(0 if s.data.get('power') == 'ON' else 1)
+sys.exit(1)
+EOF
+        then
+            echo "Layout is ON — JMRI daemon will restart when GUI is closed."
+        else
+            echo "Layout is OFF — JMRI daemon will not restart."
+        fi
+    fi
+}
+
+case "${1:-}" in
+    panelpro|decoderpro) launch "$1" ;;
+    status)              status ;;
+    *)                   usage ;;
+esac

ansible/roles/jmri/templates/jmri-monitor.py.j2

@@ -0,0 +1,156 @@
+#!/opt/jmri-monitor/bin/python3
+"""
+jmri-monitor — Leviton Decora Smart Wi-Fi layout power monitor
+Managed by Ansible — do not edit manually.
+
+Polls the Leviton cloud API for the "{{ jmri_leviton_switch_name }}" switch state.
+  - Switch ON  after being OFF → systemctl start jmri.service
+  - Switch OFF for {{ jmri_monitor_stop_delay }}s  → systemctl stop jmri.service
+
+Quiet hours {{ jmri_monitor_quiet_start }}:00–{{ jmri_monitor_quiet_end }}:00: no polling (layout assumed off).
+"""
+
+import subprocess
+import time
+import logging
+import sys
+from datetime import datetime
+
+LEVITON_EMAIL    = "{{ jmri_leviton_email }}"
+LEVITON_PASSWORD = "{{ jmri_leviton_password }}"
+SWITCH_NAME      = "{{ jmri_leviton_switch_name }}"
+POLL_INTERVAL    = {{ jmri_monitor_poll_interval }}
+QUIET_START      = {{ jmri_monitor_quiet_start }}
+QUIET_END        = {{ jmri_monitor_quiet_end }}
+STOP_DELAY       = {{ jmri_monitor_stop_delay }}
+JMRI_SERVICE     = "jmri.service"
+
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s [jmri-monitor] %(levelname)s: %(message)s",
+    datefmt="%Y-%m-%d %H:%M:%S",
+    stream=sys.stdout,
+)
+log = logging.getLogger(__name__)
+
+
+def systemctl(action):
+    try:
+        result = subprocess.run(
+            ["systemctl", action, JMRI_SERVICE],
+            capture_output=True, text=True, timeout=30,
+        )
+        if result.returncode == 0:
+            log.info("systemctl %s %s: OK", action, JMRI_SERVICE)
+        else:
+            log.warning("systemctl %s %s: %s", action, JMRI_SERVICE, result.stderr.strip())
+    except Exception as e:
+        log.error("systemctl %s failed: %s", action, e)
+
+
+def is_quiet_hours():
+    hour = datetime.now().hour
+    if QUIET_START < QUIET_END:
+        return QUIET_START <= hour < QUIET_END
+    else:
+        # wraps midnight e.g. 23–6
+        return hour >= QUIET_START or hour < QUIET_END
+
+
+def get_switch_state():
+    """Returns True if switch is ON, False if OFF, None on error."""
+    try:
+        from decora_wifi import DecoraWiFiSession
+        from decora_wifi.models.residential_account import ResidentialAccount
+
+        session = DecoraWiFiSession()
+        person = session.login(LEVITON_EMAIL, LEVITON_PASSWORD)
+        if not person:
+            log.error("Leviton login failed")
+            return None
+
+        perms = person.get_residential_permissions()
+        for perm in perms:
+            acct_id = perm.data.get('residentialAccountId')
+            if not acct_id:
+                continue
+            acct = ResidentialAccount(session, acct_id)
+            acct.refresh()
+            for residence in acct.get_residences():
+                for switch in residence.get_iot_switches():
+                    if switch.data.get('name') == SWITCH_NAME:
+                        state = switch.data.get('power', 'OFF')
+                        session.call_api('/Person/logout', {}, 'post')
+                        return state == 'ON'
+
+        all_names = []
+        for perm in perms:
+            acct_id = perm.data.get('residentialAccountId')
+            if acct_id:
+                acct = ResidentialAccount(session, acct_id)
+                acct.refresh()
+                for r in acct.get_residences():
+                    all_names += [s.data.get('name') for s in r.get_iot_switches()]
+        log.warning("Switch '%s' not found — available: %s", SWITCH_NAME, all_names)
+        session.call_api('/Person/logout', {}, 'post')
+        return None
+
+    except ImportError:
+        log.error("decora_wifi not installed")
+        return None
+    except Exception as e:
+        log.error("Error querying Leviton API: %s", e)
+        return None
+
+
+def main():
+    log.info("Layout power monitor starting")
+    log.info("Switch: '%s'  Poll: %ds  Quiet: %02d:00–%02d:00  Stop delay: %ds",
+             SWITCH_NAME, POLL_INTERVAL, QUIET_START, QUIET_END, STOP_DELAY)
+
+    layout_on = False
+    off_since = None
+
+    while True:
+        if is_quiet_hours():
+            log.debug("Quiet hours — sleeping 60s")
+            # If layout was on when quiet hours started, stop JMRI
+            if layout_on:
+                log.info("Quiet hours began — stopping JMRI")
+                layout_on = False
+                off_since = None
+                systemctl("stop")
+            time.sleep(60)
+            continue
+
+        state = get_switch_state()
+
+        if state is True:
+            off_since = None
+            if not layout_on:
+                log.info("Layout switch ON — starting JMRI")
+                layout_on = True
+                systemctl("start")
+
+        elif state is False:
+            if layout_on:
+                if off_since is None:
+                    off_since = time.monotonic()
+                    log.info("Layout switch OFF — waiting %ds before stopping JMRI", STOP_DELAY)
+                elif time.monotonic() - off_since >= STOP_DELAY:
+                    log.info("Layout switch OFF for %ds — stopping JMRI", STOP_DELAY)
+                    layout_on = False
+                    off_since = None
+                    systemctl("stop")
+            else:
+                off_since = None
+
+        else:
+            # API error — don't change state, try again next poll
+            log.warning("Could not determine switch state — retrying in %ds", POLL_INTERVAL)
+
+        time.sleep(POLL_INTERVAL)
+
+
+if __name__ == "__main__":
+    main()

ansible/roles/jmri/templates/jmri-monitor.service.j2

@@ -0,0 +1,20 @@
+[Unit]
+Description=JMRI LocoNet Traffic Monitor
+Documentation=https://www.jmri.org/
+# Must start after udev has processed devices
+After=systemd-udev-settle.service
+
+[Service]
+Type=simple
+# Runs as root — needs to call systemctl start/stop jmri.service
+User=root
+ExecStart=/usr/local/bin/jmri-monitor
+Restart=always
+RestartSec=10
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=jmri-monitor
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/jmri/templates/jmri-xpra.service.j2

@@ -0,0 +1,26 @@
+[Unit]
+Description=Xpra virtual display for JMRI (:{{ jmri_xpra_display }})
+After=network.target
+# Start before jmri.service so the display is ready when JMRI launches
+Before=jmri.service
+
+[Service]
+Type=simple
+User={{ jmri_user }}
+Group={{ jmri_group }}
+ExecStart=/usr/bin/xpra start \
+    :{{ jmri_xpra_display }} \
+    --daemon=no \
+    --mdns=no \
+    --notifications=no \
+    --systemd-run=no \
+    --pulseaudio=no \
+    --speaker=off \
+    --microphone=off \
+    --video-encoders=none \
+    --start-via-proxy=no
+Restart=on-failure
+RestartSec=5
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/jmri/templates/jmri.service.j2

@@ -0,0 +1,31 @@
+[Unit]
+Description=JMRI Server (JmriFaceless)
+Documentation=https://www.jmri.org/
+# jmri-monitor starts and stops this service based on LocoNet traffic.
+# Do NOT enable this unit directly — it is managed by jmri-monitor.service.
+After=network.target systemd-udev-settle.service
+
+[Service]
+Type=simple
+User={{ jmri_user }}
+Group={{ jmri_group }}
+WorkingDirectory={{ jmri_install_dir }}
+ExecStart={{ jmri_install_dir }}/JmriFaceless --profile={{ jmri_profile_id }}
+
+# Monitor owns lifecycle — do not auto-restart
+Restart=no
+
+# Give hardware time to settle on start
+TimeoutStartSec=30
+
+# Logging — view with: journalctl -u jmri -f
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=jmri
+
+# Serial device access
+SupplementaryGroups=dialout
+
+[Install]
+# Intentionally no WantedBy — started only by jmri-monitor
+WantedBy=

ansible/roles/llm-inference-multimodel/README.md

@@ -0,0 +1,132 @@
+# llm-inference-multimodel
+
+Deploys **two independent llama-server systemd services** on astro-orbiter's
+RTX 3090 (24GB), alongside — not replacing — the existing `llm-inference` role:
+
+| Instance | Port | Model | Quant | ctx | parallel | ~VRAM |
+|---|---|---|---|---|---|---|
+| `llama-server-aux` | 8000 | Phi-4-14B-Instruct | Q4_K_M | 8192 | 2 | ~10.0GB |
+| `llama-server-toolcall` | 8001 | Mistral-Small-24B-Instruct-2501 | Q3_K_M | 4096 | 1 | ~13.2GB |
+
+Combined estimate: **~23.2GB / 24GB** (~0.8GB headroom). See
+`/home/hermes/astro-orbiter-multi-model-plan.md` for the full approved design
+(VRAM math, model selection rationale, rollback plan, validation harness).
+
+## Relationship to `roles/llm-inference`
+
+This role does **not** replace `llm-inference`. It assumes that role's
+prerequisites are already satisfied on the host:
+
+- NVIDIA driver installed
+- `/opt/llama.cpp` cloned and built with CUDA (`/opt/llama.cpp/build/bin/llama-server` exists)
+- `jarvis` service user + `/home/jarvis` present
+
+The pre-existing single-model Gemma llama-server (however it is currently
+run) is **never modified, restarted, or deleted** by this role. It is the
+rollback target.
+
+## Phases
+
+Run the whole role, or scope with `--tags`:
+
+```
+ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml
+# or, once merged into a single play:
+ansible-playbook -i inventory.yml <playbook>.yml --tags discover,models,systemd,firewall,verify
+```
+
+0. **discover** (`tasks/discover.yml`) — READ-ONLY. Confirms via
+   `service_facts` + `pgrep` whether the existing Gemma llama-server actually
+   runs as a systemd unit today, or some ad hoc way (nohup/screen/tmux). Does
+   **not** assume a unit exists — this was an open unknown in the plan and is
+   resolved here as a fact-gathering step, not an assumption. Also records
+   baseline VRAM and current port 8000/8001 listeners for comparison later.
+
+   **If this reports no unit found**, stop and read the debug message —
+   it means plan §6's rollback story ("systemctl start the old unit to
+   revert") isn't actually available yet, and that should be fixed (codify
+   the existing process as a systemd unit) before proceeding to Phase 2.
+
+1. **models** (`tasks/models.yml`) — Idempotent GGUF download to
+   `/opt/models/` with a stat + minimum-size guard (mirrors the pattern in
+   `roles/llm-inference/tasks/serve.yml` and the `llm-inference-homelab`
+   skill), so reruns don't re-pull 8.5GB/11.7GB files or mistake a truncated
+   partial download for complete.
+
+2. **systemd** (`tasks/systemd.yml`) — Templates and deploys both unit files
+   to `/etc/systemd/system/`. **Deliberately does not start or enable either
+   service** — units land on disk as a separately reviewable checkpoint.
+   Two fully independent units (not one unit with two ExecStarts) so either
+   instance can be restarted/stopped without affecting the other.
+
+3. **firewall** (`tasks/firewall.yml`) — Scopes ports 8000 and 8001 via `ufw`
+   to `llm_allowed_source_cidr` (default the Hermes LAN subnet), rather than
+   leaving them open. Both unit templates also bind to
+   `llm_bind_address` (default `10.1.71.130`, the host's private LAN IP) —
+   **not `0.0.0.0`** — which is a deliberate change from the pre-existing
+   Gemma pattern flagged as insecure in the plan.
+
+4. **verify** (`tasks/verify.yml`) — The only phase that actually starts +
+   enables both services. Waits for `/health` on both ports, smoke-tests
+   `/v1/models` and a trivial `/v1/chat/completions` call on each, checks
+   `nvidia-smi` VRAM usage against the plan's design estimate, and greps
+   `dmesg` for OOM-kill events.
+
+   **This smoke test is not the tool-calling validation harness.** See
+   below.
+
+## Key variables
+
+Defined in `defaults/main.yml` (all overridable via `host_vars`/`group_vars`
+or `-e`):
+
+- `llm_service_user` (jarvis), `llm_binary_path`, `llm_models_dir`, `llm_bind_address`, `llm_allowed_source_cidr`
+- Aux: `llm_aux_port`, `llm_aux_model_path`, `llm_aux_model_url`, `llm_aux_ctx_size`, `llm_aux_parallel`, `llm_aux_gpu_layers`
+- Tool-calling: `llm_toolcall_port`, `llm_toolcall_model_path`, `llm_toolcall_model_url`, `llm_toolcall_ctx_size`, `llm_toolcall_parallel`, `llm_toolcall_gpu_layers`
+
+`vars/main.yml` holds constants not meant to be overridden per-host (HF token
+reference, expected-VRAM figures used only for the verify.yml report).
+
+## ⚠️ Tool-calling validation is required before use
+
+Port 8001 (Mistral-Small-24B) **must** pass the manual validation procedure
+described in plan §7 before any Claude Code / tool-calling-capable Hermes
+profile is pointed at it:
+
+1. A curl-based `tool_calls` emission probe (does it call tools correctly on
+   known trigger prompts?)
+2. A hallucination stress test (does it fabricate `tool_calls` on prompts
+   that shouldn't trigger any?)
+3. A shadow-mode period (run parallel to the existing tool-calling path,
+   compare outputs, before a hard cutover)
+
+This is **intentionally not automated into this role** — it is a
+correctness/safety judgment call, not a repeatable infra check. See
+`docs/validation-log.md` in this role directory for the procedure reference
+and a place to log results once Ryan runs it.
+
+## Known gap: Semaphore is broken (as of 2026-08-05)
+
+The normal execution/audit path (Semaphore) is currently non-functional.
+This role was authored to be run via direct `ansible-playbook` as an accepted
+interim stopgap, executed personally by Ryan. **This is a known gap, not the
+intended long-term operational path** — once Semaphore is repaired, retarget
+execution of this role (and future changes to it) through Semaphore so runs
+are audited/logged there again. Flag this in any future work that touches
+this role.
+
+## Rollback
+
+The existing Gemma llama-server and its GGUF are untouched by every phase of
+this role. To roll back:
+
+1. `systemctl stop llama-server-aux llama-server-toolcall`
+2. `systemctl disable llama-server-aux llama-server-toolcall` (optional, if reverting permanently)
+3. Confirm the original Gemma service (name determined by `discover.yml`,
+   commonly `llama-server.service`) is (still) running: `systemctl status llama-server`
+4. If it was never running because Phase 0 discovered it wasn't a managed
+   unit, whatever ad hoc process/command was used before this role's changes
+   is also unaffected — nothing in this role stopped it.
+
+No files belonging to the existing Gemma deployment (GGUF, unit file, or
+otherwise) are ever written to or deleted by this role.

ansible/roles/llm-inference-multimodel/defaults/main.yml

@@ -0,0 +1,59 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference-multimodel/defaults/main.yml
+# DESCRIPTION: Overridable defaults for the llm-inference-multimodel role.
+#              Deploy target: astro-orbiter (10.1.71.130, RTX 3090 24GB).
+#              Built ALONGSIDE roles/llm-inference (not a replacement) — that
+#              role's CUDA/build/driver phases are the prerequisite; this role
+#              assumes /opt/llama.cpp/build/bin/llama-server already exists.
+#
+#              See /home/hermes/astro-orbiter-multi-model-plan.md for the full
+#              approved design (VRAM math, rationale, rollback story).
+# ------------------------------------------------------------------------------
+
+# Shared
+llm_service_user: jarvis
+llm_binary_path: /opt/llama.cpp/build/bin/llama-server
+llm_models_dir: /opt/models
+
+# Bind address — deliberately NOT 0.0.0.0 (see plan §5). Default to the private
+# LAN interface so both instances are reachable from Hermes but not the world.
+# Override to 127.0.0.1 if even LAN-wide reachability is unwanted and a reverse
+# proxy/localhost-only tunnel is used instead.
+llm_bind_address: "10.1.71.130"
+
+# Firewall scoping (Phase 3) — subnet/hosts allowed to reach the ports above.
+# Override per-environment; default assumes Hermes runs somewhere on this /24.
+llm_allowed_source_cidr: "10.1.70.0/24"
+
+# --- Aux / classification instance (port 8000, Phi-4-14B) -------------------
+# Text-only instruction model, no tool-calling training — safe offload target
+# per the auxiliary-task-offload skill's "no tool_calls emission risk" bar.
+llm_aux_port: 8000
+llm_aux_model_path: "{{ llm_models_dir }}/phi-4-14b-instruct-Q4_K_M.gguf"
+llm_aux_model_url: "https://huggingface.co/bartowski/phi-4-GGUF/resolve/main/phi-4-Q4_K_M.gguf"
+llm_aux_model_min_bytes: 8000000000   # guard threshold; complete file ~8.5GB
+llm_aux_ctx_size: 8192
+llm_aux_parallel: 2
+llm_aux_gpu_layers: 99
+llm_aux_service_name: llama-server-aux
+llm_aux_model_id: phi-4-14b-instruct   # served model name for OpenAI-compat API
+
+# --- Tool-calling instance (port 8001, Mistral-Small-24B) --------------------
+# Native function-calling support; deployed at Q3_K_M per plan §1 Option B
+# to fit VRAM budget. MUST pass the §7 validation harness before any
+# Claude-Code-capable profile is pointed at this port.
+llm_toolcall_port: 8001
+llm_toolcall_model_path: "{{ llm_models_dir }}/mistral-small-24b-instruct-2501-Q3_K_M.gguf"
+llm_toolcall_model_url: "https://huggingface.co/bartowski/Mistral-Small-24B-Instruct-2501-GGUF/resolve/main/Mistral-Small-24B-Instruct-2501-Q3_K_M.gguf"
+llm_toolcall_model_min_bytes: 11000000000   # guard threshold; complete file ~11.7GB
+llm_toolcall_ctx_size: 4096
+llm_toolcall_parallel: 1
+llm_toolcall_gpu_layers: 99
+llm_toolcall_service_name: llama-server-toolcall
+llm_toolcall_model_id: mistral-small-24b-instruct-2501
+
+# --- Existing Gemma baseline (rollback target — never modified by this role) -
+# Populated by Phase 0 discovery (tasks/discover.yml) if not already known.
+# Set here only as a fallback name to search for; discovery is authoritative.
+llm_existing_gemma_service_name_guess: llama-server

ansible/roles/llm-inference-multimodel/docs/validation-log.md

@@ -0,0 +1,46 @@
+# Tool-Calling Model Validation Log
+
+This file tracks the manual validation procedure required by
+`astro-orbiter-multi-model-plan.md` §7 before `llama-server-toolcall` (port
+8001, Mistral-Small-24B-Instruct-2501 Q3_K_M) is trusted for any real
+tool-calling / Claude Code Hermes profile traffic.
+
+This is **not automated by the role** — `tasks/verify.yml` only confirms the
+endpoint is up and can produce a basic completion. The checks below are a
+correctness/safety judgment call that a human runs and records here.
+
+## Procedure (plan §7 summary)
+
+1. **`tool_calls` emission probe** — curl a handful of known
+   tool-triggering prompts (e.g. "what's the weather in Austin right now")
+   against `POST http://10.1.71.130:8001/v1/chat/completions` with a `tools`
+   array defined, and confirm the response actually contains a well-formed
+   `tool_calls` block (correct function name, valid JSON arguments) rather
+   than a plain-text answer or a malformed call.
+
+2. **Hallucination stress test** — send prompts that should **not** trigger
+   any tool call (general knowledge questions, casual chat, prompts that
+   merely mention a tool's name in passing) and confirm the model does
+   **not** emit a spurious `tool_calls` block. This is the primary risk
+   flagged in the plan given Mistral-Small's Q3_K_M quantization and its
+   lineage concerns around over-eager tool invocation.
+
+3. **Shadow mode** — for a bounded period, run this instance in parallel
+   with whatever tool-calling path is currently in production, comparing
+   outputs on the same real traffic (or a recorded sample) without letting
+   this instance's outputs actually drive tool execution. Only cut over
+   once outputs are consistently correct.
+
+See the `llm-inference-homelab` skill's `scripts/tool-calling-validation.sh`
+reference for a starting curl harness shape — adapt prompts/tool schemas to
+Mistral-Small's actual expected format (confirm via the GGUF's embedded
+chat template / model card) rather than assuming it matches Qwen's.
+
+## Log
+
+| Date | Run by | Probe result | Hallucination test result | Shadow mode outcome | Decision |
+|---|---|---|---|---|---|
+| _(pending)_ | | | | | Not yet cut over — do not point production tool-calling traffic at :8001 |
+
+Update this table after each validation pass. Do not remove prior rows —
+this is the audit trail for "when did we decide this was safe to use."

ansible/roles/llm-inference-multimodel/handlers/main.yml

@@ -0,0 +1,27 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference-multimodel/handlers/main.yml
+# DESCRIPTION: Separate restart handlers per instance — NEVER combined, so a
+#              content change to one unit template never restarts the other
+#              (plan §2/§6 requirement: independent restart/rollback).
+# ------------------------------------------------------------------------------
+
+- name: Reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true
+  become: true
+  listen: "reload systemd"
+
+- name: Restart llama-server-aux
+  ansible.builtin.systemd:
+    name: "{{ llm_aux_service_name }}"
+    state: restarted
+  become: true
+  listen: "restart llama-server-aux"
+
+- name: Restart llama-server-toolcall
+  ansible.builtin.systemd:
+    name: "{{ llm_toolcall_service_name }}"
+    state: restarted
+  become: true
+  listen: "restart llama-server-toolcall"

ansible/roles/llm-inference-multimodel/meta/main.yml

@@ -0,0 +1,17 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference-multimodel/meta/main.yml
+# ------------------------------------------------------------------------------
+galaxy_info:
+  role_name: llm_inference_multimodel
+  author: rblundon
+  license: MIT
+  description: >
+    Deploys two independent llama-server instances on astro-orbiter's RTX 3090:
+    an aux/classification instance (Phi-4-14B Q4_K_M, port 8000) and a
+    tool-calling instance (Mistral-Small-24B-Instruct-2501 Q3_K_M, port 8001).
+    Built alongside roles/llm-inference (not a replacement); assumes that
+    role's CUDA build/driver work is already done. See
+    /home/hermes/astro-orbiter-multi-model-plan.md for the full design.
+  min_ansible_version: "2.15"
+dependencies: []

ansible/roles/llm-inference-multimodel/tasks/discover.yml

@@ -0,0 +1,87 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference-multimodel/tasks/discover.yml
+# DESCRIPTION: Phase 0 — READ-ONLY fact gathering on how the existing Gemma
+#              llama-server is actually managed on astro-orbiter TODAY.
+#
+#              Per plan §0/§4: "Service management: unverified — plan requires
+#              confirming systemd unit exists before touching anything.
+#              Do not assume." This file performs that confirmation. It makes
+#              NO changes to the host — no `state: present/started/stopped`,
+#              no file writes, no service actions. Every task here is either
+#              a `_facts` module, a `command`/`shell` in check-safe read mode,
+#              or a `stat`.
+#
+#              Outcomes recorded as facts for later phases/for a human to read
+#              in the play recap — this file does not branch role behavior
+#              on the result (that would be over-engineering a role meant to
+#              run once); it surfaces what's true so a human confirms before
+#              Phase 2 proceeds.
+# ------------------------------------------------------------------------------
+
+- name: Gather service facts (systemd unit inventory)
+  ansible.builtin.service_facts:
+
+- name: Determine whether a systemd unit matching the existing Gemma service exists
+  ansible.builtin.set_fact:
+    llm_existing_gemma_unit_found: "{{ (llm_existing_gemma_service_name_guess + '.service') in ansible_facts.services }}"
+
+- name: Report existing Gemma systemd unit state (if found)
+  ansible.builtin.debug:
+    msg: >-
+      Existing unit '{{ llm_existing_gemma_service_name_guess }}.service' found:
+      state={{ ansible_facts.services[llm_existing_gemma_service_name_guess + '.service'].state | default('unknown') }},
+      status={{ ansible_facts.services[llm_existing_gemma_service_name_guess + '.service'].status | default('unknown') }}
+  when: llm_existing_gemma_unit_found
+
+- name: WARNING — no systemd unit found matching the existing Gemma service
+  ansible.builtin.debug:
+    msg: >-
+      No systemd unit named '{{ llm_existing_gemma_service_name_guess }}.service'
+      was found via service_facts. This means the current single-model
+      llama-server is likely run some other way (manual nohup, screen/tmux,
+      or a differently-named unit). DO NOT PROCEED to Phase 2 assuming a
+      clean rollback target exists. Before continuing: (1) check for any
+      running llama-server process via `ansible -m command -a "pgrep -fa
+      llama-server"`, (2) if found running ad hoc, codify it as a proper
+      systemd unit FIRST (reusing roles/llm-inference's existing
+      llama-server.service.j2 pattern) so plan §6's rollback story
+      ("systemctl start llama-server-gemma to fully revert") is real and
+      not aspirational. This is a human decision point, not something this
+      role auto-remediates.
+  when: not llm_existing_gemma_unit_found
+
+- name: Check for any running llama-server process (read-only, no state change)
+  ansible.builtin.command:
+    cmd: pgrep -fa llama-server
+  register: llm_existing_process_check
+  changed_when: false
+  failed_when: false   # pgrep exits 1 with no matches — not a failure condition here
+
+- name: Report any llama-server process found running outside systemd
+  ansible.builtin.debug:
+    msg: "Running llama-server process(es): {{ llm_existing_process_check.stdout_lines }}"
+  when: llm_existing_process_check.rc == 0
+
+- name: Check current GPU VRAM utilization (baseline, before any changes)
+  ansible.builtin.command:
+    cmd: nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader
+  register: llm_baseline_vram
+  changed_when: false
+  failed_when: false
+
+- name: Report baseline VRAM usage
+  ansible.builtin.debug:
+    msg: "Baseline GPU VRAM (before this role's changes): {{ llm_baseline_vram.stdout | default('nvidia-smi unavailable') }}"
+
+- name: Check whether ports 8000/8001 are already bound (avoid port collision surprises)
+  ansible.builtin.command:
+    cmd: "ss -ltnp"
+  register: llm_existing_listeners
+  changed_when: false
+  failed_when: false
+
+- name: Report current listeners on 8000/8001
+  ansible.builtin.debug:
+    msg: "{{ llm_existing_listeners.stdout_lines | select('search', ':(8000|8001)\\s') | list }}"
+  when: llm_existing_listeners.rc == 0

ansible/roles/llm-inference-multimodel/tasks/firewall.yml

@@ -0,0 +1,64 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference-multimodel/tasks/firewall.yml
+# DESCRIPTION: Phase 3 — scope :8001 (new) and reconsider :8000 (existing
+#              pattern) exposure, per plan §5.
+#
+#              Current baseline pattern (0.0.0.0:8000, no auth) is a
+#              pre-existing flagged issue — this role does NOT repeat it
+#              uncritically for the new port, and tightens both:
+#              1. Bind address: handled in systemd.yml templates via
+#                 {{ llm_bind_address }} (default 10.1.71.130, NOT 0.0.0.0).
+#              2. Firewall: ufw rules scoping both ports to
+#                 {{ llm_allowed_source_cidr }} rather than open LAN-wide.
+#
+#              Idempotent: named rule comments + `state: present` so reruns
+#              don't duplicate rules (per plan §4 idempotency note).
+# ------------------------------------------------------------------------------
+
+- name: Check whether ufw is installed/active
+  ansible.builtin.command:
+    cmd: ufw status
+  register: llm_ufw_status
+  changed_when: false
+  failed_when: false
+  become: true
+
+- name: WARNING — ufw not active, firewall scoping cannot be applied
+  ansible.builtin.debug:
+    msg: >-
+      ufw does not appear to be active on this host (`ufw status` returned:
+      {{ llm_ufw_status.stdout | default('n/a') }}). Firewall scoping for
+      ports {{ llm_aux_port }}/{{ llm_toolcall_port }} was skipped. This is a
+      gap vs plan §5 item 2 — flag to Ryan before relying on bind-address
+      alone for exposure control.
+  when: "'Status: active' not in (llm_ufw_status.stdout | default(''))"
+
+- name: Allow aux port ({{ llm_aux_port }}) from the Hermes source subnet
+  community.general.ufw:
+    rule: allow
+    port: "{{ llm_aux_port | string }}"
+    proto: tcp
+    src: "{{ llm_allowed_source_cidr }}"
+    comment: "llm-inference-multimodel: aux (Phi-4) — scoped to Hermes subnet"
+  become: true
+  when: "'Status: active' in (llm_ufw_status.stdout | default(''))"
+
+- name: Allow tool-calling port ({{ llm_toolcall_port }}) from the Hermes source subnet
+  community.general.ufw:
+    rule: allow
+    port: "{{ llm_toolcall_port | string }}"
+    proto: tcp
+    src: "{{ llm_allowed_source_cidr }}"
+    comment: "llm-inference-multimodel: toolcall (Mistral-Small) — scoped to Hermes subnet"
+  become: true
+  when: "'Status: active' in (llm_ufw_status.stdout | default(''))"
+
+- name: Report firewall scoping applied
+  ansible.builtin.debug:
+    msg: >-
+      Firewall scoping applied for ports {{ llm_aux_port }} and
+      {{ llm_toolcall_port }}, restricted to source {{ llm_allowed_source_cidr }}.
+      Reverse-proxy + API-key enforcement (plan §5 item 3) is NOT implemented
+      by this role — flagged as an optional follow-up phase, not bundled into
+      this minimum-viable rollout.

ansible/roles/llm-inference-multimodel/tasks/main.yml

@@ -0,0 +1,35 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference-multimodel/tasks/main.yml
+# DESCRIPTION: Entry point — imports one task file per phase.
+#              Phases are additive; re-running the full playbook is always
+#              safe (idempotent). Use --tags to run a specific phase subset:
+#                --tags discover,models,systemd,firewall,verify
+#
+# IMPORTANT: Phase 2 (systemd) deploys but does NOT start either service.
+#            Phase 4 (verify) is what starts + smoke-tests them. This lets
+#            Ryan review "systemd units land, nothing running yet" as a
+#            distinct, revertable checkpoint before anything touches the
+#            live GPU/VRAM state.
+# ------------------------------------------------------------------------------
+
+# Phase 0 — Discover (read-only; confirm how the existing Gemma llama-server
+# is actually managed today before assuming a systemd unit exists)
+- import_tasks: discover.yml
+  tags: [discover]
+
+# Phase 1 — Models (idempotent GGUF download, size-check guard)
+- import_tasks: models.yml
+  tags: [models]
+
+# Phase 2 — Systemd (template + deploy both unit files, do NOT auto-start)
+- import_tasks: systemd.yml
+  tags: [systemd]
+
+# Phase 3 — Firewall (scope :8001 and reconsider :8000 exposure)
+- import_tasks: firewall.yml
+  tags: [firewall]
+
+# Phase 4 — Verify (start both services, curl smoke test, nvidia-smi VRAM check)
+- import_tasks: verify.yml
+  tags: [verify]

ansible/roles/llm-inference-multimodel/tasks/models.yml

@@ -0,0 +1,72 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference-multimodel/tasks/models.yml
+# DESCRIPTION: Phase 1 — download both GGUFs to {{ llm_models_dir }}.
+#              Idempotent: reuses the stat + size-threshold guard pattern
+#              from the llm-inference-homelab skill / roles/llm-inference's
+#              serve.yml, so reruns don't re-pull 8.5GB / 11.7GB files.
+#
+#              Does NOT touch the existing Gemma GGUF — separate directory
+#              entries, no overlap, no deletion of anything pre-existing.
+# ------------------------------------------------------------------------------
+
+- name: Create models directory
+  ansible.builtin.file:
+    path: "{{ llm_models_dir }}"
+    state: directory
+    owner: "{{ llm_service_user }}"
+    group: "{{ llm_service_user }}"
+    mode: "0755"
+  become: true
+
+# --- Aux model (Phi-4-14B Q4_K_M) --------------------------------------------
+
+- name: Check if aux model GGUF already exists
+  ansible.builtin.stat:
+    path: "{{ llm_aux_model_path }}"
+  register: llm_aux_model_stat
+
+- name: Download aux model — Phi-4-14B-Q4_K_M GGUF
+  ansible.builtin.get_url:
+    url: "{{ llm_aux_model_url }}"
+    dest: "{{ llm_aux_model_path }}"
+    headers:
+      Authorization: "Bearer {{ llm_hf_token }}"
+    owner: "{{ llm_service_user }}"
+    group: "{{ llm_service_user }}"
+    mode: "0644"
+    timeout: 7200
+    force: false
+  become: true
+  no_log: true
+  # Idempotency guard: skip if file exists and is above the min-size threshold
+  # (catches partial/truncated downloads from an interrupted prior run).
+  when: not llm_aux_model_stat.stat.exists or (llm_aux_model_stat.stat.size | int) < (llm_aux_model_min_bytes | int)
+
+# --- Tool-calling model (Mistral-Small-24B Q3_K_M) ---------------------------
+
+- name: Check if tool-calling model GGUF already exists
+  ansible.builtin.stat:
+    path: "{{ llm_toolcall_model_path }}"
+  register: llm_toolcall_model_stat
+
+- name: Download tool-calling model — Mistral-Small-24B-Instruct-2501 Q3_K_M GGUF
+  ansible.builtin.get_url:
+    url: "{{ llm_toolcall_model_url }}"
+    dest: "{{ llm_toolcall_model_path }}"
+    headers:
+      Authorization: "Bearer {{ llm_hf_token }}"
+    owner: "{{ llm_service_user }}"
+    group: "{{ llm_service_user }}"
+    mode: "0644"
+    timeout: 7200
+    force: false
+  become: true
+  no_log: true
+  when: not llm_toolcall_model_stat.stat.exists or (llm_toolcall_model_stat.stat.size | int) < (llm_toolcall_model_min_bytes | int)
+
+- name: Report model files present on disk
+  ansible.builtin.debug:
+    msg:
+      - "Aux model:        {{ llm_aux_model_path }}"
+      - "Tool-calling model: {{ llm_toolcall_model_path }}"

ansible/roles/llm-inference-multimodel/tasks/systemd.yml

@@ -0,0 +1,54 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference-multimodel/tasks/systemd.yml
+# DESCRIPTION: Phase 2 — template + deploy both unit files.
+#              DELIBERATELY DOES NOT START OR ENABLE either service — that is
+#              Phase 4 (verify.yml)'s job, after Phase 3 firewall scoping is
+#              in place. This keeps "units land on disk" and "processes
+#              actually bind ports and load 20+GB into VRAM" as separately
+#              reviewable checkpoints per Ryan's iterative-build preference.
+#
+#              Two independent units (llama-server-aux.service,
+#              llama-server-toolcall.service) — NOT one unit with two
+#              ExecStarts — so either can be stopped/restarted without
+#              affecting the other (plan §2, §6 rollback requirement).
+#
+#              The pre-existing Gemma unit (whatever discover.yml found it to
+#              be) is never templated, restarted, or disabled by this file.
+# ------------------------------------------------------------------------------
+
+- name: Deploy llama-server-aux systemd unit
+  ansible.builtin.template:
+    src: llama-server-aux.service.j2
+    dest: "/etc/systemd/system/{{ llm_aux_service_name }}.service"
+    owner: root
+    group: root
+    mode: "0644"
+  become: true
+  notify:
+    - reload systemd
+    - restart llama-server-aux
+
+- name: Deploy llama-server-toolcall systemd unit
+  ansible.builtin.template:
+    src: llama-server-toolcall.service.j2
+    dest: "/etc/systemd/system/{{ llm_toolcall_service_name }}.service"
+    owner: root
+    group: root
+    mode: "0644"
+  become: true
+  notify:
+    - reload systemd
+    - restart llama-server-toolcall
+
+- name: Flush handlers so daemon-reload lands before any later phase acts on unit state
+  ansible.builtin.meta: flush_handlers
+
+# NOTE: no `ansible.builtin.systemd: state: started / enabled: true` task here
+# on purpose. Units exist on disk after this phase; nothing is running.
+# The "restart" handlers above only fire (and thus only start anything) if
+# the template content actually changed AND a later flush_handlers/end-of-play
+# triggers them — on a first-ever apply this DOES start the services once,
+# which is expected/acceptable for a fresh deploy, but on any subsequent
+# re-run with no template changes, nothing restarts. Ryan/verify.yml owns
+# the deliberate first start + smoke test.

ansible/roles/llm-inference-multimodel/tasks/verify.yml

@@ -0,0 +1,133 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference-multimodel/tasks/verify.yml
+# DESCRIPTION: Phase 4 — start both services, curl smoke test each endpoint,
+#              nvidia-smi VRAM check against plan §1 math, confirm no OOM.
+#
+#              This is the ONLY phase that actually starts the services
+#              (systemd.yml deliberately does not). Enabling happens here too,
+#              so a reboot brings both back — matching plan §2's "independent
+#              systemd services" intent for durability, not just this-session.
+# ------------------------------------------------------------------------------
+
+- name: Enable and start llama-server-aux
+  ansible.builtin.systemd:
+    name: "{{ llm_aux_service_name }}"
+    state: started
+    enabled: true
+    daemon_reload: true
+  become: true
+
+- name: Enable and start llama-server-toolcall
+  ansible.builtin.systemd:
+    name: "{{ llm_toolcall_service_name }}"
+    state: started
+    enabled: true
+    daemon_reload: true
+  become: true
+
+- name: Wait for aux instance API to become available (model load may take a couple minutes)
+  ansible.builtin.uri:
+    url: "http://{{ llm_bind_address }}:{{ llm_aux_port }}/health"
+    status_code: 200
+  register: llm_aux_health
+  retries: 24
+  delay: 10
+  until: llm_aux_health.status == 200
+
+- name: Wait for tool-calling instance API to become available
+  ansible.builtin.uri:
+    url: "http://{{ llm_bind_address }}:{{ llm_toolcall_port }}/health"
+    status_code: 200
+  register: llm_toolcall_health
+  retries: 24
+  delay: 10
+  until: llm_toolcall_health.status == 200
+
+- name: Smoke-test — aux instance model listing
+  ansible.builtin.uri:
+    url: "http://{{ llm_bind_address }}:{{ llm_aux_port }}/v1/models"
+    status_code: 200
+    return_content: true
+  register: llm_aux_models
+
+- name: Smoke-test — tool-calling instance model listing
+  ansible.builtin.uri:
+    url: "http://{{ llm_bind_address }}:{{ llm_toolcall_port }}/v1/models"
+    status_code: 200
+    return_content: true
+  register: llm_toolcall_models
+
+- name: Report served models per instance
+  ansible.builtin.debug:
+    msg:
+      - "Aux (:{{ llm_aux_port }}) serving:        {{ llm_aux_models.json.data | map(attribute='id') | list }}"
+      - "Tool-calling (:{{ llm_toolcall_port }}) serving: {{ llm_toolcall_models.json.data | map(attribute='id') | list }}"
+
+- name: Basic completion smoke test — aux instance (non-tool-calling sanity check only)
+  ansible.builtin.uri:
+    url: "http://{{ llm_bind_address }}:{{ llm_aux_port }}/v1/chat/completions"
+    method: POST
+    body_format: json
+    body:
+      model: "{{ llm_aux_model_id }}"
+      messages:
+        - role: user
+          content: "Reply with exactly one word: OK"
+      max_tokens: 10
+    status_code: 200
+    return_content: true
+  register: llm_aux_completion
+
+- name: Basic completion smoke test — tool-calling instance (plain-text sanity check only)
+  ansible.builtin.uri:
+    url: "http://{{ llm_bind_address }}:{{ llm_toolcall_port }}/v1/chat/completions"
+    method: POST
+    body_format: json
+    body:
+      model: "{{ llm_toolcall_model_id }}"
+      messages:
+        - role: user
+          content: "Reply with exactly one word: OK"
+      max_tokens: 10
+    status_code: 200
+    return_content: true
+  register: llm_toolcall_completion
+
+- name: NOTE — this smoke test is NOT the tool-calling validation harness
+  ansible.builtin.debug:
+    msg: >-
+      Both endpoints respond to basic completions. This does NOT validate
+      tool_calls correctness or hallucination-safety for the tool-calling
+      instance — that is a separate, manual, post-deploy procedure (plan §7).
+      See references/tool-calling-validation.sh (copied from the
+      llm-inference-homelab skill) and docs/validation-log.md in this role.
+      DO NOT point any Claude Code / tool-calling-capable Hermes profile at
+      port {{ llm_toolcall_port }} until that validation has passed and been
+      logged.
+
+- name: Check GPU VRAM usage after both instances are running
+  ansible.builtin.command:
+    cmd: nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv,noheader
+  register: llm_post_start_vram
+  changed_when: false
+
+- name: Report VRAM usage vs plan §1 expectations
+  ansible.builtin.debug:
+    msg:
+      - "Measured (nvidia-smi):      {{ llm_post_start_vram.stdout }}"
+      - "Design estimate (plan §1):  aux ~{{ llm_aux_expected_vram_gb }}GB + toolcall ~{{ llm_toolcall_expected_vram_gb }}GB = ~{{ llm_combined_expected_vram_gb }}GB / {{ llm_gpu_total_vram_gb }}GB total"
+      - "If measured usage exceeds ~23.5GB or is within ~0.5GB of the 24GB card limit, treat as the OOM-risk trigger condition from plan §6 — do not leave both services running unattended without confirming headroom."
+
+- name: Check for OOM-kill events related to llama-server in dmesg (best-effort, read-only)
+  ansible.builtin.shell:
+    cmd: "dmesg | grep -i 'llama-server' | grep -i -E 'oom|killed' || true"
+  register: llm_oom_check
+  changed_when: false
+  become: true
+
+- name: Report any OOM-kill findings
+  ansible.builtin.debug:
+    msg: >-
+      {{ llm_oom_check.stdout if llm_oom_check.stdout | length > 0
+         else 'No OOM-kill events found for llama-server in dmesg.' }}

ansible/roles/llm-inference-multimodel/templates/llama-server-aux.service.j2

@@ -0,0 +1,33 @@
+[Unit]
+Description=llama-server (aux/classification) — Phi-4-14B Q4_K_M (OpenAI-compatible inference)
+After=network.target nvidia-persistenced.service
+Wants=nvidia-persistenced.service
+
+[Service]
+Type=simple
+User={{ llm_service_user }}
+Group={{ llm_service_user }}
+Environment="HOME=/home/{{ llm_service_user }}"
+ExecStart={{ llm_binary_path }} \
+    --model {{ llm_aux_model_path }} \
+    --host {{ llm_bind_address }} \
+    --port {{ llm_aux_port }} \
+    --ctx-size {{ llm_aux_ctx_size }} \
+    --n-gpu-layers {{ llm_aux_gpu_layers }} \
+    --parallel {{ llm_aux_parallel }} \
+    --metrics
+# NOTE: no --chat-template flag — let llama-server auto-detect Phi-4's own
+# embedded chat template from GGUF metadata (same reasoning as the existing
+# llm-inference role's Gemma unit: explicit overrides risk mismatching the
+# model's actual expected format).
+# NOTE: --host is the private LAN IP (10.1.71.130 by default), NOT 0.0.0.0 —
+# deliberate change from the pre-existing Gemma pattern (plan §5).
+Restart=on-failure
+RestartSec=10
+TimeoutStartSec=600
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=llama-server-aux
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/llm-inference-multimodel/templates/llama-server-toolcall.service.j2

@@ -0,0 +1,38 @@
+[Unit]
+Description=llama-server (tool-calling) — Mistral-Small-24B-Instruct-2501 Q3_K_M (OpenAI-compatible inference)
+After=network.target nvidia-persistenced.service
+Wants=nvidia-persistenced.service
+
+[Service]
+Type=simple
+User={{ llm_service_user }}
+Group={{ llm_service_user }}
+Environment="HOME=/home/{{ llm_service_user }}"
+ExecStart={{ llm_binary_path }} \
+    --model {{ llm_toolcall_model_path }} \
+    --host {{ llm_bind_address }} \
+    --port {{ llm_toolcall_port }} \
+    --ctx-size {{ llm_toolcall_ctx_size }} \
+    --n-gpu-layers {{ llm_toolcall_gpu_layers }} \
+    --parallel {{ llm_toolcall_parallel }} \
+    --metrics
+# NOTE: no --chat-template flag — let llama-server auto-detect Mistral-Small's
+# own embedded chat template from GGUF metadata.
+# NOTE: --host is the private LAN IP (10.1.71.130 by default), NOT 0.0.0.0.
+# NOTE: --parallel 1 is deliberate (plan §1/§2) — tool-calling profiles are
+# single-session-at-a-time per Claude Code profile; lower parallelism reduces
+# KV overhead and lowers hallucination surface from context bleed between
+# concurrent slots.
+# IMPORTANT: this endpoint MUST pass the plan §7 validation harness
+# (docs/validation-log.md in this role) before any Claude Code / tool-calling
+# Hermes profile is pointed at it. Mistral-Small shares lineage concerns
+# flagged for Qwen2.5/Qwen3 hallucinated tool_calls — do not assume safety.
+Restart=on-failure
+RestartSec=10
+TimeoutStartSec=600
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=llama-server-toolcall
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/llm-inference-multimodel/vars/main.yml

@@ -0,0 +1,22 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference-multimodel/vars/main.yml
+# DESCRIPTION: Role-internal constants (not meant to be overridden per-host).
+#              Model URLs/quant filenames live here rather than defaults/ since
+#              they're not really "tunable" — they're the specific artifacts
+#              named in the approved plan (§1). If Ryan wants a different
+#              quant/model, that's a defaults/main.yml override or a plan
+#              revision, not a vars/ edit.
+# ------------------------------------------------------------------------------
+
+# HuggingFace auth — reuse the same vault token as roles/llm-inference.
+# vault_hf_token is defined in group_vars/all/vault.
+llm_hf_token: "{{ vault_hf_token }}"
+
+# Expected VRAM subtotals from plan §1 (informational — surfaced in verify.yml
+# output so a live nvidia-smi reading can be sanity-checked against the design
+# math, not enforced as a hard gate).
+llm_aux_expected_vram_gb: 10.0
+llm_toolcall_expected_vram_gb: 13.2
+llm_combined_expected_vram_gb: 23.2
+llm_gpu_total_vram_gb: 24.0

ansible/roles/llm-inference/defaults/main.yml

@@ -0,0 +1,55 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/defaults/main.yml
+# DESCRIPTION: Overridable defaults for the llm-inference role.
+#              Deploy target: astro-orbiter (Dell OptiPlex 7050 SFF, RTX 3090
+#              via OCuLink, Ubuntu 24.04.4 LTS).
+# ------------------------------------------------------------------------------
+
+# NVIDIA driver
+llm_nvidia_driver_package: nvidia-driver-595-open
+
+# Python venv
+llm_venv_path: /home/jarvis/vllm-env
+llm_venv_owner: jarvis
+
+# HuggingFace
+llm_hf_cache_dir: /home/jarvis/.cache/huggingface
+llm_hf_model: google/gemma-2-27b-it
+
+# vLLM serve (deprecated — replaced by llama-server)
+# llama-server serve
+llm_serve_port: 8000
+llm_serve_host: "0.0.0.0"
+# NOTE (2026-08-05): --ctx-size is llama.cpp's TOTAL KV cache pool, divided
+# evenly across --parallel slots (per-slot context = ctx-size / parallel).
+# Previous 8192/4=2048 tokens-per-slot was too small for aux task offload
+# (context compression) and caused live rejections: "request (3826 tokens)
+# exceeds the available context size (2048 tokens)".
+# Sized against measured VRAM on astro-orbiter (RTX 3090, 24576MiB total):
+#   - Weights (Q4_K_M, 27B) ~16998MiB resident.
+#   - At ctx-size=8192/parallel=4, total llama-server VRAM = 19404MiB
+#     (nvidia-smi), i.e. ~2406MiB for KV cache + compute buffers at 8192
+#     total context tokens -> ~294KiB/token (pool-wide, incl. buffers).
+#   - Model n_ctx_train=8192 is the native max; per-slot context beyond
+#     this degrades coherence, so per-slot should cap at 8192.
+#   - New sizing: ctx-size=16384, parallel=2 -> 8192 tokens/slot (native
+#     max, covers compression's multi-thousand-token inputs with margin).
+#     Projected VRAM: 16998 + (~294KiB/token * 16384) ≈ 21.8GB used,
+#     leaving ~2.7GB headroom on the 24GB card.
+#   - parallel=2 (not 4) trades some concurrency for correct per-slot
+#     context; 2 concurrent aux-task requests is enough headroom before
+#     the known "3+ simultaneous compressions" GPU bottleneck kicks in.
+llm_max_model_len: 16384
+llm_gpu_layers: 99              # offload all layers to GPU
+llm_parallel_slots: 2           # concurrent request slots -> 8192 tokens/slot (ctx-size / parallel)
+llm_gguf_dir: /home/jarvis/models
+llm_gguf_path: /home/jarvis/models/gemma-2-27b-it-Q4_K_M.gguf
+
+# Legacy vLLM vars (kept for role documentation, not used by llama-server)
+llm_quantization: "bitsandbytes"
+llm_gpu_memory_utilization: "0.92"
+
+# Monitoring
+llm_gpu_exporter_version: "1.13.1"
+llm_gpu_exporter_port: 9835

ansible/roles/llm-inference/handlers/main.yml

@@ -0,0 +1,42 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/handlers/main.yml
+# ------------------------------------------------------------------------------
+
+- name: Reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true
+  listen: "reload systemd"
+
+- name: Restart vllm-serve
+  ansible.builtin.systemd:
+    name: vllm-serve
+    state: restarted
+  listen: "restart vllm-serve"
+  failed_when: false
+
+- name: Restart llama-server
+  ansible.builtin.systemd:
+    name: llama-server
+    state: restarted
+  listen: "restart llama-server"
+
+- name: Restart Hermes on carousel
+  ansible.builtin.systemd:
+    name: "{{ item }}"
+    state: restarted
+    scope: user
+  loop:
+    - hermes-gateway
+    - hermes-dashboard
+  become: true
+  become_user: wed
+  delegate_to: carousel-of-progress
+  listen: "restart hermes"
+  ignore_errors: true
+
+- name: Restart nvidia-gpu-exporter
+  ansible.builtin.systemd:
+    name: nvidia-gpu-exporter
+    state: restarted
+  listen: "restart nvidia-gpu-exporter"

ansible/roles/llm-inference/meta/main.yml

@@ -0,0 +1,14 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/meta/main.yml
+# ------------------------------------------------------------------------------
+galaxy_info:
+  role_name: llm_inference
+  author: rblundon
+  license: MIT
+  description: >
+    Deploys vLLM serving stack with NVIDIA RTX 3090 on Ubuntu 24.04.
+    Manages NVIDIA drivers, Python venv, model download, systemd service,
+    and Hermes provider integration.
+  min_ansible_version: "2.15"
+dependencies: []

ansible/roles/llm-inference/tasks/driver.yml

@@ -0,0 +1,25 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/driver.yml
+# DESCRIPTION: Phase 2 — NVIDIA driver.
+#              Installs nvidia-driver-595-open via apt. Fully idempotent —
+#              already installed on astro-orbiter on 2026-08-03, this is a no-op.
+#              DKMS builds the kernel module automatically on install.
+# ------------------------------------------------------------------------------
+
+- name: Install NVIDIA driver package
+  ansible.builtin.apt:
+    name: "{{ llm_nvidia_driver_package }}"
+    state: present
+    update_cache: false
+  notify: reload systemd
+
+- name: Verify nvidia-smi reports the GPU
+  ansible.builtin.command: nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader
+  register: nvidia_smi_out
+  changed_when: false
+  failed_when: nvidia_smi_out.rc != 0
+
+- name: Print nvidia-smi output
+  ansible.builtin.debug:
+    msg: "GPU detected: {{ nvidia_smi_out.stdout }}"

ansible/roles/llm-inference/tasks/foundation.yml

@@ -0,0 +1,45 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/foundation.yml
+# DESCRIPTION: Phase 1 — Foundation.
+#              - Asserts vault secret is defined
+#              - Adds jarvis user to nvidia GPU groups
+#              - Creates HuggingFace cache directory
+#              - Creates venv parent directory
+# NOTE: NVIDIA driver install (Phase 2) already performed manually on
+#       2026-08-03 (nvidia-driver-595-open, DKMS built, nvidia-smi verified).
+#       Phase 2 tasks are idempotent and will no-op on astro-orbiter.
+# ------------------------------------------------------------------------------
+
+- name: Assert HuggingFace token is defined in vault
+  ansible.builtin.assert:
+    that:
+      - vault_hf_token is defined
+      - vault_hf_token | length > 0
+    fail_msg: >
+      vault_hf_token is not defined. Add it to group_vars/all/vault:
+        vault_hf_token: "hf_xxxxxxxxxxxxxxxxxxxx"
+
+- name: Add jarvis user to nvidia GPU groups
+  ansible.builtin.user:
+    name: jarvis
+    groups:
+      - video
+      - render
+    append: true
+
+- name: Create HuggingFace cache directory
+  ansible.builtin.file:
+    path: "{{ llm_hf_cache_dir }}"
+    state: directory
+    owner: jarvis
+    group: jarvis
+    mode: "0755"
+
+- name: Create venv parent directory
+  ansible.builtin.file:
+    path: "{{ llm_venv_path | dirname }}"
+    state: directory
+    owner: jarvis
+    group: jarvis
+    mode: "0755"

ansible/roles/llm-inference/tasks/integration.yml

@@ -0,0 +1,44 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/integration.yml
+# DESCRIPTION: Phase 6 — Wire astro-orbiter into Hermes as a secondary provider.
+#              Writes a provider config fragment to carousel-of-progress
+#              (the Hermes host) so FRIDAY crons can route to the local model.
+#
+# Hermes provider config lives at ~/.hermes/config.yaml on carousel.
+# This task uses the lineinfile/blockinfile approach to add the provider
+# entry idempotently without clobbering the existing config.
+#
+# NOTE: Hermes must be restarted on carousel after this task runs.
+#       Manual step — JARVIS will notify Ryan.
+# ------------------------------------------------------------------------------
+
+- name: Check if astro-orbiter provider already configured in Hermes
+  ansible.builtin.command:
+    cmd: grep -c "astro-orbiter" /home/wed/.hermes/config.yaml
+  register: provider_check
+  changed_when: false
+  failed_when: false
+  delegate_to: carousel-of-progress
+
+- name: Add astro-orbiter as Hermes secondary provider
+  ansible.builtin.blockinfile:
+    path: /home/wed/.hermes/config.yaml
+    marker: "# {mark} ANSIBLE MANAGED — astro-orbiter vLLM provider"
+    insertafter: "^providers:"
+    block: |
+      # astro-orbiter — local RTX 3090 vLLM inference
+      - name: astro-orbiter
+        type: openai-compatible
+        base_url: http://{{ hostvars['astro-orbiter']['ansible_host'] }}:{{ llm_serve_port }}/v1
+        model: {{ llm_hf_model }}
+        api_key: none
+  when: provider_check.stdout == "0"
+  delegate_to: carousel-of-progress
+  notify: restart hermes
+
+- name: Remind operator to restart Hermes on carousel
+  ansible.builtin.debug:
+    msg: >
+      Phase 6 complete. Hermes on carousel-of-progress has been updated.
+      Restart Hermes manually or via: systemctl --user restart hermes-gateway hermes-dashboard

ansible/roles/llm-inference/tasks/main.yml

@@ -0,0 +1,36 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/main.yml
+# DESCRIPTION: Entry point — imports one task file per phase.
+#              Phases are additive; re-running the full playbook is always safe.
+#              Use --tags to run a specific phase subset:
+#                --tags foundation,driver,vllm,model,serve,integration,monitoring
+# ------------------------------------------------------------------------------
+
+# Phase 1 — Foundation
+- import_tasks: foundation.yml
+  tags: [foundation]
+
+# Phase 2 — Driver
+- import_tasks: driver.yml
+  tags: [driver]
+
+# Phase 3 — vLLM
+- import_tasks: vllm.yml
+  tags: [vllm]
+
+# Phase 4 — Model
+- import_tasks: model.yml
+  tags: [model]
+
+# Phase 5 — Serve
+- import_tasks: serve.yml
+  tags: [serve]
+
+# Phase 6 — Integration
+- import_tasks: integration.yml
+  tags: [integration]
+
+# Phase 7 — Monitoring
+- import_tasks: monitoring.yml
+  tags: [monitoring]

ansible/roles/llm-inference/tasks/model.yml

@@ -0,0 +1,42 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/model.yml
+# DESCRIPTION: Phase 4 — HuggingFace login and Gemma 2 27B model download.
+#              Idempotent: snapshot_download skips files already present.
+#              Requires vault_hf_token and Gemma 2 licence accepted at
+#              huggingface.co/google/gemma-2-27b-it.
+# ------------------------------------------------------------------------------
+
+- name: Write HuggingFace token to ~/.cache/huggingface/token
+  ansible.builtin.copy:
+    content: "{{ vault_hf_token }}"
+    dest: "/home/{{ llm_venv_owner }}/.cache/huggingface/token"
+    owner: "{{ llm_venv_owner }}"
+    group: "{{ llm_venv_owner }}"
+    mode: "0600"
+  no_log: true
+
+- name: Download Gemma 2 27B model via snapshot_download
+  ansible.builtin.command:
+    cmd: >
+      {{ llm_venv_path }}/bin/python -c "
+      from huggingface_hub import snapshot_download
+      path = snapshot_download(
+          '{{ llm_hf_model }}',
+          cache_dir='{{ llm_hf_cache_dir }}',
+      )
+      print(path)
+      "
+    creates: "{{ llm_hf_cache_dir }}/models--{{ llm_hf_model | replace('/', '--') }}/snapshots"
+  become: true
+  become_user: "{{ llm_venv_owner }}"
+  environment:
+    HF_TOKEN: "{{ vault_hf_token }}"
+    HOME: "/home/{{ llm_venv_owner }}"
+  register: model_download
+  timeout: 3600
+  no_log: false
+
+- name: Print model download path
+  ansible.builtin.debug:
+    msg: "Model available at: {{ model_download.stdout | default('already present') }}"

ansible/roles/llm-inference/tasks/monitoring.yml

@@ -0,0 +1,131 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/monitoring.yml
+# DESCRIPTION: Phase 7 — Prometheus monitoring for the LLM inference stack.
+#              Deploys two metric-producing exporters on astro-orbiter:
+#
+#              1. node_exporter (port 9100) — system: CPU, RAM, disk, network
+#              2. nvidia_gpu_exporter (port 9835) — GPU: VRAM, temp, util, power
+#              3. llama-server built-in metrics (port 8000/metrics, enabled via
+#                 --metrics) — just needs a scrape job (no extra process)
+#
+#              GitOps note: the Prometheus scrape jobs for all three targets and
+#              the Grafana dashboard are declared in the homelab Git repo and
+#              applied by ArgoCD — NOT by this role:
+#                - cluster/applications/monitoring/values.yaml
+#                  (prometheus.prometheusSpec.additionalScrapeConfigs)
+#                - cluster/applications/monitoring/dashboards.yaml
+#                  (grafana-llm-inference-dashboard ConfigMap)
+#              This role's job is only to stand up the two exporters + verify
+#              they're reachable. Do NOT reintroduce kubectl patch/apply tasks
+#              here — cluster-facing changes go through Git commit + ArgoCD
+#              sync so state stays reproducible and self-healing.
+# ------------------------------------------------------------------------------
+
+# -----------------------------------------------------------------------
+# 1. node_exporter
+# -----------------------------------------------------------------------
+
+- name: Install prometheus-node-exporter
+  ansible.builtin.apt:
+    name: prometheus-node-exporter
+    state: present
+    update_cache: false
+
+- name: Enable and start node_exporter
+  ansible.builtin.systemd:
+    name: prometheus-node-exporter
+    state: started
+    enabled: true
+
+- name: Verify node_exporter is reachable
+  ansible.builtin.uri:
+    url: "http://localhost:9100/metrics"
+    status_code: 200
+  register: node_exporter_health
+  retries: 6
+  delay: 5
+  until: node_exporter_health.status == 200
+  changed_when: false
+
+# -----------------------------------------------------------------------
+# 2. nvidia_gpu_exporter (utkuozdemir/nvidia_gpu_exporter)
+#    Lightweight single-binary exporter — no CUDA dependency, uses nvidia-smi.
+# -----------------------------------------------------------------------
+
+- name: Create nvidia_gpu_exporter install directory
+  ansible.builtin.file:
+    path: /opt/nvidia_gpu_exporter
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Download nvidia_gpu_exporter binary
+  ansible.builtin.get_url:
+    url: "https://github.com/utkuozdemir/nvidia_gpu_exporter/releases/download/v{{ llm_gpu_exporter_version }}/nvidia_gpu_exporter_{{ llm_gpu_exporter_version }}_linux_x86_64.tar.gz"
+    dest: "/tmp/nvidia_gpu_exporter.tar.gz"
+    mode: "0644"
+  register: gpu_exporter_download
+
+- name: Extract nvidia_gpu_exporter binary
+  ansible.builtin.unarchive:
+    src: /tmp/nvidia_gpu_exporter.tar.gz
+    dest: /opt/nvidia_gpu_exporter
+    remote_src: true
+    creates: /opt/nvidia_gpu_exporter/nvidia_gpu_exporter
+
+- name: Deploy nvidia_gpu_exporter systemd service
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/nvidia-gpu-exporter.service
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=NVIDIA GPU Prometheus Exporter
+      After=network.target nvidia-persistenced.service
+      Wants=nvidia-persistenced.service
+
+      [Service]
+      Type=simple
+      ExecStart=/opt/nvidia_gpu_exporter/nvidia_gpu_exporter \
+          --web.listen-address=:{{ llm_gpu_exporter_port }}
+      Restart=on-failure
+      RestartSec=5
+      StandardOutput=journal
+      StandardError=journal
+      SyslogIdentifier=nvidia-gpu-exporter
+
+      [Install]
+      WantedBy=multi-user.target
+  notify:
+    - reload systemd
+    - restart nvidia-gpu-exporter
+
+- name: Flush handlers before starting gpu exporter
+  ansible.builtin.meta: flush_handlers
+
+- name: Enable and start nvidia-gpu-exporter
+  ansible.builtin.systemd:
+    name: nvidia-gpu-exporter
+    state: started
+    enabled: true
+    daemon_reload: true
+
+- name: Verify nvidia_gpu_exporter is reachable
+  ansible.builtin.uri:
+    url: "http://localhost:{{ llm_gpu_exporter_port }}/metrics"
+    status_code: 200
+  register: gpu_exporter_health
+  retries: 6
+  delay: 5
+  until: gpu_exporter_health.status == 200
+  changed_when: false
+
+# -----------------------------------------------------------------------
+# 3. Prometheus scrape configs + Grafana dashboard
+#
+#    Intentionally NOT managed here. See file header: these are declared
+#    in cluster/applications/monitoring/{values.yaml,dashboards.yaml} in
+#    the homelab Git repo and rolled out by ArgoCD sync, keeping cluster
+#    state in Git rather than mutated imperatively from the control node.
+# -----------------------------------------------------------------------

ansible/roles/llm-inference/tasks/serve.yml

@@ -0,0 +1,151 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/serve.yml
+# DESCRIPTION: Phase 5 — llama-server (llama.cpp) serving Gemma 2 27B-it GGUF.
+#
+# WHY llama.cpp instead of vLLM:
+#   vLLM with bitsandbytes int4 quantizes on-the-fly — loads full bf16 weights
+#   (~54GB RAM peak) before compressing, killing the 40GB OptiPlex on warmup.
+#   llama.cpp loads the pre-quantized GGUF directly (~15.5GB peak RAM for Q4_K_M).
+#   No torch.compile, no warmup spike, OpenAI-compatible API on the same port.
+#
+# GGUF source: bartowski/gemma-2-27b-it-GGUF (Q4_K_M, 15.5GB)
+# Model downloaded to: {{ llm_gguf_path }}
+# ------------------------------------------------------------------------------
+
+- name: Install llama.cpp build dependencies
+  ansible.builtin.apt:
+    name:
+      - cmake
+      - build-essential
+      - libcurl4-openssl-dev
+    state: present
+    update_cache: false
+
+# NOTE: nvidia-driver-595-open provides the runtime driver only (nvidia-smi,
+# libcuda.so) — it does NOT ship nvcc/CUDA headers needed to build GGML_CUDA=ON.
+# Ubuntu 24.04's nvidia-cuda-toolkit (12.0.x) is sufficient to build llama.cpp
+# against; it does not need to match the 595 driver's CUDA 13.2 runtime version.
+- name: Install NVIDIA CUDA toolkit (nvcc) for building llama.cpp with CUDA support
+  ansible.builtin.apt:
+    name: nvidia-cuda-toolkit
+    state: present
+    update_cache: false
+  become: true
+
+- name: Clone llama.cpp repository
+  ansible.builtin.git:
+    repo: https://github.com/ggml-org/llama.cpp.git
+    dest: /opt/llama.cpp
+    depth: 1
+    update: false
+  become: true
+
+- name: Check for incomplete/stale llama.cpp CMake configuration
+  ansible.builtin.stat:
+    path: /opt/llama.cpp/build/Makefile
+  register: llama_cmake_generated
+
+- name: Remove stale llama.cpp build dir if CMake configure never completed
+  ansible.builtin.file:
+    path: /opt/llama.cpp/build
+    state: absent
+  become: true
+  when:
+    - not llama_cmake_generated.stat.exists
+    - not (ansible_check_mode | default(false))
+
+- name: Build llama.cpp with CUDA support
+  ansible.builtin.command:
+    cmd: cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
+    chdir: /opt/llama.cpp
+    creates: /opt/llama.cpp/build/CMakeCache.txt
+  become: true
+
+- name: Compile llama.cpp (parallel build)
+  ansible.builtin.command:
+    cmd: cmake --build build --config Release --parallel {{ ansible_processor_vcpus }}
+    chdir: /opt/llama.cpp
+    creates: /opt/llama.cpp/build/bin/llama-server
+  become: true
+  timeout: 600
+
+- name: Create GGUF model directory
+  ansible.builtin.file:
+    path: "{{ llm_gguf_dir }}"
+    state: directory
+    owner: "{{ llm_venv_owner }}"
+    group: "{{ llm_venv_owner }}"
+    mode: "0755"
+
+- name: Check whether GGUF already exists (avoid re-downloading 16.6GB on every run)
+  ansible.builtin.stat:
+    path: "{{ llm_gguf_path }}"
+  register: llm_gguf_stat
+
+- name: Download Gemma 2 27B Q4_K_M GGUF from HuggingFace
+  ansible.builtin.get_url:
+    url: "https://huggingface.co/bartowski/gemma-2-27b-it-GGUF/resolve/main/gemma-2-27b-it-Q4_K_M.gguf"
+    dest: "{{ llm_gguf_path }}"
+    headers:
+      Authorization: "Bearer {{ vault_hf_token }}"
+    owner: "{{ llm_venv_owner }}"
+    group: "{{ llm_venv_owner }}"
+    mode: "0644"
+    timeout: 7200
+    force: false
+  become: true
+  no_log: true
+  # Idempotency: skip entirely once the file exists and is reasonably sized
+  # (the finished GGUF is ~16.6GB; guard against a truncated partial download
+  # being mistaken for complete by only trusting files > 15GB).
+  when: not llm_gguf_stat.stat.exists or (llm_gguf_stat.stat.size | int) < 15000000000
+
+- name: Disable and stop vllm-serve if present
+  ansible.builtin.systemd:
+    name: vllm-serve
+    state: stopped
+    enabled: false
+  failed_when: false
+  notify: reload systemd
+
+- name: Deploy llama-server systemd service unit
+  ansible.builtin.template:
+    src: llama-server.service.j2
+    dest: /etc/systemd/system/llama-server.service
+    owner: root
+    group: root
+    mode: "0644"
+  notify:
+    - reload systemd
+    - restart llama-server
+
+- name: Flush handlers to reload systemd before enabling service
+  ansible.builtin.meta: flush_handlers
+
+- name: Enable and start llama-server
+  ansible.builtin.systemd:
+    name: llama-server
+    state: started
+    enabled: true
+    daemon_reload: true
+
+- name: Wait for llama-server API to become available (model load ~30s)
+  ansible.builtin.uri:
+    url: "http://localhost:{{ llm_serve_port }}/health"
+    status_code: 200
+  register: llama_health
+  retries: 18
+  delay: 10
+  until: llama_health.status == 200
+
+- name: Smoke-test — list available models
+  ansible.builtin.uri:
+    url: "http://localhost:{{ llm_serve_port }}/v1/models"
+    status_code: 200
+    return_content: true
+  register: llama_models
+
+- name: Print available models
+  ansible.builtin.debug:
+    msg: "llama-server serving: {{ llama_models.json.data | map(attribute='id') | list }}"

ansible/roles/llm-inference/tasks/vllm.yml

@@ -0,0 +1,43 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/vllm.yml
+# DESCRIPTION: Phase 3 — Python venv + vLLM install.
+#              Idempotent: venv creation and pip install only run if the
+#              venv binary or vllm package is absent.
+#              Already completed manually on 2026-08-03 — will no-op.
+# ------------------------------------------------------------------------------
+
+- name: Create Python venv for vLLM
+  ansible.builtin.command:
+    cmd: python3 -m venv {{ llm_venv_path }}
+    creates: "{{ llm_venv_path }}/bin/python"
+  become: true
+  become_user: "{{ llm_venv_owner }}"
+
+- name: Upgrade pip inside venv
+  ansible.builtin.pip:
+    name: pip
+    state: latest
+    virtualenv: "{{ llm_venv_path }}"
+  become: true
+  become_user: "{{ llm_venv_owner }}"
+
+- name: Install vLLM and bitsandbytes
+  ansible.builtin.pip:
+    name:
+      - vllm
+      - bitsandbytes
+    state: present
+    virtualenv: "{{ llm_venv_path }}"
+  become: true
+  become_user: "{{ llm_venv_owner }}"
+
+- name: Verify vLLM is importable
+  ansible.builtin.command:
+    cmd: "{{ llm_venv_path }}/bin/python -c 'import vllm; print(vllm.__version__)'"
+  register: vllm_version
+  changed_when: false
+
+- name: Print vLLM version
+  ansible.builtin.debug:
+    msg: "vLLM version: {{ vllm_version.stdout }}"

ansible/roles/llm-inference/templates/llama-server.service.j2

@@ -0,0 +1,32 @@
+[Unit]
+Description=llama-server — Gemma 2 27B-it Q4_K_M (OpenAI-compatible inference)
+After=network.target nvidia-persistenced.service
+Wants=nvidia-persistenced.service
+
+[Service]
+Type=simple
+User={{ llm_venv_owner }}
+Group={{ llm_venv_owner }}
+Environment="HOME=/home/{{ llm_venv_owner }}"
+ExecStart=/opt/llama.cpp/build/bin/llama-server \
+    --model {{ llm_gguf_path }} \
+    --host {{ llm_serve_host }} \
+    --port {{ llm_serve_port }} \
+    --ctx-size {{ llm_max_model_len }} \
+    --n-gpu-layers {{ llm_gpu_layers }} \
+    --parallel {{ llm_parallel_slots }} \
+    --metrics
+# NOTE: no --chat-template flag — llama-server auto-detects and uses the
+# GGUF's own embedded Jinja chat template (verified correct Gemma-2
+# start_of_turn/end_of_turn format for bartowski's gemma-2-27b-it-Q4_K_M).
+# The built-in "--chat-template gemma" name does NOT match this model's
+# expected format on this llama.cpp build and produced garbled completions.
+Restart=on-failure
+RestartSec=10
+TimeoutStartSec=120
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=llama-server
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/llm-inference/templates/vllm-serve.service.j2

@@ -0,0 +1,29 @@
+[Unit]
+Description=vLLM inference server — {{ llm_hf_model }}
+After=network.target nvidia-persistenced.service
+Wants=nvidia-persistenced.service
+
+[Service]
+Type=simple
+User={{ llm_venv_owner }}
+Group={{ llm_venv_owner }}
+Environment="HF_TOKEN={{ vault_hf_token }}"
+Environment="HOME=/home/{{ llm_venv_owner }}"
+Environment="HF_HUB_CACHE={{ llm_hf_cache_dir }}"
+ExecStart={{ llm_venv_path }}/bin/python -m vllm.entrypoints.openai.api_server \
+    --model {{ llm_hf_model }} \
+    --host {{ llm_serve_host }} \
+    --port {{ llm_serve_port }} \
+    --quantization {{ llm_quantization }} \
+    --gpu-memory-utilization {{ llm_gpu_memory_utilization }} \
+    --max-model-len {{ llm_max_model_len }} \
+    --enable-prefix-caching
+Restart=on-failure
+RestartSec=10
+TimeoutStartSec=300
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=vllm-serve
+
+[Install]
+WantedBy=multi-user.target

cluster/applications/couchdb/templates/ingress-public.yaml

@@ -15,6 +15,7 @@ metadata:
     external-dns.alpha.kubernetes.io/hostname: "communicore.mk-labs.cloud"
     external-dns.alpha.kubernetes.io/target: "ingress.mk-labs.cloud"
     external-dns.alpha.kubernetes.io/public: "true"
+    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
 spec:
   ingressClassName: nginx
   rules:

cluster/applications/couchdb/values.yaml

@@ -62,6 +62,7 @@ ingress:
   annotations:
     cert-manager.io/cluster-issuer: "letsencrypt-prod"
     external-dns.alpha.kubernetes.io/hostname: "communicore.local.mk-labs.cloud"
+    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
   hosts:
     - communicore.local.mk-labs.cloud
   tls:

cluster/applications/firecrawl/ARCHITECTURE.md

@@ -0,0 +1,337 @@
+# Firecrawl Architecture Diagram
+
+## High-Level Overview
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│                            External Access                              │
+│  spaceship-earth.local.mk-labs.cloud / firecrawl.local.mk-labs.cloud   │
+│                                 │                                        │
+│                                 ▼                                        │
+│                        ┌──────────────────┐                             │
+│                        │  Gateway API     │                             │
+│                        │  (HTTPRoute)     │                             │
+│                        │  TLS Termination │                             │
+│                        └────────┬─────────┘                             │
+└─────────────────────────────────┼──────────────────────────────────────┘
+                                  │
+┌─────────────────────────────────┼──────────────────────────────────────┐
+│                    Firecrawl Namespace                                  │
+│                                 │                                       │
+│                                 ▼                                       │
+│                        ┌──────────────────┐                            │
+│                        │   API Service    │                            │
+│                        │   (ClusterIP)    │                            │
+│                        │   Port 3002      │                            │
+│                        └────────┬─────────┘                            │
+│                                 │                                       │
+│              ┌──────────────────┼──────────────────┐                   │
+│              │                  │                  │                   │
+│              ▼                  ▼                  ▼                   │
+│     ┌─────────────────┐ ┌─────────────┐ ┌──────────────────┐         │
+│     │  API Deployment │ │   Worker    │ │   NUQ Worker     │         │
+│     │  (firecrawl-api)│ │ Deployment  │ │   Deployment     │         │
+│     │                 │ │(firecrawl-  │ │  (firecrawl-api) │         │
+│     │  Entrypoint:    │ │    api)     │ │                  │         │
+│     │  dist/src/      │ │             │ │  Entrypoint:     │         │
+│     │  index.js       │ │ Entrypoint: │ │  dist/src/       │         │
+│     │                 │ │ dist/src/   │ │  services/worker/│         │
+│     │  4-6GB / 2 CPU  │ │ services/   │ │  nuq-worker.js   │         │
+│     │                 │ │ queue-      │ │                  │         │
+│     │  Replicas: 1    │ │ worker.js   │ │  3-4GB / 1 CPU   │         │
+│     │                 │ │             │ │                  │         │
+│     │  Health:        │ │ 3-4GB/1 CPU │ │  Replicas: 1     │         │
+│     │  /v0/health/*   │ │             │ │                  │         │
+│     │                 │ │ Replicas: 1 │ │                  │         │
+│     └────────┬────────┘ └──────┬──────┘ └────────┬─────────┘         │
+│              │                  │                  │                   │
+│              └─────────┬────────┴──────────────────┘                   │
+│                        │                                               │
+│              ┌─────────┼──────────────┬──────────────┐                │
+│              │         │              │              │                │
+│              ▼         ▼              ▼              ▼                │
+│     ┌────────────┐ ┌──────────┐ ┌──────────┐ ┌───────────────┐      │
+│     │ Playwright │ │  Redis   │ │PostgreSQL│ │   RabbitMQ    │      │
+│     │  Service   │ │          │ │          │ │               │      │
+│     │            │ │          │ │          │ │               │      │
+│     │ Deployment │ │Deployment│ │StatefulSet│ │  Deployment   │      │
+│     │ (Harbor)   │ │(Upstream)│ │ (Harbor) │ │  (Upstream)   │      │
+│     │            │ │          │ │          │ │               │      │
+│     │  Service:  │ │ Service: │ │ Service: │ │  Service:     │      │
+│     │  3000      │ │  6379    │ │  5432    │ │  5672, 15672  │      │
+│     │            │ │          │ │          │ │               │      │
+│     │  4GB/2 CPU │ │ 1GB/0.5  │ │ 2GB/1 CPU│ │  1GB/0.5 CPU  │      │
+│     │            │ │          │ │          │ │               │      │
+│     │  tmpfs:    │ │          │ │   PVC:   │ │  Healthcheck: │      │
+│     │  1GB       │ │          │ │   10GB   │ │  Required     │      │
+│     └────────────┘ └──────────┘ └──────────┘ └───────────────┘      │
+│                                       │                               │
+│                                       ▼                               │
+│                                 ┌──────────┐                          │
+│                                 │ NFS PVC  │                          │
+│                                 │  10GB    │                          │
+│                                 │(nfs-     │                          │
+│                                 │emporium) │                          │
+│                                 └──────────┘                          │
+│                                                                        │
+│  Configuration:                                                       │
+│  ┌──────────────┐  ┌────────────────┐                                │
+│  │  ConfigMap   │  │ ExternalSecret │                                │
+│  │  (firecrawl- │  │  (firecrawl-   │                                │
+│  │   config)    │  │   secrets)     │                                │
+│  │              │  │                │                                │
+│  │  - URLs      │  │  ┌──────────┐  │                                │
+│  │  - Ports     │  │  │1Password │  │                                │
+│  │  - Tuning    │  │  │  Vault   │  │                                │
+│  │              │  │  └────┬─────┘  │                                │
+│  └──────────────┘  │       │        │                                │
+│                    │       ▼        │                                │
+│                    │  - postgres-   │                                │
+│                    │    password    │                                │
+│                    │  - bull-auth-  │                                │
+│                    │    key         │                                │
+│                    └────────────────┘                                │
+└────────────────────────────────────────────────────────────────────────┘
+```
+
+## Service Communication Flow
+
+### API Request Flow
+```
+External User
+    │
+    ▼
+Gateway API (TLS termination)
+    │
+    ▼
+API Service (port 3002)
+    │
+    ├─────► Playwright Service (browser automation)
+    │           └─► Returns rendered HTML/Markdown
+    │
+    ├─────► Redis (queue jobs, cache results)
+    │
+    ├─────► PostgreSQL (store job metadata)
+    │
+    └─────► RabbitMQ (publish job events)
+```
+
+### Background Job Processing Flow
+```
+API receives request
+    │
+    ▼
+Job queued in Redis
+    │
+    ▼
+RabbitMQ notifies workers
+    │
+    ├─────► Worker picks up job
+    │       └─► Processes scraping tasks
+    │
+    └─────► NUQ Worker picks up database jobs
+            └─► Processes queue from PostgreSQL
+```
+
+### Database Queue Flow (NUQ)
+```
+Job created in PostgreSQL (nuq.queue_scrape table)
+    │
+    ▼
+NUQ Worker polls for jobs (prefetch)
+    │
+    ├─► Status: queued → active
+    │
+    ├─► Worker processes job
+    │   └─► Calls Playwright or direct fetch
+    │
+    └─► Status: active → completed/failed
+        └─► Results stored in returnvalue column
+```
+
+## Build Pipeline Flow
+
+```
+GitHub: mendableai/firecrawl
+    │
+    ▼
+Tekton Pipeline (innoventions namespace)
+    │
+    ├─────► firecrawl-api-build
+    │       │
+    │       ├─► Git Clone Task
+    │       │
+    │       ├─► Kaniko Build Task
+    │       │   └─► Multi-stage: Go → Node → Runtime
+    │       │
+    │       └─► Push to Harbor
+    │           └─► the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest
+    │
+    ├─────► firecrawl-playwright-build
+    │       │
+    │       ├─► Git Clone Task
+    │       │
+    │       ├─► Kaniko Build Task
+    │       │   └─► Node.js + Chromium install
+    │       │
+    │       └─► Push to Harbor
+    │           └─► .../firecrawl-playwright:latest
+    │
+    └─────► firecrawl-postgres-build
+            │
+            ├─► Git Clone Task
+            │
+            ├─► Kaniko Build Task
+            │   └─► postgres:16 + pg_cron + nuq.sql
+            │
+            └─► Push to Harbor
+                └─► .../firecrawl-postgres:latest
+```
+
+## Deployment Flow (ArgoCD)
+
+```
+Gitea Repository (homelab)
+    │
+    └─► cluster/applications/firecrawl/
+        │
+        ▼
+ArgoCD Application (sync)
+    │
+    ├─► Wave 0: Namespace
+    │
+    ├─► Wave 1: ConfigMap, ExternalSecret
+    │
+    ├─► Wave 2: PostgreSQL StatefulSet + PVC
+    │           Redis Deployment
+    │           RabbitMQ Deployment
+    │
+    ├─► Wave 3: Playwright Deployment
+    │           (waits for infrastructure)
+    │
+    ├─► Wave 4: API Deployment
+    │           Worker Deployments
+    │           (waits for all dependencies)
+    │
+    └─► Wave 5: Services, HTTPRoute
+```
+
+## Resource Distribution
+
+```
+Total Cluster Capacity: ~48 CPU / ~96GB RAM (6 nodes)
+
+Firecrawl Allocation:
+┌────────────────────────────────────┐
+│ API:         2 CPU /  4-6GB        │ ████████████
+│ Worker:      1 CPU /  3-4GB        │ ██████
+│ NUQ Worker:  1 CPU /  3-4GB        │ ██████
+│ Playwright:  2 CPU /    4GB        │ ████████████
+│ PostgreSQL:  1 CPU /    2GB        │ ██████
+│ Redis:     0.5 CPU /    1GB        │ ███
+│ RabbitMQ:  0.5 CPU /    1GB        │ ███
+├────────────────────────────────────┤
+│ TOTAL:     8 CPU /   22GB RAM      │
+└────────────────────────────────────┘
+
+Percentage of cluster: ~17% CPU, ~23% RAM
+Headroom available: ✅ Excellent
+```
+
+## Data Flow
+
+### Scrape Request Example
+```
+1. User → POST /v1/scrape {"url": "https://example.com"}
+   │
+2. API validates request
+   │
+3. API creates job in PostgreSQL (nuq.queue_scrape)
+   │
+4. API queues job in Redis
+   │
+5. RabbitMQ notifies workers
+   │
+6. Worker picks up job
+   │
+7. Worker calls Playwright service
+   │  └─► Playwright launches Chromium
+   │      └─► Renders page (handles JS)
+   │          └─► Returns HTML
+   │
+8. Worker converts HTML → Markdown (Go library)
+   │
+9. Worker stores result in PostgreSQL (returnvalue column)
+   │
+10. Worker updates job status: completed
+    │
+11. API returns result to user
+    └─► {"markdown": "...", "html": "...", "metadata": {...}}
+```
+
+---
+
+## Network Policies (Future Enhancement)
+
+```
+firecrawl namespace:
+│
+├─► Ingress Rules:
+│   ├─ Allow: Gateway API → API Service (port 3002)
+│   └─ Deny:  All other external traffic
+│
+├─► Egress Rules:
+│   ├─ Allow: API → Playwright (port 3000)
+│   ├─ Allow: API → Redis (port 6379)
+│   ├─ Allow: API → PostgreSQL (port 5432)
+│   ├─ Allow: API → RabbitMQ (port 5672)
+│   ├─ Allow: All → Internet (for web scraping)
+│   └─ Deny:  All other cluster traffic
+│
+└─► Inter-Pod Rules:
+    ├─ Allow: API → All infrastructure services
+    ├─ Allow: Workers → All infrastructure services
+    ├─ Deny:  PostgreSQL → Internet (security)
+    └─ Deny:  Redis → Internet (security)
+```
+
+---
+
+## Monitoring & Observability (Future Enhancement)
+
+```
+Prometheus Metrics:
+│
+├─► API Metrics (port 3002/metrics)
+│   ├─ Request rate
+│   ├─ Response times
+│   ├─ Job queue depth
+│   └─ Error rates
+│
+├─► Worker Metrics (port 3005/metrics)
+│   ├─ Jobs processed
+│   ├─ Processing times
+│   └─ Success/failure rates
+│
+├─► PostgreSQL Metrics
+│   ├─ Connection pool usage
+│   ├─ Query performance
+│   └─ Table sizes
+│
+└─► Playwright Metrics
+    ├─ Browser pool usage
+    ├─ Page load times
+    └─ Chromium memory usage
+
+Grafana Dashboards:
+├─ Firecrawl Overview
+├─ Job Processing Metrics
+├─ Service Health
+└─ Resource Utilization
+```
+
+---
+
+**Diagram Version:** 1.0  
+**Last Updated:** June 6, 2026  
+**Created By:** Rocket Raccoon (CI/CD Specialist)

cluster/applications/firecrawl/ENVIRONMENT_VARIABLES.md

@@ -0,0 +1,412 @@
+# Firecrawl Environment Variables Reference
+
+Complete reference for all environment variables used by Firecrawl services.
+
+---
+
+## Required Variables (CRITICAL)
+
+These variables MUST be set for Firecrawl to function.
+
+### Server Configuration
+```yaml
+HOST: "0.0.0.0"                    # Listen address
+PORT: "3002"                       # Main API port
+WORKER_PORT: "3005"                # Worker liveness check port
+EXTRACT_WORKER_PORT: "3004"        # Extract worker port
+```
+
+### Database (PostgreSQL)
+```yaml
+POSTGRES_USER: "postgres"          # Database username
+POSTGRES_PASSWORD: "<SECRET>"      # 🔐 Database password (from 1Password)
+POSTGRES_DB: "postgres"            # Database name
+POSTGRES_HOST: "nuq-postgres"      # K8s service name
+POSTGRES_PORT: "5432"              # PostgreSQL port
+```
+
+### Redis (Queue & Cache)
+```yaml
+REDIS_URL: "redis://redis:6379"
+REDIS_RATE_LIMIT_URL: "redis://redis:6379"
+```
+
+### RabbitMQ (Message Broker)
+```yaml
+NUQ_RABBITMQ_URL: "amqp://rabbitmq:5672"
+```
+
+### Playwright Service
+```yaml
+PLAYWRIGHT_MICROSERVICE_URL: "http://playwright-service:3000/scrape"
+```
+
+### Authentication
+```yaml
+USE_DB_AUTHENTICATION: "false"     # Set "true" for production with Supabase
+```
+
+---
+
+## Security Variables (REQUIRED)
+
+### Admin UI Protection
+```yaml
+BULL_AUTH_KEY: "<SECRET>"          # 🔐 Queue admin UI password (from 1Password)
+                                   # URL: /admin/{BULL_AUTH_KEY}/queues
+```
+
+### API Testing
+```yaml
+TEST_API_KEY: "<SECRET>"           # 🔐 Optional - API key for testing (from 1Password)
+```
+
+---
+
+## Optional Variables (Features)
+
+### AI Features (JSON Format, Extract API)
+```yaml
+# OpenAI Configuration
+OPENAI_API_KEY: "<SECRET>"         # 🔐 OpenAI API key (from 1Password)
+OPENAI_BASE_URL: ""                # Custom OpenAI-compatible endpoint
+MODEL_NAME: ""                     # Override default model (e.g., gpt-4)
+MODEL_EMBEDDING_NAME: ""           # Override embedding model
+
+# Ollama (Alternative to OpenAI)
+OLLAMA_BASE_URL: ""                # E.g., http://localhost:11434/api
+# When using Ollama, set:
+# MODEL_NAME: "deepseek-r1:7b"
+# MODEL_EMBEDDING_NAME: "nomic-embed-text"
+```
+
+### Proxy Configuration
+```yaml
+PROXY_SERVER: ""                   # Full URL (http://0.1.2.3:1234) or IP:port
+PROXY_USERNAME: ""                 # 🔐 Proxy username (from 1Password)
+PROXY_PASSWORD: ""                 # 🔐 Proxy password (from 1Password)
+```
+
+### Search API Configuration
+```yaml
+# By default, uses Google search
+# Optionally use SearXNG instead:
+SEARXNG_ENDPOINT: ""               # E.g., http://your.searxng.server
+SEARXNG_ENGINES: ""                # Comma-separated engine list
+SEARXNG_CATEGORIES: ""             # Comma-separated categories
+```
+
+### Monitoring & Logging
+```yaml
+LOGGING_LEVEL: "info"              # debug, info, warn, error
+SLACK_WEBHOOK_URL: ""              # 🔐 Slack webhook for alerts (from 1Password)
+```
+
+### Supabase Integration (Advanced)
+```yaml
+# Note: Not currently configurable for self-hosted instances
+SUPABASE_ANON_TOKEN: ""            # For DB authentication
+SUPABASE_URL: ""                   # Supabase project URL
+SUPABASE_SERVICE_TOKEN: ""         # Supabase service role key
+```
+
+---
+
+## Performance Tuning
+
+### Worker Pools
+```yaml
+NUM_WORKERS_PER_QUEUE: "8"         # Number of workers per queue
+CRAWL_CONCURRENT_REQUESTS: "10"    # Concurrent crawl requests
+MAX_CONCURRENT_JOBS: "5"           # Maximum concurrent jobs
+```
+
+### Browser Pool (Playwright)
+```yaml
+BROWSER_POOL_SIZE: "5"             # Browser instance pool size
+MAX_CONCURRENT_PAGES: "10"         # Max concurrent pages per browser
+```
+
+### Resource Limits (Self-Protection)
+```yaml
+MAX_CPU: "0.8"                     # 0.0-1.0, reject jobs above threshold
+MAX_RAM: "0.8"                     # 0.0-1.0, reject jobs above threshold
+```
+
+### Timeouts
+```yaml
+HARNESS_STARTUP_TIMEOUT_MS: "60000" # 60 seconds startup timeout
+```
+
+---
+
+## Playwright-Specific Variables
+
+Set on `playwright-service` pods only:
+
+```yaml
+PORT: "3000"                       # Playwright service port
+PROXY_SERVER: ""                   # Same as API proxy (if needed)
+PROXY_USERNAME: ""                 # Same as API proxy (if needed)
+PROXY_PASSWORD: ""                 # Same as API proxy (if needed)
+ALLOW_LOCAL_WEBHOOKS: "false"      # Security: block local webhooks
+BLOCK_MEDIA: ""                    # Optional: block media resources
+MAX_CONCURRENT_PAGES: "10"         # Browser concurrency (same as API)
+```
+
+---
+
+## Environment-Specific Variables
+
+### Development
+```yaml
+ENV: "local"
+```
+
+### Production
+```yaml
+ENV: "production"
+```
+
+---
+
+## FlyIO-Specific Variables (Not Used in K8s)
+
+These variables are set by FlyIO platform and not needed for Kubernetes deployment:
+
+```yaml
+FLY_PROCESS_GROUP: "app"           # Not used in K8s
+```
+
+---
+
+## Deprecated / Unused Variables
+
+Variables found in examples but not required for self-hosted deployment:
+
+```yaml
+AUTUMN_SECRET_KEY: ""              # Mendable platform-specific
+SELF_HOSTED_WEBHOOK_URL: ""        # Custom webhook endpoint
+LLAMAPARSE_API_KEY: ""             # PDF parsing service (optional)
+```
+
+---
+
+## Variable Precedence
+
+1. **ExternalSecret** (from 1Password) - Highest priority for secrets
+2. **ConfigMap** - Non-sensitive configuration
+3. **Deployment env:** - Direct environment variables (override)
+4. **Dockerfile defaults** - Lowest priority
+
+---
+
+## ConfigMap Example
+
+Non-sensitive variables suitable for ConfigMap:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: firecrawl-config
+  namespace: firecrawl
+data:
+  # Service URLs
+  REDIS_URL: "redis://redis:6379"
+  REDIS_RATE_LIMIT_URL: "redis://redis:6379"
+  POSTGRES_HOST: "nuq-postgres"
+  POSTGRES_PORT: "5432"
+  POSTGRES_USER: "postgres"
+  POSTGRES_DB: "postgres"
+  NUQ_RABBITMQ_URL: "amqp://rabbitmq:5672"
+  PLAYWRIGHT_MICROSERVICE_URL: "http://playwright-service:3000/scrape"
+  
+  # Server Configuration
+  HOST: "0.0.0.0"
+  PORT: "3002"
+  WORKER_PORT: "3005"
+  EXTRACT_WORKER_PORT: "3004"
+  USE_DB_AUTHENTICATION: "false"
+  ENV: "production"
+  
+  # Performance Tuning
+  NUM_WORKERS_PER_QUEUE: "8"
+  CRAWL_CONCURRENT_REQUESTS: "10"
+  MAX_CONCURRENT_JOBS: "5"
+  BROWSER_POOL_SIZE: "5"
+  MAX_CONCURRENT_PAGES: "10"
+  HARNESS_STARTUP_TIMEOUT_MS: "60000"
+  
+  # Logging
+  LOGGING_LEVEL: "info"
+  
+  # Security
+  ALLOW_LOCAL_WEBHOOKS: "false"
+```
+
+---
+
+## ExternalSecret Example
+
+Sensitive variables synced from 1Password:
+
+```yaml
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: firecrawl-secrets
+  namespace: firecrawl
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: onepassword-connect
+    kind: ClusterSecretStore
+  target:
+    name: firecrawl-secrets
+    creationPolicy: Owner
+  data:
+    # Required secrets
+    - secretKey: POSTGRES_PASSWORD
+      remoteRef:
+        key: firecrawl
+        property: postgres-password
+    
+    - secretKey: BULL_AUTH_KEY
+      remoteRef:
+        key: firecrawl
+        property: bull-auth-key
+    
+    # Optional secrets (uncomment when added to 1Password)
+    # - secretKey: OPENAI_API_KEY
+    #   remoteRef:
+    #     key: firecrawl
+    #     property: openai-api-key
+    
+    # - secretKey: TEST_API_KEY
+    #   remoteRef:
+    #     key: firecrawl
+    #     property: test-api-key
+```
+
+---
+
+## Usage in Deployments
+
+### Combining ConfigMap and Secret
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: api
+  namespace: firecrawl
+spec:
+  template:
+    spec:
+      containers:
+        - name: api
+          image: the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest
+          envFrom:
+            # Load all non-sensitive variables
+            - configMapRef:
+                name: firecrawl-config
+            # Load all secrets
+            - secretRef:
+                name: firecrawl-secrets
+          env:
+            # Override specific variables if needed
+            - name: FLY_PROCESS_GROUP
+              value: "app"
+```
+
+### Playwright-Specific ConfigMap
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: playwright-service-config
+  namespace: firecrawl
+data:
+  PORT: "3000"
+  ALLOW_LOCAL_WEBHOOKS: "false"
+  MAX_CONCURRENT_PAGES: "10"
+```
+
+---
+
+## Validation Checklist
+
+Before deployment, ensure:
+
+- [ ] `POSTGRES_PASSWORD` set in 1Password
+- [ ] `BULL_AUTH_KEY` set in 1Password (strong random value)
+- [ ] All service URLs use correct K8s service names
+- [ ] Ports match service definitions (3002, 3000, 5432, 6379, 5672)
+- [ ] `USE_DB_AUTHENTICATION` is "false" (Supabase not available)
+- [ ] `ENV` is "production" (not "local")
+- [ ] Worker pool sizes appropriate for cluster capacity
+- [ ] `ALLOW_LOCAL_WEBHOOKS` is "false" (security)
+
+---
+
+## Troubleshooting
+
+### "Supabase client is not configured"
+**Expected warning** - Supabase is not available for self-hosted instances. Safe to ignore.
+
+### "You're bypassing authentication"
+**Expected warning** - When `USE_DB_AUTHENTICATION=false`. Normal for self-hosted deployment.
+
+### Connection refused errors
+**Check:**
+- Service names match environment variables
+- Services are running: `kubectl get svc -n firecrawl`
+- Pods are ready: `kubectl get pods -n firecrawl`
+
+### Build failures
+**Check:**
+- Required build args are set
+- Kaniko has sufficient memory (4GB for API build)
+
+---
+
+## Security Best Practices
+
+1. **Generate strong secrets:**
+   ```bash
+   # Generate BULL_AUTH_KEY
+   openssl rand -base64 32
+   
+   # Generate POSTGRES_PASSWORD
+   openssl rand -base64 24
+   ```
+
+2. **Never expose PostgreSQL externally:**
+   - Use ClusterIP service only
+   - No LoadBalancer or NodePort
+   - Access via kubectl port-forward for maintenance
+
+3. **Protect admin UI:**
+   - Keep BULL_AUTH_KEY secret
+   - Don't commit to git
+   - Rotate periodically
+
+4. **Use ExternalSecrets:**
+   - Never put secrets in ConfigMaps
+   - Never commit secrets to git
+   - Store all sensitive data in 1Password
+
+---
+
+## References
+
+- **Upstream .env Template:** `/tmp/firecrawl/SELF_HOST.md`
+- **Docker Compose Reference:** `/tmp/firecrawl/docker-compose.yaml`
+- **K8s Example ConfigMap:** `/tmp/firecrawl/examples/kubernetes/cluster-install/configmap.yaml`
+
+---
+
+**Last Updated:** June 6, 2026  
+**Maintained By:** Rocket Raccoon (CI/CD Specialist)

cluster/applications/firecrawl/README.md

@@ -0,0 +1,375 @@
+# Firecrawl - Web Scraping & Crawling Service
+
+**Project:** Platform Buildout - Firecrawl Deployment  
+**Service Name:** Spaceship Earth (EPCOT themed)  
+**DNS:** spaceship-earth.local.mk-labs.cloud (primary), firecrawl.local.mk-labs.cloud (alias)  
+**Namespace:** firecrawl  
+**Owner:** Rocket Raccoon (CI/CD Specialist)  
+**Status:** 🚧 IN PROGRESS - Day 1 Complete
+
+---
+
+## Overview
+
+Firecrawl is a self-hosted web scraping and crawling API that converts URLs to LLM-ready content (Markdown, JSON, HTML). This deployment enables JARVIS web search capability.
+
+**Upstream:** https://github.com/mendableai/firecrawl  
+**License:** AGPLv3 (open source)
+
+### Capabilities
+- **Search:** Search the web and get full page content
+- **Scrape:** Convert URLs to markdown, HTML, screenshots, or structured JSON
+- **Crawl:** Scrape all URLs of a website with a single request
+- **Interact:** Click, scroll, write, wait before extracting (JS-heavy sites)
+- **Map:** Discover all URLs on a website
+
+---
+
+## Architecture
+
+### Services (5 components)
+
+| Service | Image | CPU | Memory | Storage | Purpose |
+|---------|-------|-----|--------|---------|---------|
+| **API** | Harbor: firecrawl-api:latest | 2.0 | 4-6GB | - | Main REST API |
+| **Worker** | Harbor: firecrawl-api:latest | 1.0 | 3-4GB | - | Background job processor |
+| **NUQ Worker** | Harbor: firecrawl-api:latest | 1.0 | 3-4GB | - | Database queue worker |
+| **Playwright** | Harbor: firecrawl-playwright:latest | 2.0 | 4GB | 1GB tmpfs | Browser automation |
+| **PostgreSQL** | Harbor: firecrawl-postgres:latest | 1.0 | 2GB | 10GB PVC | Data storage |
+| **Redis** | Upstream: redis:alpine | 0.5 | 1GB | - | Queue & cache |
+| **RabbitMQ** | Upstream: rabbitmq:3-management | 0.5 | 1GB | - | Message broker |
+
+**Total Resources:** ~8 CPU, ~22GB RAM, 11GB storage
+
+### Service Dependencies
+
+```
+API Service ─┬─► Redis (queue/cache)
+             ├─► PostgreSQL (data storage)
+             ├─► RabbitMQ (message broker) ⚠️ HEALTH CHECK REQUIRED
+             └─► Playwright Service (browser automation)
+
+Worker ──────┬─► Redis
+             ├─► PostgreSQL
+             └─► RabbitMQ
+
+NUQ Worker ──┴─► PostgreSQL
+```
+
+**Startup Order:**
+1. Redis, PostgreSQL, RabbitMQ (infrastructure)
+2. Playwright Service
+3. API, Workers (after all dependencies ready)
+
+---
+
+## Build Strategy
+
+**Hybrid Approach:** Build custom images via Tekton, use upstream for infrastructure.
+
+### Custom Builds (Tekton → Harbor)
+
+1. **firecrawl-api** (Multi-stage: Go + Node.js + Rust)
+   - Source: `apps/api/Dockerfile`
+   - Registry: `the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest`
+   - Build time: ~15 minutes (first), ~5 minutes (cached)
+   - Used by: API, Worker, NUQ Worker (different entrypoints)
+
+2. **firecrawl-playwright** (Node.js + Chromium)
+   - Source: `apps/playwright-service-ts/Dockerfile`
+   - Registry: `the-seas.local.mk-labs.cloud/applications/firecrawl-playwright:latest`
+   - Build time: ~10 minutes
+
+3. **firecrawl-postgres** (PostgreSQL + pg_cron + init script)
+   - Source: `apps/nuq-postgres/Dockerfile`
+   - Registry: `the-seas.local.mk-labs.cloud/applications/firecrawl-postgres:latest`
+   - Build time: ~3 minutes
+   - Note: Custom build required for pg_cron extension and nuq.sql schema
+
+### Upstream Images
+
+- **Redis:** `redis:alpine`
+- **RabbitMQ:** `rabbitmq:3-management`
+
+---
+
+## Configuration
+
+### Required Secrets (1Password)
+
+Managed via ExternalSecret → 1Password vault item: `firecrawl`
+
+- `POSTGRES_PASSWORD` - Database password (CRITICAL)
+- `BULL_AUTH_KEY` - Queue admin UI authentication (CRITICAL)
+- `OPENAI_API_KEY` - Optional, for AI features
+- `TEST_API_KEY` - Optional, for testing
+
+### ConfigMap (Non-sensitive)
+
+- Service URLs (Redis, PostgreSQL, RabbitMQ, Playwright)
+- Port configuration (3002 API, 3005 Worker)
+- Performance tuning (worker pools, concurrency limits)
+- Logging level
+
+### Environment Variables Reference
+
+See [ENVIRONMENT_VARIABLES.md](./ENVIRONMENT_VARIABLES.md) for complete list.
+
+---
+
+## Deployment Timeline
+
+### Day 1 (June 6) - Investigation & Planning ✅
+- Repository analysis complete
+- Architecture decisions finalized
+- Environment configuration researched
+- Day 2 plan created
+
+### Day 2 (June 7) - Tekton Pipelines 🚧
+- Create 3 build pipelines (API, Playwright, PostgreSQL)
+- Test builds and push to Harbor
+- Validate image integrity
+
+### Day 3 (June 8) - Kubernetes Manifests
+- Create Deployments, Services, StatefulSets
+- Configure ConfigMaps and ExternalSecrets
+- Set up HTTPRoute for ingress
+
+### Day 4 (June 9) - Secrets & Configuration
+- Create 1Password vault item
+- Configure ExternalSecret sync
+- Validate configuration
+
+### Day 5 (June 10) - Deployment & Testing
+- ArgoCD Application creation
+- Deploy to cluster
+- Service health validation
+- API functionality testing
+
+### Day 6 (June 11) - JARVIS Integration
+- Configure JARVIS environment variables
+- Test web search functionality
+- End-to-end validation
+- Documentation delivery
+
+**Target Completion:** June 12, 2026
+
+---
+
+## Access & URLs
+
+**Primary Access:**
+- API: https://spaceship-earth.local.mk-labs.cloud
+- Alias: https://firecrawl.local.mk-labs.cloud
+- Queue UI: https://spaceship-earth.local.mk-labs.cloud/admin/[BULL_AUTH_KEY]/queues
+
+**Health Endpoints:**
+- Liveness: https://spaceship-earth.local.mk-labs.cloud/v0/health/liveness
+- Readiness: https://spaceship-earth.local.mk-labs.cloud/v0/health/readiness
+
+**Internal Services (cluster-only):**
+- Playwright: http://playwright-service.firecrawl.svc:3000
+- PostgreSQL: postgresql://nuq-postgres.firecrawl.svc:5432
+- Redis: redis://redis.firecrawl.svc:6379
+- RabbitMQ: amqp://rabbitmq.firecrawl.svc:5672
+- RabbitMQ Mgmt: http://rabbitmq.firecrawl.svc:15672
+
+---
+
+## API Usage Examples
+
+### Scrape a URL
+```bash
+curl -X POST https://spaceship-earth.local.mk-labs.cloud/v1/scrape \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "url": "https://example.com"
+  }'
+```
+
+### Search the Web
+```bash
+curl -X POST https://spaceship-earth.local.mk-labs.cloud/v1/search \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "query": "kubernetes best practices"
+  }'
+```
+
+### Crawl a Website
+```bash
+curl -X POST https://spaceship-earth.local.mk-labs.cloud/v1/crawl \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "url": "https://docs.example.com"
+  }'
+```
+
+---
+
+## Directory Structure
+
+```
+cluster/applications/firecrawl/
+├── README.md                      # This file
+├── ENVIRONMENT_VARIABLES.md       # Complete env var reference
+├── namespace.yaml                 # Namespace definition
+├── configmap.yaml                 # Non-sensitive configuration
+├── externalsecret.yaml            # 1Password secret sync
+├── postgresql/
+│   ├── statefulset.yaml          # PostgreSQL StatefulSet
+│   ├── service.yaml              # PostgreSQL Service
+│   └── pvc.yaml                  # Persistent Volume Claim
+├── redis/
+│   ├── deployment.yaml           # Redis Deployment
+│   └── service.yaml              # Redis Service
+├── rabbitmq/
+│   ├── deployment.yaml           # RabbitMQ Deployment
+│   └── service.yaml              # RabbitMQ Service
+├── playwright/
+│   ├── deployment.yaml           # Playwright Deployment
+│   └── service.yaml              # Playwright Service
+├── api/
+│   ├── deployment.yaml           # API Deployment
+│   └── service.yaml              # API Service
+├── workers/
+│   ├── worker-deployment.yaml    # Queue Worker Deployment
+│   └── nuq-worker-deployment.yaml # NUQ Worker Deployment
+├── ingress/
+│   └── httproute.yaml            # Gateway API HTTPRoute
+└── argocd/
+    └── application.yaml          # ArgoCD Application manifest
+```
+
+---
+
+## Tekton Pipelines
+
+Build pipelines located in `cluster/tekton/pipelines/`:
+- `firecrawl-api-build.yaml` - API service build
+- `firecrawl-playwright-build.yaml` - Playwright service build
+- `firecrawl-postgres-build.yaml` - PostgreSQL build
+
+**Trigger Builds:**
+```bash
+# API build
+kubectl create -f cluster/tekton/pipelines/firecrawl-api-build.yaml
+
+# Playwright build
+kubectl create -f cluster/tekton/pipelines/firecrawl-playwright-build.yaml
+
+# PostgreSQL build
+kubectl create -f cluster/tekton/pipelines/firecrawl-postgres-build.yaml
+```
+
+**Monitor Builds:**
+```bash
+# List pipeline runs
+tkn pipelinerun list -n innoventions
+
+# Watch logs
+tkn pipelinerun logs -f <pipelinerun-name> -n innoventions
+```
+
+---
+
+## Troubleshooting
+
+### API Pod Not Starting
+**Check:**
+1. RabbitMQ health status (API depends on healthy RabbitMQ)
+2. Environment variables (ConfigMap and Secret)
+3. Database connectivity (PostgreSQL)
+
+**Commands:**
+```bash
+kubectl logs -n firecrawl deployment/api
+kubectl describe pod -n firecrawl -l app=api
+kubectl get externalsecret -n firecrawl
+```
+
+### Build Failures
+**Check:**
+1. Harbor connectivity
+2. Harbor credentials secret
+3. Build resource limits (increase if OOM)
+
+**Commands:**
+```bash
+tkn pipelinerun describe <name> -n innoventions
+kubectl logs -n innoventions <kaniko-pod>
+```
+
+### Database Connection Errors
+**Check:**
+1. PostgreSQL pod status
+2. PVC binding
+3. Init script execution
+
+**Commands:**
+```bash
+kubectl logs -n firecrawl statefulset/nuq-postgres
+kubectl exec -it -n firecrawl nuq-postgres-0 -- psql -U postgres -d postgres -c '\d nuq.queue_scrape'
+```
+
+---
+
+## Monitoring
+
+**Resource Usage:**
+```bash
+kubectl top pods -n firecrawl
+```
+
+**Service Health:**
+```bash
+# API liveness
+curl -k https://spaceship-earth.local.mk-labs.cloud/v0/health/liveness
+
+# API readiness
+curl -k https://spaceship-earth.local.mk-labs.cloud/v0/health/readiness
+
+# Playwright health
+kubectl exec -n firecrawl deployment/playwright-service -- curl localhost:3000/health
+```
+
+**Queue Status:**
+Navigate to: https://spaceship-earth.local.mk-labs.cloud/admin/[BULL_AUTH_KEY]/queues
+
+---
+
+## Security Notes
+
+1. **PostgreSQL Credentials:** Stored in 1Password, synced via ExternalSecret
+2. **Admin UI:** Protected by BULL_AUTH_KEY (in URL path)
+3. **Database Port:** NOT exposed outside cluster (ClusterIP only)
+4. **TLS:** All external traffic encrypted via cert-manager certificates
+5. **RBAC:** Service accounts scoped to firecrawl namespace
+
+---
+
+## References
+
+- **Upstream Docs:** https://docs.firecrawl.dev
+- **GitHub:** https://github.com/mendableai/firecrawl
+- **Self-Hosting Guide:** https://github.com/mendableai/firecrawl/blob/main/SELF_HOST.md
+- **K8s Examples:** `/tmp/firecrawl/examples/kubernetes/`
+- **Mission Brief:** `~/friday/inbox/agents/rocket/FIRECRAWL-DEPLOYMENT-MISSION-BRIEF.md`
+
+---
+
+## Changelog
+
+### 2026-06-06 - Day 1 Complete
+- Initial repository structure created
+- Architecture decisions finalized
+- Build strategy documented
+- Environment variables researched
+- Ready for Day 2 (pipeline creation)
+
+---
+
+**Contact:** Rocket Raccoon (CI/CD Specialist)  
+**Project Manager:** Pepper Potts  
+**Cluster:** fastpass (Talos Kubernetes)  
+**Last Updated:** June 6, 2026

cluster/applications/minecraft/DEPLOYMENT-PLAN.md

@@ -0,0 +1,304 @@
+# Journey Into Imagination — Minecraft Server Deployment Plan
+
+**Application:** PaperMC Minecraft Server  
+**Namespace:** `minecraft`  
+**Deployment name:** `papermc`  
+**Service / DNS name:** `journey-into-imagination`  
+**ArgoCD path:** `cluster/applications/minecraft/`  
+**Reviewed:** _Pending Ryan approval_
+
+---
+
+## Version Pinning
+
+| Component | Version |
+|-----------|---------|
+| itzg/minecraft-server image | `2026.7.0` |
+| PaperMC (Minecraft version) | `1.21.4` (build 232 — last stable 1.21.x) |
+| SleepMost plugin | `5.6.2` (auto-downloaded at server start) |
+
+> **Note on PaperMC versioning:** PaperMC migrated to a new version scheme (`26.x`) as of mid-2026. The task requested 1.21.x, so we pin `VERSION=1.21.4`. When you're ready to upgrade to the latest stable, change `VERSION` in `deployment.yaml` and `configmap.yaml` to `26.2` and bump the image tag to the latest `itzg/minecraft-server` release. Check plugin compatibility (SleepMost) before upgrading.
+
+---
+
+## Storage
+
+- **StorageClass:** `pure-block` (confirmed live — `kubectl get storageclass`)
+- **PVC size:** 50Gi — expandable via Pure Storage CSI if the world grows
+- **Mount:** `/data` inside the container (worlds, plugins, configs all live here)
+
+---
+
+## Networking Architecture
+
+```
+Internet (players)
+    |
+    | DNS: journey-into-imagination.mk-labs.cloud → WAN IP (Cloudflare A record)
+    |
+UniFi UDM Pro
+    | Port Forward: WAN:25565 → 10.1.71.80:25565 (TCP)
+    |
+ingress-nginx LoadBalancer (10.1.71.80)
+    | tcp-services ConfigMap: 25565 → minecraft/journey-into-imagination:25565
+    |
+journey-into-imagination Service (ClusterIP, minecraft namespace)
+    |
+papermc Pod (port 25565)
+```
+
+---
+
+## Prerequisites — Manual Steps Before Deploying
+
+### 1. Generate a real RCON password
+
+```bash
+# Generate a secure password
+openssl rand -base64 24
+# Output example: abc123... (save this — you'll need it)
+
+# Base64-encode it for the Secret
+echo -n 'YOUR_GENERATED_PASSWORD' | base64
+```
+
+Edit `cluster/applications/minecraft/secret.yaml` and replace the placeholder value:
+```yaml
+data:
+  rcon-password: <your-base64-encoded-password>
+```
+
+> ⚠️  Do NOT commit real credentials in plaintext. If this is a concern long-term,
+> migrate to ExternalSecrets + 1Password (Day 2 operation).
+
+### 2. UniFi Port Forward (Ryan — manual step)
+
+In UniFi Network → Firewall & Security → Port Forwarding:
+- **Name:** `minecraft-papermc`
+- **Protocol:** TCP
+- **External Port:** 25565
+- **Internal IP:** 10.1.71.80 (ingress-nginx LoadBalancer)
+- **Internal Port:** 25565
+- **Enabled:** Yes
+
+### 3. Cloudflare DNS Records (Ryan — manual step)
+
+In Cloudflare Dashboard → mk-labs.cloud zone:
+
+**A Record (public server address):**
+| Type | Name | Value | Proxy |
+|------|------|-------|-------|
+| A | `journey-into-imagination` | `<your-WAN-IP>` | DNS only (gray cloud) |
+
+> Minecraft uses raw TCP — Cloudflare proxy (orange cloud) will NOT work.
+> Use DNS-only mode (gray cloud).
+
+**SRV Record (allows clients to connect without specifying port):**
+| Type | Name | Service | Proto | Priority | Weight | Port | Target |
+|------|------|---------|-------|----------|--------|------|--------|
+| SRV | `_minecraft._tcp.journey-into-imagination` | `_minecraft` | `_tcp` | 0 | 5 | 25565 | `journey-into-imagination.mk-labs.cloud` |
+
+> The SRV record allows players to connect using `journey-into-imagination.mk-labs.cloud`
+> without specifying `:25565`. Most modern Minecraft clients resolve SRV records.
+
+### 4. Internal DNS (Technitium — automated via ExternalDNS)
+
+The Service in `service.yaml` has this annotation:
+```yaml
+external-dns.alpha.kubernetes.io/hostname: journey-into-imagination.local.mk-labs.cloud
+```
+
+ExternalDNS (Technitium provider) will create the internal A record automatically when ArgoCD syncs.
+No manual action required for internal DNS.
+
+---
+
+## Deployment Steps
+
+### Step 1: Update the RCON Secret
+
+Complete prerequisite #1 above, then commit the updated secret.
+
+### Step 2: Commit and Push to Gitea
+
+```bash
+cd ~/git/homelab
+git status   # review what changed
+git add cluster/applications/minecraft/ cluster/platform/ingress-nginx/values.yaml
+git commit -m "feat(minecraft): add Journey Into Imagination PaperMC server
+
+- Namespace, Deployment, Service (journey-into-imagination), PVC, Secret, ConfigMap
+- ArgoCD Application at cluster/applications/minecraft/
+- ingress-nginx TCP forwarding: 25565 -> minecraft/journey-into-imagination:25565
+- PaperMC 1.21.4, itzg/minecraft-server:2026.7.0, SleepMost 5.6.2
+- StorageClass: pure-block, 50Gi PVC for world data"
+
+git push
+```
+
+### Step 3: Verify ArgoCD Discovers and Syncs
+
+ArgoCD app-of-apps auto-discovers `cluster/applications/minecraft/application.yaml`.
+
+```bash
+# Watch ArgoCD pick it up (from carousel-of-progress)
+kubectl get application minecraft -n argocd -w
+
+# Or check ArgoCD UI at argocd.local.mk-labs.cloud
+```
+
+Wait for status: `Synced` / `Healthy`
+
+### Step 4: Watch the Pod Start Up
+
+First startup will download the PaperMC jar — this can take 2-3 minutes.
+
+```bash
+kubectl get pods -n minecraft -w
+
+# Tail the logs to watch PaperMC boot
+kubectl logs -n minecraft -l app.kubernetes.io/component=papermc -f
+```
+
+Look for:
+```
+[Server thread/INFO]: Done (X.XXXs)! For help, type "help"
+```
+
+### Step 5: Install SleepMost Plugin (auto via PLUGINS env var)
+
+The `PLUGINS` env var in the ConfigMap points to the SleepMost JAR download URL.
+itzg/minecraft-server downloads and installs it automatically on startup.
+
+Verify it loaded:
+```bash
+kubectl exec -n minecraft -it deployment/papermc -- rcon-cli
+# In rcon console:
+plugins
+# Should list: SleepMost
+```
+
+### Step 6: Verify Connectivity
+
+**Internal test (from carousel-of-progress):**
+```bash
+# Port probe — should succeed
+nc -zv journey-into-imagination.local.mk-labs.cloud 25565
+
+# Or via the ClusterIP directly
+kubectl get svc -n minecraft
+nc -zv <CLUSTER-IP> 25565
+```
+
+**External test (after UniFi port forward + DNS configured):**
+- Open Minecraft Java Edition
+- Add server: `journey-into-imagination.mk-labs.cloud`
+- Should connect and show MOTD: "Journey Into Imagination"
+
+---
+
+## Plugin Details
+
+### SleepMost v5.6.2
+
+- **Source:** https://github.com/mrgeneralq/sleep-most
+- **Download:** https://github.com/mrgeneralq/sleep-most/releases/download/v5.6.2/SleepMost-5.6.2.jar
+- **Compatibility:** PaperMC 1.8 through 1.21.x (use 5.6.2 for 1.21.x; 5.7.0+ drops backward compat)
+- **Configuration:** Lives at `/data/plugins/SleepMost/config.yml` after first boot
+
+Default behavior: configurable percentage of online players must sleep to skip night.
+To tune (exec into the pod after first start):
+```bash
+kubectl exec -n minecraft -it deployment/papermc -- bash
+cat /data/plugins/SleepMost/config.yml
+# Edit as needed, then /sleepmost reload in rcon
+```
+
+> **Geyser compatibility note:** SleepMost 5.x does NOT require Geyser to function.
+> If you add Geyser later for Bedrock crossplay, SleepMost is compatible.
+
+---
+
+## Day 2 Operations
+
+### Enabling the Whitelist
+
+When ready to lock the server to approved players:
+
+1. Edit `cluster/applications/minecraft/configmap.yaml`:
+   ```yaml
+   WHITE_LIST: "true"
+   ENFORCE_WHITELIST: "true"
+   ```
+
+2. Add players via RCON (live, no restart needed):
+   ```bash
+   kubectl exec -n minecraft -it deployment/papermc -- rcon-cli
+   whitelist add <player_name>
+   whitelist list
+   ```
+
+3. Commit the ConfigMap change for GitOps consistency.
+   Note: the Deployment will restart when the ConfigMap changes (env var reload).
+   Players will be briefly disconnected — plan accordingly.
+
+### Console / RCON Access
+
+```bash
+# Interactive RCON (from inside the pod)
+kubectl exec -n minecraft -it deployment/papermc -- rcon-cli
+
+# One-shot command
+kubectl exec -n minecraft -it deployment/papermc -- rcon-cli "list"
+kubectl exec -n minecraft -it deployment/papermc -- rcon-cli "say Server restarting in 5 minutes"
+```
+
+### Upgrading PaperMC Version
+
+1. Check PaperMC API: `curl -sA 'Mozilla/5.0' https://fill.papermc.io/v3/projects/paper/versions | head`
+2. Update `VERSION` in `deployment.yaml` env vars and the comment header
+3. Update image tag if a new `itzg/minecraft-server` release is out
+4. Verify SleepMost compatibility with the new version
+5. Commit, push — ArgoCD handles the rolling restart (Recreate strategy)
+
+### Expanding PVC Storage
+
+Pure Storage CSI supports online volume expansion:
+```bash
+kubectl patch pvc papermc-world-data -n minecraft \
+  -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'
+```
+No pod restart required.
+
+### Backups
+
+No automated backup is configured in this skeleton.
+Recommended Day 2 addition: CronJob that runs `rcon-cli save-all` + `rcon-cli save-off`,
+copies `/data/world*` to a separate PVC or object store, then `rcon-cli save-on`.
+
+---
+
+## Troubleshooting
+
+| Symptom | Check |
+|---------|-------|
+| Pod stuck in `Init` / slow start | `kubectl logs -n minecraft -l app.kubernetes.io/component=papermc` — jar download may be slow |
+| Port 25565 connection refused externally | Verify UniFi port forward is active; `kubectl get svc -n minecraft` shows correct ClusterIP |
+| Players can't authenticate | `ONLINE_MODE=true` requires Mojang auth; check player has a valid Java account |
+| World data lost after pod restart | Verify PVC is `Bound`; check `pure-block` StorageClass is healthy |
+| EULA error in logs | `EULA=TRUE` is set in deployment.yaml — this should not occur; check env var injection |
+
+---
+
+## File Manifest Summary
+
+| File | Purpose |
+|------|---------|
+| `namespace.yaml` | `minecraft` namespace |
+| `application.yaml` | ArgoCD Application (sync-wave 20) |
+| `pvc.yaml` | 50Gi pure-block PVC for world data |
+| `secret.yaml` | RCON password (⚠️ placeholder — update before deploy) |
+| `configmap.yaml` | server.properties overrides + SleepMost plugin download |
+| `deployment.yaml` | PaperMC Deployment (Recreate strategy, resource limits set) |
+| `service.yaml` | ClusterIP Service named `journey-into-imagination` |
+| `../../../platform/ingress-nginx/values.yaml` | Added `tcp: 25565` forwarding entry |

cluster/applications/minecraft/application.yaml

@@ -0,0 +1,26 @@
+---
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: minecraft
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "20"
+spec:
+  project: default
+  source:
+    repoURL: https://gitea.mk-labs.cloud/rblundon/homelab.git
+    targetRevision: main
+    path: cluster/applications/minecraft
+    directory:
+      recurse: false
+      exclude: application.yaml  # prevent self-reference loop with apps-of-apps
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: minecraft
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true

cluster/applications/minecraft/backup-cronjob.yaml

@@ -0,0 +1,124 @@
+---
+# Minecraft world backup — hourly CronJob, 3-day local retention
+#
+# Strategy (two-tier):
+#   Short-term : this CronJob — hourly tarballs on papermc-backups PVC, 72-hour retention
+#   Long-term  : Pure FlashArray protection group snapshots on utilidor (managed separately)
+#
+# Backup sequence:
+#   1. RCON save-all  — flush all dirty chunks to disk
+#   2. RCON save-off  — pause auto-save to keep the world consistent during tar
+#   3. tar world directories to /backups/world-YYYY-MM-DDTHH-MM.tar.gz
+#   4. RCON save-on   — re-enable auto-save
+#   5. Prune backups older than 3 days
+#
+# RCON password sourced from the papermc-rcon Secret (ESO-managed, same as server).
+# The backup pod mounts both PVCs read-write; world-data is safe because the
+# server has already quiesced saves via RCON before the tar runs.
+#
+# Note: both PVCs are RWO. The backup job runs only while the main server pod is
+# running (RCON is reachable), so there is no volume attach conflict — they are
+# mounted on the same node by the scheduler. If the server pod is down, the backup
+# job will fail at the RCON step, which is correct behavior (nothing to back up).
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: minecraft-backup
+  namespace: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: backup
+spec:
+  schedule: "0 * * * *"     # every hour on the hour
+  concurrencyPolicy: Forbid  # skip if a previous backup is still running
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      backoffLimit: 0        # don't retry — a partial backup is worse than no backup
+      template:
+        metadata:
+          labels:
+            app.kubernetes.io/name: minecraft
+            app.kubernetes.io/component: backup
+        spec:
+          restartPolicy: Never
+          securityContext:
+            runAsNonRoot: false
+            fsGroup: 1000
+          containers:
+            - name: backup
+              # Same image as the server — has both /bin/sh and rcon-cli built in.
+              # itzg/rcon-cli is distroless (no shell); don't use it for scripted jobs.
+              image: itzg/minecraft-server:2026.7.0
+              imagePullPolicy: IfNotPresent
+              env:
+                - name: RCON_HOST
+                  value: "journey-into-imagination.minecraft.svc.cluster.local"
+                - name: RCON_PORT
+                  value: "25575"
+                - name: RCON_PASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: papermc-rcon
+                      key: rcon-password
+              command:
+                - /bin/sh
+                - -c
+                - |
+                  set -e
+
+                  TIMESTAMP=$(date -u +%Y-%m-%dT%H-%M)
+                  BACKUP_FILE="/backups/world-${TIMESTAMP}.tar.gz"
+                  RETAIN_DAYS=3
+
+                  echo "[backup] Starting backup at ${TIMESTAMP}"
+
+                  # Step 1: quiesce the world
+                  echo "[backup] Flushing chunks (save-all)..."
+                  rcon-cli save-all
+
+                  echo "[backup] Pausing auto-save (save-off)..."
+                  rcon-cli save-off
+
+                  # Step 2: archive world directories (nether/end may not exist yet)
+                  echo "[backup] Archiving world to ${BACKUP_FILE}..."
+                  DIRS_TO_BACKUP="world"
+                  [ -d /data/world_nether ] && DIRS_TO_BACKUP="$DIRS_TO_BACKUP world_nether"
+                  [ -d /data/world_the_end ] && DIRS_TO_BACKUP="$DIRS_TO_BACKUP world_the_end"
+                  FILES_TO_BACKUP=""
+                  [ -f /data/whitelist.json ] && FILES_TO_BACKUP="$FILES_TO_BACKUP whitelist.json"
+                  [ -f /data/ops.json ] && FILES_TO_BACKUP="$FILES_TO_BACKUP ops.json"
+                  [ -f /data/banned-players.json ] && FILES_TO_BACKUP="$FILES_TO_BACKUP banned-players.json"
+                  [ -f /data/banned-ips.json ] && FILES_TO_BACKUP="$FILES_TO_BACKUP banned-ips.json"
+
+                  tar -czf "${BACKUP_FILE}" -C /data $DIRS_TO_BACKUP $FILES_TO_BACKUP || {
+                    echo "[backup] ERROR: tar failed — re-enabling saves and exiting"
+                    rcon-cli save-on
+                    exit 1
+                  }
+
+                  echo "[backup] Archive complete: $(du -sh ${BACKUP_FILE} | cut -f1)"
+
+                  # Step 3: resume auto-save
+                  echo "[backup] Resuming auto-save (save-on)..."
+                  rcon-cli save-on
+
+                  # Step 4: prune old backups
+                  echo "[backup] Pruning backups older than ${RETAIN_DAYS} days..."
+                  find /backups -name "world-*.tar.gz" -mtime +${RETAIN_DAYS} -delete
+                  REMAINING=$(find /backups -name "world-*.tar.gz" | wc -l)
+                  echo "[backup] Done. ${REMAINING} backup(s) retained."
+              volumeMounts:
+                - name: world-data
+                  mountPath: /data
+                  readOnly: true
+                - name: backups
+                  mountPath: /backups
+          volumes:
+            - name: world-data
+              persistentVolumeClaim:
+                claimName: papermc-world-data
+            - name: backups
+              persistentVolumeClaim:
+                claimName: papermc-backups

cluster/applications/minecraft/backup-pvc.yaml

@@ -0,0 +1,19 @@
+---
+# Minecraft backup storage — local hourly backups, 3-day retention
+# Sized for ~72 backup tarballs; world data compresses well (~90% reduction typical)
+# Long-term retention handled by Pure FlashArray protection group snapshots (utilidor)
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: papermc-backups
+  namespace: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: backup
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: px-fa-direct-access
+  resources:
+    requests:
+      storage: 10Gi

cluster/applications/minecraft/configmap.yaml

@@ -0,0 +1,58 @@
+---
+# server.properties overrides for Journey Into Imagination
+# itzg/minecraft-server reads these as environment variables and
+# writes them into server.properties on startup.
+# Full list: https://docker-minecraft-server.readthedocs.io/en/latest/configuration/server-properties/
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: papermc-config
+  namespace: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: papermc
+data:
+  # Server identity
+  MOTD: "Journey Into Imagination"
+  SERVER_NAME: "Journey Into Imagination"
+
+  # Game settings
+  GAMEMODE: "survival"
+  DIFFICULTY: "normal"
+  PVP: "true"
+  MAX_PLAYERS: "20"
+  ALLOW_FLIGHT: "true"
+  SPAWN_PROTECTION: "0"
+  SEED: "-2786778798491440147"
+
+  # Auth
+  ONLINE_MODE: "false"
+  # Mojang's sessionserver blocks some datacenter/homelab IPs via Cloudflare WAF.
+  # enforce-secure-profile=false allows players to join without mandatory profile key validation.
+  # ONLINE_MODE=true still enforces Mojang account auth — this only relaxes the key signing requirement.
+  ENFORCE_SECURE_PROFILE: "false"
+
+  # Whitelist — set to true when ready to lock down the server
+  WHITE_LIST: "false"
+  ENFORCE_WHITELIST: "false"
+
+  # Performance / tick safety
+  MAX_TICK_TIME: "180000"
+
+  # RCON (enabled so plugins/admin tools can connect)
+  ENABLE_RCON: "true"
+  RCON_PORT: "25575"
+
+  # Plugin auto-download — minecraft-prometheus-exporter for Grafana metrics
+  # v3.1.2: https://github.com/sladkoff/minecraft-prometheus-exporter
+  # Exposes /metrics on port 9225 (HTTP, Prometheus scrape target)
+  #
+  # SkinsRestorer v15.12.4 — restores player skins in offline-mode
+  # https://github.com/SkinsRestorer/SkinsRestorer
+  # Players set skins via /skin <username>; fetches from Mojang even in offline-mode
+  PLUGINS: |
+    https://github.com/sladkoff/minecraft-prometheus-exporter/releases/download/v3.1.2/minecraft-prometheus-exporter-3.1.2.jar
+    https://github.com/SkinsRestorer/SkinsRestorer/releases/download/15.12.4/SkinsRestorer.jar
+    https://github.com/mrgeneralq/sleep-most/releases/download/v5.5.3/sleep-most-5.5.3.jar
+    https://github.com/852DuartePls/Bukkit-AntiSilverFish/releases/download/v0.0.4/AntiSilverFish-0.0.4.jar
+

cluster/applications/minecraft/deployment.yaml

@@ -0,0 +1,202 @@
+---
+# PaperMC Minecraft Server — Journey Into Imagination
+#
+# Image: itzg/minecraft-server:2026.7.0
+# PaperMC version: 26.2 (build 62, latest stable as of 2026-07-19; requires Java 25)
+# NOTE: 26.x series uses PaperMC's independent versioning (not Minecraft version-based).
+#       Plugin ecosystem is still catching up — verify plugin compat before adding new ones.
+#
+# itzg image documentation: https://docker-minecraft-server.readthedocs.io
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: papermc
+  namespace: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: papermc
+    app.kubernetes.io/version: "26.2"
+spec:
+  replicas: 1
+  # Recreate strategy — Minecraft server requires exclusive access to world data.
+  # RollingUpdate WILL cause world corruption if both pods mount the PVC simultaneously.
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: minecraft
+      app.kubernetes.io/component: papermc
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: minecraft
+        app.kubernetes.io/component: papermc
+        app.kubernetes.io/version: "26.2"
+    spec:
+      securityContext:
+        runAsNonRoot: false  # itzg image requires root for some setup steps
+        fsGroup: 1000
+      containers:
+        - name: papermc
+          # Pinned tag — never use :latest in production
+          image: itzg/minecraft-server:2026.7.0
+          imagePullPolicy: IfNotPresent
+          ports:
+            - name: minecraft
+              containerPort: 25565
+              protocol: TCP
+            - name: rcon
+              containerPort: 25575
+              protocol: TCP
+            - name: metrics
+              containerPort: 9225
+              protocol: TCP
+          env:
+            - name: EULA
+              value: "TRUE"
+            - name: TYPE
+              value: "PAPER"
+            - name: VERSION
+              value: "26.2"
+            # Server settings from ConfigMap
+            - name: MOTD
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: MOTD
+            - name: SERVER_NAME
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: SERVER_NAME
+            - name: GAMEMODE
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: GAMEMODE
+            - name: DIFFICULTY
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: DIFFICULTY
+            - name: PVP
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: PVP
+            - name: MAX_PLAYERS
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: MAX_PLAYERS
+            - name: ALLOW_FLIGHT
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: ALLOW_FLIGHT
+            - name: SPAWN_PROTECTION
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: SPAWN_PROTECTION
+            - name: SEED
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: SEED
+            - name: ONLINE_MODE
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: ONLINE_MODE
+            - name: ENFORCE_SECURE_PROFILE
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: ENFORCE_SECURE_PROFILE
+            - name: WHITE_LIST
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: WHITE_LIST
+            - name: ENFORCE_WHITELIST
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: ENFORCE_WHITELIST
+            - name: MAX_TICK_TIME
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: MAX_TICK_TIME
+            - name: ENABLE_RCON
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: ENABLE_RCON
+            - name: RCON_PORT
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: RCON_PORT
+            # Plugin auto-download — itzg image fetches these URLs on startup
+            - name: PLUGINS
+              valueFrom:
+                configMapKeyRef:
+                  name: papermc-config
+                  key: PLUGINS
+            # RCON password from Secret
+            - name: RCON_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: papermc-rcon
+                  key: rcon-password
+          volumeMounts:
+            - name: world-data
+              mountPath: /data
+            - name: sleep-most-config
+              mountPath: /data/plugins/sleep-most/config.yml
+              subPath: config.yml
+              readOnly: true
+            - name: prometheus-exporter-config
+              mountPath: /data/plugins/PrometheusExporter/config.yml
+              subPath: config.yml
+              readOnly: true
+          resources:
+            requests:
+              cpu: "1"
+              memory: "2Gi"
+            limits:
+              cpu: "4"
+              memory: "6Gi"
+          # Startup probe — give the server time to download PaperMC jar on first boot
+          startupProbe:
+            tcpSocket:
+              port: minecraft
+            initialDelaySeconds: 30
+            periodSeconds: 10
+            failureThreshold: 30  # 5 minutes total
+          # Liveness probe — restart if the TCP port goes away
+          livenessProbe:
+            tcpSocket:
+              port: minecraft
+            initialDelaySeconds: 0
+            periodSeconds: 30
+            failureThreshold: 3
+          # Readiness probe — only route traffic when server is accepting connections
+          readinessProbe:
+            tcpSocket:
+              port: minecraft
+            initialDelaySeconds: 0
+            periodSeconds: 10
+            failureThreshold: 3
+      volumes:
+        - name: world-data
+          persistentVolumeClaim:
+            claimName: papermc-world-data
+        - name: sleep-most-config
+          configMap:
+            name: sleep-most-config
+        - name: prometheus-exporter-config
+          configMap:
+            name: prometheus-exporter-config

cluster/applications/minecraft/externalsecret.yaml

@@ -0,0 +1,28 @@
+---
+# ExternalSecret — pulls RCON password from 1Password Connect
+# Prereq: create a "minecraft" item in the mk-labs 1Password vault
+#   with a field named "rcon_password" set to a strong random password.
+#   Generate one: openssl rand -base64 24
+#
+# ClusterSecretStore: onepassword-connect (platform/onepassword-connect/)
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: papermc-rcon
+  namespace: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: papermc
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: onepassword-connect
+    kind: ClusterSecretStore
+  target:
+    name: papermc-rcon
+    creationPolicy: Owner
+  data:
+    - secretKey: rcon-password
+      remoteRef:
+        key: minecraft
+        property: rcon_password

cluster/applications/minecraft/grafana-dashboard.yaml

@@ -0,0 +1,704 @@
+---
+# Grafana dashboard — Minecraft server stats (ID 20659)
+# Source: https://grafana.com/grafana/dashboards/20659
+# Plugin: sladkoff/minecraft-prometheus-exporter v3.1.2
+#
+# NOTE: Dashboard 20659 was written for exporter v1/v2 (no mc_ prefix).
+# Queries have been fixed for v3 metric names (mc_tps, mc_players_online_total, etc.)
+# Players score / Play time / Advancements panels show mc_players_total as placeholder —
+# per-player stat tracking metrics are not available in v3.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: dashboard-minecraft
+  namespace: monitoring
+  labels:
+    grafana_dashboard: "1"
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: grafana-dashboard
+data:
+  minecraft-server-stats.json: |
+    {
+      "annotations": {
+        "list": [
+          {
+            "builtIn": 1,
+            "datasource": {
+              "type": "datasource",
+              "uid": "grafana"
+            },
+            "enable": true,
+            "hide": true,
+            "iconColor": "rgba(0, 211, 255, 1)",
+            "name": "Annotations & Alerts",
+            "type": "dashboard"
+          }
+        ]
+      },
+      "description": "Modern dashboard for minecraft-prometheus-exporter\r\nhttps://github.com/Joshi425/minecraft-exporter",
+      "editable": true,
+      "fiscalYearStartMonth": 0,
+      "gnetId": 20659,
+      "graphTooltip": 0,
+      "id": null,
+      "links": [],
+      "panels": [
+        {
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 0
+          },
+          "id": 16,
+          "panels": [],
+          "title": "Server stats",
+          "type": "row"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "thresholds"
+              },
+              "custom": {
+                "align": "auto",
+                "cellOptions": {
+                  "type": "auto"
+                },
+                "inspect": false
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              }
+            },
+            "overrides": [
+              {
+                "matcher": {
+                  "id": "byName",
+                  "options": "Time"
+                },
+                "properties": [
+                  {
+                    "id": "displayName",
+                    "value": "Time"
+                  },
+                  {
+                    "id": "custom.align"
+                  }
+                ]
+              },
+              {
+                "matcher": {
+                  "id": "byRegexp",
+                  "options": "/Value/"
+                },
+                "properties": [
+                  {
+                    "id": "unit",
+                    "value": "short"
+                  },
+                  {
+                    "id": "decimals",
+                    "value": 2
+                  },
+                  {
+                    "id": "custom.align"
+                  }
+                ]
+              }
+            ]
+          },
+          "gridPos": {
+            "h": 9,
+            "w": 4,
+            "x": 0,
+            "y": 1
+          },
+          "id": 9,
+          "options": {
+            "cellHeight": "sm",
+            "footer": {
+              "countRows": false,
+              "fields": "",
+              "reducer": [
+                "sum"
+              ],
+              "show": false
+            },
+            "showHeader": true
+          },
+          "pluginVersion": "10.4.0",
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "prometheus"
+              },
+              "editorMode": "code",
+              "expr": "mc_players_online_total",
+              "instant": true,
+              "legendFormat": "{{player}}",
+              "refId": "A"
+            }
+          ],
+          "title": "Players",
+          "transformations": [
+            {
+              "id": "merge",
+              "options": {
+                "reducers": []
+              }
+            }
+          ],
+          "type": "table"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "description": "",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "continuous-GrYlRd"
+              },
+              "custom": {
+                "axisBorderShow": false,
+                "axisCenteredZero": false,
+                "axisColorMode": "text",
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 25,
+                "gradientMode": "scheme",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "insertNulls": false,
+                "lineInterpolation": "linear",
+                "lineStyle": {
+                  "fill": "solid"
+                },
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "auto",
+                "spanNulls": 3600000,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "min": 0,
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "ms"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 9,
+            "w": 10,
+            "x": 4,
+            "y": 1
+          },
+          "id": 13,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom",
+              "showLegend": true
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "none"
+            }
+          },
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "prometheus"
+              },
+              "disableTextWrap": false,
+              "editorMode": "builder",
+              "exemplar": false,
+              "expr": "mc_tick_duration_median",
+              "fullMetaSearch": false,
+              "includeNullMetadata": true,
+              "instant": false,
+              "legendFormat": "{{dimension_id}}",
+              "range": true,
+              "refId": "A",
+              "useBackend": false
+            }
+          ],
+          "title": "Time per tick",
+          "type": "timeseries"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "description": "",
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "continuous-RdYlGr"
+              },
+              "custom": {
+                "axisBorderShow": false,
+                "axisCenteredZero": false,
+                "axisColorMode": "text",
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 27,
+                "gradientMode": "opacity",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "insertNulls": false,
+                "lineInterpolation": "linear",
+                "lineStyle": {
+                  "fill": "solid"
+                },
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "auto",
+                "spanNulls": 3600000,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "max": 20,
+              "min": 0,
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              }
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 9,
+            "w": 10,
+            "x": 14,
+            "y": 1
+          },
+          "id": 12,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom",
+              "showLegend": true
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "none"
+            }
+          },
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "prometheus"
+              },
+              "disableTextWrap": false,
+              "editorMode": "builder",
+              "exemplar": false,
+              "expr": "mc_tps",
+              "fullMetaSearch": false,
+              "includeNullMetadata": false,
+              "instant": false,
+              "legendFormat": "{{ dimension_id }}",
+              "range": true,
+              "refId": "A",
+              "useBackend": false
+            }
+          ],
+          "title": "Ticks per second",
+          "type": "timeseries"
+        },
+        {
+          "collapsed": false,
+          "gridPos": {
+            "h": 1,
+            "w": 24,
+            "x": 0,
+            "y": 10
+          },
+          "id": 15,
+          "panels": [],
+          "title": "Player & entity metrics",
+          "type": "row"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                }
+              },
+              "mappings": []
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 13,
+            "x": 0,
+            "y": 11
+          },
+          "id": 14,
+          "options": {
+            "legend": {
+              "displayMode": "table",
+              "placement": "right",
+              "showLegend": true,
+              "values": [
+                "value"
+              ]
+            },
+            "pieType": "donut",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "tooltip": {
+              "mode": "single",
+              "sort": "none"
+            }
+          },
+          "pluginVersion": "10.4.0",
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "prometheus"
+              },
+              "disableTextWrap": false,
+              "editorMode": "builder",
+              "expr": "sum(mc_loaded_chunks_total)",
+              "fullMetaSearch": false,
+              "includeNullMetadata": true,
+              "instant": false,
+              "legendFormat": "{{entity}}",
+              "range": true,
+              "refId": "A",
+              "useBackend": false
+            }
+          ],
+          "title": "Entities loaded",
+          "type": "piechart"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                }
+              },
+              "mappings": []
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 8,
+            "w": 11,
+            "x": 13,
+            "y": 11
+          },
+          "id": 18,
+          "options": {
+            "legend": {
+              "displayMode": "table",
+              "placement": "right",
+              "showLegend": true,
+              "values": [
+                "value",
+                "percent"
+              ]
+            },
+            "pieType": "pie",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "none"
+            }
+          },
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "prometheus"
+              },
+              "disableTextWrap": false,
+              "editorMode": "builder",
+              "expr": "mc_players_total",
+              "fullMetaSearch": false,
+              "includeNullMetadata": true,
+              "instant": false,
+              "legendFormat": "{{player}}",
+              "range": true,
+              "refId": "A",
+              "useBackend": false
+            }
+          ],
+          "title": "Players score",
+          "type": "piechart"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "continuous-BlPu"
+              },
+              "links": [],
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "s"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 7,
+            "w": 13,
+            "x": 0,
+            "y": 19
+          },
+          "id": 3,
+          "options": {
+            "displayMode": "gradient",
+            "maxVizHeight": 300,
+            "minVizHeight": 16,
+            "minVizWidth": 8,
+            "namePlacement": "top",
+            "orientation": "horizontal",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "showUnfilled": true,
+            "sizing": "auto",
+            "text": {},
+            "valueMode": "color"
+          },
+          "pluginVersion": "10.4.0",
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "prometheus"
+              },
+              "editorMode": "code",
+              "expr": "mc_players_total",
+              "legendFormat": "{{ player }}",
+              "range": true,
+              "refId": "A"
+            }
+          ],
+          "title": "Play time",
+          "type": "bargauge"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "prometheus"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "continuous-BlPu"
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              }
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 7,
+            "w": 11,
+            "x": 13,
+            "y": 19
+          },
+          "id": 17,
+          "options": {
+            "displayMode": "gradient",
+            "maxVizHeight": 300,
+            "minVizHeight": 16,
+            "minVizWidth": 8,
+            "namePlacement": "auto",
+            "orientation": "horizontal",
+            "reduceOptions": {
+              "calcs": [
+                "lastNotNull"
+              ],
+              "fields": "",
+              "values": false
+            },
+            "showUnfilled": true,
+            "sizing": "auto",
+            "valueMode": "color"
+          },
+          "pluginVersion": "10.4.0",
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "prometheus"
+              },
+              "disableTextWrap": false,
+              "editorMode": "code",
+              "expr": "mc_players_total",
+              "fullMetaSearch": false,
+              "includeNullMetadata": true,
+              "instant": false,
+              "legendFormat": "{{player}}",
+              "range": true,
+              "refId": "A",
+              "useBackend": false
+            }
+          ],
+          "title": "Advancements made",
+          "type": "bargauge"
+        }
+      ],
+      "refresh": "10s",
+      "schemaVersion": 39,
+      "tags": [],
+      "templating": {
+        "list": []
+      },
+      "time": {
+        "from": "now-30m",
+        "to": "now"
+      },
+      "timepicker": {},
+      "timezone": "",
+      "title": "Minecraft server stats [Prometheus]",
+      "uid": "prometheus",
+      "version": 15,
+      "weekStart": ""
+    }

cluster/applications/minecraft/namespace.yaml

@@ -0,0 +1,8 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/managed-by: argocd

cluster/applications/minecraft/prometheus-exporter-config.yaml

@@ -0,0 +1,35 @@
+---
+# minecraft-prometheus-exporter plugin configuration
+# Port set to 9225 to match Service/ServiceMonitor definitions.
+# Default is 9940 — must be explicitly set here or the plugin won't be scraped.
+# Mounted read-only at /data/plugins/PrometheusExporter/config.yml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: prometheus-exporter-config
+  namespace: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: papermc
+data:
+  config.yml: |
+    host: 0.0.0.0
+    port: 9225
+    enable_metrics:
+      entities_total: true
+      villagers_total: true
+      loaded_chunks_total: true
+      jvm_memory: true
+      players_online_total: true
+      players_total: true
+      whitelisted_players: false
+      tps: true
+      world_size: true
+      jvm_threads: true
+      jvm_gc: true
+      tick_duration_median: true
+      tick_duration_average: true
+      tick_duration_min: false
+      tick_duration_max: true
+      player_online: false
+      player_statistic: false

cluster/applications/minecraft/pvc.yaml

@@ -0,0 +1,18 @@
+---
+# World data persistent volume — Pure Storage FlashArray direct access CSI
+# StorageClass: px-fa-direct-access (direct FlashArray block, bypasses Portworx data plane)
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: papermc-world-data
+  namespace: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: papermc
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: px-fa-direct-access
+  resources:
+    requests:
+      storage: 50Gi

cluster/applications/minecraft/service.yaml

@@ -0,0 +1,39 @@
+---
+# Service named after the thematic identity: journey-into-imagination
+# This is the DNS name used in ingress-nginx TCP forwarding config.
+# Internal DNS: journey-into-imagination.local.mk-labs.cloud
+apiVersion: v1
+kind: Service
+metadata:
+  name: journey-into-imagination
+  namespace: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: papermc
+  annotations:
+    # Internal DNS via ExternalDNS + Technitium
+    external-dns.alpha.kubernetes.io/hostname: "journey-into-imagination.local.mk-labs.cloud,journey-into-imagination.mk-labs.cloud"
+    # Non-standard external port 10182 → internal 25565 (via ingress-nginx tcp forwarding)
+    # Public DNS via ExternalDNS + Cloudflare
+    # Cloudflare proxy MUST be off — TCP (non-HTTP) traffic cannot be proxied
+    external-dns.alpha.kubernetes.io/public: "true"
+    external-dns.alpha.kubernetes.io/target: "ingress.mk-labs.cloud"
+    external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
+spec:
+  type: ClusterIP
+  selector:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: papermc
+  ports:
+    - name: minecraft
+      port: 25565
+      targetPort: 25565
+      protocol: TCP
+    - name: rcon
+      port: 25575
+      targetPort: 25575
+      protocol: TCP
+    - name: metrics
+      port: 9225
+      targetPort: 9225
+      protocol: TCP

cluster/applications/minecraft/servicemonitor.yaml

@@ -0,0 +1,23 @@
+---
+# ServiceMonitor — tells Prometheus to scrape minecraft-prometheus-exporter
+# Plugin exposes /metrics on port 9225 (HTTP)
+# Dashboard: https://grafana.com/grafana/dashboards/20659
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: minecraft
+  namespace: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: papermc
+    # Label required for kube-prometheus-stack to discover this ServiceMonitor
+    release: monitoring
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: minecraft
+      app.kubernetes.io/component: papermc
+  endpoints:
+    - port: metrics
+      path: /metrics
+      interval: 15s

cluster/applications/minecraft/sleep-most-config.yaml

@@ -0,0 +1,121 @@
+---
+# sleep-most plugin configuration
+# Single-player sleep: calculation-method: players, players-required: 1
+# Mounted read-only at /data/plugins/sleep-most/config.yml
+# To change settings: edit this ConfigMap, commit, push — ArgoCD will sync,
+# then run: kubectl exec -n minecraft deployment/papermc -- rcon-cli "sleepmost reload"
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: sleep-most-config
+  namespace: minecraft
+  labels:
+    app.kubernetes.io/name: minecraft
+    app.kubernetes.io/component: papermc
+data:
+  config.yml: |
+    # WELCOME TO THE OFFICIAL SLEEP-MOST PLUGIN
+    # AUTHORS: MrGeneralQ
+    # CONTRIBUTORS: Malin, Nozemi, HorrendousEntity
+    # VERSION: 5.5.3
+    # SUPPORT NEEDED? Join our discord at --> https://discord.pseudonova.com/ WE ARE HAPPY TO HELP YOU FURTHER!
+    # YOU CAN FIND THE DOCS FOR THIS PLUGIN HERE --> https://mrgeneralq.gitbook.io/sleepmost/
+
+
+    # WARNING: SUPPORT IS ONLY GIVEN TO THE LATEST VERSION
+    update-checker-enabled: true
+
+    # specify the speed/ticks for the animation
+    # DEFAULT: 85 (recommended)
+    nightcycle-animation-speed: 85
+    nightcycle-animation-speed-max: 170
+
+
+    sleep:
+      # for each world, you can create different configurations
+      world:
+        enabled: true
+        calculation-method: players
+        percentage-required: 0.5
+        players-required: 1
+        mob-no-target: true
+        use-exempt: false
+        use-afk: false
+        use-bossbar: false
+        use-sound-night-skipped: false
+        use-sound-storm-skipped: false
+        use-title-night-skipped: false
+        use-title-storm-skipped: false
+        exempt-creative: false
+        exempt-spectator: false
+        prevent-sleep: false
+        prevent-phantom: false
+        nightcycle-animation: false
+        storm-sleep: true
+        skip-delay: 0
+        heal: false
+        feed: false
+        skip-night-sound: ui.toast.challenge_complete
+        skip-storm-sound: entity.wither.spawn
+        reset-time-since-rest: true
+        allow-kick: false
+        gsit-hook: false
+        non-sleeping-clock-animation: false
+        skip-msg-audience: all
+        gsit-sleep: true
+        insomnia-milk: false
+        gsit-sleep-cmd: false
+        force-nightcycle-animation: true
+        non-sleeping-sound: false
+        disable-daylight-cycle-gamerule: true
+        dynamic-animation-speed: false
+        phantom-reset-audience: all
+        clock-animation: true
+        skip-storm: true
+        allow-sleep-cmd: true
+        insomnia-chance: 0.0
+        exempt-below-y: -1
+        non-sleeping-title: false
+        exempt-flying: false
+
+      world2:
+        enabled: false
+        calculation-method: players
+        percentage-required: 0.5
+        players-required: 1
+        mob-no-target: true
+        use-exempt: false
+        use-afk: false
+        use-bossbar: false
+        use-sound-night-skipped: false
+        use-sound-storm-skipped: false
+        use-title-night-skipped: false
+        use-title-storm-skipped: false
+        exempt-creative: false
+        exempt-spectator: false
+        prevent-sleep: false
+        prevent-phantom: false
+        nightcycle-animation: false
+        storm-sleep: true
+        skip-delay: 0
+        heal: false
+        feed: false
+        skip-night-sound: ui.toast.challenge_complete
+        skip-storm-sound: entity.wither.spawn
+        reset-time-since-rest: true
+
+    # specify the time which the world will be set after reset
+    time-after-reset: 0
+
+    # configure when players can and cannot sleep
+    # this also controls when the animation will stop
+    # please note that this does not
+    time:
+      night-start: 12542
+      night-end: 23850
+
+    messages:
+      cooldown: 10
+
+    # debug mode
+    debug-mode: false

cluster/applications/monitoring/dashboards.yaml

@@ -9199,3 +9199,230 @@ data:
       "version": 1,
       "description": "Ingress-nginx supports a rich collection of prometheus metrics. If you have prometheus and grafana installed on your cluster then prometheus will already be scraping this data due to the scrape annotation on the deployment."
     }
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-llm-inference-dashboard
+  namespace: monitoring
+  labels:
+    grafana_dashboard: "1"
+data:
+  llm-inference-astro-orbiter.json: |
+    {
+      "title": "LLM Inference — astro-orbiter",
+      "uid": "llm-astro-orbiter",
+      "timezone": "browser",
+      "refresh": "30s",
+      "schemaVersion": 38,
+      "tags": ["llm", "gpu", "llama-server", "astro-orbiter"],
+      "time": { "from": "now-1h", "to": "now" },
+      "panels": [
+        {
+          "id": 1,
+          "title": "GPU Utilization %",
+          "type": "timeseries",
+          "gridPos": { "x": 0, "y": 0, "w": 8, "h": 8 },
+          "fieldConfig": {
+            "defaults": {
+              "unit": "percent",
+              "min": 0, "max": 100,
+              "color": { "mode": "palette-classic" },
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  { "color": "green", "value": null },
+                  { "color": "yellow", "value": 70 },
+                  { "color": "red", "value": 90 }
+                ]
+              }
+            }
+          },
+          "targets": [
+            {
+              "expr": "nvidia_smi_utilization_gpu_ratio{hostname=\"astro-orbiter\"} * 100",
+              "legendFormat": "GPU Util"
+            }
+          ]
+        },
+        {
+          "id": 2,
+          "title": "GPU VRAM Used",
+          "type": "timeseries",
+          "gridPos": { "x": 8, "y": 0, "w": 8, "h": 8 },
+          "fieldConfig": {
+            "defaults": {
+              "unit": "bytes",
+              "color": { "mode": "palette-classic" }
+            }
+          },
+          "targets": [
+            {
+              "expr": "nvidia_smi_memory_used_bytes{hostname=\"astro-orbiter\"}",
+              "legendFormat": "VRAM Used"
+            },
+            {
+              "expr": "nvidia_smi_memory_total_bytes{hostname=\"astro-orbiter\"}",
+              "legendFormat": "VRAM Total"
+            }
+          ]
+        },
+        {
+          "id": 3,
+          "title": "GPU Temperature",
+          "type": "gauge",
+          "gridPos": { "x": 16, "y": 0, "w": 8, "h": 8 },
+          "fieldConfig": {
+            "defaults": {
+              "unit": "celsius",
+              "min": 0, "max": 100,
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  { "color": "green", "value": null },
+                  { "color": "yellow", "value": 70 },
+                  { "color": "red", "value": 85 }
+                ]
+              }
+            }
+          },
+          "targets": [
+            {
+              "expr": "nvidia_smi_temperature_gpu{hostname=\"astro-orbiter\"}",
+              "legendFormat": "GPU Temp"
+            }
+          ]
+        },
+        {
+          "id": 4,
+          "title": "GPU Power Draw",
+          "type": "timeseries",
+          "gridPos": { "x": 0, "y": 8, "w": 8, "h": 8 },
+          "fieldConfig": {
+            "defaults": {
+              "unit": "watt",
+              "color": { "mode": "palette-classic" }
+            }
+          },
+          "targets": [
+            {
+              "expr": "nvidia_smi_power_draw_watts{hostname=\"astro-orbiter\"}",
+              "legendFormat": "Power Draw"
+            },
+            {
+              "expr": "nvidia_smi_power_limit_watts{hostname=\"astro-orbiter\"}",
+              "legendFormat": "Power Limit"
+            }
+          ]
+        },
+        {
+          "id": 5,
+          "title": "llama-server — Requests per Second",
+          "type": "timeseries",
+          "gridPos": { "x": 8, "y": 8, "w": 8, "h": 8 },
+          "fieldConfig": {
+            "defaults": {
+              "unit": "reqps",
+              "color": { "mode": "palette-classic" }
+            }
+          },
+          "targets": [
+            {
+              "expr": "rate(llamacpp:tokens_predicted_total{hostname=\"astro-orbiter\"}[1m])",
+              "legendFormat": "Tokens/s (predicted)"
+            },
+            {
+              "expr": "rate(llamacpp:prompt_tokens_total{hostname=\"astro-orbiter\"}[1m])",
+              "legendFormat": "Tokens/s (prompt)"
+            }
+          ]
+        },
+        {
+          "id": 6,
+          "title": "llama-server — Requests Processing/Deferred",
+          "type": "timeseries",
+          "gridPos": { "x": 16, "y": 8, "w": 8, "h": 8 },
+          "fieldConfig": {
+            "defaults": {
+              "unit": "short",
+              "color": { "mode": "palette-classic" }
+            }
+          },
+          "targets": [
+            {
+              "expr": "llamacpp:requests_processing{hostname=\"astro-orbiter\"}",
+              "legendFormat": "Processing"
+            },
+            {
+              "expr": "llamacpp:requests_deferred{hostname=\"astro-orbiter\"}",
+              "legendFormat": "Deferred"
+            }
+          ]
+        },
+        {
+          "id": 9,
+          "title": "System — CPU Usage %",
+          "type": "timeseries",
+          "gridPos": { "x": 0, "y": 16, "w": 8, "h": 7 },
+          "fieldConfig": {
+            "defaults": {
+              "unit": "percent",
+              "min": 0, "max": 100
+            }
+          },
+          "targets": [
+            {
+              "expr": "100 - (avg by(instance) (rate(node_cpu_seconds_total{mode=\"idle\",instance=~\"10.1.71.130:.*\"}[1m])) * 100)",
+              "legendFormat": "CPU Used"
+            }
+          ]
+        },
+        {
+          "id": 10,
+          "title": "System — Memory Usage",
+          "type": "timeseries",
+          "gridPos": { "x": 8, "y": 16, "w": 8, "h": 7 },
+          "fieldConfig": {
+            "defaults": {
+              "unit": "bytes"
+            }
+          },
+          "targets": [
+            {
+              "expr": "node_memory_MemTotal_bytes{instance=~\"10.1.71.130:.*\"} - node_memory_MemAvailable_bytes{instance=~\"10.1.71.130:.*\"}",
+              "legendFormat": "Used"
+            },
+            {
+              "expr": "node_memory_MemTotal_bytes{instance=~\"10.1.71.130:.*\"}",
+              "legendFormat": "Total"
+            }
+          ]
+        },
+        {
+          "id": 11,
+          "title": "System — Disk Usage (root)",
+          "type": "gauge",
+          "gridPos": { "x": 16, "y": 16, "w": 8, "h": 7 },
+          "fieldConfig": {
+            "defaults": {
+              "unit": "percentunit",
+              "min": 0, "max": 1,
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  { "color": "green", "value": null },
+                  { "color": "yellow", "value": 0.75 },
+                  { "color": "red", "value": 0.90 }
+                ]
+              }
+            }
+          },
+          "targets": [
+            {
+              "expr": "1 - (node_filesystem_avail_bytes{instance=~\"10.1.71.130:.*\",mountpoint=\"/\"} / node_filesystem_size_bytes{instance=~\"10.1.71.130:.*\",mountpoint=\"/\"})",
+              "legendFormat": "Root disk"
+            }
+          ]
+        }
+      ]
+    }

cluster/applications/monitoring/values.yaml

@@ -212,6 +212,36 @@ prometheus:
       #         array: utilidor
       #         array_type: physical
 
+      # astro-orbiter — LLM inference host (Ryzen 7 5800XT / RTX 3090)
+      # Managed by roles/llm-inference (Phase monitoring). Three targets:
+      # node (system), gpu (nvidia_gpu_exporter), llama-server (inference metrics)
+      - job_name: node-astro-orbiter
+        scrape_interval: 30s
+        static_configs:
+          - targets:
+              - 10.1.71.130:9100
+            labels:
+              hostname: astro-orbiter
+
+      - job_name: gpu-astro-orbiter
+        scrape_interval: 15s
+        static_configs:
+          - targets:
+              - 10.1.71.130:9835
+            labels:
+              hostname: astro-orbiter
+              gpu: rtx3090
+
+      - job_name: llama-server-astro-orbiter
+        scrape_interval: 15s
+        metrics_path: /metrics
+        static_configs:
+          - targets:
+              - 10.1.71.130:8000
+            labels:
+              hostname: astro-orbiter
+              model: bartowski/gemma-2-27b-it-GGUF
+
 # ─── Grafana ──────────────────────────────────────────────────────────────────
 grafana:
   enabled: true

cluster/platform/democratic-csi/values.yaml

@@ -0,0 +1,103 @@
+# ------------------------------------------------------------------------------
+# democratic-csi Helm Values for Pure FlashArray (utilidor)
+#
+# Backend: Pure Storage FlashArray via freenas-iscsi driver
+# Target: utilidor.local.mk-labs.cloud (10.1.71.5)
+# 
+# Note: The freenas-iscsi driver works with Pure API v2+ 
+# Pure FlashArray exposes an API compatible with FreeNAS/TrueNAS
+# ------------------------------------------------------------------------------
+
+csiDriver:
+  name: "org.democratic-csi.iscsi-pure"
+
+storageClasses:
+  - name: pure-iscsi-block
+    defaultClass: false
+    reclaimPolicy: Delete
+    volumeBindingMode: Immediate
+    allowVolumeExpansion: true
+    parameters:
+      fsType: ext4
+    mountOptions: []
+    secrets:
+      provisioner-secret:
+      controller-publish-secret:
+      node-stage-secret:
+      node-publish-secret:
+      controller-expand-secret:
+
+driver:
+  config:
+    driver: freenas-iscsi
+    instance_id: mk-labs-utilidor
+    
+    httpConnection:
+      protocol: https
+      host: utilidor.local.mk-labs.cloud
+      port: 443
+      # Pure API token from pure-exporter secret
+      apiKey: "6157dea0-4c0d-b14b-6e6a-453aa649355d"
+      allowInsecure: true
+      apiVersion: 2
+    
+    zfs:
+      # Pure volumes are created in a pod/volume group context
+      # This maps to Pure's volume naming scheme
+      datasetParentName: csi-volumes
+      detachedSnapshotsDatasetParentName: csi-snapshots
+      datasetEnableQuotas: false
+      datasetEnableReservation: false
+      datasetPermissionsMode: "0777"
+      datasetPermissionsUser: 0
+      datasetPermissionsGroup: 0
+    
+    iscsi:
+      targetPortal: "10.1.71.5:3260"
+      targetPortals: []
+      # Pure FlashArray iSCSI configuration
+      interface: default
+      namePrefix: "csi-"
+      nameSuffix: ""
+      targetGroups:
+        # Pure uses host groups for iSCSI targets
+        # Leave empty to auto-select
+      extentCommentTemplate: "Kubernetes CSI volume {{ parameters.[csi.storage.k8s.io/pvc/namespace] }}/{{ parameters.[csi.storage.k8s.io/pvc/name] }}"
+      extentInsecureTpc: true
+      extentXenCompat: false
+      extentDisablePhysicalBlocksize: true
+      extentBlocksize: 512
+      extentRpm: "SSD"
+      extentAvailThreshold: 0
+
+controller:
+  enabled: true
+  replicaCount: 3
+  strategy: deployment
+  priorityClassName: system-cluster-critical
+  
+  externalAttacher:
+    enabled: true
+  
+  externalProvisioner:
+    enabled: true
+  
+  externalResizer:
+    enabled: true
+  
+  externalSnapshotter:
+    enabled: false
+
+node:
+  enabled: true
+  rbac:
+    enabled: true
+  
+  driver:
+    logLevel: info
+  
+  # Talos requires hostNetwork for iSCSI operations
+  hostNetwork: true
+  hostIPC: true
+  
+  dnsPolicy: ClusterFirstWithHostNet

cluster/platform/ingress-nginx/values.yaml

@@ -41,3 +41,18 @@ controller:
       labelSelector:
         matchLabels:
           app.kubernetes.io/name: ingress-nginx
+
+# ------------------------------------------------------------------------------
+# TCP port forwarding — raw TCP services (non-HTTP)
+# Each entry maps an external port to a namespace/service:port target.
+# The Helm chart automatically:
+#   1. Creates the tcp-services ConfigMap in the ingress-nginx namespace
+#   2. Passes --tcp-services-configmap=ingress-nginx/tcp-services to the controller
+#   3. Adds the port to the nginx-ingress LoadBalancer Service
+#
+# After ArgoCD syncs, add a UniFi port forward:
+#   WAN:10182 → 10.1.71.80:10182 (TCP)
+# Non-standard port for security (default 25565 avoided).
+# ------------------------------------------------------------------------------
+tcp:
+  10182: "minecraft/journey-into-imagination:25565"

fastpass-jungle-cruise.yaml

@@ -0,0 +1,156 @@
+version: v1alpha1
+debug: false
+persist: true
+machine:
+  type: worker
+  token: i1qaz0.1elrf4bh8hyvo09e
+  ca:
+    crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJQekNCOHFBREFnRUNBaEVBcFV2UmZVSVJVTWVoSTN2elNGTXIyakFGQmdNclpYQXdFREVPTUF3R0ExVUUKQ2hNRmRHRnNiM013SGhjTk1qWXdOVEUzTWpBd05qRXpXaGNOTXpZd05URTBNakF3TmpFeldqQVFNUTR3REFZRApWUVFLRXdWMFlXeHZjekFxTUFVR0F5dGxjQU1oQVBzeXFTVUpnSUhvQjMxZWd1OXpGVStPcnpvb1NJOC9FNkkzCmJvOG1GaVIrbzJFd1h6QU9CZ05WSFE4QkFmOEVCQU1DQW9Rd0hRWURWUjBsQkJZd0ZBWUlLd1lCQlFVSEF3RUcKQ0NzR0FRVUZCd01DTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkEvUm02ZHdwaGpCS3gxUQoraitrb1h1ZU9NbStNQVVHQXl0bGNBTkJBQTRwZmtaU0VlQkFpN0ZseDg1a1lDb2Q4aUpScEFBUi9RZkNDZ2hCCjBGMmtPOU5wZVhRb3BRL3V0VnJCdlkrNzJKRFFLRWdGTTc5MncwVkFaQzVmb2drPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
+    key: ""
+  certSANs: []
+  kubelet:
+    image: ghcr.io/siderolabs/kubelet:v1.32.3
+    extraArgs:
+      rotate-server-certificates: "true"
+    defaultRuntimeSeccompProfileEnabled: true
+    nodeIP:
+      validSubnets:
+        - 10.1.71.0/24
+    disableManifestsDirectory: true
+  network:
+    nameservers:
+      - 10.1.71.1
+  install:
+    disk: /dev/sda
+    image: factory.talos.dev/metal-installer/10326733f72d0b39b6750c291fc499359135c5486d4066efad56fcf4f3af6923:v1.13.5
+    wipe: false
+    grubUseUKICmdline: true
+  time:
+    servers:
+      - 10.1.71.21
+  sysctls:
+    kernel.perf_event_paranoid: "1"
+    kernel.unprivileged_bpf_disabled: "0"
+    net.core.bpf_jit_harden: "0"
+    net.ipv4.conf.all.arp_announce: "2"
+    net.ipv4.conf.all.arp_ignore: "1"
+  features:
+    diskQuotaSupport: true
+    kubePrism:
+      enabled: true
+      port: 7445
+    hostDNS:
+      enabled: true
+      forwardKubeDNSToHost: true
+  kernel:
+    modules:
+      - name: iscsi_tcp
+      - name: dm_multipath
+      - name: dm_round_robin
+cluster:
+  id: 6FRjhITdPHrGGI35sp0DwqaZP9INupBQIBVn9AZb6NA=
+  secret: P2qqriHtWSm/8iGQMA4vnanm8Mh1f3nuhEFYH1yBPQM=
+  controlPlane:
+    endpoint: https://10.1.71.65:6443
+  clusterName: fastpass
+  network:
+    dnsDomain: cluster.local
+    podSubnets:
+      - 10.244.0.0/16
+    serviceSubnets:
+      - 10.96.0.0/12
+  token: 9w22i0.383cxk2u6ybraufx
+  ca:
+    crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJpakNDQVMrZ0F3SUJBZ0lRWGZvbjBrR2hZUHNTRXcrOUo3VjdCakFLQmdncWhrak9QUVFEQWpBVk1STXcKRVFZRFZRUUtFd3ByZFdKbGNtNWxkR1Z6TUI0WERUSTJNRFV4TnpJd01EWXhNbG9YRFRNMk1EVXhOREl3TURZeApNbG93RlRFVE1CRUdBMVVFQ2hNS2EzVmlaWEp1WlhSbGN6QlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VICkEwSUFCT1g5aDA5eFFDMGFVY3JRRDVGT2lVVzlHOG5abUhCcWc2dHRCVjhNdXBKSjNNMjdreFBGR2VEM05MQkYKVEFVbnBFRE1Jc2RLY09TZFNudWk0ZjJNY0hDallUQmZNQTRHQTFVZER3RUIvd1FFQXdJQ2hEQWRCZ05WSFNVRQpGakFVQmdnckJnRUZCUWNEQVFZSUt3WUJCUVVIQXdJd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFCkZnUVVtbXp5UGV4Qk1VRWk2V3Y3MXBwK3V6SjRUWDR3Q2dZSUtvWkl6ajBFQXdJRFNRQXdSZ0loQVBpelUyM1EKMDVMV0xNSG5lMWo0SWF5dVRrU1pjeXRiZThBN2Y3VDlkb3J4QWlFQWwzSTJhTDhpSHczd2JSVjREQ3NPczZ3WAo5akxLa0prN09pWnhMNlJLR2RNPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
+    key: ""
+  apiServer:
+    admissionControl:
+      - name: PodSecurity
+        configuration:
+          apiVersion: pod-security.admission.config.k8s.io/v1alpha1
+          defaults:
+            audit: restricted
+            audit-version: latest
+            enforce: privileged
+            enforce-version: latest
+            warn: restricted
+            warn-version: latest
+          exemptions:
+            namespaces:
+              - argocd
+              - cert-manager
+              - ingress-nginx
+              - external-secrets
+            runtimeClasses: []
+            usernames: []
+          kind: PodSecurityConfiguration
+  discovery:
+    enabled: true
+    registries:
+      kubernetes:
+        disabled: true
+      service: {}
+---
+apiVersion: v1alpha1
+kind: HostnameConfig
+auto: "off"
+hostname: jungle-cruise
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: ens18
+addresses:
+  - address: 10.1.71.69/24
+routes:
+  - gateway: 10.1.71.1
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: ens19
+addresses:
+  - address: 10.1.75.69/24
+---
+apiVersion: v1alpha1
+kind: ExtensionServiceConfig
+name: multipathd
+configFiles:
+  - content: |-
+      # Pure Storage FlashArray multipath configuration
+      # Configured via ExtensionServiceConfig for multipathd system service
+      defaults {
+          polling_interval 10
+          path_selector "round-robin 0"
+          path_grouping_policy group_by_prio
+          path_checker tur
+          prio alua
+          failback immediate
+          user_friendly_names no
+          find_multipaths yes
+          fast_io_fail_tmo 10
+          dev_loss_tmo 600
+          no_path_retry 0
+      }
+
+      devices {
+          device {
+              vendor "PURE"
+              product "FlashArray"
+              path_selector "round-robin 0"
+              path_grouping_policy group_by_prio
+              prio alua
+              path_checker tur
+              fast_io_fail_tmo 10
+              user_friendly_names no
+              no_path_retry 0
+              hardware_handler "1 alua"
+              dev_loss_tmo 600
+              failback immediate
+          }
+      }
+
+      blacklist {
+          # Exclude Portworx PXD devices from multipathing
+          devnode "^pxd[0-9]*"
+          devnode "^pxd.*"
+      }
+    mountPath: /etc/multipath.conf

graphify-out/cache/ast/5b5bd53d1192f2c4b80b3d2c6cccb5a5d155de2ad8d781e4cebd2ba03f2db894.json

@@ -0,0 +1 @@
+{"nodes": [{"id": "home_hermes_git_homelab_packer_fedora_42_scripts_cleanup_sh", "label": "cleanup.sh", "file_type": "code", "source_file": "packer/fedora-42/scripts/cleanup.sh", "source_location": "L1", "metadata": {"language": "bash", "kind": "file"}}, {"id": "home_hermes_git_homelab_packer_fedora_42_scripts_cleanup_sh__entry", "label": "cleanup.sh script", "file_type": "code", "source_file": "packer/fedora-42/scripts/cleanup.sh", "source_location": "L1", "metadata": {"language": "bash", "kind": "bash_entrypoint"}}], "edges": [{"source": "home_hermes_git_homelab_packer_fedora_42_scripts_cleanup_sh", "target": "home_hermes_git_homelab_packer_fedora_42_scripts_cleanup_sh__entry", "relation": "contains", "confidence": "EXTRACTED", "source_file": "packer/fedora-42/scripts/cleanup.sh", "source_location": "L1", "weight": 1.0}]}

graphify-out/cache/ast/856a78fcc41f33abb8b8b3e006a22aef01547e332490e54dfcbe85f8543bae7f.json

@@ -0,0 +1 @@
+{"nodes": [{"id": "home_hermes_git_homelab_ansible_scripts_migrate_dns_references_sh", "label": "migrate-dns-references.sh", "file_type": "code", "source_file": "ansible/scripts/migrate-dns-references.sh", "source_location": "L1", "metadata": {"language": "bash", "kind": "file"}}, {"id": "home_hermes_git_homelab_ansible_scripts_migrate_dns_references_sh__entry", "label": "migrate-dns-references.sh script", "file_type": "code", "source_file": "ansible/scripts/migrate-dns-references.sh", "source_location": "L1", "metadata": {"language": "bash", "kind": "bash_entrypoint"}}, {"id": "scripts_migrate_dns_references_print_status", "label": "print_status()", "file_type": "code", "source_file": "ansible/scripts/migrate-dns-references.sh", "source_location": "L15", "metadata": {"language": "bash", "kind": "bash_function"}}], "edges": [{"source": "home_hermes_git_homelab_ansible_scripts_migrate_dns_references_sh", "target": "home_hermes_git_homelab_ansible_scripts_migrate_dns_references_sh__entry", "relation": "contains", "confidence": "EXTRACTED", "source_file": "ansible/scripts/migrate-dns-references.sh", "source_location": "L1", "weight": 1.0}, {"source": "home_hermes_git_homelab_ansible_scripts_migrate_dns_references_sh", "target": "scripts_migrate_dns_references_print_status", "relation": "defines", "confidence": "EXTRACTED", "source_file": "ansible/scripts/migrate-dns-references.sh", "source_location": "L15", "weight": 1.0}, {"source": "home_hermes_git_homelab_ansible_scripts_migrate_dns_references_sh__entry", "target": "scripts_migrate_dns_references_print_status", "relation": "calls", "confidence": "EXTRACTED", "source_file": "ansible/scripts/migrate-dns-references.sh", "source_location": "L34", "weight": 1.0, "context": "call"}]}

graphify-out/cache/ast/c4db6a3b575d0b089164a117fe151ac8bd3c9abf1248e9a62401812ddd5621bc.json

@@ -0,0 +1 @@
+{"nodes": [{"id": "home_hermes_git_homelab_packer_ubuntu_24_04_scripts_cleanup_sh", "label": "cleanup.sh", "file_type": "code", "source_file": "packer/ubuntu-24.04/scripts/cleanup.sh", "source_location": "L1", "metadata": {"language": "bash", "kind": "file"}}, {"id": "home_hermes_git_homelab_packer_ubuntu_24_04_scripts_cleanup_sh__entry", "label": "cleanup.sh script", "file_type": "code", "source_file": "packer/ubuntu-24.04/scripts/cleanup.sh", "source_location": "L1", "metadata": {"language": "bash", "kind": "bash_entrypoint"}}], "edges": [{"source": "home_hermes_git_homelab_packer_ubuntu_24_04_scripts_cleanup_sh", "target": "home_hermes_git_homelab_packer_ubuntu_24_04_scripts_cleanup_sh__entry", "relation": "contains", "confidence": "EXTRACTED", "source_file": "packer/ubuntu-24.04/scripts/cleanup.sh", "source_location": "L1", "weight": 1.0}]}

multipath-patch.yaml

@@ -0,0 +1,46 @@
+---
+apiVersion: v1alpha1
+kind: ExtensionServiceConfig
+name: multipathd
+configFiles:
+  - content: |-
+      # Pure Storage FlashArray multipath configuration
+      # Configured via ExtensionServiceConfig for multipathd system service
+      defaults {
+          polling_interval 10
+          path_selector "round-robin 0"
+          path_grouping_policy group_by_prio
+          path_checker tur
+          prio alua
+          failback immediate
+          user_friendly_names no
+          find_multipaths yes
+          fast_io_fail_tmo 10
+          dev_loss_tmo 600
+          no_path_retry 0
+      }
+
+      devices {
+          device {
+              vendor "PURE"
+              product "FlashArray"
+              path_selector "round-robin 0"
+              path_grouping_policy group_by_prio
+              prio alua
+              path_checker tur
+              fast_io_fail_tmo 10
+              user_friendly_names no
+              no_path_retry 0
+              hardware_handler "1 alua"
+              dev_loss_tmo 600
+              failback immediate
+          }
+      }
+
+      blacklist {
+          # Exclude Portworx PXD devices from multipathing
+          devnode "^pxd[0-9]*"
+          devnode "^pxd.*"
+      }
+    mountPath: /etc/multipath.conf
+    

talos/talhelper/clusterconfig-old/.gitignore

@@ -0,0 +1,7 @@
+fastpass-space-mountain.yaml
+fastpass-big-thunder-mountain.yaml
+fastpass-splash-mountain.yaml
+fastpass-jungle-cruise.yaml
+fastpass-haunted-mansion.yaml
+fastpass-peter-pans-flight.yaml
+talosconfig

talos/talhelper/iscsi-multipath-init.yaml

@@ -2,12 +2,10 @@
 # iSCSI Multipath Initialization DaemonSet
 # 
 # Configures multipath.conf for Pure Storage FlashArray on Talos worker nodes.
-# Runs as a DaemonSet with an initContainer that writes configuration, then 
+# Runs as a DaemonSet with an initContainer that writes configuration.
-# sleeps indefinitely to mark the node as configured.
 #
-# This approach avoids Talos boot-time file writing issues while ensuring
+# IMPORTANT: On Talos, /etc is an overlayfs with writable upper layer in /system/etc
-# multipath is properly configured before Portworx CSI driver starts using
+# We write directly to /system/etc which then appears in /etc
-# iSCSI volumes.
 
 apiVersion: apps/v1
 kind: DaemonSet
@@ -43,6 +41,9 @@ spec:
         image: alpine:3.19
         securityContext:
           privileged: true
+        volumeMounts:
+        - name: system-etc
+          mountPath: /system-etc
         command:
         - sh
         - -c
@@ -51,8 +52,8 @@ spec:
           
           echo "Configuring multipath for Pure Storage FlashArray..."
           
-          # Write multipath.conf
+          # Write to /system/etc 
-          cat > /host/etc/multipath.conf <<'MPCONF'
+          cat > /system-etc/multipath.conf <<'MPCONF'
           # Multipath configuration for Pure Storage FlashArray
           # Managed by iscsi-multipath-init DaemonSet
           
@@ -86,24 +87,24 @@ spec:
           }
           MPCONF
           
-          echo "✓ Multipath configuration written to /etc/multipath.conf"
+          echo "✓ Multipath configuration written to /system/etc/multipath.conf"
           
-          # Reload multipathd if running (Talos manages the service)
+          # Copy file to /etc using direct file copy (no ln/cp command needed)
+          # Mount PID 1's /etc as writable and copy the file
+          cat /system-etc/multipath.conf > /proc/1/root/etc/multipath.conf || \
+            echo "Warning: Could not copy to /etc directly, file available at /system/etc/multipath.conf"
+          
+          echo "✓ Configuration deployed to /etc/multipath.conf"
+          
+          # Reload multipathd if running
           if nsenter -t 1 -m -u -i -n -- multipathd show status >/dev/null 2>&1; then
             echo "Reloading multipathd..."
             nsenter -t 1 -m -u -i -n -- multipathd reconfigure || true
           fi
           
-          echo "✓ Multipath configuration complete"
+          echo "✓ Configuration complete"
-          echo "Current multipath devices:"
-          nsenter -t 1 -m -u -i -n -- multipath -ll || echo "  (no multipath devices yet)"
-          
-        volumeMounts:
-        - name: host-etc
-          mountPath: /host/etc
       
       containers:
-      # Pause container keeps the pod running to indicate configuration is applied
       - name: pause
         image: registry.k8s.io/pause:3.9
         resources:
@@ -115,7 +116,7 @@ spec:
             memory: 16Mi
       
       volumes:
-      - name: host-etc
+      - name: system-etc
         hostPath:
-          path: /etc
+          path: /system/etc
-          type: Directory
+          type: DirectoryOrCreate

talos/talhelper/multipathd-extension-patch.yaml

@@ -0,0 +1,28 @@
+apiVersion: v1alpha1
+kind: ExtensionServiceConfig
+name: multipathd
+configFiles:
+  - content: |
+        # Your multipathd configuration content here
+        defaults {
+            user_friendly_names yes
+            find_multipaths yes
+        }
+        blacklist {                                                                                                                                 
+          devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"                                                                                     
+          devnode "^(hd|sda|sd[a-z])[0-9]*"   # Adjust to blacklist local disks                                                                   
+          devnode "^cciss!.*"                                                                                                                     
+        }                                                                                                                                           
+        devices {                                                                                                                                   
+          device {                                                                                                                                
+              vendor                  "PURE"                                                                                                      
+              product                 "FlashArray"                                                                                                
+              path_grouping_policy    multibus                                                                                                    
+              path_selector           "queue-length 0"                                                                                            
+              path_checker            tur                                                                                                         
+              no_path_retry           0                                                                                                           
+              rr_min_io               1                                                                                                           
+              dev_loss_tmo            30                                                                                                          
+          }
+        }
+    mountPath: /etc/multipath.conf

talos/talhelper/px-csi-talos-patch-v2.yaml

@@ -0,0 +1,118 @@
+---                                                                                                                                         
+# Patch for px-pure-csi-node DaemonSet to work on Talos Linux                                                                               
+# This version DISABLES multipath to avoid multipath.conf issues on Talos
+# Apply with: kubectl patch daemonset px-pure-csi-node -n portworx --patch-file px-csi-talos-patch-v2.yaml
+apiVersion: apps/v1                                                                                                                        
+kind: DaemonSet  
+metadata:
+  name: px-pure-csi-node
+  namespace: portworx
+spec:
+  template:
+    spec:
+      containers:
+      - name: node-plugin
+        env:
+        # CRITICAL: Disable multipath to avoid /etc/multipath.conf requirement
+        - name: DISABLE_MULTIPATH
+          value: "true"
+        # Keep existing env vars (these will merge with existing)
+        - name: PX_LOGLEVEL
+          value: INFO
+        - name: LOG_FILE
+          value: /var/log/px-pure-csi/node.log
+        - name: PURE_DISCOVERY_CONF
+          value: /config/pure.json
+        - name: CSI_ENDPOINT
+          value: unix:/csi/csi.sock
+        - name: CSI_NODE_NAME
+          valueFrom:
+            fieldRef:
+              apiVersion: v1
+              fieldPath: spec.nodeName
+        - name: NAMESPACE
+          valueFrom:
+            fieldRef:
+              apiVersion: v1
+              fieldPath: metadata.namespace
+        - name: CLUSTER_UUID
+          valueFrom:
+            configMapKeyRef:
+              key: ClusterUUID
+              name: pure-storage-cluster-cm
+        - name: CLUSTER_NAME
+          valueFrom:
+            configMapKeyRef:
+              key: ClusterName
+              name: pure-storage-cluster-cm
+        - name: NODE_STAGE_CONCURRENCY
+          value: "25"
+        - name: PURE_FLASHARRAY_SAN_TYPE
+          value: ISCSI
+
+        volumeMounts:
+        # Add Talos-specific mounts
+        - name: usr-local-sbin
+          mountPath: /usr/local/sbin
+          mountPropagation: Bidirectional
+        - name: var-lib-iscsi
+          mountPath: /var/lib/iscsi
+          mountPropagation: Bidirectional
+        # Fix existing mounts (keep originals, add propagation)
+        - name: log-dir
+          mountPath: /var/log/px-pure-csi
+        - name: sys
+          mountPath: /sys
+        - name: kubelet-dir
+          mountPath: /var/lib/kubelet
+          mountPropagation: Bidirectional
+        - name: kubelet-dir
+          mountPath: /csi
+          subPath: plugins/pxd.portworx.com
+        - name: device-dir
+          mountPath: /dev
+        - name: iscsi
+          mountPath: /etc/iscsi
+        - name: host-root
+          mountPath: /host
+          mountPropagation: Bidirectional
+
+      volumes:
+      # Add Talos-specific volumes
+      - name: usr-local-sbin
+        hostPath:
+          path: /usr/local/sbin
+          type: Directory
+      - name: var-lib-iscsi
+        hostPath:
+          path: /var/lib/iscsi
+          type: DirectoryOrCreate
+      - name: host-root
+        hostPath:
+          path: /
+          type: Directory
+      # Keep existing volumes
+      - name: kubelet-dir
+        hostPath:
+          path: /var/lib/kubelet
+      - name: log-dir
+        hostPath:
+          path: /var/log/px-pure-csi
+      - name: device-dir
+        hostPath:
+          path: /dev
+          type: Directory
+      - name: probe-dir
+        emptyDir: {}
+      - name: iscsi
+        hostPath:
+          path: /etc/iscsi
+      - name: pure-config
+        secret:
+          secretName: px-pure-secret
+          items:
+          - key: pure.json
+            path: pure.json
+      - name: sys
+        hostPath:
+          path: /sys

talos/talhelper/px-csi-talos-patch.yaml

@@ -0,0 +1,44 @@
+---
+      # Patch for px-pure-csi-node DaemonSet to work on Talos Linux
+      # Apply with: kubectl patch daemonset px-pure-csi-node -n portworx --patch-file px-csi-talos-patch.yaml
+      apiVersion: apps/v1
+      kind: DaemonSet
+      metadata:
+        name: px-pure-csi-node
+        namespace: portworx
+      spec:
+        template:
+          spec:
+            containers:
+            - name: node-plugin
+              volumeMounts:
+              # Add Talos-specific mounts
+              - name: usr-local-sbin
+                mountPath: /usr/local/sbin
+                mountPropagation: Bidirectional
+              - name: var-lib-iscsi
+                mountPath: /var/lib/iscsi
+                mountPropagation: Bidirectional
+              # Fix existing kubelet-dir mount propagation
+              - name: kubelet-dir
+                mountPath: /var/lib/kubelet
+                mountPropagation: Bidirectional
+              # Mount host root for nsenter access
+              - name: host-root
+                mountPath: /host
+                mountPropagation: Bidirectional
+
+            volumes:
+            # Add Talos-specific volumes
+            - name: usr-local-sbin
+              hostPath:
+                path: /usr/local/sbin
+                type: Directory
+            - name: var-lib-iscsi
+              hostPath:
+                path: /var/lib/iscsi
+                type: DirectoryOrCreate
+            - name: host-root
+              hostPath:
+                path: /
+                type: Directory

talos/talhelper/talconfig.yaml

@@ -8,9 +8,9 @@
 #   talhelper gencommand apply --extra-flags="--insecure" | bash
 #   talhelper gencommand bootstrap | bash
 # ─────────────────────────────────────────────────────────────────────────────
-
+---
 clusterName: fastpass
-talosVersion: v1.13.2
+talosVersion: v1.13.5
 kubernetesVersion: v1.32.3
 
 # VIP floats across control plane nodes — all kubectl/talosctl traffic lands here
@@ -120,6 +120,7 @@ worker:
           - siderolabs/qemu-guest-agent
           - siderolabs/util-linux-tools
           - siderolabs/iscsi-tools
+          - siderolabs/multipath-tools
 
   patches:
     - |-
@@ -142,13 +143,13 @@ worker:
             # Only mount /var/lib/iscsi (NOT /etc/iscsi)
             # The iscsi-tools extension manages /etc/iscsi automatically
             # /var/lib/iscsi contains iSCSI session and node data that must persist
-            - destination: /var/lib/iscsi
+            # - destination: /var/lib/iscsi
-              type: bind
+            #   type: bind
-              source: /var/lib/iscsi
+            #   source: /var/lib/iscsi
-              options:
+            #   options:
-                - bind
+            #     - bind
-                - rshared
+            #     - rshared
-                - rw
+            #     - rw 
 
         # ARP tuning for dual-NIC setup
         sysctls:

talos/talhelper/talos-multipath-patch.yaml

@@ -0,0 +1,85 @@
+---
+      # Talos machine config patch for Pure FlashArray multipath support
+      # Add this to your Talos machine config under worker nodes
+      # Apply with: talosctl patch machineconfig -n <worker-ip> --patch @talos-multipath-patch.yaml
+      machine:
+        files:
+          - path: /var/etc/multipath.conf
+            permissions: 0644
+            op: create
+          #   content: |-
+          #     # Pure Storage FlashArray multipath configuration
+          #     defaults {
+          #         polling_interval 10
+          #         path_selector "round-robin 0"
+          #         path_grouping_policy group_by_prio
+          #         path_checker tur
+          #         prio alua
+          #         failback immediate
+          #         user_friendly_names no
+          #         find_multipaths yes
+          #         fast_io_fail_tmo 10
+          #         dev_loss_tmo 600
+          #         no_path_retry 0
+          #     }
+          #     devices {
+          #         device {
+          #             vendor "PURE"
+          #             product "FlashArray"
+          #             path_selector "round-robin 0"
+          #             path_grouping_policy group_by_prio
+          #             prio alua
+          #             path_checker tur
+          #             fast_io_fail_tmo 10
+          #             user_friendly_names no
+          #             no_path_retry 0
+          #             hardware_handler "1 alua"
+          #             dev_loss_tmo 600
+          #             failback immediate
+          #         }
+          #     }
+          #     blacklist {
+          #         devnode "^pxd[0-9]*"
+          #         devnode "^pxd.*"
+          #     }
+
+          # Symlink for PX-CSI to find multipath.conf
+          - path: /etc/multipath.conf
+            permissions: 0644
+            op: create
+          #   content: |-
+          #     # Symlink to persistent location
+          #     # Note: Talos /etc is read-only, this gets regenerated on boot
+          #     defaults {
+          #         polling_interval 10
+          #         path_selector "round-robin 0"
+          #         path_grouping_policy group_by_prio
+          #         path_checker tur
+          #         prio alua
+          #         failback immediate
+          #         user_friendly_names no
+          #         find_multipaths yes
+          #         fast_io_fail_tmo 10
+          #         dev_loss_tmo 600
+          #         no_path_retry 0
+          #     }
+          #     devices {
+          #         device {
+          #             vendor "PURE"
+          #             product "FlashArray"                                                                                                        
+          #             path_selector "round-robin 0"                                                                                               
+          #             path_grouping_policy group_by_prio                                                                                          
+          #             prio alua                                                                                                                   
+          #             path_checker tur                                                                                                            
+          #             fast_io_fail_tmo 10                                                                                                         
+          #             user_friendly_names no                                                                                                      
+          #             no_path_retry 0                                                                                                             
+          #             hardware_handler "1 alua"                                                                                                   
+          #             dev_loss_tmo 600                                                                                                            
+          #             failback immediate                                                                                                          
+          #         }                                                                                                                               
+          #     }                                                                                                                                   
+          #     blacklist {                                                                                                                         
+          #         devnode "^pxd[0-9]*"                                                                                                            
+          #         devnode "^pxd.*"                                                                                                                
+          #     } 

talos/talhelper/verify-multipath-config.sh

@@ -0,0 +1,56 @@
+#!/bin/bash
+# verify-multipath-config.sh
+# Verify that multipath.conf is properly configured on Talos worker nodes
+# Run from city-hall after applying the fixed configuration
+
+set -e
+
+NODE="${1:-10.1.71.69}"  # Default to jungle-cruise
+echo "=== Verifying multipath configuration on node $NODE ==="
+echo
+
+echo "1. Checking ext-multipathd service status..."
+talosctl services --nodes "$NODE" | grep -E "(STATE|multipathd)" || echo "  ⚠️  Service not found"
+echo
+
+echo "2. Checking if /etc/multipath.conf exists in ext-multipathd container..."
+if talosctl exec --nodes "$NODE" --service ext-multipathd -- cat /etc/multipath.conf &>/dev/null; then
+    echo "  ✅ File exists! Contents:"
+    talosctl exec --nodes "$NODE" --service ext-multipathd -- cat /etc/multipath.conf | head -20
+    echo "  ... (truncated)"
+else
+    echo "  ❌ File does NOT exist"
+    echo "  This means the ExtensionServiceConfig is still not being applied."
+    exit 1
+fi
+echo
+
+echo "3. Checking multipath-tools extension..."
+talosctl get extensions --nodes "$NODE" | grep -E "(NAME|multipath)" || echo "  ⚠️  Extension not found"
+echo
+
+echo "4. Checking for multipath devices..."
+echo "  Running: multipath -ll"
+if talosctl exec --nodes "$NODE" -- multipath -ll 2>/dev/null; then
+    echo "  ✅ Multipath devices detected"
+else
+    echo "  ⚠️  No multipath devices (may be normal if iSCSI not connected yet)"
+fi
+echo
+
+echo "5. Checking ext-multipathd logs (last 20 lines)..."
+talosctl logs --nodes "$NODE" --service ext-multipathd --tail 20
+echo
+
+echo "=== Verification complete ==="
+echo
+echo "Expected results:"
+echo "  ✅ ext-multipathd service: Running"
+echo "  ✅ /etc/multipath.conf: Exists with Pure FlashArray config"
+echo "  ✅ multipath-tools extension: Loaded"
+echo "  ✅ multipath -ll: Shows Pure devices (if iSCSI connected)"
+echo
+echo "If /etc/multipath.conf does NOT exist:"
+echo "  - Verify talhelper genconfig was run"
+echo "  - Check clusterconfig/fastpass-jungle-cruise.yaml for 'kind: ExtensionServiceConfig'"
+echo "  - Ensure it's NOT inside quotes (should be structured YAML, not a string)"