rblundon/homelab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: rblundon:d974c75d7cf2b7860136c8b06ba2d5ac9ad1f905
Branches Tags
rblundon:main rblundon:master rblundon:feature/astro-orbiter-ollama rblundon:archive/pre-mk-labs rblundon:monitoring-stack
rblundon:0.1.2
..
compare: rblundon:master
Branches Tags
rblundon:main rblundon:master rblundon:feature/astro-orbiter-ollama rblundon:archive/pre-mk-labs rblundon:monitoring-stack
rblundon:0.1.2

24 Commits
d974c75d7c ... master

Author SHA1 Message Date
Hermes Agent service account
c3755aa29e llm-inference-multimodel: role + day1 playbook (phase 0 discover approved) 2026-08-05 15:53:31 -05:00
Hermes Agent service account
782cbe33d1 llm-inference: size ctx-size/parallel for aux task offload 
Previous ctx-size=8192/parallel=4 gave 2048 tokens/slot, too small for
context compression inputs (observed live rejection at 3826 tokens).

Measured VRAM on astro-orbiter (RTX 3090 24GB): weights ~17GB resident,
~294KiB/token pool-wide for KV cache+buffers at prior sizing.

New: ctx-size=16384, parallel=2 -> 8192 tokens/slot (matches model's
native n_ctx_train max). Projected VRAM ~21.8GB, ~2.7GB headroom.

Applied directly via ansible-playbook (Semaphore currently broken --
fix tracked separately).
 2026-08-05 12:14:11 -05:00
Hermes Agent service account
aff792a061 feat(llm-inference): move astro-orbiter monitoring to GitOps (values.yaml + dashboards.yaml) 
- Prometheus scrape configs for node/gpu/llama-server exporters on
  astro-orbiter now declared in cluster/applications/monitoring/values.yaml
  (additionalScrapeConfigs), applied via ArgoCD sync instead of an
  imperative kubectl secret patch from the Ansible role.
- Grafana dashboard for astro-orbiter LLM inference added as a ConfigMap
  in cluster/applications/monitoring/dashboards.yaml (grafana_dashboard=1
  sidecar label), replacing the role's ad-hoc kubectl apply of a rendered
  Jinja template.
- ansible/roles/llm-inference/tasks/monitoring.yml: removed the kubectl
  get/patch/apply tasks and orphaned grafana-llm-dashboard.json.j2
  template; role now only stands up node_exporter + nvidia_gpu_exporter
  and verifies they're reachable — cluster-facing config lives in Git.
- host_vars/vars.yml + inventory.yml: finalize astro-orbiter as the
  llama.cpp/RTX 3090 host (jarvis user, ssh key), drop stale Ollama/AMD
  vars and ollama_server inventory group superseded by the ATX rebuild.
 2026-08-05 09:43:54 -05:00
Hermes Agent service account
aa8e229e64 fix(llm-inference): switch serve phase from vLLM+bitsandbytes to llama.cpp+GGUF 
bitsandbytes peak RAM ~54GB (bf16 load before quantize) — kills 40GB OptiPlex.
llama.cpp Q4_K_M GGUF loads pre-quantized: peak RAM ~15.5GB, fits cleanly.

Changes:
- serve.yml: build llama.cpp with CUDA, download Q4_K_M GGUF from bartowski,
  disable vllm-serve, deploy llama-server.service
- llama-server.service.j2: OpenAI-compatible server on same port 8000,
  --n-gpu-layers 99 (full GPU offload), --parallel 4, gemma chat template
- defaults: llm_gguf_dir, llm_gguf_path, llm_gpu_layers, llm_parallel_slots
- handlers: restart llama-server, vllm-serve failed_when=false (may not exist)

GGUF: bartowski/gemma-2-27b-it-Q4_K_M.gguf (15.5GB, 24GB VRAM fits w/ ~8GB headroom)
 2026-08-03 12:37:03 -05:00
Hermes Agent service account
22a020e4c7 fix(llm-inference): bitsandbytes int4 OOM — pending switch to llama.cpp+GGUF 
bitsandbytes quantizes on-the-fly: loads full bf16 weights (~54GB RAM peak)
before compressing to int4. Kills the 40GB OptiPlex on torch.compile warmup.

Fix in next commit: switch serve phase to llama.cpp + GGUF Q4_K_M.
Pre-quantized weights load directly — peak RAM ~16GB, no compile overhead.
 2026-08-03 12:35:46 -05:00
Hermes Agent service account
e879cf73d3 fix(llm-inference): gpu_exporter version 1.2.2 → 1.13.1 (correct release tag) 2026-08-03 11:57:42 -05:00
Hermes Agent service account
423891001c feat(llm-inference): Phase 7 — Prometheus monitoring + Grafana dashboard 
- Phase 7 task file: monitoring.yml
  - node_exporter (port 9100) via apt, systemd managed
  - nvidia_gpu_exporter v1.2.2 (port 9835) — GPU util, VRAM, temp, power
  - Patches kube-prometheus additionalScrapeConfigs secret with 3 new jobs:
    node-astro-orbiter, gpu-astro-orbiter, vllm-astro-orbiter
  - Deploys Grafana dashboard ConfigMap via kubectl apply

- Grafana dashboard (11 panels):
  - Row 1: GPU util %, VRAM used, GPU temp gauge
  - Row 2: GPU power draw, vLLM token throughput, request queue depth
  - Row 3: vLLM e2e latency p50/p95/p99, KV cache utilization %
  - Row 4: System CPU %, memory, root disk gauge

- defaults/main.yml: llm_gpu_exporter_version, llm_gpu_exporter_port
- handlers/main.yml: restart nvidia-gpu-exporter
 2026-08-03 11:53:36 -05:00
Hermes Agent service account
dda6b91330 feat(llm-inference): Day 1 playbook for RTX 3090 vLLM stack on astro-orbiter 
- nvidia-driver-595-open (already installed 2026-08-03, idempotent)
- Python venv + vLLM 0.26.0 (already installed, idempotent)
- Gemma 2 27B model download via HuggingFace hub
- systemd vllm-serve.service on port 8000
- Hermes provider integration on carousel-of-progress
- vault_hf_token added to group_vars/all/vault
- ansible.cfg: vault_password_file set to absolute path
- inventory: astro_orbiter group added

Run with: env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference.yml
 2026-08-03 11:51:34 -05:00
Hermes Agent service account
265d3f8fd6 jmri: remove one-shot xpra migration task (idempotency fix) 
Migration from Ubuntu 3.x to upstream 6.x is complete. The explicit
removal task was firing changed on every run. state: latest on the
install task handles upgrades going forward.
 2026-08-01 22:15:11 -05:00
Hermes Agent service account
b61d19cb91 jmri: add udev rule for LCC buffer (Microchip CDC -> jmri-lcc) 2026-08-01 22:09:19 -05:00
Hermes Agent service account
00be18b1f1 jmri: move udev symlinks to /dev/jmri-* (flat, JMRI-enumerable) 
/dev/jmri/ subdirectory is invisible to JMRI's device scanner.
Symlinks must live directly in /dev to appear in the port dropdown.
 2026-08-01 21:58:42 -05:00
Hermes Agent service account
6c7ec507ef jmri: fix NCE udev rule — FTDI FT232 (ttyUSB), not Microchip CDC (ttyACM) 2026-08-01 21:55:43 -05:00
Hermes Agent service account
63b0bc72fe jmri: deploy udev rules for stable /dev/jmri/* symlinks 
Rules were documented but never deployed — /dev/jmri/nce was missing
entirely, only /dev/jmri/loconet existed (created manually).

Adds:
- templates/99-jmri-devices.rules.j2: LocoBuffer-NG -> loconet,
  NCE Power Pro (Microchip CDC) -> nce
- Task to deploy rules + trigger udev settle
- Trigger udev handler (reload-rules alone is insufficient)
 2026-08-01 21:18:06 -05:00
Hermes Agent service account
02af5d26dc jmri: fix xpra remove task idempotency (skip if already from upstream repo) 2026-08-01 21:07:32 -05:00
Hermes Agent service account
3eb38b74bd jmri: add rblundon@laptop SSH key for xpra access 
Adds jmri_ssh_authorized_keys_extra list to support multiple authorized
keys on the jmri account. Deploys rblundon's MacBook key so xpra can
connect via SSH without a password.
 2026-08-01 21:06:07 -05:00
Hermes Agent service account
2b95acb8cc jmri: install xpra from upstream repo (v6.x, replaces Ubuntu v3.x) 
Ubuntu 24.04 packages xpra 3.1.5 which is wire-incompatible with the
v6.x macOS client from xpra.org. Add xpra.org apt repo and install
current stable release so client and server versions match.
 2026-08-01 20:25:01 -05:00
Hermes Agent service account
62e9f13a45 jmri: replace TigerVNC with Xpra for JMRI GUI display 
VNC had window management issues and some dialogs wouldn't open correctly.
Xpra runs in rootless mode — each JMRI window appears natively on the
client without a VNC client or XQuartz required.

Changes:
- Remove tigervnc-standalone-server, jmri-vnc.service, .vnc/ directory
- Install xpra, deploy jmri-xpra.service.j2 (systemd unit)
- Update jmri-gui wrapper: DISPLAY=:100, attach instructions printed on launch
- Update defaults: drop VNC vars, add jmri_xpra_display=100
- Rename handler: Restart jmri-vnc -> Restart jmri-xpra

Connect from macOS/Linux:
  xpra attach ssh://jmri@main-street-station/100
 2026-08-01 20:14:49 -05:00
Hermes Agent service account
4cb87a57ad jmri: add Phase 4 TigerVNC server on :1 (port 5901) — replaces X11 forwarding 2026-07-29 23:17:35 -05:00
Hermes Agent service account
d2eaddfd11 jmri: headless service uses Lake_Country_Railroad profile 2026-07-29 22:33:49 -05:00
Hermes Agent service account
0e741aab38 jmri: gui launcher uses last-session profile instead of --profile flag 2026-07-29 22:32:22 -05:00
Hermes Agent service account
9ebd19ab52 jmri: remove udev symlink phase — JMRI uses /dev/ttyACM* natively 2026-07-29 22:17:29 -05:00
Hermes Agent service account
e47cbf2044 jmri: add jmri_lcrr_branch var; main-street-station tracks clean-profile 2026-07-29 21:50:46 -05:00
Hermes Agent service account
ce632e88b9 jmri: upgrade to 5.16 (R909e15189e) 2026-07-29 21:35:01 -05:00
Hermes Agent service account
11d8796764 jmri: version-aware install/upgrade via marker file 
Replace binary-exists check with .jmri_installed_version marker pattern.
- Reads marker on each run; skips install if version matches
- On version mismatch: stops JMRI, wipes /opt/JMRI, downloads new archive
- Separates build hash into jmri_build_hash var (templated into download URL)
- Config is preserved — lives in git-managed .jmri symlink
To upgrade: bump jmri_version + jmri_build_hash, re-run playbook.
 2026-07-29 21:34:26 -05:00
43 changed files with 2432 additions and 543 deletions
Expand all files Collapse all files

2
ansible/ansible.cfg
View File

@@ -262,7 +262,7 @@ roles_path=./roles
# (path) The vault password file to use. Equivalent to ``--vault-password-file`` or ``--vault-id``.
# If executable, it will be run and the resulting stdout will be used as the password.
;vault_password_file=
vault_password_file=/home/hermes/.vault_pass.txt
# (integer) Sets the default verbosity, equivalent to the number of ``-v`` passed in the command line.
;verbosity=0

2
ansible/group_vars/all/vars
View File

@@ -43,3 +43,5 @@ step_ca_principal_mappings:
# Leviton My Leviton API
leviton_email: "{{ vault_leviton_email }}"
leviton_password: "{{ vault_leviton_password }}"
jmri_vnc_password: "{{ vault_jmri_vnc_password }}"

804
ansible/group_vars/all/vault
View File

@@ -1,441 +1,365 @@
$ANSIBLE_VAULT;1.1;AES256
61346566636664323837633233623331666232363731633634366538333962363966366433636232
3363303535353934306538656332633436303638373933610a326364636563386437386466653335
61326564366131383062363037663965343330663431396331343831326133393233366439646538
3435366135636162300a326161313661633134393666623964323761343139323630353938313864
38393339343231636566386534363831616237343531373835326662376162373264633038353330
33663539646165656261326663306332313932333761383937633265373330346134333532616138
36623861356264643963366138646233343436643066396430653663323039333165306536383565
66346634656131323066353837313836356162346330376232363835653331323736383839356533
36636536313630623730333631383166383032363633643263613335633938616235656166356164
37376563633634666465663332633635653736393135353538346366386630393765316364616463
64343563646233623330633765326537383739323835306566613730323630376339346139613163
39623132633830393635313537616661326536643662323237353337316234663931626261306639
32376434363964363437373165613062333966653331373664383633656130313764626639353164
61363261636362326261383430666235316538383933643365663830336665633164613435303065
65313761623032373332623932623765616263633433663466366633363333356330393161346536
32373966366330363565663632306130656161323234303666313636353166353064636266386239
38363935313564323930396531336566313066386630626431346633323331633436326237633862
33363334346264343537333065336536343264336538643862386336666635383766333235323034
65313764343637343361323064333864343464356336376333613361356530343765626166366337
30326266316135353263323131326664326635616231303831353036333439383633376161383662
34613337376237393537343932326330323564313931376261393163653464316232373238393535
38376637666533363166646336376665363538643363666537393232643762383130366165373037
39636531336664633539653162393536306531313637663039636431353365396466363236376432
66633064316161623739623138386664623165363762663637353435373863656661393362353330
62333163366438343364626432643232396133643065636133626339323935623061323761643331
38393466353833643432643265623431323436346638626661323036303334303239383838396335
33353031373435306266303466366264363737326562326666363066646362623335306465363666
34343862336264626134336338626330396130393833383434363834336133393032613237393031
62633133663433653662373636356231396630626463633432326665326461396530613162373366
34363462353865336166313639663438653839376531623533376335323863383632376433343432
63316239333566356362306133323332353937366237616237326566323363646261376434373939
65623031396139396164653939393162393930326530656135363564363565643133616261666436
39313366383334633036373832323462623161666563303336383131363163373233616435663065
33316266626162363063366539323462346464393932393165643161623964373539393964636139
34343538663533343931613439663436383531393939333239613234386465343430303833353133
64636338343032373230346562363338303166616166666438646262393333313633656162303764
61386164353835383233343535636132643831623062393539336233626235303032663464386266
66366664346364623533613832373631313336336530386162363861303333306563353439333236
65316161383538373733633137633065633362363936373264333034646135366434656534383030
35303838326361313163636432643638373832396165626532623261366332613562383261346661
32386164366264356536653439353763346539643934393466643166333761366436613165623064
38623530316264333939383932313133613730383632663830366365343937313036373439663930
66613034633532303065333935623264636534393638386433346336353765316264346662313337
34613732356265363839343434633563303665313030613535343336363432323366623234623738
38396439376335653731346464363533313161343162653930656137626337623961336232623165
37663863313532626638303937326237353235393866303063313532343164303165636262646434
33316363343265336462626630376665336162336661616436656135366137633834326465303964
62643435393537613836656334616236333131653133653065636361343261643032646163633462
38666465633531383235613732366131366637336638393639346463663830626539653538636165
34633238396233646337383936633438393039306461633631363161656465336238333434633030
37346334396538386437323337393562353463336130363837393033353834386631396164386466
39343338383634663338373536656363623234313339343363636432303937316462376139333138
36386331616639376331393963356138666463306666306436643237303764346330393332626465
62666235656334323231626635333338363736663063656664326139346363323039393565363966
36326662353631646135623832333266366139313662316333376237326433626434336531366666
63323062393030656639353634336339343034316237373932376466653930616465346436383664
31343339663333386533323261393164363039386666323766343337363033366239386266393362
35653333666432376337653332393930383561626431376364326439633331353033613361376333
36633965616237333662613931366533366636393432393863323330383431326635386466646565
31633136373466663161643965396434333437626636626237383630663730663531636139666166
63613964646133666361303430643766333063643732393062333631616132323762623735613566
62613362366536386565636538656138366532643932373163353437636562353737343566613330
34366436323063356433303863383961303130666637366235353264356433313066616262313532
37653165356230313165323330666339376337656332366531383566353536633831363430353764
61383533636263393336313631393766353638353862363666323163356631396234326262626634
64633438333438633235666361373333316265326237393134346539343361353461666664306239
30643939633462303463626237653135333863336530326438313233343232323634633935653134
65626337313266356462323039393564356638353263643566313264616239353036323063623333
32636561373166653739396635633039623464656536653531356365393362343132343631343433
30623438613137333063653061376434336233666633366530333634346136663831643866386132
66383865633531356533316437383933656439353262626138623231613532643834383238323438
63616238653338656536316662306565326233646136366634666330666263356235636231393133
62373665333336336166363936656636373865666132646465656563653464633338623831616436
38663730353764386164653166666136336665383439313135346139373239636137333035323862
62366231323434363864656532323462633061663765616365366335376533336661656536353537
37383466356438343539363935653534336332386130306235353565383933316137313233383832
30663663346339396236633733633464393631376436353833663462316533646534383134356165
33616139663236653661383063323865633339343836643134396364356230323135363731626236
65333338656430376466616137303561356433393733373961313862663430656431343363363635
63663561396438646536346237653562663661353438363237616539393335386338363132623937
35613838616134633334623939333466646131363663653961646138613064366361646235346130
64353130656532333664376635633137333434343965633361333830366232356434346263646232
33336635323062653161346138373636613638623431383538343436323737316335306334366339
37303033646363636637363333323533363037616536303737653064333230313761313366666664
39643861393636356663396431346139646562383035396461323165663665626238653536353133
38346564383761636537333961656538656366333530383831306533623532303635333539616165
65303537663934626561666335643361333339316434373263613637663037613033633930313437
62376233613638636234303264353339653963383633303761316339626434303466376431343664
62616433623431353164383735313835383935333538636432386562616438623062643866333137
65366263656631616334303538363239316632373936663431643239303034373831623765663662
65646133393263373639393664373265376664356434303131396465646161616464623734316436
30396265396536333731363535613035333066616136306538623165306462333561306337386131
38313135663030333439643361633864663061323738313862326131623637663634323131373263
31383730653838656630306138386435643435343162626638656338333131363137363063616432
33326636383530343335613765346634316530376635636233356364373066343835376633356137
61356336616138383536653161363930306364313035313135656634363632353731393934656562
32643362636466656562666466613364333739393736313130656137396437373763323138353362
65353661636537623230623666373534646635626538616431323966326230396637326335326239
30373534616362666435346263646238316637646330336635346437333537643630303062303338
66626237303830386164656531356439656162656633303564316636663639383662306462373430
30616465353461366532356339313734386137623566653262656365643136303065353834646365
66353637306139316132386239306163343564356464316537656237623734326363333137366463
63633864373431353064393561653366653233616233616536373837353065353330643863663361
63653336633135636564373632653561393862633034393266613337343132363034616139366665
32313536643066376638303465616632316533636530626165663561383562623133383834376362
37353633636561326263376366373430656562306633666431316664313832353039366437396436
36333233313765363833316536363132623633336439653734623964373461396433343536316635
33356266663630376662373632323436313431376435646561336665366634333135316334303863
66653461386632303362366535346434616438653135613264666334363363663633666130313734
36376664636363623965363630326236346566393961326332613638636163666562346661363663
37663939306338643134316163333233363164616232353063323134373565616532373636653964
34363237353234393632326438346132353036313232313231663762323131303830613532386262
65663836636266616661373939383937363330623665656139613239396231326631613461396435
30343064366431343735383135623135643165376464343138393131666235323832623034333466
65663932623335323361313930343330636232626661323464623236313262303638353933613630
38646365373164633131653430616338643163383639643134376464336537363861316137353063
62623965633339303838663434306161633733373365383863633462303161643965326433333832
62383731376261303364373931376631626138656535363638376236393133636261626234616137
30333637656139343535636534346239373631633966616462636637306639336630336337343832
66373730373161613935393230623934653530666464636363653066353566333730336261386138
39626537306532633937353761303765393530316236373432373734373965623238303430623632
35396435343065346136616632376533633465346336366137663836323234613735306537373631
39653632303133633032316634323966333762623130613161323064346562346163633361323333
37613735666361366530356363393262393233346239653137303330336161613461666330636530
30366531633538636433323831646433646230376335646535643831316238323233383534626139
37343635383333633034346436376433303634383336316535306236323061623365303432326632
63343036623136363966373363373864623262646462333332616635623831383338666438333962
35326534656537386636363234316332653232626663623261343737636361633338343933373066
39346537653261356366656135303961356162363763623532613631636239633432393266623966
31303434343438643563303463326564393632653938656661366630323636643364333664306461
37616563663630643462653634666166633333393466373462363937336333303963303735666234
65666632343036646536393965643837313234353537663839353032323362663237323338623139
32656663313266393763343133356438303637306633306231323033643535346637386332643934
36653638623334343030363330643237636635633464336561316331383432636438333839336535
37346436393066626634623136366664383734653936393335626636383736343266653863396331
34303234393663653438333466313962313731666165366233313162613437633430333562343163
36663330656639643633633336643436636564366533383235383262313865656538366531303337
66353732626333623031326261306633313666393130373562626564343539313338366364656336
61353837633765646564653936366236303463363531336537333965666437303536643135393062
38343836316265396665386664326532616632346265613334353466643165663463613661333134
33623333343037333133656434376537303462343061393838336438346635363732393337313532
66346366663538373237666565643662383665623130346466316663346162313964653830373630
65663134303466303630353764636639613661613939396438633234636437363937356465326132
38653132613066356633333434366265303235363664636432323566643838636266363762633064
30383332373631326635613266366331656664393934353731653063366537616435396432663731
36376437326237303263393334386263396334316333316132653765343564373935353563353531
63396539613739316539656332623262636136363839353339353331306439323966656530383564
34336362653064656138363861353331306164663834626662646263623561393764303864306565
33366230303663303839313364353430363034316437326135393530366335363065303335636564
31336135333163643864396264383437633736336639333539656661653638333936643030396339
64646336323338376530653735613739333939363366363333623036663861613061366531643034
36303037363361333561313961343730363533383435376362333739363863643365616166613036
64663134666166383663656338323933386362393162636132306331633065323139663162383632
38373264376666363964653138623232313732626665386333643435663531373532346539636636
34303839653338366330613734646636303163643532373030613239313239646630366438386662
36616263373562626435346638383836336635386661396565653836633339353938383164373137
39323364666636356561343139373830383865633466656532336361336363363964353933613133
64343833343330353433376166323531393066356437333132346133643364383562393731333234
39363830653233386566356130356439353535353636656362616534326336616163316233353962
32663361363836646166326663633730333436376230663235393738396639616331303066626662
31643139303835303431303535393536643163393264386463323862303439353536366131373033
31653234323165373562313063636436313538393639663962393139323231303863336165383830
32363535616361666533316132343965343661373762323064333161383463343731356161326333
32623464666562626531343664643933376162646631313166643365666465383035613464643762
30663065616166613531343339383662303438653531333866396561346637623664376266373839
31643233366363646632663635343161303033323061353737666233663561623462663837643236
64613363383631663039613031616133386130363730333361393739356361376165353036326237
34383533316436333465313939306437383337663864643935346336353935303664663866663564
34346362313730306132373936663764633662646466333135343332393765373333333363376135
61303839363236373066643066363135306266333364666132323033393836356531313232363864
61653565323263306263653535306634313662366331643663643966353164326161383530613730
65303738336561363161336330373832393062613130303131633265303037396230313839306661
62373938613034383966336137303431613161343937343461663066343436626563623366646132
32386238353662663337363363393039653330383331636430343632343064383565626338313030
65643333316431616637643162353634323333313165343833636465363461333063306538396636
34343832633565383961656537373562636433323566663830326136353765303230393939356132
62356130396264633930356362653066363163623631613434326464383361643262373936636461
61616631363335353366323865316439396335373966393464626335353631383137376532346262
62343930633430333335623639643431306533396531333666653030643161356339353933643837
33343734373435303235646239343838363439333433333437643162326539326261656330643938
38306131343464613330636636396266353638616230333263323334353862363738623362633034
38626566303336393163366265306564646436623630656533633630653836333035616364623031
35363735343835306132646664613261353230336537646161643463316566343430373461326233
62396563343939643432663062633064383738383834336634323635396461383830363063636432
36666566396535363835303034363732653433623665353437626533633032333838316564323834
30356132613563616333313039666330666565666534393139653065663739363061306365666135
63396262393464653566626635643365356533623761643763663662383832393164326462313837
35616233396165363434633434613031663739343432366263303734343635336136323830333163
33626461646661343564303134366139666639383630663438396139306566643636623431396332
35633232333064383161383933333931636561313035343561643534656635363432333266376536
37373537336239363166306430376132323862333732393164323964363966666262346564613631
39616438623830386266643634663562396336666134323137366233363061613332376661326435
66313565346663343538383235356232346438383835623364353334376565393464393966366538
65376334623438663562623930616266353961393430353361623034306231326530303438316339
34386462666166363534643764633563613934366164303531343366383464643832353739316433
36326364663737353562396261363736393562393964633539363031353432356238333461383730
39386437663434326635373465643234373034303963396434656161633839613731326365613736
36343664386565313535343635376532386331643264613563613735373631626466313964383639
65663761663033333932336665383631636563363561646635383965393039393363346166613861
66393466326634633438343632376538326634353937633562333137373565343339626635376133
36626638393932623837336162343831303436643535643232616432363061346165373965323365
63386632336532656635643863623132393931383838346263636339646332373665396632653361
32346330306362343538353833313833383334366666613436343066636661366238333764353863
37333534366536313464383364303335356232363866363737376665616438666533633066373164
61636362363237313362633865333630386430623431656437646437396131363035616434343136
33353839353765623531323061306465653336623333393837363538653830326561396137613138
64646133626130323234383763623935323433336535343631383535356437333864633330393134
66623530653636623565353830613538616433613538383632303539386362383636376465393036
36373930326430373336666138653334616530653930336364353464613163656131323566393933
62646265393539313432653462616431636232326131313239363337396635336131383534353338
61356535663130336266376431666533353864393766313962623563313131336533656461373836
34653833353730363433663435313039363861386336643937626430326563353230386630356437
63396636393561376330316232343964666337663532353438303964613365303632323263633161
36323033313935636239633232333632373637613439633332666665356331363532373138393861
32326239616338366364613765393665396461633034346436623037666133623565613266376536
37633966626636323762623965653831646434356232626338636332656661333464623261393933
38643631313063336161663830343835366336346632636663353561623835303130313163366635
38633038366234303334336432653534353734346236346638356135633234323364633937666162
62323132393032303139393139633663303138646133626364626533356465323464613338363533
37306362346431306164613637303965373132323633613439396166383134323162313164323431
34383136323636636539353264396439613231653663636638343636626562353861383736306331
31303564313861306666666332303136343330363063313964353331346433626366303935383066
35386261633137613331303630613137376434663836383930366263666262656666356139343164
35616334373931343336636637356339343266613161623561336264623866633834626664663333
66346231386433383163653437353336613731646435626431366663666465313864613837343365
36633962326633313735393238373166636435656661373732386637626264323363623632353631
61613062653164353761313730613936356632306438643431306332326363616464343237386539
63393866393461663135616235643465373838336566336330346662616130666130396364343634
64333335656565623662646136313766623031363037366535386461393430646231353536346332
64313431326164306266666539366264616139303065626536333664623737336134643761333536
66333331303831316137646137316636633161353766393136363038653765376465623561373761
30633331306565393365323735613239393338326635626136363933303862343465623166313539
38393533376630313666643335623165336165633933323036643539623136333938303233353063
30373032633036663963323831646234653537646230643363663238356631303434373265303735
39613263383261306332346264646562636432663238663536653934386630393534653832333332
39316530373861323830303136663462346164386261636666356235633563646264316636363066
39353464346131643239363736383335643132383862303435366230353631613330333133396336
34373532613134613231636636373132643462353865363439356336383266616631363139656265
65333332663139393135643366623266333561313538646639666636303038663962626438663934
35386463386635613236313666306266353064663666333861643932633862336236333435363539
66356130646235636637626339356632376638316231656266303637623965636135636333643065
34396633613536336163306561363332356430616334393133396639663731653863336531313564
35653333356232396232613864313834656533323034396134313061313731613732386366343134
35663836313639663038653036633338336666633066396136663964646137636262313762306261
31313266643663373736393637323539343035636464336563636162343830313831383631313865
66633337613632343363356331663639616231653033663962376131646533313237333536663438
39353034383661396237623438623361323232653135653430313738633835613961323735343365
63376131386339643330303938343765393365656466393965363838343431373763616139393066
37323037396165636630663733333661626462643437616633383132396532396263373936343538
30333732303235376435666565313435356137383538313039626462363066373864303633616638
61363438396634613862303965653937356662373231346539653339333034343961623435393739
62393430633365323034646233393537306266326234643339623339663764343935386430383461
65396434363634363439336137613837316532373931626434323366393564646630313330366162
36653532306439653964373631653061363664663039633134633233346166376261323938363438
65653637313432663962646536376634333735616531653133653061313534623163323738643466
65363538643336353338346632323565313234333463396464663732613332646237653332623866
38666535386337386361333565646633366538333333346336363636333237316462636365383939
61393631356163313933643230343133643539323834363437306465653438396565393532636464
62343263353638343565653065373461386461363736313931313163656138656333383733373532
62356563643663373765336334306466303437343733316431623665626465666633373463366136
63663738663431306532616431373031666462346131623665376235663634336265303035346432
37616536663965643236346264656361613732386239653234613932326462613637666231396531
33316162353038306535386661626464316130343436353230383934346263323738643731373534
36616537626437393530666162336338383563333636656230646235353266373532653431633539
34646531303630373034323932303863623536666261326463303031636666363738363664653738
31376335316239373437363431626630613161663962663934643431303966323237616339393132
33366661383263396632633838393536383335343761306461313638613236326266396363623634
37623165613935373730663139303737373864663035323030643264663932353465666138303264
35326665656538353762643637393664636137313034643637643730313336356464383131343463
30623232363838373338366163326538316465643930656230323831666462353037633131643961
39393238386637313632663336373661636230386632346432366633356139376430646338316235
66393663366232366230326262633638653863396635323539363465383535303433643061633839
61666431353737393939646638653435343039353463616165653630373130383238333336303533
36343331313235376263303633326139613734663264353330643666633234323730366134663262
66643730373761323439636161623135646337383163353361633663376466343839623437613662
62633837653464316430626266646338333530646234623834643739303738366165646235323232
63663433336664376665623734613965626464653832643366353432633437643739396265376531
66636436336135343833356334636531376131316463343162353362356439386139346365333938
35646431306532663936616264626538666638336130313133323830313966353161666236323735
36313666346638363865336331313163346635613735363634366265373363303033616663363563
63353530636566303733343962653138363532326339666230666661393766333863363539373565
32363733613164373762646633373833373466326633383339626431323462306663353235346664
62653361323631623063313435663735333638616366613634363166656431303031353833646563
39313630366430613234643361666365386462346338323330646366366335656136643462663166
63316135306437376139643337646639303261666434316266323465373832333534373730636465
34313832633839396634316561346633643734373763666136363039653136353866343132326430
37303737383030616635343539386337336236376236383961333336363030376333666432613362
61366365316364613639326236333738313936393535646636383261356635303238303938383130
33613264396339646136346631643938356138653535636636393837346430396433313661613337
31323836323162323632343462666433633537623763653566643231656134353235383131663466
62313730366338646132626338623934646364316637343364653236316566666338363339363731
38373832316239613961646432373864306235656339373963643938343664346238373761663830
35663732396430666536313239386465373235393466643930383439613862313337313638386662
34623033653238333064343861633132313762646534646263663566633264393638343730313939
63353438333331383861343135326439303965333834646430393531383762623765666237316165
65346263386636363638663530643633336331623734343231626363353165323562616138373331
33393464333464666235666539333663663061383335343933373066333966346131653361333935
62643363653236626330656561383635656431303231353731396466363232613462323938643730
63356531616364346264653666613564613239646465376238336337383938313035366636396638
65303566343738343731656338663832396330623066646233376436323334633339343162383966
35343164626365386162666162346366656539383334346238336131616338393261306537303534
62343164386462393039393763303566346163646330336264623462353464306232363934343630
38396565613830346634653934653737636262646438643866376138633065303239633538373432
32653562343666313361383331636261313331363639633465383161393063336335343938653933
39663865613236393037333031613964633961326332316662666531353839343937393761636164
63643230623332626266323061386665333862303532313530653534343762383066643736336466
36626564633265303330666331353163396130376331333136386638306233343138613664626537
31626236643637363766636635333465396233663239353865306638303331363937663936613563
30306166393033336164333432626239333338653837336234393536343438326436343434373433
62613165633736353838313439366434323664333832366132663766613236643430313764343032
32636630393032343130623438333166306564366562373764656135653730346237343737346638
66333462383865333733316237663833323036643230616531353630306565353236363734643839
38616532626537613336366637386633653065383135643135363332353731353663323365653763
34623433343135636338303662633966343965303461313136373333396538626166633330343635
61633034356135623639623636626463373339633665366362343064313262333061666637323163
38646266613461393533663061376132363234376164393939663763643962633361663665336239
64356633323730303262383766386635613361316139353331346466643661333331336161636137
63616230396361626133373934613164336335376565656230623030393861383234346138303564
39326134383063393765336461306237653839336531363538383666313832353732663536343134
38663535316333656164396562303036653137653366393462663238613632383230626130346266
33653136383237303139316133636230373866316164336663613666373565643733663739323362
37633734356234626432646566333038303465333563323931396630386430633032626536623765
38623961373063323464353465316432393134633432343465633030363539363865383661646138
64373436336132356535646164336631626639383765386166326532643436393638653431623533
65666639346664613266616364663033363730373637343962393533656463653661613737303837
31633262653361373036383636323766653538346532353038393835303931323264376535373062
36663631316234363063393938633034306465373966333062663162653266386237663137653033
66346435313334653962373935316537376132353962653535366564306132636534306164666431
38363634333635373966353636633836373865666237656336626139353130626332383338313238
62616632616337363664616165336563616161386662306535666634613965613039356165396239
32316364623534643662323334326461386335666530373662353931363131306364626638636430
61363533613638616632396463343239633865643438373837333535336662366432373738393361
31663064623431383336343034653734306635323764613031353862646663626339323764313731
33313132613138633364646532633334643663663339633264343365363231633034666265376161
37313261616230616664396138306264653139643438306662313462386439336434373664313532
61396631663663353334653961363663393330353830306130313065366330653761393436326535
38326163366564373236386431366436633138326561303237613965353266346361303063613666
66333637313061383138373661643964326162633137626163383936386132346636386335353134
37393661666465366331383164366666666431386166396337386630643438643564336139656634
61613232653433336330376433653062333062663034306630653538303337343661326634363463
35626130353561353436386261663335623964633564303365316166306237306634633464653565
61353532613631653037643162666366313339633034646435316338353034366630303736386639
66396236646132366265653536386435333066306461303334646535643835393730643338376633
37303161613837383838323961343834343562666532366136303337363832363765326637653934
66396331333938633034303761313566656533346336393531326235393862623462306434396438
33393539306234626163626664393037666533326133336464323831323132633035313561303034
38343332306532303939363038383934376136333661343435313238356536396537333066613665
31623564306435313466343133643730313262623765346464616263313831313164353162636334
64666133333330646666663336613332373664653932613834383739393762326636303466363134
33393164326232353065396361636539626531656337353363666233343139333263383664666531
63663539303165643564356261346433373438396532396439363330623035353636386531356437
30326231333333393533376366333062386464323565306463336164636434303137386661393139
65336135313462363237343861313165393862393233656561663662333162306539316365393063
62623837353437373236653965646131383262653733303038306134626430623961376366383331
62383366336637626531653532633664616537333032386434306330616439373430383664623638
65336666303263613965376664336262353139396535646662316233343764396661616131346237
62666561373866646564383634313639383636653861666164393263306339616163383965643366
34656538383934313136316231643934663963643737353637633032313930623266323466653032
36636339373034336463313862353638313538316465373132616166323863373832626330383937
30313565373731663663336466363239303734336632396636346139636335646162353166643330
30323530613736613736623238393233316635333437333938313932623834643363616239343032
34626636323461303438376635633637636232386333636139626565303635656630343062353639
39396264386638326666303438623334336163343938336566353034656638323633343837376433
65366232303838643832623332663435636530363638636433303336356531633131356266396334
37313864393761623333653638383766366261353334653066316563386437383432343964616535
30306134383166666630636666316630383030346130353865386539303537363539636161643733
62343530396563393464656566363739313666613762613565386331366365393338313237636231
65326630313630363139626161373730373962656134363733633530376261613239323631623433
61396631333736383731363135383664663632643939376334633966373231363430326663323061
31666134396439346166346430363739626665663133323930363663303164616337316233656333
39303833653366353931343936623362346163363766346461653264353733363439643439336435
32656665623532666465663834363564666634653038616361393665303766326436306533613439
64616334393539643662303036626131623137383164303566623430646337353264343030303935
33313164643439643662393632323433366238306261623335656430396437396535653335343965
31616265363437356263616539343232333539393133323334633730366263386135393864393162
33653161623735363335323838353231326166376365656661373965353433336466623134386230
61653733393737643832303562663933353935626437316538393133613063316566646565323034
32643235303038656362663561666539633432383636333463396162353035323531623133656563
65376362643432643933393063393266663563663636616261663638626238653633303630383062
35363134323131323236623563353035306430343534643962333530383836666565646163346139
65633239396437346432346233643463333831663636323833386338386430373430336536313463
31386131653831623832346564343137326163326261306463666563643839653864353764353861
61663539646431633531653465333336663733663264336565386662626335376231643563356337
65653439613164306639353862656264333733386235333732643730623639333937396236616466
36303332626163323932356338313763363563326139633836343637663832633262323065326435
31643134303737316635336534613038313362396335636164656436623764656537303732643730
35613936303762623765656265663362613736636531633335666230643335633161303738663465
61636165376538313839646133653262343530393837313465343330626362626164366165323831
34396134616138363562313934626239353135336335656331383035393763306166666533643465
34383436356162323631373164373565313563656664663762343630383161646633356434643631
64306239373363343161663030343734366239356237393033643931646434393237363239333631
66343963633266613138353433383338343737636238346433366361643436363233373034656231
37633030623232613030633832306337663336616632333730373639323261646432616531366634
63663837623235383438363533313330663135323661626236333761323165663638623930653631
37633933383562396331326264623866353966633932663961643032383363613333323937383538
34653536313866336530343832336564353036376234383538323732306565636234623832376562
64626166366164626436663564353539323765663365303332373138373034333064626164336439
64333639313832656439613532326330336630613131386463663934653539663763646138343435
37636161623363613039666334626164643464313439376330653837363661393431376632326137
34383134386563343337363037616431366239623461356563613039393936653330303830303534
30373463363265653439633736396664623661646237633537373465316362383965333738333831
37373735636362376265376363386566313663343961623064343363366339373236316434653737
35356436316230626631623130386631636664623762613438313632613134396131336534393931
61663639653166376465323162313064623836623366393464386365313739626636303265343463
30336665636238343930303539313930626439326462343865326531616239323036306230363634
33626531393332393332663137303938613163653637376264663863643163346635616564326337
65393335383534323038656464353963316164396263663533643464623561343337623661663635
39663331633239336432366466623666313036393661623966396630623431626465646437363634
31623536346231373534656262336466336465353261663433363134633030393533326539666661
39623931313339646134363337343233346334366262646638613039363730613433313863396564
64643436653637356563393131666465663332326665373937643764376231336331616361323530
35303038323739316265363863316139363033386538346139636262626633303563353737363739
35656463646364323964366533656339643466323965323661306166393933353237643834373264
66623533623837306238393234666331636566616535323231393364643039323064343433393533
66653333633931303761376162666530386161336534343932633933616139306430326430326566
63656534343362323830643831643836303331643364396566626662333439646531616465623963
64616361396635393339643035616536643435353738383437303832626135316631633166613434
64366335373365653763353436333961663265643739646536343564383062396337333362363235
38326236363564313437343832383961303032643831363563646664373338616363623935656538
35376233326434623339376463346338636162306461656434626135623263613632393135306531
38326434636637366633336337663739633232386531336162333939333132336362363430363965
64316337353637656439343233343362666130663165613335633363353339633537323436323334
61343838383365346466336332643065353836366463613736353231343665623430373232333133
34616261653862353062376231356438363038643238333833353464653138303734373735303135
37663338363737666461646161653434636237313332343838346335313363616134653262633362
33653236343930316231636534613666373061373332653434356530343334313434366137636364
64393035383635646363373732636166383931343362356666643836313065346365396162393237
64623064643831363161653034316162363935356563316536393531643164643466323432323139
63643533373431653435383136663665316131343632383366396366376237346265303634326232
37653331666434316263653531323861386339326130396563663864373364663831336465326665
35383534633433356134636632353333346633383335383039626432323932333433303663383364
34633133323166313032393031346536333863306163333462623063663239666563653365643631
33623238343335333932303836396662323832366139616334653639653838303662313162643335
63396164353038636432653332376536323366343765383034366633666639353936316436613130
33633763396366363565636532613738663963363932373061313635376338626665616131346532
65323564333465616664393965633761653962616137663336333430663433646562333364633361
31613736366462333131356131303063386633386664366166653136333266663161663332353737
31623139663130363735336233353135323166613964653839313137316239393431306239343137
65643431353936616433336263353439383763663833333031653033353464653762656433393436
32376666306637636134
66393233316132396639356564316439343234383066633231646134313361666463656536323732
6630616536646439613533363430306466306233643730350a343364633233333335643833326163
64393933613963313533623733316339396236363663343635346663323366663166363839663837
3331363062653239380a326566623264623837326636383939346430666537613361333638366630
63353062393335316663633739313532366363623739653631366539323435336361353331386230
39366634643964336233353961316630616462663166316266613037623363346335373638656365
38353733396636386133373836346336383231663661346137373164386338623733393566373563
35653933343036633365643535303934326537356136666539316137363433643266346630386439
38613332646238366536333536343031356532656336613530663830613264346339353034323362
66613530626361323535653232313730373463373332313561616631393461353730653464343063
38616338363938346161616636316232313838616463326432353639613837343162646363343232
33323064363139376566343866626364373662393138353666646234373461666163363139313631
64343261326566363265323463663538343034306136326234386664333837333937333136653563
61333531353434633339383661636363363535316366353330313566323133616438373161303135
35353630613037316466353832333033393030636331386438393133366333653832393731366363
62373638303737393162303461646239653865653834613662666636373364633165383062643831
34386232376361323638353361666530366432356331353963303930326535663536373339333062
32396266373430343339636635366434313635313766363863336464633961666332353834626163
61653637316163636465343630353431313863653033643237356434313564366361373435376662
63643737353830663236643862613533623237373531646136383763303766336139303632666235
64623834323966363363663730626437323432623966663537346162656265363562643836633731
65663631633462663764393132326165346639353033633035636432613039336164303538396632
66396264393865306666643636353638613661313230313337383634663839363439656533333932
36633432306131396539386539633063653230363932376264323537396434353364643432653661
32643162363066636432336363323534316436613838646562313538326566666239633234646236
30346534636533623365326564613561363362333364363037646561656635623935653466613565
65646361313436356261643762313339333864356338386136306162386262636464393130303963
38646464316432326431326661343632396235626234366133353461623862316662326432356234
37626366383861373831633639616465663564643866356664623066386535646163336134356534
33366664616232353863626465626364313530353335306565336665663866303736323162393362
63626261653161663664363833313461653034326330653835393737616135646462366665383935
30363639306330636634386433646231363530633061336364313338653632323831393630383934
37316362326338313733646332336263386239626539383330353362616132333161613464313066
34663434326662326233363432306433363666356132383866346336336261636435366332666135
34616231613638363339356333616536643266636363643131653330396162306264303566396461
64363763376365356533636430643866333361363062376237653237663731663934306265646630
33393637656335643366383564373966343265393630333835303731316339373133633462383364
33383435383331303264313334393532373932333334343862326635346135613932356337373034
62316262326331313135376465343336373266663338396533666431616462613932663861646238
62653563623535633738383033326235383666646333653731316233376231623661306462303732
32643064373236613336396233323435393939386530323331336138353364663762356538316562
33376530623664623733386133333433303031373337313366386236376539613964316135343865
33363963366165333238356663663435386439336366646138313034343636653463323938633136
39363966376238306662303265643034306136663661393738633436393432303139313132616534
66323432313635386162333838323136623634653264643438303264636430633232323434666532
32616664663063653735316237643539633133356661333132323238376333356464313262653836
39303566316332663737323437633031353330333365383837636336643763313433313937396531
38363536343438663966663436613132663661613134383431633765383164373762343435316161
62303631646235343063383230343232383336356562303563373933346530393333316634316437
32316330306163396434663031393965663163666537353031613365353437666466333464626238
35313739646535356665323734393965303064306132626261363062363438383164346261393463
30643438623363323161323230306230386332363635386234666639623566643536626637616533
37396136643930633262333331656363376433333234343630306535313262306235663263663362
32653434363035613732363136303363393939323337613661333439393637646262383039386661
62326163323562333339323636363565623664396164383332633666386130613766393138346134
33343338393536316431353439353062663164643634396363353131303038353965393466383030
36656465383938353936346361393963356630666630373236626237303064303062383638373730
35633866646535313432353338623462323235346433653431313031363163393666626432363238
39623361316132626230633336636163623466313666346631656134343762656566353432353264
30353436356237653231363564626134633039363035313232616333336436393638396233626638
32663230396539323761313838313466376165646430346634383332346134653662393161363337
62646161343665383364306665333164666231386531626465373366623761643161656462303733
37653438616233353432626466623163316565353764323762613635333832343634323665356336
35353162326233333836396337356466636131383838313436626336663132346339623261366465
30623261303933396562353331636638376135663330643638643536346261626632626139386535
66653332366361336636666437643165656239613031303638333232303836383132616636633938
31343034643037623731643931316463303639656266323231313666356336333133323135363330
63373365303131353161303630633738353536393631353034666139383435303461316131646138
61333731356538366366613831303565613365633965323235366166313534653965366433656533
62666136313662366638356237343734336333313034396465346632336262306531633535643238
35333831366532386235316565303936616264373337356134643066396531383533353336303131
31393837623564386535323532653733393734393164373235396566333565356237356438313762
32623765326639386262393639376461326163333237313232386138643130643231626466643663
39643061393566353434333136366335393536376234366266376265333234643536633035653933
36316132663539306465343039323935356361373439346437386234386464623962643464643562
61336434613834336161633237383361303930313464613666313834356330343138633735386530
36616233323366323961653965613438346136373738366266316134356266623664313539636235
37313033373466383134346361646562366531333338386330653736626530396238356639303131
38363738396236386461316433316261326435646130383336316234363461393237623633633336
30386630376565646337383738663939663462623232316635346635653830306664653336343033
64316430396664393532313766326437636636626232613036666666656430323136356436333564
36303334303562393832336433343438396430373833623137363736386665343866313064353063
32303634393131326464656535633734386462646339663533666430336265653965333538633866
61623666643839653239373335633735373738363736313665323365613635313766656635613832
33616461643539636165383233636533626230343138663630323731626139393230383464313430
61333438393337316239376435313337313437333931623238616133666138363235386533633437
35356163363231656536353934643539643562343732626630383565623730626533313230656164
35333735666135343364663233626163363930383262363266303265303638396239636361366534
31333863333565356135613232393165353266343632633532343061663331633337343538376265
39316630613439356262396634316361356436336634396337353339616536356336653930613966
34656439653366363562636639346430623561303463356337363830373966366632303337663564
35663632313265323365636238303364366230353039353561616636633664643233343430336237
63373264643935616331616632633065366638363833306337633563653065363464343137623533
36373231363739373335346464623533393336613634333636613937366136326464336332346166
61376263623835646163353134643963663964373732313833346163323138633230393537636664
30366234303334656130336630346130656237306161376566336534653630616439323764373665
61383338326163336164353265326163646165623235626137623237306666333832306461613630
66373331356465346261643466323662393661623433383265376666623932343861323139383531
64633536373362643935633734366235396433333237306166646164363930613862613365303663
38343833336137353634313362666665306666393635663633353934363832343739616331386130
66336561633039326434313833303465366638303961626138333165623331386230616130626639
34613962366230333065633761333335613636363533656461626632343631666563383738623330
30333834346233653938633330663166616331376436356533366461336264643264336139343262
35393665656230663232366133393037643536366234343537326631623332373131323739363638
39653162646366316639313631393631666261623230313538613666393732626438393763646330
36346661313131313630343432616365666633353762623261613039623331396330623939626132
34626333386538326434356432623965666662663437646237373537326534653634346239653634
31373038303639333037613637393862356263323066666630313262366633313932396465633337
66653930303934616236323064613761353935613835356561313334323762633064306661346666
37653262343865386236343634316336386630393739626437333065323433613531393738313432
31376233353463373237653164386363633334366332356538343966663939656165323465333030
39656532363363333432626638626438396539336461326338353732376235316133616666316261
37353063343366376433653961333233306461303133376661303332386230346231383837396133
37323137343066383966343535633363643233663530613566313330336232366638396165373631
30626531363033313833303836366434613736396339643032663066333865306535323739666162
64626133616433653864376662623464343131303938303237316264393765303035663833376464
32663264383236303766323935306463643138396237373338653238633464616238306132633735
31626538653262326533326266336633623532623935383266373533363466313033393235663538
66653038646233303665343634383666343363383238326533366136363838303332323230316662
35383235646638653539633961663036663933306463626335356631646662636230356261363261
65633261353830373865636630353932323937666331353635373736376436333361613330366633
30663939356165393132636131663966373433623063356265353131306532643066306630656363
66636636353262633437663264613266613663656137386231306231646264363661613035343538
37353633643065643236376537336238663137623735613038623766393231643131653436333262
31626463646432613563393665346532386161366435396364663239386236616233356131323536
33633936623762666534633862363466353736386137636363633733623366346337613365636439
66663035313430386464623833646135333062313830396637323961386135363461326539623432
32653865623530313637393561343465636430373162333162646631643235653931333830326266
36376631316165343631326165623838306239623764363262376634663236393933343838376663
39663834306165313330393739363133396436376437643232346336386531356638343063376465
61613037623137306666383231376539656361326132396662613061376134376266633764336266
64336334313335643635303632666431383637306334376462643630646339396435313830313363
33383162316261663035393962306234613865613366353465373035656434366261383133653331
63643235616362663663343330303765363263366130393837613939323264373937333162636639
31643438666338646135663538343231643235646364623761653064633566656663383465626133
31373935646266303565303539376162623132316438623565316537306337636630313861623937
34313832636533623033616139373965303839356530353935643363613464356364343162336466
66376130653162666661313139613530306666633432346639656466653364376435636461626362
32653633303561346233643463373534653434323134353434373839373937626663336464303866
33363264623038313835396231373132396163363662626264346461333539326365326165323066
62333139383334333334353031616430323339623066363232313937323465356266323934313761
66623835653961303830383030643537393130653935313265333062393034336562633535323263
65616237373232336534393834653162363461336262653862666637326266663966356665363036
35383437326465663635303664643236633435303862633965346133376536316233386333313634
33643565326633386565653961646463383866646636303537643436623734393234633938333933
65366164633165623333623362393639656661326332306538663738356364373734316563653038
34646531616662386232613034366332656262343164333531353037363036646262623663666236
38613238666136363431623664633863636365396236666532383930336636353031396232656435
61346238643431653231623861373964383931336535363262373437353532393165316562386134
36363263666135646237383666373833373737396330616163376439663736663937666161313831
63663531656635663339306365656663636633343733636165386230376332616331313638386538
32386466323232363533613334333333346161376430373436373961316564343061326164306138
33616263666262323430303730626266396535626439623364376239346564323730323534323938
33346364393033353865393864326361643734353234613563393138363334383536396535393166
34623163616336653436393639313965353237633566313039303137326234383230323235363234
37626161356166356365366164363863636563316332393638616535376466343537373966643839
32613930643533336264626136626465303339376632323034386161663661376466616233633065
66313739346162363838346663623266383130383736656334323430623463666439386532643630
33666639613830386136363535363830333234653961663739343537306634616531616263623762
64666230373830636238353062666330623061613663376638343763626264363130313464383661
38326530333362616163363735323861376366333665623536383566653837306131623732373639
31316661353332633630326162663738636562336666326637353764323431613666303038373532
62343661336338306561356235396636343130633365303466613637633363613862663233633731
66623530353132666261316637303763363830623734346262333633646238613131346564303734
62336434353432326239333232383833633962313537626430663130393733623162626131656366
64333535623138326239336165666562376663663334323036323539653734333835386331653438
63353861666239396437346361306634613462386335376137333963333838616138633730393865
62353539376136316564666136646639363635663736636439393462633165646632623664383663
31613137306461616361323832393036323933626531363536336261356636303531633239333362
32663134363263383039646162643539663737333861386437326337616362343963373532346238
37346137363933623839373838353939386630303461346438666534616434333031373730393537
30313134643963623564356266656430613430626238613266316335336265613132616562626261
35386435313933626634616463616166646466363939313639646264346464363337656339323366
36366665363739356564363232313762323565323134616134666337336534353464373637373130
63613265366436313131356332316531633732356461383064383031613337343363646432373936
33633339386632653032663837346130623636356464326637303338376132623734333932396232
61333033386265356630316134383066343164613130666664643732643362666561346132656266
31623633333039633837383264363937623435643061393935393762346430396335373864633634
33336136353332663366313334353739303539633364663231636539333132303966383432376262
61356563323232613433653262623663336634626532653465306638316633663564633862666666
30366133616336326661626238653933383164336366333438626235636631336165386664343736
62663961346664656333306435323833366632346366356238653731653937626333653630623334
64326662346138386433333232643262333835326263343239353264373038613634356436396630
38323931643361663238623766323930666130356339363564366661663033303831363138343737
66633535326131396236653261303836613364306537633637323031663166316338323533323731
30326235323066396663613531653061643661336631613835626266626436386662353465383065
64396562343966303362636136616438353661626466636635323961613438646634336563636534
38343566396530643961356434643933636235643561353232643062303232323437666261363061
62636530396466303466653333633930376465366561376363316137323263333561343334383364
39313863643062643766396564363137386231373136346138396162376264653538303464633161
61663363623937356138356430666461623130323466623162653863393736326264393836336637
34663931646566333535666664653237643732316663323230383239393763376266356135326438
35656266353865623663373366373830613361373664346632363031356265313364623866643438
61363866353934636337616239633330623734666138396166313864333939663563636138653930
35653137363033326432373661613434313137623163356134613265393238346438306165313639
36666662393165353565633531663536613037623230373063316639663632643139353235303462
31393263656265656131613164363035343233626433656135353331613532363236616439363731
62666432356363323937666435326638323437346136366131613636653430306131623966356263
39303739306233303862636535633431363630393432613663633836396566653039383735303336
62623331623034636363636661653236386337326666656532343737336262336462613762326531
34303066303834386366636430343161653665343362363038396562626133636135656538306435
36393364333066346238396362663664643236373532336263656233386663323663623137343462
64386337333130316434663564613665666238623132343437656637653035373738313735366630
66383639616166393265616434623463326437313530326130376339313662303836636664366232
32366634633030333130316435616233396231663937343732313066373834326464623139363663
31323931626364303230666162316436653065366137663631376265383063316534343736373261
31613637316235386539343766323439653062633137663730343236343661346162653366656332
32663932313063383561636266373766633535656131386133386135663863396261306530326632
63653936626236316539613262386231616433393064323461626536363831666461316131383837
30646266646266393666396362326238613231303335336532303836363264323233343534636635
66313538643033343262373463363866346566353263303966323933383963363463393761383865
64663932343830643531643466303438343161396133666463353762393737613036646166333265
66376231613232666164663964636134653061633330383863373836306366393838393235656331
35613231306263373230326634623262326333356263353961633836396531633431383163633361
30356534666466653734333437383964346564346165326664633738653338313263633837316531
36613034323433643839333264323864613033313137663131623265643364333664646235666232
33393039313666323266643362323337316465306564303230646561303434666630616137633831
34346439616634343337306636643733316464376631616266376437636439396337306637333432
39306333363035393436316434656436353738303861633933376531383862316466373736323639
35336137373866336631386436646231653366366435363932376434303063613961353261343661
33383638393165336438376662306431333837356435626137356130323836396335636166306662
36386161353739353637353861306666383966323339303262616239633930373633323937356632
65383032613031666665623631613430666662656336663931646533636230303261646530623765
64643939326435643539373564336531623236653731636636346361363064333963376566616530
62326639663632666634326233363635383830643163373938646165656163643864336436373466
36353832306632386230373832333234643638313238626333303963383962343265366137656136
37333261343161633562346638323632616566646162633133663466346535656463393932386135
62653332313066363965386335356430326539316366633537356364666230326237306236393563
35323363633936353034323232353366373566666332323737653237323135646665626139393436
65333762653536656161386532363765336538653763666236343166653933626666633130393033
31643531646633623663313237353333313136663863663430306131316165663765653732663164
36326531626365326330643064336230313466343731376437316563303339336333326636633066
61386135626430616661313236623030316362373338643233326365646531633265626238383830
65346132643537366537626132666165616138656139626132396639376230333262643766386363
39383034663034326165643636613237623234613666333532383733623462303331326238636461
34383139383466356139333934353837613964326538336463643832623062633034613762363061
61346361656363366136353336326433326266336564316366393565626262303637316564356566
39376661383763373436306238393666653561306538333638306233356139346634653363346164
31303835383062376638626266303237323832623735653066353936376339633637333562333561
36646462653834316131323166366661386161646538346464386232306239363030366363633663
65306439616339626635326531636435356134376561303235393337373564373937623636643432
61393535643038626562366331353831663338333838383066323632383633346564396566653330
34386563393832313537623061666466366661333934613766366165366330353835323637643635
38613363396663356564646132613536653033616337386566623662333832383938303138316662
34303339323166383863363831636233323335313565393933636435396666313337663037323432
63333139333165646262666339343736383966346133356138326437386334626461636530336432
61356631366163366561303636333230643732316261376365386463333565623533663966336365
39333365393766306536366231366435363030353263393534653534373064636361333532323735
61313163323831653362356333386638343566356261353534303738613730373632363534666337
62313339613138646361356431616236613435393233343732626263653332663265393934616134
35313165613766643938393839373261633439396661623961353934373130623865353639633038
34353631346433663131653965326337663561613330323562666336656237633163356562323931
62393833663233333538383063303937386365306135343962623333663435663431396438666362
61386266353838653532393233353939363738306634666537313761313835333864633764666262
30333032623766633334383031636633636539336237613235376466356430663938653565626235
64373537656566306136633630366130363630633462656330623633393735386630343437336436
65343635366531646130616534623136636666323139326462306533653532643962656530336633
35616537303932343539336638333730663639396330653761346136363431346536666138336462
66396565613166623934316532383835316137303134363466306163356233356530323231666464
30353933306530323734306564626234343864373964333264353366326265316333343330356532
38363837646635633461653562303264353633343461633339376665616331613733666663353130
65396363613731366234326466323738663563646166653237613364323734616465643764633537
35373865353532383566363632366564353536643739663761303565333138383638653665663664
65643366316461613630366437623736353739356538336237613431306363663234373265623962
34653565373335653563356135313835643266356261623037336536613733323733363933376538
36626134396563623733656534363331626262643339633932373035626134343531623634666463
61623036313334616639633930393562663631653565656136666537393731333430663062643362
63633663396562343965313261373965356163393538666466303661363531393266316462626166
38616536653665366462383064373766396438616665346666376232653031323566313164383164
36643231646439663637333165376439333432383532316661333766363136636236326338386537
33306130353634346136356234363438383865313136393839663066333935623565333730613538
63323831663866363831383930303434333936646564316435303931396362303534386335343330
65383635346662626363626365666166636361633365643735303762393832316436646139303835
63633733656361643233323332613632653837663262306661626438316262653931333061336366
35353939353835333361623261613738383734656132613139393264393038373765343131333330
37663962313566366463623437323965326365623437363038633661313461383634626661666236
32633335323861643037383261393164393933353531636134323765353962633732396230636331
32613865373739366530303538313566346434633933393330346637346136373036306666336164
35373732346334353432616561623031663331346431383235306537386466623339356366663335
36333733626433653465336431666530626665373564336339626163633131353330656437643638
31616563616665343635356231633135663665326131636664373338323736393364353636613762
66636135633766323866376235633535613735613465303239343036663438333331626431623435
61323537386434323638666537643236623632626430666263376534643336613635663762643736
30323237353265613062373265643562373637383337326264653639306263373865333262376665
62646331373931373762303461366163393839633135393964313937616437323865653735383630
32653936336534666565373437666130396265363561333635316461663766346336623865376133
35373665303433326265623531613038636166643130616637653165376263643634376439613765
35363031373630333966656466616235616337306335363132386335613462363664653634633864
61303635653932353730663666386263633662633736313461643932386161313762663761313336
62653665313033656537643936373465633932626166366430643763313030393838393039323230
62633334383938343433306262393536653930653030393033306661666264313630643564333166
35383237633932656331653030363434313534613637373465663264643061303538653666653861
35373936643037333866636131373338363062663035323531626431633362663364396365353139
66383737666437353764333231303662393630643933376161366430376530613365363830373534
38376633333936626430393163323830346166643537326430616236393733653761363235356363
35333131663032383861336262653936376565646662313965303265623763613330653461333835
63613563323135633438383931343731656333303362316533376339376636623037376431336366
61393236383364356162633062666265653534326363363862666539623761623065386537616563
62666561316437303763376635346536666437373361386666643139643737663333323933613661
38326566663932333930616435626133616531306461356466326437623235613233393434626563
34373966633834373430386132353163366465626262353863353335323830393266393562393133
65383632653438646435343333386261653066613663623232373564666465613136353039313036
61646462626433646330396664363938376530376438646262343231393262383733636233636333
35633862336566636439653464613564333162613836343636316334316665383164353131373431
63623030306564346562346237333934616134346536303365396533626262333937396432393830
38313763393463646437666137353835373735646365373934363936346564326362376565353133
36653362333432326133393837316331666663663263396461363239306239363733633137396633
38393865613431653337313665313762653635656531353465623436343132303064303564393066
63616639353962366666616261393766643364333634346630616436376565313236316539633537
66653239636561393433383639646462616433653166613130373134376535633937353366383230
61613335663434333835653236343633633038346335333861356637353965396632393833646635
36653034356234663831333764303338663464316362646339376338393236336161616263363538
63356631613239326161343031643936623366643432663732346438333265666535623664396333
62303239363339626566613439396234303536333333653433393666383635643235376165666234
30336236393962666335353233666463346530323531316438373130303933383465316638646461
62643066376666363236333231386237376466633932313836323163363061313333633434663763
64323365353336333736363436376232653436633739613437343538633632356665656364616637
62313666346436656564663335636635393632353430666236313863613464626434323939383538
65386265343434303632313739353239323565333734656566356164643430613538333234383566
37646431316363316139646435333732313339623666613738663039613239613738393565333330
33376432373933656435303737653762666464363865633831373330393435633332636261336139
34336236373535636165353262323966363164633135613534353661316364616637663465363864
39306163346365643339643937396165666366663339336438373031613937636464383531613962
34373433663533626665656364623634373335313033646165303764396563356235343033383138
62326361353630393938643764616636313461633734646661386536356235656665393864386465
37666262346561656436343036646330363664306135333464663265306165353039396665336664
62346631366330653762646538323565613864383534636532633033643533323736373931643130
32343435333333613734626234363132373734353035326232366264336161383631353133663230
32636533333866343763336439373336356237303636376334333433376630353338333261333037
31666537363666653238383939663464346662636133346561326335346163363061393830616237
36643430626534653331653665316535303139343763663965363164636238366533303038653935
32356432316633373137316237663331336463363431393033323635646564366639346230353363
38346663363039363962323137383366303862356530353238656563306131643236626536656334
38303462306562366532346163323061393437353063326539393466616439346564383036303235
35633731393661323962633631373061303930323638326565636162316436646337383266626561
32326430363031396530396238353862333133363731623736376239626561626165663337373261
39353461343461643238646635633562653865323336366634613264616662323232653861663038
64396330626633303031333334343335393039623135353266383561313231643433393963326637
39313530636361373831306234383166346266656261663830636631333564356536323565336266
38306561376366626236306633613564386166616630613032633163613837313462343662653261
63353437663436303634633336636532646439636465663362346138313665336334313039613631
65326135383831613531323265353831313562346161663265366434623236636635333038366536
33646631663662393331323162343438626666366636613438383665633136326439376166373462
33363864613136643461663436396362643066633437376631623031613366656238396165313832
31313131653263666334393664343239306235373862313339373563643137393633343663613936
61356564636238363136623031336638333566633766636362303938653531306131396665303033
63353362636463636236643464343562383161343432383766396330623764393837613435396162
31303162356437663932663964663239623764666366663061313535346438373334636263653531
34643133356638653031373036343162653135663734623035353033633561366266623566383233
36356434643538643430383532393762333535636639353361353763333363313131646264336332
34373331343930633962623963666365306132356334646636626461316236343839383266363635
65623434336239313330343437646333353362303232346638623161616133636636626236643465
36303636363965363765656533386534633839346363363738386532386531326538643134363132
66613235373362633166343565323766306335336365333439323764623964393263623236623832
34346132383136303038363764333039626234616132386464666633663536656230666133363533
38306665643361666539636666316432623430623939663636343164386438313765633031313534
65393633323837326166343936326263343833646331326464376138633461613532303135393036
65353830373065393038343039323937303634346665393135383639303162396565646232663736
39376434646634353330383933303164653431373433346335666131386165343035303964626665
35383035653631346638326637326235393833623264323030373238646335346332353362393230
61616664613562383639306564376661306665396138613066326631616531623132633966633832
65363637376264336635633132633332373634383864626564623966356464373864393832323738
37616338646633326636323461376137663632376262363738303336616463326238333465343533
31316132386530326539

15
ansible/host_vars/astro-orbiter/vars.yml
View File

@@ -2,11 +2,14 @@
# ------------------------------------------------------------------------------
# FILE: ansible/host_vars/astro_orbiter/vars.yml
# HOST: astro-orbiter (10.1.71.130)
# ROLE: Ollama inference host with AMD RX 5700 GPU passthrough
# ROLE: llama.cpp LLM inference host — Ryzen 7 5800XT / RTX 3090 (ATX rebuild,
# 2026-08-04). Superseded the prior AMD RX 5700 / Ollama config below;
# drive was transplanted into new hardware, not reinstalled.
# ------------------------------------------------------------------------------
ansible_host: 10.1.71.130
ansible_user: wed
ansible_user: jarvis
ansible_ssh_private_key_file: ~/.ssh/id_jarvis
ansible_become: true
# LVM root expansion — xlarge template uses sda3 partition, standard VG/LV names
@@ -15,11 +18,3 @@ common_root_pv: /dev/sda3
common_root_vg: ubuntu-vg
common_root_lv: ubuntu-lv
# Ollama — all defaults apply; explicitly documented here for visibility
ollama_rocm_version: "6.2"
ollama_default_model: "qwen3:8b"
ollama_hsa_override_gfx_version: "10.1.0"
ollama_data_disk: /dev/sdb
ollama_data_vg: ollama-vg
ollama_data_lv: ollama-lv
ollama_data_dir: /var/lib/ollama

5
ansible/host_vars/main-street-station/vars.yml
View File

@@ -1,7 +1,10 @@
---
# main-street-station — JMRI / LCRR server
jmri_profile_id: "My_JMRI_Railroad.3f178885"
jmri_profile_id: "Lake_Country_Railroad.3e8b1d4b"
jmri_lcrr_repo: "ssh://git@gitea.mk-labs.cloud:2221/rblundon/LCRR.git"
jmri_lcrr_branch: "clean-profile"
jmri_leviton_email: "{{ leviton_email }}"
jmri_leviton_password: "{{ leviton_password }}"
jmri_ssh_authorized_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINnSM/9fO8rz/amqkyoGUzUKNNzzmtSXPwOCr1O9zKNO ansible"
jmri_ssh_authorized_keys_extra:
- "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG6HaK4Y21UwPRbAZ986L7I9QnUdyq53114+9kO8X4bL rblundon@laptop"

6
ansible/inventory.yml
View File

@@ -59,12 +59,9 @@ n8n_server:
hosts:
tiki-room:
ollama_server:
astro_orbiter:
hosts:
astro-orbiter:
ansible_host: 10.1.71.130
ansible_user: wed
ansible_become: true
hermes_server:
hosts:
@@ -72,6 +69,7 @@ hermes_server:
ansible_host: 10.1.71.131
ansible_user: wed
ansible_become: true
ansible_ssh_private_key_file: ~/.ssh/ansible
honcho_server:
hosts:

25
ansible/playbooks/day1_deploy_llm_inference.yml Normal file
View File

@@ -0,0 +1,25 @@
---
# ------------------------------------------------------------------------------
# FILE: playbooks/day1_deploy_llm_inference.yml
# DESCRIPTION: Day 1 playbook for astro-orbiter LLM inference stack.
# Deploys vLLM + Gemma 2 27B on RTX 3090 via OCuLink.
#
# Usage:
# cd ~/git/homelab/ansible
# ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference.yml
#
# Phases (added incrementally — safe to re-run):
# 1. Foundation — groups, directories, vault assertion
# 2. Driver — nvidia-driver-595-open (idempotent; already installed)
# 3. vLLM — Python venv + pip install vllm
# 4. Model — HF login, Gemma 2 27B snapshot_download
# 5. Serve — systemd vllm-serve.service, health check
# 6. Integration — Hermes provider config on carousel
# ------------------------------------------------------------------------------
- name: Deploy LLM inference stack on astro-orbiter
hosts: astro_orbiter
gather_facts: true
roles:
- role: llm-inference

31
ansible/playbooks/day1_deploy_llm_inference_multimodel.yml Normal file
View File

@@ -0,0 +1,31 @@
---
# ------------------------------------------------------------------------------
# FILE: playbooks/day1_deploy_llm_inference_multimodel.yml
# DESCRIPTION: Day 1 playbook for the dual-model (aux + tool-calling) rollout
# on astro-orbiter. Builds on roles/llm-inference (CUDA/driver
# already done) — does not replace it.
#
# Usage:
# cd ~/git/homelab/ansible
# ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml
# # or scope to specific phases:
# ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml --tags discover
#
# KNOWN GAP (2026-08-05): Semaphore is currently broken; this is being run
# via direct ansible-playbook as an accepted interim stopgap. Retarget
# through Semaphore once it's repaired.
#
# Phases (see roles/llm-inference-multimodel/README.md for detail):
# 0. discover — read-only; confirm existing Gemma service management
# 1. models — idempotent GGUF downloads (Phi-4-14B, Mistral-Small-24B)
# 2. systemd — deploy both unit files, do NOT auto-start
# 3. firewall — scope ports 8000/8001, non-0.0.0.0 bind
# 4. verify — start both services, smoke test, VRAM check
# ------------------------------------------------------------------------------
- name: Deploy dual-model LLM inference stack on astro-orbiter
hosts: astro_orbiter
gather_facts: true
roles:
- role: llm-inference-multimodel

28
ansible/roles/jmri/defaults/main.yml
View File

@@ -1,8 +1,12 @@
---
# JMRI version to install
jmri_version: "5.10"
# Update both jmri_version AND jmri_build_hash together when upgrading.
# Find the build hash in the release asset filename on:
# https://github.com/JMRI/JMRI/releases
jmri_version: "5.16"
jmri_build_hash: "909e15189e"
jmri_install_dir: /opt/JMRI
jmri_download_url: "https://github.com/JMRI/JMRI/releases/download/v{{ jmri_version }}/JMRI.{{ jmri_version }}+Rca461bd266.tgz"
jmri_download_url: "https://github.com/JMRI/JMRI/releases/download/v{{ jmri_version }}/JMRI.{{ jmri_version }}+R{{ jmri_build_hash }}.tgz"
# Service user
jmri_user: jmri
@@ -14,9 +18,11 @@ jmri_profile_id: ""
# LCRR git repo (set per-host in host_vars; leave blank to skip clone)
jmri_lcrr_repo: ""
jmri_lcrr_branch: "main"
# SSH key for jmri user (for jmri-gui X11 access — set per-host in host_vars)
jmri_ssh_authorized_key: ""
jmri_ssh_authorized_keys_extra: [] # additional keys (e.g. operator laptops)
# SSH key for jmri user → Gitea
jmri_gitea_key: /home/jmri/.ssh/id_ed25519_gitea
@@ -28,6 +34,14 @@ jmri_config_src: ""
jmri_json_port: 12080
jmri_withrottle_port: 12090
# ---------------------------------------------------------------------------
# Phase 4 — Xpra virtual display
# Replaces TigerVNC with rootless Xpra — proper window management,
# persistent sessions, SSH-native attach (no VNC client needed).
# Connect from macOS/Linux: xpra attach ssh://jmri@main-street-station/100
# ---------------------------------------------------------------------------
jmri_xpra_display: "100"
# ---------------------------------------------------------------------------
# Phase 2 — Layout power monitor (Leviton Decora Smart Wi-Fi)
# ---------------------------------------------------------------------------
@@ -39,15 +53,5 @@ jmri_monitor_quiet_start: 1 # hour (24h) to stop polling
jmri_monitor_quiet_end: 10 # hour (24h) to resume polling
jmri_monitor_stop_delay: 30 # seconds of OFF state before stopping JMRI
# ---------------------------------------------------------------------------
# USB device identification (ENV{ID_SERIAL} from udev — the segment of the
# by-id symlink name before the -ifNN suffix)
# ---------------------------------------------------------------------------
# LocoBuffer-NG (LocoNet bridge) → /dev/jmri/loconet
jmri_dev_loconet_serial: "RR-CirKits_LocoBuffer-NG_CDC_ACM_SERIAL_DEVICE_AA5700218A"
# NCE command station → /dev/jmri/nce ← triggers service start/stop
jmri_dev_nce_serial: "Microchip_Technology_Inc._Simple_CDC_Device_Demo"
# FTDI LCC interface → /dev/jmri/lcc
jmri_dev_lcc_serial: "ftdi_usb_serial_converter_ftDYQHZX"

9
ansible/roles/jmri/handlers/main.yml
View File

@@ -3,6 +3,10 @@
ansible.builtin.command: udevadm control --reload-rules
changed_when: false
- name: Trigger udev
ansible.builtin.command: udevadm trigger --subsystem-match=tty
changed_when: false
- name: Reload systemd
ansible.builtin.systemd:
daemon_reload: true
@@ -17,6 +21,11 @@
name: jmri
state: restarted
- name: Restart jmri-xpra
ansible.builtin.systemd:
name: jmri-xpra
state: restarted
- name: Restart sshd
ansible.builtin.systemd:
name: ssh

160
ansible/roles/jmri/tasks/main.yml
View File

@@ -30,6 +30,13 @@
state: present
when: jmri_ssh_authorized_key | length > 0
- name: Deploy extra SSH authorized keys for jmri user
ansible.posix.authorized_key:
user: "{{ jmri_user }}"
key: "{{ item }}"
state: present
loop: "{{ jmri_ssh_authorized_keys_extra }}"
- name: Add JMRI user to dialout group (serial device access)
ansible.builtin.user:
name: "{{ jmri_user }}"
@@ -37,25 +44,20 @@
append: true
# ---------------------------------------------------------------------------
# Phase 1 — Stable udev device symlinks
# Creates /dev/jmri/nce, /dev/jmri/loconet, /dev/jmri/lcc
# Phase 1 — Stable USB device symlinks
# Creates /dev/jmri/loconet and /dev/jmri/nce via udev ID_SERIAL matching.
# ---------------------------------------------------------------------------
- name: Create /dev/jmri symlink directory (static — survives reboots)
ansible.builtin.file:
path: /etc/jmri
state: directory
owner: root
group: root
mode: '0755'
- name: Deploy udev rules for stable JMRI device symlinks
- name: Deploy udev rules for JMRI USB devices
ansible.builtin.template:
src: 99-jmri-devices.rules.j2
dest: /etc/udev/rules.d/99-jmri-devices.rules
owner: root
group: root
mode: '0644'
notify: Reload udev
notify:
- Reload udev
- Trigger udev
# ---------------------------------------------------------------------------
# Phase 2 — LocoNet traffic monitor
@@ -133,7 +135,7 @@
ansible.builtin.git:
repo: "{{ jmri_lcrr_repo }}"
dest: "{{ jmri_home }}/LCRR"
version: main
version: "{{ jmri_lcrr_branch }}"
accept_hostkey: true
key_file: "{{ jmri_gitea_key }}"
update: false
@@ -159,17 +161,55 @@
when: jmri_lcrr_repo | length > 0
notify: Restart jmri
- name: Check if JMRI is already installed at correct version
ansible.builtin.stat:
path: "{{ jmri_install_dir }}/JmriFaceless"
register: jmri_binary
# ---------------------------------------------------------------------------
# JMRI install / upgrade — version-marker pattern
# Writes {{ jmri_install_dir }}/.jmri_installed_version after each install.
# On subsequent runs: read the marker, skip everything if it matches
# jmri_version. If it differs (or is absent), stop JMRI cleanly, wipe the
# old install, download the new archive, extract, and write the new marker.
# To upgrade: bump jmri_version + jmri_build_hash in defaults/main.yml (or
# host_vars) and re-run the playbook.
# ---------------------------------------------------------------------------
- name: Read installed JMRI version marker (if present)
ansible.builtin.slurp:
src: "{{ jmri_install_dir }}/.jmri_installed_version"
register: jmri_version_marker
ignore_errors: true
- name: Determine whether JMRI install/upgrade is needed
ansible.builtin.set_fact:
jmri_needs_install: >-
{{
jmri_version_marker is failed or
(jmri_version_marker.content | b64decode | trim) != jmri_version
}}
- name: Stop JMRI service before upgrade (if running)
ansible.builtin.systemd:
name: jmri
state: stopped
failed_when: false # service may not exist yet on first install
when: jmri_needs_install
- name: Remove existing JMRI install directory (upgrade path)
ansible.builtin.file:
path: "{{ jmri_install_dir }}"
state: absent
when: jmri_needs_install
- name: Remove stale JMRI archive from /tmp (if version changed)
ansible.builtin.file:
path: "/tmp/JMRI-{{ jmri_version }}.tgz"
state: absent
when: jmri_needs_install
- name: Download JMRI release archive
ansible.builtin.get_url:
url: "{{ jmri_download_url }}"
dest: /tmp/JMRI-{{ jmri_version }}.tgz
mode: '0644'
when: not jmri_binary.stat.exists
when: jmri_needs_install
- name: Create JMRI install directory
ansible.builtin.file:
@@ -187,7 +227,16 @@
owner: "{{ jmri_user }}"
group: "{{ jmri_group }}"
extra_opts: ['--strip-components=1']
when: not jmri_binary.stat.exists
when: jmri_needs_install
- name: Write installed version marker
ansible.builtin.copy:
content: "{{ jmri_version }}\n"
dest: "{{ jmri_install_dir }}/.jmri_installed_version"
owner: "{{ jmri_user }}"
group: "{{ jmri_group }}"
mode: '0644'
when: jmri_needs_install
- name: Restore JMRI config from backup
ansible.builtin.copy:
@@ -218,9 +267,9 @@
daemon_reload: true
# ---------------------------------------------------------------------------
# Phase 3 — X11 remote GUI access
# Phase 3 — X11 remote GUI access (retained for fallback; VNC preferred)
# ---------------------------------------------------------------------------
- name: Install xauth (required for SSH X11 forwarding)
- name: Install xauth (required for SSH X11 forwarding fallback)
ansible.builtin.apt:
name: xauth
state: present
@@ -244,6 +293,75 @@
X11UseLocalhost yes
notify: Restart sshd
# ---------------------------------------------------------------------------
# Phase 4 — Xpra virtual display
# Replaces TigerVNC. Rootless mode: each JMRI window appears as a native
# window on the client. Sessions are persistent across disconnects.
# Connect: xpra attach ssh://jmri@main-street-station/{{ jmri_xpra_display }}
# ---------------------------------------------------------------------------
- name: Remove TigerVNC (replaced by Xpra)
ansible.builtin.apt:
name: tigervnc-standalone-server
state: absent
notify: Reload systemd
- name: Disable and stop jmri-vnc service if present
ansible.builtin.systemd:
name: jmri-vnc
enabled: false
state: stopped
failed_when: false
- name: Remove jmri-vnc systemd unit if present
ansible.builtin.file:
path: /etc/systemd/system/jmri-vnc.service
state: absent
notify: Reload systemd
- name: Remove jmri VNC password directory if present
ansible.builtin.file:
path: "{{ jmri_home }}/.vnc"
state: absent
- name: Install xpra.org apt signing key
ansible.builtin.get_url:
url: https://xpra.org/gpg.asc
dest: /usr/share/keyrings/xpra.asc
mode: '0644'
- name: Add xpra.org upstream apt repository
ansible.builtin.copy:
dest: /etc/apt/sources.list.d/xpra.list
owner: root
group: root
mode: '0644'
content: |
deb [arch=amd64 signed-by=/usr/share/keyrings/xpra.asc] https://xpra.org/ noble main
- name: Install Xpra from upstream repo (v6.x)
ansible.builtin.apt:
name: xpra
state: latest
update_cache: true
- name: Deploy jmri-xpra systemd unit
ansible.builtin.template:
src: jmri-xpra.service.j2
dest: /etc/systemd/system/jmri-xpra.service
owner: root
group: root
mode: '0644'
notify:
- Reload systemd
- Restart jmri-xpra
- name: Enable and start jmri-xpra service
ansible.builtin.systemd:
name: jmri-xpra
enabled: true
state: started
daemon_reload: true
- name: Deploy jmri-gui script
ansible.builtin.template:
src: jmri-gui.j2

34
ansible/roles/jmri/templates/99-jmri-devices.rules.j2
View File

@@ -1,25 +1,15 @@
# /etc/udev/rules.d/99-jmri-devices.rules
# Managed by Ansible — do not edit manually.
#
# Creates stable /dev/jmri/* symlinks for all three JMRI serial interfaces.
# Uses ID_SERIAL (full serial string) to match by-id symlink identity,
# so device numbering (ttyACM0, ttyACM1, etc.) doesn't matter after reboots.
# Ownership: root:dialout mode 0660 — jmri user is a member of dialout.
# JMRI USB device symlinks — managed by Ansible, do not edit manually.
# Creates stable /dev/jmri-* symlinks at the top level of /dev so JMRI
# can enumerate them alongside real tty devices.
# NCE command station (Microchip CDC)
# Appears when layout is powered on; disappears when layout powers off.
SUBSYSTEM=="tty", ENV{ID_SERIAL}=="{{ jmri_dev_nce_serial }}", \
SYMLINK+="jmri/nce", \
GROUP="dialout", MODE="0660"
# LocoNet interface — RR-CirKits LocoBuffer-NG (Microchip CDC)
SUBSYSTEM=="tty", ENV{ID_SERIAL}=="RR-CirKits_LocoBuffer-NG_CDC_ACM_SERIAL_DEVICE_AA5700218A", \
SYMLINK+="jmri-loconet", MODE="0666"
# RR-CirKits LocoBuffer-NG (LocoNet bridge)
# USB-powered — always present when server is running.
SUBSYSTEM=="tty", ENV{ID_SERIAL}=="{{ jmri_dev_loconet_serial }}", \
SYMLINK+="jmri/loconet", \
GROUP="dialout", MODE="0660"
# NCE Power Pro command station (FTDI FT232)
SUBSYSTEM=="tty", ENV{ID_SERIAL}=="ftdi_usb_serial_converter_ftDYQHZX", \
SYMLINK+="jmri-nce", MODE="0666"
# FTDI USB serial (LCC interface)
# USB-powered — always present when server is running.
SUBSYSTEM=="tty", ENV{ID_SERIAL}=="{{ jmri_dev_lcc_serial }}", \
SYMLINK+="jmri/lcc", \
GROUP="dialout", MODE="0660"
# LCC buffer (Microchip CDC)
SUBSYSTEM=="tty", ENV{ID_SERIAL}=="Microchip_Technology_Inc._Simple_CDC_Device_Demo", \
SYMLINK+="jmri-lcc", MODE="0666"

63
ansible/roles/jmri/templates/jmri-gui.j2
View File

@@ -1,42 +1,40 @@
#!/bin/bash
# jmri-gui — launch JMRI GUI over SSH -X (X11 forwarding)
# jmri-gui — launch JMRI GUI on Xpra virtual display
# Managed by Ansible — do not edit manually.
#
# Usage (connect as jmri user):
# ssh -X jmri@main-street-station jmri-gui panelpro
# ssh -X jmri@main-street-station jmri-gui decoderpro
# ssh -X jmri@main-street-station jmri-gui status
# jmri-gui panelpro Launch PanelPro on Xpra display
# jmri-gui decoderpro Launch DecoderPro on Xpra display
# jmri-gui status Show service status and attach command
#
# ~/.ssh/config recommended:
# Host main-street-station
# User jmri
# ForwardX11 yes
# Then attach from macOS/Linux:
# xpra attach ssh://jmri@main-street-station/{{ jmri_xpra_display }}
set -euo pipefail
JMRI_DIR="{{ jmri_install_dir }}"
JMRI_PROFILE="{{ jmri_profile_id }}"
JMRI_SERVICE="jmri.service"
MONITOR_SERVICE="jmri-monitor.service"
XPRA_SERVICE="jmri-xpra.service"
DISPLAY=":{{ jmri_xpra_display }}"
export DISPLAY
usage() {
echo "Usage: jmri-gui <panelpro|decoderpro|status>"
exit 1
}
check_display() {
if [ -z "${DISPLAY:-}" ]; then
echo "ERROR: No DISPLAY set. Connect with: ssh -X jmri@main-street-station"
exit 1
fi
}
status() {
echo "=== JMRI daemon ==="
systemctl status "$JMRI_SERVICE" --no-pager -l 2>&1 | head -8
echo ""
echo "=== Xpra display ==="
systemctl status "$XPRA_SERVICE" --no-pager -l 2>&1 | head -5
echo ""
echo "=== Layout monitor ==="
systemctl status "$MONITOR_SERVICE" --no-pager -l 2>&1 | head -5
echo ""
echo "To attach: xpra attach ssh://jmri@$(hostname -f)/{{ jmri_xpra_display }}"
}
launch() {
@@ -49,30 +47,32 @@ launch() {
*) usage ;;
esac
check_display
# Ensure Xpra display is running
if ! systemctl is-active --quiet "$XPRA_SERVICE" 2>/dev/null; then
echo "Starting Xpra display..."
sudo systemctl start "$XPRA_SERVICE"
sleep 2
fi
# Stop the daemon if running
# Stop the JMRI daemon if running (we're taking over the hardware connections)
if systemctl is-active --quiet "$JMRI_SERVICE" 2>/dev/null; then
echo "Stopping JMRI daemon..."
sudo systemctl stop "$JMRI_SERVICE"
fi
echo "Launching $binary (profile: $JMRI_PROFILE)..."
echo "Close the window to return to daemon mode."
echo ""
# SSH sets DISPLAY to localhost:N (TCP) — translate to unix socket form
# so Java's AWT xauth lookup matches the forwarded cookie.
export DISPLAY=$(echo "$DISPLAY" | sed 's/localhost:/:/')
# Force AWT out of headless mode
export JMRI_OPTIONS="-Djava.awt.headless=false"
# Run directly — we are already the jmri user, DISPLAY is set by SSH
"$JMRI_DIR/$binary" --profile="$JMRI_PROFILE"
echo "Launching $binary on Xpra display $DISPLAY..."
echo ""
echo "Attach from your workstation:"
echo " xpra attach ssh://jmri@$(hostname -f)/{{ jmri_xpra_display }}"
echo ""
"$JMRI_DIR/$binary" &
echo "$binary launched. Attach with xpra to see windows."
echo ""
echo "$binary closed."
# Restart daemon if layout switch is still on
if systemctl is-active --quiet "$MONITOR_SERVICE" 2>/dev/null; then
@@ -96,8 +96,7 @@ for perm in perms:
sys.exit(1)
EOF
then
echo "Layout is ON — restarting JMRI daemon."
sudo systemctl start "$JMRI_SERVICE"
echo "Layout is ON — JMRI daemon will restart when GUI is closed."
else
echo "Layout is OFF — JMRI daemon will not restart."
fi

26
ansible/roles/jmri/templates/jmri-xpra.service.j2 Normal file
View File

@@ -0,0 +1,26 @@
[Unit]
Description=Xpra virtual display for JMRI (:{{ jmri_xpra_display }})
After=network.target
# Start before jmri.service so the display is ready when JMRI launches
Before=jmri.service
[Service]
Type=simple
User={{ jmri_user }}
Group={{ jmri_group }}
ExecStart=/usr/bin/xpra start \
:{{ jmri_xpra_display }} \
--daemon=no \
--mdns=no \
--notifications=no \
--systemd-run=no \
--pulseaudio=no \
--speaker=off \
--microphone=off \
--video-encoders=none \
--start-via-proxy=no
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target

132
ansible/roles/llm-inference-multimodel/README.md Normal file
View File

@@ -0,0 +1,132 @@
# llm-inference-multimodel
Deploys **two independent llama-server systemd services** on astro-orbiter's
RTX 3090 (24GB), alongside — not replacing — the existing `llm-inference` role:
| Instance | Port | Model | Quant | ctx | parallel | ~VRAM |
|---|---|---|---|---|---|---|
| `llama-server-aux` | 8000 | Phi-4-14B-Instruct | Q4_K_M | 8192 | 2 | ~10.0GB |
| `llama-server-toolcall` | 8001 | Mistral-Small-24B-Instruct-2501 | Q3_K_M | 4096 | 1 | ~13.2GB |
Combined estimate: **~23.2GB / 24GB** (~0.8GB headroom). See
`/home/hermes/astro-orbiter-multi-model-plan.md` for the full approved design
(VRAM math, model selection rationale, rollback plan, validation harness).
## Relationship to `roles/llm-inference`
This role does **not** replace `llm-inference`. It assumes that role's
prerequisites are already satisfied on the host:
- NVIDIA driver installed
- `/opt/llama.cpp` cloned and built with CUDA (`/opt/llama.cpp/build/bin/llama-server` exists)
- `jarvis` service user + `/home/jarvis` present
The pre-existing single-model Gemma llama-server (however it is currently
run) is **never modified, restarted, or deleted** by this role. It is the
rollback target.
## Phases
Run the whole role, or scope with `--tags`:
```
ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml
# or, once merged into a single play:
ansible-playbook -i inventory.yml <playbook>.yml --tags discover,models,systemd,firewall,verify
```
0. **discover** (`tasks/discover.yml`) — READ-ONLY. Confirms via
`service_facts` + `pgrep` whether the existing Gemma llama-server actually
runs as a systemd unit today, or some ad hoc way (nohup/screen/tmux). Does
**not** assume a unit exists — this was an open unknown in the plan and is
resolved here as a fact-gathering step, not an assumption. Also records
baseline VRAM and current port 8000/8001 listeners for comparison later.
**If this reports no unit found**, stop and read the debug message —
it means plan §6's rollback story ("systemctl start the old unit to
revert") isn't actually available yet, and that should be fixed (codify
the existing process as a systemd unit) before proceeding to Phase 2.
1. **models** (`tasks/models.yml`) — Idempotent GGUF download to
`/opt/models/` with a stat + minimum-size guard (mirrors the pattern in
`roles/llm-inference/tasks/serve.yml` and the `llm-inference-homelab`
skill), so reruns don't re-pull 8.5GB/11.7GB files or mistake a truncated
partial download for complete.
2. **systemd** (`tasks/systemd.yml`) — Templates and deploys both unit files
to `/etc/systemd/system/`. **Deliberately does not start or enable either
service** — units land on disk as a separately reviewable checkpoint.
Two fully independent units (not one unit with two ExecStarts) so either
instance can be restarted/stopped without affecting the other.
3. **firewall** (`tasks/firewall.yml`) — Scopes ports 8000 and 8001 via `ufw`
to `llm_allowed_source_cidr` (default the Hermes LAN subnet), rather than
leaving them open. Both unit templates also bind to
`llm_bind_address` (default `10.1.71.130`, the host's private LAN IP) —
**not `0.0.0.0`** — which is a deliberate change from the pre-existing
Gemma pattern flagged as insecure in the plan.
4. **verify** (`tasks/verify.yml`) — The only phase that actually starts +
enables both services. Waits for `/health` on both ports, smoke-tests
`/v1/models` and a trivial `/v1/chat/completions` call on each, checks
`nvidia-smi` VRAM usage against the plan's design estimate, and greps
`dmesg` for OOM-kill events.
**This smoke test is not the tool-calling validation harness.** See
below.
## Key variables
Defined in `defaults/main.yml` (all overridable via `host_vars`/`group_vars`
or `-e`):
- `llm_service_user` (jarvis), `llm_binary_path`, `llm_models_dir`, `llm_bind_address`, `llm_allowed_source_cidr`
- Aux: `llm_aux_port`, `llm_aux_model_path`, `llm_aux_model_url`, `llm_aux_ctx_size`, `llm_aux_parallel`, `llm_aux_gpu_layers`
- Tool-calling: `llm_toolcall_port`, `llm_toolcall_model_path`, `llm_toolcall_model_url`, `llm_toolcall_ctx_size`, `llm_toolcall_parallel`, `llm_toolcall_gpu_layers`
`vars/main.yml` holds constants not meant to be overridden per-host (HF token
reference, expected-VRAM figures used only for the verify.yml report).
## ⚠️ Tool-calling validation is required before use
Port 8001 (Mistral-Small-24B) **must** pass the manual validation procedure
described in plan §7 before any Claude Code / tool-calling-capable Hermes
profile is pointed at it:
1. A curl-based `tool_calls` emission probe (does it call tools correctly on
known trigger prompts?)
2. A hallucination stress test (does it fabricate `tool_calls` on prompts
that shouldn't trigger any?)
3. A shadow-mode period (run parallel to the existing tool-calling path,
compare outputs, before a hard cutover)
This is **intentionally not automated into this role** — it is a
correctness/safety judgment call, not a repeatable infra check. See
`docs/validation-log.md` in this role directory for the procedure reference
and a place to log results once Ryan runs it.
## Known gap: Semaphore is broken (as of 2026-08-05)
The normal execution/audit path (Semaphore) is currently non-functional.
This role was authored to be run via direct `ansible-playbook` as an accepted
interim stopgap, executed personally by Ryan. **This is a known gap, not the
intended long-term operational path** — once Semaphore is repaired, retarget
execution of this role (and future changes to it) through Semaphore so runs
are audited/logged there again. Flag this in any future work that touches
this role.
## Rollback
The existing Gemma llama-server and its GGUF are untouched by every phase of
this role. To roll back:
1. `systemctl stop llama-server-aux llama-server-toolcall`
2. `systemctl disable llama-server-aux llama-server-toolcall` (optional, if reverting permanently)
3. Confirm the original Gemma service (name determined by `discover.yml`,
commonly `llama-server.service`) is (still) running: `systemctl status llama-server`
4. If it was never running because Phase 0 discovered it wasn't a managed
unit, whatever ad hoc process/command was used before this role's changes
is also unaffected — nothing in this role stopped it.
No files belonging to the existing Gemma deployment (GGUF, unit file, or
otherwise) are ever written to or deleted by this role.

59
ansible/roles/llm-inference-multimodel/defaults/main.yml Normal file
View File

@@ -0,0 +1,59 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference-multimodel/defaults/main.yml
# DESCRIPTION: Overridable defaults for the llm-inference-multimodel role.
# Deploy target: astro-orbiter (10.1.71.130, RTX 3090 24GB).
# Built ALONGSIDE roles/llm-inference (not a replacement) — that
# role's CUDA/build/driver phases are the prerequisite; this role
# assumes /opt/llama.cpp/build/bin/llama-server already exists.
#
# See /home/hermes/astro-orbiter-multi-model-plan.md for the full
# approved design (VRAM math, rationale, rollback story).
# ------------------------------------------------------------------------------
# Shared
llm_service_user: jarvis
llm_binary_path: /opt/llama.cpp/build/bin/llama-server
llm_models_dir: /opt/models
# Bind address — deliberately NOT 0.0.0.0 (see plan §5). Default to the private
# LAN interface so both instances are reachable from Hermes but not the world.
# Override to 127.0.0.1 if even LAN-wide reachability is unwanted and a reverse
# proxy/localhost-only tunnel is used instead.
llm_bind_address: "10.1.71.130"
# Firewall scoping (Phase 3) — subnet/hosts allowed to reach the ports above.
# Override per-environment; default assumes Hermes runs somewhere on this /24.
llm_allowed_source_cidr: "10.1.70.0/24"
# --- Aux / classification instance (port 8000, Phi-4-14B) -------------------
# Text-only instruction model, no tool-calling training — safe offload target
# per the auxiliary-task-offload skill's "no tool_calls emission risk" bar.
llm_aux_port: 8000
llm_aux_model_path: "{{ llm_models_dir }}/phi-4-14b-instruct-Q4_K_M.gguf"
llm_aux_model_url: "https://huggingface.co/bartowski/phi-4-GGUF/resolve/main/phi-4-Q4_K_M.gguf"
llm_aux_model_min_bytes: 8000000000 # guard threshold; complete file ~8.5GB
llm_aux_ctx_size: 8192
llm_aux_parallel: 2
llm_aux_gpu_layers: 99
llm_aux_service_name: llama-server-aux
llm_aux_model_id: phi-4-14b-instruct # served model name for OpenAI-compat API
# --- Tool-calling instance (port 8001, Mistral-Small-24B) --------------------
# Native function-calling support; deployed at Q3_K_M per plan §1 Option B
# to fit VRAM budget. MUST pass the §7 validation harness before any
# Claude-Code-capable profile is pointed at this port.
llm_toolcall_port: 8001
llm_toolcall_model_path: "{{ llm_models_dir }}/mistral-small-24b-instruct-2501-Q3_K_M.gguf"
llm_toolcall_model_url: "https://huggingface.co/bartowski/Mistral-Small-24B-Instruct-2501-GGUF/resolve/main/Mistral-Small-24B-Instruct-2501-Q3_K_M.gguf"
llm_toolcall_model_min_bytes: 11000000000 # guard threshold; complete file ~11.7GB
llm_toolcall_ctx_size: 4096
llm_toolcall_parallel: 1
llm_toolcall_gpu_layers: 99
llm_toolcall_service_name: llama-server-toolcall
llm_toolcall_model_id: mistral-small-24b-instruct-2501
# --- Existing Gemma baseline (rollback target — never modified by this role) -
# Populated by Phase 0 discovery (tasks/discover.yml) if not already known.
# Set here only as a fallback name to search for; discovery is authoritative.
llm_existing_gemma_service_name_guess: llama-server

46
ansible/roles/llm-inference-multimodel/docs/validation-log.md Normal file
View File

@@ -0,0 +1,46 @@
# Tool-Calling Model Validation Log
This file tracks the manual validation procedure required by
`astro-orbiter-multi-model-plan.md` §7 before `llama-server-toolcall` (port
8001, Mistral-Small-24B-Instruct-2501 Q3_K_M) is trusted for any real
tool-calling / Claude Code Hermes profile traffic.
This is **not automated by the role** — `tasks/verify.yml` only confirms the
endpoint is up and can produce a basic completion. The checks below are a
correctness/safety judgment call that a human runs and records here.
## Procedure (plan §7 summary)
1. **`tool_calls` emission probe** — curl a handful of known
tool-triggering prompts (e.g. "what's the weather in Austin right now")
against `POST http://10.1.71.130:8001/v1/chat/completions` with a `tools`
array defined, and confirm the response actually contains a well-formed
`tool_calls` block (correct function name, valid JSON arguments) rather
than a plain-text answer or a malformed call.
2. **Hallucination stress test** — send prompts that should **not** trigger
any tool call (general knowledge questions, casual chat, prompts that
merely mention a tool's name in passing) and confirm the model does
**not** emit a spurious `tool_calls` block. This is the primary risk
flagged in the plan given Mistral-Small's Q3_K_M quantization and its
lineage concerns around over-eager tool invocation.
3. **Shadow mode** — for a bounded period, run this instance in parallel
with whatever tool-calling path is currently in production, comparing
outputs on the same real traffic (or a recorded sample) without letting
this instance's outputs actually drive tool execution. Only cut over
once outputs are consistently correct.
See the `llm-inference-homelab` skill's `scripts/tool-calling-validation.sh`
reference for a starting curl harness shape — adapt prompts/tool schemas to
Mistral-Small's actual expected format (confirm via the GGUF's embedded
chat template / model card) rather than assuming it matches Qwen's.
## Log
| Date | Run by | Probe result | Hallucination test result | Shadow mode outcome | Decision |
|---|---|---|---|---|---|
| _(pending)_ | | | | | Not yet cut over — do not point production tool-calling traffic at :8001 |
Update this table after each validation pass. Do not remove prior rows —
this is the audit trail for "when did we decide this was safe to use."

27
ansible/roles/llm-inference-multimodel/handlers/main.yml Normal file
View File

@@ -0,0 +1,27 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference-multimodel/handlers/main.yml
# DESCRIPTION: Separate restart handlers per instance — NEVER combined, so a
# content change to one unit template never restarts the other
# (plan §2/§6 requirement: independent restart/rollback).
# ------------------------------------------------------------------------------
- name: Reload systemd
ansible.builtin.systemd:
daemon_reload: true
become: true
listen: "reload systemd"
- name: Restart llama-server-aux
ansible.builtin.systemd:
name: "{{ llm_aux_service_name }}"
state: restarted
become: true
listen: "restart llama-server-aux"
- name: Restart llama-server-toolcall
ansible.builtin.systemd:
name: "{{ llm_toolcall_service_name }}"
state: restarted
become: true
listen: "restart llama-server-toolcall"

17
ansible/roles/llm-inference-multimodel/meta/main.yml Normal file
View File

@@ -0,0 +1,17 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference-multimodel/meta/main.yml
# ------------------------------------------------------------------------------
galaxy_info:
role_name: llm_inference_multimodel
author: rblundon
license: MIT
description: >
Deploys two independent llama-server instances on astro-orbiter's RTX 3090:
an aux/classification instance (Phi-4-14B Q4_K_M, port 8000) and a
tool-calling instance (Mistral-Small-24B-Instruct-2501 Q3_K_M, port 8001).
Built alongside roles/llm-inference (not a replacement); assumes that
role's CUDA build/driver work is already done. See
/home/hermes/astro-orbiter-multi-model-plan.md for the full design.
min_ansible_version: "2.15"
dependencies: []

87
ansible/roles/llm-inference-multimodel/tasks/discover.yml Normal file
View File

@@ -0,0 +1,87 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference-multimodel/tasks/discover.yml
# DESCRIPTION: Phase 0 — READ-ONLY fact gathering on how the existing Gemma
# llama-server is actually managed on astro-orbiter TODAY.
#
# Per plan §0/§4: "Service management: unverified — plan requires
# confirming systemd unit exists before touching anything.
# Do not assume." This file performs that confirmation. It makes
# NO changes to the host — no `state: present/started/stopped`,
# no file writes, no service actions. Every task here is either
# a `_facts` module, a `command`/`shell` in check-safe read mode,
# or a `stat`.
#
# Outcomes recorded as facts for later phases/for a human to read
# in the play recap — this file does not branch role behavior
# on the result (that would be over-engineering a role meant to
# run once); it surfaces what's true so a human confirms before
# Phase 2 proceeds.
# ------------------------------------------------------------------------------
- name: Gather service facts (systemd unit inventory)
ansible.builtin.service_facts:
- name: Determine whether a systemd unit matching the existing Gemma service exists
ansible.builtin.set_fact:
llm_existing_gemma_unit_found: "{{ (llm_existing_gemma_service_name_guess + '.service') in ansible_facts.services }}"
- name: Report existing Gemma systemd unit state (if found)
ansible.builtin.debug:
msg: >-
Existing unit '{{ llm_existing_gemma_service_name_guess }}.service' found:
state={{ ansible_facts.services[llm_existing_gemma_service_name_guess + '.service'].state | default('unknown') }},
status={{ ansible_facts.services[llm_existing_gemma_service_name_guess + '.service'].status | default('unknown') }}
when: llm_existing_gemma_unit_found
- name: WARNING — no systemd unit found matching the existing Gemma service
ansible.builtin.debug:
msg: >-
No systemd unit named '{{ llm_existing_gemma_service_name_guess }}.service'
was found via service_facts. This means the current single-model
llama-server is likely run some other way (manual nohup, screen/tmux,
or a differently-named unit). DO NOT PROCEED to Phase 2 assuming a
clean rollback target exists. Before continuing: (1) check for any
running llama-server process via `ansible -m command -a "pgrep -fa
llama-server"`, (2) if found running ad hoc, codify it as a proper
systemd unit FIRST (reusing roles/llm-inference's existing
llama-server.service.j2 pattern) so plan §6's rollback story
("systemctl start llama-server-gemma to fully revert") is real and
not aspirational. This is a human decision point, not something this
role auto-remediates.
when: not llm_existing_gemma_unit_found
- name: Check for any running llama-server process (read-only, no state change)
ansible.builtin.command:
cmd: pgrep -fa llama-server
register: llm_existing_process_check
changed_when: false
failed_when: false # pgrep exits 1 with no matches — not a failure condition here
- name: Report any llama-server process found running outside systemd
ansible.builtin.debug:
msg: "Running llama-server process(es): {{ llm_existing_process_check.stdout_lines }}"
when: llm_existing_process_check.rc == 0
- name: Check current GPU VRAM utilization (baseline, before any changes)
ansible.builtin.command:
cmd: nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader
register: llm_baseline_vram
changed_when: false
failed_when: false
- name: Report baseline VRAM usage
ansible.builtin.debug:
msg: "Baseline GPU VRAM (before this role's changes): {{ llm_baseline_vram.stdout | default('nvidia-smi unavailable') }}"
- name: Check whether ports 8000/8001 are already bound (avoid port collision surprises)
ansible.builtin.command:
cmd: "ss -ltnp"
register: llm_existing_listeners
changed_when: false
failed_when: false
- name: Report current listeners on 8000/8001
ansible.builtin.debug:
msg: "{{ llm_existing_listeners.stdout_lines | select('search', ':(8000|8001)\\s') | list }}"
when: llm_existing_listeners.rc == 0

64
ansible/roles/llm-inference-multimodel/tasks/firewall.yml Normal file
View File

@@ -0,0 +1,64 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference-multimodel/tasks/firewall.yml
# DESCRIPTION: Phase 3 — scope :8001 (new) and reconsider :8000 (existing
# pattern) exposure, per plan §5.
#
# Current baseline pattern (0.0.0.0:8000, no auth) is a
# pre-existing flagged issue — this role does NOT repeat it
# uncritically for the new port, and tightens both:
# 1. Bind address: handled in systemd.yml templates via
# {{ llm_bind_address }} (default 10.1.71.130, NOT 0.0.0.0).
# 2. Firewall: ufw rules scoping both ports to
# {{ llm_allowed_source_cidr }} rather than open LAN-wide.
#
# Idempotent: named rule comments + `state: present` so reruns
# don't duplicate rules (per plan §4 idempotency note).
# ------------------------------------------------------------------------------
- name: Check whether ufw is installed/active
ansible.builtin.command:
cmd: ufw status
register: llm_ufw_status
changed_when: false
failed_when: false
become: true
- name: WARNING — ufw not active, firewall scoping cannot be applied
ansible.builtin.debug:
msg: >-
ufw does not appear to be active on this host (`ufw status` returned:
{{ llm_ufw_status.stdout | default('n/a') }}). Firewall scoping for
ports {{ llm_aux_port }}/{{ llm_toolcall_port }} was skipped. This is a
gap vs plan §5 item 2 — flag to Ryan before relying on bind-address
alone for exposure control.
when: "'Status: active' not in (llm_ufw_status.stdout | default(''))"
- name: Allow aux port ({{ llm_aux_port }}) from the Hermes source subnet
community.general.ufw:
rule: allow
port: "{{ llm_aux_port | string }}"
proto: tcp
src: "{{ llm_allowed_source_cidr }}"
comment: "llm-inference-multimodel: aux (Phi-4) — scoped to Hermes subnet"
become: true
when: "'Status: active' in (llm_ufw_status.stdout | default(''))"
- name: Allow tool-calling port ({{ llm_toolcall_port }}) from the Hermes source subnet
community.general.ufw:
rule: allow
port: "{{ llm_toolcall_port | string }}"
proto: tcp
src: "{{ llm_allowed_source_cidr }}"
comment: "llm-inference-multimodel: toolcall (Mistral-Small) — scoped to Hermes subnet"
become: true
when: "'Status: active' in (llm_ufw_status.stdout | default(''))"
- name: Report firewall scoping applied
ansible.builtin.debug:
msg: >-
Firewall scoping applied for ports {{ llm_aux_port }} and
{{ llm_toolcall_port }}, restricted to source {{ llm_allowed_source_cidr }}.
Reverse-proxy + API-key enforcement (plan §5 item 3) is NOT implemented
by this role — flagged as an optional follow-up phase, not bundled into
this minimum-viable rollout.

35
ansible/roles/llm-inference-multimodel/tasks/main.yml Normal file
View File

@@ -0,0 +1,35 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference-multimodel/tasks/main.yml
# DESCRIPTION: Entry point — imports one task file per phase.
# Phases are additive; re-running the full playbook is always
# safe (idempotent). Use --tags to run a specific phase subset:
# --tags discover,models,systemd,firewall,verify
#
# IMPORTANT: Phase 2 (systemd) deploys but does NOT start either service.
# Phase 4 (verify) is what starts + smoke-tests them. This lets
# Ryan review "systemd units land, nothing running yet" as a
# distinct, revertable checkpoint before anything touches the
# live GPU/VRAM state.
# ------------------------------------------------------------------------------
# Phase 0 — Discover (read-only; confirm how the existing Gemma llama-server
# is actually managed today before assuming a systemd unit exists)
- import_tasks: discover.yml
tags: [discover]
# Phase 1 — Models (idempotent GGUF download, size-check guard)
- import_tasks: models.yml
tags: [models]
# Phase 2 — Systemd (template + deploy both unit files, do NOT auto-start)
- import_tasks: systemd.yml
tags: [systemd]
# Phase 3 — Firewall (scope :8001 and reconsider :8000 exposure)
- import_tasks: firewall.yml
tags: [firewall]
# Phase 4 — Verify (start both services, curl smoke test, nvidia-smi VRAM check)
- import_tasks: verify.yml
tags: [verify]

72
ansible/roles/llm-inference-multimodel/tasks/models.yml Normal file
View File

@@ -0,0 +1,72 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference-multimodel/tasks/models.yml
# DESCRIPTION: Phase 1 — download both GGUFs to {{ llm_models_dir }}.
# Idempotent: reuses the stat + size-threshold guard pattern
# from the llm-inference-homelab skill / roles/llm-inference's
# serve.yml, so reruns don't re-pull 8.5GB / 11.7GB files.
#
# Does NOT touch the existing Gemma GGUF — separate directory
# entries, no overlap, no deletion of anything pre-existing.
# ------------------------------------------------------------------------------
- name: Create models directory
ansible.builtin.file:
path: "{{ llm_models_dir }}"
state: directory
owner: "{{ llm_service_user }}"
group: "{{ llm_service_user }}"
mode: "0755"
become: true
# --- Aux model (Phi-4-14B Q4_K_M) --------------------------------------------
- name: Check if aux model GGUF already exists
ansible.builtin.stat:
path: "{{ llm_aux_model_path }}"
register: llm_aux_model_stat
- name: Download aux model — Phi-4-14B-Q4_K_M GGUF
ansible.builtin.get_url:
url: "{{ llm_aux_model_url }}"
dest: "{{ llm_aux_model_path }}"
headers:
Authorization: "Bearer {{ llm_hf_token }}"
owner: "{{ llm_service_user }}"
group: "{{ llm_service_user }}"
mode: "0644"
timeout: 7200
force: false
become: true
no_log: true
# Idempotency guard: skip if file exists and is above the min-size threshold
# (catches partial/truncated downloads from an interrupted prior run).
when: not llm_aux_model_stat.stat.exists or (llm_aux_model_stat.stat.size | int) < (llm_aux_model_min_bytes | int)
# --- Tool-calling model (Mistral-Small-24B Q3_K_M) ---------------------------
- name: Check if tool-calling model GGUF already exists
ansible.builtin.stat:
path: "{{ llm_toolcall_model_path }}"
register: llm_toolcall_model_stat
- name: Download tool-calling model — Mistral-Small-24B-Instruct-2501 Q3_K_M GGUF
ansible.builtin.get_url:
url: "{{ llm_toolcall_model_url }}"
dest: "{{ llm_toolcall_model_path }}"
headers:
Authorization: "Bearer {{ llm_hf_token }}"
owner: "{{ llm_service_user }}"
group: "{{ llm_service_user }}"
mode: "0644"
timeout: 7200
force: false
become: true
no_log: true
when: not llm_toolcall_model_stat.stat.exists or (llm_toolcall_model_stat.stat.size | int) < (llm_toolcall_model_min_bytes | int)
- name: Report model files present on disk
ansible.builtin.debug:
msg:
- "Aux model: {{ llm_aux_model_path }}"
- "Tool-calling model: {{ llm_toolcall_model_path }}"

54
ansible/roles/llm-inference-multimodel/tasks/systemd.yml Normal file
View File

@@ -0,0 +1,54 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference-multimodel/tasks/systemd.yml
# DESCRIPTION: Phase 2 — template + deploy both unit files.
# DELIBERATELY DOES NOT START OR ENABLE either service — that is
# Phase 4 (verify.yml)'s job, after Phase 3 firewall scoping is
# in place. This keeps "units land on disk" and "processes
# actually bind ports and load 20+GB into VRAM" as separately
# reviewable checkpoints per Ryan's iterative-build preference.
#
# Two independent units (llama-server-aux.service,
# llama-server-toolcall.service) — NOT one unit with two
# ExecStarts — so either can be stopped/restarted without
# affecting the other (plan §2, §6 rollback requirement).
#
# The pre-existing Gemma unit (whatever discover.yml found it to
# be) is never templated, restarted, or disabled by this file.
# ------------------------------------------------------------------------------
- name: Deploy llama-server-aux systemd unit
ansible.builtin.template:
src: llama-server-aux.service.j2
dest: "/etc/systemd/system/{{ llm_aux_service_name }}.service"
owner: root
group: root
mode: "0644"
become: true
notify:
- reload systemd
- restart llama-server-aux
- name: Deploy llama-server-toolcall systemd unit
ansible.builtin.template:
src: llama-server-toolcall.service.j2
dest: "/etc/systemd/system/{{ llm_toolcall_service_name }}.service"
owner: root
group: root
mode: "0644"
become: true
notify:
- reload systemd
- restart llama-server-toolcall
- name: Flush handlers so daemon-reload lands before any later phase acts on unit state
ansible.builtin.meta: flush_handlers
# NOTE: no `ansible.builtin.systemd: state: started / enabled: true` task here
# on purpose. Units exist on disk after this phase; nothing is running.
# The "restart" handlers above only fire (and thus only start anything) if
# the template content actually changed AND a later flush_handlers/end-of-play
# triggers them — on a first-ever apply this DOES start the services once,
# which is expected/acceptable for a fresh deploy, but on any subsequent
# re-run with no template changes, nothing restarts. Ryan/verify.yml owns
# the deliberate first start + smoke test.

133
ansible/roles/llm-inference-multimodel/tasks/verify.yml Normal file
View File

@@ -0,0 +1,133 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference-multimodel/tasks/verify.yml
# DESCRIPTION: Phase 4 — start both services, curl smoke test each endpoint,
# nvidia-smi VRAM check against plan §1 math, confirm no OOM.
#
# This is the ONLY phase that actually starts the services
# (systemd.yml deliberately does not). Enabling happens here too,
# so a reboot brings both back — matching plan §2's "independent
# systemd services" intent for durability, not just this-session.
# ------------------------------------------------------------------------------
- name: Enable and start llama-server-aux
ansible.builtin.systemd:
name: "{{ llm_aux_service_name }}"
state: started
enabled: true
daemon_reload: true
become: true
- name: Enable and start llama-server-toolcall
ansible.builtin.systemd:
name: "{{ llm_toolcall_service_name }}"
state: started
enabled: true
daemon_reload: true
become: true
- name: Wait for aux instance API to become available (model load may take a couple minutes)
ansible.builtin.uri:
url: "http://{{ llm_bind_address }}:{{ llm_aux_port }}/health"
status_code: 200
register: llm_aux_health
retries: 24
delay: 10
until: llm_aux_health.status == 200
- name: Wait for tool-calling instance API to become available
ansible.builtin.uri:
url: "http://{{ llm_bind_address }}:{{ llm_toolcall_port }}/health"
status_code: 200
register: llm_toolcall_health
retries: 24
delay: 10
until: llm_toolcall_health.status == 200
- name: Smoke-test — aux instance model listing
ansible.builtin.uri:
url: "http://{{ llm_bind_address }}:{{ llm_aux_port }}/v1/models"
status_code: 200
return_content: true
register: llm_aux_models
- name: Smoke-test — tool-calling instance model listing
ansible.builtin.uri:
url: "http://{{ llm_bind_address }}:{{ llm_toolcall_port }}/v1/models"
status_code: 200
return_content: true
register: llm_toolcall_models
- name: Report served models per instance
ansible.builtin.debug:
msg:
- "Aux (:{{ llm_aux_port }}) serving: {{ llm_aux_models.json.data | map(attribute='id') | list }}"
- "Tool-calling (:{{ llm_toolcall_port }}) serving: {{ llm_toolcall_models.json.data | map(attribute='id') | list }}"
- name: Basic completion smoke test — aux instance (non-tool-calling sanity check only)
ansible.builtin.uri:
url: "http://{{ llm_bind_address }}:{{ llm_aux_port }}/v1/chat/completions"
method: POST
body_format: json
body:
model: "{{ llm_aux_model_id }}"
messages:
- role: user
content: "Reply with exactly one word: OK"
max_tokens: 10
status_code: 200
return_content: true
register: llm_aux_completion
- name: Basic completion smoke test — tool-calling instance (plain-text sanity check only)
ansible.builtin.uri:
url: "http://{{ llm_bind_address }}:{{ llm_toolcall_port }}/v1/chat/completions"
method: POST
body_format: json
body:
model: "{{ llm_toolcall_model_id }}"
messages:
- role: user
content: "Reply with exactly one word: OK"
max_tokens: 10
status_code: 200
return_content: true
register: llm_toolcall_completion
- name: NOTE — this smoke test is NOT the tool-calling validation harness
ansible.builtin.debug:
msg: >-
Both endpoints respond to basic completions. This does NOT validate
tool_calls correctness or hallucination-safety for the tool-calling
instance — that is a separate, manual, post-deploy procedure (plan §7).
See references/tool-calling-validation.sh (copied from the
llm-inference-homelab skill) and docs/validation-log.md in this role.
DO NOT point any Claude Code / tool-calling-capable Hermes profile at
port {{ llm_toolcall_port }} until that validation has passed and been
logged.
- name: Check GPU VRAM usage after both instances are running
ansible.builtin.command:
cmd: nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv,noheader
register: llm_post_start_vram
changed_when: false
- name: Report VRAM usage vs plan §1 expectations
ansible.builtin.debug:
msg:
- "Measured (nvidia-smi): {{ llm_post_start_vram.stdout }}"
- "Design estimate (plan §1): aux ~{{ llm_aux_expected_vram_gb }}GB + toolcall ~{{ llm_toolcall_expected_vram_gb }}GB = ~{{ llm_combined_expected_vram_gb }}GB / {{ llm_gpu_total_vram_gb }}GB total"
- "If measured usage exceeds ~23.5GB or is within ~0.5GB of the 24GB card limit, treat as the OOM-risk trigger condition from plan §6 — do not leave both services running unattended without confirming headroom."
- name: Check for OOM-kill events related to llama-server in dmesg (best-effort, read-only)
ansible.builtin.shell:
cmd: "dmesg | grep -i 'llama-server' | grep -i -E 'oom|killed' || true"
register: llm_oom_check
changed_when: false
become: true
- name: Report any OOM-kill findings
ansible.builtin.debug:
msg: >-
{{ llm_oom_check.stdout if llm_oom_check.stdout | length > 0
else 'No OOM-kill events found for llama-server in dmesg.' }}

33
ansible/roles/llm-inference-multimodel/templates/llama-server-aux.service.j2 Normal file
View File

@@ -0,0 +1,33 @@
[Unit]
Description=llama-server (aux/classification) — Phi-4-14B Q4_K_M (OpenAI-compatible inference)
After=network.target nvidia-persistenced.service
Wants=nvidia-persistenced.service
[Service]
Type=simple
User={{ llm_service_user }}
Group={{ llm_service_user }}
Environment="HOME=/home/{{ llm_service_user }}"
ExecStart={{ llm_binary_path }} \
--model {{ llm_aux_model_path }} \
--host {{ llm_bind_address }} \
--port {{ llm_aux_port }} \
--ctx-size {{ llm_aux_ctx_size }} \
--n-gpu-layers {{ llm_aux_gpu_layers }} \
--parallel {{ llm_aux_parallel }} \
--metrics
# NOTE: no --chat-template flag — let llama-server auto-detect Phi-4's own
# embedded chat template from GGUF metadata (same reasoning as the existing
# llm-inference role's Gemma unit: explicit overrides risk mismatching the
# model's actual expected format).
# NOTE: --host is the private LAN IP (10.1.71.130 by default), NOT 0.0.0.0 —
# deliberate change from the pre-existing Gemma pattern (plan §5).
Restart=on-failure
RestartSec=10
TimeoutStartSec=600
StandardOutput=journal
StandardError=journal
SyslogIdentifier=llama-server-aux
[Install]
WantedBy=multi-user.target

38
ansible/roles/llm-inference-multimodel/templates/llama-server-toolcall.service.j2 Normal file
View File

@@ -0,0 +1,38 @@
[Unit]
Description=llama-server (tool-calling) — Mistral-Small-24B-Instruct-2501 Q3_K_M (OpenAI-compatible inference)
After=network.target nvidia-persistenced.service
Wants=nvidia-persistenced.service
[Service]
Type=simple
User={{ llm_service_user }}
Group={{ llm_service_user }}
Environment="HOME=/home/{{ llm_service_user }}"
ExecStart={{ llm_binary_path }} \
--model {{ llm_toolcall_model_path }} \
--host {{ llm_bind_address }} \
--port {{ llm_toolcall_port }} \
--ctx-size {{ llm_toolcall_ctx_size }} \
--n-gpu-layers {{ llm_toolcall_gpu_layers }} \
--parallel {{ llm_toolcall_parallel }} \
--metrics
# NOTE: no --chat-template flag — let llama-server auto-detect Mistral-Small's
# own embedded chat template from GGUF metadata.
# NOTE: --host is the private LAN IP (10.1.71.130 by default), NOT 0.0.0.0.
# NOTE: --parallel 1 is deliberate (plan §1/§2) — tool-calling profiles are
# single-session-at-a-time per Claude Code profile; lower parallelism reduces
# KV overhead and lowers hallucination surface from context bleed between
# concurrent slots.
# IMPORTANT: this endpoint MUST pass the plan §7 validation harness
# (docs/validation-log.md in this role) before any Claude Code / tool-calling
# Hermes profile is pointed at it. Mistral-Small shares lineage concerns
# flagged for Qwen2.5/Qwen3 hallucinated tool_calls — do not assume safety.
Restart=on-failure
RestartSec=10
TimeoutStartSec=600
StandardOutput=journal
StandardError=journal
SyslogIdentifier=llama-server-toolcall
[Install]
WantedBy=multi-user.target

22
ansible/roles/llm-inference-multimodel/vars/main.yml Normal file
View File

@@ -0,0 +1,22 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference-multimodel/vars/main.yml
# DESCRIPTION: Role-internal constants (not meant to be overridden per-host).
# Model URLs/quant filenames live here rather than defaults/ since
# they're not really "tunable" — they're the specific artifacts
# named in the approved plan (§1). If Ryan wants a different
# quant/model, that's a defaults/main.yml override or a plan
# revision, not a vars/ edit.
# ------------------------------------------------------------------------------
# HuggingFace auth — reuse the same vault token as roles/llm-inference.
# vault_hf_token is defined in group_vars/all/vault.
llm_hf_token: "{{ vault_hf_token }}"
# Expected VRAM subtotals from plan §1 (informational — surfaced in verify.yml
# output so a live nvidia-smi reading can be sanity-checked against the design
# math, not enforced as a hard gate).
llm_aux_expected_vram_gb: 10.0
llm_toolcall_expected_vram_gb: 13.2
llm_combined_expected_vram_gb: 23.2
llm_gpu_total_vram_gb: 24.0

55
ansible/roles/llm-inference/defaults/main.yml Normal file
View File

@@ -0,0 +1,55 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference/defaults/main.yml
# DESCRIPTION: Overridable defaults for the llm-inference role.
# Deploy target: astro-orbiter (Dell OptiPlex 7050 SFF, RTX 3090
# via OCuLink, Ubuntu 24.04.4 LTS).
# ------------------------------------------------------------------------------
# NVIDIA driver
llm_nvidia_driver_package: nvidia-driver-595-open
# Python venv
llm_venv_path: /home/jarvis/vllm-env
llm_venv_owner: jarvis
# HuggingFace
llm_hf_cache_dir: /home/jarvis/.cache/huggingface
llm_hf_model: google/gemma-2-27b-it
# vLLM serve (deprecated — replaced by llama-server)
# llama-server serve
llm_serve_port: 8000
llm_serve_host: "0.0.0.0"
# NOTE (2026-08-05): --ctx-size is llama.cpp's TOTAL KV cache pool, divided
# evenly across --parallel slots (per-slot context = ctx-size / parallel).
# Previous 8192/4=2048 tokens-per-slot was too small for aux task offload
# (context compression) and caused live rejections: "request (3826 tokens)
# exceeds the available context size (2048 tokens)".
# Sized against measured VRAM on astro-orbiter (RTX 3090, 24576MiB total):
# - Weights (Q4_K_M, 27B) ~16998MiB resident.
# - At ctx-size=8192/parallel=4, total llama-server VRAM = 19404MiB
# (nvidia-smi), i.e. ~2406MiB for KV cache + compute buffers at 8192
# total context tokens -> ~294KiB/token (pool-wide, incl. buffers).
# - Model n_ctx_train=8192 is the native max; per-slot context beyond
# this degrades coherence, so per-slot should cap at 8192.
# - New sizing: ctx-size=16384, parallel=2 -> 8192 tokens/slot (native
# max, covers compression's multi-thousand-token inputs with margin).
# Projected VRAM: 16998 + (~294KiB/token * 16384) ≈ 21.8GB used,
# leaving ~2.7GB headroom on the 24GB card.
# - parallel=2 (not 4) trades some concurrency for correct per-slot
# context; 2 concurrent aux-task requests is enough headroom before
# the known "3+ simultaneous compressions" GPU bottleneck kicks in.
llm_max_model_len: 16384
llm_gpu_layers: 99 # offload all layers to GPU
llm_parallel_slots: 2 # concurrent request slots -> 8192 tokens/slot (ctx-size / parallel)
llm_gguf_dir: /home/jarvis/models
llm_gguf_path: /home/jarvis/models/gemma-2-27b-it-Q4_K_M.gguf
# Legacy vLLM vars (kept for role documentation, not used by llama-server)
llm_quantization: "bitsandbytes"
llm_gpu_memory_utilization: "0.92"
# Monitoring
llm_gpu_exporter_version: "1.13.1"
llm_gpu_exporter_port: 9835

42
ansible/roles/llm-inference/handlers/main.yml Normal file
View File

@@ -0,0 +1,42 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference/handlers/main.yml
# ------------------------------------------------------------------------------
- name: Reload systemd
ansible.builtin.systemd:
daemon_reload: true
listen: "reload systemd"
- name: Restart vllm-serve
ansible.builtin.systemd:
name: vllm-serve
state: restarted
listen: "restart vllm-serve"
failed_when: false
- name: Restart llama-server
ansible.builtin.systemd:
name: llama-server
state: restarted
listen: "restart llama-server"
- name: Restart Hermes on carousel
ansible.builtin.systemd:
name: "{{ item }}"
state: restarted
scope: user
loop:
- hermes-gateway
- hermes-dashboard
become: true
become_user: wed
delegate_to: carousel-of-progress
listen: "restart hermes"
ignore_errors: true
- name: Restart nvidia-gpu-exporter
ansible.builtin.systemd:
name: nvidia-gpu-exporter
state: restarted
listen: "restart nvidia-gpu-exporter"

14
ansible/roles/llm-inference/meta/main.yml Normal file
View File

@@ -0,0 +1,14 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference/meta/main.yml
# ------------------------------------------------------------------------------
galaxy_info:
role_name: llm_inference
author: rblundon
license: MIT
description: >
Deploys vLLM serving stack with NVIDIA RTX 3090 on Ubuntu 24.04.
Manages NVIDIA drivers, Python venv, model download, systemd service,
and Hermes provider integration.
min_ansible_version: "2.15"
dependencies: []

25
ansible/roles/llm-inference/tasks/driver.yml Normal file
View File

@@ -0,0 +1,25 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference/tasks/driver.yml
# DESCRIPTION: Phase 2 — NVIDIA driver.
# Installs nvidia-driver-595-open via apt. Fully idempotent —
# already installed on astro-orbiter on 2026-08-03, this is a no-op.
# DKMS builds the kernel module automatically on install.
# ------------------------------------------------------------------------------
- name: Install NVIDIA driver package
ansible.builtin.apt:
name: "{{ llm_nvidia_driver_package }}"
state: present
update_cache: false
notify: reload systemd
- name: Verify nvidia-smi reports the GPU
ansible.builtin.command: nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader
register: nvidia_smi_out
changed_when: false
failed_when: nvidia_smi_out.rc != 0
- name: Print nvidia-smi output
ansible.builtin.debug:
msg: "GPU detected: {{ nvidia_smi_out.stdout }}"

45
ansible/roles/llm-inference/tasks/foundation.yml Normal file
View File

@@ -0,0 +1,45 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference/tasks/foundation.yml
# DESCRIPTION: Phase 1 — Foundation.
# - Asserts vault secret is defined
# - Adds jarvis user to nvidia GPU groups
# - Creates HuggingFace cache directory
# - Creates venv parent directory
# NOTE: NVIDIA driver install (Phase 2) already performed manually on
# 2026-08-03 (nvidia-driver-595-open, DKMS built, nvidia-smi verified).
# Phase 2 tasks are idempotent and will no-op on astro-orbiter.
# ------------------------------------------------------------------------------
- name: Assert HuggingFace token is defined in vault
ansible.builtin.assert:
that:
- vault_hf_token is defined
- vault_hf_token | length > 0
fail_msg: >
vault_hf_token is not defined. Add it to group_vars/all/vault:
vault_hf_token: "hf_xxxxxxxxxxxxxxxxxxxx"
- name: Add jarvis user to nvidia GPU groups
ansible.builtin.user:
name: jarvis
groups:
- video
- render
append: true
- name: Create HuggingFace cache directory
ansible.builtin.file:
path: "{{ llm_hf_cache_dir }}"
state: directory
owner: jarvis
group: jarvis
mode: "0755"
- name: Create venv parent directory
ansible.builtin.file:
path: "{{ llm_venv_path | dirname }}"
state: directory
owner: jarvis
group: jarvis
mode: "0755"

44
ansible/roles/llm-inference/tasks/integration.yml Normal file
View File

@@ -0,0 +1,44 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference/tasks/integration.yml
# DESCRIPTION: Phase 6 — Wire astro-orbiter into Hermes as a secondary provider.
# Writes a provider config fragment to carousel-of-progress
# (the Hermes host) so FRIDAY crons can route to the local model.
#
# Hermes provider config lives at ~/.hermes/config.yaml on carousel.
# This task uses the lineinfile/blockinfile approach to add the provider
# entry idempotently without clobbering the existing config.
#
# NOTE: Hermes must be restarted on carousel after this task runs.
# Manual step — JARVIS will notify Ryan.
# ------------------------------------------------------------------------------
- name: Check if astro-orbiter provider already configured in Hermes
ansible.builtin.command:
cmd: grep -c "astro-orbiter" /home/wed/.hermes/config.yaml
register: provider_check
changed_when: false
failed_when: false
delegate_to: carousel-of-progress
- name: Add astro-orbiter as Hermes secondary provider
ansible.builtin.blockinfile:
path: /home/wed/.hermes/config.yaml
marker: "# {mark} ANSIBLE MANAGED — astro-orbiter vLLM provider"
insertafter: "^providers:"
block: |
# astro-orbiter — local RTX 3090 vLLM inference
- name: astro-orbiter
type: openai-compatible
base_url: http://{{ hostvars['astro-orbiter']['ansible_host'] }}:{{ llm_serve_port }}/v1
model: {{ llm_hf_model }}
api_key: none
when: provider_check.stdout == "0"
delegate_to: carousel-of-progress
notify: restart hermes
- name: Remind operator to restart Hermes on carousel
ansible.builtin.debug:
msg: >
Phase 6 complete. Hermes on carousel-of-progress has been updated.
Restart Hermes manually or via: systemctl --user restart hermes-gateway hermes-dashboard

36
ansible/roles/llm-inference/tasks/main.yml Normal file
View File

@@ -0,0 +1,36 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference/tasks/main.yml
# DESCRIPTION: Entry point — imports one task file per phase.
# Phases are additive; re-running the full playbook is always safe.
# Use --tags to run a specific phase subset:
# --tags foundation,driver,vllm,model,serve,integration,monitoring
# ------------------------------------------------------------------------------
# Phase 1 — Foundation
- import_tasks: foundation.yml
tags: [foundation]
# Phase 2 — Driver
- import_tasks: driver.yml
tags: [driver]
# Phase 3 — vLLM
- import_tasks: vllm.yml
tags: [vllm]
# Phase 4 — Model
- import_tasks: model.yml
tags: [model]
# Phase 5 — Serve
- import_tasks: serve.yml
tags: [serve]
# Phase 6 — Integration
- import_tasks: integration.yml
tags: [integration]
# Phase 7 — Monitoring
- import_tasks: monitoring.yml
tags: [monitoring]

42
ansible/roles/llm-inference/tasks/model.yml Normal file
View File

@@ -0,0 +1,42 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference/tasks/model.yml
# DESCRIPTION: Phase 4 — HuggingFace login and Gemma 2 27B model download.
# Idempotent: snapshot_download skips files already present.
# Requires vault_hf_token and Gemma 2 licence accepted at
# huggingface.co/google/gemma-2-27b-it.
# ------------------------------------------------------------------------------
- name: Write HuggingFace token to ~/.cache/huggingface/token
ansible.builtin.copy:
content: "{{ vault_hf_token }}"
dest: "/home/{{ llm_venv_owner }}/.cache/huggingface/token"
owner: "{{ llm_venv_owner }}"
group: "{{ llm_venv_owner }}"
mode: "0600"
no_log: true
- name: Download Gemma 2 27B model via snapshot_download
ansible.builtin.command:
cmd: >
{{ llm_venv_path }}/bin/python -c "
from huggingface_hub import snapshot_download
path = snapshot_download(
'{{ llm_hf_model }}',
cache_dir='{{ llm_hf_cache_dir }}',
)
print(path)
"
creates: "{{ llm_hf_cache_dir }}/models--{{ llm_hf_model | replace('/', '--') }}/snapshots"
become: true
become_user: "{{ llm_venv_owner }}"
environment:
HF_TOKEN: "{{ vault_hf_token }}"
HOME: "/home/{{ llm_venv_owner }}"
register: model_download
timeout: 3600
no_log: false
- name: Print model download path
ansible.builtin.debug:
msg: "Model available at: {{ model_download.stdout | default('already present') }}"

131
ansible/roles/llm-inference/tasks/monitoring.yml Normal file
View File

@@ -0,0 +1,131 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference/tasks/monitoring.yml
# DESCRIPTION: Phase 7 — Prometheus monitoring for the LLM inference stack.
# Deploys two metric-producing exporters on astro-orbiter:
#
# 1. node_exporter (port 9100) — system: CPU, RAM, disk, network
# 2. nvidia_gpu_exporter (port 9835) — GPU: VRAM, temp, util, power
# 3. llama-server built-in metrics (port 8000/metrics, enabled via
# --metrics) — just needs a scrape job (no extra process)
#
# GitOps note: the Prometheus scrape jobs for all three targets and
# the Grafana dashboard are declared in the homelab Git repo and
# applied by ArgoCD — NOT by this role:
# - cluster/applications/monitoring/values.yaml
# (prometheus.prometheusSpec.additionalScrapeConfigs)
# - cluster/applications/monitoring/dashboards.yaml
# (grafana-llm-inference-dashboard ConfigMap)
# This role's job is only to stand up the two exporters + verify
# they're reachable. Do NOT reintroduce kubectl patch/apply tasks
# here — cluster-facing changes go through Git commit + ArgoCD
# sync so state stays reproducible and self-healing.
# ------------------------------------------------------------------------------
# -----------------------------------------------------------------------
# 1. node_exporter
# -----------------------------------------------------------------------
- name: Install prometheus-node-exporter
ansible.builtin.apt:
name: prometheus-node-exporter
state: present
update_cache: false
- name: Enable and start node_exporter
ansible.builtin.systemd:
name: prometheus-node-exporter
state: started
enabled: true
- name: Verify node_exporter is reachable
ansible.builtin.uri:
url: "http://localhost:9100/metrics"
status_code: 200
register: node_exporter_health
retries: 6
delay: 5
until: node_exporter_health.status == 200
changed_when: false
# -----------------------------------------------------------------------
# 2. nvidia_gpu_exporter (utkuozdemir/nvidia_gpu_exporter)
# Lightweight single-binary exporter — no CUDA dependency, uses nvidia-smi.
# -----------------------------------------------------------------------
- name: Create nvidia_gpu_exporter install directory
ansible.builtin.file:
path: /opt/nvidia_gpu_exporter
state: directory
owner: root
group: root
mode: "0755"
- name: Download nvidia_gpu_exporter binary
ansible.builtin.get_url:
url: "https://github.com/utkuozdemir/nvidia_gpu_exporter/releases/download/v{{ llm_gpu_exporter_version }}/nvidia_gpu_exporter_{{ llm_gpu_exporter_version }}_linux_x86_64.tar.gz"
dest: "/tmp/nvidia_gpu_exporter.tar.gz"
mode: "0644"
register: gpu_exporter_download
- name: Extract nvidia_gpu_exporter binary
ansible.builtin.unarchive:
src: /tmp/nvidia_gpu_exporter.tar.gz
dest: /opt/nvidia_gpu_exporter
remote_src: true
creates: /opt/nvidia_gpu_exporter/nvidia_gpu_exporter
- name: Deploy nvidia_gpu_exporter systemd service
ansible.builtin.copy:
dest: /etc/systemd/system/nvidia-gpu-exporter.service
mode: "0644"
content: |
[Unit]
Description=NVIDIA GPU Prometheus Exporter
After=network.target nvidia-persistenced.service
Wants=nvidia-persistenced.service
[Service]
Type=simple
ExecStart=/opt/nvidia_gpu_exporter/nvidia_gpu_exporter \
--web.listen-address=:{{ llm_gpu_exporter_port }}
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=nvidia-gpu-exporter
[Install]
WantedBy=multi-user.target
notify:
- reload systemd
- restart nvidia-gpu-exporter
- name: Flush handlers before starting gpu exporter
ansible.builtin.meta: flush_handlers
- name: Enable and start nvidia-gpu-exporter
ansible.builtin.systemd:
name: nvidia-gpu-exporter
state: started
enabled: true
daemon_reload: true
- name: Verify nvidia_gpu_exporter is reachable
ansible.builtin.uri:
url: "http://localhost:{{ llm_gpu_exporter_port }}/metrics"
status_code: 200
register: gpu_exporter_health
retries: 6
delay: 5
until: gpu_exporter_health.status == 200
changed_when: false
# -----------------------------------------------------------------------
# 3. Prometheus scrape configs + Grafana dashboard
#
# Intentionally NOT managed here. See file header: these are declared
# in cluster/applications/monitoring/{values.yaml,dashboards.yaml} in
# the homelab Git repo and rolled out by ArgoCD sync, keeping cluster
# state in Git rather than mutated imperatively from the control node.
# -----------------------------------------------------------------------

151
ansible/roles/llm-inference/tasks/serve.yml Normal file
View File

@@ -0,0 +1,151 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference/tasks/serve.yml
# DESCRIPTION: Phase 5 — llama-server (llama.cpp) serving Gemma 2 27B-it GGUF.
#
# WHY llama.cpp instead of vLLM:
# vLLM with bitsandbytes int4 quantizes on-the-fly — loads full bf16 weights
# (~54GB RAM peak) before compressing, killing the 40GB OptiPlex on warmup.
# llama.cpp loads the pre-quantized GGUF directly (~15.5GB peak RAM for Q4_K_M).
# No torch.compile, no warmup spike, OpenAI-compatible API on the same port.
#
# GGUF source: bartowski/gemma-2-27b-it-GGUF (Q4_K_M, 15.5GB)
# Model downloaded to: {{ llm_gguf_path }}
# ------------------------------------------------------------------------------
- name: Install llama.cpp build dependencies
ansible.builtin.apt:
name:
- cmake
- build-essential
- libcurl4-openssl-dev
state: present
update_cache: false
# NOTE: nvidia-driver-595-open provides the runtime driver only (nvidia-smi,
# libcuda.so) — it does NOT ship nvcc/CUDA headers needed to build GGML_CUDA=ON.
# Ubuntu 24.04's nvidia-cuda-toolkit (12.0.x) is sufficient to build llama.cpp
# against; it does not need to match the 595 driver's CUDA 13.2 runtime version.
- name: Install NVIDIA CUDA toolkit (nvcc) for building llama.cpp with CUDA support
ansible.builtin.apt:
name: nvidia-cuda-toolkit
state: present
update_cache: false
become: true
- name: Clone llama.cpp repository
ansible.builtin.git:
repo: https://github.com/ggml-org/llama.cpp.git
dest: /opt/llama.cpp
depth: 1
update: false
become: true
- name: Check for incomplete/stale llama.cpp CMake configuration
ansible.builtin.stat:
path: /opt/llama.cpp/build/Makefile
register: llama_cmake_generated
- name: Remove stale llama.cpp build dir if CMake configure never completed
ansible.builtin.file:
path: /opt/llama.cpp/build
state: absent
become: true
when:
- not llama_cmake_generated.stat.exists
- not (ansible_check_mode | default(false))
- name: Build llama.cpp with CUDA support
ansible.builtin.command:
cmd: cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
chdir: /opt/llama.cpp
creates: /opt/llama.cpp/build/CMakeCache.txt
become: true
- name: Compile llama.cpp (parallel build)
ansible.builtin.command:
cmd: cmake --build build --config Release --parallel {{ ansible_processor_vcpus }}
chdir: /opt/llama.cpp
creates: /opt/llama.cpp/build/bin/llama-server
become: true
timeout: 600
- name: Create GGUF model directory
ansible.builtin.file:
path: "{{ llm_gguf_dir }}"
state: directory
owner: "{{ llm_venv_owner }}"
group: "{{ llm_venv_owner }}"
mode: "0755"
- name: Check whether GGUF already exists (avoid re-downloading 16.6GB on every run)
ansible.builtin.stat:
path: "{{ llm_gguf_path }}"
register: llm_gguf_stat
- name: Download Gemma 2 27B Q4_K_M GGUF from HuggingFace
ansible.builtin.get_url:
url: "https://huggingface.co/bartowski/gemma-2-27b-it-GGUF/resolve/main/gemma-2-27b-it-Q4_K_M.gguf"
dest: "{{ llm_gguf_path }}"
headers:
Authorization: "Bearer {{ vault_hf_token }}"
owner: "{{ llm_venv_owner }}"
group: "{{ llm_venv_owner }}"
mode: "0644"
timeout: 7200
force: false
become: true
no_log: true
# Idempotency: skip entirely once the file exists and is reasonably sized
# (the finished GGUF is ~16.6GB; guard against a truncated partial download
# being mistaken for complete by only trusting files > 15GB).
when: not llm_gguf_stat.stat.exists or (llm_gguf_stat.stat.size | int) < 15000000000
- name: Disable and stop vllm-serve if present
ansible.builtin.systemd:
name: vllm-serve
state: stopped
enabled: false
failed_when: false
notify: reload systemd
- name: Deploy llama-server systemd service unit
ansible.builtin.template:
src: llama-server.service.j2
dest: /etc/systemd/system/llama-server.service
owner: root
group: root
mode: "0644"
notify:
- reload systemd
- restart llama-server
- name: Flush handlers to reload systemd before enabling service
ansible.builtin.meta: flush_handlers
- name: Enable and start llama-server
ansible.builtin.systemd:
name: llama-server
state: started
enabled: true
daemon_reload: true
- name: Wait for llama-server API to become available (model load ~30s)
ansible.builtin.uri:
url: "http://localhost:{{ llm_serve_port }}/health"
status_code: 200
register: llama_health
retries: 18
delay: 10
until: llama_health.status == 200
- name: Smoke-test — list available models
ansible.builtin.uri:
url: "http://localhost:{{ llm_serve_port }}/v1/models"
status_code: 200
return_content: true
register: llama_models
- name: Print available models
ansible.builtin.debug:
msg: "llama-server serving: {{ llama_models.json.data | map(attribute='id') | list }}"

43
ansible/roles/llm-inference/tasks/vllm.yml Normal file
View File

@@ -0,0 +1,43 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/llm-inference/tasks/vllm.yml
# DESCRIPTION: Phase 3 — Python venv + vLLM install.
# Idempotent: venv creation and pip install only run if the
# venv binary or vllm package is absent.
# Already completed manually on 2026-08-03 — will no-op.
# ------------------------------------------------------------------------------
- name: Create Python venv for vLLM
ansible.builtin.command:
cmd: python3 -m venv {{ llm_venv_path }}
creates: "{{ llm_venv_path }}/bin/python"
become: true
become_user: "{{ llm_venv_owner }}"
- name: Upgrade pip inside venv
ansible.builtin.pip:
name: pip
state: latest
virtualenv: "{{ llm_venv_path }}"
become: true
become_user: "{{ llm_venv_owner }}"
- name: Install vLLM and bitsandbytes
ansible.builtin.pip:
name:
- vllm
- bitsandbytes
state: present
virtualenv: "{{ llm_venv_path }}"
become: true
become_user: "{{ llm_venv_owner }}"
- name: Verify vLLM is importable
ansible.builtin.command:
cmd: "{{ llm_venv_path }}/bin/python -c 'import vllm; print(vllm.__version__)'"
register: vllm_version
changed_when: false
- name: Print vLLM version
ansible.builtin.debug:
msg: "vLLM version: {{ vllm_version.stdout }}"

32
ansible/roles/llm-inference/templates/llama-server.service.j2 Normal file
View File

@@ -0,0 +1,32 @@
[Unit]
Description=llama-server — Gemma 2 27B-it Q4_K_M (OpenAI-compatible inference)
After=network.target nvidia-persistenced.service
Wants=nvidia-persistenced.service
[Service]
Type=simple
User={{ llm_venv_owner }}
Group={{ llm_venv_owner }}
Environment="HOME=/home/{{ llm_venv_owner }}"
ExecStart=/opt/llama.cpp/build/bin/llama-server \
--model {{ llm_gguf_path }} \
--host {{ llm_serve_host }} \
--port {{ llm_serve_port }} \
--ctx-size {{ llm_max_model_len }} \
--n-gpu-layers {{ llm_gpu_layers }} \
--parallel {{ llm_parallel_slots }} \
--metrics
# NOTE: no --chat-template flag — llama-server auto-detects and uses the
# GGUF's own embedded Jinja chat template (verified correct Gemma-2
# start_of_turn/end_of_turn format for bartowski's gemma-2-27b-it-Q4_K_M).
# The built-in "--chat-template gemma" name does NOT match this model's
# expected format on this llama.cpp build and produced garbled completions.
Restart=on-failure
RestartSec=10
TimeoutStartSec=120
StandardOutput=journal
StandardError=journal
SyslogIdentifier=llama-server
[Install]
WantedBy=multi-user.target

29
ansible/roles/llm-inference/templates/vllm-serve.service.j2 Normal file
View File

@@ -0,0 +1,29 @@
[Unit]
Description=vLLM inference server — {{ llm_hf_model }}
After=network.target nvidia-persistenced.service
Wants=nvidia-persistenced.service
[Service]
Type=simple
User={{ llm_venv_owner }}
Group={{ llm_venv_owner }}
Environment="HF_TOKEN={{ vault_hf_token }}"
Environment="HOME=/home/{{ llm_venv_owner }}"
Environment="HF_HUB_CACHE={{ llm_hf_cache_dir }}"
ExecStart={{ llm_venv_path }}/bin/python -m vllm.entrypoints.openai.api_server \
--model {{ llm_hf_model }} \
--host {{ llm_serve_host }} \
--port {{ llm_serve_port }} \
--quantization {{ llm_quantization }} \
--gpu-memory-utilization {{ llm_gpu_memory_utilization }} \
--max-model-len {{ llm_max_model_len }} \
--enable-prefix-caching
Restart=on-failure
RestartSec=10
TimeoutStartSec=300
StandardOutput=journal
StandardError=journal
SyslogIdentifier=vllm-serve
[Install]
WantedBy=multi-user.target

227
cluster/applications/monitoring/dashboards.yaml
View File

@@ -9198,4 +9198,231 @@ data:
"uid": "nginx",
"version": 1,
"description": "Ingress-nginx supports a rich collection of prometheus metrics. If you have prometheus and grafana installed on your cluster then prometheus will already be scraping this data due to the scrape annotation on the deployment."
}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-llm-inference-dashboard
namespace: monitoring
labels:
grafana_dashboard: "1"
data:
llm-inference-astro-orbiter.json: |
{
"title": "LLM Inference — astro-orbiter",
"uid": "llm-astro-orbiter",
"timezone": "browser",
"refresh": "30s",
"schemaVersion": 38,
"tags": ["llm", "gpu", "llama-server", "astro-orbiter"],
"time": { "from": "now-1h", "to": "now" },
"panels": [
{
"id": 1,
"title": "GPU Utilization %",
"type": "timeseries",
"gridPos": { "x": 0, "y": 0, "w": 8, "h": 8 },
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0, "max": 100,
"color": { "mode": "palette-classic" },
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 70 },
{ "color": "red", "value": 90 }
]
}
}
},
"targets": [
{
"expr": "nvidia_smi_utilization_gpu_ratio{hostname=\"astro-orbiter\"} * 100",
"legendFormat": "GPU Util"
}
]
},
{
"id": 2,
"title": "GPU VRAM Used",
"type": "timeseries",
"gridPos": { "x": 8, "y": 0, "w": 8, "h": 8 },
"fieldConfig": {
"defaults": {
"unit": "bytes",
"color": { "mode": "palette-classic" }
}
},
"targets": [
{
"expr": "nvidia_smi_memory_used_bytes{hostname=\"astro-orbiter\"}",
"legendFormat": "VRAM Used"
},
{
"expr": "nvidia_smi_memory_total_bytes{hostname=\"astro-orbiter\"}",
"legendFormat": "VRAM Total"
}
]
},
{
"id": 3,
"title": "GPU Temperature",
"type": "gauge",
"gridPos": { "x": 16, "y": 0, "w": 8, "h": 8 },
"fieldConfig": {
"defaults": {
"unit": "celsius",
"min": 0, "max": 100,
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 70 },
{ "color": "red", "value": 85 }
]
}
}
},
"targets": [
{
"expr": "nvidia_smi_temperature_gpu{hostname=\"astro-orbiter\"}",
"legendFormat": "GPU Temp"
}
]
},
{
"id": 4,
"title": "GPU Power Draw",
"type": "timeseries",
"gridPos": { "x": 0, "y": 8, "w": 8, "h": 8 },
"fieldConfig": {
"defaults": {
"unit": "watt",
"color": { "mode": "palette-classic" }
}
},
"targets": [
{
"expr": "nvidia_smi_power_draw_watts{hostname=\"astro-orbiter\"}",
"legendFormat": "Power Draw"
},
{
"expr": "nvidia_smi_power_limit_watts{hostname=\"astro-orbiter\"}",
"legendFormat": "Power Limit"
}
]
},
{
"id": 5,
"title": "llama-server — Requests per Second",
"type": "timeseries",
"gridPos": { "x": 8, "y": 8, "w": 8, "h": 8 },
"fieldConfig": {
"defaults": {
"unit": "reqps",
"color": { "mode": "palette-classic" }
}
},
"targets": [
{
"expr": "rate(llamacpp:tokens_predicted_total{hostname=\"astro-orbiter\"}[1m])",
"legendFormat": "Tokens/s (predicted)"
},
{
"expr": "rate(llamacpp:prompt_tokens_total{hostname=\"astro-orbiter\"}[1m])",
"legendFormat": "Tokens/s (prompt)"
}
]
},
{
"id": 6,
"title": "llama-server — Requests Processing/Deferred",
"type": "timeseries",
"gridPos": { "x": 16, "y": 8, "w": 8, "h": 8 },
"fieldConfig": {
"defaults": {
"unit": "short",
"color": { "mode": "palette-classic" }
}
},
"targets": [
{
"expr": "llamacpp:requests_processing{hostname=\"astro-orbiter\"}",
"legendFormat": "Processing"
},
{
"expr": "llamacpp:requests_deferred{hostname=\"astro-orbiter\"}",
"legendFormat": "Deferred"
}
]
},
{
"id": 9,
"title": "System — CPU Usage %",
"type": "timeseries",
"gridPos": { "x": 0, "y": 16, "w": 8, "h": 7 },
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0, "max": 100
}
},
"targets": [
{
"expr": "100 - (avg by(instance) (rate(node_cpu_seconds_total{mode=\"idle\",instance=~\"10.1.71.130:.*\"}[1m])) * 100)",
"legendFormat": "CPU Used"
}
]
},
{
"id": 10,
"title": "System — Memory Usage",
"type": "timeseries",
"gridPos": { "x": 8, "y": 16, "w": 8, "h": 7 },
"fieldConfig": {
"defaults": {
"unit": "bytes"
}
},
"targets": [
{
"expr": "node_memory_MemTotal_bytes{instance=~\"10.1.71.130:.*\"} - node_memory_MemAvailable_bytes{instance=~\"10.1.71.130:.*\"}",
"legendFormat": "Used"
},
{
"expr": "node_memory_MemTotal_bytes{instance=~\"10.1.71.130:.*\"}",
"legendFormat": "Total"
}
]
},
{
"id": 11,
"title": "System — Disk Usage (root)",
"type": "gauge",
"gridPos": { "x": 16, "y": 16, "w": 8, "h": 7 },
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0, "max": 1,
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 0.75 },
{ "color": "red", "value": 0.90 }
]
}
}
},
"targets": [
{
"expr": "1 - (node_filesystem_avail_bytes{instance=~\"10.1.71.130:.*\",mountpoint=\"/\"} / node_filesystem_size_bytes{instance=~\"10.1.71.130:.*\",mountpoint=\"/\"})",
"legendFormat": "Root disk"
}
]
}
]
}

30
cluster/applications/monitoring/values.yaml
View File

@@ -212,6 +212,36 @@ prometheus:
# array: utilidor
# array_type: physical
# astro-orbiter — LLM inference host (Ryzen 7 5800XT / RTX 3090)
# Managed by roles/llm-inference (Phase monitoring). Three targets:
# node (system), gpu (nvidia_gpu_exporter), llama-server (inference metrics)
- job_name: node-astro-orbiter
scrape_interval: 30s
static_configs:
- targets:
- 10.1.71.130:9100
labels:
hostname: astro-orbiter
- job_name: gpu-astro-orbiter
scrape_interval: 15s
static_configs:
- targets:
- 10.1.71.130:9835
labels:
hostname: astro-orbiter
gpu: rtx3090
- job_name: llama-server-astro-orbiter
scrape_interval: 15s
metrics_path: /metrics
static_configs:
- targets:
- 10.1.71.130:8000
labels:
hostname: astro-orbiter
model: bartowski/gemma-2-27b-it-GGUF
# ─── Grafana ──────────────────────────────────────────────────────────────────
grafana:
enabled: true