feat(llm-inference): move astro-orbiter monitoring to GitOps (values.yaml + dashboards.yaml)

- Prometheus scrape configs for node/gpu/llama-server exporters on astro-orbiter now declared in cluster/applications/monitoring/values.yaml (additionalScrapeConfigs), applied via ArgoCD sync instead of an imperative kubectl secret patch from the Ansible role. - Grafana dashboard for astro-orbiter LLM inference added as a ConfigMap in cluster/applications/monitoring/dashboards.yaml (grafana_dashboard=1 sidecar label), replacing the role's ad-hoc kubectl apply of a rendered Jinja template. - ansible/roles/llm-inference/tasks/monitoring.yml: removed the kubectl get/patch/apply tasks and orphaned grafana-llm-dashboard.json.j2 template; role now only stands up node_exporter + nvidia_gpu_exporter and verifies they're reachable — cluster-facing config lives in Git. - host_vars/vars.yml + inventory.yml: finalize astro-orbiter as the llama.cpp/RTX 3090 host (jarvis user, ssh key), drop stale Ollama/AMD vars and ollama_server inventory group superseded by the ATX rebuild.
fix(llm-inference): switch serve phase from vLLM+bitsandbytes to llama.cpp+GGUF
2026-08-05 09:43:54 -05:00 · 2026-08-03 12:37:03 -05:00 · 2026-08-03 12:35:46 -05:00 · 2026-08-03 11:57:42 -05:00 · 2026-08-03 11:53:36 -05:00 · 2026-08-03 11:51:34 -05:00
20 changed files with 1324 additions and 375 deletions
--- a/ansible/ansible.cfg
+++ b/ansible/ansible.cfg
@@ -262,7 +262,7 @@ roles_path=./roles

 # (path) The vault password file to use. Equivalent to ``--vault-password-file`` or ``--vault-id``.
 # If executable, it will be run and the resulting stdout will be used as the password.
-;vault_password_file=
+vault_password_file=/home/hermes/.vault_pass.txt

 # (integer) Sets the default verbosity, equivalent to the number of ``-v`` passed in the command line.
 ;verbosity=0
--- a/ansible/group_vars/all/vault
+++ b/ansible/group_vars/all/vault
@@ -1,361 +1,365 @@
 $ANSIBLE_VAULT;1.1;AES256
-63623430346462366164333638633439646431326632346630383066333138313438643039373630
-6230333230356433343761666335323934656463653737360a376337346163316334333266346639
-30366436623165323462623330633935313438303761343666393162303466373762623235333161
-3835323539376536300a623962323438326336343537316630626262336339353730316162623935
-64666430626164316163633064643263396238353231373438643432343535633966636430386632
-35653437383361373938353034306264633139373663353062363163623634333538623831633831
-64643261376662613763306664353566643462306431643963633261643732386131333565366539
-35663062633730326166616366323865396536363532643931386637626430373562626531376430
-39376634363634656337363038666336663365363562356664333139633833323562386562396166
-33636437346130326539623364343030326330646137653131383739663166666334623566386631
-64616239656662393536633134643163663537626435653561323066336662613536306363623536
-33396466383537613135353930353832653765313435666435313266336361656633356131653766
-30326132366535323638393137633466333337306466333265353635333762613731396663313839
-36363830653032373564306435626333646530656535396362346132663937336337376339633366
-65373630323631303062656339613536303932656135363437633934333764613939343830353432
-34313665666634336465646363356237306562653534613863353064336236343963393664323738
-38653237656534616562303439616332303134626565646261343131393162343433636232393665
-62343431343535646431363861643964643234343665343366363930303663303565636161393033
-63626638333763343139396465363165363632393664623838623065623466336136373333393263
-34316439656232653261653538396231346462653339353034313063633661376266363265636432
-64316336613736396266333937353833363230653132383339626635386563626131663063396633
-36333061396639653238373965613831653935333533386633613035643530376465366331373832
-32643936373630363539666463353839333335663763653530623037326263306636383233613037
-39376266323064633366326163666264393265306438616330613631663137303963353666353231
-30343565633235643731326136643132376135376639366139383466323333333732396231373534
-62366438376232613463653231376164643930356137336337323763633063613861636266666262
-31323934356237373964316531353537366236633162303661663964356464653561333166643432
-61376631653335633264393762636435376535316663333065313638343361366439383432656138
-38353737353865396233366231373439333863386161323532373537333934346431373664633138
-30646437643737663031613332636536313039313663653262633665633564363130643738333031
-33666232333734646531326266393934636430626332356266333461326238633262316364356336
-65326238316336323037633236383765323332633936633062353266623238633738336132383432
-61343432343035386130363462663239613166626661613332393163386232356237383464383762
-31323739643235326264316134306430356539613865353532663133313830303061396339353566
-35373062316237373232616662383663636236353365626665386532333336313864383837383035
-62653638306136316137663738616663636434303533336433373739313331616631343131353265
-37353430386330333562366132393537613465343363623838393165666538623131653164376434
-61376533353366613735626165353562626431313031633238343966623962323231626265366330
-34353165623864343031623565363235323763646638393762646565393465396534343937666335
-62643530643435343931666563396661653662396265303362663932333666383534656539663033
-61663533353333323733613933363433383033346463306430353634633466396564623837346133
-38636131393738636666313765353835343666353663396539623236313237303437666665303966
-38323364376665613731336334383535373835303138373036613665336636306364623464303963
-39393830646133363263653330306531633863333064663466346135316161376362653862313132
-61333830303765386138303865323766353663656531643965356331633532343435633461366333
-61343534613166656639643363333231653738353936393938376665616136626361653432326531
-64633861653738616237313061393132383264313964643764366534303263653233393035363435
-31376164356466323431383836326334616130363264393139316362333130643262643766376331
-35666130303636363666363932616134306665313339323231306666353236353965356332343430
-36636263353565396535383862336464646134346566626635363063613761353333326137646662
-32656535356332626166636432643635353734393564376161316134613561303931663366616366
-35393166393632326564393763366161663636336362303566316162326631663937336437623637
-39613832666462613963643834313932323837393337633735633833646265666666653339353832
-37636332633063636133663938306234316666393735643634303833666537363932323363613466
-37626235626633383639663262653138613038363737323639306234366562666236326332323932
-63323336623633303736306362326237636465666465616361666130363538643135616237316234
-37623562396363326237663665636539346166323566386365376665616236653735646637363939
-64656662353233323065303438306436623735643934613839333133356564363438653133333037
-33656238663733663665373463393038303865613838366538356538383539396262656561353563
-31666333316564336262366435306261613563376462356338613063343232393766653332643237
-34636639333962393663333566643863386261666630393362356138623732323435666534376561
-61366363313861376566316235326633383630316338346333623432303766663130333130303363
-66616336613364666165393964666665383330316232316465663561613239386434663135613231
-62623334383433373461383037343037663236646463393632636161336562663533636235663039
-65383330373039386361633363613035636135373765353331636139313838353231353731346336
-63323736636130383962363762343437383237373533396433343464383235393030373862346233
-36646238623165353264323162316362376338383463663664636132373963623938373537373037
-30353262626363666565386536626163393130306634343836663137653537393066333637313639
-31376336373565313630636436616635623633396333313036366530613630663332323134613961
-36666438656636626561303637646539656166373861626232663266386533376531346166653834
-34656530363464343131303666616237373030393863326132313764373966633365313834316438
-34396239613235623233373136313464626661303666303438613337366662306363636439613332
-33626635613039363864343233323930333731376665346138626363396336373862636565613061
-30613863356637396133656663653232366265373766636237356539623639333133346361363361
-63343639353734313163346333356138343237383238623733343331376631636632373231363834
-62626432303732663435353762346161363330373266353031633862366539333133313931396166
-38373330323661616366343063363437333436626438376665333866376338626232383735613434
-64656363633332646533376535326566336566306439373835363264373335306564616337383465
-38353966636434653064326265643933653661363439363334383537623563303066346133353431
-62343662646434633761653332323430636339303666646262636334316461383865366436383231
-61313133316332363337343963306431356231306534636231643539383733383265303662373430
-34626666333830353934626139663532663332353938393038303631633335636236393133623064
-37306337386464656339383261323464303934373337333235343635326164653762613838333030
-39653063316131616566613365383961383435636639616536316662333736396238376661386533
-34643833323164303930656165396235353531326535313165663361653939656263633638633337
-36353435613335633239316435313965613439663061313834356661376365653763323837643561
-34333833626663353438623161366537326336646630613536653831306664663636343963313165
-65323332353633323931346634336331636465396337393035613834633962343662393930646337
-36346330353031636633326532616362646465386633363333613134653261323234636539633734
-63343965356433366233646536326333313366366537343934383239383439393633396461633166
-65633439336462363061316435396235346366386335666233343963383163393465633766643731
-62326362613266653238356435633032373036623839306132333136653537306365626165616638
-64316134316134393463313165376536383031366132363831333432323466653538376433333033
-65366438353935613863356562666264643063623439383335396531613730353861313664663234
-36626533366134643164353138306234333765623634316533383264303636353039643130366433
-38313133333632663666303862323939633463633634653963323866376137376137633930313963
-61336431656332333636373638316237663861336136393036653636366566316661303034623566
-31323537356338363762643265613534316466386263666236633038386634333139636561656331
-63613636316439393435383931306530306163386266373161666139373235653935393632313261
-36343066653561363333363830313136653538366639333464376562316334343839363332643137
-39336463613631346161616534613933613234656435383234313831366366643961393633633035
-39663935626134626138383038636266356434393535353165626632633430313635623532646165
-31363166373335326433363336326662663561306166313862373161633739643662376461346266
-61663730366337376165653033353466653761613938366532653633363530616134393464303138
-66353637623135626431643835613239633033386433663735356264333734613162343234633566
-64643039616137393863396433353138373862646138633838356365373863623235376638383439
-36373732373031346132363738383635653865346637626233663537613636613862323666616438
-34333336346235636234653062613466376637623165336337643933623535353033383438646530
-36653133336637346432343136336331336464633439383938353862323365343864666533316633
-66646262343534373165366266656135666339633333626133643238386139353537353866616631
-38306335663165363138303635383164303535386536626431663661396664376433656339373934
-38396130313438653539393263323265646163633939653431393035356332336664396234383636
-32323863366237633462316536626639343335626131323536373364393266386163633433356263
-36366665373432313132363761383632343532306361346134313464653763393533613565363839
-62623938346530636562383937393534653963636365303837336165343361313734303666623739
-30653834616163636538313038636261343434326164616264356261336239313039373661396535
-34346631643337666234373933383362656266346266393035363332303636363035656538636363
-39396466353565346264353663316562623239373564616162326439383334663730326435353236
-37366238653131356339306566326534616561653637613438666430336334323031373661353636
-64356131343763653639313531303866636334326637313838623138303836666466623736623237
-39363838653431323665366634303632666337363663386261326538313635313638383764383765
-34363832366135333436303635373761653265653165316135633135373365346266356635663533
-37303134613131653563356662363965353234643765373438313835343063313931626464636562
-36313434356434656138616433666266353538306666333363363431356530313262356438623537
-33303365393061393161626665316332626238353438623466373831303939663734663038346462
-39323462663732653335373261306635303231636563373465343761613262396466623135323736
-61316638313535653031633230333638343737633163636335343864336336373437643131643335
-63653766623436323039303065646162626331336637346237656132353039623063396631666535
-36316634623034326664313766326464383337663230643530623563326333336232666334323139
-33346563353734346338333432366437353665323764366335336232353163373563326265346234
-36383437383332366234393636323034376434336461336532613663316533653463633263343261
-62333535653062663663313534303237386562623533636230626531646461613233653330386466
-32623137633136323363386232393466303537643331656464663234636336393532313831313961
-37623939323963643364636439663032386630336235623037643065356631376637333062313762
-31653661316562643633643135616634353631623463353637616131343531633266393766336232
-30616138616437323138383732306537326562666239636561383631333534633937396463613765
-62313064653838623537393839386464653539326132393732393131633335393063393636376235
-64306566343762626238386632626136396331336564326361303835613732623531396631363564
-32616631333034313563396532396564663262613733626362663632333938336331633664376638
-34626466613239353935376230653732366236643830353963653662613338616631346561623063
-61616565393965613730326530353935323432306465656464626238623438636263666530336531
-64333864636131343566643164653934306135316437383738323738616362643636383761383433
-65353762663930636366376365623535333636613766666465316134303131623637343864643461
-33663934656631333032353734333361363339336266343631363937306632323635663832643266
-30383633323735333430376662663765303561643563383466333764636131363732636134376462
-37646564386564373038393135653761373937313039623737623938363631393632346565326166
-64376637323432303865366436356264633066343466333838303164373234646661346133646561
-62383838336461306638383266373462326165313161383832316462363835366361386663353935
-35616433393963613363333965666138363365666366646163336163613237386263343737636335
-36623864346566383632643035393361616533396538356562343364386330336636383433303538
-65653630303435373336366639656137386363303439333164633634323730393337333764633461
-32396161303864333362663466303161366465386338396332333462366339353461373638343961
-31343466663537636330663962636431366235646164643332343433393063633435646565333863
-37626239396264653434363538623133366561623436633739646335383534323839663465316339
-63653731636266343130383035323962333062623730343033613663366431386366623038373238
-32666537323237336236386135626339363861636439346664376561666538636532396562333435
-32383366373437303765653664396436623861333164396563623639636131656366316138396131
-61316432663239346539343261363461643632363432653161383233643431303034353136396465
-35623836393435396234336261626162326433356134636139303261653430363835333364656161
-34326533613636366134633332393064356535373432666165386163613961613539333932643963
-32343639343931623366633764363966636164323933623734333935613462303466656439306331
-36323261623639303164646435346533353966616138326230363739303063323366323964356534
-30316534656364373633663464366161373566626639363963356131353936333366326334663438
-32646463653664656635353564323230343763303538316135643838363134393634363031646530
-37323838386134316136663139626262396239643263353432613162666235616133623937383261
-31353138396238313066643964633536636337386238616637383561383563313936373538616334
-62363233623738613064313762303463303266346538333937343566343936326363323037623032
-30346434366661643930623761666134366132373936336630623963636331653532363765363263
-66306133303634343231386164343131333234376662313663393130326664386262363035353565
-37633863353935363635303039333662366462383235636564313666333162666563376666633966
-34666130613034393439346337383862643231666632663463393330356263646165316266613936
-62303536333339303431373531303065613865633339303264323137323732653036333366373664
-64623330313464356461376662663266333430633161393437363862633564396263353333316662
-64636262333832646135626534363737656336663562333365666631646261303665333964326337
-35643964386362323161666464363037623266386539383732626230303830303662366161653063
-36383539626334376238663539336637343033343037346136383733316265383430396133336139
-64616135623230613531313733626537366430626262623836386432363561626238303132613830
-36323533656164376334643436373730653531396236633933393663656435656566626165626164
-39366465646535383532616466313665333533386230656631343332376166613033343364393361
-62656237316466663131333661333565643335646166333332393263626463633639306264656666
-36343330666462373061363466636233333165663662393933343763633333373161626235363138
-64346433663661636561613636366662376562363638356231646438653532633434623132306330
-65656433353262623661636263393737623337316535326332373861393331626563326434383266
-65383931393032376137623833303031303934636135373135626562396364383635623661303431
-65623132393961396336363763616366633064366331316330396130373239336137373964356136
-37653162383030653736626564343739383030636531346534386639313063323938363236306537
-36646132626138643462636466656363353436633863613165613362616264613434303936633661
-38376661306439313036333431363664616662646533333963656165393939303865616661313864
-65656338633864663665303961613165386335646661343438333730313138616535363164336463
-35363563356362396265333334393964356534613763656362323631363237656438653962316336
-62633330616530376537333961383461323432303161343136326363313137333333323835313065
-64666132356632613365393137363061363165623733353534303638353432393739346164346535
-39363464313239386564336436663339303535336633666364373436666635613638643061316235
-61656266623934633339643063306363633131373335333538643338666339396166626262646434
-65623962396464343532366639393339306237653633623763333165303636613834333531306235
-36636430633864396236363638393165653431313139343966316561663138313761653630623436
-33363037303334383363646337303065343239343537343737616438613165323965333334613234
-62623931343365336230343037346230366134396435383236643931366631326335626431306366
-31646339323562653737303465353738343939623037616666656432313833653233373636326130
-30613434646237306636396239363439383961376539613131333164373333313131343062366638
-34333963313163303239333439393765386234313534303834636637393965323439366235613666
-65316336393335656332626165643261316464383232396136336238656662336132646366373263
-65396534353764616134663964376135623035316432623939376231373731343136313265323862
-61646238353761646434313430663362633066663763623735343730633036333533356339393665
-30313763346166653738636234663232323033636332373861373164393737353962323066363238
-34613066323862353831653438633564633561653034356263663430366432643663633836653764
-63333964663765353135636234323330623130633736653263336238396631356436346361366463
-65393239356432386163396135663161626664626137353465333733626635373537396137353437
-64626139393164313866616132386132303837303437306365336164383265346361383661323130
-62396333333666613538363332323032653037646563396436613166323166323666363934333064
-66356665306465653661323331376337653561613761643765666636396531663930333634316230
-35343733613638626230623464663133343432663764663864343363376339333566633963326630
-36313264636638396566623334333230363137353262346531623138656364653038373034353137
-31383533303234636333346332343333336531326363636161623764333762653835396632366436
-31646566363334633336366231363163643837633564363162656130323331626464333236303637
-31663439636234623534336535393835653838643430356364353762613966346461313032353939
-38613932386564623935363135663462626663313130663737373234316231343461616133626264
-31356163363234383132373730663436333738633562336435383532366138653764306165643863
-36383362633137643931333735333264656666336363663161643862323337613536383034336462
-35333165386332366436666439323236346539366530383136626637363539656230336637333432
-35336235363065306534643762346263393935333238383533646331653332643234386539653230
-30376237666166323638303465373737383031373864373266306537373335356139646230656264
-36316536363738323362643230363064363965636563346165356565643265353332386136346435
-66313937346164313661656364373465656230613261333833323335653133396132356334353237
-61353832373063666331646664623565353030663530636135336365633366623533653462386237
-65373462336538663835303831313336396638646534343839343933383032646664306434613230
-38323333666165363662376430356435363463303936396234313931303734666333376431376562
-65636132636435396562376637396464356165363937613435626235303730636239626465306237
-62613335396161303737396231666536353230656533653239353364623733623131333861626637
-64306437633564623565333664653030366166663835353063316631393864663338656663633736
-31346166656636396565333663353562613032373462366433323266396135363430623139373963
-32393864646438326366343036386231353530356233393539646562643339643366346563393162
-39393830646565326666333063396531356336653162623465313365616564653430663264363164
-62306562313037646466366463396432373536303631653338343133393262663865636133343464
-61383237386164633536663230346461383861626666623562313737373138633239333266366132
-32306135313464363461653363636462366662393938396566333661306366383838636438353738
-32356666613238356135353536656130346134663666346362353735613338666234626538373139
-65613262636233366532323630613438663334616164656166623236353531303835326266343663
-66366364333437373039306634383737663963373663366131396566613030386661656235616662
-35663533623864303431343631386362623036343135663665383039333363336534613063313534
-61373939306536376337303561373030343337326437393636366166376630346430666539646266
-34616435346139313033353130303639393530373662616432343330653762393339396432643935
-65313638383330613464363666633965393035643037626162616339306563646130306566613064
-35656130653035363163323831323730633236633736663965653833623131383037323736663066
-35343235356164623038643230373666366265363437396231613365303935366135636135613265
-31393163323837663332623636316462656338646566363361333863336365373562666434366530
-37626236343731313065353037346437356264363233363638393238373063386530646434663564
-35353562396564316535663436376335393136613338643864323132326431633066373462323330
-36306263646434333532316365613239376237303232383531343238623833626363346437386235
-66666565656635306162646430393738666132633065376330393162633066333534383561353336
-33313162666664613636303962656438626133373137336639356133363563653431646461313538
-37316433663263333536386632646536616564316362363238626231613162383966333063636132
-31623436346138613030396462626663616233326662663634633765643864373262393634353937
-32376437363737616337373333313832373135313535666430386536303236356563613166366332
-65323332626363663535633234323364663838333862643232376237653031303137303136633462
-32356431306534626135353462643736373939666130623732303562393932636231373534626132
-32343464353538356231303535396365633465333230353261346533333063303135303135393465
-34656333613966663966646162386633333830323234613836633236626134316162663139656530
-63623666313138666130326139386636646230353637353561323736656437616438633839613837
-62353931313562383830393433313232653461363436353064653039313738396138333263633965
-32316236363233613831303436323835616332646262343336353037323362313265336634393261
-35393761396537373636313665373830326463643136303961316238336238623466396334343664
-61396234663862343937633131623939306135646231373633653534353363383439656539333331
-30303536353462666433626236343163653139333038656662666664626439613639636564393835
-30363165616263656535303439636563643663633434613665613339313162363234623131306630
-34653932376634363264343335343064343838626464666661313533383534383833633863386463
-37633664396537363735353161346238656336396565383637313230636561626362656232333036
-32386538623063613331623662666531626362616430386335333664646234613965643733656236
-31303766326536326461353632353730303434333866353730383562383533623338626232346330
-64396163613663363562633635656566313263383262656666393537653233363166656335653034
-32376335623633373431376266663036616635343231663963363936353137376263656133623735
-65656435393733613762376638613034643466313632646431656461313362363561623463393238
-34373830373633373835376234366532363763316632613766366161613130633762313839386363
-61396632333533663964373463323464323035386236376364646332663431383039623830333137
-62313062393236363338616433613033306563323264316332366161346235356539323232653931
-66313862383935316235653663336666666339343534376336336138366639633538643937303665
-39386262306566386464363337336130633038633837663166333639343161373234303230306338
-30323063393365336332663531316365356531386365343836656437613337346666653637633639
-38653465393236633962373739323331623565643938373563376364333863343030333136333863
-64663132336165616462336232323865636630373533643063383730626565666337636539373136
-32333861656163646332323239653364326163613361373435323966323964313831663138353063
-34653131383463623965386365326531616265653837363462653037386633376361383639656163
-39663338383939323535366135396465656333323331386462353630396237396261316137386165
-38383664613165356535356632376138623233386266336665386330376539373963323162613336
-62396364373031656238356163656464616430316131386537356236663536366461326462373236
-62396363303861303533643438343765356437326663316239613939386134333830303739386464
-63623834366565366163383533393532383061646163666133396136643462376537393935613039
-37623134313439393737626664663637343365373563376261666133346332373033613335316164
-39313733653931383665663238303532353165613131323731306362666138643737346337623064
-34666261323664363864373061666233613163336633383837323766386137323431626337623562
-36336234656635633830366635363463313439646165353261323030323261633737346664373738
-39633836366133633635303638366134393435393663366664623837336164323630353739333462
-36366663643931653230653562326633353637383564623266313936656239393535646131373261
-61356661663534393166383530373739343834313136663366316237333464326563336636656232
-66373035623866383931393238666132386331356530616535313164366466353563653865663832
-39353364396366663462366333306135343065373532663330643435336239353435303762326132
-65343630343532656638653462383266383937663466633034373661643862663933333838653032
-33656261396638666334626366356362623663613461303936323130353436363536663638383162
-63663931313335666435616561353266376134633132633233356435313137653132373935613931
-36623461396139633439396332393037616361316562613137646466613330653234656535623230
-66306434653462633834643036656263376366623835653163326461313831633934366663646566
-65393966373466326137353261383763346666386337666130646133643666656464616438383038
-31306130356238366261616231363933666162373639363463303962303833626135366666303962
-33316161343936393634303964396566653963333237393631353938386262623739393432383339
-37653430336265323133363937303362343336373230633333323764346139326662343434396566
-66383035613262366430303636323236663937393933646566373838383562316334633836633737
-65346537353439306363633431396338396632336565386434363530346365333765363930613063
-65333166393136633665663539656331663037353138366239383332373363366562613237373234
-64303963666462323330653364396464383464666434343831313338633764636337363761616337
-63313462303339643364383361376364366232326361613137326135333263313964616634383464
-61333539373962653338313835656632663261363766346535323735633437323736323632343434
-61643662613261613835366331303439313062313033623065323262356466656533623735643235
-61643332363333363032326434363065303566616661386261386165383066306161376531356334
-63323634356566333065383763333965613366656661313931386532623963383265613131666231
-61363465343462313531313464323731356637393061663133336665363965653035333030383062
-33653335363536656333333738356437316332313361633136396535643836633035303830323232
-65633761333135633739323061613436386166383762353564326430363965623461306235386132
-66366632643763306338373064613935643630616661353233663834396331643130396465636534
-33303830383139343836633163313737313738636365346639396162343561383965323636306466
-63343038363330313363663566656464383264326538303466323862663463353930376165326466
-66383630376465376139646562633065383434333365376362643038643064656364363538356661
-35303432646332623337353465656362656234333465383663353735663537643630616266383763
-30363239363132333166363563376665633936656535663164656532393666336265393964636333
-35323539383835366162326362333537363666386162376561373464323630303032613037636636
-37663135623933643938396635323763353931336531313732636530643865663430646530666465
-34383962353137366431633530376134323635336433653361316363316132383834663030393666
-66303935343839303337636630313138336239393062666265646163386332376536336439616463
-66316165393963333235353931326461626130306435373061626233616430343064303537373839
-63396464306164613863333434373063396534623739373730303135626639663731366666343733
-32313862306431626231373963396461656637313364343661663235323237316432353762383330
-32353736643766323964643035653661363532336261353334363936333766613739623638333861
-30343933613161643862623263623838623866356534323362623838656133663363353366633764
-38323931346630616664316365363463653637663762633138323230356263636634396165613337
-32656666363137623833366561643563613938623565636132313539383130633637313236333365
-32623663333164653733643063396230313033323636323164643534376333346637636436303935
-34346539663366623137326133383533396535376131626631326463323866353363656636313538
-33393731356632313263663533633762313534346435326533346635356162326466316662323230
-62376666666339623638356138353461313431333366633833323836313338643333366161306139
-35383163373139636137356235656564303666346531663663326134656165616435616163323361
-61346131613666653431656464353636303663653934393835373332663432303762393538616465
-63363566396464333861386135336132623539626432383762313762316135623566373937643532
-64653934383662663263303833383034366436613138373863316334376564616235313462363637
-62313166623261626537323631626432393033626233353137366262613134333565666666326332
-62616662653334323535373235616433653233336265343661363765623066363762353836666364
-35373865363165363234333038323437313561646338343930316661633365623938623665623434
-66383562653564623732356130326138346236343766306535386362373664623162393833656230
-35643535353238336361646534313133303835363837353966326538313161353737363939633961
-39626137366133373536326535613362313866303232396233313263363736646139366363303463
-65363266323565303733613065366237636233306137346233333839646438623038343764346462
-35306434336363333562356566636335613263643738313766636534316232653137346166363932
-32386336653265663665623736663330623036323762623562346164373631313533396437613438
-63373632633731333361366462653466346338366139353632616661626630306633363061373937
-36386635653537346535363030326566366165393839653534383065336638303862373366633263
-33633461663764356330333534373036343565313335363731646432323931613737353231636435
-38626361393264363039303831633535643366636365656538343763646538336138623961303064
-31343431616337316661313935333735613662383235663730373632336333346338623432623731
-39303261326363656638333165646661333735316130363234343661393463346132333333633533
-36393635396234623065626463646536633139333662396635303537653338313532313062653339
-65656639646364346265386361623330333666326338386334376366306339653230633334373438
-36333735393664626136
+66393233316132396639356564316439343234383066633231646134313361666463656536323732
+6630616536646439613533363430306466306233643730350a343364633233333335643833326163
+64393933613963313533623733316339396236363663343635346663323366663166363839663837
+3331363062653239380a326566623264623837326636383939346430666537613361333638366630
+63353062393335316663633739313532366363623739653631366539323435336361353331386230
+39366634643964336233353961316630616462663166316266613037623363346335373638656365
+38353733396636386133373836346336383231663661346137373164386338623733393566373563
+35653933343036633365643535303934326537356136666539316137363433643266346630386439
+38613332646238366536333536343031356532656336613530663830613264346339353034323362
+66613530626361323535653232313730373463373332313561616631393461353730653464343063
+38616338363938346161616636316232313838616463326432353639613837343162646363343232
+33323064363139376566343866626364373662393138353666646234373461666163363139313631
+64343261326566363265323463663538343034306136326234386664333837333937333136653563
+61333531353434633339383661636363363535316366353330313566323133616438373161303135
+35353630613037316466353832333033393030636331386438393133366333653832393731366363
+62373638303737393162303461646239653865653834613662666636373364633165383062643831
+34386232376361323638353361666530366432356331353963303930326535663536373339333062
+32396266373430343339636635366434313635313766363863336464633961666332353834626163
+61653637316163636465343630353431313863653033643237356434313564366361373435376662
+63643737353830663236643862613533623237373531646136383763303766336139303632666235
+64623834323966363363663730626437323432623966663537346162656265363562643836633731
+65663631633462663764393132326165346639353033633035636432613039336164303538396632
+66396264393865306666643636353638613661313230313337383634663839363439656533333932
+36633432306131396539386539633063653230363932376264323537396434353364643432653661
+32643162363066636432336363323534316436613838646562313538326566666239633234646236
+30346534636533623365326564613561363362333364363037646561656635623935653466613565
+65646361313436356261643762313339333864356338386136306162386262636464393130303963
+38646464316432326431326661343632396235626234366133353461623862316662326432356234
+37626366383861373831633639616465663564643866356664623066386535646163336134356534
+33366664616232353863626465626364313530353335306565336665663866303736323162393362
+63626261653161663664363833313461653034326330653835393737616135646462366665383935
+30363639306330636634386433646231363530633061336364313338653632323831393630383934
+37316362326338313733646332336263386239626539383330353362616132333161613464313066
+34663434326662326233363432306433363666356132383866346336336261636435366332666135
+34616231613638363339356333616536643266636363643131653330396162306264303566396461
+64363763376365356533636430643866333361363062376237653237663731663934306265646630
+33393637656335643366383564373966343265393630333835303731316339373133633462383364
+33383435383331303264313334393532373932333334343862326635346135613932356337373034
+62316262326331313135376465343336373266663338396533666431616462613932663861646238
+62653563623535633738383033326235383666646333653731316233376231623661306462303732
+32643064373236613336396233323435393939386530323331336138353364663762356538316562
+33376530623664623733386133333433303031373337313366386236376539613964316135343865
+33363963366165333238356663663435386439336366646138313034343636653463323938633136
+39363966376238306662303265643034306136663661393738633436393432303139313132616534
+66323432313635386162333838323136623634653264643438303264636430633232323434666532
+32616664663063653735316237643539633133356661333132323238376333356464313262653836
+39303566316332663737323437633031353330333365383837636336643763313433313937396531
+38363536343438663966663436613132663661613134383431633765383164373762343435316161
+62303631646235343063383230343232383336356562303563373933346530393333316634316437
+32316330306163396434663031393965663163666537353031613365353437666466333464626238
+35313739646535356665323734393965303064306132626261363062363438383164346261393463
+30643438623363323161323230306230386332363635386234666639623566643536626637616533
+37396136643930633262333331656363376433333234343630306535313262306235663263663362
+32653434363035613732363136303363393939323337613661333439393637646262383039386661
+62326163323562333339323636363565623664396164383332633666386130613766393138346134
+33343338393536316431353439353062663164643634396363353131303038353965393466383030
+36656465383938353936346361393963356630666630373236626237303064303062383638373730
+35633866646535313432353338623462323235346433653431313031363163393666626432363238
+39623361316132626230633336636163623466313666346631656134343762656566353432353264
+30353436356237653231363564626134633039363035313232616333336436393638396233626638
+32663230396539323761313838313466376165646430346634383332346134653662393161363337
+62646161343665383364306665333164666231386531626465373366623761643161656462303733
+37653438616233353432626466623163316565353764323762613635333832343634323665356336
+35353162326233333836396337356466636131383838313436626336663132346339623261366465
+30623261303933396562353331636638376135663330643638643536346261626632626139386535
+66653332366361336636666437643165656239613031303638333232303836383132616636633938
+31343034643037623731643931316463303639656266323231313666356336333133323135363330
+63373365303131353161303630633738353536393631353034666139383435303461316131646138
+61333731356538366366613831303565613365633965323235366166313534653965366433656533
+62666136313662366638356237343734336333313034396465346632336262306531633535643238
+35333831366532386235316565303936616264373337356134643066396531383533353336303131
+31393837623564386535323532653733393734393164373235396566333565356237356438313762
+32623765326639386262393639376461326163333237313232386138643130643231626466643663
+39643061393566353434333136366335393536376234366266376265333234643536633035653933
+36316132663539306465343039323935356361373439346437386234386464623962643464643562
+61336434613834336161633237383361303930313464613666313834356330343138633735386530
+36616233323366323961653965613438346136373738366266316134356266623664313539636235
+37313033373466383134346361646562366531333338386330653736626530396238356639303131
+38363738396236386461316433316261326435646130383336316234363461393237623633633336
+30386630376565646337383738663939663462623232316635346635653830306664653336343033
+64316430396664393532313766326437636636626232613036666666656430323136356436333564
+36303334303562393832336433343438396430373833623137363736386665343866313064353063
+32303634393131326464656535633734386462646339663533666430336265653965333538633866
+61623666643839653239373335633735373738363736313665323365613635313766656635613832
+33616461643539636165383233636533626230343138663630323731626139393230383464313430
+61333438393337316239376435313337313437333931623238616133666138363235386533633437
+35356163363231656536353934643539643562343732626630383565623730626533313230656164
+35333735666135343364663233626163363930383262363266303265303638396239636361366534
+31333863333565356135613232393165353266343632633532343061663331633337343538376265
+39316630613439356262396634316361356436336634396337353339616536356336653930613966
+34656439653366363562636639346430623561303463356337363830373966366632303337663564
+35663632313265323365636238303364366230353039353561616636633664643233343430336237
+63373264643935616331616632633065366638363833306337633563653065363464343137623533
+36373231363739373335346464623533393336613634333636613937366136326464336332346166
+61376263623835646163353134643963663964373732313833346163323138633230393537636664
+30366234303334656130336630346130656237306161376566336534653630616439323764373665
+61383338326163336164353265326163646165623235626137623237306666333832306461613630
+66373331356465346261643466323662393661623433383265376666623932343861323139383531
+64633536373362643935633734366235396433333237306166646164363930613862613365303663
+38343833336137353634313362666665306666393635663633353934363832343739616331386130
+66336561633039326434313833303465366638303961626138333165623331386230616130626639
+34613962366230333065633761333335613636363533656461626632343631666563383738623330
+30333834346233653938633330663166616331376436356533366461336264643264336139343262
+35393665656230663232366133393037643536366234343537326631623332373131323739363638
+39653162646366316639313631393631666261623230313538613666393732626438393763646330
+36346661313131313630343432616365666633353762623261613039623331396330623939626132
+34626333386538326434356432623965666662663437646237373537326534653634346239653634
+31373038303639333037613637393862356263323066666630313262366633313932396465633337
+66653930303934616236323064613761353935613835356561313334323762633064306661346666
+37653262343865386236343634316336386630393739626437333065323433613531393738313432
+31376233353463373237653164386363633334366332356538343966663939656165323465333030
+39656532363363333432626638626438396539336461326338353732376235316133616666316261
+37353063343366376433653961333233306461303133376661303332386230346231383837396133
+37323137343066383966343535633363643233663530613566313330336232366638396165373631
+30626531363033313833303836366434613736396339643032663066333865306535323739666162
+64626133616433653864376662623464343131303938303237316264393765303035663833376464
+32663264383236303766323935306463643138396237373338653238633464616238306132633735
+31626538653262326533326266336633623532623935383266373533363466313033393235663538
+66653038646233303665343634383666343363383238326533366136363838303332323230316662
+35383235646638653539633961663036663933306463626335356631646662636230356261363261
+65633261353830373865636630353932323937666331353635373736376436333361613330366633
+30663939356165393132636131663966373433623063356265353131306532643066306630656363
+66636636353262633437663264613266613663656137386231306231646264363661613035343538
+37353633643065643236376537336238663137623735613038623766393231643131653436333262
+31626463646432613563393665346532386161366435396364663239386236616233356131323536
+33633936623762666534633862363466353736386137636363633733623366346337613365636439
+66663035313430386464623833646135333062313830396637323961386135363461326539623432
+32653865623530313637393561343465636430373162333162646631643235653931333830326266
+36376631316165343631326165623838306239623764363262376634663236393933343838376663
+39663834306165313330393739363133396436376437643232346336386531356638343063376465
+61613037623137306666383231376539656361326132396662613061376134376266633764336266
+64336334313335643635303632666431383637306334376462643630646339396435313830313363
+33383162316261663035393962306234613865613366353465373035656434366261383133653331
+63643235616362663663343330303765363263366130393837613939323264373937333162636639
+31643438666338646135663538343231643235646364623761653064633566656663383465626133
+31373935646266303565303539376162623132316438623565316537306337636630313861623937
+34313832636533623033616139373965303839356530353935643363613464356364343162336466
+66376130653162666661313139613530306666633432346639656466653364376435636461626362
+32653633303561346233643463373534653434323134353434373839373937626663336464303866
+33363264623038313835396231373132396163363662626264346461333539326365326165323066
+62333139383334333334353031616430323339623066363232313937323465356266323934313761
+66623835653961303830383030643537393130653935313265333062393034336562633535323263
+65616237373232336534393834653162363461336262653862666637326266663966356665363036
+35383437326465663635303664643236633435303862633965346133376536316233386333313634
+33643565326633386565653961646463383866646636303537643436623734393234633938333933
+65366164633165623333623362393639656661326332306538663738356364373734316563653038
+34646531616662386232613034366332656262343164333531353037363036646262623663666236
+38613238666136363431623664633863636365396236666532383930336636353031396232656435
+61346238643431653231623861373964383931336535363262373437353532393165316562386134
+36363263666135646237383666373833373737396330616163376439663736663937666161313831
+63663531656635663339306365656663636633343733636165386230376332616331313638386538
+32386466323232363533613334333333346161376430373436373961316564343061326164306138
+33616263666262323430303730626266396535626439623364376239346564323730323534323938
+33346364393033353865393864326361643734353234613563393138363334383536396535393166
+34623163616336653436393639313965353237633566313039303137326234383230323235363234
+37626161356166356365366164363863636563316332393638616535376466343537373966643839
+32613930643533336264626136626465303339376632323034386161663661376466616233633065
+66313739346162363838346663623266383130383736656334323430623463666439386532643630
+33666639613830386136363535363830333234653961663739343537306634616531616263623762
+64666230373830636238353062666330623061613663376638343763626264363130313464383661
+38326530333362616163363735323861376366333665623536383566653837306131623732373639
+31316661353332633630326162663738636562336666326637353764323431613666303038373532
+62343661336338306561356235396636343130633365303466613637633363613862663233633731
+66623530353132666261316637303763363830623734346262333633646238613131346564303734
+62336434353432326239333232383833633962313537626430663130393733623162626131656366
+64333535623138326239336165666562376663663334323036323539653734333835386331653438
+63353861666239396437346361306634613462386335376137333963333838616138633730393865
+62353539376136316564666136646639363635663736636439393462633165646632623664383663
+31613137306461616361323832393036323933626531363536336261356636303531633239333362
+32663134363263383039646162643539663737333861386437326337616362343963373532346238
+37346137363933623839373838353939386630303461346438666534616434333031373730393537
+30313134643963623564356266656430613430626238613266316335336265613132616562626261
+35386435313933626634616463616166646466363939313639646264346464363337656339323366
+36366665363739356564363232313762323565323134616134666337336534353464373637373130
+63613265366436313131356332316531633732356461383064383031613337343363646432373936
+33633339386632653032663837346130623636356464326637303338376132623734333932396232
+61333033386265356630316134383066343164613130666664643732643362666561346132656266
+31623633333039633837383264363937623435643061393935393762346430396335373864633634
+33336136353332663366313334353739303539633364663231636539333132303966383432376262
+61356563323232613433653262623663336634626532653465306638316633663564633862666666
+30366133616336326661626238653933383164336366333438626235636631336165386664343736
+62663961346664656333306435323833366632346366356238653731653937626333653630623334
+64326662346138386433333232643262333835326263343239353264373038613634356436396630
+38323931643361663238623766323930666130356339363564366661663033303831363138343737
+66633535326131396236653261303836613364306537633637323031663166316338323533323731
+30326235323066396663613531653061643661336631613835626266626436386662353465383065
+64396562343966303362636136616438353661626466636635323961613438646634336563636534
+38343566396530643961356434643933636235643561353232643062303232323437666261363061
+62636530396466303466653333633930376465366561376363316137323263333561343334383364
+39313863643062643766396564363137386231373136346138396162376264653538303464633161
+61663363623937356138356430666461623130323466623162653863393736326264393836336637
+34663931646566333535666664653237643732316663323230383239393763376266356135326438
+35656266353865623663373366373830613361373664346632363031356265313364623866643438
+61363866353934636337616239633330623734666138396166313864333939663563636138653930
+35653137363033326432373661613434313137623163356134613265393238346438306165313639
+36666662393165353565633531663536613037623230373063316639663632643139353235303462
+31393263656265656131613164363035343233626433656135353331613532363236616439363731
+62666432356363323937666435326638323437346136366131613636653430306131623966356263
+39303739306233303862636535633431363630393432613663633836396566653039383735303336
+62623331623034636363636661653236386337326666656532343737336262336462613762326531
+34303066303834386366636430343161653665343362363038396562626133636135656538306435
+36393364333066346238396362663664643236373532336263656233386663323663623137343462
+64386337333130316434663564613665666238623132343437656637653035373738313735366630
+66383639616166393265616434623463326437313530326130376339313662303836636664366232
+32366634633030333130316435616233396231663937343732313066373834326464623139363663
+31323931626364303230666162316436653065366137663631376265383063316534343736373261
+31613637316235386539343766323439653062633137663730343236343661346162653366656332
+32663932313063383561636266373766633535656131386133386135663863396261306530326632
+63653936626236316539613262386231616433393064323461626536363831666461316131383837
+30646266646266393666396362326238613231303335336532303836363264323233343534636635
+66313538643033343262373463363866346566353263303966323933383963363463393761383865
+64663932343830643531643466303438343161396133666463353762393737613036646166333265
+66376231613232666164663964636134653061633330383863373836306366393838393235656331
+35613231306263373230326634623262326333356263353961633836396531633431383163633361
+30356534666466653734333437383964346564346165326664633738653338313263633837316531
+36613034323433643839333264323864613033313137663131623265643364333664646235666232
+33393039313666323266643362323337316465306564303230646561303434666630616137633831
+34346439616634343337306636643733316464376631616266376437636439396337306637333432
+39306333363035393436316434656436353738303861633933376531383862316466373736323639
+35336137373866336631386436646231653366366435363932376434303063613961353261343661
+33383638393165336438376662306431333837356435626137356130323836396335636166306662
+36386161353739353637353861306666383966323339303262616239633930373633323937356632
+65383032613031666665623631613430666662656336663931646533636230303261646530623765
+64643939326435643539373564336531623236653731636636346361363064333963376566616530
+62326639663632666634326233363635383830643163373938646165656163643864336436373466
+36353832306632386230373832333234643638313238626333303963383962343265366137656136
+37333261343161633562346638323632616566646162633133663466346535656463393932386135
+62653332313066363965386335356430326539316366633537356364666230326237306236393563
+35323363633936353034323232353366373566666332323737653237323135646665626139393436
+65333762653536656161386532363765336538653763666236343166653933626666633130393033
+31643531646633623663313237353333313136663863663430306131316165663765653732663164
+36326531626365326330643064336230313466343731376437316563303339336333326636633066
+61386135626430616661313236623030316362373338643233326365646531633265626238383830
+65346132643537366537626132666165616138656139626132396639376230333262643766386363
+39383034663034326165643636613237623234613666333532383733623462303331326238636461
+34383139383466356139333934353837613964326538336463643832623062633034613762363061
+61346361656363366136353336326433326266336564316366393565626262303637316564356566
+39376661383763373436306238393666653561306538333638306233356139346634653363346164
+31303835383062376638626266303237323832623735653066353936376339633637333562333561
+36646462653834316131323166366661386161646538346464386232306239363030366363633663
+65306439616339626635326531636435356134376561303235393337373564373937623636643432
+61393535643038626562366331353831663338333838383066323632383633346564396566653330
+34386563393832313537623061666466366661333934613766366165366330353835323637643635
+38613363396663356564646132613536653033616337386566623662333832383938303138316662
+34303339323166383863363831636233323335313565393933636435396666313337663037323432
+63333139333165646262666339343736383966346133356138326437386334626461636530336432
+61356631366163366561303636333230643732316261376365386463333565623533663966336365
+39333365393766306536366231366435363030353263393534653534373064636361333532323735
+61313163323831653362356333386638343566356261353534303738613730373632363534666337
+62313339613138646361356431616236613435393233343732626263653332663265393934616134
+35313165613766643938393839373261633439396661623961353934373130623865353639633038
+34353631346433663131653965326337663561613330323562666336656237633163356562323931
+62393833663233333538383063303937386365306135343962623333663435663431396438666362
+61386266353838653532393233353939363738306634666537313761313835333864633764666262
+30333032623766633334383031636633636539336237613235376466356430663938653565626235
+64373537656566306136633630366130363630633462656330623633393735386630343437336436
+65343635366531646130616534623136636666323139326462306533653532643962656530336633
+35616537303932343539336638333730663639396330653761346136363431346536666138336462
+66396565613166623934316532383835316137303134363466306163356233356530323231666464
+30353933306530323734306564626234343864373964333264353366326265316333343330356532
+38363837646635633461653562303264353633343461633339376665616331613733666663353130
+65396363613731366234326466323738663563646166653237613364323734616465643764633537
+35373865353532383566363632366564353536643739663761303565333138383638653665663664
+65643366316461613630366437623736353739356538336237613431306363663234373265623962
+34653565373335653563356135313835643266356261623037336536613733323733363933376538
+36626134396563623733656534363331626262643339633932373035626134343531623634666463
+61623036313334616639633930393562663631653565656136666537393731333430663062643362
+63633663396562343965313261373965356163393538666466303661363531393266316462626166
+38616536653665366462383064373766396438616665346666376232653031323566313164383164
+36643231646439663637333165376439333432383532316661333766363136636236326338386537
+33306130353634346136356234363438383865313136393839663066333935623565333730613538
+63323831663866363831383930303434333936646564316435303931396362303534386335343330
+65383635346662626363626365666166636361633365643735303762393832316436646139303835
+63633733656361643233323332613632653837663262306661626438316262653931333061336366
+35353939353835333361623261613738383734656132613139393264393038373765343131333330
+37663962313566366463623437323965326365623437363038633661313461383634626661666236
+32633335323861643037383261393164393933353531636134323765353962633732396230636331
+32613865373739366530303538313566346434633933393330346637346136373036306666336164
+35373732346334353432616561623031663331346431383235306537386466623339356366663335
+36333733626433653465336431666530626665373564336339626163633131353330656437643638
+31616563616665343635356231633135663665326131636664373338323736393364353636613762
+66636135633766323866376235633535613735613465303239343036663438333331626431623435
+61323537386434323638666537643236623632626430666263376534643336613635663762643736
+30323237353265613062373265643562373637383337326264653639306263373865333262376665
+62646331373931373762303461366163393839633135393964313937616437323865653735383630
+32653936336534666565373437666130396265363561333635316461663766346336623865376133
+35373665303433326265623531613038636166643130616637653165376263643634376439613765
+35363031373630333966656466616235616337306335363132386335613462363664653634633864
+61303635653932353730663666386263633662633736313461643932386161313762663761313336
+62653665313033656537643936373465633932626166366430643763313030393838393039323230
+62633334383938343433306262393536653930653030393033306661666264313630643564333166
+35383237633932656331653030363434313534613637373465663264643061303538653666653861
+35373936643037333866636131373338363062663035323531626431633362663364396365353139
+66383737666437353764333231303662393630643933376161366430376530613365363830373534
+38376633333936626430393163323830346166643537326430616236393733653761363235356363
+35333131663032383861336262653936376565646662313965303265623763613330653461333835
+63613563323135633438383931343731656333303362316533376339376636623037376431336366
+61393236383364356162633062666265653534326363363862666539623761623065386537616563
+62666561316437303763376635346536666437373361386666643139643737663333323933613661
+38326566663932333930616435626133616531306461356466326437623235613233393434626563
+34373966633834373430386132353163366465626262353863353335323830393266393562393133
+65383632653438646435343333386261653066613663623232373564666465613136353039313036
+61646462626433646330396664363938376530376438646262343231393262383733636233636333
+35633862336566636439653464613564333162613836343636316334316665383164353131373431
+63623030306564346562346237333934616134346536303365396533626262333937396432393830
+38313763393463646437666137353835373735646365373934363936346564326362376565353133
+36653362333432326133393837316331666663663263396461363239306239363733633137396633
+38393865613431653337313665313762653635656531353465623436343132303064303564393066
+63616639353962366666616261393766643364333634346630616436376565313236316539633537
+66653239636561393433383639646462616433653166613130373134376535633937353366383230
+61613335663434333835653236343633633038346335333861356637353965396632393833646635
+36653034356234663831333764303338663464316362646339376338393236336161616263363538
+63356631613239326161343031643936623366643432663732346438333265666535623664396333
+62303239363339626566613439396234303536333333653433393666383635643235376165666234
+30336236393962666335353233666463346530323531316438373130303933383465316638646461
+62643066376666363236333231386237376466633932313836323163363061313333633434663763
+64323365353336333736363436376232653436633739613437343538633632356665656364616637
+62313666346436656564663335636635393632353430666236313863613464626434323939383538
+65386265343434303632313739353239323565333734656566356164643430613538333234383566
+37646431316363316139646435333732313339623666613738663039613239613738393565333330
+33376432373933656435303737653762666464363865633831373330393435633332636261336139
+34336236373535636165353262323966363164633135613534353661316364616637663465363864
+39306163346365643339643937396165666366663339336438373031613937636464383531613962
+34373433663533626665656364623634373335313033646165303764396563356235343033383138
+62326361353630393938643764616636313461633734646661386536356235656665393864386465
+37666262346561656436343036646330363664306135333464663265306165353039396665336664
+62346631366330653762646538323565613864383534636532633033643533323736373931643130
+32343435333333613734626234363132373734353035326232366264336161383631353133663230
+32636533333866343763336439373336356237303636376334333433376630353338333261333037
+31666537363666653238383939663464346662636133346561326335346163363061393830616237
+36643430626534653331653665316535303139343763663965363164636238366533303038653935
+32356432316633373137316237663331336463363431393033323635646564366639346230353363
+38346663363039363962323137383366303862356530353238656563306131643236626536656334
+38303462306562366532346163323061393437353063326539393466616439346564383036303235
+35633731393661323962633631373061303930323638326565636162316436646337383266626561
+32326430363031396530396238353862333133363731623736376239626561626165663337373261
+39353461343461643238646635633562653865323336366634613264616662323232653861663038
+64396330626633303031333334343335393039623135353266383561313231643433393963326637
+39313530636361373831306234383166346266656261663830636631333564356536323565336266
+38306561376366626236306633613564386166616630613032633163613837313462343662653261
+63353437663436303634633336636532646439636465663362346138313665336334313039613631
+65326135383831613531323265353831313562346161663265366434623236636635333038366536
+33646631663662393331323162343438626666366636613438383665633136326439376166373462
+33363864613136643461663436396362643066633437376631623031613366656238396165313832
+31313131653263666334393664343239306235373862313339373563643137393633343663613936
+61356564636238363136623031336638333566633766636362303938653531306131396665303033
+63353362636463636236643464343562383161343432383766396330623764393837613435396162
+31303162356437663932663964663239623764666366663061313535346438373334636263653531
+34643133356638653031373036343162653135663734623035353033633561366266623566383233
+36356434643538643430383532393762333535636639353361353763333363313131646264336332
+34373331343930633962623963666365306132356334646636626461316236343839383266363635
+65623434336239313330343437646333353362303232346638623161616133636636626236643465
+36303636363965363765656533386534633839346363363738386532386531326538643134363132
+66613235373362633166343565323766306335336365333439323764623964393263623236623832
+34346132383136303038363764333039626234616132386464666633663536656230666133363533
+38306665643361666539636666316432623430623939663636343164386438313765633031313534
+65393633323837326166343936326263343833646331326464376138633461613532303135393036
+65353830373065393038343039323937303634346665393135383639303162396565646232663736
+39376434646634353330383933303164653431373433346335666131386165343035303964626665
+35383035653631346638326637326235393833623264323030373238646335346332353362393230
+61616664613562383639306564376661306665396138613066326631616531623132633966633832
+65363637376264336635633132633332373634383864626564623966356464373864393832323738
+37616338646633326636323461376137663632376262363738303336616463326238333465343533
+31316132386530326539
--- a/ansible/host_vars/astro-orbiter/vars.yml
+++ b/ansible/host_vars/astro-orbiter/vars.yml
@@ -2,11 +2,14 @@
 # ------------------------------------------------------------------------------
 # FILE: ansible/host_vars/astro_orbiter/vars.yml
 # HOST: astro-orbiter (10.1.71.130)
-# ROLE: Ollama inference host with AMD RX 5700 GPU passthrough
+# ROLE: llama.cpp LLM inference host — Ryzen 7 5800XT / RTX 3090 (ATX rebuild,
+#       2026-08-04). Superseded the prior AMD RX 5700 / Ollama config below;
+#       drive was transplanted into new hardware, not reinstalled.
 # ------------------------------------------------------------------------------

 ansible_host: 10.1.71.130
-ansible_user: wed
+ansible_user: jarvis
+ansible_ssh_private_key_file: ~/.ssh/id_jarvis
 ansible_become: true

 # LVM root expansion — xlarge template uses sda3 partition, standard VG/LV names
@@ -15,11 +18,3 @@ common_root_pv: /dev/sda3
 common_root_vg: ubuntu-vg
 common_root_lv: ubuntu-lv

-# Ollama — all defaults apply; explicitly documented here for visibility
-ollama_rocm_version: "6.2"
-ollama_default_model: "qwen3:8b"
-ollama_hsa_override_gfx_version: "10.1.0"
-ollama_data_disk: /dev/sdb
-ollama_data_vg: ollama-vg
-ollama_data_lv: ollama-lv
-ollama_data_dir: /var/lib/ollama
--- a/ansible/inventory.yml
+++ b/ansible/inventory.yml
@@ -59,12 +59,9 @@ n8n_server:
  hosts:
    tiki-room:

-ollama_server:
+astro_orbiter:
  hosts:
    astro-orbiter:
-      ansible_host: 10.1.71.130
-      ansible_user: wed
-      ansible_become: true

 hermes_server:
  hosts:
@@ -72,6 +69,7 @@ hermes_server:
      ansible_host: 10.1.71.131
      ansible_user: wed
      ansible_become: true
+      ansible_ssh_private_key_file: ~/.ssh/ansible

 honcho_server:
  hosts:
--- a/ansible/playbooks/day1_deploy_llm_inference.yml
+++ b/ansible/playbooks/day1_deploy_llm_inference.yml
@@ -0,0 +1,25 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: playbooks/day1_deploy_llm_inference.yml
+# DESCRIPTION: Day 1 playbook for astro-orbiter LLM inference stack.
+#              Deploys vLLM + Gemma 2 27B on RTX 3090 via OCuLink.
+#
+# Usage:
+#   cd ~/git/homelab/ansible
+#   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference.yml
+#
+# Phases (added incrementally — safe to re-run):
+#   1. Foundation  — groups, directories, vault assertion
+#   2. Driver      — nvidia-driver-595-open (idempotent; already installed)
+#   3. vLLM        — Python venv + pip install vllm
+#   4. Model       — HF login, Gemma 2 27B snapshot_download
+#   5. Serve       — systemd vllm-serve.service, health check
+#   6. Integration — Hermes provider config on carousel
+# ------------------------------------------------------------------------------
+
+- name: Deploy LLM inference stack on astro-orbiter
+  hosts: astro_orbiter
+  gather_facts: true
+
+  roles:
+    - role: llm-inference
--- a/ansible/roles/llm-inference/defaults/main.yml
+++ b/ansible/roles/llm-inference/defaults/main.yml
@@ -0,0 +1,36 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/defaults/main.yml
+# DESCRIPTION: Overridable defaults for the llm-inference role.
+#              Deploy target: astro-orbiter (Dell OptiPlex 7050 SFF, RTX 3090
+#              via OCuLink, Ubuntu 24.04.4 LTS).
+# ------------------------------------------------------------------------------
+
+# NVIDIA driver
+llm_nvidia_driver_package: nvidia-driver-595-open
+
+# Python venv
+llm_venv_path: /home/jarvis/vllm-env
+llm_venv_owner: jarvis
+
+# HuggingFace
+llm_hf_cache_dir: /home/jarvis/.cache/huggingface
+llm_hf_model: google/gemma-2-27b-it
+
+# vLLM serve (deprecated — replaced by llama-server)
+# llama-server serve
+llm_serve_port: 8000
+llm_serve_host: "0.0.0.0"
+llm_max_model_len: 8192
+llm_gpu_layers: 99              # offload all layers to GPU
+llm_parallel_slots: 4           # concurrent request slots
+llm_gguf_dir: /home/jarvis/models
+llm_gguf_path: /home/jarvis/models/gemma-2-27b-it-Q4_K_M.gguf
+
+# Legacy vLLM vars (kept for role documentation, not used by llama-server)
+llm_quantization: "bitsandbytes"
+llm_gpu_memory_utilization: "0.92"
+
+# Monitoring
+llm_gpu_exporter_version: "1.13.1"
+llm_gpu_exporter_port: 9835
--- a/ansible/roles/llm-inference/handlers/main.yml
+++ b/ansible/roles/llm-inference/handlers/main.yml
@@ -0,0 +1,42 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/handlers/main.yml
+# ------------------------------------------------------------------------------
+
+- name: Reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true
+  listen: "reload systemd"
+
+- name: Restart vllm-serve
+  ansible.builtin.systemd:
+    name: vllm-serve
+    state: restarted
+  listen: "restart vllm-serve"
+  failed_when: false
+
+- name: Restart llama-server
+  ansible.builtin.systemd:
+    name: llama-server
+    state: restarted
+  listen: "restart llama-server"
+
+- name: Restart Hermes on carousel
+  ansible.builtin.systemd:
+    name: "{{ item }}"
+    state: restarted
+    scope: user
+  loop:
+    - hermes-gateway
+    - hermes-dashboard
+  become: true
+  become_user: wed
+  delegate_to: carousel-of-progress
+  listen: "restart hermes"
+  ignore_errors: true
+
+- name: Restart nvidia-gpu-exporter
+  ansible.builtin.systemd:
+    name: nvidia-gpu-exporter
+    state: restarted
+  listen: "restart nvidia-gpu-exporter"
--- a/ansible/roles/llm-inference/meta/main.yml
+++ b/ansible/roles/llm-inference/meta/main.yml
@@ -0,0 +1,14 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/meta/main.yml
+# ------------------------------------------------------------------------------
+galaxy_info:
+  role_name: llm_inference
+  author: rblundon
+  license: MIT
+  description: >
+    Deploys vLLM serving stack with NVIDIA RTX 3090 on Ubuntu 24.04.
+    Manages NVIDIA drivers, Python venv, model download, systemd service,
+    and Hermes provider integration.
+  min_ansible_version: "2.15"
+dependencies: []
--- a/ansible/roles/llm-inference/tasks/driver.yml
+++ b/ansible/roles/llm-inference/tasks/driver.yml
@@ -0,0 +1,25 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/driver.yml
+# DESCRIPTION: Phase 2 — NVIDIA driver.
+#              Installs nvidia-driver-595-open via apt. Fully idempotent —
+#              already installed on astro-orbiter on 2026-08-03, this is a no-op.
+#              DKMS builds the kernel module automatically on install.
+# ------------------------------------------------------------------------------
+
+- name: Install NVIDIA driver package
+  ansible.builtin.apt:
+    name: "{{ llm_nvidia_driver_package }}"
+    state: present
+    update_cache: false
+  notify: reload systemd
+
+- name: Verify nvidia-smi reports the GPU
+  ansible.builtin.command: nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader
+  register: nvidia_smi_out
+  changed_when: false
+  failed_when: nvidia_smi_out.rc != 0
+
+- name: Print nvidia-smi output
+  ansible.builtin.debug:
+    msg: "GPU detected: {{ nvidia_smi_out.stdout }}"
--- a/ansible/roles/llm-inference/tasks/foundation.yml
+++ b/ansible/roles/llm-inference/tasks/foundation.yml
@@ -0,0 +1,45 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/foundation.yml
+# DESCRIPTION: Phase 1 — Foundation.
+#              - Asserts vault secret is defined
+#              - Adds jarvis user to nvidia GPU groups
+#              - Creates HuggingFace cache directory
+#              - Creates venv parent directory
+# NOTE: NVIDIA driver install (Phase 2) already performed manually on
+#       2026-08-03 (nvidia-driver-595-open, DKMS built, nvidia-smi verified).
+#       Phase 2 tasks are idempotent and will no-op on astro-orbiter.
+# ------------------------------------------------------------------------------
+
+- name: Assert HuggingFace token is defined in vault
+  ansible.builtin.assert:
+    that:
+      - vault_hf_token is defined
+      - vault_hf_token | length > 0
+    fail_msg: >
+      vault_hf_token is not defined. Add it to group_vars/all/vault:
+        vault_hf_token: "hf_xxxxxxxxxxxxxxxxxxxx"
+
+- name: Add jarvis user to nvidia GPU groups
+  ansible.builtin.user:
+    name: jarvis
+    groups:
+      - video
+      - render
+    append: true
+
+- name: Create HuggingFace cache directory
+  ansible.builtin.file:
+    path: "{{ llm_hf_cache_dir }}"
+    state: directory
+    owner: jarvis
+    group: jarvis
+    mode: "0755"
+
+- name: Create venv parent directory
+  ansible.builtin.file:
+    path: "{{ llm_venv_path | dirname }}"
+    state: directory
+    owner: jarvis
+    group: jarvis
+    mode: "0755"
--- a/ansible/roles/llm-inference/tasks/integration.yml
+++ b/ansible/roles/llm-inference/tasks/integration.yml
@@ -0,0 +1,44 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/integration.yml
+# DESCRIPTION: Phase 6 — Wire astro-orbiter into Hermes as a secondary provider.
+#              Writes a provider config fragment to carousel-of-progress
+#              (the Hermes host) so FRIDAY crons can route to the local model.
+#
+# Hermes provider config lives at ~/.hermes/config.yaml on carousel.
+# This task uses the lineinfile/blockinfile approach to add the provider
+# entry idempotently without clobbering the existing config.
+#
+# NOTE: Hermes must be restarted on carousel after this task runs.
+#       Manual step — JARVIS will notify Ryan.
+# ------------------------------------------------------------------------------
+
+- name: Check if astro-orbiter provider already configured in Hermes
+  ansible.builtin.command:
+    cmd: grep -c "astro-orbiter" /home/wed/.hermes/config.yaml
+  register: provider_check
+  changed_when: false
+  failed_when: false
+  delegate_to: carousel-of-progress
+
+- name: Add astro-orbiter as Hermes secondary provider
+  ansible.builtin.blockinfile:
+    path: /home/wed/.hermes/config.yaml
+    marker: "# {mark} ANSIBLE MANAGED — astro-orbiter vLLM provider"
+    insertafter: "^providers:"
+    block: |
+      # astro-orbiter — local RTX 3090 vLLM inference
+      - name: astro-orbiter
+        type: openai-compatible
+        base_url: http://{{ hostvars['astro-orbiter']['ansible_host'] }}:{{ llm_serve_port }}/v1
+        model: {{ llm_hf_model }}
+        api_key: none
+  when: provider_check.stdout == "0"
+  delegate_to: carousel-of-progress
+  notify: restart hermes
+
+- name: Remind operator to restart Hermes on carousel
+  ansible.builtin.debug:
+    msg: >
+      Phase 6 complete. Hermes on carousel-of-progress has been updated.
+      Restart Hermes manually or via: systemctl --user restart hermes-gateway hermes-dashboard
--- a/ansible/roles/llm-inference/tasks/main.yml
+++ b/ansible/roles/llm-inference/tasks/main.yml
@@ -0,0 +1,36 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/main.yml
+# DESCRIPTION: Entry point — imports one task file per phase.
+#              Phases are additive; re-running the full playbook is always safe.
+#              Use --tags to run a specific phase subset:
+#                --tags foundation,driver,vllm,model,serve,integration,monitoring
+# ------------------------------------------------------------------------------
+
+# Phase 1 — Foundation
+- import_tasks: foundation.yml
+  tags: [foundation]
+
+# Phase 2 — Driver
+- import_tasks: driver.yml
+  tags: [driver]
+
+# Phase 3 — vLLM
+- import_tasks: vllm.yml
+  tags: [vllm]
+
+# Phase 4 — Model
+- import_tasks: model.yml
+  tags: [model]
+
+# Phase 5 — Serve
+- import_tasks: serve.yml
+  tags: [serve]
+
+# Phase 6 — Integration
+- import_tasks: integration.yml
+  tags: [integration]
+
+# Phase 7 — Monitoring
+- import_tasks: monitoring.yml
+  tags: [monitoring]
--- a/ansible/roles/llm-inference/tasks/model.yml
+++ b/ansible/roles/llm-inference/tasks/model.yml
@@ -0,0 +1,42 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/model.yml
+# DESCRIPTION: Phase 4 — HuggingFace login and Gemma 2 27B model download.
+#              Idempotent: snapshot_download skips files already present.
+#              Requires vault_hf_token and Gemma 2 licence accepted at
+#              huggingface.co/google/gemma-2-27b-it.
+# ------------------------------------------------------------------------------
+
+- name: Write HuggingFace token to ~/.cache/huggingface/token
+  ansible.builtin.copy:
+    content: "{{ vault_hf_token }}"
+    dest: "/home/{{ llm_venv_owner }}/.cache/huggingface/token"
+    owner: "{{ llm_venv_owner }}"
+    group: "{{ llm_venv_owner }}"
+    mode: "0600"
+  no_log: true
+
+- name: Download Gemma 2 27B model via snapshot_download
+  ansible.builtin.command:
+    cmd: >
+      {{ llm_venv_path }}/bin/python -c "
+      from huggingface_hub import snapshot_download
+      path = snapshot_download(
+          '{{ llm_hf_model }}',
+          cache_dir='{{ llm_hf_cache_dir }}',
+      )
+      print(path)
+      "
+    creates: "{{ llm_hf_cache_dir }}/models--{{ llm_hf_model | replace('/', '--') }}/snapshots"
+  become: true
+  become_user: "{{ llm_venv_owner }}"
+  environment:
+    HF_TOKEN: "{{ vault_hf_token }}"
+    HOME: "/home/{{ llm_venv_owner }}"
+  register: model_download
+  timeout: 3600
+  no_log: false
+
+- name: Print model download path
+  ansible.builtin.debug:
+    msg: "Model available at: {{ model_download.stdout | default('already present') }}"
--- a/ansible/roles/llm-inference/tasks/monitoring.yml
+++ b/ansible/roles/llm-inference/tasks/monitoring.yml
@@ -0,0 +1,131 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/monitoring.yml
+# DESCRIPTION: Phase 7 — Prometheus monitoring for the LLM inference stack.
+#              Deploys two metric-producing exporters on astro-orbiter:
+#
+#              1. node_exporter (port 9100) — system: CPU, RAM, disk, network
+#              2. nvidia_gpu_exporter (port 9835) — GPU: VRAM, temp, util, power
+#              3. llama-server built-in metrics (port 8000/metrics, enabled via
+#                 --metrics) — just needs a scrape job (no extra process)
+#
+#              GitOps note: the Prometheus scrape jobs for all three targets and
+#              the Grafana dashboard are declared in the homelab Git repo and
+#              applied by ArgoCD — NOT by this role:
+#                - cluster/applications/monitoring/values.yaml
+#                  (prometheus.prometheusSpec.additionalScrapeConfigs)
+#                - cluster/applications/monitoring/dashboards.yaml
+#                  (grafana-llm-inference-dashboard ConfigMap)
+#              This role's job is only to stand up the two exporters + verify
+#              they're reachable. Do NOT reintroduce kubectl patch/apply tasks
+#              here — cluster-facing changes go through Git commit + ArgoCD
+#              sync so state stays reproducible and self-healing.
+# ------------------------------------------------------------------------------
+
+# -----------------------------------------------------------------------
+# 1. node_exporter
+# -----------------------------------------------------------------------
+
+- name: Install prometheus-node-exporter
+  ansible.builtin.apt:
+    name: prometheus-node-exporter
+    state: present
+    update_cache: false
+
+- name: Enable and start node_exporter
+  ansible.builtin.systemd:
+    name: prometheus-node-exporter
+    state: started
+    enabled: true
+
+- name: Verify node_exporter is reachable
+  ansible.builtin.uri:
+    url: "http://localhost:9100/metrics"
+    status_code: 200
+  register: node_exporter_health
+  retries: 6
+  delay: 5
+  until: node_exporter_health.status == 200
+  changed_when: false
+
+# -----------------------------------------------------------------------
+# 2. nvidia_gpu_exporter (utkuozdemir/nvidia_gpu_exporter)
+#    Lightweight single-binary exporter — no CUDA dependency, uses nvidia-smi.
+# -----------------------------------------------------------------------
+
+- name: Create nvidia_gpu_exporter install directory
+  ansible.builtin.file:
+    path: /opt/nvidia_gpu_exporter
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Download nvidia_gpu_exporter binary
+  ansible.builtin.get_url:
+    url: "https://github.com/utkuozdemir/nvidia_gpu_exporter/releases/download/v{{ llm_gpu_exporter_version }}/nvidia_gpu_exporter_{{ llm_gpu_exporter_version }}_linux_x86_64.tar.gz"
+    dest: "/tmp/nvidia_gpu_exporter.tar.gz"
+    mode: "0644"
+  register: gpu_exporter_download
+
+- name: Extract nvidia_gpu_exporter binary
+  ansible.builtin.unarchive:
+    src: /tmp/nvidia_gpu_exporter.tar.gz
+    dest: /opt/nvidia_gpu_exporter
+    remote_src: true
+    creates: /opt/nvidia_gpu_exporter/nvidia_gpu_exporter
+
+- name: Deploy nvidia_gpu_exporter systemd service
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/nvidia-gpu-exporter.service
+    mode: "0644"
+    content: |
+      [Unit]
+      Description=NVIDIA GPU Prometheus Exporter
+      After=network.target nvidia-persistenced.service
+      Wants=nvidia-persistenced.service
+
+      [Service]
+      Type=simple
+      ExecStart=/opt/nvidia_gpu_exporter/nvidia_gpu_exporter \
+          --web.listen-address=:{{ llm_gpu_exporter_port }}
+      Restart=on-failure
+      RestartSec=5
+      StandardOutput=journal
+      StandardError=journal
+      SyslogIdentifier=nvidia-gpu-exporter
+
+      [Install]
+      WantedBy=multi-user.target
+  notify:
+    - reload systemd
+    - restart nvidia-gpu-exporter
+
+- name: Flush handlers before starting gpu exporter
+  ansible.builtin.meta: flush_handlers
+
+- name: Enable and start nvidia-gpu-exporter
+  ansible.builtin.systemd:
+    name: nvidia-gpu-exporter
+    state: started
+    enabled: true
+    daemon_reload: true
+
+- name: Verify nvidia_gpu_exporter is reachable
+  ansible.builtin.uri:
+    url: "http://localhost:{{ llm_gpu_exporter_port }}/metrics"
+    status_code: 200
+  register: gpu_exporter_health
+  retries: 6
+  delay: 5
+  until: gpu_exporter_health.status == 200
+  changed_when: false
+
+# -----------------------------------------------------------------------
+# 3. Prometheus scrape configs + Grafana dashboard
+#
+#    Intentionally NOT managed here. See file header: these are declared
+#    in cluster/applications/monitoring/{values.yaml,dashboards.yaml} in
+#    the homelab Git repo and rolled out by ArgoCD sync, keeping cluster
+#    state in Git rather than mutated imperatively from the control node.
+# -----------------------------------------------------------------------
--- a/ansible/roles/llm-inference/tasks/serve.yml
+++ b/ansible/roles/llm-inference/tasks/serve.yml
@@ -0,0 +1,151 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/serve.yml
+# DESCRIPTION: Phase 5 — llama-server (llama.cpp) serving Gemma 2 27B-it GGUF.
+#
+# WHY llama.cpp instead of vLLM:
+#   vLLM with bitsandbytes int4 quantizes on-the-fly — loads full bf16 weights
+#   (~54GB RAM peak) before compressing, killing the 40GB OptiPlex on warmup.
+#   llama.cpp loads the pre-quantized GGUF directly (~15.5GB peak RAM for Q4_K_M).
+#   No torch.compile, no warmup spike, OpenAI-compatible API on the same port.
+#
+# GGUF source: bartowski/gemma-2-27b-it-GGUF (Q4_K_M, 15.5GB)
+# Model downloaded to: {{ llm_gguf_path }}
+# ------------------------------------------------------------------------------
+
+- name: Install llama.cpp build dependencies
+  ansible.builtin.apt:
+    name:
+      - cmake
+      - build-essential
+      - libcurl4-openssl-dev
+    state: present
+    update_cache: false
+
+# NOTE: nvidia-driver-595-open provides the runtime driver only (nvidia-smi,
+# libcuda.so) — it does NOT ship nvcc/CUDA headers needed to build GGML_CUDA=ON.
+# Ubuntu 24.04's nvidia-cuda-toolkit (12.0.x) is sufficient to build llama.cpp
+# against; it does not need to match the 595 driver's CUDA 13.2 runtime version.
+- name: Install NVIDIA CUDA toolkit (nvcc) for building llama.cpp with CUDA support
+  ansible.builtin.apt:
+    name: nvidia-cuda-toolkit
+    state: present
+    update_cache: false
+  become: true
+
+- name: Clone llama.cpp repository
+  ansible.builtin.git:
+    repo: https://github.com/ggml-org/llama.cpp.git
+    dest: /opt/llama.cpp
+    depth: 1
+    update: false
+  become: true
+
+- name: Check for incomplete/stale llama.cpp CMake configuration
+  ansible.builtin.stat:
+    path: /opt/llama.cpp/build/Makefile
+  register: llama_cmake_generated
+
+- name: Remove stale llama.cpp build dir if CMake configure never completed
+  ansible.builtin.file:
+    path: /opt/llama.cpp/build
+    state: absent
+  become: true
+  when:
+    - not llama_cmake_generated.stat.exists
+    - not (ansible_check_mode | default(false))
+
+- name: Build llama.cpp with CUDA support
+  ansible.builtin.command:
+    cmd: cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
+    chdir: /opt/llama.cpp
+    creates: /opt/llama.cpp/build/CMakeCache.txt
+  become: true
+
+- name: Compile llama.cpp (parallel build)
+  ansible.builtin.command:
+    cmd: cmake --build build --config Release --parallel {{ ansible_processor_vcpus }}
+    chdir: /opt/llama.cpp
+    creates: /opt/llama.cpp/build/bin/llama-server
+  become: true
+  timeout: 600
+
+- name: Create GGUF model directory
+  ansible.builtin.file:
+    path: "{{ llm_gguf_dir }}"
+    state: directory
+    owner: "{{ llm_venv_owner }}"
+    group: "{{ llm_venv_owner }}"
+    mode: "0755"
+
+- name: Check whether GGUF already exists (avoid re-downloading 16.6GB on every run)
+  ansible.builtin.stat:
+    path: "{{ llm_gguf_path }}"
+  register: llm_gguf_stat
+
+- name: Download Gemma 2 27B Q4_K_M GGUF from HuggingFace
+  ansible.builtin.get_url:
+    url: "https://huggingface.co/bartowski/gemma-2-27b-it-GGUF/resolve/main/gemma-2-27b-it-Q4_K_M.gguf"
+    dest: "{{ llm_gguf_path }}"
+    headers:
+      Authorization: "Bearer {{ vault_hf_token }}"
+    owner: "{{ llm_venv_owner }}"
+    group: "{{ llm_venv_owner }}"
+    mode: "0644"
+    timeout: 7200
+    force: false
+  become: true
+  no_log: true
+  # Idempotency: skip entirely once the file exists and is reasonably sized
+  # (the finished GGUF is ~16.6GB; guard against a truncated partial download
+  # being mistaken for complete by only trusting files > 15GB).
+  when: not llm_gguf_stat.stat.exists or (llm_gguf_stat.stat.size | int) < 15000000000
+
+- name: Disable and stop vllm-serve if present
+  ansible.builtin.systemd:
+    name: vllm-serve
+    state: stopped
+    enabled: false
+  failed_when: false
+  notify: reload systemd
+
+- name: Deploy llama-server systemd service unit
+  ansible.builtin.template:
+    src: llama-server.service.j2
+    dest: /etc/systemd/system/llama-server.service
+    owner: root
+    group: root
+    mode: "0644"
+  notify:
+    - reload systemd
+    - restart llama-server
+
+- name: Flush handlers to reload systemd before enabling service
+  ansible.builtin.meta: flush_handlers
+
+- name: Enable and start llama-server
+  ansible.builtin.systemd:
+    name: llama-server
+    state: started
+    enabled: true
+    daemon_reload: true
+
+- name: Wait for llama-server API to become available (model load ~30s)
+  ansible.builtin.uri:
+    url: "http://localhost:{{ llm_serve_port }}/health"
+    status_code: 200
+  register: llama_health
+  retries: 18
+  delay: 10
+  until: llama_health.status == 200
+
+- name: Smoke-test — list available models
+  ansible.builtin.uri:
+    url: "http://localhost:{{ llm_serve_port }}/v1/models"
+    status_code: 200
+    return_content: true
+  register: llama_models
+
+- name: Print available models
+  ansible.builtin.debug:
+    msg: "llama-server serving: {{ llama_models.json.data | map(attribute='id') | list }}"
--- a/ansible/roles/llm-inference/tasks/vllm.yml
+++ b/ansible/roles/llm-inference/tasks/vllm.yml
@@ -0,0 +1,43 @@
+---
+# ------------------------------------------------------------------------------
+# FILE: roles/llm-inference/tasks/vllm.yml
+# DESCRIPTION: Phase 3 — Python venv + vLLM install.
+#              Idempotent: venv creation and pip install only run if the
+#              venv binary or vllm package is absent.
+#              Already completed manually on 2026-08-03 — will no-op.
+# ------------------------------------------------------------------------------
+
+- name: Create Python venv for vLLM
+  ansible.builtin.command:
+    cmd: python3 -m venv {{ llm_venv_path }}
+    creates: "{{ llm_venv_path }}/bin/python"
+  become: true
+  become_user: "{{ llm_venv_owner }}"
+
+- name: Upgrade pip inside venv
+  ansible.builtin.pip:
+    name: pip
+    state: latest
+    virtualenv: "{{ llm_venv_path }}"
+  become: true
+  become_user: "{{ llm_venv_owner }}"
+
+- name: Install vLLM and bitsandbytes
+  ansible.builtin.pip:
+    name:
+      - vllm
+      - bitsandbytes
+    state: present
+    virtualenv: "{{ llm_venv_path }}"
+  become: true
+  become_user: "{{ llm_venv_owner }}"
+
+- name: Verify vLLM is importable
+  ansible.builtin.command:
+    cmd: "{{ llm_venv_path }}/bin/python -c 'import vllm; print(vllm.__version__)'"
+  register: vllm_version
+  changed_when: false
+
+- name: Print vLLM version
+  ansible.builtin.debug:
+    msg: "vLLM version: {{ vllm_version.stdout }}"
--- a/ansible/roles/llm-inference/templates/llama-server.service.j2
+++ b/ansible/roles/llm-inference/templates/llama-server.service.j2
@@ -0,0 +1,32 @@
+[Unit]
+Description=llama-server — Gemma 2 27B-it Q4_K_M (OpenAI-compatible inference)
+After=network.target nvidia-persistenced.service
+Wants=nvidia-persistenced.service
+
+[Service]
+Type=simple
+User={{ llm_venv_owner }}
+Group={{ llm_venv_owner }}
+Environment="HOME=/home/{{ llm_venv_owner }}"
+ExecStart=/opt/llama.cpp/build/bin/llama-server \
+    --model {{ llm_gguf_path }} \
+    --host {{ llm_serve_host }} \
+    --port {{ llm_serve_port }} \
+    --ctx-size {{ llm_max_model_len }} \
+    --n-gpu-layers {{ llm_gpu_layers }} \
+    --parallel {{ llm_parallel_slots }} \
+    --metrics
+# NOTE: no --chat-template flag — llama-server auto-detects and uses the
+# GGUF's own embedded Jinja chat template (verified correct Gemma-2
+# start_of_turn/end_of_turn format for bartowski's gemma-2-27b-it-Q4_K_M).
+# The built-in "--chat-template gemma" name does NOT match this model's
+# expected format on this llama.cpp build and produced garbled completions.
+Restart=on-failure
+RestartSec=10
+TimeoutStartSec=120
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=llama-server
+
+[Install]
+WantedBy=multi-user.target
--- a/ansible/roles/llm-inference/templates/vllm-serve.service.j2
+++ b/ansible/roles/llm-inference/templates/vllm-serve.service.j2
@@ -0,0 +1,29 @@
+[Unit]
+Description=vLLM inference server — {{ llm_hf_model }}
+After=network.target nvidia-persistenced.service
+Wants=nvidia-persistenced.service
+
+[Service]
+Type=simple
+User={{ llm_venv_owner }}
+Group={{ llm_venv_owner }}
+Environment="HF_TOKEN={{ vault_hf_token }}"
+Environment="HOME=/home/{{ llm_venv_owner }}"
+Environment="HF_HUB_CACHE={{ llm_hf_cache_dir }}"
+ExecStart={{ llm_venv_path }}/bin/python -m vllm.entrypoints.openai.api_server \
+    --model {{ llm_hf_model }} \
+    --host {{ llm_serve_host }} \
+    --port {{ llm_serve_port }} \
+    --quantization {{ llm_quantization }} \
+    --gpu-memory-utilization {{ llm_gpu_memory_utilization }} \
+    --max-model-len {{ llm_max_model_len }} \
+    --enable-prefix-caching
+Restart=on-failure
+RestartSec=10
+TimeoutStartSec=300
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=vllm-serve
+
+[Install]
+WantedBy=multi-user.target
--- a/cluster/applications/monitoring/dashboards.yaml
+++ b/cluster/applications/monitoring/dashboards.yaml
@@ -9198,4 +9198,231 @@ data:
      "uid": "nginx",
      "version": 1,
      "description": "Ingress-nginx supports a rich collection of prometheus metrics. If you have prometheus and grafana installed on your cluster then prometheus will already be scraping this data due to the scrape annotation on the deployment."
+    }
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-llm-inference-dashboard
+  namespace: monitoring
+  labels:
+    grafana_dashboard: "1"
+data:
+  llm-inference-astro-orbiter.json: |
+    {
+      "title": "LLM Inference — astro-orbiter",
+      "uid": "llm-astro-orbiter",
+      "timezone": "browser",
+      "refresh": "30s",
+      "schemaVersion": 38,
+      "tags": ["llm", "gpu", "llama-server", "astro-orbiter"],
+      "time": { "from": "now-1h", "to": "now" },
+      "panels": [
+        {
+          "id": 1,
+          "title": "GPU Utilization %",
+          "type": "timeseries",
+          "gridPos": { "x": 0, "y": 0, "w": 8, "h": 8 },
+          "fieldConfig": {
+            "defaults": {
+              "unit": "percent",
+              "min": 0, "max": 100,
+              "color": { "mode": "palette-classic" },
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  { "color": "green", "value": null },
+                  { "color": "yellow", "value": 70 },
+                  { "color": "red", "value": 90 }
+                ]
+              }
+            }
+          },
+          "targets": [
+            {
+              "expr": "nvidia_smi_utilization_gpu_ratio{hostname=\"astro-orbiter\"} * 100",
+              "legendFormat": "GPU Util"
+            }
+          ]
+        },
+        {
+          "id": 2,
+          "title": "GPU VRAM Used",
+          "type": "timeseries",
+          "gridPos": { "x": 8, "y": 0, "w": 8, "h": 8 },
+          "fieldConfig": {
+            "defaults": {
+              "unit": "bytes",
+              "color": { "mode": "palette-classic" }
+            }
+          },
+          "targets": [
+            {
+              "expr": "nvidia_smi_memory_used_bytes{hostname=\"astro-orbiter\"}",
+              "legendFormat": "VRAM Used"
+            },
+            {
+              "expr": "nvidia_smi_memory_total_bytes{hostname=\"astro-orbiter\"}",
+              "legendFormat": "VRAM Total"
+            }
+          ]
+        },
+        {
+          "id": 3,
+          "title": "GPU Temperature",
+          "type": "gauge",
+          "gridPos": { "x": 16, "y": 0, "w": 8, "h": 8 },
+          "fieldConfig": {
+            "defaults": {
+              "unit": "celsius",
+              "min": 0, "max": 100,
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  { "color": "green", "value": null },
+                  { "color": "yellow", "value": 70 },
+                  { "color": "red", "value": 85 }
+                ]
+              }
+            }
+          },
+          "targets": [
+            {
+              "expr": "nvidia_smi_temperature_gpu{hostname=\"astro-orbiter\"}",
+              "legendFormat": "GPU Temp"
+            }
+          ]
+        },
+        {
+          "id": 4,
+          "title": "GPU Power Draw",
+          "type": "timeseries",
+          "gridPos": { "x": 0, "y": 8, "w": 8, "h": 8 },
+          "fieldConfig": {
+            "defaults": {
+              "unit": "watt",
+              "color": { "mode": "palette-classic" }
+            }
+          },
+          "targets": [
+            {
+              "expr": "nvidia_smi_power_draw_watts{hostname=\"astro-orbiter\"}",
+              "legendFormat": "Power Draw"
+            },
+            {
+              "expr": "nvidia_smi_power_limit_watts{hostname=\"astro-orbiter\"}",
+              "legendFormat": "Power Limit"
+            }
+          ]
+        },
+        {
+          "id": 5,
+          "title": "llama-server — Requests per Second",
+          "type": "timeseries",
+          "gridPos": { "x": 8, "y": 8, "w": 8, "h": 8 },
+          "fieldConfig": {
+            "defaults": {
+              "unit": "reqps",
+              "color": { "mode": "palette-classic" }
+            }
+          },
+          "targets": [
+            {
+              "expr": "rate(llamacpp:tokens_predicted_total{hostname=\"astro-orbiter\"}[1m])",
+              "legendFormat": "Tokens/s (predicted)"
+            },
+            {
+              "expr": "rate(llamacpp:prompt_tokens_total{hostname=\"astro-orbiter\"}[1m])",
+              "legendFormat": "Tokens/s (prompt)"
+            }
+          ]
+        },
+        {
+          "id": 6,
+          "title": "llama-server — Requests Processing/Deferred",
+          "type": "timeseries",
+          "gridPos": { "x": 16, "y": 8, "w": 8, "h": 8 },
+          "fieldConfig": {
+            "defaults": {
+              "unit": "short",
+              "color": { "mode": "palette-classic" }
+            }
+          },
+          "targets": [
+            {
+              "expr": "llamacpp:requests_processing{hostname=\"astro-orbiter\"}",
+              "legendFormat": "Processing"
+            },
+            {
+              "expr": "llamacpp:requests_deferred{hostname=\"astro-orbiter\"}",
+              "legendFormat": "Deferred"
+            }
+          ]
+        },
+        {
+          "id": 9,
+          "title": "System — CPU Usage %",
+          "type": "timeseries",
+          "gridPos": { "x": 0, "y": 16, "w": 8, "h": 7 },
+          "fieldConfig": {
+            "defaults": {
+              "unit": "percent",
+              "min": 0, "max": 100
+            }
+          },
+          "targets": [
+            {
+              "expr": "100 - (avg by(instance) (rate(node_cpu_seconds_total{mode=\"idle\",instance=~\"10.1.71.130:.*\"}[1m])) * 100)",
+              "legendFormat": "CPU Used"
+            }
+          ]
+        },
+        {
+          "id": 10,
+          "title": "System — Memory Usage",
+          "type": "timeseries",
+          "gridPos": { "x": 8, "y": 16, "w": 8, "h": 7 },
+          "fieldConfig": {
+            "defaults": {
+              "unit": "bytes"
+            }
+          },
+          "targets": [
+            {
+              "expr": "node_memory_MemTotal_bytes{instance=~\"10.1.71.130:.*\"} - node_memory_MemAvailable_bytes{instance=~\"10.1.71.130:.*\"}",
+              "legendFormat": "Used"
+            },
+            {
+              "expr": "node_memory_MemTotal_bytes{instance=~\"10.1.71.130:.*\"}",
+              "legendFormat": "Total"
+            }
+          ]
+        },
+        {
+          "id": 11,
+          "title": "System — Disk Usage (root)",
+          "type": "gauge",
+          "gridPos": { "x": 16, "y": 16, "w": 8, "h": 7 },
+          "fieldConfig": {
+            "defaults": {
+              "unit": "percentunit",
+              "min": 0, "max": 1,
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  { "color": "green", "value": null },
+                  { "color": "yellow", "value": 0.75 },
+                  { "color": "red", "value": 0.90 }
+                ]
+              }
+            }
+          },
+          "targets": [
+            {
+              "expr": "1 - (node_filesystem_avail_bytes{instance=~\"10.1.71.130:.*\",mountpoint=\"/\"} / node_filesystem_size_bytes{instance=~\"10.1.71.130:.*\",mountpoint=\"/\"})",
+              "legendFormat": "Root disk"
+            }
+          ]
+        }
+      ]
    }
--- a/cluster/applications/monitoring/values.yaml
+++ b/cluster/applications/monitoring/values.yaml
@@ -212,6 +212,36 @@ prometheus:
      #         array: utilidor
      #         array_type: physical

+      # astro-orbiter — LLM inference host (Ryzen 7 5800XT / RTX 3090)
+      # Managed by roles/llm-inference (Phase monitoring). Three targets:
+      # node (system), gpu (nvidia_gpu_exporter), llama-server (inference metrics)
+      - job_name: node-astro-orbiter
+        scrape_interval: 30s
+        static_configs:
+          - targets:
+              - 10.1.71.130:9100
+            labels:
+              hostname: astro-orbiter
+
+      - job_name: gpu-astro-orbiter
+        scrape_interval: 15s
+        static_configs:
+          - targets:
+              - 10.1.71.130:9835
+            labels:
+              hostname: astro-orbiter
+              gpu: rtx3090
+
+      - job_name: llama-server-astro-orbiter
+        scrape_interval: 15s
+        metrics_path: /metrics
+        static_configs:
+          - targets:
+              - 10.1.71.130:8000
+            labels:
+              hostname: astro-orbiter
+              model: bartowski/gemma-2-27b-it-GGUF
+
 # ─── Grafana ──────────────────────────────────────────────────────────────────
 grafana:
  enabled: true
Author	SHA1	Message	Date
Hermes Agent service account	aff792a061	feat(llm-inference): move astro-orbiter monitoring to GitOps (values.yaml + dashboards.yaml) - Prometheus scrape configs for node/gpu/llama-server exporters on astro-orbiter now declared in cluster/applications/monitoring/values.yaml (additionalScrapeConfigs), applied via ArgoCD sync instead of an imperative kubectl secret patch from the Ansible role. - Grafana dashboard for astro-orbiter LLM inference added as a ConfigMap in cluster/applications/monitoring/dashboards.yaml (grafana_dashboard=1 sidecar label), replacing the role's ad-hoc kubectl apply of a rendered Jinja template. - ansible/roles/llm-inference/tasks/monitoring.yml: removed the kubectl get/patch/apply tasks and orphaned grafana-llm-dashboard.json.j2 template; role now only stands up node_exporter + nvidia_gpu_exporter and verifies they're reachable — cluster-facing config lives in Git. - host_vars/vars.yml + inventory.yml: finalize astro-orbiter as the llama.cpp/RTX 3090 host (jarvis user, ssh key), drop stale Ollama/AMD vars and ollama_server inventory group superseded by the ATX rebuild.	2026-08-05 09:43:54 -05:00
Hermes Agent service account	aa8e229e64	fix(llm-inference): switch serve phase from vLLM+bitsandbytes to llama.cpp+GGUF bitsandbytes peak RAM ~54GB (bf16 load before quantize) — kills 40GB OptiPlex. llama.cpp Q4_K_M GGUF loads pre-quantized: peak RAM ~15.5GB, fits cleanly. Changes: - serve.yml: build llama.cpp with CUDA, download Q4_K_M GGUF from bartowski, disable vllm-serve, deploy llama-server.service - llama-server.service.j2: OpenAI-compatible server on same port 8000, --n-gpu-layers 99 (full GPU offload), --parallel 4, gemma chat template - defaults: llm_gguf_dir, llm_gguf_path, llm_gpu_layers, llm_parallel_slots - handlers: restart llama-server, vllm-serve failed_when=false (may not exist) GGUF: bartowski/gemma-2-27b-it-Q4_K_M.gguf (15.5GB, 24GB VRAM fits w/ ~8GB headroom)	2026-08-03 12:37:03 -05:00
Hermes Agent service account	22a020e4c7	fix(llm-inference): bitsandbytes int4 OOM — pending switch to llama.cpp+GGUF bitsandbytes quantizes on-the-fly: loads full bf16 weights (~54GB RAM peak) before compressing to int4. Kills the 40GB OptiPlex on torch.compile warmup. Fix in next commit: switch serve phase to llama.cpp + GGUF Q4_K_M. Pre-quantized weights load directly — peak RAM ~16GB, no compile overhead.	2026-08-03 12:35:46 -05:00
Hermes Agent service account	e879cf73d3	fix(llm-inference): gpu_exporter version 1.2.2 → 1.13.1 (correct release tag)	2026-08-03 11:57:42 -05:00
Hermes Agent service account	423891001c	feat(llm-inference): Phase 7 — Prometheus monitoring + Grafana dashboard - Phase 7 task file: monitoring.yml - node_exporter (port 9100) via apt, systemd managed - nvidia_gpu_exporter v1.2.2 (port 9835) — GPU util, VRAM, temp, power - Patches kube-prometheus additionalScrapeConfigs secret with 3 new jobs: node-astro-orbiter, gpu-astro-orbiter, vllm-astro-orbiter - Deploys Grafana dashboard ConfigMap via kubectl apply - Grafana dashboard (11 panels): - Row 1: GPU util %, VRAM used, GPU temp gauge - Row 2: GPU power draw, vLLM token throughput, request queue depth - Row 3: vLLM e2e latency p50/p95/p99, KV cache utilization % - Row 4: System CPU %, memory, root disk gauge - defaults/main.yml: llm_gpu_exporter_version, llm_gpu_exporter_port - handlers/main.yml: restart nvidia-gpu-exporter	2026-08-03 11:53:36 -05:00
Hermes Agent service account	dda6b91330	feat(llm-inference): Day 1 playbook for RTX 3090 vLLM stack on astro-orbiter - nvidia-driver-595-open (already installed 2026-08-03, idempotent) - Python venv + vLLM 0.26.0 (already installed, idempotent) - Gemma 2 27B model download via HuggingFace hub - systemd vllm-serve.service on port 8000 - Hermes provider integration on carousel-of-progress - vault_hf_token added to group_vars/all/vault - ansible.cfg: vault_password_file set to absolute path - inventory: astro_orbiter group added Run with: env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference.yml	2026-08-03 11:51:34 -05:00