feat(jmri): headless JMRI server with Leviton layout power monitor and X11 GUI mode
- Stable udev device symlinks (/dev/jmri/nce, /dev/jmri/loconet, /dev/jmri/lcc) - jmri-monitor: polls Leviton Decora Smart switch to start/stop JMRI automatically - Quiet hours 1-10 AM (no polling) - 30s off-delay before shutdown - LCRR config cloned from Gitea (ssh://gitea.mk-labs.cloud:2221/rblundon/LCRR.git) - ~/.jmri symlinked to LCRR repo for GitOps config management - jmri-gui: X11 remote GUI access (PanelPro/DecoderPro) via ssh -X as jmri user - Stops daemon, launches GUI, restarts daemon on exit if layout still on - jmri user gets login shell + SSH key for GUI sessions - Full JRE installed (openjdk-21-jre) for AWT/X11 support
This commit is contained in:
160
COUCHDB-ERLANGCOOKIE-FIX.md
Normal file
160
COUCHDB-ERLANGCOOKIE-FIX.md
Normal file
@@ -0,0 +1,160 @@
|
||||
# CouchDB erlangCookie Fix - Implementation Guide
|
||||
|
||||
## Summary
|
||||
|
||||
**Problem**: CouchDB deployment fails because `erlangCookie` is missing from the ExternalSecret configuration.
|
||||
|
||||
**Decision**: Externalize `erlangCookie` to 1Password (pragmatic approach)
|
||||
|
||||
**Rationale**:
|
||||
- ExternalSecret architecture requires ownership of the entire secret
|
||||
- Mixing externalized and chart-generated fields in the same secret is not supported
|
||||
- Single-node deployment makes erlangCookie rotation unnecessary
|
||||
- This is an acceptable deviation from the pure Harbor pattern given the architectural constraints
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### 1. Generate erlangCookie Value
|
||||
|
||||
```bash
|
||||
openssl rand -hex 20
|
||||
```
|
||||
|
||||
Example output: `f4e3c2b1a9d8e7f6c5b4a3d2e1f0a9b8c7d6e5f4`
|
||||
|
||||
### 2. Add to 1Password
|
||||
|
||||
- **Vault**: `mk-labs`
|
||||
- **Item**: `couchdb`
|
||||
- **Field Name**: `erlang-cookie`
|
||||
- **Field Type**: password (concealed)
|
||||
- **Value**: `<paste generated value from step 1>`
|
||||
|
||||
### 3. Update ExternalSecret Configuration
|
||||
|
||||
File: `cluster/applications/couchdb/externalsecret.yaml`
|
||||
|
||||
```yaml
|
||||
apiVersion: external-secrets.io/v1beta1
|
||||
kind: ExternalSecret
|
||||
metadata:
|
||||
name: couchdb-credentials
|
||||
namespace: couchdb
|
||||
labels:
|
||||
app.kubernetes.io/name: couchdb
|
||||
app.kubernetes.io/part-of: mk-labs
|
||||
spec:
|
||||
refreshInterval: 1h
|
||||
secretStoreRef:
|
||||
kind: ClusterSecretStore
|
||||
name: onepassword-connect
|
||||
target:
|
||||
name: couchdb-admin
|
||||
creationPolicy: Owner
|
||||
template:
|
||||
engineVersion: v2
|
||||
data:
|
||||
adminUsername: "admin"
|
||||
adminPassword: "{{ .adminPassword }}"
|
||||
cookieAuthSecret: "{{ .cookieAuthSecret }}"
|
||||
erlangCookie: "{{ .erlangCookie }}" # ← ADD THIS LINE
|
||||
data:
|
||||
- secretKey: adminPassword
|
||||
remoteRef:
|
||||
key: couchdb
|
||||
property: admin-password
|
||||
- secretKey: cookieAuthSecret
|
||||
remoteRef:
|
||||
key: couchdb
|
||||
property: cookie-auth-secret
|
||||
- secretKey: erlangCookie # ← ADD THIS BLOCK
|
||||
remoteRef:
|
||||
key: couchdb
|
||||
property: erlang-cookie
|
||||
```
|
||||
|
||||
### 4. Update values.yaml Documentation (Optional)
|
||||
|
||||
File: `cluster/applications/couchdb/values.yaml`
|
||||
|
||||
Update the comment block at line 9-10:
|
||||
|
||||
```yaml
|
||||
# Admin credentials managed via ExternalSecret
|
||||
# See externalsecret.yaml for 1Password integration
|
||||
#
|
||||
# NOTE: erlangCookie is externalized to 1Password for architectural
|
||||
# simplicity (ExternalSecret ownership model). In a pure Harbor pattern,
|
||||
# this would be chart-generated, but single-node deployment makes this
|
||||
# acceptable. The erlangCookie is treated as an immutable infrastructure
|
||||
# secret (generate once, never rotate).
|
||||
createAdminSecret: false
|
||||
extraSecretName: "couchdb-admin"
|
||||
```
|
||||
|
||||
### 5. Commit and Push
|
||||
|
||||
```bash
|
||||
cd ~/git/homelab
|
||||
git add cluster/applications/couchdb/externalsecret.yaml
|
||||
git add cluster/applications/couchdb/values.yaml # if modified
|
||||
git commit -m "fix(couchdb): add erlangCookie to ExternalSecret from 1Password"
|
||||
git push origin main
|
||||
```
|
||||
|
||||
### 6. Verify Deployment
|
||||
|
||||
```bash
|
||||
# Watch ExternalSecret sync
|
||||
kubectl get externalsecret -n couchdb couchdb-credentials -w
|
||||
# Wait for: SecretSynced
|
||||
|
||||
# Verify secret created with all four keys
|
||||
kubectl get secret -n couchdb couchdb-admin -o yaml
|
||||
# Should contain: adminUsername, adminPassword, cookieAuthSecret, erlangCookie
|
||||
|
||||
# Watch ArgoCD sync
|
||||
kubectl get application -n argocd couchdb -w
|
||||
# Wait for: Healthy/Synced
|
||||
|
||||
# Watch pod startup
|
||||
kubectl get pods -n couchdb -w
|
||||
# Wait for: Running
|
||||
|
||||
# Test CouchDB access
|
||||
kubectl port-forward -n couchdb svc/couchdb-svc-couchdb 5984:5984 &
|
||||
curl http://localhost:5984/
|
||||
# Expected: {"couchdb":"Welcome","version":"3.5.1"}
|
||||
```
|
||||
|
||||
## Why Not Follow Harbor Pattern Exactly?
|
||||
|
||||
**Harbor Pattern**: Only user-facing credentials externalized, internal secrets chart-generated.
|
||||
|
||||
**CouchDB Constraint**: ExternalSecret uses `creationPolicy: Owner`, which takes full ownership of the target secret. This prevents the Helm chart from adding auto-generated fields to the same secret.
|
||||
|
||||
**Options Considered**:
|
||||
1. ✅ **Externalize erlangCookie** (SELECTED) - Works with current architecture
|
||||
2. ❌ Chart auto-generation - Conflicts with ExternalSecret ownership
|
||||
3. ❌ Dual-secret approach - Requires Helm chart customization
|
||||
4. ❌ Disable ExternalSecret - Loses 1Password integration for admin password
|
||||
|
||||
**Decision**: Pragmatic approach wins. erlangCookie is treated as an infrastructure secret (generate once, never rotate), which is acceptable for a single-node deployment.
|
||||
|
||||
## Secret Classification
|
||||
|
||||
| Secret | Type | 1Password? | Rationale |
|
||||
|------------------|---------------|------------|------------------------------------|
|
||||
| adminUsername | User-facing | No* | Static value, hardcoded in template |
|
||||
| adminPassword | User-facing | ✅ YES | User login credential |
|
||||
| cookieAuthSecret | Gray area | ✅ YES | Session security, periodic rotation |
|
||||
| erlangCookie | Internal | ✅ YES** | Architectural constraint |
|
||||
|
||||
\* Hardcoded in ExternalSecret template (not fetched from 1Password)
|
||||
\*\* Pragmatic deviation from Harbor pattern due to ExternalSecret architecture
|
||||
|
||||
## References
|
||||
|
||||
- Full analysis: `/home/hermes/couchdb-erlangcookie-analysis.txt`
|
||||
- Harbor pattern: `/home/hermes/harbor-simplification-complete.txt`
|
||||
- CouchDB Helm chart: `apache/couchdb` v4.6.3
|
||||
@@ -39,3 +39,7 @@ step_ca_principal_mappings:
|
||||
- ryan.blundon@protonmail.com
|
||||
- ryan.blundon
|
||||
- ryanblundon
|
||||
|
||||
# Leviton My Leviton API
|
||||
leviton_email: "{{ vault_leviton_email }}"
|
||||
leviton_password: "{{ vault_leviton_password }}"
|
||||
|
||||
@@ -1,436 +1,441 @@
|
||||
$ANSIBLE_VAULT;1.1;AES256
|
||||
65653162653536373535643935613236366335356333336264346637323164633739633965656664
|
||||
3434656433323239366665356134386564643135663135360a653232636463656332343565313763
|
||||
61326635636333393730356131356466363063613734653565643766383431363166656532393934
|
||||
3631373264353563650a616362633165666239323563623161656238633737316236303133313034
|
||||
66353461643535373638616364663365643961656635643566613336633466343739386234313532
|
||||
38353066313763643266623739366261646633363962336239623830613265383335383763376534
|
||||
62653865626662383330656530336534366136313965373764353133326139623132646364396337
|
||||
33333366366434393037373361376538376334336265663162336239646237306332623039343032
|
||||
37333832383762663534643638666234623062373734626335323537616334303638376164656634
|
||||
64623962313064363166383930636136383935613237613434373733383836666532363265643739
|
||||
37376333363736326334383864633734393263623536393239393564643131616132373230356633
|
||||
61636165333537356632303034346431353139383065393832323236626331353236666263376666
|
||||
38306631653862343064656135396638636363613139373062653735366565376165353330636162
|
||||
38653530366263356437323433653437373362356166633235386330623333326264333961643434
|
||||
30323130613431353064393138303465316630383462653062326264393035393735396134396665
|
||||
36383963633938363662633035626538663366323461656136323934353731626264396364643337
|
||||
63356337343161366237353135313966616630316362396362343962356564643365626664633165
|
||||
62383765303436303239376365316632353463666664306164313933346162343833303963313835
|
||||
62353565333231646462383664356230623165363333653335343438346233316534333132366634
|
||||
31326138663433663132393761343335656636366237656337666331633062303363633938303536
|
||||
31303238323932336330363064323963306132633961313634616232636163353439363666623230
|
||||
65666632653730303134343032616331363932616532313365633038623135396463316666373637
|
||||
34343736383731363239303330306561356162623962396564323239376661363761336434376264
|
||||
34373037623331643666366165306161356365326261366566323233656538363061636637343164
|
||||
39323130613530386661343839623361303862323763373864626536376562643766643263376665
|
||||
64373334333436653061653962373437366437643034306633343830626230346532653565353135
|
||||
33326339656266616237643362333062346564333563326463393535356465666166626463643132
|
||||
36316263323761653333363632386261633432383431386435363362663734626634353761663035
|
||||
32383464396632396361623531303937383366633131613861646364303662343761323530303139
|
||||
39333866663164363431656233663064383633386261313063646435333666376465626532343636
|
||||
62386166306635353061623365313738306237616430386461343262613039346261353662346462
|
||||
65616261656534336631646434316336646337333661336530633834633037363834346539646466
|
||||
33346466643238343133326339393264613763626532633938613765373262393236643533383661
|
||||
33306233623130666161326365366164333434373463323662316432633263323761316338623065
|
||||
66306332656438303938393430633563303761303239346266386664333837613336316539643736
|
||||
31353234613664313061306433656165373737336166343930653837303532346163613564376661
|
||||
61376237383562353639393565303332633366376632386532333736303738333362353334353366
|
||||
36343166633533303834346265666536353633343564623062613030323934653863343062323136
|
||||
34323635363238623336653062666636646263343731316537346635346235613862623530326633
|
||||
30396338386233663335613661646132386230356631326331653962393336653937376264313931
|
||||
65363465343265643034363963656262666131393632653561626330396166386566396363663538
|
||||
66323935383166623038326437323730363266316530313938373261366363613731656638386634
|
||||
31316261616630306131633936326635333632643731313730303438656563343133373732376537
|
||||
38623330633539653034343364656261316336353466653663646333653636653266373363333732
|
||||
61333461346131366233336162363033623636386634613731303032393738333537393131316539
|
||||
65643238613432306436376432373364656530323331633762393566396531396466653463353832
|
||||
34393662616435343137373364663839613464346564663631343837366530313061326561656664
|
||||
33623163333864346661393634333830336133383337373833346132353531656236363661323338
|
||||
64313663356432373137343461333539386665643130316463336134396436623730666165623939
|
||||
33373832353932386238353866336130356530613766626235316332623132343735636335633437
|
||||
37303766386434373930383663386565383732636635646133393637383963353134633433616431
|
||||
33616464633431666361393931623631313731343863636234646130313136346639323662373236
|
||||
62623830633838326432326339383533373430346635643739646639633739343639636665376333
|
||||
30396238356161373734633562643466643036363239396538613465346238356366363730623965
|
||||
32336533303331363336666238363863313066376230353932323466316432653264346334373931
|
||||
61303066313831376233313261633336643265396139663037333031636666383738313763313366
|
||||
38386138363738353265613363353339323930343934393036343234626162353037326638636236
|
||||
66343634323136663134373737366433616632653737316534633232323237343566623361666131
|
||||
34393331373835613934316235336635666633646465373866356566656261373066653937316339
|
||||
37396366643564333130383237613933613736376161616231666139643030613338333133356336
|
||||
37343362363433653638373335363130313362306236393231646637383736663236376537303533
|
||||
31633230333933666131613764336532373633353132373839326562356438386236643537386231
|
||||
35616662643431383661343937613738393963373865383465306532336137373730653832633930
|
||||
37656661326434633730346364326661653937313561333437383064313933363231393164616464
|
||||
31333664386134376264376339303838656431633966663263303465376161633866376161313531
|
||||
62346266316533326639303661396164376666383864363262316630636561326366373363633063
|
||||
33666263333132323232346235633837363138663539366131303633313662326664373332353664
|
||||
39306235313732376164306164386534346666633037316334366431363063663137356464656234
|
||||
35656164303933306661373932663831346531636530386139373837633263396632643235383361
|
||||
65393939303338656537353130316361366139363933363031393735666336323931373632303663
|
||||
63333137663265353962653861316365643239396562333933383964663730663230353331386638
|
||||
62396435323162663036303334323936633436626136663537376562363430376239356136343261
|
||||
36323062363935643562326161643065333437343331613135383932333835396565646631646437
|
||||
39633035326662653334616366323736323032616264623166326563326361363263623736623739
|
||||
62633762346135626666636463343132303433346330303631656634646537616537363764626564
|
||||
30316539303964626635613338623261613430656432376265386636613566313732386561653564
|
||||
66363538643463653233373433353965623765303038333931623036666435303330633136663966
|
||||
31313964323439376637613530343635376166333765306466353337623633343539313330373437
|
||||
64343930636465636265383261633664313762303533326234626461623563636466643039343336
|
||||
62346462343639323862353036326335383735323337396465613737336661633465363330363161
|
||||
38323561613935303334303261323032616332333231363762303831373961323033626663383637
|
||||
35323134666636353766313330316231323938326532333030383638646230643735653663346634
|
||||
38363261626663393139313031633032343038396433353563363531306232386539303734323463
|
||||
63313938373735383631373665613333303530383335363262306230326665666535333564346163
|
||||
39336132643464666538356461393330616264383632623037623634336530376164336336323336
|
||||
61646165333835333961323439663864386562326131653564623861386336303934346263643036
|
||||
32666632623537376537646634313839363038666336386238343531323664396461633030373334
|
||||
35396463373339633931383636353062396562383763373939633336346463623733303039663733
|
||||
31663834326661623064323132353431366362636466623332363532316435386333636562633562
|
||||
36613465363936623462316136643664336261353938653362393938666266663330323163653338
|
||||
63333732616330323438633038366465353865353965663333376466613834633166303761633361
|
||||
37613863313230653235653034613137303231343961616330316664653333303436356561353562
|
||||
37623839646536363036613964336230373537653130366330633137356661366561386566323165
|
||||
38653633373538633762643935346661363961663265616433383832666635666331366635636637
|
||||
32336532373164373135363533333539623739383330646431356363636430326433376364393265
|
||||
35643561613033623231366362393737663964396266313232666362326436303331653764366436
|
||||
30363135663234356363613065643763353836626133656334653138393562393032623631343130
|
||||
30303936656636366465633163323061333863393532366563316233353466613566316563336539
|
||||
33326137383263393266393166336361656538353963313366353434333864623238333435653836
|
||||
39376138356461663262356665363835653638623031396539353132623737333736353937623533
|
||||
65386333396666633365313262323739633461383464386663313163646632646335646433663834
|
||||
34616663386661363135653765386534633339653232326133396332646136653230393431626637
|
||||
30366433336633643837363434383439343562366164353235653833306138643431316537343931
|
||||
34396535393432383163653231623837633961366437653434623164333333323563666330613866
|
||||
36393561656135303731666638393737653565303836326264373137303264656334653163366433
|
||||
35376362326133666336303732656561326164663832623732613830393063343231323363353366
|
||||
66636233366436363565336231663964303532343962333732313832323762343638356231343461
|
||||
65373062383564346266323362336666613735316161366136303032313039373464383532343932
|
||||
30396530373030303933353264363330353132376164303062623538613632323932373230363838
|
||||
32633766316533316665393033326161373361333166643436356430623262663032646262616633
|
||||
32643436646264343366343862356339626561346561353463616534306139383133656433636331
|
||||
32333735346163353434616530623235373037376664326266343933326439663965653865333766
|
||||
31653163643134346331313331633561643336663130353936323439636331383134646665656630
|
||||
63663866663938663632646664323936613434333731306264306633643335393566343564346537
|
||||
35326132303632373731373532623662373861356534343536613731366666373439343563663737
|
||||
64326339313533323936393362613564333065623731653930363264373061636665313135623062
|
||||
63373565306535373763356664333465343666626439626666393734303962303762393066643139
|
||||
62663334313736663132386232613061326636313664326462353361633362323464653038396662
|
||||
64363862356439633564313065363361313161643962316166643936346539626339396230333633
|
||||
39333132393135613264653363323233643032376266313630313735373966373966386239336635
|
||||
34643739363935353463373434343430633438346136306337393061646139346230636531376537
|
||||
36633861303264633662373365623261303635363836333163623864323338613739383134636237
|
||||
62326561356535626564633038643230646431613866643239646563323035353434313837663863
|
||||
66326530353265626361643738333266346339326532663166656432613631313137396463643864
|
||||
64343339373131363066303437633436306462316166356131366134626331343635323837643136
|
||||
39306461316230663134346563313662616532616239333735303061373138383363386232373062
|
||||
61643838366266613637336433323062336434313836376434636433306135303734376661376361
|
||||
38373438623331653165643837303733393436626166623762623531316164666163316661643761
|
||||
39616539653831336261333335643834663533363233333732643431363836386633663539386638
|
||||
39356336653438663061306230316464616533363036363434663335383530326432376632396338
|
||||
38386130386532333861353362313838383964663861663962353039613565636339346162383331
|
||||
32313337353430343130326466623666326263343438663463376134336131323735663261356263
|
||||
34613638303262366666643036353766653039393739303738366330643033316632616232393032
|
||||
36336161646434373332646663313834623064336437316666623663613861333838363731336564
|
||||
63636633623664353432323836306564616239343731316335333234336465363232326430643231
|
||||
30623063316130396531376237363363386465393838303463336439616333353338366436323634
|
||||
66626463326136343533343263303033333766306632303333376136643137353831323135666430
|
||||
61313663336363356531633866663366333433343433643664663665643335633035303034666334
|
||||
62653233616136656462343930636534636432353237363538356365636635373865353532643764
|
||||
32646431376165616235396133323162373330386664636336336266363961316462613263303131
|
||||
33303635306136303366353331653239306130616464303835363437313566363436663032353061
|
||||
32356639333061636536383238396665393232663562363662366563653831343037626535666432
|
||||
33353261623936313563323564643134666232333839386162663331326461393239623135383530
|
||||
31323861666631663933636338323239386336356339623738646430333136336635643562353263
|
||||
64313931656435383134353063383665646538306661396163653434373031326134336235646461
|
||||
39343734313965313365643135643166366133313465323366633038663333366535633532623966
|
||||
36666134316165613830323933333339336663363964323764313430613130653236336664393863
|
||||
64306437316235306565333238666164353738336539353064343161343834656262653666636663
|
||||
36353063326262653466303534616530616139636166343137666439336335363539393439633436
|
||||
36623963623839623834656431396162396562373330316438383739363637613366613163336138
|
||||
35373730383165316330366332373939643839336634613934653538303866343738656534653263
|
||||
33303361343237356430373266653062313037663230366531366363393937653439363732333036
|
||||
31343365633862623038316536363031356336383461396562626236313038313239623665316439
|
||||
30336137356438343362363536303234346637373833313231663333616263303233336161383130
|
||||
34316265613736383830363639366139313537313661633736303634326237643330663735356561
|
||||
65626136613831376432383363383731663763356432663264346561646535633966643462326232
|
||||
31623930303262616535356633663938323635643462663731366433623961623435303062616263
|
||||
63353836333235336362353039343431313262646137303364613838306339623930666466636132
|
||||
32656434623738333565636164303535393734346462353436313030396264623332393932653861
|
||||
36383533363030356664326332376662616139343065343932656262623334633334633462383635
|
||||
63666439343130623936376636633635613335363264363136643366636535613938373866356336
|
||||
35666632313565633630373039316262316339333035366332356138393261613836633437363136
|
||||
33653033343536623264366466356561323063313031633464353963383661316131643166663566
|
||||
36373937326333393164353234336433656336626134363236303032396531353862333535336466
|
||||
65393865333565386538386665373166383235633739383934643334653763366161396565616166
|
||||
61333666353666393230326134636661376237353937646236343034626135366535383631333035
|
||||
61346231386233326564656636363066396636393966353539666231646464323636303038656562
|
||||
31656238373331303031393034346630643263613365643232663861373336363662356265326164
|
||||
61323866663033636534313365376466303538666463663539623231313632303262326638633539
|
||||
62313239356365313932653735363662666266643765633138393037653662656566656130663133
|
||||
61373531316261646234666361313766626139613762306563343561363961656565666163313532
|
||||
35323837333737366138316164376537613834313135306461633735613362376233306134373537
|
||||
65323339393039336230373237386434333639623762643133363339303265623065326438623531
|
||||
65356632363237613937633132326231333938336639663438653861353761396434643136643238
|
||||
62363162666366653831626531353236623532656437313134343633626666383437363761626331
|
||||
35386431646564653233613632373435623665613335316164636562356264663033366639636439
|
||||
63356231393362663736366431616161633065303537366538393935363033303636653836346132
|
||||
30633036653836656433636631313863303132666531326563303266316566623934666361653932
|
||||
35623265613635343435373361353635343261656233356535303037383433353731313539613463
|
||||
64363334383666663639623735393934646366386437633638613034316366336337316561333731
|
||||
38363339386561373834333037626565623335613563323366656365643032373834616339663838
|
||||
64646534623432323363303032323666643636323536666239616238323932353165663535633465
|
||||
64633934303565396533313165393438373231376664616237373562306130363531393833353365
|
||||
63623033343762326437306536663734386365653131346235303837336236366665346234316163
|
||||
32616539666634653839633739303837646563313364333864343231306663383530313963386434
|
||||
34313965393731653963643336336134343764343065646436303739356530333761386465333333
|
||||
35653363653361366462636162663134333431323866306235343764333035373534656138643933
|
||||
30653962613339313833343532613939636332633236303733313135313932356136383439326635
|
||||
31386565343337333662663762356361623835353830333865613437336532623936343365343231
|
||||
33336137616266303835366662643737666333346433366663666462396531643463373562623237
|
||||
38383537666639363839653538303266303965613762373561643433376664313966326561653764
|
||||
66396566366265343962393633653765643563666634646637623161323538623961393039333036
|
||||
33363530653935386331396533666432626634616463663763396663353133393461623866383061
|
||||
64366132646132663263636131333031643736663031663336623065386266653638376639336334
|
||||
36653932643831303931613532333036323766376235303433633564303062623336646166303533
|
||||
63353063303033623032643333653733393332303466366538353062373666363438366432316131
|
||||
36613335396531663238343139313061613363393463363035366132353439353433646237646265
|
||||
37633964316466636337363331383565323866653036386435376530343661323565666539663230
|
||||
38386232376436623064613232636637313931316535643661306536343661653135313335616534
|
||||
34633761663066316430313361393063623033373636633133653536383764306433346235343761
|
||||
37393764633561653632653565383937383436666662336633633261653861323533303732613738
|
||||
64653739323636613639643465316437356163356238656139663034313738363532373861393333
|
||||
64613038386535396565656538663163313161353831306237313064666566336534646233393137
|
||||
37336265316361613430663961356361653636636235323439306537666363353361643664363631
|
||||
63643631336134323032636463393639613636643130313136326538376461363937643436633561
|
||||
39663433623663346462356437666230313937396132633834386661613436653839616136343234
|
||||
35333437386663353863306166393634363734323636333137343938353461386366326239396236
|
||||
65303939623437376535633463626166653935343033303066383064306332356230623566393062
|
||||
34666566353238653730613561383434343332333732356334623836616334393464373965633334
|
||||
37313564393463663862386535626338336235613132376530666635343832386231366533363562
|
||||
32626162626434366261623431373938646237643462623337653466653538653732393634366639
|
||||
62666231376239393066366336386166646631396632613631313663633435646337313465643231
|
||||
63356566333764333238326165613832643161346366396635663032623133386132666333343337
|
||||
63323637383035363833653162326231616461653062613765363165313638313234343266333334
|
||||
62383362343232363431613364383930376338303839633261656236636264353335333938333731
|
||||
63616666626437643432393438376661616262643939363239353836323036303139666338336261
|
||||
31306437353536623561333531346630643761336235373230373135326361393739613934653361
|
||||
38346532313364633638306665306563646433353038346238313530363163396166353535343537
|
||||
34343330373063383566333335386162326339383835373665363464616139383331646436343365
|
||||
30386564323733393461653662323334656637363566343737356239306462303762383332656239
|
||||
30643439343161636434656338636533396364643632383030646639356534343963666466613566
|
||||
39373833633265623563396363653830393661633636646265643362633731626462636531313362
|
||||
35366331383635643564316461633431646565393863373635323834363934336335653831613330
|
||||
39646262633636343832626362393730643163616565363765346266316636343065393630333866
|
||||
62333931336161636230326132393739616665646462653666346532356265643235333131343635
|
||||
31326638626162356333356261623531646464633139666539373261376230336661313937363036
|
||||
32313464316364623838343231346538303237393137363233366234373031346238303932383461
|
||||
66613537346136313262313337636565383639383863346535393361363930616535626264376565
|
||||
30303236663065363161663836356130363130363339643335653366623065373566353136353933
|
||||
39363761636130646537393166623335353431616462343364383535393661313738343265313138
|
||||
38323066333239616137623233323664616232323262666530373064366138316431643430643462
|
||||
32653262386565396335646431633534663031613536656561643434366464353335313239393464
|
||||
36333866343063656533623432306231343065343530343638653739343731336131626532376566
|
||||
37653833313233386538666464386232353466303035316232646339653533663831323261636330
|
||||
32646166366633656631393762626236356633393130386165643264663437666433336138353938
|
||||
37653866646364333134313366316366636132623764366562623932383239643265646461646434
|
||||
63656233316639616134666635373434323030643136626238646637386634333439616231336136
|
||||
38326431363062326232626238613334356633353362653138643237646339333762616330383934
|
||||
63613235376530373834333834316539343330313862666262633439333062633261646138626434
|
||||
33616162316632366537393939343163333530393939613936323164663034326536666133303365
|
||||
63643337343037363363663235316330316131353064383933323131643036623862303835336633
|
||||
39376134313264333834306330366136643434386566393466313835396466306363663565303065
|
||||
33393431636163376365396139346162346239343838616561373162643931623365336436626636
|
||||
39613030356162363931316166376231303033306366616162366239396333323838363465353261
|
||||
64303561386563383834386166323739626237376164326533616533343338316430366365333966
|
||||
31343233666365663830323030336161623466633261303637373537336333663262383238393833
|
||||
33383033343630663237653338636436346565356238313434346666336436653835326133313038
|
||||
65353530333839373136326132326439306431333334633933663065316531616263633533346431
|
||||
62383862653162613532383136363830313964366161306166343934363865316533643736373431
|
||||
61323135343030346336356232326138376564343131636263336332333239303733323338663830
|
||||
32373639343063353261623831623237663133646535636338353333653237653539656438666436
|
||||
63636537336636313762336531346531313637653834386535313666306636303338306465303764
|
||||
63366133396263636164386339343131653361363831363932303734333634343734636163383539
|
||||
33613430353236616132636431656636633561316161623661373663396462633930343865336236
|
||||
32373534666135303730303663333337643033346231646661393137643161383833626335336561
|
||||
66366433303436313132636230313531316261613564323963626438623530633634363265613939
|
||||
37666636313632353831343238646237666464333039306630343531626339323562346162663130
|
||||
65633133366564306230666436663630316434616634656661306461353866396662376331666630
|
||||
30323166386535633361316633303830623031393531343532633930616439326465613631613838
|
||||
38326630366534363036666433333237363636643366333161343336363936323730313339343132
|
||||
35356231666633386537366236303136663261336532386338316565343239333261613838623861
|
||||
31366531343232653431343436343766386331303133363534616536633337656433326265656161
|
||||
66356235616135363634313835666134346232356663326361396537303339373334323861383161
|
||||
33393438313061613465666536666636643166613463363165393338643066613231636138343537
|
||||
64333731376139353934316631393561356163366237386337393061343235633131613962383731
|
||||
66376261633832323864356564326532356363343762313563393331393933316336373434333237
|
||||
35396531303330396363333163373361643837383766333730303031613665313237383532666264
|
||||
65613237623136336239653566646366376538356362643062386436363466616331336665616432
|
||||
31383365363630663530643764633335336332386332663661393733343038626265383432306630
|
||||
38653861356636313634326266633637653764396233313463393964653739306234666662336432
|
||||
33393139343864346233653763343762616230383131353166396265653031326232656333626264
|
||||
63616339663564363665626338316661623230646534373231313662343736316462663763333364
|
||||
30333137386161336664386536366538313934313039333832323763313264373861666239643466
|
||||
33383334633736333634306262316265323634386635313764376532646364356565396538656137
|
||||
63316232643832383865303934636364313036666237636632393137373933656263646534623230
|
||||
65616231323035643531353430326630353761633234636234343834656664323161383335343235
|
||||
62616533366465303961306134386265303933616236346332623536393236633366333634316264
|
||||
31396361343833623131306266336238343866303361303336346566656639616134383063396235
|
||||
31626633333665373934333961646263663730653331326137383031383736646633356536333431
|
||||
39303632383030613465656236333763393737353865386235333830613665653363393536646630
|
||||
65353733613536653963393831636530636231366266343262376631393062623163313031623230
|
||||
62326561653165306339633439376461346330393231313366666339346536313765363163616630
|
||||
35373564366563656161376336323031346339313734633139383563393966373637616266666265
|
||||
34623330313265373935396130643231353131313163393333396337396666636439633035326635
|
||||
32663762333537663839346531663132613636366639643364373333343262326264366136363331
|
||||
61613833643639656632353931663830613634383334653036323462323262386265313637656535
|
||||
36333834306438633532616335663562383730306337656336663035643136386261326130336363
|
||||
32623330333764616565333162383234656463363432313039666265653563323232613930303765
|
||||
30623834666665636332303862663832303537656664643362643636616265613739356664646462
|
||||
61326466303266626166333730343330396439356663376362666536623539353666636230373533
|
||||
35363461326263366137313037323166396133646430616664343165623533393963623535626237
|
||||
39633362393636373131626364636437343361303435396138363735326235383237633964336138
|
||||
31643935646431386537643733353862666138333238323461623638356535316365653530376464
|
||||
33343465653134613536343563376564663133336532343330666131346663363434623238303862
|
||||
31306331633238623836346466616230356532316232323734356638343532643032633937653061
|
||||
37393265333233626563353963363637653338393964363832663734303566356633623733393164
|
||||
64383536343862393134336638353733373262396563303266346535346266306238616531336162
|
||||
35363638333431396163303530353461363338656363336366376430316464353131623661346366
|
||||
35313930353538343830333337356163613765356361343663613933366464353935336361636436
|
||||
65356465346535613231363932303339646162303238376236323461303062313336656366306563
|
||||
31353464333539343839353764393665393235653537313939373563303436626635663766303336
|
||||
35333838323734386537623334366235653534623664396664656264633735353634663332353364
|
||||
65306132346464363966313963346633353538633936363164393566376630376365316139376132
|
||||
38323039653465346462666439656134393964623563393664643634313638323832356164323863
|
||||
31366535336238663236666162336166313933376531643137343665653765623961333464633332
|
||||
30306265306364643738386561303833326363303836333032393462643764663664613536363835
|
||||
30663937333038663435326637663636383165363632326337653932336130333932333861366235
|
||||
66313566613033383631376132333634653639336666386164613934353230613239383239313038
|
||||
30653162643233636638393036353032353632316166333938313966626133643237376461313065
|
||||
34323762376133393535613864636461616261343031393266666165316236323231323534376465
|
||||
63386363343933316139356163363132343666366132313038643938653865663934383335336364
|
||||
31646563346339323633643366653861323030636138663861636434306238623738636331393538
|
||||
66333765626132346533313261646434633236633432333430613266333161643566326434363633
|
||||
62373033356337393232353566666263353539326564333766303335666135343461386530626562
|
||||
63633137623361366362616432613237353231616465656232373835376463386132663331633737
|
||||
63306136326264643365356634336233303163663135366531643362303666653630363962666637
|
||||
61626539613333306263323564373435613961633439633261373530363137376132326462633035
|
||||
30353838336433313135623530643663643232653364316263356164633661383937393238363262
|
||||
34393666663361613735623435626238646266376533333963653630663462653733373130393338
|
||||
32626361366632633630323363376238343534626230373835353036333065396435653964326534
|
||||
33356466643839626162316438356437623732656664613434613165643666656230656434633638
|
||||
61316134333631326365313532313335633634356466663834666531653365333333616137643835
|
||||
30666664353134313236653337366466623966366131313962346564663432633535363938333261
|
||||
37353464383832373366613531666161323632616236613631653433643562326134623838393135
|
||||
62323533373336663434386166333862366639626562343830396533646138636261326161393339
|
||||
33613961353431653537373931396531623061653163376230663338376636613136353433646231
|
||||
33633236306238336432323134393463653161303662336266313562316337343430373532663930
|
||||
39613133646432343238393762386266303531663032306130653532376262353238636231393539
|
||||
36376332633563633661353835386364633461363437333038363865356237643739323133653235
|
||||
66373931336330626663383136363265363133393239393131623465383433363336333237653131
|
||||
64656662316665356665303765343932363733666362353031656234373363663364666537663030
|
||||
34646233386630633438313166396663383732366233373139343431373463346133316663643036
|
||||
31633536663266383132393032313563653035303034336161356139396636353365636163383031
|
||||
30666637363933643262393065633463313836653530663232303736306537636533343431333966
|
||||
32386330356564613932306466353931393839316562353362303765663263383738363765333136
|
||||
36383036376130306133393166333734316231376165393832383735623838353466353664353834
|
||||
39326136656239626434313930306435333533623533643236393437313038393235386364323766
|
||||
62323537666333613066356236313361376138656232343430363438346261646165623565313435
|
||||
65326532623761343239666664636637613034623736386665303634613737613964333630613533
|
||||
31653731353533613866663166653237663162623236353838646465373734613466616362363833
|
||||
39346335373539376439643937363233373362616334323231623232353061343830383336666466
|
||||
31303663643061653866326632333336626462623835366564356231316263326130393138306330
|
||||
66333532336164633931306136373531666131356130313262313038626138653864633734636132
|
||||
61353638333133383035333937626333376564316465333539653138323739356265613664653536
|
||||
35656136343036363532613335336230373364336432343731326132666662636664356364303335
|
||||
66613736633231393534626539333536626565663231396536363062353037346563643934313236
|
||||
32363138386430633761383065343834376631643539626330313638333534333734333239383632
|
||||
39383833656134393165666533353739303834633330653936643637316639663033363865653631
|
||||
36313735613962393264623938336432336538393937316634626165383934356562353035316533
|
||||
33353735653133636439393763356430303863326235623932623464356636666265653065383736
|
||||
31613666303065653035303131646364386636393035663865336164376536323462666239356333
|
||||
33373562383863323635613634643165336465613734613034366236633835363733306662613462
|
||||
30326565346365633131363331643165346466366666633231643538303430376530393139393063
|
||||
32666664313162303065616261326261353130623564343761633861373034646161643163373533
|
||||
36386665626530313030343032376531656333356332666437383530613834356336386465353539
|
||||
66623062646236333465316137316564363938303966613561643830353332333537373736626438
|
||||
64653239613063646430386631313435306462303566333734663462626530356530663235303166
|
||||
31633366613264316137303334613366366537646261303962646364643238383062643732386436
|
||||
39643430613562663136633130666563396538306633633034323064386136303034306531323564
|
||||
63646633303035353831653438366635646562613639313230396462373161363030386264623033
|
||||
62343031616135346238303337313931643461386363396663333231626661613938626366653837
|
||||
33303265656561313332353463363534666232363561353532643532386133306332333930373161
|
||||
66626135303036633566303961646461383236663737643265346566393963343634623164633966
|
||||
61313031333863383237313633663633333866366230313936396334383361393338656465346437
|
||||
65613763643061386463386431386539656635393435633032343030633738373936613833623938
|
||||
39316239363835633339343161623237656230633763313031643663623466323764663366376165
|
||||
36366530363535376636383561343161643037626639323830396665303238663534633531613735
|
||||
39316639633730376533643932633432353835353439646666653766666338646662643162373263
|
||||
65356563303235633963633232393736636537346637376462626637303535383063373134613732
|
||||
32663763663364633463633738343535316436303365343665356133313836616164343434613133
|
||||
34626434643531343461346332636539636463313539393830613434333933643632313737356564
|
||||
39386137383165616139626363613732626336366461663339346134656530396464323533313265
|
||||
32346535356466383135636638616166313663613565353765346264376535653135356638663538
|
||||
30623834393935636562656639643737373462643137363063663737306361336339653334633665
|
||||
65316237633436303761393766393234326538653633363936303534646566373338623935666538
|
||||
38313761643437313565663762613838636163633066313630353631353934396338353665653330
|
||||
35393661356630313436373761383462396434643535316364383338343433613061353637333563
|
||||
38376562373462316334313632343965326235323231653030666666616163346438626437386637
|
||||
65656163653935656530366239663237633937336166613132356333623964666630386331383331
|
||||
32303166386431643739303363646633376331623166313135653265656636663236363936336462
|
||||
63303135613539663233366362323565396137623264353431373836356233666332633731666364
|
||||
62376565633037373832393366306431613863353137306535393664313866616235633638306365
|
||||
32633761386462386339343066316263663438623330373733333634363736623637636661636331
|
||||
38353835333962633537633864616132643563616437336334326633323636393833363666353261
|
||||
36363732383565623233343439666430303961306263363032383238653265323637313430613133
|
||||
63633338323536386139613233343636326564356564353065386664646230666561336338366436
|
||||
35353162633864633764373535366634303738346236633362626165393632383762343036313930
|
||||
62363864626338393736373938343264356235393763653432306330363363313762333137303362
|
||||
35393061356165666230343135663438303834363533323064626532663662323063386333623166
|
||||
39353037363363373166316238393565353266353665666465333134303234373263663135346364
|
||||
36646463343032326339663130393963663861323130356365396365643265313833353234323330
|
||||
34336661363733613362343739366636336265356464613033343132353031663965656634613831
|
||||
33656466376132656465383664363231653235623036346634666166623532323635313130303836
|
||||
36393064356637623139653333633164666332303539623863383538386262363064356130353337
|
||||
62393063633637383965346139626265666463356362633163363537303166336264346332636363
|
||||
39613735626132366137356365303331633037623132623637383164636565653963626632626338
|
||||
65383663633831613666396534333061643966313366303937333261303135353762656331336638
|
||||
35643539326161646433633862623762626539396137303863616139383832613639653033653933
|
||||
39373734393066613132323739383036313830346239306132656464646462626661633931623932
|
||||
36626163353563633764323466303966636161646532316134383034393733653363363933353738
|
||||
34383563666166323865656430346661633663363635623566336632353633303838303436616266
|
||||
34326131623331616533306465373365396437303764383231356631313335393364643664663864
|
||||
33313033303463376139376466643434383461643264316534306166303163373661643962343030
|
||||
36653665353232363630363437376266306531633934626230653338383664343638313334306636
|
||||
38666162623761366237383236346564333333633162623832656462616662313031316166383465
|
||||
31663362336663353564636335393738333565356432306139363661656663313234396664623832
|
||||
38393630666338663031323464383535303539643931393732616666366661316163626339643437
|
||||
34316532613063353264303462366534613035653834356562646637633137643839653237353937
|
||||
34326230396137383531393962346137643035333834376537363865643366623932303662303839
|
||||
64373431306464363762356563663038343737356165313832613965653065326532356133646137
|
||||
30326232643334656363343533383163643130343836366430333836396463386666623465366336
|
||||
61343236303336623863303630336437613932353832623866633661653566623431653938306238
|
||||
37326630616131646634383539353234323766363633333030613937616632333432366565343537
|
||||
35613237343838333430656336313232653530373863663031386433356337643334303363666532
|
||||
65633033333463366636616138646632636534333963643864383033343463666338373630313232
|
||||
63326365666666663262383664353664623963343366626364363965626435666363306237346130
|
||||
30306661386334653137336464393465646135353436336138616563366163366664346331646666
|
||||
61613334323330663835366163623836363534666531353737656464663935333765326235306566
|
||||
61353962363034663563386137373432656166663661653861396361653930653539363333613435
|
||||
64373731616330396130363464653865303931333637393333386531303365646130646161366263
|
||||
61643234333563386338643331303164323638346639353765656632386262366666376665356164
|
||||
34393464656634303130323738353161643537613337383661343232363961613931613234643361
|
||||
34663938626364636562613332373161323932386532356261353137393533633731653034633966
|
||||
62316666613738383665303433376438303266646264353133303566636436373966343662363561
|
||||
63333662613930323666656166373763303961333332363231396532376564393966333135373966
|
||||
65623664303537376362383861663238663463396537353866666333656664323562373165646633
|
||||
35396661353639356363613834653432346539633135663565376234383866343433383339666362
|
||||
63373330336562323138343934646466373738333039346361623836653231633566316435383636
|
||||
61326535393339616639366563383662636136353162393961303362393866363436663630303762
|
||||
65343632396238623137333462633633653761336635663265326332623631393566323530623562
|
||||
36666431396532303939316662663138356631336434373732393463306336303435626232316638
|
||||
36363733383831363930666132396661633733616464393264343339333166633739346236383833
|
||||
33373962313762323430343161616361636363366334613032323637316261656333633639333066
|
||||
34653635393031323262353037396130383964363037633463653331653965363562326532623037
|
||||
383331306234353238633435663136623634
|
||||
61346566636664323837633233623331666232363731633634366538333962363966366433636232
|
||||
3363303535353934306538656332633436303638373933610a326364636563386437386466653335
|
||||
61326564366131383062363037663965343330663431396331343831326133393233366439646538
|
||||
3435366135636162300a326161313661633134393666623964323761343139323630353938313864
|
||||
38393339343231636566386534363831616237343531373835326662376162373264633038353330
|
||||
33663539646165656261326663306332313932333761383937633265373330346134333532616138
|
||||
36623861356264643963366138646233343436643066396430653663323039333165306536383565
|
||||
66346634656131323066353837313836356162346330376232363835653331323736383839356533
|
||||
36636536313630623730333631383166383032363633643263613335633938616235656166356164
|
||||
37376563633634666465663332633635653736393135353538346366386630393765316364616463
|
||||
64343563646233623330633765326537383739323835306566613730323630376339346139613163
|
||||
39623132633830393635313537616661326536643662323237353337316234663931626261306639
|
||||
32376434363964363437373165613062333966653331373664383633656130313764626639353164
|
||||
61363261636362326261383430666235316538383933643365663830336665633164613435303065
|
||||
65313761623032373332623932623765616263633433663466366633363333356330393161346536
|
||||
32373966366330363565663632306130656161323234303666313636353166353064636266386239
|
||||
38363935313564323930396531336566313066386630626431346633323331633436326237633862
|
||||
33363334346264343537333065336536343264336538643862386336666635383766333235323034
|
||||
65313764343637343361323064333864343464356336376333613361356530343765626166366337
|
||||
30326266316135353263323131326664326635616231303831353036333439383633376161383662
|
||||
34613337376237393537343932326330323564313931376261393163653464316232373238393535
|
||||
38376637666533363166646336376665363538643363666537393232643762383130366165373037
|
||||
39636531336664633539653162393536306531313637663039636431353365396466363236376432
|
||||
66633064316161623739623138386664623165363762663637353435373863656661393362353330
|
||||
62333163366438343364626432643232396133643065636133626339323935623061323761643331
|
||||
38393466353833643432643265623431323436346638626661323036303334303239383838396335
|
||||
33353031373435306266303466366264363737326562326666363066646362623335306465363666
|
||||
34343862336264626134336338626330396130393833383434363834336133393032613237393031
|
||||
62633133663433653662373636356231396630626463633432326665326461396530613162373366
|
||||
34363462353865336166313639663438653839376531623533376335323863383632376433343432
|
||||
63316239333566356362306133323332353937366237616237326566323363646261376434373939
|
||||
65623031396139396164653939393162393930326530656135363564363565643133616261666436
|
||||
39313366383334633036373832323462623161666563303336383131363163373233616435663065
|
||||
33316266626162363063366539323462346464393932393165643161623964373539393964636139
|
||||
34343538663533343931613439663436383531393939333239613234386465343430303833353133
|
||||
64636338343032373230346562363338303166616166666438646262393333313633656162303764
|
||||
61386164353835383233343535636132643831623062393539336233626235303032663464386266
|
||||
66366664346364623533613832373631313336336530386162363861303333306563353439333236
|
||||
65316161383538373733633137633065633362363936373264333034646135366434656534383030
|
||||
35303838326361313163636432643638373832396165626532623261366332613562383261346661
|
||||
32386164366264356536653439353763346539643934393466643166333761366436613165623064
|
||||
38623530316264333939383932313133613730383632663830366365343937313036373439663930
|
||||
66613034633532303065333935623264636534393638386433346336353765316264346662313337
|
||||
34613732356265363839343434633563303665313030613535343336363432323366623234623738
|
||||
38396439376335653731346464363533313161343162653930656137626337623961336232623165
|
||||
37663863313532626638303937326237353235393866303063313532343164303165636262646434
|
||||
33316363343265336462626630376665336162336661616436656135366137633834326465303964
|
||||
62643435393537613836656334616236333131653133653065636361343261643032646163633462
|
||||
38666465633531383235613732366131366637336638393639346463663830626539653538636165
|
||||
34633238396233646337383936633438393039306461633631363161656465336238333434633030
|
||||
37346334396538386437323337393562353463336130363837393033353834386631396164386466
|
||||
39343338383634663338373536656363623234313339343363636432303937316462376139333138
|
||||
36386331616639376331393963356138666463306666306436643237303764346330393332626465
|
||||
62666235656334323231626635333338363736663063656664326139346363323039393565363966
|
||||
36326662353631646135623832333266366139313662316333376237326433626434336531366666
|
||||
63323062393030656639353634336339343034316237373932376466653930616465346436383664
|
||||
31343339663333386533323261393164363039386666323766343337363033366239386266393362
|
||||
35653333666432376337653332393930383561626431376364326439633331353033613361376333
|
||||
36633965616237333662613931366533366636393432393863323330383431326635386466646565
|
||||
31633136373466663161643965396434333437626636626237383630663730663531636139666166
|
||||
63613964646133666361303430643766333063643732393062333631616132323762623735613566
|
||||
62613362366536386565636538656138366532643932373163353437636562353737343566613330
|
||||
34366436323063356433303863383961303130666637366235353264356433313066616262313532
|
||||
37653165356230313165323330666339376337656332366531383566353536633831363430353764
|
||||
61383533636263393336313631393766353638353862363666323163356631396234326262626634
|
||||
64633438333438633235666361373333316265326237393134346539343361353461666664306239
|
||||
30643939633462303463626237653135333863336530326438313233343232323634633935653134
|
||||
65626337313266356462323039393564356638353263643566313264616239353036323063623333
|
||||
32636561373166653739396635633039623464656536653531356365393362343132343631343433
|
||||
30623438613137333063653061376434336233666633366530333634346136663831643866386132
|
||||
66383865633531356533316437383933656439353262626138623231613532643834383238323438
|
||||
63616238653338656536316662306565326233646136366634666330666263356235636231393133
|
||||
62373665333336336166363936656636373865666132646465656563653464633338623831616436
|
||||
38663730353764386164653166666136336665383439313135346139373239636137333035323862
|
||||
62366231323434363864656532323462633061663765616365366335376533336661656536353537
|
||||
37383466356438343539363935653534336332386130306235353565383933316137313233383832
|
||||
30663663346339396236633733633464393631376436353833663462316533646534383134356165
|
||||
33616139663236653661383063323865633339343836643134396364356230323135363731626236
|
||||
65333338656430376466616137303561356433393733373961313862663430656431343363363635
|
||||
63663561396438646536346237653562663661353438363237616539393335386338363132623937
|
||||
35613838616134633334623939333466646131363663653961646138613064366361646235346130
|
||||
64353130656532333664376635633137333434343965633361333830366232356434346263646232
|
||||
33336635323062653161346138373636613638623431383538343436323737316335306334366339
|
||||
37303033646363636637363333323533363037616536303737653064333230313761313366666664
|
||||
39643861393636356663396431346139646562383035396461323165663665626238653536353133
|
||||
38346564383761636537333961656538656366333530383831306533623532303635333539616165
|
||||
65303537663934626561666335643361333339316434373263613637663037613033633930313437
|
||||
62376233613638636234303264353339653963383633303761316339626434303466376431343664
|
||||
62616433623431353164383735313835383935333538636432386562616438623062643866333137
|
||||
65366263656631616334303538363239316632373936663431643239303034373831623765663662
|
||||
65646133393263373639393664373265376664356434303131396465646161616464623734316436
|
||||
30396265396536333731363535613035333066616136306538623165306462333561306337386131
|
||||
38313135663030333439643361633864663061323738313862326131623637663634323131373263
|
||||
31383730653838656630306138386435643435343162626638656338333131363137363063616432
|
||||
33326636383530343335613765346634316530376635636233356364373066343835376633356137
|
||||
61356336616138383536653161363930306364313035313135656634363632353731393934656562
|
||||
32643362636466656562666466613364333739393736313130656137396437373763323138353362
|
||||
65353661636537623230623666373534646635626538616431323966326230396637326335326239
|
||||
30373534616362666435346263646238316637646330336635346437333537643630303062303338
|
||||
66626237303830386164656531356439656162656633303564316636663639383662306462373430
|
||||
30616465353461366532356339313734386137623566653262656365643136303065353834646365
|
||||
66353637306139316132386239306163343564356464316537656237623734326363333137366463
|
||||
63633864373431353064393561653366653233616233616536373837353065353330643863663361
|
||||
63653336633135636564373632653561393862633034393266613337343132363034616139366665
|
||||
32313536643066376638303465616632316533636530626165663561383562623133383834376362
|
||||
37353633636561326263376366373430656562306633666431316664313832353039366437396436
|
||||
36333233313765363833316536363132623633336439653734623964373461396433343536316635
|
||||
33356266663630376662373632323436313431376435646561336665366634333135316334303863
|
||||
66653461386632303362366535346434616438653135613264666334363363663633666130313734
|
||||
36376664636363623965363630326236346566393961326332613638636163666562346661363663
|
||||
37663939306338643134316163333233363164616232353063323134373565616532373636653964
|
||||
34363237353234393632326438346132353036313232313231663762323131303830613532386262
|
||||
65663836636266616661373939383937363330623665656139613239396231326631613461396435
|
||||
30343064366431343735383135623135643165376464343138393131666235323832623034333466
|
||||
65663932623335323361313930343330636232626661323464623236313262303638353933613630
|
||||
38646365373164633131653430616338643163383639643134376464336537363861316137353063
|
||||
62623965633339303838663434306161633733373365383863633462303161643965326433333832
|
||||
62383731376261303364373931376631626138656535363638376236393133636261626234616137
|
||||
30333637656139343535636534346239373631633966616462636637306639336630336337343832
|
||||
66373730373161613935393230623934653530666464636363653066353566333730336261386138
|
||||
39626537306532633937353761303765393530316236373432373734373965623238303430623632
|
||||
35396435343065346136616632376533633465346336366137663836323234613735306537373631
|
||||
39653632303133633032316634323966333762623130613161323064346562346163633361323333
|
||||
37613735666361366530356363393262393233346239653137303330336161613461666330636530
|
||||
30366531633538636433323831646433646230376335646535643831316238323233383534626139
|
||||
37343635383333633034346436376433303634383336316535306236323061623365303432326632
|
||||
63343036623136363966373363373864623262646462333332616635623831383338666438333962
|
||||
35326534656537386636363234316332653232626663623261343737636361633338343933373066
|
||||
39346537653261356366656135303961356162363763623532613631636239633432393266623966
|
||||
31303434343438643563303463326564393632653938656661366630323636643364333664306461
|
||||
37616563663630643462653634666166633333393466373462363937336333303963303735666234
|
||||
65666632343036646536393965643837313234353537663839353032323362663237323338623139
|
||||
32656663313266393763343133356438303637306633306231323033643535346637386332643934
|
||||
36653638623334343030363330643237636635633464336561316331383432636438333839336535
|
||||
37346436393066626634623136366664383734653936393335626636383736343266653863396331
|
||||
34303234393663653438333466313962313731666165366233313162613437633430333562343163
|
||||
36663330656639643633633336643436636564366533383235383262313865656538366531303337
|
||||
66353732626333623031326261306633313666393130373562626564343539313338366364656336
|
||||
61353837633765646564653936366236303463363531336537333965666437303536643135393062
|
||||
38343836316265396665386664326532616632346265613334353466643165663463613661333134
|
||||
33623333343037333133656434376537303462343061393838336438346635363732393337313532
|
||||
66346366663538373237666565643662383665623130346466316663346162313964653830373630
|
||||
65663134303466303630353764636639613661613939396438633234636437363937356465326132
|
||||
38653132613066356633333434366265303235363664636432323566643838636266363762633064
|
||||
30383332373631326635613266366331656664393934353731653063366537616435396432663731
|
||||
36376437326237303263393334386263396334316333316132653765343564373935353563353531
|
||||
63396539613739316539656332623262636136363839353339353331306439323966656530383564
|
||||
34336362653064656138363861353331306164663834626662646263623561393764303864306565
|
||||
33366230303663303839313364353430363034316437326135393530366335363065303335636564
|
||||
31336135333163643864396264383437633736336639333539656661653638333936643030396339
|
||||
64646336323338376530653735613739333939363366363333623036663861613061366531643034
|
||||
36303037363361333561313961343730363533383435376362333739363863643365616166613036
|
||||
64663134666166383663656338323933386362393162636132306331633065323139663162383632
|
||||
38373264376666363964653138623232313732626665386333643435663531373532346539636636
|
||||
34303839653338366330613734646636303163643532373030613239313239646630366438386662
|
||||
36616263373562626435346638383836336635386661396565653836633339353938383164373137
|
||||
39323364666636356561343139373830383865633466656532336361336363363964353933613133
|
||||
64343833343330353433376166323531393066356437333132346133643364383562393731333234
|
||||
39363830653233386566356130356439353535353636656362616534326336616163316233353962
|
||||
32663361363836646166326663633730333436376230663235393738396639616331303066626662
|
||||
31643139303835303431303535393536643163393264386463323862303439353536366131373033
|
||||
31653234323165373562313063636436313538393639663962393139323231303863336165383830
|
||||
32363535616361666533316132343965343661373762323064333161383463343731356161326333
|
||||
32623464666562626531343664643933376162646631313166643365666465383035613464643762
|
||||
30663065616166613531343339383662303438653531333866396561346637623664376266373839
|
||||
31643233366363646632663635343161303033323061353737666233663561623462663837643236
|
||||
64613363383631663039613031616133386130363730333361393739356361376165353036326237
|
||||
34383533316436333465313939306437383337663864643935346336353935303664663866663564
|
||||
34346362313730306132373936663764633662646466333135343332393765373333333363376135
|
||||
61303839363236373066643066363135306266333364666132323033393836356531313232363864
|
||||
61653565323263306263653535306634313662366331643663643966353164326161383530613730
|
||||
65303738336561363161336330373832393062613130303131633265303037396230313839306661
|
||||
62373938613034383966336137303431613161343937343461663066343436626563623366646132
|
||||
32386238353662663337363363393039653330383331636430343632343064383565626338313030
|
||||
65643333316431616637643162353634323333313165343833636465363461333063306538396636
|
||||
34343832633565383961656537373562636433323566663830326136353765303230393939356132
|
||||
62356130396264633930356362653066363163623631613434326464383361643262373936636461
|
||||
61616631363335353366323865316439396335373966393464626335353631383137376532346262
|
||||
62343930633430333335623639643431306533396531333666653030643161356339353933643837
|
||||
33343734373435303235646239343838363439333433333437643162326539326261656330643938
|
||||
38306131343464613330636636396266353638616230333263323334353862363738623362633034
|
||||
38626566303336393163366265306564646436623630656533633630653836333035616364623031
|
||||
35363735343835306132646664613261353230336537646161643463316566343430373461326233
|
||||
62396563343939643432663062633064383738383834336634323635396461383830363063636432
|
||||
36666566396535363835303034363732653433623665353437626533633032333838316564323834
|
||||
30356132613563616333313039666330666565666534393139653065663739363061306365666135
|
||||
63396262393464653566626635643365356533623761643763663662383832393164326462313837
|
||||
35616233396165363434633434613031663739343432366263303734343635336136323830333163
|
||||
33626461646661343564303134366139666639383630663438396139306566643636623431396332
|
||||
35633232333064383161383933333931636561313035343561643534656635363432333266376536
|
||||
37373537336239363166306430376132323862333732393164323964363966666262346564613631
|
||||
39616438623830386266643634663562396336666134323137366233363061613332376661326435
|
||||
66313565346663343538383235356232346438383835623364353334376565393464393966366538
|
||||
65376334623438663562623930616266353961393430353361623034306231326530303438316339
|
||||
34386462666166363534643764633563613934366164303531343366383464643832353739316433
|
||||
36326364663737353562396261363736393562393964633539363031353432356238333461383730
|
||||
39386437663434326635373465643234373034303963396434656161633839613731326365613736
|
||||
36343664386565313535343635376532386331643264613563613735373631626466313964383639
|
||||
65663761663033333932336665383631636563363561646635383965393039393363346166613861
|
||||
66393466326634633438343632376538326634353937633562333137373565343339626635376133
|
||||
36626638393932623837336162343831303436643535643232616432363061346165373965323365
|
||||
63386632336532656635643863623132393931383838346263636339646332373665396632653361
|
||||
32346330306362343538353833313833383334366666613436343066636661366238333764353863
|
||||
37333534366536313464383364303335356232363866363737376665616438666533633066373164
|
||||
61636362363237313362633865333630386430623431656437646437396131363035616434343136
|
||||
33353839353765623531323061306465653336623333393837363538653830326561396137613138
|
||||
64646133626130323234383763623935323433336535343631383535356437333864633330393134
|
||||
66623530653636623565353830613538616433613538383632303539386362383636376465393036
|
||||
36373930326430373336666138653334616530653930336364353464613163656131323566393933
|
||||
62646265393539313432653462616431636232326131313239363337396635336131383534353338
|
||||
61356535663130336266376431666533353864393766313962623563313131336533656461373836
|
||||
34653833353730363433663435313039363861386336643937626430326563353230386630356437
|
||||
63396636393561376330316232343964666337663532353438303964613365303632323263633161
|
||||
36323033313935636239633232333632373637613439633332666665356331363532373138393861
|
||||
32326239616338366364613765393665396461633034346436623037666133623565613266376536
|
||||
37633966626636323762623965653831646434356232626338636332656661333464623261393933
|
||||
38643631313063336161663830343835366336346632636663353561623835303130313163366635
|
||||
38633038366234303334336432653534353734346236346638356135633234323364633937666162
|
||||
62323132393032303139393139633663303138646133626364626533356465323464613338363533
|
||||
37306362346431306164613637303965373132323633613439396166383134323162313164323431
|
||||
34383136323636636539353264396439613231653663636638343636626562353861383736306331
|
||||
31303564313861306666666332303136343330363063313964353331346433626366303935383066
|
||||
35386261633137613331303630613137376434663836383930366263666262656666356139343164
|
||||
35616334373931343336636637356339343266613161623561336264623866633834626664663333
|
||||
66346231386433383163653437353336613731646435626431366663666465313864613837343365
|
||||
36633962326633313735393238373166636435656661373732386637626264323363623632353631
|
||||
61613062653164353761313730613936356632306438643431306332326363616464343237386539
|
||||
63393866393461663135616235643465373838336566336330346662616130666130396364343634
|
||||
64333335656565623662646136313766623031363037366535386461393430646231353536346332
|
||||
64313431326164306266666539366264616139303065626536333664623737336134643761333536
|
||||
66333331303831316137646137316636633161353766393136363038653765376465623561373761
|
||||
30633331306565393365323735613239393338326635626136363933303862343465623166313539
|
||||
38393533376630313666643335623165336165633933323036643539623136333938303233353063
|
||||
30373032633036663963323831646234653537646230643363663238356631303434373265303735
|
||||
39613263383261306332346264646562636432663238663536653934386630393534653832333332
|
||||
39316530373861323830303136663462346164386261636666356235633563646264316636363066
|
||||
39353464346131643239363736383335643132383862303435366230353631613330333133396336
|
||||
34373532613134613231636636373132643462353865363439356336383266616631363139656265
|
||||
65333332663139393135643366623266333561313538646639666636303038663962626438663934
|
||||
35386463386635613236313666306266353064663666333861643932633862336236333435363539
|
||||
66356130646235636637626339356632376638316231656266303637623965636135636333643065
|
||||
34396633613536336163306561363332356430616334393133396639663731653863336531313564
|
||||
35653333356232396232613864313834656533323034396134313061313731613732386366343134
|
||||
35663836313639663038653036633338336666633066396136663964646137636262313762306261
|
||||
31313266643663373736393637323539343035636464336563636162343830313831383631313865
|
||||
66633337613632343363356331663639616231653033663962376131646533313237333536663438
|
||||
39353034383661396237623438623361323232653135653430313738633835613961323735343365
|
||||
63376131386339643330303938343765393365656466393965363838343431373763616139393066
|
||||
37323037396165636630663733333661626462643437616633383132396532396263373936343538
|
||||
30333732303235376435666565313435356137383538313039626462363066373864303633616638
|
||||
61363438396634613862303965653937356662373231346539653339333034343961623435393739
|
||||
62393430633365323034646233393537306266326234643339623339663764343935386430383461
|
||||
65396434363634363439336137613837316532373931626434323366393564646630313330366162
|
||||
36653532306439653964373631653061363664663039633134633233346166376261323938363438
|
||||
65653637313432663962646536376634333735616531653133653061313534623163323738643466
|
||||
65363538643336353338346632323565313234333463396464663732613332646237653332623866
|
||||
38666535386337386361333565646633366538333333346336363636333237316462636365383939
|
||||
61393631356163313933643230343133643539323834363437306465653438396565393532636464
|
||||
62343263353638343565653065373461386461363736313931313163656138656333383733373532
|
||||
62356563643663373765336334306466303437343733316431623665626465666633373463366136
|
||||
63663738663431306532616431373031666462346131623665376235663634336265303035346432
|
||||
37616536663965643236346264656361613732386239653234613932326462613637666231396531
|
||||
33316162353038306535386661626464316130343436353230383934346263323738643731373534
|
||||
36616537626437393530666162336338383563333636656230646235353266373532653431633539
|
||||
34646531303630373034323932303863623536666261326463303031636666363738363664653738
|
||||
31376335316239373437363431626630613161663962663934643431303966323237616339393132
|
||||
33366661383263396632633838393536383335343761306461313638613236326266396363623634
|
||||
37623165613935373730663139303737373864663035323030643264663932353465666138303264
|
||||
35326665656538353762643637393664636137313034643637643730313336356464383131343463
|
||||
30623232363838373338366163326538316465643930656230323831666462353037633131643961
|
||||
39393238386637313632663336373661636230386632346432366633356139376430646338316235
|
||||
66393663366232366230326262633638653863396635323539363465383535303433643061633839
|
||||
61666431353737393939646638653435343039353463616165653630373130383238333336303533
|
||||
36343331313235376263303633326139613734663264353330643666633234323730366134663262
|
||||
66643730373761323439636161623135646337383163353361633663376466343839623437613662
|
||||
62633837653464316430626266646338333530646234623834643739303738366165646235323232
|
||||
63663433336664376665623734613965626464653832643366353432633437643739396265376531
|
||||
66636436336135343833356334636531376131316463343162353362356439386139346365333938
|
||||
35646431306532663936616264626538666638336130313133323830313966353161666236323735
|
||||
36313666346638363865336331313163346635613735363634366265373363303033616663363563
|
||||
63353530636566303733343962653138363532326339666230666661393766333863363539373565
|
||||
32363733613164373762646633373833373466326633383339626431323462306663353235346664
|
||||
62653361323631623063313435663735333638616366613634363166656431303031353833646563
|
||||
39313630366430613234643361666365386462346338323330646366366335656136643462663166
|
||||
63316135306437376139643337646639303261666434316266323465373832333534373730636465
|
||||
34313832633839396634316561346633643734373763666136363039653136353866343132326430
|
||||
37303737383030616635343539386337336236376236383961333336363030376333666432613362
|
||||
61366365316364613639326236333738313936393535646636383261356635303238303938383130
|
||||
33613264396339646136346631643938356138653535636636393837346430396433313661613337
|
||||
31323836323162323632343462666433633537623763653566643231656134353235383131663466
|
||||
62313730366338646132626338623934646364316637343364653236316566666338363339363731
|
||||
38373832316239613961646432373864306235656339373963643938343664346238373761663830
|
||||
35663732396430666536313239386465373235393466643930383439613862313337313638386662
|
||||
34623033653238333064343861633132313762646534646263663566633264393638343730313939
|
||||
63353438333331383861343135326439303965333834646430393531383762623765666237316165
|
||||
65346263386636363638663530643633336331623734343231626363353165323562616138373331
|
||||
33393464333464666235666539333663663061383335343933373066333966346131653361333935
|
||||
62643363653236626330656561383635656431303231353731396466363232613462323938643730
|
||||
63356531616364346264653666613564613239646465376238336337383938313035366636396638
|
||||
65303566343738343731656338663832396330623066646233376436323334633339343162383966
|
||||
35343164626365386162666162346366656539383334346238336131616338393261306537303534
|
||||
62343164386462393039393763303566346163646330336264623462353464306232363934343630
|
||||
38396565613830346634653934653737636262646438643866376138633065303239633538373432
|
||||
32653562343666313361383331636261313331363639633465383161393063336335343938653933
|
||||
39663865613236393037333031613964633961326332316662666531353839343937393761636164
|
||||
63643230623332626266323061386665333862303532313530653534343762383066643736336466
|
||||
36626564633265303330666331353163396130376331333136386638306233343138613664626537
|
||||
31626236643637363766636635333465396233663239353865306638303331363937663936613563
|
||||
30306166393033336164333432626239333338653837336234393536343438326436343434373433
|
||||
62613165633736353838313439366434323664333832366132663766613236643430313764343032
|
||||
32636630393032343130623438333166306564366562373764656135653730346237343737346638
|
||||
66333462383865333733316237663833323036643230616531353630306565353236363734643839
|
||||
38616532626537613336366637386633653065383135643135363332353731353663323365653763
|
||||
34623433343135636338303662633966343965303461313136373333396538626166633330343635
|
||||
61633034356135623639623636626463373339633665366362343064313262333061666637323163
|
||||
38646266613461393533663061376132363234376164393939663763643962633361663665336239
|
||||
64356633323730303262383766386635613361316139353331346466643661333331336161636137
|
||||
63616230396361626133373934613164336335376565656230623030393861383234346138303564
|
||||
39326134383063393765336461306237653839336531363538383666313832353732663536343134
|
||||
38663535316333656164396562303036653137653366393462663238613632383230626130346266
|
||||
33653136383237303139316133636230373866316164336663613666373565643733663739323362
|
||||
37633734356234626432646566333038303465333563323931396630386430633032626536623765
|
||||
38623961373063323464353465316432393134633432343465633030363539363865383661646138
|
||||
64373436336132356535646164336631626639383765386166326532643436393638653431623533
|
||||
65666639346664613266616364663033363730373637343962393533656463653661613737303837
|
||||
31633262653361373036383636323766653538346532353038393835303931323264376535373062
|
||||
36663631316234363063393938633034306465373966333062663162653266386237663137653033
|
||||
66346435313334653962373935316537376132353962653535366564306132636534306164666431
|
||||
38363634333635373966353636633836373865666237656336626139353130626332383338313238
|
||||
62616632616337363664616165336563616161386662306535666634613965613039356165396239
|
||||
32316364623534643662323334326461386335666530373662353931363131306364626638636430
|
||||
61363533613638616632396463343239633865643438373837333535336662366432373738393361
|
||||
31663064623431383336343034653734306635323764613031353862646663626339323764313731
|
||||
33313132613138633364646532633334643663663339633264343365363231633034666265376161
|
||||
37313261616230616664396138306264653139643438306662313462386439336434373664313532
|
||||
61396631663663353334653961363663393330353830306130313065366330653761393436326535
|
||||
38326163366564373236386431366436633138326561303237613965353266346361303063613666
|
||||
66333637313061383138373661643964326162633137626163383936386132346636386335353134
|
||||
37393661666465366331383164366666666431386166396337386630643438643564336139656634
|
||||
61613232653433336330376433653062333062663034306630653538303337343661326634363463
|
||||
35626130353561353436386261663335623964633564303365316166306237306634633464653565
|
||||
61353532613631653037643162666366313339633034646435316338353034366630303736386639
|
||||
66396236646132366265653536386435333066306461303334646535643835393730643338376633
|
||||
37303161613837383838323961343834343562666532366136303337363832363765326637653934
|
||||
66396331333938633034303761313566656533346336393531326235393862623462306434396438
|
||||
33393539306234626163626664393037666533326133336464323831323132633035313561303034
|
||||
38343332306532303939363038383934376136333661343435313238356536396537333066613665
|
||||
31623564306435313466343133643730313262623765346464616263313831313164353162636334
|
||||
64666133333330646666663336613332373664653932613834383739393762326636303466363134
|
||||
33393164326232353065396361636539626531656337353363666233343139333263383664666531
|
||||
63663539303165643564356261346433373438396532396439363330623035353636386531356437
|
||||
30326231333333393533376366333062386464323565306463336164636434303137386661393139
|
||||
65336135313462363237343861313165393862393233656561663662333162306539316365393063
|
||||
62623837353437373236653965646131383262653733303038306134626430623961376366383331
|
||||
62383366336637626531653532633664616537333032386434306330616439373430383664623638
|
||||
65336666303263613965376664336262353139396535646662316233343764396661616131346237
|
||||
62666561373866646564383634313639383636653861666164393263306339616163383965643366
|
||||
34656538383934313136316231643934663963643737353637633032313930623266323466653032
|
||||
36636339373034336463313862353638313538316465373132616166323863373832626330383937
|
||||
30313565373731663663336466363239303734336632396636346139636335646162353166643330
|
||||
30323530613736613736623238393233316635333437333938313932623834643363616239343032
|
||||
34626636323461303438376635633637636232386333636139626565303635656630343062353639
|
||||
39396264386638326666303438623334336163343938336566353034656638323633343837376433
|
||||
65366232303838643832623332663435636530363638636433303336356531633131356266396334
|
||||
37313864393761623333653638383766366261353334653066316563386437383432343964616535
|
||||
30306134383166666630636666316630383030346130353865386539303537363539636161643733
|
||||
62343530396563393464656566363739313666613762613565386331366365393338313237636231
|
||||
65326630313630363139626161373730373962656134363733633530376261613239323631623433
|
||||
61396631333736383731363135383664663632643939376334633966373231363430326663323061
|
||||
31666134396439346166346430363739626665663133323930363663303164616337316233656333
|
||||
39303833653366353931343936623362346163363766346461653264353733363439643439336435
|
||||
32656665623532666465663834363564666634653038616361393665303766326436306533613439
|
||||
64616334393539643662303036626131623137383164303566623430646337353264343030303935
|
||||
33313164643439643662393632323433366238306261623335656430396437396535653335343965
|
||||
31616265363437356263616539343232333539393133323334633730366263386135393864393162
|
||||
33653161623735363335323838353231326166376365656661373965353433336466623134386230
|
||||
61653733393737643832303562663933353935626437316538393133613063316566646565323034
|
||||
32643235303038656362663561666539633432383636333463396162353035323531623133656563
|
||||
65376362643432643933393063393266663563663636616261663638626238653633303630383062
|
||||
35363134323131323236623563353035306430343534643962333530383836666565646163346139
|
||||
65633239396437346432346233643463333831663636323833386338386430373430336536313463
|
||||
31386131653831623832346564343137326163326261306463666563643839653864353764353861
|
||||
61663539646431633531653465333336663733663264336565386662626335376231643563356337
|
||||
65653439613164306639353862656264333733386235333732643730623639333937396236616466
|
||||
36303332626163323932356338313763363563326139633836343637663832633262323065326435
|
||||
31643134303737316635336534613038313362396335636164656436623764656537303732643730
|
||||
35613936303762623765656265663362613736636531633335666230643335633161303738663465
|
||||
61636165376538313839646133653262343530393837313465343330626362626164366165323831
|
||||
34396134616138363562313934626239353135336335656331383035393763306166666533643465
|
||||
34383436356162323631373164373565313563656664663762343630383161646633356434643631
|
||||
64306239373363343161663030343734366239356237393033643931646434393237363239333631
|
||||
66343963633266613138353433383338343737636238346433366361643436363233373034656231
|
||||
37633030623232613030633832306337663336616632333730373639323261646432616531366634
|
||||
63663837623235383438363533313330663135323661626236333761323165663638623930653631
|
||||
37633933383562396331326264623866353966633932663961643032383363613333323937383538
|
||||
34653536313866336530343832336564353036376234383538323732306565636234623832376562
|
||||
64626166366164626436663564353539323765663365303332373138373034333064626164336439
|
||||
64333639313832656439613532326330336630613131386463663934653539663763646138343435
|
||||
37636161623363613039666334626164643464313439376330653837363661393431376632326137
|
||||
34383134386563343337363037616431366239623461356563613039393936653330303830303534
|
||||
30373463363265653439633736396664623661646237633537373465316362383965333738333831
|
||||
37373735636362376265376363386566313663343961623064343363366339373236316434653737
|
||||
35356436316230626631623130386631636664623762613438313632613134396131336534393931
|
||||
61663639653166376465323162313064623836623366393464386365313739626636303265343463
|
||||
30336665636238343930303539313930626439326462343865326531616239323036306230363634
|
||||
33626531393332393332663137303938613163653637376264663863643163346635616564326337
|
||||
65393335383534323038656464353963316164396263663533643464623561343337623661663635
|
||||
39663331633239336432366466623666313036393661623966396630623431626465646437363634
|
||||
31623536346231373534656262336466336465353261663433363134633030393533326539666661
|
||||
39623931313339646134363337343233346334366262646638613039363730613433313863396564
|
||||
64643436653637356563393131666465663332326665373937643764376231336331616361323530
|
||||
35303038323739316265363863316139363033386538346139636262626633303563353737363739
|
||||
35656463646364323964366533656339643466323965323661306166393933353237643834373264
|
||||
66623533623837306238393234666331636566616535323231393364643039323064343433393533
|
||||
66653333633931303761376162666530386161336534343932633933616139306430326430326566
|
||||
63656534343362323830643831643836303331643364396566626662333439646531616465623963
|
||||
64616361396635393339643035616536643435353738383437303832626135316631633166613434
|
||||
64366335373365653763353436333961663265643739646536343564383062396337333362363235
|
||||
38326236363564313437343832383961303032643831363563646664373338616363623935656538
|
||||
35376233326434623339376463346338636162306461656434626135623263613632393135306531
|
||||
38326434636637366633336337663739633232386531336162333939333132336362363430363965
|
||||
64316337353637656439343233343362666130663165613335633363353339633537323436323334
|
||||
61343838383365346466336332643065353836366463613736353231343665623430373232333133
|
||||
34616261653862353062376231356438363038643238333833353464653138303734373735303135
|
||||
37663338363737666461646161653434636237313332343838346335313363616134653262633362
|
||||
33653236343930316231636534613666373061373332653434356530343334313434366137636364
|
||||
64393035383635646363373732636166383931343362356666643836313065346365396162393237
|
||||
64623064643831363161653034316162363935356563316536393531643164643466323432323139
|
||||
63643533373431653435383136663665316131343632383366396366376237346265303634326232
|
||||
37653331666434316263653531323861386339326130396563663864373364663831336465326665
|
||||
35383534633433356134636632353333346633383335383039626432323932333433303663383364
|
||||
34633133323166313032393031346536333863306163333462623063663239666563653365643631
|
||||
33623238343335333932303836396662323832366139616334653639653838303662313162643335
|
||||
63396164353038636432653332376536323366343765383034366633666639353936316436613130
|
||||
33633763396366363565636532613738663963363932373061313635376338626665616131346532
|
||||
65323564333465616664393965633761653962616137663336333430663433646562333364633361
|
||||
31613736366462333131356131303063386633386664366166653136333266663161663332353737
|
||||
31623139663130363735336233353135323166613964653839313137316239393431306239343137
|
||||
65643431353936616433336263353439383763663833333031653033353464653762656433393436
|
||||
32376666306637636134
|
||||
|
||||
7
ansible/host_vars/main-street-station/vars.yml
Normal file
7
ansible/host_vars/main-street-station/vars.yml
Normal file
@@ -0,0 +1,7 @@
|
||||
---
|
||||
# main-street-station — JMRI / LCRR server
|
||||
jmri_profile_id: "My_JMRI_Railroad.3f178885"
|
||||
jmri_lcrr_repo: "ssh://git@gitea.mk-labs.cloud:2221/rblundon/LCRR.git"
|
||||
jmri_leviton_email: "{{ leviton_email }}"
|
||||
jmri_leviton_password: "{{ leviton_password }}"
|
||||
jmri_ssh_authorized_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINnSM/9fO8rz/amqkyoGUzUKNNzzmtSXPwOCr1O9zKNO ansible"
|
||||
@@ -9,17 +9,45 @@ jmri_user: jmri
|
||||
jmri_group: jmri
|
||||
jmri_home: /home/jmri
|
||||
|
||||
# Profile — override per-host in inventory
|
||||
# Find on the old box: ls ~/.jmri/profiles/
|
||||
# Profile — set per-host in host_vars
|
||||
jmri_profile_id: ""
|
||||
|
||||
# Config restore source (path to the backed-up .jmri directory)
|
||||
# LCRR git repo (set per-host in host_vars; leave blank to skip clone)
|
||||
jmri_lcrr_repo: ""
|
||||
|
||||
# SSH key for jmri user (for jmri-gui X11 access — set per-host in host_vars)
|
||||
jmri_ssh_authorized_key: ""
|
||||
|
||||
# SSH key for jmri user → Gitea
|
||||
jmri_gitea_key: /home/jmri/.ssh/id_ed25519_gitea
|
||||
|
||||
# Config restore source (legacy tar-based restore — leave blank to skip)
|
||||
jmri_config_src: ""
|
||||
|
||||
# Ports (for documentation / firewall rules)
|
||||
jmri_json_port: 12080
|
||||
jmri_withrottle_port: 12090
|
||||
|
||||
# USB serial device for NCE (set per-host in inventory)
|
||||
# e.g. /dev/ttyUSB0 or /dev/ttyACM0
|
||||
jmri_serial_device: /dev/ttyUSB0
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 2 — Layout power monitor (Leviton Decora Smart Wi-Fi)
|
||||
# ---------------------------------------------------------------------------
|
||||
jmri_leviton_email: "" # set via group_vars (vault-backed)
|
||||
jmri_leviton_password: "" # set via group_vars (vault-backed)
|
||||
jmri_leviton_switch_name: "Layout"
|
||||
jmri_monitor_poll_interval: 30 # seconds between polls (active hours)
|
||||
jmri_monitor_quiet_start: 1 # hour (24h) to stop polling
|
||||
jmri_monitor_quiet_end: 10 # hour (24h) to resume polling
|
||||
jmri_monitor_stop_delay: 30 # seconds of OFF state before stopping JMRI
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# USB device identification (ENV{ID_SERIAL} from udev — the segment of the
|
||||
# by-id symlink name before the -ifNN suffix)
|
||||
# ---------------------------------------------------------------------------
|
||||
# LocoBuffer-NG (LocoNet bridge) → /dev/jmri/loconet
|
||||
jmri_dev_loconet_serial: "RR-CirKits_LocoBuffer-NG_CDC_ACM_SERIAL_DEVICE_AA5700218A"
|
||||
|
||||
# NCE command station → /dev/jmri/nce ← triggers service start/stop
|
||||
jmri_dev_nce_serial: "Microchip_Technology_Inc._Simple_CDC_Device_Demo"
|
||||
|
||||
# FTDI LCC interface → /dev/jmri/lcc
|
||||
jmri_dev_lcc_serial: "ftdi_usb_serial_converter_ftDYQHZX"
|
||||
|
||||
@@ -1,9 +1,23 @@
|
||||
---
|
||||
- name: Reload udev
|
||||
ansible.builtin.command: udevadm control --reload-rules
|
||||
changed_when: false
|
||||
|
||||
- name: Reload systemd
|
||||
ansible.builtin.systemd:
|
||||
daemon_reload: true
|
||||
|
||||
- name: Restart jmri-monitor
|
||||
ansible.builtin.systemd:
|
||||
name: jmri-monitor
|
||||
state: restarted
|
||||
|
||||
- name: Restart jmri
|
||||
ansible.builtin.systemd:
|
||||
name: jmri
|
||||
state: restarted
|
||||
|
||||
- name: Restart sshd
|
||||
ansible.builtin.systemd:
|
||||
name: ssh
|
||||
state: restarted
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
---
|
||||
- name: Install Java runtime (headless)
|
||||
- name: Install Java runtime (full — required for GUI mode)
|
||||
ansible.builtin.apt:
|
||||
name: openjdk-21-jre-headless
|
||||
name:
|
||||
- openjdk-21-jre
|
||||
- openjdk-21-jdk
|
||||
state: present
|
||||
update_cache: true
|
||||
|
||||
@@ -16,17 +18,147 @@
|
||||
name: "{{ jmri_user }}"
|
||||
group: "{{ jmri_group }}"
|
||||
home: "{{ jmri_home }}"
|
||||
shell: /usr/sbin/nologin
|
||||
shell: /bin/bash
|
||||
system: true
|
||||
create_home: true
|
||||
state: present
|
||||
|
||||
- name: Deploy SSH authorized key for jmri user
|
||||
ansible.posix.authorized_key:
|
||||
user: "{{ jmri_user }}"
|
||||
key: "{{ jmri_ssh_authorized_key }}"
|
||||
state: present
|
||||
when: jmri_ssh_authorized_key | length > 0
|
||||
|
||||
- name: Add JMRI user to dialout group (serial device access)
|
||||
ansible.builtin.user:
|
||||
name: "{{ jmri_user }}"
|
||||
groups: dialout
|
||||
append: true
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 1 — Stable udev device symlinks
|
||||
# Creates /dev/jmri/nce, /dev/jmri/loconet, /dev/jmri/lcc
|
||||
# ---------------------------------------------------------------------------
|
||||
- name: Create /dev/jmri symlink directory (static — survives reboots)
|
||||
ansible.builtin.file:
|
||||
path: /etc/jmri
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: '0755'
|
||||
|
||||
- name: Deploy udev rules for stable JMRI device symlinks
|
||||
ansible.builtin.template:
|
||||
src: 99-jmri-devices.rules.j2
|
||||
dest: /etc/udev/rules.d/99-jmri-devices.rules
|
||||
owner: root
|
||||
group: root
|
||||
mode: '0644'
|
||||
notify: Reload udev
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 2 — LocoNet traffic monitor
|
||||
# Installs jmri-monitor: watches /dev/jmri/loconet, starts/stops jmri.service
|
||||
# ---------------------------------------------------------------------------
|
||||
- name: Install python3-venv (monitor virtualenv support)
|
||||
ansible.builtin.apt:
|
||||
name: python3-venv
|
||||
state: present
|
||||
|
||||
- name: Create virtualenv for jmri-monitor
|
||||
ansible.builtin.command:
|
||||
cmd: python3 -m venv /opt/jmri-monitor
|
||||
creates: /opt/jmri-monitor/bin/python3
|
||||
|
||||
- name: Install decora_wifi into jmri-monitor virtualenv
|
||||
ansible.builtin.pip:
|
||||
name: decora_wifi
|
||||
state: present
|
||||
virtualenv: /opt/jmri-monitor
|
||||
|
||||
- name: Deploy jmri-monitor script
|
||||
ansible.builtin.template:
|
||||
src: jmri-monitor.py.j2
|
||||
dest: /usr/local/bin/jmri-monitor
|
||||
owner: root
|
||||
group: root
|
||||
mode: '0755'
|
||||
notify: Restart jmri-monitor
|
||||
|
||||
- name: Deploy jmri-monitor systemd unit
|
||||
ansible.builtin.template:
|
||||
src: jmri-monitor.service.j2
|
||||
dest: /etc/systemd/system/jmri-monitor.service
|
||||
owner: root
|
||||
group: root
|
||||
mode: '0644'
|
||||
notify:
|
||||
- Reload systemd
|
||||
- Restart jmri-monitor
|
||||
|
||||
- name: Enable jmri-monitor service
|
||||
ansible.builtin.systemd:
|
||||
name: jmri-monitor
|
||||
enabled: true
|
||||
state: started
|
||||
daemon_reload: true
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 2b — LCRR config repo (clone/pull .jmri from Gitea)
|
||||
# ---------------------------------------------------------------------------
|
||||
- name: Ensure jmri .ssh directory exists
|
||||
ansible.builtin.file:
|
||||
path: /home/jmri/.ssh
|
||||
state: directory
|
||||
owner: "{{ jmri_user }}"
|
||||
group: "{{ jmri_group }}"
|
||||
mode: '0700'
|
||||
|
||||
- name: Deploy jmri SSH config for Gitea
|
||||
ansible.builtin.copy:
|
||||
dest: /home/jmri/.ssh/config
|
||||
owner: "{{ jmri_user }}"
|
||||
group: "{{ jmri_group }}"
|
||||
mode: '0600'
|
||||
content: |
|
||||
Host gitea.mk-labs.cloud
|
||||
HostName gitea.mk-labs.cloud
|
||||
User git
|
||||
Port 2221
|
||||
IdentityFile {{ jmri_gitea_key }}
|
||||
StrictHostKeyChecking accept-new
|
||||
|
||||
- name: Clone LCRR config repo if not present
|
||||
ansible.builtin.git:
|
||||
repo: "{{ jmri_lcrr_repo }}"
|
||||
dest: "{{ jmri_home }}/LCRR"
|
||||
version: main
|
||||
accept_hostkey: true
|
||||
key_file: "{{ jmri_gitea_key }}"
|
||||
update: false
|
||||
become_user: "{{ jmri_user }}"
|
||||
when: jmri_lcrr_repo | length > 0
|
||||
notify: Restart jmri
|
||||
|
||||
- name: Remove auto-generated .jmri dir if it exists (will be replaced by symlink)
|
||||
ansible.builtin.file:
|
||||
path: "{{ jmri_home }}/.jmri"
|
||||
state: absent
|
||||
when:
|
||||
- jmri_lcrr_repo | length > 0
|
||||
|
||||
- name: Link .jmri config from LCRR repo
|
||||
ansible.builtin.file:
|
||||
src: "{{ jmri_home }}/LCRR/.jmri"
|
||||
dest: "{{ jmri_home }}/.jmri"
|
||||
state: link
|
||||
owner: "{{ jmri_user }}"
|
||||
group: "{{ jmri_group }}"
|
||||
force: true
|
||||
when: jmri_lcrr_repo | length > 0
|
||||
notify: Restart jmri
|
||||
|
||||
- name: Check if JMRI is already installed at correct version
|
||||
ansible.builtin.stat:
|
||||
path: "{{ jmri_install_dir }}/JmriFaceless"
|
||||
@@ -79,9 +211,60 @@
|
||||
- Reload systemd
|
||||
- Restart jmri
|
||||
|
||||
- name: Enable and start JMRI service
|
||||
- name: Enable jmri service (monitor manages start/stop — do not start directly)
|
||||
ansible.builtin.systemd:
|
||||
name: jmri
|
||||
enabled: true
|
||||
state: started
|
||||
enabled: false
|
||||
daemon_reload: true
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 3 — X11 remote GUI access
|
||||
# ---------------------------------------------------------------------------
|
||||
- name: Install xauth (required for SSH X11 forwarding)
|
||||
ansible.builtin.apt:
|
||||
name: xauth
|
||||
state: present
|
||||
|
||||
- name: Remove old jmri X11 sshd drop-in if present (renamed)
|
||||
ansible.builtin.file:
|
||||
path: /etc/ssh/sshd_config.d/20-jmri-x11.conf
|
||||
state: absent
|
||||
notify: Restart sshd
|
||||
|
||||
- name: Deploy sshd drop-in to enable X11 forwarding (must load before hardening)
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/ssh/sshd_config.d/09-jmri-x11.conf
|
||||
owner: root
|
||||
group: root
|
||||
mode: '0644'
|
||||
content: |
|
||||
# Allow X11 forwarding for JMRI GUI sessions (jmri role)
|
||||
# Must be numbered below 10-mk-labs-hardening.conf — first match wins.
|
||||
X11Forwarding yes
|
||||
X11UseLocalhost yes
|
||||
notify: Restart sshd
|
||||
|
||||
- name: Deploy jmri-gui script
|
||||
ansible.builtin.template:
|
||||
src: jmri-gui.j2
|
||||
dest: /usr/local/bin/jmri-gui
|
||||
owner: root
|
||||
group: root
|
||||
mode: '0755'
|
||||
|
||||
- name: Deploy sudoers drop-in for jmri-gui (wed can manage jmri service)
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/sudoers.d/jmri-gui
|
||||
owner: root
|
||||
group: root
|
||||
mode: '0440'
|
||||
validate: 'visudo -cf %s'
|
||||
content: |
|
||||
# jmri user can manage its own service (for jmri-gui interactive sessions)
|
||||
jmri ALL=(root) NOPASSWD: /usr/bin/systemctl start jmri.service
|
||||
jmri ALL=(root) NOPASSWD: /usr/bin/systemctl stop jmri.service
|
||||
jmri ALL=(root) NOPASSWD: /opt/jmri-monitor/bin/python3
|
||||
# wed retains service control for automation/admin use
|
||||
wed ALL=(root) NOPASSWD: /usr/bin/systemctl start jmri.service
|
||||
wed ALL=(root) NOPASSWD: /usr/bin/systemctl stop jmri.service
|
||||
wed ALL=(root) NOPASSWD: /opt/jmri-monitor/bin/python3
|
||||
|
||||
25
ansible/roles/jmri/templates/99-jmri-devices.rules.j2
Normal file
25
ansible/roles/jmri/templates/99-jmri-devices.rules.j2
Normal file
@@ -0,0 +1,25 @@
|
||||
# /etc/udev/rules.d/99-jmri-devices.rules
|
||||
# Managed by Ansible — do not edit manually.
|
||||
#
|
||||
# Creates stable /dev/jmri/* symlinks for all three JMRI serial interfaces.
|
||||
# Uses ID_SERIAL (full serial string) to match by-id symlink identity,
|
||||
# so device numbering (ttyACM0, ttyACM1, etc.) doesn't matter after reboots.
|
||||
# Ownership: root:dialout mode 0660 — jmri user is a member of dialout.
|
||||
|
||||
# NCE command station (Microchip CDC)
|
||||
# Appears when layout is powered on; disappears when layout powers off.
|
||||
SUBSYSTEM=="tty", ENV{ID_SERIAL}=="{{ jmri_dev_nce_serial }}", \
|
||||
SYMLINK+="jmri/nce", \
|
||||
GROUP="dialout", MODE="0660"
|
||||
|
||||
# RR-CirKits LocoBuffer-NG (LocoNet bridge)
|
||||
# USB-powered — always present when server is running.
|
||||
SUBSYSTEM=="tty", ENV{ID_SERIAL}=="{{ jmri_dev_loconet_serial }}", \
|
||||
SYMLINK+="jmri/loconet", \
|
||||
GROUP="dialout", MODE="0660"
|
||||
|
||||
# FTDI USB serial (LCC interface)
|
||||
# USB-powered — always present when server is running.
|
||||
SUBSYSTEM=="tty", ENV{ID_SERIAL}=="{{ jmri_dev_lcc_serial }}", \
|
||||
SYMLINK+="jmri/lcc", \
|
||||
GROUP="dialout", MODE="0660"
|
||||
111
ansible/roles/jmri/templates/jmri-gui.j2
Normal file
111
ansible/roles/jmri/templates/jmri-gui.j2
Normal file
@@ -0,0 +1,111 @@
|
||||
#!/bin/bash
|
||||
# jmri-gui — launch JMRI GUI over SSH -X (X11 forwarding)
|
||||
# Managed by Ansible — do not edit manually.
|
||||
#
|
||||
# Usage (connect as jmri user):
|
||||
# ssh -X jmri@main-street-station jmri-gui panelpro
|
||||
# ssh -X jmri@main-street-station jmri-gui decoderpro
|
||||
# ssh -X jmri@main-street-station jmri-gui status
|
||||
#
|
||||
# ~/.ssh/config recommended:
|
||||
# Host main-street-station
|
||||
# User jmri
|
||||
# ForwardX11 yes
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
JMRI_DIR="{{ jmri_install_dir }}"
|
||||
JMRI_PROFILE="{{ jmri_profile_id }}"
|
||||
JMRI_SERVICE="jmri.service"
|
||||
MONITOR_SERVICE="jmri-monitor.service"
|
||||
|
||||
usage() {
|
||||
echo "Usage: jmri-gui <panelpro|decoderpro|status>"
|
||||
exit 1
|
||||
}
|
||||
|
||||
check_display() {
|
||||
if [ -z "${DISPLAY:-}" ]; then
|
||||
echo "ERROR: No DISPLAY set. Connect with: ssh -X jmri@main-street-station"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
status() {
|
||||
echo "=== JMRI daemon ==="
|
||||
systemctl status "$JMRI_SERVICE" --no-pager -l 2>&1 | head -8
|
||||
echo ""
|
||||
echo "=== Layout monitor ==="
|
||||
systemctl status "$MONITOR_SERVICE" --no-pager -l 2>&1 | head -5
|
||||
}
|
||||
|
||||
launch() {
|
||||
local app="$1"
|
||||
local binary
|
||||
|
||||
case "$app" in
|
||||
panelpro) binary="PanelPro" ;;
|
||||
decoderpro) binary="DecoderPro" ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
|
||||
check_display
|
||||
|
||||
# Stop the daemon if running
|
||||
if systemctl is-active --quiet "$JMRI_SERVICE" 2>/dev/null; then
|
||||
echo "Stopping JMRI daemon..."
|
||||
sudo systemctl stop "$JMRI_SERVICE"
|
||||
fi
|
||||
|
||||
echo "Launching $binary (profile: $JMRI_PROFILE)..."
|
||||
echo "Close the window to return to daemon mode."
|
||||
echo ""
|
||||
|
||||
# SSH sets DISPLAY to localhost:N (TCP) — translate to unix socket form
|
||||
# so Java's AWT xauth lookup matches the forwarded cookie.
|
||||
export DISPLAY=$(echo "$DISPLAY" | sed 's/localhost:/:/')
|
||||
|
||||
# Force AWT out of headless mode
|
||||
export JMRI_OPTIONS="-Djava.awt.headless=false"
|
||||
|
||||
# Run directly — we are already the jmri user, DISPLAY is set by SSH
|
||||
"$JMRI_DIR/$binary" --profile="$JMRI_PROFILE"
|
||||
|
||||
echo ""
|
||||
echo "$binary closed."
|
||||
|
||||
# Restart daemon if layout switch is still on
|
||||
if systemctl is-active --quiet "$MONITOR_SERVICE" 2>/dev/null; then
|
||||
if sudo /opt/jmri-monitor/bin/python3 - <<'EOF'
|
||||
from decora_wifi import DecoraWiFiSession
|
||||
from decora_wifi.models.residential_account import ResidentialAccount
|
||||
import sys
|
||||
session = DecoraWiFiSession()
|
||||
person = session.login("{{ jmri_leviton_email }}", "{{ jmri_leviton_password }}")
|
||||
perms = person.get_residential_permissions()
|
||||
for perm in perms:
|
||||
acct_id = perm.data.get('residentialAccountId')
|
||||
if not acct_id:
|
||||
continue
|
||||
acct = ResidentialAccount(session, acct_id)
|
||||
acct.refresh()
|
||||
for r in acct.get_residences():
|
||||
for s in r.get_iot_switches():
|
||||
if s.data.get('name') == '{{ jmri_leviton_switch_name }}':
|
||||
sys.exit(0 if s.data.get('power') == 'ON' else 1)
|
||||
sys.exit(1)
|
||||
EOF
|
||||
then
|
||||
echo "Layout is ON — restarting JMRI daemon."
|
||||
sudo systemctl start "$JMRI_SERVICE"
|
||||
else
|
||||
echo "Layout is OFF — JMRI daemon will not restart."
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
case "${1:-}" in
|
||||
panelpro|decoderpro) launch "$1" ;;
|
||||
status) status ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
156
ansible/roles/jmri/templates/jmri-monitor.py.j2
Normal file
156
ansible/roles/jmri/templates/jmri-monitor.py.j2
Normal file
@@ -0,0 +1,156 @@
|
||||
#!/opt/jmri-monitor/bin/python3
|
||||
"""
|
||||
jmri-monitor — Leviton Decora Smart Wi-Fi layout power monitor
|
||||
Managed by Ansible — do not edit manually.
|
||||
|
||||
Polls the Leviton cloud API for the "{{ jmri_leviton_switch_name }}" switch state.
|
||||
- Switch ON after being OFF → systemctl start jmri.service
|
||||
- Switch OFF for {{ jmri_monitor_stop_delay }}s → systemctl stop jmri.service
|
||||
|
||||
Quiet hours {{ jmri_monitor_quiet_start }}:00–{{ jmri_monitor_quiet_end }}:00: no polling (layout assumed off).
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
import time
|
||||
import logging
|
||||
import sys
|
||||
from datetime import datetime
|
||||
|
||||
LEVITON_EMAIL = "{{ jmri_leviton_email }}"
|
||||
LEVITON_PASSWORD = "{{ jmri_leviton_password }}"
|
||||
SWITCH_NAME = "{{ jmri_leviton_switch_name }}"
|
||||
POLL_INTERVAL = {{ jmri_monitor_poll_interval }}
|
||||
QUIET_START = {{ jmri_monitor_quiet_start }}
|
||||
QUIET_END = {{ jmri_monitor_quiet_end }}
|
||||
STOP_DELAY = {{ jmri_monitor_stop_delay }}
|
||||
JMRI_SERVICE = "jmri.service"
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s [jmri-monitor] %(levelname)s: %(message)s",
|
||||
datefmt="%Y-%m-%d %H:%M:%S",
|
||||
stream=sys.stdout,
|
||||
)
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def systemctl(action):
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["systemctl", action, JMRI_SERVICE],
|
||||
capture_output=True, text=True, timeout=30,
|
||||
)
|
||||
if result.returncode == 0:
|
||||
log.info("systemctl %s %s: OK", action, JMRI_SERVICE)
|
||||
else:
|
||||
log.warning("systemctl %s %s: %s", action, JMRI_SERVICE, result.stderr.strip())
|
||||
except Exception as e:
|
||||
log.error("systemctl %s failed: %s", action, e)
|
||||
|
||||
|
||||
def is_quiet_hours():
|
||||
hour = datetime.now().hour
|
||||
if QUIET_START < QUIET_END:
|
||||
return QUIET_START <= hour < QUIET_END
|
||||
else:
|
||||
# wraps midnight e.g. 23–6
|
||||
return hour >= QUIET_START or hour < QUIET_END
|
||||
|
||||
|
||||
def get_switch_state():
|
||||
"""Returns True if switch is ON, False if OFF, None on error."""
|
||||
try:
|
||||
from decora_wifi import DecoraWiFiSession
|
||||
from decora_wifi.models.residential_account import ResidentialAccount
|
||||
|
||||
session = DecoraWiFiSession()
|
||||
person = session.login(LEVITON_EMAIL, LEVITON_PASSWORD)
|
||||
if not person:
|
||||
log.error("Leviton login failed")
|
||||
return None
|
||||
|
||||
perms = person.get_residential_permissions()
|
||||
for perm in perms:
|
||||
acct_id = perm.data.get('residentialAccountId')
|
||||
if not acct_id:
|
||||
continue
|
||||
acct = ResidentialAccount(session, acct_id)
|
||||
acct.refresh()
|
||||
for residence in acct.get_residences():
|
||||
for switch in residence.get_iot_switches():
|
||||
if switch.data.get('name') == SWITCH_NAME:
|
||||
state = switch.data.get('power', 'OFF')
|
||||
session.call_api('/Person/logout', {}, 'post')
|
||||
return state == 'ON'
|
||||
|
||||
all_names = []
|
||||
for perm in perms:
|
||||
acct_id = perm.data.get('residentialAccountId')
|
||||
if acct_id:
|
||||
acct = ResidentialAccount(session, acct_id)
|
||||
acct.refresh()
|
||||
for r in acct.get_residences():
|
||||
all_names += [s.data.get('name') for s in r.get_iot_switches()]
|
||||
log.warning("Switch '%s' not found — available: %s", SWITCH_NAME, all_names)
|
||||
session.call_api('/Person/logout', {}, 'post')
|
||||
return None
|
||||
|
||||
except ImportError:
|
||||
log.error("decora_wifi not installed")
|
||||
return None
|
||||
except Exception as e:
|
||||
log.error("Error querying Leviton API: %s", e)
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
log.info("Layout power monitor starting")
|
||||
log.info("Switch: '%s' Poll: %ds Quiet: %02d:00–%02d:00 Stop delay: %ds",
|
||||
SWITCH_NAME, POLL_INTERVAL, QUIET_START, QUIET_END, STOP_DELAY)
|
||||
|
||||
layout_on = False
|
||||
off_since = None
|
||||
|
||||
while True:
|
||||
if is_quiet_hours():
|
||||
log.debug("Quiet hours — sleeping 60s")
|
||||
# If layout was on when quiet hours started, stop JMRI
|
||||
if layout_on:
|
||||
log.info("Quiet hours began — stopping JMRI")
|
||||
layout_on = False
|
||||
off_since = None
|
||||
systemctl("stop")
|
||||
time.sleep(60)
|
||||
continue
|
||||
|
||||
state = get_switch_state()
|
||||
|
||||
if state is True:
|
||||
off_since = None
|
||||
if not layout_on:
|
||||
log.info("Layout switch ON — starting JMRI")
|
||||
layout_on = True
|
||||
systemctl("start")
|
||||
|
||||
elif state is False:
|
||||
if layout_on:
|
||||
if off_since is None:
|
||||
off_since = time.monotonic()
|
||||
log.info("Layout switch OFF — waiting %ds before stopping JMRI", STOP_DELAY)
|
||||
elif time.monotonic() - off_since >= STOP_DELAY:
|
||||
log.info("Layout switch OFF for %ds — stopping JMRI", STOP_DELAY)
|
||||
layout_on = False
|
||||
off_since = None
|
||||
systemctl("stop")
|
||||
else:
|
||||
off_since = None
|
||||
|
||||
else:
|
||||
# API error — don't change state, try again next poll
|
||||
log.warning("Could not determine switch state — retrying in %ds", POLL_INTERVAL)
|
||||
|
||||
time.sleep(POLL_INTERVAL)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
20
ansible/roles/jmri/templates/jmri-monitor.service.j2
Normal file
20
ansible/roles/jmri/templates/jmri-monitor.service.j2
Normal file
@@ -0,0 +1,20 @@
|
||||
[Unit]
|
||||
Description=JMRI LocoNet Traffic Monitor
|
||||
Documentation=https://www.jmri.org/
|
||||
# Must start after udev has processed devices
|
||||
After=systemd-udev-settle.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
# Runs as root — needs to call systemctl start/stop jmri.service
|
||||
User=root
|
||||
ExecStart=/usr/local/bin/jmri-monitor
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
SyslogIdentifier=jmri-monitor
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -1,26 +1,21 @@
|
||||
[Unit]
|
||||
Description=JMRI Server (JmriFaceless)
|
||||
Documentation=https://www.jmri.org/
|
||||
After=network.target
|
||||
# Give USB devices time to enumerate after boot
|
||||
After=dev-{{ jmri_serial_device | basename }}.device
|
||||
Wants=dev-{{ jmri_serial_device | basename }}.device
|
||||
# jmri-monitor starts and stops this service based on LocoNet traffic.
|
||||
# Do NOT enable this unit directly — it is managed by jmri-monitor.service.
|
||||
After=network.target systemd-udev-settle.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User={{ jmri_user }}
|
||||
Group={{ jmri_group }}
|
||||
WorkingDirectory={{ jmri_install_dir }}
|
||||
|
||||
# JmriFaceless requires a profile ID
|
||||
# Find yours: ls ~/.jmri/profiles/
|
||||
ExecStart={{ jmri_install_dir }}/JmriFaceless --profile={{ jmri_profile_id }}
|
||||
|
||||
# Restart on failure, not on clean exit (e.g. intentional shutdown from Jython)
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
# Monitor owns lifecycle — do not auto-restart
|
||||
Restart=no
|
||||
|
||||
# Give hardware time to settle before retry
|
||||
# Give hardware time to settle on start
|
||||
TimeoutStartSec=30
|
||||
|
||||
# Logging — view with: journalctl -u jmri -f
|
||||
@@ -32,4 +27,5 @@ SyslogIdentifier=jmri
|
||||
SupplementaryGroups=dialout
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
# Intentionally no WantedBy — started only by jmri-monitor
|
||||
WantedBy=
|
||||
|
||||
337
cluster/applications/firecrawl/ARCHITECTURE.md
Normal file
337
cluster/applications/firecrawl/ARCHITECTURE.md
Normal file
@@ -0,0 +1,337 @@
|
||||
# Firecrawl Architecture Diagram
|
||||
|
||||
## High-Level Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ External Access │
|
||||
│ spaceship-earth.local.mk-labs.cloud / firecrawl.local.mk-labs.cloud │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────┐ │
|
||||
│ │ Gateway API │ │
|
||||
│ │ (HTTPRoute) │ │
|
||||
│ │ TLS Termination │ │
|
||||
│ └────────┬─────────┘ │
|
||||
└─────────────────────────────────┼──────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────────────────┼──────────────────────────────────────┐
|
||||
│ Firecrawl Namespace │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────┐ │
|
||||
│ │ API Service │ │
|
||||
│ │ (ClusterIP) │ │
|
||||
│ │ Port 3002 │ │
|
||||
│ └────────┬─────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────────────┼──────────────────┐ │
|
||||
│ │ │ │ │
|
||||
│ ▼ ▼ ▼ │
|
||||
│ ┌─────────────────┐ ┌─────────────┐ ┌──────────────────┐ │
|
||||
│ │ API Deployment │ │ Worker │ │ NUQ Worker │ │
|
||||
│ │ (firecrawl-api)│ │ Deployment │ │ Deployment │ │
|
||||
│ │ │ │(firecrawl- │ │ (firecrawl-api) │ │
|
||||
│ │ Entrypoint: │ │ api) │ │ │ │
|
||||
│ │ dist/src/ │ │ │ │ Entrypoint: │ │
|
||||
│ │ index.js │ │ Entrypoint: │ │ dist/src/ │ │
|
||||
│ │ │ │ dist/src/ │ │ services/worker/│ │
|
||||
│ │ 4-6GB / 2 CPU │ │ services/ │ │ nuq-worker.js │ │
|
||||
│ │ │ │ queue- │ │ │ │
|
||||
│ │ Replicas: 1 │ │ worker.js │ │ 3-4GB / 1 CPU │ │
|
||||
│ │ │ │ │ │ │ │
|
||||
│ │ Health: │ │ 3-4GB/1 CPU │ │ Replicas: 1 │ │
|
||||
│ │ /v0/health/* │ │ │ │ │ │
|
||||
│ │ │ │ Replicas: 1 │ │ │ │
|
||||
│ └────────┬────────┘ └──────┬──────┘ └────────┬─────────┘ │
|
||||
│ │ │ │ │
|
||||
│ └─────────┬────────┴──────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌─────────┼──────────────┬──────────────┐ │
|
||||
│ │ │ │ │ │
|
||||
│ ▼ ▼ ▼ ▼ │
|
||||
│ ┌────────────┐ ┌──────────┐ ┌──────────┐ ┌───────────────┐ │
|
||||
│ │ Playwright │ │ Redis │ │PostgreSQL│ │ RabbitMQ │ │
|
||||
│ │ Service │ │ │ │ │ │ │ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ Deployment │ │Deployment│ │StatefulSet│ │ Deployment │ │
|
||||
│ │ (Harbor) │ │(Upstream)│ │ (Harbor) │ │ (Upstream) │ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ Service: │ │ Service: │ │ Service: │ │ Service: │ │
|
||||
│ │ 3000 │ │ 6379 │ │ 5432 │ │ 5672, 15672 │ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ 4GB/2 CPU │ │ 1GB/0.5 │ │ 2GB/1 CPU│ │ 1GB/0.5 CPU │ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ tmpfs: │ │ │ │ PVC: │ │ Healthcheck: │ │
|
||||
│ │ 1GB │ │ │ │ 10GB │ │ Required │ │
|
||||
│ └────────────┘ └──────────┘ └──────────┘ └───────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────┐ │
|
||||
│ │ NFS PVC │ │
|
||||
│ │ 10GB │ │
|
||||
│ │(nfs- │ │
|
||||
│ │emporium) │ │
|
||||
│ └──────────┘ │
|
||||
│ │
|
||||
│ Configuration: │
|
||||
│ ┌──────────────┐ ┌────────────────┐ │
|
||||
│ │ ConfigMap │ │ ExternalSecret │ │
|
||||
│ │ (firecrawl- │ │ (firecrawl- │ │
|
||||
│ │ config) │ │ secrets) │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ - URLs │ │ ┌──────────┐ │ │
|
||||
│ │ - Ports │ │ │1Password │ │ │
|
||||
│ │ - Tuning │ │ │ Vault │ │ │
|
||||
│ │ │ │ └────┬─────┘ │ │
|
||||
│ └──────────────┘ │ │ │ │
|
||||
│ │ ▼ │ │
|
||||
│ │ - postgres- │ │
|
||||
│ │ password │ │
|
||||
│ │ - bull-auth- │ │
|
||||
│ │ key │ │
|
||||
│ └────────────────┘ │
|
||||
└────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Service Communication Flow
|
||||
|
||||
### API Request Flow
|
||||
```
|
||||
External User
|
||||
│
|
||||
▼
|
||||
Gateway API (TLS termination)
|
||||
│
|
||||
▼
|
||||
API Service (port 3002)
|
||||
│
|
||||
├─────► Playwright Service (browser automation)
|
||||
│ └─► Returns rendered HTML/Markdown
|
||||
│
|
||||
├─────► Redis (queue jobs, cache results)
|
||||
│
|
||||
├─────► PostgreSQL (store job metadata)
|
||||
│
|
||||
└─────► RabbitMQ (publish job events)
|
||||
```
|
||||
|
||||
### Background Job Processing Flow
|
||||
```
|
||||
API receives request
|
||||
│
|
||||
▼
|
||||
Job queued in Redis
|
||||
│
|
||||
▼
|
||||
RabbitMQ notifies workers
|
||||
│
|
||||
├─────► Worker picks up job
|
||||
│ └─► Processes scraping tasks
|
||||
│
|
||||
└─────► NUQ Worker picks up database jobs
|
||||
└─► Processes queue from PostgreSQL
|
||||
```
|
||||
|
||||
### Database Queue Flow (NUQ)
|
||||
```
|
||||
Job created in PostgreSQL (nuq.queue_scrape table)
|
||||
│
|
||||
▼
|
||||
NUQ Worker polls for jobs (prefetch)
|
||||
│
|
||||
├─► Status: queued → active
|
||||
│
|
||||
├─► Worker processes job
|
||||
│ └─► Calls Playwright or direct fetch
|
||||
│
|
||||
└─► Status: active → completed/failed
|
||||
└─► Results stored in returnvalue column
|
||||
```
|
||||
|
||||
## Build Pipeline Flow
|
||||
|
||||
```
|
||||
GitHub: mendableai/firecrawl
|
||||
│
|
||||
▼
|
||||
Tekton Pipeline (innoventions namespace)
|
||||
│
|
||||
├─────► firecrawl-api-build
|
||||
│ │
|
||||
│ ├─► Git Clone Task
|
||||
│ │
|
||||
│ ├─► Kaniko Build Task
|
||||
│ │ └─► Multi-stage: Go → Node → Runtime
|
||||
│ │
|
||||
│ └─► Push to Harbor
|
||||
│ └─► the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest
|
||||
│
|
||||
├─────► firecrawl-playwright-build
|
||||
│ │
|
||||
│ ├─► Git Clone Task
|
||||
│ │
|
||||
│ ├─► Kaniko Build Task
|
||||
│ │ └─► Node.js + Chromium install
|
||||
│ │
|
||||
│ └─► Push to Harbor
|
||||
│ └─► .../firecrawl-playwright:latest
|
||||
│
|
||||
└─────► firecrawl-postgres-build
|
||||
│
|
||||
├─► Git Clone Task
|
||||
│
|
||||
├─► Kaniko Build Task
|
||||
│ └─► postgres:16 + pg_cron + nuq.sql
|
||||
│
|
||||
└─► Push to Harbor
|
||||
└─► .../firecrawl-postgres:latest
|
||||
```
|
||||
|
||||
## Deployment Flow (ArgoCD)
|
||||
|
||||
```
|
||||
Gitea Repository (homelab)
|
||||
│
|
||||
└─► cluster/applications/firecrawl/
|
||||
│
|
||||
▼
|
||||
ArgoCD Application (sync)
|
||||
│
|
||||
├─► Wave 0: Namespace
|
||||
│
|
||||
├─► Wave 1: ConfigMap, ExternalSecret
|
||||
│
|
||||
├─► Wave 2: PostgreSQL StatefulSet + PVC
|
||||
│ Redis Deployment
|
||||
│ RabbitMQ Deployment
|
||||
│
|
||||
├─► Wave 3: Playwright Deployment
|
||||
│ (waits for infrastructure)
|
||||
│
|
||||
├─► Wave 4: API Deployment
|
||||
│ Worker Deployments
|
||||
│ (waits for all dependencies)
|
||||
│
|
||||
└─► Wave 5: Services, HTTPRoute
|
||||
```
|
||||
|
||||
## Resource Distribution
|
||||
|
||||
```
|
||||
Total Cluster Capacity: ~48 CPU / ~96GB RAM (6 nodes)
|
||||
|
||||
Firecrawl Allocation:
|
||||
┌────────────────────────────────────┐
|
||||
│ API: 2 CPU / 4-6GB │ ████████████
|
||||
│ Worker: 1 CPU / 3-4GB │ ██████
|
||||
│ NUQ Worker: 1 CPU / 3-4GB │ ██████
|
||||
│ Playwright: 2 CPU / 4GB │ ████████████
|
||||
│ PostgreSQL: 1 CPU / 2GB │ ██████
|
||||
│ Redis: 0.5 CPU / 1GB │ ███
|
||||
│ RabbitMQ: 0.5 CPU / 1GB │ ███
|
||||
├────────────────────────────────────┤
|
||||
│ TOTAL: 8 CPU / 22GB RAM │
|
||||
└────────────────────────────────────┘
|
||||
|
||||
Percentage of cluster: ~17% CPU, ~23% RAM
|
||||
Headroom available: ✅ Excellent
|
||||
```
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Scrape Request Example
|
||||
```
|
||||
1. User → POST /v1/scrape {"url": "https://example.com"}
|
||||
│
|
||||
2. API validates request
|
||||
│
|
||||
3. API creates job in PostgreSQL (nuq.queue_scrape)
|
||||
│
|
||||
4. API queues job in Redis
|
||||
│
|
||||
5. RabbitMQ notifies workers
|
||||
│
|
||||
6. Worker picks up job
|
||||
│
|
||||
7. Worker calls Playwright service
|
||||
│ └─► Playwright launches Chromium
|
||||
│ └─► Renders page (handles JS)
|
||||
│ └─► Returns HTML
|
||||
│
|
||||
8. Worker converts HTML → Markdown (Go library)
|
||||
│
|
||||
9. Worker stores result in PostgreSQL (returnvalue column)
|
||||
│
|
||||
10. Worker updates job status: completed
|
||||
│
|
||||
11. API returns result to user
|
||||
└─► {"markdown": "...", "html": "...", "metadata": {...}}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Network Policies (Future Enhancement)
|
||||
|
||||
```
|
||||
firecrawl namespace:
|
||||
│
|
||||
├─► Ingress Rules:
|
||||
│ ├─ Allow: Gateway API → API Service (port 3002)
|
||||
│ └─ Deny: All other external traffic
|
||||
│
|
||||
├─► Egress Rules:
|
||||
│ ├─ Allow: API → Playwright (port 3000)
|
||||
│ ├─ Allow: API → Redis (port 6379)
|
||||
│ ├─ Allow: API → PostgreSQL (port 5432)
|
||||
│ ├─ Allow: API → RabbitMQ (port 5672)
|
||||
│ ├─ Allow: All → Internet (for web scraping)
|
||||
│ └─ Deny: All other cluster traffic
|
||||
│
|
||||
└─► Inter-Pod Rules:
|
||||
├─ Allow: API → All infrastructure services
|
||||
├─ Allow: Workers → All infrastructure services
|
||||
├─ Deny: PostgreSQL → Internet (security)
|
||||
└─ Deny: Redis → Internet (security)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Observability (Future Enhancement)
|
||||
|
||||
```
|
||||
Prometheus Metrics:
|
||||
│
|
||||
├─► API Metrics (port 3002/metrics)
|
||||
│ ├─ Request rate
|
||||
│ ├─ Response times
|
||||
│ ├─ Job queue depth
|
||||
│ └─ Error rates
|
||||
│
|
||||
├─► Worker Metrics (port 3005/metrics)
|
||||
│ ├─ Jobs processed
|
||||
│ ├─ Processing times
|
||||
│ └─ Success/failure rates
|
||||
│
|
||||
├─► PostgreSQL Metrics
|
||||
│ ├─ Connection pool usage
|
||||
│ ├─ Query performance
|
||||
│ └─ Table sizes
|
||||
│
|
||||
└─► Playwright Metrics
|
||||
├─ Browser pool usage
|
||||
├─ Page load times
|
||||
└─ Chromium memory usage
|
||||
|
||||
Grafana Dashboards:
|
||||
├─ Firecrawl Overview
|
||||
├─ Job Processing Metrics
|
||||
├─ Service Health
|
||||
└─ Resource Utilization
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Diagram Version:** 1.0
|
||||
**Last Updated:** June 6, 2026
|
||||
**Created By:** Rocket Raccoon (CI/CD Specialist)
|
||||
412
cluster/applications/firecrawl/ENVIRONMENT_VARIABLES.md
Normal file
412
cluster/applications/firecrawl/ENVIRONMENT_VARIABLES.md
Normal file
@@ -0,0 +1,412 @@
|
||||
# Firecrawl Environment Variables Reference
|
||||
|
||||
Complete reference for all environment variables used by Firecrawl services.
|
||||
|
||||
---
|
||||
|
||||
## Required Variables (CRITICAL)
|
||||
|
||||
These variables MUST be set for Firecrawl to function.
|
||||
|
||||
### Server Configuration
|
||||
```yaml
|
||||
HOST: "0.0.0.0" # Listen address
|
||||
PORT: "3002" # Main API port
|
||||
WORKER_PORT: "3005" # Worker liveness check port
|
||||
EXTRACT_WORKER_PORT: "3004" # Extract worker port
|
||||
```
|
||||
|
||||
### Database (PostgreSQL)
|
||||
```yaml
|
||||
POSTGRES_USER: "postgres" # Database username
|
||||
POSTGRES_PASSWORD: "<SECRET>" # 🔐 Database password (from 1Password)
|
||||
POSTGRES_DB: "postgres" # Database name
|
||||
POSTGRES_HOST: "nuq-postgres" # K8s service name
|
||||
POSTGRES_PORT: "5432" # PostgreSQL port
|
||||
```
|
||||
|
||||
### Redis (Queue & Cache)
|
||||
```yaml
|
||||
REDIS_URL: "redis://redis:6379"
|
||||
REDIS_RATE_LIMIT_URL: "redis://redis:6379"
|
||||
```
|
||||
|
||||
### RabbitMQ (Message Broker)
|
||||
```yaml
|
||||
NUQ_RABBITMQ_URL: "amqp://rabbitmq:5672"
|
||||
```
|
||||
|
||||
### Playwright Service
|
||||
```yaml
|
||||
PLAYWRIGHT_MICROSERVICE_URL: "http://playwright-service:3000/scrape"
|
||||
```
|
||||
|
||||
### Authentication
|
||||
```yaml
|
||||
USE_DB_AUTHENTICATION: "false" # Set "true" for production with Supabase
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Variables (REQUIRED)
|
||||
|
||||
### Admin UI Protection
|
||||
```yaml
|
||||
BULL_AUTH_KEY: "<SECRET>" # 🔐 Queue admin UI password (from 1Password)
|
||||
# URL: /admin/{BULL_AUTH_KEY}/queues
|
||||
```
|
||||
|
||||
### API Testing
|
||||
```yaml
|
||||
TEST_API_KEY: "<SECRET>" # 🔐 Optional - API key for testing (from 1Password)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Optional Variables (Features)
|
||||
|
||||
### AI Features (JSON Format, Extract API)
|
||||
```yaml
|
||||
# OpenAI Configuration
|
||||
OPENAI_API_KEY: "<SECRET>" # 🔐 OpenAI API key (from 1Password)
|
||||
OPENAI_BASE_URL: "" # Custom OpenAI-compatible endpoint
|
||||
MODEL_NAME: "" # Override default model (e.g., gpt-4)
|
||||
MODEL_EMBEDDING_NAME: "" # Override embedding model
|
||||
|
||||
# Ollama (Alternative to OpenAI)
|
||||
OLLAMA_BASE_URL: "" # E.g., http://localhost:11434/api
|
||||
# When using Ollama, set:
|
||||
# MODEL_NAME: "deepseek-r1:7b"
|
||||
# MODEL_EMBEDDING_NAME: "nomic-embed-text"
|
||||
```
|
||||
|
||||
### Proxy Configuration
|
||||
```yaml
|
||||
PROXY_SERVER: "" # Full URL (http://0.1.2.3:1234) or IP:port
|
||||
PROXY_USERNAME: "" # 🔐 Proxy username (from 1Password)
|
||||
PROXY_PASSWORD: "" # 🔐 Proxy password (from 1Password)
|
||||
```
|
||||
|
||||
### Search API Configuration
|
||||
```yaml
|
||||
# By default, uses Google search
|
||||
# Optionally use SearXNG instead:
|
||||
SEARXNG_ENDPOINT: "" # E.g., http://your.searxng.server
|
||||
SEARXNG_ENGINES: "" # Comma-separated engine list
|
||||
SEARXNG_CATEGORIES: "" # Comma-separated categories
|
||||
```
|
||||
|
||||
### Monitoring & Logging
|
||||
```yaml
|
||||
LOGGING_LEVEL: "info" # debug, info, warn, error
|
||||
SLACK_WEBHOOK_URL: "" # 🔐 Slack webhook for alerts (from 1Password)
|
||||
```
|
||||
|
||||
### Supabase Integration (Advanced)
|
||||
```yaml
|
||||
# Note: Not currently configurable for self-hosted instances
|
||||
SUPABASE_ANON_TOKEN: "" # For DB authentication
|
||||
SUPABASE_URL: "" # Supabase project URL
|
||||
SUPABASE_SERVICE_TOKEN: "" # Supabase service role key
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### Worker Pools
|
||||
```yaml
|
||||
NUM_WORKERS_PER_QUEUE: "8" # Number of workers per queue
|
||||
CRAWL_CONCURRENT_REQUESTS: "10" # Concurrent crawl requests
|
||||
MAX_CONCURRENT_JOBS: "5" # Maximum concurrent jobs
|
||||
```
|
||||
|
||||
### Browser Pool (Playwright)
|
||||
```yaml
|
||||
BROWSER_POOL_SIZE: "5" # Browser instance pool size
|
||||
MAX_CONCURRENT_PAGES: "10" # Max concurrent pages per browser
|
||||
```
|
||||
|
||||
### Resource Limits (Self-Protection)
|
||||
```yaml
|
||||
MAX_CPU: "0.8" # 0.0-1.0, reject jobs above threshold
|
||||
MAX_RAM: "0.8" # 0.0-1.0, reject jobs above threshold
|
||||
```
|
||||
|
||||
### Timeouts
|
||||
```yaml
|
||||
HARNESS_STARTUP_TIMEOUT_MS: "60000" # 60 seconds startup timeout
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Playwright-Specific Variables
|
||||
|
||||
Set on `playwright-service` pods only:
|
||||
|
||||
```yaml
|
||||
PORT: "3000" # Playwright service port
|
||||
PROXY_SERVER: "" # Same as API proxy (if needed)
|
||||
PROXY_USERNAME: "" # Same as API proxy (if needed)
|
||||
PROXY_PASSWORD: "" # Same as API proxy (if needed)
|
||||
ALLOW_LOCAL_WEBHOOKS: "false" # Security: block local webhooks
|
||||
BLOCK_MEDIA: "" # Optional: block media resources
|
||||
MAX_CONCURRENT_PAGES: "10" # Browser concurrency (same as API)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Environment-Specific Variables
|
||||
|
||||
### Development
|
||||
```yaml
|
||||
ENV: "local"
|
||||
```
|
||||
|
||||
### Production
|
||||
```yaml
|
||||
ENV: "production"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## FlyIO-Specific Variables (Not Used in K8s)
|
||||
|
||||
These variables are set by FlyIO platform and not needed for Kubernetes deployment:
|
||||
|
||||
```yaml
|
||||
FLY_PROCESS_GROUP: "app" # Not used in K8s
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deprecated / Unused Variables
|
||||
|
||||
Variables found in examples but not required for self-hosted deployment:
|
||||
|
||||
```yaml
|
||||
AUTUMN_SECRET_KEY: "" # Mendable platform-specific
|
||||
SELF_HOSTED_WEBHOOK_URL: "" # Custom webhook endpoint
|
||||
LLAMAPARSE_API_KEY: "" # PDF parsing service (optional)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Variable Precedence
|
||||
|
||||
1. **ExternalSecret** (from 1Password) - Highest priority for secrets
|
||||
2. **ConfigMap** - Non-sensitive configuration
|
||||
3. **Deployment env:** - Direct environment variables (override)
|
||||
4. **Dockerfile defaults** - Lowest priority
|
||||
|
||||
---
|
||||
|
||||
## ConfigMap Example
|
||||
|
||||
Non-sensitive variables suitable for ConfigMap:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: firecrawl-config
|
||||
namespace: firecrawl
|
||||
data:
|
||||
# Service URLs
|
||||
REDIS_URL: "redis://redis:6379"
|
||||
REDIS_RATE_LIMIT_URL: "redis://redis:6379"
|
||||
POSTGRES_HOST: "nuq-postgres"
|
||||
POSTGRES_PORT: "5432"
|
||||
POSTGRES_USER: "postgres"
|
||||
POSTGRES_DB: "postgres"
|
||||
NUQ_RABBITMQ_URL: "amqp://rabbitmq:5672"
|
||||
PLAYWRIGHT_MICROSERVICE_URL: "http://playwright-service:3000/scrape"
|
||||
|
||||
# Server Configuration
|
||||
HOST: "0.0.0.0"
|
||||
PORT: "3002"
|
||||
WORKER_PORT: "3005"
|
||||
EXTRACT_WORKER_PORT: "3004"
|
||||
USE_DB_AUTHENTICATION: "false"
|
||||
ENV: "production"
|
||||
|
||||
# Performance Tuning
|
||||
NUM_WORKERS_PER_QUEUE: "8"
|
||||
CRAWL_CONCURRENT_REQUESTS: "10"
|
||||
MAX_CONCURRENT_JOBS: "5"
|
||||
BROWSER_POOL_SIZE: "5"
|
||||
MAX_CONCURRENT_PAGES: "10"
|
||||
HARNESS_STARTUP_TIMEOUT_MS: "60000"
|
||||
|
||||
# Logging
|
||||
LOGGING_LEVEL: "info"
|
||||
|
||||
# Security
|
||||
ALLOW_LOCAL_WEBHOOKS: "false"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ExternalSecret Example
|
||||
|
||||
Sensitive variables synced from 1Password:
|
||||
|
||||
```yaml
|
||||
apiVersion: external-secrets.io/v1beta1
|
||||
kind: ExternalSecret
|
||||
metadata:
|
||||
name: firecrawl-secrets
|
||||
namespace: firecrawl
|
||||
spec:
|
||||
refreshInterval: 1h
|
||||
secretStoreRef:
|
||||
name: onepassword-connect
|
||||
kind: ClusterSecretStore
|
||||
target:
|
||||
name: firecrawl-secrets
|
||||
creationPolicy: Owner
|
||||
data:
|
||||
# Required secrets
|
||||
- secretKey: POSTGRES_PASSWORD
|
||||
remoteRef:
|
||||
key: firecrawl
|
||||
property: postgres-password
|
||||
|
||||
- secretKey: BULL_AUTH_KEY
|
||||
remoteRef:
|
||||
key: firecrawl
|
||||
property: bull-auth-key
|
||||
|
||||
# Optional secrets (uncomment when added to 1Password)
|
||||
# - secretKey: OPENAI_API_KEY
|
||||
# remoteRef:
|
||||
# key: firecrawl
|
||||
# property: openai-api-key
|
||||
|
||||
# - secretKey: TEST_API_KEY
|
||||
# remoteRef:
|
||||
# key: firecrawl
|
||||
# property: test-api-key
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Usage in Deployments
|
||||
|
||||
### Combining ConfigMap and Secret
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: api
|
||||
namespace: firecrawl
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: api
|
||||
image: the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest
|
||||
envFrom:
|
||||
# Load all non-sensitive variables
|
||||
- configMapRef:
|
||||
name: firecrawl-config
|
||||
# Load all secrets
|
||||
- secretRef:
|
||||
name: firecrawl-secrets
|
||||
env:
|
||||
# Override specific variables if needed
|
||||
- name: FLY_PROCESS_GROUP
|
||||
value: "app"
|
||||
```
|
||||
|
||||
### Playwright-Specific ConfigMap
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: playwright-service-config
|
||||
namespace: firecrawl
|
||||
data:
|
||||
PORT: "3000"
|
||||
ALLOW_LOCAL_WEBHOOKS: "false"
|
||||
MAX_CONCURRENT_PAGES: "10"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation Checklist
|
||||
|
||||
Before deployment, ensure:
|
||||
|
||||
- [ ] `POSTGRES_PASSWORD` set in 1Password
|
||||
- [ ] `BULL_AUTH_KEY` set in 1Password (strong random value)
|
||||
- [ ] All service URLs use correct K8s service names
|
||||
- [ ] Ports match service definitions (3002, 3000, 5432, 6379, 5672)
|
||||
- [ ] `USE_DB_AUTHENTICATION` is "false" (Supabase not available)
|
||||
- [ ] `ENV` is "production" (not "local")
|
||||
- [ ] Worker pool sizes appropriate for cluster capacity
|
||||
- [ ] `ALLOW_LOCAL_WEBHOOKS` is "false" (security)
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Supabase client is not configured"
|
||||
**Expected warning** - Supabase is not available for self-hosted instances. Safe to ignore.
|
||||
|
||||
### "You're bypassing authentication"
|
||||
**Expected warning** - When `USE_DB_AUTHENTICATION=false`. Normal for self-hosted deployment.
|
||||
|
||||
### Connection refused errors
|
||||
**Check:**
|
||||
- Service names match environment variables
|
||||
- Services are running: `kubectl get svc -n firecrawl`
|
||||
- Pods are ready: `kubectl get pods -n firecrawl`
|
||||
|
||||
### Build failures
|
||||
**Check:**
|
||||
- Required build args are set
|
||||
- Kaniko has sufficient memory (4GB for API build)
|
||||
|
||||
---
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
1. **Generate strong secrets:**
|
||||
```bash
|
||||
# Generate BULL_AUTH_KEY
|
||||
openssl rand -base64 32
|
||||
|
||||
# Generate POSTGRES_PASSWORD
|
||||
openssl rand -base64 24
|
||||
```
|
||||
|
||||
2. **Never expose PostgreSQL externally:**
|
||||
- Use ClusterIP service only
|
||||
- No LoadBalancer or NodePort
|
||||
- Access via kubectl port-forward for maintenance
|
||||
|
||||
3. **Protect admin UI:**
|
||||
- Keep BULL_AUTH_KEY secret
|
||||
- Don't commit to git
|
||||
- Rotate periodically
|
||||
|
||||
4. **Use ExternalSecrets:**
|
||||
- Never put secrets in ConfigMaps
|
||||
- Never commit secrets to git
|
||||
- Store all sensitive data in 1Password
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- **Upstream .env Template:** `/tmp/firecrawl/SELF_HOST.md`
|
||||
- **Docker Compose Reference:** `/tmp/firecrawl/docker-compose.yaml`
|
||||
- **K8s Example ConfigMap:** `/tmp/firecrawl/examples/kubernetes/cluster-install/configmap.yaml`
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** June 6, 2026
|
||||
**Maintained By:** Rocket Raccoon (CI/CD Specialist)
|
||||
375
cluster/applications/firecrawl/README.md
Normal file
375
cluster/applications/firecrawl/README.md
Normal file
@@ -0,0 +1,375 @@
|
||||
# Firecrawl - Web Scraping & Crawling Service
|
||||
|
||||
**Project:** Platform Buildout - Firecrawl Deployment
|
||||
**Service Name:** Spaceship Earth (EPCOT themed)
|
||||
**DNS:** spaceship-earth.local.mk-labs.cloud (primary), firecrawl.local.mk-labs.cloud (alias)
|
||||
**Namespace:** firecrawl
|
||||
**Owner:** Rocket Raccoon (CI/CD Specialist)
|
||||
**Status:** 🚧 IN PROGRESS - Day 1 Complete
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Firecrawl is a self-hosted web scraping and crawling API that converts URLs to LLM-ready content (Markdown, JSON, HTML). This deployment enables JARVIS web search capability.
|
||||
|
||||
**Upstream:** https://github.com/mendableai/firecrawl
|
||||
**License:** AGPLv3 (open source)
|
||||
|
||||
### Capabilities
|
||||
- **Search:** Search the web and get full page content
|
||||
- **Scrape:** Convert URLs to markdown, HTML, screenshots, or structured JSON
|
||||
- **Crawl:** Scrape all URLs of a website with a single request
|
||||
- **Interact:** Click, scroll, write, wait before extracting (JS-heavy sites)
|
||||
- **Map:** Discover all URLs on a website
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### Services (5 components)
|
||||
|
||||
| Service | Image | CPU | Memory | Storage | Purpose |
|
||||
|---------|-------|-----|--------|---------|---------|
|
||||
| **API** | Harbor: firecrawl-api:latest | 2.0 | 4-6GB | - | Main REST API |
|
||||
| **Worker** | Harbor: firecrawl-api:latest | 1.0 | 3-4GB | - | Background job processor |
|
||||
| **NUQ Worker** | Harbor: firecrawl-api:latest | 1.0 | 3-4GB | - | Database queue worker |
|
||||
| **Playwright** | Harbor: firecrawl-playwright:latest | 2.0 | 4GB | 1GB tmpfs | Browser automation |
|
||||
| **PostgreSQL** | Harbor: firecrawl-postgres:latest | 1.0 | 2GB | 10GB PVC | Data storage |
|
||||
| **Redis** | Upstream: redis:alpine | 0.5 | 1GB | - | Queue & cache |
|
||||
| **RabbitMQ** | Upstream: rabbitmq:3-management | 0.5 | 1GB | - | Message broker |
|
||||
|
||||
**Total Resources:** ~8 CPU, ~22GB RAM, 11GB storage
|
||||
|
||||
### Service Dependencies
|
||||
|
||||
```
|
||||
API Service ─┬─► Redis (queue/cache)
|
||||
├─► PostgreSQL (data storage)
|
||||
├─► RabbitMQ (message broker) ⚠️ HEALTH CHECK REQUIRED
|
||||
└─► Playwright Service (browser automation)
|
||||
|
||||
Worker ──────┬─► Redis
|
||||
├─► PostgreSQL
|
||||
└─► RabbitMQ
|
||||
|
||||
NUQ Worker ──┴─► PostgreSQL
|
||||
```
|
||||
|
||||
**Startup Order:**
|
||||
1. Redis, PostgreSQL, RabbitMQ (infrastructure)
|
||||
2. Playwright Service
|
||||
3. API, Workers (after all dependencies ready)
|
||||
|
||||
---
|
||||
|
||||
## Build Strategy
|
||||
|
||||
**Hybrid Approach:** Build custom images via Tekton, use upstream for infrastructure.
|
||||
|
||||
### Custom Builds (Tekton → Harbor)
|
||||
|
||||
1. **firecrawl-api** (Multi-stage: Go + Node.js + Rust)
|
||||
- Source: `apps/api/Dockerfile`
|
||||
- Registry: `the-seas.local.mk-labs.cloud/applications/firecrawl-api:latest`
|
||||
- Build time: ~15 minutes (first), ~5 minutes (cached)
|
||||
- Used by: API, Worker, NUQ Worker (different entrypoints)
|
||||
|
||||
2. **firecrawl-playwright** (Node.js + Chromium)
|
||||
- Source: `apps/playwright-service-ts/Dockerfile`
|
||||
- Registry: `the-seas.local.mk-labs.cloud/applications/firecrawl-playwright:latest`
|
||||
- Build time: ~10 minutes
|
||||
|
||||
3. **firecrawl-postgres** (PostgreSQL + pg_cron + init script)
|
||||
- Source: `apps/nuq-postgres/Dockerfile`
|
||||
- Registry: `the-seas.local.mk-labs.cloud/applications/firecrawl-postgres:latest`
|
||||
- Build time: ~3 minutes
|
||||
- Note: Custom build required for pg_cron extension and nuq.sql schema
|
||||
|
||||
### Upstream Images
|
||||
|
||||
- **Redis:** `redis:alpine`
|
||||
- **RabbitMQ:** `rabbitmq:3-management`
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### Required Secrets (1Password)
|
||||
|
||||
Managed via ExternalSecret → 1Password vault item: `firecrawl`
|
||||
|
||||
- `POSTGRES_PASSWORD` - Database password (CRITICAL)
|
||||
- `BULL_AUTH_KEY` - Queue admin UI authentication (CRITICAL)
|
||||
- `OPENAI_API_KEY` - Optional, for AI features
|
||||
- `TEST_API_KEY` - Optional, for testing
|
||||
|
||||
### ConfigMap (Non-sensitive)
|
||||
|
||||
- Service URLs (Redis, PostgreSQL, RabbitMQ, Playwright)
|
||||
- Port configuration (3002 API, 3005 Worker)
|
||||
- Performance tuning (worker pools, concurrency limits)
|
||||
- Logging level
|
||||
|
||||
### Environment Variables Reference
|
||||
|
||||
See [ENVIRONMENT_VARIABLES.md](./ENVIRONMENT_VARIABLES.md) for complete list.
|
||||
|
||||
---
|
||||
|
||||
## Deployment Timeline
|
||||
|
||||
### Day 1 (June 6) - Investigation & Planning ✅
|
||||
- Repository analysis complete
|
||||
- Architecture decisions finalized
|
||||
- Environment configuration researched
|
||||
- Day 2 plan created
|
||||
|
||||
### Day 2 (June 7) - Tekton Pipelines 🚧
|
||||
- Create 3 build pipelines (API, Playwright, PostgreSQL)
|
||||
- Test builds and push to Harbor
|
||||
- Validate image integrity
|
||||
|
||||
### Day 3 (June 8) - Kubernetes Manifests
|
||||
- Create Deployments, Services, StatefulSets
|
||||
- Configure ConfigMaps and ExternalSecrets
|
||||
- Set up HTTPRoute for ingress
|
||||
|
||||
### Day 4 (June 9) - Secrets & Configuration
|
||||
- Create 1Password vault item
|
||||
- Configure ExternalSecret sync
|
||||
- Validate configuration
|
||||
|
||||
### Day 5 (June 10) - Deployment & Testing
|
||||
- ArgoCD Application creation
|
||||
- Deploy to cluster
|
||||
- Service health validation
|
||||
- API functionality testing
|
||||
|
||||
### Day 6 (June 11) - JARVIS Integration
|
||||
- Configure JARVIS environment variables
|
||||
- Test web search functionality
|
||||
- End-to-end validation
|
||||
- Documentation delivery
|
||||
|
||||
**Target Completion:** June 12, 2026
|
||||
|
||||
---
|
||||
|
||||
## Access & URLs
|
||||
|
||||
**Primary Access:**
|
||||
- API: https://spaceship-earth.local.mk-labs.cloud
|
||||
- Alias: https://firecrawl.local.mk-labs.cloud
|
||||
- Queue UI: https://spaceship-earth.local.mk-labs.cloud/admin/[BULL_AUTH_KEY]/queues
|
||||
|
||||
**Health Endpoints:**
|
||||
- Liveness: https://spaceship-earth.local.mk-labs.cloud/v0/health/liveness
|
||||
- Readiness: https://spaceship-earth.local.mk-labs.cloud/v0/health/readiness
|
||||
|
||||
**Internal Services (cluster-only):**
|
||||
- Playwright: http://playwright-service.firecrawl.svc:3000
|
||||
- PostgreSQL: postgresql://nuq-postgres.firecrawl.svc:5432
|
||||
- Redis: redis://redis.firecrawl.svc:6379
|
||||
- RabbitMQ: amqp://rabbitmq.firecrawl.svc:5672
|
||||
- RabbitMQ Mgmt: http://rabbitmq.firecrawl.svc:15672
|
||||
|
||||
---
|
||||
|
||||
## API Usage Examples
|
||||
|
||||
### Scrape a URL
|
||||
```bash
|
||||
curl -X POST https://spaceship-earth.local.mk-labs.cloud/v1/scrape \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"url": "https://example.com"
|
||||
}'
|
||||
```
|
||||
|
||||
### Search the Web
|
||||
```bash
|
||||
curl -X POST https://spaceship-earth.local.mk-labs.cloud/v1/search \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"query": "kubernetes best practices"
|
||||
}'
|
||||
```
|
||||
|
||||
### Crawl a Website
|
||||
```bash
|
||||
curl -X POST https://spaceship-earth.local.mk-labs.cloud/v1/crawl \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"url": "https://docs.example.com"
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
cluster/applications/firecrawl/
|
||||
├── README.md # This file
|
||||
├── ENVIRONMENT_VARIABLES.md # Complete env var reference
|
||||
├── namespace.yaml # Namespace definition
|
||||
├── configmap.yaml # Non-sensitive configuration
|
||||
├── externalsecret.yaml # 1Password secret sync
|
||||
├── postgresql/
|
||||
│ ├── statefulset.yaml # PostgreSQL StatefulSet
|
||||
│ ├── service.yaml # PostgreSQL Service
|
||||
│ └── pvc.yaml # Persistent Volume Claim
|
||||
├── redis/
|
||||
│ ├── deployment.yaml # Redis Deployment
|
||||
│ └── service.yaml # Redis Service
|
||||
├── rabbitmq/
|
||||
│ ├── deployment.yaml # RabbitMQ Deployment
|
||||
│ └── service.yaml # RabbitMQ Service
|
||||
├── playwright/
|
||||
│ ├── deployment.yaml # Playwright Deployment
|
||||
│ └── service.yaml # Playwright Service
|
||||
├── api/
|
||||
│ ├── deployment.yaml # API Deployment
|
||||
│ └── service.yaml # API Service
|
||||
├── workers/
|
||||
│ ├── worker-deployment.yaml # Queue Worker Deployment
|
||||
│ └── nuq-worker-deployment.yaml # NUQ Worker Deployment
|
||||
├── ingress/
|
||||
│ └── httproute.yaml # Gateway API HTTPRoute
|
||||
└── argocd/
|
||||
└── application.yaml # ArgoCD Application manifest
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tekton Pipelines
|
||||
|
||||
Build pipelines located in `cluster/tekton/pipelines/`:
|
||||
- `firecrawl-api-build.yaml` - API service build
|
||||
- `firecrawl-playwright-build.yaml` - Playwright service build
|
||||
- `firecrawl-postgres-build.yaml` - PostgreSQL build
|
||||
|
||||
**Trigger Builds:**
|
||||
```bash
|
||||
# API build
|
||||
kubectl create -f cluster/tekton/pipelines/firecrawl-api-build.yaml
|
||||
|
||||
# Playwright build
|
||||
kubectl create -f cluster/tekton/pipelines/firecrawl-playwright-build.yaml
|
||||
|
||||
# PostgreSQL build
|
||||
kubectl create -f cluster/tekton/pipelines/firecrawl-postgres-build.yaml
|
||||
```
|
||||
|
||||
**Monitor Builds:**
|
||||
```bash
|
||||
# List pipeline runs
|
||||
tkn pipelinerun list -n innoventions
|
||||
|
||||
# Watch logs
|
||||
tkn pipelinerun logs -f <pipelinerun-name> -n innoventions
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### API Pod Not Starting
|
||||
**Check:**
|
||||
1. RabbitMQ health status (API depends on healthy RabbitMQ)
|
||||
2. Environment variables (ConfigMap and Secret)
|
||||
3. Database connectivity (PostgreSQL)
|
||||
|
||||
**Commands:**
|
||||
```bash
|
||||
kubectl logs -n firecrawl deployment/api
|
||||
kubectl describe pod -n firecrawl -l app=api
|
||||
kubectl get externalsecret -n firecrawl
|
||||
```
|
||||
|
||||
### Build Failures
|
||||
**Check:**
|
||||
1. Harbor connectivity
|
||||
2. Harbor credentials secret
|
||||
3. Build resource limits (increase if OOM)
|
||||
|
||||
**Commands:**
|
||||
```bash
|
||||
tkn pipelinerun describe <name> -n innoventions
|
||||
kubectl logs -n innoventions <kaniko-pod>
|
||||
```
|
||||
|
||||
### Database Connection Errors
|
||||
**Check:**
|
||||
1. PostgreSQL pod status
|
||||
2. PVC binding
|
||||
3. Init script execution
|
||||
|
||||
**Commands:**
|
||||
```bash
|
||||
kubectl logs -n firecrawl statefulset/nuq-postgres
|
||||
kubectl exec -it -n firecrawl nuq-postgres-0 -- psql -U postgres -d postgres -c '\d nuq.queue_scrape'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring
|
||||
|
||||
**Resource Usage:**
|
||||
```bash
|
||||
kubectl top pods -n firecrawl
|
||||
```
|
||||
|
||||
**Service Health:**
|
||||
```bash
|
||||
# API liveness
|
||||
curl -k https://spaceship-earth.local.mk-labs.cloud/v0/health/liveness
|
||||
|
||||
# API readiness
|
||||
curl -k https://spaceship-earth.local.mk-labs.cloud/v0/health/readiness
|
||||
|
||||
# Playwright health
|
||||
kubectl exec -n firecrawl deployment/playwright-service -- curl localhost:3000/health
|
||||
```
|
||||
|
||||
**Queue Status:**
|
||||
Navigate to: https://spaceship-earth.local.mk-labs.cloud/admin/[BULL_AUTH_KEY]/queues
|
||||
|
||||
---
|
||||
|
||||
## Security Notes
|
||||
|
||||
1. **PostgreSQL Credentials:** Stored in 1Password, synced via ExternalSecret
|
||||
2. **Admin UI:** Protected by BULL_AUTH_KEY (in URL path)
|
||||
3. **Database Port:** NOT exposed outside cluster (ClusterIP only)
|
||||
4. **TLS:** All external traffic encrypted via cert-manager certificates
|
||||
5. **RBAC:** Service accounts scoped to firecrawl namespace
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- **Upstream Docs:** https://docs.firecrawl.dev
|
||||
- **GitHub:** https://github.com/mendableai/firecrawl
|
||||
- **Self-Hosting Guide:** https://github.com/mendableai/firecrawl/blob/main/SELF_HOST.md
|
||||
- **K8s Examples:** `/tmp/firecrawl/examples/kubernetes/`
|
||||
- **Mission Brief:** `~/friday/inbox/agents/rocket/FIRECRAWL-DEPLOYMENT-MISSION-BRIEF.md`
|
||||
|
||||
---
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-06-06 - Day 1 Complete
|
||||
- Initial repository structure created
|
||||
- Architecture decisions finalized
|
||||
- Build strategy documented
|
||||
- Environment variables researched
|
||||
- Ready for Day 2 (pipeline creation)
|
||||
|
||||
---
|
||||
|
||||
**Contact:** Rocket Raccoon (CI/CD Specialist)
|
||||
**Project Manager:** Pepper Potts
|
||||
**Cluster:** fastpass (Talos Kubernetes)
|
||||
**Last Updated:** June 6, 2026
|
||||
103
cluster/platform/democratic-csi/values.yaml
Normal file
103
cluster/platform/democratic-csi/values.yaml
Normal file
@@ -0,0 +1,103 @@
|
||||
# ------------------------------------------------------------------------------
|
||||
# democratic-csi Helm Values for Pure FlashArray (utilidor)
|
||||
#
|
||||
# Backend: Pure Storage FlashArray via freenas-iscsi driver
|
||||
# Target: utilidor.local.mk-labs.cloud (10.1.71.5)
|
||||
#
|
||||
# Note: The freenas-iscsi driver works with Pure API v2+
|
||||
# Pure FlashArray exposes an API compatible with FreeNAS/TrueNAS
|
||||
# ------------------------------------------------------------------------------
|
||||
|
||||
csiDriver:
|
||||
name: "org.democratic-csi.iscsi-pure"
|
||||
|
||||
storageClasses:
|
||||
- name: pure-iscsi-block
|
||||
defaultClass: false
|
||||
reclaimPolicy: Delete
|
||||
volumeBindingMode: Immediate
|
||||
allowVolumeExpansion: true
|
||||
parameters:
|
||||
fsType: ext4
|
||||
mountOptions: []
|
||||
secrets:
|
||||
provisioner-secret:
|
||||
controller-publish-secret:
|
||||
node-stage-secret:
|
||||
node-publish-secret:
|
||||
controller-expand-secret:
|
||||
|
||||
driver:
|
||||
config:
|
||||
driver: freenas-iscsi
|
||||
instance_id: mk-labs-utilidor
|
||||
|
||||
httpConnection:
|
||||
protocol: https
|
||||
host: utilidor.local.mk-labs.cloud
|
||||
port: 443
|
||||
# Pure API token from pure-exporter secret
|
||||
apiKey: "6157dea0-4c0d-b14b-6e6a-453aa649355d"
|
||||
allowInsecure: true
|
||||
apiVersion: 2
|
||||
|
||||
zfs:
|
||||
# Pure volumes are created in a pod/volume group context
|
||||
# This maps to Pure's volume naming scheme
|
||||
datasetParentName: csi-volumes
|
||||
detachedSnapshotsDatasetParentName: csi-snapshots
|
||||
datasetEnableQuotas: false
|
||||
datasetEnableReservation: false
|
||||
datasetPermissionsMode: "0777"
|
||||
datasetPermissionsUser: 0
|
||||
datasetPermissionsGroup: 0
|
||||
|
||||
iscsi:
|
||||
targetPortal: "10.1.71.5:3260"
|
||||
targetPortals: []
|
||||
# Pure FlashArray iSCSI configuration
|
||||
interface: default
|
||||
namePrefix: "csi-"
|
||||
nameSuffix: ""
|
||||
targetGroups:
|
||||
# Pure uses host groups for iSCSI targets
|
||||
# Leave empty to auto-select
|
||||
extentCommentTemplate: "Kubernetes CSI volume {{ parameters.[csi.storage.k8s.io/pvc/namespace] }}/{{ parameters.[csi.storage.k8s.io/pvc/name] }}"
|
||||
extentInsecureTpc: true
|
||||
extentXenCompat: false
|
||||
extentDisablePhysicalBlocksize: true
|
||||
extentBlocksize: 512
|
||||
extentRpm: "SSD"
|
||||
extentAvailThreshold: 0
|
||||
|
||||
controller:
|
||||
enabled: true
|
||||
replicaCount: 3
|
||||
strategy: deployment
|
||||
priorityClassName: system-cluster-critical
|
||||
|
||||
externalAttacher:
|
||||
enabled: true
|
||||
|
||||
externalProvisioner:
|
||||
enabled: true
|
||||
|
||||
externalResizer:
|
||||
enabled: true
|
||||
|
||||
externalSnapshotter:
|
||||
enabled: false
|
||||
|
||||
node:
|
||||
enabled: true
|
||||
rbac:
|
||||
enabled: true
|
||||
|
||||
driver:
|
||||
logLevel: info
|
||||
|
||||
# Talos requires hostNetwork for iSCSI operations
|
||||
hostNetwork: true
|
||||
hostIPC: true
|
||||
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
1
graphify-out/cache/ast/0a35dd44e77895b4f27d566cf738bf9bf39a40f8ab120276e6d929f228eb1a98.json
vendored
Normal file
1
graphify-out/cache/ast/0a35dd44e77895b4f27d566cf738bf9bf39a40f8ab120276e6d929f228eb1a98.json
vendored
Normal file
File diff suppressed because one or more lines are too long
1
graphify-out/cache/ast/5b5bd53d1192f2c4b80b3d2c6cccb5a5d155de2ad8d781e4cebd2ba03f2db894.json
vendored
Normal file
1
graphify-out/cache/ast/5b5bd53d1192f2c4b80b3d2c6cccb5a5d155de2ad8d781e4cebd2ba03f2db894.json
vendored
Normal file
@@ -0,0 +1 @@
|
||||
{"nodes": [{"id": "home_hermes_git_homelab_packer_fedora_42_scripts_cleanup_sh", "label": "cleanup.sh", "file_type": "code", "source_file": "packer/fedora-42/scripts/cleanup.sh", "source_location": "L1", "metadata": {"language": "bash", "kind": "file"}}, {"id": "home_hermes_git_homelab_packer_fedora_42_scripts_cleanup_sh__entry", "label": "cleanup.sh script", "file_type": "code", "source_file": "packer/fedora-42/scripts/cleanup.sh", "source_location": "L1", "metadata": {"language": "bash", "kind": "bash_entrypoint"}}], "edges": [{"source": "home_hermes_git_homelab_packer_fedora_42_scripts_cleanup_sh", "target": "home_hermes_git_homelab_packer_fedora_42_scripts_cleanup_sh__entry", "relation": "contains", "confidence": "EXTRACTED", "source_file": "packer/fedora-42/scripts/cleanup.sh", "source_location": "L1", "weight": 1.0}]}
|
||||
1
graphify-out/cache/ast/6433dbfa92d93a9746bedd5f34885ef84d2a8024c15ec237307d0b6baad1b6fd.json
vendored
Normal file
1
graphify-out/cache/ast/6433dbfa92d93a9746bedd5f34885ef84d2a8024c15ec237307d0b6baad1b6fd.json
vendored
Normal file
File diff suppressed because one or more lines are too long
1
graphify-out/cache/ast/67c9dd935a4da005d3b92d45fa7158659e5c8ac9733f050bcdb66018d805a6ae.json
vendored
Normal file
1
graphify-out/cache/ast/67c9dd935a4da005d3b92d45fa7158659e5c8ac9733f050bcdb66018d805a6ae.json
vendored
Normal file
File diff suppressed because one or more lines are too long
1
graphify-out/cache/ast/8325e3b9a39fb1929c5df57faee4688c4c5daf42422dc4dfb56561865098f4e6.json
vendored
Normal file
1
graphify-out/cache/ast/8325e3b9a39fb1929c5df57faee4688c4c5daf42422dc4dfb56561865098f4e6.json
vendored
Normal file
File diff suppressed because one or more lines are too long
1
graphify-out/cache/ast/856a78fcc41f33abb8b8b3e006a22aef01547e332490e54dfcbe85f8543bae7f.json
vendored
Normal file
1
graphify-out/cache/ast/856a78fcc41f33abb8b8b3e006a22aef01547e332490e54dfcbe85f8543bae7f.json
vendored
Normal file
@@ -0,0 +1 @@
|
||||
{"nodes": [{"id": "home_hermes_git_homelab_ansible_scripts_migrate_dns_references_sh", "label": "migrate-dns-references.sh", "file_type": "code", "source_file": "ansible/scripts/migrate-dns-references.sh", "source_location": "L1", "metadata": {"language": "bash", "kind": "file"}}, {"id": "home_hermes_git_homelab_ansible_scripts_migrate_dns_references_sh__entry", "label": "migrate-dns-references.sh script", "file_type": "code", "source_file": "ansible/scripts/migrate-dns-references.sh", "source_location": "L1", "metadata": {"language": "bash", "kind": "bash_entrypoint"}}, {"id": "scripts_migrate_dns_references_print_status", "label": "print_status()", "file_type": "code", "source_file": "ansible/scripts/migrate-dns-references.sh", "source_location": "L15", "metadata": {"language": "bash", "kind": "bash_function"}}], "edges": [{"source": "home_hermes_git_homelab_ansible_scripts_migrate_dns_references_sh", "target": "home_hermes_git_homelab_ansible_scripts_migrate_dns_references_sh__entry", "relation": "contains", "confidence": "EXTRACTED", "source_file": "ansible/scripts/migrate-dns-references.sh", "source_location": "L1", "weight": 1.0}, {"source": "home_hermes_git_homelab_ansible_scripts_migrate_dns_references_sh", "target": "scripts_migrate_dns_references_print_status", "relation": "defines", "confidence": "EXTRACTED", "source_file": "ansible/scripts/migrate-dns-references.sh", "source_location": "L15", "weight": 1.0}, {"source": "home_hermes_git_homelab_ansible_scripts_migrate_dns_references_sh__entry", "target": "scripts_migrate_dns_references_print_status", "relation": "calls", "confidence": "EXTRACTED", "source_file": "ansible/scripts/migrate-dns-references.sh", "source_location": "L34", "weight": 1.0, "context": "call"}]}
|
||||
1
graphify-out/cache/ast/c4db6a3b575d0b089164a117fe151ac8bd3c9abf1248e9a62401812ddd5621bc.json
vendored
Normal file
1
graphify-out/cache/ast/c4db6a3b575d0b089164a117fe151ac8bd3c9abf1248e9a62401812ddd5621bc.json
vendored
Normal file
@@ -0,0 +1 @@
|
||||
{"nodes": [{"id": "home_hermes_git_homelab_packer_ubuntu_24_04_scripts_cleanup_sh", "label": "cleanup.sh", "file_type": "code", "source_file": "packer/ubuntu-24.04/scripts/cleanup.sh", "source_location": "L1", "metadata": {"language": "bash", "kind": "file"}}, {"id": "home_hermes_git_homelab_packer_ubuntu_24_04_scripts_cleanup_sh__entry", "label": "cleanup.sh script", "file_type": "code", "source_file": "packer/ubuntu-24.04/scripts/cleanup.sh", "source_location": "L1", "metadata": {"language": "bash", "kind": "bash_entrypoint"}}], "edges": [{"source": "home_hermes_git_homelab_packer_ubuntu_24_04_scripts_cleanup_sh", "target": "home_hermes_git_homelab_packer_ubuntu_24_04_scripts_cleanup_sh__entry", "relation": "contains", "confidence": "EXTRACTED", "source_file": "packer/ubuntu-24.04/scripts/cleanup.sh", "source_location": "L1", "weight": 1.0}]}
|
||||
1
graphify-out/cache/ast/de122a158559c370b660f2dffa601fd3f55e275629bd76b7e542221b1260bd38.json
vendored
Normal file
1
graphify-out/cache/ast/de122a158559c370b660f2dffa601fd3f55e275629bd76b7e542221b1260bd38.json
vendored
Normal file
File diff suppressed because one or more lines are too long
1
graphify-out/cache/stat-index.json
vendored
Normal file
1
graphify-out/cache/stat-index.json
vendored
Normal file
File diff suppressed because one or more lines are too long
56
talos/talhelper/verify-multipath-config.sh
Executable file
56
talos/talhelper/verify-multipath-config.sh
Executable file
@@ -0,0 +1,56 @@
|
||||
#!/bin/bash
|
||||
# verify-multipath-config.sh
|
||||
# Verify that multipath.conf is properly configured on Talos worker nodes
|
||||
# Run from city-hall after applying the fixed configuration
|
||||
|
||||
set -e
|
||||
|
||||
NODE="${1:-10.1.71.69}" # Default to jungle-cruise
|
||||
echo "=== Verifying multipath configuration on node $NODE ==="
|
||||
echo
|
||||
|
||||
echo "1. Checking ext-multipathd service status..."
|
||||
talosctl services --nodes "$NODE" | grep -E "(STATE|multipathd)" || echo " ⚠️ Service not found"
|
||||
echo
|
||||
|
||||
echo "2. Checking if /etc/multipath.conf exists in ext-multipathd container..."
|
||||
if talosctl exec --nodes "$NODE" --service ext-multipathd -- cat /etc/multipath.conf &>/dev/null; then
|
||||
echo " ✅ File exists! Contents:"
|
||||
talosctl exec --nodes "$NODE" --service ext-multipathd -- cat /etc/multipath.conf | head -20
|
||||
echo " ... (truncated)"
|
||||
else
|
||||
echo " ❌ File does NOT exist"
|
||||
echo " This means the ExtensionServiceConfig is still not being applied."
|
||||
exit 1
|
||||
fi
|
||||
echo
|
||||
|
||||
echo "3. Checking multipath-tools extension..."
|
||||
talosctl get extensions --nodes "$NODE" | grep -E "(NAME|multipath)" || echo " ⚠️ Extension not found"
|
||||
echo
|
||||
|
||||
echo "4. Checking for multipath devices..."
|
||||
echo " Running: multipath -ll"
|
||||
if talosctl exec --nodes "$NODE" -- multipath -ll 2>/dev/null; then
|
||||
echo " ✅ Multipath devices detected"
|
||||
else
|
||||
echo " ⚠️ No multipath devices (may be normal if iSCSI not connected yet)"
|
||||
fi
|
||||
echo
|
||||
|
||||
echo "5. Checking ext-multipathd logs (last 20 lines)..."
|
||||
talosctl logs --nodes "$NODE" --service ext-multipathd --tail 20
|
||||
echo
|
||||
|
||||
echo "=== Verification complete ==="
|
||||
echo
|
||||
echo "Expected results:"
|
||||
echo " ✅ ext-multipathd service: Running"
|
||||
echo " ✅ /etc/multipath.conf: Exists with Pure FlashArray config"
|
||||
echo " ✅ multipath-tools extension: Loaded"
|
||||
echo " ✅ multipath -ll: Shows Pure devices (if iSCSI connected)"
|
||||
echo
|
||||
echo "If /etc/multipath.conf does NOT exist:"
|
||||
echo " - Verify talhelper genconfig was run"
|
||||
echo " - Check clusterconfig/fastpass-jungle-cruise.yaml for 'kind: ExtensionServiceConfig'"
|
||||
echo " - Ensure it's NOT inside quotes (should be structured YAML, not a string)"
|
||||
Reference in New Issue
Block a user