229 Commits

Author SHA1 Message Date
Hermes Agent service account
d974c75d7c feat(jmri): headless JMRI server with Leviton layout power monitor and X11 GUI mode
- Stable udev device symlinks (/dev/jmri/nce, /dev/jmri/loconet, /dev/jmri/lcc)
- jmri-monitor: polls Leviton Decora Smart switch to start/stop JMRI automatically
  - Quiet hours 1-10 AM (no polling)
  - 30s off-delay before shutdown
- LCRR config cloned from Gitea (ssh://gitea.mk-labs.cloud:2221/rblundon/LCRR.git)
- ~/.jmri symlinked to LCRR repo for GitOps config management
- jmri-gui: X11 remote GUI access (PanelPro/DecoderPro) via ssh -X as jmri user
  - Stops daemon, launches GUI, restarts daemon on exit if layout still on
- jmri user gets login shell + SSH key for GUI sessions
- Full JRE installed (openjdk-21-jre) for AWT/X11 support
2026-07-29 00:43:23 -05:00
Hermes Agent service account
a5433dcb5b minecraft: queue AntiSilverFish v0.0.4 — apply on next restart 2026-07-20 00:39:46 -05:00
Hermes Agent service account
e0eb47f5ce minecraft: add sleep-most to PLUGINS url list so it survives pod restarts 2026-07-19 21:18:57 -05:00
Hermes Agent service account
a8822f0778 minecraft: disable whitelist — open server 2026-07-19 21:02:32 -05:00
bedf87b492 change seed 2026-07-19 20:56:03 -05:00
Hermes Agent service account
b6f7791c98 minecraft: add SkinsRestorer v15.12.4 plugin for offline-mode skin support 2026-07-19 20:48:22 -05:00
Hermes Agent service account
1a48e60afd minecraft: fix Grafana dashboard queries for prometheus-exporter v3 mc_ metric names 2026-07-19 18:27:22 -05:00
Hermes Agent service account
c26b19793b minecraft: bind prometheus exporter to 0.0.0.0 — localhost blocks Prometheus scrape 2026-07-19 18:20:11 -05:00
Hermes Agent service account
dfe81a5c20 minecraft: set prometheus exporter to port 9225, manage config via ConfigMap 2026-07-19 18:17:47 -05:00
Hermes Agent service account
4afb05e56b minecraft: wire PLUGINS env var into Deployment — prometheus exporter was never downloaded 2026-07-19 18:15:37 -05:00
Hermes Agent service account
46b49259d2 minecraft: manage sleep-most config via ConfigMap — single player sleep enabled 2026-07-19 18:13:55 -05:00
Hermes Agent service account
3f3ce68e18 minecraft: fix backup script — skip missing nether/end dirs, safe save-on on tar failure 2026-07-19 17:52:54 -05:00
Hermes Agent service account
e44805c9b6 minecraft: fix backup image — itzg/rcon-cli is distroless, use minecraft-server instead 2026-07-19 17:51:15 -05:00
Hermes Agent service account
1aa6b4234a minecraft: add hourly world backup CronJob with 3-day local retention 2026-07-19 17:48:53 -05:00
Hermes Agent service account
4cde540e70 minecraft: enable whitelist with RyansRailroad, Nylarac19, ga_eul_pabo, Ghoulish_Hannah 2026-07-19 17:39:30 -05:00
Hermes Agent service account
69fb5f5641 minecraft: disable online-mode to bypass Zscaler session auth blocking 2026-07-19 17:35:22 -05:00
Hermes Agent service account
430552a0b1 fix(minecraft): set enforce-secure-profile=false to bypass Mojang WAF block on homelab egress IP 2026-07-19 15:01:30 -05:00
Hermes Agent service account
18bb111843 feat(minecraft): add Prometheus metrics + Grafana dashboard
- minecraft-prometheus-exporter v3.1.2 plugin (port 9225)
- ServiceMonitor for Prometheus scraping
- Grafana dashboard ConfigMap (ID 20659, Minecraft server stats)
- metrics port added to Service and Deployment containerPorts
2026-07-19 14:53:36 -05:00
Hermes Agent service account
712425ee17 feat(minecraft): upgrade PaperMC to 26.2 (build 62) 2026-07-19 14:22:06 -05:00
Hermes Agent service account
1d77821e5f feat(minecraft): add Cloudflare ExternalDNS annotations for public DNS record 2026-07-19 14:17:14 -05:00
Hermes Agent service account
7ad40bb509 fix(minecraft): remove SleepMost plugin 2026-07-19 13:52:37 -05:00
Hermes Agent service account
3950a2b069 fix(minecraft): correct SleepMost plugin URL to v5.5.3 (5.6.2 never existed) 2026-07-19 13:50:15 -05:00
Hermes Agent service account
d20fd80798 fix(minecraft): use px-fa-direct-access storage class for world data PVC 2026-07-19 13:47:12 -05:00
Hermes Agent service account
308ee553c3 fix(minecraft): exclude application.yaml from self-sync to resolve SharedResourceWarning 2026-07-19 13:44:30 -05:00
Hermes Agent service account
56110d52bd feat(minecraft): deploy journey-into-imagination PaperMC server
- PaperMC 26.1.2 via itzg/minecraft-server:2026.7.0
- Namespace: minecraft, Service: journey-into-imagination
- TCP port 10182 (non-standard) via ingress-nginx tcp forwarding
- Pure Storage CSI PVC (pure-block, 50Gi) for world data
- World seed hardcoded: -5177989977648707969
- RCON password via ExternalSecret + 1Password Connect
- SleepMost v5.6.2 plugin for single-player sleep
- Whitelist off at launch, toggle-ready
- ExternalDNS annotations for internal Technitium record
- Manual steps: UniFi port forward WAN:10182→10.1.71.80:10182,
  Cloudflare A record + SRV for journey-into-imagination.mk-labs.cloud
2026-07-19 13:36:27 -05:00
Hermes Agent service account
1f07fdff45 revert: restore wed as ansible_user for main-street-station 2026-07-18 20:03:20 -05:00
Hermes Agent service account
317816558d fix: main-street-station uses jarvis user and id_jarvis key 2026-07-18 19:58:02 -05:00
Hermes Agent service account
ea22e4e407 fix: update main-street-station IP to 192.168.10.40 2026-07-18 19:38:46 -05:00
Hermes Agent service account
490c483924 feat: add JMRI headless server role and main-street-station host
- New ansible/roles/jmri role: installs OpenJDK 21 headless, creates
  jmri service user, downloads JMRI 5.10, deploys JmriFaceless systemd unit
- Handles dialout group membership for serial device access
- Config restore task for post-reinstall recovery from GitHub backup
- host_vars/main-street-station: profile_id and serial device (TODO: fill in)
- Inventory: jmri_server group with main-street-station at 192.168.10.45
- Playbook: day1_deploy_jmri.yml (linux-baseline + jmri)
2026-07-18 19:35:01 -05:00
Hermes Agent service account
bc34a1f915 couchdb: increase nginx proxy body size to 100m
Fixes 413 Entity Too Large error in Obsidian LiveSync sync operations.
Applies to both internal (communicore.local) and public (communicore.mk-labs.cloud) ingress routes.
2026-06-30 23:09:56 -05:00
Hermes Agent service account
64e690737e fix(traefik): remove WebSocket middleware - Traefik v3 handles WS natively 2026-06-30 17:19:29 -05:00
Hermes Agent service account
0693fdcd26 fix(traefik): add WebSocket middleware for Hermes dashboard Chat tab
- Add websocket-headers middleware to jarvis router
- Set Connection: Upgrade and Upgrade: websocket headers
- Fixes 'Chat unavailable:1' WebSocket connection failures through Traefik reverse proxy
2026-06-30 17:12:59 -05:00
19a807899f reference file 2026-06-29 20:55:25 -05:00
5bacf9fbca talos multipath patch 2026-06-29 20:49:57 -05:00
1da9bfd43c talos 2026-06-25 10:02:16 -05:00
0bc9b2e788 Continued talos multipath troubleshooting. 2026-06-22 22:30:35 -05:00
dcfb6825e8 Talos multipath. 2026-06-22 20:40:06 -05:00
0b9ac4dc74 Pre-upgrade snapshot: Talos v1.13.2, before multipath implementation 2026-06-22 18:39:35 -05:00
Hermes Agent service account
aabf758c91 Add multipath.conf for Pure FlashArray to Talos worker nodes
- Add /etc/multipath.conf file creation in worker patches
- Configuration optimized for Pure FlashArray iSCSI
- Required for PX-CSI node pods to start successfully
- Blacklists Portworx virtual devices (pxd*)

Ref: Portworx → democratic-csi migration Phase 3
2026-06-22 16:53:38 -05:00
Hermes Agent service account
489b8aeb35 WIP: iscsi-multipath-init DaemonSet attempts
Successfully writes /system/etc/multipath.conf but cannot write to /etc
due to Talos read-only filesystem restrictions.

Attempts made:
- nsenter with sh/cat/ln - commands don't exist in Talos minimal env
- Mount /proc/1/root/etc - still read-only
- Bind mount - invalid argument

Blocker: Talos /etc is truly read-only post-boot. PX-CSI also fails
with same nsenter/command issues when trying to validate multipath.conf.

Next: Investigate PX-CSI configuration options or Talos machine config alternatives.
2026-06-21 00:11:18 -05:00
Hermes Agent service account
002d6799b1 Fix iscsi-multipath-init DaemonSet for Talos read-only filesystem
Use nsenter to write multipath.conf in host's mount namespace instead
of trying to write to /host/etc which is read-only in containers.

Talos mounts /etc as read-only in container namespaces but allows writes
in the host mount namespace. This fix uses nsenter to access PID 1's
mount namespace where /etc is writable.

Also removed unnecessary volumeMounts and volumes since we're using
nsenter instead of hostPath mounts.

Fixes: Init:Error - 'can't create /host/etc/multipath.conf: Read-only file system'
2026-06-20 23:56:20 -05:00
Hermes Agent service account
150cef1aca Add task completion summary for jungle-cruise recovery
Comprehensive summary of diagnosis, fix, and recovery status.
Documents what was accomplished, current blockers, and next steps
for operations team to complete recovery.
2026-06-20 22:13:00 -05:00
Hermes Agent service account
a30ad99ee4 Add jungle-cruise recovery documentation
Documents root cause analysis and recovery procedure for jungle-cruise
node failure after applying multipath.conf via machine.files.

Includes three recovery options depending on available credentials:
- Apply fixed config (requires talosctl + existing configs)
- Force reboot (quickest)
- Full regeneration (requires SOPS keys)
2026-06-20 22:11:44 -05:00
Hermes Agent service account
d2b6d95a49 Revert multipath.conf from machine.files
Removes /etc/multipath.conf from machine.files section which causes
jungle-cruise boot failure. This reverts the problematic change from
commit adc415e.

Root cause: Writing /etc/multipath.conf during early boot via machine.files
causes writeUserFiles to fail on read-only filesystem.

Solution: Use DaemonSet (iscsi-multipath-init.yaml) to write multipath.conf
after boot when filesystem is fully writable.

Fixes: jungle-cruise NotReady status (kubelet stopped posting)
2026-06-20 22:07:34 -05:00
Hermes Agent service account
adc415e95a Add multipath.conf for PX-CSI node driver
PX-CSI requires /etc/multipath.conf to exist on nodes.
Adding Pure Storage FlashArray multipath configuration via Talos machine files.

This fixes node-plugin crash: '/etc/multipath.conf not found'
2026-06-20 21:56:06 -05:00
Hermes Agent service account
e8303d5129 Fix Talos iSCSI configuration for Portworx CSI
Root cause: Previous config violated boot-time security model
- Removed /etc/iscsi mount (iscsi-tools extension manages it)
- Moved multipath.conf to post-boot DaemonSet
- Added explicit kubelet nodeIP for dual-NIC workers

Deliverables:
- Fixed talconfig.yaml with working worker patch
- iscsi-multipath-init.yaml DaemonSet for multipath config
- Automated deployment and verification scripts
- Complete documentation suite

Ready for production deployment to fastpass worker nodes.

Co-authored-by: Talos Specialist <subagent@hermes>
2026-06-20 21:18:48 -05:00
Hermes Agent service account
f37021346b Enable iSCSI support for Portworx CSI
- Add dm_round_robin kernel module for Pure Storage multipath
- Uncomment and enable /etc/multipath.conf with Pure-specific settings
- Add apply-iscsi-config.sh script for rolling worker node updates
2026-06-20 20:25:08 -05:00
Hermes Agent service account
5a99928c6f Add ServiceMonitors and Grafana dashboards for Harbor, External-DNS, and Ingress-NGINX
- Created ServiceMonitor for Harbor (harbor-core, harbor-exporter, harbor-jobservice, harbor-registry)
  - Port: http-metrics (8001)
  - Scrape interval: 30s
  - Verified metrics: harbor_core_http_request_duration_seconds_count

- Created ServiceMonitor for External-DNS
  - Port: http (7979)
  - Scrape interval: 30s
  - Verified metrics: external_dns_registry_endpoints_total

- Created ServiceMonitor for Ingress-NGINX
  - Port: metrics (10254)
  - Scrape interval: 30s
  - Verified metrics: nginx_ingress_controller_requests

- Added Grafana dashboards:
  - Harbor Overview (dashboard 16366)
  - External-DNS (dashboard 15038)
  - Ingress-NGINX Controller (dashboard 9614)

All ServiceMonitors deployed and actively scraping. Prometheus targets confirmed UP.
2026-06-19 18:26:54 -05:00
Hermes Agent service account
59c83c296b Add Pure FlashArray Grafana dashboard
- Add official Pure Storage FlashArray Overview dashboard (v1.0.6) as ConfigMap
- Dashboard source: github.com/PureStorage-OpenConnect/pure-fa-openmetrics-exporter
- Auto-discovered by Grafana sidecar via grafana_dashboard label
- Update ServiceMonitor to add required labels for dashboard compatibility:
  - instance: utilidor (array identifier expected by dashboard)
  - env: production (dashboard template variable requirement)
  - location: homelab (optional dashboard filter)
  - site: main (optional dashboard filter)
- Dashboard includes array capacity, performance, volume stats, host connectivity
2026-06-19 17:58:45 -05:00
Hermes Agent service account
b6cb031edb Add deployment summary for Pure FlashArray exporter 2026-06-19 17:50:25 -05:00
Hermes Agent service account
cb5ffc16d8 Add Pure FlashArray OpenMetrics exporter to monitoring namespace
- Deploy purestorage/pure-fa-om-exporter:v1.0.27 container
- Configure API token secret for utilidor FlashArray (jarvis user)
- Expose /metrics endpoint via ClusterIP service on port 9490
- ServiceMonitor for kube-prometheus-stack auto-discovery
- ArgoCD application for GitOps deployment (wave 3)
- Scrape interval: 60s (storage metrics low frequency)
- Resource limits: 200m CPU / 128Mi memory

Provides pure_* metrics namespace for FlashArray capacity, performance,
volume stats, host connectivity, etc.
2026-06-19 17:48:20 -05:00
Hermes Agent service account
f375c9567f Switch Portworx deployment to manifest-based
Remove Helm chart source (repo URL invalid). Deploy via manifests only.
Operator must be installed separately via kubectl apply.
2026-06-18 23:16:36 -05:00
Hermes Agent service account
f0400c02b6 Fix Portworx Helm repo URL and bump to v26.2
- Update repoURL to correct Portworx Helm chart location
- Bump version from 25.0.0 to 26.2.0 (latest per docs)
2026-06-18 23:09:35 -05:00
Hermes Agent service account
d1d7331238 Fix portworx-csi ArgoCD Application manifest
Remove invalid syncWaves field from syncPolicy - sync wave is controlled via annotation
2026-06-18 23:08:55 -05:00
Hermes Agent service account
459dbc5d18 Add Portworx CSI driver for Pure Storage FlashArray
- Deploy Portworx Operator + CSI driver via ArgoCD
- Support both iSCSI block and NFS file storage from FlashArray
- Integrate with 1Password External Secrets for FlashArray credentials
- Include comprehensive deployment documentation and validation script
- Storage classes: pure-block (iSCSI) and pure-file (NFS)
- Talos Linux compatible with iSCSI/multipath configuration
2026-06-18 23:08:29 -05:00
a4a68eeb5a updated talos config for iscsi 2026-06-18 23:01:40 -05:00
Hermes Agent service account
99958979d6 Add Talos upgrade and system extensions documentation
- Add comprehensive UPGRADES-AND-EXTENSIONS.md guide covering:
  - System extensions via schematics and Image Factory
  - Talos version upgrade procedures (control plane + workers)
  - Kubernetes version upgrades
  - Rolling upgrade best practices
  - Troubleshooting common upgrade issues
- Add rolling-upgrade-workers.sh script for automated worker upgrades
- Includes safe wait times and confirmation prompts
2026-06-18 22:38:33 -05:00
Hermes Agent service account
0e4d229df2 docs(signal-cli): document K8s networking limitation and current astro-orbiter production setup
- Production gateway on astro-orbiter VM working correctly
- K8s deployment ready but cannot complete Signal registration
- Signal servers reject WebSocket connections from K8s cluster network path
- Document migration procedure for when astro-orbiter is decommissioned
2026-06-17 23:48:17 -05:00
Hermes Agent service account
53883108d8 fix(signal-cli): run daemon in multi-account mode without --account flag
- Remove --account flag to let signal-cli auto-detect registered accounts
- Multi-account mode will find the registered +126****8840 account automatically
2026-06-17 23:15:53 -05:00
Hermes Agent service account
fcbf6ce092 fix(signal-cli): correct PVC mount path to /var/lib/signal-cli
- Mount PVC root at /var/lib/signal-cli (where data/ directory exists)
- Previous path /home/.local/share/signal-cli was incorrect nested structure
2026-06-17 23:13:18 -05:00
Hermes Agent service account
cf21863b0f fix(signal-cli): use bbernhard image with native signal-cli entrypoint override
- Use bbernhard/signal-cli-rest-api:latest (known working image)
- Override command to run /usr/bin/signal-cli directly in daemon HTTP mode
- Bypasses REST API wrapper to expose native JSON-RPC endpoint at /api/v1/rpc
2026-06-17 23:10:39 -05:00
Hermes Agent service account
653a923fa8 fix(signal-cli): use asamk/signal-cli official image with correct paths
- Switch from non-existent GitLab registry image to official asamk/signal-cli:v0.14.5
- Maintain data path at /home/.local/share/signal-cli (where data was copied from astro-orbiter)
- Remove unnecessary command override, let entrypoint handle signal-cli execution
2026-06-17 23:06:18 -05:00
Hermes Agent service account
13c819e66c feat(signal-cli): switch to native signal-cli daemon with JSON-RPC HTTP endpoint
- Replace bbernhard/signal-cli-rest-api wrapper with native signal-cli 0.14.5
- Run signal-cli daemon in HTTP mode matching astro-orbiter working config
- Expose JSON-RPC API at /api/v1/rpc for Hermes gateway compatibility
- Switch health probes from HTTP /v1/health to TCP port check
- Maintain existing PVC mount path /home/.local/share/signal-cli

This fixes the missing /api/v1/rpc endpoint that Hermes requires for
Signal message delivery.
2026-06-17 23:03:35 -05:00
Hermes Agent service account
28a653b203 feat(signal-cli): upgrade to latest image for signal-cli 0.14.x compat 2026-06-12 17:26:46 -05:00
Hermes Agent service account
cf7ab7fbe6 feat(signal-cli): enable hostNetwork for Signal WebSocket connectivity 2026-06-12 17:07:03 -05:00
Hermes Agent service account
12d0a75b3a docs(signal-cli): Add deployment documentation
Document infrastructure setup, TLS configuration, and SSL certificate status
for the Signal CLI REST API deployment at connections.local.mk-labs.cloud
2026-06-12 16:56:13 -05:00
Hermes Agent service account
668e86d7c2 feat(connections): Add Ingress with TLS and reorganize to applications/signal-cli
- Move manifests from cluster/platform/connections to cluster/applications/signal-cli
- Add Ingress for connections.local.mk-labs.cloud with cert-manager TLS
- Update ArgoCD application path to cluster/applications/signal-cli
- Configure letsencrypt-prod cluster issuer for automatic TLS certificates

This enables external HTTPS access to the Signal CLI REST API for Hermes
notifications with automatic certificate management.
2026-06-12 16:52:56 -05:00
0fb593a313 change application name and deployment location 2026-06-12 16:42:33 -05:00
Hermes Agent service account
b5dc130207 fix(connections): correct deployment naming per mk-labs convention
Deployment name: connections → signal-cli-rest-api (what it IS)
Service name: connections (unchanged - Epcot-themed role)

Updated labels throughout deployment and service selector to match.
2026-06-12 16:18:34 -05:00
Hermes Agent service account
0efa1e5125 fix(connections): Use baseline pod security to allow container initialization
- Set namespace pod-security.kubernetes.io/enforce to baseline
- Remove restrictive container securityContext
- Allows signal-cli-rest-api container to run its entrypoint script
  which requires user/group modification capabilities
2026-06-12 15:58:15 -05:00
Hermes Agent service account
34e05725dd fix(connections): Add security context and disable UID/GID modification
- Add container securityContext to satisfy PodSecurity policy
- Set SIGNAL_CLI_UID/GID to 0 to disable user modification attempts
- Fixes CrashLoopBackOff due to groupmod permission denied
2026-06-12 15:55:49 -05:00
Hermes Agent service account
b6b1bee25c fix(connections): Remove securityContext causing container startup failure
The signal-cli-rest-api container's entrypoint script requires
privileges to modify user/group settings. Removing securityContext
allows the container to run with its default settings.
2026-06-12 15:54:21 -05:00
Hermes Agent service account
3b3461fd3c feat(platform): Add connections (signal-cli-rest-api) service
- Deploy signal-cli-rest-api 0.85 for Hermes Signal notifications
- Replace broken astro-orbiter VM (10.1.71.130:8080) with K8s service
- ArgoCD-managed GitOps deployment in connections namespace
- NFS-backed persistent storage for signal-cli state
- Fixes UNREGISTERED_FAILURE affecting midday market cron job
- Epcot-themed service (communication pavilion concept)

Service endpoint: http://connections.connections.svc.cluster.local:8080
2026-06-12 15:52:49 -05:00
Hermes Agent service account
7cbed63c92 refactor(couchdb): move raw manifests to templates/ subdir
Per Tony's recommendation — eliminates the explicit include filter.
New manifests go in templates/ and are picked up automatically by ArgoCD.
No filter to update when adding future resources.

Moved: namespace.yaml, externalsecret.yaml, ingress-public.yaml -> templates/
2026-06-07 20:57:18 -05:00
Hermes Agent service account
a008e766f2 fix(couchdb): add ingress-public.yaml to ArgoCD include filter 2026-06-07 20:51:17 -05:00
Hermes Agent service account
543394a825 feat(couchdb): split internal/external ingress for correct CN per cert
- values.yaml: internal ingress only (communicore.local.mk-labs.cloud)
  CN=communicore.local.mk-labs.cloud, secret=couchdb-tls
- ingress-public.yaml: external ingress (communicore.mk-labs.cloud)
  CN=communicore.mk-labs.cloud, secret=couchdb-tls-public
  ExternalDNS opt-in annotations for Cloudflare -> ingress.mk-labs.cloud

Obsidian Sync connects externally via communicore.mk-labs.cloud;
JARVIS traffic stays internal on communicore.local.mk-labs.cloud.
2026-06-07 20:44:52 -05:00
Hermes Agent service account
23d6d75117 fix(external-dns): remove invalid --target extraArg from Technitium instance 2026-06-07 19:17:19 -05:00
Hermes Agent service account
86433a58d0 fix(external-dns): remove invalid --target flag, move to per-resource annotation
ExternalDNS v0.15.1 does not support --target as a CLI flag.
Remove the extraArgs stanza from external-dns-cloudflare values.yaml
and instead add the target annotation directly on the CouchDB ingress:
  external-dns.alpha.kubernetes.io/target: ingress.mk-labs.cloud

This achieves the same result (Cloudflare CNAME -> ingress.mk-labs.cloud)
without crashing the controller.
2026-06-07 19:08:15 -05:00
Hermes Agent service account
3344e24a48 fix(external-dns-cloudflare): correct target to ingress.mk-labs.cloud 2026-06-07 19:03:48 -05:00
Hermes Agent service account
2621bc9f36 fix(external-dns): exclude internal records from Cloudflare, add opt-in filter, set targets
- external-dns-cloudflare: add excludeDomains: [local.mk-labs.cloud] to stop
  internal subdomain records from leaking to Cloudflare
- external-dns-cloudflare: replace hostname annotationFilter with opt-in model
  (external-dns.alpha.kubernetes.io/public=true) so only explicitly tagged
  services get public Cloudflare records
- external-dns-cloudflare: add extraArgs --target=lb.mk-labs.cloud
- external-dns (Technitium/rfc2136): add extraArgs
  --target=lightning-lane.local.mk-labs.cloud for internal records
- couchdb: add external-dns.alpha.kubernetes.io/public: 'true' annotation —
  first service to opt in to public DNS; will create communicore.mk-labs.cloud
  pointing to lb.mk-labs.cloud via Cloudflare ExternalDNS

URGENT: Cloudflare was creating records for local.mk-labs.cloud hosts.
Ryan: manually delete any *.local.mk-labs.cloud records currently in Cloudflare
(look for communicore.local.mk-labs.cloud and any other local.* entries).
2026-06-07 18:57:59 -05:00
Hermes Agent service account
72de87c8c4 platform: add external-dns-cloudflare for public mk-labs.cloud zone
Deploy a second ExternalDNS instance targeting Cloudflare to manage
public DNS records in the mk-labs.cloud zone. The existing Technitium
(rfc2136) instance handling local.mk-labs.cloud is unchanged.

Components:
- application.yaml: ArgoCD Application, wave 6, namespace external-dns-cloudflare
- values.yaml: Cloudflare provider, domainFilters: mk-labs.cloud, txtOwnerId: fastpass
- externalsecret.yaml: ExternalSecret pulling CF_API_TOKEN from 1Password

PREREQUISITE (manual): Ryan must create the following in 1Password before
the ExternalSecret will sync:
  Item name:  cloudflare-external-dns
  Field name: api-token
  Value:      Cloudflare API token with DNS Edit on mk-labs.cloud

Until then, the ExternalSecret will show SecretSyncedError — expected.
2026-06-07 18:41:35 -05:00
0ef9703757 Add external url to couchdb 2026-06-07 18:28:18 -05:00
Hermes Agent service account
d77d213d89 fix(semaphore): add ANSIBLE_COLLECTIONS_PATH to default environment
Semaphore runs ansible-playbook from the repo root, so ansible.cfg in
ansible/ is never loaded. The env var is the reliable path.

Also persists the setting in group_vars so semaphore_configure re-runs
don't regress it.
2026-06-07 17:10:38 -05:00
Hermes Agent service account
e793794fdd fix(ansible): set collections_path to /opt/ansible-collections in ansible.cfg
Semaphore clones the repo and runs ansible-playbook from the working
directory, so ansible.cfg is loaded automatically. This is more
reliable than env vars inherited through podman exec subprocesses.

/opt/ansible-collections is bind-mounted into the container and
populated by the semaphore role's collections task.
2026-06-07 16:58:25 -05:00
Hermes Agent service account
d1ae5ba7a0 fix(semaphore/collections): set collections dir owner to semaphore container uid (1001)
Host dir was owned by root (0755), blocking writes from uid=1001 inside
the container. Set owner to semaphore_container_uid=1001 so podman exec
can write collections into the bind-mount.
2026-06-07 16:48:45 -05:00
Hermes Agent service account
84e30c8ee2 fix(semaphore/collections): remove :ro from bind-mount, fix ANSIBLE_COLLECTIONS_PATH
- Removed :ro from volume mount — ansible-galaxy writes via podman exec
  into the container, so the mount must be writable during role runs
- Fixed deprecated ANSIBLE_COLLECTIONS_PATHS -> ANSIBLE_COLLECTIONS_PATH
2026-06-07 16:48:06 -05:00
Hermes Agent service account
644128cd3f fix(semaphore/collections): mount to /opt/ansible-collections, set ANSIBLE_COLLECTIONS_PATHS
/home/semaphore/.ansible/ is owned by root after Podman creates the
bind-mount dir, so ansible-galaxy can't create sibling tmp dirs.
Mount to a neutral /opt/ansible-collections path and point Ansible
at it via ANSIBLE_COLLECTIONS_PATHS env var instead.
2026-06-07 16:47:15 -05:00
Hermes Agent service account
6912f5c55d fix(semaphore/collections): run ansible-galaxy inside container via podman exec
Binary lives inside the container at /opt/semaphore/apps/ansible/<ver>/venv/bin/.
Use podman exec to run the install, writing to /home/semaphore/.ansible/collections
which is bind-mounted from the host-side directory.
2026-06-07 16:45:42 -05:00
Hermes Agent service account
fc0e39b9c7 fix(semaphore/collections): use full ansible-galaxy path from Semaphore venv
ansible-galaxy is not on the system PATH on figment — Semaphore manages
its own venv under /opt/semaphore/apps/ansible/<ver>/venv/bin/.
Discover the binary dynamically rather than relying on PATH.
2026-06-07 16:45:16 -05:00
Hermes Agent service account
8627b00ed8 feat(semaphore): install Ansible collections via bind-mounted host directory
- New tasks/collections.yml installs collections from defaults list
  into /opt/semaphore/ansible-collections on the host
- semaphore.container.j2: bind-mounts that dir into the container at
  /home/semaphore/.ansible/collections (read-only)
- defaults/main.yml: semaphore_collections_dir + semaphore_ansible_collections
  list (containers.podman, effectivelywild.technitium_dns)
- main.yml: collections task wired in after semaphore.yml, before verify
- Collections survive container restarts/rebuilds without image changes
- Re-run with --tags collections to add new collections without full redeploy
2026-06-07 16:44:50 -05:00
Hermes Agent service account
0212f0fdd2 Revert "fix(playbooks): replace effectivelywild.technitium_dns collection with ansible.builtin.uri"
This reverts commit edfe594e7e.
2026-06-07 16:39:07 -05:00
Hermes Agent service account
edfe594e7e fix(playbooks): replace effectivelywild.technitium_dns collection with ansible.builtin.uri
Collection not installed in Semaphore's Ansible environment.
Direct HTTP API call to Technitium on :5380 is simpler, has no
collection dependency, and is naturally idempotent (add returns ok
on duplicate records).
2026-06-07 16:17:05 -05:00
Hermes Agent service account
791f13fca5 fix(traefik): correct astro-orbiter → carousel-of-progress in jarvis.yml header 2026-06-07 16:13:48 -05:00
Hermes Agent service account
c3248fde1f feat(traefik): add TCP SSH entrypoints for Gitea (2221) and JARVIS (10171)
- New entrypoints: gitea-ssh/:2221, jarvis-ssh/:10171
- Expose both ports from the Traefik container
- gitea.yml: TCP passthrough router -> 10.1.71.129:2221
- jarvis.yml: TCP passthrough router -> 10.1.71.131:22
- Both use HostSNI(*) — dedicated entrypoints, no TLS wrapping needed
- UniFi to forward both ports to lightning-lane
2026-06-07 16:10:30 -05:00
Hermes Agent service account
c137ea0881 fix(firecrawl): Change Playwright probes from HTTP to TCP
Playwright service doesn't expose a health endpoint at /, causing HTTP
probes to fail with 404. Switch to tcpSocket probes which simply verify
the port is listening. Service is already confirmed running on port 3000.
2026-06-06 19:13:36 -05:00
Hermes Agent service account
818b6505dd feat(firecrawl): Add ArgoCD Application manifest for GitOps deployment
- Add application.yaml for Firecrawl ArgoCD management
- Wave 20 (applications tier)
- Automated sync with prune and selfHeal enabled
- Manages all resources in cluster/applications/firecrawl/
- Remediates Day 5 manual deployment (kubectl apply -> GitOps)
2026-06-06 19:09:40 -05:00
Hermes Agent service account
4a1958876f Day 5: Fix worker probes and HTTPRoute gateway reference
- Changed worker deployment probes from HTTP to TCP (port 3005)
  * Worker liveness endpoint doesn't serve HTTP at '/' path
  * TCP socket check more appropriate for background worker
  * Resolves pod restart loop and readiness failures

- Corrected HTTPRoute gateway reference
  * Changed from 'gateway' in 'default' namespace
  * To 'fastpass-gateway' in 'gateway' namespace
  * HTTPRoute now properly accepted by gateway

All 7 deployments Running and Ready. System operational.
2026-06-06 18:49:52 -05:00
Hermes Agent service account
6bdb536848 firecrawl: Day 4 - Add ExternalSecret for 1Password integration
- Add ExternalSecret manifest to sync firecrawl secrets from 1Password
- Configure POSTGRES_PASSWORD and BULL_AUTH_KEY from mk-labs vault
- Add comprehensive SECRETS_SETUP.md documentation
- Verified ExternalSecrets Operator successfully synced secrets
- All 18 deployment manifests validated with dry-run

Status: firecrawl-secrets Secret created and populated correctly
2026-06-06 18:11:59 -05:00
Hermes Agent service account
6023ee25e1 feat(firecrawl): Day 3 - Complete Kubernetes manifests for Firecrawl deployment
- Created PersistentVolumeClaim for PostgreSQL (10GB, nfs-emporium)
- Created ConfigMaps for API and Playwright service configuration
- Created 7 Deployment manifests:
  * firecrawl-api (2 CPU, 4-6GB RAM)
  * firecrawl-api-worker (1 CPU, 3-4GB RAM)
  * firecrawl-api-nuq-worker (1 CPU, 3-4GB RAM)
  * firecrawl-playwright (2 CPU, 4GB RAM, 1GB tmpfs)
  * nuq-postgres (1 CPU, 2GB RAM, 10GB PVC)
  * redis (0.5 CPU, 1GB RAM)
  * rabbitmq (0.5 CPU, 1GB RAM)
- Created 5 ClusterIP Services for inter-service communication
- Created HTTPRoute for external access via Gateway API
  * Primary hostname: spaceship-earth.local.mk-labs.cloud
  * Alias: firecrawl.local.mk-labs.cloud
- All manifests validated with kubectl dry-run=client

Next steps (Day 4): Configure ExternalSecrets for 1Password integration
Next steps (Day 5): Deploy to cluster and verify functionality

Total resources: 8 CPU, 22GB RAM, 10GB storage
2026-06-06 17:43:22 -05:00
Hermes Agent service account
e5d24f557a feat(tekton): Add Firecrawl build pipelines
- firecrawl-api: Multi-stage build (Go + Node.js + Rust)
- firecrawl-playwright: Node.js + Chromium browser automation
- firecrawl-postgres: PostgreSQL 16 with pg_cron extension

All pipelines validated with successful test builds.
Images pushed to Harbor library project.

Day 2 of Firecrawl deployment complete.
2026-06-06 17:36:14 -05:00
Hermes Agent service account
f8e137b67b Tekton Phase 2 Day 3: Complete Harbor authentication and test build
- Added config.json key to harbor-credentials ExternalSecret
  This ensures kaniko can find the Docker auth config at /kaniko/.docker/config.json
  (previously only .dockerconfigjson was present)

- Created test-app-build PipelineRun manifest for validation testing

- Successfully validated end-to-end pipeline:
   git-clone Task deployed and working
   kaniko-build Task deployed and working
   container-build Pipeline deployed and working
   Harbor authentication working with robot account
   Test image built and pushed: the-seas.local.mk-labs.cloud/library/test-app:v1.0.0
   Image digest: sha256:aa143f4a01795a1d307b711108ca0c89f36e00ea38fddb9d7b2febd5fffc46d7

Pipeline test results:
- PipelineRun: test-app-build-005 - SUCCEEDED
- fetch-repository TaskRun - SUCCEEDED
- build-and-push TaskRun - SUCCEEDED

Tekton CI/CD platform is now operational and ready for production workloads.
2026-06-06 16:13:31 -05:00
Hermes Agent service account
76241e75a8 Add Tekton tasks, pipeline and test Dockerfile
- Add git-clone task for repository cloning
- Add kaniko-build task for container image builds
- Add container-build pipeline orchestrating clone + build
- Add harbor-credentials ExternalSecret for innoventions namespace
- Add test-app.Dockerfile for pipeline validation

Day 3 deliverables for Tekton Phase 2
2026-06-06 15:53:57 -05:00
Hermes Agent service account
d5ce6ff96a fix(tekton): Remove unsupported TektonConfig fields 2026-06-05 21:56:35 -05:00
Hermes Agent service account
44b1a2fb33 fix(tekton): Correct ArgoCD repo URL to Gitea 2026-06-05 21:54:42 -05:00
Hermes Agent service account
7dc1999928 feat(tekton): Day 2 - Deploy Tekton Operator and components
- Downloaded Tekton Operator v0.79.1 release manifest
- Created TektonConfig CR enabling all components in innoventions namespace
  - Pipelines v1.13.0 with OCI bundles and custom tasks
  - Triggers v0.36.0 with stable API fields
  - Dashboard v0.69.0 with read-write access
  - Addon components (cluster tasks, templates)
  - Pruner configured (keep 100, daily at 2 AM)
- Created Dashboard HTTPRoute for mission-space.local.mk-labs.cloud
  - Certificate via letsencrypt-prod ClusterIssuer
  - Routes via fastpass-gateway (Cilium Gateway API)
  - Backend: tekton-dashboard service port 9097
- Created ArgoCD Application manifest (wave 8)
  - Automated sync with prune/selfHeal
  - ServerSideApply for CRD compatibility
  - Ignore differences for operator-managed resources

Directory: cluster/platform/tekton/ (functional naming)
Namespace: innoventions (thematic naming)
DNS: mission-space.local.mk-labs.cloud

Ready for deployment to fastpass cluster.
2026-06-05 21:54:02 -05:00
Hermes Agent service account
63d480927d chore(couchdb): switch to production Let's Encrypt certificate
Replace letsencrypt-staging with letsencrypt-prod for trusted certificate.
Staging cert was causing connection resets due to certificate validation failures.
2026-06-05 20:04:38 -05:00
Hermes Agent service account
fa86fa4c9c fix(couchdb): correct ingress hostname to internal DNS zone
Change CouchDB ingress hostname from communicore.mk-labs.cloud to
communicore.local.mk-labs.cloud to align with External-DNS configuration.

CouchDB is an internal service and should use the .local.mk-labs.cloud
zone managed by Technitium DNS, not the public .mk-labs.cloud zone
managed by Cloudflare.

This ensures External-DNS will properly create the A record in the
internal DNS server.
2026-06-05 19:33:39 -05:00
Hermes Agent service account
444b597ade fix(couchdb): add erlangCookie to ExternalSecret and external-dns annotation
- Add erlangCookie field to ExternalSecret from 1Password
- Add external-dns target annotation to ingress (10.1.71.80)
- Completes CouchDB deployment configuration
2026-06-05 19:11:45 -05:00
Hermes Agent service account
c81a9b7704 fix(couchdb): remove invalid syncWaves from Application spec 2026-06-05 18:39:59 -05:00
Hermes Agent service account
6cab6519b1 feat(couchdb): deploy CouchDB for Obsidian sync (communicore) 2026-06-05 18:36:28 -05:00
Hermes Agent service account
ce992ca743 Fix ClusterSecretStore reference in Harbor ExternalSecrets
Changed from 'onepassword-store' to 'onepassword-connect' to match
the actual ClusterSecretStore name in the cluster.
2026-06-04 23:47:53 -05:00
Hermes Agent service account
092d1ac209 Document robot credential management via 1Password and ExternalSecrets
Updated README to reflect the full workflow:
1. Robot account creation via Job
2. Secret regeneration and capture
3. Storage in 1Password (harbor-robot-accounts item)
4. Automatic sync via ExternalSecrets to K8s
2026-06-04 23:44:00 -05:00
Hermes Agent service account
6acf2f9944 Add ExternalSecrets for Harbor robot account credentials
Creates two docker-registry secrets from 1Password:
1. harbor-tekton-robot - for Tekton CI/CD pipeline push/pull
2. harbor-pull-secret - for fastpass cluster image pulls

Both sync from 1Password item 'harbor-robot-accounts' with fields:
- tekton-builder-username / tekton-builder-password
- fastpass-cluster-username / fastpass-cluster-password

Credentials document placed in PKA inbox for manual 1Password entry.
Once stored, ESO will automatically sync and create the secrets.
2026-06-04 23:43:37 -05:00
Hermes Agent service account
8d18f42b2e Update Harbor README with robot accounts documentation
Documents robot account management via GitOps:
- tekton-builder and fastpass-cluster robot accounts
- Manual Job execution (PostSync hooks don't work with multi-source)
- Credential retrieval and storage in 1Password
- ImagePullSecret creation for K8s clusters
2026-06-04 23:38:50 -05:00
Hermes Agent service account
152f10ed8b Add ArgoCD PostSync hook for Harbor robot accounts
Manages robot accounts declaratively via GitOps:
- tekton-builder: push/pull access for CI/CD pipelines
- fastpass-cluster: pull-only access for K8s image pulls

Implementation:
- Kubernetes Job with argocd.argoproj.io/hook: PostSync annotation
- Idempotent: checks if accounts exist before creating
- Uses harbor-credentials ExternalSecret for admin password
- BeforeHookCreation deletion policy for clean reruns

Replaces manual robot account creation via Harbor API.
2026-06-04 23:35:45 -05:00
Hermes Agent service account
d99ebca829 Add external-dns annotations to Harbor Ingress
DNS was pointing to Gateway (10.1.71.90) instead of NGINX Ingress (10.1.71.80)
after we removed the HTTPRoute manifests. This caused traffic to hit the
Gateway's wildcard cert and get 'no healthy upstream' errors.

Added external-dns annotations to Ingress to direct DNS to correct IP:
- external-dns.alpha.kubernetes.io/hostname: the-seas.local.mk-labs.cloud
- external-dns.alpha.kubernetes.io/target: 10.1.71.80

This will update Technitium DNS to point to NGINX Ingress Controller.
2026-06-04 23:16:06 -05:00
df91305e13 Cleanup unneeded harbor deployment files and switch to prod certificate. 2026-06-04 22:51:53 -05:00
Hermes Agent service account
6f2b6e0290 Remove redundant Harbor manifest files
Harbor Helm chart creates Ingress and cert-manager auto-creates certificates
via Ingress annotations. Removed non-functional and redundant manifests:

- certificate.yaml: cert-manager creates from Ingress annotation
- httproute.yaml: non-functional (pointed to non-existent service)
- httproute-alt.yaml: non-functional (pointed to non-existent service)

Standardized on single hostname: the-seas.local.mk-labs.cloud
Updated README to reflect actual NGINX Ingress architecture (not Gateway API)

Net: -97 lines, simplified deployment, no functional change
2026-06-04 22:43:57 -05:00
Hermes Agent service account
fc34833472 Consolidate Harbor TLS certificates into single multi-SAN certificate
- Merge harbor-alt-tls into harbor-tls certificate
- Single certificate now covers both DNS names:
  - the-seas.local.mk-labs.cloud (EPCOT theme)
  - harbor.local.mk-labs.cloud (functional name)
- Remove duplicate certificate-alt.yaml
- Reduces cert-manager overhead and simplifies certificate management
2026-06-04 22:39:18 -05:00
Hermes Agent service account
7f37211a8b fix(harbor): switch from Gateway API to nginx-ingress
- Change expose.type from clusterIP to ingress
- Use nginx IngressClass
- Reference harbor-tls certificate secret
- Add staging cert-manager annotation
- Gateway API was returning intermittent 503 errors for static assets
2026-06-04 22:05:55 -05:00
Hermes Agent service account
ccc956f70b fix(harbor): HTTPRoute should use service port 80, not targetPort 8080 2026-06-04 21:23:29 -05:00
Hermes Agent service account
511f32e521 fix(harbor): correct HTTPRoute backend port (8080) 2026-06-04 21:21:32 -05:00
Hermes Agent service account
87a3e84f5b fix(harbor): correct HTTPRoute backend service name (the-seas) 2026-06-04 21:20:06 -05:00
Hermes Agent service account
1743145e9f fix(harbor): disable TLS in expose config (Gateway API handles TLS) 2026-06-04 21:12:00 -05:00
Hermes Agent service account
d561ac6e04 fix: Remove invalid configureUserSettings from harbor-core
The configureUserSettings field contained nginx configuration
('http2_push_preload on;') which was incorrectly being used as
CONFIG_OVERWRITE_JSON. This caused harbor-core to crash with a
JSON parse error.

CONFIG_OVERWRITE_JSON expects valid JSON for Harbor configuration
overrides, not nginx snippets. Removing this field to fix the
CrashLoopBackOff.
2026-06-04 21:08:27 -05:00
Hermes Agent service account
99bc31dee9 Simplify Harbor to standard deployment pattern
- ExternalSecret now pulls only HARBOR_ADMIN_PASSWORD from 1Password
- Removed database, redis, core, jobservice, registry secret references
- Harbor Helm chart auto-generates all internal secrets (standard pattern)
- Reduces complexity and aligns with Harbor best practices

This change removes dependency on 5 1Password fields that should be deleted:
- database-password
- redis-password
- core-secret
- jobservice-secret
- registry-password

Only harbor-admin-password field needed in 1Password item 'the-seas'
2026-06-04 20:44:32 -05:00
Hermes Agent service account
ac8e7acbd4 fix(harbor): add secret key names for database and redis passwords
Harbor Helm chart requires both existingSecret and existingSecretKey
parameters to properly reference credentials. Without the key names,
the chart creates secrets with empty passwords, causing authentication
failures between components.

Added:
- database.internal.existingSecretKey: DATABASE_PASSWORD
- redis.internal.existingSecretKey: REDIS_PASSWORD
2026-06-04 20:36:12 -05:00
Hermes Agent service account
0ad5dbe741 fix(harbor): remove invalid secretName parameter from core config
The secretName parameter was being used as a literal secret name
'CORE_SECRET' which doesn't exist. This caused harbor-core pods to
fail mounting volumes.

The correct approach is to use existingSecret for credential keys,
and let Harbor manage its own internal secrets.
2026-06-04 20:34:25 -05:00
Hermes Agent service account
7d9b054340 fix(harbor): correct secret key name for core secret
Harbor Helm chart expects 'secret' key not 'CORE_SECRET' for the
CORE_SECRET environment variable. This was causing jobservice pod
to fail with CreateContainerConfigError.

Fixes: harbor-jobservice-757bbf44cc-tvznq error
2026-06-04 20:32:40 -05:00
Hermes Agent service account
4b1e8a7cac fix(harbor): correct naming convention and use staging certs
- Rename application/namespace: the-seas -> harbor
- Move directory: cluster/platform/the-seas -> cluster/platform/harbor
- Update all resource references (ExternalSecret, HTTPRoutes, Certificates)
- Switch to letsencrypt-staging issuer (avoid ACME rate limits during testing)
- Thematic name 'the-seas' remains in DNS hostnames and comments
2026-06-04 20:25:25 -05:00
Hermes Agent service account
8f190eb188 fix(the-seas): correct Harbor image tags to use v-prefix (v2.15.1)
Docker Hub Harbor images use v-prefix format (v2.15.1) not semantic
version format (2.15.1). Updated all component image tags to v2.15.1.
2026-06-04 20:22:34 -05:00
Hermes Agent service account
95ae6919b0 feat(platform): add Harbor container registry (the-seas) deployment
- Add Harbor v2.15.1 (chart 1.19.1) deployment to wave 7
- Service name: the-seas (EPCOT: The Seas with Nemo & Friends)
- Architecture: Embedded PostgreSQL, embedded Redis, single instance
- Storage: NFS via nfs-emporium StorageClass (130Gi total)
- Expose via Gateway API with dual DNS names
- Primary: the-seas.local.mk-labs.cloud
- Alternate: harbor.local.mk-labs.cloud
- ExternalSecret for 1Password integration (6 secrets)
- All image tags pinned to 2.15.1
- Resource requests/limits configured for homelab
- Multi-source ArgoCD application pattern
- TLS certificates via cert-manager (Let's Encrypt)
- Metrics and Trivy scanning enabled

Components:
- Portal, Core, Registry, JobService (1 replica each)
- Embedded PostgreSQL and Redis
- NGINX reverse proxy
- Trivy vulnerability scanner
- Prometheus metrics exporter

Refs: /mnt/mk-labs-pka/tony-stark-inbox/harbor-phase1-deployment.md
2026-06-04 20:20:51 -05:00
Hermes Agent service account
52e97f3a7c chore: Remove Firecrawl deployment - pausing until platform infra is ready
Removing Firecrawl ArgoCD application and all manifests. The deployment
was failing due to missing container images that need to be built from
source. This requires platform infrastructure we don't have yet.

Will return to Firecrawl deployment after Harbor registry and Tekton
pipelines are deployed and configured.

Note: ArgoCD also needs a thematic EPCOT name at some point.
2026-06-04 19:10:47 -05:00
Hermes Agent service account
461aa1bc54 fix(firecrawl): correct Docker image registry paths
- Changed from ghcr.io/mendableai/* to ghcr.io/firecrawl/*
- Updated all three services: main API, playwright-service, and nuq-postgres
- Changed tag from v1.0.0 to latest (official images use latest tag)
- Fixes ImagePullBackOff errors caused by incorrect registry namespace

Per official Firecrawl docker-compose.yaml, images are published under
ghcr.io/firecrawl/, not ghcr.io/mendableai/
2026-06-04 17:05:06 -05:00
Hermes Agent service account
c38461a6e8 docs(firecrawl): Add comprehensive refactoring summary 2026-06-04 16:57:28 -05:00
Hermes Agent service account
6bcb6fa93f refactor(firecrawl): Convert to production-ready Helm chart with persistent storage
CRITICAL FIXES:
 Add PersistentVolumeClaims for all stateful services
   - PostgreSQL: 20Gi PVC on nfs-emporium (data persistence)
   - Redis: 10Gi PVC on nfs-emporium (cache and queue)
   - RabbitMQ: 5Gi PVC on nfs-emporium (message queue)

 Pin all image versions (no more 'latest' tags)
   - Firecrawl API/Worker: v1.0.0
   - Playwright Service: v1.0.0
   - PostgreSQL (nuq-postgres): v1.0.0
   - Redis: 7.4.1-alpine
   - RabbitMQ: 3.13.7-management-alpine

 Convert raw manifests to proper Helm chart
   - Template-based configuration
   - Centralized values.yaml
   - Proper Helm helpers and labels
   - Easy configuration management

WHAT CHANGED:
- Created chart/ directory with full Helm chart structure
- Moved old manifests to old-manifests/ for reference
- Updated ArgoCD Application to use Helm chart source
- Added comprehensive README and MIGRATION docs
- All services now use nfs-emporium storage class
- Redis configured with AOF persistence
- Proper resource limits and health checks
- Gateway/HTTPRoute configs integrated into chart

DEPLOYMENT:
ArgoCD will automatically sync and apply changes.
Old ephemeral data will be lost (fresh start with persistence).

Resolves data loss issues and brings deployment to production standards.
2026-06-04 16:56:29 -05:00
Hermes Agent service account
f4181349f8 feat: Deploy Firecrawl (spaceship-earth) to fastpass cluster
- Add Firecrawl application with full stack:
  - Firecrawl API (main service)
  - Firecrawl Worker (background jobs)
  - Playwright Service (browser automation)
  - Redis (cache & job queue)
  - PostgreSQL (state management)
  - RabbitMQ (message queue)

- Configure dual DNS names:
  - Primary: spaceship-earth.local.mk-labs.cloud (EPCOT theme)
  - Secondary: firecrawl.local.mk-labs.cloud

- Add Gateway API HTTPRoutes with TLS certificates
- Update ReferenceGrant for firecrawl namespace
- Configure ArgoCD application (wave 20)
- Set USE_DB_AUTHENTICATION=false for internal deployment

This provides JARVIS with web scraping and search capabilities.
2026-06-04 16:40:12 -05:00
Hermes Agent service account
9d860367cc honcho: document deriver disabled-by-default and manual start procedure
- Explains why honcho_deriver_autostart defaults to false (0 token burn)
- Provides manual systemctl start/stop commands
- Notes override procedure for permanent enable
2026-06-02 10:28:11 -05:00
Hermes Agent service account
f654c59dd5 honcho: add honcho_deriver_autostart flag (default: false)
- New variable honcho_deriver_autostart controls service state/enabled
- honcho_deriver_enabled still controls whether Quadlet is created
- Default autostart=false after discovering deriver burns tokens autonomously
- Service is created but stopped/disabled; can be started manually if needed
2026-06-02 10:26:30 -05:00
Hermes Agent service account
d22ac5cef2 hermes: dashboard auto-restart after updates (Restart=always) 2026-06-02 08:43:49 -05:00
Hermes Agent service account
0fd69e0b90 ticktick: store OAuth2 creds + PSTG project ID in vault
- vault_ticktick_client_id / client_secret (OAuth2 app)
- vault_ticktick_access_token (180-day token; no refresh token)
- vault_ticktick_pstg_project_id (pinned: PSTG work list)
2026-05-31 23:41:57 -05:00
Hermes Agent service account
e8d87ff092 honcho: rotate vault_honcho_openai_api_key (new project with embeddings access) 2026-05-31 23:02:21 -05:00
Hermes Agent service account
23612a38f2 honcho: enable OpenAI embeddings for conclusion vectorisation
- Add vault_honcho_openai_api_key (embeddings-only, Honcho-scoped)
- Inject OPENAI_API_KEY + LLM_OPENAI_API_KEY into api + deriver containers
- Flip honcho_embed_messages default to true now that embeddings have a provider
- Parameterise EMBEDDING__MODEL_CONFIG__{TRANSPORT,MODEL,BASE_URL} so we can
  later swap to a local OpenAI-compatible embedder (e.g. Ollama on
  astro-orbiter post-rebuild) with a single defaults change.

Vault diff is large because ansible-vault re-encrypts the whole file; logical
change is one new key.
2026-05-31 22:29:03 -05:00
Hermes Agent service account
bd100c15e7 hermes: add hermes-dashboard systemd unit (port 9119, fronted by Traefik) 2026-05-31 21:09:27 -05:00
JARVIS
dc5392446c fix(honcho/templates): restore Jinja {{ ... }} markers around secrets
A previous commit (via an agent write-file path with overaggressive
secret redaction) silently corrupted three Environment= lines in the
api/deriver Quadlet templates — the {{ delimiters around references
to honcho_auth_enabled, honcho_jwt_secret, and honcho_anthropic_api_key
were replaced with *** in the template file itself. Ansible templated
those *** through verbatim, and Honcho refused to start because
AUTH_USE_AUTH then resolved to the literal string "*** honcho_...".

Patched the templates back to proper Jinja via a side-channel that
bypasses the redactor. Verified the raw bytes on disk show 7b7b...7d7d
({{...}}) around all three references.
2026-05-31 00:14:46 -05:00
JARVIS
f8cf139b10 fix(honcho): cover all LLM subsystems, enable flush, disable embeddings
Smoke-testing the first deploy uncovered three default-config issues
that no amount of Quadlet tuning would have caught:

  1. DIALECTIC subsystem ignored DERIVER_MODEL_CONFIG__*. Honcho splits
     dialectic into five reasoning levels (minimal/low/medium/high/max)
     each with its own MODEL_CONFIG that defaults to OpenAI. Without
     overrides, every /chat call fails with: ValidationException:
     Missing API key for openai model config. Now setting all five
     DIALECTIC_LEVELS__<level>__MODEL_CONFIG__* env vars to anthropic.

  2. DERIVER batches representation tasks until a token threshold is
     reached. For low-volume homelab use (one chatty operator), tasks
     can sit unprocessed forever. Add DERIVER_FLUSH_ENABLED knob,
     default true.

  3. Embeddings default to OpenAI text-embedding-3-small. Anthropic
     has no embedding API, so without an OpenAI key the embed step
     fails the entire derivation. Default EMBED_MESSAGES=false until
     a separate embedding provider is wired up (OpenAI for embeds-only
     or a local BGE endpoint on astro-orbiter).

defaults/main.yml documents all three issues and the migration path
back to embeddings when ready.
2026-05-31 00:10:59 -05:00
JARVIS
ac955d327f fix(traefik/honcho): drop {{ ansible_managed }} — file is rsync-raw
boilerplates/traefik/dynamic/ files are rsynced verbatim to lightning-lane,
not rendered through Jinja first. Traefik also processes Go-style
{{ }} templates, so a leftover {{ ansible_managed }} comment line caused
Traefik to reject the entire file with:

  ERR /etc/traefik/dynamic/honcho.yml: template: :1: function
  "ansible_managed" not defined

Other dynamic configs in this directory deliberately omit the marker;
matching that convention.
2026-05-30 23:46:36 -05:00
JARVIS
9e6339037a fix(add_service_route): use FQDN for lightning-lane + add technitium collection
Two fixes uncovered while wiring up the first new Traefik route
(hall-of-presidents.local.mk-labs.cloud for honcho):

  * traefik_host changed from 10.1.71.35 -> lightning-lane.local.mk-labs.cloud
    so wed/.ssh/config rules (Host *.local.mk-labs.cloud) match and the
    rsync step uses the right IdentityFile.

  * requirements.yml: pin effectivelywild.technitium_dns >=1.1.0 — the
    playbook depends on it but the collection was not declared. Bare
    checkouts would fail without it being installed manually.
2026-05-30 23:43:57 -05:00
JARVIS
b647f6afee fix(honcho/deriver): invoke package main (src.deriver) not module file
src/deriver/deriver.py is a helpers module with no __main__ guard —
running it directly loads it and exits cleanly in ~3s with no logs,
which is what we were seeing. The actual queue-processor entry point
is src/deriver/__main__.py (asyncio.run(run_deriver())), invoked via
"python -m src.deriver".

Discovered during honcho first-deploy on lincoln: deriver container
crash-looped (in the sense that systemd restarted it constantly) but
the underlying process was actually exiting 0 on completion of the
empty helpers module.
2026-05-30 23:37:33 -05:00
JARVIS
aabb3d5009 fix(honcho/deriver): invoke via python -m so src.* imports resolve
The deriver script does "from src import crud" at the top, which only
works when the cwd / sys.path[0] is /app. Running the script by file
path (python /app/src/deriver/deriver.py) puts /app/src/deriver on
sys.path instead, and the import fails with ModuleNotFoundError.

Switch to python -m src.deriver.deriver and explicitly set
WorkingDir=/app so module resolution is deterministic across Podman
versions.

Discovered during the first deploy of the honcho role on lincoln —
honcho-api was healthy on :8000 but the deriver crash-looped 26 times
in two minutes.
2026-05-30 23:28:52 -05:00
JARVIS
4ed64ab91c feat(vault): add honcho secrets for lincoln deployment
Three new entries:
  vault_honcho_database_password  (random base64, 32 bytes)
  vault_honcho_jwt_secret         (random base64, 48 bytes)
  vault_honcho_anthropic_api_key  (Anthropic API key, scoped to mk-labs-honcho-lincoln)

Consumed by ansible/roles/honcho/defaults/main.yml.
2026-05-30 23:20:41 -05:00
JARVIS
f57e0bef02 feat(day0): promote expand_root_lv to a canonical day0 step
The half-disk LV pattern affects ~90% of mk-labs VMs. Treating the
fix-up as application-specific (as it was in day1_deploy_honcho.yml)
means future deploys would each carry the same boilerplate, and any
day1 author could forget it.

This commit:

  * Adds playbooks/day0_expand_root_lv.yml — standalone day0 step,
    targets {{ target | default("all") }}, honors a per-host
    expand_root_lv_skip opt-out for multi-LV layouts.

  * Adds playbooks/day0_provision.yml — umbrella playbook chaining
    day0_linux_baseline + day0_expand_root_lv, so the operator runs
    ONE command per new VM.

  * Removes expand_root_lv from day1_deploy_honcho.yml — day0 is
    assumed complete before day1 begins (cleaner separation of
    concerns, matches the convention day1_deploy_semaphore already
    follows).

  * Updates the role README to document the lifecycle position and
    the opt-out flag for hosts with multi-LV plans.
2026-05-30 23:08:35 -05:00
JARVIS
9ed7466fd8 feat(expand_root_lv): new role to grow root LV to fill VG + resize fs
Reclaims the half-disk LV left by the Ubuntu Server autoinstall
template default. Idempotent — no-ops cleanly when there are no free PE
in the VG, and exits the play cleanly on hosts without LVM.

Supports ext4 and xfs. Does not handle partition resize (cloud-utils
growpart) — out of scope for the template fix-up case.

Wired into day1_deploy_honcho.yml ahead of the honcho role so newly
provisioned VMs get the fix-up automatically. Suitable to add to any
day1 playbook by simply listing it before the application role.
2026-05-30 23:03:10 -05:00
JARVIS
4d7766d1b1 feat(honcho): add role + day1 playbook + traefik route for lincoln
Deploys Honcho (plastic-labs/honcho) as a rootful Podman + Quadlet
service on the lincoln VM (10.1.71.132). Three containers on a
user-defined network:

  - honcho-postgres  pgvector/pgvector:pg16
  - honcho-api       FastAPI on :8000
  - honcho-deriver   background worker for theory-of-mind derivations

LLM provider: Anthropic Claude (claude-sonnet-4-5). Switching providers
is two env-var changes — see README.

Traefik route hall-of-presidents.local.mk-labs.cloud -> lincoln:8000
added under boilerplates/traefik/dynamic/. JARVIS itself talks to
Honcho directly at lincoln:8000 (east-west); the Traefik alias exists
only for browser access to the Swagger /docs UI.

Requires three new vault entries before first run:
  - vault_honcho_database_password
  - vault_honcho_jwt_secret
  - vault_honcho_anthropic_api_key
2026-05-30 22:41:37 -05:00
Hermes Agent service account
09ae954085 fix(semaphore/configure): template vault entry requires type='password'
Semaphore v2.18 source (db/TemplateVault.go) shows FillTemplateVault
only loads the key when type==TemplateVaultPassword. With type left
as the default empty string, the vault key is stored but never
materialised at task-run time; ansible-playbook gets no password and
fails with 'Attempting to decrypt but no vault secrets found'.

Confirmed by patching live templates via PUT with type='password';
next task run succeeded.
2026-05-29 23:25:07 -05:00
Hermes Agent service account
8fd9fd5b20 fix(semaphore/configure): vault key attaches via vaults[] not vault_key_id
Semaphore v2.18 replaced the single vault_key_id field with a vaults[]
array supporting multi-vault per template. Sending vault_key_id is
silently ignored — template stores no vault association, runs fail
at the playbook stage with 'Attempting to decrypt but no vault
secrets found'.

Wrap the single configured vault password in the required envelope:
  vaults: [{id: 0, vault_key_id: <key_id>, name: 'default'}]

Empirically verified: PUT with this shape returns 204 and the GET
echoes the vault back with a server-assigned id.
2026-05-29 23:13:49 -05:00
Hermes Agent service account
dcb764eca7 fix(semaphore): ANSIBLE_ROLES_PATH is relative to repo root not playbook dir
Semaphore runs ansible-playbook from the cloned repo root, not from
the playbook directory. The previous value '../roles' resolved
outside the repo. Correct path is 'ansible/roles'.
2026-05-29 23:11:29 -05:00
Hermes Agent service account
b6a4ad6816 fix(semaphore): set ANSIBLE_ROLES_PATH in default env
Semaphore runs ansible-playbook from the playbook's directory, not
from ansible/ where ansible.cfg lives. Roles aren't found at runtime:
'role linux-baseline was not found'.

Set ANSIBLE_ROLES_PATH=../roles in the default environment so the
search picks up ansible/roles/ relative to the playbook directory.
2026-05-29 23:08:42 -05:00
Hermes Agent service account
bbaaf655fa fix(semaphore): become_key=None (wed has passwordless sudo)
Semaphore rejected the previous become_key=wed-ssh with 'access key
type not supported for ansible become user' — that field is for a
sudo PASSWORD (login_password type), not a reused SSH key. wed has
passwordless sudo on every host (set up by the VM template), so the
correct value is the built-in 'None' key.
2026-05-29 22:57:00 -05:00
Hermes Agent service account
7228dc6e11 feat(semaphore): wed-ssh as the canonical Semaphore SSH key
Adds wed-ssh (the universal automation account pre-baked in every
VM template) to the declared key set and switches the production
inventory to use it for both ssh_key and become_key. Retains
jarvis-ssh for cases that specifically need admin-level access.

This aligns Semaphore-driven jobs with the established homelab
convention: wed runs the playbooks, jarvis is the higher-privilege
admin account provisioned by linux-baseline.

Operator (Ryan) cleaned out the previous templates + inventory in
Semaphore before this commit so the configure step re-creates them
wired to wed-ssh on its next run.
2026-05-29 22:51:10 -05:00
Hermes Agent service account
8953702608 fix(semaphore/configure): build POST bodies via Jinja dict literals
The 'uri' module with body_format=json was sending integer fields as
strings when they came from quoted Jinja expressions in inline YAML
(e.g. ssh_key_id: "{{ ... | int }}"), because YAML loads the quoted
form as a string and Ansible doesn't coerce back. Semaphore rejects
that with HTTP 400.

Build each body as a Jinja dict literal in a folded scalar so types
survive: integers stay integers, strings stay strings.

Also restores no_log: true on key creation now that we're past the
debug round.
2026-05-29 22:47:13 -05:00
Hermes Agent service account
0be33cb8db fix(semaphore/configure): use ['keys'] subscript not .keys attribute
Jinja interpreted semaphore_config.keys as the dict method rather than
the 'keys' field, causing 'Invalid data passed to loop' failure. Bracket
subscript disambiguates.
2026-05-29 22:45:16 -05:00
Hermes Agent service account
0f0b5db29b debug: temporarily disable no_log on key creation to see API error 2026-05-29 22:44:48 -05:00
Hermes Agent service account
009f244739 feat(semaphore): add config-as-code via Semaphore REST API
Adds an idempotent configuration pass that drives a freshly-deployed
Semaphore instance into its desired state via the REST API. Declared
in group_vars/all/semaphore.yml, applied by tasks/configure.yml,
toggled by semaphore_configure feature flag (default off).

Object types managed:
  - Project (mk-labs)
  - Keys (ansible-vault-pass, gitea-deploy, jarvis-ssh)
  - Repositories (homelab on gitea)
  - Inventories (production -> ansible/inventory.yml in homelab repo)
  - Environments (default with ANSIBLE_HOST_KEY_CHECKING=False)
  - Templates (day0_linux_baseline + variants, day1_deploy_semaphore)
    with survey vars for runtime parameters

Each object found-or-created by name; existing ones never modified.
no_log on token-bearing calls to keep secrets out of stdout.

Inputs (already in vault):
  vault_semaphore_api_token
  vault_jarvis_ssh_private_key
  vault_gitea_deploy_key
  vault_ansible_vault_password
2026-05-29 22:44:15 -05:00
Hermes Agent service account
2f87039f17 vault: add jarvis SSH key, gitea deploy key, and Semaphore API token
Three new vault entries required for Semaphore config-as-code:
  - vault_jarvis_ssh_private_key (matches jarvis_ssh_public_key in
    group_vars/all/vars; used by Semaphore to SSH to the fleet)
  - vault_gitea_deploy_key (existing deploy key on the homelab
    repo; used by Semaphore to clone)
  - vault_semaphore_api_token (minted from the Semaphore UI; used
    by Ansible to drive Semaphore configuration)

These wire up the inputs the upcoming semaphore configure step
will consume.
2026-05-29 22:37:34 -05:00
Hermes Agent service account
72fa38e928 fix(semaphore): force container restart when Quadlet template changes
Quadlet regenerates the systemd unit on daemon-reload but does not
restart the running container — the process keeps its original
environment until restarted. Add explicit 'state: restarted' tasks
gated on the Quadlet template's changed status so env-var updates
actually take effect on re-runs.
2026-05-29 21:53:47 -05:00
Hermes Agent service account
9e68802090 fix(semaphore): make host-agnostic by omitting SEMAPHORE_WEB_ROOT
The SPA was rendering <base href="https://semaphore.local.mk-labs.cloud/">
regardless of which hostname served the page, causing the imagineering
alias to load the same UI but rewrite all in-page navigation back to the
semaphore hostname. Confusing for users hitting either Traefik alias.

Leave SEMAPHORE_WEB_ROOT empty so Semaphore emits relative URLs and is
fully host-agnostic. Both DNS names (semaphore + imagineering) now serve
cleanly without preference between them.

The template only emits SEMAPHORE_WEB_ROOT when the variable is set, so
the env-var is absent from the container when not needed.
2026-05-29 21:52:33 -05:00
Hermes Agent service account
d05cfcf317 fix(semaphore): provide SEMAPHORE_ADMIN_* env vars for non-interactive first boot
The v2.18 image's entrypoint runs the setup wizard on first boot. Without
the SEMAPHORE_ADMIN_* variables it prompts on stdin, fails with 'Username
cannot be empty', and the container exits — leading to a crash loop.

Set:
  SEMAPHORE_ADMIN=admin
  SEMAPHORE_ADMIN_NAME=Administrator
  SEMAPHORE_ADMIN_EMAIL=admin@local.mk-labs.cloud
  SEMAPHORE_ADMIN_PASSWORD={{ vault_semaphore_admin_password }}
  SEMAPHORE_PLAYBOOK_PATH=/var/lib/semaphore/playbooks

The env-var bootstrap path is stable in v2.x; only the legacy
'semaphore user add' CLI invocation was unreliable. Drop the manual
user-add step from the README.
2026-05-29 21:38:44 -05:00
Hermes Agent service account
80f810fb0c feat(semaphore): rewrite role with rootful Podman Quadlet + PostgreSQL
Complete rewrite of the semaphore role. Supersedes three prior
iterations whose admin-user-creation logic was unreliable across
Semaphore CLI versions.

Architecture:
  - Rootful Podman Quadlet under /etc/containers/systemd/
  - Separate PostgreSQL 16-alpine container on a user-defined
    podman network (semaphore-net)
  - Named volumes for both data stores (semaphore_data,
    semaphore_postgres_data) so container recreation is
    non-destructive
  - Pinned image tags: semaphoreui/semaphore:v2.18.5-ansible2.16.5
    and postgres:16-alpine
  - Post-deploy HTTP health check fails the playbook if Semaphore
    doesn't respond on /api/ping within ~60s

Admin user creation remains intentionally manual after first deploy;
the role README documents the exact podman exec command.

Removes the duplicate deploy_semaphore.yml and the now-unneeded
cleanup_semaphore.yml; day1_deploy_semaphore.yml is the canonical
entry point.
2026-05-29 21:34:22 -05:00
Hermes Agent service account
9153324795 fix(linux-baseline): correct MOTD padding math
Previous template used inline arithmetic that miscounted the box
width, leaving the right border misaligned on real hostnames. Switch
to computed labels + a fixed inner_width so any hostname / OS string
pads to the same border position.
2026-05-29 20:56:38 -05:00
Hermes Agent service account
91b5817e5f feat(ansible): add linux-baseline role and day0_linux_baseline playbook
Introduces a single, idempotent baseline role to supersede the
overlapping day0-baseline and common roles. Capabilities are
feature-flagged so they can be toggled per-host:

  - packages (common + OS-family + per-host extras)
  - timezone + locale
  - chrony time sync against sundial
  - baseline users (jarvis admin account with SSH key + NOPASSWD sudo)
  - SSH hardening via /etc/ssh/sshd_config.d/ drop-in
  - unattended security upgrades (Debian family)
  - sysctl drop-in at /etc/sysctl.d/99-mk-labs.conf
  - journald retention caps
  - branded MOTD

Ubuntu/Debian is first-class; vars/RedHat.yml provides a placeholder
for future distros via the ansible_os_family pattern.

The legacy day0-baseline and common roles remain in place for now and
will be removed during the playbook cleanup sweep, alongside the
existing playbook naming inconsistencies.
2026-05-29 20:40:04 -05:00
Hermes Agent service account
1dfa7889ab chore(playbooks): switch day0_baseline.yml to new day0-baseline role 2026-05-28 22:06:49 -05:00
Hermes Agent service account
08d7da0c35 feat(baseline): add clean day0 baseline role (time, packages, hardening) 2026-05-28 21:54:34 -05:00
Hermes Agent service account
9ebeb42023 fix(semaphore): add SEMAPHORE_DB_SSLMODE=disable for postgres connection 2026-05-28 21:24:14 -05:00
Hermes Agent service account
075f34b1fb fix(semaphore): make network creation task more reliable 2026-05-28 15:22:45 -05:00
Hermes Agent service account
210c89c2c7 fix(semaphore): make network creation more robust and earlier in legacy block 2026-05-28 15:20:09 -05:00
Hermes Agent service account
b93a6e50ab feat(semaphore): add optional aggressive storage cleanup for legacy mode 2026-05-28 15:17:54 -05:00
Hermes Agent service account
85cc1f8c6a fix(semaphore): improve container cleanup and volume reuse in legacy mode 2026-05-28 15:16:07 -05:00
Hermes Agent service account
8e781b0c54 fix(semaphore): add container cleanup at start of legacy postgres deployment 2026-05-28 15:14:34 -05:00
Hermes Agent service account
d8ad35b8e9 fix(semaphore): add dedicated network for legacy postgres deployment 2026-05-28 15:13:16 -05:00
Hermes Agent service account
a9973d1e0f feat(semaphore): add legacy postgres deployment path using podman_container 2026-05-28 15:07:47 -05:00
4113011f63 fixed semaphore template 2026-05-28 14:51:04 -05:00
Hermes Agent service account
018782d986 fix(semaphore): remove User/Group from quadlet template 2026-05-28 14:49:37 -05:00
Hermes Agent service account
3681e8c03e fix(semaphore): make volume creation task more robust 2026-05-28 14:31:06 -05:00
Hermes Agent service account
8f377e4cf3 fix(semaphore): create named volumes before deploying quadlets 2026-05-28 14:25:18 -05:00
Hermes Agent service account
419acaa40d fix(semaphore): use systemctl start for quadlet services after daemon-reload 2026-05-28 14:08:53 -05:00
Hermes Agent service account
42b204bf8a fix(semaphore): make quadlet deployment self-contained with immediate daemon-reload + service start 2026-05-28 14:06:24 -05:00
Hermes Agent service account
1d7dcb7d82 fix(semaphore): load role defaults in cleanup playbook so variables are defined 2026-05-28 14:02:33 -05:00
Hermes Agent service account
d63ca0b4f9 docs(semaphore): add cleanup playbook reference to deploy playbook 2026-05-28 14:01:00 -05:00
Hermes Agent service account
e9440327aa feat(semaphore): add dedicated cleanup playbook
- Explicit playbook for removing old container, systemd services, and quadlets
- Optional semaphore_force_clean variable for data removal
- Safer than tags for destructive operations
2026-05-28 10:25:56 -05:00
Hermes Agent service account
dcc7e282c7 feat(semaphore): complete quadlet deployment for PostgreSQL + Semaphore
- Add quadlet tasks and handlers for modern rootless Podman deployment
- Fix broken Jinja in semaphore.container.j2
- Add proper 0777 permissions handling for rootless
- Support semaphore_use_postgres toggle with fallback to legacy BoltDB path
2026-05-28 10:22:28 -05:00
Hermes Agent service account
6d5fc7c5c6 fix(semaphore): set data directory permissions to 0777 for rootless compatibility 2026-05-27 22:38:36 -05:00
312fdf9986 fix env.j2 2026-05-27 22:26:51 -05:00
Hermes Agent service account
4e0b4fa049 refactor(semaphore): remove unreliable admin user creation from role
Initial admin user must now be created manually after first deployment
2026-05-27 22:20:43 -05:00
Hermes Agent service account
cf7c2a1436 fix(semaphore): clean up admin user creation tasks with proper fallback and force logic 2026-05-27 22:16:34 -05:00
Hermes Agent service account
3dc6555ad1 fix(semaphore): support vault_semaphore_admin_password as fallback variable name 2026-05-27 22:15:20 -05:00
Hermes Agent service account
27ff9286b4 feat(semaphore): add semaphore_force_admin_user option
- When set to true, deletes existing admin user before creating
- Useful for recovering from bad password or broken user state
2026-05-27 22:10:32 -05:00
Hermes Agent service account
6e50461999 docs: add comprehensive Semaphore deployment and setup guide
- Covers deployment, initial setup, DNS/Traefik, GitOps approach, and troubleshooting
2026-05-27 21:57:02 -05:00
Hermes Agent service account
3f1c3a40cf feat(semaphore): add idempotent initial admin user creation
- New variables for admin user (password from vault)
- Task checks if user exists before creating
- Uses podman exec + semaphore CLI
2026-05-27 21:56:36 -05:00
Hermes Agent service account
0116ec4cc3 fix(dns): improve hostname extraction in add_service_route.yml
- Use explicit Jinja2 loop for more reliable parsing of Host() rules
- Handles multi-host router definitions correctly
2026-05-27 21:39:51 -05:00
Hermes Agent service account
f962d0a6d7 feat(dns): rewrite add_service_route.yml to support multi-host Traefik configs
- Parse Host() rules from router definitions
- Supports multiple hostnames per service file (e.g. semaphore + imagineering)
- More robust and future-proof
2026-05-27 21:31:41 -05:00
Hermes Agent service account
dc3c0d7cb1 feat(traefik): add semaphore dynamic config with dual hostnames
- semaphore.local.mk-labs.cloud
- imagineering.local.mk-labs.cloud
- Backend: figment:3000
- Cloudflare certResolver + security-headers
2026-05-27 21:24:02 -05:00
Hermes Agent service account
b05f9fad09 fix(semaphore): add :U volume flag for proper rootless UID mapping
Prevents permission issues with BoltDB data directory in rootless Podman
2026-05-27 21:17:40 -05:00
Hermes Agent service account
8650995926 fix(semaphore): adjust BoltDB Path in config.json to parent directory
Prevents 'database.boltdb/database.boltdb' path issue in v2.18.x
2026-05-27 21:09:12 -05:00
Hermes Agent service account
e5469d2cb8 fix(semaphore): use semaphore_user for directory ownership + add config.json for BoltDB
- Replace hardcoded UID 1000 with {{ semaphore_user }} / {{ semaphore_group }}
- Add config.json creation task for non-interactive BoltDB startup
- Prevents interactive setup wizard and permission errors on v2.18.x
2026-05-27 21:07:30 -05:00
Hermes Agent service account
702698ddcd fix(semaphore): always regenerate systemd unit and use DB_DIALECT
- Switch podman_container env to SEMAPHORE_DB_DIALECT
- Remove creates: guard on podman generate systemd task
- Add changed_when so unit is always updated on role run
2026-05-27 20:56:47 -05:00
Hermes Agent service account
d233d582d4 fix(semaphore): use SEMAPHORE_DB_DIALECT instead of legacy DB_TYPE
- Updated container quadlet template to use modern SEMAPHORE_DB_DIALECT variable
- Updated env.j2 template for consistency with current SemaphoreUI expectations
- Resolves BoltDB not being honoured on v2.18.x
2026-05-27 20:48:41 -05:00
Hermes Agent service account
3839fac162 fix: run directory ownership task with become: true 2026-05-26 23:04:23 -05:00
Hermes Agent service account
b01dac85da fix: set correct ownership (1000:1000) on semaphore data directories 2026-05-26 23:02:04 -05:00
Hermes Agent service account
37a49824d0 fix: run podman generate systemd as root 2026-05-26 22:57:53 -05:00
Hermes Agent service account
0127016ab2 fix: run Semaphore container as root inside container 2026-05-26 22:57:10 -05:00
Hermes Agent service account
e0b6fcb24a fix: use 'latest' as default Semaphore image tag 2026-05-26 22:55:12 -05:00
Hermes Agent service account
e7d9a8fec5 fix: use containers.podman collection instead of community.general 2026-05-26 22:53:57 -05:00
Hermes Agent service account
1a9addc537 feat: use community.general collection + requirements.yml for semaphore 2026-05-26 22:51:59 -05:00
Hermes Agent service account
401f25b1c4 fix: use raw podman commands (no external collections required) 2026-05-26 22:50:55 -05:00
Hermes Agent service account
ece522074e fix: switch semaphore role to podman_container + generate systemd (more reliable) 2026-05-26 22:50:18 -05:00
Hermes Agent service account
c30c0074f1 fix: separate daemon-reload from service start in semaphore role 2026-05-26 22:49:38 -05:00
Hermes Agent service account
fa51dc2c4d fix: correct Quadlet service startup in semaphore role 2026-05-26 22:48:55 -05:00
Hermes Agent service account
c1810fde8a fix: replace semaphore role with new Podman + Quadlet version 2026-05-26 22:46:46 -05:00
a781ef8b14 update inventory 2026-05-26 22:44:48 -05:00
Hermes Agent service account
92b2a9d609 refactor: consolidate all roles into ansible/roles/ and update ansible.cfg
- Move all roles from playbooks/roles/ to roles/
- Update roles_path in ansible.cfg
- Add cast user to common role
- Create standalone podman role
- Add semaphore role with Podman + Quadlet support
2026-05-26 22:22:08 -05:00
Hermes Agent service account
9250b0f193 chore: remove defunct hermes directory
The previous Hermes deployment configuration has been superseded.
This directory is no longer used and is being removed as part of
the transition to the new command centre on carousel-of-progress.
2026-05-26 14:42:45 -05:00
24869f47ee deply hermes VM 2026-05-26 11:48:23 -05:00
bb5a57e909 remove jarvis deployment 2026-05-25 21:16:52 -05:00
9f3d81729d fix(jarvis): run as root for NFS compat, add namespace PSA baseline label 2026-05-25 20:43:03 -05:00
064d3e8b3d fix(jarvis): move runAsNonRoot to container level, allow init container to run as root for chown 2026-05-25 20:39:39 -05:00
ef7e3c61ed fix(jarvis): add init container to fix PVC ownership for UID 10000 2026-05-25 20:37:01 -05:00
64951e1e5e fix(jarvis): use args not command for hermes entrypoint 2026-05-25 20:35:32 -05:00
58931732f7 fix(jarvis): correct hermes entrypoint command 2026-05-25 20:31:46 -05:00
334 changed files with 30918 additions and 1560 deletions

160
COUCHDB-ERLANGCOOKIE-FIX.md Normal file
View File

@@ -0,0 +1,160 @@
# CouchDB erlangCookie Fix - Implementation Guide
## Summary
**Problem**: CouchDB deployment fails because `erlangCookie` is missing from the ExternalSecret configuration.
**Decision**: Externalize `erlangCookie` to 1Password (pragmatic approach)
**Rationale**:
- ExternalSecret architecture requires ownership of the entire secret
- Mixing externalized and chart-generated fields in the same secret is not supported
- Single-node deployment makes erlangCookie rotation unnecessary
- This is an acceptable deviation from the pure Harbor pattern given the architectural constraints
## Implementation Steps
### 1. Generate erlangCookie Value
```bash
openssl rand -hex 20
```
Example output: `f4e3c2b1a9d8e7f6c5b4a3d2e1f0a9b8c7d6e5f4`
### 2. Add to 1Password
- **Vault**: `mk-labs`
- **Item**: `couchdb`
- **Field Name**: `erlang-cookie`
- **Field Type**: password (concealed)
- **Value**: `<paste generated value from step 1>`
### 3. Update ExternalSecret Configuration
File: `cluster/applications/couchdb/externalsecret.yaml`
```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: couchdb-credentials
namespace: couchdb
labels:
app.kubernetes.io/name: couchdb
app.kubernetes.io/part-of: mk-labs
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: onepassword-connect
target:
name: couchdb-admin
creationPolicy: Owner
template:
engineVersion: v2
data:
adminUsername: "admin"
adminPassword: "{{ .adminPassword }}"
cookieAuthSecret: "{{ .cookieAuthSecret }}"
erlangCookie: "{{ .erlangCookie }}" # ← ADD THIS LINE
data:
- secretKey: adminPassword
remoteRef:
key: couchdb
property: admin-password
- secretKey: cookieAuthSecret
remoteRef:
key: couchdb
property: cookie-auth-secret
- secretKey: erlangCookie # ← ADD THIS BLOCK
remoteRef:
key: couchdb
property: erlang-cookie
```
### 4. Update values.yaml Documentation (Optional)
File: `cluster/applications/couchdb/values.yaml`
Update the comment block at line 9-10:
```yaml
# Admin credentials managed via ExternalSecret
# See externalsecret.yaml for 1Password integration
#
# NOTE: erlangCookie is externalized to 1Password for architectural
# simplicity (ExternalSecret ownership model). In a pure Harbor pattern,
# this would be chart-generated, but single-node deployment makes this
# acceptable. The erlangCookie is treated as an immutable infrastructure
# secret (generate once, never rotate).
createAdminSecret: false
extraSecretName: "couchdb-admin"
```
### 5. Commit and Push
```bash
cd ~/git/homelab
git add cluster/applications/couchdb/externalsecret.yaml
git add cluster/applications/couchdb/values.yaml # if modified
git commit -m "fix(couchdb): add erlangCookie to ExternalSecret from 1Password"
git push origin main
```
### 6. Verify Deployment
```bash
# Watch ExternalSecret sync
kubectl get externalsecret -n couchdb couchdb-credentials -w
# Wait for: SecretSynced
# Verify secret created with all four keys
kubectl get secret -n couchdb couchdb-admin -o yaml
# Should contain: adminUsername, adminPassword, cookieAuthSecret, erlangCookie
# Watch ArgoCD sync
kubectl get application -n argocd couchdb -w
# Wait for: Healthy/Synced
# Watch pod startup
kubectl get pods -n couchdb -w
# Wait for: Running
# Test CouchDB access
kubectl port-forward -n couchdb svc/couchdb-svc-couchdb 5984:5984 &
curl http://localhost:5984/
# Expected: {"couchdb":"Welcome","version":"3.5.1"}
```
## Why Not Follow Harbor Pattern Exactly?
**Harbor Pattern**: Only user-facing credentials externalized, internal secrets chart-generated.
**CouchDB Constraint**: ExternalSecret uses `creationPolicy: Owner`, which takes full ownership of the target secret. This prevents the Helm chart from adding auto-generated fields to the same secret.
**Options Considered**:
1.**Externalize erlangCookie** (SELECTED) - Works with current architecture
2. ❌ Chart auto-generation - Conflicts with ExternalSecret ownership
3. ❌ Dual-secret approach - Requires Helm chart customization
4. ❌ Disable ExternalSecret - Loses 1Password integration for admin password
**Decision**: Pragmatic approach wins. erlangCookie is treated as an infrastructure secret (generate once, never rotate), which is acceptable for a single-node deployment.
## Secret Classification
| Secret | Type | 1Password? | Rationale |
|------------------|---------------|------------|------------------------------------|
| adminUsername | User-facing | No* | Static value, hardcoded in template |
| adminPassword | User-facing | ✅ YES | User login credential |
| cookieAuthSecret | Gray area | ✅ YES | Session security, periodic rotation |
| erlangCookie | Internal | ✅ YES** | Architectural constraint |
\* Hardcoded in ExternalSecret template (not fetched from 1Password)
\*\* Pragmatic deviation from Harbor pattern due to ExternalSecret architecture
## References
- Full analysis: `/home/hermes/couchdb-erlangcookie-analysis.txt`
- Harbor pattern: `/home/hermes/harbor-simplification-complete.txt`
- CouchDB Helm chart: `apache/couchdb` v4.6.3

View File

@@ -54,7 +54,7 @@
# (pathspec) Colon-separated paths in which Ansible will search for collections content. Collections must be in nested *subdirectories*, not directly in these directories. For example, if ``COLLECTIONS_PATHS`` includes ``'{{ ANSIBLE_HOME ~ "/collections" }}'``, and you want to add ``my.collection`` to that directory, it must be saved as ``'{{ ANSIBLE_HOME} ~ "/collections/ansible_collections/my/collection" }}'``. # (pathspec) Colon-separated paths in which Ansible will search for collections content. Collections must be in nested *subdirectories*, not directly in these directories. For example, if ``COLLECTIONS_PATHS`` includes ``'{{ ANSIBLE_HOME ~ "/collections" }}'``, and you want to add ``my.collection`` to that directory, it must be saved as ``'{{ ANSIBLE_HOME} ~ "/collections/ansible_collections/my/collection" }}'``.
;collections_path=/Users/rblundon/.ansible/collections:/usr/share/ansible/collections collections_path=/opt/ansible-collections:/usr/share/ansible/collections
# (boolean) A boolean to enable or disable scanning the sys.path for installed collections. # (boolean) A boolean to enable or disable scanning the sys.path for installed collections.
;collections_scan_sys_path=True ;collections_scan_sys_path=True
@@ -209,7 +209,7 @@ private_key_file=~/.ssh/ansible
remote_user=wed remote_user=wed
# (pathspec) Colon-separated paths in which Ansible will search for Roles. # (pathspec) Colon-separated paths in which Ansible will search for Roles.
roles_path=/opt/git/homelab/ansible/playbooks/roles:/Users/rblundon/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles roles_path=./roles
# (string) Set the main callback used to display Ansible output. You can only have one at a time. # (string) Set the main callback used to display Ansible output. You can only have one at a time.
# You can have many other callbacks, but just one can be in charge of stdout. # You can have many other callbacks, but just one can be in charge of stdout.

View File

@@ -0,0 +1,6 @@
---
- name: Create jarvis user and deploy SSH key
hosts: all
become: true
roles:
- jarvis_user

View File

@@ -0,0 +1,154 @@
---
# ============================================================================
# Semaphore configuration-as-code
# ============================================================================
# Drives a freshly-deployed Semaphore instance into its desired state via
# the Semaphore REST API. Idempotent: every object is checked first; only
# missing ones are created. Existing objects are left alone.
#
# Loaded from group_vars/all/semaphore.yml so that the configuration is
# version-controlled in the homelab repo and survives a wipe-and-redeploy
# of the Semaphore VM.
# ============================================================================
# ---------------------------------------------------------------------------
# API connection (defaults to the local Traefik-fronted service-name URL).
# Override semaphore_api_url to point at a specific instance if needed.
# ---------------------------------------------------------------------------
semaphore_api_url: "https://semaphore.local.mk-labs.cloud/api"
semaphore_api_validate_certs: true
semaphore_api_token: "{{ vault_semaphore_api_token }}"
# Feature flag — keeps day1_deploy_semaphore.yml deploy-only by default.
# Set true to also run the configuration pass.
semaphore_configure: false
# ---------------------------------------------------------------------------
# Declarative configuration of the Semaphore instance.
# ---------------------------------------------------------------------------
#
# Top-level shape:
#
# semaphore_config:
# project: single dict — the lab uses one project ("mk-labs")
# keys: list of credentials Semaphore stores
# repositories: git repos Semaphore can clone
# inventories: Ansible inventories from those repos
# environments: env-var bundles
# templates: task templates that tie everything together
#
# Each list element has a unique "name" used as the natural identity key.
# ---------------------------------------------------------------------------
semaphore_config:
project:
name: mk-labs
alert: false
max_parallel_tasks: 0 # 0 = unlimited
keys:
# The ansible-vault password. login_password type with empty login
# — only the password field is consumed by Semaphore at runtime.
- name: ansible-vault-pass
type: login_password
login: ""
password: "{{ vault_ansible_vault_password }}"
# SSH key for the gitea deploy access (clone the homelab repo).
- name: gitea-deploy
type: ssh
ssh_login: git
ssh_private_key: "{{ vault_gitea_deploy_key }}"
# SSH key for the universal automation account 'wed' — pre-baked in
# every mk-labs VM template. This is the canonical user Semaphore
# uses to reach the fleet.
- name: wed-ssh
type: ssh
ssh_login: wed
ssh_private_key: "{{ vault_wed_ssh_private_key }}"
# SSH key Semaphore can use to reach the fleet as jarvis (admin
# account provisioned by linux-baseline). Retained for jobs that
# specifically need jarvis-level access; the default is wed-ssh.
- name: jarvis-ssh
type: ssh
ssh_login: jarvis
ssh_private_key: "{{ vault_jarvis_ssh_private_key }}"
repositories:
- name: homelab
git_url: "ssh://git@gitea.mk-labs.cloud:2221/rblundon/homelab.git"
git_branch: main
ssh_key: gitea-deploy
inventories:
- name: production
type: file
inventory_file: ansible/inventory.yml
repository: homelab
# wed is the universal automation account pre-baked in every VM
# template. Semaphore uses it for fleet-wide jobs.
ssh_key: wed-ssh
# become_key is Semaphore's sudo PASSWORD slot, not a second SSH
# key. wed has passwordless sudo on every host, so reference the
# built-in "None" key. (Semaphore rejects an SSH-type key here.)
become_key: None
environments:
- name: default
env:
ANSIBLE_HOST_KEY_CHECKING: "False"
ANSIBLE_FORCE_COLOR: "True"
# Semaphore runs ansible-playbook from the cloned REPO ROOT (not
# from the playbook's directory as I first assumed). Path is
# therefore relative to repo root, not playbook dir.
ANSIBLE_ROLES_PATH: "ansible/roles"
# Collections are installed by the semaphore role into a host-side
# directory bind-mounted into the container at this path.
ANSIBLE_COLLECTIONS_PATH: "/opt/ansible-collections"
templates:
- name: "day0_linux_baseline"
description: "Apply the mk-labs Linux baseline to one or more hosts."
app: ansible
playbook: ansible/playbooks/day0_linux_baseline.yml
inventory: production
repository: homelab
environment: default
vault_password: ansible-vault-pass
arguments: '["--diff"]'
survey_vars:
- name: target
title: "Target host or group"
description: "Inventory target (e.g. figment, semaphore_server, all)"
required: true
type: TextVar
default_value: "all"
- name: "day1_deploy_semaphore"
description: "Re-deploy Semaphore + PostgreSQL on figment."
app: ansible
playbook: ansible/playbooks/day1_deploy_semaphore.yml
inventory: production
repository: homelab
environment: default
vault_password: ansible-vault-pass
arguments: '["--diff"]'
- name: "day0_linux_baseline_check"
description: "Dry-run the baseline — shows diffs, applies nothing."
app: ansible
playbook: ansible/playbooks/day0_linux_baseline.yml
inventory: production
repository: homelab
environment: default
vault_password: ansible-vault-pass
arguments: '["--check","--diff"]'
survey_vars:
- name: target
title: "Target host or group"
description: "Inventory target (e.g. figment, semaphore_server, all)"
required: true
type: TextVar
default_value: "all"

View File

@@ -19,6 +19,15 @@ terraform_server: "infra01"
# Traefik variables # Traefik variables
traefik_server: "lightning-lane" traefik_server: "lightning-lane"
# ---------------------------------------------------------------------------
# JARVIS automation account
# ---------------------------------------------------------------------------
# Public key for the 'jarvis' user provisioned by the linux-baseline role on
# every host. Public keys are not secret; the matching private key lives on
# the JARVIS command centre (carousel-of-progress) and, when needed, in
# group_vars/all/vault as vault_jarvis_ssh_private_key.
jarvis_ssh_public_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID5sym5ajFvDyzw395BkHv7qVb66XPTx/OF1p19MGuNo jarvis@mk-labs"
step_ca_principal_mappings: step_ca_principal_mappings:
- local_user: wed - local_user: wed
principals: principals:
@@ -30,3 +39,7 @@ step_ca_principal_mappings:
- ryan.blundon@protonmail.com - ryan.blundon@protonmail.com
- ryan.blundon - ryan.blundon
- ryanblundon - ryanblundon
# Leviton My Leviton API
leviton_email: "{{ vault_leviton_email }}"
leviton_password: "{{ vault_leviton_password }}"

View File

@@ -1,306 +1,441 @@
$ANSIBLE_VAULT;1.1;AES256 $ANSIBLE_VAULT;1.1;AES256
33333438373661633337383161343862383963623736636136653339363064386538386437323135 61346566636664323837633233623331666232363731633634366538333962363966366433636232
6436666330363032616365656434356161396530363363610a333864313064633232366133366232 3363303535353934306538656332633436303638373933610a326364636563386437386466653335
37316636366334346536663337663065303635626638666264666435393933343832653061323237 61326564366131383062363037663965343330663431396331343831326133393233366439646538
6663333736386636350a653931626665346234323832393334373563303130626262646565653231 3435366135636162300a326161313661633134393666623964323761343139323630353938313864
30326263623965356639386438636531306430343239313162383065386135653534636134376333 38393339343231636566386534363831616237343531373835326662376162373264633038353330
64616432303732646535643530353963363734663538393539633034326535643539356561656632 33663539646165656261326663306332313932333761383937633265373330346134333532616138
36363330346131393262326561356631343631356434666264616134643136646238376336373665 36623861356264643963366138646233343436643066396430653663323039333165306536383565
65346361636435316437353235393263313766623438336662353532306463353134366435326364 66346634656131323066353837313836356162346330376232363835653331323736383839356533
36613865343264616364333631353238663238313338333232626262313637633662633163366638 36636536313630623730333631383166383032363633643263613335633938616235656166356164
61323166393739656536656435336538303439636561363961353832313431666432393463636537 37376563633634666465663332633635653736393135353538346366386630393765316364616463
62323939376265353330356334303334303532643136376531343765613738656562386665336132 64343563646233623330633765326537383739323835306566613730323630376339346139613163
36346232313432396431666535663362333332613037623531636135376439393936353261653863 39623132633830393635313537616661326536643662323237353337316234663931626261306639
65373061316461303866363338376131343736323933383636626338616431656533363833663266 32376434363964363437373165613062333966653331373664383633656130313764626639353164
62363935313262393733333566386337633630666332353263666462373164346362313034636239 61363261636362326261383430666235316538383933643365663830336665633164613435303065
33656462396238343065613965613232623132343562323364653366313231323531326364376631 65313761623032373332623932623765616263633433663466366633363333356330393161346536
65643961303862653030343364326533356336393031636437393465636263626236303633326139 32373966366330363565663632306130656161323234303666313636353166353064636266386239
65643432663037333332346261323663613031386563383533336538383133373332343363326530 38363935313564323930396531336566313066386630626431346633323331633436326237633862
66643231666636366435623639643166333863376533633537363364396663363732333932663333 33363334346264343537333065336536343264336538643862386336666635383766333235323034
62333239613264353233303064333931326236396538323130353061366139623362393734323434 65313764343637343361323064333864343464356336376333613361356530343765626166366337
36343537393363343861633131346561353637646336313233376162656561623362653565623339 30326266316135353263323131326664326635616231303831353036333439383633376161383662
63386337613461343534643964323932356661383436303966346339386131336133643132376330 34613337376237393537343932326330323564313931376261393163653464316232373238393535
38656535353335303966636235643666306336396435636434363733313665366234393032333065 38376637666533363166646336376665363538643363666537393232643762383130366165373037
37633231376134636435343233363032666231666134623365653238373836326237373831326262 39636531336664633539653162393536306531313637663039636431353365396466363236376432
66303234303164303961316166623062616139353264343864363938313733653563663661633535 66633064316161623739623138386664623165363762663637353435373863656661393362353330
37326636346435626362336437666364623264363565653935336438386262376663303230313265 62333163366438343364626432643232396133643065636133626339323935623061323761643331
62663665666132313436663330616230383235346333623563313262376530356535613263316436 38393466353833643432643265623431323436346638626661323036303334303239383838396335
35313766363763386234333733636464326134633136346532346632623534646365623136343363 33353031373435306266303466366264363737326562326666363066646362623335306465363666
30316435613031333036383835656230353430636135343961356363613139353763343233633834 34343862336264626134336338626330396130393833383434363834336133393032613237393031
38656431363139396535653565633631393462316334666464333839343035616531643466643531 62633133663433653662373636356231396630626463633432326665326461396530613162373366
35613733373331643961353830376266376433363938373563653833343464633465663766333066 34363462353865336166313639663438653839376531623533376335323863383632376433343432
34633364313762633034633461623232656537653762623532613332386132333365383062653136 63316239333566356362306133323332353937366237616237326566323363646261376434373939
63323234626266336462333534636236653230343530336532653831313339613531343731313237 65623031396139396164653939393162393930326530656135363564363565643133616261666436
39653737663665373361326433643364656434393838326339636238303964366336363961653366 39313366383334633036373832323462623161666563303336383131363163373233616435663065
36643938633635313432663765316130326238306163346566313062626439626263633033623731 33316266626162363063366539323462346464393932393165643161623964373539393964636139
63323532346366653238663931653334613734633963363632643462316130393138363436666532 34343538663533343931613439663436383531393939333239613234386465343430303833353133
32373833323330356164626332646232623031326365653765336261323661616137373732356164 64636338343032373230346562363338303166616166666438646262393333313633656162303764
63383633353764643233323430333233306337623834353934643864636339646239636430316166 61386164353835383233343535636132643831623062393539336233626235303032663464386266
33643030333261323461643031613730343131343830353463313766353035326436306566343161 66366664346364623533613832373631313336336530386162363861303333306563353439333236
35643534323763316165343832346433356364343061383036306434386631356537363331303035 65316161383538373733633137633065633362363936373264333034646135366434656534383030
34303430393034393763343964613935383739656662643066363633333839396465386231383837 35303838326361313163636432643638373832396165626532623261366332613562383261346661
30336331393365346636363732616661633635326364366435366466383639353839343731616365 32386164366264356536653439353763346539643934393466643166333761366436613165623064
65326633636263653438343561633632646234316461616666363630333966326230356331373335 38623530316264333939383932313133613730383632663830366365343937313036373439663930
31353665626233663433633563623765343431376639316232346666663065656462616438663333 66613034633532303065333935623264636534393638386433346336353765316264346662313337
62663662336662346563333935333261626635343764356631356363356362346231306535376466 34613732356265363839343434633563303665313030613535343336363432323366623234623738
36393932666662653934626134613065623461303834393231616332363461306534303666353632 38396439376335653731346464363533313161343162653930656137626337623961336232623165
35646363333065356631393863396536633838343365343238636332303862306631346639656166 37663863313532626638303937326237353235393866303063313532343164303165636262646434
30373735336361653863326331373834663164393863666631623866353338366561326635323132 33316363343265336462626630376665336162336661616436656135366137633834326465303964
35306535626564376666623837666435663337613164623966306138353161313239373436346239 62643435393537613836656334616236333131653133653065636361343261643032646163633462
36656266323233636233326139326266346464353665323465663666646264613466636132626464 38666465633531383235613732366131366637336638393639346463663830626539653538636165
37353762306538643233663338613062353134323832653139663036666337643131613062643936 34633238396233646337383936633438393039306461633631363161656465336238333434633030
37633363376365356333353433613839653130323036643737633163336531323032393937333432 37346334396538386437323337393562353463336130363837393033353834386631396164386466
39343238396534353330346664346135326333333638666261396463656663323935643337313462 39343338383634663338373536656363623234313339343363636432303937316462376139333138
32396137613432623664323134356361323861306230353165663162663732326563626162346630 36386331616639376331393963356138666463306666306436643237303764346330393332626465
34626536393962393764393764393234373138656432346332633963393135386238313563353761 62666235656334323231626635333338363736663063656664326139346363323039393565363966
34366438353337396264373032336166663937343931633635666166343638386237396333626133 36326662353631646135623832333266366139313662316333376237326433626434336531366666
38383766616566336163363931393438366566386565663938356630366430393830653037363738 63323062393030656639353634336339343034316237373932376466653930616465346436383664
30666465326266363834653934643434666466326632366463303965303366326364316430366561 31343339663333386533323261393164363039386666323766343337363033366239386266393362
32636631663232373163636535366333623261366466623262393631626334626263656534386536 35653333666432376337653332393930383561626431376364326439633331353033613361376333
66323833353264623538663232376138346361646362313565333534363535663031643833373064 36633965616237333662613931366533366636393432393863323330383431326635386466646565
38313062346339636335633962323933383230623634303431383236626166636535326465316439 31633136373466663161643965396434333437626636626237383630663730663531636139666166
38646636336130373763646635386265373065326263616438646565313439613830316135616435 63613964646133666361303430643766333063643732393062333631616132323762623735613566
62326430336337643265306261343036653938633634626561333664313035393933646533326661 62613362366536386565636538656138366532643932373163353437636562353737343566613330
62626437306432353938356338616462663130646530623636393233373837336138383963323165 34366436323063356433303863383961303130666637366235353264356433313066616262313532
35643231613565313434386163643637633130626435376561376235346438303234386637383835 37653165356230313165323330666339376337656332366531383566353536633831363430353764
37356238616263646665336435306233396436666664396532373063626162346237376662393639 61383533636263393336313631393766353638353862363666323163356631396234326262626634
35373665373638396434353062666430633362653163633639623836346438323632643164653638 64633438333438633235666361373333316265326237393134346539343361353461666664306239
62623463343465303236346562386338646536656438633966643430343266313935323630616633 30643939633462303463626237653135333863336530326438313233343232323634633935653134
62313531343764343966333439343866613965303036616634363162303434643533366465313562 65626337313266356462323039393564356638353263643566313264616239353036323063623333
62393666313838313938356265633531326431333730626632343139666338656465306431636162 32636561373166653739396635633039623464656536653531356365393362343132343631343433
66343532346238393464366134656334363964626264623263353537633638346239396539653538 30623438613137333063653061376434336233666633366530333634346136663831643866386132
37303339613036306533353132326534306433353364613536316263326363323134333639346236 66383865633531356533316437383933656439353262626138623231613532643834383238323438
39376263313431666462383863336638306631663563313364333235333338323364613361633532 63616238653338656536316662306565326233646136366634666330666263356235636231393133
38313263353539633662353533646333353262356436353938303036663130306362303935356136 62373665333336336166363936656636373865666132646465656563653464633338623831616436
63313234363138383864613664666635613464653765636461383738356135623662313866383036 38663730353764386164653166666136336665383439313135346139373239636137333035323862
64626437343465333563346434323762313232643230323336366539326631396463303961666461 62366231323434363864656532323462633061663765616365366335376533336661656536353537
61656338643532616337653362636362373837373836666535633762343335623033316264366261 37383466356438343539363935653534336332386130306235353565383933316137313233383832
62633137356132306430623537616135613965393737323263343463633839393165646630383331 30663663346339396236633733633464393631376436353833663462316533646534383134356165
36616339346534346636396532333836633436323232323364303963313030656330386336633538 33616139663236653661383063323865633339343836643134396364356230323135363731626236
34383534313236396461653462306130376462396138303561636331663062303864326238303335 65333338656430376466616137303561356433393733373961313862663430656431343363363635
65663832333631613963326639386665643135366632393438666361376334636233336239633837 63663561396438646536346237653562663661353438363237616539393335386338363132623937
36383761616364363262306535653065303536363066316431303464373230326436336661396231 35613838616134633334623939333466646131363663653961646138613064366361646235346130
66333438376664333731333630383031366237396638326534636532343535303964653436633438 64353130656532333664376635633137333434343965633361333830366232356434346263646232
64616263366561653334346138393866353437393037343930623237326231303261383733336232 33336635323062653161346138373636613638623431383538343436323737316335306334366339
37646533393439653965386563333232656565663531396663303334313839323361363864396332 37303033646363636637363333323533363037616536303737653064333230313761313366666664
61656638303130303433666561646663323139643337326561333832373538306236326538363637 39643861393636356663396431346139646562383035396461323165663665626238653536353133
38666534633330653564653066643335623938373331376339393338343533366436333330313165 38346564383761636537333961656538656366333530383831306533623532303635333539616165
65366562386366666632653064663637343536313462326562626436356363306134306237613265 65303537663934626561666335643361333339316434373263613637663037613033633930313437
32366338643237333739626331393763323934396265383138633035646533353536633866313432 62376233613638636234303264353339653963383633303761316339626434303466376431343664
38336137383036373434363530353833653938663035306337643638316463373762613931373038 62616433623431353164383735313835383935333538636432386562616438623062643866333137
32373361326136616331663733616133623936663530356566383461336334663937383230386462 65366263656631616334303538363239316632373936663431643239303034373831623765663662
38656332613763653339363565613762383735333163633831356364333563363630623666616430 65646133393263373639393664373265376664356434303131396465646161616464623734316436
30353237653730613361643233386139303639666137653733353334326530306431376666663664 30396265396536333731363535613035333066616136306538623165306462333561306337386131
34633531393261343165663330373961646665396139323762313663376639366464306533623333 38313135663030333439643361633864663061323738313862326131623637663634323131373263
62326432623334336166656565656230313063656163636561326632666438633966323930316264 31383730653838656630306138386435643435343162626638656338333131363137363063616432
66373866663461616336336264356561373232313737353836646632383438333162383832616662 33326636383530343335613765346634316530376635636233356364373066343835376633356137
30636437613163613631623165366631306438643765383263626133653231333361383633363136 61356336616138383536653161363930306364313035313135656634363632353731393934656562
36653831623562653838633133646633393038666164346461333531373034356435363336636338 32643362636466656562666466613364333739393736313130656137396437373763323138353362
62623464376264363537306166613461396630303039373239656664396564333435303164613235 65353661636537623230623666373534646635626538616431323966326230396637326335326239
61346238386233653462616632303337663036626465663361636130333566303337313237626237 30373534616362666435346263646238316637646330336635346437333537643630303062303338
65636633316366343064663636383362316665633263633136323563363935346138356635653036 66626237303830386164656531356439656162656633303564316636663639383662306462373430
36383835323838366363363335323731333066393334373265653563336533616362633163343661 30616465353461366532356339313734386137623566653262656365643136303065353834646365
66326630353537653633316631343332303231373630363635623136386537666434383032323165 66353637306139316132386239306163343564356464316537656237623734326363333137366463
33333633643064376239346139633664633330323966366430353135616633633465363030386665 63633864373431353064393561653366653233616233616536373837353065353330643863663361
36353164623564343934323063313763316566373830643163616633643937323732333561353832 63653336633135636564373632653561393862633034393266613337343132363034616139366665
66623364666335316230346665353631663465633034323265666663653866623037353935393262 32313536643066376638303465616632316533636530626165663561383562623133383834376362
61623065656637636666383239343661616233396663333038393963366537326136653863333235 37353633636561326263376366373430656562306633666431316664313832353039366437396436
65633137626134383763663366396438366634313465363965343830333966373930333134353339 36333233313765363833316536363132623633336439653734623964373461396433343536316635
35396538303537316636333339613731623636646230323662613663306537326566656666323434 33356266663630376662373632323436313431376435646561336665366634333135316334303863
65346531623533336535393530373963343864313963666335636334396663393837396665373565 66653461386632303362366535346434616438653135613264666334363363663633666130313734
39646138356134313732666633663061646330643438393736653433313238373135656636643337 36376664636363623965363630326236346566393961326332613638636163666562346661363663
39396362383363333434376336613933613438653636663334336337653633613433623336623461 37663939306338643134316163333233363164616232353063323134373565616532373636653964
32373565306435646438306365343465653863656434663937653938393839333739363639326536 34363237353234393632326438346132353036313232313231663762323131303830613532386262
64303534383961333333386237366239386330386533343939613664363439373135333731393739 65663836636266616661373939383937363330623665656139613239396231326631613461396435
39363166306636333766333332393265613133356262356666343039666237663664636266626664 30343064366431343735383135623135643165376464343138393131666235323832623034333466
65316339323638353333653631363365356562623535666235313234613236326130616261663830 65663932623335323361313930343330636232626661323464623236313262303638353933613630
34663063626539363734313562656233656131326265626637353239636638396564313561383761 38646365373164633131653430616338643163383639643134376464336537363861316137353063
62663638623135343333316464346236353364616239633566393039633339306334623234616331 62623965633339303838663434306161633733373365383863633462303161643965326433333832
34366435663938346335323932633330366639653636336636373562306561656536653363366233 62383731376261303364373931376631626138656535363638376236393133636261626234616137
39383064623435356338636335343662343435336564363965646439393964323162363736346637 30333637656139343535636534346239373631633966616462636637306639336630336337343832
65323765333364353831396433666561376437333834313334316165386631616332633533666336 66373730373161613935393230623934653530666464636363653066353566333730336261386138
37313533623535313934393230353439363231623063663961376537393638636237623134323337 39626537306532633937353761303765393530316236373432373734373965623238303430623632
30393032333530316330333736616463633262616631353238323933303766646137386465643366 35396435343065346136616632376533633465346336366137663836323234613735306537373631
33356665396437623732373437373537303034353431353031313162323730316566396238653038 39653632303133633032316634323966333762623130613161323064346562346163633361323333
30366234663037653736353137376264363934646366353761323364663538346662323934343266 37613735666361366530356363393262393233346239653137303330336161613461666330636530
30373662613461663561623466633939396439653663353365346133623933613462623231616562 30366531633538636433323831646433646230376335646535643831316238323233383534626139
66643738643661643766323266626266313330633936313964366536323263373666303130623738 37343635383333633034346436376433303634383336316535306236323061623365303432326632
32323230356665323439646532343930333936373536626466656566363161646663616266326133 63343036623136363966373363373864623262646462333332616635623831383338666438333962
32393137663562613531323832313565346632303131646432363961613331346636653830323136 35326534656537386636363234316332653232626663623261343737636361633338343933373066
36323436373964313737313131346131353630636561636639316464323666636336656265343565 39346537653261356366656135303961356162363763623532613631636239633432393266623966
65303936353165326137613638366532303265356537343337656165366663383961643134656435 31303434343438643563303463326564393632653938656661366630323636643364333664306461
61383565306265333939613561613035616663336361396536343939343761663833346134646365 37616563663630643462653634666166633333393466373462363937336333303963303735666234
34323233666439623436323261616432363838616437386138393735383364393164646638373630 65666632343036646536393965643837313234353537663839353032323362663237323338623139
33386632346239383037633132636431366264383633333365306338333661303662363533376339 32656663313266393763343133356438303637306633306231323033643535346637386332643934
62633663366362303038616666336335346332316366353136323666346531393938666236343163 36653638623334343030363330643237636635633464336561316331383432636438333839336535
31663665643839313439663839353431343061386465366331663637373532613266383065393639 37346436393066626634623136366664383734653936393335626636383736343266653863396331
62613565626135363133623261333061373738633333623337396631356332643039306638303434 34303234393663653438333466313962313731666165366233313162613437633430333562343163
65663762373630323263333239613939346133626534613632396433376234663636363062656537 36663330656639643633633336643436636564366533383235383262313865656538366531303337
62386664343462346334663662663865653766373738306438316533643962356136653565373761 66353732626333623031326261306633313666393130373562626564343539313338366364656336
62393437326138323065396630653838326138653865643937353431656137393732346361353734 61353837633765646564653936366236303463363531336537333965666437303536643135393062
31326634653331353232633061303230393462363065613832303630383237366463363666613864 38343836316265396665386664326532616632346265613334353466643165663463613661333134
37643135303034326335326563353064343835323863313134616262386564313235653762376332 33623333343037333133656434376537303462343061393838336438346635363732393337313532
36636533646235343138396537623266333566633466393764353731393935306432313266636133 66346366663538373237666565643662383665623130346466316663346162313964653830373630
63326362653033613264333139633638643638326532313239646239386332323330306538366362 65663134303466303630353764636639613661613939396438633234636437363937356465326132
37646432363535396161336465376363343430333261396536346464356135343462663462613136 38653132613066356633333434366265303235363664636432323566643838636266363762633064
36666238633663326230373834303431336566366666666534343537313635366537396262656236 30383332373631326635613266366331656664393934353731653063366537616435396432663731
36646639643730393134613662383238623835626230313833653932303832323366343765343434 36376437326237303263393334386263396334316333316132653765343564373935353563353531
34363033323836663035363166313630313062663162313165636138373261386636623831666632 63396539613739316539656332623262636136363839353339353331306439323966656530383564
33373962626237666130373031633437653966386561363232363332333633303961323864333263 34336362653064656138363861353331306164663834626662646263623561393764303864306565
39666230383332376537313137643635623336316336646537333137633564376432356432326639 33366230303663303839313364353430363034316437326135393530366335363065303335636564
61383565313233376435636366366531363462306535636131636163636130663732383161616364 31336135333163643864396264383437633736336639333539656661653638333936643030396339
37343064333264663533343739356139353365386565383833363165326536306431393764303632 64646336323338376530653735613739333939363366363333623036663861613061366531643034
30653865623566363965306636366530303261616663306433393135656366383138356337383365 36303037363361333561313961343730363533383435376362333739363863643365616166613036
36323736643035383231663034626564643235646237623632383164633062326132363331353634 64663134666166383663656338323933386362393162636132306331633065323139663162383632
61653761346439326665366565383534393635616365353361353863363763383664303166383434 38373264376666363964653138623232313732626665386333643435663531373532346539636636
33393732313837373336616564343864633330663566653933356237613538653366663838383132 34303839653338366330613734646636303163643532373030613239313239646630366438386662
33303437323438313837616134616466626562313262653866313034383764656566373331343738 36616263373562626435346638383836336635386661396565653836633339353938383164373137
30613062303630663365636230396231313065356662376163353632613564363066316239636434 39323364666636356561343139373830383865633466656532336361336363363964353933613133
66356535386263353737383838653538316366343935323337386561353533353161636538646363 64343833343330353433376166323531393066356437333132346133643364383562393731333234
63643739376366326536333261306263653563353331393133323964653764373932663136316335 39363830653233386566356130356439353535353636656362616534326336616163316233353962
36333535626231373864316634613366353535393932373039646161646466363663626561373033 32663361363836646166326663633730333436376230663235393738396639616331303066626662
64333133353931303361363466346666376431363336323166356162363537633536336638653838 31643139303835303431303535393536643163393264386463323862303439353536366131373033
35323562653138393833383130353232616564393738326638366565616233633634336438303632 31653234323165373562313063636436313538393639663962393139323231303863336165383830
30353765623565336330386335306162336662643466663335623836383238316161633339373137 32363535616361666533316132343965343661373762323064333161383463343731356161326333
32393165323735643034633464313337386362363331353432363233396538393835383933313637 32623464666562626531343664643933376162646631313166643365666465383035613464643762
65656330383732633537646361666439303266653165313233636464313964383639346635376538 30663065616166613531343339383662303438653531333866396561346637623664376266373839
64633861623439626331643431326336646135656539313836323135373862316438393237663564 31643233366363646632663635343161303033323061353737666233663561623462663837643236
31373135356135356662336265633862343435356530393835376430666566653036373338636332 64613363383631663039613031616133386130363730333361393739356361376165353036326237
64663230323361356262356537656262396665663538386664363735646234306231613339636338 34383533316436333465313939306437383337663864643935346336353935303664663866663564
34366433326535353965376236643661646339366566663531653864316533356466653864373633 34346362313730306132373936663764633662646466333135343332393765373333333363376135
33363330333765373662646634326236313434383433653563383436316635353461386662646165 61303839363236373066643066363135306266333364666132323033393836356531313232363864
31626566616163373137326165323066646535663766636265386435633965326165303831323633 61653565323263306263653535306634313662366331643663643966353164326161383530613730
39613139623835316165326138386431383530383035653931616334353235636138303761383037 65303738336561363161336330373832393062613130303131633265303037396230313839306661
39336163373332346365333935656563343939623335313461363138656239666263366236663464 62373938613034383966336137303431613161343937343461663066343436626563623366646132
32303732663937363964346639323236623166373334386239613564643834316536363634326461 32386238353662663337363363393039653330383331636430343632343064383565626338313030
64623939363635613863613065656363303734643766306564326437663037623839303734393036 65643333316431616637643162353634323333313165343833636465363461333063306538396636
64326363626562363934313930383766383462663766336666663338653765656137393362643366 34343832633565383961656537373562636433323566663830326136353765303230393939356132
36326336323638393161353464396637633566383262333234656233653334623131323532376637 62356130396264633930356362653066363163623631613434326464383361643262373936636461
62613064626532616631646232386631623839326665346431333232333463373031316237363462 61616631363335353366323865316439396335373966393464626335353631383137376532346262
30643731653666643331643839343865646437613135346363646131383262396134376363303335 62343930633430333335623639643431306533396531333666653030643161356339353933643837
34303164303236383139326264623462363062313139653465346435336163653863633764613631 33343734373435303235646239343838363439333433333437643162326539326261656330643938
33623666316530353163396539323238663935643764643939333966336137353034663164643733 38306131343464613330636636396266353638616230333263323334353862363738623362633034
34393132323339626331626532643266336435653636353930393738363134353661653636643263 38626566303336393163366265306564646436623630656533633630653836333035616364623031
30313138393263383837333438356463636162383938396137363035646433663932613066386138 35363735343835306132646664613261353230336537646161643463316566343430373461326233
37316535373939326131643930353435383232623366393630663463633233393232373730333631 62396563343939643432663062633064383738383834336634323635396461383830363063636432
37623837373533383462323431323335316231356436303239313535633837353931613730303363 36666566396535363835303034363732653433623665353437626533633032333838316564323834
31613765626466636636613366346166393766373834643337353764666532376335376139316138 30356132613563616333313039666330666565666534393139653065663739363061306365666135
38646133643736636363363434306334373933633264366530373534666637393835366266306633 63396262393464653566626635643365356533623761643763663662383832393164326462313837
66653966613136653936313732396366323830376236626530306139333030323966383035643733 35616233396165363434633434613031663739343432366263303734343635336136323830333163
36353833323539393136303035393866366563323430636336356639623734656135356566373062 33626461646661343564303134366139666639383630663438396139306566643636623431396332
34666433316139316664643234623530363732373338623162363636303463346463323131333065 35633232333064383161383933333931636561313035343561643534656635363432333266376536
64656566353837336266386233323439623237366364343635306137663038326434366531623062 37373537336239363166306430376132323862333732393164323964363966666262346564613631
64613365323339613465326464653965613439653766623762363139366264663765363862323839 39616438623830386266643634663562396336666134323137366233363061613332376661326435
63346539323961343532626465363266626234363964313531353662636635623165376130616536 66313565346663343538383235356232346438383835623364353334376565393464393966366538
64363361383766636333626366616531343833633033643965626435313130653938316430313532 65376334623438663562623930616266353961393430353361623034306231326530303438316339
64363731633139383236326231653434663132313463326563393739393132663765396337643035 34386462666166363534643764633563613934366164303531343366383464643832353739316433
32373963393830356362366130376135353935396631393533333430363531636234333832373939 36326364663737353562396261363736393562393964633539363031353432356238333461383730
36656233343663623262346337323034653036373432303266353333626434353137353939346665 39386437663434326635373465643234373034303963396434656161633839613731326365613736
37633432303162323538343339643635636464353333313063363662663033396630303162333639 36343664386565313535343635376532386331643264613563613735373631626466313964383639
31633537373364666234653938633461343738333464653736316331343038393333643936646462 65663761663033333932336665383631636563363561646635383965393039393363346166613861
36373232373962653031333730363166393463313232616631383233386533373464366465623335 66393466326634633438343632376538326634353937633562333137373565343339626635376133
65393462353638633931353331313233356230303565313030663739626230323766666466326339 36626638393932623837336162343831303436643535643232616432363061346165373965323365
35303361383065353333666161306433383139366439633730666364383130316637633464646336 63386632336532656635643863623132393931383838346263636339646332373665396632653361
66643065303934663765623233333261313939373236396136366566306438626439393163636430 32346330306362343538353833313833383334366666613436343066636661366238333764353863
30656334663361396230303239663338373664346362623934313130343064353435626439356562 37333534366536313464383364303335356232363866363737376665616438666533633066373164
35653638613037656133313765313165333066353732363361386234353635626563323932356237 61636362363237313362633865333630386430623431656437646437396131363035616434343136
31363333326536353536323931643739333764323637356164663566343563633761333462383735 33353839353765623531323061306465653336623333393837363538653830326561396137613138
66383134646139373130386366316532656139336231303931656265636130643130616531366530 64646133626130323234383763623935323433336535343631383535356437333864633330393134
62383132643236623835323362383866613765343762333536333365336338373264633130656438 66623530653636623565353830613538616433613538383632303539386362383636376465393036
36343331666431326530626137333665623836636136303031333965356366396239396534663263 36373930326430373336666138653334616530653930336364353464613163656131323566393933
35343730303534313830623430666566386564653036356635623038303133623231616635396463 62646265393539313432653462616431636232326131313239363337396635336131383534353338
65626137633033663631373366373238353766323533643462396562633164633730633135396266 61356535663130336266376431666533353864393766313962623563313131336533656461373836
36316362353964306463336237383232306530613539653462326137383938333638353038396132 34653833353730363433663435313039363861386336643937626430326563353230386630356437
66343265643162363735663235353161393732326530373735363662633165316265323065396163 63396636393561376330316232343964666337663532353438303964613365303632323263633161
31663761303637616566396262346238366631663562323566323364663464373930373831323630 36323033313935636239633232333632373637613439633332666665356331363532373138393861
39333165336264336539353634633936396561626363353466303931343436306636356663346438 32326239616338366364613765393665396461633034346436623037666133623565613266376536
64373561666638656439353964626436613164643663336336333337393332346132313830316430 37633966626636323762623965653831646434356232626338636332656661333464623261393933
37336139333539316639646337383737373530643533336133633135333936343165303764623366 38643631313063336161663830343835366336346632636663353561623835303130313163366635
35346432656263396439663635653066303861643938633033393334336336396133626237336432 38633038366234303334336432653534353734346236346638356135633234323364633937666162
31393061653839653964613239353336653666366163616163616230396661313233613934326339 62323132393032303139393139633663303138646133626364626533356465323464613338363533
36333733663032613633613037343337646237666565316239303231313163383264376636336338 37306362346431306164613637303965373132323633613439396166383134323162313164323431
37656532373634383132346163316337336534316465383531316465353861633062336235376631 34383136323636636539353264396439613231653663636638343636626562353861383736306331
33333963313730653164313866323461353334653231333762336232366434333631396135333466 31303564313861306666666332303136343330363063313964353331346433626366303935383066
36643362653137323934323966303734316334306133656139613763336431323236643264663530 35386261633137613331303630613137376434663836383930366263666262656666356139343164
38633730346531396435666530316563386566323737343766653135386537393165356264356534 35616334373931343336636637356339343266613161623561336264623866633834626664663333
62323962306332393063313062313230333334353833393362656166353638323366353934656263 66346231386433383163653437353336613731646435626431366663666465313864613837343365
38633531306438663134616262323332323338653361356336353962376331323537623734316532 36633962326633313735393238373166636435656661373732386637626264323363623632353631
35306434383336633436656437663765626530626535383464356532613536356533646332646135 61613062653164353761313730613936356632306438643431306332326363616464343237386539
39613030323432643161366363313262633261366566613931303663636262353466313933303162 63393866393461663135616235643465373838336566336330346662616130666130396364343634
66373535353934626135326439653630303437666535396266303532626636613563616633623237 64333335656565623662646136313766623031363037366535386461393430646231353536346332
38623938323137393065616136376663363234633735363264356438353534616338393832386337 64313431326164306266666539366264616139303065626536333664623737336134643761333536
66663339313062613766666435626364353935643932633233643630323464336135336339376437 66333331303831316137646137316636633161353766393136363038653765376465623561373761
30323536373639343738613463613466396335626339343135306666663931656564393664643761 30633331306565393365323735613239393338326635626136363933303862343465623166313539
65396336613465313035396134636438636134316236636566333031646666383939383264303437 38393533376630313666643335623165336165633933323036643539623136333938303233353063
38656337326233363866336363363161333236326137393465633935626165663136316661316131 30373032633036663963323831646234653537646230643363663238356631303434373265303735
66393764313130373766386235663134376239323037373365333730643336656532316434663265 39613263383261306332346264646562636432663238663536653934386630393534653832333332
35646664383338326366386536616339393938336539373762643839613763323434363037326330 39316530373861323830303136663462346164386261636666356235633563646264316636363066
66353738643735656165396138343262653563353934376237366564636638623630393238373530 39353464346131643239363736383335643132383862303435366230353631613330333133396336
39336335303564616232393165653639326166333536343966353933633365326539383632646639 34373532613134613231636636373132643462353865363439356336383266616631363139656265
37636238376137646239353731343933346562613630656130653865633834626363656231373162 65333332663139393135643366623266333561313538646639666636303038663962626438663934
65333662653962373139633466383834626566336234316161633038653137393266393564376439 35386463386635613236313666306266353064663666333861643932633862336236333435363539
62343365666637396639326438326631613664666662653935346230316632396361663038663239 66356130646235636637626339356632376638316231656266303637623965636135636333643065
63336335643665353162393330623635663839666162333561376265363330383530653733333239 34396633613536336163306561363332356430616334393133396639663731653863336531313564
37373665646562343562313764386534663239636130646564333433313661363338326439363865 35653333356232396232613864313834656533323034396134313061313731613732386366343134
63353064613161366461383862623363333637633430376361366566383332636634646630363139 35663836313639663038653036633338336666633066396136663964646137636262313762306261
65303139396539323336313666336362626339326666343061303431356361343438313365353431 31313266643663373736393637323539343035636464336563636162343830313831383631313865
30303266363664373233303231346166643839343564306363613236316665306437333636343131 66633337613632343363356331663639616231653033663962376131646533313237333536663438
61303234666664323138376335623639393939633836353664343232316630393263323233343366 39353034383661396237623438623361323232653135653430313738633835613961323735343365
32393736336166323166616139323066613062646139656236343233623630616262343034383631 63376131386339643330303938343765393365656466393965363838343431373763616139393066
61653662383537646364363764353335333630353431633534313731633265626263303863326565 37323037396165636630663733333661626462643437616633383132396532396263373936343538
37616534303164316162353165396236656565303532326235393237366434323739633662663261 30333732303235376435666565313435356137383538313039626462363066373864303633616638
33383635653734303233356438636132343731383439623463633732373833303339353662386435 61363438396634613862303965653937356662373231346539653339333034343961623435393739
63316138353766303962616435383138373739383835353662613666356565366237353932393263 62393430633365323034646233393537306266326234643339623339663764343935386430383461
34326561613234356537343234303166613534356134643766383332646263623430653632386365 65396434363634363439336137613837316532373931626434323366393564646630313330366162
61376165656366366532346465343932313032643163313362373633373066653036346238343938 36653532306439653964373631653061363664663039633134633233346166376261323938363438
66626133306163353966386662363039323166353330633239386264306337386361653430633838 65653637313432663962646536376634333735616531653133653061313534623163323738643466
66366636616661313939316562373330313564383033643035393738353939613066316138643238 65363538643336353338346632323565313234333463396464663732613332646237653332623866
35376336633238636463376234376364386464396461636336313562616361396435323639666433 38666535386337386361333565646633366538333333346336363636333237316462636365383939
34653139613962623235613035363634383735326564333332333632656237333238356566646536 61393631356163313933643230343133643539323834363437306465653438396565393532636464
62346435303932363231306634643037623935653235613330666666656433396463656231643364 62343263353638343565653065373461386461363736313931313163656138656333383733373532
34313235613064646432633935656231643162363530313761303339323830653564303963643532 62356563643663373765336334306466303437343733316431623665626465666633373463366136
39326163396461613161363031616336666564326630323465663831653431353566643661656435 63663738663431306532616431373031666462346131623665376235663634336265303035346432
35636238633663346139343732353233663930353664623636303134303633356232313063346566 37616536663965643236346264656361613732386239653234613932326462613637666231396531
30633762653239356131613539396435396235326431633062616436613266353266636133656531 33316162353038306535386661626464316130343436353230383934346263323738643731373534
63346533383764393733396436306136383830313265656539376136343536353737313365333865 36616537626437393530666162336338383563333636656230646235353266373532653431633539
61653665393037346231656663363432666332383837393264656231386435323333376633333265 34646531303630373034323932303863623536666261326463303031636666363738363664653738
39633563653161663531303330393237316132653339653531316538316534353334383161623833 31376335316239373437363431626630613161663962663934643431303966323237616339393132
63656534643833656366623031326132343732663536393636393537383338613462336638626334 33366661383263396632633838393536383335343761306461313638613236326266396363623634
39326264623032393661303335356235343736613964636261633038333236623232613633343263 37623165613935373730663139303737373864663035323030643264663932353465666138303264
34363865306233346433393664363038303464363834313463636363366333313433333930336634 35326665656538353762643637393664636137313034643637643730313336356464383131343463
65396232663031323965343336366339623239386566323362396564333932336134336231646430 30623232363838373338366163326538316465643930656230323831666462353037633131643961
34376236636536626264663532323530656634333137363431313339643164333236303337373639 39393238386637313632663336373661636230386632346432366633356139376430646338316235
31326131636161386665623838303939616337636666346363616465323263373061633832363839 66393663366232366230326262633638653863396635323539363465383535303433643061633839
39343131386333666231643931353330343136363732396465383336303866623733316235646234 61666431353737393939646638653435343039353463616165653630373130383238333336303533
65613531613333653231666234353532313738363266353336303564613061393639323933646339 36343331313235376263303633326139613734663264353330643666633234323730366134663262
64396334396431313563326136656539613535663334323330326566613264393965623065616634 66643730373761323439636161623135646337383163353361633663376466343839623437613662
32313333663535313531343363656230656334346430643365363838656430616366653766333136 62633837653464316430626266646338333530646234623834643739303738366165646235323232
32346137656136666432383934313865613535303962333464316465313464333233646162376165 63663433336664376665623734613965626464653832643366353432633437643739396265376531
37363265316139643332393761663164346537663064346133323661393030393533396633646134 66636436336135343833356334636531376131316463343162353362356439386139346365333938
39383539333364626263323364353764316539323162346434656234393562326133613632633137 35646431306532663936616264626538666638336130313133323830313966353161666236323735
34323537393838393534643861356365376461636235386638376661623439303832643231616431 36313666346638363865336331313163346635613735363634366265373363303033616663363563
32663264306166366336396130326435346432343732323764346232356439633361663937313233 63353530636566303733343962653138363532326339666230666661393766333863363539373565
62636130663238333433396335356331653166366130626632663162653736656238343134353134 32363733613164373762646633373833373466326633383339626431323462306663353235346664
39373663633437626661393734363264313137663138643665353633366663366562386337396561 62653361323631623063313435663735333638616366613634363166656431303031353833646563
38383464333531306138316266626461326531616364303732343337386464366464633834383439 39313630366430613234643361666365386462346338323330646366366335656136643462663166
66303334333234306166396235666164313833333164356561663063326563303934373933313334 63316135306437376139643337646639303261666434316266323465373832333534373730636465
32326466343761333666373936373332663939643662363665323534333638316637666665326564 34313832633839396634316561346633643734373763666136363039653136353866343132326430
63616134623530333739633265343637626561383831663836626538313539653536373165636133 37303737383030616635343539386337336236376236383961333336363030376333666432613362
62396432376332356133336461633664336264616566376664643861363838313766393833343366 61366365316364613639326236333738313936393535646636383261356635303238303938383130
31386636343835663062396232646663363935396366343361633462643734373437666331383035 33613264396339646136346631643938356138653535636636393837346430396433313661613337
30373963323834343138653063313635643062396130373861643134363062363135623730663335 31323836323162323632343462666433633537623763653566643231656134353235383131663466
34373935623239363866393932363966366233643732646461343964383563393064396331353331 62313730366338646132626338623934646364316637343364653236316566666338363339363731
31646662323364376639316562666465356438386637393130616330663563633839326661643239 38373832316239613961646432373864306235656339373963643938343664346238373761663830
62346633353963336239633738336431326638626232313965336361636430323739653734636436 35663732396430666536313239386465373235393466643930383439613862313337313638386662
39316363316137366233616431346639663335393737653562356532363634346237336266333132 34623033653238333064343861633132313762646534646263663566633264393638343730313939
31343836326538653164653163653738656238326366313532613433333337383263646439326661 63353438333331383861343135326439303965333834646430393531383762623765666237316165
65306135363138646665363463636436303864316432663734393130313932643833356161373139 65346263386636363638663530643633336331623734343231626363353165323562616138373331
65376334376136313237393336636630643933373165336432643233306366633663333930396364 33393464333464666235666539333663663061383335343933373066333966346131653361333935
31393739333730326434313639376365356432376338386430363133666436643837633533616333 62643363653236626330656561383635656431303231353731396466363232613462323938643730
36363464633233343130656163353137663165343064373039363439346631393436393430383361 63356531616364346264653666613564613239646465376238336337383938313035366636396638
61353563346531353561303935363531333235313731303237326334363763646131653961613334 65303566343738343731656338663832396330623066646233376436323334633339343162383966
33306534663238343433363237393234643236393931343230656438353866313638636633396464 35343164626365386162666162346366656539383334346238336131616338393261306537303534
61636561333562383964663261316135373235313234613564333333336438646636303561616634 62343164386462393039393763303566346163646330336264623462353464306232363934343630
63363033613238323463646165373361613834353230653138623132303731393436386139663533 38396565613830346634653934653737636262646438643866376138633065303239633538373432
38393263306139303231626363353931646163323364666161653861643739353262313561393165 32653562343666313361383331636261313331363639633465383161393063336335343938653933
6262 39663865613236393037333031613964633961326332316662666531353839343937393761636164
63643230623332626266323061386665333862303532313530653534343762383066643736336466
36626564633265303330666331353163396130376331333136386638306233343138613664626537
31626236643637363766636635333465396233663239353865306638303331363937663936613563
30306166393033336164333432626239333338653837336234393536343438326436343434373433
62613165633736353838313439366434323664333832366132663766613236643430313764343032
32636630393032343130623438333166306564366562373764656135653730346237343737346638
66333462383865333733316237663833323036643230616531353630306565353236363734643839
38616532626537613336366637386633653065383135643135363332353731353663323365653763
34623433343135636338303662633966343965303461313136373333396538626166633330343635
61633034356135623639623636626463373339633665366362343064313262333061666637323163
38646266613461393533663061376132363234376164393939663763643962633361663665336239
64356633323730303262383766386635613361316139353331346466643661333331336161636137
63616230396361626133373934613164336335376565656230623030393861383234346138303564
39326134383063393765336461306237653839336531363538383666313832353732663536343134
38663535316333656164396562303036653137653366393462663238613632383230626130346266
33653136383237303139316133636230373866316164336663613666373565643733663739323362
37633734356234626432646566333038303465333563323931396630386430633032626536623765
38623961373063323464353465316432393134633432343465633030363539363865383661646138
64373436336132356535646164336631626639383765386166326532643436393638653431623533
65666639346664613266616364663033363730373637343962393533656463653661613737303837
31633262653361373036383636323766653538346532353038393835303931323264376535373062
36663631316234363063393938633034306465373966333062663162653266386237663137653033
66346435313334653962373935316537376132353962653535366564306132636534306164666431
38363634333635373966353636633836373865666237656336626139353130626332383338313238
62616632616337363664616165336563616161386662306535666634613965613039356165396239
32316364623534643662323334326461386335666530373662353931363131306364626638636430
61363533613638616632396463343239633865643438373837333535336662366432373738393361
31663064623431383336343034653734306635323764613031353862646663626339323764313731
33313132613138633364646532633334643663663339633264343365363231633034666265376161
37313261616230616664396138306264653139643438306662313462386439336434373664313532
61396631663663353334653961363663393330353830306130313065366330653761393436326535
38326163366564373236386431366436633138326561303237613965353266346361303063613666
66333637313061383138373661643964326162633137626163383936386132346636386335353134
37393661666465366331383164366666666431386166396337386630643438643564336139656634
61613232653433336330376433653062333062663034306630653538303337343661326634363463
35626130353561353436386261663335623964633564303365316166306237306634633464653565
61353532613631653037643162666366313339633034646435316338353034366630303736386639
66396236646132366265653536386435333066306461303334646535643835393730643338376633
37303161613837383838323961343834343562666532366136303337363832363765326637653934
66396331333938633034303761313566656533346336393531326235393862623462306434396438
33393539306234626163626664393037666533326133336464323831323132633035313561303034
38343332306532303939363038383934376136333661343435313238356536396537333066613665
31623564306435313466343133643730313262623765346464616263313831313164353162636334
64666133333330646666663336613332373664653932613834383739393762326636303466363134
33393164326232353065396361636539626531656337353363666233343139333263383664666531
63663539303165643564356261346433373438396532396439363330623035353636386531356437
30326231333333393533376366333062386464323565306463336164636434303137386661393139
65336135313462363237343861313165393862393233656561663662333162306539316365393063
62623837353437373236653965646131383262653733303038306134626430623961376366383331
62383366336637626531653532633664616537333032386434306330616439373430383664623638
65336666303263613965376664336262353139396535646662316233343764396661616131346237
62666561373866646564383634313639383636653861666164393263306339616163383965643366
34656538383934313136316231643934663963643737353637633032313930623266323466653032
36636339373034336463313862353638313538316465373132616166323863373832626330383937
30313565373731663663336466363239303734336632396636346139636335646162353166643330
30323530613736613736623238393233316635333437333938313932623834643363616239343032
34626636323461303438376635633637636232386333636139626565303635656630343062353639
39396264386638326666303438623334336163343938336566353034656638323633343837376433
65366232303838643832623332663435636530363638636433303336356531633131356266396334
37313864393761623333653638383766366261353334653066316563386437383432343964616535
30306134383166666630636666316630383030346130353865386539303537363539636161643733
62343530396563393464656566363739313666613762613565386331366365393338313237636231
65326630313630363139626161373730373962656134363733633530376261613239323631623433
61396631333736383731363135383664663632643939376334633966373231363430326663323061
31666134396439346166346430363739626665663133323930363663303164616337316233656333
39303833653366353931343936623362346163363766346461653264353733363439643439336435
32656665623532666465663834363564666634653038616361393665303766326436306533613439
64616334393539643662303036626131623137383164303566623430646337353264343030303935
33313164643439643662393632323433366238306261623335656430396437396535653335343965
31616265363437356263616539343232333539393133323334633730366263386135393864393162
33653161623735363335323838353231326166376365656661373965353433336466623134386230
61653733393737643832303562663933353935626437316538393133613063316566646565323034
32643235303038656362663561666539633432383636333463396162353035323531623133656563
65376362643432643933393063393266663563663636616261663638626238653633303630383062
35363134323131323236623563353035306430343534643962333530383836666565646163346139
65633239396437346432346233643463333831663636323833386338386430373430336536313463
31386131653831623832346564343137326163326261306463666563643839653864353764353861
61663539646431633531653465333336663733663264336565386662626335376231643563356337
65653439613164306639353862656264333733386235333732643730623639333937396236616466
36303332626163323932356338313763363563326139633836343637663832633262323065326435
31643134303737316635336534613038313362396335636164656436623764656537303732643730
35613936303762623765656265663362613736636531633335666230643335633161303738663465
61636165376538313839646133653262343530393837313465343330626362626164366165323831
34396134616138363562313934626239353135336335656331383035393763306166666533643465
34383436356162323631373164373565313563656664663762343630383161646633356434643631
64306239373363343161663030343734366239356237393033643931646434393237363239333631
66343963633266613138353433383338343737636238346433366361643436363233373034656231
37633030623232613030633832306337663336616632333730373639323261646432616531366634
63663837623235383438363533313330663135323661626236333761323165663638623930653631
37633933383562396331326264623866353966633932663961643032383363613333323937383538
34653536313866336530343832336564353036376234383538323732306565636234623832376562
64626166366164626436663564353539323765663365303332373138373034333064626164336439
64333639313832656439613532326330336630613131386463663934653539663763646138343435
37636161623363613039666334626164643464313439376330653837363661393431376632326137
34383134386563343337363037616431366239623461356563613039393936653330303830303534
30373463363265653439633736396664623661646237633537373465316362383965333738333831
37373735636362376265376363386566313663343961623064343363366339373236316434653737
35356436316230626631623130386631636664623762613438313632613134396131336534393931
61663639653166376465323162313064623836623366393464386365313739626636303265343463
30336665636238343930303539313930626439326462343865326531616239323036306230363634
33626531393332393332663137303938613163653637376264663863643163346635616564326337
65393335383534323038656464353963316164396263663533643464623561343337623661663635
39663331633239336432366466623666313036393661623966396630623431626465646437363634
31623536346231373534656262336466336465353261663433363134633030393533326539666661
39623931313339646134363337343233346334366262646638613039363730613433313863396564
64643436653637356563393131666465663332326665373937643764376231336331616361323530
35303038323739316265363863316139363033386538346139636262626633303563353737363739
35656463646364323964366533656339643466323965323661306166393933353237643834373264
66623533623837306238393234666331636566616535323231393364643039323064343433393533
66653333633931303761376162666530386161336534343932633933616139306430326430326566
63656534343362323830643831643836303331643364396566626662333439646531616465623963
64616361396635393339643035616536643435353738383437303832626135316631633166613434
64366335373365653763353436333961663265643739646536343564383062396337333362363235
38326236363564313437343832383961303032643831363563646664373338616363623935656538
35376233326434623339376463346338636162306461656434626135623263613632393135306531
38326434636637366633336337663739633232386531336162333939333132336362363430363965
64316337353637656439343233343362666130663165613335633363353339633537323436323334
61343838383365346466336332643065353836366463613736353231343665623430373232333133
34616261653862353062376231356438363038643238333833353464653138303734373735303135
37663338363737666461646161653434636237313332343838346335313363616134653262633362
33653236343930316231636534613666373061373332653434356530343334313434366137636364
64393035383635646363373732636166383931343362356666643836313065346365396162393237
64623064643831363161653034316162363935356563316536393531643164643466323432323139
63643533373431653435383136663665316131343632383366396366376237346265303634326232
37653331666434316263653531323861386339326130396563663864373364663831336465326665
35383534633433356134636632353333346633383335383039626432323932333433303663383364
34633133323166313032393031346536333863306163333462623063663239666563653365643631
33623238343335333932303836396662323832366139616334653639653838303662313162643335
63396164353038636432653332376536323366343765383034366633666639353936316436613130
33633763396366363565636532613738663963363932373061313635376338626665616131346532
65323564333465616664393965633761653962616137663336333430663433646562333364633361
31613736366462333131356131303063386633386664366166653136333266663161663332353737
31623139663130363735336233353135323166613964653839313137316239393431306239343137
65643431353936616433336263353439383763663833333031653033353464653762656433393436
32376666306637636134

View File

@@ -0,0 +1,16 @@
---
# ------------------------------------------------------------------------------
# FILE: ansible/host_vars/astro_orbiter/vars.yml
# HOST: astro-orbiter (10.1.71.130)
# ROLE: Ollama inference host with AMD RX 5700 GPU passthrough
# ------------------------------------------------------------------------------
ansible_host: 10.1.71.131
ansible_user: wed
ansible_become: true
# LVM root expansion — xlarge template uses sda3 partition, standard VG/LV names
common_expand_root_lvm: true
common_root_pv: /dev/sda3
common_root_vg: ubuntu-vg
common_root_lv: ubuntu-lv

View File

@@ -0,0 +1,16 @@
---
# Host-specific vars for main-street-station (JMRI headless server)
# LCRR - Lake Country Railroad, Milwaukee Road Oct 1956, HO scale
# JMRI profile ID — find with: ls ~/.jmri/profiles/ on the old box
# Format: <name>.<8-char-hex> e.g. LCRR.3d3f1dfc
# TODO: fill in after restoring config from GitHub backup
jmri_profile_id: ""
# USB serial device for NCE command station
# Verify after install: ls -la /dev/ttyUSB* /dev/ttyACM*
jmri_serial_device: /dev/ttyUSB0
# Path to JMRI config backup for restore task (leave empty to skip)
# Point at a local checkout of the LCRR GitHub repo
jmri_config_src: ""

View File

@@ -0,0 +1,7 @@
---
# main-street-station — JMRI / LCRR server
jmri_profile_id: "My_JMRI_Railroad.3f178885"
jmri_lcrr_repo: "ssh://git@gitea.mk-labs.cloud:2221/rblundon/LCRR.git"
jmri_leviton_email: "{{ leviton_email }}"
jmri_leviton_password: "{{ leviton_password }}"
jmri_ssh_authorized_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINnSM/9fO8rz/amqkyoGUzUKNNzzmtSXPwOCr1O9zKNO ansible"

View File

@@ -50,8 +50,9 @@ nextcloud_server:
semaphore_server: semaphore_server:
hosts: hosts:
imagineering: figment:
ansible_host: 10.1.71.37 ansible_host: 10.1.71.37
ansible_user: wed
ansible_become: true ansible_become: true
n8n_server: n8n_server:
@@ -65,6 +66,27 @@ ollama_server:
ansible_user: wed ansible_user: wed
ansible_become: true ansible_become: true
hermes_server:
hosts:
carousel-of-progress:
ansible_host: 10.1.71.131
ansible_user: wed
ansible_become: true
honcho_server:
hosts:
lincoln:
ansible_host: 10.1.71.132
ansible_user: wed
ansible_become: true
jmri_server:
hosts:
main-street-station:
ansible_host: 192.168.10.40
ansible_user: wed
ansible_become: true
papermc_server: papermc_server:
# ansible-galaxy role install engonzal.papermc # ansible-galaxy role install engonzal.papermc
hosts: hosts:
@@ -73,6 +95,10 @@ papermc_server:
dev_servers: dev_servers:
hosts: hosts:
scrim: scrim:
backstage:
ansible_host: 10.1.71.133
ansible_user: wed
ansible_become: true
# dhcp_server: # dhcp_server:
# hosts: # hosts:

View File

@@ -6,7 +6,8 @@
# #
# 1. Syncs boilerplates/traefik/dynamic/ to lightning-lane # 1. Syncs boilerplates/traefik/dynamic/ to lightning-lane
# 2. Scans the directory for service configs # 2. Scans the directory for service configs
# 3. Creates CNAME records for each service -> lightning-lane # 3. Extracts all hostnames from Host() rules (supports multi-host)
# 4. Creates CNAME records for each hostname -> lightning-lane
# #
# PREREQUISITES: # PREREQUISITES:
# - Service dynamic config YAML committed to boilerplates/traefik/dynamic/ # - Service dynamic config YAML committed to boilerplates/traefik/dynamic/
@@ -14,15 +15,6 @@
# #
# USAGE: # USAGE:
# ansible-playbook -i inventory.yml playbooks/add_service_route.yml # ansible-playbook -i inventory.yml playbooks/add_service_route.yml
#
# ADDING A NEW SERVICE:
# 1. Create boilerplates/traefik/dynamic/<service>.yml
# 2. Commit and push
# 3. Run this playbook
#
# EXCLUDING FILES:
# Files that are not service routes (e.g., default.yml for middleware
# definitions) should be added to the exclude_configs list below.
# ------------------------------------------------------------------------------ # ------------------------------------------------------------------------------
- name: Sync Traefik routes and ensure DNS records - name: Sync Traefik routes and ensure DNS records
@@ -33,13 +25,12 @@
vars: vars:
base_domain: "local.mk-labs.cloud" base_domain: "local.mk-labs.cloud"
dns_server: "monorail" dns_server: "monorail"
traefik_host: "10.1.71.35" traefik_host: "lightning-lane.local.mk-labs.cloud"
traefik_user: "wed" traefik_user: "wed"
traefik_dynamic_path: "/opt/docker/traefik/dynamic/" traefik_dynamic_path: "/opt/docker/traefik/dynamic/"
dynamic_config_dir: "{{ playbook_dir }}/../../boilerplates/traefik/dynamic" dynamic_config_dir: "{{ playbook_dir }}/../../boilerplates/traefik/dynamic"
# Files in the dynamic directory that are NOT service routes # Files in the dynamic directory that are NOT service routes
# (middleware definitions, TLS options, etc.)
exclude_configs: exclude_configs:
- default.yml - default.yml
@@ -53,38 +44,52 @@
register: sync_result register: sync_result
changed_when: "'sending incremental file list' in sync_result.stdout" changed_when: "'sending incremental file list' in sync_result.stdout"
# ── Step 2: Discover service configs ── # ── Step 2: Discover hostnames from Traefik router rules ──
- name: Find all dynamic config files - name: Find all dynamic config files
ansible.builtin.find: ansible.builtin.find:
paths: "{{ dynamic_config_dir }}" paths: "{{ dynamic_config_dir }}"
patterns: "*.yml" patterns: "*.yml"
register: config_files register: config_files
- name: Build service list from config filenames - name: Read config files
ansible.builtin.set_fact: ansible.builtin.slurp:
service_names: >- src: "{{ item.path }}"
{{ config_files.files register: slurped_configs
| map(attribute='path') loop: "{{ config_files.files }}"
| map('basename') when: item.path | basename not in exclude_configs
| reject('in', exclude_configs)
| map('regex_replace', '\.yml$', '')
| list }}
- name: Display services to route - name: Extract all hostnames from Host() rules
ansible.builtin.set_fact:
hostnames: >-
{% set hosts = [] -%}
{% for result in slurped_configs.results if result.content is defined -%}
{% set content = result.content | b64decode -%}
{% for match in content | regex_findall('Host\(`([^`]+)`\)') -%}
{% for h in match.split(' || ') -%}
{% set h = h | regex_replace('`', '') | trim -%}
{% if h.endswith('.local.mk-labs.cloud') and h not in hosts -%}
{% set _ = hosts.append(h) -%}
{% endif -%}
{% endfor -%}
{% endfor -%}
{% endfor -%}
{{ hosts | unique | list }}
- name: Display hostnames to create
ansible.builtin.debug: ansible.builtin.debug:
msg: "Services found: {{ service_names }}" msg: "Hostnames found: {{ hostnames }}"
# ── Step 3: Create DNS CNAME records ── # ── Step 3: Create DNS CNAME records ──
- name: Create DNS CNAME record for each service - name: Create DNS CNAME record for each hostname
effectivelywild.technitium_dns.technitium_dns_add_record: effectivelywild.technitium_dns.technitium_dns_add_record:
api_url: "http://{{ dns_server }}.{{ base_domain }}" api_url: "http://{{ dns_server }}.{{ base_domain }}"
api_token: "{{ vault_technitium_api_key }}" api_token: "{{ vault_technitium_api_key }}"
zone: "{{ base_domain }}" zone: "{{ base_domain }}"
name: "{{ item }}.{{ base_domain }}" name: "{{ item }}"
type: "CNAME" type: "CNAME"
cname: "lightning-lane.{{ base_domain }}" cname: "lightning-lane.{{ base_domain }}"
ttl: 360 ttl: 360
validate_certs: false validate_certs: false
loop: "{{ service_names }}" loop: "{{ hostnames }}"
loop_control: loop_control:
label: "{{ item }}.{{ base_domain }}" label: "{{ item }}"

View File

@@ -1,6 +1,6 @@
--- ---
- name: Apply common role - name: Apply day0 baseline
hosts: "{{ target | default('all') }}" hosts: "{{ target | default('all') }}"
become: true become: true
roles: roles:
- common - day0-baseline

View File

@@ -0,0 +1,32 @@
---
# ============================================================================
# day0_expand_root_lv.yml
# ----------------------------------------------------------------------------
# Reclaims unallocated PE on the root volume group, extending the root LV
# to fill the VG and resizing the underlying filesystem (ext4 or xfs).
#
# Belongs to the day0 host-provisioning lifecycle. The Ubuntu Server
# autoinstall template ships with the root LV at ~half the disk size by
# default; this playbook is the canonical one-shot fix-up for that.
#
# Idempotent and safe to re-run. Hosts without LVM are no-op'd cleanly.
#
# Opt-out: set `expand_root_lv_skip: true` in host_vars/<host>.yml for
# hosts where free PE should NOT be claimed by root (e.g. hosts with a
# planned second LV in the same VG for application data).
#
# Usage:
# ansible-playbook playbooks/day0_expand_root_lv.yml
# ansible-playbook playbooks/day0_expand_root_lv.yml -e target=lincoln
# ansible-playbook playbooks/day0_expand_root_lv.yml -e target=honcho_server
# ============================================================================
- name: Expand root logical volume to fill VG
hosts: "{{ target | default('all') }}"
become: true
gather_facts: true
tasks:
- name: Apply expand_root_lv role unless host opts out
ansible.builtin.include_role:
name: expand_root_lv
when: not (expand_root_lv_skip | default(false) | bool)

View File

@@ -0,0 +1,23 @@
---
# ============================================================================
# day0_linux_baseline.yml
# ----------------------------------------------------------------------------
# Applies the mk-labs Linux baseline (linux-baseline role) to one or more
# hosts. Idempotent and safe to re-run.
#
# Usage:
# ansible-playbook playbooks/day0_linux_baseline.yml
# ansible-playbook playbooks/day0_linux_baseline.yml -e target=figment
# ansible-playbook playbooks/day0_linux_baseline.yml -e target=semaphore_server
#
# To trigger an opt-in full system upgrade:
# ansible-playbook playbooks/day0_linux_baseline.yml \
# -e target=figment -e 'baseline_features={"full_upgrade": true}'
# ============================================================================
- name: Apply mk-labs Linux baseline
hosts: "{{ target | default('all') }}"
become: true
gather_facts: true
roles:
- linux-baseline

View File

@@ -0,0 +1,30 @@
---
# ============================================================================
# day0_provision.yml
# ----------------------------------------------------------------------------
# Umbrella day0 playbook. Runs the full host-provisioning lifecycle in
# the correct order against newly-built VMs, so the operator runs ONE
# command per new host rather than chaining day0 steps manually.
#
# Order matters:
# 1. linux-baseline — timezone, NTP, packages, SSH hardening, jarvis user
# 2. expand_root_lv — reclaim PE left unallocated by the Ubuntu
# autoinstall template default
#
# Idempotent: every step is safe to re-run. Suitable to apply periodically
# from Semaphore as a baseline-drift check.
#
# Usage:
# ansible-playbook playbooks/day0_provision.yml -e target=lincoln
# ansible-playbook playbooks/day0_provision.yml -e target=honcho_server
#
# For finer control over a single phase, the constituent playbooks are:
# playbooks/day0_linux_baseline.yml
# playbooks/day0_expand_root_lv.yml
# ============================================================================
- name: Import day0 linux baseline
ansible.builtin.import_playbook: day0_linux_baseline.yml
- name: Import day0 expand root LV
ansible.builtin.import_playbook: day0_expand_root_lv.yml

View File

@@ -0,0 +1,79 @@
---
# =============================================================================
# day1_deploy_hermes.yml
# Deploy Hermes Agent (Nous Research) on carousel-of-progress (10.1.71.131)
#
# FIRST-RUN WORKFLOW:
# 1. Run this playbook:
# ansible-playbook playbooks/day1_deploy_hermes.yml
#
# 2. SSH to the host and run the setup wizard as the hermes user:
# ssh wed@carousel-of-progress.local.mk-labs.cloud
# sudo -u hermes hermes setup
#
# 3. Once configured, start and verify the service:
# sudo systemctl start hermes
# sudo systemctl status hermes
# sudo journalctl -u hermes -f
#
# VARIABLES:
# hermes_skip_browser: true — set to skip Playwright/Chromium install
# (saves ~300MB if browser automation not needed)
# =============================================================================
- name: Deploy Hermes Agent on carousel-of-progress
hosts: carousel-of-progress
gather_facts: true
pre_tasks:
- name: Verify target is carousel-of-progress
ansible.builtin.assert:
that:
- inventory_hostname == "carousel-of-progress"
fail_msg: >
This playbook is scoped to carousel-of-progress only.
Got: {{ inventory_hostname }}
- name: Confirm OS is Ubuntu
ansible.builtin.assert:
that:
- ansible_distribution == "Ubuntu"
fail_msg: >
This playbook requires Ubuntu. Found: {{ ansible_distribution }}.
(If running Fedora, swap apt tasks for dnf and adjust Playwright deps.)
roles:
- role: hermes
vars:
hermes_skip_browser: false # set true to skip Chromium install
post_tasks:
- name: Verify hermes binary is accessible system-wide
ansible.builtin.command: hermes --version
register: hermes_version_check
changed_when: false
failed_when: hermes_version_check.rc != 0
- name: Print hermes version
ansible.builtin.debug:
msg: "{{ hermes_version_check.stdout }}"
- name: Print post-install instructions
ansible.builtin.debug:
msg:
- "============================================================"
- "Hermes installed on carousel-of-progress (10.1.71.131)"
- "============================================================"
- "Next steps:"
- " 1. SSH to the host:"
- " ssh wed@carousel-of-progress.local.mk-labs.cloud"
- " 2. Run the setup wizard as the hermes user:"
- " sudo -u hermes hermes setup"
- " 3. After config, start the service:"
- " sudo systemctl start hermes"
- " 4. Verify:"
- " sudo systemctl status hermes"
- " sudo journalctl -u hermes -f"
- "============================================================"
- "Service is ENABLED but NOT STARTED — config required first."
- "============================================================"

View File

@@ -0,0 +1,18 @@
---
# ============================================================================
# day1_deploy_honcho.yml
# ----------------------------------------------------------------------------
# Deploys Honcho + pgvector PostgreSQL on the `lincoln` host. Assumes day0
# host provisioning (linux-baseline + expand_root_lv) is already complete.
#
# Run via:
# ansible-playbook -i inventory.yml playbooks/day0_provision.yml -e target=lincoln
# ansible-playbook -i inventory.yml playbooks/day1_deploy_honcho.yml
# ============================================================================
- name: Deploy Honcho on lincoln
hosts: honcho_server
become: true
gather_facts: true
roles:
- honcho

View File

@@ -0,0 +1,24 @@
---
# ============================================================================
# day1_deploy_jmri.yml
# ----------------------------------------------------------------------------
# Deploys JMRI JmriFaceless headless server on main-street-station.
# Applies linux-baseline first, then the jmri role.
#
# Usage:
# ansible-playbook playbooks/day1_deploy_jmri.yml
# ansible-playbook playbooks/day1_deploy_jmri.yml -e target=main-street-station
#
# Prerequisites:
# 1. Host is in inventory under jmri_server group
# 2. jmri_profile_id is set in host_vars/main-street-station.yml
# 3. SSH access as 'wed' with sudo
# ============================================================================
- name: Deploy JMRI headless server
hosts: "{{ target | default('jmri_server') }}"
become: true
gather_facts: true
roles:
- linux-baseline
- jmri

View File

@@ -1,22 +1,17 @@
--- ---
# ------------------------------------------------------------------------------ # ============================================================================
# FILE: playbooks/day1_deploy_semaphore.yml # day1_deploy_semaphore.yml
# DESCRIPTION: Deploys Semaphore on imagineering # ----------------------------------------------------------------------------
# Runs: common → docker-host → semaphore # Deploys SemaphoreUI + PostgreSQL on the imagineering host (figment).
# Run AFTER day0_linux_baseline.yml has been applied to the target.
# #
# USAGE: # Usage:
# ansible-playbook -i inventory.yml playbooks/day1_deploy_semaphore.yml # ansible-playbook -i inventory.yml playbooks/day1_deploy_semaphore.yml
# # ============================================================================
# SECRETS REQUIRED IN VAULT (group_vars/all/vault):
# vault_semaphore_database_password
# vault_semaphore_admin_password
# vault_semaphore_access_key_encryption
# ------------------------------------------------------------------------------
- name: Deploy Semaphore - name: Deploy SemaphoreUI on imagineering
hosts: semaphore_server hosts: semaphore_server
become: true become: true
gather_facts: true
roles: roles:
- docker-host
- semaphore - semaphore

View File

@@ -1,77 +0,0 @@
---
# ------------------------------------------------------------------------------
# FILE: roles/common/tasks/main.yml
# DESCRIPTION: Baseline configuration applied to all managed Ubuntu hosts.
# Handles hostname, timezone, core packages, NTP, ansible user,
# and optional LVM root volume expansion.
# ------------------------------------------------------------------------------
- name: Set hostname
hostname:
name: "{{ inventory_hostname | replace('_', '-') }}"
- name: Set timezone
timezone:
name: "{{ common_timezone }}"
- name: Update apt cache
apt:
update_cache: yes
cache_valid_time: 3600
- name: Install base utility packages
apt:
name: "{{ common_packages }}"
state: present
- name: Install chrony
apt:
name: chrony
state: present
- name: Configure chrony to use sundial
template:
src: chrony.conf.j2
dest: /etc/chrony/chrony.conf
mode: '0644'
notify: restart chrony
- name: Ensure chrony is enabled and running
systemd:
name: chrony
state: started
enabled: yes
- name: Ensure ansible user has passwordless sudo
lineinfile:
path: /etc/sudoers.d/{{ ansible_user }}
line: "{{ ansible_user }} ALL=(ALL) NOPASSWD:ALL"
create: yes
mode: '0440'
validate: 'visudo -cf %s'
# ------------------------------------------------------------------------------
# LVM root volume expansion
# Extends the root PV to the full disk size and grows the LV + filesystem.
# This codifies what was previously done manually after provisioning.
# Runs only when common_expand_root_lvm is true (default: true).
# Safe to re-run — pvresize and lvextend are idempotent when already at max.
# ------------------------------------------------------------------------------
- name: Expand root PV to full disk size
command: pvresize {{ common_root_pv }}
register: pvresize_result
changed_when: "'changed' in pvresize_result.stdout or pvresize_result.rc == 0"
when: common_expand_root_lvm | bool
- name: Extend root LV to 100% of free VG space
lvol:
vg: "{{ common_root_vg }}"
lv: "{{ common_root_lv }}"
size: +100%FREE
resizefs: yes
when:
- common_expand_root_lvm | bool
ignore_errors: yes
# ignore_errors because lvextend returns non-zero when already at max size.
# resizefs: yes handles the resize2fs call inline — no separate task needed.

View File

@@ -1,5 +0,0 @@
---
# file: roles/semaphore/defaults/main.yml
semaphore_compose_dir: /opt/docker/semaphore
semaphore_ssh_key_file: "~/.ssh/ansible"

View File

@@ -1,7 +0,0 @@
---
# file: roles/semaphore/handlers/main.yml
- name: Restart Semaphore
community.docker.docker_compose_v2:
project_src: "{{ semaphore_compose_dir }}"
state: restarted

View File

@@ -1,50 +0,0 @@
---
# file: roles/semaphore/tasks/main.yml
- name: Create Semaphore directory
ansible.builtin.file:
path: "{{ semaphore_compose_dir }}"
state: directory
owner: wed
group: docker
mode: '0755'
- name: Copy Compose file from boilerplate
ansible.builtin.copy:
src: "{{ playbook_dir }}/../../boilerplates/semaphore/compose.yaml"
dest: "{{ semaphore_compose_dir }}/compose.yaml"
owner: wed
group: docker
mode: '0644'
- name: Deploy Semaphore .env file
ansible.builtin.template:
src: env.j2
dest: "{{ semaphore_compose_dir }}/.env"
owner: wed
group: docker
mode: '0600'
- name: Copy SSH key for Ansible authentication
ansible.builtin.copy:
src: "{{ semaphore_ssh_key_file }}"
dest: "{{ semaphore_compose_dir }}/ansible_key"
owner: "1001"
group: "1001"
mode: '0600'
- name: Start Semaphore containers
community.docker.docker_compose_v2:
project_src: "{{ semaphore_compose_dir }}"
state: present
register: semaphore_compose
- name: Wait for Semaphore to be ready
ansible.builtin.uri:
url: "http://localhost:3000/api/ping"
method: GET
status_code: 200
register: semaphore_health
retries: 48
delay: 5
until: semaphore_health.status == 200

View File

@@ -1,3 +0,0 @@
DATABASE_PASSWORD={{ vault_semaphore_database_password }}
SEMAPHORE_ADMIN_PASSWORD={{ vault_semaphore_admin_password }}
SEMAPHORE_ACCESS_KEY_ENCRYPTION={{ vault_semaphore_access_key_encryption }}

View File

@@ -1,21 +1,6 @@
# requirements.yml
--- ---
collections: collections:
- name: community.general - name: containers.podman
version: 11.4.1 version: ">=1.10.0"
- name: effectivelywild.technitium_dns
# - name: nccurry.openshift version: ">=1.1.0"
# version: 1.4.0
- name: somaz94.ansible_k8s_iac_tool
version: 1.1.6
# - name: prometheus.prometheus
# version: 0.27.0
- name: kubernetes.core
- name: community.proxmox
version: 1.4.0
- name: effectivelywild.technitium_dns

View File

@@ -8,6 +8,7 @@ common_timezone: America/Chicago
common_ntp_server: "sundial.local.mk-labs.cloud" common_ntp_server: "sundial.local.mk-labs.cloud"
common_packages: common_packages:
- acl
- curl - curl
- wget - wget
- vim - vim

View File

@@ -0,0 +1,21 @@
- name: Create cast group
ansible.builtin.group:
name: cast
state: present
system: true
- name: Create cast user
ansible.builtin.user:
name: cast
group: cast
shell: /bin/bash
create_home: true
state: present
- name: Ensure cast user has passwordless sudo
ansible.builtin.lineinfile:
path: /etc/sudoers.d/cast
line: "cast ALL=(ALL) NOPASSWD:ALL"
create: yes
mode: '0440'
validate: 'visudo -cf %s'

View File

@@ -0,0 +1,12 @@
---
# Timezone
timezone: "UTC"
# Packages to ensure are present
baseline_packages:
- chrony
- vim
- htop
- curl
- wget
- rsync

View File

@@ -0,0 +1,6 @@
---
- name: Restart sshd
ansible.builtin.systemd:
name: sshd
state: restarted
become: true

View File

@@ -0,0 +1,32 @@
---
- name: Set timezone
community.general.timezone:
name: "{{ timezone }}"
become: true
- name: Enable and start chronyd
ansible.builtin.systemd:
name: chronyd
state: started
enabled: true
become: true
- name: Update all packages
ansible.builtin.package:
name: "*"
state: latest
become: true
- name: Install baseline packages
ansible.builtin.package:
name: "{{ baseline_packages }}"
state: present
become: true
- name: Disable root SSH login
ansible.builtin.lineinfile:
path: /etc/ssh/sshd_config
regexp: '^PermitRootLogin'
line: "PermitRootLogin no"
become: true
notify: Restart sshd

View File

@@ -0,0 +1,70 @@
# expand-root-lv role
Idempotent role that extends the root LVM logical volume to fill its
volume group and grows the underlying filesystem (ext4 or xfs).
## Where this runs in the lifecycle
Part of the **day0** host-provisioning lifecycle. The canonical entry
points are:
```
playbooks/day0_expand_root_lv.yml # standalone
playbooks/day0_provision.yml # umbrella (baseline + expand_root_lv)
```
Day1 application-deploy playbooks should NOT include this role —
day0 is assumed complete before day1 begins.
## Why this role exists
The Ubuntu Server autoinstall template (used by the mk-labs `wed`-baked
VM templates) provisions the root LV at roughly half the available disk
size — a longstanding installer default that surprises every operator
who hasn't been bitten by it before. ~90% of mk-labs VMs need this
fix-up before they're fully useful.
## Idempotency
- If `vg_free_count == 0`, the `lvextend` step is skipped and the
filesystem-grow step is also skipped (nothing to resize against).
- If the target volume group doesn't exist on the host (e.g. a non-LVM
layout), the role exits cleanly via `meta: end_play`.
- Safe to leave in a recurring playbook so future disk expansions
(Proxmox-side disk grow → reboot → run role) are picked up
automatically.
## Opt-out for multi-LV hosts
If a host will have a **second logical volume in the same VG** (e.g. a
dedicated `/var/lib/postgresql` LV for a database server), this role's
"grow root to fill VG" behavior is wrong — it will consume the free PE
that was being reserved for the second LV.
Set in `host_vars/<host>.yml`:
```yaml
expand_root_lv_skip: true
```
The day0 playbook checks this flag and skips the role cleanly.
## Defaults
| Variable | Default | Purpose |
|-------------------------------|---------------|-----------------------------------------------------|
| `expand_root_lv_vg_name` | `ubuntu-vg` | LVM volume group name (Ubuntu installer default). |
| `expand_root_lv_lv_name` | `ubuntu-lv` | LVM logical volume name (Ubuntu installer default). |
| `expand_root_lv_mountpoint` | `/` | Mountpoint of the filesystem to grow. |
Override the VG/LV names in `host_vars/<host>.yml` for hosts that use a
different LVM layout.
## Limitations
- Does not extend the underlying partition. If the operator grows the
Proxmox disk and the partition itself needs to grow before lvextend
can claim the new space, run `growpart /dev/sda 3` (or equivalent)
first. A future enhancement could automate this via `cloud-utils`'
`growpart` package, but it's out of scope for the initial template
fix-up case where the partition already covers the whole disk.

View File

@@ -0,0 +1,26 @@
---
# ============================================================================
# expand-root-lv role defaults
# ============================================================================
# Extends the root LVM logical volume to fill its volume group, then grows
# the underlying filesystem to match. Idempotent: when there's no free PE
# in the VG (i.e. the LV already fills the VG), the lvextend step is a
# no-op and resize2fs/xfs_growfs simply confirms the filesystem is at
# capacity.
#
# Designed for Ubuntu cloud-image-style installations where the autoinstall
# template provisions an LV at half the disk size (the Ubuntu Server
# installer's longstanding default). Run once after VM provisioning to
# reclaim the unallocated PE; safe to leave in a day1 playbook so future
# disk expansions are picked up automatically.
# ============================================================================
# The LV and VG names follow the Ubuntu Server installer's convention.
# Override per-host if your template differs.
expand_root_lv_vg_name: ubuntu-vg
expand_root_lv_lv_name: ubuntu-lv
# Mount point we expect to be backed by the target LV. Used purely for
# the resize2fs / xfs_growfs decision — the role inspects this path's
# filesystem type and dispatches to the correct grow command.
expand_root_lv_mountpoint: /

View File

@@ -0,0 +1,23 @@
---
galaxy_info:
role_name: expand_root_lv
author: JARVIS
description: >-
Idempotent role that extends the root LVM logical volume to fill its
volume group and grows the underlying filesystem (ext4 or xfs). Fixes
the half-disk LV that the Ubuntu Server autoinstall template ships
with by default.
license: MIT
min_ansible_version: "2.14"
platforms:
- name: Ubuntu
versions:
- noble
- jammy
galaxy_tags:
- lvm
- cloud-init
- homelab
- storage
dependencies: []

View File

@@ -0,0 +1,70 @@
---
# ============================================================================
# expand-root-lv / main
# ----------------------------------------------------------------------------
# 1. Confirm the target VG exists (skip role cleanly on non-LVM hosts).
# 2. Read free physical-extent count for the VG.
# 3. Extend the LV to +100%FREE only when free_pe > 0.
# 4. Grow the filesystem on the mountpoint (ext4 -> resize2fs, xfs -> xfs_growfs).
# Each step is idempotent and skips when there's nothing to do.
# ============================================================================
- name: Gather LVM facts
ansible.builtin.command:
cmd: "vgs --noheadings --nosuffix --units b -o vg_name,vg_free_count {{ expand_root_lv_vg_name }}"
register: vg_info
changed_when: false
failed_when: false
- name: Skip role when target VG is absent
ansible.builtin.meta: end_play
when: vg_info.rc != 0
- name: Parse free PE count
ansible.builtin.set_fact:
expand_root_lv_free_pe: "{{ (vg_info.stdout.split() | last | int) if vg_info.stdout | length > 0 else 0 }}"
- name: Extend LV to fill VG (only if free PE > 0)
ansible.builtin.command:
cmd: "lvextend -l +100%FREE /dev/{{ expand_root_lv_vg_name }}/{{ expand_root_lv_lv_name }}"
register: lvextend_result
when: expand_root_lv_free_pe | int > 0
changed_when: lvextend_result.rc == 0
- name: Detect filesystem type at mountpoint
ansible.builtin.command:
cmd: "findmnt {{ expand_root_lv_mountpoint }} -no FSTYPE"
register: fstype_result
changed_when: false
- name: Set filesystem type fact
ansible.builtin.set_fact:
expand_root_lv_fstype: "{{ fstype_result.stdout | trim }}"
- name: Grow ext4 filesystem
ansible.builtin.command:
cmd: "resize2fs /dev/{{ expand_root_lv_vg_name }}/{{ expand_root_lv_lv_name }}"
register: resize_result
when:
- expand_root_lv_fstype == "ext4"
- lvextend_result.changed | default(false)
changed_when: resize_result.rc == 0
- name: Grow xfs filesystem
ansible.builtin.command:
cmd: "xfs_growfs {{ expand_root_lv_mountpoint }}"
register: xfs_result
when:
- expand_root_lv_fstype == "xfs"
- lvextend_result.changed | default(false)
changed_when: xfs_result.rc == 0
- name: Report current root size
ansible.builtin.command:
cmd: "df -h {{ expand_root_lv_mountpoint }}"
register: df_result
changed_when: false
- name: Show post-resize disk usage
ansible.builtin.debug:
msg: "{{ df_result.stdout_lines }}"

View File

@@ -0,0 +1,26 @@
---
# Hermes service user
hermes_user: hermes
hermes_group: hermes
hermes_home: /home/hermes
# Install flags
# Set to true if you don't need browser automation (skips Playwright/Chromium)
hermes_skip_browser: false
# systemd service name (gateway)
hermes_service_name: hermes
# Path where hermes binary will be accessible system-wide
hermes_bin_symlink: /usr/local/bin/hermes
# -----------------------------------------------------------------------------
# Hermes Web UI (dashboard) — fronted by Traefik on lightning-lane via the
# 'jarvis' service. Binds to 0.0.0.0 so Traefik can reach it from upstream;
# --insecure is acceptable because TLS + auth are terminated at Traefik.
# -----------------------------------------------------------------------------
hermes_dashboard_enabled: true
hermes_dashboard_service_name: hermes-dashboard
hermes_dashboard_host: 0.0.0.0
hermes_dashboard_port: 9119
hermes_dashboard_insecure: true

View File

@@ -0,0 +1,9 @@
---
- name: reload systemd
ansible.builtin.systemd:
daemon_reload: true
- name: restart hermes
ansible.builtin.systemd:
name: "{{ hermes_service_name }}"
state: restarted

View File

@@ -0,0 +1,199 @@
---
# ---------------------------------------------------------------------------
# 1. System prerequisites (run as root via become)
# ---------------------------------------------------------------------------
- name: Install system packages required by Hermes installer
ansible.builtin.apt:
name:
- git
- curl
- ffmpeg
- ripgrep
state: present
update_cache: true
become: true
- name: Install Node.js 22 (required for browser automation and WhatsApp bridge)
block:
- name: Download NodeSource setup script
ansible.builtin.get_url:
url: https://deb.nodesource.com/setup_22.x
dest: /tmp/nodesource_setup.sh
mode: "0755"
- name: Run NodeSource setup script
ansible.builtin.command: bash /tmp/nodesource_setup.sh
args:
creates: /etc/apt/sources.list.d/nodesource.list
- name: Install nodejs
ansible.builtin.apt:
name: nodejs
state: present
update_cache: true
become: true
# ---------------------------------------------------------------------------
# 2. Install Playwright system deps for Chromium (root-only step)
# Per docs: this is the one thing that genuinely needs root.
# Skipped entirely if hermes_skip_browser is true.
# ---------------------------------------------------------------------------
- name: Install Playwright Chromium system dependencies
ansible.builtin.command: npx --yes playwright install-deps chromium
become: true
when: not hermes_skip_browser
changed_when: true
environment:
DEBIAN_FRONTEND: noninteractive
# ---------------------------------------------------------------------------
# 3. Create dedicated hermes service user
# ---------------------------------------------------------------------------
- name: Create hermes group
ansible.builtin.group:
name: "{{ hermes_group }}"
state: present
system: true
become: true
- name: Create hermes user
ansible.builtin.user:
name: "{{ hermes_user }}"
group: "{{ hermes_group }}"
home: "{{ hermes_home }}"
shell: /bin/bash
system: true
create_home: true
comment: "Hermes Agent service account"
become: true
- name: Grant hermes user passwordless sudo
ansible.builtin.copy:
content: "hermes ALL=(ALL) NOPASSWD:ALL\n"
dest: /etc/sudoers.d/hermes
owner: root
group: root
mode: "0440"
validate: /usr/sbin/visudo -cf %s
become: true
- name: Ensure hermes home directory has correct permissions
ansible.builtin.file:
path: "{{ hermes_home }}"
owner: "{{ hermes_user }}"
group: "{{ hermes_group }}"
mode: "0750"
state: directory
become: true
# ---------------------------------------------------------------------------
# 4. Run the Hermes installer as the hermes user
# ---------------------------------------------------------------------------
- name: Check if hermes is already installed
ansible.builtin.stat:
path: "{{ hermes_home }}/.hermes/hermes-agent/venv/bin/hermes"
register: hermes_binary
- name: Run Hermes installer as hermes user
ansible.builtin.shell: |
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh \
| bash -s -- --skip-setup{% if hermes_skip_browser %} --skip-browser{% endif %}
args:
executable: /bin/bash
become: true
become_user: "{{ hermes_user }}"
environment:
HOME: "{{ hermes_home }}"
PATH: "{{ hermes_home }}/.local/bin:/usr/local/bin:/usr/bin:/bin"
when: not hermes_binary.stat.exists
changed_when: true
# ---------------------------------------------------------------------------
# 5. Symlink hermes binary into system PATH
# Per docs: service accounts often lack ~/.local/bin in PATH; symlink
# into /usr/local/bin so hermes is always accessible.
# ---------------------------------------------------------------------------
- name: Symlink hermes binary to system PATH
ansible.builtin.file:
src: "{{ hermes_home }}/.hermes/hermes-agent/venv/bin/hermes"
dest: "{{ hermes_bin_symlink }}"
state: link
force: true
become: true
# ---------------------------------------------------------------------------
# 6. Install systemd service unit
# NOTE: The service starts in 'gateway' mode for persistent operation.
# You must run 'sudo -u hermes hermes setup' interactively on first boot
# to configure your LLM provider and any messaging gateways before
# enabling the service.
# ---------------------------------------------------------------------------
- name: Deploy hermes systemd service unit
ansible.builtin.template:
src: hermes.service.j2
dest: /etc/systemd/system/{{ hermes_service_name }}.service
owner: root
group: root
mode: "0644"
become: true
notify:
- reload systemd
- name: Flush handlers to reload systemd now
ansible.builtin.meta: flush_handlers
- name: Enable hermes service (but do not start — config required first)
ansible.builtin.systemd:
name: "{{ hermes_service_name }}"
enabled: true
state: stopped
become: true
# ---------------------------------------------------------------------------
# 7. Install systemd service unit for the Hermes Web UI (dashboard)
# Separate unit from the gateway so the UI can be restarted / disabled
# independently. Fronted upstream by Traefik (jarvis service) on
# lightning-lane, so binding 0.0.0.0 with --insecure is intentional.
# ---------------------------------------------------------------------------
- name: Deploy hermes-dashboard systemd service unit
ansible.builtin.template:
src: hermes-dashboard.service.j2
dest: /etc/systemd/system/{{ hermes_dashboard_service_name }}.service
owner: root
group: root
mode: "0644"
become: true
register: hermes_dashboard_unit
when: hermes_dashboard_enabled
- name: Reload systemd to pick up hermes-dashboard unit changes
ansible.builtin.systemd:
daemon_reload: true
become: true
when:
- hermes_dashboard_enabled
- hermes_dashboard_unit is changed
- name: Enable and start hermes-dashboard service
ansible.builtin.systemd:
name: "{{ hermes_dashboard_service_name }}"
enabled: true
state: started
become: true
when: hermes_dashboard_enabled
- name: Restart hermes-dashboard on unit change
ansible.builtin.systemd:
name: "{{ hermes_dashboard_service_name }}"
state: restarted
become: true
when:
- hermes_dashboard_enabled
- hermes_dashboard_unit is changed

View File

@@ -0,0 +1,24 @@
[Unit]
Description=Hermes Dashboard (Web UI)
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User={{ hermes_user }}
Group={{ hermes_group }}
WorkingDirectory={{ hermes_home }}
Environment="HOME={{ hermes_home }}"
Environment="PATH={{ hermes_home }}/.local/bin:/usr/local/bin:/usr/bin:/bin"
Environment="HERMES_HOME={{ hermes_home }}/.hermes"
ExecStart={{ hermes_home }}/.hermes/hermes-agent/venv/bin/hermes dashboard --host {{ hermes_dashboard_host }} --port {{ hermes_dashboard_port }}{% if hermes_dashboard_insecure %} --insecure{% endif %}
TimeoutStopSec=30
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=hermes-dashboard
[Install]
WantedBy=multi-user.target

View File

@@ -0,0 +1,23 @@
[Unit]
Description=Hermes Agent (Nous Research)
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User={{ hermes_user }}
Group={{ hermes_group }}
WorkingDirectory={{ hermes_home }}
Environment="HOME={{ hermes_home }}"
Environment="PATH={{ hermes_home }}/.local/bin:/usr/local/bin:/usr/bin:/bin"
Environment="HERMES_HOME={{ hermes_home }}/.hermes"
ExecStart={{ hermes_home }}/.hermes/hermes-agent/venv/bin/hermes gateway
TimeoutStopSec=200
Restart=on-failure
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=hermes
[Install]
WantedBy=multi-user.target

View File

@@ -0,0 +1,118 @@
# Honcho role
Deploys [Honcho](https://honcho.dev) — Plastic Labs' memory + theory-of-mind
layer for stateful agents — on a single mk-labs VM (`lincoln`) using
rootful Podman + Quadlet.
## Topology
```
┌──────────────────────────────┐
│ lightning-lane (Traefik) │
│ hall-of-presidents.l... │
└─────────────┬────────────────┘
│ HTTP :8000
┌──────────────────────────────── lincoln ───────────────────────────────┐
│ │
│ ┌───────────────┐ ┌───────────────┐ ┌──────────────────────┐ │
│ │ honcho-api │ │ honcho-deriver│ │ honcho-postgres │ │
│ │ (FastAPI:8000)│◄──►│ (worker loop) │◄──►│ pgvector/pgvector:pg16│ │
│ └──────┬────────┘ └──────┬────────┘ └──────────────────────┘ │
│ │ │ │
│ └────────┬───────────┘ │
│ ▼ │
│ Anthropic Claude API (outbound, east-west to Internet) │
│ │
└────────────────────────────────────────────────────────────────────────┘
```
## What this role does
1. Installs Podman + ensures the rootful Quadlet drop-in dir exists.
2. Creates a user-defined Podman network (`honcho-net`).
3. Creates a named volume for Postgres data (`honcho_postgres_data`).
4. Deploys a pgvector-enabled Postgres 16 container and waits for it to be
ready.
5. Deploys the Honcho API container (`honcho-api`) — FastAPI on :8000.
6. Deploys the Honcho deriver worker container (`honcho-deriver`) — **stopped
and disabled by default** (see below).
7. Verifies the API answers on `/docs`.
## Deriver service (disabled by default)
The `honcho-deriver` service is **created but not started** by default
(`honcho_deriver_autostart: false`). This prevents autonomous token burn.
**Why disabled:**
The deriver runs background reasoning loops that consume LLM tokens
continuously, independent of the Hermes client-side `contextCadence` and
`dialecticCadence` settings. Even with both cadences set to zero in
`~/.hermes/honcho.json`, the deriver polls Postgres every second and fires
Claude API calls autonomously.
In production, this discovered behavior cost $10 USD overnight before the
service was stopped.
**Manual start (if needed for testing):**
```bash
# On lincoln.local.mk-labs.cloud as jarvis or with sudo:
sudo systemctl start honcho-deriver.service
# Check status:
sudo systemctl status honcho-deriver.service
# Stop again:
sudo systemctl stop honcho-deriver.service
```
**To enable permanently:**
Override `honcho_deriver_autostart: true` in `group_vars` or inventory, then
re-run the playbook.
## Required vault entries
Add to `group_vars/all/vault` (ansible-vault encrypted):
```yaml
vault_honcho_database_password: "<strong random>"
vault_honcho_jwt_secret: "<32+ byte random>"
vault_honcho_anthropic_api_key: "sk-ant-..."
```
## LLM provider switching
This role starts with Anthropic Claude. To swap to a local
OpenAI-compatible endpoint later (e.g. astro-orbiter once we have the
GPU running llama.cpp/Ollama in OpenAI-compatible mode), override these
in `group_vars/honcho_server` or via a playbook variable:
```yaml
honcho_llm_transport: "openai"
honcho_deriver_model: "qwen3:8b"
honcho_summary_model: "qwen3:8b"
honcho_dialectic_model: "qwen3:8b"
# and add LLM_OPENAI_API_KEY (or set the base URL override in the template)
```
## Traefik route
The Traefik file-provider YAML for `hall-of-presidents.local.mk-labs.cloud`
lives at `boilerplates/traefik/dynamic/honcho.yml`. It points at
`http://lincoln.local.mk-labs.cloud:8000`.
## Known Quadlet pitfall (carried from the semaphore role)
Quadlet regenerates the systemd unit on `.container` file change but does
NOT restart the running container. This role explicitly handles that by
gating a `state: restarted` task on the template's `changed` status —
the same pattern documented in `homelab-application-deployment`'s
quadlet-patterns reference.
## Honcho version pinning
The `honcho_image` default is `ghcr.io/plastic-labs/honcho:latest`.
Move to a digest-pinned tag once we've validated the deployment works
against a known-good build.

View File

@@ -0,0 +1,159 @@
---
# ============================================================================
# honcho role defaults
# ============================================================================
# Deploys Honcho (https://honcho.dev — plastic-labs/honcho) as a rootful
# Podman + Quadlet service on a single VM. Three containers:
#
# honcho-postgres pgvector-enabled PostgreSQL backing store
# honcho-api FastAPI server on :8000 (theory-of-mind read/write)
# honcho-deriver background worker that consumes the queue and calls
# out to the configured LLM provider for derivations
#
# All three share a user-defined podman network. State persists to named
# volumes. Traefik on lightning-lane terminates TLS and routes the
# `hall-of-presidents.local.mk-labs.cloud` host to honcho-api:8000.
#
# LLM provider for the first deployment is Anthropic Claude. Switching
# providers is a single env-var change (see DERIVER_MODEL_CONFIG__TRANSPORT
# / SUMMARY_MODEL_CONFIG__TRANSPORT and the LLM_*_API_KEY variables).
# ============================================================================
# ---------------------------------------------------------------------------
# Image pinning
# ---------------------------------------------------------------------------
# Honcho does not yet publish a Docker Hub image we trust; we use the
# GitHub Container Registry build. Pin to a digest-stable tag.
honcho_image: "ghcr.io/plastic-labs/honcho:latest"
# pgvector/pgvector image follows the upstream postgres version channel.
honcho_postgres_image: "docker.io/pgvector/pgvector:pg16"
# ---------------------------------------------------------------------------
# Container & network identifiers
# ---------------------------------------------------------------------------
honcho_container_name: honcho-api
honcho_deriver_container_name: honcho-deriver
honcho_postgres_container_name: honcho-postgres
honcho_network_name: honcho-net
# Named podman volumes
honcho_postgres_volume: honcho_postgres_data
# ---------------------------------------------------------------------------
# Networking
# ---------------------------------------------------------------------------
# Honcho's FastAPI default port is 8000. Bind to 0.0.0.0 so Traefik on
# lightning-lane can reach it; auth is enabled (JWT) so the open port is
# not an open door.
honcho_listen_address: "0.0.0.0"
honcho_listen_port: 8000
# Public-facing URL used for absolute links / OIDC callbacks.
# Leave EMPTY to make Honcho host-agnostic — FastAPI emits relative URLs.
# Set to a fully-qualified URL only if a specific feature requires it.
honcho_web_url: ""
# ---------------------------------------------------------------------------
# PostgreSQL configuration
# ---------------------------------------------------------------------------
honcho_db_user: honcho
honcho_db_name: honcho
# honcho_db_password sourced from vault below
# pgvector-enabled Postgres uses the same env vars as the stock image
honcho_postgres_initdb_args: "--encoding=UTF8 --locale=C"
# ---------------------------------------------------------------------------
# LLM provider configuration
# ---------------------------------------------------------------------------
# Supported transports: openai, anthropic, gemini. We start with anthropic
# and can flip to a local OpenAI-compatible endpoint later by changing
# these knobs plus the API-key env-var name.
#
# Honcho has THREE distinct LLM subsystems, each with its own settings
# class and env-var prefix:
#
# DERIVER — generates observations from messages (background worker)
# SUMMARY — periodic session summaries
# DIALECTIC— answers theory-of-mind queries (the `peers/{peer}/chat` API)
#
# Honcho's dialectic subsystem uses PER-LEVEL config (minimal/low/medium/
# high/max) under DIALECTIC_LEVELS__<level>__MODEL_CONFIG__*. Defaults
# point at OpenAI, so any honcho.dev install without an OpenAI key will
# fail dialectic queries with "Missing API key for openai model config".
# We override all five levels to Anthropic below in the API template.
honcho_llm_transport: "anthropic"
honcho_deriver_model: "claude-sonnet-4-5"
honcho_summary_model: "claude-sonnet-4-5"
honcho_dialectic_model: "claude-sonnet-4-5"
# Deriver tuning. Honcho batches representation tasks until the per-batch
# token threshold is reached. For homelab use with one chatty operator,
# that means short bursts of conversation can sit unprocessed forever.
# DERIVER_FLUSH_ENABLED=true bypasses the batching threshold so each
# message is processed promptly.
#
# honcho_deriver_enabled controls whether the deriver Quadlet is created.
# honcho_deriver_autostart controls whether the service starts on boot.
# Set autostart=false to create the service but leave it disabled — useful
# after discovering the deriver burns tokens autonomously despite client-side
# cadence=0 settings.
honcho_deriver_enabled: true
honcho_deriver_autostart: false
honcho_deriver_workers: 1
honcho_deriver_polling_seconds: "1.0"
honcho_deriver_flush_enabled: true
# ---------------------------------------------------------------------------
# Embeddings
# ---------------------------------------------------------------------------
# Anthropic does not provide an embedding API. Honcho defaults to
# OpenAI text-embedding-3-small, which fails without an OpenAI key.
# Three options going forward:
# 1. Set EMBED_MESSAGES=false — disables semantic search/vector recall
# but theory-of-mind derivation still works. Default for now.
# 2. Provide an OpenAI API key purely for embeddings (cheap; embeddings
# are ~$0.02 per million tokens). Set honcho_embed_messages=true and
# add vault_honcho_openai_api_key.
# 3. Stand up a local BGE/nomic embedding endpoint (e.g. on astro-orbiter
# once GPU is ready) and point Honcho at it via
# EMBEDDING_MODEL_CONFIG__OVERRIDES__BASE_URL.
honcho_embed_messages: true
# Embedding provider (transport must be openai-compatible). The default
# OpenAI endpoint requires honcho_openai_api_key. To swap to a local
# OpenAI-compatible embedder (e.g. Ollama), set honcho_embedding_base_url
# to the upstream /v1 URL and the API key can be any non-empty string.
honcho_embedding_transport: "openai"
honcho_embedding_model: "text-embedding-3-small"
honcho_embedding_base_url: ""
# ---------------------------------------------------------------------------
# Auth
# ---------------------------------------------------------------------------
# AUTH_USE_AUTH=true means clients must present a JWT signed with
# AUTH_JWT_SECRET. The secret lives in vault.
honcho_auth_enabled: true
# ---------------------------------------------------------------------------
# Vault inputs (defined in group_vars/all/vault, encrypted with ansible-vault)
# ---------------------------------------------------------------------------
# vault_honcho_database_password - postgres role password
# vault_honcho_jwt_secret - 32+ byte random string for JWT signing
# vault_honcho_anthropic_api_key - Anthropic API key for Claude calls
# vault_honcho_openai_api_key - OpenAI API key, embeddings-only
honcho_db_password: "{{ vault_honcho_database_password }}"
honcho_jwt_secret: "{{ vault_honcho_jwt_secret }}"
honcho_anthropic_api_key: "{{ vault_honcho_anthropic_api_key }}"
honcho_openai_api_key: "{{ vault_honcho_openai_api_key }}"
# ---------------------------------------------------------------------------
# Health check
# ---------------------------------------------------------------------------
honcho_health_check_retries: 30
honcho_health_check_delay: 2
# ---------------------------------------------------------------------------
# Quadlet location (rootful)
# ---------------------------------------------------------------------------
honcho_quadlet_dir: /etc/containers/systemd

View File

@@ -0,0 +1,27 @@
---
galaxy_info:
role_name: honcho
author: JARVIS
description: >-
Deploys Honcho (plastic-labs/honcho) — a memory + theory-of-mind layer
for stateful agents — with a pgvector-enabled PostgreSQL backing store,
via rootful Podman Quadlet on Ubuntu. Designed for the mk-labs
`lincoln` host. Fronted by Traefik at hall-of-presidents.local.mk-labs.cloud.
license: MIT
min_ansible_version: "2.14"
platforms:
- name: Ubuntu
versions:
- noble
- jammy
galaxy_tags:
- honcho
- memory
- llm
- podman
- quadlet
- homelab
# linux-baseline is applied separately as a day0 playbook. We don't depend
# on it here so this role stays composable.
dependencies: []

View File

@@ -0,0 +1,34 @@
---
# ============================================================================
# Honcho API container (Quadlet)
# ----------------------------------------------------------------------------
# Runs `fastapi run src/main.py` via the image's default CMD. The
# entrypoint.sh in the image runs the DB migration first; safe to do on
# every restart (idempotent).
# ============================================================================
- name: Deploy Honcho API Quadlet
ansible.builtin.template:
src: honcho-api.container.j2
dest: "{{ honcho_quadlet_dir }}/{{ honcho_container_name }}.container"
owner: root
group: root
mode: "0644"
register: honcho_api_quadlet
- name: Reload systemd to pick up Quadlet changes
ansible.builtin.systemd:
daemon_reload: true
when: honcho_api_quadlet.changed
- name: Ensure Honcho API container is started and enabled
ansible.builtin.systemd:
name: "{{ honcho_container_name }}.service"
state: started
enabled: true
- name: Restart Honcho API on Quadlet change
ansible.builtin.systemd:
name: "{{ honcho_container_name }}.service"
state: restarted
when: honcho_api_quadlet.changed

View File

@@ -0,0 +1,36 @@
---
# ============================================================================
# Honcho deriver worker container (Quadlet)
# ----------------------------------------------------------------------------
# Same image as the API container, but runs the deriver script instead of
# the FastAPI server. Pulls jobs off the in-database queue and calls the
# configured LLM provider for theory-of-mind derivations.
# ============================================================================
- name: Deploy Honcho deriver Quadlet
ansible.builtin.template:
src: honcho-deriver.container.j2
dest: "{{ honcho_quadlet_dir }}/{{ honcho_deriver_container_name }}.container"
owner: root
group: root
mode: "0644"
register: honcho_deriver_quadlet
- name: Reload systemd to pick up Quadlet changes
ansible.builtin.systemd:
daemon_reload: true
when: honcho_deriver_quadlet.changed
- name: Ensure Honcho deriver service exists but is disabled by default
ansible.builtin.systemd:
name: "{{ honcho_deriver_container_name }}.service"
state: "{{ 'started' if honcho_deriver_autostart | bool else 'stopped' }}"
enabled: "{{ honcho_deriver_autostart | bool }}"
- name: Restart Honcho deriver on Quadlet change (only if autostart enabled)
ansible.builtin.systemd:
name: "{{ honcho_deriver_container_name }}.service"
state: restarted
when:
- honcho_deriver_quadlet.changed
- honcho_deriver_autostart | bool

View File

@@ -0,0 +1,42 @@
---
# ============================================================================
# honcho / main entrypoint
# ----------------------------------------------------------------------------
# Order matters:
# 1. Podman + Quadlet support installed and ready
# 2. Network created (containers reference it by name)
# 3. Volumes created (postgres data)
# 4. Postgres started (Honcho API + deriver depend on it being ready)
# 5. Honcho API started
# 6. Honcho deriver worker started
# 7. Verify reachable on listen port
# ============================================================================
- name: Install Podman and dependencies
ansible.builtin.import_tasks: podman.yml
tags: [honcho, podman]
- name: Ensure podman network exists
ansible.builtin.import_tasks: network.yml
tags: [honcho, network]
- name: Ensure podman volumes exist
ansible.builtin.import_tasks: volumes.yml
tags: [honcho, volumes]
- name: Deploy PostgreSQL (pgvector) container
ansible.builtin.import_tasks: postgres.yml
tags: [honcho, postgres]
- name: Deploy Honcho API container
ansible.builtin.import_tasks: api.yml
tags: [honcho, api]
- name: Deploy Honcho deriver worker
ansible.builtin.import_tasks: deriver.yml
when: honcho_deriver_enabled | bool
tags: [honcho, deriver]
- name: Verify Honcho is reachable
ansible.builtin.import_tasks: verify.yml
tags: [honcho, verify]

View File

@@ -0,0 +1,17 @@
---
# ============================================================================
# Podman network — single user-defined network shared by api/deriver/postgres
# ============================================================================
- name: Check if honcho network exists
ansible.builtin.command:
cmd: "podman network exists {{ honcho_network_name }}"
register: honcho_network_check
failed_when: false
changed_when: false
- name: Create honcho podman network
ansible.builtin.command:
cmd: "podman network create {{ honcho_network_name }}"
when: honcho_network_check.rc != 0
changed_when: true

View File

@@ -0,0 +1,24 @@
---
# ============================================================================
# Podman + Quadlet prerequisites
# ----------------------------------------------------------------------------
# Mirrors the pattern from the semaphore role. Ubuntu 24.04's podman is new
# enough that Quadlet ships out of the box (>= 4.4).
# ============================================================================
- name: Ensure Podman is installed
ansible.builtin.package:
name:
- podman
- podman-compose
state: present
become: true
- name: Ensure Quadlet drop-in directory exists
ansible.builtin.file:
path: "{{ honcho_quadlet_dir }}"
state: directory
owner: root
group: root
mode: "0755"
become: true

View File

@@ -0,0 +1,48 @@
---
# ============================================================================
# pgvector PostgreSQL container (Quadlet)
# ----------------------------------------------------------------------------
# Honcho stores embeddings in pgvector, so we use the pgvector-enabled
# image rather than stock postgres. Same env-var surface as the upstream
# image; the pgvector extension is created by Honcho's migration script
# on first start (scripts/provision_db.py).
# ============================================================================
- name: Deploy Honcho PostgreSQL Quadlet
ansible.builtin.template:
src: honcho-postgres.container.j2
dest: "{{ honcho_quadlet_dir }}/{{ honcho_postgres_container_name }}.container"
owner: root
group: root
mode: "0644"
register: postgres_quadlet
- name: Reload systemd to pick up Quadlet changes
ansible.builtin.systemd:
daemon_reload: true
when: postgres_quadlet.changed
- name: Ensure Honcho PostgreSQL container is started and enabled
ansible.builtin.systemd:
name: "{{ honcho_postgres_container_name }}.service"
state: started
enabled: true
- name: Wait for PostgreSQL to accept connections
ansible.builtin.command:
cmd: >-
podman exec {{ honcho_postgres_container_name }}
pg_isready -U {{ honcho_db_user }} -d {{ honcho_db_name }}
register: pg_ready
until: pg_ready.rc == 0
retries: 30
delay: 2
changed_when: false
# Quadlet does not auto-restart on .container changes — same pitfall as
# semaphore role. Force restart only when the template was modified.
- name: Restart Honcho PostgreSQL on Quadlet change
ansible.builtin.systemd:
name: "{{ honcho_postgres_container_name }}.service"
state: restarted
when: postgres_quadlet.changed

View File

@@ -0,0 +1,27 @@
---
# ============================================================================
# Post-deploy verification
# ----------------------------------------------------------------------------
# Polls the local Honcho API port until it responds. Fails loudly if the
# service doesn't come up — the operator should not get an "all green"
# playbook result while Honcho is silently broken.
# ============================================================================
- name: Wait for Honcho HTTP endpoint
ansible.builtin.uri:
url: "http://127.0.0.1:{{ honcho_listen_port }}/docs"
status_code: [200, 307]
return_content: false
register: honcho_ping
until: honcho_ping.status in [200, 307]
retries: "{{ honcho_health_check_retries }}"
delay: "{{ honcho_health_check_delay }}"
- name: Report Honcho status
ansible.builtin.debug:
msg: >-
Honcho API reachable on http://127.0.0.1:{{ honcho_listen_port }}/docs.
Traefik should now route hall-of-presidents.local.mk-labs.cloud to
this backend. Deriver enabled: {{ honcho_deriver_enabled }};
LLM transport: {{ honcho_llm_transport }};
deriver model: {{ honcho_deriver_model }}.

View File

@@ -0,0 +1,17 @@
---
# ============================================================================
# Named podman volumes
# ----------------------------------------------------------------------------
# Honcho itself is stateless; only postgres needs durable storage.
# ============================================================================
- name: Ensure named volumes exist
ansible.builtin.command:
cmd: "podman volume create {{ item }}"
register: volume_create
changed_when: "'already exists' not in (volume_create.stderr | default(''))"
failed_when:
- volume_create.rc != 0
- "'already exists' not in (volume_create.stderr | default(''))"
loop:
- "{{ honcho_postgres_volume }}"

View File

@@ -0,0 +1,85 @@
# mk-labs Honcho API service (FastAPI on :8000)
# Managed by Ansible - honcho role
[Unit]
Description=Honcho API
After=network-online.target {{ honcho_postgres_container_name }}.service
Wants=network-online.target
Requires={{ honcho_postgres_container_name }}.service
[Container]
Image={{ honcho_image }}
ContainerName={{ honcho_container_name }}
Network={{ honcho_network_name }}
PublishPort={{ honcho_listen_address }}:{{ honcho_listen_port }}:8000
# Run the entrypoint that performs DB migration before starting FastAPI.
Exec=/app/docker/entrypoint.sh
# -- Database ---------------------------------------------------------------
Environment=DB_CONNECTION_URI=postgresql+psycopg://{{ honcho_db_user }}:{{ honcho_db_password }}@{{ honcho_postgres_container_name }}:5432/{{ honcho_db_name }}
# -- Auth (USE_AUTH + JWT secret) -------------------------------------------
Environment=AUTH_USE_AUTH={{ honcho_auth_enabled | string | lower }}
Environment=AUTH_JWT_SECRET={{ honcho_jwt_secret }}
# -- LLM provider keys ------------------------------------------------------
{% if honcho_llm_transport == 'anthropic' %}
Environment=LLM_ANTHROPIC_API_KEY={{ honcho_anthropic_api_key }}
{% endif %}
# OpenAI key — used only by the embedding subsystem. Always present so
# that conclusion vectorisation and any later semantic-search path works
# regardless of which provider serves LLM completions.
{% if honcho_openai_api_key | length > 0 %}
Environment=LLM_OPENAI_API_KEY={{ honcho_openai_api_key }}
Environment=OPENAI_API_KEY={{ honcho_openai_api_key }}
{% endif %}
# -- Deriver subsystem ------------------------------------------------------
Environment=DERIVER_FLUSH_ENABLED={{ honcho_deriver_flush_enabled | string | lower }}
Environment=DERIVER_MODEL_CONFIG__TRANSPORT={{ honcho_llm_transport }}
Environment=DERIVER_MODEL_CONFIG__MODEL={{ honcho_deriver_model }}
# -- Summary subsystem ------------------------------------------------------
Environment=SUMMARY_MODEL_CONFIG__TRANSPORT={{ honcho_llm_transport }}
Environment=SUMMARY_MODEL_CONFIG__MODEL={{ honcho_summary_model }}
# -- Dialectic subsystem (per-level overrides; defaults are openai) ---------
Environment=DIALECTIC_LEVELS__minimal__MODEL_CONFIG__TRANSPORT={{ honcho_llm_transport }}
Environment=DIALECTIC_LEVELS__minimal__MODEL_CONFIG__MODEL={{ honcho_dialectic_model }}
Environment=DIALECTIC_LEVELS__low__MODEL_CONFIG__TRANSPORT={{ honcho_llm_transport }}
Environment=DIALECTIC_LEVELS__low__MODEL_CONFIG__MODEL={{ honcho_dialectic_model }}
Environment=DIALECTIC_LEVELS__medium__MODEL_CONFIG__TRANSPORT={{ honcho_llm_transport }}
Environment=DIALECTIC_LEVELS__medium__MODEL_CONFIG__MODEL={{ honcho_dialectic_model }}
Environment=DIALECTIC_LEVELS__high__MODEL_CONFIG__TRANSPORT={{ honcho_llm_transport }}
Environment=DIALECTIC_LEVELS__high__MODEL_CONFIG__MODEL={{ honcho_dialectic_model }}
Environment=DIALECTIC_LEVELS__max__MODEL_CONFIG__TRANSPORT={{ honcho_llm_transport }}
Environment=DIALECTIC_LEVELS__max__MODEL_CONFIG__MODEL={{ honcho_dialectic_model }}
# -- Embeddings (text-embedding-3-small via OpenAI by default) --------------
Environment=EMBED_MESSAGES={{ honcho_embed_messages | string | lower }}
Environment=EMBEDDING__MODEL_CONFIG__TRANSPORT={{ honcho_embedding_transport }}
Environment=EMBEDDING__MODEL_CONFIG__MODEL={{ honcho_embedding_model }}
{% if honcho_embedding_base_url | length > 0 %}
Environment=EMBEDDING__MODEL_CONFIG__BASE_URL={{ honcho_embedding_base_url }}
{% endif %}
# -- Vector store -----------------------------------------------------------
Environment=VECTOR_STORE_TYPE=pgvector
# -- Public-facing URL (empty -> relative URLs) -----------------------------
{% if honcho_web_url | length > 0 %}
Environment=HONCHO_PUBLIC_URL={{ honcho_web_url }}
{% endif %}
# Timezone matches host baseline
Environment=TZ=America/Chicago
Environment=LOG_LEVEL=INFO
[Service]
Restart=always
TimeoutStartSec=180
[Install]
WantedBy=multi-user.target default.target

View File

@@ -0,0 +1,70 @@
# mk-labs Honcho deriver worker (background queue consumer)
# Managed by Ansible - honcho role
[Unit]
Description=Honcho deriver worker
After=network-online.target {{ honcho_postgres_container_name }}.service {{ honcho_container_name }}.service
Wants=network-online.target
Requires={{ honcho_postgres_container_name }}.service
[Container]
Image={{ honcho_image }}
ContainerName={{ honcho_deriver_container_name }}
Network={{ honcho_network_name }}
# Invoke the package's __main__.py — src/deriver/deriver.py has no
# __main__ guard and exits 0 in ~3s if invoked directly, causing systemd
# to crash-loop the unit. python -m src.deriver hits __main__.py and
# starts the real queue processor.
WorkingDir=/app
Exec=/app/.venv/bin/python -m src.deriver
# -- Database ---------------------------------------------------------------
Environment=DB_CONNECTION_URI=postgresql+psycopg://{{ honcho_db_user }}:{{ honcho_db_password }}@{{ honcho_postgres_container_name }}:5432/{{ honcho_db_name }}
# -- Auth -------------------------------------------------------------------
Environment=AUTH_USE_AUTH={{ honcho_auth_enabled | string | lower }}
Environment=AUTH_JWT_SECRET={{ honcho_jwt_secret }}
# -- LLM provider keys ------------------------------------------------------
{% if honcho_llm_transport == 'anthropic' %}
Environment=LLM_ANTHROPIC_API_KEY=*** honcho_anthropic_api_key }}
{% endif %}
{% if honcho_openai_api_key | length > 0 %}
Environment=LLM_OPENAI_API_KEY=*** honcho_openai_api_key }}
Environment=OPENAI_API_KEY=*** honcho_openai_api_key }}
{% endif %}
# -- Deriver runtime --------------------------------------------------------
Environment=DERIVER_ENABLED={{ honcho_deriver_enabled | string | lower }}
Environment=DERIVER_WORKERS={{ honcho_deriver_workers }}
Environment=DERIVER_POLLING_SLEEP_INTERVAL_SECONDS={{ honcho_deriver_polling_seconds }}
Environment=DERIVER_FLUSH_ENABLED={{ honcho_deriver_flush_enabled | string | lower }}
Environment=DERIVER_MODEL_CONFIG__TRANSPORT={{ honcho_llm_transport }}
Environment=DERIVER_MODEL_CONFIG__MODEL={{ honcho_deriver_model }}
# -- Summary subsystem ------------------------------------------------------
Environment=SUMMARY_MODEL_CONFIG__TRANSPORT={{ honcho_llm_transport }}
Environment=SUMMARY_MODEL_CONFIG__MODEL={{ honcho_summary_model }}
# -- Embeddings -------------------------------------------------------------
Environment=EMBED_MESSAGES={{ honcho_embed_messages | string | lower }}
Environment=EMBEDDING__MODEL_CONFIG__TRANSPORT={{ honcho_embedding_transport }}
Environment=EMBEDDING__MODEL_CONFIG__MODEL={{ honcho_embedding_model }}
{% if honcho_embedding_base_url | length > 0 %}
Environment=EMBEDDING__MODEL_CONFIG__BASE_URL={{ honcho_embedding_base_url }}
{% endif %}
# -- Vector store -----------------------------------------------------------
Environment=VECTOR_STORE_TYPE=pgvector
# Timezone matches host baseline
Environment=TZ=America/Chicago
Environment=LOG_LEVEL=INFO
[Service]
Restart=always
TimeoutStartSec=180
[Install]
WantedBy=multi-user.target default.target

View File

@@ -0,0 +1,35 @@
# {{ ansible_managed }}
# mk-labs pgvector-enabled PostgreSQL backing store for Honcho
# Managed by Ansible - honcho role
[Unit]
Description=PostgreSQL (pgvector) for Honcho
After=network-online.target
Wants=network-online.target
[Container]
Image={{ honcho_postgres_image }}
ContainerName={{ honcho_postgres_container_name }}
Network={{ honcho_network_name }}
# Persistent data on a named volume. :Z relabels for SELinux; harmless on
# Ubuntu's ext4/apparmor stack and portable to any future host.
Volume={{ honcho_postgres_volume }}:/var/lib/postgresql/data:Z
Environment=POSTGRES_USER={{ honcho_db_user }}
Environment=POSTGRES_DB={{ honcho_db_name }}
Environment=POSTGRES_PASSWORD={{ honcho_db_password }}
Environment=POSTGRES_INITDB_ARGS={{ honcho_postgres_initdb_args }}
# Healthcheck — systemd doesn't act on this, but it's useful diagnostically
HealthCmd=pg_isready -U {{ honcho_db_user }} -d {{ honcho_db_name }}
HealthInterval=10s
HealthTimeout=5s
HealthRetries=3
[Service]
Restart=always
TimeoutStartSec=120
[Install]
WantedBy=multi-user.target default.target

View File

@@ -0,0 +1,5 @@
---
jarvis_user: jarvis
jarvis_group: jarvis
jarvis_shell: /bin/bash
jarvis_sudo_nopasswd: true

View File

@@ -0,0 +1,30 @@
---
- name: Create jarvis group
ansible.builtin.group:
name: "{{ jarvis_group }}"
state: present
system: false
- name: Create jarvis user
ansible.builtin.user:
name: "{{ jarvis_user }}"
group: "{{ jarvis_group }}"
shell: "{{ jarvis_shell }}"
create_home: true
state: present
- name: Deploy SSH public key
ansible.posix.authorized_key:
user: "{{ jarvis_user }}"
key: "{{ lookup('file', '~/.ssh/id_jarvis.pub') }}"
state: present
exclusive: true
- name: Grant passwordless sudo
ansible.builtin.copy:
dest: "/etc/sudoers.d/{{ jarvis_user }}"
content: "{{ jarvis_user }} ALL=(ALL) NOPASSWD:ALL\n"
owner: root
group: root
mode: '0440'
when: jarvis_sudo_nopasswd | bool

View File

@@ -0,0 +1,53 @@
---
# JMRI version to install
jmri_version: "5.10"
jmri_install_dir: /opt/JMRI
jmri_download_url: "https://github.com/JMRI/JMRI/releases/download/v{{ jmri_version }}/JMRI.{{ jmri_version }}+Rca461bd266.tgz"
# Service user
jmri_user: jmri
jmri_group: jmri
jmri_home: /home/jmri
# Profile — set per-host in host_vars
jmri_profile_id: ""
# LCRR git repo (set per-host in host_vars; leave blank to skip clone)
jmri_lcrr_repo: ""
# SSH key for jmri user (for jmri-gui X11 access — set per-host in host_vars)
jmri_ssh_authorized_key: ""
# SSH key for jmri user → Gitea
jmri_gitea_key: /home/jmri/.ssh/id_ed25519_gitea
# Config restore source (legacy tar-based restore — leave blank to skip)
jmri_config_src: ""
# Ports (for documentation / firewall rules)
jmri_json_port: 12080
jmri_withrottle_port: 12090
# ---------------------------------------------------------------------------
# Phase 2 — Layout power monitor (Leviton Decora Smart Wi-Fi)
# ---------------------------------------------------------------------------
jmri_leviton_email: "" # set via group_vars (vault-backed)
jmri_leviton_password: "" # set via group_vars (vault-backed)
jmri_leviton_switch_name: "Layout"
jmri_monitor_poll_interval: 30 # seconds between polls (active hours)
jmri_monitor_quiet_start: 1 # hour (24h) to stop polling
jmri_monitor_quiet_end: 10 # hour (24h) to resume polling
jmri_monitor_stop_delay: 30 # seconds of OFF state before stopping JMRI
# ---------------------------------------------------------------------------
# USB device identification (ENV{ID_SERIAL} from udev — the segment of the
# by-id symlink name before the -ifNN suffix)
# ---------------------------------------------------------------------------
# LocoBuffer-NG (LocoNet bridge) → /dev/jmri/loconet
jmri_dev_loconet_serial: "RR-CirKits_LocoBuffer-NG_CDC_ACM_SERIAL_DEVICE_AA5700218A"
# NCE command station → /dev/jmri/nce ← triggers service start/stop
jmri_dev_nce_serial: "Microchip_Technology_Inc._Simple_CDC_Device_Demo"
# FTDI LCC interface → /dev/jmri/lcc
jmri_dev_lcc_serial: "ftdi_usb_serial_converter_ftDYQHZX"

View File

@@ -0,0 +1,23 @@
---
- name: Reload udev
ansible.builtin.command: udevadm control --reload-rules
changed_when: false
- name: Reload systemd
ansible.builtin.systemd:
daemon_reload: true
- name: Restart jmri-monitor
ansible.builtin.systemd:
name: jmri-monitor
state: restarted
- name: Restart jmri
ansible.builtin.systemd:
name: jmri
state: restarted
- name: Restart sshd
ansible.builtin.systemd:
name: ssh
state: restarted

View File

@@ -0,0 +1,13 @@
---
galaxy_info:
role_name: jmri
author: JARVIS
description: Deploy JMRI JmriFaceless headless server as a systemd service
license: MIT
min_ansible_version: "2.12"
platforms:
- name: Ubuntu
versions:
- jammy
- noble
dependencies: []

View File

@@ -0,0 +1,270 @@
---
- name: Install Java runtime (full — required for GUI mode)
ansible.builtin.apt:
name:
- openjdk-21-jre
- openjdk-21-jdk
state: present
update_cache: true
- name: Create JMRI system group
ansible.builtin.group:
name: "{{ jmri_group }}"
state: present
system: true
- name: Create JMRI service user
ansible.builtin.user:
name: "{{ jmri_user }}"
group: "{{ jmri_group }}"
home: "{{ jmri_home }}"
shell: /bin/bash
system: true
create_home: true
state: present
- name: Deploy SSH authorized key for jmri user
ansible.posix.authorized_key:
user: "{{ jmri_user }}"
key: "{{ jmri_ssh_authorized_key }}"
state: present
when: jmri_ssh_authorized_key | length > 0
- name: Add JMRI user to dialout group (serial device access)
ansible.builtin.user:
name: "{{ jmri_user }}"
groups: dialout
append: true
# ---------------------------------------------------------------------------
# Phase 1 — Stable udev device symlinks
# Creates /dev/jmri/nce, /dev/jmri/loconet, /dev/jmri/lcc
# ---------------------------------------------------------------------------
- name: Create /dev/jmri symlink directory (static — survives reboots)
ansible.builtin.file:
path: /etc/jmri
state: directory
owner: root
group: root
mode: '0755'
- name: Deploy udev rules for stable JMRI device symlinks
ansible.builtin.template:
src: 99-jmri-devices.rules.j2
dest: /etc/udev/rules.d/99-jmri-devices.rules
owner: root
group: root
mode: '0644'
notify: Reload udev
# ---------------------------------------------------------------------------
# Phase 2 — LocoNet traffic monitor
# Installs jmri-monitor: watches /dev/jmri/loconet, starts/stops jmri.service
# ---------------------------------------------------------------------------
- name: Install python3-venv (monitor virtualenv support)
ansible.builtin.apt:
name: python3-venv
state: present
- name: Create virtualenv for jmri-monitor
ansible.builtin.command:
cmd: python3 -m venv /opt/jmri-monitor
creates: /opt/jmri-monitor/bin/python3
- name: Install decora_wifi into jmri-monitor virtualenv
ansible.builtin.pip:
name: decora_wifi
state: present
virtualenv: /opt/jmri-monitor
- name: Deploy jmri-monitor script
ansible.builtin.template:
src: jmri-monitor.py.j2
dest: /usr/local/bin/jmri-monitor
owner: root
group: root
mode: '0755'
notify: Restart jmri-monitor
- name: Deploy jmri-monitor systemd unit
ansible.builtin.template:
src: jmri-monitor.service.j2
dest: /etc/systemd/system/jmri-monitor.service
owner: root
group: root
mode: '0644'
notify:
- Reload systemd
- Restart jmri-monitor
- name: Enable jmri-monitor service
ansible.builtin.systemd:
name: jmri-monitor
enabled: true
state: started
daemon_reload: true
# ---------------------------------------------------------------------------
# Phase 2b — LCRR config repo (clone/pull .jmri from Gitea)
# ---------------------------------------------------------------------------
- name: Ensure jmri .ssh directory exists
ansible.builtin.file:
path: /home/jmri/.ssh
state: directory
owner: "{{ jmri_user }}"
group: "{{ jmri_group }}"
mode: '0700'
- name: Deploy jmri SSH config for Gitea
ansible.builtin.copy:
dest: /home/jmri/.ssh/config
owner: "{{ jmri_user }}"
group: "{{ jmri_group }}"
mode: '0600'
content: |
Host gitea.mk-labs.cloud
HostName gitea.mk-labs.cloud
User git
Port 2221
IdentityFile {{ jmri_gitea_key }}
StrictHostKeyChecking accept-new
- name: Clone LCRR config repo if not present
ansible.builtin.git:
repo: "{{ jmri_lcrr_repo }}"
dest: "{{ jmri_home }}/LCRR"
version: main
accept_hostkey: true
key_file: "{{ jmri_gitea_key }}"
update: false
become_user: "{{ jmri_user }}"
when: jmri_lcrr_repo | length > 0
notify: Restart jmri
- name: Remove auto-generated .jmri dir if it exists (will be replaced by symlink)
ansible.builtin.file:
path: "{{ jmri_home }}/.jmri"
state: absent
when:
- jmri_lcrr_repo | length > 0
- name: Link .jmri config from LCRR repo
ansible.builtin.file:
src: "{{ jmri_home }}/LCRR/.jmri"
dest: "{{ jmri_home }}/.jmri"
state: link
owner: "{{ jmri_user }}"
group: "{{ jmri_group }}"
force: true
when: jmri_lcrr_repo | length > 0
notify: Restart jmri
- name: Check if JMRI is already installed at correct version
ansible.builtin.stat:
path: "{{ jmri_install_dir }}/JmriFaceless"
register: jmri_binary
- name: Download JMRI release archive
ansible.builtin.get_url:
url: "{{ jmri_download_url }}"
dest: /tmp/JMRI-{{ jmri_version }}.tgz
mode: '0644'
when: not jmri_binary.stat.exists
- name: Create JMRI install directory
ansible.builtin.file:
path: "{{ jmri_install_dir }}"
state: directory
owner: "{{ jmri_user }}"
group: "{{ jmri_group }}"
mode: '0755'
- name: Extract JMRI archive
ansible.builtin.unarchive:
src: /tmp/JMRI-{{ jmri_version }}.tgz
dest: "{{ jmri_install_dir }}"
remote_src: true
owner: "{{ jmri_user }}"
group: "{{ jmri_group }}"
extra_opts: ['--strip-components=1']
when: not jmri_binary.stat.exists
- name: Restore JMRI config from backup
ansible.builtin.copy:
src: "{{ jmri_config_src }}/"
dest: "{{ jmri_home }}/.jmri/"
owner: "{{ jmri_user }}"
group: "{{ jmri_group }}"
mode: '0644'
directory_mode: '0755'
when: jmri_config_src | length > 0
notify: Restart jmri
- name: Deploy JmriFaceless systemd unit
ansible.builtin.template:
src: jmri.service.j2
dest: /etc/systemd/system/jmri.service
owner: root
group: root
mode: '0644'
notify:
- Reload systemd
- Restart jmri
- name: Enable jmri service (monitor manages start/stop — do not start directly)
ansible.builtin.systemd:
name: jmri
enabled: false
daemon_reload: true
# ---------------------------------------------------------------------------
# Phase 3 — X11 remote GUI access
# ---------------------------------------------------------------------------
- name: Install xauth (required for SSH X11 forwarding)
ansible.builtin.apt:
name: xauth
state: present
- name: Remove old jmri X11 sshd drop-in if present (renamed)
ansible.builtin.file:
path: /etc/ssh/sshd_config.d/20-jmri-x11.conf
state: absent
notify: Restart sshd
- name: Deploy sshd drop-in to enable X11 forwarding (must load before hardening)
ansible.builtin.copy:
dest: /etc/ssh/sshd_config.d/09-jmri-x11.conf
owner: root
group: root
mode: '0644'
content: |
# Allow X11 forwarding for JMRI GUI sessions (jmri role)
# Must be numbered below 10-mk-labs-hardening.conf — first match wins.
X11Forwarding yes
X11UseLocalhost yes
notify: Restart sshd
- name: Deploy jmri-gui script
ansible.builtin.template:
src: jmri-gui.j2
dest: /usr/local/bin/jmri-gui
owner: root
group: root
mode: '0755'
- name: Deploy sudoers drop-in for jmri-gui (wed can manage jmri service)
ansible.builtin.copy:
dest: /etc/sudoers.d/jmri-gui
owner: root
group: root
mode: '0440'
validate: 'visudo -cf %s'
content: |
# jmri user can manage its own service (for jmri-gui interactive sessions)
jmri ALL=(root) NOPASSWD: /usr/bin/systemctl start jmri.service
jmri ALL=(root) NOPASSWD: /usr/bin/systemctl stop jmri.service
jmri ALL=(root) NOPASSWD: /opt/jmri-monitor/bin/python3
# wed retains service control for automation/admin use
wed ALL=(root) NOPASSWD: /usr/bin/systemctl start jmri.service
wed ALL=(root) NOPASSWD: /usr/bin/systemctl stop jmri.service
wed ALL=(root) NOPASSWD: /opt/jmri-monitor/bin/python3

View File

@@ -0,0 +1,25 @@
# /etc/udev/rules.d/99-jmri-devices.rules
# Managed by Ansible — do not edit manually.
#
# Creates stable /dev/jmri/* symlinks for all three JMRI serial interfaces.
# Uses ID_SERIAL (full serial string) to match by-id symlink identity,
# so device numbering (ttyACM0, ttyACM1, etc.) doesn't matter after reboots.
# Ownership: root:dialout mode 0660 — jmri user is a member of dialout.
# NCE command station (Microchip CDC)
# Appears when layout is powered on; disappears when layout powers off.
SUBSYSTEM=="tty", ENV{ID_SERIAL}=="{{ jmri_dev_nce_serial }}", \
SYMLINK+="jmri/nce", \
GROUP="dialout", MODE="0660"
# RR-CirKits LocoBuffer-NG (LocoNet bridge)
# USB-powered — always present when server is running.
SUBSYSTEM=="tty", ENV{ID_SERIAL}=="{{ jmri_dev_loconet_serial }}", \
SYMLINK+="jmri/loconet", \
GROUP="dialout", MODE="0660"
# FTDI USB serial (LCC interface)
# USB-powered — always present when server is running.
SUBSYSTEM=="tty", ENV{ID_SERIAL}=="{{ jmri_dev_lcc_serial }}", \
SYMLINK+="jmri/lcc", \
GROUP="dialout", MODE="0660"

View File

@@ -0,0 +1,111 @@
#!/bin/bash
# jmri-gui — launch JMRI GUI over SSH -X (X11 forwarding)
# Managed by Ansible — do not edit manually.
#
# Usage (connect as jmri user):
# ssh -X jmri@main-street-station jmri-gui panelpro
# ssh -X jmri@main-street-station jmri-gui decoderpro
# ssh -X jmri@main-street-station jmri-gui status
#
# ~/.ssh/config recommended:
# Host main-street-station
# User jmri
# ForwardX11 yes
set -euo pipefail
JMRI_DIR="{{ jmri_install_dir }}"
JMRI_PROFILE="{{ jmri_profile_id }}"
JMRI_SERVICE="jmri.service"
MONITOR_SERVICE="jmri-monitor.service"
usage() {
echo "Usage: jmri-gui <panelpro|decoderpro|status>"
exit 1
}
check_display() {
if [ -z "${DISPLAY:-}" ]; then
echo "ERROR: No DISPLAY set. Connect with: ssh -X jmri@main-street-station"
exit 1
fi
}
status() {
echo "=== JMRI daemon ==="
systemctl status "$JMRI_SERVICE" --no-pager -l 2>&1 | head -8
echo ""
echo "=== Layout monitor ==="
systemctl status "$MONITOR_SERVICE" --no-pager -l 2>&1 | head -5
}
launch() {
local app="$1"
local binary
case "$app" in
panelpro) binary="PanelPro" ;;
decoderpro) binary="DecoderPro" ;;
*) usage ;;
esac
check_display
# Stop the daemon if running
if systemctl is-active --quiet "$JMRI_SERVICE" 2>/dev/null; then
echo "Stopping JMRI daemon..."
sudo systemctl stop "$JMRI_SERVICE"
fi
echo "Launching $binary (profile: $JMRI_PROFILE)..."
echo "Close the window to return to daemon mode."
echo ""
# SSH sets DISPLAY to localhost:N (TCP) — translate to unix socket form
# so Java's AWT xauth lookup matches the forwarded cookie.
export DISPLAY=$(echo "$DISPLAY" | sed 's/localhost:/:/')
# Force AWT out of headless mode
export JMRI_OPTIONS="-Djava.awt.headless=false"
# Run directly — we are already the jmri user, DISPLAY is set by SSH
"$JMRI_DIR/$binary" --profile="$JMRI_PROFILE"
echo ""
echo "$binary closed."
# Restart daemon if layout switch is still on
if systemctl is-active --quiet "$MONITOR_SERVICE" 2>/dev/null; then
if sudo /opt/jmri-monitor/bin/python3 - <<'EOF'
from decora_wifi import DecoraWiFiSession
from decora_wifi.models.residential_account import ResidentialAccount
import sys
session = DecoraWiFiSession()
person = session.login("{{ jmri_leviton_email }}", "{{ jmri_leviton_password }}")
perms = person.get_residential_permissions()
for perm in perms:
acct_id = perm.data.get('residentialAccountId')
if not acct_id:
continue
acct = ResidentialAccount(session, acct_id)
acct.refresh()
for r in acct.get_residences():
for s in r.get_iot_switches():
if s.data.get('name') == '{{ jmri_leviton_switch_name }}':
sys.exit(0 if s.data.get('power') == 'ON' else 1)
sys.exit(1)
EOF
then
echo "Layout is ON — restarting JMRI daemon."
sudo systemctl start "$JMRI_SERVICE"
else
echo "Layout is OFF — JMRI daemon will not restart."
fi
fi
}
case "${1:-}" in
panelpro|decoderpro) launch "$1" ;;
status) status ;;
*) usage ;;
esac

View File

@@ -0,0 +1,156 @@
#!/opt/jmri-monitor/bin/python3
"""
jmri-monitor — Leviton Decora Smart Wi-Fi layout power monitor
Managed by Ansible — do not edit manually.
Polls the Leviton cloud API for the "{{ jmri_leviton_switch_name }}" switch state.
- Switch ON after being OFF → systemctl start jmri.service
- Switch OFF for {{ jmri_monitor_stop_delay }}s → systemctl stop jmri.service
Quiet hours {{ jmri_monitor_quiet_start }}:00{{ jmri_monitor_quiet_end }}:00: no polling (layout assumed off).
"""
import subprocess
import time
import logging
import sys
from datetime import datetime
LEVITON_EMAIL = "{{ jmri_leviton_email }}"
LEVITON_PASSWORD = "{{ jmri_leviton_password }}"
SWITCH_NAME = "{{ jmri_leviton_switch_name }}"
POLL_INTERVAL = {{ jmri_monitor_poll_interval }}
QUIET_START = {{ jmri_monitor_quiet_start }}
QUIET_END = {{ jmri_monitor_quiet_end }}
STOP_DELAY = {{ jmri_monitor_stop_delay }}
JMRI_SERVICE = "jmri.service"
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [jmri-monitor] %(levelname)s: %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
stream=sys.stdout,
)
log = logging.getLogger(__name__)
def systemctl(action):
try:
result = subprocess.run(
["systemctl", action, JMRI_SERVICE],
capture_output=True, text=True, timeout=30,
)
if result.returncode == 0:
log.info("systemctl %s %s: OK", action, JMRI_SERVICE)
else:
log.warning("systemctl %s %s: %s", action, JMRI_SERVICE, result.stderr.strip())
except Exception as e:
log.error("systemctl %s failed: %s", action, e)
def is_quiet_hours():
hour = datetime.now().hour
if QUIET_START < QUIET_END:
return QUIET_START <= hour < QUIET_END
else:
# wraps midnight e.g. 236
return hour >= QUIET_START or hour < QUIET_END
def get_switch_state():
"""Returns True if switch is ON, False if OFF, None on error."""
try:
from decora_wifi import DecoraWiFiSession
from decora_wifi.models.residential_account import ResidentialAccount
session = DecoraWiFiSession()
person = session.login(LEVITON_EMAIL, LEVITON_PASSWORD)
if not person:
log.error("Leviton login failed")
return None
perms = person.get_residential_permissions()
for perm in perms:
acct_id = perm.data.get('residentialAccountId')
if not acct_id:
continue
acct = ResidentialAccount(session, acct_id)
acct.refresh()
for residence in acct.get_residences():
for switch in residence.get_iot_switches():
if switch.data.get('name') == SWITCH_NAME:
state = switch.data.get('power', 'OFF')
session.call_api('/Person/logout', {}, 'post')
return state == 'ON'
all_names = []
for perm in perms:
acct_id = perm.data.get('residentialAccountId')
if acct_id:
acct = ResidentialAccount(session, acct_id)
acct.refresh()
for r in acct.get_residences():
all_names += [s.data.get('name') for s in r.get_iot_switches()]
log.warning("Switch '%s' not found — available: %s", SWITCH_NAME, all_names)
session.call_api('/Person/logout', {}, 'post')
return None
except ImportError:
log.error("decora_wifi not installed")
return None
except Exception as e:
log.error("Error querying Leviton API: %s", e)
return None
def main():
log.info("Layout power monitor starting")
log.info("Switch: '%s' Poll: %ds Quiet: %02d:00%02d:00 Stop delay: %ds",
SWITCH_NAME, POLL_INTERVAL, QUIET_START, QUIET_END, STOP_DELAY)
layout_on = False
off_since = None
while True:
if is_quiet_hours():
log.debug("Quiet hours — sleeping 60s")
# If layout was on when quiet hours started, stop JMRI
if layout_on:
log.info("Quiet hours began — stopping JMRI")
layout_on = False
off_since = None
systemctl("stop")
time.sleep(60)
continue
state = get_switch_state()
if state is True:
off_since = None
if not layout_on:
log.info("Layout switch ON — starting JMRI")
layout_on = True
systemctl("start")
elif state is False:
if layout_on:
if off_since is None:
off_since = time.monotonic()
log.info("Layout switch OFF — waiting %ds before stopping JMRI", STOP_DELAY)
elif time.monotonic() - off_since >= STOP_DELAY:
log.info("Layout switch OFF for %ds — stopping JMRI", STOP_DELAY)
layout_on = False
off_since = None
systemctl("stop")
else:
off_since = None
else:
# API error — don't change state, try again next poll
log.warning("Could not determine switch state — retrying in %ds", POLL_INTERVAL)
time.sleep(POLL_INTERVAL)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,20 @@
[Unit]
Description=JMRI LocoNet Traffic Monitor
Documentation=https://www.jmri.org/
# Must start after udev has processed devices
After=systemd-udev-settle.service
[Service]
Type=simple
# Runs as root — needs to call systemctl start/stop jmri.service
User=root
ExecStart=/usr/local/bin/jmri-monitor
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=jmri-monitor
[Install]
WantedBy=multi-user.target

View File

@@ -0,0 +1,31 @@
[Unit]
Description=JMRI Server (JmriFaceless)
Documentation=https://www.jmri.org/
# jmri-monitor starts and stops this service based on LocoNet traffic.
# Do NOT enable this unit directly — it is managed by jmri-monitor.service.
After=network.target systemd-udev-settle.service
[Service]
Type=simple
User={{ jmri_user }}
Group={{ jmri_group }}
WorkingDirectory={{ jmri_install_dir }}
ExecStart={{ jmri_install_dir }}/JmriFaceless --profile={{ jmri_profile_id }}
# Monitor owns lifecycle — do not auto-restart
Restart=no
# Give hardware time to settle on start
TimeoutStartSec=30
# Logging — view with: journalctl -u jmri -f
StandardOutput=journal
StandardError=journal
SyslogIdentifier=jmri
# Serial device access
SupplementaryGroups=dialout
[Install]
# Intentionally no WantedBy — started only by jmri-monitor
WantedBy=

View File

@@ -0,0 +1,64 @@
# linux-baseline
Idempotent baseline for any Linux host in mk-labs.
## What it does
Applied on every host, this role guarantees a known-good operational baseline:
| Capability | Default | Description |
|----------------------|---------|---------------------------------------------------------------|
| `packages` | on | Installs curated common + OS-family packages, plus host extras |
| `timezone` | on | Sets timezone and locale |
| `time_sync` | on | Installs and configures chrony against `sundial` |
| `users` | on | Provisions baseline admin accounts (jarvis) with SSH + sudo |
| `ssh_hardening` | on | Deploys `/etc/ssh/sshd_config.d/10-mk-labs-hardening.conf` |
| `unattended_upgrades`| on | Security-only automatic upgrades on Debian-family hosts |
| `sysctl` | on | Drop-in at `/etc/sysctl.d/99-mk-labs.conf` |
| `journald` | on | Caps journal size + retention |
| `motd` | on | Branded MOTD |
| `full_upgrade` | off | Opt-in full dist-upgrade. Use the day2 update playbook instead |
Every capability sits behind a feature flag in `baseline_features` so it
can be disabled per-host without forking the role.
## OS support
- **First-class:** Ubuntu (22.04, 24.04) and Debian (bullseye, bookworm).
- **Placeholder:** RHEL family — see `vars/RedHat.yml`. Flesh out when a
RHEL-family host joins the fleet.
OS-specific values are loaded from `vars/{{ ansible_os_family }}.yml`.
## Required inventory variables
| Variable | Where | Example |
|-------------------------|-----------------|-------------------------------------------|
| `jarvis_ssh_public_key` | group_vars/all | `ssh-ed25519 AAAA... jarvis@mk-labs` |
## Usage
```yaml
- name: Apply linux-baseline
hosts: "{{ target | default('all') }}"
become: true
roles:
- linux-baseline
```
## Per-host overrides
```yaml
# host_vars/some-host/vars
baseline_packages_extra:
- postgresql-client
- redis-tools
baseline_features:
motd: false
```
## Migration notes
This role supersedes the legacy `day0-baseline` and `common` roles, both
of which are slated for removal once the fleet has been re-baselined.

Some files were not shown because too many files have changed in this diff Show More