Merge branch 'main' of gitea.mk-labs.cloud:rblundon/homelab

update vault
spaceship-earth: per-host Multica PAT, root LV growth, headless Obsidian (WDWL-22)
2026-10-22 16:00:51 -05:00 · 2026-10-22 15:59:59 -05:00 · 2026-10-22 15:41:54 -05:00 · 2026-10-22 12:32:18 -05:00 · 2026-10-22 12:30:35 -05:00 · 2026-10-21 11:50:53 -05:00
1058 changed files with 64000 additions and 12217 deletions
@@ -1,15 +1,9 @@
-# macOS system files
+# macOS
 .DS_Store
 .AppleDouble
 .LSOverride
 # Icon must end with two \r
 Icon
 # Thumbnails
 ._*
 # Files that might appear in the root of a volume
 .DocumentRevisions-V100
 .fseventsd
 .Spotlight-V100
@@ -17,36 +11,53 @@ Icon
 .Trashes
 .VolumeIcon.icns
 .com.apple.timemachine.donotpresent
 # Directories potentially created on remote AFP share
 .AppleDB
 .AppleDesktop
 Network Trash Folder
 Temporary Items
 .apdisk
 # VS Code
 .vscode/
 # macOS metadata
 *.icloud
-# Terraform
+# IDE
-terraform/
+.vscode/
 .idea/
 .kiro/
 # Terraform (track .tf files, ignore state and runtime)
 .terraform/
 *.tfstate
 *.tfstate.backup
-
+# *.tfvars
-# Crash log files
+!*.tfvars.example
 !*.pkrvars.hcl
 crash.log
 # Ignore override files as they are usually used to override resources locally
 *.override.tf
 *.override.tf.json
 # Ignore CLI configuration files
 .terraformrc
-.terraformrc.json
+terraform.rc
-# Ignore secrets
+# Ansible
 *.retry
 ansible/vault/*.vault
 # Packer
 packer_cache/
 # Secrets
 *.pem
 *.key
 # Environment files with secrets
 boilerplates/**/.env
 # Local paths
 ~/
 # Misc
 doppler-token.yaml
 # talhelper generated machine configs — contain secrets, never commit
 talos/talhelper/clusterconfig/
 # talenv.yaml plaintext — only commit the SOPS-encrypted version
 talos/talhelper/talenv.yaml
 !talos/talhelper/talenv.sops.yaml
@@ -1,209 +0,0 @@
 # Design Document
 ## Overview
 The AppleScript Calendar Sync system is designed as a standalone AppleScript application that synchronizes calendar events between two calendar accounts for the current day only. The system uses macOS Calendar app's AppleScript interface to read events from a source calendar and mirror them to a destination calendar, including removal of events that no longer exist in the source.
 ## Architecture
 The system follows a simple pipeline architecture:
 ```
 [Source Calendar] → [Event Reader] → [Event Processor] → [Destination Calendar]
                                          ↓
                                   [Duplicate Detector]
                                          ↓
                                   [Cleanup Manager]
 ```
 ### Core Components
 1. **Calendar Manager**: Handles calendar account and calendar selection/validation
 2. **Event Reader**: Retrieves events from source calendar for current day
 3. **Event Processor**: Processes and transforms events for destination calendar
 4. **Sync Engine**: Coordinates the synchronization process including cleanup
 5. **Logger**: Provides user feedback and error reporting
 ## Components and Interfaces
 ### Calendar Manager
 ```applescript
 -- Validates and retrieves calendar references
 on getCalendar(accountName, calendarName)
 on validateCalendarAccess(calendar)
 on listAvailableCalendars()
 ```
 **Responsibilities:**
 - Validate calendar account and calendar names exist
 - Return calendar object references for AppleScript operations
 - Handle calendar access permissions and errors
 ### Event Reader
 ```applescript
 -- Reads events from source calendar for current day
 on getEventsForToday(sourceCalendar)
 on parseEventProperties(event)
 ```
 **Responsibilities:**
 - Query source calendar for events occurring on current date
 - Extract event properties (title, start time, end time, description, etc.)
 - Handle different event types (all-day, timed, recurring)
 ### Event Processor
 ```applescript
 -- Processes events for destination calendar
 on createEventInDestination(eventData, destinationCalendar)
 on updateExistingEvent(existingEvent, newEventData)
 on compareEvents(event1, event2)
 ```
 **Responsibilities:**
 - Create new events in destination calendar
 - Update modified events
 - Compare events for duplicate detection
 ### Sync Engine
 ```applescript
 -- Main synchronization coordinator
 on performSync(sourceCalendar, destinationCalendar)
 on cleanupRemovedEvents(sourceEvents, destinationEvents, destinationCalendar)
 on generateSyncReport(results)
 ```
 **Responsibilities:**
 - Coordinate the entire sync process
 - Manage event cleanup (removal of events not in source)
 - Generate sync reports and statistics
 ### Logger
 ```applescript
 -- Logging and user feedback
 on logMessage(message, level)
 on displayProgress(current, total)
 on showSyncSummary(summary)
 ```
 **Responsibilities:**
 - Display progress information to user
 - Log errors and warnings
 - Show final sync summary
 ## Data Models
 ### Event Data Structure
 ```applescript
 record EventData
    title: string
    startDate: date
    endDate: date
    isAllDay: boolean
    description: string
    location: string
    uid: string (for duplicate detection)
 end record
 ```
 ### Sync Result Structure
 ```applescript
 record SyncResult
    eventsCreated: integer
    eventsUpdated: integer
    eventsRemoved: integer
    eventsSkipped: integer
    errors: list of strings
 end record
 ```
 ## Error Handling
 ### Error Categories
 1. **Calendar Access Errors**: Invalid calendar names, permission issues
 2. **Event Processing Errors**: Malformed events, property access failures
 3. **Sync Operation Errors**: Network issues, calendar service unavailable
 ### Error Handling Strategy
 - Graceful degradation: Continue processing other events when individual events fail
 - Detailed error logging with specific error messages
 - User-friendly error reporting with suggested solutions
 - Rollback capability for critical failures
 ### Error Recovery
 ```applescript
 on handleCalendarError(errorMessage)
    -- Log error details
    -- Provide user-friendly error message
    -- Suggest corrective actions
 end handleCalendarError
 ```
 ## Testing Strategy
 ### Unit Testing Approach
 Since AppleScript has limited testing frameworks, testing will focus on:
 1. **Manual Testing Scenarios**:
   - Test with empty source calendar
   - Test with events spanning multiple days
   - Test with all-day events
   - Test with recurring events
   - Test calendar access errors
 2. **Integration Testing**:
   - Test full sync workflow with real calendar data
   - Test cleanup functionality (event removal)
   - Test duplicate detection accuracy
   - Test error handling with invalid inputs
 3. **Edge Case Testing**:
   - Very long event titles and descriptions
   - Events with special characters
   - Overlapping events
   - Events created/modified during sync
 ### Test Data Requirements
 - Test calendars with known event sets
 - Events with various properties (all-day, timed, recurring)
 - Events with special characters and long descriptions
 - Calendar accounts with different permission levels
 ## Implementation Considerations
 ### AppleScript Calendar Integration
 - Use `Calendar` application's AppleScript dictionary
 - Handle calendar app launch and focus management
 - Manage calendar selection and event creation timing
 ### Performance Optimization
 - Batch event operations where possible
 - Minimize calendar app UI interactions
 - Cache calendar references to avoid repeated lookups
 ### User Experience
 - Provide clear progress indicators
 - Show meaningful error messages
 - Allow user to cancel long-running operations
 - Display comprehensive sync results
 ### Security and Privacy
 - Request calendar access permissions appropriately
 - Handle sensitive calendar data securely
 - Provide clear information about what data is accessed
 ## Configuration Management
 ### User Configuration
 ```applescript
 -- Configuration properties
 property sourceAccountName : "Work Account"
 property sourceCalendarName : "Main Calendar"
 property destinationAccountName : "Personal Account"
 property destinationCalendarName : "Synced Events"
 property enableLogging : true
 ```
 ### Runtime Configuration
 - Allow users to modify calendar names without editing script
 - Provide configuration validation before sync starts
 - Save user preferences for repeated use
@@ -1,73 +0,0 @@
 # Requirements Document
 ## Introduction
 This feature enables automatic synchronization of calendar entries between two different calendar accounts using AppleScript. The system will copy events from a source calendar account to a destination calendar account, maintaining event details while avoiding duplicates and providing configurable sync options.
 ## Requirements
 ### Requirement 1
 **User Story:** As a user with multiple calendar accounts, I want to sync events from one account to another, so that I can maintain consistent scheduling across different calendar systems.
 #### Acceptance Criteria
 1. WHEN the sync script is executed THEN the system SHALL read all events from the specified source calendar
 2. WHEN events are found in the source calendar THEN the system SHALL copy them to the specified destination calendar
 3. WHEN copying events THEN the system SHALL preserve event title, date, time, duration, and description
 4. IF an event already exists in the destination calendar THEN the system SHALL skip creating a duplicate
 ### Requirement 2
 **User Story:** As a user, I want to configure which calendars to sync between, so that I can control the data flow between my accounts.
 #### Acceptance Criteria
 1. WHEN configuring the sync THEN the system SHALL allow selection of source calendar account and specific calendar
 2. WHEN configuring the sync THEN the system SHALL allow selection of destination calendar account and specific calendar
 3. WHEN invalid calendar names are provided THEN the system SHALL display an error message and exit gracefully
 4. WHEN calendar accounts are not accessible THEN the system SHALL provide clear error messaging
 ### Requirement 3
 **User Story:** As a user, I want the sync to focus on today's events only, so that I maintain current day synchronization without overwhelming the destination calendar.
 #### Acceptance Criteria
 1. WHEN the sync runs THEN the system SHALL only process events occurring on the current day
 2. WHEN determining current day THEN the system SHALL use the local system date
 3. WHEN events span multiple days THEN the system SHALL include events that start or occur on the current day
 4. WHEN no events exist for the current day THEN the system SHALL complete successfully with appropriate messaging
 ### Requirement 4
 **User Story:** As a user, I want the destination calendar to mirror the source calendar for the current day, so that removed events are also cleaned up automatically.
 #### Acceptance Criteria
 1. WHEN checking for duplicates THEN the system SHALL compare event title, start date, and start time
 2. WHEN a matching event is found in the destination THEN the system SHALL skip creating the duplicate
 3. WHEN an event exists in the destination but not in the source for the current day THEN the system SHALL remove it from the destination
 4. WHEN an event has been modified in the source THEN the system SHALL update the corresponding event in the destination
 ### Requirement 5
 **User Story:** As a user, I want to see progress and results of the sync operation, so that I can verify the synchronization was successful.
 #### Acceptance Criteria
 1. WHEN the sync starts THEN the system SHALL display the source and destination calendar information
 2. WHEN processing events THEN the system SHALL show progress indicators for each event processed
 3. WHEN the sync completes THEN the system SHALL display a summary of events copied, skipped, and any errors
 4. WHEN errors occur THEN the system SHALL log detailed error information for troubleshooting
 ### Requirement 6
 **User Story:** As a user, I want the sync to handle different event types and properties, so that all my calendar data is accurately transferred.
 #### Acceptance Criteria
 1. WHEN syncing events THEN the system SHALL handle all-day events correctly
 2. WHEN syncing events THEN the system SHALL preserve recurring event patterns when possible
 3. WHEN syncing events THEN the system SHALL handle events with attendees and meeting details
 4. WHEN event properties cannot be transferred THEN the system SHALL log which properties were skipped
@@ -1,125 +0,0 @@
 # Implementation Plan
 - [x] 1. Set up project structure and configuration
  - Create main AppleScript file with basic structure and configuration properties
  - Define configuration properties for source and destination calendars
  - Set up logging and error handling framework
  - _Requirements: 2.1, 2.2, 5.1_
 - [ ] 2. Implement Calendar Manager component
  - [ ] 2.1 Create calendar validation and access functions
    - Write functions to validate calendar account and calendar names exist
    - Implement calendar object retrieval with error handling
    - Create function to list available calendars for debugging
    - _Requirements: 2.1, 2.2, 2.3, 2.4_
  - [ ]* 2.2 Write unit tests for calendar access
    - Create test scenarios for invalid calendar names
    - Test calendar access permission handling
    - _Requirements: 2.3, 2.4_
 - [ ] 3. Implement Event Reader component
  - [ ] 3.1 Create current day event retrieval function
    - Write function to get today's date and create date range
    - Implement event query for current day from source calendar
    - Handle different event types (all-day, timed events)
    - _Requirements: 1.1, 3.1, 3.2, 3.3_
  - [ ] 3.2 Implement event property extraction
    - Create function to extract event title, dates, description, location
    - Handle event property access errors gracefully
    - Parse recurring events for current day occurrences
    - _Requirements: 1.3, 6.1, 6.2, 6.3_
  - [ ]* 3.3 Write tests for event reading functionality
    - Test event retrieval with various event types
    - Test property extraction accuracy
    - _Requirements: 1.1, 1.3_
 - [ ] 4. Implement Event Processor component
  - [ ] 4.1 Create event comparison and duplicate detection
    - Write function to compare events by title, start date, and start time
    - Implement duplicate detection logic for existing events
    - Handle event matching edge cases
    - _Requirements: 4.1, 4.2_
  - [ ] 4.2 Implement event creation and update functions
    - Create function to add new events to destination calendar
    - Implement event update functionality for modified events
    - Handle event creation errors and property limitations
    - _Requirements: 1.2, 1.3, 4.4, 6.4_
  - [ ]* 4.3 Write tests for event processing
    - Test duplicate detection accuracy
    - Test event creation with various properties
    - _Requirements: 4.1, 4.2, 4.4_
 - [ ] 5. Implement Sync Engine component
  - [ ] 5.1 Create main synchronization workflow
    - Implement the main sync function that coordinates all components
    - Add progress tracking and user feedback during sync
    - Handle sync workflow errors and recovery
    - _Requirements: 1.1, 1.2, 5.2, 5.3_
  - [ ] 5.2 Implement cleanup functionality for removed events
    - Create function to identify events in destination not in source
    - Implement event removal from destination calendar
    - Add safety checks to prevent accidental deletions
    - _Requirements: 4.3_
  - [ ] 5.3 Create sync reporting and statistics
    - Implement sync result tracking (created, updated, removed, skipped)
    - Create summary display function with detailed results
    - Add error reporting and logging
    - _Requirements: 5.3, 5.4_
  - [ ]* 5.4 Write integration tests for sync engine
    - Test complete sync workflow with test data
    - Test cleanup functionality accuracy
    - Test error handling and recovery
    - _Requirements: 4.3, 5.3_
 - [ ] 6. Implement Logger component
  - [ ] 6.1 Create logging and progress display functions
    - Implement message logging with different severity levels
    - Create progress indicator for sync operations
    - Add user-friendly error message formatting
    - _Requirements: 5.1, 5.2, 5.4_
  - [ ] 6.2 Implement sync summary display
    - Create formatted summary of sync results
    - Display statistics for events processed
    - Show any errors or warnings encountered
    - _Requirements: 5.3, 5.4_
 - [ ] 7. Integrate all components and create main script
  - [ ] 7.1 Wire together all components in main execution flow
    - Create main script entry point that calls all components
    - Implement proper error handling and user feedback flow
    - Add configuration validation before sync starts
    - _Requirements: 2.3, 2.4, 5.1_
  - [ ] 7.2 Add user interaction and configuration management
    - Implement user prompts for calendar selection if needed
    - Add configuration validation and error messaging
    - Create user-friendly script execution experience
    - _Requirements: 2.1, 2.2, 2.3, 2.4_
  - [ ]* 7.3 Create comprehensive end-to-end tests
    - Test complete sync workflow with real calendar data
    - Test all error scenarios and edge cases
    - Validate sync accuracy and cleanup functionality
    - _Requirements: 1.1, 1.2, 4.3, 5.3_
 - [ ] 8. Finalize and optimize the script
  - [ ] 8.1 Add performance optimizations and error recovery
    - Optimize calendar access and event processing performance
    - Add robust error recovery and rollback capabilities
    - Implement proper resource cleanup and calendar app management
    - _Requirements: 5.4_
  - [ ] 8.2 Create user documentation and usage instructions
    - Write clear instructions for script configuration and usage
    - Document calendar permission requirements
    - Create troubleshooting guide for common issues
    - _Requirements: 2.3, 2.4, 5.4_
@@ -1,165 +0,0 @@
 # Design Document
 ## Overview
 The `fastpass-additional-control-plane` Ansible role will enable the deployment of additional control plane nodes to an existing FastPass Kubernetes cluster. This role follows the established patterns from `fastpass-first-control-plane` but focuses on joining nodes to an already initialized cluster rather than initializing a new one. The role ensures high availability by creating redundant master nodes that can handle API requests, scheduling, and cluster management tasks.
 The key difference from the first control plane role is that this role will use `kubeadm join` with control plane flags instead of `kubeadm init`, and it will need to retrieve join tokens and certificate keys from the existing cluster.
 ## Architecture
 ### Role Structure
 The role will follow the standard Ansible role structure:
 ```
 ansible/playbooks/roles/fastpass-additional-control-plane/
 ├── defaults/
 │   └── main.yml
 ├── tasks/
 │   └── main.yml
 ├── handlers/
 │   └── main.yml (if needed)
 └── meta/
    └── main.yml (if needed)
 ```
 ### Integration Points
 - **DNS Management**: Uses the existing `dns-manager` role for consistent DNS record creation
 - **Kubeconfig Management**: Uses the existing `kubeconfig-manager` role for local kubeconfig setup
 - **Firewall Configuration**: Reuses firewall service definitions from the first control plane role
 - **Cluster Integration**: Coordinates with the first control plane node to obtain join credentials
 ### Dependencies
 - The first control plane node must be fully initialized and running
 - The `dns-manager` role must be available for DNS record creation
 - The `kubeconfig-manager` role must be available for kubeconfig setup
 - Required Kubernetes prerequisites must be installed on target nodes
 ## Components and Interfaces
 ### Main Task Flow
 1. **Pre-flight Checks**: Verify cluster readiness and node prerequisites
 2. **DNS Configuration**: Set up DNS records for the new control plane node
 3. **Firewall Configuration**: Open required ports for control plane services
 4. **Kubelet Configuration**: Create initial kubelet configuration
 5. **Join Token Retrieval**: Get join token and certificate key from first control plane
 6. **Cluster Join**: Execute kubeadm join with control plane flags
 7. **Service Management**: Ensure kubelet is enabled and running
 8. **Kubeconfig Setup**: Configure local kubeconfig access
 9. **Verification**: Validate successful cluster join
 ### Key Variables
 - `cluster_name`: Name of the Kubernetes cluster
 - `ip_address`: IP address of the current control plane node
 - `first_control_plane_host`: Hostname/IP of the first control plane node
 - `kubernetes_services_control_plane`: List of firewall services to open
 - `join_token_ttl`: TTL for join tokens (default: 24h)
 - `certificate_key_ttl`: TTL for certificate keys (default: 2h)
 ### External Role Interfaces
 - **dns-manager**: Provides DNS record creation with `host_name` variable
 - **kubeconfig-manager**: Handles kubeconfig merging with `cluster_name` variable
 - **First Control Plane**: Source for join tokens and certificate keys
 ## Data Models
 ### Join Credentials Structure
 ```yaml
 join_credentials:
  token: "abcdef.1234567890abcdef"
  discovery_token_ca_cert_hash: "sha256:..."
  certificate_key: "..."
  api_server_endpoint: "cluster-name:6443"
 ```
 ### Firewall Services
 ```yaml
 kubernetes_services_control_plane:
  - kubernetes_API      # Port 6443
  - etcd               # Ports 2379-2380
  - kubelet            # Port 10250
  - kube-scheduler     # Port 10259
  - kube-controller-manager  # Port 10257
 ```
 ### Node Status Tracking
 ```yaml
 node_status:
  joined: false
  kubelet_running: false
  dns_configured: false
  kubeconfig_ready: false
 ```
 ## Error Handling
 ### Join Token Management
 - **Token Expiration**: Automatically generate new tokens if existing ones are expired
 - **Certificate Key Rotation**: Handle certificate key expiration gracefully
 - **Network Connectivity**: Retry join operations with exponential backoff
 - **API Server Availability**: Wait for API server readiness before attempting join
 ### Idempotency Checks
 - **Already Joined Nodes**: Skip join process if node is already part of the cluster
 - **Existing Configuration**: Preserve existing kubelet configuration if valid
 - **DNS Records**: Update existing DNS records instead of creating duplicates
 - **Service Status**: Only restart services if configuration changes
 ### Failure Recovery
 - **Partial Join Failures**: Clean up partial configurations and retry
 - **Network Issues**: Provide clear error messages for connectivity problems
 - **Permission Errors**: Validate sudo/root access before attempting operations
 - **Resource Constraints**: Check system resources before proceeding
 ## Testing Strategy
 ### Unit Testing Approach
 - **Task Validation**: Test individual tasks with mock data
 - **Variable Validation**: Ensure required variables are properly defined
 - **Conditional Logic**: Test all conditional branches in tasks
 - **Error Scenarios**: Validate error handling for common failure cases
 ### Integration Testing
 - **Multi-Node Clusters**: Test with 3 and 5 control plane node configurations
 - **Network Scenarios**: Test across different network topologies
 - **OS Compatibility**: Validate on supported operating systems (Ubuntu/Debian)
 - **Version Compatibility**: Test with different Kubernetes versions
 ### Validation Checks
 - **Cluster Health**: Verify all control plane nodes are healthy after join
 - **API Availability**: Confirm API server is accessible from all nodes
 - **Etcd Cluster**: Validate etcd cluster membership and health
 - **Scheduling**: Test pod scheduling across all control plane nodes
 - **Failover**: Verify cluster continues operating if one control plane fails
 ### Test Scenarios
 1. **Fresh Join**: Join additional control plane to newly created cluster
 2. **Existing Cluster**: Add control plane to cluster with existing workloads
 3. **Network Partitions**: Test behavior during temporary network issues
 4. **Token Expiration**: Handle expired join tokens gracefully
 5. **Retry Operations**: Validate retry logic for transient failures
 ## Implementation Considerations
 ### Security
 - **Token Security**: Ensure join tokens are handled securely and not logged
 - **Certificate Management**: Properly manage and rotate certificate keys
 - **Network Security**: Validate firewall rules don't expose unnecessary ports
 - **Access Control**: Ensure proper RBAC is maintained after node joins
 ### Performance
 - **Parallel Execution**: Support joining multiple control plane nodes simultaneously
 - **Resource Usage**: Monitor CPU and memory usage during join process
 - **Network Bandwidth**: Optimize data transfer during cluster join
 - **Startup Time**: Minimize time to achieve cluster readiness
 ### Monitoring and Observability
 - **Join Progress**: Provide clear progress indicators during join process
 - **Health Checks**: Implement comprehensive health validation
 - **Logging**: Ensure adequate logging for troubleshooting
 - **Metrics**: Expose relevant metrics for monitoring cluster growth
 ### Compatibility
 - **Kubernetes Versions**: Support current and previous Kubernetes versions
 - **Operating Systems**: Maintain compatibility with Ubuntu and Debian
 - **Container Runtimes**: Work with containerd runtime configuration
 - **Network Plugins**: Compatible with Flannel CNI configuration
@@ -1,77 +0,0 @@
 # Requirements Document
 ## Introduction
 This feature involves creating an Ansible role called `fastpass-additional-control-plane` that will deploy additional control plane nodes to an existing FastPass Kubernetes cluster. The role will follow the same pattern as the existing `fastpass-first-control-plane` role but will focus on joining nodes to an already initialized cluster rather than initializing a new cluster. This ensures high availability for the Kubernetes control plane by adding redundant master nodes.
 ## Requirements
 ### Requirement 1
 **User Story:** As a DevOps engineer, I want to deploy additional control plane nodes to my FastPass Kubernetes cluster, so that I can achieve high availability and fault tolerance for the cluster control plane.
 #### Acceptance Criteria
 1. WHEN the role is executed on a node THEN the system SHALL join the node to the existing Kubernetes cluster as a control plane node
 2. WHEN the role runs THEN the system SHALL configure the necessary firewall rules for control plane services
 3. WHEN the role executes THEN the system SHALL ensure the kubelet service is properly configured and running
 4. WHEN joining the cluster THEN the system SHALL use the correct join token and certificate key from the first control plane node
 5. WHEN the role completes THEN the system SHALL verify the node has successfully joined as a control plane node
 ### Requirement 2
 **User Story:** As a system administrator, I want the additional control plane role to follow the same patterns as the first control plane role, so that the codebase remains consistent and maintainable.
 #### Acceptance Criteria
 1. WHEN the role is created THEN the system SHALL follow the same directory structure as fastpass-first-control-plane
 2. WHEN the role is implemented THEN the system SHALL use similar variable naming conventions and task organization
 3. WHEN the role runs THEN the system SHALL include proper error handling and idempotency checks
 4. WHEN the role executes THEN the system SHALL use the same firewall service definitions as the first control plane role
 5. WHEN the role is documented THEN the system SHALL include proper metadata headers with author, version, and description
 ### Requirement 3
 **User Story:** As a cluster operator, I want the additional control plane nodes to have proper DNS configuration, so that they can be reached by their cluster names and participate in load balancing.
 #### Acceptance Criteria
 1. WHEN the role runs THEN the system SHALL configure DNS records for the additional control plane nodes
 2. WHEN DNS is configured THEN the system SHALL use the dns-manager role for consistency
 3. WHEN the role executes THEN the system SHALL ensure the node can resolve the cluster endpoint
 4. WHEN DNS setup completes THEN the system SHALL verify connectivity to the cluster API endpoint
 ### Requirement 4
 **User Story:** As a DevOps engineer, I want the role to handle kubeconfig management for additional control plane nodes, so that I can manage the cluster from any control plane node.
 #### Acceptance Criteria
 1. WHEN the role completes THEN the system SHALL configure kubeconfig for the new control plane node
 2. WHEN kubeconfig is set up THEN the system SHALL use the kubeconfig-manager role for consistency
 3. WHEN the role runs THEN the system SHALL ensure proper permissions are set on kubeconfig files
 4. WHEN kubeconfig is configured THEN the system SHALL verify kubectl access works from the new node
 ### Requirement 5
 **User Story:** As a system administrator, I want the role to be idempotent and handle edge cases, so that I can run it multiple times safely without causing issues.
 #### Acceptance Criteria
 1. WHEN the role is run multiple times THEN the system SHALL not attempt to rejoin an already joined node
 2. WHEN a node is already part of the cluster THEN the system SHALL skip the join process gracefully
 3. WHEN the role encounters errors THEN the system SHALL provide clear error messages and fail gracefully
 4. WHEN prerequisites are missing THEN the system SHALL report what needs to be configured first
 5. WHEN the role runs THEN the system SHALL validate that required variables are defined
 ### Requirement 6
 **User Story:** As a cluster administrator, I want the role to integrate seamlessly with the existing FastPass deployment workflow, so that it can be used in the 4-step deployment process.
 #### Acceptance Criteria
 1. WHEN the role is created THEN the system SHALL be compatible with the deploy-fastpass-4step.yml playbook
 2. WHEN the role runs THEN the system SHALL work with the fastpass_control_plane[1:] host group
 3. WHEN integrated THEN the system SHALL not interfere with the first control plane initialization
 4. WHEN the role executes THEN the system SHALL depend on the first control plane node being ready
 5. WHEN deployment completes THEN the system SHALL allow worker nodes to join the cluster successfully
@@ -1,118 +0,0 @@
 # Implementation Plan
 - [ ] 1. Create role directory structure and basic configuration
  - Create the fastpass-additional-control-plane role directory structure
  - Set up defaults/main.yml with required variables and firewall services
  - Create meta/main.yml with role metadata and dependencies
  - _Requirements: 2.1, 2.2, 2.4_
 - [ ] 2. Implement join token and certificate key retrieval
  - [ ] 2.1 Create tasks to generate new join tokens from first control plane
    - Write Ansible tasks to execute kubeadm token create on first control plane node
    - Implement token validation and expiration checking
    - Add error handling for token generation failures
    - _Requirements: 1.4, 5.4_
  - [ ] 2.2 Implement certificate key retrieval and management
    - Create tasks to upload and retrieve certificate keys from first control plane
    - Add certificate key expiration handling and rotation
    - Implement secure handling of certificate keys in variables
    - _Requirements: 1.4, 5.1_
  - [ ] 2.3 Create discovery token CA certificate hash retrieval
    - Write tasks to extract CA certificate hash from first control plane
    - Implement validation of certificate hash format
    - Add error handling for certificate retrieval failures
    - _Requirements: 1.4, 5.4_
 - [ ] 3. Implement DNS configuration and firewall setup
  - [ ] 3.1 Configure DNS records for additional control plane nodes
    - Integrate dns-manager role for consistent DNS record creation
    - Pass appropriate host_name variable to dns-manager
    - Add DNS propagation wait and validation
    - _Requirements: 3.1, 3.2, 3.4_
  - [ ] 3.2 Set up firewall rules for control plane services
    - Reuse kubernetes_services_control_plane from defaults
    - Implement UFW firewall rule creation for Debian/Ubuntu systems
    - Add conditional logic for different operating systems
    - _Requirements: 1.2, 2.4_
 - [ ] 4. Implement kubelet configuration and cluster join
  - [ ] 4.1 Create initial kubelet configuration
    - Write kubelet config.yaml with systemd cgroup driver
    - Set containerd socket endpoint configuration
    - Ensure proper file permissions and ownership
    - _Requirements: 1.3, 2.3_
  - [ ] 4.2 Execute kubeadm join for control plane
    - Implement kubeadm join command with control-plane flag
    - Use retrieved join token, certificate key, and CA cert hash
    - Add proper command argument construction and validation
    - Include idempotency checks to prevent duplicate joins
    - _Requirements: 1.1, 1.4, 5.1, 5.2_
  - [ ] 4.3 Ensure kubelet service management
    - Enable and start kubelet systemd service
    - Add service status validation and error handling
    - Implement service restart logic if needed
    - _Requirements: 1.3, 1.5_
 - [ ] 5. Implement kubeconfig management and validation
  - [ ] 5.1 Configure kubeconfig for additional control plane nodes
    - Integrate kubeconfig-manager role for consistent configuration
    - Pass cluster_name variable to kubeconfig-manager
    - Ensure proper kubeconfig merging with existing configurations
    - _Requirements: 4.1, 4.2, 4.3_
  - [ ] 5.2 Implement cluster join validation
    - Create tasks to verify node successfully joined as control plane
    - Add kubectl commands to check node status and roles
    - Implement cluster health validation checks
    - _Requirements: 1.5, 4.4_
 - [ ] 6. Add comprehensive error handling and idempotency
  - [ ] 6.1 Implement pre-flight validation checks
    - Check if node is already joined to cluster
    - Validate required variables are defined
    - Verify first control plane node accessibility
    - Add system resource and prerequisite checks
    - _Requirements: 5.1, 5.2, 5.4, 5.5_
  - [ ] 6.2 Add retry logic and failure recovery
    - Implement retry mechanisms for transient failures
    - Add exponential backoff for network-related operations
    - Create cleanup tasks for partial join failures
    - _Requirements: 5.3, 5.4_
 - [ ] 7. Integration with FastPass deployment workflow
  - [ ] 7.1 Ensure compatibility with deploy-fastpass-4step.yml
    - Verify role works with fastpass_control_plane[1:] host group
    - Test integration with existing playbook structure
    - Validate dependency on first control plane completion
    - _Requirements: 6.1, 6.2, 6.3, 6.4_
  - [ ] 7.2 Add proper task documentation and metadata
    - Include role header with author, version, and description
    - Add inline comments for complex task logic
    - Document required variables and their purposes
    - _Requirements: 2.2, 2.5_
 - [ ]* 8. Create comprehensive testing and validation
  - [ ]* 8.1 Write unit tests for individual tasks
    - Create test cases for token retrieval logic
    - Test kubeadm join command construction
    - Validate error handling scenarios
    - _Requirements: 1.1, 1.4, 5.1_
  - [ ]* 8.2 Implement integration tests
    - Test multi-node control plane deployment
    - Validate cluster health after additional nodes join
    - Test failover scenarios and cluster resilience
    - _Requirements: 1.5, 6.5_
  - [ ]* 8.3 Add validation scripts and health checks
    - Create scripts to verify cluster state after deployment
    - Implement automated health validation
    - Add performance and resource usage monitoring
    - _Requirements: 1.5, 4.4_
@@ -1,51 +0,0 @@
 # Requirements Document
 ## Introduction
 The FastPass Kubernetes cluster deployment currently has critical DNS and load balancing configuration issues that prevent proper high availability setup. While the cluster endpoint `fastpass.local.mk-labs.cloud` is defined in the group variables, the `cluster_vip` variable required by the DNS manager role is missing, and the Traefik load balancer configuration is commented out. This means the cluster endpoint cannot resolve properly and there's no load balancing for the control plane API. This feature will fix these configuration gaps to enable true HA functionality.
 ## Requirements
 ### Requirement 1: DNS CNAME Record for Load Balancer
 **User Story:** As a DevOps engineer, I want the DNS manager to create a CNAME record for the cluster endpoint pointing to the load balancer, so that the cluster endpoint resolves through the load balancer rather than directly to node IPs.
 #### Acceptance Criteria
 1. WHEN the dns-manager role is called for a load-balanced cluster THEN the system SHALL create a CNAME record instead of an A record
 2. WHEN the CNAME record is created THEN the system SHALL point fastpass.local.mk-labs.cloud to the traefik_server (lightning_lane.local.mk-labs.cloud)
 3. WHEN the DNS record type is determined THEN the system SHALL use CNAME for load-balanced endpoints and A records for direct node access
 4. WHEN DNS propagation occurs THEN the system SHALL verify that the CNAME resolution works correctly
 ### Requirement 2: Complete Traefik Integration
 **User Story:** As a DevOps engineer, I want the Traefik load balancer to be fully integrated with the FastPass deployment, so that the cluster VIP is properly load balanced across all control plane nodes.
 #### Acceptance Criteria
 1. WHEN the traefik-manager role is called THEN the system SHALL use the traefik_server variable (lightning_lane) as the target host
 2. WHEN Traefik configuration is generated THEN the system SHALL create proper TCP routing for the cluster endpoint to all control plane nodes
 3. WHEN the cluster VIP is accessed THEN the system SHALL distribute requests across space-mountain, big-thunder-mountain, and splash-mountain
 4. WHEN Traefik configuration is applied THEN the system SHALL reload the Traefik service to activate the new configuration
 ### Requirement 3: High Availability Validation
 **User Story:** As a DevOps engineer, I want to validate that the HA setup is working correctly, so that I can be confident the cluster will survive node failures.
 #### Acceptance Criteria
 1. WHEN the deployment completes THEN the system SHALL test connectivity to the cluster endpoint
 2. WHEN connectivity tests run THEN the system SHALL verify that the endpoint resolves through the CNAME to the load balancer
 3. WHEN load balancer tests run THEN the system SHALL verify that requests are being distributed across control plane nodes
 4. WHEN a control plane node is stopped THEN the system SHALL continue to serve API requests through the remaining nodes
 ### Requirement 4: Backward Compatibility
 **User Story:** As a DevOps engineer, I want the DNS fixes to be backward compatible with existing deployments, so that current clusters continue to function during the transition.
 #### Acceptance Criteria
 1. WHEN existing clusters are updated THEN the system SHALL not break existing DNS configurations
 2. WHEN new variables are introduced THEN the system SHALL provide sensible defaults for existing deployments
 3. WHEN the update is applied THEN the system SHALL preserve existing kubeconfig files and cluster access
 4. IF migration issues occur THEN the system SHALL provide rollback procedures and documentation
@@ -0,0 +1,3 @@
 creation_rules:
  - path_regex: talos/talhelper/talenv.yaml
    age: age1xkyuv8r8ce6lu3d64jfspz4e50k6pxlaxwmuplzrtnpfnnrnycaq6mfsrn
@@ -0,0 +1,160 @@
 # CouchDB erlangCookie Fix - Implementation Guide
 ## Summary
 **Problem**: CouchDB deployment fails because `erlangCookie` is missing from the ExternalSecret configuration.
 **Decision**: Externalize `erlangCookie` to 1Password (pragmatic approach)
 **Rationale**: 
 - ExternalSecret architecture requires ownership of the entire secret
 - Mixing externalized and chart-generated fields in the same secret is not supported
 - Single-node deployment makes erlangCookie rotation unnecessary
 - This is an acceptable deviation from the pure Harbor pattern given the architectural constraints
 ## Implementation Steps
 ### 1. Generate erlangCookie Value
 ```bash
 openssl rand -hex 20
 ```
 Example output: `f4e3c2b1a9d8e7f6c5b4a3d2e1f0a9b8c7d6e5f4`
 ### 2. Add to 1Password
 - **Vault**: `mk-labs`
 - **Item**: `couchdb`
 - **Field Name**: `erlang-cookie`
 - **Field Type**: password (concealed)
 - **Value**: `<paste generated value from step 1>`
 ### 3. Update ExternalSecret Configuration
 File: `cluster/applications/couchdb/externalsecret.yaml`
 ```yaml
 apiVersion: external-secrets.io/v1beta1
 kind: ExternalSecret
 metadata:
  name: couchdb-credentials
  namespace: couchdb
  labels:
    app.kubernetes.io/name: couchdb
    app.kubernetes.io/part-of: mk-labs
 spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: onepassword-connect
  target:
    name: couchdb-admin
    creationPolicy: Owner
    template:
      engineVersion: v2
      data:
        adminUsername: "admin"
        adminPassword: "{{ .adminPassword }}"
        cookieAuthSecret: "{{ .cookieAuthSecret }}"
        erlangCookie: "{{ .erlangCookie }}"  # ← ADD THIS LINE
  data:
    - secretKey: adminPassword
      remoteRef:
        key: couchdb
        property: admin-password
    - secretKey: cookieAuthSecret
      remoteRef:
        key: couchdb
        property: cookie-auth-secret
    - secretKey: erlangCookie  # ← ADD THIS BLOCK
      remoteRef:
        key: couchdb
        property: erlang-cookie
 ```
 ### 4. Update values.yaml Documentation (Optional)
 File: `cluster/applications/couchdb/values.yaml`
 Update the comment block at line 9-10:
 ```yaml
 # Admin credentials managed via ExternalSecret
 # See externalsecret.yaml for 1Password integration
 #
 # NOTE: erlangCookie is externalized to 1Password for architectural
 # simplicity (ExternalSecret ownership model). In a pure Harbor pattern,
 # this would be chart-generated, but single-node deployment makes this
 # acceptable. The erlangCookie is treated as an immutable infrastructure
 # secret (generate once, never rotate).
 createAdminSecret: false
 extraSecretName: "couchdb-admin"
 ```
 ### 5. Commit and Push
 ```bash
 cd ~/git/homelab
 git add cluster/applications/couchdb/externalsecret.yaml
 git add cluster/applications/couchdb/values.yaml  # if modified
 git commit -m "fix(couchdb): add erlangCookie to ExternalSecret from 1Password"
 git push origin main
 ```
 ### 6. Verify Deployment
 ```bash
 # Watch ExternalSecret sync
 kubectl get externalsecret -n couchdb couchdb-credentials -w
 # Wait for: SecretSynced
 # Verify secret created with all four keys
 kubectl get secret -n couchdb couchdb-admin -o yaml
 # Should contain: adminUsername, adminPassword, cookieAuthSecret, erlangCookie
 # Watch ArgoCD sync
 kubectl get application -n argocd couchdb -w
 # Wait for: Healthy/Synced
 # Watch pod startup
 kubectl get pods -n couchdb -w
 # Wait for: Running
 # Test CouchDB access
 kubectl port-forward -n couchdb svc/couchdb-svc-couchdb 5984:5984 &
 curl http://localhost:5984/
 # Expected: {"couchdb":"Welcome","version":"3.5.1"}
 ```
 ## Why Not Follow Harbor Pattern Exactly?
 **Harbor Pattern**: Only user-facing credentials externalized, internal secrets chart-generated.
 **CouchDB Constraint**: ExternalSecret uses `creationPolicy: Owner`, which takes full ownership of the target secret. This prevents the Helm chart from adding auto-generated fields to the same secret.
 **Options Considered**:
 1. ✅ **Externalize erlangCookie** (SELECTED) - Works with current architecture
 2. ❌ Chart auto-generation - Conflicts with ExternalSecret ownership
 3. ❌ Dual-secret approach - Requires Helm chart customization
 4. ❌ Disable ExternalSecret - Loses 1Password integration for admin password
 **Decision**: Pragmatic approach wins. erlangCookie is treated as an infrastructure secret (generate once, never rotate), which is acceptable for a single-node deployment.
 ## Secret Classification
 | Secret           | Type          | 1Password? | Rationale                          |
 |------------------|---------------|------------|------------------------------------|
 | adminUsername    | User-facing   | No*        | Static value, hardcoded in template |
 | adminPassword    | User-facing   | ✅ YES     | User login credential              |
 | cookieAuthSecret | Gray area     | ✅ YES     | Session security, periodic rotation |
 | erlangCookie     | Internal      | ✅ YES**   | Architectural constraint           |
 \* Hardcoded in ExternalSecret template (not fetched from 1Password)  
 \*\* Pragmatic deviation from Harbor pattern due to ExternalSecret architecture
 ## References
 - Full analysis: `/home/hermes/couchdb-erlangcookie-analysis.txt`
 - Harbor pattern: `/home/hermes/harbor-simplification-complete.txt`
 - CouchDB Helm chart: `apache/couchdb` v4.6.3
@@ -0,0 +1,702 @@
 # Account Dossier Interactive Dashboard — Design Document
 **Version:** 0.1 (MVP)  
 **Author:** Claude  
 **Date:** 2026-10-07  
 **Status:** Design Review (Pre-Prototype)
 ---
 ## 1. Overview & Purpose
 **What:** Interactive HTML/React dashboard that surfaces account intelligence for a single Pure Storage customer or prospect. Provides a "10,000 ft view" with drill-down to ground-level detail.
 **Why:** SEs need to move quickly from account snapshot → discovery prep → conversation context without context-switching between Sheets, notes, and browser tabs. One dashboard per customer becomes the source of reference for that account.
 **MVP Scope:** The 10 essential questions (List 1 + List 2 from account intelligence framework)
 **Future Scope:** Relationships, opportunities, financial modeling, industry trends, sentiment analysis, timeline views
 ---
 ## 2. User Personas & Use Cases
 ### Primary User
 **Ryan** (You, Enterprise SE)
 - Prepping for discovery calls
 - Hunting for competitive displacement opportunities
 - Planning expansion/renewal strategy
 - Sharing account context with team (future)
 ### Use Cases (in priority order, all equally weighted for MVP)
 | Use Case | What They Need | Where Dashboard Helps |
 |----------|---|---|
 | **Pre-call Prep (15 min before discovery)** | Quick account snapshot + key metrics + conversation hooks | Overview pane + drill-down to specific List items; see what changed since last call |
 | **Competitive Displacement Hunting** | Incumbent vendors + footprint + gaps + consolidation leverage | List 2 drill-down showing Pure coverage % + high-value data not on Pure; see EMC systems aging |
 | **Expansion/Renewal Planning** | Growth trajectory + budget cycle + stakeholder sentiment + upcoming initiatives | Opportunity tab (future) + timeline + stakeholder relationships; see refresh window + budget headroom |
 ### Secondary User (Future)
 **Account team** (Sales, Solution Architect, Customer Success)
 - Read-only view of account summary
 - Understand strategic context before customer calls
 ---
 ## 3. Architecture Overview
 ### 3.1 Data Flow Diagram
 ```
 ┌─────────────────────────────────────────────────────────────┐
 │  Per-Customer Data Sources (DRY: No Duplication)            │
 ├─────────────────────────────────────────────────────────────┤
 │                                                               │
 │  📊 Google Sheet (Structured Data)                           │
 │  ├─ Account tab (List 1 metrics)                             │
 │  ├─ Pure-Deployment tab (List 2 metrics)                    │
 │  ├─ Relationships tab (Contacts, sentiment, history)        │
 │  ├─ Interactions tab (Call/email log with sentiment)        │
 │  ├─ Financials tab (ARR, contracts, opportunities)         │
 │  ├─ Industry-Trends tab (Curated insights)                 │
 │  └─ Metadata tab (Sync status, ownership, sharing)         │
 │                                                               │
 │  📝 Obsidian Markdown File (Unstructured Context)           │
 │  ├─ Discovery notes (narrative, conversation context)      │
 │  ├─ Strategy (account themes, expansion hooks)             │
 │  ├─ Relationship notes (person-by-person context)          │
 │  └─ Internal flags (competitive threats, risks)            │
 │                                                               │
 └─────────────────────────────────────────────────────────────┘
                            ↓
                   (via API or manual sync)
                            ↓
 ┌─────────────────────────────────────────────────────────────┐
 │  Interactive Dashboard (HTML/React Artifact)                │
 │  Reads from both sources, displays in 3-pane layout         │
 │  No data copying; all reads from source                     │
 └─────────────────────────────────────────────────────────────┘
                            ↓
                   (displayed to user)
                            ↓
 ┌─────────────────────────────────────────────────────────────┐
 │  User Actions:                                               │
 │  • View account snapshot                                     │
 │  • Drill into List 1 / List 2 metrics                       │
 │  • Check stakeholder sentiment                              │
 │  • Review opportunity pipeline                              │
 │  • Log new interaction                                       │
 │  • Open full Sheets / Obsidian for detailed editing         │
 └─────────────────────────────────────────────────────────────┘
 ```
 ---
 ## 4. UI/UX Design
 ### 4.1 Layout: 3-Pane Architecture
 ```
 ┌──────────────────────────────────────────────────────────────────────┐
 │ HEADER: Account Name | Status | Vertical | Location | [Share] [Edit] │
 ├─────────────────────────────────────────────────────────────────────┤
 │                                                                       │
 │  LEFT PANE              │ CENTER PANE              │ RIGHT PANE      │
 │  (Overview)             │ (Details & Drill-down)   │ (Context)       │
 │                         │                          │                 │
 │  ┌─────────────────┐    │ ┌────────────────────┐   │ ┌────────────┐ │
 │  │ Account Health  │    │ │   Tab Bar:         │   │ │ Latest     │ │
 │  │ Scorecard       │    │ │ Overview | List1   │   │ │ Interactions   │
 │  │ ◆ 4 KPIs        │    │ │ List2    | Oppty   │   │ │            │ │
 │  │   (visual)      │    │ │                    │   │ │ ┌────────┐ │ │
 │  │                 │    │ │ ┌──────────────┐   │   │ │ Sarah   │ │ │
 │  ├─────────────────┤    │ │ │ Item 1 Card  │   │   │ │ Chen    │ │ │
 │  │ Key Dates       │    │ │ │ [Summary]    │   │   │ │ Call    │ │ │
 │  │ Timeline        │    │ │ │ [10K ft]     │   │   │ │ 2d ago  │ │ │
 │  │ • Refresh: 6mo  │    │ │ │ [Status]     │   │   │ │ ⭐⭐⭐  │ │ │
 │  │ • Budget: 3mo   │    │ │ └──────────────┘   │   │ │ (Drill) │ │ │
 │  │ • Contract: 8mo │    │ │                    │   │ │└────────┘ │ │
 │  │                 │    │ │ ┌──────────────┐   │   │ │           │ │
 │  ├─────────────────┤    │ │ │ Item 2 Card  │   │   │ ├──────────┤ │
 │  │ Actions         │    │ │ │ [Summary]    │   │   │ │ [Log new]│ │
 │  │ [QBR Prep]      │    │ │ │ [10K ft]     │   │   │ │           │ │
 │  │ [Edit Account]  │    │ │ │ [Status]     │   │   │ │ Related   │ │
 │  │ [Open Sheets]   │    │ │ └──────────────┘   │   │ │ Notes     │ │
 │  │ [View Notes]    │    │ │                    │   │ │ (from MD) │ │
 │  │ [Share]         │    │ │ .... (3 more)      │   │ │           │ │
 │  │                 │    │ │                    │   │ │ [View all]│ │
 │  └─────────────────┘    │ └────────────────────┘   │ └────────────┘ │
 │                         │                          │                 │
 └──────────────────────────────────────────────────────────────────────┘
 ```
 ### 4.2 Left Pane: Account Overview (10,000 ft)
 **Purpose:** At a glance, is this account healthy? What's the most critical thing to know?
 **Components:**
 #### Health Scorecard (4 KPIs)
 Visual status cards, each with:
 - **Metric name** + current value
 - **Trend indicator** (↑ up, ↓ down, → flat)
 - **Status badge** (green / yellow / red)
 - **Drillable** — click to zoom into that metric
 **KPIs (MVP):**
 1. **Pure Coverage** — % of total storage on Pure
   - Value: 30% | Trend: ↑ +5% YoY | Status: 🟡 Opportunity
   - Rationale: Shows expansion potential; top sales driver
 2. **Fleet Age** — Average age of Pure systems
   - Value: 4.2 yr | Trend: → flat | Status: 🟡 Refresh Window Approaching
   - Rationale: Indicates refresh opportunity timing
 3. **Expansion Opportunity Size** — $ value of consolidation potential
   - Value: $3.5M | Trend: ↑ Growing | Status: 🟢 Pipeline-Ready
   - Rationale: Deal size at a glance; guides priority
 4. **Engagement Momentum** — Recency of last interaction + champion sentiment
   - Value: 5 days | Trend: ↑ Active | Status: 🟢 Engaged
   - Rationale: Is this account warm or cold?
 #### Key Dates Timeline
 Visual timeline of critical dates:
 - **Refresh Date**: Next Pure system refresh
 - **Budget Cycle**: When capex decisions happen
 - **Contract Renewal**: When Pure contract is up
 - **Next Review**: When you should reconnect
 Format: Horizontal timeline or vertical list with "X days until" callouts.
 #### Action Buttons (Quick nav)
 - **QBR Prep** → Opens checklist template
 - **Edit Account** → Opens Sheets in new tab
 - **View Notes** → Opens Obsidian vault or embedded Markdown
 - **Share** → (Future) generates shareable summary link
 - **Log Interaction** → Quick-add form (bottom right pane)
 ---
 ### 4.3 Center Pane: Details & Drill-Down
 **Purpose:** Zoom from 10K ft into List 1 and List 2 metrics. See details, trends, and rationale.
 **Tab Structure:**
 #### Tab 1: Overview (Summary of all 10 items)
 Shows all 5 List 1 items + all 5 List 2 items as **cards in a grid**:
 - Card layout: Title | 10K ft summary | Status badge | Click to drill
 - Color-coded by item type (List 1 = blue accent, List 2 = green accent)
 - Scanning-friendly; can see account at a glance
 #### Tab 2: Account Profile (List 1 Deep Dive)
 Five expandable cards, one per List 1 item:
 **Card 1: Data Footprint & Workload Breakdown**
 ```
 ┌─ Data Footprint & Workload Breakdown ───────────────────┐
 │ 10K FT VIEW:                                             │
 │ 500TB current | +18% YoY | 2yr forecast: 700TB           │
 │ Workloads: 40% OLTP | 35% Analytics | 25% Unstructured  │
 │                                                           │
 │ [CLICK TO EXPAND ▼]                                      │
 │                                                           │
 │ EXPANDED VIEW:                                           │
 │ ┌─ Breakdown Table ──────────────────────────────────┐  │
 │ │ Tier       │ Current │ Growth │ 2yr Forecast │ Note  │
 │ │ OLTP       │ 200TB   │ +15%   │ 265TB        │ TX DB │
 │ │ Analytics  │ 175TB   │ +25%   │ 275TB        │ DW    │
 │ │ Unstructured│ 125TB   │ +12%   │ 160TB        │ Lake  │
 │ └────────────────────────────────────────────────────┘  │
 │                                                           │
 │ ┌─ Capacity Growth Chart ────────────────────────────┐  │
 │ │ [Line chart: 3yr history + 2yr forecast]           │  │
 │ │ Shows growth trajectory, where Pure could capture │  │
 │ └────────────────────────────────────────────────────┘  │
 │                                                           │
 │ CONTEXT FROM NOTES:                                      │
 │ "Customer mentioned big analytics push for FY27..."      │
 │                                                           │
 │ [Related Links: 10-K | Analyst Report]                  │
 └─────────────────────────────────────────────────────────┘
 ```
 **Card 2: Current Storage Vendors & Installed Infrastructure**
 ```
 ┌─ Current Storage Vendors ──────────────────────────────┐
 │ 10K FT VIEW:                                            │
 │ NetApp 50% | Pure 30% | Dell 20% (of 1200TB total)     │
 │ Consolidation Opportunity: HIGH                         │
 │                                                          │
 │ [CLICK TO EXPAND ▼]                                     │
 │                                                          │
 │ EXPANDED VIEW:                                          │
 │ ┌─ Vendor Breakdown (Pie Chart) ─────────────────────┐ │
 │ │ [Visual pie showing % by vendor]                   │ │
 │ │ NetApp = oldest systems, refresh due              │ │
 │ └────────────────────────────────────────────────────┘ │
 │                                                          │
 │ ┌─ Incumbent Systems Detail ────────────────────────┐  │
 │ │ Vendor: EMC Symmetrix DMX-4                       │  │
 │ │ Capacity: 200TB | Age: 6 years | Status: AGING   │  │
 │ │ Use Case: Core banking OLTP                       │  │
 │ │ 💡 Displacement Target: Move to Pure FlashArray   │  │
 │ └────────────────────────────────────────────────────┘ │
 │                                                          │
 │ COMPETITIVE CONTEXT:                                    │
 │ "This EMC system is at end-of-life; customer risk-     │
 │ averse but willing to consolidate if migration is      │
 │ proven. Reference customer opportunity."                │
 │                                                          │
 │ [Related: Competitive Intel | Displacement Playbook]    │
 └──────────────────────────────────────────────────────────┘
 ```
 **Cards 3-5:** Similar drill-down structure for Business Drivers, Architecture, Budget/Authority
 #### Tab 3: Pure Deployment (List 2 Deep Dive)
 Five cards showing Pure-specific metrics, one per List 2 item.
 **Card 1: Current Pure Installed Base**
 ```
 ┌─ Current Pure Installed Base ──────────────────────────┐
 │ 10K FT VIEW:                                            │
 │ 2 FlashArray systems (150TB) | Healthy | 4.2yr avg age │
 │ Headroom: 52TB | Refresh window: 18mo                   │
 │                                                          │
 │ [CLICK TO EXPAND ▼]                                     │
 │                                                          │
 │ EXPANDED VIEW:                                          │
 │ ┌─ Systems Inventory Table ──────────────────────────┐ │
 │ │ System | Type | Capacity | Util | Headroom | Age   │ │
 │ │ FA-P1  | FA   | 75TB     | 65%  | 26TB     | 5.1yr │ │
 │ │ FA-DR  | FA   | 75TB     | 40%  | 45TB     | 3.3yr │ │
 │ │                                  Refresh Needed    │ │
 │ └────────────────────────────────────────────────────┘ │
 │                                                          │
 │ ┌─ Utilization & Headroom Heatmap ──────────────────┐ │
 │ │ [Visual: System cards with fill % + headroom bar]  │ │
 │ │ FA-P1: ████████ 65% USED (26TB headroom)          │ │
 │ │ FA-DR: █████░░░ 40% USED (45TB headroom)          │ │
 │ └────────────────────────────────────────────────────┘ │
 │                                                          │
 │ EXPANSION IMPLICATION:                                  │
 │ "FA-P1 will need refresh/capacity add in 18 months;     │
 │ FA-DR has headroom for growth without additional spend" │
 │                                                          │
 │ [Related: Pure1 Dashboard | Refresh Planning Guide]     │
 └──────────────────────────────────────────────────────────┘
 ```
 **Cards 2-5:** Performance/Satisfaction, Coverage Gaps, Refresh Cycles, Stakeholders
 ---
 ### 4.4 Right Pane: Unstructured Context & Interactions
 **Purpose:** Connect quantitative data (left/center) to narrative context (right). See "why" behind the numbers.
 **Components:**
 #### Latest Interactions (Log)
 Shows most recent 3-5 interactions with sentiment:
 ```
 ┌─ Interactions ────────────────────────────────────────┐
 │ ⭐⭐⭐⭐⭐ Sarah Chen (VP Storage)                      │
 │   Call — 2026-10-02 · 5 days ago                       │
 │   "Discussed multi-site replication, pain with NetApp" │
 │   Status: Strong Advocate for Pure adoption            │
 │                                                         │
 │ ⭐⭐⭐ Mike Lopez (CIO)                                │
 │   Email — 2026-09-15 · 22 days ago                     │
 │   "Budget constraints; interested in OpEx models"      │
 │   Status: Neutral, cost-focused                        │
 │                                                         │
 │ ⭐⭐ Jennifer Park (CFO)                              │
 │   Meeting — 2026-08-20 · 47 days ago                   │
 │   "Positive on storage consolidation, capex tight"     │
 │   Status: Positive but budget-constrained              │
 │                                                         │
 │ [+ Log New Interaction]                                │
 └───────────────────────────────────────────────────────┘
 ```
 #### Related Notes (from Markdown)
 Embedded snippet from Obsidian Markdown file:
 ```
 ┌─ Discovery Notes ──────────────────────────────────────┐
 │ STRATEGY THEME: Consolidation play                     │
 │ Customer is post-acquisition; consolidating 3 data     │
 │ centers into 2. Big opportunity to move from multi-    │
 │ vendor (EMC + NetApp) to Pure standardization.         │
 │                                                         │
 │ RISK: Customer risk-averse on primary storage.         │
 │ LEVER: Success story at reference customer + DR copy. │
 │                                                         │
 │ NEXT STEP: Sarah is scheduled for Q4 arch review.      │
 │ Come with: Multi-site replication architecture,        │
 │ consolidation cost model.                              │
 │                                                         │
 │ [View Full Notes] [Edit in Obsidian]                   │
 └───────────────────────────────────────────────────────┘
 ```
 #### Quick Actions
 - **[Log Interaction]** → Popup form to add call/email/meeting
 - **[View All Notes]** → Opens Obsidian file
 - **[View Sheets]** → Opens Google Sheets
 - **[Share Account]** → (Future) generates link
 ---
 ### 4.5 Responsive Behavior & Interactivity
 #### Drill-Down Interactions
 - **Cards expand in place** — no modal/new page; content slides down
 - **Breadcrumb trail** — e.g., "Account > List 1 > Data Footprint" so you know where you are
 - **Collapse others** — When one card expands, others collapse (reduces noise)
 - **Smooth transitions** — Cards animate open/closed
 #### Status Badges & Visual Indicators
 | Status | Color | Meaning |
 |--------|-------|---------|
 | 🟢 Green | Good / On Track / Advocate | Positive signal |
 | 🟡 Yellow | Caution / Aging / Neutral | Attention needed |
 | 🔴 Red | Risk / Critical / Urgent | Action required |
 | ⚪ Gray | Unknown / TBD | Need more info |
 #### Data-Driven Highlights
 Conditional visual emphasis on metrics:
 - **Refresh window approaching** (>4yr age) → Highlighted in yellow
 - **Coverage gap opportunity** (low % Pure) → Highlighted in orange
 - **Budget cycle in next 90 days** → Underlined with "upcoming" tag
 - **Strong champion sentiment** → Starbursts/badge next to name
 ---
 ## 5. Component Breakdown & Data Mapping
 ### 5.1 Component Hierarchy
 ```
 DashboardApp
 ├── Header
 │   ├── AccountName
 │   ├── StatusBadge (Prospect/Customer)
 │   ├── MetadataBar (Vertical, Location, Last Updated)
 │   └── HeaderActions ([Share] [Edit] [Settings])
 │
 ├── MainLayout (3-pane)
 │   │
 │   ├── LeftPane (Overview)
 │   │   ├── HealthScorecard
 │   │   │   ├── KPICard (Pure Coverage)
 │   │   │   ├── KPICard (Fleet Age)
 │   │   │   ├── KPICard (Opportunity Size)
 │   │   │   └── KPICard (Engagement Momentum)
 │   │   ├── TimelineComponent
 │   │   │   ├── TimelineEvent (Refresh Date)
 │   │   │   ├── TimelineEvent (Budget Cycle)
 │   │   │   ├── TimelineEvent (Contract Renewal)
 │   │   │   └── TimelineEvent (Next Review)
 │   │   └── ActionButtons
 │   │       ├── Button (QBR Prep)
 │   │       ├── Button (Edit Account)
 │   │       ├── Button (View Notes)
 │   │       └── Button (Share)
 │   │
 │   ├── CenterPane (Details & Drill-Down)
 │   │   ├── TabBar
 │   │   │   ├── Tab (Overview)
 │   │   │   ├── Tab (List 1: Account Profile)
 │   │   │   ├── Tab (List 2: Pure Deployment)
 │   │   │   └── Tab (Opportunities — future)
 │   │   │
 │   │   └── TabContent (varies by tab)
 │   │       └── DetailsCard (repeating)
 │   │           ├── CardHeader (title + status)
 │   │           ├── CardSummary (10K ft view)
 │   │           ├── ExpandButton
 │   │           └── ExpandedContent (on click)
 │   │               ├── DetailTable / Chart
 │   │               ├── TrendVisualization
 │   │               └── RelatedContext
 │   │
 │   └── RightPane (Context)
 │       ├── InteractionLog
 │       │   └── InteractionItem (repeating)
 │       │       ├── Sentiment (stars)
 │       │       ├── Contact Info
 │       │       ├── Timestamp
 │       │       └── Summary
 │       ├── RelatedNotes
 │       │   └── NotesSnippet
 │       └── QuickActions
 │           ├── Button (Log Interaction)
 │           ├── Button (View All Notes)
 │           └── Button (View Sheets)
 ```
 ### 5.2 Data Source Mapping
 | Component | Data Source | Sheet Location | Notes |
 |-----------|---|---|---|
 | **Account Name** | Account tab | Name | |
 | **Status Badges** | Account tab | Status, Vertical, Location | |
 | **Health Scorecard** | Account + Pure-Deployment tabs | Multiple cells + formulas | Calculated KPIs |
 | **Timeline** | Account tab | Columns: Next Refresh Date, Budget Cycle, Contract End, Next Review | |
 | **List 1 Cards** | Account tab | Tabs: Item 1-5 data | Five expandable items |
 | **List 2 Cards** | Pure-Deployment tab | Tabs: Item 1-5 data | Five expandable items |
 | **Interaction Log** | Interactions tab | Contact, Type, Date, Sentiment, Summary | Sorted by date DESC |
 | **Related Notes** | Markdown file (Obsidian) | Fetched via API or embed | Text snippet + link |
 ---
 ## 6. Visual Design Principles
 ### Color Palette (Themeable)
 - **Accent Blue** (Primary): List 1 items, account overview
 - **Accent Green** (Secondary): List 2 items, Pure deployment
 - **Status Green** (Success): Healthy, positive, on-track
 - **Status Yellow** (Warning): Approaching refresh, caution needed
 - **Status Red** (Critical): Urgent action required
 - **Neutral Gray**: Unknown, TBD, secondary content
 - **Text Dark Gray** (Light mode) / **Text Light Gray** (Dark mode)
 - **Background**: Light/dark mode adaptive
 ### Typography
 - **Header**: Large, bold, account name stands out
 - **Tab titles**: Medium weight, clear tab distinction
 - **Card titles**: Semi-bold, clickable indicator (▼)
 - **Data values**: Monospace for numbers (easier to scan)
 - **Labels**: Smaller, muted color
 ### Spacing & Layout
 - **Panes**: Equal width (33% each) or responsive flex at smaller widths
 - **Cards**: Padding ~20px, margin ~16px between cards
 - **Drill-down**: Animated 200ms expand/collapse
 - **Mobile**: Stack panes vertically (future consideration)
 ---
 ## 7. Navigation & User Flows
 ### Flow 1: Pre-Call Prep (15 min before discovery)
 ```
 User lands on dashboard
    ↓
 Glances at Health Scorecard (30 sec) — "Pure coverage 30%, fleet aging, $3.5M upside"
    ↓
 Checks Key Dates Timeline (30 sec) — "Budget cycle in 3 mo, refresh in 18 mo"
    ↓
 Clicks "QBR Prep" action → Opens checklist template in new tab
    ↓
 Scrolls to Center pane, clicks "Current Vendors" card to see EMC footprint
    ↓
 Reads right pane: "Customer risk-averse on primary storage; use reference story"
    ↓
 Click [Log Interaction] → Pre-fills with today's call, ready to document post-call
 ```
 ### Flow 2: Competitive Displacement Hunt
 ```
 User is looking for displacement opportunities
    ↓
 Opens Account dashboard
    ↓
 Glances at Scorecard: "Pure coverage 30%" → Opportunity
    ↓
 Clicks "Data Footprint" card → Sees workload breakdown
    ↓
 Clicks "Current Vendors" card → Sees EMC system (200TB, 6yr old, primary OLTP)
    ↓
 Clicks "Coverage Gaps" card → Confirms: "EMC Symmetrix not on Pure yet"
    ↓
 Reads notes: "Customer risk-averse but willing to migrate if proven"
    ↓
 Action: Find reference customer, prepare migration playbook for next call
 ```
 ### Flow 3: Expansion Planning
 ```
 User reviewing account for renewal/expansion
    ↓
 Glances at Scorecard: Fleet age, Opportunity size, Engagement momentum
    ↓
 Clicks "Fleet Age" card → Sees FA-P1 refresh window in 18 months
    ↓
 Clicks "Refresh Cycles" card → Confirms capex cycle and window
    ↓
 Checks Stakeholder info in right pane → "Sarah (VP Storage) is champion, strong advocate"
    ↓
 Clicks "Opportunities" tab (future) → See pipeline: Analytics consolidation (Q2 2027, $1.5M)
    ↓
 Action: Schedule architecture review with Sarah for Q1; prepare proposal for analytics move
 ```
 ---
 ## 8. Data Sync & Freshness
 ### MVP Approach: Manual Sync
 - **Sheet → Dashboard**: Copy-paste JSON export or Google Sheets API call (daily/on-demand)
 - **Markdown → Dashboard**: Fetch from Obsidian API or GitHub if synced
 - **Freshness**: Dashboard is "point-in-time snapshot" at time of load
 - **Update frequency**: User refreshes page or manually syncs from edit screens
 ### Future: Live Sync
 - Dashboard queries Google Sheets API in real-time
 - Interactions logged directly to Sheets (no manual re-entry)
 - Markdown fetched from Obsidian Sync API
 - Auto-refresh KPIs every 5 minutes
 ---
 ## 9. Extensibility Points (Future Phases)
 ### Phase 2: Relationships & Interaction History
 - **Tab 4: Relationships** — Org chart, sentiment timeline, interaction history
 - **Interaction log** — Searchable, filterable by contact, type, sentiment
 - **Sentiment analysis** — Chart showing mood trend over time
 ### Phase 3: Opportunities & Pipeline
 - **Tab 5: Opportunities** — Consolidation roadmap, deal timeline, revenue forecast
 - **Milestone tracking** — Countdown to RFP, contract close, etc.
 ### Phase 4: Financial Modeling
 - **10-K integration** — Embed PDF link, key metrics extracted
 - **Opportunity forecasting** — If analytics consolidates, revenue impact
 - **Budget impact model** — Capex vs. OpEx trade-off calculator
 ### Phase 5: Industry & Competitive Context
 - **Industry trends** — Curated insights relevant to account
 - **Competitor tracking** — Win/loss intelligence, displacement playbooks
 - **Market context** — Industry growth rates, regulatory changes
 ---
 ## 10. Technical Architecture
 ### Technology Stack
 - **Frontend**: React (or vanilla JS if keeping simple)
 - **Styling**: Tailwind CSS (or custom CSS-in-JS)
 - **Data sources**: 
  - Google Sheets API (for Account, Pure-Deployment, Relationships, Financials tabs)
  - Markdown file (from Obsidian or GitHub)
 - **Deployment**: Claude artifact (publishable, shareable URL) or self-hosted on homelab
 - **Build**: Vite (fast dev) or Create React App
 ### API Integration (MVP)
 ```
 Frontend (React)
    ↓
    ├─→ Google Sheets API v4
    │   └─ Fetch: Account!A:Z, PureDeployment!A:Z, Relationships!A:Z, etc.
    │      Rate: On-demand or 5-min cache
    │
    └─→ Markdown fetch (via Obsidian API or raw GitHub/Drive link)
        └─ Fetch: Markdown content, parse for snippets
 ```
 ### State Management
 - **Client-side state**: React hooks (useState, useEffect)
 - **Server state**: Google Sheets is source of truth
 - **UI state**: Which tab active, which card expanded
 - **No local storage** (MVP): Always fresh from source
 ---
 ## 11. Accessibility & Usability
 ### Keyboard Navigation
 - Tab through cards, tabs, buttons
 - Enter/Space to expand cards
 - Escape to collapse all
 - Ctrl/Cmd+Click to open links in new tab
 ### Screen Readers
 - Semantic HTML (headers, nav, main, article)
 - ARIA labels on custom components
 - Heading hierarchy (H1 = account name, H2 = tabs, H3 = card titles)
 ### Color Contrast
 - All text meets WCAG AA standards
 - Status indicators have labels (not color alone)
 - Links underlined or clearly differentiated
 ### Mobile Responsiveness (Future)
 - Stack panes vertically on small screens
 - Touch-friendly button sizes (44px minimum)
 - Collapsible sections for readability
 ---
 ## 12. Success Metrics (MVP)
 | Metric | Target | How to Measure |
 |--------|--------|---|
 | **Time to account snapshot** | <2 min to full context | User time log |
 | **Drill-down effectiveness** | Find key metric in <30 sec | User task test |
 | **Information completeness** | All 10 items readily visible | Checklist |
 | **Visual clarity** | No confusion on next steps | User feedback |
 | **Engagement** | Used for all 3 use cases equally | Usage logs |
 ---
 ## 13. Known Constraints & Trade-offs
 | Constraint | Impact | Mitigation |
 |-----------|--------|---|
 | **Manual sync (MVP)** | Data can lag if Sheets updated but dashboard not refreshed | Clear "last updated" timestamp; button to manual sync |
 | **No real-time collaboration** | If account team edits Sheets while viewing dashboard, changes won't appear until reload | Future: implement live sync |
 | **Single-customer dashboard** | Can't compare metrics across accounts in MVP | Future: roll-up view with filtering |
 | **API rate limits (Sheets)** | Can't refresh dashboard every second | Cache on client; user triggers refresh manually |
 ---
 ## 14. Success Criteria for Prototype
 ✅ **Must Have (MVP)**
 - [ ] All 10 items (List 1 + List 2) visible and organized
 - [ ] 3-pane layout functioning (left/center/right)
 - [ ] Cards expand/collapse on click with animation
 - [ ] Health scorecard with 4 KPIs calculated
 - [ ] Timeline showing key dates
 - [ ] Sample data flowing in from mock Sheets
 - [ ] Navigation between tabs (Overview, List 1, List 2)
 - [ ] Status badges and visual indicators rendering correctly
 ⚡ **Should Have (Nice to Have)**
 - [ ] Trend indicators (↑ ↓ →) on KPI cards
 - [ ] Sample charts (pie, line) in expanded views
 - [ ] Markdown snippet embedded in right pane
 - [ ] Interaction log with sentiment stars
 - [ ] Dark mode toggle
 🎯 **Nice to Have (Future)**
 - [ ] Google Sheets API actually connected (MVP uses mock data)
 - [ ] Relationships tab structure visible
 - [ ] QBR Prep modal/checklist
 - [ ] Share functionality
 - [ ] Export to PDF
 ---
 ## 15. Design Review Checklist
 **Questions for Ryan before prototype:**
 1. ✅ Is the 3-pane layout clear? Does it support the 10K → ground zoom?
 2. ✅ Are the 4 KPIs the right ones, or would you change them?
 3. ✅ Does the drill-down interaction (click card → expand) feel right, or prefer modal?
 4. ✅ Should the right pane be scrollable independently, or scroll with center pane?
 5. ✅ Any major UX concerns with the navigation flow?
 6. ✅ Is the timeline component clear (horizontal bar vs. vertical list)?
 7. ✅ Color palette acceptable, or want different accent colors?
 8. ✅ Any tabs or components you'd add/remove for MVP?
 ---
 **Document Status:** Ready for Design Review & Feedback  
 **Next Step:** Prototype build (pending approval of this design)
@@ -0,0 +1,257 @@
 # Top-5 Account Intelligence Lists for Enterprise Storage SE
 **For:** Pre-Sales Systems Engineers at Pure Storage  
 **Focus Verticals:** Financial Services, Retail/Manufacturing  
 **Target Accounts:** Fortune 1000 / Global 2000
 ---
 ## LIST 1: ANY CUSTOMER OR PROSPECT
 ### First Discovery Call + Competitive Displacement + Expansion Foundation
 ---
 ### 1. **Data Footprint and Workload Breakdown**
 **What to capture:**
 - Total addressable storage capacity (current + 2-year projection)
 - Workload tier split: critical OLTP (databases, transactional), warm operational analytics (data warehouses, ML training sets), unstructured (data lakes, backups, archives)
 - Annual growth rate
 - Typical IO patterns (sequential vs. random, read/write mix)
 - Workload sources: core business apps, digital channels, analytics platforms, third-party integrations
 **Why it ranks #1:**
 This is your credibility foundation for the discovery call. Large enterprises in financial services and retail/manufacturing operate at scale—you need to know if they're managing 10PB or 500PB of storage before you can ask intelligent questions about consolidation, resilience, or performance. For competitive displacement, it tells you the total addressable market and which workloads are most valuable to move. For expansion, it shows you where growth is (analytics workloads often outpace transactional growth in these verticals).
 **Vertical differences:**
 - **Financial services:** Regulatory data tiers (Basel III, Dodd-Frank, GDPR-driven data governance), transaction volumes, archive-to-compliance mandates, and fraud detection datasets are outsized here.
 - **Retail/manufacturing:** Omnichannel transaction data, supply chain event data, and IoT/sensor streams create different growth patterns—often unstructured-heavy.
 **Where to get it:**
 Account team, analyst kickoff, customer IT strategy documents. If public: earnings calls (capex guidance), analyst reports on data center expansion. **Hard to verify fully from public sources—lean on the account team and discovery conversation.** Pure1 for existing customers (capacity trends, growth rates).
 ---
 ### 2. **Current Storage Vendor(s) and Installed Infrastructure**
 **What to capture:**
 - Primary vendor(s), total installed capacity per vendor, system age/generation
 - Deployment model (on-prem, cloud-hosted)
 - Vendor concentration (single-vendor, multi-vendor)
 - Capacity consolidation potential (old systems due refresh, overlapping vendors)
 - Competitive systems they've ruled out or are evaluating
 **Why it ranks #2:**
 This is the competitive map. In discovery, you need to know who you're up against to ask credible questions about their stack. For displacement, this is your roadmap—old NetApp or EMC systems are your opening; vendor sprawl is your consolidation story. For expansion, it shows where Pure is (or isn't) and where you can grow.
 **Applies to both verticals:**
 Regulatory and operational requirements often enforce heterogeneous stacks (legacy core systems, newer analytics platforms), creating multi-vendor complexity and Pure's consolidation angle.
 **Where to get it:**
 Account team, customer IT contacts, Gartner/IDC reports, job postings (hiring for specific platforms signals infrastructure). Pure's competitive intel teams (if available in your region). **Harder to get from public sources—interview in discovery.**
 ---
 ### 3. **Business Drivers and Strategic Imperatives**
 **What to capture:**
 - What's moving their infrastructure strategy in the next 12–24 months?
 - Digital transformation (cloud-first, API-led architecture)
 - M&A (data center consolidation, platform rationalization)
 - Regulatory change (data residency, compliance modernization)
 - Cost reduction (capex or opex pressure)
 - Performance demands (real-time analytics, AI/ML acceleration)
 - Geographic expansion (multi-site, disaster recovery)
 **Why it ranks #3:**
 This explains *why* they'll spend and *when*. For discovery, it tells you what your conversation should be anchored to—cost pressure is a different conversation than compliance urgency. For displacement, it reveals openings (legacy vendor can't support cloud migration; incumbent doesn't scale for analytics). For expansion, it shows where budget flows and growth cycles.
 **Vertical differences:**
 - **Financial services:** Regulatory change (Fed data residency mandates, CCAR/stress test infrastructure), fraud detection (AI/ML workloads), and open banking/fintech competition drive transformation.
 - **Retail/manufacturing:** Omnichannel unification, supply chain visibility, and sustainability reporting (ESG data) are primary drivers; cost pressure is often secondary.
 **Where to get it:**
 Earnings calls (CFO and CRO commentary), industry analyst reports, customer executives in discovery (CEOs and CFOs speak to this), account team (relationship intelligence). Regulatory announcements for financial services. **Public sources vary—direct ask is usually most productive.**
 ---
 ### 4. **Infrastructure Architecture and Resilience Model**
 **What to capture:**
 - Topology: single data center, active-active multi-site, hub-and-spoke?
 - Failover strategy (RPO/RTO targets, synchronous vs. asynchronous replication)
 - Cloud presence (public cloud, hybrid, private cloud)
 - Containerization and Kubernetes adoption
 - Storage tiers and tiering automation
 - Backup/archive strategy (separate systems, integrated)
 **Why it ranks #4:**
 This determines *which* Pure solutions fit and how. Knowing they're multi-site tells you Evergreen model and replication are critical; knowing they have Kubernetes tells you Portworx is on the menu; knowing they're chasing sub-second RTO tells you FlashArray's performance story matters. For discovery, it frames the conversation around resilience or performance. For displacement, it shows where Pure's architecture outperforms incumbent (e.g., Evergreen's native replication vs. bolted-on replication). For expansion, it guides where to add capacity.
 **Vertical differences:**
 - **Financial services:** Recovery requirements are often strict (RTO <1hr, RPO <15min); multi-site is standard; compliance mandates geographic separation.
 - **Retail/manufacturing:** Multi-site is common (stores, plants, distribution); RTO is important but often longer than financial services; hybrid cloud adoption is accelerating.
 **Where to get it:**
 Account team, technical RFI/RFQ documents, infrastructure diagrams (security-sensitive—may not be public). Discovery conversation with IT infrastructure/storage teams. **Hard from public sources.**
 ---
 ### 5. **Budget Cycle, Capital Allocation, and Buying Authority**
 **What to capture:**
 - Capex budget cycle and approval gates (who approves? quarterly, annual, rolling?)
 - Capex vs. opex preference (traditional capex, Evergreen-style subscription, OpEx-only mandate)
 - Budget envelope for infrastructure over next 2 years
 - Preferred buying patterns (competitive bid, existing vendor preference, tech refresh cycle)
 - Influencers and decision-makers (CIO, VP Storage, business unit leaders, CFO)
 **Why it ranks #5:**
 This is the "when and how to sell" lever. It's less about discovery credibility (though knowing budget is essential) and more about deal realism. For discovery, it tells you timing and buying authority to route follow-ups. For displacement, it tells you if capex is available or if you need to frame Evergreen as an alternative. For expansion, it reveals budget headroom and buying rhythm.
 **Applies to both verticals:**
 Financial services often have tighter capex scrutiny and longer approval cycles; retail/manufacturing may have faster refresh cycles but more budget constraints post-acquisition integration.
 **Where to get it:**
 Account team relationship notes, past deal history, 10-K/10-Q filings (capex guidance), analyst reports, budget forecasts shared in customer relationships. **Rarely fully public—account team and direct relationship intelligence are key.**
 ---
 ## LIST 2: EXISTING CUSTOMERS — PURE STORAGE DEPLOYMENT SPECIFICS
 ### Expansion + Retention + Competitive Defense
 ---
 ### 1. **Current Pure Installed Base, Deployment Pattern, and Capacity**
 **What to capture:**
 - FlashArray systems (models, capacity per system, tier across data centers or cloud regions)
 - FlashBlade deployments (scale-out analytics use cases, capacity)
 - Portworx deployments (if any; containerized workloads, environments)
 - Age of systems (when deployed; refresh cycle status)
 - Geographic footprint (on-prem across which data centers, any cloud-hosted Pure?)
 - Utilization and headroom (at capacity, growing into headroom, or sparse?)
 **Why it ranks #1:**
 You can't expand without knowing your baseline. For expansion, this is your starting point—you need to know what they own to identify the next sale (add capacity, new tier, new workload). For retention, it's your health baseline (old systems more likely to churn). For competitive defense, it shows Pure's footprint and where competitors might try to take share.
 **Key indicators for retention/expansion:**
 Systems older than 5 years are refresh candidates; sparse utilization may indicate underutilized tier or expansion opportunity; multi-site Pure deployments are higher-value accounts.
 **Where to get it:**
 **Pure1 (your best data source—capacity, performance, age, location),** internal Pure account data, customer IT/storage teams. Direct ask in customer reviews and strategic business reviews (QBR/SBR).
 ---
 ### 2. **Performance, Utilization, and Customer Satisfaction**
 **What to capture:**
 - Pure1 performance metrics (latency, IOPS achieved vs. provisioned, dedupe/compression ratios, capacity growth rate)
 - Storage team feedback (support experience, feature requests, known pain points)
 - Performance satisfaction (are performance SLAs being met?)
 - Operational efficiency gains (cost per GB, power/cooling reduction)
 - Customer sentiment (NPS or equivalent; any churn signals?)
 **Why it ranks #2:**
 This tells you expansion potential vs. churn risk. If they're unsatisfied, expansion is at risk (competitor threat, bargaining power). If they're thriving on Pure, expansion is easier (proven business case). For retention, this is your early warning system. For expansion, high satisfaction is your foundation for upsells (new workload, new tier, cloud-native).
 **Vertical differences:**
 - **Financial services:** Customers often have strict uptime/performance mandates.
 - **Retail/manufacturing:** Often cost-conscious and may trade performance for efficiency.
 **Where to get it:**
 **Pure1 (performance analytics),** NPS/CSAT surveys, QBR discussions, support ticket sentiment, sales/SE relationship notes. Direct ask: "How is Pure performing for you?"
 ---
 ### 3. **Data Footprint Not Yet on Pure / Coverage Gap Analysis**
 **What to capture:**
 - Percentage of their total storage on Pure vs. other vendors
 - Workload segments not on Pure (e.g., older OLTP databases on EMC, analytics on Hadoop/object storage, backups on separate vendor)
 - Highest-value data not on Pure (revenue-generating apps, critical analytics)
 - Consolidation opportunities (multi-vendor overlap, legacy systems ready to refresh)
 - Workloads Pure could address but hasn't yet (emerging use cases, cloud data, new business initiatives)
 **Why it ranks #3:**
 This is your expansion TAM within the account. For expansion, it shows where growth comes from (consolidation opportunity, new workload, tier migration). For competitive defense, it identifies where competitors will attack (the gap is their opening). For retention, it's leverage—you want to own more of their data.
 **Vertical differences:**
 - **Financial services:** Critical OLTP often stays on incumbent (lower risk perception); newer analytics and fraud detection data are Pure opportunities.
 - **Retail/manufacturing:** Supply chain/IoT data and unstructured analytics are often on non-Pure platforms; consolidation gains are high here.
 **Where to get it:**
 **Pure1 (shows only Pure capacity—you need to ask for total footprint),** customer IT (data inventory), discovery/expansion conversations. **Often not transparent from public sources—direct conversation essential.**
 ---
 ### 4. **Upcoming Infrastructure Initiatives and Technology Refresh Cycles**
 **What to capture:**
 - Planned data center consolidation or modernization
 - System refresh dates (when is their oldest Pure system past end-of-support? When's the next capex cycle?)
 - Cloud migration plans (moving workloads to cloud; implications for on-prem storage)
 - New workloads (AI/ML platforms, real-time analytics, new business lines)
 - Multi-site expansion or disaster recovery modernization
 **Why it ranks #4:**
 Timing is everything in expansion. Refresh cycles are your biggest lever (old systems = new capex = new opportunity). Cloud migration can cut Pure TAM or open Portworx opportunities. New workloads are expansion hooks. For expansion, this is your calendar. For retention, knowing refresh timing lets you plan defense early.
 **Applies to both verticals:**
 Financial services often have annual capex cycles tied to business planning; retail/manufacturing may refresh faster or slower depending on store/plant footprint changes.
 **Where to get it:**
 Customer IT roadmap (often multi-year), Pure account team, QBR discussions, earnings announcements (for large public customers, CapEx guidance), customer executives. Sometimes in customer's published IT strategy.
 ---
 ### 5. **Stakeholder Relationships, Expansion Readiness, and Competitive Threats**
 **What to capture:**
 - Key stakeholders (CIO, VP Storage, business unit leaders, security/compliance officers)
 - Champion(s) (who advocates for Pure?)
 - Relationship temperature (trust, satisfaction, barriers)
 - Competitive interest (have they mentioned other vendors? RFPs? Evaluation?)
 - Budget headroom (will they invest more in storage in next 12–24 months?)
 - Expansion appetite (are they open to moving more workloads to Pure, or focused on optimization?)
 **Why it ranks #5:**
 This is your "how to expand" lever. Knowing who the champion is tells you how to navigate. Knowing if they're under competitive attack tells you how urgent your defense is. Knowing budget appetite tells you if expansion is realistic. For expansion, this is your relationship risk/opportunity assessment. For retention, it's your early warning for churn or your pathway for growth.
 **Applies to both verticals:**
 In large enterprises, storage decisions often flow through business unit leaders (not just IT); understanding where Pure fits in their strategic priorities is key.
 **Where to get it:**
 Account team relationship notes, past deal history, QBR feedback, sales/SE interactions, industry/analyst events. Direct ask in customer conversations: "Where do you see us helping you grow?"
 ---
 ## Summary: Why This Order Matters
 **List 1** (any customer/prospect) moves from **foundational understanding** (what and who) → **context** (why and how) → **execution** (when and who decides). This sequence builds credibility in discovery while revealing competitive and expansion levers.
 **List 2** (existing customer Pure deployments) moves from **your current position** (what you have) → **relationship health** (how it's performing) → **opportunity size** (what you could have) → **timing** (when they'll buy) → **execution** (how to navigate the sale). This sequence is your expansion playbook.
 ### Key Callout
 In both lists, **items 1–3 should drive 80% of your discovery and account strategy effort**. Items 4–5 refine timing and approach. **Start here, then layer in the operational details** (support SLAs, feature adoption, roadmap alignment) as you deepen the relationship.
 ---
 ## Quick Reference: Source Hierarchy by Item
 | Item | Primary Source | Secondary Source | Public Availability |
 |------|---|---|---|
 | **List 1 #1** – Data Footprint | Account team, customer IT | Earnings calls, analyst reports | Moderate (public guidance varies) |
 | **List 1 #2** – Current Vendors | Account team, IT contacts | Job postings, Gartner reports | Low (often internal knowledge) |
 | **List 1 #3** – Business Drivers | Customer executives, analyst reports | Earnings calls, regulatory filings | Moderate (CEOs/CFOs speak to this) |
 | **List 1 #4** – Architecture | Technical RFI/RFQ, account team | Discovery conversation | Low (security-sensitive) |
 | **List 1 #5** – Budget/Authority | Account team, past deal history | 10-K/10-Q, customer relationships | Low (relationship intelligence) |
 | **List 2 #1** – Pure Installed Base | **Pure1** (best source), account data | Customer IT, QBR | Account-internal only |
 | **List 2 #2** – Performance/Satisfaction | **Pure1**, NPS surveys | Support tickets, QBR discussions | Account-internal only |
 | **List 2 #3** – Coverage Gaps | Customer IT inventory, discovery | Account team insights | Low (direct conversation required) |
 | **List 2 #4** – Refresh Cycles | Customer IT roadmap, account team | QBR, earnings announcements | Low-Moderate (customer-dependent) |
 | **List 2 #5** – Stakeholder/Competitive | Account team, sales history | QBR feedback, customer conversations | Account-internal only |
 ---
 **Use this as your go-to reference before discovery calls, account strategy reviews, and expansion planning.**
@@ -1,239 +1,102 @@
-# A Homelab based on Red Hat Technologies
+# mk-labs
-A comprehensive homelab for learning advanced concepts primarily based on Red Hat OpenShift.  In addition to providing services, it is an area to incorporate container technologies, and enterprise integration scenarios.
+Automated infrastructure provisioning and configuration for a personal homelab, built on GitOps practices with clear tool responsibility boundaries.
-This implementation is built on easily accessible consumer based hardware and will focus heavily on GitOps practices and automation will be used wherever possible.
+## Architecture
-## 🎯 Project Goals
+A single operator action — setting a VM record's status to **Staged** in NetBox — triggers a fully automated provisioning pipeline:
-
+```
- **Kubernetes cluster for applications**
+NetBox (webhook) → n8n (validate & orchestrate) → Terraform (create VM + DHCP)
- **IAM integration testing**
+                                                 → Ansible (OS config + DNS + status update)
 - **GitOps and automation workflows**
 ## 📋 Documentation (Priority Order)
 ### Phase 1: Foundation
 1. **[Overview & Hardware Allocation](./docs/01-overview.md)** - Complete architecture overview
 2. **[Networking Plan](./docs/02-networking.md)** - VLAN strategy and network design
 3. **[Core Services Setup](./docs/03-core-services.md)** *(TBD)* - Ansible, DNS, monitoring
 4. **[Storage Configuration](./docs/04-storage.md)** *(TBD)* - Synology, democratic-csi
 ### Phase 2: Proxmox Platform
 5. **[Proxmox Installation](./docs/11-vsphere-setup.md)** *(TBD)* - Dell 7050 virtualization
 6. **[VM Templates & Automation](./docs/13-vm-automation.md)** *(TBD)* - Template creation
 ### Phase 3: Infrastructure Services
 7. **[Recursive DNS]
 8. **[Authoritative DNS]
 9. **[Identity Management](./docs/10-acm-setup.md)** *(TBD)*
 10. **[Matchbox] (Depreciated?)
 ### Phase 4: OpenShift Clusters (Depreciated)
 8. **[OpenShift SNO + Worker](./docs/09-openshift-sno.md)** *(TBD)* - ACM Hub Cluster
 9. **[OpenShift Cluster](./docs/08-openshift-compact.md)** *(TBD)* - 3 Master/3 Worker node (production-like)
 10. **[ACM Configuration](./docs/10-acm-setup.md)** *(TBD)* - Multi-cluster management
 11. **[HCP Cluster](./docs/10-acm-setup.md)** *(TBD)* - External app cluster
 ### Phase 5: Kubernetes Container Platform
 5. **[Container Registry Setup](./docs/05-container-registry.md)** *(TBD)* - Harbor deployment
 6. **[Git Repository Setup](./docs/06-git-repository.md)** *(TBD)* - Gitea/GitLab on Synology
 7. **[Artifact Repository](./docs/07-artifact-repository.md)** *(TBD)* - Nexus/Artifactory
 ### Phase 6: Advanced Services
 14. **[Monitoring & Observability](./docs/14-monitoring.md)** *(TBD)* - Prometheus, Grafana, Splunk
 15. **[Security & Compliance](./docs/15-security.md)** *(TBD)* - ACS, certificates, auditing
 16. **[Backup & DR](./docs/16-backup-dr.md)** *(TBD)* - Backup strategies
 ## 🚀 Quick Start Deployment
 ### Prerequisites
 - Ubiquiti UDM Pro configured
 - Hardware powered and networked
 - Initial VLAN setup (see [networking plan](./docs/02-networking.md))
 ### Ansible Collections
 - ansible-galaxy collection install freeipa.ansible_freeipa
 ### Deployment Scripts
 ```bash
 source ~/venv-ansible/bin/activate
 # Phase 1: Foundation
 ./scripts/01-network-setup.sh
 ./scripts/02-core-services-deploy.sh
 # Phase 2: Container Platform  
 ./deployment/synology/docker-compose.yml  # Git repo, Harbor, Nexus
 ./scripts/03-storage-setup.sh
 # Phase 3: OpenShift
 ./deployment/openshift/compact-cluster/
 ./deployment/openshift/sno-cluster/
 # Phase 4: Proxmox
 ./deployment/proxmox/
 ```
-## 📁 Repository Structure
+| Tool | Host | IP | Responsibility |
 |------|------|----|---------------|
 | NetBox | fire-station | 10.1.71.102 | Source of truth — VM records, IP allocation, VLAN data |
 | n8n | tiki-room | 10.1.71.23 | Event orchestration, validation, pipeline sequencing |
 | Terraform | city-hall | 10.1.71.35 | Proxmox VM lifecycle, Unifi DHCP reservations |
 | Ansible / Semaphore | imagineering | 10.1.71.22 | OS configuration, DNS records, NetBox status updates |
 | Proxmox | fantasyland | 10.1.71.13 | Target hypervisor |
 All systems on the Server Trusted VLAN (10.1.71.0/24).
 ## Repository Structure
 ```
 homelab/
-├── README.md                    # This file
+├── ansible/
-├── docs/                        # Documentation (numbered by priority)
+│   ├── inventory/          # NetBox dynamic inventory + static
-│   ├── 01-overview.md
+│   ├── playbooks/          # Runnable playbooks (vm-provision, DNS, OS updates)
-│   ├── 02-networking.md
+│   ├── roles/              # vm-baseline, dns-manager, common, haproxy, n8n, observer, etc.
-│   └── ...
+│   ├── tasks/              # Shared includable task files
-├── ansible/                     # Directory structure for Ansible 
+│   ├── group_vars/         # Group variable definitions
-├── deployment/                  # Deployment 
+│   ├── host_vars/          # Per-host variable definitions
-configurations
+│   ├── templates/          # Jinja2 templates
-│   ├── synology/                # Docker Compose files for Synology
+│   └── ansible.cfg
-│   ├── k0s/                     # k0s cluster manifests
+│
-│   ├── openshift/               # OpenShift installation configs
+├── terraform/
-│   └── proxmox/                 # Proxmox automation
+│   ├── proxmox/vm/         # bpg/proxmox provider — VM creation from templates
-├── scripts/                     # Automation scripts
+│   ├── unifi/dhcp/         # Unifi provider — DHCP static reservations on UDM Pro
-└── .gitignore                   # Excludes sensitive data
+│   └── dns/                # DNS record management
 │
 ├── packer/
 │   ├── ubuntu-24.04/       # Ubuntu 24.04 VM template (small → xlarge-plus sizes)
 │   └── fedora-42/          # Fedora 42 VM template
 │
 ├── n8n/
 │   └── workflows/          # Exported n8n workflow JSON (vm-provisioning)
 │
 ├── netbox/
 │   └── initializers/       # Custom fields, VLANs, IP prefixes as code
 │
 └── docs/
    └── decisions/          # Architecture decision records
 ```
-## 🔧 Technology Stack
+## Pipeline Flow
-### Infrastructure
+| # | System | Action |
- **Networking**: Ubiquiti UDM Pro
+|---|--------|--------|
- **Storage**: Synology DS1621+, Ubiquiti UNAS Pro
+| 1 | NetBox | Operator sets VM status to Staged → webhook fires |
- **Compute**: 3x Minisforum TH60, 2x Minisforum MS01, 7x Dell 7050 SFF
+| 2 | n8n | Validates payload (hostname, IP, VLAN, template, proxmox_node) |
 | 3 | n8n → city-hall | SSH + `terraform apply` — creates VM on Proxmox |
 | 4 | n8n | Queries Proxmox API for MAC address |
 | 5 | n8n → NetBox | Writes MAC to VM interface record |
 | 6 | n8n → city-hall | SSH + `terraform apply` — creates DHCP reservation on UDM Pro |
 | 7 | n8n → imagineering | Triggers Ansible via Semaphore API |
 | 8 | Ansible | OS baseline, SSH hardening, Technitium DNS A record |
 | 9 | Ansible → NetBox | Sets VM status to Active |
-### Container Platforms
+On any failure, NetBox status is set to **Failed**. No auto-retry — operator investigates.
 - **k8s**: Core services cluster
 - **Proxmox**: VM workloads
 ### Core Services
 - **Container Registry**: Harbor
 - **Git Repository**: Gitea/GitLab CE
 - **Artifact Repository**: Nexus/Artifactory
 - **Monitoring**: Prometheus + Grafana
 - **Logging**: Splunk Enterprise
 - **DNS**: CoreDNS
 - **Load Balancing**: MetalLB + HAProxy
 ## 🔐 Security Notes
 - **No sensitive data** is stored in this repository
 - **Secrets management** via external-secrets-operator
 - **Certificate management** via cert-manager
 - **Network segmentation** via VLANs and firewall rules
 ## 🤝 Contributing
 This is a personal homelab project, but feel free to:
 - Submit issues for questions or suggestions
 - Fork for your own homelab adaptations
 - Share improvements via pull requests
 ## 📞 Next Steps
 1. **Create GitHub repository** for version control
 2. **Start with Phase 1** foundation setup
 3. **Build deployment automation** as we go
 4. **Document lessons learned** for future reference
 ---
 **Status**: 🚧 Planning & Initial Development
 **Last Updated**: June 2025
 This repo is a mono-repo that is broken up into three sections:
 - infra-config
 - apps
 - cluster
 ## Hardware
- Dell 7050 SFF (7)
+- 7× Dell 7050 SFF
- Minisforum TH60 (3)
+- 3× Minisforum TH60
- Minisforum MS01 (2)
+- 2× Minisforum MS01
- Synology 1621+
+- Synology DS1621+
 - Ubiquiti UDM Pro
-## Software
+## Software Stack
- Proxmox (Virtualization)
+- **Virtualization**: Proxmox
- Cloudflare (Domain Hosting, Public DNS)
+- **Automation**: Terraform, Ansible, n8n, Semaphore
- Unbound (Recursive DNS)
+- **DNS**: Technitium (authoritative), Unbound (recursive)
- FreeIPA (Identity management, Authoritive DNS)
+- **IPAM/DCIM**: NetBox
- Matchbox (iPXE)
+- **Networking**: Ubiquiti UDM Pro
- Red Hat OpenShift
+- **Templates**: Packer (Ubuntu 24.04, Fedora 42)
 - OpenShift Agent Based Installer (Install OpenShift)
  - [Red Hat Advanced Cluster Management for Kubernetes](https://www.redhat.com/en/technologies/management/advanced-cluster-management)
  - [Vault](https://www.hashicorp.com/en/products/vault)
  - [OpenShift GitOps (ArgoCD)](https://www.redhat.com/en/technologies/cloud-computing/openshift/gitops)
  - [Red Hat Ansible Automation Platform](https://www.redhat.com/en/technologies/management/ansible)
 ## Prerequisites
 - Ansible user created
 - Ansible configured
 - [Networking](docs/networks.md)
 - [Proxmox](docs/proxmox.md) (In my homelab, internal DNS, identity manangement, and ipxe are hosted here.)
 - Matchbox
 - DNS
 - Domain Registration
 ## Assumptions
 There are a dozen different architectures you could use to deploy OpenShift in every which way.
 For the sake of this documentation we'll assume the following:
 ## Getting Started
-[Step-by-Step Walkthrough](step-by-step.md)
+See [docs/decisions/vm-provisioning-flow.md](docs/decisions/vm-provisioning-flow.md) for the full architecture decision record.
-### Hub Cluster
+Previous OpenShift/ACM/Fastpass content is preserved in the `archive/pre-mk-labs` branch.
-You'll need an OpenShift "Hub Cluster" with access to persistant storage.
+## Security
 A Single Node OpenShift (SNO) instance, installed on bare metal, will act as a Hub cluster and run:
- Advanced Cluster Management
+No sensitive data is stored in this repository. Secrets are managed via Ansible Vault and environment variables on pipeline hosts.
 - Ansible Automation Platform
 - Vault
 - ~~OpenShift GitOps~~
 #### Network Prerequisites
 The prerequisites for OpenShift in traditional and HCP patterns are largely the same - it just kind of depends on where your DNS records go to.
 | Cluster           | Endpoint    | VIP           | DNS A Record                      | Notes                           |
 |------------------|-------------|---------------|-----------------------------------|----------------------------------|
 | Hub Cluster (SNO) | App Ingress | 192.168.0.10 | *.apps.hub-cluster.example.com    | SNO App VIP goes to IP of node  |
 | Hub Cluster (SNO) | API         | 192.168.0.10 | api.hub-cluster.example.com       | SNO API goes to IP of node      |
 These DNS entries should be put in your Authoratitive DNS.
 #### ACM & GitOps Configuration
 Before you start creating clusters you may want to create some Policies, integrate ACM and ArgoCD, etc. This step is optional in case you're just interested in trying out Hosted Control Planes or copy/paste around a cluster for testing purposes.
 Find additional details in the ./02-rhacm-config folder.
 #### Creating a Cluster
 With everything in its right place, you can now start to declaratively create clusters
 ./05-clusters/hcp-bmh - HCP to Bare Metal Hosts
 ### Internal Cluster
 Two additional bare metal nodes, to be added to Advanced Cluster Management (ACM) running on the SNO Hub. These will be used to create another HCP cluster.
 These servers have a BMC interface with Redfish - if not, then you'll need to manually manage the boot and installation of those servers.
 This makes it to where you just need 3 bare metal nodes. You could run one HCP Bare Metal cluster with both of the other nodes, but then you have a shared storage requirement that can't be satisfied by ODF since that needs at least 3 nodes.
 ### External Cluster
 You'll also either need you just need at least 2 bare metal nodes.
 ---
-## Credits
+**Status**: 🚧 Active Development — VM Provisioning Pipeline
- Ken Moini - As I used his [repo](https://github.com/kenmoini/ztp-for-you-and-me) as the baseline for this project.
+**Last Updated**: February 2026
 - Ryan Etten
 - Andrew Potozniak
@@ -1,309 +0,0 @@
 # FastPass Kubernetes Cluster - Complete Deployment Guide
 ## Overview
 This guide provides step-by-step instructions for deploying a complete Kubernetes cluster on Fedora using Ansible automation. The FastPass cluster is designed for high availability with multiple control plane nodes and worker nodes.
 ## Prerequisites
 ### System Requirements
 - **OS**: Fedora 38+ (64-bit)
 - **Memory**: Minimum 2GB RAM per node
 - **CPU**: Minimum 2 cores per node
 - **Storage**: Minimum 10GB free space
 - **Network**: All nodes must be able to communicate
 ### Software Requirements
 - Ansible 2.15+
 - Python 3.8+
 - SSH access to all nodes with sudo privileges
 ## Quick Start
 ### 1. Validate Your Setup
 Before deploying, run the validation script:
 ```bash
 cd ansible
 ./scripts/test-fastpass-deployment.sh
 ```
 This will check:
 - ✅ Ansible installation
 - ✅ Inventory configuration
 - ✅ Playbook files
 - ✅ Role files
 - ✅ Group variables
 - ✅ Playbook syntax
 - ✅ Common issues
 ### 2. Deploy the Cluster
 ```bash
 # Deploy the complete cluster
 ansible-playbook -i inventory.yml playbooks/deploy-fastpass-cluster.yml
 ```
 ## Deployment Process
 The deployment follows this sequence:
 ### Phase 1: System Preparation
 - **Hosts**: All FastPass nodes
 - **Tasks**: 
  - Preflight checks (OS, memory, CPU, disk space)
  - Disable swap and configure kernel modules
  - Install and configure containerd
  - Install Kubernetes packages
 ### Phase 2: Control Plane Initialization
 - **Hosts**: First control plane node (`space-mountain`)
 - **Tasks**:
  - Initialize the Kubernetes cluster
  - Configure API server and etcd
  - Install Flannel CNI
  - Generate join commands
 ### Phase 3: Additional Control Plane Nodes
 - **Hosts**: Remaining control plane nodes (`big-thunder-mountain`, `splash-mountain`)
 - **Tasks**:
  - Join additional control plane nodes
  - Configure high availability
 ### Phase 4: Worker Nodes
 - **Hosts**: Worker nodes (`haunted-mansion`, `peter-pans-flight`)
 - **Tasks**:
  - Join worker nodes to the cluster
  - Apply node labels and taints
 ### Phase 5: Validation
 - **Hosts**: First control plane node
 - **Tasks**:
  - Verify all nodes are ready
  - Check all pods are running
  - Display final cluster status
 ## Configuration
 ### Inventory Structure
 Ensure your `inventory.yml` has the correct structure:
 ```yaml
 fastpass_control_plane:
  hosts:
    space-mountain:
    big-thunder-mountain:
    splash-mountain:
 fastpass_workers:
  hosts:
    haunted-mansion:
    peter-pans-flight:
 fastpass:
  children:
    fastpass_control_plane:
    fastpass_workers:
 ```
 ### Group Variables
 Key variables in `group_vars/fastpass/vars`:
 ```yaml
 # Cluster Configuration
 cluster_name: "fastpass"
 kubernetes_version: "1.33"
 pod_network_cidr: "10.244.0.0/16"
 service_cidr: "10.96.0.0/12"
 # Control Plane Configuration
 control_plane_endpoint: "fastpass.local.mk-labs.cloud"
 control_plane_port: "6443"
 # CNI Configuration
 cni_plugin: "flannel"
 # Node Labels
 node_labels:
  zone:
    # Control Plane nodes (masters) - no zone labels needed
    # space-mountain: (control plane - no zone label)
    # big-thunder-mountain: (control plane - no zone label)
    # splash-mountain: (control plane - no zone label)
    # Worker nodes
    haunted-mansion: "backstage"      # Internal/backstage workloads
    peter-pans-flight: "backstage"    # Internal/backstage workloads
    # Future node: "on-stage"         # External/on-stage workloads (future)
 ```
 ## Troubleshooting
 ### Common Issues
 #### 1. Swap Enabled
 **Problem**: `running with swap on is not supported`
 **Solution**: The playbook automatically disables swap, but if it persists:
 ```bash
 # On the problematic node
 sudo swapoff -a
 sudo systemctl restart kubelet
 ```
 #### 2. CNI Not Ready
 **Problem**: `Network plugin returns error: cni plugin not initialized`
 **Solution**: The playbook installs Flannel automatically. If issues persist:
 ```bash
 kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
 ```
 #### 3. Control Plane Endpoint Issues
 **Problem**: `unable to add a new control plane instance to a cluster that doesn't have a stable controlPlaneEndpoint`
 **Solution**: The playbook uses the first control plane node as the endpoint. For production, consider using a load balancer.
 #### 4. Permission Issues
 **Problem**: `User "kubernetes-admin" cannot create resource "secrets"`
 **Solution**: The playbook uses super-admin.conf for proper permissions.
 ### Debug Commands
 ```bash
 # Check node status
 kubectl get nodes -o wide
 # Check pod status
 kubectl get pods --all-namespaces
 # Check kubelet logs
 journalctl -u kubelet -f
 # Check API server logs
 kubectl logs -n kube-system kube-apiserver-space-mountain
 # Check CNI status
 kubectl get pods -n kube-flannel
 ```
 ## Post-Deployment
 ### 1. Verify Cluster Health
 ```bash
 # Check all nodes are ready
 kubectl get nodes -o wide
 # Check all pods are running
 kubectl get pods --all-namespaces
 # Test cluster connectivity
 kubectl cluster-info
 ```
 ### 2. Manage Kubeconfig
 Use the provided script to manage different cluster configurations:
 ```bash
 # Switch to FastPass cluster
 ./scripts/manage-kubeconfigs.sh fastpass
 # Check status
 ./scripts/manage-kubeconfigs.sh status
 # Test connectivity
 ./scripts/manage-kubeconfigs.sh test
 ```
 ### 3. Deploy Applications
 Your cluster is now ready for workloads:
 ```bash
 # Deploy a test application
 kubectl run nginx --image=nginx --port=80
 # Create a service
 kubectl expose pod nginx --port=80 --type=NodePort
 # Check the service
 kubectl get svc nginx
 ```
 ## Maintenance
 ### Adding New Nodes
 1. Add the new node to the inventory
 2. Run the preparation role:
   ```bash
   ansible-playbook -i inventory.yml playbooks/deploy_k8s.yml --limit=new-node
   ```
 3. Join the node to the cluster manually or extend the playbook
 ### Upgrading Kubernetes
 1. Update the `kubernetes_version` variable
 2. Run the preparation role on all nodes
 3. Upgrade control plane nodes first
 4. Upgrade worker nodes
 ### Backup and Recovery
 - Backup `/etc/kubernetes/` directory on control plane nodes
 - Backup etcd data: `etcdctl snapshot save backup.db`
 - Document cluster configuration
 ## Architecture
 ```
 FastPass Cluster Architecture:
 ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
 │  Control Plane  │    │  Control Plane  │    │  Control Plane  │
 │  space-mountain │    │big-thunder-mtn  │    │ splash-mountain │
 │   (Masters)     │    │   (Masters)     │    │   (Masters)     │
 │  No Zone Label  │    │  No Zone Label  │    │  No Zone Label  │
 └─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 │
         ┌───────────────────────┼───────────────────────┐
         │                       │                       │
 ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
 │   Worker Node   │    │   Worker Node   │    │   Worker Node   │
 │ haunted-mansion │    │peter-pans-flight│    │  (Future)       │
 │   (Backstage)   │    │   (Backstage)   │    │   (On-Stage)    │
 │ zone=backstage  │    │ zone=backstage  │    │ zone=on-stage   │
 └─────────────────┘    └─────────────────┘    └─────────────────┘
 ```
 ## Support
 For issues specific to this deployment:
 1. Check the troubleshooting section
 2. Review Ansible logs with `-vvv` verbosity
 3. Check system logs on affected nodes
 4. Verify network connectivity between nodes
 ## Files Structure
 ```
 ansible/
 ├── group_vars/
 │   └── fastpass/
 │       └── vars                    # Cluster configuration
 ├── playbooks/
 │   ├── deploy-fastpass-cluster.yml # Main deployment playbook
 │   ├── deploy_k8s.yml             # Legacy deployment playbook
 │   └── roles/
 │       ├── kubernetes/             # System preparation
 │       ├── fastpass-control-plane/ # Control plane setup
 │       ├── fastpass-control-plane-join/ # Additional control plane nodes
 │       └── fastpass-workers/       # Worker node setup
 ├── scripts/
 │   ├── test-fastpass-deployment.sh # Validation script
 │   └── manage-kubeconfigs.sh      # Kubeconfig management
 └── inventory.yml                   # Host inventory
 ``` 
@@ -1,228 +0,0 @@
 # FastPass Kubernetes Cluster - Fedora Deployment Guide
 ## Overview
 This guide covers the deployment of a Kubernetes cluster on Fedora using Ansible automation. The FastPass cluster is designed for high availability with multiple control plane nodes and worker nodes.
 ## Architecture
 ```
 FastPass Cluster Architecture:
 ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
 │  Control Plane  │    │  Control Plane  │    │  Control Plane  │
 │  space-mountain │    │big-thunder-mtn  │    │ splash-mountain │
 │   (Leader)      │    │                 │    │                 │
 └─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 │
         ┌───────────────────────┼───────────────────────┐
         │                       │                       │
 ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
 │   Worker Node   │    │   Worker Node   │    │   Worker Node   │
 │ haunted-mansion │    │peter-pans-flight│    │  (Future)       │
 │   (backstage)   │    │   (backstage)   │    │                 │
 └─────────────────┘    └─────────────────┘    └─────────────────┘
 ```
 ## Prerequisites
 ### System Requirements
 - **OS**: Fedora 38+ (64-bit)
 - **Memory**: Minimum 2GB RAM per node
 - **CPU**: Minimum 2 cores per node
 - **Storage**: Minimum 10GB free space
 - **Network**: All nodes must be able to communicate
 ### Software Requirements
 - Ansible 2.15+
 - Python 3.8+
 - SSH access to all nodes with sudo privileges
 ## Quick Start
 ### 1. Prepare Inventory
 Ensure your `inventory.yml` has the correct host groups:
 ```yaml
 fastpass_control_plane:
  hosts:
    space-mountain:
    big-thunder-mountain:
    splash-mountain:
 fastpass_workers:
  hosts:
    haunted-mansion:
    peter-pans-flight:
 fastpass:
  children:
    fastpass_control_plane:
    fastpass_workers:
 ```
 ### 2. Configure Variables
 Edit `group_vars/fastpass/vars` to customize your deployment:
 ```yaml
 # Cluster Configuration
 cluster_name: "fastpass"
 kubernetes_version: "1.33"
 pod_network_cidr: "10.244.0.0/16"
 service_cidr: "10.96.0.0/12"
 # Node Labels
 node_labels:
  zone:
    space-mountain: "external"
    big-thunder-mountain: "external"
    splash-mountain: "external"
    haunted-mansion: "internal"
    peter-pans-flight: "internal"
 ```
 ### 3. Deploy the Cluster
 ```bash
 # Deploy the complete cluster
 ansible-playbook -i inventory.yml playbooks/deploy-fastpass-cluster.yml
 # Or deploy step by step
 ansible-playbook -i inventory.yml playbooks/deploy_k8s.yml
 ```
 ## Deployment Process
 ### Phase 1: System Preparation
 - Preflight checks (OS, memory, CPU, disk space)
 - Disable swap and configure kernel modules
 - Install and configure containerd
 - Install Kubernetes packages
 ### Phase 2: Control Plane Setup
 - Initialize the first control plane node
 - Configure API server and etcd
 - Install Calico CNI
 - Join additional control plane nodes
 ### Phase 3: Worker Node Setup
 - Join worker nodes to the cluster
 - Apply node labels and taints
 - Verify cluster health
 ## Troubleshooting
 ### Common Issues
 #### 1. Repository Errors
 **Problem**: Docker CE repository not found
 **Solution**: Update the repository URL in `container-runtime.yml`:
 ```yaml
 baseurl: "https://download.docker.com/linux/fedora/{{ ansible_distribution_major_version }}/x86_64/stable"
 ```
 #### 2. kubeadm Init Failures
 **Problem**: kubeadm init fails with preflight errors
 **Solution**: The playbook includes `--ignore-preflight-errors=all` flag
 #### 3. Node Not Ready
 **Problem**: Worker nodes stuck in NotReady state
 **Solution**: Check CNI installation and network connectivity
 ### Debug Commands
 ```bash
 # Check node status
 kubectl get nodes -o wide
 # Check pod status
 kubectl get pods --all-namespaces
 # Check kubelet logs
 journalctl -u kubelet -f
 # Check containerd status
 systemctl status containerd
 ```
 ## Customization
 ### Network Configuration
 Modify the pod and service CIDRs in `group_vars/fastpass/vars`:
 ```yaml
 pod_network_cidr: "10.244.0.0/16"
 service_cidr: "10.96.0.0/12"
 ```
 ### CNI Selection
 The default CNI is Calico. To change, modify the control plane role:
 ```yaml
 cni_plugin: "flannel"  # or "weave", "cilium"
 ```
 ### Node Labels and Taints
 Configure node labels in `group_vars/fastpass/vars`:
 ```yaml
 node_labels:
  zone:
    space-mountain: "external"
    haunted-mansion: "internal"
  environment: "production"
  cluster: "fastpass"
 ```
 ## Maintenance
 ### Adding New Nodes
 1. Add the new node to the inventory
 2. Run the preparation role: `ansible-playbook -i inventory.yml playbooks/deploy_k8s.yml --limit=new-node`
 3. Join the node to the cluster
 ### Upgrading Kubernetes
 1. Update the `kubernetes_version` variable
 2. Run the preparation role on all nodes
 3. Upgrade control plane nodes first
 4. Upgrade worker nodes
 ### Backup and Recovery
 - Backup `/etc/kubernetes/` directory on control plane nodes
 - Backup etcd data: `etcdctl snapshot save backup.db`
 - Document cluster configuration
 ## Security Considerations
 - SELinux is set to permissive mode for compatibility
 - Firewalld is disabled (configure as needed for your environment)
 - Consider enabling RBAC and network policies
 - Regularly update Kubernetes and system packages
 ## Support
 For issues specific to this deployment:
 1. Check the troubleshooting section
 2. Review Ansible logs with `-vvv` verbosity
 3. Check system logs on affected nodes
 4. Verify network connectivity between nodes
 ## Files Structure
 ```
 ansible/
 ├── group_vars/
 │   └── fastpass/
 │       └── vars                    # Cluster configuration
 ├── playbooks/
 │   ├── deploy-fastpass-cluster.yml # Main deployment playbook
 │   ├── deploy_k8s.yml             # Legacy deployment playbook
 │   └── roles/
 │       ├── kubernetes/             # System preparation
 │       ├── fastpass-control-plane/ # Control plane setup
 │       └── fastpass-workers/       # Worker node setup
 └── inventory.yml                   # Host inventory
 ``` 
@@ -1,233 +0,0 @@
 # Kubeconfig Management - Modular & Repeatable
 This document explains the modular kubeconfig management system for your homelab's multiple Kubernetes clusters.
 ## Overview
 The kubeconfig management has been refactored into reusable, modular components that can handle multiple clusters seamlessly:
 - **FastPass** (Vanilla Kubernetes)
 - **Hub Cluster** (OpenShift SNO)
 - **Internal Cluster** (OpenShift)
 - **Future clusters** (easily extensible)
 ## Architecture
 ```
 ansible/playbooks/roles/
 ├── kubeconfig-manager/          # Core reusable role
 │   ├── tasks/main.yml          # Main entry point
 │   ├── tasks/merge-kubeconfig.yml  # Merging logic
 │   ├── defaults/main.yml       # Default variables
 │   └── README.md               # Role documentation
 ├── cluster-kubeconfig/         # Generic wrapper role
 └── fastpass-first-control-plane/  # Uses kubeconfig-manager
 ```
 ## Quick Usage
 ### 1. Using the Script (Easiest)
 ```bash
 cd ansible
 # Setup FastPass cluster kubeconfig
 ./scripts/setup-kubeconfig.sh fastpass fastpass_control_plane[0]
 # Setup Hub cluster kubeconfig  
 ./scripts/setup-kubeconfig.sh hub hub_cluster
 # Setup Internal cluster kubeconfig
 ./scripts/setup-kubeconfig.sh internal internal_cluster[0]
 ```
 ### 2. Direct Ansible Usage
 ```bash
 # Single cluster
 ansible-playbook -i inventory.yml \
  playbooks/examples/multi-cluster-kubeconfig.yml \
  --limit fastpass_control_plane[0] \
  -e target_cluster_name=fastpass
 # Multiple clusters at once
 ansible-playbook -i inventory.yml \
  playbooks/examples/multi-cluster-kubeconfig.yml
 ```
 ### 3. In Your Own Playbooks
 ```yaml
 - name: Setup kubeconfig for any cluster
  hosts: my_cluster_nodes[0]
  roles:
    - role: kubeconfig-manager
      vars:
        cluster_name: "my-cluster"
        kubeconfig_source_path: "/etc/kubernetes/admin.conf"
 ```
 ## Features
 ### ✅ **Modular & Reusable**
 - Single role works with any Kubernetes cluster
 - Easy to extend for new clusters
 - Consistent behavior across all clusters
 ### ✅ **Safe Operations**
 - Automatic backups before merging
 - Non-destructive merging
 - Preserves existing contexts
 ### ✅ **Multi-Cluster Ready**
 - Unique naming: `cluster-name-admin`
 - No conflicts between clusters
 - Easy context switching
 ### ✅ **Homelab Optimized**
 - Works with vanilla Kubernetes
 - Works with OpenShift
 - Handles different kubeconfig paths
 ## Generated Structure
 After running the kubeconfig management:
 ```bash
 ~/.kube/
 ├── config                      # Merged config with all clusters
 ├── config-fastpass            # Individual cluster configs
 ├── config-hub
 ├── config-internal
 └── config.backup.1697123456   # Timestamped backups
 ```
 ## Context Management
 ```bash
 # List all available contexts
 kubectl config get-contexts
 # Switch between clusters
 kubectl config use-context fastpass-admin
 kubectl config use-context hub-admin
 kubectl config use-context internal-admin
 # Check current context
 kubectl config current-context
 # Test connectivity
 kubectl cluster-info
 kubectl get nodes
 ```
 ## Integration with Existing Roles
 ### Before (Monolithic)
 ```yaml
 # fastpass-first-control-plane/tasks/main.yml
 - name: 50+ lines of kubeconfig logic
  # Lots of repetitive code
 ```
 ### After (Modular)
 ```yaml
 # fastpass-first-control-plane/tasks/main.yml
 - name: Setup kubeconfig for FastPass cluster
  ansible.builtin.include_role:
    name: kubeconfig-manager
  vars:
    cluster_name: "{{ cluster_name }}"
 ```
 ## Adding New Clusters
 To add a new cluster (e.g., "edge-cluster"):
 1. **Add to inventory:**
 ```yaml
 edge_cluster:
  hosts:
    edge-node-01:
 ```
 2. **Use the script:**
 ```bash
 ./scripts/setup-kubeconfig.sh edge edge_cluster[0]
 ```
 3. **Or add to playbook:**
 ```yaml
 - name: Setup kubeconfig for Edge cluster
  hosts: edge_cluster[0]
  roles:
    - role: kubeconfig-manager
      vars:
        cluster_name: "edge"
 ```
 ## Customization
 ### Different Kubeconfig Paths
 ```yaml
 - role: kubeconfig-manager
  vars:
    cluster_name: "openshift-cluster"
    kubeconfig_source_path: "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/localhost.kubeconfig"
 ```
 ### Custom Context Names
 ```yaml
 - role: kubeconfig-manager
  vars:
    cluster_name: "prod"
    context_suffix: "system:admin"  # Results in "prod-system:admin"
 ```
 ## Troubleshooting
 ### Common Issues
 1. **Permission Denied**
   ```bash
   # Fix ownership
   sudo chown -R $USER:$USER ~/.kube/
   ```
 2. **Context Not Found**
   ```bash
   # List available contexts
   kubectl config get-contexts
   # Check if cluster was added
   kubectl config view
   ```
 3. **Backup Recovery**
   ```bash
   # Restore from backup
   cp ~/.kube/config.backup.1697123456 ~/.kube/config
   ```
 ### Debug Mode
 ```bash
 # Run with verbose output
 ansible-playbook -vvv playbooks/examples/multi-cluster-kubeconfig.yml
 ```
 ## Benefits for Your Homelab
 1. **Consistency**: Same process for all clusters
 2. **Maintainability**: Single role to update/fix
 3. **Scalability**: Easy to add new clusters
 4. **Safety**: Automatic backups and safe merging
 5. **Flexibility**: Works with different Kubernetes distributions
 ## Next Steps
 1. **Test the modular approach** with your FastPass cluster
 2. **Extend to Hub and Internal clusters** using the same pattern
 3. **Create cluster-specific variables** in group_vars if needed
 4. **Add monitoring/validation** tasks to verify kubeconfig health
 This modular approach makes your homelab's multi-cluster management much more maintainable and repeatable!
@@ -54,7 +54,7 @@
 # (pathspec) Colon-separated paths in which Ansible will search for collections content. Collections must be in nested *subdirectories*, not directly in these directories. For example, if ``COLLECTIONS_PATHS`` includes ``'{{ ANSIBLE_HOME ~ "/collections" }}'``, and you want to add ``my.collection`` to that directory, it must be saved as ``'{{ ANSIBLE_HOME} ~ "/collections/ansible_collections/my/collection" }}'``.
-;collections_path=/Users/rblundon/.ansible/collections:/usr/share/ansible/collections
+collections_path=/opt/ansible-collections:/usr/share/ansible/collections
 # (boolean) A boolean to enable or disable scanning the sys.path for installed collections.
 ;collections_scan_sys_path=True
@@ -209,7 +209,7 @@ private_key_file=~/.ssh/ansible
 remote_user=wed
 # (pathspec) Colon-separated paths in which Ansible will search for Roles.
-;roles_path=/Users/rblundon/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
+roles_path=./roles
 # (string) Set the main callback used to display Ansible output. You can only have one at a time.
 # You can have many other callbacks, but just one can be in charge of stdout.
@@ -262,7 +262,7 @@ remote_user=wed
 # (path) The vault password file to use. Equivalent to ``--vault-password-file`` or ``--vault-id``.
 # If executable, it will be run and the resulting stdout will be used as the password.
-;vault_password_file=
+vault_password_file=/home/hermes/.vault_pass.txt
 # (integer) Sets the default verbosity, equivalent to the number of ``-v`` passed in the command line.
 ;verbosity=0
@@ -0,0 +1,6 @@
 ---
 - name: Create jarvis user and deploy SSH key
  hosts: all
  become: true
  roles:
    - jarvis_user
@@ -0,0 +1,193 @@
 ---
 # ============================================================================
 # Semaphore configuration-as-code
 # ============================================================================
 # Drives a freshly-deployed Semaphore instance into its desired state via
 # the Semaphore REST API. Idempotent: every object is checked first; only
 # missing ones are created. Existing objects are left alone.
 #
 # Loaded from group_vars/all/semaphore.yml so that the configuration is
 # version-controlled in the homelab repo and survives a wipe-and-redeploy
 # of the Semaphore VM.
 # ============================================================================
 # ---------------------------------------------------------------------------
 # API connection (defaults to the local Traefik-fronted service-name URL).
 # Override semaphore_api_url to point at a specific instance if needed.
 # ---------------------------------------------------------------------------
 semaphore_api_url: "https://semaphore.local.mk-labs.cloud/api"
 semaphore_api_validate_certs: true
 semaphore_api_token: "{{ vault_semaphore_api_token }}"
 # Feature flag — keeps day1_deploy_semaphore.yml deploy-only by default.
 # Set true to also run the configuration pass.
 semaphore_configure: false
 # ---------------------------------------------------------------------------
 # Declarative configuration of the Semaphore instance.
 # ---------------------------------------------------------------------------
 #
 # Top-level shape:
 #
 #   semaphore_config:
 #     project:        single dict — the lab uses one project ("mk-labs")
 #     keys:           list of credentials Semaphore stores
 #     repositories:   git repos Semaphore can clone
 #     inventories:    Ansible inventories from those repos
 #     environments:   env-var bundles
 #     templates:      task templates that tie everything together
 #
 # Each list element has a unique "name" used as the natural identity key.
 # ---------------------------------------------------------------------------
 semaphore_config:
  project:
    name: mk-labs
    alert: false
    max_parallel_tasks: 0  # 0 = unlimited
  keys:
    # The ansible-vault password. login_password type with empty login
    # — only the password field is consumed by Semaphore at runtime.
    - name: ansible-vault-pass
      type: login_password
      login: ""
      password: "{{ vault_ansible_vault_password }}"
    # SSH key for the gitea deploy access (clone the homelab repo).
    - name: gitea-deploy
      type: ssh
      ssh_login: git
      ssh_private_key: "{{ vault_gitea_deploy_key }}"
    # SSH key for the universal automation account 'wed' — pre-baked in
    # every mk-labs VM template. This is the canonical user Semaphore
    # uses to reach the fleet.
    - name: wed-ssh
      type: ssh
      ssh_login: wed
      ssh_private_key: "{{ vault_wed_ssh_private_key }}"
    # SSH key Semaphore can use to reach the fleet as jarvis (admin
    # account provisioned by linux-baseline). Retained for jobs that
    # specifically need jarvis-level access; the default is wed-ssh.
    - name: jarvis-ssh
      type: ssh
      ssh_login: jarvis
      ssh_private_key: "{{ vault_jarvis_ssh_private_key }}"
  repositories:
    - name: homelab
      git_url: "ssh://git@gitea.mk-labs.cloud:2221/rblundon/homelab.git"
      git_branch: main
      ssh_key: gitea-deploy
  inventories:
    - name: production
      type: file
      inventory_file: ansible/inventory.yml
      repository: homelab
      # wed is the universal automation account pre-baked in every VM
      # template. Semaphore uses it for fleet-wide jobs.
      ssh_key: wed-ssh
      # become_key is Semaphore's sudo PASSWORD slot, not a second SSH
      # key. wed has passwordless sudo on every host, so reference the
      # built-in "None" key. (Semaphore rejects an SSH-type key here.)
      become_key: None
  environments:
    - name: default
      env:
        ANSIBLE_HOST_KEY_CHECKING: "False"
        ANSIBLE_FORCE_COLOR: "True"
        # Semaphore runs ansible-playbook from the cloned REPO ROOT (not
        # from the playbook's directory as I first assumed). Path is
        # therefore relative to repo root, not playbook dir.
        ANSIBLE_ROLES_PATH: "ansible/roles"
        # Collections are installed by the semaphore role into a host-side
        # directory bind-mounted into the container at this path.
        ANSIBLE_COLLECTIONS_PATH: "/opt/ansible-collections"
  templates:
    - name: "day0_linux_baseline"
      description: "Apply the mk-labs Linux baseline to one or more hosts."
      app: ansible
      playbook: ansible/playbooks/day0_linux_baseline.yml
      inventory: production
      repository: homelab
      environment: default
      vault_password: ansible-vault-pass
      arguments: '["--diff"]'
      survey_vars:
        - name: target
          title: "Target host or group"
          description: "Inventory target (e.g. figment, semaphore_server, all)"
          required: true
          type: TextVar
          default_value: "all"
    - name: "day1_deploy_semaphore"
      description: "Re-deploy Semaphore + PostgreSQL on figment."
      app: ansible
      playbook: ansible/playbooks/day1_deploy_semaphore.yml
      inventory: production
      repository: homelab
      environment: default
      vault_password: ansible-vault-pass
      arguments: '["--diff"]'
    - name: "day0_linux_baseline_check"
      description: "Dry-run the baseline — shows diffs, applies nothing."
      app: ansible
      playbook: ansible/playbooks/day0_linux_baseline.yml
      inventory: production
      repository: homelab
      environment: default
      vault_password: ansible-vault-pass
      arguments: '["--check","--diff"]'
      survey_vars:
        - name: target
          title: "Target host or group"
          description: "Inventory target (e.g. figment, semaphore_server, all)"
          required: true
          type: TextVar
          default_value: "all"
    - name: "llm_inference_multimodel_stage_models"
      description: >-
        Stage additional GGUF models into /opt/models on astro-orbiter via the
        llm-inference-multimodel role (--tags models only). Idempotent: skips
        files already present at the correct byte size. Notifies the
        llama-server-router restart handler ONLY when a new GGUF is actually
        downloaded. Does NOT touch Phase 4 (verify) or the legacy
        llama-server-qwen service. Safe to run repeatedly.
      app: ansible
      playbook: ansible/playbooks/day1_deploy_llm_inference_multimodel.yml
      inventory: production
      repository: homelab
      environment: default
      vault_password: ansible-vault-pass
      arguments: '["--tags","models","--diff"]'
      # Scoped to --tags models:
      #   Phase 0 (discover) -- skipped (no tag)
      #   Phase 1 (models)   -- RUN (idempotent GGUF staging via stage_model.yml)
      #   Phase 2 (systemd)  -- skipped
      #   Phase 3 (firewall) -- skipped
      #   Phase 4 (verify)   -- SKIPPED (collision risk: verify.yml would start
      #                         llama-server-qwen on :8002, conflicting with the
      #                         production llama-server-router.service. Excluded
      #                         here deliberately. See t_730f9584.)
    - name: "llm_router_update_unit"
      description: >-
        Re-render and reload the llama-server-router systemd unit on astro-orbiter,
        then restart the live service so new args (e.g. --models-max) take effect.
        Drives playbooks/day2_bump_router_models_max.yml. Added 2026-08-12 (t_33acbb2e):
        bump --models-max 1 -> 4 with full VRAM budget note in host_vars.
      app: ansible
      playbook: ansible/playbooks/day2_bump_router_models_max.yml
      inventory: production
      repository: homelab
      environment: default
      vault_password: ansible-vault-pass
      arguments: '["--diff"]'
@@ -0,0 +1,4 @@
 ---
 step_ca_url: "https://turnstile.local.mk-labs.cloud:9000"
 step_ca_fingerprint: "f63c44e76381e359978bd2dca07c928d04ad44b575f0364fa66b5725c7e7891b"
 step_ca_provisioner_name: "admin"
@@ -18,3 +18,30 @@ terraform_server: "infra01"
 # Traefik variables
 traefik_server: "lightning-lane"
 # ---------------------------------------------------------------------------
 # JARVIS automation account
 # ---------------------------------------------------------------------------
 # Public key for the 'jarvis' user provisioned by the linux-baseline role on
 # every host. Public keys are not secret; the matching private key lives on
 # the JARVIS command centre (carousel-of-progress) and, when needed, in
 # group_vars/all/vault as vault_jarvis_ssh_private_key.
 jarvis_ssh_public_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID5sym5ajFvDyzw395BkHv7qVb66XPTx/OF1p19MGuNo jarvis@mk-labs"
 step_ca_principal_mappings:
  - local_user: wed
    principals:
      - ryan.blundon@protonmail.com
      - ryan.blundon
      - ryanblundon
  - local_user: rblundon
    principals:
      - ryan.blundon@protonmail.com
      - ryan.blundon
      - ryanblundon
 # Leviton My Leviton API
 leviton_email: "{{ vault_leviton_email }}"
 leviton_password: "{{ vault_leviton_password }}"
 jmri_vnc_password: "{{ vault_jmri_vnc_password }}"
@@ -1,170 +1,427 @@
 $ANSIBLE_VAULT;1.1;AES256
-64393362643437353731353234643535366431613861636131613139636435633465306333656161
+31356235343830393937633039386366623238373864353438353332653038616330623365343135
-3365306632386330373862363637363435633235643336610a663736396537313032336162633334
+3961393463613335663138383233323964393832626466620a393763616364363030323833386565
-32626233346233323565383030393130316532653062373730376163326134616239663865363862
+30666333366332313262313836613962363537353065646361316230386434366534666338336636
-3365316365643534610a343932623566356637663136336338386534373532383863363662326438
+3930363231393333610a613039646430663233613163376131333637383866656564333765633565
-37393331356632313163643138613330366631643465623633386433663236636334666264383033
+32333338316539363034316132313631643632316337356164373765373736366639623362343738
-34333935633131363364626333313336353733373562326636343633653838616532643737366435
+63326164623535326461323234333933656132313836643139316339333831323331393030323233
-36366330633263613938323566393063373233363538643531636131316335316461656631616362
+34643932626464353861616534326130373362646437613131326464313737623965366361616266
-30343965356363383139366534363664313831363132383337636266316538393139383232386230
+37393934333761303163346465366464393964313831393266303434313061356333373430346339
-65353364313534333632306465393264386466346365303232356263653563653338306131376131
+33393331393461303163306639386239656139613833386235363335333837316666323637346462
-66343764643639396663623330623332333466633737343738386230626463386137373536343430
+35346634383936333631303338356433323865623030663336323663373962623531313562626666
-36636430616132363161303338353033343963643338656639643662636664613965636263366135
+33393063613137646631666363386161636462343262313935386535386361346464396133613736
-64353134376162633162316166316239333963653930363830646165636537653238623537323238
+31383464613130353034396264643766646161343664363435383639656234623035343833653832
-39376338636335336338323935333439306534386539336331613961663261623962383237303632
+63356130646136396463636533616334643638333239636437663030393064326362393736373233
-34326139626366303331363337353432623636626231373731646665623831353865303130663533
+34396166653932326632376535373762663633303261303536396334303937316431346133353830
-37326639626237653361316666616161363531303961613538636663633063396363346337663239
+37626337313861306164366130643131383133323337643637666535316132373238373066306339
-64313333653766383736303161636530316538633033323063323662336335666264646564643465
+38343865613937663366376630333365633039333430653762313864303737633661356232373133
-64396436366131373061326233643562626335376263623031376532313730653363343038356135
+64303934623937313863323431396638383932383836613830336235656530613234316238353364
-30336131646635636638353934636139343662623265336338386535663536393935343132306335
+32306561343362643431623239333638316239323164386634623736313261313936616239396431
-33626632613663353661313061623732643539323637336433666330313463666661623836303462
+64373934313663313139316331623436663832393131373562393436326265656562363137636461
-61383732613139343434363662616539373837633139393764353864613062346237346661633033
+33613864356464366239306639653264363565306331636137396466336163336631353163383430
-61393639326535366266643664653439383238646132306636656130333733316638363830346433
+30303034383136356465643238663034623036363562353037343832616136363539656432346639
-38643430346263663639653762393935356662363533323766343362366532626532633530323630
+34636536383533333538346237653731633331643137303966316138333633373564383034383438
-30643436653130363066373665666434383732663837643839356539366432316539613863623839
+34666465323032356333613935363161306431666533353061626536633839396234613662623062
-35323930383437383535373330326239663966383234663233643337623330353236626464636530
+63313338346138653765393537356639373035643935373735323166376336626166396562316535
-36393735646664366235363831656262383333626265383066393363346165656265363438306135
+39343236356665363038326132313437303236646161333638393939653033623664306563613730
-34633530653837346638636665666162363062663333623263356131646661373234333939333536
+34383538656139646436333765623861303739613939633962316139386463643666366464303935
-30643639333836373439613561346130623239363737626238663436356365633562366465643561
+32366239656366393832376436623534653539376661363737343865663633316138663061643866
-32643039376234323963653733353138643138626430383639313765353035646539653466623539
+65393930386162393663303135396237623633656230303036646431633236376465633835336364
-32366135366266323261333938373837386636626232323661393035613638643335383564663462
+63643332616433303766396561386433363836343836663430636530616135653966336131383533
-31383663663063373935396439653231353962303630346338623938343064643037396562626637
+64613933306165303564303835616132613839636432653137356131343732343736313237356436
-63363363343739626639633934393734396434653262373137373539316536306663336466626639
+35663033353163346363366635333231366432316434306362616265653832313963303338343538
-34646465306439313239323365333761663161313461613766343463666134303036376338366663
+61363966306162323932393037626335613133363936393166626433653463363661343732306465
-64356364653462383534343438343933623766396232336434653765613864376235363735343032
+32373839653865326333613165353461613531383666323938653034623936303562363331643363
-32313133383731333366626534363065333738346230393636393938366633363561346436363338
+37326164323935626239396239646465616464613332636662353539363233343032616636613237
-33393932626333636166313431633339386564616466653165383962656633303161393935366635
+62316663643134663038626535663262336266646565336236643234663561313264626238313136
-65396464326333396433303966316231376432363335643732303438663131653932636564363264
+65383364643537393436666465653437653334663934333765626233303835643732373334663639
-36356663623436643835373538363662643134313030656236326335393334626634663136333137
+62643064383535366630306666383365353963306563323533653664643132383832346435323465
-38363432346663393435366536636335636465343131666562396334636530346235366365396431
+30326466623265326437383063353833383131393166313231633032333137326563633734636263
-64623739373130623436356161666139656330646661396430633264393632663366353037643130
+61623466613436393237656435623335623133343032333535373430633836346566353039646238
-35373839373430643065616630613433646536356563353133633361623235623733313234306435
+34363338343065323765313466306130643462303061336265623833343139353336383234366362
-64356635343361633462376131636362386433373339383365313436333332613333316237323632
+30383465376635356462313533643961613437333930396134626463383464323661396431643234
-33633362376331323930376130393838633333316330363362396131306230643161373636303034
+30353737396262653336376137646137353362353066366634613034313938613234323931333365
-37303738306233363937316332313830396336323639333634666364633566386261373031383563
+39323733613864346132633461323563396161643738333634316564353934376537353663386636
-36633865643634363234613463636437306236376435613034623537356162353235396665636338
+36373335343439643432396266386665396266643063616331666438363066643162356265373732
-31613733363430313532643235343438393137313136653730643364623662336236323464656465
+31616537656266363866656133316663383634343735616266653161613139636164653566303666
-65636431633464613437396561643638346565313937303731633837623930333538383335633938
+35396566653136343038636239626561303966346636333635363836366538616233386164316633
-62666461323033303937653366646432623764653566626333333766656235633662623631636636
+30626135303434636335623366383566366532313937313133313664313462316438323964376130
-35626161653736303261303233643236393038303063386532643563663065303331616637386661
+61363166306232363737616432316666623166313962366462656464626537323538316337626635
-33326637393633616430643665623431616336336666383236326538383535633364613135653131
+64366533383264356430343066373664646534396538306335333861356237346164373235643239
-38383961383264313130383263613763326666663033626338313762376336353866646634323537
+31633264396338303061376633376564356231336338376135386639393031656437363662613861
-39393565643265633437396535643165626165646537373039613061613936356434366238636562
+33373934313862396238353863663530633337383633623565623937623734616664626534383061
-61393639316531316564646334663863383836656261623230646461323566316664653965336136
+34353862393537633932393338653265353633643166383564363766636539636334643534333736
-62353839363664396632306231363031303265343131343739633835353338386335343737326633
+62666332313730666661303462656162303338326433393934313536326365663235346263313330
-36656433616135653663353166636461623337653138613435646663663939323631373063373136
+34623539383031653538333762396361336235643936386366323235346162326635393062646263
-63396331313538636437383635643831643935303932313964336532643366623434316662376632
+36343638613134353966363261613632313963363236326364366133353838616636323138316561
-63666464333736333162373461333563393138366638643039393865613238616531356165346236
+63356237663864333334343461396364656231333939636264343966343266346430363965633764
-65313432383533353738396530356465386263313939393962333637306138653939646638353235
+38636562626366636438343538666632643964393463353836323536333961613961613166373963
-37393733333032623763333632353866633866613162663833346534666237383430663838383965
+62356336396536613238353531616237616334616231363836616561333866643033666336336439
-66663332616164353539323261383364363237383064306132633530326134383561346536346636
+65646362393933306534326137396561346338336263383134383263373836323034653932646463
-32393365316435626464386631663037663934313533663030313466323134336439303362626365
+62326364336438623137623862386438396164643432333135346462363761333437653532373361
-30613764346362326130353333353663316539306437376465306332353464343530663933363562
+32306161313238363365633131663334376135613336363937313263663537313733313762656134
-39646333323631353835336461326233623963353535316633353236353236636462313163633638
+33613838326137343061306431363237373661333163373739326339303732623131393633646565
-35366538366336663534353739353231306164626664383737333565666337636164386539316632
+38663963663338623362653364343337383636323730306431313830316261666264326464303235
-64323966336637626538303830323535373533306362636236303065663161353734653633393064
+64313662326434646366386166396464653666323735323931343436653763363535396636613938
-66353938306238646238313037323163343935643863653632336462363834326634363433383338
+37386435303139663965623963383861353030626530653061636132326233373165613231316232
-35646364353834376439633136313939393064633539643462623562333832303261386331636533
+38616539303630316565366161383664316337333665323631663430363764313736643934623638
-37373837396461373064353565353438633938616162363138323561343662383330346633323133
+35346264666461623533623963306238613530633034333662653561353030646437346537623065
-36626665343636343739353732366133313434623230353732323037633236303237656364323763
+37376263366630666430383833373966353730396239626133333330306232643037643231336433
-30363530306131303630386131623866323861633636313431376161336536373436656334303036
+62653332623461653565376531386464353237383665323538363061633831353534656366333933
-61316639666136313338396233326234386634343863633632626264363336376531363461626564
+37373235376564613665303462393634623836663835383738356138323136316432663637353434
-35336335396561626339616234373265663032353432363832306666353339376338396366383839
+36336338623364383334363565643334613631386330633466396336336539383762383162363631
-62633063326135323539653133626538613331613334353430623232376336633334666433316638
+32393230623938623433333632663365313165306165643433643131366237366131616263333465
-61326563366432333632666436363761376237636434356231613061626633303737626531626236
+38333739636234326139393361386131383232373362343636653862356535613230343232333365
-62373166363432616666616630616565373934356564363266396662323963636434313735626161
+30316261623036373531383238373164313961386532393663643736363131623533373366613565
-61373738623966363032313632376465356235363834633639663733653131636531363563326662
+37653133383930663037363232353032616133653763306564383434366233613066373331363162
-62663365323934363761653536356162336463646566393838346134356536303264323065396534
+61366530303731333235313537366436363237303561336232663363396562313735613461363331
-63346531303764613634646336313530326663373433396436663162343530306131386265303137
+30336561643035336430643666643565653538616536646263303265663739653132353233343834
-32333432383966363735343331646233613631373165346131323632366165346235656330323939
+36613332313430313138636464393139316466373738313330383762623963376363663839386366
-34393030313466666265346163323561353734613434386239333630623762616566616562316265
+36636265383436643661653465383561306663303461633335373331323733663936373334353164
-38633263323339396462336166646431363336616462366463383663313839613638383663303133
+35353138306233626363616636633236643130373432636161646563333937623334636163653637
-32333835623531366236643530623961656631336139313037353962323937666465343430666336
+31613431363963396266303637363833303635376239356337613935636165326265303166383062
-33383464363930663630333164636361326334396163376661666564663538623732663839613237
+32326165636331356466363766643766363266666530363739383535623863393135663331623737
-35623237613761356235383130633733333766333236363537383862306539373665383461386534
+61303261313563643665626435343362656632363263656235316230646135616163346630363335
-32646432363339306433646139393732366464336339346161323736363937383566656663356230
+36633564383761346163383166326562313464316362613531633565646365326538356565386630
-65316562383562366366653466646461613235353061623363386331636139386136306131633232
+35396334353834343733643931376261336435373362386339353138386435663961623064343438
-32326238353861613236373964383131363537653632663532376364336166303662663336623161
+34633639383037653437653461346563373733663536313862386262393933623434313163313136
-37623838616664376561663763393534323931343163376535396662626333383730363232653036
+64333531656462353966353362646636353634646433333361633166653338373938326639343732
-39626266623666346635373630343135333532353866363234396265356631333265663432326235
+30313163613532633163663965393434393735383730653430663762336262393564303263643935
-37346439613363313334633964386632336138316662616661356637363730306164646439643765
+32313564393835353762363662373631643430393938643265393462643864633465643636306664
-62343137663137633031316665363362643165643731326464323962616330383164353861366462
+32363936303737363934373536383136613636323538363565306563653438343961373566363463
-31663965626338636534356461366362333831396464376134613331393664306639386632396236
+34323338636161316537383036643638366430303964376435356164333666313833303264666464
-66633066663831646361333730356361646133383834376230356336613038303665323732663162
+64383766356630313130626235663364653730666638363336353938323735356434336539383132
-37643130313739306638636262656632353431316531336438363636363833626237346363326461
+61633334383661613039373136613735306662633734346538663838653132343066313166333834
-31636262396632666335303566633161333937626330373835613236656130383132663863636364
+33386533306466613037383636636432363435616266373664636631623264396538626366616164
-34383661353966346362386461303062303138626536323163376362326639356336396437333530
+63363831616562653034666231386365666639343330303964313165616462623865663266326531
-66623765313638616430383838396164336565646338636430363262396636663965383531383139
+38306337376133613932656631393532613431323964643532386234303335643731373265643536
-35356663363437383132333566336131666139643639663662633363616133323031353032663438
+39336233363530313533663163616462333466656364666262313132636633613863343766656465
-39393266633639306363313530643239303132303264323063306335346233363233373865653863
+38393339386335623234646437393061366230643432636163306432633365356330306234396530
-65396430346630626566383963356264313630623336316236393730343437656435656664663266
+37346466333961383538356236303630353131376330613437656536393766383062323235643465
-32643639363536396266626435373363303234623230383035326536633631633737653739366563
+38663034393633383661363632386431343364333664396530323066306365366539366336333265
-65653932663539353436366139313635663962663330343461353864323166323733366163653162
+34343865633834333564343237336661613437363333373032393661633034643161623461306362
-66353866376134633534323366656265633930356132316137616235343866653265666161363266
+33613663613966636235333664313731333064613033613639656232626365366461373535376532
-30386638343739353965643466613639303666316432323634383464633431363566333831643134
+37613365393237333037653730323432386162616165663335653565666565656438373337376232
-65363138313439366430613837396337323436636530313664303964353966306565663630323662
+35663933646636336535653431613337313562386137363137616561346435636562656337653265
-33643639623935336235373332633739306539656235323439343331376632373538613534386430
+62396430663033373234366535633838306335306263616561323438343736326664653436663432
-30633832656532623262326662376237393262333764623937343232383861636563633730393765
+61396237383030316436306130663039623963343362326465333933373061643165353465383761
-39663032373132323735633361343933333332623465343863636263363161303165656234313338
+66336433356538313761616663363438316165653763306431353139656361346631336538343936
-65383462626465376162306639626531366161396531643364656539366233306166386234313761
+37643363363662626362643333613431363032323938396365373337643235353333393232373862
-30633666303832636561323236373363373137353032613637353637386238353234646431333661
+38316664383838623836343938663231623237306566353036633265316361636565366134646562
-32313834666237626333626437663562646438363561393364373565313536363535323536363537
+61383161313664633736613463393035363637633836636634363231326230613437663438343362
-35633834616136656261643130656131306264333131326431626439343765393031313964616639
+64313362653131316433303561366536373436383865333138376663353035343731363934633537
-33643933653033336436616635346335383335333334343836343638613264396462633530623866
+37663832316265356535643561623435336563663362663766306662326564383865373565333134
-30303462653130353430663862623261343866383335343239393435633962383864396430313536
+37343035363763373236656239376665663536366131306439643331663731656234386432653665
-66643535303437383232323564326266356636376464623563636139633237313661356330633931
+30393632653939393564363634373334623865626231653639356633653136363630306539356361
-66376631373366613731633863666132626366393532666331343361323764626539666238656262
+34396238393339616333393734633835626463353932616262383263336439366262303966333666
-66356332383235353864303562356638636130343938383530376432663866316238333434616233
+39666536363537633030393730343037623265643438313166333064303635663430323435383438
-32316432333064663764363831343464326334353737346632383565643737366233396139333639
+31313364336662363365303262653566303830633930666639656234373037366238643431626562
-35656435346335323030636363356633633332376138653930613037313537373038353433363532
+31373137663434366564373833656233636165636636653738383035393936343063303761353265
-39346161336261353463656430663033623036396365346638323832313361633866363134396165
+66393938643238393338343930363233326463643237633137393566316438393266613731346162
-65356632366463393435386366643232383162623536653965643235323934633464373061333237
+39393630316164613139393436316533333632366638663437366266323837633937613539643434
-32636564386662626365613235333664323563633466383238343336343063376531396161343530
+38316132306333633666633433623565323463636664336536616266306262653064363931316165
-61306632646133626133343132666534666337336239306137663931623637643164326538646138
+34383233343964326363363235343165653339653162326161323932373631353665353164653731
-63313037336235316164316232636262316331356363356438653235346539643065636537366661
+34333665313065306439393431336632356536646334376562646232353938313836313735616238
-37313437386432316336663366633366666434346135356462623635313531333530653866663863
+63623765313564353463653133333536343937663165396463303531616365616363383938313939
-31646438316435373637303733646564316261336233623035353564363664303437373132353166
+37373530363830386338393661306261616432363737303236613232616232643666653464613937
-33646534663130356330663565656663333765653961663835623235633665623464353862616138
+35313631616166386330653730653062646234333432336135316235356166343139316538353337
-37343531366462616439333539306265626164636233616262616563386566396536386264343137
+38663265633166383563623561656632386364663365326662323036316436386639663761323861
-30396434303231366664656334656635616334633066323062313738363431646335643132626331
+33313937343632626561663233323862636335373930366233326163623831363765643236666463
-33663038316165626365346366363731323639653837616163653561623034393064303563353232
+65386234646363313039646266636435366462383266396337383439623437306166663536363966
-30613939353336396365613662393938663933363662656634663465303134393964623738633765
+34333762366161373531633036356139646338363031366561363863393830353336343337363664
-37633832626631363736383436666263356531366237356261666538303863343033356333656138
+32366263343464663338306562636664333035616165363538333764616139663063326232326664
-64366537363939346535313732643965623332396265386636353364316630333130643833383331
+63623264303965353632303230653636313162373061396436643064353632323731633435623232
-34643563386539383235386130323065353835376564613835313934333636316464636463376537
+63313263616339613065313138613032626635646236353462306263376638643966313332393939
-32363565333038643230656266393761313064616537393830343163643535646436616230626166
+36643637653261663830313564316364643832383330353165613666333733646138633432633763
-36373531646536363664663330353864636533656261353331316364633361633934326436323933
+61633431396432313863343466306266666133646266336536646635666631613633346430633539
-39636631623430313934393033373630656132623734363835353735353935393063356261323764
+63313964353937386636626532303532353133366237383566393061326265316163613665326131
-66346265376339623265376666343865343763393833323564643161613861303065346137653339
+38613232366132313762393436646265326361323736643338623035323336326539616432386630
-34326432383230656638303963373832316438666131623539613464636631653365353331346262
+65346333643936343835376364626539303265313239376338653835343232646362656366646264
-37626130343831636133646135323564306538353439636265313239326235343232633136383663
+61626463363665363737323331356361616364626565333561303836313966323461663438333933
-62306561616564623038393631373862346535366135633836353363323330346430613466393463
+65643533346362626137363762363663383761353364363164393062383830383931653262383564
-39313766613064306566316561303761626135613063326536363761626635646134303732356130
+35333532646661623731663938656339323739663964666238653963323232666664313237363137
-38333266363639636565393737323938663938633662653231366164316262666538356162323665
+38626438313265336263626365643763393532613831343763393130343766663335343431613038
-38383263366133326333636466343835373737613333623464383139343863336234373932393239
+31393938303162346236303635306239613335656338643362376539346335373130323935393133
-36653736393732353466656334613364626339356533343363353731353033363730376565353236
+62356533393363626139323632663865373837306337393433353963396231643937373762653837
-66343332306330303662373363326236393333663565383661643763363662396265313037643432
+38306331656262343832333136333536383035323532383766383538653831386531353936363936
-63343336623663326237326365333565346335346630356439303534363561626362363833313239
+66623762663266663135373165303233646262653465633930373832383364386263383831366461
-62366436313466633534343430306461313266376363336536666162306135363434643266373338
+32643533386438613533326566336538633632313331396535326562336263646334336362346634
-65323866323537636465306636646533316466303735386336663234386538303939343661653562
+30366562623663313638356233633433306564303065316337323862663533366366353230373434
-34353564313738643665636337336536343735336436626438653335376532343137613837356363
+64656437626135633932616230333339373934613666383565626535663939313930373230313130
-31353861363733373232363865323164663532646235623533336536333932363133613830373937
+39333434643834356634383133383533396264663831383736326162323431393464613736616539
-38616137646332303934303839633336313239653064626331383432633935353538636132633134
+38326533623734353937373135386664613635636264383130346263326462303531316232386131
-61393230663536633335393330386363333162636661373938343837323736663539393636366532
+30663833633738363533333662376461313331636230346538313764633135303331613039316363
-35623637623262373332373335303833373239323430393766633939633665643236623139363834
+34623937386336646339373366663563626430343866303239363661313634636333616232336438
-30636135303634643765393433393962303138666630343936653234333730336234356234396566
+66616132363337393334646333623737336633303331393030326561613835613164303965393932
-35346331336639333237323136616337623938666561343861393563393633313261633333333434
+65313665303932373331656434353634343561353236643932623464393566333037386334363066
-38633064633538396566373730636632346234653339303533343431643266373266633836633139
+39373163313838663163323666636337663161343664646661616438393333363365363065353732
-33313234366236626338636331346565336138313136623632343735363132623964353066346233
+61666666666563626432383639393038633233663561303561353238613832373633653436316366
-66363833383432316238353364363963323762666562303030663939303162656261663664616632
+65333030623565336266646563333336373339616231366334316336666463633964323630373634
-66396466643763393130313831313438356635346230646335306237363262366132323133646535
+34346131626466316163366362326265613363333933336463623961306637353639346537316336
-30313338653262636136393133306663653331333830626534356536303532633031356435663734
+63383561353339393963396339663738333765663934383038313631363733356562373935333765
-32383035643637313863653537656665366138333933333364646561353333653661306133646361
+38663735636562336564636235306135666530303762623331663931366163363734316539333734
-32326535393333383033313765383031353233323138386365623132646636383037633537386361
+33303638643166303165653062396231613862313165373564626432313961643835383038316437
-65306166613030353033666438343530333330306532323135306363653764613638383461633964
+35653639656362633631336635363035633635623035653439353561343739626637656566333065
-63666239366638663537643636373432303164623735626634383636643836343436346161393365
+61626536366535376237366664646534383534323331633035326632656132646364363133636133
-62336462636534383766656632653833333832393263316633356235373335643436363333393335
+65643065383337356136633534383033396563353034383462386230623261636337303537646463
-62643838303131643265613861373964653464623766363664383330343138653632396365633533
+38356538356434393864303563336563343566306638353138656561663930316230353433386362
-39383535616630663564326536623537323966616436353439343630363738626131326365656132
+61643561383739366362666633616365343235393363623035316263313738633662323961613931
-32383737303733623830353366383062313134396531303831323336326137396335353763353631
+35393234373730616133316462363661663866323466333839656462636133613936643137653833
-36663336353865303562316233643762643563386665393664623138643836633165
+62333030343861356365616339363134353463656162616636303430356136363638313861313637
 37613731663639633865353332653835316631663364646139623631613138633162366432333532
 30306432653133333637623631626165633537623265313237323532303566383334633337306564
 31643163633934633161316663336334316237353735313063376436393166313033313665366265
 35363864386265386139353735663238366663396366653164613833383336653230383763316134
 37386633376538366633343963303339323338303263343862396631613733336261356536396264
 37623236383565323331306531316563356238316237636631613736316437643965613634666437
 32366661346466353261336130363233613036633863303161303931346462633763353038363136
 36363762353231373933616265643039666237333334326134396531643463336238303862313465
 62346431336663633661303034376165643539323061653363613635343761326363616139313933
 64613430353539376131653439346337353733663830306333373966393162366235393364336532
 38613936303965646231323138643464636133393037363538373562396230363434386230363934
 30656361323137326335316536663231376536646565643363323462363165633139346535313733
 61666434643537363135613862656462653732306665623037363638613832323530653836643137
 37323233316131343039643137333532653165316137303032373635346539656635363265396565
 34663263396265383532306264376430313237616438383639313863663664393066643565623636
 30376136373833653462646561396238356638336432376630306465333133623538636333623961
 33316263373862643166623030316239383433313365313439386534336532653436633034643938
 34613737333933303934653635636239363332376436303063643566663536353266303363626539
 64393737383037383336373663383163303538343438306361656164356563653236383563633735
 61376362303866646130666234616362323830643238353433613831353635373636306363623262
 65393266313430653133316639326330383936303566613534316366613637623465643739323234
 36646138353135666265646137653138656262303864653839333839633336643466343066653865
 31323538353566663638316136653438353835613137653364393032383935396134333030623831
 66333431326362303038343032316436356237373932316531313437613762636238663364356535
 30393931613231303936366436316438396535303561393538633465623064623039386430393230
 31336630663839336364336332623165613763303331383830663161613966623530323266396531
 39393931363365366334643231363732646631626533653636373364393463333661653833653964
 35656361623463396664306134313365333664616436313761313837363038343632373032356163
 66653030393665626264663761313266383362393565633864636436326238343836383066363438
 66656164373936663732333565313466366631336464316335663361663563363765333938653437
 63346563356234333936666437396462613262633834333938343635613761396632633462633837
 64303563376431643930343366626336656238616162366566613166623637656533653263383832
 66643465626636303136383235383739393265313964373238393563613438346132383566353061
 39393334643839613538343966383736646664373363373234376639636637326639383734313562
 32306163326563366534303862386638643534636162353235626438373237343135376339663164
 37313836313863646539396231353864623733396163636136313864613030326534663430633062
 62373238313435346637623334623631343339323131643363383863323530333236313632653364
 38346165396336623838623630336264313565396436646536393237646163333338346338653438
 34396537373535616339626164356635333837623234316630373333646333343034376336383739
 62346366383134626162376134643738623965653963303334353936613665313061653661616438
 37643032393033313265396536353433323735613766326164323461623238316335623131343932
 31313763303235366436613939366634646434326233303637653235323539626264306238353733
 63383666636464313064623736316466323539376365376134663763343530333839313165643432
 36663238333231306134313866303332623134323631646330373563656431363537323861353632
 30306334366131633865633366353666306238326537633461323965303437656538643136386262
 61386162626633343266666338363563663965643761636462663333373539396634313766323139
 65653564333531373561393239343036616238363162633462353962393336396530383430353265
 38373539613339313964356166666630376130313830356137326465653166633135346262613839
 34623038396635373936333535343330623338663235653137393335346536343331383034666139
 61333965386132666439616639623165663034383633303136393730396232336139306664663233
 61613731646538633265323439666535346636333531326230366238653331303830646131343430
 61383735386361356262326632663666383162316232393338363633323463613732623064623232
 39623231643734363937623134336632623562356531333635633031636536383362343634373461
 35633665383130636336363364616466346462663962376164336331663165316661393934396136
 36623836636438373761376338303534353832653433396266376638636231616631356262323736
 36383466643661363062373062663039333566633763316334626334373663633062376335663461
 61316564303432333730333638303566313132646636643639626131343165663063333838333635
 36663139643730333630633766383132396636666632366164346537363732353830613939663262
 66623433633434646432313138643639366265396333626564623562323865616261663064363630
 66313134356536646633366262356331336134366265316166633831653434646232333730356661
 66663431653363666639333632366261613764393665393535373934616433653834646361373438
 36363164363932663463623161373736353233393934633537613932333732376436623031303962
 30643830353665616636663066306536336236373064376530306135646234326131613130306433
 62653463306332346530336130623633393636393762633230323063646237626130353335336563
 65646538313634333535333564663337623233613238303638616263383764653230343661343065
 30626533643465333733366430316162316134323431643166316631623965323334316263396131
 64303932663464383131386239336166353363333664353031646462363862663165376565356562
 65346539633036343031396662623233613739656633653430663737613662303331396230353038
 62333333383462653138653162626333326466653631643634636135663232393437343332333964
 38623062656633313762353438313839373935373964393938396536633233613161666166626139
 66323363323039323666303934373631343737313866343665643131386431373336333630346261
 65646666303633623166643635326165303838646532393663666266303536343634633533346664
 31386264656165376264653266663332316536303665323239613630386233373565353763323739
 31373232623735366563303731626163356537393034313962653263333831363632343561316337
 33663430333836633061643566303638316133323361343931303331353234613065366133363931
 33316564313939383137653330353935396532386563623939363336383537386639376262376561
 31323837346166396332373735316230653531343166393539386364646461376265373564636637
 65663430333564393861343261316562386564393836303834633638346638383633353164303438
 34343463353538663431306332636166623066376135323261613735393837363666666465323866
 63356530346532656131383835363133353938636630333231353964373765636162613062363937
 32393461356530383531363131326164356562383762643832353037393931316666353331396437
 37616230383338613931646135306539636335636238373065346236333966393631346266343737
 37383135613133383763306531326436323233393231376165633538653036393833373538653030
 61336331313430326638373138376661633731306165653661313766663134383163653731393934
 63643438353663646539623935356635303236616231646431356262383233643333393531376534
 65663437633761646564333366626335336534626362353964316236643066306439366565373735
 30373965366137643363653533366430303965636631383664303239623938616135303030373962
 65346635663562306566303236663563623036366465353739626564393839643530353033323737
 66643665303132346566333866663736376637393966366263636462396637663461646434376566
 61353464313337333938303866663036646566623032393036646431343739646436356632353361
 39366262376235663139666631643734376461613862613838653062313466663266646139323831
 37313961383638353139353635633133636561306361323065323733633532333235323031653930
 39306534343635646135646337353161303737396266343135363938383664346338373730366531
 32396533323264666433616330373232623330383762323263333432383066313838663931333838
 30636230333235653238356633353030663136343462353530643531376234336161656361633438
 64336361636437636236373532346131393431373537633663396534363764653563393064313665
 36616266333938643061613361653532356133663431333435343634373564656531383035353632
 38383863616133363937633937643062623831356232346238326236313536393738346263313530
 35303865666530393330343936356131326164653765623031323535363266393265333232333464
 36356536613264353962323665333238666638373637306431343566656532383735363863313366
 61666465316437383934653534363766356635666364383036366664356534333465366530633365
 62353134313939373666623966666335663031616563373931386564383039613833323434326238
 66373132323762653031363763353831353336396137376364383138366264396134343262306133
 34393865653666663035633637373661366463303335353031373431386161616437353161326638
 32633339356434633363313030636464376535383534346436663634396636376630613634653635
 61383734313234396666303136393633363536326265643863656562356239326536353632363764
 32383663306139373532643037303365346336643331336331663630333136323437663930333366
 34366162356435316436313663633936323931383832373465323537653436623237323666333737
 35303962396266336633623563663064613934613837356563636330323431346334326234663433
 31313665653836616631653661626162613032313737373939323232386635393533363962373966
 61396639366131323131623436633439363234333032636137646230656463306265333334393736
 32323262643066306631316336356461633764646664653339626162366366353566373565653635
 63363438396264666565383164363639303637646330333064363762393731613133633934666131
 65323932373235373863623434373439343661396137656564396663653336396633393331663032
 65363432373338313137356666623338303064393232636535356262383038323166393235653338
 63613536626365666437376638633665626531366330313230663135353538303232356366326161
 64303863643030316564643262303664653463653463393461313538343464313638653461643966
 61653733343339613864666665373033353062363165376530323362663763306132336230396533
 64373762626263626638643763336635616364323561633536326165326466393736333836376462
 37303365396539376538663866633761313136326563623365373739373439393630376262373138
 35663064636561626434313035346131633530326431653664373036356464313435333333386438
 34396166643435663734363434316436386637643836313438333731323261303737653932666664
 62313761383832373235353239313830326635633430313762656265313838623163366637633736
 30316230353338626561393763663932663666613331653834346131343434663232623434353838
 66396262383937353530366163313532343731353338633065393632353264343136386538616334
 31383365666534303564616364343464313233643932326539313738386131646535316638396133
 35636339396339323765336137626438643631653963333362356438663531626634633336303733
 34626138336238323063623536333837336666613630626462636661353962303330323134326231
 66396665383033386634373766383461616334383130356337323763353130653732333136303166
 32303364616239616439336639633233666230396334666531643464653433333532313737363835
 63623737383830646334643039643739636235333963643163336464373133333731633765393764
 66616236346430323464393739326364333735663937643737356631393766623734636262336266
 31376535613266613362313363666262383837663533316636323262316364633461656262363762
 39613635343165326164646636343431623230353561363566666636656237373363646230363435
 64343435633039336337313562353638363664373137396366303839303766316230323861396331
 36346230353163336538303961626261383031366330613636323263346139356339616537393236
 30306537663861323931323536373833643330313338363435383461383330656266613439333262
 39643930313537386538323436636134643639656265336137666663666637363135643137333063
 63346133313066643566643432356537383532326163356437386363353965303737656261656132
 63386338643565383033353832663634633736323432336561383763356239303936333038396639
 33616364346165323962313033666335323233343462353434663832386561376339666366393161
 62656532636238613264643833336264616634623236643534363565303531656131646263613830
 30323161623864386130316431313435383637316337333032633035303564636534326634646565
 32333465303061626536303231386434363033636665366339376432376432383134353938663331
 31323633386631383038376438313761323839646139616564383734366534376134366330336166
 32323430663335653335376637313065383731383936663765303664356661636366616136656535
 65346637313766326233376232646137313839616339636161613066653133613230623838326662
 62313037393231643966353638316164333563343264616563633365333437356133373336386561
 65323665356463326537326239396339613839306131656538323066656134333833656539623462
 61386263646434636236643534636464363731656566383831656635336437363962306263633233
 36646263636638396461313138306563656565613336356132393532626161343863633239636438
 30393663323562623764663638646431336265303764356233323566613934306132333735623962
 36626462343039313765333535633835303664393664366635393264373139346639636537313038
 30383133376361303336363239653931653436663630666665383161333138346239616338313338
 33343138666263623534396239633462383936646564313833373435346432323532336466373937
 64386537653332646439643764643563393738323565346630616432316164663161313337366531
 38396564386635323466386339613765326638333532623639303136656331633038336539393062
 66613035306632373562666466306336353333333039346134383263313834666434313237376530
 64373539383166363864366531323333323565366465383937333732613437366264396532616433
 37623438653438656162626666343033346638636564653061396562343564626535616233353334
 63373565383039383737333534663637666231323132313162373436666362383963303162373835
 62656434313463336462643764333735653166306265636638326131613736393230326634333135
 36666633626235343630306637386164313562323439663037386166303430393065386238383434
 35396664336331336561633232333934343832626230343935393433326430653632393562366532
 37346235343762336430653733353235643831633230646135333761613832336234646262623865
 34386230623338613764303536343533623432336332396665626131363364643465666639636664
 32376461393532313839393766623463386537643733616161313330623739626533643562303262
 37356339353836303235343762393838343363623331646464646163363363663638626433636230
 38323038363931353830633737383865353934373238313634613961373566336639326131653335
 32623238313339396261333531373437336266363964386237366663336164323566313864396337
 35636566616462353565383731386236356230303333336431666633613264373331663430346434
 36663166363430373534346236363332346562653463373261346331653032613436616637623936
 63393663666230663835623835633936656538323739356231353563623939383463353330326464
 38373531653537376637633436616335633866656163303937636438643162653231393963356537
 63306630313635363637656665373337363232336135346230643765376631343734303766633764
 66356236643338356631373832323863353866336333316333626232396463303466643434646232
 65623838353539353962346661383865626532613930363330326334316565343330343538636434
 33626433623361313430313361323966353732303035626366323835383631343634316634623562
 39313166376365373638303338326263616566643566636361633332373862623037336163636366
 31626365663862303563626130393266316237643266383139643864613134373734663238633639
 33613761383431383463383037313139393833633761633635356538313635383166386639326362
 36346662626139353837376565633633363964343665356632646232393433326437353532613365
 34323666633131653233393062366637336630303662306634363262666439353964653436363534
 64656238363036343164656434343262393936306230333639393932663233363062666264656361
 64363233303636646138323662623532333063646339306635633039653931333865356535396665
 38346362366165336339646162623739653433353966613762373534306634643530316538356433
 65393034366562663466313432396431636462383239326438643437316361343561393234306338
 37333664666132343263343763333137316564363963306663363265376263393964386630646636
 66316334323439663630373131643330636266633032376236383564643138356337353566363966
 31323662613461353336396637366636366336353763666264626431653935663332363930303633
 64643862393963613936323838653661646464353662613430336535363833663161323764303036
 34396130633438323566366566303932363937653536653638303931646535353239393738323337
 31316237356531373034646565653138396536326339376135613935366130333061636135613164
 62386134626234623734333339393763393536623964343032316666366632633863643839353530
 63366631646535323133353235613338383864613265613562653861653138333066396134653833
 30313237646466333139323037376564646161613131393232333233633863396536363038396165
 66643163386337333666626261336537353461666331353162346330616332616330653936616439
 62356635323036323363653566313764396566346533646437323932366134303830356331303730
 63363933396337653239343163366236386162333032356465326237623563386234656339643536
 38393636376536616638303637393537393465656563353232623435323730353561303830363434
 64346532303834363338643838346163313963303638346239373936303335373039323636343534
 62623563343361613166633465373966343534633864303564393766656465393964396237353464
 30666532393332323162313464663931356335333263643332373431343036636536316435303765
 66633161653136653038366162353538623563653237666131383163333636373035643062626236
 36303837376161363762353964363239656565396666356462633864343833373237303362316161
 35323931616535633166646163616535363863616237363238316264363733636230323864363530
 32353533666339656464303234366562616332633434393736663631656434373730313165313062
 63373939313332346235386566613537343339363365623836313835343165386133643837336161
 31666139393664363765646233323965653831333435376565653132313861613565393833613463
 34623338313765386331643235303361363033393438353432383963663133306465613530323434
 64306364326261303433623062313261613532303333656136363536353066313630363633353938
 63623566613262646531616332376333633934666337376638616231653939646661316264376531
 37346436323339633839393665383735653465313538366262366538623462306439363730386666
 66383736623636333139346434663933303766333130323333383536316333363165633762336235
 34613638626161326165346662613433316164323533376265353261303535336431323065623964
 32636265653736363763316430636133376539376361616631643965353732656336333835356536
 38373933623063316135353762663735376533663065336233363133613833373464633365346339
 36613530663735663735353264373966643134306464343431633963366337616239623135663361
 61303532323564306266656537636264663739356235393535633139646131346662353437306138
 63313239623832376362353632326563633562663265623534643032306562343535613035656162
 31353862366233663535396134643433306338333638613737623161376334653631316236383532
 66316364313637633732363935636266393364636635373061353132383866306330333734616562
 64633132336634643034633834623536383631363933653463663364643262623565393461613735
 31343962333461393431643233396561376365666337313961616338313736373439323433393061
 34636665323331653731346566656164316163666131396431333063656634373433306262666437
 62363939326131663035633264613861376131623332666533653666373262613035636437346461
 66336362393935633862323631393163373633306334366434346434333664313732636337313165
 31356132646265346537363761393462393630653239396535383164323739636632306531626531
 63643536366464383831333663376136643435636361666266333035616530333832336163636436
 32613437333336323332376238623336373465323937353130643232656238643334313332303465
 32306539626632323062343230666636653566376530633263373964623364363234316264346462
 31656665336165643831333039376163623332633634643961316635396533356531323134356162
 36363731373337376265303261313737663235306164666532303539656565343462333034366530
 66643165646461303235653435613266383034333664323165646238616236663339633361383565
 31616438623866633664323037343238383663343936643636333361633666336638333262316531
 36663139313466303736336331326666666638333938613031353439313765343530626537393933
 38383863643337363038386135326463363736373434646434343035613063343738383965366434
 63613038353065353538323130323163383763383030666335383066373536336166393666333763
 37663861353636633464396532666434313363323838346235643364393039366566313264333538
 30353962613263623838656336353134373564383864383130356664376531626137353064643562
 33393261346662346538666130303138623765383166626537636363313138646536623436643666
 39626461383038633461306431363364636364623333663766396365653732333433346664316139
 39333961366433633238386335343934383134663133633030386133386433633261383332306163
 34373536346335616265666666386364646461653433386239653730656635663665643233323630
 39616336386238333830363465636437323762633833633938323731653538613636653239373333
 30636464343262356165313932356666663765376136663065666433616639613539323237303238
 61386637373130626265373336326531653631343339613938376136346262626362653237633164
 61616636626563613738613964376561376262343837396430653438623066663937653032326530
 33396334653131356337376638356664313333343366373139623766393736326438376130646564
 63333230306139303238323235353061616666643764333864646365356135376638643038313037
 66316636643165333739356530663962346135323431386535356161393239616231326432393439
 38326163363938373330386262386664313761613631356430633861346631623765663062643865
 61353937383533633638313138316466313863656136393666353730303434613031313731333534
 31306361613234626332386339653162393734623536613638343164663963326532623133663732
 37343336326133626132383465306266623132643861376564636461383132306561613635373135
 65386432316133393134386334653633666330623039336335396433623530633765383732333635
 6431363665353131633562626461643631353938376163613361
@@ -1,66 +0,0 @@
 ---
 # FastPass Kubernetes Cluster Variables
 # Cluster Configuration
 cluster_name: "fastpass"
 kubernetes_version: "1.33"
 pod_network_cidr: "10.244.0.0/16"
 service_cidr: "10.96.0.0/12"
 # Control Plane Configuration
 control_plane_endpoint: "{{ cluster_name}}.{{ base_domain}}"
 control_plane_port: "6443"
 # CNI Configuration
 cni_plugin: "calico"
 calico_version: "v3.30.3"
 # Node Labels and Taints
 node_labels:
  zone:
    # Control Plane nodes (masters) - no zone labels needed
    # space-mountain: (control plane - no zone label)
    # big-thunder-mountain: (control plane - no zone label)
    # splash-mountain: (control plane - no zone label)
    # Worker nodes
    haunted-mansion: "backstage"      # Internal/backstage workloads
    peter-pans-flight: "backstage"    # Internal/backstage workloads
    # Future node: "on-stage"         # External/on-stage workloads (future)
  environment: "production"
  cluster: "fastpass"
 # System Configuration
 selinux_mode: "permissive"
 firewall_enabled: false
 # Container Runtime
 container_runtime: "containerd"
 containerd_version: "latest"
 # Repository Configuration
 kubernetes_repo_url: "https://pkgs.k8s.io/core:/stable:/v1.33/rpm/"
 containerd_repo_url: "https://download.docker.com/linux/fedora/docker-ce.repo"
 # Validation
 preflight_checks: true
 validate_cluster: true
 # Kubeconfig Configuration
 kubeconfig_path: "{{ lookup('env', 'HOME') }}/.kube/config"
 # Hostname Configuration for Kubernetes
 # Override hostnames to remove dots for Kubernetes compatibility
 kubernetes_hostnames:
  space-mountain: "space-mountain"
  big-thunder-mountain: "big-thunder-mountain"
  splash-mountain: "splash-mountain"
  haunted-mansion: "haunted-mansion"
  peter-pans-flight: "peter-pans-flight"
 # Generic Kubernetes Prerequisites Configuration
 configure_firewall: true
 configure_ntp: true
 selinux_state: "permissive"
 disable_swap: true 
@@ -1,3 +0,0 @@
 ---
 #freeipa_user: "admin"
 #freeipa_password: "{{ vault_freeipa_password }}"
@@ -1,51 +0,0 @@
 # file: group_vars/hub_cluster/vars
 # Ansible vars template for OpenShift nodes created via Ansible (configure_openshift_cluster).
 # Cluster deployed via Agent Installer and Matchbox (iPXE):
 # Name of the OpenShift Cluster
 cluster_name: "hub"
 # Version of OpenShift to install.  All available versions can be found at: https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/
 cluster_version: "4.18.21"
 # Type of cluster. 
 # - none (Single Node Cluster)
 # - baremetal (Traditional and Compact Cluster)
 platform_type: "none"
 # Number of nodes to deploy
 # - Master: 1; Worker 0 (Single Node Cluster)
 # - Master: 3; Worker 0 (Compact Cluster)
 # - Master: 3; Worker 1-3 (Traditional Cluster)
 master_node_count: 1
 worker_node_count: 0
 # Subnet that cluster nodes will reside on.
 machine_network: "10.1.71.0/24"
 # OpenShift API IP Address (Reference host IP address for Single Node Cluster)
 api_address: "{{ ip_address }}" # "10.1.71.10"
 # OpenShift Apps Wildcard IP Address (Reference host IP address for Single Node Cluster)
 app_address: "{{ ip_address }}" # "10.1.71.10"
 # These are internal networks to the cluster and should not need to be changed.
 cluster_network: "10.128.0.0/14"
 cluster_network_host_prefix: 23
 service_network: "172.30.0.0/16"
 # Static Variables
 openshift_installer_file: "openshift-install-linux.tar.gz"
 openshift_client_file: "openshift-client-linux.tar.gz"
 openshift_installer_download_url: "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/{{ cluster_version }}/{{ openshift_installer_file }}"
 openshift_client_download_url: "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/{{ cluster_version }}/{{ openshift_client_file }}"
@@ -1,51 +0,0 @@
 # file: group_vars/internal_cluster/vars
 # Ansible vars template for OpenShift nodes created via Ansible (configure_openshift_cluster).
 # Cluster deployed via Agent Installer and Matchbox (iPXE):
 # Name of the OpenShift Cluster
 cluster_name: "openshift"
 # Version of OpenShift to install.  All available versions can be found at: https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/
 cluster_version: "4.18.21"
 # Type of cluster. 
 # - none (Single Node Cluster)
 # - baremetal (Traditional and Compact Cluster)
 platform_type: "baremetal"
 # Number of nodes to deploy
 # - Master: 1; Worker 0 (Single Node Cluster)
 # - Master: 3; Worker 0 (Compact Cluster)
 # - Master: 3; Worker 1-3 (Traditional Cluster)
 master_node_count: 3
 worker_node_count: 1
 # Subnet that cluster nodes will reside on.
 machine_network: "10.1.71.0/24"
 # OpenShift API IP Address
 api_address: "10.1.71.69"
 # OpenShift Apps Wildcard IP Address (These will be the same in a Single Node Cluster)
 app_address: "10.1.71.60"
 # These are internal networks to the cluster and should not need to be changed.
 cluster_network: "10.128.0.0/14"
 cluster_network_host_prefix: 23
 service_network: "172.30.0.0/16"
 # Static Variables
 openshift_installer_file: "openshift-install-linux.tar.gz"
 openshift_client_file: "openshift-client-linux.tar.gz"
 openshift_installer_download_url: "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/{{ cluster_version }}/{{ openshift_installer_file }}"
 openshift_client_download_url: "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/{{ cluster_version }}/{{ openshift_client_file }}"
@@ -1,14 +0,0 @@
 ---
 ipaadmin_password: "{{ vault_ipaadmin_password }}"
 ipadm_password: "{{ vault_ipadm_password }}"
 ipaserver_domain: int.mk-labs.cloud
 ipaserver_realm: INT.MK-LABS.CLOUD
 ipaserver_setup_firewalld: true
 ipaserver_setup_dns: true
 ipaserver_forwarders: 1.1.1.1,1.0.0.1
 ipaserver_random_serial_numbers: true
 #dns_ip_addresses: "{{ ansible_hostname }}"
 #dns_reverse_zones:
 #domain: int.mk-labs.cloud
 forwarders: 1.1.1.1,1.0.0.1
 #hostname: "{{ hostname }}"
@@ -1,4 +0,0 @@
 ---
 #freeipa_user: "admin"
 ipaadmin_password: "{{ vault_freeipa_password }}"
 ipaserver_domain: "int.mk-labs.cloud"
@@ -1,9 +0,0 @@
 ---
 # Variables for Matchbox
 assets: "/var/lib/matchbox/assets"
 http_endpoint: "http://matchbox.int.mk-labs.cloud:8080"
 rpc_endpoint: "matchbox.int.mk-labs.cloud:8081"
 # Variables for OpenShift Installer
 #openshift_installer_pull_secret: "{{ vault_pull_secret }}"
 #openshift_installer_ssh_key: "{{ vault_ssh_key }}"
@@ -0,0 +1,12 @@
 ---
 # Single-origin layout: one hostname, Caddy routes /health, /ws, /api/daemon/ws to the backend
 multica_layout: single
 multica_domain: multica.wdw-labs.cloud
 # TLS: acme_http | cloudflare_dns | internal
 caddy_tls_mode: cloudflare_dns
 caddy_acme_email: "ryan.blundon@protonmail.com"
 # Pin for reproducible deploys (git tag + matching image tag)
 # multica_version: v0.4.5
 # multica_image_tag: v0.4.5
@@ -0,0 +1,8 @@
 ---
 # Copy to vault.yml and encrypt:  ansible-vault encrypt group_vars/multica/vault.yml
 # Generate secrets with:  openssl rand -hex 32   (postgres: openssl rand -hex 16)
 vault_multica_jwt_secret: "CHANGEME"
 vault_multica_postgres_password: "CHANGEME"
 vault_multica_vcs_secret_key: "CHANGEME"
 vault_cloudflare_api_token: "CHANGEME"          # only for caddy_tls_mode: cloudflare_dns
 # vault_multica_resend_api_key: ""              # optional: email verification codes
@@ -1,54 +0,0 @@
 # file: group_vars/openshift_cluster_template/vars
 # Ansible vars template for OpenShift nodes created via Ansible (configure_openshift_cluster).
 # Cluster deployed via Agent Installer and Matchbox (iPXE):
 # Role of this node in the cluster. 
 # - master
 # - worker
 # Name of the OpenShift Cluster
 cluster_name: "openshift"
 # Version of OpenShift to install.  All available versions can be found at: https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/
 cluster_version: "4.18.21"
 # Type of cluster. 
 # - none (Single Node Cluster)
 # - baremetal (Traditional and Compact Cluster)
 platform_type: "baremetal"
 # Number of nodes to deploy
 # - Master: 1; Worker 0 (Single Node Cluster)
 # - Master: 3; Worker 0 (Compact Cluster)
 # - Master: 3; Worker 1-3 (Traditional Cluster)
 master_node_count: 3
 worker_node_count: 1
 # Subnet that cluster nodes will reside on.
 machine_network: "10.1.71.0/24"
 # OpenShift API IP Address (Comment out for Single Node Cluster)
 api_address: "10.1.71.69"
 # OpenShift Apps Wildcard IP Address (Comment out for Single Node Cluster)
 app_address: "10.1.71.60"
 # These are internal networks to the cluster and should not need to be changed.
 cluster_network: "10.128.0.0/14"
 cluster_network_host_prefix: 23
 service_network: "172.30.0.0/16"
 # Static Variables
 openshift_installer_file: "openshift-install-linux.tar.gz"
 openshift_client_file: "openshift-client-linux.tar.gz"
 openshift_installer_download_url: "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/{{ cluster_version }}/{{ openshift_installer_file }}"
 openshift_client_download_url: "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/{{ cluster_version }}/{{ openshift_client_file }}"
@@ -1,34 +0,0 @@
 # file: group_vars/prometheus_server/vars
 # Ansible vars template for Prometheus nodes created via Ansible\.
 # Version of Prometheus to install.  All available versions can be found at: https://prometheus.io/download/
 #prometheus_version: "3.5.0"
 #alertmanager_version: "0.28.1"
 # prometheus_nodes:
 # #  - '192.168.1.207'  # Worker node IP
 #   - cinderella-castle  # Master node IP 
 #   - 10.1.71.21
 #   - cinderella-castle.local.mk-labs.cloud
 # Static Variables
 # prometheus_installer_file: "prometheus-{{ prometheus_version }}.linux-amd64.tar.gz"
 # prometheus_installer_download_url: "https://github.com/prometheus/prometheus/releases/download/v{{ prometheus_version }}/{{ prometheus_installer_file }}"
 # Grafana Variables
 #grafana_port: "3000"
 #grafana_url: "{{ inventory_hostname }}.{{ base_domain }}:3000"
 # prometheus_targets:
 #   node:
 #   - targets:
 #     - localhost:9100
 #     labels:
 #       env: mk-labs
 grafana_ini:
  security:
    admin_user: admin
    admin_password: admin
@@ -0,0 +1,19 @@
 ---
 # ansible/group_vars/proxmox/oidc.yml
 #
 # Proxmox OIDC configuration for Authentik integration.
 # Client credentials come from vault.yml.
 proxmox_oidc_realm_name: "authentik"
 proxmox_oidc_issuer_url: "https://authentik.local.mk-labs.cloud/application/o/proxmox/"
 proxmox_oidc_username_claim: "username"
 proxmox_oidc_scopes: "openid email profile"
 proxmox_oidc_autocreate: true
 proxmox_oidc_default_realm: false
 proxmox_oidc_comment: "Authentik SSO"
 # ACL entries - grant your Authentik user admin access
 proxmox_oidc_acl_entries:
  - path: "/"
    user: "rblundon@authentik"
    role: "Administrator"
@@ -0,0 +1,17 @@
 $ANSIBLE_VAULT;1.1;AES256
 64613361376364346139313833613465663361336634326430393261366630306466363935613139
 3634363235343736643230623865386436333734663531310a343965396534336262356234623966
 39353332626662636666383935383530613139626439373664323063633063316264383331646533
 3366333837613731310a656462633638376432326365343135373863313665346566383933353630
 37303565323630633739396433323165326262363863386233343134636236646366633765616432
 33633165613239653762343935386637393437386431383961306436373765343736313634333933
 30376463646464303066613561613564353938306664373464333966383664383034616439343735
 38363966323366343465393530383736393364373361326234356365363163356632393834393464
 39393530356235383830323065636535383138353864373237333164323436623737383435313261
 64366362626232386438376131633533633162356432613835656465623762633464353936303236
 63353633633262633461353963653534663336396134373435303266386162393965343165303335
 65313436666334383965373837376664323861376162373834363439653539383231346634633737
 30626431373739633930313937643938343165383536373465613535366562313439396533643864
 36323032316131333962373732306334373939333363386564653465303261393337333466316130
 32326533353165393635393130396164636363623937313566343333386562616662383038613565
 35386638383365633362
@@ -1,77 +0,0 @@
 ---
 # Variables for Unbound https://github.com/aruhier/ansible-role-unbound
 ### Server ###
 unbound_chroot: '""'
 unbound_num_threads: 2
 unbound_interfaces:
  - "{{ ip_address }}"
 #unbound_do_ip4: "yes"
 unbound_do_ip6: "no"
 #unbound_do_udp: "yes"
 #unbound_do_tcp: "yes"
 # list of strings, as the order impact how policies are interpreted
 unbound_outgoing_policies: []
 unbound_access_control:
  - '192.168.1.0/24 allow'
  - '192.168.3.0/24 allow'
  - '192.168.5.0/24 allow'
  - '192.168.10.0/24 allow'
  - '10.1.71.0/24 allow'
 unbound_access_control_tag: []
 unbound_access_control_tag_action: []
 unbound_access_control_tag_data: []
 unbound_access_control_view: []
 unbound_private_addresses:
  - '192.168.1.0/24'
  - '192.168.3.0/24'
  - '192.168.5.0/24'
  - '192.168.10.0/24'
  - '10.1.71.0/24'
 unbound_private_domains:
  - 'mk-labs.cloud'
 unbound_domains_insecure: []
 unbound_do_not_query_addresses: []
 unbound_local_zones: []
 unbound_local_datas: []
 unbound_local_data_ptrs: []
 unbound_local_zone_tags: []
 unbound_local_zone_overrides: []
 unbound_trust_anchors: []
 unbound_trusted_keys_files: []
 ### Remote Control ###
 unbound_control_enable: "no"
 unbound_control_interface: []
 ### Stub, forward zones and others ###
 unbound_rpz_zones: []  # list of dicts
 unbound_stub_zones: []  # list of dicts
 unbound_forward_zones:
  - name: 'int.mk-labs.cloud'
    forward-addr:
      - '10.1.71.5'
  - name: '.'
    forward-addr:
      - '1.0.0.1@53#one.one.one.one'
      - '1.1.1.1@53#one.one.one.one'
 # For stub and forward zones, if a key inside one of the dict is an iterable,
 # it will iterate inside it to duplicate the option with all the contained
 # values.
 # Can be useful in case of multiple forward-addr for a same zone:
 #   Example:
 #     unbound_forward_zones:
 #       - {name: "test.tld", "forward-addr": ["192.0.2.5", "192.0.2.6"]}
 unbound_views: []  # list of strings, as options can be multiples
@@ -0,0 +1,235 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: ansible/host_vars/astro_orbiter/vars.yml
 # HOST: astro-orbiter (10.1.71.130)
 # ROLE: llama.cpp LLM inference host — Ryzen 7 5800XT / RTX 3090 (ATX rebuild,
 #       2026-08-04). Superseded the prior AMD RX 5700 / Ollama config below;
 #       drive was transplanted into new hardware, not reinstalled.
 # ------------------------------------------------------------------------------
 ansible_host: 10.1.71.130
 ansible_user: jarvis
 ansible_ssh_private_key_file: ~/.ssh/id_jarvis
 ansible_become: true
 # LVM root expansion — xlarge template uses sda3 partition, standard VG/LV names
 common_expand_root_lvm: true
 common_root_pv: /dev/sda3
 common_root_vg: ubuntu-vg
 common_root_lv: ubuntu-lv
 # --- Staged GGUF models for the llama.cpp router (:8002) ---------------------
 # Data-driven list consumed by roles/llm-inference-multimodel tasks/models.yml
 # (loop -> tasks/stage_model.yml). Each entry is idempotently staged into
 # /opt/models: stat + EXACT-size check vs HF manifest; skip (no download, no
 # restart) when present + size matches. Source repos are public bartowski GGUFs
 # on HuggingFace (no auth). A router restart is notified ONLY when a new GGUF
 # is actually downloaded.
 # Added 2026-08-12 (War Machine): codify Phi-3.5-mini-instruct-Q8_0 and
 # Meta-Llama-3.1-8B-Instruct-Q4_K_M as router models alongside the production
 # Qwen3.6-35B-A3B-UD-Q4_K_S. The live files were already present/correct on
 # astro-orbiter; this pass codifies them. Future adds = append to this list.
 # Router --models-max override for astro-orbiter.
 # Default in defaults/main.yml is 1 (conservative). Bumped to 4 on 2026-08-12
 # (t_33acbb2e) so the router can keep more than one GGUF resident on-demand
 # and LRU-evict when needed.
 #
 # VRAM NOTE (t_33acbb2e, updated t_55c164f5, updated t_34b96e83, updated t_f5f7e9ad, updated t_441470b9, updated t_c5cef2b2):
 # With models-max=4 and all 6 GGUFs registered, worst case is all 6 loaded simultaneously:
 #   Qwen3.8-27B Q4_K_M:               ~20.0GB (weights ~17.1GB + KV ~2.9GB @ 65536 ctx, q4_0) ← CORRECTED (ctx rolled back from 128K to 65536, t_c9fed26c 2026-08-18)
 #   Phi-3.5-mini-instruct Q8_0:       ~4.3GB  (weights ~3.8GB + KV ~0.5GB @ 32K ctx)
 #   Meta-Llama-3.1-8B Q4_K_M:        ~5.6GB  (weights ~4.6GB + KV ~0.2GB @ 8K ctx)
 #   Qwen2.5-Coder-14B Q4_K_M:        ~9.0GB  (weights ~8.4GB + KV ~0.6GB @ 16K ctx)
 #   nomic-embed-text-v1.5 Q4_K_M:    ~0.09GB (~84MB, embedding only — no KV cache)
 #   Qwen3-8B Q4_K_M:                 ~5.5GB  (weights ~4.68GB + KV ~0.5GB @ 32K ctx, q4_0)
 #   Total worst-case:                 ~44.5GB  >>  24GB RTX 3090
 #
 # OOM RISK: Full co-residency is impossible on 24GB. LRU eviction prevents this
 # in practice: models-max=4 means the router can REGISTER 6 models but only keeps
 # up to 4 LOADED simultaneously — the router will evict the LRU model when a new
 # one is needed. nomic-embed-text-v1.5 is pinned via sleep-idle-seconds=-1 and
 # load-on-startup=true but it uses only ~84MB, so it never meaningfully changes
 # the budget. In single-user homelab operation, only one generative model is active
 # at a time alongside the always-resident embedding model.
 # Qwen3.8-27B alone uses ~17,804 MiB (weights+KV @ 65536 ctx); co-residency
 # with Coder (~9GB) = ~27GB > 24GB. LRU eviction handles this automatically.
 # Ryan should be aware this means model-switching always incurs a ~30-60s
 # cold-load latency when switching between Qwen3.8-27B and any other model.
 # Proceeding to models-max=4 as instructed; flagged for Ryan's attention.
 # Router --models-max override for astro-orbiter.
 # UPDATED (t_f5f7e9ad, 2026-08-16): Set to 2 because Qwen3.8-27B-Q4_K_M
 # uses 17,804 MiB at 65536 ctx. Only nomic-embed (558MB, pinned) and ONE
 # generative model can be resident simultaneously. Co-residency of Qwen3.8
 # with any auxiliary model (Phi 8.3GB, Llama 5.9GB, Coder 9GB) exceeds 24GB.
 # models-max=2: slot 1 = nomic-embed (pinned, always loaded), slot 2 = LRU
 # generative model (Qwen3.8 primary, cold-loaded on first request ~30-60s;
 # auxiliary models evict it on demand, and vice versa).
 # NOTE: Qwen3.8 does NOT have load-on-startup — it loads on first request.
 # This avoids an LRU eviction race with nomic-embed at startup.
 # UPDATED (t_72646029, 2026-08-17): CPU offload for Coder + Llama changes the
 # constraint. Coder and Llama now use CPU inference (n-gpu-layers=0). GPU-resident
 # VRAM: Qwen3.8 (~17,804 MiB at 65536 ctx) + nomic-embed (558 MiB, pinned) plus
 # the CUDA-context buffers llama.cpp 6ea215d allocates for the CPU models (~1.4-1.7GB
 # each) = ~20,004 MiB steady-state, below the 24,576 MiB physical limit.
 # CORRECTED (t_c5cef2b2, 2026-08-19): ctx-size was rolled back from 131072 to 65536
 # (t_c9fed26c 2026-08-18). Qwen3.8 VRAM at 65536: 17,804 MiB (not 20,302 MiB).
 # models-max raised to 4: nomic (slot 1, pinned) + Qwen3.8 (slot 2, GPU) +
 # Llama (slot 3, CPU) + Coder (slot 4, CPU). Phi (GPU, ~8.3GB) and new
 # Qwen3-8B (GPU, ~5.5GB) can also be requested but evict Qwen3.8 due to VRAM.
 # models-max=4 is required so CPU-offloaded models count as loaded without
 # evicting Qwen3.8.
 llm_router_models_max: 4
 llm_staged_models:
  - filename: "Phi-3.5-mini-instruct-Q8_0.gguf"
    url: "https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF/resolve/main/Phi-3.5-mini-instruct-Q8_0.gguf"
    size_bytes: 4061222688
    source_repo: "bartowski/Phi-3.5-mini-instruct-GGUF"
  - filename: "Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
    url: "https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
    size_bytes: 4920739232
    source_repo: "bartowski/Meta-Llama-3.1-8B-Instruct-GGUF"
  - filename: "Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
    url: "https://huggingface.co/bartowski/Qwen2.5-Coder-14B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
    size_bytes: 8988111072
    source_repo: "bartowski/Qwen2.5-Coder-14B-Instruct-GGUF"
  - filename: "nomic-embed-text-v1.5-Q4_K_M.gguf"
    url: "https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF/resolve/main/nomic-embed-text-v1.5.Q4_K_M.gguf"
    size_bytes: 84106624
    source_repo: "nomic-ai/nomic-embed-text-v1.5-GGUF"
  # Added t_c5cef2b2 (2026-08-19, War Machine): Qwen3-8B dense 8B model for
  # aux tasks (routing, rewriting, structured extraction, tool-call construction).
  # Source: bartowski/Qwen_Qwen3-8B-GGUF (public, no auth). HF filename is
  # Qwen_Qwen3-8B-Q4_K_M.gguf; stored locally as Qwen3-8B-Q4_K_M.gguf.
  # Exact size verified from HF manifest (content-length): 5,027,784,224 bytes.
  # VRAM: ~4.68GB weights + ~0.5GB KV @ 32K ctx (q4_0) ≈ 5.2GB total.
  # Thinking mode ON by default; use /no_think for latency-sensitive aux tasks.
  - filename: "Qwen3-8B-Q4_K_M.gguf"
    url: "https://huggingface.co/bartowski/Qwen_Qwen3-8B-GGUF/resolve/main/Qwen_Qwen3-8B-Q4_K_M.gguf"
    size_bytes: 5027784224
    source_repo: "bartowski/Qwen_Qwen3-8B-GGUF"
 # --- deploy-vllm role: vllm_models override (t_r1d32b_swap, 2026-09-01) -----
 # Ansible's hash_behaviour is "replace" (see ansible.cfg) — a host_vars list
 # variable REPLACES the role default list wholesale, it does not deep-merge.
 #
 # SWAP (Ryan direction, 2026-09-01): Qwen2.5-32B-Instruct-AWQ retired,
 # replaced with DeepSeek-R1-Distill-Qwen-32B-AWQ, max_model_len=32768.
 # "Single model only" — nomic-embed-text-v1.5 (embedding, :8020) and
 # Qwen3-8B-AWQ (aux, :8010, already disabled) are BOTH disabled here.
 # DeepSeek gets the full 24GB card to itself. Nothing in production
 # consumed nomic-embed at the time of this swap (Hindsight uses its own
 # bundled 384-dim embedder; OpenViking pointed at the old llama-swap
 # endpoint, already stopped) — confirmed with Ryan before disabling.
 #
 # Model choice: casperhansen/deepseek-r1-distill-qwen-32b-awq — same
 # quantizer/toolchain (AutoAWQ) as the outgoing Qwen2.5-32B-Instruct-AWQ,
 # widely used, 4-bit GEMM AWQ, ~19.3GB on disk (4 safetensors shards).
 # Architecture: Qwen2ForCausalLM (DeepSeek-R1 distilled onto Qwen2.5-32B
 # base) — same vLLM code path as the outgoing model, no new serving
 # support needed. Native max_position_embeddings=131072; we cap at 32768
 # per the task's explicit max-model-len requirement.
 #
 # VRAM math: ~19.3GB weights (4-bit AWQ) + KV cache at 32768 ctx (GQA,
 # 8 KV heads, 128 head_dim, 64 layers, fp16 KV by default) ≈ 19.3GB +
 # ~4GB KV+overhead ≈ 23.3GB — tight but the FULL 24GB card is now
 # available (no co-resident nomic-embed/Qwen3-8B taking a share, unlike
 # the outgoing Qwen2.5-32B config). gpu_memory_utilization=0.95 (role
 # default) + enforce_eager retained as the proven-stable mitigation from
 # t_e6facb19/t_ca1af9fb (avoids CUDA graph capture VRAM spike; this host's
 # only validated way to avoid crash-loop-to-stabilize behavior on this
 # card). If 0.95 OOMs at 32768 ctx once tested live, drop to 0.90 next
 # (documented fallback, same pattern as the outgoing model).
 #
 # DeepSeek-R1 output note: reasoning traces stream in <think> tags before
 # the final answer — this is expected R1-distill behavior, not a bug.
 # Model card recommends temperature 0.5-0.7 (not 0, not vLLM's greedy
 # default) to avoid repetition/incoherence; not set here (server-side
 # default), left to be set client-side per the model card's guidance —
 # flagging for whoever wires this into Hermes profile configs next.
 vllm_models:
  - id: "Gemma-4-26B-A4B-it-AWQ"
    hf_repo: "cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit"
    role: primary
    # NO quantization field set (unlike the AutoAWQ-quantized DeepSeek/
    # Qwen2.5 models above) — live test (2026-09-01) found this repo's
    # config.json declares quant_method: "compressed-tensors" (llm-compressor
    # tool output, not classic AutoAWQ), even though the repo name says
    # "AWQ-4bit". Passing --quantization awq explicitly caused a hard
    # pydantic ValidationError at every single startup attempt: "Quantization
    # method specified in the model config (compressed-tensors) does not
    # match the quantization method specified in the `quantization` argument
    # (awq)." vLLM auto-detects the quant method correctly from the model's
    # own config.json when --quantization is omitted — confirmed fix, clean
    # start. Lesson: don't trust a HF repo's naming convention ("...-AWQ...")
    # for the `quantization:` field here — check config.json's quant_method.
    port: 8000
    # Ryan direction (2026-09-01, t_gemma4_swap): DeepSeek-R1-Distill-Qwen-32B
    # retired after confirming its `auto` tool-choice reliability is a known,
    # documented DeepSeek-R1-distillation limitation (trained on pure
    # reasoning traces, no function-calling data — GitHub-confirmed upstream,
    # not a vLLM config gap). Replaced with Gemma 4 26B A4B (Google,
    # Apache 2.0, US-origin — matches Ryan's standing model-origin
    # preference, unlike Qwen/DeepSeek). Chose MoE (26B A4B, 3.8B active)
    # over the dense 31B variant: ~3.7GB smaller on-disk AWQ footprint
    # (17.2GB vs 20.9GB) buys more KV-cache headroom on this tight 24GB
    # card, and decode should be faster (memory-bandwidth-bound on active
    # params, not total params). Tradeoff accepted: MoE scores lower than
    # dense on the Tau2 tool-use benchmark (68.2% vs 76.9%) but still beats
    # every other size in the family except the 31B on most reasoning
    # benchmarks. Model choice: cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit —
    # AutoAWQ 4-bit group_size=32, MoE expert layers (gate/up/down/router)
    # explicitly excluded from quantization ("ignore" list in config.json)
    # per standard llm-compressor MoE quant practice — only the dense
    # attention/projection layers are 4-bit, experts stay higher precision.
    # Native architecture: Gemma4ForConditionalGeneration (registered
    # natively in this host's installed vLLM 0.28.0 — vllm/model_executor/
    # models/registry.py line 415 — no plugin/trust-remote-code needed).
    # Native max_position_embeddings: 262144 (256K) — Hermes's 64K floor is
    # comfortably covered without any context-extension trick.
    max_model_len: 65536
    # VRAM math (not yet live-validated — see swap validation log below
    # once run): AWQ weights ~17.2GB on disk (dense attn 4-bit + MoE
    # experts higher-precision, per config.json's compressed-tensors
    # ignore list). Starting the KV cache dtype at int4_per_token_head
    # from the outset (rather than fp16 -> fp8 -> int4 trial-and-error like
    # the DeepSeek swap) since that same escalation pattern is expected to
    # repeat on this VRAM-constrained card for any 20+ GB model at >32K ctx.
    kv_cache_dtype: int4_per_token_head
    gpu_memory_utilization: 0.95
    enforce_eager: true
    # Native tool-calling + reasoning support (no `hermes` workaround
    # needed, unlike DeepSeek-R1-Distill): Gemma4EngineToolParser and
    # Gemma4ParserReasoningAdapter are both registered natively in this
    # host's vLLM 0.28.0 (vllm/tool_parsers/__init__.py,
    # vllm/reasoning/__init__.py) — purpose-built for this model's actual
    # output format, not a same-family approximation.
    enable_auto_tool_choice: true
    tool_call_parser: gemma4
    reasoning_parser: gemma4
    enabled: true
  - id: "Qwen3-8B-AWQ"
    hf_repo: "Qwen/Qwen3-8B-AWQ"
    role: aux
    quantization: awq
    port: 8010
    max_model_len: 32768
    gpu_memory_utilization: 0.15
    enforce_eager: true
    enabled: false   # single-model deployment — see swap note above
  - id: "nomic-embed-text-v1.5"
    hf_repo: "nomic-ai/nomic-embed-text-v1.5"
    role: embedding
    quantization: none
    port: 8020
    max_model_len: 2048
    gpu_memory_utilization: 0.05
    trust_remote_code: true
    enabled: false   # single-model deployment — see swap note above
 # --- deploy-vllm role: boot persistence (unchanged) -------------------------
 # Still permanent/boot-persistent — same policy as the outgoing Qwen2.5-32B
 # deployment (t_5508360a), just now serving one model instead of two.
 vllm_service_enabled: true
 vllm_service_state: started
@@ -0,0 +1,16 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: ansible/host_vars/astro_orbiter/vars.yml
 # HOST: astro-orbiter (10.1.71.130)
 # ROLE: Ollama inference host with AMD RX 5700 GPU passthrough
 # ------------------------------------------------------------------------------
 ansible_host: 10.1.71.131
 ansible_user: wed
 ansible_become: true
 # LVM root expansion — xlarge template uses sda3 partition, standard VG/LV names
 common_expand_root_lvm: true
 common_root_pv: /dev/sda3
 common_root_vg: ubuntu-vg
 common_root_lv: ubuntu-lv
@@ -1,49 +0,0 @@
 ---
 # file: host_vars/vm_template/vars
 # Ansible vars template for hosts created via cloning.
 # Supported hypervisors:
 # - Proxmox
 platform: "proxmox"
 # This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
 proxmox_clone_node: "fantasyland"
 # Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
 # Current OS offerings are:
 # - Fedora (42)
 # Current VM sizes are:
 # - Small: 2 cores, 2GB memory, 8 GiB virtual disk
 # - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
 # - Large: 4 cores, 4GB memory, 32 GiB virtual disk
 # - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
 # - Xlarge Plus: 4 cores, 8GB memory, 128 GiB virtual disk
 vm_clone_source: "fedora-42-xlarge-plus"
 # Proxmox storage target.
 vm_storage: "general"
 # Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
 # and backup groups. (ha_group will be factored out in the next functionality update.)
 ha_group: "fantasyland"
 proxmox_host_target: "fantasyland"
 # Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
 # sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
 ip_address: 10.1.71.21
 # Software
 # Future enhancement will allow specification of additional software to automatically deploy to
 # the VM after creation.
 #terraform_version: "1.11.3"
 # ---
 hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn
@@ -0,0 +1,12 @@
 ---
 # file: host_vars/guest-relations/vars
 # guest-relations — Guest Relations
 # VM provisioned by Terraform (pre-pipeline bootstrap)
 # ─── Network ─────────────────────────────────────────────────────────────────
 ip_address: 10.1.71.40
 # ─── Application ─────────────────────────────────────────────────────────────
 app_role: identity_provider
 app_name: authentik
 app_deployment: docker_compose
@@ -0,0 +1,26 @@
 ---
 # Paperclip Docker deployment
 paperclip_domain: cranium-command.wdw-labs.cloud
 paperclip_base_path: /opt/paperclip
 paperclip_port: 3100
 certbot_email: ryan.blundon@protonmail.com
 # vLLM inference endpoint (local)
 vllm_endpoint: "http://127.0.0.1:8000"
 hermes_model:
  provider: custom
  default: Gemma-4-26B-A4B-Hermes-VLM-MLX      # from http://fantasmic.wdw-labs.cloud:8000/v1/models
  base_url: http://fantasmic.wdw-labs.cloud:8000/v1
  api_key: "{{ vault_fantasmic_api_key }}"
 # # DNS entries (if using your existing dns-manager role)
 # dns_entries:
 #   - name: illuminations
 #     domain: wdw-labs.cloud
 #     type: A
 #     value: "{{ ansible_host }}"
 #   - name: muppet-vision
 #     domain: wdw-labs.cloud
 #     type: A
 #     value: "{{ ansible_host }}"
@@ -0,0 +1,244 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: ansible/host_vars/illuminations/vars.yml
 # HOST: illuminations (10.1.71.130)
 # ROLE: llama.cpp LLM inference host — Ryzen 7 5800XT / RTX 3090 (ATX rebuild,
 #       2026-08-04). Superseded the prior AMD RX 5700 / Ollama config below;
 #       drive was transplanted into new hardware, not reinstalled.
 # ------------------------------------------------------------------------------
 ansible_host: 10.1.71.130
 ansible_user: jarvis
 ansible_ssh_private_key_file: ~/.ssh/id_jarvis
 ansible_become: true
 # LVM root expansion — xlarge template uses sda3 partition, standard VG/LV names
 common_expand_root_lvm: true
 common_root_pv: /dev/sda3
 common_root_vg: ubuntu-vg
 common_root_lv: ubuntu-lv
 # --- Staged GGUF models for the llama.cpp router (:8002) ---------------------
 # Data-driven list consumed by roles/llm-inference-multimodel tasks/models.yml
 # (loop -> tasks/stage_model.yml). Each entry is idempotently staged into
 # /opt/models: stat + EXACT-size check vs HF manifest; skip (no download, no
 # restart) when present + size matches. Source repos are public bartowski GGUFs
 # on HuggingFace (no auth). A router restart is notified ONLY when a new GGUF
 # is actually downloaded.
 # Added 2026-08-12 (War Machine): codify Phi-3.5-mini-instruct-Q8_0 and
 # Meta-Llama-3.1-8B-Instruct-Q4_K_M as router models alongside the production
 # Qwen3.6-35B-A3B-UD-Q4_K_S. The live files were already present/correct on
 # astro-orbiter; this pass codifies them. Future adds = append to this list.
 # Router --models-max override for astro-orbiter.
 # Default in defaults/main.yml is 1 (conservative). Bumped to 4 on 2026-08-12
 # (t_33acbb2e) so the router can keep more than one GGUF resident on-demand
 # and LRU-evict when needed.
 #
 # VRAM NOTE (t_33acbb2e, updated t_55c164f5, updated t_34b96e83, updated t_f5f7e9ad, updated t_441470b9, updated t_c5cef2b2):
 # With models-max=4 and all 6 GGUFs registered, worst case is all 6 loaded simultaneously:
 #   Qwen3.8-27B Q4_K_M:               ~20.0GB (weights ~17.1GB + KV ~2.9GB @ 65536 ctx, q4_0) ← CORRECTED (ctx rolled back from 128K to 65536, t_c9fed26c 2026-08-18)
 #   Phi-3.5-mini-instruct Q8_0:       ~4.3GB  (weights ~3.8GB + KV ~0.5GB @ 32K ctx)
 #   Meta-Llama-3.1-8B Q4_K_M:        ~5.6GB  (weights ~4.6GB + KV ~0.2GB @ 8K ctx)
 #   Qwen2.5-Coder-14B Q4_K_M:        ~9.0GB  (weights ~8.4GB + KV ~0.6GB @ 16K ctx)
 #   nomic-embed-text-v1.5 Q4_K_M:    ~0.09GB (~84MB, embedding only — no KV cache)
 #   Qwen3-8B Q4_K_M:                 ~5.5GB  (weights ~4.68GB + KV ~0.5GB @ 32K ctx, q4_0)
 #   Total worst-case:                 ~44.5GB  >>  24GB RTX 3090
 #
 # OOM RISK: Full co-residency is impossible on 24GB. LRU eviction prevents this
 # in practice: models-max=4 means the router can REGISTER 6 models but only keeps
 # up to 4 LOADED simultaneously — the router will evict the LRU model when a new
 # one is needed. nomic-embed-text-v1.5 is pinned via sleep-idle-seconds=-1 and
 # load-on-startup=true but it uses only ~84MB, so it never meaningfully changes
 # the budget. In single-user homelab operation, only one generative model is active
 # at a time alongside the always-resident embedding model.
 # Qwen3.8-27B alone uses ~17,804 MiB (weights+KV @ 65536 ctx); co-residency
 # with Coder (~9GB) = ~27GB > 24GB. LRU eviction handles this automatically.
 # Ryan should be aware this means model-switching always incurs a ~30-60s
 # cold-load latency when switching between Qwen3.8-27B and any other model.
 # Proceeding to models-max=4 as instructed; flagged for Ryan's attention.
 # Router --models-max override for astro-orbiter.
 # UPDATED (t_f5f7e9ad, 2026-08-16): Set to 2 because Qwen3.8-27B-Q4_K_M
 # uses 17,804 MiB at 65536 ctx. Only nomic-embed (558MB, pinned) and ONE
 # generative model can be resident simultaneously. Co-residency of Qwen3.8
 # with any auxiliary model (Phi 8.3GB, Llama 5.9GB, Coder 9GB) exceeds 24GB.
 # models-max=2: slot 1 = nomic-embed (pinned, always loaded), slot 2 = LRU
 # generative model (Qwen3.8 primary, cold-loaded on first request ~30-60s;
 # auxiliary models evict it on demand, and vice versa).
 # NOTE: Qwen3.8 does NOT have load-on-startup — it loads on first request.
 # This avoids an LRU eviction race with nomic-embed at startup.
 # UPDATED (t_72646029, 2026-08-17): CPU offload for Coder + Llama changes the
 # constraint. Coder and Llama now use CPU inference (n-gpu-layers=0). GPU-resident
 # VRAM: Qwen3.8 (~17,804 MiB at 65536 ctx) + nomic-embed (558 MiB, pinned) plus
 # the CUDA-context buffers llama.cpp 6ea215d allocates for the CPU models (~1.4-1.7GB
 # each) = ~20,004 MiB steady-state, below the 24,576 MiB physical limit.
 # CORRECTED (t_c5cef2b2, 2026-08-19): ctx-size was rolled back from 131072 to 65536
 # (t_c9fed26c 2026-08-18). Qwen3.8 VRAM at 65536: 17,804 MiB (not 20,302 MiB).
 # models-max raised to 4: nomic (slot 1, pinned) + Qwen3.8 (slot 2, GPU) +
 # Llama (slot 3, CPU) + Coder (slot 4, CPU). Phi (GPU, ~8.3GB) and new
 # Qwen3-8B (GPU, ~5.5GB) can also be requested but evict Qwen3.8 due to VRAM.
 # models-max=4 is required so CPU-offloaded models count as loaded without
 # evicting Qwen3.8.
 llm_router_models_max: 4
 llm_staged_models:
  - filename: "Phi-3.5-mini-instruct-Q8_0.gguf"
    url: "https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF/resolve/main/Phi-3.5-mini-instruct-Q8_0.gguf"
    size_bytes: 4061222688
    source_repo: "bartowski/Phi-3.5-mini-instruct-GGUF"
  - filename: "Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
    url: "https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
    size_bytes: 4920739232
    source_repo: "bartowski/Meta-Llama-3.1-8B-Instruct-GGUF"
  - filename: "Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
    url: "https://huggingface.co/bartowski/Qwen2.5-Coder-14B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
    size_bytes: 8988111072
    source_repo: "bartowski/Qwen2.5-Coder-14B-Instruct-GGUF"
  - filename: "nomic-embed-text-v1.5-Q4_K_M.gguf"
    url: "https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF/resolve/main/nomic-embed-text-v1.5.Q4_K_M.gguf"
    size_bytes: 84106624
    source_repo: "nomic-ai/nomic-embed-text-v1.5-GGUF"
  # Added t_c5cef2b2 (2026-08-19, War Machine): Qwen3-8B dense 8B model for
  # aux tasks (routing, rewriting, structured extraction, tool-call construction).
  # Source: bartowski/Qwen_Qwen3-8B-GGUF (public, no auth). HF filename is
  # Qwen_Qwen3-8B-Q4_K_M.gguf; stored locally as Qwen3-8B-Q4_K_M.gguf.
  # Exact size verified from HF manifest (content-length): 5,027,784,224 bytes.
  # VRAM: ~4.68GB weights + ~0.5GB KV @ 32K ctx (q4_0) ≈ 5.2GB total.
  # Thinking mode ON by default; use /no_think for latency-sensitive aux tasks.
  - filename: "Qwen3-8B-Q4_K_M.gguf"
    url: "https://huggingface.co/bartowski/Qwen_Qwen3-8B-GGUF/resolve/main/Qwen_Qwen3-8B-Q4_K_M.gguf"
    size_bytes: 5027784224
    source_repo: "bartowski/Qwen_Qwen3-8B-GGUF"
 # --- deploy-vllm role: vllm_models override (t_r1d32b_swap, 2026-09-01) -----
 # Ansible's hash_behaviour is "replace" (see ansible.cfg) — a host_vars list
 # variable REPLACES the role default list wholesale, it does not deep-merge.
 #
 # SWAP (Ryan direction, 2026-09-01): Qwen2.5-32B-Instruct-AWQ retired,
 # replaced with DeepSeek-R1-Distill-Qwen-32B-AWQ, max_model_len=32768.
 # "Single model only" — nomic-embed-text-v1.5 (embedding, :8020) and
 # Qwen3-8B-AWQ (aux, :8010, already disabled) are BOTH disabled here.
 # DeepSeek gets the full 24GB card to itself. Nothing in production
 # consumed nomic-embed at the time of this swap (Hindsight uses its own
 # bundled 384-dim embedder; OpenViking pointed at the old llama-swap
 # endpoint, already stopped) — confirmed with Ryan before disabling.
 #
 # Model choice: casperhansen/deepseek-r1-distill-qwen-32b-awq — same
 # quantizer/toolchain (AutoAWQ) as the outgoing Qwen2.5-32B-Instruct-AWQ,
 # widely used, 4-bit GEMM AWQ, ~19.3GB on disk (4 safetensors shards).
 # Architecture: Qwen2ForCausalLM (DeepSeek-R1 distilled onto Qwen2.5-32B
 # base) — same vLLM code path as the outgoing model, no new serving
 # support needed. Native max_position_embeddings=131072; we cap at 32768
 # per the task's explicit max-model-len requirement.
 #
 # VRAM math: ~19.3GB weights (4-bit AWQ) + KV cache at 32768 ctx (GQA,
 # 8 KV heads, 128 head_dim, 64 layers, fp16 KV by default) ≈ 19.3GB +
 # ~4GB KV+overhead ≈ 23.3GB — tight but the FULL 24GB card is now
 # available (no co-resident nomic-embed/Qwen3-8B taking a share, unlike
 # the outgoing Qwen2.5-32B config). gpu_memory_utilization=0.95 (role
 # default) + enforce_eager retained as the proven-stable mitigation from
 # t_e6facb19/t_ca1af9fb (avoids CUDA graph capture VRAM spike; this host's
 # only validated way to avoid crash-loop-to-stabilize behavior on this
 # card). If 0.95 OOMs at 32768 ctx once tested live, drop to 0.90 next
 # (documented fallback, same pattern as the outgoing model).
 #
 # DeepSeek-R1 output note: reasoning traces stream in <think> tags before
 # the final answer — this is expected R1-distill behavior, not a bug.
 # Model card recommends temperature 0.5-0.7 (not 0, not vLLM's greedy
 # default) to avoid repetition/incoherence; not set here (server-side
 # default), left to be set client-side per the model card's guidance —
 # flagging for whoever wires this into Hermes profile configs next.
 vllm_models:
  - id: "Gemma-4-26B-A4B-it-AWQ"
    hf_repo: "cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit"
    role: primary
    # NO quantization field set (unlike the AutoAWQ-quantized DeepSeek/
    # Qwen2.5 models above) — live test (2026-09-01) found this repo's
    # config.json declares quant_method: "compressed-tensors" (llm-compressor
    # tool output, not classic AutoAWQ), even though the repo name says
    # "AWQ-4bit". Passing --quantization awq explicitly caused a hard
    # pydantic ValidationError at every single startup attempt: "Quantization
    # method specified in the model config (compressed-tensors) does not
    # match the quantization method specified in the `quantization` argument
    # (awq)." vLLM auto-detects the quant method correctly from the model's
    # own config.json when --quantization is omitted — confirmed fix, clean
    # start. Lesson: don't trust a HF repo's naming convention ("...-AWQ...")
    # for the `quantization:` field here — check config.json's quant_method.
    port: 8000
    # Ryan direction (2026-09-01, t_gemma4_swap): DeepSeek-R1-Distill-Qwen-32B
    # retired after confirming its `auto` tool-choice reliability is a known,
    # documented DeepSeek-R1-distillation limitation (trained on pure
    # reasoning traces, no function-calling data — GitHub-confirmed upstream,
    # not a vLLM config gap). Replaced with Gemma 4 26B A4B (Google,
    # Apache 2.0, US-origin — matches Ryan's standing model-origin
    # preference, unlike Qwen/DeepSeek). Chose MoE (26B A4B, 3.8B active)
    # over the dense 31B variant: ~3.7GB smaller on-disk AWQ footprint
    # (17.2GB vs 20.9GB) buys more KV-cache headroom on this tight 24GB
    # card, and decode should be faster (memory-bandwidth-bound on active
    # params, not total params). Tradeoff accepted: MoE scores lower than
    # dense on the Tau2 tool-use benchmark (68.2% vs 76.9%) but still beats
    # every other size in the family except the 31B on most reasoning
    # benchmarks. Model choice: cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit —
    # AutoAWQ 4-bit group_size=32, MoE expert layers (gate/up/down/router)
    # explicitly excluded from quantization ("ignore" list in config.json)
    # per standard llm-compressor MoE quant practice — only the dense
    # attention/projection layers are 4-bit, experts stay higher precision.
    # Native architecture: Gemma4ForConditionalGeneration (registered
    # natively in this host's installed vLLM 0.28.0 — vllm/model_executor/
    # models/registry.py line 415 — no plugin/trust-remote-code needed).
    # Native max_position_embeddings: 262144 (256K) — Hermes's 64K floor is
    # comfortably covered without any context-extension trick.
    max_model_len: 65536
    # VRAM math (not yet live-validated — see swap validation log below
    # once run): AWQ weights ~17.2GB on disk (dense attn 4-bit + MoE
    # experts higher-precision, per config.json's compressed-tensors
    # ignore list). Starting the KV cache dtype at int4_per_token_head
    # from the outset (rather than fp16 -> fp8 -> int4 trial-and-error like
    # the DeepSeek swap) since that same escalation pattern is expected to
    # repeat on this VRAM-constrained card for any 20+ GB model at >32K ctx.
    kv_cache_dtype: int4_per_token_head
    gpu_memory_utilization: 0.95
    enforce_eager: true
    # Native tool-calling + reasoning support (no `hermes` workaround
    # needed, unlike DeepSeek-R1-Distill): Gemma4EngineToolParser and
    # Gemma4ParserReasoningAdapter are both registered natively in this
    # host's vLLM 0.28.0 (vllm/tool_parsers/__init__.py,
    # vllm/reasoning/__init__.py) — purpose-built for this model's actual
    # output format, not a same-family approximation.
    enable_auto_tool_choice: true
    tool_call_parser: gemma4
    reasoning_parser: gemma4
    enabled: true
  - id: "Qwen3-8B-AWQ"
    hf_repo: "Qwen/Qwen3-8B-AWQ"
    role: aux
    quantization: awq
    port: 8010
    max_model_len: 32768
    gpu_memory_utilization: 0.15
    enforce_eager: true
    enabled: false   # single-model deployment — see swap note above
  - id: "nomic-embed-text-v1.5"
    hf_repo: "nomic-ai/nomic-embed-text-v1.5"
    role: embedding
    quantization: none
    port: 8020
    max_model_len: 2048
    gpu_memory_utilization: 0.05
    trust_remote_code: true
    enabled: false   # single-model deployment — see swap note above
 # --- deploy-vllm role: boot persistence (unchanged) -------------------------
 # Still permanent/boot-persistent — same policy as the outgoing Qwen2.5-32B
 # deployment (t_5508360a), just now serving one model instead of two.
 vllm_service_enabled: true
 vllm_service_state: started
 # Paperclip Docker deployment
 nginx_proxy_certbot_email: ryan.blundon@protonmail.com
 # nginx_proxy_certbot_staging: true # remove after first successful real issuance
 hermes_model:
  provider: custom
  default: Gemma-4-26B-A4B-Hermes-VLM-MLX      # from http://fantasmic.wdw-labs.cloud:8000/v1/models
  base_url: http://fantasmic.wdw-labs.cloud:8000/v1
  api_key: "{{ vault_fantasmic_api_key }}"
@@ -1,29 +0,0 @@
 ---
 # file: host_vars/int-master01/vars
 # Ansible vars template for OpenShift nodes created via Ansible (configure_openshift_cluster).
 # Cluster deployed via Agent Installer and Matchbox (iPXE):
 # Role of this node in the cluster. 
 # - master
 # - worker
 node_role: "master"
 # IP address assigned to this node on the Primary Interface
 ip_address: 10.1.71.61
 # Ansible Playbook supports booting node via one interface, but installing on a 
 # different interface. (Disabling the boot interface on install)  All these fields
 # are necessary.  In the case of a single NIC node, interfaces and MAC addresses
 # should be the same.
 boot_interface: "enp0s31f6"
 boot_mac_address: "54:bf:64:60:52:2d"
 primary_interface: "enp1s0f0"
 install_mac_address: "98:b7:85:20:47:22"
 # Static Variables
 hostname: "{{ inventory_hostname }}.{{ base_domain }}"
@@ -1,29 +0,0 @@
 ---
 # file: host_vars/int-master02/vars
 # Ansible vars template for OpenShift nodes created via Ansible (configure_openshift_cluster).
 # Cluster deployed via Agent Installer and Matchbox (iPXE):
 # Role of this node in the cluster. 
 # - master
 # - worker
 node_role: "master"
 # IP address assigned to this node on the Primary Interface
 ip_address: 10.1.71.62
 # Ansible Playbook supports booting node via one interface, but installing on a 
 # different interface. (Disabling the boot interface on install)  All these fields
 # are necessary.  In the case of a single NIC node, interfaces and MAC addresses
 # should be the same.
 boot_interface: "enp0s31f6"
 boot_mac_address: "54:bf:64:5f:a2:80"
 primary_interface: "enp1s0f0"
 install_mac_address: "98:b7:85:20:47:3a"
 # ---
 hostname: "{{ inventory_hostname }}.{{ base_domain }}"
@@ -1,29 +0,0 @@
 ---
 # file: host_vars/int-master03/vars
 # Ansible vars template for OpenShift nodes created via Ansible (configure_openshift_cluster).
 # Cluster deployed via Agent Installer and Matchbox (iPXE):
 # Role of this node in the cluster. 
 # - master
 # - worker
 node_role: "master"
 # IP address assigned to this node on the Primary Interface
 ip_address: 10.1.71.63
 # Ansible Playbook supports booting node via one interface, but installing on a 
 # different interface. (Disabling the boot interface on install)  All these fields
 # are necessary.  In the case of a single NIC node, interfaces and MAC addresses
 # should be the same.
 boot_interface: "enp0s31f6"
 boot_mac_address: "d8:9e:f3:4c:02:a9"
 primary_interface: "enp1s0f0"
 install_mac_address: "98:b7:85:1f:89:cf"
 # ---
 hostname: "{{ inventory_hostname }}.{{ base_domain }}"
@@ -1,29 +0,0 @@
 ---
 # file: host_vars/int-worker02/vars
 # Ansible vars template for OpenShift nodes created via Ansible (configure_openshift_cluster).
 # Cluster deployed via Agent Installer and Matchbox (iPXE):
 # Role of this node in the cluster. 
 # - master
 # - worker
 node_role: "worker"
 # IP address assigned to this node on the Primary Interface
 ip_address: 10.1.71.164
 # Ansible Playbook supports booting node via one interface, but installing on a 
 # different interface. (Disabling the boot interface on install)  All these fields
 # are necessary.  In the case of a single NIC node, interfaces and MAC addresses
 # should be the same.
 boot_interface: "enp87s0"
 boot_mac_address: "36:47:ca:76:13:49"
 primary_interface: "enp2s0f0"
 install_mac_address: "36:47:ca:76:13:47"
 # ---
 hostname: "{{ inventory_hostname }}.{{ base_domain }}"
@@ -1,29 +0,0 @@
 ---
 # file: host_vars/int-worker02/vars
 # Ansible vars template for OpenShift nodes created via Ansible (configure_openshift_cluster).
 # Cluster deployed via Agent Installer and Matchbox (iPXE):
 # Role of this node in the cluster. 
 # - master
 # - worker
 node_role: "worker"
 # IP address assigned to this node on the Primary Interface
 ip_address: 10.1.71.65
 # Ansible Playbook supports booting node via one interface, but installing on a 
 # different interface. (Disabling the boot interface on install)  All these fields
 # are necessary.  In the case of a single NIC node, interfaces and MAC addresses
 # should be the same.
 boot_interface: "enp87s0"
 boot_mac_address: "58:47:ca:76:13:49"
 primary_interface: "enp2s0f0"
 install_mac_address: "58:47:ca:76:13:47"
 # ---
 hostname: "{{ inventory_hostname }}.{{ base_domain }}"
@@ -1,33 +1,12 @@
 ---
 # file: host_vars/lightning_lane/vars
-# Traefik Load Balancer Server
+# lightning-lane — Traefik Reverse Proxy / Load Balancer
 # VM provisioned by Terraform (pre-pipeline bootstrap)
-# Supported hypervisors:
+# ─── Network ─────────────────────────────────────────────────────────────────
-# - Proxmox
+ip_address: 10.1.71.35
-platform: "proxmox"
+# ─── Application ─────────────────────────────────────────────────────────────
-
+app_role: reverse_proxy
-# This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
+app_name: traefik
-proxmox_clone_node: "fantasyland"
+app_deployment: docker_compose
 # VM Configuration
 vm_os_distribution: "ubuntu"
 vm_os_version: "24.04"
 vm_size: "small"
 vm_storage: "general"
 # Proxmox HA Configuration
 ha_group: "pve02"
 proxmox_host_target: "pve02"
 # Network Configuration
 # IP address for the Traefik load balancer
 ip_address: 10.1.71.24
 #vm_mac_address: 'BC:24:11:50:00:80'
 # Software Configuration
 # Traefik will be installed and configured on this host
 # VM Template Configuration
 #vm_clone_source: "{{ vm_os_distribution }}-{{ vm_os_version }}-{{ vm_size }}"
 hostname: "{{ inventory_hostname }}.{{ base_domain }}"
@@ -0,0 +1,16 @@
 ---
 # Host-specific vars for main-street-station (JMRI headless server)
 # LCRR - Lake Country Railroad, Milwaukee Road Oct 1956, HO scale
 # JMRI profile ID — find with: ls ~/.jmri/profiles/ on the old box
 # Format: <name>.<8-char-hex>  e.g. LCRR.3d3f1dfc
 # TODO: fill in after restoring config from GitHub backup
 jmri_profile_id: ""
 # USB serial device for NCE command station
 # Verify after install: ls -la /dev/ttyUSB* /dev/ttyACM*
 jmri_serial_device: /dev/ttyUSB0
 # Path to JMRI config backup for restore task (leave empty to skip)
 # Point at a local checkout of the LCRR GitHub repo
 jmri_config_src: ""
@@ -0,0 +1,10 @@
 ---
 # main-street-station — JMRI / LCRR server
 jmri_profile_id: "Lake_Country_Railroad.3e8b1d4b"
 jmri_lcrr_repo: "ssh://git@gitea.mk-labs.cloud:2221/rblundon/LCRR.git"
 jmri_lcrr_branch: "clean-profile"
 jmri_leviton_email: "{{ leviton_email }}"
 jmri_leviton_password: "{{ leviton_password }}"
 jmri_ssh_authorized_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINnSM/9fO8rz/amqkyoGUzUKNNzzmtSXPwOCr1O9zKNO ansible"
 jmri_ssh_authorized_keys_extra:
  - "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG6HaK4Y21UwPRbAZ986L7I9QnUdyq53114+9kO8X4bL rblundon@laptop"
@@ -1,17 +0,0 @@
 ---
 # file: host_vars/monitor/vars
 # VM Definition
 platform: "proxmox"
 proxmox_clone_node: "pve03"
 vm_clone_source: "fedora-42-large"
 vm_storage: "pve-general"
 ha_group: "pve03"
 # Networking
 ip_address: 10.1.71.249
 # Software
 #terraform_version: "1.11.3"
 hostname: "{{ inventory_hostname }}.{{ base_domain }}" #change variable to fqdn
@@ -1,31 +0,0 @@
 ---
 # file: host_vars/ocp-hub/vars
 # file: host_vars/int-master01/vars
 # Ansible vars template for OpenShift nodes created via Ansible (configure_openshift_cluster).
 # Cluster deployed via Agent Installer and Matchbox (iPXE):
 # Role of this node in the cluster. 
 # - master
 # - worker
 node_role: "master"
 # IP address assigned to this node on the Primary Interface
 ip_address: 10.1.71.10
 # Ansible Playbook supports booting node via one interface, but installing on a 
 # different interface. (Disabling the boot interface on install)  All these fields
 # are necessary.  In the case of a single NIC node, interfaces and MAC addresses
 # should be the same.
 boot_interface: "enp0s31f6"
 boot_mac_address: "50:9A:4C:52:A8:5D"
 primary_interface: "enp1s0f0"
 install_mac_address: "98:b7:85:1e:c6:f1"
 # Static Variables
 hostname: "{{ inventory_hostname }}.{{ base_domain }}"
@@ -1,29 +0,0 @@
 ---
 # file: host_vars/openshift_node_template/vars
 # Ansible vars template for OpenShift nodes created via Ansible (configure_openshift_cluster).
 # Cluster deployed via Agent Installer and Matchbox (iPXE):
 # Role of this node in the cluster. 
 # - master
 # - worker
 node_role: "master"
 # IP address assigned to this node on the Primary Interface
 ip_address: 10.1.71.61
 # Ansible Playbook supports booting node via one interface, but installing on a 
 # different interface. (Disabling the boot interface on install)  All these fields
 # are necessary.  In the case of a single NIC node, interfaces and MAC addresses
 # should be the same.
 boot_interface: "enp0s31f6"
 boot_mac_address: "54:bf:64:60:52:2d"
 primary_interface: "enp1s0f0"
 install_mac_address: "98:b7:85:20:47:22"
 # Static Variables
 hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn
@@ -0,0 +1,27 @@
 ---
 # file: host_vars/turnstile/vars
 # turnstile — Smallstep step-ca SSH Certificate Authority
 # VM provisioned by Terraform (pre-pipeline bootstrap)
 # ─── Network ─────────────────────────────────────────────────────────────────
 ip_address: 10.1.71.34
 # ─── Application ─────────────────────────────────────────────────────────────
 app_role: ssh_certificate_authority
 app_name: step-ca
 app_deployment: docker_compose
 # ─── step-ca Configuration ───────────────────────────────────────────────────
 stepca_hostname: turnstile.local.mk-labs.cloud
 stepca_dns_names: "turnstile.local.mk-labs.cloud,10.1.71.34"
 stepca_ssh_enabled: true
 stepca_listen_port: 9000
 # ─── OIDC Provisioner (Authentik) ────────────────────────────────────────────
 # Client ID and secret stored in vault
 stepca_oidc_provisioner_name: authentik
 stepca_oidc_configuration_endpoint: "https://authentik.local.mk-labs.cloud/application/o/step-ca/.well-known/openid-configuration"
 stepca_oidc_listen_address: ":10000"
 stepca_oidc_domains:
  - "local.mk-labs.cloud"
  - "protonmail.com"
@@ -0,0 +1,17 @@
 $ANSIBLE_VAULT;1.1;AES256
 30383930323363386234333433636465393263613336646464666365643730386430353864616334
 3865343031653162313736656437373666636136653263360a613961646133376262633164393535
 31363434633634303035663133316230633538363936303266373931313661346563333832653032
 6239376363646530620a303231363333363233623038326165316463383662656565626534396232
 62333133623530643932643430663166373930353733363866336533643233373261333130613635
 31633337376337663833613634666436383862386431636537373363343333323932363761653366
 61353166613162346436396138316561646165303566376366323462663861616364336539313138
 30386430373566353761383636626335393463376661356666303861353564313832346333396134
 30653861363866336331336164323130623230666237356165613934333239386536373664643065
 33643139346562346663313533643433353462363665323166313364373335366665373435643935
 65303863663864393238393732303065343364306264396333376233666233346664323334336532
 38373164383835663163363137643531303163396236623565666436363261393563343133333161
 36663934303633663062346161636333396135336135616136303664636562363862353764343562
 39643430396636313139336130656363306562363430346233313530663963366238326334623135
 62383632613061316262323038303333323739303137363334626637666663666263613132336433
 65346266393733633632
@@ -1,47 +0,0 @@
 ---
 # file: host_vars/unbound01/vars
 # Supported hypervisors:
 # - Proxmox
 platform: "proxmox"
 # This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
 proxmox_clone_node: "pve03"
 # Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
 # Current OS offerings are:
 # - Fedora (42)
 # Current VM sizes are:
 # - Small: 2 cores, 2GB memory, 8 GiB virtual disk
 # - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
 # - Large: 4 cores, 4GB memory, 32 GiB virtual disk
 # - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
 vm_clone_source: "fedora-42-small"
 # Proxmox storage target.
 vm_storage: "pve-general"
 # Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
 # and backup groups. (ha_group will be factored out in the next functionality update.)
 ha_group: "pve03"
 proxmox_host_target: "pve03"
 # Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
 # sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
 ip_address: 10.1.71.251
 # Software
 # Future enhancement will allow specification of additional software to automatically deploy to
 # the VM after creation.
 #terraform_version: "1.11.3"
 # ---
 hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn
@@ -1,47 +0,0 @@
 ---
 # file: host_vars/unbound02/vars
 # Supported hypervisors:
 # - Proxmox
 platform: "proxmox"
 # This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
 proxmox_clone_node: "pve03"
 # Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
 # Current OS offerings are:
 # - Fedora (42)
 # Current VM sizes are:
 # - Small: 2 cores, 2GB memory, 8 GiB virtual disk
 # - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
 # - Large: 4 cores, 4GB memory, 32 GiB virtual disk
 # - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
 vm_clone_source: "fedora-42-small"
 # Proxmox storage target.
 vm_storage: "pve-general"
 # Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
 # and backup groups. (ha_group will be factored out in the next functionality update.)
 ha_group: "pve03"
 proxmox_host_target: "pve03"
 # Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
 # sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
 ip_address: 10.1.71.252
 # Software
 # Future enhancement will allow specification of additional software to automatically deploy to
 # the VM after creation.
 #terraform_version: "1.11.3"
 # ---
 hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn
@@ -1,15 +1,12 @@
 # file: inventory.yml
 proxmox:
  hosts:
-    pve0[1:3]:
+    main-street-usa:
-
+      ansible_host: 10.1.71.11
-ntp_servers:
+      ansible_user: wed
-  hosts:
+      ansible_become: true
-    sundial:
+    tomorrowland:
-    
+    fantasyland:
 dns_server:
  hosts:
    monorail:
 magic_kingdom:
  hosts:
@@ -17,47 +14,120 @@ magic_kingdom:
    tomorrowland:
    fantasyland:
-prometheus_server:
+dns_server:
  hosts:
-    cinderella-castle:
+    monorail:
-prometheus_nodes:
+ntp_servers:
  hosts:
-    cinderella-castle:
+    sundial:
 load_balancers:
  hosts:
    lightning-lane:
      ansible_host: 10.1.71.24
      ansible_become: true
 # node_explorer:
 #   hosts:
 #     cinderella-castle:
-# alertmanager:
+step_ca_server:
-#   hosts:
+  hosts:
-#     cinderella-castle:
+    turnstile:
      ansible_become: true
-# grafana:
+authentik_server:
-#   hosts:
+  hosts:
-#     cinderella-castle:
+    guest-relations:
      ansible_become: true
 gitea_servers:
  hosts:
    mad-tea-party:
      ansible_user: wed
      ansible_become: true
 nextcloud_server:
  hosts:
    the-grid:
      ansible_become: true
 semaphore_server:
  hosts:
    figment:
      ansible_host: 10.1.71.37
      ansible_user: wed
      ansible_become: true
 n8n_server:
  hosts:
    tiki-room:
 astro_orbiter:
  hosts:
    astro-orbiter:
 claude_servers:
  hosts:
    haunted-mansion:
      ansible_host: 10.1.71.137
      ansible_user: wed
      ansible_become: true
      ansible_ssh_private_key_file: ~/.ssh/ansible
 hermes_server:
  hosts:
    carousel-of-progress:
      ansible_host: 10.1.71.131
      ansible_user: wed
      ansible_become: true
      ansible_ssh_private_key_file: ~/.ssh/ansible
 honcho_server:
  hosts:
    lincoln:
      ansible_host: 10.1.71.132
      ansible_user: wed
      ansible_become: true
 paperclip_server:
  hosts:
    illuminations:
      ansible_host: 10.1.71.130
      ansible_user: wed
      ansible_become: true
 multica_agent_hosts:
  hosts:
    haunted-mansion:
      ansible_host: haunted-mansion.wdw-labs.cloud
      multica_device_name: haunted-mansion
    spaceship-earth:
      ansible_host: spaceship-earth.wdw-labs.cloud
      multica_device_name: spaceship-earth
 multica:
  hosts:
    muppet-vision:
      ansible_host: 10.1.71.136
      ansible_user: wed
      ansible_become: true
 jmri_server:
  hosts:
    main-street-station:
      ansible_host: 192.168.10.40
      ansible_user: wed
      ansible_become: true
 papermc_server:
 # ansible-galaxy role install engonzal.papermc
  hosts:
    arcade:
-semaphore_server:
+dev_servers:
  hosts:
-    imagineering:
+    scrim:
-
+    backstage:
-n8n_server:
+      ansible_host: 10.1.71.133
-  hosts:
+      ansible_user: wed
-    tiki-room:
+      ansible_become: true
 traefik_server:
  hosts:
    lightning_lane:
 # dhcp_server:
 #   hosts:
@@ -67,13 +137,13 @@ backup_servers:
  hosts:
    timekeeper:
-# freeipa:
+talos_control:
-#   hosts:
+  hosts:
-#     infra01:
+    city-hall:
-
+      ansible_become: true
-# ipaserver:
+  vars:
-#   hosts:
+    talosctl_version: "v1.12.4"
-#     infra01.int.mk-labs.cloud:
+    talos_cluster_name: "fastpass"
 # matchbox_server:
 #   hosts:
@@ -82,51 +152,3 @@ backup_servers:
 terraform_server:
  hosts:
    infra01:
 # hub_cluster:
 #   hosts:
 #     ocp-hub:
 # internal_cluster:
 #   hosts:
 #     int-master01:
 #     int-master02:
 #     int-master03:
 # #    int-worker01:
 #     int-worker02:
 # talos_cluster:
 #   hosts:
 #     splash:
 # fastpass_control_plane:
 #   hosts:
 #     space-mountain:
 #     big-thunder-mountain:
 #     splash-mountain:
 # fastpass_workers:
 #   hosts:
 #     haunted-mansion: # backstage
 #     peter-pans-flight: # backstage
 # fastpass:
 #   children:
 #     fastpass_control_plane:
 #     fastpass_workers:
 # sql_servers:
 #   hosts:
 #     sql01:
 # docker_servers:
 #   hosts:
 #     docker01:
 # observer:
 #   hosts:
 #     prometheus:
 # target:
 #   hosts:
 #     prometheus:
@@ -0,0 +1,95 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/add_service_route.yml
 # DESCRIPTION: Ensures all services in the Traefik dynamic config directory
 #              are routed and have DNS CNAME records on monorail.
 #
 #              1. Syncs boilerplates/traefik/dynamic/ to lightning-lane
 #              2. Scans the directory for service configs
 #              3. Extracts all hostnames from Host() rules (supports multi-host)
 #              4. Creates CNAME records for each hostname -> lightning-lane
 #
 # PREREQUISITES:
 #   - Service dynamic config YAML committed to boilerplates/traefik/dynamic/
 #   - vault_technitium_api_key defined in group_vars/all/vault
 #
 # USAGE:
 #   ansible-playbook -i inventory.yml playbooks/add_service_route.yml
 # ------------------------------------------------------------------------------
 - name: Sync Traefik routes and ensure DNS records
  hosts: localhost
  connection: local
  gather_facts: false
  vars:
    base_domain: "local.mk-labs.cloud"
    dns_server: "monorail"
    traefik_host: "lightning-lane.local.mk-labs.cloud"
    traefik_user: "wed"
    traefik_dynamic_path: "/opt/docker/traefik/dynamic/"
    dynamic_config_dir: "{{ playbook_dir }}/../../boilerplates/traefik/dynamic"
    # Files in the dynamic directory that are NOT service routes
    exclude_configs:
      - default.yml
  tasks:
    # ── Step 1: Sync dynamic config to lightning-lane ──
    - name: Sync Traefik dynamic configuration to lightning-lane
      ansible.builtin.shell: >
        rsync -av --delete
        {{ dynamic_config_dir }}/
        {{ traefik_user }}@{{ traefik_host }}:{{ traefik_dynamic_path }}
      register: sync_result
      changed_when: "'sending incremental file list' in sync_result.stdout"
    # ── Step 2: Discover hostnames from Traefik router rules ──
    - name: Find all dynamic config files
      ansible.builtin.find:
        paths: "{{ dynamic_config_dir }}"
        patterns: "*.yml"
      register: config_files
    - name: Read config files
      ansible.builtin.slurp:
        src: "{{ item.path }}"
      register: slurped_configs
      loop: "{{ config_files.files }}"
      when: item.path | basename not in exclude_configs
    - name: Extract all hostnames from Host() rules
      ansible.builtin.set_fact:
        hostnames: >-
          {% set hosts = [] -%}
          {% for result in slurped_configs.results if result.content is defined -%}
            {% set content = result.content | b64decode -%}
            {% for match in content | regex_findall('Host\(`([^`]+)`\)') -%}
              {% for h in match.split(' || ') -%}
                {% set h = h | regex_replace('`', '') | trim -%}
                {% if h.endswith('.local.mk-labs.cloud') and h not in hosts -%}
                  {% set _ = hosts.append(h) -%}
                {% endif -%}
              {% endfor -%}
            {% endfor -%}
          {% endfor -%}
          {{ hosts | unique | list }}
    - name: Display hostnames to create
      ansible.builtin.debug:
        msg: "Hostnames found: {{ hostnames }}"
    # ── Step 3: Create DNS CNAME records ──
    - name: Create DNS CNAME record for each hostname
      effectivelywild.technitium_dns.technitium_dns_add_record:
        api_url: "http://{{ dns_server }}.{{ base_domain }}"
        api_token: "{{ vault_technitium_api_key }}"
        zone: "{{ base_domain }}"
        name: "{{ item }}"
        type: "CNAME"
        cname: "lightning-lane.{{ base_domain }}"
        ttl: 360
        validate_certs: false
      loop: "{{ hostnames }}"
      loop_control:
        label: "{{ item }}"
@@ -0,0 +1,40 @@
 ---
 - name: Build Paperclip Docker image from source
  hosts: illuminations
  become: true
  vars:
    paperclip_repo_dir: /opt/paperclip-repo
    paperclip_image_name: paperclip-local
    paperclip_image_tag: latest
  tasks:
    - name: Install git (if not present)
      apt:
        name: git
        state: present
    - name: Clone or update Paperclip repository
      git:
        repo: https://github.com/paperclipai/paperclip.git
        dest: "{{ paperclip_repo_dir }}"
        update: yes
      register: git_clone
    - name: Build Paperclip Docker image
      shell: |
        cd {{ paperclip_repo_dir }}
        docker build --target production -t {{ paperclip_image_name }}:{{ paperclip_image_tag }} .
      register: docker_build
      changed_when: "'Successfully tagged' in docker_build.stdout"
    - name: Show build result
      debug:
        msg: "Built image: {{ paperclip_image_name }}:{{ paperclip_image_tag }}"
    - name: Show image details
      shell: docker images {{ paperclip_image_name }}:{{ paperclip_image_tag }}
      register: image_info
    - name: Image info
      debug:
        msg: "{{ image_info.stdout_lines }}"
@@ -1,218 +0,0 @@
 ---
 - name: Configure DNS entries for cluster nodes
  hosts: internal_cluster
  gather_facts: false
  become: true
  tasks:
    - name: Create DNS entry for each cluster node
      ansible.builtin.include_tasks: tasks/create_dns_record.yml
      vars:
        host_ip_address: "{{ ip_address }}"
 - name: Configure DNS entries for OpenShift cluster services
  hosts: internal_cluster[0]
  gather_facts: false
  become: true
  tasks:
    - name: Create DNS entries for OpenShift cluster services
      ansible.builtin.include_tasks: tasks/create_dns_record.yml
      vars:
        dns_name: "{{ item.name }}"
        dns_address: "{{ item.address }}"
      loop:
        - name: "api.{{ cluster_name }}"
          address: "{{ api_address }}"
        - name: "api-int.{{ cluster_name }}"
          address: "{{ api_address }}"
        - name: "*.apps.{{ cluster_name }}"
          address: "{{ app_address }}"
      loop_control:
        label: "{{ item.name }}"
 - name: Configure DHCP entries for cluster nodes
  hosts: internal_cluster
  gather_facts: false
  become: true
  tasks:
    - name: Configure DHCP entry for node
      ansible.builtin.include_tasks: tasks/configure_dhcp_entry.yml
      vars:
        host_mac_address: "{{ install_mac_address }}"
        host_ip_address: "{{ ip_address }}"
  handlers:
    - name: Restart dnsmasq
      delegate_to: "{{ dhcp_server }}"
      become: true
      ansible.builtin.service:
        name: dnsmasq
        state: restarted
 - name: Create OpenShift installer files
  hosts: matchbox
  gather_facts: true
  vars:
    cluster_group: "internal_cluster"
    download_dir: "/var/cache/openshift-install"
  tasks:
    - name: Remove previous directory
      ansible.builtin.file:
        path: ~/homelab/terraform/{{ cluster_group }}
        state: absent
    - name: Create directory
      ansible.builtin.file:
        path: ~/homelab/terraform/{{ cluster_group }}
        state: directory
        mode: '0755'
    - name: OpenShift Installer install-config.yaml
      ansible.builtin.template:
        src: templates/install-config.yaml.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/install-config.yaml
        mode: '0644'
    - name: OpenShift Installer agent-config.yaml
      ansible.builtin.template:
        src: templates/agent-config.yaml.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/agent-config.yaml
        mode: '0644'
    - name: Create directory for OpenShift Installer files
      become: true
      ansible.builtin.file:
        path: "{{ download_dir }}/{{ hostvars[groups[cluster_group][0]].cluster_version }}"
        state: directory
        mode: '0755'
        owner: root
        group: root
    - name: Check if OpenShift installer file exists
      ansible.builtin.stat:
        path: "{{ download_dir }}/{{ hostvars[groups[cluster_group][0]].cluster_version }}/{{ hostvars[groups[cluster_group][0]].openshift_installer_file }}"
      register: openshift_installer_file
    - name: Download and install OpenShift installer package
      when: not openshift_installer_file.stat.exists
      block:
        - name: Download OpenShift Installer file
          become: true
          ansible.builtin.get_url:
            url: "{{ hostvars[groups[cluster_group][0]].openshift_installer_download_url }}"
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            mode: '0644'
        - name: Unarchive installer file
          become: true
          ansible.builtin.unarchive:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/openshift-install-linux.tar.gz
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            remote_src: true
        - name: Copy install binary
          become: true
          ansible.builtin.copy:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/openshift-install
            dest: /usr/local/bin
            remote_src: true
            owner: wed
            group: wed
            mode: '0755'
    - name: Check if OpenShift client file exists
      ansible.builtin.stat:
        path: "{{ download_dir }}/{{ hostvars[groups[cluster_group][0]].cluster_version }}/{{ hostvars[groups[cluster_group][0]].openshift_client_file }}"
      register: openshift_client_file
    - name: Download and install OpenShift client package
      when: not openshift_client_file.stat.exists
      block:
        - name: Download OpenShift Client file
          become: true
          ansible.builtin.get_url:
            url: "{{ hostvars[groups[cluster_group][0]].openshift_client_download_url }}"
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            mode: '0644'
        - name: Unarchive client file
          become: true
          ansible.builtin.unarchive:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/openshift-client-linux.tar.gz
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            remote_src: true
        - name: Copy client binary
          become: true
          ansible.builtin.copy:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/oc
            dest: /usr/local/bin
            remote_src: true
            owner: wed
            group: wed
            mode: '0755'
    - name: Check if OpenShift image exists
      ansible.builtin.stat:
        path: ~/homelab/terraform/{{ cluster_group }}/boot-artifacts/agent.x86_64-initrd.img
      register: openshift_image_exists
    - name: OpenShift image creation
      ansible.builtin.command: /usr/local/bin/openshift-install agent create pxe-files
      args:
        chdir: ~/homelab/terraform/{{ cluster_group }}
      when: not openshift_image_exists.stat.exists
      changed_when: true
    - name: Create a directory if it does not exist
      become: true
      ansible.builtin.file:
        path: "{{ hostvars[groups['matchbox_server'][0]].assets }}/{{ hostvars[groups[cluster_group][0]].cluster_name }}"
        state: directory
        owner: matchbox
        group: matchbox
        mode: '0755'
    - name: Copy file with owner and permissions
      become: true
      ansible.builtin.copy:
        src: "{{ ansible_env['HOME'] }}/homelab/terraform/{{ cluster_group }}/boot-artifacts/"
        dest: "{{ hostvars[groups['matchbox_server'][0]].assets }}/{{ hostvars[groups[cluster_group][0]].cluster_name }}/"
        remote_src: true
        owner: matchbox
        group: matchbox
        mode: '0644'
    - name: Verify directory permissions
      become: true
      ansible.builtin.file:
        path: "{{ hostvars[groups['matchbox_server'][0]].assets }}/{{ hostvars[groups[cluster_group][0]].cluster_name }}"
        state: directory
        owner: matchbox
        group: matchbox
        mode: '0755'
    - name: Create Terraform matchbox groups file from template
      ansible.builtin.template:
        src: templates/groups.tf.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/groups.tf
        mode: '0644'
    - name: Create Terraform matchbox profiles file from template
      ansible.builtin.template:
        src: templates/profiles.tf.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/profiles.tf
        mode: '0644'
    - name: Create Terraform matchbox provider file from template
      ansible.builtin.template:
        src: templates/provider.tf.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/provider.tf
        mode: '0644'
    - name: Configure Matchbox via Terraform
      community.general.terraform:
        project_path: ~/homelab/terraform/{{ cluster_group }}
        state: present
        force_init: true
@@ -1,221 +0,0 @@
 ---
 - name: Configure DNS entries for cluster nodes
  hosts: hub_cluster
  gather_facts: false
  become: true
  tasks:
    - name: Create DNS entry for each cluster node
      ansible.builtin.include_tasks: tasks/create_dns_record.yml
      vars:
        host_ip_address: "{{ ip_address }}"
 - name: Configure DNS entries for OpenShift cluster services
  hosts: hub_cluster[0]
  gather_facts: false
  become: true
  tasks:
    - name: Create DNS entries for OpenShift cluster services
      ansible.builtin.include_tasks: tasks/create_dns_record.yml
      vars:
        dns_name: "{{ item.name }}"
        dns_address: "{{ item.address }}"
      loop:
        - name: "api.{{ cluster_name }}"
          address: "{{ api_address }}"
        - name: "api-int.{{ cluster_name }}"
          address: "{{ api_address }}"
        - name: "*.apps.{{ cluster_name }}"
          address: "{{ app_address }}"
      loop_control:
        label: "{{ item.name }}"
 - name: Configure DHCP entries for cluster nodes
  hosts: hub_cluster
  gather_facts: false
  become: true
  tasks:
    - name: Configure DHCP entry for node
      ansible.builtin.include_tasks: tasks/configure_dhcp_entry.yml
      vars:
        host_mac_address: "{{ install_mac_address }}"
        host_ip_address: "{{ ip_address }}"
  handlers:
    - name: Restart dnsmasq
      delegate_to: "{{ dhcp_server }}"
      become: true
      ansible.builtin.service:
        name: dnsmasq
        state: restarted
 - name: Create OpenShift installer files
  hosts: matchbox
  gather_facts: true
  vars:
    cluster_group: "hub_cluster"
    download_dir: "/var/cache/openshift-install"
  tasks:
    - name: Remove previous directory
      ansible.builtin.file:
        path: ~/homelab/terraform/{{ cluster_group }}
        state: absent
    - name: Create directory
      ansible.builtin.file:
        path: ~/homelab/terraform/{{ cluster_group }}
        state: directory
        mode: '0755'
    - name: OpenShift Installer install-config.yaml
      ansible.builtin.template:
        src: templates/install-config.yaml.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/install-config.yaml
        mode: '0644'
    - name: OpenShift Installer agent-config.yaml
      ansible.builtin.template:
        src: templates/agent-config.yaml.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/agent-config.yaml
        mode: '0644'
    # - name: Quit
    #   ansible.builtin.meta: end_play
    - name: Create directory for OpenShift Installer files
      become: true
      ansible.builtin.file:
        path: "{{ download_dir }}/{{ hostvars[groups[cluster_group][0]].cluster_version }}"
        state: directory
        mode: '0755'
        owner: root
        group: root
    - name: Check if OpenShift installer file exists
      ansible.builtin.stat:
        path: "{{ download_dir }}/{{ hostvars[groups[cluster_group][0]].cluster_version }}/{{ hostvars[groups[cluster_group][0]].openshift_installer_file }}"
      register: openshift_installer_file
    - name: Download and install OpenShift installer package
      when: not openshift_installer_file.stat.exists
      block:
        - name: Download OpenShift Installer file
          become: true
          ansible.builtin.get_url:
            url: "{{ hostvars[groups[cluster_group][0]].openshift_installer_download_url }}"
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            mode: '0644'
        - name: Unarchive installer file
          become: true
          ansible.builtin.unarchive:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/openshift-install-linux.tar.gz
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            remote_src: true
        - name: Copy install binary
          become: true
          ansible.builtin.copy:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/openshift-install
            dest: /usr/local/bin
            remote_src: true
            owner: wed
            group: wed
            mode: '0755'
    - name: Check if OpenShift client file exists
      ansible.builtin.stat:
        path: "{{ download_dir }}/{{ hostvars[groups[cluster_group][0]].cluster_version }}/{{ hostvars[groups[cluster_group][0]].openshift_client_file }}"
      register: openshift_client_file
    - name: Download and install OpenShift client package
      when: not openshift_client_file.stat.exists
      block:
        - name: Download OpenShift Client file
          become: true
          ansible.builtin.get_url:
            url: "{{ hostvars[groups[cluster_group][0]].openshift_client_download_url }}"
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            mode: '0644'
        - name: Unarchive client file
          become: true
          ansible.builtin.unarchive:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/openshift-client-linux.tar.gz
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            remote_src: true
        - name: Copy client binary
          become: true
          ansible.builtin.copy:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/oc
            dest: /usr/local/bin
            remote_src: true
            owner: wed
            group: wed
            mode: '0755'
    - name: Check if OpenShift image exists
      ansible.builtin.stat:
        path: ~/homelab/terraform/{{ cluster_group }}/boot-artifacts/agent.x86_64-initrd.img
      register: openshift_image_exists
    - name: OpenShift image creation
      ansible.builtin.command: /usr/local/bin/openshift-install agent create pxe-files
      args:
        chdir: ~/homelab/terraform/{{ cluster_group }}
      when: not openshift_image_exists.stat.exists
      changed_when: true
    - name: Create a directory if it does not exist
      become: true
      ansible.builtin.file:
        path: "{{ hostvars[groups['matchbox_server'][0]].assets }}/{{ hostvars[groups[cluster_group][0]].cluster_name }}"
        state: directory
        owner: matchbox
        group: matchbox
        mode: '0755'
    - name: Copy file with owner and permissions
      become: true
      ansible.builtin.copy:
        src: "{{ ansible_env['HOME'] }}/homelab/terraform/{{ cluster_group }}/boot-artifacts/"
        dest: "{{ hostvars[groups['matchbox_server'][0]].assets }}/{{ hostvars[groups[cluster_group][0]].cluster_name }}/"
        remote_src: true
        owner: matchbox
        group: matchbox
        mode: '0644'
    - name: Verify directory permissions
      become: true
      ansible.builtin.file:
        path: "{{ hostvars[groups['matchbox_server'][0]].assets }}/{{ hostvars[groups[cluster_group][0]].cluster_name }}"
        state: directory
        owner: matchbox
        group: matchbox
        mode: '0755'
    - name: Create Terraform matchbox groups file from template
      ansible.builtin.template:
        src: templates/groups.tf.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/groups.tf
        mode: '0644'
    - name: Create Terraform matchbox profiles file from template
      ansible.builtin.template:
        src: templates/profiles.tf.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/profiles.tf
        mode: '0644'
    - name: Create Terraform matchbox provider file from template
      ansible.builtin.template:
        src: templates/provider.tf.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/provider.tf
        mode: '0644'
    - name: Configure Matchbox via Terraform
      community.general.terraform:
        project_path: ~/homelab/terraform/{{ cluster_group }}
        state: present
        force_init: true
@@ -1,218 +0,0 @@
 ---
 - name: Configure DNS entries for cluster nodes
  hosts: internal_cluster
  gather_facts: false
  become: true
  tasks:
    - name: Create DNS entry for each cluster node
      ansible.builtin.include_tasks: tasks/create_dns_record.yml
      vars:
        host_ip_address: "{{ ip_address }}"
 - name: Configure DNS entries for OpenShift cluster services
  hosts: internal_cluster[0]
  gather_facts: false
  become: true
  tasks:
    - name: Create DNS entries for OpenShift cluster services
      ansible.builtin.include_tasks: tasks/create_dns_record.yml
      vars:
        dns_name: "{{ item.name }}"
        dns_address: "{{ item.address }}"
      loop:
        - name: "api.{{ cluster_name }}"
          address: "{{ api_address }}"
        - name: "api-int.{{ cluster_name }}"
          address: "{{ api_address }}"
        - name: "*.apps.{{ cluster_name }}"
          address: "{{ app_address }}"
      loop_control:
        label: "{{ item.name }}"
 - name: Configure DHCP entries for cluster nodes
  hosts: internal_cluster
  gather_facts: false
  become: true
  tasks:
    - name: Configure DHCP entry for node
      ansible.builtin.include_tasks: tasks/configure_dhcp_entry.yml
      vars:
        host_mac_address: "{{ install_mac_address }}"
        host_ip_address: "{{ ip_address }}"
  handlers:
    - name: Restart dnsmasq
      delegate_to: "{{ dhcp_server }}"
      become: true
      ansible.builtin.service:
        name: dnsmasq
        state: restarted
 - name: Create OpenShift installer files
  hosts: matchbox
  gather_facts: true
  vars:
    cluster_group: "internal_cluster"
    download_dir: "/var/cache/openshift-install"
  tasks:
    - name: Remove previous directory
      ansible.builtin.file:
        path: ~/homelab/terraform/{{ cluster_group }}
        state: absent
    - name: Create directory
      ansible.builtin.file:
        path: ~/homelab/terraform/{{ cluster_group }}
        state: directory
        mode: '0755'
    - name: OpenShift Installer install-config.yaml
      ansible.builtin.template:
        src: templates/install-config.yaml.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/install-config.yaml
        mode: '0644'
    - name: OpenShift Installer agent-config.yaml
      ansible.builtin.template:
        src: templates/agent-config.yaml.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/agent-config.yaml
        mode: '0644'
    - name: Create directory for OpenShift Installer files
      become: true
      ansible.builtin.file:
        path: "{{ download_dir }}/{{ hostvars[groups[cluster_group][0]].cluster_version }}"
        state: directory
        mode: '0755'
        owner: root
        group: root
    - name: Check if OpenShift installer file exists
      ansible.builtin.stat:
        path: "{{ download_dir }}/{{ hostvars[groups[cluster_group][0]].cluster_version }}/{{ hostvars[groups[cluster_group][0]].openshift_installer_file }}"
      register: openshift_installer_file
    - name: Download and install OpenShift installer package
      when: not openshift_installer_file.stat.exists
      block:
        - name: Download OpenShift Installer file
          become: true
          ansible.builtin.get_url:
            url: "{{ hostvars[groups[cluster_group][0]].openshift_installer_download_url }}"
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            mode: '0644'
        - name: Unarchive installer file
          become: true
          ansible.builtin.unarchive:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/openshift-install-linux.tar.gz
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            remote_src: true
        - name: Copy install binary
          become: true
          ansible.builtin.copy:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/openshift-install
            dest: /usr/local/bin
            remote_src: true
            owner: wed
            group: wed
            mode: '0755'
    - name: Check if OpenShift client file exists
      ansible.builtin.stat:
        path: "{{ download_dir }}/{{ hostvars[groups[cluster_group][0]].cluster_version }}/{{ hostvars[groups[cluster_group][0]].openshift_client_file }}"
      register: openshift_client_file
    - name: Download and install OpenShift client package
      when: not openshift_client_file.stat.exists
      block:
        - name: Download OpenShift Client file
          become: true
          ansible.builtin.get_url:
            url: "{{ hostvars[groups[cluster_group][0]].openshift_client_download_url }}"
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            mode: '0644'
        - name: Unarchive client file
          become: true
          ansible.builtin.unarchive:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/openshift-client-linux.tar.gz
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            remote_src: true
        - name: Copy client binary
          become: true
          ansible.builtin.copy:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/oc
            dest: /usr/local/bin
            remote_src: true
            owner: wed
            group: wed
            mode: '0755'
    - name: Check if OpenShift image exists
      ansible.builtin.stat:
        path: ~/homelab/terraform/{{ cluster_group }}/boot-artifacts/agent.x86_64-initrd.img
      register: openshift_image_exists
    - name: OpenShift image creation
      ansible.builtin.command: /usr/local/bin/openshift-install agent create pxe-files
      args:
        chdir: ~/homelab/terraform/{{ cluster_group }}
      when: not openshift_image_exists.stat.exists
      changed_when: true
    - name: Create a directory if it does not exist
      become: true
      ansible.builtin.file:
        path: "{{ hostvars[groups['matchbox_server'][0]].assets }}/{{ hostvars[groups[cluster_group][0]].cluster_name }}"
        state: directory
        owner: matchbox
        group: matchbox
        mode: '0755'
    - name: Copy file with owner and permissions
      become: true
      ansible.builtin.copy:
        src: "{{ ansible_env['HOME'] }}/homelab/terraform/{{ cluster_group }}/boot-artifacts/"
        dest: "{{ hostvars[groups['matchbox_server'][0]].assets }}/{{ hostvars[groups[cluster_group][0]].cluster_name }}/"
        remote_src: true
        owner: matchbox
        group: matchbox
        mode: '0644'
    - name: Verify directory permissions
      become: true
      ansible.builtin.file:
        path: "{{ hostvars[groups['matchbox_server'][0]].assets }}/{{ hostvars[groups[cluster_group][0]].cluster_name }}"
        state: directory
        owner: matchbox
        group: matchbox
        mode: '0755'
    - name: Create Terraform matchbox groups file from template
      ansible.builtin.template:
        src: templates/groups.tf.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/groups.tf
        mode: '0644'
    - name: Create Terraform matchbox profiles file from template
      ansible.builtin.template:
        src: templates/profiles.tf.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/profiles.tf
        mode: '0644'
    - name: Create Terraform matchbox provider file from template
      ansible.builtin.template:
        src: templates/provider.tf.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/provider.tf
        mode: '0644'
    - name: Configure Matchbox via Terraform
      community.general.terraform:
        project_path: ~/homelab/terraform/{{ cluster_group }}
        state: present
        force_init: true
@@ -0,0 +1,20 @@
 ---
 # ansible/playbooks/configure_proxmox_oidc.yml
 #
 # Configures Proxmox OIDC authentication with Authentik.
 # Only targets one node since realm config is cluster-wide.
 #
 # Usage:
 #   cd ansible
 #   ansible-playbook -i inventory.yml playbooks/configure_proxmox_oidc.yml
 #
 # To also set up ACL entries for your user:
 #   ansible-playbook -i inventory.yml playbooks/configure_proxmox_oidc.yml \
 #     -e '{"proxmox_oidc_acl_entries": [{"path": "/", "user": "rblundon@authentik", "role": "Administrator"}]}'
 - name: Configure Proxmox Authentik OIDC
  hosts: main-street-usa
  become: true
  roles:
    - role: proxmox
      tags: [proxmox-oidc]
@@ -0,0 +1,6 @@
 ---
 - name: Apply day0 baseline
  hosts: "{{ target | default('all') }}"
  become: true
  roles:
    - day0-baseline
@@ -0,0 +1,32 @@
 ---
 # ============================================================================
 # day0_expand_root_lv.yml
 # ----------------------------------------------------------------------------
 # Reclaims unallocated PE on the root volume group, extending the root LV
 # to fill the VG and resizing the underlying filesystem (ext4 or xfs).
 #
 # Belongs to the day0 host-provisioning lifecycle. The Ubuntu Server
 # autoinstall template ships with the root LV at ~half the disk size by
 # default; this playbook is the canonical one-shot fix-up for that.
 #
 # Idempotent and safe to re-run. Hosts without LVM are no-op'd cleanly.
 #
 # Opt-out: set `expand_root_lv_skip: true` in host_vars/<host>.yml for
 # hosts where free PE should NOT be claimed by root (e.g. hosts with a
 # planned second LV in the same VG for application data).
 #
 # Usage:
 #   ansible-playbook playbooks/day0_expand_root_lv.yml
 #   ansible-playbook playbooks/day0_expand_root_lv.yml -e target=lincoln
 #   ansible-playbook playbooks/day0_expand_root_lv.yml -e target=honcho_server
 # ============================================================================
 - name: Expand root logical volume to fill VG
  hosts: "{{ target | default('all') }}"
  become: true
  gather_facts: true
  tasks:
    - name: Apply expand_root_lv role unless host opts out
      ansible.builtin.include_role:
        name: expand_root_lv
      when: not (expand_root_lv_skip | default(false) | bool)
@@ -0,0 +1,23 @@
 ---
 # ============================================================================
 # day0_linux_baseline.yml
 # ----------------------------------------------------------------------------
 # Applies the mk-labs Linux baseline (linux-baseline role) to one or more
 # hosts. Idempotent and safe to re-run.
 #
 # Usage:
 #   ansible-playbook playbooks/day0_linux_baseline.yml
 #   ansible-playbook playbooks/day0_linux_baseline.yml -e target=figment
 #   ansible-playbook playbooks/day0_linux_baseline.yml -e target=semaphore_server
 #
 # To trigger an opt-in full system upgrade:
 #   ansible-playbook playbooks/day0_linux_baseline.yml \
 #     -e target=figment -e 'baseline_features={"full_upgrade": true}'
 # ============================================================================
 - name: Apply mk-labs Linux baseline
  hosts: "{{ target | default('all') }}"
  become: true
  gather_facts: true
  roles:
    - linux-baseline
@@ -0,0 +1,30 @@
 ---
 # ============================================================================
 # day0_provision.yml
 # ----------------------------------------------------------------------------
 # Umbrella day0 playbook. Runs the full host-provisioning lifecycle in
 # the correct order against newly-built VMs, so the operator runs ONE
 # command per new host rather than chaining day0 steps manually.
 #
 # Order matters:
 #   1. linux-baseline      — timezone, NTP, packages, SSH hardening, jarvis user
 #   2. expand_root_lv      — reclaim PE left unallocated by the Ubuntu
 #                            autoinstall template default
 #
 # Idempotent: every step is safe to re-run. Suitable to apply periodically
 # from Semaphore as a baseline-drift check.
 #
 # Usage:
 #   ansible-playbook playbooks/day0_provision.yml -e target=lincoln
 #   ansible-playbook playbooks/day0_provision.yml -e target=honcho_server
 #
 # For finer control over a single phase, the constituent playbooks are:
 #   playbooks/day0_linux_baseline.yml
 #   playbooks/day0_expand_root_lv.yml
 # ============================================================================
 - name: Import day0 linux baseline
  ansible.builtin.import_playbook: day0_linux_baseline.yml
 - name: Import day0 expand root LV
  ansible.builtin.import_playbook: day0_expand_root_lv.yml
@@ -0,0 +1,400 @@
 ---
 # ansible/playbooks/day1_configure_netbox_catalog.yml
 #
 # Creates the tag taxonomy and custom fields in NetBox for service catalog
 # documentation. Run once (idempotent — uses name-based checks).
 #
 # Usage (from ansible/ directory):
 #   ansible-playbook playbooks/day1_configure_netbox_catalog.yml
 #
 # Requires:
 #   - vault_netbox_token in Ansible Vault
 #   - netbox reachable at http://fire-station.local.mk-labs.cloud
 - name: Configure NetBox service catalog taxonomy
  hosts: localhost
  gather_facts: false
  vars_files:
    - "{{ playbook_dir }}/../group_vars/all/vault"
  vars:
    netbox_url: "http://fire-station.local.mk-labs.cloud"
    netbox_token: "{{ vault_netbox_token }}"
    netbox_api: "{{ netbox_url }}/api"
    headers:
      Authorization: "Token {{ netbox_token }}"
      Content-Type: "application/json"
      Accept: "application/json"
    # ── Tag definitions ───────────────────────────────────────────────────
    tags:
      # Infrastructure type
      - name: k8s
        slug: k8s
        color: "2196f3"   # blue
        description: "Workload running in the fastpass Kubernetes cluster"
      - name: vm
        slug: vm
        color: "4caf50"   # green
        description: "Traditional VM or LXC on Proxmox"
      # Service tier
      - name: platform
        slug: platform
        color: "9c27b0"   # purple
        description: "Platform/infrastructure service (not user-facing)"
      - name: application
        slug: application
        color: "ff9800"   # orange
        description: "User-facing application workload"
      # Service categories
      - name: monitoring
        slug: monitoring
        color: "607d8b"   # grey
        description: "Metrics, logging, alerting"
      - name: auth
        slug: auth
        color: "607d8b"
        description: "Authentication and SSO"
      - name: gitops
        slug: gitops
        color: "607d8b"
        description: "GitOps and CI/CD"
      - name: dashboard
        slug: dashboard
        color: "607d8b"
        description: "Dashboard and portal services"
      - name: storage
        slug: storage
        color: "607d8b"
        description: "Storage and file services"
      - name: dns
        slug: dns
        color: "607d8b"
        description: "DNS and name resolution"
      - name: automation
        slug: automation
        color: "607d8b"
        description: "Automation and orchestration"
      - name: networking
        slug: networking
        color: "607d8b"
        description: "Network infrastructure services"
      - name: inference
        slug: inference
        color: "607d8b"
        description: "AI/ML inference workloads"
    # ── Custom field definitions ──────────────────────────────────────────
    # object_types use app_label.model format
    custom_fields:
      - name: hostnames
        label: Hostnames
        type: longtext
        object_types:
          - ipam.ipaddress
          - virtualization.virtualmachine
        description: "All DNS names that resolve to this service (comma-separated)"
        ui_visible: always
        ui_editable: yes
      - name: namespace
        label: Namespace
        type: text
        object_types:
          - ipam.ipaddress
          - virtualization.virtualmachine
        description: "Kubernetes namespace (blank for VM-based services)"
        ui_visible: always
        ui_editable: yes
      - name: managed_by
        label: Managed By
        type: select
        object_types:
          - ipam.ipaddress
          - virtualization.virtualmachine
        description: "How this service is managed"
        choices:
          - ArgoCD
          - Ansible
          - Manual
        ui_visible: always
        ui_editable: yes
      - name: thematic_name
        label: Thematic Name
        type: text
        object_types:
          - ipam.ipaddress
          - virtualization.virtualmachine
        description: "Disney/Magic Kingdom thematic hostname for this service"
        ui_visible: always
        ui_editable: yes
  tasks:
    # ── Tags ─────────────────────────────────────────────────────────────
    - name: Fetch existing tags
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/tags/?limit=200"
        method: GET
        headers: "{{ headers }}"
        return_content: true
        status_code: 200
      register: existing_tags_response
    - name: Set existing tag slugs fact
      ansible.builtin.set_fact:
        existing_tag_slugs: "{{ existing_tags_response.json.results | map(attribute='slug') | list }}"
    - name: Create tags
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/tags/"
        method: POST
        headers: "{{ headers }}"
        body_format: json
        body:
          name: "{{ item.name }}"
          slug: "{{ item.slug }}"
          color: "{{ item.color }}"
          description: "{{ item.description }}"
        status_code: 201
      loop: "{{ tags }}"
      when: item.slug not in existing_tag_slugs
      register: tag_creation
      changed_when: tag_creation.status == 201
    # ── Custom Fields ─────────────────────────────────────────────────────
    - name: Fetch existing custom fields
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-fields/?limit=200"
        method: GET
        headers: "{{ headers }}"
        return_content: true
        status_code: 200
      register: existing_cf_response
    - name: Set existing custom field names fact
      ansible.builtin.set_fact:
        existing_cf_names: "{{ existing_cf_response.json.results | map(attribute='name') | list }}"
    - name: Create custom field — hostnames
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-fields/"
        method: POST
        headers: "{{ headers }}"
        body_format: json
        body:
          name: hostnames
          label: Hostnames
          type: longtext
          object_types: "{{ custom_fields[0].object_types }}"
          description: "{{ custom_fields[0].description }}"
          ui_visible: always
          ui_editable: yes
        status_code: 201
      when: "'hostnames' not in existing_cf_names"
      register: cf_hostnames
      changed_when: cf_hostnames.status == 201
    - name: Create custom field — namespace
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-fields/"
        method: POST
        headers: "{{ headers }}"
        body_format: json
        body:
          name: namespace
          label: Namespace
          type: text
          object_types: "{{ custom_fields[1].object_types }}"
          description: "{{ custom_fields[1].description }}"
          ui_visible: always
          ui_editable: yes
        status_code: 201
      when: "'namespace' not in existing_cf_names"
      register: cf_namespace
      changed_when: cf_namespace.status == 201
    - name: Create choice set for managed_by field
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-field-choice-sets/"
        method: POST
        headers: "{{ headers }}"
        body_format: json
        body:
          name: managed-by-choices
          extra_choices:
            - - ArgoCD
              - ArgoCD
            - - Ansible
              - Ansible
            - - Manual
              - Manual
        status_code: [201, 400]
      register: choice_set
      changed_when: choice_set.status == 201
    - name: Fetch choice set ID
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-field-choice-sets/?name=managed-by-choices"
        method: GET
        headers: "{{ headers }}"
        return_content: true
        status_code: 200
      register: choice_set_response
    - name: Create custom field — managed_by
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-fields/"
        method: POST
        headers: "{{ headers }}"
        body_format: json
        body:
          name: managed_by
          label: Managed By
          type: select
          object_types: "{{ custom_fields[2].object_types }}"
          description: "{{ custom_fields[2].description }}"
          choice_set: "{{ choice_set_response.json.results[0].id }}"
          ui_visible: always
          ui_editable: yes
        status_code: 201
      when: "'managed_by' not in existing_cf_names"
      register: cf_managed_by
      changed_when: cf_managed_by.status == 201
    - name: Create custom field — thematic_name
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-fields/"
        method: POST
        headers: "{{ headers }}"
        body_format: json
        body:
          name: thematic_name
          label: Thematic Name
          type: text
          object_types: "{{ custom_fields[3].object_types }}"
          description: "{{ custom_fields[3].description }}"
          ui_visible: always
          ui_editable: yes
        status_code: 201
      when: "'thematic_name' not in existing_cf_names"
      register: cf_thematic_name
      changed_when: cf_thematic_name.status == 201
    # ── Provisioning Pipeline Fields ──────────────────────────────────────
    # These fields drive the NetBox → n8n → Terraform pipeline.
    # proxmox_datastore already exists — PATCH it to add utilidor choice.
    # data_disk_enabled and data_disk_size_gb are new POSTs.
    - name: Fetch proxmox_datastore field ID
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-fields/?name=proxmox_datastore"
        method: GET
        headers: "{{ headers }}"
        return_content: true
        status_code: 200
      register: proxmox_datastore_cf_response
    - name: Fetch proxmox-datastore-choices choice set ID
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-field-choice-sets/?name=proxmox-datastore-choices"
        method: GET
        headers: "{{ headers }}"
        return_content: true
        status_code: 200
      register: datastore_choice_set_response
    - name: Create proxmox-datastore-choices choice set if missing
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-field-choice-sets/"
        method: POST
        headers: "{{ headers }}"
        body_format: json
        body:
          name: proxmox-datastore-choices
          extra_choices:
            - - liberty-tree
              - liberty-tree
            - - utilidor
              - utilidor
        status_code: [201, 400]
      register: datastore_choice_set_create
      changed_when: datastore_choice_set_create.status == 201
      when: datastore_choice_set_response.json.count == 0
    - name: Re-fetch proxmox-datastore-choices choice set ID after possible creation
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-field-choice-sets/?name=proxmox-datastore-choices"
        method: GET
        headers: "{{ headers }}"
        return_content: true
        status_code: 200
      register: datastore_choice_set_response
    - name: Patch proxmox_datastore field to use choice set with utilidor
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-fields/{{ proxmox_datastore_cf_response.json.results[0].id }}/"
        method: PATCH
        headers: "{{ headers }}"
        body_format: json
        body:
          choice_set: "{{ datastore_choice_set_response.json.results[0].id }}"
        status_code: 200
      when: proxmox_datastore_cf_response.json.count > 0
      register: cf_datastore_patch
      changed_when: cf_datastore_patch.status == 200
    - name: Create custom field — data_disk_enabled
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-fields/"
        method: POST
        headers: "{{ headers }}"
        body_format: json
        body:
          name: data_disk_enabled
          label: Data Disk Enabled
          type: boolean
          object_types:
            - virtualization.virtualmachine
          description: "When true, Terraform provisions a second disk for application data storage."
          default: false
          ui_visible: always
          ui_editable: yes
        status_code: 201
      when: "'data_disk_enabled' not in existing_cf_names"
      register: cf_data_disk_enabled
      changed_when: cf_data_disk_enabled.status == 201
    - name: Create custom field — data_disk_size_gb
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-fields/"
        method: POST
        headers: "{{ headers }}"
        body_format: json
        body:
          name: data_disk_size_gb
          label: Data Disk Size (GB)
          type: integer
          object_types:
            - virtualization.virtualmachine
          description: "Size in GB for the optional second data disk. Only used when data_disk_enabled is true."
          ui_visible: always
          ui_editable: yes
        status_code: 201
      when: "'data_disk_size_gb' not in existing_cf_names"
      register: cf_data_disk_size
      changed_when: cf_data_disk_size.status == 201
    # ── Summary ───────────────────────────────────────────────────────────
    - name: Summary
      ansible.builtin.debug:
        msg:
          - "Tags created: {{ tag_creation.results | selectattr('status', 'equalto', 201) | list | length }}"
          - "Tags skipped (already exist): {{ tag_creation.results | selectattr('skipped', 'defined') | list | length }}"
          - "Custom fields configured: hostnames, namespace, managed_by, thematic_name, proxmox_datastore (patched), data_disk_enabled, data_disk_size_gb"
@@ -0,0 +1,18 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/deploy_authentik.yml
 # DESCRIPTION: Deploys Authentik identity provider on guest-relations.
 #              Installs Docker and configures Authentik with PostgreSQL and Redis.
 #
 # USAGE:
 #   ansible-playbook -i inventory.yml playbooks/deploy_authentik.yml
 # ------------------------------------------------------------------------------
 - name: Deploy Authentik identity provider
  hosts: authentik_server
  become: true
  roles:
    - common
    - docker-host
    - authentik
@@ -0,0 +1,23 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: ansible/playbooks/day1_deploy_gitea.yml
 # DESCRIPTION: Deploys Gitea + PostgreSQL on mad-tea-party (10.1.71.129)
 #              Role chain: common → docker-host → gitea
 #
 # USAGE:
 #   ansible-playbook -i ansible/inventory.yml ansible/playbooks/day1_deploy_gitea.yml
 #
 # SECRETS REQUIRED IN VAULT (group_vars/all/vault):
 #   vault_gitea_db_password
 # ------------------------------------------------------------------------------
 - name: Deploy Gitea on mad-tea-party
  hosts: gitea_servers
  become: true
  vars:
    gitea_db_password: "{{ vault_gitea_db_password }}"
  roles:
    - role: docker-host
    - role: gitea
@@ -0,0 +1,79 @@
 ---
 # =============================================================================
 # day1_deploy_hermes.yml
 # Deploy Hermes Agent (Nous Research) on carousel-of-progress (10.1.71.131)
 #
 # FIRST-RUN WORKFLOW:
 #   1. Run this playbook:
 #        ansible-playbook playbooks/day1_deploy_hermes.yml
 #
 #   2. SSH to the host and run the setup wizard as the hermes user:
 #        ssh wed@carousel-of-progress.local.mk-labs.cloud
 #        sudo -u hermes hermes setup
 #
 #   3. Once configured, start and verify the service:
 #        sudo systemctl start hermes
 #        sudo systemctl status hermes
 #        sudo journalctl -u hermes -f
 #
 # VARIABLES:
 #   hermes_skip_browser: true   — set to skip Playwright/Chromium install
 #                                 (saves ~300MB if browser automation not needed)
 # =============================================================================
 - name: Deploy Hermes Agent on carousel-of-progress
  hosts: carousel-of-progress
  gather_facts: true
  pre_tasks:
    - name: Verify target is carousel-of-progress
      ansible.builtin.assert:
        that:
          - inventory_hostname == "carousel-of-progress"
        fail_msg: >
          This playbook is scoped to carousel-of-progress only.
          Got: {{ inventory_hostname }}
    - name: Confirm OS is Ubuntu
      ansible.builtin.assert:
        that:
          - ansible_distribution == "Ubuntu"
        fail_msg: >
          This playbook requires Ubuntu. Found: {{ ansible_distribution }}.
          (If running Fedora, swap apt tasks for dnf and adjust Playwright deps.)
  roles:
    - role: hermes
      vars:
        hermes_skip_browser: false   # set true to skip Chromium install
  post_tasks:
    - name: Verify hermes binary is accessible system-wide
      ansible.builtin.command: hermes --version
      register: hermes_version_check
      changed_when: false
      failed_when: hermes_version_check.rc != 0
    - name: Print hermes version
      ansible.builtin.debug:
        msg: "{{ hermes_version_check.stdout }}"
    - name: Print post-install instructions
      ansible.builtin.debug:
        msg:
          - "============================================================"
          - "Hermes installed on carousel-of-progress (10.1.71.131)"
          - "============================================================"
          - "Next steps:"
          - "  1. SSH to the host:"
          - "       ssh wed@carousel-of-progress.local.mk-labs.cloud"
          - "  2. Run the setup wizard as the hermes user:"
          - "       sudo -u hermes hermes setup"
          - "  3. After config, start the service:"
          - "       sudo systemctl start hermes"
          - "  4. Verify:"
          - "       sudo systemctl status hermes"
          - "       sudo journalctl -u hermes -f"
          - "============================================================"
          - "Service is ENABLED but NOT STARTED — config required first."
          - "============================================================"
@@ -0,0 +1,7 @@
 ---
 # Run after day1_deploy_nginx_proxy.yml (the role publishes its vhost through it).
 - name: Deploy Hermes Agent (Docker)
  hosts: illuminations
  become: true
  roles:
    - hermes_docker
@@ -0,0 +1,18 @@
 ---
 # ============================================================================
 # day1_deploy_honcho.yml
 # ----------------------------------------------------------------------------
 # Deploys Honcho + pgvector PostgreSQL on the `lincoln` host. Assumes day0
 # host provisioning (linux-baseline + expand_root_lv) is already complete.
 #
 # Run via:
 #   ansible-playbook -i inventory.yml playbooks/day0_provision.yml -e target=lincoln
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_honcho.yml
 # ============================================================================
 - name: Deploy Honcho on lincoln
  hosts: honcho_server
  become: true
  gather_facts: true
  roles:
    - honcho
@@ -0,0 +1,24 @@
 ---
 # ============================================================================
 # day1_deploy_jmri.yml
 # ----------------------------------------------------------------------------
 # Deploys JMRI JmriFaceless headless server on main-street-station.
 # Applies linux-baseline first, then the jmri role.
 #
 # Usage:
 #   ansible-playbook playbooks/day1_deploy_jmri.yml
 #   ansible-playbook playbooks/day1_deploy_jmri.yml -e target=main-street-station
 #
 # Prerequisites:
 #   1. Host is in inventory under jmri_server group
 #   2. jmri_profile_id is set in host_vars/main-street-station.yml
 #   3. SSH access as 'wed' with sudo
 # ============================================================================
 - name: Deploy JMRI headless server
  hosts: "{{ target | default('jmri_server') }}"
  become: true
  gather_facts: true
  roles:
    - linux-baseline
    - jmri
@@ -0,0 +1,25 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day1_deploy_llm_inference.yml
 # DESCRIPTION: Day 1 playbook for astro-orbiter LLM inference stack.
 #              Deploys vLLM + Gemma 2 27B on RTX 3090 via OCuLink.
 #
 # Usage:
 #   cd ~/git/homelab/ansible
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference.yml
 #
 # Phases (added incrementally — safe to re-run):
 #   1. Foundation  — groups, directories, vault assertion
 #   2. Driver      — nvidia-driver-595-open (idempotent; already installed)
 #   3. vLLM        — Python venv + pip install vllm
 #   4. Model       — HF login, Gemma 2 27B snapshot_download
 #   5. Serve       — systemd vllm-serve.service, health check
 #   6. Integration — Hermes provider config on carousel
 # ------------------------------------------------------------------------------
 - name: Deploy LLM inference stack on astro-orbiter
  hosts: astro_orbiter
  gather_facts: true
  roles:
    - role: llm-inference
@@ -0,0 +1,35 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day1_deploy_llm_inference_multimodel.yml
 # DESCRIPTION: Day 1 playbook for the dual-model (aux + tool-calling) rollout
 #              on astro-orbiter. Builds on roles/llm-inference (CUDA/driver
 #              already done) — does not replace it.
 #
 # Usage:
 #   cd ~/git/homelab/ansible
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml
 #   # or scope to specific phases:
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml --tags discover
 #
 # EXECUTION CHANNEL (2026-08-12, War Machine): run via the Semaphore template
 # "llm_inference_multimodel_stage_models" (scoped to --tags models). Do NOT
 # run this via direct ansible-playbook or ad-hoc ssh/curl/systemctl — all
 # homelab inference changes go through Ansible roles executed by Semaphore for
 # audit/visibility. Phase 1 (models) is idempotent: it only downloads/stages a
 # GGUF when missing or size-mismatched, and only restarts the router when a new
 # GGUF is detected (normal re-runs that find the files correct touch nothing).
 #
 # Phases (see roles/llm-inference-multimodel/README.md for detail):
 #   0. discover  — read-only; confirm existing Gemma service management
 #   1. models    — idempotent GGUF downloads (Phi-4-14B, Mistral-Small-24B)
 #   2. systemd   — deploy both unit files, do NOT auto-start
 #   3. firewall  — scope ports 8000/8001, non-0.0.0.0 bind
 #   4. verify    — start both services, smoke test, VRAM check
 # ------------------------------------------------------------------------------
 - name: Deploy dual-model LLM inference stack on astro-orbiter
  hosts: astro_orbiter
  gather_facts: true
  roles:
    - role: llm-inference-multimodel
@@ -0,0 +1,106 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day1_deploy_llm_router_shadow.yml
 # DESCRIPTION: Deploy llama-server in router mode on a shadow port (8003).
 #
 # This playbook deploys and validates the llama.cpp router mode supervisor on
 # astro-orbiter (10.1.71.130) WITHOUT touching the production endpoint
 # (llama-server-qwen, port 8002). All 7 dependent Hermes profiles
 # (bruce-banner, groot, happy, heimdall, rocket-raccoon, war-machine, wong)
 # remain pointing at port 8002 throughout this run.
 #
 # Usage (from ~/git/homelab/ansible):
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_router_shadow.yml
 #
 # Tag-scoped runs (if you need to re-run one phase):
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_router_shadow.yml \
 #     --tags router_systemd,router_firewall,router_verify
 #
 # Execution path (Ryan-approved 2026-08-12, task t_0cca74a2):
 #   Direct ansible-playbook as documented exception — Semaphore template for
 #   this role does not exist yet. Create template after cutover is confirmed.
 #   This is the same exception pattern used in prior sessions on this box.
 #
 # Pre-requisites:
 #   1. llama-server binary at /opt/llama.cpp/build/bin/llama-server supports
 #      router mode (confirmed 2026-08-12: --models-dir flag present in --help).
 #   2. /opt/models/ contains ONLY Qwen3.6-35B-A3B-UD-Q4_K_S.gguf
 #      (confirmed 2026-08-12: directory is clean, Phi-4/Mistral already deleted).
 #   3. Port 8002 is in use by the production llama-server-qwen service —
 #      this playbook does NOT touch it.
 #
 # Validation gates this playbook runs (all hard gates EXCEPT Gate 4):
 #   Gate 1: /v1/models reports Qwen with n_ctx >= 64000 (64K Hermes floor)
 #   Gate 2: Tool-calling probe through router returns finish_reason=tool_calls
 #   Gate 2b: Hallucination stress test does NOT trigger spurious tool_calls
 #   Gate 3: nvidia-smi VRAM <= 23,000 MiB (--models-max 1 confirmed effective)
 #   Gate 4: Bundled SvelteKit UI check (nice-to-have, non-blocking)
 #
 # What happens after this playbook:
 #   War Machine posts validation gate results to Ryan.
 #   Ryan reviews and signs off on cutover (or requests changes).
 #   War Machine then runs day2_cutover_qwen_to_router.yml (not yet created)
 #   to promote the router to port 8002 and retire the bare llama-server-qwen.
 #
 # Reference: proposal at
 #   ~/friday/system/inbox/agents/war-machine/2026-08-12-qwen-router-mode-proposal.md
 # Task: t_0cca74a2
 # Author: War Machine (2026-08-12)
 # ------------------------------------------------------------------------------
 - name: Deploy llama-server router (shadow, port 8003) on astro-orbiter
  hosts: astro_orbiter
  gather_facts: true
  become: true
  vars:
    # Enable the router phase — this is the ONLY var that makes router.yml run.
    # Default in defaults/main.yml is false (no-op). Flip here for the shadow run.
    llm_router_enabled: true
    # Qwen model ID as it appears in /v1/models from the router.
    # llama-server router uses the GGUF filename (without .gguf) as the model id.
    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
  roles:
    - role: llm-inference-multimodel
  # No --tags needed here: router.yml is included dynamically from main.yml
  # whenever llm_router_enabled: true. The full role runs but the
  # discover/models/systemd/verify phases are gated on their own vars
  # (llm_qwen_service_enabled etc.) and are idempotent. The stale
  # models.yml (Phi-4/Mistral download tasks) uses variables no longer
  # defined — a follow-up cleanup task should update that file.
 - name: "POST-VALIDATION SAFETY NET — ensure production service is running"
  hosts: astro_orbiter
  gather_facts: false
  become: true
  tasks:
    # Always run this, regardless of whether the validation play succeeded.
    # If the router.yml play stopped llama-server-qwen for VRAM validation
    # and then a gate failed (play aborted), this play ensures it comes back up.
    - name: "Ensure llama-server-qwen (port 8002) is running after validation (always)"
      ansible.builtin.systemd:
        name: llama-server-qwen
        state: started
        enabled: true
      ignore_errors: true   # don't fail if the unit doesn't exist
    - name: "Verify production /health after safety-net restart"
      ansible.builtin.uri:
        url: "http://10.1.71.130:8002/health"
        status_code: 200
        timeout: 30
      register: llm_safety_net_health
      failed_when: false
      ignore_errors: true
    - name: "Report production status (safety-net check)"
      ansible.builtin.debug:
        msg: >-
          Safety-net: llama-server-qwen :8002 health check returned
          {{ llm_safety_net_health.status | default('UNREACHABLE') }}.
          {{ 'OK — production is up.' if (llm_safety_net_health.status | default(0) | int == 200)
             else 'WARNING — production may not be healthy. Check manually.' }}
@@ -0,0 +1,91 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day1_deploy_nextcloud.yml
 # DESCRIPTION: Deploys Nextcloud on the-grid
 #              Runs: docker-host → nextcloud
 #
 # USAGE:
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_nextcloud.yml
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_nextcloud.yml --limit the-grid
 #
 # SECRETS REQUIRED IN VAULT (group_vars/all/vault):
 #   vault_nextcloud_db_root_password
 #   vault_nextcloud_db_password
 #   vault_nextcloud_admin_user
 #   vault_nextcloud_admin_password
 # ------------------------------------------------------------------------------
 - name: Deploy Nextcloud on the-grid
  hosts: the-grid
  become: true
  vars:
    # NFS prerequisite
    nfs_packages:
      - nfs-common
    # Compose stack location
    nextcloud_base_dir: /opt/docker/nextcloud
    # Secrets from vault
    nextcloud_db_root_password: "{{ vault_nextcloud_db_root_password }}"
    nextcloud_db_password:      "{{ vault_nextcloud_db_password }}"
    nextcloud_admin_user:       "{{ vault_nextcloud_admin_user }}"
    nextcloud_admin_password:   "{{ vault_nextcloud_admin_password }}"
  pre_tasks:
    - name: Install NFS client
      ansible.builtin.apt:
        name: "{{ nfs_packages }}"
        state: present
        update_cache: true
  roles:
    - role: docker-host
  tasks:
    - name: Create Nextcloud directory
      ansible.builtin.file:
        path: "{{ nextcloud_base_dir }}"
        state: directory
        owner: "{{ ansible_user }}"
        group: docker
        mode: "0775"
    - name: Deploy Compose file
      ansible.builtin.copy:
        src: "{{ playbook_dir }}/../../boilerplates/nextcloud/compose.yml"
        dest: "{{ nextcloud_base_dir }}/compose.yml"
        owner: "{{ ansible_user }}"
        group: docker
        mode: "0644"
    - name: Deploy .env from vault
      ansible.builtin.copy:
        content: |
          # Managed by Ansible — do not edit manually
          MYSQL_ROOT_PASSWORD={{ nextcloud_db_root_password }}
          MYSQL_PASSWORD={{ nextcloud_db_password }}
          NEXTCLOUD_ADMIN_USER={{ nextcloud_admin_user }}
          NEXTCLOUD_ADMIN_PASSWORD={{ nextcloud_admin_password }}
        dest: "{{ nextcloud_base_dir }}/.env"
        owner: "{{ ansible_user }}"
        group: docker
        mode: "0600"
    - name: Start Nextcloud stack
      community.docker.docker_compose_v2:
        project_src: "{{ nextcloud_base_dir }}"
        state: present
    - name: Wait for Nextcloud to become ready
      ansible.builtin.uri:
        url: "http://the-grid.local.mk-labs.cloud/status.php"
        status_code: 200
        return_content: true
      register: nextcloud_status
      until: >
        nextcloud_status.status == 200 and
        (nextcloud_status.content | from_json).installed == true
      retries: 20
      delay: 15
@@ -0,0 +1,6 @@
 ---
 - name: Deploy shared nginx reverse proxy and certbot
  hosts: illuminations
  become: true
  roles:
    - nginx-proxy
@@ -0,0 +1,35 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: ansible/playbooks/day1_deploy_ollama.yml
 # DESCRIPTION: Deploys Ollama with ROCm GPU acceleration on astro-orbiter.
 #              Assumes day0_baseline.yml has already run (common role complete).
 #              PCIe passthrough for the RX 5700 must be configured in Proxmox
 #              and the GPU must be visible to the VM before running this playbook.
 #
 # Pre-flight check:
 #   ssh wed@astro-orbiter 'lspci | grep -i amd'
 #   Should show the RX 5700 before proceeding.
 #
 # Usage (from ansible/ directory):
 #   ansible-playbook playbooks/day1_deploy_ollama.yml
 # ------------------------------------------------------------------------------
 - name: Deploy Ollama with ROCm on astro-orbiter
  hosts: astro-orbiter
  become: true
  pre_tasks:
    - name: Verify AMD GPU is visible to the VM
      command: lspci
      register: lspci_output
      changed_when: false
    - name: Fail if no AMD GPU detected
      fail:
        msg: >
          No AMD GPU detected via lspci. Verify PCIe passthrough is configured
          in Proxmox and the RX 5700 is visible to the VM before proceeding.
      when: "'AMD' not in lspci_output.stdout and 'Radeon' not in lspci_output.stdout"
  roles:
    - ollama
@@ -0,0 +1,17 @@
 ---
 # ============================================================================
 # day1_deploy_semaphore.yml
 # ----------------------------------------------------------------------------
 # Deploys SemaphoreUI + PostgreSQL on the imagineering host (figment).
 # Run AFTER day0_linux_baseline.yml has been applied to the target.
 #
 # Usage:
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_semaphore.yml
 # ============================================================================
 - name: Deploy SemaphoreUI on imagineering
  hosts: semaphore_server
  become: true
  gather_facts: true
  roles:
    - semaphore
@@ -0,0 +1,18 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/deploy_traefik.yml
 # DESCRIPTION: Deploys Traefik reverse proxy on lightning-lane.
 #              Installs Docker and configures Traefik with Cloudflare DNS-01
 #              certificate resolution for *.local.mk-labs.cloud
 #
 # USAGE:
 #   ansible-playbook -i inventory.yml playbooks/deploy_traefik.yml
 # ------------------------------------------------------------------------------
 - name: Deploy Traefik reverse proxy
  hosts: load_balancers
  become: true
  roles:
    - docker-host
    - traefik
@@ -0,0 +1,18 @@
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day1_deploy_vllm.yml
 # Deploy vLLM to a target host via roles/deploy-vllm.
 #
 # Staging run (deploy + validate WITHOUT touching production traffic):
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_vllm.yml --limit astro-orbiter
 #
 # Cutover run (once staging is validated and Ryan/JARVIS approve flipping
 # traffic — starts and enables the systemd unit(s), runs Phase 5 verification):
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_vllm.yml \
 #     --limit astro-orbiter --extra-vars "vllm_service_state=started"
 # ------------------------------------------------------------------------------
 - name: Deploy vLLM inference serving stack
  hosts: astro-orbiter
  become: false
  gather_facts: true
  roles:
    - deploy-vllm
@@ -0,0 +1,7 @@
 ---
 - name: Apply common role
  hosts: "{{ target | default('all') }}"
  become: true
  roles:
    - common
    - docker-host
@@ -0,0 +1,259 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day2_add_coder_alias.yml
 # DESCRIPTION: Add Qwen2.5-Coder-14B-Instruct-Q4_K_M to the llama-server-router
 #              on astro-orbiter (10.1.71.130:8002).
 #
 # Context (t_55c164f5, 2026-08-13):
 #   Ryan requested a Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf be added to the
 #   astro-orbiter router with:
 #     alias = "Qwen2.5-Coder-14B-Instruct-4bit"
 #     n_gpu_layers = 99
 #     ctx_size = 16384
 #     flash_attn = true
 #   Deployed GitOps-style via this role; no hand-editing of the live preset.
 #
 # What this playbook does:
 #   1. Downloads Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf into /opt/models if
 #      not already present (idempotent: size-check guard, no re-pull on match).
 #   2. Redeploys the preset INI (adding the [Qwen2.5-Coder-14B-Instruct-Q4_K_M]
 #      section with alias = Qwen2.5-Coder-14B-Instruct-4bit).
 #   3. Restarts llama-server-router to pick up the new model entry.
 #   4. Verifies /v1/models returns all 4 models including the new Coder entry.
 #
 # VRAM context note (t_55c164f5):
 #   Qwen2.5-Coder-14B Q4_K_M: ~8.4GB weights + ~0.6GB KV @ 16K ctx ≈ 9.0GB
 #   Qwen3.6-35B-A3B: ~21.5GB
 #   Full co-residency is impossible on 24GB. LRU eviction handles this:
 #   when Coder is requested, Qwen3.6-35B is evicted (and vice versa).
 #   Model-switching incurs ~30-60s cold-load latency — expected and acceptable.
 #   Phi (~4.3GB) or Llama (~5.6GB) can co-reside with Coder (total ~14GB).
 #
 # Usage (from ~/git/homelab/ansible):
 #   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook -i inventory.yml \
 #     playbooks/day2_add_coder_alias.yml
 #
 # Semaphore note: Semaphore SSH key for jarvis user is not loaded in the
 # container (known pitfall, homelab-llm-serving skill). Run via CLI with
 # id_jarvis key; document as exception per Ryan's standing CLI fallback directive.
 #
 # Author: War Machine (2026-08-13, t_55c164f5)
 # ------------------------------------------------------------------------------
 - name: "Add Qwen2.5-Coder-14B-Instruct-4bit alias to astro-orbiter router"
  hosts: astro_orbiter
  gather_facts: false
  become: true
  vars:
    # Activate preset mode
    llm_router_preset_enabled: true
    llm_router_preset_path: /opt/llama-server-router-preset.ini
    # Production port (router is on 8002 since t_cd0d5388)
    llm_router_port: 8002
    # Per-model ctx-size settings (carried from t_ryan_per_model_ctx; Coder new)
    llm_router_llama_ctx_size: 8192
    llm_router_llama_flash_attn: "true"
    llm_router_phi_ctx_size: 32768
    llm_router_phi_flash_attn: "true"
    llm_router_coder_ctx_size: 16384
    llm_router_coder_flash_attn: "true"
    # All other vars inherit from host_vars + defaults/main.yml.
    # Explicitly set the ones needed by the unit/template tasks for clarity:
    llm_router_enabled: true
    llm_service_user: jarvis
    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
    llm_models_dir: /opt/models
    llm_bind_address: "10.1.71.130"
    llm_allowed_source_cidr: "10.1.70.0/24"
    llm_router_service_name: llama-server-router
    llm_router_bind_address: "10.1.71.130"
    llm_router_allowed_source_cidr: "10.1.70.0/24"
    llm_router_models_dir: /opt/models
    llm_router_models_max: 4            # from host_vars; bumped by t_33acbb2e
    llm_router_ctx_size: 65536          # Qwen3.6-35B default; per-model overrides above
    llm_router_parallel: 1
    llm_router_gpu_layers: 99
    llm_router_batch_size: 2048
    llm_router_ubatch_size: 512
    llm_router_cache_type_k: q4_0
    llm_router_cache_type_v: q4_0
    llm_router_flash_attn: "auto"
    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
    llm_router_vram_max_mib: 23000
    # Coder model staging entry (used below)
    coder_filename: "Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
    coder_url: "https://huggingface.co/bartowski/Qwen2.5-Coder-14B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
    coder_size_bytes: 8988111072
  handlers:
    - name: reload systemd
      ansible.builtin.systemd:
        daemon_reload: true
      become: true
      listen: "reload systemd"
    - name: restart router
      ansible.builtin.systemd:
        name: llama-server-router
        state: restarted
      become: true
      listen: "restart router"
  tasks:
    # ==========================================================================
    # PHASE 1: Download Coder GGUF if not present / size mismatch
    # ==========================================================================
    - name: "[coder] Stat existing GGUF"
      ansible.builtin.stat:
        path: "{{ llm_models_dir }}/{{ coder_filename }}"
        get_checksum: false
      register: coder_stat
    - name: "[coder] Download GGUF (skip if present and size matches)"
      ansible.builtin.get_url:
        url: "{{ coder_url }}"
        dest: "{{ llm_models_dir }}/{{ coder_filename }}"
        owner: "{{ llm_service_user }}"
        group: "{{ llm_service_user }}"
        mode: "0644"
        timeout: 3600
      when: >
        not coder_stat.stat.exists or
        coder_stat.stat.size != coder_size_bytes
      register: coder_download
      notify: restart router
    - name: "[coder] Confirm GGUF size post-download"
      ansible.builtin.stat:
        path: "{{ llm_models_dir }}/{{ coder_filename }}"
        get_checksum: false
      register: coder_stat_post
    - name: "[coder] FAIL if GGUF size mismatch after download"
      ansible.builtin.fail:
        msg: >-
          GGUF size mismatch: expected {{ coder_size_bytes }} bytes,
          got {{ coder_stat_post.stat.size }} bytes.
          Re-download may be needed.
      when: coder_stat_post.stat.size != coder_size_bytes
    # ==========================================================================
    # PHASE 2: Deploy updated preset INI (adds Coder section)
    # ==========================================================================
    - name: "[coder] Deploy preset INI to {{ llm_router_preset_path }}"
      ansible.builtin.template:
        src: "../roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2"
        dest: "{{ llm_router_preset_path }}"
        owner: root
        group: root
        mode: "0644"
      register: coder_preset_deployed
      notify: restart router
    # ==========================================================================
    # PHASE 3: Redeploy systemd unit (unchanged flags, but ensures unit is fresh)
    # ==========================================================================
    - name: "[coder] Deploy llama-server-router unit"
      ansible.builtin.template:
        src: "../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
        dest: /etc/systemd/system/llama-server-router.service
        owner: root
        group: root
        mode: "0644"
      register: coder_unit_deployed
      notify:
        - reload systemd
        - restart router
    - name: "[coder] Flush handlers (daemon-reload + router restart)"
      ansible.builtin.meta: flush_handlers
    # ==========================================================================
    # PHASE 4: Verify router is up and Coder model appears in /v1/models
    # ==========================================================================
    - name: "[coder] Wait for /health (router supervisor)"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
        status_code: 200
        timeout: 30
      retries: 12
      delay: 5
      register: coder_health
      until: coder_health.status == 200
    - name: "[coder] Query /v1/models"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
        status_code: 200
        return_content: true
        timeout: 30
      register: coder_models
    - name: "[coder] Extract model IDs and aliases"
      ansible.builtin.set_fact:
        coder_model_ids: "{{ coder_models.json.data | map(attribute='id') | list }}"
        coder_all_aliases: "{{ coder_models.json.data | map(attribute='aliases') | flatten | list }}"
    - name: "[coder] FAIL if Coder primary ID missing"
      ansible.builtin.fail:
        msg: >-
          'Qwen2.5-Coder-14B-Instruct-Q4_K_M' not in /v1/models.
          IDs: {{ coder_model_ids }}
      when: "'Qwen2.5-Coder-14B-Instruct-Q4_K_M' not in coder_model_ids"
    - name: "[coder] FAIL if Coder alias missing"
      ansible.builtin.fail:
        msg: >-
          'Qwen2.5-Coder-14B-Instruct-4bit' not found as ID or alias in /v1/models.
          IDs:     {{ coder_model_ids }}
          Aliases: {{ coder_all_aliases }}
      when:
        - "'Qwen2.5-Coder-14B-Instruct-4bit' not in coder_model_ids"
        - "'Qwen2.5-Coder-14B-Instruct-4bit' not in coder_all_aliases"
    - name: "[coder] FAIL if Qwen3.6-35B missing"
      ansible.builtin.fail:
        msg: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in /v1/models. IDs: {{ coder_model_ids }}"
      when: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in coder_model_ids"
    - name: "[coder] FAIL if Phi missing"
      ansible.builtin.fail:
        msg: "'Phi-3.5-mini-instruct-Q8_0' not in /v1/models. IDs: {{ coder_model_ids }}"
      when: "'Phi-3.5-mini-instruct-Q8_0' not in coder_model_ids"
    - name: "[coder] FAIL if Llama missing"
      ansible.builtin.fail:
        msg: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in /v1/models. IDs: {{ coder_model_ids }}"
      when: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in coder_model_ids"
    - name: "[coder] PASS — full /v1/models summary"
      ansible.builtin.debug:
        msg:
          - "========================================================================"
          - "QWEN2.5-CODER-14B ALIAS DEPLOYMENT — COMPLETE"
          - ""
          - "  Mode:     --models-preset ({{ llm_router_preset_path }})"
          - "  Service:  llama-server-router.service (:{{ llm_router_port }})"
          - ""
          - "  /v1/models IDs:     {{ coder_model_ids }}"
          - "  /v1/models aliases: {{ coder_all_aliases }}"
          - ""
          - "  VERIFY:"
          - "  Qwen3.6-35B-A3B-UD-Q4_K_S:          {{ 'PRESENT' if 'Qwen3.6-35B-A3B-UD-Q4_K_S' in coder_model_ids else 'MISSING' }}"
          - "  Phi-3.5-mini-instruct-Q8_0:          {{ 'PRESENT' if 'Phi-3.5-mini-instruct-Q8_0' in coder_model_ids else 'MISSING' }}"
          - "  Meta-Llama-3.1-8B-Instruct-Q4_K_M:  {{ 'PRESENT' if 'Meta-Llama-3.1-8B-Instruct-Q4_K_M' in coder_model_ids else 'MISSING' }}"
          - "  Qwen2.5-Coder-14B-Instruct-Q4_K_M:  {{ 'PRESENT' if 'Qwen2.5-Coder-14B-Instruct-Q4_K_M' in coder_model_ids else 'MISSING' }}"
          - "  Qwen2.5-Coder-14B-Instruct-4bit:    {{ 'PRESENT (ID)' if 'Qwen2.5-Coder-14B-Instruct-4bit' in coder_model_ids else ('PRESENT (alias)' if 'Qwen2.5-Coder-14B-Instruct-4bit' in coder_all_aliases else 'MISSING') }}"
          - ""
          - "  GGUF download: {{ 'NEW DOWNLOAD' if (coder_download is defined and coder_download.changed) else 'ALREADY PRESENT (skipped)' }}"
          - "========================================================================"
@@ -0,0 +1,313 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day2_add_nomic_embed.yml
 # DESCRIPTION: Add nomic-embed-text-v1.5-Q4_K_M to the llama-server-router
 #              on astro-orbiter (10.1.71.130:8002).
 #
 # Context (t_34b96e83, 2026-08-13, OpenViking Phase 1b):
 #   Ryan approved adding nomic-embed-text-v1.5-Q4_K_M as an embedding model
 #   after Phase 0 follow-up confirmed embedding models fold cleanly into the
 #   existing router preset via embedding=true. Model ID is "nomic-embed-text-v1.5".
 #   No alias needed — peter-parker and Honcho consumers will call it by the section
 #   name directly.
 #
 # What this playbook does:
 #   1. Downloads nomic-embed-text-v1.5-Q4_K_M.gguf into /opt/models if not
 #      already present (idempotent: exact size-check guard, no re-pull on match).
 #   2. Redeploys the preset INI (adding the [nomic-embed-text-v1.5] section with
 #      embedding=true, n-gpu-layers=99, ctx-size=8192, load-on-startup=true,
 #      sleep-idle-seconds=-1).
 #   3. Restarts llama-server-router to pick up the new model entry.
 #   4. Verifies /v1/models returns all 5 models including the new nomic entry.
 #   5. Runs a /v1/embeddings smoke test to confirm the model actually embeds.
 #
 # VRAM context note (t_34b96e83):
 #   nomic-embed-text-v1.5 Q4_K_M: ~84MB weights, embedding model (no KV cache).
 #   VRAM impact is negligible — always pinned via sleep-idle-seconds=-1.
 #   The 4 generative models remain unchanged (OOM analysis unchanged from t_55c164f5).
 #
 # Usage (from ~/git/homelab/ansible):
 #   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook -i inventory.yml \
 #     playbooks/day2_add_nomic_embed.yml
 #
 # Semaphore note: Semaphore SSH key for jarvis user is not loaded in the
 # container (known pitfall, homelab-llm-serving skill). Run via CLI with
 # id_jarvis key; document as exception per Ryan's standing CLI fallback directive.
 #
 # Author: War Machine (2026-08-13, t_34b96e83)
 # ------------------------------------------------------------------------------
 - name: "Add nomic-embed-text-v1.5 embedding model to astro-orbiter router"
  hosts: astro_orbiter
  gather_facts: false
  become: true
  vars:
    # Activate preset mode
    llm_router_preset_enabled: true
    llm_router_preset_path: /opt/llama-server-router-preset.ini
    # Production port (router is on 8002 since t_cd0d5388)
    llm_router_port: 8002
    # Per-model ctx-size settings (carried from t_55c164f5; nomic new)
    llm_router_llama_ctx_size: 8192
    llm_router_llama_flash_attn: "true"
    llm_router_phi_ctx_size: 32768
    llm_router_phi_flash_attn: "true"
    llm_router_coder_ctx_size: 16384
    llm_router_coder_flash_attn: "true"
    llm_router_nomic_ctx_size: 8192
    # NOTE (2026-08-14, t_openviking_embed_batch): per-model batch-size/
    # ubatch-size lines in the preset INI are NOT honored by llama-server's
    # router — only ctx-size is applied per-model; batch-size/ubatch-size for
    # every spawned child come from the router's own global CLI flags
    # (confirmed via `ps aux` on astro-orbiter: child process launched with
    # the router's --batch-size/--ubatch-size regardless of the INI values).
    # Kept below for documentation/future-proofing but the REAL fix is the
    # global llm_router_batch_size / llm_router_ubatch_size override further
    # down, which raises the physical batch for ALL models on this router
    # (Qwen3.6-35B, Phi, Llama, Coder, nomic).
    llm_router_nomic_batch_size: 4096
    llm_router_nomic_ubatch_size: 4096
    # All other vars inherit from host_vars + defaults/main.yml.
    llm_router_enabled: true
    llm_service_user: jarvis
    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
    llm_models_dir: /opt/models
    llm_bind_address: "10.1.71.130"
    llm_allowed_source_cidr: "10.1.70.0/24"
    llm_router_service_name: llama-server-router
    llm_router_bind_address: "10.1.71.130"
    llm_router_allowed_source_cidr: "10.1.70.0/24"
    llm_router_models_dir: /opt/models
    llm_router_models_max: 4            # from host_vars; bumped by t_33acbb2e
    llm_router_ctx_size: 65536          # Qwen3.6-35B default; per-model overrides above
    llm_router_parallel: 1
    llm_router_gpu_layers: 99
    # FIX (2026-08-14, t_openviking_embed_batch): raised from 512 to 4096.
    # This is a GLOBAL router flag applied to every spawned model process
    # (per-model INI batch-size/ubatch-size overrides are not honored by
    # llama-server's router — see note above nomic vars). 512 tokens was too
    # small for OpenViking's chunked-document embedding inputs (observed
    # 2000-3400 tokens/chunk), causing hard 500 errors ("input (N tokens) is
    # too large to process") that tripped OpenViking's circuit breaker into a
    # permanent fail/re-enqueue loop. 4096 comfortably covers observed chunk
    # sizes and stays under nomic's ctx-size=8192. VRAM impact of raising
    # ubatch-size is in compute-buffer scratch space, not KV cache; monitored
    # post-deploy against the 23000 MiB budget (host_vars/astro-orbiter).
    llm_router_batch_size: 4096
    llm_router_ubatch_size: 4096
    llm_router_cache_type_k: q4_0
    llm_router_cache_type_v: q4_0
    llm_router_flash_attn: "auto"
    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
    llm_router_vram_max_mib: 23000
    # nomic model staging
    nomic_filename: "nomic-embed-text-v1.5-Q4_K_M.gguf"
    nomic_url: "https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF/resolve/main/nomic-embed-text-v1.5.Q4_K_M.gguf"
    nomic_size_bytes: 84106624
  handlers:
    - name: reload systemd
      ansible.builtin.systemd:
        daemon_reload: true
      become: true
      listen: "reload systemd"
    - name: restart router
      ansible.builtin.systemd:
        name: llama-server-router
        state: restarted
      become: true
      listen: "restart router"
  tasks:
    # ==========================================================================
    # PHASE 1: Download nomic GGUF if not present / size mismatch
    # ==========================================================================
    - name: "[nomic] Stat existing GGUF"
      ansible.builtin.stat:
        path: "{{ llm_models_dir }}/{{ nomic_filename }}"
        get_checksum: false
      register: nomic_stat
    - name: "[nomic] Download GGUF (skip if present and size matches)"
      ansible.builtin.get_url:
        url: "{{ nomic_url }}"
        dest: "{{ llm_models_dir }}/{{ nomic_filename }}"
        owner: "{{ llm_service_user }}"
        group: "{{ llm_service_user }}"
        mode: "0644"
        timeout: 300
      when: >
        not nomic_stat.stat.exists or
        nomic_stat.stat.size != nomic_size_bytes
      register: nomic_download
      notify: restart router
    - name: "[nomic] Confirm GGUF size post-download"
      ansible.builtin.stat:
        path: "{{ llm_models_dir }}/{{ nomic_filename }}"
        get_checksum: false
      register: nomic_stat_post
    - name: "[nomic] FAIL if GGUF size mismatch after download"
      ansible.builtin.fail:
        msg: >-
          GGUF size mismatch: expected {{ nomic_size_bytes }} bytes,
          got {{ nomic_stat_post.stat.size }} bytes.
          Re-download may be needed.
      when: nomic_stat_post.stat.size != nomic_size_bytes
    # ==========================================================================
    # PHASE 2: Deploy updated preset INI (adds nomic-embed-text-v1.5 section)
    # ==========================================================================
    - name: "[nomic] Deploy preset INI to {{ llm_router_preset_path }}"
      ansible.builtin.template:
        src: "../roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2"
        dest: "{{ llm_router_preset_path }}"
        owner: root
        group: root
        mode: "0644"
      register: nomic_preset_deployed
      notify: restart router
    # ==========================================================================
    # PHASE 3: Redeploy systemd unit (ensures unit is fresh; no flag changes)
    # ==========================================================================
    - name: "[nomic] Deploy llama-server-router unit"
      ansible.builtin.template:
        src: "../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
        dest: /etc/systemd/system/llama-server-router.service
        owner: root
        group: root
        mode: "0644"
      register: nomic_unit_deployed
      notify:
        - reload systemd
        - restart router
    - name: "[nomic] Flush handlers (daemon-reload + router restart)"
      ansible.builtin.meta: flush_handlers
    # ==========================================================================
    # PHASE 4: Verify router is up and nomic model appears in /v1/models
    # ==========================================================================
    - name: "[nomic] Wait for /health (router supervisor)"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
        status_code: 200
        timeout: 30
      retries: 12
      delay: 5
      register: nomic_health
      until: nomic_health.status == 200
    - name: "[nomic] Query /v1/models"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
        status_code: 200
        return_content: true
        timeout: 30
      register: nomic_models
    - name: "[nomic] Extract model IDs and aliases"
      ansible.builtin.set_fact:
        nomic_model_ids: "{{ nomic_models.json.data | map(attribute='id') | list }}"
        nomic_all_aliases: "{{ nomic_models.json.data | map(attribute='aliases') | flatten | list }}"
    - name: "[nomic] FAIL if nomic primary ID missing"
      ansible.builtin.fail:
        msg: >-
          'nomic-embed-text-v1.5' not in /v1/models.
          IDs: {{ nomic_model_ids }}
      when: "'nomic-embed-text-v1.5' not in nomic_model_ids"
    - name: "[nomic] FAIL if Qwen3.6-35B missing"
      ansible.builtin.fail:
        msg: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in /v1/models. IDs: {{ nomic_model_ids }}"
      when: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in nomic_model_ids"
    - name: "[nomic] FAIL if Phi missing"
      ansible.builtin.fail:
        msg: "'Phi-3.5-mini-instruct-Q8_0' not in /v1/models. IDs: {{ nomic_model_ids }}"
      when: "'Phi-3.5-mini-instruct-Q8_0' not in nomic_model_ids"
    - name: "[nomic] FAIL if Llama missing"
      ansible.builtin.fail:
        msg: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in /v1/models. IDs: {{ nomic_model_ids }}"
      when: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in nomic_model_ids"
    - name: "[nomic] FAIL if Coder missing"
      ansible.builtin.fail:
        msg: "'Qwen2.5-Coder-14B-Instruct-Q4_K_M' not in /v1/models. IDs: {{ nomic_model_ids }}"
      when: "'Qwen2.5-Coder-14B-Instruct-Q4_K_M' not in nomic_model_ids"
    # ==========================================================================
    # PHASE 5: /v1/embeddings smoke test — confirm model actually embeds
    # ==========================================================================
    - name: "[nomic] POST /v1/embeddings smoke test"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/embeddings"
        method: POST
        body_format: json
        body:
          model: "nomic-embed-text-v1.5"
          input: "The dog ran across the park."
        status_code: 200
        return_content: true
        timeout: 120
      register: nomic_embed_result
    - name: "[nomic] Extract embedding vector length"
      ansible.builtin.set_fact:
        nomic_embed_dims: >-
          {{ (nomic_embed_result.json.data | first).embedding | length }}
      when:
        - nomic_embed_result.status == 200
        - nomic_embed_result.json.data is defined
        - nomic_embed_result.json.data | length > 0
    - name: "[nomic] FAIL if embedding vector is empty or missing"
      ansible.builtin.fail:
        msg: >-
          Embedding smoke test returned no vector.
          Response: {{ nomic_embed_result.json }}
      when: >-
        nomic_embed_result.status != 200 or
        nomic_embed_result.json.data is not defined or
        nomic_embed_result.json.data | length == 0 or
        (nomic_embed_result.json.data | first).embedding | length == 0
    - name: "[nomic] PASS — full summary"
      ansible.builtin.debug:
        msg:
          - "========================================================================"
          - "NOMIC-EMBED-TEXT-V1.5 DEPLOYMENT — COMPLETE"
          - ""
          - "  Mode:     --models-preset ({{ llm_router_preset_path }})"
          - "  Service:  llama-server-router.service (:{{ llm_router_port }})"
          - ""
          - "  /v1/models IDs:     {{ nomic_model_ids }}"
          - ""
          - "  VERIFY:"
          - "  Qwen3.6-35B-A3B-UD-Q4_K_S:          {{ 'PRESENT' if 'Qwen3.6-35B-A3B-UD-Q4_K_S' in nomic_model_ids else 'MISSING' }}"
          - "  Phi-3.5-mini-instruct-Q8_0:          {{ 'PRESENT' if 'Phi-3.5-mini-instruct-Q8_0' in nomic_model_ids else 'MISSING' }}"
          - "  Meta-Llama-3.1-8B-Instruct-Q4_K_M:  {{ 'PRESENT' if 'Meta-Llama-3.1-8B-Instruct-Q4_K_M' in nomic_model_ids else 'MISSING' }}"
          - "  Qwen2.5-Coder-14B-Instruct-Q4_K_M:  {{ 'PRESENT' if 'Qwen2.5-Coder-14B-Instruct-Q4_K_M' in nomic_model_ids else 'MISSING' }}"
          - "  nomic-embed-text-v1.5:               {{ 'PRESENT' if 'nomic-embed-text-v1.5' in nomic_model_ids else 'MISSING' }}"
          - ""
          - "  Embedding smoke test: PASS"
          - "  Vector dimensions: {{ nomic_embed_dims | default('unknown') }}"
          - ""
          - "  GGUF download: {{ 'NEW DOWNLOAD' if (nomic_download is defined and nomic_download.changed) else 'ALREADY PRESENT (skipped)' }}"
          - "========================================================================"
@@ -0,0 +1,203 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day2_add_phi_alias.yml
 # DESCRIPTION: Add Phi-3.5-mini-instruct-8bit alias to the llama-server-router
 #              by switching from --models-dir to --models-preset INI mode.
 #
 # Context (t_9adf0889, 2026-08-12):
 #   Ryan's Hermes config (auxiliary.title_generation.model) points to
 #   "Phi-3.5-mini-instruct-8bit" but the router only exposes the GGUF
 #   filename-derived ID "Phi-3.5-mini-instruct-Q8_0". They are the same file.
 #   This playbook adds the alias so both names work without changing Ryan's
 #   Hermes config.
 #
 # What this playbook does:
 #   1. Deploys the preset INI template (llama-server-router-preset.ini.j2)
 #      to /opt/llama-server-router-preset.ini on astro-orbiter.
 #   2. Redeploys the systemd unit (llama-server-router.service) with
 #      --models-preset instead of --models-dir.
 #   3. Restarts llama-server-router to pick up the new flag.
 #   4. Verifies that /v1/models returns:
 #      - Phi-3.5-mini-instruct-Q8_0 (original ID — must still work)
 #      - Phi-3.5-mini-instruct-8bit (new alias — Ryan's config target)
 #      - Qwen3.6-35B-A3B-UD-Q4_K_S (unchanged)
 #      - Meta-Llama-3.1-8B-Instruct-Q4_K_M (unchanged)
 #
 # Known upstream behavior:
 #   GH #22364: --models-preset creates an extra "default" entry in /v1/models.
 #   This is cosmetic only and does not affect model selection by name.
 #
 # Usage (from ~/git/homelab/ansible):
 #   ansible-playbook -i inventory.yml playbooks/day2_add_phi_alias.yml
 #
 # Semaphore note (t_9adf0889): Semaphore SSH key for jarvis user is not loaded
 # in the container (known pitfall, homelab-llm-serving skill). Run via CLI with
 # id_jarvis key; document as exception per Ryan's standing CLI fallback directive.
 #
 # Author: War Machine (2026-08-12, t_9adf0889)
 # ------------------------------------------------------------------------------
 - name: "Add Phi-3.5-mini-instruct-8bit alias — switch router to preset mode"
  hosts: astro_orbiter
  gather_facts: false
  become: true
  vars:
    # Activate preset mode and provide the on-disk INI path
    llm_router_preset_enabled: true
    llm_router_preset_path: /opt/llama-server-router-preset.ini
    # Production port (router is already on 8002 since t_cd0d5388)
    llm_router_port: 8002
    # All other vars inherit from host_vars + defaults/main.yml.
    # Explicitly set the ones needed by the unit template for clarity:
    llm_router_enabled: true
    llm_service_user: jarvis
    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
    llm_models_dir: /opt/models
    llm_bind_address: "10.1.71.130"
    llm_allowed_source_cidr: "10.1.70.0/24"
    llm_router_service_name: llama-server-router
    llm_router_bind_address: "10.1.71.130"
    llm_router_allowed_source_cidr: "10.1.70.0/24"
    llm_router_models_dir: /opt/models
    llm_router_models_max: 4            # from host_vars; bumped by t_33acbb2e
    llm_router_ctx_size: 65536
    llm_router_parallel: 1
    llm_router_gpu_layers: 99
    llm_router_batch_size: 2048
    llm_router_ubatch_size: 512
    llm_router_cache_type_k: q4_0
    llm_router_cache_type_v: q4_0
    llm_router_flash_attn: "auto"
    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
    llm_router_vram_max_mib: 23000
  handlers:
    - name: reload systemd
      ansible.builtin.systemd:
        daemon_reload: true
      become: true
      listen: "reload systemd"
    - name: restart router
      ansible.builtin.systemd:
        name: llama-server-router
        state: restarted
      become: true
      listen: "restart router"
  tasks:
    # ==========================================================================
    # PHASE 1: Deploy the preset INI
    # ==========================================================================
    - name: "[phi-alias] Deploy preset INI to {{ llm_router_preset_path }}"
      ansible.builtin.template:
        src: "../roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2"
        dest: "{{ llm_router_preset_path }}"
        owner: root
        group: root
        mode: "0644"
      register: phi_alias_preset_deployed
      notify:
        - restart router
    # ==========================================================================
    # PHASE 2: Redeploy systemd unit with --models-preset flag
    # ==========================================================================
    - name: "[phi-alias] Deploy llama-server-router unit (--models-preset mode)"
      ansible.builtin.template:
        src: "../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
        dest: /etc/systemd/system/llama-server-router.service
        owner: root
        group: root
        mode: "0644"
      register: phi_alias_unit_deployed
      notify:
        - reload systemd
        - restart router
    - name: "[phi-alias] Flush handlers (daemon-reload + router restart)"
      ansible.builtin.meta: flush_handlers
    # ==========================================================================
    # PHASE 3: Verify alias is present
    # ==========================================================================
    - name: "[phi-alias] Wait for /health (router supervisor, no model needed)"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
        status_code: 200
        timeout: 30
      retries: 12
      delay: 5
      register: phi_alias_health
      until: phi_alias_health.status == 200
    - name: "[phi-alias] Query /v1/models"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
        status_code: 200
        return_content: true
        timeout: 30
      register: phi_alias_models
    - name: "[phi-alias] Extract model IDs and aliases"
      ansible.builtin.set_fact:
        phi_alias_model_ids: "{{ phi_alias_models.json.data | map(attribute='id') | list }}"
        phi_alias_all_aliases: "{{ phi_alias_models.json.data | map(attribute='aliases') | flatten | list }}"
        phi_alias_model_sources: "{{ phi_alias_models.json.data | map(attribute='source') | list }}"
    - name: "[phi-alias] FAIL if Phi original ID missing"
      ansible.builtin.fail:
        msg: >-
          'Phi-3.5-mini-instruct-Q8_0' not in /v1/models.
          IDs: {{ phi_alias_model_ids }}
      when: "'Phi-3.5-mini-instruct-Q8_0' not in phi_alias_model_ids"
    - name: "[phi-alias] FAIL if Phi alias missing"
      ansible.builtin.fail:
        msg: >-
          'Phi-3.5-mini-instruct-8bit' not found as ID or alias in /v1/models.
          IDs:     {{ phi_alias_model_ids }}
          Aliases: {{ phi_alias_all_aliases }}
      when:
        - "'Phi-3.5-mini-instruct-8bit' not in phi_alias_model_ids"
        - "'Phi-3.5-mini-instruct-8bit' not in phi_alias_all_aliases"
    - name: "[phi-alias] FAIL if Qwen missing"
      ansible.builtin.fail:
        msg: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in /v1/models. IDs: {{ phi_alias_model_ids }}"
      when: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in phi_alias_model_ids"
    - name: "[phi-alias] FAIL if Llama missing"
      ansible.builtin.fail:
        msg: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in /v1/models. IDs: {{ phi_alias_model_ids }}"
      when: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in phi_alias_model_ids"
    - name: "[phi-alias] PASS — full /v1/models summary"
      ansible.builtin.debug:
        msg:
          - "========================================================================"
          - "PHI ALIAS DEPLOYMENT — COMPLETE"
          - ""
          - "  Mode:     --models-preset ({{ llm_router_preset_path }})"
          - "  Service:  llama-server-router.service (:{{ llm_router_port }})"
          - ""
          - "  /v1/models IDs:     {{ phi_alias_model_ids }}"
          - "  /v1/models aliases: {{ phi_alias_all_aliases }}"
          - "  Sources:            {{ phi_alias_model_sources }}"
          - ""
          - "  VERIFY:"
          - "  Phi-3.5-mini-instruct-Q8_0:  {{ 'PRESENT' if 'Phi-3.5-mini-instruct-Q8_0' in phi_alias_model_ids else 'MISSING' }}"
          - "  Phi-3.5-mini-instruct-8bit:  {{ 'PRESENT (ID)' if 'Phi-3.5-mini-instruct-8bit' in phi_alias_model_ids else ('PRESENT (alias)' if 'Phi-3.5-mini-instruct-8bit' in phi_alias_all_aliases else 'MISSING') }}"
          - "  Qwen3.6-35B-A3B-UD-Q4_K_S:  {{ 'PRESENT' if 'Qwen3.6-35B-A3B-UD-Q4_K_S' in phi_alias_model_ids else 'MISSING' }}"
          - "  Meta-Llama-3.1-8B-Instruct-Q4_K_M: {{ 'PRESENT' if 'Meta-Llama-3.1-8B-Instruct-Q4_K_M' in phi_alias_model_ids else 'MISSING' }}"
          - ""
          - "  GH #22364: if 'default' appears in IDs above, that is expected"
          - "  in --models-preset mode. Cosmetic only."
          - "========================================================================"
@@ -0,0 +1,161 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day2_bump_router_models_max.yml
 # DESCRIPTION: Bump --models-max on the production llama-server-router unit.
 #
 # Context: t_33acbb2e (2026-08-12) — Ryan requested --models-max raised from 1
 # to 4 so the router can keep multiple GGUFs resident on-demand (LRU eviction
 # when the cap is reached). The actual var change lives in:
 #   host_vars/astro-orbiter/vars.yml  (llm_router_models_max: 4)
 #
 # This playbook:
 #   1. Re-renders llama-server-router.service.j2 with the updated var value.
 #   2. Reloads systemd (daemon-reload handler) if the unit changed.
 #   3. Restarts llama-server-router so the new --models-max takes effect on the
 #      live process. Router holds no resident model (all-unloaded) so restart
 #      is sub-second and non-disruptive.
 #   4. Verifies /health returns 200 and /v1/models still lists all three GGUFs.
 #
 # VRAM NOTE: --models-max 4 allows up to all 3 current GGUFs to co-reside on
 # a 24GB card simultaneously. Worst-case combined footprint is ~31GB which
 # EXCEEDS 24GB — OOM is possible if all 3 are loaded concurrently. In normal
 # single-user homelab operation this is very unlikely. Full VRAM breakdown
 # documented in host_vars/astro-orbiter/vars.yml. Ryan approved (t_33acbb2e).
 #
 # Execution channel: Semaphore template "llm_router_update_unit" (project mk-labs).
 # Do NOT run via direct ansible-playbook or ad-hoc ssh/systemctl.
 #
 # Author: War Machine (2026-08-12, t_33acbb2e)
 # ------------------------------------------------------------------------------
 - name: "Bump llama-server-router --models-max to 4 on astro-orbiter"
  hosts: astro_orbiter
  gather_facts: true
  become: true
  vars:
    # Production vars — router is live on :8002 (post-cutover t_cd0d5388)
    llm_router_port: 8002
    llm_router_bind_address: "10.1.71.130"
    # llm_router_models_max is 4 via host_vars/astro-orbiter/vars.yml.
    # Remaining role vars come from host_vars + defaults/main.yml via the
    # inventory — we only explicitly set vars this playbook needs for its
    # own tasks (health/models check URIs).
    # Needed by the template task (mirrors defaults set in role defaults/main.yml)
    llm_service_user: jarvis
    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
    llm_models_dir: /opt/models
    llm_router_service_name: llama-server-router
    llm_router_models_dir: /opt/models
    llm_router_gpu_layers: 99
    llm_router_ctx_size: 65536
    llm_router_flash_attn: "auto"
    llm_router_cache_type_k: q4_0
    llm_router_cache_type_v: q4_0
    llm_router_batch_size: 2048
    llm_router_ubatch_size: 512
    llm_router_parallel: 1
  tasks:
    # -------------------------------------------------------------------------
    # Phase 1: Re-render the router unit file
    # Template src path is relative to the role's templates/ dir; we reference
    # it with a relative path that Ansible resolves from the role directory.
    # -------------------------------------------------------------------------
    - name: "Deploy updated llama-server-router unit (--models-max {{ llm_router_models_max }})"
      ansible.builtin.template:
        src: "{{ playbook_dir }}/../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
        dest: "/etc/systemd/system/{{ llm_router_service_name }}.service"
        owner: root
        group: root
        mode: "0644"
      register: llm_router_unit_updated
      notify:
        - reload systemd
      tags: [always]
    - name: "Flush handlers — ensure daemon-reload lands before restart"
      ansible.builtin.meta: flush_handlers
      tags: [always]
    # -------------------------------------------------------------------------
    # Phase 2: Restart the router so the new --models-max takes effect.
    # Always restart (even if unit unchanged) to ensure live process matches.
    # -------------------------------------------------------------------------
    - name: "Restart llama-server-router so --models-max {{ llm_router_models_max }} takes effect"
      ansible.builtin.systemd:
        name: "{{ llm_router_service_name }}"
        state: restarted
        enabled: true
      tags: [always]
    # -------------------------------------------------------------------------
    # Phase 3: Verify /health returns 200
    # -------------------------------------------------------------------------
    - name: "Wait for /health to return 200 after restart"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
        status_code: 200
        timeout: 30
      register: bump_health_check
      retries: 10
      delay: 3
      until: bump_health_check.status == 200
      tags: [always]
    # -------------------------------------------------------------------------
    # Phase 4: Verify /v1/models lists all three GGUFs
    # -------------------------------------------------------------------------
    - name: "Check /v1/models — all three GGUFs should appear"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
        status_code: 200
        timeout: 30
        return_content: true
      register: bump_models_check
      tags: [always]
    - name: "Display /v1/models summary"
      ansible.builtin.debug:
        msg:
          - "======================================================================"
          - "--models-max BUMP VERIFICATION (t_33acbb2e)"
          - ""
          - "  /health:               HTTP {{ bump_health_check.status }}"
          - "  /v1/models HTTP:       {{ bump_models_check.status }}"
          - "  Models listed:         {{ bump_models_check.json.data | map(attribute='id') | list | join(', ') }}"
          - ""
          - "  --models-max now:      {{ llm_router_models_max }}"
          - "  --parallel (unchanged): {{ llm_router_parallel }}"
          - ""
          - "  VRAM WARNING: worst-case 3-model co-residency ~31GB > 24GB RTX 3090."
          - "  OOM risk if all 3 load concurrently. LRU eviction mitigates in practice."
          - "  Full breakdown: host_vars/astro-orbiter/vars.yml"
          - "======================================================================"
      when: bump_models_check is defined
      tags: [always]
    - name: "GATE: confirm all 3 expected GGUFs appear in /v1/models"
      ansible.builtin.assert:
        that:
          - "'Qwen3.6-35B-A3B-UD-Q4_K_S' in (bump_models_check.json.data | map(attribute='id') | list)"
          - "'Phi-3.5-mini-instruct-Q8_0' in (bump_models_check.json.data | map(attribute='id') | list)"
          - "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' in (bump_models_check.json.data | map(attribute='id') | list)"
        fail_msg: >-
          /v1/models did not return all 3 expected GGUFs after --models-max bump.
          Check router logs: journalctl -u llama-server-router -n 50
        success_msg: "GATE PASSED: all 3 GGUFs listed in /v1/models."
      when: bump_models_check is defined
      tags: [always]
  handlers:
    - name: reload systemd
      ansible.builtin.systemd:
        daemon_reload: true
      listen: "reload systemd"
@@ -0,0 +1,59 @@
 ---
 # ------------------------------------------------------------------------------
 # Playbook: day2_cpu_offload_aux_models.yml
 # Purpose:  CPU-offload Qwen2.5-Coder-14B and Meta-Llama-3.1-8B on
 #           astro-orbiter's production router (port 8002).
 #
 # What this playbook does:
 #   1. Re-renders llama-server-router-preset.ini (Coder + Llama sections now
 #      use per-model n-gpu-layers vars = 0 -> full CPU inference).
 #   2. Re-renders the router unit (--models-max now 4 via host_vars, global
 #      --n-gpu-layers removed per t_72646029 unit template fix) and restarts
 #      llama-server-router so both changes take effect.
 #   3. Verifies per the role's router_preset phase.
 #
 # Context (2026-08-17):
 #   - RAM/model-swap audit, TIER 1 (Coder-14B CPU offload) + TIER 2
 #     (Llama-3.1-8B CPU offload) — Ryan approved 1 & 2 on 2026-08-17.
 #     See inbox/ryan/2026-08-17-llm-system-ram-model-swap.md.
 #   - Unit template fix (t_72646029): global --n-gpu-layers removed from
 #     ExecStart in preset mode. Each INI section now sets n-gpu-layers
 #     explicitly (Qwen3.8=99, Phi=99, nomic=99, Coder=0, Llama=0).
 #   - Concurrent residency after change: Qwen3.8-27B (20,302 MiB @ 128K ctx)
 #     + nomic-embed (558 MiB, pinned) + Coder (CPU, ~1,390 MiB CUDA ctx) +
 #     Llama (CPU, ~1,706 MiB CUDA ctx) = ~24,004 MiB. NOTE: llama.cpp 6ea215d
 #     allocates CUDA-context VRAM even at n-gpu-layers=0, so CPU models are not
 #     0-VRAM; total sits at the 24,576 MiB physical limit (headroom ~572 MiB).
 #     Qwen3.8 is never evicted for a CPU aux model; Phi-3.5-mini (GPU, 8.3GB)
 #     still evicts as before.
 #   - CPU speed (8-core Ryzen 7 5800XT): ~5-10 tok/s (14B), ~10-20 tok/s (8B).
 #   - Semaphore SSH gap for astro-orbiter still applies (t_730f9584 /
 #     t_33acbb2e); running direct CLI Ansible per standing exception.
 #
 # Run:
 #   cd /home/hermes/git/homelab/ansible
 #   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook \
 #     -i inventory.yml \
 #     playbooks/day2_cpu_offload_aux_models.yml
 #
 # Rollback:
 #   git checkout -- \
 #     roles/llm-inference-multimodel/templates/llama-server-router.service.j2 \
 #     roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2 \
 #     roles/llm-inference-multimodel/defaults/main.yml \
 #     host_vars/astro-orbiter/vars.yml
 #   (restores n-gpu-layers=99 global flag, models-max=2, all GPU)
 #   then re-run this playbook to redeploy rollback state.
 #   Note: playbooks/day2_cpu_offload_aux_models.yml is untracked — left on disk.
 # ------------------------------------------------------------------------------
 - name: CPU-offload Coder-14B and Llama-3.1-8B on astro-orbiter
  hosts: astro-orbiter
  become: true
  vars:
    llm_router_preset_enabled: true
    llm_router_enabled: true
    llm_router_port: 8002
  roles:
    - role: llm-inference-multimodel
      tags: [always]
@@ -0,0 +1,511 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day2_cutover_qwen_to_router.yml
 # DESCRIPTION: Promote llama-server-router to production on port 8002.
 #
 # Context: Router-mode shadow deployment (t_0cca74a2) validated 2026-08-12:
 #   all 4 hard gates PASSED (n_ctx 65536, tool-calling PASS, hallucination-stress
 #   PASS, VRAM 20410 MiB / 1 process). Ryan approved cutover.
 #
 # This playbook makes the router the permanent production endpoint:
 #
 #   1. Stop + disable llama-server-qwen (:8002). Unit file is PRESERVED on disk
 #      as the rollback target (same pattern as prior role history).
 #   2. Redeploy llama-server-router unit file with --port 8002 (production port).
 #      PORT DECISION: we rebind the router to :8002 rather than updating 8
 #      dependent Hermes profiles' base_url. One unit file change beats 8
 #      config.yaml updates — atomic, GitOps-clean, zero profile drift.
 #   3. Enable + start llama-server-router on :8002.
 #   4. Re-run validation gates 1-3 against the NOW-production endpoint.
 #      (Same logic as Phase R / router_verify in tasks/router.yml — hard gates.)
 #   5. Run Gate 4: verify bundled SvelteKit UI is reachable.
 #
 # Usage (from ~/git/homelab/ansible):
 #   ansible-playbook -i inventory.yml playbooks/day2_cutover_qwen_to_router.yml
 #
 # Rollback (if gates fail or any time after):
 #   ansible-playbook -i inventory.yml playbooks/day2_cutover_qwen_to_router.yml \
 #     --tags cutover_rollback
 #
 # Author: War Machine (2026-08-12, t_cd0d5388)
 # Approved by: Ryan (cutover authorization, 2026-08-12)
 # ------------------------------------------------------------------------------
 - name: "CUTOVER — Promote llama-server-router to production (:8002) on astro-orbiter"
  hosts: astro_orbiter
  gather_facts: true
  become: true
  vars:
    # ----------------------------------------------------------------
    # PORT DECISION:
    # We rebind the router to :8002 (production port) rather than
    # updating 8 dependent Hermes profiles' base_url to :8003.
    # Rationale: one unit file change is atomic and GitOps-clean.
    # Updating 8 config.yaml files risks drift and requires per-profile
    # activation tests. The template renders llm_router_port as the
    # --port argument; we just override it here to 8002.
    # ----------------------------------------------------------------
    # Router port override: take over production port
    llm_router_port: 8002
    # All other role defaults needed by the template (mirrors defaults/main.yml)
    llm_service_user: jarvis
    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
    llm_models_dir: /opt/models
    llm_bind_address: "10.1.71.130"
    llm_allowed_source_cidr: "10.1.70.0/24"
    llm_router_enabled: true
    llm_router_service_name: llama-server-router
    llm_router_models_dir: /opt/models
    llm_router_models_max: 1               # CRITICAL: RTX 3090 24GB, single model only
    llm_router_ctx_size: 65536
    llm_router_parallel: 1
    llm_router_gpu_layers: 99
    llm_router_batch_size: 2048
    llm_router_ubatch_size: 512
    llm_router_cache_type_k: q4_0
    llm_router_cache_type_v: q4_0
    llm_router_flash_attn: "auto"
    llm_router_bind_address: "10.1.71.130"
    llm_router_allowed_source_cidr: "10.1.70.0/24"
    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
    llm_router_vram_max_mib: 23000
    llm_qwen_service_name: llama-server-qwen
    llm_qwen_port: 8002
  tasks:
    # =======================================================================
    # PHASE 1 — Stop and disable llama-server-qwen (bare single-model)
    # Preserve unit file on disk — rollback target per existing role pattern.
    # =======================================================================
    - name: "[cutover] PHASE 1: Confirm llama-server-qwen current state"
      ansible.builtin.systemd:
        name: llama-server-qwen
      register: cutover_qwen_status
      tags: [cutover_stop_qwen, cutover]
    - name: "[cutover] PHASE 1: Report current llama-server-qwen status"
      ansible.builtin.debug:
        msg: >-
          llama-server-qwen: ActiveState={{ cutover_qwen_status.status.ActiveState | default('unknown') }},
          UnitFileState={{ cutover_qwen_status.status.UnitFileState | default('unknown') }}.
          Will stop + disable. Unit file preserved at /etc/systemd/system/llama-server-qwen.service as rollback target.
      tags: [cutover_stop_qwen, cutover]
    - name: "[cutover] PHASE 1: Stop llama-server-qwen (:8002, bare single-model)"
      ansible.builtin.systemd:
        name: llama-server-qwen
        state: stopped
      register: cutover_qwen_stopped
      tags: [cutover_stop_qwen, cutover]
    - name: "[cutover] PHASE 1: Disable llama-server-qwen (prevent auto-start on reboot)"
      ansible.builtin.systemd:
        name: llama-server-qwen
        enabled: false
      tags: [cutover_stop_qwen, cutover]
    - name: "[cutover] PHASE 1: Wait 5s for VRAM to be released"
      ansible.builtin.pause:
        seconds: 5
      when: cutover_qwen_stopped.changed | default(false)
      tags: [cutover_stop_qwen, cutover]
    - name: "[cutover] PHASE 1: Verify port 8002 is now free"
      ansible.builtin.command:
        cmd: ss -ltnp
      register: cutover_port_check
      changed_when: false
      tags: [cutover_stop_qwen, cutover]
    - name: "[cutover] PHASE 1: Fail if port 8002 is still bound"
      ansible.builtin.fail:
        msg: >-
          Port 8002 is still bound after stopping llama-server-qwen.
          Check 'ss -ltnp | grep :8002' and resolve before the router can bind.
      when:
        - "':8002 ' in (cutover_port_check.stdout | default('')) or ':8002:' in (cutover_port_check.stdout | default(''))"
      tags: [cutover_stop_qwen, cutover]
    - name: "[cutover] PHASE 1: Report VRAM state (should be empty)"
      ansible.builtin.command:
        cmd: nvidia-smi --query-compute-apps=pid,name,used_memory --format=csv,noheader
      register: cutover_vram_free_check
      changed_when: false
      tags: [cutover_stop_qwen, cutover]
    - name: "[cutover] PHASE 1: Print VRAM state"
      ansible.builtin.debug:
        msg: >-
          VRAM after stopping llama-server-qwen:
          {{ cutover_vram_free_check.stdout if (cutover_vram_free_check.stdout | length > 0)
             else '(no GPU processes — VRAM free)' }}
      tags: [cutover_stop_qwen, cutover]
    # =======================================================================
    # PHASE 2 — Redeploy llama-server-router unit with --port 8002
    # =======================================================================
    - name: "[cutover] PHASE 2: Deploy llama-server-router unit file (port 8002 — production)"
      ansible.builtin.template:
        src: "../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
        dest: /etc/systemd/system/llama-server-router.service
        owner: root
        group: root
        mode: "0644"
      register: cutover_router_unit_deployed
      notify:
        - reload systemd
      tags: [cutover_deploy_unit, cutover]
    - name: "[cutover] PHASE 2: Flush handlers (daemon-reload before start)"
      ansible.builtin.meta: flush_handlers
      tags: [cutover_deploy_unit, cutover]
    # =======================================================================
    # PHASE 3 — Enable + start llama-server-router on :8002
    # =======================================================================
    - name: "[cutover] PHASE 3: Enable + start llama-server-router (production, :8002)"
      ansible.builtin.systemd:
        name: llama-server-router
        state: "{{ 'restarted' if (cutover_router_unit_deployed.changed | default(false)) else 'started' }}"
        enabled: true
        daemon_reload: true
      tags: [cutover_start_router, cutover]
    # =======================================================================
    # PHASE 4 — Validation gates 1-3 (hard gates against now-production :8002)
    # =======================================================================
    - name: "[cutover] GATE 1a: Wait for router /health on :8002 (up to 5min — cold model load)"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
        status_code: 200
      register: cutover_health
      retries: 30
      delay: 10
      until: cutover_health.status == 200
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 1a: Trigger model load (router lazy-loads on first request)"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/chat/completions"
        method: POST
        body_format: json
        body:
          model: "{{ llm_router_expected_model_id }}"
          messages:
            - role: user
              content: "Reply with one word: hello"
          max_tokens: 5
          temperature: 0.0
        status_code: 200
        return_content: true
        timeout: 300
      register: cutover_warmup
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 1a: Report warmup"
      ansible.builtin.debug:
        msg:
          - "Model loaded. finish_reason={{ cutover_warmup.json.choices[0].finish_reason | default('unknown') }}"
          - "Response: {{ cutover_warmup.json.choices[0].message.content | default('(empty)') | truncate(100) }}"
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 1b: Query /v1/models on :8002"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
        status_code: 200
        return_content: true
      register: cutover_models
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 1b: Fail if expected model ID not found"
      ansible.builtin.fail:
        msg: >-
          GATE 1 FAIL: '{{ llm_router_expected_model_id }}' not found in /v1/models.
          Returned: {{ cutover_models.json.data | map(attribute='id') | list }}
      when:
        - cutover_models.json.data | selectattr('id', 'equalto', llm_router_expected_model_id) | list | length == 0
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 1b: Extract ctx-size from router model args"
      ansible.builtin.set_fact:
        cutover_qwen_n_ctx: >-
          {%- set model = cutover_models.json.data | selectattr('id', 'equalto', llm_router_expected_model_id) | first -%}
          {%- set args = model.status.args -%}
          {%- set ctx_idx = args.index('--ctx-size') if '--ctx-size' in args else -1 -%}
          {{ args[ctx_idx + 1] | int if ctx_idx >= 0 else 0 }}
      when:
        - cutover_models.json.data | selectattr('id', 'equalto', llm_router_expected_model_id) | list | length > 0
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 1b: Fail if n_ctx < 64000"
      ansible.builtin.fail:
        msg: "GATE 1 FAIL: --ctx-size={{ cutover_qwen_n_ctx }} < 64000 (Hermes 64K floor)."
      when:
        - cutover_qwen_n_ctx is defined
        - cutover_qwen_n_ctx | int < 64000
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 1b: PASS — n_ctx >= 64K"
      ansible.builtin.debug:
        msg: "GATE 1 PASS: --ctx-size={{ cutover_qwen_n_ctx }} >= 64000."
      when:
        - cutover_qwen_n_ctx is defined
        - cutover_qwen_n_ctx | int >= 64000
      tags: [cutover_validate, cutover]
    # --- Gate 2: Tool-calling through router proxy ---
    - name: "[cutover] GATE 2: Tool-calling probe"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/chat/completions"
        method: POST
        body_format: json
        body:
          model: "{{ llm_router_expected_model_id }}"
          messages:
            - role: user
              content: "What is the current weather in Chicago? Use the provided tool."
          tools:
            - type: function
              function:
                name: get_weather
                description: "Get current weather conditions for a city"
                parameters:
                  type: object
                  properties:
                    city:
                      type: string
                      description: "The city name"
                  required:
                    - city
          temperature: 0.0
        status_code: 200
        return_content: true
        timeout: 120
      register: cutover_toolcall_probe
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 2: Fail if not finish_reason=tool_calls"
      ansible.builtin.fail:
        msg: >-
          GATE 2 FAIL: finish_reason={{ cutover_toolcall_probe.json.choices[0].finish_reason | default('(missing)') }}
          (expected tool_calls). Response: {{ cutover_toolcall_probe.json | to_json }}
      when:
        - cutover_toolcall_probe.json.choices[0].finish_reason | default('') != 'tool_calls'
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 2: PASS"
      ansible.builtin.debug:
        msg:
          - "GATE 2 PASS: finish_reason=tool_calls"
          - "function: {{ cutover_toolcall_probe.json.choices[0].message.tool_calls[0].function.name | default('(unknown)') }}"
          - "arguments: {{ cutover_toolcall_probe.json.choices[0].message.tool_calls[0].function.arguments | default('(none)') }}"
      when:
        - cutover_toolcall_probe.json.choices[0].finish_reason | default('') == 'tool_calls'
      tags: [cutover_validate, cutover]
    # --- Gate 2b: Hallucination stress ---
    - name: "[cutover] GATE 2b: Hallucination stress probe"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/chat/completions"
        method: POST
        body_format: json
        body:
          model: "{{ llm_router_expected_model_id }}"
          messages:
            - role: user
              content: "Tell me a brief fact about the planet Mars. Do not call any functions."
          tools:
            - type: function
              function:
                name: get_weather
                description: "Get current weather conditions for a city"
                parameters:
                  type: object
                  properties:
                    city:
                      type: string
                  required:
                    - city
          temperature: 0.1
        status_code: 200
        return_content: true
        timeout: 120
      register: cutover_halluc_probe
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 2b: Fail if spurious tool_calls"
      ansible.builtin.fail:
        msg: >-
          GATE 2b FAIL: finish_reason=tool_calls on unrelated prompt (Mars fact).
          Over-triggering through router. Response: {{ cutover_halluc_probe.json | to_json }}
      when:
        - cutover_halluc_probe.json.choices[0].finish_reason | default('') == 'tool_calls'
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 2b: PASS"
      ansible.builtin.debug:
        msg: "GATE 2b PASS: finish_reason={{ cutover_halluc_probe.json.choices[0].finish_reason }} — no spurious tool_calls."
      when:
        - cutover_halluc_probe.json.choices[0].finish_reason | default('') != 'tool_calls'
      tags: [cutover_validate, cutover]
    # --- Gate 3: VRAM guard ---
    - name: "[cutover] GATE 3: Check VRAM usage (--models-max 1 guard)"
      ansible.builtin.command:
        cmd: nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv,noheader
      register: cutover_vram_post
      changed_when: false
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 3: Parse VRAM used MiB"
      ansible.builtin.set_fact:
        cutover_vram_used_mib: "{{ cutover_vram_post.stdout.split(',')[0].strip().split(' ')[0] | int }}"
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 3: Fail if VRAM exceeds ceiling"
      ansible.builtin.fail:
        msg: >-
          GATE 3 FAIL: {{ cutover_vram_used_mib }} MiB > {{ llm_router_vram_max_mib }} MiB ceiling.
          Full: {{ cutover_vram_post.stdout }}
      when:
        - cutover_vram_used_mib | int > llm_router_vram_max_mib | int
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 3: Count GPU processes"
      ansible.builtin.command:
        cmd: nvidia-smi --query-compute-apps=pid,name --format=csv,noheader
      register: cutover_gpu_procs
      changed_when: false
      failed_when: false
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 3: PASS"
      ansible.builtin.debug:
        msg:
          - "GATE 3 PASS: {{ cutover_vram_used_mib }} MiB / {{ llm_router_vram_max_mib }} MiB ceiling."
          - "GPU processes: {{ cutover_gpu_procs.stdout_lines | default(['(none)']) }}"
          - "Full nvidia-smi: {{ cutover_vram_post.stdout }}"
      when:
        - cutover_vram_used_mib | int <= llm_router_vram_max_mib | int
      tags: [cutover_validate, cutover]
    # =======================================================================
    # PHASE 5 — Gate 4: Bundled SvelteKit Web UI (required this time)
    # =======================================================================
    - name: "[cutover] GATE 4: Check bundled SvelteKit UI at :8002"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/"
        status_code: [200, 301, 302]
        return_content: true
        timeout: 30
      register: cutover_ui_check
      failed_when: false
      tags: [cutover_validate, cutover_ui, cutover]
    - name: "[cutover] GATE 4: Inspect UI content"
      ansible.builtin.set_fact:
        cutover_ui_is_html: "{{ 'html' in (cutover_ui_check.content | default('') | lower) or '<!doctype' in (cutover_ui_check.content | default('') | lower) }}"
        cutover_ui_has_model_select: "{{ 'select' in (cutover_ui_check.content | default('') | lower) or 'model' in (cutover_ui_check.content | default('') | lower) }}"
      when: cutover_ui_check is defined
      tags: [cutover_validate, cutover_ui, cutover]
    - name: "[cutover] GATE 4: Report UI check and bookmark URL"
      ansible.builtin.debug:
        msg:
          - "======================================================================"
          - "GATE 4 UI CHECK:"
          - "  HTTP status:          {{ cutover_ui_check.status | default('UNREACHABLE') }}"
          - "  Is HTML:              {{ cutover_ui_is_html | default(false) }}"
          - "  Contains model/select: {{ cutover_ui_has_model_select | default(false) }}"
          - "  BOOKMARK URL:         http://{{ llm_router_bind_address }}:{{ llm_router_port }}/"
          - "  {{ 'GATE 4 PASS — UI serving HTML at :8002.' if (cutover_ui_check.status | default(0) | int in [200, 301, 302]) else 'GATE 4 WARN — UI not reachable (HTTP ' + (cutover_ui_check.status | default('FAIL') | string) + ').' }}"
          - "======================================================================"
      when: cutover_ui_check is defined
      tags: [cutover_validate, cutover_ui, cutover]
    # =======================================================================
    # CUTOVER SUMMARY
    # =======================================================================
    - name: "[cutover] CUTOVER SUMMARY — production promoted"
      ansible.builtin.debug:
        msg:
          - "======================================================================"
          - "CUTOVER COMPLETE: llama-server-router is now production."
          - ""
          - "  Service:  llama-server-router.service (enabled, running)"
          - "  Port:     8002 (unchanged for all 8 Hermes profiles)"
          - "  Model:    {{ llm_router_expected_model_id }}"
          - "  Mode:     Router/supervisor (--models-dir /opt/models, --models-max 1)"
          - ""
          - "  Gate 1 (n_ctx >= 64K):     PASS ({{ cutover_qwen_n_ctx | default('N/A') }})"
          - "  Gate 2 (tool-calling):     PASS (finish_reason=tool_calls)"
          - "  Gate 2b (halluc stress):   PASS (no spurious tool_calls)"
          - "  Gate 3 (VRAM <= 23000MiB): PASS ({{ cutover_vram_used_mib | default('N/A') }} MiB)"
          - "  Gate 4 (Web UI):           HTTP {{ cutover_ui_check.status | default('N/A') }}"
          - ""
          - "  ROLLBACK TARGET: /etc/systemd/system/llama-server-qwen.service (unit preserved)"
          - "  ROLLBACK CMD: sudo systemctl enable --now llama-server-qwen"
          - "               sudo systemctl disable --now llama-server-router"
          - "  Or: ansible-playbook -i inventory.yml day2_cutover_qwen_to_router.yml --tags cutover_rollback"
          - ""
          - "  Web UI bookmark: http://{{ llm_router_bind_address }}:{{ llm_router_port }}/"
          - "======================================================================"
      tags: [cutover]
    # =======================================================================
    # ROLLBACK — tag cutover_rollback reverses the cutover
    # Run: ansible-playbook -i inventory.yml day2_cutover_qwen_to_router.yml --tags cutover_rollback
    # WARNING: rollback_task has no dependency on cutover tags — safe to run standalone.
    # =======================================================================
    - name: "[cutover_rollback] Stop + disable llama-server-router"
      ansible.builtin.systemd:
        name: llama-server-router
        state: stopped
        enabled: false
      tags: [cutover_rollback, never]    # 'never' = only runs with explicit --tags cutover_rollback
    - name: "[cutover_rollback] Enable + start llama-server-qwen (restore bare :8002)"
      ansible.builtin.systemd:
        name: llama-server-qwen
        state: started
        enabled: true
      tags: [cutover_rollback, never]
    - name: "[cutover_rollback] Verify rollback /health"
      ansible.builtin.uri:
        url: "http://{{ llm_bind_address | default('10.1.71.130') }}:8002/health"
        status_code: 200
        timeout: 30
      register: cutover_rollback_health
      failed_when: false
      tags: [cutover_rollback, never]
    - name: "[cutover_rollback] Report rollback result"
      ansible.builtin.debug:
        msg: >-
          ROLLBACK: llama-server-qwen :8002 health returned
          {{ cutover_rollback_health.status | default('UNREACHABLE') }}.
          {{ 'OK — production restored to bare qwen.' if (cutover_rollback_health.status | default(0) | int == 200)
             else 'WARNING — health check failed. Check manually.' }}
      tags: [cutover_rollback, never]
  handlers:
    - name: reload systemd
      ansible.builtin.systemd:
        daemon_reload: true
@@ -0,0 +1,80 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: ansible/playbooks/day2_install_node_exporter.yml
 # DESCRIPTION: Install and configure Prometheus Node Exporter on Linux hosts.
 #
 # Targets two groups with different firewall managers:
 #   - proxmox_nodes  : Proxmox hypervisors (Debian, firewalld not present,
 #                      uses iptables/no firewall — just open the port via UFW
 #                      if present, otherwise skip)
 #   - monitored_vms  : Ubuntu VMs managed by UFW
 #
 # After installing on new hosts, re-runs day1_deploy_monitoring.yml to refresh
 # the Prometheus scrape config is NOT needed — targets are already statically
 # defined in prometheus.yaml.j2 for the Proxmox nodes. For new VMs, add the
 # IP to the prometheus.yaml.j2 proxmox-vms job and re-run day1_deploy_monitoring.yml.
 #
 # Usage:
 #   ansible-playbook -i inventory.yml playbooks/day2_install_node_exporter.yml
 #   ansible-playbook -i inventory.yml playbooks/day2_install_node_exporter.yml --limit proxmox_nodes
 #   ansible-playbook -i inventory.yml playbooks/day2_install_node_exporter.yml --limit monitored_vms
 # ------------------------------------------------------------------------------
 # ── Play 1: Proxmox hypervisors ───────────────────────────────────────────────
 # Proxmox runs Debian. The prometheus.prometheus.node_exporter role installs
 # a binary + systemd service without touching apt, which is what we want on
 # hypervisors (keep the package footprint clean).
 # Note: Proxmox does not run UFW. Port 9100 is open by default on VLAN 71.
 - name: Install Node Exporter on Proxmox hypervisors
  hosts: proxmox_nodes
  become: true
  vars:
    node_exporter_version: "1.9.1"
    node_exporter_web_listen_address: "0.0.0.0:9100"
    node_exporter_enabled_collectors:
      - systemd
      - processes
      - filesystem
      - meminfo
      - cpu
      - diskstats
      - netdev
      - loadavg
      - uname
  roles:
    - role: prometheus.prometheus.node_exporter
 # ── Play 2: Ubuntu VMs ────────────────────────────────────────────────────────
 # Standard Ubuntu hosts with UFW. Same role, adds UFW allow rule for 9100.
 - name: Install Node Exporter on monitored VMs
  hosts: monitored_vms
  become: true
  vars:
    node_exporter_version: "1.9.1"
    node_exporter_web_listen_address: "0.0.0.0:9100"
    node_exporter_enabled_collectors:
      - systemd
      - processes
      - filesystem
      - meminfo
      - cpu
      - diskstats
      - netdev
      - loadavg
      - uname
  pre_tasks:
    - name: Open Node Exporter port in UFW
      community.general.ufw:
        rule: allow
        port: "9100"
        proto: tcp
        comment: "Prometheus Node Exporter"
  roles:
    - role: prometheus.prometheus.node_exporter
@@ -0,0 +1,277 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day2_per_model_ctx_size.yml
 # DESCRIPTION: Right-size --ctx-size per model workload on llama-server-router
 #              (already in --models-preset mode since t_9adf0889).
 #
 # Context (t_ryan_per_model_ctx, 2026-08-13, requested by Ryan via JARVIS):
 #   All 3 preset models currently launch with a uniform --ctx-size 65536.
 #   This playbook narrows two of them to match actual workload:
 #     - Meta-Llama-3.1-8B-Instruct-Q4_K_M (alias Meta-Llama-3.1-8B-Instruct-4bit):
 #         ctx-size 65536 -> 8192  (tool-routing / micro-tasks: title gen, MCP
 #         tool calls, approval checks)
 #     - Phi-3.5-mini-instruct-Q8_0 (alias Phi-3.5-mini-instruct-8bit):
 #         ctx-size 65536 -> 32768 (long web scrapes / session-log compression)
 #   Both also move flash-attn from "auto" to explicit "true" per Ryan's spec.
 #   Qwen3.6-35B-A3B-UD-Q4_K_S is INTENTIONALLY left untouched at 65536/auto.
 #
 #   Existing aliases (Meta-Llama-3.1-8B-Instruct-4bit, Phi-3.5-mini-instruct-8bit)
 #   are PRESERVED as-is. Ryan's pasted TOML used different alias strings
 #   ("llama-3.1-8b", "phi-3.5-mini") but renaming aliases was not explicitly
 #   requested and would break live Hermes custom_providers routing — flagged
 #   in the deployment report rather than applied silently.
 #
 # IMPORTANT — Hermes side effect: /home/hermes/.hermes/config.yaml declares
 # context_length: 65536 for both these models under custom_providers. This
 # playbook does NOT touch that file (out of role/agent scope) but the value
 # becomes STALE the moment this playbook lands. Flag to JARVIS/Maria Hill.
 #
 # Usage (from ~/git/homelab/ansible):
 #   ansible-playbook -i inventory.yml playbooks/day2_per_model_ctx_size.yml
 #
 # Author: War Machine (2026-08-13, t_ryan_per_model_ctx)
 # ------------------------------------------------------------------------------
 - name: "Right-size per-model ctx-size on llama-server-router (Llama 8k, Phi 32k)"
  hosts: astro_orbiter
  gather_facts: false
  become: true
  vars:
    # Preset mode already active in production (t_9adf0889) — keep it on.
    llm_router_preset_enabled: true
    llm_router_preset_path: /opt/llama-server-router-preset.ini
    llm_router_enabled: true
    # Production port
    llm_router_port: 8002
    llm_router_bind_address: "10.1.71.130"
    llm_router_allowed_source_cidr: "10.1.70.0/24"
    llm_bind_address: "10.1.71.130"
    llm_allowed_source_cidr: "10.1.70.0/24"
    llm_service_user: jarvis
    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
    llm_models_dir: /opt/models
    llm_router_service_name: llama-server-router
    llm_router_models_dir: /opt/models
    llm_router_models_max: 4
    llm_router_parallel: 1
    llm_router_gpu_layers: 99
    llm_router_batch_size: 2048
    llm_router_ubatch_size: 512
    llm_router_cache_type_k: q4_0
    llm_router_cache_type_v: q4_0
    # Qwen — untouched baseline (also used as router-wide fallback default)
    llm_router_ctx_size: 65536
    llm_router_flash_attn: "auto"
    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
    llm_router_vram_max_mib: 23000
    # --- THE CHANGE: per-model overrides ---
    llm_router_llama_ctx_size: 8192
    llm_router_llama_flash_attn: "true"
    llm_router_phi_ctx_size: 32768
    llm_router_phi_flash_attn: "true"
  handlers:
    - name: reload systemd
      ansible.builtin.systemd:
        daemon_reload: true
      become: true
      listen: "reload systemd"
    - name: restart router
      ansible.builtin.systemd:
        name: llama-server-router
        state: restarted
      become: true
      listen: "restart router"
  tasks:
    # ==========================================================================
    # PHASE 1: Deploy the preset INI with new per-model ctx-size/flash-attn
    # ==========================================================================
    - name: "[ctx-resize] Deploy preset INI to {{ llm_router_preset_path }}"
      ansible.builtin.template:
        src: "../roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2"
        dest: "{{ llm_router_preset_path }}"
        owner: root
        group: root
        mode: "0644"
      register: ctx_resize_preset_deployed
      notify:
        - restart router
    - name: "[ctx-resize] Deploy router systemd unit (drop global --ctx-size/--flash-attn in preset mode)"
      ansible.builtin.template:
        src: "../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
        dest: /etc/systemd/system/llama-server-router.service
        owner: root
        group: root
        mode: "0644"
      register: ctx_resize_unit_deployed
      notify:
        - reload systemd
        - restart router
    - name: "[ctx-resize] Flush handlers (daemon-reload + router restart if changed)"
      ansible.builtin.meta: flush_handlers
    # ==========================================================================
    # PHASE 2: Verify
    # ==========================================================================
    - name: "[ctx-resize] Wait for /health"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
        status_code: 200
        timeout: 30
      retries: 12
      delay: 5
      register: ctx_resize_health
      until: ctx_resize_health.status == 200
    - name: "[ctx-resize] Query /v1/models"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
        status_code: 200
        return_content: true
        timeout: 30
      register: ctx_resize_models
    - name: "[ctx-resize] Trigger load — Llama (confirms actual load + captures live args)"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/chat/completions"
        method: POST
        body_format: json
        body:
          model: "Meta-Llama-3.1-8B-Instruct-Q4_K_M"
          messages:
            - role: user
              content: "Reply with one word: hello"
          max_tokens: 5
          temperature: 0.0
        status_code: 200
        return_content: true
        timeout: 120
      register: ctx_resize_llama_warmup
    - name: "[ctx-resize] Trigger load — Phi (confirms actual load + captures live args)"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/chat/completions"
        method: POST
        body_format: json
        body:
          model: "Phi-3.5-mini-instruct-Q8_0"
          messages:
            - role: user
              content: "Reply with one word: hello"
          max_tokens: 5
          temperature: 0.0
        status_code: 200
        return_content: true
        timeout: 120
      register: ctx_resize_phi_warmup
    - name: "[ctx-resize] Re-query /v1/models after warmup (final state)"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
        status_code: 200
        return_content: true
        timeout: 30
      register: ctx_resize_models_final
    - name: "[ctx-resize] Extract Llama args"
      ansible.builtin.set_fact:
        ctx_resize_llama_args: >-
          {{ (ctx_resize_models_final.json.data | selectattr('id', 'equalto', 'Meta-Llama-3.1-8B-Instruct-Q4_K_M') | first).status.args }}
        ctx_resize_llama_status: >-
          {{ (ctx_resize_models_final.json.data | selectattr('id', 'equalto', 'Meta-Llama-3.1-8B-Instruct-Q4_K_M') | first).status.value }}
    - name: "[ctx-resize] Extract Phi args"
      ansible.builtin.set_fact:
        ctx_resize_phi_args: >-
          {{ (ctx_resize_models_final.json.data | selectattr('id', 'equalto', 'Phi-3.5-mini-instruct-Q8_0') | first).status.args }}
        ctx_resize_phi_status: >-
          {{ (ctx_resize_models_final.json.data | selectattr('id', 'equalto', 'Phi-3.5-mini-instruct-Q8_0') | first).status.value }}
    - name: "[ctx-resize] Extract Qwen args (must be unchanged)"
      ansible.builtin.set_fact:
        ctx_resize_qwen_args: >-
          {{ (ctx_resize_models_final.json.data | selectattr('id', 'equalto', 'Qwen3.6-35B-A3B-UD-Q4_K_S') | first).status.args }}
    - name: "[ctx-resize] GATE — Llama ctx-size must be 8192"
      ansible.builtin.assert:
        that:
          - "'8192' in ctx_resize_llama_args"
          - ctx_resize_llama_args[ctx_resize_llama_args.index('--ctx-size') + 1] == '8192'
        fail_msg: "Llama ctx-size not 8192. Args: {{ ctx_resize_llama_args }}"
        success_msg: "Llama ctx-size confirmed 8192."
    - name: "[ctx-resize] GATE — Llama flash-attn must be true"
      ansible.builtin.assert:
        that:
          - ctx_resize_llama_args[ctx_resize_llama_args.index('--flash-attn') + 1] == 'true'
        fail_msg: "Llama flash-attn not true. Args: {{ ctx_resize_llama_args }}"
        success_msg: "Llama flash-attn confirmed true."
    - name: "[ctx-resize] GATE — Llama loaded successfully"
      ansible.builtin.assert:
        that:
          - ctx_resize_llama_status == 'loaded'
        fail_msg: "Llama status is '{{ ctx_resize_llama_status }}', expected 'loaded'."
        success_msg: "Llama status confirmed 'loaded'."
    - name: "[ctx-resize] GATE — Phi ctx-size must be 32768"
      ansible.builtin.assert:
        that:
          - ctx_resize_phi_args[ctx_resize_phi_args.index('--ctx-size') + 1] == '32768'
        fail_msg: "Phi ctx-size not 32768. Args: {{ ctx_resize_phi_args }}"
        success_msg: "Phi ctx-size confirmed 32768."
    - name: "[ctx-resize] GATE — Phi flash-attn must be true"
      ansible.builtin.assert:
        that:
          - ctx_resize_phi_args[ctx_resize_phi_args.index('--flash-attn') + 1] == 'true'
        fail_msg: "Phi flash-attn not true. Args: {{ ctx_resize_phi_args }}"
        success_msg: "Phi flash-attn confirmed true."
    - name: "[ctx-resize] GATE — Phi loaded successfully"
      ansible.builtin.assert:
        that:
          - ctx_resize_phi_status == 'loaded'
        fail_msg: "Phi status is '{{ ctx_resize_phi_status }}', expected 'loaded'."
        success_msg: "Phi status confirmed 'loaded'."
    - name: "[ctx-resize] GATE — Qwen ctx-size UNCHANGED at 65536"
      ansible.builtin.assert:
        that:
          - ctx_resize_qwen_args[ctx_resize_qwen_args.index('--ctx-size') + 1] == '65536'
        fail_msg: "Qwen ctx-size changed unexpectedly! Args: {{ ctx_resize_qwen_args }}"
        success_msg: "Qwen ctx-size confirmed UNCHANGED at 65536."
    - name: "[ctx-resize] PASS — summary"
      ansible.builtin.debug:
        msg:
          - "================================================================"
          - "PER-MODEL CTX-SIZE DEPLOYMENT — COMPLETE"
          - ""
          - "  Llama-3.1-8B (Meta-Llama-3.1-8B-Instruct-Q4_K_M):"
          - "    status:     {{ ctx_resize_llama_status }}"
          - "    args:       {{ ctx_resize_llama_args }}"
          - ""
          - "  Phi-3.5-mini (Phi-3.5-mini-instruct-Q8_0):"
          - "    status:     {{ ctx_resize_phi_status }}"
          - "    args:       {{ ctx_resize_phi_args }}"
          - ""
          - "  Qwen3.6-35B-A3B-UD-Q4_K_S: UNCHANGED (ctx-size 65536, args: {{ ctx_resize_qwen_args }})"
          - ""
          - "  ACTION NEEDED: /home/hermes/.hermes/config.yaml custom_providers"
          - "  context_length: 65536 for both Meta-Llama-3.1-8B-Instruct-4bit and"
          - "  Phi-3.5-mini-instruct-8bit is now STALE (actual: 8192 / 32768)."
          - "  Flag to JARVIS/Maria Hill for correction — NOT done by this playbook."
          - "================================================================"
@@ -0,0 +1,38 @@
 ---
 # ------------------------------------------------------------------------------
 # Playbook: day2_qwen38_ctx128k.yml
 # Purpose:  Bump Qwen3.8-27B-Q4_K_M ctx-size from 32768 to 131072 (128K)
 #           on astro-orbiter's production router (port 8002).
 #
 # What this playbook does:
 #   1. Renders the updated llama-server-router-preset.ini.j2 (now with
 #      llm_router_qwen38_ctx_size: 131072) to /opt/llama-server-router-preset.ini.
 #   2. Restarts llama-server-router.service.
 #   3. Verifies the router loads Qwen3.8-27B at ctx=131072 in status.args.
 #
 # Context:
 #   - Empirical VRAM test (t_4455a44c): 131072 ctx = 20,282 MiB Qwen3.8
 #     + 558 MiB nomic-embed = ~20.8GB total; ~3.2GB headroom on 24GB RTX 3090.
 #     Co-resident with nomic-embed: comfortably fits.
 #   - Ryan approved this deployment.
 #   - Semaphore SSH gap for astro-orbiter still applies (t_730f9584 / t_33acbb2e);
 #     running direct CLI Ansible per standing exception.
 #
 # Run:
 #   cd /home/hermes/git/homelab/ansible
 #   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook \
 #     -i inventory.yml \
 #     playbooks/day2_qwen38_ctx128k.yml
 #
 # Task reference: t_441470b9 — War Machine, 2026-08-16
 # ------------------------------------------------------------------------------
 - name: Bump Qwen3.8-27B ctx-size to 131072 on astro-orbiter
  hosts: astro-orbiter
  become: true
  vars:
    llm_router_preset_enabled: true
    llm_router_qwen38_ctx_size: 131072
  roles:
    - role: llm-inference-multimodel
      tags: [preset, systemd, verify]
@@ -0,0 +1,65 @@
 ---
 # ------------------------------------------------------------------------------
 # Playbook: day2_qwen38_ctx128k_rollback.yml
 # Purpose:  Roll back Qwen3.8-27B-Q4_K_M ctx-size from 131072 back to 65536
 #           on astro-orbiter's production router (port 8002).
 #
 # What this playbook does:
 #   1. Renders the updated llama-server-router-preset.ini.j2 (now with
 #      llm_router_qwen38_ctx_size: 65536) to
 #      /opt/llama-server-router-preset.ini.
 #   2. Restarts llama-server-router.service.
 #   3. Verifies the router loads Qwen3.8-27B at ctx=65536 in status.args.
 #
 # Context:
 #   - t_441470b9 (2026-08-16): ctx-size bumped 32768 -> 131072. Verified VRAM
 #     at 131072 ctx with only Qwen3.8 + nomic-embed co-resident: ~20,282 MiB
 #     + 558 MiB = ~20.8 GB on 24 GB RTX 3090. Comfortably safe.
 #   - t_72646029 (2026-08-17): Phi-3.5mini moved to GPU (n-gpu-layers=99)
 #     to enable concurrent residency with CPU-offloaded Coder-14B and
 #     Llama-3.1-8B. This added ~2GB CUDA context buffers for Phi + shifted
 #     Phi's model weights onto the GPU (~3.8GB).
 #   - NEW steady-state VRAM: Qwen3.8 @ 131072 ctx (~20,282 MiB) + nomic-embed
 #     (~558 MiB) + Llama CUDA ctx (~1,706 MiB) + Coder CUDA ctx (~1,390 MiB)
 #     = ~24,004 MiB. Adding Phi-3.5 (~3,800 MiB weights + ~1.4 GB CUDA ctx)
 #     pushes total to ~29,000+ MiB — exceeding the 24,576 MiB RTX 3090 limit.
 #     Qwen3.8-27B-131072 now fails to load (HTTP 500, OOM before llama.cpp
 #     reaches the model-loading phase).
 #   - FIX: reduce Qwen3.8 ctx-size 131072 -> 65536. This reduces KV cache
 #     from ~6GB to ~3GB, freeing ~3GB of VRAM. New estimated steady-state:
 #     Qwen3.8 @ 65536 ctx (~17,068 MiB) + nomic (~558) + Llama ctx (~1,706)
 #     + Coder ctx (~1,390) + Phi-3.5 (~3,800 + ~1,400 CUDA ctx) = ~25,922 MiB.
 #     Still over 24,576 — see "Phase 2" below for the secondary fix.
 #
 #   IMPORTANT: Rolling back ctx-size alone may NOT be sufficient. The
 #   hardware reference (astro-orbiter-hardware.md line 166, t_72646029)
 #   states steady-state ~24,004 MiB WITHOUT Phi on GPU. Adding Phi-3.5 back
 #   to GPU tips it over. This playbook handles the context rollback; if Qwen3.8
 #   still fails to load after Phase R, Wong should escalate to Ryan for a
 #   decision on either (a) offloading Phi-3.5mini to CPU (n-gpu-layers=0),
 #   or (b) adding a second GPU. Document the Phase 2 finding as a separate
 #   follow-up task if needed.
 #
 #   The 64K floor from the 2026-08-12 cutover validation (t_cd0d5388, Gate 1)
 #   still applies — ctx-size=65536 satisfies it.
 #
 # Run:
 #   cd /home/hermes/git/homelab/ansible
 #   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook \
 #     -i inventory.yml \
 #     playbooks/day2_qwen38_ctx128k_rollback.yml
 #
 # Task reference: t_c9fed26c — War Machine benchmark, 2026-08-18
 # Root cause: t_72646029 CPU-offload deployment added Phi-3.5 to GPU,
 #   shifting total VRAM past the 24,576 MiB ceiling when Qwen3.8 runs at 128K.
 # ------------------------------------------------------------------------------
 - name: Roll back Qwen3.8-27B ctx-size to 65536 on astro-orbiter
  hosts: astro-orbiter
  become: true
  vars:
    llm_router_preset_enabled: true
    llm_router_qwen38_ctx_size: 65536
  roles:
    - role: llm-inference-multimodel
      tags: [preset, systemd, verify]
@@ -0,0 +1,36 @@
 ---
 # ------------------------------------------------------------------------------
 # Playbook: day2_swap_qwen38.yml
 # Purpose:  Swap the primary production model on astro-orbiter router from
 #           Qwen3.6-35B-A3B-UD-Q4_K_S to Qwen3.8-27B-Q4_K_M.
 #           This is a GitOps-encoded record of the swap performed 2026-08-16
 #           per Ryan's direction (kanban task t_f5f7e9ad).
 #
 # What this playbook does:
 #   1. Renders the updated llama-server-router-preset.ini.j2 to
 #      /opt/llama-server-router-preset.ini on astro-orbiter.
 #   2. Reloads the llama-server-router service (SIGHUP / restart as needed).
 #   3. Verifies the new model ID appears in /v1/models.
 #
 # Prerequisites:
 #   - Qwen3.8-27B-Q4_K_M.gguf must be present in /opt/models on astro-orbiter.
 #     (Downloaded out-of-band via wget during the swap task.)
 #   - roles/llm-inference-multimodel/defaults/main.yml updated to reference
 #     Qwen3.8-27B-Q4_K_M (done in this same commit).
 #
 # Run:
 #   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook \
 #     -i inventory.yml \
 #     playbooks/day2_swap_qwen38.yml
 #
 # Task reference: t_f5f7e9ad — War Machine, 2026-08-16
 # ------------------------------------------------------------------------------
 - name: Swap primary model to Qwen3.8-27B-Q4_K_M on astro-orbiter
  hosts: astro-orbiter
  become: true
  vars:
    llm_router_preset_enabled: true
  roles:
    - role: llm-inference-multimodel
      tags: [preset, systemd, verify]
@@ -0,0 +1,22 @@
 ---
 # Playbook: day3_deploy_qwen38_ctx131k.yml
 # Purpose: Deploy Qwen3.8-27B-Q4_K_M ctx-size 65536 -> 131072 to astro-orbiter
 #         via llama-swap config re-render + restart.
 #
 # The git change to defaults/main.yml (line 235: ctx_size: 131072) is already staged.
 # This playbook renders /etc/llama-swap/config.yaml from the updated defaults
 # and restarts llama-swap to load the new ctx-size.
 #
 # Run:
 #   cd /home/hermes/git/homelab/ansible
 #   ansible-playbook -i inventory.yml playbooks/day3_deploy_qwen38_ctx131k.yml
 #
 - name: Deploy Qwen3.8 ctx-size 131072 to astro-orbiter
  hosts: astro-orbiter
  become: true
  vars:
    llm_swapmode_enabled: true
  roles:
    - role: llm-inference-multimodel
      tags: [swapmode_config, swapmode_systemd, swapmode_verify]
@@ -1,30 +0,0 @@
 ---
 # - name: Step 1 - Install Prerequisites
 #   hosts: fastpass
 #   become: true
 #   gather_facts: true
 #   roles:
 #     - role: kubernetes-prerequisites
 # - name: Step 2 - Deploy First Control Plane Node
 #   hosts: fastpass_control_plane[0]
 #   become: true
 #   gather_facts: false
 #   roles:
 #     - role: fastpass-first-control-plane
 - name: Step 3 - Deploy Additional Control Plane Nodes
  hosts: fastpass_control_plane[1:]
  become: true
  gather_facts: true # false
  roles:
    - role: fastpass-additional-control-plane
  vars:
    cluster_name: "fastpass"
 # - name: Step 4 - Deploy Worker Nodes
 #   hosts: fastpass_workers
 #   become: true
 #   gather_facts: false
 #   roles:
 #     - role: fastpass-workers
@@ -1,157 +0,0 @@
 ---
 # FastPass Kubernetes Cluster Deployment
 # This playbook deploys a complete Kubernetes cluster on Fedora
 - name: 1. Preflight checks and system preparation
  hosts: fastpass
  become: true
  gather_facts: true
  roles:
    - role: kubernetes
 - name: 2. Initialize first control plane node
  hosts: fastpass_control_plane[0]
  become: true
  gather_facts: false
  roles:
    - role: fastpass-control-plane
 - name: 3. Install Calico CNI
  hosts: fastpass_control_plane[0]
  become: false
  gather_facts: false
  environment:
    KUBECONFIG: "{{ kubeconfig_path }}"
  tasks:
    - name: Check if Calico is already installed
      ansible.builtin.shell: kubectl get pods -n kube-system -l k8s-app=calico-node --no-headers | wc -l
      delegate_to: localhost
      register: calico_check
      ignore_errors: true
    - name: Display Calico check result
      ansible.builtin.debug:
        msg: "Calico pods found: {{ calico_check.stdout | trim }}"
      delegate_to: localhost
    - name: Install Calico CNI
      ansible.builtin.command: kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/{{ calico_version | default('v3.28.0') }}/manifests/calico.yaml
      delegate_to: localhost
      when: calico_check.stdout | trim == "0"
      register: calico_install_result
      ignore_errors: true
    - name: Display Calico installation result
      ansible.builtin.debug:
        msg: "Calico install stdout: {{ calico_install_result.stdout }}"
      delegate_to: localhost
      when: calico_check.stdout | trim == "0"
    - name: Wait for Calico node pods to be ready
      ansible.builtin.command: kubectl wait --for=condition=ready pod -l k8s-app=calico-node -n kube-system --timeout=300s
      delegate_to: localhost
      when: calico_check.stdout | trim == "0"
 - name: 4. Wait for first control plane to be fully ready
  hosts: fastpass_control_plane[0]
  become: false
  gather_facts: false
  environment:
    KUBECONFIG: "{{ kubeconfig_path }}"
  tasks:
    - name: Wait for API server to be ready on first node
      ansible.builtin.wait_for:
        host: "{{ inventory_hostname }}"
        port: "{{ control_plane_port | default('6443') }}"
        timeout: 300
      delegate_to: localhost
    - name: Wait for first control plane node to be ready
      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} wait --for=condition=ready node {{ inventory_hostname }} --timeout=300s
      delegate_to: localhost
    - name: Verify control plane status
      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} get nodes -o wide
      delegate_to: localhost
      register: node_status
    - name: Display cluster status
      ansible.builtin.debug:
        msg: "{{ node_status.stdout_lines }}"
 - name: 5. Join additional control plane nodes
  hosts: fastpass_control_plane[1:]
  become: true
  gather_facts: false
  roles:
    - role: fastpass-control-plane-join
 - name: 6. Wait for all control plane nodes to be ready
  hosts: fastpass_control_plane[0]
  become: false
  gather_facts: false
  environment:
    KUBECONFIG: "{{ kubeconfig_path }}"
  tasks:
    - name: Wait for all control plane nodes to be ready
      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} wait --for=condition=ready node --selector=node-role.kubernetes.io/control-plane --timeout=600s
      delegate_to: localhost
    - name: Display all control plane nodes
      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} get nodes --selector=node-role.kubernetes.io/control-plane -o wide
      delegate_to: localhost
      register: control_plane_status
    - name: Show control plane status
      ansible.builtin.debug:
        msg: "{{ control_plane_status.stdout_lines }}"
 - name: 7. Join worker nodes
  hosts: fastpass_workers
  become: true
  gather_facts: false
  roles:
    - role: fastpass-workers
 - name: 8. Final cluster validation
  hosts: fastpass_control_plane[0]
  become: false
  gather_facts: false
  environment:
    KUBECONFIG: "{{ kubeconfig_path }}"
  tasks:
    - name: Wait for all nodes to be ready
      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} wait --for=condition=ready node --all --timeout=300s
      delegate_to: localhost
    - name: Verify all pods are running
      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} get pods --all-namespaces
      delegate_to: localhost
      register: pod_status
    - name: Display final cluster status
      ansible.builtin.debug:
        msg: |
          ========================================
          FastPass Kubernetes Cluster Status
          ========================================
          {{ pod_status.stdout }}
          ========================================
    - name: Show final node status
      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} get nodes -o wide
      delegate_to: localhost
      register: final_node_status
    - name: Display final node status
      ansible.builtin.debug:
        msg: "{{ final_node_status.stdout_lines }}"
    - name: Test DNS-based control plane endpoint
      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} cluster-info
      delegate_to: localhost
      register: cluster_info
    - name: Display cluster info
      ansible.builtin.debug:
        msg: "{{ cluster_info.stdout_lines }}"