Merge branch 'main' of gitea.mk-labs.cloud:rblundon/homelab

added claude role
same for open-webui
2026-10-21 11:17:17 -05:00 · 2026-10-21 11:17:14 -05:00 · 2026-10-20 23:16:10 -05:00 · 2026-10-20 23:13:51 -05:00 · 2026-10-21 03:55:57 +00:00 · 2026-10-20 14:23:27 -05:00
1015 changed files with 61151 additions and 12220 deletions
@@ -1,15 +1,9 @@
-# macOS system files
+# macOS
 .DS_Store
 .AppleDouble
 .LSOverride
 # Icon must end with two \r
 Icon
 # Thumbnails
 ._*
 # Files that might appear in the root of a volume
 .DocumentRevisions-V100
 .fseventsd
 .Spotlight-V100
@@ -17,36 +11,53 @@ Icon
 .Trashes
 .VolumeIcon.icns
 .com.apple.timemachine.donotpresent
 # Directories potentially created on remote AFP share
 .AppleDB
 .AppleDesktop
 Network Trash Folder
 Temporary Items
 .apdisk
 # VS Code
 .vscode/
 # macOS metadata
 *.icloud
-# Terraform
+# IDE
-terraform/
+.vscode/
 .idea/
 .kiro/
 # Terraform (track .tf files, ignore state and runtime)
 .terraform/
 *.tfstate
 *.tfstate.backup
-
+# *.tfvars
-# Crash log files
+!*.tfvars.example
 !*.pkrvars.hcl
 crash.log
 # Ignore override files as they are usually used to override resources locally
 *.override.tf
 *.override.tf.json
 # Ignore CLI configuration files
 .terraformrc
-.terraformrc.json
+terraform.rc
-# Ignore secrets
+# Ansible
-doppler-token.yaml
+*.retry
 ansible/vault/*.vault
 # Packer
 packer_cache/
 # Secrets
 *.pem
 *.key
 # Environment files with secrets
 boilerplates/**/.env
 # Local paths
 ~/
 # Misc
 doppler-token.yaml
 # talhelper generated machine configs — contain secrets, never commit
 talos/talhelper/clusterconfig/
 # talenv.yaml plaintext — only commit the SOPS-encrypted version
 talos/talhelper/talenv.yaml
 !talos/talhelper/talenv.sops.yaml
@@ -1,209 +0,0 @@
 # Design Document
 ## Overview
 The AppleScript Calendar Sync system is designed as a standalone AppleScript application that synchronizes calendar events between two calendar accounts for the current day only. The system uses macOS Calendar app's AppleScript interface to read events from a source calendar and mirror them to a destination calendar, including removal of events that no longer exist in the source.
 ## Architecture
 The system follows a simple pipeline architecture:
 ```
 [Source Calendar] → [Event Reader] → [Event Processor] → [Destination Calendar]
                                          ↓
                                   [Duplicate Detector]
                                          ↓
                                   [Cleanup Manager]
 ```
 ### Core Components
 1. **Calendar Manager**: Handles calendar account and calendar selection/validation
 2. **Event Reader**: Retrieves events from source calendar for current day
 3. **Event Processor**: Processes and transforms events for destination calendar
 4. **Sync Engine**: Coordinates the synchronization process including cleanup
 5. **Logger**: Provides user feedback and error reporting
 ## Components and Interfaces
 ### Calendar Manager
 ```applescript
 -- Validates and retrieves calendar references
 on getCalendar(accountName, calendarName)
 on validateCalendarAccess(calendar)
 on listAvailableCalendars()
 ```
 **Responsibilities:**
 - Validate calendar account and calendar names exist
 - Return calendar object references for AppleScript operations
 - Handle calendar access permissions and errors
 ### Event Reader
 ```applescript
 -- Reads events from source calendar for current day
 on getEventsForToday(sourceCalendar)
 on parseEventProperties(event)
 ```
 **Responsibilities:**
 - Query source calendar for events occurring on current date
 - Extract event properties (title, start time, end time, description, etc.)
 - Handle different event types (all-day, timed, recurring)
 ### Event Processor
 ```applescript
 -- Processes events for destination calendar
 on createEventInDestination(eventData, destinationCalendar)
 on updateExistingEvent(existingEvent, newEventData)
 on compareEvents(event1, event2)
 ```
 **Responsibilities:**
 - Create new events in destination calendar
 - Update modified events
 - Compare events for duplicate detection
 ### Sync Engine
 ```applescript
 -- Main synchronization coordinator
 on performSync(sourceCalendar, destinationCalendar)
 on cleanupRemovedEvents(sourceEvents, destinationEvents, destinationCalendar)
 on generateSyncReport(results)
 ```
 **Responsibilities:**
 - Coordinate the entire sync process
 - Manage event cleanup (removal of events not in source)
 - Generate sync reports and statistics
 ### Logger
 ```applescript
 -- Logging and user feedback
 on logMessage(message, level)
 on displayProgress(current, total)
 on showSyncSummary(summary)
 ```
 **Responsibilities:**
 - Display progress information to user
 - Log errors and warnings
 - Show final sync summary
 ## Data Models
 ### Event Data Structure
 ```applescript
 record EventData
    title: string
    startDate: date
    endDate: date
    isAllDay: boolean
    description: string
    location: string
    uid: string (for duplicate detection)
 end record
 ```
 ### Sync Result Structure
 ```applescript
 record SyncResult
    eventsCreated: integer
    eventsUpdated: integer
    eventsRemoved: integer
    eventsSkipped: integer
    errors: list of strings
 end record
 ```
 ## Error Handling
 ### Error Categories
 1. **Calendar Access Errors**: Invalid calendar names, permission issues
 2. **Event Processing Errors**: Malformed events, property access failures
 3. **Sync Operation Errors**: Network issues, calendar service unavailable
 ### Error Handling Strategy
 - Graceful degradation: Continue processing other events when individual events fail
 - Detailed error logging with specific error messages
 - User-friendly error reporting with suggested solutions
 - Rollback capability for critical failures
 ### Error Recovery
 ```applescript
 on handleCalendarError(errorMessage)
    -- Log error details
    -- Provide user-friendly error message
    -- Suggest corrective actions
 end handleCalendarError
 ```
 ## Testing Strategy
 ### Unit Testing Approach
 Since AppleScript has limited testing frameworks, testing will focus on:
 1. **Manual Testing Scenarios**:
   - Test with empty source calendar
   - Test with events spanning multiple days
   - Test with all-day events
   - Test with recurring events
   - Test calendar access errors
 2. **Integration Testing**:
   - Test full sync workflow with real calendar data
   - Test cleanup functionality (event removal)
   - Test duplicate detection accuracy
   - Test error handling with invalid inputs
 3. **Edge Case Testing**:
   - Very long event titles and descriptions
   - Events with special characters
   - Overlapping events
   - Events created/modified during sync
 ### Test Data Requirements
 - Test calendars with known event sets
 - Events with various properties (all-day, timed, recurring)
 - Events with special characters and long descriptions
 - Calendar accounts with different permission levels
 ## Implementation Considerations
 ### AppleScript Calendar Integration
 - Use `Calendar` application's AppleScript dictionary
 - Handle calendar app launch and focus management
 - Manage calendar selection and event creation timing
 ### Performance Optimization
 - Batch event operations where possible
 - Minimize calendar app UI interactions
 - Cache calendar references to avoid repeated lookups
 ### User Experience
 - Provide clear progress indicators
 - Show meaningful error messages
 - Allow user to cancel long-running operations
 - Display comprehensive sync results
 ### Security and Privacy
 - Request calendar access permissions appropriately
 - Handle sensitive calendar data securely
 - Provide clear information about what data is accessed
 ## Configuration Management
 ### User Configuration
 ```applescript
 -- Configuration properties
 property sourceAccountName : "Work Account"
 property sourceCalendarName : "Main Calendar"
 property destinationAccountName : "Personal Account"
 property destinationCalendarName : "Synced Events"
 property enableLogging : true
 ```
 ### Runtime Configuration
 - Allow users to modify calendar names without editing script
 - Provide configuration validation before sync starts
 - Save user preferences for repeated use
@@ -1,73 +0,0 @@
 # Requirements Document
 ## Introduction
 This feature enables automatic synchronization of calendar entries between two different calendar accounts using AppleScript. The system will copy events from a source calendar account to a destination calendar account, maintaining event details while avoiding duplicates and providing configurable sync options.
 ## Requirements
 ### Requirement 1
 **User Story:** As a user with multiple calendar accounts, I want to sync events from one account to another, so that I can maintain consistent scheduling across different calendar systems.
 #### Acceptance Criteria
 1. WHEN the sync script is executed THEN the system SHALL read all events from the specified source calendar
 2. WHEN events are found in the source calendar THEN the system SHALL copy them to the specified destination calendar
 3. WHEN copying events THEN the system SHALL preserve event title, date, time, duration, and description
 4. IF an event already exists in the destination calendar THEN the system SHALL skip creating a duplicate
 ### Requirement 2
 **User Story:** As a user, I want to configure which calendars to sync between, so that I can control the data flow between my accounts.
 #### Acceptance Criteria
 1. WHEN configuring the sync THEN the system SHALL allow selection of source calendar account and specific calendar
 2. WHEN configuring the sync THEN the system SHALL allow selection of destination calendar account and specific calendar
 3. WHEN invalid calendar names are provided THEN the system SHALL display an error message and exit gracefully
 4. WHEN calendar accounts are not accessible THEN the system SHALL provide clear error messaging
 ### Requirement 3
 **User Story:** As a user, I want the sync to focus on today's events only, so that I maintain current day synchronization without overwhelming the destination calendar.
 #### Acceptance Criteria
 1. WHEN the sync runs THEN the system SHALL only process events occurring on the current day
 2. WHEN determining current day THEN the system SHALL use the local system date
 3. WHEN events span multiple days THEN the system SHALL include events that start or occur on the current day
 4. WHEN no events exist for the current day THEN the system SHALL complete successfully with appropriate messaging
 ### Requirement 4
 **User Story:** As a user, I want the destination calendar to mirror the source calendar for the current day, so that removed events are also cleaned up automatically.
 #### Acceptance Criteria
 1. WHEN checking for duplicates THEN the system SHALL compare event title, start date, and start time
 2. WHEN a matching event is found in the destination THEN the system SHALL skip creating the duplicate
 3. WHEN an event exists in the destination but not in the source for the current day THEN the system SHALL remove it from the destination
 4. WHEN an event has been modified in the source THEN the system SHALL update the corresponding event in the destination
 ### Requirement 5
 **User Story:** As a user, I want to see progress and results of the sync operation, so that I can verify the synchronization was successful.
 #### Acceptance Criteria
 1. WHEN the sync starts THEN the system SHALL display the source and destination calendar information
 2. WHEN processing events THEN the system SHALL show progress indicators for each event processed
 3. WHEN the sync completes THEN the system SHALL display a summary of events copied, skipped, and any errors
 4. WHEN errors occur THEN the system SHALL log detailed error information for troubleshooting
 ### Requirement 6
 **User Story:** As a user, I want the sync to handle different event types and properties, so that all my calendar data is accurately transferred.
 #### Acceptance Criteria
 1. WHEN syncing events THEN the system SHALL handle all-day events correctly
 2. WHEN syncing events THEN the system SHALL preserve recurring event patterns when possible
 3. WHEN syncing events THEN the system SHALL handle events with attendees and meeting details
 4. WHEN event properties cannot be transferred THEN the system SHALL log which properties were skipped
@@ -1,125 +0,0 @@
 # Implementation Plan
 - [x] 1. Set up project structure and configuration
  - Create main AppleScript file with basic structure and configuration properties
  - Define configuration properties for source and destination calendars
  - Set up logging and error handling framework
  - _Requirements: 2.1, 2.2, 5.1_
 - [ ] 2. Implement Calendar Manager component
  - [ ] 2.1 Create calendar validation and access functions
    - Write functions to validate calendar account and calendar names exist
    - Implement calendar object retrieval with error handling
    - Create function to list available calendars for debugging
    - _Requirements: 2.1, 2.2, 2.3, 2.4_
  - [ ]* 2.2 Write unit tests for calendar access
    - Create test scenarios for invalid calendar names
    - Test calendar access permission handling
    - _Requirements: 2.3, 2.4_
 - [ ] 3. Implement Event Reader component
  - [ ] 3.1 Create current day event retrieval function
    - Write function to get today's date and create date range
    - Implement event query for current day from source calendar
    - Handle different event types (all-day, timed events)
    - _Requirements: 1.1, 3.1, 3.2, 3.3_
  - [ ] 3.2 Implement event property extraction
    - Create function to extract event title, dates, description, location
    - Handle event property access errors gracefully
    - Parse recurring events for current day occurrences
    - _Requirements: 1.3, 6.1, 6.2, 6.3_
  - [ ]* 3.3 Write tests for event reading functionality
    - Test event retrieval with various event types
    - Test property extraction accuracy
    - _Requirements: 1.1, 1.3_
 - [ ] 4. Implement Event Processor component
  - [ ] 4.1 Create event comparison and duplicate detection
    - Write function to compare events by title, start date, and start time
    - Implement duplicate detection logic for existing events
    - Handle event matching edge cases
    - _Requirements: 4.1, 4.2_
  - [ ] 4.2 Implement event creation and update functions
    - Create function to add new events to destination calendar
    - Implement event update functionality for modified events
    - Handle event creation errors and property limitations
    - _Requirements: 1.2, 1.3, 4.4, 6.4_
  - [ ]* 4.3 Write tests for event processing
    - Test duplicate detection accuracy
    - Test event creation with various properties
    - _Requirements: 4.1, 4.2, 4.4_
 - [ ] 5. Implement Sync Engine component
  - [ ] 5.1 Create main synchronization workflow
    - Implement the main sync function that coordinates all components
    - Add progress tracking and user feedback during sync
    - Handle sync workflow errors and recovery
    - _Requirements: 1.1, 1.2, 5.2, 5.3_
  - [ ] 5.2 Implement cleanup functionality for removed events
    - Create function to identify events in destination not in source
    - Implement event removal from destination calendar
    - Add safety checks to prevent accidental deletions
    - _Requirements: 4.3_
  - [ ] 5.3 Create sync reporting and statistics
    - Implement sync result tracking (created, updated, removed, skipped)
    - Create summary display function with detailed results
    - Add error reporting and logging
    - _Requirements: 5.3, 5.4_
  - [ ]* 5.4 Write integration tests for sync engine
    - Test complete sync workflow with test data
    - Test cleanup functionality accuracy
    - Test error handling and recovery
    - _Requirements: 4.3, 5.3_
 - [ ] 6. Implement Logger component
  - [ ] 6.1 Create logging and progress display functions
    - Implement message logging with different severity levels
    - Create progress indicator for sync operations
    - Add user-friendly error message formatting
    - _Requirements: 5.1, 5.2, 5.4_
  - [ ] 6.2 Implement sync summary display
    - Create formatted summary of sync results
    - Display statistics for events processed
    - Show any errors or warnings encountered
    - _Requirements: 5.3, 5.4_
 - [ ] 7. Integrate all components and create main script
  - [ ] 7.1 Wire together all components in main execution flow
    - Create main script entry point that calls all components
    - Implement proper error handling and user feedback flow
    - Add configuration validation before sync starts
    - _Requirements: 2.3, 2.4, 5.1_
  - [ ] 7.2 Add user interaction and configuration management
    - Implement user prompts for calendar selection if needed
    - Add configuration validation and error messaging
    - Create user-friendly script execution experience
    - _Requirements: 2.1, 2.2, 2.3, 2.4_
  - [ ]* 7.3 Create comprehensive end-to-end tests
    - Test complete sync workflow with real calendar data
    - Test all error scenarios and edge cases
    - Validate sync accuracy and cleanup functionality
    - _Requirements: 1.1, 1.2, 4.3, 5.3_
 - [ ] 8. Finalize and optimize the script
  - [ ] 8.1 Add performance optimizations and error recovery
    - Optimize calendar access and event processing performance
    - Add robust error recovery and rollback capabilities
    - Implement proper resource cleanup and calendar app management
    - _Requirements: 5.4_
  - [ ] 8.2 Create user documentation and usage instructions
    - Write clear instructions for script configuration and usage
    - Document calendar permission requirements
    - Create troubleshooting guide for common issues
    - _Requirements: 2.3, 2.4, 5.4_
@@ -1,165 +0,0 @@
 # Design Document
 ## Overview
 The `fastpass-additional-control-plane` Ansible role will enable the deployment of additional control plane nodes to an existing FastPass Kubernetes cluster. This role follows the established patterns from `fastpass-first-control-plane` but focuses on joining nodes to an already initialized cluster rather than initializing a new one. The role ensures high availability by creating redundant master nodes that can handle API requests, scheduling, and cluster management tasks.
 The key difference from the first control plane role is that this role will use `kubeadm join` with control plane flags instead of `kubeadm init`, and it will need to retrieve join tokens and certificate keys from the existing cluster.
 ## Architecture
 ### Role Structure
 The role will follow the standard Ansible role structure:
 ```
 ansible/playbooks/roles/fastpass-additional-control-plane/
 ├── defaults/
 │   └── main.yml
 ├── tasks/
 │   └── main.yml
 ├── handlers/
 │   └── main.yml (if needed)
 └── meta/
    └── main.yml (if needed)
 ```
 ### Integration Points
 - **DNS Management**: Uses the existing `dns-manager` role for consistent DNS record creation
 - **Kubeconfig Management**: Uses the existing `kubeconfig-manager` role for local kubeconfig setup
 - **Firewall Configuration**: Reuses firewall service definitions from the first control plane role
 - **Cluster Integration**: Coordinates with the first control plane node to obtain join credentials
 ### Dependencies
 - The first control plane node must be fully initialized and running
 - The `dns-manager` role must be available for DNS record creation
 - The `kubeconfig-manager` role must be available for kubeconfig setup
 - Required Kubernetes prerequisites must be installed on target nodes
 ## Components and Interfaces
 ### Main Task Flow
 1. **Pre-flight Checks**: Verify cluster readiness and node prerequisites
 2. **DNS Configuration**: Set up DNS records for the new control plane node
 3. **Firewall Configuration**: Open required ports for control plane services
 4. **Kubelet Configuration**: Create initial kubelet configuration
 5. **Join Token Retrieval**: Get join token and certificate key from first control plane
 6. **Cluster Join**: Execute kubeadm join with control plane flags
 7. **Service Management**: Ensure kubelet is enabled and running
 8. **Kubeconfig Setup**: Configure local kubeconfig access
 9. **Verification**: Validate successful cluster join
 ### Key Variables
 - `cluster_name`: Name of the Kubernetes cluster
 - `ip_address`: IP address of the current control plane node
 - `first_control_plane_host`: Hostname/IP of the first control plane node
 - `kubernetes_services_control_plane`: List of firewall services to open
 - `join_token_ttl`: TTL for join tokens (default: 24h)
 - `certificate_key_ttl`: TTL for certificate keys (default: 2h)
 ### External Role Interfaces
 - **dns-manager**: Provides DNS record creation with `host_name` variable
 - **kubeconfig-manager**: Handles kubeconfig merging with `cluster_name` variable
 - **First Control Plane**: Source for join tokens and certificate keys
 ## Data Models
 ### Join Credentials Structure
 ```yaml
 join_credentials:
  token: "abcdef.1234567890abcdef"
  discovery_token_ca_cert_hash: "sha256:..."
  certificate_key: "..."
  api_server_endpoint: "cluster-name:6443"
 ```
 ### Firewall Services
 ```yaml
 kubernetes_services_control_plane:
  - kubernetes_API      # Port 6443
  - etcd               # Ports 2379-2380
  - kubelet            # Port 10250
  - kube-scheduler     # Port 10259
  - kube-controller-manager  # Port 10257
 ```
 ### Node Status Tracking
 ```yaml
 node_status:
  joined: false
  kubelet_running: false
  dns_configured: false
  kubeconfig_ready: false
 ```
 ## Error Handling
 ### Join Token Management
 - **Token Expiration**: Automatically generate new tokens if existing ones are expired
 - **Certificate Key Rotation**: Handle certificate key expiration gracefully
 - **Network Connectivity**: Retry join operations with exponential backoff
 - **API Server Availability**: Wait for API server readiness before attempting join
 ### Idempotency Checks
 - **Already Joined Nodes**: Skip join process if node is already part of the cluster
 - **Existing Configuration**: Preserve existing kubelet configuration if valid
 - **DNS Records**: Update existing DNS records instead of creating duplicates
 - **Service Status**: Only restart services if configuration changes
 ### Failure Recovery
 - **Partial Join Failures**: Clean up partial configurations and retry
 - **Network Issues**: Provide clear error messages for connectivity problems
 - **Permission Errors**: Validate sudo/root access before attempting operations
 - **Resource Constraints**: Check system resources before proceeding
 ## Testing Strategy
 ### Unit Testing Approach
 - **Task Validation**: Test individual tasks with mock data
 - **Variable Validation**: Ensure required variables are properly defined
 - **Conditional Logic**: Test all conditional branches in tasks
 - **Error Scenarios**: Validate error handling for common failure cases
 ### Integration Testing
 - **Multi-Node Clusters**: Test with 3 and 5 control plane node configurations
 - **Network Scenarios**: Test across different network topologies
 - **OS Compatibility**: Validate on supported operating systems (Ubuntu/Debian)
 - **Version Compatibility**: Test with different Kubernetes versions
 ### Validation Checks
 - **Cluster Health**: Verify all control plane nodes are healthy after join
 - **API Availability**: Confirm API server is accessible from all nodes
 - **Etcd Cluster**: Validate etcd cluster membership and health
 - **Scheduling**: Test pod scheduling across all control plane nodes
 - **Failover**: Verify cluster continues operating if one control plane fails
 ### Test Scenarios
 1. **Fresh Join**: Join additional control plane to newly created cluster
 2. **Existing Cluster**: Add control plane to cluster with existing workloads
 3. **Network Partitions**: Test behavior during temporary network issues
 4. **Token Expiration**: Handle expired join tokens gracefully
 5. **Retry Operations**: Validate retry logic for transient failures
 ## Implementation Considerations
 ### Security
 - **Token Security**: Ensure join tokens are handled securely and not logged
 - **Certificate Management**: Properly manage and rotate certificate keys
 - **Network Security**: Validate firewall rules don't expose unnecessary ports
 - **Access Control**: Ensure proper RBAC is maintained after node joins
 ### Performance
 - **Parallel Execution**: Support joining multiple control plane nodes simultaneously
 - **Resource Usage**: Monitor CPU and memory usage during join process
 - **Network Bandwidth**: Optimize data transfer during cluster join
 - **Startup Time**: Minimize time to achieve cluster readiness
 ### Monitoring and Observability
 - **Join Progress**: Provide clear progress indicators during join process
 - **Health Checks**: Implement comprehensive health validation
 - **Logging**: Ensure adequate logging for troubleshooting
 - **Metrics**: Expose relevant metrics for monitoring cluster growth
 ### Compatibility
 - **Kubernetes Versions**: Support current and previous Kubernetes versions
 - **Operating Systems**: Maintain compatibility with Ubuntu and Debian
 - **Container Runtimes**: Work with containerd runtime configuration
 - **Network Plugins**: Compatible with Flannel CNI configuration
@@ -1,77 +0,0 @@
 # Requirements Document
 ## Introduction
 This feature involves creating an Ansible role called `fastpass-additional-control-plane` that will deploy additional control plane nodes to an existing FastPass Kubernetes cluster. The role will follow the same pattern as the existing `fastpass-first-control-plane` role but will focus on joining nodes to an already initialized cluster rather than initializing a new cluster. This ensures high availability for the Kubernetes control plane by adding redundant master nodes.
 ## Requirements
 ### Requirement 1
 **User Story:** As a DevOps engineer, I want to deploy additional control plane nodes to my FastPass Kubernetes cluster, so that I can achieve high availability and fault tolerance for the cluster control plane.
 #### Acceptance Criteria
 1. WHEN the role is executed on a node THEN the system SHALL join the node to the existing Kubernetes cluster as a control plane node
 2. WHEN the role runs THEN the system SHALL configure the necessary firewall rules for control plane services
 3. WHEN the role executes THEN the system SHALL ensure the kubelet service is properly configured and running
 4. WHEN joining the cluster THEN the system SHALL use the correct join token and certificate key from the first control plane node
 5. WHEN the role completes THEN the system SHALL verify the node has successfully joined as a control plane node
 ### Requirement 2
 **User Story:** As a system administrator, I want the additional control plane role to follow the same patterns as the first control plane role, so that the codebase remains consistent and maintainable.
 #### Acceptance Criteria
 1. WHEN the role is created THEN the system SHALL follow the same directory structure as fastpass-first-control-plane
 2. WHEN the role is implemented THEN the system SHALL use similar variable naming conventions and task organization
 3. WHEN the role runs THEN the system SHALL include proper error handling and idempotency checks
 4. WHEN the role executes THEN the system SHALL use the same firewall service definitions as the first control plane role
 5. WHEN the role is documented THEN the system SHALL include proper metadata headers with author, version, and description
 ### Requirement 3
 **User Story:** As a cluster operator, I want the additional control plane nodes to have proper DNS configuration, so that they can be reached by their cluster names and participate in load balancing.
 #### Acceptance Criteria
 1. WHEN the role runs THEN the system SHALL configure DNS records for the additional control plane nodes
 2. WHEN DNS is configured THEN the system SHALL use the dns-manager role for consistency
 3. WHEN the role executes THEN the system SHALL ensure the node can resolve the cluster endpoint
 4. WHEN DNS setup completes THEN the system SHALL verify connectivity to the cluster API endpoint
 ### Requirement 4
 **User Story:** As a DevOps engineer, I want the role to handle kubeconfig management for additional control plane nodes, so that I can manage the cluster from any control plane node.
 #### Acceptance Criteria
 1. WHEN the role completes THEN the system SHALL configure kubeconfig for the new control plane node
 2. WHEN kubeconfig is set up THEN the system SHALL use the kubeconfig-manager role for consistency
 3. WHEN the role runs THEN the system SHALL ensure proper permissions are set on kubeconfig files
 4. WHEN kubeconfig is configured THEN the system SHALL verify kubectl access works from the new node
 ### Requirement 5
 **User Story:** As a system administrator, I want the role to be idempotent and handle edge cases, so that I can run it multiple times safely without causing issues.
 #### Acceptance Criteria
 1. WHEN the role is run multiple times THEN the system SHALL not attempt to rejoin an already joined node
 2. WHEN a node is already part of the cluster THEN the system SHALL skip the join process gracefully
 3. WHEN the role encounters errors THEN the system SHALL provide clear error messages and fail gracefully
 4. WHEN prerequisites are missing THEN the system SHALL report what needs to be configured first
 5. WHEN the role runs THEN the system SHALL validate that required variables are defined
 ### Requirement 6
 **User Story:** As a cluster administrator, I want the role to integrate seamlessly with the existing FastPass deployment workflow, so that it can be used in the 4-step deployment process.
 #### Acceptance Criteria
 1. WHEN the role is created THEN the system SHALL be compatible with the deploy-fastpass-4step.yml playbook
 2. WHEN the role runs THEN the system SHALL work with the fastpass_control_plane[1:] host group
 3. WHEN integrated THEN the system SHALL not interfere with the first control plane initialization
 4. WHEN the role executes THEN the system SHALL depend on the first control plane node being ready
 5. WHEN deployment completes THEN the system SHALL allow worker nodes to join the cluster successfully
@@ -1,118 +0,0 @@
 # Implementation Plan
 - [ ] 1. Create role directory structure and basic configuration
  - Create the fastpass-additional-control-plane role directory structure
  - Set up defaults/main.yml with required variables and firewall services
  - Create meta/main.yml with role metadata and dependencies
  - _Requirements: 2.1, 2.2, 2.4_
 - [ ] 2. Implement join token and certificate key retrieval
  - [ ] 2.1 Create tasks to generate new join tokens from first control plane
    - Write Ansible tasks to execute kubeadm token create on first control plane node
    - Implement token validation and expiration checking
    - Add error handling for token generation failures
    - _Requirements: 1.4, 5.4_
  - [ ] 2.2 Implement certificate key retrieval and management
    - Create tasks to upload and retrieve certificate keys from first control plane
    - Add certificate key expiration handling and rotation
    - Implement secure handling of certificate keys in variables
    - _Requirements: 1.4, 5.1_
  - [ ] 2.3 Create discovery token CA certificate hash retrieval
    - Write tasks to extract CA certificate hash from first control plane
    - Implement validation of certificate hash format
    - Add error handling for certificate retrieval failures
    - _Requirements: 1.4, 5.4_
 - [ ] 3. Implement DNS configuration and firewall setup
  - [ ] 3.1 Configure DNS records for additional control plane nodes
    - Integrate dns-manager role for consistent DNS record creation
    - Pass appropriate host_name variable to dns-manager
    - Add DNS propagation wait and validation
    - _Requirements: 3.1, 3.2, 3.4_
  - [ ] 3.2 Set up firewall rules for control plane services
    - Reuse kubernetes_services_control_plane from defaults
    - Implement UFW firewall rule creation for Debian/Ubuntu systems
    - Add conditional logic for different operating systems
    - _Requirements: 1.2, 2.4_
 - [ ] 4. Implement kubelet configuration and cluster join
  - [ ] 4.1 Create initial kubelet configuration
    - Write kubelet config.yaml with systemd cgroup driver
    - Set containerd socket endpoint configuration
    - Ensure proper file permissions and ownership
    - _Requirements: 1.3, 2.3_
  - [ ] 4.2 Execute kubeadm join for control plane
    - Implement kubeadm join command with control-plane flag
    - Use retrieved join token, certificate key, and CA cert hash
    - Add proper command argument construction and validation
    - Include idempotency checks to prevent duplicate joins
    - _Requirements: 1.1, 1.4, 5.1, 5.2_
  - [ ] 4.3 Ensure kubelet service management
    - Enable and start kubelet systemd service
    - Add service status validation and error handling
    - Implement service restart logic if needed
    - _Requirements: 1.3, 1.5_
 - [ ] 5. Implement kubeconfig management and validation
  - [ ] 5.1 Configure kubeconfig for additional control plane nodes
    - Integrate kubeconfig-manager role for consistent configuration
    - Pass cluster_name variable to kubeconfig-manager
    - Ensure proper kubeconfig merging with existing configurations
    - _Requirements: 4.1, 4.2, 4.3_
  - [ ] 5.2 Implement cluster join validation
    - Create tasks to verify node successfully joined as control plane
    - Add kubectl commands to check node status and roles
    - Implement cluster health validation checks
    - _Requirements: 1.5, 4.4_
 - [ ] 6. Add comprehensive error handling and idempotency
  - [ ] 6.1 Implement pre-flight validation checks
    - Check if node is already joined to cluster
    - Validate required variables are defined
    - Verify first control plane node accessibility
    - Add system resource and prerequisite checks
    - _Requirements: 5.1, 5.2, 5.4, 5.5_
  - [ ] 6.2 Add retry logic and failure recovery
    - Implement retry mechanisms for transient failures
    - Add exponential backoff for network-related operations
    - Create cleanup tasks for partial join failures
    - _Requirements: 5.3, 5.4_
 - [ ] 7. Integration with FastPass deployment workflow
  - [ ] 7.1 Ensure compatibility with deploy-fastpass-4step.yml
    - Verify role works with fastpass_control_plane[1:] host group
    - Test integration with existing playbook structure
    - Validate dependency on first control plane completion
    - _Requirements: 6.1, 6.2, 6.3, 6.4_
  - [ ] 7.2 Add proper task documentation and metadata
    - Include role header with author, version, and description
    - Add inline comments for complex task logic
    - Document required variables and their purposes
    - _Requirements: 2.2, 2.5_
 - [ ]* 8. Create comprehensive testing and validation
  - [ ]* 8.1 Write unit tests for individual tasks
    - Create test cases for token retrieval logic
    - Test kubeadm join command construction
    - Validate error handling scenarios
    - _Requirements: 1.1, 1.4, 5.1_
  - [ ]* 8.2 Implement integration tests
    - Test multi-node control plane deployment
    - Validate cluster health after additional nodes join
    - Test failover scenarios and cluster resilience
    - _Requirements: 1.5, 6.5_
  - [ ]* 8.3 Add validation scripts and health checks
    - Create scripts to verify cluster state after deployment
    - Implement automated health validation
    - Add performance and resource usage monitoring
    - _Requirements: 1.5, 4.4_
@@ -1,51 +0,0 @@
 # Requirements Document
 ## Introduction
 The FastPass Kubernetes cluster deployment currently has critical DNS and load balancing configuration issues that prevent proper high availability setup. While the cluster endpoint `fastpass.local.mk-labs.cloud` is defined in the group variables, the `cluster_vip` variable required by the DNS manager role is missing, and the Traefik load balancer configuration is commented out. This means the cluster endpoint cannot resolve properly and there's no load balancing for the control plane API. This feature will fix these configuration gaps to enable true HA functionality.
 ## Requirements
 ### Requirement 1: DNS CNAME Record for Load Balancer
 **User Story:** As a DevOps engineer, I want the DNS manager to create a CNAME record for the cluster endpoint pointing to the load balancer, so that the cluster endpoint resolves through the load balancer rather than directly to node IPs.
 #### Acceptance Criteria
 1. WHEN the dns-manager role is called for a load-balanced cluster THEN the system SHALL create a CNAME record instead of an A record
 2. WHEN the CNAME record is created THEN the system SHALL point fastpass.local.mk-labs.cloud to the traefik_server (lightning_lane.local.mk-labs.cloud)
 3. WHEN the DNS record type is determined THEN the system SHALL use CNAME for load-balanced endpoints and A records for direct node access
 4. WHEN DNS propagation occurs THEN the system SHALL verify that the CNAME resolution works correctly
 ### Requirement 2: Complete Traefik Integration
 **User Story:** As a DevOps engineer, I want the Traefik load balancer to be fully integrated with the FastPass deployment, so that the cluster VIP is properly load balanced across all control plane nodes.
 #### Acceptance Criteria
 1. WHEN the traefik-manager role is called THEN the system SHALL use the traefik_server variable (lightning_lane) as the target host
 2. WHEN Traefik configuration is generated THEN the system SHALL create proper TCP routing for the cluster endpoint to all control plane nodes
 3. WHEN the cluster VIP is accessed THEN the system SHALL distribute requests across space-mountain, big-thunder-mountain, and splash-mountain
 4. WHEN Traefik configuration is applied THEN the system SHALL reload the Traefik service to activate the new configuration
 ### Requirement 3: High Availability Validation
 **User Story:** As a DevOps engineer, I want to validate that the HA setup is working correctly, so that I can be confident the cluster will survive node failures.
 #### Acceptance Criteria
 1. WHEN the deployment completes THEN the system SHALL test connectivity to the cluster endpoint
 2. WHEN connectivity tests run THEN the system SHALL verify that the endpoint resolves through the CNAME to the load balancer
 3. WHEN load balancer tests run THEN the system SHALL verify that requests are being distributed across control plane nodes
 4. WHEN a control plane node is stopped THEN the system SHALL continue to serve API requests through the remaining nodes
 ### Requirement 4: Backward Compatibility
 **User Story:** As a DevOps engineer, I want the DNS fixes to be backward compatible with existing deployments, so that current clusters continue to function during the transition.
 #### Acceptance Criteria
 1. WHEN existing clusters are updated THEN the system SHALL not break existing DNS configurations
 2. WHEN new variables are introduced THEN the system SHALL provide sensible defaults for existing deployments
 3. WHEN the update is applied THEN the system SHALL preserve existing kubeconfig files and cluster access
 4. IF migration issues occur THEN the system SHALL provide rollback procedures and documentation
@@ -0,0 +1,3 @@
 creation_rules:
  - path_regex: talos/talhelper/talenv.yaml
    age: age1xkyuv8r8ce6lu3d64jfspz4e50k6pxlaxwmuplzrtnpfnnrnycaq6mfsrn
@@ -0,0 +1,160 @@
 # CouchDB erlangCookie Fix - Implementation Guide
 ## Summary
 **Problem**: CouchDB deployment fails because `erlangCookie` is missing from the ExternalSecret configuration.
 **Decision**: Externalize `erlangCookie` to 1Password (pragmatic approach)
 **Rationale**: 
 - ExternalSecret architecture requires ownership of the entire secret
 - Mixing externalized and chart-generated fields in the same secret is not supported
 - Single-node deployment makes erlangCookie rotation unnecessary
 - This is an acceptable deviation from the pure Harbor pattern given the architectural constraints
 ## Implementation Steps
 ### 1. Generate erlangCookie Value
 ```bash
 openssl rand -hex 20
 ```
 Example output: `f4e3c2b1a9d8e7f6c5b4a3d2e1f0a9b8c7d6e5f4`
 ### 2. Add to 1Password
 - **Vault**: `mk-labs`
 - **Item**: `couchdb`
 - **Field Name**: `erlang-cookie`
 - **Field Type**: password (concealed)
 - **Value**: `<paste generated value from step 1>`
 ### 3. Update ExternalSecret Configuration
 File: `cluster/applications/couchdb/externalsecret.yaml`
 ```yaml
 apiVersion: external-secrets.io/v1beta1
 kind: ExternalSecret
 metadata:
  name: couchdb-credentials
  namespace: couchdb
  labels:
    app.kubernetes.io/name: couchdb
    app.kubernetes.io/part-of: mk-labs
 spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: onepassword-connect
  target:
    name: couchdb-admin
    creationPolicy: Owner
    template:
      engineVersion: v2
      data:
        adminUsername: "admin"
        adminPassword: "{{ .adminPassword }}"
        cookieAuthSecret: "{{ .cookieAuthSecret }}"
        erlangCookie: "{{ .erlangCookie }}"  # ← ADD THIS LINE
  data:
    - secretKey: adminPassword
      remoteRef:
        key: couchdb
        property: admin-password
    - secretKey: cookieAuthSecret
      remoteRef:
        key: couchdb
        property: cookie-auth-secret
    - secretKey: erlangCookie  # ← ADD THIS BLOCK
      remoteRef:
        key: couchdb
        property: erlang-cookie
 ```
 ### 4. Update values.yaml Documentation (Optional)
 File: `cluster/applications/couchdb/values.yaml`
 Update the comment block at line 9-10:
 ```yaml
 # Admin credentials managed via ExternalSecret
 # See externalsecret.yaml for 1Password integration
 #
 # NOTE: erlangCookie is externalized to 1Password for architectural
 # simplicity (ExternalSecret ownership model). In a pure Harbor pattern,
 # this would be chart-generated, but single-node deployment makes this
 # acceptable. The erlangCookie is treated as an immutable infrastructure
 # secret (generate once, never rotate).
 createAdminSecret: false
 extraSecretName: "couchdb-admin"
 ```
 ### 5. Commit and Push
 ```bash
 cd ~/git/homelab
 git add cluster/applications/couchdb/externalsecret.yaml
 git add cluster/applications/couchdb/values.yaml  # if modified
 git commit -m "fix(couchdb): add erlangCookie to ExternalSecret from 1Password"
 git push origin main
 ```
 ### 6. Verify Deployment
 ```bash
 # Watch ExternalSecret sync
 kubectl get externalsecret -n couchdb couchdb-credentials -w
 # Wait for: SecretSynced
 # Verify secret created with all four keys
 kubectl get secret -n couchdb couchdb-admin -o yaml
 # Should contain: adminUsername, adminPassword, cookieAuthSecret, erlangCookie
 # Watch ArgoCD sync
 kubectl get application -n argocd couchdb -w
 # Wait for: Healthy/Synced
 # Watch pod startup
 kubectl get pods -n couchdb -w
 # Wait for: Running
 # Test CouchDB access
 kubectl port-forward -n couchdb svc/couchdb-svc-couchdb 5984:5984 &
 curl http://localhost:5984/
 # Expected: {"couchdb":"Welcome","version":"3.5.1"}
 ```
 ## Why Not Follow Harbor Pattern Exactly?
 **Harbor Pattern**: Only user-facing credentials externalized, internal secrets chart-generated.
 **CouchDB Constraint**: ExternalSecret uses `creationPolicy: Owner`, which takes full ownership of the target secret. This prevents the Helm chart from adding auto-generated fields to the same secret.
 **Options Considered**:
 1. ✅ **Externalize erlangCookie** (SELECTED) - Works with current architecture
 2. ❌ Chart auto-generation - Conflicts with ExternalSecret ownership
 3. ❌ Dual-secret approach - Requires Helm chart customization
 4. ❌ Disable ExternalSecret - Loses 1Password integration for admin password
 **Decision**: Pragmatic approach wins. erlangCookie is treated as an infrastructure secret (generate once, never rotate), which is acceptable for a single-node deployment.
 ## Secret Classification
 | Secret           | Type          | 1Password? | Rationale                          |
 |------------------|---------------|------------|------------------------------------|
 | adminUsername    | User-facing   | No*        | Static value, hardcoded in template |
 | adminPassword    | User-facing   | ✅ YES     | User login credential              |
 | cookieAuthSecret | Gray area     | ✅ YES     | Session security, periodic rotation |
 | erlangCookie     | Internal      | ✅ YES**   | Architectural constraint           |
 \* Hardcoded in ExternalSecret template (not fetched from 1Password)  
 \*\* Pragmatic deviation from Harbor pattern due to ExternalSecret architecture
 ## References
 - Full analysis: `/home/hermes/couchdb-erlangcookie-analysis.txt`
 - Harbor pattern: `/home/hermes/harbor-simplification-complete.txt`
 - CouchDB Helm chart: `apache/couchdb` v4.6.3
@@ -0,0 +1,702 @@
 # Account Dossier Interactive Dashboard — Design Document
 **Version:** 0.1 (MVP)  
 **Author:** Claude  
 **Date:** 2026-10-07  
 **Status:** Design Review (Pre-Prototype)
 ---
 ## 1. Overview & Purpose
 **What:** Interactive HTML/React dashboard that surfaces account intelligence for a single Pure Storage customer or prospect. Provides a "10,000 ft view" with drill-down to ground-level detail.
 **Why:** SEs need to move quickly from account snapshot → discovery prep → conversation context without context-switching between Sheets, notes, and browser tabs. One dashboard per customer becomes the source of reference for that account.
 **MVP Scope:** The 10 essential questions (List 1 + List 2 from account intelligence framework)
 **Future Scope:** Relationships, opportunities, financial modeling, industry trends, sentiment analysis, timeline views
 ---
 ## 2. User Personas & Use Cases
 ### Primary User
 **Ryan** (You, Enterprise SE)
 - Prepping for discovery calls
 - Hunting for competitive displacement opportunities
 - Planning expansion/renewal strategy
 - Sharing account context with team (future)
 ### Use Cases (in priority order, all equally weighted for MVP)
 | Use Case | What They Need | Where Dashboard Helps |
 |----------|---|---|
 | **Pre-call Prep (15 min before discovery)** | Quick account snapshot + key metrics + conversation hooks | Overview pane + drill-down to specific List items; see what changed since last call |
 | **Competitive Displacement Hunting** | Incumbent vendors + footprint + gaps + consolidation leverage | List 2 drill-down showing Pure coverage % + high-value data not on Pure; see EMC systems aging |
 | **Expansion/Renewal Planning** | Growth trajectory + budget cycle + stakeholder sentiment + upcoming initiatives | Opportunity tab (future) + timeline + stakeholder relationships; see refresh window + budget headroom |
 ### Secondary User (Future)
 **Account team** (Sales, Solution Architect, Customer Success)
 - Read-only view of account summary
 - Understand strategic context before customer calls
 ---
 ## 3. Architecture Overview
 ### 3.1 Data Flow Diagram
 ```
 ┌─────────────────────────────────────────────────────────────┐
 │  Per-Customer Data Sources (DRY: No Duplication)            │
 ├─────────────────────────────────────────────────────────────┤
 │                                                               │
 │  📊 Google Sheet (Structured Data)                           │
 │  ├─ Account tab (List 1 metrics)                             │
 │  ├─ Pure-Deployment tab (List 2 metrics)                    │
 │  ├─ Relationships tab (Contacts, sentiment, history)        │
 │  ├─ Interactions tab (Call/email log with sentiment)        │
 │  ├─ Financials tab (ARR, contracts, opportunities)         │
 │  ├─ Industry-Trends tab (Curated insights)                 │
 │  └─ Metadata tab (Sync status, ownership, sharing)         │
 │                                                               │
 │  📝 Obsidian Markdown File (Unstructured Context)           │
 │  ├─ Discovery notes (narrative, conversation context)      │
 │  ├─ Strategy (account themes, expansion hooks)             │
 │  ├─ Relationship notes (person-by-person context)          │
 │  └─ Internal flags (competitive threats, risks)            │
 │                                                               │
 └─────────────────────────────────────────────────────────────┘
                            ↓
                   (via API or manual sync)
                            ↓
 ┌─────────────────────────────────────────────────────────────┐
 │  Interactive Dashboard (HTML/React Artifact)                │
 │  Reads from both sources, displays in 3-pane layout         │
 │  No data copying; all reads from source                     │
 └─────────────────────────────────────────────────────────────┘
                            ↓
                   (displayed to user)
                            ↓
 ┌─────────────────────────────────────────────────────────────┐
 │  User Actions:                                               │
 │  • View account snapshot                                     │
 │  • Drill into List 1 / List 2 metrics                       │
 │  • Check stakeholder sentiment                              │
 │  • Review opportunity pipeline                              │
 │  • Log new interaction                                       │
 │  • Open full Sheets / Obsidian for detailed editing         │
 └─────────────────────────────────────────────────────────────┘
 ```
 ---
 ## 4. UI/UX Design
 ### 4.1 Layout: 3-Pane Architecture
 ```
 ┌──────────────────────────────────────────────────────────────────────┐
 │ HEADER: Account Name | Status | Vertical | Location | [Share] [Edit] │
 ├─────────────────────────────────────────────────────────────────────┤
 │                                                                       │
 │  LEFT PANE              │ CENTER PANE              │ RIGHT PANE      │
 │  (Overview)             │ (Details & Drill-down)   │ (Context)       │
 │                         │                          │                 │
 │  ┌─────────────────┐    │ ┌────────────────────┐   │ ┌────────────┐ │
 │  │ Account Health  │    │ │   Tab Bar:         │   │ │ Latest     │ │
 │  │ Scorecard       │    │ │ Overview | List1   │   │ │ Interactions   │
 │  │ ◆ 4 KPIs        │    │ │ List2    | Oppty   │   │ │            │ │
 │  │   (visual)      │    │ │                    │   │ │ ┌────────┐ │ │
 │  │                 │    │ │ ┌──────────────┐   │   │ │ Sarah   │ │ │
 │  ├─────────────────┤    │ │ │ Item 1 Card  │   │   │ │ Chen    │ │ │
 │  │ Key Dates       │    │ │ │ [Summary]    │   │   │ │ Call    │ │ │
 │  │ Timeline        │    │ │ │ [10K ft]     │   │   │ │ 2d ago  │ │ │
 │  │ • Refresh: 6mo  │    │ │ │ [Status]     │   │   │ │ ⭐⭐⭐  │ │ │
 │  │ • Budget: 3mo   │    │ │ └──────────────┘   │   │ │ (Drill) │ │ │
 │  │ • Contract: 8mo │    │ │                    │   │ │└────────┘ │ │
 │  │                 │    │ │ ┌──────────────┐   │   │ │           │ │
 │  ├─────────────────┤    │ │ │ Item 2 Card  │   │   │ ├──────────┤ │
 │  │ Actions         │    │ │ │ [Summary]    │   │   │ │ [Log new]│ │
 │  │ [QBR Prep]      │    │ │ │ [10K ft]     │   │   │ │           │ │
 │  │ [Edit Account]  │    │ │ │ [Status]     │   │   │ │ Related   │ │
 │  │ [Open Sheets]   │    │ │ └──────────────┘   │   │ │ Notes     │ │
 │  │ [View Notes]    │    │ │                    │   │ │ (from MD) │ │
 │  │ [Share]         │    │ │ .... (3 more)      │   │ │           │ │
 │  │                 │    │ │                    │   │ │ [View all]│ │
 │  └─────────────────┘    │ └────────────────────┘   │ └────────────┘ │
 │                         │                          │                 │
 └──────────────────────────────────────────────────────────────────────┘
 ```
 ### 4.2 Left Pane: Account Overview (10,000 ft)
 **Purpose:** At a glance, is this account healthy? What's the most critical thing to know?
 **Components:**
 #### Health Scorecard (4 KPIs)
 Visual status cards, each with:
 - **Metric name** + current value
 - **Trend indicator** (↑ up, ↓ down, → flat)
 - **Status badge** (green / yellow / red)
 - **Drillable** — click to zoom into that metric
 **KPIs (MVP):**
 1. **Pure Coverage** — % of total storage on Pure
   - Value: 30% | Trend: ↑ +5% YoY | Status: 🟡 Opportunity
   - Rationale: Shows expansion potential; top sales driver
 2. **Fleet Age** — Average age of Pure systems
   - Value: 4.2 yr | Trend: → flat | Status: 🟡 Refresh Window Approaching
   - Rationale: Indicates refresh opportunity timing
 3. **Expansion Opportunity Size** — $ value of consolidation potential
   - Value: $3.5M | Trend: ↑ Growing | Status: 🟢 Pipeline-Ready
   - Rationale: Deal size at a glance; guides priority
 4. **Engagement Momentum** — Recency of last interaction + champion sentiment
   - Value: 5 days | Trend: ↑ Active | Status: 🟢 Engaged
   - Rationale: Is this account warm or cold?
 #### Key Dates Timeline
 Visual timeline of critical dates:
 - **Refresh Date**: Next Pure system refresh
 - **Budget Cycle**: When capex decisions happen
 - **Contract Renewal**: When Pure contract is up
 - **Next Review**: When you should reconnect
 Format: Horizontal timeline or vertical list with "X days until" callouts.
 #### Action Buttons (Quick nav)
 - **QBR Prep** → Opens checklist template
 - **Edit Account** → Opens Sheets in new tab
 - **View Notes** → Opens Obsidian vault or embedded Markdown
 - **Share** → (Future) generates shareable summary link
 - **Log Interaction** → Quick-add form (bottom right pane)
 ---
 ### 4.3 Center Pane: Details & Drill-Down
 **Purpose:** Zoom from 10K ft into List 1 and List 2 metrics. See details, trends, and rationale.
 **Tab Structure:**
 #### Tab 1: Overview (Summary of all 10 items)
 Shows all 5 List 1 items + all 5 List 2 items as **cards in a grid**:
 - Card layout: Title | 10K ft summary | Status badge | Click to drill
 - Color-coded by item type (List 1 = blue accent, List 2 = green accent)
 - Scanning-friendly; can see account at a glance
 #### Tab 2: Account Profile (List 1 Deep Dive)
 Five expandable cards, one per List 1 item:
 **Card 1: Data Footprint & Workload Breakdown**
 ```
 ┌─ Data Footprint & Workload Breakdown ───────────────────┐
 │ 10K FT VIEW:                                             │
 │ 500TB current | +18% YoY | 2yr forecast: 700TB           │
 │ Workloads: 40% OLTP | 35% Analytics | 25% Unstructured  │
 │                                                           │
 │ [CLICK TO EXPAND ▼]                                      │
 │                                                           │
 │ EXPANDED VIEW:                                           │
 │ ┌─ Breakdown Table ──────────────────────────────────┐  │
 │ │ Tier       │ Current │ Growth │ 2yr Forecast │ Note  │
 │ │ OLTP       │ 200TB   │ +15%   │ 265TB        │ TX DB │
 │ │ Analytics  │ 175TB   │ +25%   │ 275TB        │ DW    │
 │ │ Unstructured│ 125TB   │ +12%   │ 160TB        │ Lake  │
 │ └────────────────────────────────────────────────────┘  │
 │                                                           │
 │ ┌─ Capacity Growth Chart ────────────────────────────┐  │
 │ │ [Line chart: 3yr history + 2yr forecast]           │  │
 │ │ Shows growth trajectory, where Pure could capture │  │
 │ └────────────────────────────────────────────────────┘  │
 │                                                           │
 │ CONTEXT FROM NOTES:                                      │
 │ "Customer mentioned big analytics push for FY27..."      │
 │                                                           │
 │ [Related Links: 10-K | Analyst Report]                  │
 └─────────────────────────────────────────────────────────┘
 ```
 **Card 2: Current Storage Vendors & Installed Infrastructure**
 ```
 ┌─ Current Storage Vendors ──────────────────────────────┐
 │ 10K FT VIEW:                                            │
 │ NetApp 50% | Pure 30% | Dell 20% (of 1200TB total)     │
 │ Consolidation Opportunity: HIGH                         │
 │                                                          │
 │ [CLICK TO EXPAND ▼]                                     │
 │                                                          │
 │ EXPANDED VIEW:                                          │
 │ ┌─ Vendor Breakdown (Pie Chart) ─────────────────────┐ │
 │ │ [Visual pie showing % by vendor]                   │ │
 │ │ NetApp = oldest systems, refresh due              │ │
 │ └────────────────────────────────────────────────────┘ │
 │                                                          │
 │ ┌─ Incumbent Systems Detail ────────────────────────┐  │
 │ │ Vendor: EMC Symmetrix DMX-4                       │  │
 │ │ Capacity: 200TB | Age: 6 years | Status: AGING   │  │
 │ │ Use Case: Core banking OLTP                       │  │
 │ │ 💡 Displacement Target: Move to Pure FlashArray   │  │
 │ └────────────────────────────────────────────────────┘ │
 │                                                          │
 │ COMPETITIVE CONTEXT:                                    │
 │ "This EMC system is at end-of-life; customer risk-     │
 │ averse but willing to consolidate if migration is      │
 │ proven. Reference customer opportunity."                │
 │                                                          │
 │ [Related: Competitive Intel | Displacement Playbook]    │
 └──────────────────────────────────────────────────────────┘
 ```
 **Cards 3-5:** Similar drill-down structure for Business Drivers, Architecture, Budget/Authority
 #### Tab 3: Pure Deployment (List 2 Deep Dive)
 Five cards showing Pure-specific metrics, one per List 2 item.
 **Card 1: Current Pure Installed Base**
 ```
 ┌─ Current Pure Installed Base ──────────────────────────┐
 │ 10K FT VIEW:                                            │
 │ 2 FlashArray systems (150TB) | Healthy | 4.2yr avg age │
 │ Headroom: 52TB | Refresh window: 18mo                   │
 │                                                          │
 │ [CLICK TO EXPAND ▼]                                     │
 │                                                          │
 │ EXPANDED VIEW:                                          │
 │ ┌─ Systems Inventory Table ──────────────────────────┐ │
 │ │ System | Type | Capacity | Util | Headroom | Age   │ │
 │ │ FA-P1  | FA   | 75TB     | 65%  | 26TB     | 5.1yr │ │
 │ │ FA-DR  | FA   | 75TB     | 40%  | 45TB     | 3.3yr │ │
 │ │                                  Refresh Needed    │ │
 │ └────────────────────────────────────────────────────┘ │
 │                                                          │
 │ ┌─ Utilization & Headroom Heatmap ──────────────────┐ │
 │ │ [Visual: System cards with fill % + headroom bar]  │ │
 │ │ FA-P1: ████████ 65% USED (26TB headroom)          │ │
 │ │ FA-DR: █████░░░ 40% USED (45TB headroom)          │ │
 │ └────────────────────────────────────────────────────┘ │
 │                                                          │
 │ EXPANSION IMPLICATION:                                  │
 │ "FA-P1 will need refresh/capacity add in 18 months;     │
 │ FA-DR has headroom for growth without additional spend" │
 │                                                          │
 │ [Related: Pure1 Dashboard | Refresh Planning Guide]     │
 └──────────────────────────────────────────────────────────┘
 ```
 **Cards 2-5:** Performance/Satisfaction, Coverage Gaps, Refresh Cycles, Stakeholders
 ---
 ### 4.4 Right Pane: Unstructured Context & Interactions
 **Purpose:** Connect quantitative data (left/center) to narrative context (right). See "why" behind the numbers.
 **Components:**
 #### Latest Interactions (Log)
 Shows most recent 3-5 interactions with sentiment:
 ```
 ┌─ Interactions ────────────────────────────────────────┐
 │ ⭐⭐⭐⭐⭐ Sarah Chen (VP Storage)                      │
 │   Call — 2026-10-02 · 5 days ago                       │
 │   "Discussed multi-site replication, pain with NetApp" │
 │   Status: Strong Advocate for Pure adoption            │
 │                                                         │
 │ ⭐⭐⭐ Mike Lopez (CIO)                                │
 │   Email — 2026-09-15 · 22 days ago                     │
 │   "Budget constraints; interested in OpEx models"      │
 │   Status: Neutral, cost-focused                        │
 │                                                         │
 │ ⭐⭐ Jennifer Park (CFO)                              │
 │   Meeting — 2026-08-20 · 47 days ago                   │
 │   "Positive on storage consolidation, capex tight"     │
 │   Status: Positive but budget-constrained              │
 │                                                         │
 │ [+ Log New Interaction]                                │
 └───────────────────────────────────────────────────────┘
 ```
 #### Related Notes (from Markdown)
 Embedded snippet from Obsidian Markdown file:
 ```
 ┌─ Discovery Notes ──────────────────────────────────────┐
 │ STRATEGY THEME: Consolidation play                     │
 │ Customer is post-acquisition; consolidating 3 data     │
 │ centers into 2. Big opportunity to move from multi-    │
 │ vendor (EMC + NetApp) to Pure standardization.         │
 │                                                         │
 │ RISK: Customer risk-averse on primary storage.         │
 │ LEVER: Success story at reference customer + DR copy. │
 │                                                         │
 │ NEXT STEP: Sarah is scheduled for Q4 arch review.      │
 │ Come with: Multi-site replication architecture,        │
 │ consolidation cost model.                              │
 │                                                         │
 │ [View Full Notes] [Edit in Obsidian]                   │
 └───────────────────────────────────────────────────────┘
 ```
 #### Quick Actions
 - **[Log Interaction]** → Popup form to add call/email/meeting
 - **[View All Notes]** → Opens Obsidian file
 - **[View Sheets]** → Opens Google Sheets
 - **[Share Account]** → (Future) generates link
 ---
 ### 4.5 Responsive Behavior & Interactivity
 #### Drill-Down Interactions
 - **Cards expand in place** — no modal/new page; content slides down
 - **Breadcrumb trail** — e.g., "Account > List 1 > Data Footprint" so you know where you are
 - **Collapse others** — When one card expands, others collapse (reduces noise)
 - **Smooth transitions** — Cards animate open/closed
 #### Status Badges & Visual Indicators
 | Status | Color | Meaning |
 |--------|-------|---------|
 | 🟢 Green | Good / On Track / Advocate | Positive signal |
 | 🟡 Yellow | Caution / Aging / Neutral | Attention needed |
 | 🔴 Red | Risk / Critical / Urgent | Action required |
 | ⚪ Gray | Unknown / TBD | Need more info |
 #### Data-Driven Highlights
 Conditional visual emphasis on metrics:
 - **Refresh window approaching** (>4yr age) → Highlighted in yellow
 - **Coverage gap opportunity** (low % Pure) → Highlighted in orange
 - **Budget cycle in next 90 days** → Underlined with "upcoming" tag
 - **Strong champion sentiment** → Starbursts/badge next to name
 ---
 ## 5. Component Breakdown & Data Mapping
 ### 5.1 Component Hierarchy
 ```
 DashboardApp
 ├── Header
 │   ├── AccountName
 │   ├── StatusBadge (Prospect/Customer)
 │   ├── MetadataBar (Vertical, Location, Last Updated)
 │   └── HeaderActions ([Share] [Edit] [Settings])
 │
 ├── MainLayout (3-pane)
 │   │
 │   ├── LeftPane (Overview)
 │   │   ├── HealthScorecard
 │   │   │   ├── KPICard (Pure Coverage)
 │   │   │   ├── KPICard (Fleet Age)
 │   │   │   ├── KPICard (Opportunity Size)
 │   │   │   └── KPICard (Engagement Momentum)
 │   │   ├── TimelineComponent
 │   │   │   ├── TimelineEvent (Refresh Date)
 │   │   │   ├── TimelineEvent (Budget Cycle)
 │   │   │   ├── TimelineEvent (Contract Renewal)
 │   │   │   └── TimelineEvent (Next Review)
 │   │   └── ActionButtons
 │   │       ├── Button (QBR Prep)
 │   │       ├── Button (Edit Account)
 │   │       ├── Button (View Notes)
 │   │       └── Button (Share)
 │   │
 │   ├── CenterPane (Details & Drill-Down)
 │   │   ├── TabBar
 │   │   │   ├── Tab (Overview)
 │   │   │   ├── Tab (List 1: Account Profile)
 │   │   │   ├── Tab (List 2: Pure Deployment)
 │   │   │   └── Tab (Opportunities — future)
 │   │   │
 │   │   └── TabContent (varies by tab)
 │   │       └── DetailsCard (repeating)
 │   │           ├── CardHeader (title + status)
 │   │           ├── CardSummary (10K ft view)
 │   │           ├── ExpandButton
 │   │           └── ExpandedContent (on click)
 │   │               ├── DetailTable / Chart
 │   │               ├── TrendVisualization
 │   │               └── RelatedContext
 │   │
 │   └── RightPane (Context)
 │       ├── InteractionLog
 │       │   └── InteractionItem (repeating)
 │       │       ├── Sentiment (stars)
 │       │       ├── Contact Info
 │       │       ├── Timestamp
 │       │       └── Summary
 │       ├── RelatedNotes
 │       │   └── NotesSnippet
 │       └── QuickActions
 │           ├── Button (Log Interaction)
 │           ├── Button (View All Notes)
 │           └── Button (View Sheets)
 ```
 ### 5.2 Data Source Mapping
 | Component | Data Source | Sheet Location | Notes |
 |-----------|---|---|---|
 | **Account Name** | Account tab | Name | |
 | **Status Badges** | Account tab | Status, Vertical, Location | |
 | **Health Scorecard** | Account + Pure-Deployment tabs | Multiple cells + formulas | Calculated KPIs |
 | **Timeline** | Account tab | Columns: Next Refresh Date, Budget Cycle, Contract End, Next Review | |
 | **List 1 Cards** | Account tab | Tabs: Item 1-5 data | Five expandable items |
 | **List 2 Cards** | Pure-Deployment tab | Tabs: Item 1-5 data | Five expandable items |
 | **Interaction Log** | Interactions tab | Contact, Type, Date, Sentiment, Summary | Sorted by date DESC |
 | **Related Notes** | Markdown file (Obsidian) | Fetched via API or embed | Text snippet + link |
 ---
 ## 6. Visual Design Principles
 ### Color Palette (Themeable)
 - **Accent Blue** (Primary): List 1 items, account overview
 - **Accent Green** (Secondary): List 2 items, Pure deployment
 - **Status Green** (Success): Healthy, positive, on-track
 - **Status Yellow** (Warning): Approaching refresh, caution needed
 - **Status Red** (Critical): Urgent action required
 - **Neutral Gray**: Unknown, TBD, secondary content
 - **Text Dark Gray** (Light mode) / **Text Light Gray** (Dark mode)
 - **Background**: Light/dark mode adaptive
 ### Typography
 - **Header**: Large, bold, account name stands out
 - **Tab titles**: Medium weight, clear tab distinction
 - **Card titles**: Semi-bold, clickable indicator (▼)
 - **Data values**: Monospace for numbers (easier to scan)
 - **Labels**: Smaller, muted color
 ### Spacing & Layout
 - **Panes**: Equal width (33% each) or responsive flex at smaller widths
 - **Cards**: Padding ~20px, margin ~16px between cards
 - **Drill-down**: Animated 200ms expand/collapse
 - **Mobile**: Stack panes vertically (future consideration)
 ---
 ## 7. Navigation & User Flows
 ### Flow 1: Pre-Call Prep (15 min before discovery)
 ```
 User lands on dashboard
    ↓
 Glances at Health Scorecard (30 sec) — "Pure coverage 30%, fleet aging, $3.5M upside"
    ↓
 Checks Key Dates Timeline (30 sec) — "Budget cycle in 3 mo, refresh in 18 mo"
    ↓
 Clicks "QBR Prep" action → Opens checklist template in new tab
    ↓
 Scrolls to Center pane, clicks "Current Vendors" card to see EMC footprint
    ↓
 Reads right pane: "Customer risk-averse on primary storage; use reference story"
    ↓
 Click [Log Interaction] → Pre-fills with today's call, ready to document post-call
 ```
 ### Flow 2: Competitive Displacement Hunt
 ```
 User is looking for displacement opportunities
    ↓
 Opens Account dashboard
    ↓
 Glances at Scorecard: "Pure coverage 30%" → Opportunity
    ↓
 Clicks "Data Footprint" card → Sees workload breakdown
    ↓
 Clicks "Current Vendors" card → Sees EMC system (200TB, 6yr old, primary OLTP)
    ↓
 Clicks "Coverage Gaps" card → Confirms: "EMC Symmetrix not on Pure yet"
    ↓
 Reads notes: "Customer risk-averse but willing to migrate if proven"
    ↓
 Action: Find reference customer, prepare migration playbook for next call
 ```
 ### Flow 3: Expansion Planning
 ```
 User reviewing account for renewal/expansion
    ↓
 Glances at Scorecard: Fleet age, Opportunity size, Engagement momentum
    ↓
 Clicks "Fleet Age" card → Sees FA-P1 refresh window in 18 months
    ↓
 Clicks "Refresh Cycles" card → Confirms capex cycle and window
    ↓
 Checks Stakeholder info in right pane → "Sarah (VP Storage) is champion, strong advocate"
    ↓
 Clicks "Opportunities" tab (future) → See pipeline: Analytics consolidation (Q2 2027, $1.5M)
    ↓
 Action: Schedule architecture review with Sarah for Q1; prepare proposal for analytics move
 ```
 ---
 ## 8. Data Sync & Freshness
 ### MVP Approach: Manual Sync
 - **Sheet → Dashboard**: Copy-paste JSON export or Google Sheets API call (daily/on-demand)
 - **Markdown → Dashboard**: Fetch from Obsidian API or GitHub if synced
 - **Freshness**: Dashboard is "point-in-time snapshot" at time of load
 - **Update frequency**: User refreshes page or manually syncs from edit screens
 ### Future: Live Sync
 - Dashboard queries Google Sheets API in real-time
 - Interactions logged directly to Sheets (no manual re-entry)
 - Markdown fetched from Obsidian Sync API
 - Auto-refresh KPIs every 5 minutes
 ---
 ## 9. Extensibility Points (Future Phases)
 ### Phase 2: Relationships & Interaction History
 - **Tab 4: Relationships** — Org chart, sentiment timeline, interaction history
 - **Interaction log** — Searchable, filterable by contact, type, sentiment
 - **Sentiment analysis** — Chart showing mood trend over time
 ### Phase 3: Opportunities & Pipeline
 - **Tab 5: Opportunities** — Consolidation roadmap, deal timeline, revenue forecast
 - **Milestone tracking** — Countdown to RFP, contract close, etc.
 ### Phase 4: Financial Modeling
 - **10-K integration** — Embed PDF link, key metrics extracted
 - **Opportunity forecasting** — If analytics consolidates, revenue impact
 - **Budget impact model** — Capex vs. OpEx trade-off calculator
 ### Phase 5: Industry & Competitive Context
 - **Industry trends** — Curated insights relevant to account
 - **Competitor tracking** — Win/loss intelligence, displacement playbooks
 - **Market context** — Industry growth rates, regulatory changes
 ---
 ## 10. Technical Architecture
 ### Technology Stack
 - **Frontend**: React (or vanilla JS if keeping simple)
 - **Styling**: Tailwind CSS (or custom CSS-in-JS)
 - **Data sources**: 
  - Google Sheets API (for Account, Pure-Deployment, Relationships, Financials tabs)
  - Markdown file (from Obsidian or GitHub)
 - **Deployment**: Claude artifact (publishable, shareable URL) or self-hosted on homelab
 - **Build**: Vite (fast dev) or Create React App
 ### API Integration (MVP)
 ```
 Frontend (React)
    ↓
    ├─→ Google Sheets API v4
    │   └─ Fetch: Account!A:Z, PureDeployment!A:Z, Relationships!A:Z, etc.
    │      Rate: On-demand or 5-min cache
    │
    └─→ Markdown fetch (via Obsidian API or raw GitHub/Drive link)
        └─ Fetch: Markdown content, parse for snippets
 ```
 ### State Management
 - **Client-side state**: React hooks (useState, useEffect)
 - **Server state**: Google Sheets is source of truth
 - **UI state**: Which tab active, which card expanded
 - **No local storage** (MVP): Always fresh from source
 ---
 ## 11. Accessibility & Usability
 ### Keyboard Navigation
 - Tab through cards, tabs, buttons
 - Enter/Space to expand cards
 - Escape to collapse all
 - Ctrl/Cmd+Click to open links in new tab
 ### Screen Readers
 - Semantic HTML (headers, nav, main, article)
 - ARIA labels on custom components
 - Heading hierarchy (H1 = account name, H2 = tabs, H3 = card titles)
 ### Color Contrast
 - All text meets WCAG AA standards
 - Status indicators have labels (not color alone)
 - Links underlined or clearly differentiated
 ### Mobile Responsiveness (Future)
 - Stack panes vertically on small screens
 - Touch-friendly button sizes (44px minimum)
 - Collapsible sections for readability
 ---
 ## 12. Success Metrics (MVP)
 | Metric | Target | How to Measure |
 |--------|--------|---|
 | **Time to account snapshot** | <2 min to full context | User time log |
 | **Drill-down effectiveness** | Find key metric in <30 sec | User task test |
 | **Information completeness** | All 10 items readily visible | Checklist |
 | **Visual clarity** | No confusion on next steps | User feedback |
 | **Engagement** | Used for all 3 use cases equally | Usage logs |
 ---
 ## 13. Known Constraints & Trade-offs
 | Constraint | Impact | Mitigation |
 |-----------|--------|---|
 | **Manual sync (MVP)** | Data can lag if Sheets updated but dashboard not refreshed | Clear "last updated" timestamp; button to manual sync |
 | **No real-time collaboration** | If account team edits Sheets while viewing dashboard, changes won't appear until reload | Future: implement live sync |
 | **Single-customer dashboard** | Can't compare metrics across accounts in MVP | Future: roll-up view with filtering |
 | **API rate limits (Sheets)** | Can't refresh dashboard every second | Cache on client; user triggers refresh manually |
 ---
 ## 14. Success Criteria for Prototype
 ✅ **Must Have (MVP)**
 - [ ] All 10 items (List 1 + List 2) visible and organized
 - [ ] 3-pane layout functioning (left/center/right)
 - [ ] Cards expand/collapse on click with animation
 - [ ] Health scorecard with 4 KPIs calculated
 - [ ] Timeline showing key dates
 - [ ] Sample data flowing in from mock Sheets
 - [ ] Navigation between tabs (Overview, List 1, List 2)
 - [ ] Status badges and visual indicators rendering correctly
 ⚡ **Should Have (Nice to Have)**
 - [ ] Trend indicators (↑ ↓ →) on KPI cards
 - [ ] Sample charts (pie, line) in expanded views
 - [ ] Markdown snippet embedded in right pane
 - [ ] Interaction log with sentiment stars
 - [ ] Dark mode toggle
 🎯 **Nice to Have (Future)**
 - [ ] Google Sheets API actually connected (MVP uses mock data)
 - [ ] Relationships tab structure visible
 - [ ] QBR Prep modal/checklist
 - [ ] Share functionality
 - [ ] Export to PDF
 ---
 ## 15. Design Review Checklist
 **Questions for Ryan before prototype:**
 1. ✅ Is the 3-pane layout clear? Does it support the 10K → ground zoom?
 2. ✅ Are the 4 KPIs the right ones, or would you change them?
 3. ✅ Does the drill-down interaction (click card → expand) feel right, or prefer modal?
 4. ✅ Should the right pane be scrollable independently, or scroll with center pane?
 5. ✅ Any major UX concerns with the navigation flow?
 6. ✅ Is the timeline component clear (horizontal bar vs. vertical list)?
 7. ✅ Color palette acceptable, or want different accent colors?
 8. ✅ Any tabs or components you'd add/remove for MVP?
 ---
 **Document Status:** Ready for Design Review & Feedback  
 **Next Step:** Prototype build (pending approval of this design)
@@ -0,0 +1,257 @@
 # Top-5 Account Intelligence Lists for Enterprise Storage SE
 **For:** Pre-Sales Systems Engineers at Pure Storage  
 **Focus Verticals:** Financial Services, Retail/Manufacturing  
 **Target Accounts:** Fortune 1000 / Global 2000
 ---
 ## LIST 1: ANY CUSTOMER OR PROSPECT
 ### First Discovery Call + Competitive Displacement + Expansion Foundation
 ---
 ### 1. **Data Footprint and Workload Breakdown**
 **What to capture:**
 - Total addressable storage capacity (current + 2-year projection)
 - Workload tier split: critical OLTP (databases, transactional), warm operational analytics (data warehouses, ML training sets), unstructured (data lakes, backups, archives)
 - Annual growth rate
 - Typical IO patterns (sequential vs. random, read/write mix)
 - Workload sources: core business apps, digital channels, analytics platforms, third-party integrations
 **Why it ranks #1:**
 This is your credibility foundation for the discovery call. Large enterprises in financial services and retail/manufacturing operate at scale—you need to know if they're managing 10PB or 500PB of storage before you can ask intelligent questions about consolidation, resilience, or performance. For competitive displacement, it tells you the total addressable market and which workloads are most valuable to move. For expansion, it shows you where growth is (analytics workloads often outpace transactional growth in these verticals).
 **Vertical differences:**
 - **Financial services:** Regulatory data tiers (Basel III, Dodd-Frank, GDPR-driven data governance), transaction volumes, archive-to-compliance mandates, and fraud detection datasets are outsized here.
 - **Retail/manufacturing:** Omnichannel transaction data, supply chain event data, and IoT/sensor streams create different growth patterns—often unstructured-heavy.
 **Where to get it:**
 Account team, analyst kickoff, customer IT strategy documents. If public: earnings calls (capex guidance), analyst reports on data center expansion. **Hard to verify fully from public sources—lean on the account team and discovery conversation.** Pure1 for existing customers (capacity trends, growth rates).
 ---
 ### 2. **Current Storage Vendor(s) and Installed Infrastructure**
 **What to capture:**
 - Primary vendor(s), total installed capacity per vendor, system age/generation
 - Deployment model (on-prem, cloud-hosted)
 - Vendor concentration (single-vendor, multi-vendor)
 - Capacity consolidation potential (old systems due refresh, overlapping vendors)
 - Competitive systems they've ruled out or are evaluating
 **Why it ranks #2:**
 This is the competitive map. In discovery, you need to know who you're up against to ask credible questions about their stack. For displacement, this is your roadmap—old NetApp or EMC systems are your opening; vendor sprawl is your consolidation story. For expansion, it shows where Pure is (or isn't) and where you can grow.
 **Applies to both verticals:**
 Regulatory and operational requirements often enforce heterogeneous stacks (legacy core systems, newer analytics platforms), creating multi-vendor complexity and Pure's consolidation angle.
 **Where to get it:**
 Account team, customer IT contacts, Gartner/IDC reports, job postings (hiring for specific platforms signals infrastructure). Pure's competitive intel teams (if available in your region). **Harder to get from public sources—interview in discovery.**
 ---
 ### 3. **Business Drivers and Strategic Imperatives**
 **What to capture:**
 - What's moving their infrastructure strategy in the next 12–24 months?
 - Digital transformation (cloud-first, API-led architecture)
 - M&A (data center consolidation, platform rationalization)
 - Regulatory change (data residency, compliance modernization)
 - Cost reduction (capex or opex pressure)
 - Performance demands (real-time analytics, AI/ML acceleration)
 - Geographic expansion (multi-site, disaster recovery)
 **Why it ranks #3:**
 This explains *why* they'll spend and *when*. For discovery, it tells you what your conversation should be anchored to—cost pressure is a different conversation than compliance urgency. For displacement, it reveals openings (legacy vendor can't support cloud migration; incumbent doesn't scale for analytics). For expansion, it shows where budget flows and growth cycles.
 **Vertical differences:**
 - **Financial services:** Regulatory change (Fed data residency mandates, CCAR/stress test infrastructure), fraud detection (AI/ML workloads), and open banking/fintech competition drive transformation.
 - **Retail/manufacturing:** Omnichannel unification, supply chain visibility, and sustainability reporting (ESG data) are primary drivers; cost pressure is often secondary.
 **Where to get it:**
 Earnings calls (CFO and CRO commentary), industry analyst reports, customer executives in discovery (CEOs and CFOs speak to this), account team (relationship intelligence). Regulatory announcements for financial services. **Public sources vary—direct ask is usually most productive.**
 ---
 ### 4. **Infrastructure Architecture and Resilience Model**
 **What to capture:**
 - Topology: single data center, active-active multi-site, hub-and-spoke?
 - Failover strategy (RPO/RTO targets, synchronous vs. asynchronous replication)
 - Cloud presence (public cloud, hybrid, private cloud)
 - Containerization and Kubernetes adoption
 - Storage tiers and tiering automation
 - Backup/archive strategy (separate systems, integrated)
 **Why it ranks #4:**
 This determines *which* Pure solutions fit and how. Knowing they're multi-site tells you Evergreen model and replication are critical; knowing they have Kubernetes tells you Portworx is on the menu; knowing they're chasing sub-second RTO tells you FlashArray's performance story matters. For discovery, it frames the conversation around resilience or performance. For displacement, it shows where Pure's architecture outperforms incumbent (e.g., Evergreen's native replication vs. bolted-on replication). For expansion, it guides where to add capacity.
 **Vertical differences:**
 - **Financial services:** Recovery requirements are often strict (RTO <1hr, RPO <15min); multi-site is standard; compliance mandates geographic separation.
 - **Retail/manufacturing:** Multi-site is common (stores, plants, distribution); RTO is important but often longer than financial services; hybrid cloud adoption is accelerating.
 **Where to get it:**
 Account team, technical RFI/RFQ documents, infrastructure diagrams (security-sensitive—may not be public). Discovery conversation with IT infrastructure/storage teams. **Hard from public sources.**
 ---
 ### 5. **Budget Cycle, Capital Allocation, and Buying Authority**
 **What to capture:**
 - Capex budget cycle and approval gates (who approves? quarterly, annual, rolling?)
 - Capex vs. opex preference (traditional capex, Evergreen-style subscription, OpEx-only mandate)
 - Budget envelope for infrastructure over next 2 years
 - Preferred buying patterns (competitive bid, existing vendor preference, tech refresh cycle)
 - Influencers and decision-makers (CIO, VP Storage, business unit leaders, CFO)
 **Why it ranks #5:**
 This is the "when and how to sell" lever. It's less about discovery credibility (though knowing budget is essential) and more about deal realism. For discovery, it tells you timing and buying authority to route follow-ups. For displacement, it tells you if capex is available or if you need to frame Evergreen as an alternative. For expansion, it reveals budget headroom and buying rhythm.
 **Applies to both verticals:**
 Financial services often have tighter capex scrutiny and longer approval cycles; retail/manufacturing may have faster refresh cycles but more budget constraints post-acquisition integration.
 **Where to get it:**
 Account team relationship notes, past deal history, 10-K/10-Q filings (capex guidance), analyst reports, budget forecasts shared in customer relationships. **Rarely fully public—account team and direct relationship intelligence are key.**
 ---
 ## LIST 2: EXISTING CUSTOMERS — PURE STORAGE DEPLOYMENT SPECIFICS
 ### Expansion + Retention + Competitive Defense
 ---
 ### 1. **Current Pure Installed Base, Deployment Pattern, and Capacity**
 **What to capture:**
 - FlashArray systems (models, capacity per system, tier across data centers or cloud regions)
 - FlashBlade deployments (scale-out analytics use cases, capacity)
 - Portworx deployments (if any; containerized workloads, environments)
 - Age of systems (when deployed; refresh cycle status)
 - Geographic footprint (on-prem across which data centers, any cloud-hosted Pure?)
 - Utilization and headroom (at capacity, growing into headroom, or sparse?)
 **Why it ranks #1:**
 You can't expand without knowing your baseline. For expansion, this is your starting point—you need to know what they own to identify the next sale (add capacity, new tier, new workload). For retention, it's your health baseline (old systems more likely to churn). For competitive defense, it shows Pure's footprint and where competitors might try to take share.
 **Key indicators for retention/expansion:**
 Systems older than 5 years are refresh candidates; sparse utilization may indicate underutilized tier or expansion opportunity; multi-site Pure deployments are higher-value accounts.
 **Where to get it:**
 **Pure1 (your best data source—capacity, performance, age, location),** internal Pure account data, customer IT/storage teams. Direct ask in customer reviews and strategic business reviews (QBR/SBR).
 ---
 ### 2. **Performance, Utilization, and Customer Satisfaction**
 **What to capture:**
 - Pure1 performance metrics (latency, IOPS achieved vs. provisioned, dedupe/compression ratios, capacity growth rate)
 - Storage team feedback (support experience, feature requests, known pain points)
 - Performance satisfaction (are performance SLAs being met?)
 - Operational efficiency gains (cost per GB, power/cooling reduction)
 - Customer sentiment (NPS or equivalent; any churn signals?)
 **Why it ranks #2:**
 This tells you expansion potential vs. churn risk. If they're unsatisfied, expansion is at risk (competitor threat, bargaining power). If they're thriving on Pure, expansion is easier (proven business case). For retention, this is your early warning system. For expansion, high satisfaction is your foundation for upsells (new workload, new tier, cloud-native).
 **Vertical differences:**
 - **Financial services:** Customers often have strict uptime/performance mandates.
 - **Retail/manufacturing:** Often cost-conscious and may trade performance for efficiency.
 **Where to get it:**
 **Pure1 (performance analytics),** NPS/CSAT surveys, QBR discussions, support ticket sentiment, sales/SE relationship notes. Direct ask: "How is Pure performing for you?"
 ---
 ### 3. **Data Footprint Not Yet on Pure / Coverage Gap Analysis**
 **What to capture:**
 - Percentage of their total storage on Pure vs. other vendors
 - Workload segments not on Pure (e.g., older OLTP databases on EMC, analytics on Hadoop/object storage, backups on separate vendor)
 - Highest-value data not on Pure (revenue-generating apps, critical analytics)
 - Consolidation opportunities (multi-vendor overlap, legacy systems ready to refresh)
 - Workloads Pure could address but hasn't yet (emerging use cases, cloud data, new business initiatives)
 **Why it ranks #3:**
 This is your expansion TAM within the account. For expansion, it shows where growth comes from (consolidation opportunity, new workload, tier migration). For competitive defense, it identifies where competitors will attack (the gap is their opening). For retention, it's leverage—you want to own more of their data.
 **Vertical differences:**
 - **Financial services:** Critical OLTP often stays on incumbent (lower risk perception); newer analytics and fraud detection data are Pure opportunities.
 - **Retail/manufacturing:** Supply chain/IoT data and unstructured analytics are often on non-Pure platforms; consolidation gains are high here.
 **Where to get it:**
 **Pure1 (shows only Pure capacity—you need to ask for total footprint),** customer IT (data inventory), discovery/expansion conversations. **Often not transparent from public sources—direct conversation essential.**
 ---
 ### 4. **Upcoming Infrastructure Initiatives and Technology Refresh Cycles**
 **What to capture:**
 - Planned data center consolidation or modernization
 - System refresh dates (when is their oldest Pure system past end-of-support? When's the next capex cycle?)
 - Cloud migration plans (moving workloads to cloud; implications for on-prem storage)
 - New workloads (AI/ML platforms, real-time analytics, new business lines)
 - Multi-site expansion or disaster recovery modernization
 **Why it ranks #4:**
 Timing is everything in expansion. Refresh cycles are your biggest lever (old systems = new capex = new opportunity). Cloud migration can cut Pure TAM or open Portworx opportunities. New workloads are expansion hooks. For expansion, this is your calendar. For retention, knowing refresh timing lets you plan defense early.
 **Applies to both verticals:**
 Financial services often have annual capex cycles tied to business planning; retail/manufacturing may refresh faster or slower depending on store/plant footprint changes.
 **Where to get it:**
 Customer IT roadmap (often multi-year), Pure account team, QBR discussions, earnings announcements (for large public customers, CapEx guidance), customer executives. Sometimes in customer's published IT strategy.
 ---
 ### 5. **Stakeholder Relationships, Expansion Readiness, and Competitive Threats**
 **What to capture:**
 - Key stakeholders (CIO, VP Storage, business unit leaders, security/compliance officers)
 - Champion(s) (who advocates for Pure?)
 - Relationship temperature (trust, satisfaction, barriers)
 - Competitive interest (have they mentioned other vendors? RFPs? Evaluation?)
 - Budget headroom (will they invest more in storage in next 12–24 months?)
 - Expansion appetite (are they open to moving more workloads to Pure, or focused on optimization?)
 **Why it ranks #5:**
 This is your "how to expand" lever. Knowing who the champion is tells you how to navigate. Knowing if they're under competitive attack tells you how urgent your defense is. Knowing budget appetite tells you if expansion is realistic. For expansion, this is your relationship risk/opportunity assessment. For retention, it's your early warning for churn or your pathway for growth.
 **Applies to both verticals:**
 In large enterprises, storage decisions often flow through business unit leaders (not just IT); understanding where Pure fits in their strategic priorities is key.
 **Where to get it:**
 Account team relationship notes, past deal history, QBR feedback, sales/SE interactions, industry/analyst events. Direct ask in customer conversations: "Where do you see us helping you grow?"
 ---
 ## Summary: Why This Order Matters
 **List 1** (any customer/prospect) moves from **foundational understanding** (what and who) → **context** (why and how) → **execution** (when and who decides). This sequence builds credibility in discovery while revealing competitive and expansion levers.
 **List 2** (existing customer Pure deployments) moves from **your current position** (what you have) → **relationship health** (how it's performing) → **opportunity size** (what you could have) → **timing** (when they'll buy) → **execution** (how to navigate the sale). This sequence is your expansion playbook.
 ### Key Callout
 In both lists, **items 1–3 should drive 80% of your discovery and account strategy effort**. Items 4–5 refine timing and approach. **Start here, then layer in the operational details** (support SLAs, feature adoption, roadmap alignment) as you deepen the relationship.
 ---
 ## Quick Reference: Source Hierarchy by Item
 | Item | Primary Source | Secondary Source | Public Availability |
 |------|---|---|---|
 | **List 1 #1** – Data Footprint | Account team, customer IT | Earnings calls, analyst reports | Moderate (public guidance varies) |
 | **List 1 #2** – Current Vendors | Account team, IT contacts | Job postings, Gartner reports | Low (often internal knowledge) |
 | **List 1 #3** – Business Drivers | Customer executives, analyst reports | Earnings calls, regulatory filings | Moderate (CEOs/CFOs speak to this) |
 | **List 1 #4** – Architecture | Technical RFI/RFQ, account team | Discovery conversation | Low (security-sensitive) |
 | **List 1 #5** – Budget/Authority | Account team, past deal history | 10-K/10-Q, customer relationships | Low (relationship intelligence) |
 | **List 2 #1** – Pure Installed Base | **Pure1** (best source), account data | Customer IT, QBR | Account-internal only |
 | **List 2 #2** – Performance/Satisfaction | **Pure1**, NPS surveys | Support tickets, QBR discussions | Account-internal only |
 | **List 2 #3** – Coverage Gaps | Customer IT inventory, discovery | Account team insights | Low (direct conversation required) |
 | **List 2 #4** – Refresh Cycles | Customer IT roadmap, account team | QBR, earnings announcements | Low-Moderate (customer-dependent) |
 | **List 2 #5** – Stakeholder/Competitive | Account team, sales history | QBR feedback, customer conversations | Account-internal only |
 ---
 **Use this as your go-to reference before discovery calls, account strategy reviews, and expansion planning.**
@@ -1,239 +1,102 @@
-# A Homelab based on Red Hat Technologies
+# mk-labs
-A comprehensive homelab for learning advanced concepts primarily based on Red Hat OpenShift.  In addition to providing services, it is an area to incorporate container technologies, and enterprise integration scenarios.
+Automated infrastructure provisioning and configuration for a personal homelab, built on GitOps practices with clear tool responsibility boundaries.
-This implementation is built on easily accessible consumer based hardware and will focus heavily on GitOps practices and automation will be used wherever possible.
+## Architecture
-## 🎯 Project Goals
+A single operator action — setting a VM record's status to **Staged** in NetBox — triggers a fully automated provisioning pipeline:
-
+```
- **Kubernetes cluster for applications**
+NetBox (webhook) → n8n (validate & orchestrate) → Terraform (create VM + DHCP)
- **IAM integration testing**
+                                                 → Ansible (OS config + DNS + status update)
 - **GitOps and automation workflows**
 ## 📋 Documentation (Priority Order)
 ### Phase 1: Foundation
 1. **[Overview & Hardware Allocation](./docs/01-overview.md)** - Complete architecture overview
 2. **[Networking Plan](./docs/02-networking.md)** - VLAN strategy and network design
 3. **[Core Services Setup](./docs/03-core-services.md)** *(TBD)* - Ansible, DNS, monitoring
 4. **[Storage Configuration](./docs/04-storage.md)** *(TBD)* - Synology, democratic-csi
 ### Phase 2: Proxmox Platform
 5. **[Proxmox Installation](./docs/11-vsphere-setup.md)** *(TBD)* - Dell 7050 virtualization
 6. **[VM Templates & Automation](./docs/13-vm-automation.md)** *(TBD)* - Template creation
 ### Phase 3: Infrastructure Services
 7. **[Recursive DNS]
 8. **[Authoritative DNS]
 9. **[Identity Management](./docs/10-acm-setup.md)** *(TBD)*
 10. **[Matchbox] (Depreciated?)
 ### Phase 4: OpenShift Clusters (Depreciated)
 8. **[OpenShift SNO + Worker](./docs/09-openshift-sno.md)** *(TBD)* - ACM Hub Cluster
 9. **[OpenShift Cluster](./docs/08-openshift-compact.md)** *(TBD)* - 3 Master/3 Worker node (production-like)
 10. **[ACM Configuration](./docs/10-acm-setup.md)** *(TBD)* - Multi-cluster management
 11. **[HCP Cluster](./docs/10-acm-setup.md)** *(TBD)* - External app cluster
 ### Phase 5: Kubernetes Container Platform
 5. **[Container Registry Setup](./docs/05-container-registry.md)** *(TBD)* - Harbor deployment
 6. **[Git Repository Setup](./docs/06-git-repository.md)** *(TBD)* - Gitea/GitLab on Synology
 7. **[Artifact Repository](./docs/07-artifact-repository.md)** *(TBD)* - Nexus/Artifactory
 ### Phase 6: Advanced Services
 14. **[Monitoring & Observability](./docs/14-monitoring.md)** *(TBD)* - Prometheus, Grafana, Splunk
 15. **[Security & Compliance](./docs/15-security.md)** *(TBD)* - ACS, certificates, auditing
 16. **[Backup & DR](./docs/16-backup-dr.md)** *(TBD)* - Backup strategies
 ## 🚀 Quick Start Deployment
 ### Prerequisites
 - Ubiquiti UDM Pro configured
 - Hardware powered and networked
 - Initial VLAN setup (see [networking plan](./docs/02-networking.md))
 ### Ansible Collections
 - ansible-galaxy collection install freeipa.ansible_freeipa
 ### Deployment Scripts
 ```bash
 source ~/venv-ansible/bin/activate
 # Phase 1: Foundation
 ./scripts/01-network-setup.sh
 ./scripts/02-core-services-deploy.sh
 # Phase 2: Container Platform  
 ./deployment/synology/docker-compose.yml  # Git repo, Harbor, Nexus
 ./scripts/03-storage-setup.sh
 # Phase 3: OpenShift
 ./deployment/openshift/compact-cluster/
 ./deployment/openshift/sno-cluster/
 # Phase 4: Proxmox
 ./deployment/proxmox/
 ```
-## 📁 Repository Structure
+| Tool | Host | IP | Responsibility |
 |------|------|----|---------------|
 | NetBox | fire-station | 10.1.71.102 | Source of truth — VM records, IP allocation, VLAN data |
 | n8n | tiki-room | 10.1.71.23 | Event orchestration, validation, pipeline sequencing |
 | Terraform | city-hall | 10.1.71.35 | Proxmox VM lifecycle, Unifi DHCP reservations |
 | Ansible / Semaphore | imagineering | 10.1.71.22 | OS configuration, DNS records, NetBox status updates |
 | Proxmox | fantasyland | 10.1.71.13 | Target hypervisor |
 All systems on the Server Trusted VLAN (10.1.71.0/24).
 ## Repository Structure
 ```
 homelab/
-├── README.md                    # This file
+├── ansible/
-├── docs/                        # Documentation (numbered by priority)
+│   ├── inventory/          # NetBox dynamic inventory + static
-│   ├── 01-overview.md
+│   ├── playbooks/          # Runnable playbooks (vm-provision, DNS, OS updates)
-│   ├── 02-networking.md
+│   ├── roles/              # vm-baseline, dns-manager, common, haproxy, n8n, observer, etc.
-│   └── ...
+│   ├── tasks/              # Shared includable task files
-├── ansible/                     # Directory structure for Ansible 
+│   ├── group_vars/         # Group variable definitions
-├── deployment/                  # Deployment 
+│   ├── host_vars/          # Per-host variable definitions
-configurations
+│   ├── templates/          # Jinja2 templates
-│   ├── synology/                # Docker Compose files for Synology
+│   └── ansible.cfg
-│   ├── k0s/                     # k0s cluster manifests
+│
-│   ├── openshift/               # OpenShift installation configs
+├── terraform/
-│   └── proxmox/                 # Proxmox automation
+│   ├── proxmox/vm/         # bpg/proxmox provider — VM creation from templates
-├── scripts/                     # Automation scripts
+│   ├── unifi/dhcp/         # Unifi provider — DHCP static reservations on UDM Pro
-└── .gitignore                   # Excludes sensitive data
+│   └── dns/                # DNS record management
 │
 ├── packer/
 │   ├── ubuntu-24.04/       # Ubuntu 24.04 VM template (small → xlarge-plus sizes)
 │   └── fedora-42/          # Fedora 42 VM template
 │
 ├── n8n/
 │   └── workflows/          # Exported n8n workflow JSON (vm-provisioning)
 │
 ├── netbox/
 │   └── initializers/       # Custom fields, VLANs, IP prefixes as code
 │
 └── docs/
    └── decisions/          # Architecture decision records
 ```
-## 🔧 Technology Stack
+## Pipeline Flow
-### Infrastructure
+| # | System | Action |
- **Networking**: Ubiquiti UDM Pro
+|---|--------|--------|
- **Storage**: Synology DS1621+, Ubiquiti UNAS Pro
+| 1 | NetBox | Operator sets VM status to Staged → webhook fires |
- **Compute**: 3x Minisforum TH60, 2x Minisforum MS01, 7x Dell 7050 SFF
+| 2 | n8n | Validates payload (hostname, IP, VLAN, template, proxmox_node) |
 | 3 | n8n → city-hall | SSH + `terraform apply` — creates VM on Proxmox |
 | 4 | n8n | Queries Proxmox API for MAC address |
 | 5 | n8n → NetBox | Writes MAC to VM interface record |
 | 6 | n8n → city-hall | SSH + `terraform apply` — creates DHCP reservation on UDM Pro |
 | 7 | n8n → imagineering | Triggers Ansible via Semaphore API |
 | 8 | Ansible | OS baseline, SSH hardening, Technitium DNS A record |
 | 9 | Ansible → NetBox | Sets VM status to Active |
-### Container Platforms
+On any failure, NetBox status is set to **Failed**. No auto-retry — operator investigates.
 - **k8s**: Core services cluster
 - **Proxmox**: VM workloads
 ### Core Services
 - **Container Registry**: Harbor
 - **Git Repository**: Gitea/GitLab CE
 - **Artifact Repository**: Nexus/Artifactory
 - **Monitoring**: Prometheus + Grafana
 - **Logging**: Splunk Enterprise
 - **DNS**: CoreDNS
 - **Load Balancing**: MetalLB + HAProxy
 ## 🔐 Security Notes
 - **No sensitive data** is stored in this repository
 - **Secrets management** via external-secrets-operator
 - **Certificate management** via cert-manager
 - **Network segmentation** via VLANs and firewall rules
 ## 🤝 Contributing
 This is a personal homelab project, but feel free to:
 - Submit issues for questions or suggestions
 - Fork for your own homelab adaptations
 - Share improvements via pull requests
 ## 📞 Next Steps
 1. **Create GitHub repository** for version control
 2. **Start with Phase 1** foundation setup
 3. **Build deployment automation** as we go
 4. **Document lessons learned** for future reference
 ---
 **Status**: 🚧 Planning & Initial Development
 **Last Updated**: June 2025
 This repo is a mono-repo that is broken up into three sections:
 - infra-config
 - apps
 - cluster
 ## Hardware
- Dell 7050 SFF (7)
+- 7× Dell 7050 SFF
- Minisforum TH60 (3)
+- 3× Minisforum TH60
- Minisforum MS01 (2)
+- 2× Minisforum MS01
- Synology 1621+
+- Synology DS1621+
 - Ubiquiti UDM Pro
-## Software
+## Software Stack
- Proxmox (Virtualization)
+- **Virtualization**: Proxmox
- Cloudflare (Domain Hosting, Public DNS)
+- **Automation**: Terraform, Ansible, n8n, Semaphore
- Unbound (Recursive DNS)
+- **DNS**: Technitium (authoritative), Unbound (recursive)
- FreeIPA (Identity management, Authoritive DNS)
+- **IPAM/DCIM**: NetBox
- Matchbox (iPXE)
+- **Networking**: Ubiquiti UDM Pro
- Red Hat OpenShift
+- **Templates**: Packer (Ubuntu 24.04, Fedora 42)
 - OpenShift Agent Based Installer (Install OpenShift)
  - [Red Hat Advanced Cluster Management for Kubernetes](https://www.redhat.com/en/technologies/management/advanced-cluster-management)
  - [Vault](https://www.hashicorp.com/en/products/vault)
  - [OpenShift GitOps (ArgoCD)](https://www.redhat.com/en/technologies/cloud-computing/openshift/gitops)
  - [Red Hat Ansible Automation Platform](https://www.redhat.com/en/technologies/management/ansible)
 ## Prerequisites
 - Ansible user created
 - Ansible configured
 - [Networking](docs/networks.md)
 - [Proxmox](docs/proxmox.md) (In my homelab, internal DNS, identity manangement, and ipxe are hosted here.)
 - Matchbox
 - DNS
 - Domain Registration
 ## Assumptions
 There are a dozen different architectures you could use to deploy OpenShift in every which way.
 For the sake of this documentation we'll assume the following:
 ## Getting Started
-[Step-by-Step Walkthrough](step-by-step.md)
+See [docs/decisions/vm-provisioning-flow.md](docs/decisions/vm-provisioning-flow.md) for the full architecture decision record.
-### Hub Cluster
+Previous OpenShift/ACM/Fastpass content is preserved in the `archive/pre-mk-labs` branch.
-You'll need an OpenShift "Hub Cluster" with access to persistant storage.
+## Security
 A Single Node OpenShift (SNO) instance, installed on bare metal, will act as a Hub cluster and run:
- Advanced Cluster Management
+No sensitive data is stored in this repository. Secrets are managed via Ansible Vault and environment variables on pipeline hosts.
 - Ansible Automation Platform
 - Vault
 - ~~OpenShift GitOps~~
 #### Network Prerequisites
 The prerequisites for OpenShift in traditional and HCP patterns are largely the same - it just kind of depends on where your DNS records go to.
 | Cluster           | Endpoint    | VIP           | DNS A Record                      | Notes                           |
 |------------------|-------------|---------------|-----------------------------------|----------------------------------|
 | Hub Cluster (SNO) | App Ingress | 192.168.0.10 | *.apps.hub-cluster.example.com    | SNO App VIP goes to IP of node  |
 | Hub Cluster (SNO) | API         | 192.168.0.10 | api.hub-cluster.example.com       | SNO API goes to IP of node      |
 These DNS entries should be put in your Authoratitive DNS.
 #### ACM & GitOps Configuration
 Before you start creating clusters you may want to create some Policies, integrate ACM and ArgoCD, etc. This step is optional in case you're just interested in trying out Hosted Control Planes or copy/paste around a cluster for testing purposes.
 Find additional details in the ./02-rhacm-config folder.
 #### Creating a Cluster
 With everything in its right place, you can now start to declaratively create clusters
 ./05-clusters/hcp-bmh - HCP to Bare Metal Hosts
 ### Internal Cluster
 Two additional bare metal nodes, to be added to Advanced Cluster Management (ACM) running on the SNO Hub. These will be used to create another HCP cluster.
 These servers have a BMC interface with Redfish - if not, then you'll need to manually manage the boot and installation of those servers.
 This makes it to where you just need 3 bare metal nodes. You could run one HCP Bare Metal cluster with both of the other nodes, but then you have a shared storage requirement that can't be satisfied by ODF since that needs at least 3 nodes.
 ### External Cluster
 You'll also either need you just need at least 2 bare metal nodes.
 ---
-## Credits
+**Status**: 🚧 Active Development — VM Provisioning Pipeline
- Ken Moini - As I used his [repo](https://github.com/kenmoini/ztp-for-you-and-me) as the baseline for this project.
+**Last Updated**: February 2026
 - Ryan Etten
 - Andrew Potozniak
@@ -1,309 +0,0 @@
 # FastPass Kubernetes Cluster - Complete Deployment Guide
 ## Overview
 This guide provides step-by-step instructions for deploying a complete Kubernetes cluster on Fedora using Ansible automation. The FastPass cluster is designed for high availability with multiple control plane nodes and worker nodes.
 ## Prerequisites
 ### System Requirements
 - **OS**: Fedora 38+ (64-bit)
 - **Memory**: Minimum 2GB RAM per node
 - **CPU**: Minimum 2 cores per node
 - **Storage**: Minimum 10GB free space
 - **Network**: All nodes must be able to communicate
 ### Software Requirements
 - Ansible 2.15+
 - Python 3.8+
 - SSH access to all nodes with sudo privileges
 ## Quick Start
 ### 1. Validate Your Setup
 Before deploying, run the validation script:
 ```bash
 cd ansible
 ./scripts/test-fastpass-deployment.sh
 ```
 This will check:
 - ✅ Ansible installation
 - ✅ Inventory configuration
 - ✅ Playbook files
 - ✅ Role files
 - ✅ Group variables
 - ✅ Playbook syntax
 - ✅ Common issues
 ### 2. Deploy the Cluster
 ```bash
 # Deploy the complete cluster
 ansible-playbook -i inventory.yml playbooks/deploy-fastpass-cluster.yml
 ```
 ## Deployment Process
 The deployment follows this sequence:
 ### Phase 1: System Preparation
 - **Hosts**: All FastPass nodes
 - **Tasks**: 
  - Preflight checks (OS, memory, CPU, disk space)
  - Disable swap and configure kernel modules
  - Install and configure containerd
  - Install Kubernetes packages
 ### Phase 2: Control Plane Initialization
 - **Hosts**: First control plane node (`space-mountain`)
 - **Tasks**:
  - Initialize the Kubernetes cluster
  - Configure API server and etcd
  - Install Flannel CNI
  - Generate join commands
 ### Phase 3: Additional Control Plane Nodes
 - **Hosts**: Remaining control plane nodes (`big-thunder-mountain`, `splash-mountain`)
 - **Tasks**:
  - Join additional control plane nodes
  - Configure high availability
 ### Phase 4: Worker Nodes
 - **Hosts**: Worker nodes (`haunted-mansion`, `peter-pans-flight`)
 - **Tasks**:
  - Join worker nodes to the cluster
  - Apply node labels and taints
 ### Phase 5: Validation
 - **Hosts**: First control plane node
 - **Tasks**:
  - Verify all nodes are ready
  - Check all pods are running
  - Display final cluster status
 ## Configuration
 ### Inventory Structure
 Ensure your `inventory.yml` has the correct structure:
 ```yaml
 fastpass_control_plane:
  hosts:
    space-mountain:
    big-thunder-mountain:
    splash-mountain:
 fastpass_workers:
  hosts:
    haunted-mansion:
    peter-pans-flight:
 fastpass:
  children:
    fastpass_control_plane:
    fastpass_workers:
 ```
 ### Group Variables
 Key variables in `group_vars/fastpass/vars`:
 ```yaml
 # Cluster Configuration
 cluster_name: "fastpass"
 kubernetes_version: "1.33"
 pod_network_cidr: "10.244.0.0/16"
 service_cidr: "10.96.0.0/12"
 # Control Plane Configuration
 control_plane_endpoint: "fastpass.local.mk-labs.cloud"
 control_plane_port: "6443"
 # CNI Configuration
 cni_plugin: "flannel"
 # Node Labels
 node_labels:
  zone:
    # Control Plane nodes (masters) - no zone labels needed
    # space-mountain: (control plane - no zone label)
    # big-thunder-mountain: (control plane - no zone label)
    # splash-mountain: (control plane - no zone label)
    # Worker nodes
    haunted-mansion: "backstage"      # Internal/backstage workloads
    peter-pans-flight: "backstage"    # Internal/backstage workloads
    # Future node: "on-stage"         # External/on-stage workloads (future)
 ```
 ## Troubleshooting
 ### Common Issues
 #### 1. Swap Enabled
 **Problem**: `running with swap on is not supported`
 **Solution**: The playbook automatically disables swap, but if it persists:
 ```bash
 # On the problematic node
 sudo swapoff -a
 sudo systemctl restart kubelet
 ```
 #### 2. CNI Not Ready
 **Problem**: `Network plugin returns error: cni plugin not initialized`
 **Solution**: The playbook installs Flannel automatically. If issues persist:
 ```bash
 kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
 ```
 #### 3. Control Plane Endpoint Issues
 **Problem**: `unable to add a new control plane instance to a cluster that doesn't have a stable controlPlaneEndpoint`
 **Solution**: The playbook uses the first control plane node as the endpoint. For production, consider using a load balancer.
 #### 4. Permission Issues
 **Problem**: `User "kubernetes-admin" cannot create resource "secrets"`
 **Solution**: The playbook uses super-admin.conf for proper permissions.
 ### Debug Commands
 ```bash
 # Check node status
 kubectl get nodes -o wide
 # Check pod status
 kubectl get pods --all-namespaces
 # Check kubelet logs
 journalctl -u kubelet -f
 # Check API server logs
 kubectl logs -n kube-system kube-apiserver-space-mountain
 # Check CNI status
 kubectl get pods -n kube-flannel
 ```
 ## Post-Deployment
 ### 1. Verify Cluster Health
 ```bash
 # Check all nodes are ready
 kubectl get nodes -o wide
 # Check all pods are running
 kubectl get pods --all-namespaces
 # Test cluster connectivity
 kubectl cluster-info
 ```
 ### 2. Manage Kubeconfig
 Use the provided script to manage different cluster configurations:
 ```bash
 # Switch to FastPass cluster
 ./scripts/manage-kubeconfigs.sh fastpass
 # Check status
 ./scripts/manage-kubeconfigs.sh status
 # Test connectivity
 ./scripts/manage-kubeconfigs.sh test
 ```
 ### 3. Deploy Applications
 Your cluster is now ready for workloads:
 ```bash
 # Deploy a test application
 kubectl run nginx --image=nginx --port=80
 # Create a service
 kubectl expose pod nginx --port=80 --type=NodePort
 # Check the service
 kubectl get svc nginx
 ```
 ## Maintenance
 ### Adding New Nodes
 1. Add the new node to the inventory
 2. Run the preparation role:
   ```bash
   ansible-playbook -i inventory.yml playbooks/deploy_k8s.yml --limit=new-node
   ```
 3. Join the node to the cluster manually or extend the playbook
 ### Upgrading Kubernetes
 1. Update the `kubernetes_version` variable
 2. Run the preparation role on all nodes
 3. Upgrade control plane nodes first
 4. Upgrade worker nodes
 ### Backup and Recovery
 - Backup `/etc/kubernetes/` directory on control plane nodes
 - Backup etcd data: `etcdctl snapshot save backup.db`
 - Document cluster configuration
 ## Architecture
 ```
 FastPass Cluster Architecture:
 ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
 │  Control Plane  │    │  Control Plane  │    │  Control Plane  │
 │  space-mountain │    │big-thunder-mtn  │    │ splash-mountain │
 │   (Masters)     │    │   (Masters)     │    │   (Masters)     │
 │  No Zone Label  │    │  No Zone Label  │    │  No Zone Label  │
 └─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 │
         ┌───────────────────────┼───────────────────────┐
         │                       │                       │
 ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
 │   Worker Node   │    │   Worker Node   │    │   Worker Node   │
 │ haunted-mansion │    │peter-pans-flight│    │  (Future)       │
 │   (Backstage)   │    │   (Backstage)   │    │   (On-Stage)    │
 │ zone=backstage  │    │ zone=backstage  │    │ zone=on-stage   │
 └─────────────────┘    └─────────────────┘    └─────────────────┘
 ```
 ## Support
 For issues specific to this deployment:
 1. Check the troubleshooting section
 2. Review Ansible logs with `-vvv` verbosity
 3. Check system logs on affected nodes
 4. Verify network connectivity between nodes
 ## Files Structure
 ```
 ansible/
 ├── group_vars/
 │   └── fastpass/
 │       └── vars                    # Cluster configuration
 ├── playbooks/
 │   ├── deploy-fastpass-cluster.yml # Main deployment playbook
 │   ├── deploy_k8s.yml             # Legacy deployment playbook
 │   └── roles/
 │       ├── kubernetes/             # System preparation
 │       ├── fastpass-control-plane/ # Control plane setup
 │       ├── fastpass-control-plane-join/ # Additional control plane nodes
 │       └── fastpass-workers/       # Worker node setup
 ├── scripts/
 │   ├── test-fastpass-deployment.sh # Validation script
 │   └── manage-kubeconfigs.sh      # Kubeconfig management
 └── inventory.yml                   # Host inventory
 ``` 
@@ -1,228 +0,0 @@
 # FastPass Kubernetes Cluster - Fedora Deployment Guide
 ## Overview
 This guide covers the deployment of a Kubernetes cluster on Fedora using Ansible automation. The FastPass cluster is designed for high availability with multiple control plane nodes and worker nodes.
 ## Architecture
 ```
 FastPass Cluster Architecture:
 ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
 │  Control Plane  │    │  Control Plane  │    │  Control Plane  │
 │  space-mountain │    │big-thunder-mtn  │    │ splash-mountain │
 │   (Leader)      │    │                 │    │                 │
 └─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 │
         ┌───────────────────────┼───────────────────────┐
         │                       │                       │
 ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
 │   Worker Node   │    │   Worker Node   │    │   Worker Node   │
 │ haunted-mansion │    │peter-pans-flight│    │  (Future)       │
 │   (backstage)   │    │   (backstage)   │    │                 │
 └─────────────────┘    └─────────────────┘    └─────────────────┘
 ```
 ## Prerequisites
 ### System Requirements
 - **OS**: Fedora 38+ (64-bit)
 - **Memory**: Minimum 2GB RAM per node
 - **CPU**: Minimum 2 cores per node
 - **Storage**: Minimum 10GB free space
 - **Network**: All nodes must be able to communicate
 ### Software Requirements
 - Ansible 2.15+
 - Python 3.8+
 - SSH access to all nodes with sudo privileges
 ## Quick Start
 ### 1. Prepare Inventory
 Ensure your `inventory.yml` has the correct host groups:
 ```yaml
 fastpass_control_plane:
  hosts:
    space-mountain:
    big-thunder-mountain:
    splash-mountain:
 fastpass_workers:
  hosts:
    haunted-mansion:
    peter-pans-flight:
 fastpass:
  children:
    fastpass_control_plane:
    fastpass_workers:
 ```
 ### 2. Configure Variables
 Edit `group_vars/fastpass/vars` to customize your deployment:
 ```yaml
 # Cluster Configuration
 cluster_name: "fastpass"
 kubernetes_version: "1.33"
 pod_network_cidr: "10.244.0.0/16"
 service_cidr: "10.96.0.0/12"
 # Node Labels
 node_labels:
  zone:
    space-mountain: "external"
    big-thunder-mountain: "external"
    splash-mountain: "external"
    haunted-mansion: "internal"
    peter-pans-flight: "internal"
 ```
 ### 3. Deploy the Cluster
 ```bash
 # Deploy the complete cluster
 ansible-playbook -i inventory.yml playbooks/deploy-fastpass-cluster.yml
 # Or deploy step by step
 ansible-playbook -i inventory.yml playbooks/deploy_k8s.yml
 ```
 ## Deployment Process
 ### Phase 1: System Preparation
 - Preflight checks (OS, memory, CPU, disk space)
 - Disable swap and configure kernel modules
 - Install and configure containerd
 - Install Kubernetes packages
 ### Phase 2: Control Plane Setup
 - Initialize the first control plane node
 - Configure API server and etcd
 - Install Calico CNI
 - Join additional control plane nodes
 ### Phase 3: Worker Node Setup
 - Join worker nodes to the cluster
 - Apply node labels and taints
 - Verify cluster health
 ## Troubleshooting
 ### Common Issues
 #### 1. Repository Errors
 **Problem**: Docker CE repository not found
 **Solution**: Update the repository URL in `container-runtime.yml`:
 ```yaml
 baseurl: "https://download.docker.com/linux/fedora/{{ ansible_distribution_major_version }}/x86_64/stable"
 ```
 #### 2. kubeadm Init Failures
 **Problem**: kubeadm init fails with preflight errors
 **Solution**: The playbook includes `--ignore-preflight-errors=all` flag
 #### 3. Node Not Ready
 **Problem**: Worker nodes stuck in NotReady state
 **Solution**: Check CNI installation and network connectivity
 ### Debug Commands
 ```bash
 # Check node status
 kubectl get nodes -o wide
 # Check pod status
 kubectl get pods --all-namespaces
 # Check kubelet logs
 journalctl -u kubelet -f
 # Check containerd status
 systemctl status containerd
 ```
 ## Customization
 ### Network Configuration
 Modify the pod and service CIDRs in `group_vars/fastpass/vars`:
 ```yaml
 pod_network_cidr: "10.244.0.0/16"
 service_cidr: "10.96.0.0/12"
 ```
 ### CNI Selection
 The default CNI is Calico. To change, modify the control plane role:
 ```yaml
 cni_plugin: "flannel"  # or "weave", "cilium"
 ```
 ### Node Labels and Taints
 Configure node labels in `group_vars/fastpass/vars`:
 ```yaml
 node_labels:
  zone:
    space-mountain: "external"
    haunted-mansion: "internal"
  environment: "production"
  cluster: "fastpass"
 ```
 ## Maintenance
 ### Adding New Nodes
 1. Add the new node to the inventory
 2. Run the preparation role: `ansible-playbook -i inventory.yml playbooks/deploy_k8s.yml --limit=new-node`
 3. Join the node to the cluster
 ### Upgrading Kubernetes
 1. Update the `kubernetes_version` variable
 2. Run the preparation role on all nodes
 3. Upgrade control plane nodes first
 4. Upgrade worker nodes
 ### Backup and Recovery
 - Backup `/etc/kubernetes/` directory on control plane nodes
 - Backup etcd data: `etcdctl snapshot save backup.db`
 - Document cluster configuration
 ## Security Considerations
 - SELinux is set to permissive mode for compatibility
 - Firewalld is disabled (configure as needed for your environment)
 - Consider enabling RBAC and network policies
 - Regularly update Kubernetes and system packages
 ## Support
 For issues specific to this deployment:
 1. Check the troubleshooting section
 2. Review Ansible logs with `-vvv` verbosity
 3. Check system logs on affected nodes
 4. Verify network connectivity between nodes
 ## Files Structure
 ```
 ansible/
 ├── group_vars/
 │   └── fastpass/
 │       └── vars                    # Cluster configuration
 ├── playbooks/
 │   ├── deploy-fastpass-cluster.yml # Main deployment playbook
 │   ├── deploy_k8s.yml             # Legacy deployment playbook
 │   └── roles/
 │       ├── kubernetes/             # System preparation
 │       ├── fastpass-control-plane/ # Control plane setup
 │       └── fastpass-workers/       # Worker node setup
 └── inventory.yml                   # Host inventory
 ``` 
@@ -1,233 +0,0 @@
 # Kubeconfig Management - Modular & Repeatable
 This document explains the modular kubeconfig management system for your homelab's multiple Kubernetes clusters.
 ## Overview
 The kubeconfig management has been refactored into reusable, modular components that can handle multiple clusters seamlessly:
 - **FastPass** (Vanilla Kubernetes)
 - **Hub Cluster** (OpenShift SNO)
 - **Internal Cluster** (OpenShift)
 - **Future clusters** (easily extensible)
 ## Architecture
 ```
 ansible/playbooks/roles/
 ├── kubeconfig-manager/          # Core reusable role
 │   ├── tasks/main.yml          # Main entry point
 │   ├── tasks/merge-kubeconfig.yml  # Merging logic
 │   ├── defaults/main.yml       # Default variables
 │   └── README.md               # Role documentation
 ├── cluster-kubeconfig/         # Generic wrapper role
 └── fastpass-first-control-plane/  # Uses kubeconfig-manager
 ```
 ## Quick Usage
 ### 1. Using the Script (Easiest)
 ```bash
 cd ansible
 # Setup FastPass cluster kubeconfig
 ./scripts/setup-kubeconfig.sh fastpass fastpass_control_plane[0]
 # Setup Hub cluster kubeconfig  
 ./scripts/setup-kubeconfig.sh hub hub_cluster
 # Setup Internal cluster kubeconfig
 ./scripts/setup-kubeconfig.sh internal internal_cluster[0]
 ```
 ### 2. Direct Ansible Usage
 ```bash
 # Single cluster
 ansible-playbook -i inventory.yml \
  playbooks/examples/multi-cluster-kubeconfig.yml \
  --limit fastpass_control_plane[0] \
  -e target_cluster_name=fastpass
 # Multiple clusters at once
 ansible-playbook -i inventory.yml \
  playbooks/examples/multi-cluster-kubeconfig.yml
 ```
 ### 3. In Your Own Playbooks
 ```yaml
 - name: Setup kubeconfig for any cluster
  hosts: my_cluster_nodes[0]
  roles:
    - role: kubeconfig-manager
      vars:
        cluster_name: "my-cluster"
        kubeconfig_source_path: "/etc/kubernetes/admin.conf"
 ```
 ## Features
 ### ✅ **Modular & Reusable**
 - Single role works with any Kubernetes cluster
 - Easy to extend for new clusters
 - Consistent behavior across all clusters
 ### ✅ **Safe Operations**
 - Automatic backups before merging
 - Non-destructive merging
 - Preserves existing contexts
 ### ✅ **Multi-Cluster Ready**
 - Unique naming: `cluster-name-admin`
 - No conflicts between clusters
 - Easy context switching
 ### ✅ **Homelab Optimized**
 - Works with vanilla Kubernetes
 - Works with OpenShift
 - Handles different kubeconfig paths
 ## Generated Structure
 After running the kubeconfig management:
 ```bash
 ~/.kube/
 ├── config                      # Merged config with all clusters
 ├── config-fastpass            # Individual cluster configs
 ├── config-hub
 ├── config-internal
 └── config.backup.1697123456   # Timestamped backups
 ```
 ## Context Management
 ```bash
 # List all available contexts
 kubectl config get-contexts
 # Switch between clusters
 kubectl config use-context fastpass-admin
 kubectl config use-context hub-admin
 kubectl config use-context internal-admin
 # Check current context
 kubectl config current-context
 # Test connectivity
 kubectl cluster-info
 kubectl get nodes
 ```
 ## Integration with Existing Roles
 ### Before (Monolithic)
 ```yaml
 # fastpass-first-control-plane/tasks/main.yml
 - name: 50+ lines of kubeconfig logic
  # Lots of repetitive code
 ```
 ### After (Modular)
 ```yaml
 # fastpass-first-control-plane/tasks/main.yml
 - name: Setup kubeconfig for FastPass cluster
  ansible.builtin.include_role:
    name: kubeconfig-manager
  vars:
    cluster_name: "{{ cluster_name }}"
 ```
 ## Adding New Clusters
 To add a new cluster (e.g., "edge-cluster"):
 1. **Add to inventory:**
 ```yaml
 edge_cluster:
  hosts:
    edge-node-01:
 ```
 2. **Use the script:**
 ```bash
 ./scripts/setup-kubeconfig.sh edge edge_cluster[0]
 ```
 3. **Or add to playbook:**
 ```yaml
 - name: Setup kubeconfig for Edge cluster
  hosts: edge_cluster[0]
  roles:
    - role: kubeconfig-manager
      vars:
        cluster_name: "edge"
 ```
 ## Customization
 ### Different Kubeconfig Paths
 ```yaml
 - role: kubeconfig-manager
  vars:
    cluster_name: "openshift-cluster"
    kubeconfig_source_path: "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/localhost.kubeconfig"
 ```
 ### Custom Context Names
 ```yaml
 - role: kubeconfig-manager
  vars:
    cluster_name: "prod"
    context_suffix: "system:admin"  # Results in "prod-system:admin"
 ```
 ## Troubleshooting
 ### Common Issues
 1. **Permission Denied**
   ```bash
   # Fix ownership
   sudo chown -R $USER:$USER ~/.kube/
   ```
 2. **Context Not Found**
   ```bash
   # List available contexts
   kubectl config get-contexts
   # Check if cluster was added
   kubectl config view
   ```
 3. **Backup Recovery**
   ```bash
   # Restore from backup
   cp ~/.kube/config.backup.1697123456 ~/.kube/config
   ```
 ### Debug Mode
 ```bash
 # Run with verbose output
 ansible-playbook -vvv playbooks/examples/multi-cluster-kubeconfig.yml
 ```
 ## Benefits for Your Homelab
 1. **Consistency**: Same process for all clusters
 2. **Maintainability**: Single role to update/fix
 3. **Scalability**: Easy to add new clusters
 4. **Safety**: Automatic backups and safe merging
 5. **Flexibility**: Works with different Kubernetes distributions
 ## Next Steps
 1. **Test the modular approach** with your FastPass cluster
 2. **Extend to Hub and Internal clusters** using the same pattern
 3. **Create cluster-specific variables** in group_vars if needed
 4. **Add monitoring/validation** tasks to verify kubeconfig health
 This modular approach makes your homelab's multi-cluster management much more maintainable and repeatable!
@@ -54,7 +54,7 @@
 # (pathspec) Colon-separated paths in which Ansible will search for collections content. Collections must be in nested *subdirectories*, not directly in these directories. For example, if ``COLLECTIONS_PATHS`` includes ``'{{ ANSIBLE_HOME ~ "/collections" }}'``, and you want to add ``my.collection`` to that directory, it must be saved as ``'{{ ANSIBLE_HOME} ~ "/collections/ansible_collections/my/collection" }}'``.
-;collections_path=/Users/rblundon/.ansible/collections:/usr/share/ansible/collections
+collections_path=/opt/ansible-collections:/usr/share/ansible/collections
 # (boolean) A boolean to enable or disable scanning the sys.path for installed collections.
 ;collections_scan_sys_path=True
@@ -209,7 +209,7 @@ private_key_file=~/.ssh/ansible
 remote_user=wed
 # (pathspec) Colon-separated paths in which Ansible will search for Roles.
-;roles_path=/Users/rblundon/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
+roles_path=./roles
 # (string) Set the main callback used to display Ansible output. You can only have one at a time.
 # You can have many other callbacks, but just one can be in charge of stdout.
@@ -262,7 +262,7 @@ remote_user=wed
 # (path) The vault password file to use. Equivalent to ``--vault-password-file`` or ``--vault-id``.
 # If executable, it will be run and the resulting stdout will be used as the password.
-;vault_password_file=
+vault_password_file=/home/hermes/.vault_pass.txt
 # (integer) Sets the default verbosity, equivalent to the number of ``-v`` passed in the command line.
 ;verbosity=0
@@ -0,0 +1,6 @@
 ---
 - name: Create jarvis user and deploy SSH key
  hosts: all
  become: true
  roles:
    - jarvis_user
@@ -0,0 +1,193 @@
 ---
 # ============================================================================
 # Semaphore configuration-as-code
 # ============================================================================
 # Drives a freshly-deployed Semaphore instance into its desired state via
 # the Semaphore REST API. Idempotent: every object is checked first; only
 # missing ones are created. Existing objects are left alone.
 #
 # Loaded from group_vars/all/semaphore.yml so that the configuration is
 # version-controlled in the homelab repo and survives a wipe-and-redeploy
 # of the Semaphore VM.
 # ============================================================================
 # ---------------------------------------------------------------------------
 # API connection (defaults to the local Traefik-fronted service-name URL).
 # Override semaphore_api_url to point at a specific instance if needed.
 # ---------------------------------------------------------------------------
 semaphore_api_url: "https://semaphore.local.mk-labs.cloud/api"
 semaphore_api_validate_certs: true
 semaphore_api_token: "{{ vault_semaphore_api_token }}"
 # Feature flag — keeps day1_deploy_semaphore.yml deploy-only by default.
 # Set true to also run the configuration pass.
 semaphore_configure: false
 # ---------------------------------------------------------------------------
 # Declarative configuration of the Semaphore instance.
 # ---------------------------------------------------------------------------
 #
 # Top-level shape:
 #
 #   semaphore_config:
 #     project:        single dict — the lab uses one project ("mk-labs")
 #     keys:           list of credentials Semaphore stores
 #     repositories:   git repos Semaphore can clone
 #     inventories:    Ansible inventories from those repos
 #     environments:   env-var bundles
 #     templates:      task templates that tie everything together
 #
 # Each list element has a unique "name" used as the natural identity key.
 # ---------------------------------------------------------------------------
 semaphore_config:
  project:
    name: mk-labs
    alert: false
    max_parallel_tasks: 0  # 0 = unlimited
  keys:
    # The ansible-vault password. login_password type with empty login
    # — only the password field is consumed by Semaphore at runtime.
    - name: ansible-vault-pass
      type: login_password
      login: ""
      password: "{{ vault_ansible_vault_password }}"
    # SSH key for the gitea deploy access (clone the homelab repo).
    - name: gitea-deploy
      type: ssh
      ssh_login: git
      ssh_private_key: "{{ vault_gitea_deploy_key }}"
    # SSH key for the universal automation account 'wed' — pre-baked in
    # every mk-labs VM template. This is the canonical user Semaphore
    # uses to reach the fleet.
    - name: wed-ssh
      type: ssh
      ssh_login: wed
      ssh_private_key: "{{ vault_wed_ssh_private_key }}"
    # SSH key Semaphore can use to reach the fleet as jarvis (admin
    # account provisioned by linux-baseline). Retained for jobs that
    # specifically need jarvis-level access; the default is wed-ssh.
    - name: jarvis-ssh
      type: ssh
      ssh_login: jarvis
      ssh_private_key: "{{ vault_jarvis_ssh_private_key }}"
  repositories:
    - name: homelab
      git_url: "ssh://git@gitea.mk-labs.cloud:2221/rblundon/homelab.git"
      git_branch: main
      ssh_key: gitea-deploy
  inventories:
    - name: production
      type: file
      inventory_file: ansible/inventory.yml
      repository: homelab
      # wed is the universal automation account pre-baked in every VM
      # template. Semaphore uses it for fleet-wide jobs.
      ssh_key: wed-ssh
      # become_key is Semaphore's sudo PASSWORD slot, not a second SSH
      # key. wed has passwordless sudo on every host, so reference the
      # built-in "None" key. (Semaphore rejects an SSH-type key here.)
      become_key: None
  environments:
    - name: default
      env:
        ANSIBLE_HOST_KEY_CHECKING: "False"
        ANSIBLE_FORCE_COLOR: "True"
        # Semaphore runs ansible-playbook from the cloned REPO ROOT (not
        # from the playbook's directory as I first assumed). Path is
        # therefore relative to repo root, not playbook dir.
        ANSIBLE_ROLES_PATH: "ansible/roles"
        # Collections are installed by the semaphore role into a host-side
        # directory bind-mounted into the container at this path.
        ANSIBLE_COLLECTIONS_PATH: "/opt/ansible-collections"
  templates:
    - name: "day0_linux_baseline"
      description: "Apply the mk-labs Linux baseline to one or more hosts."
      app: ansible
      playbook: ansible/playbooks/day0_linux_baseline.yml
      inventory: production
      repository: homelab
      environment: default
      vault_password: ansible-vault-pass
      arguments: '["--diff"]'
      survey_vars:
        - name: target
          title: "Target host or group"
          description: "Inventory target (e.g. figment, semaphore_server, all)"
          required: true
          type: TextVar
          default_value: "all"
    - name: "day1_deploy_semaphore"
      description: "Re-deploy Semaphore + PostgreSQL on figment."
      app: ansible
      playbook: ansible/playbooks/day1_deploy_semaphore.yml
      inventory: production
      repository: homelab
      environment: default
      vault_password: ansible-vault-pass
      arguments: '["--diff"]'
    - name: "day0_linux_baseline_check"
      description: "Dry-run the baseline — shows diffs, applies nothing."
      app: ansible
      playbook: ansible/playbooks/day0_linux_baseline.yml
      inventory: production
      repository: homelab
      environment: default
      vault_password: ansible-vault-pass
      arguments: '["--check","--diff"]'
      survey_vars:
        - name: target
          title: "Target host or group"
          description: "Inventory target (e.g. figment, semaphore_server, all)"
          required: true
          type: TextVar
          default_value: "all"
    - name: "llm_inference_multimodel_stage_models"
      description: >-
        Stage additional GGUF models into /opt/models on astro-orbiter via the
        llm-inference-multimodel role (--tags models only). Idempotent: skips
        files already present at the correct byte size. Notifies the
        llama-server-router restart handler ONLY when a new GGUF is actually
        downloaded. Does NOT touch Phase 4 (verify) or the legacy
        llama-server-qwen service. Safe to run repeatedly.
      app: ansible
      playbook: ansible/playbooks/day1_deploy_llm_inference_multimodel.yml
      inventory: production
      repository: homelab
      environment: default
      vault_password: ansible-vault-pass
      arguments: '["--tags","models","--diff"]'
      # Scoped to --tags models:
      #   Phase 0 (discover) -- skipped (no tag)
      #   Phase 1 (models)   -- RUN (idempotent GGUF staging via stage_model.yml)
      #   Phase 2 (systemd)  -- skipped
      #   Phase 3 (firewall) -- skipped
      #   Phase 4 (verify)   -- SKIPPED (collision risk: verify.yml would start
      #                         llama-server-qwen on :8002, conflicting with the
      #                         production llama-server-router.service. Excluded
      #                         here deliberately. See t_730f9584.)
    - name: "llm_router_update_unit"
      description: >-
        Re-render and reload the llama-server-router systemd unit on astro-orbiter,
        then restart the live service so new args (e.g. --models-max) take effect.
        Drives playbooks/day2_bump_router_models_max.yml. Added 2026-08-12 (t_33acbb2e):
        bump --models-max 1 -> 4 with full VRAM budget note in host_vars.
      app: ansible
      playbook: ansible/playbooks/day2_bump_router_models_max.yml
      inventory: production
      repository: homelab
      environment: default
      vault_password: ansible-vault-pass
      arguments: '["--diff"]'
@@ -0,0 +1,4 @@
 ---
 step_ca_url: "https://turnstile.local.mk-labs.cloud:9000"
 step_ca_fingerprint: "f63c44e76381e359978bd2dca07c928d04ad44b575f0364fa66b5725c7e7891b"
 step_ca_provisioner_name: "admin"
@@ -17,4 +17,31 @@ base_domain: "local.mk-labs.cloud"
 terraform_server: "infra01"
 # Traefik variables
-traefik_server: "lightning-lane"
+traefik_server: "lightning-lane"
 # ---------------------------------------------------------------------------
 # JARVIS automation account
 # ---------------------------------------------------------------------------
 # Public key for the 'jarvis' user provisioned by the linux-baseline role on
 # every host. Public keys are not secret; the matching private key lives on
 # the JARVIS command centre (carousel-of-progress) and, when needed, in
 # group_vars/all/vault as vault_jarvis_ssh_private_key.
 jarvis_ssh_public_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID5sym5ajFvDyzw395BkHv7qVb66XPTx/OF1p19MGuNo jarvis@mk-labs"
 step_ca_principal_mappings:
  - local_user: wed
    principals:
      - ryan.blundon@protonmail.com
      - ryan.blundon
      - ryanblundon
  - local_user: rblundon
    principals:
      - ryan.blundon@protonmail.com
      - ryan.blundon
      - ryanblundon
 # Leviton My Leviton API
 leviton_email: "{{ vault_leviton_email }}"
 leviton_password: "{{ vault_leviton_password }}"
 jmri_vnc_password: "{{ vault_jmri_vnc_password }}"
@@ -1,170 +1,418 @@
 $ANSIBLE_VAULT;1.1;AES256
-64393362643437353731353234643535366431613861636131613139636435633465306333656161
+39633233363564393465633065323730343130653362373436356334323930393036656638303033
-3365306632386330373862363637363435633235643336610a663736396537313032336162633334
+6333636536643835303936633039633138336434663565320a643961663732373630333238633466
-32626233346233323565383030393130316532653062373730376163326134616239663865363862
+62633236666331326165653637353038313163643162323763313635306230333338613164316337
-3365316365643534610a343932623566356637663136336338386534373532383863363662326438
+6133396638336465350a306163643138653639646133346561323333313332336639393764343364
-37393331356632313163643138613330366631643465623633386433663236636334666264383033
+30666535323264363661653236333839643631376337303563383739656238343732396364366437
-34333935633131363364626333313336353733373562326636343633653838616532643737366435
+33333437643930656564376236333736336465356134653239393036313830333462363633653366
-36366330633263613938323566393063373233363538643531636131316335316461656631616362
+62623863613261323135633262633066383363313731646361326232366661356261316633323164
-30343965356363383139366534363664313831363132383337636266316538393139383232386230
+34653661383366643531353939303333363832323731623830393262373538613631613631616463
-65353364313534333632306465393264386466346365303232356263653563653338306131376131
+66396163613965343833356432643534393539643961373263323533316664386665383330326665
-66343764643639396663623330623332333466633737343738386230626463386137373536343430
+35393537363734333961393931386161386234323264316466303361663139323431303034393035
-36636430616132363161303338353033343963643338656639643662636664613965636263366135
+33353432363162353261646231313136643533363563343537666565306638353564646235623835
-64353134376162633162316166316239333963653930363830646165636537653238623537323238
+33653663393838303364363762613663646139326135313438663838323064326265303733346164
-39376338636335336338323935333439306534386539336331613961663261623962383237303632
+62386130613134653739353930623330663135633632346531306535643332303939356263636232
-34326139626366303331363337353432623636626231373731646665623831353865303130663533
+34313939333263313133373635396138643838396235626662343664316632663930313038643863
-37326639626237653361316666616161363531303961613538636663633063396363346337663239
+65653630373437373631633032333663353866636630306235363035623837656661323735356361
-64313333653766383736303161636530316538633033323063323662336335666264646564643465
+33343961373235613262613363636565396234613865303937323432656433343836373831343162
-64396436366131373061326233643562626335376263623031376532313730653363343038356135
+37356234363662333764656530333139653434336137326631386366306162646433333239316133
-30336131646635636638353934636139343662623265336338386535663536393935343132306335
+35303136633664336433386331316662313330643734383139633938636462346539623835653738
-33626632613663353661313061623732643539323637336433666330313463666661623836303462
+63373439623866363863633963633265373339613130343966656637376632396139646266306666
-61383732613139343434363662616539373837633139393764353864613062346237346661633033
+62316430343338626331326433366266396131666264633234323233646639643236323937396431
-61393639326535366266643664653439383238646132306636656130333733316638363830346433
+62323738376664303237653563323164396166313266613937383431653766616234653762383163
-38643430346263663639653762393935356662363533323766343362366532626532633530323630
+65666138353262663536623036623439363463306136363930343633383436333565363232376162
-30643436653130363066373665666434383732663837643839356539366432316539613863623839
+35343035306230666432666131643634383263663838626466633133653830623330656164313465
-35323930383437383535373330326239663966383234663233643337623330353236626464636530
+32363031313266396532313563343963613133396232353832636464663462636138623637646135
-36393735646664366235363831656262383333626265383066393363346165656265363438306135
+64356364363862653465303837663261333933623266333736343938643065393031343535633762
-34633530653837346638636665666162363062663333623263356131646661373234333939333536
+62363436323931373563323863643066653237393339653363656133316365346262306463336364
-30643639333836373439613561346130623239363737626238663436356365633562366465643561
+66303732366437633461373065363730336235313732623131633366633834376563663734653063
-32643039376234323963653733353138643138626430383639313765353035646539653466623539
+31656638623933376666393232613238323239623836313430356639613131303761346535336338
-32366135366266323261333938373837386636626232323661393035613638643335383564663462
+65613363366331613865343163623863636666303336646636663465373065393731306634663239
-31383663663063373935396439653231353962303630346338623938343064643037396562626637
+32653666343631666235383863336338326438646137656562646431623436323037313438343839
-63363363343739626639633934393734396434653262373137373539316536306663336466626639
+34663062396564306138356465623032396239396562643933306365613033333065323630393665
-34646465306439313239323365333761663161313461613766343463666134303036376338366663
+66346262303965353539663463626261633332373833346430346235376636323633623864353133
-64356364653462383534343438343933623766396232336434653765613864376235363735343032
+34666265373936376339646363383834393166376434653233306632373966336232316339336465
-32313133383731333366626534363065333738346230393636393938366633363561346436363338
+63313866653732386234666531663030313364626534343235333463376334663061316461613331
-33393932626333636166313431633339386564616466653165383962656633303161393935366635
+64303962393930373036643561393762666334666538626233333937653038633734626430623837
-65396464326333396433303966316231376432363335643732303438663131653932636564363264
+33303463386363386562333061353963303538366362313131656433383263666631666566633765
-36356663623436643835373538363662643134313030656236326335393334626634663136333137
+65613238393736646637346333323933326664663862326563363566613361343539646264626561
-38363432346663393435366536636335636465343131666562396334636530346235366365396431
+37323034323136376433343433316266386135333765643036373565326539643066336533343431
-64623739373130623436356161666139656330646661396430633264393632663366353037643130
+34316264393261373130333930383932343864386534376633336239303066356666363438356632
-35373839373430643065616630613433646536356563353133633361623235623733313234306435
+63313637616465386363346162653064613236383166353432636530336437376665643830356134
-64356635343361633462376131636362386433373339383365313436333332613333316237323632
+35366239326637373366383962376165643932336632373330326533366535666434343161373264
-33633362376331323930376130393838633333316330363362396131306230643161373636303034
+30323130626236356666663230613837373666666334646366616339313661613238666138353832
-37303738306233363937316332313830396336323639333634666364633566386261373031383563
+37633066303139313961643935383632646662613766303366663335306263646439383534373639
-36633865643634363234613463636437306236376435613034623537356162353235396665636338
+33643733626531376565623431666637323736393361373566626666633232383832656333333566
-31613733363430313532643235343438393137313136653730643364623662336236323464656465
+65616662383561626161363061646334633033386534633533393032386532643939373736356532
-65636431633464613437396561643638346565313937303731633837623930333538383335633938
+31633430613338343762656630303139316634323635653763386366633366366336383064613333
-62666461323033303937653366646432623764653566626333333766656235633662623631636636
+32373164633066643064316535333364366461666332663165346462386563373864663565613736
-35626161653736303261303233643236393038303063386532643563663065303331616637386661
+64636131346133623166656235306230353733373263313166383761303364366434656666303732
-33326637393633616430643665623431616336336666383236326538383535633364613135653131
+34313234663139616466633562383663316630666261336463653236393832303363316438326635
-38383961383264313130383263613763326666663033626338313762376336353866646634323537
+32343135396333613936383239633938393261393435653239643763373065623531336633313130
-39393565643265633437396535643165626165646537373039613061613936356434366238636562
+64323761646666636464623734656236653439646433303934633536333862653930303133336264
-61393639316531316564646334663863383836656261623230646461323566316664653965336136
+66333361323735386338643933336636303263613739666432626137393061343833636537396539
-62353839363664396632306231363031303265343131343739633835353338386335343737326633
+62626534663636323764376437393237663839356534306438306431663937323364316431323264
-36656433616135653663353166636461623337653138613435646663663939323631373063373136
+38353234303364653833393733383736636139336237386137313937656137636562303036666135
-63396331313538636437383635643831643935303932313964336532643366623434316662376632
+62626530383037613439646261346130353430303131303936393230333662386533356631326533
-63666464333736333162373461333563393138366638643039393865613238616531356165346236
+65643931313036623766316137353130363535383834323363623334376234393265326362653838
-65313432383533353738396530356465386263313939393962333637306138653939646638353235
+34353665666537343232626137383166383761643232313539346461393937353737373935323462
-37393733333032623763333632353866633866613162663833346534666237383430663838383965
+38303631383465623865323230333330383730333536346366616332343539313237333538333431
-66663332616164353539323261383364363237383064306132633530326134383561346536346636
+65636235343938316635633732393430323932626339323236323663663932306463373039343363
-32393365316435626464386631663037663934313533663030313466323134336439303362626365
+66363235366539386430333365613338646166313133346239346537313666396435343064643537
-30613764346362326130353333353663316539306437376465306332353464343530663933363562
+30363236396563346165353032373463386630646134356634646465353136643635373164643263
-39646333323631353835336461326233623963353535316633353236353236636462313163633638
+31396333353232323937346530303334623163316538353937396337636165356163396639373731
-35366538366336663534353739353231306164626664383737333565666337636164386539316632
+64363036346434623366376261316632343238306564646234383433343137396462303466383635
-64323966336637626538303830323535373533306362636236303065663161353734653633393064
+38353161383933336264393332386233643666656132633038323936353630313635313365353663
-66353938306238646238313037323163343935643863653632336462363834326634363433383338
+30353734363435373265616333653737383838376463646466633732333537346565666433656137
-35646364353834376439633136313939393064633539643462623562333832303261386331636533
+62333234653264626532323064386639643136336264346232663537393065386666623463336433
-37373837396461373064353565353438633938616162363138323561343662383330346633323133
+38613966376535303961343361363930313664633030353538306633656335326430653534346161
-36626665343636343739353732366133313434623230353732323037633236303237656364323763
+31613166366231386239366631623839316361303630663063336637303832373463613735356364
-30363530306131303630386131623866323861633636313431376161336536373436656334303036
+39323238656232383632643630386261323764643861353063616130323939383732343863356633
-61316639666136313338396233326234386634343863633632626264363336376531363461626564
+33303965363565313535386463613030373634653763313430643335393032663839636164303630
-35336335396561626339616234373265663032353432363832306666353339376338396366383839
+38653930383661386331613763316365653538386266396430633638633638653134633365343937
-62633063326135323539653133626538613331613334353430623232376336633334666433316638
+30313565663139366139623065623730356363313731663034663362313335346561323130396333
-61326563366432333632666436363761376237636434356231613061626633303737626531626236
+37343165333434613235363964323564306330393738333635303763633466366664616138613862
-62373166363432616666616630616565373934356564363266396662323963636434313735626161
+61623464656630343361353433313836316230653435356661326366366131386637363334326336
-61373738623966363032313632376465356235363834633639663733653131636531363563326662
+34313334386464363261343831356163613834353639323562333930363563343963633833623832
-62663365323934363761653536356162336463646566393838346134356536303264323065396534
+37323964336565383062373739346261306139376637306461336265316336666439353536313063
-63346531303764613634646336313530326663373433396436663162343530306131386265303137
+31653432373235666235396239363261316333373332333234376635323261306234383836383161
-32333432383966363735343331646233613631373165346131323632366165346235656330323939
+33303136363335353539663833306166643737333238643066313633323031613237666264666365
-34393030313466666265346163323561353734613434386239333630623762616566616562316265
+63333766616631366639396235633466383062623462363638363866346462306535623839626231
-38633263323339396462336166646431363336616462366463383663313839613638383663303133
+63666633393137316637616634653139316439626432366236663738316166643335333534626164
-32333835623531366236643530623961656631336139313037353962323937666465343430666336
+62623861356134663637663665313934313833636165643732376438316563383539663037373432
-33383464363930663630333164636361326334396163376661666564663538623732663839613237
+30326233373936346433323934663633363166376463623135386238393334396466316265613965
-35623237613761356235383130633733333766333236363537383862306539373665383461386534
+33623362653436613835386535393038363064373233623266643466633732366632346330343438
-32646432363339306433646139393732366464336339346161323736363937383566656663356230
+65383938363063666263616330356661343732313435383137313435656337353063613230656462
-65316562383562366366653466646461613235353061623363386331636139386136306131633232
+63383465343932303832313335346230313164333362303935343836363764313933623831306130
-32326238353861613236373964383131363537653632663532376364336166303662663336623161
+36333339373230386162323337386661303663616530383863333332396439313833303461643662
-37623838616664376561663763393534323931343163376535396662626333383730363232653036
+37336231343065623635376564393934316333626566623765353265613737646131653934343631
-39626266623666346635373630343135333532353866363234396265356631333265663432326235
+32613463636130333561653066343839303332396239656534303733643639343337363437656662
-37346439613363313334633964386632336138316662616661356637363730306164646439643765
+63303732326235643830303038663131613330303863353231313561376231666135353464303864
-62343137663137633031316665363362643165643731326464323962616330383164353861366462
+66633332613139386136303264366464356434633434656536393664313130646234393634326432
-31663965626338636534356461366362333831396464376134613331393664306639386632396236
+63663938373932386564363834373134356464633662383130386337633564643930613734383631
-66633066663831646361333730356361646133383834376230356336613038303665323732663162
+38643239663733393336643834643438303737636635396630333565656264653866366166623332
-37643130313739306638636262656632353431316531336438363636363833626237346363326461
+35653430323365313834643862393065323531633230613630343738333934383161613264356265
-31636262396632666335303566633161333937626330373835613236656130383132663863636364
+66626561386138396639353934653465373162323838303233636565306534633236343066623638
-34383661353966346362386461303062303138626536323163376362326639356336396437333530
+38653132376663666136383235663337343762646137643935626365343162363439303061393563
-66623765313638616430383838396164336565646338636430363262396636663965383531383139
+32633366623834613739303139333038303762316663663136656636663663386362613264663238
-35356663363437383132333566336131666139643639663662633363616133323031353032663438
+35323038326239643062666336333932383065383461373933626339383434363736323536313533
-39393266633639306363313530643239303132303264323063306335346233363233373865653863
+65363565306239363935666234653438393039303663623636376262666164393334363839393665
-65396430346630626566383963356264313630623336316236393730343437656435656664663266
+62326533613639643938613838396565376334333035333962626135356265353639653633373438
-32643639363536396266626435373363303234623230383035326536633631633737653739366563
+37316463303838633335306333346465386461373766356533313863333534326231353864303531
-65653932663539353436366139313635663962663330343461353864323166323733366163653162
+32663461623731653138646336663266303065306562643961393936373937653664323032626339
-66353866376134633534323366656265633930356132316137616235343866653265666161363266
+39373539363664316432333931633866646333346363336466393963656637366562336162613465
-30386638343739353965643466613639303666316432323634383464633431363566333831643134
+32663230613335393839633633633063636635373363653234663564663264353036393434663963
-65363138313439366430613837396337323436636530313664303964353966306565663630323662
+66653631663062666338326133636236303862356662666533326236613965323462303839623331
-33643639623935336235373332633739306539656235323439343331376632373538613534386430
+35626238323837343738356264313266383135626239363731353639633837323132623635633435
-30633832656532623262326662376237393262333764623937343232383861636563633730393765
+61386663366533326130626332613562633965633837346237633334303564363136633733323361
-39663032373132323735633361343933333332623465343863636263363161303165656234313338
+32353361396166396334386630353738643438643537303034623830303939326436363239343664
-65383462626465376162306639626531366161396531643364656539366233306166386234313761
+35633731396161623934633864313863323336326535313637653935376436363162363565666634
-30633666303832636561323236373363373137353032613637353637386238353234646431333661
+33613435326539343865653862613263383236313130383234623766333637646464613365346532
-32313834666237626333626437663562646438363561393364373565313536363535323536363537
+34653035393862643039623763323138333334623866316437613435666530396134306665316334
-35633834616136656261643130656131306264333131326431626439343765393031313964616639
+31373364653865323765646535303166373638636134326538343563363732636337653931373234
-33643933653033336436616635346335383335333334343836343638613264396462633530623866
+35303433656137333635383334376237616261393334623835353866323333393630373765613835
-30303462653130353430663862623261343866383335343239393435633962383864396430313536
+33363636323664383136353863666635646563393764633531616135313935333539333664343664
-66643535303437383232323564326266356636376464623563636139633237313661356330633931
+62633465356563353433633564303531633236656631326331323638643666643636396162316532
-66376631373366613731633863666132626366393532666331343361323764626539666238656262
+36313839363164663336326639313066316331303365386130666662366638353834306662303336
-66356332383235353864303562356638636130343938383530376432663866316238333434616233
+34393538346338376330333332383231333630353334646635313037313736613139383832663566
-32316432333064663764363831343464326334353737346632383565643737366233396139333639
+30653137386339623136356663613431303564633966383633643830393636383734306238333831
-35656435346335323030636363356633633332376138653930613037313537373038353433363532
+64363933386532346264666638316161393435653661356162306262323131643535636134396439
-39346161336261353463656430663033623036396365346638323832313361633866363134396165
+64656538313938623938323638393734343464653362636532386663303762326234373930363866
-65356632366463393435386366643232383162623536653965643235323934633464373061333237
+32306235363837396339383461396430373964396533616434373937386266396439303964356534
-32636564386662626365613235333664323563633466383238343336343063376531396161343530
+65663262343461343864633934636536366335326162393830633232633832383166366430313934
-61306632646133626133343132666534666337336239306137663931623637643164326538646138
+64643262313163323531326163323431386532386530326332326130353230653263616139303566
-63313037336235316164316232636262316331356363356438653235346539643065636537366661
+64386535326262326463663734303061613435653964303764386537636132646430626530313331
-37313437386432316336663366633366666434346135356462623635313531333530653866663863
+38323563303738353630653631333535376239393730633535353934316131333039353931616439
-31646438316435373637303733646564316261336233623035353564363664303437373132353166
+35343531353332303932353636643763643035376161643536333365666466616234326237633430
-33646534663130356330663565656663333765653961663835623235633665623464353862616138
+62666464613334633537306431396534306237313032306632323266613134363231313331313139
-37343531366462616439333539306265626164636233616262616563386566396536386264343137
+63316661303830653035376261633539386533373235653839333562656135636433333263303030
-30396434303231366664656334656635616334633066323062313738363431646335643132626331
+65333136653939616231363835616134373038633634656366363266386534343233393730386436
-33663038316165626365346366363731323639653837616163653561623034393064303563353232
+61366536333730646562343333313234663037616461643864613334313464326661366133613966
-30613939353336396365613662393938663933363662656634663465303134393964623738633765
+32343831376266376636646439323566373033643939663261356432623865313833316238363363
-37633832626631363736383436666263356531366237356261666538303863343033356333656138
+34353765396233336263373161323433396437363463626464386463346330653861656162346639
-64366537363939346535313732643965623332396265386636353364316630333130643833383331
+32653738353738396630383431346535616165306638666137616664336235623834616239613131
-34643563386539383235386130323065353835376564613835313934333636316464636463376537
+33643866396530643062633537376565306538313437623462363963363364303738376662323237
-32363565333038643230656266393761313064616537393830343163643535646436616230626166
+37313438633138666434646633333262643333633930396439333565663930613263363638366265
-36373531646536363664663330353864636533656261353331316364633361633934326436323933
+61376165353939633165646330656631323765313430323630303335643362393763663134333333
-39636631623430313934393033373630656132623734363835353735353935393063356261323764
+64383338343939616362363930626235303532666538653561656162396566383865363463373233
-66346265376339623265376666343865343763393833323564643161613861303065346137653339
+62363363623938383234343639656631343831616564633136363332393131356531343137396534
-34326432383230656638303963373832316438666131623539613464636631653365353331346262
+65626639613663323139393061353636313036396163646434613431323864303130663531343964
-37626130343831636133646135323564306538353439636265313239326235343232633136383663
+35363339336436336532303763333564363962313637653438626235353339333438376433396565
-62306561616564623038393631373862346535366135633836353363323330346430613466393463
+32646536383036353864356134623564653436326232643330613736386162656161336165363463
-39313766613064306566316561303761626135613063326536363761626635646134303732356130
+37646138323937343839373736656332643430386630383463393637643463396231303730656530
-38333266363639636565393737323938663938633662653231366164316262666538356162323665
+61633563633338663635353235663931303964333839663333323062646266646466356434343432
-38383263366133326333636466343835373737613333623464383139343863336234373932393239
+38303762336132333132373532623637353731306161316564373935303366666530376639613166
-36653736393732353466656334613364626339356533343363353731353033363730376565353236
+66383533663634333162373436313031643533336436373563356364323236333632303663393761
-66343332306330303662373363326236393333663565383661643763363662396265313037643432
+32343631643566626433306532343764373263666537346538303561323465646237396639363065
-63343336623663326237326365333565346335346630356439303534363561626362363833313239
+64646334633033653962303437396430646332306562623763323733303930623763393537646532
-62366436313466633534343430306461313266376363336536666162306135363434643266373338
+38336439636134666264663339346230326435383365616433363535613235626439623037666234
-65323866323537636465306636646533316466303735386336663234386538303939343661653562
+33393434306662333966316364303261643137356136326637346539363538353465313662393534
-34353564313738643665636337336536343735336436626438653335376532343137613837356363
+34306132613636633936616339636632623733616432323564393565336633366464306661306130
-31353861363733373232363865323164663532646235623533336536333932363133613830373937
+32373465656131346138343537373463663937636337303765366337376666346136643139356234
-38616137646332303934303839633336313239653064626331383432633935353538636132633134
+37376436313834366538636136303965383330663437343363633861396634373764373863613734
-61393230663536633335393330386363333162636661373938343837323736663539393636366532
+39363837613363663337323137616139633935643437353838383735383561346365333565316336
-35623637623262373332373335303833373239323430393766633939633665643236623139363834
+37373231343665343838663235656132613331663638323331363763373539323239306232656338
-30636135303634643765393433393962303138666630343936653234333730336234356234396566
+33643632333863333064666332663437613961393530616330643139663736313733306339346535
-35346331336639333237323136616337623938666561343861393563393633313261633333333434
+31386639363338656462663566313836313066623661653139306662636331373038396636386362
-38633064633538396566373730636632346234653339303533343431643266373266633836633139
+63623431336139316233396432396166633762363662616432616634363639616530663931366230
-33313234366236626338636331346565336138313136623632343735363132623964353066346233
+37396434343039303232363633626532383235376538626438356639386661373330663838663134
-66363833383432316238353364363963323762666562303030663939303162656261663664616632
+65666437363139663932383035303036343633303766343836363435313739363839373938643562
-66396466643763393130313831313438356635346230646335306237363262366132323133646535
+33393733303165373364643961326634396265323962313433653634326436336564313338323631
-30313338653262636136393133306663653331333830626534356536303532633031356435663734
+63666636646638626639323631323335653961363964616461393632316363623162653665333532
-32383035643637313863653537656665366138333933333364646561353333653661306133646361
+34346532313436356235343539343330656235376261326262646364393866343562323663363232
-32326535393333383033313765383031353233323138386365623132646636383037633537386361
+32636539343063323734663732303430306261366361383461363963346633316632623531363530
-65306166613030353033666438343530333330306532323135306363653764613638383461633964
+34343633666136323238336530653335643730326439666637663266363563636466306531393431
-63666239366638663537643636373432303164623735626634383636643836343436346161393365
+33306662653537643835656637636466383965356663373530386431383837373438373436383461
-62336462636534383766656632653833333832393263316633356235373335643436363333393335
+66653735376535656637626231613437336663333132386135666432656638636138383939386665
-62643838303131643265613861373964653464623766363664383330343138653632396365633533
+35346636313439613335653635653861303231663239613563333438303738356439343933326332
-39383535616630663564326536623537323966616436353439343630363738626131326365656132
+35653764613061356465666361323134323331326137353863666364333332326133343039353164
-32383737303733623830353366383062313134396531303831323336326137396335353763353631
+35663264326432366330353230643762623933346363623039656137356135613063366265626334
-36663336353865303562316233643762643563386665393664623138643836633165
+64666539373230376335623365373866613761373637393066343536386134633131393339343439
 39353938643233636562353131336262616439623461363531383064616564616462383833363338
 31396436303330336532333661333263346233626532653437633766306230643032633035636132
 65343938643630303063336664393532363131333532323566306338343837323431373062393265
 31653964376339626632313734366665613563383266376533353664363764366639393563313164
 63646661373461323039636338313165306435323530633535353331343166636565373630333530
 63343262656538616232313633616166623534643834313138326634356334636635363935343030
 61373533373863633630616135626335376136336261646465666135383130633032656465623938
 32306433303937363133623537313535633164346536626465366663623430383639386662396665
 65393038383136306331323035373432666565373934653763336336323131353732356164366262
 64396137353838303934363033393363666632323137663762626163303963386461373166323734
 30336135396262323736313037653133623237336532663765323034383766363932666562343335
 36623731326232653363663937626136386139663630376564333230313632646334353664303137
 62363261316161356164313433666562303965646362626331323538663865386530393864373932
 63326161656561393665396461313938316430353131643035353933326430653439643634316435
 61626437376565363365303061346536386134356335646137626162663739656631343334653866
 31323564663165376266656233316266363232333336356137353665613961303065373735353735
 30663339343631633931376562316262356462656132666661383064333838323061616361383264
 64666664616231363661636337323839613830363366373530313963613132643836353062646561
 32396565313631623639666433663037336338313033373938653336366539653166376237326536
 35323134313962613962663630376663333334373431383731383038323965333736356462303566
 32633164663734366262383264376539313438343530303364623566383564376361343231386564
 39316366373964333837656536393961613466353436333839396331326463356636623738613563
 66333030376364383535633637383439383237636539623930623966356461366436323933646266
 63363264643238333432346432363339323461646533333266666630663161653031626361366364
 62613433333235356438326439306132376234326437316437373537346363303537626632373537
 33346534306363323133353163663932633632626561313566383865393461316163616532666463
 36333966626638336437666338356437643862303332353663303537613933643338396432663336
 65613565353536396132373663663730386664323732616533303037363362663335656236383838
 32666463323463363736316534393538373938303062396564333435383834646664346261383535
 65653931623934363330366361303030306132616637636366313333303664656363623061346638
 38626261313765303832333562383763363935656632343735613935356333346237313436326562
 65353230353537373737326239636465656566346635303134363739333139666362333964393465
 35663837633132663461383036353065656334303730636661363236646462353238343737633635
 38663762656234376233623264343762303434353366363262353232386131303231373832336431
 35636434346531376461393330363837633564613363373235623837373130336465383933353039
 31363732656164333037316438393532336435663035346139373439326565653363343336306638
 30383635656336643934623132383433333962323438303733396131333761643730343464633539
 32306664393738353334333465363966613261333532393130366437353433643439316135613861
 63303863313133333339323665343539343930356637356262316138613733613465343031376538
 30633537396262636262663033656430313831616635363239646334356334323731626530303034
 61343362383935343835316561333231323261643838616435313032313636613933363837343336
 34613131316239653861626434366338343232326531396530373165663931396136373536366339
 64306363393061333333663435666130366630633838333533306163313237653462306461303030
 61353830326337333563356165663062316165373939333938373637636164646536616639366634
 65353035353836356264346234653431383565373637383035323234316637356530346332643635
 37326161333838643166313664616331623532356235323564356261663734613464666133313230
 35393763353639323666313430666164323836633138373961666162316433396364393563626265
 64373565393965313134613365653062323764346432653630643662663930353738353038643661
 34376363633338353132363061363964346162663438626638373032326536363733396634303461
 39306432646362666336393437396330643864353437376235313966393837393435373437313065
 33303439343736626436613663353739656264313639626633323265643966643566623765653739
 31336564393138356265623636306266363861323961636131336331653632626439613461383735
 61616265646365636133393031333733363537366262326632363962333433656366313965303164
 66663335636363343739313838633235323561343833346332353532663065363439303936303266
 34356331633961383264393765316236376130323233346163306536386436366636636139363035
 34623938366266343164633532646161633264316334313930383565623766396165663664393537
 32373266656134333136393965366538316366313735653535626635663462623036323832373964
 61343236616264373563386265656364333230396462653833373338356233393962663965346636
 39313261353263373364373034653338383632643430323335306632353331633939383137396164
 65313166633632366465663466633733663939303039316265313539303564613038313136623861
 37383837653262633365333063646331663562356162333937383538623662633134613364653033
 32613537633639656361336236323039383533303661366432376665323432386464393534633066
 32616666613938373066313866306330393061396263386563306661643831303266633465396364
 30613961636333373462386531373264386332303063663264393861363263666638643836396234
 66613235643365316233353262663739313831373561633339396165386530313538653538633039
 36353364656136306161636135663464343033306161653365653166643762343035336166316164
 66346363646238636438353036353631663135356538663762323030353965386635306630643131
 36666330663063626334386431663635646561643464616636633735626132383961663265626631
 61653365373863313832366538363864333537643336623633343334633333323561373638613961
 34396433366665333531373031336362306364633537646565656630323334643038383535383036
 36633431633930313065663337366533323463333162326435626264393162333162656236306536
 62323032616162313666303265636636323163303234656230313061363862343736373630626435
 37616662393737656132326537336534623633613133343833313135336330303536393364336363
 34383136376363306264306237653236326166323032383939616563353432323565666263646164
 38346663373335636131366236313134623435633766653266383661363532353931323831363065
 34663732643435623436323236316365313861346162613636323138303831353062616263626436
 37356230643833323062336465306366613637633834303762323838656132333130646462653766
 32393532636463363634636665643439366531303162306133626138653839316433343532646530
 36383066303539386136643436393764613937313933396335376262323064663363396133373739
 66643332333135333437623266626563356337366665336632333035323537343762376132373331
 66303630353430313961356234616438363436323938656466636264636364313431396165623530
 33306334656338623237313333386135313961633666363031376334653433366330303262363832
 32333139643434623031653137393134343961353261366464383537643237623166396431356430
 64356435356566336461663766393537303737393139653132363737343238366361316463306136
 38313365333863303734336162666130393531356531623865356431653237343432613238633566
 35323166623935336361653530373466653063383339396466356334663066393539356432616362
 65643536316137326533363930333966666632393334333536326431633962353362313532383033
 63376566643166353136353733353862643033363035353534643062303133633064366332623938
 33363862306263663737616364633462363736666530613562643865393466656131663663663166
 33313839623739386565323133323263613939623838396365313265363466666235316263313535
 64393936396332376635343362396564346634336634336663383566376365353836383739333665
 30653038346161623536633361363961623931326265666566316565636662633032663639373361
 62653665366338656532373734663138353238356535353036336331383635306236656130613633
 34323962316636333664616664313163643061623836316432366664623164303365613161653466
 39333538343435383066343336313564386136363363326231353838373437643136616336393561
 62626430393735326366353839393166313030376435303039643739373337653635316466643236
 64646237303064306663396439316431393931626431616630396538626639636432326561653562
 61363338623863363731356566306231366336363739623735313134613933373238303765386234
 61363034313139626636643533643738663266343235333465663632343533646131393130666632
 31366562643166616430613030333832303636616231646239646435616530343032376663653031
 65636561353734656562626463643361633362313861373233346633636438653335323761373165
 31613130306437346630313466393735643434376133336565656566323132653962636630313431
 30386136643462306261383135336362356161666435323132626637346365396436653835363733
 33393732633464626437366439303931623962313433626533363136653939316633326432356565
 62393332656238303763363732623163653663373536393265343763373365316337323564353064
 31616634653837396438303938343562613761363766616561643032643065346238383734376433
 34613931366331366138346636376566646539303430353665626263643465313133323939613361
 30613063656665666439646362656136656439643934623730383736353031363633653036643161
 39366163663035663135343462336564383862336531656138313431316263653064306334633062
 39386538616361623265663536636130383362633934313764373538643636353237323133303266
 65613765313838363236656334663635636264356331666134643063636334633131353734623632
 38316637306163363839366536333365663337383632376230373137386535303666343362316663
 62663333313634313361373463313566363063363665323565633935303738316135393264663935
 63643662333430633535663130343930373763366430306337656162383537376564653439343430
 35656430366365636165386333663566313034646564636261333361373030663633373861323065
 33363463303135336436613563633135353531623339616530353538356130656263383265353735
 66383964313665623133303130323335393339663331323431363438613637636665623634313862
 66336533626266343831363732613331653431306462313830343830336339663636396566343134
 34396137396461376433653962613361323334393766303330633237623963653033363532613761
 37316139336234366633343064313965393965353461383433366630633935336233383962616662
 30356234663661343232633435346532313737376234623336366161303937623964656331633935
 62333537303536333062613132646237373031663766623730666330666364313634366362363437
 34643634303366633236363665643934323864303732613461326238323637393733393964663562
 30386664636139313335333238373632353936646432326436666665326365646565383766316362
 62376463626338643937663163336166663465353936626332303739393762313235323535366636
 63666438636230383031333838613333633734623134363833383035623564613637633562323263
 64356662376532333261316537656234636338393738613730656530613232373662363431393630
 32323230346332613038373530643563353162366366636338623566373065373637623638333533
 34633333663832393438616266653563623338626431636634303234333636613033333831316264
 36333664366262343261373163373763373036653866663539343337323761633431313435323465
 35386539636338613666393064303363303263323565613837623866633031303834363837626531
 31363838326235643163353731333938386264323730323036333066326463326537613530373832
 65303663613361336565643366316238336135396334303432393936336635306434653930643631
 65393438643630313330383061643031343663393265646664356636313035313961383634316466
 36633139373564623466386264303664373763386534326234323337633732633232346136366133
 61363939336531633265396136623031396534363031393164646661383339653232393463636439
 63306438613837313136646232333434323430643130313937376230613531383461306262366636
 61633137643164653739343663393335373065386339393531343930373161376434356666306132
 35383763613966353662333362396265326532303238656133316332363335346437306164653230
 38333263323064653533373365336166613431303231373062376663343261663135623937373163
 63613239343034303263306138376430613335643862303633343637373430336131623037373537
 64323838306233396435656339306132623266363961643166303733373537333135303665336239
 62623439623865363736343631613432633264363733353834386335333330313933383433346137
 32386432393765356133353736306430376665626665663964313037323136326164653435383332
 61346361313063333137356662383533623734633435646638363537303037393638316439316561
 62376361363532623266383635323330633033333536636262386239353732633237663139393839
 33633537353136376163613334393430663366636533616265633133373563653962393238373862
 61376639663631323231376631353964643162616538353835363937663661313632333333373933
 62323539643630303062333063386163656530623061613664376436623732386666336162396533
 36343563643032356261363234663264303762663439363664353966313561623761336138313165
 37633630663131313565346539396234363565646530386266666232306262663961663636326536
 64653638643935636365313561313964653764363766623433353365376161623333356164346232
 36646630666333383234343931343161646463633134316466616436336634336661356161303764
 62646362333139373761333035386666663561643837623665656630393432303839666666303339
 31646461613838353033343665396239616538356164333436353133653134313738343530313366
 33646530373561633834383865393961353032656339633732646533363535343630613937323461
 65363262643937373638633838343266323966623039616333656538363362333962353066323363
 36326138323132333837343130363735623861616334343130363064303030306361336161303832
 32343534623662393566653131363866343637373930643434326338333362646233383231636235
 33636433346464363766376634636533366163333064313534313831636238376639393533616566
 66333633613261336361366234643966663536653736323235663030626234386166663033376234
 32623435383837336330616235313165636537623462626230346535663662386337366532363733
 39306631626262623630393137653763326234366139626461643931613736653932316466396163
 66643038343336326533366438356562333561616138383330643235663633336262303266613630
 61346237313730303666346639366439313865663430643536336338313130656438613831383633
 30613862363961373732333863636134623431626361396562393437613533383738613761363835
 30363831363963623133653038383031323364346237326164313232363936656132353732373463
 39383737303034346163623737353537646433366539663963313566363032343836383134636131
 66363664336435376563343765623737366632383431656166303639366661633565623331613530
 39396131613161663436333736666536656364663361626633333637316130646266346165323632
 35623062353732616535373166336564383166663861613239373836646637616333613639613862
 66396434333039393135623733653938336233613235363134636362666363303337383335303136
 66313061323938643432623833363930303236633462363663323362626332363565656135363464
 63376266653132333565616363383564353135656431393062393333626336663738366331383737
 30626239366666366263353662333166303063386364366530653731353135313761343461646238
 39353531656232616530633330393130663862623964363565326233643431343432313039333234
 63336232336535373032383933393763623538353061396534373632323165666134333465303066
 39333366303231363863333133643062333230316361623635656663323737643965663563376564
 34343134306333386663356339616461646639303835333333623363383364666436643436343533
 38373364363765343265663138616439316633623865343939353465353466613065653366336230
 39636664303335316538326136356361633366373761333635613139633665616433386162623965
 32636266303031336162626132363931363163313235363865353334616137643934353637643633
 33626137376132353735303737343433323462343830633731396565383037313232313862396330
 65383564386564353634636430393562623833373533626632333530336266633038663333663063
 32313534666162666134373533313730663730333933613762303238323065363236363166326361
 39633938373861366133316261303665356661393830366131343138623138616233623961386233
 65356634333833313061313134333364306238623637333733666366373762383732663037643338
 33333263356262633366346130386337396263646333636134333935343835386366306663663962
 62306232653735656365643533653635366232326638386563393365376464653532656136656266
 62393830656339393033343638643334313765383766336338363331386135373864383939643166
 66316437666434643266326235663632363161663338336434313134643533383634633936336665
 62653638346633336331333364623363376538373562663437373933323932663533333733633036
 39386333653636646133366461633237656166396664623136633863653339633463346135393566
 39316161613532386432306364333662313261346631636430363061376332633431356132663066
 65643137303961383539663733653863346131396232346263393934326239383735353565376533
 33356438336463323531326561326264396635623734663835363039613162613762393662363562
 32646634643934393263616335633933386564373162303766376661323864306230376533633539
 38363032353639633735356266616330396333373665373035643334336566636534383831323862
 33623866386539376131393531653562643430663765316539613438663733366566366533393761
 31303836636533656535383165323232353065663338646337323530363133393263653930666334
 62323836353563366637383138383864323438353339653434306532336164306238633437646235
 64366536393039323339616130663939393562363935633435373932396462323533363838356663
 34343138353130383734363234366661386638653834643661353031353031343562613461393966
 31386463366635353965386462653461653866393864613063636564313638623734663864333334
 36386239336431316338643761363566636137333161663833623633376262623830383961613931
 33666136653631663131626631643266656430356364343138353663326435656663373035333934
 33636339386639666532333763393735386231376666323733333030353239373333383264323130
 35303437616433393231653739653230326263616233373166316237346138336364353339653339
 33353132636262323030643465313231393763613861343734363031386563646337336562336463
 37303232663235313335336435333032626432633531306638313561646236616230306135313639
 63633938326266656138383635653339393238303964386663366638643734633866333432373737
 34636433613566366132613864643135356136336162643066633035663533666237376437333762
 64376262633064333635623438353732376466366338323636643662323234326139316534646533
 33666432303636373963666334376263636634323532643837316361376338616430346464656465
 33373336653539303163336239613038386230363839353532336335303564306363666665333530
 63393863363232653138356234323161323931353932326137653430363836323863393035356230
 61633161623761303335646435383934663135633734346433643639383932643662393264626430
 38303735343033303765316365353834633461386636653662393538643339346230643335656438
 66386164326535646335356661383735366134613939636130383034633734313739333635376233
 35623130353239323237386337623962616536643463376632613631646436353630366137343838
 30376136616261303235643437613864613031346564363465646533363033303566613061363330
 63306530323938383737313438333139363862366331336665623139626135353236316664643661
 38343165383537643136623661663730383635376263346131303332646464376263303464373734
 62366331316166633631663163306237343862306365326339653363336638633830646233363665
 61393461326535653736363834346163343736666665623838643937306539376266393761633263
 39316163356262633137313261353463373166633735663035663135326437383534616239343661
 64633762613833653365353234363539643533353239316238636537323064353337316361353630
 39343463643236656431623861333734313838346430356630656131383763386534373635666331
 34333137346266353265376431623935336366623461666237666334616634613162623761346161
 35366334313837333939306563316236313839623236373234623635656138656165313662656632
 31343961363864346438333932656264363932646132366136663339363930386435666639663135
 61313037303762626537393330366131616532646666613338393032353837643835353436383539
 30323064656161636534653664336232613565626162383035636131393539386633643436653633
 37343339663531396533323432643464393431623338343537646138653236366331366261663631
 65386666313531323031353762363437653561653534316463666539636439383361373861646331
 66626434326566653534393964643938393463303930613734336161356431363230363261383139
 38356663323539636533663237306566623062333036666636663637306332626139333738386365
 38363635386439646363626631366430393636613763646539303939643764643966353461343466
 39363331343334653637656561383638626637646230346665393366313366633366653839663839
 39333263313566626135326633653238626566633536353461376462613462303830353638393161
 36366433656261313436663466366433663138613561386336383262646663313836633564353266
 61623139323961346564623939303339366233643231373031626165646364376138383935626166
 38373833613033633630313965336438626363353031346630376532393932353961363732623631
 35613038366664663337653230653134373861666430343666623165306130306137383163396235
 66633035353534386262663336646361303236623365613736383938333261653462386636376531
 37616539376637386462373063316535636365626634613462623232333435313733326261373230
 66373061336231663230386133613665396263646563376534313830316439623363653165613666
 62373063633863656566643333636461353339396635396331633734623266306430
@@ -1,66 +0,0 @@
 ---
 # FastPass Kubernetes Cluster Variables
 # Cluster Configuration
 cluster_name: "fastpass"
 kubernetes_version: "1.33"
 pod_network_cidr: "10.244.0.0/16"
 service_cidr: "10.96.0.0/12"
 # Control Plane Configuration
 control_plane_endpoint: "{{ cluster_name}}.{{ base_domain}}"
 control_plane_port: "6443"
 # CNI Configuration
 cni_plugin: "calico"
 calico_version: "v3.30.3"
 # Node Labels and Taints
 node_labels:
  zone:
    # Control Plane nodes (masters) - no zone labels needed
    # space-mountain: (control plane - no zone label)
    # big-thunder-mountain: (control plane - no zone label)
    # splash-mountain: (control plane - no zone label)
    # Worker nodes
    haunted-mansion: "backstage"      # Internal/backstage workloads
    peter-pans-flight: "backstage"    # Internal/backstage workloads
    # Future node: "on-stage"         # External/on-stage workloads (future)
  environment: "production"
  cluster: "fastpass"
 # System Configuration
 selinux_mode: "permissive"
 firewall_enabled: false
 # Container Runtime
 container_runtime: "containerd"
 containerd_version: "latest"
 # Repository Configuration
 kubernetes_repo_url: "https://pkgs.k8s.io/core:/stable:/v1.33/rpm/"
 containerd_repo_url: "https://download.docker.com/linux/fedora/docker-ce.repo"
 # Validation
 preflight_checks: true
 validate_cluster: true
 # Kubeconfig Configuration
 kubeconfig_path: "{{ lookup('env', 'HOME') }}/.kube/config"
 # Hostname Configuration for Kubernetes
 # Override hostnames to remove dots for Kubernetes compatibility
 kubernetes_hostnames:
  space-mountain: "space-mountain"
  big-thunder-mountain: "big-thunder-mountain"
  splash-mountain: "splash-mountain"
  haunted-mansion: "haunted-mansion"
  peter-pans-flight: "peter-pans-flight"
 # Generic Kubernetes Prerequisites Configuration
 configure_firewall: true
 configure_ntp: true
 selinux_state: "permissive"
 disable_swap: true 
@@ -1,3 +0,0 @@
 ---
 #freeipa_user: "admin"
 #freeipa_password: "{{ vault_freeipa_password }}"
@@ -1,51 +0,0 @@
 # file: group_vars/hub_cluster/vars
 # Ansible vars template for OpenShift nodes created via Ansible (configure_openshift_cluster).
 # Cluster deployed via Agent Installer and Matchbox (iPXE):
 # Name of the OpenShift Cluster
 cluster_name: "hub"
 # Version of OpenShift to install.  All available versions can be found at: https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/
 cluster_version: "4.18.21"
 # Type of cluster. 
 # - none (Single Node Cluster)
 # - baremetal (Traditional and Compact Cluster)
 platform_type: "none"
 # Number of nodes to deploy
 # - Master: 1; Worker 0 (Single Node Cluster)
 # - Master: 3; Worker 0 (Compact Cluster)
 # - Master: 3; Worker 1-3 (Traditional Cluster)
 master_node_count: 1
 worker_node_count: 0
 # Subnet that cluster nodes will reside on.
 machine_network: "10.1.71.0/24"
 # OpenShift API IP Address (Reference host IP address for Single Node Cluster)
 api_address: "{{ ip_address }}" # "10.1.71.10"
 # OpenShift Apps Wildcard IP Address (Reference host IP address for Single Node Cluster)
 app_address: "{{ ip_address }}" # "10.1.71.10"
 # These are internal networks to the cluster and should not need to be changed.
 cluster_network: "10.128.0.0/14"
 cluster_network_host_prefix: 23
 service_network: "172.30.0.0/16"
 # Static Variables
 openshift_installer_file: "openshift-install-linux.tar.gz"
 openshift_client_file: "openshift-client-linux.tar.gz"
 openshift_installer_download_url: "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/{{ cluster_version }}/{{ openshift_installer_file }}"
 openshift_client_download_url: "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/{{ cluster_version }}/{{ openshift_client_file }}"
@@ -1,51 +0,0 @@
 # file: group_vars/internal_cluster/vars
 # Ansible vars template for OpenShift nodes created via Ansible (configure_openshift_cluster).
 # Cluster deployed via Agent Installer and Matchbox (iPXE):
 # Name of the OpenShift Cluster
 cluster_name: "openshift"
 # Version of OpenShift to install.  All available versions can be found at: https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/
 cluster_version: "4.18.21"
 # Type of cluster. 
 # - none (Single Node Cluster)
 # - baremetal (Traditional and Compact Cluster)
 platform_type: "baremetal"
 # Number of nodes to deploy
 # - Master: 1; Worker 0 (Single Node Cluster)
 # - Master: 3; Worker 0 (Compact Cluster)
 # - Master: 3; Worker 1-3 (Traditional Cluster)
 master_node_count: 3
 worker_node_count: 1
 # Subnet that cluster nodes will reside on.
 machine_network: "10.1.71.0/24"
 # OpenShift API IP Address
 api_address: "10.1.71.69"
 # OpenShift Apps Wildcard IP Address (These will be the same in a Single Node Cluster)
 app_address: "10.1.71.60"
 # These are internal networks to the cluster and should not need to be changed.
 cluster_network: "10.128.0.0/14"
 cluster_network_host_prefix: 23
 service_network: "172.30.0.0/16"
 # Static Variables
 openshift_installer_file: "openshift-install-linux.tar.gz"
 openshift_client_file: "openshift-client-linux.tar.gz"
 openshift_installer_download_url: "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/{{ cluster_version }}/{{ openshift_installer_file }}"
 openshift_client_download_url: "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/{{ cluster_version }}/{{ openshift_client_file }}"
@@ -1,14 +0,0 @@
 ---
 ipaadmin_password: "{{ vault_ipaadmin_password }}"
 ipadm_password: "{{ vault_ipadm_password }}"
 ipaserver_domain: int.mk-labs.cloud
 ipaserver_realm: INT.MK-LABS.CLOUD
 ipaserver_setup_firewalld: true
 ipaserver_setup_dns: true
 ipaserver_forwarders: 1.1.1.1,1.0.0.1
 ipaserver_random_serial_numbers: true
 #dns_ip_addresses: "{{ ansible_hostname }}"
 #dns_reverse_zones:
 #domain: int.mk-labs.cloud
 forwarders: 1.1.1.1,1.0.0.1
 #hostname: "{{ hostname }}"
@@ -1,4 +0,0 @@
 ---
 #freeipa_user: "admin"
 ipaadmin_password: "{{ vault_freeipa_password }}"
 ipaserver_domain: "int.mk-labs.cloud"
@@ -1,9 +0,0 @@
 ---
 # Variables for Matchbox
 assets: "/var/lib/matchbox/assets"
 http_endpoint: "http://matchbox.int.mk-labs.cloud:8080"
 rpc_endpoint: "matchbox.int.mk-labs.cloud:8081"
 # Variables for OpenShift Installer
 #openshift_installer_pull_secret: "{{ vault_pull_secret }}"
 #openshift_installer_ssh_key: "{{ vault_ssh_key }}"
@@ -0,0 +1,12 @@
 ---
 # Single-origin layout: one hostname, Caddy routes /health, /ws, /api/daemon/ws to the backend
 multica_layout: single
 multica_domain: multica.wdw-labs.cloud
 # TLS: acme_http | cloudflare_dns | internal
 caddy_tls_mode: cloudflare_dns
 caddy_acme_email: "ryan.blundon@protonmail.com"
 # Pin for reproducible deploys (git tag + matching image tag)
 # multica_version: v0.4.5
 # multica_image_tag: v0.4.5
@@ -0,0 +1,8 @@
 ---
 # Copy to vault.yml and encrypt:  ansible-vault encrypt group_vars/multica/vault.yml
 # Generate secrets with:  openssl rand -hex 32   (postgres: openssl rand -hex 16)
 vault_multica_jwt_secret: "CHANGEME"
 vault_multica_postgres_password: "CHANGEME"
 vault_multica_vcs_secret_key: "CHANGEME"
 vault_cloudflare_api_token: "CHANGEME"          # only for caddy_tls_mode: cloudflare_dns
 # vault_multica_resend_api_key: ""              # optional: email verification codes
@@ -1,54 +0,0 @@
 # file: group_vars/openshift_cluster_template/vars
 # Ansible vars template for OpenShift nodes created via Ansible (configure_openshift_cluster).
 # Cluster deployed via Agent Installer and Matchbox (iPXE):
 # Role of this node in the cluster. 
 # - master
 # - worker
 # Name of the OpenShift Cluster
 cluster_name: "openshift"
 # Version of OpenShift to install.  All available versions can be found at: https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/
 cluster_version: "4.18.21"
 # Type of cluster. 
 # - none (Single Node Cluster)
 # - baremetal (Traditional and Compact Cluster)
 platform_type: "baremetal"
 # Number of nodes to deploy
 # - Master: 1; Worker 0 (Single Node Cluster)
 # - Master: 3; Worker 0 (Compact Cluster)
 # - Master: 3; Worker 1-3 (Traditional Cluster)
 master_node_count: 3
 worker_node_count: 1
 # Subnet that cluster nodes will reside on.
 machine_network: "10.1.71.0/24"
 # OpenShift API IP Address (Comment out for Single Node Cluster)
 api_address: "10.1.71.69"
 # OpenShift Apps Wildcard IP Address (Comment out for Single Node Cluster)
 app_address: "10.1.71.60"
 # These are internal networks to the cluster and should not need to be changed.
 cluster_network: "10.128.0.0/14"
 cluster_network_host_prefix: 23
 service_network: "172.30.0.0/16"
 # Static Variables
 openshift_installer_file: "openshift-install-linux.tar.gz"
 openshift_client_file: "openshift-client-linux.tar.gz"
 openshift_installer_download_url: "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/{{ cluster_version }}/{{ openshift_installer_file }}"
 openshift_client_download_url: "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/{{ cluster_version }}/{{ openshift_client_file }}"
@@ -1,34 +0,0 @@
 # file: group_vars/prometheus_server/vars
 # Ansible vars template for Prometheus nodes created via Ansible\.
 # Version of Prometheus to install.  All available versions can be found at: https://prometheus.io/download/
 #prometheus_version: "3.5.0"
 #alertmanager_version: "0.28.1"
 # prometheus_nodes:
 # #  - '192.168.1.207'  # Worker node IP
 #   - cinderella-castle  # Master node IP 
 #   - 10.1.71.21
 #   - cinderella-castle.local.mk-labs.cloud
 # Static Variables
 # prometheus_installer_file: "prometheus-{{ prometheus_version }}.linux-amd64.tar.gz"
 # prometheus_installer_download_url: "https://github.com/prometheus/prometheus/releases/download/v{{ prometheus_version }}/{{ prometheus_installer_file }}"
 # Grafana Variables
 #grafana_port: "3000"
 #grafana_url: "{{ inventory_hostname }}.{{ base_domain }}:3000"
 # prometheus_targets:
 #   node:
 #   - targets:
 #     - localhost:9100
 #     labels:
 #       env: mk-labs
 grafana_ini:
  security:
    admin_user: admin
    admin_password: admin
@@ -0,0 +1,19 @@
 ---
 # ansible/group_vars/proxmox/oidc.yml
 #
 # Proxmox OIDC configuration for Authentik integration.
 # Client credentials come from vault.yml.
 proxmox_oidc_realm_name: "authentik"
 proxmox_oidc_issuer_url: "https://authentik.local.mk-labs.cloud/application/o/proxmox/"
 proxmox_oidc_username_claim: "username"
 proxmox_oidc_scopes: "openid email profile"
 proxmox_oidc_autocreate: true
 proxmox_oidc_default_realm: false
 proxmox_oidc_comment: "Authentik SSO"
 # ACL entries - grant your Authentik user admin access
 proxmox_oidc_acl_entries:
  - path: "/"
    user: "rblundon@authentik"
    role: "Administrator"
@@ -0,0 +1,17 @@
 $ANSIBLE_VAULT;1.1;AES256
 64613361376364346139313833613465663361336634326430393261366630306466363935613139
 3634363235343736643230623865386436333734663531310a343965396534336262356234623966
 39353332626662636666383935383530613139626439373664323063633063316264383331646533
 3366333837613731310a656462633638376432326365343135373863313665346566383933353630
 37303565323630633739396433323165326262363863386233343134636236646366633765616432
 33633165613239653762343935386637393437386431383961306436373765343736313634333933
 30376463646464303066613561613564353938306664373464333966383664383034616439343735
 38363966323366343465393530383736393364373361326234356365363163356632393834393464
 39393530356235383830323065636535383138353864373237333164323436623737383435313261
 64366362626232386438376131633533633162356432613835656465623762633464353936303236
 63353633633262633461353963653534663336396134373435303266386162393965343165303335
 65313436666334383965373837376664323861376162373834363439653539383231346634633737
 30626431373739633930313937643938343165383536373465613535366562313439396533643864
 36323032316131333962373732306334373939333363386564653465303261393337333466316130
 32326533353165393635393130396164636363623937313566343333386562616662383038613565
 35386638383365633362
@@ -1,77 +0,0 @@
 ---
 # Variables for Unbound https://github.com/aruhier/ansible-role-unbound
 ### Server ###
 unbound_chroot: '""'
 unbound_num_threads: 2
 unbound_interfaces:
  - "{{ ip_address }}"
 #unbound_do_ip4: "yes"
 unbound_do_ip6: "no"
 #unbound_do_udp: "yes"
 #unbound_do_tcp: "yes"
 # list of strings, as the order impact how policies are interpreted
 unbound_outgoing_policies: []
 unbound_access_control:
  - '192.168.1.0/24 allow'
  - '192.168.3.0/24 allow'
  - '192.168.5.0/24 allow'
  - '192.168.10.0/24 allow'
  - '10.1.71.0/24 allow'
 unbound_access_control_tag: []
 unbound_access_control_tag_action: []
 unbound_access_control_tag_data: []
 unbound_access_control_view: []
 unbound_private_addresses:
  - '192.168.1.0/24'
  - '192.168.3.0/24'
  - '192.168.5.0/24'
  - '192.168.10.0/24'
  - '10.1.71.0/24'
 unbound_private_domains:
  - 'mk-labs.cloud'
 unbound_domains_insecure: []
 unbound_do_not_query_addresses: []
 unbound_local_zones: []
 unbound_local_datas: []
 unbound_local_data_ptrs: []
 unbound_local_zone_tags: []
 unbound_local_zone_overrides: []
 unbound_trust_anchors: []
 unbound_trusted_keys_files: []
 ### Remote Control ###
 unbound_control_enable: "no"
 unbound_control_interface: []
 ### Stub, forward zones and others ###
 unbound_rpz_zones: []  # list of dicts
 unbound_stub_zones: []  # list of dicts
 unbound_forward_zones:
  - name: 'int.mk-labs.cloud'
    forward-addr:
      - '10.1.71.5'
  - name: '.'
    forward-addr:
      - '1.0.0.1@53#one.one.one.one'
      - '1.1.1.1@53#one.one.one.one'
 # For stub and forward zones, if a key inside one of the dict is an iterable,
 # it will iterate inside it to duplicate the option with all the contained
 # values.
 # Can be useful in case of multiple forward-addr for a same zone:
 #   Example:
 #     unbound_forward_zones:
 #       - {name: "test.tld", "forward-addr": ["192.0.2.5", "192.0.2.6"]}
 unbound_views: []  # list of strings, as options can be multiples
@@ -0,0 +1,235 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: ansible/host_vars/astro_orbiter/vars.yml
 # HOST: astro-orbiter (10.1.71.130)
 # ROLE: llama.cpp LLM inference host — Ryzen 7 5800XT / RTX 3090 (ATX rebuild,
 #       2026-08-04). Superseded the prior AMD RX 5700 / Ollama config below;
 #       drive was transplanted into new hardware, not reinstalled.
 # ------------------------------------------------------------------------------
 ansible_host: 10.1.71.130
 ansible_user: jarvis
 ansible_ssh_private_key_file: ~/.ssh/id_jarvis
 ansible_become: true
 # LVM root expansion — xlarge template uses sda3 partition, standard VG/LV names
 common_expand_root_lvm: true
 common_root_pv: /dev/sda3
 common_root_vg: ubuntu-vg
 common_root_lv: ubuntu-lv
 # --- Staged GGUF models for the llama.cpp router (:8002) ---------------------
 # Data-driven list consumed by roles/llm-inference-multimodel tasks/models.yml
 # (loop -> tasks/stage_model.yml). Each entry is idempotently staged into
 # /opt/models: stat + EXACT-size check vs HF manifest; skip (no download, no
 # restart) when present + size matches. Source repos are public bartowski GGUFs
 # on HuggingFace (no auth). A router restart is notified ONLY when a new GGUF
 # is actually downloaded.
 # Added 2026-08-12 (War Machine): codify Phi-3.5-mini-instruct-Q8_0 and
 # Meta-Llama-3.1-8B-Instruct-Q4_K_M as router models alongside the production
 # Qwen3.6-35B-A3B-UD-Q4_K_S. The live files were already present/correct on
 # astro-orbiter; this pass codifies them. Future adds = append to this list.
 # Router --models-max override for astro-orbiter.
 # Default in defaults/main.yml is 1 (conservative). Bumped to 4 on 2026-08-12
 # (t_33acbb2e) so the router can keep more than one GGUF resident on-demand
 # and LRU-evict when needed.
 #
 # VRAM NOTE (t_33acbb2e, updated t_55c164f5, updated t_34b96e83, updated t_f5f7e9ad, updated t_441470b9, updated t_c5cef2b2):
 # With models-max=4 and all 6 GGUFs registered, worst case is all 6 loaded simultaneously:
 #   Qwen3.8-27B Q4_K_M:               ~20.0GB (weights ~17.1GB + KV ~2.9GB @ 65536 ctx, q4_0) ← CORRECTED (ctx rolled back from 128K to 65536, t_c9fed26c 2026-08-18)
 #   Phi-3.5-mini-instruct Q8_0:       ~4.3GB  (weights ~3.8GB + KV ~0.5GB @ 32K ctx)
 #   Meta-Llama-3.1-8B Q4_K_M:        ~5.6GB  (weights ~4.6GB + KV ~0.2GB @ 8K ctx)
 #   Qwen2.5-Coder-14B Q4_K_M:        ~9.0GB  (weights ~8.4GB + KV ~0.6GB @ 16K ctx)
 #   nomic-embed-text-v1.5 Q4_K_M:    ~0.09GB (~84MB, embedding only — no KV cache)
 #   Qwen3-8B Q4_K_M:                 ~5.5GB  (weights ~4.68GB + KV ~0.5GB @ 32K ctx, q4_0)
 #   Total worst-case:                 ~44.5GB  >>  24GB RTX 3090
 #
 # OOM RISK: Full co-residency is impossible on 24GB. LRU eviction prevents this
 # in practice: models-max=4 means the router can REGISTER 6 models but only keeps
 # up to 4 LOADED simultaneously — the router will evict the LRU model when a new
 # one is needed. nomic-embed-text-v1.5 is pinned via sleep-idle-seconds=-1 and
 # load-on-startup=true but it uses only ~84MB, so it never meaningfully changes
 # the budget. In single-user homelab operation, only one generative model is active
 # at a time alongside the always-resident embedding model.
 # Qwen3.8-27B alone uses ~17,804 MiB (weights+KV @ 65536 ctx); co-residency
 # with Coder (~9GB) = ~27GB > 24GB. LRU eviction handles this automatically.
 # Ryan should be aware this means model-switching always incurs a ~30-60s
 # cold-load latency when switching between Qwen3.8-27B and any other model.
 # Proceeding to models-max=4 as instructed; flagged for Ryan's attention.
 # Router --models-max override for astro-orbiter.
 # UPDATED (t_f5f7e9ad, 2026-08-16): Set to 2 because Qwen3.8-27B-Q4_K_M
 # uses 17,804 MiB at 65536 ctx. Only nomic-embed (558MB, pinned) and ONE
 # generative model can be resident simultaneously. Co-residency of Qwen3.8
 # with any auxiliary model (Phi 8.3GB, Llama 5.9GB, Coder 9GB) exceeds 24GB.
 # models-max=2: slot 1 = nomic-embed (pinned, always loaded), slot 2 = LRU
 # generative model (Qwen3.8 primary, cold-loaded on first request ~30-60s;
 # auxiliary models evict it on demand, and vice versa).
 # NOTE: Qwen3.8 does NOT have load-on-startup — it loads on first request.
 # This avoids an LRU eviction race with nomic-embed at startup.
 # UPDATED (t_72646029, 2026-08-17): CPU offload for Coder + Llama changes the
 # constraint. Coder and Llama now use CPU inference (n-gpu-layers=0). GPU-resident
 # VRAM: Qwen3.8 (~17,804 MiB at 65536 ctx) + nomic-embed (558 MiB, pinned) plus
 # the CUDA-context buffers llama.cpp 6ea215d allocates for the CPU models (~1.4-1.7GB
 # each) = ~20,004 MiB steady-state, below the 24,576 MiB physical limit.
 # CORRECTED (t_c5cef2b2, 2026-08-19): ctx-size was rolled back from 131072 to 65536
 # (t_c9fed26c 2026-08-18). Qwen3.8 VRAM at 65536: 17,804 MiB (not 20,302 MiB).
 # models-max raised to 4: nomic (slot 1, pinned) + Qwen3.8 (slot 2, GPU) +
 # Llama (slot 3, CPU) + Coder (slot 4, CPU). Phi (GPU, ~8.3GB) and new
 # Qwen3-8B (GPU, ~5.5GB) can also be requested but evict Qwen3.8 due to VRAM.
 # models-max=4 is required so CPU-offloaded models count as loaded without
 # evicting Qwen3.8.
 llm_router_models_max: 4
 llm_staged_models:
  - filename: "Phi-3.5-mini-instruct-Q8_0.gguf"
    url: "https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF/resolve/main/Phi-3.5-mini-instruct-Q8_0.gguf"
    size_bytes: 4061222688
    source_repo: "bartowski/Phi-3.5-mini-instruct-GGUF"
  - filename: "Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
    url: "https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
    size_bytes: 4920739232
    source_repo: "bartowski/Meta-Llama-3.1-8B-Instruct-GGUF"
  - filename: "Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
    url: "https://huggingface.co/bartowski/Qwen2.5-Coder-14B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
    size_bytes: 8988111072
    source_repo: "bartowski/Qwen2.5-Coder-14B-Instruct-GGUF"
  - filename: "nomic-embed-text-v1.5-Q4_K_M.gguf"
    url: "https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF/resolve/main/nomic-embed-text-v1.5.Q4_K_M.gguf"
    size_bytes: 84106624
    source_repo: "nomic-ai/nomic-embed-text-v1.5-GGUF"
  # Added t_c5cef2b2 (2026-08-19, War Machine): Qwen3-8B dense 8B model for
  # aux tasks (routing, rewriting, structured extraction, tool-call construction).
  # Source: bartowski/Qwen_Qwen3-8B-GGUF (public, no auth). HF filename is
  # Qwen_Qwen3-8B-Q4_K_M.gguf; stored locally as Qwen3-8B-Q4_K_M.gguf.
  # Exact size verified from HF manifest (content-length): 5,027,784,224 bytes.
  # VRAM: ~4.68GB weights + ~0.5GB KV @ 32K ctx (q4_0) ≈ 5.2GB total.
  # Thinking mode ON by default; use /no_think for latency-sensitive aux tasks.
  - filename: "Qwen3-8B-Q4_K_M.gguf"
    url: "https://huggingface.co/bartowski/Qwen_Qwen3-8B-GGUF/resolve/main/Qwen_Qwen3-8B-Q4_K_M.gguf"
    size_bytes: 5027784224
    source_repo: "bartowski/Qwen_Qwen3-8B-GGUF"
 # --- deploy-vllm role: vllm_models override (t_r1d32b_swap, 2026-09-01) -----
 # Ansible's hash_behaviour is "replace" (see ansible.cfg) — a host_vars list
 # variable REPLACES the role default list wholesale, it does not deep-merge.
 #
 # SWAP (Ryan direction, 2026-09-01): Qwen2.5-32B-Instruct-AWQ retired,
 # replaced with DeepSeek-R1-Distill-Qwen-32B-AWQ, max_model_len=32768.
 # "Single model only" — nomic-embed-text-v1.5 (embedding, :8020) and
 # Qwen3-8B-AWQ (aux, :8010, already disabled) are BOTH disabled here.
 # DeepSeek gets the full 24GB card to itself. Nothing in production
 # consumed nomic-embed at the time of this swap (Hindsight uses its own
 # bundled 384-dim embedder; OpenViking pointed at the old llama-swap
 # endpoint, already stopped) — confirmed with Ryan before disabling.
 #
 # Model choice: casperhansen/deepseek-r1-distill-qwen-32b-awq — same
 # quantizer/toolchain (AutoAWQ) as the outgoing Qwen2.5-32B-Instruct-AWQ,
 # widely used, 4-bit GEMM AWQ, ~19.3GB on disk (4 safetensors shards).
 # Architecture: Qwen2ForCausalLM (DeepSeek-R1 distilled onto Qwen2.5-32B
 # base) — same vLLM code path as the outgoing model, no new serving
 # support needed. Native max_position_embeddings=131072; we cap at 32768
 # per the task's explicit max-model-len requirement.
 #
 # VRAM math: ~19.3GB weights (4-bit AWQ) + KV cache at 32768 ctx (GQA,
 # 8 KV heads, 128 head_dim, 64 layers, fp16 KV by default) ≈ 19.3GB +
 # ~4GB KV+overhead ≈ 23.3GB — tight but the FULL 24GB card is now
 # available (no co-resident nomic-embed/Qwen3-8B taking a share, unlike
 # the outgoing Qwen2.5-32B config). gpu_memory_utilization=0.95 (role
 # default) + enforce_eager retained as the proven-stable mitigation from
 # t_e6facb19/t_ca1af9fb (avoids CUDA graph capture VRAM spike; this host's
 # only validated way to avoid crash-loop-to-stabilize behavior on this
 # card). If 0.95 OOMs at 32768 ctx once tested live, drop to 0.90 next
 # (documented fallback, same pattern as the outgoing model).
 #
 # DeepSeek-R1 output note: reasoning traces stream in <think> tags before
 # the final answer — this is expected R1-distill behavior, not a bug.
 # Model card recommends temperature 0.5-0.7 (not 0, not vLLM's greedy
 # default) to avoid repetition/incoherence; not set here (server-side
 # default), left to be set client-side per the model card's guidance —
 # flagging for whoever wires this into Hermes profile configs next.
 vllm_models:
  - id: "Gemma-4-26B-A4B-it-AWQ"
    hf_repo: "cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit"
    role: primary
    # NO quantization field set (unlike the AutoAWQ-quantized DeepSeek/
    # Qwen2.5 models above) — live test (2026-09-01) found this repo's
    # config.json declares quant_method: "compressed-tensors" (llm-compressor
    # tool output, not classic AutoAWQ), even though the repo name says
    # "AWQ-4bit". Passing --quantization awq explicitly caused a hard
    # pydantic ValidationError at every single startup attempt: "Quantization
    # method specified in the model config (compressed-tensors) does not
    # match the quantization method specified in the `quantization` argument
    # (awq)." vLLM auto-detects the quant method correctly from the model's
    # own config.json when --quantization is omitted — confirmed fix, clean
    # start. Lesson: don't trust a HF repo's naming convention ("...-AWQ...")
    # for the `quantization:` field here — check config.json's quant_method.
    port: 8000
    # Ryan direction (2026-09-01, t_gemma4_swap): DeepSeek-R1-Distill-Qwen-32B
    # retired after confirming its `auto` tool-choice reliability is a known,
    # documented DeepSeek-R1-distillation limitation (trained on pure
    # reasoning traces, no function-calling data — GitHub-confirmed upstream,
    # not a vLLM config gap). Replaced with Gemma 4 26B A4B (Google,
    # Apache 2.0, US-origin — matches Ryan's standing model-origin
    # preference, unlike Qwen/DeepSeek). Chose MoE (26B A4B, 3.8B active)
    # over the dense 31B variant: ~3.7GB smaller on-disk AWQ footprint
    # (17.2GB vs 20.9GB) buys more KV-cache headroom on this tight 24GB
    # card, and decode should be faster (memory-bandwidth-bound on active
    # params, not total params). Tradeoff accepted: MoE scores lower than
    # dense on the Tau2 tool-use benchmark (68.2% vs 76.9%) but still beats
    # every other size in the family except the 31B on most reasoning
    # benchmarks. Model choice: cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit —
    # AutoAWQ 4-bit group_size=32, MoE expert layers (gate/up/down/router)
    # explicitly excluded from quantization ("ignore" list in config.json)
    # per standard llm-compressor MoE quant practice — only the dense
    # attention/projection layers are 4-bit, experts stay higher precision.
    # Native architecture: Gemma4ForConditionalGeneration (registered
    # natively in this host's installed vLLM 0.28.0 — vllm/model_executor/
    # models/registry.py line 415 — no plugin/trust-remote-code needed).
    # Native max_position_embeddings: 262144 (256K) — Hermes's 64K floor is
    # comfortably covered without any context-extension trick.
    max_model_len: 65536
    # VRAM math (not yet live-validated — see swap validation log below
    # once run): AWQ weights ~17.2GB on disk (dense attn 4-bit + MoE
    # experts higher-precision, per config.json's compressed-tensors
    # ignore list). Starting the KV cache dtype at int4_per_token_head
    # from the outset (rather than fp16 -> fp8 -> int4 trial-and-error like
    # the DeepSeek swap) since that same escalation pattern is expected to
    # repeat on this VRAM-constrained card for any 20+ GB model at >32K ctx.
    kv_cache_dtype: int4_per_token_head
    gpu_memory_utilization: 0.95
    enforce_eager: true
    # Native tool-calling + reasoning support (no `hermes` workaround
    # needed, unlike DeepSeek-R1-Distill): Gemma4EngineToolParser and
    # Gemma4ParserReasoningAdapter are both registered natively in this
    # host's vLLM 0.28.0 (vllm/tool_parsers/__init__.py,
    # vllm/reasoning/__init__.py) — purpose-built for this model's actual
    # output format, not a same-family approximation.
    enable_auto_tool_choice: true
    tool_call_parser: gemma4
    reasoning_parser: gemma4
    enabled: true
  - id: "Qwen3-8B-AWQ"
    hf_repo: "Qwen/Qwen3-8B-AWQ"
    role: aux
    quantization: awq
    port: 8010
    max_model_len: 32768
    gpu_memory_utilization: 0.15
    enforce_eager: true
    enabled: false   # single-model deployment — see swap note above
  - id: "nomic-embed-text-v1.5"
    hf_repo: "nomic-ai/nomic-embed-text-v1.5"
    role: embedding
    quantization: none
    port: 8020
    max_model_len: 2048
    gpu_memory_utilization: 0.05
    trust_remote_code: true
    enabled: false   # single-model deployment — see swap note above
 # --- deploy-vllm role: boot persistence (unchanged) -------------------------
 # Still permanent/boot-persistent — same policy as the outgoing Qwen2.5-32B
 # deployment (t_5508360a), just now serving one model instead of two.
 vllm_service_enabled: true
 vllm_service_state: started
@@ -0,0 +1,16 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: ansible/host_vars/astro_orbiter/vars.yml
 # HOST: astro-orbiter (10.1.71.130)
 # ROLE: Ollama inference host with AMD RX 5700 GPU passthrough
 # ------------------------------------------------------------------------------
 ansible_host: 10.1.71.131
 ansible_user: wed
 ansible_become: true
 # LVM root expansion — xlarge template uses sda3 partition, standard VG/LV names
 common_expand_root_lvm: true
 common_root_pv: /dev/sda3
 common_root_vg: ubuntu-vg
 common_root_lv: ubuntu-lv
@@ -1,49 +0,0 @@
 ---
 # file: host_vars/vm_template/vars
 # Ansible vars template for hosts created via cloning.
 # Supported hypervisors:
 # - Proxmox
 platform: "proxmox"
 # This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
 proxmox_clone_node: "fantasyland"
 # Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
 # Current OS offerings are:
 # - Fedora (42)
 # Current VM sizes are:
 # - Small: 2 cores, 2GB memory, 8 GiB virtual disk
 # - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
 # - Large: 4 cores, 4GB memory, 32 GiB virtual disk
 # - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
 # - Xlarge Plus: 4 cores, 8GB memory, 128 GiB virtual disk
 vm_clone_source: "fedora-42-xlarge-plus"
 # Proxmox storage target.
 vm_storage: "general"
 # Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
 # and backup groups. (ha_group will be factored out in the next functionality update.)
 ha_group: "fantasyland"
 proxmox_host_target: "fantasyland"
 # Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
 # sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
 ip_address: 10.1.71.21
 # Software
 # Future enhancement will allow specification of additional software to automatically deploy to
 # the VM after creation.
 #terraform_version: "1.11.3"
 # ---
 hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn
@@ -0,0 +1,12 @@
 ---
 # file: host_vars/guest-relations/vars
 # guest-relations — Guest Relations
 # VM provisioned by Terraform (pre-pipeline bootstrap)
 # ─── Network ─────────────────────────────────────────────────────────────────
 ip_address: 10.1.71.40
 # ─── Application ─────────────────────────────────────────────────────────────
 app_role: identity_provider
 app_name: authentik
 app_deployment: docker_compose
@@ -0,0 +1,26 @@
 ---
 # Paperclip Docker deployment
 paperclip_domain: cranium-command.wdw-labs.cloud
 paperclip_base_path: /opt/paperclip
 paperclip_port: 3100
 certbot_email: ryan.blundon@protonmail.com
 # vLLM inference endpoint (local)
 vllm_endpoint: "http://127.0.0.1:8000"
 hermes_model:
  provider: custom
  default: Gemma-4-26B-A4B-Hermes-VLM-MLX      # from http://fantasmic.wdw-labs.cloud:8000/v1/models
  base_url: http://fantasmic.wdw-labs.cloud:8000/v1
  api_key: "{{ vault_fantasmic_api_key }}"
 # # DNS entries (if using your existing dns-manager role)
 # dns_entries:
 #   - name: illuminations
 #     domain: wdw-labs.cloud
 #     type: A
 #     value: "{{ ansible_host }}"
 #   - name: muppet-vision
 #     domain: wdw-labs.cloud
 #     type: A
 #     value: "{{ ansible_host }}"
@@ -0,0 +1,244 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: ansible/host_vars/illuminations/vars.yml
 # HOST: illuminations (10.1.71.130)
 # ROLE: llama.cpp LLM inference host — Ryzen 7 5800XT / RTX 3090 (ATX rebuild,
 #       2026-08-04). Superseded the prior AMD RX 5700 / Ollama config below;
 #       drive was transplanted into new hardware, not reinstalled.
 # ------------------------------------------------------------------------------
 ansible_host: 10.1.71.130
 ansible_user: jarvis
 ansible_ssh_private_key_file: ~/.ssh/id_jarvis
 ansible_become: true
 # LVM root expansion — xlarge template uses sda3 partition, standard VG/LV names
 common_expand_root_lvm: true
 common_root_pv: /dev/sda3
 common_root_vg: ubuntu-vg
 common_root_lv: ubuntu-lv
 # --- Staged GGUF models for the llama.cpp router (:8002) ---------------------
 # Data-driven list consumed by roles/llm-inference-multimodel tasks/models.yml
 # (loop -> tasks/stage_model.yml). Each entry is idempotently staged into
 # /opt/models: stat + EXACT-size check vs HF manifest; skip (no download, no
 # restart) when present + size matches. Source repos are public bartowski GGUFs
 # on HuggingFace (no auth). A router restart is notified ONLY when a new GGUF
 # is actually downloaded.
 # Added 2026-08-12 (War Machine): codify Phi-3.5-mini-instruct-Q8_0 and
 # Meta-Llama-3.1-8B-Instruct-Q4_K_M as router models alongside the production
 # Qwen3.6-35B-A3B-UD-Q4_K_S. The live files were already present/correct on
 # astro-orbiter; this pass codifies them. Future adds = append to this list.
 # Router --models-max override for astro-orbiter.
 # Default in defaults/main.yml is 1 (conservative). Bumped to 4 on 2026-08-12
 # (t_33acbb2e) so the router can keep more than one GGUF resident on-demand
 # and LRU-evict when needed.
 #
 # VRAM NOTE (t_33acbb2e, updated t_55c164f5, updated t_34b96e83, updated t_f5f7e9ad, updated t_441470b9, updated t_c5cef2b2):
 # With models-max=4 and all 6 GGUFs registered, worst case is all 6 loaded simultaneously:
 #   Qwen3.8-27B Q4_K_M:               ~20.0GB (weights ~17.1GB + KV ~2.9GB @ 65536 ctx, q4_0) ← CORRECTED (ctx rolled back from 128K to 65536, t_c9fed26c 2026-08-18)
 #   Phi-3.5-mini-instruct Q8_0:       ~4.3GB  (weights ~3.8GB + KV ~0.5GB @ 32K ctx)
 #   Meta-Llama-3.1-8B Q4_K_M:        ~5.6GB  (weights ~4.6GB + KV ~0.2GB @ 8K ctx)
 #   Qwen2.5-Coder-14B Q4_K_M:        ~9.0GB  (weights ~8.4GB + KV ~0.6GB @ 16K ctx)
 #   nomic-embed-text-v1.5 Q4_K_M:    ~0.09GB (~84MB, embedding only — no KV cache)
 #   Qwen3-8B Q4_K_M:                 ~5.5GB  (weights ~4.68GB + KV ~0.5GB @ 32K ctx, q4_0)
 #   Total worst-case:                 ~44.5GB  >>  24GB RTX 3090
 #
 # OOM RISK: Full co-residency is impossible on 24GB. LRU eviction prevents this
 # in practice: models-max=4 means the router can REGISTER 6 models but only keeps
 # up to 4 LOADED simultaneously — the router will evict the LRU model when a new
 # one is needed. nomic-embed-text-v1.5 is pinned via sleep-idle-seconds=-1 and
 # load-on-startup=true but it uses only ~84MB, so it never meaningfully changes
 # the budget. In single-user homelab operation, only one generative model is active
 # at a time alongside the always-resident embedding model.
 # Qwen3.8-27B alone uses ~17,804 MiB (weights+KV @ 65536 ctx); co-residency
 # with Coder (~9GB) = ~27GB > 24GB. LRU eviction handles this automatically.
 # Ryan should be aware this means model-switching always incurs a ~30-60s
 # cold-load latency when switching between Qwen3.8-27B and any other model.
 # Proceeding to models-max=4 as instructed; flagged for Ryan's attention.
 # Router --models-max override for astro-orbiter.
 # UPDATED (t_f5f7e9ad, 2026-08-16): Set to 2 because Qwen3.8-27B-Q4_K_M
 # uses 17,804 MiB at 65536 ctx. Only nomic-embed (558MB, pinned) and ONE
 # generative model can be resident simultaneously. Co-residency of Qwen3.8
 # with any auxiliary model (Phi 8.3GB, Llama 5.9GB, Coder 9GB) exceeds 24GB.
 # models-max=2: slot 1 = nomic-embed (pinned, always loaded), slot 2 = LRU
 # generative model (Qwen3.8 primary, cold-loaded on first request ~30-60s;
 # auxiliary models evict it on demand, and vice versa).
 # NOTE: Qwen3.8 does NOT have load-on-startup — it loads on first request.
 # This avoids an LRU eviction race with nomic-embed at startup.
 # UPDATED (t_72646029, 2026-08-17): CPU offload for Coder + Llama changes the
 # constraint. Coder and Llama now use CPU inference (n-gpu-layers=0). GPU-resident
 # VRAM: Qwen3.8 (~17,804 MiB at 65536 ctx) + nomic-embed (558 MiB, pinned) plus
 # the CUDA-context buffers llama.cpp 6ea215d allocates for the CPU models (~1.4-1.7GB
 # each) = ~20,004 MiB steady-state, below the 24,576 MiB physical limit.
 # CORRECTED (t_c5cef2b2, 2026-08-19): ctx-size was rolled back from 131072 to 65536
 # (t_c9fed26c 2026-08-18). Qwen3.8 VRAM at 65536: 17,804 MiB (not 20,302 MiB).
 # models-max raised to 4: nomic (slot 1, pinned) + Qwen3.8 (slot 2, GPU) +
 # Llama (slot 3, CPU) + Coder (slot 4, CPU). Phi (GPU, ~8.3GB) and new
 # Qwen3-8B (GPU, ~5.5GB) can also be requested but evict Qwen3.8 due to VRAM.
 # models-max=4 is required so CPU-offloaded models count as loaded without
 # evicting Qwen3.8.
 llm_router_models_max: 4
 llm_staged_models:
  - filename: "Phi-3.5-mini-instruct-Q8_0.gguf"
    url: "https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF/resolve/main/Phi-3.5-mini-instruct-Q8_0.gguf"
    size_bytes: 4061222688
    source_repo: "bartowski/Phi-3.5-mini-instruct-GGUF"
  - filename: "Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
    url: "https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
    size_bytes: 4920739232
    source_repo: "bartowski/Meta-Llama-3.1-8B-Instruct-GGUF"
  - filename: "Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
    url: "https://huggingface.co/bartowski/Qwen2.5-Coder-14B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
    size_bytes: 8988111072
    source_repo: "bartowski/Qwen2.5-Coder-14B-Instruct-GGUF"
  - filename: "nomic-embed-text-v1.5-Q4_K_M.gguf"
    url: "https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF/resolve/main/nomic-embed-text-v1.5.Q4_K_M.gguf"
    size_bytes: 84106624
    source_repo: "nomic-ai/nomic-embed-text-v1.5-GGUF"
  # Added t_c5cef2b2 (2026-08-19, War Machine): Qwen3-8B dense 8B model for
  # aux tasks (routing, rewriting, structured extraction, tool-call construction).
  # Source: bartowski/Qwen_Qwen3-8B-GGUF (public, no auth). HF filename is
  # Qwen_Qwen3-8B-Q4_K_M.gguf; stored locally as Qwen3-8B-Q4_K_M.gguf.
  # Exact size verified from HF manifest (content-length): 5,027,784,224 bytes.
  # VRAM: ~4.68GB weights + ~0.5GB KV @ 32K ctx (q4_0) ≈ 5.2GB total.
  # Thinking mode ON by default; use /no_think for latency-sensitive aux tasks.
  - filename: "Qwen3-8B-Q4_K_M.gguf"
    url: "https://huggingface.co/bartowski/Qwen_Qwen3-8B-GGUF/resolve/main/Qwen_Qwen3-8B-Q4_K_M.gguf"
    size_bytes: 5027784224
    source_repo: "bartowski/Qwen_Qwen3-8B-GGUF"
 # --- deploy-vllm role: vllm_models override (t_r1d32b_swap, 2026-09-01) -----
 # Ansible's hash_behaviour is "replace" (see ansible.cfg) — a host_vars list
 # variable REPLACES the role default list wholesale, it does not deep-merge.
 #
 # SWAP (Ryan direction, 2026-09-01): Qwen2.5-32B-Instruct-AWQ retired,
 # replaced with DeepSeek-R1-Distill-Qwen-32B-AWQ, max_model_len=32768.
 # "Single model only" — nomic-embed-text-v1.5 (embedding, :8020) and
 # Qwen3-8B-AWQ (aux, :8010, already disabled) are BOTH disabled here.
 # DeepSeek gets the full 24GB card to itself. Nothing in production
 # consumed nomic-embed at the time of this swap (Hindsight uses its own
 # bundled 384-dim embedder; OpenViking pointed at the old llama-swap
 # endpoint, already stopped) — confirmed with Ryan before disabling.
 #
 # Model choice: casperhansen/deepseek-r1-distill-qwen-32b-awq — same
 # quantizer/toolchain (AutoAWQ) as the outgoing Qwen2.5-32B-Instruct-AWQ,
 # widely used, 4-bit GEMM AWQ, ~19.3GB on disk (4 safetensors shards).
 # Architecture: Qwen2ForCausalLM (DeepSeek-R1 distilled onto Qwen2.5-32B
 # base) — same vLLM code path as the outgoing model, no new serving
 # support needed. Native max_position_embeddings=131072; we cap at 32768
 # per the task's explicit max-model-len requirement.
 #
 # VRAM math: ~19.3GB weights (4-bit AWQ) + KV cache at 32768 ctx (GQA,
 # 8 KV heads, 128 head_dim, 64 layers, fp16 KV by default) ≈ 19.3GB +
 # ~4GB KV+overhead ≈ 23.3GB — tight but the FULL 24GB card is now
 # available (no co-resident nomic-embed/Qwen3-8B taking a share, unlike
 # the outgoing Qwen2.5-32B config). gpu_memory_utilization=0.95 (role
 # default) + enforce_eager retained as the proven-stable mitigation from
 # t_e6facb19/t_ca1af9fb (avoids CUDA graph capture VRAM spike; this host's
 # only validated way to avoid crash-loop-to-stabilize behavior on this
 # card). If 0.95 OOMs at 32768 ctx once tested live, drop to 0.90 next
 # (documented fallback, same pattern as the outgoing model).
 #
 # DeepSeek-R1 output note: reasoning traces stream in <think> tags before
 # the final answer — this is expected R1-distill behavior, not a bug.
 # Model card recommends temperature 0.5-0.7 (not 0, not vLLM's greedy
 # default) to avoid repetition/incoherence; not set here (server-side
 # default), left to be set client-side per the model card's guidance —
 # flagging for whoever wires this into Hermes profile configs next.
 vllm_models:
  - id: "Gemma-4-26B-A4B-it-AWQ"
    hf_repo: "cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit"
    role: primary
    # NO quantization field set (unlike the AutoAWQ-quantized DeepSeek/
    # Qwen2.5 models above) — live test (2026-09-01) found this repo's
    # config.json declares quant_method: "compressed-tensors" (llm-compressor
    # tool output, not classic AutoAWQ), even though the repo name says
    # "AWQ-4bit". Passing --quantization awq explicitly caused a hard
    # pydantic ValidationError at every single startup attempt: "Quantization
    # method specified in the model config (compressed-tensors) does not
    # match the quantization method specified in the `quantization` argument
    # (awq)." vLLM auto-detects the quant method correctly from the model's
    # own config.json when --quantization is omitted — confirmed fix, clean
    # start. Lesson: don't trust a HF repo's naming convention ("...-AWQ...")
    # for the `quantization:` field here — check config.json's quant_method.
    port: 8000
    # Ryan direction (2026-09-01, t_gemma4_swap): DeepSeek-R1-Distill-Qwen-32B
    # retired after confirming its `auto` tool-choice reliability is a known,
    # documented DeepSeek-R1-distillation limitation (trained on pure
    # reasoning traces, no function-calling data — GitHub-confirmed upstream,
    # not a vLLM config gap). Replaced with Gemma 4 26B A4B (Google,
    # Apache 2.0, US-origin — matches Ryan's standing model-origin
    # preference, unlike Qwen/DeepSeek). Chose MoE (26B A4B, 3.8B active)
    # over the dense 31B variant: ~3.7GB smaller on-disk AWQ footprint
    # (17.2GB vs 20.9GB) buys more KV-cache headroom on this tight 24GB
    # card, and decode should be faster (memory-bandwidth-bound on active
    # params, not total params). Tradeoff accepted: MoE scores lower than
    # dense on the Tau2 tool-use benchmark (68.2% vs 76.9%) but still beats
    # every other size in the family except the 31B on most reasoning
    # benchmarks. Model choice: cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit —
    # AutoAWQ 4-bit group_size=32, MoE expert layers (gate/up/down/router)
    # explicitly excluded from quantization ("ignore" list in config.json)
    # per standard llm-compressor MoE quant practice — only the dense
    # attention/projection layers are 4-bit, experts stay higher precision.
    # Native architecture: Gemma4ForConditionalGeneration (registered
    # natively in this host's installed vLLM 0.28.0 — vllm/model_executor/
    # models/registry.py line 415 — no plugin/trust-remote-code needed).
    # Native max_position_embeddings: 262144 (256K) — Hermes's 64K floor is
    # comfortably covered without any context-extension trick.
    max_model_len: 65536
    # VRAM math (not yet live-validated — see swap validation log below
    # once run): AWQ weights ~17.2GB on disk (dense attn 4-bit + MoE
    # experts higher-precision, per config.json's compressed-tensors
    # ignore list). Starting the KV cache dtype at int4_per_token_head
    # from the outset (rather than fp16 -> fp8 -> int4 trial-and-error like
    # the DeepSeek swap) since that same escalation pattern is expected to
    # repeat on this VRAM-constrained card for any 20+ GB model at >32K ctx.
    kv_cache_dtype: int4_per_token_head
    gpu_memory_utilization: 0.95
    enforce_eager: true
    # Native tool-calling + reasoning support (no `hermes` workaround
    # needed, unlike DeepSeek-R1-Distill): Gemma4EngineToolParser and
    # Gemma4ParserReasoningAdapter are both registered natively in this
    # host's vLLM 0.28.0 (vllm/tool_parsers/__init__.py,
    # vllm/reasoning/__init__.py) — purpose-built for this model's actual
    # output format, not a same-family approximation.
    enable_auto_tool_choice: true
    tool_call_parser: gemma4
    reasoning_parser: gemma4
    enabled: true
  - id: "Qwen3-8B-AWQ"
    hf_repo: "Qwen/Qwen3-8B-AWQ"
    role: aux
    quantization: awq
    port: 8010
    max_model_len: 32768
    gpu_memory_utilization: 0.15
    enforce_eager: true
    enabled: false   # single-model deployment — see swap note above
  - id: "nomic-embed-text-v1.5"
    hf_repo: "nomic-ai/nomic-embed-text-v1.5"
    role: embedding
    quantization: none
    port: 8020
    max_model_len: 2048
    gpu_memory_utilization: 0.05
    trust_remote_code: true
    enabled: false   # single-model deployment — see swap note above
 # --- deploy-vllm role: boot persistence (unchanged) -------------------------
 # Still permanent/boot-persistent — same policy as the outgoing Qwen2.5-32B
 # deployment (t_5508360a), just now serving one model instead of two.
 vllm_service_enabled: true
 vllm_service_state: started
 # Paperclip Docker deployment
 nginx_proxy_certbot_email: ryan.blundon@protonmail.com
 # nginx_proxy_certbot_staging: true # remove after first successful real issuance
 hermes_model:
  provider: custom
  default: Gemma-4-26B-A4B-Hermes-VLM-MLX      # from http://fantasmic.wdw-labs.cloud:8000/v1/models
  base_url: http://fantasmic.wdw-labs.cloud:8000/v1
  api_key: "{{ vault_fantasmic_api_key }}"
@@ -1,29 +0,0 @@
 ---
 # file: host_vars/int-master01/vars
 # Ansible vars template for OpenShift nodes created via Ansible (configure_openshift_cluster).
 # Cluster deployed via Agent Installer and Matchbox (iPXE):
 # Role of this node in the cluster. 
 # - master
 # - worker
 node_role: "master"
 # IP address assigned to this node on the Primary Interface
 ip_address: 10.1.71.61
 # Ansible Playbook supports booting node via one interface, but installing on a 
 # different interface. (Disabling the boot interface on install)  All these fields
 # are necessary.  In the case of a single NIC node, interfaces and MAC addresses
 # should be the same.
 boot_interface: "enp0s31f6"
 boot_mac_address: "54:bf:64:60:52:2d"
 primary_interface: "enp1s0f0"
 install_mac_address: "98:b7:85:20:47:22"
 # Static Variables
 hostname: "{{ inventory_hostname }}.{{ base_domain }}"
@@ -1,29 +0,0 @@
 ---
 # file: host_vars/int-master02/vars
 # Ansible vars template for OpenShift nodes created via Ansible (configure_openshift_cluster).
 # Cluster deployed via Agent Installer and Matchbox (iPXE):
 # Role of this node in the cluster. 
 # - master
 # - worker
 node_role: "master"
 # IP address assigned to this node on the Primary Interface
 ip_address: 10.1.71.62
 # Ansible Playbook supports booting node via one interface, but installing on a 
 # different interface. (Disabling the boot interface on install)  All these fields
 # are necessary.  In the case of a single NIC node, interfaces and MAC addresses
 # should be the same.
 boot_interface: "enp0s31f6"
 boot_mac_address: "54:bf:64:5f:a2:80"
 primary_interface: "enp1s0f0"
 install_mac_address: "98:b7:85:20:47:3a"
 # ---
 hostname: "{{ inventory_hostname }}.{{ base_domain }}"
@@ -1,29 +0,0 @@
 ---
 # file: host_vars/int-master03/vars
 # Ansible vars template for OpenShift nodes created via Ansible (configure_openshift_cluster).
 # Cluster deployed via Agent Installer and Matchbox (iPXE):
 # Role of this node in the cluster. 
 # - master
 # - worker
 node_role: "master"
 # IP address assigned to this node on the Primary Interface
 ip_address: 10.1.71.63
 # Ansible Playbook supports booting node via one interface, but installing on a 
 # different interface. (Disabling the boot interface on install)  All these fields
 # are necessary.  In the case of a single NIC node, interfaces and MAC addresses
 # should be the same.
 boot_interface: "enp0s31f6"
 boot_mac_address: "d8:9e:f3:4c:02:a9"
 primary_interface: "enp1s0f0"
 install_mac_address: "98:b7:85:1f:89:cf"
 # ---
 hostname: "{{ inventory_hostname }}.{{ base_domain }}"
@@ -1,29 +0,0 @@
 ---
 # file: host_vars/int-worker02/vars
 # Ansible vars template for OpenShift nodes created via Ansible (configure_openshift_cluster).
 # Cluster deployed via Agent Installer and Matchbox (iPXE):
 # Role of this node in the cluster. 
 # - master
 # - worker
 node_role: "worker"
 # IP address assigned to this node on the Primary Interface
 ip_address: 10.1.71.164
 # Ansible Playbook supports booting node via one interface, but installing on a 
 # different interface. (Disabling the boot interface on install)  All these fields
 # are necessary.  In the case of a single NIC node, interfaces and MAC addresses
 # should be the same.
 boot_interface: "enp87s0"
 boot_mac_address: "36:47:ca:76:13:49"
 primary_interface: "enp2s0f0"
 install_mac_address: "36:47:ca:76:13:47"
 # ---
 hostname: "{{ inventory_hostname }}.{{ base_domain }}"
@@ -1,29 +0,0 @@
 ---
 # file: host_vars/int-worker02/vars
 # Ansible vars template for OpenShift nodes created via Ansible (configure_openshift_cluster).
 # Cluster deployed via Agent Installer and Matchbox (iPXE):
 # Role of this node in the cluster. 
 # - master
 # - worker
 node_role: "worker"
 # IP address assigned to this node on the Primary Interface
 ip_address: 10.1.71.65
 # Ansible Playbook supports booting node via one interface, but installing on a 
 # different interface. (Disabling the boot interface on install)  All these fields
 # are necessary.  In the case of a single NIC node, interfaces and MAC addresses
 # should be the same.
 boot_interface: "enp87s0"
 boot_mac_address: "58:47:ca:76:13:49"
 primary_interface: "enp2s0f0"
 install_mac_address: "58:47:ca:76:13:47"
 # ---
 hostname: "{{ inventory_hostname }}.{{ base_domain }}"
@@ -1,33 +1,12 @@
 ---
 # file: host_vars/lightning_lane/vars
-# Traefik Load Balancer Server
+# lightning-lane — Traefik Reverse Proxy / Load Balancer
 # VM provisioned by Terraform (pre-pipeline bootstrap)
-# Supported hypervisors:
+# ─── Network ─────────────────────────────────────────────────────────────────
-# - Proxmox
+ip_address: 10.1.71.35
-platform: "proxmox"
+# ─── Application ─────────────────────────────────────────────────────────────
-
+app_role: reverse_proxy
-# This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
+app_name: traefik
-proxmox_clone_node: "fantasyland"
+app_deployment: docker_compose
 # VM Configuration
 vm_os_distribution: "ubuntu"
 vm_os_version: "24.04"
 vm_size: "small"
 vm_storage: "general"
 # Proxmox HA Configuration
 ha_group: "pve02"
 proxmox_host_target: "pve02"
 # Network Configuration
 # IP address for the Traefik load balancer
 ip_address: 10.1.71.24
 #vm_mac_address: 'BC:24:11:50:00:80'
 # Software Configuration
 # Traefik will be installed and configured on this host
 # VM Template Configuration
 #vm_clone_source: "{{ vm_os_distribution }}-{{ vm_os_version }}-{{ vm_size }}"
 hostname: "{{ inventory_hostname }}.{{ base_domain }}"
@@ -0,0 +1,16 @@
 ---
 # Host-specific vars for main-street-station (JMRI headless server)
 # LCRR - Lake Country Railroad, Milwaukee Road Oct 1956, HO scale
 # JMRI profile ID — find with: ls ~/.jmri/profiles/ on the old box
 # Format: <name>.<8-char-hex>  e.g. LCRR.3d3f1dfc
 # TODO: fill in after restoring config from GitHub backup
 jmri_profile_id: ""
 # USB serial device for NCE command station
 # Verify after install: ls -la /dev/ttyUSB* /dev/ttyACM*
 jmri_serial_device: /dev/ttyUSB0
 # Path to JMRI config backup for restore task (leave empty to skip)
 # Point at a local checkout of the LCRR GitHub repo
 jmri_config_src: ""
@@ -0,0 +1,10 @@
 ---
 # main-street-station — JMRI / LCRR server
 jmri_profile_id: "Lake_Country_Railroad.3e8b1d4b"
 jmri_lcrr_repo: "ssh://git@gitea.mk-labs.cloud:2221/rblundon/LCRR.git"
 jmri_lcrr_branch: "clean-profile"
 jmri_leviton_email: "{{ leviton_email }}"
 jmri_leviton_password: "{{ leviton_password }}"
 jmri_ssh_authorized_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINnSM/9fO8rz/amqkyoGUzUKNNzzmtSXPwOCr1O9zKNO ansible"
 jmri_ssh_authorized_keys_extra:
  - "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG6HaK4Y21UwPRbAZ986L7I9QnUdyq53114+9kO8X4bL rblundon@laptop"
@@ -1,17 +0,0 @@
 ---
 # file: host_vars/monitor/vars
 # VM Definition
 platform: "proxmox"
 proxmox_clone_node: "pve03"
 vm_clone_source: "fedora-42-large"
 vm_storage: "pve-general"
 ha_group: "pve03"
 # Networking
 ip_address: 10.1.71.249
 # Software
 #terraform_version: "1.11.3"
 hostname: "{{ inventory_hostname }}.{{ base_domain }}" #change variable to fqdn
@@ -1,31 +0,0 @@
 ---
 # file: host_vars/ocp-hub/vars
 # file: host_vars/int-master01/vars
 # Ansible vars template for OpenShift nodes created via Ansible (configure_openshift_cluster).
 # Cluster deployed via Agent Installer and Matchbox (iPXE):
 # Role of this node in the cluster. 
 # - master
 # - worker
 node_role: "master"
 # IP address assigned to this node on the Primary Interface
 ip_address: 10.1.71.10
 # Ansible Playbook supports booting node via one interface, but installing on a 
 # different interface. (Disabling the boot interface on install)  All these fields
 # are necessary.  In the case of a single NIC node, interfaces and MAC addresses
 # should be the same.
 boot_interface: "enp0s31f6"
 boot_mac_address: "50:9A:4C:52:A8:5D"
 primary_interface: "enp1s0f0"
 install_mac_address: "98:b7:85:1e:c6:f1"
 # Static Variables
 hostname: "{{ inventory_hostname }}.{{ base_domain }}"
@@ -1,29 +0,0 @@
 ---
 # file: host_vars/openshift_node_template/vars
 # Ansible vars template for OpenShift nodes created via Ansible (configure_openshift_cluster).
 # Cluster deployed via Agent Installer and Matchbox (iPXE):
 # Role of this node in the cluster. 
 # - master
 # - worker
 node_role: "master"
 # IP address assigned to this node on the Primary Interface
 ip_address: 10.1.71.61
 # Ansible Playbook supports booting node via one interface, but installing on a 
 # different interface. (Disabling the boot interface on install)  All these fields
 # are necessary.  In the case of a single NIC node, interfaces and MAC addresses
 # should be the same.
 boot_interface: "enp0s31f6"
 boot_mac_address: "54:bf:64:60:52:2d"
 primary_interface: "enp1s0f0"
 install_mac_address: "98:b7:85:20:47:22"
 # Static Variables
 hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn
@@ -0,0 +1,27 @@
 ---
 # file: host_vars/turnstile/vars
 # turnstile — Smallstep step-ca SSH Certificate Authority
 # VM provisioned by Terraform (pre-pipeline bootstrap)
 # ─── Network ─────────────────────────────────────────────────────────────────
 ip_address: 10.1.71.34
 # ─── Application ─────────────────────────────────────────────────────────────
 app_role: ssh_certificate_authority
 app_name: step-ca
 app_deployment: docker_compose
 # ─── step-ca Configuration ───────────────────────────────────────────────────
 stepca_hostname: turnstile.local.mk-labs.cloud
 stepca_dns_names: "turnstile.local.mk-labs.cloud,10.1.71.34"
 stepca_ssh_enabled: true
 stepca_listen_port: 9000
 # ─── OIDC Provisioner (Authentik) ────────────────────────────────────────────
 # Client ID and secret stored in vault
 stepca_oidc_provisioner_name: authentik
 stepca_oidc_configuration_endpoint: "https://authentik.local.mk-labs.cloud/application/o/step-ca/.well-known/openid-configuration"
 stepca_oidc_listen_address: ":10000"
 stepca_oidc_domains:
  - "local.mk-labs.cloud"
  - "protonmail.com"
@@ -0,0 +1,17 @@
 $ANSIBLE_VAULT;1.1;AES256
 30383930323363386234333433636465393263613336646464666365643730386430353864616334
 3865343031653162313736656437373666636136653263360a613961646133376262633164393535
 31363434633634303035663133316230633538363936303266373931313661346563333832653032
 6239376363646530620a303231363333363233623038326165316463383662656565626534396232
 62333133623530643932643430663166373930353733363866336533643233373261333130613635
 31633337376337663833613634666436383862386431636537373363343333323932363761653366
 61353166613162346436396138316561646165303566376366323462663861616364336539313138
 30386430373566353761383636626335393463376661356666303861353564313832346333396134
 30653861363866336331336164323130623230666237356165613934333239386536373664643065
 33643139346562346663313533643433353462363665323166313364373335366665373435643935
 65303863663864393238393732303065343364306264396333376233666233346664323334336532
 38373164383835663163363137643531303163396236623565666436363261393563343133333161
 36663934303633663062346161636333396135336135616136303664636562363862353764343562
 39643430396636313139336130656363306562363430346233313530663963366238326334623135
 62383632613061316262323038303333323739303137363334626637666663666263613132336433
 65346266393733633632
@@ -1,47 +0,0 @@
 ---
 # file: host_vars/unbound01/vars
 # Supported hypervisors:
 # - Proxmox
 platform: "proxmox"
 # This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
 proxmox_clone_node: "pve03"
 # Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
 # Current OS offerings are:
 # - Fedora (42)
 # Current VM sizes are:
 # - Small: 2 cores, 2GB memory, 8 GiB virtual disk
 # - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
 # - Large: 4 cores, 4GB memory, 32 GiB virtual disk
 # - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
 vm_clone_source: "fedora-42-small"
 # Proxmox storage target.
 vm_storage: "pve-general"
 # Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
 # and backup groups. (ha_group will be factored out in the next functionality update.)
 ha_group: "pve03"
 proxmox_host_target: "pve03"
 # Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
 # sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
 ip_address: 10.1.71.251
 # Software
 # Future enhancement will allow specification of additional software to automatically deploy to
 # the VM after creation.
 #terraform_version: "1.11.3"
 # ---
 hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn
@@ -1,47 +0,0 @@
 ---
 # file: host_vars/unbound02/vars
 # Supported hypervisors:
 # - Proxmox
 platform: "proxmox"
 # This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
 proxmox_clone_node: "pve03"
 # Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
 # Current OS offerings are:
 # - Fedora (42)
 # Current VM sizes are:
 # - Small: 2 cores, 2GB memory, 8 GiB virtual disk
 # - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
 # - Large: 4 cores, 4GB memory, 32 GiB virtual disk
 # - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
 vm_clone_source: "fedora-42-small"
 # Proxmox storage target.
 vm_storage: "pve-general"
 # Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
 # and backup groups. (ha_group will be factored out in the next functionality update.)
 ha_group: "pve03"
 proxmox_host_target: "pve03"
 # Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
 # sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
 ip_address: 10.1.71.252
 # Software
 # Future enhancement will allow specification of additional software to automatically deploy to
 # the VM after creation.
 #terraform_version: "1.11.3"
 # ---
 hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn
@@ -1,15 +1,12 @@
 # file: inventory.yml
 proxmox:
  hosts:
-    pve0[1:3]:
+    main-street-usa:
-
+      ansible_host: 10.1.71.11
-ntp_servers:
+      ansible_user: wed
-  hosts:
+      ansible_become: true
-    sundial:
+    tomorrowland:
-    
+    fantasyland:
 dns_server:
  hosts:
    monorail:
 magic_kingdom:
  hosts:
@@ -17,47 +14,111 @@ magic_kingdom:
    tomorrowland:
    fantasyland:
-prometheus_server:
+dns_server:
  hosts:
-    cinderella-castle:
+    monorail:
-prometheus_nodes:
+ntp_servers:
  hosts:
-    cinderella-castle:
+    sundial:
-
+    
 load_balancers:
  hosts:
    lightning-lane:
      ansible_host: 10.1.71.24
      ansible_become: true
 # node_explorer:
 #   hosts:
 #     cinderella-castle:
-# alertmanager:
+step_ca_server:
-#   hosts:
+  hosts:
-#     cinderella-castle:
+    turnstile:
      ansible_become: true
-# grafana:
+authentik_server:
-#   hosts:
+  hosts:
-#     cinderella-castle:
+    guest-relations:
      ansible_become: true
 gitea_servers:
  hosts:
    mad-tea-party:
      ansible_user: wed
      ansible_become: true
 nextcloud_server:
  hosts:
    the-grid:
      ansible_become: true
 semaphore_server:
  hosts:
    figment:
      ansible_host: 10.1.71.37
      ansible_user: wed
      ansible_become: true
 n8n_server:
  hosts:
    tiki-room:
 astro_orbiter:
  hosts:
    astro-orbiter:
 claude_servers:
  hosts:
    haunted-mansion:
      ansible_host: 10.1.71.137
      ansible_user: wed
      ansible_become: true
      ansible_ssh_private_key_file: ~/.ssh/ansible
 hermes_server:
  hosts:
    carousel-of-progress:
      ansible_host: 10.1.71.131
      ansible_user: wed
      ansible_become: true
      ansible_ssh_private_key_file: ~/.ssh/ansible
 honcho_server:
  hosts:
    lincoln:
      ansible_host: 10.1.71.132
      ansible_user: wed
      ansible_become: true
 paperclip_server:
  hosts:
    illuminations:
      ansible_host: 10.1.71.130
      ansible_user: wed
      ansible_become: true
 multica:
  hosts:
    muppet-vision:
      ansible_host: 10.1.71.136
      ansible_user: wed
      ansible_become: true
 jmri_server:
  hosts:
    main-street-station:
      ansible_host: 192.168.10.40
      ansible_user: wed
      ansible_become: true
 papermc_server:
 # ansible-galaxy role install engonzal.papermc
  hosts:
    arcade:
-semaphore_server:
+dev_servers:
  hosts:
-    imagineering:
+    scrim:
-
+    backstage:
-n8n_server:
+      ansible_host: 10.1.71.133
-  hosts:
+      ansible_user: wed
-    tiki-room:
+      ansible_become: true
 traefik_server:
  hosts:
    lightning_lane:
 # dhcp_server:
 #   hosts:
@@ -67,13 +128,13 @@ backup_servers:
  hosts:
    timekeeper:
-# freeipa:
+talos_control:
-#   hosts:
+  hosts:
-#     infra01:
+    city-hall:
-
+      ansible_become: true
-# ipaserver:
+  vars:
-#   hosts:
+    talosctl_version: "v1.12.4"
-#     infra01.int.mk-labs.cloud:
+    talos_cluster_name: "fastpass"
 # matchbox_server:
 #   hosts:
@@ -82,51 +143,3 @@ backup_servers:
 terraform_server:
  hosts:
    infra01:
 # hub_cluster:
 #   hosts:
 #     ocp-hub:
 # internal_cluster:
 #   hosts:
 #     int-master01:
 #     int-master02:
 #     int-master03:
 # #    int-worker01:
 #     int-worker02:
 # talos_cluster:
 #   hosts:
 #     splash:
 # fastpass_control_plane:
 #   hosts:
 #     space-mountain:
 #     big-thunder-mountain:
 #     splash-mountain:
 # fastpass_workers:
 #   hosts:
 #     haunted-mansion: # backstage
 #     peter-pans-flight: # backstage
 # fastpass:
 #   children:
 #     fastpass_control_plane:
 #     fastpass_workers:
 # sql_servers:
 #   hosts:
 #     sql01:
 # docker_servers:
 #   hosts:
 #     docker01:
 # observer:
 #   hosts:
 #     prometheus:
 # target:
 #   hosts:
 #     prometheus:
@@ -0,0 +1,95 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/add_service_route.yml
 # DESCRIPTION: Ensures all services in the Traefik dynamic config directory
 #              are routed and have DNS CNAME records on monorail.
 #
 #              1. Syncs boilerplates/traefik/dynamic/ to lightning-lane
 #              2. Scans the directory for service configs
 #              3. Extracts all hostnames from Host() rules (supports multi-host)
 #              4. Creates CNAME records for each hostname -> lightning-lane
 #
 # PREREQUISITES:
 #   - Service dynamic config YAML committed to boilerplates/traefik/dynamic/
 #   - vault_technitium_api_key defined in group_vars/all/vault
 #
 # USAGE:
 #   ansible-playbook -i inventory.yml playbooks/add_service_route.yml
 # ------------------------------------------------------------------------------
 - name: Sync Traefik routes and ensure DNS records
  hosts: localhost
  connection: local
  gather_facts: false
  vars:
    base_domain: "local.mk-labs.cloud"
    dns_server: "monorail"
    traefik_host: "lightning-lane.local.mk-labs.cloud"
    traefik_user: "wed"
    traefik_dynamic_path: "/opt/docker/traefik/dynamic/"
    dynamic_config_dir: "{{ playbook_dir }}/../../boilerplates/traefik/dynamic"
    # Files in the dynamic directory that are NOT service routes
    exclude_configs:
      - default.yml
  tasks:
    # ── Step 1: Sync dynamic config to lightning-lane ──
    - name: Sync Traefik dynamic configuration to lightning-lane
      ansible.builtin.shell: >
        rsync -av --delete
        {{ dynamic_config_dir }}/
        {{ traefik_user }}@{{ traefik_host }}:{{ traefik_dynamic_path }}
      register: sync_result
      changed_when: "'sending incremental file list' in sync_result.stdout"
    # ── Step 2: Discover hostnames from Traefik router rules ──
    - name: Find all dynamic config files
      ansible.builtin.find:
        paths: "{{ dynamic_config_dir }}"
        patterns: "*.yml"
      register: config_files
    - name: Read config files
      ansible.builtin.slurp:
        src: "{{ item.path }}"
      register: slurped_configs
      loop: "{{ config_files.files }}"
      when: item.path | basename not in exclude_configs
    - name: Extract all hostnames from Host() rules
      ansible.builtin.set_fact:
        hostnames: >-
          {% set hosts = [] -%}
          {% for result in slurped_configs.results if result.content is defined -%}
            {% set content = result.content | b64decode -%}
            {% for match in content | regex_findall('Host\(`([^`]+)`\)') -%}
              {% for h in match.split(' || ') -%}
                {% set h = h | regex_replace('`', '') | trim -%}
                {% if h.endswith('.local.mk-labs.cloud') and h not in hosts -%}
                  {% set _ = hosts.append(h) -%}
                {% endif -%}
              {% endfor -%}
            {% endfor -%}
          {% endfor -%}
          {{ hosts | unique | list }}
    - name: Display hostnames to create
      ansible.builtin.debug:
        msg: "Hostnames found: {{ hostnames }}"
    # ── Step 3: Create DNS CNAME records ──
    - name: Create DNS CNAME record for each hostname
      effectivelywild.technitium_dns.technitium_dns_add_record:
        api_url: "http://{{ dns_server }}.{{ base_domain }}"
        api_token: "{{ vault_technitium_api_key }}"
        zone: "{{ base_domain }}"
        name: "{{ item }}"
        type: "CNAME"
        cname: "lightning-lane.{{ base_domain }}"
        ttl: 360
        validate_certs: false
      loop: "{{ hostnames }}"
      loop_control:
        label: "{{ item }}"
@@ -0,0 +1,40 @@
 ---
 - name: Build Paperclip Docker image from source
  hosts: illuminations
  become: true
  vars:
    paperclip_repo_dir: /opt/paperclip-repo
    paperclip_image_name: paperclip-local
    paperclip_image_tag: latest
  tasks:
    - name: Install git (if not present)
      apt:
        name: git
        state: present
    - name: Clone or update Paperclip repository
      git:
        repo: https://github.com/paperclipai/paperclip.git
        dest: "{{ paperclip_repo_dir }}"
        update: yes
      register: git_clone
    - name: Build Paperclip Docker image
      shell: |
        cd {{ paperclip_repo_dir }}
        docker build --target production -t {{ paperclip_image_name }}:{{ paperclip_image_tag }} .
      register: docker_build
      changed_when: "'Successfully tagged' in docker_build.stdout"
    - name: Show build result
      debug:
        msg: "Built image: {{ paperclip_image_name }}:{{ paperclip_image_tag }}"
    - name: Show image details
      shell: docker images {{ paperclip_image_name }}:{{ paperclip_image_tag }}
      register: image_info
    - name: Image info
      debug:
        msg: "{{ image_info.stdout_lines }}"
@@ -1,218 +0,0 @@
 ---
 - name: Configure DNS entries for cluster nodes
  hosts: internal_cluster
  gather_facts: false
  become: true
  tasks:
    - name: Create DNS entry for each cluster node
      ansible.builtin.include_tasks: tasks/create_dns_record.yml
      vars:
        host_ip_address: "{{ ip_address }}"
 - name: Configure DNS entries for OpenShift cluster services
  hosts: internal_cluster[0]
  gather_facts: false
  become: true
  tasks:
    - name: Create DNS entries for OpenShift cluster services
      ansible.builtin.include_tasks: tasks/create_dns_record.yml
      vars:
        dns_name: "{{ item.name }}"
        dns_address: "{{ item.address }}"
      loop:
        - name: "api.{{ cluster_name }}"
          address: "{{ api_address }}"
        - name: "api-int.{{ cluster_name }}"
          address: "{{ api_address }}"
        - name: "*.apps.{{ cluster_name }}"
          address: "{{ app_address }}"
      loop_control:
        label: "{{ item.name }}"
 - name: Configure DHCP entries for cluster nodes
  hosts: internal_cluster
  gather_facts: false
  become: true
  tasks:
    - name: Configure DHCP entry for node
      ansible.builtin.include_tasks: tasks/configure_dhcp_entry.yml
      vars:
        host_mac_address: "{{ install_mac_address }}"
        host_ip_address: "{{ ip_address }}"
  handlers:
    - name: Restart dnsmasq
      delegate_to: "{{ dhcp_server }}"
      become: true
      ansible.builtin.service:
        name: dnsmasq
        state: restarted
 - name: Create OpenShift installer files
  hosts: matchbox
  gather_facts: true
  vars:
    cluster_group: "internal_cluster"
    download_dir: "/var/cache/openshift-install"
  tasks:
    - name: Remove previous directory
      ansible.builtin.file:
        path: ~/homelab/terraform/{{ cluster_group }}
        state: absent
    - name: Create directory
      ansible.builtin.file:
        path: ~/homelab/terraform/{{ cluster_group }}
        state: directory
        mode: '0755'
    - name: OpenShift Installer install-config.yaml
      ansible.builtin.template:
        src: templates/install-config.yaml.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/install-config.yaml
        mode: '0644'
    - name: OpenShift Installer agent-config.yaml
      ansible.builtin.template:
        src: templates/agent-config.yaml.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/agent-config.yaml
        mode: '0644'
    - name: Create directory for OpenShift Installer files
      become: true
      ansible.builtin.file:
        path: "{{ download_dir }}/{{ hostvars[groups[cluster_group][0]].cluster_version }}"
        state: directory
        mode: '0755'
        owner: root
        group: root
    - name: Check if OpenShift installer file exists
      ansible.builtin.stat:
        path: "{{ download_dir }}/{{ hostvars[groups[cluster_group][0]].cluster_version }}/{{ hostvars[groups[cluster_group][0]].openshift_installer_file }}"
      register: openshift_installer_file
    - name: Download and install OpenShift installer package
      when: not openshift_installer_file.stat.exists
      block:
        - name: Download OpenShift Installer file
          become: true
          ansible.builtin.get_url:
            url: "{{ hostvars[groups[cluster_group][0]].openshift_installer_download_url }}"
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            mode: '0644'
        - name: Unarchive installer file
          become: true
          ansible.builtin.unarchive:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/openshift-install-linux.tar.gz
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            remote_src: true
        - name: Copy install binary
          become: true
          ansible.builtin.copy:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/openshift-install
            dest: /usr/local/bin
            remote_src: true
            owner: wed
            group: wed
            mode: '0755'
    - name: Check if OpenShift client file exists
      ansible.builtin.stat:
        path: "{{ download_dir }}/{{ hostvars[groups[cluster_group][0]].cluster_version }}/{{ hostvars[groups[cluster_group][0]].openshift_client_file }}"
      register: openshift_client_file
    - name: Download and install OpenShift client package
      when: not openshift_client_file.stat.exists
      block:
        - name: Download OpenShift Client file
          become: true
          ansible.builtin.get_url:
            url: "{{ hostvars[groups[cluster_group][0]].openshift_client_download_url }}"
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            mode: '0644'
        - name: Unarchive client file
          become: true
          ansible.builtin.unarchive:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/openshift-client-linux.tar.gz
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            remote_src: true
        - name: Copy client binary
          become: true
          ansible.builtin.copy:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/oc
            dest: /usr/local/bin
            remote_src: true
            owner: wed
            group: wed
            mode: '0755'
    - name: Check if OpenShift image exists
      ansible.builtin.stat:
        path: ~/homelab/terraform/{{ cluster_group }}/boot-artifacts/agent.x86_64-initrd.img
      register: openshift_image_exists
    - name: OpenShift image creation
      ansible.builtin.command: /usr/local/bin/openshift-install agent create pxe-files
      args:
        chdir: ~/homelab/terraform/{{ cluster_group }}
      when: not openshift_image_exists.stat.exists
      changed_when: true
    - name: Create a directory if it does not exist
      become: true
      ansible.builtin.file:
        path: "{{ hostvars[groups['matchbox_server'][0]].assets }}/{{ hostvars[groups[cluster_group][0]].cluster_name }}"
        state: directory
        owner: matchbox
        group: matchbox
        mode: '0755'
    - name: Copy file with owner and permissions
      become: true
      ansible.builtin.copy:
        src: "{{ ansible_env['HOME'] }}/homelab/terraform/{{ cluster_group }}/boot-artifacts/"
        dest: "{{ hostvars[groups['matchbox_server'][0]].assets }}/{{ hostvars[groups[cluster_group][0]].cluster_name }}/"
        remote_src: true
        owner: matchbox
        group: matchbox
        mode: '0644'
    - name: Verify directory permissions
      become: true
      ansible.builtin.file:
        path: "{{ hostvars[groups['matchbox_server'][0]].assets }}/{{ hostvars[groups[cluster_group][0]].cluster_name }}"
        state: directory
        owner: matchbox
        group: matchbox
        mode: '0755'
    - name: Create Terraform matchbox groups file from template
      ansible.builtin.template:
        src: templates/groups.tf.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/groups.tf
        mode: '0644'
    - name: Create Terraform matchbox profiles file from template
      ansible.builtin.template:
        src: templates/profiles.tf.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/profiles.tf
        mode: '0644'
    - name: Create Terraform matchbox provider file from template
      ansible.builtin.template:
        src: templates/provider.tf.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/provider.tf
        mode: '0644'
    - name: Configure Matchbox via Terraform
      community.general.terraform:
        project_path: ~/homelab/terraform/{{ cluster_group }}
        state: present
        force_init: true
@@ -1,221 +0,0 @@
 ---
 - name: Configure DNS entries for cluster nodes
  hosts: hub_cluster
  gather_facts: false
  become: true
  tasks:
    - name: Create DNS entry for each cluster node
      ansible.builtin.include_tasks: tasks/create_dns_record.yml
      vars:
        host_ip_address: "{{ ip_address }}"
 - name: Configure DNS entries for OpenShift cluster services
  hosts: hub_cluster[0]
  gather_facts: false
  become: true
  tasks:
    - name: Create DNS entries for OpenShift cluster services
      ansible.builtin.include_tasks: tasks/create_dns_record.yml
      vars:
        dns_name: "{{ item.name }}"
        dns_address: "{{ item.address }}"
      loop:
        - name: "api.{{ cluster_name }}"
          address: "{{ api_address }}"
        - name: "api-int.{{ cluster_name }}"
          address: "{{ api_address }}"
        - name: "*.apps.{{ cluster_name }}"
          address: "{{ app_address }}"
      loop_control:
        label: "{{ item.name }}"
 - name: Configure DHCP entries for cluster nodes
  hosts: hub_cluster
  gather_facts: false
  become: true
  tasks:
    - name: Configure DHCP entry for node
      ansible.builtin.include_tasks: tasks/configure_dhcp_entry.yml
      vars:
        host_mac_address: "{{ install_mac_address }}"
        host_ip_address: "{{ ip_address }}"
  handlers:
    - name: Restart dnsmasq
      delegate_to: "{{ dhcp_server }}"
      become: true
      ansible.builtin.service:
        name: dnsmasq
        state: restarted
 - name: Create OpenShift installer files
  hosts: matchbox
  gather_facts: true
  vars:
    cluster_group: "hub_cluster"
    download_dir: "/var/cache/openshift-install"
  tasks:
    - name: Remove previous directory
      ansible.builtin.file:
        path: ~/homelab/terraform/{{ cluster_group }}
        state: absent
    - name: Create directory
      ansible.builtin.file:
        path: ~/homelab/terraform/{{ cluster_group }}
        state: directory
        mode: '0755'
    - name: OpenShift Installer install-config.yaml
      ansible.builtin.template:
        src: templates/install-config.yaml.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/install-config.yaml
        mode: '0644'
    - name: OpenShift Installer agent-config.yaml
      ansible.builtin.template:
        src: templates/agent-config.yaml.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/agent-config.yaml
        mode: '0644'
    # - name: Quit
    #   ansible.builtin.meta: end_play
    - name: Create directory for OpenShift Installer files
      become: true
      ansible.builtin.file:
        path: "{{ download_dir }}/{{ hostvars[groups[cluster_group][0]].cluster_version }}"
        state: directory
        mode: '0755'
        owner: root
        group: root
    - name: Check if OpenShift installer file exists
      ansible.builtin.stat:
        path: "{{ download_dir }}/{{ hostvars[groups[cluster_group][0]].cluster_version }}/{{ hostvars[groups[cluster_group][0]].openshift_installer_file }}"
      register: openshift_installer_file
    - name: Download and install OpenShift installer package
      when: not openshift_installer_file.stat.exists
      block:
        - name: Download OpenShift Installer file
          become: true
          ansible.builtin.get_url:
            url: "{{ hostvars[groups[cluster_group][0]].openshift_installer_download_url }}"
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            mode: '0644'
        - name: Unarchive installer file
          become: true
          ansible.builtin.unarchive:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/openshift-install-linux.tar.gz
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            remote_src: true
        - name: Copy install binary
          become: true
          ansible.builtin.copy:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/openshift-install
            dest: /usr/local/bin
            remote_src: true
            owner: wed
            group: wed
            mode: '0755'
    - name: Check if OpenShift client file exists
      ansible.builtin.stat:
        path: "{{ download_dir }}/{{ hostvars[groups[cluster_group][0]].cluster_version }}/{{ hostvars[groups[cluster_group][0]].openshift_client_file }}"
      register: openshift_client_file
    - name: Download and install OpenShift client package
      when: not openshift_client_file.stat.exists
      block:
        - name: Download OpenShift Client file
          become: true
          ansible.builtin.get_url:
            url: "{{ hostvars[groups[cluster_group][0]].openshift_client_download_url }}"
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            mode: '0644'
        - name: Unarchive client file
          become: true
          ansible.builtin.unarchive:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/openshift-client-linux.tar.gz
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            remote_src: true
        - name: Copy client binary
          become: true
          ansible.builtin.copy:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/oc
            dest: /usr/local/bin
            remote_src: true
            owner: wed
            group: wed
            mode: '0755'
    - name: Check if OpenShift image exists
      ansible.builtin.stat:
        path: ~/homelab/terraform/{{ cluster_group }}/boot-artifacts/agent.x86_64-initrd.img
      register: openshift_image_exists
    - name: OpenShift image creation
      ansible.builtin.command: /usr/local/bin/openshift-install agent create pxe-files
      args:
        chdir: ~/homelab/terraform/{{ cluster_group }}
      when: not openshift_image_exists.stat.exists
      changed_when: true
    - name: Create a directory if it does not exist
      become: true
      ansible.builtin.file:
        path: "{{ hostvars[groups['matchbox_server'][0]].assets }}/{{ hostvars[groups[cluster_group][0]].cluster_name }}"
        state: directory
        owner: matchbox
        group: matchbox
        mode: '0755'
    - name: Copy file with owner and permissions
      become: true
      ansible.builtin.copy:
        src: "{{ ansible_env['HOME'] }}/homelab/terraform/{{ cluster_group }}/boot-artifacts/"
        dest: "{{ hostvars[groups['matchbox_server'][0]].assets }}/{{ hostvars[groups[cluster_group][0]].cluster_name }}/"
        remote_src: true
        owner: matchbox
        group: matchbox
        mode: '0644'
    - name: Verify directory permissions
      become: true
      ansible.builtin.file:
        path: "{{ hostvars[groups['matchbox_server'][0]].assets }}/{{ hostvars[groups[cluster_group][0]].cluster_name }}"
        state: directory
        owner: matchbox
        group: matchbox
        mode: '0755'
    - name: Create Terraform matchbox groups file from template
      ansible.builtin.template:
        src: templates/groups.tf.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/groups.tf
        mode: '0644'
    - name: Create Terraform matchbox profiles file from template
      ansible.builtin.template:
        src: templates/profiles.tf.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/profiles.tf
        mode: '0644'
    - name: Create Terraform matchbox provider file from template
      ansible.builtin.template:
        src: templates/provider.tf.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/provider.tf
        mode: '0644'
    - name: Configure Matchbox via Terraform
      community.general.terraform:
        project_path: ~/homelab/terraform/{{ cluster_group }}
        state: present
        force_init: true
@@ -1,218 +0,0 @@
 ---
 - name: Configure DNS entries for cluster nodes
  hosts: internal_cluster
  gather_facts: false
  become: true
  tasks:
    - name: Create DNS entry for each cluster node
      ansible.builtin.include_tasks: tasks/create_dns_record.yml
      vars:
        host_ip_address: "{{ ip_address }}"
 - name: Configure DNS entries for OpenShift cluster services
  hosts: internal_cluster[0]
  gather_facts: false
  become: true
  tasks:
    - name: Create DNS entries for OpenShift cluster services
      ansible.builtin.include_tasks: tasks/create_dns_record.yml
      vars:
        dns_name: "{{ item.name }}"
        dns_address: "{{ item.address }}"
      loop:
        - name: "api.{{ cluster_name }}"
          address: "{{ api_address }}"
        - name: "api-int.{{ cluster_name }}"
          address: "{{ api_address }}"
        - name: "*.apps.{{ cluster_name }}"
          address: "{{ app_address }}"
      loop_control:
        label: "{{ item.name }}"
 - name: Configure DHCP entries for cluster nodes
  hosts: internal_cluster
  gather_facts: false
  become: true
  tasks:
    - name: Configure DHCP entry for node
      ansible.builtin.include_tasks: tasks/configure_dhcp_entry.yml
      vars:
        host_mac_address: "{{ install_mac_address }}"
        host_ip_address: "{{ ip_address }}"
  handlers:
    - name: Restart dnsmasq
      delegate_to: "{{ dhcp_server }}"
      become: true
      ansible.builtin.service:
        name: dnsmasq
        state: restarted
 - name: Create OpenShift installer files
  hosts: matchbox
  gather_facts: true
  vars:
    cluster_group: "internal_cluster"
    download_dir: "/var/cache/openshift-install"
  tasks:
    - name: Remove previous directory
      ansible.builtin.file:
        path: ~/homelab/terraform/{{ cluster_group }}
        state: absent
    - name: Create directory
      ansible.builtin.file:
        path: ~/homelab/terraform/{{ cluster_group }}
        state: directory
        mode: '0755'
    - name: OpenShift Installer install-config.yaml
      ansible.builtin.template:
        src: templates/install-config.yaml.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/install-config.yaml
        mode: '0644'
    - name: OpenShift Installer agent-config.yaml
      ansible.builtin.template:
        src: templates/agent-config.yaml.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/agent-config.yaml
        mode: '0644'
    - name: Create directory for OpenShift Installer files
      become: true
      ansible.builtin.file:
        path: "{{ download_dir }}/{{ hostvars[groups[cluster_group][0]].cluster_version }}"
        state: directory
        mode: '0755'
        owner: root
        group: root
    - name: Check if OpenShift installer file exists
      ansible.builtin.stat:
        path: "{{ download_dir }}/{{ hostvars[groups[cluster_group][0]].cluster_version }}/{{ hostvars[groups[cluster_group][0]].openshift_installer_file }}"
      register: openshift_installer_file
    - name: Download and install OpenShift installer package
      when: not openshift_installer_file.stat.exists
      block:
        - name: Download OpenShift Installer file
          become: true
          ansible.builtin.get_url:
            url: "{{ hostvars[groups[cluster_group][0]].openshift_installer_download_url }}"
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            mode: '0644'
        - name: Unarchive installer file
          become: true
          ansible.builtin.unarchive:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/openshift-install-linux.tar.gz
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            remote_src: true
        - name: Copy install binary
          become: true
          ansible.builtin.copy:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/openshift-install
            dest: /usr/local/bin
            remote_src: true
            owner: wed
            group: wed
            mode: '0755'
    - name: Check if OpenShift client file exists
      ansible.builtin.stat:
        path: "{{ download_dir }}/{{ hostvars[groups[cluster_group][0]].cluster_version }}/{{ hostvars[groups[cluster_group][0]].openshift_client_file }}"
      register: openshift_client_file
    - name: Download and install OpenShift client package
      when: not openshift_client_file.stat.exists
      block:
        - name: Download OpenShift Client file
          become: true
          ansible.builtin.get_url:
            url: "{{ hostvars[groups[cluster_group][0]].openshift_client_download_url }}"
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            mode: '0644'
        - name: Unarchive client file
          become: true
          ansible.builtin.unarchive:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/openshift-client-linux.tar.gz
            dest: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}
            remote_src: true
        - name: Copy client binary
          become: true
          ansible.builtin.copy:
            src: /var/cache/openshift-install/{{ hostvars[groups[cluster_group][0]].cluster_version }}/oc
            dest: /usr/local/bin
            remote_src: true
            owner: wed
            group: wed
            mode: '0755'
    - name: Check if OpenShift image exists
      ansible.builtin.stat:
        path: ~/homelab/terraform/{{ cluster_group }}/boot-artifacts/agent.x86_64-initrd.img
      register: openshift_image_exists
    - name: OpenShift image creation
      ansible.builtin.command: /usr/local/bin/openshift-install agent create pxe-files
      args:
        chdir: ~/homelab/terraform/{{ cluster_group }}
      when: not openshift_image_exists.stat.exists
      changed_when: true
    - name: Create a directory if it does not exist
      become: true
      ansible.builtin.file:
        path: "{{ hostvars[groups['matchbox_server'][0]].assets }}/{{ hostvars[groups[cluster_group][0]].cluster_name }}"
        state: directory
        owner: matchbox
        group: matchbox
        mode: '0755'
    - name: Copy file with owner and permissions
      become: true
      ansible.builtin.copy:
        src: "{{ ansible_env['HOME'] }}/homelab/terraform/{{ cluster_group }}/boot-artifacts/"
        dest: "{{ hostvars[groups['matchbox_server'][0]].assets }}/{{ hostvars[groups[cluster_group][0]].cluster_name }}/"
        remote_src: true
        owner: matchbox
        group: matchbox
        mode: '0644'
    - name: Verify directory permissions
      become: true
      ansible.builtin.file:
        path: "{{ hostvars[groups['matchbox_server'][0]].assets }}/{{ hostvars[groups[cluster_group][0]].cluster_name }}"
        state: directory
        owner: matchbox
        group: matchbox
        mode: '0755'
    - name: Create Terraform matchbox groups file from template
      ansible.builtin.template:
        src: templates/groups.tf.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/groups.tf
        mode: '0644'
    - name: Create Terraform matchbox profiles file from template
      ansible.builtin.template:
        src: templates/profiles.tf.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/profiles.tf
        mode: '0644'
    - name: Create Terraform matchbox provider file from template
      ansible.builtin.template:
        src: templates/provider.tf.j2
        dest: ~/homelab/terraform/{{ cluster_group }}/provider.tf
        mode: '0644'
    - name: Configure Matchbox via Terraform
      community.general.terraform:
        project_path: ~/homelab/terraform/{{ cluster_group }}
        state: present
        force_init: true
@@ -0,0 +1,20 @@
 ---
 # ansible/playbooks/configure_proxmox_oidc.yml
 #
 # Configures Proxmox OIDC authentication with Authentik.
 # Only targets one node since realm config is cluster-wide.
 #
 # Usage:
 #   cd ansible
 #   ansible-playbook -i inventory.yml playbooks/configure_proxmox_oidc.yml
 #
 # To also set up ACL entries for your user:
 #   ansible-playbook -i inventory.yml playbooks/configure_proxmox_oidc.yml \
 #     -e '{"proxmox_oidc_acl_entries": [{"path": "/", "user": "rblundon@authentik", "role": "Administrator"}]}'
 - name: Configure Proxmox Authentik OIDC
  hosts: main-street-usa
  become: true
  roles:
    - role: proxmox
      tags: [proxmox-oidc]
@@ -0,0 +1,6 @@
 ---
 - name: Apply day0 baseline
  hosts: "{{ target | default('all') }}"
  become: true
  roles:
    - day0-baseline
@@ -0,0 +1,32 @@
 ---
 # ============================================================================
 # day0_expand_root_lv.yml
 # ----------------------------------------------------------------------------
 # Reclaims unallocated PE on the root volume group, extending the root LV
 # to fill the VG and resizing the underlying filesystem (ext4 or xfs).
 #
 # Belongs to the day0 host-provisioning lifecycle. The Ubuntu Server
 # autoinstall template ships with the root LV at ~half the disk size by
 # default; this playbook is the canonical one-shot fix-up for that.
 #
 # Idempotent and safe to re-run. Hosts without LVM are no-op'd cleanly.
 #
 # Opt-out: set `expand_root_lv_skip: true` in host_vars/<host>.yml for
 # hosts where free PE should NOT be claimed by root (e.g. hosts with a
 # planned second LV in the same VG for application data).
 #
 # Usage:
 #   ansible-playbook playbooks/day0_expand_root_lv.yml
 #   ansible-playbook playbooks/day0_expand_root_lv.yml -e target=lincoln
 #   ansible-playbook playbooks/day0_expand_root_lv.yml -e target=honcho_server
 # ============================================================================
 - name: Expand root logical volume to fill VG
  hosts: "{{ target | default('all') }}"
  become: true
  gather_facts: true
  tasks:
    - name: Apply expand_root_lv role unless host opts out
      ansible.builtin.include_role:
        name: expand_root_lv
      when: not (expand_root_lv_skip | default(false) | bool)
@@ -0,0 +1,23 @@
 ---
 # ============================================================================
 # day0_linux_baseline.yml
 # ----------------------------------------------------------------------------
 # Applies the mk-labs Linux baseline (linux-baseline role) to one or more
 # hosts. Idempotent and safe to re-run.
 #
 # Usage:
 #   ansible-playbook playbooks/day0_linux_baseline.yml
 #   ansible-playbook playbooks/day0_linux_baseline.yml -e target=figment
 #   ansible-playbook playbooks/day0_linux_baseline.yml -e target=semaphore_server
 #
 # To trigger an opt-in full system upgrade:
 #   ansible-playbook playbooks/day0_linux_baseline.yml \
 #     -e target=figment -e 'baseline_features={"full_upgrade": true}'
 # ============================================================================
 - name: Apply mk-labs Linux baseline
  hosts: "{{ target | default('all') }}"
  become: true
  gather_facts: true
  roles:
    - linux-baseline
@@ -0,0 +1,30 @@
 ---
 # ============================================================================
 # day0_provision.yml
 # ----------------------------------------------------------------------------
 # Umbrella day0 playbook. Runs the full host-provisioning lifecycle in
 # the correct order against newly-built VMs, so the operator runs ONE
 # command per new host rather than chaining day0 steps manually.
 #
 # Order matters:
 #   1. linux-baseline      — timezone, NTP, packages, SSH hardening, jarvis user
 #   2. expand_root_lv      — reclaim PE left unallocated by the Ubuntu
 #                            autoinstall template default
 #
 # Idempotent: every step is safe to re-run. Suitable to apply periodically
 # from Semaphore as a baseline-drift check.
 #
 # Usage:
 #   ansible-playbook playbooks/day0_provision.yml -e target=lincoln
 #   ansible-playbook playbooks/day0_provision.yml -e target=honcho_server
 #
 # For finer control over a single phase, the constituent playbooks are:
 #   playbooks/day0_linux_baseline.yml
 #   playbooks/day0_expand_root_lv.yml
 # ============================================================================
 - name: Import day0 linux baseline
  ansible.builtin.import_playbook: day0_linux_baseline.yml
 - name: Import day0 expand root LV
  ansible.builtin.import_playbook: day0_expand_root_lv.yml
@@ -0,0 +1,400 @@
 ---
 # ansible/playbooks/day1_configure_netbox_catalog.yml
 #
 # Creates the tag taxonomy and custom fields in NetBox for service catalog
 # documentation. Run once (idempotent — uses name-based checks).
 #
 # Usage (from ansible/ directory):
 #   ansible-playbook playbooks/day1_configure_netbox_catalog.yml
 #
 # Requires:
 #   - vault_netbox_token in Ansible Vault
 #   - netbox reachable at http://fire-station.local.mk-labs.cloud
 - name: Configure NetBox service catalog taxonomy
  hosts: localhost
  gather_facts: false
  vars_files:
    - "{{ playbook_dir }}/../group_vars/all/vault"
  vars:
    netbox_url: "http://fire-station.local.mk-labs.cloud"
    netbox_token: "{{ vault_netbox_token }}"
    netbox_api: "{{ netbox_url }}/api"
    headers:
      Authorization: "Token {{ netbox_token }}"
      Content-Type: "application/json"
      Accept: "application/json"
    # ── Tag definitions ───────────────────────────────────────────────────
    tags:
      # Infrastructure type
      - name: k8s
        slug: k8s
        color: "2196f3"   # blue
        description: "Workload running in the fastpass Kubernetes cluster"
      - name: vm
        slug: vm
        color: "4caf50"   # green
        description: "Traditional VM or LXC on Proxmox"
      # Service tier
      - name: platform
        slug: platform
        color: "9c27b0"   # purple
        description: "Platform/infrastructure service (not user-facing)"
      - name: application
        slug: application
        color: "ff9800"   # orange
        description: "User-facing application workload"
      # Service categories
      - name: monitoring
        slug: monitoring
        color: "607d8b"   # grey
        description: "Metrics, logging, alerting"
      - name: auth
        slug: auth
        color: "607d8b"
        description: "Authentication and SSO"
      - name: gitops
        slug: gitops
        color: "607d8b"
        description: "GitOps and CI/CD"
      - name: dashboard
        slug: dashboard
        color: "607d8b"
        description: "Dashboard and portal services"
      - name: storage
        slug: storage
        color: "607d8b"
        description: "Storage and file services"
      - name: dns
        slug: dns
        color: "607d8b"
        description: "DNS and name resolution"
      - name: automation
        slug: automation
        color: "607d8b"
        description: "Automation and orchestration"
      - name: networking
        slug: networking
        color: "607d8b"
        description: "Network infrastructure services"
      - name: inference
        slug: inference
        color: "607d8b"
        description: "AI/ML inference workloads"
    # ── Custom field definitions ──────────────────────────────────────────
    # object_types use app_label.model format
    custom_fields:
      - name: hostnames
        label: Hostnames
        type: longtext
        object_types:
          - ipam.ipaddress
          - virtualization.virtualmachine
        description: "All DNS names that resolve to this service (comma-separated)"
        ui_visible: always
        ui_editable: yes
      - name: namespace
        label: Namespace
        type: text
        object_types:
          - ipam.ipaddress
          - virtualization.virtualmachine
        description: "Kubernetes namespace (blank for VM-based services)"
        ui_visible: always
        ui_editable: yes
      - name: managed_by
        label: Managed By
        type: select
        object_types:
          - ipam.ipaddress
          - virtualization.virtualmachine
        description: "How this service is managed"
        choices:
          - ArgoCD
          - Ansible
          - Manual
        ui_visible: always
        ui_editable: yes
      - name: thematic_name
        label: Thematic Name
        type: text
        object_types:
          - ipam.ipaddress
          - virtualization.virtualmachine
        description: "Disney/Magic Kingdom thematic hostname for this service"
        ui_visible: always
        ui_editable: yes
  tasks:
    # ── Tags ─────────────────────────────────────────────────────────────
    - name: Fetch existing tags
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/tags/?limit=200"
        method: GET
        headers: "{{ headers }}"
        return_content: true
        status_code: 200
      register: existing_tags_response
    - name: Set existing tag slugs fact
      ansible.builtin.set_fact:
        existing_tag_slugs: "{{ existing_tags_response.json.results | map(attribute='slug') | list }}"
    - name: Create tags
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/tags/"
        method: POST
        headers: "{{ headers }}"
        body_format: json
        body:
          name: "{{ item.name }}"
          slug: "{{ item.slug }}"
          color: "{{ item.color }}"
          description: "{{ item.description }}"
        status_code: 201
      loop: "{{ tags }}"
      when: item.slug not in existing_tag_slugs
      register: tag_creation
      changed_when: tag_creation.status == 201
    # ── Custom Fields ─────────────────────────────────────────────────────
    - name: Fetch existing custom fields
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-fields/?limit=200"
        method: GET
        headers: "{{ headers }}"
        return_content: true
        status_code: 200
      register: existing_cf_response
    - name: Set existing custom field names fact
      ansible.builtin.set_fact:
        existing_cf_names: "{{ existing_cf_response.json.results | map(attribute='name') | list }}"
    - name: Create custom field — hostnames
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-fields/"
        method: POST
        headers: "{{ headers }}"
        body_format: json
        body:
          name: hostnames
          label: Hostnames
          type: longtext
          object_types: "{{ custom_fields[0].object_types }}"
          description: "{{ custom_fields[0].description }}"
          ui_visible: always
          ui_editable: yes
        status_code: 201
      when: "'hostnames' not in existing_cf_names"
      register: cf_hostnames
      changed_when: cf_hostnames.status == 201
    - name: Create custom field — namespace
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-fields/"
        method: POST
        headers: "{{ headers }}"
        body_format: json
        body:
          name: namespace
          label: Namespace
          type: text
          object_types: "{{ custom_fields[1].object_types }}"
          description: "{{ custom_fields[1].description }}"
          ui_visible: always
          ui_editable: yes
        status_code: 201
      when: "'namespace' not in existing_cf_names"
      register: cf_namespace
      changed_when: cf_namespace.status == 201
    - name: Create choice set for managed_by field
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-field-choice-sets/"
        method: POST
        headers: "{{ headers }}"
        body_format: json
        body:
          name: managed-by-choices
          extra_choices:
            - - ArgoCD
              - ArgoCD
            - - Ansible
              - Ansible
            - - Manual
              - Manual
        status_code: [201, 400]
      register: choice_set
      changed_when: choice_set.status == 201
    - name: Fetch choice set ID
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-field-choice-sets/?name=managed-by-choices"
        method: GET
        headers: "{{ headers }}"
        return_content: true
        status_code: 200
      register: choice_set_response
    - name: Create custom field — managed_by
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-fields/"
        method: POST
        headers: "{{ headers }}"
        body_format: json
        body:
          name: managed_by
          label: Managed By
          type: select
          object_types: "{{ custom_fields[2].object_types }}"
          description: "{{ custom_fields[2].description }}"
          choice_set: "{{ choice_set_response.json.results[0].id }}"
          ui_visible: always
          ui_editable: yes
        status_code: 201
      when: "'managed_by' not in existing_cf_names"
      register: cf_managed_by
      changed_when: cf_managed_by.status == 201
    - name: Create custom field — thematic_name
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-fields/"
        method: POST
        headers: "{{ headers }}"
        body_format: json
        body:
          name: thematic_name
          label: Thematic Name
          type: text
          object_types: "{{ custom_fields[3].object_types }}"
          description: "{{ custom_fields[3].description }}"
          ui_visible: always
          ui_editable: yes
        status_code: 201
      when: "'thematic_name' not in existing_cf_names"
      register: cf_thematic_name
      changed_when: cf_thematic_name.status == 201
    # ── Provisioning Pipeline Fields ──────────────────────────────────────
    # These fields drive the NetBox → n8n → Terraform pipeline.
    # proxmox_datastore already exists — PATCH it to add utilidor choice.
    # data_disk_enabled and data_disk_size_gb are new POSTs.
    - name: Fetch proxmox_datastore field ID
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-fields/?name=proxmox_datastore"
        method: GET
        headers: "{{ headers }}"
        return_content: true
        status_code: 200
      register: proxmox_datastore_cf_response
    - name: Fetch proxmox-datastore-choices choice set ID
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-field-choice-sets/?name=proxmox-datastore-choices"
        method: GET
        headers: "{{ headers }}"
        return_content: true
        status_code: 200
      register: datastore_choice_set_response
    - name: Create proxmox-datastore-choices choice set if missing
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-field-choice-sets/"
        method: POST
        headers: "{{ headers }}"
        body_format: json
        body:
          name: proxmox-datastore-choices
          extra_choices:
            - - liberty-tree
              - liberty-tree
            - - utilidor
              - utilidor
        status_code: [201, 400]
      register: datastore_choice_set_create
      changed_when: datastore_choice_set_create.status == 201
      when: datastore_choice_set_response.json.count == 0
    - name: Re-fetch proxmox-datastore-choices choice set ID after possible creation
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-field-choice-sets/?name=proxmox-datastore-choices"
        method: GET
        headers: "{{ headers }}"
        return_content: true
        status_code: 200
      register: datastore_choice_set_response
    - name: Patch proxmox_datastore field to use choice set with utilidor
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-fields/{{ proxmox_datastore_cf_response.json.results[0].id }}/"
        method: PATCH
        headers: "{{ headers }}"
        body_format: json
        body:
          choice_set: "{{ datastore_choice_set_response.json.results[0].id }}"
        status_code: 200
      when: proxmox_datastore_cf_response.json.count > 0
      register: cf_datastore_patch
      changed_when: cf_datastore_patch.status == 200
    - name: Create custom field — data_disk_enabled
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-fields/"
        method: POST
        headers: "{{ headers }}"
        body_format: json
        body:
          name: data_disk_enabled
          label: Data Disk Enabled
          type: boolean
          object_types:
            - virtualization.virtualmachine
          description: "When true, Terraform provisions a second disk for application data storage."
          default: false
          ui_visible: always
          ui_editable: yes
        status_code: 201
      when: "'data_disk_enabled' not in existing_cf_names"
      register: cf_data_disk_enabled
      changed_when: cf_data_disk_enabled.status == 201
    - name: Create custom field — data_disk_size_gb
      ansible.builtin.uri:
        url: "{{ netbox_api }}/extras/custom-fields/"
        method: POST
        headers: "{{ headers }}"
        body_format: json
        body:
          name: data_disk_size_gb
          label: Data Disk Size (GB)
          type: integer
          object_types:
            - virtualization.virtualmachine
          description: "Size in GB for the optional second data disk. Only used when data_disk_enabled is true."
          ui_visible: always
          ui_editable: yes
        status_code: 201
      when: "'data_disk_size_gb' not in existing_cf_names"
      register: cf_data_disk_size
      changed_when: cf_data_disk_size.status == 201
    # ── Summary ───────────────────────────────────────────────────────────
    - name: Summary
      ansible.builtin.debug:
        msg:
          - "Tags created: {{ tag_creation.results | selectattr('status', 'equalto', 201) | list | length }}"
          - "Tags skipped (already exist): {{ tag_creation.results | selectattr('skipped', 'defined') | list | length }}"
          - "Custom fields configured: hostnames, namespace, managed_by, thematic_name, proxmox_datastore (patched), data_disk_enabled, data_disk_size_gb"
@@ -0,0 +1,18 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/deploy_authentik.yml
 # DESCRIPTION: Deploys Authentik identity provider on guest-relations.
 #              Installs Docker and configures Authentik with PostgreSQL and Redis.
 #
 # USAGE:
 #   ansible-playbook -i inventory.yml playbooks/deploy_authentik.yml
 # ------------------------------------------------------------------------------
 - name: Deploy Authentik identity provider
  hosts: authentik_server
  become: true
  roles:
    - common
    - docker-host
    - authentik
@@ -0,0 +1,23 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: ansible/playbooks/day1_deploy_gitea.yml
 # DESCRIPTION: Deploys Gitea + PostgreSQL on mad-tea-party (10.1.71.129)
 #              Role chain: common → docker-host → gitea
 #
 # USAGE:
 #   ansible-playbook -i ansible/inventory.yml ansible/playbooks/day1_deploy_gitea.yml
 #
 # SECRETS REQUIRED IN VAULT (group_vars/all/vault):
 #   vault_gitea_db_password
 # ------------------------------------------------------------------------------
 - name: Deploy Gitea on mad-tea-party
  hosts: gitea_servers
  become: true
  vars:
    gitea_db_password: "{{ vault_gitea_db_password }}"
  roles:
    - role: docker-host
    - role: gitea
@@ -0,0 +1,79 @@
 ---
 # =============================================================================
 # day1_deploy_hermes.yml
 # Deploy Hermes Agent (Nous Research) on carousel-of-progress (10.1.71.131)
 #
 # FIRST-RUN WORKFLOW:
 #   1. Run this playbook:
 #        ansible-playbook playbooks/day1_deploy_hermes.yml
 #
 #   2. SSH to the host and run the setup wizard as the hermes user:
 #        ssh wed@carousel-of-progress.local.mk-labs.cloud
 #        sudo -u hermes hermes setup
 #
 #   3. Once configured, start and verify the service:
 #        sudo systemctl start hermes
 #        sudo systemctl status hermes
 #        sudo journalctl -u hermes -f
 #
 # VARIABLES:
 #   hermes_skip_browser: true   — set to skip Playwright/Chromium install
 #                                 (saves ~300MB if browser automation not needed)
 # =============================================================================
 - name: Deploy Hermes Agent on carousel-of-progress
  hosts: carousel-of-progress
  gather_facts: true
  pre_tasks:
    - name: Verify target is carousel-of-progress
      ansible.builtin.assert:
        that:
          - inventory_hostname == "carousel-of-progress"
        fail_msg: >
          This playbook is scoped to carousel-of-progress only.
          Got: {{ inventory_hostname }}
    - name: Confirm OS is Ubuntu
      ansible.builtin.assert:
        that:
          - ansible_distribution == "Ubuntu"
        fail_msg: >
          This playbook requires Ubuntu. Found: {{ ansible_distribution }}.
          (If running Fedora, swap apt tasks for dnf and adjust Playwright deps.)
  roles:
    - role: hermes
      vars:
        hermes_skip_browser: false   # set true to skip Chromium install
  post_tasks:
    - name: Verify hermes binary is accessible system-wide
      ansible.builtin.command: hermes --version
      register: hermes_version_check
      changed_when: false
      failed_when: hermes_version_check.rc != 0
    - name: Print hermes version
      ansible.builtin.debug:
        msg: "{{ hermes_version_check.stdout }}"
    - name: Print post-install instructions
      ansible.builtin.debug:
        msg:
          - "============================================================"
          - "Hermes installed on carousel-of-progress (10.1.71.131)"
          - "============================================================"
          - "Next steps:"
          - "  1. SSH to the host:"
          - "       ssh wed@carousel-of-progress.local.mk-labs.cloud"
          - "  2. Run the setup wizard as the hermes user:"
          - "       sudo -u hermes hermes setup"
          - "  3. After config, start the service:"
          - "       sudo systemctl start hermes"
          - "  4. Verify:"
          - "       sudo systemctl status hermes"
          - "       sudo journalctl -u hermes -f"
          - "============================================================"
          - "Service is ENABLED but NOT STARTED — config required first."
          - "============================================================"
@@ -0,0 +1,7 @@
 ---
 # Run after day1_deploy_nginx_proxy.yml (the role publishes its vhost through it).
 - name: Deploy Hermes Agent (Docker)
  hosts: illuminations
  become: true
  roles:
    - hermes_docker
@@ -0,0 +1,18 @@
 ---
 # ============================================================================
 # day1_deploy_honcho.yml
 # ----------------------------------------------------------------------------
 # Deploys Honcho + pgvector PostgreSQL on the `lincoln` host. Assumes day0
 # host provisioning (linux-baseline + expand_root_lv) is already complete.
 #
 # Run via:
 #   ansible-playbook -i inventory.yml playbooks/day0_provision.yml -e target=lincoln
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_honcho.yml
 # ============================================================================
 - name: Deploy Honcho on lincoln
  hosts: honcho_server
  become: true
  gather_facts: true
  roles:
    - honcho
@@ -0,0 +1,24 @@
 ---
 # ============================================================================
 # day1_deploy_jmri.yml
 # ----------------------------------------------------------------------------
 # Deploys JMRI JmriFaceless headless server on main-street-station.
 # Applies linux-baseline first, then the jmri role.
 #
 # Usage:
 #   ansible-playbook playbooks/day1_deploy_jmri.yml
 #   ansible-playbook playbooks/day1_deploy_jmri.yml -e target=main-street-station
 #
 # Prerequisites:
 #   1. Host is in inventory under jmri_server group
 #   2. jmri_profile_id is set in host_vars/main-street-station.yml
 #   3. SSH access as 'wed' with sudo
 # ============================================================================
 - name: Deploy JMRI headless server
  hosts: "{{ target | default('jmri_server') }}"
  become: true
  gather_facts: true
  roles:
    - linux-baseline
    - jmri
@@ -0,0 +1,25 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day1_deploy_llm_inference.yml
 # DESCRIPTION: Day 1 playbook for astro-orbiter LLM inference stack.
 #              Deploys vLLM + Gemma 2 27B on RTX 3090 via OCuLink.
 #
 # Usage:
 #   cd ~/git/homelab/ansible
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference.yml
 #
 # Phases (added incrementally — safe to re-run):
 #   1. Foundation  — groups, directories, vault assertion
 #   2. Driver      — nvidia-driver-595-open (idempotent; already installed)
 #   3. vLLM        — Python venv + pip install vllm
 #   4. Model       — HF login, Gemma 2 27B snapshot_download
 #   5. Serve       — systemd vllm-serve.service, health check
 #   6. Integration — Hermes provider config on carousel
 # ------------------------------------------------------------------------------
 - name: Deploy LLM inference stack on astro-orbiter
  hosts: astro_orbiter
  gather_facts: true
  roles:
    - role: llm-inference
@@ -0,0 +1,35 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day1_deploy_llm_inference_multimodel.yml
 # DESCRIPTION: Day 1 playbook for the dual-model (aux + tool-calling) rollout
 #              on astro-orbiter. Builds on roles/llm-inference (CUDA/driver
 #              already done) — does not replace it.
 #
 # Usage:
 #   cd ~/git/homelab/ansible
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml
 #   # or scope to specific phases:
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference_multimodel.yml --tags discover
 #
 # EXECUTION CHANNEL (2026-08-12, War Machine): run via the Semaphore template
 # "llm_inference_multimodel_stage_models" (scoped to --tags models). Do NOT
 # run this via direct ansible-playbook or ad-hoc ssh/curl/systemctl — all
 # homelab inference changes go through Ansible roles executed by Semaphore for
 # audit/visibility. Phase 1 (models) is idempotent: it only downloads/stages a
 # GGUF when missing or size-mismatched, and only restarts the router when a new
 # GGUF is detected (normal re-runs that find the files correct touch nothing).
 #
 # Phases (see roles/llm-inference-multimodel/README.md for detail):
 #   0. discover  — read-only; confirm existing Gemma service management
 #   1. models    — idempotent GGUF downloads (Phi-4-14B, Mistral-Small-24B)
 #   2. systemd   — deploy both unit files, do NOT auto-start
 #   3. firewall  — scope ports 8000/8001, non-0.0.0.0 bind
 #   4. verify    — start both services, smoke test, VRAM check
 # ------------------------------------------------------------------------------
 - name: Deploy dual-model LLM inference stack on astro-orbiter
  hosts: astro_orbiter
  gather_facts: true
  roles:
    - role: llm-inference-multimodel
@@ -0,0 +1,106 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day1_deploy_llm_router_shadow.yml
 # DESCRIPTION: Deploy llama-server in router mode on a shadow port (8003).
 #
 # This playbook deploys and validates the llama.cpp router mode supervisor on
 # astro-orbiter (10.1.71.130) WITHOUT touching the production endpoint
 # (llama-server-qwen, port 8002). All 7 dependent Hermes profiles
 # (bruce-banner, groot, happy, heimdall, rocket-raccoon, war-machine, wong)
 # remain pointing at port 8002 throughout this run.
 #
 # Usage (from ~/git/homelab/ansible):
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_router_shadow.yml
 #
 # Tag-scoped runs (if you need to re-run one phase):
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_router_shadow.yml \
 #     --tags router_systemd,router_firewall,router_verify
 #
 # Execution path (Ryan-approved 2026-08-12, task t_0cca74a2):
 #   Direct ansible-playbook as documented exception — Semaphore template for
 #   this role does not exist yet. Create template after cutover is confirmed.
 #   This is the same exception pattern used in prior sessions on this box.
 #
 # Pre-requisites:
 #   1. llama-server binary at /opt/llama.cpp/build/bin/llama-server supports
 #      router mode (confirmed 2026-08-12: --models-dir flag present in --help).
 #   2. /opt/models/ contains ONLY Qwen3.6-35B-A3B-UD-Q4_K_S.gguf
 #      (confirmed 2026-08-12: directory is clean, Phi-4/Mistral already deleted).
 #   3. Port 8002 is in use by the production llama-server-qwen service —
 #      this playbook does NOT touch it.
 #
 # Validation gates this playbook runs (all hard gates EXCEPT Gate 4):
 #   Gate 1: /v1/models reports Qwen with n_ctx >= 64000 (64K Hermes floor)
 #   Gate 2: Tool-calling probe through router returns finish_reason=tool_calls
 #   Gate 2b: Hallucination stress test does NOT trigger spurious tool_calls
 #   Gate 3: nvidia-smi VRAM <= 23,000 MiB (--models-max 1 confirmed effective)
 #   Gate 4: Bundled SvelteKit UI check (nice-to-have, non-blocking)
 #
 # What happens after this playbook:
 #   War Machine posts validation gate results to Ryan.
 #   Ryan reviews and signs off on cutover (or requests changes).
 #   War Machine then runs day2_cutover_qwen_to_router.yml (not yet created)
 #   to promote the router to port 8002 and retire the bare llama-server-qwen.
 #
 # Reference: proposal at
 #   ~/friday/system/inbox/agents/war-machine/2026-08-12-qwen-router-mode-proposal.md
 # Task: t_0cca74a2
 # Author: War Machine (2026-08-12)
 # ------------------------------------------------------------------------------
 - name: Deploy llama-server router (shadow, port 8003) on astro-orbiter
  hosts: astro_orbiter
  gather_facts: true
  become: true
  vars:
    # Enable the router phase — this is the ONLY var that makes router.yml run.
    # Default in defaults/main.yml is false (no-op). Flip here for the shadow run.
    llm_router_enabled: true
    # Qwen model ID as it appears in /v1/models from the router.
    # llama-server router uses the GGUF filename (without .gguf) as the model id.
    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
  roles:
    - role: llm-inference-multimodel
  # No --tags needed here: router.yml is included dynamically from main.yml
  # whenever llm_router_enabled: true. The full role runs but the
  # discover/models/systemd/verify phases are gated on their own vars
  # (llm_qwen_service_enabled etc.) and are idempotent. The stale
  # models.yml (Phi-4/Mistral download tasks) uses variables no longer
  # defined — a follow-up cleanup task should update that file.
 - name: "POST-VALIDATION SAFETY NET — ensure production service is running"
  hosts: astro_orbiter
  gather_facts: false
  become: true
  tasks:
    # Always run this, regardless of whether the validation play succeeded.
    # If the router.yml play stopped llama-server-qwen for VRAM validation
    # and then a gate failed (play aborted), this play ensures it comes back up.
    - name: "Ensure llama-server-qwen (port 8002) is running after validation (always)"
      ansible.builtin.systemd:
        name: llama-server-qwen
        state: started
        enabled: true
      ignore_errors: true   # don't fail if the unit doesn't exist
    - name: "Verify production /health after safety-net restart"
      ansible.builtin.uri:
        url: "http://10.1.71.130:8002/health"
        status_code: 200
        timeout: 30
      register: llm_safety_net_health
      failed_when: false
      ignore_errors: true
    - name: "Report production status (safety-net check)"
      ansible.builtin.debug:
        msg: >-
          Safety-net: llama-server-qwen :8002 health check returned
          {{ llm_safety_net_health.status | default('UNREACHABLE') }}.
          {{ 'OK — production is up.' if (llm_safety_net_health.status | default(0) | int == 200)
             else 'WARNING — production may not be healthy. Check manually.' }}
@@ -0,0 +1,91 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day1_deploy_nextcloud.yml
 # DESCRIPTION: Deploys Nextcloud on the-grid
 #              Runs: docker-host → nextcloud
 #
 # USAGE:
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_nextcloud.yml
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_nextcloud.yml --limit the-grid
 #
 # SECRETS REQUIRED IN VAULT (group_vars/all/vault):
 #   vault_nextcloud_db_root_password
 #   vault_nextcloud_db_password
 #   vault_nextcloud_admin_user
 #   vault_nextcloud_admin_password
 # ------------------------------------------------------------------------------
 - name: Deploy Nextcloud on the-grid
  hosts: the-grid
  become: true
  vars:
    # NFS prerequisite
    nfs_packages:
      - nfs-common
    # Compose stack location
    nextcloud_base_dir: /opt/docker/nextcloud
    # Secrets from vault
    nextcloud_db_root_password: "{{ vault_nextcloud_db_root_password }}"
    nextcloud_db_password:      "{{ vault_nextcloud_db_password }}"
    nextcloud_admin_user:       "{{ vault_nextcloud_admin_user }}"
    nextcloud_admin_password:   "{{ vault_nextcloud_admin_password }}"
  pre_tasks:
    - name: Install NFS client
      ansible.builtin.apt:
        name: "{{ nfs_packages }}"
        state: present
        update_cache: true
  roles:
    - role: docker-host
  tasks:
    - name: Create Nextcloud directory
      ansible.builtin.file:
        path: "{{ nextcloud_base_dir }}"
        state: directory
        owner: "{{ ansible_user }}"
        group: docker
        mode: "0775"
    - name: Deploy Compose file
      ansible.builtin.copy:
        src: "{{ playbook_dir }}/../../boilerplates/nextcloud/compose.yml"
        dest: "{{ nextcloud_base_dir }}/compose.yml"
        owner: "{{ ansible_user }}"
        group: docker
        mode: "0644"
    - name: Deploy .env from vault
      ansible.builtin.copy:
        content: |
          # Managed by Ansible — do not edit manually
          MYSQL_ROOT_PASSWORD={{ nextcloud_db_root_password }}
          MYSQL_PASSWORD={{ nextcloud_db_password }}
          NEXTCLOUD_ADMIN_USER={{ nextcloud_admin_user }}
          NEXTCLOUD_ADMIN_PASSWORD={{ nextcloud_admin_password }}
        dest: "{{ nextcloud_base_dir }}/.env"
        owner: "{{ ansible_user }}"
        group: docker
        mode: "0600"
    - name: Start Nextcloud stack
      community.docker.docker_compose_v2:
        project_src: "{{ nextcloud_base_dir }}"
        state: present
    - name: Wait for Nextcloud to become ready
      ansible.builtin.uri:
        url: "http://the-grid.local.mk-labs.cloud/status.php"
        status_code: 200
        return_content: true
      register: nextcloud_status
      until: >
        nextcloud_status.status == 200 and
        (nextcloud_status.content | from_json).installed == true
      retries: 20
      delay: 15
@@ -0,0 +1,6 @@
 ---
 - name: Deploy shared nginx reverse proxy and certbot
  hosts: illuminations
  become: true
  roles:
    - nginx-proxy
@@ -0,0 +1,35 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: ansible/playbooks/day1_deploy_ollama.yml
 # DESCRIPTION: Deploys Ollama with ROCm GPU acceleration on astro-orbiter.
 #              Assumes day0_baseline.yml has already run (common role complete).
 #              PCIe passthrough for the RX 5700 must be configured in Proxmox
 #              and the GPU must be visible to the VM before running this playbook.
 #
 # Pre-flight check:
 #   ssh wed@astro-orbiter 'lspci | grep -i amd'
 #   Should show the RX 5700 before proceeding.
 #
 # Usage (from ansible/ directory):
 #   ansible-playbook playbooks/day1_deploy_ollama.yml
 # ------------------------------------------------------------------------------
 - name: Deploy Ollama with ROCm on astro-orbiter
  hosts: astro-orbiter
  become: true
  pre_tasks:
    - name: Verify AMD GPU is visible to the VM
      command: lspci
      register: lspci_output
      changed_when: false
    - name: Fail if no AMD GPU detected
      fail:
        msg: >
          No AMD GPU detected via lspci. Verify PCIe passthrough is configured
          in Proxmox and the RX 5700 is visible to the VM before proceeding.
      when: "'AMD' not in lspci_output.stdout and 'Radeon' not in lspci_output.stdout"
  roles:
    - ollama
@@ -0,0 +1,17 @@
 ---
 # ============================================================================
 # day1_deploy_semaphore.yml
 # ----------------------------------------------------------------------------
 # Deploys SemaphoreUI + PostgreSQL on the imagineering host (figment).
 # Run AFTER day0_linux_baseline.yml has been applied to the target.
 #
 # Usage:
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_semaphore.yml
 # ============================================================================
 - name: Deploy SemaphoreUI on imagineering
  hosts: semaphore_server
  become: true
  gather_facts: true
  roles:
    - semaphore
@@ -0,0 +1,18 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/deploy_traefik.yml
 # DESCRIPTION: Deploys Traefik reverse proxy on lightning-lane.
 #              Installs Docker and configures Traefik with Cloudflare DNS-01
 #              certificate resolution for *.local.mk-labs.cloud
 #
 # USAGE:
 #   ansible-playbook -i inventory.yml playbooks/deploy_traefik.yml
 # ------------------------------------------------------------------------------
 - name: Deploy Traefik reverse proxy
  hosts: load_balancers
  become: true
  roles:
    - docker-host
    - traefik
@@ -0,0 +1,18 @@
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day1_deploy_vllm.yml
 # Deploy vLLM to a target host via roles/deploy-vllm.
 #
 # Staging run (deploy + validate WITHOUT touching production traffic):
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_vllm.yml --limit astro-orbiter
 #
 # Cutover run (once staging is validated and Ryan/JARVIS approve flipping
 # traffic — starts and enables the systemd unit(s), runs Phase 5 verification):
 #   ansible-playbook -i inventory.yml playbooks/day1_deploy_vllm.yml \
 #     --limit astro-orbiter --extra-vars "vllm_service_state=started"
 # ------------------------------------------------------------------------------
 - name: Deploy vLLM inference serving stack
  hosts: astro-orbiter
  become: false
  gather_facts: true
  roles:
    - deploy-vllm
@@ -0,0 +1,7 @@
 ---
 - name: Apply common role
  hosts: "{{ target | default('all') }}"
  become: true
  roles:
    - common
    - docker-host
@@ -0,0 +1,259 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day2_add_coder_alias.yml
 # DESCRIPTION: Add Qwen2.5-Coder-14B-Instruct-Q4_K_M to the llama-server-router
 #              on astro-orbiter (10.1.71.130:8002).
 #
 # Context (t_55c164f5, 2026-08-13):
 #   Ryan requested a Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf be added to the
 #   astro-orbiter router with:
 #     alias = "Qwen2.5-Coder-14B-Instruct-4bit"
 #     n_gpu_layers = 99
 #     ctx_size = 16384
 #     flash_attn = true
 #   Deployed GitOps-style via this role; no hand-editing of the live preset.
 #
 # What this playbook does:
 #   1. Downloads Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf into /opt/models if
 #      not already present (idempotent: size-check guard, no re-pull on match).
 #   2. Redeploys the preset INI (adding the [Qwen2.5-Coder-14B-Instruct-Q4_K_M]
 #      section with alias = Qwen2.5-Coder-14B-Instruct-4bit).
 #   3. Restarts llama-server-router to pick up the new model entry.
 #   4. Verifies /v1/models returns all 4 models including the new Coder entry.
 #
 # VRAM context note (t_55c164f5):
 #   Qwen2.5-Coder-14B Q4_K_M: ~8.4GB weights + ~0.6GB KV @ 16K ctx ≈ 9.0GB
 #   Qwen3.6-35B-A3B: ~21.5GB
 #   Full co-residency is impossible on 24GB. LRU eviction handles this:
 #   when Coder is requested, Qwen3.6-35B is evicted (and vice versa).
 #   Model-switching incurs ~30-60s cold-load latency — expected and acceptable.
 #   Phi (~4.3GB) or Llama (~5.6GB) can co-reside with Coder (total ~14GB).
 #
 # Usage (from ~/git/homelab/ansible):
 #   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook -i inventory.yml \
 #     playbooks/day2_add_coder_alias.yml
 #
 # Semaphore note: Semaphore SSH key for jarvis user is not loaded in the
 # container (known pitfall, homelab-llm-serving skill). Run via CLI with
 # id_jarvis key; document as exception per Ryan's standing CLI fallback directive.
 #
 # Author: War Machine (2026-08-13, t_55c164f5)
 # ------------------------------------------------------------------------------
 - name: "Add Qwen2.5-Coder-14B-Instruct-4bit alias to astro-orbiter router"
  hosts: astro_orbiter
  gather_facts: false
  become: true
  vars:
    # Activate preset mode
    llm_router_preset_enabled: true
    llm_router_preset_path: /opt/llama-server-router-preset.ini
    # Production port (router is on 8002 since t_cd0d5388)
    llm_router_port: 8002
    # Per-model ctx-size settings (carried from t_ryan_per_model_ctx; Coder new)
    llm_router_llama_ctx_size: 8192
    llm_router_llama_flash_attn: "true"
    llm_router_phi_ctx_size: 32768
    llm_router_phi_flash_attn: "true"
    llm_router_coder_ctx_size: 16384
    llm_router_coder_flash_attn: "true"
    # All other vars inherit from host_vars + defaults/main.yml.
    # Explicitly set the ones needed by the unit/template tasks for clarity:
    llm_router_enabled: true
    llm_service_user: jarvis
    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
    llm_models_dir: /opt/models
    llm_bind_address: "10.1.71.130"
    llm_allowed_source_cidr: "10.1.70.0/24"
    llm_router_service_name: llama-server-router
    llm_router_bind_address: "10.1.71.130"
    llm_router_allowed_source_cidr: "10.1.70.0/24"
    llm_router_models_dir: /opt/models
    llm_router_models_max: 4            # from host_vars; bumped by t_33acbb2e
    llm_router_ctx_size: 65536          # Qwen3.6-35B default; per-model overrides above
    llm_router_parallel: 1
    llm_router_gpu_layers: 99
    llm_router_batch_size: 2048
    llm_router_ubatch_size: 512
    llm_router_cache_type_k: q4_0
    llm_router_cache_type_v: q4_0
    llm_router_flash_attn: "auto"
    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
    llm_router_vram_max_mib: 23000
    # Coder model staging entry (used below)
    coder_filename: "Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
    coder_url: "https://huggingface.co/bartowski/Qwen2.5-Coder-14B-Instruct-GGUF/resolve/main/Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf"
    coder_size_bytes: 8988111072
  handlers:
    - name: reload systemd
      ansible.builtin.systemd:
        daemon_reload: true
      become: true
      listen: "reload systemd"
    - name: restart router
      ansible.builtin.systemd:
        name: llama-server-router
        state: restarted
      become: true
      listen: "restart router"
  tasks:
    # ==========================================================================
    # PHASE 1: Download Coder GGUF if not present / size mismatch
    # ==========================================================================
    - name: "[coder] Stat existing GGUF"
      ansible.builtin.stat:
        path: "{{ llm_models_dir }}/{{ coder_filename }}"
        get_checksum: false
      register: coder_stat
    - name: "[coder] Download GGUF (skip if present and size matches)"
      ansible.builtin.get_url:
        url: "{{ coder_url }}"
        dest: "{{ llm_models_dir }}/{{ coder_filename }}"
        owner: "{{ llm_service_user }}"
        group: "{{ llm_service_user }}"
        mode: "0644"
        timeout: 3600
      when: >
        not coder_stat.stat.exists or
        coder_stat.stat.size != coder_size_bytes
      register: coder_download
      notify: restart router
    - name: "[coder] Confirm GGUF size post-download"
      ansible.builtin.stat:
        path: "{{ llm_models_dir }}/{{ coder_filename }}"
        get_checksum: false
      register: coder_stat_post
    - name: "[coder] FAIL if GGUF size mismatch after download"
      ansible.builtin.fail:
        msg: >-
          GGUF size mismatch: expected {{ coder_size_bytes }} bytes,
          got {{ coder_stat_post.stat.size }} bytes.
          Re-download may be needed.
      when: coder_stat_post.stat.size != coder_size_bytes
    # ==========================================================================
    # PHASE 2: Deploy updated preset INI (adds Coder section)
    # ==========================================================================
    - name: "[coder] Deploy preset INI to {{ llm_router_preset_path }}"
      ansible.builtin.template:
        src: "../roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2"
        dest: "{{ llm_router_preset_path }}"
        owner: root
        group: root
        mode: "0644"
      register: coder_preset_deployed
      notify: restart router
    # ==========================================================================
    # PHASE 3: Redeploy systemd unit (unchanged flags, but ensures unit is fresh)
    # ==========================================================================
    - name: "[coder] Deploy llama-server-router unit"
      ansible.builtin.template:
        src: "../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
        dest: /etc/systemd/system/llama-server-router.service
        owner: root
        group: root
        mode: "0644"
      register: coder_unit_deployed
      notify:
        - reload systemd
        - restart router
    - name: "[coder] Flush handlers (daemon-reload + router restart)"
      ansible.builtin.meta: flush_handlers
    # ==========================================================================
    # PHASE 4: Verify router is up and Coder model appears in /v1/models
    # ==========================================================================
    - name: "[coder] Wait for /health (router supervisor)"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
        status_code: 200
        timeout: 30
      retries: 12
      delay: 5
      register: coder_health
      until: coder_health.status == 200
    - name: "[coder] Query /v1/models"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
        status_code: 200
        return_content: true
        timeout: 30
      register: coder_models
    - name: "[coder] Extract model IDs and aliases"
      ansible.builtin.set_fact:
        coder_model_ids: "{{ coder_models.json.data | map(attribute='id') | list }}"
        coder_all_aliases: "{{ coder_models.json.data | map(attribute='aliases') | flatten | list }}"
    - name: "[coder] FAIL if Coder primary ID missing"
      ansible.builtin.fail:
        msg: >-
          'Qwen2.5-Coder-14B-Instruct-Q4_K_M' not in /v1/models.
          IDs: {{ coder_model_ids }}
      when: "'Qwen2.5-Coder-14B-Instruct-Q4_K_M' not in coder_model_ids"
    - name: "[coder] FAIL if Coder alias missing"
      ansible.builtin.fail:
        msg: >-
          'Qwen2.5-Coder-14B-Instruct-4bit' not found as ID or alias in /v1/models.
          IDs:     {{ coder_model_ids }}
          Aliases: {{ coder_all_aliases }}
      when:
        - "'Qwen2.5-Coder-14B-Instruct-4bit' not in coder_model_ids"
        - "'Qwen2.5-Coder-14B-Instruct-4bit' not in coder_all_aliases"
    - name: "[coder] FAIL if Qwen3.6-35B missing"
      ansible.builtin.fail:
        msg: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in /v1/models. IDs: {{ coder_model_ids }}"
      when: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in coder_model_ids"
    - name: "[coder] FAIL if Phi missing"
      ansible.builtin.fail:
        msg: "'Phi-3.5-mini-instruct-Q8_0' not in /v1/models. IDs: {{ coder_model_ids }}"
      when: "'Phi-3.5-mini-instruct-Q8_0' not in coder_model_ids"
    - name: "[coder] FAIL if Llama missing"
      ansible.builtin.fail:
        msg: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in /v1/models. IDs: {{ coder_model_ids }}"
      when: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in coder_model_ids"
    - name: "[coder] PASS — full /v1/models summary"
      ansible.builtin.debug:
        msg:
          - "========================================================================"
          - "QWEN2.5-CODER-14B ALIAS DEPLOYMENT — COMPLETE"
          - ""
          - "  Mode:     --models-preset ({{ llm_router_preset_path }})"
          - "  Service:  llama-server-router.service (:{{ llm_router_port }})"
          - ""
          - "  /v1/models IDs:     {{ coder_model_ids }}"
          - "  /v1/models aliases: {{ coder_all_aliases }}"
          - ""
          - "  VERIFY:"
          - "  Qwen3.6-35B-A3B-UD-Q4_K_S:          {{ 'PRESENT' if 'Qwen3.6-35B-A3B-UD-Q4_K_S' in coder_model_ids else 'MISSING' }}"
          - "  Phi-3.5-mini-instruct-Q8_0:          {{ 'PRESENT' if 'Phi-3.5-mini-instruct-Q8_0' in coder_model_ids else 'MISSING' }}"
          - "  Meta-Llama-3.1-8B-Instruct-Q4_K_M:  {{ 'PRESENT' if 'Meta-Llama-3.1-8B-Instruct-Q4_K_M' in coder_model_ids else 'MISSING' }}"
          - "  Qwen2.5-Coder-14B-Instruct-Q4_K_M:  {{ 'PRESENT' if 'Qwen2.5-Coder-14B-Instruct-Q4_K_M' in coder_model_ids else 'MISSING' }}"
          - "  Qwen2.5-Coder-14B-Instruct-4bit:    {{ 'PRESENT (ID)' if 'Qwen2.5-Coder-14B-Instruct-4bit' in coder_model_ids else ('PRESENT (alias)' if 'Qwen2.5-Coder-14B-Instruct-4bit' in coder_all_aliases else 'MISSING') }}"
          - ""
          - "  GGUF download: {{ 'NEW DOWNLOAD' if (coder_download is defined and coder_download.changed) else 'ALREADY PRESENT (skipped)' }}"
          - "========================================================================"
@@ -0,0 +1,313 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day2_add_nomic_embed.yml
 # DESCRIPTION: Add nomic-embed-text-v1.5-Q4_K_M to the llama-server-router
 #              on astro-orbiter (10.1.71.130:8002).
 #
 # Context (t_34b96e83, 2026-08-13, OpenViking Phase 1b):
 #   Ryan approved adding nomic-embed-text-v1.5-Q4_K_M as an embedding model
 #   after Phase 0 follow-up confirmed embedding models fold cleanly into the
 #   existing router preset via embedding=true. Model ID is "nomic-embed-text-v1.5".
 #   No alias needed — peter-parker and Honcho consumers will call it by the section
 #   name directly.
 #
 # What this playbook does:
 #   1. Downloads nomic-embed-text-v1.5-Q4_K_M.gguf into /opt/models if not
 #      already present (idempotent: exact size-check guard, no re-pull on match).
 #   2. Redeploys the preset INI (adding the [nomic-embed-text-v1.5] section with
 #      embedding=true, n-gpu-layers=99, ctx-size=8192, load-on-startup=true,
 #      sleep-idle-seconds=-1).
 #   3. Restarts llama-server-router to pick up the new model entry.
 #   4. Verifies /v1/models returns all 5 models including the new nomic entry.
 #   5. Runs a /v1/embeddings smoke test to confirm the model actually embeds.
 #
 # VRAM context note (t_34b96e83):
 #   nomic-embed-text-v1.5 Q4_K_M: ~84MB weights, embedding model (no KV cache).
 #   VRAM impact is negligible — always pinned via sleep-idle-seconds=-1.
 #   The 4 generative models remain unchanged (OOM analysis unchanged from t_55c164f5).
 #
 # Usage (from ~/git/homelab/ansible):
 #   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook -i inventory.yml \
 #     playbooks/day2_add_nomic_embed.yml
 #
 # Semaphore note: Semaphore SSH key for jarvis user is not loaded in the
 # container (known pitfall, homelab-llm-serving skill). Run via CLI with
 # id_jarvis key; document as exception per Ryan's standing CLI fallback directive.
 #
 # Author: War Machine (2026-08-13, t_34b96e83)
 # ------------------------------------------------------------------------------
 - name: "Add nomic-embed-text-v1.5 embedding model to astro-orbiter router"
  hosts: astro_orbiter
  gather_facts: false
  become: true
  vars:
    # Activate preset mode
    llm_router_preset_enabled: true
    llm_router_preset_path: /opt/llama-server-router-preset.ini
    # Production port (router is on 8002 since t_cd0d5388)
    llm_router_port: 8002
    # Per-model ctx-size settings (carried from t_55c164f5; nomic new)
    llm_router_llama_ctx_size: 8192
    llm_router_llama_flash_attn: "true"
    llm_router_phi_ctx_size: 32768
    llm_router_phi_flash_attn: "true"
    llm_router_coder_ctx_size: 16384
    llm_router_coder_flash_attn: "true"
    llm_router_nomic_ctx_size: 8192
    # NOTE (2026-08-14, t_openviking_embed_batch): per-model batch-size/
    # ubatch-size lines in the preset INI are NOT honored by llama-server's
    # router — only ctx-size is applied per-model; batch-size/ubatch-size for
    # every spawned child come from the router's own global CLI flags
    # (confirmed via `ps aux` on astro-orbiter: child process launched with
    # the router's --batch-size/--ubatch-size regardless of the INI values).
    # Kept below for documentation/future-proofing but the REAL fix is the
    # global llm_router_batch_size / llm_router_ubatch_size override further
    # down, which raises the physical batch for ALL models on this router
    # (Qwen3.6-35B, Phi, Llama, Coder, nomic).
    llm_router_nomic_batch_size: 4096
    llm_router_nomic_ubatch_size: 4096
    # All other vars inherit from host_vars + defaults/main.yml.
    llm_router_enabled: true
    llm_service_user: jarvis
    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
    llm_models_dir: /opt/models
    llm_bind_address: "10.1.71.130"
    llm_allowed_source_cidr: "10.1.70.0/24"
    llm_router_service_name: llama-server-router
    llm_router_bind_address: "10.1.71.130"
    llm_router_allowed_source_cidr: "10.1.70.0/24"
    llm_router_models_dir: /opt/models
    llm_router_models_max: 4            # from host_vars; bumped by t_33acbb2e
    llm_router_ctx_size: 65536          # Qwen3.6-35B default; per-model overrides above
    llm_router_parallel: 1
    llm_router_gpu_layers: 99
    # FIX (2026-08-14, t_openviking_embed_batch): raised from 512 to 4096.
    # This is a GLOBAL router flag applied to every spawned model process
    # (per-model INI batch-size/ubatch-size overrides are not honored by
    # llama-server's router — see note above nomic vars). 512 tokens was too
    # small for OpenViking's chunked-document embedding inputs (observed
    # 2000-3400 tokens/chunk), causing hard 500 errors ("input (N tokens) is
    # too large to process") that tripped OpenViking's circuit breaker into a
    # permanent fail/re-enqueue loop. 4096 comfortably covers observed chunk
    # sizes and stays under nomic's ctx-size=8192. VRAM impact of raising
    # ubatch-size is in compute-buffer scratch space, not KV cache; monitored
    # post-deploy against the 23000 MiB budget (host_vars/astro-orbiter).
    llm_router_batch_size: 4096
    llm_router_ubatch_size: 4096
    llm_router_cache_type_k: q4_0
    llm_router_cache_type_v: q4_0
    llm_router_flash_attn: "auto"
    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
    llm_router_vram_max_mib: 23000
    # nomic model staging
    nomic_filename: "nomic-embed-text-v1.5-Q4_K_M.gguf"
    nomic_url: "https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF/resolve/main/nomic-embed-text-v1.5.Q4_K_M.gguf"
    nomic_size_bytes: 84106624
  handlers:
    - name: reload systemd
      ansible.builtin.systemd:
        daemon_reload: true
      become: true
      listen: "reload systemd"
    - name: restart router
      ansible.builtin.systemd:
        name: llama-server-router
        state: restarted
      become: true
      listen: "restart router"
  tasks:
    # ==========================================================================
    # PHASE 1: Download nomic GGUF if not present / size mismatch
    # ==========================================================================
    - name: "[nomic] Stat existing GGUF"
      ansible.builtin.stat:
        path: "{{ llm_models_dir }}/{{ nomic_filename }}"
        get_checksum: false
      register: nomic_stat
    - name: "[nomic] Download GGUF (skip if present and size matches)"
      ansible.builtin.get_url:
        url: "{{ nomic_url }}"
        dest: "{{ llm_models_dir }}/{{ nomic_filename }}"
        owner: "{{ llm_service_user }}"
        group: "{{ llm_service_user }}"
        mode: "0644"
        timeout: 300
      when: >
        not nomic_stat.stat.exists or
        nomic_stat.stat.size != nomic_size_bytes
      register: nomic_download
      notify: restart router
    - name: "[nomic] Confirm GGUF size post-download"
      ansible.builtin.stat:
        path: "{{ llm_models_dir }}/{{ nomic_filename }}"
        get_checksum: false
      register: nomic_stat_post
    - name: "[nomic] FAIL if GGUF size mismatch after download"
      ansible.builtin.fail:
        msg: >-
          GGUF size mismatch: expected {{ nomic_size_bytes }} bytes,
          got {{ nomic_stat_post.stat.size }} bytes.
          Re-download may be needed.
      when: nomic_stat_post.stat.size != nomic_size_bytes
    # ==========================================================================
    # PHASE 2: Deploy updated preset INI (adds nomic-embed-text-v1.5 section)
    # ==========================================================================
    - name: "[nomic] Deploy preset INI to {{ llm_router_preset_path }}"
      ansible.builtin.template:
        src: "../roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2"
        dest: "{{ llm_router_preset_path }}"
        owner: root
        group: root
        mode: "0644"
      register: nomic_preset_deployed
      notify: restart router
    # ==========================================================================
    # PHASE 3: Redeploy systemd unit (ensures unit is fresh; no flag changes)
    # ==========================================================================
    - name: "[nomic] Deploy llama-server-router unit"
      ansible.builtin.template:
        src: "../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
        dest: /etc/systemd/system/llama-server-router.service
        owner: root
        group: root
        mode: "0644"
      register: nomic_unit_deployed
      notify:
        - reload systemd
        - restart router
    - name: "[nomic] Flush handlers (daemon-reload + router restart)"
      ansible.builtin.meta: flush_handlers
    # ==========================================================================
    # PHASE 4: Verify router is up and nomic model appears in /v1/models
    # ==========================================================================
    - name: "[nomic] Wait for /health (router supervisor)"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
        status_code: 200
        timeout: 30
      retries: 12
      delay: 5
      register: nomic_health
      until: nomic_health.status == 200
    - name: "[nomic] Query /v1/models"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
        status_code: 200
        return_content: true
        timeout: 30
      register: nomic_models
    - name: "[nomic] Extract model IDs and aliases"
      ansible.builtin.set_fact:
        nomic_model_ids: "{{ nomic_models.json.data | map(attribute='id') | list }}"
        nomic_all_aliases: "{{ nomic_models.json.data | map(attribute='aliases') | flatten | list }}"
    - name: "[nomic] FAIL if nomic primary ID missing"
      ansible.builtin.fail:
        msg: >-
          'nomic-embed-text-v1.5' not in /v1/models.
          IDs: {{ nomic_model_ids }}
      when: "'nomic-embed-text-v1.5' not in nomic_model_ids"
    - name: "[nomic] FAIL if Qwen3.6-35B missing"
      ansible.builtin.fail:
        msg: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in /v1/models. IDs: {{ nomic_model_ids }}"
      when: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in nomic_model_ids"
    - name: "[nomic] FAIL if Phi missing"
      ansible.builtin.fail:
        msg: "'Phi-3.5-mini-instruct-Q8_0' not in /v1/models. IDs: {{ nomic_model_ids }}"
      when: "'Phi-3.5-mini-instruct-Q8_0' not in nomic_model_ids"
    - name: "[nomic] FAIL if Llama missing"
      ansible.builtin.fail:
        msg: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in /v1/models. IDs: {{ nomic_model_ids }}"
      when: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in nomic_model_ids"
    - name: "[nomic] FAIL if Coder missing"
      ansible.builtin.fail:
        msg: "'Qwen2.5-Coder-14B-Instruct-Q4_K_M' not in /v1/models. IDs: {{ nomic_model_ids }}"
      when: "'Qwen2.5-Coder-14B-Instruct-Q4_K_M' not in nomic_model_ids"
    # ==========================================================================
    # PHASE 5: /v1/embeddings smoke test — confirm model actually embeds
    # ==========================================================================
    - name: "[nomic] POST /v1/embeddings smoke test"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/embeddings"
        method: POST
        body_format: json
        body:
          model: "nomic-embed-text-v1.5"
          input: "The dog ran across the park."
        status_code: 200
        return_content: true
        timeout: 120
      register: nomic_embed_result
    - name: "[nomic] Extract embedding vector length"
      ansible.builtin.set_fact:
        nomic_embed_dims: >-
          {{ (nomic_embed_result.json.data | first).embedding | length }}
      when:
        - nomic_embed_result.status == 200
        - nomic_embed_result.json.data is defined
        - nomic_embed_result.json.data | length > 0
    - name: "[nomic] FAIL if embedding vector is empty or missing"
      ansible.builtin.fail:
        msg: >-
          Embedding smoke test returned no vector.
          Response: {{ nomic_embed_result.json }}
      when: >-
        nomic_embed_result.status != 200 or
        nomic_embed_result.json.data is not defined or
        nomic_embed_result.json.data | length == 0 or
        (nomic_embed_result.json.data | first).embedding | length == 0
    - name: "[nomic] PASS — full summary"
      ansible.builtin.debug:
        msg:
          - "========================================================================"
          - "NOMIC-EMBED-TEXT-V1.5 DEPLOYMENT — COMPLETE"
          - ""
          - "  Mode:     --models-preset ({{ llm_router_preset_path }})"
          - "  Service:  llama-server-router.service (:{{ llm_router_port }})"
          - ""
          - "  /v1/models IDs:     {{ nomic_model_ids }}"
          - ""
          - "  VERIFY:"
          - "  Qwen3.6-35B-A3B-UD-Q4_K_S:          {{ 'PRESENT' if 'Qwen3.6-35B-A3B-UD-Q4_K_S' in nomic_model_ids else 'MISSING' }}"
          - "  Phi-3.5-mini-instruct-Q8_0:          {{ 'PRESENT' if 'Phi-3.5-mini-instruct-Q8_0' in nomic_model_ids else 'MISSING' }}"
          - "  Meta-Llama-3.1-8B-Instruct-Q4_K_M:  {{ 'PRESENT' if 'Meta-Llama-3.1-8B-Instruct-Q4_K_M' in nomic_model_ids else 'MISSING' }}"
          - "  Qwen2.5-Coder-14B-Instruct-Q4_K_M:  {{ 'PRESENT' if 'Qwen2.5-Coder-14B-Instruct-Q4_K_M' in nomic_model_ids else 'MISSING' }}"
          - "  nomic-embed-text-v1.5:               {{ 'PRESENT' if 'nomic-embed-text-v1.5' in nomic_model_ids else 'MISSING' }}"
          - ""
          - "  Embedding smoke test: PASS"
          - "  Vector dimensions: {{ nomic_embed_dims | default('unknown') }}"
          - ""
          - "  GGUF download: {{ 'NEW DOWNLOAD' if (nomic_download is defined and nomic_download.changed) else 'ALREADY PRESENT (skipped)' }}"
          - "========================================================================"
@@ -0,0 +1,203 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day2_add_phi_alias.yml
 # DESCRIPTION: Add Phi-3.5-mini-instruct-8bit alias to the llama-server-router
 #              by switching from --models-dir to --models-preset INI mode.
 #
 # Context (t_9adf0889, 2026-08-12):
 #   Ryan's Hermes config (auxiliary.title_generation.model) points to
 #   "Phi-3.5-mini-instruct-8bit" but the router only exposes the GGUF
 #   filename-derived ID "Phi-3.5-mini-instruct-Q8_0". They are the same file.
 #   This playbook adds the alias so both names work without changing Ryan's
 #   Hermes config.
 #
 # What this playbook does:
 #   1. Deploys the preset INI template (llama-server-router-preset.ini.j2)
 #      to /opt/llama-server-router-preset.ini on astro-orbiter.
 #   2. Redeploys the systemd unit (llama-server-router.service) with
 #      --models-preset instead of --models-dir.
 #   3. Restarts llama-server-router to pick up the new flag.
 #   4. Verifies that /v1/models returns:
 #      - Phi-3.5-mini-instruct-Q8_0 (original ID — must still work)
 #      - Phi-3.5-mini-instruct-8bit (new alias — Ryan's config target)
 #      - Qwen3.6-35B-A3B-UD-Q4_K_S (unchanged)
 #      - Meta-Llama-3.1-8B-Instruct-Q4_K_M (unchanged)
 #
 # Known upstream behavior:
 #   GH #22364: --models-preset creates an extra "default" entry in /v1/models.
 #   This is cosmetic only and does not affect model selection by name.
 #
 # Usage (from ~/git/homelab/ansible):
 #   ansible-playbook -i inventory.yml playbooks/day2_add_phi_alias.yml
 #
 # Semaphore note (t_9adf0889): Semaphore SSH key for jarvis user is not loaded
 # in the container (known pitfall, homelab-llm-serving skill). Run via CLI with
 # id_jarvis key; document as exception per Ryan's standing CLI fallback directive.
 #
 # Author: War Machine (2026-08-12, t_9adf0889)
 # ------------------------------------------------------------------------------
 - name: "Add Phi-3.5-mini-instruct-8bit alias — switch router to preset mode"
  hosts: astro_orbiter
  gather_facts: false
  become: true
  vars:
    # Activate preset mode and provide the on-disk INI path
    llm_router_preset_enabled: true
    llm_router_preset_path: /opt/llama-server-router-preset.ini
    # Production port (router is already on 8002 since t_cd0d5388)
    llm_router_port: 8002
    # All other vars inherit from host_vars + defaults/main.yml.
    # Explicitly set the ones needed by the unit template for clarity:
    llm_router_enabled: true
    llm_service_user: jarvis
    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
    llm_models_dir: /opt/models
    llm_bind_address: "10.1.71.130"
    llm_allowed_source_cidr: "10.1.70.0/24"
    llm_router_service_name: llama-server-router
    llm_router_bind_address: "10.1.71.130"
    llm_router_allowed_source_cidr: "10.1.70.0/24"
    llm_router_models_dir: /opt/models
    llm_router_models_max: 4            # from host_vars; bumped by t_33acbb2e
    llm_router_ctx_size: 65536
    llm_router_parallel: 1
    llm_router_gpu_layers: 99
    llm_router_batch_size: 2048
    llm_router_ubatch_size: 512
    llm_router_cache_type_k: q4_0
    llm_router_cache_type_v: q4_0
    llm_router_flash_attn: "auto"
    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
    llm_router_vram_max_mib: 23000
  handlers:
    - name: reload systemd
      ansible.builtin.systemd:
        daemon_reload: true
      become: true
      listen: "reload systemd"
    - name: restart router
      ansible.builtin.systemd:
        name: llama-server-router
        state: restarted
      become: true
      listen: "restart router"
  tasks:
    # ==========================================================================
    # PHASE 1: Deploy the preset INI
    # ==========================================================================
    - name: "[phi-alias] Deploy preset INI to {{ llm_router_preset_path }}"
      ansible.builtin.template:
        src: "../roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2"
        dest: "{{ llm_router_preset_path }}"
        owner: root
        group: root
        mode: "0644"
      register: phi_alias_preset_deployed
      notify:
        - restart router
    # ==========================================================================
    # PHASE 2: Redeploy systemd unit with --models-preset flag
    # ==========================================================================
    - name: "[phi-alias] Deploy llama-server-router unit (--models-preset mode)"
      ansible.builtin.template:
        src: "../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
        dest: /etc/systemd/system/llama-server-router.service
        owner: root
        group: root
        mode: "0644"
      register: phi_alias_unit_deployed
      notify:
        - reload systemd
        - restart router
    - name: "[phi-alias] Flush handlers (daemon-reload + router restart)"
      ansible.builtin.meta: flush_handlers
    # ==========================================================================
    # PHASE 3: Verify alias is present
    # ==========================================================================
    - name: "[phi-alias] Wait for /health (router supervisor, no model needed)"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
        status_code: 200
        timeout: 30
      retries: 12
      delay: 5
      register: phi_alias_health
      until: phi_alias_health.status == 200
    - name: "[phi-alias] Query /v1/models"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
        status_code: 200
        return_content: true
        timeout: 30
      register: phi_alias_models
    - name: "[phi-alias] Extract model IDs and aliases"
      ansible.builtin.set_fact:
        phi_alias_model_ids: "{{ phi_alias_models.json.data | map(attribute='id') | list }}"
        phi_alias_all_aliases: "{{ phi_alias_models.json.data | map(attribute='aliases') | flatten | list }}"
        phi_alias_model_sources: "{{ phi_alias_models.json.data | map(attribute='source') | list }}"
    - name: "[phi-alias] FAIL if Phi original ID missing"
      ansible.builtin.fail:
        msg: >-
          'Phi-3.5-mini-instruct-Q8_0' not in /v1/models.
          IDs: {{ phi_alias_model_ids }}
      when: "'Phi-3.5-mini-instruct-Q8_0' not in phi_alias_model_ids"
    - name: "[phi-alias] FAIL if Phi alias missing"
      ansible.builtin.fail:
        msg: >-
          'Phi-3.5-mini-instruct-8bit' not found as ID or alias in /v1/models.
          IDs:     {{ phi_alias_model_ids }}
          Aliases: {{ phi_alias_all_aliases }}
      when:
        - "'Phi-3.5-mini-instruct-8bit' not in phi_alias_model_ids"
        - "'Phi-3.5-mini-instruct-8bit' not in phi_alias_all_aliases"
    - name: "[phi-alias] FAIL if Qwen missing"
      ansible.builtin.fail:
        msg: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in /v1/models. IDs: {{ phi_alias_model_ids }}"
      when: "'Qwen3.6-35B-A3B-UD-Q4_K_S' not in phi_alias_model_ids"
    - name: "[phi-alias] FAIL if Llama missing"
      ansible.builtin.fail:
        msg: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in /v1/models. IDs: {{ phi_alias_model_ids }}"
      when: "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' not in phi_alias_model_ids"
    - name: "[phi-alias] PASS — full /v1/models summary"
      ansible.builtin.debug:
        msg:
          - "========================================================================"
          - "PHI ALIAS DEPLOYMENT — COMPLETE"
          - ""
          - "  Mode:     --models-preset ({{ llm_router_preset_path }})"
          - "  Service:  llama-server-router.service (:{{ llm_router_port }})"
          - ""
          - "  /v1/models IDs:     {{ phi_alias_model_ids }}"
          - "  /v1/models aliases: {{ phi_alias_all_aliases }}"
          - "  Sources:            {{ phi_alias_model_sources }}"
          - ""
          - "  VERIFY:"
          - "  Phi-3.5-mini-instruct-Q8_0:  {{ 'PRESENT' if 'Phi-3.5-mini-instruct-Q8_0' in phi_alias_model_ids else 'MISSING' }}"
          - "  Phi-3.5-mini-instruct-8bit:  {{ 'PRESENT (ID)' if 'Phi-3.5-mini-instruct-8bit' in phi_alias_model_ids else ('PRESENT (alias)' if 'Phi-3.5-mini-instruct-8bit' in phi_alias_all_aliases else 'MISSING') }}"
          - "  Qwen3.6-35B-A3B-UD-Q4_K_S:  {{ 'PRESENT' if 'Qwen3.6-35B-A3B-UD-Q4_K_S' in phi_alias_model_ids else 'MISSING' }}"
          - "  Meta-Llama-3.1-8B-Instruct-Q4_K_M: {{ 'PRESENT' if 'Meta-Llama-3.1-8B-Instruct-Q4_K_M' in phi_alias_model_ids else 'MISSING' }}"
          - ""
          - "  GH #22364: if 'default' appears in IDs above, that is expected"
          - "  in --models-preset mode. Cosmetic only."
          - "========================================================================"
@@ -0,0 +1,161 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day2_bump_router_models_max.yml
 # DESCRIPTION: Bump --models-max on the production llama-server-router unit.
 #
 # Context: t_33acbb2e (2026-08-12) — Ryan requested --models-max raised from 1
 # to 4 so the router can keep multiple GGUFs resident on-demand (LRU eviction
 # when the cap is reached). The actual var change lives in:
 #   host_vars/astro-orbiter/vars.yml  (llm_router_models_max: 4)
 #
 # This playbook:
 #   1. Re-renders llama-server-router.service.j2 with the updated var value.
 #   2. Reloads systemd (daemon-reload handler) if the unit changed.
 #   3. Restarts llama-server-router so the new --models-max takes effect on the
 #      live process. Router holds no resident model (all-unloaded) so restart
 #      is sub-second and non-disruptive.
 #   4. Verifies /health returns 200 and /v1/models still lists all three GGUFs.
 #
 # VRAM NOTE: --models-max 4 allows up to all 3 current GGUFs to co-reside on
 # a 24GB card simultaneously. Worst-case combined footprint is ~31GB which
 # EXCEEDS 24GB — OOM is possible if all 3 are loaded concurrently. In normal
 # single-user homelab operation this is very unlikely. Full VRAM breakdown
 # documented in host_vars/astro-orbiter/vars.yml. Ryan approved (t_33acbb2e).
 #
 # Execution channel: Semaphore template "llm_router_update_unit" (project mk-labs).
 # Do NOT run via direct ansible-playbook or ad-hoc ssh/systemctl.
 #
 # Author: War Machine (2026-08-12, t_33acbb2e)
 # ------------------------------------------------------------------------------
 - name: "Bump llama-server-router --models-max to 4 on astro-orbiter"
  hosts: astro_orbiter
  gather_facts: true
  become: true
  vars:
    # Production vars — router is live on :8002 (post-cutover t_cd0d5388)
    llm_router_port: 8002
    llm_router_bind_address: "10.1.71.130"
    # llm_router_models_max is 4 via host_vars/astro-orbiter/vars.yml.
    # Remaining role vars come from host_vars + defaults/main.yml via the
    # inventory — we only explicitly set vars this playbook needs for its
    # own tasks (health/models check URIs).
    # Needed by the template task (mirrors defaults set in role defaults/main.yml)
    llm_service_user: jarvis
    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
    llm_models_dir: /opt/models
    llm_router_service_name: llama-server-router
    llm_router_models_dir: /opt/models
    llm_router_gpu_layers: 99
    llm_router_ctx_size: 65536
    llm_router_flash_attn: "auto"
    llm_router_cache_type_k: q4_0
    llm_router_cache_type_v: q4_0
    llm_router_batch_size: 2048
    llm_router_ubatch_size: 512
    llm_router_parallel: 1
  tasks:
    # -------------------------------------------------------------------------
    # Phase 1: Re-render the router unit file
    # Template src path is relative to the role's templates/ dir; we reference
    # it with a relative path that Ansible resolves from the role directory.
    # -------------------------------------------------------------------------
    - name: "Deploy updated llama-server-router unit (--models-max {{ llm_router_models_max }})"
      ansible.builtin.template:
        src: "{{ playbook_dir }}/../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
        dest: "/etc/systemd/system/{{ llm_router_service_name }}.service"
        owner: root
        group: root
        mode: "0644"
      register: llm_router_unit_updated
      notify:
        - reload systemd
      tags: [always]
    - name: "Flush handlers — ensure daemon-reload lands before restart"
      ansible.builtin.meta: flush_handlers
      tags: [always]
    # -------------------------------------------------------------------------
    # Phase 2: Restart the router so the new --models-max takes effect.
    # Always restart (even if unit unchanged) to ensure live process matches.
    # -------------------------------------------------------------------------
    - name: "Restart llama-server-router so --models-max {{ llm_router_models_max }} takes effect"
      ansible.builtin.systemd:
        name: "{{ llm_router_service_name }}"
        state: restarted
        enabled: true
      tags: [always]
    # -------------------------------------------------------------------------
    # Phase 3: Verify /health returns 200
    # -------------------------------------------------------------------------
    - name: "Wait for /health to return 200 after restart"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
        status_code: 200
        timeout: 30
      register: bump_health_check
      retries: 10
      delay: 3
      until: bump_health_check.status == 200
      tags: [always]
    # -------------------------------------------------------------------------
    # Phase 4: Verify /v1/models lists all three GGUFs
    # -------------------------------------------------------------------------
    - name: "Check /v1/models — all three GGUFs should appear"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
        status_code: 200
        timeout: 30
        return_content: true
      register: bump_models_check
      tags: [always]
    - name: "Display /v1/models summary"
      ansible.builtin.debug:
        msg:
          - "======================================================================"
          - "--models-max BUMP VERIFICATION (t_33acbb2e)"
          - ""
          - "  /health:               HTTP {{ bump_health_check.status }}"
          - "  /v1/models HTTP:       {{ bump_models_check.status }}"
          - "  Models listed:         {{ bump_models_check.json.data | map(attribute='id') | list | join(', ') }}"
          - ""
          - "  --models-max now:      {{ llm_router_models_max }}"
          - "  --parallel (unchanged): {{ llm_router_parallel }}"
          - ""
          - "  VRAM WARNING: worst-case 3-model co-residency ~31GB > 24GB RTX 3090."
          - "  OOM risk if all 3 load concurrently. LRU eviction mitigates in practice."
          - "  Full breakdown: host_vars/astro-orbiter/vars.yml"
          - "======================================================================"
      when: bump_models_check is defined
      tags: [always]
    - name: "GATE: confirm all 3 expected GGUFs appear in /v1/models"
      ansible.builtin.assert:
        that:
          - "'Qwen3.6-35B-A3B-UD-Q4_K_S' in (bump_models_check.json.data | map(attribute='id') | list)"
          - "'Phi-3.5-mini-instruct-Q8_0' in (bump_models_check.json.data | map(attribute='id') | list)"
          - "'Meta-Llama-3.1-8B-Instruct-Q4_K_M' in (bump_models_check.json.data | map(attribute='id') | list)"
        fail_msg: >-
          /v1/models did not return all 3 expected GGUFs after --models-max bump.
          Check router logs: journalctl -u llama-server-router -n 50
        success_msg: "GATE PASSED: all 3 GGUFs listed in /v1/models."
      when: bump_models_check is defined
      tags: [always]
  handlers:
    - name: reload systemd
      ansible.builtin.systemd:
        daemon_reload: true
      listen: "reload systemd"
@@ -0,0 +1,59 @@
 ---
 # ------------------------------------------------------------------------------
 # Playbook: day2_cpu_offload_aux_models.yml
 # Purpose:  CPU-offload Qwen2.5-Coder-14B and Meta-Llama-3.1-8B on
 #           astro-orbiter's production router (port 8002).
 #
 # What this playbook does:
 #   1. Re-renders llama-server-router-preset.ini (Coder + Llama sections now
 #      use per-model n-gpu-layers vars = 0 -> full CPU inference).
 #   2. Re-renders the router unit (--models-max now 4 via host_vars, global
 #      --n-gpu-layers removed per t_72646029 unit template fix) and restarts
 #      llama-server-router so both changes take effect.
 #   3. Verifies per the role's router_preset phase.
 #
 # Context (2026-08-17):
 #   - RAM/model-swap audit, TIER 1 (Coder-14B CPU offload) + TIER 2
 #     (Llama-3.1-8B CPU offload) — Ryan approved 1 & 2 on 2026-08-17.
 #     See inbox/ryan/2026-08-17-llm-system-ram-model-swap.md.
 #   - Unit template fix (t_72646029): global --n-gpu-layers removed from
 #     ExecStart in preset mode. Each INI section now sets n-gpu-layers
 #     explicitly (Qwen3.8=99, Phi=99, nomic=99, Coder=0, Llama=0).
 #   - Concurrent residency after change: Qwen3.8-27B (20,302 MiB @ 128K ctx)
 #     + nomic-embed (558 MiB, pinned) + Coder (CPU, ~1,390 MiB CUDA ctx) +
 #     Llama (CPU, ~1,706 MiB CUDA ctx) = ~24,004 MiB. NOTE: llama.cpp 6ea215d
 #     allocates CUDA-context VRAM even at n-gpu-layers=0, so CPU models are not
 #     0-VRAM; total sits at the 24,576 MiB physical limit (headroom ~572 MiB).
 #     Qwen3.8 is never evicted for a CPU aux model; Phi-3.5-mini (GPU, 8.3GB)
 #     still evicts as before.
 #   - CPU speed (8-core Ryzen 7 5800XT): ~5-10 tok/s (14B), ~10-20 tok/s (8B).
 #   - Semaphore SSH gap for astro-orbiter still applies (t_730f9584 /
 #     t_33acbb2e); running direct CLI Ansible per standing exception.
 #
 # Run:
 #   cd /home/hermes/git/homelab/ansible
 #   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook \
 #     -i inventory.yml \
 #     playbooks/day2_cpu_offload_aux_models.yml
 #
 # Rollback:
 #   git checkout -- \
 #     roles/llm-inference-multimodel/templates/llama-server-router.service.j2 \
 #     roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2 \
 #     roles/llm-inference-multimodel/defaults/main.yml \
 #     host_vars/astro-orbiter/vars.yml
 #   (restores n-gpu-layers=99 global flag, models-max=2, all GPU)
 #   then re-run this playbook to redeploy rollback state.
 #   Note: playbooks/day2_cpu_offload_aux_models.yml is untracked — left on disk.
 # ------------------------------------------------------------------------------
 - name: CPU-offload Coder-14B and Llama-3.1-8B on astro-orbiter
  hosts: astro-orbiter
  become: true
  vars:
    llm_router_preset_enabled: true
    llm_router_enabled: true
    llm_router_port: 8002
  roles:
    - role: llm-inference-multimodel
      tags: [always]
@@ -0,0 +1,511 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day2_cutover_qwen_to_router.yml
 # DESCRIPTION: Promote llama-server-router to production on port 8002.
 #
 # Context: Router-mode shadow deployment (t_0cca74a2) validated 2026-08-12:
 #   all 4 hard gates PASSED (n_ctx 65536, tool-calling PASS, hallucination-stress
 #   PASS, VRAM 20410 MiB / 1 process). Ryan approved cutover.
 #
 # This playbook makes the router the permanent production endpoint:
 #
 #   1. Stop + disable llama-server-qwen (:8002). Unit file is PRESERVED on disk
 #      as the rollback target (same pattern as prior role history).
 #   2. Redeploy llama-server-router unit file with --port 8002 (production port).
 #      PORT DECISION: we rebind the router to :8002 rather than updating 8
 #      dependent Hermes profiles' base_url. One unit file change beats 8
 #      config.yaml updates — atomic, GitOps-clean, zero profile drift.
 #   3. Enable + start llama-server-router on :8002.
 #   4. Re-run validation gates 1-3 against the NOW-production endpoint.
 #      (Same logic as Phase R / router_verify in tasks/router.yml — hard gates.)
 #   5. Run Gate 4: verify bundled SvelteKit UI is reachable.
 #
 # Usage (from ~/git/homelab/ansible):
 #   ansible-playbook -i inventory.yml playbooks/day2_cutover_qwen_to_router.yml
 #
 # Rollback (if gates fail or any time after):
 #   ansible-playbook -i inventory.yml playbooks/day2_cutover_qwen_to_router.yml \
 #     --tags cutover_rollback
 #
 # Author: War Machine (2026-08-12, t_cd0d5388)
 # Approved by: Ryan (cutover authorization, 2026-08-12)
 # ------------------------------------------------------------------------------
 - name: "CUTOVER — Promote llama-server-router to production (:8002) on astro-orbiter"
  hosts: astro_orbiter
  gather_facts: true
  become: true
  vars:
    # ----------------------------------------------------------------
    # PORT DECISION:
    # We rebind the router to :8002 (production port) rather than
    # updating 8 dependent Hermes profiles' base_url to :8003.
    # Rationale: one unit file change is atomic and GitOps-clean.
    # Updating 8 config.yaml files risks drift and requires per-profile
    # activation tests. The template renders llm_router_port as the
    # --port argument; we just override it here to 8002.
    # ----------------------------------------------------------------
    # Router port override: take over production port
    llm_router_port: 8002
    # All other role defaults needed by the template (mirrors defaults/main.yml)
    llm_service_user: jarvis
    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
    llm_models_dir: /opt/models
    llm_bind_address: "10.1.71.130"
    llm_allowed_source_cidr: "10.1.70.0/24"
    llm_router_enabled: true
    llm_router_service_name: llama-server-router
    llm_router_models_dir: /opt/models
    llm_router_models_max: 1               # CRITICAL: RTX 3090 24GB, single model only
    llm_router_ctx_size: 65536
    llm_router_parallel: 1
    llm_router_gpu_layers: 99
    llm_router_batch_size: 2048
    llm_router_ubatch_size: 512
    llm_router_cache_type_k: q4_0
    llm_router_cache_type_v: q4_0
    llm_router_flash_attn: "auto"
    llm_router_bind_address: "10.1.71.130"
    llm_router_allowed_source_cidr: "10.1.70.0/24"
    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
    llm_router_vram_max_mib: 23000
    llm_qwen_service_name: llama-server-qwen
    llm_qwen_port: 8002
  tasks:
    # =======================================================================
    # PHASE 1 — Stop and disable llama-server-qwen (bare single-model)
    # Preserve unit file on disk — rollback target per existing role pattern.
    # =======================================================================
    - name: "[cutover] PHASE 1: Confirm llama-server-qwen current state"
      ansible.builtin.systemd:
        name: llama-server-qwen
      register: cutover_qwen_status
      tags: [cutover_stop_qwen, cutover]
    - name: "[cutover] PHASE 1: Report current llama-server-qwen status"
      ansible.builtin.debug:
        msg: >-
          llama-server-qwen: ActiveState={{ cutover_qwen_status.status.ActiveState | default('unknown') }},
          UnitFileState={{ cutover_qwen_status.status.UnitFileState | default('unknown') }}.
          Will stop + disable. Unit file preserved at /etc/systemd/system/llama-server-qwen.service as rollback target.
      tags: [cutover_stop_qwen, cutover]
    - name: "[cutover] PHASE 1: Stop llama-server-qwen (:8002, bare single-model)"
      ansible.builtin.systemd:
        name: llama-server-qwen
        state: stopped
      register: cutover_qwen_stopped
      tags: [cutover_stop_qwen, cutover]
    - name: "[cutover] PHASE 1: Disable llama-server-qwen (prevent auto-start on reboot)"
      ansible.builtin.systemd:
        name: llama-server-qwen
        enabled: false
      tags: [cutover_stop_qwen, cutover]
    - name: "[cutover] PHASE 1: Wait 5s for VRAM to be released"
      ansible.builtin.pause:
        seconds: 5
      when: cutover_qwen_stopped.changed | default(false)
      tags: [cutover_stop_qwen, cutover]
    - name: "[cutover] PHASE 1: Verify port 8002 is now free"
      ansible.builtin.command:
        cmd: ss -ltnp
      register: cutover_port_check
      changed_when: false
      tags: [cutover_stop_qwen, cutover]
    - name: "[cutover] PHASE 1: Fail if port 8002 is still bound"
      ansible.builtin.fail:
        msg: >-
          Port 8002 is still bound after stopping llama-server-qwen.
          Check 'ss -ltnp | grep :8002' and resolve before the router can bind.
      when:
        - "':8002 ' in (cutover_port_check.stdout | default('')) or ':8002:' in (cutover_port_check.stdout | default(''))"
      tags: [cutover_stop_qwen, cutover]
    - name: "[cutover] PHASE 1: Report VRAM state (should be empty)"
      ansible.builtin.command:
        cmd: nvidia-smi --query-compute-apps=pid,name,used_memory --format=csv,noheader
      register: cutover_vram_free_check
      changed_when: false
      tags: [cutover_stop_qwen, cutover]
    - name: "[cutover] PHASE 1: Print VRAM state"
      ansible.builtin.debug:
        msg: >-
          VRAM after stopping llama-server-qwen:
          {{ cutover_vram_free_check.stdout if (cutover_vram_free_check.stdout | length > 0)
             else '(no GPU processes — VRAM free)' }}
      tags: [cutover_stop_qwen, cutover]
    # =======================================================================
    # PHASE 2 — Redeploy llama-server-router unit with --port 8002
    # =======================================================================
    - name: "[cutover] PHASE 2: Deploy llama-server-router unit file (port 8002 — production)"
      ansible.builtin.template:
        src: "../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
        dest: /etc/systemd/system/llama-server-router.service
        owner: root
        group: root
        mode: "0644"
      register: cutover_router_unit_deployed
      notify:
        - reload systemd
      tags: [cutover_deploy_unit, cutover]
    - name: "[cutover] PHASE 2: Flush handlers (daemon-reload before start)"
      ansible.builtin.meta: flush_handlers
      tags: [cutover_deploy_unit, cutover]
    # =======================================================================
    # PHASE 3 — Enable + start llama-server-router on :8002
    # =======================================================================
    - name: "[cutover] PHASE 3: Enable + start llama-server-router (production, :8002)"
      ansible.builtin.systemd:
        name: llama-server-router
        state: "{{ 'restarted' if (cutover_router_unit_deployed.changed | default(false)) else 'started' }}"
        enabled: true
        daemon_reload: true
      tags: [cutover_start_router, cutover]
    # =======================================================================
    # PHASE 4 — Validation gates 1-3 (hard gates against now-production :8002)
    # =======================================================================
    - name: "[cutover] GATE 1a: Wait for router /health on :8002 (up to 5min — cold model load)"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
        status_code: 200
      register: cutover_health
      retries: 30
      delay: 10
      until: cutover_health.status == 200
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 1a: Trigger model load (router lazy-loads on first request)"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/chat/completions"
        method: POST
        body_format: json
        body:
          model: "{{ llm_router_expected_model_id }}"
          messages:
            - role: user
              content: "Reply with one word: hello"
          max_tokens: 5
          temperature: 0.0
        status_code: 200
        return_content: true
        timeout: 300
      register: cutover_warmup
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 1a: Report warmup"
      ansible.builtin.debug:
        msg:
          - "Model loaded. finish_reason={{ cutover_warmup.json.choices[0].finish_reason | default('unknown') }}"
          - "Response: {{ cutover_warmup.json.choices[0].message.content | default('(empty)') | truncate(100) }}"
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 1b: Query /v1/models on :8002"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
        status_code: 200
        return_content: true
      register: cutover_models
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 1b: Fail if expected model ID not found"
      ansible.builtin.fail:
        msg: >-
          GATE 1 FAIL: '{{ llm_router_expected_model_id }}' not found in /v1/models.
          Returned: {{ cutover_models.json.data | map(attribute='id') | list }}
      when:
        - cutover_models.json.data | selectattr('id', 'equalto', llm_router_expected_model_id) | list | length == 0
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 1b: Extract ctx-size from router model args"
      ansible.builtin.set_fact:
        cutover_qwen_n_ctx: >-
          {%- set model = cutover_models.json.data | selectattr('id', 'equalto', llm_router_expected_model_id) | first -%}
          {%- set args = model.status.args -%}
          {%- set ctx_idx = args.index('--ctx-size') if '--ctx-size' in args else -1 -%}
          {{ args[ctx_idx + 1] | int if ctx_idx >= 0 else 0 }}
      when:
        - cutover_models.json.data | selectattr('id', 'equalto', llm_router_expected_model_id) | list | length > 0
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 1b: Fail if n_ctx < 64000"
      ansible.builtin.fail:
        msg: "GATE 1 FAIL: --ctx-size={{ cutover_qwen_n_ctx }} < 64000 (Hermes 64K floor)."
      when:
        - cutover_qwen_n_ctx is defined
        - cutover_qwen_n_ctx | int < 64000
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 1b: PASS — n_ctx >= 64K"
      ansible.builtin.debug:
        msg: "GATE 1 PASS: --ctx-size={{ cutover_qwen_n_ctx }} >= 64000."
      when:
        - cutover_qwen_n_ctx is defined
        - cutover_qwen_n_ctx | int >= 64000
      tags: [cutover_validate, cutover]
    # --- Gate 2: Tool-calling through router proxy ---
    - name: "[cutover] GATE 2: Tool-calling probe"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/chat/completions"
        method: POST
        body_format: json
        body:
          model: "{{ llm_router_expected_model_id }}"
          messages:
            - role: user
              content: "What is the current weather in Chicago? Use the provided tool."
          tools:
            - type: function
              function:
                name: get_weather
                description: "Get current weather conditions for a city"
                parameters:
                  type: object
                  properties:
                    city:
                      type: string
                      description: "The city name"
                  required:
                    - city
          temperature: 0.0
        status_code: 200
        return_content: true
        timeout: 120
      register: cutover_toolcall_probe
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 2: Fail if not finish_reason=tool_calls"
      ansible.builtin.fail:
        msg: >-
          GATE 2 FAIL: finish_reason={{ cutover_toolcall_probe.json.choices[0].finish_reason | default('(missing)') }}
          (expected tool_calls). Response: {{ cutover_toolcall_probe.json | to_json }}
      when:
        - cutover_toolcall_probe.json.choices[0].finish_reason | default('') != 'tool_calls'
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 2: PASS"
      ansible.builtin.debug:
        msg:
          - "GATE 2 PASS: finish_reason=tool_calls"
          - "function: {{ cutover_toolcall_probe.json.choices[0].message.tool_calls[0].function.name | default('(unknown)') }}"
          - "arguments: {{ cutover_toolcall_probe.json.choices[0].message.tool_calls[0].function.arguments | default('(none)') }}"
      when:
        - cutover_toolcall_probe.json.choices[0].finish_reason | default('') == 'tool_calls'
      tags: [cutover_validate, cutover]
    # --- Gate 2b: Hallucination stress ---
    - name: "[cutover] GATE 2b: Hallucination stress probe"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/chat/completions"
        method: POST
        body_format: json
        body:
          model: "{{ llm_router_expected_model_id }}"
          messages:
            - role: user
              content: "Tell me a brief fact about the planet Mars. Do not call any functions."
          tools:
            - type: function
              function:
                name: get_weather
                description: "Get current weather conditions for a city"
                parameters:
                  type: object
                  properties:
                    city:
                      type: string
                  required:
                    - city
          temperature: 0.1
        status_code: 200
        return_content: true
        timeout: 120
      register: cutover_halluc_probe
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 2b: Fail if spurious tool_calls"
      ansible.builtin.fail:
        msg: >-
          GATE 2b FAIL: finish_reason=tool_calls on unrelated prompt (Mars fact).
          Over-triggering through router. Response: {{ cutover_halluc_probe.json | to_json }}
      when:
        - cutover_halluc_probe.json.choices[0].finish_reason | default('') == 'tool_calls'
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 2b: PASS"
      ansible.builtin.debug:
        msg: "GATE 2b PASS: finish_reason={{ cutover_halluc_probe.json.choices[0].finish_reason }} — no spurious tool_calls."
      when:
        - cutover_halluc_probe.json.choices[0].finish_reason | default('') != 'tool_calls'
      tags: [cutover_validate, cutover]
    # --- Gate 3: VRAM guard ---
    - name: "[cutover] GATE 3: Check VRAM usage (--models-max 1 guard)"
      ansible.builtin.command:
        cmd: nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv,noheader
      register: cutover_vram_post
      changed_when: false
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 3: Parse VRAM used MiB"
      ansible.builtin.set_fact:
        cutover_vram_used_mib: "{{ cutover_vram_post.stdout.split(',')[0].strip().split(' ')[0] | int }}"
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 3: Fail if VRAM exceeds ceiling"
      ansible.builtin.fail:
        msg: >-
          GATE 3 FAIL: {{ cutover_vram_used_mib }} MiB > {{ llm_router_vram_max_mib }} MiB ceiling.
          Full: {{ cutover_vram_post.stdout }}
      when:
        - cutover_vram_used_mib | int > llm_router_vram_max_mib | int
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 3: Count GPU processes"
      ansible.builtin.command:
        cmd: nvidia-smi --query-compute-apps=pid,name --format=csv,noheader
      register: cutover_gpu_procs
      changed_when: false
      failed_when: false
      tags: [cutover_validate, cutover]
    - name: "[cutover] GATE 3: PASS"
      ansible.builtin.debug:
        msg:
          - "GATE 3 PASS: {{ cutover_vram_used_mib }} MiB / {{ llm_router_vram_max_mib }} MiB ceiling."
          - "GPU processes: {{ cutover_gpu_procs.stdout_lines | default(['(none)']) }}"
          - "Full nvidia-smi: {{ cutover_vram_post.stdout }}"
      when:
        - cutover_vram_used_mib | int <= llm_router_vram_max_mib | int
      tags: [cutover_validate, cutover]
    # =======================================================================
    # PHASE 5 — Gate 4: Bundled SvelteKit Web UI (required this time)
    # =======================================================================
    - name: "[cutover] GATE 4: Check bundled SvelteKit UI at :8002"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/"
        status_code: [200, 301, 302]
        return_content: true
        timeout: 30
      register: cutover_ui_check
      failed_when: false
      tags: [cutover_validate, cutover_ui, cutover]
    - name: "[cutover] GATE 4: Inspect UI content"
      ansible.builtin.set_fact:
        cutover_ui_is_html: "{{ 'html' in (cutover_ui_check.content | default('') | lower) or '<!doctype' in (cutover_ui_check.content | default('') | lower) }}"
        cutover_ui_has_model_select: "{{ 'select' in (cutover_ui_check.content | default('') | lower) or 'model' in (cutover_ui_check.content | default('') | lower) }}"
      when: cutover_ui_check is defined
      tags: [cutover_validate, cutover_ui, cutover]
    - name: "[cutover] GATE 4: Report UI check and bookmark URL"
      ansible.builtin.debug:
        msg:
          - "======================================================================"
          - "GATE 4 UI CHECK:"
          - "  HTTP status:          {{ cutover_ui_check.status | default('UNREACHABLE') }}"
          - "  Is HTML:              {{ cutover_ui_is_html | default(false) }}"
          - "  Contains model/select: {{ cutover_ui_has_model_select | default(false) }}"
          - "  BOOKMARK URL:         http://{{ llm_router_bind_address }}:{{ llm_router_port }}/"
          - "  {{ 'GATE 4 PASS — UI serving HTML at :8002.' if (cutover_ui_check.status | default(0) | int in [200, 301, 302]) else 'GATE 4 WARN — UI not reachable (HTTP ' + (cutover_ui_check.status | default('FAIL') | string) + ').' }}"
          - "======================================================================"
      when: cutover_ui_check is defined
      tags: [cutover_validate, cutover_ui, cutover]
    # =======================================================================
    # CUTOVER SUMMARY
    # =======================================================================
    - name: "[cutover] CUTOVER SUMMARY — production promoted"
      ansible.builtin.debug:
        msg:
          - "======================================================================"
          - "CUTOVER COMPLETE: llama-server-router is now production."
          - ""
          - "  Service:  llama-server-router.service (enabled, running)"
          - "  Port:     8002 (unchanged for all 8 Hermes profiles)"
          - "  Model:    {{ llm_router_expected_model_id }}"
          - "  Mode:     Router/supervisor (--models-dir /opt/models, --models-max 1)"
          - ""
          - "  Gate 1 (n_ctx >= 64K):     PASS ({{ cutover_qwen_n_ctx | default('N/A') }})"
          - "  Gate 2 (tool-calling):     PASS (finish_reason=tool_calls)"
          - "  Gate 2b (halluc stress):   PASS (no spurious tool_calls)"
          - "  Gate 3 (VRAM <= 23000MiB): PASS ({{ cutover_vram_used_mib | default('N/A') }} MiB)"
          - "  Gate 4 (Web UI):           HTTP {{ cutover_ui_check.status | default('N/A') }}"
          - ""
          - "  ROLLBACK TARGET: /etc/systemd/system/llama-server-qwen.service (unit preserved)"
          - "  ROLLBACK CMD: sudo systemctl enable --now llama-server-qwen"
          - "               sudo systemctl disable --now llama-server-router"
          - "  Or: ansible-playbook -i inventory.yml day2_cutover_qwen_to_router.yml --tags cutover_rollback"
          - ""
          - "  Web UI bookmark: http://{{ llm_router_bind_address }}:{{ llm_router_port }}/"
          - "======================================================================"
      tags: [cutover]
    # =======================================================================
    # ROLLBACK — tag cutover_rollback reverses the cutover
    # Run: ansible-playbook -i inventory.yml day2_cutover_qwen_to_router.yml --tags cutover_rollback
    # WARNING: rollback_task has no dependency on cutover tags — safe to run standalone.
    # =======================================================================
    - name: "[cutover_rollback] Stop + disable llama-server-router"
      ansible.builtin.systemd:
        name: llama-server-router
        state: stopped
        enabled: false
      tags: [cutover_rollback, never]    # 'never' = only runs with explicit --tags cutover_rollback
    - name: "[cutover_rollback] Enable + start llama-server-qwen (restore bare :8002)"
      ansible.builtin.systemd:
        name: llama-server-qwen
        state: started
        enabled: true
      tags: [cutover_rollback, never]
    - name: "[cutover_rollback] Verify rollback /health"
      ansible.builtin.uri:
        url: "http://{{ llm_bind_address | default('10.1.71.130') }}:8002/health"
        status_code: 200
        timeout: 30
      register: cutover_rollback_health
      failed_when: false
      tags: [cutover_rollback, never]
    - name: "[cutover_rollback] Report rollback result"
      ansible.builtin.debug:
        msg: >-
          ROLLBACK: llama-server-qwen :8002 health returned
          {{ cutover_rollback_health.status | default('UNREACHABLE') }}.
          {{ 'OK — production restored to bare qwen.' if (cutover_rollback_health.status | default(0) | int == 200)
             else 'WARNING — health check failed. Check manually.' }}
      tags: [cutover_rollback, never]
  handlers:
    - name: reload systemd
      ansible.builtin.systemd:
        daemon_reload: true
@@ -0,0 +1,80 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: ansible/playbooks/day2_install_node_exporter.yml
 # DESCRIPTION: Install and configure Prometheus Node Exporter on Linux hosts.
 #
 # Targets two groups with different firewall managers:
 #   - proxmox_nodes  : Proxmox hypervisors (Debian, firewalld not present,
 #                      uses iptables/no firewall — just open the port via UFW
 #                      if present, otherwise skip)
 #   - monitored_vms  : Ubuntu VMs managed by UFW
 #
 # After installing on new hosts, re-runs day1_deploy_monitoring.yml to refresh
 # the Prometheus scrape config is NOT needed — targets are already statically
 # defined in prometheus.yaml.j2 for the Proxmox nodes. For new VMs, add the
 # IP to the prometheus.yaml.j2 proxmox-vms job and re-run day1_deploy_monitoring.yml.
 #
 # Usage:
 #   ansible-playbook -i inventory.yml playbooks/day2_install_node_exporter.yml
 #   ansible-playbook -i inventory.yml playbooks/day2_install_node_exporter.yml --limit proxmox_nodes
 #   ansible-playbook -i inventory.yml playbooks/day2_install_node_exporter.yml --limit monitored_vms
 # ------------------------------------------------------------------------------
 # ── Play 1: Proxmox hypervisors ───────────────────────────────────────────────
 # Proxmox runs Debian. The prometheus.prometheus.node_exporter role installs
 # a binary + systemd service without touching apt, which is what we want on
 # hypervisors (keep the package footprint clean).
 # Note: Proxmox does not run UFW. Port 9100 is open by default on VLAN 71.
 - name: Install Node Exporter on Proxmox hypervisors
  hosts: proxmox_nodes
  become: true
  vars:
    node_exporter_version: "1.9.1"
    node_exporter_web_listen_address: "0.0.0.0:9100"
    node_exporter_enabled_collectors:
      - systemd
      - processes
      - filesystem
      - meminfo
      - cpu
      - diskstats
      - netdev
      - loadavg
      - uname
  roles:
    - role: prometheus.prometheus.node_exporter
 # ── Play 2: Ubuntu VMs ────────────────────────────────────────────────────────
 # Standard Ubuntu hosts with UFW. Same role, adds UFW allow rule for 9100.
 - name: Install Node Exporter on monitored VMs
  hosts: monitored_vms
  become: true
  vars:
    node_exporter_version: "1.9.1"
    node_exporter_web_listen_address: "0.0.0.0:9100"
    node_exporter_enabled_collectors:
      - systemd
      - processes
      - filesystem
      - meminfo
      - cpu
      - diskstats
      - netdev
      - loadavg
      - uname
  pre_tasks:
    - name: Open Node Exporter port in UFW
      community.general.ufw:
        rule: allow
        port: "9100"
        proto: tcp
        comment: "Prometheus Node Exporter"
  roles:
    - role: prometheus.prometheus.node_exporter
@@ -0,0 +1,277 @@
 ---
 # ------------------------------------------------------------------------------
 # FILE: playbooks/day2_per_model_ctx_size.yml
 # DESCRIPTION: Right-size --ctx-size per model workload on llama-server-router
 #              (already in --models-preset mode since t_9adf0889).
 #
 # Context (t_ryan_per_model_ctx, 2026-08-13, requested by Ryan via JARVIS):
 #   All 3 preset models currently launch with a uniform --ctx-size 65536.
 #   This playbook narrows two of them to match actual workload:
 #     - Meta-Llama-3.1-8B-Instruct-Q4_K_M (alias Meta-Llama-3.1-8B-Instruct-4bit):
 #         ctx-size 65536 -> 8192  (tool-routing / micro-tasks: title gen, MCP
 #         tool calls, approval checks)
 #     - Phi-3.5-mini-instruct-Q8_0 (alias Phi-3.5-mini-instruct-8bit):
 #         ctx-size 65536 -> 32768 (long web scrapes / session-log compression)
 #   Both also move flash-attn from "auto" to explicit "true" per Ryan's spec.
 #   Qwen3.6-35B-A3B-UD-Q4_K_S is INTENTIONALLY left untouched at 65536/auto.
 #
 #   Existing aliases (Meta-Llama-3.1-8B-Instruct-4bit, Phi-3.5-mini-instruct-8bit)
 #   are PRESERVED as-is. Ryan's pasted TOML used different alias strings
 #   ("llama-3.1-8b", "phi-3.5-mini") but renaming aliases was not explicitly
 #   requested and would break live Hermes custom_providers routing — flagged
 #   in the deployment report rather than applied silently.
 #
 # IMPORTANT — Hermes side effect: /home/hermes/.hermes/config.yaml declares
 # context_length: 65536 for both these models under custom_providers. This
 # playbook does NOT touch that file (out of role/agent scope) but the value
 # becomes STALE the moment this playbook lands. Flag to JARVIS/Maria Hill.
 #
 # Usage (from ~/git/homelab/ansible):
 #   ansible-playbook -i inventory.yml playbooks/day2_per_model_ctx_size.yml
 #
 # Author: War Machine (2026-08-13, t_ryan_per_model_ctx)
 # ------------------------------------------------------------------------------
 - name: "Right-size per-model ctx-size on llama-server-router (Llama 8k, Phi 32k)"
  hosts: astro_orbiter
  gather_facts: false
  become: true
  vars:
    # Preset mode already active in production (t_9adf0889) — keep it on.
    llm_router_preset_enabled: true
    llm_router_preset_path: /opt/llama-server-router-preset.ini
    llm_router_enabled: true
    # Production port
    llm_router_port: 8002
    llm_router_bind_address: "10.1.71.130"
    llm_router_allowed_source_cidr: "10.1.70.0/24"
    llm_bind_address: "10.1.71.130"
    llm_allowed_source_cidr: "10.1.70.0/24"
    llm_service_user: jarvis
    llm_binary_path: /opt/llama.cpp/build/bin/llama-server
    llm_models_dir: /opt/models
    llm_router_service_name: llama-server-router
    llm_router_models_dir: /opt/models
    llm_router_models_max: 4
    llm_router_parallel: 1
    llm_router_gpu_layers: 99
    llm_router_batch_size: 2048
    llm_router_ubatch_size: 512
    llm_router_cache_type_k: q4_0
    llm_router_cache_type_v: q4_0
    # Qwen — untouched baseline (also used as router-wide fallback default)
    llm_router_ctx_size: 65536
    llm_router_flash_attn: "auto"
    llm_router_expected_model_id: "Qwen3.6-35B-A3B-UD-Q4_K_S"
    llm_router_vram_max_mib: 23000
    # --- THE CHANGE: per-model overrides ---
    llm_router_llama_ctx_size: 8192
    llm_router_llama_flash_attn: "true"
    llm_router_phi_ctx_size: 32768
    llm_router_phi_flash_attn: "true"
  handlers:
    - name: reload systemd
      ansible.builtin.systemd:
        daemon_reload: true
      become: true
      listen: "reload systemd"
    - name: restart router
      ansible.builtin.systemd:
        name: llama-server-router
        state: restarted
      become: true
      listen: "restart router"
  tasks:
    # ==========================================================================
    # PHASE 1: Deploy the preset INI with new per-model ctx-size/flash-attn
    # ==========================================================================
    - name: "[ctx-resize] Deploy preset INI to {{ llm_router_preset_path }}"
      ansible.builtin.template:
        src: "../roles/llm-inference-multimodel/templates/llama-server-router-preset.ini.j2"
        dest: "{{ llm_router_preset_path }}"
        owner: root
        group: root
        mode: "0644"
      register: ctx_resize_preset_deployed
      notify:
        - restart router
    - name: "[ctx-resize] Deploy router systemd unit (drop global --ctx-size/--flash-attn in preset mode)"
      ansible.builtin.template:
        src: "../roles/llm-inference-multimodel/templates/llama-server-router.service.j2"
        dest: /etc/systemd/system/llama-server-router.service
        owner: root
        group: root
        mode: "0644"
      register: ctx_resize_unit_deployed
      notify:
        - reload systemd
        - restart router
    - name: "[ctx-resize] Flush handlers (daemon-reload + router restart if changed)"
      ansible.builtin.meta: flush_handlers
    # ==========================================================================
    # PHASE 2: Verify
    # ==========================================================================
    - name: "[ctx-resize] Wait for /health"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/health"
        status_code: 200
        timeout: 30
      retries: 12
      delay: 5
      register: ctx_resize_health
      until: ctx_resize_health.status == 200
    - name: "[ctx-resize] Query /v1/models"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
        status_code: 200
        return_content: true
        timeout: 30
      register: ctx_resize_models
    - name: "[ctx-resize] Trigger load — Llama (confirms actual load + captures live args)"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/chat/completions"
        method: POST
        body_format: json
        body:
          model: "Meta-Llama-3.1-8B-Instruct-Q4_K_M"
          messages:
            - role: user
              content: "Reply with one word: hello"
          max_tokens: 5
          temperature: 0.0
        status_code: 200
        return_content: true
        timeout: 120
      register: ctx_resize_llama_warmup
    - name: "[ctx-resize] Trigger load — Phi (confirms actual load + captures live args)"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/chat/completions"
        method: POST
        body_format: json
        body:
          model: "Phi-3.5-mini-instruct-Q8_0"
          messages:
            - role: user
              content: "Reply with one word: hello"
          max_tokens: 5
          temperature: 0.0
        status_code: 200
        return_content: true
        timeout: 120
      register: ctx_resize_phi_warmup
    - name: "[ctx-resize] Re-query /v1/models after warmup (final state)"
      ansible.builtin.uri:
        url: "http://{{ llm_router_bind_address }}:{{ llm_router_port }}/v1/models"
        status_code: 200
        return_content: true
        timeout: 30
      register: ctx_resize_models_final
    - name: "[ctx-resize] Extract Llama args"
      ansible.builtin.set_fact:
        ctx_resize_llama_args: >-
          {{ (ctx_resize_models_final.json.data | selectattr('id', 'equalto', 'Meta-Llama-3.1-8B-Instruct-Q4_K_M') | first).status.args }}
        ctx_resize_llama_status: >-
          {{ (ctx_resize_models_final.json.data | selectattr('id', 'equalto', 'Meta-Llama-3.1-8B-Instruct-Q4_K_M') | first).status.value }}
    - name: "[ctx-resize] Extract Phi args"
      ansible.builtin.set_fact:
        ctx_resize_phi_args: >-
          {{ (ctx_resize_models_final.json.data | selectattr('id', 'equalto', 'Phi-3.5-mini-instruct-Q8_0') | first).status.args }}
        ctx_resize_phi_status: >-
          {{ (ctx_resize_models_final.json.data | selectattr('id', 'equalto', 'Phi-3.5-mini-instruct-Q8_0') | first).status.value }}
    - name: "[ctx-resize] Extract Qwen args (must be unchanged)"
      ansible.builtin.set_fact:
        ctx_resize_qwen_args: >-
          {{ (ctx_resize_models_final.json.data | selectattr('id', 'equalto', 'Qwen3.6-35B-A3B-UD-Q4_K_S') | first).status.args }}
    - name: "[ctx-resize] GATE — Llama ctx-size must be 8192"
      ansible.builtin.assert:
        that:
          - "'8192' in ctx_resize_llama_args"
          - ctx_resize_llama_args[ctx_resize_llama_args.index('--ctx-size') + 1] == '8192'
        fail_msg: "Llama ctx-size not 8192. Args: {{ ctx_resize_llama_args }}"
        success_msg: "Llama ctx-size confirmed 8192."
    - name: "[ctx-resize] GATE — Llama flash-attn must be true"
      ansible.builtin.assert:
        that:
          - ctx_resize_llama_args[ctx_resize_llama_args.index('--flash-attn') + 1] == 'true'
        fail_msg: "Llama flash-attn not true. Args: {{ ctx_resize_llama_args }}"
        success_msg: "Llama flash-attn confirmed true."
    - name: "[ctx-resize] GATE — Llama loaded successfully"
      ansible.builtin.assert:
        that:
          - ctx_resize_llama_status == 'loaded'
        fail_msg: "Llama status is '{{ ctx_resize_llama_status }}', expected 'loaded'."
        success_msg: "Llama status confirmed 'loaded'."
    - name: "[ctx-resize] GATE — Phi ctx-size must be 32768"
      ansible.builtin.assert:
        that:
          - ctx_resize_phi_args[ctx_resize_phi_args.index('--ctx-size') + 1] == '32768'
        fail_msg: "Phi ctx-size not 32768. Args: {{ ctx_resize_phi_args }}"
        success_msg: "Phi ctx-size confirmed 32768."
    - name: "[ctx-resize] GATE — Phi flash-attn must be true"
      ansible.builtin.assert:
        that:
          - ctx_resize_phi_args[ctx_resize_phi_args.index('--flash-attn') + 1] == 'true'
        fail_msg: "Phi flash-attn not true. Args: {{ ctx_resize_phi_args }}"
        success_msg: "Phi flash-attn confirmed true."
    - name: "[ctx-resize] GATE — Phi loaded successfully"
      ansible.builtin.assert:
        that:
          - ctx_resize_phi_status == 'loaded'
        fail_msg: "Phi status is '{{ ctx_resize_phi_status }}', expected 'loaded'."
        success_msg: "Phi status confirmed 'loaded'."
    - name: "[ctx-resize] GATE — Qwen ctx-size UNCHANGED at 65536"
      ansible.builtin.assert:
        that:
          - ctx_resize_qwen_args[ctx_resize_qwen_args.index('--ctx-size') + 1] == '65536'
        fail_msg: "Qwen ctx-size changed unexpectedly! Args: {{ ctx_resize_qwen_args }}"
        success_msg: "Qwen ctx-size confirmed UNCHANGED at 65536."
    - name: "[ctx-resize] PASS — summary"
      ansible.builtin.debug:
        msg:
          - "================================================================"
          - "PER-MODEL CTX-SIZE DEPLOYMENT — COMPLETE"
          - ""
          - "  Llama-3.1-8B (Meta-Llama-3.1-8B-Instruct-Q4_K_M):"
          - "    status:     {{ ctx_resize_llama_status }}"
          - "    args:       {{ ctx_resize_llama_args }}"
          - ""
          - "  Phi-3.5-mini (Phi-3.5-mini-instruct-Q8_0):"
          - "    status:     {{ ctx_resize_phi_status }}"
          - "    args:       {{ ctx_resize_phi_args }}"
          - ""
          - "  Qwen3.6-35B-A3B-UD-Q4_K_S: UNCHANGED (ctx-size 65536, args: {{ ctx_resize_qwen_args }})"
          - ""
          - "  ACTION NEEDED: /home/hermes/.hermes/config.yaml custom_providers"
          - "  context_length: 65536 for both Meta-Llama-3.1-8B-Instruct-4bit and"
          - "  Phi-3.5-mini-instruct-8bit is now STALE (actual: 8192 / 32768)."
          - "  Flag to JARVIS/Maria Hill for correction — NOT done by this playbook."
          - "================================================================"
@@ -0,0 +1,38 @@
 ---
 # ------------------------------------------------------------------------------
 # Playbook: day2_qwen38_ctx128k.yml
 # Purpose:  Bump Qwen3.8-27B-Q4_K_M ctx-size from 32768 to 131072 (128K)
 #           on astro-orbiter's production router (port 8002).
 #
 # What this playbook does:
 #   1. Renders the updated llama-server-router-preset.ini.j2 (now with
 #      llm_router_qwen38_ctx_size: 131072) to /opt/llama-server-router-preset.ini.
 #   2. Restarts llama-server-router.service.
 #   3. Verifies the router loads Qwen3.8-27B at ctx=131072 in status.args.
 #
 # Context:
 #   - Empirical VRAM test (t_4455a44c): 131072 ctx = 20,282 MiB Qwen3.8
 #     + 558 MiB nomic-embed = ~20.8GB total; ~3.2GB headroom on 24GB RTX 3090.
 #     Co-resident with nomic-embed: comfortably fits.
 #   - Ryan approved this deployment.
 #   - Semaphore SSH gap for astro-orbiter still applies (t_730f9584 / t_33acbb2e);
 #     running direct CLI Ansible per standing exception.
 #
 # Run:
 #   cd /home/hermes/git/homelab/ansible
 #   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook \
 #     -i inventory.yml \
 #     playbooks/day2_qwen38_ctx128k.yml
 #
 # Task reference: t_441470b9 — War Machine, 2026-08-16
 # ------------------------------------------------------------------------------
 - name: Bump Qwen3.8-27B ctx-size to 131072 on astro-orbiter
  hosts: astro-orbiter
  become: true
  vars:
    llm_router_preset_enabled: true
    llm_router_qwen38_ctx_size: 131072
  roles:
    - role: llm-inference-multimodel
      tags: [preset, systemd, verify]
@@ -0,0 +1,65 @@
 ---
 # ------------------------------------------------------------------------------
 # Playbook: day2_qwen38_ctx128k_rollback.yml
 # Purpose:  Roll back Qwen3.8-27B-Q4_K_M ctx-size from 131072 back to 65536
 #           on astro-orbiter's production router (port 8002).
 #
 # What this playbook does:
 #   1. Renders the updated llama-server-router-preset.ini.j2 (now with
 #      llm_router_qwen38_ctx_size: 65536) to
 #      /opt/llama-server-router-preset.ini.
 #   2. Restarts llama-server-router.service.
 #   3. Verifies the router loads Qwen3.8-27B at ctx=65536 in status.args.
 #
 # Context:
 #   - t_441470b9 (2026-08-16): ctx-size bumped 32768 -> 131072. Verified VRAM
 #     at 131072 ctx with only Qwen3.8 + nomic-embed co-resident: ~20,282 MiB
 #     + 558 MiB = ~20.8 GB on 24 GB RTX 3090. Comfortably safe.
 #   - t_72646029 (2026-08-17): Phi-3.5mini moved to GPU (n-gpu-layers=99)
 #     to enable concurrent residency with CPU-offloaded Coder-14B and
 #     Llama-3.1-8B. This added ~2GB CUDA context buffers for Phi + shifted
 #     Phi's model weights onto the GPU (~3.8GB).
 #   - NEW steady-state VRAM: Qwen3.8 @ 131072 ctx (~20,282 MiB) + nomic-embed
 #     (~558 MiB) + Llama CUDA ctx (~1,706 MiB) + Coder CUDA ctx (~1,390 MiB)
 #     = ~24,004 MiB. Adding Phi-3.5 (~3,800 MiB weights + ~1.4 GB CUDA ctx)
 #     pushes total to ~29,000+ MiB — exceeding the 24,576 MiB RTX 3090 limit.
 #     Qwen3.8-27B-131072 now fails to load (HTTP 500, OOM before llama.cpp
 #     reaches the model-loading phase).
 #   - FIX: reduce Qwen3.8 ctx-size 131072 -> 65536. This reduces KV cache
 #     from ~6GB to ~3GB, freeing ~3GB of VRAM. New estimated steady-state:
 #     Qwen3.8 @ 65536 ctx (~17,068 MiB) + nomic (~558) + Llama ctx (~1,706)
 #     + Coder ctx (~1,390) + Phi-3.5 (~3,800 + ~1,400 CUDA ctx) = ~25,922 MiB.
 #     Still over 24,576 — see "Phase 2" below for the secondary fix.
 #
 #   IMPORTANT: Rolling back ctx-size alone may NOT be sufficient. The
 #   hardware reference (astro-orbiter-hardware.md line 166, t_72646029)
 #   states steady-state ~24,004 MiB WITHOUT Phi on GPU. Adding Phi-3.5 back
 #   to GPU tips it over. This playbook handles the context rollback; if Qwen3.8
 #   still fails to load after Phase R, Wong should escalate to Ryan for a
 #   decision on either (a) offloading Phi-3.5mini to CPU (n-gpu-layers=0),
 #   or (b) adding a second GPU. Document the Phase 2 finding as a separate
 #   follow-up task if needed.
 #
 #   The 64K floor from the 2026-08-12 cutover validation (t_cd0d5388, Gate 1)
 #   still applies — ctx-size=65536 satisfies it.
 #
 # Run:
 #   cd /home/hermes/git/homelab/ansible
 #   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook \
 #     -i inventory.yml \
 #     playbooks/day2_qwen38_ctx128k_rollback.yml
 #
 # Task reference: t_c9fed26c — War Machine benchmark, 2026-08-18
 # Root cause: t_72646029 CPU-offload deployment added Phi-3.5 to GPU,
 #   shifting total VRAM past the 24,576 MiB ceiling when Qwen3.8 runs at 128K.
 # ------------------------------------------------------------------------------
 - name: Roll back Qwen3.8-27B ctx-size to 65536 on astro-orbiter
  hosts: astro-orbiter
  become: true
  vars:
    llm_router_preset_enabled: true
    llm_router_qwen38_ctx_size: 65536
  roles:
    - role: llm-inference-multimodel
      tags: [preset, systemd, verify]
@@ -0,0 +1,36 @@
 ---
 # ------------------------------------------------------------------------------
 # Playbook: day2_swap_qwen38.yml
 # Purpose:  Swap the primary production model on astro-orbiter router from
 #           Qwen3.6-35B-A3B-UD-Q4_K_S to Qwen3.8-27B-Q4_K_M.
 #           This is a GitOps-encoded record of the swap performed 2026-08-16
 #           per Ryan's direction (kanban task t_f5f7e9ad).
 #
 # What this playbook does:
 #   1. Renders the updated llama-server-router-preset.ini.j2 to
 #      /opt/llama-server-router-preset.ini on astro-orbiter.
 #   2. Reloads the llama-server-router service (SIGHUP / restart as needed).
 #   3. Verifies the new model ID appears in /v1/models.
 #
 # Prerequisites:
 #   - Qwen3.8-27B-Q4_K_M.gguf must be present in /opt/models on astro-orbiter.
 #     (Downloaded out-of-band via wget during the swap task.)
 #   - roles/llm-inference-multimodel/defaults/main.yml updated to reference
 #     Qwen3.8-27B-Q4_K_M (done in this same commit).
 #
 # Run:
 #   env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook \
 #     -i inventory.yml \
 #     playbooks/day2_swap_qwen38.yml
 #
 # Task reference: t_f5f7e9ad — War Machine, 2026-08-16
 # ------------------------------------------------------------------------------
 - name: Swap primary model to Qwen3.8-27B-Q4_K_M on astro-orbiter
  hosts: astro-orbiter
  become: true
  vars:
    llm_router_preset_enabled: true
  roles:
    - role: llm-inference-multimodel
      tags: [preset, systemd, verify]
@@ -0,0 +1,22 @@
 ---
 # Playbook: day3_deploy_qwen38_ctx131k.yml
 # Purpose: Deploy Qwen3.8-27B-Q4_K_M ctx-size 65536 -> 131072 to astro-orbiter
 #         via llama-swap config re-render + restart.
 #
 # The git change to defaults/main.yml (line 235: ctx_size: 131072) is already staged.
 # This playbook renders /etc/llama-swap/config.yaml from the updated defaults
 # and restarts llama-swap to load the new ctx-size.
 #
 # Run:
 #   cd /home/hermes/git/homelab/ansible
 #   ansible-playbook -i inventory.yml playbooks/day3_deploy_qwen38_ctx131k.yml
 #
 - name: Deploy Qwen3.8 ctx-size 131072 to astro-orbiter
  hosts: astro-orbiter
  become: true
  vars:
    llm_swapmode_enabled: true
  roles:
    - role: llm-inference-multimodel
      tags: [swapmode_config, swapmode_systemd, swapmode_verify]
@@ -1,30 +0,0 @@
 ---
 # - name: Step 1 - Install Prerequisites
 #   hosts: fastpass
 #   become: true
 #   gather_facts: true
 #   roles:
 #     - role: kubernetes-prerequisites
 # - name: Step 2 - Deploy First Control Plane Node
 #   hosts: fastpass_control_plane[0]
 #   become: true
 #   gather_facts: false
 #   roles:
 #     - role: fastpass-first-control-plane
 - name: Step 3 - Deploy Additional Control Plane Nodes
  hosts: fastpass_control_plane[1:]
  become: true
  gather_facts: true # false
  roles:
    - role: fastpass-additional-control-plane
  vars:
    cluster_name: "fastpass"
 # - name: Step 4 - Deploy Worker Nodes
 #   hosts: fastpass_workers
 #   become: true
 #   gather_facts: false
 #   roles:
 #     - role: fastpass-workers
@@ -1,157 +0,0 @@
 ---
 # FastPass Kubernetes Cluster Deployment
 # This playbook deploys a complete Kubernetes cluster on Fedora
 - name: 1. Preflight checks and system preparation
  hosts: fastpass
  become: true
  gather_facts: true
  roles:
    - role: kubernetes
 - name: 2. Initialize first control plane node
  hosts: fastpass_control_plane[0]
  become: true
  gather_facts: false
  roles:
    - role: fastpass-control-plane
 - name: 3. Install Calico CNI
  hosts: fastpass_control_plane[0]
  become: false
  gather_facts: false
  environment:
    KUBECONFIG: "{{ kubeconfig_path }}"
  tasks:
    - name: Check if Calico is already installed
      ansible.builtin.shell: kubectl get pods -n kube-system -l k8s-app=calico-node --no-headers | wc -l
      delegate_to: localhost
      register: calico_check
      ignore_errors: true
    - name: Display Calico check result
      ansible.builtin.debug:
        msg: "Calico pods found: {{ calico_check.stdout | trim }}"
      delegate_to: localhost
    - name: Install Calico CNI
      ansible.builtin.command: kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/{{ calico_version | default('v3.28.0') }}/manifests/calico.yaml
      delegate_to: localhost
      when: calico_check.stdout | trim == "0"
      register: calico_install_result
      ignore_errors: true
    - name: Display Calico installation result
      ansible.builtin.debug:
        msg: "Calico install stdout: {{ calico_install_result.stdout }}"
      delegate_to: localhost
      when: calico_check.stdout | trim == "0"
    - name: Wait for Calico node pods to be ready
      ansible.builtin.command: kubectl wait --for=condition=ready pod -l k8s-app=calico-node -n kube-system --timeout=300s
      delegate_to: localhost
      when: calico_check.stdout | trim == "0"
 - name: 4. Wait for first control plane to be fully ready
  hosts: fastpass_control_plane[0]
  become: false
  gather_facts: false
  environment:
    KUBECONFIG: "{{ kubeconfig_path }}"
  tasks:
    - name: Wait for API server to be ready on first node
      ansible.builtin.wait_for:
        host: "{{ inventory_hostname }}"
        port: "{{ control_plane_port | default('6443') }}"
        timeout: 300
      delegate_to: localhost
    - name: Wait for first control plane node to be ready
      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} wait --for=condition=ready node {{ inventory_hostname }} --timeout=300s
      delegate_to: localhost
    - name: Verify control plane status
      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} get nodes -o wide
      delegate_to: localhost
      register: node_status
    - name: Display cluster status
      ansible.builtin.debug:
        msg: "{{ node_status.stdout_lines }}"
 - name: 5. Join additional control plane nodes
  hosts: fastpass_control_plane[1:]
  become: true
  gather_facts: false
  roles:
    - role: fastpass-control-plane-join
 - name: 6. Wait for all control plane nodes to be ready
  hosts: fastpass_control_plane[0]
  become: false
  gather_facts: false
  environment:
    KUBECONFIG: "{{ kubeconfig_path }}"
  tasks:
    - name: Wait for all control plane nodes to be ready
      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} wait --for=condition=ready node --selector=node-role.kubernetes.io/control-plane --timeout=600s
      delegate_to: localhost
    - name: Display all control plane nodes
      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} get nodes --selector=node-role.kubernetes.io/control-plane -o wide
      delegate_to: localhost
      register: control_plane_status
    - name: Show control plane status
      ansible.builtin.debug:
        msg: "{{ control_plane_status.stdout_lines }}"
 - name: 7. Join worker nodes
  hosts: fastpass_workers
  become: true
  gather_facts: false
  roles:
    - role: fastpass-workers
 - name: 8. Final cluster validation
  hosts: fastpass_control_plane[0]
  become: false
  gather_facts: false
  environment:
    KUBECONFIG: "{{ kubeconfig_path }}"
  tasks:
    - name: Wait for all nodes to be ready
      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} wait --for=condition=ready node --all --timeout=300s
      delegate_to: localhost
    - name: Verify all pods are running
      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} get pods --all-namespaces
      delegate_to: localhost
      register: pod_status
    - name: Display final cluster status
      ansible.builtin.debug:
        msg: |
          ========================================
          FastPass Kubernetes Cluster Status
          ========================================
          {{ pod_status.stdout }}
          ========================================
    - name: Show final node status
      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} get nodes -o wide
      delegate_to: localhost
      register: final_node_status
    - name: Display final node status
      ansible.builtin.debug:
        msg: "{{ final_node_status.stdout_lines }}"
    - name: Test DNS-based control plane endpoint
      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} cluster-info
      delegate_to: localhost
      register: cluster_info
    - name: Display cluster info
      ansible.builtin.debug:
        msg: "{{ cluster_info.stdout_lines }}"