homelab/.kiro/specs/fastpass-additional-control-plane/design.md

Design Document

Overview

The fastpass-additional-control-plane Ansible role will enable the deployment of additional control plane nodes to an existing FastPass Kubernetes cluster. This role follows the established patterns from fastpass-first-control-plane but focuses on joining nodes to an already initialized cluster rather than initializing a new one. The role ensures high availability by creating redundant master nodes that can handle API requests, scheduling, and cluster management tasks.

The key difference from the first control plane role is that this role will use kubeadm join with control plane flags instead of kubeadm init, and it will need to retrieve join tokens and certificate keys from the existing cluster.

Architecture

Role Structure

The role will follow the standard Ansible role structure:

ansible/playbooks/roles/fastpass-additional-control-plane/
├── defaults/
│   └── main.yml
├── tasks/
│   └── main.yml
├── handlers/
│   └── main.yml (if needed)
└── meta/
    └── main.yml (if needed)

Integration Points

• DNS Management: Uses the existing dns-manager role for consistent DNS record creation
• Kubeconfig Management: Uses the existing kubeconfig-manager role for local kubeconfig setup
• Firewall Configuration: Reuses firewall service definitions from the first control plane role
• Cluster Integration: Coordinates with the first control plane node to obtain join credentials

Dependencies

• The first control plane node must be fully initialized and running
• The dns-manager role must be available for DNS record creation
• The kubeconfig-manager role must be available for kubeconfig setup
• Required Kubernetes prerequisites must be installed on target nodes

Components and Interfaces

Main Task Flow

1. Pre-flight Checks: Verify cluster readiness and node prerequisites
2. DNS Configuration: Set up DNS records for the new control plane node
3. Firewall Configuration: Open required ports for control plane services
4. Kubelet Configuration: Create initial kubelet configuration
5. Join Token Retrieval: Get join token and certificate key from first control plane
6. Cluster Join: Execute kubeadm join with control plane flags
7. Service Management: Ensure kubelet is enabled and running
8. Kubeconfig Setup: Configure local kubeconfig access
9. Verification: Validate successful cluster join

Key Variables

• cluster_name: Name of the Kubernetes cluster
• ip_address: IP address of the current control plane node
• first_control_plane_host: Hostname/IP of the first control plane node
• kubernetes_services_control_plane: List of firewall services to open
• join_token_ttl: TTL for join tokens (default: 24h)
• certificate_key_ttl: TTL for certificate keys (default: 2h)

External Role Interfaces

• dns-manager: Provides DNS record creation with host_name variable
• kubeconfig-manager: Handles kubeconfig merging with cluster_name variable
• First Control Plane: Source for join tokens and certificate keys

Data Models

Join Credentials Structure

join_credentials:
  token: "abcdef.1234567890abcdef"
  discovery_token_ca_cert_hash: "sha256:..."
  certificate_key: "..."
  api_server_endpoint: "cluster-name:6443"

Firewall Services

kubernetes_services_control_plane:
  - kubernetes_API      # Port 6443
  - etcd               # Ports 2379-2380
  - kubelet            # Port 10250
  - kube-scheduler     # Port 10259
  - kube-controller-manager  # Port 10257

Node Status Tracking

node_status:
  joined: false
  kubelet_running: false
  dns_configured: false
  kubeconfig_ready: false

Error Handling

Join Token Management

• Token Expiration: Automatically generate new tokens if existing ones are expired
• Certificate Key Rotation: Handle certificate key expiration gracefully
• Network Connectivity: Retry join operations with exponential backoff
• API Server Availability: Wait for API server readiness before attempting join

Idempotency Checks

• Already Joined Nodes: Skip join process if node is already part of the cluster
• Existing Configuration: Preserve existing kubelet configuration if valid
• DNS Records: Update existing DNS records instead of creating duplicates
• Service Status: Only restart services if configuration changes

Failure Recovery

• Partial Join Failures: Clean up partial configurations and retry
• Network Issues: Provide clear error messages for connectivity problems
• Permission Errors: Validate sudo/root access before attempting operations
• Resource Constraints: Check system resources before proceeding

Testing Strategy

Unit Testing Approach

• Task Validation: Test individual tasks with mock data
• Variable Validation: Ensure required variables are properly defined
• Conditional Logic: Test all conditional branches in tasks
• Error Scenarios: Validate error handling for common failure cases

Integration Testing

• Multi-Node Clusters: Test with 3 and 5 control plane node configurations
• Network Scenarios: Test across different network topologies
• OS Compatibility: Validate on supported operating systems (Ubuntu/Debian)
• Version Compatibility: Test with different Kubernetes versions

Validation Checks

• Cluster Health: Verify all control plane nodes are healthy after join
• API Availability: Confirm API server is accessible from all nodes
• Etcd Cluster: Validate etcd cluster membership and health
• Scheduling: Test pod scheduling across all control plane nodes
• Failover: Verify cluster continues operating if one control plane fails

Test Scenarios

1. Fresh Join: Join additional control plane to newly created cluster
2. Existing Cluster: Add control plane to cluster with existing workloads
3. Network Partitions: Test behavior during temporary network issues
4. Token Expiration: Handle expired join tokens gracefully
5. Retry Operations: Validate retry logic for transient failures

Implementation Considerations

Security

• Token Security: Ensure join tokens are handled securely and not logged
• Certificate Management: Properly manage and rotate certificate keys
• Network Security: Validate firewall rules don't expose unnecessary ports
• Access Control: Ensure proper RBAC is maintained after node joins

Performance

• Parallel Execution: Support joining multiple control plane nodes simultaneously
• Resource Usage: Monitor CPU and memory usage during join process
• Network Bandwidth: Optimize data transfer during cluster join
• Startup Time: Minimize time to achieve cluster readiness

Monitoring and Observability

• Join Progress: Provide clear progress indicators during join process
• Health Checks: Implement comprehensive health validation
• Logging: Ensure adequate logging for troubleshooting
• Metrics: Expose relevant metrics for monitoring cluster growth

Compatibility

• Kubernetes Versions: Support current and previous Kubernetes versions
• Operating Systems: Maintain compatibility with Ubuntu and Debian
• Container Runtimes: Work with containerd runtime configuration
• Network Plugins: Compatible with Flannel CNI configuration