rblundon/homelab
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

sync

Browse Source
This commit is contained in:
Ryan Blundon
2025-10-19 17:02:16 -05:00
parent 3f31f77bc8
commit 702c71fcff
222 changed files with 2834 additions and 10845 deletions
Download Patch File Download Diff File Expand all files Collapse all files

165
.kiro/specs/fastpass-additional-control-plane/design.md Normal file
View File

@@ -0,0 +1,165 @@
# Design Document
## Overview
The `fastpass-additional-control-plane` Ansible role will enable the deployment of additional control plane nodes to an existing FastPass Kubernetes cluster. This role follows the established patterns from `fastpass-first-control-plane` but focuses on joining nodes to an already initialized cluster rather than initializing a new one. The role ensures high availability by creating redundant master nodes that can handle API requests, scheduling, and cluster management tasks.
The key difference from the first control plane role is that this role will use `kubeadm join` with control plane flags instead of `kubeadm init`, and it will need to retrieve join tokens and certificate keys from the existing cluster.
## Architecture
### Role Structure
The role will follow the standard Ansible role structure:
```
ansible/playbooks/roles/fastpass-additional-control-plane/
├── defaults/
│ └── main.yml
├── tasks/
│ └── main.yml
├── handlers/
│ └── main.yml (if needed)
└── meta/
└── main.yml (if needed)
```
### Integration Points
- **DNS Management**: Uses the existing `dns-manager` role for consistent DNS record creation
- **Kubeconfig Management**: Uses the existing `kubeconfig-manager` role for local kubeconfig setup
- **Firewall Configuration**: Reuses firewall service definitions from the first control plane role
- **Cluster Integration**: Coordinates with the first control plane node to obtain join credentials
### Dependencies
- The first control plane node must be fully initialized and running
- The `dns-manager` role must be available for DNS record creation
- The `kubeconfig-manager` role must be available for kubeconfig setup
- Required Kubernetes prerequisites must be installed on target nodes
## Components and Interfaces
### Main Task Flow
1. **Pre-flight Checks**: Verify cluster readiness and node prerequisites
2. **DNS Configuration**: Set up DNS records for the new control plane node
3. **Firewall Configuration**: Open required ports for control plane services
4. **Kubelet Configuration**: Create initial kubelet configuration
5. **Join Token Retrieval**: Get join token and certificate key from first control plane
6. **Cluster Join**: Execute kubeadm join with control plane flags
7. **Service Management**: Ensure kubelet is enabled and running
8. **Kubeconfig Setup**: Configure local kubeconfig access
9. **Verification**: Validate successful cluster join
### Key Variables
- `cluster_name`: Name of the Kubernetes cluster
- `ip_address`: IP address of the current control plane node
- `first_control_plane_host`: Hostname/IP of the first control plane node
- `kubernetes_services_control_plane`: List of firewall services to open
- `join_token_ttl`: TTL for join tokens (default: 24h)
- `certificate_key_ttl`: TTL for certificate keys (default: 2h)
### External Role Interfaces
- **dns-manager**: Provides DNS record creation with `host_name` variable
- **kubeconfig-manager**: Handles kubeconfig merging with `cluster_name` variable
- **First Control Plane**: Source for join tokens and certificate keys
## Data Models
### Join Credentials Structure
```yaml
join_credentials:
token: "abcdef.1234567890abcdef"
discovery_token_ca_cert_hash: "sha256:..."
certificate_key: "..."
api_server_endpoint: "cluster-name:6443"
```
### Firewall Services
```yaml
kubernetes_services_control_plane:
- kubernetes_API # Port 6443
- etcd # Ports 2379-2380
- kubelet # Port 10250
- kube-scheduler # Port 10259
- kube-controller-manager # Port 10257
```
### Node Status Tracking
```yaml
node_status:
joined: false
kubelet_running: false
dns_configured: false
kubeconfig_ready: false
```
## Error Handling
### Join Token Management
- **Token Expiration**: Automatically generate new tokens if existing ones are expired
- **Certificate Key Rotation**: Handle certificate key expiration gracefully
- **Network Connectivity**: Retry join operations with exponential backoff
- **API Server Availability**: Wait for API server readiness before attempting join
### Idempotency Checks
- **Already Joined Nodes**: Skip join process if node is already part of the cluster
- **Existing Configuration**: Preserve existing kubelet configuration if valid
- **DNS Records**: Update existing DNS records instead of creating duplicates
- **Service Status**: Only restart services if configuration changes
### Failure Recovery
- **Partial Join Failures**: Clean up partial configurations and retry
- **Network Issues**: Provide clear error messages for connectivity problems
- **Permission Errors**: Validate sudo/root access before attempting operations
- **Resource Constraints**: Check system resources before proceeding
## Testing Strategy
### Unit Testing Approach
- **Task Validation**: Test individual tasks with mock data
- **Variable Validation**: Ensure required variables are properly defined
- **Conditional Logic**: Test all conditional branches in tasks
- **Error Scenarios**: Validate error handling for common failure cases
### Integration Testing
- **Multi-Node Clusters**: Test with 3 and 5 control plane node configurations
- **Network Scenarios**: Test across different network topologies
- **OS Compatibility**: Validate on supported operating systems (Ubuntu/Debian)
- **Version Compatibility**: Test with different Kubernetes versions
### Validation Checks
- **Cluster Health**: Verify all control plane nodes are healthy after join
- **API Availability**: Confirm API server is accessible from all nodes
- **Etcd Cluster**: Validate etcd cluster membership and health
- **Scheduling**: Test pod scheduling across all control plane nodes
- **Failover**: Verify cluster continues operating if one control plane fails
### Test Scenarios
1. **Fresh Join**: Join additional control plane to newly created cluster
2. **Existing Cluster**: Add control plane to cluster with existing workloads
3. **Network Partitions**: Test behavior during temporary network issues
4. **Token Expiration**: Handle expired join tokens gracefully
5. **Retry Operations**: Validate retry logic for transient failures
## Implementation Considerations
### Security
- **Token Security**: Ensure join tokens are handled securely and not logged
- **Certificate Management**: Properly manage and rotate certificate keys
- **Network Security**: Validate firewall rules don't expose unnecessary ports
- **Access Control**: Ensure proper RBAC is maintained after node joins
### Performance
- **Parallel Execution**: Support joining multiple control plane nodes simultaneously
- **Resource Usage**: Monitor CPU and memory usage during join process
- **Network Bandwidth**: Optimize data transfer during cluster join
- **Startup Time**: Minimize time to achieve cluster readiness
### Monitoring and Observability
- **Join Progress**: Provide clear progress indicators during join process
- **Health Checks**: Implement comprehensive health validation
- **Logging**: Ensure adequate logging for troubleshooting
- **Metrics**: Expose relevant metrics for monitoring cluster growth
### Compatibility
- **Kubernetes Versions**: Support current and previous Kubernetes versions
- **Operating Systems**: Maintain compatibility with Ubuntu and Debian
- **Container Runtimes**: Work with containerd runtime configuration
- **Network Plugins**: Compatible with Flannel CNI configuration

77
.kiro/specs/fastpass-additional-control-plane/requirements.md Normal file
View File

@@ -0,0 +1,77 @@
# Requirements Document
## Introduction
This feature involves creating an Ansible role called `fastpass-additional-control-plane` that will deploy additional control plane nodes to an existing FastPass Kubernetes cluster. The role will follow the same pattern as the existing `fastpass-first-control-plane` role but will focus on joining nodes to an already initialized cluster rather than initializing a new cluster. This ensures high availability for the Kubernetes control plane by adding redundant master nodes.
## Requirements
### Requirement 1
**User Story:** As a DevOps engineer, I want to deploy additional control plane nodes to my FastPass Kubernetes cluster, so that I can achieve high availability and fault tolerance for the cluster control plane.
#### Acceptance Criteria
1. WHEN the role is executed on a node THEN the system SHALL join the node to the existing Kubernetes cluster as a control plane node
2. WHEN the role runs THEN the system SHALL configure the necessary firewall rules for control plane services
3. WHEN the role executes THEN the system SHALL ensure the kubelet service is properly configured and running
4. WHEN joining the cluster THEN the system SHALL use the correct join token and certificate key from the first control plane node
5. WHEN the role completes THEN the system SHALL verify the node has successfully joined as a control plane node
### Requirement 2
**User Story:** As a system administrator, I want the additional control plane role to follow the same patterns as the first control plane role, so that the codebase remains consistent and maintainable.
#### Acceptance Criteria
1. WHEN the role is created THEN the system SHALL follow the same directory structure as fastpass-first-control-plane
2. WHEN the role is implemented THEN the system SHALL use similar variable naming conventions and task organization
3. WHEN the role runs THEN the system SHALL include proper error handling and idempotency checks
4. WHEN the role executes THEN the system SHALL use the same firewall service definitions as the first control plane role
5. WHEN the role is documented THEN the system SHALL include proper metadata headers with author, version, and description
### Requirement 3
**User Story:** As a cluster operator, I want the additional control plane nodes to have proper DNS configuration, so that they can be reached by their cluster names and participate in load balancing.
#### Acceptance Criteria
1. WHEN the role runs THEN the system SHALL configure DNS records for the additional control plane nodes
2. WHEN DNS is configured THEN the system SHALL use the dns-manager role for consistency
3. WHEN the role executes THEN the system SHALL ensure the node can resolve the cluster endpoint
4. WHEN DNS setup completes THEN the system SHALL verify connectivity to the cluster API endpoint
### Requirement 4
**User Story:** As a DevOps engineer, I want the role to handle kubeconfig management for additional control plane nodes, so that I can manage the cluster from any control plane node.
#### Acceptance Criteria
1. WHEN the role completes THEN the system SHALL configure kubeconfig for the new control plane node
2. WHEN kubeconfig is set up THEN the system SHALL use the kubeconfig-manager role for consistency
3. WHEN the role runs THEN the system SHALL ensure proper permissions are set on kubeconfig files
4. WHEN kubeconfig is configured THEN the system SHALL verify kubectl access works from the new node
### Requirement 5
**User Story:** As a system administrator, I want the role to be idempotent and handle edge cases, so that I can run it multiple times safely without causing issues.
#### Acceptance Criteria
1. WHEN the role is run multiple times THEN the system SHALL not attempt to rejoin an already joined node
2. WHEN a node is already part of the cluster THEN the system SHALL skip the join process gracefully
3. WHEN the role encounters errors THEN the system SHALL provide clear error messages and fail gracefully
4. WHEN prerequisites are missing THEN the system SHALL report what needs to be configured first
5. WHEN the role runs THEN the system SHALL validate that required variables are defined
### Requirement 6
**User Story:** As a cluster administrator, I want the role to integrate seamlessly with the existing FastPass deployment workflow, so that it can be used in the 4-step deployment process.
#### Acceptance Criteria
1. WHEN the role is created THEN the system SHALL be compatible with the deploy-fastpass-4step.yml playbook
2. WHEN the role runs THEN the system SHALL work with the fastpass_control_plane[1:] host group
3. WHEN integrated THEN the system SHALL not interfere with the first control plane initialization
4. WHEN the role executes THEN the system SHALL depend on the first control plane node being ready
5. WHEN deployment completes THEN the system SHALL allow worker nodes to join the cluster successfully

118
.kiro/specs/fastpass-additional-control-plane/tasks.md Normal file
View File

@@ -0,0 +1,118 @@
# Implementation Plan
- [ ] 1. Create role directory structure and basic configuration
- Create the fastpass-additional-control-plane role directory structure
- Set up defaults/main.yml with required variables and firewall services
- Create meta/main.yml with role metadata and dependencies
- _Requirements: 2.1, 2.2, 2.4_
- [ ] 2. Implement join token and certificate key retrieval
- [ ] 2.1 Create tasks to generate new join tokens from first control plane
- Write Ansible tasks to execute kubeadm token create on first control plane node
- Implement token validation and expiration checking
- Add error handling for token generation failures
- _Requirements: 1.4, 5.4_
- [ ] 2.2 Implement certificate key retrieval and management
- Create tasks to upload and retrieve certificate keys from first control plane
- Add certificate key expiration handling and rotation
- Implement secure handling of certificate keys in variables
- _Requirements: 1.4, 5.1_
- [ ] 2.3 Create discovery token CA certificate hash retrieval
- Write tasks to extract CA certificate hash from first control plane
- Implement validation of certificate hash format
- Add error handling for certificate retrieval failures
- _Requirements: 1.4, 5.4_
- [ ] 3. Implement DNS configuration and firewall setup
- [ ] 3.1 Configure DNS records for additional control plane nodes
- Integrate dns-manager role for consistent DNS record creation
- Pass appropriate host_name variable to dns-manager
- Add DNS propagation wait and validation
- _Requirements: 3.1, 3.2, 3.4_
- [ ] 3.2 Set up firewall rules for control plane services
- Reuse kubernetes_services_control_plane from defaults
- Implement UFW firewall rule creation for Debian/Ubuntu systems
- Add conditional logic for different operating systems
- _Requirements: 1.2, 2.4_
- [ ] 4. Implement kubelet configuration and cluster join
- [ ] 4.1 Create initial kubelet configuration
- Write kubelet config.yaml with systemd cgroup driver
- Set containerd socket endpoint configuration
- Ensure proper file permissions and ownership
- _Requirements: 1.3, 2.3_
- [ ] 4.2 Execute kubeadm join for control plane
- Implement kubeadm join command with control-plane flag
- Use retrieved join token, certificate key, and CA cert hash
- Add proper command argument construction and validation
- Include idempotency checks to prevent duplicate joins
- _Requirements: 1.1, 1.4, 5.1, 5.2_
- [ ] 4.3 Ensure kubelet service management
- Enable and start kubelet systemd service
- Add service status validation and error handling
- Implement service restart logic if needed
- _Requirements: 1.3, 1.5_
- [ ] 5. Implement kubeconfig management and validation
- [ ] 5.1 Configure kubeconfig for additional control plane nodes
- Integrate kubeconfig-manager role for consistent configuration
- Pass cluster_name variable to kubeconfig-manager
- Ensure proper kubeconfig merging with existing configurations
- _Requirements: 4.1, 4.2, 4.3_
- [ ] 5.2 Implement cluster join validation
- Create tasks to verify node successfully joined as control plane
- Add kubectl commands to check node status and roles
- Implement cluster health validation checks
- _Requirements: 1.5, 4.4_
- [ ] 6. Add comprehensive error handling and idempotency
- [ ] 6.1 Implement pre-flight validation checks
- Check if node is already joined to cluster
- Validate required variables are defined
- Verify first control plane node accessibility
- Add system resource and prerequisite checks
- _Requirements: 5.1, 5.2, 5.4, 5.5_
- [ ] 6.2 Add retry logic and failure recovery
- Implement retry mechanisms for transient failures
- Add exponential backoff for network-related operations
- Create cleanup tasks for partial join failures
- _Requirements: 5.3, 5.4_
- [ ] 7. Integration with FastPass deployment workflow
- [ ] 7.1 Ensure compatibility with deploy-fastpass-4step.yml
- Verify role works with fastpass_control_plane[1:] host group
- Test integration with existing playbook structure
- Validate dependency on first control plane completion
- _Requirements: 6.1, 6.2, 6.3, 6.4_
- [ ] 7.2 Add proper task documentation and metadata
- Include role header with author, version, and description
- Add inline comments for complex task logic
- Document required variables and their purposes
- _Requirements: 2.2, 2.5_
- [ ]* 8. Create comprehensive testing and validation
- [ ]* 8.1 Write unit tests for individual tasks
- Create test cases for token retrieval logic
- Test kubeadm join command construction
- Validate error handling scenarios
- _Requirements: 1.1, 1.4, 5.1_
- [ ]* 8.2 Implement integration tests
- Test multi-node control plane deployment
- Validate cluster health after additional nodes join
- Test failover scenarios and cluster resilience
- _Requirements: 1.5, 6.5_
- [ ]* 8.3 Add validation scripts and health checks
- Create scripts to verify cluster state after deployment
- Implement automated health validation
- Add performance and resource usage monitoring
- _Requirements: 1.5, 4.4_