309 lines
8.9 KiB
Markdown
Executable File
309 lines
8.9 KiB
Markdown
Executable File
# FastPass Kubernetes Cluster - Complete Deployment Guide
|
|
|
|
## Overview
|
|
|
|
This guide provides step-by-step instructions for deploying a complete Kubernetes cluster on Fedora using Ansible automation. The FastPass cluster is designed for high availability with multiple control plane nodes and worker nodes.
|
|
|
|
## Prerequisites
|
|
|
|
### System Requirements
|
|
- **OS**: Fedora 38+ (64-bit)
|
|
- **Memory**: Minimum 2GB RAM per node
|
|
- **CPU**: Minimum 2 cores per node
|
|
- **Storage**: Minimum 10GB free space
|
|
- **Network**: All nodes must be able to communicate
|
|
|
|
### Software Requirements
|
|
- Ansible 2.15+
|
|
- Python 3.8+
|
|
- SSH access to all nodes with sudo privileges
|
|
|
|
## Quick Start
|
|
|
|
### 1. Validate Your Setup
|
|
|
|
Before deploying, run the validation script:
|
|
|
|
```bash
|
|
cd ansible
|
|
./scripts/test-fastpass-deployment.sh
|
|
```
|
|
|
|
This will check:
|
|
- ✅ Ansible installation
|
|
- ✅ Inventory configuration
|
|
- ✅ Playbook files
|
|
- ✅ Role files
|
|
- ✅ Group variables
|
|
- ✅ Playbook syntax
|
|
- ✅ Common issues
|
|
|
|
### 2. Deploy the Cluster
|
|
|
|
```bash
|
|
# Deploy the complete cluster
|
|
ansible-playbook -i inventory.yml playbooks/deploy-fastpass-cluster.yml
|
|
```
|
|
|
|
## Deployment Process
|
|
|
|
The deployment follows this sequence:
|
|
|
|
### Phase 1: System Preparation
|
|
- **Hosts**: All FastPass nodes
|
|
- **Tasks**:
|
|
- Preflight checks (OS, memory, CPU, disk space)
|
|
- Disable swap and configure kernel modules
|
|
- Install and configure containerd
|
|
- Install Kubernetes packages
|
|
|
|
### Phase 2: Control Plane Initialization
|
|
- **Hosts**: First control plane node (`space-mountain`)
|
|
- **Tasks**:
|
|
- Initialize the Kubernetes cluster
|
|
- Configure API server and etcd
|
|
- Install Flannel CNI
|
|
- Generate join commands
|
|
|
|
### Phase 3: Additional Control Plane Nodes
|
|
- **Hosts**: Remaining control plane nodes (`big-thunder-mountain`, `splash-mountain`)
|
|
- **Tasks**:
|
|
- Join additional control plane nodes
|
|
- Configure high availability
|
|
|
|
### Phase 4: Worker Nodes
|
|
- **Hosts**: Worker nodes (`haunted-mansion`, `peter-pans-flight`)
|
|
- **Tasks**:
|
|
- Join worker nodes to the cluster
|
|
- Apply node labels and taints
|
|
|
|
### Phase 5: Validation
|
|
- **Hosts**: First control plane node
|
|
- **Tasks**:
|
|
- Verify all nodes are ready
|
|
- Check all pods are running
|
|
- Display final cluster status
|
|
|
|
## Configuration
|
|
|
|
### Inventory Structure
|
|
|
|
Ensure your `inventory.yml` has the correct structure:
|
|
|
|
```yaml
|
|
fastpass_control_plane:
|
|
hosts:
|
|
space-mountain:
|
|
big-thunder-mountain:
|
|
splash-mountain:
|
|
|
|
fastpass_workers:
|
|
hosts:
|
|
haunted-mansion:
|
|
peter-pans-flight:
|
|
|
|
fastpass:
|
|
children:
|
|
fastpass_control_plane:
|
|
fastpass_workers:
|
|
```
|
|
|
|
### Group Variables
|
|
|
|
Key variables in `group_vars/fastpass/vars`:
|
|
|
|
```yaml
|
|
# Cluster Configuration
|
|
cluster_name: "fastpass"
|
|
kubernetes_version: "1.33"
|
|
pod_network_cidr: "10.244.0.0/16"
|
|
service_cidr: "10.96.0.0/12"
|
|
|
|
# Control Plane Configuration
|
|
control_plane_endpoint: "fastpass.local.mk-labs.cloud"
|
|
control_plane_port: "6443"
|
|
|
|
# CNI Configuration
|
|
cni_plugin: "flannel"
|
|
|
|
# Node Labels
|
|
node_labels:
|
|
zone:
|
|
# Control Plane nodes (masters) - no zone labels needed
|
|
# space-mountain: (control plane - no zone label)
|
|
# big-thunder-mountain: (control plane - no zone label)
|
|
# splash-mountain: (control plane - no zone label)
|
|
|
|
# Worker nodes
|
|
haunted-mansion: "backstage" # Internal/backstage workloads
|
|
peter-pans-flight: "backstage" # Internal/backstage workloads
|
|
# Future node: "on-stage" # External/on-stage workloads (future)
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
#### 1. Swap Enabled
|
|
**Problem**: `running with swap on is not supported`
|
|
**Solution**: The playbook automatically disables swap, but if it persists:
|
|
```bash
|
|
# On the problematic node
|
|
sudo swapoff -a
|
|
sudo systemctl restart kubelet
|
|
```
|
|
|
|
#### 2. CNI Not Ready
|
|
**Problem**: `Network plugin returns error: cni plugin not initialized`
|
|
**Solution**: The playbook installs Flannel automatically. If issues persist:
|
|
```bash
|
|
kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
|
|
```
|
|
|
|
#### 3. Control Plane Endpoint Issues
|
|
**Problem**: `unable to add a new control plane instance to a cluster that doesn't have a stable controlPlaneEndpoint`
|
|
**Solution**: The playbook uses the first control plane node as the endpoint. For production, consider using a load balancer.
|
|
|
|
#### 4. Permission Issues
|
|
**Problem**: `User "kubernetes-admin" cannot create resource "secrets"`
|
|
**Solution**: The playbook uses super-admin.conf for proper permissions.
|
|
|
|
### Debug Commands
|
|
|
|
```bash
|
|
# Check node status
|
|
kubectl get nodes -o wide
|
|
|
|
# Check pod status
|
|
kubectl get pods --all-namespaces
|
|
|
|
# Check kubelet logs
|
|
journalctl -u kubelet -f
|
|
|
|
# Check API server logs
|
|
kubectl logs -n kube-system kube-apiserver-space-mountain
|
|
|
|
# Check CNI status
|
|
kubectl get pods -n kube-flannel
|
|
```
|
|
|
|
## Post-Deployment
|
|
|
|
### 1. Verify Cluster Health
|
|
|
|
```bash
|
|
# Check all nodes are ready
|
|
kubectl get nodes -o wide
|
|
|
|
# Check all pods are running
|
|
kubectl get pods --all-namespaces
|
|
|
|
# Test cluster connectivity
|
|
kubectl cluster-info
|
|
```
|
|
|
|
### 2. Manage Kubeconfig
|
|
|
|
Use the provided script to manage different cluster configurations:
|
|
|
|
```bash
|
|
# Switch to FastPass cluster
|
|
./scripts/manage-kubeconfigs.sh fastpass
|
|
|
|
# Check status
|
|
./scripts/manage-kubeconfigs.sh status
|
|
|
|
# Test connectivity
|
|
./scripts/manage-kubeconfigs.sh test
|
|
```
|
|
|
|
### 3. Deploy Applications
|
|
|
|
Your cluster is now ready for workloads:
|
|
|
|
```bash
|
|
# Deploy a test application
|
|
kubectl run nginx --image=nginx --port=80
|
|
|
|
# Create a service
|
|
kubectl expose pod nginx --port=80 --type=NodePort
|
|
|
|
# Check the service
|
|
kubectl get svc nginx
|
|
```
|
|
|
|
## Maintenance
|
|
|
|
### Adding New Nodes
|
|
|
|
1. Add the new node to the inventory
|
|
2. Run the preparation role:
|
|
```bash
|
|
ansible-playbook -i inventory.yml playbooks/deploy_k8s.yml --limit=new-node
|
|
```
|
|
3. Join the node to the cluster manually or extend the playbook
|
|
|
|
### Upgrading Kubernetes
|
|
|
|
1. Update the `kubernetes_version` variable
|
|
2. Run the preparation role on all nodes
|
|
3. Upgrade control plane nodes first
|
|
4. Upgrade worker nodes
|
|
|
|
### Backup and Recovery
|
|
|
|
- Backup `/etc/kubernetes/` directory on control plane nodes
|
|
- Backup etcd data: `etcdctl snapshot save backup.db`
|
|
- Document cluster configuration
|
|
|
|
## Architecture
|
|
|
|
```
|
|
FastPass Cluster Architecture:
|
|
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
|
│ Control Plane │ │ Control Plane │ │ Control Plane │
|
|
│ space-mountain │ │big-thunder-mtn │ │ splash-mountain │
|
|
│ (Masters) │ │ (Masters) │ │ (Masters) │
|
|
│ No Zone Label │ │ No Zone Label │ │ No Zone Label │
|
|
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
|
│ │ │
|
|
└───────────────────────┼───────────────────────┘
|
|
│
|
|
┌───────────────────────┼───────────────────────┐
|
|
│ │ │
|
|
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
|
│ Worker Node │ │ Worker Node │ │ Worker Node │
|
|
│ haunted-mansion │ │peter-pans-flight│ │ (Future) │
|
|
│ (Backstage) │ │ (Backstage) │ │ (On-Stage) │
|
|
│ zone=backstage │ │ zone=backstage │ │ zone=on-stage │
|
|
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
|
```
|
|
|
|
## Support
|
|
|
|
For issues specific to this deployment:
|
|
1. Check the troubleshooting section
|
|
2. Review Ansible logs with `-vvv` verbosity
|
|
3. Check system logs on affected nodes
|
|
4. Verify network connectivity between nodes
|
|
|
|
## Files Structure
|
|
|
|
```
|
|
ansible/
|
|
├── group_vars/
|
|
│ └── fastpass/
|
|
│ └── vars # Cluster configuration
|
|
├── playbooks/
|
|
│ ├── deploy-fastpass-cluster.yml # Main deployment playbook
|
|
│ ├── deploy_k8s.yml # Legacy deployment playbook
|
|
│ └── roles/
|
|
│ ├── kubernetes/ # System preparation
|
|
│ ├── fastpass-control-plane/ # Control plane setup
|
|
│ ├── fastpass-control-plane-join/ # Additional control plane nodes
|
|
│ └── fastpass-workers/ # Worker node setup
|
|
├── scripts/
|
|
│ ├── test-fastpass-deployment.sh # Validation script
|
|
│ └── manage-kubeconfigs.sh # Kubeconfig management
|
|
└── inventory.yml # Host inventory
|
|
``` |