homelab/ansible/FASTPASS_DEPLOYMENT_GUIDE.md

FastPass Kubernetes Cluster - Complete Deployment Guide

Overview

This guide provides step-by-step instructions for deploying a complete Kubernetes cluster on Fedora using Ansible automation. The FastPass cluster is designed for high availability with multiple control plane nodes and worker nodes.

Prerequisites

System Requirements

• OS: Fedora 38+ (64-bit)
• Memory: Minimum 2GB RAM per node
• CPU: Minimum 2 cores per node
• Storage: Minimum 10GB free space
• Network: All nodes must be able to communicate

Software Requirements

• Ansible 2.15+
• Python 3.8+
• SSH access to all nodes with sudo privileges

Quick Start

1. Validate Your Setup

Before deploying, run the validation script:

cd ansible
./scripts/test-fastpass-deployment.sh

This will check:

• ✅ Ansible installation
• ✅ Inventory configuration
• ✅ Playbook files
• ✅ Role files
• ✅ Group variables
• ✅ Playbook syntax
• ✅ Common issues

2. Deploy the Cluster

# Deploy the complete cluster
ansible-playbook -i inventory.yml playbooks/deploy-fastpass-cluster.yml

Deployment Process

The deployment follows this sequence:

Phase 1: System Preparation

• Hosts: All FastPass nodes
• Tasks:
  • Preflight checks (OS, memory, CPU, disk space)
  • Disable swap and configure kernel modules
  • Install and configure containerd
  • Install Kubernetes packages

Phase 2: Control Plane Initialization

• Hosts: First control plane node (space-mountain)
• Tasks:
  • Initialize the Kubernetes cluster
  • Configure API server and etcd
  • Install Flannel CNI
  • Generate join commands

Phase 3: Additional Control Plane Nodes

• Hosts: Remaining control plane nodes (big-thunder-mountain, splash-mountain)
• Tasks:
  • Join additional control plane nodes
  • Configure high availability

Phase 4: Worker Nodes

• Hosts: Worker nodes (haunted-mansion, peter-pans-flight)
• Tasks:
  • Join worker nodes to the cluster
  • Apply node labels and taints

Phase 5: Validation

• Hosts: First control plane node
• Tasks:
  • Verify all nodes are ready
  • Check all pods are running
  • Display final cluster status

Configuration

Inventory Structure

Ensure your inventory.yml has the correct structure:

fastpass_control_plane:
  hosts:
    space-mountain:
    big-thunder-mountain:
    splash-mountain:

fastpass_workers:
  hosts:
    haunted-mansion:
    peter-pans-flight:

fastpass:
  children:
    fastpass_control_plane:
    fastpass_workers:

Group Variables

Key variables in group_vars/fastpass/vars:

# Cluster Configuration
cluster_name: "fastpass"
kubernetes_version: "1.33"
pod_network_cidr: "10.244.0.0/16"
service_cidr: "10.96.0.0/12"

# Control Plane Configuration
control_plane_endpoint: "fastpass.local.mk-labs.cloud"
control_plane_port: "6443"

# CNI Configuration
cni_plugin: "flannel"

# Node Labels
node_labels:
  zone:
    # Control Plane nodes (masters) - no zone labels needed
    # space-mountain: (control plane - no zone label)
    # big-thunder-mountain: (control plane - no zone label)
    # splash-mountain: (control plane - no zone label)
    
    # Worker nodes
    haunted-mansion: "backstage"      # Internal/backstage workloads
    peter-pans-flight: "backstage"    # Internal/backstage workloads
    # Future node: "on-stage"         # External/on-stage workloads (future)

Troubleshooting

Common Issues

1. Swap Enabled

Problem: running with swap on is not supported Solution: The playbook automatically disables swap, but if it persists:

# On the problematic node
sudo swapoff -a
sudo systemctl restart kubelet

2. CNI Not Ready

Problem: Network plugin returns error: cni plugin not initialized Solution: The playbook installs Flannel automatically. If issues persist:

kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

3. Control Plane Endpoint Issues

Problem: unable to add a new control plane instance to a cluster that doesn't have a stable controlPlaneEndpoint Solution: The playbook uses the first control plane node as the endpoint. For production, consider using a load balancer.

4. Permission Issues

Problem: User "kubernetes-admin" cannot create resource "secrets" Solution: The playbook uses super-admin.conf for proper permissions.

Debug Commands

# Check node status
kubectl get nodes -o wide

# Check pod status
kubectl get pods --all-namespaces

# Check kubelet logs
journalctl -u kubelet -f

# Check API server logs
kubectl logs -n kube-system kube-apiserver-space-mountain

# Check CNI status
kubectl get pods -n kube-flannel

Post-Deployment

1. Verify Cluster Health

# Check all nodes are ready
kubectl get nodes -o wide

# Check all pods are running
kubectl get pods --all-namespaces

# Test cluster connectivity
kubectl cluster-info

2. Manage Kubeconfig

Use the provided script to manage different cluster configurations:

# Switch to FastPass cluster
./scripts/manage-kubeconfigs.sh fastpass

# Check status
./scripts/manage-kubeconfigs.sh status

# Test connectivity
./scripts/manage-kubeconfigs.sh test

3. Deploy Applications

Your cluster is now ready for workloads:

# Deploy a test application
kubectl run nginx --image=nginx --port=80

# Create a service
kubectl expose pod nginx --port=80 --type=NodePort

# Check the service
kubectl get svc nginx

Maintenance

Adding New Nodes

1. Add the new node to the inventory
2. Run the preparation role:

   ansible-playbook -i inventory.yml playbooks/deploy_k8s.yml --limit=new-node
   

3. Join the node to the cluster manually or extend the playbook

Upgrading Kubernetes

1. Update the kubernetes_version variable
2. Run the preparation role on all nodes
3. Upgrade control plane nodes first
4. Upgrade worker nodes

Backup and Recovery

• Backup /etc/kubernetes/ directory on control plane nodes
• Backup etcd data: etcdctl snapshot save backup.db
• Document cluster configuration

Architecture

FastPass Cluster Architecture:
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Control Plane  │    │  Control Plane  │    │  Control Plane  │
│  space-mountain │    │big-thunder-mtn  │    │ splash-mountain │
│   (Masters)     │    │   (Masters)     │    │   (Masters)     │
│  No Zone Label  │    │  No Zone Label  │    │  No Zone Label  │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 │
         ┌───────────────────────┼───────────────────────┐
         │                       │                       │
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Worker Node   │    │   Worker Node   │    │   Worker Node   │
│ haunted-mansion │    │peter-pans-flight│    │  (Future)       │
│   (Backstage)   │    │   (Backstage)   │    │   (On-Stage)    │
│ zone=backstage  │    │ zone=backstage  │    │ zone=on-stage   │
└─────────────────┘    └─────────────────┘    └─────────────────┘

Support

For issues specific to this deployment:

1. Check the troubleshooting section
2. Review Ansible logs with -vvv verbosity
3. Check system logs on affected nodes
4. Verify network connectivity between nodes

Files Structure

ansible/
├── group_vars/
│   └── fastpass/
│       └── vars                    # Cluster configuration
├── playbooks/
│   ├── deploy-fastpass-cluster.yml # Main deployment playbook
│   ├── deploy_k8s.yml             # Legacy deployment playbook
│   └── roles/
│       ├── kubernetes/             # System preparation
│       ├── fastpass-control-plane/ # Control plane setup
│       ├── fastpass-control-plane-join/ # Additional control plane nodes
│       └── fastpass-workers/       # Worker node setup
├── scripts/
│   ├── test-fastpass-deployment.sh # Validation script
│   └── manage-kubeconfigs.sh      # Kubeconfig management
└── inventory.yml                   # Host inventory