sync current state

2025-09-15 22:27:42 -05:00
parent b7e090ed7c
commit 405aae5209
41 changed files with 2360 additions and 198 deletions
--- a/ansible/FASTPASS_DEPLOYMENT_GUIDE.md
+++ b/ansible/FASTPASS_DEPLOYMENT_GUIDE.md
@@ -0,0 +1,309 @@
+# FastPass Kubernetes Cluster - Complete Deployment Guide
+
+## Overview
+
+This guide provides step-by-step instructions for deploying a complete Kubernetes cluster on Fedora using Ansible automation. The FastPass cluster is designed for high availability with multiple control plane nodes and worker nodes.
+
+## Prerequisites
+
+### System Requirements
+- **OS**: Fedora 38+ (64-bit)
+- **Memory**: Minimum 2GB RAM per node
+- **CPU**: Minimum 2 cores per node
+- **Storage**: Minimum 10GB free space
+- **Network**: All nodes must be able to communicate
+
+### Software Requirements
+- Ansible 2.15+
+- Python 3.8+
+- SSH access to all nodes with sudo privileges
+
+## Quick Start
+
+### 1. Validate Your Setup
+
+Before deploying, run the validation script:
+
+```bash
+cd ansible
+./scripts/test-fastpass-deployment.sh
+```
+
+This will check:
+- ✅ Ansible installation
+- ✅ Inventory configuration
+- ✅ Playbook files
+- ✅ Role files
+- ✅ Group variables
+- ✅ Playbook syntax
+- ✅ Common issues
+
+### 2. Deploy the Cluster
+
+```bash
+# Deploy the complete cluster
+ansible-playbook -i inventory.yml playbooks/deploy-fastpass-cluster.yml
+```
+
+## Deployment Process
+
+The deployment follows this sequence:
+
+### Phase 1: System Preparation
+- **Hosts**: All FastPass nodes
+- **Tasks**: 
+  - Preflight checks (OS, memory, CPU, disk space)
+  - Disable swap and configure kernel modules
+  - Install and configure containerd
+  - Install Kubernetes packages
+
+### Phase 2: Control Plane Initialization
+- **Hosts**: First control plane node (`space-mountain`)
+- **Tasks**:
+  - Initialize the Kubernetes cluster
+  - Configure API server and etcd
+  - Install Flannel CNI
+  - Generate join commands
+
+### Phase 3: Additional Control Plane Nodes
+- **Hosts**: Remaining control plane nodes (`big-thunder-mountain`, `splash-mountain`)
+- **Tasks**:
+  - Join additional control plane nodes
+  - Configure high availability
+
+### Phase 4: Worker Nodes
+- **Hosts**: Worker nodes (`haunted-mansion`, `peter-pans-flight`)
+- **Tasks**:
+  - Join worker nodes to the cluster
+  - Apply node labels and taints
+
+### Phase 5: Validation
+- **Hosts**: First control plane node
+- **Tasks**:
+  - Verify all nodes are ready
+  - Check all pods are running
+  - Display final cluster status
+
+## Configuration
+
+### Inventory Structure
+
+Ensure your `inventory.yml` has the correct structure:
+
+```yaml
+fastpass_control_plane:
+  hosts:
+    space-mountain:
+    big-thunder-mountain:
+    splash-mountain:
+
+fastpass_workers:
+  hosts:
+    haunted-mansion:
+    peter-pans-flight:
+
+fastpass:
+  children:
+    fastpass_control_plane:
+    fastpass_workers:
+```
+
+### Group Variables
+
+Key variables in `group_vars/fastpass/vars`:
+
+```yaml
+# Cluster Configuration
+cluster_name: "fastpass"
+kubernetes_version: "1.33"
+pod_network_cidr: "10.244.0.0/16"
+service_cidr: "10.96.0.0/12"
+
+# Control Plane Configuration
+control_plane_endpoint: "fastpass.local.mk-labs.cloud"
+control_plane_port: "6443"
+
+# CNI Configuration
+cni_plugin: "flannel"
+
+# Node Labels
+node_labels:
+  zone:
+    # Control Plane nodes (masters) - no zone labels needed
+    # space-mountain: (control plane - no zone label)
+    # big-thunder-mountain: (control plane - no zone label)
+    # splash-mountain: (control plane - no zone label)
+    
+    # Worker nodes
+    haunted-mansion: "backstage"      # Internal/backstage workloads
+    peter-pans-flight: "backstage"    # Internal/backstage workloads
+    # Future node: "on-stage"         # External/on-stage workloads (future)
+```
+
+## Troubleshooting
+
+### Common Issues
+
+#### 1. Swap Enabled
+**Problem**: `running with swap on is not supported`
+**Solution**: The playbook automatically disables swap, but if it persists:
+```bash
+# On the problematic node
+sudo swapoff -a
+sudo systemctl restart kubelet
+```
+
+#### 2. CNI Not Ready
+**Problem**: `Network plugin returns error: cni plugin not initialized`
+**Solution**: The playbook installs Flannel automatically. If issues persist:
+```bash
+kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
+```
+
+#### 3. Control Plane Endpoint Issues
+**Problem**: `unable to add a new control plane instance to a cluster that doesn't have a stable controlPlaneEndpoint`
+**Solution**: The playbook uses the first control plane node as the endpoint. For production, consider using a load balancer.
+
+#### 4. Permission Issues
+**Problem**: `User "kubernetes-admin" cannot create resource "secrets"`
+**Solution**: The playbook uses super-admin.conf for proper permissions.
+
+### Debug Commands
+
+```bash
+# Check node status
+kubectl get nodes -o wide
+
+# Check pod status
+kubectl get pods --all-namespaces
+
+# Check kubelet logs
+journalctl -u kubelet -f
+
+# Check API server logs
+kubectl logs -n kube-system kube-apiserver-space-mountain
+
+# Check CNI status
+kubectl get pods -n kube-flannel
+```
+
+## Post-Deployment
+
+### 1. Verify Cluster Health
+
+```bash
+# Check all nodes are ready
+kubectl get nodes -o wide
+
+# Check all pods are running
+kubectl get pods --all-namespaces
+
+# Test cluster connectivity
+kubectl cluster-info
+```
+
+### 2. Manage Kubeconfig
+
+Use the provided script to manage different cluster configurations:
+
+```bash
+# Switch to FastPass cluster
+./scripts/manage-kubeconfigs.sh fastpass
+
+# Check status
+./scripts/manage-kubeconfigs.sh status
+
+# Test connectivity
+./scripts/manage-kubeconfigs.sh test
+```
+
+### 3. Deploy Applications
+
+Your cluster is now ready for workloads:
+
+```bash
+# Deploy a test application
+kubectl run nginx --image=nginx --port=80
+
+# Create a service
+kubectl expose pod nginx --port=80 --type=NodePort
+
+# Check the service
+kubectl get svc nginx
+```
+
+## Maintenance
+
+### Adding New Nodes
+
+1. Add the new node to the inventory
+2. Run the preparation role:
+   ```bash
+   ansible-playbook -i inventory.yml playbooks/deploy_k8s.yml --limit=new-node
+   ```
+3. Join the node to the cluster manually or extend the playbook
+
+### Upgrading Kubernetes
+
+1. Update the `kubernetes_version` variable
+2. Run the preparation role on all nodes
+3. Upgrade control plane nodes first
+4. Upgrade worker nodes
+
+### Backup and Recovery
+
+- Backup `/etc/kubernetes/` directory on control plane nodes
+- Backup etcd data: `etcdctl snapshot save backup.db`
+- Document cluster configuration
+
+## Architecture
+
+```
+FastPass Cluster Architecture:
+┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
+│  Control Plane  │    │  Control Plane  │    │  Control Plane  │
+│  space-mountain │    │big-thunder-mtn  │    │ splash-mountain │
+│   (Masters)     │    │   (Masters)     │    │   (Masters)     │
+│  No Zone Label  │    │  No Zone Label  │    │  No Zone Label  │
+└─────────────────┘    └─────────────────┘    └─────────────────┘
+         │                       │                       │
+         └───────────────────────┼───────────────────────┘
+                                 │
+         ┌───────────────────────┼───────────────────────┐
+         │                       │                       │
+┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
+│   Worker Node   │    │   Worker Node   │    │   Worker Node   │
+│ haunted-mansion │    │peter-pans-flight│    │  (Future)       │
+│   (Backstage)   │    │   (Backstage)   │    │   (On-Stage)    │
+│ zone=backstage  │    │ zone=backstage  │    │ zone=on-stage   │
+└─────────────────┘    └─────────────────┘    └─────────────────┘
+```
+
+## Support
+
+For issues specific to this deployment:
+1. Check the troubleshooting section
+2. Review Ansible logs with `-vvv` verbosity
+3. Check system logs on affected nodes
+4. Verify network connectivity between nodes
+
+## Files Structure
+
+```
+ansible/
+├── group_vars/
+│   └── fastpass/
+│       └── vars                    # Cluster configuration
+├── playbooks/
+│   ├── deploy-fastpass-cluster.yml # Main deployment playbook
+│   ├── deploy_k8s.yml             # Legacy deployment playbook
+│   └── roles/
+│       ├── kubernetes/             # System preparation
+│       ├── fastpass-control-plane/ # Control plane setup
+│       ├── fastpass-control-plane-join/ # Additional control plane nodes
+│       └── fastpass-workers/       # Worker node setup
+├── scripts/
+│   ├── test-fastpass-deployment.sh # Validation script
+│   └── manage-kubeconfigs.sh      # Kubeconfig management
+└── inventory.yml                   # Host inventory
+``` 
--- a/ansible/FASTPASS_README.md
+++ b/ansible/FASTPASS_README.md
@@ -0,0 +1,228 @@
+# FastPass Kubernetes Cluster - Fedora Deployment Guide
+
+## Overview
+
+This guide covers the deployment of a Kubernetes cluster on Fedora using Ansible automation. The FastPass cluster is designed for high availability with multiple control plane nodes and worker nodes.
+
+## Architecture
+
+```
+FastPass Cluster Architecture:
+┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
+│  Control Plane  │    │  Control Plane  │    │  Control Plane  │
+│  space-mountain │    │big-thunder-mtn  │    │ splash-mountain │
+│   (Leader)      │    │                 │    │                 │
+└─────────────────┘    └─────────────────┘    └─────────────────┘
+         │                       │                       │
+         └───────────────────────┼───────────────────────┘
+                                 │
+         ┌───────────────────────┼───────────────────────┐
+         │                       │                       │
+┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
+│   Worker Node   │    │   Worker Node   │    │   Worker Node   │
+│ haunted-mansion │    │peter-pans-flight│    │  (Future)       │
+│   (backstage)   │    │   (backstage)   │    │                 │
+└─────────────────┘    └─────────────────┘    └─────────────────┘
+```
+
+## Prerequisites
+
+### System Requirements
+- **OS**: Fedora 38+ (64-bit)
+- **Memory**: Minimum 2GB RAM per node
+- **CPU**: Minimum 2 cores per node
+- **Storage**: Minimum 10GB free space
+- **Network**: All nodes must be able to communicate
+
+### Software Requirements
+- Ansible 2.15+
+- Python 3.8+
+- SSH access to all nodes with sudo privileges
+
+## Quick Start
+
+### 1. Prepare Inventory
+
+Ensure your `inventory.yml` has the correct host groups:
+
+```yaml
+fastpass_control_plane:
+  hosts:
+    space-mountain:
+    big-thunder-mountain:
+    splash-mountain:
+
+fastpass_workers:
+  hosts:
+    haunted-mansion:
+    peter-pans-flight:
+
+fastpass:
+  children:
+    fastpass_control_plane:
+    fastpass_workers:
+```
+
+### 2. Configure Variables
+
+Edit `group_vars/fastpass/vars` to customize your deployment:
+
+```yaml
+# Cluster Configuration
+cluster_name: "fastpass"
+kubernetes_version: "1.33"
+pod_network_cidr: "10.244.0.0/16"
+service_cidr: "10.96.0.0/12"
+
+# Node Labels
+node_labels:
+  zone:
+    space-mountain: "external"
+    big-thunder-mountain: "external"
+    splash-mountain: "external"
+    haunted-mansion: "internal"
+    peter-pans-flight: "internal"
+```
+
+### 3. Deploy the Cluster
+
+```bash
+# Deploy the complete cluster
+ansible-playbook -i inventory.yml playbooks/deploy-fastpass-cluster.yml
+
+# Or deploy step by step
+ansible-playbook -i inventory.yml playbooks/deploy_k8s.yml
+```
+
+## Deployment Process
+
+### Phase 1: System Preparation
+- Preflight checks (OS, memory, CPU, disk space)
+- Disable swap and configure kernel modules
+- Install and configure containerd
+- Install Kubernetes packages
+
+### Phase 2: Control Plane Setup
+- Initialize the first control plane node
+- Configure API server and etcd
+- Install Calico CNI
+- Join additional control plane nodes
+
+### Phase 3: Worker Node Setup
+- Join worker nodes to the cluster
+- Apply node labels and taints
+- Verify cluster health
+
+## Troubleshooting
+
+### Common Issues
+
+#### 1. Repository Errors
+**Problem**: Docker CE repository not found
+**Solution**: Update the repository URL in `container-runtime.yml`:
+```yaml
+baseurl: "https://download.docker.com/linux/fedora/{{ ansible_distribution_major_version }}/x86_64/stable"
+```
+
+#### 2. kubeadm Init Failures
+**Problem**: kubeadm init fails with preflight errors
+**Solution**: The playbook includes `--ignore-preflight-errors=all` flag
+
+#### 3. Node Not Ready
+**Problem**: Worker nodes stuck in NotReady state
+**Solution**: Check CNI installation and network connectivity
+
+### Debug Commands
+
+```bash
+# Check node status
+kubectl get nodes -o wide
+
+# Check pod status
+kubectl get pods --all-namespaces
+
+# Check kubelet logs
+journalctl -u kubelet -f
+
+# Check containerd status
+systemctl status containerd
+```
+
+## Customization
+
+### Network Configuration
+Modify the pod and service CIDRs in `group_vars/fastpass/vars`:
+
+```yaml
+pod_network_cidr: "10.244.0.0/16"
+service_cidr: "10.96.0.0/12"
+```
+
+### CNI Selection
+The default CNI is Calico. To change, modify the control plane role:
+
+```yaml
+cni_plugin: "flannel"  # or "weave", "cilium"
+```
+
+### Node Labels and Taints
+Configure node labels in `group_vars/fastpass/vars`:
+
+```yaml
+node_labels:
+  zone:
+    space-mountain: "external"
+    haunted-mansion: "internal"
+  environment: "production"
+  cluster: "fastpass"
+```
+
+## Maintenance
+
+### Adding New Nodes
+1. Add the new node to the inventory
+2. Run the preparation role: `ansible-playbook -i inventory.yml playbooks/deploy_k8s.yml --limit=new-node`
+3. Join the node to the cluster
+
+### Upgrading Kubernetes
+1. Update the `kubernetes_version` variable
+2. Run the preparation role on all nodes
+3. Upgrade control plane nodes first
+4. Upgrade worker nodes
+
+### Backup and Recovery
+- Backup `/etc/kubernetes/` directory on control plane nodes
+- Backup etcd data: `etcdctl snapshot save backup.db`
+- Document cluster configuration
+
+## Security Considerations
+
+- SELinux is set to permissive mode for compatibility
+- Firewalld is disabled (configure as needed for your environment)
+- Consider enabling RBAC and network policies
+- Regularly update Kubernetes and system packages
+
+## Support
+
+For issues specific to this deployment:
+1. Check the troubleshooting section
+2. Review Ansible logs with `-vvv` verbosity
+3. Check system logs on affected nodes
+4. Verify network connectivity between nodes
+
+## Files Structure
+
+```
+ansible/
+├── group_vars/
+│   └── fastpass/
+│       └── vars                    # Cluster configuration
+├── playbooks/
+│   ├── deploy-fastpass-cluster.yml # Main deployment playbook
+│   ├── deploy_k8s.yml             # Legacy deployment playbook
+│   └── roles/
+│       ├── kubernetes/             # System preparation
+│       ├── fastpass-control-plane/ # Control plane setup
+│       └── fastpass-workers/       # Worker node setup
+└── inventory.yml                   # Host inventory
+``` 
--- a/ansible/group_vars/fastpass/vars
+++ b/ansible/group_vars/fastpass/vars
@@ -0,0 +1,66 @@
+---
+# FastPass Kubernetes Cluster Variables
+
+# Cluster Configuration
+cluster_name: "fastpass"
+kubernetes_version: "1.33.4"
+pod_network_cidr: "10.244.0.0/16"
+service_cidr: "10.96.0.0/12"
+
+# Control Plane Configuration
+control_plane_endpoint: "fastpass.local.mk-labs.cloud"
+control_plane_port: "6443"
+
+# CNI Configuration
+cni_plugin: "calico"
+calico_version: "v3.28.0"
+
+# Node Labels and Taints
+node_labels:
+  zone:
+    # Control Plane nodes (masters) - no zone labels needed
+    # space-mountain: (control plane - no zone label)
+    # big-thunder-mountain: (control plane - no zone label)
+    # splash-mountain: (control plane - no zone label)
+    
+    # Worker nodes
+    haunted-mansion: "backstage"      # Internal/backstage workloads
+    peter-pans-flight: "backstage"    # Internal/backstage workloads
+    # Future node: "on-stage"         # External/on-stage workloads (future)
+  
+  environment: "production"
+  cluster: "fastpass"
+
+# System Configuration
+selinux_mode: "permissive"
+firewall_enabled: false
+
+# Container Runtime
+container_runtime: "containerd"
+containerd_version: "latest"
+
+# Repository Configuration
+kubernetes_repo_url: "https://pkgs.k8s.io/core:/stable:/v1.33/rpm/"
+containerd_repo_url: "https://download.docker.com/linux/fedora/docker-ce.repo"
+
+# Validation
+preflight_checks: true
+validate_cluster: true
+
+# Kubeconfig Configuration
+kubeconfig_path: "{{ lookup('env', 'HOME') }}/.kube/config"
+
+# Hostname Configuration for Kubernetes
+# Override hostnames to remove dots for Kubernetes compatibility
+kubernetes_hostnames:
+  space-mountain: "space-mountain"
+  big-thunder-mountain: "big-thunder-mountain"
+  splash-mountain: "splash-mountain"
+  haunted-mansion: "haunted-mansion"
+  peter-pans-flight: "peter-pans-flight"
+
+# Generic Kubernetes Prerequisites Configuration
+configure_firewall: true
+configure_ntp: true
+selinux_state: "permissive"
+disable_swap: true 
--- a/ansible/group_vars/ntp_servers/vars
+++ b/ansible/group_vars/ntp_servers/vars
@@ -0,0 +1,18 @@
+---
+# file: group_vars/ntp_servers/vars
+#Ansible vars template for NTP servers.
+
+ntp_servers:
+  - 0.us.pool.ntp.org
+  - 1.us.pool.ntp.org
+
+allowed_networks:
+  - "192.168.1.0/24"
+  - "192.168.2.0/24"
+  - "192.168.3.0/24"
+  - "192.168.5.0/24"
+  - "192.168.9.0/24"
+  - "192.168.10.0/24"
+  - "192.168.250.0/24"
+  - "10.1.71.0/24"
+  - "10.1.82.0/24"
--- a/ansible/group_vars/prometheus_server/vars
+++ b/ansible/group_vars/prometheus_server/vars
@@ -0,0 +1,34 @@
+# file: group_vars/prometheus_server/vars
+# Ansible vars template for Prometheus nodes created via Ansible\.
+
+# Version of Prometheus to install.  All available versions can be found at: https://prometheus.io/download/
+
+#prometheus_version: "3.5.0"
+#alertmanager_version: "0.28.1"
+
+# prometheus_nodes:
+# #  - '192.168.1.207'  # Worker node IP
+#   - cinderella-castle  # Master node IP 
+#   - 10.1.71.21
+#   - cinderella-castle.local.mk-labs.cloud
+
+# Static Variables
+# prometheus_installer_file: "prometheus-{{ prometheus_version }}.linux-amd64.tar.gz"
+# prometheus_installer_download_url: "https://github.com/prometheus/prometheus/releases/download/v{{ prometheus_version }}/{{ prometheus_installer_file }}"
+
+# Grafana Variables
+
+#grafana_port: "3000"
+#grafana_url: "{{ inventory_hostname }}.{{ base_domain }}:3000"
+
+# prometheus_targets:
+#   node:
+#   - targets:
+#     - localhost:9100
+#     labels:
+#       env: mk-labs
+
+grafana_ini:
+  security:
+    admin_user: admin
+    admin_password: admin
--- a/ansible/host_vars/cinderella-castle/vars
+++ b/ansible/host_vars/cinderella-castle/vars
@@ -0,0 +1,49 @@
+---
+# file: host_vars/vm_template/vars
+# Ansible vars template for hosts created via cloning.
+
+# Supported hypervisors:
+# - Proxmox
+
+platform: "proxmox"
+
+# This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
+
+proxmox_clone_node: "fantasyland"
+
+# Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
+# Current OS offerings are:
+# - Fedora (42)
+# Current VM sizes are:
+# - Small: 2 cores, 2GB memory, 8 GiB virtual disk
+# - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
+# - Large: 4 cores, 4GB memory, 32 GiB virtual disk
+# - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
+# - Xlarge Plus: 4 cores, 8GB memory, 128 GiB virtual disk
+
+vm_clone_source: "fedora-42-xlarge-plus"
+
+# Proxmox storage target.
+
+vm_storage: "general"
+
+# Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
+# and backup groups. (ha_group will be factored out in the next functionality update.)
+
+ha_group: "fantasyland"
+proxmox_host_target: "fantasyland"
+
+# Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
+# sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
+
+ip_address: 10.1.71.21
+
+# Software
+# Future enhancement will allow specification of additional software to automatically deploy to
+# the VM after creation.
+
+#terraform_version: "1.11.3"
+
+# ---
+
+hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn
--- a/ansible/host_vars/sundial/vars
+++ b/ansible/host_vars/sundial/vars
@@ -0,0 +1,50 @@
+---
+# file: host_vars/vm_template/vars
+# Ansible vars template for hosts created via cloning.
+
+# Supported hypervisors:
+# - Proxmox
+
+platform: "proxmox"
+
+# This is the Proxmox node where all the VM templates are stored. (Templates are not global.)
+
+proxmox_clone_node: "fantasyland"
+
+# Templates are named in the following format (all lower case): <OS Distribution>-<OS Version>-<VM Size>
+# Current OS offerings are:
+# - Fedora (42)
+# Current VM sizes are:
+# - Small: 2 cores, 2GB memory, 8 GiB virtual disk
+# - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
+# - Large: 4 cores, 4GB memory, 32 GiB virtual disk
+# - Large Plus: 4 cores, 4GB memory, 48 GiB virtual disk
+# - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
+# - Xlarge Plus: 4 cores, 8GB memory, 128 GiB virtual disk
+
+vm_clone_source: "fedora-42-small"
+
+# Proxmox storage target.
+
+vm_storage: "general"
+
+# Proxmox does not yet do dynamic load balancing, the host target sets the target for HA groups
+# and backup groups. (ha_group will be factored out in the next functionality update.)
+
+ha_group: "fantasyland"
+proxmox_host_target: "fantasyland"
+
+# Currently, only single NIC VMs using IPv4 are supported via cloning.  The IP address also 
+# sets the Proxmox VMID.  The VMID is a combination of the 3rd and 4th octet of the IPv4 address.
+
+ip_address: 10.1.71.21
+
+# Software
+# Future enhancement will allow specification of additional software to automatically deploy to
+# the VM after creation.
+
+#terraform_version: "1.11.3"
+
+# ---
+
+hostname: "{{ inventory_hostname }}.{{ base_domain }}" # Change variable to fqdn
--- a/ansible/host_vars/vm_template/vars
+++ b/ansible/host_vars/vm_template/vars
@@ -18,7 +18,9 @@ proxmox_clone_node: "pve03"
 # - Small: 2 cores, 2GB memory, 8 GiB virtual disk
 # - Medium: 2 cores, 4GB memory, 16 GiB virtual disk
 # - Large: 4 cores, 4GB memory, 32 GiB virtual disk
+# - Large Plus: 4 cores, 4GB memory, 48 GiB virtual disk
 # - Xlarge: 4 cores, 8GB memory, 64 GiB virtual disk
+# - Xlarge Plus: 4 cores, 8GB memory, 128 GiB virtual disk

 vm_clone_source: "fedora-42-large"

--- a/ansible/inventory.yml
+++ b/ansible/inventory.yml
@@ -3,45 +3,52 @@ proxmox:
  hosts:
    pve0[1:3]:

-magic-kingdom:
+magic_kingdom:
  hosts:
    main-street-usa:
    tomorrowland:
    fantasyland:

+ntp_servers:
+  hosts:
+    sundial:
+
+prometheus_server:
+  hosts:
+    cinderella-castle:
+
+prometheus_nodes:
+  hosts:
+    cinderella-castle:
+
+# node_explorer:
+#   hosts:
+#     cinderella-castle:
+
+# alertmanager:
+#   hosts:
+#     cinderella-castle:
+
+# grafana:
+#   hosts:
+#     cinderella-castle:
+
+semaphore_server:
+  hosts:
+    imagineering:
+
 dhcp_server:
  hosts:
    matchbox:

-# unbound_servers:
-#   hosts:
-#     unbound01:
-#     unbound02:
-
 backup_servers:
  hosts:
-    backup:
+    timekeeper:

 freeipa:
  hosts:
    infra01:

-prometheus_server:
-  hosts:
-    monitor:
-
-node_explorer:
-  hosts:
-    monitor:
-
-alertmanager:
-  hosts:
-    monitor:
-
-grafana:
-  hosts:
-    monitor:
-
 ipaserver:
  hosts:
    infra01.int.mk-labs.cloud:
@@ -64,37 +71,36 @@ internal_cluster:

 fastpass_control_plane:
  hosts:
-    space-mountain: # ansible_host=10.1.71.50
-    big-thunder-mountain: # ansible_host=10.1.71.51
-    splash-mountain: # ansible_host=10.1.71.52
+    space-mountain:
+    big-thunder-mountain:
+    splash-mountain:

 fastpass_workers:
  hosts:
    haunted-mansion: # backstage
    peter-pans-flight: # backstage

-
 fastpass:
  children:
    fastpass_control_plane:
    fastpass_workers:

-sql_servers:
-  hosts:
-    sql01:
+# sql_servers:
+#   hosts:
+#     sql01:

-n8n_servers:
-  hosts:
-    n8n:
+# n8n_servers:
+#   hosts:
+#     n8n:

-docker_servers:
-  hosts:
-    docker01:
+# docker_servers:
+#   hosts:
+#     docker01:

-observer:
-  hosts:
-    prometheus:
+# observer:
+#   hosts:
+#     prometheus:

-target:
-  hosts:
-    prometheus:
+# target:
+#   hosts:
+#     prometheus:
--- a/ansible/mk
+++ b/ansible/mk
@@ -0,0 +1,5 @@
+space-mountain
+splash-mountain
+big-thunder-mountain
+peter-pans-flight
+haunted-mansion
--- a/ansible/playbooks/configure_prometheus.yml
+++ b/ansible/playbooks/configure_prometheus.yml
@@ -1,16 +0,0 @@
---
- name: Master playbook to install and configure prometheus
-  hosts: prometheus_server
-  become: true
-
-  tasks:
-    - name: Permit traffic in default zone for dns service
-      firewalld:
-        port: 9090/tcp
-        permanent: true
-        immediate: true
-        state: enabled
-
-    - name: Install unbound via role
-      ansible.builtin.import_role:
-        name: prometheus.prometheus.prometheus
--- a/ansible/playbooks/deploy-fastpass-4step.yml
+++ b/ansible/playbooks/deploy-fastpass-4step.yml
@@ -0,0 +1,44 @@
+---
+- name: Step 1 - Install Prerequisites
+  hosts: fastpass
+  become: true
+  gather_facts: true
+  roles:
+    - role: kubernetes-prerequisites
+
+- name: Step 2 - Deploy First Control Plane Node
+  hosts: fastpass_control_plane[0]
+  become: true
+  gather_facts: false
+  roles:
+    - role: fastpass-first-control-plane
+
+- name: Step 3 - Deploy Additional Control Plane Nodes
+  hosts: fastpass_control_plane[1:]
+  become: true
+  gather_facts: false
+  roles:
+    - role: fastpass-additional-control-plane
+
+- name: Step 4 - Deploy Worker Nodes
+  hosts: fastpass_workers
+  become: true
+  gather_facts: false
+  roles:
+    - role: fastpass-workers
+
+- name: Final Cluster Validation
+  hosts: fastpass_control_plane[0]
+  become: false
+  gather_facts: false
+  environment:
+    KUBECONFIG: "{{ kubeconfig_path }}"
+  tasks:
+    - name: Display cluster status
+      ansible.builtin.command: kubectl get nodes
+      delegate_to: localhost
+
+    - name: Display all pods
+      ansible.builtin.command: kubectl get pods -A
+      delegate_to: localhost
+
--- a/ansible/playbooks/deploy-fastpass-cluster.yml
+++ b/ansible/playbooks/deploy-fastpass-cluster.yml
@@ -0,0 +1,157 @@
+---
+# FastPass Kubernetes Cluster Deployment
+# This playbook deploys a complete Kubernetes cluster on Fedora
+
+- name: 1. Preflight checks and system preparation
+  hosts: fastpass
+  become: true
+  gather_facts: true
+  roles:
+    - role: kubernetes
+
+- name: 2. Initialize first control plane node
+  hosts: fastpass_control_plane[0]
+  become: true
+  gather_facts: false
+  roles:
+    - role: fastpass-control-plane
+
+- name: 3. Install Calico CNI
+  hosts: fastpass_control_plane[0]
+  become: false
+  gather_facts: false
+  environment:
+    KUBECONFIG: "{{ kubeconfig_path }}"
+  tasks:
+    - name: Check if Calico is already installed
+      ansible.builtin.shell: kubectl get pods -n kube-system -l k8s-app=calico-node --no-headers | wc -l
+      delegate_to: localhost
+      register: calico_check
+      ignore_errors: true
+
+    - name: Display Calico check result
+      ansible.builtin.debug:
+        msg: "Calico pods found: {{ calico_check.stdout | trim }}"
+      delegate_to: localhost
+
+    - name: Install Calico CNI
+      ansible.builtin.command: kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/{{ calico_version | default('v3.28.0') }}/manifests/calico.yaml
+      delegate_to: localhost
+      when: calico_check.stdout | trim == "0"
+      register: calico_install_result
+      ignore_errors: true
+
+    - name: Display Calico installation result
+      ansible.builtin.debug:
+        msg: "Calico install stdout: {{ calico_install_result.stdout }}"
+      delegate_to: localhost
+      when: calico_check.stdout | trim == "0"
+
+    - name: Wait for Calico node pods to be ready
+      ansible.builtin.command: kubectl wait --for=condition=ready pod -l k8s-app=calico-node -n kube-system --timeout=300s
+      delegate_to: localhost
+      when: calico_check.stdout | trim == "0"
+
+- name: 4. Wait for first control plane to be fully ready
+  hosts: fastpass_control_plane[0]
+  become: false
+  gather_facts: false
+  environment:
+    KUBECONFIG: "{{ kubeconfig_path }}"
+  tasks:
+    - name: Wait for API server to be ready on first node
+      ansible.builtin.wait_for:
+        host: "{{ inventory_hostname }}"
+        port: "{{ control_plane_port | default('6443') }}"
+        timeout: 300
+      delegate_to: localhost
+
+    - name: Wait for first control plane node to be ready
+      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} wait --for=condition=ready node {{ inventory_hostname }} --timeout=300s
+      delegate_to: localhost
+
+    - name: Verify control plane status
+      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} get nodes -o wide
+      delegate_to: localhost
+      register: node_status
+
+    - name: Display cluster status
+      ansible.builtin.debug:
+        msg: "{{ node_status.stdout_lines }}"
+
+- name: 5. Join additional control plane nodes
+  hosts: fastpass_control_plane[1:]
+  become: true
+  gather_facts: false
+  roles:
+    - role: fastpass-control-plane-join
+
+- name: 6. Wait for all control plane nodes to be ready
+  hosts: fastpass_control_plane[0]
+  become: false
+  gather_facts: false
+  environment:
+    KUBECONFIG: "{{ kubeconfig_path }}"
+  tasks:
+    - name: Wait for all control plane nodes to be ready
+      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} wait --for=condition=ready node --selector=node-role.kubernetes.io/control-plane --timeout=600s
+      delegate_to: localhost
+
+    - name: Display all control plane nodes
+      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} get nodes --selector=node-role.kubernetes.io/control-plane -o wide
+      delegate_to: localhost
+      register: control_plane_status
+
+    - name: Show control plane status
+      ansible.builtin.debug:
+        msg: "{{ control_plane_status.stdout_lines }}"
+
+- name: 7. Join worker nodes
+  hosts: fastpass_workers
+  become: true
+  gather_facts: false
+  roles:
+    - role: fastpass-workers
+
+- name: 8. Final cluster validation
+  hosts: fastpass_control_plane[0]
+  become: false
+  gather_facts: false
+  environment:
+    KUBECONFIG: "{{ kubeconfig_path }}"
+  tasks:
+    - name: Wait for all nodes to be ready
+      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} wait --for=condition=ready node --all --timeout=300s
+      delegate_to: localhost
+
+    - name: Verify all pods are running
+      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} get pods --all-namespaces
+      delegate_to: localhost
+      register: pod_status
+
+    - name: Display final cluster status
+      ansible.builtin.debug:
+        msg: |
+          ========================================
+          FastPass Kubernetes Cluster Status
+          ========================================
+          {{ pod_status.stdout }}
+          ========================================
+
+    - name: Show final node status
+      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} get nodes -o wide
+      delegate_to: localhost
+      register: final_node_status
+
+    - name: Display final node status
+      ansible.builtin.debug:
+        msg: "{{ final_node_status.stdout_lines }}"
+
+    - name: Test DNS-based control plane endpoint
+      ansible.builtin.command: kubectl --kubeconfig={{ kubeconfig_path }} cluster-info
+      delegate_to: localhost
+      register: cluster_info
+
+    - name: Display cluster info
+      ansible.builtin.debug:
+        msg: "{{ cluster_info.stdout_lines }}" 
--- a/ansible/playbooks/deploy_k8s.yml
+++ b/ansible/playbooks/deploy_k8s.yml
@@ -2,7 +2,7 @@
 # FILE: deploy_k8s.yml
 # ------------------------------------------------------------------------------
 - name: 1. Prepare all nodes for Kubernetes
-  hosts: kube_cluster
+  hosts: fastpass
  roles:
    - role: kubernetes

@@ -12,6 +12,6 @@
    - role: fastpass-control-plane

 - name: 3. Join worker nodes to the cluster
-  hosts: fastpass-workers
+  hosts: fastpass_workers
  roles:
    - role: fastpass-workers
--- a/ansible/playbooks/deploy_ntp_servers.yml
+++ b/ansible/playbooks/deploy_ntp_servers.yml
@@ -0,0 +1,5 @@
+- name: 1. Deploy NTP servers
+  hosts: ntp_servers
+  gather_facts: true
+  roles:
+    - role: ntp-server
--- a/ansible/playbooks/install_monitoring.yml
+++ b/ansible/playbooks/install_monitoring.yml
@@ -0,0 +1,67 @@
+---
+- name: Master playbook to install and configure prometheus
+  hosts: prometheus_server
+  become: true
+
+  tasks:
+    - name: Install prerequisite software
+      ansible.builtin.dnf:
+        name:
+          - tar
+          - python3-dnf
+        state: present
+
+    - name: Permit prometheus metrics in default zone for dns service
+      ansible.posix.firewalld:
+        port: 9100/tcp
+        permanent: true
+        immediate: true
+        state: enabled
+
+    - name: Permit traffic in default zone for dns service
+      ansible.posix.firewalld:
+        port: 9090/tcp
+        permanent: true
+        immediate: true
+        state: enabled
+
+    - name: Install prometheus via role
+      ansible.builtin.import_role:
+        name: prometheus.prometheus.prometheus
+      vars:
+        prometheus_targets:
+          node:
+          - targets:
+            - localhost:9100
+            labels:
+              env: mk-labs
+
+    - name: Permit grafana in default zone for dns service
+      ansible.posix.firewalld:
+        port: 3000/tcp
+        permanent: true
+        immediate: true
+        state: enabled
+
+    - name: Install grafana via role
+      ansible.builtin.import_role:
+        name: grafana.grafana.grafana
+
+    # - name: Create/Update Data sources
+    #   grafana.grafana.datasource:
+    #     dataSource: |
+    #       {
+    #         "name": "Prometheus",
+    #         "type": "prometheus",
+    #         "access": "proxy",
+    #         "url": "http://localhost:9090",
+    #         "jsonData": {
+    #           "httpMethod": "POST",
+    #           "manageAlerts": true,
+    #           "prometheusType": "Prometheus",
+    #           "cacheLevel": "High"
+    #         }
+    #       }
+    #     grafana_url: "{{ grafana_url }}"
+    #     grafana_api_key: "{{ grafana_api_key }}"
+    #     state: present
--- a/ansible/playbooks/node_explorer.yml
+++ b/ansible/playbooks/node_explorer.yml
@@ -0,0 +1,16 @@
+---
+- name: Master playbook to install and configure prometheus node explorer
+  hosts: prometheus_server
+  become: true
+
+  tasks:
+    - name: Permit prometheus metrics in default zone for dns service
+      ansible.posix.firewalld:
+        port: 9100/tcp
+        permanent: true
+        immediate: true
+        state: enabled
+
+    - name: Install prometheus node exporter via role
+      ansible.builtin.import_role:
+        name: prometheus.prometheus.node_exporter
--- a/ansible/playbooks/prometheus_grafana_server.yml
+++ b/ansible/playbooks/prometheus_grafana_server.yml
@@ -0,0 +1,143 @@
+---
+- name: Install Prometheus and Grafana on control node
+  hosts: prometheus_server
+  become: true
+  # vars_files:
+  #   - vars.yml
+  tasks:
+    - name: Install prerequisite software
+      ansible.builtin.dnf:
+        name: 
+          - tar
+          - wget
+        state: present
+
+    - name: Download Prometheus
+      ansible.builtin.get_url:
+        url: "{{ prometheus_installer_download_url }}"
+        dest: "/tmp/{{ prometheus_installer_file }}"
+
+    - name: Extract Prometheus
+      ansible.builtin.unarchive:
+        src: "/tmp/{{ prometheus_installer_file }}"
+        dest: "/usr/local/bin/"
+        remote_src: true
+
+    - name: Create Prometheus user
+      user:
+        name: prometheus
+        shell: /bin/false
+        state: present
+
+    - name: Create Prometheus directories
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: directory
+        owner: prometheus
+        group: prometheus
+      with_items:
+        - /etc/prometheus
+        - /var/lib/prometheus
+
+    - name: Move Prometheus binaries
+      ansible.builtin.command: "mv /usr/local/bin/prometheus-{{ prometheus_version }}.linux-amd64/prometheus /usr/local/bin/prometheus"
+
+    - name: Move Prometheus tool
+      ansible.builtin.command: "mv /usr/local/bin/prometheus-{{ prometheus_version }}.linux-amd64/promtool /usr/local/bin/promtool"
+
+    - name: Create Prometheus configuration file
+      ansible.builtin.copy:
+        dest: "/etc/prometheus/prometheus.yml"
+        content: |
+          global:
+            scrape_interval: 15s
+            evaluation_interval: 15s
+
+          scrape_configs:
+            - job_name: 'prometheus'
+              static_configs:
+                - targets: ['localhost:9090']
+
+            - job_name: 'nodes'
+              static_configs:
+                - targets:
+                  {% for ip in prometheus_nodes %}
+                  - '{{ ip }}:9100'
+                  {% endfor %}
+
+    - name: Create Prometheus service file
+      ansible.builtin.copy:
+        dest: "/etc/systemd/system/prometheus.service"
+        content: |
+          [Unit]
+          Description=Prometheus
+          Wants=network-online.target
+          After=network-online.target
+
+          [Service]
+          User=prometheus
+          Group=prometheus
+          Type=simple
+          ExecStart=/usr/local/bin/prometheus \
+            --config.file=/etc/prometheus/prometheus.yml \
+            --storage.tsdb.path=/var/lib/prometheus/ \
+            --web.console.templates=/etc/prometheus/consoles \
+            --web.console.libraries=/etc/prometheus/console_libraries
+
+          [Install]
+          WantedBy=multi-user.target
+
+      register: prometheus_service_status
+      ignore_errors: true
+
+    - name: Check if Prometheus service is running
+      ansible.builtin.command: systemctl is-active prometheus
+      register: prometheus_status
+      changed_when: false
+      ignore_errors: true
+
+    - name: Permit prometheus endpoint in default zone for dns service
+      firewalld:
+        port: 9090/tcp
+        permanent: true
+        immediate: true
+        state: enabled
+
+    - name: Permit prometheus metrics in default zone for dns service
+      firewalld:
+        port: 9100/tcp
+        permanent: true
+        immediate: true
+        state: enabled
+
+    - name: Permit grafana traffic in default zone for dns service
+      firewalld:
+        port: 3000/tcp
+        permanent: true
+        immediate: true
+        state: enabled
+
+    - name: Manage Prometheus service state
+      systemd:
+        name: prometheus
+        state: "{{ 'restarted' if prometheus_status.rc == 0 else 'started' }}"
+        enabled: true
+
+    - name: Add repository
+      ansible.builtin.yum_repository:
+        name: grafana
+        description: Grafana OSS repo
+        baseurl: https://rpm.grafana.com
+        gpgkey: https://rpm.grafana.com/gpg.key
+
+    - name: Install Grafana
+      ansible.builtin.dnf:
+        name: 
+          - grafana
+        state: present
+
+    - name: Start Grafana service
+      systemd:
+        name: grafana-server
+        state: started
+        enabled: true
--- a/ansible/playbooks/proxmox_clone_vm.yml
+++ b/ansible/playbooks/proxmox_clone_vm.yml
@@ -22,7 +22,7 @@
            name: "{{ inventory_hostname }}"
            node: "{{ proxmox_clone_node }}"
            storage: "{{ vm_storage }}"
-            format: raw
+            format: qcow2 # raw
            timeout: 600

        # - name: Add VM to HA group
--- a/ansible/playbooks/proxmox_start_vm.yml
+++ b/ansible/playbooks/proxmox_start_vm.yml
@@ -15,18 +15,26 @@
        node: "{{ proxmox_clone_node }}"
        state: started

-    # - name: Wait for the VM to start
-    #   delegate_to: "localhost"
-    #   wait_for:
-    #     port: 22
-    #     host: "{{ inventory_hostname }}"
-    #     search_regex: OpenSSH
-    #     delay: 10
-    #     timeout: 60
+    - name: Wait for SSH service to be ready
+      delegate_to: "localhost"
+      when: platform is defined and platform == "proxmox"
+      ansible.builtin.wait_for:
+        port: 22
+        host: "{{ inventory_hostname }}"
+        search_regex: OpenSSH
+        timeout: 300
+        msg: "Waiting for SSH service to be ready on {{ inventory_hostname }}"

-    # - name: Wait for the reboot to complete if there was a change.
-    #   wait_for_connection:
-    #     connect_timeout: 10
-    #     sleep: 5
-    #     delay: 5
-    #     timeout: 300
+    - name: Wait for VM to be fully booted and responsive
+      delegate_to: "localhost"
+      when: platform is defined and platform == "proxmox"
+      ansible.builtin.wait_for_connection:
+        connect_timeout: 10
+        sleep: 5
+        timeout: 300
+
+    - name: Display VM startup completion
+      delegate_to: "localhost"
+      when: platform is defined and platform == "proxmox"
+      ansible.builtin.debug:
+        msg: "✅ {{ inventory_hostname }} is now running and accessible via SSH"
--- a/ansible/playbooks/reset-fastpass-cluster.yml
+++ b/ansible/playbooks/reset-fastpass-cluster.yml
@@ -0,0 +1,115 @@
+---
+- name: Reset FastPass Kubernetes Cluster
+  hosts: fastpass
+  become: true
+  gather_facts: false
+  tasks:
+    - name: Stop and disable kubelet service
+      ansible.builtin.systemd:
+        name: kubelet
+        state: stopped
+        enabled: false
+      ignore_errors: true
+
+    - name: Stop and disable containerd service
+      ansible.builtin.systemd:
+        name: containerd
+        state: stopped
+        enabled: false
+      ignore_errors: true
+
+    - name: Remove Kubernetes packages
+      ansible.builtin.dnf:
+        name:
+          - kubelet
+          - kubeadm
+          - kubectl
+          - kubernetes-cni
+          - containernetworking-plugins
+        state: absent
+      ignore_errors: true
+
+    - name: Remove containerd
+      ansible.builtin.dnf:
+        name: containerd
+        state: absent
+      ignore_errors: true
+
+    - name: Remove Docker repository
+      ansible.builtin.file:
+        path: /etc/yum.repos.d/docker-ce.repo
+        state: absent
+      ignore_errors: true
+
+    - name: Remove Kubernetes repository
+      ansible.builtin.file:
+        path: /etc/yum.repos.d/kubernetes.repo
+        state: absent
+      ignore_errors: true
+
+    - name: Remove Kubernetes directories
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: absent
+      loop:
+        - /etc/kubernetes
+        - /var/lib/kubelet
+        - /var/lib/etcd
+        - /etc/cni/net.d
+        - /opt/cni/bin
+        - /var/lib/containerd
+        - /etc/containerd
+      ignore_errors: true
+
+    - name: Remove CNI plugins
+      ansible.builtin.file:
+        path: /opt/cni
+        state: absent
+      ignore_errors: true
+
+    - name: Remove iptables rules
+      ansible.builtin.shell: |
+        iptables -F
+        iptables -t nat -F
+        iptables -t mangle -F
+        iptables -X
+      ignore_errors: true
+
+    - name: Remove firewall Kubernetes service
+      ansible.builtin.file:
+        path: /etc/firewalld/services/kubernetes.xml
+        state: absent
+      ignore_errors: true
+
+    - name: Remove firewall rules for Kubernetes
+      ansible.builtin.shell: |
+        firewall-cmd --permanent --remove-service=kubernetes || true
+        firewall-cmd --reload || true
+      ignore_errors: true
+
+    - name: Reset network interfaces
+      ansible.builtin.shell: |
+        ip link delete cni0 || true
+        ip link delete flannel.1 || true
+        ip link delete cali* || true
+      ignore_errors: true
+
+    - name: Clean up systemd drop-in files
+      ansible.builtin.file:
+        path: /usr/lib/systemd/system/kubelet.service.d
+        state: absent
+      ignore_errors: true
+
+    - name: Reset hostname to original
+      ansible.builtin.hostname:
+        name: "{{ inventory_hostname }}"
+
+    - name: Clean package cache
+      ansible.builtin.dnf:
+        clean: all
+      ignore_errors: true
+
+    - name: Reboot system
+      ansible.builtin.reboot:
+        reboot_timeout: 300
+
--- a/ansible/playbooks/roles/fastpass-control-plane/tasks/main.yml
+++ b/ansible/playbooks/roles/fastpass-control-plane/tasks/main.yml
@@ -1,59 +0,0 @@
-# ------------------------------------------------------------------------------
-# FILE: roles/control-plane/tasks/main.yml
-# ------------------------------------------------------------------------------
- name: Check if cluster is already initialized
-  ansible.builtin.stat:
-    path: /etc/kubernetes/admin.conf
-  register: kube_init_stat
-
- name: Initialize the Kubernetes cluster
-  when: not kube_init_stat.stat.exists
-  ansible.builtin.command: >
-    kubeadm init
-    --pod-network-cidr=10.244.0.0/16
-    --control-plane-endpoint={{ inventory_hostname }}
-  register: kubeadm_init
-
- name: Create .kube directory for the user
-  become: false
-  ansible.builtin.file:
-    path: "{{ ansible_env.HOME }}/.kube"
-    state: directory
-    mode: '0755'
-  delegate_to: localhost
-  run_once: true
-
- name: Copy admin.conf to user's .kube directory on controller
-  when: not kube_init_stat.stat.exists
-  ansible.builtin.fetch:
-    src: /etc/kubernetes/admin.conf
-    dest: "{{ ansible_env.HOME }}/.kube/config"
-    flat: true
-
- name: Install Calico CNI
-  become: false
-  ansible.builtin.command: kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml
-  delegate_to: localhost
-  run_once: true
-  when: not kube_init_stat.stat.exists
-
- name: Generate join command for control plane nodes
-  when: not kube_init_stat.stat.exists
-  ansible.builtin.command: kubeadm token create --print-join-command
-  register: control_plane_join_command_raw
-
- name: Generate certificate key for control plane join
-  when: not kube_init_stat.stat.exists
-  ansible.builtin.command: kubeadm init phase upload-certs --upload-certs
-  register: control_plane_cert_key
-
- name: Store control plane join command
-  ansible.builtin.set_fact:
-    control_plane_join_command: "{{ control_plane_join_command_raw.stdout }} --control-plane --certificate-key {{ control_plane_cert_key.stdout_lines[-1] }}"
-  when: not kube_init_stat.stat.exists
-
- name: Join other control plane nodes to the cluster
-  when:
-    - inventory_hostname != groups['fastpass_control_plane'][0]
-    - not kube_init_stat.stat.exists
-  ansible.builtin.command: "{{ hostvars[groups['fastpass_control_plane'][0]].control_plane_join_command }}"
--- a/ansible/playbooks/roles/fastpass-workers/tasks/main.yml
+++ b/ansible/playbooks/roles/fastpass-workers/tasks/main.yml
@@ -1,35 +0,0 @@
-# ------------------------------------------------------------------------------
-# FILE: roles/fastpass_workers/tasks/main.yml
-# ------------------------------------------------------------------------------
- name: Generate join command for worker nodes
-  ansible.builtin.command: kubeadm token create --print-join-command
-  delegate_to: "{{ groups['fastpass_control_plane'][0] }}"
-  run_once: true
-  register: worker_join_command
-
- name: Join worker nodes to the cluster
-  ansible.builtin.command: "{{ hostvars[groups['fastpass_control_plane'][0]].worker_join_command.stdout }}"
-
- name: Label and Taint on-stage worker nodes
-  become: false
-  ansible.builtin.command: "kubectl {{ item }}"
-  delegate_to: localhost
-  run_once: true
-  loop:
-    - label node seven-dwarfs-mine-train zone=on-stage --overwrite
-    - taint node seven-dwarfs-mine-train dedicated=external:NoSchedule --overwrite
-  when: "'seven-dwarfs-mine-train' in inventory_hostname"
-
- name: Label backstage worker node haunted-mansion
-  become: false
-  ansible.builtin.command: "kubectl label node haunted-mansion zone=backstage --overwrite"
-  delegate_to: localhost
-  run_once: true
-  when: "'haunted-mansion' in inventory_hostname"
-
- name: Label backstage worker node peter-pans-flight
-  become: false
-  ansible.builtin.command: "kubectl label node peter-pans-flight zone=backstage --overwrite"
-  delegate_to: localhost
-  run_once: true
-  when: "'peter-pans-flight' in inventory_hostname"
--- a/ansible/playbooks/roles/kubernetes-prerequisites/defaults/main.yml
+++ b/ansible/playbooks/roles/kubernetes-prerequisites/defaults/main.yml
@@ -0,0 +1,47 @@
+---
+# Kubernetes Prerequisites Role Defaults
+
+# Kubernetes version to install (should be overridden in group_vars)
+kubernetes_version: "v1.33"
+
+# Container runtime (containerd is default)
+container_runtime: "containerd"
+
+# CNI plugin to install
+cni_plugin: "calico"
+
+# Firewall configuration
+configure_firewall: true
+
+# Time synchronization
+configure_ntp: true
+
+# SELinux configuration
+selinux_state: "permissive"
+
+# Swap configuration
+disable_swap: true
+
+# Kernel modules to load
+required_kernel_modules:
+  - overlay
+  - br_netfilter
+
+# Sysctl parameters for Kubernetes
+k8s_sysctl_params:
+  net.bridge.bridge-nf-call-iptables: 1
+  net.bridge.bridge-nf-call-ip6tables: 1
+  net.ipv4.ip_forward: 1
+
+# Kubernetes firewall ports
+k8s_firewall_ports:
+  - { protocol: "tcp", port: "6443" }    # API server
+  - { protocol: "tcp", port: "2379" }    # etcd client
+  - { protocol: "tcp", port: "2380" }    # etcd peer
+  - { protocol: "tcp", port: "10250" }   # kubelet
+  - { protocol: "tcp", port: "10251" }   # kube-scheduler
+  - { protocol: "tcp", port: "10252" }   # kube-controller-manager
+  - { protocol: "tcp", port: "10255" }   # kubelet read-only
+  - { protocol: "tcp", port: "30000-32767" }  # NodePort services
+  - { protocol: "udp", port: "4789" }    # VXLAN (Flannel)
+  - { protocol: "tcp", port: "179" }     # BGP (Calico)
--- a/ansible/playbooks/roles/kubernetes-prerequisites/tasks/main.yml
+++ b/ansible/playbooks/roles/kubernetes-prerequisites/tasks/main.yml
@@ -0,0 +1,210 @@
+---
+- name: Set Kubernetes-compatible hostname
+  ansible.builtin.hostname:
+    name: "{{ kubernetes_hostnames[inventory_hostname] | default(inventory_hostname) }}"
+
+- name: Remove zram-generator-defaults
+  ansible.builtin.file:
+    path: /etc/systemd/zram-generator.conf
+    state: absent
+
+- name: Disable swap
+  ansible.builtin.shell: swapoff -a
+  when: disable_swap | default(true)
+
+- name: Persist swap off by commenting out swap in fstab
+  ansible.builtin.replace:
+    path: /etc/fstab
+    regexp: '^([^#].*?\sswap\s+sw\s+.*)$'
+    replace: '# \1'
+  when: disable_swap | default(true)
+
+- name: Load required kernel modules
+  ansible.builtin.modprobe:
+    name: "{{ item }}"
+  loop: "{{ required_kernel_modules }}"
+
+- name: Persist kernel modules on boot
+  ansible.builtin.copy:
+    dest: /etc/modules-load.d/k8s.conf
+    content: "{{ required_kernel_modules | join('\n') }}"
+    owner: root
+    group: root
+    mode: '0644'
+
+- name: Configure required sysctl params for Kubernetes
+  ansible.builtin.template:
+    dest: /etc/sysctl.d/k8s.conf
+    src: k8s-sysctl.conf.j2
+    owner: root
+    group: root
+    mode: '0644'
+
+- name: Apply sysctl params without reboot
+  ansible.builtin.shell: sysctl --system
+
+- name: Install python3-libselinux for managing selinux
+  ansible.builtin.dnf:
+    name: python3-libselinux
+    state: present
+
+- name: Set SELinux to permissive mode
+  ansible.posix.selinux:
+    policy: targeted
+    state: "{{ selinux_state }}"
+
+- name: Install and configure chrony for time synchronization
+  ansible.builtin.dnf:
+    name: chrony
+    state: present
+  when: configure_ntp | default(true)
+
+- name: Start and enable chronyd
+  ansible.builtin.systemd:
+    name: chronyd
+    state: started
+    enabled: true
+  when: configure_ntp | default(true)
+
+- name: Force time synchronization
+  ansible.builtin.shell: chronyc makestep
+  ignore_errors: true
+  when: configure_ntp | default(true)
+
+- name: Install firewalld
+  ansible.builtin.dnf:
+    name: firewalld
+    state: present
+  when: configure_firewall | default(true)
+
+- name: Start and enable firewalld
+  ansible.builtin.systemd:
+    name: firewalld
+    state: started
+    enabled: true
+  when: configure_firewall | default(true)
+
+- name: Create Kubernetes service definition
+  ansible.builtin.template:
+    dest: /etc/firewalld/services/kubernetes.xml
+    owner: root
+    group: root
+    mode: '0644'
+    src: kubernetes-firewall-service.xml.j2
+  when: configure_firewall | default(true)
+
+- name: Reload firewalld to load new service
+  ansible.builtin.shell: firewall-cmd --reload
+  when: configure_firewall | default(true)
+
+- name: Add Kubernetes service to default zone
+  ansible.builtin.shell: firewall-cmd --permanent --add-service=kubernetes
+  when: configure_firewall | default(true)
+
+- name: Add SSH service to default zone (ensure SSH access)
+  ansible.builtin.shell: firewall-cmd --permanent --add-service=ssh
+  when: configure_firewall | default(true)
+
+- name: Reload firewalld to apply changes
+  ansible.builtin.shell: firewall-cmd --reload
+  when: configure_firewall | default(true)
+
+- name: Verify Kubernetes service is active
+  ansible.builtin.shell: firewall-cmd --list-services
+  register: active_services
+  when: configure_firewall | default(true)
+
+- name: Display active firewall services
+  ansible.builtin.debug:
+    msg: "Active firewall services: {{ active_services.stdout }}"
+  when: configure_firewall | default(true)
+
+- name: Install DNF plugins core for managing repositories
+  ansible.builtin.dnf:
+    name: dnf-plugins-core
+    state: present
+
+- name: Add Docker CE repository
+  ansible.builtin.shell: |
+    curl -fsSL https://download.docker.com/linux/fedora/docker-ce.repo -o /etc/yum.repos.d/docker-ce.repo
+
+- name: Install containerd
+  ansible.builtin.dnf:
+    name: containerd
+    state: present
+
+- name: Create containerd config directory
+  ansible.builtin.file:
+    path: /etc/containerd
+    state: directory
+    owner: root
+    group: root
+    mode: '0755'
+
+- name: Generate default containerd config and enable SystemdCgroup
+  ansible.builtin.shell: |
+    containerd config default > /etc/containerd/config.toml
+    sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
+
+- name: Restart and enable containerd service
+  ansible.builtin.systemd:
+    name: containerd
+    state: restarted
+    enabled: true
+
+- name: Add Kubernetes repository
+  ansible.builtin.template:
+    dest: /etc/yum.repos.d/kubernetes.repo
+    owner: root
+    group: root
+    mode: '0644'
+    src: kubernetes.repo.j2
+
+- name: Install Kubernetes packages
+  ansible.builtin.dnf:
+    name:
+      - kubelet
+      - kubeadm
+      - kubectl
+      - kubernetes-cni
+    state: present
+
+- name: Create kubelet config directory
+  ansible.builtin.file:
+    path: /var/lib/kubelet
+    state: directory
+    owner: root
+    group: root
+    mode: '0755'
+
+- name: Create initial kubelet config file
+  ansible.builtin.copy:
+    dest: /var/lib/kubelet/config.yaml
+    owner: root
+    group: root
+    mode: '0644'
+    content: |
+      apiVersion: kubelet.config.k8s.io/v1beta1
+      kind: KubeletConfiguration
+      cgroupDriver: systemd
+      containerRuntimeEndpoint: unix:///run/containerd/containerd.sock
+
+- name: Install CNI plugins via dnf
+  ansible.builtin.dnf:
+    name: containernetworking-plugins
+    state: present
+
+- name: Verify CNI plugins installation
+  ansible.builtin.shell: ls -la /opt/cni/bin/ || echo "CNI plugins not found in /opt/cni/bin/"
+  register: cni_plugins
+  ignore_errors: true
+
+- name: Display installed CNI plugins
+  ansible.builtin.debug:
+    msg: "CNI plugins status: {{ cni_plugins.stdout }}"
+
+- name: Enable kubelet service (but don't start yet)
+  ansible.builtin.systemd:
+    name: kubelet
+    enabled: true
+    state: stopped
--- a/ansible/playbooks/roles/kubernetes-prerequisites/templates/kubernetes-firewall-service.xml.j2
+++ b/ansible/playbooks/roles/kubernetes-prerequisites/templates/kubernetes-firewall-service.xml.j2
@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="utf-8"?>
+<service>
+  <short>Kubernetes</short>
+  <description>Kubernetes cluster communication ports</description>
+{% for port in k8s_firewall_ports %}
+  <port protocol="{{ port.protocol }}" port="{{ port.port }}"/>
+{% endfor %}
+</service>
+
--- a/ansible/playbooks/roles/kubernetes-prerequisites/templates/kubernetes.repo.j2
+++ b/ansible/playbooks/roles/kubernetes-prerequisites/templates/kubernetes.repo.j2
@@ -0,0 +1,7 @@
+[kubernetes]
+name=Kubernetes
+baseurl=https://pkgs.k8s.io/core:/stable:/v{{ kubernetes_version | default('1.33') | regex_replace('^v?(.*)$', '\\1') | regex_replace('^(\\d+\\.\\d+).*$', '\\1') }}/rpm/
+enabled=1
+gpgcheck=1
+gpgkey=https://pkgs.k8s.io/core:/stable:/v{{ kubernetes_version | default('1.33') | regex_replace('^v?(.*)$', '\\1') | regex_replace('^(\\d+\\.\\d+).*$', '\\1') }}/rpm/repodata/repomd.xml.key
+
--- a/ansible/playbooks/roles/kubernetes/tasks/main.yml
+++ b/ansible/playbooks/roles/kubernetes/tasks/main.yml
@@ -1,11 +1,25 @@
 # ------------------------------------------------------------------------------
 # FILE: roles/common/tasks/main.yml
 # ------------------------------------------------------------------------------
+- name: Set Kubernetes-compatible hostname
+  become: true
+  ansible.builtin.hostname:
+    name: "{{ kubernetes_hostnames[inventory_hostname] | default(inventory_hostname) }}"
+  when: kubernetes_hostnames is defined
+
+- name: Remove zram-generator-defaults
+  become: true
+  ansible.builtin.dnf:
+    name: zram-generator-defaults
+    state: absent
+
 - name: Disable swap
  ansible.builtin.command: swapoff -a
+  become: true
  changed_when: false

 - name: Persist swap off by commenting out swap in fstab
+  become: true
  ansible.builtin.replace:
    path: /etc/fstab
    regexp: '^(\s*)([^#\n]+\s+swap\s+.*)$'
@@ -13,6 +27,7 @@
    backup: true

 - name: Load required kernel modules
+  become: true
  community.general.modprobe:
    name: "{{ item }}"
    state: present
@@ -21,50 +36,121 @@
    - br_netfilter

 - name: Persist kernel modules on boot
+  become: true
  ansible.builtin.copy:
    dest: /etc/modules-load.d/k8s.conf
+    mode: '0644'
    content: |
      overlay
      br_netfilter

 - name: Configure required sysctl params for Kubernetes
+  become: true
  ansible.builtin.copy:
    dest: /etc/sysctl.d/k8s.conf
+    mode: '0644'
    content: |
      net.bridge.bridge-nf-call-iptables  = 1
      net.bridge.bridge-nf-call-ip6tables = 1
      net.ipv4.ip_forward                 = 1

 - name: Apply sysctl params without reboot
+  become: true
  ansible.builtin.command: sysctl --system
  changed_when: false

+- name: Install python3-libselinux for managing selinux
+  become: true
+  ansible.builtin.dnf:
+    name: python3-libselinux
+    state: present
+
 - name: Set SELinux to permissive mode
-  selinux:
+  become: true
+  ansible.posix.selinux:
    policy: targeted
    state: permissive

+- name: Create Kubernetes service definition
+  become: true
+  ansible.builtin.copy:
+    dest: /etc/firewalld/services/kubernetes.xml
+    owner: root
+    group: root
+    mode: '0644'
+    content: |
+      <?xml version="1.0" encoding="utf-8"?>
+      <service>
+        <short>Kubernetes</short>
+        <description>Kubernetes cluster communication ports</description>
+        <port protocol="tcp" port="6443"/>
+        <port protocol="tcp" port="2379"/>
+        <port protocol="tcp" port="2380"/>
+        <port protocol="tcp" port="10250"/>
+        <port protocol="tcp" port="10251"/>
+        <port protocol="tcp" port="10252"/>
+        <port protocol="tcp" port="10255"/>
+        <port protocol="tcp" port="30000-32767"/>
+        <port protocol="udp" port="4789"/>
+        <port protocol="tcp" port="179"/>
+      </service>
+
+- name: Reload firewalld to load new service
+  become: true
+  ansible.builtin.command: firewall-cmd --reload
+
+- name: Add Kubernetes service to default zone
+  become: true
+  ansible.builtin.command: firewall-cmd --permanent --add-service=kubernetes
+
+- name: Add SSH service to default zone (ensure SSH access)
+  become: true
+  ansible.builtin.command: firewall-cmd --permanent --add-service=ssh
+
+- name: Reload firewalld to apply changes
+  become: true
+  ansible.builtin.command: firewall-cmd --reload
+
+- name: Verify Kubernetes service is active
+  become: true
+  ansible.builtin.command: firewall-cmd --list-services
+  register: firewall_services
+  changed_when: false
+
+- name: Display active firewall services
+  ansible.builtin.debug:
+    msg: "Active firewall services: {{ firewall_services.stdout }}"
+
 - name: Install DNF plugins core for managing repositories
+  become: true
  ansible.builtin.dnf:
    name: dnf-plugins-core
    state: present

 - name: Add Docker CE repository
-  ansible.builtin.command: dnf config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo
-  args:
-    creates: /etc/yum.repos.d/docker-ce.repo
+  become: true
+  ansible.builtin.yum_repository:
+    name: docker-ce
+    baseurl: https://download.docker.com/linux/fedora/docker-ce.repo
+    gpgcheck: false
+    enabled: true
+    description: "Docker repository"

 - name: Install containerd
+  become: true
  ansible.builtin.dnf:
-    name: containerd.io
+    name: containerd
    state: present

 - name: Create containerd config directory
+  become: true
  ansible.builtin.file:
    path: /etc/containerd
+    mode: '0755'
    state: directory

 - name: Generate default containerd config and enable SystemdCgroup
+  become: true
  ansible.builtin.shell: |
    containerd config default > /etc/containerd/config.toml
    sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
@@ -72,6 +158,7 @@
    creates: /etc/containerd/config.toml

 - name: Restart and enable containerd service
+  become: true
  ansible.builtin.systemd:
    name: containerd
    state: restarted
@@ -79,6 +166,7 @@
    daemon_reload: true

 - name: Add Kubernetes repository
+  become: true
  ansible.builtin.yum_repository:
    name: kubernetes
    description: Kubernetes
@@ -87,6 +175,7 @@
    gpgcheck: true

 - name: Install Kubernetes packages
+  become: true
  ansible.builtin.dnf:
    name:
      - kubelet
@@ -95,7 +184,73 @@
    state: present
    disable_excludes: kubernetes

- name: Enable the kubelet service
+- name: Create kubelet config directory
+  become: true
+  ansible.builtin.file:
+    path: /var/lib/kubelet
+    state: directory
+    owner: root
+    group: root
+    mode: '0755'
+
+- name: Create initial kubelet config file
+  become: true
+  ansible.builtin.copy:
+    dest: /var/lib/kubelet/config.yaml
+    owner: root
+    group: root
+    mode: '0644'
+    content: |
+      apiVersion: kubelet.config.k8s.io/v1beta1
+      kind: KubeletConfiguration
+      authentication:
+        anonymous:
+          enabled: false
+        webhook:
+          enabled: true
+        x509:
+          clientCAFile: /etc/kubernetes/pki/ca.crt
+      authorization:
+        mode: Webhook
+      clusterDomain: cluster.local
+      clusterDNS:
+        - 10.96.0.10
+      cpuManagerPolicy: none
+      evictionHard:
+        imagefs.available: 15%
+        memory.available: 100Mi
+        nodefs.available: 10%
+        nodefs.inodesFree: 5%
+      maxPods: 110
+      podCIDR: {{ pod_network_cidr | default('10.244.0.0/16') }}
+      resolvConf: /etc/resolv.conf
+      runtimeRequestTimeout: 2m
+      staticPodPath: /etc/kubernetes/manifests
+      streamingConnectionIdleTimeout: 4h0m0s
+      syncFrequency: 1m0s
+      volumeStatsAggPeriod: 1m0s
+
+- name: Install CNI plugins via dnf
+  become: true
+  ansible.builtin.dnf:
+    name: containernetworking-plugins
+    state: present
+
+- name: Verify CNI plugins installation
+  become: true
+  ansible.builtin.command: ls -la /usr/libexec/cni/
+  register: cni_plugins_list
+  changed_when: false
+
+
+
+- name: Display installed CNI plugins
+  ansible.builtin.debug:
+    msg: "Installed CNI plugins: {{ cni_plugins_list.stdout_lines }}"
+
+- name: Enable and start the kubelet service
+  become: true
  ansible.builtin.systemd:
    name: kubelet
-    enabled: true
+    enabled: true
+    state: started
--- a/ansible/playbooks/roles/ntp-server/tasks/main.yml
+++ b/ansible/playbooks/roles/ntp-server/tasks/main.yml
@@ -0,0 +1,46 @@
+---
+# Role to install and configure an Chrony NTP server
+
+- name: 1. Install Chrony NTP for time synchronization
+  become: true
+  ansible.builtin.dnf:
+    name: chrony
+    state: present
+
+- name: 2.1 Remove DHCP NTP servers from config
+  become: true
+  ansible.builtin.lineinfile:
+    path: /etc/chrony.conf
+    line: "server unifi.int.mk-labs.cloud iburst"
+    state: absent
+
+- name: 2.2 Ensure specific NTP servers are present
+  become: true
+  ansible.builtin.lineinfile:
+    path: /etc/chrony.conf
+    line: "server {{ item }} iburst"
+    state: present
+  loop: "{{ ntp_servers }}"
+
+- name: 3. Ensure network allowed are present
+  become: true
+  ansible.builtin.lineinfile:
+    path: /etc/chrony.conf
+    line: "allow {{ item }}"
+    state: present
+  loop: "{{ allowed_networks }}"
+
+- name: 4.Restart ntpd service
+  become: true
+  ansible.builtin.systemd:
+    name: chronyd
+    state: restarted
+    enabled: true
+
+- name: 5. Allow NTP traffic
+  become: true
+  ansible.posix.firewalld:
+    service: ntp
+    permanent: true
+    immediate: true
+    state: enabled
--- a/ansible/playbooks/set_hostname.yml
+++ b/ansible/playbooks/set_hostname.yml
@@ -9,3 +9,8 @@
      delegate_to: "{{ inventory_hostname }}"
      ansible.builtin.hostname:
        name: "{{ hostname }}"
+
+    - name: Set timezone to US/Central
+      become: true
+      community.general.timezone:
+        name: US/Central
--- a/ansible/playbooks/templates/kubeadm-config.yaml.j2
+++ b/ansible/playbooks/templates/kubeadm-config.yaml.j2
@@ -0,0 +1,28 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: kubeadm-config
+  namespace: kube-system
+data:
+  ClusterConfiguration: |
+    apiServer: {}
+    apiVersion: kubeadm.k8s.io/v1beta4
+    caCertificateValidityPeriod: 87600h0m0s
+    certificateValidityPeriod: 8760h0m0s
+    certificatesDir: /etc/kubernetes/pki
+    clusterName: kubernetes
+    controllerManager: {}
+    dns: {}
+    encryptionAlgorithm: RSA-2048
+    etcd:
+      local:
+        dataDir: /var/lib/etcd
+    imageRepository: registry.k8s.io
+    kind: ClusterConfiguration
+    kubernetesVersion: {{ kubernetes_version | default('v1.33.4') }}
+    networking:
+      dnsDomain: cluster.local
+      serviceSubnet: {{ service_cidr | default('10.96.0.0/12') }}
+    proxy: {}
+    scheduler: {}
+    controlPlaneEndpoint: {{ control_plane_endpoint | default('fastpass.local.mk-labs.cloud') }}:{{ control_plane_port | default('6443') }}
--- a/ansible/playbooks/test-control-plane.yml
+++ b/ansible/playbooks/test-control-plane.yml
@@ -0,0 +1,8 @@
+---
+- name: Test Control Plane Initialization on Single Node
+  hosts: space-mountain
+  become: true
+  gather_facts: false
+  roles:
+    - role: fastpass-control-plane
+
--- a/ansible/playbooks/test-kubernetes-role.yml
+++ b/ansible/playbooks/test-kubernetes-role.yml
@@ -0,0 +1,8 @@
+---
+- name: Test Kubernetes Role on Single Node
+  hosts: space-mountain
+  become: true
+  gather_facts: true
+  roles:
+    - role: kubernetes
+
--- a/ansible/playbooks/test-prerequisites.yml
+++ b/ansible/playbooks/test-prerequisites.yml
@@ -0,0 +1,8 @@
+---
+- name: Test Kubernetes Prerequisites Role
+  hosts: space-mountain
+  become: true
+  gather_facts: true
+  roles:
+    - role: kubernetes-prerequisites
+
--- a/ansible/playbooks/test.yml
+++ b/ansible/playbooks/test.yml
@@ -1,27 +1,25 @@
 ---
- name: Interactively configure an OpenShift cluster
-  # Run this play on the controller to orchestrate the changes
-  hosts: localhost
-  gather_facts: false
-
-  vars_prompt:
-    - name: cluster_group
-      prompt: "What is the name of the Cluster Group?"
-      private: false
+- name: Test Ansible Playbook
+  hosts: all
+  gather_facts: yes
+  become: false

  tasks:
-    - name: Create DNS entry for each cluster node
-      # Loop over every hostname in the user-provided group
-      loop: "{{ groups[cluster_group] }}"
-      loop_control:
-        # Use a descriptive name for the loop variable instead of 'item'
-        loop_var: node_hostname
+    - name: Ping all hosts
+      ansible.builtin.ping:

+    - name: Display hostname
      ansible.builtin.debug:
-        msg: "Using {{ item, dns_address }}"
+        msg: "The hostname of this system is {{ inventory_hostname }}"

-#      ansible.builtin.include_tasks: tasks/create_dns_record.yml
-      vars:
-        # Access the 'ip_address' variable of the specific host in the loop
-        dns_address: "{{ hostvars[node_hostname]['ip_address'] }}"
-        dns_name: "{{ hostvars[node_hostname] }}"
+    - name: Show OS distribution
+      ansible.builtin.debug:
+        msg: "This host is running {{ ansible_distribution }} {{ ansible_distribution_version }}"
+
+    - name: Run uptime command
+      ansible.builtin.command: uptime
+      register: uptime_result
+
+    - name: Display uptime result
+      ansible.builtin.debug:
+        var: uptime_result.stdout
--- a/ansible/roles/fastpass-kubernetes/tasks/container-runtime.yml
+++ b/ansible/roles/fastpass-kubernetes/tasks/container-runtime.yml
@@ -0,0 +1,72 @@
+---
+# Container runtime setup for Kubernetes
+
+- name: Add containerd repository
+  become: true
+  ansible.builtin.yum_repository:
+    name: docker-ce
+    description: "Docker CE Repository"
+    baseurl: "https://download.docker.com/linux/fedora/{{ ansible_distribution_major_version }}/x86_64/stable"
+    gpgkey: "https://download.docker.com/linux/fedora/gpg"
+    gpgcheck: true
+    enabled: true
+    state: present
+
+- name: Install containerd
+  become: true
+  ansible.builtin.dnf:
+    name: containerd
+    state: present
+    disable_excludes: docker-ce-stable
+
+- name: Create containerd config directory
+  become: true
+  ansible.builtin.file:
+    path: /etc/containerd
+    state: directory
+    mode: '0755'
+
+- name: Generate default containerd config
+  become: true
+  ansible.builtin.shell: |
+    containerd config default > /etc/containerd/config.toml
+  args:
+    creates: /etc/containerd/config.toml
+
+- name: Configure containerd for Kubernetes
+  become: true
+  ansible.builtin.replace:
+    path: /etc/containerd/config.toml
+    regexp: 'SystemdCgroup = false'
+    replace: 'SystemdCgroup = true'
+
+- name: Create containerd service override directory
+  become: true
+  ansible.builtin.file:
+    path: /etc/systemd/system/containerd.service.d
+    state: directory
+    mode: '0755'
+
+- name: Configure containerd service override
+  become: true
+  ansible.builtin.copy:
+    dest: /etc/systemd/system/containerd.service.d/override.conf
+    content: |
+      [Service]
+      ExecStartPre=-/sbin/modprobe overlay
+      ExecStartPre=-/sbin/modprobe br_netfilter
+    mode: '0644'
+
+- name: Restart and enable containerd service
+  become: true
+  ansible.builtin.systemd:
+    name: containerd
+    state: restarted
+    enabled: true
+    daemon_reload: true
+
+- name: Verify containerd is running
+  become: true
+  ansible.builtin.systemd:
+    name: containerd
+    state: started 
--- a/ansible/roles/fastpass-kubernetes/tasks/main.yml
+++ b/ansible/roles/fastpass-kubernetes/tasks/main.yml
@@ -0,0 +1,16 @@
+---
+# FastPass Kubernetes Cluster Preparation for Fedora
+# This role prepares all nodes for Kubernetes installation
+
+- name: Include preflight checks
+  ansible.builtin.include_tasks: preflight.yml
+  when: preflight_checks | default(true)
+
+- name: Include system preparation
+  ansible.builtin.include_tasks: system-prep.yml
+
+- name: Include container runtime setup
+  ansible.builtin.include_tasks: container-runtime.yml
+
+- name: Include Kubernetes installation
+  ansible.builtin.include_tasks: kubernetes-install.yml 
--- a/ansible/roles/fastpass-kubernetes/tasks/preflight.yml
+++ b/ansible/roles/fastpass-kubernetes/tasks/preflight.yml
@@ -0,0 +1,40 @@
+---
+# Preflight checks for Kubernetes installation
+
+- name: Check if running on supported OS
+  ansible.builtin.assert:
+    that:
+      - ansible_os_family == "RedHat"
+      - ansible_distribution == "Fedora"
+    fail_msg: "This role only supports Fedora"
+    success_msg: "OS check passed"
+
+- name: Check minimum memory requirement (2GB)
+  ansible.builtin.assert:
+    that: ansible_memtotal_mb >= 2048
+    fail_msg: "Minimum 2GB RAM required, found {{ ansible_memtotal_mb }}MB"
+    success_msg: "Memory check passed: {{ ansible_memtotal_mb }}MB"
+
+- name: Check minimum CPU cores (2)
+  ansible.builtin.assert:
+    that: ansible_processor_cores >= 2
+    fail_msg: "Minimum 2 CPU cores required, found {{ ansible_processor_cores }}"
+    success_msg: "CPU check passed: {{ ansible_processor_cores }} cores"
+
+- name: Check available disk space (10GB)
+  ansible.builtin.assert:
+    that: ansible_mounts | selectattr('mount', 'equalto', '/') | map(attribute='size_available') | first >= 10737418240
+    fail_msg: "Minimum 10GB free space required on root filesystem"
+    success_msg: "Disk space check passed"
+
+- name: Check if system is 64-bit
+  ansible.builtin.assert:
+    that: ansible_architecture in ['x86_64', 'amd64']
+    fail_msg: "Only 64-bit architectures are supported"
+    success_msg: "Architecture check passed: {{ ansible_architecture }}"
+
+- name: Check if running as root or with sudo
+  ansible.builtin.assert:
+    that: ansible_become | default(false)
+    fail_msg: "This role requires root privileges (become: true)"
+    success_msg: "Privilege check passed" 
--- a/ansible/roles/fastpass-kubernetes/tasks/system-prep.yml
+++ b/ansible/roles/fastpass-kubernetes/tasks/system-prep.yml
@@ -0,0 +1,83 @@
+---
+# System preparation for Kubernetes
+
+- name: Disable swap
+  ansible.builtin.command: swapoff -a
+  become: true
+  changed_when: false
+  when: disable_swap | default(true)
+
+- name: Persist swap off by commenting out swap in fstab
+  become: true
+  ansible.builtin.replace:
+    path: /etc/fstab
+    regexp: '^(\s*)([^#\n]+\s+swap\s+.*)$'
+    replace: '#\2'
+    backup: true
+  when: disable_swap | default(true)
+
+- name: Load required kernel modules
+  become: true
+  community.general.modprobe:
+    name: "{{ item }}"
+    state: present
+  loop:
+    - overlay
+    - br_netfilter
+
+- name: Persist kernel modules on boot
+  become: true
+  ansible.builtin.copy:
+    dest: /etc/modules-load.d/k8s.conf
+    content: |
+      overlay
+      br_netfilter
+    mode: '0644'
+
+- name: Configure required sysctl params for Kubernetes
+  become: true
+  ansible.builtin.copy:
+    dest: /etc/sysctl.d/k8s.conf
+    content: |
+      net.bridge.bridge-nf-call-iptables  = 1
+      net.bridge.bridge-nf-call-ip6tables = 1
+      net.ipv4.ip_forward                 = 1
+    mode: '0644'
+
+- name: Apply sysctl params without reboot
+  become: true
+  ansible.builtin.command: sysctl --system
+  changed_when: false
+
+- name: Install required packages
+  become: true
+  ansible.builtin.dnf:
+    name:
+      - python3-libselinux
+      - dnf-plugins-core
+      - curl
+      - wget
+      - ca-certificates
+    state: present
+
+- name: Set SELinux to permissive mode
+  become: true
+  ansible.posix.selinux:
+    policy: targeted
+    state: "{{ selinux_mode | default('permissive') }}"
+
+- name: Stop and disable firewalld
+  become: true
+  ansible.builtin.systemd:
+    name: firewalld
+    state: stopped
+    enabled: false
+  when: not firewall_enabled | default(false)
+
+- name: Update system packages
+  become: true
+  ansible.builtin.dnf:
+    name: "*"
+    state: latest
+  async: 300
+  poll: 10 
--- a/ansible/scripts/test-fastpass-deployment.sh
+++ b/ansible/scripts/test-fastpass-deployment.sh
@@ -0,0 +1,210 @@
+#!/bin/bash
+
+# FastPass Kubernetes Cluster Deployment Test Script
+# This script validates the Ansible playbooks before deployment
+
+set -e
+
+echo "🧪 FastPass Kubernetes Cluster Deployment Test"
+echo "=============================================="
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m' # No Color
+
+# Function to print colored output
+print_status() {
+    local status=$1
+    local message=$2
+    case $status in
+        "PASS")
+            echo -e "${GREEN}✅ PASS${NC}: $message"
+            ;;
+        "FAIL")
+            echo -e "${RED}❌ FAIL${NC}: $message"
+            ;;
+        "WARN")
+            echo -e "${YELLOW}⚠️  WARN${NC}: $message"
+            ;;
+    esac
+}
+
+# Test 1: Check if Ansible is installed
+echo ""
+echo "1. Checking Ansible installation..."
+if command -v ansible-playbook &> /dev/null; then
+    ANSIBLE_VERSION=$(ansible-playbook --version | head -n1)
+    print_status "PASS" "Ansible found: $ANSIBLE_VERSION"
+else
+    print_status "FAIL" "Ansible not found. Please install Ansible."
+    exit 1
+fi
+
+# Test 2: Check inventory file
+echo ""
+echo "2. Checking inventory file..."
+if [ -f "inventory.yml" ]; then
+    print_status "PASS" "Inventory file found: inventory.yml"
+    
+    # Check if fastpass groups exist
+    if grep -q "fastpass_control_plane:" inventory.yml; then
+        print_status "PASS" "fastpass_control_plane group found"
+    else
+        print_status "FAIL" "fastpass_control_plane group not found in inventory"
+    fi
+    
+    if grep -q "fastpass_workers:" inventory.yml; then
+        print_status "PASS" "fastpass_workers group found"
+    else
+        print_status "FAIL" "fastpass_workers group not found in inventory"
+    fi
+else
+    print_status "FAIL" "Inventory file not found: inventory.yml"
+    exit 1
+fi
+
+# Test 3: Check playbook files
+echo ""
+echo "3. Checking playbook files..."
+if [ -f "playbooks/deploy-fastpass-cluster.yml" ]; then
+    print_status "PASS" "Main deployment playbook found"
+else
+    print_status "FAIL" "Main deployment playbook not found"
+fi
+
+if [ -f "playbooks/deploy_k8s.yml" ]; then
+    print_status "PASS" "Legacy deployment playbook found"
+else
+    print_status "WARN" "Legacy deployment playbook not found"
+fi
+
+# Test 4: Check role files
+echo ""
+echo "4. Checking role files..."
+ROLES=(
+    "playbooks/roles/kubernetes/tasks/main.yml"
+    "playbooks/roles/fastpass-control-plane/tasks/main.yml"
+    "playbooks/roles/fastpass-control-plane-join/tasks/main.yml"
+    "playbooks/roles/fastpass-workers/tasks/main.yml"
+    "playbooks/roles/fastpass-workers/tasks/node-labels.yml"
+)
+
+for role in "${ROLES[@]}"; do
+    if [ -f "$role" ]; then
+        print_status "PASS" "Role file found: $role"
+    else
+        print_status "FAIL" "Role file not found: $role"
+    fi
+done
+
+# Test 5: Check group_vars
+echo ""
+echo "5. Checking group variables..."
+if [ -f "group_vars/fastpass/vars" ]; then
+    print_status "PASS" "FastPass group variables found"
+    
+    # Check for required variables
+    if grep -q "pod_network_cidr:" group_vars/fastpass/vars; then
+        print_status "PASS" "pod_network_cidr variable found"
+    else
+        print_status "WARN" "pod_network_cidr variable not found"
+    fi
+    
+    if grep -q "control_plane_endpoint:" group_vars/fastpass/vars; then
+        print_status "PASS" "control_plane_endpoint variable found"
+    else
+        print_status "WARN" "control_plane_endpoint variable not found"
+    fi
+else
+    print_status "FAIL" "FastPass group variables not found"
+fi
+
+# Test 6: Validate playbook syntax
+echo ""
+echo "6. Validating playbook syntax..."
+if ansible-playbook --syntax-check playbooks/deploy-fastpass-cluster.yml > /dev/null 2>&1; then
+    print_status "PASS" "Main playbook syntax is valid"
+else
+    print_status "FAIL" "Main playbook syntax is invalid"
+    echo "Running syntax check with verbose output:"
+    ansible-playbook --syntax-check playbooks/deploy-fastpass-cluster.yml
+fi
+
+# Test 7: Check for common issues
+echo ""
+echo "7. Checking for common issues..."
+
+# Check for hardcoded values
+if grep -r "10.244.0.0/16" playbooks/ | grep -v "group_vars" > /dev/null; then
+    print_status "WARN" "Hardcoded pod network CIDR found in playbooks"
+else
+    print_status "PASS" "No hardcoded pod network CIDR found"
+fi
+
+# Check for proper kubeconfig usage
+if grep -r "kubectl --kubeconfig" playbooks/ > /dev/null; then
+    print_status "PASS" "Proper kubeconfig usage found"
+else
+    print_status "WARN" "No explicit kubeconfig usage found"
+fi
+
+# Test 8: DNS validation
+echo ""
+echo "8. Validating DNS setup..."
+CONTROL_PLANE_ENDPOINT="fastpass.local.mk-labs.cloud"
+if nslookup "$CONTROL_PLANE_ENDPOINT" > /dev/null 2>&1; then
+    print_status "PASS" "DNS resolution successful for $CONTROL_PLANE_ENDPOINT"
+    
+    # Get A records
+    A_RECORDS=$(nslookup "$CONTROL_PLANE_ENDPOINT" | grep -A 10 "Name:" | grep "Address:" | awk '{print $2}' | sort)
+    RECORD_COUNT=$(echo "$A_RECORDS" | wc -l)
+    
+    if [ "$RECORD_COUNT" -ge 1 ]; then
+        print_status "PASS" "Found $RECORD_COUNT A record(s) for load balancing"
+        echo "   DNS records:"
+        echo "$A_RECORDS" | while read -r ip; do
+            echo "   - $ip"
+        done
+    else
+        print_status "WARN" "No A records found for $CONTROL_PLANE_ENDPOINT"
+    fi
+else
+    print_status "FAIL" "DNS resolution failed for $CONTROL_PLANE_ENDPOINT"
+    echo "   Please run: ./scripts/validate-dns-setup.sh for detailed DNS validation"
+fi
+
+# Test 9: Check SSH connectivity (if hosts are available)
+echo ""
+echo "9. Checking SSH connectivity..."
+if [ -n "$1" ]; then
+    echo "Testing SSH connectivity to specified host: $1"
+    if ssh -o ConnectTimeout=5 -o BatchMode=yes "$1" exit 2>/dev/null; then
+        print_status "PASS" "SSH connectivity to $1 successful"
+    else
+        print_status "FAIL" "SSH connectivity to $1 failed"
+    fi
+else
+    print_status "WARN" "No host specified for SSH test. Use: $0 <hostname>"
+fi
+
+# Summary
+echo ""
+echo "=============================================="
+echo "🎯 Deployment Test Summary"
+echo "=============================================="
+echo ""
+echo "To deploy the FastPass cluster:"
+echo "1. Ensure all tests above pass"
+echo "2. Run: ansible-playbook -i inventory.yml playbooks/deploy-fastpass-cluster.yml"
+echo ""
+echo "To test with a specific host:"
+echo "   $0 <hostname>"
+echo ""
+echo "To validate DNS setup:"
+echo "   ./scripts/validate-dns-setup.sh"
+echo ""
+echo "To validate the deployment:"
+echo "   ./scripts/manage-kubeconfigs.sh test"
+echo "" 
--- a/infra-config/04-vm-templates/fedora-42/proxmox.pkvars.hcl
+++ b/infra-config/04-vm-templates/fedora-42/proxmox.pkvars.hcl
@@ -10,7 +10,7 @@ ssh_username       = "wed"
 # Proxmox Objects
 proxmox_node                    = "fantasyland"
 proxmox_domain                  = "local.mk-labs.cloud"
-proxmox_vm_storage_pool         = "mk-templates"
+proxmox_vm_storage_pool         = "templates"
 #proxmox_insecure_connection     = false #Default: true
 proxmox_iso_storage_pool        = "local"