homelab/talos/UPGRADES-AND-EXTENSIONS.md

Talos Linux Upgrades and System Extensions

Guide for upgrading Talos versions and adding system extensions to the fastpass cluster.

Table of Contents

• System Extensions
• Upgrading Talos Versions
• Rolling Upgrades (Workers)
• Troubleshooting

---

System Extensions

System extensions in Talos v1.13+ are baked into custom boot images via schematics and the Talos Image Factory, not loaded at runtime.

Adding Extensions

Edit talhelper/talconfig.yaml and add extensions to the worker or control plane section:

# ─── Worker patches ──────────────────────────────────────────────────────────
worker:
  schematic:
    customization:
      systemExtensions:
        officialExtensions:
          - siderolabs/iscsi-tools
          - siderolabs/util-linux-tools
  patches:
    - |-
      machine:
        kernel:
          modules:
            - name: iscsi_tcp
            - name: dm_multipath

Common extensions:

• siderolabs/iscsi-tools - iSCSI initiator utilities
• siderolabs/util-linux-tools - Multipath and storage utilities
• siderolabs/qemu-guest-agent - Proxmox/QEMU guest agent
• siderolabs/nvidia-container-toolkit - NVIDIA GPU support

Browse available extensions:

curl -s https://factory.talos.dev/schematics | jq '.extensions[] | .name'

Applying Extensions

Extensions require an upgrade (new OS image), not apply-config:

cd /opt/git/homelab/talos/talhelper

# 1. Generate configs with schematic
talhelper genconfig

# 2. Generate upgrade commands
talhelper gencommand upgrade --extra-flags="--preserve"

# 3. Copy the output commands and run them one node at a time
# Example output:
# talosctl upgrade --nodes 10.1.71.69 --image factory.talos.dev/installer/<schematic>:v1.13.2 --preserve

Manual rolling upgrade for workers:

# Worker 1
talosctl upgrade --nodes 10.1.71.69 \
  --image factory.talos.dev/installer/<schematic-id>:v1.13.2 \
  --preserve
kubectl wait --for=condition=Ready node/jungle-cruise --timeout=10m
sleep 30

# Worker 2
talosctl upgrade --nodes 10.1.71.70 \
  --image factory.talos.dev/installer/<schematic-id>:v1.13.2 \
  --preserve
kubectl wait --for=condition=Ready node/haunted-mansion --timeout=10m
sleep 30

# Worker 3
talosctl upgrade --nodes 10.1.71.71 \
  --image factory.talos.dev/installer/<schematic-id>:v1.13.2 \
  --preserve
kubectl wait --for=condition=Ready node/peter-pans-flight --timeout=10m

Verify extensions:

# List installed extensions
talosctl -n 10.1.71.69 get extensions

# Verify specific modules
talosctl -n 10.1.71.69 read /proc/modules | grep iscsi

---

Upgrading Talos Versions

Pre-Upgrade Checklist

1. Review release notes: https://github.com/siderolabs/talos/releases
2. Check Kubernetes compatibility: Ensure your K8s version is supported
3. Backup etcd: Control plane data
4. Take VM snapshots: If running on Proxmox/vSphere
5. Check extension compatibility: Ensure extensions exist for new Talos version

Upgrade Process

Step 1: Update talconfig.yaml

clusterName: fastpass
talosVersion: v1.14.0  # <- Update this
kubernetesVersion: v1.32.3

Step 2: Regenerate Configs

cd /opt/git/homelab/talos/talhelper
talhelper genconfig

Step 3: Upgrade Control Plane (One at a Time)

# Generate upgrade commands
talhelper gencommand upgrade --extra-flags="--preserve"

# Control plane node 1
talosctl upgrade --nodes 10.1.71.66 \
  --image factory.talos.dev/installer/<schematic>:v1.14.0 \
  --preserve
kubectl wait --for=condition=Ready node/space-mountain --timeout=10m
sleep 60

# Control plane node 2
talosctl upgrade --nodes 10.1.71.67 \
  --image factory.talos.dev/installer/<schematic>:v1.14.0 \
  --preserve
kubectl wait --for=condition=Ready node/big-thunder-mountain --timeout=10m
sleep 60

# Control plane node 3
talosctl upgrade --nodes 10.1.71.68 \
  --image factory.talos.dev/installer/<schematic>:v1.14.0 \
  --preserve
kubectl wait --for=condition=Ready node/splash-mountain --timeout=10m
sleep 60

IMPORTANT: Always upgrade control plane nodes one at a time with sufficient wait between each. Etcd requires quorum (2 of 3 nodes) to remain operational.

Step 4: Upgrade Workers (Rolling)

After all control plane nodes are upgraded:

# Worker nodes (can be faster since no etcd)
talosctl upgrade --nodes 10.1.71.69 \
  --image factory.talos.dev/installer/<schematic>:v1.14.0 \
  --preserve
kubectl wait --for=condition=Ready node/jungle-cruise --timeout=10m
sleep 30

talosctl upgrade --nodes 10.1.71.70 \
  --image factory.talos.dev/installer/<schematic>:v1.14.0 \
  --preserve
kubectl wait --for=condition=Ready node/haunted-mansion --timeout=10m
sleep 30

talosctl upgrade --nodes 10.1.71.71 \
  --image factory.talos.dev/installer/<schematic>:v1.14.0 \
  --preserve
kubectl wait --for=condition=Ready node/peter-pans-flight --timeout=10m

Step 5: Verify Cluster

# Check all nodes
kubectl get nodes -o wide

# Verify Talos version
talosctl -n 10.1.71.66,10.1.71.67,10.1.71.68 version

# Check cluster health
kubectl get pods -A | grep -v Running
talosctl -n 10.1.71.66 service kubelet status

---

Rolling Upgrades (Workers)

Automated Rolling Upgrade Script

#!/bin/bash
# rolling-upgrade-workers.sh

set -e

IMAGE="$1"
if [ -z "$IMAGE" ]; then
  echo "Usage: $0 <image-url>"
  echo "Example: $0 factory.talos.dev/installer/abc123:v1.14.0"
  exit 1
fi

declare -A WORKERS=(
  ["jungle-cruise"]="10.1.71.69"
  ["haunted-mansion"]="10.1.71.70"
  ["peter-pans-flight"]="10.1.71.71"
)

for worker in "${!WORKERS[@]}"; do
  ip="${WORKERS[$worker]}"
  echo "=========================================="
  echo "Upgrading $worker ($ip)..."
  echo "=========================================="
  
  talosctl upgrade --nodes ${ip} --image ${IMAGE} --preserve
  
  echo "Waiting for $worker to be Ready..."
  kubectl wait --for=condition=Ready node/${worker} --timeout=10m
  
  echo "$worker upgrade complete. Sleeping 30s before next node..."
  sleep 30
done

echo "=========================================="
echo "All workers upgraded successfully!"
echo "=========================================="

Usage:

cd /opt/git/homelab/talos/talhelper
chmod +x rolling-upgrade-workers.sh

# Get the image URL from talhelper
IMAGE=$(grep "image:" clusterconfig/fastpass-jungle-cruise.yaml | head -1 | awk '{print $2}')

# Run the rolling upgrade
./rolling-upgrade-workers.sh "$IMAGE"

---

Upgrading Kubernetes Version

Talos manages Kubernetes. To upgrade K8s:

Step 1: Update talconfig.yaml

clusterName: fastpass
talosVersion: v1.14.0
kubernetesVersion: v1.33.0  # <- Update this

Step 2: Regenerate and Apply

cd /opt/git/homelab/talos/talhelper
talhelper genconfig

# Apply config changes (does NOT require node reboot)
talosctl -n 10.1.71.66,10.1.71.67,10.1.71.68 upgrade-k8s --to v1.33.0

The upgrade-k8s command:

• Upgrades control plane components (apiserver, scheduler, controller-manager) first
• Then upgrades kubelet on all nodes
• Does NOT reboot nodes (in-place upgrade)
• Automatic rolling update

Verify:

kubectl get nodes -o wide
# Check both Talos version and Kubernetes version columns

---

Troubleshooting

Extension Not Found Error

Error: module not found when loading kernel modules

Cause: Module isn't in the base Talos image or extension wasn't applied

Fix:

1. Check if extension exists: curl -s https://factory.talos.dev/schematics | jq '.extensions[] | select(.name | contains("iscsi"))'
2. Verify schematic in config: grep -A10 "schematic:" talhelper/talconfig.yaml
3. Confirm extension is loaded: talosctl -n <node-ip> get extensions

Upgrade Fails with "invalid reference format"

Error: failed to parse image reference

Cause: Incorrect image URL format or missing schematic

Fix:

1. Manually check generated image URL: grep "image:" clusterconfig/fastpass-<node>.yaml
2. Use talhelper's upgrade command generator: talhelper gencommand upgrade
3. Ensure format is: factory.talos.dev/installer/<schematic-id>:<version>

Node Stuck in "NotReady" After Upgrade

Symptoms: Node shows NotReady for >5 minutes post-upgrade

Debug:

# Check node status
kubectl describe node <node-name>

# Check Talos services
talosctl -n <node-ip> service kubelet status
talosctl -n <node-ip> service containerd status

# Check logs
talosctl -n <node-ip> logs kubelet
talosctl -n <node-ip> dmesg | tail -50

Common causes:

• Network policy blocking CNI (Cilium)
• iSCSI/storage mounts failing
• Kernel module load failure
• Certificate rotation issues

Control Plane Quorum Lost

Symptoms: etcdserver: no leader errors, apiserver unavailable

Prevention:

• NEVER upgrade more than 1 control plane node simultaneously
• Always wait for full Ready status + 60s between CP upgrades
• Verify etcd health before proceeding: talosctl -n 10.1.71.66 service etcd status

Recovery: If quorum is lost (2 of 3 nodes down):

# Wait for at least 2 nodes to come back online
# Check etcd cluster status
talosctl -n 10.1.71.66,10.1.71.67,10.1.71.68 service etcd status

# If completely broken, may require etcd disaster recovery
# See: https://www.talos.dev/v1.13/advanced/disaster-recovery/

---

Quick Reference

Check Current Versions

# Talos version on all nodes
talosctl -n 10.1.71.66,10.1.71.67,10.1.71.68,10.1.71.69,10.1.71.70,10.1.71.71 version

# Kubernetes version
kubectl version

# Installed extensions
talosctl -n 10.1.71.69 get extensions

Safe Upgrade Order

1. ✅ Control plane node 1 → wait → verify
2. ✅ Control plane node 2 → wait → verify
3. ✅ Control plane node 3 → wait → verify
4. ✅ Worker nodes (can be parallel or rolling)

Config Changes vs. OS Upgrades

Change Type	Command	Reboot Required?
Network config, sysctls, files	talosctl apply-config --mode=reboot	Yes
Kubernetes version	talosctl upgrade-k8s	No
Talos version	talosctl upgrade	Yes
System extensions	talosctl upgrade	Yes

---

Links

• Talos Releases: https://github.com/siderolabs/talos/releases
• Talos Image Factory: https://factory.talos.dev
• Extension Catalog: https://github.com/siderolabs/extensions
• Upgrade Guide: https://www.talos.dev/v1.13/talos-guides/upgrading-talos/
• talhelper Docs: https://budimanjojo.github.io/talhelper/latest/