homelab/docs/astro-orbiter-gpu-passthrough.md

# astro-orbiter: AMD GPU Passthrough & Ollama/ROCm Setup

**Host:** `tomorrowland` (10.1.71.12) — Proxmox node  
**VM:** `astro-orbiter` (10.1.71.130, VM ID 71130)  
**GPU:** MSI Radeon RX 5700 (Navi 10, gfx1010) via Minisforum DEG1 eGPU dock + OCuLink  
**Purpose:** Ollama inference host for `jarvis` AI assistant  
**Date:** 2026-05-25

---

## Hardware

- Minisforum MS-01 (`tomorrowland`)
- Minisforum DEG1 eGPU dock connected via M.2 OCuLink
- MSI Radeon RX 5700 8GB
- Montech BETA 2 650W ATX PSU powering the eGPU dock

---

## Part 1: tomorrowland Host Setup

### 1.1 Network Interface Rename (eGPU side effect)

Adding the eGPU dock shifted PCIe bus numbers, renaming all NICs. Fix:

```bash
# Back up interfaces config
cp /etc/network/interfaces /etc/network/interfaces.bak

# Apply renames (adjust if hardware differs)
sed -i 's/enp87s0/enp93s0/g' /etc/network/interfaces
sed -i 's/enp2s0f0np0/enp5s0f0np0/g' /etc/network/interfaces
sed -i 's/enp2s0f0np1/enp5s0f0np1/g' /etc/network/interfaces

# Reboot to apply (do from console/JetKVM, not SSH)
reboot
```

### 1.2 Verify GPU Enumeration

After reboot, confirm the GPU is visible:

```bash
lspci | grep -i amd
# Expected:
# 01:00.0 PCI bridge: AMD/ATI Navi 10 XL Upstream Port
# 02:00.0 PCI bridge: AMD/ATI Navi 10 XL Downstream Port
# 03:00.0 VGA compatible controller: AMD/ATI Navi 10 [Radeon RX 5600/5700]
# 03:00.1 Audio device: AMD/ATI Navi 10 HDMI Audio
```

### 1.3 Verify IOMMU

```bash
dmesg | grep -i iommu | head -5
# Must show: iommu: Default domain type: Translated

# Verify GPU is in its own IOMMU groups (no ACS override needed)
find /sys/kernel/iommu_groups/*/devices/ -name "*03:00*"
# Expected: groups 17 and 18, one device each
```

### 1.4 Blacklist amdgpu and Bind vfio-pci

```bash
# Blacklist host GPU drivers
echo "blacklist amdgpu
blacklist radeon" > /etc/modprobe.d/blacklist-amdgpu.conf

# Bind vfio-pci to the GPU at boot
# 1002:731f = RX 5700 VGA, 1002:ab38 = Navi 10 HDMI Audio
echo "options vfio-pci ids=1002:731f,1002:ab38" > /etc/modprobe.d/vfio.conf

# Load vfio modules early
echo "vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd" > /etc/modules-load.d/vfio.conf

# Rebuild initramfs and reboot
update-initramfs -u -k all
reboot
```

Verify after reboot:
```bash
lspci -k -s 03:00.0
# Must show: Kernel driver in use: vfio-pci

lspci -k -s 03:00.1
# Must show: Kernel driver in use: vfio-pci
```

### 1.5 Fix GPU D3 Power State on VM Restart (Required — RX 5700 lacks FLR)

The RX 5700 does not support PCIe Function Level Reset (FLR). Without a fix,
the GPU gets stuck in D3 power state on every VM restart, requiring a full host
reboot to recover.

**Fix: `pcie_port_pm=off` kernel parameter**

`vendor-reset` was attempted but causes a QEMU PCI IRQ assertion failure
(`pci_irq_handler: Assertion 0 <= irq_num`) on PVE 7.0.2 with this hardware.
`pcie_port_pm=off` prevents the card from entering D3 entirely and is the
correct solution for this setup.

```bash
sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="/GRUB_CMDLINE_LINUX_DEFAULT="pcie_port_pm=off /' /etc/default/grub
update-grub
reboot
```

Verify after reboot — VM should restart cleanly without host reboot:
```bash
qm reboot 71130
# wait 30 seconds
ssh wed@astro-orbiter 'lsmod | grep amdgpu'
# Should be non-empty
```

---

## Part 2: astro-orbiter VM Provisioning

### 2.1 Terraform Provisioning

VM was provisioned manually (not via NetBox pipeline) due to PCIe passthrough
requirements. Run from `city-hall`:

```bash
cd /opt/git/homelab/terraform/proxmox/vm

terraform apply \
  -var='proxmox_api_url=https://fantasyland.local.mk-labs.cloud:8006' \
  -var='proxmox_api_token=terraform@pve!terraform-token=<secret>' \
  -var='vm_id=71130' \
  -var='hostname=astro-orbiter' \
  -var='ip_address=10.1.71.130' \
  -var='target_node=tomorrowland' \
  -var='template_name=ubuntu-24.04-large' \
  -var='datastore=liberty-tree' \
  -var='data_disk_enabled=true' \
  -var='data_disk_size=128' \
  -var='data_disk_datastore=utilidor'
```

**Notes:**
- `datastore=liberty-tree` for OS disk — iSCSI storage does not support EFI disk
  cloning from templates. OS disk stays on Ceph.
- `data_disk_datastore=utilidor` for model storage — fresh disk creation works fine
  on iSCSI.
- `pcie_passthrough_enabled` is NOT passed — the `bpg/proxmox` provider's
  `dynamic "hostpci"` block does not work reliably. PCIe passthrough is applied
  manually via `qm set` after provisioning (see 2.2).

### 2.2 Post-Provisioning: PCIe and Machine Type

The template uses `i440fx` machine type. PCIe passthrough requires `q35`.
Run after Terraform provisioning, with VM shut down:

```bash
# Switch to q35 machine type (required for PCIe passthrough)
qm set 71130 -machine q35

# Add GPU passthrough (both VGA and audio functions)
qm set 71130 -hostpci0 0000:03:00,pcie=1,rombar=1

# Add data disk on utilidor
qm set 71130 -scsi1 utilidor:128,format=raw,discard=on

# Increase RAM to 8GB
qm set 71130 -memory 8192

# Start the VM
qm start 71130
```

Verify config landed:
```bash
qm config 71130 | grep -E "hostpci|machine|memory|scsi1"
```

### 2.3 Ansible Deployment

From `city-hall` in the `ansible/` directory:

```bash
# OS baseline
ansible-playbook playbooks/day0_baseline.yml --limit astro-orbiter --ask-vault-pass

# Ollama + ROCm
ansible-playbook playbooks/day1_deploy_ollama.yml --ask-vault-pass
```

The `day1_deploy_ollama.yml` playbook includes a pre-flight `lspci` check that
will fail fast if the GPU is not visible to the VM.

---

## Part 3: Verification

### 3.1 GPU visible in VM
```bash
lspci | grep -i amd
# Expected: 01:00.0 VGA ... Navi 10 [Radeon RX 5600/5700]
#           01:00.1 Audio ... Navi 10 HDMI Audio
```

### 3.2 amdgpu driver loaded
```bash
lsmod | grep amdgpu
# Should show amdgpu and dependent modules
```

### 3.3 ROCm sees the GPU
```bash
/opt/rocm/bin/rocm-smi
# Should show Device 0 with VRAM%, GPU%, temperature
```

### 3.4 Ollama GPU inference
```bash
ollama run qwen3:8b "say hello"
# While running, check rocm-smi — GPU% and power draw should spike
/opt/rocm/bin/rocm-smi
# GPU% > 0%, power > 18W idle = GPU is doing inference
```

### 3.5 VM restart test
```bash
# On tomorrowland:
qm reboot 71130
# Wait 30 seconds, then on astro-orbiter:
lsmod | grep amdgpu
# Should be non-empty — if empty, vendor-reset is not working
```

---

## Known Issues & Lessons Learned

| Issue | Cause | Fix |
|---|---|---|
| NIC names changed after eGPU install | PCIe bus renumbering | Update `/etc/network/interfaces` with new names |
| `can't allocate space in iscsi storage` on clone | iSCSI doesn't support EFI disk cloning | Use `liberty-tree` for OS disk |
| `q35 machine model is not enabled` on hostpci | Template built with i440fx | `qm set <vmid> -machine q35` before adding hostpci |
| GPU stuck in D3 on VM restart | RX 5700 lacks FLR support | `pcie_port_pm=off` kernel param on host. `vendor-reset` builds but causes QEMU PCI IRQ assertion failure on PVE 7.0.2 — do not use. |
| `dynamic "hostpci"` block not applying | `bpg/proxmox` provider limitation | Use `qm set` workaround post-provisioning |
| `rocm-smi` not in PATH | ROCm installs to `/opt/rocm-6.2.0/bin/` | `/etc/profile.d/rocm.sh` sets PATH (via Ansible role) |
| Ollama install script 404 | `OLLAMA_VERSION=latest` is not a valid version string | Omit env var when version is `latest` — script self-determines |

---

## ROCm Configuration

The RX 5700 is RDNA 1 (gfx1010). ROCm's official support starts at gfx1030
(RDNA 2). The `HSA_OVERRIDE_GFX_VERSION=10.1.0` environment variable tells ROCm
to treat the GPU as a supported target.

This is set via systemd drop-in at:
`/etc/systemd/system/ollama.service.d/override.conf`

```ini
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_MODELS=/var/lib/ollama"
Environment="HSA_OVERRIDE_GFX_VERSION=10.1.0"
Environment="ROCR_VISIBLE_DEVICES=0"
```

---

## Parking Lot

- **P.PACKER:** Ubuntu VM templates should be built with `machine: q35` by default
  to support PCIe passthrough without manual intervention.
- **P.TERRAFORM:** Fix `bpg/proxmox` provider `hostpci` dynamic block or find
  correct syntax for the installed provider version.
- **P.PIPELINE:** Update n8n workflow to pass `data_disk_enabled`, `data_disk_size`,
  and `pcie_device_id` from NetBox custom fields to Terraform for pipeline-driven
  GPU VM provisioning.