Files

Ryan Blundon be8e50d590 deploy hermes

2026-05-25 20:21:24 -05:00

8.2 KiB

Raw Blame History

astro-orbiter: AMD GPU Passthrough & Ollama/ROCm Setup

Host: tomorrowland (10.1.71.12) — Proxmox node
VM: astro-orbiter (10.1.71.130, VM ID 71130)
GPU: MSI Radeon RX 5700 (Navi 10, gfx1010) via Minisforum DEG1 eGPU dock + OCuLink
Purpose: Ollama inference host for jarvis AI assistant
Date: 2026-05-25

Hardware

Minisforum MS-01 (tomorrowland)
Minisforum DEG1 eGPU dock connected via M.2 OCuLink
MSI Radeon RX 5700 8GB
Montech BETA 2 650W ATX PSU powering the eGPU dock

Part 1: tomorrowland Host Setup

1.1 Network Interface Rename (eGPU side effect)

Adding the eGPU dock shifted PCIe bus numbers, renaming all NICs. Fix:

# Back up interfaces config
cp /etc/network/interfaces /etc/network/interfaces.bak

# Apply renames (adjust if hardware differs)
sed -i 's/enp87s0/enp93s0/g' /etc/network/interfaces
sed -i 's/enp2s0f0np0/enp5s0f0np0/g' /etc/network/interfaces
sed -i 's/enp2s0f0np1/enp5s0f0np1/g' /etc/network/interfaces

# Reboot to apply (do from console/JetKVM, not SSH)
reboot

1.2 Verify GPU Enumeration

After reboot, confirm the GPU is visible:

lspci | grep -i amd
# Expected:
# 01:00.0 PCI bridge: AMD/ATI Navi 10 XL Upstream Port
# 02:00.0 PCI bridge: AMD/ATI Navi 10 XL Downstream Port
# 03:00.0 VGA compatible controller: AMD/ATI Navi 10 [Radeon RX 5600/5700]
# 03:00.1 Audio device: AMD/ATI Navi 10 HDMI Audio

1.3 Verify IOMMU

dmesg | grep -i iommu | head -5
# Must show: iommu: Default domain type: Translated

# Verify GPU is in its own IOMMU groups (no ACS override needed)
find /sys/kernel/iommu_groups/*/devices/ -name "*03:00*"
# Expected: groups 17 and 18, one device each

1.4 Blacklist amdgpu and Bind vfio-pci

# Blacklist host GPU drivers
echo "blacklist amdgpu
blacklist radeon" > /etc/modprobe.d/blacklist-amdgpu.conf

# Bind vfio-pci to the GPU at boot
# 1002:731f = RX 5700 VGA, 1002:ab38 = Navi 10 HDMI Audio
echo "options vfio-pci ids=1002:731f,1002:ab38" > /etc/modprobe.d/vfio.conf

# Load vfio modules early
echo "vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd" > /etc/modules-load.d/vfio.conf

# Rebuild initramfs and reboot
update-initramfs -u -k all
reboot

Verify after reboot:

lspci -k -s 03:00.0
# Must show: Kernel driver in use: vfio-pci

lspci -k -s 03:00.1
# Must show: Kernel driver in use: vfio-pci

1.5 Fix GPU D3 Power State on VM Restart (Required — RX 5700 lacks FLR)

The RX 5700 does not support PCIe Function Level Reset (FLR). Without a fix, the GPU gets stuck in D3 power state on every VM restart, requiring a full host reboot to recover.

Fix: pcie_port_pm=off kernel parameter

vendor-reset was attempted but causes a QEMU PCI IRQ assertion failure (pci_irq_handler: Assertion 0 <= irq_num) on PVE 7.0.2 with this hardware. pcie_port_pm=off prevents the card from entering D3 entirely and is the correct solution for this setup.

sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="/GRUB_CMDLINE_LINUX_DEFAULT="pcie_port_pm=off /' /etc/default/grub
update-grub
reboot

Verify after reboot — VM should restart cleanly without host reboot:

qm reboot 71130
# wait 30 seconds
ssh wed@astro-orbiter 'lsmod | grep amdgpu'
# Should be non-empty

Part 2: astro-orbiter VM Provisioning

2.1 Terraform Provisioning

VM was provisioned manually (not via NetBox pipeline) due to PCIe passthrough requirements. Run from city-hall:

cd /opt/git/homelab/terraform/proxmox/vm

terraform apply \
  -var='proxmox_api_url=https://fantasyland.local.mk-labs.cloud:8006' \
  -var='proxmox_api_token=terraform@pve!terraform-token=<secret>' \
  -var='vm_id=71130' \
  -var='hostname=astro-orbiter' \
  -var='ip_address=10.1.71.130' \
  -var='target_node=tomorrowland' \
  -var='template_name=ubuntu-24.04-large' \
  -var='datastore=liberty-tree' \
  -var='data_disk_enabled=true' \
  -var='data_disk_size=128' \
  -var='data_disk_datastore=utilidor'

Notes:

datastore=liberty-tree for OS disk — iSCSI storage does not support EFI disk cloning from templates. OS disk stays on Ceph.
data_disk_datastore=utilidor for model storage — fresh disk creation works fine on iSCSI.
pcie_passthrough_enabled is NOT passed — the bpg/proxmox provider's dynamic "hostpci" block does not work reliably. PCIe passthrough is applied manually via qm set after provisioning (see 2.2).

2.2 Post-Provisioning: PCIe and Machine Type

The template uses i440fx machine type. PCIe passthrough requires q35. Run after Terraform provisioning, with VM shut down:

# Switch to q35 machine type (required for PCIe passthrough)
qm set 71130 -machine q35

# Add GPU passthrough (both VGA and audio functions)
qm set 71130 -hostpci0 0000:03:00,pcie=1,rombar=1

# Add data disk on utilidor
qm set 71130 -scsi1 utilidor:128,format=raw,discard=on

# Increase RAM to 8GB
qm set 71130 -memory 8192

# Start the VM
qm start 71130

Verify config landed:

qm config 71130 | grep -E "hostpci|machine|memory|scsi1"

2.3 Ansible Deployment

From city-hall in the ansible/ directory:

# OS baseline
ansible-playbook playbooks/day0_baseline.yml --limit astro-orbiter --ask-vault-pass

# Ollama + ROCm
ansible-playbook playbooks/day1_deploy_ollama.yml --ask-vault-pass

The day1_deploy_ollama.yml playbook includes a pre-flight lspci check that will fail fast if the GPU is not visible to the VM.

Part 3: Verification

3.1 GPU visible in VM

lspci | grep -i amd
# Expected: 01:00.0 VGA ... Navi 10 [Radeon RX 5600/5700]
#           01:00.1 Audio ... Navi 10 HDMI Audio

3.2 amdgpu driver loaded

lsmod | grep amdgpu
# Should show amdgpu and dependent modules

3.3 ROCm sees the GPU

/opt/rocm/bin/rocm-smi
# Should show Device 0 with VRAM%, GPU%, temperature

3.4 Ollama GPU inference

ollama run qwen3:8b "say hello"
# While running, check rocm-smi — GPU% and power draw should spike
/opt/rocm/bin/rocm-smi
# GPU% > 0%, power > 18W idle = GPU is doing inference

3.5 VM restart test

# On tomorrowland:
qm reboot 71130
# Wait 30 seconds, then on astro-orbiter:
lsmod | grep amdgpu
# Should be non-empty — if empty, vendor-reset is not working

Known Issues & Lessons Learned

Issue	Cause	Fix
NIC names changed after eGPU install	PCIe bus renumbering	Update `/etc/network/interfaces` with new names
`can't allocate space in iscsi storage` on clone	iSCSI doesn't support EFI disk cloning	Use `liberty-tree` for OS disk
`q35 machine model is not enabled` on hostpci	Template built with i440fx	`qm set <vmid> -machine q35` before adding hostpci
GPU stuck in D3 on VM restart	RX 5700 lacks FLR support	`pcie_port_pm=off` kernel param on host. `vendor-reset` builds but causes QEMU PCI IRQ assertion failure on PVE 7.0.2 — do not use.
`dynamic "hostpci"` block not applying	`bpg/proxmox` provider limitation	Use `qm set` workaround post-provisioning
`rocm-smi` not in PATH	ROCm installs to `/opt/rocm-6.2.0/bin/`	`/etc/profile.d/rocm.sh` sets PATH (via Ansible role)
Ollama install script 404	`OLLAMA_VERSION=latest` is not a valid version string	Omit env var when version is `latest` — script self-determines

ROCm Configuration

The RX 5700 is RDNA 1 (gfx1010). ROCm's official support starts at gfx1030 (RDNA 2). The HSA_OVERRIDE_GFX_VERSION=10.1.0 environment variable tells ROCm to treat the GPU as a supported target.

This is set via systemd drop-in at: /etc/systemd/system/ollama.service.d/override.conf

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_MODELS=/var/lib/ollama"
Environment="HSA_OVERRIDE_GFX_VERSION=10.1.0"
Environment="ROCR_VISIBLE_DEVICES=0"

Parking Lot

P.PACKER: Ubuntu VM templates should be built with machine: q35 by default to support PCIe passthrough without manual intervention.
P.TERRAFORM: Fix bpg/proxmox provider hostpci dynamic block or find correct syntax for the installed provider version.
P.PIPELINE: Update n8n workflow to pass data_disk_enabled, data_disk_size, and pcie_device_id from NetBox custom fields to Terraform for pipeline-driven GPU VM provisioning.

8.2 KiB Raw Blame History