deploy hermes

2026-05-25 20:21:24 -05:00
parent e309acd67d
commit be8e50d590
40 changed files with 2420 additions and 1159 deletions
--- a/docs/astro-orbiter-gpu-passthrough.md
+++ b/docs/astro-orbiter-gpu-passthrough.md
@@ -0,0 +1,281 @@
+# astro-orbiter: AMD GPU Passthrough & Ollama/ROCm Setup
+
+**Host:** `tomorrowland` (10.1.71.12) — Proxmox node  
+**VM:** `astro-orbiter` (10.1.71.130, VM ID 71130)  
+**GPU:** MSI Radeon RX 5700 (Navi 10, gfx1010) via Minisforum DEG1 eGPU dock + OCuLink  
+**Purpose:** Ollama inference host for `jarvis` AI assistant  
+**Date:** 2026-05-25
+
+---
+
+## Hardware
+
+- Minisforum MS-01 (`tomorrowland`)
+- Minisforum DEG1 eGPU dock connected via M.2 OCuLink
+- MSI Radeon RX 5700 8GB
+- Montech BETA 2 650W ATX PSU powering the eGPU dock
+
+---
+
+## Part 1: tomorrowland Host Setup
+
+### 1.1 Network Interface Rename (eGPU side effect)
+
+Adding the eGPU dock shifted PCIe bus numbers, renaming all NICs. Fix:
+
+```bash
+# Back up interfaces config
+cp /etc/network/interfaces /etc/network/interfaces.bak
+
+# Apply renames (adjust if hardware differs)
+sed -i 's/enp87s0/enp93s0/g' /etc/network/interfaces
+sed -i 's/enp2s0f0np0/enp5s0f0np0/g' /etc/network/interfaces
+sed -i 's/enp2s0f0np1/enp5s0f0np1/g' /etc/network/interfaces
+
+# Reboot to apply (do from console/JetKVM, not SSH)
+reboot
+```
+
+### 1.2 Verify GPU Enumeration
+
+After reboot, confirm the GPU is visible:
+
+```bash
+lspci | grep -i amd
+# Expected:
+# 01:00.0 PCI bridge: AMD/ATI Navi 10 XL Upstream Port
+# 02:00.0 PCI bridge: AMD/ATI Navi 10 XL Downstream Port
+# 03:00.0 VGA compatible controller: AMD/ATI Navi 10 [Radeon RX 5600/5700]
+# 03:00.1 Audio device: AMD/ATI Navi 10 HDMI Audio
+```
+
+### 1.3 Verify IOMMU
+
+```bash
+dmesg | grep -i iommu | head -5
+# Must show: iommu: Default domain type: Translated
+
+# Verify GPU is in its own IOMMU groups (no ACS override needed)
+find /sys/kernel/iommu_groups/*/devices/ -name "*03:00*"
+# Expected: groups 17 and 18, one device each
+```
+
+### 1.4 Blacklist amdgpu and Bind vfio-pci
+
+```bash
+# Blacklist host GPU drivers
+echo "blacklist amdgpu
+blacklist radeon" > /etc/modprobe.d/blacklist-amdgpu.conf
+
+# Bind vfio-pci to the GPU at boot
+# 1002:731f = RX 5700 VGA, 1002:ab38 = Navi 10 HDMI Audio
+echo "options vfio-pci ids=1002:731f,1002:ab38" > /etc/modprobe.d/vfio.conf
+
+# Load vfio modules early
+echo "vfio
+vfio_iommu_type1
+vfio_pci
+vfio_virqfd" > /etc/modules-load.d/vfio.conf
+
+# Rebuild initramfs and reboot
+update-initramfs -u -k all
+reboot
+```
+
+Verify after reboot:
+```bash
+lspci -k -s 03:00.0
+# Must show: Kernel driver in use: vfio-pci
+
+lspci -k -s 03:00.1
+# Must show: Kernel driver in use: vfio-pci
+```
+
+### 1.5 Fix GPU D3 Power State on VM Restart (Required — RX 5700 lacks FLR)
+
+The RX 5700 does not support PCIe Function Level Reset (FLR). Without a fix,
+the GPU gets stuck in D3 power state on every VM restart, requiring a full host
+reboot to recover.
+
+**Fix: `pcie_port_pm=off` kernel parameter**
+
+`vendor-reset` was attempted but causes a QEMU PCI IRQ assertion failure
+(`pci_irq_handler: Assertion 0 <= irq_num`) on PVE 7.0.2 with this hardware.
+`pcie_port_pm=off` prevents the card from entering D3 entirely and is the
+correct solution for this setup.
+
+```bash
+sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="/GRUB_CMDLINE_LINUX_DEFAULT="pcie_port_pm=off /' /etc/default/grub
+update-grub
+reboot
+```
+
+Verify after reboot — VM should restart cleanly without host reboot:
+```bash
+qm reboot 71130
+# wait 30 seconds
+ssh wed@astro-orbiter 'lsmod | grep amdgpu'
+# Should be non-empty
+```
+
+---
+
+## Part 2: astro-orbiter VM Provisioning
+
+### 2.1 Terraform Provisioning
+
+VM was provisioned manually (not via NetBox pipeline) due to PCIe passthrough
+requirements. Run from `city-hall`:
+
+```bash
+cd /opt/git/homelab/terraform/proxmox/vm
+
+terraform apply \
+  -var='proxmox_api_url=https://fantasyland.local.mk-labs.cloud:8006' \
+  -var='proxmox_api_token=terraform@pve!terraform-token=<secret>' \
+  -var='vm_id=71130' \
+  -var='hostname=astro-orbiter' \
+  -var='ip_address=10.1.71.130' \
+  -var='target_node=tomorrowland' \
+  -var='template_name=ubuntu-24.04-large' \
+  -var='datastore=liberty-tree' \
+  -var='data_disk_enabled=true' \
+  -var='data_disk_size=128' \
+  -var='data_disk_datastore=utilidor'
+```
+
+**Notes:**
+- `datastore=liberty-tree` for OS disk — iSCSI storage does not support EFI disk
+  cloning from templates. OS disk stays on Ceph.
+- `data_disk_datastore=utilidor` for model storage — fresh disk creation works fine
+  on iSCSI.
+- `pcie_passthrough_enabled` is NOT passed — the `bpg/proxmox` provider's
+  `dynamic "hostpci"` block does not work reliably. PCIe passthrough is applied
+  manually via `qm set` after provisioning (see 2.2).
+
+### 2.2 Post-Provisioning: PCIe and Machine Type
+
+The template uses `i440fx` machine type. PCIe passthrough requires `q35`.
+Run after Terraform provisioning, with VM shut down:
+
+```bash
+# Switch to q35 machine type (required for PCIe passthrough)
+qm set 71130 -machine q35
+
+# Add GPU passthrough (both VGA and audio functions)
+qm set 71130 -hostpci0 0000:03:00,pcie=1,rombar=1
+
+# Add data disk on utilidor
+qm set 71130 -scsi1 utilidor:128,format=raw,discard=on
+
+# Increase RAM to 8GB
+qm set 71130 -memory 8192
+
+# Start the VM
+qm start 71130
+```
+
+Verify config landed:
+```bash
+qm config 71130 | grep -E "hostpci|machine|memory|scsi1"
+```
+
+### 2.3 Ansible Deployment
+
+From `city-hall` in the `ansible/` directory:
+
+```bash
+# OS baseline
+ansible-playbook playbooks/day0_baseline.yml --limit astro-orbiter --ask-vault-pass
+
+# Ollama + ROCm
+ansible-playbook playbooks/day1_deploy_ollama.yml --ask-vault-pass
+```
+
+The `day1_deploy_ollama.yml` playbook includes a pre-flight `lspci` check that
+will fail fast if the GPU is not visible to the VM.
+
+---
+
+## Part 3: Verification
+
+### 3.1 GPU visible in VM
+```bash
+lspci | grep -i amd
+# Expected: 01:00.0 VGA ... Navi 10 [Radeon RX 5600/5700]
+#           01:00.1 Audio ... Navi 10 HDMI Audio
+```
+
+### 3.2 amdgpu driver loaded
+```bash
+lsmod | grep amdgpu
+# Should show amdgpu and dependent modules
+```
+
+### 3.3 ROCm sees the GPU
+```bash
+/opt/rocm/bin/rocm-smi
+# Should show Device 0 with VRAM%, GPU%, temperature
+```
+
+### 3.4 Ollama GPU inference
+```bash
+ollama run qwen3:8b "say hello"
+# While running, check rocm-smi — GPU% and power draw should spike
+/opt/rocm/bin/rocm-smi
+# GPU% > 0%, power > 18W idle = GPU is doing inference
+```
+
+### 3.5 VM restart test
+```bash
+# On tomorrowland:
+qm reboot 71130
+# Wait 30 seconds, then on astro-orbiter:
+lsmod | grep amdgpu
+# Should be non-empty — if empty, vendor-reset is not working
+```
+
+---
+
+## Known Issues & Lessons Learned
+
+| Issue | Cause | Fix |
+|---|---|---|
+| NIC names changed after eGPU install | PCIe bus renumbering | Update `/etc/network/interfaces` with new names |
+| `can't allocate space in iscsi storage` on clone | iSCSI doesn't support EFI disk cloning | Use `liberty-tree` for OS disk |
+| `q35 machine model is not enabled` on hostpci | Template built with i440fx | `qm set <vmid> -machine q35` before adding hostpci |
+| GPU stuck in D3 on VM restart | RX 5700 lacks FLR support | `pcie_port_pm=off` kernel param on host. `vendor-reset` builds but causes QEMU PCI IRQ assertion failure on PVE 7.0.2 — do not use. |
+| `dynamic "hostpci"` block not applying | `bpg/proxmox` provider limitation | Use `qm set` workaround post-provisioning |
+| `rocm-smi` not in PATH | ROCm installs to `/opt/rocm-6.2.0/bin/` | `/etc/profile.d/rocm.sh` sets PATH (via Ansible role) |
+| Ollama install script 404 | `OLLAMA_VERSION=latest` is not a valid version string | Omit env var when version is `latest` — script self-determines |
+
+---
+
+## ROCm Configuration
+
+The RX 5700 is RDNA 1 (gfx1010). ROCm's official support starts at gfx1030
+(RDNA 2). The `HSA_OVERRIDE_GFX_VERSION=10.1.0` environment variable tells ROCm
+to treat the GPU as a supported target.
+
+This is set via systemd drop-in at:
+`/etc/systemd/system/ollama.service.d/override.conf`
+
+```ini
+[Service]
+Environment="OLLAMA_HOST=0.0.0.0:11434"
+Environment="OLLAMA_MODELS=/var/lib/ollama"
+Environment="HSA_OVERRIDE_GFX_VERSION=10.1.0"
+Environment="ROCR_VISIBLE_DEVICES=0"
+```
+
+---
+
+## Parking Lot
+
+- **P.PACKER:** Ubuntu VM templates should be built with `machine: q35` by default
+  to support PCIe passthrough without manual intervention.
+- **P.TERRAFORM:** Fix `bpg/proxmox` provider `hostpci` dynamic block or find
+  correct syntax for the installed provider version.
+- **P.PIPELINE:** Update n8n workflow to pass `data_disk_enabled`, `data_disk_size`,
+  and `pcie_device_id` from NetBox custom fields to Terraform for pipeline-driven
+  GPU VM provisioning.