1af645d272
Critical finding: flipping vllm_service_enabled/state=true/started and restarting llama-swap alongside it broke llama-swap's ability to load ANY of its own generative models -- every /v1/chat/completions request against Qwen3.8-27B-Q4_K_M or Qwen3-8B aux models failed with 'upstream command exited prematurely' (llama-server OOM at spawn, ~1.8GB free on this 24GB card once vLLM's ~22.8GB was claimed). Confirmed by direct A/B: same request 500s with vLLM running, 200s seconds after stopping it. This breaks 21 Hermes agent profiles' aux-model tasks (skills_hub, approval, mcp, title_generation, profile_describer, compression) plus OpenViking's VLM -- a far larger blast radius than Hindsight's single LLM endpoint. Reverted: - vllm_service_enabled/state back to role defaults (false/stopped) -- vLLM stays staged, startable for a brief validated shadow window, NOT safe to leave resident in production. - Hindsight's HINDSIGHT_API_LLM_BASE_URL back to llama-swap (astro-orbiter:8001, Qwen3.8-27B-Q4_K_M) and the API key secret source back to the Nous fallback item (pre-task state) -- the vLLM cutover, while functionally validated in isolation (health, /v1/chat/completions, and a live hindsight_retain+recall round-trip all succeeded), requires continuous vLLM availability which is now known to be unsafe on this card. Comment posted on t_6dff1ecc: teardown remains correctly blocked -- full cutover is not achievable within this card's VRAM budget as currently scoped. Needs a human decision on aux-model migration strategy (see roles/deploy-vllm README's 'Critical architectural finding' section) before any further progress.
mk-labs
Automated infrastructure provisioning and configuration for a personal homelab, built on GitOps practices with clear tool responsibility boundaries.
Architecture
A single operator action — setting a VM record's status to Staged in NetBox — triggers a fully automated provisioning pipeline:
NetBox (webhook) → n8n (validate & orchestrate) → Terraform (create VM + DHCP)
→ Ansible (OS config + DNS + status update)
| Tool | Host | IP | Responsibility |
|---|---|---|---|
| NetBox | fire-station | 10.1.71.102 | Source of truth — VM records, IP allocation, VLAN data |
| n8n | tiki-room | 10.1.71.23 | Event orchestration, validation, pipeline sequencing |
| Terraform | city-hall | 10.1.71.35 | Proxmox VM lifecycle, Unifi DHCP reservations |
| Ansible / Semaphore | imagineering | 10.1.71.22 | OS configuration, DNS records, NetBox status updates |
| Proxmox | fantasyland | 10.1.71.13 | Target hypervisor |
All systems on the Server Trusted VLAN (10.1.71.0/24).
Repository Structure
homelab/
├── ansible/
│ ├── inventory/ # NetBox dynamic inventory + static
│ ├── playbooks/ # Runnable playbooks (vm-provision, DNS, OS updates)
│ ├── roles/ # vm-baseline, dns-manager, common, haproxy, n8n, observer, etc.
│ ├── tasks/ # Shared includable task files
│ ├── group_vars/ # Group variable definitions
│ ├── host_vars/ # Per-host variable definitions
│ ├── templates/ # Jinja2 templates
│ └── ansible.cfg
│
├── terraform/
│ ├── proxmox/vm/ # bpg/proxmox provider — VM creation from templates
│ ├── unifi/dhcp/ # Unifi provider — DHCP static reservations on UDM Pro
│ └── dns/ # DNS record management
│
├── packer/
│ ├── ubuntu-24.04/ # Ubuntu 24.04 VM template (small → xlarge-plus sizes)
│ └── fedora-42/ # Fedora 42 VM template
│
├── n8n/
│ └── workflows/ # Exported n8n workflow JSON (vm-provisioning)
│
├── netbox/
│ └── initializers/ # Custom fields, VLANs, IP prefixes as code
│
└── docs/
└── decisions/ # Architecture decision records
Pipeline Flow
| # | System | Action |
|---|---|---|
| 1 | NetBox | Operator sets VM status to Staged → webhook fires |
| 2 | n8n | Validates payload (hostname, IP, VLAN, template, proxmox_node) |
| 3 | n8n → city-hall | SSH + terraform apply — creates VM on Proxmox |
| 4 | n8n | Queries Proxmox API for MAC address |
| 5 | n8n → NetBox | Writes MAC to VM interface record |
| 6 | n8n → city-hall | SSH + terraform apply — creates DHCP reservation on UDM Pro |
| 7 | n8n → imagineering | Triggers Ansible via Semaphore API |
| 8 | Ansible | OS baseline, SSH hardening, Technitium DNS A record |
| 9 | Ansible → NetBox | Sets VM status to Active |
On any failure, NetBox status is set to Failed. No auto-retry — operator investigates.
Hardware
- 7× Dell 7050 SFF
- 3× Minisforum TH60
- 2× Minisforum MS01
- Synology DS1621+
- Ubiquiti UDM Pro
Software Stack
- Virtualization: Proxmox
- Automation: Terraform, Ansible, n8n, Semaphore
- DNS: Technitium (authoritative), Unbound (recursive)
- IPAM/DCIM: NetBox
- Networking: Ubiquiti UDM Pro
- Templates: Packer (Ubuntu 24.04, Fedora 42)
Getting Started
See docs/decisions/vm-provisioning-flow.md for the full architecture decision record.
Previous OpenShift/ACM/Fastpass content is preserved in the archive/pre-mk-labs branch.
Security
No sensitive data is stored in this repository. Secrets are managed via Ansible Vault and environment variables on pipeline hosts.
Status: 🚧 Active Development — VM Provisioning Pipeline
Last Updated: February 2026