homelab/docs/vm-provisioning-flow.md

ADR: VM Provisioning Pipeline Flow

Status: Accepted
Date: February 2026
Supersedes: PRD v1.0 Section 3.2 (adds DHCP step, adjusts tool boundaries)
Last Updated: February 28, 2026 — revised flow for DHCP-before-boot, HA affinity via Proxmox API, secrets via n8n globals

Context

The mk-labs homelab needs an automated VM provisioning pipeline triggered by a single operator action. The original PRD defined the flow; this ADR documents the adjusted architecture where n8n owns all orchestration including DHCP reservation (previously Ansible-rendered Terraform templates).

Decision

System Context

Tool	Host	IP Address	Role in Pipeline
NetBox	fire-station	10.1.71.102	Source of truth — VM records, IP allocation, VLAN data, webhook emitter
n8n	tiki-room	10.1.71.23	Event orchestration, payload validation, pipeline sequencing
Terraform	city-hall	10.1.71.35	Proxmox VM lifecycle + Unifi DHCP reservations (repo: /opt/git/homelab)
Ansible / Semaphore	imagineering	10.1.71.22	OS configuration, DNS record creation, NetBox status updates
Proxmox	fantasyland	10.1.71.13	Target hypervisor for all new VM deployments

Status State Machine

PLANNED  →(operator)→  STAGED  →(pipeline)→  ACTIVE
                                    |
                                on error
                                    ↓
                                  FAILED

• Planned: All fields populated, operator not yet ready. No trigger.
• Staged: Operator validated and ready. Triggers pipeline.
• Active: Set by Ansible on successful OS config + DNS registration.
• Failed: Set by n8n or Ansible on any error. Manual investigation.

Provisioning Flow (Revised)

#	System	Action	Detail
1	NetBox	Operator sets VM status to Staged	Triggers webhook to n8n on tiki-room
2	n8n	Receive & validate webhook payload	Check: hostname, IP, template, proxmox_node, proxmox_datastore all present. Status must be exactly Staged. On failure → set NetBox status to Failed, stop.
3	n8n → city-hall	SSH to city-hall, run terraform apply (proxmox/vm)	Creates VM clone on fantasyland in stopped state. Uses -state=states/{hostname}.tfstate for per-VM state isolation. Passes secrets from n8n Global Constants via -var flags.
4	n8n	Parse Terraform output for MAC address	Reads mac_address from terraform output -json.
5	n8n → NetBox	Write MAC address to VM interface	PATCH to NetBox API to update VM interface record with MAC.
6	n8n → city-hall	SSH to city-hall, run terraform apply (unifi/dhcp)	Creates static DHCP reservation on UDM Pro. Uses -state=states/{hostname}.tfstate. MAC and IP passed as variables.
7	n8n → Proxmox API	Add VM to HA resource + node affinity rule	POST to /cluster/ha/resources to register VM as HA resource. PUT to /cluster/ha/rules/{prefer_node} to add VM to the target node's affinity rule.
8	n8n → Proxmox API	Start VM	POST to /nodes/{node}/qemu/{vmid}/status/start. VM boots and receives IP via DHCP (reservation already in place).
9	n8n → imagineering	Call Semaphore API to trigger Ansible	Launches vm-provision playbook with VM details as extra vars.
10	Ansible	OS configuration & DNS record creation	Applies vm-baseline role: packages, SSH keys, Technitium DNS A record creation.
11	Ansible → NetBox	Set VM status to Active	Final step of playbook updates NetBox VM status via API.
ERR	n8n / Ansible	Any step fails	NetBox VM status set to Failed. Pipeline halts. No cleanup — operator investigates manually.

Key Design Decisions

VM created in stopped state (Step 3) Cloud-init uses ip=dhcp. The DHCP reservation must exist on the UDM Pro before the VM's first boot, otherwise the VM gets a random DHCP lease instead of its assigned IP. The VM stays stopped until after the Unifi DHCP reservation is confirmed (Step 6).

HA affinity via Proxmox API, not Terraform (Step 7) Proxmox 9.x replaced HA groups with HA node affinity rules. The bpg/proxmox Terraform provider does not yet support these (tracked in bpg/terraform-provider-proxmox#2097). n8n calls the Proxmox API directly to add the VM to the appropriate prefer_{node} affinity rule. This also handles VM migration from fantasyland (clone source) to the target node.

Per-VM Terraform state files (Steps 3, 6) Each VM gets its own state file via -state=states/{hostname}.tfstate to avoid state conflicts when managing multiple VMs. The states/ directory is in .gitignore.

Secrets via n8n Global Constants All secrets (Proxmox API token, Unifi credentials, NetBox API token) are stored in n8n's encrypted credential store using the n8n-nodes-globals community node. Secrets are passed to Terraform via -var flags in SSH commands. No secrets are stored on city-hall's filesystem. See docs/n8n-setup.md for details.

Tool Responsibility Boundary (Adjusted)

Tool	Owns	Does Not Own
Terraform	Proxmox VM lifecycle (bpg/proxmox), UDM Pro DHCP reservations (unifi provider)	OS config, DNS records, application deployment, NetBox data, HA placement
Ansible	OS configuration, DNS A records (Technitium API), NetBox status updates	Infrastructure provisioning, DHCP reservations, event orchestration
n8n	Event orchestration, API glue, pipeline sequencing, HA affinity rule management, triggering all Terraform and Ansible steps, secrets distribution	Configuration management, infrastructure state
NetBox	Source of truth for all assets	Execution of any automation — it only holds data

Dependencies

Component	Dependency	Purpose
n8n	n8n-nodes-globals (community node)	Encrypted secrets storage for pipeline credentials
Terraform (proxmox/vm)	bpg/proxmox provider >= 0.74.0	Proxmox VM lifecycle management
Terraform (unifi/dhcp)	paultyng/unifi provider >= 0.41.0	UDM Pro DHCP static reservations
Proxmox	API token: terraform@pve!terraform-token	Terraform + n8n API access (requires VM.GuestAgent.Audit for Proxmox 9)
NetBox	Event Rule + Webhook (VM Staged Trigger)	Pipeline trigger on status change to Staged
NetBox	REST API token	n8n reads/writes VM data

Change from PRD v1.0

What changed (original): DHCP reservation moved from "existing Terraform/UDM Pro integration" (out of scope) to an explicit pipeline step owned by n8n.

What changed (this revision):

1. VM is created in stopped state. DHCP reservation is created before VM boots.
2. HA node affinity rule assignment added as an explicit step (Proxmox API, not Terraform).
3. VM start is a separate explicit step after HA rule assignment.
4. Secrets managed via n8n Global Constants instead of filesystem-based .tfvars files.
5. Per-VM Terraform state files via -state flag.

Why: Cloud-init uses ip=dhcp, so the DHCP reservation must exist before first boot. The bpg/proxmox Terraform provider doesn't support Proxmox 9 HA affinity rules, so n8n handles placement via API. Secrets in n8n's credential store keeps city-hall stateless.

Git Repository Structure

homelab/
├── ansible/
│   ├── inventory/
│   │   ├── netbox.yml              # NetBox dynamic inventory plugin
│   │   └── static.yml              # Legacy static inventory
│   ├── playbooks/
│   │   ├── vm-provision.yml        # Main playbook (Semaphore trigger)
│   │   ├── add_technitium_dns_entry.yml
│   │   ├── add_dhcp_reservation.yml
│   │   ├── update-linux-os.yml
│   │   └── set_hostname.yml
│   ├── roles/
│   │   ├── vm-baseline/            # NEW — packages, SSH, DNS, NetBox status
│   │   ├── dns-manager/            # Existing Technitium DNS role
│   │   ├── common/                 # Base system config
│   │   ├── haproxy/
│   │   ├── n8n/
│   │   ├── observer/
│   │   └── time-sync/
│   ├── tasks/                      # Shared task files
│   ├── group_vars/
│   ├── host_vars/
│   ├── templates/
│   ├── scripts/
│   ├── ansible.cfg
│   └── requirements.yml
│
├── terraform/
│   ├── proxmox/
│   │   └── vm/
│   │       ├── main.tf             # bpg/proxmox VM resource (creates stopped)
│   │       ├── variables.tf        # hostname, ip, template, proxmox_node
│   │       ├── outputs.tf          # vm_id, mac_address
│   │       ├── providers.tf
│   │       ├── terraform.tfvars.example
│   │       └── states/             # Per-VM state files (.gitignore)
│   ├── unifi/
│   │   └── dhcp/
│   │       ├── main.tf             # Unifi user resource (DHCP reservation)
│   │       ├── variables.tf        # hostname, mac, ip
│   │       ├── providers.tf
│   │       ├── terraform.tfvars.example
│   │       └── states/             # Per-VM state files (.gitignore)
│   └── dns/
│       ├── dns.tf                  # Existing DNS terraform
│       └── provider.tf
│
├── packer/
│   ├── ubuntu-24.04/               # Ubuntu template build
│   └── fedora-42/                  # Fedora template build
│
├── n8n/
│   └── workflows/
│       └── vm-provisioning.json    # Exported n8n workflow
│
├── netbox/
│   └── initializers/
│       ├── custom_fields.yml
│       ├── vlans.yml
│       └── prefixes.yml
│
├── docs/
│   ├── decisions/
│   │   ├── vm-provisioning-flow.md # This document
│   │   └── tool-responsibility-matrix.md
│   └── n8n-setup.md               # n8n dependencies and configuration
│
├── README.md
├── LICENSE
└── .gitignore

Consequences

• Positive: Single visual workflow in n8n shows entire pipeline. Ansible simplified to pure configuration management. Terraform usage consistent (always invoked via n8n SSH, never rendered from templates). Secrets never stored on disk outside n8n's encrypted database. Per-VM state files prevent conflicts.
• Negative: n8n becomes a critical dependency — if tiki-room is down, no provisioning. Two separate terraform apply calls per VM (Proxmox + Unifi) instead of one. HA rule management is imperative API calls rather than declarative Terraform.
• Mitigated by: n8n workflow JSON committed to Git. Both Terraform modules can be run manually from city-hall if n8n is unavailable. HA rules can be managed via ha-manager CLI on any Proxmox node.