homelab

rblundon/homelab

Fork 0

Commit Graph

Author	SHA1	Message	Date
Hermes Agent service account	aff792a061	feat(llm-inference): move astro-orbiter monitoring to GitOps (values.yaml + dashboards.yaml) - Prometheus scrape configs for node/gpu/llama-server exporters on astro-orbiter now declared in cluster/applications/monitoring/values.yaml (additionalScrapeConfigs), applied via ArgoCD sync instead of an imperative kubectl secret patch from the Ansible role. - Grafana dashboard for astro-orbiter LLM inference added as a ConfigMap in cluster/applications/monitoring/dashboards.yaml (grafana_dashboard=1 sidecar label), replacing the role's ad-hoc kubectl apply of a rendered Jinja template. - ansible/roles/llm-inference/tasks/monitoring.yml: removed the kubectl get/patch/apply tasks and orphaned grafana-llm-dashboard.json.j2 template; role now only stands up node_exporter + nvidia_gpu_exporter and verifies they're reachable — cluster-facing config lives in Git. - host_vars/vars.yml + inventory.yml: finalize astro-orbiter as the llama.cpp/RTX 3090 host (jarvis user, ssh key), drop stale Ollama/AMD vars and ollama_server inventory group superseded by the ATX rebuild.	2026-08-05 09:43:54 -05:00
Hermes Agent service account	aa8e229e64	fix(llm-inference): switch serve phase from vLLM+bitsandbytes to llama.cpp+GGUF bitsandbytes peak RAM ~54GB (bf16 load before quantize) — kills 40GB OptiPlex. llama.cpp Q4_K_M GGUF loads pre-quantized: peak RAM ~15.5GB, fits cleanly. Changes: - serve.yml: build llama.cpp with CUDA, download Q4_K_M GGUF from bartowski, disable vllm-serve, deploy llama-server.service - llama-server.service.j2: OpenAI-compatible server on same port 8000, --n-gpu-layers 99 (full GPU offload), --parallel 4, gemma chat template - defaults: llm_gguf_dir, llm_gguf_path, llm_gpu_layers, llm_parallel_slots - handlers: restart llama-server, vllm-serve failed_when=false (may not exist) GGUF: bartowski/gemma-2-27b-it-Q4_K_M.gguf (15.5GB, 24GB VRAM fits w/ ~8GB headroom)	2026-08-03 12:37:03 -05:00
Hermes Agent service account	dda6b91330	feat(llm-inference): Day 1 playbook for RTX 3090 vLLM stack on astro-orbiter - nvidia-driver-595-open (already installed 2026-08-03, idempotent) - Python venv + vLLM 0.26.0 (already installed, idempotent) - Gemma 2 27B model download via HuggingFace hub - systemd vllm-serve.service on port 8000 - Hermes provider integration on carousel-of-progress - vault_hf_token added to group_vars/all/vault - ansible.cfg: vault_password_file set to absolute path - inventory: astro_orbiter group added Run with: env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference.yml	2026-08-03 11:51:34 -05:00

Author

SHA1

Message

Date

Hermes Agent service account

aff792a061

feat(llm-inference): move astro-orbiter monitoring to GitOps (values.yaml + dashboards.yaml)

- Prometheus scrape configs for node/gpu/llama-server exporters on
  astro-orbiter now declared in cluster/applications/monitoring/values.yaml
  (additionalScrapeConfigs), applied via ArgoCD sync instead of an
  imperative kubectl secret patch from the Ansible role.
- Grafana dashboard for astro-orbiter LLM inference added as a ConfigMap
  in cluster/applications/monitoring/dashboards.yaml (grafana_dashboard=1
  sidecar label), replacing the role's ad-hoc kubectl apply of a rendered
  Jinja template.
- ansible/roles/llm-inference/tasks/monitoring.yml: removed the kubectl
  get/patch/apply tasks and orphaned grafana-llm-dashboard.json.j2
  template; role now only stands up node_exporter + nvidia_gpu_exporter
  and verifies they're reachable — cluster-facing config lives in Git.
- host_vars/vars.yml + inventory.yml: finalize astro-orbiter as the
  llama.cpp/RTX 3090 host (jarvis user, ssh key), drop stale Ollama/AMD
  vars and ollama_server inventory group superseded by the ATX rebuild.

2026-08-05 09:43:54 -05:00

Hermes Agent service account

aa8e229e64

fix(llm-inference): switch serve phase from vLLM+bitsandbytes to llama.cpp+GGUF

bitsandbytes peak RAM ~54GB (bf16 load before quantize) — kills 40GB OptiPlex.
llama.cpp Q4_K_M GGUF loads pre-quantized: peak RAM ~15.5GB, fits cleanly.

Changes:
- serve.yml: build llama.cpp with CUDA, download Q4_K_M GGUF from bartowski,
  disable vllm-serve, deploy llama-server.service
- llama-server.service.j2: OpenAI-compatible server on same port 8000,
  --n-gpu-layers 99 (full GPU offload), --parallel 4, gemma chat template
- defaults: llm_gguf_dir, llm_gguf_path, llm_gpu_layers, llm_parallel_slots
- handlers: restart llama-server, vllm-serve failed_when=false (may not exist)

GGUF: bartowski/gemma-2-27b-it-Q4_K_M.gguf (15.5GB, 24GB VRAM fits w/ ~8GB headroom)

2026-08-03 12:37:03 -05:00

Hermes Agent service account

dda6b91330

feat(llm-inference): Day 1 playbook for RTX 3090 vLLM stack on astro-orbiter

- nvidia-driver-595-open (already installed 2026-08-03, idempotent)
- Python venv + vLLM 0.26.0 (already installed, idempotent)
- Gemma 2 27B model download via HuggingFace hub
- systemd vllm-serve.service on port 8000
- Hermes provider integration on carousel-of-progress
- vault_hf_token added to group_vars/all/vault
- ansible.cfg: vault_password_file set to absolute path
- inventory: astro_orbiter group added

Run with: env -u ANSIBLE_VAULT_PASSWORD_FILE ansible-playbook -i inventory.yml playbooks/day1_deploy_llm_inference.yml

2026-08-03 11:51:34 -05:00

3 Commits