A local LLM in a dedicated VM keeps it isolated, snapshot-able, and easy to rebuild. The big decision is whether to pass through a GPU.
CPU-only: easy, slow, fine for small jobs
Ollama runs on CPU with no extra setup. It works for small quantized models and occasional use, such as summarising notes or short Home Assistant prompts, but responses are much slower than on a GPU.
Give the VM plenty of RAM (the model must fit in memory alongside the OS) and use CPU type host so the guest sees your CPU’s instruction-set extensions. See QEMU/KVM VMs for CPU type options.
GPU passthrough: fast, more setup
Proxmox’s PCI Passthrough guide covers the full process. In short:
- Enable VT-d/AMD-Vi (IOMMU) in BIOS and verify IOMMU is active on the host.
- Load the VFIO modules and make sure the GPU is in a clean IOMMU group.
- Stop the host driver from claiming the GPU (bind it to
vfio-pci). - Use the
q35machine type and OVMF (UEFI) for the VM, then add the GPU as a PCI device with PCI-Express enabled.
Note that a passed-through GPU belongs to that one VM. The host and other guests can’t use it at the same time.
Install Ollama in the VM
Follow the Linux install docs. For GPU use, install the vendor driver in the guest and check supported GPUs before buying hardware.
Check where the model actually runs
ollama run llama3.2 "hello"
ollama psollama ps shows whether the loaded model is on GPU, CPU, or split between them. A split means the model doesn’t fit in VRAM. Choose a smaller model or quantization for full speed.
Snapshot before upgrades
Snapshot the VM before driver or Ollama updates. GPU driver upgrades are the most common way this setup breaks.
Quick decision
- CPU-only: low cost, low power, light use.
- GPU passthrough: interactive chat, voice assistants, bigger models.