The recommendation
Use Ollama as the reference runtime for single-user local inference and Hermes Agent. Keep llama.cpp as the optimization laboratory for GGUF, Metal, CPU, and unusual offload layouts. Use vLLM for Linux servers where continuous batching, multiple users, or multi-GPU throughput matters. vLLM's native macOS path is experimental and its Metal acceleration is community-maintained, so it is not the Mac default.
Get more guides like this in your inbox
No spam. Unsubscribe anytime.
Separate hype from alpha
Alpha is a model/runtime pair that passes the same acceptance checks on named hardware. Hype is a vendor benchmark, a short chat demo, an unpinned installer, or a model that fits only until a real context window is allocated. Qwen 3.8 27B is genuine ecosystem momentum: it is prominent in current model discovery, has official 256K context and agent/tool positioning, and ships directly through Ollama. But our 36 GB Mac test also showed the important counter-signal: direct inference passed while the full Hermes 64K call timed out.
Runtime roles
Ollama wins on maintenance cost and harness integration. llama.cpp wins when you need control over quantization, GPU layers, KV cache, or a specific GGUF build. vLLM wins when a dedicated Linux GPU host serves concurrent clients. LM Studio and other GUIs can be excellent personal tools, but maintaining parallel tutorials for every wrapper dilutes verification quality; they are intentionally outside the core tutorial set.
The model-profile contract
Every published profile records: exact model identifier and manifest digest; runtime version; hardware and operating system; allocated context; model bytes and peak memory; direct inference result; Hermes result; cold and warm latency; tool-use test; known failures; and verification date. A profile is PASS only when the harness test passes. Direct-only success is EXPERIMENTAL.
Current maintained profiles
Qwen 3.5 9B + Ollama 0.32.14 + Hermes 0.20.1 is PASS on Ubuntu 24.04.4, 6 vCPU, 16 GiB, CPU-only, with a 64K context. Qwen 3.8 27B MLX + Ollama 0.32.14 is DIRECT PASS on a 36 GB M4 Max Mac, but EXPERIMENTAL for Hermes because the 64K harness call exceeded seven minutes. No throughput claim is generalized from one machine or one short marker.
Hardware decision in one minute
Already own a 16 GB machine: start with a 7B–9B quantized model and expect limited agent speed. Buying a quiet appliance: choose an Apple-silicon Mac with at least 32 GB, preferably a 48/64 GB M5 Pro Mac mini for context headroom. Want upgradeable NVIDIA acceleration: choose a Windows/Linux mini PC with OCuLink plus a discrete GPU, or a small desktop. Apple-silicon Macs do not support eGPUs; Thunderbolt ports do not change that.
Privacy boundary
Local inference means prompts sent to the loopback model endpoint are not processed by a hosted LLM provider. It does not make the whole agent offline. Web search, remote MCP servers, telemetry, messaging bridges, package downloads, and cloud fallbacks can still transmit data. Keep Ollama on loopback, review every Hermes tool, and mount only the directories a task needs.
Start here
Use the maintained Ollama install guide first. If your goal is an agent rather than chat, continue with the Hermes + Ollama baseline and do not skip the 64K context or exact-marker checks. Those two tutorials are the source of truth; this page remains the strategy layer so it does not duplicate commands that can drift.