Running an LLM locally means your prompts never leave your machine, you pay no per-token fees, and you can run models that are unavailable through any API. In 2026, local LLMs have reached a quality threshold where a 32B parameter model running on consumer hardware genuinely rivals GPT-5.4 for most everyday tasks. This guide covers every layer of the stack: hardware selection, runtime choice, model selection, and connecting your existing tools to a local endpoint.
Get more guides like this in your inbox
No spam. Unsubscribe anytime.
Why Run Locally?
The three most common reasons are privacy, cost, and control. Privacy: every prompt you send to OpenAI, Anthropic, or Google is processed on their servers and may be used for safety review, abuse detection, or model improvement. For legal documents, medical records, proprietary code, or personal journals, that is unacceptable. Cost: at scale, API costs compound quickly — a developer making 1,000 API calls per day at $0.01 per call spends $3,650 per year. A one-time hardware investment pays for itself within months. Control: local models can be fine-tuned, run offline, and are not subject to API rate limits, outages, or policy changes.
Step 1: Assess Your Hardware
The single most important variable in local LLM performance is VRAM — the memory on your GPU. Models must fit entirely in VRAM to run at GPU speed; if they overflow to system RAM or CPU, inference slows by 5–20x. The practical formula: a model needs approximately (parameter count in billions × quantization bits / 8) GB of VRAM. A 7B model at 4-bit quantization needs ~4.4 GB. A 70B model at 4-bit needs ~40 GB. Apple Silicon Macs are a special case — their unified memory architecture means GPU and CPU share the same RAM pool, so a 64 GB MacBook Pro can run a 70B model at GPU speed.
Hardware Recommendations by Budget
For new hardware purchases in 2026, the GPU market is in a difficult state due to memory shortages and RTX 5090 scalping. The RTX 5060 Ti 16 GB ($539) is the best budget GPU. For the $1,500–$2,000 range, the Mac Studio M4 Max (36 GB unified memory, $1,999) offers better value than any consumer GPU for LLM inference. The RTX 5090 is technically the best GPU for LLMs but is selling at $3,798 — nearly double MSRP. For Apple Silicon laptops, the M4 Pro MacBook Pro with 48 GB unified memory is the best option; the M5 Max Mac Studio (shipping mid-2026) will be the best desktop option.
Step 2: Choose a Runtime
The runtime is the software that loads the model and handles inference. There are three main options in 2026, each with a different target use case.
Install Ollama (Recommended for Most Users)
Ollama is the recommended starting point for most users. It handles model management, GPU detection, and API serving automatically:
# Install via Homebrew
brew install ollama
# Pull and run a model
ollama run qwen3.5:9b
# The API is now available at http://localhost:11434Install MLX (Apple Silicon Only)
MLX is Apple's machine learning framework, optimized for the unified memory architecture of M-series chips. For Apple Silicon users, MLX inference is 3–4x faster than Ollama/llama.cpp for the same model:
# Install mlx-lm
pip install mlx-lm
# Run a model (downloads from Hugging Face automatically)
mlx_lm.generate --model mlx-community/Qwen3.5-9B-4bit --prompt "Hello, world"
# Serve as an OpenAI-compatible API
mlx_lm.server --model mlx-community/Qwen3.5-9B-4bit --port 8080Step 3: Choose a Model
Model selection depends on your hardware, use case, and quality requirements. In 2026, the Qwen3.5 family from Alibaba leads most benchmarks at every size class. DeepSeek-R1 distilled models are the best option for reasoning and math tasks. Llama 4 Scout (17B active parameters via mixture-of-experts) is Meta's latest and runs efficiently on 24 GB VRAM. For coding specifically, DeepSeek V3 and Qwen3.5-Coder are the strongest options.
Step 4: Connect Your Tools
Once your local LLM is running, connect it to your existing workflow. The OpenAI-compatible API means most tools work with a simple base URL change:
# Test your local API directly
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen3.5:9b",
"messages": [{"role": "user", "content": "Hello"}]
}'
# VS Code + Continue.dev: set provider to "ollama", base URL to http://localhost:11434
# Open WebUI: set Ollama base URL to http://localhost:11434
# LangChain: use ChatOllama(model="qwen3.5:9b", base_url="http://localhost:11434")
# AnythingLLM: set LLM provider to Ollama, base URL to http://localhost:11434Running on a Cloud GPU
If your local hardware is insufficient — or you need more VRAM for a larger model — RunPod Secure Cloud is the recommended option. It provides dedicated GPU pods (not shared with other tenants) starting at $0.59/hour for an RTX 4090 (24 GB VRAM). The setup is identical to local: install Ollama on the pod, pull your model, and use SSH port forwarding to expose the API securely to your local machine. Your prompts travel over an encrypted SSH tunnel and are processed on hardware you've rented exclusively.