Run a Private Local AI on Raspberry Pi 5

Before You Start: What the Pi 5 Can and Can't Do

The Raspberry Pi 5 is a genuinely capable single-board computer. At $80 for the 8GB model, it is the cheapest way to run a language model with zero cloud dependency, zero data leaving your home, and zero monthly cost after purchase. That is a real achievement.

But it is not a desktop workstation. The Pi 5 uses a 4-core ARM Cortex-A76 CPU running at 2.4GHz with no discrete GPU. Language models run entirely on the CPU, which means inference speed is measured in tokens per second rather than tokens per millisecond. In practical terms, a good model on a Pi 5 types out responses at roughly the speed of a fast human typist — around 5 to 10 words per second. That is readable and usable. It is not instant.

Here is what you can realistically expect from the 8GB Pi 5:

• Models that work well: 1B–2B parameter models (Gemma 3 1B, Qwen3.5 2B, Qwen 2.5 1.5B) run at 5–15 tokens/sec. Responses feel natural.
• Models that work but feel slow: 3B parameter models (Llama 3.2 3B, Qwen 2.5 3B) run at 4–7 tokens/sec. Usable for non-interactive tasks.
• Models that technically run but aren't practical: 7B+ models run at 1–3 tokens/sec. A single response can take several minutes. Not recommended.
• Models that won't run at all: Anything requiring more than ~6GB of RAM after the OS overhead (~1.5GB) will either fail to load or cause the system to swap to the SD card, which is effectively unusable.

The 8GB model is the only Pi 5 worth buying for AI work. The 4GB model leaves too little headroom once the OS and Ollama are loaded.

What the Pi 5 Is Actually Good For

The Pi 5's constraints push you toward a specific and genuinely useful niche: an always-on, low-power private AI endpoint for your home network. At 5–10W under load, it costs roughly $5–8/year in electricity to run 24/7. No cloud account, no API key, no data leaving your network.

Concrete use cases that work well at 1B–3B model scale:

Offline personal assistant: Ask questions, get summaries, draft short messages. The model doesn't know today's news, but it knows how to write, reason, and explain. Works well for anything that doesn't require real-time information.

Document Q&A: Paste in a contract, a manual, or a config file and ask questions about it. At 3B scale, models handle several thousand tokens of context reliably. This is one of the most privacy-sensitive use cases — you're not sending your documents to OpenAI.

Home automation scripting: Use the Pi as a local API endpoint for Home Assistant or n8n. When a sensor triggers, the automation calls the local Ollama API to classify, summarize, or generate a response. The latency of 5–10 seconds per call is acceptable for background automation tasks.

Code explanation and small edits: Paste a function, ask what it does or ask for a fix. Works well for Python, bash, and JavaScript at 2B–3B scale. Not a replacement for Copilot on complex codebases, but useful for quick offline lookups.

Image analysis (Qwen3.5 2B only): Describe photos, read text from images, or analyse what a Pi camera sees — entirely offline. See the Qwen3.5 2B section below for details.

Learning and experimentation: If you want to understand how LLMs work, run evals, or test prompts without paying per token, the Pi 5 is an excellent sandbox. You can run hundreds of prompts for free.

What it is not good for: real-time conversation with sub-second latency, processing large documents (>10,000 tokens) quickly, running multiple models simultaneously, or anything requiring a 7B+ model for quality reasons.

What You Need

This guide assumes you are starting from scratch. Here is the complete hardware list with honest notes on each component.

| Component | Recommended | Notes | |---|---|---| | Raspberry Pi 5 | 8GB model | The 4GB model is not sufficient for AI workloads | | Storage | Samsung 990 Pro NVMe + M.2 HAT+ | Fastest option. USB 3.0 SSD (e.g. Samsung T7) is a good budget alternative | | Power supply | Official Pi 5 27W USB-C PSU | The Pi 5 draws more power than older models; underpowered PSUs cause throttling | | Cooling | Active cooler (official or Argon Neo 5) | The CPU runs hot under sustained LLM inference. Passive cooling is not sufficient | | OS | Raspberry Pi OS (64-bit, Bookworm) | 64-bit is required. Lite version is fine if you prefer headless | | Network | Ethernet (preferred) or Wi-Fi | Ethernet is more stable for a server setup |

You do not need a keyboard, mouse, or monitor if you are comfortable with SSH. The guide uses SSH throughout.

Step 1: Install Raspberry Pi OS

Download Raspberry Pi Imager from the official site (raspberrypi.com/software) and flash Raspberry Pi OS (64-bit) to your storage device. When configuring the image, click the gear icon to pre-configure your hostname, SSH access, Wi-Fi credentials, and username/password. This saves you from needing a monitor during setup.

Insert the storage, connect power, and wait 60–90 seconds for first boot. Then SSH in:

ssh [email protected]
# or use the IP address if mDNS doesn't resolve:
ssh [email protected]

Once connected, update the system before doing anything else:

sudo apt update && sudo apt full-upgrade -y
sudo reboot

After the reboot, reconnect and verify you are running 64-bit OS:

uname -m
# Should output: aarch64

Step 2: Install Ollama

Ollama is the simplest way to run language models on the Pi 5. It handles model downloading, quantization selection, and provides an OpenAI-compatible HTTP API that other tools (Open WebUI, Home Assistant, n8n) can connect to.

Install with the official one-liner:

curl -fsSL https://ollama.com/install.sh | sh

Ollama installs as a systemd service and starts automatically. Verify it is running:

systemctl status ollama
# Should show: active (running)

Before pulling any models, configure Ollama for the Pi 5's constraints. The defaults are tuned for desktop hardware and will cause problems on the Pi. Create an override file:

sudo systemctl edit ollama

Add these environment variables in the editor that opens:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_MAX_LOADED_MODELS=1"
Environment="OLLAMA_MAX_QUEUE=2"
Environment="OLLAMA_KEEP_ALIVE=0"
Environment="OLLAMA_CONTEXT_LENGTH=4096"

Save and reload:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Step 3: Pull Your First Model

Start with Gemma 3 1B. It is the fastest model that produces genuinely useful output on the Pi 5, running at 10–15 tokens/sec with only 1.5GB of RAM. It is a good baseline to confirm everything is working before trying larger models.

ollama pull gemma3:1b

The download is about 815MB. Once complete, run it interactively:

ollama run gemma3:1b

Type a question and press Enter. You should see tokens streaming within 1–2 seconds. If the first response feels slow, that is normal — the model is loading from disk. Subsequent responses in the same session will be faster. To exit: type /bye or press Ctrl+D.

Run this benchmark to see your exact tokens/sec:

ollama run gemma3:1b "Explain what a transformer neural network is in 3 sentences." --verbose 2>&1 | grep "eval rate"

Once Gemma 3 1B is working, here are the models worth trying on Pi 5, in order of recommendation:

| Model | Pull command | RAM (Q4) | Speed (t/s) | Best for | |---|---|---|---|---| | Gemma 3 1B | ollama pull gemma3:1b | ~1.5GB | 10–15 | Fastest, text-only Q&A | | Qwen3.5 2B | ollama pull qwen3.5:2b | ~1.6GB | 5–8 | Best overall — vision + reasoning | | Qwen 2.5 1.5B | ollama pull qwen2.5:1.5b | ~1.8GB | 10–13 | Structured output, coding | | Phi-3 Mini 3.8B | ollama pull phi3:mini | ~3.5GB | 5–8 | Reasoning, longer context | | Llama 3.2 3B | ollama pull llama3.2:3b | ~3.8GB | 4–7 | General purpose | | Qwen 2.5 3B | ollama pull qwen2.5:3b | ~4.2GB | 4–6 | Best quality at 3B scale |

Do not pull a 7B model expecting it to be usable. At 1–3 tokens/sec, a 200-word response takes 3–5 minutes. It will run, but it is not a practical tool.

Qwen3.5 2B: The Recommended Model for Pi 5

Qwen3.5 2B is the most capable model you can run comfortably on the Pi 5 8GB. Released in early 2026, it uses a hybrid DeltaNet + Gated Attention architecture that is meaningfully more capable than the previous generation Qwen 2.5 1.5B — better reasoning, better instruction following, and crucially, multimodal vision support. It uses only ~1.6GB of RAM at Q4 quantization, leaving plenty of headroom for the OS and other processes.

Pull it with:

ollama pull qwen3.5:2b

Vision capability: Qwen3.5 2B can analyse images — a unique capability at this model size. You can pass an image file directly from the command line:

# Describe an image
ollama run qwen3.5:2b "What is in this image?" /path/to/photo.jpg

# Read text from a photo (OCR)
ollama run qwen3.5:2b "Extract all text from this image" /path/to/document.jpg

Thinking mode warning: Qwen3.5 has a "thinking mode" where it reasons through a problem step-by-step before answering. This is disabled by default in Ollama for small models, which is the right call for Pi 5. The official model documentation explicitly warns that the 2B model is "more prone to entering thinking loops" — meaning it can generate thousands of tokens without stopping, which will cause your Pi to appear frozen.

Thinking is off by default, but you can confirm this and control it explicitly:

# Run with thinking explicitly disabled (recommended for Pi 5)
ollama run qwen3.5:2b --think=false

# In an interactive session, disable thinking:
/set nothink

# To enable thinking for a complex problem (use with caution on Pi 5):
/set think

Custom Modelfile for Pi 5: A Modelfile lets you bake Pi-specific settings into a named model so you never have to remember flags. This is the most effective way to optimise Qwen3.5 2B for your hardware.

Create a file called Modelfile in your home directory:

cat > ~/Modelfile << 'EOF'
FROM qwen3.5:2b
PARAMETER num_ctx 4096
PARAMETER num_thread 4
PARAMETER temperature 1.0
PARAMETER top_k 20
PARAMETER top_p 0.95
PARAMETER presence_penalty 1.5
SYSTEM "You are a helpful, concise assistant. Keep responses focused and brief."
EOF

Then build and run your Pi-optimised model:

ollama create qwen35-pi -f ~/Modelfile
ollama run qwen35-pi

Step 4: Access from Your Network

With OLLAMA_HOST=0.0.0.0:11434 set, Ollama is now accessible from any device on your local network. Find your Pi's IP address:

hostname -I | awk '{print $1}'

From any other machine on your network, you can call the API directly:

curl http://192.168.1.XXX:11434/api/generate -d '{
  "model": "qwen35-pi",
  "prompt": "What is a VPN?",
  "stream": false
}'

For a proper chat interface, install Open WebUI. It runs in Docker and gives you a ChatGPT-like UI that connects to your local Ollama instance:

# Install Docker first if you haven't
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
# Log out and back in, then:
docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

Keeping It Running Well

A few things that will save you headaches over time.

Monitor temperature: The Pi 5 will throttle the CPU at 80°C and hard-limit at 85°C. Under sustained LLM inference, a Pi 5 without active cooling will hit these limits within minutes.

vcgencmd measure_temp
# Healthy range under load: 55–72°C
# Concerning: above 78°C

Check RAM usage: Before pulling a new model, verify you have enough headroom. With the OS and Ollama idle, you should have ~6GB free. A 3B model needs ~4GB, so the math works — but only one model at a time.

free -h

Manage model storage: Each model is stored in ~/.ollama/models/. List what you have and remove models you are not using:

ollama list
ollama rm modelname:tag

Advanced: ik_llama.cpp for More Speed

If you want to squeeze more performance out of Qwen3.5 2B, the ik_llama.cpp fork by ikawrakow contains SIMD optimisations specifically written for Qwen3.5's DeltaNet architecture that mainline llama.cpp does not yet have. On ARM processors like the Pi 5's Cortex-A76, this can yield a 1.5–1.7x speedup — pushing Qwen3.5 2B from 5–8 t/s toward 8–13 t/s.

The trade-off is that there are no pre-built binaries for the Pi 5. You must compile from source, which takes approximately 20–30 minutes on the Pi itself.

# Install build dependencies
sudo apt install -y cmake build-essential

# Clone ik_llama.cpp
git clone https://github.com/ikawrakow/ik_llama.cpp
cd ik_llama.cpp

# Build with ARM NEON optimisations enabled
cmake -B build -DGGML_NATIVE=ON -DGGML_ARM_NEON=ON
cmake --build build --config Release -j4

# Run with a GGUF model file
./build/bin/llama-cli -m ~/.ollama/models/blobs/<model-blob> -p "Hello" -n 100

What to Try Next

Once your Pi 5 is running Ollama reliably with Qwen3.5 2B, here are the most useful next steps depending on what you want to build.

Connect to Home Assistant: The Ollama integration in Home Assistant lets you use your local model as an AI assistant for automations. Go to Settings → Integrations → Add Integration → Ollama, and point it at http://192.168.1.XXX:11434. Use the qwen35-pi Modelfile variant for this — the capped context and presence penalty make it more reliable for automation triggers.

Run a local RAG pipeline: Tools like AnythingLLM can connect to your Ollama instance and let you upload documents (PDFs, text files) and ask questions about them. This is the most privacy-preserving way to do document Q&A — your documents never leave your network.

Use it as a coding assistant: Configure your editor's AI extension to point at your local Ollama endpoint instead of OpenAI. Continue (VS Code extension) and Twinny both support custom Ollama endpoints. Qwen3.5 2B is the best model for this at Pi 5 scale.

Build a scheduled automation: The Pi 5 excels at batch, asynchronous tasks — cron jobs that pull RSS feeds, summarise documents, or analyse logs overnight while you sleep. The latency doesn't matter when the output is consumed the next morning.

Explore the OpenAI-compatible API: Ollama's API is compatible with the OpenAI client library. Any Python script that uses openai.ChatCompletion.create() can be redirected to your Pi by changing the base_url to http://192.168.1.XXX:11434/v1. This makes it easy to test prompts and automation scripts without spending API credits.

The Pi 5 is not the right tool for every AI task. But for a private, always-on, zero-cost inference endpoint on your home network, it is hard to beat. The constraint of working within 8GB and 4 CPU cores also teaches you a lot about what actually matters in a language model: quantization, context length, and prompt efficiency. Those lessons transfer directly to working with larger hardware.