Run CrewAI Privately with Ollama

CrewAI v1.12.2 (released March 26, 2026) supports Ollama as a first-class LLM provider via LiteLLM. Every prompt, every tool call, and every agent-to-agent message stays on your machine — nothing is sent to OpenAI, Anthropic, or any third-party API. This guide walks through the complete setup: installing CrewAI with the litellm extra, pulling a model with Ollama, wiring it into a two-agent crew, and verifying that zero traffic leaves your machine.

Why CrewAI + Ollama?

Most multi-agent frameworks default to cloud LLMs. CrewAI is different: it treats the LLM as a swappable dependency. Swap in Ollama and you get a fully private agent stack — no API key, no usage logs, no data retention policy to audit. The tradeoff is speed and model quality: a local Llama 3.3 70B will be slower than GPT-5.4, but for sensitive workloads (legal document analysis, internal code review, private research) the privacy guarantee is worth it. CrewAI's role-based agent model also maps well to real workflows: a Researcher agent and a Writer agent running locally is a practical pattern for many private use cases.

Prerequisites

You need Python 3.10–3.13, uv (the fast Python package manager recommended by CrewAI), and Ollama installed. If you don't have uv, install it with: curl -LsSf https://astral.sh/uv/install.sh | sh. If you don't have Ollama, install it with: curl -fsSL https://ollama.com/install.sh | sh. Verify both are working before proceeding.

# Verify uv
uv --version
# Expected: uv 0.6.x or later

# Verify Ollama
ollama --version
# Expected: ollama version 0.6.x or later

# Pull the model you'll use (Llama 3.3 70B, Q4_K_M quantization)
# Requires ~40 GB disk and 48 GB+ VRAM (2x 3090 or A100 80GB)
# For a single 24 GB GPU, use a 14B model instead:
ollama pull llama3.3:70b-instruct-q4_K_M
# OR for a 24 GB GPU:
ollama pull qwen3.5:9b

Step 1: Create a New CrewAI Project

Use the CrewAI CLI to scaffold a new project. The CLI creates the correct directory structure, agents.yaml, tasks.yaml, and crew.py files. The --provider flag sets the default LLM provider in your .env file.

# Install the CrewAI CLI
uv tool install crewai

# Create a new project (replace 'private_crew' with your project name)
crewai create crew private_crew

# When prompted for a model, enter: ollama/llama3.3:70b-instruct-q4_K_M
# (or your chosen model — see Step 2 for the exact format)

cd private_crew

Step 2: Install CrewAI with the LiteLLM Extra

Ollama is powered by LiteLLM in CrewAI. You must install the litellm extra — it is not included in the base crewai package. Without it, you'll get an ImportError when CrewAI tries to route to Ollama. Pin litellm to v1.83.0 or later — versions 1.82.7 and 1.82.8 were compromised in a supply chain attack on March 24, 2026 and have been removed from PyPI.

# Inside your project directory
# Pin litellm to a verified safe version (1.82.7 and 1.82.8 were compromised)
uv add 'crewai[litellm]' 'litellm>=1.83.0'

# Verify the install — confirm you are NOT on 1.82.7 or 1.82.8
uv run python -c "import litellm; print(litellm.__version__)"

Alternative: Fully Private Setup Without LiteLLM

If you want zero third-party Python dependencies, CrewAI can connect to Ollama directly through its OpenAI-compatible API — no litellm package required at all. Ollama exposes an OpenAI-compatible endpoint at http://localhost:11434/v1. Use the openai/ model prefix with a custom base_url to route directly to it. This is the recommended approach for air-gapped environments or regulated industries where every dependency must be audited.

# Install base crewai only — no litellm extra needed
uv add crewai

# In your crew.py, configure each agent's LLM directly:
# from crewai import LLM
# llm = LLM(
#     model="openai/qwen3.5:9b",          # openai/ prefix required
#     base_url="http://localhost:11434/v1",
#     api_key="ollama"                    # Ollama ignores this value
# )

# Verify Ollama's OpenAI-compatible endpoint is working
curl http://localhost:11434/v1/models

Step 3: Configure Ollama as the LLM Provider

Open the .env file in your project root. Set MODEL to the Ollama model string format: ollama/<model_name>:<tag>. Set OPENAI_API_KEY to any non-empty string — LiteLLM requires this variable to exist, even though Ollama doesn't use it. Optionally set OLLAMA_HOST if your Ollama server is not running on localhost.

# .env
MODEL=ollama/llama3.3:70b-instruct-q4_K_M
OPENAI_API_KEY=not-used  # Required by LiteLLM, ignored by Ollama

# Optional: if Ollama is running on a remote machine
# OLLAMA_HOST=http://192.168.1.100:11434

Step 4: Define Your Agents

Edit src/private_crew/config/agents.yaml. Each agent gets a role, goal, and backstory. The llm field overrides the default model for that specific agent — useful if you want different agents to use different local models. If you omit the llm field, the agent uses the MODEL from your .env.

# src/private_crew/config/agents.yaml
researcher:
  role: >-
    Senior Research Analyst
  goal: >-
    Gather and synthesize information from provided documents and context.
    Never access the internet or external APIs.
  backstory: >-
    You are a meticulous analyst who works exclusively with information
    provided in the task context. You do not browse the web.
  llm: ollama/llama3.3:70b-instruct-q4_K_M

writer:
  role: >-
    Technical Writer
  goal: >-
    Transform research findings into clear, structured reports.
  backstory: >-
    You produce precise, well-structured documents from research summaries.
  llm: ollama/llama3.3:70b-instruct-q4_K_M

Step 5: Define Tasks and Run the Crew

Edit src/private_crew/config/tasks.yaml to define what each agent does, then run the crew. The first run will be slower as Ollama loads the model into VRAM. Subsequent runs are faster.

# Run the crew
uv run crewai run

# To verify no network traffic is leaving your machine during the run,
# open a second terminal and monitor outbound connections:
watch -n 1 "ss -tnp | grep python"

Choosing the Right Local Model

Model choice depends on your available VRAM. Llama 3.3 70B (Q4_K_M) requires approximately 40 GB VRAM — suitable for an A100 80 GB or two RTX 3090s. For a single 24 GB GPU (RTX 4090 or RTX 3090), Qwen3.5 27B (Q4_K_M) is a strong choice: it fits in VRAM with headroom, runs at 40–60 tokens/second, and performs well on instruction-following tasks. For CPU-only machines, Llama 3.2 3B or Phi-4-mini (3.8B) are usable but slow. All models are pulled with ollama pull <model>:<tag>.

Power User: Hybrid Crew — Cloud Brain + Local Workers

For maximum efficiency and output quality, the most effective CrewAI pattern is a hybrid crew: a capable cloud model acts as the orchestrator (the "brain") while local Ollama models handle the high-volume, repetitive, or sensitive subtasks (the "workers"). The cloud orchestrator plans the work, decomposes tasks, evaluates agent outputs, and makes final decisions. The local workers do the heavy lifting — document extraction, code generation, classification, structured output — without sending raw data to any external API. The cloud brain sees only task descriptions and synthesised results, not the underlying documents or sensitive content.

from crewai import Agent, Crew, Task, LLM

# Cloud brain — strong instruction-following for planning and evaluation
# GPT-5.4, Claude Sonnet 4.6, or Gemini 3 Pro all work well here
orchestrator_llm = LLM(model="gpt-5.4")              # or "anthropic/claude-sonnet-4-6"

# Local workers — high volume, privacy-sensitive, or repetitive tasks
# These never send raw data to the cloud
worker_llm = LLM(
    model="openai/qwen3.5:9b",
    base_url="http://localhost:11434/v1",
    api_key="ollama"
)

orchestrator = Agent(
    role="Project Manager",
    goal="Decompose the task, delegate to specialists, synthesise their outputs into a final report.",
    backstory="You are a senior strategist. You plan work, review outputs critically, and produce the final deliverable.",
    llm=orchestrator_llm,
)

researcher = Agent(
    role="Document Analyst",
    goal="Extract key facts, figures, and entities from the provided document. Be exhaustive.",
    backstory="You are a meticulous analyst. You work only with the context provided — you do not browse the web.",
    llm=worker_llm,
)

writer = Agent(
    role="Technical Writer",
    goal="Transform research findings into a clear, structured summary.",
    backstory="You produce precise, well-structured documents from raw research notes.",
    llm=worker_llm,
)

crew = Crew(
    agents=[orchestrator, researcher, writer],
    tasks=[...],  # define your tasks as normal
    verbose=True,
)

Hybrid Crew: What Stays Local, What Goes to the Cloud

The key privacy property of the hybrid pattern is that the cloud orchestrator only ever sees what you explicitly pass it — typically task descriptions, agent role definitions, and the final summaries produced by local workers. The raw document content, database records, or sensitive files that local workers process never leave your machine. This makes the hybrid pattern suitable for workloads where some orchestration intelligence is acceptable to route through a cloud API, but the underlying data must remain private.

# What the cloud orchestrator sees:
# - Your task description ("Analyse this contract and identify risk clauses")
# - The local worker's output summary ("Found 3 risk clauses: indemnity (p.4), IP assignment (p.7), auto-renewal (p.12)")
# - Agent role definitions

# What stays on your machine:
# - The actual contract PDF
# - All intermediate extraction steps
# - Any credentials or secrets in the document

# To verify: monitor outbound traffic during a run
watch -n 1 "ss -tnp | grep python"
# You should see:
#   127.0.0.1:11434  ← local worker calls to Ollama
#   api.openai.com   ← orchestrator calls (task descriptions + summaries only)

Troubleshooting: LiteLLM Import Error

If you see 'ModuleNotFoundError: No module named litellm' when running the crew, you installed crewai without the litellm extra. Fix it by running: uv add 'crewai[litellm]' inside your project directory. If you see 'Connection refused' errors pointing to localhost:11434, Ollama is not running — start it with: ollama serve.

# Fix missing litellm
uv add 'crewai[litellm]'

# Fix Ollama not running
ollama serve &

# Verify Ollama is accepting connections
curl http://localhost:11434/api/version