Run Ollama on RunPod: A Complete Privacy-First Setup

RunPod's Secure Cloud gives you a dedicated GPU that never shares hardware with other tenants. Combined with Ollama, you get a fully private LLM API endpoint that you control end-to-end — no data leaves your pod. This guide walks through the complete setup: provisioning the pod, installing Ollama, pulling a model, and exposing a secure API endpoint back to your local machine.

Why RunPod + Ollama?

Most cloud LLM APIs (OpenAI, Anthropic, Google) log your prompts for safety review and model improvement. If you're working with sensitive business data, legal documents, medical records, or personal projects you'd rather keep private, that's a non-starter. RunPod Secure Cloud gives you a bare-metal GPU pod — no shared tenancy, no hypervisor logging your traffic. Ollama is the simplest way to run open-weight models (Llama 4, Mistral, DeepSeek-R1, Gemma 3) with a clean HTTP API that mirrors the OpenAI spec, so your existing tooling works without modification.

Step 1: Provision a RunPod Secure Cloud Pod

Log into RunPod and navigate to Secure Cloud. Select a GPU — for most 7B-14B models, an RTX 4090 (24 GB VRAM, ~$0.59/hr) is sufficient. For 70B+ models, you'll want an A100 80 GB or H100. Choose the Ubuntu 22.04 base template. Set the container disk to at least 50 GB to accommodate model weights. Under 'Expose HTTP Ports', add port 11434 (Ollama's default). Under 'TCP Ports', add port 22 for SSH access. Deploy the pod and wait for it to reach Running status.

Step 2: Install Ollama

SSH into your pod using the credentials shown in the RunPod console. Then run the official Ollama installer:

# SSH into your pod
ssh root@<pod-ip> -p <ssh-port>

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Start Ollama as a background service
OLLAMA_HOST=0.0.0.0 ollama serve &

# Verify it's running
curl http://localhost:11434/api/version

Step 3: Pull a Model

With Ollama running, pull the model you want. For a good balance of capability and speed on a 24 GB GPU, Llama 3.3 70B in Q4 quantization fits in VRAM and runs at approximately 25-35 tokens/second:

# Pull Llama 3.3 70B (Q4_K_M quantization, ~40 GB)
# Requires A100 80GB or 2x 3090s
ollama pull llama3.3:70b-instruct-q4_K_M

# For a 24 GB GPU (RTX 4090), use a 14B or 32B model
ollama pull qwen3.5:27b

# Verify the model is available
ollama list

Step 4: Expose a Secure Endpoint

RunPod automatically exposes port 11434 via their proxy. Your endpoint URL will look like: https://<pod-id>-11434.proxy.runpod.net. However, this is publicly accessible to anyone who knows the URL. For a private setup, use SSH port forwarding instead — this tunnels the API through your encrypted SSH connection and never exposes it to the public internet:

# On your local machine — forward RunPod's port 11434 to localhost:11434
ssh -L 11434:localhost:11434 root@<pod-ip> -p <ssh-port> -N &

# Test from your local machine
curl http://localhost:11434/api/generate \
  -d '{"model": "qwen3.5:27b", "prompt": "Hello, are you private?", "stream": false}'

Step 5: Connect Your Tools

Because Ollama's API is OpenAI-compatible, you can point any tool that supports a custom base URL at your private endpoint. In AnythingLLM, set the LLM provider to 'Ollama' and the base URL to http://localhost:11434. In Continue.dev (VS Code), set the model provider to 'ollama' with the same URL. For the OpenAI Python SDK, set base_url='http://localhost:11434/v1' and api_key='ollama' (any non-empty string works).

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # required but ignored
)

response = client.chat.completions.create(
    model="qwen3.5:27b",
    messages=[{"role": "user", "content": "Summarize this contract clause: ..."}],
)
print(response.choices[0].message.content)

Cost Estimate

An RTX 4090 Secure Cloud pod on RunPod costs approximately $0.59/hour. If you run it 8 hours/day for a month, that's roughly $142/month — less than most enterprise LLM API subscriptions, with complete data privacy. You can also stop the pod when not in use and only pay for active compute time. Model weights are stored on the pod's persistent volume, so you don't need to re-download on restart.