Local models with Ollama.

You probably want Ollama if any of these apply:

• Privacy matters. Source documents never leave your machine. Useful for confidential research, legal notes, medical records, anything you wouldn't paste into ChatGPT.
• You ingest a lot. Pay-per-token costs add up if you're feeding the wiki hundreds of sources. Local is free per query (electricity only) after the one-time model download.
• You want offline capability. Once a model is pulled, Ollama works without internet. Useful on flights, in secure environments, or when your connection is flaky.
• You're experimenting. Try different model sizes, compare output styles, learn how LLMs actually behave — all without burning API credits.

You probably want OpenRouter (cloud) if:

• Quality matters most — frontier models (Claude 4.6, GPT-4o, Gemini 2.5 Pro) are still substantially smarter than the best open models you can run locally.
• Your hardware is modest. A 5-year-old laptop with 8 GB RAM can run small local models, but slowly and badly. Cloud is always fast.
• You only need it occasionally. $5 of OpenRouter credit can last weeks at default models. Below that threshold, the mental cost of managing a local model is more than the dollar cost of the cloud one.

Mixed usage works fine. You can set Ingest to Ollama (heavy, you-don't-want-to-watch-it) and Chat to OpenRouter (interactive, want frontier quality). LLM Wiki picks per-slot.

One-time install. Pick your OS:

macOS

# With Homebrew (recommended — auto-starts a background service)
brew install ollama

# Or download the .dmg installer from https://ollama.com/download

Apple Silicon (M1+) gets GPU acceleration out of the box. Intel Macs work but slower.

Linux

curl -fsSL https://ollama.com/install.sh | sh

NVIDIA GPUs auto-detected via CUDA. AMD has limited support — see Ollama GPU docs.

Windows

Download the installer from ollama.com/download. Runs natively on Windows 10/11; works inside WSL too if you prefer.

Verify it's running

curl http://localhost:11434/api/version

Should print something like {"version":"0.x.x"}. If you get connection refused, run ollama serve in a terminal to start the service manually.

Models aren't included with Ollama itself — you download each one once and they're cached locally. From a terminal:

# General-purpose, fast, good default
ollama pull llama3

# Smaller + faster, lower quality
ollama pull phi3

# Vision-capable (for PDFs and images)
ollama pull llava

# Full library: https://ollama.com/library

Download size + speed depends on your connection — count on 1-2 minutes for a 4-5 GB model on broadband. Models persist in ~/.ollama/models/ and you only need to pull each one once.

To see what you've pulled: ollama list. To remove one: ollama rm <name>.

The single biggest factor in whether a local model is useful or painful is whether your hardware can run it comfortably. Numbers below assume 4-bit quantized versions (Ollama's default — half the memory of full precision, near-identical quality for most use cases).

Speed numbers are tokens/sec on the listed hardware, rough order-of-magnitude. Real-world varies ±50%.

Quick picker

• Most modern laptop (8-16 GB RAM) → llama3 or mistral. Reliable, balanced.
• Old / underpowered machine (≤8 GB RAM) → phi3. Still usable. Output quality drops.
• Apple Silicon 16-32 GB → llama3 for general, phi3:medium when you want more quality and don't mind slower.
• Apple Silicon 64+ GB unified, or workstation with 64+ GB RAM → llama3:70b or mixtral. Real frontier-ish quality, fully local.
• Need vision (PDFs, images) → llava for quality, moondream for speed.

Rule of thumb on RAM: the model needs roughly its file-size in RAM, plus 2-4 GB for the OS, plus context window overhead. Running a model that's bigger than your free RAM will use swap, which drops speeds by 10-50x and is usually painful enough to be unusable.

With Ollama running and at least one model pulled:

1. Open Settings → Models
2. For any operation slot (ingest / query / chat / lint / vision): change the Provider dropdown from OpenRouter to Ollama (Local)
3. Pick a model from the dropdown — only pick models you've actually pulled via ollama pull <name>
4. Click Save. New operations on that slot route to Ollama immediately.

You can mix providers per slot. A common pattern: Ingest on Ollama (slow but free, good for batch work), Chat on OpenRouter (fast and smart for interactive use), Vision on whichever has the better vision model for your case.

If you set ALL slots to Ollama, you don't need an OpenRouter API key at all. The first-run wizard's key step becomes optional once at least one slot is Ollama.

Ollama runs on http://localhost:11434 by default. If yours runs elsewhere (different port, on another machine via tunnel, inside a Docker network), set the OLLAMA_BASE_URL environment variable before starting LLM Wiki:

# Example: Ollama running on a different port
export OLLAMA_BASE_URL=http://localhost:12345
llm-wiki start

# Example: Ollama on another machine on your LAN
export OLLAMA_BASE_URL=http://192.168.1.100:11434
llm-wiki start

# Example: Ollama exposed via a tunnel
export OLLAMA_BASE_URL=https://my-tunnel.example.com
llm-wiki start

LLM Wiki appends /v1 to whatever you set, matching Ollama's OpenAI-compatible API. No need to include it yourself.

• “Connection error” when running an operation

  Ollama isn't running. Try curl http://localhost:11434/api/version — if it fails, run ollama serve in a terminal.

• “Model not found” or 404 from Ollama

  You picked a model in Settings that you haven't pulled. Run ollama list to see what's available; ollama pull <name> to add one.

• Responses are very slow (under 5 tokens/sec, painful to read)

  Your hardware is below the model's comfortable range. Try a smaller model (phi3 instead of mistral, mistral instead of mixtral). Or accept that batch operations (ingest, lint) work fine and only chat is painful — and use OpenRouter for chat.

• Out of memory / system swap-thrashing during operations

  Same as slow: pick a smaller model. Or close other apps to free RAM. As a hard rule, the model's file size + 4 GB should comfortably fit in your free RAM.

• GPU isn't being used (CPU pegged, GPU idle)

  NVIDIA: check nvidia-smi while a query runs. AMD: support is incomplete. Apple Silicon: GPU is always used, no toggle. See Ollama GPU docs.

• “OpenRouter API key not configured” even though only some slots use OpenRouter

  At least one slot is still set to OpenRouter and needs the key. Either set ALL slots to Ollama, or add an OpenRouter key in Settings → API.

• ollama.com/library — full list of available models with sizes + benchmarks
• github.com/ollama/ollama — source code + issue tracker
• GPU compatibility docs — what works on what
• Chatbot Arena Leaderboard — independent quality rankings of LLMs (open + closed)