Run LLMs Locally with Ollama

Why run a model locally?

Most AI tutorials assume a cloud API, but you can run capable open models on your own machine — no API key, no per-token bill, and your data never leaves your computer. Ollama makes this almost trivial: one command downloads a model and starts chatting. It's perfect for private data, offline work, experimentation, and learning how models behave without watching a meter run.

Install and run

After installing Ollama from ollama.com, a single command pulls a model and drops you into a chat:

# after installing from ollama.com
ollama run llama3.2

The first run downloads the model (a few gigabytes); after that it's fully offline. Try other models — mistral, gemma, qwen — with the same command.

Use it from code

Here's what makes Ollama genuinely useful for builders: it exposes an OpenAI-compatible endpoint, so the exact code you'd write for a cloud API works locally by changing the base URL:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
print(client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Hello!"}],
).choices[0].message.content)

That means you can prototype against a free local model and switch to a hosted one later by changing two lines.

When to use local models

Local models are the right call when you value:

• Privacy — sensitive data (medical, legal, personal) never leaves your machine.
• Cost — no per-token charges, ideal for high-volume or experimentation.
• Offline — they work on a plane or behind a firewall.

The trade-off: a model that fits on your laptop is smaller and generally less capable than a frontier cloud model, and speed depends on your hardware. For many tasks — drafting, summarizing, classification — that trade is well worth it.

The takeaway

Ollama turns "run an LLM yourself" from a weekend project into one command, with an API that mirrors the cloud. It's the easiest on-ramp to private, free, offline AI — and a great way to learn by tinkering.

Try it now: Run ollama run llama3.2, ask it something, then point the Python snippet at it. Realizing your local model speaks the same API as the cloud is what makes it click as a real building block.