Ollama: Running Open LLMs Locally

Running an open-weight model yourself normally means wrangling weight files, quantisation formats, and a serving runtime. Ollama collapses that into one command: name a model, and it downloads, prepares, and serves it, exposing both a chat CLI and a local HTTP API that speaks the shape most client libraries already expect.

ollama pull llama3.1:8b
ollama run llama3.1:8b          # interactive session
ollama serve                     # HTTP API on localhost:11434

Because the API is OpenAI-compatible, pointing an existing application at a local model is usually a base URL change rather than a rewrite:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")  # key unused
response = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Summarise this ticket in one line."}],
)

That compatibility is the practical reason to start here: you can prototype against a hosted provider and swap in a local model later, or the reverse, without restructuring the application.

Memory is the whole constraint

The question that decides everything is whether the model fits in fast memory. A rough estimate: parameters multiplied by bytes per parameter, plus roughly a fifth again for the KV cache and overhead. At the common 4-bit quantisation that is about half a byte per parameter, so a 7-to-8-billion-parameter model wants somewhere around 5–6 GB, a 13B around 9–10 GB, and a 70B around 40 GB or more.

The cliff matters more than the numbers. A model that fits entirely in GPU memory — or in unified memory on Apple silicon — runs at interactive speed. A model that spills to system RAM does not run somewhat slower; it runs at a fraction of the speed, because generation becomes bound by the slowest memory in the path. The symptom is tokens arriving at reading pace or worse, and the fix is a smaller model or a heavier quantisation, not patience.

Context length is the part people forget. The KV cache grows with the conversation, so a model that fits comfortably at 4k context can exceed memory at 32k, and long-document workloads hit this well before the weights are the problem.

Reading a quantisation tag

Tags like q4_K_M encode the compression. The number is bits per weight; the suffix is the scheme and size variant.

Full precision is the reference and rarely worth running locally. 8-bit is essentially indistinguishable from it for most tasks at half the memory. 4-bit is the sweet spot and where you should start — quality loss is small enough to be hard to detect in normal use, and it is what makes a capable model fit on ordinary hardware. Below 4-bit, degradation becomes noticeable, and it shows up first in exactly the places that matter: instruction following, structured output, long-context coherence, and code.

The useful rule when choosing between two options that both fit: a larger model at 4-bit generally beats a smaller model at 8-bit. Parameter count buys more than precision does.

What belongs local

Local inference wins on four axes and loses on one.

It wins on privacy, because nothing leaves the machine, which is often decisive for regulated data. On cost at volume, because after the hardware there is no per-token charge, and a high-volume classification or extraction job that a small model handles well can be dramatically cheaper. On stability, because the model does not change underneath you when a provider ships an update. And on availability, because it works offline and has no rate limits.

It loses on peak capability. The largest hosted models remain ahead of what runs on a laptop, and for the hardest reasoning and coding tasks the gap is still visible.

So the split that works: local for high-volume narrow tasks, privacy-sensitive data, offline development, and anything where a stable model matters more than the best model. Hosted for the hard tail. Many teams run both behind one interface and route by task, which the OpenAI-compatible API makes nearly free to implement.

Common mistakes

Choosing the model by what fits rather than what the task needs. Both directions hurt — too small produces disappointing output, too large produces unusable speed.

Ignoring context length in the memory estimate. The KV cache is what pushes a working setup over the edge.

Quantising below 4-bit to fit. Instruction following and structured output degrade first, which is usually what you needed.

Assuming local means capable. Benchmark on your actual task before replacing a hosted call.