Type ollama into your terminal today. What starts is not a local model.

Since version 0.32.0 of 11 July 2026, the bare command opens an agent — and Ollama’s own release notes say what with: “Chat, Code, & Work (glm-5.2:cloud)”. A cloud model. The most popular tool for local AI sends you online on its default invocation.

That’s not a scandal, it’s a product decision, and you can switch it off. But it’s the perfect opening for this piece. Because “local” has become a word you have to check rather than believe.

Why you’d do this at all

Let’s start with what actually holds up — and it isn’t the money.

If you run an AI on your own servers exclusively for your own purposes, then in the view of the German data protection authorities the processor relationship disappears. You are the sole controller. No data processing agreement under Art. 28(3) GDPR, no sub-processor chain, no third-country transfer, no adequacy decision that might collapse next year.

That’s the point worth understanding: local processing is not a risk reduction. It’s the structural removal of an entire chain of obligations. The supervisory authorities explicitly recommend local processing, for instance for further training and fine-tuning.

And the common worry that running your own AI drags regulatory duties into your home misses the private user entirely: under Art. 2(10) the AI Act does not apply at all to purely personal, non-professional use, and Art. 2(12) generally exempts free open-source systems.

(This is a rendering of authority guidance and statutory text as of 24 July 2026 — not legal advice. For an actual business case, this belongs with your data protection officer.)

What fits on your card

The most honest VRAM information officially available is the file sizes in Ollama’s library. Roughly: you need about as much video memory as the file is large, plus headroom.

ModelSize (Q4)Fits on
Gemma 4, 12B7.6 GB8 GB card, barely
gpt-oss-20bdesigned for 16 GB16 GB card
Qwen3.6-27B17 GB24 GB card
Gemma 4, 31B20 GB24 GB card
Laguna XS 2.1 (q4)20 GB24 GB card
Qwen3.6-35B24 GB24 GB, very tight

The realistic entry threshold is therefore a 16 GB card. At a large German retailer an RTX 5060 Ti with 16 GB was listed at €555.85 (promotional price), the cheapest regular 16 GB card at €559.64. The 32 GB card (RTX 5090) costs a multiple of that, at €4,248.99.

Two things about this are rarely presented correctly:

Quantisation doesn’t just make models smaller, it makes them faster. On the standard Llama-3.1-8B example, the 4-bit variant generates 71.93 tokens per second against 29.17 for the 16-bit variant — a factor of 2.5. The reason: text generation is memory-bandwidth-bound. Fewer bits means less to read. The widespread notion that quantisation is a pure quality compromise you accept only for lack of space is simply not right.

Electricity doesn’t matter. At one hour of full load per day, the 5060 Ti costs about €2.22 a month, the 5090 about €7.09 — calculated with the official German household electricity price of 40.55 cents per kilowatt-hour. Even at four hours a day it’s €8.88 and €28.35. The argument that “local AI doesn’t pay off because of power costs” is off the table. The decision is made purely on purchase price.

The arithmetic almost nobody writes down

And the purchase price decides against you.

€555.85 corresponds — at the exchange rate on the retrieval date — to roughly 253 million output tokens on Gemini 3.5 Flash-Lite. On Claude Haiku 4.5 it’s 126 million, on Claude Sonnet 5 still 63 million, on GPT-5.6 Sol a respectable 21 million.

Sit with what that means. Someone generating 5,000 tokens a day — and that is already intensive daily use — would need roughly 138 years to burn through 253 million tokens.

Expressed as break-even: the entry-level card pays for itself after around 71 months, six years, at €10 of monthly API spend. At €25, after a good two years. Only at €50 a month does it get interesting, at just under twelve months. The 5090 at €50 a month takes about eight years.

Local AI is almost never the cheaper option financially. Anyone justifying it with “it saves money in the long run” is, as a rule, calculating wrongly. You buy it for data sovereignty — which is a real value, but a different one.

One honest exception: if compliance already requires European data residency, both large US providers charge exactly a 10% surcharge — OpenAI calls it “a 10% uplift”, Anthropic a factor of 1.1. That shifts the arithmetic slightly, but not dramatically.

The capability gap — and the trick inside the number

A lot gets conflated here, and the distinction is decisive.

“Open models have almost caught up” is true — in Artificial Analysis’s Intelligence Index, Kimi K2.6 and MiMo V2.5 Pro sit at 54 points, DeepSeek V4 Pro at 52, against 60 for GPT-5.5 at the top. A year earlier the gap was considerably wider.

“I can run that at home” is a different statement about different models. Kimi K2.6 has a trillion parameters. DeepSeek V4 Pro 1.6 trillion. These are datacentre models. They’re open — but that doesn’t make them runnable on your graphics card.

What actually fits on a single card sits further back: Qwen3.6-27B scores 37 index points — rank 1 in its size class, but 37 against 60. gpt-oss-20b sits at 15.

And the cross-check in blind comparison is starker still: in the LMArena leaderboard there is not a single open model in the top 15. The best open model sits at rank 29.

In fairness: that arena’s methodology has been documented and criticised (“The Leaderboard Illusion”), partly because a very large share of the data comes from a few commercial providers. The critique, however, comes from authors themselves close to a model provider. Both belong side by side.

My read: for summarising, rephrasing, structuring, simple coding, a 27B model on a 24 GB card is entirely sufficient. For the hard cases — long reasoning, nested tasks, agentic work — you feel those 23 points immediately.

Where “local” isn’t local

This is the part no guide writes, because it’s inconvenient.

Ollama’s default invocation. See above: since v0.32.0, ollama starts an agent on a cloud model. Cloud models sit in the same library listing as local ones, distinguished only by the tag suffix — gemma4:31b runs on your machine, gemma4:31b-cloud does not. One character of difference.

The agent’s web search runs through Ollama’s own servers and requires an account and an API key.

Even purely local use produces metadata. Ollama’s privacy policy separates this cleanly — content is not collected locally, explicitly: “We do not collect, store, transmit, or have access to your prompts, responses, model interactions, or other content you process locally.” Device and usage metadata, however, are.

LM Studio declares itself telemetry-free and at the same time openly names the five functions that need the internet: model search, model download, catalogue statistics, runtime downloads, update checks. That’s the exemplary version — the boundary is in the docs.

And with one of the best-known models, the first step towards sovereignty is handing over your identity: to download the “open” Llama 4 weights, Meta requires your full legal name and date of birth. The Llama licence isn’t an open-source licence in any case — cross 700 million monthly active users and you must ask Meta for permission, granted at its sole discretion.

Which model to pick if the licence matters to you

If “open” is supposed to mean actually open, the licence is the first criterion — and something has changed here that almost every older guide still gets wrong.

Gemma 4 has been under Apache 2.0 since its release on 31 March 2026 — the licence page carries the full, unmodified text. All earlier Gemma generations had a restrictive house licence in which Google even reserved the right to restrict usage remotely. That still applies to Gemma 3 and older. If you’re reading a guide that says “Gemma has its own licence”, you’re reading an outdated guide.

Qwen3.6-27B is under unmodified Apache 2.0 with no additional clauses — and is simultaneously the strongest model in its size class. If I had to recommend one thing for a 24 GB card, it would be this.

gpt-oss-20b is likewise Apache 2.0 and, per its model card, explicitly designed for 16 GB — the most comfortable entry point on the entry-level card.

What you give up

So this doesn’t turn into an advert, here’s the honest price:

Speed. An entry-level card is not a datacentre. With long contexts and long reasoning, you wait.

Capability. 37 against 60 index points is not a rounding error. Your local model handles the hard tasks worse.

Maintenance. Drivers, quantisations, model updates, broken dependencies. That’s work nobody bills you for with an API, because someone else does it.

And one gap I couldn’t close: I found no statement in the official documentation of either Ollama or LM Studio about whether downloaded model weights are cryptographically verified for provenance. For a file you execute on your own machine, that’s a detail I would have liked to have.

My take

Don’t buy a graphics card to save money. The arithmetic almost never works out, and anyone showing you otherwise has flattered either your usage or the card price.

Buy one if it matters to you that certain texts never leave your house — client files, patient records, manuscripts, contracts, anything where “where is this right now?” needs a real answer. For that, local AI isn’t the cheaper solution but the only clean one, because then no third party sits in the chain at all.

And if you’re starting out: start small. A 16 GB card, gpt-oss-20b or a mid-sized Gemma 4, and spend a week honestly watching what you actually use. After that you’ll know whether the 24 GB card is worth it — better than guessing beforehand.

The tools around all this are in my continuously maintained index of 137 AI tools; which of the commercial models is good for what is in the comparison of ChatGPT, Claude and Gemini.

Sources

All checked on 24 July 2026. Prices, model states and licences change; hardware prices are retailer prices as of the retrieval date.

If something here looks outdated or wrong to you: get in touch.