Local LLMs (Ollama, LM Studio) vs a Private Gateway:
An Honest Speed, Quality and Privacy Test

Published August 6, 2026

Installing a local model has never been easier. You download Ollama or LM Studio, type one command or click one button, and a few minutes later a chatbot is answering you from your own machine with the Wi-Fi switched off. Nothing is transmitted. No account, no provider, no terms of service. For anyone who has ever hesitated before pasting something sensitive into a cloud chatbot, that first offline reply feels like the whole problem solved.

Then you ask it something hard, and the feeling changes.

This article is the honest version of that comparison — local models on your own hardware against a privacy-focused gateway to the big commercial models — measured on the three axes people actually care about: speed, quality and privacy. It contains one conclusion most write-ups tiptoe around, so here it is up front: if you want answers at the level of GPT, Claude or Gemini running on your own hardware, you do not need a good computer. You need a very, very serious server — and in 2026 the parts for it got harder to buy, not easier.

First, What Are You Actually Running at Home?

There is a gap between the model people believe they are running locally and the model that is really loaded.

When you pull a model in Ollama without specifying anything, you get a 4-bit quantized build — Q4_K_M is the library default. Quantization compresses the numbers inside the model so it fits in less memory; the rule of thumb is a little under one gigabyte per billion parameters at 4 bits, plus headroom for the conversation itself. That is why the popular local models cluster where they do: on a 16 GB laptop, a 7–8 billion parameter model lands around 5 GB and leaves room to work. A 32 GB machine reaches into the 20–30 billion range. That is the honest consumer envelope.

Those models are not toys. An 8B model is genuinely good at summarizing a document you paste in, reformatting messy notes, drafting boilerplate, extracting fields from text, renaming things, and answering questions about material you supply in the prompt. If that is your workload, stop reading and go install one — you will be happy, and your data will never leave the room.

But an 8B model is not what people mean when they say "I'll just run it locally instead of paying for ChatGPT." They mean the reasoning, the long-context synthesis, the code that compiles, the answer that shows judgement. That lives somewhere else entirely.

The Models That Rival GPT, Claude and Gemini Are Server-Class. Full Stop.

Here is the fact that reframes the whole debate. The open-weight models that genuinely compete with the frontier do exist — you can download them today, free, no permission needed. They are also enormous.

• Kimi K3 (Moonshot AI) is a sparse mixture-of-experts model with 2.8 trillion total parameters, 896 experts of which 16 are selected per token, and 104B parameters activated per token, published with MXFP4 weights. Even in that compressed 4-bit format the weight files run to roughly 1.5 terabytes. They do not fit on one GPU. They do not fit on one eight-GPU node. Serving this model is a multi-node cluster job measured in dozens of accelerators.
• GLM-5.2 (Z.ai) is around 744 billion total parameters with roughly 40B active.
• DeepSeek's flagship line has sat in the same territory since the 671B generation, whose 4-bit build alone occupies about 404 GB — before you have loaded a single token of context.

Read those numbers next to the consumer envelope above. The distance between "a 5 GB file on my laptop" and "1.5 TB spread across a cluster" is not a gap you close with a better graphics card. It is a difference of roughly three hundred times, and it is why the phrase "run it locally" quietly means two completely unrelated things depending on who is saying it.

So the honest sentence, the one worth remembering: the open-weight models are free, but the hardware to run them at frontier quality is a data-centre purchase. You are not buying a model. You are buying a machine room.

The Speed Test, With Real Numbers

Suppose you go most of the way anyway and buy the strongest single box a person can reasonably own. What actually happens?

The clearest published measurement is a Mac Studio with M3 Ultra and 512 GB of unified memory — the configuration people bought specifically because its memory pool could swallow a 671B model — running DeepSeek 671B in 4-bit through llama.cpp. Feed it an 8,001-token prompt (a long email thread, a contract, a few pages of code) and, as measured:

• Prompt processing: 888 seconds — 14.8 minutes before the first word appears, at 9.01 tokens/second.
• Generation: 6.21 tokens/second, taking another 2.4 minutes.
• Total: 17.2 minutes for one answer.

Fairness matters here, so: with Apple's own MLX runtime instead of llama.cpp, the same class of machine has been reported at around 20 tokens/second of generation on a DeepSeek model — perfectly usable — but that figure applies to small context sizes. And that is precisely the trap. Two different walls exist, and only one of them is the one everybody benchmarks:

• Generation speed is governed by memory bandwidth. It is what every YouTube demo shows, and it looks fine.
• Prompt processing — reading everything you sent before it may answer — is governed by raw compute, and it is where long prompts go to die. A short "write me a haiku" prompt hides this completely. A real working prompt does not.

The picture on NVIDIA's compact machines is the same story with different scenery. The DGX Spark packs 128 GB of unified memory into a small box at $4,699, but that memory runs at 273 GB/s, and bandwidth is destiny. Benchmarked, it generates at 4.42 tokens/second on a 70B dense model in 4-bit — slower than a person types. On a 120B mixture-of-experts model, where only a fraction of the parameters fire per token, the same box manages 38.55 tokens/second, while a Mac Studio M3 Ultra with 819 GB/s of bandwidth reaches 70.79.

Which points at the one genuinely encouraging fact in this whole comparison: sparse mixture-of-experts models run far better on modest hardware than their size suggests, because only a few experts activate per token. That is real, and it is why local AI has improved so much. But it rescues the speed problem only. The capacity problem is untouched: every one of those 2.8 trillion parameters still has to be resident in memory, because the model chooses which experts to use per token and cannot know in advance which ones. You still need the whole 1.5 TB. You just get to read it more slowly.

2026 Made the Hardware Harder to Buy, Not Easier

The standard reply to all this is "hardware gets cheaper, wait a year." That has not been true this year. The memory shortage rewrote the shopping list in both directions:

• Apple removed the big memory options. The 512 GB Mac Studio configuration disappeared in March 2026 amid the global DRAM shortage, and 256 GB was cut in May. The machine that made "run a 671B model at home" a real sentence is no longer on sale in that form — the M3 Ultra now tops out at 96 GB.
• The big professional GPU went up, not down. NVIDIA's RTX PRO 6000 Blackwell with 96 GB launched in March 2025 at an $8,565 MSRP; by August 2026 NVIDIA's own marketplace listed it near $13,250 — roughly a 55% increase in sixteen months, driven by the GDDR7 supply crunch rather than manufacturing cost.

Put those together and the arithmetic gets uncomfortable. At around 96 GB per card and roughly thirteen thousand dollars each, assembling the ~1.6 TB of memory that a top open-weight model needs means somewhere near seventeen of them — a couple of hundred thousand dollars in graphics cards alone, before the chassis, the interconnect, the power delivery, the cooling, or the person who keeps it running. That is the "very, very strong server" in plain figures. It is not hyperbole and it is not a straw man; it is what the leaderboard-topping open model requires to answer one question.

Quality: The Gap Narrowed — But Not Where Your Laptop Lives

Now the part that is genuinely good news for open models, and which makes the hardware conclusion sharper rather than softer.

The capability distance between open-weight models and closed frontier models has shrunk dramatically. It is no longer a matter of years. In August 2026 TechCrunch reported on a SaferAI evaluation finding GLM-5.2 only a few months behind the leading closed models on the cybersecurity and biology capabilities it tested — while noting that safety mitigations are diverging even as capabilities converge, with no published safety framework or pre-deployment testing for that release.

But look at which models closed that gap: 744 billion parameters. 2.8 trillion. The convergence happened at the top of the size chart, not at the bottom. The gap between the best open model and the best closed model narrowed to months; the gap between the best open model and the model on your laptop stayed exactly as wide as it always was. Those are two different lines on two different charts, and conflating them is how a genuinely exciting headline turns into a disappointing weekend.

Judge a small local model by what it is, and it is excellent: fast, free, offline, private, and completely adequate for transformation work on text you supply. Judge it against a frontier model on a hard reasoning question and you will feel the difference in the first paragraph.

Privacy: Local Wins the Data, and We Will Say So Plainly

We sell a gateway, so let us be direct about the one axis where we do not win.

A model running offline on your own machine transmits nothing. There is no provider, no retention policy to read, no account, nobody to breach, and nobody to subpoena for your prompts. For material whose exposure would be catastrophic or unlawful, that certainty is the feature, and no cloud arrangement — ours included — can match it. If that is your threat model, the right answer is a local model and this article has just told you to use one.

What a gateway offers instead is a different kind of protection, and it should be described precisely rather than sold as the same thing. Secret Chat AI is an anonymizer, not a content filter: it removes you from your queries — it does not remove the data from your messages. Concretely:

• We build no profile of you. There is no behavioural model of your interests assembled from what you ask.
• No chat is ever associated with you. Your conversations live in your own browser's local storage; a prompt exists on our side only for as long as it takes to fetch your answer. There is no stored chat archive on our servers.
• Your queries reach the top models anonymously. Whatever a provider holds under its own terms and configuration, the record on that side carries our gateway's credentials and server address — not your name, your account, or your IP. You use the model as a stranger.
• Your queries are never used for training.

Two caveats that belong in the same breath. "Anonymously" describes the link, not the words: no account identifier travels with your prompt, but it is not a claim that the text stops being identifying — write your own name or a case number into a message and it is all still sitting there in the message. And registration does take an email; it is used only for account access and payment, never stored against your prompts or associated with them. Retention may still apply at the provider, but your query reaches the model anonymized, not linked to your identity.

Anonymity is not privilege, not a legal exemption, and not a way to keep anything from a court that is entitled to it. Redacting identifying details before you send remains your responsibility, and pasting someone else's personal data makes you responsible for that disclosure. We keep ordinary infrastructure telemetry — IP addresses, request timestamps, error codes — for at most 30 days, which is not a record of what you asked.

The local footguns people forget

Running locally removes the provider from the picture. It does not automatically make you private, and three things catch people out:

• The runner is software with a network stack. "The model runs offline" and "this application never contacts anything" are separate claims. Check your tool's settings and its actual network behaviour yourself rather than assuming — and if it matters, verify it with the network disconnected.
• Your disk becomes the archive. The cloud chat history you were trying to avoid now exists as local files instead. That is better — but only if the drive is encrypted, backed up sensibly, and not sitting in a laptop that can be lost, stolen, borrowed by a colleague, imaged by an employer, or produced in discovery.
• Downloaded weights come with their own baggage. Open weights move the hosting, not the training. Whatever a model was trained to refuse, believe, or slant travels with the file — which is a separate discussion we work through in DeepSeek without the Chinese servers.

The Cost Nobody Budgets

The hardware price is the visible number. The invisible ones matter more:

• Electricity and heat, continuously, for a machine that idles expensively and peaks worse.
• Your time — quantization formats, runtime updates, context settings, driver breakage, and the evening you lose to discovering that the model fits but the context window does not.
• Depreciation on a fast-moving target. This is the cruel one. Buy a machine sized precisely for today's best open model and a new release arrives a few months later that is better and bigger, and your very expensive box is now one generation behind on a curve that has not flattened.

A gateway inverts that shape: no capital outlay, no depreciation, and model upgrades arrive as a menu item rather than a purchase order. Our own plans are on the pricing page — but the comparison worth making is not "subscription versus free," it is "subscription versus a server, its power bill, and its obsolescence schedule."

The Split That Actually Works

Most people who have run both for a while end up in the same place, and it is not a compromise so much as correct tool selection: a small local model for the confidential and the mechanical, a gateway to the frontier models for the hard thinking — with identifiers stripped either way.

Made concrete, that means the local model does the first pass on anything sensitive: extracting, restructuring, summarizing and de-identifying, all offline. Only the residue that genuinely needs frontier reasoning travels, and it travels without the details that made the original sensitive. A prompt to run in your local model to do exactly that:

The output of that prompt is what you paste into a cloud model. The input never leaves.

How to Choose

• How bad is a leak, honestly? Catastrophic or unlawful — keep it local and offline. Unwelcome but survivable, and de-identifiable first — a gateway fits.
• Is the task transformation or reasoning? Reformatting, summarizing and extracting from text you supply is comfortably local work. Nuanced judgement, hard analysis and long-context synthesis is where the frontier models still earn their keep.
• How long is your prompt? This is the question people skip. Short prompts hide the prompt-processing wall completely; long ones run straight into it. If you routinely paste in pages, benchmark that before you buy anything.
• Do you want to own a machine, or use one? A server is a standing commitment — power, upkeep, and a depreciation curve. If nobody owns that upkeep, it will quietly rot.

For the conceptual version of this trade-off — control versus capability, and where each genuinely fits — see our companion piece on secure AI gateways versus self-hosting. For the wider field of privacy-focused options, the best private AI chatbots in 2026 roundup covers the alternatives, and our model line-up shows what running anonymously through a gateway actually gets you.

The Bottom Line

Local models won a real victory, and it deserves to be stated without qualification: an offline model on your own hardware transmits nothing, and for genuinely sensitive material that is the strongest posture available. Ollama and LM Studio made it a ten-minute setup. The small models are far better than they were, and mixture-of-experts architectures have made them faster on ordinary hardware than anyone expected.

What did not change is the ceiling. Frontier-level answers come from models measured in hundreds of billions to trillions of parameters, all of which must be resident in memory at once — 1.5 terabytes of it for the current leader — while the hardware to hold that has been getting scarcer and more expensive through 2026. To have GPT-, Claude- or Gemini-class quality answering to you alone, you must buy a very, very strong server: not a good laptop, not a maxed-out desktop, but a cluster. Almost nobody's threat model justifies that, and the ones that do usually have a budget line for it.

For everyone else, the sensible arrangement is the split: keep the crown jewels on a small local model that never speaks to the network, and reach the frontier models through a gateway that keeps you unlinked from what you ask — Secret Chat AI being built for exactly that job. It will never be as absolute as an offline machine, and we would rather say so than pretend otherwise. What it is, is the version of cloud AI where nobody is building a profile of you — at a fraction of the price of the machine room it would take to do better.

Frequently Asked Questions

1. Can I run a GPT- or Claude-level model on my laptop with Ollama or LM Studio?

   No. The open-weight models that genuinely rival the frontier are enormous — Kimi K3 carries 2.8 trillion parameters and roughly 1.5 TB of weights even in 4-bit, and comparable models sit in the hundreds of billions. All of it has to be resident in memory at once. A laptop realistically runs a 4-bit 7–8B model, which is a useful tool for summarizing and reformatting but is not in the same class. Reaching frontier quality on your own hardware means a multi-GPU server, not a better laptop.

2. What hardware would I actually need to run the best open model at home?

   Roughly 1.6 TB of aggregate GPU memory and dozens of accelerators, deployed across more than one node. With the largest professional cards at 96 GB each and listed near $13,250 as of August 2026, that is well over a hundred thousand dollars in GPUs before the chassis, interconnect, power and cooling. Smaller open models run on far less, but they are correspondingly less capable.

3. Is a local model actually more private than a privacy-focused gateway?

   For the data itself, yes — an offline model transmits nothing, so there is no provider to retain, breach or subpoena. A gateway cannot match that and should not claim to. What a gateway offers instead is unlinkability: no profile is built of you, no chat is associated with you, queries reach the models anonymously, and they are never used for training. The provider still reads your prompt to answer it, so what protects you there is anonymity, not content secrecy.

4. Why is my local model fast in demos but slow on my real work?

   Because two different bottlenecks exist and demos only exercise one. Generation speed depends on memory bandwidth and looks fine on short prompts. Prompt processing — reading everything you sent before answering — depends on raw compute, and it dominates on long inputs. On a 512 GB Mac Studio running a 671B model, an 8,001-token prompt took about 14.8 minutes before the first word appeared. Benchmark with prompts the length you actually use.

5. What is the practical way to use both?

   Run a small local model on anything confidential, using it to extract, restructure and de-identify offline. Send only the de-identified residue — the part that genuinely needs deeper reasoning — to a frontier model through a privacy-focused gateway. The sensitive original never leaves your machine, and the hard thinking still gets done by the strongest model available.