The Local Model Playbook: when running AI on your own hardware actually beats renting it

Every few weeks a frontier-class model ships with open weights, the internet declares AI is now free, and a few thousand people spend a weekend discovering that free weights and free inference are completely different things. The download costs nothing. The machine that can hold the thing in memory costs somewhere between a nice dinner and a used car.

This playbook is the decision in order: does local make sense for you at all, what size model your hardware can physically hold, which runner to install, what it really costs per million tokens, and how to build the hybrid setup most people end up wanting anyway. Work through it once and you will never again buy a graphics card because of a headline.

Part 1 — The only three reasons that justify going local

Start here, because if none of these describe you, the rest of the playbook is a hobby, not a decision. There are exactly three durable reasons to run a model on hardware you own.

1. The data legally cannot leave

This is the strongest reason and the one that survives every price war. If you handle protected health information, privileged legal material, unreleased financials, or client data under a contract that forbids third-party processing, on-premise inference means no model provider ever touches the data — which, in a US healthcare context, removes the need for a business associate agreement with a model vendor entirely.

The catch nobody mentions in the enthusiast forums: you inherit the security burden you just took back. Encryption, access control, audit logging, patching, and log retention are now yours. “It runs on my laptop” is not a compliance posture. Budget for the boring infrastructure, not just the GPU.

2. Volume, at high and steady utilization

Per-token pricing is brutally efficient when you use AI in bursts and merciless when you run a pipeline that never sleeps. Classification over millions of records, document extraction at scale, embedding regeneration, synthetic data — these are the workloads where owning the machine wins. Note the word steady: local economics are a utilization story, and Part 5 shows why that single variable decides everything.

3. Availability you control

Rate limits, deprecation notices, capacity pauses, and price changes are all decisions someone else makes about your workflow. A model on your own disk keeps working when the vendor is having a bad week — the argument we made at length in the resilience playbook. A local fallback is cheap insurance even if it is not your daily driver.

Part 2 — The VRAM formula that governs everything

One piece of arithmetic decides what you can run. Memory required scales with parameter count and precision:

Run it and the landscape becomes obvious. A 7B model at Q4 needs roughly 4–5 GB. A 14B needs about 8–10 GB to sit entirely on the GPU. A 30B wants 16–20 GB. A 70B needs 48 GB or more at Q4.

That phrase “entirely on the GPU” is the one that matters. The moment a model does not fit in video memory, layers spill to system RAM and generation speed falls off a cliff — not by 10%, by an order of magnitude. Local inference is a game of fitting inside a memory budget. Everything else is detail.

Which quantization to pick

Ignore the alphabet soup and remember three:

• Q4_K_M — the default. Best size-to-quality tradeoff for most work, and what you should try first.
• Q5_K_M — roughly 20% more memory for a modest quality bump. Worth it if you have the headroom.
• Q8_0 — near full precision. Use when memory is abundant and output quality is the whole point.

Do not chase precision you cannot verify. Run ten of your real prompts against Q4 and Q5 and see whether you can tell the difference on your work. Most people cannot, and the ones who can usually find the gap is in the prompt, not the bits.

Part 3 — The four hardware tiers

Apple Silicon changes the math. M-series chips use unified memory, so system RAM is usable as video memory — a well-specified Mac can hold models that would need an expensive multi-card rig on the PC side, at lower speed but far lower hassle. If you already own a high-memory Mac, you own a local inference machine and did not know it. Check before you buy anything.

Which model to actually download

The open-weight field moves monthly, so anchor on families rather than version numbers. As of mid-2026 the reasonable defaults: Qwen3 is the best all-around starting point across sizes and licenses permissively; Gemma 3 at 4B is the laptop sweet spot, with smaller variants for thin hardware; gpt-oss ships 20B and 120B reasoning models under Apache 2.0; Mistral Small is the speed pick on mid-range hardware; Llama remains the easiest on-ramp. Chinese labs keep pushing the frontier of what open weights can do — and also of what consumer hardware simply cannot hold, which is the quiet limit on “open” meaning “yours.”

Part 4 — Pick your runner in thirty seconds

People burn weekends on this choice. They should not, because the tools are not substitutes — they sit at different layers. llama.cpp and MLX are engines. Ollama and LM Studio are experience layers wrapping those engines. vLLM is a serving system for many simultaneous users.

• One person, any OS, command line fine → Ollama. Recent versions use MLX under the hood on Apple Silicon, so you get the fast path for free.
• One person, wants a GUI to browse and try models → LM Studio.
• Serving a team or an app with concurrent requests → vLLM. Its throughput advantage at high concurrency is real and large — the widely quoted benchmark is roughly 16–20x — but that gap lives entirely at peak load. At a single user, vLLM and Ollama finish in a near dead heat, which is why installing vLLM for personal use is a common and pointless act of self-punishment.
• Embedded, air-gapped, or unusual hardware → drop to llama.cpp directly.

Pick in thirty seconds, install, move on. The runner is not where your quality comes from.

Part 5 — The break-even calculator

This is the part that saves people the most money, because it usually tells them not to buy. Local cost per million tokens has two components — amortized hardware and electricity — and both are dominated by how many hours a year you actually generate tokens.

Worked example, with every assumption labeled so you can swap in your own measured numbers. Say a $2,000 machine, 300 watts under sustained load, 16 cents per kilowatt-hour, and a measured 30 tokens per second — about 108,000 tokens per hour.

• Light use (4 hours a day): $667/year ÷ 1,460 hours = $0.46/hour of hardware, plus about $0.05 of electricity. That is roughly $4.70 per million tokens.
• Heavy use (12 hours a day): $667/year ÷ 4,380 hours = $0.15/hour, plus the same nickel of power. Roughly $1.85 per million tokens.

Read those two lines again — they contain the entire economic argument. The hardware did not change. The electricity did not change. Tripling utilization cut the unit cost by 60%. Local inference is not cheap or expensive; it is a fixed cost pretending to be a variable one, and it only beats the meter if you keep the machine busy.

Which is why the honest answer for most individuals is that your cost per million tokens lands above what a provider would charge, and the gap is not close. Go local for reason one or reason three from Part 1, and treat reason two as something you prove with a utilization number rather than assume.

Part 6 — The 90-minute setup

1. Find your memory ceiling (5 min). Look up your GPU’s VRAM, or your Mac’s unified memory. Multiply by 0.8 for headroom. That number is your budget.
2. Pick a model that fits with room to spare (5 min). Use the Part 2 formula. If your budget is 19 GB, take the 14B, not the 32B you have to cripple to load.
3. Install one runner (10 min). Ollama or LM Studio. Do not evaluate five.
4. Measure, do not guess (15 min). Record tokens per second and memory actually used. These are your Part 5 inputs, and numbers from the internet will be wrong for your machine.
5. Build a ten-prompt bench from your real work (30 min). Not trivia — the actual tasks you want to move off the meter. Run them local and cloud. Score each output pass/fail on whether you would ship it.
6. Decide with the scores in front of you (15 min). What passes locally moves. What fails stays on the meter. Write the split down.
7. Set a recheck date (5 min). Ninety days. Models improve; so does your bench.

That last step is the one people skip and the one that compounds. A ten-prompt bench you keep is the difference between an opinion about local models and a measurement — and it doubles as the verification layer described in the AI trust ladder.

Part 7 — Six ways this goes wrong

1. Buying before benching. The card arrives, then you discover the 8B model was fine and what you needed was better prompts. Test on borrowed or rented hardware first.
2. Loading a model that does not fit. It still runs — spilled into system RAM at a fraction of the speed — and you will blame the model. Watch memory, not just output.
3. Ignoring context memory. The weights fit, then you paste a long document and the KV cache blows past your budget mid-generation. Size for your longest real input.
4. Comparing against the wrong cloud model. A 14B local model is not competing with a frontier model. It competes with the cheap, fast tier — the honest comparison, and often a winnable one.
5. Counting the GPU and forgetting the rest. Power supply, cooling, noise, the electricity line, and — if compliance drove this — logging and access control. Budget the system, not the card.
6. Letting it rot. An unmaintained local stack is worse than none, because you will reach for it during an outage and find it broken. Ten minutes a month, or delete it.

Part 8 — The answer most people should actually build

Almost nobody should run everything locally, and almost everybody benefits from running something locally. The durable architecture is a three-way split:

• Local, always: anything touching sensitive data, plus high-volume mechanical work — classification, extraction, tagging, redaction, first-pass summarization. Bulk and boring.
• Cloud, always: the hardest reasoning, the longest context, anything where quality is the product, and anything you do rarely enough that owning the capacity makes no sense.
• Local as fallback: a small model kept current and tested quarterly, so an outage or a price shock is an inconvenience instead of a stoppage.

Sort your workflows into those buckets and the hardware question answers itself — you will know what size model you need because you will know what you are asking it to do. That inventory pairs naturally with an audit of what you already pay and the delegation system in the $2 Test.

What to watch

• Memory, not parameters. The interesting frontier is models that do more inside a fixed memory budget, not models that need more of it.
• Unified-memory machines. Every generation that raises the ceiling on consumer unified memory moves the line between what you rent and what you own.
• The size of what actually ships open. When flagship open weights need a rack to load, “open” stops meaning “yours” for anyone without a data center.
• Your own bench scores. The only benchmark that decides your stack is the one built from your work. Rerun it quarterly.