Which local model fits your hardware

The most common disappointment with local AI is not that it does not work. It is that somebody pulls the largest model they saw recommended somewhere, and the machine then either crashes or produces one word per second. After that the verdict is "local is useless", when all that was wrong was the size.

This playbook turns the order around. Look at your hardware first, derive the size class from it, and only then choose within that class by task. At the end you have two candidates and a method for comparing them on your own work.

1. Work out how much memory is actually available

What counts is not the memory in the machine, but the memory the model actually gets.

On a Mac with Apple Silicon, processor and graphics share the same memory, and only part of it goes to the model. As a rough guide reckon with two thirds to three quarters: 32 GB gives you around 22 GB usable, 64 GB about 45. The exact share is set by your runtime and has changed between versions, so treat the figure as a guide rather than a promise.

With a dedicated graphics card only its own memory counts, regardless of how much RAM sits in the machine. It is not entirely yours there either, though: display output and drivers take half a gigabyte to a full one. So a 12 GB card leaves you roughly 11 GB. If the model does not fit, part of it is offloaded to the main processor, and then speed collapses.

Tip: If you only keep one number, keep this one: usable memory, not installed memory.

2. Estimate a model's size in your head

A rule of thumb for a rough estimate, without hunting for a table. At the usual Q4 compression a model needs roughly half as many gigabytes as it has billions of parameters. An 8B model lands around 4 to 5 GB, a 32B model around 18 to 20 GB. That estimates the weights, not the whole requirement, and the exact download size is stated on every model anyway. Use the formula to shortlist and the real number to decide.

On top of that comes room for context, meaning what you type in and what the model holds on to. For short conversations about one extra gigabyte is enough, for long documents considerably more.

Tip: Always plan two to three gigabytes of headroom. A model that fits exactly has no room left for your text and gets slow on the first longer document.

3. Place yourself in one of four classes

As of August 2026, at the usual Q4 compression:

The figures below are usable memory, so a graphics card's VRAM or the Apple Silicon share from step 1, not the number on the box.

With 8 GB usable a model in the 7 to 9 billion class runs. It works, but with no reserve for long texts. Fine for summaries and short tasks.

With 16 GB usable it gets comfortable. The same class runs smoothly with decent context, or you move up to 14 billion.

With 24 to 32 GB usable you reach the 32 billion class. This is where local starts being serious on substance.

From 44 GB usable models in the 70 billion class become possible, which at Q4 occupy roughly 38 to 42 GB depending on the model. On Apple Silicon that makes 64 GB installed the absolute floor, and a tight one: at a two-thirds usable share that is around 43 GB, so barely the model and hardly any context. That class only gets comfortable there with more. It is the class where you stop noticing the gap to the cloud on every answer.

Tip: These thresholds move, and they move in your favour, because models keep getting more efficient. A 30B model today beats a 70B model from eighteen months ago.

4. Choose within your class by task

Inside a size class there are families with different strengths. The landscape changes fast, so here is the pattern rather than a ranking that will be wrong in three months.

For general text work and multilingual use, Alibaba's Qwen family is the solid 2026 default, partly because many of its models ship under Apache 2.0 and come in many sizes. The licence applies per model and per revision, though, not to a family as a whole, so the look at the model card is still due. For programming there are dedicated coder variants inside the families that are clearly better than the general model at the same size. For small devices and tight memory, Google's Gemma line is built for frugality. If you want models that expose their reasoning, look at the reasoning variants, for example OpenAI's gpt-oss line.

Tip: Never take a general model for code when a coder variant of the same family exists. At identical memory footprint that difference is bigger than the one between two families.

5. Check the licence before it turns commercial

Open weights does not automatically mean free use. Apache 2.0 and MIT are uncomplicated, you may work commercially with those. Other licences carry restrictions, on user numbers or attribution for instance.

Licence terms apply privately too, they just surface less often. The moment a client is involved or the model sits inside a product, the formality turns into a risk.

Tip: The licence is on the model card at Hugging Face. One look, thirty seconds, saves an awkward question in a sales meeting.

6. Download two candidates, not one

The mistake is deciding beforehand. Pull two models from your class that come from different families. The disk space costs you nothing permanent, and comparing on your own work is worth more than any leaderboard.

Tip: Do not take two sizes from the same family. The bigger one wins predictably. Take two different families at the same size, that is the question you actually have.

7. Build yourself ten real test cases

Collect ten tasks from your daily work where you can judge whether the result is good. Rewrite an email, summarize minutes, interpret a table, whatever you actually do.

Put them in a file and send each to both models. Do not judge while reading. Collect all twenty answers first, then go through them.

Tip: Include tasks that the cloud models struggle with. That is exactly where it becomes clear whether local is enough for you.

8. Measure speed, but the right one

Two numbers matter differently. Time to the first word you notice immediately in conversation. Words per second after that you only notice when they drop below your reading speed.

A model that starts after one second and then writes steadily feels better than one that thinks for three seconds and then types faster. For bulk processing in the background it is exactly the other way round, there only throughput counts.

Tip: My personal threshold sits around ten words per second, roughly reading speed. Below that the work feels sluggish. That is a rule of thumb rather than a measured standard; runtimes report tokens rather than words anyway, and German needs more tokens per word than English. If it feels sluggish, go down a class, the overall result gets better.

9. Decide, and write down the reason

Pick the model that convinced you more often across your ten test cases. Not the one with the better benchmark scores, not the one recommended in a forum.

Note in two sentences why. You will need that note in three months when the next generation lands and you compare again. Then you have both the test cases and the yardstick ready.

Tip: Keep the ten test cases. They are your personal benchmark and worth more than any public leaderboard, because they measure your work and not somebody else's.

10. Plan the switch from the start

The model you pick today will not be the best one in six months. That is fine, as long as you have not hard-wired it.

Keep the model name in one place in your configuration, not scattered across ten scripts. Then switching is a single line. Whoever writes the name everywhere stops switching out of inconvenience and ends up two generations behind.

Tip: The same principle applies to cloud models. How to catch model releases at all instead of discovering them by accident is in Track model availability.

What next

If you have no model running yet, start with Your first local AI model in 30 minutes. If the hardware question is settled and it is now about company use, Local AI and the GDPR covers the legal part. And if you want to know whether your own machine pays off against the cloud at all, the arithmetic is in Local or cloud, the honest arithmetic.

Source

Memory figures are approximations at Q4 compression, as of August 2026, and vary with model architecture and context length. Check the concrete value on the model card.

• Ollama, model library with size figures: https://ollama.com/library
• Hugging Face, model cards with licence and weights: https://huggingface.co/models
• llama.cpp, documentation on quantization levels: https://github.com/ggml-org/llama.cpp