Your first local AI model in 30 minutes

There are two reasons to run a language model on your own machine. One is privacy, and in Germany that carries more weight than elsewhere. The other is that you want to see for yourself what this thing actually is when no company sits in between. Both are good reasons.

What you get is a model that answers offline. No account, no per-token billing, no transfer to a provider. What you do not get is the quality of Opus 5 or GPT-5.6. That is not a small thing, and I will say plainly in step 9 where the limit sits. But for a surprising number of tasks the local option is enough, and getting there takes half an hour.

1. Check whether your machine can play along at all

Before you install anything, look up how much memory you have. On a Mac with Apple Silicon what counts is the shared memory, so the number that said 16 GB or 32 GB when you bought it. On a Windows or Linux machine with a dedicated graphics card, what counts is the memory on that card, not the machine's.

The rough shape of it, as of August 2026, split by build. With a dedicated graphics card almost all of its memory is yours: 8 GB is enough for a small model, 16 GB is comfortable, from 24 GB it gets serious on substance. On Apple Silicon processor and graphics share the same memory and only part of it goes to the model, roughly two thirds to three quarters. So 16 GB becomes about 11 GB usable, 32 GB about 22. On a Mac, budget one step more generously than the number on the box suggests. Without a graphics card it works too, just slower, because the main processor does the maths.

Tip: If you are unsure, start anyway. The first attempt costs you nothing but disk space, and you get that back with one command.

2. Install Ollama

Ollama is the most comfortable way in. It downloads models, handles the format and exposes an interface that feels like OpenAI's to other programs. That is more useful than it sounds, because it lets a lot of tools talk to your local model without any rework.

There are installers for Mac, Windows and Linux on ollama.com. After installing, a small service runs in the background, reachable only from your own machine by default. You can check it is there from a terminal:

ollama --version

Tip: If you prefer clicking to typing, take LM Studio instead. It does the same thing and has a built-in model browser. Watch out with the commands below: LM Studio starts its server by hand from the interface and listens on a different port than Ollama. The address is shown there in the server tab, and that is what goes into step 8 instead of the Ollama address. The ollama commands are for Ollama only.

3. Pull a sensible first model

Start small. A model with seven to nine billion parameters at the usual compression fits into eight gigabytes and answers fluently on a normal machine.

ollama pull qwen3:8b

The download is a few gigabytes. You can read step 4 while it runs.

Tip: The number before the b is billions of parameters, so roughly the size class. What follows it, something like q4_K_M, is the compression. More on that in Which local model fits your hardware.

4. Understand quantization in one sentence

A model is trained with very precise numbers. For running it, those numbers can be stored more coarsely, a bit like saving a photo as JPEG instead of RAW. The result is smaller and faster, and most of the time you cannot see the difference.

The usual setting is called Q4_K_M. Against the uncompressed original the memory footprint shrinks to roughly a quarter, and the quality loss is small enough that it does not show up in everyday use. It is not exactly four bits per number, scaling values come on top, which is why the real file size is always a little above the plain arithmetic. Ollama picks it automatically, you do not have to configure anything. The only thing that matters is that you know what is meant when you read the label somewhere.

Tip: Do not go below Q4. Q2 and Q3 save more memory, but that is where the model starts getting noticeably dumber. How much depends on the model and the task, and for code and tool calls it degrades earlier than for prose.

5. Talk to the model for the first time

ollama run qwen3:8b

You land in a chat in your terminal. Ask something you already know the answer to, so you can judge the quality. Something from your own field works better than a general knowledge question, because with domain material you notice immediately whether there is substance or just fluency.

You exit with /bye.

Tip: The first answer takes longer because the model is being loaded into memory. From the second one on it moves. So if your first impression is that this is unusably slow, wait for the second try.

6. Put an interface in front of it

Chatting in a terminal is fine for testing and awkward for daily work. Open WebUI gives you a browser interface that looks like the usual chat services but runs entirely on your machine. It finds your Ollama models by itself.

With Docker it is one command, but it looks different on Mac and Windows than on Linux. That difference is not cosmetic: Ollama listens on 127.0.0.1 only, and a container cannot reach that address on the host by itself.

On Mac and Windows Docker Desktop provides the name host.docker.internal, through which the container reaches the host. The command is on one line here so it pastes cleanly into Terminal, PowerShell and Command Prompt alike:

docker run -d -p 127.0.0.1:3000:8080 --add-host=host.docker.internal:host-gateway -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:v0.6.31

The -p 127.0.0.1:3000:8080 is the important part: it makes the interface reachable only from your own machine.

On Linux that name does not exist, and the workaround you read everywhere is a trap. --network=host does let the container reach the Ollama on 127.0.0.1, but it removes the localhost binding: the interface then listens on all interfaces. Putting a firewall in front sounds like a fix but is not a reliable one, because ufw status: active says nothing about whether that particular port is closed. And the risk is concrete: on a fresh Open WebUI installation the first person to register becomes the administrator. Someone else on the same network may have been quicker.

So that route is not listed here. On Linux take the image that brings Ollama along:

docker run -d -p 127.0.0.1:3000:8080 -p 127.0.0.1:11435:11434 -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama

This container has its own Ollama, so nothing has to be passed through to the host, and both bindings to 127.0.0.1 hold without any firewall. The second port exposes the container's Ollama interface, which you need in step 8. It deliberately sits on 11435 rather than 11434 so it does not clash with an Ollama still installed on the machine. You pull models here in the interface or with docker exec -it open-webui ollama pull qwen3:8b. The Ollama from step 2 is then no longer needed for the interface; it was still useful for terminal steps 3 to 5, and you can leave it or uninstall it.

If you want neither Docker nor two Ollamas, take LM Studio on Linux. It brings interface and model management in one.

Then open http://localhost:3000 in your browser. The fixed version in the Mac and Windows command is deliberate: :main shifts underneath you. The Linux tag :ollama unfortunately moves as well, there is no fixed equivalent. Pinning a build exactly only works via its digest anyway.

Tip: This is the point where it pays to have a conversation with a real task instead of test questions. Only then do you find out whether the model works for you.

7. Give the model your own documents

The point where local models get interesting is not general knowledge. There they lose against the big providers. It gets interesting when they see something you do not want to hand to a provider.

Drag a PDF into the chat in Open WebUI and have it summarized. A contract, minutes of a meeting, an invoice. Exactly the things you hesitate over with ChatGPT.

One caveat belongs here before you feed in anything genuinely sensitive. What is local is the model, not automatically the whole interface. If web search is switched on there, a cloud embedding service is configured for document retrieval, or a third-party model is registered as an alternative, parts of your data do go out. Walk through the settings once and switch off whatever points outward. After that the sentence holds: the file never leaves the machine.

Tip: Keep documents short at first. Not because small models inherently have a small context window, that depends on the model rather than the parameter count, but because long documents get chunked and only read in excerpts. For an 80-page contract that means the summary can simply omit an important clause without saying so. So spot-check long documents against the original.

8. Make the model reachable for other programs

Ollama exposes its interface on port 11434 and speaks the format most tools know from OpenAI. That means in many programs you can simply swap the address and suddenly the same application runs against your local model.

The command below is for the Ollama on your machine. With the Linux container that bundles Ollama it is the same interface, just on the port you exposed in step 6: use 11435 there instead of 11434. Nothing else changes.

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3:8b","messages":[{"role":"user","content":"Say hello"}]}'

If an answer comes back, the interface is open and you can build on it.

Tip: By default the port is only reachable from your own machine, and that is a good thing. If you expose it to the network, think first about who else is sitting in that network.

9. Expect the right thing, or you will be disappointed

Here is the honest assessment, without which this playbook would be incomplete.

What works well locally: summarizing, rewriting, translating, sorting, classifying, pulling structure out of a text, simple questions about an attached document. So the diligence half of text work.

What works worse locally: long chains of reasoning, hard software architecture, working independently across many steps, anything that needs genuinely deep thinking. There the big models are not slightly better, they are clearly better.

This is not a temporary weakness that gets fixed next week. It is the price of something running on a device under your desk instead of in a data centre. Accept that and you have a very useful tool. Expect a replacement for Opus 5 and you will put it away after two days.

Tip: The most productive setup is usually mixed. Local for anything with sensitive data and for bulk work, cloud for the hard cases. The arithmetic for that is in Local or cloud, the honest arithmetic.

10. Clean up if you do not want to keep it

Models take up space, and while experimenting several pile up quickly. With the Ollama on your machine it goes like this:

ollama list
ollama rm qwen3:8b

With the Linux container that bundles Ollama, the same commands but inside the container:

docker exec -it open-webui ollama list
docker exec -it open-webui ollama rm qwen3:8b

Either way that removes the model only, though. Your uploaded documents, the conversation histories and the search indexes live elsewhere, in the interface's data area. Which is exactly the material you went local for in the first place. Delete only the model and you leave the contracts and medical letters sitting there.

Fully gone looks like this:

docker rm -f open-webui
docker volume rm open-webui

If you took the Linux container with the bundled Ollama, a second copy of the models sits in its own volume:

docker volume rm ollama

With LM Studio the equivalent sits in the program folder under your user directory; chats and attached files belong in that deletion too.

That gets rid of your content. What stays behind is infrastructure: the Docker image, the Ollama installation and any further models you have not removed yet. None of that holds your documents, but it does take up space. What there is not: an account to cancel or a subscription still running. That is the other pleasant side of local.

Tip: Before you delete, write down which model was good for which task. That note is worth more next time than the model itself, because the lineup changes every few months.

What next

If you now want to know which model gives the best result on your specific hardware, Which local model fits your hardware takes over. If you want to use this in a company, Local AI and the GDPR sorts out what actually applies and what is merely claimed. And if your local model should not only talk but also use tools, MCP with local models shows how it connects to the rest of your setup.

Source

Numbers on memory requirements and model sizes change with every model generation. Check them at the source before a purchase decision, not in blog posts.

• Ollama, install and model library: https://ollama.com
• LM Studio, desktop interface with a local server: https://lmstudio.ai
• Open WebUI, browser interface for local models: https://openwebui.com
• llama.cpp, the engine underneath Ollama: https://github.com/ggml-org/llama.cpp