Local models in your own stack

This level has been about portability so far. The same memory in Claude, Cursor and Codex, the same tools over an open protocol. This lesson draws the line to its end, because portability does not stop at the provider. It stops at the question of whether a provider needs to be involved at all.

Why this is the real test of the standard

A protocol is open when you can run it against something its inventor did not plan for. MCP comes from Anthropic. If the same servers work against a model sitting on your desk that has nothing to do with Anthropic, the openness is proven rather than merely claimed.

That is exactly what works. It is the same server software, the same memory tool, just a different model behind it. What changes is the model's ability to use the tools intelligently, and that is what the rest of this lesson is about. That you CAN use the same servers, by the way, does not mean you should register them everywhere. More on that at the end.

Separating the roles cleanly

In Claude Desktop two things blur into one application, and that causes confusion the moment you pull them apart.

The protocol knows host, client and server. The host is the application you sit in; it starts one client per connection, and each client talks to one server that provides the tools. The model is not a participant in the protocol: the host puts the tool list in front of it and executes whatever it wants to call.

That separation is exactly why the same server software works behind any model. In a local setup the model lives in a runtime such as Ollama, and the host is a separate program, a web interface for instance. You swap a layer, not the setup. That only means you COULD use the same servers, though. Which ones you actually register in which environment is a security decision, more on that below. What you do have to check is the transport: a server entry configured as a local process cannot be copied unchanged into a web interface.

Where the limit sits

Here is the difference that matters in practice. Check the setup first: chat template, tool descriptions, quantization, whether the interface passes calls through at all. If the behaviour survives that, it is the model, and that is the part you will not configure away.

A large model reliably decides whether a question needs a tool. A small local model does not always. It occasionally answers from its own knowledge although the right tool is sitting there. It sometimes reaches for the wrong one. And when a task needs three tools in the right order, it loses the thread more often.

Selecting a tool is a reasoning task, but unlike text comprehension it is not primarily size that decides. Below roughly seven billion parameters the capability is largely missing. Above that, what counts is whether the model was explicitly trained for tool calling: in comparisons, 8B and 14B models with that training land close together, and a considerably larger model without it can do worse than a small one with it. Picking by parameter count alone therefore misses regularly here.

What follows for your setup

Three consequences, all pointing the same way.

Fewer tools at once. Three well described ones beat twenty. With large models a long list is harmless, with small ones the hit rate drops with every additional entry.

Descriptions that explain the when. "Searches the database" is enough for a strong model. A local one needs "use this when the question is about a customer, an invoice or a case". With local models the description is not decoration, it is the most important lever.

As little access as possible, on both sides. A model that occasionally reaches wrong being allowed to delete or send is the obvious bad combination, and a confirmation requirement handles it. The underrated part is reading: a read tool scoped too widely pulls keys and client data into context and logs, and the moment that same interface runs against a cloud model, that is the route outward. So a single directory rather than the home folder, and confirmation for anything that writes.

Why memory matters even more locally

In this level the thesis has run like this so far: a model that knows your context beats a stronger one that has to guess. With local models that stops being an optimisation and becomes the precondition for the setup being any good at all.

A small model knows little about the world and nothing about you. What it does not know it can only invent. Give it access to your notes, decisions and project state, and it no longer has to guess, only look up and phrase. Looking up and phrasing is exactly the discipline small models are good at.

The size disadvantage thereby shifts from "does not know it" to "has to fetch it". And that is a problem tools solve.

The mixed setup

In practice almost nobody picks only one side, and there is no need to.

Everything with sensitive data and all bulk work runs against the local model. Everything hard goes to the cloud. Because it is the same server software either way, setting up the second environment costs little.

One boundary belongs with it, otherwise the mixed setup undoes exactly the privacy it exists for: two separate installations, not two windows and not two profiles of the same interface either. There, servers and model credentials are usually registered globally, so a second profile is not a boundary. If a read tool has pulled keys or client data into the history and you then switch to the cloud model, the history travels with you. So: register the sensitive servers only in the local installation, the cloud access only in the other.

To keep it that way, hold the model choice in one place within each of the two environments rather than scattered across scripts. Then a model change there is one line and not a project. It is the same principle as with model migrations inside one provider, just one level up. Between the two environments you do not switch by line, you move.

What this means for your independence

The side effect is strategic. Anyone who keeps their setup such that a local model can step in is insured against a whole class of disruptions: price changes, retired models, blocks, provider outages.

The local model does not have to be equally good for that. It only has to be good enough for the work to continue. This way of thinking is the same one shown in level 1 with the Fable episode: build so that losing a single model does not knock you over.

Getting started in practice

The technical procedure is in the playbook MCP with local models, from the runtime through the interface to the first tool. If no model is running yet, Your first local AI model in 30 minutes is the starting point. And before you attach third-party servers to a setup that is explicitly local for privacy reasons, Vetting third-party MCP servers safely belongs in between.

Quiz

1. What stays the same when you switch from a cloud model to a local one?

• A) The server software and the memory tool
• B) The quality on hard reasoning tasks
• C) The reliability of tool selection

A is correct. That is precisely the value of the open protocol: it is the same software. Which servers you then actually register in a cloud environment stays a separate decision. B and C get worse, that is the price.

2. Why does the hit rate drop with local models the more tools are active?

• A) The runtime gets slower
• B) Tool selection is a reasoning task, and small models are limited at it
• C) MCP only allows a limited number of servers

B is correct. It is not a technical limit but a capability limit of the model.

3. Why is memory especially valuable with a small local model?

• A) It makes the model faster
• B) It replaces the tool descriptions
• C) Knowledge does not have to sit inside the model, it gets looked up

C is correct. Looking up and phrasing is what small models do well, carrying knowledge is what they do badly.

Source

• Model Context Protocol, specification: modelcontextprotocol.io
• Ollama, models with tool support: ollama.com/search?c=tools
• Open WebUI, MCP connection: docs.openwebui.com