MCP with local models, tools without the cloud

A language model on its own is a conversation partner without hands. It cannot look anything up, cannot compute, cannot fetch, it can only answer from what is in the conversation. It gets interesting when it is given tools, and the protocol for that is MCP.

What is taken for granted with Claude or ChatGPT also works entirely locally. Model on your machine, tools on your machine, data on your machine. This playbook shows the way there and says at the right point where it gets rough, because at one point it genuinely does.

1. Work out which three parts you need

The protocol knows three roles, and the confusion usually comes from throwing the model in with them.

The host is the application you sit in, Claude Desktop or a local web interface for instance. It starts one client per connection, and each client talks to exactly one server. The server provides the tools, file access or a database for instance.

The model is deliberately not in that list. It is not a participant in the protocol. The host collects the tool list from the servers, puts it in front of the model, and when the model wants to call one, the host executes the call. That sounds like hair-splitting, but it is the reason the same server works behind any model.

In Claude Desktop the host and the model access sit in one application. In a local setup you separate them: Ollama supplies the model, an interface or your own script is the host. It is the same server software as with Claude, and that is the entire point of an open protocol. Which servers you actually register in which environment is independent of that and a security question, see step 10.

Tip: If the protocol itself is still unfamiliar, read the lesson What is MCP first. Everything here assumes it.

2. Check whether your model can use tools at all

Not every local model handles tool calling. Some were never trained for it, others can do it formally and then use it wrongly. That is the first hurdle, and it eliminates a good share of the candidates.

In the Ollama library, models with tool support are labelled accordingly. Stick to that instead of trying and then spending half an hour looking for why nothing happens.

Tip: For tool calling do not take the smallest model you can find. Below roughly seven billion parameters the capability is largely missing and you get malformed calls. Above that, what decides is not size but whether the model was explicitly trained for it: in comparisons, 8B and 14B models with tool training land close together, while considerably larger models without that training do worse. Bigger is not automatically better here.

3. Take an interface that already speaks MCP

The fastest route is an interface that brings the protocol along. Open WebUI has supported MCP natively since version 0.6.31 and is built for local models. That saves you writing the client yourself.

The setup is then: Ollama runs in the background, Open WebUI connects to it, and in its settings you register your MCP servers.

One detail catches nearly everyone on the first attempt: the servers are the same, the registration is not. What Claude Desktop configures as a locally started process (stdio, so a command plus arguments) needs a reachable address in a web interface. Some servers can do both, others only one.

If you need a bridge for a stdio server, that is not a mere format change but a new open door. Bind it to 127.0.0.1, give it an access token, and do not put it on the network. A file server exposed through a bridge without authentication is file access for anyone who knows the address.

Tip: Before you build your own client, try the ready-made one. Most people notice at that point that they do not need their own.

4. Start with a harmless server

Take a read-only server first. File access restricted to a single test directory is ideal. That way you see whether the chain works without a misfire doing any damage.

Then ask the model a question it cannot answer without the tool. "What does the file notes.txt say" is a good test, because the answer is unambiguously right or wrong.

Tip: If the model answers without having used the tool, it made the answer up. That is exactly why a test with an unambiguous answer beats an open-ended one.

5. Expect the model to overlook tools

Here is the point where local setups differ from Claude, and it matters more than anything technical.

A large model reliably decides when a tool is needed. A small local model does not always. It sometimes answers from its gut although the right tool is sitting there. It sometimes calls the wrong one. And on a chain of several steps it can lose the thread.

Before you blame the model: it can also come from the setup. A wrong chat template, a server that does not describe its tools cleanly, too coarse a quantization, an interface that does not pass tool calls through at all. Check that first. If the behaviour survives, it is a capability limit of the model, and you will not configure that away.

Tip: The fewer tools are active at once, the more reliable the selection gets. Three well described tools beat twenty, and with small models by a wide margin.

6. Write tool descriptions for a weaker model

With a strong model a tool description may be terse, it works out the context by itself. With a local model the description is your most important lever.

Write into it when the tool is to be used, not only what it does. So instead of "searches the database", write "use this when the question is about a customer, an invoice or a case". The difference in hit rate is substantial.

Tip: If you build your own servers, that is better practice anyway. The rules for good tool design are in Tool design, and with local models they pay off twice over.

7. Attach your memory

The most valuable server in a local setup is the one that remembers things. A small model knows little about the world and even less about you. A memory partly compensates for both, because then the knowledge does not have to sit inside the model, it gets looked up.

That is also the point where the size question loses its edge. An 8 billion model with access to your notes beats a large model without that access on questions about your project.

Tip: This is precisely the core thesis of this academy. A model that knows your context beats a stronger one that has to guess. The setup is in Setting up memory.

8. Keep writing tools on a short leash

Writing tools are the obvious risk. A model that can delete files, send mail or change records and occasionally grabs the wrong tool is an uncomfortable combination. Those need a confirmation, always.

That does not make reading harmless, and this is the mistake you read in guides constantly. A read tool with too wide a scope pulls SSH keys, credentials, tax paperwork or client data into the model context, and from there into logs and histories. If you later run that same interface against a cloud model once, that is exactly the route outward you were trying to avoid. On top of that: what is named "read" need not be free of side effects, the protocol does not enforce that.

So the rule is not "reading free, writing locked" but as little access as possible on both sides. Give the file server a single directory rather than your home folder, and have writing actions confirmed.

Tip: Start with a single directory in read access and widen only once you know what the setup actually touches. Not the other way round.

9. Vet third-party servers before you attach them

An MCP server is executable code on your machine. Your model running locally does not make a third-party server any safer. The privacy gain from the local model is gone if the attached server phones home.

Tip: The vetting procedure is in Vetting third-party MCP servers safely. In a setup that is explicitly local for privacy reasons, this step is not optional.

10. Decide honestly whether it is enough

After a week of daily use you know whether the setup carries. Two outcomes are normal and both are fine.

Either it is enough, and then you have an assistant with tools where nothing leaves the house. Or it is not enough, because the model reaches wrong too often. Then the answer is not "local is useless" but a mixed setup: the local one for everything with sensitive data, the large model for the hard cases.

What step 8 warns about applies here, though. If you switch between local and cloud model inside the same profile, the tools and the existing history come along, and with them everything a read tool pulled in earlier. So separate at the level that actually separates: two separate installations, or separate configuration directories. A second window is not enough, and neither is a second user profile in the same interface: the MCP servers and model credentials are usually registered globally there, so both profiles can reach them. The local installation holds the sensitive servers and no cloud access; the other holds the cloud access and simply does not list those servers.

Tip: That an MCP server COULD run on both sides does not mean it should. Portability saves you the learning curve when setting up the second environment, not the decision about which servers get registered there at all. More on the principle in Tool portability.

What next

If the model itself is still missing, start with Your first local AI model in 30 minutes. The size question is covered by Which local model fits your hardware. And if you eventually want to build your own server instead of attaching other people's, level 6 with Planning an MCP server is the way in.

Source

• Model Context Protocol, official specification: https://modelcontextprotocol.io
• Ollama, models with tool support: https://ollama.com/search?c=tools
• Open WebUI, documentation on the MCP connection: https://docs.openwebui.com