Having AI summarize long documents without it inventing things

There is this moment. You have a long document in front of you, a rental contract, a funding decision, a forty-page tender, and you just want to know what it is about and whether there is a trap hidden somewhere. The obvious idea is to throw the thing into an AI and type "summarize this". That often works surprisingly well. And sometimes the AI invents a clause that is not in there at all, or it leaves out exactly the one sentence that mattered.

The problem is not that the AI is stupid. It is that summarizing and inventing are the same activity for a language model, namely producing plausible text. If you do nothing about it, you get a summary that feels right, whether or not it is. These ten steps are my routine for getting a short version out of a long document that I actually trust, without having to read every word myself.

Step 1, understand why long texts fall apart

A language model has a context window, a maximum amount of text it can look at in one go. Short documents fit in completely. With very long ones it can happen that parts get cut off, or that the model loses the thread in the middle while it holds on to the beginning and the end just fine.

That is exactly where the errors come from. Not because the model wants to lie, but because information is missing in the middle and it fills the gap with something plausible. Once you have that in your head, the next steps make sense, because they all aim at making guessing as hard as possible for the model. The mechanics behind it are in the lesson Context and tokens.

Step 2, feed the document in properly

Where you can, paste the text into the chat window as real text or attach the file, instead of taking a screenshot. A screenshot forces the AI to read the text out of an image first, and that is where misreadings creep in, especially with tables, numbers and small print.

If you have a PDF, check whether you can select the text. If you can, it is a real text document and everything is fine. If you cannot, it is a scanned image, and you should explicitly ask the AI to read the text out cleanly first and to flag any passage it is unsure about. Garbage in, garbage out, and most bad summaries already start here.

Step 3, phrase the job precisely

"Summarize this" is not enough. The AI does not know what you need the summary for, so it guesses that too. Tell it the purpose. "I might sign this contract, tell me the obligations I am taking on and all the costs." Or "This is a funding decision, I want the deadlines and the conditions under which I have to pay the money back."

The clearer the purpose, the more targeted the summary. And a targeted job leaves less room for invented decoration, because the model has a narrow corridor to answer in.

Step 4, demand evidence instead of mere statements

This is the most important step of all. Ask the AI to name the place in the original for every point of the summary, so the literal quote or at least the page or the section. The wording is simple. "For every point: name the exact passage from the document as evidence."

The effect is twofold. First, you can look up every point in the original in seconds. Second, and this is the real trick, it gets harder for the model to invent something when it has to deliver a literal quote for it. It cannot produce a quote for a clause that does not exist. You go deeper into the principle behind this in the playbook Fact-checking an AI answer.

Step 5, let the AI show its uncertainty

Models sound confident even when they are guessing. Take that away from them by explicitly giving permission to doubt. "If something in the document is unclear or contradictory, write that down instead of smoothing it over. If you cannot find something with certainty, say so."

Without that instruction, the model presents everything to you in the same confident tone. With it, you get an honest split between "it says so" and "I am unsure here". That second category is exactly the set of passages you have to read yourself, and those are the ones you do not want hidden.

Step 6, cut very long documents into parts

If a document is so long that the AI gets weak in the middle, do not hand it over in one piece. Split it into sections, chapter by chapter for example, and have each part summarized on its own. At the end you give the partial summaries back and have an overall picture built from them.

That sounds like more work, but it is more reliable, because every part fits completely into the context window and nothing gets lost in the middle. With an important contract, those ten extra minutes are well invested. Why long texts fall apart in the first place is in the same lesson, Context and tokens.

Step 7, ask specifically about the small print

A summary smooths things out by its nature. But the unpleasant part is often in the small print, and that is exactly what tends to fall out during the smoothing. So ask for it specifically in a second pass. "Are there deadlines, notice periods, automatic renewals, penalties or hidden costs? List them one by one, with the passage."

These targeted questions bring out what a general summary passes over. With contracts and official decisions, that is the part that costs you money later if you miss it.

Step 8, spot-check the summary against the original

Do not trust blindly, not even with evidence. Take three or four points from the summary, ideally the most important ones and the ones that surprise you, and look up the quoted passages in the original. Does the quote match word for word? Does it really say that in context?

If the spot checks hold up, you can trust the rest considerably more. If one does not hold up, if a quote is slightly twisted or cannot be found at all, then treat the whole summary as unreliable and go through it more strictly. An invented quote is an alarm signal, not a slip.

Step 9, hold a second AI or a second run against it

For something that really counts, a contract you are signing, a deadline that means money, a second look is worth it. Feed the same document with the same job in a second time, ideally into a different model, and compare the two summaries.

Where both say the same thing, the odds are high that it is correct. Where they differ, you have found exactly the passages you need to check yourself. That is the cheapest safeguard there is, it costs you five minutes and catches most inventions.

Step 10, know when the AI is no longer enough

An AI summary is a tool for understanding quickly and for finding the right questions. It is not a substitute for legal advice, not a binding statement and not a licence to sign blindly. With a rental contract, an employment contract or an official decision that has real consequences, you use the summary to walk prepared into a conversation with a human being who is liable for it.

That is not a weakness of the method, that is its proper use. You save the hour you would spend grinding through forty pages, and you arrive with pointed questions at the place where an expert is really needed.

What next

If you want to understand why AI invents things at all, work through the lesson Hallucinations, that is the foundation underneath this whole playbook. For the technique of checking any AI answer, Fact-checking an AI answer is the next step. And if your concern is conversations and appointments rather than documents, AI notes for meetings helps.