What Is a Context Window in AI?
Erwin Berkouwer
A context window is the amount of text an AI model can process and remember at once during a conversation or task. It is the model's working memory: whatever falls outside it, the model no longer has. For a business owner, this decides what you can and can't hand to an AI tool, from a single question to an entire contract.
How does a context window work?
A language model doesn't read words, it reads tokens: chunks of text from a few letters up to a whole word. As a rule of thumb, one word is roughly one and a half tokens. The context window is the maximum number of tokens the model can handle in one go. That includes your question, any attached documents, the system instructions running in the background and the answer the model gives back.
Go past that limit and the model doesn't just get vague. It loses access to part of the information, usually the start of the conversation or document. Think of a postman who can only carry a certain number of letters at a time. Add one more to a full bag and an old one falls out. How a tool deals with this differs: some show an error, others quietly leave something out.
Why are context windows growing so fast?
Context windows have grown enormously in recent years, from a few thousand tokens to models that handle hundreds of thousands or even millions (limits differ per model and change often). That sounds impressive, but a window fills up faster than you'd think. A long contract, a few email threads and a tool's system instructions already take up a big share of the space.
A bigger context window doesn't automatically mean a smarter answer. Research suggests that models are more likely to overlook information in the middle of a long document than information at the start or the end.
Why does the context window matter for SMEs?
For most SME use cases, the context window isn't an abstract concept. It is the hard limit on what an AI tool can do for you. Think of:
- Reviewing contracts and quotes: a 40-page supplier contract has to fit in full, otherwise the model misses clauses further down.
- Summarising customer conversations: a long phone call or chat history has to be included completely to get a good summary.
- Combining several documents: for example a quote request next to earlier correspondence, so the answer stays consistent.
With AI agents that carry out tasks on their own, every step counts too: each intermediate step, each tool result and each earlier action stays part of the running conversation until the window is full.
A concrete example
An accountancy firm uses an AI assistant to check incoming invoices and the related email exchanges for discrepancies. As long as the invoice, the quote and the relevant emails fit in the context window together, the model can compare them in one pass and give a well-founded answer. If the email chain gets too long, you have a choice: send only the latest messages, or have the conversation summarised first. You can read more about this kind of use case under invoices and administration.
When does the context window become a problem?
For short tasks, like rewriting a LinkedIn post or answering a single question, it is rarely an issue. It becomes relevant with:
| Situation | Risk of the window filling up |
|---|---|
| A single question or short text | Low |
| One document of a few pages | Low to medium |
| Several documents or a long conversation | Medium to high |
| AI agent with multiple steps and tools | High, grows with every step |
If you regularly hit this limit, it is often a sign your workflow can be smarter, for example by supplying only the relevant fragments instead of complete documents.
Which terms go with it?
- Tokens: the unit the window is measured in.
- RAG (Retrieval-Augmented Generation): a technique that pulls up only the most relevant pieces from a large body of documentation, so you don't have to cram everything into the window. See also What is semantic search?.
- Vector database: the storage where RAG systems find relevant text fragments by meaning, not just by keyword.
- Prompt engineering: wording your question and context compactly on purpose, so you use the available window efficiently.
A well-built AI setup respects these limits and delivers only relevant information. Many implementations go wrong exactly here: not because of the quality of the AI, but because of the workflow around it.
Curious how this plays out in your business? An AI scan shows you in a few minutes where AI makes the biggest difference in your processes. If you'd rather talk through a specific use case first, book an introduction call or read more about AI consultancy.
Frequently asked questions
Short, clear answers so you can decide faster.
What happens if I go over the context window limit?
The model loses access to part of the conversation or document, usually the oldest part. Depending on the tool, you get an error message, or the model answers as if that information never existed, often without telling you.
Is a bigger context window always better?
Not necessarily. Bigger windows cost more computing power, so often more time or money per question, and models sometimes do worse with information buried in the middle of a long document.
Roughly how much text fits in a context window?
That differs a lot per model and changes regularly. As a rule of thumb, count on about one and a half tokens per word, so a 10,000-word document quickly uses 15,000 tokens.
Do I need to think about the context window myself as a business owner?
Only indirectly. What matters more is that the tool or workflow you use handles it well, for example by supplying only relevant fragments instead of complete archives.
