AI tokens and context windows explained simply

What tokens are, how context windows limit what an AI model can consider at once, and how both affect cost, speed and the quality of answers.

By the UK Technology Hub teamPublished 4 minute read

The short answer

Tokens are the small chunks of text that AI models read and write; in English, a token averages roughly three-quarters of a word. The context window is the maximum number of tokens a model can consider at once, including your instructions, documents, conversation history and its answer. Tokens determine cost and speed, and context limits shape how you design AI features.

If you use AI through an API or build AI features, you'll quickly meet two terms: tokens and context windows. They sound technical, but they explain a lot about cost, speed and why AI sometimes "forgets" things.

What is a token?

Language models don't read words the way people do. Text is split into tokens: common words may be a single token, while longer or rarer words are split into pieces. Punctuation and spaces count too. As a rough rule of thumb for English, 100 tokens is around 75 words, though this varies by language and content. Code, numbers and some languages use more tokens per word.

Every request has:

  • Input tokens: your instructions, any documents or data you include, and conversation history.
  • Output tokens: the model's answer.

Why tokens matter for cost

AI providers usually price API usage per million tokens, with separate rates for input and output. Output tokens often cost more than input tokens. Larger, more capable models cost more per token than smaller ones. So the cost of a feature depends on:

  1. How much text you send each time.
  2. How long the answers are.
  3. Which model you use.
  4. How many requests you make.

Our guide to AI cost optimisation covers practical ways to bring these down.

Why tokens matter for speed

Models generate answers token by token. Longer outputs take longer. Very long inputs also take time to process. For interactive features like chat, streaming the answer as it's generated makes the wait feel shorter.

What is a context window?

The context window is the maximum number of tokens the model can handle in a single request, including the answer. Think of it as the model's working memory for one task. Context windows have grown enormously; many current models can handle hundreds of thousands of tokens, enough for long reports or large parts of a codebase. Limits vary by model and provider.

What happens at the limit

If a conversation or document exceeds the context window, something has to give. Applications typically:

  • Trim older conversation history.
  • Summarise earlier parts of a conversation.
  • Retrieve only the most relevant passages from documents instead of sending everything.

That's why a chatbot may "forget" something from much earlier in a long conversation.

Bigger isn't automatically better

Large context windows are useful, but they come with trade-offs:

  • Cost: sending a huge document with every question is expensive.
  • Speed: long inputs take longer to process.
  • Attention: models can be less reliable at using information buried in the middle of very long inputs.

For most business applications, retrieving the right few passages beats stuffing everything into the context. Our guide to embeddings and vector databases explains how retrieval finds those passages.

Designing AI features around tokens

  • Keep instructions concise but complete.
  • Send only relevant data, not entire databases.
  • Summarise long histories periodically.
  • Set maximum output lengths where appropriate.
  • Cache repeated content if your provider supports prompt caching, which can reduce the cost of sending the same instructions repeatedly.
  • Choose the right model for each task; see our guide to small language models.

A worked example

Say a support assistant sends 1,500 tokens of instructions, 2,000 tokens of retrieved help articles and 500 tokens of conversation with each question, and writes a 300-token answer. That's 4,300 tokens per reply. At 10,000 replies a month, that's 43 million tokens. Trimming instructions, retrieving fewer but better passages and using a smaller model for simple questions can cut that figure substantially without hurting quality.

Counting tokens

Providers offer tokenizer tools and APIs to count tokens before sending a request. Building token counting into your application helps you predict costs, stay within limits and spot unexpectedly large requests.

Tokens beyond text

Multimodal models also convert images and audio into tokens, so a high-resolution image can use many tokens. Our guide to multimodal AI covers how these models work.

How we help

Our AI solutions team designs AI features that balance quality, speed and cost, including retrieval, caching and model selection. Our web app developers build token tracking and limits into your software, and our AI chatbot builds manage long conversations sensibly.

Frequently asked questions

How many words is 1,000 tokens?

Roughly 750 English words, although it varies with vocabulary, formatting and language.

Does a bigger context window mean better answers?

Not necessarily. It lets the model see more, but focused, relevant context usually produces better and cheaper results.

Why do AI chatbots forget earlier messages?

Long conversations can exceed the context window, so older messages are trimmed or summarised.

This guide is general information, not legal, tax or financial advice, and details such as prices, features and rules can change. Product and company names mentioned are trademarks of their respective owners and are used only to identify those products; UK Technology Hub is not affiliated with or endorsed by them.

Related guides

All AI & automation guides