What is a token in AI? Tokens and context windows, explained

A token is the small piece of text an AI language model reads and writes: a whole word, part of a word, a number or a punctuation mark. Models don’t see letters or words the way we do; they see a sequence of tokens. As a rule of thumb, a token is about four characters of English, and 100 tokens are about 60 to 80 words (Google). Everything about a model’s limits is counted in tokens: how much it can read at once (its context window), how long a reply can be, how fast it writes, and, for cloud models, what it costs.

This guide explains tokens and context windows in plain words, with real numbers from models you can run on an iPhone or a Mac.

What is a token?

Before a model sees your message, a tokenizer cuts the text into pieces from a fixed vocabulary, and each piece gets a number. Common words are usually one token. Long or rare words are split into several. Spaces, punctuation and digits count too.

For example, a sentence like “Unbelievably, the cat sat.” might be split roughly like this:

Un believ ably , the cat sat .

That’s 8 tokens for 4 words. The exact split depends on the model: each model family has its own tokenizer and vocabulary. Microsoft’s Phi 4 Mini, for instance, has a vocabulary of about 200,000 tokens (Microsoft), so the same text can come out as a different number of tokens in another model.

A few things follow from this:

• The rule of thumb is for English. Vocabularies are built from the text a model was trained on, so a language it saw less of can be split into more, smaller pieces.
• Code, numbers and unusual words are token-hungry. A long serial number can take many tokens.
• Token counts are estimates across models. A “1,000-token” document in one model may be 1,100 in another.

Why tokens matter

What	Measured in tokens because…
Context window	It’s the most text the model can consider at once
Reply length	Apps cap how many tokens a reply may have
Speed	Models write one token at a time, so speed is tokens per second
Price (cloud models)	Providers charge per million tokens sent in and written out
Thinking	A reasoning model’s thinking is extra tokens before the answer

When you use a cloud model through a service like OpenRouter, prices are listed per million tokens, separately for the tokens you send and the tokens the model writes. With a model running on your own device, tokens cost nothing; they only cost time and memory.

What is a context window?

The context window is the most tokens a model can work with at once: your current message, the earlier messages of the chat, any instructions, any document passages, and the reply it’s writing. Anything outside the window simply isn’t seen.

Model makers publish the longest context a model was built for. These are the figures in Modelbase’s catalog for a few models you can run on a phone:

Model	Context length the model supports
Qwen 3.5 (0.8B, 2B, 4B)	262,144 tokens
Llama 3.2 (1B, 3B)	131,072 tokens
Phi 4 Mini (3.8B)	131,072 tokens
Gemma 3 (1B)	32,768 tokens
SmolVLM 2 (256M, 500M)	8,192 tokens

128K tokens is a whole book’s worth of text. But that’s what the model can handle, not what fits in a phone’s memory, which brings us to the part most explainers skip.

Why your phone uses a smaller context window

While a model writes, it keeps a working memory of every token in the context (the “KV cache”), and that memory grows with each token. For a small model like Qwen 3 (0.6B), each token takes about 112 KB: 28 layers × 8 key-value heads × 128 values × 2 (keys and values) × 2 bytes, from the model’s published configuration (Qwen). So 8,192 tokens of context need about 0.9 GB of memory on top of the model itself, and 131,072 tokens would need about 15 GB, far more than any iPhone gives an app.

That’s why Modelbase picks the context window by your device’s memory when it’s set to Automatic:

Device memory	Context window (Automatic)
iPhones with about 4 GB	2,048 tokens
iPhones and iPads with about 6 GB	4,096 tokens
8 GB and more	8,192 tokens
Mac	8,192 tokens

Models that see images keep a little memory back for the picture, so on the smallest iPhones they get a slightly smaller window.

8,192 tokens is still a lot of conversation: at 60 to 80 words per 100 tokens, it’s several thousand words. You can change it under Settings > Generation > Context window (Generation is in the Advanced section): 2K, 4K, 8K, 16K or 32K tokens. The app warns that larger windows use more memory and can crash on small phones, so raise it only on a device with memory to spare, such as a Mac.

What happens when a chat gets longer than the context window

A long chat would eventually overflow any window. Instead of silently dropping the oldest messages, Modelbase summarizes earlier messages and keeps the summary in the context, so the model still knows what you talked about an hour ago, in a shorter form.

You can see this at any time. The small ring at the end of the message field shows how much of the context your next message would fill. Tap it to open Token Usage:

• Tokens used: the last prompt plus its reply in this chat;
• Context Length: the window the current model has on this device;
• Context used: the share of it;
• Summarized: whether earlier messages were folded into a summary, and how many.

The app counts tokens with the current model’s own tokenizer, so the numbers are that model’s real count.

How documents fit in a small context window

If 8,192 tokens is a few thousand words, how can a phone answer questions about a 300-page PDF? It doesn’t put the whole PDF in the context. Modelbase splits the document into short passages and indexes them on the device; for each question, it finds the few passages that matter and gives only those to the model. Our guide on chatting with a PDF on iPhone explains it step by step.

Tokens, thinking and reply length

Reasoning models write their thinking in tokens before the answer, and those tokens take room in the context and time to generate. That’s why Modelbase’s Medium thinking level stops the thinking at 1,024 tokens on device, and why a reasoning reply takes longer than a plain one. See what a reasoning model is for the details.

Context windows of cloud models

Cloud models often have much larger windows, because they run on servers with far more memory. In Modelbase, an OpenRouter model uses the context length OpenRouter reports for it, and for a server of your own you set each model’s context length yourself (from 4,096 to 131,072 tokens), because such servers don’t say. If you need a model to read a very long text in one go, that’s where a cloud model or your own computer helps; see free LLM APIs for ways to try cloud models without paying.

For the basics of running models on your own device, start with how to run an LLM locally on iPhone, iPad and Mac and what a small language model is. Modelbase is free on the App Store for iPhone, iPad and Mac.

FAQ

How many words is a token?

About three quarters of a word in English, on average. Google puts it at about 4 characters per token, with 100 tokens equal to about 60 to 80 English words. Other languages and code usually need more tokens per word.

What is a context window in AI?

It’s the most text, counted in tokens, that a model can take into account at once: the conversation so far, any instructions or documents, and the reply it’s writing.

Why does my iPhone have a smaller context window than the model supports?

Because every token in the context takes memory while the model works. A phone gives an app a few gigabytes at most, so Modelbase sets 2,048 to 8,192 tokens depending on the device’s memory, even for models built for 128K or more.

What happens when a conversation is longer than the context window?

In Modelbase, earlier messages are summarized and the summary stays in the context, so the model keeps the gist of the conversation. The Token Usage sheet shows how many messages were summarized.

Do tokens cost money?

Only with cloud models that charge per token. A model running on your own iPhone or Mac costs nothing per token.

Is a bigger context window always better?

Not on a phone. A bigger window holds more conversation but uses more memory and slows replies down, and on a small phone it can make the app run out of memory. For long documents, picking the relevant passages works better than a huge window.