What is a Token? LLM Tokenization Explained
How GPT and other LLMs split text into tokens with Byte Pair Encoding, why token count drives API cost and context limits, and which models you can count offline.
A token is the unit an LLM actually reads and writes: not a character, not a word, but a piece produced by an algorithm called Byte Pair Encoding (BPE). Before any text reaches the model, a tokenizer breaks it into a sequence of these pieces and converts each one to an integer ID, since the model's underlying math only operates on numbers.
BPE is trained ahead of time on a huge corpus of text: it starts from individual bytes, then repeatedly merges whichever adjacent pair appears most often, until it reaches a fixed vocabulary size (100,000 to 200,000 entries for OpenAI's models). Common English words end up as a single token; rarer words, misspellings, and most non-English text get split into several.
A real example
This is the actual token split for a short sentence, produced by OpenAI's o200k_base encoding:
9 tokens for 53 characters. Two things worth noticing:
- "Tokenization" splits into "Token" + "ization" (a less common word), while every other word here stays whole (all common English words).
- Most tokens carry their leading space as part of the token itself (" splits", not "splits"). This is why token counts are sensitive to whitespace and formatting, not just to the words themselves.
Paste your own text into the GPT Token Counter to see its real count instead of estimating from the word-count rule of thumb.
Why token count matters
| Constraint | How tokens factor in |
|---|---|
| API pricing | Billed per token, usually per million, with separate input and output rates. Token count converts directly into cost. |
| Context window | Every model has a fixed token budget covering prompt plus completion combined. Dense content (code, JSON, non-English text) can exceed the limit well before it "feels" long. |
| Latency | Generation time scales roughly with output token count, since most models generate one token at a time. |
Which encoding does which model use?
| Encoding | Vocabulary size | Used by |
|---|---|---|
o200k_base | ~200,000 | GPT-4o, o1, o3 |
cl100k_base | ~100,000 | GPT-4, GPT-3.5 |
These two encodings are not interchangeable. The same input text produces a different token sequence, and usually a different total count, depending on which one a given model actually uses.
Not every model can be tokenized offline
Counting tokens client-side, without sending the text anywhere, requires the tokenizer's vocabulary data to be available to run locally. That is true for OpenAI's encodings, but not universally true across providers:
- Anthropic (Claude 3 and later) ships no local tokenizer at all. The only supported method is their Count Tokens API, which needs a network call and an API key.
- Google (Gemini) provides a local tokenizer, but only in the Python and Go SDKs, and it does not cover the newest model releases at launch.
- Open-weight models (Kimi K2 and similar) publish their tokenizer files, but a 160,000-entry vocabulary means several megabytes of data just to load, which works against the idea of a lightweight, instant browser tool.
This is why a tool promising accurate, offline token counts for every model at once cannot actually deliver on that promise honestly today.
Frequently asked questions
No. For English text, one token is closer to 0.75 words, or about 4 characters. Common short words are usually a single token, but longer or rarer words often split into two or three ("tokenization" might become "token" + "ization").
The ratio gets worse for anything that is not everyday English: source code, JSON, and non-Latin scripts typically need more tokens per character, since the BPE vocabulary was trained mostly on natural-language web text.