LLM tokenization explained: BPE, vocabularies and why token counts differ

Key points

• The tokenizer is trained before the model and frozen with it. The model only ever sees token IDs, never characters.
• Byte-pair encoding starts from characters or bytes and repeatedly merges the most frequent adjacent pair; the number of merges sets the vocabulary size.
• Byte-level BPE starts from the 256 possible byte values, so any string can be encoded and there is no unknown token.
• The same text costs different token counts on different models and in different languages. Petrov and colleagues found gaps of up to 15 times between languages.
• Digits, whitespace and code split in tokenizer-specific ways, which affects arithmetic and how much code fits in the context window.
• Count tokens with the target model's own tokenizer or counting endpoint, never with another model's tokenizer.

What a token is

A language model does not read characters. Before text reaches the model, a tokenizer splits it into tokens and maps each token to an integer ID from a fixed vocabulary. The model looks up a learned vector for each ID, predicts the next ID, and the tokenizer turns predicted IDs back into text. A token can be a common word, a piece of a rarer word, a punctuation mark, a space joined to the word after it, or a few raw bytes of a character the vocabulary has no better unit for.

Whole-word vocabularies fail because they would be enormous and still miss new names, typos and other languages. Single characters make sequences very long, and the model's work grows with sequence length. Subword tokenization sits between the two: frequent strings get one token each and rare strings are built from smaller pieces, so any input can be represented. The GPT-2 authors describe BPE as a practical middle ground between character-level and word-level language modeling. The tiktoken README says that on average, in practice, each token corresponds to about 4 bytes; that average hides wide variation by language and content, as the sections below show.

• The tokenizer is trained separately, before the model, and is fixed once the model is trained. Changing it means retraining the model's embeddings.
• For a given tokenizer, encoding is deterministic: the same string gives the same IDs every time.
• Good tokenizers are lossless. Decoding the IDs returns the exact input, spaces included.
• Embedding models have tokenizers and token limits too; see text embeddings explained.

How byte-pair encoding works

Byte-pair encoding began as a compression method. Philip Gage described it in 1994: find the most frequent pair of adjacent bytes and replace it with a single byte that does not occur in the data, then repeat. Sennrich, Haddow and Birch adapted it for neural machine translation (ACL 2016). Instead of bytes they merge characters and character sequences: split every word into characters plus an end-of-word marker, count every adjacent pair across the corpus weighted by word frequency, merge the most frequent pair into a new symbol, and repeat. Each merge adds one symbol, so the final vocabulary is the starting characters plus the number of merges, which the authors call the only hyperparameter of the algorithm. For efficiency they do not count pairs that cross word boundaries.

Here is the whole training loop, run on the four-word example corpus from the Sennrich paper. It is plain Python, and the output in the comment is what it prints.

from collections import Counter

def merge(symbols, pair):
    out, i = [], 0
    while i < len(symbols):
        if symbols[i:i + 2] == pair:
            out.append(symbols[i] + symbols[i + 1]); i += 2
        else:
            out.append(symbols[i]); i += 1
    return tuple(out)

def train_bpe(word_counts, num_merges):
    vocab = {tuple(w) + ('</w>',): c for w, c in word_counts.items()}
    merges = []
    for _ in range(num_merges):
        pairs = Counter()
        for symbols, count in vocab.items():
            for pair in zip(symbols, symbols[1:]):
                pairs[pair] += count
        best = max(pairs, key=pairs.get)  # ties go to the pair seen first
        merges.append(best)
        vocab = {merge(s, best): c for s, c in vocab.items()}
    return merges

merges = train_bpe({'low': 5, 'lower': 2, 'newest': 6, 'widest': 3}, 6)
print(merges)
# [('e', 's'), ('es', 't'), ('est', '</w>'), ('l', 'o'), ('lo', 'w'), ('n', 'e')]

The first merge is e + s, because that pair appears 9 times: 6 in "newest" and 3 in "widest". The pair s + t also appears 9 times; this code breaks the tie by taking the pair it counted first, and real trainers each have their own tie rule. After three merges, est</w> is a single symbol, the suffix the corpus repeats. After six, "low" is one token and "newest" is three: ne, w and est</w>.

Encoding new text replays the learned merges on each word, in the order they were learned. A word the trainer never saw still encodes, because every single character stays in the vocabulary:

def merge(symbols, pair):
    out, i = [], 0
    while i < len(symbols):
        if symbols[i:i + 2] == pair:
            out.append(symbols[i] + symbols[i + 1]); i += 2
        else:
            out.append(symbols[i]); i += 1
    return tuple(out)

def encode(word, merges):
    symbols = tuple(word) + ('</w>',)
    for pair in merges:  # replay merges in the order they were learned
        symbols = merge(symbols, pair)
    return symbols

merges = [('e', 's'), ('es', 't'), ('est', '</w>'), ('l', 'o'), ('lo', 'w'), ('n', 'e')]
print(encode('lowest', merges))   # ('low', 'est</w>')
print(encode('slower', merges))   # ('s', 'low', 'e', 'r', '</w>')
print(encode('zest', merges))     # ('z', 'est</w>')

"lowest" never appeared in training, yet it becomes two tokens, low and est</w>. "slower" falls back to single characters where no merge applies. That fallback is how subword vocabularies handle open-ended input, and it is also why rare words, names and unfamiliar languages cost more tokens: they match fewer merges.

Byte-level BPE

Character-level BPE has a gap: its base alphabet is whatever characters appeared in training, so an unseen character, such as one from a rare script or a new emoji, has no symbol at all. Radford and colleagues closed it in the GPT-2 paper (2019) by running BPE over UTF-8 bytes. They note that BPE over Unicode code points would need a base vocabulary of over 130,000 symbols before any merges, while a byte-level version needs only 256. Every string is a sequence of bytes, so a byte-level tokenizer never meets an unknown token.

Applied naively, byte-level BPE made poor merges, the authors report: because the merge rule is greedy and frequency-based, it learned many versions of common words, such as "dog" followed by different punctuation marks, wasting vocabulary slots. Their fix was to stop BPE from merging across character categories, so letters, digits and punctuation stay apart, with an exception for spaces that they say significantly improves compression. GPT-2 ended up with a vocabulary of 50,257 tokens and a context of 1,024 tokens.

• In GPT-2-style tokenizers a space is usually merged with the word that follows it, so hello with a leading space and hello at the start of a line are different tokens with different IDs.
• UTF-8 uses one byte for ASCII and two to four bytes for other characters. A character that no merge covers can cost a token per byte.
• The base of 256 bytes is fixed; everything else in the vocabulary depends on the training data, which is why two byte-level tokenizers can split the same text very differently.

WordPiece, SentencePiece and the Unigram model

WordPiece comes from Google's speech recognition and translation work. In the GNMT paper, Wu and colleagues describe a wordpiece model built with a data-driven approach that maximizes the language-model likelihood of the training data, plus a special symbol that marks the start of each word so the original words can be recovered without ambiguity. They found that vocabularies of 8k to 32k wordpieces gave both good accuracy and fast decoding across the language pairs they tried. BERT used WordPiece embeddings with a 30,000-token vocabulary, and its tokenizer marks a piece that continues a word, as in flight ##less.

Kudo and Richardson released SentencePiece (EMNLP 2018) to train subword models directly from raw sentences, with no language-specific pre-tokenization. That matters for languages written without spaces between words, such as Japanese and Chinese. SentencePiece treats input as a plain sequence of Unicode characters and handles whitespace as an ordinary symbol, escaping it as ▁ (U+2581), so decoding is a simple join and tokenization is lossless. It implements two algorithms: BPE, and the unigram language model.

The unigram model (Kudo, ACL 2018) works in the opposite direction from BPE. Instead of growing a vocabulary by merges, it starts from a large seed vocabulary of frequent substrings, estimates each piece's probability with the EM algorithm, and repeatedly drops the pieces whose removal lowers the corpus likelihood least, always keeping single characters, until it reaches the target size. To encode, it picks the most probable segmentation with the Viterbi algorithm. Because it is a probabilistic model it can also sample other segmentations of the same text. Kudo calls training on those samples subword regularization and reports consistent gains, especially in low-resource and out-of-domain settings.

Vocabulary size trade-offs

Vocabulary size has costs on both sides. The GPT-2 paper mentions 32,000 to 64,000 tokens as typical for BPE vocabularies of its time; BERT used 30,000, GPT-2 50,257, and the GNMT authors found 8k to 32k worked well for translation. Larger vocabularies have become common since, mostly to cover more languages and code with fewer tokens.

For an application developer the vocabulary comes with the model, so it is not a setting you change. It does explain why moving to a model with a different tokenizer changes token counts, context fit and usage for exactly the same prompts.

Why the same text costs different token counts

Three things decide how many tokens a text costs: the tokenizer's vocabulary, the data it was trained on, and the text itself. A tokenizer trained mostly on English learns long merges for English words and few for languages it rarely saw, which then fall back to short pieces or single bytes.

Petrov, La Malfa, Torr and Bibi measured this in "Language Model Tokenizers Introduce Unfairness Between Languages" (NeurIPS 2023). They found that the same text translated into different languages can differ in tokenized length by up to 15 times, that tokenizers built for multilingual use still show the gap, and that even character-level and byte-level models differ by more than 4 times for some language pairs. Because usage is metered and limited per token, the authors point out that speakers of some languages pay more for the same content, wait longer for responses and fit less into the context window.

Counts also move between models from one vendor. Anthropic's token counting documentation says Claude 4.7 and later models use a newer tokenizer, and that the same input text produces approximately 30 percent more tokens than on earlier models, with the exact increase depending on content. It tells developers to recount prompts against the model they plan to use instead of reusing counts from an earlier one.

The sketch below compares any set of tokenizers on the same texts. With only built-in stand-ins (characters, UTF-8 bytes and whitespace-separated words) it already shows the base problem: the Hindi and Japanese sentences need about three bytes per character, so a byte-level tokenizer starts from roughly three times as many symbols before any merges, and splitting on spaces finds just one "word" in Japanese. The counts in the comments are the real output. Add the actual tokenizers of the models you are comparing and run it on your own prompts.

def compare(tokenizers, texts):
    # tokenizers: name -> function that turns text into a list of tokens
    for label, text in texts.items():
        counts = {name: len(fn(text)) for name, fn in tokenizers.items()}
        print(label, counts)

tokenizers = {
    'chars': list,
    'utf8_bytes': lambda t: list(t.encode('utf-8')),
    'words': str.split,
    # add real tokenizers here, e.g. 'model_a': enc_a.encode
}
compare(tokenizers, {
    'English': 'The weather is nice today.',
    'Hindi': 'आज मौसम अच्छा है।',
    'Japanese': '今日はいい天気です。',
})
# English {'chars': 26, 'utf8_bytes': 26, 'words': 5}
# Hindi {'chars': 17, 'utf8_bytes': 45, 'words': 4}
# Japanese {'chars': 10, 'utf8_bytes': 30, 'words': 1}

Numbers, code and whitespace

Tokenizers split numbers differently. Singh and Strouse (2024) note that LLaMA and PaLM split numbers into single digits, while GPT-3.5 and GPT-4 have separate tokens for 1-, 2- and 3-digit numbers. Chunking from the left means chunks do not line up with place value: in a scheme of up to three digits read left to right, 1234 splits as 123 and 4, while 56789 splits as 567 and 89, so the last digit is a token of its own in one number and part of a two-digit token in the other. The authors found that forcing right-to-left chunks, by writing numbers with comma separators, largely improved arithmetic for GPT-3.5 and GPT-4, that the errors from left-to-right chunks follow systematic patterns, and that the gap shrinks as models get larger.

The practical rule: do not rely on a model for exact arithmetic over long numbers. Give it a calculator or code tool, and format numbers in prompts consistently.

Code is dense with symbols, identifiers and indentation. Names in camelCase or snake_case split into several pieces, and indentation can cost tokens on every line, depending on whether the tokenizer learned tokens for runs of spaces and tabs. Text with few repeated substrings, such as minified code, long hex strings, base64 and UUIDs, tokenizes poorly; a base64-encoded file can cost far more tokens than the text it encodes.

Whitespace is part of the text. Because a space usually attaches to the word after it, a prompt that ends in a trailing space can leave the model starting its reply from an unusual token boundary, and two strings that look identical but differ in spaces, tabs or Unicode normalization tokenize differently. Normalize text the same way wherever it is produced and compared, and trim trailing spaces from prompts. Token boundaries also matter when output is constrained to a format; see constrained decoding and structured outputs.

Special tokens and chat templates

Beside the learned pieces, every vocabulary reserves special tokens that ordinary text never produces: markers for the end of a document, for the start and end of each chat message, for padding, and in some models for tool calls. GPT-2-era tokenizers use <|endoftext|>; BERT starts every input with a classification token called CLS and separates segments with one called SEP.

Chat models are trained on conversations rendered through a template that wraps each message in special tokens naming its role. The exact template differs by model family. An illustrative ChatML-style rendering looks like this:

<|im_start|>system
Answer briefly.<|im_end|>
<|im_start|>user
What is a token?<|im_end|>
<|im_start|>assistant

• Hosted chat APIs apply the template for you, and the wrapping tokens count toward usage, which is one reason a message costs a few more tokens than its text alone.
• When you serve an open-weight model yourself, use the template that ships with its tokenizer exactly. A missing or extra newline can lower output quality without raising any error.
• Never let user text become special tokens. If a user types the literal string <|im_end|> and your code encodes it as the real special token, they can close their own message early and write text that looks like a system or assistant turn. tiktoken guards against this: by default its encode raises an error when the text contains a special token, and its docstring explains that special tokens can be used to trick a model into doing something you do not want.

Counting tokens before you send, and why limits are in tokens

Context windows, maximum output lengths, rate limits and usage are all measured in tokens because tokens are what the model processes. Every token in the context is a position the model attends over, and during generation each one holds memory in the KV cache, as prompt caching and the KV cache explains. A limit stated in tokens therefore holds a different amount of text for each model and each language.

• For OpenAI models, tiktoken runs locally: tiktoken.encoding_for_model("gpt-4o") returns the matching encoding, and len(enc.encode(text)) counts a string. Chat formatting and tool definitions add tokens on top.
• For Claude models, Anthropic provides a count_tokens endpoint that accepts the same inputs as a message, including the system prompt, tools, images and PDFs, and returns input_tokens. Its documentation says the count is an estimate that can differ from actual usage by a small amount, that the endpoint is free to use but rate limited, and that it does not apply prompt caching.
• For open-weight models, load the tokenizer that ships with the model and apply its chat template before counting.
• Never estimate one model's count with another model's tokenizer. A characters-divided-by-four rule is fine for a rough English budget and wrong for other languages and for code.

Use the counts to manage a budget: reserve room for the reply, trim or summarize history before a request hits the limit rather than after it fails, and size retrieval chunks in tokens, as in chunking strategies for RAG. Most of the levers in how to reduce LLM costs come down to sending fewer tokens.

If you are building an AI agent or LLM feature and want help with token budgets, context management and evaluation, see our AI agent development service: the code lives in your GitHub from day one, AI-assisted work is reviewed by an engineer, and the IP is assigned to you.