What Is a Token? LLM Tokenization Explained

The quick answer

A token is the smallest unit a language model processes. For English text, 1 token ≈ 4 characters ≈ 0.75 words. "The quick brown fox" is 5 tokens. A 10,000-word document is roughly 13,000 tokens.

Why tokens, not words?

Models use tokenizers (algorithms like BPE — Byte Pair Encoding) that split text into subwords. Common words are single tokens; rare words split into 2–4 tokens. "tokenization" might be "token" + "ization" = 2 tokens. This is why token counts don't map 1:1 to word counts.

Token counts differ by model

GPT models use the cl100k_base or o200k_base tokenizer. Claude uses its own tokenizer. Llama uses SentencePiece. The same sentence produces different token counts across models — typically within ±15%, but can differ more for code, numbers, or non-English text.

Why tokens matter for cost

Every API call bills you for input tokens (your prompt) + output tokens (the response). At $3.00/M input tokens, 1 million tokens = $3.00. A 2,000-token system prompt sent 50,000 times/month = 100M tokens = $300. Token awareness is cost awareness.

Code, numbers, and non-English text use more tokens

Python code, JSON, and SQL typically run 1.2–1.5× the token count of equivalent English prose. Non-Latin scripts (Chinese, Japanese, Arabic) are particularly token-heavy — a Chinese character may be 1–3 tokens vs 0.25 for an English letter. Run your specific content through a tokenizer counter before estimating costs.