LLM Token Counts Across Languages — Why Chinese Costs More
The same information expressed in different languages can produce 2–5× different token counts, directly affecting API cost. This guide explains why and provides multipliers for 10+ languages.
The core problem
LLM tokenizers are trained predominantly on English. English words tokenize efficiently — common words are single tokens. Languages with different scripts or morphological structures tokenize less efficiently, producing more tokens for the same semantic content.
Language token multipliers (vs English)
These are approximate multipliers for the same content expressed in each language: English: 1.0× (baseline). Spanish/French: 1.1–1.2×. German: 1.1–1.3× (compound words). Russian: 1.4–1.6×. Arabic: 1.5–2.0×. Japanese: 1.2–1.5× (Katakana efficient, Kanji less so). Chinese (simplified): 1.5–2.5× (characters may be 1–2 tokens each). Korean: 1.7–2.5×. Thai/Hindi: 2.0–3.0×.
Why code tokenizes differently
Programming languages have highly consistent syntax. Python and JavaScript tokenize at roughly 1.2–1.5× English prose for equivalent functionality. SQL often tokenizes at 1.0–1.1× (many short keywords). JSON with verbose keys can be 1.5–2.0×.
Impact on your budget
A multilingual customer support bot handling 50% non-English queries at 2× token efficiency effectively runs at 1.5× the cost of an English-only bot. Budget accordingly. Also: multilingual content that uses Unicode escapes (\uXXXX) tokenizes extremely inefficiently — always use native Unicode.
Related calculators
Related guides
Frequently Asked Questions
Which LLM handles non-English text most efficiently?
Models specifically optimized for multilingual content (Qwen 2.5, Mistral multilingual) often have more efficient tokenizers for non-Latin scripts. Claude and GPT models handle multilingual content well quality-wise, but may be less token-efficient than purpose-built multilingual models.
Guide updated 2026-08-20. Pricing data verified weekly. See methodology · sources.