Definition
A tokenizer maps text to discrete token IDs for LLM input/output. Different vendors and model generations use different vocabularies, so the same bytes can produce different token counts — directly affecting billed cost and context-window usage.
Key Points
- Anthropic Sonnet 5 / Opus 4.8 / Fable 5 use a newer tokenizer (~30% more tokens vs prior Anthropic tokenizer; content-dependent)
- OpenAI o200k-family tokenizers remain a common cross-vendor baseline for comparisons
- Effective price = list $/MTok × (tokens_new / tokens_old)