Definition
Tokenization is the process of breaking text into smaller units called tokens, which might be whole words, parts of words, or punctuation marks. LLMs process tokens, not raw text, and pricing is typically based on token counts.
Example
The sentence "AI is powerful" contains approximately four tokens. The word "unbelievable" might be split into three tokens: "un," "believ," and "able."
Why it matters
Understanding tokens explains why longer prompts cost more, why there are limits to context window size, and why some words are more expensive to process than others.