Language models don’t read words — they read tokens. A token is the small unit a model actually processes, and almost everything practical about working with AI (cost, context limits, speed) is measured in them. Understanding tokens is the fastest way to stop being surprised by an AI system.
What a token is
Tokenization splits text into pieces — sometimes a whole word, often a fragment. “Tokenization” might become token + ization. A rough rule of thumb for English: one token is about four characters, and 100 tokens is roughly 75 words.
Tokens beyond text
- Images — split into patches, each turned into a token a vision model can read.
- Audio — sampled into discrete units representing slices of sound.
- Code — tokenized much like text, with structure-aware splitting.
Why tokens matter in practice
- Context window — a model’s memory is a token budget; overflow it and the oldest content falls out.
- Cost — you pay per token, in and out, so prompt design is cost design.
- Latency — more tokens to generate means more time to wait.
“You can’t reason about an AI system’s cost, speed or limits until you think in tokens. It’s the unit the whole field is priced in.”
KnackLabs
.png)