Before a computer can understand a single word you write, it has to do something much more basic.
It has to figure out where one piece of text ends and the next begins.
That process is called tokenization, and it's the first step in almost every NLP system ever built. Whether you're using a spam filter, a search engine, or ChatGPT, tokenization happened before anything else. It's the foundation everything else is built on.
This post explains what tokenization is, why it matters, and how it has evolved from simple word splitting into the sophisticated subword methods that power modern AI. Along the way, we'll also cover the related concepts you'll encounter whenever you work with text: stemming, lemmatization, and n-grams.
What Is Tokenization?
Tokenization is the process of breaking text into smaller units called tokens.
A token is just a piece of text. It might be a word, a punctuation mark, a part of a word, or even a single character. The important thing is that tokens are the units a model actually works with. Before any analysis, translation, or generation can happen, the raw text needs to be split into these pieces.
Take this sentence:
"NLP is surprisingly powerful."
After word tokenization, it becomes:
`["NLP", "is", "surprisingly", "powerful", "."]`
That's it. Five tokens. The model now has something it can process.
Simple in concept. But the details turn out to matter enormously.
Why Tokenization Is Harder Than It Looks
You might be thinking: can't you just split on spaces?
For English, that gets you most of the way there. But language is messier than that.
What do you do with "don't"? Is it one token or two ("do" and "n't")? What about "New York"? Splitting on spaces gives you two tokens, but it's one concept. What about emojis, URLs, email addresses, or code snippets mixed into text?
And that's just English. Languages like Chinese and Japanese have no spaces between words at all. Arabic and Hebrew write without vowels. German creates long compound words that are really several concepts joined together.
Tokenization that works well across all of these is a genuinely hard problem. And how you solve it has downstream consequences for everything the model learns.
Word Tokenization
The most intuitive approach is word tokenization: split the text into individual words.
This works well for clean, formal text. It's fast, easy to understand, and produces tokens that map directly to human intuitions about language.
The limitation is vocabulary size. If you train a model on word tokens, it can only handle words it has seen before. Give it "unimaginably" when it's only ever seen "imagine" and "imaginable", and it's stuck. This is called the out-of-vocabulary (OOV) problem, and it's one of the main reasons word tokenization has been largely superseded for modern AI systems.
Stemming and Lemmatization: Reducing Words to Their Core
Before diving into more advanced tokenization, it's worth understanding two related text processing techniques you'll encounter constantly: stemming and lemmatization.
Both answer the same question: when should "running", "runs", and "ran" be treated as the same word?
Stemming
Stemming is the blunter approach. It chops off the ends of words using simple rules until you're left with a root form called a stem.
"Running" becomes "run". "Fishing" becomes "fish". "Better" becomes "better" (stemming can't handle irregular forms).
It's fast and simple, but it's crude. The stem doesn't have to be a real word. "Arguing" might become "argu". "Studies" might become "studi". That's fine for some applications, like topic modelling or search indexing, where you just need to group related words together. It falls apart when you need the actual meaning of a word.
Lemmatization
Lemmatization is the smarter approach. Instead of blindly chopping characters, it uses vocabulary and grammar rules to find the true base form of a word, called the lemma.
"Running" becomes "run". "Better" becomes "good". "Was" becomes "be".
Lemmatization understands that "better" is a form of "good", something stemming can never figure out. It's slower and requires more linguistic knowledge, but the output is always a real, meaningful word.
The key difference:
"Stemming is fast and crude. Lemmatization is slower and correct. Choose based on what your task actually needs."
When to use each:
- Stemming: search indexing, topic modelling, situations where speed matters and rough grouping is enough
- Lemmatization: sentiment analysis, question answering, any task where word meaning matters
Subword Tokenization: The Modern Solution
Neither word tokenization nor character tokenization is quite right for modern AI. Word tokenization has the OOV problem. Character tokenization produces sequences so long they're computationally impractical.
The solution the field landed on is subword tokenization: split words into pieces, but not all the way down to individual characters.
The idea is elegant. Common words stay as single tokens. Rare or unknown words get split into recognisable subword pieces. "Tokenization" might become "token" + "ization". "Unbelievable" might become "un" + "believ" + "able".
This gives the model the best of both worlds. It can handle any word, including ones it has never seen, because it can always fall back to subword pieces. And it keeps vocabulary size manageable.
Byte Pair Encoding (BPE)
Byte Pair Encoding (BPE) is the most widely used subword tokenization algorithm. It powers GPT, GPT-2, GPT-3, GPT-4, and many other major models.
The algorithm works like this:
Start with every character as its own token. Then repeatedly find the most common pair of adjacent tokens and merge them into a single new token. Repeat until you reach your desired vocabulary size.
So if "lo" and "w" appear together constantly, they get merged into "low". If "low" and "er" appear together constantly, they get merged into "lower". The algorithm keeps building up frequent sequences into single tokens.
The result is a vocabulary that naturally captures common words and common word fragments, while still being able to handle anything new by falling back to shorter pieces.
WordPiece
WordPiece is a similar approach used by BERT and many Google models. The main difference is in how it decides what to merge. BPE merges the most frequent pairs. WordPiece merges the pairs that increase the likelihood of the training data the most.
In practice, the results look similar. Subword pieces in WordPiece are often prefixed with `##` to indicate they're continuations of a word. "playing" might become "play" + "##ing".
SentencePiece
SentencePiece takes a slightly different approach: it treats the input as a raw stream of characters (including spaces) rather than pre-tokenized words. This makes it language-agnostic. It works equally well on English, Japanese, Chinese, or any other language without needing language-specific rules. It's used in models like T5 and many multilingual systems.
N-grams: Capturing Context Between Tokens
Once you have tokens, a natural next question is: does the order matter?
Almost always, yes. "Not good" means something very different from "good". "Bank robbery" means something different from "robbery bank".
N-grams are a simple way to capture this. An n-gram is a sequence of n consecutive tokens.
- A unigram is a single token: "good"
- A bigram is two consecutive tokens: "not good"
- A trigram is three: "not very good"
By looking at n-grams instead of individual tokens, models can capture short-range context. "New York" as a bigram carries different meaning than "New" and "York" separately.
N-grams were the backbone of language modelling before neural networks. They're still used in text classification, spam detection, and search systems where simplicity and speed matter more than capturing long-range context.
The limitation is obvious: n-grams only capture context within a short window. A trigram can't know that the word at the start of a paragraph shapes the meaning of a word at the end. That's what neural networks, and ultimately the Transformer, were built to solve.
How Modern LLMs Tokenize Text
If you've ever used ChatGPT or Claude and noticed that pricing is measured in "tokens," this is where that comes from.
Modern large language models use BPE or similar subword methods. Every piece of text you send gets broken into tokens before the model ever sees it. The model generates its response one token at a time.
On average, one token is roughly four characters in English, or about three-quarters of a word. "Tokenization" is two tokens. "Hello" is one. A typical paragraph is around 100 tokens.
This is also why LLMs sometimes handle unusual words or names strangely. If a word gets split into many subword pieces, the model has to work harder to reason about it as a coherent unit. The tokenization shapes what the model finds easy or difficult.
Putting It Together
Here's a quick summary of the landscape:
- Word tokenization: simple, intuitive, but can't handle unknown words
- Stemming: fast root-finding, crude, sometimes produces non-words
- Lemmatization: accurate root-finding, slower, always produces real words
- Subword tokenization (BPE, WordPiece, SentencePiece): the modern standard, handles any word, used in all major LLMs
- N-grams: sequences of tokens that capture short-range context, still used in classical NLP
Each approach has its place. For production NLP systems today, subword tokenization is almost always the right choice. For understanding older systems or building lightweight classifiers, word tokens and n-grams still have their role.
What Comes Next
Tokenization turns raw text into something a model can process. But once you have tokens, the next question is: what role does each one play in the sentence?
That's what Part of Speech Tagging covers. It's the process of labelling each token with its grammatical role: noun, verb, adjective, and so on. It sounds simple, but it's one of the foundational steps that allows NLP systems to understand the structure of language, not just the words themselves.
In the next post in this series, we'll break down how POS tagging works, why it matters, and where it shows up in real NLP pipelines.
Learn This at Fondra Labs
This post is part of our NLP Foundations series, where we cover everything from the basics of text processing to the architectures powering modern AI systems.
At Fondra Labs, we teach AI from production reality, not hype. Every concept in this series is chosen because it matters in practice, not just in theory. Tokenization, embeddings, retrieval, evaluation: these are the building blocks that engineers actually use when they build real systems.
If you found this useful, explore the rest of the blog. We cover machine learning, deep learning, NLP, and the practical skills that bridge the gap between understanding AI and building with it.
Frequently Asked Questions
Javier Aguirre
Teaching AI from production reality, not hype.
Build your AI Foundations
Get the free checklist that production engineers use to avoid these common RAG pitfalls.
Get the Free Guide