Definition
The text processing step of breaking down a stream of text into smaller, meaningful units called tokens, which can be words, subwords, or individual characters.
In short: The text processing step of breaking down a stream of text into smaller, meaningful units called tokens, which can be words, subwords, or individual characters.