Definition
An unsupervised text tokenizer that learns to break text into subword units, capable of handling inputs from multiple languages without reliance on language-specific pre-tokenization.
In short: An unsupervised text tokenizer that learns to break text into subword units, capable of handling inputs from multiple languages without reliance on language-specific pre-tokenization.