Definition
The fundamental units of text or data—such as words, punctuation marks, or subword pieces—that an AI model processes and generates.
In short: The fundamental units of text or data—such as words, punctuation marks, or subword pieces—that an AI model processes and generates.