Tokenization

In natural language processing, tokenization converts text into units such as words, subwords, or individual characters. Modern language models commonly use subword tokenization, allowing them to represent unfamiliar words and efficiently process large vocabularies. The resulting tokens are converted into numerical representations that the model can process.

Tokenization determines how text is represented and processed by a language model. It affects context length, computational requirements, model performance, and how efficiently information can be encoded. Tokenization behavior can also be relevant to AI security because security controls may interpret tokenized input differently from the underlying text.