Toward a Theory of Tokenization in LLMs.

Nived Rajaraman Jiantao Jiao Kannan Ramchandran

Published in: CoRR (2024)

Keyphrases

theoretical framework
named entities
theoretical basis
social networks
image processing
expert systems
email
knowledge discovery
information theory
formal theory
biomedical text