Tokenization Is More Than Compression.

Craig W. Schmidt Varshini Reddy Haoran Zhang Alec Alameddine Omri Uzan Yuval Pinter Chris Tanner

Published in: CoRR (2024)

Keyphrases

data compression
image compression
named entities
compression algorithm
compression ratio
biomedical information retrieval
image processing
database
multiresolution
lossless compression
lossy compression
compression rate
text compression