Information-Theoretic Foundations for Neural Scaling Laws.
Hong Jun JeonBenjamin Van RoyPublished in: CoRR (2024)
Keyphrases
- information theoretic
- mutual information
- information theory
- theoretic framework
- information theoretic measures
- jensen shannon divergence
- neural network
- information bottleneck
- minimum description length
- entropy measure
- kullback leibler divergence
- multi modality
- computational learning theory
- kl divergence
- log likelihood
- relative entropy
- probability distribution
- pattern recognition
- machine learning
- data mining