Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs.
Angelica ChenRavid Schwartz-ZivKyunghyun ChoMatthew L. LeavittNaomi SaphraPublished in: CoRR (2023)
Keyphrases
- phase transition
- random constraint satisfaction problems
- constraint satisfaction
- satisfiability problem
- combinatorial problems
- randomly generated
- hard problems
- np complete
- sat problem
- graph coloring
- np complete problems
- frequent itemsets
- random instances
- stochastic local search
- itemsets
- deep structure
- random graphs
- context free grammars
- sat instances
- reinforcement learning
- objective function