Jeffrey Pennington

Publication Activity (10 Years)

Years Active: 2011-2024
Publications (10 Years): 67

Top Topics

Learning Curves

Random Matrix Theory

Top Venues

Publications

Atish Agarwala, Jeffrey Pennington
High dimensional analysis reveals conservative sharpening and a stochastic edge of stability. CoRR (2024)
Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie E. Everett, Alexander A. Alemi, Ben Adlam, John D. Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, Simon Kornblith
Small-scale proxies for large-scale Transformer training instabilities. ICLR (2024)
Elliot Paquette, Courtney Paquette, Lechao Xiao, Jeffrey Pennington
4+3 Phases of Compute-Optimal Neural Scaling Laws. CoRR (2024)
Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron T. Parisi, Abhishek Kumar, Alexander A. Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Fathy Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, Noah Fiedel
Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models. Trans. Mach. Learn. Res. 2024 (2024)
Brian Lester, Jaehoon Lee, Alex Alemi, Jeffrey Pennington, Adam Roberts, Jascha Sohl-Dickstein, Noah Constant
Training LLMs over Neurally Compressed Text. CoRR (2024)
Katie Everett, Lechao Xiao, Mitchell Wortsman, Alexander A. Alemi, Roman Novak, Peter J. Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, Jeffrey Pennington
Scaling Exponents Across Parameterizations and Optimizers. CoRR (2024)
Atish Agarwala, Fabian Pedregosa, Jeffrey Pennington
Second-order regression models exhibit progressive sharpening to the edge of stability. ICML (2023)
Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin F. Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, Noah Fiedel
Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models. CoRR (2023)
Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D. Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, Simon Kornblith
Small-scale proxies for large-scale Transformer training instabilities. CoRR (2023)
Atish Agarwala, Samuel Stern Schoenholz, Jeffrey Pennington, Yann N. Dauphin
Temperature check: theory and practice for training models with softmax-cross-entropy losses. Trans. Mach. Learn. Res. 2023 (2023)
C. Daniel Freeman, Laura Culp, Aaron Parisi, Maxwell L. Bileschi, Gamaleldin F. Elsayed, Alex Rizkowsky, Isabelle Simpson, Alex Alemi, Azade Nova, Ben Adlam, Bernd Bohnet, Gaurav Mishra, Hanie Sedghi, Igor Mordatch, Izzeddin Gur, Jaehoon Lee, John D. Co-Reyes, Jeffrey Pennington, Kelvin Xu, Kevin Swersky, Kshiteej Mahajan, Lechao Xiao, Rosanne Liu, Simon Kornblith, Noah Constant, Peter J. Liu, Roman Novak, Yundi Qian, Noah Fiedel, Jascha Sohl-Dickstein
Frontier Language Models are not Robust to Adversarial Arithmetic, or "What do I need to say so you agree 2+2=5? CoRR (2023)
Courtney Paquette, Elliot Paquette, Ben Adlam, Jeffrey Pennington
Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions. CoRR (2022)
Lechao Xiao, Jeffrey Pennington
Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression. CoRR (2022)
Lechao Xiao, Jeffrey Pennington
Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithm. CoRR (2022)
Courtney Paquette, Elliot Paquette, Ben Adlam, Jeffrey Pennington
Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions. NeurIPS (2022)
Lechao Xiao, Hong Hu, Theodor Misiakiewicz, Yue Lu, Jeffrey Pennington
Precise Learning Curves and Higher-Order Scalings for Dot-product Kernel Regression. NeurIPS (2022)
Jiri Hron, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein
Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling. ICML (2022)
Ben Adlam, Jake A. Levinson, Jeffrey Pennington
A Random Matrix Perspective on Mixtures of Nonlinearities in High Dimensions. AISTATS (2022)
Gabriel Mel, Jeffrey Pennington
Anisotropic Random Feature Regression in High Dimensions. ICLR (2022)
Atish Agarwala, Fabian Pedregosa, Jeffrey Pennington
Second-order regression models exhibit progressive sharpening to the edge of stability. CoRR (2022)
Jiri Hron, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein
Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling. CoRR (2022)
Lechao Xiao, Jeffrey Pennington
Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithm. ICML (2022)
Jeffrey Pennington, Rose Hartman, Ashwini Davison, Ali Shokoufandeh, Joy Payton, Daniel Chen, André Dietrich
Online education for data science: Opportunities and challenges. AMIA (2022)
Nilesh Tripuraneni, Ben Adlam, Jeffrey Pennington
Overparameterization Improves Robustness to Covariate Shift in High Dimensions. NeurIPS (2021)
Nilesh Tripuraneni, Ben Adlam, Jeffrey Pennington
Covariate Shift in High-Dimensional Random Feature Regression. CoRR (2021)
Ben Adlam, Jaehoon Lee, Lechao Xiao, Jeffrey Pennington, Jasper Snoek
Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit. ICLR (2021)
Ben Adlam, Jeffrey Pennington
Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition. CoRR (2020)
Ben Adlam, Jeffrey Pennington
The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization. CoRR (2020)
Jaehoon Lee, Samuel S. Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, Jascha Sohl-Dickstein
Finite Versus Infinite Neural Networks: an Empirical Study. NeurIPS (2020)
Wei Hu, Lechao Xiao, Jeffrey Pennington
Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks. CoRR (2020)
Atish Agarwala, Jeffrey Pennington, Yann N. Dauphin, Samuel S. Schoenholz
Temperature check: theory and practice for training models with softmax-cross-entropy losses. CoRR (2020)
Ben Adlam, Jeffrey Pennington
The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization. ICML (2020)
Lechao Xiao, Jeffrey Pennington, Samuel Schoenholz
Disentangling Trainability and Generalization in Deep Neural Networks. ICML (2020)
Wei Hu, Lechao Xiao, Ben Adlam, Jeffrey Pennington
The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks. NeurIPS (2020)
Ben Adlam, Jaehoon Lee, Lechao Xiao, Jeffrey Pennington, Jasper Snoek
Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit. CoRR (2020)
Wei Hu, Lechao Xiao, Ben Adlam, Jeffrey Pennington
The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks. CoRR (2020)
Ben Adlam, Jeffrey Pennington
Understanding Double Descent Requires A Fine-Grained Bias-Variance Decomposition. NeurIPS (2020)
Wei Hu, Lechao Xiao, Jeffrey Pennington
Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks. ICLR (2020)
Jaehoon Lee, Samuel S. Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, Jascha Sohl-Dickstein
Finite Versus Infinite Neural Networks: an Empirical Study. CoRR (2020)
Jiri Hron, Yasaman Bahri, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein
Exact posterior distributions of wide Bayesian neural networks. CoRR (2020)
Roman Novak, Lechao Xiao, Yasaman Bahri, Jaehoon Lee, Greg Yang, Jiri Hron, Daniel A. Abolafia, Jeffrey Pennington, Jascha Sohl-Dickstein
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes. ICLR (Poster) (2019)
Lechao Xiao, Jeffrey Pennington, Samuel S. Schoenholz
Disentangling trainability and generalization in deep learning. CoRR (2019)
Ben Adlam, Jake Levinson, Jeffrey Pennington
A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learning. CoRR (2019)
Krzysztof Choromanski, Aldo Pacchiano, Jeffrey Pennington, Yunhao Tang
KAMA-NNs: Low-dimensional Rotation Based Neural Networks. AISTATS (2019)
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Jascha Sohl-Dickstein, Jeffrey Pennington
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent. CoRR (2019)
Dar Gilboa, Bo Chang, Minmin Chen, Greg Yang, Samuel S. Schoenholz, Ed H. Chi, Jeffrey Pennington
Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs. CoRR (2019)
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, Samuel S. Schoenholz
A Mean Field Theory of Batch Normalization. ICLR (Poster) (2019)
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, Jeffrey Pennington
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent. NeurIPS (2019)
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, Samuel S. Schoenholz
A Mean Field Theory of Batch Normalization. CoRR (2019)
Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, Jascha Sohl-Dickstein
Sensitivity and Generalization in Neural Networks: an Empirical Study. ICLR (Poster) (2018)
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S. Schoenholz, Jeffrey Pennington
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10, 000-Layer Vanilla Convolutional Neural Networks. CoRR (2018)
Minmin Chen, Jeffrey Pennington, Samuel S. Schoenholz
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks. ICML (2018)
Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli
The emergence of spectral universality in deep networks. AISTATS (2018)
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein
Deep Neural Networks as Gaussian Processes. ICLR (Poster) (2018)
Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli
The Emergence of Spectral Universality in Deep Networks. CoRR (2018)
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S. Schoenholz, Jeffrey Pennington
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10, 000-Layer Vanilla Convolutional Neural Networks. ICML (2018)
Minmin Chen, Jeffrey Pennington, Samuel S. Schoenholz
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks. CoRR (2018)
Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, Jascha Sohl-Dickstein
Bayesian Convolutional Neural Networks with Many Channels are Gaussian Processes. CoRR (2018)
Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, Jascha Sohl-Dickstein
Sensitivity and Generalization in Neural Networks: an Empirical Study. CoRR (2018)
Jeffrey Pennington, Pratik Worah
The Spectrum of the Fisher Information Matrix of a Single-Hidden-Layer Neural Network. NeurIPS (2018)
Jeffrey Pennington, Yasaman Bahri
Geometry of Neural Network Loss Surfaces via Random Matrix Theory. ICML (2017)
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein
Deep Neural Networks as Gaussian Processes. CoRR (2017)
Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. CoRR (2017)
Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. NIPS (2017)
Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein
A Correspondence Between Random Neural Networks and Statistical Field Theory. CoRR (2017)
Jeffrey Pennington, Pratik Worah
Nonlinear random matrix theory for deep learning. NIPS (2017)
Kevin Murphy, Jeffrey Pennington, Aaron N. Browne, Byron Ruth, Ritu Khare, LeMar Davidson, Patricia K. Morris, Levon H. Utidjian, Charles Bailey
Clinical Data Research Network Lessons Learned. CRI (2016)
Jeffrey Pennington, Felix X. Yu, Sanjiv Kumar
Spherical Random Features for Polynomial Kernels. NIPS (2015)
Jeffrey Pennington, Richard Socher, Christopher D. Manning
Glove: Global Vectors for Word Representation. EMNLP (2014)
Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, Christopher D. Manning
Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. NIPS (2011)
Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, Christopher D. Manning
Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions. EMNLP (2011)