Sashank J. Reddi

Publication Activity (10 Years)

Years Active: 2013-2024
Publications (10 Years): 73

Top Topics

High Dimensions

Stochastic Optimization

Heterogeneous Distributed

Top Venues

Publications

Abhishek Panigrahi, Nikunj Saunshi, Kaifeng Lyu, Sobhan Miryoosefi, Sashank J. Reddi, Satyen Kale, Sanjiv Kumar
Efficient Stagewise Pretraining via Progressive Subnetworks. CoRR (2024)
Sashank J. Reddi, Sobhan Miryoosefi, Stefani Karp, Shankar Krishnan, Satyen Kale, Seungyeon Kim, Sanjiv Kumar
Efficient Training of Language Models using Few-Shot Learning. ICML (2023)
Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J. Reddi, Ke Ye, Felix Chern, Felix X. Yu, Ruiqi Guo, Sanjiv Kumar
The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers. ICLR (2023)
Samy Jelassi, Boris Hanin, Ziwei Ji, Sashank J. Reddi, Srinadh Bhojanapalli, Sanjiv Kumar
Depth Dependence of μP Learning Rates in ReLU MLPs. CoRR (2023)
Tian Li, Manzil Zaheer, Ken Liu, Sashank J. Reddi, Hugh Brendan McMahan, Virginia Smith
Differentially Private Adaptive Optimization with Delayed Preconditioners. ICLR (2023)
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Seungyeon Kim, Sashank J. Reddi, Sanjiv Kumar
In defense of dual-encoders for neural ranking. ICML (2022)
Tian Li, Manzil Zaheer, Sashank J. Reddi, Virginia Smith
Private Adaptive Optimization with Side Information. CoRR (2022)
Tian Li, Manzil Zaheer, Ken Ziyu Liu, Sashank J. Reddi, H. Brendan McMahan, Virginia Smith
Differentially Private Adaptive Optimization with Delayed Preconditioners. CoRR (2022)
Zhiyuan Li, Srinadh Bhojanapalli, Manzil Zaheer, Sashank J. Reddi, Sanjiv Kumar
Robust Training of Neural Networks using Scale Invariant Architectures. CoRR (2022)
Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J. Reddi, Ke Ye, Felix X. Chern, Felix X. Yu, Ruiqi Guo, Sanjiv Kumar
Large Models are Parsimonious Learners: Activation Sparsity in Trained Transformers. CoRR (2022)
Jianyu Wang, Hang Qi, Ankit Singh Rawat, Sashank J. Reddi, Sagar Waghmare, Felix X. Yu, Gauri Joshi
FedLite: A Scalable Approach for Federated Learning on Resource-constrained Clients. CoRR (2022)
Zhiyuan Li, Srinadh Bhojanapalli, Manzil Zaheer, Sashank J. Reddi, Sanjiv Kumar
Robust Training of Neural Networks Using Scale Invariant Architectures. ICML (2022)
Tian Li, Manzil Zaheer, Sashank J. Reddi, Virginia Smith
Private Adaptive Optimization with Side information. ICML (2022)
Han Nguyen, Hai Pham, Sashank J. Reddi, Barnabás Póczos
On the Algorithmic Stability and Generalization of Adaptive Optimization Methods. CoRR (2022)
Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H. Brendan McMahan, Blaise Agüera y Arcas, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, Suhas N. Diggavi, Hubert Eichner, Advait Gadhikar, Zachary Garrett, Antonious M. Girgis, Filip Hanzely, Andrew Hard, Chaoyang He, Samuel Horvath, Zhouyuan Huo, Alex Ingerman, Martin Jaggi, Tara Javidi, Peter Kairouz, Satyen Kale, Sai Praneeth Karimireddy, Jakub Konecný, Sanmi Koyejo, Tian Li, Luyang Liu, Mehryar Mohri, Hang Qi, Sashank J. Reddi, Peter Richtárik, Karan Singhal, Virginia Smith, Mahdi Soltanolkotabi, Weikang Song, Ananda Theertha Suresh, Sebastian U. Stich, Ameet Talwalkar, Hongyi Wang, Blake E. Woodworth, Shanshan Wu, Felix X. Yu, Honglin Yuan, Manzil Zaheer, Mi Zhang, Tong Zhang, Chunxiang Zheng, Chen Zhu, Wennan Zhu
A Field Guide to Federated Optimization. CoRR (2021)
Sashank J. Reddi, Rama Kumar Pasumarthi, Aditya Krishna Menon, Ankit Singh Rawat, Felix X. Yu, Seungyeon Kim, Andreas Veit, Sanjiv Kumar
RankDistil: Knowledge Distillation for Ranking. AISTATS (2021)
Honglin Yuan, Manzil Zaheer, Sashank J. Reddi
Federated Composite Optimization. ICML (2021)
Ankit Singh Rawat, Aditya Krishna Menon, Wittawat Jitkrittum, Sadeep Jayasumana, Felix X. Yu, Sashank J. Reddi, Sanjiv Kumar
Disentangling Sampling and Labeling Bias for Learning in Large-output Spaces. ICML (2021)
Ankit Singh Rawat, Aditya Krishna Menon, Wittawat Jitkrittum, Sadeep Jayasumana, Felix X. Yu, Sashank J. Reddi, Sanjiv Kumar
Disentangling Sampling and Labeling Bias for Learning in Large-Output Spaces. CoRR (2021)
Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konecný, Sanjiv Kumar, Hugh Brendan McMahan
Adaptive Federated Optimization. ICLR (2021)
Andrew Cotter, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, Sashank J. Reddi, Yichen Zhou
Distilling Double Descent. CoRR (2021)
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, Seungyeon Kim, Sanjiv Kumar
A statistical perspective on distillation. ICML (2021)
Erik Lindgren, Sashank J. Reddi, Ruiqi Guo, Sanjiv Kumar
Efficient Training of Retrieval Models using Negative Cache. NeurIPS (2021)
Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, Ananda Theertha Suresh
Breaking the centralized barrier for cross-device federated learning. NeurIPS (2021)
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank J. Reddi, Sanjiv Kumar, Suvrit Sra
Why are Adaptive Methods Good for Attention Models? NeurIPS (2020)
Srinadh Bhojanapalli, Chulhee Yun, Ankit Singh Rawat, Sashank J. Reddi, Sanjiv Kumar
Low-Rank Bottleneck in Multi-head Attention Models. CoRR (2020)
Ankit Singh Rawat, Aditya Krishna Menon, Andreas Veit, Felix X. Yu, Sashank J. Reddi, Sanjiv Kumar
Doubly-stochastic mining for heterogeneous retrieval. CoRR (2020)
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, Seungyeon Kim, Sanjiv Kumar
Why distillation helps: a statistical perspective. CoRR (2020)
Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, Ananda Theertha Suresh
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning. CoRR (2020)
Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konecný, Sanjiv Kumar, H. Brendan McMahan
Adaptive Federated Optimization. CoRR (2020)
Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi, Sanjiv Kumar
Are Transformers universal approximators of sequence-to-sequence functions? ICLR (2020)
Srinadh Bhojanapalli, Chulhee Yun, Ankit Singh Rawat, Sashank J. Reddi, Sanjiv Kumar
Low-Rank Bottleneck in Multi-head Attention Models. ICML (2020)
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, Ananda Theertha Suresh
SCAFFOLD: Stochastic Controlled Averaging for Federated Learning. ICML (2020)
Chulhee Yun, Yin-Wen Chang, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi, Sanjiv Kumar
$O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers. CoRR (2020)
Ilqar Ramazanli, Han Nguyen, Hai Pham, Sashank J. Reddi, Barnabás Póczos
Adaptive Sampling Distributed Stochastic Variance Reduced Gradient for Heterogeneous Distributed Datasets. CoRR (2020)
Chulhee Yun, Yin-Wen Chang, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi, Sanjiv Kumar
O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformers. NeurIPS (2020)
Honglin Yuan, Manzil Zaheer, Sashank J. Reddi
Federated Composite Optimization. CoRR (2020)
Yangjun Ruan, Yuanhao Xiong, Sashank J. Reddi, Sanjiv Kumar, Cho-Jui Hsieh
Learning to Learn by Zeroth-Order Oracle. ICLR (2020)
Yang You, Jing Li, Sashank J. Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, Cho-Jui Hsieh
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes. ICLR (2020)
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, Sanjiv Kumar
Can gradient clipping mitigate label noise? ICLR (2020)
Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi, Sanjiv Kumar
Are Transformers universal approximators of sequence-to-sequence functions? CoRR (2019)
Sashank J. Reddi, Satyen Kale, Sanjiv Kumar
On the Convergence of Adam and Beyond. CoRR (2019)
Matthew Staib, Sashank J. Reddi, Satyen Kale, Sanjiv Kumar, Suvrit Sra
Escaping Saddle Points with Adaptive Gradient Methods. CoRR (2019)
Venkatadheeraj Pichapati, Ananda Theertha Suresh, Felix X. Yu, Sashank J. Reddi, Sanjiv Kumar
AdaCliP: Adaptive Clipping for Private SGD. CoRR (2019)
Sashank J. Reddi, Satyen Kale, Felix X. Yu, Daniel Niels Holtmann-Rice, Jiecao Chen, Sanjiv Kumar
Stochastic Negative Mining for Learning with Large Output Spaces. AISTATS (2019)
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank J. Reddi, Sanjiv Kumar, Suvrit Sra
Why ADAM Beats SGD for Attention Models. CoRR (2019)
Yangjun Ruan, Yuanhao Xiong, Sashank J. Reddi, Sanjiv Kumar, Cho-Jui Hsieh
Learning to Learn by Zeroth-Order Oracle. CoRR (2019)
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, Ananda Theertha Suresh
SCAFFOLD: Stochastic Controlled Averaging for On-Device Federated Learning. CoRR (2019)
Chuan Guo, Ali Mousavi, Xiang Wu, Daniel Niels Holtmann-Rice, Satyen Kale, Sashank J. Reddi, Sanjiv Kumar
Breaking the Glass Ceiling for Embedding-Based Classifiers for Large Output Spaces. NeurIPS (2019)
Matthew Staib, Sashank J. Reddi, Satyen Kale, Sanjiv Kumar, Suvrit Sra
Escaping Saddle Points with Adaptive Gradient Methods. ICML (2019)
Sashank J. Reddi, Manzil Zaheer, Suvrit Sra, Barnabás Póczos, Francis R. Bach, Ruslan Salakhutdinov, Alexander J. Smola
A Generic Approach for Escaping Saddle points. AISTATS (2018)
Sashank J. Reddi, Satyen Kale, Felix X. Yu, Daniel N. Holtmann-Rice, Jiecao Chen, Sanjiv Kumar
Stochastic Negative Mining for Learning with Large Output Spaces. CoRR (2018)
Manzil Zaheer, Sashank J. Reddi, Devendra Singh Sachan, Satyen Kale, Sanjiv Kumar
Adaptive Methods for Nonconvex Optimization. NeurIPS (2018)
Sashank J. Reddi, Satyen Kale, Sanjiv Kumar
On the Convergence of Adam and Beyond. ICLR (2018)
Sashank J. Reddi, Manzil Zaheer, Suvrit Sra, Barnabás Póczos, Francis R. Bach, Ruslan Salakhutdinov, Alexander J. Smola
A Generic Approach for Escaping Saddle points. CoRR (2017)
Sashank J. Reddi, Ahmed Hefny, Suvrit Sra, Barnabás Póczos, Alexander J. Smola
Stochastic Variance Reduction for Nonconvex Optimization. CoRR (2016)
Hongyi Zhang, Sashank J. Reddi, Suvrit Sra
Fast stochastic optimization on Riemannian manifolds. CoRR (2016)
Sashank J. Reddi, Suvrit Sra, Barnabás Póczos, Alexander J. Smola
Stochastic Frank-Wolfe Methods for Nonconvex Optimization. CoRR (2016)
Sashank J. Reddi, Suvrit Sra, Barnabás Póczos, Alexander J. Smola
Proximal Stochastic Methods for Nonsmooth Nonconvex Finite-Sum Optimization. NIPS (2016)
Sashank J. Reddi, Suvrit Sra, Barnabás Póczos, Alexander J. Smola
Stochastic Frank-Wolfe methods for nonconvex optimization. Allerton (2016)
Kumar Avinava Dubey, Sashank J. Reddi, Sinead A. Williamson, Barnabás Póczos, Alexander J. Smola, Eric P. Xing
Variance Reduction in Stochastic Gradient Langevin Dynamics. NIPS (2016)
Sashank J. Reddi, Suvrit Sra, Barnabás Póczos, Alexander J. Smola
Fast incremental method for smooth nonconvex optimization. CDC (2016)
Hongyi Zhang, Sashank J. Reddi, Suvrit Sra
Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds. NIPS (2016)
Sashank J. Reddi, Ahmed Hefny, Suvrit Sra, Barnabás Póczos, Alexander J. Smola
Stochastic Variance Reduction for Nonconvex Optimization. ICML (2016)
Sashank J. Reddi, Suvrit Sra, Barnabás Póczos, Alexander J. Smola
Fast Incremental Method for Nonconvex Optimization. CoRR (2016)
Sashank J. Reddi, Jakub Konecný, Peter Richtárik, Barnabás Póczos, Alexander J. Smola
AIDE: Fast and Communication Efficient Distributed Optimization. CoRR (2016)
Sashank J. Reddi, Suvrit Sra, Barnabás Póczos, Alexander J. Smola
Fast Stochastic Methods for Nonsmooth Nonconvex Optimization. CoRR (2016)
Sashank J. Reddi, Barnabás Póczos, Alexander J. Smola
Communication Efficient Coresets for Empirical Loss Minimization. UAI (2015)
Sashank J. Reddi, Ahmed Hefny, Suvrit Sra, Barnabás Póczos, Alexander J. Smola
On Variance Reduction in Stochastic Gradient Descent and its Asynchronous Variants. CoRR (2015)
Aaditya Ramdas, Sashank J. Reddi, Barnabás Póczos, Aarti Singh, Larry A. Wasserman
Adaptivity and Computation-Statistics Tradeoffs for Kernel and Distance based High Dimensional Two Sample Testing. CoRR (2015)
Sashank J. Reddi, Ahmed Hefny, Suvrit Sra, Barnabás Póczos, Alexander J. Smola
On Variance Reduction in Stochastic Gradient Descent and its Asynchronous Variants. NIPS (2015)
Sashank J. Reddi, Aaditya Ramdas, Barnabás Póczos, Aarti Singh, Larry A. Wasserman
On the High Dimensional Power of a Linear-Time Two Sample Test under Mean-shift Alternatives. AISTATS (2015)
Sashank J. Reddi, Ahmed Hefny, Carlton Downey, Avinava Dubey, Suvrit Sra
Large-scale randomized-coordinate descent methods with non-separable linear constraints. UAI (2015)
Sashank J. Reddi, Aaditya Ramdas, Barnabás Póczos, Aarti Singh, Larry A. Wasserman
Kernel MMD, the Median Heuristic and Distance Correlation in High Dimensions. CoRR (2014)
Sashank J. Reddi, Barnabás Póczos
k-NN Regression on Functional Data with Incomplete Observations. UAI (2014)
Aaditya Ramdas, Sashank J. Reddi, Barnabás Póczos, Aarti Singh, Larry A. Wasserman
On the High-dimensional Power of Linear-time Kernel Two-Sample Testing under Mean-difference Alternatives. CoRR (2014)
Sashank J. Reddi, Barnabás Póczos
Scale Invariant Conditional Dependence Measures. ICML (3) (2013)