Samyam Rajbhandari

Publication Activity (10 Years)

Years Active: 2012-2024
Publications (10 Years): 40

Top Topics

Top Venues

USENIX Annual Technical Conference

Publications

Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Xiaoxia Wu, Connor Holmes, Zhewei Yao, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, Yuxiong He
ZeRO++: Extremely Efficient Collective Communication for Large Model Training. ICLR (2024)
Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, Yuxiong He
DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference. CoRR (2024)
Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Reza Yazdani Aminabadi, Shuaiwen Leon Song, Samyam Rajbhandari, Yuxiong He
System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. IPDPS (Workshops) (2024)
Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Reza Yazdani Aminadabi, Shuaiwen Leon Song, Samyam Rajbhandari, Yuxiong He
System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. PODC (2024)
Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He, Abhinav Bhatele
A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training. ICS (2023)
Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Connor Holmes, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, Yuxiong He
ZeRO++: Extremely Efficient Collective Communication for Giant Model Training. CoRR (2023)
Zhewei Yao, Xiaoxia Wu, Conglong Li, Minjia Zhang, Heyang Qin, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He
DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attention. CoRR (2023)
Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, Zhongzhu Zhou, Michael Wyatt, Molly Smith, Lev Kurilenko, Heyang Qin, Masahiro Tanaka, Shuai Che, Shuaiwen Leon Song, Yuxiong He
DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales. CoRR (2023)
Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He, Abhinav Bhatele
A Novel Tensor-Expert Hybrid Parallelism Approach to Scale Mixture-of-Experts Training. CoRR (2023)
Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, Yuxiong He
DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. CoRR (2023)
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zheng, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, Bryan Catanzaro
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model. CoRR (2022)
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, Yuxiong He
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale. ICML (2022)
Conglong Li, Ammar Ahmad Awan, Hanlin Tang, Samyam Rajbhandari, Yuxiong He
1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed. HIPC (2022)
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, Yuxiong He
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale. CoRR (2022)
Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, Yuxiong He
DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale. SC (2022)
Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, Yuxiong He
DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale. CoRR (2022)
Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He
ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning. CoRR (2021)
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, Yuxiong He
ZeRO-Offload: Democratizing Billion-Scale Model Training. USENIX Annual Technical Conference (2021)
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, Yuxiong He
ZeRO-Offload: Democratizing Billion-Scale Model Training. CoRR (2021)
Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, Yuxiong He
1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed. ICML (2021)
Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andrés Felipe Cruz-Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, Hany Hassan Awadalla
Scalable and Efficient MoE Training for Multitask Multilingual Models. CoRR (2021)
Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, Yuxiong He
1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed. CoRR (2021)
Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He
ZeRO-infinity: breaking the GPU memory wall for extreme scale deep learning. SC (2021)
Conglong Li, Ammar Ahmad Awan, Hanlin Tang, Samyam Rajbhandari, Yuxiong He
1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed. CoRR (2021)
Heyang Qin, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, Yuxiong He
SimiGrad: Fine-Grained Adaptive Batching for Large Scale Training using Gradient Similarity Measurement. NeurIPS (2021)
Yang You, Yuxiong He, Samyam Rajbhandari, Wenhan Wang, Cho-Jui Hsieh, Kurt Keutzer, James Demmel
Fast LSTM by dynamic decomposition on cloud and distributed systems. Knowl. Inf. Syst. 62 (11) (2020)
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He
ZeRO: memory optimizations toward training trillion parameter models. SC (2020)
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, Yuxiong He
DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters. KDD (2020)
Hanlin Tang, Shaoduo Gan, Samyam Rajbhandari, Xiangru Lian, Ji Liu, Yuxiong He, Ce Zhang
APMSqueeze: A Communication Efficient Adam-Preconditioned Momentum SGD Algorithm. CoRR (2020)
Yang You, Yuxiong He, Samyam Rajbhandari, Wenhan Wang, Cho-Jui Hsieh, Kurt Keutzer, James Demmel
Fast LSTM Inference by Dynamic Decomposition on Cloud Systems. ICDM (2019)
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He
ZeRO: Memory Optimization Towards Training A Trillion Parameter Models. CoRR (2019)
Minjia Zhang, Samyam Rajbhandari, Wenhan Wang, Elton Zheng, Olatunji Ruwase, Jeff Rasley, Jason Li, Junhua Wang, Yuxiong He
Accelerating Large Scale Deep Learning Inference through DeepCPU at Microsoft. OpML (2019)
Samyam Rajbhandari, Harsh Shrivastava, Yuxiong He
AntMan: Sparse Low-Rank Compression to Accelerate RNN inference. CoRR (2019)
Wei Wen, Yuxiong He, Samyam Rajbhandari, Minjia Zhang, Wenhan Wang, Fang Liu, Bin Hu, Yiran Chen, Hai Li
Learning Intrinsic Sparse Structures within Long Short-Term Memory. ICLR (Poster) (2018)
Minjia Zhang, Samyam Rajbhandari, Wenhan Wang, Yuxiong He
DeepCPU: Serving RNN-based Deep Learning Models 10x Faster. USENIX Annual Technical Conference (2018)
Samyam Rajbhandari, Fabrice Rastello, Karol Kowalski, Sriram Krishnamoorthy, P. Sadayappan
Optimizing the Four-Index Integral Transform Using Data Movement Lower Bounds Analysis. PPOPP (2017)
Wei Wen, Yuxiong He, Samyam Rajbhandari, Wenhan Wang, Fang Liu, Bin Hu, Yiran Chen, Hai Li
Learning Intrinsic Sparse Structures within Long Short-term Memory. CoRR (2017)
Samyam Rajbhandari, Yuxiong He, Olatunji Ruwase, Michael Carbin, Trishul M. Chilimbi
Optimizing CNNs on Multicores for Scalability, Performance and Goodput. ASPLOS (2017)
Samyam Rajbhandari, Jinsung Kim, Sriram Krishnamoorthy, Louis-Noël Pouchet, Fabrice Rastello, Robert J. Harrison, P. Sadayappan
A domain-specific compiler for a parallel multiresolution adaptive numerical simulation environment. SC (2016)
Samyam Rajbhandari, Jinsung Kim, Sriram Krishnamoorthy, Louis-Noël Pouchet, Fabrice Rastello, Robert J. Harrison, P. Sadayappan
On fusing recursive traversals of K-d trees. CC (2016)
Samyam Rajbhandari, Akshay Nikam, Pai-Wei Lai, Kevin Stock, Sriram Krishnamoorthy, P. Sadayappan
CAST: Contraction Algorithm for Symmetric Tensors. ICPP (2014)
Samyam Rajbhandari, Akshay Nikam, Pai-Wei Lai, Kevin Stock, Sriram Krishnamoorthy, P. Sadayappan
A Communication-Optimal Framework for Contracting Distributed Tensors. SC (2014)
Pai-Wei Lai, Kevin Stock, Samyam Rajbhandari, Sriram Krishnamoorthy, P. Sadayappan
A framework for load balancing of tensor contraction expressions via dynamic task partitioning. SC (2013)
Pai-Wei Lai, Huaijian Zhang, Samyam Rajbhandari, Edward F. Valeev, Karol Kowalski, P. Sadayappan
International Conference on Computational Science, ICCS 2012. ICCS (2012)