Mohammad Shoeybi

Publication Activity (10 Years)

Years Active: 2006-2024
Publications (10 Years): 62

Top Topics

Text Generation

Technical Report

Question Answering

Top Venues

Publications

Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping
NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models. CoRR (2024)
Jupinder Parmar, Sanjeev Satheesh, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models. CoRR (2024)
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, Bryan Catanzaro
Retrieval meets Long Context Large Language Models. ICLR (2024)
Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, Bryan Catanzaro
RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs. CoRR (2024)
Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro
ChatQA: Building GPT-4 Level Conversational QA Models. CoRR (2024)
Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, Garvit Kulshreshtha, Vartika Singh, Jared Casper, Jan Kautz, Mohammad Shoeybi, Bryan Catanzaro
An Empirical Study of Mamba-based Language Models. CoRR (2024)
Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, Bryan Catanzaro
ODIN: Disentangled Reward Mitigates Hacking in RLHF. CoRR (2024)
Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Mostofa Patwary, Sandeep Subramanian, Dan Su, Chen Zhu, Deepak Narayanan, Aastha Jhunjhunwala, Ayush Dattagupta, Vibhu Jawa, Jiwei Liu, Ameya Mahabaleshwarkar, Osvald Nitski, Annika Brundyn, James Maki, Miguel Martinez, Jiaxuan You, John Kamalu, Patrick LeGresley, Denys Fridman, Jared Casper, Ashwath Aithal, Oleksii Kuchaiev, Mohammad Shoeybi, Jonathan M. Cohen, Bryan Catanzaro
Nemotron-4 15B Technical Report. CoRR (2024)
Peng Xu, Wei Ping, Xianchao Wu, Zihan Liu, Mohammad Shoeybi, Bryan Catanzaro
ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities. CoRR (2024)
Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan M. Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Aleksander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grzegorzek, Robert Hero, Jining Huang, Vibhu Jawa, Joseph Jennings, Aastha Jhunjhunwala, John Kamalu, Sadaf Khan, Oleksii Kuchaiev, Patrick LeGresley, Hui Li, Jiwei Liu, Zihan Liu, Eileen Long, Ameya Sunil Mahabaleshwarkar, Somshubra Majumdar, James Maki, Miguel Martinez, Maer Rodrigues de Melo, Ivan Moshkov, Deepak Narayanan, Sean Narenthiran, Jesus Navarro, Phong Nguyen, Osvald Nitski, Vahid Noroozi, Guruprasad Nutheti, Christopher Parisien, Jupinder Parmar, Mostofa Patwary, Krzysztof Pawelec, Wei Ping, Shrimai Prabhumoye, Rajarshi Roy, Trisha Saar, Vasanth Rao Naik Sabavat, Sanjeev Satheesh, Jane Polak Scowcroft, Jason Sewall, Pavel Shamis, Gerald Shen, Mohammad Shoeybi, Dave Sizer, Misha Smelyanskiy, Felipe Soares, Makesh Narsimhan Sreedhar, Dan Su, Sandeep Subramanian, Shengyang Sun, Shubham Toshniwal, Hao Wang, Zhilin Wang, Jiaxuan You, Jiaqi Zeng, Jimmy Zhang, Jing Zhang, Vivienne Zhang, Yian Zhang, Chen Zhu
Nemotron-4 340B Technical Report. CoRR (2024)
Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Bo Li, Aastha Jhunjhunwala, Zhilin Wang, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
Data, Data Everywhere: A Guide for Pretraining Dataset Construction. CoRR (2024)
Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, Pavlo Molchanov
Compact Language Models via Pruning and Knowledge Distillation. CoRR (2024)
Zhuolin Yang, Wei Ping, Zihan Liu, Vijay Korthikanti, Weili Nie, De-An Huang, Linxi Fan, Zhiding Yu, Shiyi Lan, Bo Li, Ming-Yu Liu, Yuke Zhu, Mohammad Shoeybi, Bryan Catanzaro, Chaowei Xiao, Anima Anandkumar
Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning. CoRR (2023)
Boxin Wang, Wei Ping, Lawrence McAfee, Peng Xu, Bo Li, Mohammad Shoeybi, Bryan Catanzaro
InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining. CoRR (2023)
Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei Xiao, Anima Anandkumar, Bryan Catanzaro
Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study. CoRR (2023)
Dan Su, Mostofa Patwary, Shrimai Prabhumoye, Peng Xu, Ryan Prenger, Mohammad Shoeybi, Pascale Fung, Anima Anandkumar, Bryan Catanzaro
Context Generation Improves Open Domain Question Answering. EACL (Findings) (2023)
Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei Xiao, Anima Anandkumar, Bryan Catanzaro
Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study. EMNLP (2023)
Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, Song Han
VILA: On Pre-training for Visual Language Models. CoRR (2023)
Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
Adding Instructions during Pretraining: Effective Way of Controlling Toxicity in Language Models. CoRR (2023)
Zhuolin Yang, Wei Ping, Zihan Liu, Vijay Korthikanti, Weili Nie, De-An Huang, Linxi Fan, Zhiding Yu, Shiyi Lan, Bo Li, Mohammad Shoeybi, Ming-Yu Liu, Yuke Zhu, Bryan Catanzaro, Chaowei Xiao, Anima Anandkumar
Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning. EMNLP (Findings) (2023)
Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, Bryan Catanzaro
Reducing Activation Recomputation in Large Transformer Models. MLSys (2023)
Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
Adding Instructions during Pretraining: Effective way of Controlling Toxicity in Language Models. EACL (2023)
Jie Huang, Wei Ping, Peng Xu, Mohammad Shoeybi, Kevin Chen-Chuan Chang, Bryan Catanzaro
RAVEN: In-Context Learning with Retrieval Augmented Encoder-Decoder Language Models. CoRR (2023)
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, Bryan Catanzaro
Retrieval meets Long Context Large Language Models. CoRR (2023)
Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, Naveen Mellempudi, Stuart F. Oberman, Mohammad Shoeybi, Michael Y. Siu, Hao Wu
FP8 Formats for Deep Learning. CoRR (2022)
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zheng, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, Bryan Catanzaro
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model. CoRR (2022)
Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale Fung, Mohammad Shoeybi, Bryan Catanzaro
Factuality Enhanced Language Models for Open-Ended Text Generation. NeurIPS (2022)
Boxin Wang, Wei Ping, Chaowei Xiao, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Bo Li, Anima Anandkumar, Bryan Catanzaro
Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models. NeurIPS (2022)
Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, Bryan Catanzaro
Reducing Activation Recomputation in Large Transformer Models. CoRR (2022)
David Wingate, Mohammad Shoeybi, Taylor Sorensen
Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models. CoRR (2022)
Boxin Wang, Wei Ping, Chaowei Xiao, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Bo Li, Anima Anandkumar, Bryan Catanzaro
Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models. CoRR (2022)
David Wingate, Mohammad Shoeybi, Taylor Sorensen
Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models. EMNLP (Findings) (2022)
Zihan Liu, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye, Wei Ping, Mohammad Shoeybi, Bryan Catanzaro
Multi-Stage Prompting for Knowledgeable Dialogue Generation. CoRR (2022)
Peng Xu, Mostofa Patwary, Shrimai Prabhumoye, Virginia Adams, Ryan J. Prenger, Wei Ping, Nayeon Lee, Mohammad Shoeybi, Bryan Catanzaro
Evaluating Parameter Efficient Learning for Generation. CoRR (2022)
Peng Xu, Mostofa Patwary, Shrimai Prabhumoye, Virginia Adams, Ryan Prenger, Wei Ping, Nayeon Lee, Mohammad Shoeybi, Bryan Catanzaro
Evaluating Parameter Efficient Learning for Generation. EMNLP (2022)
Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
Factuality Enhanced Language Models for Open-Ended Text Generation. CoRR (2022)
Dan Su, Mostofa Patwary, Shrimai Prabhumoye, Peng Xu, Ryan Prenger, Mohammad Shoeybi, Pascale Fung, Anima Anandkumar, Bryan Catanzaro
Context Generation Improves Open Domain Question Answering. CoRR (2022)
Zihan Liu, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye, Wei Ping, Mohammad Shoeybi, Bryan Catanzaro
Multi-Stage Prompting for Knowledgeable Dialogue Generation. ACL (Findings) (2022)
Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, Bryan Catanzaro
Long-Short Transformer: Efficient Transformers for Language and Vision. CoRR (2021)
Shrimai Prabhumoye, Rafal Kocielnik, Mohammad Shoeybi, Anima Anandkumar, Bryan Catanzaro
Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases. CoRR (2021)
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, Matei Zaharia
Efficient Large-Scale Language Model Training on GPU Clusters. CoRR (2021)
Devendra Singh Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamilton, Bryan Catanzaro
End-to-End Training of Neural Retrievers for Open-Domain Question Answering. CoRR (2021)
Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, Bryan Catanzaro
Long-Short Transformer: Efficient Transformers for Language and Vision. NeurIPS (2021)
Devendra Singh Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamilton, Bryan Catanzaro
End-to-End Training of Neural Retrievers for Open-Domain Question Answering. ACL/IJCNLP (1) (2021)
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, Matei Zaharia
Efficient large-scale language model training on GPU clusters using megatron-LM. SC (2021)
Raul Puri, Ryan Spring, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
Training Question Answering Models From Synthetic Data. CoRR (2020)
Raul Puri, Ryan Spring, Mohammad Shoeybi, Mostofa Patwary, Bryan Catanzaro
Training Question Answering Models From Synthetic Data. EMNLP (1) (2020)
Alex Boyd, Raul Puri, Mohammad Shoeybi, Mostofa Patwary, Bryan Catanzaro
Large Scale Multi-Actor Generative Dialog Modeling. CoRR (2020)
Hoo-Chang Shin, Yang Zhang, Evelina Bakhturina, Raul Puri, Mostofa Patwary, Mohammad Shoeybi, Raghav Mani
BioMegatron: Larger Biomedical Domain Language Model. CoRR (2020)
Sashank Santhanam, Wei Ping, Raul Puri, Mohammad Shoeybi, Mostofa Patwary, Bryan Catanzaro
Local Knowledge Powered Conversational Agents. CoRR (2020)
Hoo-Chang Shin, Yang Zhang, Evelina Bakhturina, Raul Puri, Mostofa Patwary, Mohammad Shoeybi, Raghav Mani
BioMegatron: Larger Biomedical Domain Language Model. EMNLP (1) (2020)
Alex Boyd, Raul Puri, Mohammad Shoeybi, Mostofa Patwary, Bryan Catanzaro
Large Scale Multi-Actor Generative Dialog Modeling. ACL (2020)
Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar, Bryan Catanzaro
MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models. CoRR (2020)
Kuo-Hao Zeng, Mohammad Shoeybi, Ming-Yu Liu
Style Example-Guided Text Generation using Generative Adversarial Transformers. CoRR (2020)
Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar, Bryan Catanzaro
MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models. EMNLP (1) (2020)
Fitsum A. Reda, Deqing Sun, Aysegul Dundar, Mohammad Shoeybi, Guilin Liu, Kevin J. Shih, Andrew Tao, Jan Kautz, Bryan Catanzaro
Unsupervised Video Interpolation Using Cycle Consistency. CoRR (2019)
Fitsum A. Reda, Deqing Sun, Aysegul Dundar, Mohammad Shoeybi, Guilin Liu, Kevin J. Shih, Andrew Tao, Jan Kautz, Bryan Catanzaro
Unsupervised Video Interpolation Using Cycle Consistency. ICCV (2019)
Rafael Valle, Fitsum A. Reda, Mohammad Shoeybi, Patrick LeGresley, Andrew Tao, Bryan Catanzaro
Neural ODEs for Image Segmentation with Level Sets. CoRR (2019)
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, Bryan Catanzaro
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. CoRR (2019)
Sercan Ömer Arik, Mike Chrzanowski, Adam Coates, Gregory Frederick Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Andrew Y. Ng, Jonathan Raiman, Shubho Sengupta, Mohammad Shoeybi
Deep Voice: Real-time Neural Text-to-Speech. ICML (2017)
Markus Kliegl, Siddharth Goyal, Kexin Zhao, Kavya Srinet, Mohammad Shoeybi
Trace norm regularization and faster inference for embedded speech recognition RNNs. CoRR (2017)
Sercan Ömer Arik, Mike Chrzanowski, Adam Coates, Greg Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Jonathan Raiman, Shubho Sengupta, Mohammad Shoeybi
Deep Voice: Real-time Neural Text-to-Speech. CoRR (2017)
Mohammad Shoeybi, Magnus Svärd, Frank E. Ham, Parviz Moin
An adaptive implicit-explicit scheme for the DNS and LES of compressible flows on unstructured grids. J. Comput. Phys. 229 (17) (2010)
Ken Mattsson, Magnus Svärd, Mohammad Shoeybi
Stable and accurate schemes for the compressible Navier-Stokes equations. J. Comput. Phys. 227 (4) (2008)
Jeremy A. Templeton, Mohammad Shoeybi
Towards Wall-Normal Filtering for Large-Eddy Simulation. Multiscale Model. Simul. 5 (2) (2006)