Noam Shazeer

Publication Activity (10 Years)

Years Active: 2010-2023
Publications (10 Years): 55

Top Topics

Designing Effective

Weakly Supervised

Sequence Prediction

Top Venues

J. Mach. Learn. Res.

Publications

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, Noah Fiedel
PaLM: Scaling Language Modeling with Pathways. J. Mach. Learn. Res. 24 (2023)
William Fedus, Barret Zoph, Noam Shazeer
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. J. Mach. Learn. Res. 23 (2022)
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, Andrea Gesmundo
Scaling Up Models and Data with t5x and seqio. CoRR (2022)
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, Quoc Le
LaMDA: Language Models for Dialog Applications. CoRR (2022)
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, William Fedus
Designing Effective Sparse Expert Models. CoRR (2022)
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, Noah Fiedel
PaLM: Scaling Language Modeling with Pathways. CoRR (2022)
William Fedus, Barret Zoph, Noam Shazeer
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. CoRR (2021)
David R. So, Wojciech Manke, Hanxiao Liu, Zihang Dai, Noam Shazeer, Quoc V. Le
Primer: Searching for Efficient Transformers for Language Modeling. CoRR (2021)
Sharan Narang, Hyung Won Chung, Yi Tay, Liam Fedus, Thibault Févry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, Yanqi Zhou, Wei Li, Nan Ding, Jake Marcus, Adam Roberts, Colin Raffel
Do Transformer Modifications Transfer Across Implementations and Applications? EMNLP (1) (2021)
Sharan Narang, Hyung Won Chung, Yi Tay, William Fedus, Thibault Févry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, Yanqi Zhou, Wei Li, Nan Ding, Jake Marcus, Adam Roberts, Colin Raffel
Do Transformer Modifications Transfer Across Implementations and Applications? CoRR (2021)
Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake A. Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, Zhifeng Chen
GSPMD: General and Scalable Parallelization for ML Computation Graphs. CoRR (2021)
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, Zhifeng Chen
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. ICLR (2021)
David R. So, Wojciech Manke, Hanxiao Liu, Zihang Dai, Noam Shazeer, Quoc V. Le
Searching for Efficient Transformers for Language Modeling. NeurIPS (2021)
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res. 21 (2020)
Ciprian Chelba, Mia Xu Chen, Ankur Bapna, Noam Shazeer
Faster Transformer Decoding: N-gram Masked Self-Attention. CoRR (2020)
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, Zhifeng Chen
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. CoRR (2020)
Noam Shazeer
GLU Variants Improve Transformer. CoRR (2020)
Noam Shazeer, Zhenzhong Lan, Youlong Cheng, Nan Ding, Le Hou
Talking-Heads Attention. CoRR (2020)
Adam Roberts, Colin Raffel, Noam Shazeer
How Much Knowledge Can You Pack Into the Parameters of a Language Model? CoRR (2020)
Adam Roberts, Colin Raffel, Noam Shazeer
How Much Knowledge Can You Pack Into the Parameters of a Language Model? EMNLP (1) (2020)
Le Hou, Youlong Cheng, Noam Shazeer, Niki Parmar, Yeqing Li, Panagiotis Korfiatis, Travis M. Drucker, Daniel J. Blezek, Xiaodan Song
High Resolution Medical Image Analysis with Spatial Partitioning. CoRR (2019)
Jared Lichtarge, Chris Alberti, Shankar Kumar, Noam Shazeer, Niki Parmar, Simon Tong
Corpora Generation for Grammatical Error Correction. NAACL-HLT (1) (2019)
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. CoRR (2019)
Noam Shazeer
Fast Transformer Decoding: One Write-Head is All You Need. CoRR (2019)
Jared Lichtarge, Christopher Alberti, Shankar Kumar, Noam Shazeer, Niki Parmar, Simon Tong
Corpora Generation for Grammatical Error Correction. CoRR (2019)
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, Douglas Eck
Music Transformer: Generating Music with Long-Term Structure. ICLR (Poster) (2019)
Lukasz Kaiser, Aurko Roy, Ashish Vaswani, Niki Parmar, Samy Bengio, Jakob Uszkoreit, Noam Shazeer
Fast Decoding in Sequence Models using Discrete Latent Variables. CoRR (2018)
Ravi Teja Mullapudi, William R. Mark, Noam Shazeer, Kayvon Fatahalian
HydraNets: Specialized Dynamic Architectures for Efficient Inference. CVPR (2018)
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku
Image Transformer. CoRR (2018)
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Curtis Hawthorne, Andrew M. Dai, Matthew D. Hoffman, Douglas Eck
An Improved Relative Self-Attention Mechanism for Transformer with Application to Music Generation. CoRR (2018)
Mitchell Stern, Noam Shazeer, Jakob Uszkoreit
Blockwise Parallel Decoding for Deep Autoregressive Models. CoRR (2018)
Mitchell Stern, Noam Shazeer, Jakob Uszkoreit
Blockwise Parallel Decoding for Deep Autoregressive Models. NeurIPS (2018)
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, Blake A. Hechtman
Mesh-TensorFlow: Deep Learning for Supercomputers. CoRR (2018)
Noam Shazeer, Mitchell Stern
Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. CoRR (2018)
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, Noam Shazeer
Fast Decoding in Sequence Models Using Discrete Latent Variables. ICML (2018)
Jared Lichtarge, Christopher Alberti, Shankar Kumar, Noam Shazeer, Niki Parmar
Weakly Supervised Grammatical Error Correction using Iterative Decoding. CoRR (2018)
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, Noam Shazeer
Generating Wikipedia by Summarizing Long Sequences. CoRR (2018)
Ashish Vaswani, Samy Bengio, Eugene Brevdo, François Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, Jakob Uszkoreit
Tensor2Tensor for Neural Machine Translation. CoRR (2018)
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, Blake A. Hechtman
Mesh-TensorFlow: Deep Learning for Supercomputers. NeurIPS (2018)
Noam Shazeer, Mitchell Stern
Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. ICML (2018)
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, Dustin Tran
Image Transformer. ICML (2018)
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, Noam Shazeer
Generating Wikipedia by Summarizing Long Sequences. ICLR (Poster) (2018)
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George F. Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, Macduff Hughes
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation. ACL (1) (2018)
Ashish Vaswani, Samy Bengio, Eugene Brevdo, François Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, Jakob Uszkoreit
Tensor2Tensor for Neural Machine Translation. AMTA (1) (2018)
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, Jeff Dean
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR (Poster) (2017)
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, Jeff Dean
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. CoRR (2017)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
Attention is All you Need. NIPS (2017)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
Attention Is All You Need. CoRR (2017)
Lukasz Kaiser, Aidan N. Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, Jakob Uszkoreit
One Model To Learn Them All. CoRR (2017)
Babak Damavandi, Shankar Kumar, Noam Shazeer, Antoine Bruguier
NN-grams: Unifying neural network and n-gram language models for Speech Recognition. CoRR (2016)
Joris Pelemans, Noam Shazeer, Ciprian Chelba
Sparse Non-negative Matrix Language Modeling. Trans. Assoc. Comput. Linguistics 4 (2016)
Georg Heigold, Ignacio Moreno, Samy Bengio, Noam Shazeer
End-to-end text-dependent speaker verification. ICASSP (2016)
Babak Damavandi, Shankar Kumar, Noam Shazeer, Antoine Bruguier
NN-Grams: Unifying Neural Network and n-Gram Language Models for Speech Recognition. INTERSPEECH (2016)
Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, Yonghui Wu
Exploring the Limits of Language Modeling. CoRR (2016)
Noam Shazeer, Ryan Doherty, Colin Evans, Chris Waterson
Swivel: Improving Embeddings by Noticing What's Missing. CoRR (2016)
Joris Pelemans, Noam Shazeer, Ciprian Chelba
Pruning sparse non-negative matrix n-gram language models. INTERSPEECH (2015)
Noam Shazeer, Joris Pelemans, Ciprian Chelba
Sparse non-negative matrix language modeling for skip-grams. INTERSPEECH (2015)
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam Shazeer
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. CoRR (2015)
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam Shazeer
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. NIPS (2015)
Ciprian Chelba, Noam Shazeer
Sparse non-negative matrix language modeling for geo-annotated query session data. ASRU (2015)
Georg Heigold, Ignacio Moreno, Samy Bengio, Noam Shazeer
End-to-End Text-Dependent Speaker Verification. CoRR (2015)
Noam Shazeer, Joris Pelemans, Ciprian Chelba
Skip-gram Language Modeling Using Sparse Non-negative Matrix Probability Estimation. CoRR (2014)
Georges Harik, Noam Shazeer
Variational Program Inference CoRR (2010)