Joshua Ainslie

Publication Activity (10 Years)

Years Active: 2020-2024
Publications (10 Years): 48

Top Topics

Avoid Overfitting

Semantic Parsing

Top Venues

Publications

Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontañón, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, Srinadh Bhojanapalli
Functional Interpolation for Relative Positions improves Long Context Transformers. ICLR (2024)
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyanskiy, David C. Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, Yun-Hsuan Sung, Sumit Sanghai
CoLT5: Faster Long-Range Transformers with Conditional Computation. EMNLP (2023)
Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Joshua Ainslie, Sumit Sanghai, Fei Sha, William W. Cohen
Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your compute. ICML (2023)
Michiel de Jong, Yury Zemlyanskiy, Joshua Ainslie, Nicholas FitzGerald, Sumit Sanghai, Fei Sha, William W. Cohen
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference. ACL (Findings) (2023)
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, Sumit Sanghai
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. EMNLP (2023)
David C. Uthus, Santiago Ontañón, Joshua Ainslie, Mandy Guo
mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences. EMNLP (Findings) (2023)
Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Sumit Sanghai, William W. Cohen, Joshua Ainslie
GLIMMER: generalized late-interaction memory reranker. CoRR (2023)
Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan A. Plummer, Kate Saenko, Jianmo Ni, Mandy Guo
A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding. EMNLP (2023)
Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan A. Plummer, Kate Saenko, Jianmo Ni, Mandy Guo
WikiWeb2M: A Page-Level Multimodal Wikipedia Dataset. CoRR (2023)
Chen-Yu Lee, Chun-Liang Li, Hao Zhang, Timothy Dozat, Vincent Perot, Guolong Su, Xiang Zhang, Kihyuk Sohn, Nikolai Glushnev, Renshen Wang, Joshua Ainslie, Shangbang Long, Siyang Qin, Yasuhisa Fujii, Nan Hua, Tomas Pfister
FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction. CoRR (2023)
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, Neil Houlsby
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints. ICLR (2023)
Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontañón, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, Srinadh Bhojanapalli
Functional Interpolation for Relative Positions Improves Long Context Transformers. CoRR (2023)
David C. Uthus, Santiago Ontañón, Joshua Ainslie, Mandy Guo
mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences. CoRR (2023)
Tao Lei, Junwen Bai, Siddhartha Brahma, Joshua Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Y. Zhao, Yuexin Wu, Bo Li, Yu Zhang, Ming-Wei Chang
Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference. CoRR (2023)
Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Joshua Ainslie, Sumit Sanghai, Fei Sha, William Cohen
Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your compute. CoRR (2023)
Tao Lei, Junwen Bai, Siddhartha Brahma, Joshua Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Y. Zhao, Yuexin Wu, Bo Li, Yu Zhang, Ming-Wei Chang
Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference. NeurIPS (2023)
Chen-Yu Lee, Chun-Liang Li, Hao Zhang, Timothy Dozat, Vincent Perot, Guolong Su, Xiang Zhang, Kihyuk Sohn, Nikolay Glushnev, Renshen Wang, Joshua Ainslie, Shangbang Long, Siyang Qin, Yasuhisa Fujii, Nan Hua, Tomas Pfister
FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction. ACL (1) (2023)
Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan A. Plummer, Kate Saenko, Jianmo Ni, Mandy Guo
A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding. CoRR (2023)
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, Sumit Sanghai
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. CoRR (2023)
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyanskiy, David C. Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, Yun-Hsuan Sung, Sumit Sanghai
CoLT5: Faster Long-Range Transformers with Conditional Computation. CoRR (2023)
Yury Zemlyanskiy, Michiel de Jong, Luke Vilnis, Santiago Ontañón, William W. Cohen, Sumit Sanghai, Joshua Ainslie
MEMORY-VQ: Compression for Tractable Internet-Scale Memory. CoRR (2023)
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, Neil Houlsby
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints. CoRR (2022)
James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontañón
FNet: Mixing Tokens with Fourier Transforms. NAACL-HLT (2022)
Chen-Yu Lee, Chun-Liang Li, Timothy Dozat, Vincent Perot, Guolong Su, Nan Hua, Joshua Ainslie, Renshen Wang, Yasuhisa Fujii, Tomas Pfister
FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extraction. CoRR (2022)
Chen-Yu Lee, Chun-Liang Li, Timothy Dozat, Vincent Perot, Guolong Su, Nan Hua, Joshua Ainslie, Renshen Wang, Yasuhisa Fujii, Tomas Pfister
FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extraction. ACL (1) (2022)
Mandy Guo, Joshua Ainslie, David C. Uthus, Santiago Ontañón, Jianmo Ni, Yun-Hsuan Sung, Yinfei Yang
LongT5: Efficient Text-To-Text Transformer for Long Sequences. NAACL-HLT (Findings) (2022)
Yury Zemlyanskiy, Michiel de Jong, Joshua Ainslie, Panupong Pasupat, Peter Shaw, Linlu Qiu, Sumit Sanghai, Fei Sha
Generate-and-Retrieve: Use Your Predictions to Improve Retrieval for Semantic Parsing. COLING (2022)
James Lee-Thorp, Joshua Ainslie
Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT. CoRR (2022)
Santiago Ontañón, Joshua Ainslie, Zachary Fisher, Vaclav Cvicek
Making Transformers Solve Compositional Tasks. ACL (1) (2022)
Michiel de Jong, Yury Zemlyanskiy, Joshua Ainslie, Nicholas FitzGerald, Sumit Sanghai, Fei Sha, William Cohen
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference. CoRR (2022)
James Lee-Thorp, Joshua Ainslie
Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT. EMNLP (Findings) (2022)
Santiago Ontañón, Joshua Ainslie, Vaclav Cvicek, Zachary Fisher
LogicInference: A New Dataset for Teaching Logical Inference to seq2seq Models. CoRR (2022)
Yury Zemlyanskiy, Michiel de Jong, Joshua Ainslie, Panupong Pasupat, Peter Shaw, Linlu Qiu, Sumit Sanghai, Fei Sha
Generate-and-Retrieve: use your predictions to improve retrieval for semantic parsing. CoRR (2022)
Yury Zemlyanskiy, Joshua Ainslie, Michiel de Jong, Philip Pham, Ilya Eckstein, Fei Sha
ReadTwice: Reading Very Large Documents with Memories. CoRR (2021)
Juyong Kim, Pradeep Ravikumar, Joshua Ainslie, Santiago Ontañón
Improving Compositional Generalization in Classification Tasks via Structure Annotations. CoRR (2021)
Yury Zemlyanskiy, Joshua Ainslie, Michiel de Jong, Philip Pham, Ilya Eckstein, Fei Sha
ReadTwice: Reading Very Large Documents with Memories. NAACL-HLT (2021)
Juyong Kim, Pradeep Ravikumar, Joshua Ainslie, Santiago Ontañón
Improving Compositional Generalization in Classification Tasks via Structure Annotations. ACL/IJCNLP (2) (2021)
Santiago Ontañón, Joshua Ainslie, Vaclav Cvicek, Zachary Fisher
Making Transformers Solve Compositional Tasks. CoRR (2021)
James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontañón
FNet: Mixing Tokens with Fourier Transforms. CoRR (2021)
Mandy Guo, Joshua Ainslie, David C. Uthus, Santiago Ontañón, Jianmo Ni, Yun-Hsuan Sung, Yinfei Yang
LongT5: Efficient Text-To-Text Transformer for Long Sequences. CoRR (2021)
Luana Ruiz, Joshua Ainslie, Santiago Ontañón
Iterative Decoding for Compositional Generalization in Transformers. CoRR (2021)
Ruining He, Anirudh Ravula, Bhargav Kanagal, Joshua Ainslie
RealFormer: Transformer Likes Residual Attention. ACL/IJCNLP (Findings) (2021)
Gurmeet Manku, James Lee-Thorp, Bhargav Kanagal, Joshua Ainslie, Jingchen Feng, Zach Pearson, Ebenezer Anjorin, Sudeep Gandhe, Ilya Eckstein, Jim Rosswog, Sumit Sanghai, Michael Pohl, Larry Adams, D. Sivakumar
ShopTalk: A System for Conversational Faceted Search. CoRR (2021)
Joshua Ainslie, Santiago Ontañón, Chris Alberti, Philip Pham, Anirudh Ravula, Sumit Sanghai
ETC: Encoding Long and Structured Data in Transformers. CoRR (2020)
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, Amr Ahmed
Big Bird: Transformers for Longer Sequences. NeurIPS (2020)
Ruining He, Anirudh Ravula, Bhargav Kanagal, Joshua Ainslie
RealFormer: Transformer Likes Residual Attention. CoRR (2020)
Joshua Ainslie, Santiago Ontañón, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, Li Yang
ETC: Encoding Long and Structured Inputs in Transformers. EMNLP (1) (2020)
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, Amr Ahmed
Big Bird: Transformers for Longer Sequences. CoRR (2020)