Tri Dao

Publication Activity (10 Years)

Years Active: 2017-2024
Publications (10 Years): 69

Top Topics

Structured Representations

Top Venues

Publications

Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision. CoRR (2024)
Sukjun Hwang, Aakash Lahoti, Tri Dao, Albert Gu
Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers. CoRR (2024)
Tri Dao, Albert Gu
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. CoRR (2024)
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, Tri Dao
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. CoRR (2024)
Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, Garvit Kulshreshtha, Vartika Singh, Jared Casper, Jan Kautz, Mohammad Shoeybi, Bryan Catanzaro
An Empirical Study of Mamba-based Language Models. CoRR (2024)
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian J. McAuley, Han Hu, Torsten Scholak, Sébastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, et al.
StarCoder 2 and The Stack v2: The Next Generation. CoRR (2024)
Tri Dao
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. ICLR (2024)
James Liu, Guangxuan Xiao, Kai Li, Jason D. Lee, Song Han, Tri Dao, Tianle Cai
BitDelta: Your Fine-Tune May Only Be Worth One Bit. CoRR (2024)
Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, Volodymyr Kuleshov
Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling. CoRR (2024)
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Ré, Beidi Chen
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time. ICML (2023)
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, Christopher Ré
Hyena Hierarchy: Towards Larger Convolutional Language Models. CoRR (2023)
Daniel Y. Fu, Elliot L. Epstein, Eric Nguyen, Armin W. Thomas, Michael Zhang, Tri Dao, Atri Rudra, Christopher Ré
Simple Hardware-Efficient Long Convolutions for Sequence Modeling. ICML (2023)
Michael Zhang, Khaled Kamal Saab, Michael Poli, Tri Dao, Karan Goel, Christopher Ré
Effectively Modeling Time Series with Simple Discrete State Spaces. CoRR (2023)
Albert Gu, Tri Dao
Mamba: Linear-Time Sequence Modeling with Selective State Spaces. CoRR (2023)
Michael Zhang, Khaled Kamal Saab, Michael Poli, Tri Dao, Karan Goel, Christopher Ré
Effectively Modeling Time Series with Simple Discrete State Spaces. ICLR (2023)
Daniel Y. Fu, Tri Dao, Khaled Kamal Saab, Armin W. Thomas, Atri Rudra, Christopher Ré
Hungry Hungry Hippos: Towards Language Modeling with State Space Models. ICLR (2023)
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy V, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, Harm de Vries
StarCoder: may the source be with you! CoRR (2023)
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason T. Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, Harm de Vries
StarCoder: may the source be with you! Trans. Mach. Learn. Res. 2023 (2023)
Daniel Y. Fu, Elliot L. Epstein, Eric Nguyen, Armin W. Thomas, Michael Zhang, Tri Dao, Atri Rudra, Christopher Ré
Simple Hardware-Efficient Long Convolutions for Sequence Modeling. CoRR (2023)
Tri Dao
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. CoRR (2023)
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, Christopher Ré
Hyena Hierarchy: Towards Larger Convolutional Language Models. ICML (2023)
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Ré, Beidi Chen
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time. CoRR (2023)
Binhang Yuan, Yongjun He, Jared Davis, Tianyi Zhang, Tri Dao, Beidi Chen, Percy Liang, Christopher Ré, Ce Zhang
Decentralized Training of Foundation Models in Heterogeneous Environments. NeurIPS (2022)
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS (2022)
Chenlin Meng, Linqi Zhou, Kristy Choi, Tri Dao, Stefano Ermon
ButterflyFlow: Building Invertible Layers with Butterfly Matrices. ICML (2022)
Eric Nguyen, Karan Goel, Albert Gu, Gordon W. Downs, Preey Shah, Tri Dao, Stephen A. Baccus, Christopher Ré
S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spaces. CoRR (2022)
Beidi Chen, Tri Dao, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, Christopher Ré
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models. ICLR (2022)
Eric Nguyen, Karan Goel, Albert Gu, Gordon W. Downs, Preey Shah, Tri Dao, Stephen Baccus, Christopher Ré
S4ND: Modeling Images and Videos as Multidimensional Signals with State Spaces. NeurIPS (2022)
Binhang Yuan, Yongjun He, Jared Quincy Davis, Tianyi Zhang, Tri Dao, Beidi Chen, Percy Liang, Christopher Ré, Ce Zhang
Decentralized Training of Foundation Models in Heterogeneous Environments. CoRR (2022)
Tri Dao, Beidi Chen, Nimit Sharad Sohoni, Arjun D. Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, Christopher Ré
Monarch: Expressive Structured Matrices for Efficient and Accurate Training. ICML (2022)
Chenlin Meng, Linqi Zhou, Kristy Choi, Tri Dao, Stefano Ermon
ButterflyFlow: Building Invertible Layers with Butterfly Matrices. CoRR (2022)
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. CoRR (2022)
Michael Poli, Stefano Massaroli, Federico Berto, Jinkyoo Park, Tri Dao, Christopher Ré, Stefano Ermon
Transform Once: Efficient Operator Learning in Frequency Domain. NeurIPS (2022)
Jue Wang, Binhang Yuan, Luka Rimanic, Yongjun He, Tri Dao, Beidi Chen, Christopher Ré, Ce Zhang
Fine-tuning Language Models over Slow Networks using Activation Quantization with Guarantees. NeurIPS (2022)
Jue Wang, Binhang Yuan, Luka Rimanic, Yongjun He, Tri Dao, Beidi Chen, Christopher Ré, Ce Zhang
Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees. CoRR (2022)
Tri Dao, Daniel Y. Fu, Khaled Kamal Saab, Armin W. Thomas, Atri Rudra, Christopher Ré
Hungry Hungry Hippos: Towards Language Modeling with State Space Models. CoRR (2022)
Michael Poli, Stefano Massaroli, Federico Berto, Jinkyoo Park, Tri Dao, Christopher Ré, Stefano Ermon
Transform Once: Efficient Operator Learning in Frequency Domain. CoRR (2022)
Tri Dao, Beidi Chen, Nimit Sharad Sohoni, Arjun D. Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, Christopher Ré
Monarch: Expressive Structured Matrices for Efficient and Accurate Training. CoRR (2022)
Tri Dao, Govinda M. Kamath, Vasilis Syrgkanis, Lester Mackey
Knowledge Distillation as Semiparametric Inference. ICLR (2021)
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, Christopher Ré
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State Space Layers. NeurIPS (2021)
Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, Christopher Ré
Scatterbrain: Unifying Sparse and Low-rank Attention. NeurIPS (2021)
Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, Christopher Ré
Scatterbrain: Unifying Sparse and Low-rank Attention Approximation. CoRR (2021)
Nicholas Roberts, Mikhail Khodak, Tri Dao, Liam Li, Christopher Ré, Ameet Talwalkar
Rethinking Neural Operations for Diverse Tasks. CoRR (2021)
Nicholas Roberts, Mikhail Khodak, Tri Dao, Liam Li, Christopher Ré, Ameet Talwalkar
Rethinking Neural Operations for Diverse Tasks. NeurIPS (2021)
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, Christopher Ré
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers. CoRR (2021)
Beidi Chen, Tri Dao, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, Christopher Ré
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models. CoRR (2021)
Jared Quincy Davis, Albert Gu, Krzysztof Choromanski, Tri Dao, Christopher Ré, Chelsea Finn, Percy Liang
Catformer: Designing Stable Transformers via Sensitivity Analysis. ICML (2021)
Beidi Chen, Zichang Liu, Binghui Peng, Zhaozhuo Xu, Jonathan Lingjie Li, Tri Dao, Zhao Song, Anshumali Shrivastava, Christopher Ré
MONGOOSE: A Learnable LSH Framework for Efficient Neural Network Training. ICLR (2021)
Tri Dao, Govinda M. Kamath, Vasilis Syrgkanis, Lester Mackey
Knowledge Distillation as Semiparametric Inference. CoRR (2021)
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, Christopher Ré
HiPPO: Recurrent Memory with Optimal Polynomial Projections. NeurIPS (2020)
Tri Dao, Nimit Sharad Sohoni, Albert Gu, Matthew Eichhorn, Amit Blonder, Megan Leszczynski, Atri Rudra, Christopher Ré
Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps. CoRR (2020)
Tri Dao, Nimit Sharad Sohoni, Albert Gu, Matthew Eichhorn, Amit Blonder, Megan Leszczynski, Atri Rudra, Christopher Ré
Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps. ICLR (2020)
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, Christopher Ré
HiPPO: Recurrent Memory with Optimal Polynomial Projections. CoRR (2020)
Jonathan Kuck, Tri Dao, Hamid Rezatofighi, Ashish Sabharwal, Stefano Ermon
Approximating the Permanent by Sampling from Adaptive Partitions. CoRR (2019)
Tri Dao, Albert Gu, Matthew Eichhorn, Atri Rudra, Christopher Ré
Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations. CoRR (2019)
Tri Dao, Albert Gu, Matthew Eichhorn, Atri Rudra, Christopher Ré
Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations. ICML (2019)
Jian Zhang, Avner May, Tri Dao, Christopher Ré
Low-Precision Random Fourier Features for Memory-constrained Kernel Approximation. AISTATS (2019)
Jonathan Kuck, Tri Dao, Hamid Rezatofighi, Ashish Sabharwal, Stefano Ermon
Approximating the Permanent by Sampling from Adaptive Partitions. NeurIPS (2019)
Avner May, Jian Zhang, Tri Dao, Christopher Ré
On the Downstream Performance of Compressed Word Embeddings. CoRR (2019)
Jonathan Kuck, Tri Dao, Shenjia Zhao, Burak Bartan, Ashish Sabharwal, Stefano Ermon
Adaptive Hashing for Model Counting. UAI (2019)
Avner May, Jian Zhang, Tri Dao, Christopher Ré
On the Downstream Performance of Compressed Word Embeddings. NeurIPS (2019)
Tri Dao, Albert Gu, Alexander Ratner, Virginia Smith, Chris De Sa, Christopher Ré
A Kernel Theory of Modern Data Augmentation. ICML (2019)
Anna T. Thomas, Albert Gu, Tri Dao, Atri Rudra, Christopher Ré
Learning Invariance with Compact Transforms. ICLR (Workshop) (2018)
Anna T. Thomas, Albert Gu, Tri Dao, Atri Rudra, Christopher Ré
Learning Compressed Transforms with Low Displacement Rank. CoRR (2018)
Tri Dao, Albert Gu, Alexander J. Ratner, Virginia Smith, Christopher De Sa, Christopher Ré
A Kernel Theory of Modern Data Augmentation. CoRR (2018)
Jian Zhang, Avner May, Tri Dao, Christopher Ré
Low-Precision Random Fourier Features for Memory-Constrained Kernel Approximation. CoRR (2018)
Anna T. Thomas, Albert Gu, Tri Dao, Atri Rudra, Christopher Ré
Learning Compressed Transforms with Low Displacement Rank. NeurIPS (2018)
Tri Dao, Christopher De Sa, Christopher Ré
Gaussian Quadrature for Kernel Features. NIPS (2017)
Tri Dao, Christopher De Sa, Christopher Ré
Gaussian Quadrature for Kernel Features. CoRR (2017)