Tri Dao

tri_photo_2021_04.jpeg
tri [at] tridao (dot) me

Assistant Professor of Computer Science at Princeton University.
Chief Scientist at Together AI.

CV

Previously: PhD, Department of Computer Science, Stanford University

Research Interests

Machine learning and systems, with a focus on efficient training and long-range context:

  • Efficient Transformer training and inference.
  • Sequence models with long-range memory.
  • Structured sparsity for compact deep learning models.

latest posts

Jul 11, 2024 FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
May 31, 2024 State Space Duality (Mamba-2) Part IV - The Systems
May 31, 2024 State Space Duality (Mamba-2) Part III - The Algorithm
May 31, 2024 State Space Duality (Mamba-2) Part II - The Theory
May 31, 2024 State Space Duality (Mamba-2) Part I - The Model

selected publications

  1. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
    Jay Shah*, Ganesh Bikshandi*, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao
    2024
    Code
  2. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
    Tri Dao*, and Albert Gu*
    In International Conference on Machine Learning (ICML) , 2024
    arXiv Code
  3. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré
    In Advances in Neural Information Processing Systems , 2022
    arXiv Code
  4. Mamba: Linear-Time Sequence Modeling with Selective State Spaces
    Albert Gu*, and Tri Dao*
    Conference on Language Modeling (COLM), 2023
    arXiv Code
  5. Monarch: Expressive Structured Matrices for Efficient and Accurate Training
    Tri Dao, Beidi Chen, Nimit Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher Ré
    In International Conference on Machine Learning (ICML) , 2022
    arXiv Code