David Harwath

Publication Activity (10 Years)

Years Active: 2013-2024
Publications (10 Years): 87

Top Topics

Home Environments

Speech Recognition

Visual Representation

Top Venues

Publications

Jordan Voas, Raymond Mooney, David Harwath
Multimodal Contextualized Semantic Parsing from Speech. CoRR (2024)
Yuan Tseng, Layne Berry, Yiting Chen, I-Hsiang Chiu, Hsuan-Hao Lin, Max Liu, Puyuan Peng, Yi-Jen Shih, Hung-Yu Wang, Haibin Wu, Poyao Huang, Chun-Mao Lai, Shang-Wen Li, David Harwath, Yu Tsao, Abdelrahman Mohamed, Chi-Luen Feng, Hung-Yi Lee
AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models. ICASSP (2024)
Changan Chen, Puyuan Peng, Ami Baid, Zihui Xue, Wei-Ning Hsu, David Harwath, Kristen Grauman
Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos. CoRR (2024)
Changan Chen, Kumar Ashutosh, Rohit Girdhar, David Harwath, Kristen Grauman
SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos. CoRR (2024)
Hsuan-Fu Wang, Yi-Jen Shih, Heng-Jui Chang, Layne Berry, Puyuan Peng, Hung-yi Lee, Hsin-Min Wang, David Harwath
SpeechCLIP+: Self-supervised multi-task representation learning for speech via CLIP and speech-image data. CoRR (2024)
Yi-Jen Shih, David Harwath
Interface Design for Self-Supervised Speech Models. CoRR (2024)
Zhisheng Zheng, Puyuan Peng, Ziyang Ma, Xie Chen, Eunsol Choi, David Harwath
BAT: Learning to Reason about Spatial Sounds with Large Language Models. CoRR (2024)
Jordan Voas, David Harwath, Raymond Mooney
Multimodal Contextualized Semantic Parsing from Speech. ACL (1) (2024)
Puyuan Peng, Po-Yao Huang, Shang-Wen Li, Abdelrahman Mohamed, David Harwath
VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild. ACL (1) (2024)
Puyuan Peng, Po-Yao Huang, Daniel Li, Abdelrahman Mohamed, David Harwath
VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild. CoRR (2024)
Hung-Chieh Fang, Nai-Xuan Ye, Yi-Jen Shih, Puyuan Peng, Hsuan-Fu Wang, Layne Berry, Hung-yi Lee, David Harwath
Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Model. CoRR (2024)
Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, James R. Glass
Contrastive Audio-Visual Masked Autoencoder. ICLR (2023)
Puyuan Peng, Brian Yan, Shinji Watanabe, David Harwath
Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization. INTERSPEECH (2023)
Debadatta Dash, Paul Ferrari, Abbas Babajani-Feremi, David Harwath, Amir Borna, Jun Wang
Subject Generalization in Classifying Imagined and Spoken Speech with MEG. NER (2023)
Reem Gody, David Harwath
Unsupervised Fine-Tuning Data Selection for ASR Using Self-Supervised Speech Models. ICASSP (2023)
Cheng-I Jeff Lai, Freda Shi, Puyuan Peng, Yoon Kim, Kevin Gimpel, Shiyu Chang, Yung-Sung Chuang, Saurabhchand Bhati, David D. Cox, David Harwath, Yang Zhang, Karen Livescu, James R. Glass
Audio-Visual Neural Syntax Acquisition. CoRR (2023)
Anuj Diwan, Ching-Feng Yeh, Wei-Ning Hsu, Paden Tomasello, Eunsol Choi, David Harwath, Abdelrahman Mohamed
Continual Learning for On-Device Speech Recognition Using Disentangled Conformers. ICASSP (2023)
Layne Berry, Yi-Jen Shih, Hsuan-Fu Wang, Heng-Jui Chang, Hung-Yi Lee, David Harwath
M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrieval. ICASSP (2023)
Chiori Hori, Puyuan Peng, David Harwath, Xinyu Liu, Kei Ota, Siddarth Jain, Radu Corcodel, Devesh K. Jha, Diego Romeres, Jonathan Le Roux
Style-transfer based Speech and Audio-visual Scene Understanding for Robot Action Sequence Acquisition from Videos. CoRR (2023)
Puyuan Peng, Brian Yan, Shinji Watanabe, David Harwath
Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization. CoRR (2023)
Dawei Liang, Zifan Xu, Yinuo Chen, Rebecca Adaimi, David Harwath, Edison Thomaz
A Dataset for Foreground Speech Analysis With Smartwatches In Everyday Home Environments. ICASSP Workshops (2023)
Yuan Tseng, Layne Berry, Yi-Ting Chen, I-Hsiang Chiu, Hsuan-Hao Lin, Max Liu, Puyuan Peng, Yi-Jen Shih, Hung-Yu Wang, Haibin Wu, Po-Yao Huang, Chun-Mao Lai, Shang-Wen Li, David Harwath, Yu Tsao, Shinji Watanabe, Abdelrahman Mohamed, Chi-Luen Feng, Hung-yi Lee
AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models. CoRR (2023)
Puyuan Peng, Shang-Wen Li, Okko Räsänen, Abdelrahman Mohamed, David Harwath
Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Model. INTERSPEECH (2023)
Anuj Diwan, Eunsol Choi, David Harwath
When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants. ACL (2) (2023)
Changan Chen, Wei Sun, David Harwath, Kristen Grauman
Learning Audio-Visual Dereverberation. ICASSP (2023)
Puyuan Peng, Shang-Wen Li, Okko Räsänen, Abdelrahman Mohamed, David Harwath
Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Mode. CoRR (2023)
Anuj Diwan, Anirudh Srinivasan, David Harwath, Eunsol Choi
Unit-based Speech-to-Speech Translation Without Parallel Data. CoRR (2023)
Cheng-I Jeff Lai, Freda Shi, Puyuan Peng, Yoon Kim, Kevin Gimpel, Shiyu Chang, Yung-Sung Chuang, Saurabhchand Bhati, David D. Cox, David Harwath, Yang Zhang, Karen Livescu, James R. Glass
Audio-Visual Neural Syntax Acquisition. ASRU (2023)
Xixi Hu, Senthil Purushwalkam, David Harwath, Kristen Grauman
Learning to Map Efficiently by Active Echolocation. IROS (2023)
Chiori Hori, Puyuan Peng, David Harwath, Xinyu Liu, Kei Ota, Siddarth Jain, Radu Corcodel, Devesh K. Jha, Diego Romeres, Jonathan Le Roux
Style-transfer based Speech and Audio-visual Scene understanding for Robot Action Sequence Acquisition from Videos. INTERSPEECH (2023)
Andrew Rouditchenko, Yung-Sung Chuang, Nina Shvetsova, Samuel Thomas, Rogério Feris, Brian Kingsbury, Leonid Karlinsky, David Harwath, Hilde Kuehne, James R. Glass
C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval. ICASSP (2023)
Andrew Rouditchenko, Sameer Khurana, Samuel Thomas, Rogério Feris, Leonid Karlinsky, Hilde Kuehne, David Harwath, Brian Kingsbury, James R. Glass
Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages. CoRR (2023)
Andrew Rouditchenko, Sameer Khurana, Samuel Thomas, Rogério Feris, Leonid Karlinsky, Hilde Kuehne, David Harwath, Brian Kingsbury, James R. Glass
Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages. INTERSPEECH (2023)
Anuj Diwan, Eunsol Choi, David Harwath
When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants. CoRR (2023)
Anuj Diwan, Ching-Feng Yeh, Wei-Ning Hsu, Paden Tomasello, Eunsol Choi, David Harwath, Abdelrahman Mohamed
Continual Learning for On-Device Speech Recognition using Disentangled Conformers. CoRR (2022)
Alan Baade, Puyuan Peng, David Harwath
MAE-AST: Masked Autoencoding Audio Spectrogram Transformer. INTERSPEECH (2022)
Alan Baade, Puyuan Peng, David Harwath
MAE-AST: Masked Autoencoding Audio Spectrogram Transformer. CoRR (2022)
Dawei Liang, Zifan Xu, Yinuo Chen, Rebecca Adaimi, David Harwath, Edison Thomaz
Automated detection of foreground speech with wearable sensing in everyday home environments: A transfer learning approach. CoRR (2022)
Nina Shvetsova, Brian Chen, Andrew Rouditchenko, Samuel Thomas, Brian Kingsbury, Rogério Feris, David Harwath, James R. Glass, Hilde Kuehne
Everything at Once - Multi-modal Fusion Transformer for Video Retrieval. CVPR (2022)
Puyuan Peng, David Harwath
Word Discovery in Visually Grounded, Self-Supervised Speech Models. CoRR (2022)
Tyler Miller, David Harwath
Exploring Few-Shot Fine-Tuning Strategies for Models of Visually Grounded Speech. INTERSPEECH (2022)
Christopher Song, David Harwath, Tuka Alhanai, James R. Glass
Speak: A Toolkit Using Amazon Mechanical Turk to Collect and Validate Speech Audio Recordings. LREC (2022)
Luke Strgar, David Harwath
Phoneme Segmentation Using Self-Supervised Speech Models. CoRR (2022)
Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, James R. Glass
Contrastive Audio-Visual Masked Autoencoder. CoRR (2022)
Reem Gody, David Harwath
Unsupervised Fine-Tuning Data Selection for ASR Using Self-Supervised Speech Models. CoRR (2022)
Puyuan Peng, David Harwath
Fast-Slow Transformer for Visually Grounding Speech. ICASSP (2022)
Anuj Diwan, Layne Berry, Eunsol Choi, David Harwath, Kyle Mahowald
Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality. EMNLP (2022)
Yi-Jen Shih, Hsuan-Fu Wang, Heng-Jui Chang, Layne Berry, Hung-yi Lee, David Harwath
SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model. SLT (2022)
David Xu, David Harwath
Adversarial Input Ablation for Audio-Visual Learning. ICASSP (2022)
Puyuan Peng, David Harwath
Word Discovery in Visually Grounded, Self-Supervised Speech Models. INTERSPEECH (2022)
Andrew Rouditchenko, Yung-Sung Chuang, Nina Shvetsova, Samuel Thomas, Rogério Feris, Brian Kingsbury, Leonid Karlinsky, David Harwath, Hilde Kuehne, James R. Glass
C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval. CoRR (2022)
Yi-Jen Shih, Hsuan-Fu Wang, Heng-Jui Chang, Layne Berry, Hung-yi Lee, David Harwath
SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model. CoRR (2022)
Layne Berry, Yi-Jen Shih, Hsuan-Fu Wang, Heng-Jui Chang, Hung-yi Lee, David Harwath
M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrieval. CoRR (2022)
Luke Strgar, David Harwath
Phoneme Segmentation Using Self-Supervised Speech Models. SLT (2022)
Anuj Diwan, Layne Berry, Eunsol Choi, David Harwath, Kyle Mahowald
Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality. CoRR (2022)
Puyuan Peng, David Harwath
Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling. CoRR (2022)
Mathew Monfort, SouYoung Jin, Alexander H. Liu, David Harwath, Rogério Feris, James R. Glass, Aude Oliva
Spoken Moments: Learning Joint Audio-Visual Representations From Video Descriptions. CVPR (2021)
Andrew Rouditchenko, Angie W. Boggust, David Harwath, Brian Chen, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Hilde Kuehne, Rameswar Panda, Rogério Schmidt Feris, Brian Kingsbury, Michael Picheny, Antonio Torralba, James R. Glass
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos. Interspeech (2021)
Nina Shvetsova, Brian Chen, Andrew Rouditchenko, Samuel Thomas, Brian Kingsbury, Rogério Feris, David Harwath, James R. Glass, Hilde Kuehne
Everything at Once - Multi-modal Fusion Transformer for Video Retrieval. CoRR (2021)
Mathew Monfort, SouYoung Jin, Alexander H. Liu, David Harwath, Rogério Feris, James R. Glass, Aude Oliva
Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptions. CoRR (2021)
Brian Chen, Andrew Rouditchenko, Kevin Duarte, Hilde Kuehne, Samuel Thomas, Angie W. Boggust, Rameswar Panda, Brian Kingsbury, Rogério Schmidt Feris, David Harwath, James R. Glass, Michael Picheny, Shih-Fu Chang
Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos. CoRR (2021)
Kevin Duarte, Brian Chen, Nina Shvetsova, Andrew Rouditchenko, Samuel Thomas, Alexander Liu, David Harwath, James R. Glass, Hilde Kuehne, Mubarak Shah
Routing with Self-Attention for Multimodal Capsule Networks. CoRR (2021)
Brian Chen, Andrew Rouditchenko, Kevin Duarte, Hilde Kuehne, Samuel Thomas, Angie W. Boggust, Rameswar Panda, Brian Kingsbury, Rogério Feris, David Harwath, James R. Glass, Michael Picheny, Shih-Fu Chang
Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos. ICCV (2021)
Andrew Rouditchenko, Angie W. Boggust, David Harwath, Samuel Thomas, Hilde Kuehne, Brian Chen, Rameswar Panda, Rogério Feris, Brian Kingsbury, Michael Picheny, James R. Glass
Cascaded Multilingual Audio-Visual Learning from Videos. Interspeech (2021)
Changan Chen, Wei Sun, David Harwath, Kristen Grauman
Learning Audio-Visual Dereverberation. CoRR (2021)
Andrew Rouditchenko, Angie W. Boggust, David Harwath, Samuel Thomas, Hilde Kuehne, Brian Chen, Rameswar Panda, Rogério Feris, Brian Kingsbury, Michael Picheny, James R. Glass
Cascaded Multilingual Audio-Visual Learning from Videos. CoRR (2021)
Puyuan Peng, David Harwath
Fast-Slow Transformer for Visually Grounding Speech. CoRR (2021)
Wei-Ning Hsu, David Harwath, Tyler Miller, Christopher Song, James R. Glass
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units. ACL/IJCNLP (1) (2021)
Yasunori Ohishi, Akisato Kimura, Takahito Kawanishi, Kunio Kashino, David Harwath, James R. Glass
Trilingual Semantic Embeddings of Visually Grounded Speech with Self-Attention Mechanisms. ICASSP (2020)
David Harwath, Wei-Ning Hsu, James R. Glass
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech. ICLR (2020)
Wei-Ning Hsu, David Harwath, Christopher Song, James R. Glass
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units. CoRR (2020)
Andrew Rouditchenko, Angie W. Boggust, David Harwath, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Rogério Feris, Brian Kingsbury, Michael Picheny, Antonio Torralba, James R. Glass
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos. CoRR (2020)
David Harwath, Adrià Recasens, Dídac Surís, Galen Chuang, Antonio Torralba, James R. Glass
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input. Int. J. Comput. Vis. 128 (3) (2020)
Yasunori Ohishi, Akisato Kimura, Takahito Kawanishi, Kunio Kashino, David Harwath, James R. Glass
Pair Expansion for Learning Multilingual Semantic Embeddings Using Disjoint Visually-Grounded Speech Audio Datasets. INTERSPEECH (2020)
Emmanuel Azuh, David Harwath, James R. Glass
Towards Bilingual Lexicon Discovery From Visually Grounded Speech Audio. INTERSPEECH (2019)
Wei-Ning Hsu, David Harwath, James R. Glass
Transfer Learning from Audio-Visual Grounding to Speech Recognition. INTERSPEECH (2019)
Angie W. Boggust, Kartik Audhkhasi, Dhiraj Joshi, David Harwath, Samuel Thomas, Rogério Schmidt Feris, Danny Gutfreund, Yang Zhang, Antonio Torralba, Michael Picheny, James R. Glass
Grounding Spoken Words in Unlabeled Video. CVPR Workshops (2019)
David Harwath, James R. Glass
Towards Visually Grounded Sub-word Speech Unit Discovery. ICASSP (2019)
David Harwath, James R. Glass
Towards Visually Grounded Sub-Word Speech Unit Discovery. CoRR (2019)
Didac Suris, Adrià Recasens, David Bau, David Harwath, James R. Glass, Antonio Torralba
Learning Words by Drawing Images. CVPR (2019)
David Harwath, Wei-Ning Hsu, James R. Glass
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech. CoRR (2019)
David Harwath, Adrià Recasens, Dídac Surís, Galen Chuang, Antonio Torralba, James R. Glass
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input. CoRR (2018)
David Harwath, Galen Chuang, James R. Glass
Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech. ICASSP (2018)
David Harwath, Adrià Recasens, Dídac Surís, Galen Chuang, Antonio Torralba, James R. Glass
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input. ECCV (6) (2018)
Kenneth Leidal, David Harwath, James R. Glass
Learning Modality-Invariant Representations for Speech and Images. CoRR (2017)
David Harwath, James R. Glass
Learning Word-Like Units from Joint Audio-Visual Analysis. ACL (1) (2017)
Kenneth Leidal, David Harwath, James R. Glass
Learning modality-invariant representations for speech and images. ASRU (2017)
David Harwath, Alexander Gruenstein, Ian McGraw
Choosing useful word alternates for automatic speech recognition correction interfaces. INTERSPEECH (2014)
Aren Jansen, Emmanuel Dupoux, Sharon Goldwater, Mark Johnson, Sanjeev Khudanpur, Kenneth Church, Naomi Feldman, Hynek Hermansky, Florian Metze, Richard C. Rose, Mike Seltzer, Pascal Clark, Ian McGraw, Balakrishnan Varadarajan, Erin Bennett, Benjamin Börschinger, Justin T. Chiu, Ewan Dunbar, Abdellah Fourtassi, David Harwath, Chia-ying Lee, Keith D. Levin, Atta Norouzian, Vijayaditya Peddinti, Rachael Richardson, Thomas Schatz, Samuel Thomas
A summary of the 2012 JHU CLSP workshop on zero resource speech technologies and models of early language acquisition. ICASSP (2013)