Kazuki Irie

Publication Activity (10 Years)

Years Active: 2014-2024
Publications (10 Years): 61

Top Topics

Rapidly Changing

Speech Recognition

Top Venues

Publications

Lorenzo Tiberi, Francesca Mignacco, Kazuki Irie, Haim Sompolinsky
Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers. CoRR (2024)
Kazuki Irie, Anand Gopalakrishnan, Jürgen Schmidhuber
Exploring the Promise and Limits of Real-Time Recurrent Learning. ICLR (2024)
Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber, Christopher Potts, Christopher D. Manning
MoEUT: Mixture-of-Experts Universal Transformers. CoRR (2024)
Kazuki Irie, Jürgen Schmidhuber
Accelerating Neural Self-Improvement via Bootstrapping. CoRR (2023)
Kazuki Irie, Róbert Csordás, Jürgen Schmidhuber
Topological Neural Discrete Representation Learning à la Kohonen. CoRR (2023)
Kazuki Irie, Róbert Csordás, Jürgen Schmidhuber
Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions. EMNLP (2023)
Aleksandar Stanic, Anand Gopalakrishnan, Kazuki Irie, Jürgen Schmidhuber
Contrastive Training of Complex-Valued Autoencoders for Object Discovery. NeurIPS (2023)
Róbert Csordás, Piotr Piekos, Kazuki Irie, Jürgen Schmidhuber
SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention. CoRR (2023)
Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber
Approximating Two-Layer Feedforward Networks for Efficient Transformers. CoRR (2023)
Anand Gopalakrishnan, Kazuki Irie, Jürgen Schmidhuber, Sjoerd van Steenkiste
Unsupervised Learning of Temporal Abstractions With Slot-Based Transformers. Neural Comput. 35 (4) (2023)
Kazuki Irie, Róbert Csordás, Jürgen Schmidhuber
Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions. CoRR (2023)
Kazuki Irie, Anand Gopalakrishnan, Jürgen Schmidhuber
Exploring the Promise and Limits of Real-Time Recurrent Learning. CoRR (2023)
Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber
Approximating Two-Layer Feedforward Networks for Efficient Transformers. EMNLP (Findings) (2023)
Mingchen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R. Ashley, Róbert Csordás, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Irie, Louis Kirsch, Bing Li, Guohao Li, Shuming Liu, Jinjie Mai, Piotr Piekos, Aditya Ramesh, Imanol Schlag, Weimin Shi, Aleksandar Stanic, Wenyi Wang, Yuhui Wang, Mengmeng Xu, Deng-Ping Fan, Bernard Ghanem, Jürgen Schmidhuber
Mindstorms in Natural Language-Based Societies of Mind. CoRR (2023)
Kazuki Irie, Róbert Csordás, Jürgen Schmidhuber
Automating Continual Learning. CoRR (2023)
Aleksandar Stanic, Anand Gopalakrishnan, Kazuki Irie, Jürgen Schmidhuber
Contrastive Training of Complex-Valued Autoencoders for Object Discovery. CoRR (2023)
Kazuki Irie, Jürgen Schmidhuber
Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules. ICLR (2023)
Kazuki Irie, Róbert Csordás, Jürgen Schmidhuber
The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention. CoRR (2022)
Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber
CTL++: Evaluating Generalization on Never-Seen Compositional Patterns of Known Functions, and Compatibility of Neural Representations. EMNLP (2022)
Anand Gopalakrishnan, Kazuki Irie, Jürgen Schmidhuber, Sjoerd van Steenkiste
Unsupervised Learning of Temporal Abstractions with Slot-based Transformers. CoRR (2022)
Kazuki Irie, Francesco Faccio, Jürgen Schmidhuber
Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules. CoRR (2022)
Kazuki Irie, Francesco Faccio, Jürgen Schmidhuber
Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules. NeurIPS (2022)
Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber
CTL++: Evaluating Generalization on Never-Seen Compositional Patterns of Known Functions, and Compatibility of Neural Representations. CoRR (2022)
Kazuki Irie, Róbert Csordás, Jürgen Schmidhuber
The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention. ICML (2022)
Kazuki Irie, Imanol Schlag, Róbert Csordás, Jürgen Schmidhuber
A Modern Self-Referential Weight Matrix That Learns to Modify Itself. ICML (2022)
Kazuki Irie, Imanol Schlag, Róbert Csordás, Jürgen Schmidhuber
A Modern Self-Referential Weight Matrix That Learns to Modify Itself. CoRR (2022)
Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber
The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization. ICLR (2022)
Kazuki Irie, Jürgen Schmidhuber
Learning to Control Rapidly Changing Synaptic Connections: An Alternative Type of Memory in Sequence Processing Artificial Neural Networks. CoRR (2022)
Kazuki Irie, Jürgen Schmidhuber
Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules. CoRR (2022)
Kazuki Irie, Jürgen Schmidhuber
Training and Generating Neural Networks in Compressed Weight Space. CoRR (2021)
Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber
The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization. CoRR (2021)
Imanol Schlag, Kazuki Irie, Jürgen Schmidhuber
Linear Transformers Are Secretly Fast Weight Programmers. ICML (2021)
Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber
The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers. EMNLP (1) (2021)
Kazuki Irie, Imanol Schlag, Róbert Csordás, Jürgen Schmidhuber
Improving Baselines in the Wild. CoRR (2021)
Kazuki Irie, Imanol Schlag, Róbert Csordás, Jürgen Schmidhuber
Going Beyond Linear Transformers with Recurrent Fast Weight Programmers. CoRR (2021)
Kazuki Irie, Imanol Schlag, Róbert Csordás, Jürgen Schmidhuber
Going Beyond Linear Transformers with Recurrent Fast Weight Programmers. NeurIPS (2021)
Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber
The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers. CoRR (2021)
Imanol Schlag, Kazuki Irie, Jürgen Schmidhuber
Linear Transformers Are Secretly Fast Weight Memory Systems. CoRR (2021)
Kazuki Irie, Alexander Gerstenberger, Ralf Schlüter, Hermann Ney
How Much Self-Attention Do We Needƒ Trading Attention for Feed-Forward Layers. ICASSP (2020)
Alexander Gerstenberger, Kazuki Irie, Pavel Golik, Eugen Beck, Hermann Ney
Domain Robust, Fast, and Compact Neural Language Models. ICASSP (2020)
Wei Zhou, Wilfried Michel, Kazuki Irie, Markus Kitza, Ralf Schlüter, Hermann Ney
The Rwth Asr System for Ted-Lium Release 2: Improving Hybrid Hmm With Specaugment. ICASSP (2020)
Wei Zhou, Wilfried Michel, Kazuki Irie, Markus Kitza, Ralf Schlüter, Hermann Ney
The RWTH ASR System for TED-LIUM Release 2: Improving Hybrid HMM with SpecAugment. CoRR (2020)
Christoph Lüscher, Eugen Beck, Kazuki Irie, Markus Kitza, Wilfried Michel, Albert Zeyer, Ralf Schlüter, Hermann Ney
RWTH ASR Systems for LibriSpeech: Hybrid vs Attention - w/o Data Augmentation. CoRR (2019)
Albert Zeyer, Parnia Bahar, Kazuki Irie, Ralf Schlüter, Hermann Ney
A Comparison of Transformer and LSTM Encoder Decoder Models for ASR. ASRU (2019)
Kazuki Irie, Albert Zeyer, Ralf Schlüter, Hermann Ney
Language Modeling with Deep Transformers. CoRR (2019)
Kazuki Irie, Rohit Prabhavalkar, Anjuli Kannan, Antoine Bruguier, David Rybach, Patrick Nguyen
On the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition. INTERSPEECH (2019)
Kazuki Irie, Albert Zeyer, Ralf Schlüter, Hermann Ney
Language Modeling with Deep Transformers. INTERSPEECH (2019)
Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, Mia Xu Chen, Ye Jia, Anjuli Kannan, Tara N. Sainath, Yuan Cao, Chung-Cheng Chiu, Yanzhang He, Jan Chorowski, Smit Hinsu, Stella Laurenzo, James Qin, Orhan Firat, Wolfgang Macherey, Suyog Gupta, Ankur Bapna, Shuyuan Zhang, Ruoming Pang, Ron J. Weiss, Rohit Prabhavalkar, Qiao Liang, Benoit Jacob, Bowen Liang, HyoukJoong Lee, Ciprian Chelba, Sébastien Jean, Bo Li, Melvin Johnson, Rohan Anil, Rajat Tibrewal, Xiaobing Liu, Akiko Eriguchi, Navdeep Jaitly, Naveen Ari, Colin Cherry, Parisa Haghani, Otavio Good, Youlong Cheng, Raziel Alvarez, Isaac Caswell, Wei-Ning Hsu, Zongheng Yang, Kuan-Chieh Wang, Ekaterina Gonina, Katrin Tomanek, Ben Vanik, Zelin Wu, Llion Jones, Mike Schuster, Yanping Huang, Dehao Chen, Kazuki Irie, George F. Foster, John Richardson, Klaus Macherey, Antoine Bruguier, Heiga Zen, Colin Raffel, Shankar Kumar, Kanishka Rao, David Rybach, Matthew Murray, Vijayaditya Peddinti, Maxim Krikun, Michiel Bacchiani, Thomas B. Jablin, Robert Suderman, Ian Williams, Benjamin Lee, Deepti Bhatia, Justin Carlson, Semih Yavuz, Yu Zhang, Ian McGraw, Max Galkin, Qi Ge, Golan Pundak, Chad Whipkey, Todd Wang, Uri Alon, Dmitry Lepikhin, Ye Tian, Sara Sabour, William Chan, Shubham Toshniwal, Baohua Liao, Michael Nirschl, Pat Rondon
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling. CoRR (2019)
Kazuki Irie, Rohit Prabhavalkar, Anjuli Kannan, Antoine Bruguier, David Rybach, Patrick Nguyen
Model Unit Exploration for Sequence-to-Sequence Speech Recognition. CoRR (2019)
Christoph Lüscher, Eugen Beck, Kazuki Irie, Markus Kitza, Wilfried Michel, Albert Zeyer, Ralf Schlüter, Hermann Ney
RWTH ASR Systems for LibriSpeech: Hybrid vs Attention. INTERSPEECH (2019)
Kazuki Irie, Albert Zeyer, Ralf Schlüter, Hermann Ney
Training Language Models for Long-Span Cross-Sentence Evaluation. ASRU (2019)
Albert Zeyer, Kazuki Irie, Ralf Schlüter, Hermann Ney
Improved training of end-to-end attention models for speech recognition. CoRR (2018)
Kazuki Irie, Zhihong Lei, Liuhui Deng, Ralf Schlüter, Hermann Ney
Investigation on Estimation of Sentence Probability by Combining Forward, Backward and Bi-directional LSTM-RNNs. INTERSPEECH (2018)
Kazuki Irie, Zhihong Lei, Ralf Schlüter, Hermann Ney
Prediction of LSTM-RNN Full Context States as a Subtask for N-Gram Feedforward Language Models. ICASSP (2018)
Albert Zeyer, Kazuki Irie, Ralf Schlüter, Hermann Ney
Improved Training of End-to-end Attention Models for Speech Recognition. INTERSPEECH (2018)
Kazuki Irie, Shankar Kumar, Michael Nirschl, Hank Liao
RADMM: Recurrent Adaptive Mixture Model with Applications to Domain Robust Language Modeling. ICASSP (2018)
Kazuki Irie, Pavel Golik, Ralf Schlüter, Hermann Ney
Investigations on byte-level convolutional neural networks for language modeling in low resource speech recognition. ICASSP (2017)
Pavel Golik, Zoltán Tüske, Kazuki Irie, Eugen Beck, Ralf Schlüter, Hermann Ney
The 2016 RWTH Keyword Search System for Low-Resource Languages. SPECOM (2017)
Kazuki Irie, Zoltán Tüske, Tamer Alkhouli, Ralf Schlüter, Hermann Ney
LSTM, GRU, Highway and a Bit of Attention: An Empirical Overview for Language Modeling in Speech Recognition. INTERSPEECH (2016)
Zoltán Tüske, Kazuki Irie, Ralf Schlüter, Hermann Ney
Investigation on log-linear interpolation of multi-domain neural network language model. ICASSP (2016)
Ralf Schlüter, Patrick Doetsch, Pavel Golik, Markus Kitza, Tobias Menne, Kazuki Irie, Zoltán Tüske, Albert Zeyer
Automatic Speech Recognition Based on Neural Networks. SPECOM (2016)
Kazuki Irie, Ralf Schlüter, Hermann Ney
Bag-of-words input for long history representation in neural network-based language models for speech recognition. INTERSPEECH (2015)
Rami Botros, Kazuki Irie, Martin Sundermeyer, Hermann Ney
On efficient training of word classes and their application to recurrent neural network language models. INTERSPEECH (2015)
Simon Wiesler, Kazuki Irie, Zoltán Tüske, Ralf Schlüter, Hermann Ney
The RWTH English lecture recognition system. ICASSP (2014)