Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers
Published in In the proceedings of Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026
Use Google Scholar for full citation
Recommended citation: Anrui Chen, Ruijun Huang, Xin Zhang, Fang Dong, Hengjie Cao, Zhendong Huang, Yifeng Yang, Mengyi Chen, Jixian Zhou, Mingzhi Dong, Yujiang Wang, Jinlong Hou, Qin Lv, Robert Dick, Yuan Cheng, Tun Lu, Fan Yang, Li Shang, "Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers." In the proceedings of Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.