Authors
Jianping Gou, Liyuan Sun, Baosheng Yu, Philip H S Torr, Ming-Hsuan Yang, Dacheng Tao
Published in
IEEE transactions on pattern analysis and machine intelligence. Volume PP. Nov 06, 2026. Epub Nov 06, 2026.
Abstract
Transformers have gained widespread recognition across sectors and academic fields for their innovative self-attention mechanism, which effectively captures long-range dependencies, and for their scalability and exceptional performance across diverse applications. However, their high computational cost and time requirements limit their adoption in resource-constrained environments. Knowledge distillation provides a straightforward yet effective approach for compressing traditional network architectures, such as CNNs, by transferring knowledge from a large teacher model to a smaller student model. This paper explores the potential for model compression by combining novel Transformer architectures with knowledge distillation. Specifically, we categorize, summarize, and present the substantial body of work on Transformer knowledge and distillation schemes. Furthermore, this survey extends its coverage to widely used knowledge-distillation pipelines for large language models (LLMs) and systematically analyses six major distillation targets. We also review popular applications in various domains, including natural language processing, computer vision, and speech recognition. Finally, we provide forward-looking perspectives, highlighting potential directions for future research on developing highly efficient yet performance-competitive Transformer models through knowledge distillation techniques.
PMID:
42993647
Bibliographic data and abstract were imported from PubMed on 07 Nov 2026.
Read full publication at:
Please sign in
to see all details.
Advertisement
Stats
- Recommendations n/a n/a positive of 0 vote(s)
- Views 35
- Comments 0