So sánh phương pháp

	Chưng cất tri thức	Mixture of Experts
Lĩnh vực	Học sâu	Học sâu
Họ	Machine learning	Machine learning
Năm ra đời	2015	2017
Người khởi xướng	Hinton, G., Vinyals, O. & Dean, J.	Shazeer, N. et al.
Loại	Neural network compression (teacher–student)	Sparse neural network architecture (conditional computation)
Công trình gốc	Hinton, G., Vinyals, O. & Dean, J. (2015). Distilling the Knowledge in a Neural Network. NeurIPS Deep Learning Workshop.	Shazeer, N. et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR. arXiv:1701.06538
Tên gọi khác	Bilgi Damıtma (Knowledge Distillation), bilgi damıtma, teacher-student distillation, model distillation	Uzman Karışımı (Mixture of Experts — MoE), uzman karışımı, MoE, sparse mixture of experts
Liên quan	5	3
Tóm tắt	Knowledge Distillation is a model-compression technique, introduced by Geoffrey Hinton and colleagues in 2015, that trains a small student model using the soft-label outputs of a large teacher model. Distilled models such as DistilBERT and TinyBERT reach roughly 97% of the larger model's performance while running far faster.	Mixture of Experts (MoE) is a sparse neural-network architecture, introduced by Shazeer and colleagues in 2017 with the sparsely-gated MoE layer, in which only a subset of expert sub-networks is activated for each input. As seen in models such as Switch Transformer and Mixtral, it holds computation cost fixed even as the total parameter count grows.