Knowledge Distillation in Artificial Intelligence

Knowledge distillation is a model-compression method in which a capable teacher model guides the training of a smaller student model. Instead of copying the teacher's parameters, the student learns to reproduce its outputs, internal representations, or relationships among examples, thereby acquiring part of its learned behaviour.

Core transfer mechanism

The teacher processes a transfer dataset and generates target outputs for the student. These outputs often contain more information than ordinary class labels.

• A temperature parameter can soften the teacher's output probabilities, revealing similarities among alternative classes that a single hard label conceals.
• The student is trained using a distillation loss that measures divergence between teacher and student outputs, often combined with supervised loss from true labels.
• After training, only the student is generally deployed, so the teacher's capabilities are approximated with lower computational requirements.

What knowledge can be distilled

Distillation transfers learned response patterns rather than a literal copy of the teacher. The method chosen depends on whether the trainer can access only outputs or also the teacher's internal layers.

• Response-based distillation matches predicted probabilities or logits and can operate with output access.
• Feature-based distillation aligns selected intermediate representations of the teacher and student.
• Relation-based distillation teaches relationships among samples or internal features, such as relative similarities.
• In self-distillation, a model or its earlier stages can provide training targets for another version of the same architecture.

Benefits and limits

A successful student can retain useful performance while requiring less memory, computation, energy, or inference time. This supports deployment on constrained devices and can reduce barriers to local adaptation of advanced AI systems.

• The student may not reproduce capabilities that exceed its capacity or are poorly represented in the transfer data.
• Distillation can reproduce the teacher's errors and biases, because these are embedded in its outputs.
• Output-only distillation may facilitate model imitation, creating concerns about intellectual property, access control, and strategic technology diffusion.