A simple ensemble learning knowledge distillation

Himel Das Gupta, Kun Zhang, Victor S. Sheng

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review


Deep neural network (DNN) has shown significant improvement in learning and generalizing different machine learning tasks over the years. But it comes with an expense of heavy computational power and memory requirements. We can see that machine learning applications are even running in portable devices like mobiles and embedded systems nowadays, which generally have limited resources regarding computational power and memory and thus can only run small machine learning models. However, smaller networks usually do not perform very well. In this paper, we have implemented a simple ensemble learning based knowledge distillation network to improve the accuracy of such small models. Our experimental results prove that the performance enhancement of smaller models can be achieved through distilling knowledge from a combination of small models rather than using a cumbersome model for the knowledge transfer. Besides, the ensemble knowledge distillation network is simpler, time-efficient, and easy to implement.

Original languageEnglish
Title of host publicationMachine Learning and Artificial Intelligence - Proceedings of MLIS 2020
EditorsAntonio J. Tallon-Ballesteros, Chi-Hua Chen
PublisherIOS Press BV
Number of pages7
ISBN (Electronic)9781643681368
StatePublished - Dec 2 2020
Event2020 International Conference on Machine Learning and Intelligent Systems, MLIS 2020 - Virtual, Online, Korea, Republic of
Duration: Oct 25 2020Oct 28 2020

Publication series

NameFrontiers in Artificial Intelligence and Applications
ISSN (Print)0922-6389


Conference2020 International Conference on Machine Learning and Intelligent Systems, MLIS 2020
Country/TerritoryKorea, Republic of
CityVirtual, Online


  • Bagging
  • Ensemble
  • Knowledge distillation


Dive into the research topics of 'A simple ensemble learning knowledge distillation'. Together they form a unique fingerprint.

Cite this