Integrating active learning with supervision for crowdsourcing generalization

Zhenyu Shu, Victor S. Sheng, Yang Zhang, Dianhong Wang, Jing Zhang, Heng Chen

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

4 Scopus citations

Abstract

With various online crowdsourcing platforms, it is easy to collect multiple labels for the same examples from the crowd. Consensus integration algorithms can infer the estimated ground truths from the multiple label sets of these crowdsourcing datasets. However, it couldn't be avoided that these integrated (estimated) labels still contain noises. In order to further improve the performance of a model learned from data with these integrated labels, we propose an active learning framework to further improve the data quality, such that to improve the model quality, through acquiring limited true labels from experts (the oracle). We further investigate two active learning strategies in terms of two uncertainty measures (i.e., CLUE and MUE) within the active learning framework. From our experimental results on eight simulation crowdsourcing datasets and four real-world crowdsourcing datasets with three popular consensus integration algorithms, we draw several conclusions as follows. (i) Our active learning framework with the input from the oracle significantly improves the generalization ability of the model learned from crowdsourcing data. (ii) Our two active learning strategies outperform a random active learning strategy.

Original languageEnglish
Title of host publicationProceedings - 2015 IEEE 14th International Conference on Machine Learning and Applications, ICMLA 2015
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages232-237
Number of pages6
ISBN (Electronic)9781509002870
DOIs
StatePublished - Mar 2 2016
EventIEEE 14th International Conference on Machine Learning and Applications, ICMLA 2015 - Miami, United States
Duration: Dec 9 2015Dec 11 2015

Publication series

NameProceedings - 2015 IEEE 14th International Conference on Machine Learning and Applications, ICMLA 2015

Conference

ConferenceIEEE 14th International Conference on Machine Learning and Applications, ICMLA 2015
CountryUnited States
CityMiami
Period12/9/1512/11/15

Keywords

  • Active learning
  • Crowdsourcing
  • Supervised classification

Fingerprint Dive into the research topics of 'Integrating active learning with supervision for crowdsourcing generalization'. Together they form a unique fingerprint.

Cite this