Improving label quality in crowdsourcing using noise correction

Jing Zhang, Victor S. Sheng, Jian Wu, Xiaoqin Fu, Xindong Wu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

10 Scopus citations

Abstract

This paper proposes a novel framework that introduces noise correction techniques to further improve label quality after ground truth inference in crowdsourcing. In the framework, an adaptive voting noise correction algorithm (AVNC) is proposed to identify and correct the most likely noises with the help of estimated qualities of labelers provided by the ground truth inference. The experimental results on two real-world datasets show that (1) the framework can improve label quality regardless of inference algorithms, especially under the circumstance that each example has a few noisy labels; and (2) since the algorithm AVNC considers both the number of and the probability of potential noises, it outperforms a baseline noise correction algorithm.

Original languageEnglish
Title of host publicationCIKM 2015 - Proceedings of the 24th ACM International Conference on Information and Knowledge Management
PublisherAssociation for Computing Machinery
Pages1931-1934
Number of pages4
ISBN (Electronic)9781450337946
DOIs
StatePublished - Oct 17 2015
Event24th ACM International Conference on Information and Knowledge Management, CIKM 2015 - Melbourne, Australia
Duration: Oct 19 2015Oct 23 2015

Publication series

NameInternational Conference on Information and Knowledge Management, Proceedings
Volume19-23-Oct-2015

Conference

Conference24th ACM International Conference on Information and Knowledge Management, CIKM 2015
CountryAustralia
CityMelbourne
Period10/19/1510/23/15

Keywords

  • Crowdsourcing
  • Inference
  • Label quality
  • Noise correction

Fingerprint Dive into the research topics of 'Improving label quality in crowdsourcing using noise correction'. Together they form a unique fingerprint.

  • Cite this

    Zhang, J., Sheng, V. S., Wu, J., Fu, X., & Wu, X. (2015). Improving label quality in crowdsourcing using noise correction. In CIKM 2015 - Proceedings of the 24th ACM International Conference on Information and Knowledge Management (pp. 1931-1934). (International Conference on Information and Knowledge Management, Proceedings; Vol. 19-23-Oct-2015). Association for Computing Machinery. https://doi.org/10.1145/2806416.2806627