A threshold method for imbalanced multiple noisy labeling

Jing Zhang, Xindong Wu, Victor S. Sheng

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations

Abstract

Internet-based crowdsourcing systems can be viewed as a kind of loosely coupled social networks. With these systems, it is easy to collect multiple noisy labels for the same object when conducting annotation for supervised learning. Because non-expert labelers lack expertise and dedication, and have strong personal preference, they may have bias when labeling. These cause Imbalanced Multiple Noisy Labeling. In this paper, we propose an agnostic algorithm Positive LAbel frequency Threshold (PLAT) to deal with imbalanced labeling. Because of the dynamics of social networks, in most cases no information about the qualities of labelers and underlying class distributions can be acquired. PLAT does not require prior knowledge of the labeling qualities of labelers, the underlying class distributions, and the level of labeling imbalance. Simulations on eight real-world datasets with different underlying class distributions demonstrate that PLAT not only effectively deals with the imbalanced multiple noisy labeling that off-the-shelf agnostic methods cannot cope with, but also performs nearly the same as majority voting under the circumstances that labelers have no bias.

Original languageEnglish
Title of host publicationProceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2013
PublisherAssociation for Computing Machinery
Pages61-65
Number of pages5
ISBN (Print)9781450322409
DOIs
StatePublished - 2013
Event2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2013 - Niagara Falls, ON, Canada
Duration: Aug 25 2013Aug 28 2013

Publication series

NameProceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2013

Conference

Conference2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2013
CountryCanada
CityNiagara Falls, ON
Period08/25/1308/28/13

Keywords

  • Classification
  • Crowdsourcing
  • Imbalance labeling
  • Multiple noisy labels
  • Outsourcing

Fingerprint Dive into the research topics of 'A threshold method for imbalanced multiple noisy labeling'. Together they form a unique fingerprint.

  • Cite this

    Zhang, J., Wu, X., & Sheng, V. S. (2013). A threshold method for imbalanced multiple noisy labeling. In Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2013 (pp. 61-65). (Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2013). Association for Computing Machinery. https://doi.org/10.1145/2492517.2492640