Imbalanced multiple noisy labeling

Jing Zhang, Xindong Wu, Victor S. Sheng

Research output: Contribution to journalArticlepeer-review

61 Scopus citations


It can be easy to collect multiple noisy labels for the same object via Internet-based crowdsourcing systems. Labelers may have bias when labeling, due to lacking expertise, dedication, and personal preference. These cause Imbalanced Multiple Noisy Labeling. In most cases, we have no information about the labeling qualities of labelers and the underlying class distributions. It is important to design agnostic solutions to utilize these noisy labels for supervised learning. We first investigate how imbalanced multiple noisy labeling affects the class distributions of training sets and the performance of classification. Then, an agnostic algorithm Positive LAbel frequency Threshold (PLAT) is proposed to deal with the imbalanced labeling issue. Simulations on eight UCI data sets with different underlying class distributions show that PLAT not only effectively deals with the imbalanced multiple noisy labeling problems that off-the-shelf agnostic methods cannot cope with, but also performs nearly the same as majority voting under the circumstances without imbalance. We also apply PLAT to eight real-world data sets with imbalanced labels collected from Amazon Mechanical Turk, and the experimental results show that PLAT is efficient and better than other ground truth inference algorithms.

Original languageEnglish
Article number6823124
Pages (from-to)489-503
Number of pages15
JournalIEEE Transactions on Knowledge and Data Engineering
Issue number2
StatePublished - Feb 1 2015


  • Imbalanced noisy labeling
  • crowdsourcing
  • imbalanced learning
  • repeated labeling


Dive into the research topics of 'Imbalanced multiple noisy labeling'. Together they form a unique fingerprint.

Cite this