Crowdsourced Label Aggregation Using Bilayer Collaborative Clustering

Jing Zhang, Victor S. Sheng, Jian Wu

Research output: Contribution to journalArticlepeer-review

23 Scopus citations


With online crowdsourcing platforms, labels can be acquired at relatively low costs from massive nonexpert workers. To improve the quality of labels obtained from these imperfect crowdsourced workers, we usually let different workers provide labels for the same instance. Then, the true labels for all instances are estimated from these multiple noisy labels. This traditional general-purpose label aggregation process, solely relying on the collected noisy labels, cannot significantly improve the accuracy of integrated labels under a low labeling quality circumstance. This paper proposes a novel bilayer collaborative clustering (BLCC) method for the label aggregation in crowdsourcing. BLCC first generates the conceptual-level features for the instances from their multiple noisy labels and infers the initially integrated labels by performing clustering on the conceptual-level features. Then, it performs another clustering on the physical-level features to form the estimations of the true labels on the physical layer. The clustering results on both layers can facilitate in tracking the changes in the uncertainties of the instances. Finally, the initially integrated labels that are likely to be wrongly inferred on the conceptual layer can be addressed using the estimated labels on the physical layer. The clustering processes on both layers can keep providing guidance information for each other in the multiple label remedy rounds. The experimental results on 12 real-world crowdsourcing data sets show that the performance of the proposed method in terms of accuracy is better than that of the state-of-The-Art methods.

Original languageEnglish
Article number8626164
Pages (from-to)3172-3185
Number of pages14
JournalIEEE Transactions on Neural Networks and Learning Systems
Issue number10
StatePublished - Oct 2019


  • Clustering
  • crowdsourcing
  • label aggregation
  • label noise handling
  • truth inference


Dive into the research topics of 'Crowdsourced Label Aggregation Using Bilayer Collaborative Clustering'. Together they form a unique fingerprint.

Cite this