Managing Uncertainty in Crowdsourcing with Interval-Valued Labels

Crowdsourcing has been an emerging machine learning paradigm. It collects labels from human crowds as inputs typically through the Internet. Due to limitations on knowledge, social-economic status, and other factors, participants may often have ambiguity in labeling some instances in practice. In this work, we propose interval-valued labels (IVLs), instead of commonly used binary-valued ones, to manage such kind of uncertainty in crowdsourcing. IVLs possess interval specific statistic and probabilistic properties. With them, this work presents an algorithm that is able to make an inference with a favorable matching probability as a main result. The algorithm also implies an index, which measures the overall uncertainty of collected IVLs quantitatively. Reported computational experiments further evidence that we may better manage uncertainty in crowdsourcing with IVLs than without.