Supervised Learning from Multiple Experts: Whom to Trust When Everyone Lies a Bit

We describe a probabilistic approach for supervised learning when we have multiple experts/annotators providing (possibly noisy) labels but no absolute gold standard. The proposed algorithm evaluates