Active Content-Based Crowdsourcing Task Selection

Crowdsourcing has long established itself as a viable alternative to corpus annotation by domain experts for tasks such as document relevance assessment. The crowdsourcing process traditionally relies on high degrees of label redundancy in order to mitigate the detrimental effects of individually noisy worker submissions. Such redundancy comes at the cost of increased label volume, and, subsequently, monetary requirements. In practice, especially as the size of datasets increases, this is undesirable.
In this paper, we focus on an alternate method that exploits document information instead, to infer relevance labels for unjudged documents. We present an active learning scheme for document selection that aims at maximising the overall relevance label prediction accuracy, for a given budget of available relevance judgements by exploiting system-wide estimates of label variance and mutual information. Our experiments are based on TREC 2011 Crowdsourcing Track data and show that our method is able to achieve state-of-the-art performance while requiring 17 – 25% less budget.

This paper has been accepted for presentation at the 25th ACM International Conference on Information and Knowledge Management (CIKM).

SIGIR 2016 – Pisa, Italy

My personal highlights among the full paper presentations:

As well as some promising short papers:

  • Tetsuya Sakai
    Two Sample T-tests for IR Evaluation: Student or Welch?
    The author contests the widely-accepted notion that a Welch t-test be unconditionally preferable over a two-sample Student test. The investigation concludes that, if sample sizes differ substantially, and if the larger sample has a substantially larger variance, Welch’s t-test may not be reliable.
  • Bevan Koopman et al.
    A Test Collection for Matching Patients to Clinical Trials
    . The authors annotated 60 existing TREC CDS patient descriptions in terms of their eligibility for participation in a wide range of publicly advertised clinical trials.
  • Sumit Sidana et al.
    Health Monitoring on Social Media over Time
    . Using a spatio-temporal topic modeling approach, the authors investigate which medical conditions people manifest on social media at different geographical locations as well as points in time (e.g., throughout the seasons).