These are my personal (biased by interest and jet lag) highlights of the oral paper presentations:
- Tetsuya Sakai – Designing Test Collections for Comparing many Systems. The author presents a statistical analysis of the question how many topics (e.g., queries) an experimental collection should comprise in order to give a reliable performance estimate for a fixed number of participating systems/methods as well as a required target confidence level. The insights of their research are made available in the form of conveniently applicable spreadsheets.
- Anne Schuth et al. – Multileaved Comparisons for Fast Online Evaluation. The authors introduce a modification to the established team-draft and optimized interleafing methods for online evaluation. While previous evaluation schemes allowed only for direct evaluation of 2 systems at a time, the presented modification efficiently yields direct comparisons for arbitrary numbers of systems in a performance preserving manner.
- Nikita Spirin et al. – Large Scale Analysis of People Search in an Online Social Network. The authors present a broad analysis of how the community makes use of the a social networking platform’s graph-based people search feature. The study is based on the proprietary log files of Facebook.
- Julia Kiseleva et al. – Modelling and Detecting Changes in User Satisfaction. The authors discuss the phenomenon of “concept drift” on dynamic Web Search Engines. SERPs to popular queries sometimes experience drastic changes in quality when late-breaking news or previously unseen facets of the overall topic are not included due to negative reinforcement by the user community (the filter bubble). The authors show how this effect can be detected and how appropriate diversification measures can help. This study is based on the proprietary log files of Microsoft Bing.
- Philip McParlane et al. – Picture the scene…, Visually Summarizing Social Media Events. The authors present a topicality and diversity aware summarization scheme for creating comprehensive visual digests of social media collections. An especially interesting aspect to this paper is the detailed discussion of the various data cleaning efforts necessary to embark on this task.