The 20th edition of the Text REtrieval Conference (TREC) for the first time featured a dedicated crowdsourcing track. As a consequence of the various crowdsourcing experiments conducted for the PuppyIR project, we hypothesised that a game-based crowdsourcing approach could proof advantageous for the collection of relevance assessments. In order to explore the tradeoff between pay rate and entertainment, TU Delft participated in a joint effort with the University of Iowa and proposed the GeAnn annotation game. Within the course of one week, we were able to collect 10,000 relevance judgements for the assigned query-document pairs. Using GeAnn, we were able to finance the entire assessment process with as little as $ 3.74. This unmatched cost-efficiency compared very beneficially to other participants who typically spent between $30 and $70 to achieve comparable result quality. Especially with regard to final system evaluation with children, the results looked very encouraging.
Across the other tracks, numerous state of the art challenges to IR systems such as the retrieval of chemical compounds, micro blog posts or medical records from patient files. In dedicated planning sessions for TREC 2012, the refining of current tracks as well as the introduction of 2 entirely new ones was discussed.
The Contextual Suggestion Track aims to entertain people at a given spatio-temporal position and a known set of preferences. Rather than the varying queries of most traditional tracks, this new setting focuses entirely on the user and his specific context.
The Knowledge-Base Acceleration Track addresses the task of automatically curating information for online encyclopedias such as Wikipedia. The challenge is to identify previously uncovered and appropriate news stories that would make a valid extension to the current Wiki article.
Finally, in a TREC anniversary historical panel, an interesting (and occasionally smirk-inducing) review of 20 years of TREC was given.
CIKM 2011 in Glasgow, UK, just closed its gates. Over 700 delegates spent a great time between research and fresh Haggis in Scotland’s co-capital.
For all those who could not attend, here is a short list of my personal highlights:
- Yuanhua Lv et al. – Lower-Bounding Term Frequency Normalization (Best Student Paper)
The authors argue that term frequency normalization may not always be optimal. Especially long documents are shown to be too strongly penalized by a number of well-known bag-of-words-based retrieval models.
- Maryam Karimzadehgan et al. – Improving Retrieval Accuracy of Difficult Queries through Generalizing Negative Document Language Models
Using information of negative examples (e.g., skipped documents on previous result pages) can be used to reorder the result list. In this work, the authors create a generalized model to capture undesired query facets which are subsequently demoted in the ranking.
- Katja Hofmann et al. – A probabilistic Method for Inferring Preferences from Clicks
The authors present a probabilistic approach to interleaving ranked result lists created by different ranking schemes. Based on click-through information, a learning to rank method is applied to infer the interleaving probabilities.
- Jiafeng Guo et al. – Intent-Aware Query Similarity (Best Paper)
The authors employ auxiliary information sources (search result snippets & click-through information) in order to determine query intent. Subsequently, surface feature-based query similarity measures can be augmented by search intent.
- Barbara Poblete et al. – Do all Birds Tweet the Same? Characterizing Twitter Around the World.
An interesting large-scale analysis of tweeting behaviour according to language families and countries of origin.
- Sergio Duarte Torres et al. – What and How Children Search on the Web
The authors analyze query logs of Yahoo’s US users to determine differences in the querying behaviour of different user age groups. The findings give quantitative evidence to hypotheses from cognitive science and children’s psychology.
- Kevyn Collins-Thompson et al. – Personalizing Web Search Results by Reading Level
The authors conduct a large-scale reading level analysis of web search engine results and the related snippets. They find that easy-to-read snippets leading to more difficult documents result in significantly shorter dwell times as the user’s expectation may be dissatisfied.
- Dimitrios Lymberopoulos et al. – Location-Aware Click Prediction in Mobile Local Search
The authors analyze location-aware web searches originating from local devices. They find significant differences in the locality preferences of inhabitants of different US states. While some states are more primarily interested in local resources, others are willing to travel further. Adjusting for said preferences was able to improve ranking quality.
As the successor of SIREN//NL and STW.ICT, starting in 2011, the ICT.OPEN aims to serve as a major hub for interaction and exchange in the Dutch ICT research community. The even will be held on November 14th and 15th in Veldhoven.
I will present parts of the research done in PuppyIR in one of the oral sessions. In particular, I will address the task of content selection on social video sharing platforms such as YouTube. The task of determining which video to show to children of a given age is not an entirely straightforward one. It has been shown previously, that children are not reliably able to make these choices themselves. At the same time, parental supervision can not always be granted. We therefore propose an automatic suitability classification method for YouTube videos based on video meta data and complementary community information.
The European Summit on Networked and Electronic Media (NEM) each year brings together industry, academia and user parties of media-centered technologies, applications and visions to drive European collaboration and advance the general field of information science. This year’s edition of the summit was held in Turin, Italy. Andreas Lingnau and I presented the mission and current state of the PuppyIR project. In particular, we demonstrated a number of metaphorical search interfaces that guide the information search process along easily understandable, yet entertaining stories and tasks.
The 8th European Summer School on Information Retrieval (ESSIR) just closed its gates. This year’s edition was held in Koblenz, Germany and featured an interesting and well-balanced program taught by some of the leading minds in IR research and application.
- Hinrich Schütze – IR Foundations & Formal Models
- Vladimir Batagelj – Graph models and methods in IR
- Stefan Rüger – Multimedia IR
- Ricardo Baeza-Yates – Web Retrieval, IR in the Cloud
- Andreas Hotho – IR in Social Media
- Peter Ingwersen – Evaluation Methodology and Experimental Design
- Duc Thanh Tran – IR for Structured Data
This broad range of lectures was accompanied by a keynote address on future directions in IR by Nigel Shadbolt, two co-located workshops and a beautiful late summer in the Rhine-Mosel area.
During the past years, the Web culture has been growing more and more enticing, centering many services around social media and collaboratively shared content. The vast range of possible exploitations of such community platforms include viral marketing, collaborative tagging, recommendation or content creation. BooksOnline’11 aims to offer a forum for bringing together expertise from academia, industry and library professionals to facilitate exchange on research and application of social media in the field of digital libraries.
A special form of collaboration on the Web that has been emerging lately is crowdsourcing. It is being used with increasing frequency and volume in numerous research fields and aspires to become a dedicated field of research at the intersection of social and computer science. Crowdsourcing has been shown to be a valid means of addressing large-scale system evaluation, cold start and sparsity issues as well as facilitating usability studies.
BooksOnline’11 will encourage strong exploitation of the incentives and benefits of these major forms of massive on-line collaboration for digital libraries.
BooksOnline’11 will be co-located to the 20th ACM Conference on Information and Knowledge Management (CIKM) in Glasgow, Scotland on 24th October 2011
ECIR 2011, hosted in Dublin, Ireland, is over. It was a great week with lots of opportunities for scientific exchange, discussion and the occasional pint of fresh Guinness.
While there was a large number of interesting publications presented at the conference, this is a short list of my personal favourite talks:
- Stephen Robertson – On the Contribution of Topics to System Evaluation
An analysis of the usefulness of given topics for IR system evaluation. The conclusion is, that there does not seem to be a general property that makes a topic more or less discriminative in terms of system evaluation. An interesting example of a “negative results paper” constructing a strong case and being accepted for publication.
- Jaime Arguello et al. – A Methodology for Evaluating Aggregated Search Results (Best Student Paper)
A method for evaluation of search interfaces that aggregate results from multiple verticals based on pairwise comparison. User studies show a high correlation between the proposed method and holistic, page-wide human judgements.
- Jagadeesh Jagarlamudi et al. – Fractional Similarity: Cross-Lingual Feature Selection for Search
A method for improving foreign (as in non-English) search result ranking by gathering hints from equivalent rankings in more frequently-observed languages.
- Kamran Massoudi et al. – Incorporating Query Expansion and Quality Indicators in Searching Microblog Posts
The authors identify a range of features (general textual vs. micro blog-specific ones) to identify credible posts from which subsequent query expansion terms are extracted.
- Gabriel Dulac-Arnold et al. – Text Classification: A Sequential Reading Approach
The authors present a sequential reading problem of text classification. They propose using a probabilistic model to determine at which point the classification does not improve significantly by reading further lines of the document.
- Jinyoung Kim et al. – An Analysis of Time-Instability in Web Search Results
A nice survey of the frequency at which top-ranked result lists change for major search engines.
- Wolfgang Gatterbauer – Rules of Thumb for Information Acquisition from Large and Redundant Data
An overview of clever sampling heuristics to approximate exhaustive reads of textual collections.
- Elena Smirnova et al. – A User-Oriented Model for Expert Finding (Best Paper)
The authors introduce novel considerations into expert finding based on the searcher. An expert’s proximity to the searcher (with respect to a social network, organizational chart of an institution or a building’s office layout) as well as the relative expertise given the searcher are discussed.
PuppyIR presented their query assistance demonstrator for children. The open source code base of the demo can be obtained from Sourceforge.
Poster session at S3MR 2010 in Interlaken, Switzerland.