Download or read book Methods for Evaluating Interactive Information Retrieval Systems with Users written by Diane Kelly and published by Now Publishers Inc. This book was released on 2009 with total page 246 pages. Available in PDF, EPUB and Kindle. Book excerpt: Provides an overview and instruction on the evaluation of interactive information retrieval systems with users.
Download or read book Development of IR Evaluation Methods written by Stephen Edward Robertson and published by . This book was released on 1999 with total page 134 pages. Available in PDF, EPUB and Kindle. Book excerpt:
Download or read book Introduction to Information Retrieval written by Christopher D. Manning and published by Cambridge University Press. This book was released on 2008-07-07 with total page pages. Available in PDF, EPUB and Kindle. Book excerpt: Class-tested and coherent, this textbook teaches classical and web information retrieval, including web search and the related areas of text classification and text clustering from basic concepts. It gives an up-to-date treatment of all aspects of the design and implementation of systems for gathering, indexing, and searching documents; methods for evaluating systems; and an introduction to the use of machine learning methods on text collections. All the important ideas are explained using examples and figures, making it perfect for introductory courses in information retrieval for advanced undergraduates and graduate students in computer science. Based on feedback from extensive classroom experience, the book has been carefully structured in order to make teaching more natural and effective. Slides and additional exercises (with solutions for lecturers) are also available through the book's supporting website to help course instructors prepare their lectures.
Download or read book Information Retrieval Evaluation in a Changing World written by Nicola Ferro and published by Springer. This book was released on 2019-08-13 with total page 595 pages. Available in PDF, EPUB and Kindle. Book excerpt: This volume celebrates the twentieth anniversary of CLEF - the Cross-Language Evaluation Forum for the first ten years, and the Conference and Labs of the Evaluation Forum since – and traces its evolution over these first two decades. CLEF’s main mission is to promote research, innovation and development of information retrieval (IR) systems by anticipating trends in information management in order to stimulate advances in the field of IR system experimentation and evaluation. The book is divided into six parts. Parts I and II provide background and context, with the first part explaining what is meant by experimental evaluation and the underlying theory, and describing how this has been interpreted in CLEF and in other internationally recognized evaluation initiatives. Part II presents research architectures and infrastructures that have been developed to manage experimental data and to provide evaluation services in CLEF and elsewhere. Parts III, IV and V represent the core of the book, presenting some of the most significant evaluation activities in CLEF, ranging from the early multilingual text processing exercises to the later, more sophisticated experiments on multimodal collections in diverse genres and media. In all cases, the focus is not only on describing “what has been achieved”, but above all on “what has been learnt”. The final part examines the impact CLEF has had on the research world and discusses current and future challenges, both academic and industrial, including the relevance of IR benchmarking in industrial settings. Mainly intended for researchers in academia and industry, it also offers useful insights and tips for practitioners in industry working on the evaluation and performance issues of IR tools, and graduate students specializing in information retrieval.
Download or read book Test Collection Based Evaluation of Information Retrieval Systems written by Mark Sanderson and published by Now Publishers Inc. This book was released on 2010-06-03 with total page 143 pages. Available in PDF, EPUB and Kindle. Book excerpt: Use of test collections and evaluation measures to assess the effectiveness of information retrieval systems has its origins in work dating back to the early 1950s. Across the nearly 60 years since that work started, use of test collections is a de facto standard of evaluation. This monograph surveys the research conducted and explains the methods and measures devised for evaluation of retrieval systems, including a detailed look at the use of statistical significance testing in retrieval experimentation. This monograph reviews more recent examinations of the validity of the test collection approach and evaluation measures as well as outlining trends in current research exploiting query logs and live labs. At its core, the modern-day test collection is little different from the structures that the pioneering researchers in the 1950s and 1960s conceived of. This tutorial and review shows that despite its age, this long-standing evaluation method is still a highly valued tool for retrieval research.
Download or read book Interactive IR User Study Design Evaluation and Reporting written by Jiqun Liu and published by Springer Nature. This book was released on 2022-05-31 with total page 75 pages. Available in PDF, EPUB and Kindle. Book excerpt: Since user study design has been widely applied in search interactions and information retrieval (IR) systems evaluation studies, a deep reflection and meta-evaluation of interactive IR (IIR) user studies is critical for sharpening the instruments of IIR research and improving the reliability and validity of the conclusions drawn from IIR user studies. To this end, we developed a faceted framework for supporting user study design, reporting, and evaluation based on a systematic review of the state-of-the-art IIR research papers recently published in several top IR venues (n=462). Within the framework, we identify three major types of research focuses, extract and summarize facet values from specific cases, and highlight the under-reported user study components which may significantly affect the results of research. Then, we employ the faceted framework in evaluating a series of IIR user studies against their respective research questions and explain the roles and impacts of the underlying connections and "collaborations" among different facet values. Through bridging diverse combinations of facet values with the study design decisions made for addressing research problems, the faceted framework can shed light on IIR user study design, reporting, and evaluation practices and help students and young researchers design and assess their own studies.
Download or read book Online Evaluation for Information Retrieval written by Katja Hofmann and published by . This book was released on 2016-06-07 with total page 134 pages. Available in PDF, EPUB and Kindle. Book excerpt: Provides a comprehensive overview of the topic. It shows how online evaluation is used for controlled experiments, segmenting them into experiment designs that allow absolute or relative quality assessments. It also includes an extensive discussion of recent work on data re-use, and experiment estimation based on historical data.
Download or read book Information Retrieval written by Ayse Goker and published by John Wiley & Sons. This book was released on 2009-12-15 with total page 320 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book is an essential reference to cutting-edge issues and future directions in information retrieval Information retrieval (IR) can be defined as the process of representing, managing, searching, retrieving, and presenting information. Good IR involves understanding information needs and interests, developing an effective search technique, system, presentation, distribution and delivery. The increased use of the Web and wider availability of information in this environment led to the development of Web search engines. This change has brought fresh challenges to a wider variety of users’ needs, tasks, and types of information. Today, search engines are seen in enterprises, on laptops, in individual websites, in library catalogues, and elsewhere. Information Retrieval: Searching in the 21st Century focuses on core concepts, and current trends in the field. This book focuses on: Information Retrieval Models User-centred Evaluation of Information Retrieval Systems Multimedia Resource Discovery Image Users’ Needs and Searching Behaviour Web Information Retrieval Mobile Search Context and Information Retrieval Text Categorisation and Genre in Information Retrieval Semantic Search The Role of Natural Language Processing in Information Retrieval: Search for Meaning and Structure Cross-language Information Retrieval Performance Issues in Parallel Computing for Information Retrieval This book is an invaluable reference for graduate students on IR courses or courses in related disciplines (e.g. computer science, information science, human-computer interaction, and knowledge management), academic and industrial researchers, and industrial personnel tracking information search technology developments to understand the business implications. Intermediate-advanced level undergraduate students on IR or related courses will also find this text insightful. Chapters are supplemented with exercises to stimulate further thinking.
Download or read book Information Retrieval written by William Hersh and published by Springer Science & Business Media. This book was released on 2006-05-04 with total page 524 pages. Available in PDF, EPUB and Kindle. Book excerpt: Coupled with the growth of the World Wide Web, the topic of health information retrieval has had a tremendous impact on consumer health information. With the aid of newly added questions and discussions at the end of each chapter, this Second Edition covers theory practical applications, evaluation, and research directions of all aspects of medical information retireval systems.
Download or read book Information Retrieval Uncertainty and Logics written by Fabio Crestani and published by Springer Science & Business Media. This book was released on 1998-10-31 with total page 362 pages. Available in PDF, EPUB and Kindle. Book excerpt: A collection of papers proposing, developing, and implementing logical IR models. After an introductory chapter on non-classical logic as the appropriate formalism with which to build IR models, papers are divided into groups on three approaches: logical models, uncertainty models, and meta-models. Topics include preferential models of query by navigation, a logic for multimedia information retrieval, logical imaging and probabilistic information retrieval, and an axiomatic aboutness theory for information retrieval. Can be used as a text for a graduate course on information retrieval or database systems, and as a reference for researchers and practitioners in industry. Annotation copyrighted by Book News, Inc., Portland, OR
Download or read book Information Retrieval Evaluation in a Changing World written by and published by . This book was released on 2019 with total page 597 pages. Available in PDF, EPUB and Kindle. Book excerpt: This volume celebrates the twentieth anniversary of CLEF - the Cross-Language Evaluation Forum for the first ten years, and the Conference and Labs of the Evaluation Forum since - and traces its evolution over these first two decades. CLEF's main mission is to promote research, innovation and development of information retrieval (IR) systems by anticipating trends in information management in order to stimulate advances in the field of IR system experimentation and evaluation. The book is divided into six parts. Parts I and II provide background and context, with the first part explaining what is meant by experimental evaluation and the underlying theory, and describing how this has been interpreted in CLEF and in other internationally recognized evaluation initiatives. Part II presents research architectures and infrastructures that have been developed to manage experimental data and to provide evaluation services in CLEF and elsewhere. Parts III, IV and V represent the core of the book, presenting some of the most significant evaluation activities in CLEF, ranging from the early multilingual text processing exercises to the later, more sophisticated experiments on multimodal collections in diverse genres and media. In all cases, the focus is not only on describing "what has been achieved", but above all on "what has been learnt". The final part examines the impact CLEF has had on the research world and discusses current and future challenges, both academic and industrial, including the relevance of IR benchmarking in industrial settings. Mainly intended for researchers in academia and industry, it also offers useful insights and tips for practitioners in industry working on the evaluation and performance issues of IR tools, and graduate students specializing in information retrieval.
Download or read book Experimental IR Meets Multilinguality Multimodality and Interaction written by Fabio Crestani and published by Springer Nature. This book was released on 2019-09-02 with total page 434 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book constitutes the refereed proceedings of the 10th International Conference of the CLEF Association, CLEF 2019, held in Lugano, Switzerland, in September 2019. The conference has a clear focus on experimental information retrieval with special attention to the challenges of multimodality, multilinguality, and interactive search ranging from unstructured to semi structures and structured data. The 7 full papers and 8 short papers presented in this volume were carefully reviewed and selected from 30 submissions. This year, many contributions tackle the social networks with the detection of stances or early identification of depression signs on Twitter in a cross-lingual context. Further this volume presents 7 “best of the labs” papers which were reviewed as a full paper submission with the same review criteria. The labs represented scientific challenges based on new data sets and real world problems in multimodal and multilingual information access. In addition to this, 9 benchmarking labs reported results of their yearlong activities in overview talks and lab sessions.
Download or read book Experimental IR Meets Multilinguality Multimodality and Interaction written by Lorraine Goeuriot and published by Springer Nature. This book was released on with total page 286 pages. Available in PDF, EPUB and Kindle. Book excerpt:
Download or read book Information Retrieval Interaction written by Peter Ingwersen and published by . This book was released on 1992 with total page 264 pages. Available in PDF, EPUB and Kindle. Book excerpt:
Download or read book A Collaborative Approach to IR Evaluation written by Sheshadri Aashish and published by . This book was released on 2014 with total page 124 pages. Available in PDF, EPUB and Kindle. Book excerpt: In this thesis we investigate two main problems: 1) inferring consensus from disparate inputs to improve quality of crowd contributed data; and 2) developing a reliable crowd-aided IR evaluation framework. With regard to the first contribution, while many statistical label aggregation methods have been proposed, little comparative benchmarking has occurred in the community making it difficult to determine the state-of-the-art in consensus or to quantify novelty and progress, leaving modern systems to adopt simple control strategies. To aid the progress of statistical consensus and make state-of-the-art methods accessible, we develop a benchmarking framework in SQUARE, an open source shared task framework including benchmark datasets, defined tasks, standard metrics, and reference implementations with empirical results for several popular methods. Through the development of SQUARE we propose a crowd simulation model that emulates real crowd environments to enable rapid and reliable experimentation of collaborative methods with different crowd contributions. We apply the findings of the benchmark to develop reliable crowd contributed test collections for IR evaluation. As our second contribution, we describe a collaborative model for distributing relevance judging tasks between trusted assessors and crowd judges. Based on prior work's hypothesis of judging disagreements on borderline documents, we train a logistic regression model to predict assessor disagreement, prioritizing judging tasks by expected disagreement. Judgments are generated from different crowd models and intelligently aggregated. Given a priority queue, a judging budget, and a ratio for expert vs. crowd judging costs, critical judging tasks are assigned to trusted assessors with the crowd supplying remaining judgments. Results on two TREC datasets show significant judging burden can be confidently shifted to the crowd, achieving high rank correlation and often at lower cost vs. exclusive use of trusted assessors.
Download or read book Learning to Rank for Information Retrieval written by Tie-Yan Liu and published by Springer Science & Business Media. This book was released on 2011-04-29 with total page 282 pages. Available in PDF, EPUB and Kindle. Book excerpt: Due to the fast growth of the Web and the difficulties in finding desired information, efficient and effective information retrieval systems have become more important than ever, and the search engine has become an essential tool for many people. The ranker, a central component in every search engine, is responsible for the matching between processed queries and indexed documents. Because of its central role, great attention has been paid to the research and development of ranking technologies. In addition, ranking is also pivotal for many other information retrieval applications, such as collaborative filtering, definition ranking, question answering, multimedia retrieval, text summarization, and online advertisement. Leveraging machine learning technologies in the ranking process has led to innovative and more effective ranking models, and eventually to a completely new research area called “learning to rank”. Liu first gives a comprehensive review of the major approaches to learning to rank. For each approach he presents the basic framework, with example algorithms, and he discusses its advantages and disadvantages. He continues with some recent advances in learning to rank that cannot be simply categorized into the three major approaches – these include relational ranking, query-dependent ranking, transfer ranking, and semisupervised ranking. His presentation is completed by several examples that apply these technologies to solve real information retrieval problems, and by theoretical discussions on guarantees for ranking performance. This book is written for researchers and graduate students in both information retrieval and machine learning. They will find here the only comprehensive description of the state of the art in a field that has driven the recent advances in search engine development.
Download or read book Language Modeling for Information Retrieval written by W. Bruce Croft and published by Springer Science & Business Media. This book was released on 2013-04-17 with total page 253 pages. Available in PDF, EPUB and Kindle. Book excerpt: A statisticallanguage model, or more simply a language model, is a prob abilistic mechanism for generating text. Such adefinition is general enough to include an endless variety of schemes. However, a distinction should be made between generative models, which can in principle be used to synthesize artificial text, and discriminative techniques to classify text into predefined cat egories. The first statisticallanguage modeler was Claude Shannon. In exploring the application of his newly founded theory of information to human language, Shannon considered language as a statistical source, and measured how weH simple n-gram models predicted or, equivalently, compressed natural text. To do this, he estimated the entropy of English through experiments with human subjects, and also estimated the cross-entropy of the n-gram models on natural 1 text. The ability of language models to be quantitatively evaluated in tbis way is one of their important virtues. Of course, estimating the true entropy of language is an elusive goal, aiming at many moving targets, since language is so varied and evolves so quickly. Yet fifty years after Shannon's study, language models remain, by all measures, far from the Shannon entropy liInit in terms of their predictive power. However, tbis has not kept them from being useful for a variety of text processing tasks, and moreover can be viewed as encouragement that there is still great room for improvement in statisticallanguage modeling.
