On the Volatility of Commercial Search Engines and its Impact on Information Retrieval Research

Jimmy, Guido Zuccon, and Gianluca Demartini

Appendix

A. Literature in the ACM SIGIR Conference between 2006 and 2016 that Used Commercial Search Engines

1. Review of MSR-Bing Web Scale Speller Challenge
2. Adaptive Query Suggestion for Difficult Queries
3. Manhattan Hashing for Large-Scale Image Retrieval
4. Explicit Relevance Models in Intent-Oriented Information Retrieval Diversification
5. AspecTiles: Tile-based Visualization of Diversified Web Search Results
6. Combining Inverted Indices and Structured Search for Ad-hoc Object Retrieval
7. Modeling the Impact of Short- and Long-Term Behavior on Search Personalization
8. Improving Searcher Models Using Mouse Cursor Activity
9. User Evaluation of Query Quality
10. Index Maintenance for Time-Travel Text Search
11. Studies of the Onset and Persistence of Medical Concerns in Search Logs
12. A Semi-Supervised Approach to Modeling Web Search Satisfaction
13. Image Ranking Based on User Browsing Behavior
14. What Reviews are Satisfactory: Novel Features for Automatic Helpfulness Voting
15. Prefetching Query Results and its Impact on Search Engines
16. Online Result Cache Invalidation for Real-time Web Search
17. Mining the Web for Points of Interest
18. When Web Search Fails, Searchers Become Askers: Understanding the Transition
19. Personalized Diversification of Search Results
20. An IR-based Evaluation Framework for Web Search Query Segmentation
21. Extending BM25 with Multiple Query Operators
22. Creating Temporally Dynamic Web Search Snippets
23. Learning-Based Time-Sensitive Re-Ranking for Web Search
24. On Judgments Obtained from a Commercial Search Engine
25. Survival Analysis of Click Logs
26. Text Selections as Implicit Relevance Feedback
27. Unity: Relevance Feedback using User Query Logs
28. Beliefs and Biases in Web Search
29. Utilizing Query Change for Session Search
30. Query Expansion Using Path-Constrained Random Walks
31. Personalized Ranking Model Adaptation for Web Search
32. Task-Aware Query Recommendation
33. An Effective Implicit Relevance Feedback Technique Using Affective, Physiological and Behavioural Features
34. Fighting Search Engine Amnesia: Reranking Repeated Results
35. News Vertical Search: When and What to Display to Users
36. Modeling Click-through Based Word-pairs for Web Search
37. Search Result Diversification in Resource Selection for Federated Search
38. Improving Search Result Summaries by Using Searcher Behavior Data
39. Timeline Generation with Social Attention
40. Displaying Relevance Scores for Search Results
41. Report from the NTCIR-10 1CLICK-2 Japanese Subtask: Baselines, Upperbounds and Evaluation Robustness
42. Commodity Query by Snapping
43. Explicit Feedback in Local Search Tasks
44. Building a Web Test Collection using Social Media
45. Hybrid Retrieval Approaches to Geospatial Music Recommendation
46. Recommending personalized touristic sights using Google Places
47. Relevance Dimensions in Preference-based IR Evaluation
48. From Keywords to Keyqueries: Content Descriptors for the Web
49. Online Matching of Web Content to Closed Captions in IntoNow
50. Rendering Expressions to Improve Accuracy of Relevance Assessment for Math Search
51. Examining Collaborative Query Reformulation: A Case of Travel Information Searching
52. Web Search without ’Stupid’ Results
53. Evaluating Mobile Web Search Performance by Taking Good Abandonment into Account
54. Analyzing the Content Emphasis of Web Search Engines
55. Effects of Task and Domain on Searcher Attention
56. To Hint or Not: Exploring the Effectiveness of Search Hints for Complex Informational Tasks
57. Personalized Query Expansion for the Web
58. A POMDP Model for Content-Free Document Re-ranking
59. On Correlation of Absence Time and Search Effectiveness
60. Multidimensional Relevance Modeling via Psychometrics and Crowdsourcing
61. Characterizing Multi-Click Search Behavior and the Risks and Opportunities of Changing Results during Use
62. User-Driven System-Mediated Collaborative Information Retrieval
63. Interactions between Health Searchers and Search Engines
64. Context-Aware Web Search Abandonment Prediction
65. Modeling Action-level Satisfaction for Search Task Satisfaction Prediction
66. Cohort Modeling for Enhanced Personalized Search
67. SearchPanel: Framing Complex Search Needs
68. Using Information Scent and Need for Cognition to Understand Online Search Behavior
69. Searching, Browsing, and Clicking in a Search Session: Changes in User Behavior by Task and Over Time
70. Injecting User Models and Time into Precision via Markov Chains
71. A Perspective-Aware Approach to Search: Visualizing Perspectives in News Search Results
72. Unconscious Physiological Effects of Search Latency on Users and Their Click Behaviour
73. Mining, Ranking and Recommending Entity Aspects
74. Personalizing Search on Shared Devices
75. Differences in the Use of Search Assistance for Tasks of Varying Complexity
76. Studying the Use of Popular Destinations to Enhance Web Search Interaction
77. Inferring Searcher Attention by Jointly Modeling User Interactions and Content Salience
78. Neighborhood Restrictions in Geographic IR
79. Exploring Opportunities to Facilitate Serendipity in Search
80. Cognitive Activity during Web Search
81. Search Engine Evaluation based on Search Engine Switching Prediction
82. Understanding Temporal Query Intent
83. Semantic Term Matching in Axiomatic Approaches to Information Retrieval
84. Sign-Aware Periodicity Metrics of User Engagement for Online Search Quality Evaluation
85. Multiple-Signal Duplicate Detection for Search Evaluation
86. Towards Understanding the Impact of Length in Web Search Result Summaries over a Speech-only Communication Channel
87. Anchoring and Adjustment in Relevance Estimation
88. Investigating the Querying and Browsing Behavior of Advanced Search Engine Users
89. Where the Event Lies: Predicting Event Occurrence in Textual Documents
90. InfoScout
91. Predicting User Engagement with Direct Displays Using Mouse Cursor Information
92. Search Result Prefetching Using Cursor Movement
93. Interesting Nuggets and Their Impact on Definitional Question Answering
94. A Probabilistic Graphical Model for Joint Answer Ranking in Question Answering
95. When a Knowledge Base Is Not Enough: Question Answering over Knowledge Bases with External Text Data
96. Learning for Efficient Supervised Query Expansion via Two-stage Feature Selection
97. Building Bridges for Web Query Classification
98. Evaluating Retrieval over Sessions: The TREC Session Track 2011–2014: The TREC Session Track 2011–2014
99. ArabicWeb16: A New Crawl for Today’s Arabic Web: A New Crawl for Today’s Arabic Web
100. On the Applicability of Delicious for Temporal Search on Web Archives
101. “Is Sven Seven?”: A Search Intent Module for Children
102. To Blend or Not to Blend? Perceptual Speed, Visual Memory and Aggregated Search
103. Topic Model based Privacy Protection in Personalized Web Search
104. A Music Search Engine Built upon Audio-based and Web-based Similarity Measures
105. Cross-Lingual Query Suggestion Using Query Logs of Different Languages
106. Principles of Hash-based Text Retrieval
107. How Well does Result Relevance Predict Session Satisfaction?
108. Improving the Estimation of Relevance Models Using Large External Corpora
109. ARSA: A Sentiment-Aware Model for Predicting Sales Performance Using Blogs
110. Web Text Retrieval with a P2P Query-Driven Index
111. An Effective Snippet Generation Method Using the Pseudo Relevance Feedback Technique
112. The Relationship between IR Effectiveness Measures and User Satisfaction
113. Varying Approaches to Topical Web Query Classification
114. Probabilistic Model for Definitional Question Answering
115. Investigating the Relevance of Sponsored Results for Web Ecommerce Queries
116. Query Suggestion Based on User Landing Pages
117. Towards Breaking the Quality Curse. A Web-Querying Approach to Web People Search
118. Enhancing Web Search by Promoting Multiple Search Engine Use
119. ResIn: A Combination of Results Caching and Index Pruning for High-performance Web Search Engines
120. User Adaptation: Good Results from Poor Systems
121. Retrieval and Feedback Models for Blog Feed Search
122. Classifiers Without Borders: Incorporating Fielded Text From Neighboring Web Pages
123. Emulating Query-Biased Summaries Using Document Titles
124. Task-aware Search Personalization
125. Searching Blogs and News: A Study on Popular Queries
126. SOPING: A Chinese Customer Review Mining System
127. Query Preprocessing: Improving Web Search Through a Vietnamese Word Tokenization Approach
128. WISA: A Novel Web Image Semantic Analysis System
129. One-button Search Extracts Wider Interests: An Empirical Study with Video Bookmarking Search
130. Mining Dependency Relations for Query Expansion in Passage Retrieval
131. Generating Diverse Katakana Variants Based on Phonemic Mapping
132. Less is More Probabilistic Models for Retrieving Fewer Relevant Documents
133. Predicting Searcher Frustration
134. Ready to Buy or Just Browsing? Detecting Web Searcher Goals from Interaction Data
135. Evaluating Verbose Query Processing Techniques
136. A Comparison of General vs Personalised Affective Models for the Prediction of Topical Relevance
137. Studying Trailfinding Algorithms for Enhanced Web Search
138. Visual Summarization of Web Pages
139. A User Behavior Model for Average Precision and its Generalization to Graded Judgments
140. Assessing the Scenic Route: Measuring the Value of Search Trails in Web Logs
141. Automatic Content Linking: Speech-based Just-in-time Retrieval for Multimedia Archives
142. Inferring User Intent in Web Search by Exploiting Social Annotations
143. Using Local Precision to Compare Search Engines in Consumer Health Information Retrieval
144. Web Page Publication Time Detection and its Application for Page Rank
145. Aspect Presence Verification Conditional on Other Aspects
146. Inferring and Using Location Metadata to Personalize Web Search
147. Active Learning to Maximize Accuracy vs. Effort in Interactive Information Retrieval
148. Why Searchers Switch: Understanding and Predicting Engine Switching Rationales
149. Find It If You Can: A Game for Modeling Different Types of Web Search Success Using Interaction Data
150. Measuring Improvement in User Search Performance Resulting from Optimal Search Tips
151. ViewSer: Enabling Large-Scale Remote User Studies of Web Search Examination and Interaction
152. Predicting Web Searcher Satisfaction with Existing Community-based Answers
153. UPS: Efficient Privacy Protection in Personalized Web Search
154. An Exploratory Web Log Study of Multitasking
155. Multi-Layer Graph-Based Semi-Supervised Learning for Large-Scale Image Datasets Using MapReduce
156. Location Disambiguation for Geo-tagged Images
157. How Diverse are Web Search Results?
158. Evolution of Web Search Results within Years

B. Volatility of Search Results: New URLs Retrieved

• Average New Documents from Start Date: WEB2013-2014
• Average New Documents from Start Date: CLEF2016
• Average New Documents from Previous Date: WEB2013-2014
• Average New Documents from Previous Date: CLEF2016
• Average New Documents All Date Pairs: WEB2013-2014
• Average New Documents All Date Pairs: CLEF2016

C. Volatility of Search Results: Rank Movement

• Average Rank Distance from Start Date: WEB2013-2014
• Average Rank Distance from Start Date: CLEF2016
• Average Rank Distance from Previous Date: WEB2013-2014
• Average Rank Distance from Previous Date: CLEF2016
• Average Rank Distance All Date Pairs: WEB2013-2014
• Average Rank Distance All Date Pairs: CLEF2016

D. Impact of Result Volatility on Search Effectiveness

WEB2013-2014: P@10

P@10 for WEB2013-2014 queries over time. Each box plot refers to a query; queries are ordered in decreasing effectiveness as returned on day 1. The red shaded area indicates the effectiveness gap between results for day 1 and those for the day with the biggest average effectiveness gap over the query set.

WEB2013-2014: ERR@10

ERR@10 for WEB2013-2014 queries over time. Each box plot refers to a query; queries are ordered in decreasing effectiveness as returned on day 1. The red shaded area indicates the effectiveness gap between results for day 1 and those for the day with the biggest average effectiveness gap over the query set.