BrowseComp-Plus: retriever, reranker and agent baselines

Search Leaderboard

One fixed setting, with one part swapped at a time: the retriever behind the search tool, a reranker over it, or the agent model. Every row is one experiment file you can run with SkimSearchAgent.

Default agent Tongyi-DeepResearch-30B-A3B Questions 830 Corpus 100,195 docs Judge Qwen3-32B Updated –

How to run a row Experiment files

BrowseComp-Plus leaderboard SkimSearchAgent library

Default agent, no reranker
Default agent with a reranker
Another agent model
Pareto frontier

#	All	All	All
Loading…

Views

Each view changes one part

All

Every row on the board.

Fixed: nothing
Varies: agent, retriever, reranker

Retrievers

The retriever behind the search tool.

Fixed: Tongyi-DeepResearch-30B-A3B, no reranker
Varies: retriever

Rerankers

A reranker over the retriever's top 100.

Fixed: Tongyi-DeepResearch-30B-A3B, Qwen3-Embedding-0.6B
Varies: reranker

Agent backbones

The agent model that runs the search.

Fixed: Qwen3-Embedding-0.6B, no reranker, 100 turns
Varies: agent model

A reranker over another retriever, or an agent with another retriever, is a row in All only. In All, the menus in the Agent, Retriever and Reranker headers choose what the table includes.

Metrics

What each column means

Accuracy

The share of all 830 questions the Qwen3-32B judge marks correct, with the BrowseComp-Plus judge prompt. An answer without a verdict counts as wrong.

Recall

For each question, the share of its evidence documents (gold and evidence together) that any search result showed the agent, averaged over all 830 questions.

Search calls

The mean number of search calls per question.

Tokens

The mean tokens per question, each counted once: the largest prompt the episode held plus every token the model generated.

Rows rank by accuracy inside the view shown. Click a column header to sort by it instead. The chart plots the Accuracy and Tokens columns of the rows in the table.

Setting

What stays fixed

Budget: 100 turns and a 131,072-token context window, for every agent on the board.
Default agent: Tongyi-DeepResearch-30B-A3B served with vLLM, temperature 0.6, seed 42.
Other agents: Each runs with its own prompt family, sampling settings and tool-call format. An agent fine-tuned on its own tools keeps those tools and its prompt. The budget, the corpus and the retriever do not change.
Corpus: The original BrowseComp-Plus text: 100,195 documents, all 830 questions.
Strategy: Search-Visit, for every row. The agent searches, reads snippets, and opens the documents it picks. No row uses Sieve or another strategy. A fine-tuned agent does the same two steps through its own tool names.
Tools: search returns the top 10 documents with a 64-token snippet each. get_document reads a document, cut at 12,000 tokens.
Encoding: Every learned retriever encodes a document's first 512 tokens.
Rerankers: Each reorders the retriever's top 100. The agent sees the top 10.
Base retriever: Qwen3-Embedding-0.6B in the Rerankers and Agent backbones views.
Lists of queries: When the agent sends several queries in one call, the first one runs.

Run a row

Every row is one file

A row's file is configs/baselines/browsecomp_plus_<row>_tongyi.yaml. A backbone row's file is configs/baselines/browsecomp_plus_<retriever>_<backbone>.yaml. Build its index, serve the agent model, run the file and judge the answers. docs/BASELINES.md has every step and command.

shell

skimsearchagent run configs/baselines/browsecomp_plus_colbert_tongyi.yaml \
  model.backend=api model.api_base=http://127.0.0.1:8000/v1
# after the three judges: the row's accuracy, recall and search calls
python scripts/leaderboard_row.py <the run's results directory>