Swap. Run. Compare.
Deep-search agents over your own corpus. Swap the corpus, the retriever, the tools, the backbone or the evaluation. One agent loop, one run record and one evaluation layer stay fixed underneath, so every comparison is like for like.
- 10
- strategy families, from Search-Visit and Sieve to shell-based DCI, ITER's search and one-shot RAG
- 4
- backbone providers: OpenAI, Gemini, any OpenAI-compatible server, and vLLM in-process
- 40
- settings on the BrowseComp-Plus leaderboard: retrievers, rerankers and agent backbones
- 1
- run record that every strategy writes and one evaluation layer reads the same way
Retrievers, rerankers and agent backbones on BrowseComp-Plus
One fixed setting, with one part swapped per row: the retriever behind the search tool, a reranker over it, or the agent model. Every row runs Search-Visit with a 100-turn budget. Accuracy is the Qwen3-32B judge over all 830 questions. Tokens are per question, each token counted once.
Top retrieval settings agent: Tongyi-DeepResearch-30B-A3B
| # | retriever and reranker | acc. (%) | tokens |
|---|---|---|---|
| Loading… | |||
Top agent backbones retriever: Qwen3-Embedding-0.6B
| # | agent | acc. (%) | tokens | |
|---|---|---|---|---|
| Loading… | ||||
Six modules behind one set of contracts
One component changes at a time; the rest of the experiment stays fixed. Documents go in, a run record comes out, and the same evaluation layer reads every strategy's output the same way.
Corpus
Documents become units: an id, a title, a body, and optional sections. One dataset registry loads them; a corpus too large for memory is served from disk instead of held in RAM.
agent_search/corpus 02Retrieval and reranking
Every retriever indexes once and searches by query. Lucene BM25, dense encoders, BQL's fielded Boolean retrieval and Indri-style structured retrieval share one engine registry per corpus; a hybrid is any of them fused by one method, and a reranked retriever is one of them with a reranker over its pool.
agent_search/retrievers 03Tools, tasks, strategies
A tool is one atomic action with its own declaration and code, and it names the snippet its listing shows. A task is the goal and the answer protocol. A strategy combines tools with options and names its harness. This is where most extension work happens.
docs/EXTENDING.md 04Harness
How the model is put to work, one file each: ReAct, the default loop where the model picks each step; one-shot RAG; a plan-and-search team whose members are conditions run as agents. Every harness writes the same record, with every limit measured in tokens.
agent_search/harness 05Evaluation and evidence
Every run writes the same record: a full trajectory, its configuration, and its metrics. Metrics, judges and paired statistics all read that one record format, whichever strategy produced it.
agent_search/evaluation 06Training and rollouts
Every episode is a full trajectory, so it is also training data. Triples with tiered negatives come out of any run directory and train a dense retriever that plugs back in as any strategy's dense model.
docs/TRAINING.mdWhat ships with the library
Each part below is a registered name. Naming it in an experiment file is all it takes to use it.
- lexical retrievers
- BM25 on Lucene
- the Boolean method BQL
- Indri query language
- grep
- dense retrievers
- BGE
- Qwen3-Embedding (0.6B, 4B, 8B)
- RepLLaMA
- LRAT
- Agentic-R
- AgentIR
- the ITER checkpoints
- CodeRankEmbed
- learned sparse and multi-vector
- SPLADE++
- ColBERTv2
- Agent-ModernColBERT
- DiffRetriever
- rerankers
- bge-reranker-v2-m3
- Qwen3-Reranker
- monoT5
- the Laya decision models
- agent backbones
- Tongyi-DeepResearch-30B-A3B
- Qwen3.5 (4B, 9B, 27B)
- Qwen3.6-27B
- Qwen3.8-27B
- gpt-oss (20b, 120b)
- Qwen-AgentWorld-35B
- OpenResearcher-30B
- OpenSeeker-v2-30B
- MiroThinker-1.7-mini
- QUEST-35B
- judges
- the BrowseComp-Plus judge prompt with gpt-4o-mini or a served model
- DIVER's judge template
- exact match and F1
- datasets
- BrowseComp-Plus
- HotpotQA
- MuSiQue
- 2WikiMultihopQA
- InfoSeek
- SWE-bench Lite and Verified
- LocBench
The nouns
Four words cover everything you build on top of the library.
Tool
One atomic action the agent can call. A tool owns its declaration (the name and description the model sees), its code (a run function that returns text, never an exception), and, when it needs one, a manual rendered into the prompt.
Task
The goal and the answer protocol: the prompt template, the domain, and the terminal that ends an episode, an <answer>, a <fix>, or a patch.
Strategy
A named combination of tools with their options and the harness that runs them, or a retrieval-only floor.
agent_search/strategiesCondition
A task paired with a strategy. What a run names, what a config file's strategy and dataset keys resolve to, what the run record carries.
Built-in strategies by family
- search_visit
a search that lists documents, then visits the whole document
- search_visit
- search_visit_dense
- search_visit_hybrid
- search_visit_snippets
- autoread
one search that returns the full text of its hits
- autoread
- autoread_dense
- autoread_hybrid
- search_fetch
a search that lists structure, then fetches one section
- search_fetch
- search_fetch_dense
- search_fetch_hybrid
- search_fetch_bm25_plain
- search_fetch_dense_plain
- sieve
Boolean candidate filtering, one ranking model, result cards, section fetch
- sieve_bm25
- sieve
- sieve_dense
- sieve_nosnip
- sieve_plain
- sieve_v2
- sieve_visit
- sieve_visit_fused
- sieve_visit_dense
- indri
Indri-QL retrieval with cards and section fetch
- indri
- indri_plain
- indri_visit
- dci
shell commands over the exported corpus
- dci
- bounded_dci
- iter
ITER's search, the standard top ten or with already-listed documents dropped, then a document read by id
- iter_dense
- iter_bm25
- iter_reranked
- iter_splade
- iter_colbert
- iter_diffretriever
- dedup_dense
- dedup_bm25
- codefix
Boolean code search or grep, then read a function and commit a fix
- codefix
- codefix_grep
- rag
rank once, put the top documents in one prompt, one model call
- rag
- rag_dense
- rag_hybrid
- retrieval-only floors
rank once, no agent loop, no model
- bm25
- dense
- bql
- grep
- hybrid
Backbones are served through four providers: OpenAI's chat and reasoning models, Gemini, any OpenAI-compatible server, and vLLM running in-process. Every one of them emits the same tool-call text the loop parses, so swapping the backbone is one YAML key.
Bring your corpus, run in minutes
One experiment is one file. The first command runs a scripted policy against a three-document fixture with no API key and no GPU. The second names a real model. You can also call the library from Python with your own documents.
# install and run, no model, no GPU python -m pip install -e ".[api]" skimsearchagent run configs/fixtures/smoke_doc_fixture_sieve_bm25.yaml # the same run, with a real model export OPENAI_API_KEY=... skimsearchagent run configs/fixtures/doc_fixture_sieve_bm25_gpt4omini.yaml
# or call it from Python, with your own documents
from agent_search import research
docs = [{"_id": "d1", "title": "Treaty of Guadalupe Hidalgo",
"text": "The Treaty of Guadalupe Hidalgo ended the Mexican-American War in 1848."}]
result = research("Which treaty ended the Mexican-American War?", docs,
strategy="sieve_bm25", model="gpt-4o-mini")
print(result.answer, result.ranking, result.usage)
One tool, one strategy, one condition
Every extension point is a small class in one file. Here is a tool that looks documents up by title, a strategy that offers only that tool, and a condition that runs it. No edits anywhere else in the codebase.
# a tool, a strategy, a condition from agent_search.tools.base import Tool from agent_search.strategies.base import Strategy, register_strategy from agent_search.strategies.conditions import condition class TitleLookup(Tool): name = "title_lookup" description = "Documents whose title contains the words." parameters = {"type": "object", "properties": {"words": {"type": "string"}}, "required": ["words"]} def run(self, args): # self.units, self.ubyid, self.state, self.engine[...] hits = [u for u in self.units if all(w in (u.title or "").lower() for w in args["words"].lower().split())] self.state.seen.update(u.doc_id for u in hits) # first-seen order = the agent's ranking return "\n".join(f"{u.doc_id} {u.title}: {u.body}" for u in hits) or "no match" register_strategy(Strategy(name="title_only", description="look up by title", tools=(TitleLookup(name="title_lookup"),))) condition("title_agent", task="research", strategy="title_only") # run it: strategy=title_agent
Built on SkimSearchAgent
Two research projects are developed on the library. Each has its own page with its method, results and citation.
Sieve: Search, Inspect, Fetch
Boolean-filtered search for deep-research agents: fielded candidate selection, one ranking model, compact result cards with query-biased snippets, and reading one section instead of a whole document.
Sieve page arXiv 2608.27912, cs.IRITER: Interaction-Aware Retrieval
A dense retriever trained from search-agent trajectories. It conditions on the agent's earlier searches and returns documents the agent hasn't read yet.
ITER page