AI Search Agents Don't Fail at Searching — They Fail at Asking the Right Questions

Researchers at Tencent Hunyuan and Tsinghua University built DiscoBench, a benchmark designed to test whether AI search agents can handle ambiguous queries in real-world multi-step research tasks. Unlike existing benchmarks such as GAIA or BrowseComp — which assume queries are complete and unambiguous — DiscoBench tests agents on the messy reality of vague, underspecified, or factually incorrect inputs. The benchmark contains 211 tasks with 463 ambiguous decision points spanning 11 domains including gaming, sports, music, film, science, and politics. At each checkpoint, an agent must choose between continuing to search, asking the user for clarification, or giving an answer.

The core finding: when a model picks the wrong entity early in a multi-step chain, it continues searching with syntactically clean queries but misses the actual target entirely. Every unresolved ambiguity compounds, derailing the full reasoning chain even when individual steps appear correct.

DiscoBench's behavioral analysis reveals a counterintuitive pattern: agents that search repeatedly but still guess instead of asking (SearchHeavyGuess) average only 51.9% success at ambiguous checkpoints — worse than agents that just guess outright (DirectGuess at 56.5%). The best strategy, SearchThenAsk, achieves a 93.4% checkpoint success rate, underscoring that the act of asking is the decisive skill. The repeated searches indicate models sense something is wrong but never translate that uncertainty into a user interaction.

This also explains why raw tool-call volume doesn't predict accuracy. Claude Opus 4.7 makes more search calls than most tested models but still trails Gemini 3.1 Pro Preview and Doubao Seed 2.0 Pro in end-to-end accuracy.

Eleven frontier models were tested without any explicit ambiguity hint. Doubao Seed 2.0 Pro led with 43.1% end-to-end accuracy, followed by Gemini 3.1 Pro Preview at 40.8% and Claude Opus 4.7 at 39.8%. Weaker models like Qwen3.6 Max (12.3%) and MiniMax M2.7 (16.1%) lagged far behind. Adding a guided system prompt telling agents to watch for ambiguity raised average end-to-end accuracy only modestly — from 28.6% to 33.7% — while detection F1 jumped more sharply from 45.3% to 64.9%, showing the hint helps agents spot ambiguity without reliably helping them resolve it.

Without access to search tools entirely, results collapsed: Doubao Seed 2.0 Pro fell from 43.1% to just 2.4%, confirming DiscoBench cannot be solved from stored knowledge alone. Conversely, when ambiguity was removed from queries, accuracy jumped by 26.8 to 40.2 points depending on the model — a clear ceiling showing how much performance is currently left on the table.

DiscoBench exposes a gap between recognizing ambiguity and acting on it effectively. Qwen3.6 Max detects ambiguity only 16% of the time and asks just 0.07 follow-up questions per task — but when it does ask, 94.7% of its questions are factually correct and 89.5% drive progress. MiniMax M2.7 asks far more often but achieves a follow-through rate of only 60.7–66.5%. A useful research agent needs both: knowing when to ask and framing the question so the answer actually advances the search.

Factual errors are the easiest ambiguity type to catch because they create direct contradictions during research. Entity and criteria ambiguities are harder — multiple plausible candidates or unclear evaluation standards can coexist without any obvious signal that something is wrong.

DiscoBench fits into a growing body of evidence that current AI search agents have systematic weaknesses beyond raw retrieval. Research using LiveBrowseComp — a benchmark testing facts beyond model knowledge cutoffs — found all leading systems dropped 25–40 points versus standard benchmarks. The Halluhard benchmark showed Claude Opus 4.5 with web search hallucinates roughly 30% of the time, mainly when verifying cited sources. The DiscoBench authors conclude that future search agents need mechanisms that convert search uncertainty into structured user interaction — on top of their existing retrieval and reasoning capabilities.

Anthropic's Claude Opus 4.8 update attempts to address related issues by flagging uncertainties more proactively. Perplexity's Search as Code approach takes a different angle, letting models write their own search workflows as Python programs rather than calling a fixed API.