Taste-Bench Exposes AI Agents’ Weak Long-Term Decision-Making

AI agents increasingly handle multi-step work such as research and software projects, where early choices can shape the final outcome. Taste-Bench focuses on the quality of those choices, not just whether a task is eventually completed. The benchmark defines this judgment as an agent’s “taste”: the ability to pick the better strategic path before the consequences are obvious.

The practical takeaway is clear: end-result benchmarks may miss whether an agent is making durable, efficient decisions along the way.

The researchers use recorded agent trajectories and identify “decision forks,” where two possible paths split from the same task context. They freeze the run at the fork, hide later work, and ask a model to choose between the two directions. Taste-Bench contains 502 questions built from coding and AI research tasks, including SWE-bench Pro and METR’s public transcript release MALT.

This setup turns hindsight into a scalable test: the final result labels which earlier path was better, while the evaluated model only sees the information available at the decision point.

The team tested 14 current models from the Claude, GPT, Grok, DeepSeek, GLM, MiniMax, and Mistral families. Each question was asked twice with the answer order flipped, and a model only received credit if it was correct both times. GPT-5.6 Sol led with 59.7 percent average accuracy, followed by GPT-5.5 at 59.5 percent, while pure guessing scores 25 percent under this evaluation.

The result suggests that even strong models have limited ability to judge which long-term path will work better when the task is still unfolding.

Performance falls sharply when the clue that reveals the better path appears later in the trajectory. When the key evidence is already visible, average accuracy across models reaches 62.3 percent; when it appears only after extensive later work, accuracy drops to 21 percent. The researchers also found that higher reasoning settings barely changed accuracy for two tested models.

For teams deploying agents, the warning is practical: letting a model “think longer” may not help if the information needed to choose correctly is not yet available.

The researchers also found that decision quality can be improved through training. Using Qwen3.6-27B as a base, they distilled a teacher model’s judgment into a student model, raising accuracy on unseen tasks from 30 to 47.9 percent. When the student’s judgment was added as advice for a separate agent, success on 41 held-out SWE-bench Pro tasks rose from 14.6 to 33.7 percent.

The broader implication is that long-term judgment can be measured and improved, making it a concrete target for future agent development rather than an invisible failure mode.