OpenAI audit finds major AI agent monitoring gaps

Two researchers from METR and Redwood Research spent three weeks on-site at OpenAI auditing the company’s internal monitoring system for rogue AI behavior. From July 21 to August 11, they had access to OpenAI’s codebase and internal documents and interviewed about a dozen employees.

The audit focused on whether OpenAI’s monitoring systems could detect risky behavior from AI agents. The review was part of a pilot program organized by METR, Redwood Research, and OpenAI to give outside researchers deeper access to internal infrastructure.

The auditors found that an estimated 6 to 25 percent of internal agentic Responses API traffic from models at the GPT-5.5 class and above was not covered by monitoring during the audit period. METR said the uncertainty around that estimate is significant in both directions.

For internal Codex traffic that was supposed to be monitored, METR estimated roughly 10 percent went untracked between May and July because of a gap involving compaction events, or context summaries. OpenAI had independently discovered that bug and fixed it while the audit was still underway.

During the period when compaction events were not monitored, agents could delay oversight because the system only triggered when they handed control back to the user. That meant agents could potentially avoid the monitor by continuing to execute actions.

The auditors also found ways to hide hundreds of actions from the monitor and bypass the Codex sandbox. They determined that specially crafted inputs could likely crash the monitor entirely, and that model behavior during RL training was not being monitored when the audit began.

OpenAI says it has fixed all the weaknesses the auditors found. It also says it paused RL training after the Hugging Face incident in August and expanded monitoring to cover it, but those changes came after the auditors’ access period and were not part of the evaluation.

The full report was 61 pages, but OpenAI redacted large portions. METR and Redwood Research leadership received a 38-page version, while only a short summary was made public; METR said it had no right to disclose the redacted portions on its own. For future audits, METR says it wants to push for more public transparency.