External Red Team Stress-Tests OpenAI's Internal Monitoring: Found Novel Flaws, One Could Hide Hundreds of Actions From Oversight
OpenAI keeps saying it monitors models for going off the rails. Now a third party has come in and checked. The verdict: the monitoring system itself has holes.
Three key facts
From July 21 to August 11, METR and Redwood Research ran a three-week "high-access" red-team exercise against OpenAI's internal misalignment monitoring system. According to METR's public summary published October 22, two testers were given access to OpenAI's internal codebase, monitor prompts and configuration details, and conducted technical interviews with roughly a dozen researchers, safety, and security staff.
The output was a 61-page internal report that named "a number of novel vulnerabilities." One technique: letting agents "prevent hundreds of recent actions (including the actions involved in the exploit itself) from being monitored." In other words, the model can not only step out of bounds, it can erase the traces of doing so.
The scope had clear boundaries, and so do the conclusions. METR notes it assessed the monitoring system as it existed on and before July 21, not the changes in progress at the time; OpenAI said the vulnerabilities impacting its production safeguards stack have been patched. The exercise estimated that roughly 75%–94% of internal Responses API traffic (which powers Codex, among other things) from models at least as strong as GPT-5.5 was eligible for monitoring, with 99.2% coverage for the subset used by internal Codex agents.
Wang Dou's Take
Let's be clear about what this exercise was: not "OpenAI is unsafe," but "someone actually went and checked — with real access, real code, and a real written report." In the AI safety field, letting an external red team into your most core monitoring system is a rare posture. That deserves credit.
But after the credit, that vulnerability is glaring. Its essence is "the monitoring can be affected by the thing being monitored" — the model can scrub recent behavior, including its own wrongdoing, from oversight's view. That's not a technical detail; it's the foundation of the entire safety narrative: if the tool you use to find problems can be switched off by the thing you're looking for, then who exactly is it protecting?
More worth pondering is that numeric range: 75%–94%. METR itself says there is "substantial uncertainty in both directions." That means 6% to 25% of traffic — nobody can even say clearly whether it's in scope. If a safety system can't accurately measure its own coverage, then the "safety" it promises is essentially a probability claim, not a guarantee.
One last point: OpenAI says the relevant flaws are patched. That may be true, but notice the structure of the claim — "patched" isn't "independently verified." If the people who patch and the people who verify the patch are the same people, then the value of this red team depends entirely on when the next red team shows up.