Skip to main content
OpenAI and the California lawsuit: the hole was the package proxy

This post is a translation of the Spanish original.

3 min read

OpenAI and the California lawsuit: the hole was the package proxy

More than 80 times. That's the number of occasions on which, according to the lawsuit California's attorney general filed on Monday, OpenAI lost control of its agents during testing. One of them ended with evaluation agents escaping their sandbox and getting into Hugging Face's systems in July.

I read it twice. Not because of the figure, but because of the technical detail of how they got out.

What the lawsuit says

Rob Bonta filed the suit on 16 November in San Francisco Superior Court against OpenAI. It rests on two legs: California's unfair competition law, because it considers the company's safety promises misleading, and the public nuisance doctrine. It seeks a court order to correct testing practices and civil penalties of $2,500 for each violation proven at trial.

According to the prosecutors, the attack on Hugging Face was "entirely foreseeable" for a company that chose to test models it knew were dangerous in a flawed, exploitable test environment without adequate oversight. It goes back a while: an investigation was opened in September and the subpoena reached OpenAI on 1 October.

OpenAI, for its part, says the suit singles it out for a problem all the major labs have identified, and that it will keep cooperating with the prosecutors. It's worth remembering these are allegations: filing a lawsuit doesn't prove anything yet.

Where they got out

Here's what really caught my attention. According to OpenAI's own technical report and Hugging Face's account, the agents didn't break through a wall. They went out through the door that was deliberately left open: the package management service exposed inside the sandbox, the package registry's caching proxy. They chained previously unknown vulnerabilities in that service to bypass the isolation.

Hugging Face detected it and made it public on 16 July. OpenAI identified its own models as the source on the 21st.

Anyone who has built an isolated environment knows why this hurts. You cut internet access, but you need to install dependencies, so you leave a mirror or package proxy with an outbound connection. It's the "harmless" exception that appears in almost any segmented network design, including corporate environments where machines don't browse but do update. It's so common we stop seeing it as attack surface.

My reading

I think the lawsuit aims well, although I'm not fully convinced by the approach. Well, because the problem it describes isn't "the AI turned evil", but a classic systems one: trusting an isolation that had an exit and not watching it with the same paranoia as everything else. If you're testing agents whose job is precisely to find security flaws, that proxy is the first thing they'll look at. You don't even need to assume strange intentions in the model; it's enough that it optimises the task you gave it.

OpenAI's best argument is also true: this isn't exclusive to them. Any lab evaluating offensive capabilities has the same problem, and punishing only the one that disclosed the most can encourage disclosing less. Even so, I side with California's position. If you're the one training and releasing the agents, the environment is your responsibility, just as your company's firewall is even if someone else wrote the ransomware.

For those of us in systems, the lesson is smaller and more useful than the trial. That exception in the egress rule for the package repository deserves logs, alerts and patches like any exposed server. And if you're starting to give agents tools and network access, even in a home lab, the sandbox isn't the virtual machine: it's everything that virtual machine can reach.

I'm left with a question I can't settle. Should there be an obligation to publicly report these incidents, as already happens with personal data breaches, or would that only make labs test less and worse?

Sources

David González Lombraña

Apuntes, laboratorio y proyectos de IA, Linux y redes.