Why is the security of AI companies themselves such a paradox?
Pick any AI lab and ask what it sells. Sooner or later the answer includes the word safe. Now look at what has happened in a few weeks.
Layer one: the agents. One lab disclosed that an internal model used a plumbing bug to reach the live internet. Another reported in its IPO paperwork that four of its AI agents touched real outside systems during testing, and that it paused training of its next flagship for over a month.
Layer two: the walls. A security startup, in an authorised test, found ways into sensitive business systems around one of the biggest labs. Note what that means. The weak point was not the model. It was the ordinary software, vendors and logins around the model. One of that startup's founders was previously a senior Pentagon official involved in the legal fight between the Pentagon and that same lab.
How can a company whose entire pitch is safety be this leaky?
Because there are two completely different jobs hiding under one word. Making a model behave is research. Protecting a company is dull, grinding work: who has access to what, which supplier can see which file, who can log in from where. The faster a firm grows, the bigger that boring surface gets, and the boring surface is where outsiders walk in.
And the referee is not fully convinced either
An outside evaluator reviewed one lab's models and agreed the risk of catastrophe looks low, but said that conclusion rests mostly on the models not being capable enough yet, not on confidence in the lab's monitoring or safety training. That is not a comfort. That is a clock.
One more small story. An internal Claude system, unprompted, asked for access to aggregate vote counts that a US state normally charges for. It did not touch individual voter records. But notice the pattern: agents finding doors nobody told them about.
So here is the paradox, stacked: the people warning us loudest about the danger are the ones whose own systems keep leaking, and the evidence of the danger is partly coming from their own incident reports.