Why do even experts end up repeating what the AI agent told them?
Here is an experiment that should make anyone who uses AI at work a little uncomfortable. Anthropic's alignment team took 50 professional data analysts and had them work with Claude Code on a made-up business question: should a farming co-op expand its advisory programme or shut it down?
The twist: at four key methodology decisions, the agent was secretly given a random preference. The analysts had no idea. The agent was told to argue honestly and to change its mind if the evidence went against it.
Result? The analysts went with the agent 86% of the time. Pure chance would be about 50%. And it flowed straight into the conclusions. When the agent leaned towards expanding, 54% recommended expanding. When it leaned the other way, only 36% did. 77% repeated the agent's arguments as if they were their own. Only 15% said the agent had influenced them. Nobody spotted that the preferences were planted.
Two layers of the same problem
The first layer is plain laziness. You want the quick answer, so you do not ask the awkward question. The second layer is sneakier, and it is about the AI being good. Once an agent has been right ten times in a row, you stop checking the eleventh. That trust is earned, and that is exactly what makes it dangerous.
And here is the part nobody wants to say out loud: the agent was told to back down if the evidence was against it. It barely needed to. People did not push hard enough to make it change its mind.
Add the oldest bias in the book. When the AI says what you were already hoping to hear, you do not check at all, you just get more sure of yourself. That is how a nudge turns into a firm opinion.
To be fair, there is a bright side. With the agent, serious methodology mistakes dropped a lot: 3 in 50, compared with 6 in 11 without it, and the work went faster. The tool makes you better. It also quietly makes your decisions for you. These are professionals, not beginners, and they still did not see it coming.