In July 2025, a nonprofit research group called METR published the result nobody in the industry wanted to hear. In a randomized controlled trial — the same design used to test drugs, and a rarity in software research — experienced open-source developers were handed real tasks from their own repositories, with AI tools randomly allowed or forbidden. Going in, the developers predicted the AI would make them 24 percent faster. Instead it made them 19 percent slower — and, most damning of all, even after finishing they still believed the AI had sped them up. The gap between what they felt and what the stopwatch said was almost 40 percentage points.
That study became a Rorschach test. AI skeptics waved it around as proof the whole coding-assistant boom was a mass delusion. Boosters dismissed it as a fluke that would evaporate as the tools improved. As usual, the people treating it as a verdict were both missing the more interesting thing: what happened next.
The number that flipped
Here is the part that rarely makes the headlines, because it complicates a clean story. METR did not stop at the scary result. The organization has been careful to frame the finding as a snapshot of early-2025 tools rather than a permanent law of nature, and it kept measuring. When it ran the numbers again on later tools, the same study’s estimate flipped from a 19 percent slowdown to roughly an 18 percent speedup within about a year. The tools got better, and the developers got better at knowing when to reach for them and when not to.
But even METR would tell you not to over-read the reversal. The organization has been unusually honest about the limits of its own follow-up work, publicly noting that a newer experiment gave an unreliable signal — partly because AI adoption among developers had grown so pervasive that finding a clean control group who weren’t using the tools became genuinely hard. Sit with that irony for a second: the technology saturated the workflow so completely that measuring its effect got harder, not easier. That is not what a fad looks like.
Why the honest number is smaller than everyone’s story
So who is right — the 2x-productivity crowd or the it-makes-you-slower crowd? The best answer available is: neither, and the truth is less exciting than both. When you set aside the vendor demos and the viral anecdotes and look at independent measurement, the durable productivity gain keeps landing in the same unglamorous range. One analysis of the independent data put the real-world figure closer to 10 percent even as adoption climbed into the ninetieth percentile, with the flashy 2x-to-3x claims failing to survive contact with controlled measurement. A large McKinsey study of 4,500 developers found the same shape: big savings of around 46 percent on routine tasks, but under 10 percent on complex work. The gains are real. They are just concentrated in the boring parts of the job.
This is the actual paradox, and it is worth stating plainly because it resolves most of the shouting: the productivity boost is simultaneously real and much smaller than it feels. It feels enormous because the moments AI helps most — scaffolding a file, writing boilerplate, remembering an API signature — are also the moments that used to be the most tedious. Relief from tedium reads as speed even when the clock disagrees. Meanwhile the hard parts of software — understanding a gnarly system, deciding what to build, debugging the subtle thing — are exactly where the models help least and can actively mislead.
The quality bill nobody’s reading yet
There is a second half to the paradox that is easy to miss because it shows up later than the speed does. More code, faster, is not the same as more value, faster. An analysis of hundreds of millions of lines of code found that the average developer checked in 75 percent more code in 2025 than in 2022 — roughly the volume jump you would expect if the speed gains are genuine. But volume is not the same as quality, and independent benchmark researchers warn that AI-inflated line counts and rising code churn can manufacture the illusion of productivity even where the delivered value has not moved.
The uncomfortable pattern several teams are now noticing: some of the apparent speed is borrowed from the future. Code generated faster than it is reasoned about gets paid back later, in review load, in churn, in the subtle production bug that a junior would have caught while writing it slowly by hand. That does not make the tools bad. It means the true productivity number has to be measured after the maintenance bill arrives, not on the afternoon the pull request was opened.
What to actually believe
Strip it all down and the reasonable position is narrower than any of the camps want it to be. AI coding tools are not a hoax; the people getting real value from them are not lying. But they are also not the 10x revolution the earnings calls describe. The measured, durable, whole-pipeline gain for experienced engineers on real work is modest — think incremental, not transformational — and it is unevenly distributed across tasks, wildly overestimated by the people using it, and partly offset by quality costs that surface on a delay.
The practical takeaway for anyone running a team is to stop trusting the feeling and start trusting the baseline. Measure your own delivery, not seat counts or lines generated or how fast your best engineer says the tools make them. The single most robust finding in this entire literature is that developers are terrible judges of their own AI-assisted speed — they were wrong by nearly 40 points in a controlled setting. If the professionals closest to the work cannot eyeball this accurately, neither can a dashboard that only knows the tool was opened. The gains are worth having. They are just smaller, later, and lumpier than the hype — and knowing that is the difference between deploying these tools well and paying for a revolution you never actually received.
Related reading: Everyone’s Adopting AI. Almost Nobody’s Profiting. That’s the Real 2026 Story · Open vs. Closed AI Models: The Gap Closed, the Trade-Offs Didn’t
