Almost every engineering team now uses AI. The 2025 DORA report found that 90% of respondents use AI in their work, with a median of about 2 hours a day. Board members and CFOs have moved on from asking whether the company uses AI coding tools. They now ask what the company gets for the money. Most CTOs cannot answer with confidence, because the evidence is mixed and the usual engineering metrics were not built for agent-written code.

This guide sets out what the research says, which metrics prove value, which ones mislead and how to run a fair measurement in one quarter.

What does the research say about AI coding productivity?

The data points in different directions, and a CTO needs all of it in view.

• Self-reported gains are high. In the Jellyfish State of Engineering Management 2026 survey of more than 600 engineering leaders, 64% reported at least a 25% increase in developer velocity from AI, and 80% said AI improves team productivity.
• Controlled measurement is more sobering. A randomized trial by METR found that 16 experienced open-source developers took 19% longer on real tasks when AI tools were allowed, while believing AI had made them 20% faster. METR's February 2026 update found the gap narrowing with late-2025 tools and noted that many developers now refuse to work without AI at all, which makes clean experiments harder.
• Speed and stability move separately. The 2025 DORA research found AI adoption now correlates with higher throughput but still correlates with higher instability. AI amplifies whatever engineering foundation already exists.
• Trust is falling as use rises. In the Stack Overflow 2025 Developer Survey, more than 84% of developers used or planned to use AI tools, but only 29% said they trust AI output, down 11 points from 2024.

The lesson for CTOs: perception is not evidence, and speed without stability is not productivity.

Which metrics prove AI coding ROI?

Measure outcomes at the team and system level, not activity at the individual level.

1. Throughput that reaches customers

Track deployment frequency and lead time for changes, the two DORA throughput measures. More pull requests mean nothing if they do not ship.

2. Stability

Track change failure rate, failed deployment recovery time and rework rate. If throughput rises and stability falls, the agents are moving cost downstream into incidents and on-call load.

3. Review load

Agents shift work from writing code to reviewing it. Measure review time per pull request, pull request size and the share of senior engineer time spent reviewing. Rising review load is the most common hidden cost of agent adoption.

4. Cost per shipped change

Add tool licences, model usage and the engineering time spent on agent workflows, then divide by changes shipped to production. In the Jellyfish survey, 42% of leaders cited rising AI tool costs as a barrier, so expect your CFO to ask for this number.

5. Developer experience

Run a short quarterly survey on flow, cognitive load and confidence in the codebase. Falling confidence often predicts quality problems before incident data shows them.

Which metrics mislead?

• Lines of code or percentage of code written by AI. Agents produce volume cheaply. Volume is a cost, not a benefit, once someone has to maintain the code.
• Acceptance rate of AI suggestions. A high rate shows developers accept output, not that the output was correct or valuable.
• Self-reported time saved. The METR findings show how far perception drifts from measured reality.
• Individual developer leaderboards. These invite gaming and damage trust in the program.

How should a CTO run a fair measurement in one quarter?

1. Set a baseline. Capture 8 to 12 weeks of DORA metrics, review load and cost before expanding agent use.
2. Pick comparable teams. Compare teams with similar codebases and work types, or compare the same team across periods with similar roadmap complexity.
3. Define the work types. Agents help most on well-specified, test-covered tasks such as migrations, test generation and boilerplate. Report results by work type, not as one average.
4. Track stability for a full cycle. Quality problems from agent code often surface weeks after merge. Do not call the result before one full release cycle.
5. Report one page to the executive team. Show throughput, stability, cost per change and developer experience together, with a clear recommendation to expand, adjust or stop.
   [Table 1]

What should CTOs fix before scaling agents?

DORA's 2025 research describes AI as an amplifier. Teams with strong testing, small batch sizes, clear platform standards and fast code review gain the most. Teams with weak foundations get more output and more incidents. Before scaling agents, invest in automated testing coverage, a paved-road platform, clear ownership of services and review practices sized for higher pull request volume. Our guide to engineering org design for B2B SaaS CTOs covers where platform ownership fits.

What does this mean for engineering leadership roles?

Measuring AI ROI is becoming a core expectation of the CTO and VP of Engineering roles. Boards now ask for it, and search committees test for it. For hiring guidance, see our CTO hiring resources, and for how AI scope changes pay, see AI-native CTO compensation in 2026.

[Table 2]

‍

Frequently asked questions

What is a realistic productivity gain from AI coding agents?

Results vary by team and work type. Self-reported gains of 25% or more are common, while controlled studies show smaller or mixed effects. Measure your own baseline rather than relying on vendor figures.

Should we measure individual developers' AI usage?

Measure at the team level. Individual metrics invite gaming and say little about customer outcomes.

How often should we report AI ROI to the board?

Quarterly, alongside delivery and reliability metrics, so AI spend is judged by the same outcomes as the rest of engineering.

Compare notes with other engineering leaders: apply to join CTO Club.