Measuring AI ROI in Software Engineering: What Actually Works in the Enterprise

Enterprises are spending serious money on AI coding tools. Licenses, pilot programs, internal enablement, security reviews, policy work. The pitch is always the same: developers ship faster, teams get more leverage, and the business gets more output.

But there’s a basic problem: in a lot of companies, nobody can clearly answer whether the tools are producing real gains or just creating new kinds of churn.

This blog walks through a practical approach to measuring AI impact in software engineering based on a research-driven framework: comparing AI vs non-AI teams over time, separating usage from outcomes, and avoiding the trap of shallow metrics that look good while quality quietly degrades.

Why “AI Productivity” Is Hard to Prove

AI image generated by Gemini

The cleanest way to measure ROI would be to say:

We gave engineers AI tools, and revenue increased.

That’s the ideal. It’s also usually impossible to isolate.

Business outcomes have too many confounding variables between “AI tool adoption” and “money,” including:

• product strategy and prioritization

• sales execution

• macro environment

• marketing, pricing, and packaging changes

• headcount changes and reorgs

• seasonality

• customer mix shifts

• platform incidents and reliability issues

So while business KPIs are the end goal, the measurement path that actually works today is to track engineering outcomes directly, and treat business outcomes as the longer-term validation.

That doesn’t mean engineering metrics are perfect. It means they’re less noisy than revenue when you’re trying to understand whether AI is helping engineers produce more value or just more code.

The Baseline: AI Lift Exists, But the Gap Between Teams Is Growing

When you look across teams using AI tools, you don’t see one consistent “AI productivity boost.” You see a distribution.

In one research setup, teams using AI were matched against similar teams not using AI, and net productivity gains were measured quarterly. Median lift landed around ~10% for that cohort (as of July in the dataset described), but the bigger story was this:

The spread between top performers and bottom performers kept widening.

That matters because it suggests AI is not a uniform accelerator. It’s more like a compounding advantage for teams that adopt it well, while teams that adopt it poorly may stall or go backward.

If you’re leading engineering, you don’t just want to know if AI is “good.” You want to know which cohort you’re in right now:

• Are you in the group compounding gains?

• Or are you shipping more while quality drops and rework rises?

You can’t course-correct without measurement.

Why Token Spend Doesn’t Predict Outcomes

A lot of companies start by measuring usage: tokens, prompts, tool sessions, seats assigned. That’s necessary, but it’s not enough.

One surprising insight from the transcript: when teams were plotted by token usage per engineer per month, the correlation with productivity lift was weak. There was even a “death valley” zone around a certain usage level where some

teams performed worse than teams using fewer tokens.

The takeaways:

• Usage volume isn’t the same as usage quality.

• Spending more tokens can mean:

  • more iteration,

  • more churn,

  • more back-and-forth,

  • more time cleaning up AI output.

So, yes: measure usage. But don’t confuse it with outcomes.

Codebase Hygiene Is a Real Predictor of AI Gains

AI image generated by Gemini

If token usage isn’t strongly predictive, what is?

One factor that correlated much more strongly was the “environment” developers work inside. The transcript describes an experimental environment cleanliness index, a composite score based on things like:

• tests

• types

• documentation

• modularity

• code quality

In that dataset, cleanliness showed a meaningfully stronger correlation with productivity lift than token volume did.

This is intuitive if you’ve watched AI tools operate inside real codebases:

• In clean, modular systems with tests and clear contracts, AI can safely contribute.

• In messy systems with unclear boundaries, missing tests, and hidden coupling, AI tends to produce code that compiles but increases entropy.

A useful mental model: every codebase sits on a spectrum where the percentage of tasks that AI can handle well changes depending on hygiene. As entropy increases, the set of tasks where AI is useful shrinks.

And there’s a feedback loop:

• AI can accelerate delivery,

• but unchecked AI can also accelerate codebase entropy (tech debt, duplication, churn),

• which reduces future AI gains unless humans actively invest in hygiene.

So one of the most practical “AI ROI” strategies is boring:

Spend time improving codebase hygiene.

It’s not just good engineering. It’s also how you unlock AI gains.

Adoption Isn’t Just “Do People Have Access?”

A huge trap in enterprise rollouts is equating:

• “we bought licenses”

• “people have access”

• “we’re adopting AI”

Access is not adoption. And adoption is not effective use.

The transcript describes an “AI engineering practices benchmark” that looks for AI fingerprints in a codebase to infer how teams are using AI patterns over time. The maturity model described runs roughly like:

• Level 0: no AI usage (human-only)

• Level 1: personal AI use (not shared, not versioned)

• Level 2: team usage (shared prompts/rules)

• Level 3: AI handles specific tasks autonomously

• Level 4: agentic orchestration (AI runs a larger slice of workflow)

One case study in the transcript shows two business units with the same tools and access, but very different adoption patterns: one unit used AI in a much larger share of active work than the other.

So leaders need to understand not just:

• Are people using AI?

But also:

• How are they using AI?

• Are they sharing workflows?

• Are prompts and rules versioned?

• Is there a consistent method?

• Are there guardrails?

Measuring ROI: Separate Usage Metrics From Outcome Metrics

AI image generated by Gemini

A practical AI ROI measurement system has two parts:

1. Usage measurement (treatment)

2. Engineering outcome measurement (effects)

1) Usage measurement

There are two broad approaches:

Access-based measurement

• who got access, when

• compare pilot group vs control group

• or compare the same team before/after access

This is easy to set up but noisy, because getting access doesn’t mean usage, and usage doesn’t mean results.

Usage-based measurement

• telemetry from tools and APIs

• who used AI, how often, where

• ideally granular enough to analyze team patterns

The transcript points out a real constraint: vendors differ in telemetry quality. Some aggregate heavily; others provide more granular data. So usage measurement is partly a tooling reality.

A key point: if you already adopted AI, you can still measure impact retroactively by pairing tool usage (as available) with git history and engineering data. You don’t necessarily need to wait six months to design a new experiment.

2) Engineering outcome measurement

This is where most companies get it wrong, because they pick metrics that are easy to count, not metrics that reflect value.

Why PR Counts and Lines of Code Don’t Work

Teams love metrics like:

• PR count

• commit count

• lines of code

• “DORA metrics, but used incorrectly”

These can be useful signals in the right context, but they are not reliable as direct productivity measures for AI adoption.

Why?

Because AI can inflate them without producing real value.

• More PRs can mean more shipping, or it can mean more fragmentation and review burden.

• More code can mean more features, or it can mean duplication and churn.

• Faster merges can mean better flow, or it can mean weaker review standards.

This is why you need outcome metrics that capture effective engineering output and guardrails.

A Practical Framework: One Primary Metric + Guardrails

A strong approach is:

Primary metric: Engineering output (not raw volume)

The transcript describes using a machine learning model trained to replicate a panel of expert reviewers who rate changes across dimensions like:

• implementation quality

• maintainability

• complexity (and implicitly time/cost to maintain)

The point isn’t that ML is magic. The point is that:

• expert review at scale is impossible,

• so you need a consistent proxy that correlates with expert judgment,

• and you can validate it by sampling real expert panels when needed.

This primary metric tries to answer:Did we produce meaningful, maintainable engineering output?

Not “did we write code.”

Guardrails: Keep the system healthy while output rises

Guardrails are metrics you want to keep within healthy bounds, not maximize blindly.

The transcript groups them into buckets like:

• rework / refactoring

• quality tech and risk

• people / DevOps signals

The mindset is:

• grow effective output,

• while preventing rework, quality degradation, and operational risk from blowing up.

Case Study: When AI “Improved Productivity” on Paper but ROI Was Negative

AI image generated by Gemini

The transcript gives a clean example of how companies fool themselves.

A large enterprise team (about 350 people) adopted AI tools.

Looking at the four months before and four months after adoption:

• PRs increased by 14%

• Leadership could easily interpret that as “we’re 14% more productive”

But deeper measurement showed:

• code quality decreased (maintainability score dropped)

• quality became more erratic after adoption

• rework increased dramatically (about 2.5x)

• effective output did not meaningfully increase

That’s the nightmare scenario: activity rises, but value doesn’t. And because rework surged, the hidden cost shows up later in:

• review burden

• defect rates

• reliability incidents

• slower future delivery

• worse onboarding

• growing tech debt

The conclusion in the transcript is blunt:ROI might be negative, and without deeper measurement the company would have celebrated the rollout.

The important nuance: the right move is not “abandon AI.” The right move is to use the data to diagnose what’s going wrong and improve adoption patterns.

What Usually Goes Wrong When AI ROI Turns Negative

When enterprises adopt AI and see flat or negative ROI, common causes include:

1) AI is used where it shouldn’t be

Engineers use AI for tasks that require deep system context and subtle constraints, and the output looks plausible but is wrong.

2) The codebase lacks guardrails

Missing tests, weak typing, weak linting, unclear patterns. AI ships faster into an environment that can’t catch mistakes early.

3) Review processes don’t evolve

AI increases output. Review burden rises. Quality gates get overloaded. Teams either slow down or let more issues through.

4) “Usage” is mistaken for “progress”

Companies celebrate token consumption, prompt counts, or PR volume, without tracking rework, maintainability, or downstream risk.

5) No shared practices

Everyone uses AI differently. Prompts aren’t shared. Rules aren’t versioned. Teams don’t converge on effective workflows.

How to Use This Measurement to Improve, Not Punish

A lot of teams avoid measuring AI impact because they fear metrics will be weaponized. That’s a real risk. But “no measurement” is worse, because then you can’t tell if you’re compounding gains or drifting into churn.

A few principles to keep measurement constructive:

• Use metrics to improve systems, not rank individuals

• Look at trends at team/org level, not “developer scorecards”

• Pair output metrics with guardrails to avoid Goodhart effects

• Use sampling and qualitative review to validate what numbers suggest

• Treat measurement as iteration: refine the model as you learn

The goal is to answer:

• What workflows produce gains?

• Where is AI hurting us?

• What investments unlock better ROI?

The Playbook: Getting to Real AI ROI

AI image generated by Gemini

If you want AI ROI that holds up in an enterprise, the path looks like this:

1) Instrument usage (as best your tools allow)

Even imperfect telemetry is better than guessing.

2) Measure outcomes beyond PR volume

Track maintainability/quality proxies and rework signals.

3) Invest in codebase hygiene

Tests, modularity, types, docs where it matters. Hygiene isn’t optional if AI is writing more code.

4) Build shared practices

Move from personal usage to team usage:

• shared prompts

• shared rules

• reusable workflows

• consistent standards

5) Teach “when not to use AI”

If engineers lose trust because outputs get rejected or rewritten, adoption collapses and gains vanish.

6) Keep guardrails healthy

If rework rises, or quality drops, treat it like a production incident: root cause and fix the system.

Bottom Line

AI tools can absolutely create productivity lift in software engineering. But the lift is not automatic, and it’s not evenly distributed.

The teams that win tend to do a few things consistently:

• measure outcomes, not just activity

• keep codebases clean enough for AI to contribute safely

• evolve practices and workflows as output increases

• manage entropy and rework aggressively

Without that, it’s easy to ship more while building a slower, riskier, more expensive codebase underneath.

If you’re spending millions on AI tools, the question isn’t “are we using them?” The

question is:

Are we getting durable engineering output without exploding rework and quality risk?