Measure the productivity impact of AI tools

Combine developer experience surveys, adoption metrics, and usage patterns to understand how AI coding tools play into your software organization’s productivity.

Everyone’s excited about the potential of generative AI in software development, and for good reason. Tools like GitHub Copilot, Cursor, Claude Code, and many other tools are changing how developers write code. Engineering leaders we work with are seeing real productivity gains, but they also ask an important question: “How do we actually measure the impact?”

It’s a natural question. As engineering leaders, we want to understand the return on investment (ROI) of any new tool or practice. We want evidence that our investments are paying off and insights into optimizing their usage. But measuring the productivity impact of AI tools isn’t straightforward:

• Many teams lack a clear baseline to compare against.

• Developers use a fragmented mix of tools, and it’s hard to track them all.

• Gains in one metric can unintentionally hurt others.

• Early adopters tend to be high performers, skewing results (self-selection bias).

• Overreliance and inadequate reviews can reduce code understanding and increase tech debt.

• There’s no single measure of productivity and, thus, no simple definition of ROI.

That said, there are still lots you can do to better understand the impact of AI coding tools. In this guide, we walk through the different things to measure, their limitations, as well as actions you can take based on the results.

Prerequisites & Setup

General prerequisites

• GitHub connected

• Teams created in Swarmia

Connecting AI coding tools into Swarmia

To enable GitHub Copilot, Cursor, and Claude Code metrics, see our instructions for AI coding tool integrations.

Using Swarmia

Track AI tool adoption and licenses

Track the adoption and licenses of GitHub Copilot, Cursor, or Claude Code across all your teams over time at AI tools → AI adoption.

See how many developers have the AI assistants enabled versus how many are using them at any time period. This makes it easy to spot adoption trends, identify teams leading the charge, and find unused licenses that might need attention.

Read more about AI adoption metrics.

Study AI tool activity patterns in more detail

If you're looking for a productivity boost from an AI coding tool, it's not enough to enable the tool for your teams and then forget it. People need to pick up the new habit and invest the time to learn how to use it effectively. To support that, you need proper visibility into the current activity patterns.

Read more:

• GitHub Copilot activity breakdown

• Cursor activity breakdown

• Claude Code activity breakdown

Understand how AI impacts your developer productivity

There’s no single measure of developer productivity, so don’t expect to find one magic number that tells you the impact of AI tools either. Sure, that’s something your CEO or board might ask for from you, but the real world is more complex than that. Be wary of claims like “this tool makes your developers 55% more productive” or “GenAI gives you annualized savings of $475,728”, since they’re usually based on narrow definitions, wild assumptions, or flawed statistics.

Instead, here are some dimensions to explore for a comprehensive view of AI coding tools' impact on your engineering organization. With the added nuance, you should be much better equipped to answer the “now what” questions and take action.

• Speed: Use the AI tools → AI impact view to see how pull request cycle time and throughput change depending on AI use. What kind of differences can you see between teams with different levels of adoption? Pay attention also to the share of review time versus time in progress.

• Batch size: Use the AI tools → AI impact view to monitor pull request batch sizes to keep review quality high. AI assistants make it easy to generate big pull requests with lots of changes. In the worst case, some of the code that ends up in your product hasn’t been read by any human, neither the author nor the reviewer.

• Collaboration: Use the work log to track activity patterns and the distribution of work. Is collaboration increasing, or are more people working alone? Watch out for knowledge silos that form when individual developers generate and ship code without proper team understanding.

• Quality: Quality often suffers when teams focus solely on speed. Track the proportion of maintenance work in your investment balance and the change failure rate and mean time to recovery of your deployments to spot hidden technical debt.

Read more about AI impact on PR metrics.

Analyze AI coding agents' work

AI coding agents are AI tools responsible for creating pull requests end-to-end — planning work, editing code, running tests, and responding to reviews — not just suggesting code in your editor.

Use the AI tools → Coding agents view to see how these agents are contributing across your organization:

• Usage patterns:

  • Track how many agent PRs get merged versus closed to understand whether they’re mostly throwaway experiments or actually end up in production.

  • Follow the share of PRs created entirely by agents to judge how often engineers need to intervene manually.

  • Examine the batch size distribution to understand the complexity of the tasks the agents are able to complete.

• Trends over time: Understand whether agent usage is growing, and what’s driving it.

• Team comparisons: See which teams are getting the most value from agents.

• Drill-downs: Look at individual PRs to understand what’s working and what isn’t.

Read more about AI coding agent metrics.

Capture developer sentiment and experience with surveys and retrospectives

AI tools are not just for code generation, and not all developer time goes into writing code. Even if you’d create new code faster, your bottleneck could be in your product discovery, review process, or deployment pipeline. Hence, you might want to keep an eye on a wider array of tools for different uses, such as taking meeting notes, improving documentation, retrieving knowledge, analyzing technical debt, suggesting missing test cases, and identifying risky changes.

To get the richest view on how developers use AI tools, simply ask them directly. This has several benefits:

• You can gauge the usage of any tools without the need to get telemetry data from each.

• You can assess the perceived effects on speed and quality, which can be quite elusive when using only system metrics.

• Engineers can add their comments, and you can analyze common patterns in them.

You can create a survey in just a couple of minutes. Swarmia offers a selection of built-in AI tool questions with links to system metrics to analyze them hand-in-hand.

Taking action

Spreading best practices

Allow teams to find their own path to effective AI tool usage. Some developers and tasks will benefit more than others, and that’s okay.

• Make progress visible, while emphasizing learning rather than comparison. Ensure your teams have access to the AI assistant activity metrics and survey results. Without transparency, people who don’t use these tools might assume others aren’t using them either.

• Spot early adopters. Maybe they could organize an internal knowledge-sharing session to demonstrate effective usage patterns?

• Collect successes, failures, tips, and observations in a shared document, wiki, or Slack channel.

• Set aside dedicated time to experiment with AI tools.

• Regularly reassess your approach as AI capabilities evolve — what wasn’t effective six months ago might be transformative today.

Finding adoption bottlenecks

Invest in overcoming setup hurdles and make it easy to get started.

• Identify teams with low AI assistant usage or unused licences and find out what’s blocking them from taking the tools into use. Is it perhaps a lack of knowledge, access, or time?

• Revise your codebase documentation to make it easier for AI agents to understand.

• Configure good defaults for AI tools in your development environments.

Agreeing on ways of working

When taking any new tool into use, there are going to be unintended side effects. Be prepared to spot and address them early, before they become real problems.

• Establish clear guidelines for code review of AI-generated content.

• Create policies around AI tool usage with sensitive code.

• If you notice knowledge silos, set up a working agreement to avoid working alone on issues.

• If AI tool usage tends to result in too big pull requests, set up a working agreement to limit batch size.

• Discuss the topic in retrospectives. See our guide on running survey retrospectives.

• Focus on team-level improvements rather than individual performance.

Further reading

Swarmia blog:

• A staged approach to AI adoption for engineering teams by Rebecca Murphey, Field CTO · Dec 5, 2025

• What the 2025 DORA report tells us about AI readiness by Rebecca Murphey, Field CTO · Oct 22, 2025

• Small teams, big bets: Lessons from three Nordic AI startups by Erin Backlund, Content Marketing Manager · Oct 13, 2025

• Code faster, ship ... the same? by Rebecca Murphey, Field CTO · Aug 20, 2025

• Measuring AI impact like it’s 1995 by Rebecca Murphey, Field CTO · Aug 5, 2025

• Measuring the productivity impact of AI coding tools: A practical guide for engineering leaders by Otto Hilska, Founder & CEO · Feb 6, 2025