Understand the impact of AI tools

Combine developer experience surveys, adoption metrics, and usage patterns to understand how AI coding tools play into your software organization's productivity.

Tools like GitHub Copilot, Cursor, Claude Code, and other AI coding tools are changing how engineers write and review code. Many teams see real productivity gains, but they also ask an important question: How do we measure the impact?

It's a natural question. Engineering leaders want evidence that their investments are paying off. Yet, there's no single measure of developer productivity, so don't expect one number to tell you the impact of AI tools. Be wary of claims like "this tool makes your developers 55% more productive" or "GenAI gives you annualized savings of $475,728", since they're usually based on narrow definitions, broad assumptions, or flawed statistics.

Here's why measuring the productivity impact of AI tools isn't straightforward.

Many teams lack a clear baseline to compare against.
Engineers use a fragmented mix of tools, and it's hard to track them all.
Gains in one metric can unintentionally hurt others.
Early adopters tend to be high performers, skewing results.
Overreliance and inadequate reviews can reduce code understanding and increase tech debt.
There isn't one metric that can tell the full story.

You can still build a useful picture of how AI coding tools affect your organization. This guide explains what to measure, where the data has limitations, and what to do with the results.

Using Swarmia

Track AI tool adoption and licenses

Use AI adoption to see how many people have GitHub Copilot, Cursor, or Claude Code enabled, how many actively use them, and where you may have idle licenses.

Study AI tool activity patterns

Use the activity breakdowns to understand how people use AI tools in practice: which features they adopt, how usage develops over time, and where teams may need more support.

Understand how AI impacts developer productivity

Use AI impact to compare AI-assisted and non-AI-assisted pull requests across throughput, cycle time, review time, and batch size. The page helps you understand where AI changes the flow of work and where you may need to look deeper.

Analyze AI cloud agents' work

Use Cloud agents to see how AI agents that create pull requests end-to-end are contributing across your organization. Track merged and closed agent PRs, the share of work done entirely by agents, batch size, and team-level adoption.

Track review agent coverage

Use Review agents to understand which AI agents review your pull requests, how much of your code they cover, and how many findings they leave per PR.

Capture developer sentiment with surveys and retrospectives

AI tools affect more than code generation. They can change discovery, review, testing, documentation, and knowledge sharing. To understand those changes, ask engineers directly.

Developer experience surveys help you collect feedback on AI tool usage, perceived speed, quality, and friction. Use retrospectives to discuss the patterns and agree on what to change next.

Understand collaboration patterns

Use the work log to see how AI tools affect collaboration and work distribution. Watch for changes in how much work happens alone, how often people collaborate on the same issues, and whether knowledge is spreading across the team.

If AI-assisted work creates new silos, use the data as a starting point for team discussions and working agreements.

Track quality signals

AI tools can speed up code changes, but faster output doesn't automatically mean better outcomes. Use investment balance to monitor the share of maintenance work, and use DORA metrics to track change failure rate and mean time to recovery.

These signals help you spot whether AI-assisted work is creating hidden quality costs, such as more rework, incidents, or technical debt.

Taking action

Spreading best practices

Allow teams to find their own path to effective AI tool usage. Some engineers and tasks will benefit more than others, and that's okay.

Make progress visible while emphasizing learning rather than comparison.
Spot early adopters who can share practical examples.
Collect successes, failures, tips, and observations in a shared document, wiki, or Slack channel.
Set aside time to experiment with AI tools.
Reassess your approach as AI capabilities evolve.

Finding adoption bottlenecks

Invest in overcoming setup hurdles and make it easy to get started.

Identify teams with low AI assistant usage or unused licenses, and find out what's blocking adoption.
Improve codebase documentation so AI agents can understand your system.
Configure good defaults for AI tools in your development environments.

Agreeing on ways of working

New tools can create unintended side effects. Spot and address them early, before they become real problems.

Establish clear guidelines for reviewing AI-generated code.
Create policies for using AI tools with sensitive code.
If you notice knowledge silos, set up a working agreement to avoid working alone on issues.
If AI tool usage results in overly large pull requests, set up a working agreement to limit batch size.
Discuss the topic in retrospectives. See our guide on running survey retrospectives.
Focus on team-level improvements rather than individual performance.