> For the complete documentation index, see [llms.txt](https://help.swarmia.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://help.swarmia.com/guides/understand-the-impact-of-ai-tools.md).
# Understand the impact of AI tools
Tools like GitHub Copilot, Cursor, Claude Code, and other AI coding tools are changing how engineers write and review code. Many teams see real productivity gains, but they also ask an important question: How do we measure the impact?
It's a natural question. Engineering leaders want evidence that their investments are paying off. Yet, [there's no single measure of developer productivity](https://queue.acm.org/detail.cfm?id=3454124), so don't expect one number to tell you the impact of AI tools. Be wary of claims like *"this tool makes your developers 55% more productive"* or *"GenAI gives you annualized savings of $475,728"*, since they're usually based on narrow definitions, broad assumptions, or flawed statistics.
Here's why measuring the productivity impact of AI tools isn't straightforward.
* Many teams lack a clear baseline to compare against.
* Engineers use a fragmented mix of tools, and it's hard to track them all.
* Gains in one metric can unintentionally hurt others.
* Early adopters tend to be high performers, skewing results.
* Overreliance and inadequate reviews can reduce code understanding and increase tech debt.
* There isn't one metric that can tell the full story.
You can still build a useful picture of how AI coding tools affect your organization. This guide explains what to measure, where the data has limitations, and what to do with the results.
## Using Swarmia
### Track AI tool adoption and licenses
Use [AI adoption](/features/ai-tools/ai-adoption.md) to see how many people have GitHub Copilot, Cursor, or Claude Code enabled, how many actively use them, and where you may have idle licenses.
### Study AI tool activity patterns
Use the activity breakdowns to understand how people use AI tools in practice: which features they adopt, how usage develops over time, and where teams may need more support.
Read more:
* [GitHub Copilot activity](/features/ai-tools/github-copilot-activity.md)
* [Cursor activity](/features/ai-tools/cursor-activity.md)
* [Claude Code activity](/features/ai-tools/claude-code-activity.md)
### Understand how AI impacts developer productivity
Use [AI impact](/features/ai-tools/ai-impact.md) to compare AI-assisted and non-AI-assisted pull requests across throughput, cycle time, review time, and batch size. The page helps you understand where AI changes the flow of work and where you may need to look deeper.
### Analyze AI cloud agents' work
Use [Cloud agents](/features/ai-tools/cloud-agents.md) to see how AI agents that create pull requests end-to-end are contributing across your organization. Track merged and closed agent PRs, the share of work done entirely by agents, batch size, and team-level adoption.
### Track review agent coverage
Use [Review agents](/features/ai-tools/review-agents.md) to understand which AI agents review your pull requests, how much of your code they cover, and how many findings they leave per PR.
### Capture developer sentiment with surveys and retrospectives
AI tools affect more than code generation. They can change discovery, review, testing, documentation, and knowledge sharing. To understand those changes, ask engineers directly.
Developer experience [surveys](/features/run-developer-experience-surveys.md) help you collect feedback on AI tool usage, perceived speed, quality, and friction. Use retrospectives to discuss the patterns and agree on what to change next.
### Understand collaboration patterns
Use the [work log](/features/focus/analyzing-the-activity-patterns-on-work-log.md) to see how AI tools affect collaboration and work distribution. Watch for changes in how much work happens alone, how often people collaborate on the same issues, and whether knowledge is spreading across the team.
If AI-assisted work creates new silos, use the data as a starting point for team discussions and working agreements.
### Track quality signals
AI tools can speed up code changes, but faster output doesn't automatically mean better outcomes. Use [investment balance](/features/focus/balance-engineering-investments.md) to monitor the share of maintenance work, and use [DORA metrics](/features/metrics/track-dora-metrics.md) to track change failure rate and mean time to recovery.
These signals help you spot whether AI-assisted work is creating hidden quality costs, such as more rework, incidents, or technical debt.
## Taking action
### Spreading best practices
Allow teams to find their own path to effective AI tool usage. Some engineers and tasks will benefit more than others, and that's okay.
* Make progress visible while emphasizing learning rather than comparison.
* Spot early adopters who can share practical examples.
* Collect successes, failures, tips, and observations in a shared document, wiki, or Slack channel.
* Set aside time to experiment with AI tools.
* Reassess your approach as AI capabilities evolve.
### Finding adoption bottlenecks
Invest in overcoming setup hurdles and make it easy to get started.
* Identify teams with low AI assistant usage or unused licenses, and find out what's blocking adoption.
* Improve codebase documentation so AI agents can understand your system.
* Configure good defaults for AI tools in your development environments.
### Agreeing on ways of working
New tools can create unintended side effects. Spot and address them early, before they become real problems.
* Establish clear guidelines for reviewing AI-generated code.
* Create policies for using AI tools with sensitive code.
* If you notice knowledge silos, [set up a working agreement](/features/working-agreements.md) to avoid working alone on issues.
* If AI tool usage results in overly large pull requests, [set up a working agreement](/features/working-agreements.md) to limit batch size.
* Discuss the topic in retrospectives. See our [guide on running survey retrospectives](https://www.swarmia.com/blog/developer-survey-retrospectives/).
* Focus on team-level improvements rather than individual performance.
## Further reading
Swarmia blog:
* [Five levels of AI coding agent autonomy, and why higher isn't always better](https://www.swarmia.com/blog/five-levels-ai-agent-autonomy/) by Miikka Holkeri, Product Manager · Mar 19, 2026
* [A staged approach to AI adoption for engineering teams](https://www.swarmia.com/blog/staged-approach-AI-adoption-for-engineering) by Rebecca Murphey, Field CTO · Dec 5, 2025
* [What the 2025 DORA report tells us about AI readiness](https://www.swarmia.com/blog/dora-2025-report-ai-readiness/) by Rebecca Murphey, Field CTO · Oct 22, 2025
* [Small teams, big bets: Lessons from three Nordic AI startups](https://www.swarmia.com/blog/small-teams-big-bets/) by Erin Backlund, Content Marketing Manager · Oct 13, 2025
* [Code faster, ship ... the same?](https://www.swarmia.com/blog/code-faster-ship-the-same/) by Rebecca Murphey, Field CTO · Aug 20, 2025
* [Measuring AI impact like it's 1995](https://www.swarmia.com/blog/measuring-ai-impact-like-1995/) by Rebecca Murphey, Field CTO · Aug 5, 2025
* [Measuring the productivity impact of AI coding tools: A practical guide for engineering leaders](https://www.swarmia.com/blog/productivity-impact-of-ai-coding-tools/) by Otto Hilska, Founder & CEO · Feb 6, 2025
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://help.swarmia.com/guides/understand-the-impact-of-ai-tools.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.