Automatic change failure detection

Automatically detect DORA change failures from application version rollback, pull request reverts, and advanced hotfix filters.

About automatic change failure detection

When enabled, Swarmia automatically looks into every created deployment to see if it includes fixes to a previous change. When a fix is detected, Swarmia automatically records a change failure for the deployment that shipped the failing change to production.

Automatic change failure detection supports identifying version rollbacks, pull request reverts, and advanced hotfix filters.

Automatic change failure detection works with all available deployment sources.

Enabling automatic change failure detection

Automatic change failure detection is enabled by default for all new applications.

You can enable or disable automatic change failure detection by navigating to your application in Settings / Deployments and toggling the "Automatically detect rollbacks, reverts and hotfixes" checkbox.

Rollback detection

Rollback detection checks if the exact same version has been previously deployed to the same environment (e.g. production).

If Swarmia detects a rollback, a change failure is recorded to the first deployment after the deployment you rolled back to.

Pull request revert detection

To detect reverts, Swarmia checks if any of the pull requests included in your deployment were reverting a previously deployed pull request.

Swarmia first looks into all merge commit messages in the base branch to find a merge commit for a pull request that was reverted through the GitHub UI.

If none is found, we check pull request branch names (format: revert-PULL_REQUEST_NUMBER-) and descriptions (format: Reverts #PULL_REQUEST_NUMBER) for reverted pull request numbers.

If we find a reverted pull request, we record a change failure for the deployment where the pull request was deployed.

For applications using the Deployment API, the payload you send must include commit and repository information. This data is automatically included if you have been using the examples from our latest documentation.

Advanced hotfix detection options

Swarmia supports two additional methods for hotfix detection, used to calculate change failure metrics and available in Settings/Deployments.

Pull request filters

You can identify change failures based on deployments linked to specific PRs using criteria like pull request title or branch name. Matching PR deployments will be considered fixes, marking the preceding deployment as a failure.

Issue filters

Similarly, it's possible to detect change failures in deployments linked to specific issues in your issue tracker using criteria such as issue title, label, or custom field. Matching deployments will be considered fixes, marking the preceding deployment as a failure.

Limitations

• Automatic change failure detection only looks into the first 1000 commits per deployment. If your deployment includes more commits, those are not checked.

• We record a single change failure per deployment. If your deployment includes fixes for multiple change failures, only one of them will be marked as failed.