Change Fail Rate

The percentage of changes that result in negative customer impact, or rollback.

changeFailRate = failedChangeCount / changeCount

What is the intended behavior?

Reduce the percentage of failed changes.

How to improve it

  • Release more, smaller changes to make quality steps more effective and reduce the impact of failure.
  • Identify root cause for each failure and improve the automated quality checks.

How to game it

  • Deploy fixes without recording the defect.
  • Create defect review meetings and re-classify defects as feature requests.
  • Re-deploy the latest working version to increase deploy count.

Guardrail Metrics

Metrics to use in combination with this metric to prevent unintended consequences.

  • Delivery frequency can decrease if focus is placed on “zero defect” changes.
  • Defect rates can increase as reduced delivery frequency increases code change batch size and delivery risk.

References

Last modified February 2, 2022: Switch to hugo (#2) (a244f74)