Testing Metrics: Why Measure, What to Measure, and How to Measure

read

In my experience, one of the most challenging conversations an organization has around quality is how to measure it.  You can google testing metrics and find yourself drowning in hundreds of key performance indicator (KPI) options.  The reason there is no one size fits all is simple; the culture and what is important to each organization is unique.  The old adage: what gets measured, gets done is certainly true in this context.  Understanding why you are measuring, what you are measuring, and what you will do with the data, is important when identifying what KPIs to implement for your organization.

Why Are You Measuring?

Before deciding on what to measure, it is important to stop and contemplate why you want to measure quality in the first place?  Understanding the why upfront will help guide you in selecting the best KPIs for your organization.  There are many reasons your organization could be driven to use KPIs:

  • Your C-Level suite wants an additional perspective on customer satisfaction: In addition to knowing when a release has shipped to customers, C-Level leaders also want insight when customers report defects, on any downtime that might impact customers, and how the company can avoid delays moving forward.  In addition, they may want to ensure efficiencies across their teams; if you have major quality issues that block or move release dates, that may raise major red flags for your C-Level suite.
  • Your QA manager wants to gauge the effectiveness of your testing suite: Ensuring your testing suite is effective in catching defects, and catching them as soon as possible, is a critical piece of a QA manager’s job.  Using KPIs to track and trend defect activity can help drive effectiveness over time.
  • Your organization has quality problems that you want to resolve: Using KPIs to gauge where you are, and track trends based on implemented changes, can help guide decision making and ultimately resolve quality issues.  Knowing where the quality issues are stemming from is an important piece of resolution.
  • Intense release timelines make it imperative that you make certain those releases aren’t defect heavy: As more pressure is put on teams to quickly deliver new features to customers, the lure of cutting corners and the risk for defects may increase.  KPIs can help you in becoming proactive in your goal to get features to customers rapidly.
  • You want to keep the customers you have and continue to grow: There’s nothing worse than downloading an app, or going to a website, and the enormous amounts of defects make almost impossible to use, much less enjoy.  Using KPIs to ensure every corner of your application is useable is important to your organization’s long-term growth plan.
  • You want to create a culture of accountability: To hold someone accountable, you must first communicate their success measurements.  Ensuring you have a standard set of KPIs, tracked and communicated consistently, will allow you to hold people accountable, and do so fairly.
  • You want a way to track release progress: As you’re moving through your final testing stages and preparing your release to customers, knowing what milestones have been completed is important.  These milestones might include zero critical defects, completion of test case writing, or hitting specific test execution percentage markers. 

Pitfalls of KPIs

Before we jump into specific KPIs, let’s walk through some common pitfalls to avoid.  Using KPIs for the sole purpose of informing and improving is ideal.  However, using them out of context or applying them incorrectly, paints an inaccurate picture.  Avoid these pitfalls when you’re implementing your KPIs:

  • Don’t focus solely on one area: Even if your goal is to measure quality, your focus should be throughout the entire software development lifecycle (SDLC).  Focusing in on testers, or only the testing phase, will not give you the big picture view of quality.
  • KPIs aren’t helpful if taken out of context: As you’re considering what KPIs will be useful to your organization, and your ability to paint a whole picture, make certain you are deriving useful contextual information.
  • Using metrics to spy on teams: Honestly, this is a common pitfall that we’ve all found ourselves trapped in.  There are two ways to measure success:  let teams use KPIs internally and self-manage OR have a set of KPIs that you routinely monitor to help guide customer satisfaction, process or tool effectiveness, or coaching opportunities.  Don’t fall into the trap of using KPIs solely to manage productivity.  KPIs should be used as a starting point and lead to further conversations. 

What Could You Measure?

So, you’ve worked through the reasons you want to measure quality and understand the pitfalls to avoid.  Next step is determining what KPIs help you meet those goals.  Here are some top testing KPIs you may decide are a good fit for your organization:

  1. Defect to User Story Percentage:

     What Does It Tell Me?

    This metric will give you a visual on how many defects your team is raising each sprint.  It also considers the unique number of user stories in a sprint to give you a consistent percentage that you can track from sprint to sprint.  If you see a spike, you may conclude that a specific feature your team is testing is complex, or maybe it wasn’t well understood by the team.  Either way, using this metric gives you a starting point to dig in and be proactive.

    How Do You Calculate It?

    Defect to User Story % = # of defects opened / # of user stories) * 100

  2. Automation Script Effectiveness

     What Does It Tell Me?

    This metric provides clarity on how your defects are being found.  If you are investing large dollars in automation, but your automation scripts aren’t finding defects, you may want to dig in and understand the effectiveness of those scripts.  Also, if you use different testing environments, like integration and staging, you may expect to see lower effectiveness in your integration environment because scripting hasn’t been completed for new features.  Likewise, in your staging environment, you would expect to see automation finding the majority of the defects as you’re typically focused on regression testing.

    How Do You Calculate It?

    Automation Script Effectiveness = (# of defects found by automation / # of defects opened) * 100

  3. Mean Time to Detect (MTTD)

    What Does It Tell Me?

    This metric provides insight into how long it takes QA to find a defect.  Reducing this number over time will help provide feedback on the efficiency of your processes and execution strategy. 

    How Do You Calculate It?

    This calculation takes a bit of upfront work to gather the data needed.  First, pull a list of all defects found.  Next, note what day/time your team started execution.  Finally, note what day/time each defect was found.  Using this data, get the duration of each defect by measuring the time between the start and detection times.  Finally, use that to calculate the mean time:

    Mean Time to Detect = total time to detect (duration) / # of defects opened

    Using an example, let’s walk through this calculation:

    Start Time

    Detection Time

    Duration (in mins)

    7:15 AM

    8:00 AM

    45

    8:45 AM

    12:00 PM

    195

    2:30 PM (5/11/20)

    2:30 PM (5/12/20)

    1440

    In this example, our durations add up to 1680 minutes (45 + 195 + 1440).  Divide that sum by 3 (# of defects opened) and our MTTD = 560 minutes (or just over 9 hours) 

  4. Mean Time to Fix (MTTF)

    What Does It Tell Me?

    This metric partners nicely with MTTD in that it gives further visibility into how long it takes your development team to fix a defect once it’s been found. 

    How Do You Calculate It?

    Using a similar method as MTTD, in this calculation, you have to gather the total durations to fix all defects.  You can calculate the total time to fix using a couple of methods.  It’s important to pick one method to use, as that consistency gives you the ability to use this metric to track trends over time:

    • Total coding time (time spent coding it, not including prioritization, build/deploy activities)
    • Total time starting when the defect was identified to when the defect fix is ready to be tested (incorporates everything it takes to get a defect fixed and deployed to your testing environment).

    Once you have your total durations, you simply add those up and divide by the total defects. 

    MTTF = total time to fix (duration) / # of defects opened

    Using our previous example, we can take it a step further to include MTTF.

    Detection Time

    Detection Time

    Duration (in mins)

    8:00 AM

    8:00 PM

    720

    12:00 PM

    6:15 PM

    375

    2:30 PM (5/12/20)

    3:30 PM (5/14/20)

    2940

    In this example, our durations add up to 4035 minutes (720 + 375 + 2940).  Divide that sum by 3 (# of defects opened) and our MTTD = 1345 minutes (or just over 22 hours)

  5. Defect Leakage Rate in Production:

    What Does It Tell Me?

    This metric is perhaps the most widely used metric as it gives anyone from senior leadership, all the way down to individual contributors, a good indication of the efficiency of your overall processes.  The defect leakage rate calculation will tell you what percentage of defects made it to production.  Defects found in production are costly to fix and could damage your brand, therefore, striving for a 0% defect leakage rate is desirable.  In addition to production, you can modify this to understand defect leakage across any testing environments you may have. 

    How Do You Calculate It?

    Defect Leakage Rate = (# of production defects / # of defects opened) * 100

  6. Defect Severity

    What Does It Tell Me?

    This metric is important as it measures the levels of defect severity.  If you have a large percentage of defects that are critical or high severity you may conclude that your release is risky. 

    How Do You Calculate It?

    This requires a bit more effort than a single calculation.  First, you will need to identify the severity of all defects identified.  Next, count the number of defects in each priority bucket.  Lastly, calculate what percentage each bucket accounts for in the overall defect count.

    Defect Severity = total(defects in each severity bucket) / # of defects opened) * 100

  7. Downtime

    What Does It Tell Me?

    This metric is an organizational level metric that accounts for downtime across all teams, in production.  It is a simple calculation that can help guide leaders on their overall development effectiveness.  Every organization aims for this metric to be 0%.

    How Do You Calculate It?

    Downtime % = (amount of system down time in production / system up time in production) * 100

  8. Fixed Defects Percentage

    What Does It Tell Me?

    This metric is important in understanding how many defects your organization is closing versus how many they are opening.  If you are consistently opening more defects than you can close, over time you will have a degraded application riddled with bugs.   

    How Do You Calculate It?

    Fixed Defect % = (defects fixed / # of defects opened) * 100

  9. Automation Pass Rate

    What Does It Tell Me?

    This metric will give you a standard percentage of how many automation tests passed.  It is helpful in understanding the stability of your automation suite as well as its effectiveness.  Having a low pass rate requires more time spent validating failures.  If your failures turn out to be false failures, that’s an early indicator that your automation suite isn’t reliable.  In addition, if you see this number dip after an automation execution, this might be a quick indicator that your release contains higher than normal defects. 

    How Do You Calculate It?

    Pass Rate % = (# of cases that passed / # of test cases executed) * 100

  10. Invalid Defect %

    What Does It Tell Me?

    This metric will provide some insight into how well your testers understand the features they are testing.  A higher invalid defect percentage might indicate your testers don’t understand the feature or your testing isn’t covering the areas it should.

    However, zero invalid defects isn’t the number we are shooting for either.  Having testers that find the clear-cut bugs is great, but taking that a step further and reporting other suspicious activity is good too, even if that bug turns out to be invalid.

    How Do You Calculate It?

    Invalid Defect % = (# of defects marked as invalid by team / # of defects opened) * 100

  11. Critical defect rate

    What Does It Tell Me?

    This metric is used to help teams understand how many defects are being opened that are identified as critical.

    How Do You Calculate It?

    Critical defect rate % = (critical defects / # of defects opened) * 100

  12. Test Case Count Per User Story

    What Does It Tell Me?

    This metric can help you understand the average number of test cases per user story (US). It doesn’t tell you if that number is good or bad, as it relates to each user story, but over time it can help you understand the level of effort in writing test cases per user story.  In addition, if you have defect leakage issues with a person or two, you may use this metric to help you understand if their test coverage is adequate. For example, if an underperformer is averaging 2 test cases per US and you have an over performer who averages 10 test cases per US, you could make the argument that their testing isn’t adequality covering the changes, therefore, leading to defect leakage.  This metric shouldn’t be used as a measurement for productivity, as tempting as it may be.  It is a useful conversation starter in coaching employees.

    How Do You Calculate It?

    Average Test Case Count per User Story = (# of user stories / # of test cases created)

  13. % of Test Cases Executed

    What Does It Tell Me?

    This is a helpful measurement to track during your regression cycle.  As your testing team progresses, you can run this calculation to determine the percentage of testing completed.

    How Do You Calculate It?

    % of Tests Executed = (# of test cases executed / total test case count) * 100

Conclusion

There are a lot of options for testing KPIs and as you read through each one, keep in mind why you’re wanting to implement testing KPIs in the first place.  Avoid falling victim to the common pitfalls, especially using these as a measurement of productivity.  There are so many variables in the mentioned metrics and taking them out of context can lead you to focus on the wrong things.  Once you identify what KPIs help meet your goals, make sure you’re communicating appropriately so people understand what KPIs may be used as a success measurement.  Last, but certainly not least, use these consistently so you develop a trend to help identify anomalies over time.  When implemented correctly, testing KPIs can help drive operational effectiveness and deliver quality features to your customers.

About Author

Johanna South

Johanna South has been actively testing or evangelizing software testing concepts for a little over a decade. She has worked in various roles throughout the Software Development Lifecycle, including manual and automation testing, systems analyst, and leading full-stack software development teams. Her broad understanding of delivering quality software gives her a unique perspective in which to pull ideas. Johanna is currently a Sr. Director of Engineering, Mobile Applications for TwinSpires.com. Also, she leads her organization’s QA Center of Excellence and is responsible for ensuring application performance across multiple platforms.

Subscribe to Newsletter

Popular Posts