How Do You Estimate Test Automation Coverage?

[Allan]

There are tools for measuring unit test coverage but how do you plan and measure regression / automated test coverage?
One way is to manually exhaustively list (using good old Excel) all the functionality /unique elements / strings etc and then manually tick off all the parts that are covered by your autotest suite.
There must be some dymamic way to estimate how much of the product (not just lines of code) have been covered.

[Jesper]

Coverage is too easily gamed – and says nothing about how well you help the customer reaching her goal

Don’t get excited about code coverage numbers. The only thing you know from 80% coverage is that 20% of your product is completely untested.

Alan Page
https://twitter.com/alanpage/status/688434850601263104

[Allan]

Yup.. ‘coverage’ is a misleading word here 🙂
I think that post is referring mostly to scanning tools for actual code coverage. I am thinking more about how to manually plan a test suite against a complex product. Identifying where to focus your regression tests for best time spent and value.

Take, for example, the Google.com front page. It’s just one simple page so I could list all the elements to be ‘verified’
8 clickable links
1 searchbox
1 virtual keyboard popup
1 popup menu with 11 clickable icons

I want to know how other testers organise their thoughts when tackling a very complex product with 10s or 100s of pages/screens/options. Not for reports to management but to increase confidence in the product quality.

To play Devil’s Advocate … When I think about it, if I had a report that told me that 80% of the product had been verified (at a basic level) with regression tests then that information would at least point me to the 20% which needs consideration and possibly tests added.

[Author]

Posts