How Do You Estimate Test Automation Coverage?

Home Forums Software Testing Discussions How Do You Estimate Test Automation Coverage?

This topic contains 2 replies, has 2 voices, and was last updated by  Allan 3 years, 8 months ago.

Viewing 3 posts - 1 through 3 (of 3 total)
  • Author
    Posts
  • #10562

    Allan
    Participant
    @allanwilson

    There are tools for measuring unit test coverage but how do you plan and measure regression / automated test coverage?
    One way is to manually exhaustively list (using good old Excel) all the functionality /unique elements / strings etc and then manually tick off all the parts that are covered by your autotest suite.
    There must be some dymamic way to estimate how much of the product (not just lines of code) have been covered.

    #10569

    Jesper
    Moderator
    @jesper-lindholt-ottosen

    Coverage is too easily gamed – and says nothing about how well you help the customer reaching her goal

    Don’t get excited about code coverage numbers. The only thing you know from 80% coverage is that 20% of your product is completely untested.

    Alan Page
    https://twitter.com/alanpage/status/688434850601263104

    #10571

    Allan
    Participant
    @allanwilson

    Yup.. ‘coverage’ is a misleading word here 🙂
    I think that post is referring mostly to scanning tools for actual code coverage. I am thinking more about how to manually plan a test suite against a complex product. Identifying where to focus your regression tests for best time spent and value.

    Take, for example, the Google.com front page. It’s just one simple page so I could list all the elements to be ‘verified’
    8 clickable links
    1 searchbox
    1 virtual keyboard popup
    1 popup menu with 11 clickable icons

    I want to know how other testers organise their thoughts when tackling a very complex product with 10s or 100s of pages/screens/options. Not for reports to management but to increase confidence in the product quality.

    To play Devil’s Advocate … When I think about it, if I had a report that told me that 80% of the product had been verified (at a basic level) with regression tests then that information would at least point me to the 20% which needs consideration and possibly tests added.

Viewing 3 posts - 1 through 3 (of 3 total)

You must be logged in to reply to this topic.