March 16, 2018 at 4:15 pm #18794@ronanOnly available when logged in
What are your thoughts on 100% coverage?
Is it something that testers should always be striving for or is it just an objective that is unrealistic that has been put out there?
I read a recent post on the idea of 100% coverage and how going for 100% coverage can just lead testing to a numbers game. What do you think?March 17, 2018 at 5:22 pm #18798Only available when logged in
Oh, come on… the question was so much covered in classic testing books and lectures that I thought I won’t see it again in any community reading the books and listening to the lectures.
You can not test everything even with simple system(s), and you don’t have to. The test design techniques like equivalence classes, path coverage, decision coverage, pairwise, happy path etc aimed at reducing the number of your checks and testing. IMO in lots of cases coverage is not about numbers at all, but about testing every important thing in the time frame you have for that, or even about testing as much things as we can in strict time frame under some time pressure. It’s good to have covered what you think is important, but just forget about 100% coverage.March 29, 2018 at 8:59 am #18964@kalpeshdarjiOnly available when logged in
100% code coverage is not possible in real time but if you are doing TDD, you should always end up with 80-90% code coverage because you will never add new functionality to the application, a new branch in code, without first having failing tests for that specific new piece of code.April 10, 2018 at 12:19 am #19109Only available when logged in
Code coverage may not mean full coverage or even 90% coverage.
… so there are screens A,B,C which may follow in sequence, and you may move between them. You cover code in A, B and C. You think you have code coverage. Then I take the application and do ABA. Or ABABA. Or ABCBABC. Have you covered those? I bet you haven’t. But such paths in mobile application may lead to crashes. I found craEshes with paths like that. Now imagine that application can have more screens than there are letters in English alphabet… Will you cover all possible paths between screens? I don’t think so.April 12, 2018 at 11:26 am #19133@paulsOnly available when logged in
”Is 100% coverage possible?” Can I ask coverage of what. I see from responses that code coverage is assumed but it’s not stated in your question. Even within the idea that you might be focusing on code coverage, does it have to be the complete code base? That appears to be implied in responses but not stated in your question. The idea of covering something 100% predicates that we have decided what the whole is. Could it not be a subset of, for this instance, code? If it is something we can cover 100%, then 100% coverage is possible. Why does the notion of 100% coverage have to mean everything possible? Could we instead define this to mean covering everything we believe represents business value to our clients and covering that in ways that give us valuable information about the software? I think we can talk in terms of 100% coverage if we can define the basis of our coverage (which also requires making it clear what we mean when we say “coverage”).April 12, 2018 at 6:41 pm #19142@jerryweinbergOnly available when logged in
I see at least two diffeerent meanings of “code coverage” here. It would be well to distinguish them.
1. Every line of code has been run through at least once in some test. Clearly you have not completely tested some code if that hasn’t been done. So, it ensures the negative: you are not done.
2. Every possible sequence through the code has been run through at least one in some test. There are very few programs for which it will be mathematically possible to do this in one human lifetime.
This second meaning suggests you are never done testing, but the term “complete code coverage” falsely suggests that you are done. We should never claim “complete code coverage,” just as we should never claim to have built a perpetual motion machine. Both are impossible in any meaningful sense.April 12, 2018 at 10:33 pm #19143@michaelaboltonOnly available when logged in
In addition to what Jerry has said, I would add this:
X coverage is how much testing we’ve done with respect to some model of X. This applies to code in just the way Jerry pointed out in his first point: you can indeed cover every line of code by running it at least once in some test. But that’s just one way to model the code; as a set of lines. Another way to model the code is as a set of branches, or conditions, or sequences, which is where Jerry’s second point comes in.
Code, though, is only one way to model the product and its relationships to data, platforms, risks, and everything else—including people. Risk coverage is how much testing we’ve done with respect to some model of risk; performance coverage is how much testing we’ve done with respect to some model of performance — and test coverage is how much testing we’ve done with respect to some model of testing.
If you have a sufficiently narrow way to model what you’re testing, you can achieve coverage 100% of that. For instance, if you have a checkbox with two given states, and perform one test with the checkbox checked and another with it unchecked, you can say that you’ve covered both states of that checkbox. Yet you haven’t covered all the states of all the possible factors in the product, on all the different platforms, with all possible data, at every possible pace, for every possible person who might use the product for all possible times… So, Jerry’s point (2) is only the beginning of the impossibilities.
Since we’re dealing with infinities here, coverage in any broad sense is not something that we can quantify successfully. Coverage of the test space is always asymptotically close to zero. But we can model model, and we can discuss its quality — its value to some person or persons who matter.
As you’ll notice in the linked post, I owe much of my thinking on this to Jerry. 🙂
—Michael B.April 13, 2018 at 11:23 am #19145@msalazar18Only available when logged in
My opinion is that 100% coverage is not possible in software testing, if we consider to cover all the combination and aspect of a software, specially cross compatibilities features. Maybe the concept of “Minimum Viable Product” help us to have a sense of a target to obtain a software with considerably quality. But not 100% coverage though.April 13, 2018 at 3:49 pm #19166@shaneksaltaireOnly available when logged in
We should only test where the defects are and all other tests are superfluous. Testing functionality that works is wasted effort and provides us with nothing other than the information we should already have, the software works. I think we should be striving for 0% test coverage from “testers”, where the tests are already built into the code and we need nothing other than user interaction and clever monitoring to confirm this assumption.April 13, 2018 at 9:44 pm #19167Only available when logged in
We should only test where the defects are and all other tests are superfluous.
- I love an example from @jerryweinberg cited once by @michaelabolton (AFAIR): imagine a program without bugs. It is absolutely correct, perfect, no problems, and it outputs a horoscope for a person who died in 1990s. The program may have no bugs, but has no value.
- Apart from bugs we may have, say, performance problems, which are not bugs, but still problems.
We may not know whether particular thing is a bug or not (have no oracle). Or our oracle may be “tell customer and ask how they like it”. But we can’t tell if we don’t know and we don’t know if we don’t test.
- It was said countless times about regression and automation of it. Functionality may change, and the change may affect other code, and to have lots of automated checks may be better than to miss some issue or to re-check many things manually.
This way or that, at any given moment if we change, investigate, develop something, there is something we can’t know, including if there are bugs, or even if something is bug or not.April 13, 2018 at 10:25 pm #19168Only available when logged in
I received a political bug report once. A Chinese tester filed as top priority issue Program thinks Taiwan is a country. It was a news client. User could choose a country they are in to get weather reports. Taiwan was in the countries list. How one could cover that? To cover that one must know and think of that. None of us did.April 13, 2018 at 10:57 pm #19169@michaelaboltonOnly available when logged in
We should only test where the defects are and all other tests are superfluous.
That sounds like a great idea. One small problem: how do we know where the defects are and where they’re not until we’ve tested?
Testing functionality that works is wasted effort and provides us with nothing other than the information we should already have, the software works.
This also sounds like a great idea. Where do we get the knowledge that the software works? Should we be confident in that knowledge, or should we be skeptical and challenge it?
There are lots of times when I’ve believed that I’ve removed the last problem from the code and prevented all the other ones, only to find that I’ve made three mistakes: the first is a problem in the product itself; the second is my false belief that the problem wasn’t there; and the third is my confidence in my belief about the problem.
I think we should be striving for 0% test coverage from “testers”, where the tests are already built into the code and we need nothing other than user interaction and clever monitoring to confirm this assumption.
This too sounds like a great idea. Historically, many bugs have existed because of errors in code, and it’s a great idea to check the code automatically for problems. We might want to think carefully about that, though, since those checks are expressed as… code. Also, we should remember that confirming our beliefs that we’ve done things correctly is risky for all kinds of reasons. (https://en.wikipedia.org/wiki/Confirmation_bias)
—Michael B.April 20, 2018 at 12:43 pm #19219@alishahendersonOnly available when logged in
In the past, it didn’t matter how long it took for software to go through the testing process since users were waiting with baited inspiration for the latest version.
Nowadays, users know that the first version of any software is not reliable, so they will usually opt to wait for the second or third iteration rather than, which results in fewer sales for the first version.
Therefore, it’s really necessary to ensure the first version of your software is reliable. Follow the guidance in this blog, thus keeping everyone happy.
Software Processes which are part of the Software Testing Life Cycle:
Test Environment Management: The setup and configuration of a test environment are necessary for the procedure of software testing.
Virtualization tools are utilized to make sure perfect environment replications and timely deployment of the Test Lab.
Defect Management: Defect management includes tracking & analysis of the errors found during the testing period. This procedure needs tools like Rational Clear Quest & other bug and defect tracking tools.
Test Automation: Test automation consists of automated tools for the creation of test cases and their analysis. Testing on e-commerce applications also called e-testing is also a part of test automation.
You must be logged in to reply to this topic.