- This topic has 19 replies, 1 voice, and was last updated 4 years, 10 months ago by
.
-
Posts
-
What are your thoughts on 100% coverage?
Is it something that testers should always be striving for or is it just an objective that is unrealistic that has been put out there?
I read a recent post on the idea of 100% coverage and how going for 100% coverage can just lead testing to a numbers game. What do you think?
Oh, come on… the question was so much covered in classic testing books and lectures that I thought I won’t see it again in any community reading the books and listening to the lectures.
You can not test everything even with simple system(s), and you don’t have to. The test design techniques like equivalence classes, path coverage, decision coverage, pairwise, happy path etc aimed at reducing the number of your checks and testing. IMO in lots of cases coverage is not about numbers at all, but about testing every important thing in the time frame you have for that, or even about testing as much things as we can in strict time frame under some time pressure. It’s good to have covered what you think is important, but just forget about 100% coverage.
100% code coverage is not possible in real time but if you are doing TDD, you should always end up with 80-90% code coverage because you will never add new functionality to the application, a new branch in code, without first having failing tests for that specific new piece of code.
Code coverage may not mean full coverage or even 90% coverage.
… so there are screens A,B,C which may follow in sequence, and you may move between them. You cover code in A, B and C. You think you have code coverage. Then I take the application and do ABA. Or ABABA. Or ABCBABC. Have you covered those? I bet you haven’t. But such paths in mobile application may lead to crashes. I found crashes with paths like that. Now imagine that application can have more screens than there are letters in English alphabet… Will you cover all possible paths between screens? I don’t think so.
”Is 100% coverage possible?” Can I ask coverage of what. I see from responses that code coverage is assumed but it’s not stated in your question. Even within the idea that you might be focusing on code coverage, does it have to be the complete code base? That appears to be implied in responses but not stated in your question. The idea of covering something 100% predicates that we have decided what the whole is. Could it not be a subset of, for this instance, code? If it is something we can cover 100%, then 100% coverage is possible. Why does the notion of 100% coverage have to mean everything possible? Could we instead define this to mean covering everything we believe represents business value to our clients and covering that in ways that give us valuable information about the software? I think we can talk in terms of 100% coverage if we can define the basis of our coverage (which also requires making it clear what we mean when we say “coverage”).
I see at least two diffeerent meanings of “code coverage” here. It would be well to distinguish them.
1. Every line of code has been run through at least once in some test. Clearly you have not completely tested some code if that hasn’t been done. So, it ensures the negative: you are not done.
2. Every possible sequence through the code has been run through at least one in some test. There are very few programs for which it will be mathematically possible to do this in one human lifetime.
This second meaning suggests you are never done testing, but the term “complete code coverage” falsely suggests that you are done. We should never claim “complete code coverage,” just as we should never claim to have built a perpetual motion machine. Both are impossible in any meaningful sense.
In addition to what Jerry has said, I would add this:
X coverage is how much testing we’ve done with respect to some model of X. This applies to code in just the way Jerry pointed out in his first point: you can indeed cover every line of code by running it at least once in some test. But that’s just one way to model the code; as a set of lines. Another way to model the code is as a set of branches, or conditions, or sequences, which is where Jerry’s second point comes in.
Code, though, is only one way to model the product and its relationships to data, platforms, risks, and everything else—including people. Risk coverage is how much testing we’ve done with respect to some model of risk; performance coverage is how much testing we’ve done with respect to some model of performance — and test coverage is how much testing we’ve done with respect to some model of testing.
If you have a sufficiently narrow way to model what you’re testing, you can achieve coverage 100% of that. For instance, if you have a checkbox with two given states, and perform one test with the checkbox checked and another with it unchecked, you can say that you’ve covered both states of that checkbox. Yet you haven’t covered all the states of all the possible factors in the product, on all the different platforms, with all possible data, at every possible pace, for every possible person who might use the product for all possible times… So, Jerry’s point (2) is only the beginning of the impossibilities.
Since we’re dealing with infinities here, coverage in any broad sense is not something that we can quantify successfully. Coverage of the test space is always asymptotically close to zero. But we can model model, and we can discuss its quality — its value to some person or persons who matter.
As you’ll notice in the linked post, I owe much of my thinking on this to Jerry. 🙂
—Michael B.
My opinion is that 100% coverage is not possible in software testing, if we consider to cover all the combination and aspect of a software, specially cross compatibilities features. Maybe the concept of “Minimum Viable Product” help us to have a sense of a target to obtain a software with considerably quality. But not 100% coverage though.
We should only test where the defects are and all other tests are superfluous. Testing functionality that works is wasted effort and provides us with nothing other than the information we should already have, the software works. I think we should be striving for 0% test coverage from “testers”, where the tests are already built into the code and we need nothing other than user interaction and clever monitoring to confirm this assumption.
We should only test where the defects are and all other tests are superfluous.
- I love an example from @jerryweinberg cited once by @michaelabolton (AFAIR): imagine a program without bugs. It is absolutely correct, perfect, no problems, and it outputs a horoscope for a person who died in 1990s. The program may have no bugs, but has no value.
- Apart from bugs we may have, say, performance problems, which are not bugs, but still problems.
We may not know whether particular thing is a bug or not (have no oracle). Or our oracle may be “tell customer and ask how they like it”. But we can’t tell if we don’t know and we don’t know if we don’t test.
- It was said countless times about regression and automation of it. Functionality may change, and the change may affect other code, and to have lots of automated checks may be better than to miss some issue or to re-check many things manually.
This way or that, at any given moment if we change, investigate, develop something, there is something we can’t know, including if there are bugs, or even if something is bug or not.
I received a political bug report once. A Chinese tester filed as top priority issue Program thinks Taiwan is a country. It was a news client. User could choose a country they are in to get weather reports. Taiwan was in the countries list. How one could cover that? To cover that one must know and think of that. None of us did.
We should only test where the defects are and all other tests are superfluous.
That sounds like a great idea. One small problem: how do we know where the defects are and where they’re not until we’ve tested?
Testing functionality that works is wasted effort and provides us with nothing other than the information we should already have, the software works.
This also sounds like a great idea. Where do we get the knowledge that the software works? Should we be confident in that knowledge, or should we be skeptical and challenge it?
There are lots of times when I’ve believed that I’ve removed the last problem from the code and prevented all the other ones, only to find that I’ve made three mistakes: the first is a problem in the product itself; the second is my false belief that the problem wasn’t there; and the third is my confidence in my belief about the problem.I think we should be striving for 0% test coverage from “testers”, where the tests are already built into the code and we need nothing other than user interaction and clever monitoring to confirm this assumption.
This too sounds like a great idea. Historically, many bugs have existed because of errors in code, and it’s a great idea to check the code automatically for problems. We might want to think carefully about that, though, since those checks are expressed as… code. Also, we should remember that confirming our beliefs that we’ve done things correctly is risky for all kinds of reasons. (https://en.wikipedia.org/wiki/Confirmation_bias)
—Michael B.
The idea consists of something 100% predicates that we have decided what is the whole. It may not be a subset of, for this case. I think we should strive for 0% of the test from “testers”.
[commercial link/modified by mod]
We should only test where the defects are and all other tests are superfluous.
- I love an example from @jerryweinberg cited once by @michaelabolton (AFAIR): imagine a program without bugs. It is absolutely correct, perfect, no problems, and it outputs a horoscope for a person who died in 1990s. The program may have no bugs, but has no value.[/quoxte]
Interesting example of what you might find in testing.
What is a bug? I have found that the definition varies a lot from person to person, from company to company.
Now after 10 years of working with testing, I have almost stopped talking about bugs. I call everything “findings”, and report/list those findings in the appropriate way (could be in bug reports, could be verbally in status meetings, could be in a list, in some kind of admin system, mail, etc etc). Any finding will have to be addressed (could be a code change, could be a deployment recommendation, guideline, even a process improvement, etc etc).
What is a bug? I have found that the definition varies a lot from person to person, from company to company.
Hello Alex. Good question. I have a good answer, picked from teachings of @michaelabolton again: “A bug is something wrong in the product for someone whose opinion matters“.
At any given moment one can tell if there is something wrong in the product for someone. If nothing is wrong for anyone important at the moment, than at the moment there are no known bugs (may be known issues, you know 😉 ). If nobody complained about some piece of software for years, we can say that there are no (known) bugs again. At some earlier point of time by someone whose opinion matters the program was described as perfect, no need to add or remove or change something. Until anyone else complains.A small story to illustrate. Our team implemented a piece of functionality, call it FT, according to requirements in August 2017. It was released, all was okay. Until BA came to us in February 2018 and said that what we implemented was a bug. “But wait!” screamed we. “This is exactly what we implemented according to your requirements back in August! And we even protested to do it like that”. “I know”, he replied. “It was initial idea. Now some important customer complained, and so it became a bug”.
I really like the use of the term “finding” to characterize what a test reports out to the world.
I will try to use that term from now on. IOW, “we found this, and though you might be interested.”
One of the nice things about “finding” is it allows you to report things that are not faults, failures, errors, bugs, etc. You could for example, find that “I enjoyed interacting with your app.”
Careful, there, Roman. I don’t think I’ve ever said that a bug is “something wrong in the product for someone whose opinion matters”. I have said, and do say, that a bug is a problem that threatens the value of the product to someone who matters. This might sound like nitpicking, but there’s an important nuance there. There might not be anything in particular wrong in the product, but if someone declares that they perceive a bug, there is a problem in the relationship between the product and that person.
Jerry himself tells a wonderful story about a program that crashed intermittently at unpredictable times. To the people who wrote that program, it was deeply problematic, of course. Yet it turned out to be a wonderful program for training computer operators, who had to learn to deal with the unexpected.
There might not be anything in particular wrong in the product, but if someone declares that they perceive a bug, there is a problem in the relationship between the product and that person.
I meant that there might not be anything wrong in the product on itself, but someone thinks that something is wrong.
There is a cultural issue here. My native languages are Ukrainian and Russian, and in these we do not apply native word meaning “value” to software and products in general. Using the word for value we talk about “spiritual values”, “family values”, we use “values” with meaning “valuables”. But when we talk about software we don’t use word meaning “value”. When my manager wants to talk to me using “value”, he uses English word, not the Russian equivalent. That’s because if you ask someone in Russian or Ukrainian “What is the value of Firefox for you?” using word meaning “value”, you sound like “What is the greatness of Firefox to you?”. It’s not like you can’t speak like that, just nobody usually does it, so it sounds strange. Just like most people never speak like that in our languages, most people, including me, don’t think like that too. So my brain stored the definition “translated” into terms we usually use.
I also thought about “minimal case” when the “product” is some software script I wrote for myself. If I see that something in script work and/or results of that work does not match my expectations, that means that something is wrong for me in the script (and my opinion matters as at some point I can be the only owner and user of that). But I just can’t think of “threatening the value of script for me”. It is just a script, it can work up to my expectations or not. There is nothing about it I perceive as “value”.
Hey Guys,
let me just give you a quick look into my thoughts and experience regarding this topic. I was once hired to spend all-out efforts into a company to make 100% of the software work. This took our time about 3 years. But what was the result of it? In the end they had a well working software – yes. But in the end, it made everything harder for the developers and of course new members, because every single or small change of code leads to the test failed constantly. So every change in the software ends up in more effort than it should have taken.
With further development and the given problem that QA isn’t sexy and nobody wants to actually pay for it. The company also realized that 80% of the software is not even used by most of the customers. So they swap the idea of a 100% working software to software that is usable and perfect at the endpoints a User actually needs. So we followed the Lean testing approach by just checking for the most important Features and searching for symptoms instead of the source of the bug. You should take a look at this topic. It saved a lot of time and money with and no impact on frustrating customers using the software.
But I agree with everyone saying, that it depends on what software you are developing. There is a huge difference between a banking software and an online shop.
Kind regards
Aurel
Hello,
According to me, it depends but in the real-time world it is very difficult to achieve 100% test coverage. It varies from a different application to applications…and major think is application size some test coverage are different as per the application size, code, bandwidth…In the case of a small size project, you can achieve the maximum number of test coverage
One more thing is, With the help of test automation more test coverage, is achievable.
Thanks and Regards,
Priti
- You must be logged in to reply to this topic.