When I read this year’s EuroSTAR Conference theme – “Learning to test, testing to learn” – I thought about the possible most important thing I’ve learned through my testing career so far. To my surprise, the answer was: the more I was testing, the more I realized that the right thing to do was to stop testing.
Whoa, this is a weird thing for a tester to say. After all, would you tell a doctor to stop healing people? Or a teacher, to stop teaching students? But, I kept thinking, and found some good analogies in real life. For instance, testing is a bit like driving. By driving a lorry from point A to B, the driver achieves his objective, which usually is getting some cargo transported. Likewise, testing is not an objective, but a means to achieve this objective, which is to improve the product quality.
OK, I know what you think now. Testing cannot improve the quality, right? Of course it can’t; but I have not yet seen a project which was cancelled because of the bugs found. Rather, the bugs are fixed, and quality improves to an acceptable level. Even if your objective is acceptance testing of someone else’s product, most likely you’re not doing it for gathering evidence against the vendor, but to make the final product better.
Quality counts
Have look at your smartphone. Or your laptop, tablet, or whatever you’re reading this on. Do you know how it was tested? What strategies, processes, techniques, metrics, standards were used during testing process? You might try to find out, out of pure curiosity; but the vast majority of end user’s don’t care at all. For them, there might have been no testing whatsoever, they get the product, it works fine, they’re happy and might buy another one from you. This is what C.E.O’s mean when they talk about quality: make the customers happy, make them want our products, make these products work well. And testing? If you need it, go on. But it’s only a means, not an objective.
Testing is like a gearbox in a car. Everyone takes for granted that you need one. But have you ever wondered what is the reason you need that big, complex, and failure-prone lump of machinery between the engine and wheels? Here’s why: because of the characteristics of engines we use in cars. The combustion engines work well only in a limited range of RPM, and gearing is a smart way of using this narrow range to achieve satisfactory performance at both low and high speed of the car.
What if you could make an engine which has a different characteristics and works well no matter how fast it revolves? Obviously, there would be no need for any gearbox any more. Cars could be less complicated, lighter, and more spacious, just by getting rid of one part.
Does it mean that testing will not be necessary in the future? No, I don’t think so, although it may change. Maybe testing will remain an inherent part of software development. Maybe it will change to some different activity. Or maybe we won’t even be developing software any more in the future, who knows. James Whittaker tackled this topic in his (excellent, in my opinion; watch it if you haven’t yet) talk titled “All that testing is getting in the way of quality” on StarWest 2011.
Air war
The problem with testing is that it finds problems which are already in the product. Despite the apparently common understanding that testing should start as soon as the project does, many test teams get involved only when software is built. Testers are happy to be finding bugs, because sometimes it’s the only way they can influence the product. The uglier the bug, and the more panic it causes, the better, even if this is a day before deadline and everyone is working overtime.
This is similar to a case from 1940s, when air war was fought over Europe, and American aircraft were sent every day to bomb German war facilities. Obviously, the Germans did not like the idea, so they sent their own fighters up there to shoot down the bombers. At first the Americans believed that the bombers are sufficiently armed to defend themselves against the fighters, but this turned out to be a very costly mistake and great losses forced the bombing raids to be stopped altogether in late 1943.
The solution was to send long-range fighters along with the bombers, which would attack enemy fighters and shoot them down. This worked reasonably well at first, but German air force called Luftwaffe also quickly adapted their tactics and sent the fighters in large groups, where the light ones engaged American counterparts, and the heavy ones made their way to the bombers. So still, despite shooting down German fighters, Americans did not really achieved victory, and the bombers kept going down in flames. Furthermore, Luftwaffe pilots bailed out from their aircraft, landed on parachutes, and if they weren’t hurt, next day they were up there again, piloting another fighter. On the contrary, losing one B-17 caused 10 Allied airmen, in the best scenario, to land somewhere on enemy territory, where police or soldiers took them to POW camps.
Software development also went this way, of painful field failures of software (bombers) released without testing. Then a position of software tester (fighter) was created, and the picture improved. But as long as the testers are focusing on executing tests and hunting bugs (Luftwaffe fighters), no matter how many they find, they keep coming. We testers are sometimes like American fighter aces, proud to come back from each raid with another Bf 109 shot down, despite another bomber and ten men lost.
Victory
So how did the Americans win the air war? The answer allegedly came from general James Doolittle, best known for his crazy air raid over Tokyo in 1942. Americans realised something which may seem obvious now: the German fighters didn’t materialise out of thin air; they had some airfields, fuel depots, and needed some time to build the formation before heading towards the bombers. So, instead of keeping the escorting fighters near the bombers (which was perfectly in line with the idea of protecting the bombers, it just didn’t work), US command sent them ahead at low altitude with the mission to seek and destroy any German fighter they could. This tactics (which was not exactly in line with the idea of protecting bombers) worked marvellously, with Luftwaffe fighters not even seeing the bombers, being engaged right after taking off or even destroyed on the ground.
What can we testers learn from that quick history lesson? First of all, we need to keep in mind that there’s some big picture, usually some business or mission, which our testing must serve. If we keep finding critical bugs just before deadline, we’ll end up in not releasing anything; this can keep the customer-originated bug count at zero, but the project will never go into production and become a complete waste. In other words, if we keep shooting down enemy fighters after the bomber is already down, this won’t do well for the campaign.
The other lesson is to go and look for the enemy airfields. Do you know where your quality problems come from? If not, keep some squadron ahead of the main forces, and let them look for the sources. There’s plenty of information in your organisation. If you can’t collect it automatically, just ask people: developers, architects, analysts, managers, whoever you know and trust. You will be surprised how quickly you’ll be able to see some trends. Many people miss this step and, again speaking figuratively, fire at anything that moves, losing all the ammo hunting enemy cows. For instance, 100% unit test coverage is completely useless if the product specification is wrong.
Don’t go armed with machine gun against concrete hangars. Arm the fighters with heavy guns, rockets, or even bombs. That is to say, empower the testers to be able to make a difference. Let them actively participate in development process, and you will be surprised how much they can achieve, before even installing the software.
There is also a hidden catch. The bomber crews cannot see fighters in action any more, so from their perspective, fighters aren’t doing anything related to the mission. Logically, since you aim not to let the bugs in the software, you won’t top the bug hunting monthly ratings any more. Here’s where your communication becomes even more important than ever; you need to find clever ways to tell the organisation what you’re doing and why.
Finally, be realistic. Some fighters will take off and approach the formation, and most likely, they will be the best pilots in best airplanes. You need ace pilots to engage them. Yes, flying up there is not nearly as fun as wreaking havoc on the ground, and uses up precious fuel, but it is still necessary. Even the best software may have bugs inside, and the role of tester is still indispensable, at least until the development process is tough as the force fields in “Independence Day”. Yet, whoever has seen this film knows that even force fields have their weaknesses.
Happy hunting, testers!
