Kim Herzig

Microsoft

Testing as a bottleneck – How testing slows down modern development processes and how to compensate

We often claim the purpose of testing is to verify that software meets a desired level of quality. Frequently, the term “testing” is associated with checking for functional correctness. However, in large, complex software systems with an established user-base, it is also important to verify system constraints such as backward compatibility, reliability, security, accessibility, usability.

The time spend on verification is a lower bound on how fast we can ship software. Anything we can do to increase the effectiveness and efficiency of test execution has immediate effect on product development. Moreover, short release cycles which some projects are on, while very beneficial also force the verification process to be substantially different. To achieve this, we often can’t afford executing the full complement of tests. We need to think of testing as a risk management tool: select the right tests for the right changes and we need to ensure that the test we are executing are highly effective and highly reliable. However, deciding the effectiveness of tests and when to execute them is not trivial.