It can be difficult to think about how to improve testing efficiency and quality. A recent EuroSTAR blog post on the Myths around software testing addressed several common myths and challenges facing testing professionals had me thinking about it. The following are a few quotes that specifically speak to some challenges that faced our team.
“Some common myths are that testing is time-consuming, it is expensive, it is tedious, testers are responsible for the decreased quality of application, testing in agile environment is completely ad-hoc etc.”
“If an application is a large one with a lot of functions and features, it would be quite interesting and challenging to test. It requires applying creativity to be productive.”
“Some clients, project managers, and the management team think that a tester’s only job is to identify bugs. If any bugs are missed, the testers would be accountable for the decreased quality of an application.”
I want to share the story of our journey to overcome these myths and improve the quality and efficiency of our testing. Initially, our team had the same issues — testing was considered a bottleneck to releasing code. Project planning generally only accounted for the time needed to code the project. The testers were expected to identify bugs within whatever small amount of time that was left before a predefined release date arrived. Testing always came at the end. Sound familiar?
We knew we needed to change our culture around testing. We knew we needed to shift our testing left to enable continuous delivery. This was a daunting challenge in terms of where to start. We decided to begin by addressing our test design practices. Only by first addressing our test design practices could we enable the automation and execution cadences needed for frequent, high quality releases.
Where we started on Testing Quality Journey
When we began our journey, our test design methods were to write new tests for each new feature or defect correction. Often we spent more time writing the test and getting it reviewed than actually testing the code change. This new test was then added to an ever-growing suite of tests that were used for passivity/regression testing later. The tests in the suite of regression tests were rarely maintained to remove duplicates or conflicts when functionality evolved. When regression-testing time came, there was never enough time to execute the regression suite and there was rarely time for exploratory testing. When issues were found, it was very late in the cycle – causing undue pressure to release with known issues or rushing costly fixes late in the cycle.
The World Quality report 2015-2016 posted by Capgemini, HP and Sogeti earlier this year recommended a focusing more on customer-experience-driven testing style.
“The world is moving towards the customer-experience-driven testing, which means combination of behavior driven testing (where the focus is determined by understanding or analyzing the actual end-user usage of different software application features) and more exploratory user scenario based testing. Furthermore, testing must provide insight into the assurance levels of the different customer journey steps affected by the new/changed IT solution.”
As we considered this recommendation, we decided to take a Living Test approach in combination with exploratory testing to achieve this goal. This improved our efficiency in test planning, maintenance and execution.
What is a Living Test?
A Living Test is a term to describe an approach to designing and maintaining tests for a solution. The test (or group of tests) is modified/expanded over time as the functionality for a solution advances. In other words, the test “lives.”
Contrast that to our original approach where a new test was written for each enhancement project or defect correction. Over time, this created an unsustainable repository of tests.
Automation has always been a big challenge in this kind of test environment. Challenges include:
- To decide what to automate and what not to automate
- No time to automate
- Is it really worth of our time to automate small, single-use functional tests
The Living Test approach helped us overcome a number of challenges – one of which was not equating volume of tests with quality. More is not necessarily better. Sometimes, more is just more. By ensuring we had a maintainable set of tests, we afforded ourselves the time to automate them.
When a defect is fixed or an enhancement is added to a solution, based on the “customer-experience-driven testing” above, we update the existing test if the change occurs within the common workflow. If not, we test the defect utilizing Exploratory Testing methods and document the evidence. This gave us time to execute regression testing early in the development lifecycle to test the impact of change.
Suddenly, we started finding more defects early in the development cycle.
As we teach this approach to other teams, we use the following diagram to help teams transition to this style of testing:
How to Improve Testing Efficiency
This approach has dispelled the notion that testers are a bottleneck to releasing code. The friction between engineering and testing started to fade. The testers are now viewed as partners and engineers look forward to the early feedback that they receive for their code. Management is pleased that testing completes more quickly and when issues are identified, they are caught early in the SDLC.
