This paper highlights the effects and challenges of moving to agile in software testing. It starts with a brief explanation of all Software Development Lifecycle’s (SDLC’s) and what is involved in moving onto Agile. It continues to explain the basics of test management in Agile Projects and the important aspects to be kept in mind. It also touches upon factors that make agile testing different from traditional methods and the factors to be observed to make sure that testing in agile is successful. Challenges that a software tester faces when testing in agile are discussed. Finally it also points out the things to keep in mind to avoid failure of agile test methods and for achieving excellence in agile test management.
Software Development Lifecycle
The changes in focus of software development have given rise to many different software development lifecycles (SDLCs). Each model has its own characteristics that influence the way of testing and test management.
The oldest model, waterfall, is a sequential model in which each phase in the cycle begins only when the phase before it completes. This makes test management an easy task as there is ample clarity of requirements and time to generate test cases and scripts and for it execution. Since the functionality is frozen, there is very little flexibility that can be expected. Hence, along with the pre-set formal rounds of testing, regression tests form a major chunk of the testing.
The V model which is an extension of the waterfall model is also sequential but offers more flexibility than waterfall model. During each phase, use cases corresponding to that level are derived but the test execution starts only after a formal functional requirements release. Since there is a feedback channel, some requirements may get affected which may in-turn trigger test specification and/or test script updates.
Iterative (time-boxed) model delivers fully functional versions to the customer at fixed intervals where the iterations use V-model and have their own set of requirements that do not change. For each version, you not only have to test the new functionalities added but also its impact on the already existing functionality and also the open defects from previous versions.
Why Agile Testing?
Agile is a fast paced development methodology that is seeing increasingly widespread adoption. Users aren’t questioned extensively for exact requirements for the whole system while the developers are given the freedom to express their creative talents and imagination to elaborate on the requirement in a way which would suit the end user.
The most commonly used agile methodology is SCRUM where fully functional sub-versions will be released after each phase/sprint. This approach poses a threat to the tester as now they will not have a fully finished product to verify against a complete requirement specification – they have to test and validate a changing target against a moving backdrop!! Also responding to change over following a plan.
Agile Test Management
The most important thing to keep in mind while managing software testing activities in an agile project is that when a sprint finishes, all test activities for that subset of requirements including testing have to be finished. There is no specific period to write test cases or scripts. In such a scenario, testers usually resort to ad hoc testing. But missing test documentation poses a huge threat on regression testing which can be said to be the most important tests given the nature of agile development.
Some important aspects of agile testing to be kept in mind are:
1. Define the test scope
The testing scope should clearly specify what will be tested and what not, for each iteration. This can be decided based on the changes or additions in each sprint.
2. Define the test goal
Consensus has to be there among all the stakeholders as to what testing is needed to achieve the desired goal of the project. The test intensity, levels and types of testing needed to achieve the goal have to be identified.
3. Define test Strategies
Efficient test strategies have to be in place. Decisions such as, what test data/environment will be used, what tools should be used to prepare and run the tests, what qualify factors should be used based on quality risks to design the test cases etc. should be made well in advance.
4. Specify metrics and stopping point
Clarify about what metrics are to be used for follow-up of testing and to decide whether to continue or stop testing. A final acceptance criteria sheet should be prepared by the team with inputs from all the stakeholders for the output of the iterations as well as for the final delivery.
Is Agile Test Management same as Traditional?
Traditional requirement-based software testing techniques such as state transition testing and use case test design techniques will be less frequently used and replaced with more agile testing techniques such as risk based and exploratory testing.
The two principles of the Agile Manifesto that will change the traditional testing ways are:
- Working software over comprehensive documentation
- In order to overcome the lack of detailed requirement and to produce working software, testers will have to rely on Business Analysts or Product Owners and subject matter experts (Architects) to determine the test scenarios for the application under test. User stories often contain inadequate information for the test cases to be determined. This is where the use of exploratory test charters to document the test execution over following a pre-scripted test script would be beneficial. The results of the exploratory test charters can be discussed and verified by the business analyst. The beauty of exploratory testing is that we can begin testing with a minimum amount of test documentation and guide the testing and document the results as we go. With the focus of Agile on “Just enough documentation” at the early product development phase. After the input documents are freezed, detailed documentation is recommended and to be made available “just in time.” Hence the structure should be created and made ready with the quality test scenarios.
- Responding to change over following a planThe agile development lifecycle has proven that it can deliver applications into production in a relatively short space of time. Testing needs to adapt and change with the project. The tester would need to identify the right techniques to get the job done. As more is demanded from the testing team in less time, testers should come up with a plan to work smarter with the help of the right testing tools and test strategy.
- Testers should write a brief test plan for every iteration with just enough information to guide the testing through each sprint. Test objectives, test deliverables and test milestone for each sprint should need to be clearly distinguished, if testing is to add value to the agile project.
Testing in an organisation going Agile
When an organization chooses to embrace agile methods, testing must undergo a lot of changes to fit in and add the most value. Testers must possess the proper mindset and expertise. Some insights that may help the organizations make the changes in testing successful are:
- Tester involvement all through
We all know that early testing can save lots of time. Agile teams can take this to extremes. Providing test scenarios to developers during design and testing code immediately after it is integrated. Testers should talk to customers or have more face time with POs and help improve requirements long before they enter the iteration. As such the added value of testing is undeniable.
- Testers with ‘Agile Gene’
Training should be given in the basics and the underlying principles of the agile way of working, to ensure that it is properly understood and executed. As a tester, your primary function is to facilitate the team by giving fast feedback. This stresses the value of test execution over test design. Often you will have to be flexible and change testing tasks at a moment’s notice. Doing that which adds the most value to the team at any given point in time, that’s what agile working is all about.
- Tester without ‘acclimatization’
Tester’s overtime with repeated exposure to the system will become more tolerant of failures. Always testing should be done with a tester’s mindset. The final validation and doneness test should be performed on a clean build without allowing any workarounds.
How Agile challenges Test Management
One of the principles of agile development is that project teams should “welcome changing requirements, even late in development”. Analytical requirements-based testing strategies become quite inefficient in such situations. However, risk-based testing accommodates change, since we can always add risks, remove risks, change risks, and adjust the level of risk. Since risk-based testing provides an intelligent way to decide what to test, how much, and in what order, we can always revise those decisions based on new information or direction from the project team. However, change can impose challenges for testing that are independent of the use of these test strategies. Many of these challenges arise from change in the definition of the product and its correct behaviour. When the test team is not kept informed of these changes, or when the rate of change is very high, this can impose inefficiencies on the development, execution, and maintenance of tests.
Agile methodologies are less formal and faster moving. The sprints use short, fast-paced iterations. This pace and brevity further squeeze the test team’s ability to develop and maintain test systems, compounding the effects of change noted earlier.
The risk-based element of the recommended strategy can help. Risk-based testing focuses on the important areas of test coverage, and de-emphasises or even cuts less important areas, relieving some of the pressure created by the short iterations. This ability to focus proves especially helpful for test teams also under tight resources constraints. Test teams in an agile world should develop, maintain, and execute tests in risk priority order. Using risk priority to sequence development and maintenance efforts allows the test team to have the most important tests ready at the beginning of each sprint’s test execution period.
In iterative lifecycles, though, code that worked in previous sprints has behavior changes by new features in each subsequent sprint. This increases the risk of regression. Agile methodology advocates emphasize good automated unit testing in part to manage the regression risk inherent in such case. However, good unit testing has limited defect removal effectiveness. So, automated regression testing via unit tests will likely miss most of the regression bugs. Therefore, we need effective regression testing at the System test level, which has a higher level of defect detection effectiveness. By combining risk-based testing with the automated regression testing, test teams can effectively manage the increased regression risk.
Test basis for each requirement should be defined. The test basis also provides a means to measure the results. In a requirements-based test strategy, testers can’t report test results accurately if the requirements are missing or poor. Software testers can’t report the percentage of the test basis covered by passed tests, because the requirements won’t provide enough detail for meaningful coverage analysis. With the risk-based testing strategy, testers use the quality risk items. They design and implement test cases based on the quality risk items. The level of risk associated with each risk item determines the number of test cases and the priority of the test cases derived from that risk item. The test team can report test results in terms of quality risks mitigated versus not mitigated.
The challenges cannot be overcome, if an independent test team exists. The test team, lead by a good test manager, can introduce centripetal forces that will bind the team together and makes its actions consistent and congruent. One approach to managing these challenges is to have a focus on the corrective action to get a test project on track by monitoring test metrics and reporting. A test Manager needs to check the premature entry of product into different test levels and unnecessary delays. Also have a separate test cycle that follows the development sprint.
Why does Test Management in Agile fail? – Things to avoid…
Agile projects are in fact an excellent opportunity for the testers to take leadership of the agile processes, who is better placed to bridge the gap between users and developers, understand what is required, how it can be achieved and how it can be assured prior to deployment. But it requires focusing on techniques to optimize a new strategy to testing.
Why Testing at the End Doesn’t Work in Agile:
- Implementing Agile through traditional methods
The biggest and most important reason for failure is that people try to implement Agile Testing with traditional methods in their minds. Following waterfall model with exhaustive planning, designing test case by requirement-based testing techniques and improper priotization of the tasks with no transparency in work assignments. This ends up in partially done work that is not completely done as per your Definition of Done. When you have stories lined up (committed) for a sprint, do not try to manipulate them during the sprint. It will add unnecessary rework and will lead to a loss in velocity. The keys to prioritizing these stories are value, cost, and risk.
- Team Spirit
Agile projects involve pairing up and integration and they require team spirit from all team members. That is, the team members should not only be technically very sound, they should be socially sound as well. Further, communication also plays an important role in agile projects for asking and giving help. Not using the knowledge that is available within the team members, trying to reinvent the wheel lead to waste. Here, all you have to do is communicate, complete & deliver. An Agile Testing project will fail if the team members are not socially communicative and have poor social skills.
- Mistakes continue unnoticed
- Ad-hoc testing with no-strategy and no-plan for the testing. One must plan for an Agile Testing project for every sprint. The planning may include first test-driven development, unit testing, test-first approach like Bring up testing, testing early and iteratively, testing frequently, use of methods like regression testing, exploratory testing, acceptance test testing and finally following a context-driven approach to testing.
- It is hard to improve the quality of an existing product
- Testing is more likely to be cut, because of deadline pressure, work that is planned to happen at the end of the project is more likely to be dropped or reduced. Hence not enough testing performed. Missing acceptance criteria and the spillover of the testing activities adds to product risk. It is important for team to plan and execute the strict discipline with no delays in the builds for enough testing, including regression testing after the retesting for the fixes provided.
- Proper requirements and early feedback is all that is needed
Agile projects are different in that you cannot expect loads of requirement documents. The fundamental principle of agile is based on collaboration and integration and not documentation. And this does not spare the customer too because he is the one driving the project and defining the requirements. The customer ideally should have his acceptance ready and should be clear about his requirements. He should be available to give early feedback and clarify acceptance criteria, so that frequently delivered pieces of software remain in sync with what he wants.
Basically, any project will fail if you do not follow the basic principles of the methodology you are using. Test Management in Agile requires you to think about Agile. All flavors of Agile emphasize collaboration, communication, integration, simplicity, respect, feedback and the ability to adapt to changing business requirements. If you miss any of these, your project may fail.
References
- Agile Testing: A Practical Guide for Testers & Agile Teams – Lisa Crispin & Janet Gregory
- Succeeding with Agile – Mike Cohn
Additional Reading
If you liked this article on Test Management in Agile, we recommend you download this free agile testing eBook from Lisa Crispin & Janet Gregory which looks at Roles & Competencies on Agile Teams.
