My story is from a quite ordinary project where I was in the test manager role. The project was missing experienced test resources and the quality of the requirements was poor or missing. To ensure acceptable test coverage and that we could get the system tested well enough, I had to focus on getting the best out of time and resources. This is the reality a test manager is often presented with. In this article, I will get into some of the decisions we made and why with some examples from the project. Many of the challenges we encountered in this project probably could apply to other types of projects.
[Tweet “The best training is the training you actually do”]
The Best Training…
Oddvar Braa was not impressed by fitness theories and discussions about what you should do to be a better cross-country skier. Oh yes, he had his own training plans and had faith in basic theory. He was more focused on spending his time training. Instead of talking about when, where and how one should work out. I think it is the same with testing. Planned testing does not provide any value. It is the test work the project has delivered that matters. The real challenge is to optimize the test work, with the resources that are assigned to you as a test manager.
[Tweet “The best testing is testing that is actually performed”]
The Project
- Off the shelf IT-solution – Receive project
- Public sector – Healthcare
- Large organization – 4000 end users
- Nurses, doctors, social workers, executive officers
- Migration of sensitive data from existing system
Some of the challenges in the receive project
The test organization was immature. The quality of the test basis was not good enough. The business processes were not documented, nor testable. The project had access to domain knowledge but the availability was limited in certain business areas. Limited time to release was also a key factor. The project organization did not have a testing framework in place. The project resources that were allocated to contribute to the testing process needed guidance and training. The test basis required a lot of work to be testable and to be able to derive test conditions from.
Input to test the strategy- “what should we test and how do we test it”
To be able to present a realistic test strategy, the test manager must consider numerous input variables. Test planning without regard to these inputs may result in wrong decisions for the selection of the test strategy
Fig 1: The availability of resources weighs a little heavier than the quality of the test basis.
On the left side- The Test basis has poor quality; the business processes are not testable. Test conditions are not obvious
On the right side- Domain knowledge is good, but how to convert this knowledge into testable business processes and other test cases will be a challenge
A Realistic Test Strategy
It is very important to present a realistic test strategy. If it’s not realistic, the value of the test process will be reduced if it’s not possible to follow the test plan
Based on these input factors, the test strategy below was chosen. It was important to be clear and avoid complexity. The main reason was avoiding waste of valuable time for functional project resources.
- Test design-“keep it simple” but fit to different groups of testers.
- Test reporting-“keep it simple”, but keep up to date.
- Test coverage-risk based, “happy-path” take priority.
- Documentation of business processes-“keep it simple”, and visually.
- The top level in the requirements hierarchy is the business processes.
To be able to make quick adjustments, the test reporting would have to be understandable and updated all the times. It was necessary to prioritize by using a risk based approach. This was crucial to have sufficient time to work on documenting and testing the most important business processes.
Test Plan for the Receiving Project
The table below shows the phases that followed the planning phase and the test implementation had started. The plan shows the 16 weeks of the completion of the 3 test levels in parallel with the documentation and the development of training materials. To be able to do this successfully, there had to be a high degree of collaboration between the different contributors to this process. It was also important that the various groups respect and understand the work they all do in parallel in this process. The table 1. also shows the main contributors to the phases
Fig 2. shows the sequences in the test plan.
The circle in the middle illustrates that in all the test levels something new was learned and the business process documentation was updated. Defects were fixed and functional design were changed when necessary
Establishing the Test Organization: Test Training
The establishment of the test organizations means to motivate and train those who will help in the testing process. I especially focus on the domain experts and technical project resources that don’t have specific test experience. It is important to restrict the training to specific tasks as it is they are going to perform in the testing process.
- They need to learn how to write test cases directly in a template that can be imported into the test tool.
- They need to learn how to register test results in the test tool, as well as to be able to record defects and help in the error handling process.
It is important to remind the project why we test.
- The best way to learn the system is actually to use the system, not to write documents.
- You will find deficiencies and weaknesses and that we then get the opportunity to reduce the risks.
- By testing and adjusting, the detailed system-specific documentation will get better and that there will be more efficiency in training the end users.
Keep it simple, but repeat often.
Establish a Test Framework
To ensure effective test implementation and execution, it is smart to develop necessary import templates so that more people can contribute in the testing process without relying on knowledge of test tool. We must also prepare the test tool for reporting on both the requirement coverage and test progress.
In the receiving project, we chose to divide the requirements in 3 groups.
- The highest level in the hierarchy is the business processes.
- The next level is the integrations
- The lowest level is the risk components.
It is very important to have a test framework in place before the test implementation starts.
Documenting the business processes
In the receive project, it was important to establish the appropriate level of detail to document business processes. The highest level was visually. It was chosen to use BPMN as architecture language. We thought this was something everyone should be able to relate to and understand, including end users, the business side and the technical resources.
This level should only describe the business process without being system specific. It was chosen to plan and conduct workshops on each business process. The workshops were facilitated by architects who was good at drawing and structuring processes. Contributors were the functional domain experts.
The next phase of the documentation was to make the business processes testable for the end users. This was done by writing step-by-step guides while actually using and learning the system. The documentation was by now, system-specific.
This work was done individually by the domain experts but included a quality control. According to the test strategy we chose to import the activities in the BPMN diagrams as test steps in the high-level test cases.
It was extremely important to instruct the testers to follow the detailed guides even though the details were not recorded as detailed test steps in the test tool. The reason was that we believed that it was more important to focus on working with and documenting the business processes rather than allocating resources in updating the testcases according to the system specific detailed business process guides.
Fig.3 Business process-high level-Not system-specific
Table 2 Business process-activity detailed level-System-specific
Work with integrations and risk components
In parallel with the documentation of the business processes, the technical resources worked with getting the integrations ready for system integration testing. This work did not involve the functional domain experts more than absolutely necessary.
We needed some functional input but tried to get this from other business sources so the domain experts could focus on working with the business processes. The Integrations were tested and fixed mainly by technical project resources.
The risk components were identified and assessed in a workshop involving the domain experts. By risk components I mean any risk that should be tested.
For example:
- We may experience a problem with access to sensitive information
- We may experience a problem with lack of access to the sensitive information
This must be checked.
Test levels – “when should we test what”
Even with limited time and resources, it is important to follow a documented standardized test process in which each test level has clear approval criteria.
- Integrations were documented, tested and prepared for the next level of testing. This work was done by the technical project resources without involvement of the functional domain experts.
- During system integration, the business processes were tested end to end, with no user access control. The tests were conducted by the functional domain experts, who had been involved in clarifying and visualizing processes. The tests on this test level is not system-specific. The actual test execution was carried out in workshops and necessary adjustments in work processes were recorded carefully.
- Testing of risk components were also tested in this test level. These tests were performed using the exploratory test sessions.
- Finally, the business processes were tested in the acceptance test level by end users, using actual access control.
The test and documentation process had by now delivered detailed system-specific step by step guides which were based on the business processes.
Fig 4. Test levels with reference to some of the test goals in each level
Lessons learned
We successfully implemented the new system as planned. The main reason, obviously, was that the project managed to follow the plan and make the deliveries according to the plan.
From a test management perspective, I think that the most important success factor was that we could start focusing on clarifying, visualizing and ultimately, testing the business processes end-to-end. The domain experts who then became expert users of the new system completed this task.
In parallel we focused on identified risk components and testing integrations with minimal involvement of functional domain experts who are often a scarce resource.
Domain experts executed the Acceptance tests by using the low-level documentation they developed. The low-level documentation was made while learning the system and using it to support the existing business processes.
After acceptance testing, the low-level documentation was adjusted and used as training material for the end users. This was possible because the quality had been continuously improved during the process of testing the system and developing the documentation.
- Focus on testing business processes end to end – not only functionality
- Look at test conditions – not test steps
- Target low level user documentation – not test cases
“The best training is the training you actually do” (“the best testing is testing that is actually performed”)
