There is no doubt that the QA process needs to change when a software development project goes to production environment used by real users. This is true for both agile and waterfall projects. This process usually benefits from being transformed to be more formal, well documented and controlled at the cost of speed and efficiency.
Although it is one of the key success factors for the product, it is very uncommon to have a well thought through strategy or technique for it.
These are some of the aspects that need to be considered:
- Agree, plan and communicate process changes
- The whole process with all its details might be affected by the process change
- Change in process for change and release control
- Change in process for defect handling
- Defining clear roles for transition
- Build a bridge from user feedback to the project
- Automation of tests and performance tests
Agree, Plan and Communicate Process Changes
I recommend bringing up the process changes needed in the sprint planning meetings, so not only the stories and the tasks should be discussed in the planning meeting, also the process and the techniques. A good example is that it might not be enough that bugs found in production are just verbally communicated to the developers; they might need to be recorded in a tool where they can be viewed, tracked and handled by all stakeholders. That process change needs to be agreed on and clearly communicated to all relevant people involved.
The Whole Process Will Be Affected By Process Change
The transformation stretches over a big variety of high level and low level QA areas. For example these areas are might all be subject to process change when moving into maintenance mode:
- How detailed the functional specifications should be
- What tools to use when registering defects and stories
- How to communicate
- What information to include when reporting defects
- Process for prioritizing defects
- Documentation of releases
- Test ambitions for verifying changes and scope of regression testing
Every Project is Different
The scope, speed and technique for the process change varies from project to project.
Fewer Changes Needed in Some Projects
Some projects work from day one in a strict and controlled way. They might need less adjustments when going into production, but even the old fashion project will need to change their process in some way. For example, it might be wise to split the team into two – one team working on hot fixes urgently needed in production and one team working on the next big release. Of course a split like that needs to be handled.
Case 1: No maintenance responsibility
A project is temporary in that it has a defined beginning and end time. When the project ends, some kind of maintenance is needed and sometimes it can be quickly handed over to another organisation with an existing process. If that is the case, no transformation of the process is needed.
Case 2: Big bang release in production
If there is a big release date, for example where 1000 of active users are expected the day after the system goes live: a total transformation needs to be planned for. I strongly recommend to make that change gradually and not from one day to another. The closer the project moves to installing the system in production the more controlled the process needs to be. By controlled I mean: well documented, authorization controlled, change controlled and careful.
Even if the project chooses the approach to split the teams and the processes: one maintenance team/process for production hot fixes and one project team/process for the next big release, the project team for the next big release will probably need to change the way they work significantly to a more controlled and restricted way compared to the situation before the system was installed in production with many active users.
Case 3: Small changes in functions and/or user base
If the changes in functions and/or user base are small but constant, I still recommend the same approach as for the big bang release: make small changes constantly.
Change in Process for Change and Release Control
I recommend a Change Control Board (CBB) or similar to be appointed in and early stage, but give the board gradually more and more influence. When the project reaches the point where many users are active, it is probably a good idea to let the CCB or similar decide priorities for most high level changes.
I recommend releases to be well defined, labeled and documented a bit earlier than the point where many users are actively uses the system.
Change in Process for Defect Handling
Defects in an early stage are usually most efficiently handled in an informal way and with direct communication between testers and developers. Later when closer to installing a version in production, more control is warranted and requirements like this for defect handling might be recommended:
- Defects should be reported in a tool designed to handle defects, for example Bugzilla, Jira or HP Quality center.
- Defects should have at least the following attributes: Unique ID, Status, Summary, Description, Reported by, Assigned to, Comments, Priority, and Planned to be fixed in release.
- CCB or other relevant stakeholders should prioritize the defects.
- The fixed defects should be verified by the person reporting the defect or by QA.
In the early stages of a project or sprint all this might not be needed and might make the process slow down with no or only small benefits.
Defining Clear Roles
I think the best way to provide for the transition is to define two different test roles: Sprint Tester and System Tester. By defining the sprint tester’s role as a good fit for the early stages of the project and by defining the system tester’s role as more fit for later stages in the project, the transition can be clearly accomplished by simply gradually adjusting the planned number of man days spent in the sprint tester role and the number of man days spent in the system tester role.
Here is an example how the sprint tester and system tester roles can be defined:
| Description | Sprint test | System test* |
| Main purpose of test | 1. Finding bugs related to changes planned for current sprint 2. Verifying changes planned for the current sprint |
1. Verifying functions from previous sprint and earlier sprints that have not been verified yet (in best case scenario everything relevant has been verified at least once and only ‘2. Regression testing’ needs to be done.) 2. Regression testing functions that might have been affected by previous sprints or earlier sprints |
| Preferred seating location | Beside sprint developer | Beside maintenance developer or beside the rest of the QA team. |
| Participate in sprint planning? | Yes | No |
| When test starts | As soon as function is partially testable | After sprint is completed |
| Documents used when verifying | Jira stories and tasks | Functional specification, User manuals and requirement specifications |
| Test environment used | Local | UAT |
| Tool for reporting defects | Talk to the developer, Skype, Excel (Defects not fixed before the day ends are reported in Jira) | Jira |
| Advantages | Fast feedback to developers, efficient | Good control of what functions are verified, how it is tested and in what version it is tested (tests are well documented). |
| Communication cultureDes | Talk directly with sprint developer and BSA | Talk directly with maintenance developer and BSA |
| Activities controlled by | Scrum team | QA manager |
| Deliverables from planning | Post it notes with planned QA activities for the sprint | Test plan |
| Reporting deliverables | Weekly report, sprint test report | Weekly report, system test report |
| Test cases | No | Yes |
| Regression test | No | Yes |
| Tested version can be recreated | No | Yes. Tests performed on a defined version that should be possible to recreate (many times all regression tests are performed on the same version and that version is a release candidate). |
| Automated testing | Preferable | Preferable |
- ISTQB definition of System testing is: “The process of testing an integrated system to verify that it meets specified requirements”. Ref: http://www.istqb.org/downloads/finish/20/101.html
- Sometimes it is enough to just have sprint test without system test, for example in an early stage of the project before it is close to being in production. Also sometimes instead of having a pure system tester as defined above, a sprint tester and a hybrid (sprint-system) tester can be the best solution. That hybrid tester can, for example, test in a nightly environment doing regression tests without test cases and talking to sprint developers since maintenance developers might not exist yet.
User Feedback
Once the system is used by many active users, it is many times wise to move the main focus from verifying functionality and looking for defects to monitoring production and getting user feedback.
All changes made of course need to be verified before being installed in production. Hopefully less frequent and dramatic changes will be made after the system is in production and therefore also less new/changed functions need to be verified, even if the need for regression tests increases.
The bridge from user feedback can be built by analysing the support organisations incidents. It is therefore important that all incidents/questions/suggestions from production environment are logged in a tool fit for handling such issues. The issues recorded need to be recorded and categorised in such a way that they can help to give a good overview of the issues in production. By making a more detailed analysis and summary of the issues, presumably caused by defects in the system, very valuable information can be assembled for the prioritising activity. I suggest that this activity should be regularly performed and the project/system owner should appoint a responsible person for summarising and presenting this information to the defect prioritising forum.
Automation of Tests and Performance Tests
The expected return of investment for automation should always be considered when making a call on how much resource that should be invested in automation. I think some level of automated unit tests can be motivated in most projects through the whole system lifecycle. Automated UI tests are cheaper to maintain when the UI is changing less frequently and to a smaller extent, that is usually the case when the project move closer to maintenance mode.
Early performance test might be motivated to find performance and architectural problems in an early stage. Once the complexity that might affect the performance has been added, performance tests are even more motivated, that usually happens in a later stage of the project.
