A few years into my career as a functional tester, a customer hired me as a test manager for a new project labeled as “extremely agile”. What I learned earlier was that the way to develop a project from start to finish has changed.
The customer needed a new case management system for their customer service users. They wanted a diverse team consisting of UX-designers, developers, architects and a single test resource (me) delivering fast and continuous in short time periods. Every prototype and design was its own delivery.
To achieve this, we implemented a process called “conceptual development”; a design-driven, lean development process. It focuses on customer collaboration and pushing through frequent deliveries from concept to design to development.
Sounds perfect, so what was the problem?
First, conceptual development turns traditional development upside down. Traditional agile development demands requirements to be planned, developed and then tested at the end of each delivery. But now testing should start right away, starting with the concepts and prototypes first.
In layman’s terms: “Everything is testable”. Testing is extended to ideas, concepts and prototypes as well as the system being developed.
The problem was how we in the project dealt with:
- A new mindset for experienced workers (us)
- A new process for a customer inexperienced with agile methodology
- Repeating common mistakes without realizing it
Now, you may ask yourself: Did we succeed? No, but let´s learn from that.
Teaching an old dog new tricks
Many in the team had not worked with the new process and had to adapt to completely new mindset of agile development.
To further complicate the situation; the customer had never worked in a traditional agile manner, which made the transition bigger
Since there was so much new to both supplier and customer, we should more closely agree on training, expectations management and learning points we learned along the way. This would create greater acceptance for both risk and time, and not least the control of delivery on delivery.
If we had managed expectations earlier and more clear to our customer, we would have shown a better development process.
Know your history (dummy)
Caught up in the new way to develop we made a rookie mistake. We underestimated the need to analyze and utilize past requirements of the system we were replacing.
We started the project with mapping out different concepts, narrowing them down, and then started to develop prototypes. This was a daily process with the customer, but what we forgot in the heat of the moment was to ask: What features needed to be inherited from the old system?
This mistake haunted us later during the test phase when everything was rejected by the customer. We managed to design and develop something the customer wanted, but at the same time couldn´t be used. It only had the new fancy functionality they wished the old case management offered.
When they started the project with us, the customer´s order was a short bullet point list while their heads were filled with knowledge that we failed to extract.
In retrospect, we should have increased our efforts with these actions:
- Analyze the system we were replacing together with the customer/end users.
- Analyze past documentation.
- Create a list of needs, wants and desires based on today´s system and summarize what the new system needs to provide for the users.
- Better learn the domain knowledge to further understand the customer/end user´s needs
Here it is important to fail fast and learn quickly. We should have remembered to ask questions like: Are there any laws or regulations that we need to take into account during our design? Can the customer or the end user complete important everyday processes with the new design?
These are questions that can be answered in simple and low key user testing starting with the concepts and prototypes
Test fast and furious
However, old habits die hard, and test got a dedicated test period starting right after development had finished up their deliveries. That was a mistake.
With an extremely agile method like this everyone should have been able to test, and testing should have started from day one when the concepts were sketched. Instead they were discussed in a more superficial manner and accepted into further development.
This gave the customer a false sense of productivity. The team produced a lot of value for our customer and did the prototype->design->develop phase in an efficient way. But everything halted as abruptly as a car crash when we reached the test phase.
First and foremost, introducing a dedicated test period contradicts the conceptual development process. Secondly, for each phase a testable element passed through, there should be a list of things to check before delivering it to the next phase. This would allow each element to go through a “QA-sub phase” focusing on fail fast-testing and verifying user needs before advancing into further development.
Through workshops, the customer should also have been trained to test and be involved, not only in discussions and decision making, but also in QA. Instead of a traditional customer acceptance testing phase, the customer should test frequently from the start alongside the team members.
In conclusion, what will I do differently the next time?
From the three categories, some suggestions are:
- Learn new tricks:
- Expectation management
- Customer training workshops
- Know your history
- Analyze former requirements
- Map out the new and fancy stuff, but don´t forget the tested and true features the customer knows and loves.
- Test fast and furious
- Early “QA-sub phase” testing
- Customer test training
- Let everyone test
This seems like common knowledge, but that was the main realization after failing the project. Dealing with the change of a new methodology made us forget our experience and make rookie mistakes.
