Last month I wrote about a revised approach for obtaining a test strategy. I explained a three-step approach in which we first defined persona’s and then searched for features that these persona’s would love about the product, or features that would make them throw away the product. We called these features qualifiers and disqualifiers. You can read the previous column here on the community pages: G(r)ood testing 9: a test strategy revisited. In this month’s column we’ll take the story a little further and see how we can use this information as basis for user centered risk based test reporting
In my spare time I am an enthusiastic home barista. A barista is someone who likes coffee, and that describes me pretty well. I have very much pleasure in trying to brew the God Shot espresso, make a beautiful café latte and I occasionally roast my own beans. But ultimately I like to taste. I always surprised by the different flavors that a coffee can have. Depending on the bean, the brewing method and the roosting even an espresso can have citrus, red berries or nuts flavors.
When tasting I regularly use the taste wheel. This a picture that notes flavors that coffee can have. These flavors are neatly grouped together; the berries like flavors with the berries and the chocolate flavors just off the nuts. The test wheel helps you to identify and recognize flavors and it helps you to express what you taste.
All rights: ©1995 Specialty Coffee Association of America
Testers are also tasters. They taste the application and try to explain what benefits the application has (the good tastes) and what might put the users off (the bad tastes). With mobile apps, it’s the qualifiers that will make you want to show your app at, lets say, a birthday party. It’s de disqualifiers that make you remove the app from your device. But the principle holds with desktop applications, or big services as well. Testers try to identify/locate the flavors and report them.
But how do we do our reporting? How can we be accurate, complete and report is a language that is understood by the stakeholders? Many testers still report a list of ticked off test cases accompanied with a list with found errors. But I think we should challenge ourselves to tell a more colorful story. We have to. Bugs are no longer in long lists but on volatile post-its. Tests are not completely written down and UX is becoming more important. We need alternatives. We need to tell an appealing story in explains why we think we did the right tests. We should express what we think of the system. For this we need to do the right tests, of course, but a good test story starts with a good strategy. The taste wheel for testers can benefit us is several ways.
First it can help us with identifying tests that we can execute. It serves as a checklist suggesting test topics that are valid within our organization. Just like the taste wheel helps me to search for certain flavors; “Do I taste a hint of citrus?” A testers taste wheel can help us to look for particular attributes in a targeted way. I imagine a tester reasoning in the following way: ”The taste wheel states that unwanted sharing of private data over the network is regarded as a disqualifier, can I define a test mission to cover this item?”
Secondly, including this information in our test story makes that we can communicate what we did a tester. In our example, simply explain what test you did in order to verify that no unwanted data is being shared.
Thirdly if we sort and group the attributes, we get a feel of the weight of the identified items. In the workshops that I give on this topic we define personas as a group and brainstorm on qualifiers and disqualifiers. We put them on the board and sort them by category and user group (remember we started with persona’s as a starting point). This instantly makes it visible what attributes are regarded as a big influence on the user rating of the system. It also visualizes what user-groups are affected in a positive of negative manner. This enables us to decide on an effective test strategy: where do we want to put our effort?
Example of a workshop output: Qualifiers and Disqualifiers grouped by category on a improved ‘taste wheel’
If we involve the stakeholders in our workshop, they will understand the personas, the qualifiers and disqualifiers. Therefore they understand why we do our tests. We already saw that the (dis)qualifiers are a great starting point for us to tell our test story, so we can share our experience more easy and in a language that is understood by our beholders. Stakeholders in their turn can chose how many details they want. The story thus becoming a dialogue. It can be expected that they want to hear more about those items that affect a large group of users, or that were estimated to have a great impact. And that completes the circle, using the taste wheel as basis for our risk analysis and test strategy. Test reporting transfers from boring lists to a user centered risk based dialogue about quality.
