This is the second post in a four-part series on Session based test Management. Read the first post in the series here. In this post Mark will share how to identify test sessions from a feature file.
Feature files are a great tool for team discussion and collaboration that testers should get involved in. We can use our analytical skills and knowledge of the product to ask questions during collaborative sessions to learn about the business’ vision for a new feature, challenge assumptions and establish shared understanding across the team. However, the value of feature files can extend beyond those activities and help us to identify a wide range of exploratory testing activities that we want to execute.
Identifying risks using feature files
As testers, we want to identify as many relevant risks to our product as possible and then explore our product to discover information related to those risks. Feature files are an excellent source to discover different risks by analysing a feature file in different ways. To demonstrate how we could deconstruct a feature file, we will use a series of different techniques to analyse the example scenario below:
User story:
As a hotel manager
I want to give loyalty club members free room upgrades to generate repeat business
Scenario: Gold member checks in and rooms available
Given a guest with a standard room reservation
And the guest is a gold loyalty member
And there are superior rooms available for the length of their stay
When the guest checks in
Then the guest should be upgraded to a superior room
Modelling the system
As we learn about a new feature we can model the system in different ways to observe how the same feature might work in different contexts. There are multiple ways in which we can do this, first by mentally modelling the system. If we look at a feature’s behaviour and think about how it might work in different contexts, we can identify risks based upon that thinking. For example, we could consider:
- How might this feature handle data? There might be risks around:
- Upgrading the booking corrupts the data
- Upgrading the booking deletes the data
- Data being out of date and users being offered an upgrade when there isn’t one available
- How is this feature technically structured? There might be risks around:
- Our back end service going down
- The hotel that we are connecting to being unavailable
- Scheduling maintenance when a user is halfway through an upgrade
- How is this feature working over time? There might be risks around:
- Login sessions timing out during the upgrade process
- Someone else upgrading at the same time as me
- Slow response times upsetting the user
All of these different mental models are based upon a mnemonic from the Heuristic Test Strategy Model called SFDIPOT (or San Francisco depot). It allows you to think about the features in the context of structure, functionality, data, interface, platform, operations and time, to help identify risks.
Another method is to visualise your models. Mapping out your feature or product in different ways will help you identify risks. You could diagram the feature as:
- A user flowchart, to discover:
- Risks around a user deviating off expected paths
- Risks around a user dropping out of the user flow at a specific point
- Risks around too many upgrade steps putting off users upgrading or coming back for repeat business
- A wireframe of the user interface, to discover:
- Risks around accessibility of the upgrade form
- Risks around form validation
- Risks around how it might render across different browsers
Headline game
Sometimes working backwards can help you discover risks that you might not otherwise find with other techniques. With the headline game, you’re encouraged to generate hypothetical news headlines that are reporting something negative that has affected your product or business. From there you work back and think of risks in your product that could exist to cause such a headline to be written.
For example, a headline for our scenario could be:
‘Hotel company loses millions as users exploit free upgrades’
This headline can help identify risks in our product such as:
- A risk that a user could upgrade their account to gold member for free
- A risk that the gold member codes can be reused and are shared on social media
- A risk that the upgrade feature incorrectly validates if you are gold member or not
- A risk that the upgrade feature is hidden but can be revealed via HTML or API manipulation
This is just a small sample of the different risks identified with one headline. Trying different types of headlines might reveal many different risks to test for. You can also try out this activity with user reviews instead of headlines. Mike Talks has collated a series of different user reviews specifically for this task called Oblique Testing.
Personas
Not only are personas a useful tool for identifying and understanding requirements, but similar to the headline game they can be used to work backwards to identify risks. For example, if we have a persona around someone with accessibility needs we could consider:
- A risk that the upgrade form can’t be accessed by keyboard or screen readers
- A risk that the colour scheme of the form isn’t suitable for colour blind users
- A risk that an animation in the form moves so quickly it could trigger an epileptic seizure
We can also use personas to think about different roles people might have when interacting with our scenario. For example, what type of employee interacting with our scenario might make us consider:
- A risk that as a receptionist only partial data is submitted making it harder to book rooms
- A risk that the marketing manager isn’t informed reliably who upgraded and who didn’t
- A risk that the upgrade is not logged, meaning an auditor doesn’t have an official record of the upgrade
It’s a common habit to slip into thinking about risks that might affect you alone. Using personas can help you to be more empathetic of other people’s contexts, which in turn can help reveal more risks. You will want to try this approach with lots of different personas which you will then have to manage. Using a tool such as persona cards can help with triggering ideas for new personas as well as ordering and managing them for future use.
Asking questions
During your collaborative session with the team, you will have asked questions to help the team understand and agree on the behaviour of the new feature. This will bring up risks that you should note down for later, but there might be more risks to identify through questioning.
Taking advantage of pairing sessions with developers along with questioning techniques such as the ‘Five Ws’ (who, what, where, when, why and how) can help reveal new risks such as:
- What part of the feature will handle data validation? Is there:
- A risk that if the front end handles data validation the feature could be hacked with a man-in-the-middle attack
- A risk that if the back end handles data validation the front end won’t render validation errors correctly
- A risk that if there is no data validation then incorrect data could be submitted and affect both the original and upgraded booking
- When will the hotel be notified of the upgrade? Is there:
- A risk that delaying the notification means the upgrade will be lost
- A risk that too many upgrades sent in a short space of time could crash the hotel system
- A risk that if the system goes down any scheduled upgrades might be lost
- How are we going to handle upgrades if the system crashes? Is there:
- A risk around storing draft upgrades in another part of the system
- A risk around allowing an upgrade to be rolled back to the start
- A risk that sensitive information could be leaked into log files
Interestingly, asking these questions during a pairing session might mean specific risks are resolved before exploratory testing occurs. However, it may be dangerous to assume that the risk has been mitigated completely. You may want to carry out exploratory testing nonetheless, just as a lower priority to other exploratory testing sessions.
Creating test charters and prioritising exploratory testing sessions
Once you have successfully identified a series of risks to investigate, the next step is to create test charters around which you will run one or more exploratory test sessions. You can use the following template to create your charters:
Explore <risks>
Using <tool>
To discover <information>
So based on one of my previously identified risks I could create the charter:
Explore the accessibility of the upgrade form
Using only the keyboard
To discover whether I can submit an upgrade or not
With this charter generated, I can then run multiple exploratory testing, sessions based around it. However, before starting any exploratory testing you will also want to discuss your test charters with your team to determine a priority order. Some risks will be more severe to the product or more complex to fix, so you will want to consider running them sooner than other charters. Once you have organised your list of charters into a priority order, you can then begin exploratory testing sessions to satisfy each charter.
Read Part 3 of Session Based Test Management Here
About The Author
Mark is a tester, teacher, mentor, coach and international speaker, presenting workshops and talks on technical testing techniques. He has worked on award-winning projects across a wide variety of technology sectors ranging from broadcast, digital, financial and public sector working with various web, mobile and desktop technologies.
Mark is part of the Hindsight Software team, sharing his expertise in technical testing and test automation and advocating for risk-based automation and automation in testing. He regularly blogs at mwtestconsultancy.co.uk and is also the co-founder of the Software Testing Clinic in London, a regular workshop for new and junior testers to receive free mentoring and lessons in software testing. Mark has a keen interest in various technologies, regularly developing new apps and devices for the Internet of things. You can get in touch with Mark on twitter: @2bittester
