Kimberly Snoyl of the Netherlands was awarded the EuroSTAR 2022 RisingSTAR Award for her idea her idea to create a HeuriChecker to Automate the Usability Heuristics. See more details. Please see below the latest update on her progress. You can find out more about the RisingSTAR Award on our main award page.
It has been a while since I updated you about what I have been up to since winning the RisingSTAR award in June 2022. In this blog I would like to take you along the journey so far.
Is there a need for a “UX Tester”?
First of all, thank you to those who filled in the questionnaire on Testing Usability in the previous update. I’m very happy with your response to my request; 101 people filled in the form. Although this is technically too few for quantitative research, and more than is needed for qualitative research, the rich responses provided a lot of information and useful indicators.
My main hypothesis for the survey was:
“Most testers don’t test the usability or general UX of the applications they test because they don’t have the knowledge of how to test it. “
The questions I asked in the survey included: Do you test the usability of the front-end applications you test and what training or certificates do you have in the field of testing?
Your answers indicated that my hypothesis is partially correct, and also showed some other potential blockers to UX testing.
Key findings:
- Many testers would welcome more education and training about UX and usability;
- Organizations and managers need more education and understanding about the value of UX and usability to the customer and the business.
About half of the testers test Usability
What came out of the survey is that about half of the testers who responded DO test usability (52%, see Figure 1). This means that the other half of the testers (48%) don’t (always) test usability.
Reasons why testers don’t test UX
There could be different reasons to why they don’t test the usability. My assumptions while putting together the survey were:
- Testers have not learned how to test usability enough in their education in becoming a tester. This was the case for me, I have a TMAP certificate (2014), but Usability was a very small part of the syllabus back then.
- Testers don’t have enough time to test usability. In the test education I had, the focus was on how to test the functionality and what test techniques to use. When we talk about usability or UX in general, there were no techniques mentioned on how to test it in a structured way. So we go with our gut feeling and use exploratory testing and use the application as we think a user would: press buttons you normally wouldn’t press or press a button 20 times. For me that is not what UX is about. Usually when users do unexpected things it is because they don’t understand the application, or the application does not give appropriate feedback. It should be made more clear what is expected from them and better yet: don’t offer functionality they are not allowed to use in certain moments!
- UX is not a main priority to the business. I noticed that when I found a usability issue or improvement, it was usually not important enough to the business. They would say that the functionality works, so it is good for the MVP and when there is time left, it can be improved. Also, in some companies the pace is quite high, with a focus on test automation – and since a lot of testers (like me) are not developers, test automation takes up a lot of time. In this case, the priority of the tester is making sure that the functionality works according to the functional requirements and automating the tests. Usability testing does not really make it into the priority list for the tester. In some projects I could bring my team to do exploratory testing with the entire team, but if the usability findings are not encouraged by the business, it is inevitable that they will not keep testing usability. However, I did not know if this was a greater or lesser blocker than knowledge of UX and usability.
Usability is not a priority to the business
According to Figure 2 there are five reasons why people don’t test usability, but a large number of the responding testers (42%) say that it is not a priority to the business, which I found is an interesting finding.
Even though I noticed this in my own career as well, somehow, I thought the problem was with the education of the tester and that they needed a structured way of testing usability. That’s why I want to develop the UX-Tester tool to help testers learn more about testing usability and speeding up the process through automation. While the survey shows that is still a valid motivation for this project, the result shows that is even more pressing that the business needs to become more aware of the importance of UX. Therefore the “UX tester” tool should also support the testers in convincing the business. Thinking about this result and discussing with Isabel Evans who helped me review the data, it is actually not such a surprising result at all, because what is important to the business reflects in the education which is offered to us. I will write another blog about this, because I think it is important to address this issue that most of the priority of the business is on test automation, while this might not be the best way to make a difference for the end user.
A significant proportion of testers struggle with testing usability
The next part of the survey explores the following questions: do people struggle with testing usability and if yes, do feel that they need some guidance? In my talk “UX Testing in Agile teams”, I talk about different ways in which testers can test the usability of their applications. One of the ways is using the 10 usability heuristics as a checklist. This is because I think that testers like to test in a structured way. Testing is a quite repetitive process: for each functionality and each release, you must do the same things: make test cases from the acceptance criteria, think about which test-techniques to use and think about which quality attributes should be applied. Structuring your work helps you not to forget anything, and checklists are a way to support that.
About half of the testers test Usability/UX (Figure 1), but that does not mean that they don’t struggle with it. Of the testers who test UX, Figure 3 shows that more than half of them (65%) struggle with it.
How come people struggle with UX testing? As I mentioned: in my opinion, this is not extensively treated in the training that I personally got when I started testing in 2014. Looking at the training people had combined with their years of experience in the field, we can see that many people have ISTQB training, or other certification/training, but there are also a lot who don’t have any training at all (see Figure 4).
Test training does not cover usability testing enough
I was wondering if they feel that usability testing was covered enough in their training, so that was also a question in the survey. According to Figure 5 this is NOT the case for 75%!
Since some people don’t have any test specific education at all, this can be more nuanced:
Combined with the years of experience, we can still derive that most of the people that had a certificate don’t think that the training covered Usability testing enough (see Figure 6).
This confirms my hypothesis that testers don’t really know how to test usability because it was not covered enough in their training. Although I know that ISTQB has recently a Usability Testing certificate to their offerings, we could take a closer look at people who are new to the field. You can see in Figure 6 that the people with 10 years and above (yellow bar) indicate that that their training has not covered Usability testing enough but also of the people with less years of experience, most of them feel the same way. It would be interesting to dive deeper into the reasons behind this. Since UX is not a priority to the business AND test automation takes a lot of time to learn for testers who are not really developers, I can imagine they’d rather take a course on test automation than on usability testing. But this is again my assumption. I would love to hear your take on this as well!
How do testers test usability?
In the survey, I also asked how testers go about testing usability to see if they do this in a structured way, or mainly using their intuition as I suspect.
Most testers use their intuition when testing usability
Figure 7 shows that 42% of the testers use their product under test as a user would (use their intuition). I also count exploratory testing as part of this category:
Some testers work together with a UX professional
Another interesting finding is that 20% has indicated that they work together with a UX professional (Figure 7), but in Figure 2 some people have indicated that there is a dedicated UX team so they don’t need to focus on UX. I would say that is there is a dedicated UX team, I agree that the testers would not need to focus on UX testing as much (and it shows that UX is a priority to the business), but in the case of there only being one UX professional they work with, I would still recommend to test UX as well, since the single-expert UXer on a team can be biased, especially if this person does not have a lot of experience, or did not take the time to do usability testing (it happens!). Having more people take a look at the design can have a positive effect on the user experience.
Some testers use tools for usability testing
17% (26 testers) have indicated that they use “other” measures. When checking what these “other” measures are I can re-design this graph into 4 groups (Figure 8):
- people use tools
- people don’t use tools
- people who ask actual users or representatives and
- people who do all or a mix of those options
Figure 8 shows that still most of the people who test UX don’t use any tools (56%) and the rest does use tools like the heuristics and even automatic tools (19%). What didn’t surprise me is that only 4% has time or makes time to test with real users, even though that is considered by Usability experts to be very effective.
It is interesting that 19% already use some tools! Let’s check out what tools they use (some of them are also new to me):
- Accessibility tools:
- WCAG (Web Content Accessibility Guidelines) or VPAT (Voluntary Product Accessibility Template). I didn’t know about the VPAT, but the WCAG is known to me.
- Lighthouse
- Contrast checker
- Wave tool
- Axe
- Ghost inspector
I know that there are a lot of tools in place for accessibility already, but remember UX and usability are about more than accessibility. There are not enough tools for usability or other UX measures, which is why I think a tool especially for usability would be well received.
- UXcheck and Heurio: I knew that these tools exist, but I have not used them myself yet. They can help you test the Usability Heuristics but they are not automated.
- CAN I USE THIS Mnemonic: Wow, I did not know about this, but happy to see that there is someone else out there who is working on the topic of making Usability Testing easier for Software Testers: David Greenlees. The Mnemonic is quite long though, so I wonder if people can really remember it, but it suits to be used as a checklist as well. It contains interesting material, including the 10 Usability Heuristics!
- General GUI test automation tools which people have mentioned and which I have used in the past as well. These tools can be used to automate functional testing, but do not specifically point out usability issues by themselves (io, Cypress, Applitools Eyes, Galen framework, Ghost inspector, Squish).
I was positively surprised that there are testers who user UXcheck and Heurio and about the Mnemonic and I am curious to hear about their experiences with these tools! However, I still see potential for the UX-Tester tool since these tools are not testing usability automatically. The UX-Tester tool would work in a similar way as the accessibility tools that already exist: automatically checking certain acceptance criteria, pointing out any violations and foremost informing the user about the guidelines. Accessibility is a part of usability, so I am happy that these tools already exist, but there is more we could automate when it comes to usability.
Next steps
Unfortunately, I have not yet found a developer to help me develop the UX-Tester, so I got the tip from the RisingSTAR support team to make it an open-source project! I don’t have any experience with open-source, so if there is a member of the EuroSTAR community who can introduce me to the open-source community, that would be awesome! Please contact me on email.
In the next update blog, I will go into the topic: should we automate the usability heuristics, since some people say they are subjective and dependent on the context. Some people even say they are outdated. I have done some research on this and even given a workshop on this topic for my colleagues as Capgemini, so I would like to go in depth in the next update, very soon!
* I would like to thank my dear friend and supporter, Isabel Evans, for helping me make sense of all of the data that came out of the survey and who also helped me by reviewing this blog.
The 2023 RisingSTAR Award will soon be opening for entries. The RisingSTAR Award recognises innovation in the software testing and quality assurance industry. It is self-nominated and the winner is chosen by the RisingSTAR supporter group who provide mentorship and guidance to the winner as they develop their idea into being.
