Usability Testing in Agile: Best Practice and How to Mix & Match DIY Usability Testing

Introduction: “We are all Users under the skin”

In the era of global digitisation, living in society that is always online, we can easily witness the appearance of a new widespread cohort called the User. And there can be no geographical, ethnic or cultural boundaries, as they are all connected with the same “navel cord” officially named “Internet”. Putting aside all the metaphorical aspect of this subject we can add that today’s User is a key element that influences and determines the value of a product and Usability testing. This post will examine Usability testing and how to carry our DIY Usability Testing.

Furthermore everybody knows perfectly that Usability Testing is intended as a permanent learning process, where the best idea is that you don’t have to worry about making mistakes, because there is no right answer at all. So it is like the cognition process in itself. But how often do you hear that User Experience is something that can be taught and learned?

The main fact is that this kind of learning is a mutual process and this global mutual learning from “tell me what I want to learn” to “got it” moment eventually builds to create some universal reservoir of UX assets too valuable not to profit from it. In the time where international teams become a part of objective reality it creates the idea of global UX reservoir that is closer to life.

So basically if you don’t have a right user you don’t really need it to provide Usability Testing. There are many ways to escape it but all of them have the same direction – getting rid of everything that you can.

“User or not, Here I come, You can’t hide”

Speaking about Usability Testing two words come firstly to mind: “time consuming” and “complex”. It doesn’t sound positive way but everybody loves a challenge (that’s why we are all here) especially when there are so many “helping hands”.

And the first thing to help is DIY approach which gives a clear vision and understanding of the essence of Usability Testing process being composed of 5 simple and well-known steps:

1. Having this strategy in mind and seeing all redundancy of tests defined by their repetitiveness getting rid of it should be the next step in current challenge. Here comes these “Just do it” (so adored by Steve Krug) vs “Just don’t do it” (reducing time waste and money spent) moments that can be resolved in a number of ways:

  •  Rest upon previously-received results regardless its source (personal experience, global reservoir of valuable UX assets, trends, competitor site feedback etc). What is known for sure, sometimes you just don’t need to have Usability tests to understand your user.

Next ways of narrowing testing scope can be suggested:

  •  Descope testing of something that is well-known for the benefit of more ‘risky’ and unstable cases;
  •  Costs and time can be saved on conducting phase by blurring the boundaries between user (object of the test) & specialist which can also be treated as a user (see Case study for more details)


2. When we are speaking about today’s software development which is based on principles of ‘agility’ and ‘ready to deliver’ we should tune down our “perfection” detector for the benefit of rough but early deliverables.

So when you get rid from all that you can, take what rests and use… Agile.

[Tweet “Sometimes you just don’t need to have Usability tests to understand your user.”]

But even more Usability Testing is close to Lean. And here comes the idea of Lean UX with Sprint Zero as an extra opportunity to speed-up with Usability Testing, UX assets backlog and early deliverables making the whole process more “UX savvy”.


Case study: “When there is no place for Usability, let’s build it in”

Working on USA-located project that incorporates media & entertainment sites dedicated to African American society of different ages and gender I used to be a part of two web-site redesigns.
usability testing in agile

So having the task, a group of entirely Caucasian team members and no plans for Usability Testing at all was really challenging. But keeping in mind principles that were described previously and English as universal mediator a solution came on its own.

The idea was all about substitution of Usability testing flow by means of regular functional testing with different Usability practices (e.g. DIY, Lean UX).

Using Agile as our main methodology, having no free time in Release plan the idea of encouragement was the first step in the right direction. Overall emotions are one of the moving part for all Usability processes, especially nowadays when Emotional design is one of the main trends.Therefore, solution came in the next way:

1) The first decision that was taken was to combine DIY Usability Testing with Requirement analysis phase.

Analysis of provided draft version of Specification (many TBD moments) gave us the possibility to:

  •        Start early study of the materials,
  •       Define basic metrics (like ‘time on task’, ‘confidence’, ‘task difficulty’, ‘success rate’) by simple Heuristic evaluation identifying first usability concerns,
  •        Handle these concerns in form of Q&A sent directly to the customers and designers, sometimes even in real-time mode,
  •        Create some sort of Usability issues backlog as an on-going list of issues discovered during evaluation and eye-tracking,
  •        Map out in advance where formative and summative Usability tests will occur.


2) Collaboration with third-party front-end team that had started its work before gave us the idea of Sprint Zero.

So at the moment of beginning of the testing we had:

  • Specification (UX Wires) keeping updating,
  • prototypes (updated based on first round of Q&A)
  • static pages with UI code (keeping updating) and…
  • A lot of discrepancies.

The main concern was to turn all these inconsistencies in valuable materials and assets. And fortunately all this multiplicity (comps, UX Wires, static) made possible to use 3×3 way testing which main idea consists of reviewing, comparing and testing of alternative versions of the same design. Again, each controversial point was assigned to Customers for feedback and coordination. So it helped to decide what to develop and what to put in UX backlog.

All this allowed us to start doing functional testing of UI code and formative Usability tests at the same time.

3) By the moment of backend code first delivery we had already established an efficient process of coordination with customers.

This helped to proceed to the summative Usability Testing due to:

  • Day-to-day discussions and feedback receiving;
  • Use of metrics that were established in the point 1), which could be compared from rough to improved.


Giving a summary, next milestones can be highlighted:

  • Know your user and its surroundings, think like your user.
  • Compare functionality from past designs and prepare backlog of assets. Test & benchmark competitive sites against your site. Likeability of UX based on likeness (recognition processes in human mind).
  • Requirement analysis can be a phase of Usability Testing with Q&A form of work and baseline metrics defining;
  • Working code is not crucial to start (Sprint Zero).
  • Team communication and day-to-day coordination mean.


Usability Testing in Agile

It has to be admitted that nowadays the User continues to be a crucial element of the whole development process. Having this idea in mind and a clear vision of your target user is already valuable enough so keep learning from it. Furthermore keep thinking about testing (especially Usability) as a learning opportunity that helps your mind to be updated and ready to surf. Let the idea of encouragement create a clear mind; alert and user-friendly and make Usability Testing in Agile work.
P.S. On the whole we took a risk at some point but when we received the first feedback from real target users saying that “this site is so good, that it should be made definitively by true black women” (direct citation) – all the risks and efforts paid off 🙂

About the Author


Find out more about @ka-sido