Testing Vs Quality Management – What Is Quality

What is Quality

In my most recent blog posts ‘What is Testing’ and ‘Testing Activities’, I shared my thoughts on testing. There is a close relationship between testing and quality, but they are not the same. Testing in some ways, assesses quality, but only to a certain degree.

My intention was to make this the final post in this series and look purely at quality and quality management. As it turns out, I have lots of thoughts on this topic as well, so I divided it into two posts. This first part is how I look at quality.

My preference is to use the term quality management rather than quality assurance. Information from testing can help improve the quality but cannot ‘assure’ the quality is there. When I obtained a quality manager certification from ASQ, I learned that quality assurance was ‘supposed to be’ focused on good processes. However, the term has been subverted and misused, so I don’t think it is appropriate any longer – at least to software development.

There are many different contexts, and each may need a different way of looking at quality. I believe that in all cases, quality needs to be built into the product from the beginning, and I like to make the customers part of the process.

Quality Definitions

The most popular definition I hear these days is from Jerry Weinberg; “Quality is value to some person.” I like it, but I think may be too simplistic and understates some of the dimensions that we should be thinking about.

In the last few years, Alan Page and Brent Jensen have been talking about their ‘Modern Testing Principles’. and their 5^th principle is: “We believe that the customer is the only one capable to judge and evaluate the quality of our product”. Like Jerry’s, it is a simplistic quality model. Personally, I need something more meaningful so I can focus what the quality strategy should be for the product I’m working on.

If we go a bit further back, W. Edwards Deming defined quality as “Good quality means a predictable degree of uniformity and dependability with a quality standard suited to the customer.” Again – based on the customer. Many of Deming’s 14 principles in his book Out of Crisis are still applicable today and bear looking at further. For example, he talked about building quality in and ceasing dependence on inspection.

Dan Ashby wrote a post about Crosby’s 4 absolutes of quality showing how they are still applicable but need to be adjusted to fit into a software context.

A few years ago when I was preparing a talk about quality processes, I had a conversation with Isabel Evans, and she pointed me to this article by David A. Garvin  It got me thinking more about quality really means. I share some of his ideas below but encourage you to read his full paper. You can also read this article which adds in his eight quality dimensions.

People talk about quality as if there is only one kind – all encompassing. Most of what we measure is how well we adhere to our processes – what I call ‘process quality’. To me, ‘product quality’ is about the quality of the finished product – what our customers experience.

Approaches to Quality

Product quality can be looked at from many perspectives. The customer’s perspective is only one – perhaps fitness for use or meets their needs, and different customers have different needs. We also need to consider other stakeholders who have an interest in our product’s quality.

David Garvin talks about five approaches to quality and I use the diagram below as one way to look at them.

We have different lenses in how we view quality. We can look at it differently depending on our circumstances. I start with the inside circle since that is the easiest to think about, and then work towards the outside.

Manufacturing-based quality

If we equate manufacturing to our development cycle, the emphasis is on the practices we follow and conformance to requirements. Excellence is equated with meeting specifications. The focus is on defect prevention and limiting rework. This is often where many teams spend their time building it right and measure the quality of the process. Many teams target testing to this perspective.

We need to recognize that software development is not like manufacturing which builds the same thing over and over. We are continually building on to what we have, changing and adapting.

Some processes that support development (manufacturing) quality are: TDD (test-driven development), coding for maintainability, code reviews, continuous integration, exploratory testing or even conforming to the definition of Done. We are measuring process quality and we answer the question – “Are we building it right? “

Product-based quality

Garvin talks about product-based quality is about the quality of the attributes, the pieces that put together to make the whole. The thought is that better ingredients make better products.

Because this approach to quality reflects the quality of attributes that a product contains, and attributes are considered to be costly to produce, higher-quality goods will be more expensive. If quality is viewed as an inherent characteristic of goods, higher quality means better performance, enhanced features – things that may increase the cost of the product.

Paul Seaman and I had a conversation about this and used this metaphor to think about it – an ordinary cook with good ingredients may not have a good outcome, but a great chef with ordinary ingredients may produce something magical. We need to understand what our product is and how it is put together.

Some processes that teams can do to help support product quality are: ATDD (acceptance test-driven development) or BDD (behavioural-driven development), as well as testing quality attributes such as security, performance or reliability. We answer the question – “Does it work as expected (or desired)?”

User-based quality

The user-based perspective is what we see used most often to talk about quality, and it is highly subjective and highly personal.

It assumes that consumers possess sufficient information to evaluate product quality. If they do not, they will rely on other cues when making that assessment. Let’s consider a cup of coffee. I prefer a simple black medium roast coffee to a well-made cappuccino, but my sister will take the cappuccino every time. What does this mean to you? Who is your consumer? Who is evaluating your product’s quality? Do you need to satisfy the majority of consumers? Or target a specific group and satisfy that group only?

 

Some of the processes that I think help support user-based quality are: user experience (UX) designers working with customers to learn what they want, ATDD, testing using customer personas, A/B testing, accessibility testing, observability and running the analytics on production usage. We try to answer the question “Is this what our consumers want?”

Value-based quality

Value is defined in terms of cost and price. Does the product provide performance at an acceptable price?  That cup of coffee – some people only want to pay 50 cents while others will pay $5.00. That is 1000% difference in price. Is there that much difference in the actual bean? Or the roasting? … Or is it something different?

Marketing specialists are often the folks who care about this. They may do research on price points, give customer surveys to understand what customers think, run analytics on number of products sold, or determine what the profit of specific features might be. These are tests that validate hypothesis and help organizations understand how people use their products and what they find value in.

Product management folks also care about value. They must find that price point that gives value to the customer, but also makes them the profit they need to stay in business.  The question we need answered is “Does the customer find good value in our product?”

Transcendent

I’ve left transcendent quality until last. Garvin describes it as ‘innate excellence’ – universally recognizable, a mark of uncompromising standards and high achievement. It’s hard to quantify or define, but you know it when you see it. Your senses tell you. Perhaps its the ambiance of that great cup of cappuccino you keep going back for.

When organizations enable teams to create something special, to go beyond the norm, that’s when you get transcendent quality. An example might be some unexpected amazing graphics for a new game. I once worked on a system that was replacing one that was no longer supported. When I asked about how the users liked it, I was told “they loved it”, because it followed their work process and not the other way around. A little bit of transcendent quality.

Summary

If I got you thinking a little bit more about your definition of quality, that is a good thing. It is not a simple conversation so many folks don’t talk about it. In one workshop I facilitated, I asked participants to come up with their definition for quality of a specific product. After you’ve read this, go back to your team and ask that question. Check to see if the definitions cover how well you develop (process quality), or are you really defining product quality.

You can check out some of my thoughts from an old blog post about quality and its cost I wrote in 2010. P.S.  I still drive a Mini Cooper although a new version with more bling than I wrote about in that post. If you do read it, I suggest you read the comments too.

 

About the Author

Janet Gregory is an agile testing coach and process consultant with DragonFire Inc. She is the co-author with Lisa Crispin of Agile Testing Condensed: A Brief Introduction (LeanPub 2019), More Agile Testing: Learning Journeys for the Whole Team (Addison-Wesley 2014), and Agile Testing: A Practical Guide for Testers and Agile Teams (Addison-Wesley, 2009), the Live Lessons Agile Testing Essentials video course, and “Agile Testing for the Whole Team” 3-day training course that is delivered both live and remotely. Janet specialises in showing agile teams how testing activities are necessary for the whole team to develop good quality products.  Janet has a degree in Computing Science from the University of Alberta, an Information Management Certificate from the University of Calgary, Scrum Master Certification, and took her Quality Management Certification from the ASQ (American Society for Quality).