Why do we need metrics?
When you need to communicate the results of your testing, specially as you start interacting with managers or with peers who don’t have a lot of time, you tend to hear this request:
“Can you provide a summary with numbers
and the high level status?”
It’s not that these people don’t care about your testing… but they care more about their time. And so, before they are willing to hear you go in detail about the problems, they want to get a high level picture of the project’s status.
Don’t take this personally and definitely not in the wrong way, you are working with people who are busy, and they trust you enough so that if you tell them “All is Good” they will move forward without a second thought.
But based on this same principle, when things “aren’t that good” and you do want to point their attention to a problem, they will want to understand what is wrong in the quickest and most effective way.
They will also want to “grade” how big the problem is by comparing it to other problems in the past, or to other parts of the project that do not have problems.
This is the reason most projects after a certain size and complexity will require metrics to communicate effectively.
Wait! But Metrics are EVIL… Right?
Many times it may seem as if metrics are an evil tool – causing more harm than help.
If you search on the Internet for “Software Metrics Evil” you will get many results explaining why your project should not use metrics, and how you should avoid working with metrics at all costs!
But the truth is that Metrics, just like a knife, fire or TNT are not intrinsically evil. The problem is that just like any great tool it can be very dangerous when it “falls on the wrong hands”.
And wrong many times is not necessarily someone with evil intentions, but it may also be someone who does not think “all the way through” before defining and publishing metrics.
| Here are some benefits of using metrics: | Some cons of using metrics: |
|
#1 Metric are an effective means of communication and are easy to distribute Metrics can quickly communicate the overall status, but they can also convey what is important and where efforts are needed. |
#1 People don’t like being measured Whenever you introduce metrics, the first response of many people will be to reject them and to try to stop their implementation as they believe they will be unfairly measured by them. |
|
#2 Metrics are clear and help us track our progress Good metrics are easy to understand for both team insiders and external readers. They also make it easy to understand where you stand based on past results. |
#2 Metrics tend to present only one side of the story
Each metric tends to present only one dimension of a complex issue, and this can create a challenge in presenting the whole picture of your project. |
| #3 Metrics assist in comparing
You can benchmark and compare different teams and products by using the correct sets of metrics. |
#3 Metrics are gonna be used by “Managers” Many times people who are “away from the trenches” think that their personal experience is valid in all situations. This can make them jump to the wrong conclusions based on the raw numbers, and without hearing the relevant explanations behind them. |
| #4 Managers like metrics Busy people, like your managers, tend to like metrics because they are compact. |
Some interesting thoughts and observations about metrics…
From working with hundreds of teams and helping them to develop their metrics we came up with something we call:
The “Laws” of (Conservation of) Metics
- Anything that is measured will improve over time.
- Every improvement (see previous) comes at the expense of something else that will worsen.
- When setting up your metrics, know that you can only measure between 3 and 5 aspects of anything concurrently. Trying to measure more than this will lead to confusion and failure.
- Metrics will become ineffective after a number of iterations, hence you will need to shift them when they stop being effective.
Planning your metrics program
By now you know you need metrics, and you should also understand they are not evil, but that you need to be careful about the metrics you want to implement.
So Far, So Good!
Now let us take you over some steps that will help you to define the metrics you want to implement, and also to review how to keep them relevant over time.
Step 1:
Define your (Metric’s) Objectives
Ask yourself who is your audience and what information do they need from you? Every stakeholder has its own need for information, so creating a single metric is never the best solution.
Some examples for the different needs of information for your stakeholders are:
The Management Team
They want to know if we are able to release the project on time and to live up to the customers’ expectations. In case the answer is NO, they want to know when is the most probable date of release and if there are any unexpected costs to it.
Your Development Team
They wish to know where should we put in more effort and if there are risks and dangers to be handled?
Your Product Team
They are also want to know if the project will be released on time, but in case the answer for this is NO, they wish to know what can compromises can be reached in order to release as close to the release date and with as close to the functionality as originally intended.
Your Support Team
They are interested in learning what issues are going to be fixed by the release, and if there are potential limitations they need to prepare up front
Step 2:
Make sure you have the right metrics in place
There are a number of methods to understand if you have the correct metrics in place. One of them is the one called “The 5 Whys” where the aim is to ask yourself iteratively the question Why in order to look for the root metric that will give you the information you are looking for – you can read more about it here.
For example, when deciding to measure the number of reported bugs, ask why knowing this will help you achieve your goal. In the above mentioned example in many cases , you will realize that you might wish to identify the number of unrecognized bugs, or the number of bugs you have identified and decided not to fix, etc.
Step 3:
Once you have your metrics, make sure they are S.M.A.R.T.
The objective of the SMART approach is to make sure your metrics are both effective and efficient, and to achieve this you validate the following 5 points:
Simple- You should be able to describe it and understand it easily
Measureable – Concise and speaks in standard units
Actionable- Once we see the metric we know what we needs to be done
Repetitive- Can be reproduced on a routinely basis
Timely- It should be provided at the time when actions can be made
Step 4:
Think about metrics maintenance
Metrics need to be constantly reviewed for:
- Accuracy – did something change in my product or process to make the metric I defined change? If so, change the way you capture this metric.
- Relevancy – is the potential issue I am looking into still relevant? If so, understand what is currently relevant and make sure that is covered. As important is the decision to drop irrelevant metrics so you won’t grow the feeling that it’s redundant and obsolete.
Remember that metrics need to be updated constantly based on changes in your project: Over the course of years, technology will most likely vary, and so will the people who work on the project. Experienced professionals can benefit from different metrics than a starting tester and as your team grow in tenure, so should your metrics.
Most importantly:
Don’t fall in love with your metrics!
Be ready to change metrics if needed,
even half way during your project.
