$ Value to testing?

[Stephen]

Hi, I know this subject has been discussed before, and will be discussed again; but I can’t find anything relevant at the moment and I thought it would be good to get a view from today.

I’ve been asked to quantify the value of testing; particularly the Service Acceptance Testing phase that I have been managing for the last 3 years. Whilst the organisation recognises the need for testing, and the transition/project managers know they have to include UAT in their budget, they don’t want to pay for it: Isn’t that the way for everything – money for nothing. They seem to be willing to budget for development, hardware, operations teams; but not testing. I know it’s because: development are perceived to produce a product that can be sold; and operations provide a service that can be billed for; but testing provides a service that is conceptual to most PM’s and not tangible.

I’ve suggested we analyse the last 3 years bugs and provide a cost to find v cost to release. e.g. Estimate how much to manage the calls to the Service Desk, man-hours to log – investigate – fix – functionally test – release/change management etc. That’s going to be a huge job, considering they have not invested in a proper test management tool and it’s all on spread-sheets. Also: How do you quantify customer satisfaction in a quality deployment, when they never reference testing as the reason they are happy; or quantify company reputation when testing is a hidden service?

How have you provided a quantified $ value for acceptance testing?

[Padmaraj]

Hi,
UAT is final testing phase. It validation that a developed system or app will meet user needs in the operational or business environment.

According to my experience, Almost all organizations, work according there needs with testing product. Few stable companies focus more on UAT and allocate the time and budget.

UAT done in many way – with intended users or involving real user partial beta testing.

UAT main achievements – identifying issues with usability of the system; and system response time and performance.

[Stephen]

Thanks Padmaraj for your attention to the thread. I was beginning to think nobody wanted to talk to me.

UAT is indeed the final test (unless you consider a pilot an informal test). We choose to utilise the end user, whether that is the operations team or the customer representatives as a formal test, with Test Scenarios (not scripts). We don’t treat it as a Beta, with no particular test, we have guided scenarios that are agreed as part of the exit/success criteria.

The intangible benefits are known and understood by all, e.g. Usability, putting the all systems and operations together as a final end to end process test, providing evidence that the solution will satisfy the end users business needs and support their daily lives. Capture of critical issues that would adversely affect both our company and the customers business. Provide as much information as possible to the stakeholders enabling them to make an informed decision on the risk of going live with the service and products.

All of these benefits are intangible values. i.e. there is not identified $ value gain to them. Each one incurs a $ value cost, but cannot be linked directly to the $ value benefit. e.g. We are going to release, testing provides information on the risk to release. No $ Value, the information does not generate revenue; Again the solution is going to be delivered, showing that it will support the customer doesn’t add revenue.

At the end of the day UAT is more of an enabler than a generator of revenue. We support the generation of revenue without adding income.
Whilst everyone agrees Testing, and particularly UAT, is necessary I need to justify how much it costs to execute UAT. I think we can all agree that $1,000,000 cost of UAT for a £60,000 product that cost £20,000 to develop is unreasonable and not justified. However if the cost of testing was $10,000 and we can show a value of £15,000 then UAT is more than worth while, as is the cost of the resources. I can easily find the cost of UAT, we know the charge rates of the resources and the time taken/allotted to UAT. But I need to find a way of demonstrating the $ value, the saving or even revenue generated by UAT.

Has anyone performed a similar investigation and analysis on $ Cost v Value of testing in any projects before?

[Padmaraj]

Hi, Stephen,

I like to represent my own formula for you, It may be help you to analysis on $ Cost v Value of testing, from above your own statistics.

UAT Value (product development value or end customer buying value) = perceived benefits received, cost to develop ($15,000) / perceived price paid ($10,000)

* Use above formula and get UAT value in each software test life cycle.
* After 3 months show UAT value in graph, to bring a light on real benefit for UAT.

[Stephen]

Padmaraj

Thanks for your input. I’m not sure I understand how I can apply this. You refer to ‘perceived’ benefit and ‘perceived’ price. Perceived value is still conceptual or someone’s opinion not realised, and who’s perception/opinion should I solicit?

Plus your example would suggest a UAT Value = $1.5. ie. 15,000/10,000 = 1.5.

My perception of value would always be higher than the PM’s. The developer may have perceive a high value or low value depending on their self confidence and/or experience. The customer will demand UAT and there imply a high perceived value for UAT.

If you mean actual benefits, that’s hard to measure as 1 – nobody has or wants to spend time estimating how much a defect found would have cost if it escaped. 2- I don’t have this history of defect potential costs.

If I had historic data and estimates on cost should a defect escape the formula would easily be:

UAT Value = (Cost of supporting a defect + Cost of resolving defect found after testing) – (Cost to find Defect + Cost to fix defect found in testing)
Of course this doesn’t account for the value of providing information and confidence to release. e.g. Testing complete, but minimal defects found. No potential escapes saved so no real cost saved. There is still a conceptual value to providing information that suggests the solution is fit for purpose and risks are low enough to release.

I guess that there has to be a question asked: What if we don’t do testing, what would happen. Then cost out testing v’s expense incurred due to not testing. Again very conceptual and difficult to ascertain.

[Author]

Posts