- This topic has 8 replies, 8 voices, and was last updated 9 years ago by
.
-
Posts
-
….for improving product quality and time to market and ultimately reducing cost (more margin)
I “know someone”, a certified ISEB Test Practitioner who has tried metrics of every kind to demonstrate the considerable and shocking waste in his situation in terms of dollars and wasted man hours doing this testing in a manual environment.
He has more and more complex products coming and the only investment seems to be throwing “manual” bodies at each new product, (obviously) from a different continent. The product functionalities are apparently 90% common across the range and lend themselves exceptionally well to automated testing.
It is thought that the biggest issue that the more senior management don’t understand testing particularly well and that the test management below seem incapable of transmitting the message up the food chain to persuade the case for investment in automation and to convince business as usual activity as being relatively unimportant and wasteful
I would be interested in what “robust” metrics are likely to have an impact in this situation?
Hi Mike,
it’s always difficult to come up with metrics defining the need of test automation. In my company we’re implementing agile and continuous delivery. What I normally say to my management: Continuous Delivery means Continuous Testing; Continuous Testing means Automated Testing. They buy this statement because I use the terminology that they want to hear: continuous delivery, Agile, Time to Market. I could be that simple…Hi Mike
Is the “fear factor” at play.
Businesses that are running at a million miles an hour with a portfolio of projects (some of which might already have been subject to delay) sometimes don’t listen very well and/or just want to keep things simple.
We’ve tried the concept of a fixed price, short (10 working day) proof of concept with a well defined set out outputs. Doesn’t stress budgets too much and using 3rd parties should mean that it won’t impact existing project timelines.
I do agree with Henk as well – massage ego’s.
Alex
Hi guys,
i will add few of my thoughts on this topic as what i have learned in the past 6-8 years leading big QA group of 5 teams including automation team.
First of all i would like to say that Continuous Integration is far much better solution than Continuous Delivery depends on the product and your needs as i am not familiar with it maybe a little too aggressive statement 🙂
CI provides you quality product at the end of dev cycle which will shorten your manual test cycles by having less bugs. CD will focus on delivery automation which does not help you to escape from situations of having broken build/code/product in QA.
CD will not reduce nr. of issues reported by QA CI actually does and focus on that.
If you are delivery director then yes you focus on CD because i want to deliver faster and etc…but you need to work your ideas few steps back actually to achieve this or be in situation that your CD will be effective once you use it.They all say that in Agile the team must delivery working deployable product at the end of the sprint but in my experience if the product has even a little of complexity in the code then no tester or developer is able to catch these during the sprint. There are always to many things to focus during the sprint: Testers/Devs are writing unit test or automation test to cover the area what have been developed. But what about the all things around the development ? This is what usually fails after any sprint… so how to ensure this will not fail is actually write automation tests to be a part of CI.
5 Years ago we jumped into automation world and wanted to automate our platform fully which we achieved in a way… but we had major downside as well. The cost of maintaining these test were way too expensive and time consuming. We did use mainstream frameworks and approaches available that time. We actually did not improve our product either we shortened our QA or UAT cycles at all which we thought will happen… as everyone was hyping it that this is the key. As far as i understand the you need to understand your product and do your analysis about what you want to achieve with this ? Will you do this because everyone are talking about it ? Or you think it will help you ? What exactly are your goals ?
What i can say is that we did tried both CD and CI and our case going CI was far better than CD. We still trying to figure out how CD will fit in our environment.
We did lots of manual test process improvement and i can say that we had far more gains by improving our manual test by using different tools and doing integration between QA tools and JIRA and etc…. this was boosting much faster defect entry much faster feedback from devs. I do have to mention that tools in QA department secifcaly test management tools will improve productivity a lot.
Coming to the back to the point again about the matrix… then you need to start from development. Only measure you can use in my opinion is time. Faster you can fix and find the issue less it will cost. By going CI and Agile you will achieve this partly for sure. CI provides you better code together with automation which will improve your code quality and will shorten QA or UAT cycles at the end.
To be honest i am moving in parallel line between automation solutions and manual solutions because our biggest gains have been by improving our manual test processes and help development department to understand why they fail in the first place. Many many years ago developer did his own testing and they were responsible of their code — these days its QA/tester/engineer responsible of quality of code or product… which is kind of wrong. This was a major thought change for developers to understand and to brake out of this paradigm.
.Can i ask what kind of product you are working on ? Maybe i can suggest ideas from where to start ? If needed i can help and guide to set up CI and implementing automation test into it.
Anyway i might not give you right answers but maybe you see things from different angle which maybe help ? 🙂 There are way too much ideas/thoughts in my head that can be said now 🙂
Viljam
I hae very simple prescription: Calculates all yours expenses for manuals testing and multiple it too all your circle and send this information to your CTO or othe person how care about budget.
Hi Mike!
When starting with test automation take small steps, like Viljam already mentioned, don’t go to your management and state that you want to do Continuous Delivery now – that will be overwhelming.From my experience the most important thing for convincing management is to make them understand the situation. Give them a concisely example on how to smoothly integrate test automation to your test process and which impact that will have on the costs and response time on bugs found by tests. It’s hard to make someone who may not know much about the importance of testing understand your point of view. Therefore be prepared with some good illustrations. They say more than metrics. Metrics are good and important, but maybe not for that specific situation.
For starting with test automation you have to know the value of your tests (in terms of risk, frequency of usage, bugs found by test …) and the effort it will take to automate them.
Prioritize your tests according to those criteria. Tests with high value and low effort should be automated first. Tests with low value and high effort should be automated last, if at all.Knowing that, you can present the Return on Investment (ROI) of test automation.
You can find Test Automation ROI Calculators online to get some good numbers for your specific case.Cheers, Larissa.
Thank you all for your answers and comments – there is quite a range of opinion which suggests my “problem” is not unique.
Embedded systems (as with other genres) do come with their own specific issues.
I have an ISEB Practitioner ticket so I’ve figured out most of the above over my lengthy career; …. but I still find it exceedingly difficult to accept that development/test management would not find the above arguments way beyond compelling and actually something they would want to invest in.
This is especially true when one spends a large slice of my time trying to make the justification when I could be specifying/creating more test content to improve the general situation!!
..But I’m not giving up!! ……..I will give some of the ideas a bash and report back to this thread should any of them be effective! 😉
MikeMaybe it can be just a case of aiming too high and going at it with metrics instead of actual results. My experience with starting with automation was to convince my manager “ok, we did the last project with manual tests, what about doing the next one with automation?”. Just start one level up and see how that goes and treat it as a POC with immediate and visible results. Working in agile helps and having CI also. The most important metric for me would be that the sooner the bug is found the cheaper is to fix. It is all about the money in the end but things other people said on the thread, like time-to-market, or shorter release cycles are all valid points too
Hey Mike,
The stakeholders/decision makers involved each have a value proposition all their own. My experience is that it may even be irrational and rooted in their own personal experiences and their experience may be totally irrelevant to the current business situation you are trying to address. So this means a rational approach to justifying the investment in test automation or any other process improvement won’t resonate as it doesn’t address their personal value proposition and consequently this will not motivate them to move to your side. You are selling “Your” perspective on value! Just because it resonates with you doesn’t mean it is a persuasive argument to the decision makers involved.
My experience, in the past 35 years, suggests that decisions are based on emotional perspectives more often than rational perspectives. Perhaps it is possible to get some one-on-one time with key decision and explore what their personal value proposition is on your proposal and attack it from there.
Just a thought and maybe irrelevant.
- You must be logged in to reply to this topic.