This is the first post in our new series on myths in testing. Like any profession, there are beliefs and myths about what a tester does or what should a tester do. There are ideas about processes that many outside testing might accept as usual practise but that might not often be the case. Each instalment will feature a post on a myth or perceived myth about testing that should be cleared up for the wider (software development) world.
It is accepted that automated testing is a ‘must’ if you want to be able to effectively regression test your software, and to be able to follow a continuous integration delivery approach. This makes perfect sense and is completely logical, however with this push towards automated testing comes a myth that 100% test automation is desirable and achievable, it solves all your problems and there is no requirement to manually test anything.
I believe this is a wrong assumption.
100% Test Automation
Firstly, it is not usually possible, or necessarily desirable, to achieve 100% test automation. There are certain tests that cannot easily be automated – e.g. checking the background colour of a window or dialog box. Why would you do that? What would be the benefit? Does it make sense to even try? It will take far more effort to write certain automation tests than it would ever save in manual effort in their life time.
A far better approach is to identify the key tests that need to be run repeatedly, and verify that there is a defined return on investment (ROI) if automating them:
Consider the case where an automated test takes 180 minutes to write, the time to manually test is 10 minutes, and the automated test run time is 1 minute. Each time the test runs, 9 minutes of manual effort are saved. So, in order to achieve an ROI, the tests need to be run 20 times (180/9 = 20).
If the time to write the test was 240 minutes, the time to manually test is 5 minutes and the automated test run time is 3 minutes, then the saving is only 2 minutes. The ROI is 240/2=120, so the test needs to be executed 120 times before any return is achieved.
This of course is only considering the time taken to write a test – maintenance time needs adding back into the equation, and the ROI recalculated.
Automated Testing Problems
Secondly, automated testing cannot solve all your problems – it actually creates new ones. The automated test pack becomes outdated the minute another change is made to the application, and maintenance of the tests is often overlooked. I would be surprised if there were that many organisations who bake-in the removal of outdated tests and maintenance of existing tests in their processes – the focus is usually around creating new tests. Testing debt is not often viewed as a priority.
Automated test coverage is also key, as each test is only as good and thorough as the person who wrote it. If the test does not fully cover all the different scenarios for any reason, then a ‘passed’ automation test can give you a false sense of security. A 100% pass rate may only mean that the areas covered are fine, but others could be missed.
There are also test failures to consider, as each one needs to be investigated. Some tests are ‘false negatives’ – e.g. a test which has timed-out for some reason due to an environment issue. This costs time rather than saving it.
Pre-Selected User Behaviour
Thirdly, an automated test cannot think for itself. It will blindly follow a path through the application as coded to do. But this is not how a human being will interact with the application – humans can (and do) divert off, click on things that they shouldn’t, therefore automated tests are only covering certain pre-selected user behaviours.
Manual ad-hoc and edge case tests can help uncover additional scenarios that need to be added to the automation pack, which may otherwise have been missed.
So, to summarise:
- Not all tests can or should be automated.
- Automated tests will not solve all your problems.
- Automated tests are best used to cover repetitive tests and to save manual test execution effort.
- Automated tests come at a price and require investment in order to remain current.
- There will always be a place for manual exploratory testing, supported by automated regression tests.
Steve Watson, June 2017
