- This topic is empty.
-
Posts
-
Lately I see and hear a lot of opinions that suggest that test automation can / will or should replace testing.
There is a problem with this. Test automation is not testing. Test automation is programming.
I do both but it is hardly the same. I do different things, I do things differently and I even think different when I am programming compared to when I am testing.So for everybody who wants to know if automated testing can replace (manual) testing the answer is :
NO!
I’m not completely sure to agree here. Sure, the way to the test is different with different concepts etc. But at the end your automated test should mirror your manual test.
I might miss a point here, but for me, an automated test is a replacement of a manual test. For a manual test you have your test description you execute, the automated test should follow the same principal, regardless that it was programmed.
So yes, the path to the goal is different, but you should have the same goal at the end.
Agreed that your automated test should mirror your manual test. But is it really possible to cover as many scenarios in automation testing as in manual testing especially exploratory testing? I don’t think so.
So for everybody who wants to know if automated testing can replace (manual) testing the answer is : NO!
So, I agree that manual testing cannot be replaced with automated testing
@bygones I think we have a misunderstanding here.
I do not say that there should be no test automation, or that test automation would not free up testers to do more worthwhile things than run the same tests over, and over, and over again. But that is not replacing testers that is taking the most boring activities out of their hands so they can do something productive – like test software.
The opinions I am referring to when I state: “I see and hear a lot of opinions that suggest that test automation can / will or should replace testing” go a lot further than this.
Basically the opinion is that based on the requirements (if at all possible in BDD language like Gherkin) and the code the test cases can be generated, test data can be generated and test scripts can be populated and then run with just a little input from a developer – you don’t need testers anymore.
Since I perceive testing to be a craft / state of mind / more Zen like state of being I strongly disagree with that.
I do lots of test automation, but then I am just a programmer. The difference is that my stakeholder is the tester whose test I automate and not an outside client. I will use different tools like WebDriver compared to other programmers, but I am still just writing code, be it in Java, C#, Python or shell scripting in bash or Powershell.
When I am testing however I have a completely different mindset.
I want to find out the nooks and crannies of the software under test. I want to crawl in the dark innards, I want to know way more than the coders are willing to divulge about the inner workings of the software. I do not only want to know What the code does but How and Why it does what it does. And somewhere along the line I will even take notice of the requirements and run some tests to make sure the software meets them.
But checking requirements is not where the passion is, or the most interesting, most difficult and most damaging bugs are.
The passion is in the investigation and discovery of bugs outside of the requirements.
I am still convinced that you can not automate that kind of tester / testing.
ps. Sorry for the rambling: it’s been a long day…
Hej,
I agree with your statement and I’m happy that you elaborated more on this, so that the discussion is clear and one can actually see what your point is.
It should be obvious (maybe only to me though), that an automated test suite does not replace any manual testing or vice versa. Automated testing is happy path testing – we make sure that the things work as expected. Manual testing is more than that (it is also happy path testing) – but manual testing always things like monkey testing and I guess what you mean with your craft/Zen etc statement.
Have you ever tried recording tools like Ranorex ? This allows to create automated tests by manual testing and therefore (at least to a certain degree) removes the “programming” part of automated testing.
But also here is at the end a test case that will only make sure that the recorded actions will work later on – leaving the path is not possible.
So, no worries about being rambling, this is a valuable discussion 🙂
You can automate all the testing some of the time,
and automate some of the testing all the time,
but you cannot automate all the testing all the time
(Word police: checking is a part of testing)
(See also: Shift LEFT)
Revisit the webinar: Do we need testers on agile teams?
https://huddle.eurostarsoftwaretesting.com/resource/agile-testing/need-testers-agile-teams/?@kasper touched the hot topic and some kind of nightmare for the people who cannot automate. So I’d say that automation helps the manual testers to get their best tie back. Automated test cases suites do all the boring, repeatable job and then the manual testers have time for exploration, ‘real testing’, discovering new features etc. 🙂
What do you think @bygones?
I think @jesper-lindholt-ottosen had some kind of that in mind?
@kasper touched the hot topic and some kind of nightmare for the people who cannot automate. So I’d say that automation helps the manual testers to get their best tie back. Automated test cases suites do all the boring, repeatable job and then the manual testers have time for exploration, ‘real testing’, discovering new features etc. 🙂 What do you think @bygones?
I’m not so fond with the wording as it implies automated testing is the “boring” part and the manual test is the “real” one – but besides that I get your point and agree on that
There is no test automation. Just a bunch of scripts executing regression checks. The naming is a scam by tool vendors and consultants.
That being said. If you previously did a lot of hands-on regression testing. Then yes, there might be a possibility to replace some of that testing with those regression checks (not that it will ever be the same thing). And there might be a possibility to reduce staff (to much more expensive staff though).
@bygones I have used tools like Ranorex, but they are extremely limited in what they can do. If you have a small static application they are somewhat usable.
But if you have large, dynamically generated applications on top of a framework – not so much. Or if you have systems with no GUI.
I mostly work on large applications and / or back end systems without GUI.
For example: I am now working on a global application in the financial markets on top of a dynamic framework.
The number of test functions I write in Java is more than 500. I actually have a manual tester testing the automated test framework I am developing.
@axeloi The framework is more than regression testing however.
There are just tests like testing risk models and authorization matrices and performance and static code analysis and entering and checking test data that are much faster and more reliable tested by automated scripts than by humans.
So I respectfully disagree that there is no test automation but only scripts for executing regression tests.BUT: Automated tests can not replace human testers, just like expert systems can not replace real MD’s.
Look at the Microsoft chat bot Tay – even in harmless conversation software can not replace humans at the moment, but there are things better done by software – use it or lose out.ps. Yes, I am a consultant
The testing profession is changing at an incredible pace these days and this question reflects, to my mind, the uncertainty and inevitable internal divergence that naturally occur when the pace of evolution is so rapid. The necessity of the tester is called into question again and again, and defended just as often. Whether automated testing can or should replace manual testing is an extremely controversial topic in various forums, at many conferences and in organizations of all types and sizes.
I believe that the dominant feature of this rapid evolution is that programming and testing are, in many ways, converging. They have different cultures, foundations and goals, all of which are being challenged by modern approaches to testing (such as cross-functional teams), the increasing complexity and magnitude of software systems, and the rising diversity in software platforms.
The question, by the way, does not apply only to testers. Programmers are facing similar dilemmas. Should programmers test? What kind of testing should they do? Is their testing “adequate”? Can a programmer who does not test be considered a good programmer? Divergence on these issues is just as rife amongst programmers.
I would suggest a different way to approach the topic that I believe is simple and effective. I distinguish between skills and roles. I think this changes the whole conversation.
I identify two ideal roles at opposite ends of a continuum. All programmers and testers are on this continuum and almost none are at the extremes.
- Programmers solve problems, implement abstract ideas and identify and automate processes.
- Testers identify and investigate problems, manage and monitor quality, and ensure compliance with requirements and specifications.
I think that the essentials of these roles has not changed much in the past couple decades but the skills that they rely on have changed significantly.
So here are a few skills. Needless to say, there are many others. Some of these are implicit skills.
- Coding is a skill.
- Validation is a skill.
- Verification is a skill.
- Test design is a skill.
- Design patterns is a skill.
- Estimation is a skill.
- Abstraction is a skill.
- Empathy is a skill.
- Investigation is a skill.
- Identifying processes is a skill.
A coder is someone who can read and write code in at least one language, is familiar with those languages’ ecosystems, and is able to use the tools and practices associated with those languages effectively (including for example, integration tools). Coding is essential to programming so a good programmer is almost certainly a good coder, but the opposite is not necessarily true. A good coder is not necessarily a good programmer because programming relies on other skills as well.
Test automation testers must possess coding skills, though this can be mitigated to some extent because different tools require different levels of skill. They do not have to be master coders but they do have to be competent coders. In any case, coding is not the only skill that test automation testers must have. An automated tester who doesn’t know what equivalence partitions are is probably not much of a tester, in exactly the same way as a manual tester.
In light of this approach, I think that the “manual tester” vs. “automated tester” dichotomy is false. There are testers (a role) who employ different skill sets to practice their profession and achieve their goals. A tester who does a two-week course to learn to code has not become a programmer because that tester did not acquire all the skills a programmer relies on and has not yet mastered the coding skill. But as that tester becomes more proficient in coding and perhaps learns other skills strongly associated with programming, they can be said to move along the programmer-tester continuum. The continuum is horizontal.
The line between “automated” and “manual” is an increasingly gray line as well. A tester who writes code that utilizes the WebDriver API directly can be said to be an automated tester writing automated tests. But if that same tester writes Gherkin on top of a framework somebody else wrote that utilizes WebDriver, then the tests are still automated but it’s not clear at all if that tester is an “automated tester”. The same is true for MTM fast-forward automation, Ranorex Recording modules, numerous web API recording tools, and a host of other tools that try to compensate for the low or absent coding skills of most testers. It’s an ambiguous term, at the very least.
I personally believe that the false dichotomy also plays into the traditionally highly meritocratic aspect of our industry, often placing “automated testers” higher on the ladder than “manual testers”. Familiarity with code does not on its own create a better tester. Proficiency in testing skills such as verification and validation, however, do, regardless of whether code is employed. It seems inevitable to me that coding will eventually become an essential testing skill that all testers will be expected to learn. In the meantime, testers who learn to code should do so in the pursuit of improving the profession rather than stratifying it.
In Agile parlance, testers are developers just as much as programmers are developers. This is becoming more evident, in my opinion, as the two professions continue to converge and skills once associated with one role are adopted by the other.
So test automation is testing that employs a convergent set of testing and programming skills.
@kasper Sorry, but using a script for checking 1000 input-output combinations rather than checking them one by one hands-on, does not qualify as automation. That would be like saying that a carpenter who uses an electric drill does automated carpentry.
Sure, scripts (scheduled regularly for regression control or not) is a good tool to have. But calling that automation just gives the false impression that you’re replacing manual work.
In the ideal case, the test automation should not be the development work. The test declaration should replace the test coding. Here are the pillar of this approach:
1. To reduce complexity use the test hierarchy, for example: test suite, use case, test case, test action
2. To improve maintenance use unconditional test expectable sequence
3. To bring efficiency of test automation use keywords driven approach. Use the following keyword types:
a. Test oriented keywords: result selections from the output buffers to be compare; compare possibilities with expected results; control statements, including time measurements
b. Interface oriented keywords: send stimulus and extract/ capture results
c. Business oriented keyword: use cases/ scenarios related to the object-to-test specificsHere is a typical process:
A tester defines the keywords syntax based on his/ her Test Plans
A developer creates scripts/ drives and add them to the respective libraries.
A testers declares the test cases by coping/pasting the use keywords with necessary parameters.As a result, hundreds test cases can be created daily, and, moreover, they come readable and maintainable, as a package. Robot Framework is a good example of such approach.
In Rapid Software Testing, we say that testing is evaluating a product by learning about it through exploration and experimentation, which includes to some degree questioning, study, modeling, observation and inference. Testing involves many other things, too, including checking, which we say is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.
Checking—one of many things that happen during testing—can be entirely performed by machinery and algorithms. But analyzing risk, designing checks, programming checks, and interpreting their results cannot be done by machinery.
When a human writes a program or configures a tool, software can check, which aids in testing. There are other ways in which tools can help people to test, in many wonderful ways. But tools don’t do anything on their own. Humans wield tools to extend or enhance or enable or accelerate or intensify human capabilities.
Although they might iteratively refine algorithms in powerful ways, machines don’t learn as humans do. Machines don’t explore. Machines don’t experiment. Machines don’t question, study, model, observe, or make inferences in any kind of human sense. Emotional engagement is a critical part of testing, but machines don’t feel displeasure, nor pain, nor satisfaction, nor delight. So whatever is being automated, it’s not the testing. “Automated” and “manual” testing is indeed a false dichotomy.
This prompts a cautionary note: if we keep using the expression “automated testing”, naive or ignorant people will find it easy to believe that testing can be automated. Those people will be vulnerable to being suckered by irresponsible tool vendors. And we will all pay by being subjected to software and products that have only been checked, and that have been negligently tested.
@axeloi I am curious about your definition of automation but I guess it is somewhat different than I learned at my first year Technical University. I am also curious how you jumped to the conclusion that I (only) write scripts to check thousands of input – output combinations – which would still constitute automation IMHO.
As stated earlier I work a lot on systems without GUI – so I use the API, which means programming, which just about guarantees automation.
Since I have been programming since 1985 I hope I learned a little bit about automation – but I am happy to be corrected if you have factual arguments.
@michaelabolton As I stated in the first post:<span class=”ms-translatable”>There is a problem with this. Test automation is not testing. Test automation is programming.</span>
I guess we agree on this?
<span style=”color: #e4af0a;”>@bygones</span> I have used tools like Ranorex, but they are extremely limited in what they can do. If you have a small static application they are somewhat usable. But if you have large, dynamically generated applications on top of a framework – not so much. Or if you have systems with no GUI. I mostly work on large applications and / or back end systems without GUI. For example: I am now working on a global application in the financial markets on top of a dynamic framework. The number of test functions I write in Java is more than 500. I actually have a manual tester testing the automated test framework I am developing.
I dont quite understand, why you like to use UI automation tools like Ranorex for non-Ui systems ? Of course they are not helpful here, as they have a completely other area of use. This said does not mean that non-ui systems do not have any automated tests, but of course than other technical orientated ones.
And having a person testing the automated test framework sounds wrong to me, but I dont know the setup nor the framework and no context, so not judging from my side.
<span style=”color: #e4af0a;”>@kasper</span> Sorry, but using a script for checking 1000 input-output combinations rather than checking them one by one hands-on, does not qualify as automation. That would be like saying that a carpenter who uses an electric drill does automated carpentry. Sure, scripts (scheduled regularly for regression control or not) is a good tool to have. But calling that automation just gives the false impression that you’re replacing manual work.
of course you are replacing manual work for the (tragic) person, who had checked that 1000 input-output combinations before automation. This lucky guy can now focus on the more elaborated testing stuff, rather than blindly check combinations. Also your comparison is wrong. Test automation is not simply that a person is using a tool, test automation is the automation of a process.
For me it seems that you expect from automation some magic, that replaces something like exploratory testing done by a person – and this is not the case and no one is claiming this in my eyes. Of course automation is nothing else than reducing the man-power of regression testing. No system (yet) is able to perform monkey testing or similar – it might come in the future, as self-writing applications – but now – automating your test suite helps to focus on other areas of testing.
Regarding the whole automation vs manual false dichotomy problem – it is in my eyes not a question of manual vs automations that needs to be clarified, but testing… what is testing and what it is not ? as it seems that there are several (different ?) definitions existing, that results in confusion/discussion whether automation is testing or not.
@Kasper: In Rapid Software Testing, we say this:
A tool is any contrivance used to fulfill a human purpose (and a “test tool” is any tool used in the service of testing).
Automation is any process entirely performed by a tool OR a tool capable of such performance.
To avoid confusion, we don’t talk about “test automation”. We want to reduce the risk that people might believe that testing is a process that can be entirely performed by a tool. Automation can’t do testing. We do talk about automated checking (automation can do checking), and we do talk about tool-assisted testing.
Since we don’t use “test automation” in our way of talking about testing, we can say that “test automation is not testing”, in the same way we could “unicorn training is not training”. However, we can’t say “test automation is programming”, because to us test automation isn’t anything at all.
What we can say is that programming is a way (or a set of ways) in which we can use or create tools for testing. We can use programming in automated checking, and we can apply programming in other forms of tool use as well.
Where we do agree, I believe, is here: in testing, excellent automated checking and excellent use of other tools requires a set of skills, and programming is a very useful set of skills for a tester to develop.
There’s lots more in http://www.developsense.com/blog/2016/01/a-context-driven-approach-to-automation-in-testing/
—Michael B.
Where we do agree, I believe, is here: in testing, excellent automated checking and excellent use of other tools requires a set of skills, and programming is a very useful set of skills for a tester to develop.
I can live with that.
<span class=”ms-translatable”>List All Your “definitions” first. It will reduce the amount of confusions.</span>
No, they are just the definitions as described by ISTQB / TMap / Technical University. I am not about to post all because that would be silly. If you have a specific question for me I am willing to discuss that question. The rest can be learned or searched for.
…scripts to check thousands of input – output combinations – which would still constitute automation IMHO. …
This is actually why I call this a scam. Calling something “automation” that was never done by humans in the first place.
It’s not that much about definitions, but more about how people interpret “automation”. And in general the interpretation is that you automate something you previously did by hand.
@axeloi You unmasked me! I am a scam artist. 😉
@kasper No worries. Who knows where and when this false dichotomy (as some say here) was invented. But it wasn’t by you 🙂 Possibly it just came to be with the best of intentions.
Thank you guys, I like this thread! For me, automation is “anything that was/is done manually and is/will be done by some kind of tool while I go for coffee or lunch, or just lean back for a while.” So using an automatic drill is not automation indeed.
Some additional thoughts:
– It’s not about manual or automated, it’s about brains or no brains involved
– Ever heard of AuTEST or Specialisterren? A sound alternative to Test Automation. Same low-cost, same reproduceability, Con: slower and difficult to put in your daily build & CI script. Pro: some brains, sometimes particularly clever brains
– Brains for test design are a must: the simple part may be replaced by MBT, but not exceptions, nooks & crannies.
– Brains for test execution are a bonus: Automated testing misses the exploratory ‘test design while executing’ and continuous context driven improvement part.
– It’s not only changing roles. Testers mindsets are different. When the risks are real, you need real testers.
– Aut(est)omation is fine, because it makes brain power avaliable for the creative part
– Too bad, tools & script maintenance often consume a part of the testers brainpower and creative risk focus. Let developers do the tools & scripting as much as possible.
Interesting upcoming webinar titled ‘Why You Shouldn’t Automate… But You Will Anyway’ presented by Kevin Pyles if anyone is interested.
Sign up here!
I agree to
Test automation is not testing.
Test automation is regression checking.
But I would add that regression checking is version control in disguise, i.e. it is about controlling changes to the dynamic properties of the software (the look and behavior). Because existing version control systems can’t do that directly, automated tests simply pin those dynamic properties down and transform them to a static artifact (e.g. a test script), which again can be governed by current version control systems.
I agree to avoid confusion and stop talking about “test automation”. Instead let’s talk about “version control of dynamic properties”.
Regression checking is part of test automation but it is certainly not all test automation does.
I agree: there is also performance / load testing and ATDD/BDD until implementation is completed (then it becomes regression testing).
There is also some automation in the broader field of testing, such as:
- Generating data
- Obfuscating real-life data
- Converting test data from one format to another
- Setup and configuration
- Sorting and searching logs
- …
But I would consider these as supporting activities, not testing by itself.
Is there anything else that you think test automation does?
<span class=”ms-translatable”>Hi Jeremias, I partly agree with your point. But I’m afraid “version control of dynamic properties” instead of “</span><span class=”ms-translatable”><span class=”ms-translatable”>test automation” is more regression than progression (I stick to our jargon), simply because it’s too complex wording. I don’t think it’s helpful to bring in the whole ‘testing vs checking’ discussion here. That discussion definitely makes some sense, but is hardly relevant to our stakeholders. It’s just an insiders topic.
</span></span>Imho it’s enough to clarify that dumb (regression) testing and clever (exploratory) testing are different indeed, but still both are testing and both have value and must be applied in the right combination.
- You must be logged in to reply to this topic.