What is best tool to start with Automation testing?

[Joey]

There are many tools available in the market for automation testing. But which is the best tool to start with who does not have experience in Automation testing but has worked on manual testing.

As I know, here’s the list of some automation testing tools that i think they are the best at this moment:

1. Testcomplete https://smartbear.com (It’s a paid automation tool. It’s an little tricky to learn and use but quite similar to Selenium)

2. Katalon Studio: http://www.katalon.com (It’s a free automation tool – enhances Selenium and Appium with a complete test automation framework)

3. Tosca Testsuit: http://www.tricentis.com (There are many more features that are useful for automation but there can really be a need for some coding effort to integrate with your application)

And anything else?

[Jesper]

Hi Joey

It absolutely depends in the development stack of your environment.
4) Visual Studio (online) for Desktop applications
5) Docker / Jenkins for Continuous Integration

[Archana]

HP QTP and Selenium are the market leaders in automation testing
Ranorex and IBM Rational Robot are also popular

If you are looking for something that requires minimum coding, you can also try TestingWhiz
http://www.testing-whiz.com/

[Jari]

I’d like to try HPE Unified Functional Testing, UFT (used to be called HP QuickTest Professional, QTP) but I don’t think I will in the near future due to:

1. Price/Cost compared with added value
2. I want fast execution time (most of our tests anyway run without any GUI)
3. I want to have Linux support
4. I want to have multiple instances/threads (spread to multiple test servers)

Selenium…that’s more for web stuff, so not my current game.

Visual Studio would be interesting to try. TestComplete is fine, but has some annoying features; mostly when we want to automate testing new builds.

Jenkins is really good. It does have annoying bugs, at least on the GUI (e.g. sometimes when a server job fails, it’s not directly visible, only from the Console output), but does the job quite well.

But to answer what is the best to start with, we would need to know a lot more about the context. Basic points to start with:

1. Web/Mobile/Desktop support needs?
2. Knowledge of programming?
3. Who does the testing? (For example, does a tester get help from the programmers?)
4. What is the goal for automation?
5. Is it more important to create tests fast than to create them robust?
6. Is the test automation throw-away code?
7. Will Record and Playback suffice?
8. What are the plans to go forward?

It’s entirely possible that a tool like ZAP from OWASP could meet the current needs, so it’s good to keep eyes open.

[Nisha]

Any suggestion for Mobile Automation which is the best tool  for Native app ?

 

[Archana]

I have heard a lot about Appium. TestingWhiz is another good tool which is integrated with Appium.

[barret]

I’d also like to add Ranorex to your list because it is suitable for both coders and non-coders, regardless of whether a desktop, web or mobile application needs to be tested.

Nisha wrote:

Any suggestion for Mobile Automation which is the best tool for Native app ?

You can also use it for mobile automation testing.

[Erkki]

I have good confidence on the Robit Framework. It is quite logical, well organised and has extensive libraries available. And now with Jyhton support it is possible to build your integration also on Java (if Python is any problem). Also there is good amount of support available. Locally in Finland it is a skill with high demand now.

It is not dependent on any target technology (like Selenium in WWW) or platform, has good plugin support and is free. What else would you need? 🙂

[Erkki]

(Naturally I meant the Robot Framework, bloody phone has too small buttons for my fingers.)

[Lila]

Thanks @huynduong. Just tried Katalon Studio since it’s free 😉 With limited coding skills, I thought this one is extremely useful for me to start the automation testing.

Highly recommend, especially to manual testers who would like to transform to automation testing.

[Monica]

Qualitia gives you simple step wise approach to quickly build your test automation and handle complexities easily with its unique scriptless approach. It gives you smart alerts and analysis to adopt changes effectively, making it agile friendly.

http://www.qualitiasoft.com/

 

 

[Joerg]

Hi,

we have switched from Selenium to SAHI Pro because the creation of test cases saved us 50% of time. Also it is more rock solid in AJAX environments and it has been written from (test) user centric point of view. Very easy to start and you can use Java, Ruby or JavaScript to write your test cases. The recording can be enhanced by using AR (Accessor Repository) – to define once the accessors (which are NOT XPATH and NOT CSS) and use them also in recordings. The recorder is very nice because you can add asserts, define manually checks etc.

Yours,

Jogi

[Raj]

First of all, it is a good question. Thanks for posting it.

Secondly, the question is very broad in the sense,

• When you say “Automated Testing” are you talking about Native app automation or Mobile web app automation?
• When you say “best” tool, I think there is no such thing, as it is based on the context of what you want to do. With any automation tool comes the usual discussion of cost vs value, time to maintain them, stability etc.

Just wanted to give you something to think about.

That being said, I am not a big fan of vendor tools (I have used many and evaluated them in detail). The reason is, you are overcharged for their services and you are confined by whatever they provide within their IDE and framework. It is really difficult to add some new plugins to it or feature to it based on what you actually want to do.  They will say that, “they can definitely add new features we want in their next release” which never happens.

NOTE: I do respect vendor tools a great deal as a lot of hardwork goes into it and lot of bright people create it. But for small business or individual contributors wanting to do automation, it becomes more expensive. This is ONLY MY POINT OF VIEW based on MY EXPERIENCE doing 4 years of mobile app automation.

That being said I have used the below tools/frameworks (which helps to do automation) before for different purposes and all of them are free (at least the basic features).

• Appium (everyone talked about it here)
• Robotium framework
• Calabash framework
• Adobe Edge Inspect (The basic feature is free I think 1 or 2 devices, please check and they may have stopped more development in Nov 2015)
• Sikuli (Python based GUI Automation tool)
• Selenium (If doing web automation)
• Monkey Talk

And there are much more…

Personally I have been using Appium+Cucumber+Java for my native app automation and EVERYTHING IS FREE.

Hope this helps.

Also, before doing automated testing, keep in mind on what you want to achieve by it. If you have a valid reason like Smoke Testing, Regression Testing etc. then it could be a good reason.

If you have more questions contact me at [email protected] and we can talk more about it.

NOTE: These are all my point of view and by no means I am an expert  🙂

[Raj]

Also, I have evaluated some of the tools in my blog posts, you can check it out on my website- http://www.rajsubra.com. If that helps.

[Gregory]

There are following aspects that are vital for the tester who does not have experience in automation testing, but has worked on manual testing.
1. First of all, there should be the close relationship with developers. It is a long way to make your first test run automatically. You need to set an environment, install a test tool and applications to test, run a connection script to link the test tool to the object to test, access the available interfaces (API, CLI, GUI, web, etc.) And there, the developer help is priceless.
2. Make preliminary selection of available on the market test tools, based on the constrains: operation systems, test types (functional, performance), available interfaces, company culture, available budget
3. Select from the initial set the best tool, which is more popular tool. It is because as a package you get access to discussions, blogs, answers on stackoverflow.com, etc. Because the most questions you will have are already answered there.
4. Find the popular framework to the tool of your choice. The modern trend in test automation is the shift from writing tests to declaring them. It is achieved by using the keywords. The existing test framework already have keywords libraries that cover 90% of a tester’s wild needs. For instance, the Robot Framework has libraries to work with all major interfaces. If you will need some exotic test with access to the object resource, you always can ask developer to write it for you and you can create new keyword that will call this script.

[Sbora Lili]

Hello @huynduong

Awesome and very usefull automation testing tools are :

– test complete : you have the possibilty to choose among JavaScript, Python, VBScript, JScript, DelphiScript, C++Script, and C#
– test studio : for web & mobile ,

– SoapAPI ( open source) : cross- platform for API testing , such as SOAP and Rest interfaces to ensure the interoperability of dif. applications

– Selenium ( open source) : a web testing tool

– and many other cool automation testing tools

[Jari]

When you say “Automated Testing” are you talking about Native app automation or Mobile web app automation?

Are we really sure what the person means with “automation”, at the end? For me, there is a certain level of automation for example when using spreadsheets.

In my current teams, “automation testing” is used mostly how different products integrate with other products, none of having  a GUI. Well there are GUI parts, but we focus on the algo’s and decision making code. Automation for us is not a question of a tool, but more like what we will build to test the software with. It does make me feel a bit alone here, since other people use tools made by 3rd parties.

 

Happy to see so many different experiences from people! I’d love to hear more about Robot Framework and the learning curve it has for starters.

[Lilis AyumSari]

it’s a nice discussion, reveal so many automatic tools recently used, I’m just new, use robotframework and testanywhere to do automatic testing in our project. Mostly testing for web apps project and it’s flexible since we don’t need more tools. tbh I don’t meet much person who joins test automatisation in my country (Indonesia), so  I think it’s hard to have a deep discussion for this.

[Andy]

I think Jari has a really good point here that it strongly depends on several factors and your individual answers, which tool is the best on automation testing

Jari wrote:

1. Web/Mobile/Desktop support needs?
2. Knowledge of programming?
3. Who does the testing? (For example, does a tester get help from the programmers?)
4. What is the goal for automation?
5. Is it more important to create tests fast than to create them robust?
6. Is the test automation throw-away code?
7. Will Record and Playback suffice?
8. What are the plans to go forward?

Especially his question #6 goes along with how good it is in it’s own version control or can be embedded in another solution.

1. could be more generally seen as the platform question. For Desktop e.g. it depends on which OS your application runs, also this might limit the choices you have. And also the platforms you want to cover for test creation (some tools offer less platforms for it’s designer/studio than for the actual test client).

In our case we are looking into leaptest: https://leaptest.com

It seems to fit best for our case, here a few facts why that is so:

• it’s very roughly comparable to SIKULI, but node based (building blocks), so testers don’t need coding knowledge (some have it some not), furthermore it’s very intuitive. One of our QA members took 20min from download, trial account creation, first try to a working automated test for our application.
• Testclients available for our relevant platforms (Windows and OS X)
• Offers real GUI interaction, which is vital for our tests
• features of buildblocks and the whole system seem to cover most of our required cases – details need to be found out, as well as…
• … how maintainable it is, how well we can avoid false positives and still detect reliable enough bad changes

To my mind anyway a challenge with test automation, especially if you have an application that has as one of it’s main functions to render any graphics content, and it’s acceptable if this looks slightly different but not subjectively different. For OpenGL drawing the results can vary in an acceptable way quite a bit, also due to newer drivers your product uses (would be a false positive if detected), but if a certain tiny part due to a bug is not drawn at all it can be very disturbing and an issue. When evaluating test automation tools and trying to estimate how good they are at distinguishing those important edge cases, it’s important but also hard to predict typical edge cases that could appear. Likely you might simulate defects from the past to find out if it would really detect what you want.

Regarding maintainability I found the approach from Applitools eyes very good (visual comparison, the strongest points: you quickly vote if the new result is better than the old one, then it becomes automatically your new reference image). For our use case it had to many limitations, it seems it focuses much on webpage testing, but there it might be definitely worth a look.

Last but not least some of the important automation tools we create in house our selfs. I prefer “off the shelf” software as long as it somehow fits, but in several cases we need to compare images on a 32bit color depth (per channel, so 3x32bit for RGB, not all channels summed up) basis. The very few tools that offer it in our last evaluation where not sufficient for our case.

So far my thoughts on that topic 🙂

Cheers,

Andy

[Andy]

…hm, I do really like leaptest, but I dislike that advertisement overlay on my reply 😉

Any idea how I can prevent this, it came by just mentioning their URL in my text.

[Jesper]

Andy : Fixed it by creating the link manually. /MOD:JO

[Jeremy]

Interesting discussion, but the question is very broad.  If talking about the best test tool to start Automation testing…

• in your QA group?
• coming from a manual tester stand point to learn automation testing
  • with some, or more, programming experience?
  • with no programming experience?
• coming from a development standpoint?
• to integrate manual and automated testing?
  • Will separate tools suffice, or is a tool suite with a central reporting point required?
  • Do you need/want hybrid tests?
  • Will it integrate with API testing?
  • Will it integrate with Performance Testing?
  • Will it integrate with Load Testing?

These are the first questions that came to mind.  It is a perspective issue regarding as to which tool to adopt.  As Jari and Andy have written, there are additional questions to ask as you look at automation.

My recommendation is to think of all the requirements you need the automated tests to perform, and then compare the suites that are available.

I’ve started Automated testing in multiple groups.  There isn’t a one size fits all. Here’s what I’ve setup in the past:

• In two instances we programmed our own test automation from the ground up
  • This was needed as we were testing down to the voltage level of processors and solid state drives
• Selenium
  • Automation was not in the budget and we needed to get moving quickly. Most of our QA team had a development background and we were able to get up and running quickly.
• Ranorex
  • We were needed a proprietery tool that was within budget. After researching mulitple tools this is the one that worked for all of our products.  We received excellent support where/when we needed it.
• Test Studio
  • After researching mulitple tools this allowed both manual and automation with hybrid capabilities wrapped into one, it was a nice low entry point for many of our non-development background people and allowed for integration with Visual Studio for our team members that had more of a coding background. It was a nice package for what we needed.

So, what’s the best?  The best is the one that fits what you need and the background of who is going to be using it.

[stefan]

I’m amazed no one said anything yet about a text-editor and terminal/command line -> best tools to start with.

Code scripts in text-editor, run them in command line.

[Archana]

I recently tried Tricentis Tosca and found it to be pretty good. It works for a variety of protocols / applications and is easy to get started with as no scripting is needed.

[Paul]

This recent article on test automation tool comparisons on the Huddle blog may be useful

[Dan]

I have just released the free automated functional web app testing software Automate Your Tests which is perfect for someone like you @huynduong who hasn’t got<span style=”line-height: 1.5;”> much automated testing experience.</span>

It should only take you 10 minutes to get your own custom test suite configured and running.

[Yevheniia Jane]

Main criteria of most popular testing and developing services:

• test management system – Create and manage projects, members.
• test planning opportunity – creating test plans, test cases, test scenarios or user stories.
• test run availability– creating test runs, managing members of the testing process, usability and efficiency.
• making test reports on work quality and quantity.
• task management tool – the bug tracker availability and its speed.
• using additional features – import/export, integration with other systems, crash analytics etc.I want to recommend you EasyQA, which I use.

  EasyQA has a great advantage – EasyQA SDK. If your test product is a mobile app for Androidor iOS, it proves useful. Our library is able to catch and send crashes to dashboard.It allows taking screenshots, recording video, making bugs and sending all of this to our bug tracker from the phone

  Also you can read about 10 test management tools.

[Yevheniia Jane]

Main criteria:

• test management system – Create and manage projects, members.
• test planning opportunity – creating test plans, test cases, test scenarios or user stories.
• test run availability– creating test runs, managing members of the testing process, usability and efficiency.
• making test reports on work quality and quantity.
• task management tool  – the bug tracker availability and its speed.
• using additional features – import/export, integration with other systems, crash analytics etc.

[commercial content removed /mod:jo]

[sneha shinde]

there are many tools for automation testing! I can recommend :

• Appium Automation Framework
• UI Automator Tool:

[edited mod/jo]

[iain]

It is a good question and for a manual tester wishing to become an Automation Tester, you need to consider:

– do you have any development experience? If Yes, then try Selenium Webdriver. If No try HP UFT or SmartBear TestComplete and use the record & playback features to show what can be done.

– is there a dominant technology at work? E.g. some automation work better on certain platforms and technology.

Hope this helps and update the post when you make a choice.

 

[Author]

Posts