Test automation tools: open source vs. paid license

[Alin]

Test automation is one of the most challenging topics in testing. There are a lot of automation tools for web, desktop and mobile testing. Some of them are free, others require a paid license. I have worked with the following tools:
– Selenium as a free tool for web applications
– HP UFT as a paid tool for web and desktop applications
– SoapUI for API testing with a paid pro license but it can also be used as a free tool because only the pro version that has some extra features requires purchasing a license.
Both categories (free and paid) have their own pros and cons. I find Selenium faster when running tests than HP tool and more flexible with programming languages (Java, C#, Ruby, Python) while on UFT only VBScript is available. But with UFT you can test desktop applications while Selenium cannot handle them and also HP offer support when using its products.
What do you think about the differences between using free and paid tools? What are the other advantages and disadvantages for each category? It would be nice to have here as many opinions as possible from your experience with some examples because there are many tools available. You can take into consideration general aspects like bug tracking, supported programming languages, version control, continuous integration, maintainability of tests, support when using the tool. I have talked once to a test manager and he said he decided to choose a paid tool because it offers dedicated support.
I will also be interesting in your opinions regarding mobile applications.
Regards,
Alin

[Edward]

Unfortunately this is never an easy question to answer the usual response is; It depends?

Price advantages in using free verses paid tools is an obvious areas where people concentrate their initial thoughts, but this does not give a true picture, there are various other factors that all affect the choice in free or paid tools, but lets look at a few:

1 Cost – What’s the tool budget – Is there a tool budget? Has anyone worked out the cost verses reward on using a tool to determine a budget (ROI) return on investment?
2 Security – Are there security restrictions, can the tools be used are they on the clients approved tool list?
3 Time – A couple of factors here how quickly can we get the tools, install them, set them up? How quickly can our people use them and create usable automation scripts. how much time is required to maintain the scripts?
4 Infrastructure – can we load the free tools on our infrastructure – approval required may affect time and cost may be affected if you have to host independently. What infrastructure do they need, local install, server install?
5 Skills – Do we have people experienced in using the skills, are they readily available, can they be trained if not, how long does this take how much will it cost, are some tools are easier to recruit people for as they have more practitioners?
6 Support – Are the tools supported, are there non IPR protected examples Is there training available, are there forums, support lines etc..?
7 How flexible are the tools or how flexible do they have to be? You don’t need an expensive do everything tool if you have a very specific test request that can be met by a simple free tool.
8 Who will do the automation, in the past this has usually been reserved for Technical Testers or Test Engineers, but more and more clients are leaning towards non scripting automation such as Turnkey that can be picked up by non-technical business users.
9 Is the client insistent on a type of tool – many specify what you use and there are no options available?

All the tools you mention are valid and used by many clients I design testing solutions for and key is understanding the client requirements, timescales costs and business priorities and these will usually be reviewed against all the factors above to ultimately lead to a tool choice.

Regards

Edward Patmore
Enterprise Test Consultant

[Jon]

First, there is no such thing as a free lunch. Almost anything “free” has some price to pay.

Edward has a good list. I like it.

On cost for “free tools”, I expand and add:
1. It may be “free” up front, but if you must customize, fixes issues, or have create “work around” actions, these are not free
2. If there is no fast support “help line”, like you get from a vendor, this may be another cost
3. If the tool does something like happened to Apple Apps recently (inserting malware), this is impacts cost and security issues (not to forget impact time and PR)

However, that is not to say I recommend vendor software tools totally either. Issues to consider here include:
1. Software vendor tool is Proprietary so you can’t modify their code/system to add a feature the team might want (you can do this in open source free tools)
2. Vendor tools can be “closed” on the output data structures so one can’t move between different tools quickly.
3. Will the vendor be their in the future to support their tool (vendors go out of business, and this may be a big issue for mobile).

So, the “it depends” is what a team must consider. I recommend a trade study be done since there is not “best” answer or tools. This is true for both traditional computer systems and mobile.

The tool-scape in the mobile world has matured in the last few years, but now there are almost “to many” vendors and “free” options in tools to choose from. This means a trade study will take longer and have a larger trade space.

This is why we should get paid for the testing we do.

[Janez]

I won’t name any tools since I produce and market testing tool (free and paid version), but in my 20 years of professional development/testing I stick to the following principle:
If the tool/component is absolutely essential for your business, then go with the paid version, if possible.

The major reason for such opinion is continuous support you (usually) get with the paid version. Too many times I have experienced situation where I would use some free tool and then its development stopped or slowed down often because there was no motivation anymore for the guys behind the project or they simply went into other projects. But since I stick to the principle above, this was never a serious problem.

Not having a source code for the tool was never a problem for me; if the tool is dying, source code does not help you much; you will switch to some other tool sooner or later. If the tool is alive and well, you usually do not need to fix things on your own.

In short, for some non-essential support tasks, go with the free tools, for something essential to your business, go with the paid. This conclusion is drawn completely from my long experience, nothing else.

[Aleksandr]

The all tools use the same methods at the end. The free tools is need more attentions from your side. And this is all differents. If you have in your team the specialist with lowest experience leven when use the tool with high prices per licence, but if your team have experience in testing the free tool is always welcome.. You can save buget in this case for othe activities.

[Mark]

It is always an interesting debate free versus paid. I think the major points have been covered by the previous responses. But here is my 2 cents.
A while ago I looked at why automation fails to meet expectations and in a lot of cases dies quickly and two significant things became apparent.

1. The first is that automation is oversold, have you heard the salesman say ‘With our automation tool you can do 3 weeks testing in 3 hours overnight after the code is delivered’. Well this is just not true (unless you don’t find a single problem), you may be able to execute the tests but you still need to identify and double check the error, raise a defect, work with the developer, retest the fix etc. So you can’t sack half your testers!
2. The second is that there is always an ongoing maintenance cost and this is the killer. In a world where we work with lots of integrated systems consuming each other and each system has regular updates to keep up with current market fashions the level of ongoing maintenance is high.
The problem some companies have is that they bring in a resource to do the automation for a project and when that project is finished the resource leaves. The automated test assets are then needed again, but they weren’t written to any standards so there is no supporting documentation and they are almost impossible to revive.

The Analogy – I love analogies!
Building a House – You can use the cheap hammer to build your house or you can invest in a nail gun. One is low cost but labour intensive the other is high cost but faster and easier to use. In some cases you may need to use a specialist hammer for a particular job.. If you have lots of labour sitting around doing nothing the cheap hammer makes sense, but if you have time and resource constraints then the nail gun will get the job done faster and usually produces a better quality job.

But the fact is, the thing that defines what the house looks like, how well it is built, how efficiently it is built, how easy it is to maintain and modify is not the hammer but the plan and the building standards!!

Too often automation is produced without a plan or standards and without a view on how it will grow and be maintained going forward, as a result it often can’t be.

When we train up our test automators or performance testers we always concentrate on the big picture, the pitfalls and factors that affect what and how we produce scripts. We don’t want a script monkey (God knows there are plenty out there), we want to produce an automator or a performance tester. We also have generic standards that we tailor to meet each customer’s needs.

Conclusion
So for me the tool is a tool, if we want to improve our profession and the service we provide we need to throw our efforts into the plan and the standards and stop over selling automation. In short we need to be more professional.

Answer to the original question – Given the choice (all things been equal) I would choose a bought tool as I can usually deliver a better quality (maintainable) automation suite in a quicker time.

[Roman]

Not every free tool is open source, but in general in most cases I’m interested in free tools.

Let me start from rather distant topics and then close on main question. Do you read this forum using free or paid browser? I bet you use free one. Even while IE may be considered paid (as part of paid OS), Chrome and Firefox are more popular. Browser is a tool as well. And it takes part in automated testing, it really does: so many locators for Selenium Webdriver I took or composed and tested using Firefox / Firebug and Chrome. Do you remember paid browsers? I do. I remember that 10+ years ago Opera was shareware and ad-supported. Now what? Did it help Opera to have a paid version? It didn’t.

Do you see around yourself more people with Android or iPhones or WinPhones? AFAIK, iPhone is more popular with US market, but there are many not that rich markets, and Android is free, what makes Android devices cheaper, and more popular, as manufacturers are not required to pay license fees and obey Apple or Microsoft etc. There was a time I tested products for Windows Mobile phones, but most of them are history now, and Android is alive and popular.

Do you know that Linux and Linux-based products occupy substantial share of server OS markets? I think you do.

I named browsers, mobiles, servers. This itself occupies substantial share of markets, and you have to use it if you are going to make popular products for global markets, do you like it or not.

Now close to the tools.

What makes me opponent of paid tools.

Entry fee and community size There was a time when I was testing client-server news aggregator client applications for different platforms. I had a possibility to compare Android and Apple (and Microsoft) developing environments. Of course, Android had a lot of problems and still has. But to develop for iOS at the time (2011) meant that you have to buy Mac, pay for Mac OS X, pay for Apple developer license, etc. This was pretty hefty sum, and when I learned that there are more Android than iPhone activations and more Android than iPhone apps, I took it as expected. Free tool is going to have a larger community.

Education and chicken-and-egg problem
Educational organization and students are often not that rich. And to learn a tool (with some course or on your own) you have to use it. If it is free — no problem. If it is without free license, than to learn it you must pay for it or someone must pay for it for you. This, again, makes free tools more popular and community for free tools larger. From the business point of view that means that there are more people who have experience with free tools, and they may be hired cheaper. If you want to narrow community, make every member pay. Be it club or game or whatever, paid-only membership makes community smaller, any type of community, including development and testing ones.

License and instance cost Paid tools in general are licensed per-install, per-seat, per-user, per whatever they want to get money from you. If you want to use 20 instances of Selenium, it bears not additional costs. If you want to use 20 instances of some paid tool — it may cost you 20x tool cost.

Per-period or renewing fee Some tools are on SaaS or other subscription-based pricing model. If you use it, you are on a hook. Nobody likes to be on a hook. “My monthly bills are high enough”.

License restrictions Adding to instance cost, license may prohibit you from doing what you want. Say, for one tool I investigated it was forbidden to use it for servicing clients. But my job was exactly servicing clients!

Commercial tools must be paid upfront… but a project of yours may be experimental and you or your partners/ customers may be not interested in substantial investment with unpredictable return.

Project may not justify tool price I tested a lot of project which development time was 2-6 months. Why even consider to buy something for project lasting that long (and you are not sure that similar project follow).

Now for the names:

Various “monkey” automation tools (starting with Microsoft Hopper Test Tool for Windows mobile)
Selenium — many know it
Robot Framework — free keyword-driven framework with libraries for various purposes, extensible
Sikuli / SikuliX — cross-platform image-based automation tool
Appium — mobile automation tool
Specflow — ATDD / BDD tool

[Kasper]

Pretty much a moot discussion.
You can get paid support for Open Source Tools. Think Red Hat, Canonical or IBM for starters.
Just use the tool that is right for the job. You will need a sys admin anyway (if your organization has some size) and you can pay for support if you feel you need that.

That said I like to be in control of my tools and tend to use Open Source where possible.

ps. Open Source or Free is not the same as free of cost.

[Christy]

Hi everyone i am looking for a load testing tool preferably an open source tool for load testing.Presently i use test complete(Smartbear) for managing regression test.
Can you please tell me any load testing tool for a desktop appliaction

[Kasper]

@Christy: JMeter…
I am pulling your leg here of course. But you don’t give us much to go on so I just give my personal favourite with good performance, easy to extend with plugins and loads of supported protocols.

But I suggest you read
http://www.opensourcetesting.org/performance.php or the shorter
https://blazemeter.com/blog/open-source-load-testing-tools-which-one-should-you-use

[Pavan]

Testing a software is an essential activity for the businesses to provide high quality experience for the end users. There are a lot of open source software testing tools in which Selenium, Sikuli and Watir are usually used open source automated testing tools. Selenium is the most used open source tool for automated testing of Websites. Selenium is very popular and it is the first choice of businesses for automating the testing of Web-based applications for both the GUI and also for the functionality. Over a last few years, open source tools have gave better capabilities than the commercial tools. All commercial tools have some positives and negatives. It all depends on the automation engineers who can understand the concept and leverages the tool to the fullest. They mostly go after open source, because they can customize the open source to their needs.

[Edited by moderator: please see the community guidelines]

[barret]

We have both worked with paid and open source testing tools. In both areas there are really good tools available, but in the end we decided to invest in a paid license. The main reason for that is the support. Of course some free tools (e.g. Selenium) have many forums where users can ask questions. However, there might also be problems others have not had before, so professional support is needed.

There are many comparisons between free and paid tools available, here are some examples:

https://www.linkedin.com/pulse/open-source-testing-tool-vs-paid-anjan-biswas

http://www.ranorex.com/ranorex-vs-selenium.html

http://www.testing-whiz.com/blog/comparing-qtp-selenium-and-testingwhiz

[Paul]

Hi Barret, there’s also a blog article on Huddle comparing software testing tools added in the last month or so.

[nikhila]

hi everyone,
Try this AnyAUT test automation tool which makes automationTesting easy and fast.At AnyAUT we take care of both these aspects, delivering accurate test results and to deliver it fast.
AnyAUT can be used to automate most tests for web based applications. AnyAUT can also help you in migrating your tests from licensed tools such as HP QTP/UFT to free and open source tools such as Selenium and JAVA.For more about this click here

[sneha shinde]

There is a wide range of tools available for automation testing. Some of them are free, while some are quite costly. Some automation tools were developed years prior; some have quite recently showed up in the market. Every tool is novel and has certain qualities.Wide variety of accessible automation tools makes it hard to choose the most appropriate ones for a project. The issue is that scarcely any of the current tools completely corresponds to the necessities of a project.

1. UI Automator Tool:

This tool has been as of late expounded by Google. It bolsters Android versions starting from 4.1. One ought to choose another Android application testing tool keeping in mind the end goal to automate tests for prior renditions.

2. Appium Automation Framework:

It’s a system for creation of automated tests for Android and iOS. It is a free to use tool. It underpins Android variants from 2.3 and later. Appium uses WebDriver interfaces for tests running. It is compatible with programming languages like Java, C#, Ruby and other which are in the WebDriver library.

3. Robotium:

Robotium is a free Android UI testing tool. It is appropriate for test automation for various Android versions and sub-renditions. Automation engineers regularly portray it as Selenium for Android. Tests made by Robotium are composed in Java. Indeed, Robotium is a library for unit tests.

4. Monkey Runner:

This tool is more low-level than Robotium is. One doesn’t need to manage the source code so as to automate tests. The tests are written in Python, one may utilize a recording tool for developing tests.

5. Ranorex:

Ranorex is a decent tool for tests automation for the most recent, as well as for early versions and sub-versions of Android, starting from Android 2.2.

One of Ranorex preferences is its definite reports with screen-shots. It can link a cell phone or a tablet to Internet by means of WiFi.

[commercial content removed /mod]

[Author]

Posts