Here it is! Each month on the first Tuesday of the month, we will post a new article to take you through a step-by-step guide on how to address a particular aspect of using Selenium as part of our How To series. This post continues Anton’s guide on using Page Objects with Selenium.
________________________________________________________________________________________
Delivery cycles will get even shorter. More than half of respondents want to have shorter delivery cycles in the next two years. Sixty-eight percent (68%) want to deploy weekly, daily, or multiple times a day, as compared to the 41% who do so already. (The State of Testing Report for 2017 by SmartBear)
How Do People Run Their ‘Heavy’ Tests?
Here, by heavy test, I mean tests that take more than 1 second to execute. Usually, these are DB, UI, API and system integration tests. The execution time starts from 1 second to a couple of minutes. On average, we can accept that the completion time is approximately 30 seconds. In the different channels, Automate The Planet has over 25000 followers. To prepare for the talk, we asked them to participate in a short survey. I will use the data to illustrate some points. It summarizes the practices from more than 50 companies.
The first question in the survey was- “How many tests do you have and what type?” I have summarized the results in 5 groups based on the count of these heavy tests.
As you can see, almost half the respondents mention they have between 500 and 1000 heavy tests. If we accept that the average tests number is 750 (seven hundred fifty) and the average tests execution time is 30 seconds. We can make a simple calculation. The test run will need more than 6 hours not complete if the run is sequential.
In this case, executing all tests in CI or using them for CD is impossible. Why is that? If you want to be able to release each day, your whole tests suite should be able to complete at least two/three times a day or under 3 hours. This is necessary, because often, there is a bug in the tests or there is an issue during the run, and you may need to repeat the whole process.
Advantages Distributed Parallel Test Execution
The most obvious reason is speed. Instead of executing the tests in 16 hours, they complete in under 4 hours. When all required tests can be run a couple of times in a business day, you will be able to release your application every day (if we talk about web projects). Even if it is not web, you can improve the quality of your app and shorten the testing cycle by executing all your tests as part of the CI process. This means you will have higher coverage in shorter throughput time. As you know, each time your tests execute, their ROI (return of investment) increases. Last, the more often you run all your tests, the probability of locating unstable/not-well-written tests rises.
Parallel vs Distributed Test Execution
I can define at least three types of parallel test execution:
Parallel- Single Machine – Multiple Processes
Some unit test frameworks have native parallel execution support like NUnit. You have, for example, 100 tests. If you run them on a single machine with 5 CPU Cores, on each core, 20 tests will be executed simultaneously. However, not all test frameworks support this option, and there are some major problems related to it, depending on the type of tests you want to run.
Distributed Testing- Multiple Machines – Single Process
Your second option is to run your tests at the same time on multiple machines and merge the results at the end. Usually, you need an additional complex tooling, for example Microsoft Test Controller/Agents setup.
Parallel Distributed Testing- Multiple Machines – Multiple Processes
You can mix both approaches. In this case, you will use the complex tooling and, at the same time, run the tests in parallel on each machine.
You can find a complete comparison between some of the available tools about distributed testing in the article- Distributed and Parallel Tests Execution of WebDriver UI Tests
Meissa Distributed Test Runner Meissa is an open source distributed tests runner. It is built using the latest technologies such as .NET Core, ASPNET.Core and more. Therefore, it is completely cross-platform. It is designed to be programming language agnostic, which means that it can run tests written in different languages.
Why Did We Build Meissa?
To understand why we created the tool, I need to tell you what we needed to test. Our company specializes in building test automation framework and tooling for web, mobile, desktop, and API. It is
not a visual application rather than a set of C# libraries and other tools to speed up writing of tests scripts. My belief is that, when you build a testing tool, you need to make sure it is covered with lots of tests. For example, for the web part, we support 8 browsers, the five major ones and three headless. The web library uses WebDriver in its core but adds many capabilities on top of it, so we have lots of tests related to Selenium grid mode and the integration with the cloud providers. We have tons of other logic that needs to be tested, so we ended up with over 5000 UI tests. I skip all the unit tests, since they execute quickly. Actually, most of the UI tests are quite fast too, since we use Bellatrix to test itself. But, if we execute them sequentially on a single machine, it takes more than 16 hours to execute them.
We need to be able to deliver updates a few times a day so this was unacceptable.
Now with Meissa, we execute them in under 4 hours, and this is without using the in-parallel Bellatrix feature. We only distribute all tests on four virtual machines in Azure. Find out how you can execute 100000 Tests Under 20 Minutes With Meissa Distributed Runner in our biggest experiment.
Common Challenges Running WebDriver UI Tests in Parallel
There are many benefits in running tests in parallel. However, you need to know there are many challenges to overcome before you are ready to speed up your test runs this way. Some of them are:
Preparing Test Environment and Managing Tests Data – one of the most significant challenges in writing maintainable automated tests has been making the tests autonomous, not depending on each other.
Keeping the Clean State of Agents
Video Recording during Testing in Multiple Browsers- if you run multiple browsers on a single machine, the video recording of the tests become useless.
Making Screenshots during Testing in Multiple Browsers – the same is valid for taking screenshots for some drivers that take screenshots of the entire desktop instead of the browser tab.
Changing Port on WebDriver Initialization- another problem we faced trying to run our UI tests in parallel on a single machine was that not all drivers get a free port on initialization. If you attempt to create the driver a second time before disposing of the first one, it tells you that the port is already in use. The same may happen if you use proxies.
Edge Driver Cannot Start Multiple Instances – once the driver is disposed, it closes all currently opened browsers (even not opened by WebDriver), which makes the execution in parallel on a single machine impossible. Moreover, it is designed in a way that does not support multiple WebDriver instances, even if you assign a different port on initialize. Read even more details about the above issues and their sample solutions here.
About the Author
Anton Angelov
