Test automation is not a new topic, however, mobile automation has become a trend relatively recently. And the demand for service is constantly growing. It is caused by the fact that handheld mobile applications become more functional, taking on the role of desktop PCs.
Mobile app automation testing is a software verification process that involves performing such basic functions and test steps as running, initializing, executing, analyzing, and issuing a result, automatically, using specialized tools. It is applied in order to smooth and enhance solution’s functionality, usability and consistency. Testing applications on mobile devices is generally consistent with the general principles of testing, but there are also a number of features that are specific to mobile applications testing. In order to understand the features of mobile app testing services, it is necessary to take into account the factors that distinguish a mobile application from a desktop one, namely: specific and diverse operating systems for mobile platforms, different configurations of components, functionality of devices such as communicators, etc.
Due to these factors, the approach to testing applications developed for handheld mobile devices is quite different from the desktop one. A large number of additional nuances and requirements that need to be tested appear, so that to meet the expectations of the end user. Considering increasing functionality and frequent updates it is not always possible for a manual tester to cover all regression cases in a short period of time. And this is where automation comes to the rescue.
Advantages of mobile automation
The advantages of automated testing of mobile applications largely coincide with the advantages of automated testing of web applications:
- Quick results;
- The exclusion of the human factor;
- Parallel testing on multiple devices;
- Ability to cover a lot of localizations.
The advantages are significant, yet there are enough of…
Disadvantages:
Autotesting stops working with an operating system update. Therefore, after each update, the OS will have to wait until the tools for developing automatic tests are updated. Difficulty in writing tests. Compare: it takes an average of 8 hours to write one autotest for a web application, and a mobile one takes about 20 hours.
Unstable work of many tools. For mobile platforms, there are many different UI components. By virtue of diversity, it is physically impossible to develop a single tool capable of working with any type of UI, or its use is extremely time-consuming, technically difficult and, as a result, economically unprofitable. Requirements for iOS: all ipa-files must be signed by developers.
Selection of approach and tools for automation
With the automation of web application testing, everything is more or less clear: Selenium WebDriver is the most popular tool on the market that all professional testing teams have mastered. But when automating the testing of mobile applications, there is no clear answer when choosing a tool. However, the main question concerns not the choice of instrument, but the approach. It is the chosen approach that will predetermine the tool for the subsequent development of autotests.
When automating the testing of mobile applications, you can use one of the following approaches:
- Record and Play
- Screen object
The first method is simple and comes down to recording all user (tester) actions in the application. After recording the actions, the tool generates a code that it understands and creates autotests.
Pros: fast implementation, no programming knowledge required;
Cons: the slightest changes in the application will require you to create a new autotest.
The second approach is Screen Object – a pattern intended for the organization of the autotest architecture in the form of interaction between application screens. Screen Object models the screens (or pages) of the application under test as objects in the code. As a result, we get a set of classes, each of which is responsible for working with a separate screen of the application.
Pros:
- It is possible to reuse the code;
- The reliability of the code, a small sensitivity to changes in the application;
- Clear structure.
Cons:
- Requires knowledge of programming languages;
- Development speed.
Of course, for a complete and high-quality testing of the product, it is recommended to use the second approach. If your testers do not operate any programming language, we advise you to consider the option of attracting a third-party team. Well, approach is selected. Now is the time to pick a tool.
There is a kaleidoscope of automation tools. But it is important to note that some of them are suitable for applications written in a specific programming language. For example, Xamarin is only suitable for C # applications, whereas some tools may cost you an arm and a leg (for example, the Ranorex utility). Therefore, it is very important to weigh the pros and cons of not only the process itself, but also of the tool for its implementation, prior to the introduction of automation.
TOP 6 Best Open-Source Mobile Testing Frameworks
Appium
Appium is an open source, cross-platform tool that helps automate for both Android and iOS. It is one of the most widely used tools for creating automated tests for smartphones and tablets. The undoubted advantages of Appium are ease of use, as well as it support many programming languages: Java, Ruby, Python, C #, PHP.
At the moment, Appium is one of the best solutions in its niche. Designed to help with UI-testing mobile applications.
Pros:
- Search for application elements via XPath, selector or ID, and work with them (press, enter text, scroll)
- Less flexible functions are also supported, such as pressing or swiping along coordinates on the screen, or emulating pressing hardware buttons (back, home, etc.).
- Supports popular mobile OS (Android, IOS)
- Supports the most current PL
- Good community
- Easy to install
- Documentation covers most emerging issues.
Cons:
- Does not recognize interface elements in programs / games created on multiplatform engines (for example, Unity, UnrealEngine)
- Sometimes there are problems with synchronization.
- Sometimes it provides poor, insufficient reports.
Calabash
Calabash works efficiently with .NET, Ruby, Flex, Java and other programming languages. It tests native and hybrid mobile apps. Programmers can have APIs that enable native apps to run on touch screen devices. Calabash involves libraries that permit test-code to interact with hybrid and native apps programmatically. It also supports the framework Cucumber. Calabash is a framework for automating functional testing, which, in essence, is the driver that controls the operation of an application on a device or simulator.
Pros:
- The user can perform Calabash test case execution on more than 1,000 real mobile devices in the Xamarin Test Cloud, a dedicated cloud lab for testing.
- Clear and simple documentation regarding the operation of the application.
- Cucumber support for conducting development testing based on functioning (Behavior-Driven Development).
- Parallel execution of test cases.
- Mobile functions such as gestures, scrolling, etc. are widely supported.
- Scrolling can be automated.
Cons:
- Various startup commands (cucumber, calabash-android run)
- There is no way to work in debug mode
- It takes time to run on an emulator or device as it always installs the app first before starting the first scenario.
- Test cases are written only in Ruby.
- Special targeted training is required for testing iOS applications.
- There are no recording and playback functions.
- There are no code generation tools.
Selendroid
Carrying out automated testing of mobile applications of different types, you can use this tool. With the help of Selendroid, you can conduct web testing, as well as validation of Android native and hybrid apps. Moreover, it holds exceptional capabilities to interact with multiple Android devices at the same time. In addition, this framework can be used for testing both on real devices and on emulators.
Pros:
- Android testing with Selendroid does not require any changes in the application itself. You only need to install a binary file (APK) on your computer.
- Compatibility testing will not be difficult with this framework, since Selendroid can interact with different devices and emulators at the same time.
- The tool can simulate user actions – tab, drag, scroll, etc.
- You can enable and disable plug-ins directly during testing without restarting or stopping tests. Selendroid recognizes new devices automatically.
- The framework supports new versions of the Android API.
- Thanks to the integrated monitoring tool, UI testing will be much easier. This tool identifies user interface elements (text fields, buttons, etc.).
Cons:
- Direct support for Android applications only. Although, it should be mentioned that you can use ios-driver to test native, mobile and hybrid applications.
- The user does not have the ability to automate actions outside the application – camera, maps, etc.
- Selendroid is quite slow. It is difficult to use on devices with RAM less than 4GB.
Robotium
Robotium is one of the leading frameworks for automated testing of Android applications.
Pros:
- Robotium is an easy to write test case.
- With the help of Robotium, the user can write scripts to conduct functional, system and acceptance testing, while covering several Android actions.
- Fast execution of test cases.
- It can be used for testing applications with available source code, as well as for applications where only an APK file and execution details are available.
- Robotium supports Android features such as actions, pop-up messages, menus, and context menus.
- Integration with Maven or Ant.
- Automatic synchronization and latency.
Cons:
- The ability to conduct only one test at a time.
- Robotium does not work with Flash or Web components.
Ranorex
Ranorex is an easy-to-use, automatic testing tool for developing and managing projects in teams consisting of testers and developers.
Pros:
- Easy installation even for those who are not programmers.
- “Click & Go” xpath editor.
- User Interface Map Editor.
- Dynamic IDs support.
- Easy maintenance.
- Simple execution of files with the extension (.exe).
- Professional service support.
Cons:
- Does not support Mac, as well as testing hybrid and Android web applications.
- There is no option to export scripts to various programming languages – Java, VBscript, etc.
- There is no support for gestures, as well as “drag and drop” actions.
- In the event of a crash or hangup, the automatic launch of a closed application is not supported.
- Long waiting time for object identification.
Espresso
Espresso is designed more for white-box testing and was created as a tool for developers. In general, if you want to test only the application, without taking into account its interaction with the system, and there is a desire and opportunity to work with the sources, it is better to use Espresso.
Pros:
- It implements useful functions like automatic synchronization of tests with the application’s UI, so you do not need to write various wait-commands.
- It supports older APIs starting at API level 10 (Android 2.3.3).
- This tool has a recorder via which you can record simple scripts and use them at the initial stage of automation.
Cons:
- it requires access to the source code to run.
- can not work independently with other applications and the Android system.
- If you need to test the application in conjunction with others or with system functions, and there is only access to .apk, then better to choose UIAutomator.
EarlGrey
EarlGrey is a tool for automated UI testing. It was created by Google. Performing mobile testing with EarlGrey, experts can create simple, easily understandable and compact autotests. The EarlGrey tool has a lot in common with Espresso, so it is often called Espresso for iOS. This tool is used to test native iOS applications, including Google Calendar, YouTube, etc. EarlGrey can also be integrated with the Xcode test navigator. Thus, you can run UI tests straight from Xcode or via the command line.
Pros:
- Automatic synchronization with user interface and network requests. Perhaps this is one of the main advantages of this tool.
- Using a special screenshots comparison utility, EarlGrey determines the visibility of the interface elements of products before interacting with them. Thus, you can easily make sure that the user can see and interact with the product interface.
- With the help of EarlGrey, you can check the peculiarities of user interaction with the application by simulating the actions of the user.
Cons:
- Earl Grey is for iOS only, and tests must be written in Objective-C or Swift
- EarlGrey finds multiple elements even though we know that there is only one element.
- Swipes, taps, and searches take too much time to be checked.
- Cucumber is not supported, that is it could be hard to write tests for non-devs, and what’s more, Cucumber test reports are not available.
- May require additional coding in your APP for using EarlGrey feature like ‘wait APP to idle’.
- Has to be in the same repo as the main APP.
- Have to manage schemes to run tests on CI.
Overall, it is difficult to overestimate the importance of the testing process in the world of mobile applications. An error, or bug or flaw in the application within a few hours in the case of Google Play and within a few weeks in the case of the App Store will cost the developer tens and hundreds of negative user reviews, low ratings and no audience growth. In other words – a complete collapse. If you have decided that mobile testing automation is what you need for your project, make sure that the chosen approach and tools meet all your requirements. And be flawless!
