Mobile Testing Basics – What You Need to Know

With the rise in Internet of Things (IOT), one can expect the use of mobile to continue exploding. As with testing other applications, improving time to market as well as application quality, while keeping testing costs low, remains a key driver for organisations looking to roll out mobile applications to the marketplace.

Mobile consumers however have significantly lower tolerance for buggy mobile application as compared to desktop or web users. Remember blue screens that you used to encounter while navigating desktop applications. After a while, you got so used to them that you actually started caring less about them. Mobile users however aren’t that forgiving. In fact, a recent TechCrunch article suggested how only 16% of mobile users will give the app another try after it has failed once.                                                                                                       

On testing tool vendor side, the market continues to evolve rapidly and is not yet standardized, resulting in various vendors offering varied solutions for mobile testing. The growing role of open source tools such as Appium, Robotium, and MonkeyTalk makes the marketplace even more dynamic and testing tool selection further complicated. It thus becomes critical to understand key characteristics that make vendors different and then map these features to your mobile testing strategy.

Mobile necessitates adjustments to your current desktop/web testing strategy            

It is crucial to realize that testing mobile applications requires a different approach than testing desktop or web applications. This is primarily because of the following:

1. Fragmented Marketplace: Android is the main culprit here, after having introduced a variety of operating systems such as Jelly Bean, KitKat, and Ice Cream Sandwich over the past few years. The additional fragmentation resulting from screen size, device type, and hardware specifications can easily make testing messier.And with the Internet of Things, gadgets such as Fitbit becoming more prominent, rampant device fragmentation means that testing mobile applications needs to be device specific, often resulting in longer testing cycles than that of desktop or web.                                   
2. Increased API dependency: Behind every mobile application or a connected device, there is at least one API that enables the front-end graphical user interface (GUI) to interact with other resources.  These API’s and GUI are interdependent on each other to collect information. For instance, in case of Yelp, google maps API determines which nearby restaurants need to be shown on the mobile screen. Similarly, depending the information (Restaurants, Bars, Coffee places), you type on the Yelp GUI, API generates a different search result.      Testing front-end GUI of mobile applications for different API responses or testing API for variety of GUI parameters can become more cumbersome when third-party API’s are involved.  This is primarily because third-party API’s such as google maps are often not available for testing or even when they are, calling them from within automated tests can be really expensive.
3. Sensor data determines front- end GUI behavior: The front-end graphical user interface (GUI) of mobile application depends not only on the API responses, but also on mobile device hardware/sensor data. For instance, sensor data such as Wi-Fi, accelerometers, and compasses controls GUI behavior. Additionally, access to real-world sensor device data may be needed for testing location-based actions such as ensuring locale of the app is set depending on the place it is being accessed or downloaded from.                                                                                                                                                                                                                                                                                                                                                              
4. Dealing with Variety of Input Types: Varying gesture sets such as swipe, pinch, long touch, drag and drop, scroll, etc. across multiple operating systems, screen orientations, and devices necessitate UI automation in case of mobile. Additionally, depending on the application being developed, testing alternative input types such as voice and camera may be essential.        
5. Performance and nonfunctional testing: Nonfunctional areas like performance are critical in determining the adoption rates of mobile application. The performance of an app is also governed by external factors such as whether the app is being accessed through a cellular network or Wi-Fi. It is entirely plausible that a mobile application may behave differently on a Sprint network than it does on an AT&T network. Another nonfunctional area that needs to be focused heavily while testing mobile app is security.                    

Solution to some of the challenges that mobile poses

1. Move testing to the left: Shift testing closer to development and ensure quality is built-in from the very beginning to reduce technical debt. Some of the ways to infuse quality and reduce testing time during the later portions of the project could be:
   •  Follow behavior-driven development (BDD): Ensure QA teams, developers, and business analysts work together from the start of the project in order to align business requirements and testing needs. Approaches such as BDD ensure that the team can define test scenarios for various business requirements even before development starts writing automation code for different scenarios. Open source tools such as Cucumber can come in real handy here.                                                                                                                                                         
   • Ensure code quality: Improve communication and collaboration across development teams using peer code review tools.  Performance optimization problems arising due to code changes can be avoided by setting up workflow-based reviews in peer code review tools. Additionally, static and dynamic analysis tools can come in handy to ensure there is architectural stability and security/performance problems are not encountered when the code is checked in.
   • Invest in Service Virtualization: Simulate full functionality of an API or a service that is still in development or design stage using service virtualization tools. Developers and QA professionals can thereby reduce their dependence on API’s or services being developed by other teams and start testing earlier. Service virtualization tools may also come in handy to create different scenarios for testing third-party API’s that may be costly to call or even not available.
2. Don’t treat GUI and API testing in silos:  Since mobile necessities use of APIs and other third-party services, it is extremely essential that the automated solution you select allows you to go beyond the GUI layer.
   •  Integrated GUI and API testing: An integrated GUI and API testing tool can come in real handy while quickly finding the root cause. An ability to pinpoint issue at the GUI or the API layer allows the GUI testers to cut down on intermediary steps.  For instance, on realizing the issue is at the API layer, the GUI tester can reach out to the API developer without going to the GUI developer first. This saves testing time, which is of essence in case of mobile.                                                                                                                        
   •  Focus on Test Automation Pyramid: To reduce maintenance costs and create solid test automation strategy, focus on creating more unit tests, followed by API tests, and then fewer UI tests.  Do ensure security and performance tests are a part of test automation pyramid strategy.                                                                                                                                                                                                                                    An integrated GUI and API testing solution can make the process of implementing a test automation pyramid easier as it ensures changes made to the API test cases are automatically reflected at the GUI level as well.
3. Support for multiple skill sets: Automation skill sets of team members testing mobile application vary widely. Manual tests just moving into automation might need record and replay to get started. However, as team matures, tools must support multiple scripting languages so that team members can work using the languages they are most comfortable with.And with Python becoming more widely used, it is important your automation tool supports its use. Additionally, if the team is more technically savvy, integration with open source solutions such as SoapUI may be required. Out of the box plugins for continuous integration systems such as Jenkins may come in handy while implementing continuous delivery.
4. Get a single view of your testing efforts: Ensure proper coverage exists for requirements at the GUI and API levels. Over time as you write more automated (GUI and API) as well as manual tests, maintaining traceability between the requirements and defects raised in correspondence can become tough. Test management systems can help ensure proper API and functional GUI test coverage exists for requirements.                                                                  
5. Focus on Reusability for DevOps: Improve reusability of scripts at both upstream and downstream side of application life-cycle management. For instance, for upstream, an ability to record repeatable accurate gestures and then replay these gestures in a consistent manner across multiple mobile devices can be helpful in overcoming device specific mobile testing, resulting from device fragmentation. Similarly, downstream, API tests created using automation pyramid approach can be used for monitoring by operations.
6. Recognize Automation has limitations:  While reducing manual testing in lieu of automation is crucial to test mobile applications in a shorter release cycle, it’s necessary to realize manual testing is still required. Exploratory and Usability testing are two areas where  manual testing is essential for mobile.

Over time as your testing matures, you can move testing to the right and harness data related to consumer usage in order to provide developers customer feedback. Additionally, initiatives like A/B as well as multivariate testing can be rolled out to improve monetization of the app.