This is the question that app developers, testers and product managers ask themselves as app quality becomes a major focus for not only development and testing teams, but for the entire organization. Getting app testing right can be difficult. The stakes are raised for delivering a high-quality end user experience across mobile and web apps. In short, the quality of digital experiences, across mobile and web, matter more than ever. So how can you know what the right mobile devices, operating systems and web browsers are for testing your app?
- Start with what you know. As organizations seek to understand their end-users’ app usage, they obtain unique web traffic analytics and other insights coming directly from their customers specific devices. Such information is extremely useful for making decisions around the next app capability, enhanced quality for specific devices in specific locations and more. With that in mind, I recommend you use this level of analytics for your mobile, web, responsive web strategies. It only makes sense, organizations should test for the conditions, devices, and platforms that its customers are using.
- Pair with industry data. There’s no better guide for your testing than real enterprise data. I’ve just published the third edition of the Digital Test Coverage Index to help you determine the right test matrix of mobile devices, operating systems and web browsers that represents your customers, markets and geographies. We aggregated data from our customers on over 4,000 devices and combined it with mobile market sales data and our own analysis, to create a list of the most relevant devices, OS versions and web browsers. It guides you in the right direction based on target geographies and market data so you can optimize your test coverage strategy.
For instance, did you know it only takes 32 mobile devices to achieve 80% test coverage for the mobile market? And did you know that these 32 devices vary widely between the US, Germany and Australia? Here are some key trends I found to share with you – more are available in the index:
Devices: Legacy devices are still popular in major markets. For example, Samsung Galaxy S3, HTC One M7, iPhone 5S, iPad 2 and iPad Air remain important for various mobile and web apps and need to be part of organizations’ testing labs.
Operating systems: Three months after its release, iOS9 has stabilized with a 70% market share. Of users who are not on iOS9, 22% are using iOS8. It’s critical to test on both of these major platforms.
Regional trend: Apple continues to dominate in the U.S., Canada and Australia markets with more than 40% mobile OS share in each region. But in Europe, Android has a clear advantage, with an aggregated 72% market share.
Pair data like this with your own customer data to optimize your organization’s test coverage metrics.
- Keep an eye out for the future. Test coverage planning is a continuous project. For instance, did you know the average shelf life for a device is less than 9 months and shrinking, fast? What was the top device a year ago is now totally irrelevant, and your test coverage plan must reflect these constant changes. In the Digital Test Coverage Index, we’ve included a calendar showing what mobile devices and OS versions are either new to the market or releasing in Q1 2016, and a 2015 Mobile Retrospective that looks back at the year’s major releases. This should help you not only react when new devices, OS versions or browser versions hit, but help you proactively prepare for the market updates.
App Testing
Optimizing a test coverage strategy that matches end user scenarios, keeps up with the constantly evolving mobile and digital markets, and is prepared for what is yet to come is definitely a challenging part of mobile testing. Developing a test coverage strategy that addresses customer usage data, market data and the future of the market, however, will set your app apart from the rest and lead to the highest-quality end user experience.
