Figuring out the how to define the right test coverage and lab configuration for a mobile project is one of the biggest frustrations for today’s digital teams.

Organizations struggle to assure high quality for the mobile apps but at the end of the day, how do they know that they have tested their apps on the right platforms?

Some of the key questions I hear in the digital space associated with test coverage are:

Which Devices and Browsers?

This question is the toughest since each organisation who develops a mobile app has unique target customers and geographies. In addition, in these target geographies there are varying trends and network conditions. If this is not enough, the mobile space is constantly changing making this entire test coverage maintenance a daily challenge.

While each organisation mobile app would look different, and will have different functionalities, there are some elements within the mobile landscape which can be predicted in advance and planned for.

Each year there are certain mobile market milestones which repeats themselves therefore making them accessible to test coverage owners (see below 2016 projected mobile market calendar).

In the image below of the 2016 calendar, there are clear market noises which repeats themselves.

• Post mobile world congress device launches (March-April timeframe)
• New Android OS rollout to major device vendors around 6 months’ post Nexus launches (April time frame) – Samsung, LG and HTC getting Android 6 as an example
• Major Apple OS and devices launch during September each year
• Major Android OS releases around October each year

With the above observations market does seem a bit more predictable. Teams who develop and test mobile apps can surely plan against the above milestones and assure lab refresh, OS upgrades and of course test plans updates.

Test plans and test scenarios are also being impacted by some of the above market changes. Since new OSes introduces new UI flows, and new functionalities it forces teams to adapt their testing to match these changes.

In addition, since iOS 9, apple has made a change in the way it introduces Beta versions to the market. All of its iOS 9.X Beta versions were made publically available to the end-users. Such market change also requires test plan change and lab change– Organizations cannot remain exposed to risks with a public version out there that is not being covered.

How Many Platforms Are Sufficient?

In addition to leveraging market calendars and market events, getting the right number of platforms which consist of Device/OS/Screen Parameters/Device Aging and locations remains a great challenge.

Organizations who are mature enough and have ongoing conversations and open channels to the business units, can usually get web traffic analysis or analytics (Google analytics/Adobe Analytics). This kind of insights are very powerful in determining the top most used devices and OS versions in a given geography.
Analytics however are not a silver bullet for defining a winning test lab coverage which is also future ready. Most common pitfalls around organizations who uses analytics are the following:

• Planning forward data isn’t available with such reports, only current insights
• Teams uses the analytics in a point in time and often fail to re-examine such reports, leaving their labs outdated.

To address the above Perfecto has developed a test coverage model which recommends a set of usage based sufficient device/OS buckets which gradually enhances test coverage as you move from one level to the next.

In the above model, when organizations merge either through analytics or through analytics and market reports like Perfecto’s quarterly Index report the data and assures that each group include devices from different types with different OS/Device characteristics – the lab will be well configured.

 

Here are some parameters which you would like to include in each mix:

• Device and OS popularity (market share)
• Screen sizes, resolution and other screen attributes such as pixel per inch (PPI)
• Device age (launch date)
• New and trending devices and platforms
• Operating system version update rate (e.g. reference devices like Android Nexus get a higher score)
  • This relates to reference devices (iPhone 6S/Ios9.X, Google Nexus/Android latest)
• Unique device properties important for testing purposes – chipset, CPU, memory
• Audience demographics

When Should I Refresh My Test Lab & Who Is in Charge?

This question can be answered in different ways, however the recommendation is to validate and adjust your test lab on a quarterly basis together with tools such as the above market calendar which help reflect market events throughout the year.

When organizations assign a coverage owner that sync’s the various teams such as Dev, Test, Ops etc. it makes the overall lab management strategy and maintenance easier.

In various organizations, the owner would be the head of TCOE (Testing center of excellence), in others it would be the Test Analyst, Strategy consultant or Head of QA.

No matter who is assigned, as long as he gets complete visibility into the market trends, product requirements, analytics and any other relevant technical details which can help in taking the right decisions.

The Digital Test Coverage Toolkit

Aiming to address some of the above challenges, we have developed a unique test coverage toolkit which consists of 3 useful assets:

• The quarterly test coverage index report
• The online Digital Test Coverage Optimizer
• The ultimate Test Coverage Guide

Each of the above has its unique purpose and value to the test coverage owner, and the teams which it serves.

The Index report is a static multi-geography report recommending based on usage, trends and testing considerations the top 32 devices (based on the above 3-layer model) the right device/OS mix. This report is a static one, hence cannot merge dynamically an organizational analytics report and other product requirements into a tailored set of device’s and OS mix. This report, can help teams who are starting with their mobile projects, or are already developing and testing a mobile app and have no insights and analytics about the target markets that they serve.

The online tool, is a much more advance and dynamic tool (responsive) which offers a simple wizard for any team who develops and test a mobile app, where they can enter specific requirements (geography, mobile OS, smartphone vs. tablet, unique devices which are must based on analytics or product needs) and get automatically a lab configuration recommendation with various risk levels.

 

To complete the entire challenge of setting a mobile test lab, the only thing organizations would need to do is to size their lab and match their team sizes, release velocity, and various test type allocation per devices.

The above toolkit will output a list of unique Device/OS recommendation per geography (e.g. top 25) where the various teams would need to allocate these unique models among themselves – in some cases, due to project needs there would need to be duplicates of these unique models (e.g. – 3 iPhone 6S/iOS9.x) because f multiple mobile apps, different usage of these devices and in the lab etc. an example to such math can look like the below image describes.

As seen in the image above, in a lab with 20 unique device models, where there are 2000 automated tests to be executed with a time constrain of 2 weeks – it may require based on each test execution length to size the lab with a total of actually 29 devices. As mentioned – each organization would have to define its lab size based on the unique device models tools like the above recommends.

Conclusion

As various teams evolve and grow their mobile business, test coverage will become more critical to their overall deployment success and revenue growth. Using tools such as market predictable calendars, periodic lab refresh, traffic analytics, and market related tools like the optimizer and Index can provide guidance and risk mitigation on an ongoing basis. Test coverage matching is not a one-time effort, but rather a daily challenge which requires an owner and the right tools that assures right decisions making.

 

About The Author

Eran Kinsbruner is Director and Mobile Technical Evangelist of Perfecto Mobile. He regularly blogs on mobile testing and you can find his blog here.