Those of us working in QA, or as a developer in a larger company (for example 100 people or more), will likely have had access to a dedicated set of test devices. Quite often, you will have two or three devices on your desk at any given time. Quite often, a developer or tester colleague will either ask to take the device, or “borrow” it before you are done. This is where you have to start thinking about how to manage a Device Lab.
Someone borrowing a device might not be a large issue if your QA department is not large, or located in very close proximity to one another but these sort of incidents can cause office tension, as devices are returned uncharged, or you have to chase down a specific device for a particular test with a stringent deadline. Device charging can be a serious issue – where iOS devices tend to boot quickly when plugged in, some Android devices can take up to 20 minutes. If you have a limited time between meetings, and just want to quickly test your latest code on an older device or OS version, this wait can seem interminable.
We at testmunk know that a device cloud with a large set of real devices (both iOS and Android) can be a strong solution. Automated testing on such a cloud lab has even saved clients a great deal of money. However, even with this option open, we know that at times, manual testing is just plain necessary, and that organizations reaching a certain size know the importance of ready availability. If your organization is reaching this point, we hope to help, by providing some real world examples of existing solutions, as well as some insights from our own lab.
Challenges when Managing a Device lab
Strong mobile testing, as we discussed in a previous article, needs to include real device testing. This means that you need a mobile device lab, and one that provides a somewhat representative selection of devices. This in turn means that if relying on internal resources entirely, labs need to be larger than ever before. Because larger labs often require dedicated bandwidth and storage space, not to mention power demands and other issues, there are some logistical issues that must be dealt with. You must have a plan in place not only to power and provide connectivity, but to keep track of the devices, and to maintain them (latest OS, updates, etc.). It is precisely these issues that are pushing many organizations towards cloud device labs.
Many organizations do not have formal device labs (ie a “large” set of devices, managed centrally, used exclusively for testing) right out of the gate, particularly startups. After a few devices disappear, after replacing cords, after chasing around the office for a particular device several times in a week, organizations come to the realization that things need to change…but what do they do about it? Well, this article hopes to shed some light on this, by presenting a few solutions seen “in the field.”
What Devices to Test On?
If developing for Android at all, it is safe to say that the bulk of your testing devices will be Android. With over 24000 distinct models on the market, to have any true coverage will require several devices, both tablets and phones.
The days when you can get by testing a handful of devices are gone. According to data from OpenSignal, it can be argued that to cover 50% of the market, you needed approximately 60 devices last year. For 20% of the market, you could get by with 12. While a large device lab is certainly helpful, there comes a point where logistically speaking you are doing more harm than good. It is more important that you select the right devices, rather than a large number of devices. Analytics is key to this. Determine the top devices that are using your apps, or driving traffic to your site, and focus there. This was a primary selection criteria of Etsy, in the formation of their mobile lab. We at testmunk consider an in-house lab of 30 to 40 devices fairly large, and that you can represent enough of the available market with this number (this total includes iOS devices), provided you select based on analytical data relevant to your organization. (This is particularly true if you also avail yourself of automated testing with a device cloud lab as well.)
Coverage of the Android user-base with 12 devices can certainly exceed 20%, if they are the right devices. The following devices are a good starting point:
- Samsung Galaxy S3, S4 and S5 – Samsung Galaxy S3 and S4 remain some of the most popular smartphones. The S6 hasn’t sold as well as expected, making the S5 a more representative model.
Going by the top ten device list for Android, the majority of devices to test would be Samsung. However, sticking to one manufacturer is risky in and of itself. Hence, we recommend skipping to some of the up and coming models from other manufacturers.
- LG G3 – Another high-end choice, with a 5.5-inch high-quality HD screen that makes it a suitable choice for mobile users that use multimedia on the go.
- Google Nexus 5 – Stock OS, smart design, great battery life and high-end features make this a good bet.
- Motorola Moto G – Low end price tag, but mid-end features make this a tester’s dream
- Sony Xperia Z2, Z3 – high end capacity, the best battery life available, and loved by the multimedia savvy.
- HTC One M8 – Another high-end option, but one that has come down in price a bit, making it more attractive for testing purposes.
- LG Nexus 4 – when testing compatibility for older OS versions, this is a go-to model at a fairly reasonable price. Used devices can be found for $90, sometimes less. Just be aware of battery issues.
For tablets, it is a good idea to cover not only a couple of different screen sizes, but also high-end and low-end performance.
- Google Nexus 9 and 10 – it is always a good idea to go with a device that comes with stock OS and high-end capacity.
- Samsung Galaxy Tab 4 – An excellent 7 inch screen budget tablet, with a decent market share.
- LG G Pad 7 – This budget 7 inch device is said to outperform its peers at a lower price. If building a lab on a budget, this is an excellent option.
- LG G Pad 10 – same reasons as above, just a larger screen.
Testing for iOS is much more straightforward, with far fewer devices to test, but you still need to cover more than just the latest model and OS. According to Forbes, half of iOS users are running their iOS devices on an out-dated version. This means that we cannot afford to simply test the latest version of iOS, but also the last generation – ideally both in terms of devices and OS.
The above information assumes that your primary testing is Android and iOS – which is an assumption that is designed, like selecting key Android devices, to cover a majority of those reading this article. However, some organizations may need to cover more than this. It is important to remember that your organization’s needs and the purpose of the lab (open or private) are paramount to deciding the devices you need to support.
Security and Device Management
If running a private lab, security and device management are important considerations. Not only do you need to be able to find individual devices when needed, but you need to know they are safe when not in use. Sometimes this means physical security, rather than simple process-based tracking. It is for this reason that organizations such as Phunware (one of testmunk’s clients) keep their devices locked up when not in use.
Not only are devices locked away, but they are kept organized via OS-based labels, and managed by a dedicated resource. Devices are signed in and out via an app designed specifically for internal use by Phunware, with access to the devices handled by a single point of contact (ie, one employee has access – if said employee is away, the key is handed to another in his absence). This not only ensures that devices are protected from being lost or stolen, but also that at any given point, the last known user can be pinpointed by asking the “keymaster,” so the device can be recovered by the next person.
