Managing a Mobile Device Lab

Those of us working in QA, or as a developer in a larger company (for example 100 people or more), will likely have had access to a dedicated set of test devices. Quite often, you will have two or three devices on your desk at any given time. Quite often, a developer or tester colleague will either ask to take the device, or “borrow” it before you are done.

This might not be a large issue if your QA department is not large, or located in very close proximity to one another. Even so, the issue can cause office tension, as devices are returned uncharged, or you have to chase down a specific device for a particular test with a stringent deadline. Device charging can be a serious issue – where iOS devices tend to boot quickly when plugged in, some Android devices can take up to 20 minutes. If you have a limited time between meetings, and just want to quickly test your latest code on an older device or OS version, this wait can seem interminable.

We at testmunk know that a device cloud with a large set of real devices (both iOS and Android) can be a strong solution. Automated testing on such a cloud lab has even saved clients a great deal of money. However, even with this option open, we know that at times, manual testing is just plain necessary, and that organizations reaching a certain size know the importance of ready availability. If your organization is reaching this point, we hope to help, by providing some real world examples of existing solutions, as well as some insights from our own lab.

Challenges when Managing a Device Lab

Strong mobile testing, as we discussed in a previous article, needs to include real device testing. This means that you need a mobile device lab, and one that provides a somewhat representative selection of devices. This in turn means that if relying on internal resources entirely, labs need to be larger than ever before. Because larger labs often require dedicated bandwidth and storage space, not to mention power demands and other issues, there are some logistical issues that must be dealt with. You must have a plan in place not only to power and provide connectivity, but to keep track of the devices, and to maintain them (latest OS, updates, etc.). It is precisely these issues that are pushing many organizations towards cloud device labs.

Many organizations do not have formal device labs (ie a “large” set of devices, managed centrally, used exclusively for testing) right out of the gate, particularly startups. After a few devices disappear, after replacing cords, after chasing around the office for a particular device several times in a week, organizations come to the realization that things need to change…but what do they do about it? Well, this article hopes to shed some light on this, by presenting a few solutions seen “in the field.”

What Devices to Test On?

If developing for Android at all, it is safe to say that the bulk of your testing devices will be Android. With over 24000 distinct models on the market, to have any true coverage will require several devices, both tablets and phones.

The days when you can get by testing a handful of devices are gone. According to data from OpenSignal, it can be argued that to cover 50% of the market, you needed approximately 60 devices last year. For 20% of the market, you could get by with 12. While a large device lab is certainly helpful, there comes a point where logistically speaking you are doing more harm than good. It is more important that you select the right devices, rather than a large number of devices. Analytics is key to this. Determine the top devices that are using your apps, or driving traffic to your site, and focus there. This was a primary selection criteria of Etsy, in the formation of their mobile lab. We at testmunk consider an in-house lab of 30 to 40 devices fairly large, and that you can represent enough of the available market with this number (this total includes iOS devices), provided you select based on analytical data relevant to your organization. (This is particularly true if you also avail yourself of automated testing with a device cloud lab as well.)

Coverage of the Android user-base with 12 devices can certainly exceed 20%, if they are the right devices. The following devices are a good starting point:

Samsung Galaxy S3, S4 and S5 – Samsung Galaxy S3 and S4 remain some of the most popular smartphones. The S6 hasn’t sold as well as expected, making the S5 a more representative model.

Samsung Devices provide strongest testing coverage

Going by the top ten device list for Android, the majority of devices to test would be Samsung. However, sticking to one manufacturer is risky in and of itself. Hence, we recommend skipping to some of the up and coming models from other manufacturers.

Other strong testing options

LG G3 – Another high-end choice, with a 5.5-inch high-quality HD screen that makes it a suitable choice for mobile users that use multimedia on the go.
Google Nexus 5 – Stock OS, smart design, great battery life and high-end features make this a good bet.
Motorola Moto G – Low end price tag, but mid-end features make this a tester’s dream
Sony Xperia Z2, Z3 – high end capacity, the best battery life available, and loved by the multimedia savvy.
HTC One M8 – Another high-end option, but one that has come down in price a bit, making it more attractive for testing purposes.
LG Nexus 4 – when testing compatibility for older OS versions, this is a go-to model at a fairly reasonable price. Used devices can be found for $90, sometimes less. Just be aware ofbattery issues.

For tablets, it is a good idea to cover not only a couple of different screen sizes, but also high-end and low-end performance.

Google Nexus 9 and 10 – it is always a good idea to go with a device that comes with stock OS and high-end capacity.
Samsung Galaxy Tab 4 – An excellent 7 inch screen budget tablet, with a decent market share.
LG G Pad 7 – This budget 7 inch device is said to outperform its peers at a lower price. If building a lab on a budget, this is an excellent option.
LG G Pad 10 – same reasons as above, just a larger screen.

Testing for iOS is much more straightforward, with far fewer devices to test, but you still need to cover more than just the latest model and OS. According to Forbes, half of iOS users are running their iOS devices on an out-dated version. This means that we cannot afford to simply test the latest version of iOS, but also the last generation – ideally both in terms of devices and OS.

The above information assumes that your primary testing is Android and iOS – which is an assumption that is designed, like selecting key Android devices, to cover a majority of those reading this article. However, some organizations may need to cover more than this. It is important to remember that your organization’s needs and the purpose of the lab (open or private) are paramount to deciding the devices you need to support.

Security and Device Management

If running a private lab, security and device management are important considerations. Not only do you need to be able to find individual devices when needed, but you need to know they are safe when not in use. Sometimes this means physical security, rather than simple process-based tracking. It is for this reason that organizations such as Phunware (one of testmunk’s clients) keep their devices locked up when not in use.

Phunware Device Lab Locker

Not only are devices locked away, but they are kept organized via OS-based labels, and managed by a dedicated resource. Devices are signed in and out via an app designed specifically for internal use by Phunware, with access to the devices handled by a single point of contact (ie, one employee has access – if said employee is away, the key is handed to another in his absence). This not only ensures that devices are protected from being lost or stolen, but also that at any given point, the last known user can be pinpointed by asking the “keymaster,” so the device can be recovered by the next person to book it.

This methodical process and stringent control over testing devices is very effective in keeping tabs on numerous devices used for individual testing, in particular if the devices are often taken to an employee’s desk for testing, or even outside the office. Knowing who has the device is often half the battle.

However, if you are testing numerous devices at once, this solution can be problematic. If all devices are locked away between uses, not only might booking of a large number of devices be tedious (if done individually), but so will setup. If hooking up numerous devices in a central location, you will need to find cables, power strips, and USB hubs, set up stands, and connect all the devices. You will then need to tear this down, and organize the devices, cabling and stands into their secured location when done.

For this reason, some organizations are moving towards an open device lab design, whether or not it is truly open. Etsy’s shelf solution is a prime example. As you can see from the before picture, security was a primary concern. However, in creating their shelf solution, they moved away from this locked down approach in favor of a central location (visible to all), library cards for device sign-out , labelling, and an efficient cable management system.

Etsy Device Lab: Before and After

Etsy used library cards for signing out devices that leave the room, color coded by operating system. Labelled pockets were attached to the front of the shelf holding the device. On the front of the library card pockets they included:

device name and number
screen resolution
pixel density
operating system version

“Each device gets a cable drop to hold its cable in place so they don’t fall behind the shelves, a library card with its information, Adobe Edge Inspect installed, an asset tag, and a stand that fits it.”

Eventually, this library card sign-out process was replaced by a system using RFID tags for device management.

Phunware too, has revised their device management and check-in/checkout process for lab devices. To improve visibility, they created an app, listing all lab devices, allowing their users to check devices out, then back in simply by scanning a barcode on the device, and entering their username.

Phunware Inventory App Login

Not only did the app help to streamline the check-out/check-in process, but it also helped increase visibility, ensuring that each developer/QA member could see at a glance what was available (and who had what).

Phunware App: Barcode Scanner and Device List Screen

Power

When setting up your device lab, power will be of increasing concern, as the number of devices in the lab grow. Etsy discovered this first hand, particularly in light of their efforts to uphold the values of a certified B corporation (which includes a mandate to be environmentally conscious). Part of their ongoing effort is to track energy usage, and to set up power strips with timers to limit the number of hours per day the devices are plugged in and charging. (Considering Etsy’s solution and our own experiences at Testmunk, this timer solution might also help avoid situations such as our potentially explosive Nexus 4 batteries.)

In addition to limiting power consumption, logistics can be a concern as well. USB hubs must be able to support the large number of connections, and still provide adequate charge. Some of the hubs initially used would conk out after the devices were connected. A 32 port hub was obtained that provided 500 mA per port (16 A total), and this solved the charging problem.

Finding a place to put cables for charging devices was also a concern. The shelving solution chosen by Etsy above handled this nicely, by placing the power supplies and USB connections along the bottom shelf.

Managing test devices in use:

Because the selection of devices can be rather eclectic, encompassing tablet devices alongside Android and iOS phones (and possibly some netbooks and non-iOS/Android devices), it is important to find a system that allows visibility of the devices being tested. For example, Etsy’s shelf-system allowed the devices to be displayed upright, and had room for cable management. This is the ideal for an open lab design, allowing you to view multiple devices at once while testing.

If your device lab is closer in size, design, and scope to Phunware’s, and you do not run tests on your full range of devices often, but still test on several devices at once, it might be a good idea to look at some of the commercially available desktop stands to complement your lab solution. These stands allow you to stand 4-6 devices comfortably on your desk, with full visibility during testing.

Desktop Device Lab