In parts one and two of this series, I introduced some ideas on building a physical test lab for testing mobile apps. In part 3, I’m going to talk about devices. This is all based off of my own experience.

Purchasing Devices

This is the the most difficult aspect of setting up a test lab. I’m not going to make any hard and fast recommendations, because by tomorrow the market will probably change. You will need to research and adjust depending on what your current needs are.

Emulators are fine for basic functional testing, but you’re going to need to do testing on the real thing, whether your focus is on automated or manual testing. The devices have sensor and touch interaction, as well as network and other physical aspects that need to have real-world testing.

You will need to consider the following:

• both smartphones and tablets
• different hardware and operating systems or versions, per framework
• devices with different screen sizes
• different wireless broadband carriers
• cables and other accessories for charging and syncing (never hurts to have extras)

Each of these areas will yield different results when testing. For example, I frequently test with a smartphone and a tablet at the same time, and it’s amazing how the app can behave or appear differently on one or the other.

Also remember that a platform for testing is a combination of a hardware model, operating system version, and if you are using network connections with your app, wifi, wireless broadband, different carriers and any special software carriers install.

Purchasing devices is a balancing act, because the market is so dynamic. You will need to get a reasonable number of devices for testing the real thing, but you can’t buy them all. I look at the following to help guide what I should buy:

• What platforms are our customers using?
• What are the most popular smartphones and tablets on the market?
• What are the most popular mobile browsers?
• What platforms are known to be problematic? (aka. problem child devices)
• When is the newest thing coming out? Do we need it for testing?

To figure out what you need to do using this information can be challenging. Here is one example. If you don’t know what your target customer base are using, you can start with information you already have. If you already have a web presence, or a web app, find out your web traffic stats. Are people using mobile browsers? What brand is the most popular? Can you get a visual breakdown? This information will usually breakdown according to brand, such as Android or Apple iOS, but it won’t give you specifics, such as an iPad, with 1st gen. hardware running iOS version 5.1.1. But, if you know that Apple iOS devices are the most popular, you can find out information about what the most popular devices are in the market, and assume that your customer breakdown is likely very similar. Look at a chart of popular versions, and try to replicate that in your lab – ie. stock up on the more popular devices, with less of the less popular devices. Not perfect, but it’s a good heuristic.

If you are looking at mass market applications, there is some information out there to help. For example, this chart this chart by Net Marketshare would indicate we should focus a good deal of our efforts with iOS device testing. Here is some more information to help deal with proportions of devices to purchase: comScore Reports May 2012 U.S. Mobile Subscriber Market Share. Akamai has an interesting report on mobile web browser breakdowns here: Akamai IO.

Notice that these reports are already out of date. This information is difficult to get, and I doubt anyone really knows exact numbers. It’s important to research and look at a lot of data to help inform you. When you spot trends, that should help you with proportions.

Some people think that Apple devices are much easier because there are fewer hardware devices, and they are all made by one company. However, even with Apple, you can end up with a lot of platform combinations. Take the iPhone for instance. Let’s say we are going to have 3 devices: an iPhone 3GS, an iPhone 4, and an iPhone 4S. Since Apple encourages people to upgrade their OS, we have decided to stick with iOS 5, the latest version. At the time of writing, there are 4 iOS versions out there: iOS 5, iOS 5.0.1, iOS 5.1.1. Also assume our app requires an internet connection. If we choose 3G, the most popular wireless broadband networking type, and wifi, then we have 24 platform combinations. What if we now need to test on 4G now? Add 12 to the mix for a total of 36. That can be a lot of regression testing. If you want to optimize, you may decide to elminate everything but the latest version, 5.1.1. If you do that, try to find something that gives you an indication of upgrade proportions, like this information: iOS 5.1.1 Upgrade Stats. You may not want to upgrade all of your devices at once, but keep some back to mimic what is happening in the real world.

Here is a recent plan for an iOS test lab:
Apple iOS (all on the latest, 5.1.1):

• (1) iPhone 3GS
• (2) iPhone 4
• (3) iPhone 4S
• (1) iPod touch (they aren’t as powerful, and are great for finding problems you don’t see on other devices)
• (1) iPad first gen
• (2) iPad second gen
• (3) iPad third gen
• (at least 1) whatever the newest iOS device is when it comes out(these are always hugely popular)

Android, Windows Phone and Blackberry are more difficult, not just because of the different handset manufacturers, but there is more of a proportion of devices across versions of their OS, or OS family. For example, Android has a nice graphic of their Android OS Distribution. When setting up a lab for Android devices, I would try to replicate that pie chart in my own device distribution. For older platform families like Windows and Blackberry, you will have even more variations to consider, but temper that with how popular they are in your target market.

For mass market applications, or web testing, Brad Frost has a fabulous blog post test on real mobile devices without breaking the bank, and this blog post from the Mobile in Higher Ed blog is also very useful.

It sounds expensive, and it can be, but compared to test workstations and servers, it isn’t that bad. You can get multiple test devices for the price of many test servers when you take hardware, OS and required software into account. You will have to research using your favorite search engine, and you’ll need to stay on top of new developments, because you will regularly need to acquire new devices.

Looking for ideas on how to test mobile devices? Check out my I SLICED UP FUN mnemonic to help generate ideas.