Category Archives: mobile testing

exploratory testing, functional testing, heuristics, idea generation, listicle, mnemonics, mobile application testing, mobile testing, product management, systems

I SLICED UP FUN Mobile Testing Infographic

Leave a comment

Twelve years on since I created and shared the mobile testing mnemonic I SLICED UP FUN, I see that people are still using it and finding it valuable. I still use it myself on projects, so I decided to create an infographic to make it more shareable.

I call this a mnemonic because it is a memory aid to help me with my work. A catchy phrase helps me remember everything I need to think about to be thorough when testing mobile apps. Sometimes these are called heuristics, or listicles. Whatever you want to call it, it’s a helpful thinking framework to help quickly generate lots of useful testing ideas.

I SLICED UP FUN is a testing framework for mobile apps, but I use it for more than testing. As a product manager I use it in a generative or creative way as well, not only to help evaluate an existing app design, but to create something new.

If you haven’t used a thinking framework like this before, it’s quite simple to use. Read each section, and determine which ones apply to your product. If a section doesn’t apply, skip it and move to the next. Once you have a list that is applicable to your work, use each item in the list to generate ideas for that category. Once you have a few relevant ideas under that section, move to the next. Then review what you have, and see if there are gaps. Whenever you’re able, include other people to help you generate more and better ideas.

Once you have generated enough ideas, put them into action, whether it is testing, design, or other work you need to do.

You can download the infographic here:
ISLICEDUPFUN mobile testing infographic

books, mobile, mobile design, mobile testing, product management, Tap Into Mobile Application Design, Tap Into Mobile Application Testing, usability, UX

New Book Published: Tap Into Mobile Application Design

2 Comments

After a long delay due to some health issues, I have finally finished the first version of Tap Into Mobile Application Design. This book is only available in PDF and epub electronic formats which should work on most e-reader apps on most mobile devices. I used Leanpub as the publishing platform, as with Tap Into Mobile Application Testing. I like this platform because it’s faster for me to get content out there, changes and updates are instant, and customers can download updated copies for free.

Tap Into Mobile Application Design book cover

This book is long and detailed, a result of me trying to capture most of my thoughts on designing software for mobile apps. When I started the book, I had planned for something much shorter, but as I worked through the content, it felt abbreviated and overly simplified. In Tap Into Mobile Application Testing, I made a couple of mistakes trying to overly simplify complex technical issues with regards to wireless technology. I updated some of the content to be more accurate, but I didn’t want to repeat that problem with this book, so I went deeper in some technical areas.

To help illustrate the challenges we have on these projects, I decided to use an example app project in the book. This helps to ground the content, moving from initial idea, to a full user experience design process and ending with user testing. The example project helps illustrate what a real world project can look like, but with the benefit of time I was able to capture many design project issues, rather than the few you encounter on a rapidly developed app. You get it all, including the positives and the negatives of the example app.

Furthermore, the context of app design changed as I was writing, and I felt I should capture some of those changes in the book as well. The legal landscape has changed, and there is a much better awareness of ethics and the long term effect of our designs on people. With the benefit of a side project to use as the example in the book, I was able to capture these issues as they happened on that project. The project had to adjust, and that is reflected in the book. Unexpected issues are common on mobile projects, and the example app shows how we adjusted. Initial attempts often fail due to oversight, legal rulings have an impact, and the timing of what you do on a product is crucial.

The book is also longer because it doesn’t just follow a happy path. There are lots of great books out there that fit that model. Instead, this book covers false starts, changes in direction and a completely reworked interaction design. That’s right, I cover how we almost went to market with one design, hit a snag, and completely redesigned the example app from the ground up. It’s difficult to capture the non-linearity of design in a book, and that results in some awkward flow and a couple of extra long chapters. I apologize for that, but I had hoped this would be an honest and detailed account of what can happen when you are creating an app.

I have also created a book bundle, combining both my “tap into” books called Tap Into Mobile Apps, where you can buy both the books for about the same cost as the full price of either book. The books are very different, but are complementary. In Tap Into Mobile Application Testing, the reader follows Tracy, a tester who is learning about mobile testing approaches. In Tap Into Mobile Application Design, the reader follows an example mobile app project called “Reporter” throughout the book. The design book is more heavy and dense content-wise, and that is reflected in the tone. The testing book is lighter in content and tone and an easier read. Both books cover technical issues to help inform your work. The combination of designing for or testing for people in social contexts, with a deep understanding of the technical underpinnings of the technology, within real world environments is my core differentiator. When I help teams develop that three pronged approach themselves, they build better software and have happier customers.

Both of these books represent my approach to working on mobile apps, which people can utilize as they see fit. These books aren’t for everyone. They are long and detailed and don’t provide easy answers. What they do provide is context and details that are important to understand on mobile projects, especially when you are having trouble. In spite of it representing a more difficult approach to your work, Tap Into Mobile Application Testing has been used by people all over the world, and influenced many mobile projects. It was highly praised when released and even now people contact me to tell me about how much it helped them. People still use it, they still talk about it at conferences and on projects, and several years on, find it relevant and helpful. I hope as many people find the design book to be as useful as they found the testing book.

app development, books, mobile, mobile testing, Tap Into Mobile Application Testing

Book Excerpt: The Seven Deadly Sins of Mobile Apps

5 Comments

This is an excerpt from my book: “Tap Into Mobile Application Testing“, from Chapter 10, pp. 431-432:

Here is a simple summary I created to help you think of different areas an app can fail. This is another mind hack you can use to quickly organize your thoughts, and analyze an application quickly.

  1. Lust: the app advertises that it can do a lot more than it actually can. It leaves you feeling unfulfilled and wanting more.
  2. Gluttony: it uses far too many resources. It uses up your device memory, downloads large image and other files, and slows down your device, eats up your data plan and kills your battery.
  3. Greed: the assumes you have a strong network connection, and would love to use as much of your network resources as possible. The app can’t handle poor or weak connections, or transitions between network types when you’re on the move.
  4. Sloth: the app performs very poorly. It is far too slow to respond to your interactions, and takes too long to do anything useful.
  5. Wrath: it doesn’t play well with other apps. It has special settings that override your defaults, and doesn’t allow other services to work well with it. It may even cause other apps to malfunction because of its behavior.
  6. Envy: The app is too close to other available apps out there that you would prefer to use instead. It’s a copycat. You just wasted precious time and resources to get an app that wishes it was something else, but it just can’t deliver on its features, and usefulness.
  7. Pride: The app is difficult to use, expecting users to adapt to its way instead of helping you be more effective.You are subordinate to it, and you have trouble using it, and end up feeling stupid. This is most likely due to designers and developers assuming they know better than the users, and ignoring valuable usability feedback and usability bugs.

Thanks to Shannon Hale and Jared Quinert for their help with this list.

This is now a talk!

Note: I have created a one hour talk on this topic if your company, user group or conference are interested. I provide tips for designers, developers and testers on how to create apps that are engaging and reliable for mobile users.

 

books, mobile, mobile application testing, mobile testing, Tap Into Mobile Application Testing, testing mobile apps

Tap Into Mobile Application Testing Book Now Available in Beta

9 Comments

Update: The book is no longer in beta. The final version was published in September 2013 and is available here: Tap Into Mobile Application Testing.

How did you spend your summer? I spent mine writing a book: Tap Into Mobile Application Testing. Now that smartphones and tablets are taking over the mobile space, there has been tremendous demand for ideas and training on how to test on these devices. I started out with my Testing Mobile Apps course, but that only scales so far. Many of you have asked me about writing a book on the topic, so I did. It’s not completely finished yet, but I managed to create most of the content I had hoped to share.

I have two audiences with this book:

  1. Mobilists without much testing experience
  2. Professional testers without much mobile experience

I wrote the book with both of these audiences in mind, and I was also pleased to learn that even deeply experienced mobile testers told me they learned a lot of new ideas and approaches for their own testing projects after reviewing the book.

It is only available in electronic formats right now: PDF, epub and mobi, which should work on most e-reader apps on most mobile devices. I chose mobile first because that’s the medium many of my readers will choose first. It’s also faster for me to get content out there, and many traditional publishers print dead tree copies first, then look at electronic versions. I am using Leanpub, which allows me to publish free updates to every reader who has purchased a copy, even the finalized version that I am targeting for a release later in the year. Once I have a finalized version, I will provide print copies using a print-on-demand service.

The Beta version is a bit rough around the edges. I still have the following on my to-do list:

  • Copy editing and attribution clean up
  • Creating figures and images
  • Section rewrites and new material where needed

If you notice something wrong, misattributed, or missing feel free to contact me and I can fix it before the final version is complete in December.

If you are curious about mobile apps testing on modern devices, I hope you find it useful.

mobile, mobile testing, testing

Creating a Test Lab for Mobile Apps Testing – Part 3

5 Comments

In parts one and two of this series, I introduced some ideas on building a physical test lab for testing mobile apps. In part 3, I’m going to talk about devices. This is all based off of my own experience.

Purchasing Devices

This is the the most difficult aspect of setting up a test lab. I’m not going to make any hard and fast recommendations, because by tomorrow the market will probably change. You will need to research and adjust depending on what your current needs are.

Emulators are fine for basic functional testing, but you’re going to need to do testing on the real thing, whether your focus is on automated or manual testing. The devices have sensor and touch interaction, as well as network and other physical aspects that need to have real-world testing.

You will need to consider the following:

  • both smartphones and tablets
  • different hardware and operating systems or versions, per framework
  • devices with different screen sizes
  • different wireless broadband carriers
  • cables and other accessories for charging and syncing (never hurts to have extras)

Each of these areas will yield different results when testing. For example, I frequently test with a smartphone and a tablet at the same time, and it’s amazing how the app can behave or appear differently on one or the other.

Also remember that a platform for testing is a combination of a hardware model, operating system version, and if you are using network connections with your app, wifi, wireless broadband, different carriers and any special software carriers install.

Purchasing devices is a balancing act, because the market is so dynamic. You will need to get a reasonable number of devices for testing the real thing, but you can’t buy them all. I look at the following to help guide what I should buy:

  • What platforms are our customers using?
  • What are the most popular smartphones and tablets on the market?
  • What are the most popular mobile browsers?
  • What platforms are known to be problematic? (aka. problem child devices)
  • When is the newest thing coming out? Do we need it for testing?

To figure out what you need to do using this information can be challenging. Here is one example. If you don’t know what your target customer base are using, you can start with information you already have. If you already have a web presence, or a web app, find out your web traffic stats. Are people using mobile browsers? What brand is the most popular? Can you get a visual breakdown? This information will usually breakdown according to brand, such as Android or Apple iOS, but it won’t give you specifics, such as an iPad, with 1st gen. hardware running iOS version 5.1.1. But, if you know that Apple iOS devices are the most popular, you can find out information about what the most popular devices are in the market, and assume that your customer breakdown is likely very similar. Look at a chart of popular versions, and try to replicate that in your lab – ie. stock up on the more popular devices, with less of the less popular devices. Not perfect, but it’s a good heuristic.

If you are looking at mass market applications, there is some information out there to help. For example, this chart this chart by Net Marketshare would indicate we should focus a good deal of our efforts with iOS device testing. Here is some more information to help deal with proportions of devices to purchase: comScore Reports May 2012 U.S. Mobile Subscriber Market Share. Akamai has an interesting report on mobile web browser breakdowns here: Akamai IO.

Notice that these reports are already out of date. This information is difficult to get, and I doubt anyone really knows exact numbers. It’s important to research and look at a lot of data to help inform you. When you spot trends, that should help you with proportions.

Some people think that Apple devices are much easier because there are fewer hardware devices, and they are all made by one company. However, even with Apple, you can end up with a lot of platform combinations. Take the iPhone for instance. Let’s say we are going to have 3 devices: an iPhone 3GS, an iPhone 4, and an iPhone 4S. Since Apple encourages people to upgrade their OS, we have decided to stick with iOS 5, the latest version. At the time of writing, there are 4 iOS versions out there: iOS 5, iOS 5.0.1, iOS 5.1.1. Also assume our app requires an internet connection. If we choose 3G, the most popular wireless broadband networking type, and wifi, then we have 24 platform combinations. What if we now need to test on 4G now? Add 12 to the mix for a total of 36. That can be a lot of regression testing. If you want to optimize, you may decide to elminate everything but the latest version, 5.1.1. If you do that, try to find something that gives you an indication of upgrade proportions, like this information: iOS 5.1.1 Upgrade Stats. You may not want to upgrade all of your devices at once, but keep some back to mimic what is happening in the real world.

Here is a recent plan for an iOS test lab:
Apple iOS (all on the latest, 5.1.1):

  • (1) iPhone 3GS
  • (2) iPhone 4
  • (3) iPhone 4S
  • (1) iPod touch (they aren’t as powerful, and are great for finding problems you don’t see on other devices)
  • (1) iPad first gen
  • (2) iPad second gen
  • (3) iPad third gen
  • (at least 1) whatever the newest iOS device is when it comes out(these are always hugely popular)

Android, Windows Phone and Blackberry are more difficult, not just because of the different handset manufacturers, but there is more of a proportion of devices across versions of their OS, or OS family. For example, Android has a nice graphic of their Android OS Distribution. When setting up a lab for Android devices, I would try to replicate that pie chart in my own device distribution. For older platform families like Windows and Blackberry, you will have even more variations to consider, but temper that with how popular they are in your target market.

For mass market applications, or web testing, Brad Frost has a fabulous blog post test on real mobile devices without breaking the bank, and this blog post from the Mobile in Higher Ed blog is also very useful.

It sounds expensive, and it can be, but compared to test workstations and servers, it isn’t that bad. You can get multiple test devices for the price of many test servers when you take hardware, OS and required software into account. You will have to research using your favorite search engine, and you’ll need to stay on top of new developments, because you will regularly need to acquire new devices.

Looking for ideas on how to test mobile devices? Check out my I SLICED UP FUN mnemonic to help generate ideas.

mobile, mobile testing, testing

Creating a Test Lab for Mobile Apps Testing – Part 2

6 Comments

In Part 1 of this series, I introduced some ideas on building a physical test lab for testing mobile apps. In part 2, I expand on those ideas.

Storage

You will want a clean, dry, secure storage area for devices to be kept in when not in use. I’ve used a drawer that locks with toolbox liner at the bottom to protect the cases and keep them from moving around. Here is an example: toolbox liner. The devices are small and disappear easily, so secure storage is a must.

Set up a sign-out sheet with each device noted by OS, version and serial number. Also create tags for each power/sync cable for each device so they can be kept track of if they leave the test lab. I used a label maker for this. There is nothing worse than losing your hard-earned test devices to other groups, or not having devices around when you need them. Cables disappear the most, so you need to make ownership for your lab visible. Think of the washroom key for a service station on the side of a busy road – the small key is often connected to something larger to make it more visible to help prevent it from getting lost. I always chuckle when I see a key on a chain with a hubcap attached, and I try to make my lab cables visible in some easily identifiable way.

Power Station

You will need access to power to charge the devices so they are available for testing when you are. It’s incredibly frustrating when you are getting ready to test on a device and find out that the battery is dead. If you can’t get power into your device storage area, set up a charging station in the lab.

Set aside counter or desk space near a power source, and add the following:

  • at least one power bar to plug in multiple devices
  • several power cables for the devices
  • toolbox liner or something similar on the surface to keep them from moving around

Lab PCs

You will need computers to manage the devices for testing, troubleshooting and other tasks. Consider at least one machine for each platform family. Here are four popular platforms. Your team will have their own mix of devices that they are developing for. Get the most powerful workstation you can afford for each platform; mobile app development tools can be large and complex.

iOS (Apple)

You will need at least one Mac for emulators and device management, and pulling screenshots and other information from the devices. Install XCode for emulator use, Also consider installing the Network Link Conditioner to help simulate different network connection conditions with an emulator. Install the iPhone Configuration Utility for getting stack traces, managing licenses, etc.

Android

You will need at least one machine with the Android SDK installed for managing Android devices and emulator use. Many developers use this within the Eclipse IDE, so talk to them about the correct combination of tools for your shop. The operating system doesn’t matter for Android development.

Windows Phone

If you are supporting Windows Phones, you will need a Windows machine with at least Windows 7 and the Windows Phone SDK and related tools.

BlackBerry

The new BlackBerry SDK supports several languages and implementations, so you’ll need to find out which version to install, and what other libraries need to be installed. I don’t think OS matters, but I have seen installation instructions recommending at least Windows XP or Vista.

Wireless Broadband

Data plans or other contracts for monthly wireless broadband access are important to test. I would get a reasonable plan for each device, from at least two local telecom providers. Try a larger provider, and a smaller provider (like a discount provider) to get more variation in wireless broadband technology. There is a huge diversity in what different carriers use to deliver this to the end user, so try to get some variation

Other

You will need internal test email addresses, and phone numbers for each smartphone for testing. Also get at least one developer license for each platform your team supports for your test lab. You will need test ids with the App Store on Apple for making purchases, or installing apps from the app store. You will likely need to do this with other stores as well such as Google Play and others for each device type you have.

Looking for ideas on how to test mobile devices? Check out my I SLICED UP FUN mnemonic to help generate ideas.

mobile, mobile testing, testing

Creating a Test Lab for Mobile Apps Testing – Part 1

4 Comments

Some of you have asked me to expand on the physical aspects of mobile app development that I described in my article: Mobile Challenges for Project Management. If we have a testing burden when we develop mobile apps, what sort of considerations should we take into account when we are setting up test labs, when using real devices? Here are some pointers from my own experience.

This is part 1 of a 3 part series.  Feel free to add comments for anything I’ve missed.

The Room

You’ll need space where you can set up PCs for managing devices and using productivity tools, a place to store devices when they aren’t in use, space for charging devices, and plenty of space for people to move around with the devices while testing. (The mobility side of mobile devices needs to be tested.)

Network

Wifi

I prefer to have at least two dedicated test wi-fi beacons to test network connections and transitions between wifi, and wifi to wireless broadband. I prefer small, cheaper devices that have weaker strength so it is easier to set up test conditions for apps that require network connections. I like to have these for test purposes only, and it is better that they have weaker signals (so testers can walk away and get weaker wifi for testing), and can be turned off or unplugged for testing outages, etc. Also since they are smaller and weaker, you can set up two in a lab and transition between the two by moving in the lab.

Simulate Dead Spots

If you don’t have an office dead spot (no wireless broadband or wifi connectivity), or one that is convenient for testing, you may need to set something up yourself. If you have an elevator or other known dead spot near your office, that works well. In some buildings you don’t.

Consider building a faraday cage to test for dead spots. Try doing network/web searches or submissions within a deadspot, or transitioning into or out of one. You can use an old microwave, but make sure you cut off the power cord so it can’t ever be powered on accidentally. They are simple to use, try a web search or submission, or any action that requires a network connection, and put the device into the microwave. It will instantly be in a dead spot.

You can also buy them from providers, or make one out of wood and wire mesh:

  • build a box out of 2×4’s, hinges and wire mesh
  • create a door in the front
  • wrap the box with wire or brass mesh
  • attach a ground wire to the box

There are different instructions online on how to make them.

To test outages, or performance issues, and to monitor and measure performance, page sizes, security, and other things,  you may want your own network for testing. It’s a good idea to have a separate test network, so you can quickly create scenarios, outages, or easily monitor mobile app traffic, whether it is a physically separate network, or a virtual network without interrupting other work.  Your administrators can figure out the implementation.

Stay tuned for parts 2 and 3.

Looking for ideas on how to test mobile devices? Check out my I SLICED UP FUN mnemonic to help generate ideas.

mobile, mobile testing, mobile training, tutorials

Public Course: Testing Mobile Applications, in London, UK, May 10-11

Leave a comment

Working with Electromind, I’m offering a 2 day public course on Testing Mobile Applications in London, UK, May 10-11, 2012.

Register here: Testing Mobile Apps Registration

Testing Mobile Applications Workshop Description

2 day tutorial by Jonathan Kohl.

Applications for smartphones and tablets are incredibly popular. Teams are facing pressure to deliver testing quickly and successfully on mobile devices. When faced with a mobile testing project, many testers find it tempting to apply the same methods and techniques used for desktop applications. Although some of these concepts transfer directly, testing mobile applications presents its own special challenges. If you follow the same practices and techniques as you have before, you will miss critical bugs. Learn how to effectively test mobile applications, and how to add more structure and organization to generate effective test ideas to exploit the capabilities and weaknesses of mobile devices. Hear first-hand experiences with testing mobile applications and discuss how to address various challenges.

Note: Participants must bring their own mobile device for course exercises. This is a hands-on course.

mobile, mobile testing, mobile training, speaking, tutorials

Public Workshop: Testing Mobile Applications, in Vancouver, May 25

Leave a comment

As part of the Agile Vancouver Quality in Agile conference, I’m offering a 1 day public workshop on Testing Mobile Applications in Vancouver, BC, Canada, May 25, 2012.

I will also be speaking about testing and value during the conference on May 26.

Register here: Quality in Agile Conference Registration

Testing Mobile Applications Workshop Description

1 day tutorial by Jonathan Kohl.

Applications for smartphones and tablets are incredibly popular. Teams are facing pressure to deliver testing quickly and successfully on mobile devices. When faced with a mobile testing project, many testers find it tempting to apply the same methods and techniques used for desktop applications. Although some of these concepts transfer directly, testing mobile applications presents its own special challenges. If you follow the same practices and techniques as you have before, you will miss critical bugs. Learn how to effectively test mobile applications, and how to add more structure and organization to generate effective test ideas to exploit the capabilities and weaknesses of mobile devices. Hear first-hand experiences with testing mobile applications and discuss how to address various challenges.

Note: Participants must bring their own mobile device for course exercises. This is a hands-on course.