All posts by jonathankohl

gamification, software testing, testing

Applying Gamification to Software Testing

Leave a comment

I wrote an article for Better Software magazine this month called “Software Testing is a Game”, available here in PDF format. I wrote about using gamification as an approach to analyze and help make software testing more engaging. I encouraged readers to apply some ideas from gamification to their own testing efforts. Now, why would I do a thing like that? And what do I mean by using game mechanics when we are testing? Games are all well and good, and I may enjoy them, but we are talking about serious work here, why would we make it look like a game?

Let me give you a bit of background information.

I was working with my friends Monroe Thomas and David McFadzean on product strategy when they started bringing up my gamification design ideas. I use gamification in mobile app design to help them be more engaging for users. That doesn’t mean that I make an app look like a game, it means I use ideas from games to help make the app more interesting and easier to use. However, we weren’t talking about mobile apps, so I was a bit surprised. They pointed out that the same concepts that make gamification in mobile apps apply to other apps, after all, David and I even wrote an article about using gaming when creating software processes. Why couldn’t I use those ideas in a product strategy meeting for something else?

Good point.

In fact, they even urged me to look at some of my other prior app designs, they felt I would find gamification-style aspects in those as well, because I always worry about making apps more engaging. Once I started thinking about the implications of what they were saying, an entire new world of possibility opened up. I felt like they had just kicked open a big door of perception for me.

But wait a minute. What is this business about games? Well, the thing with gamification is that when I use those tools correctly in an app, you don’t know it is there. I don’t put childish badges and leaderboards in a productivity app and then say: “Look! gamification at work!” for example. Andrzej Marczewski describes gamification mechanics in terms we can relate to in his blog Game Mechanics in Gamification as: Desired Behavior, Motivation and Supporters.

Andrzej uses a game format to illustrate his point, but it should be obvious that these three themes are not limited to games. Where game designers shine, and where policy wonks and enterprise or productivity designers tend to fail is in the structure around desired behavior. Too often, we just expect people to excel in a work place environment with little support. Games on the other hand tickle our emotions, they captivate us, and they encourage us to work hard at solving problems and reaching goals.

Framing something like software testing in terms of gaming, and borrowing some of their ideas and mechanics, applying them and experimenting can be incredibly worthwhile. After all, as I state in the article, it is difficult to get people involved in software testing, and as technology becomes more pervasive and more enmeshed in our every day lives, it has more potential to do harm. We need new people and new ideas and new approaches, and I want to figure out how to make it more engaging for people. Why can’t effective testing be fun?

It can.

If you work on a team with me, you will notice that there is a lot of laughter, a lot of collaboration, a lot of discovery and learning. And everyone tests from time to time. Sometimes, it can be difficult to get the coders to code, the designers to design and the managers to manage, because everyone wants to test. Why is that? Well, gamification can help provide a structure to analyze what we do and learn why some things are fun and help us work hard, while others cause us to avoid them.

Speaking of analyzing something from a gamification perspective, remember in the Better Software article how I described several aspects from gaming and asked you to apply it to your testing work? Prior to writing the article, I did exactly that with a product I designed called Session Tester. Aaron West and I developed a tool to help testers capture information while using an approach called Session-Based Testing. We had high hopes for the project, but after several setbacks, it’s now dormant. However, a back of the napkin analysis of the tool using a gamification approach was incredibly useful. This is what we came up with, using game concepts from Michael Wilson’s “Gamification: You’re Doing it Wrong!” presentation:

  1. Guidelines and Behaviors:
    Context and rules around the tool was hit and miss. The tool enforces the basic form of session-based testing which helps people learn how to approach testing from this perspective. People are required to fill in the minimum information to create a session sheet. There are strategy ideas readily at hand, and the elements are easily added by using tags. The tool was helpful to teach beginners on the basic form of SBT, but we didn’t enforce the original SBTM rules as set out by James and Jon Bach. This hurt the tool’s effectiveness. While we value the ability for people to modify and adapt, we should have started with the known rules and then provided the ability to adapt, rather than design it from an adapted view. This caused confusion and controversy.
  2. Strategies and Tasks:
    Elisabeth Hendrickson’s ET Heuristics Cheatsheet is provided in the tool to help people think about strategy, and there are oblique strategies to help create test ideas using the Prime Me! button. There could be more resources added to help with strategy, and in fact a lot of the strategy work can be done outside of the tool. We could have done more feature-wise to help with strategy. Tasks can be pre-planned outside of the tool, or done on the fly and recorded with the @tasks tag, which is saved in session sheets. We could also have done more to support tasks.
  3. Risks and Rewards:
    There is a risk that you don’t have a productive session, or your session sheet is woefully inadequate. The timer was a good motivator since you run the risk of running out of time, so there was a bit of a game there with trying to beat the clock and have a focused, productive session. I designed that to be analogous to the “red bar green bar game” used in Test Driven Development tools. There is a reward inherent in getting your mission completed and having a good session sheet you can be proud to share, but it is completely intrinsic. You are also rewarded a bit with the Prime Me! button to help you get a new idea, or break a creativity log jam. We could have done a lot more to help people plan and manage risks, and add features to reward testers for using a good assortment of tags, or a peer-reference or reward system for great testing. The full bar showing once time has run out helps tickle an intrinsic reward of completion. As a tester, I did all I could in that session, and now I can move on to other things.
  4. Skill and Chance Events:
    Skilled testers often like to record what they discover, to have the freedom to investigate areas of high value, and take pride in having a varied approach to their testing. However, there is no extrinsic reward for completion of session sheets. Sheets with more tags having a higher score might have been a good option to add,to help people learn how to improve what they record. Outside of discovering bugs, chance events are brought in by the Prime Me! button. Like rolling a dice, people can click the button until an oblique strategy jiggles their brain in a different direction. The Prime Me! button is the most popular feature of the tool and is still demonstrated at testing conferences by people like Jon Bach. People find it fun and useful.
  5. Cheating and Compliance:
    Cheating: Anyone who uses a test case management system will have a high degree of cheating. People just get tired of the regression tests they run over and over and start clicking pass or fail to show progress. They are very easy to cheat, but a session-based approach is much more difficult to cheat, because you have to show a description of a testing session. However, there is nothing to prevent people from saving an empty session sheet. I have seen this happen on over worked teams, and it wasn’t discovered for weeks. We could possibly have looked at flagging incomplete or blank session sheets in the system so there is visibility on them /prior/ to an audit, or encourage people to do something about it within the tool. Compliance was a big miss because we altered the original SBTM rules, which caused a lot of controversy and prevented more widespread adoption. We should have enforced the original rules by supporting the Bach SBTM format first, then added the ability to adapt it instead of approaching it from the other direction.

It’s interesting to note that the aspects that made this tool popular and engaging can also be viewed in terms of gaming mechanics. A couple of them were there by design, but the others were just there because I was trying to make the app more engaging. However, if we had used this gamification structure during design of the tool, we would have had different results, and arguably a better tool, because it provides a more thorough structure. Areas of fun such as the Prime Me! button, and trying to automate some of the processes of SBTM helped make the experience more enjoyable for our users.

However, if you didn’t look at the tool from a gaming perspective, you wouldn’t notice that there are game mechanics at play within it. This is an example of using a gamification approach that goes beyond superficial leaderboards and rewards, and I encourage you to try it not only with your testing tools, but your processes and practices in testing. Use it as a system to analyze: What is working well? Where are you lacking? It’s a useful, systematic approach.

That analysis doesn’t look like a childish game does it? Bottom line: if you aren’t a gamer, you probably won’t notice the gaming aspects I bring into testing process and tools. If you are a gamer, you’ll notice the parallels right away, and will hopefully appreciate them. For both groups, hopefully gamification will be one tool we can use to help make testing more engaging and fun.

gamification, software testing, testing

Software Testing Training and Gaming

3 Comments

If you spend time at conferences, or hire a well-known testing consultant to provide some training for your company, it’s likely that one or more of them have used game mechanics as teaching tools. In fact, they probably used them on you. You may not be aware that they did, but they used gaming mechanics to help you learn something important.

James Bach is famous for using magic tricks and puzzle solving as teaching tools. When I spent time with James learning about how to be a more effective trainer, he told me that magic tricks are great teaching tools because we all love to be fooled. When we are fooled by something, we are entertained, and our mind is primed for learning about what we missed during the trick. That is an ideal state for the introduction to new ideas. If you spend any time with James or any of his adherents at a conference or peer workshop, you will likely be inundated with puzzles to solve. There is always a testing lesson to be learned at the end, and it is a novel way of helping people learn through solving a tangible problem. If you love to solve puzzles and learning about testing, you’ll enjoy these experiences.

Dorothy Graham has a board game that she developed for testing tutorials. It’s a traditional style game that she created as a training aid, and Dot loves to deliver this course. The tutorial attendees have a lot of fun, and they learn some important lessons, but Dot admits she may even have more fun than they do. Dot loves training, and the game takes the entertainment value of learning up a few notches. I’ve taught next door to Dot and heard attendees as they play the game and learn with her, and I’ve seen their smiling faces during breaks and after the course. There is something inherently positive about using a real, physical game, designed for a specific purpose (and fun) in this way.

Fiona Charles and Michael Bolton also created a board game for a software development game workshop they facilitated in 2006. Fiona says:  “Our experience with the game highlighted the power of games and simulations in teaching: their ability to teach the participants (and the teachers) more than was consciously intended.”

Ben Simo uses a variation on a board game. I’m not going to give it away, since it’s highly effective, but he used it on me when I was moving from a dabbler in performance and load testing to working on some serious projects. Ben is an experienced and talented performance tester, and he has taught a lot of people how to do the job well. Ben spent hours with me using pieces from a board game, and posing problems for me and having me work on solving them. It was highly interactive, was chock full of performance testing analysis lessons, and we enjoyed working together on it. He would set up the scenario, enhanced by the board game, and I would work on approaches to solve it. I had about 15 pages of notes from this game play activity to take back and apply to my work on Monday. After playing this training game with Ben, I had much more confidence and I was able to spot far more performance anomaly patterns than I had prior to working with him. (We worked through this in a hotel lounge, and we got a lot of weird looks. We didn’t care, we were having fun! Besides, channeling Ralph Wiggum: I was “learnding”!)

James Lyndsay developed a fascinating course on exploratory testing, and with it, simple “black box test machines” that he developed in Flash to aid in experiential learning. These machines had no text on them, and they are difficult to start using, because there are no outward signs of what they are for. This is done on purpose, and each machine helps each class participant experience the lesson through their own exploration and discovery. This is one of my favorite game-like experiences in a testing training course. The machine exercises remind me of a puzzle adventure game. One of my favorites of this type of game is Myst. You have to explore and go off of your observations and clues to figure out what to do, and the possibilities for application and experience are wide open. James managed to create 4 incredibly simple programs that can replicate this sort of game experience during training. Simply brilliant.

Those of you who follow Jerry Weinberg, or the many consultants who have been influenced by him have likely worked through simulations during a workshop or tutorial. Much like an RPG (role playing game), attendees are organized around different goals, roles, activities and tasks to create an improvised simulation of a real-life problem. This involves drawing on improvisation, your “pretending” skills and applying your problem solving techniques in a different context than a work context. Many people report having very positive experiences and “aha!” moments when learning from these sorts of activities.

Another theme in Jerry’s people working is physical activity. Jerry gets people to move around, and he can influence the mood of the room by adding in physical activity to a workshop. In the book, the Gift of Time, Fiona Charles shares a poignant story about Jerry using a movement activity to calm down a room full of people during a workshop when they first learned about the events of September 11. Michael Bolton has told me several stories of how Jerry changes the learning dynamic by getting people to move and work in different parts of the room, or grouping people and having them move and work with others in creative combinations. Movement is a huge part of many games, especially sports and outdoor activities, and it gets different parts of our brain working. If you couple movement with learning concepts, it brings together more of your senses to help with concept retention. It is also associated with good health, a sense of well being and fun.

(Speaking of experiential learning, pretty much everyone I have mentioned here, including me, (and a lot more trainers you have heard of) have been influenced either directly or indirectly by Jerry Weinberg’s work on experiential learning. He even has a series books on the topic on Leanpub. The first one: Experiential Learning: Beginning , the second: Experiential Learning: Inventing and the third: Experiential Learning: Simulation.)

There are other examples of trainers using game structures in software testing, and I’ve probably missed some obvious ones. (I haven’t even told you about the ones I use, but that doesn’t matter.) These are some good examples off the top of my head that demonstrate the use of game mechanics in teaching.

I wanted to point out that each of them use game mechanics to teach serious lessons. While people may have fun, they come away with real-world skills that they can apply to their work as soon as they are back in the office.

Don’t be turned off by the term “game” when it comes to serious business – if you look at gaming with an open mind, you’ll see that it is all around us, being used in effective ways.

Did I miss a good software testing training gaming example? Please add them in the comments.

Edit: I just discovered an interesting post on games and learning on the testinggeek.com blog: Software Testing Games – Do They Help?

gamification, process, productivity, test automation, testing

Test Automation Games

2 Comments

As I mentioned in a prior post: Software Testing is a Game, two dominant manual testing approaches to the software testing game are scripted and exploratory testing. In the test automation space, we have other approaches. I look at three main contexts for test automation:

  1. Code context – eg. unit testing
  2. System context – eg. protocol or message level testing
  3. Social context – eg. GUI testing

In each context, the automation approach, tools and styles differ. (Note: I first introduced this idea publicly in my keynote “Test Automation: Why Context Matters” at the Alberta Workshop on Software Testing, May 2005)

In the code context, we are dominated now by automated unit tests written in some sort of xUnit framework. This type of test automation is usually carried out by programmers who write tests to check their code as they develop products, and to provide a safety net to detect changes and failures that might get introduced as the code base changes over the course of a release. We’re concerned that our code works sufficiently well in this context. These kinds of tests are less about being rewarded for finding bugs – “Cool! Discovery!” and more about providing a safety net for coding, which is a different high value activity that can hold our interest.

In the social context, we are concerned about automating the software from a user’s perspective, which means we are usually creating tests using libraries that drive a User Interface, or GUI. This approach of testing is usually dominated by regression testing. People would rather get the tool to repeat the tests than deal with the repetition inherent in regression testing, so they use tools to try to automate that repetition. In other words, regression testing is often outsourced to a tool. In this context, we are concerned that the software works reasonably well for end users in the places that they use it, which are social situations. The software has emergent properties by combining code, system and user expectations and needs at this level. We frequently look to automate away the repetition of manual testing. In video game design terms, we might call repetition that isn’t very engaging as “grinding“. (David McFadzean introduced this idea to me during a design session.)

The system context is a bit more rare, but we test machine to machine interaction, or simulate various messaging or user traffic by sending messages to machines without using the GUI. There are integration paths and emergent properties that we can catch at this level that we will miss with unit testing, but by stripping the UI away, we can create tests that run faster, and track down intermittent or other bugs that might be masked by the GUI. In video or online games, some people use tools to help enhance their game play at this level, sometimes circumventing rules. In the software testing world, we don’t have explicit rules against testing at this level, but we aren’t often rewarded for it either. People often prefer we look at the GUI, or the code level of automation. However, you can get a lot of efficiency for testing at this level by cutting out the slow GUI, and we can explore the emergent properties of a system that we don’t see at the unit level.

We also have other types of automation to consider.

Load and performance testing is a fascinating approach to test automation. As performance thought leaders like Scott Barber will tell you, performance testing is roughly 20% of the automation code development and load generation work, and 80% interpreting results and finding problem areas to address. It’s a fascinating puzzle to solve – we simulate real-world or error conditions, look at the data, find anomalies and investigate the root cause. We combine a quest with discovery and puzzle solving game styles.

If we look at Test-Driven Development with xUnit tools, we even get an explicit game metaphor: The “red bar/green bar game.” TDD practitioners I have worked with have used this to describe the red bar (test failed), green bar (test passed) and refactor (improve the design of existing code, using the automated tests as a safety net.) I was first introduced to the idea of TDD being a game by John Kordyback. Some people argue that TDD is primarily a design activity, but it also has interesting testing implications, which I wrote about here: Test-Driven Development from a Conventional Software Testing Perspective Part 1, here: Test-Driven Development from a Conventional Software Testing Perspective Part 2, and here: Test-Driven Development from a Conventional Software Testing Perspective Part 3.
(As an aside, the Session Tester tool was inspired by the fun that programmers express while coding in this style.)

Cem Kaner often talks about high volume test automation, which is another approach to automation. If you automate a particular set of steps, or a path through a system and run it many times, you will discover information you might otherwise miss. In game design, one way to deal with the boredom of grinding is to add in surprises or rewarding behavior when people repeat things. That keeps the repetitiveness from getting boring. In automation terms, high volume test automation is an incredibly powerful tool to help discover important information. I’ve used this particularly in systems that do a lot of transactions. We may run a manual test several dozen times, and maybe an automated test several hundred or a thousand times in a release. With high volume test automation, I will run a test thousands of times a day or overnight. This greatly increases my chance of finding problems that only appear in very rare events, and forces seemingly intermittent problems to show themselves in a pattern. I’ve enhanced this approach to mutate messages in a system using fuzzing tools, which helps me greatly extend my reach as a tester over both manual testing, and conventional user or GUI-based regression automated testing.

Similarly, creating simulators or emulators to help generate real-world or error conditions that are impossible to create manually are powerful approaches to enhance our testing game play. In fact, I have written about some of these other approaches are about enhancing our manual testing game play. I wrote about “interactive automated testing” in my “Man and Machine” article and in Chapter 19 of the book “Experiences of Test Automation“. This was inspired by looking at alternatives to regression testing that could help testers be more effective in their work.

In many cases, we attempt to automate what the manual testers do, and we fail because the tests are much richer when exercised by humans, because they were written by humans. Instead of getting the computer to do things that we are poor at executing (lots of simple repetition, lots of math, asynchronous actions, etc.) we try to do an approximation of what the humans do. Also, since the humans interact with a constantly changing interface, the dependency of our automation code on a changing product creates a maintenance nightmare. This is all familiar, and I looked at other options. However, another inspiration was code one of my friends wrote to help him play an online game more effectively. He actually created code to help him do better in gaming activities, and then used that algorithm to create an incredibly powerful Business Intelligence engine. Code he wrote to enhance his manual game play in an online game was so powerful at helping him do better work in a gaming context, when he applied it to a business context, it was also powerful.

Software test automation has a couple of gaming aspects:

  1. To automate parts of the manual software testing game we don’t enjoy
  2. It’s own software testing game based on our perceived benefits and rewards

In number 1 above, it’s interesting to analyze why we are automating something. Is it to help our team and system quality goals, or are we merely trying to outsource something we don’t like to a tool, rather than look at what alternatives fit our problem best? In number 2, if we look at how we reward people who do automation, and map automation styles and approaches to our quality goals or quality criteria, not to mention helping our teams work with more efficiency, discover more important information, and make the lives of our testers better, there are a lot of fascinating areas to explore.

design, mobile, news

It’s Not About Mobile App Download Numbers, It’s About User Engagement

Leave a comment

Peggy Anne Salz has blogged about using SMS to help encourage user engagement with mobile apps, based on research by her firm and Tyntec. The white paper she links to is an interesting read, but the message that sticks out for me is when Peggy says:

“The bottom line: app developers need to work out a strategy to increase engagement, not just downloads.”

How many apps do you have on your smartphone or tablet currenty? I want you to check right now, and get a rough number in your mind. (Humour me, it’s worth it to see.)

Now, how many of those apps do you use regularly? Conversely, how many are just sitting there on your device, but you rarely if ever use them?

I have close to 100 apps, and some of my colleagues think I am conservative – they have far more on their devices.  I probably use about a dozen of them regularly. As mobile consumers, we have a lot of choice for content, entertainment and productivity applications, all competing for our attention on mobile devices. We can’t use them all regularly, so we optimize our time and focus on the ones that we like, the ones we find engaging and return to again and again.

While getting people to download and install an app that you have worked so hard to build  is important, Peggy points out that an app’s success is part of a long-term relationship:

“Smart developers understand that selling apps is a serious business. But a raft of research suggests a singular focus on driving downloads is patently flawed. It’s really how well app developers can persuade us to make their app part of our regular routine that will make or break their app business. Winning is all about making the right choices to delight us again and again.”

 

If people merely download and install our apps and then forget about them, we lose out in the long term. Sure, we might get some initial sales, but customers will forget about our brand. They won’t come back to us when they need to have a product or service solve a problem for them, and we as suppliers are lost in the great sea of available apps and service providers. We might lose many of our initial customers for good, and that hurts. When I worked in sales, I worked very hard to have repeat customers – that’s what got me through the slow times – people coming back and asking specifically for me, Jonathan, and spending their time and hard earned cash in my direction.

In her blog post, Peggy points out that using SMS (Short Message Service) is one way app developers can use mobile technology to enhance engagement and app use. With SMS integration, customers are reminded of the app, get notices on new features, and are encouraged to use it in different ways. As consumers, we may have forgotten about a great app, and a reminder once in a while might be what we need to go back. She  provides some examples of how app developers can use SMS: “…to deliver valuable content, drive traffic to the community website, and reactivate users who haven’t interacted with the app in while.”

The lesson for app developers is that app downloads are well and good, but we need to strive for user engagement even more. Peggy makes a good case for using SMS, but what other ways can you use technology and app design to drive continued use? Another area to look at is in your app design. Is it usable and user friendly? Does it solve problems that your customers have in mobile contexts? Is it reliable under different conditions in weather, temperature, lighting, and with different network connections, latency and speed?

I have some ideas on design: Three Keys to Mobile Application Design (and soon there will be much more in my upcoming book: Tap Into Mobile Application Design), and a lot of ideas on app reliability in my e-book: Tap Into Mobile Application Testing. There are others.

In fact, the sky is the limit in mobile app development to explore engagement models. We just need to set our sights on user engagement and be creative with our use of the technology. It is not enough to just build a mobile app or mobile web presence. Long term engagement trumps downloads and installs because that means people will actually use your app more than once, and hopefully engage with your brand, and buy more products and services from you later on. Take the long view, even though we are in a pressure cooker of short term wins on many of our mobile projects.

app development, books, mobile, mobile testing, Tap Into Mobile Application Testing

Book Excerpt: The Seven Deadly Sins of Mobile Apps

5 Comments

This is an excerpt from my book: “Tap Into Mobile Application Testing“, from Chapter 10, pp. 431-432:

Here is a simple summary I created to help you think of different areas an app can fail. This is another mind hack you can use to quickly organize your thoughts, and analyze an application quickly.

  1. Lust: the app advertises that it can do a lot more than it actually can. It leaves you feeling unfulfilled and wanting more.
  2. Gluttony: it uses far too many resources. It uses up your device memory, downloads large image and other files, and slows down your device, eats up your data plan and kills your battery.
  3. Greed: the assumes you have a strong network connection, and would love to use as much of your network resources as possible. The app can’t handle poor or weak connections, or transitions between network types when you’re on the move.
  4. Sloth: the app performs very poorly. It is far too slow to respond to your interactions, and takes too long to do anything useful.
  5. Wrath: it doesn’t play well with other apps. It has special settings that override your defaults, and doesn’t allow other services to work well with it. It may even cause other apps to malfunction because of its behavior.
  6. Envy: The app is too close to other available apps out there that you would prefer to use instead. It’s a copycat. You just wasted precious time and resources to get an app that wishes it was something else, but it just can’t deliver on its features, and usefulness.
  7. Pride: The app is difficult to use, expecting users to adapt to its way instead of helping you be more effective.You are subordinate to it, and you have trouble using it, and end up feeling stupid. This is most likely due to designers and developers assuming they know better than the users, and ignoring valuable usability feedback and usability bugs.

Thanks to Shannon Hale and Jared Quinert for their help with this list.

This is now a talk!

Note: I have created a one hour talk on this topic if your company, user group or conference are interested. I provide tips for designers, developers and testers on how to create apps that are engaging and reliable for mobile users.

 

exploratory testing, gamification, scripted testing, testing

Software Testing is a Game

14 Comments

David McFadzean and I wrote an article for Better Software magazine called The Software Development Game, which was published in the September/October edition. We describe applying game-like structures and approaches to software development teams. I’ve been asked for another article on applying game-like approaches to testing, so look for more in this space.

In the meantime, here are some of my current thoughts.

How is software testing like a game? If we think of software development as a game, or a situation involving cooperation and conflict with different different actors with different motivations and goals, software testing fits as a game within the larger software development game (SDG). While the SDG focuses more on policy, practices and determining an ideal mix of process, tools and technology, software testing is often an individual pursuit within a larger game format. There are two distinct styles of playing that game today: scripted testing and exploratory testing.

In the scripted testing game, we are governed by a test plan and a test case management system. We are rewarded for going through a management system and repeating test scripts and marking them as passed or failed. We are measured on our coverage of the tests that are stored in this test case management system. It is very important to stakeholders that we get as close to 100% coverage of the tests in that system as possible. This is a repetitive task, in a highly constricted environment. Anything that threatens the test team goal of high coverage of the tests within a test case management system tends to be discouraged, sometimes just implicitly. The metrics are what matter the most, so any activity, even more and better testing will be discouraged if it takes away from reaching that objective. “You, tester are rewarded on how many test cases you follow and mark as pass or fail in the system. If there is time after we get this number, you may look at other testing activities.”

In the exploratory testing game, testers are given some degree of self-determination. In fact, the latest version of the definition of exploratory testing emphasizes this:
“a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

The game player in exploratory testing is rewarded on the quality of their work, their approach, and the quality of the information that they provide. Exploratory testing works by being adaptable and relevant, so traditional ideas of metrics are often downplayed in favor of qualitative information. If a tester changes their mind when they are working on an assignment, and they have good reason to do so, and can make their case defensible, they are rewarded for changing things up because it helps make the product and our approach better. Coverage is important, but testers are rewarded on using multiple models of coverage (and discovering important new ones) rather than following the regression tests and counting that as complete coverage.

When I think about exploratory testing, I am reminded of the game Nomic where changing the rules is considered a valid move. Instead of a group effort like Nomic, a tester has the freedom to change their own course, and to help improve the approach of the overall team by demonstrating excellent work. “You, tester are rewarded by excellent testing, and your skill and ability to find and report important problems. We don’t care if you work through one test, or a thousand tests to get that information. The information you provide and how it improves the quality and value of our products is what we value.” It is deeply important for exploratory testers to be able to adapt to changing conditions, to uncover important information to help the team create value with their product, and for many, to get endorsement and the respect of their peers.

Now think of game play activities. Both scripted and exploratory testing approaches involve repetitive work. In the gamification space, we can explore these activities, and how we can apply game concepts to enhance the work. One area we can explore are video games. Imagine we are are playing a massively multiplayer online role-playing game (MMORPG). Some players like to perform repeated tasks that are unrelated to individual and shared game objectives. They like to repeat tasks to unlock features related to their character, such as acquiring new outfits or accessories for their character.

Other players are very achievement goal oriented – they try to reach individual goals and gain more points and unlock more achievement-based features in the game, and learn that when they co-operate with others, that they can achieve even more. One player type gets rewarded more for repeated individual tasks, while the other gets rewarded for trying to create achievement or quest value for themselves and others. The two types blend, as any player will have a subgoal of changing the appearance of my character, or acquiring accessories or currency to enhance quest or other achievement game play. People who are less adventurous and are more caught up in acquiring personal character attributes will also take part in quest events.

If you play these games and try to collaborate, you will often find other players who are at different points on this spectrum. It can be frustrating when your team members are more interested in building up their own character’s numbers than in helping the team as a whole. For example, if you are off on a raid, and one or more team members constantly stop to pick up health or currency points, your team will be let down and may have to regroup. It can be difficult to work together, since the perceived value of the game and the reward structures don’t match. One group want personal metrics and accessories, while the other care about non-character pursuits and the respect of their peers more.

In the software testing game, it is difficult to try to play an exploratory testing game in a system that is rigid, restrictive, and rewards testers for playing a scripted game. Conversely, those who are used to being rewarded on acquiring coverage counts in test case management systems can be very threatened when they are now rewarded on the quality of their testing skills and the information they provide. Testing tradition and common practice tends to think of testing as one sort of game, of which a test case management system is central. What they fail to realize is that there are many ways of playing this game, not just one. I want to explore this other alternatives in more detail through a gaming lens.

Testers and test managers, there is a lot more to our world than scripted testing and test case management tools. They are part of an ancient game (from at least the 1970s), which is struggling and showing its age in the face of new technology and new approaches to work that mobile technology and other styles of work are bringing. Modern testers work hard to play a game that fits the world around us. Are you measuring and rewarding them for playing the right kind of game, or the one you are most used to? Like the video game character who has gathered metrics and accessories, are you gathering these at the expense of the rest of the team’s goals, or are these measures and rewards helping the entire software development team create a better product?

Watch for more in this space as I explore software testing from the perspective of gamification, game theory and other game activities more deeply.

books, mobile, mobile application testing, mobile testing, Tap Into Mobile Application Testing, testing mobile apps

Tap Into Mobile Application Testing Book Now Available in Beta

9 Comments

Update: The book is no longer in beta. The final version was published in September 2013 and is available here: Tap Into Mobile Application Testing.

How did you spend your summer? I spent mine writing a book: Tap Into Mobile Application Testing. Now that smartphones and tablets are taking over the mobile space, there has been tremendous demand for ideas and training on how to test on these devices. I started out with my Testing Mobile Apps course, but that only scales so far. Many of you have asked me about writing a book on the topic, so I did. It’s not completely finished yet, but I managed to create most of the content I had hoped to share.

I have two audiences with this book:

  1. Mobilists without much testing experience
  2. Professional testers without much mobile experience

I wrote the book with both of these audiences in mind, and I was also pleased to learn that even deeply experienced mobile testers told me they learned a lot of new ideas and approaches for their own testing projects after reviewing the book.

It is only available in electronic formats right now: PDF, epub and mobi, which should work on most e-reader apps on most mobile devices. I chose mobile first because that’s the medium many of my readers will choose first. It’s also faster for me to get content out there, and many traditional publishers print dead tree copies first, then look at electronic versions. I am using Leanpub, which allows me to publish free updates to every reader who has purchased a copy, even the finalized version that I am targeting for a release later in the year. Once I have a finalized version, I will provide print copies using a print-on-demand service.

The Beta version is a bit rough around the edges. I still have the following on my to-do list:

  • Copy editing and attribution clean up
  • Creating figures and images
  • Section rewrites and new material where needed

If you notice something wrong, misattributed, or missing feel free to contact me and I can fix it before the final version is complete in December.

If you are curious about mobile apps testing on modern devices, I hope you find it useful.

gamification, process, process fusion

Content Pointer: New Article – The Software Development Game

1 Comment

Better Software magazine has published a new article that I co-authored with David McFadzean called “The Software Development Game.” A PDF version is available here: SDG Feature in PDF.

David approached me in the summer of 2008 and pitched the idea of co-authoring a piece about using game-like concepts on software development teams. It turned out that David and I had both been influenced by game theory when creating policy and strategy on software development teams, but David had taken it a step further than I had: he had formalized an actual game-like structure on several teams.

It took us a while to write the piece. First of all, we wanted to observe two SDG instances that were newly created, so we took our time. Secondly, we found that the topic is massive. It was very difficult to fit our ideas into a 3000 word article. Also, we wanted to incorporate more ideas from the gamification of work movement into game play. Finally, our ideas had to pass the review of peers that we trusted, and their feedback took time to address and incorporate.

The final result is a very brief introduction to the topic. It is one powerful tool, particularly for self-organizing teams to help determine their own destiny. Instead of being told what to do by a coach, process consultant or manager, a team can use a simple framework to determine their own optimal mix of processes, tools and practices at a particular point in time. The game structure provides visibility on decisions, and can help teams align their technology focus with the visions of leadership.

We hope you find it as interesting as we do, and if you try it out on your own team, let people know how it works for you and your team.

design, development, mobile

Three Keys to Mobile Application Design – Part 4 Conclusion

2 Comments

In Part 1 of this series, we looked at mobility and app design. In Part 2, we looked at social aspects. In Part 3, we looked at gaming and entertainment features to consider with mobile app design. In Part 4, I conclude with some final thoughts, a simple example, and three people who inspire me.

Conclusion

What is it about some mobile applications that we use over and over, to the point of addiction? What is it about others that we seldom use? I’ve found that a mix of mobility features (and usability!), social features, and gaming/entertainment are some of the reasons why we keep coming back. We enjoy the experience of the application. It is easy to use and helps us solve problems or reach goals while we’re on the move. It helps us feel connected to our friends, family and coworkers, so we never feel alone. The app helps us quickly access information we need. It is entertaining, so we enjoy spending time in the application, and feel drawn to it when we’re away from it.

Our mobile devices have a lot of applications on them, so we have a huge amount of choice on where to spend out time. To keep people coming back to use your application, consider using the mix I’ve recommended above. To figure out that mix, make sure you observe people (but don’t creep them out!), ask questions, and spend time away from your desk in different contexts. As I mentioned in an earlier post, when I ride public transit, I feel like I’m surrounded by people using tablets with e-readers, people texting on mobile phones, or using social networking services and commenting about their trip. Others are watching TV shows or movies, or simply listening to music. When I’m in an airport, I jostle with others flocking to power stations prior to boarding the aircraft. In a restaurant or coffee shop, the people around me are staring into their screens. I observe what they are doing, but I take it a step further. If the moment is right, I strike up a conversation. Many people are happy to demonstrate their device and the programs they use, and tell you what they enjoy about them, and where they use them. As a designer (and tester), this kind of information is gold.

I also look for apps that demonstrate good mobile design, and look at what they have done well. One that has caught my eye in the enterprise space is Rypple, now owned by Salesforce. It’s a talent management app that has a mix of the three keys: mobility, social interactions and gaming/entertainment. It is easy to use anywhere, and has features to keep drawing you in. In the mass market, I look at popular games like Angry Birds, and social networking apps. What motivations to people have to use these apps? What do they do well? Conversely, what are people complaining about? Do they crash too much, or are they too slow?

Putting it All Together

Let’s try applying this thinking to a simple app. Imagine you are designing an app that provides a listing and related information for local coffee shops. Here is a brief listing of features that can help tie together a good user experience that will keep people coming back for more:

1. Mobility:

  • map integration
  • information: ratings, etc

2. Social:

  • camera and photo integration
  • social media support
  • people need to take pictures of their food and drinks so they can post them and chat about the experience with their friends!

3. Gamification:

  • unlock premium content after usage – ie. after visiting all the shops listed in the app, or for multiple visits
  • look at coupon or other deals to integrate with after app usage, or other ways to co-ordinate with local businesses for cross-promotion

A word of caution: be sure to implement these features in a way that will resonate with your user community. Make sure your mobility features work well, don’t mislead your user, and don’t have irritating defaults, such as always defaulting to their home address, even when they are traveling. With social, don’t just copy what is out there and popular and think people will like it. Take time to model the existing social connections your app users will have, and make sure your app plugs into and enhances that. With gamification and entertainment, don’t put in childish rewards for an enterprise app, or app aimed for adults, or people will just think it is lame. Again, model the space and find out intrinsic motivations, and take the context and users into account.

Good Design References

These are some abstract ideas to help you model and create your app. For implementation ideas, I look to people who specialize in mobility and are pushing the technology. Here are some people I look to for help, innovation and inspiration:

Luke Wroblewski
Brad Frost
Josh Clark

The always excellent Smashing Magazine has a fabulous piece on gamification: Gamification and UX: Where Users Win or Lose with tips on where to use gamification, and where not to use it. Smashing Mag also has a lot of great mobile design information here: The Elements Of The Mobile User Experience .

I hope you find these ideas useful as you design your own mobile apps, or work to help others. Happy app designing!

design, development, mobile

Three Keys to Mobile Application Design – Part 3 Gaming and Entertainment

2 Comments

In Part 1 of this series, we looked at mobility and app design. In Part 2, we looked at social aspects. In Part 3, we will look at gaming and entertainment features to consider with mobile app design.

Gaming and Entertainment

We spend an enormous amount of time playing games or using entertainment apps and services on our mobile devices. When I ride my local commuter train, I see people using e-readers on tablets all around me. I also notice people playing games on their smartphones. When I am on an airplane, there are people all around me with tablets or smartphones reading, playing games, watching TV shows or movies, or simply listening to music.

Entertainment and games tap into a different part of our brains than other activities. We enjoy them, and we find them addictive. They appeal to our emotions: a TV show or movie give us a brief respite from stresses in life. A book requires our imagination, but it too helps provide a break and triggers creative thoughts as our brains fill in the visuals of the story for us.

With games, we feel challenged and a sense of accomplishment when we complete a level or finish a quest. We will spend hours doing repetitive actions in a game so we can get small rewards within it. In a social application, we tolerate repetitive tasks, such as uploading all of our vacation photos, because the intrinsic value overcomes our feelings of boredom or frustration. We know that our friends and family will enjoy seeing them (as will we) and that they will spark comments and conversations.

Contrast this with tasks that we encounter at work. We often procrastinate over repetitive or tasks we find boring, because there is little motivation for us, or the reward won’t be realized in the short term. Application designers and process wonks have noticed this contrast, and have begun to apply game-like processes to applications. There is enormous productivity in gaming and entertainment applications, so how do we tap into that for our corporate apps?

One movement that has become popular lately is called Gamification. Gamification involves imposing game-like structures on work tasks to help make work more entertaining, and help workers become more productive. Another concept is game theory which uses mathematical models to study decision making. Understanding both concepts can help us as we design apps.

Here are a couple of concepts to think about when designing your mobile app:

  • Gamification (provide incentives to use the app, or to take some of the tedium out of repetitive tasks)
  • Interactivity, media, other features of entertainment (tap into emotions)

Gamification can be as simple as providing rewards in an application after you complete a certain number of repetitive tasks. It might be a visual representation, such as note that says: “good job” or a graphic equivalent of a gold star. More sophisticated implementations may unlock new features or content in the application for you, or provide a media break, such as a short video that is entertaining, but provides information that users will find useful. When thinking of a game in this context, think of games that have quests and achievements, where players repeat tasks to score points, or acquire goods or credibility within the game. Some app developers are even looking at enormously popular social games and modeling their entire workflow in a similar manner.

Interactivity, media, and other forms of entertainment are features that allow us to tap into people’s emotions. We like nice colors, sounds and things that move. Features that stimulate our senses (see, hear, touch, smell, taste) tend to evoke emotions. (I haven’t figured out how to appeal to taste and smell with apps, at least not yet. 🙂 ) A clean design with great graphics and appealing colors will evoke an emotion. An app that provides tactile feedback to help train you to work with it in certain ways helps us understand concepts more quickly, and reinforces emotional responses. A bit of video or sound can go a long way. On one app I worked on, we replaced long paragraphs of text with short videos where a professional speaker provided the same content in an entertaining way. They were easier to consume and understand, and fit the devices better since they reduced scrolling and eye fatigue that we experienced with the wall of text effect.

The devices themselves provide a lot of affordances for gaming and entertainment:

  • Game-specific animation and graphics libraries
  • Media: camera, video and sound recording and playback
  • Rich graphics support with high resolution screens
  • Networking to connect to information and different people
  • Natural User Interfaces to support touching and gesturing
  • Movement sensors to support different kinds of inputs or control

Gamification of tasks, and adding entertainment features can help with serious applications as well. One aspect is to reward the completion mentality. When I do chores around the house, or work on my car, I sit back and admire my work when I’m finished, and bask in the sense of satisfaction for a job well done. This is more difficult with knowledge work. It is virtual, so I can’t sit there and look at the job I completed. We can put these affordances in our apps. Furthermore, if there is a sense of reaching a level and getting some sort of completion notification, I may stick with a task and finish it, rather than procrastinate or engage in distracting activities. A user might just spend a few more minutes with your app if they feel they can get some sort of reward for completing a level or a task.

I also like gamification and entertainment features to help inject some variation to keep people interested. lately, I’ve been advising a startup that is developing a mission-critical app. Gamification is incredibly important here because the information and activities in the app are very important, and the people using the app need to be brain-engaged and paying careful attention, or learning something important. If the same old same old pops up in the app all the time, people will just tune it out and click to dismiss, much like we do with terms of service or end user license agreements. We do what we can to dismiss it and do something else. (I remember a popular personal firewall program that popped up with so many messages, people would just turn it off.) Variation and interaction is important to hold our attention, and so that people with different learning styles can synthesize and retain information.

Gamification and entertainment also entice people to use our apps. Many people have over 100 apps on their smartphones or tablets. The difference between them using our app or a competitor’s app can come down to how well we entice them to use our app. If there is little to entertain them, there are no rewards, or no sense of completion when working through the app, why would they swipe three screens over to start our app, when they can use a similar app that is on their home screen?

Charge Extra for Cheating

There are also revenue opportunities to upsell premium content. One app that I designed unlocked premium content at various levels of completion. As a user worked their way through a workflow, premium subject matter expert content was made available to them. After so many steps of activities or completion, consider providing something to reward the users. However, I advise allowing people to cheat. Let them unlock the premium content without going through all the steps, but for a premium price. That appeals to our laziness, and instant gratification culture, and you can charge a lot more money for these sorts of features than you can with apps themselves. Just look at popular game platforms. They have enormous markets related to buying things that are unrelated to the game or quest. You can buy a shield, or different armor for your character, or pets, and other status items just like in real life. A user might only be willing to pay $0.99 for a game (or get a free version), but they might spend $20.00 accessorizing, or for getting other premium content. Look at how popular gaming platforms have their own marketplaces, and the kinds of things they sell. These items appeal to status (I can brag socially about it, or have status over other players because I am unique), and to intrinsic motivations that users may put more of a dollar value on than our actual app. There are even eBay and online classified services that allow you to purchase virtual currency, or a player that is at a higher level of game play that already has worked through many levels within the game. Some people will pay a premium for shortcuts, so tap into that.

The key point here is that we have a lot of choices in the apps we use, so we need to put in mechanisms into our apps that draw people back into them. If we don’t provide incentives to use the app, they may go to an app that is more entertaining and rewards them for repetitive tasks, or is simply more enjoyable to use. Furthermore, if app usage and the information in the app is important, you don’t want them to tune it out. If you want people to keep going back and using your app more and more, you will need to figure out how to tap into gaming and entertainment.

Stay tuned for the conclusion of this series next week.