Category Archives: test quests

gamification, test quests, testing, testing debt, tools, UX

Designing a Gamification Productivity Tool

4 Comments

Gamification and Software Testing

I haven’t spoken about this project publicly because we never got to a public release. Software testing tools represent a tiny market, so they are incredibly difficult to fund. Some of you have asked me about gamification tools with testing, so I thought I would share this brain dump.

A few years ago, I was asked to help a development team that had significant regulatory issues, and frequently accrued testing debt. The product owner’s solution was to periodically have “testing sprints” where other team members helped the over burdened test teams catch up. There was just one problem: the developers HATED helping out with 2 weeks of testing, so I was asked to do what I could to help.

A couple of the senior architects at this company were very interested in Session Tester and asked me why I had put game mechanics in a testing tool. I didn’t really realize at the time I had put game mechanics in, I was just trying to make something useful and engaging for people. So I started talking with them more about game design, and they encouraged me to look into MMOs and co-operative games. The team played games together a great deal, so I learned about the games they enjoyed and tried to incorporate mechanics

I set up a game-influenced process to help structure testing for the developers, and taught them the basics of SBTM. They LOVED it, and started having fun little side contests to try to find bugs in each other’s code. In fact, they were enjoying testing so much, they would complain about having to go back to coding to fix bugs. They didn’t want to do it full time, but a two week testing sprint under a gamified, co-operative model with some structure (and no horrible boring test cases) really did the trick.

Eventually, I worked with some of the team members with a side-project, and the team lead proposed creating a tool to capture what I had implemented. This was actually extremely difficult.  We started with what had been done with Session Tester, and went far beyond that, looking at a full stack testing productivity tool. One of the key aspects of our approach that differed from the traditional ET and scripted testing approaches was the test quest. As I was designing this test tool, I stumbled on Jane McGonigal’s work and found it really inspiring. She was also a big proponent of the quest as a model for getting things done in the real world. Also, we were very careful in how we measured testing progress. Bug counts are easily gamed and have a lot of chance. I have worked in departments that measured on bug counts in the past, and they are depressing if you are working on a mature product while your coworkers are working on a buggy version 1.0.

One thing Cem Kaner taught me was to reward testers based on approach rather than easily counted results, because they can’t control how many bugs there may or may not be in a system. So we set up a system around test quests. Also, many people find pure exploratory testing (ET) too free form and it doesn’t provide a sense of completion the way scripted test case management tools do. And when you are in a regulatory environment, you can’t do ET all the time, and test cases are too onerous and narrow focused. We were doing something else that wasn’t pure ET and it wasn’t traditional scripted testing. It turns out test quest was a perfect repository for everything that we needed to be done. Also, you didn’t finish the quest until you cleaned up data, entered bugs and other things people might find unpleasant after a test session or two. There is more here on quests: Test Quests – Gamification Applied to Software Test Execution

As I point out in that post, Chore Wars is interesting, but it was challenging for sustained testing because of different personalities and motivations of different people. So we used some ideas from ARGs to sprinkle within our process rather than use it as a foundation. Certain gamer types are attracted to things like Chore Wars, but others are turned off by them, so you have to be careful with a productivity tool.

We set up a reward system that reminded people to do a more thorough job. Was there a risk assessment? Were there coverage outlines? Session sheets? How were they filled out? Were they complete? What about bug reports? Were they complete and clear?  I fought with the architects over having a leaderboard, but eventually I relented and we reached a compromise. Superstar testers can dominate a system like this, causing others to feel demoralized and not want to try anymore. We decided to overcome that by looking at chance events, which are a huge part of what makes games fun, so no one could stay and dominate the testing leaderboard, they would get knocked to the bottom randomly and would have to work their way back up. Unfortunately, we ran into regulatory issues with the leaderboard – while we forbade the practice of ranking employees based on the tool, this sort of thing can run afoul of labor laws in some countries, so we were working on alternatives but ran out of resources before we could get it completed.

Social aspects of gaming are a massive part of online games in particular, but board games are more fun with more people too. We set up a communication system similar to a company IRC system we had developed in the past. We also designed a way to ask for help and for senior testers to provide mentoring, and like MMOs, we rewarded people who worked together more than if they worked alone. Like developer tools, we set up flags for review to help get more eyes on a problem.

We also set up a voting system so testers could nominate each other for best bug, or best bug report, best bug video, and encouraged sharing bug stories and technical information with each other within the tool.

An important design aspect was interoperability with other tools, so we designed testing products to be easily exported so they could be incorporated with tools people already use. Rather than try to compete or replace, we wanted to complement what testers were already doing in many organizations, and have an alternative to the tired and outdated test case management systems. However, if you had one of those systems, we wanted to work with it, rather than against it.

Unfortunately, we ran out of resources and weren’t able to get the tool off the ground. It had the basics of Session Tester embedded in it, with improvements and a lot of game approaches mixed in with testing fundamentals.

We learned three lessons with all of this:

  1. Co-operative game play works well for productivity over a sustained period of time, while competitive game play can be initially productive, but over time it can be destructive. Competition is something that has to be developed within a co-operative structure with a lot of care. Many people shut down when others get competitive, and rewarding for things like bugs found, or bugs fixed causes people to game the system, rather than focus on value.
  2. Each team is different, and there are different personalities and player types. You have to design accordingly and make implementations customizable and flexible. If you design too narrowly, the software testing game will no longer be relevant. If design is more flexible and customizable from the beginning, the tool has a much better chance of sustained use, even if the early champions move on to other companies. I’ve had people ask me for simple approaches and get disappointed when I don’t have a pat answer on how to gamify their testing team approach without observing and working with them first. There is no simple approach that fits all.
  3. Designing a good productivity tool is very difficult, and game approaches are much more complex than you might anticipate. There were unintended consequences when using certain approaches, and we really had to take different personality and player styles into account. (There are also labour and other game-related laws to explore.) Thin layer gamification (points, badges, leaderboards) had limited value over time and only appealed to a narrow group of people.

 If some of you are looking at gamification and testing productivity, I hope you find some of these ideas useful. If you are interested in some of the approaches we used, these Gamification Inspiration Cards are a good place to start.

exploratory testing, gamification, software testing, test quests

Test Quests – Gamification Applied to Software Test Execution

3 Comments

I decided to analyze a game feature, the “quest“, which is used in popular video games, particularly MMORPGs. Quests have some compelling aspects for structuring testing activitues. Jane McGonigal‘s book “Reality is Broken” provided me with a solid analysis of quests, and how they can be adapted to real life activities. Working from her example of a quest (ch. 3 pp. 56) , I created a basic test quest format:

  1. Goal statement (what we intend to accomplish with our testing work)
  2. Why the goal matters (why are we testing this?)
  3. Where to go in the application (what technique or approach are we using to test?)
  4. Guidance (not detailed steps, but enough to help. Bonus points for using video or other rich media examples.)
  5. Proof of completion (how do you know when you are finished?)

A quest is larger than a single testing mission (or a test case), but is smaller than a test plan. It’s a way we can organize testing tasks to help provide a sense of completion and interest, but in areas that require exploration and creativity. Just like in a video game, there are multiple ways to satisfy a quest. Once we have fulfilled a quest, which might take days or hours, depending on how it is created, we can move on to another one. It’s another way of organizing people, with the added bonus of leveraging years of game design success. Furthermore, modern technology involves a lot of collaboration between people in different locations, using different technology to reach a common goal, and we need to adapt testing to meet that. Testing a mobile app in your lab, one tester at a time, won’t really provide useful testing for an app that requires real-time communication and collaboration for people all over the world. MMO’s do a fabulous job of getting people to work hard and co-ordinate activities in a virtual world, and people have fun doing it. I decided to apply it to testing.

Where do quests fit? Think in terms of a hierarchy of activities:

  • test strategy and plan
  • risks that are mitigated through testing
  • different models of coverage that map to risk mitigation
  • test quests
  • sessions, tours, tasks
  • feedback and reporting

A good test approach will have more than one model of coverage (check I SLICED UP FUN for 12 mobile coverage models), and under each model of coverage, there will be multiple quests. Sometimes quests will be repeated when regressions are required.

So why add this structure?

One area I have worked on over the years is using structure and guidance to help manage exploratory testing efforts. In the past, test case management systems provided some measure of coverage and oversight, but they have little in the way of intrinsic value for testers. People get tired of repeating the same tests over and over, but management love the metrics and they provide even though they are incredibly easy to cheat with. Furthermore, from a tester’s perspective there is an extrinsic reward that is inherent in the design of the tools, and they are easy to use. There is also a sense of completion, once I have run through X number of test cases, I feel like I have accomplished something.

With exploratory testing, the rewards are more intrinsic. The approach can be more fulfilling; I personally feel like I am approaching testing in a more effective way, and I can spend my time on high value activities. However, it is harder to measure coverage, and it is more difficult to direct people in areas where coverage is required without adding some guidance. There have been a lot of different approaches to adding structure to exploratory testing over the years to find a balance. Test quests are another approach to adding structure and finding that balance between the intrinsic rewards of pure exploratory testing, and the extrinsic rewards of scripted testing. This is an idea to provide a blend.

As many of you have heard me argue over the years, test cases and test case management systems are merely one form of guidance, there are others. In the exploratory testing community, you will see coverage outlines, checklists, mind maps, charter lists, session sheets, and media such as video demonstrations and all sorts of alternatives. When it comes to managing exploratory testing, one of the first places we start is to use session-based testing management. This approach helps us focus testing in particular areas, and provides a reviewable result, which makes our auditors and stakeholders happy. I’ve used it a lot over the years.

I’ve also used Bach’s General Functionality and Stability Procedure for over a decade to help organize exploratory testing. However, through experience, unique projects and contexts, I have adapted and moved away from the orthodoxy where I saw fit. However, when I started analyzing why people on my teams have fun with testing, SBTM and Bach’s General Functionality and Stability Procedures were big reasons why. Even though I often use a much more lightweight version of SBTM than he has created, people appreciate the structure. The General Functionality and Stability Procedures is a great example of guidance for analysis, exploration, and great things to do as testers.

The other side of fun on the teams I work on are related to humour, collaboration and technology. We often come up with nicknames, and divide up testing into teams and hold contests. Who can come up with the best test approach? Who recorded the best bug report video? Who found the most difficult to find bug last week? What team has the most pop culture references in their work? Testing is filled with laughter, excitement and learning, and some good plain old fashioned silly fun. We communicate constantly using technology to help stay up to speed on changes and progress, and often other team members want to get in on the action. Sometimes, it’s hard to get the coders to code, the product owners to product own, and the managers to manage, because everyone wants in on the fun. In the midst of this fun is incredibly valuable testing. Stakeholders are blown away by the productivity of testing, the volume of useful information produced, the quality of bugs, and the detailed, useful information from bug reports to status reports and quality criteria that is produced. While there is laughter and fun, there is hard work going on. I learned why this is so effective reading Jane McGonigal’s work.

In Reality is Broken, Jane McGonigal describes Augmented Reality Games (ARGs). These are real life activities that are gamified – they have a game-like structure applied to them. She mentions Chore Wars, and how gamifiying something as mundane as household chores can turn it into a fun activity. She mentions that since cleaning the bathroom is a high value activity in the game, her and her husband have to work hard to try to clean it before the other does. McGonigal explains that since there is a choice, and meaning attached to the task, people choose to do it under the mechanism of the game. It’s not that awful thing no one wants to do anymore because it is unpleasant, when framed within a game context, it is a highly sought after quest or task to complete. You get points in the game, you get bragging rights, you get intrinsic rewards as well as the extrinsic clean bathroom. Amazing.

If we apply that to testing, how about using lessons from ARGs to gamify things like regression testing, or test data creation, or other maintenance tasks we don’t like doing? One way we can do this is to sprinkle these tasks within quests. You can only complete the quest by finishing up one of these less desirable tasks.

In Reality is Broken, McGonigal defines a game as having four traits: a goal, rules, a feedback system, and voluntary participation (pp.21). Working backwards, in exploratory testing, a lot of what we do is voluntary because testers have some degree freedom to make decisions about what they are going to test, even if it is within narrow parameters of coverage. Furthermore, we can choose a different model of coverage to reach a goal. For example, I was working with an e-commerce testing team who were bored to death of testing the purchasing engine because they were following the same set of functional test scripts. To help them be more effective and to enjoy what they were doing, I introduced a new model of coverage to test the purchasing engine: user scenarios. Suddenly, they were engaged and interested and found bugs they had previously missed. I then helped them develop more models of coverage so that they could change their perspective and test the same thing, but with variation to keep them engaged and interested while still satisfying coverage requirements. As humans, we need to mix things up. Previously, they had no choice – they were told to execute the tests in the test case management system, and that was the end of it.

Feedback systems are often linked to bug reporting systems in testing. But I like to go beyond that. Bring in other people to test with you in pairs, trios or whatever combination to bring more ideas to the table. This isn’t duplicated testing, but a redoubling of brain power and effort. I also utilize instant messaging, IRC, and big visible charts to help encourage feedback across functional areas of teams.

Rules in testing are often related to what is dictated to us by managers, developers, and tradition. It boggles my mind how many so-called Agile programmers will demand their testers work in un-Agile ways, expecting them to create test plans, test cases and use test case management systems. When I ask the programmers if they would like to work that way, they usually say no. Well guess what, not many other homo sapiens like to work that way either. I prefer to have rules around approach. We have identified risks, and models of coverage to mitigate those risks, and we use people, tools and automation to help us reach our goals. Rather than count test cases and bugs, we rate our team on our ability to get great coverage and information that helps stakeholders make quality-related decisions.

Finally, a goal in testing needs to be project-specific. If you want to fail, you just copy what you did last time on your test project. The problem with that is you are unaware of any new risks or changes and you’ll likely be blind to them. Every project has a goal, a way we can measure whether we did the right sort of work to help reach that goal, rather than “run the regression tests, automate as many as possible, and if there is time, do other testing”, we have something specific that helps ensure we aren’t doing busy work, but we’re creating value.

When it comes to quests, they can have this format as well. A goal, a feedback system, rules or parameters on where to test, and voluntary participation. As long as all the quests are fulfilled for a project, it doesn’t matter who did them.

It turns out that my application of SBTM, Bach’s General Funcationality and Stability Procedure, plus some zany fun and utilizing technology to help socialize, report and record information, I was right next door to gamification. Using gamification as a guide, I hope to provide tools for others who also want to make testing effective and fun. A test quest is one option to try. Consider using avatars, fun names and anything that resonates with your team members to help make the activity more fun. Also consider rewards for difficult quests and tasks such as a free meal, public kudos, or time off in lieu. Get creative and use as much or as little from the video game world as you like.

Some of my goals with test quests are:

  • Enough structure to provide guidance to testers so they know where to focus efforts
  • Not so much structure (like scripted test cases) that personal choice, creativity and exploration are discouraged or forbidden
  • Guidance and structure is lightweight so that it doesn’t become a maintenance burden like our scripted regression test cases become (both manual and automated)
  • Testers get a sense of purpose, they get a sense of meaning in their work, and completion by completing a set of tasks in a quest
  • Utilize tools (automated tests, automated tasks, simulators, high volume test automation, monitoring and reporting) to help boost the power of the testers and be more efficient and effective, and to do things no human could do on their own
  • Encourage collaboration and sharing information so that testers can provide feedback to other project team members on the quality of the products, but also get feedback on their own work and approaches
  • Encourage test teams to use multiple models of coverage (changing perspectives, using different testing techniques and tools) on a project instead of thinking of coverage as a singular thing
  • Utilize an effective gaming structure to augment reality and encourage people to have fun working hard at testing activities

I am encouraging testing teams to use this as a structure for organizing test execution to help make testing more engaging and fun. Feel free to add as many (or few) elements from video game quests as you see fit, and alter to match the unique personalities and goals of the people on your team. Or, study them and analyze how you organize your testing work for you and your teams. Does your structure encourage people to have fun and work hard at accomplishing something great? If not, you might learn something from how others have managed to get people to work hard in games.

Happy questing!