Category Archives: pair testing

agile, agile testing, context-driven testing, pair testing

Conventional Testers on Agile Projects – Getting Started Continued

Some of what you find out about agile methods may sound familiar. In fact, many development projects have adopted solutions that some agile methods employ. You may have already adjusted to some agile practices as a conventional tester without realizing it. For example, before the term “agile” was formally adopted, I was on more traditional projects that had some “agile” elements:

  • when I started as a tester, I spent a lot of my first year pair testing with developers
  • during the dot com bubble, we adopted an iterative life cycle with rapid releases at least every two weeks
  • one project required quick builds, so the team developed something very similar to a continuous integration build system with heavy test automation
  • developers I worked with had been doing refactoring since the early ’80s. they didn’t call it by that name, and used checkpoints in their code instead of xUnit tests that would be used now
  • in a formal waterfall project, we had a customer representative on the team, and did quick iterations in between the formal signoffs from phase to phase
  • one project adapted Open Source-inspired practices and rapid prototyping

These actions were done by pragmatic, product-focused companies who needed to get something done to please the customer. Many of these projects would not consider themselves to be “agile” – they were just getting the job done. The difference between them and an agile development team is that the agile methods are a complete methodology driven towards a certain goal rather than a team who has adjusted some practices to improve what they are doing.

Other conventional testers tell me about projects they were on that were not agile, but did agile-like things. This shouldn’t be surprising. The iterative lifecycle has been around for many years (at least back to the 1940s). There are a lot of methodologies that people have used, but not necessarily codified into a formal method within the iterative lifecycle as some agile champions have. A lot of what agile methods talk about isn’t new. Jerry Weinberg has said that methods employed on the Mercury project team he was on in the early ’60s looks to be indistinguishable from what is now known as Extreme Programming.1

Another familiar aspect of agile methods is the way projects are managed. Much of the agile management theory draws very heavily from the quality movement, lean manufacturing, and what some might call Theory Y management. Like the quality pundits of past, many agile management writers are once again educating workers about the problems of Taylorism or Theory X management.

What is new with agile methods, are comprehensive methodology descriptions that are driven from experience. From these practices, discplined design and development methodologies have improved rapidly, such as Test-Driven Development. Most importantly, a shared language has emerged for practices like “unit testing”, “refactoring”, “continuous integration” and others – many of which might have been widely practiced but called different things. This shared language helps a community of practice share and improve ideas much more efficiently. Common goals are much more easily identified when everyone involved is using the same terminology. As a result, the needs of the community have been quickly addressed by tool makers, authors, consultants and practitioners.

This has several implications for conventional testers that require some adjustments:

  • a new vocabulary of practices, rituals, tools and roles
  • getting involved in testing from day one
  • testing in iterations which are often 2-4 weeks long
  • an absence of detailed, formalized requirements documents developed up front
  • requirements done by iteration in backlogs or on 3×5 story cards
  • often, a lack of a formal bug-tracking system
  • working knowledge of tools such as refactoring and TDD-based IDEs, xUnit automation and continuous integration build tools
  • a team focus over individual performance
  • developers who are obsessed with testing
  • working closely with the entire team in the same work area
  • not focusing on individual bug counts or lines of code
  • less emphasis on detailed test plans and scripted test cases
  • heavy emphasis on test automation using Open Source tools

Some of these changes sound shocking to a conventional tester. Without a detailed requirements document, how can we test? Why would a team not have a bug tracking database? What about my comprehensive test plans and detailed manual regression test suites? Where are the expensive capture/replay GUI automation tools? How can we keep up with testing when the project is moving so quickly?

A good place to address some of these questions is: Lessons Learned in Software Testing: A Context-Driven Approach by Cem Kaner, James Bach and Bret Pettichord.

We’ll address some of these challenges in this series, as well as examples of testing activities that conventional testers can engage in on agile projects.

1 p. 48 “Iterative and Incremental Development: A Brief History”, Larman and Basili, 2003

bug reporting, bugs, pair testing, test cases, testing

Virtual Bugs

Mark McSweeny and I were talking about some of the challenges facing conventional testers on agile projects. One such challenge is what to do with bugs found during development, such as during Test-Driven Development when a tester and developer are pairing. It doesn’t seem fair to the developer to formally log bugs on a story before they have completed development on it. Many of them will be moot once the story is done, but some of them might squeak through. How do we keep track of them in a constructive way?

When they are pairing and developing tests, new test cases are added as they are generated, and the code is added to make them pass. However, sometimes when bugs are found during story development, the tester can overwhelm the developer and impede development progress. At this point, the developer and tester pair can split up, and the develoepr can pair with another developer and work towards completing the story. As a tester, what do I do with the bugs we discovered but couldn’t get the unit tests finished for?

On small teams, I will keep a running tab in my notes of these bugs. When the story is complete, I check my notes and test these scenarios first, and then log the ones that weren’t fixed during the story as “bug stories”. This is fine if there are a small number of developers, and I’m the only tester. This doesn’t scale well though. On slightly larger teams, I have also used a wiki to record these bugs which the other testers also reviewed and used. When they tested a story when it was complete, they would check the wiki first for any of these bugs. Any of these that weren’t addressed in development were then logged as bug stories or in a fault-tracking system. This can create classes of bugs which can create problems. It was hard to maintain the two systems, the wiki and the bug tracker.

As I was describing some of the problems I’ve come across with bugs found during story development, Mark cut through my usual verbosity with clarity, and said I was describing: “virtual bugs”. This is a lot more concise than my five minute hand waving explanation of this different class of bugs.

I have started calling the bugs found during story development “virtual bugs”. My question to other conventional testers on agile projects is: “How do you deal with virtual bugs”? Please share your experiences.

pair testing, TDD, test-driven development

Adriaan Brae Comments on TDD Postings

TDD developer Adriaan Brae writes:

I think the separation of tests into ‘generative’  and ‘elaborative’ is a very good concept. I can definitely feel the difference myself.

I like the generative tests much better – they are part of the creative process. I find elaborative
tests much harder to write. This is probably because I lack some of the QA skills you were talking  about in your Article about test pairing with developers. It sounds like that would be an excellent learning experience. I think it would be great to spend, say, 1-2 hours a week pair testing.

I am thinking it would be good to keep the pair testing times short. This would allow for cross-training, but also require that each individual go back and implement that training alone.  Longer times pairing might end up with each side dependent on the other for ideas.

(posted with permission)

pair testing, TDD, test-driven development

Collaboration and Human Dynamics

Brian Marick commented on some of my recent TDD pairing posts. He raises some good points:

…managing the human dynamics of the tester/programmer interaction is a key issue

Sometimes we forget that in the end, we are all people on development teams with different styles, habits, mannerisms and interact with others differently. Any pairing relationship is unique, and we have to work together towards a common goal. Some of the generalizations I make based on my own experience may not apply to others. I hope others can share their experiences to help broaden our knowledge.

Brian also comments on critical thinking and disruption:

One can be a critical thinker but not a critical speaker. One can post-process the thoughts and use them to guide progress in a non-disruptive way. That’s not traditionally been the role of testers: it’s been their job to have critical thoughts, to evaluate the product against them, to report the results clearly and dispassionately, and to let someone else decide what to do about it. In an agile project, it seems to me that the tester’s responsibility has to go a bit further.

We still have a lot of work to do as testers when working on Agile projects, but the exploration, inquiry and discovery is a lot of fun. It will be interesting to look back on what we are doing now in several years and see how far we’ve come.

pair testing, TDD, test-driven development

Testers, TDD and Disruption

James Bach writes:

“It was interesting to me that the developers who came to the Agile Fusion conference initially professed interest in pairing with testers, but their enthusiasm seemed to wane as they learned that testers are critical thinkers– and critical thinkers are disruptive. When I paired with programmers, I mostly adopted a programmer mindset, because I didn’t want to freak them out.

On the context-driven testing mailing list, James elaborates on the disruptive nature of testing:

Try this experiment. Stand behind someone who is writing a memo. Every time they mistype something, tell them. Also tell them immediately about any grammar mistakes. Also comment on their writing style. Periodically stop them and question the whole purpose of the memo. Speak any risks you think of along the way.

I think you will find that the person you are “helping” gets annoyed. Their train of thought will be broken. The same kind of thing happens if we sit with programmers who are writing code. The same kind of thing happens in requirements and design meetings. There is a value to proceeding speculatively and creatively for protected periods of time, without being assailed by critical thoughts.

(posted with permission)

This sentence really got me thinking about tester/developer pairing, and how we can add value to the development process: “There is a value to proceeding speculatively and creatively for protected periods of time, without being assailed by critical thoughts.” I’m not sure that we could or even should pair constantly with a programmer, but there are times when it seems to work very well.

pair testing, TDD, test-driven development

More on Test Driven Development:

William Wake writes:

At least half the time in TDD, I feel like I’m doing the design part. I talk of generative tests, that yield design, and elaborative tests, that test it “the rest of the way”; the generative tests are pushing the design forward, and it’s somewhat of a distraction to stop and do elaborative tests too soon.

(posted with permission)

This was in response to a note by me describing some of the challenges facing programmers and testers pairing together in TDD activities. I really like the terms: “generative tests” that yield design, and “elaborative tests” that yield more verification or critiquing tests.

As a tester pairing in TDD efforts, I found some of our test development to be a natural fit, while some of it seemed more difficult. When William emailed me with the above comment, I had an “aha!” moment. The natural fit I’d encountered was in test idea generation when we worked on elaborative test development. The generative test development was new to me. When I was the one driving developing new functionality, I found it hard to write tests for classes and methods that hadn’t been developed yet. I clearly need to do more tester developer pairing in TDD activities, and need to work more on TDD when I’m coding. As my pairing partner says, the only key is practice.

Another area of difficulty is the disruptive nature of testing. Some developers might appreciate the disruption as they create code, but others have told me that they wouldn’t want a tester there during what William calls the generative test development. It seems that developers need a protected time of creativity and development to get something together that can be tested. Having a tester interrupting them in this process may not be helpful. Furthermore, when developers spend time making interfaces more testable, the dedicated tester may not be the best person for them to pair with. The tester may not add a lot of value until the interface has been refactored. At that point, they could step in and see if it is testable or not.

Testers who have strong development and testing skills (and who understand and have insight into TDD, bad coding smells, interface discovery, etc.) should fit well into both categories – generative and elaborative. Most testers I might wager, will fit in better in the elaborative test development aspect of TDD. We are used to Business-Facing Product Critiques, and test idea generation activities. Generative test development will probably be new to many dedicated testers. If you want to pair with a developer doing TDD, this might be something to keep in mind.

It may be a good practice to have developer-tester pairing during the elaborative test development activities of TDD, and not during the generative test development. More testers and developers pairing and reporting their experiences may help us learn if we can do this well.

I encourage testers and developers to pair together, and feel free to let me know how you do with TDD pairing. What did you struggle with? What worked well?