Category Archives: test-driven development

agile, post-agilism, process fusion, TDD, test-driven development

Interesting Posts on Agile Challenges

Scrum Challenges

A couple of posts that describe how many teams are flailing and failing with Scrum:

I’ve observed similar patterns. However, my gripe with an oft-heard Agilist response: “they are failing because they just don’t get Agile” or “if Agile failed for you, it’s your (or management’s) fault” smacks of blaming the victim(s). An emerging response that I support is:

Maybe a pure form of ‘Agile’ isn’t appropriate for that team, in their context.

(Time for some Process fusion?) Philippe Kruchten has a great talk on this: Situated Agility – Context Does Matter, a Lot.

It’s also very difficult dealing with the scorched earth of a failed Scrum project after the Scrum trainers have left and the team is struggling on their own, feeling humiliated. “Are we the only ones failing? Why do we hear all these wonderful reports of how Scrum would solve all process ills? What’s wrong with us? We’re trying…” It’s hard to get them to retain the good practices they learned from Scrum and to encourage them not to throw out everything and return to a system that wasn’t working before either, but is more familiar, so it feels safer.

Rumours of Practice

TDD – more of a rumour of practice than actual practice? (much like some of what is described in the two posts above.)

Roy Osherove: Goodbye mocks, Farewell stubs

My own observations about these and other Agile practices being more of a rumour of practice than an actual practice leads me to wonder if Agile practices are another flavour of a bubble. Time will tell, but some of the behavior is troubling. It still galls me that many blindly parrot TDD as an un-alloyed good practice, instead of TDD as another tool to think about using, particularly when people might be basing their conclusions off of rumours, rather than personal experience. This irrational exhuberance is one reason why stock markets ramp up on empty speculation, real estate prices boom on over-valued properties (using mortgages that people can’t afford to pay back), and tulips are bought with abandon. (At least you can plant your tulip bulbs and enjoy beautiful flowers when the bubble bursts. What do you do with your old un-maintainable tests?)

My advice to those who may be struggling? Don’t worry about being “Agile”, (particularly if you’re trying and failing) and worry about providing value. That’s what really matters anyway. (That, and enjoying your work.) Providing value to the users of your software, and valuing the people you work with is important. Value, coupled with the skill and interest level of the team members, will trump methodology in the long run.

exploratory testing, TDD, test-driven development

Test-Driven Development and Exploratory Testing

I’ve spent time working on teams practicing Test-Driven Development. Since I am a tester, and the practice has the word “test” in it, I’d only been on Agile teams for a short time before I was enlisted by programmers to help them out with test idea generation during Test-Driven Development.

At first, it felt a bit foreign, particularly when the programmers were using tools and programming languages I wasn’t that familiar with. In fact, it didn’t really feel like testing at all; it seemed like a programming style that was heavily influenced by object-oriented programming techniques. Once in a while though, it really felt like testing, the kind of testing I was familiar with. The programmers would turn to me to get me to drive when pairing, and were acutely interested in test idea generation. This was where using exploratory testing techniques, particularly rapid testing techniques really shone. Since we were developing new features quickly, the faster I could generate useful and diverse testing ideas, the quicker we could generate tests, strengthen the code, and improve the design. When we went back to developing a new feature though, I again felt out of place, particularly when I distracted and frustrated development at first.

I decided that the only way to understand Test-Driven Development was to immerse myself in the practice, so I paired with an expert programmer who had experience teaching TDD. I describe some of my experience in this article: Test-Driven Development from a Conventional Software Testing Perspective, Part 1. I found that I was still struggling with parts of TDD, and I felt like I would contribute very little at some points, and contribute a great deal at others. William Wake cleared this up for me. He explained two phases of Test-Driven Development: the generative phase and elaborative phase.

Generative tests are the first tests that are written, primarily about design. They’re “shallow sweep” tests that yield the initial design. Elaborative tests are deeper, more detailed tests that play out the design and variations.

Transitions between these phases can occur at any time, which is why I hadn’t noticed the change between my testing idea generation providing more or less value. At some points I would actually distract the programmer with too many test ideas, and at others, they would sponge up everything I could generate and would want more. I had to learn to watch for patterns in the code, for patterns in our development, and to work on my timing for test idea generation.

When we were clearly in the elaborative phase, our testing was much more investigative. When we were in the generative phase, it was much more confirmation-based. In the elaborative phase, we were less concerned with following TDD practice and generating code than we were with generating test ideas and seeing if the tools could support those tests or not.

To understand more, I decided to do TDD myself on a test toolsmith project. I describe those experiences in more detail here: Test-Driven Development from a Conventional Software Testing Perspective, Part 2. This experience gave me more insight into the testing side of TDD.

One day while developing in the generative phase, I was suddenly reminded of pre-scripting test cases, a practice I did religiously in the beginning of my career before fully embracing investigative exploratory testing. Instead of recording tests in a document or in a test case management system, I was pre-scripting them in code. Instead of waiting for a build from the development team to run the scripts, I was running them almost immediately after conceiving them and recording them. I was also recording them one at a time and growing a regression test suite at the same time as creating scripts. I realized that an assertion in my tests was analogous to the “Expected Results” section of a test script, an idea popularized by Glenford Myers in his book “The Art of Software Testing”, published in 1979.

When I moved to the elaborative phase and used different kinds of testing, glaring weaknesses in my test-driven code became apparent. In the elaborative phase, I didn’t care about expected results nearly as much as the unexpected results. The testing here was investigative, and if I pulled off my programmer hat long enough, I could go quite deeply with exploratory testing techniques. When a weakness was exposed, that meant I had missed test(s) in the generative phase, and I would add new test(s) to cover those missed cases to my unit testing code.

The main difference between what I was doing in code and what I would normally do with a program was the interface I was using for testing (I was using code-level interfaces, not the GUI), and I was the one fixing the code whenever I found a problem. Since the tests were highly coupled, and expressed in the same language as the program code, test case execution and maintenance wasn’t as disconnected.

I was also less concerned with my test case design up-front in the elaborative phase. It was much more important to use the programming tools to allow me create different kinds of test cases quickly, or on the fly. Once I found something interesting, and decided to record a test as a regression test, I would convert the test into a proper unit test with setup method, a test method with an assertion, and a teardown method.

Now that I’ve had some experience myself with TDD, and have worked with skilled practitioners, I have a better idea of what it is, and how it can relate to exploratory testing and scripted testing. There are a lot of elements of pre-scripting test cases in the generative phase, or scripted testing, including some of the drawbacks of that practice. Over several releases, pre-scripted test cases tend to have an increasing maintenance cost. (Note, Story-Test Driven Development is another form of pre-scripted, confirmatory testing which can exacerbate the maintenance problems we had with pre-scripted tests in pre-Agile days. They can be extremely maintenance intensive, particularly over long projects, or over multiple releases, robbing resources from new feature development and investigative testing activities.)

The elaborative phase was a natural fit for exploratory testing, and a great place for non-technical testers to jump in and help generate testing ideas. The challenge with this is figuring out what is good enough, or when to stop. Investigative testing is only limited by the imagination of the people involved in the activity. When there are two of you, a tester and a developer, it is easy to quickly generate a lot of great ideas.

It is also common for TDD practitioners to skip the elaborative phase for the most part. In this case, TDD is almost exclusively about design, with little testing going on. The “tests” are mostly a type of requirement, or examples. I’ve also seen some programmers abandon elaborative tests because they failed, while the generative tests passed. Others just didn’t do much elaboration at all, partly because they were so focused on velocity that they wanted to get a story written as quickly as possible.
In other cases, the phases were more blurred, particularly with programmers who felt that testability improved their design, and were open to a lot of different kinds of testing techniques. In fact, because the feedback loop with pre-scripted generative tests is so tight, a programmer who is experiencing flow may be utilizing exploratory thinking while developing new features.

I talk more about some of my opinions on TDD from a tester’s perspective here: Test-Driven Development from a Conventional Software Testing Perspective, Part 3. This reflective article spawned some interesting ideas, and while recording the skepticism and cautionary points, I had a lot of ideas for new areas of work and research.

Several TDD practitioners have approached me about exploring the elaborative side of TDD with skilled exploratory testing to address some of the problems they were having with their code. One area I consistently hear about are tests that yield a brittle design. Once the code is put into a production-like environment, testing revealed a lot of errors they felt would have been better addressed during development. The programmers felt that with more exploratory testing with more elaboration would combat this. Using more elaborative tests, more often, might be one way to maintain exploratory thinking in an emerging new design. Instead of a narrow focus of “write a test, run it, when it fails, write code so it passes”, they felt that exploratory testing would expand their exploratory thinking while programming, with a better overall design as the desired result. Rapidly transitioning from generative design thinking to elaborative, exploratory testing is one area to explore.

This intersection of investigative testing and programming holds a lot of promise for practice and research. I encourage interested testers and programmers to work on this together. I’ve certainly learned a tremendous amount by working in this space, and the testing community has benefited from the subsequent improvement in practices and tools. In fact, recently as I was rapidly writing web services tests in Java/JUnit, I recalled how much work it was eight years ago for a testing department to create a custom test harness in Java before we could even write tests.

agile, extreme programming, post-agilism, TDD, test-driven development

Who Do We Serve?

Bob and Doug are programmers on an Extreme Programming team. Their customer representative has written a story card for a new feature the company would like to see developed in the system to help with calclulations undertaken by the finance group. Currently, they have spreadsheets and a manual process that is time-consuming and somewhat error prone. Bob and Doug pair up to work on the story. They feel confident – they are both experienced with Test-Driven Development, and they constantly use design patterns in their code.

They sit at a machine, check out the latest version of the code from source control, and start to program. They begin by writing a test and forming the code test-first, to meet the feature request. Over time, they accumulate a good number of test cases and corresponding code. Running their automated unit test harness gives them feedback on whether their tests passed or not. Sometimes the tests fail sporadically for no apparent reason, but over time the failures appear less frequently.

As they are programming and have their new suite of automated test cases passing, they begin refactoring. They decide to refactor to a design pattern that seems suitable. It takes some rework, but the code seems to meet the pattern definition and all the automated unit tests are passing. They check the code in, and move on to a new story.

The next day, they demonstrate the feature to the customer representative, but the customer representative isn’t happy with the feature. It feels awkward to them. Bob and Doug explain what pattern they were using, and talk about facades, persistence layers, inversion of control, design patterns, and demonstrate that the automated unit tests all pass by showing the green bar in their IDE. The customer representative eventually reluctantly agrees with them that the feature has been implemented correctly.

Months later, the finance department complains that the feature is wrong almost half of the time, and is causing them to make some embarrassing mistakes. They aren’t pleased. Wasn’t XP supposed to solve these kinds of problems?

Looking into the problem revealed that the implementation was incomplete – several important scenarios had been missed. A qualitative review of the automated tests showed that they were quite simple, worked only at the code level, and no tests had been done in a production-like environment. Furthermore, no thought had been put into the design for the future when the system would be under more load due to more data and usage. “That isn’t the simplest thing that could possibly work!” they retorted. However, the team knew that data load would greatly increase over the coming months. If they weren’t going to deal with that inevitability in the design right then, they needed to address it soon after that. They never had, and blamed the non-technical customer representative for not writing a story to address it.

Finally, when the customer representative tried to express that something wasn’t quite right with the design, they were outnumbered and shot down by technical explanations. One representative against the six technical programmers on the team who were using jargon and technical bullying didn’t stand much of a chance. After the problem was discovered in production, Bob and Doug admitted that things felt a little awkward even to them when they initially implemented the code for the story, but since they were following practices by the book, they ignored signs to the contrary. The automated tests and textbook refactoring to patterns had given them a false sense of security. In fact, the automated tests trumped the manual tests done by a human, an actual user, in their eyes.

What went wrong? Bob and Doug measured their project success on their adoption of their preferred development practices. They worried more about satisfying their desire to use design patterns, and to follow Test-Driven Development and XP practices than they did about the customer. Oh sure, they talked about providing business value, worked with the customer representative on the team and wrote a lot of automated unit tests. The test strategy was sorely lacking, and their blind faith in test automation let them down.

Due to this incident, the development team lost credibility with the business and faced a difficult uphill battle to regain their trust. In fact, they had been so effective at marketing these development methods as the answer to all of the company’s technical problems that “Agile” was blamed, and to many in the business, it became a dirty word.

Sadly, this kind of scenario is far too common. On software development teams, we seem especially susceptible to desire silver-bullet solutions. Sometimes it seems we try to force every programming problem into a pattern whether it needs it or not. We suggest tools to take care of testing, or deployments, or any of the tasks that we don’t understand that well, or don’t enjoy doing as much as programming. We expect that following a process will prevail, particularly a good one like Extreme Programming.

Through all of this we can lose sight of what is really important – satisfying and impressing the customer. In the end, the process doesn’t matter as much as getting that result. The process and tools we use are our servants, not our masters. If we forget that and stop thinking about what we’re trying to solve, and who we are trying to solve it for, the best tools and processes at our disposal won’t help us.

TDD, test automation, test-driven development

TDD in Test Automation Projects

I’ve written before about pairing with developers during Test-Driven Development(TDD), and I’ve been fortunate to work with very talented TDD developers who are apt to teach. I’ve learned a lot, and decided to try TDD in a programming role. Recently, I’ve taken off my tester hat and started doing TDD myself with test automation projects. I’m not completely there yet – I often need to do an architectural spike first when I’m developing something new. Once I have figured out a general design, or have learned how a particular library works, I throw away the spike code and start off development by writing a test. I then write enough code to get the test to pass, write a new test, add new code and repeat until the design is where I need it to be.

So what does this gain in test automation projects? I’m loathe to have test cases that are so complex that they themselves require testing. If our test cases are so complex that they are causing problems themselves, that’s a test design smell. However, there are other kinds of software in our automation projects than just the test cases. In automation frameworks, we need special libraries, or adaptors, or a way to access an application we need to test, and all sorts of utilities that help us with automation. Since these utilities are still software, they are subject to the same problems that any other software development effort is. Sometimes there is nothing more frustrating than buggy test code, so we need to do what we can to make it as reliable as possible.

In my own development, I’m finding a lot of benefits doing TDD. My designs improve, because if they aren’t testable, I know there is a problem. When I make my code testable, it suddenly becomes more usable and more reliable. Too often testers aren’t given time to refactor test code. I’ve found that refactoring usually starts when technical debt in a test harness and custom test library are starting to interfere with productivity. When there are no unit tests for custom test library code, it can involve a few minutes to change the code, and several hours testing it. Having a safety net of unit tests helps immensely with refactoring. You can refactor your test code with greater confidence, and when it’s done consistently with automated unit tests, with much greater speed. It just becomes a normal part of development.

Recently, I’ve found several bugs in my test library code in the elaborative phase of TDD that I didn’t find testing manually. The opposite is also true, I tested libraries I was developing manually and found a couple of bugs that my unit tests didn’t uncover. The TDD-discovered bugs required design changes that helped my design immensely. The tests guided the design into something different (and better) than what I had in my head when I started. However, after a couple of days of only running the unit tests to satisfy the “green bar”, I found a big hole that was only uncovered by using the library the way an end user would. The balance of testing techniques is helpful. I have also adopted a practice of pairing a positive test with a negative test, a technique I learned from John Kordyback. If I do an assert_equal for example, I also do an assert_not_equal (using test::unit style). This has really come in handy at times where one assertion would work, but the other would fail.

TDD may not be for everyone, but I find it is a nice complement to other kinds of testing we can do. For my own work, it seems to suit the way I think about development. Even if the test cases I develop while programming are trivial and small, there is strength in numbers, and writing and using them helps assuage that tester voice in the back of my head that comes out when programming. I encourage other conventional testers who work on automation projects to give it a try. You may find that your designs improve, and you have a safety net of automated tests to give you more confidence in your automation code, especially when you need to enhance it down the road. At the very least, it helps you gain an appreciation and helps you communicate when working with TDD folks on a project.

TDD, test-driven development

TDD – A Fifth School of Software Testing

Bret Pettichord has a presentation on the Four Schools of Software Testing. He covers the schools of testing identified by Cem Kaner, James Bach and himself: Analytic, Factory, Quality and Context-Driven. (For a brief time, Bret had renamed “Factory” school to the “Routine” school, but has since reverted back to “Factory”.) This breaking down of popular testing ideas into schools is a thought-provoking concept, and just reading his presentation notes should get testers thinking about what they do on projects. “What school do I identify with?” a reader might ask themselves.

Cem Kaner has identified Test-Driven Development as another school of software testing. He talks about TDD as a big force in software development from the past decade in this paper: The Ongoing Revolution in Software Testing. On the context-driven testing mailing list, Cem Kaner explained why he identifies TDD as a testing school:

…TDD isn’t completely inside my [schools of software testing] paradigm. At [this] point, I ask:

(a) Is it coherent enough, popular enough, and looked to for guidance–a foundation for teaching about testing– enough to be called a school?

I think this is a clear yes. The fact that courses in TDD exist and are fashionable demonstrates that it’s a source of knowledge, inspiration, and guidance (which is at the essence of what I think of in a school). And there’s clearly an active group of collaborators who jointly develop and push the ideas.

(b) Is it testing?

It’s not inconsistent with my definition of testing, and a lot of people who advocate it think it is testing. OK, I accept it as testing.

Is it a unified field theory of testing? No. But neither is anything I’ve done. TDD focuses on problems that many of us haven’t focused on before, and sheds remarkably little light (or interest) on problems that many of us take very seriously. To the extent that it thinks its problems are the primary interesting problems, it asserts itself as a paradigm–elucidates some key issues and puts on blinders with respect to the rest. But I have my own blinders, so the mere fact that TDD is blind to some of the problems I find most interesting can’t be a basis for invalidating it as a school of testing. It can only be a basis for rejecting it as the driving philosophy of my school of testing.

Along with techniques, TDD advocates often espouse a set of values, a set of broader ideas about development, an attitude about people and how they should use their time and how they should be held accountable, an attitude about the use of tools, about the types of skills people doing this work should possess and how they should/could grow them–for sure, these are fuzzy sets, not everybody adopts all of them. It’s all of this stuff that separates schools — it’s also this type of stuff that gives context/interpretation for the use of the tools/techniques and the directions of advancement of the use and enhancement of these tools/techniques.

I think these illustrate an approach to testing that includes some deep and careful thinking, a rich set of practices, and attention to a lot of basic issues. I think I see different answers than I would expect from the factory school. I think it is much more brain-engaged than factory school, and much more about the cognition of the test creator than about the detail of procedures to be followed by the test runner.

So, I see a thoughtful, coherent view. A lot of practitioners. A body of work and advocates that guide thinking, practice and the teaching of thinking about how to envision, research, and practice testing. Looks like a school to me. And a welcome one.”

(posted with permission)

I agree with what he has said. In my own forays into pair testing with test-driven developers and learning how to do TDD from skilled practitioners, I have met people who are really dedicated to testing. They tend to seek out conventional testers who are also passionate about testing. This intersection of schools can sometimes have some confusing results at first.

I’ve witnessed confusion when conventional testers and TDD testers are working together. That prompted a blog post on vague words used in testing. Those I would characterize as Quality school testers often feel responsible for the TDD process and unit tests even though they aren’t programmers and haven’t tried TDD themselves. I have heard of TDD testers and Quality school testers attending the same meeting, using the same language, agreeing to move forward together, and then independently moving in completely different directions. In one case, a TDD developer and I spent several sessions where I “translated” the language that the Quality school folks were using. He started changing his use of the shared terminology by using synonyms to improve communication and reduce confusion. They managed to work things out reasonably well.

Is it helpful to make distinctions of testing schools like this? Here is one area where it certainly helps: when communicating testing concepts. When a word can mean different things depending on how a practitioner defines their role, it helps to understand where they are coming from. Cem Kaner has provided further insight on TDD, and we can use that to enhance our communication and collaboration with developers who are honing their own testing craft.

interfaces, TDD, test-driven development

Expressing Interfaces

Mark McSweeny comments on Discovering Interfaces:

While testers work to discover interfaces in programs, developers work to express interfaces. This can be through developing an API, an interface for unit testing, all the way down to the data structures and algorithms you use. Well designed interfaces are testable interfaces, so a tester can tell you whether the interfaces you have expressed are testable or not.

The meeting of the roles through the program is interesting to me. The two meet in the middle of creation and inquiry. Sometimes as testers we forget that there are people behind what we are testing who have poured a lot of effort into the interfaces we are testing. The more testers can supply positive feedback to the developers, the more confidence they can get from their development efforts.

While testers may be satisfied when a program does not fail after they have attempted to make it break, a developer is satisfied when they have solved a problem using technology. Together, there is expression, inquiry, discovery, feedback, communication, and collaboration.

What is interesting with methods such as TDD is that they attempt to combine interface expression and discovery. One attempts to express an interface while also attempting to critique it.

pair testing, TDD, test-driven development

Adriaan Brae Comments on TDD Postings

TDD developer Adriaan Brae writes:

I think the separation of tests into ‘generative’  and ‘elaborative’ is a very good concept. I can definitely feel the difference myself.

I like the generative tests much better – they are part of the creative process. I find elaborative
tests much harder to write. This is probably because I lack some of the QA skills you were talking  about in your Article about test pairing with developers. It sounds like that would be an excellent learning experience. I think it would be great to spend, say, 1-2 hours a week pair testing.

I am thinking it would be good to keep the pair testing times short. This would allow for cross-training, but also require that each individual go back and implement that training alone.  Longer times pairing might end up with each side dependent on the other for ideas.

(posted with permission)

pair testing, TDD, test-driven development

Collaboration and Human Dynamics

Brian Marick commented on some of my recent TDD pairing posts. He raises some good points:

…managing the human dynamics of the tester/programmer interaction is a key issue

Sometimes we forget that in the end, we are all people on development teams with different styles, habits, mannerisms and interact with others differently. Any pairing relationship is unique, and we have to work together towards a common goal. Some of the generalizations I make based on my own experience may not apply to others. I hope others can share their experiences to help broaden our knowledge.

Brian also comments on critical thinking and disruption:

One can be a critical thinker but not a critical speaker. One can post-process the thoughts and use them to guide progress in a non-disruptive way. That’s not traditionally been the role of testers: it’s been their job to have critical thoughts, to evaluate the product against them, to report the results clearly and dispassionately, and to let someone else decide what to do about it. In an agile project, it seems to me that the tester’s responsibility has to go a bit further.

We still have a lot of work to do as testers when working on Agile projects, but the exploration, inquiry and discovery is a lot of fun. It will be interesting to look back on what we are doing now in several years and see how far we’ve come.

pair testing, TDD, test-driven development

Testers, TDD and Disruption

James Bach writes:

“It was interesting to me that the developers who came to the Agile Fusion conference initially professed interest in pairing with testers, but their enthusiasm seemed to wane as they learned that testers are critical thinkers– and critical thinkers are disruptive. When I paired with programmers, I mostly adopted a programmer mindset, because I didn’t want to freak them out.

On the context-driven testing mailing list, James elaborates on the disruptive nature of testing:

Try this experiment. Stand behind someone who is writing a memo. Every time they mistype something, tell them. Also tell them immediately about any grammar mistakes. Also comment on their writing style. Periodically stop them and question the whole purpose of the memo. Speak any risks you think of along the way.

I think you will find that the person you are “helping” gets annoyed. Their train of thought will be broken. The same kind of thing happens if we sit with programmers who are writing code. The same kind of thing happens in requirements and design meetings. There is a value to proceeding speculatively and creatively for protected periods of time, without being assailed by critical thoughts.

(posted with permission)

This sentence really got me thinking about tester/developer pairing, and how we can add value to the development process: “There is a value to proceeding speculatively and creatively for protected periods of time, without being assailed by critical thoughts.” I’m not sure that we could or even should pair constantly with a programmer, but there are times when it seems to work very well.