bug hunting, bugs, intermittence, unrepeatable bugs

Superbugs

My wife and I have friends and family in the health-care profession who tell us about “superbugs” – bacteria which are resistant to antibiotics. In spite of all the precautions, new technology and the enormous efforts of health care professionals, bugs still manage to mutate and respond to the environment they are in and still pose a threat to human health. In software development projects, I have encountered bugs that at least on the surface appear to exhibit this “superbug” behavior.

Development environments that utilize test-driven development, automated unit testing tools and other test-infected development techniques, in my experience, tend to generate very robust applications. When I see how much the developers are testing, and how good the tests are, I wonder if I’ll be able to find any bugs in their code at all. I do find bugs (sometimes to my surprise), but it can be much harder than in traditional development environments. Gone are the easy bugs an experienced tester can find in minutes in a newly developed application component. These include bounds conditions tests, integration tests and others that may not be what first come to mind to a developer testing their own code. However, in a test-infected development environment, most of these have already been thought of, and tested for by developers. As a tester, I have to get creative and inventive to find bugs in code that has already been thoroughly tested by the developers.

In some cases, I have collaborated with the developer to help in their unit test development efforts. <shameless plug> I talk about this more in next month’s edition of Better Software. </shameless plug> The resulting code is very hard for me to find bugs in. Sometimes to find any bugs at all, I have to collaborate with the developer to generate new testing ideas based on their knowledge of interactions in the code itself. The bugs that are found in these efforts are often tricky, time consuming and difficult to replicate. Nailing down the cause of these bugs often requires testers and developers pair testing. These bugs are not only hard to find, they are often difficult to fix, and seem to be resistant to the development efforts which are so successful in catching many bugs during the coding process. I’ve started calling these bugs “superbugs”.

It may be the case that certain bugs are resistant to the developer testing techniques, but I’m not sure if this is the case or not. I’ve thought until recently that these bugs also exist in traditionally developed code, but since testers spend so much time dealing with the bugs that test-infected development techniques tend to catch, they don’t have the time in the life of the project to find these types of bugs as frequently. Similarily, since they are difficult to replicate, they may not get reported as much by actual users, or several users may report the same problem in the form of several “unrepeatable” bugs.

Another reason *I* find them difficult to find might be due to my own testing habits and rules of thumb, particularly if the developer and I are working together quite closely. When we test together, I teach the developer some of my techniques, and they teach me theirs. When I finally test the code, both of our usual techniques have been tested quite well in development. Now I’m left with usability problems, some integration bugs that the unit testing doesn’t catch, and these so-called “superbugs”. Maybe the superbugs aren’t superbugs at all. Another tester might think of them as regular bugs, and may find them much more easily than I can because of their own toolkit of testing techniques and rules of thumb.

This behavior intrigues me none the less. Are we now able to find bugs that we didn’t have the time to find before, or are we now having to work harder as testers and push the bounds of our knowledge to find bugs in thoroughly developer-tested code? Or is it possible that our test-infected development efforts have resulted in a new strain of bugs?

ruby, tools, Watir

A Testing Win Using Ruby

Brian Marick has written lately about Exploratory Testing using Ruby. I decided to try this out on a web application using Ruby and the WTR(Web Testing with Ruby) IE Controller. I’m not at the level with Ruby yet where I can test an application using the Ruby Command interpreter like Brian does, but I thought I could write a short script and run it in an Exploratory manner. I decided to automate steps that would be tedious, time consuming and as a result, very much prone to error if run by a human. After a few minutes, I felt I had a script that was testing the application in a way I hadn’t tested manually. I ran it and watched it play back on my monitor.

When I ran it for the first time, the application responded in a strange manner. I had seen this behavior a couple of days previously when testing it manually, but couldn’t repeat it. When I re-ran the script, the behavior occurred again. I edited the script to hone in on the problem area and was able to repeat the problem each time. I edited it again to narrow down the cause, and I talked to the developer about the behavior. He had a suggestion to check a similar action, so I edited the script to try out the developer’s idea. In the end, I probably spent about a half hour on script development.

In a short period of time, I was able to track down a defect using Ruby that I had been trying to replicate manually. I’m confident that I would have spent a much longer time replicating the problem manually, and probably would have narrowed it down much later in the release when I was more familiar with the product. The test script that capitalized on the computer’s strengths over my human abilities run in a “what would happen if” scenario really paid off.

agile, agile testing, business

Presenting Testing Activities to Business Stakeholders

Brian Marick’s series on Agile Testing Directions begins with a test matrix that describes testing activities as “Business Facing”, “Technology Facing”, “Support Programming” and “Critique Product”. This resonated with me, but it wasn’t until he pointed out that in my pair work with developers I did both Business Facing and Technology Facing activities that this seemed to click. I think this matrix he has developed provides testers with a common language to identify and communicate these activities.

I recently did presentations to business stakeholders on testing activities in Agile projects. I’ve generally found it difficult to explain the testing activities I engage in to fellow testers, let alone business stakeholders. In one meeting, I thought of the test matrix and brought up the “Business Facing” testing and “Technology Facing” areas of testing while I was explaining how I test on Agile projects. People seemed to understand this, so I started working on it more.

I started thinking of the matrix rotated on its side with Technology Facing on the left and Business Facing on the right. Instead of “support programming”, I went with “support” to capture both areas. The “business support” would involve activities like setting up meetings with developers and business stakeholders after each development iteration to ensure that the working code is what the business expects, and to get people communicating. I also thought that business support would involve helping the business people with acceptance tests and things like that.

I initially thought of naming each quadrant of the matrix, but when explaining it to my wife Elizabeth, she said: “Why don’t you just put that in a tree diagram?” I did just that, and presented Agile Testing activities like this:

I felt that “technical testing” was a simple way to describe “technology-facing product critiques”, and “business testing” would describe “business-facing product critiques”. Keeping it simple seems to work well when communicating testing concepts to non-technical people.

I described some of the testing techniques in each area. For example, a technical testing activity I use involves collaboration with the developers to write tests that can be run in the absence of a user interface. This can involve adding tests to drive a layer of the application at the controller level. Once the developers make this area testable, we co-design and develop a test case. I can then own the test case and run it with different kinds of test data.

Under the programmer support activity, we can pair together to generate testing ideas. In a test-driven development environment, we can pair program to come up with tests that drive the code, or the tester can use a scripting language to write the tests for the developers.

Business people and technical people seemed to understand this tree diagram and the explanations I gave. I heard later that business stakeholders were starting to use this language in other contexts when they were talking about testing.

agile, agile testing

Testers on Agile Projects

When I began reading articles and books on Agile Development and attended lectures by well-known experts in the subject, I was impressed. This style of development resonated with me, combining what I had learned from the Open Source world and aspects of successful projects I had been on. As a believer in W. Edwards Deming’s 14 Points, I felt that Agile Methods seemed be addressing many of the same issues.

I welcomed Agile Development, and have championed it now that I’ve experienced it. I didn’t feel a threat to my job as a tester, but I knew things were going to change. The only thing that bothered me was a new impression towards testers that seemed to be emerging. The attitude sounded like: “we’re doing testing now thank you very much, so we don’t know where you will fit in Agile projects”. This attitude is changing, but the role of testers on Agile teams is still emerging.

Since I am a tester working on Agile projects, I want to share my experiences. When I first thought about Agile Development not needing dedicated testers, my intitial reaction was to think of a writer/editor analogy. As my experience with Agile projects grows, I am less confident of the need for a dedicated tester. I still think there is a need, but I have to be willing to admit that there may be no role for dedicated testers on Agile Projects, even if I want there to be. The role may be a diminished one, or as Brian Marick points out in his blog series on Agile Testing Directions, testers may be specialists called in for certain tasks like Security or Performance testing. However, the agile testing role might evolve and change into something completely different from what we know of as testing today. My goal is to see if and where I fit in on Agile projects. I’m relying on my development colleagues to provide me with honest feedback which I will try to share here.

One aspect of the tester role I’m exploring is pair testing with developers. I can support the programmers and help them generate test ideas, especially if they are using test driven development. We can also test together on a machine to test the product – simultaneously generating testing ideas. A senior developer noted that pair programming with a tester provides a developer with someone to help them generate tests. After all, the testers see the world in terms of tests, and the developers see the world in terms of patterns or objects or algorithms. One creates code while the other creates testing ideas. It sounds like a good match. It will be interesting to see how this type of testing pans out.

Thoughts on product development, management, design, mobile and other topics.