All posts by jonathankohl

ethics, testing

Traits of Good Testers

People frequently ask me what to look for in potential candidates when hiring testers. Michael Hunter “The Braidy Tester” has a great post on The Hallmarks of a Great Tester. He has some good thoughts in this post. I recommend checking it out.

I would add the traits honesty and integrity, as well as someone who has courage to the list. Testers seem to end up in situations where ethical concerns may arise. A tester needs to know what their ethics are, and to stand for them. As Lesson 227 of Lessons Learned in Software Testing says:

You don’t have to lie. You don’t have to cover up problems. In fact, as a tester, the essence of your job is to expose problems and not cover them up. You don’t have to, and you never should compromise your integrity. Your power comes from having the freedom to say what’s so, to the people who need to hear it. If you compromise your reputation for integrity, you’ll weaken one of the core supports of your power.*

Discovering these traits in a potential candidate can be difficult, but there are effective ways of interviewing to tell if someone is a great tester or not. Check out Johanna Rothman’s Hiring Technical People blog for more information on hiring techies.

*Excerpted from p.200 of Lessons Learned in Software Testing.

extreme programming, ruby, test automation, XP

Gerard Meszaros on Computer Assisted Testing

At last night’s Extreme Tuesday Club meeting, Gerard Meszaros had a great way of describing what I do using automated scripts with Ruby combined with exploratory testing. (See my blog post about this here: Test Automation as a Testing Tool.)

As I was describing using a Ruby script to automate some parts of the application I’m testing, and then taking over manually with exploratory testing, Gerard said:

Sounds like you are using your Ruby program to set up the test fixture that you will then use as a starting point for exploratory testing.

I hadn’t thought of what I was doing in this way before, and I like the idea. It is a great way to succinctly explain how to complement the strengths of a testing tool and a brain-engaged tester.

Check out Gerard’s Patterns of xUnit Test Automation site for more of his thoughts on testing.

agile, user profiles, user stories

Dehumanizing the User

As Mike Cohn points out in Advantages of User Stories for Requirements, writing requirements documents with the phrase: “the system shall” causes us to lose focus on the user. We aren’t writing something to satisfy the system, we are writing something to satisfy the needs of the user, the customer.

Maybe this is another encapsulation endeavor. Is using this kind of language an implicit (or unconscious) attempt to hide the complexity of working with people into a simplified interface called “the system”? As we encapsulate the needs and wants of the user within this black box, is this system an accurate representation of the customer’s original knowledge? Does it meet their needs? How can we tell?

What do we lose when we shift focus from being customer-obsessed to talking more in terms of the system? It’s one thing to satisfy a system, and quite another to satisfy the customer. Language is very important, and if the “user” or “customer” are being talked about less on a project, there is a danger that we may think about the customer and their needs less. Furthermore, if we refer less to the user and more to the system we are developing when looking at requirements and needs, we dehumanize the user. The context of satisfying a requirement at the system level and at the customer level can be two very different things. Customers worry about how usable the software is, sometimes more so than the fact that the requirement exists in the system concept.

If we dehumanize the user, why shouldn’t the user dehumanize us? Instead of a development relationship based on collaboration focused on meeting the needs of the customer, we run the risk of becoming commodities to each other. What’s to stop the customer from treating us as a commodity if we treat them as one? If communication breaks down and we dehumanize each other, how can we meet the needs of the customer? How can they be sure their needs are met? Can we trust each other?

Even when we use methodologies that are based on collaborating with the customer, we should be careful not to force them into a position they aren’t comfortable with. A key for overcoming technical bullying is effective communication. We need to treat the customer as a human being who has needs and feelings like we do, and communicate with them. With proper two-way communication, we can make sure they understand what we are doing, and enable them to ensure we are meeting their needs. Words are important, and listening to the customer is even more important. If we really listen, we can find out what the customer needs and develop that. The customer needs to be satisfied, not necessarily the system. Our language should reflect this.

test cases, tools

Dehumanizing Software Testing

I was talking with Daniel Gackle, a software developer and philosophical thinker, about developing software and systems. Daniel mentioned in passing that we often forget about the humanity in the system – the motivation of the human who is using the tool, or doing the work. This comment resonated with me, particularly with regards to some software testing practices.

Sometimes when we design test plans, test cases, and use testing tools, we try to minimize the humanity of the tester. The feeling might be that humans are prone to error, and we want solid, repeatable tests, so we do things to try to minimize human error. One practice is the use of procedural test scripts. The motivation often is that if the test scripts are detailed enough, anyone can repeat them, with a benefit that there will be less chance for variability or error. The people who write these tests try to be as detailed as possible with one motivation being that the test must be repeated the same way each time. It is not desirable to have variation because we want this exact test case to be run the same way every time it is run.

This type of thinking spills over into test automation as well. There is a push to have tools that can be used by anyone. These tools take care of the details, all a user needs to do is learn the basics and point and click and the tool does the rest for us. We don’t need specialists then, the tool will handle test case design, development and execution.

Without going into the drawbacks that both of these approaches entail, I want to focus on what I would call the “dehumanizing of software testing”. When we place less value on the human doing the work, and try to minimize their direct interaction with software under test, what do we gain, and what do we lose?

In The Dumbing Down of Programming, Ellen Ullman describes some of what is lost when using tools that do more work for us. Ullman says:

the desire to encapsulate complexity behind a simplified set of visual representations, the desire to make me resist opening that capsule — is now in the tools I use to write programs for the system.

It is today, (as it was in 1998 when this article was written), also in procedural test cases (or scripts) and in the popular “capture/replay” or event recording testing tools. Both seek to “encapsulate complexity” behind an interface. Both practices I would argue, lead to dehumanizing software testing.

Ullman mentions that the original knowledge is something we give up when we encapsulate complexity:

Over time, the only representation of the original knowledge becomes the code itself, which by now is something we can run but not exactly understand.

When we don’t think about the human interaction that I believe is so valuable to software testing, we need to be aware of what we are giving up. There is a trade-off between having interactive, cognitive testing, and testing through interfaces where the complexity is hidden from us. There are cases where this is useful, we don’t always need to understand the underlying complexity of an application when we are testing against an interface. However, as Ullman says, we should be aware of what is going on:

Yet, when we allow complexity to be hidden and handled for us, we should at least notice what we’re giving up. We risk becoming users of components, handlers of black boxes that don’t open or don’t seem worth opening.

When is hiding the complexity dehumanizing software testing, and when is it a valuable practice? I would argue that as soon as we encapsulate complexity to the point that the tester is discouraged from interacting with the software with their mind fully engaged, we are dehumanizing software testing. If a tester is merely following a script for most of their testing work, they are going to have trouble being completely mentally engaged. If a testing tool is trying to do the work of test design and execution, it will be difficult for the tester to be completely engaged with testing. I believe we miss out on their skills of observation, inquiry and discovery when we script what the tester needs to do and they do most of their work by rote.

The important thing to me is to understand what the trade-offs are before we adopt a testing practice. When we forget the humanity of the system, we will probably get results other than what we intended. We need to be aware of how people impact the use of tools, and how people work on projects, because after all, it’s still the people who are key to project success.

agile, agile testing, test automation

“Test” and “generalist” are vague words

Brian Marick has posted his Methodology Work Is Ontology Work paper. As I was reading it, I came across this part on page 3 which puts into words what I have been thinking about for a while:

… there are two factions. One is the “conventional” testers, to whom testing is essentially about finding bugs. The other is the Agile programmers, to whom testing is essentially not about bugs. Rather, it’s about providing examples that guide the programming process.

Brian uses this example as an ontology conflict, but has provided me with a springboard to talk about vague words. Not only conflicts, but vague terms can cause breakdowns in communication which can be frustrating to the parties involved.

Tests

When I was studying philosophy at university, we talked a lot about vague words, which are words that can have more than one meaning. We were taught to state assumptions before crafting an argument to drive out any vague interpretations. Some work has been done in this area with the word “testing” on agile projects. Brian Marick has talked about using the term “checked examples“, and others have grappled with this as well.

“Test” is a term that seems to work in the agile development context, so it has stuck. Attempts at changing the terminology haven’t worked. For those of us who have testing backgrounds and experience on agile teams, we automatically determine the meaning of the term based on the context. If I’m talking to developers on an agile team and they use the word “test”, it often means the example they wrote during TDD to develop software. This and other tests are automated and run constantly as a safety net or “change detectors” when developing software. If I’m talking to testers, a test is something that is developed to find defects in the software, or in the case of some regression tests, to show that defects aren’t there. There are many techniques and many different kinds of tests in this context.

Generalists

Another vague term that can cause communication breakdowns is “generalist”. An expression that comes up a lot in the agile community is that agile projects prefer generalists to specialists. What does that mean in that context? Often, when I talk with developers on agile teams, I get the impression that they would prefer working with a tester who is writing automated tests and possibly even contributing production code. As James Bach has pointed out to me, this concept of a project generalist to a tester would be “…an automated testing specialist, not a project generalist.” Sometimes independent testers express confusion to me when they get the impression that on some agile teams, testing specialists need not apply, the team needs to be made up of generalists. A tester may look at the generalist term differently than a developer. To a tester, a generalist may do some programming, testing, documentation, work with the customer – a little bit of everything. A tester feels that by their very dilletante nature on projects that their role by definition truly is a generalist one. Again, we’re using the same word, but it can mean different things to different people.

For those who have software testing backgrounds, working at the intersection of “conventional” testing and agile development is challenging. This is a challenge I enjoy, but I find that sometimes testers and developers are using the same words and talking about two different things. Testers may be intially drawn to the language of agile developers only to be confused when they feel the developers are expecting them to provide a different service than they are used to. Agile developers may initially welcome testers because they value the expertise they hope to gain by collaborating, but may find that the independent tester knows little about xUnit, green bars and FIT. They may be saying the same words, but talking about completely different things. It can be especially frustrating on a project when people don’t notice this is happening.

Other vague words?

I’m trying to make sense of this intersection and share ideas. If you have other vague words you’ve come across in this intersection of conventional testers and agile developers, please drop me a line.

agile, agile testing, user stories

But it’s not in the story…

Sometimes when I’m working as a tester on an agile project that is using story cards and I come across a bug, a response from the developers is something like this: “That’s not in the story.”, or: “That case wasn’t stated in the story, so we didn’t write code to cover that.”

I’ve heard this line of reasoning before on non-agile projects, and I used to think it sounded like an excuse. The response then to a bug report was: ‘That wasn’t in the specification.” As a tester, my thought is that the specification might be wrong. Maybe we missed a specification. Maybe it’s a fault of omission.

However, when I talk to developers, or wear the developer hat myself, I can see their point. It’s frustrating to put a lot of effort into developing something and have the specification change, particularly when you feel like you’re almost finished. Maybe to the developer, as a tester I’m coming across as the annoying manager type who constantly comes up with half-baked ideas that fuel scope creep.

But as a software tester, I get nervous when people point to documents as the final say. As Brian Marick points out, expressing documentation in a written form is often a poor representation of the tacit knowledge of an expert. Part of what I do as a tester is to look for things that might be missing. Automated tests cannot catch something that is missing in the first place.

I found a newsletter entry talking about “implied specifications” on Michael Bolton’s site that underscores this. On an agile project, replace “specification” with “story” and this point is very appropriate:

…So when I’m testing, even if I have a written specification, I’m also dealing with what James Bach has called “implied specifications” and what other people sometimes call “reasonable expectations”. Those expectations inform the work of any tester. As a real tester in the real world, sometimes the things I know and the program are all I have to work with.

(This was excerpted from the “How to Break Software” book review section of the January 2004 DevelopSense Newsletter, Volume 1, Number 1)

So where do we draw the line? When are testers causing “scope creep”, and when are they providing valuable feedback to the developers? One way to deal with the issue is to realize that feedback for developers is effective when given quickly. If feedback can be given earlier on, the testers and developers can collaborate to deal with these issues. The later the feedback occurs, the more potential there is for the developer to feel that someone is springing the dreaded scope creep on them. But if testers bring these things up, and sometimes it’s annoying, why work with them at all?

Testers often work in areas of uncertainty. In many cases that’s where the showstopper bugs live. Testers think about testing on projects all the time, and have a unique perspective, a toolkit of testing techniques, and a catalog of “reasonable expectations”. Good testers have developed the skill of realizing what implied specs or reasonable expectations are to a high level. They can often help articulate something that is missing on a project that developers or customers may not be able to commit to paper.

When testers find bugs that may not seem important to developers (and sometimes to the customer), we need to be careful not to dismiss them as not worth fixing. As a developer, make sure your testers have demonstrated through serious inquiry into the application that this bug isn’t worth fixing. It might simply be a symptom of a larger issue in the project. The bug on its own might seem trivial to the customer and the developer may not feel it’s worth spending the time fixing. Maybe they are right, but a tester knows that bugs tend to cluster, can investigate whether this bug is a corner case, or a symptom of a larger issue that needs to be discovered and addressed. Faults of omission at some level are often to blame for particularly nasty or “unrepeatable” bugs. If it does turn out to be a corner case that isn’t worth fixing, the work of the tester can help build confidence in the developer’s and customer’s decision to not fix it.

*With thanks to John Kordyback for his review and comments.

agile, agile testing, ruby, test automation, tools, Watir

Automating Tests at the Browser Layer

With the rise in popularity of agile development, there is much work being done with various kinds of testing in the software development process. Developers as well as testers are looking at creative solutions for test automation. With the popularity of Open Source xUnit test framework tools such as JUnit, NUnit, HTTPUnit, JWebUnit and others, testing when developing software can be fun. Getting a “green bar” (all automated unit tests in the harness passed) has become a game, and folks are getting more creative about bringing these test tools to new areas of applications.

One area of web applications that is difficult to test is the browser layer. We can easily use JUnit or NUnit at the code level, we can create a testable interface and some fixtures for FIT or FITnesse to drive the code with tabular data, and we can run tests at the HTTP layer using HTTPUnit or JWebUnit. Where do we go from there, particularly in a web application that relies on JavaScript and CSS?

Historically, the well-known “capture/replay” testing tools have owned this market. These are an option, but do have some drawbacks. Following the home brew test automation vein that many agile development projects use, there is another option using a scripting language and Internet Explorer.

IE can be controlled using its COM interface (also referred to as OLE or ActiveX) which allows a user to access the IE DOM. This means that all users of IE have an API that is tested, published, and quite stable. In short, the vendor supplies us with a testable interface. We can use a testable interface that is published with the product, and maintained by the vendor. This provides a more stable interface than building one at run-time against the objects in the GUI, and we can use any kind of language we want to drive the API. I’m part of a group that prefers the scripting language Ruby.

A Simple Example

How does it work? We can find methods for the IE COM On the MSDN web site, and use these to create tests. I’ll provide a simple example using Ruby. If you have Ruby installed on your machine, open up the command interpreter, the Interactive Ruby Shell. At the prompt, enter the following (after Brian Marick’s example in Bypassing the GUI, what we type is in bold, while the response from the interpreter is in regular font).

irb> require ‘win32ole’
=>true
irb> ie = WIN32OLE.new(‘InternetExplorer.Application’)
=>#<WIN32OLE:0x2b9caa8>
irb> ie.visible = true
#You should see a new Internet Explorer application appear. Now let’s direct our browser to Google:
irb> ie.navigate(“http://www.google.com”)
=>nil
#now that we are on the Google main page, let’s try a search:
irb> ie.document.all[“q”].value = “pickaxe”
=>”pickaxe”
irb> ie.document.all[“btnG”].click
=>nil

#You should now see a search returned, with “Programming Ruby” high up on the results page. If you click that link, you will be taken to the site with the excellent “Programming Ruby” book known as the “pickaxe” book.

Where do we go from here?

Driving tests this way through the Interactive Ruby Shell may look a little cryptic, and the tests aren’t in a test framework. However, it shows us we can develop tests using those methods, and is a useful tool for computer-assisted exploratory testing, or for trying out new script ideas.

This approach for testing web applications was pioneered by Chris Morris, and taught by Bret Pettichord. Building from both of those sources, Paul Rogers has developed a sophisticated library of Ruby methods for web application testing. An Open Source development group has grown up around this method of testing first known as “WTR” (Web Testing with Ruby). Bret Pettichord and Paul Rogers are spearheading the latest effort known as WATIR. Check out the WATIR details here, and the RubyForge project here. If this project interests you, join the mailing list and ask about contributing.

*This post was made with thanks to Paul Rogers for his review and corrections.

testing

MythBusters and Software Testing

I enjoy the Discovery Channel show MythBusters. For those who may not be familiar with the show, it is hosted by two film-industry techies: Adam Savage and Jamie Hyneman who test popular urban legends to see if they are plausible or not. It’s almost like an extreme snopes. They take a popular urban legend, design a concept on how to recreate it, and test to see if they can disprove it or not.

What I like about the show is the process that they follow when testing out whether a myth is plausible or not. Once a myth has been selected, they treat it like a theory (or hypothesis) that needs to be shown it can’t be disproven. They design and build tools to simulate a particular environment, and use observation to see whether the myth holds up. They usually improvise to create an environment that simulates the conditions needed to test the myth. They may not have the exact tools or objects readily at hand to use, so they will design objects to get the desired effect, and measure key elements to ensure they will help create the conditions that are needed.

The process the MythBusters follow isn’t perfect, and the simulations they create are not necessarily identical to the original objects in the myth, but the environments and tests they create and execute (as software testers often find on their projects) are generally good enough.

I think they would make great software testers. What they do reminds me of software testing. We often have to build specialized tools to simulate appropriate conditions, and we use observation and some measurement tools to test whether a theory can be reasonably proven to be false, under certain conditions. If we can demonstrate that the theory is falsifiable,1 we gather the data from our observations and demonstrate how the theory was shown to be false under certain conditions. For software testers, this data is usually gathered into a bug report or bug story card. On the show, they say whether a myth is plausible or “busted”.

What’s more, the MythBusters have that certain testing spirit that says: “What would happen if?….” which compels them to push things to the limit, and quite literally blow things up. Testers have fun pushing something to the brink, and then giving it that one last shove over the edge where the server dies, or the application runs out of memory. Not only are we having a bit of fun to satisfy curiosity, but we are gathering important data about what happens when something is pushed to perhaps extreme limits. Not only do we get a certain enjoyment out of watching things break, we add that observed behavior to our catalog of behavior. When we see it again, we can draw on some of those patterns and predict what might happen. Sometimes we can spot a bug waiting to happen based on the observation of what happens before the application blows up.

A related testing example might be a development theory we are testing such as: “the web application can handle 30 concurrent users”. We may have 8 people on our development team, so we can’t test 30 concurrent users with so few people. Instead, like the MythBusters we use, develop or modify a tool to simulate this. We might develop a test case that threads several sessions on a machine to simulate the 30 concurrent users, or use a test tool designed for this purpose. If the application fails to hold up under the simulated conditions in a repeatable fashion, other factors equal, we have evidence that this theory is falsifiable. If the reverse occurs, and the application holds up perfectly fine, we know the theory is not likely to be false under the conditions we ran the test. We may alter other variables to design new test cases, and observe what happens. As testers, we usually only record the results that demonstrate falsifiability which is a bit different from the MythBusters, and from others who follow the general pattern of the Scientific Method.

1 Check out Chapter 2 in Lessons Learned in Software Testing for more on testing and inference theory.

documentation, writing

Jennitta Andrea on Tests as Documentation

Jennitta Andrea sent me her position paper developed for the Tests as Documentation workshop. Unfortunately, she was unable to attend the workshop, but sent this paper with test examples. I like her approach, and encourage you to take a look at her examples. Here are some of her thoughts in the paper on the Tests as Documentation concept:

  • You must work VERY hard for your tests to be actually used as documentation for the system requirements.
    • They must be both organized and named extremely well so that someone looking for the details on some aspect of the system can find them.
    • They must be written with the reader in mind … all of the well known test automation patterns describe these details.
  • The automated acceptance tests must be built upon a clear and concise domain specific testing language (this is a superset of the ubiquitous language described by Evans).
  • The system requirements are spread out amongst a number of different kinds of tests
    • User acceptance tests cover the big-picture work flow and business process description
    • Unit tests cover the specific business rules for individual / small groups of components.
  • I still think you need some actual textual / graphical documentation that serves as an overview of the business process and the system. We recently put this kind of thing on a wiki at the very end of the project for the support team (should have done it incrementally during the project)
  • Stories should be thrown away once they are implemented. They should not be used as a permanent artifact for describing the system. A story is only a small chunk of the big picture, and it may contradict/overturn an earlier story. We need to create the big picture as it evolves when we implement various stories.

It’s interesting to note her emphasis on writing tests with an audience in mind which was a major theme in the workshop. She also noted that user acceptance tests document the business side of the product while the unit tests document the system process side of the product. This was also noted by one of the groups in the workshop.

I especially like the approach she has used for tabular test data. Check out “Example 2: Tabular Test Specification” of her test examples in the paper. The format for the test data in a spreadsheet is very intuitive and reads nicely.

exploratory testing

Exploratory Testing

Exploratory Testing is an effective way of thinking about software testing that skilled testers can use to think of techniques to find important bugs quickly, provide rapid feedback to the rest of the team, and add diversity to their testing activities.

To learn more about Exploratory Testing, check out James Bach’s site for articles. Cem Kaner has also written about it. Either of those sources can explain it better than I can.

I sometimes get confused looks from some practitioners when I tell them that I’ve found Exploratory Testing to be effective for my own testing. Some of the confusion may come from not knowing exactly what Exploratory Testing is.

I’ve worked as a testing lead on projects and have directed others to do Exploratory Testing to complement automated tests, and have sometimes met with resistance. Those who resisted later told me that they weren’t used to finding so many defects when they tested, but still were uncomfortable doing unscripted testing even though it seemed to be more effective. When I repeat what James Bach says, that “…testing is an interactive cognitive activity” and I value them for their brains and expertise, they are smart testers who can add a lot of value and I am pleased with the results, it’s rewarding to watch their confidence grow.

Some of the confusion may come from working in an unscripted environment when one is used to following a script. When confidence begins to displace confusion, testers often get more creative, they seem to improvise more when testing, and they seem to want to collaborate and communicate more with the rest of the team. When they are finding important issues quickly, sharing them with the developers and getting rapid positive feedback on their own work, it seems to help build team cohesion as well as individual confidence.
Here’s where I think another source of some of the confusion is. Some people seem to think that you can only do Exploratory Testing on software you haven’t used before. At least that’s the impression I get. If I say I will do Exploratory Testing on each build after the automated test suite runs, some people act puzzled that I would be doing Exploratory Testing on a program I am already familiar with.

As a software tester, I can still explore, inquire and discover testing software I’m familiar with. If we think about it, that’s often how scientific research works. Scientists deal with the familiar and look for patterns or occurrences that don’t seem to fit known models. Reviewing the familiar to verify that the models still hold true is common. Discovery can readily come from exploring something we are familiar with. A known behavior might change under certain conditions that we haven’t seen yet. We may try a new experiment (test) in a different way that yields results that we haven’t seen before.

Exploratory Testing isn’t always working as a discoverer working through a program for the first time. Maybe we are thinking of an explorer analogy like Lewis and Clark when we should be thinking of a scientific method exploration analogy. Sometimes Exploratory Testing can be like a voyage of discovery charting the unknown, but it is very often like a scientific experiment where we are exploring changing variables based on observed behavior.

The pursuit of knowledge and creative problem solving are facilitated with Exploratory Testing. Scripted testing or Directed Observation is a common “best practice” in software testing. How often do we miss problems because we direct what we want to see in a program and miss the obvious? Exploratory Testing is one way to help test with diversity and look at the familiar in new ways.