All posts by jonathankohl

agile, agile testing

The Role of a Tester on Agile Projects

I have been dealing with this question for some time now: “What is the role of a tester on Agile projects?” and I’m beginning to wonder if I’m thinking about this in the right way. I’ve been exploring by doing, by thinking and talking to practitioners about whether dedicated testers have a place on Agile teams. However, most of the questions I get asked by practitioners and developers on Agile teams are about dealing with testing on an Agile project right now, not whether a tester should be put on the team. The testers are here, or the need for testers on the team has been proscribed, so they are looking for answers on how to deal with issues they are facing right now.

Has the ship sailed on the question: “Is there room for dedicated testers on Agile projects?” already? Is it time to rephrase the question to: “What are roles that dedicated testers have added value with on Agile teams?” followed by “What are some good techniques to deal with the unique team conditions on Agile projects?”.

I’m willing to accept that some methodologies may not be compatible with this notion. The question remains, what are testers doing on real-world Agile projects, and what methodologies don’t seem to be amenable to dedicated testers? Of those, are dedicated testers pressured out due to team development philosophy, or are dedicated testers simply not needed? Real-world experience is what we as a community need to keep sharing.

Have the “specialized testers” arrived already, and has the question of whether they should be brought on teams become academic, or are we answering the question by doing? The question will answer itself anyway as time goes on, and experience tends to trump theory alone.

I have been a bit reluctant to put a stake in the ground about the role of testers on Agile projects without more experience myself, but judging by the questions I am getting and the constructive criticism that I have recieved, I should probably share more of my own experiences. I think it’s time for testers on Agile projects to start talking about techniques and what roles they have filled on Agile teams. From that we can gather a set of values that describe the roles, techniques and mindsets of those who are testers on Agile projects. Answering the question by exploration and doing is much more exciting to me than an academic debate.

ruby, tools, Watir, web applications

Visible Tests and Web Testing with Ruby

Announcing a new project and blogging my notes for the Scripting Web Tests Tutorial which I prepared for XP Agile Universe:

Visible Tests

Automated test results are often difficult to communicate to business stakeholders on a project. People frequently ask me about the visibility of tests on a project saying: “The customer doesn’t understand JUnit tests” or “The JUnit and FIT tests mean very little to the customer. The customer doesn’t usually think in terms of green bars, or table-driven tests like developers and technical testers do. They believe we are testing, but they can’t see the tests run in a way that they relate to. How do we raise the visibility of tests?”

The WTR IE Controller has an advantage in test visibility since the testable interface we use is about as close as we can get to interacting with a program the way an end-user would. Business stakeholders understand testing at the GUI layer because that’s the way they relate to the program. These tests can be played back to the business stakeholders so they can see the tests run in a way they relate to. Technical people on a project team relate to the program at various layers and sometimes we forget about the business problems we are solving. Techies look at the backend and the front-end, business users usually only see and understand the front-end of the application. The preferred method of interaction with the program can differ between the groups.

Business stakeholders can watch the tests play back on a computer which provides rich, visual feedback. If the test fails, they see it fail. If it passes, they see it pass while exercising the application in a way that they would themselves. While they have faith in the ability of the technical members of the team, and will accept the testing numbers and their word, nothing replaces the assurance they get from seeing it work, and from manipulating the product themselves. With IE Controller tests, the customer can see if the tests pass or fail by watching how the application works in the Internet Explorer browser. Tests can also be designed to mimic business interaction, and provide results logging that non-technical project stakeholders can understand. The tests will demo the business solutions that the customer needs the application for in the first place. These kinds of automated tests help provide more assurance in what the technical members of the team are doing.

At an end of iteration meeting, the application can be demonstrated using these tests if the group wanted to give a quick demonstration.

Announcing WATIR

The latest version of the Web Testing with Ruby project has begun under the WATIR project spearheaded by Paul Rogers and Bret Pettichord. I’m excited by the prospect of a more sophisticated Open Source web testing solution using Ruby. I’ve had success with the IE Controller tests I’ve written to date, and look forward to using a more complete solution.

coverage, strategy

Testing and Numbers

Michael Bolton said this about numbers on testing projects:

In my experience, the trusted people don’t earn their trust by presenting numbers; they earn trust by preventing and mitigating bad outcomes, and by creating and maintaining good ones.

I agree. Lately, I’ve been thinking about how we report numbers on testing projects. I was recently in a meeting of Software Quality Assurance professionals and a phrase kept coming up that bothered me: “What percent complete are you on your project?” I think they mean that they have exercised a certain percentage of test cases on the development project. I don’t feel like I can know what “percent complete” of test cases I am on a project, so I’m uncomfortable giving out a number such as “90%” complete. How can I know what 100% of all test cases on a project are? Cem Kaner’s paper: Impossibility of Complete Testing shows us how vast the possible tests that can be run on a project are.

For all the projects I’ve been on that claimed to have 100% test coverage, each one had a major bug discovered by a customer in the field that required a patch. We obviously at best had only covered 100%-1 of the possible test cases. That one test case that the customer found a bug that we did not find was not in our set of test cases, so how could we claim that we had 100% completion? How many more are we missing?

Reporting numbers like this are dangerous in that they can create a false sense of security for testers and project stakeholders alike. If we as testers make measurement claims without looking at the complexity of measurement, we had better be prepared to lose credibility when bugs are found after we report a high “percent complete” number prior to shipping. Worse still, if testers feel that they are 90% complete of all tests on a project, the relentless pursuit of knowledge and test idea generation is easily replaced by apathy.

Cem Kaner points out many variables in measuring testing efforts in this paper: Measurement Issues and Software Testing. Accurate, meaningful measurement of testing activities is not a simple thing, so why the propensity for providing simple numbers?

I look at a testing project as a statistical problem. How many test cases could be in this project if I knew the bounds of the project? Since I don’t usually know the bounds of the entire project, it is difficult to do an accurate statistical analysis using formulas. Instead, I can estimate based on what I know about a project now, and use heuristics to help deal with the vast numbers of possible tests that would need to be covered to get a good percentage. As the project progresses, I learn more about it, and use risk-based techniques to try to mitigate the risk to the customer. I can’t know all the possible test cases at any given time. I may have a number at a particular point in the project, so of the test cases that I know of, right now, we may have a percentage of completion. However, there may be a lot of important ones that I haven’t, or the testing team together haven’t thought of. That is why I don’t like to quote numbers of “percent complete” without providing a context, and even then I don’t present just numbers.

The Software Quality Assurance school of thought seems to be numbers obsessed these days. I am interested in accurate numbers, but I don’t think we have enough information on testing projects to be using many of the numbers we have conditioned project stakeholders to rely on. Numbers are only part of the picture – we need to realize that positive project outcomes are what are really important to project stakeholders.

Numbers without a context and careful analysis and thought can give project stakeholders a false sense of security. This brings me back to Michael Bolton’s thought: what is more important to a stakeholder? A number, or the opinion of a competent professional? In my experience, the latter outweighs the former. Trust is built by delivering on your word, and helping stakeholders realize the results they need. Numbers may be useful when helping provide information to project stakeholders, but we need to be careful how we use them.

interfaces, TDD, test-driven development

Expressing Interfaces

Mark McSweeny comments on Discovering Interfaces:

While testers work to discover interfaces in programs, developers work to express interfaces. This can be through developing an API, an interface for unit testing, all the way down to the data structures and algorithms you use. Well designed interfaces are testable interfaces, so a tester can tell you whether the interfaces you have expressed are testable or not.

The meeting of the roles through the program is interesting to me. The two meet in the middle of creation and inquiry. Sometimes as testers we forget that there are people behind what we are testing who have poured a lot of effort into the interfaces we are testing. The more testers can supply positive feedback to the developers, the more confidence they can get from their development efforts.

While testers may be satisfied when a program does not fail after they have attempted to make it break, a developer is satisfied when they have solved a problem using technology. Together, there is expression, inquiry, discovery, feedback, communication, and collaboration.

What is interesting with methods such as TDD is that they attempt to combine interface expression and discovery. One attempts to express an interface while also attempting to critique it.

writing

Go Flames Go (Edit)

One win away from the Stanley Cup.

Go Flames!!

edit — It was a heartbreaker last night, but Calgary and Canada are proud of the Flames amazing playoff run.

In related news, Tim Van Tongeren has a post on his blog about a publishing error where the wrong article ran in a Tampa newspaper. They accidentally published the version about the Lightning losing the cup. Sounds like they needed a second set of eyes in production last night to help with testing.

reliability, security

Simplicity and Reliability

From the National Post:

Systems crash inevitable report: Networks for banks, hospitals, power lines at risk within the next five years

Michael Friscolanti
National Post

June 8, 2004

Computer networks that support Canada’s critical services — from hospitals to banks to power lines — will undoubtedly crash in the next five years, warns a government-commissioned report that says even immediate action cannot stop the inevitable.

Shoddy software has left the Internet and other parts of our telecommunications systems vulnerable to a massive meltdown, the report concludes. No corrective action can avert “a major failure,” but the authors say both the government and the private sector must act quickly if they are to prevent subsequent collapses.

[….]

With respect to software that has evolved to a high level of complexity, there may exist no single individual who grasps the entire program, let alone one who can keep track of all those who have contributed to its various components,” the report reads. ” [emphasis added]

I’m reminded of Ralph Johnson speaking on security last week. He noted that secure systems are usually known by one person; they have a simple enough design to be understood by one person (usually the designer). To be secure, a system must be reliable.

design

Software Design & Testability

I recently attended a talk by Ralph Johnson which I enjoyed.

One theme that Ralph talked about was simplicity. He was talking about security patterns, and how simplicity of design implies knowledge. If a design is knowable, it can be made secure. That led to a thought for me on testability. This is also true of a testable design; if a design is simple, it is knowable and therefore testable.

For example, as a tester one is often asked to look at requirements or user stories to determine if they are testable. If you as a tester can understand them, that’s a good indication that they are testable. If you can’t, how can you (or others) test? In my experience, when no one person understands or knows the project, it is difficult to test. If I need to go around to seven different people to understand the design of a project and none of them understand it in its entirety, chances are the project will be equally as difficult to test. Legacy systems suffer from this, but poorly designed new software projects are also difficult to test.

As a tester, if you don’t understand the design, don’t automatically think that you lack the necessary technical skills to understand. The architectural designs should be understandable to any stakeholder involved with the project (once the convention is explained to them). If the design isn’t understandable at a high level by the entire team, this is probably a bad design smell.

Ralph also mentioned that tools such as UML or DFDs serve to express program design to other people. (The opposing view is that these tools are used to generate code.) I agree with Ralph that these tools express design to be communicated to other people. If the architectural diagrams are easily understood, they will not only be testable, but more easily developed and communicated to the business. Tool generated code is extremely difficult to know and to test. People design software, people develop software, people test software, and people use software. The tools are there to help express each of those roles.