All posts by jonathankohl

agile, process, value

Content Pointer: The Next Wave: Valuable Products First, Process Second

I wrote an article for Modern Analyst that describes some of my process thinking over the past several years. They asked for a piece on post-Agilism, but I prefer talking about value now, so I wrote this piece for them instead.

I introduce my thoughts on a trend I have witnessed for a while now where people move from software ideas (let’s build this killer app!), to process (let’s go Agile!), to value (let’s ensure our application blows our customers away, and everything we do feeds that effort.)

Three years ago, I didn’t hear people talk about value that much at all. It was all process, process, process, and how following an Agile or other process would lead us to success. Now, I am seeing the “value” word pop up more and more, and more teams are using an overall vision to help focus their efforts (process and otherwise) towards creating value for their customers and themselves.

test automation

Content Pointer: Automation Politics 101

I wrote an article for Automated Software Testing magazine on test automation politics for the November issue. A PDF copy of the article is available here: Test Automation Politics 101.

Article back story: I had a nice dinner with Dot Graham and Dion Johnson this spring. As we were sharing automation stories, Dion asked if I would be interested in writing an article on politics and test automation. Since most automation writing focuses on tools, results and ideals, there isn’t a lot out there on the social aspects of automation. I thought I’d provide my take on some social challenges I learned about the hard way. I was inspired by Bach’s Test Automation Snake Oil. Bach doesn’t pull punches, and I tried not to either. I’ve often thought that it would have been nice to have been advised on potential controversies when I was starting out in test automation, so we took a handbook approach with the article.

I hope you enjoy it, and if you’re starting out in test automation, you find the article informative. If you’re currently facing any of the resistance I outlined, I hope you find the content encouraging.

fuzzing, security, tools, XML

Use of Fuzzers Helps Discover XML Security Threats

I posted about fuzzing a few days ago, and I think the tools are neat, and in the hands of good testers can be powerful. They are a nice way to augment existing security testing, to test data transfers or messaging, or to simply generate test data. However, some of my readers basically said:

Big deal. It’s a neat toy for you, or some of your clients who have the time for that sort of thing, but why would I want to use one?

The Register has posted an interesting article where fuzzers were used to discover potential security holes in XML libraries. Check it out here: XML flaws threaten ‘enormous’ array of apps. If my babblings about fuzzing don’t get your attention, maybe this article will. The potential to use fuzzers (in conjunction with other security tools and techniques) to help people catch problems that bad guys will exploit is enormous.

Codemonicon, the company cited in the article, have a lot of interesting information and expertise in this area. They have a nice introduction to fuzzing on their website. This paper by Rauli Kaksonen has a lot of technical detail on fuzzing if you’d like to learn more.

SBTM, Session Based Testing

Session-Based Testing Using Instant Messaging

Here’s a scenario: a programmer friend is creating a new web-based product for his company. He wanted someone to augment the existing testing that was already underway, and hired me to add more firepower. It is a distributed environment, which means that everyone in the office telecommutes. We decided to start with me doing an initial session, via instant messaging. We blocked out a time for the session, determined a focus (“mission”) and I started testing.

As I tested, I typed what I was doing in the instant messaging (IM) client, along with questions, concerns, crashes and bug ideas. He followed along through the application as I tested, asked clarifying questions as I discovered problems.

After ten minutes, I had generated enough ideas for that mission. He logged 6 bugs in his fault-tracking system. I had discovered 5, he had discovered 1 himself while following me along and monitoring the server and database.

I’ve done session-based testing over IM quite a lot with different distributed projects, like open source projects such as Watir and Session Tester, and with some of my clients that use telecommuting, or are completely virtual. We’ve used and adapted some of the ideas from SBTM to help provide more structure around the actual test sessions.

Using session-based testing with IM seems to work best in pairs. Once a focus and rough timebox are determined, test setup is out of the way, both pairs initiate a session through an IM client. One drives by testing, and the other saves the conversation locally, and creates notes, logs bugs, investigates underlying behavior, etc.

This style is natural (we communicate a lot with IM clients anyway) and doesn’t interrupt the natural testing flow if one person acts more as the primary test idea generator/executor and the other acts as the primary scribe. If you want to add some variation to your testing approach, give it a try.

exploratory testing

New Exploratory Testing Tutorials

I’ve developed some new ET tutorials. The newest is “Managing Exploratory Testing”, addressing questions I hear the most from managers. Since I’ve done a lot of this sort of training already, it made sense to start offering this publicly. I’ll be teaching it at EuroSTAR and STAR West this fall.

The other tutorial is an evolved version of my take on “Exploratory Testing Explained”. Due to questions and interests of people who have taken the course, it has evolved into a hands-on, experiential workshop: “Exploratory Testing Interactive”. I’m also teaching it at STAR West.

If you’d like to work with me to get a glimpse into my take on exploratory testing, and learn some new skills, drop in at one of the conferences. If you don’t or can’t attend these conferences, you can also consider bringing me in to work with you and your team one-on-one.

fuzzing, security

Fuzzing Through the Side Door

I’ve been looking into testing with fuzzers lately, and finally got the chance to do this on a live project. While there are a good deal of black-box fuzzing tools out there, if you want to go beyond that you are often on your own. At the other end of the spectrum, MSDN has a nice article on white box fuzzing.

What I needed to do was somewhere in the middle. I needed to test a data transport layer in a large, complicated system, and fuzz the data that was sent in messages from internal systems as well as from 3rd parties. This required some knowledge of the code internals, the protocol, and messaging APIs used, the architecture, etc. Some people call this gray box testing, and I like to call it “testing through the side door”, a term I got from Jennitta Andrea. (The front door is the UI, the back door is the database, the side door are other testable interfaces in-between like messaging APIs, etc.)

I couldn’t find any tools (open source or proprietary) that did what I needed, so I started a search for a Java fuzzing library. (I was doing the work in a Java shop, so Java-based tools were my first choice.) I couldn’t find anything, so I contacted the people on the JBroFuzz project and told them what I wanted to do. I like OWASP, and I liked what I saw in the JBroFuzz product. Thankfully, it is pretty easy to use JBroFuzz as a library, in addition to the already-powerful HTTP(S) fuzzing the tool provides through its UI.

Yiannis Pavlosoglou bent over backwards to help me. Yiannis sent me examples to try out, explained how to create my own fuzzers, and even helped me fix a silly iterator error I had in my own code. Thanks Yiannis!

In relatively short order, I integrated JBroFuzz into the existing test framework for messaging, and the first afternoon I had it running, it helped us uncover problems.
If you’re looking for a fuzzing library for Java, here are directions for using JBroFuzz as a fuzzing library.

Thanks to Yiannis Pavlosoglou for all the help, and the JBroFuzz team for a great tool. If you haven’t started looking at fuzzers and fault injection tools on your project, give them a try. Happy fuzzing.

bugs, politics, testing

I’m Finding Bugs, But It’s Pissing People Off. What Am I Doing Wrong?

A rookie tester asked me this question. They are experienced in software development, and their grad school work was in a highly specialized area of mathematics. They have been hired to do a very particular type of testing that they are uniquely qualified for. After getting over the mindset shift required to be an effective tester, they were pleased that they were finding bugs that they’d been hired to find. Then a wrinkle appeared. The very people who hired them to find the bugs got angry when they found the bugs they were being paid to find. Sound strange? It might, but it’s actually a common response to effective testing.
Here are some of my clarifying questions:

  • Are the bugs you are finding important, or trivial? Stakeholders on teams can get irritated if you inundate them with only trivial bugs.
  • How are your bug reports? Are they thorough? Do you provide enough information for devs to accurately reproduce the bugs?Programmers get irritated with bug reports they can’t reproduce and track down.
  • How’s your attitude? Do you laugh with glee at the misfortune of the dev team? Are you empathetic, or a condescending jerk? Is your language accusatory. blaming or condescending in any way?No one wants to be around someone who enjoys the schadenfreude.

No, the devs like my bug reports, and I’ve been as low-key and empathetic as possible. These are serious bugs that have probably been in the application since it was released several years ago. I’m tempted to stop logging the kinds of bugs that cause them to get mad at me.

Ok, you’re probably not doing anything wrong. In fact, you are probably doing something right. Don’t stop! You aren’t failing in your work, you are succeeding.

Since they seem to be doing some of the right things and aren’t knowingly or obviously antagonizing stakeholders, I assume that they really are finding important bugs, and that reaction is one telltale sign that they are being effective as a tester. I also encouraged them not to give up, but to do as Peter Block would recommend: move towards the resistance. That is a heuristic that tells us we are doing our jobs, and highlighting the really important problems. The resistance to being confronted with difficult problems that aren’t trivial to solve is just human nature.

As testers, if we are finding good problems, and we’re working with the team to help them, moving away from the resistance is a sure-fire way to relieve the pressure in the short-term, but lose credibility in the long term. If stakeholders realize we can’t be trusted, we’ve lost out ability to effectively test, observe information, and provide information that is useful for them.

I have a personal rule: if I am being pressured not to talk about something, or I feel like I should avoid a contentious issue, that means I absolutely must deal with it and talk about it. I try to be empathetic, understanding, use the right kind of language and not be a jerk, but I bring it up. Every time I do, something important is brought to the attention of the people who need to hear it. They may not like it, and they may not like to see me coming, but they know that I will always tell them the truth.
Early in my career, I was told not to bring up issues that were met with resistance. I was told that was a “career-limiting move.” My career is a testament to the opposite. Whenever I have faced resistance, stuck to my integrity and ethics, and talked about the hard problems the team had been taught to ignore, it has been a career-catapulting move.

So testers, if your work is pissing people off, the problems you are observing and reporting are important, and you aren’t being a jerk, don’t give up. It may hurt to point them out in the short-term, but it pays off handsomely in the long-term.

*Note: This doesn’t just apply to testing work, but any type of work that requires pointing out and helping solve problems. Whatever it is you do, don’t be discouraged if you are working on real problems and people are behaving strangely. Use that resistance to tell you that you are doing something right. The worst thing to do is give up and become silent.