Monthly Archives: December 2006

heuristics, mnemonics

Modeling Test Heuristics

Mike Kelly has created an effective software testing mnemonic based around James Bach’s software touring heuristic. It is now burned indelibly on my brain: FCC CUTS VIDS. I think Mike hit a home run with this one – I use it a lot myself. The mnemonic makes more sense when you review Mike’s excellent explanation, and when he spells out the mnemonic into a list:

Feature tour
Complexity tour
Claims tour

Configuration tour
User tour
Testability tour
Scenario tour

Variability tour
Interoperability tour
Data tour
Structure tour

Like all good mnemonics, it is easy to memorize because it evokes familiarity, imagery, and it has a nice cadence. I should qualify that. It is easy for /me/ to memorize, and of course, as the creator, easy for Mike to memorize. I relate to words and lists, and a nice cadence helps me repeat it silently to myself, in English, the language I tend to think in. My friend Steve, one of the smartest people I know, does not have as much self-talk going on in his brain. He doesn’t relate much to lists and words, he relates to images and colors much more than I do. He thinks in pictures more than words. If I were teaching Steve the touring heuristic, and hoped he’d memorize it, I’d take a different tack.

To explore Mike’s test heuristic in an alternate form, I created the following image, in the form of a mind map, as an example:

This image is a model my friends who think more in terms of images and colors might be more comfortable with than a list.
As I look at it, I see some cool side effects. As a mind map, it looks unfinished. This is partly by design; I didn’t expand any of the idea points as you normally would, and there is a blank spot at the bottom right hand corner, an area that our eyes are naturally drawn to as we scan an image.

Contrast the image with the list above. In the list, I get a sense of completion – each letter of the mnemonic is spelled out. When I get to the final “S”, standing for “Structure Tour”, I get a sense of satisfaction. The spelling out of the mnemonic is complete. When I look at the image, I am struck by how unfinished it looks. I want to draw it on a whiteboard and expand each idea bubble. I want to fill in that blank spot at the bottom right-hand corner. I want to explore the heuristic further, and I want my image to be symmetrical so I can easily remember the details by the shape and colors in the model.

My next urge is to re-create the mind map using only images instead of text. That would be an interesting experiment. Next, I want to convert the list into a tree diagram, with all the relevant testing ideas I can fit under each item. A tree diagram is also an effective way to quickly spot areas that need to be expanded. This appeals to my pragmatic testing approach – more analysis can help me be that much more thorough the next time I use the mnemonic when testing.

Different models help us look at problems differently, and they will reveal or obscure different information and ideas. As my father taught me years ago, involving all your senses when learning is effective. I constantly analyze my own models, and try to improve on my technique. If a model is getting stale, all I need to do is change the medium, and something extraordinary usually clicks in my brain as a result.

What do you see in the image and list above? Do they evoke different reactions in your mind? How would you model Mike’s mnemonic? What tools would you use, and what would you learn about testing and thinking by employing them? How many of you would write a song to accompany it? There’s only one way to find out. Model away.

*Thanks to Sherry Heinze for reawakening my interest in mind maps.

heuristics, mnemonics, testing

Learning Testing Heuristics

When I was in junior high school, one of my tasks was to learn metric conversions. My science teacher at the time took this task seriously. To help us remember the kilo, hecto, deca, deci, centi, milli prefixes to meter, liter and gram, he had us memorize a mnemonic. The mnemonic looked like this: KHDMDCM, and there was a sort of song that went through it: “King Henry Danced Merrily…something something something”. Trouble was, I barely had the mnemonic memorized in time for a pop quiz, and I couldn’t remember what each letter stood for. I ended up only being able to write the King Henry song out on my test paper and handing it in. I couldn’t do the required conversions such as decimeters to meters, or kilometers to centimeters. My teacher gave me a cold look when he handed the exams back, and admonished me to stop fooling around and actually start trying. Trouble was, I was trying. I just couldn’t get that stupid mnemonic to make sense in my brain.

At home that night, I sang the King Henry song over and over, repeating what each letter stood for. It was hopeless. I had to constantly look at my cue cards.

My older sister happened along and asked: “What on EARTH are you doing?”

I explained.

“Don’t do it that way,” she said, and quickly sketched out a diagram.

She wrote out all the measurement prefixes from largest to smallest, with “meter, liter, gram” in the center, and drew an arrow going up to the left on one side, and down to the right on the other. I stared at the diagram for a few seconds, and she had me reproduce it cold. Within minutes, I had the image memorized. She then taught me how to use it for conversion. It was a snap. On my metric conversion exam later that week, I ploughed through the questions with ease. “Forget King Henry,” was my thought, “this works way better.” And, forget him I did.

However, over twenty years later, I can still scratch out that conversion diagram and convert deciliters to decaliters, or kilograms to grams on the back of a napkin with relative ease.

“Different people learn differently” my father once told me. His first career was a school teacher, and over twenty five years he taught hundreds of different students. He was surprised by little, and was quite pragmatic. If colors helped you learn and remember, use colors. If mnemonics and little songs worked, try them. If images worked, go with the images. Try to involve most of your senses if you want to remember something, he said. Furthermore, if the system you are trying to use doesn’t work for you, make sure you clearly understand the concept, and then create your own. (These didactic sessions usually came from the depths of a newspaper, my Dad gazing at me over the rim of his glasses with CBC Radio blasting in the background.)

When I started working with James Bach, he talked about using mnemonics for testing. I was familiar with, and had used most of his published mnemonics, but I had never memorized them. I used them more for analysis prior to testing. They seemed suited for analysis and test planning, and I was able to use them to generate extremely effective test strategies and plans.

Then one day I saw James at work, testing software. He was thinking out loud, using test heuristics, rattling off testing mnemonics, and simultaneously designing and executing tests under each letter of a mnemonic in real time. He would pause periodically to point out his findings or jot down notes. He would generate and execute many test ideas in different categories in a short period of time. This resulted in surprisingly thorough testing, with test ideas no on else had thought to try. James’s testing provided a veritable treasure trove of bugs within minutes. Whoa.

James was using SFDPOT (San Francisco Depot), and he would say the first letter of the mnemonic out loud like this:”S – SYSTEM!” Then he would mutter and brainstorm and figure out ways to test the system part of the application. He might alter the files in an installed app, or remove dependencies. He would use system tools to monitor memory use and CPU usage for example, while testing. If he saw that memory use jumped up when doing a particular action, he would repeat it rapidly to see if memory use kept increasing, and if it fell back down to where it was before. He would uncover installation issues, improper dependencies, memory leaks, and all sorts of things in a very short period of time.

Once he was satisfied with those tests he would sound out “F- FUNCTION!” Then he would generate a few ideas of functional tests, and he would examine areas that seemed out of the ordinary and push them. If he found an awkward workflow, he would note it down. If he also noticed memory leaks or CPU usage spikes, he would note what he was doing that caused them. He would continue on through Data, Platforms, Operations and Time, then he would stop testing, consult his notes and start logging bugs.

I’d never seen anything like it, and I knew right then I had to master his approach. But memorizing mnemonics was a bit of a blind spot for me. I needed to figure out how to memorize those mnemonics, so I too could quickly slice through a testing problem (any testing problem) with thorough ease. However, the spectre of King Henry loomed above me, reminding me of my early mnemonic memorization failure.

James peered down at me over the rim of his glasses: “You need to memorize testing mnemonics.”

His tone was casual, but firm. To him it was like telling me I needed to learn to cross the street. Obvious.

I complained about how I often found it hard to memorize mnemonics. Then, much like my sister did back when I was in junior high, James told me not to do it that way anymore.

“That’s because you are memorizing someone else’s. You need to create your own. If they are yours, you’ll remember them.”

He then spent time with me helping me practice memorizing a mnemonic, using the mnemonic to test in real time, and then encouraged me to personalize it and make it my own.

It turned out I could learn new tricks. After all, I had used mnemonics as a student that were quite useful and effective. Who can forget SOHCAHTOA, an effective way to remember how to calculate the sine, cosine and tangent of an angle? I remembered memorizing that mnemonic and using it to great effect. “Soak-a-toe-ah” I repeated in a unique cadence, and imagined a big toe in a wash basin. The imagery and the rhythmic cadence of “soak-a-toe-ah” helped me remember the mnemonic, and practice helped me remember the application. I could memorize testing mnemonics just as easily, and banish King Henry from my memorization nightmares. It turns out that motivation and seeking results can go a long way to memorizing a mnemonic.

I was faithful to James’s advice, and mnemonics have served me well. While I use other people’s mnemonics in my own work, I find I return to my own more often. I’ve added mnemonic memorization and creation to my testing thinking repertoire. I use ones from others that are particularly effective, and I create my own as I need to as well.

When I teach testing techniques, I find, as my father did many years ago, that some thinking models appeal to different people in different ways than others. Mnemonics and other thinking aids don’t need to be lists, they can be imagery, visual patterns, sounds, or anything else that helps you remember to be more thorough. Any thinking model that gets the job done is valid. Furthermore, using different thinking models along with memorized test heuristics helps me analyze them, enhance them, and learn more about my own work. It also means my implementation looks different from someone else’s, based on my own experience, insight and skill. Now, I don’t just use test mnemonics to help plan, I use them during test analysis and planning, and during test execution.

EDIT Update: I SLICED UP FUN is a popular mnemonic I created. I developed it first for myself, then shared it with my team, then with everyone else. It has become quite popular on mobile software development projects the world over.

Are you curious about using mnemonics in your work? If so, imagine me peering over my glasses at you, expecting you to make them your own.

mnemonics, models, puzzles

Speeding Up Observation with Models

You just generated four possible ways to solve the problem in about thirty seconds!

One of my brilliant programmer friends had just posed one of those useless interviewing puzzles that Microsoft made famous. We were discussing that the only thing they test is a candidate’s ability to solve those sorts of problems, and really tell you nothing about their ability to do software development tasks. He was frustrated by a recent interviewer who had merely posed a puzzle, wrinkled up their nose at his attempt to solve it in the interview, and had then lectured him on the “right” answer. To illustrate, he had posed the question to me.

Fortunately, it was a type of puzzle that I could easily apply a model against. Also in my favor, I had spent a recent evening teaching a software testing student that very model. I smiled inwardly, and with nonchalance outlined four possible ways of attacking the problem. He stared at me almost dumb-founded for a moment, and then I had to let him off the hook. I laughed as he stared at me in wonder. “There’s a trick to this” I told him. I revealed the model, which is the software testing mnemonic HICCUPPS. I suddenly had a brilliant programmer with a much stronger intellect than mine very interested in software testing. “Teach it to me!” he chortled, with relief and delight.

We both agreed that the “right” answer the interviewer posed was a complete joke, and he was justified in concluding that this was not a company he wanted to work for. That company lost out on hiring one of the best problem-solvers I know.

“Teach me more about these testing mnemonics!” he admonished as we parted ways that day.

* * *

How did you generate so much thorough analysis so quickly? I’ve been here for three weeks, and you’ve been here for three days!

I was doing analysis work with a friend of mine, who like my friend above, is also a brilliant programmer. When I was still in short pants, he was creating artificial intelligence programs in Lisp.

“Part of the reason is that I have your three weeks of work to draw on.” I replied. “Of course, but you have touched on areas I hadn’t thought about at all. I would have thought of them eventually, but how did you think of them so quickly?” he said with a questioning grin. “I use models”, I replied, and brought out a testing problem to demonstrate my thinking. He grasped the object I use in that problem for a second, then looked at me, handed it back and said: “Oh, I see how you do it. Raw observation is very slow. Your models help frame your thinking and speed up observation.” He didn’t even try to solve the problem, and in his characteristic way, rapidly understood a lesson that had taken me much longer to learn.

“Good work” he said with a chuckle as he got up to leave for home. My speech on the wonders of James Bach’s San Francisco Depot (SFDPOT) mnemonic would have to wait for another day.

* * *
Models are useful for generating ideas when solving problems. They aren’t fool-proof though. I am careful to not ignore, but to absorb and test my first instinctive thoughts when solving a problem. Often, that intuition helps, but sometimes I discard it, as careful scrutiny denies the plausibility of the initial thought impulse. Intuition can be both powerful and perilous. There’s nothing wrong with intuition, as long as you can make your conclusions defensible.

It doesn’t hurt to complement intuitive thought with a model, and I find using both helps kick start my thinking processes. Sometimes, the most lucid and effective ideas sprout out of nowhere from a sea of wonder and half-baked thoughts. They frequently pop into my head as I am using a model, and they often turn out to be unrelated to my current train of thought, but incredibly useful. I don’t ignore them merely because I can’t explain them. Instead, I try to adapt my models to increase the chances of them occurring.

Models also add consistency when solving problems. Instead of a formulaic, step-by-step, “one size fits all” approach to solve a problem, they can be used to consistently remind our brains to look into diverse areas for clues and impulses. This is much more powerful than a rote process that may, or as in most cases, may not work. It greatly increases our chances of coming up with a unique solution for a unique problem. I find that after a considerable degree of rumination, I have an “aha” moment, but using a model to complement my natural thinking usually brings me to an “aha – why didn’t I think of that before?!!” thought much more quickly.

Models also help build confidence. If I am posed with a difficult problem, and I have no idea where to start, I can nervously waver and pause, or I can jump in with confidence and use a model. It doesn’t matter if it’s the wrong one, the thinking drives out the fear and uncertainty and leads me in a direction towards solving the problem. Fear steals energy, while models give me a place to start in the face of uncertainty. I constantly learn from experience, and develop new models when I learn a new lesson. Variety of models makes the problem-solving space much richer, and less intimidating.

A good friend of mine is a firefighter, and he also uses mnemonics in his work. His son had recently taken an emergency services technician course, and he was drilling his knowledge one evening. They both have been taught mnemonics to help frame their thinking in emergency situations. In fact, they and others have told me that attempts at scripting this kind of problem solving ends up in farcical situations, or worse, loss of life.

That sobering thought aside, it was a pleasant evening together, and I sat amazed as they rattled off one mnemonic after the other. They even had mnemonics within mnemonics. While one would set the scene for a new problem, the other would rattle off a mnemonic, and explain how they would use it to observe something different to help them get a fuller picture of the problem. They are terrified of having narrow observation within the small window of time they are usually faced with. “But now you notice this in your patient. What do you do now?” From within one letter of one mnemonic, the other would rattle off another mnemonic to use to help jog their observation and thinking to solve that problem, with speed, coherence, and confidence. They would also frequently throw the mnemonics aside altogether, and use other heuristic-based thinking techniques.

They were both in the heat of the discussion, each trying to trip the other up, and the father was attempting to pass on wisdom and experience to his son in a curious Socratic style. I chuckled to myself. I felt if at that instant I were to keel over with some health-related malady, I would be in very capable hands, and someone would learn a new thinking technique in the process.

agile

What’s New?

When I was in university ten years ago, it seemed that Total Quality Management was all the rage in business schools. Terms like employee empowerment, quality control, quality circles, lean manufacturing and statistical process control seemed to be on the lips of the wise.

Nowadays, I am seeing the same ideas that were put forward by people ten or twenty years ago called fancy things like “Agile Software Management.” Some of these recent thoughts would seem at home in Feigenbaum’s Total Quality Management from the 1950s. Some of them were good ideas in the past, and are good ideas now, even if they are rehashed under a new name. We see this kind of behavior repeated over time.

Many early large organizations in North America were railroad companies. They were among the first to discover that large companies faced unique problems. Over one hundred and fifty years ago, Daniel McCallum defined a management structure that was widely adopted and that we are familiar with today. The next time you look at an org chart, feel free to silently thank him.

Prior to the great railroad empires of the 19th century, mercantilism was a dominant economic belief. Adam Smith and his colleagues attacked this model and wrote of a free market system. The system was driven by consumer wants and competition. To survive as a viable business in this system, you needed to be productive. This obsession with productivity is still with us today.

In the early 20th century, Frederick Taylor and the Gilbreths made this into an art form. They employed Time and Motion studies; they lectured and wrote on how workers could increase efficiency, and thereby improve productivity. Today, many decry the work of Taylor and the Gilbreths for various reasons, but we still feel the pressure of efficiency. In fact, not unlike the poor bricklayers, assembly-line workers, or coal shovelers, knowledge-based workers find themselves in a culture obsessed with speed of development and zero defects. Today, we give it names like “Lean” or “Evo”.

As organizations grew larger, thinkers like Henri Fayol came up with systems to deal with the administrative side of management. It emerged as a system of planning, organizing, directing, co-ordinating and controlling. Max Weber was an early thinker who felt that organizations should be socially responsible. It’s interesting that almost one hundred years ago Fayol and Weber expounded on beliefs that we hear echoed today.

Somewhere along the way, management thinkers realized that people do the work, and that no matter what kind of machines we have, or automation we employ, an organization is only as good as its people. Fritz Roethlisberger described the all-too-human Hawthorne Effect. His contemporaries felt that human factors contributed to productivity. Abraham Maslow introduced his hierarchy of needs theory, and now we have volumes of work in this area. Today we are well aware of human factors in organizations. Just talk to someone in Human Resources about their typical day.

Management theory began to embrace innovation as a key component. Over the past twenty years (at least) this has been a major theme. When was the last time you attended a management seminar that didn’t extol the virtues of Toyota, Federal Express and Wal-Mart? Often, the ability to harness technology is trotted out as a factor in the success of those companies. (As a side note, Toyota’s quality scores are carefully examined, but their product recalls are not usually discussed in those venues. Toyota makes a great product, but they aren’t perfect. I have jokingly threatened colleagues that if I hear one more software process wonk drag out Toyota as an example, I will scream. It is an empty threat though – I haven’t yet screamed in a meeting.)

In software development circles, we are faced with some interesting new ideas. Let’s examine the Agile software development movement. The Agile Manifesto was thrust on the scene by respected software thinkers over five years ago. I, like many others, was excited by this statement. Here was an acknowledgment of the humanistic endeavor of software development. It not only made sense but it resonated with my beliefs. Something sounded familiar, though. Agile Manufacturing was an idea that was brought forward in 1995 in a book: “Agile Competitors and Virtual Organizations: Strategies for Enriching the Customer” by Steven L. Goldman, Roger N. Nagel and Kenneth Preiss. According to wikipedia, these are four key attributes in the book:

  1. delivering value to the customer;
  2. being ready for change;
  3. valuing human knowledge and skills;
  4. forming virtual partnerships.

Those sound like good ideas. I don’t know if they inspired the Agile software development movement or not, but even if they did, it doesn’t detract from the points of Agile manifesto. It may just tell us that a lot of people have been struggling for a long time with the complexity of organizations trying to work together towards common goals. Maybe they just came to similar conclusions.
Another new idea on the software development front is adapted from lean manufacturing. The Poppendiecks are at the forefront of this push in the software movement. I like a lot of what Mary and Tom say. It is thoughtful, makes a lot of sense, and they have the experience to back up their claims. However, I get concerned about doing a copy/paste from the manufacturing world and applying it to software development. David Benoit sums this up eloquently:

[I]t still amazes me how many people don’t seem to understand that software development has very little in common with real-world-object development. The creation of software is most like the creation of art. To be good it must be inspired. To be of interest it must be unique. To be worth something it must be different.

As time marches on, I find I am faced with an uncomfortable truth: software development, like any other human organizational effort, is a complex endeavor. There aren’t formulas to guide us that will always work in every situation. And, as much as we have progressed into a modern age, we still struggle with the same things organizations have struggled with for centuries, or millennia. We are still re-inventing the same solutions and are still trying to come up with new ones. Sometimes what is “new” is merely the “old” with a different name. Maybe the author of Ecclesiastes was right when they said: “there is nothing new under the sun.”