Category Archives: test ideas

idea generation, models, test ideas, web applications

Combating Model Myopia in Testing

Part 1

In my last post, I posed a testing problem. This post begins to address how I view the problem, and my motivation behind posing it.

Models

In software testing, a “model” is a representation of something we are trying to understand. It might be a representation in our mind (an idea), a diagram (like an Entity-Relationship diagram), or it might be a physical representation. A model helps us think of a system or concept in a certain way, and as a representation is not going to be perfect or complete. We can have many models for the same thing – our only limitation is our imagination.

Here’s an example:

How many ways can you model a web page? I can think of at least three categories:

  1. as an end-user, or consumer view where I am using a web page for some purpose. I see the webpage as having text, colors, pictures and hyperlinks. I visualize that web page, served up in a web browser when I think about a particular page I find useful, and want to go to it. I interface with it and relate to it as it is shown through a web browser.
  2. as a web developer. I tend to develop web pages using a text editor and do all of the HTML, CSS, etc. by hand. I interface with the web page with a text editor, and I visualize it and relate to it by the HTML view which is made up of markup tags. I interface with the web page primarily through a text editor.
  3. as a web designer. I either have a visual picture in my mind of the layout and design of a web site I want to create and I follow that, or sometimes I draw it out using a tool. Even a rough sketch of a general layout on a piece of paper or on a whiteboard will convey that idea. I visualize the web page according to that layout.

These are three simple examples of modeling a web page. I added a couple of dimensions to help describe the models – a user, and an interface. Each one is different, but I may switch between all three as I’m developing a web page, or when I’m testing.

Since I often test web applications, I think of these different views a lot. I think of the first two very often, especially since I did a lot of web development in the mid-late ’90s. When I started writing web applications, I also learned to model from a programming context. My HTML tags, page layout, and content would be generated according to program logic, so I had a fourth model category derived from my role as a web application programmer. Understanding all the components used in rendering a web page is important to me when testing a web application. Everything from the web browser to the application server, web server, database if one is used, any drivers that are used, as well as the program code design is important information that can help with testing. I actually visualize what is happening across the entire application path when data is input into a web page, sent to the web server, processed, and the page I am on is updated. Even a simple HTML page has an interesting path. From the web server we have a text document that contains HTML markup, with certain tags that denote it as such. When a web browser asks for that page, the web server takes the tags, processes them and sends them via HTTP to the machine that is asking for it. The web browser takes that HTTP response, interprets it, and if it comes through without an error, it processes the HTML and displays it in a web browser.

Why is this important?

Modeling applications like this helps me imagine areas of potential problems based on experience and technical knowledge of the various technologies used, and also helps me narrow down or get repeatable cases for bugs I find in the application. It also helps me generate different testing ideas instead of one “happy path” based off of requirements documents, I can think of other potential usage of the software that might have the potential for problems. When I work with software users in their own environment, I am always amazed at how they use the software in unintended ways which they are perfectly happy doing.

What other models can you think of when testing web applications? Who are the users? Not all of them will be human. What are the interfaces? Not all of them will involve a Graphical User Interface like most modern, familiar web browsers.

But I’ve Run Out of Ideas!

The problem I described was something I deliberately chose as a modeling problem. Notice that in the energy industry, when I got a failure on a search attempt that contained the word “well”, I immediately had a suspicion of what the problem might be. But if I was in another industry, “well” may not be in my common vocabulary as a noun, but primarily an adverb. (Note that language is important when testing.) In my description here, we have at least two modeling categories of the software. One as “enterprise search”, and one as “my energy company search”. The ideas that each model represent are going to be subtly different. That subtle shift between models can be the difference between tracking down the source of a bug or not. But if I didn’t have energy experience, how can I find that model that will help me track down the bug?

I call this so-called running out of ideas “testing model myopia”. If we were omniscient, we would know all the possible ways to model the software, and we’d know where all the bugs were. If we were semi-omniscient and merely knew all the possible ways to model our software, we’d have an easier time. Since we aren’t, we have to use our skill as testing thinkers to break out of mental blocks. This can be very difficult if the entire team has testing model myopia as I described in the problem. When under project stress, with constraints on a development team such as time and resources, a tester may have to build a case in order to be able to pursue a testing idea. The predominant model of the software as shared by the team can be very entrenched, and it can be threatening to have that model questioned or analyzed. Nevertheless, as a tester, we are what Jerry Weinberg has described: “someone who can imagine that there might be another way.”

One of the first places I would start to tackle this problem is to imagine other models of the software that we aren’t addressing when testing.

Stay tuned for Part 2.

scenarios, test ideas, testing

Well, well, well

Living in Alberta, I do testing work in the energy industry, or with companies that support the energy industry with software or services. Alberta is Canada’s energy capital, the largest oil and gas producer in the country. Rudyard Kipling said when he visited Alberta that we “…seem to have all hell for a basement”.

As a kid growing up in the ’70s and early ’80s, it seemed every farmer’s field in our area had a pumpjack servicing an oil well, or had a gas well feeding into a pipeline. In the energy industry, the term “well” is common, but vital. Wells are one source from which oil and gas flow, so a lot of resources are put behind drilling to find new sources, or to extract more oil and gas from known producing reservoirs beneath the surface.

I was recently testing an application that serves this industry. It had a neat search function, so I typed in the name of one of my test wells I had set up. I immediately got an error, with messaging informing me that the application could not complete my search because it had “common” words in it. The example it gave was “the”, and the messaging said to try it again without a common word. I immediately suspected that it was filtering out the word “well”, so I repeated the search without the term “well”, and it worked like a charm.

I logged the issue and the developers investigated the cause. (I was off-site, so I didn’t get all of the details I normally would.) They told me that this problem was due to the database server configuration that filters out common words on queries. The software that managed the database had a feature that they tought would be helpful in many contexts. Sadly, this feature the database vendor thought would be helpful caused problems in our context. An assumption with the term “well” as always being a common word for all contexts caused this feature to be useless in our context. Context is important, and assumptions are frequently a source of bugs.

In what other contexts would this be a problem? I can think of at least one. If I had this feature turned on, my database and I worked for a company that managed water sources, I imagine I would run into problems as well with searching in this way, since water wells are also common, particularly in rural areas. But this is a context where the problem is going to be obvious.

When I saw the error message, I used inference to generate the best explanation. My inference immediately told me that it was probably filtering out the term “well”, so I tested a conjecture (“they are thinking well is a common word”) and as far as I can tell, it turned out to be the case. But what if I was in an industry that was one step away, where people aren’t talking about wells, and reciting well ids, or talking about drilling, extraction, and other tasks related to wells? That’s where these kinds of problems turn into what might seem to be a sporadic or “unrepeatable bug”. We know there is a problem, but it isn’t obvious to us because our context doesn’t make it obvious. It would be obvious to someone with experience in the right context, it just isn’t obvious to us. Our observations are filtered due to our context, our knowledge and our biases. How do we get beyond this when dealing with a problem that seems mysterious to us?

Here is an example that I am dreaming up to fit this scenario. Let’s pretend that we have written a cool search tool that is fast, accurate and works well in corporate environments. We have sold this tool to companies in the transportation and financial industries. We also sold it to a couple of companies in the energy industry. Our tool works like this: it is installed on application servers in the corporate head office and is managed by the IT department. It uses a database to store information gathered from disparate sources, and updates the data found on the network, intranet, etc, on a set time. It might update every half hour, or on a nightly job. Our customers are thrilled with the tool, they now have a simple interface into this scattered data, but a couple of them are complaining about not being able to find a small amount of critical data. It appears to be lost. You, the reader are the tester working for the search company and are helping investigate this problem.

The team is frustrated, they just can’t get a repeatable case with the information they have. You have tried all your existing test data that you’ve generated and normally use with regression tests, but the software works fine in all your tests. The team feels that they have exhausted testing ideas, but a couple of clients are still complaining, and getting less patient with your company.

Given the information you know from this post, what would you do? Feel free to email me your ideas, and I’ll post a follow up with some of your suggestions as well as my own.