By Jerry Weinberg
In addition to her many other talents, my wife, Dani, trains dogs. Her speciality is “problem dogs,” which really means “problem owners.”
Rather than retrain the dogs, Dani retrains the owners, who then retrain the dogs. In that sense, her dog work is very much like my program testing work. When people come to me with “problem programs,” I work on the people, not the programs. After all, it was the people who made the programs into problems.
Of all the questions dog owners ask Dani, the most frequent is, “When do we start training the puppy?” Unfortunately, few people ask me the corresponding question, “When is the right time to start testing a program?”
What is the right time to start training a puppy? As Dani explains, the question is meaningless because “you start training a puppy the moment you set eyes on it, whether you intend to or not. And the puppy starts training you at the same time.”
The same is true of testing programs. You start testing a program the moment you start to think of the possibility of writing a program. Actually, it starts even earlier.
How can program testing start even before you start thinking of writing the program? Think about the dog owners. Their troubles arise from bad habits they’ve acquired over a lifetime, before they ever set eyes upon their new puppy.
Your good and bad habits will have more influence than any other factor in the success or failure of your program testing efforts. It is no accident that certain programmers and testers perform consistently better than others in the production of well-tested programs. Those who believe it is an accident will learn nothing about program testing until they convince themselves it is possible to learn how to do a better testing job.
I’m not denying that a measure of basic intelligence is required to be a successful program tester. Even some forms of “genius” might help. But those who subscribe to the “native genius” school of programming or program testing should study the lives of geniuses.
For example, Einstein once remarked that he really didn’t do anything but take the work of others and put it together in some way that had not been done before. If we are to emulate Einstein and put together the work of others—instead of starting from scratch with each new program—perhaps we can approach the genius level in our testing.
Scientists know that there is honor, not shame, in taking the work of others— probably others who are smarter than ourselves—as the starting point for our work. For the most part, this previous work is available to us in libraries of one form or another.
Libraries may store millions of useful programming ideas, but their very size can make it difficult to know what they contain. In order to make effective use of libraries in program testing, we have to know what kinds of libraries are available, and we have to know what those libraries contain.
One library that every programmer uses is the one contained in the programming language. Even if your program is in binary machine code, you’re using a library— the one the machine designers thought useful for programming. It is theoretically possible to design a general purpose computer with but a single instruction—a kind of “conditional branch and subtract.” Therefore, any instruction set bigger than that must be considered a library—a way of giving the programmer a set of pre-tested building blocks.
Higher Level
By “higher-level” language, we essentially mean a language with a library that allows programs to be built with fewer building blocks. Fewer building blocks means fewer connections between building blocks—which in turn means fewer places to search for errors. Unfortunately, many programmers are not well acquainted with the contents of their language library.
Here’s just one example. A programmer came to me with a program that was mysteriously losing parts of strings. I helped him narrow down the problem to a statement that was trying to replace two characters with asterisks, but when the original string was longer than 10 characters, the excess portion was truncated. When I asked him why he had used the “magic number” 10, he explained that at the time he wrote the statement he didn’t think he would have strings longer than that.
He didn’t know his language library contained the ability to do substring assignment. Had he known this, he could have written a much simpler and more reliable statement. That approach—because it was a more direct expression of what he wanted to do rather than how he wanted it done—would have prevented the error and other errors from ever occurring in the first place.
Another genius, Newton, said he could see so far because “I am a midget standing on the shoulders of giants.” Dick Hamming once told me the programmers are more like midgets standing on the toes of other midgets. It may be true that our language designers lack the mental stature of a Newton, but that’s no reason to stand on their toes by ignoring the hidden libraries they have created for us. You can still see farther standing on shoulders than on toes.
But how can you come to know the contents of your hidden library? The obvious answer is to read the manual. Ugh! We complain that manuals are badly written, and many are. But a more serious problem is that manuals are written for reference, not to be read like novels. If you know there is such a thing as substring assignment, and you know its name, you can look it up and learn how it works. But if you don’t suspect such a function exists, how can you find it in the manual?
Another problem with manual reading is their emphasis on how things work, rather than why you might want to use them in the first place. They are solution-oriented, rather than problem-oriented, leaving their readers to bridge the gap between what they’re trying to accomplish and how the language could make it easier and more accurate.
Still, there are ways of using manuals to overcome some of this shortcoming. Two or more testers can play a game. One takes the manual and reads the name of some feature to the others, who must then describe how that feature works. “What happens when we execute this?” Everyone writes down an answer, after which the example is executed and the competitive members of the team can award points for correct answers.
This game increases everyone’s familiarity with parts of the hidden library—or any application, but still leaves the “why” largely unanswered. The game can be extended to “why?” by asking each player to come up with a problem for which his feature is the best solution. Or, in testing, “what problem was the programmer trying to solve?” For substring assignment, a problem might be “to insert asterisks in the third and fourth positions of a string,” or “to replace a blank at position N with a comma.”
Technical Reviews
The technical review is even more effective at vocabulary building. Technical reviews have the further advantage that managers who might be troubled to see programmers and testers “playing games” have little trouble accepting a technical review for its obvious function of finding errors. In my experience, however, the long-range error-prevention benefits of technical reviews far outweigh their short-range debugging efforts.
When several people sit down to review code, each person can quietly witness all the good techniques used in that code. They can also witness poor techniques, and if anyone in the group happens to know a better technique, all can learn it. And, if the technique has a fault, then can all learn how to recognize and even correct it.
Moreover they can learn these things without having to embarrass themselves by confessing they didn’t know them before the review.
Another problem: though manuals may be comprehensive, they tend to put the emphasis on what is most difficult to explain, rather than what is most useful to know. Because technical reviews are focussed on current work, the learning from reviews tend to be more relevant than what you can learn from playing games with manuals.
There are, of course, other types of program library besides the hidden library of the programming language. There are procedure libraries, subroutine libraries, libraries containing tables of common data, utility libraries, libraries of parameters to direct those utilities in common tasks, macro libraries containing common code sequences, data dictionaries, and the inventory of existing applications—often a huge library of useful code for reuse.
The amount of useful, pre-tested material in such libraries is so overwhelming that programmers and testers sometimes forget about that other sort of library—the kind containing good old-fashioned books. A caution, though. Even if we could somehow guarantee perfect copies of the author’s examples, we must remember that examples presented in a book are ordinarily designed—or should be—for maximum teaching effect, not for maximum generality or performance. Books are terrific ways to obtain new ideas of what be in other types of library, which is where you should look for hopefully reliable pieces of code and design.
The subject of early testing would be incomplete without the mention of one more type of library—the library of work habits you carry around in your head, or wherever it is that work habits are carried. For instance, have you ever noticed how pig-pen workers invariably spend more time debugging than those with tidier habits? I say this not as Mr. Clean, but as an old pig-pen programmer and tester myself. It took me thirty years of effort to clean up my act reasonably well, but I’ve never completely eliminated slovenly habits of my youth. Still, little by little, I’ve eliminated slovenly, lazy practices that had previously wasted thousands of hours of precious time.
Good Habits For Software Testers
The Chinese say, “a journey of 10,000 miles starts with a single step,” and even one fault prevented is a step in the right direction of more reliable software. So, here are a few of the pre-programmed habits I’ve noticed in effective program testers:
Keeping a Journal
Like all professional engineers, professional programmers and testers should keep a bound journal in which to record all those tiny but significant events that occur every day. (I prefer a bound journal to a digital record because it prevents me from revising history to make myself look better than I really was.)
Ideas, observations, events, successes, failures, puzzles—all are grist for the journal. You don’t have to realize the importance of something at the time you write it down. Indeed, the puzzling things are the most important to record. Often, faults are found only when they have managed to repeat themselves in a dozen ways, and only with a journal will you have all the necessary data in one place. Memory isn’t sufficiently reliable. Scraps of paper don’t stay in one place. Your digital device may be in the next county when you have an idea, so carry a journal.
Record Dates and Times
Someone once said, “Time is God’s way of making sure everything doesn’t happen at once.” Effective testers use God’s great invention to ensure they always know the order of past events. Put the date and time on everything you write down, everything you put in the computer, and everything the computer puts out. Record both the time of writing and the time of the event itself. It’s an investment like insurance—it may seem a bit troublesome in the moment, but when the accident occurs, you’ll be glad you paid the small price. And even if there’s never an accident, you’ve bought some peace of mind.
Standardizing Formats
When I write down the same thing in different ways, I always seem to have trouble later. Europeans write dates as day/month; Americans use month/day. Obviously, either method is workable, but used together they make a mess. After living in Europe, I came to realize that the European method was clearly more logical (for sorting order, at least), but that didn’t mean I could adopt it when I returned to the US. It confuses Americans. Instead, I evolved a standard (for me) dating method— 11 October 2017) which is far less likely to confuse either Europeans or Americans than 10/11, or is it 11/10? My standard method also forces me to write the year. You’d be surprised how many errors I’ve had that survived more than a year—or maybe you wouldn’t be surprised at all.
Over long periods of time, my ideas about certain standards have changed, which some people use as an argument against any kind of standard. But one advantage of standard formats is that they allow easier transformation to a new standard, should that be necessary. And, if I’ve dated everything and recorded the change in standards in my journal, I can always figure out which standard applied to a particular item.
Filing
One of the areas in which I still could use some improvement is my filing. My journal often helps as an index to filed items, and dating every filed item was a great leap forward. But I still find myself too frequently in the situation where I know I’ve put away the solution to a problem but can’t find it in my files.
Clearly, I’m in no position to tell other people how to file their myriad documents —test results, bug reports, articles, correction letters, and so forth—that might be needed when testing a system. But at least I speak with the experience of a common person rather than a filing genius (like a former secretary who filed the cans of food in her kitchen alphabetically).
Probably the most important principle I’ve learned is to think, when putting something away, about the question I’ll be asking when I want to retrieve it. That question is my best guide as to where to file it. Recently, when I’ve been unable to reduce an important item to a single question, I’ve taken to filing a reference to the item in extra places. Or, if it’s filed digitally, I add at least one keyword for each possible question.
Throwing Things Away
Filing multiple references tends to compound another problem of mine, based on the universal law: “Files Fill.” I’ve noticed that good testers have a sense of when to rid themselves of things. I don’t have this sense, and it has always cost me much wasted effort when searching for something.
When I used to work for IBM (which stood for “I’ve Been Moved”), every time I packed my files to move I would purge about 1/3 of my filed material. Now that I don’t move as frequently, I tend to drown in my files. Once in a while, though, I become seized with a cleaning fit and go through a drawer or two, or tidy up a few folders. Still, I’m still trying to learn something good about filing—or, rather, unfiling.
Learn Something New Before Sleeping
My father was an extraordinarily smart man. Somewhere in my early years, he told me that he would never go to bed at night unless he had learned something new that day. Somehow that made sense to me, perhaps because I wanted to be smart like my father. For whatever reason, I have followed this habit for my entire life. I was never one of those people who felt he could learn only in a classroom, so every night, before hitting the pillows, I review my day for learnings. My journal helps. If I cannot think of anything worth writing down, I go to my library and grab a book. Or, nowadays, I’ll go online and browse until I find something of value.
Sometimes, it’s something new. Others, it’s a reminder of something I used to know, but have forgotten. Once in a great while, I try to retire without this learning, but I’ve discovered that this habit is so ingrained that I cannot fall asleep. So, I get up from bed and start the learning process over again.
Learning From Others
Which brings me to the final habit on my list, undoubtedly the one that has done more than all the others to make me a better tester.
I don’t know how I acquired this habit—I certainly didn’t have it when I was in school. Somewhere along the line, I developed the habit of listening to other people. And watching them.
Of course, I try to learn from my own mistakes, but my ego makes it hard to learn from them. Besides there are so many other people making mistakes that I can learn even more from their blunders. I think that’s how Dani learned how to train dogs—and dog owners.
