Taxonomy of Software Testing Terms

[Paul]

@Keith – I guess it’s a relatively small sum of money but my point was that it appears people are not prepared to spend it in order to access the standards. I doubt the sum is the sole barrier to testers and organisations using these standards?

[Paul]

@Stephen, Stuart Reid was involved in both the original British standards and the ISO 29119 standards. I’m going to get in touch with him to ask for his views on this conversation.

[Paul]

Stuart actually presented a webinar on ISO 29119 last year: View it here

[Erkki]

A taxonomy works best the items in it could be considered as coordinates for nuggets of useful material, so that any single item could have few coordinate values in place, and then same item could be listed on all relevant places where the material is relevant. Hardest would be the artificial constraint where the taxonomy is assumed to be a single hierarchy and that each material would be relevant on only one point of hierarchy / locatable via only one dimension only. For example there could be checklists for social aspects of test planning: they have both testing life-cycle releveance, relevance to “Test management” and also in “People” dimension. As there really (and fortunately) is no single testing approach guiding we all to assume certain hiearchy in our heads, so there also is no single dimension to have all relevant classifications in it as one value, but more like items float in multi-dimensional space and several coordinates are potential ways to find the material.

It looks like the Non-funtional testing is a subset of Acceptance testing (a simple copy paste error?). I’d suggest to adjust the upper organisation as follows:

————
Testing through the SDLC (avoiding acronymes would be better here, I suppose, like “Testing through the software life-cycle”)
(Each waterfall phase / test level listed here, from Requirements to Maintenance)

Testing types
Functional testing
Non-functional testing
———-

Then from testing methods onward it seems mostly logical, except 2 things:

1. Only place I find questionable is the “results analysis” alone with no context. This points out the missing of the lack of plan-driven testing continuum, or testing life-cycle. Testing life-cycle in all its artificial simplicity is great for organising different testing practices (methods / tools / techniques). For example the TMap had these 5 that encompass useful concepts and logical continuum of important testing activities and decisions — as long nobody assumes they follow each other linearly or that all should be put into some document 🙂

Testing life-cycle
Planning & Control
Preparation
Specification
Execution
Completion

2. The division of testing techniques and methods is IMHO somewhat artificial and they might be grouped as “Testing practices”.

BTW, if the proposed changes in all this thread posts have been implemented somewhere else than the original post, how about giving the link on this thread?

Cheers, Erkki

[Rainer]

Some more classifications of testing methods or kinds of testing (might overlap with threads shown above)

acceptance testing
accessibility testing
aggressive testing
ad hoc testing
agile testing
all-pairs testing
alpha testing
API-based testing
arc testing
automated testing
back-to-back testing
basis path testing
behavioral testing
best representative testing
beta testing
big-bang testing
black-box testing
bottom-up testing
boundary value testing
branch condition combination testing
branch testing
build verification testing
business process-based testing
claims-based testing
coverage-based-testing
code-based testing
combination testing
compatibility testing
complete testing
complex testing
compliance testing
component integration testing
component testing
concurrency testing
condition determination testing
condition testing
configuration testing
confirmation testing
conformance testing
context driven testing
control flow based testing
conversion testing
data driven testing
data flow testing
data integrity testing
database integrity testing
decision condition testing
decision table testing
decision testing
design-based testing
development testing
deversified testing
dirty testing
documentation testing
domain testing
dynamic testing
efficiency testing
endurance testing
equivalence class testing
evaluation based testing
exhaustive testing
expert testing
exploratory testing
extreme value testing
feature or function integration testing
field testing
finite state testing
follow up testing
functional testing
functionality testing
generic testing
glass box testing
gray box testing
guerilla testing
incremental testing
installability testing
installation testing
integration testing
integration testing in the large
integration testing in the small
interface testing
interoperability testing
invalid testing
isolation testing
keyword driven testing
LCSAJ testing
link testing
load testing
logic-coverage testing
logic-driven testing
logic testing
long sequence testing
maintenance testing
maintainability testing
manual testing
maturity testing
migration testing
modified condition decision testing
modified multiple condition testing
module testing
multiple condition testing
mutation testing
N-switch testing
negative testing
non-functional testing
operational profile testing
operational testing
pair(ed) testing
partition testing
path testing
performance testing
portability testing
program testing
prototype testing
random testing
recoverability testing
recovery testing
regression testing
regulation testing
reliability testing
requirements-based testing
resource utilization testing
risk-based testing
robustness testing
safety testing
scalability testing
scenario testing
scripted testing
security testing
serviceability testing
session testing
site acceptance testing
smoke testing
specification-based testing
standards testing
state transition testing
statement testing
static testing
statistical testing
storage testing
stress testing
structural testing
subject-matter expert testing
subpath testing
sympathetic testing
syntax testing
system integration testing
system testing
thread testing
top-down testing
unit integration testing
unit testing
usability testing
use case testing
user acceptance testing
user scenario testing
user testing
volume testing
white-box testing

[Paul]

Hi Erkki, thanks for your input! Can I summarise your suggestions as follows?

1. Testing through the sofware life-cycle
to include:
Requirements
Design
Development
Testing (as a lifecycle phase)
Results Analysis
Deployment
Acceptance Testing
Maintenance

2. Testing Types
to include:
Functional testing
Non-Functional Testing

3. Test Practices

to include Methods and Techniques as above

4. Test Process as above

5. Test Automation as above

6. Testing Artefacts as above

7. People as above

Seven overarching areas like this would be a great starting point for getting a taxonomy that works on this platform but if we can apply relevant tags to all content resources, then the idea of material being discoverable by different coordinates becomes possible.

[Paul]

Thanks Rainer, 159 terms I count there. I wonder where they would sit in a taxonomy as opposed to the alphabetical list? I also wonder how many more terms are commonly used? Have we come to a near exhaustive list?

I’d also be interested to get the thoughts of the few hundred people following this thread but so far not joining the conversation too 🙂

[Geoff]

@Paul, Erikki’s puts forth a schema that is reasonably harmonious with ISO concepts for product (i.e., systems and software) development and reasonably harmonious with the concept of a taxonomic hierarchy where one employes generalization and specialization. Your thread has surfaced the fact that a taxonomic schema isn’t universally understood for its role in a vocabulary of a domain discourse.

Even in the light of the recent turmoil and dissension regarding ISO/IEC 29119 by key principles in the Test community, the concepts of classification developed over the years by ISO/IEC should not be ignored. IMHO

ISO 15288 “Life Cycle” “stages” of a system = Concept, Development, Production, Utilization, Support, Retirement

During the “Development” stage the following technical transformation processes may be executed = Stakeholder Requirements Definition, Requirements Analysis, Architectural Design, Implementation, Integration, Verification, Transition, Validation

During the “Development” stage the following project governance processes may be executed = Planning, Assessment and Control, Decision Management, Risk Management, Configuration Management, Information Management, Measurement

During the “Development” stage the following project-enabling processes may be executed = infrastructure management, human resource management, quality management.

I tend to place “processes” as a central concept in my thinking. Why? Firstly, because I am a Systems Engineer. Secondly, because it is through the execution of a process that transformation occurs. It is how value is created. Processes define “What”, they do not define “How”. “How” is defined in the lower elements of the behavioral hierarchy by process activty, tasks, methods, and procedures. Next, I have structure (e.g., specifications, architecture descriptions, designs, reports, roles, etc…) as these things are inputs and outputs of processes (i.e., technical transformation, project governance and project-enabling). This is how I might start at building a taxonomy. But this is what I am comfortable with, it is how I think. It is how I have been conditioned to think of my domain, mostly through my exposure to ISO/IEC standards.

This figure is an attempt to illustrate how I think about a process and its elements . This view of the model is not complete, it is merely one perspective of my process concept model.

Hopefully one perceives the harmony with classification ideas put forth by Erikki. They are not equivalent, but they are not discordant either. “Test Artifacts” are structural elements. “People” relate to Roles. “1. Testing through the software life-cycle” relates to processes in general. Methods relate to @Erikki’s concepts of testing practices and behavioral aspects of testing. The “How” of a process.

@Paul, you state in your reply to @Erikki “discoverable by different coordinates” which suggests, to me, that an entity might be discoverable by way of more than following a hierarchical classification tree. Was this your intent? If it was, then aren’t you really speaking of an ontology now? A much more complex concept, at least to my understanding.

[Paul]

Geoff, an ontology is a stretch too far I think! What I meant was that a piece of material would be discoverable in more than one way – it would be found at relevant points in the taxonomy and not just one.

[Paul]

Given the interest here in the topic of a standard language, it would be good to get your thoughts on the ISO 29119 standards – Ronan has started a discussion on this. Why are some testers so vehemently opposed to this while others are ‘ok’ with the development of standards?

[Claudiu]

Hei,

I have been working in the past year or so on a taxonomy in regards to software testing information but I believe it can be applied in general for software testing.

It is is a visual shape: The Testing Map

If you consider it useful I can also provide you the whole tree structure. There are around 300 areas on the map.

[Adam Hepner]

I have a quiet proposition: this form of discussion makes me feel a bit… dizzy – it’s hard to follow, does not let itself be linked properly, and change propositions are not really properly managed. The developers have perfect tool for managing collaborative discussion and changes to text documents. It’s obviously GitHub :). I’ve created a quick repository here: https://github.com/AdamHepner/taxonomy-of-software-testing-terms (it is under creative commons, and I will initiate it with content from this discussion in the coming days). I encourage everyone to clone the repo, provide additional explanations, and submit merge requests and/or issues. Together we might make it one of the best repositories of the knowledge in the web!

[Paul]

This is a great idea Adam and you are right, the conversation has many tangents. I look forward to the next version.

[Ronan Healy]

I thoughts this was an interesting article by Donald Firesmith on taxonomy of software testing. He has divided it up into a number of different parts.

[Jeba Qpt]

It involves different phases such as Initial/Planning phase, Requirement analysis phase, Design phase, Coding phase, Testing phase, Delivery and Maintenance phase.
Initial Phase involves gathering requirements by interacting with the Customer.
Requirement analysis phase involves doing detailed study of the customer requirements and judging possibilities and scope of the requirements.
SRS (System Requirement Specification) will be created in this phase
Design phase involves dividing the whole project into modules and sub-modules by doing High level Designing (H.L.D) and Low level Designing (L.L.D).
Coding Phase involves creating source code or program by the programmers by referring the design document.
Testing Phase involves getting clarification for the unclear requirements and then writing test cases by Testing Team based on the requirements.
Delivery & Maintenance phase involves installing the application in the customer place and providing the details such as release notes to the customer.
Read and learn about http://qtpbook.com/2011/08/13/software-testing-terms/

QTP and Software Testing

[Geoff]

When I first read Donald’s paper, his taxonomy didn’t sit well with me. Flattening the multiple dimensions the way the taxonomy does just doesn’t seem to be the right way to conceptualize test. Each to his own.

The contributions of Jeba and several others are excellent examples of the challenges of this proposal. ISO standards have already established system, software and test vocabularies, yet it is clear from several terms used by others in this thread that these international standards are not having the influence they should be having. I suspect it is mostly due to the “pay to play” business model that ISO, IEC, and IEEE observe. The expense of accessing these standards is prohibitive for most. While I do not assert ISO standards to be the “Holy Grail”, they have a place in the ecosystem.

We are all entitled to our opinion, but whether that opinion is acknowledged as having relevance and value by our peers is another matter.

Ronan, Has there been any real movement on your proposal?

[Paul]

Hi all,

This discussion petered out somewhat I guess – it’s pretty daunting but I think @stevean’s suggestion is a good starting point:

Stephen wrote:

@Paul

A Glossary is a good starting point. Thrash out all of the terms and short definitions.
This can then be expanded in two routes. 1 – A Wiki to take the definition and explanation further and 2- a Model to show when and how to use each term.
I think in Testing a Taxonomy is limited as it’s a bit 2 dimensional. A full model could be multi-dimensional showing all the contextual relationships and enabling the user to identify the appropriate terms and methods for each project and situation.

If there’s genuine interest in this being a member-driven project a glossary of terms with short definitions sounds like a good place to start. Any volunteers to get us started?

We could start by having terms defined in this thread and then branching the conversation into different threads as it grows?

Open to other suggestions of course.

[Author]

Posts