Wim Decoutere spoke at EuroSTAR 2022 at the closing keynote on the topic of “What’s your Test Design Debt? Avoid the Inevitable!“. He explored the concept of test design debt and explain several techniques on how to create more lightweight test cases with the same level of detail and coverage as your current test cases. This post is a collection of some of the questions that Wim was asked at his talk that he did not get to answer at the event.
Do you believe being multiple testers in a team is only for bad?
No, definitely not. I really believe 2 know more than 1 and I’m a huge fan of pairwise testing. I can only recommend working together with other testers to help the number of heuristics in your test toolbox grow even more.
What you should keep an eye open for is that when multiple people work on the same thing, you should come to an agreement on how you will do things. If every person or team just acts like they think is best, you end up with chaos. It’s very hard to align everyone once they’ve found their personal way of working. Better to start right than having to fix things later on.
“Use a glossary in your test cases” Would you recommended a keyword driven framework and/or BDD as glossary?
These frameworks and the keywords they require can be useful to keep in mind when setting up a glossary, though I would not recommend this as mandatory. They’re set up with the goal of automation and once you start using technical approaches, you might scare away less technical people. Keep in mind that testing is not about automation or tools. Those are only helpful to get the test job done.
What approach would you take to fix a large number of test cases that are not linked to requirements? but still use actively in the company?
It depends. What do you want to fix? Why do you need that link?
If it’s for administration and reporting, just create a fake requirement and link your test to this ‘empty’ requirement. Same for defects that you cannot link directly to a test case. Don’t create a useless test case that describes the scenario of this defect, just link the defect to a fake ‘empty’ test case.
If it’s for coverage and impact analysis, you should try to retrieve the missing requirements that are covered with these tests. This might seem like a waste of time, but next time one of these requirements change, you know which test cases are impacted.
When multiple testers work on the same project but on different features, any recommendations how to review each other’s test cases?
A checklist with some basic criteria is a good starting point. The bare minimum is that you understand the test case and know what to do after reading it. You could also use some of the criteria that I’ve presented during my talk. Important is that you come to an agreement on what to focus. This avoids heavy discussions afterwards.
You could train your testers in reviewing test cases by showing them examples of bad test cases (test smells) and discuss with the team what is wrong with them and how they should be corrected.
When talking about lifespan of test, how do you approach test cases written for a mostly immutable feature? Keep and never run, or get rid of them? “How to make this scenario live together with exploratory testing?”
A mostly immutable feature is still part of the system and thus has to be part of your regression set for as long as this feature is part of the system. The amount of effort you should spend on testing this feature depends on the importance of this feature and the associated risk.
I wouldn’t use exploratory testing as an approach for regression testing. Exploratory testing is useful to explore, learn and discover the exact behavior of your system. Once a feature has been explored and you learned everything about it, there’s not much new to explore about it anymore. If you know for sure it is not going to change –and this is not an assumption- the test can safely go into your regression set (which can be automated).
How about the effort needed to do this review f and maintenance?
There exist no disadvantages to reviewing. It may seem to delay things because it takes more time before you actually start building something visible, but since you work with better test cases, the time you’ve ‘lost’ will be won in multifold later in the project.
If you do things right from the start -thus keeping in mind test case maintainability- you reduce your test design debt and thus the cost of test case maintenance will be a lot lower.
Did you somehow quantitively measured benefits / effects of your criteria?
No, I never measured the number of redundant steps that were never written or the increased readability. It is an interesting question though so if somebody has some spare time, let’s do some research together!
Would you recommend a mind map tool to create a blueprint of the test to visualise all alternatives?
Would you recommend using e.g. a Mind Mapping tool to visualize the Test Case Blueprint? or another tool?
It’s not about the tools you use, but about the processing that goes in your mind and that you share with other people. If a mindmap is helpful to you, you should definitely use it. Make sure that everyone involved has access to this blueprint, this includes licences to specific tools or access rights to certain shared drives. Don’t forget about readability. Don’t make your blueprint too technical because you scare away part of your stakeholders.
If we have a combination of automation and manual testing, what should be the approach for documenting test cases and how much should we document? Should we only document small test cases and implement E2e test case as code?
As said in my talk, the order should be in the execution schedule, so also for your automated test cases, you should code small test cases/scripts, which you execute in the correct order.
How would you optimise / adapt those criteria / advises for automated tests (non-human execution)?
I strongly believe that the first step of test automation should the design of test cases, where you apply the practices I’ve shared. Designing test cases is not the same as setting up a full decision table or elaborating all equivalence classes. It’s about what happens in your head and the questions you ask yourself about the system.
Since test case automation is a form of coding, you can have a look at design patterns for code and in particular design patterns for test automation code. Dorothy Graham & Seretta Gamba recently published an ebook on test automation patterns on the Eurostar Huddle.
What about testing a big workflow in regression test? Do you still recommend to create small test cases for this case?
Yes, since you use small test cases to reconstruct this big workflow in your test execution schedule.
Clear language: what is correct? QA engineer, software tester, tester?
QA is more than testing alone, so I never use any term referring to QA to describe what I do.
Since our (yours and mine) main domain is software, I prefer the term software tester over tester, as the latter is more general applicable.
What do you think about the usage of “and/or”?
AND/OR is not specific and should be avoided at all times. It is popular in natural language, but you cannot do anything with it from a tester perspective. AND, OR, XAND, XOR and other logical operators should be used correctly.
Test data parametrization leads to different test scenarios. How this can handled within same test case. Do we need multiple tests to handle different test scenarios.
If you use parameters, would you create separate test cases for bad weather scenarios or would you add the result as a parameter?
I would only use parameters on the input side and only if the different inputs result in exactly the same behavior in the system.
I would not parametrize the results of your test. If a different result is expected, you’re testing a different objective and you thus need to create a different test case. It will be very difficult to report on execution if you mix both of them.
For automation different concepts apply, but that was not the scope of my talk.
No numbers in the titles? I found it useful to use number from Jira.
Exactly, every test needs a unique identifier, and if you work with tools like JIRA, these identifiers are created automatically. These ID numbers are not part of your title. What I’m talking about is that you shouldn’t include these numbers or tags in the title of your test cases. So no titles like ‘1.2.1.3.2_XYZ_AAZ_Withdrawal with an expired VISA card’.
Would you cover 1 req with 2 test cases? This is done to split the test, when different feature is involved.
Requirements are typically not that fine grained that you can have a 1 on 1 mapping between requirements and test cases, so for me it’s the normal case that you cover 1 requirement with multiple test cases. Don’t forget to link them correctly for traceability.
If test design debts are existing. What is your recommendation when to update them? Like prior or during the execution of a regression test phase?
If you review or execute a test and you notice something that should be updated from a test case maintainability point-of-view, just do it right away. It probably takes more time to mark the test for updates than to fix the issue immediately.
If your test design debt is that big that it requires some time to fix the problems, try to get some funding for a maintenance project by creating a business case for this.
If you have to make assumptions, like assuming the executer of the test knows how to do something, is it good to call these assumptions out in the preconditions, like “Assumption: Tester knows how to do x”
Asking questions is one of the most important tasks of a tester. Putting this ‘requirement’ for the person executing the test can be helpful to force whoever executes it to ask how it should be done. I wouldn’t use this however to exclude people from executing the test, as this will prevent them from learning something new.
Check out all the software testing webinars and eBooks here on EuroSTARHuddle.com
