You can imagine a test strategy as a map. It should help you to easily find your place. You can manage even without the map; however, your journey can be long enough so that you run out of your stamina and fail just before reaching the end. Imagine that your stamina is your project budget. In a real business world spending real money without delivering a quality product, is not an entertaining fact.
The crucial point about a testing process is that it starts even before the first line of code is written. The quality will not be achieved only by performing UI testing. The quality must be built in, it cannot be added on. You have probably heard this statement before and you maybe thought that it was easier said than done. This article gives you some recommendations in terms of making this task achievable and it shares some advice that can help you to define your test strategy. The emphasis is on structuring a continuous testing process for Agile because Agile does not mean unstructured.
Mostly, when we start to think about the test strategy the following questions come to mind: What should be automated? When should investments in automation be made? Is automation the most cost-effective solution to delivering quality? To help your judgment, think about test automation as a means of transportation. It is very convenient to choose a luxury means of transport, but the key question is: “Which means of transport best suits your case or your budget and timeline?” Sometimes it is silly not to book a flight if you are traveling to a distant place. On the other hand, if you need to reach a not far away destination today and your flight is tomorrow, then the choice is more than logical.
To answer the questions mentioned above is important, but what is more important, is to think about reaching your goal, your desired destination or if we come back from dreaming of a vacation to reality, it is more important to align the complete process that ensures you delivering a useful product on time. Therefore, we need to dynamically make decisions that best suit our case according to the logistic support we have or the conditions in which we find ourselves, like resources, timeline etc.
Since we are talking about a dynamic process that involves a lot of dependencies, and as you might expect, there is no unique recipe. This brings up another important point about defining a testing process. I have been asked several times to help teams in terms of providing the knowledge of the standard testing process. There is no unique answer that will work in all situations. Although some considerations are good to have in mind and based on my experience, I would like to share the following points that can help you define and structure your test process.
Understand your domain
Testing is hardly possible without understanding what end users expect from your application. This is a well-known statement, but still very often put aside and testers are missing the overall knowledge and they are acting more like the hands of a robot that automates what someone imagined. Wrong. Test automation is not all about the implementation of UI scripts which manual testers can rely on.
Perform exploratory testing to “feel” your system
How to learn about your application? Forget the test automation, and manually test your application and do the database testing. The activities will help you get to know the weaknesses of the application, notice the dependencies, see which the key features are from the users’ perspective etc.
Perform the risk-based analysis
When you mark the key features, you can do the prioritization of your scope. Bear in mind that your budget is limited, and your product owner would like to see that the core of the functionalities is working. That increases your mutual trust and improve your further collaboration.
Understand the system architecture
The above-mentioned points guide you towards better answering the question “What to test?” and make us think about “How to test?”. Look at your system under the hood. You should figure out how your system works, what it is happening when an end user performs an action. Follow the requests or messages analyzing all flows. Sometimes, software architecture needs to be adjusted to increase testability of the system or to enable continuous testing. Consequently, even in agile, you should define test architecture according to HLA (High-level architecture) at the beginning of your project and adapt it in time if the requirements are changed.
Proper test automation is more than implementing UI scripts
An effective test automation strategy calls for automating tests at various levels. Testing pyramid illustrates tests distribution between different layers.
Your test script is finished once it is passing on your build server
If your script is working on your machine, you are not done. Your test is not meant to be executed locally, same as developer code is not meant to work locally. You need to take into consideration the entire continuous pipeline. Your low-level test like unit tests should be executed as a part of your build process, while your high-level tests like e2e tests should target real system usage and they should be executed under the deployed system. Therefore, you are done when your continuous pipeline is successfully executed, and your build server is green. And not green just after the first execution, moreover, it should be green after several executions are performed. With this last checking, you are ensuring that your test is not flaky.
Flaky tests are your enemies
True. They are like bacteria, they sometimes spread very fast, like a serious infection. And your test automation will die. You need to react very fast, investigate the reason and eliminate it. Otherwise, the invasion on your test scripts will soon be out of your control.
Make sure that test data is independent
One of the most common reasons why our tests are flaky is the improper management of test data. If you share test data between tests, sooner or later, your tests will fail because they are working with an unexpected set of data. Invest your effort at the beginning of the test automation process and set the solution that supports data independence. There are different approaches depending on test types and some aspects of your project, such as using a proper framework e.g. DbUnit, in-memory databases etc.
Automate your test environment
Apart from data collision issues, environmental issues are a very often reason why tests are fragile. Setting up your test environment is an important building block of your test automation strategy. To avoid that environment configuration and deploying your application are manual and error-prone tasks, think about Infrastructure as Code (IAC) approach. Infrastructure as Code (IAC) is a type of IT infrastructure that operations teams can automatically manage and provision through the code, rather than using a manual process. As the name suggests, infrastructure as code is a concept of managing your operations environment in the same way you implement the code of your application. Rather than manually making configuration changes or using one-off scripts to make infrastructure adjustments, the infrastructure operations are managed using the same rules and structures that govern code development.
Insist on Green build policy
The policy will ensure the full benefit from the continuous process and test automation. The several items presented below should be met for the build to be considered a green build:
- All tests executed during build process pass
- A quality gate concerning static program analysis has a pass
- The quality gate includes an agreed percentage of code coverage is achieved
To ensure the policy is applied, a merge request should not be approved if the mentioned items are not fulfilled. Not having a “Green Build” for an extended period indicates that the product is unstable. The lack of a stable product increases the unpredictability factor in any given release. In these situations, it is impossible to know when the product will be ready to be released without a large testing phase at the end. Not having a “Green Build” leads to a sequential project lifecycle, aka a waterfall.
You should have a quick insight into the quality of your application
The stable tests and the green build policy will help you to have a fast feedback on the quality of your application. To see the status, you should not look at the huge list of the failed tests to be sure of is it a bug or just an environmental issue. Moreover, the preparation of a concise test-execution report increases the test execution transparency.
Do not insist on test automation
Accept the fact that it is more efficient to test some features manually regardless of the test automation trends unless you are testing amazon.com and your team is releasing an updated version every few seconds. This list is by no means all-encompassing, but it can give you a starting point for planning your journey to help you define your test strategy. Enjoy your trip!
