In today’s fast paced industry, where accelerating delivery is the only way to survive, testing everything or every combination is simply impossible. Even in most favourable cases, where we assume that we tested everything, there will be always be a chance of missing something. In both cases, continuous monitoring will assist to uncover missing cases well in advance and proactively before the actual end-users are affected.
Testing everything is impossible
In this accelerated release era, all organisations are concerned about the increasing rate of change in the market, in order to compete & adapt to time-to-market pressures. Overall, project timelines are very strict to survive in this market. Whether using waterfall or today’s agile/devops methodologies, testing everything is not realistic due to having short testing duration. Specially for agile/devops, testing timelines are so strict that test everything is like a dream even with full intention. Keeping any unimaginable scenarios outside, there will be many combinations to test. In today’s different technology platforms, different operating systems, different browsers, different devices, different versions, different networks – there are so many combinations that testing all of these are simply impossible. Yes, leveraging automation adds value and can save an incredible amount of time but even still, testing all of these are not possible to conduct in shorter sprint duration. Even adding more resources is not possible due to project budget. Test early, test often, test fast, test as much as possible, test even in production and test the right things will be the overall objective rather than test everything.
What needs to be done
As testing everything is not possible, testing on short time span must be very much effective to go ahead with production deployment. It should have as wide a coverage as possible. It should be end-user centric (both external and internal end-users). It should be well planned, properly discussed with stakeholders and the product team. Collaboration between the development/application teams and the testing teams is absolutely essential. Both need to work as a single team. While testing, thinking like a real user adds value and helps identifies the priorities. Test scenarios which are business critical, mostly navigated by many users, continuously used, financially impacted are examples of higher priorities. Higher priorities should always be tested first, followed by medium priorities and the lowest priorities last. Lowest priorities are the ones which normally gets missed as they may have the least impact. Re-prioritisation of test cases can be possible while conducting testing to ensure no surprises in production. Combining test cases and parallel testing accelerates the testing process to cover more test cases in a shorter time. Continuous testing which includes shift-left and shift-right testing can assist to increase the coverage.
In shift-left testing started during the early phases of SDLC- software development life cycle to fully utilise the project timings, identifies the issues early and can be more cost effective too. Component driven and bottom-up approach can be used for early testing processes. To accelerate the testing process, service virtualisation or stubbing or drivers can be used. This is for those functionalities which are dependent on other components but may not be developed at the time of component testing. Shift-right testing or testing in the live environment assists to uncover many things from real users and the real environment perspective. Testing in production is mostly about real user feedback and it is costlier. Overall, the best approach is to test as much as possible despite knowing that testing everything is impossible.
Continuous monitoring
First, continuous project monitoring of the test process helps to track progress, compare with original plan and re-prioritise to meet the overall deadline. Though testing everything can’t be done, this does ensure there are no major surprises in production.
Secondly, continuous application monitoring in the testing environment is very important to know each of the components, APIs, integrated components, overall application performance (whether it is functionally working fine, performance is within SLA-service level agreement) for different devices, OSs, browsers, networks etc. Even for interfaces, continuous monitoring shows whether the interfaces are up and running and provides optimum response time. Low priority test cases which are missed can be monitored via continuous performance monitoring. It monitors each of the components, interfaces, end to end application and identifies if any low critical components are throwing errors or having issues. If they are, then these need to be re-prioritised. Continuous monitoring tools provide dashboards reporting, trending charts and send alerts as soon as the problem appears as ell as pinpointing where exactly the problem lies. Deploying monitoring tests both from inside and outside the firewall will ensure full coverage.
Thirdly, continuous security monitoring solutions assists to identify security vulnerabilities, source code, attacks and is very helpful for shorter sprints. Overall, these monitoring solutions assists application owners to understand and resolve the problems. Despite being unable to test everything both in terms of functional and non-functional, continuous monitoring ensures no surprises in production and end-user happiness.
Lastly, today systems can go with wrong in many ways. Full stack observability provides full context across the whole system. This full stack observability keeps track of everything about the whole system. Observability tells why an issue is appearing, how it appears. Though full stack observability in testing environment is still unavailable but getting popular in production environment. It will be helpful for shift- right testing perspective.
Conclusion
A tester’s goal is always to test everything before production. However, as described in this blog, it is not possible to test everything given the limited time available, even when availing of automated testing. Testing with continuous monitoring gives the green light to application teams for deployment in the most efficient way.
See more software testing resources on EuroSTAR Huddle.
