Who hasn’t heard of end-to-end testing? It’s been the standard prescription for success in the testing trade for decades. There’s lots of information readily available on the practice, and it’s hard to imagine anyone skilled in business process testing unfamiliar with the term or the methodology behind it.
I’m not a fan: End-to-end testing was never great; it was just the best we had.
Throughout my career in software development and testing, and later business process optimization, I came to know end-to-end well. The logic behind it is easy to understand: first document business processes, then ensure they work as designed when moved into automation tools like ERP systems by performing them from start to finish. Whenever change is scheduled for the underlying system we run them again to help ensure against unintended outcomes. Who would argue against that?
Don’t get me wrong here. I am not advocating against business process discovery and documentation, value stream mapping, or any activity associated with helping organizations optimize the intersection of people, process and technology. I did this kind of work for many years and believe in it passionately. However, I am going to argue against it for the sake of SAP regression testing. End-to-end remains relevant for disciplines such as UAT and smoke testing, but new technology means it is no longer necessary as a methodology for robust SAP regression testing – and may be counter-productive.
End-to-end testing
End-to-end testing became a thing because the test industry couldn’t envision another way to do it. At the time, best-practice depended on well-documented, linear processes for outsourced manual testing. When using automation, there were few other ways to ensure the availability of test data and other functional prerequisites (scenarios) necessary for the performance of a test. In fact, it’s no surprise test data management solutions often cost more than test automation solutions, and a significant portion of their claim is reducing this exact problem.
But the misery of end-to-end testing doesn’t end there! When tests are end-to-end, each test case becomes much larger and longer, often filled with repeating steps to account for prerequisite actions and other non-value add activity. Coverage at the end of test cases becomes “starved” since failures earlier in the end-to-end execution prevent the full performance of the end-to-end process. This leaves development teams stuck in a find-and-fix loop early in the business process until testing time runs out or a release is delayed.
Additionally, coverage in general is difficult to maintain. A change to the behavior of a common prerequisite step in a complex business process may cause a test-breaking ricochet across the entire test documentation and automation library, instantly invalidating even years of existing work. And, let’s not forget that end-to-end testing cannot begin until the entire infrastructure of every component part of the business process is fully assembled in a QA environment, pushing regression testing and defect detection very late in the software development life cycle.
So, how do we get around these challenges?
Challenges
At least within SAP, a recent innovation called Robotic Test Automation (RTA) changed the available options for regression testing. RTA creates a test harness by recording actual utilization of a production system and combining this with a data mirror-image copy of the same system. The resulting test library is unconstrained by typical testing requirements such as prerequisite test data and business activities because it begins with a test environment already in a production-like state, with a specific set of tests built from real production business activity.
RTA takes advantage of the fact most business processes are not infrequent ‘one-offs’ but are instead executed many times in a given period. At any one moment, multiple concurrent executions of the same process may be in-flight, with each having reached a different point in the workflow. RTA captures each part of all such executions and uses this data to test business activities that would not have been reached in a single end-to-end test due to early-stage failures.
In effect, the individual business activity, rather than an entire end-to-end process, becomes the atomic unit of any test. This eliminates the issue with starvation of testing coverage towards the end of long end-to-end tests and in the process decouples the data issues typical of an end-to-end approach.
What’s more, where end-to-end requires the definition and execution of a specific test for every variant of a single process, RTA automatically validates all versions occurring in the live production system when the test harness is created – a huge advantage for high-volume activities like sales and distribution, or materials management, where many valid process variants exist.
Test data management also becomes a non-issue, since the state of the production environment and the recording of activity within it walk together as the regression test harness. This benefit alone often justifies the investment in RTA, since testing for major upgrades and migrations requires comprehensive regression testing well beyond the scope typical of an individual release or periodic change.
When facing testing challenges in SAP, don’t settle for anything less than a robust, fully automated, more comprehensive approach to eliminate the constraints imposed by end-to-end testing. Demand regression testing without scripting, test data management, and other non-value-add activity.
To learn more about Robotic Test Automation – a low-touch, high coverage approach to regression testing – get in touch with us.
Check out all the software testing webinars and eBooks here on EuroSTARHuddle.com
