The main success criteria for a successful IT/information system implementation project, is that all of the ”end to end” business processes must be able run correctly with the expected output. For the organisation, the most valuable products of the information system, is the transactional data it produces. Our test strategy reflects this with end-to-end testing.
Some takeaways from this article:
- How to learn more about the system under test
- How to establish if our system is testable
- How to share knowledge on integrations
- How to increase test coverage by early end-to-end testing
Many projects include a number of integration points, including those of third party service providers and other types of integrations, where data passes between components. A business process is always dependent on availability of updated master data, in the expected format and with the required quality. The data handling in these integrations must also comply with the required level of security and accuracy.
Is well-documented business processes, business rules and documentation of the integration points good enough as test basis for our test cases? Probably not. The risk is also that the technical implementation will not comply with correct data handling across the system and may risk delay in getting the test environments testable.
With the assumption that some of the end-to-end processes will fail due to incompatible integrations and incorrect data handling, the test strategy should be to plan and execute end-to-end testing as early as possible. The goal for the test team is to gain knowledge about the data handling by the integration points and add to the test basis, for both the integrations and system integrations test levels. Equally important is involving key project resources with this knowledge. The reason is efficient use of test and technical project resources.
Improving quality through testing
One way to improve quality by test is to discover faults and weaknesses in both the test basis and the product under test, and then contribute to do something about it. The most important task for the test team is to record and report findings and share it with the project organization. We are likely to discover different issues in different test phases or test levels.
Requirement Quality
Can we rely on all the available requirements to be good enough to use as test basis, in order to obtain sufficient test coverage? Probably not.
Even when the requirements for each component and the corresponding integration design documentation is up to date there is still a risk that the not all of business processes that are expected to run from start to finish are able to run as expected by the user. With this in mind, we can use these assumptions as input to the test strategy and the test plans.
As testers, we know that we will find bugs and risks in all phases of the test process. We also know that functional requirements needs reviews and work for them to have enough detail to use as test basis and in securing required test coverage. The most important input must come from end users in how they interact with the system, but we must also understand how master data and transactional data flows through the connected components in “end to end” business processes.
[Tweet “We must also understand how master data and transactional data flows through the connected components in “end to end” business processes.”]
Managing and sharing knowledge about integrations
Many system integration projects struggle with a gap of required functional knowledge between the technical and functional people, within the project organization. Especially if the system environment is a hybrid, composed of modern technology and integrated with several legacy systems. There will also be knowledge gaps inside these two main domains.
Most projects may also include integrations from a number of third party service providers with a possible mix of technology. In addition to “internal” integrations. This means many integrations, where things could go wrong. How can our test strategy best deal with this landscape?
The risk is that the project will not be able to manage and control testing of all the integrations the system needs in order to test and use all of the ”end to end” business processes.
The project could be dependent on contribution from a number of product vendors and their representatives. Not all of them will be able to implement the systems business rules. The risk is that needed data quality and information breath exchanged between components will fail.
The project communication strategy should describe how and to whom we must share key functional knowledge about business rules, constrains, master data attributes and how the users expect the system to work.
This is especially useful information to the technical team in preparing the test environment for test of the individual integrations, in addition to system integration test.
The development team should also verify that known constraints, business rules and master data attributes implemented in schema validation and database constraints.
Test environment Configuration – testable Environments
As with all testing, one of the most underestimated testing tasks is to get the test environment ready for test. The System Integration level is the most interesting test levels for gaining knowledge about the system. Mainly because when the project reach this test level the test environment set-up must be correct in order for the test execution to start. In addition, this is a good opportunity for the whole project team to practice environment setup. The ultimate set up of the production environment or if the system is to integrate to an existing system will get input from the experiences made from this collaboration process.
Introducing a new “test level” – Early End-to-end testing
The integrations are often be tested separately, but managed in a dedicated test level with own test plan. When our basic integration tests passes, we are ready to test if the integrations work together by exercising business processes.
When testing end-to-end processes we will discover what parts of the system that are not ready for test. This is very important in many aspects.
- Which modules are actually ready for test?
- Which modules are not testable?
- What is the quality of the test basis?
- Is our test plan on track?
To be able to perform early end-to-end testing we have to implement this as a separate test level in our project. All the test phases must be in use, but our emphasis should not be test coverage. The scope of this test level is to identify, risk assess and test the business processes. If the risk is high, we will test some alternative and fault paths, else only the main paths. Our Evaluation of exit criteria from this test level is to the test planning phases in the System Integration and Acceptance test levels. We may also discover input to better test coverage for some of the integrations, which may need more testing.
Integration test level
At this test level, the typical contribution of new knowledge is to the technical domain. Testing input data validation by using EP and BVA to pass data from one component to the next. The expected result verified along with verifying that the output actually “fit” in the receiving component. Using other test techniques can also test validation of valid and invalid combinations of input data.
These tests could potentially lead to a recommendation for Schemas improvement. This because of new information discovered about handling of input and output data. The schemas contain rules and logic about what is to pass through an integration. Schemas are a powerful means for communicating requirements to avoid misunderstandings. It is a good idea to use the schemas as a main source of rules for this particular integration. By discovering defects or weakness in the rules by thorough testing, we can improve the quality of the system by preventing incorrect or missing data sent to the next component. The rules and business logic found in the schemas are a great way to share requirements throughout the project. The information can convert to a well-understood format.
System Integration test level
Functional compatibility between system components belongs in the System integration test level. If we have made a good effort in testing and building stable and robust integrations, we will have no trouble, using the same input data sets from the integration level. The most important entry criteria for this test level is to be able to run the planned tests without having to worry about bugs we discovered in the integration level.
We should conduct Early End-to-end testing to gain knowledge about the system, then refine the requirements and share key information with the people who can contribute the most preparing for the next test levels.
About the Author
Øyvind Woll is a Senior Consultant at Vivento AS based in Norway.
