Data centres have been identified as one of the key focus areas of IT organization’s Green efficiency campaigns with the other two being networks and buildings (data centres consume 2.5% of the world’s total grid power). A premier Tier 1 Telco had a vision “To be number one for energy, carbon and environmental management globally” as part of their social awareness and responsibility campaign and data-centres were top priority. This was the subject of much debate when they had to validate and benchmark the performance of a major data migration solution. The data migration exercise was estimated to have an impact on 6 million+ LIVE broadband customers and any change to production without sufficient performance testing was deemed a major risk to business. This meant setting up of a Production like scalable test bed, which translated to an additional 722 servers. An implementation of this scale for a one time performance test activity went against the organizations Green vision.
The total cost of implementing a new setup was estimated to be in the range of approximately £3 Million with additional Opex costs for first time implementation and commissioning. The costs apart, the time for implementing this was an estimated 6 months from the time of procuring the hardware.
The solution to this requirement was to arrive at a setup without adding any new infrastructure to the existing data centres. This meant there was a need to conceptualize and implement a Virtualized Performance Test Bed that would replicate production in terms of configuration and capacity but at significantly lower costs. The virtualization of servers would effectively reduce potential energy consumption of approx. 364 KW which equates to 0.23 KT of CO2 emissions. This was to be achieved by stitching together diverse components from existing environments to create a virtually connected E2E (End to End) test bed – supplemented further by the use of simulation and data anonymization techniques. The Virtualized Performance Test Bed would also need to provide the scale and the flexibility to toggle between other forms of testing (system test and system integration test), along with the ability to plug in-out any component/s.
A fundamental challenge with introducing an E2E performance test bed within an already crowded data centre was with provisioning of additional resources and space. Hence the guiding principle behind this setup was avoid procuring new infrastructure, reuse existing servers and virtualize the remaining components, effectively reducing the energy consumption of the whole setup and reducing the carbon footprint.
The following sequence of activities formed the crux of the solution that helped realize a reliable and cost effective performance test bed in very challenging timescales:
- A systematic assessment of existing hardware and software specifications of critical applications involved in the migration process– in order to arrive at a blue-print of the desired setup.
- Consolidation of specs for legacy systems and modelling of capacity to establish accurate scaling factors. This would help to account for the difference in size and scale of test and Production servers.
- Integration of interfacing components to achieve next box performance testing.
- Integration of multiple remotely located components to achieve E2E flows and journeys.
- Creation of a Virtualized Stub Framework (VSF) to emulate components those not available in the virtual test bed (Explained in detail below).
- Calibration of application configurations to help realize realistic test output and make the tests more meaningful to operational stakeholders.
- Data cloning to replicate production data volumes without violation of data integrity and security rules which were key to maintaining customer confidentiality.
- Automated scripts to regenerate data using industry standard practices for data masking and anonymization.
- Instrumentation of a mechanism for performance monitoring of all affected services and components so as to cover the entire landscape.
The key to all of the above was right sizing of the various components to ensure that results of these tests could be further extrapolated to provide a representative E2E view of performance in Production. Given below is a high level view of the test bed in its current form.
Virtualized Stub Framework
The Virtualized Stub Framework (VSF) was conceptualized and implemented in two variants to support both Windows and Unix operating systems. It could be deployed and configured on a plug and play basis with very little development effort. This ensured the VSF was flexible to adapt to the code changes in an agile delivery model where a new release was deployed in production every 6-9 weeks. The overall implementation of the Framework comprised of the following elements:
- A lightweight core engine that supports one or more communication types of varying degrees and types. This was to facilitate simulation of the multiple interfaces types that FOS was communicating with as part of completing an order journey.
- A configuration matrix with GUI – that defines the number of threads for each communication type, the ramp-up /ramp-down profiles and the logging mechanism for each engine. An easy to use GUI allowed new users of the framework to manipulate and configure new values on the matrix as per the requirement of each test run.
- A backend database – that stores the response message relevant to each business transaction supported by the Virtualized Stub Framework. The Framework picks up messages based on the unique identifiers in each request and responds with the corresponding response message.
- Interface Manager – that would allow communication across multiple channels regardless of type.
As there was a requirement to support synchronous and asynchronous communication, the concept of state transitions was used in the VSF logic – i.e. the last known state of the order journey and the sequence of transitions anticipated to complete a specific journey was recorded and used as input to the core engine of the VSF.
A multi-threading feature was provided via the configuration matrix which facilitated the creation of concurrent connections to support the high volume of traffic between the FOS (system under test) and the Virtualized Stub Framework.
This environment was key to the delivery of a performance compliant migration solution which could process up to 7.5 million customers (which was higher than original planned volume of 6 million customers).
While Green environments and Virtualization are much sought after topics by enterprises, the practical implementation has proved to be a major impediment. The solution articulated here is just one example of the capabilities that can be brought into a testing environment in process reducing the overall carbon footprint.
About the Author:
Mallikarjun Patil is a Test Consultant working with TechMahindra based in the U.K. Mallikarjun area’s of interest include performance and non-functional testing.