High Volume Performance Testing driven by Virtualization of a data centre

OBJECTIVE

Data centres have been identified as one of the key focus areas of IT organization’s Green efficiency campaigns with the other two being networks and buildings (data centres consume 2.5% of the world’s total grid power). A premier Tier 1 Telco had a vision “To be number one for energy, carbon and environmental management globally” as part of their social awareness and responsibility campaign and data-centres were top priority. This was the subject of much debate when they had to validate and benchmark the performance of a major data migration solution. The data migration exercise was estimated to have an impact on 6 million+ LIVE broadband customers and any change to production without sufficient performance testing was deemed a major risk to business. This meant setting up of a Production like scalable test bed, which translated to an additional 722 servers. An implementation of this scale for a one time performance test activity went against the organizations Green vision.

The total cost of implementing a new setup was estimated to be in the range of approximately £3 Million with additional Opex costs for first time implementation and commissioning. The costs apart, the time for implementing this was an estimated 6 months from the time of procuring the hardware.

PROPOSED SOLUTION

The solution to this requirement was to arrive at a setup without adding any new infrastructure to the existing data centres. This meant there was a need to conceptualize and implement a Virtualized Performance Test Bed that would replicate production in terms of configuration and capacity but at significantly lower costs. The virtualization of servers would effectively reduce potential energy consumption of approx. 364 KW which equates to 0.23 KT of CO2 emissions. This was to be achieved by stitching together diverse components from existing environments to create a virtually connected E2E (End to End) test bed – supplemented further by the use of simulation and data anonymization techniques. The Virtualized Performance Test Bed would also need to provide the scale and the flexibility to toggle between other forms of testing (system test and system integration test), along with the ability to plug in-out any component/s.

 

IMPLEMENTATION

A fundamental challenge with introducing an E2E performance test bed within an already crowded data centre was with provisioning of additional resources and space. Hence the guiding principle behind this setup was avoid procuring new infrastructure, reuse existing servers and virtualize the remaining components, effectively reducing the energy consumption of the whole setup and reducing the carbon footprint.

The following sequence of activities formed the crux of the solution that helped realize a reliable and cost effective performance test bed in very challenging timescales:

• A systematic assessment of existing hardware and software specifications of critical applications involved in the migration process– in order to arrive at a blue-print of the desired setup.
• Consolidation of specs for legacy systems and modelling of capacity to establish accurate scaling factors. This would help to account for the difference in size and scale of test and Production servers.
• Integration of interfacing components to achieve next box performance testing.
• Integration of multiple remotely located components to achieve E2E flows and journeys.
• Creation of a Virtualized Stub Framework (VSF) to emulate components those not available in the virtual test bed (Explained in detail below).
• Calibration of application configurations to help realize realistic test output and make the tests more meaningful to operational stakeholders.
• Data cloning to replicate production data volumes without violation of data integrity and security rules which were key to maintaining customer confidentiality.
• Automated scripts to regenerate data using industry standard practices for data masking and anonymization.
• Instrumentation of a mechanism for performance monitoring of all affected services and components so as to cover the entire landscape.

The key to all of the above was right sizing of the various components to ensure that results of these tests could be further extrapolated to provide a representative E2E view of performance in Production. Given below is a high level view of the test bed in its current form.

 Virtualized Stub Framework

 The Virtualized Stub Framework (VSF) was conceptualized and implemented in two variants to support both Windows and Unix operating systems. It could be deployed and configured on a plug and play basis with very little development effort. This ensured the VSF was flexible to adapt to the code changes in an agile delivery model where a new release was deployed in production every 6-9 weeks. The overall implementation of the Framework comprised of the following elements:

• A lightweight core engine that supports one or more communication types of varying degrees and types. This was to facilitate simulation of the multiple interfaces types that FOS was communicating with as part of completing an order journey.
• A configuration matrix with GUI – that defines the number of threads for each communication type, the ramp-up /ramp-down profiles and the logging mechanism for each engine. An easy to use GUI allowed new users of the framework to manipulate and configure new values on the matrix as per the requirement of each test run.
• A backend database – that stores the response message relevant to each business transaction supported by the Virtualized Stub Framework. The Framework picks up messages based on the unique identifiers in each request and responds with the corresponding response message.
• Interface Manager – that would allow communication across multiple channels regardless of type.

As there was a requirement to support synchronous and asynchronous communication, the concept of state transitions was used in the VSF logic – i.e. the last known state of the order journey and the sequence of transitions anticipated to complete a specific journey was recorded and used as input to the core engine of the VSF.

A multi-threading feature was provided via the configuration matrix which facilitated the creation of concurrent connections to support the high volume of traffic between the FOS (system under test) and the Virtualized Stub Framework.

 

BENEFITS

At an organization level, a 6-8% reduction in energy consumption and a 42 per cent reduction (323kt) in absolute CO2e emissions was achieved compared to the previous year. This was possible only through a concerted effort by numerous projects such as this, where projects minimized spend on new infrastructure and encouraged reuse and recycle. This was a standout entry from the QA department. The graph shows an year on year reduction in energy consumption over a period of 5 years (blue indicating electricity, red for gas) across the Telco’s IT space. Year on year OPEX savings – a new test bed would have required new personnel for ongoing deployment and maintenance. A flexible and modular test bed that paved the way for establishing a reliable performance test process where none existed. This was key to providing an improved quality of test output which in turn ensured conformance to the desired level of performance to business and end users. Significant cost savings achieved as a result of the following:

The solution provided for re-use of existing hardware and software elements leading to cost savings close to £3.0 Million Capex (which was the cost for a new environment to be setup). Systematic assessment of existing capacity and infrastructure conducted during this setup; there were further savings of around £300K. This was primarily due to multiple instances of hardware, software and storage consolidation and recovery – which had a direct impact on reducing the carbon footprint. The Virtualized Stub Framework solution also provided a costs effective alternative to procuring expensive third party licensed software for virtualization. These software have been proven for low volume functional testing, but unlike our solution, their use for performance testing remains largely untested (this licenses cost approx. £60K to £200K annually). Apart from its primary role as that of simulating interfacing components, the Virtualized Stub Framework also performed the role of a load injector and consumer, a function that is typically performed by expensive third party load testing tools and software (the cost of these tools range from £100k – £500K annually). The modular nature of the VSF meant there was minimal development effort required to ensure fitment to subsequent releases and windows for performance testing did not stretch beyond agreed timelines due to high scripting or development efforts. It also offered the flexibility to further enhance and configure new technology/domain areas with minimal amount of rework. The lightweight build of the VSF resulted in zero investment on hardware, as it could be hosted on basic infrastructure within the same network or data centre, at the same time providing seamless integration into diverse technology areas. The architecture and design of the VSF also ensured there is up to 40% re-usability while implementing other systems as well as other forms of testing (functional and regression testing). Much beyond the benefits of realizing a cost effective test environment, the subsequent testing on this setup resulted in a superior quality of code being, delivered to production, best reflected by the below : An improvement of 10% to 63% in response times of key end user transactions after performance testing. A 60% improvement in overall migration window as a result of optimizing individual transactions. The throughput of volumes processed by the migration solution improved by 20%. This environment was key to the delivery of a performance compliant migration solution which could process up to 7.5 million customers (which was higher than original planned volume of 6 million customers).

 

CONCLUSION

 While Green environments and Virtualization are much sought after topics by enterprises, the practical implementation has proved to be a major impediment. The solution articulated here is just one example of the capabilities that can be brought into a testing environment in process reducing the overall carbon footprint.

About the Author:

Mallikarjun Patil

Mallikarjun Patil is a Test Consultant working with TechMahindra based in the U.K. Mallikarjun area’s of interest include performance and non-functional testing.