Overview
Nowadays, traditional performance testing is obsolete. Essentially, the traditional way of conducting performance testing is outdated. This full-blown traditional performance testing normally conducted at the end of software development life cycle at pre-production stage. Now, it is replaced by DevOps performance testing. DevOps performance testing is modularized, component driven continuous performance testing with proactive performance monitoring in production environment.
Introduction
I am in the performance testing field and have been conducting traditional performance testing over the last decade. For last few years, it seems that traditional performance testing is obsolete. It is now replaced by DevOps performance testing, looking at the benefits of DevOps methodologies and industry wide it’s practices.
Traditional Performance Testing to DevOps Performance Testing – Industry Shift
Traditional performance testing has its own advantages & disadvantages. It is suitable for traditional performance testing methodologies. Once industry started using agile methodologies, traditional performance testing is upgraded to agile performance testing. It moves towards continuous performance testing (mostly shift-left performance testing). Then comes, DevOps methodologies. It is based on top of agile methodologies. It always points towards automation. In DevOps performance testing, we do comprehensive continuous performance testing. Comprehensive continuous performance testing combines both shift-left & shift-right performance testing concepts.
As industry started using DevOps performance testing, slowly traditional performance testing becomes obsolete. When I say traditional performance testing is obsolete, I mean the traditional method is outdated. Now, performance testing is not conducted at the end of software development life cycle at preproduction stage. Even, traditional full-blown performance testing is also not executed.
DevOps Performance Testing – Today’s solution
DevOps performance testing is modularized, component driven continuous performance testing with proactive performance monitoring in production environment.
In DevOps, the development team is responsible for new and updated features. Whereas the operation team is responsible for existing application. From a performance testing perspective, the testing team is responsible for new enhancement performance testing. They are responsible for the overall system performance (in terms of speed, scalability, stability, spike, size) or regression performance or proactively monitoring end to end system. Overall, the testing team is accountable for both performance testing & performance monitoring.
Production Performance Monitoring – crucial in DevOps
Currently, production performance monitoring is critical. Shift-right performance testing perspective, we need to conduct controlled performance testing and continuous performance monitoring in production. Production performance monitoring covers both real user monitoring (RUM) & synthetic monitoring (SM).
RUM is a passive monitoring technology that records all end-user interactions. RUM is important to know whether end-users are being functioned speedily and without any issues. If any performance issues are raised, then RUM proactively identify the issue and resolves immediately before actual end-users get affected.
Whereas, SM is an active monitoring technology, scripted and executed in a more controlled manner in production. SM helps to identify potential application performance problems, slowness, any downtime experiences etc. well in advance. SM plays an important role for first time releases, major upgrades or migrations.
Together, RUM & SM can provide deeper application visibility into end to end application performance, regardless of where and how they are running.
Traditional Performance Testing – Slow
Traditionally the performance testing team would take a large amount of time for conducting full blown performance testing. Also, in case of failures, it is not easy to identify the potential root cause of the failures. This is because the replication of test execution with similar performance data takes time. And most importantly, even if you find performance issues, it takes time to resolve these. Development teams need to identify the performance issue and then resolve followed by testing, retesting and even regression testing. These additional steps can make the overall project schedule slippage. It can also frustrate all the stakeholders. Overall, traditional performance testing is slow.
Traditional Performance Testing – Time Consuming
Performance testing can be time consuming due to its category of testing. So, including performance testing in sprint releases for agile methodologies in its initial days were little tough. When release cycles are tight & frequent, stakeholders try to exclude performance testing in SDLC process.
Performance Testing is now part of Automated Deployment Process
Whereas, DevOps performance testing is less time consuming. Due to this, DevOps automated continuous build and deployment concepts include performance testing. Performance testing is now an integrated part of continuous integration and the continuous development (CI/CD) process. Conducting small low volume continuous performance testing fits perfectly into the CI/CD pipeline concept. For, large performance testing, we need to break them into multiple modules or components. Overall, objective is to conduct component level performance testing. And execute many component performance tests in parallel. This is done at API layer so that different services can be performance tested at same time. But all these needs to be executed independently to each other to save overall execution time.
Traditional Performance scripting – Obsolete
The traditional method of script development is now old-fashioned. Creating script is much easier & less time-consuming in DevOps performance testing. Good thing is that these fast, small scoped, low volumes performance testing can be executed in parallel to confirm overall system performance before actual production release. DevOps performance testing is executed automatically during the pipeline process to save time, parallelization saves even more time.
Traditional Performance Test Results Analysis – Outdated
Even performance test result analysis are now automated. If any deployment is unable to meet the expected performance, deployment will automatically fail. Performance test results must be within the specified performance threshold or within the business specified SLA- service level agreement. These automated pipeline jobs execute performance test at component level/API level as and when required as part of build pipeline. These jobs also share the test results automatically as dashboard. Developers quickly understand the application performance looking at those performance test results or dashboards. Stakeholders can easily take the decision without waiting for full-scale performance testing results.
Traditional Performance Testing is now replaced by DevOps Performance Testing
Organisations are now concerned about increasing rate of change in the market. They not only need to compete in the market but also familiarise with speed of changes to adapt time-to-market pressures. Organizations also need to ensure improved performance every time to satisfy end-user. Consequences are that traditional full-blown performance testing is obsolete and is replaced by DevOps performance testing. DevOps performance testing is essentially continuous component performance testing in CI/CD pipeline.
In DevOps performance testing, proactively monitoring the servers is important. This is done via application performance monitoring (APM) tools. These tools assist to pinpoint the potential performance problems proactively. These tools have minimal overhead and without any additional code. They also assist in server log analysis. Log analysis assist to prepare for next releases in comprehensive continuous performance testing.
Together, CI/CD pipeline component performance testing and continuous application performance monitoring ensures full benefits of comprehensive continuous performance testing (i.e. DevOps performance testing)
Conclusion
The traditional method of conducting performance testing is now outdated. Organisations are more interested in DevOps performance testing. DevOps performance testing ensures full benefits of comprehensive continuous performance testing by conducting:
- Continuous component level, low volume performance testing in parallel (Dev)
- Proactive application performance monitoring – both RUM & SM (Ops).
Check out DevOps webinars and eBooks on here on Huddle
