Traditional Performance Testing is now Obsolete – A Thought

Reading Time: 4 minutes

Traditional Performance Testing is now Obsolete – A Thought

 Overview:

Now a day, traditional performance testing is obsolete. Basically, traditional way of conducting performance testing is outdated. This full-blown traditional performance testing normally conducted at the end of software development life cycle at preproduction stage. Now, it is replaced by devops performance testing. Devops performance testing is modularized, component driven continuous performance testing with proactive performance monitoring in production environment.

 

 

Introduction:

I am in performance testing field and conducting traditional performance testing over the last decade. For last few years, it seems that traditional performance testing is obsolete. It is now replaced by devops performance testing, looking at the benefits of devops methodologies and industry wide it’s practices.

Traditional Performance Testing to Devops Performance Testing- Industry Shift:

Traditional performance testing has its own advantages & disadvantages. It is suitable for traditional performance testing methodologies. Once industry started using agile methodologies, traditional performance testing is upgraded to agile performance testing. It moves towards continuous performance testing (mostly shift-left performance testing). Then comes, devops methodologies. It is based on top of agile methodologies. It always points towards automation. In devops performance testing, we do comprehensive continuous performance testing. Comprehensive continuous performance testing combines both shift-left & shift-right performance testing concepts.

As industry started using devops performance testing, slowly traditional performance testing becomes obsolete. When I say traditional performance testing is obsolete, I mean traditional way of conducting

performance testing is outdated. Now, performance testing is not conducted at the end of software development life cycle at preproduction stage.  Even, traditional full-blown performance testing is also not executed.

 Devops Performance Testing- Today’s solution:

Devops performance testing is modularized, component driven continuous performance testing with proactive performance monitoring in production environment.

In devops, development team is responsible for new and updated features. Whereas operation team is responsible for existing application. From performance testing perspective, performance testing team is responsible for new enhancement performance testing. They are responsible for overall system performance (in terms of speed, scalability, stability, spike, size) or regression performance or proactively monitoring end to end system. Overall, performance testing team is accountable for both performance testing & performance monitoring.

Production Performance Monitoring – crucial in Devops:

Now, production performance monitoring is highly critical. Shift-right performance testing perspective, we need to conduct controlled performance testing and continuous performance monitoring in production. Production performance monitoring covers both real user monitoring (RUM) & synthetic monitoring (SM).

RUM is a passive monitoring technology that records all end-user interactions. RUM is important to know whether end-users are being functioned speedily and without any issues. If any performance issue raised, then RUM proactively identify the issue and resolves immediately before actual end-users get affected.

Whereas, SM is an active monitoring technology, scripted and executed in more controlled manner in production. SM helps to identify potential application performance problems, slowness, any downtime experiences etc. well in advance. SM plays an important role for first time releases, major upgradations or migrations.

Together, RUM & SM can provide deeper application visibility into end to end application performance, regardless of where and how they are running.

Traditional Performance Testing- Slow:

Traditional performance testing team normally takes a big amount of time for conducting full blown

performance testing. Also, in case of any failure it is not easy to identify potential root cause of the failures. This is due to replication of test execution with similar performance data takes time. And most importantly, even if you find a performance issue, it takes time to resolve the performance issues. Development team needs to identify the performance issue and then resolve followed by testing, retesting and even regression testing. These additional works can make overall project schedule slippage. This can even frustrate all the stakeholders. Overall, traditional performance testing is slow.

Traditional Performance Testing – Time Consuming:

Traditionally performance testing is always time consuming due to its category of testing. So, including performance testing in sprint releases for agile methodologies in its initial days were little tough. when release cycles are tight & frequent, stakeholders try to exclude performance testing in SDLC process.

Performance Testing is now part of Automated Deployment Process:

Whereas, devops performance testing is less time consuming. Due to this, devops automated continuous build and deployment concepts includes performance testing.  Performance testing is now an integrated part of continuous integration and continuous development (CI/CD) process. Conducting small low volume continuous performance testing perfectly fits into CI/CD pipeline concepts. For, large performance testing, we need to break them into multiple modules or components. Overall, objective is to conduct component level performance testing. And execute many component performance tests in parallel. This is done at API layer so that different services can be performance tested at same time. But all these needs to be executed independently to each other to save overall execution time.

Traditional way of Performance scripting – Obsolete:

Traditional way of script development is now old-fashioned. Creating script is much easier & less time-consuming in devops performance testing. Good thing is that these fast, small scoped, low volumes performance testing can be executed in parallel to confirm overall system performance before actual production release. Devops performance testing is executed automatically in pipeline process to save time, parallelization makes it save more time.

Traditional way of Performance Test Results Analysis – Outdated:

Even performance test result analysis is now automated. If any deployment is unable to meet the expected performance, deployment will automatically fail. Performance test results must be within specified performance threshold or within business specified SLA- service level agreement. These automated pipeline jobs execute performance test at component level/API level as and when required as part of build pipeline. These jobs also share the test results automatically as dashboard. Developers quickly understand the application performance looking at those performance test results or dashboards. Stakeholders can easily take the decision without waiting for full-scale performance testing results.

 Traditional Performance Testing is now replaced by Devops Performance Testing-

Organizations are now concerned about increasing rate of changes in market. They not only need to compete in the market but also familiarize with speed of changes to adapt time-to-market pressures. Organizations also need to ensure improved performance every time to satisfy end-user. Consequences of that traditional full-blown performance testing is obsolete.  And is replaced by devops performance testing. Devops performance testing is basically continuous component performance testing in CI/CD pipeline.

In devops performance testing, proactively monitoring the servers are important. These are done via application performance monitoring (APM) tools. These tools assist to pinpoint the potential performance problems proactively. These tools are having minimal overhead and without any additional code. They also assist in server log analysis. Log analysis assist to prepare for next releases in comprehensive continuous performance testing.

Together, CI/CD pipeline component performance testing and continuous application performance monitoring ensures full benefits of comprehensive continuous performance testing (i.e. devops performance testing)

Conclusion:

Traditional way of conducting performance testing is now outdated. Organizations are more interested in devops performance testing. Devops performance testing ensures full benefits of comprehensive continuous performance testing by conducting:

  • Continuous component level, low volume performance testing in parallel (Dev)
  • Proactive application performance monitoring – both RUM & SM (Ops).
About the Author

Arun Kumar

I earned a degree in Computer science from Govt. Engineering College, India (college topper). I have 13.5 years experience of managing E2E testing delivery in different types of applications. I have a keen interest in reading and writing different technical papers. I was selected to speak at multiple international conferences; global webinars and my papers have been published in multiple forums. Currently, I am working as a Senior Test Manager in Atos-NAO  & Global subdomain leader for Atos Expert: Application-Testing.
Find out more about @arun2005413gmail-com

One Response to “Traditional Performance Testing is now Obsolete – A Thought”

  1. I’d rather see it as we have more options for performance testing now and it is getting more integrated with overall performance engineering efforts. There are still risks that could be mitigated only by full-blown performance testing (when applicable) and some tests probably should be kept outside CI (for example, tests for memory leaks or sizing / capacity planning tests – of course, depends on the context).

Leave a Reply