Given the rapid migration to cloud-based architectures, the underlying IT that supports new ways of doing business – including quality engineering – needs to transform. Cloud-based platforms and solutions provide an array of benefits, from accelerated business growth and increased agility to cost reductions and run-time resource provisioning. Cloud can also deliver a superior and seamless user experience across channels.

Yet the quality of that user journey is vital to improving customer acquisition and retention. It is therefore critical for businesses to test appropriately. Quality testing must take place across the user journey, not just for new system functionality. And with the increasing complexity of cloud-based user journeys, non-functional testing (all the ways a system operates) and performance testing (how well it works) need a renewed focus.

Best Practices for Non-functional and Performance Testing on the Cloud

Performance testing on the cloud is distinctive and differs significantly from traditional load testing in its objectives and non-functional dimensions.. While conventional simulated load tests and system monitoring must continue for legacy software (enterprise systems), the focus for cloud should be on resilience and reliability of the highly distributed (event-driven, API, microservices) platforms. Chaos engineering and gamification tests can shed light on the weakness in design-code-deployment, but testing for resilience requires checking the right levels of performance, throughput under varying workloads, and the data coming from multiple channels. Accounting for peak time loads with variable host infrastructure elements has also become mandatory to assure reliability. 

When planning performance testing for cloud environments, quality engineering teams should consider the following four best practices.

1. Define desired non-functional attributes for the user journeys in a steady state. It’s also important to be aware of the difference between a steady and unsteady state. Engineering teams are normally better equipped or at least expecting failures in an unsteady state; steady-state is where a different approach is needed.

2. Introduce reliability as a core principle for DevOps. Ensure that changes in the system are gradual. The impact should be detectable and measurable. Measure all attributes, be it related to system or engineering, to determine risks and toil and ultimately service level objectives and error budgets.

3. Testing in production is critical. 91¿ì»îÁÖ’s Quality Engineering in a Cloud-Centric World report found that 23% of companies found more value in shift-right activities (when compared to the shift-left approach) in cloud ecosystems for non-functional and performance attributes. Utilize chaos engineering principles to experiment in production by introducing variables that can impact the cloud, the network or the application, and observe the overall system behavior. Integrate non-functional tests and chaos tests in the continuous delivery pipeline immediately after a change is introduced in the system.

4. Deploy the right delivery model based on skill sets. Performance skill sets are still very niche, and 46% of organizations still offer it in a shared-service model. It may stay this way until more organizations advance their cloud maturity.

The new cloud paradigm is evolving rapidly, and so is the need for quality engineering. No two organizations are the same. What works for one will not necessarily work for the other. However, adopting these best practices requires a certain discipline in terms of the delivery model, skill sets availability and shift left/right approach