This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
A few points to add to Scott’s presentation…A few points to add to Scott’s presentation…
Scott said…– Virtualization does not change the core principles of
successful performance testing
Why?– Hardware is not virtual
– Whose environment is it, anyway?
– Rapid Bottleneck Identification Tuning a Virtualized Context
Hardware is not virtualHardware is not virtual
Virtualization is app/OS or hardware emulation specific
Underneath it all is a fixed pool of HW resources!
Performance Testing is designed to understand the impact on the hardware – as seen from the perspective of the user
Common Virtualization Types. Source: QAZone Whitepaper, Virtualization from the perspective of a QA/IT Operation Professional, http://qazone,empirix.com.
The Best Performance Data in a Virtual EnvironmentThe Best Performance Data in a Virtual Environment
Understand the impact on the hardware…– Requires accurate workload stimulus representing activity on and
through all VMs associated with that device
– Requires measurement of hardware characteristics at the base layer as well as OS and app-level characteristics in each VM
– Requires real-time collaboration during load testing to differentiate intra-VM bottlenecks (business logic) vs Hardware bottlenecks (CPU, Memory, NIC)
…as seen from the perspective of the user!– Your users don’t care about Virtualization
– Metrics that matter are end-user perceived performance
– Performance testing from inside the VM helps create stimulus but doesn’t report on end-user-perceived performance
Rapid Bottleneck Identification Load Testing in a Virtual EnvironmentRapid Bottleneck Identification Load Testing in a Virtual Environment
What is Load testing?
“Testing conducted to isolate and identify the system and application issues (bottlenecks) that will keep the application from scaling to meet its performance requirements”
What is a Bottleneck?What is a Bottleneck?
Any resource (hardware, software or bandwidth) that places defining limits on data flow or processing speed: your application is only as efficient as its least efficient element
On the Web, bottlenecks directly affect performance and scalability
Most untested systems have more than one bottleneck, but they can only be identified and resolved one at a time.
Where are the Bottlenecks?Where are the Bottlenecks?
The vast majority of bottlenecks are
caused by throughput
limitations – not concurrency.
Throughput80%
Concurrency20%
Source: Empirix
Sample throughput limitations in virtual environmentsSample throughput limitations in virtual environments
Disk I/O– Multiple applications accessing shared disk resources
– Virtual Memory Paging on shared disk resources
NIC– Multiple VMs shipping data through the same physical NIC
CPU– Processor-intensive (i.e., database) applications will compete for shared
CPU time.
Memory– Depends on type of VM configuration how much of an issue this can be
Whose Environment Is It Anyway?Whose Environment Is It Anyway?
Virtualization blurs the line between Dev/Test/Ops environments and ownership– Can exacerbate “silo” problems between Dev/Test/Ops
However, there is an opportunity…
…A common set of measurements can un-blur the line– Test data should inform production SLAs
– Common measurement/metrics can bridge the gap
– Measurement consistency is the cornerstone of ALM