BENCHMARK TESTING LOAD TESTING PERFORMANCE TESTING STRESS TESTING VOLUME TESTING UNIT & ALGORITHM EFFICIENCY (1) A standard against which measurements or comparisons can be made. (2) A test that is used to compare components or systems with each other or to a standard as in (1). E.g. very roughly „The new release must be at least as efficient as the previous release.“ Typical Metric: Comparison to the competitor, comparison to the predecessor. Testing where the system is subjected to large volumes of data. Typical Metric: amount of registered users, database migration/upgrade/size, response time in dependency to data amount. The process of testing to determine the resource-utilization of a software product. Typical Metric: cycles of search algorithm, re-entry of data regions, memory allocation, etc. The process of testing to determine the performance of a software product as the round trip time, system response time, etc. E.g. within unit testing this can be the run time of an algorithm. Typical Metric: delay, run time measurements, response time, network latency. A type of effiency testing conducted to evaluate the behavior of a component or system with increasing load, e.g. numbers of parallel users and/or numbers of transactions, to determine what load can be handled by the component or system. This kind of testing is used to measure the system capability and identify the system bottlenecks based on load specific metrics. Typical Metric: transactions per second, usage attempts per hour, failure rate. A type of efficiency testing conducted to evaluate a system or component at or beyond the limits of its anticipated or specified workloads, or with reduced availability of resources such as access to memory or servers. Typical Metric: administration and maintenance attempts must be possible under over load situations. RESOURCES TIME [email protected] www.bbv.ch FINAL REPORTING INITIALIZE Requirements User Stories Business Cases Non-functional Requirements • Expected Load • Required Performance • Reaction Time • Overload Control • User Expectations IDENTIFY GOALS Non-functional Requirements (NFR) Risks Benchmarking PLAN TESTS "Efficiency Tests" Testing Stages IMPLEMENT Load Model Scaling Test Data Tools & Scripts Test Harness Efficiency Acceptance Stress Load Performance Volume Re-Entry Algorithm Efficiency PROD SYSTEM End-to-End SYSTEM Integration INTEGRATION UNIT Test Planning • Cost-Benefit Analysis • Test Scope • Timeline • Amount • Resources • Efficiency Profile (e.g. Load Model) • Definition of Acceptance Criteria EXECUTE & EVALUATE FINALIZE Feedback • Quick Feedback to Stakeholders • Detailed Evaluation • Process Improvement • Product Improvement Outcome • Final Reporting • Backup • Retrospective Time Transactions per Sec. Time Transactions per Sec. PRODUCT IMPROVEMENT PROCESS IMPROVEMENT Network Engineer Application Developer Application Developer PERFORMANCE TUNING AREAS B1 B2 B3 B4 B5 B6 B7 B8 B9 Network Component tuning, Increase Internet bandwidth Configuration (methods, e.g. Round Robin) Caching configuration / Kernel tuning OS-Update / Firmware update Configuration / Communication protocol / Fixing memory leaks / Content optimization / Concurrency / Dedicated system Indexing / Clustering / Hashing Reduce monitoring level / Change monitoring software / Drop monitoring Interface tuning, Subsystem tuning Increase I/O, Optimize file cache Load Balancer Reverse Proxy Operation System & HW Resources Application: Config/Programming Database Monitoring Interface Storage Network Engineer Possible Bottlenecks Measures Role in Technical Team Network Engineer System Engineer System Engineer Application Developer DB Administrator All Application Developer System Engineer Stakeholder Stakeholders require concise information that clearly highlights key points. Intuitive visual representations, consolidated data and summarized information are given preference, as the information is usually required less frequently than by other team members. Project Team Project team members – project manager, development lead, and the test manager – have the same needs as stakeholders but require data more frequently and in greater detail. Technical Team Technical team members are mainly interested in a continuous flow of information related to monitored data, observations and test results. CHECKLIST NON-FUNCTIONAL REQUIREMENTS • Expected user characteristics, determine expected load Number of users Number of (business) transactions per time Sizing and memory requirements • Definition of load scenarios / perturbance scenarios / load peaks • Performance Response times of functions: max. and average Response times on load: max. and average Execution times of business cases: max. and average Execution times of business cases on load: max. and average • System availability Robustness Behavior on overload / overload protection Restart time on load • Consideration of legal and contractual restraints • Considerate performance expectations and performance needs of users Legal Disclaimer: While we have made every attempt to ensure that the information in this publication has been obtained from reliable sources, bbv Software Services (bbv) is not responsible for any errors or omissions, or for the results obtained from the use of this information. All information is provided with no guarantee of completeness or accuracy, and without warranty of any kind. In no event will bbv or its employees therefore be liable to you or anyone else for any decision made or action taken in reliance on the information in this publication. The information in this publication should not be used as a substitute for consultation with professional bbv advisors. Before making any decision or taking any action, you should consult a bbv professional. The names of actual companies and products (e.g. ScrumAlliance, bbv Software Services) mentioned in this publication may be the trademarks of their respective owners. ©2014 bbv Software Services www.zenwis.ch PERFORMANCE TESTING PERFORMANCE TESTING BUSINESS VIEW OPERATIONAL VIEW BUSINESS VIEW OPERATIONAL VIEW PROCESS & PLANNING VIEW