CONNECT WITH US:
Testing as a Managed Service Using SLA’s and KPI’s
Testing Conference, Chicago
September 2014
CONNECT WITH US: 2
What is it?
Why should I care? Knowing the skillset increases
your worth to the business Not all Outsource Groups are
created equal The business will still look to
you to ensure over all quality
Your tools for success? Proper Governance
Specific KPIs Custom SLAs
How do you successfully manage outsourced testing?
Outsource testing is when a company decides to do all or part of there testing from a 3rd party specialist either onsite, near shore or offshore.
Manage Services is when the 3rd party handles everything from Strategy through Sunset
CONNECT WITH US:
Accelerate Problem
Identification and
contribute to the
development process
Make Process Improvements and ensure accountability on both sides
Governance both for
internal QA team and/or
Outsourced Vendors
Improve Quality of
Software as well as Service Delivery
Adhere to
Internal
and
External
Complianc
e
Standards
Measure to avoid the common mistakes
7
The Aberdeen Group in 2010 estimated that nearly 50% of all outsource projects fail outright or fail to meet expectations
CONNECT WITH US: 4
KPIs
Guidelines
Key Performance Indicators vs Metrics
KPIs highlight the things that are important to your business and should tie directly to business goals Financial Goals - QA costs as a % of total IT
costs and as a % of revenue Staffing Goals – QA FTE to total FTE Ratio Operational Goals – Business Process Quality
and Continuity as a % of Up Time
Guidelines for creating KPIS The success measure must show clear, specific
and measureable performance indicators Costs to monitor can not exceed the value that
will known from measuring Linked to a strategic objective and easily
understood within its context Credible and under the control of the team
attributed to
CONNECT WITH US: 5
SLA’s are vital signs. You can tell where a testing organization is headed and also measure it’s maturity level.
Service Level Agreements
CONNECT WITH US: 6
Understand your KPIs and determine what you need specific to your environment
Remember to include items you will typically delegate to internal staff to self manage
Service Level Agreements (SLA’s)
Before you begin…..
CONNECT WITH US: 7
What are some of the Key go to items to have in your SLAs?
Product quality measurements DRE (Defect Removal
Efficiency) Traceability, test case
efficiency and code coverage Defect identification quality
(valid bugs identified % of the time)
Test plan quality and strategy will require no more than X reviews
Reporting and project management timelines
Defects will be reported within a specified time window by severity
Defects assigned for re-test are regressed within x hours
Testers will design X number of test cases per day per tester based on complexity
Manual testers will execute X number of test cases per day
Automation testers will automate X number of test cases a day
Knowledge Transfer and Core business understanding
Vendor staff are required to maintain X experts and Y intermediates in our core business process
CONNECT WITH US: 8
DRE Process Compliance Business Process Knowledge
General Incident Mgmt. Automated Regression Library - Execution
Regression Library - Updates
100%
98%
100%
91%
100%
100%
Snapshot of a SLA Metrics Dashboard
CONNECT WITH US: 9
Metrics
Cost On-Cost Delivery (Accuracy
of cost estimation) Estimated Effort Index
(Actual vs. planned hours) Time
On-Time Delivery (Accuracy of schedule estimation)
Schedule Variance Productivity
Percentage Rework Quality
Weighted Average Defect Density
Defect detection rate (based on function points or person hours)
Defect Management Defects detected by phase
and by type Root cause distribution
SLAs
Delivery within budget Defect Removal Effectiveness Estimation and Sizing
Turnaround Test Design Efficiency
Process Compliance Score Staff’s knowledge of business
process Accuracy of Metrics General Incident Management
Automated Regression Test Library
Regression Library Maintenance
KPIs
Cost of testing as a % of overall project cost
Test Case Effectiveness
Test Progress – Plan vs. Actual
Skill Set Matrix
Quality of Development
Quantifiable Regression Execution
Number of regression test cases updated by release/ project
Illustrative list of typical performance measures
How KPIs, SLAs, and Metrics work together
Individual Project PerformanceMain Project DriversBusiness Goals
CONNECT WITH US: 10
Our Governance Process at McKesson
We have a three-tier model for leading the McKesson -Prolifics partnership and providing comprehensive program
oversight.
Set strategic direction Review long term goals Review areas of
improvement Optimize alignment
across partners
Steering Committee Meeting
Operational planning Account management SLA Reviews Escalation management New Initiatives
Leadership Team Meeting
Project progress review Resource management Prioritization Knowledge management Issue management
Test Managers Meeting
McKesson
Head of testing practice
Account Executive
VP McKesson IT Generics
Director SV&T Senior
Director SV& T Director
Frequency
Quarterly Meetings
Weekly Meetings
Monthly Meetings
Account Executive
Onsite Test Manager
Generics Director
SVCS Sr. Director
SV&T Senior Director
SV& T Director
Account Executive
Onsite Test Manager
Project /Program Manager
Prolifics
McKesson Prolifics
McKesson Prolifics
CONNECT WITH US: 11
Automated SLA Reporting to Track Events in Real Time
CONNECT WITH US: 12
Sample SLA Report
CONNECT WITH US:Public | Copyright © 2014 Prolifics 13
Q & A
Thank You!
CONNECT WITH US:Public | Copyright © 2014 Prolifics 14
Contact US
www.prolifics.com
+1 510 689 8712