V1.1 | 2017-01-30
Christof Ebert, Vector Consulting Services
Benchmarking – Industry Data and ImprovementTutorial, OOP, Munich, 30. Jan. 2017
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Vector Consulting Services
Welcome
…supports clients worldwide
…improves product development and product strategy, IT and provides interim management
…is as Vector Group globally present with 1700 employees and over 400 Mio. € sales
…offers with the Vector Group a portfolio of tools, software components and services
…is growing and thus continuously hiring
www.vector.com/consultingRailway
IT & Finance
Automotive
Aerospace
DigitalTransformation
Medical
2/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Christof EBERT
Welcome
[email protected] @christofebertwww.vector.com/consulting
Christof Ebert is managing director at Vector Consulting Services.
He supports clients to improve product strategy and product development and to manage organizational changes.
Prior to that, he held senior management positions for ten years at a global ICT leader, with worldwide responsibility for software technologies.
A trusted advisor for companies around the world, member of industry boards, and professor at the University of Stuttgart and Sorbonne in Paris, Dr. Ebert authored several books.
3/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Welcome
Performance Improvement
Measurement
Benchmarking
Data Analytics
Case Study: Lean Development
Case Study: Technical Debt
Summary
Agenda
Projects without clear goalswill not achieve their goals clearly.
Tom Gilb's principle of fuzzy targets
5/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Software and IT Generate Value – But How is it Measured?
Performance Improvement
A modern car has 50-100 electronic control units
Electrical equipment amounts to 30% of cost
Software delivers 90% of the innovation and value
Increasing complexity and callbacks, liability risks
But not enough measurement to learn and progress
Systematic measurement today is used in all engineering disciplines – except software and IT development
6/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Software-Driven Value – Innovation and Complexity
Performance Improvement
“At Apple, we are committed to world-class products. With the launch of our new maps we could not keep this commitment. We will do everything possible to make maps better.”
Tim Cook, CEO
…Complexity
The recall rate of cars is exploding. In 2014 5 times the number of sold cars are recalled in the same timeframe.
50% of the recalls are due to software problems.
Smartphones worldwide annual sales
• 2005: 50 Mio
• 2010: 300 Mio
• 2015: 1,4 Bio.
• 2020: 1,7 Bio.
Unlike the awkward-looking first trials, the incumbent automotive companies, are showing autonomous vehicles that just look like cars.
Innovation…
7/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Challenge “Complexity Explosion”
Performance Improvement
High complexity
Long life-time
Mostly maintenance
Source: Vector Benchmarking Suite, Ebert 2016
8/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Performance Improvement
Value: Goals
Waste: Fouls
Process: Passes
People: Teamwork
Continuous improvement: Training
Measurements and Benchmarking
Analytics: Goals, speed, corners, …
Improve:Concentration, blood, …
Performance Improvement
9/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Measuring the Business Impact: Value and Waste
Performance Improvement
Department Ping Pong
Added value30%
Overheads40%
Pure waste30%
Source: Vector Benchmarking Suite, Ebert 2016
10/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Two thirds of performance improvement projects fail
Productivity improvement means a substantial culture change
The change itself is often underestimated and handled ad-hoc
Reasons No management leadership Insufficient communication Lack of internal know-how
Performance Improvement Is Not Easy
Performance Improvement
Sources: Vector Consulting Services 2016
48%
Internalresistance
Insufficient know-how
Pressurewithoutownership
Other reasons
22%
17%
13%
Performance improvement needs top-down focus and change management.
11/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Industry Challenges 2017
Performance Improvement
Innovative Products
Others
Connectivity
Distributed Development
Efficiencyand Cost
Digital Transformation
Governance and Compliance
ComplexityManagement
Securityand Safety
0%
10%
20%
30%
40%
50%
60%
70%
0% 10% 20% 30% 40% 50% 60% 70%
Cost remains biggest short-term challenge across all industries. Connectivity and Digital Transformation evolved as a major challenge.
Mid
-ter
m c
hal
lenges
Short-term challenges
Vector Client Survey 2017. Details: www.vector.com/trends. Horizontal axis shows short-term challenges; vertical axis shows mid-term challenges. Sum > 100% due to 3 answers per question. Strong validity with >4% response rate of 1500 recipients from different industries worldwide.
12/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Welcome
Performance Improvement
Measurement
Benchmarking
Data Analytics
Case Study: Lean Development
Case Study: Technical Debt
Summary
Agenda
You can’t control what you cannot measure.
Tom DeMarco
14/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Measurement – Motivation
Measurement
What is not measurable, make measurable.
Galileo GalileiA science is as mature as its measurement tools.
Louis Pasteur
If you can‘t measure it, you can‘t manage it.
Peter Drucker
15/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Bloom (1960-1980) Source code metrics (first publication 1968) Volume, complexity (Halstead, McCabe), Reliability Goal: complexity reduction, release criteria
Boom (1980-2000) Software engineering as a comprehensive discipline Process frameworks like CMMI push for measurement Cost models (Boehm, Putnam) and estimation (FP, COSMIC) Goal orientation (top down definition of metrics) Experiments and validation of data
Doom (2000-2015) Erosion of disciplined software engineering and back to trial and error Agility and time to market more relevant than focused learning Measurement seen as nice to have; enterprise databases disappear
Today… Data analytics and big data suddenly are hype But where are the experts?
A Brief History of Software Measurement
Measurement
16/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Estimation and Performance Control
Measurement
Relationship of Function Points to duration and effort needs always to be tailored to environmental conditions.
Possible explanations for lower effort compared to predictive model:
Unrealistic estimation
Bad reporting
Better performance
Lower overhead cost
Experience database (same context)Predicted project parametersCurrently planned project parameters
Project effort
Person years
Project size (FPs)
?
17/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Assess supplier viability, contract reliability, cost and product quality – independent whether it is components or outsourcing
Standard estimation techniques and tools (e.g., COSMIC FP, QSM) for feasibility, health and cost
Risk assessments upfront and during the project
Predictions for progress related to your needs
External audits if you don’t trust process quality
Supplier Management and Project Control
Measurement
Earned Value
100%
Pro-gress
0%
Planned value
Actual cost
0% Calendar time today 100%
Current value: -15% Schedule variance
10% Cost varianceEarned value
Project start Calendar time
100%
Defects
0%
Expected defects Tolerance intervalQuality
Detected defects
Size (FPs)
Effort
50 100 5001
5
10
50 HistoryForecast
Current plan
Feasibility
18/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Example: Working with Measurements
Measurement
Total cum normalized defects
0
600
S 2 3 4 5 6 7 8 9S 2 4 6 7 8 9 D
efects
1 4 7 10 13 16 *Apr‘15
Jul Oct Jan’16
Apr Jul
Total cum effort
0
150
S 2 3 4 5 6 7 8 9S 2 4 6 7 8 9
PM
1 4 7 10 13 16 *Apr'15
Jul Oct Jan’16
Apr Jul
Total defect rate
0
80
S 2 3 4 5 6 7 8 9S 2 4 6 7 8 9 D
efects1 4 7 10 13 16 *
Apr'15
Jul Oct Jan’16
Apr Jul
Aggregate staffing rate
01530
S 2 3 4 5 6 7 8 9S 2 4 6 7 8 9 Peop
le
1 4 7 10 13 16 *Apr'15
Jul Oct Jan’16
Apr Jul
Total Cum Normal DefectsTotal Defect RateTotal Cum Effort (PM)Agg. StaffSize (ESLOC(K))PI 12.6 14.6MBI 3.0 3.5
Date 05.09.2016 (17.16 mos)
PlanActual/
ForecastEst. to
Complete613 769 1
1 1123.89 201.93 0.00
0.2137.30 67.30 0.00
Current Plan Actual
Size
04080
S 2 3 4 5 6 7 8 9S 2 4 6 7 8 9
ESLO
C (th
ousan
ds)
1 4 7 10 13 16 *Apr'15
Jul Oct Jan’16
Apr Jul
Interpolated Current Forecast Green Control Bound Yellow Contr. BoundS = Start, 2 = CDR, 3 = FCC, 4 = SIT, 5 = Freeze, 6 = IOC, 7 = Pilot, 8 = 15R, 9 = Release
Actuals, History, Outlookcompared to approved plan
19/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Welcome
Performance Improvement
Measurement
Benchmarking
Data Analytics
Case Study: Lean Development
Case Study: Technical Debt
Summary
Agenda
Poor management can increase software costsmore rapidly than any other factor.
Barry Boehm
21/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Benchmarking – Motivation
Benchmarking
How much time does the blue car need to its final destination?
It depends on several parameters:
Distance to drive (quantitative: Km, absolute scale)
Average speed (quantitative, ratio scale)
Weather(qualitative: ordinal scale)
Traffic and road(unpredictable: risk to be evaluated)
Car(condition, engine)
Driver(condition, competences)
22/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Benchmarking is the structured and systematic learning from the best in class.
Benchmarking of IT and Software applies to different dimensions
Products (e.g., quality requirements, product complexity, variance, platforms, technical debt, architecture)
Processes (e.g., roles, methods, tools, value stream, productivity)
Projects (e.g., estimation, supplier management, interfaces)
People (e.g., competences, distributed teams, organization)
What is Benchmarking?
Benchmarking
Prod
uct
ivity
Time
Improvementneeds
SustainableImprovements
Learning, Transitition
23/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Business Performance and Improvement
Benchmarking
Business Performance
BusinessPain Points
Marketing, Ops, Product Mgmt
(Customer Satis-faction, Prizing, Service, Revenues, Organization, Corporate Culture)
Sales, Marketing, Finance …
IT/Software Impact
Pain Points Performance
Change Program
Continuous Improvement
Vector Benchmarking Suite
Measurements(Gap analysis, Engineering and IT performance, contribution to business, improvement progress)
Industry benchmarks(Product, Defects, Rework, Complexity, Locations, Competences, People management, Processes)
Productivity levers: Value orientation, Waste reduction, People focus
Best practices for change
24/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Balanced Scorecard
Benchmarking
Financial PerspectiveTargets Performance MeasuresStrength Cash flow
Profit vs. targetsGrowth Sales growthShareholder value Increase of ROCE
Customer PerspectiveTargets Performance MeasuresNew products Share of new products
among all salesFast sales Delivery accuracyPrefered supplier Share of sales to long-
term clients
Innovation and Know how PerspectiveTargets Performance MeasuresTechnologyleadership
Cycle time for new productgeneration
Org. learning inproduction
Lead time until matureproduct
Time to marketaccess
Benchmark of productintroduction with competiton
Internal PerspectiveTargets Performance MeasuresTechnologicalcompetence
Development or productionbenchmark with best of class
Development Development efficiencyProduction know-how
Cycle time, marginal cost,operating profit
25/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Example: Product Data and Usage
Benchmarking
Always7%
Often13%
Sometimes16%
Rarely19%
Never45%
Source: Ebert 2016; Standish Group Study Reported since 2003 in Chaos Reports.
Goal: Reduce complexity, focus on value Question: How many of delivered features are seen as valuable?Approach: Survey and activation counts with logsAnalytics: Frequency and clusters
26/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Welcome
Performance Improvement
Measurement
Benchmarking
Data Analytics
Case Study: Lean Development
Case Study: Technical Debt
Summary
Agenda
If you cannot measure it, you cannot manage it.
Peter Drucker
28/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Smart solutionsProducts adapt to the changing environment and use> Connectivity and collaboration
Smart networkingSoftware, mechanics and electronics to create new technologies and business models> Usage data
Smart environmentsCooperating systems to communicate and exchange information> Predictive maintenance
Smart production systemsFlexible production systems enable resource-efficient production and custom products> Flexible integration
Data Analytics – Motivation
Data Analytics
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Data analyticsalso: Business Intelligence (BI) or Business Analytics (BA)
Emerging technique that dives into a data set without prior set of hypotheses
The data derive meaningful trends or intriguing findings that were not previously seen or empirically validated.
Data analytics enables quick decisions or help change policies due to trends observed
Approach Accumulation of raw data captured from various sources (i.e. discussion
boards, emails, exam logs, chat logs in e-learning systems) can be used to identify fruitful patterns and relationships
Exploratory visualization – uses exploratory data analytics by capturing relationships that are perhaps unknown or at least less formally formulated
Confirmatory visualization – theory-driven
Data Analytics Approach
Data Analytics
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Example: Market Data and Sales
Data Analytics
Usability 0 100%
Performance
Automation
Pricingscheme
Functionality
License model
Service
Installation, administration
Competitors product
New entrant
Goal: Identify New Segments and Buying CriteriaQuestion: Which buying criteria matter for which segments?Approach: Survey and sales figures across similar productsAnalytics: Multivariate analyses
29/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Welcome
Performance Improvement
Measurement
Benchmarking
Data Analytics
Case Study: Lean Development
Case Study: Technical Debt
Summary
Agenda
As far as I can tell “engineering judgement” just means they’re going to make up numbers
Richard Feynman
31/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Method: Dedicated Techniques from Lean Development
Case Study: Lean Development
Focus on value
Continuouslyimprove
Empowerpeople
Eliminatewaste
Optimize value streams
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Always7%
Often13%
Sometimes16%
Rarely19%
Never45%
Practice: Eliminate Waste
Case Study: Lean Development
Feature UsageSources:
Vector Benchmarking Suite, Microsoft, Daimler, BITKOM 2012,
Ebert 2016
Our recommendations:
Set a clear goal: Reduce waste
Analyze waste in work flows such as requirements value, requirements changes, duration of open defects, defect detection effectiveness.
Learn from industry best practices and extract concrete prioritized actions
Project: Use quality gates as defined thresholds and to ensure transparent governance.
Product: Reduce technical debt such as insufficient architecture with concrete refactoring actions.
Case Study Focus
33/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Implement the Change Project
Case Study: Lean Development
From Classic Development
Client Benefits
25% reduction of lead time and effort before project start
Reduction of hand-offs and ping-pong between project functions
20% reduction of project lead time by concurrent engineering
To Lean Agile
Vector Contribution
Streamlined development processes and interfaces
Value-orientation end-to-end
Product line engineering from requirements onwards
Efficient yet effective safety processes and governance
35/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Welcome
Performance Improvement
Measurement
Benchmarking
Data Analytics
Case Study: Lean Development
Case Study: Technical Debt
Summary
Agenda
As far as I can tell “engineering judgement” just means they’re going to make up numbers
Richard Feynman
36/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Situation: Market leader in critical infrastructure solutions
Challenge: Complexity, legacy code, volatile requirements
Business needs: Flexible solution and service development, schedule adherence, cycle time reduction
Approach: External benchmarking and internal improvement program to reduce and control technical debt
Goal: Systematically Reduce Technical Debt
Case Study: Technical Debt
1) Make technical debt visible 2) Evaluate trade-offs 3) Control technical debt
Target zone
Effort
Impact
Low
High
High
0%
50%
100%Perfor-mance
Safety
Robust-ness
Port-ability Defects
Maintain-ability
Security
Innovation
Testability
Reliability
02468
1012141618
Q1-2009
Q2-2009
Q3-2009
Q4-2009
Q1-2010
Q2-2010
Q3-2010
Q4-2010
Q1-2011
050
100150200250300350400450
Q1-2009
Q2-2009
Q3-2009
Q4-2009
Q1-2010
Q2-2010
Q3-2010
Q4-2010
Q1-2011
Impact:
0
50
100
150
200
250
300
350
Q1-2009
Q2-2009
Q3-2009
Q4-2009
Q1-2010
Q2-2010
Q3-2010
Q4-2010
Q1-2011
Cost:
0
1
2
3
4
5
6
Q1-2009
Q2-2009
Q3-2009
Q4-2009
Q1-2010
Q2-2010
Q3-2010
Q4-2010
Q1-2011
Deployment:
37/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Approach: Combination of Different Techniques
Case Study: Technical Debt
Development project
Component design
Component test
Product test
System test
Solu
tions
Develop
men
t
Component development
Commercial start
System approval
End-to-end customer focus: Solutions
Custom
er
SolutionDevelopment
Early defect detection
Incrementaldevelopment
Teamwork and ownership
System specification
38/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Systematically decreased technical debt – where it mattered most
Efficiency in development and approval, e.g. schedule adherence
Performance Improvement: Control Results
Case Study: Technical Debt
Target zone
Delivery date
Delays
39/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Welcome
Performance Improvement
Measurement
Benchmarking
Data Analytics
Case Study: Lean Development
Case Study: Technical Debt
Summary
Agenda
What gets measured, gets done.
Tom DeMarco
40/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
It is vital in today business climate to understand and manage based on measurements and benchmarks.
Three key principles Systematically measure, analyze and
benchmark Pragmatically evaluate trade-offs and
decide Sustainably control and take action
Don’t exaggerate measurements. It’s one of several tools.Dogmatism will distract potential users.
Measurement: Ambition
Summary
Our own simple law for managing technical debt: RACE
Reduce Accidents and Control Essence
41/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Analysis:What are the opportunities and threats to my business?Where do I need to be in two years?What do I need to do better to stay competitive?
Assessment:What does that mean to product innovation, development and IT?Which improvements and changes are necessary?
Alignment:Which quantitative targets?Which measurements and analytics?
Your Own Action ListSummary
42/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Concrete Benchmarks and Measurements
Summary
Software MeasurementEstablish, Extract, Evaluate, Execute
Christof Ebert and Reiner DumkeThird edition, Springer, 2007www.vector.com/books
“Few organizations have really institutionalized measurement of their products and processes. This book is bang up-to-date in both fields and packed with practical advice. For every software engineer."
Charles R. Symons, Inventor of Function Points
44/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
Optimizing Global Projects and Teams
Summary
Global Software and IT
Christof Ebert2. extended Edition, Wiley, 2011 Discount: http://bit.ly/cSjZgD
"This book stands out as the best source of information on distributed software development. Seldom do we see a book with the concepts completely backed by industry experiences and views. Software developers and managers benefit from the broad case studies.“S M Balasubramanian, Vice President, Wipro Technologies
45/47
© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30
For more information about Vector Consulting Services please visit
www.vector.com/consulting
Passion. Partner. Value.
Vector Consulting Services
Phone +49 711 80670-0 www.vector.com/consultingFax +49 711 80670-444 [email protected]