Kelly Weyrauch Agile Quality Systems LLC 2015 Opinion-Based Metrics Easy to Collect & Powerful Drivers of Process Improvement
Kelly WeyrauchAgile Quality Systems LLC2015
Opinion-Based Metrics Easy to Collect &
Powerful Drivers of Process Improvement
? Metrics are an important element to a Process Improvement Initiative• Agree• Somewhat Agree• Somewhat Disagree• Disagree
What went through you mind when I asked you to state your opinion?
What did we learn about this crowd from the answers?
Reason for Metrics
Agile Quality Systems 2015
? Metrics are an important element to a Process Improvement Initiative
Why did I ask? Why was a subjective question like this sufficient? Wanted to know if we all agreed
If so, I don’t need any more information If not, I would need to ask further questions
Sometimes, an opinion is all you need
Reason for Metrics
Agile Quality Systems 2015
? My organization has good Metrics to help us in our Process Improvement Initiatives• Agree• Somewhat Agree• Somewhat Disagree• Disagree
What did you consider to form your opinion? What did we learn about this crowd from the
answers?
Metrics Programs
Agile Quality Systems 2015
? My organization has good Metrics to help us in our Process Improvement Initiatives
Why did I ask? I want you to think about Metrics from your perspective Wanted to know if we all agreed or differed
Perspective is key to Subjective Questions
Metrics Programs
Agile Quality Systems 2015
Benefits of Process Improvement
Agile Quality Systems 2015
Productivity Predictability Team Health
Quality
? What do you use to measure Quality?
? What’s good about those?? What’s not so good?
Quality Metrics
Agile Quality Systems 2015
? What do you use to measure Productivity?
? What’s good about those?? What’s not so good?
Productivity Metrics
Agile Quality Systems 2015
? What do you use to measure Predictability?
? What’s good about those?? What’s not so good?
Predictability Metrics
Agile Quality Systems 2015
? What do you use to measure Team Health?
? What’s good about those?? What’s not so good?
Team Health Metrics
Agile Quality Systems 2015
Quality Productivity Predictability Team Health
? We are good at ___• Agree• Somewhat Agree• Somewhat Disagree• Disagree
Benefits of Process Improvement
Agile Quality Systems 2015
Quality Productivity Predictability Team Health
? We are good at ___
What would it mean if: We all strongly agreed we were good at it? We all strongly disagreed?
Benefits of Process Improvement
Agile Quality Systems 2015
“WE” = The entire team, not individualsNeed to scope the survey – define who the “we” is
“ARE” = Where we are right nowSo we can decide whether a change is warranted
“GOOD” = What we want to be
Team owns the metricTeam owns the action to improve it
Power of the question:“We are good at ___”
Agile Quality Systems 2015
Quality Productivity Predictability Team Health
? My organization is good at ___
Good categories, but too broad to be actionable
Benefits of Process Improvement
Agile Quality Systems 2015
Benefits
Agile Quality Systems 2015
Productivity Predictability Team Health
Quality
Benefits
Agile Quality Systems 2015
Quality Productivity
Predictability
Team Health
• Customer facing• Meets customer expectations of
“good”• Does what they want / need• Bug free• …
• Internal facing• Reliable / stable / maintainable• Find bugs early• Confident in assessing quality• …
• …
Benefits
Agile Quality Systems 2015
Quality
Productivity Predictability
Team Health
Productivity = Value / Cost
• Value• Deliver high value to customer
(features / functionality)• Deliver high value to business
(deliverables meeting requirements of process/phases/business)
• Prioritize well• …
• Cost• Efficient (lean)• Fast• Right amount of rework• …
• …
Benefits
Agile Quality Systems 2015
Quality
Productivity
Predictability Team Health
Predictability = Making good commitments, meeting them with reasonable adjustments
• Awareness• Visibility• Feedback
• Control• Reality-based estimation and
planning• Manage all levers of project
management• Reasonable responses to change
(new needs / problems)
• …
Benefits
Agile Quality Systems 2015
Quality
Productivity
Predictability
Team Health
Team Health = The soft benefits that lead to more tangible ones
• See my impact on the project• Accountable & Empowered• Aligning skills & interests• Enjoyment• Teamwork (intra-team, across
teams)• …
Defining the topics for a survey forces a definition of, or even a recognition of, what matters to you
Asking about those topics tells everyone what mattersProviding focus to process-improvement effortsProvides motivation - “Want to move the dial”
Choosing the Topics for a Subjective Survey
Agile Quality Systems 2015
Establishes what matters to us, what we care about being good at
Provides a baseline for improvement Motivates and empowers us to improve Shows whether we improved
Benefits Survey
Agile Quality Systems 2015
For each of the Topics, two questions “We are good at ___” “Our practices are helping with ___”Are we set up for success, or are we
hindered by our processes / practices / behaviors / …
Adds information on another factor for determining the path for improvement if we aren’t where we want to be
Survey Structure
Agile Quality Systems 2015
Six-point scale for answer Strongly Disagree Disagree Slightly Disagree Slightly Agree Agree Strongly Agree
“Agreement” – your opinionEncourages thought to explain/defend the opinion
Forces a choice– no middle groundEnough choices to differentiate, not too many to
overwhelm or quibble
Survey Structure
Agile Quality Systems 2015
An Example - Quality
Agile Quality Systems 2015
The product quality is high.
Our practices help us get better quality.
We satisfy our customers.
Our practices help us satisfy our customers.
We deliver bug-free software.
Our practices help us deliver bug-free software.
We deliver safe and effective software.
Our practices help us deliver safe and effective software.
We deliver software that is reliable, stable, maintainable.
Our practices help us deliver software that is reliable, stable, maintainable.
We find bugs early enough.
Our practices help us find bugs earlier.
We are confident in assessing quality.
Our practices increase our confidence in assessing quality.
Strongly Disagree Disagree Slightly Disagree Slightly Agree Agree Strongly Agree
4.5
4.2
4.4
3.7
3.5
3.8
4.4
4.2
4.4
3.8
3.5
3.5
3.9
3.7
4.0
Productivity – Team
Agile Quality Systems 2015
Our productivity is high.
Our practices help us increase productivity.
We deliver high value to our customers.
Our practices help us deliver high value to our customers.
We deliver high value to the business.
Our practices help us deliver high value to the business.
We prioritize well.
Our practices help us prioritize our work.
We are efficient.
Our practices help us be efficient.
We are fast.
Our practices help us be faster.
We do the right amount of rework.
Our practices help us optimize the amount of rework.
Strongly Disagree Disagree Slightly Disagree Slightly Agree Agree Strongly Agree
3.9
3.8
4.2
3.8
4.7
4.1
3.2
3.4
3.2
3.5
3.2
3.3
3.2
3.4
3.6
Productivity – Management
Agile Quality Systems 2015
Our productivity is high.
Our practices help us increase productivity.
We deliver high value to our customers.
Our practices help us deliver high value to our customers.
We deliver high value to the business.
Our practices help us deliver high value to the business.
We prioritize well.
Our practices help us prioritize our work.
We are efficient.
Our practices help us be efficient.
We are fast.
Our practices help us be faster.
We do the right amount of rework.
Our practices help us optimize the amount of rework.
Strongly Disagree Disagree Slightly Disagree Slightly Agree Agree Strongly Agree
2.9
2.3
4.6
3.7
4.0
3.4
3.2
2.8
2.6
2.4
2.2
2.1
2.6
2.1
2.9
An Example - Predictability
Agile Quality Systems 2015
Our projects are predictable.
Our practices help us be predictable.
Project progress is visible.
Our practices help us make project progress visible.
We gather and make use of feedback.
Our practices help us gather and make use of feedback.
We are good at estimating.
Our practices help us make better estimates.
We manage all levers of project management.
Our practices help us manage all levers of project management.
We respond reasonably to changes.
Our practices help us respond reasonably to changes.
Strongly Disagree Disagree Slightly Disagree Slightly Agree Agree Strongly Agree
3.1
3.2
3.2
3.5
3.6
3.3
2.2
2.8
2.2
2.4
4.2
3.8
3.1
An Example – Team Health
Agile Quality Systems 2015
Our team is healthy.
Our practices help us improve the health of the team.
I see my impact on the project.
Our practices help me see my impact on the project.
I am accountable and empowered to take action.
Our practices help me be accountable and empowered to take action.
I can align my skills and interests with the needs of the team.
Our practices help me align my skills and interests with the needs of the team.
I enjoy what I work on.
Our practices help me enjoy what I work on.
My team works well together.
Our practices help my team work well together.
My team works well with other teams.
Our practices help my team work well with other teams.
Strongly Disagree Disagree Slightly Disagree Slightly Agree Agree Strongly Agree
4.6
4.6
4.4
4.1
4.6
3.9
4.6
4.3
4.8
3.8
4.9
4.4
4.1
3.5
4.3
Summarization:Radar Graph
Agile Quality Systems 2015
PRODUCT QUALITY
PRODUCTIVITY
PREDICTABILITY
TEAM HEALTH
Benefits Survey
Initial
Predictability Comparison
Agile Quality Systems 2015
1.0 2.0 3.0 4.0 5.0 6.0 1.0
2.0
3.0
4.0
5.0
6.0 Agile Benefits - PREDICTABILITY
Our projects are predictable.
Project progress is visible.
We gather and make use of feedback.
We are good at estimating.
We manage all levers of project management.
We respond reasonably to changes.
We are good at ___
Pra
cti
ce
s a
re h
elp
ing
wit
h _
__
_
? “Subjective questions are useful”• Agree• Somewhat Agree• Somewhat Disagree• Disagree
Subjective, and that’s a good thingThe opinion of the team mattersThe opinion of stakeholders mattersBetter if supported by objective evidence
Subjective Questions
Agile Quality Systems 2015
The “How Do You KNOW…” story Do you trust the opinion of the team? Are they qualified to have such an opinion? If the team is to own process improvement, they
must also own the opinion on the need for it Is the team’s opinion consistent with:
Other metrics? Reality? (Who defines “reality”?) Stakeholders (yours, management, customers, …)?
How Do You “KNOW”?
Agile Quality Systems 2015
? “Surveys must be anonymous”• Agree• Somewhat Agree• Somewhat Disagree• Disagree
? Why? Why Not? (columns on whiteboard)Surveys are made to be anonymous when:
There is a perceived “right or wrong” answerUndesired consequence comes from answering wrong
To the survey taker, or someone they care aboutUndesired action after the survey
Need to be anonymous suggests mistrust
Anonymous Surveys
Agile Quality Systems 2015
Surveys described here must not be anonymousTeam should be comfortable sharing honest opinions
within the team and with stakeholdersTeam must own their opinions
And the desire to make improvementsWant to identify & explore different perspectivesWant to evaluate changes over time
Anonymous Surveys
Agile Quality Systems 2015
How much do you care about this topic? 5 = My reason for living, the real #1 on a priority list 4 = A top priority, but accepting there are some ahead
of it 3 = This matters, but it’s not a huge thing 2 = “Would be nice to be good at this, but….” 1 = This doesn’t matter to me, but it’s okay if it matters
to someone else 0 = This doesn’t matter to me, and it shouldn’t matter
to anyone else either
Additional Survey Questions:Interest Level, Importance
Agile Quality Systems 2015
A variation of “how much do you care” could be “how good do you want to be” 5 = I want us to be excellent at this (A Student) 4 = I want us to be good at this (B Student) 3 = I want to be good at this, but not right now 2 = There is some good and some bad in this 1 = I think this will hurt us, so if it matters to someone
else, keep it away from us 0 = I don’t want us to care about this at all, it would
hurt us to be good at this
Additional Survey Questions:Interest Level, Importance
Agile Quality Systems 2015
How strongly do you believe your opinion is right, reasonable, defendable?
5 = Adamant. Can make a strong/complete case that would convince most. Either have strong supporting
evidence, or such an easy case that you don’t need evidence. Can easily explain why, and would not be swayed by differing opinions or perspectives.
4 = Certain, but… Strong from what you can see, but possibly incomplete. As strong as #5 from your perspective, but
you accept that there could be other perspectives with differing opinions.
3 = Convinced, but hard time convincing others. You have a strong belief, but would have a hard time explaining it to others, and would change your
opinion if faced with a strong counter argument.
2 = Lots of holes in the case, but a reasonable “gut feel” about it Would readily accept a disagreeing opinion, would be easily swayed
1 = Nothing to back it up, it’s just what I think. 0 = Don’t count my vote, my opinion is meaningless.
Additional Survey Questions:Strength of Opinion
Agile Quality Systems 2015
Data-Driven Metrics can be hard, but Subjective Surveys are easy to do
Opinions may be all you need Discussions of the opinions has huge value Combine Subjective Surveys with other “hard
data” to tell a more complete story
Summary
Agile Quality Systems 2015
Kelly Weyrauch [email protected] 763-688-0980
www.AgileQualitySystems.com
Connect with me on LinkedIn
The End
Agile Quality Systems 2015