What is (good) performance?
Tal Olier, HP SoftwareJan, 2010
(Israel’s SQL Server UG)
Who am I?
Tal Olier
Works for HP-SW, R&D
• Database expert• System architect• Performance theme
manager
Scope of this lecture
ISV – Independent software provider
Product / Release
Rich functionality
Today I am going to tell you my story - a story about the product's/release's performance and what do we actually mean when aiming for good performance...
So I went looking for a performance problem...
Demo
Simple application that loads objects to memory
At object 10,000 there is an important development ...
Okay, Enough with It
You have just been subjected to a behavior experiment without you knowing about it
Demo source
kse_demo_application.exe is a simple sleep.exe that sleeps %1 miliseconds.
run_demo.bat is a script that displays numbers from 1 .. 9958 and gets stuck
End of the run_demo.bat script:...echo 9955 kse_demo_application.exe -m 8000echo 9956 kse_demo_application.exe -m 16000echo 9957kse_demo_application.exe -m 32000echo 9958kse_demo_application.exe -m 3200000 #not going anywhere from here ...
Okay, enough with It
• Demo goal was to let us experience on our own time what our customers are experiencing in their meetings (and they pay for our products…)
So what does makes good performance?
Anybody?
So I asked god..
So I asked god
And god said..
And god said..
And god said..
And god said..
I guess god is not the answer to everything, let's ask people...
So what does makes good performance?
Database CPU ?
So what does makes good performance?
Server memory ?
So what does makes good performance?
How many disks I have?
So what does makes good performance?
The size of my window?
So what does makes good performance?
Who am I married to?
So what does makes good performance?
Who am I?
So what does makes good performance?
What am I?
So what does make good performance?
???
This is not getting anywhere...
I turn to "observing"...
It was awkward at the beginning...
The family was looking funny...
Then I learn some beautiful things...
Although the smell sometimes was weird...
I finaly managed to have some fun...
I found it…
So what does make good performance?
Customer perception...
So what does make good performance?
Customer perception...
Of the application performance
Who is the customer?
Who is the customer?
• Developer• Help desk person• CIO• Analyst• Tester• Manager• ...
Who is the customer?
• Developer• Help desk person• CIO• Analyst• Tester• Manager• ...
= USERS
Who is the customer?
• Developer• Help desk person• CIO• Analyst• Tester• Manager• ...
= USERS
Who is the customer? (cont.)
Usability, this is it?
What about servers, scalability, network, all of the stuff that we can't live without?
Who is the customer? (cont.)
Who is the customer? (cont.)
Application Administrator
Who is the customer? (cont.)
?= Application Administrator
Who is the customer? (cont.)
Organization perspective:• Mainly an IT person• Responsible for the
application firmness inside the organization
• A mixture of a technical-geek and a functional architect
• Provides initial aid for any problem inside the organization (including performance of our product)
= Application Administrator
Who is the customer? (cont.)
Organization perspective:• Mainly an IT person• Responsible for the
application firmness inside the organization
• A mixture of a technical-geek and a functional architect
• Provides initial aid for any problem inside the organization (including performance of our product)
Provider perspective:• Our best friend inside the
organization• The guy who collect info
for• The guy who read our
capacity planning guides and plans the next phase
• Single point of perception
The inevitable equation :)
+ =
“Customer perception”, I need elaboration...
Customer perception of the application performance
What is it composed of?
Customer perception of the application performance
Customer perception of the application performance
What drives the end user?
Customer perception of the application performance
9 2 5
Customer perception of the application performance
• 9 2 5 (09:00 - 17:00)• Let me complete my job
ASAP and go home
:)
Customer perception of the application performance
• 9 2 5 (09:00 - 17:00)• Let me complete my job
ASAP and go home
:)
What drives the application administrator ?
Customer perception of the application performance
• 9 2 5 (09:00 - 17:00)• Let me complete my job
ASAP and go home
:)
9 2 5
Customer perception of the application performance
• 9 2 5 (09:00 - 17:00)• Let me complete my job
ASAP and go home
:)
• 9 2 5 (09:00 - 17:00)• Let me complete my job
ASAP and go home
:)
Customer perception of the application performance
I need a usable application now !
I need all my users to be able to use the application, all the time and for a long time.
Customer perception of the application performance
? ? ? ? ? ?
Customer perception of the application performance
• Product’s usability
? ? ?
Customer perception of the application performance
• Product’s usability • Product’s scalability • Product’s robustness• Product’s support /
supportability
Usability, ... Supportabilty - I am not sure I know those words
Customer perception of the application performance
Product’s usability -
?
Customer perception of the application performance
Product’s usability -
From Wikipedia: In human-computer interaction and computer science, usability usually refers to the elegance and clarity with which the interaction with a Computer program or a web site is designed
"It is impossible to work this way..."
"S**t !!! It is impossible to work this way..."
Customer perception of the application performance
Product’s scalability -
?
Customer perception of the application performance
Product’s scalability -
From Wikipedia: The capability of a system to increase total throughput under an increased load when resources (typically hardware) are added
“Mother f***er!!! I don't know what HW I need to run my site?”
Customer perception of the application performance
Product’s robustness -
?
Customer perception of the application performance
Product’s robustness -
From Wikipedia: Robustness is the quality of being able to withstand stresses, pressures, or changes in procedure or circumstance. A system, organism or design may be said to be "robust“ if it is capable of coping well with variations (sometimes unpredictable variations) in its operating environment with minimal damage, alteration or loss of functionality.
“Damn, p re ssing this one a lway s stuck it a ll...”
Customer perception of the application performance
Product’s support / supportability -
?
Customer perception of the application performance
Product’s support / supportability -
Pinpointing the problem as fast as possible providing an accurate, cost saving and low risk solution that requires minimum customer activity
“This support case is open for a long time and up until now those ma***cs
were unable to pinpoint the problem” ...
Customer perception of the application performance
• Product’s usability • Product’s scalability • Product’s robustness• Product’s support /
supportability
This is a great discovery, let's make a poster...
Performance axis
Customer perception of the application performance
• Product’s usability• Product’s scalability • Product’s robustness• Product’s support /
supportability
Customer perception of the application performance
• Product’s usability• Product’s scalability • Product’s robustness• Product’s support /
supportability
So I am a project manager and would like to use this information in order to make my product better...
Customer perception of the application performance
• Product’s usability• Product’s scalability • Product’s robustness• Product’s support /
supportability
So I am a project manager and would like to use this information in order to make my product better...
Can we break it down to something practical?
Sounds like I am going to do some additional research...
Customer perception of the application performanceProduct’s usability - itemization
Customer perception of the application performanceProduct’s usability - itemization
• Embarrassment – “waiting for a long time for such a simple operation is a major embarrassment”
Customer perception of the application performanceProduct’s usability - itemization
• Embarrassment – “waiting for a long time for such a simple operation is a major embarrassment”
• Operation usage – differentiation between activities based on “amount/time” e.g. operations being performed many times during the day and one time daily operations
Customer perception of the application performanceProduct’s usability - itemization
• Embarrassment – “waiting for a long time for such a simple operation is a major embarrassment”
• Operation usage – differentiation between activities based on “amount/time” e.g. operations being performed many times during the day and one time daily operations
• Non-Intuitiveness of current – inability to agree with current solution model (current performance penalty) based on one’s intuition
Customer perception of the application performanceProduct’s usability - itemization
• Embarrassment – “waiting for a long time for such a simple operation is a major embarrassment”
• Operation usage – differentiation between activities based on “amount/time” e.g. operations being performed many times during the day and one time daily operations
• Non-Intuitiveness of current – inability to agree with current solution model (current performance penalty) based on one’s intuition
• Expensiveness non-acceptance – “it is so easy to fix, why isn’t that fixed?”
Customer perception of the application performanceProduct’s scalability - itemization
Customer perception of the application performanceProduct’s scalability - itemization
• Pure scalability – existing performance load issue which is not solved if more "machines power" is added
Customer perception of the application performanceProduct’s scalability - itemization
• Pure scalability – existing performance issue which is not solved if more "machines power" is added
• Suitability discrepancy – mismatch between the customer scale and his machinery strength causing issues
Customer perception of the application performanceProduct’s scalability - itemization
• Pure scalability – existing performance issue which is not solved if more "machines power" is added
• Suitability discrepancy – mismatch between the customer scale and his machinery strength causing issues
• Inability to predict – missing the ability to predict that a scale up or out is required
Customer perception of the application performance
Product’s robustness - itemization
Customer perception of the application performance
Product’s robustness - itemization
• Pure robustness – existing performance issue caused by stress, pressure, or changes in procedure or circumstance.
Customer perception of the application performance
Product’s robustness - itemization
• Pure robustness – existing performance issue caused by stress, pressure, or changes in procedure or circumstance.
• Robustness problem prevention - blocking robustness issues from happening e.g. preventing a server resource from being abused by a client
Customer perception of the application performance
Product’s support / supportability - itemization
Customer perception of the application performance
Product’s support / supportability - itemization
• Lengthy resolution - it takes a long time until get the root cause and fix
Customer perception of the application performance
Product’s support / supportability - itemization
• Lengthy resolution - it takes a long time until get the root cause and fix
• Risky solution/resolution - o Understanding what's wrong (sometimes) involves
"tweaking" in production siteo Solution provided (frequently) imposes a certain amount
of risk
Customer perception of the application performance
Product’s support / supportability - itemization
• Lengthy resolution - it takes a long time until get the root cause and fix
• Risky solution/resolution - o Understanding what's wrong (sometimes) involves
"tweaking" in production siteo Solution provided (frequently) imposes a certain amount
of risk• Expensive solution - In many cases the solution price is
high e.g. buy new hardware
Performance axis
Performance axis
I am practical, now what?
Translating into actions (cont.)
“Hooking” into the development processes:
• “Performance Risk” attribute in the ALM system• Address feature leaders• Technical and functional design reviews• Addressed by DEV - technical consulting
Translating into actions
Research aspects (Scope: R&D):
Methodology Aspect
Performance usability Response time validation
Performance scalability and robustness Resource consumption analysis
Performance support Architecture, flexibility
Let's summ up
Customer perception
2 types of "customers"
Methodology:• Response• Resource• Architectural
Flexibility
Performance is like...A poem by Alex (Bummer) Perfomsky. (the 2nd)
Sometimes you cry, but no one will notice your tears.
Sometimes you laugh, but no one will notice your smile.
Sometimes you are scared, but no one will notice you are shivering.
Sometimes you fall, but no one will notice your pain.
But if you fart once…
Performance is like...A poem by Alex (Bummer) Perfomsky. (the 2nd)
Sometimes you cry, but no one will notice your tears.
Sometimes you laugh, but no one will notice your smile.
Sometimes you are scared, but no one will notice you are shivering.
Sometimes you fall, but no one will notice your pain.
But if you fart once…
The End
for comments / questions:
[email protected]@hp.com