The benefits of using an APM solution while performance testing

DevOpsGuys

@DevOpsGuys www.devopsguys.com 1

DevOpsGuysThe benefits of using an

APM solution while performance testing

Why Perf testing without APM is like Corona without the Lime…

DevOpsGuys

@DevOpsGuys www.devopsguys.com 2http://www.flickr.com/photos/lendog64/5781379311/

DevOpsGuys


Perf Testing & APM in 30mins

• Why Performance Test? • What is APM anyway? • Benefits of APM when Perf Testing• With some case studies along the

way!• Q&A

DevOpsGuys


Why Performance Test?

1. Ensure that we meet our customer’s expectations of website response time– So that they convert and we make £££

2. Ensure that we have sufficient computing capacity to meet #1 under load (load testing)

3. Understand how the system reacts when things go wrong (stress testing)

DevOpsGuys


DevOpsGuys


APM Defined

“In the fields of information technology and systems management, application performance management (APM), is the monitoring and managing of performance and availability of software applications. APM strives to detect and diagnose application performance problems to maintain an expected level of service. APM is the translation of IT metrics into business meaning (i.e. value).”http://en.wikipedia.org/wiki/Application_Performance_Management

DevOpsGuys


“5 Dimensions of APM*”

1. End user experience monitoring - (Active and passive)

2. Application runtime architecture discovery and modeling

3. User-defined transaction profiling (also called business transaction management)

4. Application component monitoring5. Reporting & Application data analytics

*According to Gartner

DevOpsGuys

@DevOpsGuys www.devopsguys.com

CA

QuestHP

OpTier

Opnet

IBM

Compuware

Oracle

MicrosoftSL

Inetco

NetScout

eG InnovationsProgress

ASG

IderaVisual networks

Info vista

Network Instruments

Knoa

Aternity

Coradient

CorrelsenseNastel

dynaTracePrecise

Arctures Technologies

BMC

Managed Engine

New RelicAppDynamics

2011Riverbed

OpTier

2012

AppNeta

Dell

2013

Gartner APM Magic Quadrant

Legac

y Ven

dors

DevOpsGuys


BENEFITS OF LOAD TESTING WITH APM TOOLS

Never leave home without it…

DevOpsGuys


#1See the Big Picture

(aka Systems Thinking)

DevOpsGuys


Systems Thinking 101

“Systems thinking is a framework for seeing interrelationships rather than things, for seeing patterns rather than static snapshots. It is a set of general principles spanning fields as diverse as physical and social sciences, engineering and management.” - Peter Senge

DevOpsGuys


Command & Control vs Systems Thinking

http://www.systemsthinking.co.uk/2-1.asp

DevOpsGuys


CASE STUDY #12 WEEKS WASTED…

“any improvement not made at the constraint is an illusion”

DevOpsGuys


The Scenario

• E-commerce customer• New website • High data-start times on page load• Poor load testing results• Initial conversation with client• 2 week delay while hosting provider

scaled out the hardware…• Problem still not fixed…

DevOpsGuys


What’s the root cause of the ~1sec Data Start?

Waterfall Diagram of page load times

DevOpsGuys


DevOpsGuys


#2Drill down to the

Detail(Code-level)

DevOpsGuys


Find the slow code!

DevOpsGuys


#3Faster Iteration= Better Value

DevOpsGuys


Maximise Testing Value

Test

AnalyseFix

Deploy

DevOpsGuys


CASE STUDY #156X FASTER IN A DAY…

9am to 9pm. 12hrs. 56x faster!

DevOpsGuys


The Scenario

• New website pre-launch testing• Customer Portal (SaaS solution)• Start Load testing 9am• 4 “Test & Fix” cycles in the day • Final testing completed at 9pm

DevOpsGuys


The End Results = 56x Average

ResponseThroughput

(Req/sec)Initial Testing 3024ms 16/minFinal Result 54ms 59/min

56x faster 3.5x throughput

Before 3024ms After

54ms

DevOpsGuys


Nice flat line!

Before

After

DevOpsGuys


#4Team Work“The Enemy is Performance,

not each other…”

DevOpsGuys


The Blame Game…

Ops DevQA

DevOpsGuys


The Fix-It Game…

Ops

DevQA

DevOpsGuys


DevOpsGuys


Summary

1. See the Big Picture (systems thinking)

2. Drill down to the details3. Faster Iteration = Better Value4. Stop the “Blame Game”– aka “There’s your Problem!”

DevOpsGuys


DevOpsGuys

Thank YouWorkshop Session @

1345End-to-end testing & APM demooperations | availability | performance | scalability | migration | strategy

DevOpsGuys


DevOpsGuys provide development & operations support (DevOps) services

for your online applications.

You run your business. We run your website.

It’s as simple as that.

What We Do

DevOpsGuys


Our service portfolio

• 24 x 7 Application-Centric web operations

• DevOps Transformation & Training

• Continuous Delivery implementation

• Infrastructure Automation & “Infrastructure as Code”

• Application Performance Management

• AppDynamics Implementation & Consultancy

• Web Performance consulting

• Web monitoring• Load Testing• Test Automation and

scripting• JetBrains Product

Implementation & Consultancy

• Octopus Deploy implementation

DevOpsGuys


James Smith (@TheDevMgr)

• 16 years of software delivery experience.

• Proven track record– Pizza Hut– KFC– Dell– RAC– Aviva– Fujitsu Siemens– Whitbread

Steve Thair (@TheOpsMgr)

• 25 years of IT Operations experience.

• Proven track record– Vodafone– Credit Suisse Group– IBM– BNP Paribas– Zurich Financial Services– Totaljobs Group– TSL Education

The Leadership Team

DevOpsGuys


Contact UsVisit us at www.devopsguys.com

Call our team on +44 (0)20 8798 3079

Email our engineers at [email protected]

The benefits of using an APM solution while performance testing

Technology

performance testing

perf testing apm

load load testing

apm solution

benefits of apm

itbenefits of load testing

apm tools

dimensions of apm