Whipping Your Apps Into Shape Michael Lee, VP Technology
Whipping Your Apps Into Shape Michael Lee, VP Technology
• Introduction – What is Beachbody? – Our Growing Pain – Our Problem
• Decide. – Why We Chose AppDynamics
• Commit. – Governance Plan – Low barrier to entry: Make it accessible
and consumable • Succeed.
– Lean Out – Explosive Speed – Strength Training – Our Solution
• Q&A
What new features would you like to see in AppDynamics or features Beachbody should take a closer look at? Text the keyword BEACH and your response to 22333
$1.3B Grew exponentially 1998-2015 • Fitness company with four major lines of business
– Direct to consumer – Network – Certifications – On Demand
• Our programs – 1998 Turbo Jam – 2003 P90x – 2007 Team Beachbody – 2010 Shakeology – 2014 Beachbody Live – 2015 Apparel
What new features would you like to see in AppDynamics or features Beachbody should take a closer look at? Text the keyword BEACH and your response to 22333
Our challenge
• Growth led to sprawl which led to knowledge isolation
• Architecture was not ready for our increasing data and customer volumes – Many key services SaaS hosted, with companies that did not grow with us
• Bringing applications in house with a team more accustomed to hosted applications
• Need to deliver user facing, web speed experience with marketing driven traffic spikes across eCommerce, CMS, digital distribution, and analytics – Need to be able to monitor those businesses on more than just technical metrics in real time
What new features would you like to see in AppDynamics or features Beachbody should take a closer look at? Text the keyword BEACH and your response to 22333
Warehouse
Tax
Orders
Accounting
Payroll
Payments
Media
Customer Service
Procurement
Continuities
Telemarketing
Billing
1998
Text the keyword BEACH and your response to 22333
Warehouse
Tax
Orders
Call Switch
Accounting
Payroll
Payments
Media
Customer Service
Procurement
Continuities
Telemarketing
Billing
Community
eCommerce
1998 2000
Text the keyword BEACH and your response to 22333
Warehouse
Tax
Orders
Call Switch
Accounting
Payroll
Payments
Media
Customer Service
Procurement
Continuities
Telemarketing
Billing
Community
eCommerce
1998 2000 2005
Genealogy
Coach
Club
Commission
Content
Coach Reports
Chat
Affiliate
Text the keyword BEACH and your response to 22333
Warehouse
Tax
Orders
Call Switch
Accounting
Payroll
Payments
Media
Customer Service
Procurement
Continuities
Telemarketing
Billing
Community
eCommerce
1998 2000 2005 2010
Assets
Genealogy
Revenue
Revenue Recognition
Cash
Reporting
Accounts Payable
Coach
Club
Commission
Email Management
Content
Coach Reports
Coach Relations
General Ledger
Chat
Accounts Receivable
Replicated Sites
Affiliate
Text the keyword BEACH and your response to 22333
Warehouse
Tax
Orders
Call Switch
Accounting
Payroll
Payments
Media
Customer Service
Procurement
Continuities
Telemarketing
Billing
Community
eCommerce
1998 2000 2005 2010 2015
Assets
Genealogy Order Reporting
Properties
Marketing Database
Projects
Revenue
Revenue Recognition
Video
Workshops
Cash
Reporting
Accounts Payable
Coach
Club
Commission
Email Management
Fitness Tracking
Content
Coach Reports
Coach Relations
General Ledger
Chat
Accounts Receivable
Replicated Sites
Affiliate Email Marketing
Text the keyword BEACH and your response to 22333
Warehouse
Tax
Orders
Call Switch
Accounting
Payroll
Payments
Media
Customer Service
Procurement
Continuities
Telemarketing
Billing
Community
eCommerce
1998 2000 2005 2010 2015 2016+
Assets
Genealogy Order Reporting
Properties
Marketing Database
Projects
Revenue
Revenue Recognition
Video
Workshops
Cash
Reporting
Accounts Payable
Coach
Club
Commission
Email Management
Fitness Tracking
Content
Coach Reports
Coach Relations
General Ledger
Chat
Accounts Receivable
Replicated Sites
Affiliate Email Marketing
International
Text the keyword BEACH and your response to 22333
i3c
Media
Quickbooks
Warehouse Telemarketing
Initial Years
Text the keyword BEACH and your response to 22333
Four Resources in IT Basic Accounting <3 3rd Party Integrations
i3c
Media
Quickbooks
Warehouse Telemarketing eCommerce
Call Switch
Message Boards
Customer Service eCometry
mybeachbody
Enterprise Service Bus
Growing Pains
Text the keyword BEACH and your response to 22333
3rd Party Partner Integrations 150 Resources in IT
i3c
Media
Quickbooks
Warehouse Telemarketing eCommerce
Call Switch
Message Boards
Customer Service eCometry
mybeachbody Coach CRM Certification Digital
Enterprise Service Bus
Today
Text the keyword BEACH and your response to 22333
3rd Party Partners Critical to our business 300+ Resources in IT
Media Agencies
Channels & Affiliates
Dist Center (3PL) Telemarketing Manufacturer Supplier B2B
Countries COO BB in a Box BB Live bb.com
External Service Layer
CRM Email Delivery
Content Management MDM
Commerce Engine
ERP
MLM Engine
Non-Transactional
Corporate Systems
Data Asset Management
Media Mgmt (Pioneer)
Merchant Portal
Campaign & Test
Management BI Portal C/S Portal R&D
(Agile PLM)
Payment Service
Internal Service Layer
Streaming Players
Club.tbb.com
2018
Micro-service Architecture API –driven Ready for International Exponential Revenue Growth Target Partner Development Centers
Obese Too many tools, platforms, and services. PCI Compliance, SoX, Security, & International Growth
Slow MTTR on $150,000/hour. Business growing and we couldn’t keep up
Weak 3rd Party Integrations were owning us. We were losing focus on the user.
Our Problem: No Foundation. No Anchor. No DevOps Culture
Text the keyword BEACH and your response to 22333
Decide. Choose your program.
Text the keyword BEACH and your response to 22333
Why we chose AppDynamics
End-to-end visibility, from NOC to Engineer to Product Manager
• We have a team focused on integration with a large enterprise software footprint – Wanted instant value without code changes
• End user monitoring was key to our strategy of providing higher levels of service to our users – Bringing monitoring beyond DevOps all the way to Business Value
• Integrated triage – Minimize cultural impact while rolling out DevOps to a company unaccustomed to the level of responsibility
• Partnership and support – Software with Service, not just SaaS
Text the keyword BEACH and your response to 22333
Commit. Implement the solution.
Text the keyword BEACH and your response to 22333
Drive Engagement
Culture = Values + Execution
• Training
• Governance plan – Two TAM’s + a project manager coordinating with our internal teams – Across Eight Environments – One Dev, one Dev/Int, three QA, one Staging, one UAT, one Production
• Innovation Hackday – iPad to the winner of the AppD prize – Release Engineer automated
rollout to our systems
• DevOps contest – 7 Teams – 15 Individuals
• Beachbody takeover of LA AppDynamics User Group – Over 50 People show up
Text the keyword BEACH and your response to 22333
So, what are we monitoring? What apps? What databases? What sources? What scale?
1,600
250 0
400
800
1200
1600
2000
Text the keyword BEACH and your response to 22333
Succeed. The “after” photo.
Text the keyword BEACH and your response to 22333
Copyright © 2015 AppDynamics. All rights reserved. 22
Text the keyword BEACH and your response to 22333
Governance & DevOps Culture
Lean Out. Focus & Anchor on the Core User
Text the keyword BEACH and
your response to 22333
ADOPT 6. AppDynamics
Copyright © 2015 AppDynamics. All rights reserved. 24
Text the keyword BEACH and your response to 22333
BB Live Dashboard (Eng/Product) Business Metrics Tech Metrics
Product Metrics
Responsive
Desktop
Copyright © 2015 AppDynamics. All rights reserved. 25
Text the keyword BEACH and your response to 22333
Executive Dashboard
Copyright © 2015 AppDynamics. All rights reserved. 26
Text the keyword BEACH and your response to 22333
Explosive Speed. Improve Goodwill and Increase Revenue
Leverage Advanced AppDynamics. DevOps Solutions Must Empower The Delivery Team
Copyright © 2015 AppDynamics. All rights reserved. 27
Text the keyword BEACH and your response to 22333
Automatic Application Mapping
Copyright © 2015 AppDynamics. All rights reserved. 28
Text the keyword BEACH and your response to 22333
That’s not all…. + =
So, what does this really mean? 1. AppD detects MySQL is down 2. It clears out the bad bin files 3. It restarts MySQL 4. You know it happened, but nobody had to wake up!
Automated Remediation
Copyright © 2015 AppDynamics. All rights reserved. 29
Automated Remediation
DevOps + AppDynamics = a better night’s sleep
• Imagine keeping the entire application relationship in your mind • How do you when the application slowness you’re seeing is your
app or a lower tier? • With dynamic mapping, you can see how the apps are related,
where slowness in one app is propagating up • This allows you to scale up the correct tier, focus on indices in
the correct DB, and not waste your time in examining a victim rather than the perpetrator
• You can have your NOC call the right team to fix the underlying problem, rather than waking somebody up to triage a symptom
Copyright © 2015 AppDynamics. All rights reserved. 30
Text the keyword BEACH and your response to 22333
Copyright © 2015 AppDynamics. All rights reserved. 31
Text the keyword BEACH and your response to 22333
War Room
• Beachbody is heavily dependent on consulting and third parties. • Driving consensus and a consolidated view of what’s interesting
during triage is key to resolving a problem • War room allows for a single channel to bring people on
disparate systems together looking at the same data, talking in the same place.
• It’s a great read-only place to share data with third parties you’re troubleshooting with – Show them yours through the built-in widgets
• Pull in theirs using iFrames
Copyright © 2015 AppDynamics. All rights reserved. 32
Text the keyword BEACH and your response to 22333
What’s your normal bridge look like?
Copyright © 2015 AppDynamics. All rights reserved. 33
Who’s on the bridge?
What’s been
done? Who’s on the bridge?
What’s been
done?
Who’s on the bridge?
What’s been
done?
What’s been
done?
Who’s on the bridge?
Text the keyword BEACH and your response to 22333
Wouldn’t this be better….?
Copyright © 2015 AppDynamics. All rights reserved. 34
What’s been done?
What logs are pertinent?
What’s our checklist?
What’ve we looked at so far?
Copyright © 2015 AppDynamics. All rights reserved. 35
Text the keyword BEACH and your response to 22333
Strength Training. Pack the Muscle. Hold Your Partner Accountable.
Trust but Verify. Measure, Monitor, and Test what you Manage.
End User Monitoring
• Auto-baselining for: – Requests per Minute – Errors per Minute – Average Response Time per minute
• Coupled with Health Rules for: – One standard deviation from the baseline – Replaces [Insert Your Synthetic Monitoring Here] with something that
has fewer false positives • Why use synthetics then?
– For sites with little traffic or very spiky traffic -- as a secondary complement to EUM, rather than the other way around
Copyright © 2015 AppDynamics. All rights reserved. 36
Text the keyword BEACH and your response to 22333
Good news about Synthetics
We’ve been using the beta, working closely with AppD, and it’s getting there. • The good:
– Seeing it alongside our existing metrics – Using it as complement to EUM on our newer sites
• What we’re excited about using: – Script based testing
• Vanilla Python environment – Automatic retest on failure
• No more guessing if it’s the node or the test
Copyright © 2015 AppDynamics. All rights reserved. 37
Text the keyword BEACH and your response to 22333
EUM vs. Synthetics
Copyright © 2015 AppDynamics. All rights reserved. 38
Test Locations Reflective of your actual users, and scales as your users scale
Test Timing Near-real-time based on actual users
Works with low traffic and 3rd party sites
You need enough consistent traffic to baseline; must be on your system
False Positives Less frequent with sites with enough traffic
Not necessarily reflective of where your end users are located
Scheduled intervals
Not traffic or hosting dependent
Dependent on whether individual nodes are working
EUM Synthetics
Complementary strengths and roles
Text the keyword BEACH and your response to 22333
Accountability of 3rd Parties
• Synthetic monitoring provides a consistent measure of the performance of all 3rd party services
Copyright © 2015 AppDynamics. All rights reserved. 39
Identify deviations from normal performance using consistent measurements
Obese Too many tools, platforms, and services. PCI Compliance, SoX, Security, International
Slow MTTR on $150,000/hour. Business growing and we couldn’t keep up
Weak 3rd Party Integrations were owning us. We were losing focus on the user.
Our Problem: No Foundation. No Anchor. No DevOps Culture
Text the keyword BEACH and your response to 22333
Copyright © 2015 AppDynamics. All rights reserved. 41
Text the keyword BEACH and your response to 22333
Our Solution: Core. Anchor. DevOps Culture.
Lean Out Focus on the Core User. Governance & DevOps Culture Result: Maximized our Investment.
Explosive Speed
Leverage Advanced AppDynamics. DevOps Solutions Must Empower The Delivery Team Result: MTTR down from hours to minutes
Strength Training
Trust but Verify. Measure, Monitor, and Test what you Manage. Result: 3rd Party Vendors/Partners Improved Stability
Shakeology
Copyright © 2015 AppDynamics. All rights reserved. 42
Text the keyword BEACH and your response to 22333
Governance & Culture
Dashboards Automated Remediation & Automatic App
Mapping
War Room End User Monitoring
Synthetics
Questions?
Text the keyword BEACH and your response to 22333
Answer a Question. Win a Program!
Copyright © 2015 AppDynamics. All rights reserved. 44
1. How many AppD features are you using? 2. How many systems are you monitoring? 3. What have you found works in your dashboard? What gets
your stakeholders engaged? 4. How are you using APM or DevOps to support a major
business goal that IT has to deliver? 5. What is your favorite AppD feature and why?
Text the keyword BEACH and your response to 22333
Appendix
Text the keyword BEACH and your response to 22333
The results: un-Photoshopped A war isn’t won in a single battle. • We reached Engineering and Product
– Drink our own champagne and start looking at our products more deeply – This is the dashboard we’ll see on slide 13:
• https://beachbody.saas.appdynamics.com/controller/#/location=CDASHBOARD_DETAIL&timeRange=last_12_hours.BEFORE_NOW.-1.-1.720&mode=MODE_DASHBOARD&dashboard=47
• We reached executives – Show the power of near-real-time analytics vs. 24hrs to upload analytics data – This the dashboard we’ll see on slide 14:
• [Insert link to executive dashboard] • We reached Ops
– Give our Ops team a chance to sleep at night – This is the workflow we’ll see on slide 14:
• [Insert link here to automated triage flow] Copyright © 2015 AppDynamics. All rights reserved. 46
Copyright © 2015 AppDynamics. All rights reserved. 47
Our footprint
Copyright © 2015 AppDynamics. All rights reserved. 48
Server side
Java
PHP
Node
MySQL (Percona)
Applications Databases
More than 1600 nodes, including 200+ databases
Client side
EUM
Synthetics
Oracle
Isolate test where errors were verified
Copyright © 2015 AppDynamics. All rights reserved. 49
Compare content across geography and time
Copyright © 2015 AppDynamics. All rights reserved. 50