AppSphere 15 - Mining the World’s Largest Healthcare Data Warehouse while Ensuring Great Consumer Experience

Mining the world’s largest healthcare data warehouse while ensuring great consumer experience Siva Balu, Managing Director, Blue Cross Blue Shield Association

PERFORMANCE What does it mean?

BACKGROUND Who are we?

BlueCross BlueShield •  Federation of 36 separate United States health insurance organizations and

companies

•  Providing health insurance to 105+ million Americans

•  Trusted for 80+ years

•  Presence in all 50 states

•  Accepted by over 90% of doctors and specialists

Copyright © 2015 AppDynamics. All rights reserved. 5

By the numbers

SHIFT IN STRATEGY Data as a Strategic Asset

Data as A Strategic Asset

Data as A Strategic Asset •  Board aligned that the Blue System should have a broader, aggregated data

asset as a foundation to drive value and enable future competitiveness

•  The System has many of the key data assets to enable immediate Plan business needs and Plans prioritized potential uses of collected data

•  Focus on Consumer


WEB SERVICES Data as a Strategic Asset

What is Web Service Platform?

•  Web-Services Platform is an enabler to provide Access to the said Data in a real-time, secure, responsive and reliable manner

•  Different software systems often need to exchange Data with each other, and a Web-Service is a method of communication that allows two disparate software systems to exchange this Data over the network


Why is there a need for Web Services?

•  Once data is collected, cleansed and organized…

•  … a reliable and secure method is necessary to access this data

•  … and mask all the data storage and retrieval details

•  … and provide fast, real-time access to meaningful and actionable information

•  A Web-Service (using Application Programming Interface – API) is an industry standard to accomplish this


What types of Data is available via Web Services? •  Consumer Cost •  Patient Review of Providers •  Provider Data •  Provider Faceted Search •  Member out of Pocket •  Eligibility •  Claims Status •  Subscriber information •  And more …


ACHITECTURE Web services

Web Services Architecture


Web Service Request

Private Internet

Load Balancer (External)

Gateway

Web Server

AP

I Key

M

anag

emen

t

Database Layer

Load Balancer (Internal)

Applica<on Layer

Web Service Response End User

Web Services Technology Stack


Web Service Request

BluesNet Internet

Load Balancer (F5)

Intel Enterprise Services Gateway (ESG)

HTTPS Layer -‐ IBM HTTP Server (IHS)

Mas

hery

AP

I

Database Layer (Oracle & SOLR)

Load Balancer (F5)

Applica<on Layer (WebSphere)


How do users get what they want?

17

Internet

Loca l BCBS

Internet / BluesNet

Core Da ta Warehouse

BCBSA

A Brief History of Web Service Volume…

~50k ~900k ~7.5M ~18M

~80M

~250M

0 50000000

100000000 150000000 200000000 250000000 300000000

2010 2011 2012 2013 2014 2015

Volume


So, what’s the problem?

Capacity: Unexpected exponential volume increase Performance: System Performance issues Response Time: Need for sub-second response for 100% of requests Increased Research Time: Excess time taken to research root cause Customer Satisfaction: User experience impacts Copyright © 2015 AppDynamics. All rights reserved. 19

Where does come in?


Existing monitoring tools had limitations Need for a APM tool that will quickly point us to the problem Fairly new to using AppDynamics Still in the infancy stages of adoption Started with web services, expanding to Tier-1 Business Applications Working on a long-term strategy to integrate AppDynamics into SDLC and Operations

Web Services - AppDynamics


Web Service Request

BluesNet Internet

Load Balancer (F5)

Intel Enterprise Services Gateway (ESG)

HTTPS Layer -‐ IBM HTTP Server (IHS)

Mas

hery

AP

I

Database Layer (Oracle & SOLR)

Load Balancer (F5)

Applica<on Layer (WebSphere)


What we learned… couple of examples:

Application: Code inefficiencies

Example: Zip code search Database: Query

Example: Queries mining large data volumes for small inquiries


What we did…

Added Intelligence into the code Changed the way code returned results to users Significant query tuning Expanded the environment to highly clustered Separated web-service calls into smaller service methods Moved from Core Warehouse to DataMart model Copyright © 2015 AppDynamics. All rights reserved. 23

Before

24

Before

25

After

26

Web Services – Single Function DataMarts


BUSINESS APPLICATIONS Expanding the usage of AppDynamics

Business Application - Rapidly growing volume


0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

3,000,000

3,500,000

4,000,000 N

ov-0

9

Jan-

10

Mar

-10

May

-10

Jul-1

0

Sep

-10

Nov

-10

Jan-

11

Mar

-11

May

-11

Jul-1

1

Sep

-11

Nov

-11

Jan-

12

Mar

-12

May

-12

Jul-1

2

Sep

-12

Nov

-12

Jan-

13

Mar

-13

May

-13

Jul-1

3

Sep

-13

Nov

-13

Jan-

14

Mar

-14

May

-14

Jul-1

4

Sep

-14

Nov

-14

Jan-

15

Mar

-15

May

-15

Jul-1

5

Sep

-15

6000% claim volume growth

Tier-1 Business Application - Highly clustered


What we did so far…


Dec Nov Oct Sep Aug Jul Jun May Apr Mar Feb Jan

2015

Proof of Concept

Performance Engineering

Production Implementation

Controller Upgrade

Production Implementation

Beyond Web Services

Additional Apps

Proof of Concept (POC)

New Web Services

New Web Services

Windows to Linux

All Web Services

Applications

More Apps

Takeaways… Define Scope: Identify what you want to monitor and how to use the product POC: Do a Proof-of-Concept Decide the implementation model: On-premise vs. Cloud Strategy: Define and Execute a long-term Strategy for APM People: Educate your team, peers and customers Process: Devise processes for feedback to Dev, provisioning mechanism, etc. Technology: Look at other AppDynamics offerings and extensions Copyright © 2015 AppDynamics. All rights reserved. 32

Thank You

AppSphere 15 - Mining the World’s Largest Healthcare Data Warehouse while Ensuring Great Consumer Experience

Documents