Case 2-3 Ford Motor Company

Case: CareGroupCase: CareGroup

Steven AdamsTravis Donselman

Ruwani Palihapitiya

9 November 2009

Case: GroupCare/Adams et al. 1

2

Waltham

Deaconess Hospital Needham

Case: GroupCare/Adams et al.

SWOT AnalysisSWOT Analysis

Case: GroupCare/Adams et al. 3

Strengths Weaknesses

• Halamka’s leadership • Y2K Disaster Recovery Plan• Info. Week’s #1 IT organization• 90% cut in capital expenditures• 2nd largest healthcare system in region

• Outdated network components• Weak financial statements• Inadequate security controls• Inadequate change controls• Lack of networking knowledge• Outdated Disaster Recovery Plan

Opportunities Threats

• Partnership with Cisco• Recovering economy (Q4, 2002)• Vendor to Waltham and Glover• Synergies from merger• Further cost cuts due to leveraging IT• Many hospitals closing, leading to less market saturations

• Volatile time for industry• Stagnant industry growth • Mass. Gen.-Brigham and Women’s merger

Symptoms and ProblemsSymptoms and Problems

Case: GroupCare/Adams et al. 4

Symptoms Problems Evaluation Criteria

Consolidation in industry Mass. Gen.-B&W Hosp.Price Pressure HMOs want cost cuts Hospitals losing money MAH, BIDH, NEBH, WH

Excess Supply in health care industry < 60% bed occupancyFierce Competition Many hospitals closing

Efficiency increase of 20% Debt-equity ratio under 1 Cost cuts limited to 5% All CG hospitals earn profit for Q4, 2003 10% ROE for FY 2005

Uncertainty about future New CEO, CFO, CIOLack of centralized IT system Legacy IT systems

New business strategy Merger of different hospitals MAH, BIDH, NEBH, DNH, DGH, WH

95% of personnel know CG mission and departmental goals and objectives All hospitals on Meditech software by end of 2001

Collapse of IT network Out of spec network Layer 2 based Lack of IT knowledge No network specialistLack of Controls No record of changes Napster-like attack

“Five nines” uptime for a period of two years No unscheduled downtime for two years Network is “in-spec” at all times for two years Meet Gartner benchmark

Theory and ConceptsTheory and Concepts

Case: GroupCare/Adams et al. 5

HIGH PATH TO BUSINESS TRANSFORMATION

IT Impact on Core Capabilities

Business Process Design/Reengineering Centralized IT System Improved efficiency

Business Transformation Merger of five hospitals Greater contracting power against HMOs

Incremental Improvement Use of synergies: strong image (Beth Israel), lower debt costs

Emerging Opportunities Vendor to divested Waltham and Nashoba branches Reorganization of Beth Israel and Deaconess-Glover

LOW IT IMPACT ON CORE STRATEGY HIGH

Timeline of Network Timeline of Network CollapseCollapse

Case: GroupCare/Adams et al. 6

Napster-like internal attack

Wed., afternoon

Service restoredThurs., 4:00 am

Intermittent failures

Thursday

CAP declared Thurs., 4:00pm

Network blocked; backup procedures

implemented Fri.,

morning

Redundant core builtSaturday

Network fully

operational Sunday,

night

The network had grown haphazardly due to the Caregroup merger, the standardization of Meditech in all hospitals, and the lack of change controls and networking knowledge. Significantly, legacy applications required a flat topology.

Additionally, the use of aged outdated networks components contributed to the failure and prolonged the recovery process. This is a result of not meeting the Gartner benchmark for IT spending as a percentage of revenue (1.9% vs. 2.7% target).

Reboot of core and Layer 3

Friday

Case: GroupCare/Adams et al. 7

RenaissanceParkswitch-rca switch-rcb

switch-rcc

5500 5500

5500

EastCampus

switch-ccell118

switch-rob05

switch-ly030

SiSi

switch-br203

5500

5509

5500

5500ATM7/1

FEC 9/1-2

FEC10/5-6

WestCampus

switch-spg06b

switch-spg06a

switch-ccw00m4

5500

5500

5500

SiSi

ATM 5/1

SiSi

FEC 8/1-3

FEC 10/1-4FEC 8/3-4

FEC 9/1-4

FEC 8/1-2

FEC 8/1-2

SiSi

FEC 10/1-2

SiSi

FEC 11/5-6

FEC 8/1-2

FEC 9/1-2

FEC 10/5-6FEC10/1-4

FEC 11/1-2

FEC 11/3-4

ATM10/1ATM10/1

FEC 6/23-24 FEC 6/23-24

FEC 3/21-24FEC 4/21-24

FEC 3/21-24FEC 4/21-24

14

8 12

12

12

12

1212

8 12

14

14 14

12

8 8

8

(8) 5505 Switches

(1) 5505 Switch

(15) 5505 Switches(1) 5500 Switch

(37) 5505 Switches

(1) 5500 Switch(21) 5505 Switches(2) 5509 Switches

(1) 3500 XL Switch (PACS)

(18) 5505 Switches

(21) 5505 Switches(1) 3500XL

(4) Ren Ctr (rc5, rc6, rc7, rcc)(1) Mount Auburn (Remote)

(1) 5500 Switch(1) 5505 Switch

(3) 5509 Switches

SiSi

(12) CC East Campus(4) HIM

(3) 109 Brookline Ave(2) 2127 Burlington(3) Research North

SiSi

(24) CC West Campus - - 1 is Dual Homed w/spg06b(1) PACS

(1) Research East

(1) 5500 Switch(1) 5505 Switch(1) 5509 Switch

(4) Dana(3) East(7) Feldberg(2) Finard(3) Kirstein(4) Reisman(5) Rose(1) Service(5) Stoneham(2) Yamens

(1) Baker(4) Deaconess(7) Lowry Medical(1) Maintenance(3) Palmer(1) CC West (Dual-homed w/ccw00m4)

(13) Farr(6) Kennedy(2) Lowry Medical(1) Masco

(3) Ren Ctr (rc7, rc8, rcc)

SiSi

ATM OC-3 (155Mbps) over Sonet

ATM OC-3 (155Mbps) dark fiber

Fast Etherchannel (400 Mbps)Fast Etherchannel (800 Mbps)

Not Active

SiSi

Technical ExplanationTechnical Explanation Layers 2 and 3 were out of specification with 802.1d

standards:

◦ The standard calls for a maximum of 7 hops.

◦ Multiple bridges were 10 hops from the bridge root

◦ Bridge loops present

◦ Insufficient use of routers

Case: GroupCare/Adams et al. 8

802.1d Standards CG Network

One VLAN per subnet One VLAN for multiple subnets

Limited or no bridging Extensive use of bridging

Switching limited to Layer 2 Switching used for Layers 2 and 3

Alternatives and Alternatives and EvaluationEvaluationAlternatives Pros ConsMitigation• Disaster Recovery Plan• Emergency Access System

• Limited access available during outages• Outages run more smoothly

• Moderate capital expenditures• Moderate time requirements• Limited risk involved

Avoidance• Contract with Cisco• Upgrade Components• Sec./Change Controls• SETA

• Minimizes risk of outages• Expertise and knowledge of Cisco• Changes are documented• Employees are less likely to harm the network

• High expenditures $300,000/yr. for Cisco• High level of network knowledge required• Significant time investment• Some uncertainty involved

Acceptance• Do nothing

• No project planning• No capital expenditures• No time required

• System subject to future failures• Threat to company’s reputation and profitability

Transference• Outsource Network Management

• Skilled administrators• Can focus on core competency (knowledge management)

• Need to trust another company• Very expensive• Loss of control over process Case: GroupCare/Adams et al. 9

ImplementationImplementation

Case: GroupCare/Adams et al. 10

RecommendationsRecommendationsUse multiple spanning tree protocol Implement monitoring and diagnostic tools to

maintain the health and hygiene of the network:◦ NATkit◦ CiscoWorks◦ HP Network Node Manager

Implement exclusively routers in Layer 3Replace all legacy applicationsReview and, if necessary, replace Meditech PAC

softwareUse dense multicast instead of broadcastHire network VP, network managers, network

administrators, and network technicians

Case: GroupCare/Adams et al. 11

Case: GroupCare/Adams et al. 12

SiSi

Renaissance Park

6500 6500

East Campus West Campus

SiSi

SiSi

SiSi

SiSiSiSi

SiSi

SiSiSiSi

Server Server Server

Data Center

ClosetSwitches

Typical Access LayerConnection to the Core

Typical Access LayerConnection to the Core

Remote LocationWAN Connectivity

Firewall Firewall

Internet

109Brookline

Shapiro ClinicalCtr

Rose

West ClinicalCenter

Palmer

SiSi

SiSiSiSi

Review of CaseReview of Case How did CareGroup respond to the disaster?

◦ Implemented backup procedures◦ Brought in Cisco’s Emergency Response Team◦ Reverted to paper system◦ Installed routers, restored critical systems, reviewed crisis

Your thoughts on how CareGrop handled the crisis.◦ Well organized, decisive action◦ Cisco Emergency Response Team was the right decision◦ Should NOT have changed system before ERT arrived

Your thoughts on network management in light of the case.◦ Vigilant system maintenance and upgrades are critical◦ Control access to the main system at all times◦ Make sure backup plans are in place and well-practiced

Case: GroupCare/Adams et al. 13