VMworld 2013: How SRP Delivers More Than Power to Their Customers

How SRP Delivers More Than Power to

Their Customers

Sheldon Brown, SRP

Girish Manmadkar, VMware

VAPP5105

#VAPP5105

2

Agenda

Introduction

Before We Start - Homework

Figuring It All Out

What It All Looks Like Now

After the Dust Settled

Summary

Questions

3

Introduction

4

Girish Manmadkar – Consulting Architect

With VMware since 2007 in various roles, focusing on providing

VMware solutions around Business Critical Application with major

focus around SAP

More than 22+ years experience starting with SAP R4.6

Regular speaker at VMworld since 2007

Recent VMware Partner Exchange, EMCWorld events

5

Sheldon Brown – Manager SAP Technology

Responsible for SAP Technology (BASIS,Development/Integration)

21 years with SRP

10 years with Server and Storage Team

2 years with SAP implementation team

9 of 10 years at VMWorld

[email protected]

6

Who We Are – Salt River Project Agricultural and Power District (SRP)

Power and Water supplier to the 1,000,000+ customers in the

Phoenix metro area

Third largest public power company in the country

100+ years old

5000+ employees

Slow moving, risk adverse

70% virtualized

http://www.srpnet.com

7

Home Work

VMware Best Practices

8

Best Practices

Use the latest processor generations, especially with EPT (Intel

XEON 55xx, 65xx, 75xx and newer) or RVI (AMD Opteron 83xx,

84xx, 61xx and newer) which bring performance benefits

Assigning vCPUs to virtual machines

• ESXi 5.1: shows scaling for virtual machines > 8 vCPUs

• Understand the v/NUMA architecture for larger VMs

You can overcommit CPUs, but reasonably

• Start with no CPU overcommitment. When the average utilization of SAP

instances and the average idle time on the host are known, accordingly raise

the number of vCPUs on the host (for example, adding virtual machines). Use

VMware vSphere Distributed Resource Scheduler (DRS) capabilities to control

and balance overall workload.

• If performance problems are seen, first reserve CPU to see if the performance

problems are a result of resource contention

9

Best Practices (cont.)

• Do not overcommit memory. This is not recommended at all with SAP.

Reserve 100% of the assigned memory.

• Installation of VMware Tools is mandatory to avoid time conflicts and have all

necessary drivers (for example, new network cards)

• Timekeeping: Use NTP (Network Time Protocol) on the ESX host and guest

as per Timekeeping best practices for Windows, including NTP and

SAP Note 989963

http://kb.vmware.com/kb/1318

• Use multiple virtual SCSI controllers for the database virtual machine and

virtual machines with high I/O load. The use of multiple virtual SCSI controllers

allows the execution of several parallel I/O operations inside the guest OS.

• SAP requires the implementation of the new virtualization aware monitoring.

See SAP Note 1409604.

• When using VMware snapshots, follow Best practices for virtual machine

snapshots in the VMware environment. Make sure that the virtual machine has

no active snapshot before reporting performance problems.

http://kb.vmware.com/kb/1025279

10

Best Practices for Linux

Disabling the Linux I/O scheduler

• Set kernel parameter "elevator=noop"

• Red Hat KB 5428 (applies to RHEL and SLES)

Choose the optimal SAP memory model

• MAP memory model (SAP Note 386605) for CPUs without EPT/RVI

• STD memory model (SAP Note 941735) for CPUs with EPT/RVI

11

vSphere – ESX CPU Scheduling

Determine size of hardware NUMA nodes on the server

Size the virtual machine as small as possible

• Too many vCPUs creates scheduling overhead

• Too much RAM results in accessing RAM from a far NUMA node

Enable simultaneous multithreading

• More hardware execution contexts are available

Generally CPU affinity not required

12

Best Practices for Databases

Oracle

• SAP Note 1173954 states support with minor restrictions

• Oracle on VMware vSphere Essential Database Deployment Tips http://www.vmware.com/resources/techresources/10101

• VMware Alliances statement http://www.vmware.com/solutions/partners/alliances/oracle-vmware-support.html

13

Best Practices for Databases – I/O Subsystem

Store data files and log files on separate physical devices and distribute

data files as per SAN Volume Controller as defined by the storage

vendor’s best practices

Quote from SAP on SQL Server best practices guide

• “The number of disks is determined by the throughput in I/O operations per

seconds (IOPS). Therefore discussions around storage investments or

configurations should be driven by IOPS”

• http://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/4ab89e84-0d01-0010-

cda2-82ddc3548c65

IOPS used by storage and OEM vendors to size SAP storage on

VMware – Similar to physical

The standard SAP rules for databases apply!

• SAP Note 806554 for general I/O intensive DB tasks

• SAP Note 592393 and 793113 for Oracle

15

Best Practices for vStorage APIs (cont.)

VMware vSphere® Storage APIs – Array Integration (VAAI)

• Has nothing to do with backup – it is a better way of integrating storage

capabilities into vSphere

• vStorage APIs for Array Integration FAQ

http://kb.vmware.com/kb/1021976

• Follows the vSphere lifecycle

• Offloads specific storage operations to compliant storage hardware

• Write Same/Zero

• Eliminating redundant and repetitive write commands

• Fast/Full Copy

• Leveraging array ability to mass copy, snapshot, and move blocks

• Hardware Offloaded Locking (ATS)

• Stop locking LUNs and start locking only blocks

16

Best Practices for Network

Standard vSphere networking guidelines apply

• Separate infrastructure traffic from virtual machine traffic: use physical NICs or VLANs

• Use NIC teaming for availability and load balancing (group physical NICs connected to

the same physical network)

• Take advantage of network I/O controls to converge network and storage I/O onto

10GbE

• Take advantage of the dvSwitch and/or Cisco Nexus 1000v to simplfy the architecture

and support

Use VMXNET3 adapter in guest OS

• Improved performance

• Helps with network traffic in three-tier setup between application and DB VM

• SAP Batch Job Performance on vSphere

http://blogs.vmware.com/performance/2010/02/sap-batch-job-performance-on-

vsphere.html

17

Best Practices – Further Information

SDN Forum SAP on VMware

• http://forums.sdn.sap.com/forum.jspa?forumID=471

• Most complete link collection around SAP on VMware

SAP Notes

• Following the SAP Notes is mandatory!

VMworld Sessions

• Various documents – from high level overview to technical deep dive

Performance Tests

• Continuous performance measurement for databases and SAP software

18

Figuring It All Out

19

Figuring It All Out – SAP Architecture and Components

20

Figuring It All Out – the Team

Team Effort

Consultants

• Accenture

• SAP

• HP

• VMware

• NetApp

• Cisco

SRP Employees

• Server

• Storage

• Network

• BASIS

• Strategy and Planning

• Executive Management

21

Figuring It All Out – What we needed to decide

Datacenters

Host Servers

Virtual Servers

Storage

Network

Database

Operating System

Infrastructure Security

22

Figuring It All Out – SAP Environment Description

ECC, SRM, PI, BW, SolMan, BOBJ, GRC, Portal, Data Services

Bolt Ons

• Vertex

• OpenText

• BPC

Landscapes (Production, Quality, Test, Development, Sandbox, Training)

• Production

• Quality

• Test

• Development

• Sandbox (Technical and Functional)

• Training

5000+ users

850+ unique per day

23

Figuring It All Out – Quick Sizer Results (nothing “quick about it”)

User Based SAPs

Memory and Disk Sizing

24

What It Looks Like Now

25

What It All Looks Like Now – Datacenters and Network

Two datacenters

• 30 miles by the wire apart

• Connected by 20Gbps (Layer 2) 16 Gbps (Layer 3)

Network

• Cisco 7k series core

• Cisco 5k series edge

• 10Gbps network

26

What It All Looks Like Now – Physical Servers

24 ESX Hosts (2 clusters of 12 at each data center)

• HP DL580 G7

• vSphere 5.1 U1

• 4 proc 10 core

• 512 GB RAM

• 2 10GBps ports

6 ESX Host Stretch Cluster (3 at each data center)

• HP DL580 G7

• vSphere 5.1 U1

• 4 proc 10 core

• 256GB RAM

• 2 10GBps ports

27

What It All Looks Like Now – Virtual Servers

Production – 76 servers

Quality – 59 servers

Test – 27 servers

Development – 32 servers

Sandbox (Technical and Functional) – 40 servers

Training – 3 servers

Administration – 3 servers

Total – 240

Operating Systems – Red Hat Enterprise Linux (RHEL) 5.x to 6.2

and Windows 2008 R2

28

What It All Looks Like Now – Storage

4 NetApp 6280’s (2 at each data center)

• NetApp Data ONTAP Release 8.1.2P3 7-Mode

• Theoretical capacity 77,000 IOPS (440 disk * 175 IOPS)

• Highest recorded capacity 8212 IOPS

• Raw Capacity:154,493GB or 150TB

• Consumed Capacity: 61519GB or 60TB

• Extensive use of Snap Shot Manager for SAP

29

What It All Looks Like Now – Network Architecture

Datacenter 2 Datacenter 1

30

What It All Looks Like Now – SAP vSphere Host Configuration

Datacenter 1 Datacenter 2

31

What It All Looks Like Now – Storage Architecture

32

What It All Looks Like Now – Monitoring

SAP Solution Manager – Application

Quest vFoglight – Virtual Server Performance

Top, Satellite – Operating System

CA Wily Interscope – Java

NetApp ONCommand Unified Manager – Storage

HP SIM – Server Hardware

Q1, Cascade Shark, eHealth – Network

CA Spectrum – Operations

33

What It All Looks Like Now – Database

Oracle 11g (Corporate Standard)

DataGuard for replication

One virtual server per database

Unlimited Oracle for SAP

34

What It All Looks Like Now – Disaster Recovery/High Availability

Completed a Business Impact Analysis

12 hour RTO

Close to zero RPO

Load-balanced dialog instances

Central Messaging instance at primary datacenter

Database instance at primary datacenter

Database replication happening real time

35

Datacenter 1

What It All Looks Like Now – Disaster Recovery/High Availability

Replication

Log Shipping

Datacenter 1 Datacenter 2

36

What It All Looks Like Now – Why We Virtualized SAP on VMware

Cost Effective

Flexibility

Higher Availability

Faster Deployment

Easier to recover in disaster

Higher Server Density = Lower Data Center Costs

Virtualization is the SRP corporate standard

VMware is the leader

Favorable license agreement with SAP

37

What It All Looks Like Now – Refresh Methodology

Prepare the source and target systems

Snap Clone production using Snapshot Manager for SAP

• Mount cloned copies to targeted servers

• Changes the SID

• Minutes instead of hours

Post clone cleanup

Validate target systems

Scrub production data

Enjoy

38

After the Dust Settled

39

After the Dust Settled - Lessons Learned

Overbuild

Plan for disaster now

Plan out storage configuration early

Pay attention to security early

• Infrastructure

• OS and Application hardening

40

After the Dust Settled - Successes

Flexibility

Speed in reaction

Environment Refresh rate

Spin up environments quickly

Saved $1,000,000 in hardware by virtualizing Oracle

41

After the Dust Settled - Things to Come

Release 2

• Work Management?

• HR

• More Supply Chain

• More Finance

• Customer Information?

Mobility?

HANA?

42

Summary

Now that we are done – don’t forget your home work

Figure it all out

What It All Looks Like Now

After the Dust Settled

43

Questions

44

Other VMware Activities Related to This Session

HOL:

HOL-SDC-1301

Applied Cloud Operations

VAPP5105

THANK YOU

How SRP Delivers More Than Power to

Their Customers

Sheldon Brown, SRP

Girish Manmadkar, VMware

VAPP5105

#VAPP5105