VMware Validated Design for Software-Defined Data Center ......management cluster to provide east/west routing across all regions. Using the UDLR reduces the hop count between nodes

Mike BrownSenior SDDC Integration Architect, VMware, Inc.VCDX4/5/6-DCV, VCIX6-NV

@vMikeBrown

PBO1721BE

#VMworld #PBO1721BE

VMware Validated Design for Software-Defined Data Center:Technical Deep Dive

VMworld 2017 Content: Not fo

r publication or distri

bution

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

2#PBO1721BE CONFIDENTIAL



bution

Building the Cloud is Often

the Bottleneck

3



bution

VMware Validated DesignsPrescriptive Blueprints with Comprehensive Deployment and Operational Practices

✓Broad Use Cases

ComprehensiveDocumentation

Proven & RobustStandardized

Designs

4



bution

VMware Validated Designs

5

A History Lesson

1.0

2.0

3.0

3.0.2

4.0

4.1

February 2016▪ 12mo of Engineering▪ Release to PSO

and Partners

September 2016

▪ 2mo of Engineering

▪ Dual Region with DR

▪ Two Pod Architecture

July 2016

▪ 3.5mo of Engineering

▪ Smaller scope.(i.e. Dual Region + DR)

November 2016

▪ 1.5mo of Engineering

▪ Added M-Seg Use Case

▪ Added IT Automating IT Guide

March 2017

▪ 1.5mo of Engineering.

▪ Major Product Updates

▪ Added ROBO

August 2017

▪ 4mo of Engineering• Minor Product Updates• Consolidated Pod

Option

#PBO1721BE CONFIDENTIAL



bution

Design Objectives

Overall Objective SDDC capable of automated provisioning of workloads

Type of Deployment Greenfield and Brownfield

Cloud Type Private Cloud

Regions and Disaster Recovery Dual-region SDDC that Supports Disaster Recovery

▪ Guidance for an SDDC whose management components are designed to operate in the event of

planned migration or disaster recovery.

▪ Guidance for an SDDC that supports two regions for both management and tenant workloads.

▪ Operations guidance for disaster recovery and planned migration

Pods Two Pod

▪ Management Pod – Runs the management stack.

▪ Shared Edge and Compute Pod – Runs tenant workloads, and services for north-south plus east-

west routing.

One Pod

▪ Consolidated Pod – Runs the management stack, tenant workloads, and services for north-south

plus east-west routing.

Max Number of VM ▪ 10,000 Running VMs (Two Pod) / 1,500 (One Pod)

▪ 150 VM deployments/hour (Two Pod) / 60 (One Pod)

Design ObjectivesVMware Validated Design for SDDC

Design Objectives

Overall Availability 99%

= 3.65 days downtime/year

= 1.7 hours downtime/week

Planned downtime expected for upgrades, patching, on-going maintenance.

Authentication, Authorization, and

Access Control

▪ Use of Microsoft Active Directory as a central user repository

▪ Use of service accounts with minimum required authentication and Access Control List

configuration.

▪ Use of basic tenant accounts.

Certificate Signing Certificates are signed by an external certificate authority (CA) that consists of a root and intermediate

authority layers

Hardening Tenant workload traffic can be separated from the management traffic.

The design uses a distributed firewall to protect all management applications. To secure the SDDC, only

other management solutions and approved administration IP addresses can directly communicate with

individual components.

#PBO1721BE CONFIDENTIAL 6



bution

Design Decisions

7

290+ in VMware Validated Design for SDDC

Reduces risk by providinga baseline of standardization.

Ensures the design meets the design objectives.

Reinforces standardization with justification and

implications.Easy to follow checklist form.




bution

Example Design DecisionsVMware Validated Design for SDDC

NSX Design > Routing Design > Routing Model Design Decisions (4.1)

Decision ID Design Decision Design Justification Design Implication

SDDC-VI-SDN-017 Deploy NSX Edge Services Gateways

in an ECMP configuration for

north/south routing in both

management and shared edge and

compute clusters.

The NSX ESG is the recommended

device for managing north/south

traffic. Using ECMP provides multiple

paths in and out of the SDDC. This

results in faster failover times than

deploying Edge service gateways in

HA mode.

ECMP requires 2 VLANS for uplinks

which adds an additional VLAN over

traditional HA ESG configurations.

SDDC-VI-SDN-018 Deploy a single NSX UDLR for the

management cluster to provide

east/west routing across all regions.

Using the UDLR reduces the hop

count between nodes attached to it to

1. This reduces latency and improves

performance.

UDLRs are limited to 1,000 logical

interfaces. When that limit is reached,

a new UDLR must be deployed.

SDDC-VI-SDN-019 Deploy a single NSX UDLR for the

shared edge and compute, and

compute clusters to provide east/west

routing across all regions for

workloads that require mobility across

regions.

Using the UDLR reduces the hop

count between nodes attached to it to

1. This reduces latency and improves

performance.

UDLRs are limited to 1,000 logical

interfaces. When that limit is reached

a new UDLR must be deployed.




bution

vRealize Business 7.3for Cloud

vRealize Automation 7.3vSphere 6.5 U1

vSAN 6.6.1

Site Recovery Manager 6.5.1 vRealize Log Insight 4.5and Content Packs

vRealize Operations 6.6.1and Management Packs

NSX 6.3.3

Bill of MaterialsVMware Validated Design for SDDC 4.1

For a complete list refer to the release notes.




bution

Environmental and External Systems Requirements

10

VMware Validated Design for SDDC

Active Directory

Certificate Authority

DNS and NTP

SMTP Relay

SFTP

Rack Space

Power

Cooling




bution

Dual-Region Deployment ReadyVMware Validated Design for SDDC

Characteristics & Restrictions

▪ Regional Distance is Rather Large

▪ A Region May Be Treated as an SDDC

▪ Multiple Regions are Not Treated as a Single SDDC

Workload Placement Closer to Customer

▪ Northern California and Southern California

▪ US East Coast and US West Coast

▪ US Region and EU Region

Common Uses

▪ Disaster Recovery: One region can be the primary site

and another region can be the recovery site.

▪ Data Privacy: Address laws & restrictions in some

countries by keeping tenant data within a region in the

same country.

San Francisco, CAPrimary Region

Los Angeles, CASecondary Region




bution

Availability ZonesVMware Validated Design for SDDC

AVAILABILIITY ZONE

AVAILABILIITY ZONE Characteristics

• “Islands” of infrastructure for physical isolation or

building-level redundancy and high-availability.

• Positioned within “metro” distance to allow

synchronous storage replication. (~50km/30mi with low single-digit latency and large bandwidth)

• Allows the SDDC equipment across the availability

zone to operate in an active/active manner as a single

virtual data center or region.

• Isolated enough from each other to stop the

propagation of failure or outage across their

boundaries.

Early Access Preview

• Guidance for vSAN Stretched Clusters

within a region.




bution

13

StandardizedElevation

NetworkTransport

Out-of-BandManagement

FunctionalRoles

PodsVMware Validated Design for SDDC




bution

Two Pod – Distributed Management and WorkloadVMware Validated Design for SDDC




bution

One Pod – Consolidated Management and WorkloadVMware Validated Design for SDDC

New in Version 4.1

▪ Consolidates Management, Edge, and Workload into a single pod.

▪ Requires only a minimum of 4 ESXi hosts

▪ All functional testing and validation of the design is done using vSAN.

▪ Any supported storage may be used. Adjust the operations guidance.

▪ Network Transport

▪ Supports both L2 and L3 transport services.

▪ Scalable and vendor-neutral network, use an L3 transport.

▪ Ready for Scale

▪ Expandable to a 32 ESXi host pod.

▪ SDDC solutions easily scale – deployed w/ native or NSX load balancing in place.

▪ Transitions to Two-Pod Distributed Management and Workload (Standard)

▪ Downtime Required

▪ Single Region and Single Availability Zone

▪ License Flexibility for NSX (No Universal Objects)

External

Connection

WAN/LAN




bution

High-Level Deployment Objectives

16


Two-Pod / Standard Architecture One-Pod / Consolidated Architecture

Minimum Hosts 8 4

Management VMs420 GB vRAM,

2TB VSAN, 6 TB NFS50% - 70% less

Recoverability Dual Region Single Region (DR to cloud)

Scale (VMs) Up to 10,000 Up to 1,500

Churn Medium (up to 150/hr) Low (up to 50/hr)

Availability 99% 95%

Modularity Foundation Cloud Operations Cloud Management Foundation Cloud Operations Cloud Management

Expansion options Additional Compute Pods (Up to 32 Hosts Each) Expand Pod to 32 Hosts or Grow to 2-Pod




bution

Racks

17


42

41

40

39

38

37

36

35

34

33

32

31

30

29

28

27

26

25

24

23

22

21

20

19

18

17

16

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

42

41

40

39

38

37

36

35

34

33

32

31

30

29

28

27

26

25

24

23

22

21

20

19

18

17

16

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

C24 M3UCS

241 8 16

C24 M3UCS

241 8 16

C24 M3UCS

241 8 16

C24 M3UCS

241 8 16

C24 M3UCS

241 8 16

C24 M3UCS

241 8 16

C24 M3UCS

241 8 16

17 18 19 20 21 22 23 24 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 4825 26 27 28 29 30 31 32

CISCO NEXUS 2248PQSTAT

1 2 3 4

ID

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

4321

4321

4321

4321

17 18 19 20 21 22 23 24 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 4825 26 27 28 29 30 31 32


1 2 3 4

ID

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

4321

4321

4321

4321

17 18 19 20 21 22 23 24 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 4825 26 27 28 29 30 31 32


1 2 3 4

ID

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

4321

4321

4321

4321

C24 M3UCS

241 8 16

C24 M3UCS

241 8 16

C24 M3UCS

241 8 16

C24 M3UCS

241 8 16

C24 M3UCS

241 8 16

C24 M3UCS

241 8 16

C24 M3UCS

241 8 16

C24 M3UCS

241 8 16

C24 M3UCS

241 8 16

C24 M3UCS

241 8 16

C24 M3UCS

241 8 16

C24 M3UCS

241 8 16

Top-of-Rack Switches for Server Connectivity

Uplinks to Spine or Corefor Inter-Pod Connectivity

ServersTwo 40GbE Uplinks

to Spine or CoreOne 48 x 1 GbE

Management SwitchTwo 48 x 10 GbE

Top-of-Rack Switches

Compatible or Certified

ConfigurationsManagement Switch for

Out-of-Band Connectivity

Two PowerFeeds

Redundant Power




bution

Servers

18

Management Pod – Example

1 2 3 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

SD

▪ SD/USB or SATADOM Recommended

▪ Remote Syslog

Boot

▪ On Compatibility Guide

▪ RackmountSingle or Multi-Node

▪ Blade

Form Factor

▪ 2 x Sockets min.

▪ Intel XD or AMD NX Set

▪ High-Performance in BIOS

▪ 192GB RAM min.

Compute

▪ 2 x 10 GbE to

Top-of-Rack Leaf Switches

▪ Jumbo Frames

▪ 1x 1 GbE BMC to

Out-of-Band Switch

Network

Host

▪ vSAN Hybrid or All-Flash1 x Disk Groups min

▪ Flash Device for Cache Tier1 x 200GB Flash Device min.

▪ SAS for Capacity Tier2 x 1TB SAS min.

▪ Or Any Supported Storage

Storage

SATA DOM CPU MEMORY NIC IPMIAny SupportedvSAN




bution

Servers

19

Shared Edge/Compute and Compute Only Pod(s) – Example

SD

▪ SD/USB or SATADOM Recommended

▪ Remote Syslog

Boot

▪ 2 x Sockets min.

▪ Intel XD or AMD NX Set

▪ High-Performance in BIOS

▪ 128GB RAM min.

Compute

Host

▪ vSAN Hybrid or All-Flashn Disk Groups

▪ SSD for Caching Tier

▪ Flash Device for

Capacity Tier

▪ Or Any Supported Storage

Storage

SATA DOM CPU MEMORY NIC IPMI

▪ On Compatibility Guide

▪ RackmountSingle or Multi-Node

▪ Blade

Form Factor

Any SupportedvSAN

▪ 2 x 10 GbE to

Top-of-Rack Leaf Switches

▪ Jumbo Frames

▪ 1x 1 GbE IPMI to

Out-of-Band Switch

Network

1 2 3 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Using vSAN? Visit vsanreadynode.vmware.com #PBO1721BE CONFIDENTIAL



bution

Network Transport Services

▪ The VMware Validated Designs supports both L2 and L3 transport services.

▪ For a scalable and vendor-neutral data center network, use an L3 transport.

▪ When deciding to use L2 or L3, consider the following:

▪ The NSX ECMP Edge devices establish L3 routing adjacency with the first upstream L3 device to provide equal cost routing for management and workload virtual machine traffic.

▪ The investment you have today in your current physical network infrastructure.

▪ All design documentation is provided for an L3 transport. You must appropriately adjustthe design deployment and day-two operations guidance under the context of an L2 transport.

Minimal Design Requirements

▪ One 10 GbE port on each ToR for host uplinks.

▪ Host uplinks are not configured in an ether-channel (LAG/vPC)

▪ Layer 3 device that supports BGP

▪ IGMP support required by vSAN and NSX Hybrid Mode

20





bution

Example: L3 Leaf-and-Spine Topology

21





bution

Example: L3 Leaf-and-Spine Topology

22


Design for Workload Requirements

Compute Only Pods and Clusters May Span Racks

Management and Shared Edge/Compute Pods and ClustersMay Span Racks if L2 Network Transport is Used. Peer with Upstream L3.

Homogenous Nodes within the Pod

May Be Heterogeneous Pod to Pod




bution

vSphere Clusters

23


ESXi ESXi ESXi ESXi

Management Distributed Switch

Universal Management Transport Zone in Hybrid Mode

Any Supported Storage

plus NFS

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

ESXi ESXi ESXi ESXi

Compute Distributed Switch



ESXi ESXi ESXi ESXi

Compute n Distributed Switch



Universal Compute Transport Zone in Hybrid Mode

plus NFS

Management ClusterMinimum 4 Nodes | vSphere HA and DRS Enabled

vSAN Ready Nodes Recommended

Shared Edge and Compute ClusterBusiness Workload Requirements | Minimum 4 Nodes

vSphere HA and DRS Enabled | Edge Resource Pool and Anti-Affinity Rules

Compute Cluster nBusiness Workload Requirements | Minimum 4 Nodes

vSAN Ready Nodes Recommended

Management StackManaged by Management Stack vCenter Server

Compute StackManaged by Compute Stack vCenter Server

Management Pod Shared Edge and Compute Pod Compute Pod n

plus NFS




bution

Distributed Switches

24





bution

Storage

25


Primary Storage

VMware vSAN is Recommended

for Management Pod


for Shared Edge and Compute Pod


for Compute Only Pod(s)

Secondary Storage

NFS for Backups

NFS for Log Archives

NFS for Content Library and Templates




bution

vSAN Optional for Management Pod Primary Storage

▪ Previous releases required vSAN as primary storage in the management pod. This requirement has been relaxed inthe 4.1 release.

▪ All functional testing and validation of the design is done using vSAN.

▪ Although the VMware Validated Designs highly recommend the use of vSAN, in particular for the management pods, any supported storage solution may be used.

▪ If a storage solution other than vSAN is selected:

▪ You must appropriately adjust the design deployment and day-two operations guidance under the context of vSAN.

▪ The storage design must match or exceed the capacity and performance capabilities of the vSAN configuration in the design.


0%

50%

100%

25%

75%

0%

50%

100%

25%

75%

vSAN

Datastore

Non-vSAN

Datastore




bution

NFS Secondary Storage

27


Volume 1 Volume 2

NFS Storage Array

Region A

Export for Backups

Export for

Content Library

and Templates

Export for

Log Archives

Volume 1 Volume 2

NFS Storage Array

Region B

Export for Backups

Export for

Content Library

and Templates

Export for

Log Archives




bution

vCenter Server and Platform Services

28





bution

Two-Layer Certificate Authority + Certificate Replacement

CertGenVVD tool saves you time when creating signed certificates. See VMware Knowledge Base article 2146215.

Certificate Mode

▪ VMCA Hybrid Mode

▪ All user-facing certificates are signed by a certificate authority (CA).

▪ All virtual infrastructure management components use TLS/SSL certificates that are signed by the VMware Certificate Authority (VMCA).

▪ Supports a Two-Layer CA environment.

Certificate Replacement

▪ If the CA-signed certificates expire after you deploy the SDDC, you must replace them individually on each affected component.

▪ Provides guidance for replacing all CA-signed certificates that are expiring. *





bution

NSX

30





bution

Distributed Logical Networking

31


Universal Distributed Logical Router

UDLRCTRL

ECMPESG

ECMPESG

ECMPESG

ECMPESG

LEAF LEAF LEAF LEAF

BGP BGP BGP BGP

Universal Transit Logical Switch

VC PSC NSXM NSXC VC PSC NSXM

SPINESPINERegion A Region B

Virtual Networks




bution

32



bution

Cloud Operations

33

VMware Validated Design for SDDC // vRealize Operations




bution

Cloud Operations

34

VMware Validated Design for SDDC // vRealize Log Insight




bution

Management Packs and Content PacksVMware Validated Design for SDDC

Now Included by default with product deployment.

▪ Management Pack for vCenter Server

▪ Management Pack for vRealize Log Insight

▪ Management Pack for vSAN (New in v6.6)

▪ Management Pack for vRealize Automation (New in v6.6)

▪ Management Pack for vRealize Business for Cloud (New in v6.6)

Installed post-deployment.

▪ Management Pack for NSX for vSphere

▪ Management Pack for Storage Devices

Now Included by default with product deployment.

▪ General

▪ Content Pack for vSphere

▪ Content Pack for vSAN (New in v4.5)

▪ Content Pack for vRealize Operations

Installed post-deployment.

▪ Content Pack for NSX for vSphere

▪ Content Pack for vRealize Automation 7

▪ Content Pack for vRealize Orchestrator 7.0.1+

▪ Content Pack for Linux (Added to the Architecture)

▪ Content Pack for Microsoft SQL Server

vRealize Operations 6.6.1Management Packs

vRealize Log Insight 4.5Content Packs



bution

Region A

vRealize Automation vRealize Business

BUC

BUS

vRO

vRA IWS IMS DEM IAS SQL

vRA IWS IMS DEM IAS

Region B

IAS

Cloud Management Platform Components

36


BUS vRealize Business Appliance

BUC vRealize Business Data Collector

SQL Microsoft SQL Server Database

VRA vRealize Automation Appliance

IWS vRealize Automation IaaS Web Server

IMS vRealize Automation IaaS Manager Service

IAS vRealize Automation IaaS vSphere Proxy Agent

DEM vRealize Automation Distributed Execution Manager

BUC

IAS

vRealize Automation

and Business




bution

Distributed Deployment with NSX

37

VMware Validated Design for SDDCVMware Validated Design for SDDC // Cloud Operations




bution

Disaster Recovery of SDDC Solutions

38


Region A Non-Replicated

vRealize Log Insight

Region A Infrastructure Management

vSphere

NSX

Site Recovery Manager

vSphere Data Protection or VADP-Based Solution

Region A Replicated

vRealize Operations

vRealize Automation

vRealize Business

SRM(using vSphere Replication)

Region B Infrastructure Management

vSphere

NSX

Site Recovery Manager

vSphere Data Protection or VADP-Based Solution

Region B Non-Replicated

vRealize Log Insight

Region B Replicated

vRealize Operations

vRealize Automation

vRealize Business

SRM (using vSphere Replication)




bution

DocumentationVMware Validated Design for SDDC

≠

Release Notes

Architecture Details

Architecture Diagrams

Planning and Preparation Guides

Step-by-Step Deployment Guides

Operations Guides

▪ Monitoring and Alerting

▪ Business Continuity

▪ Disaster Recovery

▪ Plus more Operations Add-ons

Download the Docs Todayvmware.com/go/vvd-docs

VMware Validated Design 4.1 Documentation Center




bution

Sizing Tool on VMware Code

Provide qualified partners a sizing tool for deployments adhering to the VMware Validated Design for SDDC.

Online web based tool which integrates with VMware Compatibility Guide to pull certified vSAN ReadyNodedetails.

vmware.com/go/vvd-sizing





bution

41

Download the Poster Todayvmware.com/go/vvd-sddc-poster



bution



bution



bution

VMware Validated Design for Software-Defined Data Center ......management cluster to provide east/west routing across all regions. Using the UDLR reduces the hop count between nodes

Documents