VMware Validated Design for Software-Defined Data Center 4.2 Copyright © 2018 VMware, Inc. All rights reserved. Reference Secondary Storage @vmwcf | vmware.com/go/vvd-docs SSD PCIe Read and Write Cache Capacity NVMe Caching Tier Capacity Tier The design uses NFS as a secondary storage tier. NFS is used for the content library and templates consumed by vRealize Automation blueprints and for vRealize Log Insight log archives. NFS is also used by any vSphere APIs for Data Protection compatible solution to store backups. Storage Internet or Enterprise WAN/MPLS 172.16.11.0/24 192.168.11.0/24 APP OS APP OS APP OS APP OS APP OS Region B Region A ECMP NSX Edge Services Gateways NSXM OS VDP OS PSC OS VC OS Top-of-Rack Leaf Switches L3 L2 BGP Peering Spine Switches NSX Edge Services Gateway One-Arm Load Balancer To Shared Edge and Compute Domain Workload Domains Internet or Enterprise WAN/MPLS 192.168.11.0/24 NSX Edge Services Gateway One-Arm Load Balancer Internet or Enterprise WAN/MPLS ECMP NSX Edge Services Gateways Top-of-Rack Leaf Switches L3 L2 BGP Peering Spine Switches Internet or Enterprise WAN/MPLS 172.17.11.0/24 PSC OS VDP OS VC OS NSXM OS Management Universal Distributed Logical Router 192.168.10.0/24 192.168.10.0/24 192.168.31.0/24 APP OS APP OS APP OS APP OS APP OS 192.168.32.0/24 APP OS APP OS APP OS APP OS APP OS Universal Transit Network Universal Logical Switch / VXLAN Segment Reserved for Disaster Recovery IWS IMS VRA VIP: 192.168.11.56 192.168.11.54 > Active Node 192.168.11.55 > Active Node VIP: 192.168.11.59 192.168.11.57 > Active Node 192.168.11.58 > Passive Node VIP: 192.168.11.53 192.168.11.51 > Active Node 192.168.11.52 > Active Node Region B 192.168.11.0/24 NSX Edge Services Gateway One-Arm Load Balancer ECMP NSX Edge Services Gateways Region Independent Application Virtual Network 192.168.10.0/24 192.168.32.0/24 IAS APP OS IAS APP OS BUC APP OS 192.168.31.0/24 IAS APP OS IAS APP OS BUC APP OS Region Independent Application Virtual Network 192.168.11.0/24 Region A ECMP NSX Edge Services Gateways NSX Edge Services Gateway One-Arm Load Balancer 192.168.10.0/24 VRA IWS IMS DEM IWS IMS DEM APP OS APP OS APP OS APP OS APP OS OS APP OS APP S SQL BUS APP OS APP OS APP OS VRA IWS IMS DEM SQL VRA IWS IMS DEM BUS APP OS APP OS APP OS APP OS APP OS APP OS OS APP OS APP APP OS APP OS Management Universal Distributed Logical Router VRA Distributed Logical Routing and Application Virtual Networks for Management, Operations and Automation Solutions vRealize Automation Appliance VRA vRealize Automation IaaS Web Server IWS vRealize Automation IaaS Manager Service IMS vRealize Automation IaaS vSphere Proxy Agent IAS vRealize Automation Distributed Execution Manager DEM vRealize Business Appliance BUS vRealize Business Data Collector BUC Microsoft SQL Server Database SQL Networks Notable Acronyms Management Application Virtual Network VXLAN Universal Transit Network VXLAN External Transit Network(s) Management Distributed Port Group Logical Component Architecture In a dual-region Software-Defined Data Center, the two Platform Service Controllers and two vCenter Server instances are deployed in each region. This includes a vCenter Server for the management domain and a vCenter Server for the shared edge and compute domains. Each vCenter Server instance is connected to a load-balanced pair of Platform Services Controllers using an NSX Edge Services Gateway. To enable enhanced linked mode, the design joins the Platform Services Controller instances into a unified Single Sign-On domain. Region A Common vCenter Single Sign-On Domain (Ring Topology) Region B Management Domain vCenter Server Appliance Platform Services Controller Appliance Compute Domain vCenter Server Appliance Platform Services Controller Appliance Platform Services Controller Appliance vSphere Update Manager Download Service Compute Domain vCenter Server Appliance In a dual-region Software-Defined Data Center, two primary NSX Manager instances are deployed in Region A. One for the management domain and one for the shared edge and compute domains, along with associated NSX Universal Controller Clusters. In Region B, secondary NSX Manager instances automatically import the configurations of the NSX Universal Controller Clusters from Region A. Region B Management Domain Region A Management Domain Region B Shared Edge and Compute Domain (Edge Resource Pool) Region A Shared Edge and Compute Domain (Edge Resource Pool) NSX Edge Services Gateways (N/S Routing) NSX Edge Services Gateway w/ HA (One-Arm Load Balancer) NSX Edge Services Gateways (N/S Routing) NSX Edge Services Gateway w/ HA (One-Arm Load Balancer) Management Domain NSX Manager (Primary) Compute Domain NSX Manager (Primary) Management Domain NSX Universal Controller Cluster Management Domain NSX Manager (Secondary) Compute Domain NSX Manager (Secondary) Import of Management Domain NSX Controller Configuration from Primary NSX Manager Compute Domain NSX Universal Controller Cluster Import of Compute Domain NSX Controller Configuration from Primary NSX Manager NSX Manager Pairing NSX Manager Pairing Region A Region B NSX Edge Services Gateways (N/S Routing) NSX Edge Services Gateways (N/S Routing) In a dual-region Software-Defined Data Center, a vRealize Log Insight cluster is deployed in each region. Each cluster consists of three nodes, enabling continued availability and increased log ingestion rates. vRealize Log Insight collects and analyzes log data across the domain using the syslog protocol and the ingestion API. vRealize Log Insight also integrates with vRealize Operations Manager to facilitate root cause analysis. Region A Region B Management / Compute vCenter Servers NSX vSAN vRealize Log Insight Cluster vRealize Log Insight Cluster NSX Master Node Worker Node Worker Node Master Node Worker Node Worker Node vRealize Automation Management / Compute vCenter Servers vRealize Operations Any Supported Primary Storage NFS Log Archives Any Supported Primary Storage NFS Log Archives Event Forwarding via Ingestion API vSAN vRealize Log Insight BUS BUC vRealize Business VRA IWS IMS DEM IAS VRA IWS IMS DEM IAS SQL vRealize Automation BUC vRealize Business IAS IAS vRealize Automation BGP Peering BGP Peering BGP Peering BGP Peering BGP Peering BGP Peering Refer to the design release notes for products and versions included in the design. Replicated for Disaster Recovery vRealize Automation / vRealize Orchestrator vRealize Business for Cloud Core vSphere Management NSX vRealize Automation, vRealize Orchestrator and vRealize Business for Cloud Distributed Logical Routing and Application Virtual Networks vRealize Operations and vRealize Log Insight vRealize Automation and vRealize Business for Cloud Primary Storage Core and Domain Architecture Management Distributed Switch Universal Management Transport Zone plus NFS Any Supported Storage vSAN Recommended Management Cluster Minimum 4 Nodes | vSAN ReadyNodes Recommended vSphere HA and DRS Enabled ESXi VTEP VTEP ESXi VTEP VTEP ESXi VTEP VTEP ESXi VTEP VTEP Management Domain Management Domain The management domain hosts the infrastructure components used to instantiate, manage and monitor the SDDC. This includes the core infrastructure components, such as the Platform Services Controllers, vCenter Server instances, NSX Managers, NSX Controllers for the management domain, vSphere Replication, Site Recovery Manager, as well as the SDDC monitoring and automation solutions like vRealize Operations, vRealize Log Insight and vRealize Automation. Managed by Management Domain vCenter Server Workloads running in the SDDC do not have direct access to external networks. To access external networks, tra c is routed through distributed routing to the NSX Edge Services Gateways in the shared edge and compute domain. Expansions beyond the initial shared domain are simply compute Domains. plus NFS Shared Edge and Compute Cluster & Compute Cluster n Minimum 4 Nodes | vSAN ReadyNodes Recommended vSphere HA and DRS Enabled | Business Workload Requirements Compute Distributed Switch ESXi ESXi ESXi ESXi VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP Shared Edge and Compute Domain & Compute Domain n Compute Domain Managed by Compute Domain vCenter Server L3 L2 The design supports L3 or L2 network transport services. For a scalable and vendor-neutral data center network, use an L3 transport. All design documentation is provided for an L3 transport. Adjust the deployment and operations guidance under the context of an L2 transport. . Spine Spine Spine vSphere Update Manager Download Service vRealize Automation Proxy Agents vRealize Operations Remote Collectors vSphere Update Manager Download Service, vRealize Operations Analytics Cluster and Remote Collectors, Regional vRealize Log Insight Cluster, Distributed vRealize Automation and Proxy Agents, and vRealize Business for Cloud Server and Collector. vSphere Update Manager Download Service, vRealize Operations Remote Collectors, Regional vRealize Log Insight Cluster, vRealize Automation Proxy Agents and vRealize Business for Cloud Collector. Disaster Recovery vRealize Operations Analytics Cluster, Distributed vRealize Automation, and vRealize Business for Cloud Server. Application Virtual Networks for SDDC Management Solutions in Region A Application Virtual Networks for SDDC Management Solutions in Region B Universal Transit Network Universal Logical Switch / VXLAN Segment Universal Transit Network Universal Logical Switch / VXLAN Segment Universal Transit Network Universal Logical Switch / VXLAN Segment Universal Logical Switch / VXLAN Segment Universal Logical Switch / VXLAN Segment Region Dependent Application Virtual Network Universal Logical Switch / VXLAN Segment Region Dependent Application Virtual Network Universal Logical Switch / VXLAN Segment Region Independent Application Virtual Network Universal Logical Switch / VXLAN Segment Region Independent Application Virtual Network Universal Logical Switch / VXLAN Segment Region Dependent Application Virtual Network Universal Logical Switch / VXLAN Segment Region Dependent Application Virtual Network Universal Logical Switch / VXLAN Segment Any Supported Storage vSAN Recommended All design documentation for is provided for an L3 transport with BGP based peering. A TechNote is provided for the alternative mixed-use or end-to-end use of OSPF. Platform Services Controller Appliance The design uses standardized building blocks called workload domains. Below is the standard design based on a two domain model with a dedicated management domain and shared edge/compute domain. To Shared Edge and Compute Domain Workload Domains Management Domain vCenter Server Appliance Management Domain vCenter Server Appliance Compute Domain vCenter Server Appliance Management Domain vCenter Server Appliance Compute Domain vCenter Server Appliance Server 10 GigE L3 L2 L3 L2 10 GigE Additional Compute Domains Shared Edge and Compute Domain (4+ Hosts) Management Domain (4+ Hosts) Leaf Leaf Leaf Leaf Leaf Leaf IGMP IGMP IGMP IGMP IGMP IGMP Leaf Leaf IGMP IGMP Host Connectivity Management Distributed Switch Management Domain ESXi Host nic0 nic1 VLAN NFS VLAN Management VLAN vMotion VLAN VTEP (VXLAN) VLAN vSAN VMK MTU 9000 VMK MTU 9000 VMK MTU 9000 VMK MTU 9000 VMK MTU 9000 10 GigE 10 GigE vDS MTU 9000 Compute Distributed Switch Shared Edge and Compute Domain ESXi Host nic0 vDS MTU 9000 nic1 10 GigE 10 GigE VLAN NFS VLAN Management VLAN vMotion VLAN VTEP (VXLAN) VLAN vSAN VMK MTU 9000 VMK MTU 9000 VMK MTU 9000 VMK MTU 9000 VMK MTU 9000 VLAN Uplink 02 VLAN Uplink 01 Layer 3 ToR Switch VLAN 1611 VLAN 1612 VLAN 1613 VLAN 1614 Management 172.16.11.0/24 DGW: 172.16.11.253 vMotion 172.16.12.0/24 DGW: 172.16.12.253 VXLAN 172.16.13.0/24 vSAN 172.16.14.0/24 ESXi Host Routed Uplinks (ECMP) VLAN Trunk (802.1Q) L2 L3 Span of VLANs Span of VLANs When using the recommended L3 network transport, the top-of-rack leaf switches of each rack act as the corresponding L3 interface for the associated subnets. The management domain and the shared edge and compute domain are provided with externally accessible VLANs to access to the Internet and corporate networks. The two 10GbE NICs on each host are connected across the top-of-rack leaf switches and teamed on the vSphere Distributed Switch via an active-active configuration. All port groups, except for the ones that carry VXLAN tra c, are configured for the 'Route based on physical NIC load' teaming algorithm. VTEP kernel ports and VXLAN tra c use the ’Route based on SRC-ID' algorithm. The vSphere Distributed Switch has a MTU of 9000 configured for Jumbo Frames along with with necessary VMkernel ports. Spine Spine 40 GigE 40 GigE Network Transport Region Protection and Disaster Recovery Workload Domains Region A Non-Replicated vRealize Log Insight Protection Groups • vRealize Automation • vRealize Business for Cloud • vRealize Operations Protection Groups • vRealize Automation • vRealize Business for Cloud • vRealize Operations Region B Non-Replicated vRealize Log Insight Region A Infrastructure Management vSphere NSX Site Recovery Manager Region B Infrastructure Management vSphere NSX Site Recovery Manager Region A Replicated Region B Replicated Domains Management North/South Uplink(s) vMotion vSAN Region Dependent VXLAN Any Supported Storage ESXi-MGMT-01 ESXi-MGMT-02 ESXi-MGMT-03 VTEPs VTEPs VTEPs Management North/South Uplink(s) vMotion vSAN VXLAN xxxx Any Supported Storage ESX-COMP-01 VTEPs VLAN vSphere Replication VLAN Uplink 02 VLAN Uplink 01 VMK MTU 9000 Region B Management Domain Region A Management Domain Region A Management Domain Region B Management Domain Region A Management Domain Region B Management Domain NSX Controllers (Compute) Region A Export for Content Library and Templates Export for Log Archives Export for Backups NFS Storage Array Volume 2 Volume 1 Region B NFS Storage Array Volume 2 Volume 1 vCenter (Compute) PSC (Compute) NSX Manager (Compute) NSX Manager (Management) PSC (Management) vCenter (Management) VDP (Management) SRM (Management) VR (Management) External Networks N/S NSX EDGE (Compute) vRealize Automation Business Groups & Reservations The design integrates solutions for compute, storage, network, cloud operations, and cloud management. A single vRealize Operations analytics cluster monitors and performs diagnostics across the Software-Defined Data Center by using a series of remote collectors and solution management packs. vRealize Operations Region Independent VXLAN VXLAN xxxx Core and Domain Architecture Core and Domain Architecture L3 L2 L3 L2 APP OS APP OS APP OS APP OS APP OS Universal Logical Switch APP OS APP OS APP OS APP OS APP OS APP OS Universal Logical Switch APP OS APP OS APP OS APP OS Universal Logical Switch APP OS APP OS APP OS Universal Logical Switch APP OS L2 L2 UDLR UDLR UDLR Edge Resource Pool UDLR UDLR UDLR UDLR & DLR N/S NSX EDGE (Management) UDLR & DLR L3 L3 L2 L2 L3 L3 L3 L3 L3 L3 Distributed Switches Universal Transport Zones Core Platform Services Application Virtual Networks for SDDC Solutions Workload Virtual Networks North/South Routing NSX Controllers (Management) Management Custer Edge/Compute Cluster The design establishes a Cloud Management Platform with vRealize Automation to provide a service catalog and self-service portal to deploy, update, and manage the workloads. Its embedded instance of vRealize Orchestrator provides a repository of extensible workflows and integrations. vRealize Business for Cloud provides visibility into the financial aspects of the cloud infrastructure, allowing cost to be tracked and optimized. The design implements a single vRealize Automation tenant. Business groups can be created to fit your needs. Within each business group the tenant administrators are able to manage users and groups, apply tenant-specific branding, enable notifications, configure business policies, and manage the service catalog. The IT Automating IT Use Case documenation provides implementation steps for a set of scenarios. One region is designated as the primary region and the other as the secondary region. SDDC management, automation and operations solutions are deployed in the primary region and configured to migrate to the secondary region in the event of a disaster. All regions actively run business workloads. https://my.sddc.local/vcac/org/company Sign In Business Group Manager Business Group Manager Tenant Admin IaaS Admin Tenant Admin IaaS Admin Business Group Reservation Business Group Reservation Edge Reservation Region B Data Center Infrastructure Fabric Region B Fabric Group Fabric Admin Additional Compute Domain(s) UDLR & DLR UDLR & DLR Export for Content Library and Templates Export for Log Archives Export for Backups Shared Edge/Compute Domain Edge Reservation Business Group Reservation Business Group Reservation Region A Data Center Infrastructure Fabric Region A Fabric Group Fabric Admin Additional Compute Domain(s) Shared Edge/Compute Domain Universal Compute Transport Zone Platform Services Controller PSC NSX Manager NSXM Site Recovery Manager SRM Universal Distributed Logical Router UDLR VXLAN Tunnel Endpoint VTEP vSphere Data Protection VDP vSphere Replication VR 192.168.11.0/24 Region A ECMP NSX Edge Services Gateways NSX Edge Services Gateway One-Arm Load Balancer 192.168.10.0/24 192.168.31.0/24 Master Node APP OS APP OS APP OS Replica Node Data Node APP OS APP OS Collector Node Collector Node vRealize Operations vRealize Operations vRealize Log Insight Cluster VIP APP OS APP OS APP OS Master Node Worker Node Worker Node Region B 192.168.11.0/24 NSX Edge Services Gateway One-Arm Load Balancer ECMP NSX Edge Services Gateways 192.168.10.0/24 192.168.32.0/24 APP OS APP OS Collector Node Collector Node vRealize Operations vRealize Log Insight Replicated for Disaster Recovery Cluster VIP APP OS APP OS APP OS Master Node Worker Node Worker Node Master Node APP OS APP OS APP OS APP OS Replica Node Data Node Management Universal Distributed Logical Router Local Compute Transport Zone Universal Transit Network Universal Logical Switch / VXLAN Segment Universal Transit Network Universal Logical Switch / VXLAN Segment Region Independent Application Virtual Network Universal Logical Switch / VXLAN Segment Region Independent Application Virtual Network Universal Logical Switch / VXLAN Segment NSX Edge Services Gateway w/ HA (Load Balancer) NSX Edge Services Gateway w/ HA (Load Balancer) Shared Storage Systems Region A Region B Management / Compute vCenter Servers NSX Remote Collectors Remote Collectors NSX Shared Storage Systems Analytics Cluster Master Node Replica Node Data Node Clctr Node Clctr Node Clctr Node Clctr Node vRealize Automation Management / Compute vCenter Servers Region B Management Domain Region A Management Domain Region Dependent Application Virtual Network Universal Logical Switch / VXLAN Segment Region Dependent Application Virtual Network Universal Logical Switch / VXLAN Segment All design documentation and validation is provided using vSAN as the primary storage system. vSAN enables both all-flash and hybrid architectures. Adjust deployment and operations for supported storage systems. Use of vSAN ReadyNodes is recommended to ensure seamless compatibility and support. The configuration and assembly of the components are standardized to eliminate system variability. A consolidated management and compute design is also available. Refer to the VVD documentation. or (failover/failback) SRM vSphere Replication when using vSAN Replication