Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail Powered by Dell EMC PowerSwitch and Pluribus Netvisor ONE Abstract Design Guide This deployment guide describes how to deploy the Pluribus Adaptive Cloud Fabric to interconnect multiple geographically distributed data center sites running Dell EMC SmartFabric OS10 in the leaf/spine switches with an application focus on stretched VxRail Clusters to support workload mobility and active-active resiliency. April 2021 H18513 Dell Technologies Networking Infrastructure Solutions
32
Embed
DE-PN-DG Dell EMC Data Center Interconnect Fabric For OS10 ...
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRailPowered by Dell EMC PowerSwitch and Pluribus Netvisor ONE
Abstract
Design Guide
Networking from Dell Technologieswww.DellTechnologies.com/Networking
This deployment guide describes how to deploy the Pluribus Adaptive Cloud Fabric to interconnect multiple geographically distributed data center sites running Dell EMC SmartFabric OS10 in the leaf/spine switches with an application focus on stretched VxRail Clusters to support workload mobility and active-active resiliency.
Pluribus UNUM – Unified Automation, Management and Analytics 26
Support and references 31
A NOTE indicates important information that helps you make better use of your product.A CAUTION indicates either potential damage to hardware or loss of data and tells you how to avoid the problem.A WARNING indicates a potential for property damage, personal injury, or death.
Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail | 2
Overview This guide describes how to configure and implement the Pluribus Edge Data Center Interconnect (DCI) solution to interconnect multiple data centers running Dell EMC OS10 on the leaf/spine switches. The DCI solution uses the Pluribus Adaptive Cloud Fabric (ACF) so�ware to provide an SDN-automated VXLAN over IP DCI fabric between all of the DC sites. The application focus is on stretching layer 2 VLANs to support VxRail vSAN clusters across multiple datacenters to support workload mobility and active-active resiliency in conjunction with a VxRail Witness Host.
The local fabrics in each data center are interconnected across the WAN by means of connecting from the border leaf switch pair at each DC location with standard VLANs to pair of collocated Dell ON switches which are running Pluribus Netvisor ONE operating system and acting as the DCI switches. The Pluribus Adaptive Cloud Fabric so�ware is deployed between all of the DCI switches and is a distributed architecture which delivers a radically simplified approach to building large scale DCI solutions leveraging industry standard Virtual Extensible LAN (VxLAN) encapsulation virtualized overlay network featuring SDN automation, transported over an IP underlay network.
The Pluribus DCI solution enables organizations to simplify their data center network with a distributed fabric bringing unified management across multiple datacenters and unparalleled operational simplification and agility.
Supporting up to 64 geographically distributed data centers, the Pluribus Edge DCI solution allows the extension of Layer 2 & Layer-3 services across multiple locations connected via an existing WAN orcore network.
Pluribus ACF can be managed via CLI, RESTful API or Pluribus UNUM virtual appliance and also supports scripting languages such as Python and automation frameworks such as Ansible. UNUM is a unified fabric management, automation and analytics platform that o�ers a rich graphical user interface and enables administrators to further reduce day 0, 1 and 2 DCI operations.
Figure 1 Pluribus So�ware-Defined DCI Solution
3 | Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail
Dell switches used for DCI must support Pluribus Netvisor ONE R5.2 or later. A list of compatible switches can be found here https://www.pluribusnetworks.com/assets/PN-DS-Pluribus_Netvisor_Open_Networking_Edition.pdf. Pluribus recommends S52xx switches or later be used when possible.
Hardware and So�ware Compatibility
There are several methods to interconnect data center sites with VxRail clusters directly connected to Dell OS10 or SFS fabrics. In this design guide we are assuming that each site VxRail site fabric is based on a leaf-spine fabric topology (based on one of the Dell Technologies network solutions mentioned above). Each site fabric has a designated pair of border leaf switches, which connect via industry standard layer 2 VLANs to a pair of Dell Open Networking switches running Pluribus Netvisor ONE OS with ACF activated. These switch pairs, running Pluribus ACF at each site, then connect to the WAN network and stretch the VxRail networks over the WAN using a VXLAN based overlay.
Edge DCI Interconnection for multiple local data center fabrics
The reference topology used in this design guide is shown in the following figure, which describes the physical interconnections between the border leaf and the Edge DCI switches in each of the two data centers with the VxRail infrastructure, as well as the third site hosting the vSAN witness host, whose function is to protect against split-brain in a vSAN Stretched Clusters.
Pluribus Edge DCI Solution Overview
Pluribus Edge DCI solution provides a highly available, highly scalable distributed fabric to stretch a VxRail vSAN cluster across the WAN interconnecting the datacenters. Based on Pluribus AdaptiveCloud Fabric, the DCI solution allows organizations to simplify the deployment and migration of VMware workloads and vSAN storage across any site to optimize compute resources and enhance business continuity.
The Pluribus Edge DCI solution allows the extension of Layer-2 and Layer-3 network service across the IP WAN to support stretching vSAN services across multiple locations connected via any existing WAN or Core network.
The Pluribus Edge DCI solution interconnection provides the following features: • Unified and centralized management among all the data centers • Simple, cloud-style provisioning and monitoring the overlay network services • Adapts and interoperate with any WAN network • Scales to a large number (64) of sites • Provides site-to-site tra�ic visibility and analytic
The Pluribus Edge DCI with Dell OS10 Hands-on Lab is available on the Demo Center and can help you understand how to configure a data center fabric to achieve so�ware-defined networks. See Demo Center for information about the Hands-on Lab.
Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail | 4
Pluribus Edge DCI Deployment Overview Demo Center Hands-on Lab includes full configurations for all the Pluribus Edge DCI nodes. The configuration steps assume Netvisor ONE OS is already loaded on the Dell switches and is shown both for CLI as well as via the UNUM Fabric Manager and cover the following tasks: • Creation of underlay connections for Pluribus Edge DCI fabric • Creation of VTEP (Virtual Tunnel End Point) to support the overlay network required to stretch the VxRail cluster • Creation of stretched VLANs and subnet objects • Creation of a distributed VRF service to provide inter-vlan routing for the VxRail components
The following figure shows configuration details for the pair of Pluribus Edge DCI nodes referred to as Core-1 (for Sites A), Core-3 (for Site-B) and Core-2 (for Site-C). The three sites are unified and managed with a single ACF instance.
Each of the Dell PODs in the primary and secondary datacenters has a pair of designated leaf switches acting as border nodes to hand-o� the vSAN VxRail tra�ic to the Pluribus Edge DCI Fabric as depictedin Figure 1.
The following configuration details refers to the four switches, running Dell OS10, directly connected to the Dell DCI switches running Pluribus Netvisor.
Dell EMC SmartFabric OS10 POD Switch Configuration
Figure 2 Pluribus Interconnection details
5 | Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail
4112F-01# show running-configuration
S4112F-02# show running-configuration
S4128F-01# show running-configuration
S4128F-02# show running-configuration
! Version 10.5.2.2!ip vrf default!iscsi enableiscsi target port 860iscsi target port 3260hostname S4112F-01interface breakout 1/1/13 map 100g-1xinterface breakout 1/1/14 map 100g-1xinterface breakout 1/1/15 map 100g-1xno multicast snooping flood-restrictsystem-user linuxadmin password ****username admin password **** role sysadmin priv-lvl 15aaa authentication login default localaaa authentication login console local!class-map type application class-iscsi!policy-map type application policy-iscsi!interface vlan1 no shutdown!interface vlan30 description Engineering- VLAN
! Version 10.5.2.2!ip vrf default!iscsi enableiscsi target port 860iscsi target port 3260hostname S4112F-02interface breakout 1/1/13 map 100g-1xinterface breakout 1/1/14 map 100g-1xinterface breakout 1/1/15 map 100g-1xno multicast snooping flood-restrictsystem-user linuxadmin password ****username admin password **** role sysadmin priv-lvl 15aaa authentication login default localaaa authentication login console local!class-map type application class-iscsi!policy-map type application policy-iscsi!interface vlan1 no shutdown!interface vlan30 description Engineering- VLAN
! Version 10.5.2.2!ip vrf default!iscsi enableiscsi target port 860iscsi target port 3260hostname S4128F-01interface breakout 1/1/25 map 100g-1xinterface breakout 1/1/26 map 100g-1x
no multicast snooping flood-restrictsystem-user linuxadmin password ****username admin password **** role sysadmin priv-lvl 15aaa authentication login default localaaa authentication login console local!class-map type application class-iscsi!policy-map type application policy-iscsi!interface vlan1 no shutdown!interface vlan30 description Engineering- VLAN
! Version 10.5.2.2!ip vrf default!iscsi enableiscsi target port 860iscsi target port 3260hostname S4128F-02interface breakout 1/1/25 map 100g-1xinterface breakout 1/1/26 map 100g-1x
no multicast snooping flood-restrictsystem-user linuxadmin password ****username admin password **** role sysadmin priv-lvl 15aaa authentication login default localaaa authentication login console local!class-map type application class-iscsi!policy-map type application policy-iscsi!interface vlan1 no shutdown!interface vlan30 description Engineering- VLAN
4112F-01# show running-configuration
S4112F-02# show running-configuration
S4128F-01# show running-configuration
S4128F-02# show running-configuration
no shutdown mtu 9216!interface vlan40 description Marketing-VLAN no shutdown mtu 9216!interface vlan1111 description Mgmt-VLAN no shutdown mtu 9216!interface vlan1312 description vSAN-VLAN no shutdown mtu 9216!interface vlan1412 description vMotion-VLAN no shutdown mtu 9216!interface vlan3939 description VxRail-Dis-VLAN no shutdown mtu 9216!interface port-channel101 description Channel-2- Pluri-Core-1 no shutdown switchport mode trunk switchport access vlan 1 switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216 vlt-port-channel 101!interface mgmt1/1/1 no shutdown no ip address dhcp ip address 172.17.180.31/24 ipv6 address autoconfig!interface ethernet1/1/1 no shutdown switchport mode trunk switchport access vlan 1 switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216 flowcontrol receive on!
interface ethernet1/1/2 no shutdown
no shutdown mtu 9216!interface vlan40 description Marketing-VLAN no shutdown mtu 9216!interface vlan1111 description Mgmt-VLAN no shutdown mtu 9216!interface vlan1312 description vSAN-VLAN no shutdown mtu 9216!interface vlan1412 description vMotion-VLAN no shutdown mtu 9216!interface vlan3939 description VxRail-Dis-VLAN no shutdown mtu 9216!interface port-channel101 description Channel-2- Pluri-Core-1 no shutdown switchport mode trunk switchport access vlan 1 switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216 vlt-port-channel 101!interface mgmt1/1/1 no shutdown no ip address dhcp ip address 172.17.180.32/24 ipv6 address autoconfig!interface ethernet1/1/1 no shutdown switchport mode trunk switchport access vlan 1 switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216 flowcontrol receive on!
interface ethernet1/1/2 no shutdown
no shutdown mtu 9216!interface vlan40 description Marketing-VLAN no shutdown mtu 9216!interface vlan1111 description Mgmt-VLAN no shutdown mtu 9216!interface vlan1312 description vSAN-VLAN no shutdown mtu 9216!interface vlan1412 description vMotion-VLAN no shutdown mtu 9216!interface vlan3939 description VxRail-Disc-VLAN no shutdown mtu 9216!interface port-channel102 description Po-2-Core-3
no shutdown switchport mode trunk switchport access vlan 1 switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216 vlt-port-channel 102!interface mgmt1/1/1 no shutdown no ip address dhcp ip address 172.17.180.36/24 ipv6 address autoconfig!interface ethernet1/1/1 no shutdown switchport access vlan 3939 mtu 9216 flowcontrol receive on
!interface ethernet1/1/2 description Link-2-VxRail- Nodes no shutdown switchport mode trunk
no shutdown mtu 9216!interface vlan40 description Marketing-VLAN no shutdown mtu 9216!interface vlan1111 description Mgmt-VLAN no shutdown mtu 9216!interface vlan1312 description vSAN-VLAN no shutdown mtu 9216!interface vlan1412 description vMotion-VLAN no shutdown mtu 9216!interface vlan3939 description VxRail-Disc-VLAN no shutdown mtu 9216!interface port-channel102 description Upstream- LAG-102 no shutdown switchport mode trunk switchport access vlan 1 switchport trunk allowed vlan 30,40,1111,1312,1412, 3939 mtu 9216 vlt-port-channel 102!interface mgmt1/1/1 no shutdown no ip address dhcp ip address 172.17.180.35/24 ipv6 address autoconfig!interface ethernet1/1/1 no shutdown switchport access vlan 3939 mtu 9216 flowcontrol receive on
!interface ethernet1/1/2 description Link-2-VxRail- Nodes no shutdown switchport mode trunk
Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail | 6
The Pluribus Ede DCI switches in Figure 2 are directly connected to a border-leaf (Dell) devices in a datacenter. All the Pluribus core switches in Figure 2 are running Pluribus Netvisor operating system and provides connectivity between data centers.
Configuring the underlay network involves the configuration of (at least) the following base features:
The following configuration is for switches Site-A:Core-1, Site-C:Core-2(witness) and Site-B:Core-3 only.
Detailed Underlay (Day-0) configurations steps:
Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail | 10
Enable physical port, speed & mtu for jumbo frames
Configure port channel
Configure the in-band IP for fabric communication
Enable OSPF to ensure in-band IP reachability across the all core node
Create the Fabric and define all the in-band subnets of the nodes that will be part of the Fabric
Configure the rest of the core switch across sites to join the existing Fabric
Configure the VXLAN Tunnel End-Points (VTEPs)
[Optional] Reserve a loopback port for VXLAN routing packet recirculation. Not required with Dell S52xx and S41xx ON switches.
Step 1Step 2Step 3
Step 4
Step 5
Step 6Step 7Step 8
About this taskNOTE: The Pluribus cluster (High-Availability) configuration is not described in this guide, however Pluribus recommended to use cluster for redundancy and high-availability.
Core-1 Core-3 Core-2 (witness)
!Enable Port connecting to Core-1 and Leaf1 (S4112F-01) & Leaf2 (S4112F-02)
CLI (network-admin@Core-1) > port-config-modify port 33 speed 100g enable jumbo
CLI (network-admin@Core-1) > port-config-modify port 41 speed 100g enable jumbo
!Enable Port connecting to Core-2
CLI (network-admin@Core-1) > port-config-modify port 9 speed 100g enable jumbo
!Enable Port connecting to Core-3
CLI (network-admin@Core-1) > port-config-modify port 89 speed 100g enable jumbo
CLI (network-admin@Core-1) > port-config-modify port 97 speed 100g enable jumbo
Enable Port connecting to Core-3 and Leaf3 (s4128F-01) & Leaf4 (s4128F-02)
CLI (network-admin@Core-3) > port-config-modify port 113 speed 100g enable jumbo
CLI (network-admin@Core-3) > port-config-modify port 121 speed 100g enable jumbo
!Enable Port connecting to Core-2
CLI (network-admin@Core-3) > port-config-modify port 9 speed 100g enable jumbo
!Enable Port connecting to Core-1
CLI (network-admin@Core-3) > port-config-modify port 89 speed 100g enable jumbo
CLI (network-admin@Core-3) > port-config-modify port 97 speed 100g enable jumbo
!Enable Port connecting to Core-2 and Leaf1
CLI (network-admin@Core-2) > port-config-modify port 1 speed 10g enable jumbo
CLI (network-admin@Core-2) > port-config-modify port 2 speed 10g enable jumbo
!Enable Port connecting to Core-1
CLI (network-admin@Core-2) > port-config-modify port 61 speed 100g enable jumbo
!Enable Port connecting to Core-3
CLI (network-admin@Core-2) > port-config-modify port 57 speed 100g enable jumbo
Core-1 Core-3 Core-2 (witness)
!Port-phy-show
CLI (network-admin@Core-1) > port-phy-show port 9,33,41,89port state speed eth-mode max-frame def-vlan ---- ----- ------ -------- --------- -------- 9 up 40000 sr4 9412 4092 33 up 100000 caui4 9412 1 41 up 100000 caui4 9412 1 89 up 40000 sr4 9412 1
!Port-phy-show
CLI (network-admin@Core-3) > port-phy-show port 9,89,113,121port state speed eth-mode max-frame def-vlan ---- ----- ------ -------- --------- -------- 9 up 40000 sr4 9412 4092 89 up 40000 sr4 9412 1 113 up 100000 caui4 9412 1 121 up 100000 caui4 9412 1
!Port-phy-show
CLI (network-admin@Core-2) > port-phy-show port 1,2,57,61port state speed eth-mode max-frame def-vlan ---- ----- ----- -------- --------- -------- 1 up 10000 xfi 9412 1 2 up 10000 xfi 9412 1 57 up 40000 sr4 9412 4091 61 up 40000 sr4 9412 4092
11 | Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail
Step 1 : Enable Physical port, speed and MTU for jumbo frames
A�er plugging in the cables, login to each Pluribus core switch and use the following command to enable port, speed, and increase the MTU value to jumbo.
Use the following sample configuration for Core-1, Core-2 (witness)and Core-3
Validation Steps:Use the following command to verify on Core-1, Core-2 (witness)and Core-3
NOTE: VXLAN encapsulation adds between 50 and 54 bytes of additional header information to the original Ethernet frame. Because this can result in Ethernet frames that exceed the default 1514-byte MTU, best practice is to implement jumbo frames throughout the network.
Core-1
!Enable Port-101 connecting to Core-1 and Leaf1 (S4112F-01) & Leaf2 (S4112F-02)
CLI (network-admin@Core-1) >trunk-create name PO-101 ports 33,41 jumbo lacp-mode active
Core-3
!Enable Port-102 connecting to Core-2 and Leaf3 (s4128F-01) & Leaf4 (s4128F-02)
CLI (network-admin@Core-3) >trunk-create name PO-102 ports 113,121 jumbo lacp-mode active
Step 2 : Configure Port-Channel between Core and Border-Leaf’s
Configure Port-channel with LACP to provide more bandwidth, load balance tra�ic across multiple physical connected links and redundancy.
Use the following sample configuration for Core-1 and Core-3
Step 3 : Configure the in-band IP for fabric communication
This step focuses on setting up the Adaptive Cloud Fabric control plane communication using the in-band management interface which is reachable through the switch front panel ports carrying the network tra�ic.
In order to behave as a unified single-logical system, all switches that are part of an Adaptive Cloud Fabric need to exchange fabric-related messages such as: control-plane messages, configuration transactions, network and endpoint state notifications, remote procedure calls and file system replication messages. This communication is always encrypted and can be enabled via an out-of-band management network or via the in-band network. In-band communication has two main advantages: first it removes the dependence on an external management network, second it leverages the high-availability and resilience of production network.
The following steps configure a dedicated in-band management network on each switch using an internal VLAN.
Use the following sample configuration for Core-1,Core-2 (witness) and Core-3
Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail | 12
Validation Steps:Use the following command to verify Port-Channel on Core-1 and Core-3
America/Los_Angelesdate: 2021-04-03,05:52:47hostid: 184558815location-id: 2enable-host-ports: yesbanner: * Welcome to Pluribus Networks Inc. Netvisor(R). This is a monitored system. *device-id: GGLQG02ntp: onbanner: * ACCESS RESTRICTED TO AUTHORIZED USERS ONLY *banner: * By using the Netvisor(R) CLI,you agree to the terms of the Pluribus Networks *banner: * End User License Agreement (EULA). The EULA can be accessed via *banner: * http://www.pluribusnet-works.com/eula or by using the command "eula-show" *
CLI (network-admin@Core-1*) >
!Vrouter-interface-show
CLI (network-admin@Core-1) > vrouter-interface-show nic eth1.4050 vrouter-name nic ip mac vlan vlan-type nic-state fabric-nic mtu priority-tag ------------ --------- ------------ ----------------- ---- --------- --------- ---------- ---- ------------ Core-1 eth1.4050 10.20.1.2/24 66:0e:94:df:f4:f5 4050 public up fabric-nic 1500 off CLI (network-admin@Core-1) >
Netvisor(R). This is a monitored system. *device-id: GGHQG02ntp: onbanner: * ACCESS RESTRICTED TO AUTHORIZED USERS ONLY *banner: * By using the Netvisor(R) CLI,you agree to the terms of the Pluribus Networks *banner: * End User License Agreement (EULA). The EULA can be accessed via *banner: * http://www.pluribusnet-works.com/eula or by using the command "eula-show" *CLI (network-admin@Core-3*) >
!Vrouter-interface-show
CLI (network-admin@Core-3) > vrouter-interface-show nic eth1.4050 vrouter-name nic ip mac vlan vlan-type nic-state fabric-nic mtu priority-tag ------------ --------- ------------ ----------------- ---- --------- --------- ---------- ---- ------------ Core-3 eth1.4050 10.20.3.2/24 66:0e:94:de:6c:fc 4050 public up fabric-nic 1500 off CLI (network-admin@Core-3) >
enable-host-ports: yesbanner: * Welcome to Pluribus Networks Inc. Netvisor(R). This is a monitored system. *device-id: 9K31XC2ntp: onbanner: * ACCESS RESTRICTED TO AUTHORIZED USERS ONLY *banner: * By using the Netvisor(R) CLI,you agree to the terms of the Pluribus Networks *banner: * End User License Agreement (EULA). The EULA can be accessed via *banner: * http://www.pluribusnet-works.com/eula or by using the command "eula-show" *
CLI (network-admin@Core-2) >
!Vrouter-interface-show
CLI (network-admin@Core-2*) > vrouter-interface-show nic eth0.4050 vrouter-name nic ip mac vlan vlan-type nic-state fabric-nic mtu priority-tag ------------ --------- ------------ ----------------- ---- --------- --------- ---------- ---- ------------ Core-2 eth0.4050 10.20.2.2/24 66:0e:94:f8:98:4a 4050 public up fabric-nic 1500 off CLI (network-admin@Core-2) >
e,mtunetwork-admin@Core-1) > vrouter-interface-show format nic,ip,vlan,nic-state,mtuvrouter-name nic ip vlan nic-state mtu ------------ --------- ---------------- ---- --------- ---- Core-1 eth0.4092 10.1.2.1/30 4092 up 9216 Core-1 eth2.4000 10.1.3.1/30 4000 up 9216
!Vrouter-ospf-interface-show
CLI (network-admin@Core-1) > vrouter-ospf-interface-show vrouter-name nic l3-port area mtu state hello-due neighbor ------------ --------- ------- ------- ---- -------------- --------- -------- Core-1 eth0.4092 9 0.0.0.0 9216 point-to-point 0.864s 1 Core-1 eth2.4000 0.0.0.0 9216 point-to-point 1.168s 1
CLI (network-admin@Core-3) > vrouter-interface-show format nic,ip,vlan,nic-state,mtu vrouter-name nic ip vlan nic-state mtu ------------ --------- --------------- ---- --------- ---- Core-3 eth2.4000 10.1.3.2/30 4000 up 9216 Core-3 eth1.4092 10.2.3.2/30 4092 up 9216
!Vrouter-ospf-interface-show
CLI (network-admin@Core-3) > vrouter-ospf-interface-showvrouter-name nic l3-port area mtu state hello-due neighbor ------------ --------- ------- ------- ---- -------------- --------- -------- Core-3 eth1.4092 9 0.0.0.0 9216 point-to-point 0.357s 1 Core-3 eth2.4000 0.0.0.0 9216 point-to-point 0.463s 1
!Vrouter-ospf-neighbor-show
CLI (network-admin@Core-3) > vrouter-ospf-neighbor-show format neighbor,pri,State,address,interface,vlan
CLI (network-admin@Core-2) > vrouter-interface-show format nic,ip,vlan,nic-state,mtu vrouter-name nic ip vlan nic-state mtu ------------ --------- ---------------- ---- --------- ---- Core-2 eth0.4092 10.1.2.2/30 4092 up 9216 Core-2 eth1.4091 10.2.3.1/30 4091 up 9216
!Vrouter-ospf-interface-show
CLI (network-admin@Core-2) > vrouter-ospf-interface-show vrouter-name nic l3-port area mtu state hello-due neighbor ------------ --------- ------- ------- ---- -------------- --------- -------- Core-2 eth0.4092 61 0.0.0.0 9216 point-to-point 0.068s 1 Core-2 eth1.4091 57 0.0.0.0 9216 point-to-point 0.068s 1
!Vrouter-ospf-neighbor-show
CLI (network-admin@Core-2) > vrouter-ospf-neighbor-show format neighbor,pri,State,address,interface,vlan
17 | Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail
Step 5 : Create the Fabric and define all the in-band subnets of the nodes that will be part of the fabric
A�er completing the initial setup, the OSPF underlay, and the in-band properties, it is now time to create a new fabric instance to subsequently add all other switches.
Choose any switch to create the fabric instance and use the optional password option, so that other switches can join the fabric only by providing the assigned password.
Joined fabric PN-DCI, Restarting nvOS…Please enter username and password: Username (network-admin): Password:
Step 6 : Configure the rest of the core switches across all sites to join the existing Fabric
Use below command to join fabric on Core-2 (witness) and Core-3
Step 7 : Configure the VXLAN Tunnel End-Points (VTEPs) Each Core is configured with a VTEP. VxLAN uses VxLAN tunnel end-point (VTEP) to map tenants to segments and performs VXLAN encapsulation and de-capsulation on all packets. By creating the VTEP objects above, the Fabric will dynamically create a mesh of interconnecting tunnels among them without any additional operational e�ort.
Use the following sample configuration for Core-1, Core-2 (witness) and Core-3
Validation Steps:Use below command to verify fabric-node-show on any Core-1, Core-2 (witness) or Core-3
Core-1
!fabric-node-show
CLI (network-admin@Core-1) > fabric-node-showname fab-name mgmt-ip in-band-ip in-band-vlan-type fab-tid out-port version state firmware-upgrade device-state ------ -------- ---------------- ------------ ----------------- ------- -------- ---------------- ------ Core-1 PN-DCI 172.17.180.14/24 10.20.1.1/24 public 13 6.0.3-6000317001 online not-required ok Core-3 PN-DCI 172.17.180.15/24 10.20.3.1/24 public 13 89 6.0.3-6000317001 online not-required ok Core-2 PN-DCI 10.20.2.1/24 public 13 61 6.0.3-6000317001 online not-required ok CLI (network-admin@Core-1) >
Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail | 18
NOTE: A�er fabric-join, it will restart nvOS and the switch join the fabric.
NOTE: Pluribus logical VTEP is not described in this guide, however Pluribus recommends using cluster for VTEP-HA (high-availability). More information can be found here.
NOTE: Single command can be also used to create the VLAN with fabric-scope, and configure SVI individually using switch-local.
Validation Steps:Use below command to verify vlan-show <id> on any Core-1
Core-1
!Vlan-show <id> CLI (network-admin@Core-1) > vlan-show id 4040switch id type auto-vxlan replicators scope description active stats ports untagged-ports active-edge-ports------ ---- ------ ---------- ----------- ----- ----------- ------ ----- ----- -------------- -----------------Core-1 4040 public no none fabric VTEP-VLAN yes yes 0 none 0Core-3 4040 public no none fabric VTEP-VLAN yes yes 0 none 0Core-2 4040 public no none fabric VTEP-VLAN yes yes 0 none 0
Validation Steps:Use below command to verify vtep-show on any Core-1, Core-2 (witness) or Core-3
Core-1 Core-3 Core-2 (witness)
!Run commands locally on a switch by using theCLI (network-admin@Core-1) > switch-local
!Create VTEP-VLAN-SVICLI (network-admin@Core-1)*> vrouter-interface-add vrouter-name Core-1 vlan 4040 ip 40.40.1.1/29 mtu 9216
CLI (network-admin@Core-1*) > vtep-showscope name location vrouter-name ip virtual-ip mac-learning ------ ----------- -------- ------------ --------- ---------- ------------ fabric VTEP-Core-1 Core-1 Core-1 40.40.1.1 :: on fabric VTEP-Core-3 Core-3 Core-3 40.40.3.1 :: on fabric VTEP-Core-2 Core-2 Core-2 40.40.2.1 :: on
!Vtep-show
CLI (network-admin@Core-3*) > vtep-showscope name location vrouter-name ip virtual-ip mac-learning ------ ----------- -------- ------------ --------- ---------- ------------ fabric VTEP-Core-1 Core-1 Core-1 40.40.1.1 :: on fabric VTEP-Core-3 Core-3 Core-3 40.40.3.1 :: on fabric VTEP-Core-2 Core-2 Core-2 40.40.2.1 :: on
!Vtep-show
CLI (network-admin@Core-2*) > vtep-showscope name location vrouter-name ip virtual-ip mac-learning ------ ----------- -------- ------------ --------- ---------- ------------ fabric VTEP-Core-1 Core-1 Core-1 40.40.1.1 :: on fabric VTEP-Core-3 Core-3 Core-3 40.40.3.1 :: on fabric VTEP-Core-2 Core-2 Core-2 40.40.2.1 :: on
19 | Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail
NOTE: Switch <switch-name> command can be also used to login into another switch in the fabric, but in this configuration will login individually and use the <switch-local> to configure locally.
!Vrouter-interface show
CLI (network-admin@Core-2) > vrouter-interface-show format nic,ip,vlan,nic-state,mtu vrouter-name nic ip vlan nic-state mtu ------------ --------- ---------------- ---- --------- ---- Core-2 eth0.4092 10.1.2.2/30 4092 up 9216 Core-2 eth1.4091 10.2.3.1/30 4091 up 9216
!Vrouter-ospf-interface-show
CLI (network-admin@Core-2) > vrouter-ospf-interface-show vrouter-name nic l3-port area mtu state hello-due neighbor ------------ --------- ------- ------- ---- -------------- --------- -------- Core-2 eth0.4092 61 0.0.0.0 9216 point-to-point 0.068s 1 Core-2 eth1.4091 57 0.0.0.0 9216 point-to-point 0.068s 1
!Vrouter-ospf-neighbor-show
CLI (network-admin@Core-2) > vrouter-ospf-neighbor-show format neighbor,pri,State,address,interface,vlan
!Tunnel-showCLI (network-admin@Core-1*) > tunnel-show format name type,local-ip,remote-ip, state,name type local-ip remote-ip state ------------------------------- ----- --------- --------- ----- auto-tunnel- 40.40.1.1_40.40.3.1 vxlan 40.40.1.1 40.40.3.1 ok auto-tunnel- 40.40.1.1_40.40.2.1 vxlan 40.40.1.1 40.40.2.1 ok
!Tunnel-showCLI (network-admin@Core-3*) > tunnel-show format name type,local-ip,remote-ip, state,name type local-ip remote-ip state ------------------------------- ----- --------- --------- ----- auto-tunnel- 40.40.3.1_40.40.1.1 vxlan 40.40.3.1 40.40.1.1 ok auto-tunnel- 40.40.3.1_40.40.2.1 vxlan 40.40.3.1 40.40.2.1 ok
!Tunnel-showCLI (network-admin@Core-2*) > tunnel-show format name type,local-ip,remote-ip, state,name type local-ip remote-ip state ------------------------------- ----- --------- --------- ----- auto-tunnel- 40.40.2.1_40.40.1.1 vxlan 40.40.2.1 40.40.1.1 ok auto-tunnel- 40.40.2.1_40.40.3.1 vxlan 40.40.2.1 40.40.3.1 ok
!Trunk-showCLI (network-admin@Core-1*) > trunk-show name vxlan-loopback-trunk format trunk-id,switch,name,ports,vxlan-termination,statustrunk-id switch name ports vxlan-termination status -------- ------ -------------------- ----- ----------------- ---------------- 397 Core-1 vxlan-loopback-trunk 69 off up,stp-edge-port
!Trunk-showCLI (network-admin@Core-3*) > trunk-show name vxlan-loopback-trunk format trunk-id,switch,name,ports,vxlan-termination,statustrunk-id switch name ports vxlan-termination status -------- ------ -------------------- ----- ----------------- ---------------- 397 Core-3 vxlan-loopback-trunk 69 off up,stp-edge-port
!Trunk-showCLI (network-admin@Core-2*) > trunk-show name vxlan-loopback-trunk format trunk-id,switch,name,ports,vxlan-termination,statustrunk-id switch name ports vxlan-termination status -------- ------ -------------------- ----- ----------------- ---------------- 397 Core-2 vxlan-loopback-trunk 69 off up,stp-edge-port
Step 8 : [Optional] Reserve a port for VXLAN loopback / packet recirculation This step is required only if the ACF switches are not part of the S52xx or S41xx family.
It is necessary to dedicate a front-panel port for handling broadcast and unknown multicast (BUM) tra�ic that needs to be sent across the mesh of VXLAN tunnels. In fact, an internal physical loopback facility is required to recirculate this tra�ic in order to perform Head End Replication (HER) and send a unicast packet to each remote VTEP associated to the same Layer 2 extended segment or VNI.
NOTE: Certain hardware platforms have internal ports available for this task, so this step is not necessary with those devices.
Use the following sample configuration for Core-1, Core-2 (witness) and Core-3
Validation Steps:Use below command to verify trunk-show on any Core-1, Core-2 (witness) or Core-3
Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail | 20
Step 9 : Configure stretched VLANs with VxLAN (VNI) & Auto-VxLAN VxLAN overlay networks, each stretched VLAN is mapped to a VNI number representing a VxLAN segment.
Create a VLAN either by specifying a VLAN ID and auto-vlxan identifier for the tunnel with no ports tagging,
NOTE: VxLAN (VNI) ID will auto map with VTEP.
Use the following commands with fabric scope for configuring the stretched VLANs with associated anycast gateway addresses and the associated VNIs for Core-1, Core-2 (witness) and Core-3
Figure 3 Stretched VLANs (layer-2) details
21 | Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail
Configuring the Overlay network involves the configuration of (at least) the following base features:Detailed Overlay (Day-1/2) configurations steps:
Configure stretched VLANs with VxLAN (VNI) & Auto-VxLAN
Configure routing and stretched subnets across sites
Step 9Step 10
Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail | 10
Core-1 Core-3 Core-2 (witness)
!Run commands locally on a switch by using theCLI (network-admin@Core-1) > switch-local
Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail | 22
Core-1 Core-3 Core-2 (witness)
!Create Layer-3 GW interfaceCLI (network-admin@Core-1)* > vrouter-interface-add vrouter-name Core-1 nic eth1.1312 ip 10.13.12.254/24 vlan 1312 mtu 9216
CLI (network-admin@Core-1)* > vrouter-interface-add vrouter-name Core-1 nic eth1.1412 ip 10.14.12.254/24 vlan 1412 mtu 9216
CLI (network-admin@Core-1)* > vrouter-interface-add vrouter-name Core-1 nic eth3.1111 ip 172.11.11.254/24 vlan 1111 mtu 9216
!Create Layer-3 GW interfaceCLI (network-admin@Core-3)* > vrouter-interface-add vrouter-name Core-3 nic eth0.1312 ip 10.13.12.254/24 vlan 1312 mtu 9216
CLI (network-admin@Core-3)* > vrouter-interface-add vrouter-name Core-3 nic eth0.1412 ip 10.14.12.254/24 vlan 1412 mtu 9216
!Create Layer-3 GW interfaceCLI (network-admin@Core-2)* > vrouter-interface-add vrouter-name Core-2 nic eth4.1112 ip 172.11.12.254/24 vlan 1112 mtu 9216
CLI (network-admin@Core-2)* > vrouter-interface-add vrouter-name Core-2 nic eth0.1311 ip 10.13.11.254/24 vlan 1311 vlan-type public if data mtu 9216 mtu 9216
Use the following sample configuration using fabric scope form any Core-1, Core-2(witness) and Core-3
Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail | 24
Validation Steps:Use the command below to verify vrf-show and subnet-show from any Core-1, Core-2(witness) or Core-3
Step 9 : Routing between Stretched VLAN across sites vSAN VLANs between site 1 and site 2 are stretched, however vSAN VLAN on the witness site is routed to site 1 and site 2.
L3VNI (VRF) is used to route between L2VNIs (VLAN-1111, 1112, 1311, 1312, 30 & 40). The ingress VTEP will perform IP destination routing lookup. The encapsulated VXLAN tra�ic is sent between VTEPs where it is associated with L3VNI the VRF-1 in which VLAN-1111, 1112, 1311, 1312, 30 & 40 reside.
Figure 4 Layer-3 (L3VNI) details
25 | Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail
To provide inter-vlan routing, Pluribus supports the recirculation technique with the dedicated for any fabric packets that need to be routed between two hosts in di�erent VNI-mapped VLANs.
Use the following sample configuration using switch-local form Core-1, Core-2(witness) and Core-3
Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail | 26
Pluribus Networks’ UNUM is a unified fabric management, automation and analytics platform that enables administrators to significantly reduce fabric deployment times, execute everyday fabric management lifecycle tasks and conduct e�ective tra�ic analysis, all from the centralized graphical interface.
A single instance of Pluribus UNUM can seamlessly scale to visualize multiple sites data center interconnected in a single unified topology view.
Pluribus UNUM – Unified Automation, Management and Analytics
Validation Steps:Use below command to verify VxLAN-loopback-trunk from any Core-1, Core-2(witness) or Core-3
UNUM provides day-0 automation capabilities with respect to underlay fabric deployments and playbooks for Layer-2/3 and Overlay configuration.
Depending on the actual topology connections between devices are represented in the topology map in that case UNUM may display a topology as shown in the following example.
27 | Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail
How UNUM simplifies stretched VLANs configuration across Multi-sites In this example, we are going to extend VLAN-30 (Eng) across site-A to site-B over VxLAN.
Select the Fabric from the le� side navigation bar to create VLAN across all switch in fabric or select an individual Switch from the fabric to create only the VLAN for that switch.
Figure 5 UNUM Deploy, Manage, Visualize Multi-Sites (DC-I) from single from pane of glass
NOTE: UNUM requires all hosts, switches, and servers to run LLDP if they are to appear in the Topology.
Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail | 26
Click on Create a VLAN by id
Enter the parameter settings for: • Switch – select any switch in fabric • Scope – select fabric scope to extend VLAN across multi-sites • ID – configure VLAN ID • Description – Eng VLAN • Ports – None
Check the Stats – VLAN statistics box to receive VLAN statistical information.
Step 1: Click on Manager → Layer 2 → Manage VLAN
Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail | 28
NOTE: We can add VLAN to port now, but in this example we will cover in next-step.
• VXLAN – VXLAN 1000030 mapped to VLAN for tunnel • Select Auto-VXLAN-Mode – VXLAN encapsulation mode
Additional fields are configured by clicking on the icon for Additional Fields. Enter the parameter settings for:
29 | Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail
Click Save to continue
Step 2: Click on VLAN Ports to add VLAN to port
Then select the desired switch (Core-1) from the fabric and the dashboard updates automatically with VLAN PORT settings.
The dashboard displays a list of existing VLAN Ports and a drop-down selection list. The dashboard displays a list of existing VLAN Ports entries. Parameters include: ports and untagged-ports.
Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail | 30
Select the VLAN using the up / down arrows in each of the columns as required.
Interactive Port Selector The Interactive Port Selector widget displays a grid accurately representing switch port availability and shows the number of physical ports on a given switch. Gray depicts down ports and white, up ports. Selected ports appear in blue.
Click the Edit icon to display and select a range of ports. Click the desired ports and click Apply to continue.
Dell Technologies + Pluribus Networks | Dell EMC Data Center Interconnect Fabric Design Guide for OS10 and Stretched VxRail | 28
Follow the above steps by choosing selecting Site-B Core-3 switch and add the physical ports.
Click on Apply and Save to continue.
More details can find on www.pluribusnetworks.com/UNUM
Pluribus Networks, Inc. www.pluribusnetworks.com
Networking from Dell Technologieswww.DellTechnologies.com/Networking
The following resources provide support and additional information: • The Dell Demo Center platform provides several SmartFabric OS10 virtual labs. These vLabs provide sample OS10 based configurations that help the user get familiar with Dell EMC SmartFabric OS10 CLI and features. • The Dell Technologies online Support page, which focuses on meeting customer needs with proven services and support. • Dell Community, which is an online technical community where IT professionals have access to numerous resources for Dell EMC so�ware, hardware, and services. • Dell Technologies online support Knowledge Base, which provides storage solutions technical documents and expertise that helps to ensure success on Dell EMC Storage platforms. • The Networking Solutions Info Hub, which provides access to networking guides. • Pluribus Networks technical documentation for Netvisor ONE, the Adaptive Cloud Fabric and UNUM Management Platform. • The Pluribus partner portal. • Additional Pluribus support inquiries can be sent to [email protected]