VALIDATED REFERENCE DESIGN GUIDE DEPLOYING EBGP EVPN … · • EBGP is enabled between the border leaf switches (Leaf2a/Leaf2b) and the DC core switches to advertise the server subnets
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Notices The information contained herein is subject to change without notice. The only warranties for Hewlett Packard Enterprise products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. Hewlett Packard Enterprise shall not be liable for technical or editorial errors or omissions contained herein. Confidential computer software. Valid license from Hewlett Packard Enterprise required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license. Links to third-party websites take you outside the Hewlett Packard Enterprise website. Hewlett Packard Enterprise has no control over and is not responsible for information outside the Hewlett Packard Enterprise website.
This document provides guidance on deploying an AOS-CX powered L3 Spine/Leaf EBGP EVPN (dual-AS) VXLAN with VSX Centralized L3 Gateway architecture using NetEdit.
The AOS-CX switches to be deployed will depend on interfaces, scale and features required. Aruba networks provides a diverse product portfolio to meet different customer requirements.
NetEdit empowers IT teams to orchestrate multiple switch configurations with intelligent capabilities including search, edit, validation (including conformance checking), deployment and audit. Using NetEdit, the network admin can configure and validate multiple switches simultaneously, while specifying unique settings for each switch.
The architecture as shown in Figure 1 provides the following benefits: • Provides a loop free L3 underlay network fabric and overlay network for East-West L2/L3 connectivity between
racks (Rack 1 = Leaf1a/Leaf1b, Rack 2 = Leaf2a/2b) with all links active • Provides the capability to add more spine switches in future to provide additional fabric bandwidth • Distributed scale out control plane • Provides maximum network High Availability and uptime with VSX Live Upgrades • Provides built in support for Network Analytics Engine • Supports multi-tenancy via VXLAN and VRFs • Provides a centralized firewall inspection chokepoint for traffic entering/leaving the POD/Zone
Figure 1. POD/Zone DC architecture
This architecture can be replicated to other PODs/Zones to create separate failure domains. A L3 DC core will connect all the zones together, each POD/Zone is assigned multiple AS#s and EBGP is recommended as the routing protocol to route traffic between the PODs/Zones. The POD/Zone architecture enables each POD/Zone to have different architectures if
desired, e.g. Zone12 requires a simple collapsed core while Zone1 requires a L3 Spine/Leaf EBGP EVPN (dual-AS) VXLAN with VSX Centralized L3 Gateway.
For the sake of completeness, configs for the L3 DC Core are also included to show the routes learnt from the border leaf switches in the DC POD/Zone.
In the DC core switches: • 200.200.200.0/24 is used to simulate an external subnet the DC-Core has learnt • Only one DC-Core switch is shown in this guide, in a production network redundant DC-Core switches placed in the
same AS# should be deployed for additional path redundancy
In the POD/Zone: • EBGP (IPv4 address family) is enabled within the POD/ZONE between the leaf and spine switches
o Using the directly connected interfaces o For underlay routing, all loopbacks and /31 links are advertised via EBGP o bgp bestpath as-path multipath-relax is enabled for ECMP
• EBGP (EVPN address family) is enabled within the POD/ZONE between the leaf and spine switches o Using Lo0 with EBGP multi-hop o For EVPN route exchange
• Spines are assigned to the same AS#, e.g. AS#65101, they do not need to peer to each other o next-hop-unchanged is enabled within the EVPN address family on the spines to ensure the VTEP next
hop (tunnel Lo1 on the leaf switches) do not change • Leafs across the racks are assigned to the same AS#, e.g.
o Leaf1a/Leaf1b in rack 1, Leaf2a/Leaf2b in rack 2 are in AS#65001 • IBGP is enabled between VSX switches in a rack in the same AS# for underlay connectivity • BGP “allowas-in” is required on leafs in order to accept routes from other leafs in the same AS • Leaf switches in a rack do not peer to other leaf switches in other racks • Leaf1a/Leaf1b in rack 1 function only as L2 VTEPs, there can be other racks with similar L2 VTEPs if desired • L2 connectivity between racks with L2 VTEPs is achieved via direct VXLAN tunnels • Redundant VSX ISL links in a LAG should be used for maximum VSX availability • Dedicated VSX Keepalive link is recommended between VSX switches in a rack • VSX system-mac is enabled so that LACP peers think they are connected to the same remote switch • VSX logical VTEP in a leaf rack share an anycast Lo1 IP address • Split-recovery is disabled for VSX to ensure only the primary VSX switch will keep the logical VTEP up during a VSX
split • Single homed servers should only be connected to the primary VSX switch • Should uplinks fail on any leaf switch, VLAN4000 and IBGP will be used as the transit link to reroute traffic to the
redundant leaf switch within the rack • The border leaf switches (Leaf2a/Leaf2b) in rack 2 function as centralized L3 gateway VSX switches and performs
all overlay L3 routing between subnets • EBGP is enabled between the border leaf switches (Leaf2a/Leaf2b) and the DC core switches to advertise the
server subnets (11.1.1.0/24, 12.1.1.0/24 etc) out, the server subnets should be summarized if possible • “bgp fast-external-fallover” is recommended on the all switches for fast failover between different AS# • Large MTU is enabled to support applications that require it • The server facing ports should be set to maximum MTU supported by the server (e.g. 9000) if large MTU
applications need to be transported across the VSX switches • Active-gateway is required on the centralized L3 gateway for default gateway redundancy • On the leaf switches, southbound lag 10 is used for server connectivity and enabled with LACP fallback to allow an
active LACP interface to establish a Link Aggregation (LAG) before it receives LACP PDUs from its peer, this feature is useful in environments if Preboot Execution Environment (PXE) Servers are used
The IP address assignments and interface details used in this guide are shown in Figure 2.
Figure 2. Interface and IP address details
1) INITIAL SETUP (OOB, INITIAL CONFIGS)
In the DC, we recommend switches connect their management ports to a separate Out Of Band (OOB) management network as shown in Figure 3, this allows the switches to be manageable if there is an issue with In Band network connectivity.
Once the switches are configured and physically connected, ensure NetEdit has IP connectivity to switch management IPs and add all devices into NetEdit [Devices -> Action -> Discover Devices]
2) MODIFY CHANGE VALIDATION SETTINGS
To help with change validation, you can add or modify the change validation commands used by NetEdit.
This is done in NetEdit [Settings -> Validation -> Change Validation -> Command Scripts]
The screenshots below showcase the verification commands used in this guide.
interface 1/1/2 no shutdown mtu 9198 description Leaf1b ip mtu 9198 ip address 192.168.2.5/31 interface 1/1/3 no shutdown mtu 9198 description Leaf2a ip mtu 9198 ip address 192.168.2.9/31 interface 1/1/4 no shutdown mtu 9198 description Leaf2b ip mtu 9198 ip address 192.168.2.13/31
And right click to modify IPs assigned to each switch
You can click on “Change Validation” to compare configs.
As the leaf switches are not connected, you will not be able to validate BGP neighbor peering.
After validation, you can choose to “COMMIT” to save the desired configs or “ROLLBACK” to revert configs before the configs were deployed to make further desired changes.
interface lag 1 no shutdown description VSX ISL LAG no routing vlan trunk native 1 tag vlan trunk allowed all lacp mode active interface 1/1/48 no shutdown mtu 9198 description VSX ISL lag 1
Add VSX (select 1 pair at a time)
vsx system-mac 00:00:00:00:02:12 inter-switch-link lag 1 role primary keepalive peer 10.1.2.1 source 10.1.2.0 no split-recovery
Select leafs in a different rack to modify the shared loopback 1 IP.
Configure uplinks towards spine switches
interface 1/1/49 no shutdown mtu 9198 description Spine1 ip mtu 9198 ip address 192.168.2.0/31 interface 1/1/50 no shutdown mtu 9198 description Spine2 ip mtu 9198 ip address 192.168.2.2/31
And right click to modify IPs assigned to each switch
Add VLANs and configure transit VLAN between Leaf switches
vlan 11-13,4000 ! interface vlan4000 description Transit VLAN ip mtu 9198 ip address 192.168.3.0/31
And right click to modify IPs assigned to each switch
Configure EBGP IPv4 (advertise all /31 and loopbacks) and EVPN (using EBGP multi-hop Lo0) towards Spine switches, allowas-in is required in order to accept routes from other leafs in the same AS.
Configure VRFs on border “Leaf2a/Leaf2b” towards DC Core, this RT policy allows “VRFa”, “VRFb” to access VRF “External” but does not allow connectivity between “VRFa” and “VRFb”.
Configure the centralized L3 gateways on “Leaf2a/Leaf2b” with VRFs, IPs and active-gateway.
interface vlan11 vrf attach VRFa ip mtu 9198 ip address 11.1.1.2/24 active-gateway ip mac 00:00:00:00:01:01 active-gateway ip 11.1.1.1 interface vlan12 vrf attach VRFa ip mtu 9198 ip address 12.1.1.2/24 active-gateway ip mac 00:00:00:00:01:01 active-gateway ip 12.1.1.1 interface vlan13 vrf attach VRFb ip mtu 9198 ip address 13.1.1.2/24 active-gateway ip mac 00:00:00:00:01:01 active-gateway ip 13.1.1.1 And right click to modify the IP address on each “Int VLAN”
Configure ports and IP addresses on “Leaf2a/Leaf2b” towards DC Core
interface 1/1/53 no shutdown vrf attach External description L3 Core ip address 10.1.1.0/31
And right click to modify IPs assigned to each switch
Configure these additional BGP configs on “Leaf2a/Leaf2b”, “fast-external-fallover” helps to speed up convergence towards DC-Core in a different AS, “bestpath as-path multipath-relax” allows ECMP paths to be used by BGP.
interface lag 10 multi-chassis no shutdown description Server no routing vlan trunk native 1 vlan trunk allowed 11-13 lacp mode active lacp fallback ! interface 1/1/51 no shutdown mtu 9198 description Server lag 10
After validation, you can choose to “COMMIT” to save the desired configs or “ROLLBACK” to revert configs before the configs were deployed to make further desired changes.
DC-Core1# sh bgp ipv4 u s VRF : default BGP Summary ----------- Local AS : 65100 BGP Router Identifier : 9.9.9.9 Peers : 2 Log Neighbor Changes : No Cfg. Hold Time : 180 Cfg. Keep Alive : 60 Neighbor Remote-AS MsgRcvd MsgSent Up/Down Time State AdminStatus 10.1.1.0 65001 106 124 00h:10m:50s Established Up 10.1.1.2 65001 108 131 00h:10m:31s Established Up DC-Core1# sh bgp ipv4 u Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, e external S Stale, R Removed Origin codes: i - IGP, e - EGP, ? - incomplete VRF : default Local Router-ID 9.9.9.9 Network Nexthop Metric LocPrf Weight Path *=e 11.1.1.0/24 10.1.1.0 0 100 0 65001 i *>e 11.1.1.0/24 10.1.1.2 0 100 0 65001 i *=e 12.1.1.0/24 10.1.1.0 0 100 0 65001 i *>e 12.1.1.0/24 10.1.1.2 0 100 0 65001 i *=e 13.1.1.0/24 10.1.1.0 0 100 0 65001 i *>e 13.1.1.0/24 10.1.1.2 0 100 0 65001 i *> 200.200.200.1/32 0.0.0.0 0 100 0 i DC-Core1# sh ip ro b Displaying ipv4 routes selected for forwarding '[x/y]' denotes [distance/metric] 11.1.1.0/24, vrf default via 10.1.1.2, [20/0], bgp via 10.1.1.0, [20/0], bgp 12.1.1.0/24, vrf default via 10.1.1.2, [20/0], bgp via 10.1.1.0, [20/0], bgp 13.1.1.0/24, vrf default via 10.1.1.2, [20/0], bgp via 10.1.1.0, [20/0], bgp
Zone1-Spine1# sh run Current configuration: ! !Version ArubaOS-CX GL.10.04.0001 !export-password: default hostname Zone1-Spine1 user admin group administrators password ciphertext AQBape!snip cli-session timeout 0 ! ! ! ssh server vrf mgmt ! ! ! ! ! vlan 1 interface mgmt no shutdown ip static 10.10.10.41/24 default-gateway 10.10.10.254 interface 1/1/1 no shutdown mtu 9198 description Leaf1a ip mtu 9198 ip address 192.168.2.1/31 interface 1/1/2 no shutdown mtu 9198 description Leaf1b ip mtu 9198 ip address 192.168.2.5/31 interface 1/1/3 no shutdown mtu 9198 description Leaf2a ip mtu 9198 ip address 192.168.2.9/31 interface 1/1/4 no shutdown mtu 9198 description Leaf2b ip mtu 9198 ip address 192.168.2.13/31 interface loopback 0 ip address 192.168.1.11/32 router bgp 65101
interface 1/1/47 no shutdown description VSX KA ip address 10.1.2.2/31 interface 1/1/48 no shutdown mtu 9198 description VSX ISL lag 1 interface 1/1/49 no shutdown mtu 9198 description Spine1 ip mtu 9198 ip address 192.168.2.8/31 interface 1/1/50 no shutdown mtu 9198 description Spine2 ip mtu 9198 ip address 192.168.2.10/31 interface 1/1/51 no shutdown mtu 9198 description Server lag 10 interface 1/1/53 no shutdown vrf attach External description L3 Core ip address 10.1.1.0/31 interface loopback 0 ip address 192.168.1.3/32 interface loopback 1 ip address 192.168.100.2/32 interface vlan11 vrf attach VRFa ip mtu 9198 ip address 11.1.1.2/24 active-gateway ip mac 00:00:00:00:01:01 active-gateway ip 11.1.1.1 interface vlan12 vrf attach VRFa ip mtu 9198 ip address 12.1.1.2/24 active-gateway ip mac 00:00:00:00:01:01 active-gateway ip 12.1.1.1 interface vlan13 vrf attach VRFb ip mtu 9198 ip address 13.1.1.2/24 active-gateway ip mac 00:00:00:00:01:01 active-gateway ip 13.1.1.1 interface vlan4000