Suresh Thiru Sridhar Subramanian NET1949BU VMworld 2017 - NET1949BU Seamless Network Connectivity for Virtual and Bare-metal Workloads with NSX VMworld 2017 Content: Not for publication or distribution
Suresh ThiruSridhar Subramanian
NET1949BU
VMworld 2017 - NET1949BU
Seamless Network Connectivity for Virtual and Bare-metal Workloads with NSX
VMworld 2017 Content: Not fo
r publication or distri
bution
• This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.
• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
• Technical feasibility and market demand will affect final delivery.
• Pricing and packaging for any new technologies or features discussed or presented have not been determined.
Disclaimer
2
VMworld 2017 Content: Not fo
r publication or distri
bution
Agenda
1 NSX Everywhere
2 Bare-metal Use Cases
3 NSX Solutions
4 Design Considerations and Best Practices
3
VMworld 2017 Content: Not fo
r publication or distri
bution
Works across Hypervisors, Clouds,
Application frameworks
Infrastructure
independent
Security wrapped around the VM, container, microservice
Application Drives InfrastructureWhat does this mean for Networking and Security?
VMworld 2017 Content: Not fo
r publication or distri
bution
Evolution of Server Computing
Explosion of VM and
mobility led to network
virtualization
Native container network
with multi-tenancy,
micro-segmentation, and
common tools for day 2
operations
Full visibility and control
with consistent operation
across private and public
cloud
Seamless connectivity and
security for physical
workloads (Legacy App, DB,
Storage, Security Appliance)
Container Workload
Dynamic Static
Bare-metal Workload
Public Cloud Workload
VM Workload
Networking And Security Services
Introduces new Networking and Security requirements
VMworld 2017 Content: Not fo
r publication or distri
bution
Container Workload
Bare-metal Workload
Public Cloud Workload
VM Workload
New Silos Leads to Operational InefficienciesChallenges: Different technology stacks, processes, teams, and expertise
VMworld 2017 Content: Not fo
r publication or distri
bution
• Uniform Networking & Security services across private & public clouds
• Single pane of glass management
• Supports any physical network infrastructure
PV FW
Container Workload
Bare-metal Workload
Public Cloud Workload
VM Workload
NSX Everywhere: A Platform For All Workloads
VMworld 2017 Content: Not fo
r publication or distri
bution
8
Cloud
Consumption
Control Plane
Management
Plane
Data Plane
Hypervisor
Virtual Switch
SW RT FW
Container on
Hypervisor or
Bare-metal*
Virtual Switch
SW RT FW
Guest VM in
Public Cloud
Virtual Switch
SW RT FW
NSX Edge
Services
Edge Router
RT FW LB VPN
ESXi
Bare-metal Server
PV
Virtual
Switch
OVSDB
TOR
NSX Manager
NSX Controller
* NSX support for containers on bare-metal is planned for future release
NSX Architecture Extended to Support All WorkloadsCentral Management to manage networking and security policies
VMworld 2017 Content: Not fo
r publication or distri
bution
NSX Platform Journey
Public
Cloud
Multi
Hypervisor ContainersvSphere Baremetal
Delivered entire
networking and
security services in
software for vSphere
Extended NSX to
KVM and Openstack
Integrated NSX with
PaaS and Container
orchestrator for
cloud-native apps
Extended NSX to
native cloud
workload and cloud
services
NSX benefits extended to
bare-metal
VMworld 2017 Content: Not fo
r publication or distri
bution
Focus For Rest of The Session
Public
Cloud
Multi
Hypervisor ContainersvSphere Baremetal
Refer to
NET1510BU -
Introduction to NSX-T
Architecture
Refer to
NET1535BU -
NSX Design—
Reference Design for
SDDC
with NSX and vSphere
Refer to
MMC2046BU -Using VMware NSX for
Enhanced Networking and
Security for AWS Native
Workloads
Refer to
CNA1091BU -One-Stop Container
Networking: Cloud Foundry,
Kubernetes, Docker, and More
This SessionVMworld 2017 Content: Not fo
r publication or distri
bution
Bare-metal Use Cases
VMworld 2017 Content: Not fo
r publication or distri
bution
Use Case 1: Integration of non-Virtualized Workloads
• Typically necessary for integrating a non-virtualized appliance
• L2 as well as L3
• A gateway takes care of the on ramp/off ramp
12
Overlay-backed Workloads
OverlayVLAN
Virtual To PhysicalGATEWAY
PhysicalWorkloads
VMworld 2017 Content: Not fo
r publication or distri
bution
Use Case 2: Migration Of Physical To Virtual
• Physical workloads migrated in phases to virtual form factor
• Temporary, bandwidth not critical
BEFORE DURING MIGRATION AFTER
3 Physical Workloads
Virtual To PhysicalGATEWAY
OverlayVLAN
2 Physical Workloads
1 VirtualWorkloads
Virtual To PhysicalGATEWAY
OverlayVLAN
0 Physical Workloads
3 VirtualWorkloads
Virtual To PhysicalGATEWAY
OverlayVLAN
0 VirtualWorkloads
VMworld 2017 Content: Not fo
r publication or distri
bution
Use Case 3: Migration Of VLAN-Backed Virtual Workloads
• VLAN-backed Virtual workloads Migrated in phases to Overly-backed Virtual workloads
• Temporary, bandwidth not critical
BEFORE DURING MIGRATION AFTER
3 VLAN-backed
Workloads
Virtual To PhysicalGATEWAY
OverlayVLAN
2 VLAN-backed
Workloads
1 Overlay-backedWorkload
Virtual To PhysicalGATEWAY
OverlayVLAN
0 VLAN-backed
Workloads
3 Overlay-backedWorkload
Virtual To PhysicalGATEWAY
OverlayVLAN
0 Overlay-backedWorkloads
VMworld 2017 Content: Not fo
r publication or distri
bution
NSX Solutions
VMworld 2017 Content: Not fo
r publication or distri
bution
Guiding Principle: Routing Vs Bridging
Routed Connectivity to Physical workloads
• Standard Routing protocols (OSPF and BGP)
• ECMP Scale-out, failure isolation with routing
Bridged Connectivity to Physical workloads
• Flat Broadcast domain limiting size and scale
• Single Active bridge for a VXLAN-VLAN pair
16
Route when you Can, Bridge when you Must !
Overlay VLAN
Virtual To PhysicalGATEWAY
Physical Workloads
VLAN-backed Virtual workload
L2
…
Overlay VLAN
Physical Workloads
L3
L2L2 VMworld 2017 Content: Not fo
r publication or distri
bution
Web1 App1
External
Network
Oracle ExadataPhysical Server
L3
Edge
Services
GatewayVPN
Distributed Logical Router
NET1416BU NSX-T Logical Routing
Practical Example with Exadata Server in separate L3 Subnet
• Web and App Tier in Overlay
• App Tier and Exadata Are in different Subnets
• Edge Gateway provides routed North-South to physical network
• Performance & Scale with ECMP
• Most commonly deployed by Oracle & Enterprise
VMworld 2017 Content: Not fo
r publication or distri
bution
Web1 App1
External
Network
L3
Edge
Services
GatewayVPN
Distributed Logical Router
Virtual To PhysicalGATEWAY
Oracle ExadataBare-metal Server
App tier and Exadatashare the same subnet
Practical Example when Exadata and APP Server in same Subnet
• Web and App Tier in NSX Overlay
• App to Exadata Bridging via the “Virtual to Physical Gateway” realizable in Two ways
1. NSX Software Bridge Design in a separate VM
2. HW Gateway Design by enabling Top-Of-Rack Network switch to provide the function
VMworld 2017 Content: Not fo
r publication or distri
bution
Summary of Bridging Options For Virtual To Physical Connectivity
19
SW Agent*
Pros
• Common NSX Stack for workload connectivity across Bare-metal servers, Hypervisors, Containers and Public Cloud
• Paves way for security Of workload at OS layer
Cons
• Legacy OS versions not supported
*This is NOT a shipping option today and is in exploration stage
SW Bridge
Pros
• Independent Of Physical Switch
HW or SW
• Scale-out with little investment
• High performance VXLAN to VLAN
gateway in hypervisor kernel
Cons
• Density of Physical workloads mapping to different VxLAN –VLAN pairs
HW Gateway
Pros
• Offers Higher Bandwidth and port-density for workloads
• Useful in racks where no Hypervisor can be deployed
• Fast Failover and Redundancy features from HW Vendors
Cons
• Reduces Virtualization benefits by introducing Hardware dependency
VMworld 2017 Content: Not fo
r publication or distri
bution
Software Bridge - Recorded Demo
192.168.1.10192.168.1.20
VLAN 16
Overlay-backed Workload
PhysicalWorkload
OverlayVLAN
NSX SW Bridge hosted in a
Hypervisor Instance
VMworld 2017 Content: Not fo
r publication or distri
bution
Software Bridge DEMO With NSX-T
VMworld 2017 Content: Not fo
r publication or distri
bution
Hardware Bridge – Recorded Demo
22
172.16.10.10
VLAN 160
Ethernet18
Database Logical Switch
VNI 5000
PhysicalWorkload
NSX Controllers10.114.221.235-237
10.114.221.196Arista Switch as Hardware Gateway10.114.211.105
HV1
VTEP
Overlay-backed Workload
172.16.10.11
VMworld 2017 Content: Not fo
r publication or distri
bution
Up Next: Configuration of the Arista Hardware Gateway
23
Configuration of the Replication Cluster
VMworld 2017 Content: Not fo
r publication or distri
bution
Up Next: Registration of the Arista Hardware Gateway into NSX
24
Configuration of the Arista Hardware Gateway
VMworld 2017 Content: Not fo
r publication or distri
bution
Up Next: Binding a Logical Switch to a Physical Port/VLAN
25
Registration of the Arista Hardware Gateway into NSX
VMworld 2017 Content: Not fo
r publication or distri
bution
26
Binding a Logical Switch to a Physical Port/VLAN
VMworld 2017 Content: Not fo
r publication or distri
bution
Customer Case Study
• Deployment Region: Global
• Deployment Scale :
– 1st Phase: 26 Hosts
– 2nd Phase: 30 Hosts in 2nd DC
• Management: Log Insight
• NSX version: 6.2.3
• 3rd Party Integration: Arista Hardware VTEP
• NSX Features Used:
– HW Gateway
– DFW
27
Large Electronics Manufacturing Company
VMworld 2017 Content: Not fo
r publication or distri
bution
Network Topology for Case Study
28
Key Takeaways
• Use Case:- Shared storage service
with controlled access for compute
Rack VMs.
• Problem: Compute rack VMs need
shared storage access from a non-
virtualized disk.
• Conditions: VMs cannot talk to each
other
• Solution:
– HW Gateway Solution used to
bridge VM traffic to VLAN X on
which storage disk attached to get
shared service
– DFW used to prevent VM to VM
communication
Storage
Disk
VLAN XDatabase RacksCompute Racks
VXLAN
VLAN
VXLAN ID 500X
Arista HW Gateway
VMworld 2017 Content: Not fo
r publication or distri
bution
Design Considerations and Best Practices
VMworld 2017 Content: Not fo
r publication or distri
bution
Software Bridge vs. Hardware Gateway
30
• A single bridging instance per Logical Switch
• Bandwidth limited by single bridging instance
• L2 network must be extended to reach all the
physical devices
• Several Hardware Gateways can be deployed at several
locations simultaneously
• With Hardware Gateways, VLANs can be kept local to a
rack and don’t need to be extended
VXLAN
VLAN
VLAN 10 VLAN 20VLAN 10 VLAN 10Database Racks
VLAN extended
between racks
Non-virtualized
devices (part of the
same L2 segment)
Database RacksCompute Racks Compute Racks
VMworld 2017 Content: Not fo
r publication or distri
bution
Redundancy considerations With Software Bridge
31
PhysicalWorkload VLAN Y
Control VM-0
SW Bridge
Hypervisor
ACTIVE
Control VM-1
SW Bridge
Hypervisor
STANDBY VirtualWorkload
(Logical Switch X)
PhysicalWorkload VLAN Y
Control VM-0
SW Bridge
Hypervisor
DOWN
Control VM-1
SW Bridge
Hypervisor
ACTIVE VirtualWorkload
(Logical Switch X)
• SW Bridge functionality for a given VLAN/VXLAN par can only be active on a single Hypervisor
Recommendation:
• Introduce redundancy by selecting Standby Hypervisor that will host SW Bridge and take over upon failure
• Standby is determined by location of Control VM that the User configures
VMworld 2017 Content: Not fo
r publication or distri
bution
Hardware Based Solution re-Introduces Hardware Dependency
32
The Hardware Based model invalidates the benefits of virtualization
VXLAN
VLAN
VLAN 10 VLAN 20Database Racks
Compute Racks
Physical switch HW and SW versions need to be certified with NSX
Hardware Gateway does not natively support Distributed Routing or Distributed Firewall
VMworld 2017 Content: Not fo
r publication or distri
bution
Redundancy Consideration With Hardware Gateway
• The OVSDB based mechanism is currently not aware of any form of redundancy
• Several Hardware Gateways can be active for the same Logical Switch
• A backdoor connection could result in a loop
• Recommendation: Only connect hosts to the Hardware Gateway
33
VLAN 10Hardware
Gateway1
LS VNI:5000
VLAN 10
Network
Switch in
backdoorPhysical
workload 1
Physical
workload 2
Loop
Hardware
Gateway2
VMworld 2017 Content: Not fo
r publication or distri
bution
Best Practices For Redundancy With Hardware Gateway
34
• Active/Standby uplink
• No L2 connection must be made between switches
Host Based Redundancy
VLAN 10
Hardware
Gateway1
Hardware
Gateway2
LS VNI:5000
VLAN 10
Physical
workload
VMworld 2017 Content: Not fo
r publication or distri
bution
Best Practices For Redundancy With Hardware GatewayPort Channel Based Redundancy
Data Plane: Physical View
HV
Several physical Hardware
Gateways presented as a
single logical one to NSX
Data Plane: Logical View
HV
Most Hardware vendors offer a distributed port channel based solution
VMworld 2017 Content: Not fo
r publication or distri
bution
Security Considerations For Bare-metal Workloads
36
• Distributed or Edge FW can regulate V-P traffic
• NSX integration with Partner FW manager can regulate V-P traffic closest to the Physical workload
PhysicalWorkload
VPN
Edge Firewall
PhysicalNetwork
STOP
STOP
Virtual To Physical communication
STOP
PhysicalWorkload
VPN
Edge Firewall
NSX Manager
Security Groups
PhysicalNetwork
STOP
Eg.Partner Firewall
Mangement Console
STOP STOP
VMworld 2017 Content: Not fo
r publication or distri
bution
Security Considerations For Bare-metal Workloads
37
IPFIX (from
vSphere)
NetFlow (from
physical)
Search, Analytics and Micro-segmentation Modeling
Across Virtual, Physical & Cloud
Public Cloud VirtualPhysical
AWS Flow
Logs
Physical To Physical Flows Analysis & ACL Recommendations*
*This is NOT a shipping option today and is currently under development
• vRealize Network Insight (vRNI) leverageable analyzing flows from virtual, physical (Netflow) and cloud. V-to-V, V-to-P and P-to-P
• Micro-segmentation models, application tier definition and firewall/ACL rules recommendation for physical end points / IPs
• Scale out architecture for large scale flow collection. No agents.
PAR4377BU NSX Advanced Security
VMworld 2017 Content: Not fo
r publication or distri
bution
Key Takeaways
38
PV FW
ContainerWorkload
Bare-metalWorkload
PublicCloudWorkload
VMWorkload
• Route when you can and Bridge only when you must
• Recommended Order Of Bridging Solutions For Bare-metal workloads
• SW Bridge
• Hardware Gateway
• Secure Bare-metal servers with native NSX solution or with NSX integrated partner solution
VMworld 2017 Content: Not fo
r publication or distri
bution
Relevant Sessions and References
▪ Sessions
▪ References
NSX for vSphere Network Virtualization Design Guide (Ver 3.0)
https://communities.vmware.com/docs/DOC-27683
39
NET1535BU
NET1536BU
Reference Design for SDDC with NSX and vSphere: Part 1 & 2
NET1863BU NSX-T Advanced Architecture Concepts
NET1416BU NSX-T Logical Routing
CNA1091BU One-Stop Container Networking: Cloud Foundry, Kubernetes, Docker,
and More
MMC2046BU Using VMware NSX for Enhanced Networking and Security for AWS
Native Workloads
VMworld 2017 Content: Not fo
r publication or distri
bution
VMworld 2017 Content: Not fo
r publication or distri
bution
VMworld 2017 Content: Not fo
r publication or distri
bution