1-855-MIKROTIK Network · PDF fileNetwork Architecture – Validated designs utilizing MikroTik in the Data Center ... •VPN Aggregation ... •MikroTik routers can be used in...

Network Architecture – Validated designs utilizing MikroTik in the Data Center P R E S E N T E D B Y: K E V I N M Y E R S , N E T W O R K A R C H I T E C T / M A N AG I N G PA R T NE R I P A R C H I T E C H S M A N AG E D S E R V I C E S

www.iparchitechs.com 1-855-MIKROTIK

24/7/365 MikroTik TAC Nationwide Private 4G LTE MPLS Proactive Network Monitoring Design / Engineering / Operations

www.iparchitechs.com

Background

• Kevin Myers

• 16 + years in IT/Network Engineering

• Designed and implemented networks in Service Provider, Enterprise, Ecommerce and Government environments

• Areas of Design Focus:

• MikroTik integration with multi-vendor networks

• Design of BGP/MPLS/OSPF Service Provider Triple-Play networks

• Design of large enterprise Data Center networks

• Certifications

• MTCINE #1409INE006

• Certified – CCNP, CCNA, MCP, MTCRE, MTCTCE, MTCNA

1-855-MIKROTIK

IP ArchiTechs Managed Services

• Exhibitor at 2013 and 2014 MUM – Please stop by our exhibitor booth and register to win an RC Helicopter !

• The first Carrier-Grade 24/7/365 MikroTik TAC (Technical Assistance Center)

• Three tiers of engineering support

• Monthly and per incident pricing available

• 1-855-MIKROTIK or support.iparchitechs.com

• AirMPLS - Private Nationwide 4G LTE MPLS backbone

• Partnership with Verizon Wireless - available anywhere in the Verizon service area

• Not Internet facing – privately routed over our MPLS infrastructure

• Multiple Deployment options to carry public and private traffic including L2 adjacency

• Proactive Monitoring / Ticketing / Change Control / IPAM (IP Address Management)

• Carrier-Grade Network Engineering / Design in large (100,000+ nodes) environments



Introduction – The MikroTik enabled Data Center

• Role within the Data Center

• Layer 3 Core – Designs using CCRs with 10 Gbps interfaces

• Top of rack / End of Row – L3 options for core connectivity

• External / Internal Firewall – Internet reachability / protect critical internal networks with multiple layers (PCI)

• VPN Aggregation – Multiple Vendors / Remote Mgmt Access

• MPLS P/PE router - Segregation of traffic within Data Center

• Role between Data Centers

• MPLS L2 VPN – VLAN extension between Data Centers for VM mobility

• MPLS L3 VPN – Segregate traffic as it routes between data centers

• VLAN Rewrites – Used to deal with VLAN overlap between two or more sites

• Multiple Gateways for the same subnet at more than one site



Conventional Data Center



Multi-Million dollar DCs - Where does MikroTik fit in ?

• MikroTik routers can be used in different areas of the Data Center and compete with mainstream vendors like Cisco, Juniper and HP within a specific set of design parameters.

• The goal of this presentation is to display the design elements required to build a Layer 3 infrastructure capable of up to 320 Gbps forwarding with off the shelf 10 Gig switches

• Why 320 Gbps? • Relies on ECMP (server side) – assuming 16 BGP paths

• 16 Paths is a conservative value for ECMP – some go as high as 128 paths

• 16 CCRs with 20 Gbps LACP channels = 320 Gbps

• Design validation was tested with 2 CCRs which yielded 40 Gbps between servers



Multi-Million dollar DCs - Where does MikroTik fit in ?

•Why?

• The business case for MikroTik in the Data Center •CAPEX (Capital Expenditure) Savings

•Lower hardware replacement cost when a node fails

•Cisco Nexus Deployment for 320 Gbps

•$2,000,000 to $5,000,000 CAPEX

•MikroTik Deployment for 320 Gbps

•$50,000 to $100,000 CAPEX



The MikroTik enabled Data Center



Part 1 – Desigining for High Availability – 99.999% uptime

• Getting to five 9’s isn’t easy – can only have 5 minutes of unplanned downtime per year – maintenance windows aren’t included

• HA design elements

• Stackable switches – enables multi-chassis LAG for CCRs and servers to provide survivability in the event of a failure of one of the switches

• LACP – channeling/bonding at Layer 2 allows devices to aggregate speeds as well as prevent routing topology changes when a link fails

• Load Balancers – Provide the ability to use multiple CCR chassis as a single firewall without breaking state. The LB has the ability to return traffic to the same source and track that relationship dynamically.

• BFD – Bi-Directonal Forwarding Detection allows a network t o converge much more quickly than standard timers

• Multiple Internet BGP Peers – When used along with BFD if the upstream carrier supports it, multiple tier 1 peerings provide a level of redundancy to ensure Internet traffic is uninterrupted



Part 2 – Achieving 320 Gbps throughput

• Two mechanisms for achieving high throughput

• Method 1 - ECMP

• Equal Cost Multipath (ECMP) on the CCR – RouterOS is capable of up to 128 gateways .

• Example below shows 16 gateways for one route

• Routes can be installed by either OSPF or Static. BGP can also be run on top of OSPF and utilize ECMP as well.

• ECMP Route with 16 Gateways




• ECMP Continued

• Using multiple gateway allows traffic egressing the router to balance along multiple paths but what about ingress?

• Server side ECMP is the key to scaling throughput when using independent routers.

• Support in multiple operating systems – Microsoft and Linux both support ECMP in static routes along with OSPF and BGP



Achieving 320 Gbps throughput




• Utilizing BGP and OSPF at the server for dynamic ECMP

• Role of OSPF

• Converges quickly using adjusted standard timers (1 second hello 3 second dead)

• Converges even faster with Bidirectional Forwarding Detection (BFP)

• Provides Loopback reachability for BGP

• Is needed to implement ECMP dynamically through MikroTik routers until ECMP is added to BGP.

• Role of BGP

• Advertise data center subnets for servers, databases, web apps, etc, to the 16 router CCR core

• Can be utilized for traffic management



Achieving 320 Gbps throughput




• Method 2 – Offset VRRP Gateways

• Each CCR is the VRRP master for one or more VLANS

• Requires setting priority for each VLAN/CCR

• Can be used in conjunction with ECMP when servers cannot be setup for ECMP

• CCR 1 – Master for VLAN 100 Backup for VLAN 200

• CCR2 – Backup for VLAN 100 Master for VLAN 200




• LACP Channels for Routers and Servers

• LACP is an open standard for aggregating Layer 2 links – 802.3ad

• Referred to as Channeling, Bonding, Teaming, Link Aggregation

• Can be trunked with multiple VLANs and multiple Layer 3 gateways

• Can be used with VRRP

• 20 Gigabit LACP channel on CCR-1036-8G-2S+





• Microsoft LACP example:





• Linux LACP example:



Part 2 – Achieving 320 Gbps throughput • Final Result – 40 Gbps throughput

• Only 2 CCRs in the Core – with 16 CCRs, the throughput will be roughly 320 Gbps



Part 3 – Multiple Data Centers




•Using MPLS in the Data Center

• CCRs can be used as MPLS edge routers to connect Data Centers.

• Used to segregate traffic within and between Data Centers

• L2VPN (VPLS ) – Provides Layer 2 Connectivity and isolation

• L3VPN – Provides Layer 3 connectivity and isolation

• VRF (Routing Marks) – Used to separate customer routing tables so that more than one customer can use the same subnet without overlap






• MPLS Customer Isolation at Layer 3




EoIP provides Layer 2 Connectivity and will allow MPLS to function across an encrypted internet link. Either EoIP or VPLS can be used for L2 connectivity.


•VLAN Rewrites • Problem:

• Data Center 1 uses Vlan 100 for web Servers on 10.1.1.0/24

• Data Center 2 uses Vlan 100 for storage replication on 192.168.222.0/24

• When extending the VLAN between Data Centers, one side must be rewritten

• CCRs can do this via bridging

• MikroTik routers with switch chips can use /switch to perform vlan rewrites




•VLAN Rewrites – change VLAN 100 traffic to VLAN 3100




•VLAN Rewrites – change VLAN 100 traffic to VLAN 3100

•Create VLAN 100 and 3100 interface VLANs

Create Bridge and add VLAN interface ports




•Dual VRRP Gateways • Problem – when extending VLANs between Data Centers, If there is

not a local gateway for hosts in that subnet, traffic must go all the way to the other Data Center via Layer 2 to hit the default gateway

• Solution: Dual VRRP gateways

• Data Center 1 – VRRP GW – 100.64.100.1/24

• Data Center 2 – VRRP GW – 100.64.100.2/24

• These are duplicate IPs – How can this work?

• Because VRRP uses MAC addresses derived from the VRRP Group Number – Hosts will always find the gateway in their own data center before going to the other Data Center




•Dual VRRP Gateways

• Add input filter for VRRP on both edge routers – IP Protocol 112 to prevent either gateway from becoming master for the other (bridges must be set to use IP Firewall)

Add VRRP Gateway for 100.64.100.1 in both Data Centers




•Dual VRRP Gateways



24/7/365 MikroTik TAC | Nationwide Private 4G LTE MPLS | Proactive Network Monitoring | Design / Engineering / Operations

www.iparchitechs.com

1-855-MIKROTIK

2014 Pittsburgh MUM RC Heli Giveaway !! • 4 To Give Away!!! 17” RC Helicopters

Questions?

• The content of this presentation will be available at mum.iparchitechs.com

• Please come see us at the IP ArchiTechs booth in the Exhibitor Hall

• Email: [email protected]

• Office: (303) 590-9943

• Web: www.iparchitechs.com

•Thank you for your time and enjoy the MUM!!



mailto:[email protected]

http://www.iparchitechs.com/

1-855-MIKROTIK Network · PDF fileNetwork Architecture – Validated designs utilizing MikroTik in the Data Center ... •VPN Aggregation ... •MikroTik routers can be used in...

Documents