Using BGP To Manage Dual Internet Connections

SDCUGSept 10, 2014

Meredith Rose, CCIE#4617

Using BGP to Manage Dual

Internet connections

Copyright © SIGMAnet ® 2012. All rights reserved. Proprietary & Confidential.2

Intro

• Meredith Rose, CCIE#4617 Emeritus• Currently a Solutions Architect for SIGAMnet• Disaster Recovery and Redundancy are

recurring themes requested by customers striving to improve their network uptime• Internet access has become

better/faster/cheaper, causing more companies to rely on it and expect 5-nines uptime.• Not planning on reviewing the BGP protocol

details, but please ask questions any time.


High Level Agenda

• The need for Corp Internet x 2• What you need to use BGP• Key considerations• BGP routes offered by ISPs• Influencing traffic flows


Does a Company Need 2 Connections to the Internet?

• Internet access is business-critical• Apps, data exist in the cloud (ie AWS, WebEx)• Branch connectivity via VPNs over the Internet• Remote access, B2B connections• eCommerce hosted at Corp data center

• Redundancy is a must; the less $ the better• BGP can give you tools for utilizing the

bandwidth of both Internet connections simultaneously and/or dynamic failover with 1 connection backing up the other


“I want to use BGP to Load Balance my Internet Connections”

• The BGP protocol does NOT know how to “load balance” your Internet traffic!• BGP’s job is to select the single best path to a destination among the BGP

paths that are learned from different sources/ISP’s. • BGP is not aware which link is “full” (oversubscribed) or “faster” (lower latency)• Load sharing across your redundant Internet connections is a manual process

done on a per prefix basis that takes some TLC.• Inbound and Outbound traffic loads of each link are tuned separately by

manipulating BGP attributes


One Internet Connection

Internet Connection

ISP Router

Corporate LAN

Corp Router

ISP

Global Internet

• Static routes to Corp on ISP router• static default route to ISP on Corp router• No need for BGP


Redundant Internet Connections

InternetConnection#1

ISP#1 Router

Corporate LAN

Corp Router#1

ISP#1

Global Internet


ISP#2 Router

Corp Router#2

ISP#2

eBGP eBGP

iBGP

L3

FHRP/OSPF/etc


Review of Recovery from Failure

• ISP failure– Internet handoff– Router failure– Upstream peering issues

• Corp Router failure– Internet handoff– Router failure– Connection to Corp LAN


Getting started with BGP to the Internet

• You will need an ASN (Autonomous System Number). AKA “AS number”– This can be private if using redundant

connections to the same ISP. Obtain from ISP Will be removed by ISP before being advertised to

global Internet Note: impacts ability to influence inbound traffic with

as-path pre-pending

– This will be a public ASN if connecting to diverse ISPs. Obtain from ARIN More flexibility, ISP-independent


Getting started with BGP to the Internet

• You will need a public IP address “block” to advertise• /24 minimum

– This can be assigned/leased to you from your ISP Easy if both Internet connections are from same ISP Make sure the ISP that allocated the block to you advertises your

specific subnet (ie /24) and not just their supernet block. If using diverse ISP’s, must check with both to make sure it is ok to

advertise IP block from ISP#1 IP space through ISP#2. More convenient, but less portable

– This IP block can be owned by your company. You can advertise your block to both ISPs. More mobility if change ISP’s

– Make sure you only advertise your assigned, routable IP address space!

– You will advertise the SAME IP block out to BOTH ISP’s Can do some tricks with splitting into sub-prefixes and advertising

smaller, more specific chunks. Always >=/24


Key Considerations

• Ingress and Egress “traffic engineering” managed separately

• OUTBOUND traffic influencing– Get your Corp traffic to its destination on the Internet– Want to send traffic out the “best” ISP

Shortest AS path is usually best

– Want to avoid oversubscribing a link

• INBOUND traffic influencing– Packets from everywhere on the global Internet have to find your

Corp network. ISP advertises your IP block(s) to global Internet– Asymmetric is usually OK here (out one ISP, in the other)

Caveat: not ok if you have non-stateful firewalls

– Want to take “best” route from global Internet to Corp Shortest AS path wins in most cases by default

– Want to avoid oversubscribing a link


Key Considerations (Continued)

• Redundancy protocols on Corp routers.– HSRP/VRRP if L2 connected– Or use L3 dynamic protocol like OSPF. Internet

routers can be in different Corp locations, L3 connected. Each Corp BGP router can originate a default route in Corp-wide OSPF.

• Corp routers need to know how to get to ISP router’s peering IP address (or use next-hop-self on iBGP session). If iBGP routers peer on loopback, must be reachable (use IGP + update-source loopback0)

• Get Corp traffic destined for Internet to one of the Corp Internet routers. It doesn’t really matter which one. BGP will take it from there.

• It’s about manual traffic load distribution; BGP does not know how to do dynamic Load Balancing to multiple ISPs on its own

• You do not want your Corp to be come a “Transit” path between your two ISP’s!


“Transit” - What’s the big deal?

Corporate LAN

ISP#1

Global Internet

ISP#2

ISP#1 Routes

ISP#2 says “Hey Global Internet!

Here’s a quick way to reach ISP#1 customers!”


Don’t be a Transit!

Corporate LAN

ISP#1

Global Internet

ISP#2

ISP#2 Routes ISP#1 Routes

Only send routes

originating from your Corp

ASN to each ISP

iBGP full route exchange


What Routes to Take in from ISP• Remember: this affects OUTBOUND decisions (not inbound), ie

which ISP your Corp will use to make a connection to a site on the Internet. Most common options:

• Option#1: Full Internet routes from each ISP• Option#2: Default/0.0.0.0 only from each ISP

– Tune so use one link as primary, other as backup

• Option#3: ISP’s Customer Routes Only– AKA “Partial Routes”– Get each ISP’s local customer routes only. Use a default

route to put the rest of the outbound traffic on one ISP’s link, backup by other ISP.

– Or use just one ISP link to receive that ISP’s directly connected customers, use default route to put the rest of the outbound traffic on the other link


Option#1: Full Routes from Both

Corporate LAN

ISP#1

Global Internet

ISP#2

Full Routes from ISP#1

iBGP full route exchange

Full Routes from ISP#2


What Routes to Take In from ISP• Option#1: Full Internet routes from each ISP

– Need a lot of memory for this. Each router will have 2xfull Internet routing table (table>450k routes)!

– Let it play out and monitor for over-utilization of one link– Tune to balance links better if necessary– Use route-map + as-path access list to make sure you

do not become a transit between ISP’s. Do not advertise routes to ISP#2 that you learned from ISP#1

and vice versa apply a similar route map outbound to each ISP neighbor so

that only locally originated BGP routes are advertised– route-map localonly permit 10– match as-path 10– ip as-path access-list 10 permit ^$

– Not a bad idea to take a default from each ISP as well


Option#2: Default from Both

Corporate LAN

ISP#1

Global Internet

ISP#2

0.0.0.0

iBGP exchange default received

routes w/preferences

Configure iBGP to prefer default

route from ISP#1

0.0.0.0

iBGP will agree to prefer 0.0.0.0 from

ISP#1 over ISP#2


What Routes to Take In from ISP• Option#2: Default only from each ISP– Tune BGP (local pref is common) so use one link as

primary, other as backup (again, only applies to OUTBOUND traffic)

– Tell your ISPs you only want them to send you the default route

– Use an inbound prefix-list on route-map inbound on the ISP neighbor statement or similar filter to make sure to drop every route except default just in case ip prefix-list default-only seq 5 permit 0.0.0.0/0

– Still only advertise prefixes originated by your AS to ISP#1 and ISP#2 (by default, BGP won’t send them each other’s 0.0.0.0 that you learned –phew!)


Option#3: ISP Local Routes Only

Corporate LAN

ISP#1

Global Internet

ISP#2

Routes from ISP#1 customers + 0.0.0.0

iBGP will naturally send traffic for local

routes to its corresponding ISP

Configure iBGP to prefer default route from ISP#1 to catch routes not local

to either ISP

iBGP will agree to prefer ISP#1 for everything not

local to ISP#2

Routes from ISP#2 customers + 0.0.0.0


What Routes to Take In from ISP

• Option#3: ISP’s routes only + Default– Only receive routes from an ISP of that ISP’s directly

connected customers (think of how many big companies host with ATT, etc)

– You can ask your ISP to send you just their customer routes– Filter routes not sourced from that ISP just in case (in this

example, ISP = AS100, route-map is inbound on neighbor statement to ISP): ip as-path access-list 20 permit ^100$ route-map as100only permit 10 match as-path 20

– Use one link for one directly connected ISP’s customers (more local provider), use default route to prefer to put the rest of the outbound traffic on the other link or similar combo


Influencing Traffic Flows: OUTBOUND

• OUTBOUND Traffic Control is easier than INBOUND. It’s all on you.

• All you have to control is how attractive a destination looks to your Corp BGP routers.

• You can only control the next AS in the path (ie ISP#1 vs ISP#2), not the entire path through the global Internet to the destination.

• Most common OUTBOUND:– Local preference

Outbound traffic flows to one of your Corp BGP routers. BGP will have used the “local preference” attribute to tell that router which route to take (ISP#1 vs ISP#2) to reach the destination.

Monitor regularly and tweak/tune local pref of prefixes as desired Look for popular, heavily-used prefixes to influence to get the

most bang for your buck (or increase local pref of big /4 chunks)


Influencing Traffic Flows: INBOUND

• Most common INBOUND:– AS-Path prepend

Backup Path: If you don’t want traffic to come in on a link for a prefix (or the entire IP block), use Prepending feature to add AS Path length to your outbound advertisement, making this link the less preferred path for traffic to your IP block. Aka “padding”. 4xAS# is generally sufficient

Primary Path: Use standard advertisement (no prepending) for the link you prefer to use for inbound traffic to your company

Still have (pre-pended, valid) advertisement from backup path if primary path fails.– Example: set as-path prepend 130 130 130 (add to route-map and apply to neighbor statement to backup ISP)

– Communities Community = instructions from you to your ISP on how to tweak what you advertise ISP will let you know definition of communities they honor You will attach a community to a prefix that you are advertising to your ISP(s) Consists of a series of numbers that correspond to handling instructions for that prefix (such as set

local pref within provider’s AS) Communities can also be used internally to identify routes. For example, you can assign all routes that came

from ISP#1 with a community and routes that came from ISP#2 with a different community. That community identifier can then be used by your company to assign preferences to routes advertised internally via iBGP. For example, I want all traffic destined for YouTube’s /16 IP Block to use ISP#2, even though ISP#1 has a shorter AS-Path for the YouTube subnet (perhaps bandwidth is greater to ISP#2). So use the community to set a better metric on that route when it comes in from ISP#2. Remember, weight and local pref take precedence over AS-path length.

– Prefix-splitting ie /192x/20 subnets. Advertise one to each ISP, both also advertise complete /19 aggregate as a

safety-net to cover failure of one ISP. Remember: most specific advertisement always wins! Works best when you own your IP Space (splits still >=/24) Use a BGP Looking Glass or Route Server to see how to get to your Corp AS’s prefixes

SDCUGSept 10, 2014

Meredith Rose, CCIE#4617

Thank You!


Redundant Internet Connections


ISP#1 Router

Corporate LAN

Corp Router#1

ISP#1

Global Internet


ISP#2 Router

Corp Router#2

ISP#2

eBGP eBGP

iBGP

L3

FHRP/OSPF/etc

Using BGP To Manage Dual Internet Connections

Technology

internet traffic

corp internet x

corp bgp router

corp traffic

connections tothe internet

isp easy

corp shortest

internet remote access