© 2002 Foundry Networks, Inc. Técnicas de Alta Disponibilidade para NAPs Marcelo Molinari – Foundry Networks do Brasil [email protected] [email protected].

© 2002 Foundry Networks, Inc.

Técnicas de Alta Disponibilidade para NAPs

Marcelo Molinari – Foundry Networks do [email protected]

©2002 Foundry Networks, Inc.

Agenda

• LINX topology overview

• AMS-IX topology overview

• Metro Ring Protocol

• Virtual Switching Redundancy Protocol


London Internet Exchange (LINX)


LINX Topology


LINX Topology• The LINX Network consists of two separate high-

performance Ethernet switching platforms installed across seven locations.

• Switches from two equipment vendors are deployed in two separate networks to provide an extra level of fault-tolerance, the logic being that both systems shouldn't develop the same fault at the same time.


LINX Topology

• Two switches are installed in every LINX location, and the locations are interconnected by multiple 10 gigabit Ethernet circuits to form two physically separate backbone rings.

• Most LINX members connect to both switching platforms, which reduces the impact of any downtime on a single network element.

• Management of the logical redundancy of the network is done using MRP (Metro Ring Protocol). In the event of the loss of a network segment, MRP activates a redundant link within tenths of a second and restore connectivity.


LINX Aggregated Traffic Statistics


Amsterdam Exchange (AMS-IX)


AMS-IX Topology


AMS-IX Topology• AMS-IX is a distributed exchange, currently present at five

independent co-location facilities in Amsterdam.

• The AMS-IX topology is built around two hub/spoke arrangements.

• The core switches are Foundry Networks NetIron MLX-32 switches.

• Members connected with GigE, 100Base-TX or 10Base-T ports are connected to Foundry Networks BigIron 15000 and BigIron RX-8 switches.

• Members with a 10GbE port are connected to Glimmerglass Networks photonic cross-connects. These L1 switches connect the member 10GbE ports to BigIron RX-16 or NetIron MLX-16 switches.

http://www.foundrynet.com/products/enterprise/agg-bb-l23/bi-rx.html


AMS-IX Topology• The two core switches run VSRP (Virtual Switch

Redundancy Protocol) to define the active hub/spoke and to automatically fail over to the other, based on pre-defined triggers (e.g. link failure).

• All edge switches from Foundry follow VSRP automatically. The Glimmerglass switches follow the VSRP failover based on software developed at AMS-IX.

• Members can connect to the AMS-IX infrastructure at any of the five AMS-IX co-locations, at 100 Mbit/s, 1 Gbit/s or 10 Gbit/s.


AMS-IX Traffic Statistics


Metro Ring Protocol


Metro Ring Protocol (MRP)

• Metro Ring Protocol is a Layer 2 protocol designed to provide SONET-like, high speed, fault tolerant, fast recovery for Metro Ethernet networks.

• MRP SONET-like features provide: – Sub-second failover– Efficient use of bandwidth with topology groups (802.1s

based)– Scalable protection for multiple VLANs– Large Scale L2 MANs with multi-ring support– Highly flexible network designs– Works with other L2 features

• Runs on all Ethernet and PoS/SDH interfaces, including 10 Gigabit


How it works

• A single node is defined as the Ring Master Node

• All other nodes are defined as Ring Member Nodes

• The Master Node prevents loops by blocking its secondary port

• Ring Hello Packets are generated by the Master Node to check ring integrity

• As long as the master sees its own Hello packets on the secondary port, ring health is verified, and secondary port remains blocked

Secondary (Blocking)

Primary (Forwarding)

RHPRHP

RHPRHP


Rapid Failover• Hello packets are hardware

forwarded by the nodes in the ring to ensure fastest failure detection.

• Master considers ring broken if no Hello packets are received within 300 ms (3 consecutive Hellos are lost).

• If no Hellos are received, Master transitions secondary port into forwarding state to restore ring connectivity.

• To provide reliable flushing of stale MAC entries, Master sends 3 consecutive TCN notifications.

• By changing timers and using messages sent by the node where the ring broke, it is possible to achieve recovery times from 150 ms to 200 ms.

RHP

NetIron 400

NetIron 400

NetIron 400 NetIron 400

RHP

RHPRHP

TC

NetIron 400

NetIron 400


TC

FAULT

TC


Link RestorationPreventing Temporary Loops• When an MRP port goes up, it goes

into pre-forwarding mode to avoid the creation of temporary loops.

• In pre-forwarding mode the port forwards no data, but only the ring hello packets from the master.

• Master sees its own RHP, detects that ring integrity has been restored, puts its secondary port in blocking mode.

• From that point onwards, master sends RHPs with the Forwarding flag bit set, indicating that members should transition their ports from pre-forwarding to forwarding.

• The Forwarding flag bit is always set, as long as the master is blocking its secondary port.

NetIron 400

NetIron 400


RHPRHP

NetIron 400

NetIron 400


Link restored

RHPRHP

RHPRHP

PF

PF

RHPRHP

F

F


Topology Groups

Topology Group

Master VLAN

Member VLAN

Member VLAN Group

A VLAN running a control protocol (or more) that controls the active topology for the whole topology group. Control protocols: STP, RSTP, MRP, VSRP.

A VLAN running NO control protocol of its own but rather follows the active topology of the master VLAN..

A group of VLANs running NO control protocol of their own but rather follow the active topology of the master VLAN. VLAN groups are defined via the “vlan-group” command.


Efficient use of Ring Bandwidth• MRP Supports Multiple Topology groups within a Ring

• An MRP Node can be both a Master node and Member Node for different topology groups

• Each Topology group contains a Master VLAN and Member VLANs

• Master VLANs generate Hello packets and block secondary ports

• 4094 VLANs can be divided among up to 255 Topology groups

VLAN Group 2

All links are utilized!VLAN Group 1

2

BigIron 4000BigIron 4000



BigIron 4000BigIron 4000BigIron 4000BigIron 4000







1

TopoGroup 1

TopoGroup 2


Using Multiple Rings• There are 3 ring scenarios:

• Single ring

• Rings that don’t overlap

• Overlapping rings that share links

• Each Ring runs its own instance of MRP

• A ring node can be Master for Multiple rings




BigIron 4000BigIron 4000 BigIron 4000BigIron 4000












Single Ring Non-Overlapping Ring Overlapping Rings

Phase I

Phase II


Example Scenarios – Phase I

– High-speed 10 GE trunks for Metro rings or IXPs

– Provides sub-second fault-detection and fail-over

– Superior scalability: no limit on maximum number of nodes per ring

– Counter rotating topology groups provide efficient use of bandwidth

BigIron 4000

BigIron 8000

BigIron 4000

BigIron 4000

BigIron 8000

BigIron 4000

S5

S6

S1

S4

S3

S2

MasternodePrimary

port

Secondary port

BigIron 4000

BigIron 4000

BigIron 4000

BigIron 4000

BigIron 4000

Ring1

Ring4

Ring2

Masternode

BigIron 4000

BigIron 4000

S5

S4

Ring3


Example Scenarios – Phase II

– Shared ring support– Increased reliability – Increased bandwidth

BigIron 4000

BigIron 8000

BigIron 4000

BigIron 4000

BigIron 8000

BigIron 4000

S5

S6

S1

S4

S3

S2

Masternode

Primary port

Secondary port

BigIron 4000

BigIron 4000

BigIron 4000

BigIron 4000

BigIron 4000

BigIron 4000

Ring1

Ring4

Ring2

Masternode

Masternode

BigIron 4000

BigIron 4000

S5

S4

Ring3


Interface Flexibility

BigIron 4000

BigIron 8000

BigIron 4000

BigIron 4000

BigIron 8000

BigIron 4000

S5

S6

S1

S4

S3

S2

Masternode

Primary port

Secondary port

BigIron 4000

BigIron 4000

BigIron 8000

S5

S6

S4

BigIron 4000

S3

Masternode

Primary port

Secondary port

BigIron 8000

S2

BigIron 4000

S1

• Support for Mixed interfaces- 10Gig & Gig- Gig & 10/100- 10Gig & PoS/SDH

Slower link

• Support for Trunked interfaces- 10Gig & Gig- PoS/SDH


MRP – Summary of Benefits

• Fast, sub-second, predictable fail-over functionality

• Maximizes ring bandwidth utilization

• Cost effective scalable solution for MAN resiliency

• Attractive alternative to STP

• Utilizes 802.3 Ethernet Packet standards and MACs

• Can be combined with other Foundry features to provide complete end to end MAN designs


Virtual Switch Redundancy Protocol


Virtual Switch Redundancy Protocol• VSRP provides an alternative to Rapid Spanning Tree

Protocol (RSTP) in dual homed/mesh configurations, providing sub-second fail-over and recovery.

• VSRP features provide: – Sub-second fail-over– Efficient use of mesh bandwidth – no blocked links– Block and unblock ports at the per-VLAN group level– Large scale L2 MANs with multi-tiered support– Highly flexible network designs– Configurable tracking options– Works with other L2 features

• Works with all Ethernet interfaces, including 10 Gigabit

• VSRP is based on VRRP-E & can provide L2 and L3 backup


How it works

BigIron 8000

S1

BigIron 4000

S3

BigIron 4000

S5

VSRPMaster

• VSRP uses an election process to select a Master switch and up to 4 backup switches for each VLAN: higher configured priority wins; if equal, higher IP address wins.

• Only the Master switch forwards data, while Backup switches block traffic on all VSRP configured interfaces within the VLAN (or the topology group).

• Master switch sends Hello packets to all backup switches

• Switches do not have to be VSRP aware. VSRP aware provides faster failover.

• VSRP can track ports and decrease the priority of VSRP active switch in case a tracked port goes down.

BigIron 4000

S4

S2

VSRPAware

Hello

Hello

FBBFF B

BigIron 8000

VSRPBackup


Rapid Failover• A VSRP Backup switch monitors Hellos

from the Master.

• If no Hellos are received for Master Dead Interval (default 300 ms), Backup goes into Hold Down state, starts sending periodic Hellos.

• Hold Down interval is by default 300 ms, and it allows for the election of a new master.

• If the switch is elected as Master, it sets its port into forwarding state, sends 3 TCNs.

• A VSRP aware switch receives TCN, and looks for the new master. Hellos of the new master will be received on a different port.

• A VSRP aware switch shifts the MAC addresses learned on the failed port to the new port.

Mac Type:D = Dynamic, S = Static, H = Host, R = Router

MAC Port Age Type

0060.f320.23a8 2 0 D

0030.1b07.0694 2 0 D

00d0.b758.88dc 2 0 D

0004.8039.5f00 2 0 D

NetIron 400

NetIron 800 NetIron 800 NetIron 800

BF B

Master Backup Backup

Hello

HelloHe

llo

NetIron 400


BDn F

Master Backup

Hello

FAULT

FAULT

Hello


Link RestorationSwitching Back to Original Master• When the failed link is restored, the

original Master remains as a Backup.

• Original Master receives inferior Hello from the current Master, so it immediately replies with its own Hello, switches into Hold Down state (300 ms), starts sending periodic Hellos

• Current Master receives superior Hello, so it switches into Backup mode.

• If no superior Hellos are received during Hold Down interval, original Master considers itself the new current Master, and sets its port in Forwarding mode.

• New Master sends out 3 TCNs.

• A VSRP aware switch receives TCN, and looks for the new master. Hellos of the new master will be received on a different port.

Mac Type:D = Dynamic, S = Static, H = Host, R = Router

MAC Port Age Type

0060.f320.23a8 1 0 D

0030.1b07.0694 1 0 D

00d0.b758.88dc 1 0 D

0004.8039.5f00 1 0 D

NetIron 400


BB F

Master(Orig. Master)

Backup Backup

Hello

Hello

NetIron 400


BF B

Master Backup

Hello HelloHello

Hello

Backup

Link restored

Hello

Hello

Hello

• A VSRP aware switch shifts the MAC addresses to the new port.


Efficient use of Uplink Bandwidth


S1

BigIron 4000

S3

BigIron 4000

S5

Master topology group 1Backup topology group 2

HelloPackets

BigIron 4000

S4

S2

Master topology group 2Backup topology group 1

VSRPAware

Topology group 1 = Master VLAN 1Member VLANs 2 to 2048

Topology group 2 = Master VLAN 2049Member VLANs 2050 to 4096

• VSRP supports topology groups to fully utilize switches and links

• Topology groups are a collection of VLANs

• Each yopology group contains a Master VLAN and Member VLANs

• VSRP configured switches can be Master for some topology groups while backup other for others

• 4094 VLANs can be divided among up to 255 Topology groups


VSRP Domains• VSRP can be configured in

separate domains within the same VLAN to allow for larger topologies.

• Topology groups can be designed to use unique paths in each domain.

• A TTL value within the VSRP Hello packet controls how deep the packet goes into the network.

• TTL is being decremented by 1 at each VSRP aware switch.

• Default TTL is 2, which allows Hello to traverse one VSRP aware switch to go to another VSRP active switch.

NetIron 400


NetIron 400



VSRP Aware

VSRP Active

VSRP Active

VSRP Active

VSRP Aware

VSRP Aware

VSRP Domain 1(VRID 1)




Intelligent Port Level Control

• VSRP can be configured to run only on designated ports

• Only VSRP configured ports are placed in blocking for Backup switches

• Supports host attachments and dual port NICs: host facing ports are configured as VSRP free.

• Works in combination with MRP to provide flexible Metro / Enterprise Ethernet designs.


MRPMaster Backup

VSRP AwareDual Homed Servers




VSRP-Aware Switches

• Both VSRP-Aware and Non VSRP-Aware switches can be used as edge devices

• VSRP-Aware switches recognize the VSRP Hello packet sent by the Master, and then create an table which contains the VRID of the VLAN which sent the VSRP Hello plus the incoming port where the VSRP Hello was received.

• When the VSRP-Aware switch sees a Hello packet coming from a different port, it quickly moves the MAC address table entries to the new port.

BigIron 8000

S1

BigIron 4000

S3

BigIron 4000

S5

VSRPMaster

BigIron 4000

S4

S2

VSRPAware

Hello

Hello

FBBFF B

BigIron 8000

VSRPBackup

VSRPAware

VSRPAware


Non VSRP-Aware Switches

• Non VSRP-Aware switches can be used as edge devices, but they do not recognize VSRP Hello packets.

• MAC entries will age out or they will eventually be learned from the new port.

• This results in slow convergence when the Master fails.

• Solution is to configure “VSRP Fast Start” in the VSRP Master and Backup nodes.

• VSRP Fast Start disables and re-enable the ports before transitioning from Master to Backup. This causes a MAC address flush in the edge devices which makes convergence faster.

BigIron 8000

S1

BigIron 4000

S3

BigIron 4000

S5

VSRPMaster

BigIron 4000

S4

S2

Non VSRPAware

FBBFF B

BigIron 8000

VSRPBackup

Non VSRPAware

Non VSRPAware


VSRP – Summary of Benefits

• Fast, sub-second protection without Spanning Tree

• Combines both switching and routing redundancy– Provides default gateway redundancy if

needed

• Supports topology groups for full link utilization

• Can be combined with other Foundry features to provide complete end to end MAN designs


References

• LINX - https://www.linx.net/

• AMS-IX - http://www.ams-ix.net/

• “I can feel your traffic” - http://www.jasinska.de/

• MRP- http://www.foundrynet.com/pdf/wp-mrp.pdf

• VSRP - http://www.foundrynet.com/pdf/wp-vsrp.pdf


Thank You !Marcelo Molinari – Foundry Networks do Brasil

[email protected]

© 2002 Foundry Networks, Inc. Técnicas de Alta Disponibilidade para NAPs Marcelo Molinari – Foundry Networks do Brasil [email protected] [email protected].

Documents

foundry networks bigiron

foundry networks netiron

metro ring protocol

separate networks

linx topology slide

metro ethernet networks

ring master node

ring connectivity