1 © 2001, Cisco Systems, Inc. All rights reserved. © 2001, Cisco Systems, Inc. All rights reserved. © 2002, Cisco Systems, Inc. All rights reserved. 12K Support Training
Mar 26, 2015
1© 2001, Cisco Systems, Inc. All rights reserved.© 2001, Cisco Systems, Inc. All rights reserved.© 2002, Cisco Systems, Inc. All rights reserved.
12K Support Training
© 2001, Cisco Systems, Inc. All rights reserved. 2© 2001, Cisco Systems, Inc. All rights reserved. 2© 2002, Cisco Systems, Inc. All rights reserved. 2
Goals
Deepen global 12k support expertise through architecture discussion and hands-on troubleshooting
© 2001, Cisco Systems, Inc. All rights reserved. 3© 2001, Cisco Systems, Inc. All rights reserved. 3© 2002, Cisco Systems, Inc. All rights reserved. 3
Agenda
• 12K Product Overview
• System Architecture
• Forwarding Architecture
• Services and Applications
• Troubleshooting
4© 2001, Cisco Systems, Inc. All rights reserved.© 2001, Cisco Systems, Inc. All rights reserved.© 2002, Cisco Systems, Inc. All rights reserved.
Module I – Overview and System Architecture
5© 2001, Cisco Systems, Inc. All rights reserved.© 2001, Cisco Systems, Inc. All rights reserved.© 2002, Cisco Systems, Inc. All rights reserved.
Part I - 12K Product Overview
© 2001, Cisco Systems, Inc. All rights reserved. 6© 2001, Cisco Systems, Inc. All rights reserved. 6© 2002, Cisco Systems, Inc. All rights reserved. 6
12K Architecture Overview
• Fully distributed, multi-gigabit IP Router
RP provides routing and control services
Line cards perform IP forwarding
• Advanced QoS capabilities
• Bandwidth scalable (OC12, OC48, OC192)
© 2001, Cisco Systems, Inc. All rights reserved. 7© 2001, Cisco Systems, Inc. All rights reserved. 7© 2002, Cisco Systems, Inc. All rights reserved. 7
Cisco 12008 Product Highlights
• Crossbar switch fabric architecture
• 8 slot card cage (7 for interfaces)
• Components:Switch Fabric Cards (SFC)
Clock and Scheduler Cards (CSC)
Route Processor (RP)
Line Cards (LCs)
© 2001, Cisco Systems, Inc. All rights reserved. 8© 2001, Cisco Systems, Inc. All rights reserved. 8© 2002, Cisco Systems, Inc. All rights reserved. 8
Cisco 12012 Product Highlights
• Crossbar switch fabric architecture
• 12 slot card cage (11 for interfaces)
• Components:Switch Fabric Cards (SFC)
Clock and Scheduler Cards (CSC)
Route Processor (RP)
Line Cards (LCs)
© 2001, Cisco Systems, Inc. All rights reserved. 9© 2001, Cisco Systems, Inc. All rights reserved. 9© 2002, Cisco Systems, Inc. All rights reserved. 9
Cisco 12016 Product Highlights
• Switching performance16 Slot System, 2.5Gbps switching capacity/slot – can support 10Gb LCs if fabric is upgraded
• Increased number of linecards
• Configuration2 Interface Shelves16 slots1 Fabric Shelf, with 5 slots2 Alarm cards – 1 top shelf, 1 bottom shelf
© 2001, Cisco Systems, Inc. All rights reserved. 10© 2001, Cisco Systems, Inc. All rights reserved. 10© 2002, Cisco Systems, Inc. All rights reserved. 10
• Switching performance16 Slot System each with 10Gbps switching capacity/slot
• Supports 10G linecardsSupport for existing 12k line cards Slots are wider to accommodate 10 Gb LCs
• Configuration2 interface shelves16 slots1 fabric shelf, with 5 slots2 Alarm cards – 1 top shelf, 1 bottom shelf
Cisco 12416 - Product Overview
© 2001, Cisco Systems, Inc. All rights reserved. 11© 2001, Cisco Systems, Inc. All rights reserved. 11© 2002, Cisco Systems, Inc. All rights reserved. 11
• 10 X OC192 capable slots• 8 Slots are wider to accommodate
10 Gb Lcs• 2 x Legacy slots (narrower slots 8 and 9)• 7 card fabric – 2 CSCs & 5 SFCs
Cisco 12410 Product Highlights
© 2001, Cisco Systems, Inc. All rights reserved. 12© 2001, Cisco Systems, Inc. All rights reserved. 12© 2002, Cisco Systems, Inc. All rights reserved. 12
Cisco 12406 Product Highlights
• 6 slot card cage 1 narrow slot dedicated for RP
5 for redundant RP and Line Cards
• Components:Switch Fabric Cards (SFC)
Clock and Scheduler Cards (CSC)
1 or 2 Route Processors (RP)
Up to 5 Line Cards (LCs)
1 or 2 Alarm Cards
• 1/3 rack height
© 2001, Cisco Systems, Inc. All rights reserved. 13© 2001, Cisco Systems, Inc. All rights reserved. 13© 2002, Cisco Systems, Inc. All rights reserved. 13
Cisco 12404 GSR Product Highlights
• 4 slot card cage
1 narrow slot for RP
3 10G capable slots
• Components:1 Consolidated Fabric Card : CSC-4
(CSC, SFC, Alarm built in)Route Processors (RP)
Up to 3 Line Cards (LCs)
FABRIC IS NOT RESILIENT
© 2001, Cisco Systems, Inc. All rights reserved. 14© 2001, Cisco Systems, Inc. All rights reserved. 14© 2002, Cisco Systems, Inc. All rights reserved. 14
• Switching performance16 Slot System each with 40Gbps switching capacity/slot
• Supports 20G and future 40G linecardsSupport for existing GSR line cards Slots are wider to accommodate 10/20Gb LCsRequires PRP
• Configuration2 interface shelves16 slots1 fabric shelf, with 5 slots2 Alarm cards – 1 top shelf, 1 bottom shelf
Cisco 12816 - Product Overview
© 2001, Cisco Systems, Inc. All rights reserved. 15© 2001, Cisco Systems, Inc. All rights reserved. 15© 2002, Cisco Systems, Inc. All rights reserved. 15
• 10 X 40Gb capable slots• 8 Slots are wider to accommodate
10/20 Gb LcsRequires PRP
• 2 x Legacy slots (narrower slots 8 and 9)• 7 card fabric – 2 CSCs & 5 SFCs
Cisco 12810 Product Highlights
16© 2001, Cisco Systems, Inc. All rights reserved.© 2001, Cisco Systems, Inc. All rights reserved.© 2002, Cisco Systems, Inc. All rights reserved.
Part II - 12K System Architecture
17© 2001, Cisco Systems, Inc. All rights reserved.© 2001, Cisco Systems, Inc. All rights reserved.© 2002, Cisco Systems, Inc. All rights reserved.
12K Components
© 2001, Cisco Systems, Inc. All rights reserved. 18© 2001, Cisco Systems, Inc. All rights reserved. 18© 2002, Cisco Systems, Inc. All rights reserved. 18
System Components
• Route Processor
• Switching fabric
• Line cards
• Power/Environmental Subsystems
• Maintenance BUS
© 2001, Cisco Systems, Inc. All rights reserved. 19© 2001, Cisco Systems, Inc. All rights reserved. 19© 2002, Cisco Systems, Inc. All rights reserved. 19
• • • • • •Switch Fabric
Switch Fabric
Maintenance Bus
PowerSuppliesPower
SuppliesFan/Blower
SystemFan/Blower
System
12k Architecture - Components
Line CardLine Card
Line CardLine Card
Line CardLine Card
Route Processor
Route Processor
Line CardLine Card
Line CardLine Card
Line CardLine Card
Route Processor
Route Processor
© 2001, Cisco Systems, Inc. All rights reserved. 20© 2001, Cisco Systems, Inc. All rights reserved. 20© 2002, Cisco Systems, Inc. All rights reserved. 20
Route Processor
• Boots and manages line cards
• Provides and coordinates routing services
• Builds, distributes, and maintains FIB
Adjacency table, FIB table, MPLS label table
• Provides out-of-band console/aux ports
• Provides intelligence behind system monitoring and access
© 2001, Cisco Systems, Inc. All rights reserved. 21© 2001, Cisco Systems, Inc. All rights reserved. 21© 2002, Cisco Systems, Inc. All rights reserved. 21
RP - System Monitor/Controller
• • • • • •
Line CardLine Card
Line CardLine Card
Line CardLine Card
LineCardLineCard
Switch Fabric
Switch Fabric
Line CardLine Card
Line CardLine Card
Line CardLine Card
RouteProcessor
RouteProcessor
Maintenance Bus
Routing Protocol UpdatesProcess-level Traffic
System health monitoringInterface Status Msgs
Statistics
Temperature,Voltage,
Current MonitoringPower
SuppliesPower
SuppliesFan/Blower
SystemFan/Blower
System
© 2001, Cisco Systems, Inc. All rights reserved. 22© 2001, Cisco Systems, Inc. All rights reserved. 22© 2002, Cisco Systems, Inc. All rights reserved. 22
Line Cards
• Perform all packet switching
• Statistics collection and reporting
• Run IOS
• Six different forwarding architectures
© 2001, Cisco Systems, Inc. All rights reserved. 23© 2001, Cisco Systems, Inc. All rights reserved. 23© 2002, Cisco Systems, Inc. All rights reserved. 23
• Out-of-band communications channel to linecards
• 1 Mbps - 2 wire serial interface
• Based on Controller Area Network (CAN) 2.0 Spec. (ISO 11898)
http://www.can-cia.org/can/
• A daughter card on each linecard having it’s own CPU w/ integrated CAN controller, A/D converter and other peripherals, dual CAN interface, SRAM, Flash and Serial EEPROM.
CSCs and BusBoard can proxy and/or multiplex MBUS signals for power supplies
• Control pins reach into LED, Serial ID EEPROM, DC/DC power converter, clock select FPGA, temp sensor, voltage sensor
Very large set of functions
PowerSupply
PowerSupply
Fan/BlowerSystem
Fan/BlowerSystem
• • • • • •Multigigabit Crossbar
Fabric
Multigigabit Crossbar
Fabric
Line CardLine Card
Route ProcessorRoute Processor
Line CardLine Card
Line CardLine Card
Line CardLine Card
Line CardLine Card
Line CardLine Card
Route Processor
Route Processor
SchedulerScheduler
Maintenance Bus
MBUS
© 2001, Cisco Systems, Inc. All rights reserved. 24© 2001, Cisco Systems, Inc. All rights reserved. 24© 2002, Cisco Systems, Inc. All rights reserved. 24
MBUS Functions
• Power and boot LC
• Device Discovery
• RP arbitration
• OIR management
• Environmental monitoring
• Diagnostics download
• LC console access
Via “attach” command
• Logging
© 2001, Cisco Systems, Inc. All rights reserved. 25© 2001, Cisco Systems, Inc. All rights reserved. 25© 2002, Cisco Systems, Inc. All rights reserved. 25
Alarm Cards
• LED display for fabric card status
• External alarm connection
• Power conversion/supply for 5v MBUS power plane
On the 12008, this functionality is on the CSC.
© 2001, Cisco Systems, Inc. All rights reserved. 26© 2001, Cisco Systems, Inc. All rights reserved. 26© 2002, Cisco Systems, Inc. All rights reserved. 26
Switch Fabric - Overview
• Provides the data path connecting the LCs and the RP
• Active CSC card provides the master clock for the system
• Everything traverses fabric in Cisco cell.
- Data is 8B/10B encoded
• Two components
- Clock & Scheduler Cards (CSC)
- Switch Fabric Cards (SFC)
© 2001, Cisco Systems, Inc. All rights reserved. 27© 2001, Cisco Systems, Inc. All rights reserved. 27© 2002, Cisco Systems, Inc. All rights reserved. 27
ciscoCell
• Packet are chopped into ciscoCells before they are sent across the switching fabric.
• A ciscoCell is 64bytes of data consisting of 48bytes of IP payload and 8bytes of header and 8bytes of CRC.
© 2001, Cisco Systems, Inc. All rights reserved. 28© 2001, Cisco Systems, Inc. All rights reserved. 28© 2002, Cisco Systems, Inc. All rights reserved. 28
• Scheduler (SCA)
Handles scheduling requests and issues grants to access the crossbar switching fabric
• Cross-bar (XBAR)
Sets the fabric lines for transmissions following the scheduling decision
Clock Scheduler Card (CSC)
© 2001, Cisco Systems, Inc. All rights reserved. 29© 2001, Cisco Systems, Inc. All rights reserved. 29© 2002, Cisco Systems, Inc. All rights reserved. 29
• Each fabric card provides a slice of the Cisco cell data path
• Up to 5 data paths are available – for up to 4+1 redundancy
• The 5th data path carries an XOR of other streams
Used for recovery of a errored stream No 5th path = no recovery capability
• ‘Grants’ travel exclusively between the LC and the active CSC using separate communication lines
Never traverse the SFC cards
Fabric Redundancy
© 2001, Cisco Systems, Inc. All rights reserved. 30© 2001, Cisco Systems, Inc. All rights reserved. 30© 2002, Cisco Systems, Inc. All rights reserved. 30
Scheduling Algorithm (“ESLIP”)
• RequestEach input LC makes request to output highest priority queued cell (unicast or multicast)
• GrantEach destination LC grants the request to the highest priority request
• AcceptEach input LC selects the highest grant
• TransmitXBAR set and cells transmitted
© 2001, Cisco Systems, Inc. All rights reserved. 31© 2001, Cisco Systems, Inc. All rights reserved. 31© 2002, Cisco Systems, Inc. All rights reserved. 31
ESLIP Illustrated
SwitchFabricSwitchFabric
SchedulerScheduler
DRR
Each line card utilizesDRR to select a setof packets from theVoQs.
Request is sent to Scheduler on CSCto obtain a grant
Request
Scheduler (for eachoutput) selectshighest priority packet fromrequests and determines if outputcan grant request
Grant
Schedulersend multiple grants(for multiple outputs)to slot
Slot select the highestgrant and accepts theconnection
Accept
32© 2001, Cisco Systems, Inc. All rights reserved.© 2001, Cisco Systems, Inc. All rights reserved.© 2002, Cisco Systems, Inc. All rights reserved.
Bootup process
© 2001, Cisco Systems, Inc. All rights reserved. 33© 2001, Cisco Systems, Inc. All rights reserved. 33© 2002, Cisco Systems, Inc. All rights reserved. 33
Startup/Boot Process
• Initial Power On
• RP Boot Process
• Clock Scheduler Boot Process
• Line Card Boot Process
• Switch Fabric Boot Process
• Fabric Initialization
• IOS Download
© 2001, Cisco Systems, Inc. All rights reserved. 34© 2001, Cisco Systems, Inc. All rights reserved. 34© 2002, Cisco Systems, Inc. All rights reserved. 34
Initial Power On
• When the chassis is powered on, the Mbus module on each card is powered on.
• After the Mbus module powers on its processor it boots from a module on EEPROM.
• Card power up order varies depending on linecard type.
© 2001, Cisco Systems, Inc. All rights reserved. 35© 2001, Cisco Systems, Inc. All rights reserved. 35© 2002, Cisco Systems, Inc. All rights reserved. 35
RP Boot Process
• Mbus module powers first
• Board logic starts, image begins booting and Mbus code is loaded to the Mbus module
• The CPU, Memory controller ASIC, cell-handler ASICs and FIA ASICs are then issued power for startup
• RP arbitration process is executed using the Mbus
• Master RP instructs Line Cards and Switch Fabric Cards to power on.
• RP waits for Line Cards to power and finish booting
© 2001, Cisco Systems, Inc. All rights reserved. 36© 2001, Cisco Systems, Inc. All rights reserved. 36© 2002, Cisco Systems, Inc. All rights reserved. 36
Switch Fabric Card Startup/Boot
• Master RP instructs each SFC Mbus module to power on at the same time the Line Card Mbus modules are told
• SFC obtains clock the same way each LC does
• The SLI ASICs and XBAR initialize and power up
• SFC Mbus code is downloaded from the RP
• All cards are now powered on but not usable
© 2001, Cisco Systems, Inc. All rights reserved. 37© 2001, Cisco Systems, Inc. All rights reserved. 37© 2002, Cisco Systems, Inc. All rights reserved. 37
Line Card Startup/Boot
• Each LC Mbus module powers up after being told to do so by the RP
• Clock selection takes place
• The Line Card CPU is powered on and boots
• Mbus module code is loaded
• The Line Cards CPU notifies the RP it has booted
• Switch Fabric access is not available yet
© 2001, Cisco Systems, Inc. All rights reserved. 38© 2001, Cisco Systems, Inc. All rights reserved. 38© 2002, Cisco Systems, Inc. All rights reserved. 38
Line Card IOS Downloads
• The Line Card may already have enough code in its flash to become operational on the Switch Fabric, or it may require an Mbus download.
• Only enough code for the Line Card to become operational on the fabric will be loaded using the Mbus.
• Once all cards are operational on the fabric, the fabric is initialized and the main IOS software is downloaded.
39© 2001, Cisco Systems, Inc. All rights reserved.© 2001, Cisco Systems, Inc. All rights reserved.© 2002, Cisco Systems, Inc. All rights reserved.
IPC Services
© 2001, Cisco Systems, Inc. All rights reserved. 40© 2001, Cisco Systems, Inc. All rights reserved. 40© 2002, Cisco Systems, Inc. All rights reserved. 40
IPC Overview
• The 12k is a distributed multiprocessor system. The processors communicate via IPC … an essential architectural service
• IPC has a reliable (acknowledged) and unreliable mode of transport (with or without sequence number or notification). The application uses an appropriate method.
© 2001, Cisco Systems, Inc. All rights reserved. 41© 2001, Cisco Systems, Inc. All rights reserved. 41© 2002, Cisco Systems, Inc. All rights reserved. 41
IPC Clients
• Applications (clients) can build their own queue structures and feed the IPC queue/cache as well as choose to block or not until an ACK or imbedded response is received.
e.g. … CEF uses a multi-priority queue and it’s own cache in front of the IPC queue (controlled by “ip cef linecard ipc memory”) … it’s got it’s own message handling routines defined in the same registry as direct IPC interrupt or process level message handling.
Many (most) applications use the CEF packaging (XDR) message types and queues as an interface to IPC.
e.g. … route-map updates and acl updates to linecards
• Applications are also responsible for being “well-behaved”.
• Utility applications like slavelog and slavecore use IPC directly.
42© 2001, Cisco Systems, Inc. All rights reserved.© 2001, Cisco Systems, Inc. All rights reserved.© 2002, Cisco Systems, Inc. All rights reserved.
Module 2 – Forwarding Architecture
43© 2001, Cisco Systems, Inc. All rights reserved.© 2001, Cisco Systems, Inc. All rights reserved.© 2002, Cisco Systems, Inc. All rights reserved.
Route Processor
© 2001, Cisco Systems, Inc. All rights reserved. 44© 2001, Cisco Systems, Inc. All rights reserved. 44© 2002, Cisco Systems, Inc. All rights reserved. 44
The Route Processor (RP)
• The RP’s control path for Line Cards uses IPC via the switch fabric or Mbus
• The switch fabric connection is the main data path for route table distribution
• The Mbus connection enables the RP to download a bootstrap image, collect or load diagnostic information, and perform general maintenance operations
© 2001, Cisco Systems, Inc. All rights reserved. 45© 2001, Cisco Systems, Inc. All rights reserved. 45© 2002, Cisco Systems, Inc. All rights reserved. 45
RP Responsibilities
• Running routing protocols
• Builds and distributes the routing tables to Line Cards (i.e. routing table maintenance)
• Provides general maintenance functions (i.e. Booting Line Card processors)
© 2001, Cisco Systems, Inc. All rights reserved. 46© 2001, Cisco Systems, Inc. All rights reserved. 46© 2002, Cisco Systems, Inc. All rights reserved. 46
RP Routing Table Maintenance
• Using the RIB the RP maintains a complete forwarding table of its own (RP-FIB)
• Routing updates are forwarded from RP-RIB to each Line Card (LC-FIB)
• Each LC-FIB entry corresponds to an interface which contains a MAC encapsulation string, output interface and MTU
© 2001, Cisco Systems, Inc. All rights reserved. 47© 2001, Cisco Systems, Inc. All rights reserved. 47© 2002, Cisco Systems, Inc. All rights reserved. 47
RP Routing Table Maintenance
• FIB distribution is done through reliable IPC updates
• When the routing protocol triggers an update, it is placed into the FIB of the RP then sent to the Line Cards
• Updates are unicast across the fabric to all Line Cards
48© 2001, Cisco Systems, Inc. All rights reserved.© 2001, Cisco Systems, Inc. All rights reserved.© 2002, Cisco Systems, Inc. All rights reserved.
GRP
© 2001, Cisco Systems, Inc. All rights reserved. 49© 2001, Cisco Systems, Inc. All rights reserved. 49© 2002, Cisco Systems, Inc. All rights reserved. 49
Major Components: The GRP
• R5000 CPU (a.k.a. P4)
• Mbus Module
• Tiger ASIC
• CSAR ASIC
• FIA ASIC
• SLI ASIC
• Power Modules
© 2001, Cisco Systems, Inc. All rights reserved. 50© 2001, Cisco Systems, Inc. All rights reserved. 50© 2002, Cisco Systems, Inc. All rights reserved. 50
GRP ComponentsPower Units
DRAM
Tiger ASIC
CPU
Fabric Interface ASIC (FIA)
Serial Line Interface ASIC (SLI)
Cisco cell Segment And Reassembly (CSAR)
© 2001, Cisco Systems, Inc. All rights reserved. 51© 2001, Cisco Systems, Inc. All rights reserved. 51© 2002, Cisco Systems, Inc. All rights reserved. 51
GRP Component Groups
I/O Sub-system
Fabric
Mbus
Logic
© 2001, Cisco Systems, Inc. All rights reserved. 52© 2001, Cisco Systems, Inc. All rights reserved. 52© 2002, Cisco Systems, Inc. All rights reserved. 52
CSAR (ciscoCell Segmentation and Reassembly) ASIC
• Buffer manager ASIC for the GRP (equivalent to Rx and Tx BMA on Engine 0 LCs)
• The CSAR contains two 64k buffers
• Messages are placed in a hold queue if these buffers are full
• An interrupt is sent to the CPU when the buffers are free
• The CSAR contains 32 reassembly areas when receiving ciscoCells from the fabric for unicast and multicast providing 64 areas
• Connects to fabric at OC12
53© 2001, Cisco Systems, Inc. All rights reserved.© 2001, Cisco Systems, Inc. All rights reserved.© 2002, Cisco Systems, Inc. All rights reserved.
Performance RP 1(PRP-1)
© 2001, Cisco Systems, Inc. All rights reserved. 54© 2001, Cisco Systems, Inc. All rights reserved. 54© 2002, Cisco Systems, Inc. All rights reserved. 54
PRP-1 Architecture
CHOPPERXCVR
ASSEMBLER
2M L3Cache
Main(RAM)
Memory
CPUVoyager
(PPC7450)
I/O Bus
SystemController
(Discovery)
XCVR
XCVR
XCVR
XCVR
MBUS2x10/100 DUART PCMCIA NVRAMBootflash BootPROM
FUSILLI
64@133Mhz
© 2001, Cisco Systems, Inc. All rights reserved. 55© 2001, Cisco Systems, Inc. All rights reserved. 55© 2002, Cisco Systems, Inc. All rights reserved. 55
Performance Route Processor (PRP)
• The PRP is fully compatible with the GRP at the hardware level
• One of the major differences with the PRP is the use of the V’ger processor, a Motorola PPC processor running at 655MHz
• The future Apollo processor running at 1GHz will replace V’ger
• Connects to fabric at OC48 – requires at least 1CSC and 3 SFCs to operate
© 2001, Cisco Systems, Inc. All rights reserved. 56© 2001, Cisco Systems, Inc. All rights reserved. 56© 2002, Cisco Systems, Inc. All rights reserved. 56
Performance Route Processor (PRP)
• The PPC CPU also supports on-chip 32Kbs of Layer 1 cache and on-chip 256Kb of Layer 2 cache with an external 2MB of Layer 3 cache controller.
• The realized performance improvement is 4 – 5 times that of the current GRP
© 2001, Cisco Systems, Inc. All rights reserved. 57© 2001, Cisco Systems, Inc. All rights reserved. 57© 2002, Cisco Systems, Inc. All rights reserved. 57
Performance Route Processor (PRP)
• Default 512Mb DRAM upgradeable to 2Gb
• 2 10/100 Enet ports
• RJ-45 Console port
• 64Mb Flash Disk as standard
58© 2001, Cisco Systems, Inc. All rights reserved.© 2001, Cisco Systems, Inc. All rights reserved.© 2002, Cisco Systems, Inc. All rights reserved.
Line card concepts
© 2001, Cisco Systems, Inc. All rights reserved. 59© 2001, Cisco Systems, Inc. All rights reserved. 59© 2002, Cisco Systems, Inc. All rights reserved. 59
Line Card Concepts
Components:
• PLIM - Physical Layer
Optics, Framer, SAR, etc.
• Layer 3 Forwarding Engine
IP/MPLS Switching and Services
• Fabric Interface
Transmission
Physical Layer
(Optics)
Physical Layer
(Optics)
Layer 3 Engine
Layer 3 Engine
Fabric InterfaceFabric
Interface
RXRX
TXTX
CPUCPU
To Fabric
To Fabric
From FabricFrom Fabric
© 2001, Cisco Systems, Inc. All rights reserved. 60© 2001, Cisco Systems, Inc. All rights reserved. 60© 2002, Cisco Systems, Inc. All rights reserved. 60
• Handle L2 protocol encap/decap
- SONET/SDH framing- ATM cell segmentation/re-assembly- Channelization
• Receives packet off the wire and passes it to the forwarding engine
PLIM – Physical Interfaces
© 2001, Cisco Systems, Inc. All rights reserved. 61© 2001, Cisco Systems, Inc. All rights reserved. 61© 2002, Cisco Systems, Inc. All rights reserved. 61
• Runs IOS and maintains CEF tables
• Provides CEF switching services, feature capabilities
• Provides queuing and QoS services (through the RX and TX queue managers)
NOTE – QoS will be covered in detail in the ‘Applications section’
FE - Forwarding Engine
© 2001, Cisco Systems, Inc. All rights reserved. 62© 2001, Cisco Systems, Inc. All rights reserved. 62© 2002, Cisco Systems, Inc. All rights reserved. 62
• Provides fabric transmission services
• Two components:
1. FIA – interface between forwarding engine and fabric interface
2. SLI - does 8B/10B encoding and decoding of Cisco cells
FIM - Fabric Interface Module
63© 2001, Cisco Systems, Inc. All rights reserved.© 2001, Cisco Systems, Inc. All rights reserved.© 2002, Cisco Systems, Inc. All rights reserved.
Line Card Concepts:A Reference Architecture
© 2001, Cisco Systems, Inc. All rights reserved. 64© 2001, Cisco Systems, Inc. All rights reserved. 64© 2002, Cisco Systems, Inc. All rights reserved. 64
Line Card Reference Architecture
PLIMForwarding &
Feature Complex
CPU
ToFab queue mgr
ToFab packet memory
FrFab queue mgr
FrFab packet memory
ToFabFabric
Interface
ForwardingLookup Tables
FrFabFabric
Interface
© 2001, Cisco Systems, Inc. All rights reserved. 65© 2001, Cisco Systems, Inc. All rights reserved. 65© 2002, Cisco Systems, Inc. All rights reserved. 65
Various routing protocols maintain individual routing databases.
Forwarding Architecture
The routing table is built by using the best available paths from the routing protocols.
From the IP routing table, we pre-resolve recursive routes and build the CEF table (a.k.a. FIB table)
The CEF table is pushed down from the GRP to each linecard via IPC
From the CEF table, HW-based linecards will build their own hardware forwarding tables
© 2001, Cisco Systems, Inc. All rights reserved. 66© 2001, Cisco Systems, Inc. All rights reserved. 66© 2002, Cisco Systems, Inc. All rights reserved. 66
Summary
• Multiple levels of routing/forwarding information
• RP provides control plane services
- IP routing protocols- MPLS label exchange protocols
• RP maintains RIB, FIB, LFIB
• LC have a copy of FIB and LFIB
E2/3/4/4+/6 have a HW forwarding FIB and LFIB as well
67© 2001, Cisco Systems, Inc. All rights reserved.© 2001, Cisco Systems, Inc. All rights reserved.© 2002, Cisco Systems, Inc. All rights reserved.
Engine Architectures
© 2001, Cisco Systems, Inc. All rights reserved. 68© 2001, Cisco Systems, Inc. All rights reserved. 68© 2002, Cisco Systems, Inc. All rights reserved. 68
Line Card - Switching Engines & ASICs
• Engine 0 – BMA – 622Mb
• Engine 1 - Salsa/BMA48 – 2.5Gb
• Engine 2 - PSA/TBM/RBM – 2.5Gb
• Engine 3 (aka ISE) – Alpha/Conga/Radar – 2.5Gb
• Engine 4 – RX/MCC/TX – 10Gb
• Engine 4+ - RX+/MCC/TX+ - 10Gb
• Engine 6 – Hermes/Ares/Hera – 20 Gb
69© 2001, Cisco Systems, Inc. All rights reserved.© 2001, Cisco Systems, Inc. All rights reserved.© 2002, Cisco Systems, Inc. All rights reserved.
Engine 0 Architecture
© 2001, Cisco Systems, Inc. All rights reserved. 70© 2001, Cisco Systems, Inc. All rights reserved. 70© 2002, Cisco Systems, Inc. All rights reserved. 70
Engine 0 - Components
• R5000 CPU + L3FE ASIC
• BMA
QoS support with performance hit
• Main Memory
Up to 256MB of DRAM
• Packet Memory
Up to 256MB SDRAM split equally between Rx and Tx
© 2001, Cisco Systems, Inc. All rights reserved. 71© 2001, Cisco Systems, Inc. All rights reserved. 71© 2002, Cisco Systems, Inc. All rights reserved. 71
Engine 0 Architecture
ToFabBMA
ToFabBMA
L3FEL3FE
CPUCPU
FrFabBMA
FrFabBMA
PacketMemoryPacket
Memory
PacketMemoryPacket
Memory
LC
IOS
Me
mo
ryL
C IO
S M
em
ory ToFab
FIAToFab
FIA SLISLI
FrFabFIA
FrFabFIA
SLISLI
PLIM L3 Engine Fabric Interface
RxPOSRx
POS
TxPOSTx
POS
FramerFramerOpticsOpticsX
C
V
R
S
© 2001, Cisco Systems, Inc. All rights reserved. 72© 2001, Cisco Systems, Inc. All rights reserved. 72© 2002, Cisco Systems, Inc. All rights reserved. 72
1 port OC12 Engine 0 line card
Optics
Mbus Agent Module
L3FE
SLI
FIA
RxBMARx Packet Memory
TxBMA
Tx Packet Memory
CPU
© 2001, Cisco Systems, Inc. All rights reserved. 73© 2001, Cisco Systems, Inc. All rights reserved. 73© 2002, Cisco Systems, Inc. All rights reserved. 73
Engine 0 – OC12 with Features
• CPU-based switching
• Provides OC-12 performance with features
• Extensible/flexible architecture- easy to add more features
• WRED/MDRR in HW with performance hit
• Performance:
No features - ~ 420 kpps
With features - ~ 250 kpps
74© 2001, Cisco Systems, Inc. All rights reserved.© 2001, Cisco Systems, Inc. All rights reserved.© 2002, Cisco Systems, Inc. All rights reserved.
Engine 1 Architecture
© 2001, Cisco Systems, Inc. All rights reserved. 75© 2001, Cisco Systems, Inc. All rights reserved. 75© 2002, Cisco Systems, Inc. All rights reserved. 75
Engine 1 - Components
• R5000 CPU + Salsa ASIC
Salsa = Enhanced Layer 3 Fetch Engine (L3FE)
Hardware IP lookup with software re-write
• BMA48
Performance enhanced BMA
No QoS support
• Main Memory
Up to 256MB of DRAM
• Packet Memory
Up to 256MB SDRAM split equally between Rx and Tx
© 2001, Cisco Systems, Inc. All rights reserved. 76© 2001, Cisco Systems, Inc. All rights reserved. 76© 2002, Cisco Systems, Inc. All rights reserved. 76
ToFabBMA48ToFabBMA48
SalsaSalsa
CPUCPU
FrFabBMA48FrFab
BMA48
PacketMemoryPacket
Memory
PacketMemoryPacket
Memory
LC
IOS
Me
mo
ryL
C IO
S M
em
ory ToFab
FIA48ToFabFIA48 SLISLI
FrFabFIA48FrFabFIA48
SLISLI
PLIM L3 Engine Fabric Interface
RxSOPRx
SOP
TxSOPTx
SOP
X
C
V
R
S
Engine 1 Architecture
OpticsOptics
GigaMACGigaMAC
RxTrans
RxTrans
TxTrans
TxTrans
© 2001, Cisco Systems, Inc. All rights reserved. 77© 2001, Cisco Systems, Inc. All rights reserved. 77© 2002, Cisco Systems, Inc. All rights reserved. 77
1 port GigE Engine 1 line card
Rx BMA48
Salsa
Tx BMA48
CPU
© 2001, Cisco Systems, Inc. All rights reserved. 78© 2001, Cisco Systems, Inc. All rights reserved. 78© 2002, Cisco Systems, Inc. All rights reserved. 78
Engine 1- Salsa
• Hardware enhancements to IP packet validation and FIB lookup assist
• Verify
• packet is IPv4 packets with no options.
• Identify that packet is PPP/HDLC encapsulated.
• checksum, length, TTL
• Update IP header (TTL, checksum)
• Perform IP lookup and cache FIB pointer for CPU re-write operation
79© 2001, Cisco Systems, Inc. All rights reserved.© 2001, Cisco Systems, Inc. All rights reserved.© 2002, Cisco Systems, Inc. All rights reserved.
E0/1 - Life of a Packet:Watching the Queues
© 2001, Cisco Systems, Inc. All rights reserved. 80© 2001, Cisco Systems, Inc. All rights reserved. 80© 2002, Cisco Systems, Inc. All rights reserved. 80
Qnum Head Tail #Qelem LenThresh---- ---- ---- ------ ---------
4 non-IPC free queues:26626/26626 (buffers specified/carved), 50.90%, 80 byte data size1 101 26726 26625 65535 16184/16184 (buffers specified/carved), 30.94%, 608 byte data size2 26727 42910 16184 65535 7831/7831 (buffers specified/carved), 14.97%, 1568 byte data size3 42911 50741 7831 65535
IPC Queue:100/100 (buffers specified/carved), 0.19%, 4112 byte data size30 67 66 100 65535
Raw Queue:31 0 0 0 65535
ToFab Queues:Slot0 0 0 0 65535 1 0 0 0 655352 0 0 0 65535 3 0 0 0 65535 4 0 0 0 65535 5 0 0 0 65535 6 0 0 0 65535 7 0 0 0 65535
Mcast 0 0 0 65535
Packet Arrives on Line Card (tofab)
© 2001, Cisco Systems, Inc. All rights reserved. 81© 2001, Cisco Systems, Inc. All rights reserved. 81© 2002, Cisco Systems, Inc. All rights reserved. 81
Move the Buffer onto the Raw Q (tofab)Qnum Head Tail #Qelem LenThresh---- ---- ---- ------ ---------
4 non-IPC free queues:26626/26626 (buffers specified/carved), 50.90%, 80 byte data size1 101 26726 26625 65535 16184/16184 (buffers specified/carved), 30.94%, 608 byte data size2 26727 42910 16184 65535 7831/7831 (buffers specified/carved), 14.97%, 1568 byte data size3 42911 50741 7831 65535
IPC Queue:100/100 (buffers specified/carved), 0.19%, 4112 byte data size30 67 66 100 65535
Raw Queue:31 0 0 1 65535
ToFab Queues:Slot0 0 0 0 65535 1 0 0 0 655352 0 0 0 65535 3 0 0 0 65535 4 0 0 0 65535 5 0 0 0 65535 6 0 0 0 65535 7 0 0 0 65535
Mcast 0 0 0 65535
© 2001, Cisco Systems, Inc. All rights reserved. 82© 2001, Cisco Systems, Inc. All rights reserved. 82© 2002, Cisco Systems, Inc. All rights reserved. 82
Qnum Head Tail #Qelem LenThresh---- ---- ---- ------ ---------
4 non-IPC free queues:26626/26626 (buffers specified/carved), 50.90%, 80 byte data size1 101 26726 26625 65535 16184/16184 (buffers specified/carved), 30.94%, 608 byte data size2 26727 42910 16184 65535 7831/7831 (buffers specified/carved), 14.97%, 1568 byte data size3 42911 50741 7831 65535
IPC Queue:100/100 (buffers specified/carved), 0.19%, 4112 byte data size30 67 66 100 65535
Raw Queue:31 0 0 0 65535
ToFab Queues:Slot0 0 0 0 65535 1 0 0 0 655352 0 0 0 65535 3 0 0 0 65535 4 0 0 0 65535 5 0 0 0 65535 6 0 0 1 65535 7 0 0 0 65535
Mcast 0 0 0 65535
FIB Result and ToFab Queuing (tofab)
© 2001, Cisco Systems, Inc. All rights reserved. 83© 2001, Cisco Systems, Inc. All rights reserved. 83© 2002, Cisco Systems, Inc. All rights reserved. 83
Qnum Head Tail #Qelem LenThresh---- ---- ---- ------ ---------
4 non-IPC free queues:26626/26626 (buffers specified/carved), 50.90%, 80 byte data size1 101 26726 26626 65535 16184/16184 (buffers specified/carved), 30.94%, 608 byte data size2 26727 42910 16184 65535 7831/7831 (buffers specified/carved), 14.97%, 1568 byte data size3 42911 50741 7831 65535
IPC Queue:100/100 (buffers specified/carved), 0.19%, 4112 byte data size30 67 66 100 65535
Raw Queue:31 0 0 0 65535
ToFab Queues:Slot0 0 0 0 65535 1 0 0 0 655352 0 0 0 65535 3 0 0 0 65535 4 0 0 0 65535 5 0 0 0 65535 6 0 0 0 65535 7 0 0 0 65535
Mcast 0 0 0 65535
Return the Buffer to the Free Q (tofab)
© 2001, Cisco Systems, Inc. All rights reserved. 84© 2001, Cisco Systems, Inc. All rights reserved. 84© 2002, Cisco Systems, Inc. All rights reserved. 84
Qnum Head Tail #Qelem LenThresh---- ---- ---- ------ ---------
4 non-IPC free queues:26560/26560 (buffers specified/carved), 50.90%, 80 byte data size1 101 26660 26559 65535 16144/16144 (buffers specified/carved), 30.94%, 608 byte data size2 26661 42804 16144 65535 7811/7811 (buffers specified/carved), 14.97%, 1568 byte data size3 42805 50615 7811 65535 1562/1562 (buffers specified/carved), 2.99%, 4544 byte data size4 50616 52177 1562 65535
IPC Queue:100/100 (buffers specified/carved), 0.19%, 4112 byte data size30 78 77 100 65535
Raw Queue:31 0 83 0 65535
Interface Queues:0 0 0 0 65535 1 0 0 0 65535 2 0 0 0 65535 3 0 0 0 65535
Egress Card Receives the Packet (frfab)
© 2001, Cisco Systems, Inc. All rights reserved. 85© 2001, Cisco Systems, Inc. All rights reserved. 85© 2002, Cisco Systems, Inc. All rights reserved. 85
Queuing for Transmission (frfab)Qnum Head Tail #Qelem LenThresh---- ---- ---- ------ ---------
4 non-IPC free queues:26560/26560 (buffers specified/carved), 50.90%, 80 byte data size1 101 26660 26559 65535 16144/16144 (buffers specified/carved), 30.94%, 608 byte data size2 26661 42804 16144 65535 7811/7811 (buffers specified/carved), 14.97%, 1568 byte data size3 42805 50615 7811 65535 1562/1562 (buffers specified/carved), 2.99%, 4544 byte data size4 50616 52177 1562 65535
IPC Queue:100/100 (buffers specified/carved), 0.19%, 4112 byte data size30 78 77 100 65535
Raw Queue:31 0 83 0 65535
Interface Queues:0 0 0 1 65535 1 0 0 0 65535 2 0 0 0 65535 3 0 0 0 65535
© 2001, Cisco Systems, Inc. All rights reserved. 86© 2001, Cisco Systems, Inc. All rights reserved. 86© 2002, Cisco Systems, Inc. All rights reserved. 86
Return the Buffer to the Free Q (frfab)Qnum Head Tail #Qelem LenThresh---- ---- ---- ------ ---------
4 non-IPC free queues:26560/26560 (buffers specified/carved), 50.90%, 80 byte data size1 101 26660 26560 65535 16144/16144 (buffers specified/carved), 30.94%, 608 byte data size2 26661 42804 16144 65535 7811/7811 (buffers specified/carved), 14.97%, 1568 byte data size3 42805 50615 7811 65535 1562/1562 (buffers specified/carved), 2.99%, 4544 byte data size4 50616 52177 1562 65535
IPC Queue:100/100 (buffers specified/carved), 0.19%, 4112 byte data size30 78 77 100 65535
Raw Queue:31 0 83 0 65535
Interface Queues:0 0 0 0 65535 1 0 0 0 65535 2 0 0 0 65535 3 0 0 0 65535
87© 2001, Cisco Systems, Inc. All rights reserved.© 2001, Cisco Systems, Inc. All rights reserved.© 2002, Cisco Systems, Inc. All rights reserved.
Engine 2 Architecture
© 2001, Cisco Systems, Inc. All rights reserved. 88© 2001, Cisco Systems, Inc. All rights reserved. 88© 2002, Cisco Systems, Inc. All rights reserved. 88
Engine 2 Overview
• First programmable, hardware-based forwarding engine
• Multi-million PPS with some features
• Up to 4Mpps performance (no features)
© 2001, Cisco Systems, Inc. All rights reserved. 89© 2001, Cisco Systems, Inc. All rights reserved. 89© 2002, Cisco Systems, Inc. All rights reserved. 89
RBMRBM
PacketMemoryPacket
Memory
Engine 2 Architecture
PSAPSA
TBMTBM
PacketMemoryPacket
Memory
PSAMemory
PSAMemory
ToFabFIA48ToFabFIA48 SLISLI
FrFabFIA48FrFabFIA48
SLISLI
PLIM L3 Engine Fabric Interface
RxSOPRx
SOP
TxSOPTx
SOP
FramerFramerOpticsOpticsX
C
V
R
S
LC
IOS
Me
mo
ry
SalsaSalsaCPUCPU
© 2001, Cisco Systems, Inc. All rights reserved. 90© 2001, Cisco Systems, Inc. All rights reserved. 90© 2002, Cisco Systems, Inc. All rights reserved. 90
1 port OC48 POS Engine 2 line card
TBM
RBM
PSA
© 2001, Cisco Systems, Inc. All rights reserved. 91© 2001, Cisco Systems, Inc. All rights reserved. 91© 2002, Cisco Systems, Inc. All rights reserved. 91
Engine 2 - Components
• R5000 CPU -> Slow Path
Slow path (CPU computed) CEF tables, ICMPs, IP options, etc…
• PSA (Packet Switched ASIC) -> Fast Path
Microcoded IP/MPLS lookup & feature processing
• RBM/TBM (Receive/Transmit Buffer Manager)
Hardware WRED, MDRR
• Packet Memory
256MB SDRAM can be upgraded to 512MB SDRAM
• PSA Memory
PSA copy of FIB table
© 2001, Cisco Systems, Inc. All rights reserved. 92© 2001, Cisco Systems, Inc. All rights reserved. 92© 2002, Cisco Systems, Inc. All rights reserved. 92
RBMRBM
PacketMemoryPacket
Memory
Engine 2 – Rx Packet flow
PSAPSA
TBMTBM
PacketMemoryPacket
Memory
PSAMemory
PSAMemory
ToFabFIA48ToFabFIA48 SLISLI
FrFabFIA48FrFabFIA48
SLISLI
PLIM L3 Engine Fabric Interface
RxSOPRx
SOP
TxSOPTx
SOP
FramerFramerOpticsOpticsX
C
V
R
S
LC
IOS
Me
mo
ry
SalsaSalsaCPUCPU
•SONET/SDH framer
•Extract packets from SONET/SDH payload
•Pass indication of input interface and packet header to PSA
•Payload passed to RBM
•Packet validation •IP/MPLS lookup
•Feature processing (ACLs, CAR, Netflow, etc...)
•Append buffer header
•Determine loq, oq and freeq for packet
•Tofab queueing
•WRED
•MDRR
•Segment packet into ciscoCell
•add CRC to ciscoCell
•send transmission request to SCA
•8B/10B encoding
•send cells to fabric
© 2001, Cisco Systems, Inc. All rights reserved. 93© 2001, Cisco Systems, Inc. All rights reserved. 93© 2002, Cisco Systems, Inc. All rights reserved. 93
Engine 2 – PSA Forwarding
• The Packet Switching ASIC is an IP and TAG forwarding engine
• The ASIC contains a 6 stage pipeline, Pointer and Table Lookup memory
• As packets move through the PSA pipeline, the forwarding decision and feature processing is completed
© 2001, Cisco Systems, Inc. All rights reserved. 94© 2001, Cisco Systems, Inc. All rights reserved. 94© 2002, Cisco Systems, Inc. All rights reserved. 94
PSA Architecture
MAC header checking, protocol ID checking, IP header checking, extraction of IP/MPLS address fields
Microcode engine which performs checks on the packet (protocol, length, TTL, IP CHKSUM) and extracts the appropriate address(es) for the main lookup. Some feature processing.
IP/MPLS lookup machine
Adjacency Lookup, Per Adjacency Counters
Fetch PreP PLU TLU Gather
FIB TREE(256K) LEAVES/ADJ/STATS(256K)
PoP
Ext. SSRAM
Microcode engine which applies the results of the PLU/TLU lookup to the packet. Tasks include COS handling, MTU check, special case tests, setup of gather stage, feature processing, etc...
Modifications to packet header (e.g. pushing MPLS Labels). Prepare packet for transmission to RBM
Each stage has a 25 clock budget @100MHz = 250ns, i.e. 4Mpps
© 2001, Cisco Systems, Inc. All rights reserved. 95© 2001, Cisco Systems, Inc. All rights reserved. 95© 2002, Cisco Systems, Inc. All rights reserved. 95
RBM: Rx Queue Manager
• The RBM manages the linecard’s receive packet memory buffers and queues
• There are two major types of queues in RBM:
LowQs (16 FreeQs, 1 RAWQs, an IPC FreeQ and spare queues)
2048 unicast Output Queues and 8 multicast queues
• 16 slots per chassis, 16 ports per slot, 8 queues per port = 2048 queues
• One hpr (high priority) queue is allocated per destination slot/port.
© 2001, Cisco Systems, Inc. All rights reserved. 96© 2001, Cisco Systems, Inc. All rights reserved. 96© 2002, Cisco Systems, Inc. All rights reserved. 96
TBM: Tx Queue Manager
• The TBM manages the linecard’s transmit packet memory buffers and queues
• Three types of queues:
Non-IPC freeQs, 1 CPU RawQ, IPC FreeQ
128 Output Queues
Multicast RawQ
• 8 CoS queues per output port, 16 ports = 128 queues
© 2001, Cisco Systems, Inc. All rights reserved. 97© 2001, Cisco Systems, Inc. All rights reserved. 97© 2002, Cisco Systems, Inc. All rights reserved. 97
RBMRBM
PacketMemoryPacket
Memory
Engine 2 – Tx Packet flow
PSAPSA
TBMTBM
PacketMemoryPacket
Memory
PSAMemory
PSAMemory
ToFabFIA48ToFabFIA48 SLISLI
FrFabFIA48FrFabFIA48
SLISLI
PLIM L3 Engine Fabric Interface
RxSOPRx
SOP
TxSOPTx
SOP
FramerFramerOpticsOpticsX
C
V
R
S
LC
IOS
Me
mo
ry
SalsaSalsaCPUCPU
•Remove 8B/10B encoding
•Verify and remove CRC from ciscoCell
•send cells to TBM
•Re-assemble packet from ciscoCells
•FrFab queueing
•WRED
•MDRR
•Append L2 header and send packet to PLIM
•Mcast duplication
•Put packets in SONET/SDH payload
•SONET/SDH framer
© 2001, Cisco Systems, Inc. All rights reserved. 98© 2001, Cisco Systems, Inc. All rights reserved. 98© 2002, Cisco Systems, Inc. All rights reserved. 98
E2 Feature Support
• Designed to be forwarding ASIC on a “backbone” card, ie does not natively support any features
• Features like ACLs, SNF, BGP PA added later on, but take performance hit
• Most new features require a separate ucode load and are mutually exclusive
• Performance varies with features (eg. ACLs):128 line iACLs – 800kpps
128 line oACLs – 675 kpps
448 line iACLs – 690 kpps
448 line oACLs – 460 kpps
99© 2001, Cisco Systems, Inc. All rights reserved.
Session NumberPresentation_ID
Engine 3 - IP Services Engine (ISE)
100100100© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
ISE Overview
• Programmable, hardware-based forwarding engine
• Up to 4Mpps performance (with features)
Uses TCAMs for advanced feature processing
• Traffic shaping and advanced QoS support
Flexible mapping of queues
101101101© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
ISE – Architecture
ALPHA
PICANTECPU
CONGA
PacketMemory
FIB TableMemory
X
C
V
R
S
PLIM L3 Engine Fabric Interface
Optics
SPECTRA
RADAR
PacketMemory
LC
IOS
Me
m
ALPHA
SLIFIA
SLIFIA
FUSCILLIGULF
TCAM
TCAM
102102102© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
4xOC12 POS ISE Linecard
TX TCAM
RX TCAM
RX ALPHA
SPECTRA
GULFTX ALPHA
CONGA
FUSCILLI
R7K
PICANTE
RADAR
103103103© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
ISE – Rx Packet flow
ALPHA
PICANTECPU
CONGA
PacketMemory
FIB TableMemory
X
C
V
R
S
PLIM L3 Engine Fabric Interface
Optics
SPECTRA
RADAR
PacketMemory
LC
IOS
Me
m
ALPHA
SLIFIA
SLIFIA
FUSCILLIGULF
TCAM
TCAM
•Packet validation •IP/MPLS lookup
•Feature processing (ACLs, CAR, Netflow, etc...)
•Append buffer header
•Determine loq, oq and freeq for packet
•SONET/SDH framer
•Handle channelization
•Extract packets from SONET/SDH payload
•Pass indication of input interface and packet header to RX Alpha
•Payload based to Radar
•Tofab queueing
•Input rate shaping
•WRED
•MDRR
•Segment packet into ciscoCell
•add CRC to ciscoCell
•send transmission request to SCA
•8B/10B encoding
•send cells to fabric
104104104© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
ALPHA• Advanced Layer 3 Packet Handling ASIC
• Performs forwarding, classification, policing and accounting
• Two ALPHA chips, one in the receive path, one in the transmit path. This allows features to be implemented in both the ingress (RX) and egress (TX) paths
• 11 pipeline stages 3 micro-code stages for future expandability
• Utilizes TCAMs to perform high-speed feature processing. Each ALPHA has its own TCAM
ISE - ALPHA
105105105© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
11 stages of ALPHA Pipeline
Fetch PreP PLU PCM TLU
Gather
FIB TREELEAVES/ADJ/STATS
PoP
Ext. SSRAM
CAMP3 stages
Ext. CAM + SSRAM
MIP
MAC header checking, protocol ID checking, IP header checking, extraction of IP/MPLS address fields
Microcoded stage which is capable of any general purpose activity on the packet
MTRIE lookup machine
TCAM access for altering PLU results (PBR, MPLS)
Adjacency Lookup, Per Adjacency Counters
Microcoded stage which is capable of any general purpose activity on the packet
CAM Processor – Lookups for xACL (permit/deny), CAR token bucket maintenance, Netflow counters update
Processing packet structure, including stripping the old input encapsulation, stripping old MPLS labels if necessary, pushing new labels and computation of the new IP checksum
Microcoded stage – Performs feature actions, handling exception packets
106106106© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
RADAR: Rx Queue Manager
• The RADAR manages the linecard’s receive packet memory buffers and queues
• There are three major types of queues in RADAR:
LowQs (16 FreeQs, 3 RAWQs, an IPC FreeQ and spare queues)
2048 Input Shape Queues (rate-shaping)
2048 unicast Output Queues (16 unicast high priority queues)
and 8 multicast queues
• One local output-queue is allocated per destination interface
• One hpr (high priority) queue is allocated per destination slot
107107107© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
RADAR: Input Shape Queues
• There are 2048 queues dedicated to ingress traffic shaping each with an independent ‘leaky bucket’ circuit.
• Each flow can be shaped in increments of 64kbps (from 64kbps up to line rate)
108108108© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
RADAR: Rx Queue Manager
• The Rx ALPHA decides which type of queue will be used for each packet.
Rx PacketProcessor
IputShapeQueue
VirtualOutputQueue
SwitchFabric
Interface
RawQueue
LinecardCPU
Rx QueueManager
The Packet Processordecides which patheach packet will take
109109109© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
ISE – TX packet flow
ALPHA
PICANTECPU
CONGA
PacketMemory
FIB TableMemory
X
C
V
R
S
PLIM L3 Engine Fabric Interface
Optics
SPECTRA
RADAR
PacketMemory
LC
IOS
Me
m
ALPHA
SLIFIA
SLIFIA
FUSCILLIGULF
TCAM
TCAM
•Remove 8B/10B encoding
•Verify and remove CRC from ciscoCell
•send cells to TX ALPHA•Adjacency lookup
•Feature processing (ACLs, CAR, MQC, etc...)
•Update output_info field of buffer header with info from adjacency
•Re-assemble packet from ciscoCells
•FrFab queueing
•Output rate shaping
•WRED
•MDRR
•Append L2 header and send packet to PLIM
•Mcast duplication
•Handle channelization
•Put packets in SONET/SDH payload
•SONET/SDH framer
110110110© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
CONGA: Tx Queue Manager
• The CONGA manages the linecard’s transmit packet memory buffers and queues
• Three types of queues:
Non-IPC freeQs, 3 CPU RawQs, IPC FreeQ
2048 Output Queues (2 leaky buckets per queue for rate-shaping)
Multicast RawQ
• Output queues divided equally among output ports
• Support for 512 logical interfaces
• Max Bandwidth shaping per port, Min and Max Bandwidth shaping per queue
111111111© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
CONGA
• Each ‘Shaped Output’ queue has a built in dual-leaky bucket mechanism with a programmable maximum and minimum rate (I.e. a DRR bandwidth guarantee)
• A second level of shaping is available per ‘Port’.
Port 0 ShapedOutput Queues
HP
LP0
LPn
To PLIM
To PLIM
DRR
DRR
Port 511 ShapedOutput Queues
HP
LP0
LPnRepresentsA Shaper
112© 2001, Cisco Systems, Inc. All rights reserved.
Session NumberPresentation_ID
Engine 4+ - 10G Edge Services
113113113© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
RX+RX+
TX+TX+
PacketMemoryPacket
Memory
LookupMTRIE
LookupMTRIE
10GFIA
10GFIA
PLIM L3 Engine Fabric Interface
MCCMCC
PacketMemoryPacket
Memory
PicantePicanteCPUCPU
LC
IOS
Me
mo
ry SerDes
BACKPLANE
PHAD
(OpticalInterface
ASIC)
PHAD
(OpticalInterface
ASIC)
OpticsOptics
Engine4+ “Edge Services”
FramerFramer
114114114© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
Engine 4+ - Components
• R5000 CPU -> Slow Path
Slow path (CPU computed) CEF tables, ICMPs
• RX+ ASIC -> Fast Path
Hardware IP/MPLS lookup inc. Multicast, CAR, ACLs, MPLS-PE
• MCC ASIC
Hardware WRED, MDRR
• Packet Memory
256MB SDRAM can be upgraded to 512MB SDRAM and 1024 in future
• TX+ ASIC -> Fast Path
Hardware outbound traffic shaping, ACLs,
115115115© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
10x1 GigE Engine 4 linecard
TX
10x1GE PLIM
MCC
RXPicante
116116116© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
RX+ - Packet processing ASIC
• Non-programmable high-speed ASIC providing 25 Mpps switching capacity
• Virtual CAM (vCAM) for features
ACLs, CAR and PBR
• Line-rate for 40 byte packets at /32 FIB lookup
117117117© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
MCC – ToFab Queueing ASIC
• Manages receive packet memory and queues
• Three types of queues:
Non-IPC freeQs, 8 CPU RawQs, IPC FreeQ
2048 Unicast VOQs, 8 multicast VOQs
• One high priority queue per destination slot/port
118118118© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
RX+RX+
TX+TX+
PacketSRAMPacketSRAM
LookupMTRIE
LookupMTRIE
10GFIA
10GFIA
PLIM L3 Engine Fabric Interface
MCCMCC
PacketSDRAMPacketSDRAM
PicantePicanteCPUCPU
LC
IOS
Me
mo
ry SerDes
BACKPLANE
PHAD
(OpticalInterface
ASIC)
PHAD
(OpticalInterface
ASIC)
OpticsOptics
Engine4+ - RX Packet Flow
FramerFramer
• SONET/SDH framer
•Extract packets from SONET/SDH payload
•Protocol identification
•Verify packet length
•Append PLIM header
•IP unicast and multicast lookup
•MPLS lookup
•CAR/ACL feature processing
•Append buffer header and update loq, oq, and ideal freeq values
•Manage packet buffers
•Perform WRED, MDRR
•Packets segmented into cells
•Make packet transmission request
•Append ciscoCell CRC
• 8B/10B encoding
•Transmit cell over fabric
119119119© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
RX+RX+
TX+TX+
PacketSRAMPacketSRAM
LookupMTRIE
LookupMTRIE
10GFIA
10GFIA
PLIM L3 Engine Fabric Interface
MCCMCC
PacketSRAMPacketSRAM
PicantePicanteCPUCPU
LC
IOS
Me
mo
ry SerDes
BACKPLANE
PHAD
(OpticalInterface
ASIC)
PHAD
(OpticalInterface
ASIC)
OpticsOptics
Engine4+ - TX Packet Flow
FramerFramer
•PLIM header removed
•Packets segments queued to SONET channels
•Packets sent within SONET payloads (POS)
•Multicast packets duplicated
•Header re-written for output(MAC re-write)
•ACL/CAR performed
•RED/WRED performed
•Packets queued for output(16 ports. 8 queued/port)
•MDRR scheduling and output shaping performed
•Cells re-assembled into packets •CRC checked
•Packet header reconstructed
•Packets scheduled to TX
• SONET/SDH framer
120120120© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
TX+ - TX Queueing ASIC
• Manages transmit packet memory and queues
• Four types of queues:
Non-IPC freeQs, 8 CPU RawQs, IPC FreeQ,
Multicast Raw Queue
128 Unicast OQs, 8 multicast OQs
• Per-destination port LLQ
• Performs output CAR, rate-shaping
121© 2001, Cisco Systems, Inc. All rights reserved.
Session NumberPresentation_ID
Engine 6 – 20Gb Edge Services
122122122© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
HermesHermes
HERAHERA
PacketMemoryPacket
Memory
LookupMTRIE
LookupMTRIE
TFIA
FFIA
PLIM L3 Engine Fabric Interface
AresAres
PacketMemoryPacket
Memory
PicantePicanteCPUCPU
LC
IOS
Me
mo
ry
EROS
SERDES
BACKPLANE
Zeus
Framer& PHAD
Zeus
Framer& PHAD
OpticsOptics
Engine 6
TCAMTCAM
TCAMTCAMFFIA
TFIA
123123123© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
Engine 6 - Components
• RM7000 CPU -> Slow PathSlow path (CPU computed) CEF tables, ICMPs
• Hermes ASIC -> Fast Path – 50Mpps @ 40BytesHardware IP/MPLS lookup inc. Multicast, CAR, ACLs, MPLS-PE, 6-PE, PBR, Loose & Strict uRPF
• Ares ASICHardware WRED, MDRR
• Hera ASIC -> Fast PathHardware outbound traffic shaping, ACLs, SNF, Mcast
• 512Mb Dram Route Memory• 512Mb RLDRAM Packet Memory• TCAM4 ASICs
Attached to Hermes and Hera for feature processing
124124124© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
Engine6 2xOC192 layout
TCAM
Optics
Power
Supply
CPU
memory
CPU
PICANTE
Zeus
Hermes Ares
Eros
Hera
MBUS
TCAM
Optics
Zeus
TCAM
Hermes Ares
Eros
Hera
Power Supply
PICANTE
CPUCPU memory
MBUS
125125125© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
Engine6 8xOC48 layout
SFP Pluggable Optics
Surface mounted RLDRAMs
126126126© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
HermesHermes
HERAHERA
PacketMemoryPacket
Memory
LookupMTRIE
LookupMTRIE
TFIA
FFIA
PLIM L3 Engine Fabric Interface
AresAres
PacketMemoryPacket
Memory
PicantePicanteCPUCPU
LC
IOS
Me
mo
ry
EROS
SERDES
BACKPLANE
Zeus
Framer& PHAD
Zeus
Framer& PHAD
OpticsOptics
Engine 6
TCAMTCAM
TCAMTCAMFFIA
TFIA
Framer + PHAD integrated•Layer-1 processing alarms ,crc check, APS..•Pkts buffering•Verify packet length•Append PLIM header
•IP/MPLS Lookup•TCAM based feature processing (ACL/CAR/PBR/VRFs)•PKT MOD TTL adj, ToS adj,IP checksum adj.•Append buffer header and update loq, oq etc..
•Queueing ASIC •Manage packet buffers•Perform WRED, MDRR
•TFIA + FFIA ASIC•Packets segmented into cells•Make packet transmission request•Append ciscoCell CRC
•Cells re-assembled into packets CRC checked•Packet header reconstructed•Packets scheduled to TX
•Multicast packets duplication•MAC rewrite for output•TCAM based output feature - ACL/CAR•Output packet queuing•RED/WRED performed•MDRR scheduling •Output traffic shaping
•Layer1 processing•PLIM header removed•Packets segments queued to SONET channels•Packets sent within SONET payloads (POS)
127127127© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
TCAM based feature
• TCAM used to implement key features
• 32000 ingress entries, 32000 egress entries shared between…
ACL
CAR – 32 car rules per port
PBR
VRF Selection
• Security ACLs are not merged
128128128© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
TCAM Basics
• TCAM - Ternary Content Addressable Memory – Match on 0 , 1 , X (don’t care)
• ACL/CAR/PBR/VRFs rules from CLI converted into Value Mask Result (VMR) format to be inserted in TCAM
Value cells – key values
ACL/CAR/PBR/VRFs values
Mask cells = Significant Value Bits to be matched
Result = Value && Mask - Action
Security ACL – permit/deny
CAR - Pointer to CAR buckets
PBR – Adjacency
VRF Selection – VRF Root
129129129© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
QoS flow
ACL iCAR WRED
drop drop dropCOS Queues
WRED oCAR ACL
drop drop dropCOS Queues
MDRR
Shaping
Hermes Ares
Hera
MDRR
tofab
frfab
130130130© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
QoS support
• 2064 tofab queues
16x16x8 = 2048 unicast queues
8 local CPU queues
8 multicast queues
Per priority queue per destination port
• 136 frfab queues
16x8 = 128 unicast queues
8 local CPU queues
Per priority queue per port
131131131© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID 131131131© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
132132132© 2001, Cisco Systems, Inc. All rights reserved.Presentation_ID
Engine 4+ - Line Card Family
• OC192 POS
• 4 x OC48 POS
• 1 x 10 GE
• 10 x 1 GE - EOS
• Modular GE
• 2 x OC48 DPT*
• OC192 DPT*