Washington WASHINGTON UNIVERSITY IN ST LOUIS [email protected] LC/NPE Substrate Control: Substrate Control Daemon Fred Kuhns [email protected] Applied Research Laboratory Washington University in St. Louis
Jan 19, 2016
WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
fredkarlwustledu
LCNPE Substrate ControlSubstrate Control Daemon
Fred Kuhnsfredkarlwustledu
Applied Research Laboratory
Washington University in St Louis
2WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Single Interface Example
wrr
qxsn
qxs1
qxs2 SchedNPE1
qpsn
qps1
qps2
SchedGPE1
qBE
Ingress
Map fpMI to Qdip dp pr
wrr
SchedCP
qsn
qs1
qs2
wrr
qGPE
qCP
SchedI1
Egress
src addrproto
porticmp
LC
inte
rfac
e 1
GPE
bull LC Ingressndash One queue per slice with reserved bandwidth
(really one per scheduler)
ndash One queue for best effort traffic to each GPE
ndash One scheduler for CP with queues for reserved traffic plus BE
bull LC Egressndash At least one scheduler for each physical interface
ndash One queue for each active slice with MI defined for the associated scheduler
ndash One best effort queue for each board (GPE CP NPE)
bull Substrate sets scheduler rates according to aggregate allocations
ndash Manage scheduler rates to control aggregate traffic to interfaces and boards
BWI1
GPE
CP
NPE
NPE
3WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Common Definitions and Typesbull See slides titled Typesppt
bull For all commands the message context ID is used to identify the context within which a command is to be executedndash The special value of 0 (cid = 0) indicates a privileged operation
performed by the substrate
ndash Otherwise the context ID is an identifier indicating a user specific context in the command is to be interpreted and executing In most cases this is either the fastpath ID or an internal slice ID
bull uint16_t miid Meta-Interface numberndash If fpid == 0 then miid is the endpoint ID (epid)
bull int32_t retCode_t Most calls return success (0) or an error codendash If a command results in an error then the first 4-Byte of the reply
message contains an error code
ndash If a command doesnrsquot return any data then it should at least return a retCode_t code of 0 but this is not required
4WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE Tables
VLAN
VLAN Table
031
copt (8b)not used81624
Start of SRAM Block (Physical Address)SRAM Block SizeMax Buffer Limit
Current Buffer CountBuffer Limit Exceeded Count
copt (8b)not usedStart of SRAM Block (Physical Address)
SRAM Block SizeMax Buffer Limit
Current Buffer CountBuffer Limit Exceeded Count
sid(Tunnel) Source IP Address
Destination MAC Address High Order 4 Bytes
Dest MAC Not Used
(Tunnel) Source IP Address
Destination MAC Address High Order 4 Bytes
Dest MAC Not Used
031 81624
Encap Control Block
VLAN
GPE Info table
031GPE IP Address (tunnel)
81624
NPE IP Address (tunnel)
Exception UDP SPort Exception UDP DPort Local UDP SPort Local UDP DPort
sid Exception qidqmnot usedsid Local Delivery qidqmnot used
(Tunnel) GPE IP Address(Tunnel) NPE IP Address
Exception UDP SPort Exception UDP DPort Local UDP SPort Local UDP DPort
sidqmnot usedsidqmnot used
Exception qidLocal Delivery qid
sidinterface number rate (Kbps683)
031 81624
Scheduler Rate table
interface number rate (Kbps683)
qidLength in Pkts (28b)
Length in Bytes (32b)
Threshold in Bytes (32b)
031 81624
QParams Table
unused
Quantum (32b)
Length in Pkts (28b)
Length in Bytes (32b)
Threshold in Bytes (32b)
unused
Quantum (32b)
5WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Line Card Tables or Both
sid destination Mac (high order 4 Bytes)
dest MAC Not Used
031 81624
IngressEgress Dynamic MAC Table
source Mac (high order 4 Bytes)
src MAC Not Used
destination Mac (high order 4 Bytes)
dest MAC Not Used
source Mac (high order 4 Bytes)
src MAC Not Used
sidinterface number rate (Kbps683)
031 81624
Scheduler Rate table
interface number rate (Kbps683)
qidLength in Pkts (28b)
Length in Bytes (32b)
Threshold in Bytes (32b)
031 81624
QParams Table
unused
Quantum (32b)
Length in Pkts (28b)
Length in Bytes (32b)
Threshold in Bytes (32b)
unused
Quantum (32b)
6WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Fast Path Commands SRM to SCDNPE Onlybull retCode_t set_fastpath(fpid copt_t VLAN rcnts_t Mem[])
Message context ID = 0 fpid Fast-Path ID Unique within SPP nodecopt Code option identifier 0 is invalid IPv4 = 1 I3 = 2uint16_t VLAN Ethernet VLAN tag used in datapath to identify fastpath instancercnts_t = cntr_t Qs Fltrs Buffers Statsstruct uint32_t offset size Mem[2] SRAM and DRAM allocation values are
relative to the start of an SRAM block set aside for fastpath useMem = sram(offset size) SRAM version 1 all same size dram(offset size) DRAM not used in version 1
ndash Create fastpath specific mappings for queues TCAM filters and the stats tablendash Initialize queue parameters to ldquoreasonablerdquo values (threshold weight)ndash Update VLAN Table with the code option ID (copt) and SRAM Address
bull SCD knows starting address of SRAM so add offsetndash Assume there is an SRAM table read by the microengines which maps VLAN to buffer limits
this may be the VLAN tablendash Treat as a transaction either it completes fully or no changes are recordeduint32_t retCode_t 0 = Success app defined
7WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Fast Path Commands SRM to SCDNPE Onlybull retCode_t enable_fastpath(fpid_t)
Message context ID = 0 fpid_t fast path identifieruint32_t retCode_t 0 = Success app definedndash enable by setting code option field of VLAN table entry for fp
bull retCode_t disable_fastpath(fpid)Message context ID = 0 uint16_t fpid fast path identifierretCode_t 0 = Success 1 = Pending 2 = Invalid fpid else error codendash First disable fp by setting code option field of VLAN table to 0ndash It is not an error to call disable an already disabled fastpath
bull SRM will periodically call disable_fastpath() until Success or Errorndash Then check all fast path queue lengths
bull If any queue has non-zero length then return Pendingbull If all queues are empty then return Success
bull retCode_t rem_fastpath(fpid_t)Message context ID = 0 fpid_t fast path identifierretCode_t 0 = Success 2 = Invalid fpid else error codendash Fastpath must be disabled (disable_fastpath) before it can be removedndash Remove all allocations associated with fpid
8WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
BW and Scheduler Control SRM to SCDbull Common Message
Message context ID = 0uint16_t sid Scheduler Id There are 5 schedulers per QM and 4 QMs so 0 lt= sid lt 20 (NBO)uint8_t MAC[6] 6 Byte Ethernet address
LC and NPE schedule parametersbull retCode_t set_sched_params(sid ifn BWmax BWmin)
int32_t ifn Interface numberuint32_t BWmax Kbps Maximum scheduler rateuint32_t BWmin Kbps Minimum assignable rate for this scheduler
ndash If BWmin is different from a previous call then may need to reassign weights for all associated queuesndash Must convert the rate I send (Kbps) to that expected by the hardware
Value = X Mbps 0683 = BW Kbps 683 (So no floating point needed)ndash Update scheduler parameters table in SRAM The low 16-bit word is the converted rate value and the upper 16-bits is the interface
number (ifn) The interface number must be the same value that TX uses To set only the Rate (preserves the existing interface number) SParams[sid] = (SchedParams[sid] amp 0xFFFF0000) | (0xFFFF amp Value)To set the interface number (ifn preserves the existing rate value) SParams[sid] = (SParams[sid] amp 0xFFFF) | (ifn ltlt 16)To set both SParams[sid] = (ifn ltlt 16) | (0xFFFF amp Value)
NPE Encap Control Blockbull retCode_t set_encap_cb(sid srcIP dMAC)
ndash Must update the table which associates encapsulation header IP source addresses with a schedulerSchedAddrs[sid] = ipaddr destination MAC
bull retCode_t create_mi(fpid mi sid)bull retCode_t delete_mi(fpid mi)bull retCode_t set_mi_bw(fpid mi bw)LC Dynamic MAC Tablebull retCode_t set_sched_mac(sid dstMAC srcMAC)
ndash Must update the table which associates Ethernet source and destination addresses with a schedulerSchedAddrs[sid]smac = smac (6 Bytes over 2 4B words)SchedAddrs[sid]dmac = dmac (6 Bytes over 2 4B words)
9WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
System Configuration SRM to SCD
bull Common Message context ID = 0
SRM to SCD NPE Onlybull retCode_t start_mes()
ndash The SCD starts the IXP microengines running the code that is automatically loaded when the SCD is launched
ndash Returns Success (0) Error (-1)
bull retCode_t stop_mes()ndash The SCD stops the IXP microengines if they are running
ndash Returns Success (0) Error (-1)
10WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE GPE Infobull Common Message context ID = 0
uint32_t IP 4-Byte IP address (NBO)
uint16_t PORT UDP Port number (NBO)
uint16_t QID Slice relative queue id must be converted by SCD to the tuple of qid QM id and scheduler ID
bull SRM to SCDNPE Only
retCode_t set_encap_gpe(fpid gpeIP npeIP)
retCode_t unset_encap_gpe(fpid)
unset_encap_gpe() == set_encap_gpe(fpid00)
uint16_t fpid Fast path id globally unique
bull RMP to SCDNPE Only
retCode_t set_gpe_info(exPort ldPort exQID ldQID)
retCode_t unset_gpe_info()
Set context ID to the global fast path ID
11WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Queuesbull Common
ndash if cid == 0 then qid is absolute otherwise SCD must convert from fastpath relativeuint16_t miid Meta-Interface numberuint32_t qlen Length of packet queue in Bytesuint32_t threshold the maximum number of packets queued before droppinguint32_t bw Kbs SCD must convert to the associated weightuint8_t list_type Explicit list (0) or range specification (1)uint16_t qid_list[] List of queue identifiers to associate with meta-interface mi In C or C++ this would
be an array of integers 0 n If using a range then the list has the first and one past the last id of the range For example 4 8 is the same as 4 5 6 7
bull RMP to SCDNPE Onlyndash retCode_t bind_queue(u16 miid u8 list_type u16[] qid_list)ndash retCode_t unbind_queue(u8 list_type u16[] qid_list)LC and NPEndash bw_t actual_bw set_queue_params(u16 qid u32 threshold u32 bw)
bull If a parameter is -1 then do not update in tablendash u32 threshold u32 bw get_queue_params(u16 qid)ndash u32 pktCnt u32 byteCnt get_queue_len(u16 qid)
bull SRM to SCD LC (Ingress and Egress)ndash retCode_t bind_queue_sched(u16 qid u16 sid)
bull Assign qid to scheduler sidndash retCode_t unbind_queue_sched(u16 qid)
bull Deallocate queuersquos bw from its assigned scheduler and disassociate it from that scheduler
12WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Lookup Table (TCAM)bull Common LC cid = 0 NPE cid = fpid
ndash If cid == 0 then fid is absolute otherwise it is fastpath relative and must be converteduint32_t fid Filter ID If cid == 0 then absolute otherwise relativeuint16_t dbid Database ID NPE dbid=0 LC Ingress dbid=0 Egress dbid=1struct fltr uint8_t key[N] N-Byte key value defined by code option uint8_t mask[N] N-Byte mask for key lookups defined by code option uint8_t result[M] M-Byte result vector defined by code option
bull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)LC and NPEndash ret_t write_fltr(dbid fid key mask result)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash ret_t update_result(dbid fid result)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_fltr_bykey(dbid key)fltr get_fltr_byfid(dbid fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash result lookup_fltr(dbid key)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash retcode rem_fltr_bykey(dbid key)
retcode rem_fltr_byfid(dbid fid)ndash Returns fid if filter removed otherwise -1
13WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE Filter Interfacebull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)
LC and NPEsubKeyWrap_t type rxip rxport coptKey[14] subResult_t actions sindx daddr dport sport qidsubFltr_t subKeyWrap coptMask[14] subResult ndash scdCode_t write_npe_fltr(fid subFltr)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash scdCode_t update_npe_result(fid subResult)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_npe_fltr_bykey(subKeyWrap)fltr get_npe_fltr_byfid(fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash scdCode_t lookup_npe_fltr(subKeyWrap)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash scdCode_t rem_npe_fltr_bykey(subKeyWrap)
scdCode_t rem_npe_fltr_byfid(fid)ndash Returns 0 on success or an error
14WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
IPv4 TCAM Filter Formats (on NPE)
6 82flags
2 12
0100
2
TCP RSV proto00TCP
daddr saddr sport dport tcpproto
Defined by the IPv4 Code Option 112bits
32 32 16 16 16
vlan
11
if
T = 0 Normal LookupT = 1 substrate only lookup
T
1
RX port
Substrate defined
164
TX IP daddr TX dport TX sport rsv32 16 12 1516
QM3
D Drop packetL Local delivery
rsv113 1
L1
Drsv sindx Sch2
qid16
20-bit internal qid(SCD maps slicersquos miidto QM and Sch SCD Also
maps slicersquos qid toglobal qid value)
TX IP address and sport representsthe output meta-interface The
dport is provided by the slice (RMP maps miid to tx tunnel params
use dport provided by slice)
Result 128 bits
Represents input meta-interface
global statsindex
(SCD mapsslicersquos sindx
to global value)
Key Input miid IPv4 fltr daddr saddr sport dport tcpproto
Result Flags Drop GPE sindx Output miID QID
Slice parameters
15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Reading Statisticsbull Common
ndash if cid = 0 then sindx is absolute otherwiseSCD must convert fastpath relative index
uint8_t flagsuint32_t handle opaque reference to periodic eventuint32_t sindx stats index
bull RMP to SCD RLIGIU to SCDLC and NPEndash stats = read_stats(sindx flags)
bull struct stats uint32_t cnt tstampbull Uses flags L and W
ndash result = clear_stats(sindx)bull result success or error code
ndash handle create_periodic(sindx P cnt flags)bull uint32_t P periodic interval in millisecondsbull uint16_t cnt Number of samples to keep in a history bufferbull Uses all 3 flags
ndash retcode del_periodic(handle)ndash retcode set_callback(handle udp_port)
bull push model every (cntperiod) milliseconds send the last cnt samples to the clientrsquos UDP port udp_port
ndash stats = get_periodic(handle)bull pull model returns the last cnt samples
FlagsW 0 - Packts 1 - BytesL 0 - PreQ 1 - PostQT 0 - Push 1 - PullX Donrsquot Care
06 3 12457WLTXXXXX
16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Accessing Memorybull common
Message context id NPE equals fpid for LC must equal 0uint32_t offset Offset in from start of assigned SRAM block uint32_t len Number of bytes to readwriteuint8_t data[X] Data buffer with X Bytesuint32_t kpa Kernel physical address
bull SRM to SCD LC and NPEndash result write_mem(kpa len data)
bull Message context ID must == 0bull Can read any valid physical address on the xscale
ndash data read_mem(kpa len)bull Message context ID must == 0bull Can read any valid physical address on the xscale
bull RMP to SCD NPE Onlyndash result write_sram(offset len data)
bull Offset is relative to the starting address of SRAM block allocated to slice SCD must verify the write operation is within bounds
ndash data read_sram(offset len)bull Read len bytes from SRAM block and return to client First verify offset and len with within
bounds for slice
17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
New Commands
bull NPE Onlybull retCode_t set_src_hwaddr(hwaddr_t)
ndash context must be 0
bull retCode_t set_iface_table(ipAddr_t[16])ndash context must be 0
2WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Single Interface Example
wrr
qxsn
qxs1
qxs2 SchedNPE1
qpsn
qps1
qps2
SchedGPE1
qBE
Ingress
Map fpMI to Qdip dp pr
wrr
SchedCP
qsn
qs1
qs2
wrr
qGPE
qCP
SchedI1
Egress
src addrproto
porticmp
LC
inte
rfac
e 1
GPE
bull LC Ingressndash One queue per slice with reserved bandwidth
(really one per scheduler)
ndash One queue for best effort traffic to each GPE
ndash One scheduler for CP with queues for reserved traffic plus BE
bull LC Egressndash At least one scheduler for each physical interface
ndash One queue for each active slice with MI defined for the associated scheduler
ndash One best effort queue for each board (GPE CP NPE)
bull Substrate sets scheduler rates according to aggregate allocations
ndash Manage scheduler rates to control aggregate traffic to interfaces and boards
BWI1
GPE
CP
NPE
NPE
3WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Common Definitions and Typesbull See slides titled Typesppt
bull For all commands the message context ID is used to identify the context within which a command is to be executedndash The special value of 0 (cid = 0) indicates a privileged operation
performed by the substrate
ndash Otherwise the context ID is an identifier indicating a user specific context in the command is to be interpreted and executing In most cases this is either the fastpath ID or an internal slice ID
bull uint16_t miid Meta-Interface numberndash If fpid == 0 then miid is the endpoint ID (epid)
bull int32_t retCode_t Most calls return success (0) or an error codendash If a command results in an error then the first 4-Byte of the reply
message contains an error code
ndash If a command doesnrsquot return any data then it should at least return a retCode_t code of 0 but this is not required
4WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE Tables
VLAN
VLAN Table
031
copt (8b)not used81624
Start of SRAM Block (Physical Address)SRAM Block SizeMax Buffer Limit
Current Buffer CountBuffer Limit Exceeded Count
copt (8b)not usedStart of SRAM Block (Physical Address)
SRAM Block SizeMax Buffer Limit
Current Buffer CountBuffer Limit Exceeded Count
sid(Tunnel) Source IP Address
Destination MAC Address High Order 4 Bytes
Dest MAC Not Used
(Tunnel) Source IP Address
Destination MAC Address High Order 4 Bytes
Dest MAC Not Used
031 81624
Encap Control Block
VLAN
GPE Info table
031GPE IP Address (tunnel)
81624
NPE IP Address (tunnel)
Exception UDP SPort Exception UDP DPort Local UDP SPort Local UDP DPort
sid Exception qidqmnot usedsid Local Delivery qidqmnot used
(Tunnel) GPE IP Address(Tunnel) NPE IP Address
Exception UDP SPort Exception UDP DPort Local UDP SPort Local UDP DPort
sidqmnot usedsidqmnot used
Exception qidLocal Delivery qid
sidinterface number rate (Kbps683)
031 81624
Scheduler Rate table
interface number rate (Kbps683)
qidLength in Pkts (28b)
Length in Bytes (32b)
Threshold in Bytes (32b)
031 81624
QParams Table
unused
Quantum (32b)
Length in Pkts (28b)
Length in Bytes (32b)
Threshold in Bytes (32b)
unused
Quantum (32b)
5WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Line Card Tables or Both
sid destination Mac (high order 4 Bytes)
dest MAC Not Used
031 81624
IngressEgress Dynamic MAC Table
source Mac (high order 4 Bytes)
src MAC Not Used
destination Mac (high order 4 Bytes)
dest MAC Not Used
source Mac (high order 4 Bytes)
src MAC Not Used
sidinterface number rate (Kbps683)
031 81624
Scheduler Rate table
interface number rate (Kbps683)
qidLength in Pkts (28b)
Length in Bytes (32b)
Threshold in Bytes (32b)
031 81624
QParams Table
unused
Quantum (32b)
Length in Pkts (28b)
Length in Bytes (32b)
Threshold in Bytes (32b)
unused
Quantum (32b)
6WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Fast Path Commands SRM to SCDNPE Onlybull retCode_t set_fastpath(fpid copt_t VLAN rcnts_t Mem[])
Message context ID = 0 fpid Fast-Path ID Unique within SPP nodecopt Code option identifier 0 is invalid IPv4 = 1 I3 = 2uint16_t VLAN Ethernet VLAN tag used in datapath to identify fastpath instancercnts_t = cntr_t Qs Fltrs Buffers Statsstruct uint32_t offset size Mem[2] SRAM and DRAM allocation values are
relative to the start of an SRAM block set aside for fastpath useMem = sram(offset size) SRAM version 1 all same size dram(offset size) DRAM not used in version 1
ndash Create fastpath specific mappings for queues TCAM filters and the stats tablendash Initialize queue parameters to ldquoreasonablerdquo values (threshold weight)ndash Update VLAN Table with the code option ID (copt) and SRAM Address
bull SCD knows starting address of SRAM so add offsetndash Assume there is an SRAM table read by the microengines which maps VLAN to buffer limits
this may be the VLAN tablendash Treat as a transaction either it completes fully or no changes are recordeduint32_t retCode_t 0 = Success app defined
7WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Fast Path Commands SRM to SCDNPE Onlybull retCode_t enable_fastpath(fpid_t)
Message context ID = 0 fpid_t fast path identifieruint32_t retCode_t 0 = Success app definedndash enable by setting code option field of VLAN table entry for fp
bull retCode_t disable_fastpath(fpid)Message context ID = 0 uint16_t fpid fast path identifierretCode_t 0 = Success 1 = Pending 2 = Invalid fpid else error codendash First disable fp by setting code option field of VLAN table to 0ndash It is not an error to call disable an already disabled fastpath
bull SRM will periodically call disable_fastpath() until Success or Errorndash Then check all fast path queue lengths
bull If any queue has non-zero length then return Pendingbull If all queues are empty then return Success
bull retCode_t rem_fastpath(fpid_t)Message context ID = 0 fpid_t fast path identifierretCode_t 0 = Success 2 = Invalid fpid else error codendash Fastpath must be disabled (disable_fastpath) before it can be removedndash Remove all allocations associated with fpid
8WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
BW and Scheduler Control SRM to SCDbull Common Message
Message context ID = 0uint16_t sid Scheduler Id There are 5 schedulers per QM and 4 QMs so 0 lt= sid lt 20 (NBO)uint8_t MAC[6] 6 Byte Ethernet address
LC and NPE schedule parametersbull retCode_t set_sched_params(sid ifn BWmax BWmin)
int32_t ifn Interface numberuint32_t BWmax Kbps Maximum scheduler rateuint32_t BWmin Kbps Minimum assignable rate for this scheduler
ndash If BWmin is different from a previous call then may need to reassign weights for all associated queuesndash Must convert the rate I send (Kbps) to that expected by the hardware
Value = X Mbps 0683 = BW Kbps 683 (So no floating point needed)ndash Update scheduler parameters table in SRAM The low 16-bit word is the converted rate value and the upper 16-bits is the interface
number (ifn) The interface number must be the same value that TX uses To set only the Rate (preserves the existing interface number) SParams[sid] = (SchedParams[sid] amp 0xFFFF0000) | (0xFFFF amp Value)To set the interface number (ifn preserves the existing rate value) SParams[sid] = (SParams[sid] amp 0xFFFF) | (ifn ltlt 16)To set both SParams[sid] = (ifn ltlt 16) | (0xFFFF amp Value)
NPE Encap Control Blockbull retCode_t set_encap_cb(sid srcIP dMAC)
ndash Must update the table which associates encapsulation header IP source addresses with a schedulerSchedAddrs[sid] = ipaddr destination MAC
bull retCode_t create_mi(fpid mi sid)bull retCode_t delete_mi(fpid mi)bull retCode_t set_mi_bw(fpid mi bw)LC Dynamic MAC Tablebull retCode_t set_sched_mac(sid dstMAC srcMAC)
ndash Must update the table which associates Ethernet source and destination addresses with a schedulerSchedAddrs[sid]smac = smac (6 Bytes over 2 4B words)SchedAddrs[sid]dmac = dmac (6 Bytes over 2 4B words)
9WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
System Configuration SRM to SCD
bull Common Message context ID = 0
SRM to SCD NPE Onlybull retCode_t start_mes()
ndash The SCD starts the IXP microengines running the code that is automatically loaded when the SCD is launched
ndash Returns Success (0) Error (-1)
bull retCode_t stop_mes()ndash The SCD stops the IXP microengines if they are running
ndash Returns Success (0) Error (-1)
10WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE GPE Infobull Common Message context ID = 0
uint32_t IP 4-Byte IP address (NBO)
uint16_t PORT UDP Port number (NBO)
uint16_t QID Slice relative queue id must be converted by SCD to the tuple of qid QM id and scheduler ID
bull SRM to SCDNPE Only
retCode_t set_encap_gpe(fpid gpeIP npeIP)
retCode_t unset_encap_gpe(fpid)
unset_encap_gpe() == set_encap_gpe(fpid00)
uint16_t fpid Fast path id globally unique
bull RMP to SCDNPE Only
retCode_t set_gpe_info(exPort ldPort exQID ldQID)
retCode_t unset_gpe_info()
Set context ID to the global fast path ID
11WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Queuesbull Common
ndash if cid == 0 then qid is absolute otherwise SCD must convert from fastpath relativeuint16_t miid Meta-Interface numberuint32_t qlen Length of packet queue in Bytesuint32_t threshold the maximum number of packets queued before droppinguint32_t bw Kbs SCD must convert to the associated weightuint8_t list_type Explicit list (0) or range specification (1)uint16_t qid_list[] List of queue identifiers to associate with meta-interface mi In C or C++ this would
be an array of integers 0 n If using a range then the list has the first and one past the last id of the range For example 4 8 is the same as 4 5 6 7
bull RMP to SCDNPE Onlyndash retCode_t bind_queue(u16 miid u8 list_type u16[] qid_list)ndash retCode_t unbind_queue(u8 list_type u16[] qid_list)LC and NPEndash bw_t actual_bw set_queue_params(u16 qid u32 threshold u32 bw)
bull If a parameter is -1 then do not update in tablendash u32 threshold u32 bw get_queue_params(u16 qid)ndash u32 pktCnt u32 byteCnt get_queue_len(u16 qid)
bull SRM to SCD LC (Ingress and Egress)ndash retCode_t bind_queue_sched(u16 qid u16 sid)
bull Assign qid to scheduler sidndash retCode_t unbind_queue_sched(u16 qid)
bull Deallocate queuersquos bw from its assigned scheduler and disassociate it from that scheduler
12WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Lookup Table (TCAM)bull Common LC cid = 0 NPE cid = fpid
ndash If cid == 0 then fid is absolute otherwise it is fastpath relative and must be converteduint32_t fid Filter ID If cid == 0 then absolute otherwise relativeuint16_t dbid Database ID NPE dbid=0 LC Ingress dbid=0 Egress dbid=1struct fltr uint8_t key[N] N-Byte key value defined by code option uint8_t mask[N] N-Byte mask for key lookups defined by code option uint8_t result[M] M-Byte result vector defined by code option
bull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)LC and NPEndash ret_t write_fltr(dbid fid key mask result)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash ret_t update_result(dbid fid result)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_fltr_bykey(dbid key)fltr get_fltr_byfid(dbid fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash result lookup_fltr(dbid key)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash retcode rem_fltr_bykey(dbid key)
retcode rem_fltr_byfid(dbid fid)ndash Returns fid if filter removed otherwise -1
13WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE Filter Interfacebull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)
LC and NPEsubKeyWrap_t type rxip rxport coptKey[14] subResult_t actions sindx daddr dport sport qidsubFltr_t subKeyWrap coptMask[14] subResult ndash scdCode_t write_npe_fltr(fid subFltr)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash scdCode_t update_npe_result(fid subResult)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_npe_fltr_bykey(subKeyWrap)fltr get_npe_fltr_byfid(fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash scdCode_t lookup_npe_fltr(subKeyWrap)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash scdCode_t rem_npe_fltr_bykey(subKeyWrap)
scdCode_t rem_npe_fltr_byfid(fid)ndash Returns 0 on success or an error
14WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
IPv4 TCAM Filter Formats (on NPE)
6 82flags
2 12
0100
2
TCP RSV proto00TCP
daddr saddr sport dport tcpproto
Defined by the IPv4 Code Option 112bits
32 32 16 16 16
vlan
11
if
T = 0 Normal LookupT = 1 substrate only lookup
T
1
RX port
Substrate defined
164
TX IP daddr TX dport TX sport rsv32 16 12 1516
QM3
D Drop packetL Local delivery
rsv113 1
L1
Drsv sindx Sch2
qid16
20-bit internal qid(SCD maps slicersquos miidto QM and Sch SCD Also
maps slicersquos qid toglobal qid value)
TX IP address and sport representsthe output meta-interface The
dport is provided by the slice (RMP maps miid to tx tunnel params
use dport provided by slice)
Result 128 bits
Represents input meta-interface
global statsindex
(SCD mapsslicersquos sindx
to global value)
Key Input miid IPv4 fltr daddr saddr sport dport tcpproto
Result Flags Drop GPE sindx Output miID QID
Slice parameters
15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Reading Statisticsbull Common
ndash if cid = 0 then sindx is absolute otherwiseSCD must convert fastpath relative index
uint8_t flagsuint32_t handle opaque reference to periodic eventuint32_t sindx stats index
bull RMP to SCD RLIGIU to SCDLC and NPEndash stats = read_stats(sindx flags)
bull struct stats uint32_t cnt tstampbull Uses flags L and W
ndash result = clear_stats(sindx)bull result success or error code
ndash handle create_periodic(sindx P cnt flags)bull uint32_t P periodic interval in millisecondsbull uint16_t cnt Number of samples to keep in a history bufferbull Uses all 3 flags
ndash retcode del_periodic(handle)ndash retcode set_callback(handle udp_port)
bull push model every (cntperiod) milliseconds send the last cnt samples to the clientrsquos UDP port udp_port
ndash stats = get_periodic(handle)bull pull model returns the last cnt samples
FlagsW 0 - Packts 1 - BytesL 0 - PreQ 1 - PostQT 0 - Push 1 - PullX Donrsquot Care
06 3 12457WLTXXXXX
16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Accessing Memorybull common
Message context id NPE equals fpid for LC must equal 0uint32_t offset Offset in from start of assigned SRAM block uint32_t len Number of bytes to readwriteuint8_t data[X] Data buffer with X Bytesuint32_t kpa Kernel physical address
bull SRM to SCD LC and NPEndash result write_mem(kpa len data)
bull Message context ID must == 0bull Can read any valid physical address on the xscale
ndash data read_mem(kpa len)bull Message context ID must == 0bull Can read any valid physical address on the xscale
bull RMP to SCD NPE Onlyndash result write_sram(offset len data)
bull Offset is relative to the starting address of SRAM block allocated to slice SCD must verify the write operation is within bounds
ndash data read_sram(offset len)bull Read len bytes from SRAM block and return to client First verify offset and len with within
bounds for slice
17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
New Commands
bull NPE Onlybull retCode_t set_src_hwaddr(hwaddr_t)
ndash context must be 0
bull retCode_t set_iface_table(ipAddr_t[16])ndash context must be 0
3WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Common Definitions and Typesbull See slides titled Typesppt
bull For all commands the message context ID is used to identify the context within which a command is to be executedndash The special value of 0 (cid = 0) indicates a privileged operation
performed by the substrate
ndash Otherwise the context ID is an identifier indicating a user specific context in the command is to be interpreted and executing In most cases this is either the fastpath ID or an internal slice ID
bull uint16_t miid Meta-Interface numberndash If fpid == 0 then miid is the endpoint ID (epid)
bull int32_t retCode_t Most calls return success (0) or an error codendash If a command results in an error then the first 4-Byte of the reply
message contains an error code
ndash If a command doesnrsquot return any data then it should at least return a retCode_t code of 0 but this is not required
4WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE Tables
VLAN
VLAN Table
031
copt (8b)not used81624
Start of SRAM Block (Physical Address)SRAM Block SizeMax Buffer Limit
Current Buffer CountBuffer Limit Exceeded Count
copt (8b)not usedStart of SRAM Block (Physical Address)
SRAM Block SizeMax Buffer Limit
Current Buffer CountBuffer Limit Exceeded Count
sid(Tunnel) Source IP Address
Destination MAC Address High Order 4 Bytes
Dest MAC Not Used
(Tunnel) Source IP Address
Destination MAC Address High Order 4 Bytes
Dest MAC Not Used
031 81624
Encap Control Block
VLAN
GPE Info table
031GPE IP Address (tunnel)
81624
NPE IP Address (tunnel)
Exception UDP SPort Exception UDP DPort Local UDP SPort Local UDP DPort
sid Exception qidqmnot usedsid Local Delivery qidqmnot used
(Tunnel) GPE IP Address(Tunnel) NPE IP Address
Exception UDP SPort Exception UDP DPort Local UDP SPort Local UDP DPort
sidqmnot usedsidqmnot used
Exception qidLocal Delivery qid
sidinterface number rate (Kbps683)
031 81624
Scheduler Rate table
interface number rate (Kbps683)
qidLength in Pkts (28b)
Length in Bytes (32b)
Threshold in Bytes (32b)
031 81624
QParams Table
unused
Quantum (32b)
Length in Pkts (28b)
Length in Bytes (32b)
Threshold in Bytes (32b)
unused
Quantum (32b)
5WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Line Card Tables or Both
sid destination Mac (high order 4 Bytes)
dest MAC Not Used
031 81624
IngressEgress Dynamic MAC Table
source Mac (high order 4 Bytes)
src MAC Not Used
destination Mac (high order 4 Bytes)
dest MAC Not Used
source Mac (high order 4 Bytes)
src MAC Not Used
sidinterface number rate (Kbps683)
031 81624
Scheduler Rate table
interface number rate (Kbps683)
qidLength in Pkts (28b)
Length in Bytes (32b)
Threshold in Bytes (32b)
031 81624
QParams Table
unused
Quantum (32b)
Length in Pkts (28b)
Length in Bytes (32b)
Threshold in Bytes (32b)
unused
Quantum (32b)
6WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Fast Path Commands SRM to SCDNPE Onlybull retCode_t set_fastpath(fpid copt_t VLAN rcnts_t Mem[])
Message context ID = 0 fpid Fast-Path ID Unique within SPP nodecopt Code option identifier 0 is invalid IPv4 = 1 I3 = 2uint16_t VLAN Ethernet VLAN tag used in datapath to identify fastpath instancercnts_t = cntr_t Qs Fltrs Buffers Statsstruct uint32_t offset size Mem[2] SRAM and DRAM allocation values are
relative to the start of an SRAM block set aside for fastpath useMem = sram(offset size) SRAM version 1 all same size dram(offset size) DRAM not used in version 1
ndash Create fastpath specific mappings for queues TCAM filters and the stats tablendash Initialize queue parameters to ldquoreasonablerdquo values (threshold weight)ndash Update VLAN Table with the code option ID (copt) and SRAM Address
bull SCD knows starting address of SRAM so add offsetndash Assume there is an SRAM table read by the microengines which maps VLAN to buffer limits
this may be the VLAN tablendash Treat as a transaction either it completes fully or no changes are recordeduint32_t retCode_t 0 = Success app defined
7WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Fast Path Commands SRM to SCDNPE Onlybull retCode_t enable_fastpath(fpid_t)
Message context ID = 0 fpid_t fast path identifieruint32_t retCode_t 0 = Success app definedndash enable by setting code option field of VLAN table entry for fp
bull retCode_t disable_fastpath(fpid)Message context ID = 0 uint16_t fpid fast path identifierretCode_t 0 = Success 1 = Pending 2 = Invalid fpid else error codendash First disable fp by setting code option field of VLAN table to 0ndash It is not an error to call disable an already disabled fastpath
bull SRM will periodically call disable_fastpath() until Success or Errorndash Then check all fast path queue lengths
bull If any queue has non-zero length then return Pendingbull If all queues are empty then return Success
bull retCode_t rem_fastpath(fpid_t)Message context ID = 0 fpid_t fast path identifierretCode_t 0 = Success 2 = Invalid fpid else error codendash Fastpath must be disabled (disable_fastpath) before it can be removedndash Remove all allocations associated with fpid
8WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
BW and Scheduler Control SRM to SCDbull Common Message
Message context ID = 0uint16_t sid Scheduler Id There are 5 schedulers per QM and 4 QMs so 0 lt= sid lt 20 (NBO)uint8_t MAC[6] 6 Byte Ethernet address
LC and NPE schedule parametersbull retCode_t set_sched_params(sid ifn BWmax BWmin)
int32_t ifn Interface numberuint32_t BWmax Kbps Maximum scheduler rateuint32_t BWmin Kbps Minimum assignable rate for this scheduler
ndash If BWmin is different from a previous call then may need to reassign weights for all associated queuesndash Must convert the rate I send (Kbps) to that expected by the hardware
Value = X Mbps 0683 = BW Kbps 683 (So no floating point needed)ndash Update scheduler parameters table in SRAM The low 16-bit word is the converted rate value and the upper 16-bits is the interface
number (ifn) The interface number must be the same value that TX uses To set only the Rate (preserves the existing interface number) SParams[sid] = (SchedParams[sid] amp 0xFFFF0000) | (0xFFFF amp Value)To set the interface number (ifn preserves the existing rate value) SParams[sid] = (SParams[sid] amp 0xFFFF) | (ifn ltlt 16)To set both SParams[sid] = (ifn ltlt 16) | (0xFFFF amp Value)
NPE Encap Control Blockbull retCode_t set_encap_cb(sid srcIP dMAC)
ndash Must update the table which associates encapsulation header IP source addresses with a schedulerSchedAddrs[sid] = ipaddr destination MAC
bull retCode_t create_mi(fpid mi sid)bull retCode_t delete_mi(fpid mi)bull retCode_t set_mi_bw(fpid mi bw)LC Dynamic MAC Tablebull retCode_t set_sched_mac(sid dstMAC srcMAC)
ndash Must update the table which associates Ethernet source and destination addresses with a schedulerSchedAddrs[sid]smac = smac (6 Bytes over 2 4B words)SchedAddrs[sid]dmac = dmac (6 Bytes over 2 4B words)
9WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
System Configuration SRM to SCD
bull Common Message context ID = 0
SRM to SCD NPE Onlybull retCode_t start_mes()
ndash The SCD starts the IXP microengines running the code that is automatically loaded when the SCD is launched
ndash Returns Success (0) Error (-1)
bull retCode_t stop_mes()ndash The SCD stops the IXP microengines if they are running
ndash Returns Success (0) Error (-1)
10WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE GPE Infobull Common Message context ID = 0
uint32_t IP 4-Byte IP address (NBO)
uint16_t PORT UDP Port number (NBO)
uint16_t QID Slice relative queue id must be converted by SCD to the tuple of qid QM id and scheduler ID
bull SRM to SCDNPE Only
retCode_t set_encap_gpe(fpid gpeIP npeIP)
retCode_t unset_encap_gpe(fpid)
unset_encap_gpe() == set_encap_gpe(fpid00)
uint16_t fpid Fast path id globally unique
bull RMP to SCDNPE Only
retCode_t set_gpe_info(exPort ldPort exQID ldQID)
retCode_t unset_gpe_info()
Set context ID to the global fast path ID
11WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Queuesbull Common
ndash if cid == 0 then qid is absolute otherwise SCD must convert from fastpath relativeuint16_t miid Meta-Interface numberuint32_t qlen Length of packet queue in Bytesuint32_t threshold the maximum number of packets queued before droppinguint32_t bw Kbs SCD must convert to the associated weightuint8_t list_type Explicit list (0) or range specification (1)uint16_t qid_list[] List of queue identifiers to associate with meta-interface mi In C or C++ this would
be an array of integers 0 n If using a range then the list has the first and one past the last id of the range For example 4 8 is the same as 4 5 6 7
bull RMP to SCDNPE Onlyndash retCode_t bind_queue(u16 miid u8 list_type u16[] qid_list)ndash retCode_t unbind_queue(u8 list_type u16[] qid_list)LC and NPEndash bw_t actual_bw set_queue_params(u16 qid u32 threshold u32 bw)
bull If a parameter is -1 then do not update in tablendash u32 threshold u32 bw get_queue_params(u16 qid)ndash u32 pktCnt u32 byteCnt get_queue_len(u16 qid)
bull SRM to SCD LC (Ingress and Egress)ndash retCode_t bind_queue_sched(u16 qid u16 sid)
bull Assign qid to scheduler sidndash retCode_t unbind_queue_sched(u16 qid)
bull Deallocate queuersquos bw from its assigned scheduler and disassociate it from that scheduler
12WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Lookup Table (TCAM)bull Common LC cid = 0 NPE cid = fpid
ndash If cid == 0 then fid is absolute otherwise it is fastpath relative and must be converteduint32_t fid Filter ID If cid == 0 then absolute otherwise relativeuint16_t dbid Database ID NPE dbid=0 LC Ingress dbid=0 Egress dbid=1struct fltr uint8_t key[N] N-Byte key value defined by code option uint8_t mask[N] N-Byte mask for key lookups defined by code option uint8_t result[M] M-Byte result vector defined by code option
bull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)LC and NPEndash ret_t write_fltr(dbid fid key mask result)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash ret_t update_result(dbid fid result)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_fltr_bykey(dbid key)fltr get_fltr_byfid(dbid fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash result lookup_fltr(dbid key)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash retcode rem_fltr_bykey(dbid key)
retcode rem_fltr_byfid(dbid fid)ndash Returns fid if filter removed otherwise -1
13WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE Filter Interfacebull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)
LC and NPEsubKeyWrap_t type rxip rxport coptKey[14] subResult_t actions sindx daddr dport sport qidsubFltr_t subKeyWrap coptMask[14] subResult ndash scdCode_t write_npe_fltr(fid subFltr)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash scdCode_t update_npe_result(fid subResult)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_npe_fltr_bykey(subKeyWrap)fltr get_npe_fltr_byfid(fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash scdCode_t lookup_npe_fltr(subKeyWrap)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash scdCode_t rem_npe_fltr_bykey(subKeyWrap)
scdCode_t rem_npe_fltr_byfid(fid)ndash Returns 0 on success or an error
14WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
IPv4 TCAM Filter Formats (on NPE)
6 82flags
2 12
0100
2
TCP RSV proto00TCP
daddr saddr sport dport tcpproto
Defined by the IPv4 Code Option 112bits
32 32 16 16 16
vlan
11
if
T = 0 Normal LookupT = 1 substrate only lookup
T
1
RX port
Substrate defined
164
TX IP daddr TX dport TX sport rsv32 16 12 1516
QM3
D Drop packetL Local delivery
rsv113 1
L1
Drsv sindx Sch2
qid16
20-bit internal qid(SCD maps slicersquos miidto QM and Sch SCD Also
maps slicersquos qid toglobal qid value)
TX IP address and sport representsthe output meta-interface The
dport is provided by the slice (RMP maps miid to tx tunnel params
use dport provided by slice)
Result 128 bits
Represents input meta-interface
global statsindex
(SCD mapsslicersquos sindx
to global value)
Key Input miid IPv4 fltr daddr saddr sport dport tcpproto
Result Flags Drop GPE sindx Output miID QID
Slice parameters
15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Reading Statisticsbull Common
ndash if cid = 0 then sindx is absolute otherwiseSCD must convert fastpath relative index
uint8_t flagsuint32_t handle opaque reference to periodic eventuint32_t sindx stats index
bull RMP to SCD RLIGIU to SCDLC and NPEndash stats = read_stats(sindx flags)
bull struct stats uint32_t cnt tstampbull Uses flags L and W
ndash result = clear_stats(sindx)bull result success or error code
ndash handle create_periodic(sindx P cnt flags)bull uint32_t P periodic interval in millisecondsbull uint16_t cnt Number of samples to keep in a history bufferbull Uses all 3 flags
ndash retcode del_periodic(handle)ndash retcode set_callback(handle udp_port)
bull push model every (cntperiod) milliseconds send the last cnt samples to the clientrsquos UDP port udp_port
ndash stats = get_periodic(handle)bull pull model returns the last cnt samples
FlagsW 0 - Packts 1 - BytesL 0 - PreQ 1 - PostQT 0 - Push 1 - PullX Donrsquot Care
06 3 12457WLTXXXXX
16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Accessing Memorybull common
Message context id NPE equals fpid for LC must equal 0uint32_t offset Offset in from start of assigned SRAM block uint32_t len Number of bytes to readwriteuint8_t data[X] Data buffer with X Bytesuint32_t kpa Kernel physical address
bull SRM to SCD LC and NPEndash result write_mem(kpa len data)
bull Message context ID must == 0bull Can read any valid physical address on the xscale
ndash data read_mem(kpa len)bull Message context ID must == 0bull Can read any valid physical address on the xscale
bull RMP to SCD NPE Onlyndash result write_sram(offset len data)
bull Offset is relative to the starting address of SRAM block allocated to slice SCD must verify the write operation is within bounds
ndash data read_sram(offset len)bull Read len bytes from SRAM block and return to client First verify offset and len with within
bounds for slice
17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
New Commands
bull NPE Onlybull retCode_t set_src_hwaddr(hwaddr_t)
ndash context must be 0
bull retCode_t set_iface_table(ipAddr_t[16])ndash context must be 0
4WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE Tables
VLAN
VLAN Table
031
copt (8b)not used81624
Start of SRAM Block (Physical Address)SRAM Block SizeMax Buffer Limit
Current Buffer CountBuffer Limit Exceeded Count
copt (8b)not usedStart of SRAM Block (Physical Address)
SRAM Block SizeMax Buffer Limit
Current Buffer CountBuffer Limit Exceeded Count
sid(Tunnel) Source IP Address
Destination MAC Address High Order 4 Bytes
Dest MAC Not Used
(Tunnel) Source IP Address
Destination MAC Address High Order 4 Bytes
Dest MAC Not Used
031 81624
Encap Control Block
VLAN
GPE Info table
031GPE IP Address (tunnel)
81624
NPE IP Address (tunnel)
Exception UDP SPort Exception UDP DPort Local UDP SPort Local UDP DPort
sid Exception qidqmnot usedsid Local Delivery qidqmnot used
(Tunnel) GPE IP Address(Tunnel) NPE IP Address
Exception UDP SPort Exception UDP DPort Local UDP SPort Local UDP DPort
sidqmnot usedsidqmnot used
Exception qidLocal Delivery qid
sidinterface number rate (Kbps683)
031 81624
Scheduler Rate table
interface number rate (Kbps683)
qidLength in Pkts (28b)
Length in Bytes (32b)
Threshold in Bytes (32b)
031 81624
QParams Table
unused
Quantum (32b)
Length in Pkts (28b)
Length in Bytes (32b)
Threshold in Bytes (32b)
unused
Quantum (32b)
5WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Line Card Tables or Both
sid destination Mac (high order 4 Bytes)
dest MAC Not Used
031 81624
IngressEgress Dynamic MAC Table
source Mac (high order 4 Bytes)
src MAC Not Used
destination Mac (high order 4 Bytes)
dest MAC Not Used
source Mac (high order 4 Bytes)
src MAC Not Used
sidinterface number rate (Kbps683)
031 81624
Scheduler Rate table
interface number rate (Kbps683)
qidLength in Pkts (28b)
Length in Bytes (32b)
Threshold in Bytes (32b)
031 81624
QParams Table
unused
Quantum (32b)
Length in Pkts (28b)
Length in Bytes (32b)
Threshold in Bytes (32b)
unused
Quantum (32b)
6WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Fast Path Commands SRM to SCDNPE Onlybull retCode_t set_fastpath(fpid copt_t VLAN rcnts_t Mem[])
Message context ID = 0 fpid Fast-Path ID Unique within SPP nodecopt Code option identifier 0 is invalid IPv4 = 1 I3 = 2uint16_t VLAN Ethernet VLAN tag used in datapath to identify fastpath instancercnts_t = cntr_t Qs Fltrs Buffers Statsstruct uint32_t offset size Mem[2] SRAM and DRAM allocation values are
relative to the start of an SRAM block set aside for fastpath useMem = sram(offset size) SRAM version 1 all same size dram(offset size) DRAM not used in version 1
ndash Create fastpath specific mappings for queues TCAM filters and the stats tablendash Initialize queue parameters to ldquoreasonablerdquo values (threshold weight)ndash Update VLAN Table with the code option ID (copt) and SRAM Address
bull SCD knows starting address of SRAM so add offsetndash Assume there is an SRAM table read by the microengines which maps VLAN to buffer limits
this may be the VLAN tablendash Treat as a transaction either it completes fully or no changes are recordeduint32_t retCode_t 0 = Success app defined
7WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Fast Path Commands SRM to SCDNPE Onlybull retCode_t enable_fastpath(fpid_t)
Message context ID = 0 fpid_t fast path identifieruint32_t retCode_t 0 = Success app definedndash enable by setting code option field of VLAN table entry for fp
bull retCode_t disable_fastpath(fpid)Message context ID = 0 uint16_t fpid fast path identifierretCode_t 0 = Success 1 = Pending 2 = Invalid fpid else error codendash First disable fp by setting code option field of VLAN table to 0ndash It is not an error to call disable an already disabled fastpath
bull SRM will periodically call disable_fastpath() until Success or Errorndash Then check all fast path queue lengths
bull If any queue has non-zero length then return Pendingbull If all queues are empty then return Success
bull retCode_t rem_fastpath(fpid_t)Message context ID = 0 fpid_t fast path identifierretCode_t 0 = Success 2 = Invalid fpid else error codendash Fastpath must be disabled (disable_fastpath) before it can be removedndash Remove all allocations associated with fpid
8WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
BW and Scheduler Control SRM to SCDbull Common Message
Message context ID = 0uint16_t sid Scheduler Id There are 5 schedulers per QM and 4 QMs so 0 lt= sid lt 20 (NBO)uint8_t MAC[6] 6 Byte Ethernet address
LC and NPE schedule parametersbull retCode_t set_sched_params(sid ifn BWmax BWmin)
int32_t ifn Interface numberuint32_t BWmax Kbps Maximum scheduler rateuint32_t BWmin Kbps Minimum assignable rate for this scheduler
ndash If BWmin is different from a previous call then may need to reassign weights for all associated queuesndash Must convert the rate I send (Kbps) to that expected by the hardware
Value = X Mbps 0683 = BW Kbps 683 (So no floating point needed)ndash Update scheduler parameters table in SRAM The low 16-bit word is the converted rate value and the upper 16-bits is the interface
number (ifn) The interface number must be the same value that TX uses To set only the Rate (preserves the existing interface number) SParams[sid] = (SchedParams[sid] amp 0xFFFF0000) | (0xFFFF amp Value)To set the interface number (ifn preserves the existing rate value) SParams[sid] = (SParams[sid] amp 0xFFFF) | (ifn ltlt 16)To set both SParams[sid] = (ifn ltlt 16) | (0xFFFF amp Value)
NPE Encap Control Blockbull retCode_t set_encap_cb(sid srcIP dMAC)
ndash Must update the table which associates encapsulation header IP source addresses with a schedulerSchedAddrs[sid] = ipaddr destination MAC
bull retCode_t create_mi(fpid mi sid)bull retCode_t delete_mi(fpid mi)bull retCode_t set_mi_bw(fpid mi bw)LC Dynamic MAC Tablebull retCode_t set_sched_mac(sid dstMAC srcMAC)
ndash Must update the table which associates Ethernet source and destination addresses with a schedulerSchedAddrs[sid]smac = smac (6 Bytes over 2 4B words)SchedAddrs[sid]dmac = dmac (6 Bytes over 2 4B words)
9WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
System Configuration SRM to SCD
bull Common Message context ID = 0
SRM to SCD NPE Onlybull retCode_t start_mes()
ndash The SCD starts the IXP microengines running the code that is automatically loaded when the SCD is launched
ndash Returns Success (0) Error (-1)
bull retCode_t stop_mes()ndash The SCD stops the IXP microengines if they are running
ndash Returns Success (0) Error (-1)
10WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE GPE Infobull Common Message context ID = 0
uint32_t IP 4-Byte IP address (NBO)
uint16_t PORT UDP Port number (NBO)
uint16_t QID Slice relative queue id must be converted by SCD to the tuple of qid QM id and scheduler ID
bull SRM to SCDNPE Only
retCode_t set_encap_gpe(fpid gpeIP npeIP)
retCode_t unset_encap_gpe(fpid)
unset_encap_gpe() == set_encap_gpe(fpid00)
uint16_t fpid Fast path id globally unique
bull RMP to SCDNPE Only
retCode_t set_gpe_info(exPort ldPort exQID ldQID)
retCode_t unset_gpe_info()
Set context ID to the global fast path ID
11WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Queuesbull Common
ndash if cid == 0 then qid is absolute otherwise SCD must convert from fastpath relativeuint16_t miid Meta-Interface numberuint32_t qlen Length of packet queue in Bytesuint32_t threshold the maximum number of packets queued before droppinguint32_t bw Kbs SCD must convert to the associated weightuint8_t list_type Explicit list (0) or range specification (1)uint16_t qid_list[] List of queue identifiers to associate with meta-interface mi In C or C++ this would
be an array of integers 0 n If using a range then the list has the first and one past the last id of the range For example 4 8 is the same as 4 5 6 7
bull RMP to SCDNPE Onlyndash retCode_t bind_queue(u16 miid u8 list_type u16[] qid_list)ndash retCode_t unbind_queue(u8 list_type u16[] qid_list)LC and NPEndash bw_t actual_bw set_queue_params(u16 qid u32 threshold u32 bw)
bull If a parameter is -1 then do not update in tablendash u32 threshold u32 bw get_queue_params(u16 qid)ndash u32 pktCnt u32 byteCnt get_queue_len(u16 qid)
bull SRM to SCD LC (Ingress and Egress)ndash retCode_t bind_queue_sched(u16 qid u16 sid)
bull Assign qid to scheduler sidndash retCode_t unbind_queue_sched(u16 qid)
bull Deallocate queuersquos bw from its assigned scheduler and disassociate it from that scheduler
12WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Lookup Table (TCAM)bull Common LC cid = 0 NPE cid = fpid
ndash If cid == 0 then fid is absolute otherwise it is fastpath relative and must be converteduint32_t fid Filter ID If cid == 0 then absolute otherwise relativeuint16_t dbid Database ID NPE dbid=0 LC Ingress dbid=0 Egress dbid=1struct fltr uint8_t key[N] N-Byte key value defined by code option uint8_t mask[N] N-Byte mask for key lookups defined by code option uint8_t result[M] M-Byte result vector defined by code option
bull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)LC and NPEndash ret_t write_fltr(dbid fid key mask result)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash ret_t update_result(dbid fid result)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_fltr_bykey(dbid key)fltr get_fltr_byfid(dbid fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash result lookup_fltr(dbid key)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash retcode rem_fltr_bykey(dbid key)
retcode rem_fltr_byfid(dbid fid)ndash Returns fid if filter removed otherwise -1
13WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE Filter Interfacebull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)
LC and NPEsubKeyWrap_t type rxip rxport coptKey[14] subResult_t actions sindx daddr dport sport qidsubFltr_t subKeyWrap coptMask[14] subResult ndash scdCode_t write_npe_fltr(fid subFltr)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash scdCode_t update_npe_result(fid subResult)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_npe_fltr_bykey(subKeyWrap)fltr get_npe_fltr_byfid(fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash scdCode_t lookup_npe_fltr(subKeyWrap)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash scdCode_t rem_npe_fltr_bykey(subKeyWrap)
scdCode_t rem_npe_fltr_byfid(fid)ndash Returns 0 on success or an error
14WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
IPv4 TCAM Filter Formats (on NPE)
6 82flags
2 12
0100
2
TCP RSV proto00TCP
daddr saddr sport dport tcpproto
Defined by the IPv4 Code Option 112bits
32 32 16 16 16
vlan
11
if
T = 0 Normal LookupT = 1 substrate only lookup
T
1
RX port
Substrate defined
164
TX IP daddr TX dport TX sport rsv32 16 12 1516
QM3
D Drop packetL Local delivery
rsv113 1
L1
Drsv sindx Sch2
qid16
20-bit internal qid(SCD maps slicersquos miidto QM and Sch SCD Also
maps slicersquos qid toglobal qid value)
TX IP address and sport representsthe output meta-interface The
dport is provided by the slice (RMP maps miid to tx tunnel params
use dport provided by slice)
Result 128 bits
Represents input meta-interface
global statsindex
(SCD mapsslicersquos sindx
to global value)
Key Input miid IPv4 fltr daddr saddr sport dport tcpproto
Result Flags Drop GPE sindx Output miID QID
Slice parameters
15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Reading Statisticsbull Common
ndash if cid = 0 then sindx is absolute otherwiseSCD must convert fastpath relative index
uint8_t flagsuint32_t handle opaque reference to periodic eventuint32_t sindx stats index
bull RMP to SCD RLIGIU to SCDLC and NPEndash stats = read_stats(sindx flags)
bull struct stats uint32_t cnt tstampbull Uses flags L and W
ndash result = clear_stats(sindx)bull result success or error code
ndash handle create_periodic(sindx P cnt flags)bull uint32_t P periodic interval in millisecondsbull uint16_t cnt Number of samples to keep in a history bufferbull Uses all 3 flags
ndash retcode del_periodic(handle)ndash retcode set_callback(handle udp_port)
bull push model every (cntperiod) milliseconds send the last cnt samples to the clientrsquos UDP port udp_port
ndash stats = get_periodic(handle)bull pull model returns the last cnt samples
FlagsW 0 - Packts 1 - BytesL 0 - PreQ 1 - PostQT 0 - Push 1 - PullX Donrsquot Care
06 3 12457WLTXXXXX
16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Accessing Memorybull common
Message context id NPE equals fpid for LC must equal 0uint32_t offset Offset in from start of assigned SRAM block uint32_t len Number of bytes to readwriteuint8_t data[X] Data buffer with X Bytesuint32_t kpa Kernel physical address
bull SRM to SCD LC and NPEndash result write_mem(kpa len data)
bull Message context ID must == 0bull Can read any valid physical address on the xscale
ndash data read_mem(kpa len)bull Message context ID must == 0bull Can read any valid physical address on the xscale
bull RMP to SCD NPE Onlyndash result write_sram(offset len data)
bull Offset is relative to the starting address of SRAM block allocated to slice SCD must verify the write operation is within bounds
ndash data read_sram(offset len)bull Read len bytes from SRAM block and return to client First verify offset and len with within
bounds for slice
17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
New Commands
bull NPE Onlybull retCode_t set_src_hwaddr(hwaddr_t)
ndash context must be 0
bull retCode_t set_iface_table(ipAddr_t[16])ndash context must be 0
5WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Line Card Tables or Both
sid destination Mac (high order 4 Bytes)
dest MAC Not Used
031 81624
IngressEgress Dynamic MAC Table
source Mac (high order 4 Bytes)
src MAC Not Used
destination Mac (high order 4 Bytes)
dest MAC Not Used
source Mac (high order 4 Bytes)
src MAC Not Used
sidinterface number rate (Kbps683)
031 81624
Scheduler Rate table
interface number rate (Kbps683)
qidLength in Pkts (28b)
Length in Bytes (32b)
Threshold in Bytes (32b)
031 81624
QParams Table
unused
Quantum (32b)
Length in Pkts (28b)
Length in Bytes (32b)
Threshold in Bytes (32b)
unused
Quantum (32b)
6WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Fast Path Commands SRM to SCDNPE Onlybull retCode_t set_fastpath(fpid copt_t VLAN rcnts_t Mem[])
Message context ID = 0 fpid Fast-Path ID Unique within SPP nodecopt Code option identifier 0 is invalid IPv4 = 1 I3 = 2uint16_t VLAN Ethernet VLAN tag used in datapath to identify fastpath instancercnts_t = cntr_t Qs Fltrs Buffers Statsstruct uint32_t offset size Mem[2] SRAM and DRAM allocation values are
relative to the start of an SRAM block set aside for fastpath useMem = sram(offset size) SRAM version 1 all same size dram(offset size) DRAM not used in version 1
ndash Create fastpath specific mappings for queues TCAM filters and the stats tablendash Initialize queue parameters to ldquoreasonablerdquo values (threshold weight)ndash Update VLAN Table with the code option ID (copt) and SRAM Address
bull SCD knows starting address of SRAM so add offsetndash Assume there is an SRAM table read by the microengines which maps VLAN to buffer limits
this may be the VLAN tablendash Treat as a transaction either it completes fully or no changes are recordeduint32_t retCode_t 0 = Success app defined
7WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Fast Path Commands SRM to SCDNPE Onlybull retCode_t enable_fastpath(fpid_t)
Message context ID = 0 fpid_t fast path identifieruint32_t retCode_t 0 = Success app definedndash enable by setting code option field of VLAN table entry for fp
bull retCode_t disable_fastpath(fpid)Message context ID = 0 uint16_t fpid fast path identifierretCode_t 0 = Success 1 = Pending 2 = Invalid fpid else error codendash First disable fp by setting code option field of VLAN table to 0ndash It is not an error to call disable an already disabled fastpath
bull SRM will periodically call disable_fastpath() until Success or Errorndash Then check all fast path queue lengths
bull If any queue has non-zero length then return Pendingbull If all queues are empty then return Success
bull retCode_t rem_fastpath(fpid_t)Message context ID = 0 fpid_t fast path identifierretCode_t 0 = Success 2 = Invalid fpid else error codendash Fastpath must be disabled (disable_fastpath) before it can be removedndash Remove all allocations associated with fpid
8WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
BW and Scheduler Control SRM to SCDbull Common Message
Message context ID = 0uint16_t sid Scheduler Id There are 5 schedulers per QM and 4 QMs so 0 lt= sid lt 20 (NBO)uint8_t MAC[6] 6 Byte Ethernet address
LC and NPE schedule parametersbull retCode_t set_sched_params(sid ifn BWmax BWmin)
int32_t ifn Interface numberuint32_t BWmax Kbps Maximum scheduler rateuint32_t BWmin Kbps Minimum assignable rate for this scheduler
ndash If BWmin is different from a previous call then may need to reassign weights for all associated queuesndash Must convert the rate I send (Kbps) to that expected by the hardware
Value = X Mbps 0683 = BW Kbps 683 (So no floating point needed)ndash Update scheduler parameters table in SRAM The low 16-bit word is the converted rate value and the upper 16-bits is the interface
number (ifn) The interface number must be the same value that TX uses To set only the Rate (preserves the existing interface number) SParams[sid] = (SchedParams[sid] amp 0xFFFF0000) | (0xFFFF amp Value)To set the interface number (ifn preserves the existing rate value) SParams[sid] = (SParams[sid] amp 0xFFFF) | (ifn ltlt 16)To set both SParams[sid] = (ifn ltlt 16) | (0xFFFF amp Value)
NPE Encap Control Blockbull retCode_t set_encap_cb(sid srcIP dMAC)
ndash Must update the table which associates encapsulation header IP source addresses with a schedulerSchedAddrs[sid] = ipaddr destination MAC
bull retCode_t create_mi(fpid mi sid)bull retCode_t delete_mi(fpid mi)bull retCode_t set_mi_bw(fpid mi bw)LC Dynamic MAC Tablebull retCode_t set_sched_mac(sid dstMAC srcMAC)
ndash Must update the table which associates Ethernet source and destination addresses with a schedulerSchedAddrs[sid]smac = smac (6 Bytes over 2 4B words)SchedAddrs[sid]dmac = dmac (6 Bytes over 2 4B words)
9WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
System Configuration SRM to SCD
bull Common Message context ID = 0
SRM to SCD NPE Onlybull retCode_t start_mes()
ndash The SCD starts the IXP microengines running the code that is automatically loaded when the SCD is launched
ndash Returns Success (0) Error (-1)
bull retCode_t stop_mes()ndash The SCD stops the IXP microengines if they are running
ndash Returns Success (0) Error (-1)
10WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE GPE Infobull Common Message context ID = 0
uint32_t IP 4-Byte IP address (NBO)
uint16_t PORT UDP Port number (NBO)
uint16_t QID Slice relative queue id must be converted by SCD to the tuple of qid QM id and scheduler ID
bull SRM to SCDNPE Only
retCode_t set_encap_gpe(fpid gpeIP npeIP)
retCode_t unset_encap_gpe(fpid)
unset_encap_gpe() == set_encap_gpe(fpid00)
uint16_t fpid Fast path id globally unique
bull RMP to SCDNPE Only
retCode_t set_gpe_info(exPort ldPort exQID ldQID)
retCode_t unset_gpe_info()
Set context ID to the global fast path ID
11WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Queuesbull Common
ndash if cid == 0 then qid is absolute otherwise SCD must convert from fastpath relativeuint16_t miid Meta-Interface numberuint32_t qlen Length of packet queue in Bytesuint32_t threshold the maximum number of packets queued before droppinguint32_t bw Kbs SCD must convert to the associated weightuint8_t list_type Explicit list (0) or range specification (1)uint16_t qid_list[] List of queue identifiers to associate with meta-interface mi In C or C++ this would
be an array of integers 0 n If using a range then the list has the first and one past the last id of the range For example 4 8 is the same as 4 5 6 7
bull RMP to SCDNPE Onlyndash retCode_t bind_queue(u16 miid u8 list_type u16[] qid_list)ndash retCode_t unbind_queue(u8 list_type u16[] qid_list)LC and NPEndash bw_t actual_bw set_queue_params(u16 qid u32 threshold u32 bw)
bull If a parameter is -1 then do not update in tablendash u32 threshold u32 bw get_queue_params(u16 qid)ndash u32 pktCnt u32 byteCnt get_queue_len(u16 qid)
bull SRM to SCD LC (Ingress and Egress)ndash retCode_t bind_queue_sched(u16 qid u16 sid)
bull Assign qid to scheduler sidndash retCode_t unbind_queue_sched(u16 qid)
bull Deallocate queuersquos bw from its assigned scheduler and disassociate it from that scheduler
12WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Lookup Table (TCAM)bull Common LC cid = 0 NPE cid = fpid
ndash If cid == 0 then fid is absolute otherwise it is fastpath relative and must be converteduint32_t fid Filter ID If cid == 0 then absolute otherwise relativeuint16_t dbid Database ID NPE dbid=0 LC Ingress dbid=0 Egress dbid=1struct fltr uint8_t key[N] N-Byte key value defined by code option uint8_t mask[N] N-Byte mask for key lookups defined by code option uint8_t result[M] M-Byte result vector defined by code option
bull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)LC and NPEndash ret_t write_fltr(dbid fid key mask result)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash ret_t update_result(dbid fid result)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_fltr_bykey(dbid key)fltr get_fltr_byfid(dbid fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash result lookup_fltr(dbid key)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash retcode rem_fltr_bykey(dbid key)
retcode rem_fltr_byfid(dbid fid)ndash Returns fid if filter removed otherwise -1
13WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE Filter Interfacebull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)
LC and NPEsubKeyWrap_t type rxip rxport coptKey[14] subResult_t actions sindx daddr dport sport qidsubFltr_t subKeyWrap coptMask[14] subResult ndash scdCode_t write_npe_fltr(fid subFltr)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash scdCode_t update_npe_result(fid subResult)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_npe_fltr_bykey(subKeyWrap)fltr get_npe_fltr_byfid(fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash scdCode_t lookup_npe_fltr(subKeyWrap)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash scdCode_t rem_npe_fltr_bykey(subKeyWrap)
scdCode_t rem_npe_fltr_byfid(fid)ndash Returns 0 on success or an error
14WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
IPv4 TCAM Filter Formats (on NPE)
6 82flags
2 12
0100
2
TCP RSV proto00TCP
daddr saddr sport dport tcpproto
Defined by the IPv4 Code Option 112bits
32 32 16 16 16
vlan
11
if
T = 0 Normal LookupT = 1 substrate only lookup
T
1
RX port
Substrate defined
164
TX IP daddr TX dport TX sport rsv32 16 12 1516
QM3
D Drop packetL Local delivery
rsv113 1
L1
Drsv sindx Sch2
qid16
20-bit internal qid(SCD maps slicersquos miidto QM and Sch SCD Also
maps slicersquos qid toglobal qid value)
TX IP address and sport representsthe output meta-interface The
dport is provided by the slice (RMP maps miid to tx tunnel params
use dport provided by slice)
Result 128 bits
Represents input meta-interface
global statsindex
(SCD mapsslicersquos sindx
to global value)
Key Input miid IPv4 fltr daddr saddr sport dport tcpproto
Result Flags Drop GPE sindx Output miID QID
Slice parameters
15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Reading Statisticsbull Common
ndash if cid = 0 then sindx is absolute otherwiseSCD must convert fastpath relative index
uint8_t flagsuint32_t handle opaque reference to periodic eventuint32_t sindx stats index
bull RMP to SCD RLIGIU to SCDLC and NPEndash stats = read_stats(sindx flags)
bull struct stats uint32_t cnt tstampbull Uses flags L and W
ndash result = clear_stats(sindx)bull result success or error code
ndash handle create_periodic(sindx P cnt flags)bull uint32_t P periodic interval in millisecondsbull uint16_t cnt Number of samples to keep in a history bufferbull Uses all 3 flags
ndash retcode del_periodic(handle)ndash retcode set_callback(handle udp_port)
bull push model every (cntperiod) milliseconds send the last cnt samples to the clientrsquos UDP port udp_port
ndash stats = get_periodic(handle)bull pull model returns the last cnt samples
FlagsW 0 - Packts 1 - BytesL 0 - PreQ 1 - PostQT 0 - Push 1 - PullX Donrsquot Care
06 3 12457WLTXXXXX
16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Accessing Memorybull common
Message context id NPE equals fpid for LC must equal 0uint32_t offset Offset in from start of assigned SRAM block uint32_t len Number of bytes to readwriteuint8_t data[X] Data buffer with X Bytesuint32_t kpa Kernel physical address
bull SRM to SCD LC and NPEndash result write_mem(kpa len data)
bull Message context ID must == 0bull Can read any valid physical address on the xscale
ndash data read_mem(kpa len)bull Message context ID must == 0bull Can read any valid physical address on the xscale
bull RMP to SCD NPE Onlyndash result write_sram(offset len data)
bull Offset is relative to the starting address of SRAM block allocated to slice SCD must verify the write operation is within bounds
ndash data read_sram(offset len)bull Read len bytes from SRAM block and return to client First verify offset and len with within
bounds for slice
17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
New Commands
bull NPE Onlybull retCode_t set_src_hwaddr(hwaddr_t)
ndash context must be 0
bull retCode_t set_iface_table(ipAddr_t[16])ndash context must be 0
6WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Fast Path Commands SRM to SCDNPE Onlybull retCode_t set_fastpath(fpid copt_t VLAN rcnts_t Mem[])
Message context ID = 0 fpid Fast-Path ID Unique within SPP nodecopt Code option identifier 0 is invalid IPv4 = 1 I3 = 2uint16_t VLAN Ethernet VLAN tag used in datapath to identify fastpath instancercnts_t = cntr_t Qs Fltrs Buffers Statsstruct uint32_t offset size Mem[2] SRAM and DRAM allocation values are
relative to the start of an SRAM block set aside for fastpath useMem = sram(offset size) SRAM version 1 all same size dram(offset size) DRAM not used in version 1
ndash Create fastpath specific mappings for queues TCAM filters and the stats tablendash Initialize queue parameters to ldquoreasonablerdquo values (threshold weight)ndash Update VLAN Table with the code option ID (copt) and SRAM Address
bull SCD knows starting address of SRAM so add offsetndash Assume there is an SRAM table read by the microengines which maps VLAN to buffer limits
this may be the VLAN tablendash Treat as a transaction either it completes fully or no changes are recordeduint32_t retCode_t 0 = Success app defined
7WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Fast Path Commands SRM to SCDNPE Onlybull retCode_t enable_fastpath(fpid_t)
Message context ID = 0 fpid_t fast path identifieruint32_t retCode_t 0 = Success app definedndash enable by setting code option field of VLAN table entry for fp
bull retCode_t disable_fastpath(fpid)Message context ID = 0 uint16_t fpid fast path identifierretCode_t 0 = Success 1 = Pending 2 = Invalid fpid else error codendash First disable fp by setting code option field of VLAN table to 0ndash It is not an error to call disable an already disabled fastpath
bull SRM will periodically call disable_fastpath() until Success or Errorndash Then check all fast path queue lengths
bull If any queue has non-zero length then return Pendingbull If all queues are empty then return Success
bull retCode_t rem_fastpath(fpid_t)Message context ID = 0 fpid_t fast path identifierretCode_t 0 = Success 2 = Invalid fpid else error codendash Fastpath must be disabled (disable_fastpath) before it can be removedndash Remove all allocations associated with fpid
8WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
BW and Scheduler Control SRM to SCDbull Common Message
Message context ID = 0uint16_t sid Scheduler Id There are 5 schedulers per QM and 4 QMs so 0 lt= sid lt 20 (NBO)uint8_t MAC[6] 6 Byte Ethernet address
LC and NPE schedule parametersbull retCode_t set_sched_params(sid ifn BWmax BWmin)
int32_t ifn Interface numberuint32_t BWmax Kbps Maximum scheduler rateuint32_t BWmin Kbps Minimum assignable rate for this scheduler
ndash If BWmin is different from a previous call then may need to reassign weights for all associated queuesndash Must convert the rate I send (Kbps) to that expected by the hardware
Value = X Mbps 0683 = BW Kbps 683 (So no floating point needed)ndash Update scheduler parameters table in SRAM The low 16-bit word is the converted rate value and the upper 16-bits is the interface
number (ifn) The interface number must be the same value that TX uses To set only the Rate (preserves the existing interface number) SParams[sid] = (SchedParams[sid] amp 0xFFFF0000) | (0xFFFF amp Value)To set the interface number (ifn preserves the existing rate value) SParams[sid] = (SParams[sid] amp 0xFFFF) | (ifn ltlt 16)To set both SParams[sid] = (ifn ltlt 16) | (0xFFFF amp Value)
NPE Encap Control Blockbull retCode_t set_encap_cb(sid srcIP dMAC)
ndash Must update the table which associates encapsulation header IP source addresses with a schedulerSchedAddrs[sid] = ipaddr destination MAC
bull retCode_t create_mi(fpid mi sid)bull retCode_t delete_mi(fpid mi)bull retCode_t set_mi_bw(fpid mi bw)LC Dynamic MAC Tablebull retCode_t set_sched_mac(sid dstMAC srcMAC)
ndash Must update the table which associates Ethernet source and destination addresses with a schedulerSchedAddrs[sid]smac = smac (6 Bytes over 2 4B words)SchedAddrs[sid]dmac = dmac (6 Bytes over 2 4B words)
9WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
System Configuration SRM to SCD
bull Common Message context ID = 0
SRM to SCD NPE Onlybull retCode_t start_mes()
ndash The SCD starts the IXP microengines running the code that is automatically loaded when the SCD is launched
ndash Returns Success (0) Error (-1)
bull retCode_t stop_mes()ndash The SCD stops the IXP microengines if they are running
ndash Returns Success (0) Error (-1)
10WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE GPE Infobull Common Message context ID = 0
uint32_t IP 4-Byte IP address (NBO)
uint16_t PORT UDP Port number (NBO)
uint16_t QID Slice relative queue id must be converted by SCD to the tuple of qid QM id and scheduler ID
bull SRM to SCDNPE Only
retCode_t set_encap_gpe(fpid gpeIP npeIP)
retCode_t unset_encap_gpe(fpid)
unset_encap_gpe() == set_encap_gpe(fpid00)
uint16_t fpid Fast path id globally unique
bull RMP to SCDNPE Only
retCode_t set_gpe_info(exPort ldPort exQID ldQID)
retCode_t unset_gpe_info()
Set context ID to the global fast path ID
11WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Queuesbull Common
ndash if cid == 0 then qid is absolute otherwise SCD must convert from fastpath relativeuint16_t miid Meta-Interface numberuint32_t qlen Length of packet queue in Bytesuint32_t threshold the maximum number of packets queued before droppinguint32_t bw Kbs SCD must convert to the associated weightuint8_t list_type Explicit list (0) or range specification (1)uint16_t qid_list[] List of queue identifiers to associate with meta-interface mi In C or C++ this would
be an array of integers 0 n If using a range then the list has the first and one past the last id of the range For example 4 8 is the same as 4 5 6 7
bull RMP to SCDNPE Onlyndash retCode_t bind_queue(u16 miid u8 list_type u16[] qid_list)ndash retCode_t unbind_queue(u8 list_type u16[] qid_list)LC and NPEndash bw_t actual_bw set_queue_params(u16 qid u32 threshold u32 bw)
bull If a parameter is -1 then do not update in tablendash u32 threshold u32 bw get_queue_params(u16 qid)ndash u32 pktCnt u32 byteCnt get_queue_len(u16 qid)
bull SRM to SCD LC (Ingress and Egress)ndash retCode_t bind_queue_sched(u16 qid u16 sid)
bull Assign qid to scheduler sidndash retCode_t unbind_queue_sched(u16 qid)
bull Deallocate queuersquos bw from its assigned scheduler and disassociate it from that scheduler
12WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Lookup Table (TCAM)bull Common LC cid = 0 NPE cid = fpid
ndash If cid == 0 then fid is absolute otherwise it is fastpath relative and must be converteduint32_t fid Filter ID If cid == 0 then absolute otherwise relativeuint16_t dbid Database ID NPE dbid=0 LC Ingress dbid=0 Egress dbid=1struct fltr uint8_t key[N] N-Byte key value defined by code option uint8_t mask[N] N-Byte mask for key lookups defined by code option uint8_t result[M] M-Byte result vector defined by code option
bull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)LC and NPEndash ret_t write_fltr(dbid fid key mask result)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash ret_t update_result(dbid fid result)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_fltr_bykey(dbid key)fltr get_fltr_byfid(dbid fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash result lookup_fltr(dbid key)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash retcode rem_fltr_bykey(dbid key)
retcode rem_fltr_byfid(dbid fid)ndash Returns fid if filter removed otherwise -1
13WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE Filter Interfacebull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)
LC and NPEsubKeyWrap_t type rxip rxport coptKey[14] subResult_t actions sindx daddr dport sport qidsubFltr_t subKeyWrap coptMask[14] subResult ndash scdCode_t write_npe_fltr(fid subFltr)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash scdCode_t update_npe_result(fid subResult)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_npe_fltr_bykey(subKeyWrap)fltr get_npe_fltr_byfid(fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash scdCode_t lookup_npe_fltr(subKeyWrap)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash scdCode_t rem_npe_fltr_bykey(subKeyWrap)
scdCode_t rem_npe_fltr_byfid(fid)ndash Returns 0 on success or an error
14WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
IPv4 TCAM Filter Formats (on NPE)
6 82flags
2 12
0100
2
TCP RSV proto00TCP
daddr saddr sport dport tcpproto
Defined by the IPv4 Code Option 112bits
32 32 16 16 16
vlan
11
if
T = 0 Normal LookupT = 1 substrate only lookup
T
1
RX port
Substrate defined
164
TX IP daddr TX dport TX sport rsv32 16 12 1516
QM3
D Drop packetL Local delivery
rsv113 1
L1
Drsv sindx Sch2
qid16
20-bit internal qid(SCD maps slicersquos miidto QM and Sch SCD Also
maps slicersquos qid toglobal qid value)
TX IP address and sport representsthe output meta-interface The
dport is provided by the slice (RMP maps miid to tx tunnel params
use dport provided by slice)
Result 128 bits
Represents input meta-interface
global statsindex
(SCD mapsslicersquos sindx
to global value)
Key Input miid IPv4 fltr daddr saddr sport dport tcpproto
Result Flags Drop GPE sindx Output miID QID
Slice parameters
15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Reading Statisticsbull Common
ndash if cid = 0 then sindx is absolute otherwiseSCD must convert fastpath relative index
uint8_t flagsuint32_t handle opaque reference to periodic eventuint32_t sindx stats index
bull RMP to SCD RLIGIU to SCDLC and NPEndash stats = read_stats(sindx flags)
bull struct stats uint32_t cnt tstampbull Uses flags L and W
ndash result = clear_stats(sindx)bull result success or error code
ndash handle create_periodic(sindx P cnt flags)bull uint32_t P periodic interval in millisecondsbull uint16_t cnt Number of samples to keep in a history bufferbull Uses all 3 flags
ndash retcode del_periodic(handle)ndash retcode set_callback(handle udp_port)
bull push model every (cntperiod) milliseconds send the last cnt samples to the clientrsquos UDP port udp_port
ndash stats = get_periodic(handle)bull pull model returns the last cnt samples
FlagsW 0 - Packts 1 - BytesL 0 - PreQ 1 - PostQT 0 - Push 1 - PullX Donrsquot Care
06 3 12457WLTXXXXX
16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Accessing Memorybull common
Message context id NPE equals fpid for LC must equal 0uint32_t offset Offset in from start of assigned SRAM block uint32_t len Number of bytes to readwriteuint8_t data[X] Data buffer with X Bytesuint32_t kpa Kernel physical address
bull SRM to SCD LC and NPEndash result write_mem(kpa len data)
bull Message context ID must == 0bull Can read any valid physical address on the xscale
ndash data read_mem(kpa len)bull Message context ID must == 0bull Can read any valid physical address on the xscale
bull RMP to SCD NPE Onlyndash result write_sram(offset len data)
bull Offset is relative to the starting address of SRAM block allocated to slice SCD must verify the write operation is within bounds
ndash data read_sram(offset len)bull Read len bytes from SRAM block and return to client First verify offset and len with within
bounds for slice
17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
New Commands
bull NPE Onlybull retCode_t set_src_hwaddr(hwaddr_t)
ndash context must be 0
bull retCode_t set_iface_table(ipAddr_t[16])ndash context must be 0
7WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Fast Path Commands SRM to SCDNPE Onlybull retCode_t enable_fastpath(fpid_t)
Message context ID = 0 fpid_t fast path identifieruint32_t retCode_t 0 = Success app definedndash enable by setting code option field of VLAN table entry for fp
bull retCode_t disable_fastpath(fpid)Message context ID = 0 uint16_t fpid fast path identifierretCode_t 0 = Success 1 = Pending 2 = Invalid fpid else error codendash First disable fp by setting code option field of VLAN table to 0ndash It is not an error to call disable an already disabled fastpath
bull SRM will periodically call disable_fastpath() until Success or Errorndash Then check all fast path queue lengths
bull If any queue has non-zero length then return Pendingbull If all queues are empty then return Success
bull retCode_t rem_fastpath(fpid_t)Message context ID = 0 fpid_t fast path identifierretCode_t 0 = Success 2 = Invalid fpid else error codendash Fastpath must be disabled (disable_fastpath) before it can be removedndash Remove all allocations associated with fpid
8WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
BW and Scheduler Control SRM to SCDbull Common Message
Message context ID = 0uint16_t sid Scheduler Id There are 5 schedulers per QM and 4 QMs so 0 lt= sid lt 20 (NBO)uint8_t MAC[6] 6 Byte Ethernet address
LC and NPE schedule parametersbull retCode_t set_sched_params(sid ifn BWmax BWmin)
int32_t ifn Interface numberuint32_t BWmax Kbps Maximum scheduler rateuint32_t BWmin Kbps Minimum assignable rate for this scheduler
ndash If BWmin is different from a previous call then may need to reassign weights for all associated queuesndash Must convert the rate I send (Kbps) to that expected by the hardware
Value = X Mbps 0683 = BW Kbps 683 (So no floating point needed)ndash Update scheduler parameters table in SRAM The low 16-bit word is the converted rate value and the upper 16-bits is the interface
number (ifn) The interface number must be the same value that TX uses To set only the Rate (preserves the existing interface number) SParams[sid] = (SchedParams[sid] amp 0xFFFF0000) | (0xFFFF amp Value)To set the interface number (ifn preserves the existing rate value) SParams[sid] = (SParams[sid] amp 0xFFFF) | (ifn ltlt 16)To set both SParams[sid] = (ifn ltlt 16) | (0xFFFF amp Value)
NPE Encap Control Blockbull retCode_t set_encap_cb(sid srcIP dMAC)
ndash Must update the table which associates encapsulation header IP source addresses with a schedulerSchedAddrs[sid] = ipaddr destination MAC
bull retCode_t create_mi(fpid mi sid)bull retCode_t delete_mi(fpid mi)bull retCode_t set_mi_bw(fpid mi bw)LC Dynamic MAC Tablebull retCode_t set_sched_mac(sid dstMAC srcMAC)
ndash Must update the table which associates Ethernet source and destination addresses with a schedulerSchedAddrs[sid]smac = smac (6 Bytes over 2 4B words)SchedAddrs[sid]dmac = dmac (6 Bytes over 2 4B words)
9WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
System Configuration SRM to SCD
bull Common Message context ID = 0
SRM to SCD NPE Onlybull retCode_t start_mes()
ndash The SCD starts the IXP microengines running the code that is automatically loaded when the SCD is launched
ndash Returns Success (0) Error (-1)
bull retCode_t stop_mes()ndash The SCD stops the IXP microengines if they are running
ndash Returns Success (0) Error (-1)
10WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE GPE Infobull Common Message context ID = 0
uint32_t IP 4-Byte IP address (NBO)
uint16_t PORT UDP Port number (NBO)
uint16_t QID Slice relative queue id must be converted by SCD to the tuple of qid QM id and scheduler ID
bull SRM to SCDNPE Only
retCode_t set_encap_gpe(fpid gpeIP npeIP)
retCode_t unset_encap_gpe(fpid)
unset_encap_gpe() == set_encap_gpe(fpid00)
uint16_t fpid Fast path id globally unique
bull RMP to SCDNPE Only
retCode_t set_gpe_info(exPort ldPort exQID ldQID)
retCode_t unset_gpe_info()
Set context ID to the global fast path ID
11WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Queuesbull Common
ndash if cid == 0 then qid is absolute otherwise SCD must convert from fastpath relativeuint16_t miid Meta-Interface numberuint32_t qlen Length of packet queue in Bytesuint32_t threshold the maximum number of packets queued before droppinguint32_t bw Kbs SCD must convert to the associated weightuint8_t list_type Explicit list (0) or range specification (1)uint16_t qid_list[] List of queue identifiers to associate with meta-interface mi In C or C++ this would
be an array of integers 0 n If using a range then the list has the first and one past the last id of the range For example 4 8 is the same as 4 5 6 7
bull RMP to SCDNPE Onlyndash retCode_t bind_queue(u16 miid u8 list_type u16[] qid_list)ndash retCode_t unbind_queue(u8 list_type u16[] qid_list)LC and NPEndash bw_t actual_bw set_queue_params(u16 qid u32 threshold u32 bw)
bull If a parameter is -1 then do not update in tablendash u32 threshold u32 bw get_queue_params(u16 qid)ndash u32 pktCnt u32 byteCnt get_queue_len(u16 qid)
bull SRM to SCD LC (Ingress and Egress)ndash retCode_t bind_queue_sched(u16 qid u16 sid)
bull Assign qid to scheduler sidndash retCode_t unbind_queue_sched(u16 qid)
bull Deallocate queuersquos bw from its assigned scheduler and disassociate it from that scheduler
12WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Lookup Table (TCAM)bull Common LC cid = 0 NPE cid = fpid
ndash If cid == 0 then fid is absolute otherwise it is fastpath relative and must be converteduint32_t fid Filter ID If cid == 0 then absolute otherwise relativeuint16_t dbid Database ID NPE dbid=0 LC Ingress dbid=0 Egress dbid=1struct fltr uint8_t key[N] N-Byte key value defined by code option uint8_t mask[N] N-Byte mask for key lookups defined by code option uint8_t result[M] M-Byte result vector defined by code option
bull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)LC and NPEndash ret_t write_fltr(dbid fid key mask result)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash ret_t update_result(dbid fid result)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_fltr_bykey(dbid key)fltr get_fltr_byfid(dbid fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash result lookup_fltr(dbid key)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash retcode rem_fltr_bykey(dbid key)
retcode rem_fltr_byfid(dbid fid)ndash Returns fid if filter removed otherwise -1
13WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE Filter Interfacebull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)
LC and NPEsubKeyWrap_t type rxip rxport coptKey[14] subResult_t actions sindx daddr dport sport qidsubFltr_t subKeyWrap coptMask[14] subResult ndash scdCode_t write_npe_fltr(fid subFltr)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash scdCode_t update_npe_result(fid subResult)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_npe_fltr_bykey(subKeyWrap)fltr get_npe_fltr_byfid(fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash scdCode_t lookup_npe_fltr(subKeyWrap)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash scdCode_t rem_npe_fltr_bykey(subKeyWrap)
scdCode_t rem_npe_fltr_byfid(fid)ndash Returns 0 on success or an error
14WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
IPv4 TCAM Filter Formats (on NPE)
6 82flags
2 12
0100
2
TCP RSV proto00TCP
daddr saddr sport dport tcpproto
Defined by the IPv4 Code Option 112bits
32 32 16 16 16
vlan
11
if
T = 0 Normal LookupT = 1 substrate only lookup
T
1
RX port
Substrate defined
164
TX IP daddr TX dport TX sport rsv32 16 12 1516
QM3
D Drop packetL Local delivery
rsv113 1
L1
Drsv sindx Sch2
qid16
20-bit internal qid(SCD maps slicersquos miidto QM and Sch SCD Also
maps slicersquos qid toglobal qid value)
TX IP address and sport representsthe output meta-interface The
dport is provided by the slice (RMP maps miid to tx tunnel params
use dport provided by slice)
Result 128 bits
Represents input meta-interface
global statsindex
(SCD mapsslicersquos sindx
to global value)
Key Input miid IPv4 fltr daddr saddr sport dport tcpproto
Result Flags Drop GPE sindx Output miID QID
Slice parameters
15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Reading Statisticsbull Common
ndash if cid = 0 then sindx is absolute otherwiseSCD must convert fastpath relative index
uint8_t flagsuint32_t handle opaque reference to periodic eventuint32_t sindx stats index
bull RMP to SCD RLIGIU to SCDLC and NPEndash stats = read_stats(sindx flags)
bull struct stats uint32_t cnt tstampbull Uses flags L and W
ndash result = clear_stats(sindx)bull result success or error code
ndash handle create_periodic(sindx P cnt flags)bull uint32_t P periodic interval in millisecondsbull uint16_t cnt Number of samples to keep in a history bufferbull Uses all 3 flags
ndash retcode del_periodic(handle)ndash retcode set_callback(handle udp_port)
bull push model every (cntperiod) milliseconds send the last cnt samples to the clientrsquos UDP port udp_port
ndash stats = get_periodic(handle)bull pull model returns the last cnt samples
FlagsW 0 - Packts 1 - BytesL 0 - PreQ 1 - PostQT 0 - Push 1 - PullX Donrsquot Care
06 3 12457WLTXXXXX
16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Accessing Memorybull common
Message context id NPE equals fpid for LC must equal 0uint32_t offset Offset in from start of assigned SRAM block uint32_t len Number of bytes to readwriteuint8_t data[X] Data buffer with X Bytesuint32_t kpa Kernel physical address
bull SRM to SCD LC and NPEndash result write_mem(kpa len data)
bull Message context ID must == 0bull Can read any valid physical address on the xscale
ndash data read_mem(kpa len)bull Message context ID must == 0bull Can read any valid physical address on the xscale
bull RMP to SCD NPE Onlyndash result write_sram(offset len data)
bull Offset is relative to the starting address of SRAM block allocated to slice SCD must verify the write operation is within bounds
ndash data read_sram(offset len)bull Read len bytes from SRAM block and return to client First verify offset and len with within
bounds for slice
17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
New Commands
bull NPE Onlybull retCode_t set_src_hwaddr(hwaddr_t)
ndash context must be 0
bull retCode_t set_iface_table(ipAddr_t[16])ndash context must be 0
8WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
BW and Scheduler Control SRM to SCDbull Common Message
Message context ID = 0uint16_t sid Scheduler Id There are 5 schedulers per QM and 4 QMs so 0 lt= sid lt 20 (NBO)uint8_t MAC[6] 6 Byte Ethernet address
LC and NPE schedule parametersbull retCode_t set_sched_params(sid ifn BWmax BWmin)
int32_t ifn Interface numberuint32_t BWmax Kbps Maximum scheduler rateuint32_t BWmin Kbps Minimum assignable rate for this scheduler
ndash If BWmin is different from a previous call then may need to reassign weights for all associated queuesndash Must convert the rate I send (Kbps) to that expected by the hardware
Value = X Mbps 0683 = BW Kbps 683 (So no floating point needed)ndash Update scheduler parameters table in SRAM The low 16-bit word is the converted rate value and the upper 16-bits is the interface
number (ifn) The interface number must be the same value that TX uses To set only the Rate (preserves the existing interface number) SParams[sid] = (SchedParams[sid] amp 0xFFFF0000) | (0xFFFF amp Value)To set the interface number (ifn preserves the existing rate value) SParams[sid] = (SParams[sid] amp 0xFFFF) | (ifn ltlt 16)To set both SParams[sid] = (ifn ltlt 16) | (0xFFFF amp Value)
NPE Encap Control Blockbull retCode_t set_encap_cb(sid srcIP dMAC)
ndash Must update the table which associates encapsulation header IP source addresses with a schedulerSchedAddrs[sid] = ipaddr destination MAC
bull retCode_t create_mi(fpid mi sid)bull retCode_t delete_mi(fpid mi)bull retCode_t set_mi_bw(fpid mi bw)LC Dynamic MAC Tablebull retCode_t set_sched_mac(sid dstMAC srcMAC)
ndash Must update the table which associates Ethernet source and destination addresses with a schedulerSchedAddrs[sid]smac = smac (6 Bytes over 2 4B words)SchedAddrs[sid]dmac = dmac (6 Bytes over 2 4B words)
9WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
System Configuration SRM to SCD
bull Common Message context ID = 0
SRM to SCD NPE Onlybull retCode_t start_mes()
ndash The SCD starts the IXP microengines running the code that is automatically loaded when the SCD is launched
ndash Returns Success (0) Error (-1)
bull retCode_t stop_mes()ndash The SCD stops the IXP microengines if they are running
ndash Returns Success (0) Error (-1)
10WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE GPE Infobull Common Message context ID = 0
uint32_t IP 4-Byte IP address (NBO)
uint16_t PORT UDP Port number (NBO)
uint16_t QID Slice relative queue id must be converted by SCD to the tuple of qid QM id and scheduler ID
bull SRM to SCDNPE Only
retCode_t set_encap_gpe(fpid gpeIP npeIP)
retCode_t unset_encap_gpe(fpid)
unset_encap_gpe() == set_encap_gpe(fpid00)
uint16_t fpid Fast path id globally unique
bull RMP to SCDNPE Only
retCode_t set_gpe_info(exPort ldPort exQID ldQID)
retCode_t unset_gpe_info()
Set context ID to the global fast path ID
11WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Queuesbull Common
ndash if cid == 0 then qid is absolute otherwise SCD must convert from fastpath relativeuint16_t miid Meta-Interface numberuint32_t qlen Length of packet queue in Bytesuint32_t threshold the maximum number of packets queued before droppinguint32_t bw Kbs SCD must convert to the associated weightuint8_t list_type Explicit list (0) or range specification (1)uint16_t qid_list[] List of queue identifiers to associate with meta-interface mi In C or C++ this would
be an array of integers 0 n If using a range then the list has the first and one past the last id of the range For example 4 8 is the same as 4 5 6 7
bull RMP to SCDNPE Onlyndash retCode_t bind_queue(u16 miid u8 list_type u16[] qid_list)ndash retCode_t unbind_queue(u8 list_type u16[] qid_list)LC and NPEndash bw_t actual_bw set_queue_params(u16 qid u32 threshold u32 bw)
bull If a parameter is -1 then do not update in tablendash u32 threshold u32 bw get_queue_params(u16 qid)ndash u32 pktCnt u32 byteCnt get_queue_len(u16 qid)
bull SRM to SCD LC (Ingress and Egress)ndash retCode_t bind_queue_sched(u16 qid u16 sid)
bull Assign qid to scheduler sidndash retCode_t unbind_queue_sched(u16 qid)
bull Deallocate queuersquos bw from its assigned scheduler and disassociate it from that scheduler
12WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Lookup Table (TCAM)bull Common LC cid = 0 NPE cid = fpid
ndash If cid == 0 then fid is absolute otherwise it is fastpath relative and must be converteduint32_t fid Filter ID If cid == 0 then absolute otherwise relativeuint16_t dbid Database ID NPE dbid=0 LC Ingress dbid=0 Egress dbid=1struct fltr uint8_t key[N] N-Byte key value defined by code option uint8_t mask[N] N-Byte mask for key lookups defined by code option uint8_t result[M] M-Byte result vector defined by code option
bull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)LC and NPEndash ret_t write_fltr(dbid fid key mask result)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash ret_t update_result(dbid fid result)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_fltr_bykey(dbid key)fltr get_fltr_byfid(dbid fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash result lookup_fltr(dbid key)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash retcode rem_fltr_bykey(dbid key)
retcode rem_fltr_byfid(dbid fid)ndash Returns fid if filter removed otherwise -1
13WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE Filter Interfacebull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)
LC and NPEsubKeyWrap_t type rxip rxport coptKey[14] subResult_t actions sindx daddr dport sport qidsubFltr_t subKeyWrap coptMask[14] subResult ndash scdCode_t write_npe_fltr(fid subFltr)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash scdCode_t update_npe_result(fid subResult)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_npe_fltr_bykey(subKeyWrap)fltr get_npe_fltr_byfid(fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash scdCode_t lookup_npe_fltr(subKeyWrap)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash scdCode_t rem_npe_fltr_bykey(subKeyWrap)
scdCode_t rem_npe_fltr_byfid(fid)ndash Returns 0 on success or an error
14WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
IPv4 TCAM Filter Formats (on NPE)
6 82flags
2 12
0100
2
TCP RSV proto00TCP
daddr saddr sport dport tcpproto
Defined by the IPv4 Code Option 112bits
32 32 16 16 16
vlan
11
if
T = 0 Normal LookupT = 1 substrate only lookup
T
1
RX port
Substrate defined
164
TX IP daddr TX dport TX sport rsv32 16 12 1516
QM3
D Drop packetL Local delivery
rsv113 1
L1
Drsv sindx Sch2
qid16
20-bit internal qid(SCD maps slicersquos miidto QM and Sch SCD Also
maps slicersquos qid toglobal qid value)
TX IP address and sport representsthe output meta-interface The
dport is provided by the slice (RMP maps miid to tx tunnel params
use dport provided by slice)
Result 128 bits
Represents input meta-interface
global statsindex
(SCD mapsslicersquos sindx
to global value)
Key Input miid IPv4 fltr daddr saddr sport dport tcpproto
Result Flags Drop GPE sindx Output miID QID
Slice parameters
15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Reading Statisticsbull Common
ndash if cid = 0 then sindx is absolute otherwiseSCD must convert fastpath relative index
uint8_t flagsuint32_t handle opaque reference to periodic eventuint32_t sindx stats index
bull RMP to SCD RLIGIU to SCDLC and NPEndash stats = read_stats(sindx flags)
bull struct stats uint32_t cnt tstampbull Uses flags L and W
ndash result = clear_stats(sindx)bull result success or error code
ndash handle create_periodic(sindx P cnt flags)bull uint32_t P periodic interval in millisecondsbull uint16_t cnt Number of samples to keep in a history bufferbull Uses all 3 flags
ndash retcode del_periodic(handle)ndash retcode set_callback(handle udp_port)
bull push model every (cntperiod) milliseconds send the last cnt samples to the clientrsquos UDP port udp_port
ndash stats = get_periodic(handle)bull pull model returns the last cnt samples
FlagsW 0 - Packts 1 - BytesL 0 - PreQ 1 - PostQT 0 - Push 1 - PullX Donrsquot Care
06 3 12457WLTXXXXX
16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Accessing Memorybull common
Message context id NPE equals fpid for LC must equal 0uint32_t offset Offset in from start of assigned SRAM block uint32_t len Number of bytes to readwriteuint8_t data[X] Data buffer with X Bytesuint32_t kpa Kernel physical address
bull SRM to SCD LC and NPEndash result write_mem(kpa len data)
bull Message context ID must == 0bull Can read any valid physical address on the xscale
ndash data read_mem(kpa len)bull Message context ID must == 0bull Can read any valid physical address on the xscale
bull RMP to SCD NPE Onlyndash result write_sram(offset len data)
bull Offset is relative to the starting address of SRAM block allocated to slice SCD must verify the write operation is within bounds
ndash data read_sram(offset len)bull Read len bytes from SRAM block and return to client First verify offset and len with within
bounds for slice
17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
New Commands
bull NPE Onlybull retCode_t set_src_hwaddr(hwaddr_t)
ndash context must be 0
bull retCode_t set_iface_table(ipAddr_t[16])ndash context must be 0
9WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
System Configuration SRM to SCD
bull Common Message context ID = 0
SRM to SCD NPE Onlybull retCode_t start_mes()
ndash The SCD starts the IXP microengines running the code that is automatically loaded when the SCD is launched
ndash Returns Success (0) Error (-1)
bull retCode_t stop_mes()ndash The SCD stops the IXP microengines if they are running
ndash Returns Success (0) Error (-1)
10WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE GPE Infobull Common Message context ID = 0
uint32_t IP 4-Byte IP address (NBO)
uint16_t PORT UDP Port number (NBO)
uint16_t QID Slice relative queue id must be converted by SCD to the tuple of qid QM id and scheduler ID
bull SRM to SCDNPE Only
retCode_t set_encap_gpe(fpid gpeIP npeIP)
retCode_t unset_encap_gpe(fpid)
unset_encap_gpe() == set_encap_gpe(fpid00)
uint16_t fpid Fast path id globally unique
bull RMP to SCDNPE Only
retCode_t set_gpe_info(exPort ldPort exQID ldQID)
retCode_t unset_gpe_info()
Set context ID to the global fast path ID
11WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Queuesbull Common
ndash if cid == 0 then qid is absolute otherwise SCD must convert from fastpath relativeuint16_t miid Meta-Interface numberuint32_t qlen Length of packet queue in Bytesuint32_t threshold the maximum number of packets queued before droppinguint32_t bw Kbs SCD must convert to the associated weightuint8_t list_type Explicit list (0) or range specification (1)uint16_t qid_list[] List of queue identifiers to associate with meta-interface mi In C or C++ this would
be an array of integers 0 n If using a range then the list has the first and one past the last id of the range For example 4 8 is the same as 4 5 6 7
bull RMP to SCDNPE Onlyndash retCode_t bind_queue(u16 miid u8 list_type u16[] qid_list)ndash retCode_t unbind_queue(u8 list_type u16[] qid_list)LC and NPEndash bw_t actual_bw set_queue_params(u16 qid u32 threshold u32 bw)
bull If a parameter is -1 then do not update in tablendash u32 threshold u32 bw get_queue_params(u16 qid)ndash u32 pktCnt u32 byteCnt get_queue_len(u16 qid)
bull SRM to SCD LC (Ingress and Egress)ndash retCode_t bind_queue_sched(u16 qid u16 sid)
bull Assign qid to scheduler sidndash retCode_t unbind_queue_sched(u16 qid)
bull Deallocate queuersquos bw from its assigned scheduler and disassociate it from that scheduler
12WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Lookup Table (TCAM)bull Common LC cid = 0 NPE cid = fpid
ndash If cid == 0 then fid is absolute otherwise it is fastpath relative and must be converteduint32_t fid Filter ID If cid == 0 then absolute otherwise relativeuint16_t dbid Database ID NPE dbid=0 LC Ingress dbid=0 Egress dbid=1struct fltr uint8_t key[N] N-Byte key value defined by code option uint8_t mask[N] N-Byte mask for key lookups defined by code option uint8_t result[M] M-Byte result vector defined by code option
bull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)LC and NPEndash ret_t write_fltr(dbid fid key mask result)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash ret_t update_result(dbid fid result)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_fltr_bykey(dbid key)fltr get_fltr_byfid(dbid fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash result lookup_fltr(dbid key)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash retcode rem_fltr_bykey(dbid key)
retcode rem_fltr_byfid(dbid fid)ndash Returns fid if filter removed otherwise -1
13WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE Filter Interfacebull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)
LC and NPEsubKeyWrap_t type rxip rxport coptKey[14] subResult_t actions sindx daddr dport sport qidsubFltr_t subKeyWrap coptMask[14] subResult ndash scdCode_t write_npe_fltr(fid subFltr)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash scdCode_t update_npe_result(fid subResult)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_npe_fltr_bykey(subKeyWrap)fltr get_npe_fltr_byfid(fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash scdCode_t lookup_npe_fltr(subKeyWrap)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash scdCode_t rem_npe_fltr_bykey(subKeyWrap)
scdCode_t rem_npe_fltr_byfid(fid)ndash Returns 0 on success or an error
14WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
IPv4 TCAM Filter Formats (on NPE)
6 82flags
2 12
0100
2
TCP RSV proto00TCP
daddr saddr sport dport tcpproto
Defined by the IPv4 Code Option 112bits
32 32 16 16 16
vlan
11
if
T = 0 Normal LookupT = 1 substrate only lookup
T
1
RX port
Substrate defined
164
TX IP daddr TX dport TX sport rsv32 16 12 1516
QM3
D Drop packetL Local delivery
rsv113 1
L1
Drsv sindx Sch2
qid16
20-bit internal qid(SCD maps slicersquos miidto QM and Sch SCD Also
maps slicersquos qid toglobal qid value)
TX IP address and sport representsthe output meta-interface The
dport is provided by the slice (RMP maps miid to tx tunnel params
use dport provided by slice)
Result 128 bits
Represents input meta-interface
global statsindex
(SCD mapsslicersquos sindx
to global value)
Key Input miid IPv4 fltr daddr saddr sport dport tcpproto
Result Flags Drop GPE sindx Output miID QID
Slice parameters
15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Reading Statisticsbull Common
ndash if cid = 0 then sindx is absolute otherwiseSCD must convert fastpath relative index
uint8_t flagsuint32_t handle opaque reference to periodic eventuint32_t sindx stats index
bull RMP to SCD RLIGIU to SCDLC and NPEndash stats = read_stats(sindx flags)
bull struct stats uint32_t cnt tstampbull Uses flags L and W
ndash result = clear_stats(sindx)bull result success or error code
ndash handle create_periodic(sindx P cnt flags)bull uint32_t P periodic interval in millisecondsbull uint16_t cnt Number of samples to keep in a history bufferbull Uses all 3 flags
ndash retcode del_periodic(handle)ndash retcode set_callback(handle udp_port)
bull push model every (cntperiod) milliseconds send the last cnt samples to the clientrsquos UDP port udp_port
ndash stats = get_periodic(handle)bull pull model returns the last cnt samples
FlagsW 0 - Packts 1 - BytesL 0 - PreQ 1 - PostQT 0 - Push 1 - PullX Donrsquot Care
06 3 12457WLTXXXXX
16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Accessing Memorybull common
Message context id NPE equals fpid for LC must equal 0uint32_t offset Offset in from start of assigned SRAM block uint32_t len Number of bytes to readwriteuint8_t data[X] Data buffer with X Bytesuint32_t kpa Kernel physical address
bull SRM to SCD LC and NPEndash result write_mem(kpa len data)
bull Message context ID must == 0bull Can read any valid physical address on the xscale
ndash data read_mem(kpa len)bull Message context ID must == 0bull Can read any valid physical address on the xscale
bull RMP to SCD NPE Onlyndash result write_sram(offset len data)
bull Offset is relative to the starting address of SRAM block allocated to slice SCD must verify the write operation is within bounds
ndash data read_sram(offset len)bull Read len bytes from SRAM block and return to client First verify offset and len with within
bounds for slice
17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
New Commands
bull NPE Onlybull retCode_t set_src_hwaddr(hwaddr_t)
ndash context must be 0
bull retCode_t set_iface_table(ipAddr_t[16])ndash context must be 0
10WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE GPE Infobull Common Message context ID = 0
uint32_t IP 4-Byte IP address (NBO)
uint16_t PORT UDP Port number (NBO)
uint16_t QID Slice relative queue id must be converted by SCD to the tuple of qid QM id and scheduler ID
bull SRM to SCDNPE Only
retCode_t set_encap_gpe(fpid gpeIP npeIP)
retCode_t unset_encap_gpe(fpid)
unset_encap_gpe() == set_encap_gpe(fpid00)
uint16_t fpid Fast path id globally unique
bull RMP to SCDNPE Only
retCode_t set_gpe_info(exPort ldPort exQID ldQID)
retCode_t unset_gpe_info()
Set context ID to the global fast path ID
11WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Queuesbull Common
ndash if cid == 0 then qid is absolute otherwise SCD must convert from fastpath relativeuint16_t miid Meta-Interface numberuint32_t qlen Length of packet queue in Bytesuint32_t threshold the maximum number of packets queued before droppinguint32_t bw Kbs SCD must convert to the associated weightuint8_t list_type Explicit list (0) or range specification (1)uint16_t qid_list[] List of queue identifiers to associate with meta-interface mi In C or C++ this would
be an array of integers 0 n If using a range then the list has the first and one past the last id of the range For example 4 8 is the same as 4 5 6 7
bull RMP to SCDNPE Onlyndash retCode_t bind_queue(u16 miid u8 list_type u16[] qid_list)ndash retCode_t unbind_queue(u8 list_type u16[] qid_list)LC and NPEndash bw_t actual_bw set_queue_params(u16 qid u32 threshold u32 bw)
bull If a parameter is -1 then do not update in tablendash u32 threshold u32 bw get_queue_params(u16 qid)ndash u32 pktCnt u32 byteCnt get_queue_len(u16 qid)
bull SRM to SCD LC (Ingress and Egress)ndash retCode_t bind_queue_sched(u16 qid u16 sid)
bull Assign qid to scheduler sidndash retCode_t unbind_queue_sched(u16 qid)
bull Deallocate queuersquos bw from its assigned scheduler and disassociate it from that scheduler
12WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Lookup Table (TCAM)bull Common LC cid = 0 NPE cid = fpid
ndash If cid == 0 then fid is absolute otherwise it is fastpath relative and must be converteduint32_t fid Filter ID If cid == 0 then absolute otherwise relativeuint16_t dbid Database ID NPE dbid=0 LC Ingress dbid=0 Egress dbid=1struct fltr uint8_t key[N] N-Byte key value defined by code option uint8_t mask[N] N-Byte mask for key lookups defined by code option uint8_t result[M] M-Byte result vector defined by code option
bull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)LC and NPEndash ret_t write_fltr(dbid fid key mask result)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash ret_t update_result(dbid fid result)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_fltr_bykey(dbid key)fltr get_fltr_byfid(dbid fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash result lookup_fltr(dbid key)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash retcode rem_fltr_bykey(dbid key)
retcode rem_fltr_byfid(dbid fid)ndash Returns fid if filter removed otherwise -1
13WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE Filter Interfacebull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)
LC and NPEsubKeyWrap_t type rxip rxport coptKey[14] subResult_t actions sindx daddr dport sport qidsubFltr_t subKeyWrap coptMask[14] subResult ndash scdCode_t write_npe_fltr(fid subFltr)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash scdCode_t update_npe_result(fid subResult)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_npe_fltr_bykey(subKeyWrap)fltr get_npe_fltr_byfid(fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash scdCode_t lookup_npe_fltr(subKeyWrap)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash scdCode_t rem_npe_fltr_bykey(subKeyWrap)
scdCode_t rem_npe_fltr_byfid(fid)ndash Returns 0 on success or an error
14WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
IPv4 TCAM Filter Formats (on NPE)
6 82flags
2 12
0100
2
TCP RSV proto00TCP
daddr saddr sport dport tcpproto
Defined by the IPv4 Code Option 112bits
32 32 16 16 16
vlan
11
if
T = 0 Normal LookupT = 1 substrate only lookup
T
1
RX port
Substrate defined
164
TX IP daddr TX dport TX sport rsv32 16 12 1516
QM3
D Drop packetL Local delivery
rsv113 1
L1
Drsv sindx Sch2
qid16
20-bit internal qid(SCD maps slicersquos miidto QM and Sch SCD Also
maps slicersquos qid toglobal qid value)
TX IP address and sport representsthe output meta-interface The
dport is provided by the slice (RMP maps miid to tx tunnel params
use dport provided by slice)
Result 128 bits
Represents input meta-interface
global statsindex
(SCD mapsslicersquos sindx
to global value)
Key Input miid IPv4 fltr daddr saddr sport dport tcpproto
Result Flags Drop GPE sindx Output miID QID
Slice parameters
15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Reading Statisticsbull Common
ndash if cid = 0 then sindx is absolute otherwiseSCD must convert fastpath relative index
uint8_t flagsuint32_t handle opaque reference to periodic eventuint32_t sindx stats index
bull RMP to SCD RLIGIU to SCDLC and NPEndash stats = read_stats(sindx flags)
bull struct stats uint32_t cnt tstampbull Uses flags L and W
ndash result = clear_stats(sindx)bull result success or error code
ndash handle create_periodic(sindx P cnt flags)bull uint32_t P periodic interval in millisecondsbull uint16_t cnt Number of samples to keep in a history bufferbull Uses all 3 flags
ndash retcode del_periodic(handle)ndash retcode set_callback(handle udp_port)
bull push model every (cntperiod) milliseconds send the last cnt samples to the clientrsquos UDP port udp_port
ndash stats = get_periodic(handle)bull pull model returns the last cnt samples
FlagsW 0 - Packts 1 - BytesL 0 - PreQ 1 - PostQT 0 - Push 1 - PullX Donrsquot Care
06 3 12457WLTXXXXX
16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Accessing Memorybull common
Message context id NPE equals fpid for LC must equal 0uint32_t offset Offset in from start of assigned SRAM block uint32_t len Number of bytes to readwriteuint8_t data[X] Data buffer with X Bytesuint32_t kpa Kernel physical address
bull SRM to SCD LC and NPEndash result write_mem(kpa len data)
bull Message context ID must == 0bull Can read any valid physical address on the xscale
ndash data read_mem(kpa len)bull Message context ID must == 0bull Can read any valid physical address on the xscale
bull RMP to SCD NPE Onlyndash result write_sram(offset len data)
bull Offset is relative to the starting address of SRAM block allocated to slice SCD must verify the write operation is within bounds
ndash data read_sram(offset len)bull Read len bytes from SRAM block and return to client First verify offset and len with within
bounds for slice
17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
New Commands
bull NPE Onlybull retCode_t set_src_hwaddr(hwaddr_t)
ndash context must be 0
bull retCode_t set_iface_table(ipAddr_t[16])ndash context must be 0
11WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Queuesbull Common
ndash if cid == 0 then qid is absolute otherwise SCD must convert from fastpath relativeuint16_t miid Meta-Interface numberuint32_t qlen Length of packet queue in Bytesuint32_t threshold the maximum number of packets queued before droppinguint32_t bw Kbs SCD must convert to the associated weightuint8_t list_type Explicit list (0) or range specification (1)uint16_t qid_list[] List of queue identifiers to associate with meta-interface mi In C or C++ this would
be an array of integers 0 n If using a range then the list has the first and one past the last id of the range For example 4 8 is the same as 4 5 6 7
bull RMP to SCDNPE Onlyndash retCode_t bind_queue(u16 miid u8 list_type u16[] qid_list)ndash retCode_t unbind_queue(u8 list_type u16[] qid_list)LC and NPEndash bw_t actual_bw set_queue_params(u16 qid u32 threshold u32 bw)
bull If a parameter is -1 then do not update in tablendash u32 threshold u32 bw get_queue_params(u16 qid)ndash u32 pktCnt u32 byteCnt get_queue_len(u16 qid)
bull SRM to SCD LC (Ingress and Egress)ndash retCode_t bind_queue_sched(u16 qid u16 sid)
bull Assign qid to scheduler sidndash retCode_t unbind_queue_sched(u16 qid)
bull Deallocate queuersquos bw from its assigned scheduler and disassociate it from that scheduler
12WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Lookup Table (TCAM)bull Common LC cid = 0 NPE cid = fpid
ndash If cid == 0 then fid is absolute otherwise it is fastpath relative and must be converteduint32_t fid Filter ID If cid == 0 then absolute otherwise relativeuint16_t dbid Database ID NPE dbid=0 LC Ingress dbid=0 Egress dbid=1struct fltr uint8_t key[N] N-Byte key value defined by code option uint8_t mask[N] N-Byte mask for key lookups defined by code option uint8_t result[M] M-Byte result vector defined by code option
bull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)LC and NPEndash ret_t write_fltr(dbid fid key mask result)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash ret_t update_result(dbid fid result)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_fltr_bykey(dbid key)fltr get_fltr_byfid(dbid fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash result lookup_fltr(dbid key)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash retcode rem_fltr_bykey(dbid key)
retcode rem_fltr_byfid(dbid fid)ndash Returns fid if filter removed otherwise -1
13WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE Filter Interfacebull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)
LC and NPEsubKeyWrap_t type rxip rxport coptKey[14] subResult_t actions sindx daddr dport sport qidsubFltr_t subKeyWrap coptMask[14] subResult ndash scdCode_t write_npe_fltr(fid subFltr)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash scdCode_t update_npe_result(fid subResult)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_npe_fltr_bykey(subKeyWrap)fltr get_npe_fltr_byfid(fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash scdCode_t lookup_npe_fltr(subKeyWrap)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash scdCode_t rem_npe_fltr_bykey(subKeyWrap)
scdCode_t rem_npe_fltr_byfid(fid)ndash Returns 0 on success or an error
14WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
IPv4 TCAM Filter Formats (on NPE)
6 82flags
2 12
0100
2
TCP RSV proto00TCP
daddr saddr sport dport tcpproto
Defined by the IPv4 Code Option 112bits
32 32 16 16 16
vlan
11
if
T = 0 Normal LookupT = 1 substrate only lookup
T
1
RX port
Substrate defined
164
TX IP daddr TX dport TX sport rsv32 16 12 1516
QM3
D Drop packetL Local delivery
rsv113 1
L1
Drsv sindx Sch2
qid16
20-bit internal qid(SCD maps slicersquos miidto QM and Sch SCD Also
maps slicersquos qid toglobal qid value)
TX IP address and sport representsthe output meta-interface The
dport is provided by the slice (RMP maps miid to tx tunnel params
use dport provided by slice)
Result 128 bits
Represents input meta-interface
global statsindex
(SCD mapsslicersquos sindx
to global value)
Key Input miid IPv4 fltr daddr saddr sport dport tcpproto
Result Flags Drop GPE sindx Output miID QID
Slice parameters
15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Reading Statisticsbull Common
ndash if cid = 0 then sindx is absolute otherwiseSCD must convert fastpath relative index
uint8_t flagsuint32_t handle opaque reference to periodic eventuint32_t sindx stats index
bull RMP to SCD RLIGIU to SCDLC and NPEndash stats = read_stats(sindx flags)
bull struct stats uint32_t cnt tstampbull Uses flags L and W
ndash result = clear_stats(sindx)bull result success or error code
ndash handle create_periodic(sindx P cnt flags)bull uint32_t P periodic interval in millisecondsbull uint16_t cnt Number of samples to keep in a history bufferbull Uses all 3 flags
ndash retcode del_periodic(handle)ndash retcode set_callback(handle udp_port)
bull push model every (cntperiod) milliseconds send the last cnt samples to the clientrsquos UDP port udp_port
ndash stats = get_periodic(handle)bull pull model returns the last cnt samples
FlagsW 0 - Packts 1 - BytesL 0 - PreQ 1 - PostQT 0 - Push 1 - PullX Donrsquot Care
06 3 12457WLTXXXXX
16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Accessing Memorybull common
Message context id NPE equals fpid for LC must equal 0uint32_t offset Offset in from start of assigned SRAM block uint32_t len Number of bytes to readwriteuint8_t data[X] Data buffer with X Bytesuint32_t kpa Kernel physical address
bull SRM to SCD LC and NPEndash result write_mem(kpa len data)
bull Message context ID must == 0bull Can read any valid physical address on the xscale
ndash data read_mem(kpa len)bull Message context ID must == 0bull Can read any valid physical address on the xscale
bull RMP to SCD NPE Onlyndash result write_sram(offset len data)
bull Offset is relative to the starting address of SRAM block allocated to slice SCD must verify the write operation is within bounds
ndash data read_sram(offset len)bull Read len bytes from SRAM block and return to client First verify offset and len with within
bounds for slice
17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
New Commands
bull NPE Onlybull retCode_t set_src_hwaddr(hwaddr_t)
ndash context must be 0
bull retCode_t set_iface_table(ipAddr_t[16])ndash context must be 0
12WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Lookup Table (TCAM)bull Common LC cid = 0 NPE cid = fpid
ndash If cid == 0 then fid is absolute otherwise it is fastpath relative and must be converteduint32_t fid Filter ID If cid == 0 then absolute otherwise relativeuint16_t dbid Database ID NPE dbid=0 LC Ingress dbid=0 Egress dbid=1struct fltr uint8_t key[N] N-Byte key value defined by code option uint8_t mask[N] N-Byte mask for key lookups defined by code option uint8_t result[M] M-Byte result vector defined by code option
bull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)LC and NPEndash ret_t write_fltr(dbid fid key mask result)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash ret_t update_result(dbid fid result)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_fltr_bykey(dbid key)fltr get_fltr_byfid(dbid fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash result lookup_fltr(dbid key)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash retcode rem_fltr_bykey(dbid key)
retcode rem_fltr_byfid(dbid fid)ndash Returns fid if filter removed otherwise -1
13WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE Filter Interfacebull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)
LC and NPEsubKeyWrap_t type rxip rxport coptKey[14] subResult_t actions sindx daddr dport sport qidsubFltr_t subKeyWrap coptMask[14] subResult ndash scdCode_t write_npe_fltr(fid subFltr)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash scdCode_t update_npe_result(fid subResult)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_npe_fltr_bykey(subKeyWrap)fltr get_npe_fltr_byfid(fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash scdCode_t lookup_npe_fltr(subKeyWrap)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash scdCode_t rem_npe_fltr_bykey(subKeyWrap)
scdCode_t rem_npe_fltr_byfid(fid)ndash Returns 0 on success or an error
14WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
IPv4 TCAM Filter Formats (on NPE)
6 82flags
2 12
0100
2
TCP RSV proto00TCP
daddr saddr sport dport tcpproto
Defined by the IPv4 Code Option 112bits
32 32 16 16 16
vlan
11
if
T = 0 Normal LookupT = 1 substrate only lookup
T
1
RX port
Substrate defined
164
TX IP daddr TX dport TX sport rsv32 16 12 1516
QM3
D Drop packetL Local delivery
rsv113 1
L1
Drsv sindx Sch2
qid16
20-bit internal qid(SCD maps slicersquos miidto QM and Sch SCD Also
maps slicersquos qid toglobal qid value)
TX IP address and sport representsthe output meta-interface The
dport is provided by the slice (RMP maps miid to tx tunnel params
use dport provided by slice)
Result 128 bits
Represents input meta-interface
global statsindex
(SCD mapsslicersquos sindx
to global value)
Key Input miid IPv4 fltr daddr saddr sport dport tcpproto
Result Flags Drop GPE sindx Output miID QID
Slice parameters
15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Reading Statisticsbull Common
ndash if cid = 0 then sindx is absolute otherwiseSCD must convert fastpath relative index
uint8_t flagsuint32_t handle opaque reference to periodic eventuint32_t sindx stats index
bull RMP to SCD RLIGIU to SCDLC and NPEndash stats = read_stats(sindx flags)
bull struct stats uint32_t cnt tstampbull Uses flags L and W
ndash result = clear_stats(sindx)bull result success or error code
ndash handle create_periodic(sindx P cnt flags)bull uint32_t P periodic interval in millisecondsbull uint16_t cnt Number of samples to keep in a history bufferbull Uses all 3 flags
ndash retcode del_periodic(handle)ndash retcode set_callback(handle udp_port)
bull push model every (cntperiod) milliseconds send the last cnt samples to the clientrsquos UDP port udp_port
ndash stats = get_periodic(handle)bull pull model returns the last cnt samples
FlagsW 0 - Packts 1 - BytesL 0 - PreQ 1 - PostQT 0 - Push 1 - PullX Donrsquot Care
06 3 12457WLTXXXXX
16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Accessing Memorybull common
Message context id NPE equals fpid for LC must equal 0uint32_t offset Offset in from start of assigned SRAM block uint32_t len Number of bytes to readwriteuint8_t data[X] Data buffer with X Bytesuint32_t kpa Kernel physical address
bull SRM to SCD LC and NPEndash result write_mem(kpa len data)
bull Message context ID must == 0bull Can read any valid physical address on the xscale
ndash data read_mem(kpa len)bull Message context ID must == 0bull Can read any valid physical address on the xscale
bull RMP to SCD NPE Onlyndash result write_sram(offset len data)
bull Offset is relative to the starting address of SRAM block allocated to slice SCD must verify the write operation is within bounds
ndash data read_sram(offset len)bull Read len bytes from SRAM block and return to client First verify offset and len with within
bounds for slice
17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
New Commands
bull NPE Onlybull retCode_t set_src_hwaddr(hwaddr_t)
ndash context must be 0
bull retCode_t set_iface_table(ipAddr_t[16])ndash context must be 0
13WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
NPE Filter Interfacebull SRM to SCD (ctx = 0) RMP to SCD (ctx = fpid)
LC and NPEsubKeyWrap_t type rxip rxport coptKey[14] subResult_t actions sindx daddr dport sport qidsubFltr_t subKeyWrap coptMask[14] subResult ndash scdCode_t write_npe_fltr(fid subFltr)
bull The SCD can lookup the database object with id = dbid to get the key and result byte widthsbull The RMP will prepend the VLAN tag and change the slice MI numbers to the correct values
ndash scdCode_t update_npe_result(fid subResult)bull If fid is a valid entry then updates result vector Otherwise no change to the database
ndash fltr get_npe_fltr_bykey(subKeyWrap)fltr get_npe_fltr_byfid(fid)
bull Returns a filter if keyfid matches a valid entry Otherwise 0 (does not return an error)ndash scdCode_t lookup_npe_fltr(subKeyWrap)
bull Returns a result if keyfid matches a valid entry otherwise 0 (does not return an error)ndash scdCode_t rem_npe_fltr_bykey(subKeyWrap)
scdCode_t rem_npe_fltr_byfid(fid)ndash Returns 0 on success or an error
14WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
IPv4 TCAM Filter Formats (on NPE)
6 82flags
2 12
0100
2
TCP RSV proto00TCP
daddr saddr sport dport tcpproto
Defined by the IPv4 Code Option 112bits
32 32 16 16 16
vlan
11
if
T = 0 Normal LookupT = 1 substrate only lookup
T
1
RX port
Substrate defined
164
TX IP daddr TX dport TX sport rsv32 16 12 1516
QM3
D Drop packetL Local delivery
rsv113 1
L1
Drsv sindx Sch2
qid16
20-bit internal qid(SCD maps slicersquos miidto QM and Sch SCD Also
maps slicersquos qid toglobal qid value)
TX IP address and sport representsthe output meta-interface The
dport is provided by the slice (RMP maps miid to tx tunnel params
use dport provided by slice)
Result 128 bits
Represents input meta-interface
global statsindex
(SCD mapsslicersquos sindx
to global value)
Key Input miid IPv4 fltr daddr saddr sport dport tcpproto
Result Flags Drop GPE sindx Output miID QID
Slice parameters
15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Reading Statisticsbull Common
ndash if cid = 0 then sindx is absolute otherwiseSCD must convert fastpath relative index
uint8_t flagsuint32_t handle opaque reference to periodic eventuint32_t sindx stats index
bull RMP to SCD RLIGIU to SCDLC and NPEndash stats = read_stats(sindx flags)
bull struct stats uint32_t cnt tstampbull Uses flags L and W
ndash result = clear_stats(sindx)bull result success or error code
ndash handle create_periodic(sindx P cnt flags)bull uint32_t P periodic interval in millisecondsbull uint16_t cnt Number of samples to keep in a history bufferbull Uses all 3 flags
ndash retcode del_periodic(handle)ndash retcode set_callback(handle udp_port)
bull push model every (cntperiod) milliseconds send the last cnt samples to the clientrsquos UDP port udp_port
ndash stats = get_periodic(handle)bull pull model returns the last cnt samples
FlagsW 0 - Packts 1 - BytesL 0 - PreQ 1 - PostQT 0 - Push 1 - PullX Donrsquot Care
06 3 12457WLTXXXXX
16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Accessing Memorybull common
Message context id NPE equals fpid for LC must equal 0uint32_t offset Offset in from start of assigned SRAM block uint32_t len Number of bytes to readwriteuint8_t data[X] Data buffer with X Bytesuint32_t kpa Kernel physical address
bull SRM to SCD LC and NPEndash result write_mem(kpa len data)
bull Message context ID must == 0bull Can read any valid physical address on the xscale
ndash data read_mem(kpa len)bull Message context ID must == 0bull Can read any valid physical address on the xscale
bull RMP to SCD NPE Onlyndash result write_sram(offset len data)
bull Offset is relative to the starting address of SRAM block allocated to slice SCD must verify the write operation is within bounds
ndash data read_sram(offset len)bull Read len bytes from SRAM block and return to client First verify offset and len with within
bounds for slice
17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
New Commands
bull NPE Onlybull retCode_t set_src_hwaddr(hwaddr_t)
ndash context must be 0
bull retCode_t set_iface_table(ipAddr_t[16])ndash context must be 0
14WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
IPv4 TCAM Filter Formats (on NPE)
6 82flags
2 12
0100
2
TCP RSV proto00TCP
daddr saddr sport dport tcpproto
Defined by the IPv4 Code Option 112bits
32 32 16 16 16
vlan
11
if
T = 0 Normal LookupT = 1 substrate only lookup
T
1
RX port
Substrate defined
164
TX IP daddr TX dport TX sport rsv32 16 12 1516
QM3
D Drop packetL Local delivery
rsv113 1
L1
Drsv sindx Sch2
qid16
20-bit internal qid(SCD maps slicersquos miidto QM and Sch SCD Also
maps slicersquos qid toglobal qid value)
TX IP address and sport representsthe output meta-interface The
dport is provided by the slice (RMP maps miid to tx tunnel params
use dport provided by slice)
Result 128 bits
Represents input meta-interface
global statsindex
(SCD mapsslicersquos sindx
to global value)
Key Input miid IPv4 fltr daddr saddr sport dport tcpproto
Result Flags Drop GPE sindx Output miID QID
Slice parameters
15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Reading Statisticsbull Common
ndash if cid = 0 then sindx is absolute otherwiseSCD must convert fastpath relative index
uint8_t flagsuint32_t handle opaque reference to periodic eventuint32_t sindx stats index
bull RMP to SCD RLIGIU to SCDLC and NPEndash stats = read_stats(sindx flags)
bull struct stats uint32_t cnt tstampbull Uses flags L and W
ndash result = clear_stats(sindx)bull result success or error code
ndash handle create_periodic(sindx P cnt flags)bull uint32_t P periodic interval in millisecondsbull uint16_t cnt Number of samples to keep in a history bufferbull Uses all 3 flags
ndash retcode del_periodic(handle)ndash retcode set_callback(handle udp_port)
bull push model every (cntperiod) milliseconds send the last cnt samples to the clientrsquos UDP port udp_port
ndash stats = get_periodic(handle)bull pull model returns the last cnt samples
FlagsW 0 - Packts 1 - BytesL 0 - PreQ 1 - PostQT 0 - Push 1 - PullX Donrsquot Care
06 3 12457WLTXXXXX
16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Accessing Memorybull common
Message context id NPE equals fpid for LC must equal 0uint32_t offset Offset in from start of assigned SRAM block uint32_t len Number of bytes to readwriteuint8_t data[X] Data buffer with X Bytesuint32_t kpa Kernel physical address
bull SRM to SCD LC and NPEndash result write_mem(kpa len data)
bull Message context ID must == 0bull Can read any valid physical address on the xscale
ndash data read_mem(kpa len)bull Message context ID must == 0bull Can read any valid physical address on the xscale
bull RMP to SCD NPE Onlyndash result write_sram(offset len data)
bull Offset is relative to the starting address of SRAM block allocated to slice SCD must verify the write operation is within bounds
ndash data read_sram(offset len)bull Read len bytes from SRAM block and return to client First verify offset and len with within
bounds for slice
17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
New Commands
bull NPE Onlybull retCode_t set_src_hwaddr(hwaddr_t)
ndash context must be 0
bull retCode_t set_iface_table(ipAddr_t[16])ndash context must be 0
15WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Reading Statisticsbull Common
ndash if cid = 0 then sindx is absolute otherwiseSCD must convert fastpath relative index
uint8_t flagsuint32_t handle opaque reference to periodic eventuint32_t sindx stats index
bull RMP to SCD RLIGIU to SCDLC and NPEndash stats = read_stats(sindx flags)
bull struct stats uint32_t cnt tstampbull Uses flags L and W
ndash result = clear_stats(sindx)bull result success or error code
ndash handle create_periodic(sindx P cnt flags)bull uint32_t P periodic interval in millisecondsbull uint16_t cnt Number of samples to keep in a history bufferbull Uses all 3 flags
ndash retcode del_periodic(handle)ndash retcode set_callback(handle udp_port)
bull push model every (cntperiod) milliseconds send the last cnt samples to the clientrsquos UDP port udp_port
ndash stats = get_periodic(handle)bull pull model returns the last cnt samples
FlagsW 0 - Packts 1 - BytesL 0 - PreQ 1 - PostQT 0 - Push 1 - PullX Donrsquot Care
06 3 12457WLTXXXXX
16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Accessing Memorybull common
Message context id NPE equals fpid for LC must equal 0uint32_t offset Offset in from start of assigned SRAM block uint32_t len Number of bytes to readwriteuint8_t data[X] Data buffer with X Bytesuint32_t kpa Kernel physical address
bull SRM to SCD LC and NPEndash result write_mem(kpa len data)
bull Message context ID must == 0bull Can read any valid physical address on the xscale
ndash data read_mem(kpa len)bull Message context ID must == 0bull Can read any valid physical address on the xscale
bull RMP to SCD NPE Onlyndash result write_sram(offset len data)
bull Offset is relative to the starting address of SRAM block allocated to slice SCD must verify the write operation is within bounds
ndash data read_sram(offset len)bull Read len bytes from SRAM block and return to client First verify offset and len with within
bounds for slice
17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
New Commands
bull NPE Onlybull retCode_t set_src_hwaddr(hwaddr_t)
ndash context must be 0
bull retCode_t set_iface_table(ipAddr_t[16])ndash context must be 0
16WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
Accessing Memorybull common
Message context id NPE equals fpid for LC must equal 0uint32_t offset Offset in from start of assigned SRAM block uint32_t len Number of bytes to readwriteuint8_t data[X] Data buffer with X Bytesuint32_t kpa Kernel physical address
bull SRM to SCD LC and NPEndash result write_mem(kpa len data)
bull Message context ID must == 0bull Can read any valid physical address on the xscale
ndash data read_mem(kpa len)bull Message context ID must == 0bull Can read any valid physical address on the xscale
bull RMP to SCD NPE Onlyndash result write_sram(offset len data)
bull Offset is relative to the starting address of SRAM block allocated to slice SCD must verify the write operation is within bounds
ndash data read_sram(offset len)bull Read len bytes from SRAM block and return to client First verify offset and len with within
bounds for slice
17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
New Commands
bull NPE Onlybull retCode_t set_src_hwaddr(hwaddr_t)
ndash context must be 0
bull retCode_t set_iface_table(ipAddr_t[16])ndash context must be 0
17WashingtonWASHINGTON UNIVERSITY IN ST LOUIS
Fred Kuhns - 042123
New Commands
bull NPE Onlybull retCode_t set_src_hwaddr(hwaddr_t)
ndash context must be 0
bull retCode_t set_iface_table(ipAddr_t[16])ndash context must be 0