Transcript
7/30/2019 RNIC Verbs Overview2
1/28
4/29/2001
RDMA enabled NIC
(RNIC) Verbs Overview
Renato Recio
7/30/2019 RNIC Verbs Overview2
2/28
4/29/200
RNIC Verbs
!The RDMA Protocol Verbs Specification describes the behavior ofRNIC hardware, firmware, and software as viewed by the host,"not the host software itself, and
"not the programming interface viewed by the host.
!The behavioral description is specified in the form of an RNIC
Interface (RI) and a set of RNIC Verbs:
"A RNIC Interface defines the semantics of the RDMA services that are providedby an RNIC that supports the RNIC Verb Specification. The RI can be
implemented through a combination of hardware, firmware, and software.
"A Verb is an operation which an RNIC Interface is expected to be able toperform.
7/30/2019 RNIC Verbs Overview2
3/28
4/29/200
First Principles
!Strove to minimize the number of options in the RNIC Verbs Spec.
!Strove to minimize semantic and interface delta from existing standards
(i.e. InfiniBand).
!RNIC Verbs Specification supports TCP transport."
Some effort was placed on provisioning for SCTP,but additional work would be needed for SCTP.
!Define a common RNIC Interface that can be used by O/Ss and
appliances to access RNIC functions.
!Consumer does not directly access queue elements.
"IB style queue model (vs VIA style queue model).
7/30/2019 RNIC Verbs Overview2
4/28
4/29/200
Emerging NIC ModelExample of service offered by
a netw ork stack off load NI C.
Host
!Background on RNIC Verb scope.
"NICs are in the process of incorporatinglayer 3 and layer 4 functions.
"The scope of the RNIC Verbs Spec isLayer 4 access to RDMA functions:
# Definition of the verbs (and theirassociated semantics) needed to accessRDMA Protocol Layer
functions.semantics.
# Except for connection management andteardown semantics, access to other
layers is not semantically defined by theRNIC verbs.NI C
IP
DDP
RDMA
MPA
TCP
Ethernet
L2Ethe
rnetAc
cessM
echa
nism
L3IPv4
AccessM
echa
nism
L3IPv6
AccessM
echa
nism
L3IPSecAc
cessM
echa
nism
L4TOE
AccessM
echa
nism
L4RDM
AA
ccessM
echa
nism
7/30/2019 RNIC Verbs Overview2
5/28
4/29/200
RNIC Model Overview
RNI C Driver/ Library
Verb consumerVerbs
!Verb consumer SW that uses RNI
to communicate to other nodes.!Communication is thru verbs, which:
"Manage connection state.
"Manage memory and queue access.
"Submit work to RNIC."Retrieve work and events from RNIC.
!RNIC Interface (RI) performs workon behalf of the consumer."Consists of software, firmware, and
hardware.
"Performs queue and memory mgt.
"Converts Work Requests (WRs) to Work
Queue Elements (WQEs)."Supports the standard RNIC layers
(RDMA, DDP, MPA, TCP, IP, andEthernet).
"Converts Completion Queue Elements(CQEs) to Work Completions (WCs).
"Generates asynchronous events.
RNIC
Data Engine Layer
QPContext(QPC)
RDMA/ DDP/ MPA/ TCP/ I P
RI
SQ Send QueueRQ Receive QueueSRQ Shared RQ
CQRQSQ AE
MemoryTranslation
andProtection
Table
(TPT)
SRQ
QP Queue PairQP = SQ + RQCQ = Completion Queue
7/30/2019 RNIC Verbs Overview2
6/28
4/29/200
Send/Receive Work Submission Model
RNI C Driver/ Library
Verb consumer
Verbs
"
Consumer work submission model:# For all outbound messages, Consumer uses SQ.
# For inbound Send message types, Consumer uses:
$ RQ, or
$ SRQ (if QP was created with SRQ association)."Work is submitted in the form of a Work Request,
which includes:
# Scatter/gather list of local data segments,
each represented by Local: STag, TO, and Length# Other modifiers (see specification).
"WR types:
# Send (four types), RDMA Write, RDMA Read, Bind
MW, Fast-Register MR, and Invalidate STag"RI converts work requests into WQEs and
processes the WQEs.
# RI returns control to consumer immediately after W
has been converted to WQE and submitted to WQ.# After control is returned to consumer, RI can not
modify WR.RNI C
Data Engine LayerQPC
MemoryTPT
RDMA/ DDP/ MPA/ TCP/ I P
CQSRQSQ
WQE WQE
WR WR
RQ
WQE
WR
# One of the two,but not both.
7/30/2019 RNIC Verbs Overview2
7/284/29/200
Completion and Event Model!RI completion processing model:
"Consumer sets Completion Event Handler."WQE processing model:
# For SQ,
$ RNIC performs operation.
$ When operation completes, if WQE wassignaled (or completed in error), a CQE isgenerated.
# For RQ,
$ If no SRQ is associated with QP, when
operation completes, WQE is convertedinto CQE.
$ If SRQ is associated with QP, RNICbehaves as if WQE is pulled from SRQ andmoved to RQ, and then moved from RQ to
CQ when the message completes."Consumer polls CQ to retrieve CQEs as WC.
!RI event processing model, includes:"Consumer sets Asynchronous Event
Handler."Asynchronous events are sent to consumer
through Async Event Handler.
RNI C Driver/ Library
Verb consumer
Verbs
RNI C
Data Engine LayerQPC
MemoryTPT
RDMA/ DDP/ MPA/ TCP/ I P
CQSRQSQ
WQE WQE
WR WR
RQ
WQE
WR
# One of the two,but not both.
Async
Events
CQE
WC
7/30/2019 RNIC Verbs Overview2
8/284/29/200
Motivation for Shared RQSend/Receive skew between connections may result in
inefficient memory or wire usage.
RNICRQ
WQE
RQ
WQE
RQ
WQE
RQ
WQE
WQE WQE WQE WQE
WQE WQE WQE WQE
WQE WQE WQE WQE
WRWRWRWRWR
WRWRWRWRWR
WRWRWRWRWR
WRWRWRWRWR
!Under traditional RQ models (e.g. VIA, IB), for eachconnection the Consumer posts the number of receiveWRs necessary to handle incoming receives on that
connection.! If the Data Sink Consumer cannot predict the incoming
rate on a given connection, the Data Sink Consumermust either:
A. Post the a sufficient number of RQ WQEs to handle thehighest incoming ratefor each connection.
Post RQ WQE Rate >= Incoming rate
Where Incoming rate may equal link B/W.
B. Let message flow control cause the Data Source to backoff until Data Sink posts more WRs.
Either approach is inefficient:
A. Holding WQEs in RQs that are relatively inactive wastesthe memory space associated with the SGE of the WQEs.
B. Data Sink Consumer may be unaware that the RQ isstarving.
WRWRWRWRMRs
WRWRWRWRMRs
WRWRWRWRMRs
WRWRWRWRMRs
WRSGLs
WRSGLs
WRSGLs
WRSGLs
7/30/2019 RNIC Verbs Overview2
9/284/29/200
Conceptual RNIC Shared RQ WR ModelShared RQ enables more efficient use of memory or wire.
RNIC
!Some Consumers (e.g. storage subsystems)manage a fixed-size pool of buffers amongstseveral different transport connections toreduce buffer space requirements." The SRQ model, enables these to post receive
WRs to a queue that is shared by a (Consumerdefined) set of connections.
!Under the SRQ model the Consumer postsreceive WRs to the SRQ and, as incomingsegments arrive on a given QP that isassociated with the SRQ, the SRQ WQEsare transferred from the SRQ to the RQ ofthe given QP." The maximum incoming rate is bounded by the
link rate:
# Post SRQ WQE Rate =< Link B/W.SRQ RQ
WQE
RQ RQ
WQE
WQEWQE
WQEWQE
WQEWQE
WRWRWRWRWR
WRWRWRWRMRs
WR
SGLs
RQ WQEsare returnedto Consumerusing the CQ.
7/30/2019 RNIC Verbs Overview2
10/284/29/20010
!RNIC SRQ processing semantics:
"On a given RQ, message WCs are returned inthe order the associated messages were sent.
"For a Send type message that arrives in orderon a given RDMAP Stream:# Next available WQE is pulled from SRQ,
# RNIC must behaves as if WQE is moved to RQ
$ Whether it actually does or not isimplementers choice.
"For a message that arrives out of order, two
options are allowed:# Sequential ordering:
$ RNIC dequeues one WQE for the incomingmessage plus one WQE each message withan MSN lower than the out-of-order message
that doesnt already have a WQE.# Arrival ordering:
$ RNIC dequeues one WQE. WQEs requiredfor messages with lower MSNs, will bedequeued when those messages arrive.
Query RNIC has output modifier stating whichof the above options is supported by the RNIC.
SRQ RQ RQ
SRQ RQ RQ
WQE
WQE
WQE
WQE
WQE
E
WQE
WQE
WQE
WQE
1
WQE 2
WQE
3
WQE
4
WQE
2
WQE
Number represents arrival order.
Color represents when WQE is dequeued.
Order, from bottom, in RQ represents MSN order.
WQE
WQE
1
3
4
Shared RQ Processing ModelSequential
Ordering.
WQ
RNIC
ArrivalOrdering.
RNIC
Work Request Types and
7/30/2019 RNIC Verbs Overview2
11/284/29/20011
Work Request Types andWork Request Posting Mechanisms
If QP is not associated with SRQ,
WR posted to RQ.
If QP associated with SRQ,WR posted SRQ.
Posted as single WR, or
List of WRs
Receive QueueSend Queue
Receive
WR Posting Attributes
WRs types
Bind
Fast Register MR
Invalidate Local STag
Memory
RDMA Write
RDMA Read
RDMA Read with Invalidate
RDMA
Send
Send with SE
Send with Invalidate
Send with SE and Invalidate
Send/Receive
7/30/2019 RNIC Verbs Overview2
12/284/29/20012
Summary Error and Event Classes!RNIC Verb Errors and Events:
RQ
SQ
SQ
RQWQE WQE
(2) (3)WQE WQE
(B)(C)
CQ CQCQE CQE
RQ Errors,by where detected, and how it is returned:
A. Local Immediate, returned before WQEposted.
Returned through CQE on associated CQ:
B. Local Completion, pre-WQE processingC. Local Completion, post WQE processing
D. Remote Completion
SQ Errors,by where detected, and how it is returned:
1) Local Immediate, returned before WQEposted.
Returned through CQE on associated CQ:
2) Local Completion, pre-WQE processing3) Local Completion, post WQE processing
4) Remote Completion
(D)(1) (A)V
ER
bs
VER
bs
WRs WRs
(4)RI RIWCs WCs
AEQE AEQEAEQ AEQAEs AEs
RNI C - Requestor RNI C - Responder
Asynchronous Error and Event:- Locally detected SQ, RQ, or RNIC errors or event that cannot be returned through CQ.
- Returned instead through RNICs Asynchronous Event Queue.
U S M M t M d l
7/30/2019 RNIC Verbs Overview2
13/284/29/20013
User Space Memory Management Model
User
Kernel
!Memory Windows"Windows enable flexible & efficient
dynamic RDMA access control tounderlying Memory Regions"Consumer uses Send Queue to
bind a pre-allocated Window to aspecified portion of an existing
Region."QP access to Windows managedthrough QP ID.
Page 1
Page 3
Region
RegisteredAddress
Space
Consumer Managed
PrivilegedConsumerManaged
!Memory Regions
"Base TO to physical mapping of a(portion of) consumer processaddress space
"RNIC Driver is responsible forpinning and translation.
"Explicit registration by consumerwith the RNIC Driver through RIregistration mechanisms.
"QP access to Regions managedthrough Protection Domains
"QP consumers use Base TOaddressing, RNIC performs Base TOto Physical mapping
User Space Consumer
RNICLibrary
RNIC
Driver
RNIC
Window
Page 0
Page 2
Wh A i MW i h QP ID?
7/30/2019 RNIC Verbs Overview2
14/284/29/20014
Why Associate MW with QP ID?
RNIC Driver and Library
Programming Interface
RNIC
Transport/Network/Link
RNIC Driver and Library
Programming Interface
RNIC
QPC MemoryTPT
Transport/Network/Link
CQRQSQ AE
QPC MemoryTPT
CQRQSQ AECQRQSQ
XX
!Considered associating MWs with PDs, but encountered problems."When a single server process communicates with multiple processes at the clients, two
options are available from protecting STag access control:# Option 1: Use the same PD on all QPs.
$ Problem: When multiple clients, or when multiple processes running on the same client, connectto the server process, process A can access the STag exposed to process B.
# Option 2: Use different PDs on each QP.
$ Problem: If the same region is to be exposed to multiple clients/processes, the server processwould need to create multiple windows, one for each client. Each MW would consumes oneMemory Translation and Protection Table entry.
"Net:# PD based MW access control is not fine grained enough to support 1:N server:client models.
!RNIC MW bind and access semantics:"At bind time QP ID is associated with the MW (vs the PD).
"At access time, QP ID of MW must match QP ID of QP.
Server ProcessProcess A Process B
RNIC Driver and Library
Programming Interface
RNIC
Transport/Network/Link
QPC MemoryTPT
CQRQSQ AE
Process B
Privileged Space Memory Mgt Model
7/30/2019 RNIC Verbs Overview2
15/284/29/20015
Privileged Space Memory Mgt Model(Superset of user space model)
!Memory Windows"Consumer uses Send Queue to
bind a pre-allocated Window to aspecified portion of an existingRegion.
"QP access to Windows managedthrough QP ID.
Page 1
Page 3
Region
RegisteredAddress
Space
Consumer Managed
PrivilegedConsumerManaged
!Memory Regions"Driver performs Base TO to physical
address mapping"Driver is responsible for pinning andtranslation."Consumer registers mappings with
RNIC through:
"RI registration mechanisms"Post SQ Fast-Register mechanism"STag 0 is a special STag thatrequires no registration.
"QP access to Regions managed throughProtection Domains
"Consumer uses Base TO, RNICperforms Base TO to physical mapping
Privileged Consumer
RNICDriver
RNIC
Window
Page 0
Page 2
Terms Associated with Buffers
7/30/2019 RNIC Verbs Overview2
16/284/29/20016
Terms Associated with Buffers
!Base Tagged Offset (Base TO) - The offsetassigned to the first byte of a Memory Region.
RNICs support two addressing types:
"Virtual Address Based Tagged Offset (VA Base TO) -The Base TO of an MR or MW that starts at a non-zeroTO (and the address is associated with an address of thelocal hosts processor).
"Zero Based Tagged Offset (Zero Base TO) - A Regionor Window that starts with a TO of zero.
# Zero Based TO Created to meet the needs of Consumers
that dont want to exchange addresses in theirrequest/respond handshakes.
$ For example, iSCSI Extensions for RDMA dontexchange a TO in the iSCSI Command or iSCSIResponse. For iSER, the TO is only used in the RDMA
Write and Read operations (more will be said later oniSER).
Zero based TO
First addresshas a zerovalue.
To offset intoaddress space:
TO = offset
VA based TO
First addresshas a non-zerovalue.To offset intoaddress space:
TO = Base VA + offset
7/30/2019 RNIC Verbs Overview2
17/28
4/29/20017
Terms Associated with Buffers Cont.!
Physical Buffer - A set of physically contiguous memory locationsthat can be directly accessed by the RNIC through Physical Addresses."Physical Buffer List (PBL) - A list of Physical Buffers.
# The input modifier to register Non-Shared Memory Regions.
$ Each Physical Buffer List Entry references a physical buffer."RNICs support two physical buffer list types:
# Page List - Created to support consumers that have page aligned physical buffers.
# Block List - Created to support consumers (e.g. storage) that have control information
before and/or after each block on the list and the control information is not transferred.
!First Byte Offset (FBO) - The offset into the first Physical Buffer of a
Memory Region.
!Scatter/Gather List (SGL)"The input modifier to Post Send and Post Receive Verbs which indicate the data
buffer location information.
# Each Scatter/Gather List Entry references an existing Memory Region.
Physical Buffer List: Page List
7/30/2019 RNIC Verbs Overview2
18/28
4/29/20018
Physical Buffer List: Page List!Page list attributes:
"Page size:# Size is an integral power of 2
# All pages have the same size
"Data boundaries:
# Data can start at an offset into the firstpage (First Byte Offset)
# Data can end on a non-page boundary(i.e. last page may be partially filled)
"Page (starting) addresses:# Must be integral number of page size.
# Doesnt have to be contiguous.
!Page list modifiers:"Page size
"FBO
"Length
# From FBO to last byte of data in the lastpage.
"Address list
Starting TO
- 0 if zero based TO- VA if VA based TO
Page 1
Page 2
Page N-1
Page N- Pink = Data- Blue = Reserved
Page Size =
x**2
Legend:
Physical Buffer List: Block List
7/30/2019 RNIC Verbs Overview2
19/28
4/29/20019
Physical Buffer List: Block List!Block list attributes:
"Block size:# Arbitrary (depends on sizes supported by
RNIC)
# All pages have the same size
"Data boundaries:# Data can start at an offset into the first
block (First Byte Offset)
# Data can end at an offset into the last block
(i.e. last block may be partially filled)"Block (starting) addresses:
# Arbitrary.
!Page list modifiers:"Block size
"FBO
"Length
#From FBO to last byte of data in the lastblock.
"Address list
Starting TO
- 0 if zero based TO- VA if VA based TO
Block
1
Block
2
Block
N-1
BlockN- Pink = Data- Blue = Reserved
Block Size =
Arbitrary
Legend:
7/30/2019 RNIC Verbs Overview2
20/28
4/29/2000
Memory Management
!Memory Region (MR) - An area of memory that the Consumer wantsthe RNIC to be able to (locally or locally and remotely) access directlyin a logically contiguous fashion."A Memory Region is identified by an STag, a Base TO, and a length.
"A Memory Region is associated with a Physical Buffer List through the STag.
!Two types of MRs:
"Non-Shared Memory Region - A Memory Region that solely owns the PhysicalBuffer List associated with the Memory Region. Specifically, the PBL is notshared and has never been shared with another Memory Region.
"Shared Memory Region - An MR that currently shares, or at one time shared, thePhysical Buffer List associated with the Memory Region. Specifically, the PBL iscurrently shared or was previously shared with another Memory Region.
M d W k R R l d T
7/30/2019 RNIC Verbs Overview2
21/28
4/29/2001
Memory and Work Request Related Terms!
Problem addressed by Fast-Register:"For privileged consumer, compared to I/O transaction models over PCI, the
synchronous RI-Registration mechanism is inefficient:
# Hosts wastes CPU resources to perform registration
$
RNIC driver issues PIO Writes to create the MR.# Latency incurred waiting for RNIC to confirm the MR has been registered.
$ Specially if the RNIC is attached through a PCI bus.
!More efficient mechanism is needed"Allocate STag - A mechanism used to allocate memory registration resources for
use in subsequent Fast-Registrations or RI-Reregistrations.
# Resources that get allocated are:
$ Protection Domain, PBL, and Remote Access Enablement
"Fast-Register - A mechanism used to register a Memory Region through the SendQueue.
# The Fast Register Work Request uses a Memory Region STag, that is in the Invalidstate, to register the PBL and access controls passed in through Post Send Queue verb.
F R i C U O i
7/30/2019 RNIC Verbs Overview2
22/28
4/29/2002
Fast-Register Consumer Usage Options
!At initialization (and possibly, at other intervals), consumer invokesAllocates STag with several size classes."For example:
# Large Size (65536 PhyPages)
# Medium-Large (1024 PhyPages)
# Medium (64 PhyPages)
# Medium-Small (16 PhyPages)
# Small (1 Physical Page)
!At run-time, consumer chooses a previously allocated STag that has asize class which is higher or equal to the size of the Memory Region
that is to be registered.
However, Verbs spec is silent on the above (it is up to the consumer to
decide how to use the allocation).
Animated Introductory Slide Depicting
7/30/2019 RNIC Verbs Overview2
23/28
4/29/2003
Animated Introductory Slide Depicting10,000 Foot Overview of Memory Region
Fast-Registration and Local Access
RNI C Driver
and Library
Verbconsumer
Verbs
Data Engine Layer
MemoryTPT
1. Allocate (PBLcapacit y, PD, STag Key)
2. Program RNI C(STag, PD, PBL cap,
Remot e Access Right )
3. Return ( STag)
SQ or RQ
4. Fast -Reg WR(TO, lengt h, STag, et c..)
WR 1
QPC
5. Reg MR(STag, TO, et c)
WQE 1
7. Access Data(Physical
Addresses)
WQE 2
6. Submit WR( that uses MR)
WR 2
MPA/ TCP/ I P
RNIC
PostSend Verb WR Register MR
7/30/2019 RNIC Verbs Overview2
24/28
4/29/2004
PostSend Verb WR Register MR
!Privileged mode operation.
!WR Type = Register MR"Function:
# Registers a Memory Region using an existing Non-Shared MR STag.
# If total # of pages
7/30/2019 RNIC Verbs Overview2
25/28
4/29/2005
Invalidate STag Rationale
!Problem with RI de-allocate semantics:
"Some consumers (esp. privileged mode storage) use MRs dynamically.# For these consumers the MR is created, used once, and never used again.
"RI MR de/re-registration is a synchronous, inefficient operation that incurs:
# Host CPU resources (e.g. memory stores and PIO Writes), and
latency associated with waiting for RNIC to confirm the MR has been deregistered.
!RNIC Local and Remote Invalidate semantics:"Assuming all STag checks pass, an incoming Send with Invalidate or Send with SE
and Invalidate, invalidates access to an MR or MW."An Invalidate STag Post SQ WR invalidates a local MR or MW through an
asynchronous process.
RNIC Driver and LibraryProgramming Interface
RNIC
QPC MemoryTPT
Transport/Network/Link
CQRQSQ AE
RNIC Driver and Library
Programming Interface
RNIC
Transport/Network/Link
QPC MemoryTPT
CQRQSQ AE
ProcessMiddleware Application
PostSend Verb WR Invalidate STag
7/30/2019 RNIC Verbs Overview2
26/28
4/29/2006
PostSend Verb WR Invalidate STag
RNI C Driverand
Library
Verb consumer
Verbs
RNI C
Data EngineLayer
SQ
RDMA/ DDP
WQEs
!
WR Type = Invalidate STag"Function:
# Invalidates a Local Memory Region.
# For an STag that is associated with an MR:
$ If QP PD matches PD associated with the STag, then the RNICdisables access to the MR referenced by the STag.
# For an STag that is associated with an MW:
$ If QP ID matches the QP ID associated with the STag, then theRNIC disables access to the MW referenced by the STag.
"Key Input Modifiers:
# Invalidate STag is the WR type
# STag for the MR or MW
"
Key Completion Results returned through CQE:# Invalid STag
# Invalid PD
# MR still has MWs bound to it
Inval idateMR WR
Overview of MR/MW Attributes
7/30/2019 RNIC Verbs Overview2
27/28
4/29/2007
Overview of MR/MW Attributes
VA Based
Zero Based
VA Based
Zero Based
STag zero
AddressingTypes
Page
Block
PBL Types
WindowRegion
STagTagged Offset
Length
STagTagged Offset
Length
Referenced by
ByteByteAccessgranularity
PD on Bind
QP ID on AccessAccess Rights
Base & bounds
PD
Access Rights
Base & bounds
Access control
checks
Attribute
YesVA basedVA basedYesZero basedVA based
NoVA basedZero based
NoZero basedZero based
ValidMemoryWindowTO base
UnderlyingMemory
Region TO
base
Summary
7/30/2019 RNIC Verbs Overview2
28/28
4/29/2008
Summary
!The RNIC Verbs Specification"Provides a rich set of semantics that meet the needs of several
application environments:#
General networking,# Storage networking, and
# Cluster networking.
"Enables integration with existing RDMA APIs/KPIs,
and provides extensions to those APIs/KPIs that improveperformance.
"Provides NIC Vendors with implementation flexibility, for:# Queue management and memory management structures,
as well as, integration with other NIC layer 3 and 4 offload functions.
top related