Virtual Machine Queue Driver Development Sambhrama Mundkur Sr. Software Design Engineer Core Networking sammun@microsoft.com.
Post on 05-Jan-2016
213 Views
Preview:
Transcript
Virtual Machine Queue Driver Development
Sambhrama MundkurSr. Software Design EngineerCore Networkingsammun@microsoft.com
Agenda
• Overview of Virtual Machine Queue (VMQ)• VMQ capability advertisement• VM Queue allocation and deletion• Setting and clearing VMQ filters• Shared memory resources allocation• VMQ receive path• Embedded NIC switch • Diagnostics and Tools• Questions
VMQ Architecture
Parent PartitionVM1 VM2
Ethernet
VM BUS
TCP/IP TCP/IP
VM NIC 1 VM NIC 2
Network Virtual Service Provider
MiniportDriver
Switch/Routing unit
Q1 DefaultQueueQ2
RoutingVLAN filtering
Data Copy Port 1Port 2
NIC
Overview of Virtual Machine Queue
• Performance bottlenecks in receive processing in virtualized environment• Each incoming packet has to be examined to identify the
target virtual machine (VM)• Received data has to be copied from the parent partition’s
memory to the child partition’s memory address space• Lack of concurrency of interrupts and DPCs
Overview of Virtual Machine Queue (contd.)
• How does VMQ help?• Determine the target VM by MAC filtering in hardware• DMA received packets directly to the child partition’s memory
address space• Provide interrupt and DPC concurrency by indicating received
packets based on the target VM• Use switch capabilities of the NIC to provide VM to VM
loopback functionality
VMQ Capability Advertisement
• Receive filter capabilities advertised during miniport adapter initialization• Use standardized keywords to enable or disable VMQ
functionality
• Capabilities advertised• Enabled filters and queue types• Number of queues and filters• Supported queue properties• Lookahead-split support• NIC switch capabilities
Queue State Transitions
Queue Allocation
• Protocol driver issues an OID to allocate a queue and sets the parameters for the queue• Per-queue receive indication flag• Processor affinity• Queue name• Lookahead-split parameters
• NDIS generates a unique Queue ID and sets it before issuing the OID to the miniport driver
• Protocol driver can set filters after allocating the queue• Protocol driver issues an OID to complete the queue allocation
• Miniport driver can allocate shared memory and other resources
• Default queue (Queue ID 0) is always allocated
Queue Deletion
• Protocol driver clears all filters before freeing the queue• Protocol driver issues an OID to free the queue• Miniport driver
• Must immediately stop DMA to shared memory resources allocated for this queue
• Must generate status indication to indicate that the DMA has been stopped
• Waits for all outstanding NET_BUFFER_LIST (NBLs) for this queue to be returned
• Frees the allocated shared memory and hardware resources
• Default queue can never be freed
Updating Queue Parameters
• Protocol driver can issue an OID to update the queue parameters
• Queue parameters that can be updated are• Processor affinity• Queue name
VMQ Filter Operations
Setting and Clearing Filters
• Filters can be set and cleared any time after the queue has been allocated
• Protocol issues an OID to set the filter and the filter’s parameters• Queue ID• Array of header field test parameters. • For VMQ two header field tests can be set:
• Destination MAC address in the packet == specified MAC address
• VLAN ID in the packet == specified VLAN ID
• NDIS generates a unique filter ID and issues the OID to the miniport driver
Setting and Clearing Filters
• Miniport driver programs the hardware based on the filters• All header field tests of a particular filter must match to
indicate the packet to that queue• Multiple filters can be set on a queue• Packets must be indicated from the queue if any of the filters
pass
• Protocol driver can set filters on the default queue, too• Protocol driver issues an OID to clear a filter
• Provides the Filter ID of the filter
Shared Memory Resource Allocation
• Miniport driver calls the new DDI, NdisAllocateSharedMemory, to allocate shared memory resources for a queue• Initial allocation occurs when processing the queue allocation complete OID • Miniport driver can allocate shared memory as long as the queue has not been
freed
• Miniport driver specifies:• Queue ID• Length • Usage – Lookahead, post-Lookahead etc.
• NdisAllocateSharedMemory returns:• Virtual address of the allocated memory• Scatter/gather List• Shared memory handle (used in receive indications)• Allocation handle (used to free the memory)
Shared Memory Resources
• If lookahead-split is not enabled, specify that the memory will be used for receive buffer
• If lookahead-split has been enabled for the queue, allocate two shared memory buffers• Specify that usage is for the lookahead buffer• Specify that usage is for the post-lookahead (data) buffer
• Call NdisFreeSharedMemory to free the memory• Driver frees the resources when the queue is freed• Driver must wait for all NBLs using the shared memory to be
returned before calling this DDI
VMQ Packet Structures
VMQ Receive Path
• Indicate packets on a queue only if it passes all the filter field tests for a particular filter
• If per-queue receive indication flag is set: • NBLs from this queue must not be mixed with others• Must set ‘single queue indication’ flag in the Receive Flags
• Must set flag to indicate that the shared memory information is valid
• Out-of-band (OOB) information• Specify the Queue ID in packet OOB information• For VMQ, set the Filter ID in the OOB to zero
VMQ Receive Path
• Specify the SharedMemoryInfo in the NET_BUFFER• Set SharedMemoryHandle returned by NdisAllocateSharedMemory• Offset is the offset from the beginning of the shared memory buffer
• If lookahead split is enabled for the queue, each NET_BUFFER: • Must have two MDLs and corresponding SharedMemoryInfo structures• The post-lookahead buffer must have a backfill
• The protocol driver copies the contents of the lookahead buffer to the backfill
• Backfill must exist even if the packet is entirely in the lookahead buffer
• MDL Length and DataLength in the packet must be set as before
Queue Statistics
• NIC must maintain following per-queue statistics:• Number of transmitted and received packets• Number of transmitted and received bytes
• NIC can maintain per-packet type statistics• The statistics can be queried using the new statistics OID
• This is a mandatory Direct OID request• VMQ miniport driver must provide a Direct OID request
handler• Return statistics for a particular queue based on Queue ID• Default queue can be queried, too
Embedded NIC Switch
• Embedded switch on the NIC can assist in advanced features
• Advertise support during miniport adapter initialization• Loopback support
• Protocol driver issues an OID to enable these features• Loopback
• On transmit, Queue ID of the outgoing queue is set in packet OOB data
• NIC indicates the packet if it matches filters set on any other queue
• If no match is found, NIC transmits the packet on the wire
Diagnostics
• OIDs and WMI GUIDs • Enumerate the queues allocated on a miniport adapter• Query the parameters of a particular queue• Enumerate the filters allocated on a particular queue• Query the parameters of a particular filter• Handled by NDIS
• NDIS kernel debugger extensions• Performance counters exposed by NDIS
Call to Action
• Update drivers to NDIS 6.20 to take advantage of Windows 7 functionality
• Implement VMQ support in the miniport driver• Attend other related sessions in the DDC
Resources
• Windows 7 WDK will be available as of WinHEC• Windows Logo Program Web site:
http://www.microsoft.com/whdc/winlogo/default.mspx • NDIS 6 feedback alias:
ndis6fb@microsoft.com
Related Sessions
Session Day / Time
Network Power Management for Windows 7 Tues. 5:15-6:15
Windows Logo Program Tests for NDIS Mon. 11-12 andWed. 9:45-10:45
Discussion: Porting Drivers to NDIS 6.20 Wed. 8:30-9:30
Lab: How to Debug an NDIS 6.20 Driver Mon. 2:45-3:45 andWed. 9:45-10:45
Mobile Broadband Driver Development for Windows 7 Mon. 1:30-2:30
Virtual Machine Queue Architecture Review Tues. 2:45-3:45
Virtual Machine Queue Driver Development Tues. 4-5
Implementing Efficient RSS Capable Hardware and Drivers for Windows 7 Tues. 1:30-2:30
Developing TCP Chimney Drivers for Windows 7 Tues. 11-12
Driver Scalability Mon. 11-12 andTues. 11-12
Questions?
top related