DirectPlay ® 8 In Depth Michael Narayan Development Lead DirectPlay Core Microsoft ® Corporation.

DirectPlayDirectPlay® ® 8 In Depth 8 In Depth

Michael NarayanMichael NarayanDevelopment LeadDevelopment LeadDirectPlay CoreDirectPlay Core

MicrosoftMicrosoft®® Corporation Corporation

Overview

DirectPlay8 Architecture API Usage Expected Behavior Common Problems DirectPlay8 Protocol

ApplicationApplication

NetworkNetwork

DirectPlay8DirectPlay8

DirectPlay8 ArchitectureDirectPlay8 ArchitectureOverviewOverview

API callsAPI calls CallbacksCallbacks

ApplicationApplication

Network(s)Network(s)

CoreCore

DirectPlay8 ArchitectureDirectPlay8 ArchitectureComponentsComponents

Service Provider(s)Service Provider(s)

LobbyLobby

DPNSVRDPNSVRProtocolProtocol

Lobby ClientLobby ClientLobby ClientLobby Client

API UsageAPI Usage

Initialize()

Must call this before using DirectPlay Message handler function Context value is unique for each

interface Cannot change message handler or

context value

Close()

Matched call for Initialize() Can call Initialize() after Close()

Release() will call Close() Synchronous

Callbacks may occur Outstanding operations will complete

Cannot call Close() from message handler function

EnumServiceProviders()

Enumerate service providers and adapters for a given service provider

Determine which SP’s are available DirectPlay8 will attempt to load the SP

Determine which SP’s are installed Use DPNENUMSERVICEPROVIDERS_ALL

Call twice (size and data)

Host()

May specify several device addresses May specify an address with only an SP Using DPNHOST_OKTOQUERYFORADDRESSING

flag may cause a dialog box If no port is specified, DirectPlay will

pick one between 2302-2400 DPNSVR listens on 6073

Host() (cont’d)

Synchronous All listens will start before returning May receive enum queries before returning Will cause CREATE_PLAYER notification

for host player to be posted to message handler

EnumHosts()

Synchronous or Asynchronous Multiple concurrent enum’s allowed

Use context values to differentiate

Broadcast or target a known host Uses DPNSVR port (6073) if none

specified Uses all adapters if none specified DPNENUMHOSTS_OKTOQUERYFORADDRESSING

flag may cause a dialog box

EnumHosts() (cont’d)

Send application data to server Entire enum packet must fit within a

frame (about 1300 bytes on IP) Use GetSPCaps() to determine

maximum payload size

typedef struct _DPN_SP_CAPS {…

DWORD dwDefaultEnumCount;DWORD dwDefaultEnumRetryInterval;DWORD dwDefaultEnumTimeout;DWORD dwMaxEnumPayloadSize;

…} DPN_SP_CAPS, *PDPN_SP_CAPS;

EnumHosts() (cont’d)

Asynchronous completion/termination Time out Cancel with CancelAsyncOperation() Cancel by returning non-DPN_OK value

from message handler (DPN_MSGID_ENUM_HOSTS_RESPONSE)

Successfully connecting to a host Calling Close()

EnumHosts() (cont’d)

Synchronous completion/termination Time out Cancel by returning non-DPN_OK value

from message handler (DPN_MSGID_ENUM_HOSTS_RESPONSE)

Successfully connecting to a host Calling Close()

Connect()

Synchronous or asynchronous Only host address is required

Device address will be extracted from host address if not supplied

Host address may be known or discovered via EnumHosts()

All adapters used if none specified DPNCONNECT_OKTOQUERYFORADDRESSING

may cause dialog box

Connect() (cont’d)

Use host and device addresses from DPN_MSGID_ENUM_HOSTS_RESPONSE notification

typedef struct _DPNMSG_ENUM_HOSTS_RESPONSE{

DWORD dwSize;IDirectPlay8Address *pAddressSender;IDirectPlay8Address *pAddressDevice;const DPN_APPLICATION_DESC *pApplicationDescription;PVOID pvResponseData;

DWORD dwResponseDataSize;PVOID pvUserContext;

DWORD dwRoundTripLatencyMS;} DPNMSG_ENUM_HOSTS_RESPONSE;

Connect() (cont’d)

Able to pass application data to/from host

First to succeed on all adapters Others are cancelled

Successful connect will: Cancel all outstanding enum’s Generate DPN_MSGID_CONNECT_COMPLETE

notification

Connect() (cont’d)

IDirectPlay8Peer::Connect() DPN_MSGID_CREATE_PLAYER

notification always generated Use pvPlayerContext to preset local

player’s context value Completes when player is fully in the

session

Connect() (cont’d)

IDirectPlay8Client::Connect() No DPN_MSGID_CREATE_PLAYER

notification generated Completes when player is connected to the

server

Send() / SendTo()

IDirectPlay8Client::Send() IDirectPlay8Server::SendTo() IDirectPlay8Peer::SendTo() SendTo() requires DPNID of target

Clients only have knowledge of server and can’t target other players

Send() / SendTo() (cont’d)

Synchronous or asynchronous Best practice is asynchronous

May specify: Guaranteed / non-guaranteed Sequential / non-sequential Normal / high / low priority Completion / no completion (async case) Default: non-guaranteed, sequential,

normal priority with completion

Send() / SendTo() (cont’d)

If DPNSEND_NOCOPY is specified: DirectPlay owns the send buffer until the

send completes Application should not change buffer

contents (or free the buffer) Don’t allocate the send buffer on the stack!

Best practice – saves memcpy() call

Send() / SendTo() (cont’d)

If DPNSEND_COMPLETEONPROCESS is specified: Completion will not be posted until

recipient’s message handler is invoked Does not guarantee processing!

Send() / SendTo() (cont’d)

Sends complete when: Get onto the wire (non-guaranteed) ACK’d by receiver (guaranteed) Time out Get processed by recipient’s message

handler (DPNSEND_COMPLETEONPROCESS)

All asynchronous sends complete unless DPNSEND_NOCOMPLETE is specified

Use completion to free send buffers if DPNSEND_NOCOPY is specified

Expected BehaviorExpected Behavior

Expected Behavior

Hosting a session Enumerating sessions Connecting to a session Sending data Receiving data Dropped connections Host migration

Hosting a session

DirectPlay will: Launch DPNSVR (if required) Listen for enumeration queries Listen for incoming connections Post DPN_MSGID_CREATE_PLAYER

notification for the local player to message handler

Update session info and return

Hosting (cont’d)

DPNSVR: Listens on UDP port 6073 for enum queries Forwards enum queries to the appropriate

application (port) based on application GUID

Host sends enum responses back to client directly (not through DPNSVR)

Only one instance of DPNSVR per system Self-terminates after 10 minutes of

inactivity Force termination with: dpnsvr /kill

Hosting (cont’d)

Enumeration queries: Cause DPN_MSGID_ENUM_HOSTS_QUERY to

be posted to message handler

typedef struct _DPNMSG_ENUM_HOSTS_QUERY{

DWORD dwSize;IDirectPlay8Address *pAddressSender;IDirectPlay8Address *pAddressDevice;PVOID pvReceivedData;DWORD dwReceivedDataSize;DWORD

dwMaxResponseDataSize;PVOID pvResponseData;DWORD dwResponseDataSize;PVOID pvResponseContext;

} DPNMSG_ENUM_HOSTS_QUERY;

Hosting (cont’d)

Enumeration queries: Pre-filtered to ensure matching application

GUID, instance GUID (if specified) Host will only generate responses if

message handler returns DPN_OK Host is informed of sender’s address Host is informed of device address enum

query arrived on

Hosting (cont’d)

Enumeration queries: Client may send enum data to host via

pvReceivedData, dwReceivedDataSize Host may send reply via pvResponseData,

dwResponseDataSize and pvResponseContext

Response size must be limited to dwMaxResponseDataSize

Reply will generate DPN_MSGID_RETURN_BUFFER with matching context

Hosting (cont’d)

Incoming connections: Cause DPN_MSGID_INDICATE_CONNECT to be

posted to message handler

typedef struct _DPNMSG_INDICATE_CONNECT{

DWORD dwSize;PVOID pvUserConnectData;DWORD dwUserConnectDataSize;PVOID pvReplyData;DWORD dwReplyDataSize;PVOID pvReplyContext;PVOID pvPlayerContext;IDirectPlay8Address *pAddressPlayer;IDirectPlay8Address *pAddressDevice;

} DPNMSG_INDICATE_CONNECT;

Hosting (cont’d)

Incoming connections: Pre-filtered to ensure matching application

GUID, instance GUID, password, availability

Host will only accept connections if message handler returns DPN_OK

Accepted connections will cause DPN_MSGID_CREATE_PLAYER or DPN_MSGID_INDICATED_CONNECT_ABORTED

Pre-allocate resources Pre-set pvPlayerContext

Hosting (cont’d)

Incoming connections: Client may send connect data to host via

pvUserConnectData, dwUserConnectDataSize

Host may send reply data to client via pvReplyData, dwReplyDataSize, pvReplyContext

Reply will always be sent, even if the connection is denied by the host

Reply will generate DPN_MSGID_RETURN_BUFFER with matching context

Connecting to a session

IDirectPlay8Client::Connect() DirectPlay performs one connection

(server)

IDirectPlay8Peer::Connect() DirectPlay performs multiple connections

(host and other players)

Anatomy of a connection

Connection established at the SP layer Indication provided to protocol layer

3 way exchange at protocol layer Required for round-trip time calculation First indication of link speed Indication provided to core layer

3 way exchange at the core layer Exchange player and session information

Connecting (cont’d)

IDirectPlay8Client::Connect() Connection is established to the server Connect info is sent to the server for

verification Server returns name table or failure

response Client installs name table Client ACK’s name table Completion is posted to the message

handler with result and possible reply

ApplicationApplication DirectPlay8DirectPlay8

ConnectingConnectingClient Successfully ConnectingClient Successfully Connecting

Connect()Connect() Protocol ConnectionProtocol Connection

DirectPlay8DirectPlay8 ApplicationApplication

Name TableName Table

ConnectConnectCompleteComplete

IndicateIndicateConnectConnectConnect DataConnect Data

Name Table ACKName Table ACK

CreateCreatePlayerPlayer(client)(client)

Connecting (cont’d)

IDirectPlay8Peer::Connect() Connection is established to the host Connect info is sent to the host for

verification Host sends name table or failure response

to joining player and name table update to existing players

Joining player installs name table DPN_MSGID_CREATE_PLAYER notification

posted to message handler of joining player for host and joining player

Joining player ACK’s name table to host

Connecting (cont’d)

IDirectPlay8Peer::Connect() (cont’d) DPN_MSGID_CREATE_PLAYER notification

posted to message handler of host for joining player

Host instructs existing players to connect to joining player

Existing players connect to the joining player

DPN_MSGID_CREATE_PLAYER notification posted to message handler of existing player for joining player

Connecting (cont’d)

IDirectPlay8Peer::Connect() (cont’d) DPN_MSGID_CREATE_PLAYER notification

posted to message handler of joining player for existing player

Completion is posted to the message handler with result and possible reply

If an existing player cannot connect to the joining player, the entire connect operation will fail with DPNERR_PLAYERNOTREACHABLE

ConnectingConnectingPeer Successfully ConnectingPeer Successfully Connecting

NewNewPlayerPlayer

HostHostPlayerPlayer

ExistingExistingPlayerPlayer

Connect InfoConnect Info

Name TableName Table

Name Table ACKName Table ACK

Name Table UpdateName Table Update

Instruct to ConnectInstruct to Connect

Connect to new playerConnect to new player

Create Player Create Player (new-player, (new-player,

host)host)

Create Player Create Player (new-player)(new-player)

Create Player Create Player (new-player)(new-player)

Create Player Create Player (existing-(existing-

player)player)

Connecting (cont’d)

Connect completion will always be generated Even for synchronous or failed connects Reply from host included

Clients can only send or receive messages after DPN_MSGID_CONNECT_COMPLETE has been posted

Peers can only send or receive messages after DPN_MSGID_CREATE_PLAYER has been posted

Sending Data

SendTo() / Send() called DirectPlay core layer makes a copy of the

send buffer unless DPNSEND_NOCOPY flag specified

Core layer hands the send buffer to the protocol layer

Protocol layer breaks the send down into frames and hands the frames to the service provider layer

Service provider layer puts the frames onto the wire

Sending Data (cont’d)

Non-guaranteed sends Send completion

(DPN_MSGID_SEND_COMPLETE) posted to message handler as soon as data gets on the wire

Guaranteed sends Send completion posted to message

handler when receiver ACK’s receipt of the data

Only guarantees delivery to receiver Does not guarantee indication to receiver’s

message handler, or processing

Sending Data (cont’d)

Complete-on-process sends Send completion posted to message

handler when receiver’s message handler has returned from DPN_MSGID_RECEIVE notification

Handled behind the scenes by DirectPlay Application does not need to implement

this functionality

ApplicationApplication DirectPlay8DirectPlay8

Sending Data (cont’d)Sending Data (cont’d)Guaranteed Send ExampleGuaranteed Send Example

SendTo()SendTo() FramesFrames

DirectPlay8DirectPlay8 ApplicationApplication

ACKACKMessageMessageHandlerHandler(Send(Send

Complete)Complete)

MessageMessageHandlerHandler

(Receive)(Receive)

Sending Data (cont’d)

DirectPlay throttles sent messages Processing rate of the receiver Transmission characteristics of the link

Processing rate of the receiver is a dependent upon application design Speed of servicing message handler

notifications (DPN_MSGID_RECEIVE)

Sending Data (cont’d)

Transmission characteristics include: Physical connection limitations (modem,

LAN, DSL, etc.) Temporary conditions (network drops,

congestion, etc.)

Sending Data (cont’d)

Avoid sending data faster than the link can handle Use GetSendQueueInfo() Use GetConnectionInfo() Use timeouts

GetSendQueueInfo() Data (bytes, messages) queued to be sent

GetConnectionInfo() Transmission characteristics of the

connection

Sending Data (cont’d)

Timeouts Time before placed on wire Prevent send queue from growing without

bound Enforce timeliness of data Provide connection feedback

Receiving Data

Received messages posted to message handler with DPN_MSGID_RECEIVE

Receive buffer may not be aligned Application can:

Process the message immediately, and return DPN_OK

Retain the message and return DPNSUCCESS_PENDING

If the message is retained, it should be returned to DirectPlay with ReturnBuffer()

Receiving Data (cont’d)

Thread used to indicate message belongs to DirectPlay Holding it will prevent work from being

done, lowering throughput Avoid blocking on callback threads Number of threads available for work may

be set via SetSPCaps() Performance and scalability issues

Dropped Connections

Calling Close() will gracefully terminate connections

DirectPlay handles ungraceful terminations by timing out the connection Guaranteed sends fail to be ACK’d after a

certain number of retries Keep-alive traffic fails to be ACK’d

Dropped ConnectionsPeer-peer sessions Host detects dropped connection

Removes dropped player from session by sending DESTROY_PLAYER messages

Non-host player detects dropped connection to host player If player is the next oldest player, initiates

host migration

Dropped ConnectionsPeer-peer sessions (cont’d) Non-host player detects dropped

connection to another non-host player Informs host player of drop, and requests

integrity check Host queries “dropped” player If “dropped” player responds, host drops

detecting player Otherwise, query will result in host

detecting drop as well

Name Table

Name table contains all players and groups in the session

In peer-peer, all players keep a synchronized copy Controlled by the host player

In client-server, only server player keeps a copy

Name Table (cont’d)

All transactions are ordered including: Create player Destroy player Create group Destroy group Add player to group Remove player from group Update player info Update group info

Transactions are processed in-order on all players

Host Migration

Must set DPNSESSION_MIGRATE_HOST in application description when calling Host()

Migration is based on join order Host is the oldest player in the session

When a player is disconnected from the session All players determine if they are the oldest

remaining player Oldest remaining player is host-elect

Host Migration (cont’d)

Host-elect: Requests name table versions from all

players Ensures local name table is up-to-date Sends missing name table transactions to

remaining players Informs remaining players that updates

have been transacted

All players perform unfinished transactions

Common ProblemsCommon Problems

Common Problems

Unnecessary EnumHosts() calls prior to Connect() If you know the address of the host/server

player, you don’t need to call EnumHosts() before Connect()

Calling Close() between EnumHosts() and Connect() You can connect on the same interface you

enumerate on Successful Connect() will terminate

enumerations

Common Problems (cont’d)

Rebuilding host address between EnumHosts() and Connect() Addresses contain DirectPlay specific,

non-user data required for NAT and firewall traversal

Reuse addresses handed back from message handler (DPN_MSGID_ENUM_HOSTS_RESPONSE)

Not taking a reference on addresses returned in message handler (DPN_MSGID_ENUM_HOSTS_REPSONSE)

Common Problems (cont’d)

Modifying / releasing the send buffer when using DPNSEND_NOCOPY

Sending infrequent traffic prevents link from tuning effectively

Not monitoring send queue Send queue may grow without bound Use timeouts

Sending very small messages Inefficient Coalesce data before sending

Common Problems (cont’d)

Holding message handler call-back threads Prevent DirectPlay from performing work Result in lower throughput

Tuning (increasing) thread count Increase performance if I/O bound Lower performance if CPU bound

Calling Close() on UI thread Close() is synchronous No UI can be performed

Common Problems (cont’d)Common Problems (cont’d)

Forgetting to set Forgetting to set DPNINITIALIZE_DISABLEPARAMVAL DPNINITIALIZE_DISABLEPARAMVAL for shipping codefor shipping code Disables parameter validationDisables parameter validation Makes significant performance differenceMakes significant performance difference Don’t set while debugging though!Don’t set while debugging though!

DirectPlay8 ProtocolDirectPlay8 Protocol

DirectPlay8 Protocol

Both guaranteed and non-guaranteed on same connection UDP over IP

Messages are sent as frames All frames have a DirectPlay protocol

header DirectPlay protocol header is typically 4

bytes + UDP header + IP header

DirectPlay8 Protocol (cont’d)

Header includes: Beginning and end of message flags Reliable message flag Sequential message flag Poll message flag Command message flag Sequence numbers Other DirectPlay internal state information

DirectPlay8 Protocol (cont’d)

Frames are transmitted in a send window Maximum frame size is fixed

About 1400 bytes on IP

Frames may be smaller than maximum size for small messages

No coalescence of messages Window size is variable

DirectPlay8 Protocol (cont’d)

Send window size adjusts for connection quality and receiver Initial window size is 2 frames Minimum window size is 1 frame Maximum window size is 64 frames DirectPlay attempts to grow the window

without incurring drops or increased RTT Dropped frames shrink the window Receiver may fail to ACK frames if busy

DirectPlay8 Protocol (cont’d)

Multiple frames are ACK’d at once (TCP) One dropped frame will block all

further transmissions DirectPlay ACK’s all frames in the window

Sequence number if all arrive Selective ACK (SACK) with bit-mask (NACK) if

some are missing

ACK’s are piggybacked on data frames DirectPlay will send out a command frame

if no data frame is being sent

DirectPlay8 Protocol (cont’d)

Guaranteed messages Missing frames must be retried until they

arrive Non-guaranteed messages

Missing frames need to be NACK’d to advance the window

Sequential messages Require all previous frames to be ACK’d or

NACK’d before being indicated Non-sequential messages

Need all current message’s frames before being indicated

DirectPlay8 Protocol (cont’d)

Retries Guaranteed frames get re-transmitted Non-guaranteed frames are not sent

Send mask is used to advance the send window If no ACK or SACK is received for a given

interval, all frames in the window will be retried

Several retries may be attempted Retry interval grows between retries for a

given frame If several retries fail to generate an ACK or

SACK, the connection will be dropped

DirectPlay8 Protocol (cont’d)

Keep-alives Prevents connection from timing-out

during periods of inactivity If no guaranteed traffic on the link for a

specified period of time (see DPN_SP_CAPS), a keep-alive will be sent

Guaranteed message Generates an ACK or SACK from the

receiver

Summary

Continual work by DirectPlay team Large test organization Save development time and resources

QuestionsQuestions

[email protected]@microsoft.com