Proposed Diffserv Evaluation for Emerge Sites

Proposed Diffserv Evaluation for Emerge

SitesVolker Sander

sander@mcs.anl.gov

Argonne National Laboratory1/20/2000

2

Outline

Basic GARA overview New GARA-features Basic set of experiments Installation of GARA Running the experiments using GARA Timeline

3

The Basic Diffserv Concept

Aggregate based scheduling implements EF PHB

Classify and Mark

packets

Shape Traffic

Flow based

Classify and Mark

packets

Shape Traffic

4

Proposed Implementation

Client

GARA API

GARA Diffserv

Resource Manager

Server

5

The Multidomain Extension

Server

GARA Diffserv

Resource

Manager

Client

GARA API GARA Diffserv

Resource

Manager

6

GARA Globus Architecture for Reservation and

Allocation Three important contributions:

– Support for Advance Reservations (AG)

– Uniform treatment of underlying resources> Network, disk, CPU etc…

> Currently supported:

Differentiated Services, DPSS, and DSRT

– Layered architecture enables generic co-reservation agents

> User is able to select API best suitable for his need

7

GARA Basics—Reservations There is a generic “reservation”, which has

several properties:– Start Time (“now” or future) and Duration

– Resource type/Underlying resource identifier

– Resource-specific (bandwidth, % CPU…) All reservations are treated uniformly:

– Create/Modify (Given properties)=> Returns Reservation Handle

– Destroy

– Monitor (Callbacks or Polling)

8

A picture of what happens(creating a reservation)

Gatekeeper(Authenticate and Authorize)

CreateUser Program or Agent

Reservation Handle

Resource Manager

Gatekeeper Service

Local Resource Manager API (LRAM)

Resource (Router...)

9

A picture of what happens(claiming a reservation)

User Programbind(handle)

LRAM API

Resource Manager

Setup reservation

ResourceEnforce Reservation

Note: GARA supports third party binding

10

11

Example: Network reservation In advance, make a reservation:

RSL:& (reservation-type=network)

(start-time=now) (duration=3600)

(endpoint-a=128.135.11.4) (endpoint-b=128.135.11.1)

(bandwidth=1000) (protocol=udp)

(reservation-subtype=low-latency)

Handle: Opaque string H1

At run-time:

– Bind(H1,RSL)

RSL:& (which-endpoint=a) (endpoint-a-port=5000)

(endpoint-b-port=5000)

12

The Resource Manager To track advanced reservations, we use a simple “slot

table” We assume exclusive access to resource through the

resource manager We aren’t tied to this particular resource manager

% of available resource allocated

0%

100%

Time

13

End-to-End Network Reservations Algorithm for a single network reservation

– Discover which gatekeepers/resource managers need to be contacted (MDS)

– Make reservation at each one.

User process Workstation

Resource Manager Resource Manager

MDSEnd-to-End Co-Reservation Agent

14

The GARA API

globus_gara_reservation_create() Gatekeeper contact RSL reservation specification Reservation Handle or Error

globus_gara_reservation_modify() Old Reservation Handle RSL reservation specification New Reservation Handle or Error

15

The GARA API (continued) globus_gara_reservation_cancel()

Reservation Handle Error

globus_gara_reservation_status() Reservation Handle Status/Error

globus_gara_reservation_callback_register() Reservation Handle Callback Function/User Parameter Error

16

New GARA features New Reservation-Subtypes

– low-latency> Used to support jitter sensitive applications using the same EF

aggregate behavior as high-throughput TCP applications

> Based on traffic shaping and Priority-Queuing

> Currently being evaluated

– background> Used to support bulk transfers, including deadline support. Takes at

least the amount of premium traffic required to fulfill the deadline; more if not used actively

Monitoring of the network edges– Provides feedback to applications when they send too

fast

17

Network + CPU Reservations

0

20000

40000

60000

80000

100000

0 50 100 150 200 250

Time

Ban

dw

idth

(K

bp

s)

TCP Flow Competitive Traffic

18

Example of bulk transfer (LAN)

0

20000

40000

60000

80000

100000

0 50 100 150 200 250

Time

Ban

dw

idth

(K

bp

s)

backgroundforegroundcompetitive

19

Example of bulk transfer (WAN)

0

10000

20000

30000

40000

50000

0 20 40 60 80 100 120 140 160 180

Time

Ban

dw

idth

(K

bp

s)

backgroundforegroundcompetitive

20

Feedback from Monitoring the Edges

Receiver Bound BW

0

1000

2000

3000

4000

5000

6000

7000

8000

90001

11

21

31

41

51

61

71

81

91

10

1

11

1

12

1

13

1

14

1

15

1

16

1

17

1

18

1

Seconds

KB

/s

Series2

21

Evaluation Tools UDP Traffic Generator

– Modified Version of Andy Adamson’s gen_send and gen_recv

> Evaluate admission control

> Creating competing traffic

– MGEN/Drec> Evaluate delay and Jitter for Premium UDP Flow

Modified Version of ttcp – GARA-enabled (wait for reservation)

– Support for a desired application rate> long term controllong term control

> short term controlshort term control

– Consecutive bandwidth reporting

– Bulk transfer ttcp

22

Basic Experiments Evaluate Admission Control Evaluate premium TCP Flows under Congestion

– Analyze network traces with tcpdump Evaluate UDP

– Delay

– Jitter Evaluate Inter-Domain Issues

Fundamental idea: Get the necessary knowledge to be able to experiment with REAL applications

23

Basic Experiment I Goal: Proof of Admission Control

– Gather packet statistics from gen_recv for different configurations and different traffic profiles:

UDP Rate comparison for different CAR buffer sizes(Desired rate: >40Mbps; Average rate limit 40 Mbps)

Packet size 530 Bytes

36500

37000

37500

38000

38500

39000

39500

40000

40500

1 3 5 7 9

11

13

15

17

19

21

23

25

27

29

Time (s)

Ac

tua

l R

ate

(K

bp

s)

100KB

1MB

2MB

Buffer size

24

Basic Experiment I

gen_send

GARA Diffserv

Resource Manager

gen_recv

25

Basic Experiment II Goal: Proof of Premium Quality for TCP flows

– Run modified ttcp with different traffic and reservation profiles; gather receiver bound statistics

– Create congestion using gen_send/recvStable Long Term TCP Traffic

(Attempted Rate: 6000KB/s, Buffer size: 2000000 bytes)

2000

3000

4000

5000

6000

7000

8000

9000

1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97Time

Ac

tua

l R

ate

(K

bs

)

Rate

Limit

26

Basic Experiment II Enhancement

– Use tcpdump and tcptrace get a deep overview on the behavior

– Use netlogger_enabled ttcp_gara and gen_recv to publish data into MDS. “Real Time Visualization”

27

Basic Experiment II

GARA Diffserv

Resource Managerttcp -t

ttcp -r

gen_send gen_recv

28

Basic Experiment III Goal: Demonstrate Premium Quality for UDP flows

– Run mgen/drec with different traffic and reservation profiles; gather receiver bound statistics

– Create congestion using gen_send/recv

29

Basic Experiment III

GARA Diffserv

Resource Manager

mgen

drec

gen_send gen_recv

30

Basic Experiment IV Goal: Proof Premium Quality for UDP flows

sharing one EF aggregate– Run mgen/drec with different traffic and

reservation profiles; gather receiver bound statistics

– Run high-throughput ttcp using a correct reservation

– Create congestion using gen_send/recv

31

Basic Experiment IV

GARA Diffserv

Resource Manager

ttcp -t

ttcp -r

mgen drec

gen_recv

gen_send

32

Additional Experiments

Bulk-Transfer Reevaluate inter-domain behavior

– Add Traffic-Shaping between domains Test Inter-Domain Admission control Prepare the Instrumentation Go with real applications

33

Installing GARA Get GARA

– (currently through E-Mail from sander/roy @mcs.anl.gov) Get SSLeay 0.9.0 and install it Get Globus 1.1 configure and compile it

deploy is not necessary for doing tests but recommended

Run grid-cert-request to apply for a Grid X509 Certificate

add “gara-service stderr_log,local_cred - $GARA_DIR/gara/programs/globus_gatekeeper_gara_service -d”

to $GLOBUS_DEPLOY_DIR/etc/globus-services Read gara/docs/* Edit gara/build Run gara/build Customize GARA

– gara/resource_manager/programs/diffserv_manager.conf

– gara/resource_manager/programs/setup_flow.cfg

– (Samples are in the documentation directory)

34

Building GARA $GARA_DIR/build #!/bin/ksh CC=cc CXX=CC CFLAGS="-g -v -mt" GLOBUS_DIR=/soft/pub/packages/globus-1.1.0/development/sparc-sun-solaris2.7_pt hreads_standard_debug SSL_LIBRARY_DIR=/soft/pub/packages/SSLeay-0.9.0/lib

export CC CXX CFLAGS GLOBUS_DIR SSL_LIBRARY_DIR

configure make clean make depend make all # # For portability reasons you might not want to do a make withdsrt.. # in case: just comment out this line. #make withdsrt

Run build

35

Configuring GARA resource_manager/programs/diffserv_manager.confPublicationMethod web mds

LDAPDistinguishedName "dslnet1.mcs.anl.gov:34880:/C=US/O=Globus/O=Argonne Nati

onal Laboratory/OU=MCS/CN=roy”

MDSPublishPassword “password”

WEBPublishFile /home/sander/public_html/diffserv_manager_slots.html

NoOfRouters 2

IPAddressesServed[1] 140.221.48.162,140.221.48.146,140.221.48.164,140.221.48.1

65,140.221.48.166,140.221.48.167,140.221.48.34

IPAddressesServed[2] 140.221.48.98,140.221.48.114,140.221.48.102,140.221.48.10

3,140.221.48.104,140.221.48.105

Parameters: "LDAPDistinguishedName",

"MDSPublishPassword",

"WEBPublishFile",

"FILEPublishFile",

"Quantity",

"ClearSlotTable",

"SlotTableFilename",

"Verbose",

"PublicationMethod",

"DontReallyBindReservations"

36

Configuring GARA resource_manager/programs/setup_flow.conf1_ROUTER 140.221.48.145

1_USERNAME my_userid

1_PASSWORD password

1_INTERFACE1 FastEthernet1/0/1 140.221.48.162

1_INTERFACE2 FastEthernet1/0/1 140.221.48.164

1_INTERFACE3 FastEthernet1/0/1 140.221.48.165

1_INTERFACE4 FastEthernet1/0/1 140.221.48.166

1_INTERFACE5 FastEthernet1/0/1 140.221.48.167

1_INTERFACE6 FastEthernet1/0/0 140.221.48.146

#

#

2_ROUTER baku

2_USERNAME my_userid

2_PASSWORD password

2_INTERFACE1 FastEthernet1/0/1 140.221.48.98

2_INTERFACE2 FastEthernet1/0/1 140.221.48.102

2_INTERFACE3 FastEthernet1/0/1 140.221.48.103

2_INTERFACE4 FastEthernet1/0/1 140.221.48.104

2_INTERFACE5 FastEthernet1/0/1 140.221.48.105

2_INTERFACE6 FastEthernet1/0/0 140.221.48.114

37

Setting up the routers Get new IOS XE beta release

IOS (tm) RSP Software (RSP-PV-M), Experimental Version 12.0(19991129:193607) [rolsen-conn_isp_llq 144]

Available at ftp://ftpeng.cisco.com/rolsen/rsp-pv-mz.120699

Edge-Routers (Update of setup_flow required)class-map match-all EF

match ip dscp 46

class-map match-all priority-qos

match qos-group 99

policy-map emerge

class EF

bandwidth 30000

queue-limit 500

class priority-qos

priority

policy-map set-precedence

– Attach service-policy set-precendence to every input

– Attach service-policy emerge to every output interface

38

Setting up the routers

Interior-Routers (Update of setup_flow required)class-map match-all EF

match ip dscp 46

policy-map emerge

class EF

bandwidth 30000

queue-limit 500

– Attach service-policy emerge to every output interface

39

Updating setup_flow

setup_flow is the script which is configuring the router Possible update of setup_flow (is)

197 expect_after default { close; continue }

198

199 expect "name: " { send "$username\r" }

200 expect "word: " { send "$password\r" }

Might be (without tacacs+)197 expect_after default { close; continue }

198

199 expect ”word: " { send "$password\r” }

200 expect “#” { send “enable\r” }

201 expect "word: " { send "$password\r" }

40

Updating the MDS

End-to-end API searches for end-systems in MDS Publish information for all end-systems into MDS

dslnet1.cldif:

---------------------

dn: hn=dslnet1.mcs.anl.gov, ou=Mathematics and Computer Science Division,

o=Argonne National Laboratory, o=Globus, c=US

…. If the machine is already in MDS this is enough

networkreservationcontact:

dslnet1.mcs.anl.gov:30000:/C=US/O=Globus/O=Argonne National

Laboratory/OU=MCS/CN=roy197 expect_after default { close; continue }

– run grid-info-update Tools will be available soon

41

The GARA Source Tree Subdirectories represent different layers

– end2end

– gara (gara/java GUI)

– lram

– resource_manager

– slot_manager

– common

– logging

– simple_multi

– (dsrt)

We are providing examples for every layer docs directory contains lots of useful

information

42

Doing Further Tests

Select your favorite application Determine port numbers edit (set endpoint systems) and run script:

gara/gara/utilities/demo_make_reservation Create congestion Try demo_modify_reservation for getting

different premium bandwidth Try demo_cancel_reservation Or use the GUI

43

Timeline Establish connectivity (January)

– mostly done!

– Setup accounts for Emerge-User (January!)

– Check connectivity Install and customize GARA (Jan - Feb)

– partly done

– Configure and check with GUI without any application (Just look at differv_manager and router)

Do basic experiments (February-March) Start with multi-domain issues (March-April) Port real applications to the Testbed (Feb-?)

44

Questions...

Now or later… (sander/roy @ mcs.anl.gov)