Page 1
University Computing ServicesUniversity Computing Services
EDUCAUSE Mid-Atlantic Regional Conference16 January 2003
An Infrastructure and An Infrastructure and Accounting Response to Accounting Response to
Peer to Peer Traffic VolumePeer to Peer Traffic Volume
Dr. Michael R MundraneDirector of Telecommunications
Rutgers University Computing Services
Page 2
University Computing ServicesUniversity Computing Services
CopyrightCopyright
Copyright Michael R Mundrane 2003. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.
Page 3
University Computing ServicesUniversity Computing Services
AgendaAgenda
• Statement of Problem
• Objectives
• Approach
• Results
• Conclusions
Page 4
University Computing ServicesUniversity Computing Services
Statement of ProblemStatement of Problem
Is he kidding? P2P is the problem!
Page 5
University Computing ServicesUniversity Computing Services
Network EvolutionNetwork Evolution
• Sporadic
• Unequally funded
• Unstructured
• Immediacy
• Complex
• Point services
• Faculty centric
Page 6
University Computing ServicesUniversity Computing Services
Application ModelsApplication Models
• Limited customer interface • Minimal administration• Centralized management• Centralized storage• hub and spoke infrastructure• Minimal bandwidth
Terminal Host
Page 7
University Computing ServicesUniversity Computing Services
Application ModelsApplication Models
• Rich customer interface
• Medium administration
• Centralized management
• Hybrid storage (server and client)
• Tiered network infrastructure
• Bandwidth server/s dependant
Client Server
Page 8
University Computing ServicesUniversity Computing Services
Application ModelsApplication Models
• Rich user interface
• High touch administration
• Distributed management (costly)
• Distributed storage (difficult to manage)
• Complex mesh infrastructure
• High bandwidth
Peer Peer
Page 9
University Computing ServicesUniversity Computing Services
Cooperative?!?Cooperative?!?
A. Badges? We don’t see no stinking badges!
Q. Excuse me, would you please forward the business activity associated with your traffic so that we can adjust our records?
Page 10
University Computing ServicesUniversity Computing Services
ObjectivesObjectives
More than near term survival!
Page 11
University Computing ServicesUniversity Computing Services
Essential CharacteristicsEssential Characteristics
• Preserve behavior
• Ensure access
• Moderate impact
• Protect privacy
• Avoid value judgments
• Apply to new applications
Page 12
University Computing ServicesUniversity Computing Services
AssumptionsAssumptions
• Large number of hosts
• Small number of problems
• Service consumers
• Many random light providers
• Few heavy providers
• Responsive community
Page 13
University Computing ServicesUniversity Computing Services
Just Use Traffic ShapingJust Use Traffic Shaping
Page 14
University Computing ServicesUniversity Computing Services
Just Use Traffic ShapingJust Use Traffic Shaping
• Cisco routers
• CAR – traffic class
• MicroCAR – identified flow
day
Gigabytes
bits
byte
M
G
K
M
day
onds
ond
32.1
8024,1024,1
sec400,86
sec
Kilobits128
Page 15
University Computing ServicesUniversity Computing Services
Just Use QoSJust Use QoS
Page 16
University Computing ServicesUniversity Computing Services
Just Use QoSJust Use QoS
• Classification
• Differentiation
• Admission control
• Provisioning
• Bandwidth
• Latency
• Jitter
Page 17
University Computing ServicesUniversity Computing Services
QoS DifferentiationQoS Differentiation
P2P Other
10Mbit 90Mbit
Page 18
University Computing ServicesUniversity Computing Services
QoS DifferentiationQoS Differentiation
10Mbit
Differentiation w/o admission control only
defers the problem!
Page 19
University Computing ServicesUniversity Computing Services
Rutgers NetworkRutgers Network
• 40,000+ hosts
• 1200+ networks
• 200+ routers
• 17 zones
• 7 campuses
• 3 regions
• 1 autonomous system
Page 20
University Computing ServicesUniversity Computing Services
ApproachApproach
No single solution!
Page 21
University Computing ServicesUniversity Computing Services
Best Network PracticesBest Network Practices
• Modular
• Layered
• Aggregated
• Scalable
• Uniform
• Deterministic
• Comprehensible
Page 22
University Computing ServicesUniversity Computing Services
DeviceDevice DeviceDevice
DeviceDevice DeviceDevice
Intra-building Backbone
Building
Intra-building BackboneIntra-building Backbone
RUNet ~ 1200
Page 23
University Computing ServicesUniversity Computing Services
BuildingBuilding BuildingBuilding
BuildingBuilding BuildingBuilding
Inter-building Backbone
Zone
Inter-building BackboneInter-building Backbone
RUNet 17
Page 24
University Computing ServicesUniversity Computing Services
ZoneZone ZoneZone
ZoneZone ZoneZone
Intra-campus Backbone
Campus
Intra-campus BackboneIntra-campus Backbone
RUNet 7
Page 25
University Computing ServicesUniversity Computing Services
CampusCampus CampusCampus
CampusCampus CampusCampus
Inter-campus backbone
Region
Inter-campus BackboneInter-campus Backbone
RUNet 3
Page 26
University Computing ServicesUniversity Computing Services
MANMAN MANMAN
MANMAN MANMAN
Inter-region Backbone
Autonomous System
Inter-region BackboneInter-region Backbone
RUNet 1
Page 27
University Computing ServicesUniversity Computing Services
CharacteristicsCharacteristics
• Geographic independence
• Shallow topology
• Similar (not optimal) paths
• Low latency
• Uniform characteristics
• 1 autonomous system
Page 28
University Computing ServicesUniversity Computing Services
Collect DataCollect Data
• Netflow
• Source/Destination address
• Source/Destination ports
• Protocol
• Packets/Octets/Flows
• Start/End time
Page 29
University Computing ServicesUniversity Computing Services
Raw DataRaw Data
• 10 minute granularity
• Each source
• Each destination
• 1,000,000 addresses
• 10,000,000 records
• 1 Gigabytes, 1 day
Page 30
University Computing ServicesUniversity Computing Services
Rollup DataRollup Data
• Rutgers sources/sinks
• Data >= 1024, 10 minutes
• Data >= 6*1024, 1 hour
• Data >= 24*6*1024, 1 day
• 20,000 unique hosts
• 20,000 records
• 1 Megabyte
Page 31
University Computing ServicesUniversity Computing Services
Filtered DataFiltered Data
• Rutgers sources/sinks
• Data >= 512 Megabytes, 1 Day
• 125 unique hosts
• 125 records
• 50 Kilobytes
Page 32
University Computing ServicesUniversity Computing Services
ReductionReduction
10,000,000 99.799%20,000 0.200%125 0.001%
10,020,125
Addresses
1,000,000 98.027%20,000 1.961%125 0.012%
1,020,125
Records1,073,741,824 99.898%
1,048,576 0.098%51,200 0.005%
1,074,841,600
Size
1,000 90.090%100 9.009%10 0.901%
1,110
Model
Page 33
University Computing ServicesUniversity Computing Services
DistributionDistribution
• Reread entire data set
• Limit to filtered only
• Rollup based on external address
• Preserve individual distributions
• Useful to reduce contact
Page 34
University Computing ServicesUniversity Computing Services
Questionable DistributionQuestionable Distribution
Page 35
University Computing ServicesUniversity Computing Services
Good DistributionGood Distribution
Page 36
University Computing ServicesUniversity Computing Services
Storage
Process ModelProcess Model
Rollup
Internet
NetflowFilterDistribution
Analyze
Page 37
University Computing ServicesUniversity Computing Services
Residence AssumptionsResidence Assumptions
• RFC1918 address space
• Large number of hosts
• Small number of problems
• Service consumers
• No service providers
• Unresponsive community
Page 38
University Computing ServicesUniversity Computing Services
Set LimitsSet Limits
• 2048 MB download
• 512 MB upload
• 7 day granularity
• Sliding window
• Enforcement
Page 39
University Computing ServicesUniversity Computing Services
ReferenceReference
• 4 movies
• 400 songs
• 45,000 web pages
• 2048 Megabytes
Page 40
University Computing ServicesUniversity Computing Services
Oracle
Process ModelProcess Model
Table
Rollup
Table
Enforce
Table
GatherInternet
Netflow
WWW
Custom ACL
Page 41
University Computing ServicesUniversity Computing Services
Traffic ShapingTraffic Shaping
• 1 Day on
• 7 Days off
• Multiplexed
• 1:8 ratio
• Automatic
• Aggregated
• Not legalistic
Load
Impact
Page 42
University Computing ServicesUniversity Computing Services
Differentiated ServiceDifferentiated Service
• Residence facilities
• Other locations
• Two traffic classes
• 1:2 host distribution
• 1:1 bandwidth allocation
• CAR enforced
Page 43
University Computing ServicesUniversity Computing Services
ResultsResults
Some pains, some gains!
Page 44
University Computing ServicesUniversity Computing Services
Extra EffortsExtra Efforts
• Registration
• Port Address Translation
• Split horizon DNS
• Help desk/Appeals
• Address hopping
• Proxy services
• Oracle
Page 45
University Computing ServicesUniversity Computing Services
90% Data Sinks90% Data Sinks
Page 46
University Computing ServicesUniversity Computing Services
99.99% Data Sinks99.99% Data Sinks
Page 47
University Computing ServicesUniversity Computing Services
90% Data Sources90% Data Sources
Page 48
University Computing ServicesUniversity Computing Services
99.99% Data Sources99.99% Data Sources
Page 49
University Computing ServicesUniversity Computing Services
Internet TrafficInternet Traffic
Page 50
University Computing ServicesUniversity Computing Services
ConclusionsConclusions
• Modest applications with broad demographics have profound impact.
• Students have free time.
• Network best practices never more important.
• Cooperative generic methods can be effective (w/ encouragement).
• No magic bullet.
Page 51
University Computing ServicesUniversity Computing Services
Questions?
[email protected]