1 State of Network Monitoring and Analysis in the US Les Cottrell, KC Claffy, Brian Tierney, Ronn Ritke, Hans-Werner Braun Prepared for the LSN meeting at NSF Washington 6/10/03 Partially funded by DOE/MICS Field Work Proposal on Internet End-to-end Performance Monitoring (IEPM), by the SciDAC base program
22
Embed
1 State of Network Monitoring and Analysis in the US Les Cottrell, KC Claffy, Brian Tierney, Ronn Ritke, Hans-Werner Braun Prepared for the LSN meeting.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
State of Network Monitoring and Analysis in
the US
Les Cottrell, KC Claffy, Brian Tierney, Ronn Ritke, Hans-Werner
BraunPrepared for the LSN meeting at NSF Washington
6/10/03Partially funded by DOE/MICS Field Work
Proposal on Internet End-to-end Performance Monitoring (IEPM), by the SciDAC base program
2
Outline• Goal: for network monitoring & analysis talk:
– identify the R&D gaps and large-scale deployment issues for DOE, NSF, DARPA, NASA, NSA, NIST, etc. – the federal agencies that fund network research in US
• Two complementary presentations– High performance networking measurement needs for Science (E2E)-
• Science network measurement needs– The end-to-end challenge, illustrations– Solution– End to end Monitoring Goals– Current issues
• Problem analysis, measurement infrastructure, analysis tools, standards, collaborations
– Benefits to Science– Consequences of not addressing issues– Why not leave to industry– Appendix
• What is being done today– Who is measuring?– Who is using the measurements?– What is being measured?– What tools are being used?
3
The Problem• Distributed systems are very hard
– A distributed system is one in which I can't get my work done because a computer I've never heard of has failed. Butler Lampson
• When building distributed systems, we often observe unexpectedly low performance
• the reasons for which are usually not obvious
• The bottlenecks can be in any of the following components:– the applications– the operating systems– the disks, network adapters, bus, memory, etc. on either the
sending or receiving host– the network switches and routers, and so on
• Problems may not be logical– Most problems are operator errors, configurations, bugs
4
Anatomy of a Problem
Applications Developer
System Administrator
LAN Administrator
CampusNetworking
Gigapop Gigapop
Backbone
CampusNetworking
LAN Administrator
System Administrator
Applications Developer
How do you solvea problem along a path?
Hey, this is not
working right!
The computerIs working
OK
Talk to the other guys
Everything isAOK
No othercomplaints
The network is lightly loaded
All the lights are green
We don’t see anything wrong
Looks fine
Others are getting in ok
Not our problem
5
Problem examples: Help, it’s not working• I’ve lost my connection • Despite over-provisioned
networks user cannot get throughput expected– Wizard gap
• What should I expect the performance to be?
• It sometimes works …• What am I, as a scientist,
supposed to do?• Need
tools/measurements to detect problems, identify location, cause and time of occurrence
Is Grid server down, is the network partitioned, is there heavy congestion, did DNS fail, is a firewall preventing access …
WizardWizard
Typical userTypical userMb
its/
sM
bit
s/s
6
The Solution• A complete End-to-End monitoring framework that includes the
following components: – instrumentation tools (application, middleware, and OS monitoring)– host and network sensors (host and network monitoring)– sensor management / activation tools– event publication service– event archive service– event analysis and visualization tools– a common set of protocols for describing, exchanging, and locating
monitoring data• Need for applications (e.g. Grid middleware), diagnosis, perf. analysis
– toolkit for streamlined problem diagnosis: detection, location, isolation & reporting
• glue to multiple sources of information, traceroute archives, router info, delay/loss archives, on-demand tests, baselines
• analysis and heuristics– E2EPi working on solution, but only funded for
coordination not for all the underlying work
7
End-2-End Monitoring Goals• Have to solve the E2E performance, it is THE
critical metric for user, not just a backbone bandwidth problem
• Improve end-to-end data throughput for data intensive applications in a high-speed WAN environments
• Provide the ability to do performance analysis and fault detection in a Grid computing environment
• Provide accurate, detailed, and adaptive monitoring of all of distributed computing components, including the network
• Unfortunately, network management research has historically been very under-funded, because it is difficult to get funding bodies to recognize this as legitimate networking research, IAB Concerns & Recommendations Regarding Internet Research & Evolution
8
Current Issues 1: Problem Analysis• Cultivate systematic studies of problems,
causes, how to discover, how to report, how to by-pass– Analysis to help in deciding what are the
most important problems, see how they are tackled manually today
– Decide on which problems are most cost-effective to assist in developing tools to assist in diagnosis
9
Current issues 2: Measurement Infrastructures
• Need to build infrastructure to support troubleshooting: – Requires repetitive and on-demand measurements with
appropriate security model. – Provide recommended/accepted set of tools for delay, RTT,
loss, route tracking, "bandwidth" estimation. • Include archiving and access to data, analysis and reporting of
repetitive data. – Allow for evaluation, validation and comparison of new
SCNM), remote testing of connection (ANL NDT), – Traceroute archives– Make tools easier to comprehend and use by scientists– Encourage efforts such as Internet2 E2Epi efforts to provide
measurements inside the cloud• Extend to ESnet & other NRNs, and beyond• Fund collaboration across boundaries
– Ubiquitous coverage (require multiple toolkits): Inter agency, international, hi-speed, digital divide, long term and current
10
Current issues 3: Analysis tools• Provide measurement tools to accurately
& quickly identify performance problems, – to automatically take action to investigate
and provide information for:• Scientist• Grid support “NOC”• Network administrator or network person
– Promote well understood, accepted metrics for customers for realistic, enforceable SLAs, • provide acceptable limits, • provide tools to track
11
Current issues 4: Standards• All the above requires:
– easy to use standard ways (e.g.web services) for applications to access data from existing and new monitoring projects.
– standard naming conventions and schemas.
• This will provide the ability to share information from multiple measurement infrastructure projects
12
Current issues 5: Collaboration• Need to build multi-disciplinary teams (incent
orthogonal groups to work with one another): – include people close to eventual customers
(scientists, operational folks)• to ensure what is developed is useful, tested out in realistic
environments
– include vendors and providers in funded projects to bridge the gaps
• E2Epi is funded to provide coordination• Multi agency funding!
– This is not a problem a single agency can address– Science applications cross multi-agency networks,
but barriers to interagency network monitoring collaborations
13
Benefits to Science• Network reaches its potential
– enable new ways of doing science: • data intensive science (astrophysics, global
weather, seismology, medicine), • remote instrument control (SNS, fusion(ITER),