(C) 2003 Mulitfacet Project University of Wisconsin-Madison Evaluating a $2M Commercial Server on a $2K PC and Related Challenges Mark D. Hill Multifacet Project (www.cs.wisc.edu/multifacet) Computer Sciences Department University of Wisconsin—Madison February 2003
33
Embed
(C) 2003 Mulitfacet ProjectUniversity of Wisconsin-Madison Evaluating a $2M Commercial Server on a $2K PC and Related Challenges Mark D. Hill Multifacet.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
(C) 2003 Mulitfacet Project University of Wisconsin-Madison
Evaluating a $2M Commercial Server on a $2K PC
and Related Challenges
Mark D. Hill
Multifacet Project (www.cs.wisc.edu/multifacet)
Computer Sciences Department
University of Wisconsin—Madison
February 2003
Wisconsin Multifacet Project2 Methods
• Commercial Servers– Processors, memory, disks $2M– Run large multithreaded transaction-oriented workloads– Use commercial applications on commercial OS
• To Simulate on $2K PC– Scale & tune workloads– Manage simulation complexity– Cope with workload variability
• NSF Challenges in Computer Architecture Evaluation
Advice researchers, program committees, & funders basically “know," but often forget to heed
Wisconsin Multifacet Project3 Methods
Multifacet: Commercial Server Design
• Wisconsin Multifacet Project– Directed by Mark D. Hill & David A. Wood– Sponsors: NSF, WI, IBM, Intel, & Sun– Current Contributors: Alaa Alameldeen, Brad Beckman,
Milo Martin, Mike Marty, Kevin Moore, & Min Xu
• Commercial Server Availability– SafetyNet tolerates some transient faults [ISCA 2002]
• Commercial Server Software Complexity– Flight Data Recorder aids debugging of multithreaded programs
• Thread-level parallelism– Hardware Multi-threading– Traditional Multi-processing
Wisconsin Multifacet Project18 Methods
Managing Simulator Complexity
Functional Simulator
Timing Simulator Functional-First (Trace-driven)
- Timing feedback
+ Timing feedback- Tight Coupling- Performance?
Timing and FunctionalSimulator Integrated (SimOS)
- Complex
Timing-DirectedFunctional Simulator
Timing Simulator
Complete TimingNo? Function
No TimingComplete Function
Timing-First (Multifacet)Functional Simulator
Timing Simulator
Complete TimingPartial Function
No TimingComplete Function
Wisconsin Multifacet Project19 Methods
Timing-First Operation
Timing Simulator
Functional Simulator
CPUSystem
RAMNet
wor
k
addload
Cache
CPU
Execute Commit
Reload
Verify
• Timing Simulator runs speculatively ahead• On commit, calls Functional Simulator to verify• Reload Timing Simulator state if necessary,
e.g., interrupt, unimplemented instruction
Wisconsin Multifacet Project20 Methods
Timing-First Discussion
• Supports speculative multi-processor timing models• Leverages existing simulators• Rapid development time (e.g., immediate checks)• Has low simulation overhead (18% uniprocessor)• Introduces relatively little performance error (< 3%)• BUT duplicates some code & function
Timing-First SimulationFunctional Simulator
Timing Simulator
Complete TimingPartial Function
No TimingComplete Function
Wisconsin Multifacet Project21 Methods
Outline
• Workload & Simulation Methods
• Separate Timing & Functional Simulation
• Cope with Workload Variability– Variability in Multithreaded Workloads– Coping in Simulation
• NSF Challenges in Computer Architecture Evaluation
Wisconsin Multifacet Project22 Methods
What is Happening Here?
OLTP
Wisconsin Multifacet Project23 Methods
What is Happening Here?
• How can slower memory lead to faster workload?
• Answer: Multithreaded workload takes different path– Different lock race outcomes– Different scheduling decisions
• (1) Does this happen for real hardware?
• (2) If so, what should we do about it?
Wisconsin Multifacet Project24 Methods
One Second Intervals (on real hardware)
OLTP
Wisconsin Multifacet Project25 Methods
60 Second Intervals (on real hardware)
16-day simulation
OLTP
Wisconsin Multifacet Project26 Methods
Coping with Workload Variability
• Running (simulating) long enough not appealing
• Need to separate coincidental & real effects• Standard statistics on real hardware
– Variation within base system runs
vs. variation between base & enhanced system runs– But deterministic simulation has no “within” variation
• Solution with deterministic simulation– Add pseudo-random delay on L2 misses– Simulate base (enhanced) system many times– Use simple or complex statistics
Wisconsin Multifacet Project27 Methods
Confidence Interval Example
• Estimate #runs to getnon-overlapping confidence intervals
ROB
Wisconsin Multifacet Project28 Methods
Outline
• Workload & Simulation Methods
• Separate Timing & Functional Simulation
• Cope with Workload Variability
• NSF Challenges in Computer Architecture EvaluationAdvice researchers, program committees, & funders
basically “know," but often forget to heed
Wisconsin Multifacet Project29 Methods
NSF Challenges in Computer Architecture Evaluation
• Dec 2001 NSF Computer Systems Architecture Workshop– Report in IEEE Computer, Aug 2003
– By Kevin Skadon, Margaret Martonosi,David August,Mark Hill, David Lilja, & Vijay Pai
• Simulation Frameworks– P (Problem): Need more modularity, portability, & reuse
– R (Recommendation): More simulations frameworks,e.g., ASIM & Liberty
• Benchmarking– P: Benchmarks for too few domains
– R: Reward benchmark development & characterization; consider micro- and synthetic benchmarks
Wisconsin Multifacet Project30 Methods
NSF Challenges in Computer Architecture Evaluation
• Abstractions & Methodology– P: Believe simulation too much; other methods insufficiently
• 1985 ISCA: 30% simulation & 30% modeling
• 2001 ISCA: 90% simulation & 0% modeling
– R: Push analytic models for insight, cross validation, & far—reaching research
• Metrics, Accuracy, & Validation– P: Too dependent on relative & aggregate metrics– R: More metrics & statistical methods, especially when
– Cope w/ workload variability via randomness & statistics
• References (www.cs.wisc.edu/multifacet/papers)– Simulating a $2M Commercial Server on a $2K PC [Computer 2/03]– Full-System Timing-First Simulation [Sigmetrics 02]– Variability in Architectural Simulations … [HPCA 03]
• NSF Panel– Challenges in Computer Architecture Evaluation [Computer 8/03]