The Case For Prediction-based Best-effort Real-time Peter A. Dinda Bruce Lowekamp Loukas F. Kallivokas David R. O’Hallaron Carnegie Mellon University.

The Case For Prediction-based

Best-effort Real-time

Peter A. Dinda

Bruce Lowekamp

Loukas F. Kallivokas

David R. O’Hallaron

Carnegie Mellon University

2

Overview

• Distributed interactive applications• Could benefit from best-effort real-time• Example: QuakeViz (Earthquake Visualization) and

the DV (Distributed Visualization) framework

• Evidence for feasibility of prediction-based best-effort RT service for these applications

• Mapping algorithms• Execution time model• Host load prediction

3

Application Characteristics• Interactivity

• Users initiate tasks with deadlines• Timely, consistent, and predictable feedback

• Resilience• Missed deadlines are acceptable

• Distributability• Tasks can be initiated on any host

• Adaptability• Task computation and communication can be adjusted

Shared, unreserved computing environments

4

Teora, Italy1980

Motivation for QuakeViz

5

Northridge Earthquake Simulation

40 seconds of an aftershock of Jan 17, 1994 Northridge quake in San Fernando Valley of Southern California

50 x 50 x 10 km region13,422,563 nodes76,778,630 tetrahedrons1 Hz frequency resolution20 meter spatial resolution

16,666 40M x 40M SMVPs15 GBytes of RAM6.5 hours on 256 T3D PEs 80 trillion (1012) FLOPs3.5 sustained GFLOP/s1.4 peak GB/s

16,666 time steps13,422,563 3-tuples

per step

6 Terabytes6 Terabytes

Real Event

Huge Model

High Perf. Simulation

HUGE OUTPUT

6

Must Visualize Massive Remote Datasets

ProblemOne Month Turnaround Time

Datasets must be kept at remote supercomputing site due to their sheer size

Visualization is inherently distributed

7

QuakeViz: Distributed Interactive Visualizationof Massive Remote Earthquake Datasets

GoalInteractive manipulation of massive remote datasets from arbitrary clients

Sample 2 host visualization of Northridge Earthquake

8

DV: A Framework For Building Distributed Interactive Visualizations of Massive Remote Datasets

Dataset

interpolationinterpolation isosurfaceextraction

isosurfaceextraction

scenesynthesis

scenesynthesis

interpolationinterpolation morphologyreconstruction

morphologyreconstruction

localdisplay

anduser

renderingrenderingreadingreading

ROI resolution contours

•Logical View: Distributed pipelines of vtk* modules

*Visualization Toolkit, open source C++ library

User feedback and quality settings

Display update latency

•Example:

deadline

9

DV: A Framework For Building Distributed Interactive Visualizations of Massive Remote Datasets

Dataset

interpolationinterpolation isosurfaceextraction

isosurfaceextraction

scenesynthesis

scenesynthesis

interpolationinterpolation morphologyreconstruction

morphologyreconstruction

localdisplay

anduser

renderingrenderingreadingreading

ROI resolution contours

•Logical View: Distributed pipelines of vtk* modules

*Visualization Toolkit, open source C++ library

User feedback and quality settings

Display update latency

•Example:

deadline

10

Active Frames

Active Frame

n+2?

interpolationinterpolation isosurfaceextraction

isosurfaceextraction

scenesynthesis

scenesynthesis

Physical View of Example Pipeline:

deadlineActive Frame

n+1?

deadlineActive Frame

n?

deadline

•Encapsulates data, computation, and path through pipeline•Launched from server by user interaction•Dynamically chose on which host each pipeline stage will execute and what quality settings to use

11

Active Frames

Active Frame

n+2?

interpolationinterpolation isosurfaceextraction

isosurfaceextraction

scenesynthesis

scenesynthesis

Physical View of Example Pipeline:

deadlineActive Frame

n+1?

deadlineActive Frame

n?

deadline

•Encapsulates data, computation, and path through pipeline•Launched from server by user interaction•Dynamically chose on which host each pipeline stage will execute and what quality settings to use

12

Active Frame Execution ModelActive Frame

Host LoadMeasurement

NetworkMeasurementRemos

Measurement Infrastructure

Mapping Algorithm

Prediction Prediction

CMU Remos API

ResourcePredictionsExec Time Model

•pipeline stage•quality params

deadline

13

Active Frame Execution ModelActive Frame

Host LoadMeasurement

NetworkMeasurementRemos

Measurement Infrastructure

Mapping Algorithm

Prediction Prediction

CMU Remos API

ResourcePredictionsExec Time Model

•pipeline stage•quality params

deadline

14

Active Frame Execution ModelActive Frame

Host LoadMeasurement

NetworkMeasurementRemos

Measurement Infrastructure

Mapping Algorithm

Prediction Prediction

CMU Remos API

ResourcePredictionsExec Time Model

•pipeline stage•quality params

deadline

15

Feasibility of Best-effort Mapping Algorithms

0

10

20

30

40

50

60

70

80

90

100

0.01 0.1 1 10 100tnominal (seconds)

Random

Best Individual Host

RangeCounter(50)

Optimal (RC)

16

Active Frame Execution ModelActive Frame

Host LoadMeasurement

NetworkMeasurementRemos

Measurement Infrastructure

Mapping Algorithm

Prediction Prediction

CMU Remos API

ResourcePredictionsExec Time Model

•pipeline stage•quality params

deadline

17

Feasibility of Execution Time Models

1 3 5 7Measured Load

0

5

10

15

20

25E

xecu

tion

TIm

e (S

econ

ds)

42,000 pointsCoefficient of Correlation = 0.998

nominal

tt

t

tdttload

execnow

now

)(1

1

18

Active Frame Execution ModelActive Frame

Host LoadMeasurement

NetworkMeasurementRemos

Measurement Infrastructure

Mapping Algorithm

Prediction Prediction

CMU Remos API

ResourcePredictionsExec Time Model

•pipeline stage•quality params

deadline

19

Why Is Prediction Important?Bad Prediction

No obvious choiceGood PredictionTwo good choices

Pre

dict

ed E

xec

Tim

e

Good predictions result in smaller confidence intervals

Smaller confidence intervals simplify mapping decision

Pre

dict

ed E

xec

Tim

e

deadline

20

Feasibility of Host Load Prediction

1 2 3 4 5 6 7 8 9 10 11 12 13 14 150

0.5

1

1.5

2

2.5

3

3.5

4

4.5

Lead (seconds)

testcase 1681847547 from unix16 trace from all4.txt - resampled

By signal variance

By AR(9)

21

Comparing Prediction Models

Good models achieve consistently low error

Mea

n S

quar

ed E

rror

Model A Model B Model C

Inconsistentlow error

Consistent low error

Consistent high error

Run 1000s of randomized testcases, measure prediction error for each, datamine results:

2.5%

25%

50%

Mean

75%

97.5%

22

Comparing Linear Models for Host Load Prediction15 second predictions for one host

Title:axp0_lead15_8to8.epsCreator:MATLAB, The Mathworks, Inc.Preview:This EPS picture was not savedwith a preview included in it.Comment:This EPS picture will print to aPostScript printer, but not toother types of printers.

2.5%

25%

50%

Mean

75%

97.5%

Raw Cheap Expensive Very $

23

Conclusions

• Identified and described class of applications that benefit from best-effort real-time

• Distributed interactive applications• Example: QuakeViz / DV

• Showed feasibility of prediction-based best-effort real-time systems

• Mapping algorithms, execution time model, host load prediction

24

Status - http://www.cs.cmu.edu/~cmcl• QuakeViz / DV

• Overview: PDPTA'99, Aeschlimann, et al• http://www.cs.cmu.edu/~quake • Currently under construction

• Remos• Overview: HPDC’98, DeWitt, et al• Available from http://www.cs.cmu.edu/~cmcl/remulac/remos.html• Integrating prediction services

• Network measurement and analysis• HPDC’98, DeWitt, et al; HPDC’99, Lowekamp, et al• Currently studying network prediction

• Host load measurement and analysis• LCR’98, Dinda; SciProg’99, Dinda

• Host load prediction• HPDC’99, Dinda, et al

25

Feasibility of Best-effort Mapping Algorithms

0

10

20

30

40

50

60

70

80

90

100

1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3tmax/tnominal

Random

Best Individual Host

RangeCounter(50)

Optimal(RC)

26

Feasibility of Host Load Prediction

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 250

0.5

1

1.5

2

2.5

Lead (seconds)

testcase -1619784968 from axpfea.psc trace

By signal variance

By AR(18)

27

Comparing Linear Models for Host Load Prediction

Title:all_lead15_8to8.epsCreator:MATLAB, The Mathworks, Inc.Preview:This EPS picture was not savedwith a preview included in it.Comment:This EPS picture will print to aPostScript printer, but not toother types of printers.

15 second predictions aggregated over 38 hosts

2.5%

25%

50%

Mean

75%

97.5%

Raw Cheap Expensive Very $