Top Banner
NIMBUS www.nimbusproject.org Evalua&ng Streaming Strategies for Event Processing across Infrastructure Clouds (joint work) Radu Tudoran, Gabriel Antoniu (INRIA, University of Rennes) Kate Keahey, Pierre Riteau (ANL, University of Chicago) Sergey Panitkin (Brookhaven National Laboratory) Presented by Kate Keahey 1 12/1/13
17

Evalua&ng)Streaming)Strategies)) for)Event)Processing ... · Evalua&ng)Streaming)Strategies)) for)Event)Processing)across)) Infrastructure)Clouds (joint)work)) Radu Tudoran, Gabriel

Jan 05, 2020

Download

Documents

dariahiddleston
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Evalua&ng)Streaming)Strategies)) for)Event)Processing ... · Evalua&ng)Streaming)Strategies)) for)Event)Processing)across)) Infrastructure)Clouds (joint)work)) Radu Tudoran, Gabriel

NIMBUS www.nimbusproject.org  

Evalua&ng  Streaming  Strategies    for  Event  Processing  across    

Infrastructure  Clouds  (joint  work)  

Radu Tudoran, Gabriel Antoniu (INRIA, University of Rennes) Kate Keahey, Pierre Riteau (ANL, University of Chicago)

Sergey Panitkin (Brookhaven National Laboratory) Presented by Kate Keahey

1  12/1/13  

Page 2: Evalua&ng)Streaming)Strategies)) for)Event)Processing ... · Evalua&ng)Streaming)Strategies)) for)Event)Processing)across)) Infrastructure)Clouds (joint)work)) Radu Tudoran, Gabriel

NIMBUS www.nimbusproject.org  

Cloud  versus  Cloud  

12/1/13   2  

Custom  user  environments!  

On-­‐demand  access!  

Elas5c  compu5ng!  

Isola5on!  

Capital  expense  -­‐>  opera5onal  expense!  

Too  complex:  do  I  need  to  become  a  sys  admin?  

What  is  the  best  programming  model,  what  are  the  tools  I  need  to  make  effec5ve  use  of  them?    

It  costs  too  much!  And  what  if  Amazon  raises  prices?  

Performance:  especially  I/O,  especially  Big  Data!  

Page 3: Evalua&ng)Streaming)Strategies)) for)Event)Processing ... · Evalua&ng)Streaming)Strategies)) for)Event)Processing)across)) Infrastructure)Clouds (joint)work)) Radu Tudoran, Gabriel

NIMBUS www.nimbusproject.org  

Cloud  Storage  Basics  

•  Ephemeral/Transient Storage –  Local virtual disk attached to an instance –  Persists only for the lifetime of an instance –  Included in the cost of an instance –  Varying capacity, e.g.,160 GB-48 TB on AWS

•  Persistent attached storage –  Block storage volumes that can be attached to an instance –  Lifetime independent of a particular instance, can be mounted by many –  Price based on space and time used –  E.g., AWS Elastic Block Storage (EBS), Azure drives

•  Storage Clouds –  Data storage as binary objects (BLOBs) –  Price differs based levels of service, e.g., access time or reliability,

space used and time –  E.g., AWS Simple Storage Service (S3), AWS Glacier, Azure BLOBs,

Google Cloud Storage

12/1/13   3  

Page 4: Evalua&ng)Streaming)Strategies)) for)Event)Processing ... · Evalua&ng)Streaming)Strategies)) for)Event)Processing)across)) Infrastructure)Clouds (joint)work)) Radu Tudoran, Gabriel

NIMBUS www.nimbusproject.org  

Streaming  Applica&ons  •  Repeatedly apply an

operation to a stream of data (time events)

•  Examples: –  Virtual Observatories: OOI,

Forest project at ANL, IFC –  Experiment processing: STAR,

APS •  Requirements:

–  An “always-on” service –  Real-time event-based data

stream processing capabilities –  Highly volatile need for data

distribution and processing

12/1/13   4  

Page 5: Evalua&ng)Streaming)Strategies)) for)Event)Processing ... · Evalua&ng)Streaming)Strategies)) for)Event)Processing)across)) Infrastructure)Clouds (joint)work)) Radu Tudoran, Gabriel

NIMBUS www.nimbusproject.org  

ATLAS  Data  Analysis  

12/1/13   5  

Data  analysis  searches  in  a  channel  where    the  Higgs  decays  into  t-­‐an4-­‐t  quarks  

Collected  as  successive  4me  events,  each  event    corresponding  to  the  aggregated  readings    from  the  ATLAS  sensors  at  a  given  moment  

Size:  ~10s  of  PBs  

Page 6: Evalua&ng)Streaming)Strategies)) for)Event)Processing ... · Evalua&ng)Streaming)Strategies)) for)Event)Processing)across)) Infrastructure)Clouds (joint)work)) Radu Tudoran, Gabriel

NIMBUS www.nimbusproject.org  

Streaming  Scenarios  

12/1/13   6  

Page 7: Evalua&ng)Streaming)Strategies)) for)Event)Processing ... · Evalua&ng)Streaming)Strategies)) for)Event)Processing)across)) Infrastructure)Clouds (joint)work)) Radu Tudoran, Gabriel

NIMBUS www.nimbusproject.org  

Streaming  Scenarios  (2)  

Stream&Compute (SC) •  Simpler model with fewer

moving parts •  Potentially better

response time •  Overlap computation and

communication (potentially faster)

•  Uses ephemeral storage (potentially cheaper)

Copy&Compute (CC) •  Independent of network

saturation •  Persistent storage: less

liable to data loss

12/1/13   7  

Page 8: Evalua&ng)Streaming)Strategies)) for)Event)Processing ... · Evalua&ng)Streaming)Strategies)) for)Event)Processing)across)) Infrastructure)Clouds (joint)work)) Radu Tudoran, Gabriel

NIMBUS www.nimbusproject.org  

Experimental  Configura&on  

12/1/13   8  

•  Compute rate: events processed per time unit •  Data rate: amount of data acquired per time unit

Page 9: Evalua&ng)Streaming)Strategies)) for)Event)Processing ... · Evalua&ng)Streaming)Strategies)) for)Event)Processing)across)) Infrastructure)Clouds (joint)work)) Radu Tudoran, Gabriel

NIMBUS www.nimbusproject.org  

SC  versus  CC  (FutureGrid)  

12/1/13   9  

CC  outperforms  SC  by  almost  4  5mes  in  both    compute  rates  and  data  rates!  

Page 10: Evalua&ng)Streaming)Strategies)) for)Event)Processing ... · Evalua&ng)Streaming)Strategies)) for)Event)Processing)across)) Infrastructure)Clouds (joint)work)) Radu Tudoran, Gabriel

NIMBUS www.nimbusproject.org  

SC  versus  CC  (Azure)  

12/1/13   10  

Why?  

Page 11: Evalua&ng)Streaming)Strategies)) for)Event)Processing ... · Evalua&ng)Streaming)Strategies)) for)Event)Processing)across)) Infrastructure)Clouds (joint)work)) Radu Tudoran, Gabriel

NIMBUS www.nimbusproject.org  

Data  Throughput  vs  CPU  Load  

12/1/13   11  

Page 12: Evalua&ng)Streaming)Strategies)) for)Event)Processing ... · Evalua&ng)Streaming)Strategies)) for)Event)Processing)across)) Infrastructure)Clouds (joint)work)) Radu Tudoran, Gabriel

NIMBUS www.nimbusproject.org  

Mul&-­‐Core  and  Stream&Compute  

12/1/13   12  

What  is  the  impact  of  increasing  the  number  of  cores  in  instances  on  Stream&Compute?  

Page 13: Evalua&ng)Streaming)Strategies)) for)Event)Processing ... · Evalua&ng)Streaming)Strategies)) for)Event)Processing)across)) Infrastructure)Clouds (joint)work)) Radu Tudoran, Gabriel

NIMBUS www.nimbusproject.org  

Scalability  for  Stream&Compute  

12/1/13   13  

Page 14: Evalua&ng)Streaming)Strategies)) for)Event)Processing ... · Evalua&ng)Streaming)Strategies)) for)Event)Processing)across)) Infrastructure)Clouds (joint)work)) Radu Tudoran, Gabriel

NIMBUS www.nimbusproject.org  

Scaling  Data  Sources  

12/1/13   14  

Page 15: Evalua&ng)Streaming)Strategies)) for)Event)Processing ... · Evalua&ng)Streaming)Strategies)) for)Event)Processing)across)) Infrastructure)Clouds (joint)work)) Radu Tudoran, Gabriel

NIMBUS www.nimbusproject.org  

Cost  

12/1/13   15  

•  Cost of instance: ~$0.1 per hour •  Cost of storage: ~$0.1 per 1GB month •  In our case (320M events & 5 GB attached storage)

–  Stream&Compute: $1.33 –  Copy&Compute: $0.48 –  Overall: SC is 2.77 times more expensive

Page 16: Evalua&ng)Streaming)Strategies)) for)Event)Processing ... · Evalua&ng)Streaming)Strategies)) for)Event)Processing)across)) Infrastructure)Clouds (joint)work)) Radu Tudoran, Gabriel

NIMBUS www.nimbusproject.org  

Related  Work  •  Data management strategies for large unstructured sets of static

data – we focus on dynamic time events –  I/O Performance of Virtualized Cloud Environments, Ghoshal et al., DataCloud-SC ’11 –  A Survey of Large Scale Data Management Approaches in Cloud Environments, S. Sakr et al.

IEEE Communications Surveys and Tutorials

•  Performance evaluations about data analysis in the clouds focus on the MapReduce processing paradigm - we focus on the stream processing model –  On the Performance and Energy Efficiency of Hadoop Deployment Models, E. Feller et al., IEEE

BigData 2013 –  Evaluating Hadoop for Data-Intensive Scientific Operations. Z. Fadika et al. CLOUD ’12

•  Stream processing studies – we focus on multi-site processing –  GeoStreaming in Cloud, S. J. Kazemitabar et al. 2011 –  Scheduling processing of real-time data streams on heterogeneous multi-GPU systems, U.

Verner et al., SYSTOR ’12

12/1/13   16  

Page 17: Evalua&ng)Streaming)Strategies)) for)Event)Processing ... · Evalua&ng)Streaming)Strategies)) for)Event)Processing)across)) Infrastructure)Clouds (joint)work)) Radu Tudoran, Gabriel

NIMBUS www.nimbusproject.org  

Conclusions  

•  To stream or not to stream? – Not to stream! – Difference of ~4x in performance and ~3x in cost

•  Amplification of virtualization performance trade-offs in the presence of remote traffic

•  Hypervisor design – Need for controlled allocation of CPU to I/O

processing •  Paper: Tudoran et al., “Evaluating Streaming

Strategies for Event Processing across Infrastructure Clouds”, submitted to CCGrid

12/1/13   17  


Related Documents
Instrucciones Evalua

Instrucciones Evalua

Category: Documents
Laminas Evalua 0

Laminas Evalua 0

Category: Documents
Evalua&ng)Evidence) - University of California, Berkeleystage.cchem.berkeley.edu/~bcgc/sites/default/files/Class 20.pdf · Evalua&ng)Evidence) Making&Decisions&in&Uncertain&Condi?ons&

Evalua&ng)Evidence) - University of California,...

Category: Documents
Evalua&ng the MJO forecast skill in the NCEP GEFS 35-day … · 2018-04-13 · Evalua&ng the MJO forecast skill in the NCEP GEFS 35-day Experiments Wei Li, Yuejian Zhu, Xiaqiong Zhou,

Evalua&ng the MJO forecast skill in the NCEP GEFS 35-day...

Category: Documents
Manual Evalua 2

Manual Evalua 2

Category: Documents
Evalua&ng Farms for Pollinator and Beneficial Insect Habitat

Evalua&ng Farms for Pollinator and Beneficial Insect...

Category: Documents
Original Research Evalua ng Experimental Design of ERT for ... et al. 2012.pdf · Evalua ng Experimental Design of ERT for Soil Moisture Monitoring in Contour Hedgerow Intercropping

Original Research Evalua ng Experimental Design of ERT for.....

Category: Documents
Que Evalua El Evalua

Que Evalua El Evalua

Category: Documents
Evalua&ng)Ammonium,)Nitrate)and)Sulfate)Aerosols… · Evalua&ng)Ammonium,)Nitrate)and)Sulfate)Aerosols)in)38Dimensions)) References))) Setup: • 2°x2.5 ...

Evalua&ng)Ammonium,)Nitrate)and)Sulfate)Aerosols… ·...

Category: Documents
Manual Evalua 7

Manual Evalua 7

Category: Documents
EVALUATION OF RESEARCH · evalua on of research in energe cs” (hereina L er Steering Group); − to visit the ins tu ons par cipa ng in the evalua on for the purpose of obtaining

EVALUATION OF RESEARCH · evalua on of research in energe.....

Category: Documents
cuadernillo evalua

cuadernillo evalua

Category: Documents