1 An Overview of Cloud Computing @ Yahoo! Raghu Ramakrishnan Chief Scientist, Audience and Cloud Computing Research Fellow, Yahoo! Research Reflects many discussions with: Eric Baldeschwieler, Jay Kistler, Chuck Neerdaels, Shelton Shugar, and Raymie Stata and joint work with the Sherpa team, in particular: Brian Cooper, Utkarsh Srivastava, Adam Silberstein, Rodrigo Fonseca and Nick Puz in Y! Chuck Neerdaels, P.P. Suryanarayanan and many others in CCDI
An Overview of Cloud Computing @ Yahoo! Raghu Ramakrishnan Chief Scientist, Audience and Cloud Computing Research Fellow, Yahoo! Research. Reflects many discussions with: Eric Baldeschwieler, Jay Kistler, Chuck Neerdaels, Shelton Shugar, and Raymie Stata - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
An Overview of Cloud Computing @ Yahoo!
Raghu RamakrishnanChief Scientist, Audience and Cloud Computing
Research Fellow, Yahoo! Research
Reflects many discussions with: Eric Baldeschwieler, Jay Kistler, Chuck Neerdaels, Shelton Shugar, and Raymie Stata
and joint work with the Sherpa team, in particular:Brian Cooper, Utkarsh Srivastava, Adam Silberstein, Rodrigo Fonseca and Nick Puz in Y! ResearchChuck Neerdaels, P.P. Suryanarayanan and many others in CCDI
2
Questions
• What is cloud computing?– Horizontal and functional services
• What’s it going to change?– Software business models, science, life
• How many clouds will there be?– 1, 2, 3, infinity
• What’s new in cloud computing?– HPC grids, ASPs, hosted services, Multics (!)– Emerging “cloud stack” to support a broad class of
programs, including data intensive applications
3
SCENARIOSPie-in-the-sky
4
Living in the Clouds
• We want to start a new website, FredsList.com• Our site will provide listings of items for sale, jobs,
etc.• As time goes on, we’ll add more features
– And illustrate how more cloud capabilities (and corresponding infrastructure components) are used as needed
• List of capabilities/components is illustrative, not exhaustive
• Our cloud provides a “dataset” abstraction– FredsList doesn’t worry about the underlying components
• Very different workloads, requirements• Data from serving system is one of many
kinds of data (click streams are another common kind, as are syndicated feeds) to be analyzed and integrated
• The result of analysis often goes right back into serving system
12
EYES TO THE SKIESMotherhood-and-Apple-Pie
13
Why Clouds?
• On-demand infrastructure to create a fundamental shift in the OE curve:
– Do things we can’t do
– Build more robustly, more efficiently, more globally, more completely, more quickly, for a given budget
• Cloud services should do heavy lifting of heavy-lifting of scaling & high-availability– Today, this is done at the
app-level, which is not productive
14
Requirements for Cloud Services
• Multitenant. A cloud service must support multiple, organizationally distant customers.
• Elasticity. Tenants should be able to negotiate and receive resources/QoS on-demand.
• Resource Sharing. Ideally, spare cloud resources should be transparently applied when a tenant’s negotiated QoS is insufficient, e.g., due to spikes.
• Horizontal scaling. It should be possible to add cloud capacity in small increments; this should be transparent to the tenants of the service.
• Metering. A cloud service must support accounting that reasonably ascribes operational and capital expenditures to each of the tenants of the service.
• Security. A cloud service should be secure in that tenants are not made vulnerable because of loopholes in the cloud.
• Availability. A cloud service should be highly available.• Operability. A cloud service should be easy to operate, with few
operators. Operating costs should scale linearly or better with the capacity of the service.
15
Types of Cloud Services
• Two kinds of cloud services:– Horizontal (“Platform”) Cloud Services
• Functionality enabling tenants to build applications or new services on top of the cloud
– Functional Cloud Services • Functionality that is useful in and of itself to tenants. E.g., various
SaaS instances, such as Saleforce.com; Google Analytics and Yahoo!’s IndexTools; Yahoo! properties aimed at end-users and small businesses, e.g., flickr, Groups, Mail, News, Shopping
• Could be built on top of horizontal cloud services or from scratch• Yahoo! has been offering these for a long while (e.g., Mail for
SMB, Groups, Flickr, BOSS, Ad exchanges)
16
Opening Up Yahoo! Search
Phase 1 Phase 2
Giving site owners and developers control over the appearance of Yahoo!
Search results.
BOSS takes Yahoo!’s open strategy to the next level by providing Yahoo!
Search infrastructure and technology to developers and companies to help them
build their own search experiences.
18
BOSS Offerings
API
A self-service, web services model for developers and start-ups to quickly build and deploy new search experiences.
BOSS offers two options for companies and developers and has partnered with top technology universities to drive search experimentation, innovation and research into next generation search.
• University of Illinois Urbana Champaign• Carnegie Mellon University
• Stanford University
• Purdue University
• MIT
• Indian Institute of
Technology Bombay
• University of
Massachusetts
CUSTOM
Working with 3rd parties to build a more relevant, brand/site specific web search experience.
This option is jointly built by Yahoo! and select partners.
ACADEMIC
Working with the following universities to allow for wide-scale research in the search field:
(Slide courtesy Prabhakar Raghavan)
19
Partner Examples
20
Horizontal Cloud Services
• Horizontal cloud services are foundations on which tenants build applications or new services. They should be:– Semantics-free. Must be "generic infrastructure,” and not tied to
specific app-logic. • May provide the ability to inject application logic through well-defined
APIs
– Broadly applicable. Must be broadly applicable (i.e., it can't be intended for just one or two properties).
– Fault-tolerant over commodity hardware. Must be built using inexpensive commodity hardware, and should mask component failures.
• While each cloud service provides value, the power of the cloud paradigm will depend on a collection of well-chosen, loosely coupled services that collectively make it easy to quickly develop and operate innovative web applications.
22
Yahoo! Cloud StackPr
ovis
ioni
ng (
Self-
serv
e)
Horizontal Cloud Services …YCS YCPI Brooklyn
EDGEM
onito
ring/
Met
erin
g/Se
curit
y
Horizontal Cloud Services…Hadoop
BATCH
Horizontal Cloud Services…Sherpa MOBStor
STORAGE
Horizontal Cloud ServicesVM/OS …
APP
Horizontal Cloud ServicesVM/OS yApache
WEB
Dat
a H
ighw
ay
Serving Grid
PHP App Engine
23
Yahoo! CCDI Thrust Areas
• Fast Provisioning and Machine Virtualization: On demand, deliver a set of hosts imaged with desired software and configured against standard services– Multiple hosts may be multiplexed onto the same physical
machine.
• Batch Storage and Processing: Scalable data storage optimized for batch processing, together with computational capabilities
• Operational Storage: Persistent storage that supports low-latency updates and flexible retrieval
• Edge Content Services: Support for dealing with network topology, communication protocols, caching, and BCP
Rest of today’s talk
24
Web Data Management
Large data analysis(Hadoop)
Structured record storage
(PNUTS/Sherpa)
Blob storage(SAN/NAS)
• Scan oriented workloads
• Focus on sequential disk I/O
• $ per cpu cycle
• CRUD • Point lookups
and short scans
• Index organized table and random I/Os
• $ per latency
• Object retrieval and streaming
• Scalable file storage
• $ per GB
25
[Workflow][Workflow]
Hadoop: Batch Storage/Analysis
Why is batch processing important?
• Whether it’s – response-prediction for advertising– machine-learned relevance for Search, or– content optimization for audience, – data-intensive computing is increasingly
central to everything Yahoo! does– Hadoop is central to addressing this need
• Hadoop is a case-study in our cloud vision– Processes enormous amounts of data– Provides horizontal scaling and fault-
tolerance for our users– Allows those users to focus on their app
logic
HDFSHDFS
Map-ReduceMap-Reduce
High-level query layer (Pig)
High-level query layer (Pig)
26
The World Has Changed
• Web serving applications need:– Scalability!
• Preferably elastic
– Flexible schemas– Geographic distribution– High availability– Reliable storage
• Web serving applications can do without:– Complicated queries– Strong transactions
2727
MObStor
• Yahoo!’s next-generation globally replicated, virtualized media object storage service
• Better provisioning, easy migration, replication, better BCP, and performance
• New features (Evergreen URLs, CDN integration, REST API, …)
• The object metadata problem addressed using Sherpa, though MObStor is focused on blob storage.
28
Storage & Delivery Stack
29
PNUTS /
SHERPA
To Help You Scale Your Mountains of Data
30
CCDI—Research Collaboration
Yahoo! Research
• Raghu Ramakrishnan • Brian Cooper• Utkarsh Srivastava• Adam Silberstein• Rodrigo Fonseca
CCDI
• Chuck Neerdaels • P.P.S. Narayan • Kevin Athey• Toby Negrin• Plus Dev/QA teams
31
Yahoo! Serving Storage Problem
– Small records – 100KB or less
– Structured records – lots of fields, evolving
– Extreme data scale - Tens of TB
– Extreme request scale - Tens of thousands of requests/sec
• Per-record mastering– Each record is assigned a “master region”
• May differ between records
– Updates to the record forwarded to the master region– Ensures consistent ordering of updates
• Tablet-level mastering– Each tablet is assigned a “master region”– Inserts and deletes of records forwarded to the master region– Master region decides tablet splits
• These details are hidden from the application– Except for the latency impact!
5959
Mastering
A 42342 EB 42521 W
C 66354 W
D 12352 EE 75656 C
F 15677 E A 42342 EB 42521 W
C 66354 W
D 12352 EE 75656 C
F 15677 EA 42342 EB 42521 W
C 66354 W
D 12352 EE 75656 C
F 15677 E
Tablet master
60
Bulk Insert/Update/Replace
Client
Source Data
Bulk manager
1. Client feeds records to bulk manager
2. Bulk loader transfers records to SU’s in batches• Bypass routers and
message brokers• Efficient import into
storage unit
61
Bulk Load in YDOT
• YDOT bulk inserts can cause performance hotspots
• Solution: preallocate tablets
62
Index Maintenance
• How to have lots of interesting indexes and views, without killing performance?
• Solution: Asynchrony!– Indexes/views updated asynchronously when
base table updated
63
SHERPAIN CONTEXT
63
64
Types of Record Stores
• Query expressiveness
Simple Feature rich
Object retrieval
Retrieval from single table of
objects/records
SQL
S3 PNUTS Oracle
65
Types of Record Stores
• Consistency model
Best effort Strong guaranteesEventual
consistencyTimeline
consistencyACID
S3 PNUTS Oracle
Program centric
consistency
Program centric
consistencyObject-centric consistency
Object-centric consistency
66
Types of Record Stores
• Data model
Flexibility,Schema evolution
Optimized forFixed schemas
CouchDB
PNUTS
Oracle
Consistency spans objectsConsistency
spans objectsObject-centric consistency
Object-centric consistency
67
Types of Record Stores
• Elasticity (ability to add resources on demand)
Inelastic Elastic
Limited (via data
distribution)
VLSD(Very Large
Scale Distribution /Replication)
OraclePNUTS
S3
68
Data Stores Comparison
• User-partitioned SQL stores– Microsoft Azure SDS– Amazon SimpleDB
• Multi-tenant application databases– Salesforce.com– Oracle on Demand
• Mutable object stores– Amazon S3
Versus PNUTS
• More expressive queries• Users must control partitioning• Limited elasticity
• Highly optimized for complex workloads
• Limited flexibility to evolving applications
• Inherit limitations of underlying data management system
• Object storage versus record management
69
Application Design Space
Records Files
Get a few things
Scan everything
Sherpa MObStor
Everest Hadoop
YMDBMySQL
Filer
Oracle
BigTable
69
70
Alternatives Matrix
Ela
stic
Ope
rabi
lity
Glo
bal l
ow
late
ncy
Ava
ilab
ilit
y
Stru
ctur
ed
acce
ss
Sherpa
Y! UDB
MySQL
Oracle
HDFS
BigTable
DynamoU
pdat
esCassandra
Con
sist
ency
m
odel
SQL
/AC
ID
70
71
Further Reading
Efficient Bulk Insertion into a Distributed Ordered Table (SIGMOD 2008)Adam Silberstein, Brian Cooper, Utkarsh Srivastava, Erik Vee, Ramana Yerneni, Raghu Ramakrishnan
PNUTS: Yahoo!'s Hosted Data Serving Platform (VLDB 2008)Brian Cooper, Raghu Ramakrishnan, Utkarsh Srivastava, Adam Silberstein, Phil Bohannon, Hans-Arno Jacobsen, Nick Puz, Daniel Weaver, Ramana Yerneni
Asynchronous View Maintenance for VLSD Databases,Parag Agrawal, Adam Silberstein, Brian F. Cooper, Utkarsh Srivastava and Raghu RamakrishnanSIGMOD 2009 (to appear)
Cloud Storage Design in a PNUTShellBrian F. Cooper, Raghu Ramakrishnan, and Utkarsh SrivastavaBeautiful Data, O’Reilly Media, 2009 (to appear)