Page 1
AN OPEN-SOURCE BENCHMARK SUITE FOR MICROSERVICES ANDTHEIR HARDWARE-SOFTWARE IMPLICATIONS FOR CLOUD AND
EDGE SYSTEMS
Yu Gan, Yanqi Zhang, Dailun Cheng, Ankitha Shetty, Priyal Rathi, Nayantara Katarki, Ariana Bruno, Justin Hu, Brian Ritchken, Brendon Jackson, Kelvin Hu, Meghna Pancholi, Yuan He, Brett Clancy, Chris Colen, Fukang Wen, Catherine Leung, Siyuan Wang, Leon
Zaruvinsky, Mateo Espinosa, Rick Lin, Zhongling Liu, Jake Padilla and Christina Delimitrou
Cornell University
ASPLOS 2019
Session Cloud I
Page 2
EXECUTIVE SUMMARY
Cloud applications migrating from monoliths to microservices• Monoliths: all functionality in a single service
• Microservices: many single-concerned, loosely-coupled services
• Modularity, specialization, faster development
• Datacenters designed for monoliths microservices have different requirements
An end-to-end benchmark suite for large-scale microservices
Architectural and system implications• Hardware design
• OS/networking overheads
• Cluster management
• Application & programming frameworks
• Tail at scale
1
Page 3
FROM MONOLITHS TO MICROSERVICES
Monolithic applications• Single binary with entire business logic
Limitations• Too complex for continuous development
• Obstacle to adopting new frameworks
• Poor scalability & elasticity
2
Monolith Application
login
orders
payments
shipping
Page 4
FROM MONOLITHS TO MICROSERVICES
Microservices• Fine-grained, loosely-coupled, and single-
concerned
• Communicate with RPCs or RESTful APIs
Pros• Agile development
• Better modularity & elasticity
• Testing and debugging in isolation
Cons• Different hardware & software constraints
• Dependencies complicate cluster management
3
loginorders payments
shipping
Page 5
FROM MONOLITHS TO MICROSERVICES
4
Page 6
MOTIVATION
Explore implications of microservices across the system stack
5
1. Hardware design
2. OS/Network overheads
3. Cluster management
4.
1. Hardware design
2. OS/Network overheads
3. Cluster management
4. Application and frameworks
5. Tail at scale
Page 7
MOTIVATION
Explore implications of microservices across the system stack
6
1. Hardware design
2. OS/Network overheads
3. Cluster management
4.
1. Hardware design
2. OS/Network overheads
3. Cluster management
4. Application and frameworks
5. Tail at scale
Need representative, end-to-end applicationsbuilt with microservices
Page 8
MOTIVATION
Previous work in cloud benchmarking• CloudSuite [ASPLOS’12]
• Sirius [ASPLOS’15]
• TailBench [IISWC’17]
• μSuite [IISWC’18]
DeathStarBench suite• Focus on large-scale microservices that stress typical datacenter design
7
Focus either on monolithic applications or applications with few tiers
Page 9
DEATHSTARBENCH SUITE
Design principles• Representativeness
» Use of popular open-source applications and frameworks
» Service architecture following public documentation of real systems using microservices
8
Page 10
DEATHSTARBENCH SUITE
Design principles• Representativeness
• End-to-end operation» Full functionality using microservices
9
Page 11
DEATHSTARBENCH SUITE
Design principles• Representativeness
• End-to-end operation
• Heterogeneity» Wide range of programming languages and microservices frameworks
10
Page 12
DEATHSTARBENCH SUITE
Design principles• Representativeness
• End-to-end operation
• Heterogeneity
• Modularity» Single-concerned and loosely-coupled services
11
Page 13
DEATHSTARBENCH SUITE
Design principles• Representativeness
• End-to-end operation
• Heterogeneity
• Modularity
• Reconfigurability» Easy to update or change components with minimal effort
12
Page 14
DEATHSTARBENCH SUITE
5 end-to-end applications, tens of unique microservices each• Social Network
• Media Service
• E-Commerce Service
• Banking System
• Drone Coordination System
13
Page 15
DEATHSTARBENCH SUITE
Social network
14
Page 16
DEATHSTARBENCH SUITE
Media service
15
Page 17
DEATHSTARBENCH SUITE
E-commerce service
16
Page 18
DEATHSTARBENCH SUITE
Banking system
17
Page 19
DEATHSTARBENCH SUITE
Drone coordination system
18
ClientLoad
BalancerNGINX Controller
Construct
Route
TargetDB
ImageDB
VideoDB
LocationDB
SpeedDB
LuminosityDB
OrientationDB
Image
Video
Location
Speed
Luminosity
Orientation
Stocking
ImageDB
MotionCtrl
Image
Recognition
Obstacle
Avoidance
Controller
Log(node.js)
Edge Router
Edge Swarm
Frontend EdgeCloud
Page 20
MongoDB
MongoDB
MongoDB
MongoDB
MongoDB
MongoDB
Memcached
Redis
Memcached
Memcached
Memcached
Memcached
Memcached
RabbitMQ
Read
Post
User
Timeline
Write Home
Timeline
Unique ID
URL Shorten
Image
Video
Text
User Tag
Favorite
Search
Recommender
User
Compose
Post
Post
StorageRead Home
Timeline
Social
Graph
Load
BalancerNGINX
Video Store
Frontend
Image Store
Frontend
Client
Video storage
Image storage
Social graph
storage
Home timeline
storage
User timeline
storage
Post storage
User storage
Frontend Logic Caching & Storage
Index1IndexnIndex0
CASE STUDY: SOCIAL NETWORK
User sign up/login
19
Page 21
CASE STUDY: SOCIAL NETWORK
Write posts
20
ClientLoad
BalancerNGINX
Unique ID
URL Shorten
Image
Video
Text
User Tag
Favorite
Search
Recommender
User
Compose
Post
Post
StorageRead Home
Timeline
Social
Graph
Memcached MongoDB
Redis
Memcached MongoDB
Read
PostMemcached MongoDB
Memcached MongoDBUser
Timeline
RabbitMQ
Write Home
Timeline
Frontend Logic Caching & Storage
Memcached MongoDB
Memcached MongoDB
Video storage
Image storage
Social graph
storage
Home timeline
storage
User timeline
storage
Post storage
User storage
Video Store
Frontend
Image Store
Frontend Index1IndexnIndex0
Page 22
CASE STUDY: SOCIAL NETWORK
Read home timeline
21
MongoDB
MongoDB
MongoDB
MongoDB
MongoDB
MongoDB
Memcached
Redis
Memcached
Memcached
Memcached
Memcached
Memcached
RabbitMQ
Read
Post
User
Timeline
Write Home
Timeline
Unique ID
URL Shorten
Image
Video
Text
User Tag
Favorite
Search
Recommender
User
Compose
Post
Post
StorageRead Home
Timeline
Social
Graph
Load
BalancerNGINX
Video Store
Frontend
Image Store
Frontend
Client
Video storage
Image storage
Social graph
storage
Home timeline
storage
User timeline
storage
Post storage
User storage
Frontend Logic Caching & Storage
Index1IndexnIndex0
Page 23
CASE STUDY: SOCIAL NETWORK
Search
22
Index1IndexnIndex0
MongoDB
MongoDB
MongoDB
MongoDB
MongoDB
MongoDB
Memcached
Redis
Memcached
Memcached
Memcached
Memcached
Memcached
RabbitMQ
Unique ID
URL Shorten
Image
Video
Text
User Tag
Favorite
Search
Recommender
User
Compose
Post
Post
StorageRead Home
Timeline
Social
Graph
Read
Post
User
Timeline
Write Home
Timeline
Load
BalancerNGINXClient
Frontend Logic Caching & Storage
Video storage
Image storage
Social graph
storage
Home timeline
storage
User timeline
storage
Post storage
User storage
Video Store
Frontend
Image Store
Frontend
Page 24
CASE STUDY: SOCIAL NETWORK
Recommendation
23
MongoDB
MongoDB
MongoDB
MongoDB
MongoDB
MongoDB
Memcached
Redis
Memcached
Memcached
Memcached
Memcached
Memcached
RabbitMQ
Unique ID
URL Shorten
Image
Video
Text
User Tag
Favorite
Search
Recommender
User
Compose
Post
Post
StorageRead Home
Timeline
Social
Graph
Read
Post
User
Timeline
Write Home
Timeline
Load
BalancerNGINXClient
Frontend Logic Caching & Storage
Video storage
Image storage
Social graph
storage
Home timeline
storage
User timeline
storage
Post storage
User storage
Video Store
Frontend
Image Store
Frontend Index1IndexnIndex0
Page 25
ARCHITECTURAL AND SYSTEM IMPLICATIONS
Explore implications of microservices across the system stack
24
1. Hardware design
2. OS/Network overheads
3. Cluster management
4.
1. Hardware design
2. OS/Network overheads
3. Cluster management
4. Application and frameworks
5. Tail at scale
Page 26
HARDWARE DESIGN
Brawny vs. wimpy cores• Microservices are more sensitive to performance
unpredictability than monoliths
25
Microservices Monoliths
1. Hardware design
2. OS/Network overheads
3. Cluster management
4. Application and frameworks
5. Tail at scale
Page 27
HARDWARE DESIGN
Brawny vs. wimpy cores• Microservices are more sensitive to performance
unpredictability than monoliths
26
Microservices
Monoliths
1. Hardware design
2. OS/Network overheads
3. Cluster management
4. Application and frameworks
5. Tail at scale
Page 28
HARDWARE DESIGN
Brawny vs. wimpy cores• Microservices are more sensitive to performance
unpredictability than monolithic apps
• Xeon vs Cavium servers
Cycle breakdown of each microservice• Smaller fraction of frontend stalls than monoliths
I-cache pressure• Lower I-cache pressure than monoliths
27
1. Hardware design
2. OS/Network overheads
3. Cluster management
4. Application and frameworks
5. Tail at scale
Page 29
OS/NETWORK OVERHEADS
RPC overheads• A large fraction of time spent in network stack
FPGA network acceleration• Offload TCP stack on FPGA
• 10 − 68x improvement on network processing latency
• 43% - 2.2x improvement on end-to-end latency
28
1. Hardware design
2. OS/Network overheads
3. Cluster management
4. Application and frameworks
5. Tail at scale
Page 30
MemcachedNGINX
CLUSTER MANAGEMENT
Latency back-pressure• Bottleneck services pressure upstreaming services
• Cause: Imperfect pipelining» HTTP/TCP HoL blocking
» Limited number of worker threads/connections
29
NGINX bottleneck NGINX Memcached
🔥
NGINX Memcached
🔥Memcached bottleneck
Example: HTTP 1.1 HoL blocking
1. Hardware design
2. OS/Network overheads
3. Cluster management
4. Application and frameworks
5. Tail at scale
Page 31
CLUSTER MANAGEMENT
Cascading QoS violations• Hotspots propagating along the dependency graph
• No obvious correlation to CPU utilization
• Difficulty in discovering the bottleneck and long time to recover from QoS violations
30
1. Hardware design
2. OS/Network overheads
3. Cluster management
4. Application and frameworks
5. Tail at scale
Page 32
APPLICATION AND FRAMEWORKS
Serverless frameworks• Compared long-running microservices on EC2 with short–
running microservices on AWS Lambda
• Agile resource adjustments with diurnal load pattern
• Higher performance variability due to
» No control of lambda placement
» Communication through S3
» Loading of dependencies
31
1. Hardware design
2. OS/Network overheads
3. Cluster management
4. Application and frameworks
5. Tail at scale
Page 33
TAIL AT SCALE
Impact of slow servers• Larger cluster larger impact of slow servers
• More severe tail latency increase compared to monoliths
32
1. Hardware design
2. OS/Network overheads
3. Cluster management
1. Hardware design
2. OS/Network overheads
3. Cluster management
4. Application and frameworks
5. Tail at Scale
Page 34
CONCLUSIONS
• Cloud applications from monoliths to microservices
• Study implications of microservices across the system stack
• Open-source benchmark suite for cloud and IoT microservices
• Explored the implications of microservices• More sensitive to performance unpredictability
• Potential of hardware acceleration for networking
• Need for cluster managers that account for dependencies
• Tail at scale effects more prominent in microservices
33
Page 35
QUESTIONS?• Cloud applications from monoliths to microservices
• Study implications of microservices across the system stack
• Open-source benchmark suite for cloud and IoT microservices
• Explored the implications of microservices• More sensitive to performance unpredictability
• Potential of hardware acceleration for networking
• Need for cluster managers that account for dependencies
• Tail at scale effects more prominent in microservices
34