Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University.

Automatic Configuration of Internet Services

Wei Zheng, Ricardo Bianchini, and Thu Nguyen

Department of Computer ScienceRutgers University

Motivation Downtime and management are expensive for services

• Downtime can cost millions of dollars per hour [IW00]

• Management can account for 50—80% of IT budgets [Brown05]

Operator mistakes are a major source of downtime

• Cause up to 36% of service failures [Oppenheimer03]

Misconfiguration is the most common type of mistake

• More than 50% of service failures [Oppenheimer03]

• 24 out of 42 mistakes in a multi-tier service [Nagaraja04]

Understanding Misconfigurations Misconfigurations occur frequently for two main

reasons:

• Servers and services are becoming increasingly complex

• Operators modify servers’ configs often, as service evolves via software and hardware upgrades, node additions and removals

Example: Apache

• One config file has 240+ lines relating to performance, structure, modules, and others

• Config files must change every time a node is added or removed in the 2nd tier

Our Work Idea: Automatically generate server config files as

the system and its workload evolve

Goals: Reduce config complexity and operator effort, and generate optimized config files efficiently

Contributions:

• Infrastructure for automatic configuration of services

• Heuristics for efficient parameter tuning

• Quantitative evaluation for a realistic service

Outline Motivation Approach and Contributions Automatic Configuration Infrastructure

(ACI) Tuning Parameters Evaluation Conclusions

Key Observations Behind ACI Most of each server’s config doesn’t change

Node additions and removals require tedious and mistake-prone reconfigurations

Changes to a config parameter affect only a few other parameters

Key Observations Behind ACI Most of each server’s config doesn’t change

• Config file templates and generation scripts

Node additions and removals require tedious and mistake-prone reconfigurations

Changes to a config parameter affect only a few other parameters

Key Observations Behind ACI Most of each server’s config doesn’t change

• Config file templates and generation scripts

Node additions and removals require tedious and mistake-prone reconfigurations

• Network of membership daemons

Changes to a config parameter affect only a few other parameters

Key Observations Behind ACI Most of each server’s config doesn’t change

• Config file templates and generation scripts

Node additions and removals require tedious and mistake-prone reconfigurations

• Network of membership daemons

Changes to a config parameter affect only a few other parameters

• Parameter dependency graph and parameter-tuning heuristics

Global View of ACI (3-tier Service)

Local View of ACI

Daemon

Scripts + templates

Config

files

Node i

Global membership info + tuned parameter values

Outline Motivation Approach and Contributions Automatic Configuration Infrastructure

(ACI) Tuning Parameters Evaluation Conclusions

Tuning Parameters Tune parameters for some metric, as service

evolves

Evolution may cause current config to behave poorly• Ex: when throughput is the metric, more memory may

allow more threads and higher throughput

Problem• Hard to know what parameters are affected• Hard to select new values for the parameters

Parameter Dependency Graph Idea: Constrain the search space with a parameter

dependency graph and value ranges

Vertex = parameter, directed edge = dependency

After each change, ACI traverses the graph to find the affected parameters and tune them experimentally• Ex: if Apache node is upgraded, make Apache

parameters “sources” and tune reachable parameters

Overhead of generating graph can be amortized

Generating the Graph 1st step: find important parameters using CART

2nd step: find dependencies between important parametersPair-wise test: parameter A depends on B iff different settings

of B lead to different best values for A; check min, mid, max values

0

1

2

3

4

5

6

7

1 3Parameter A

Met

ric R

espo

nse

Parameter B=1

Parameter B=3

0

1

2

3

4

5

6

7

1 3Parameter B

Met

ric R

espo

nse

Parameter A=1

Parameter A=3

Generating the Graph 1st step: find important parameters using CART

2nd step: find dependencies between important parametersPair-wise test: parameter A depends on B iff different settings

of B lead to different best values for A; check min, mid, max values

0

1

2

3

4

5

6

7

1 3Parameter A

Met

ric R

espo

nse

Parameter B=1

Parameter B=3

0

1

2

3

4

5

6

7

1 3Parameter B

Met

ric R

espo

nse

Parameter A=1

Parameter A=3

A depends on B

Generating the Graph 1st step: find important parameters using CART

2nd step: find dependencies between important parametersPair-wise test: parameter A depends on B iff different settings

of B lead to different best values for A; check min, mid, max values

0

1

2

3

4

5

6

7

1 3Parameter A

Met

ric R

espo

nse

Parameter B=1

Parameter B=3

0

1

2

3

4

5

6

7

1 3Parameter B

Met

ric R

espo

nse

Parameter A=1

Parameter A=3

A depends on B B doesn’t depend on A

Generating the Graph (cont.) An evolution may change the bottleneck tier

Dependencies may be affected by provisioning of tiers• Ex: if throughput is the metric, different bottleneck tiers

may lead to different dependencies

Run dependency checking once per tier, each time forcing the tier to become the bottleneck

Dependency graph is the union of these results

Tuning = Traversing the Graph In a cycle, tune all member vertices at the same

time

In an acyclic chain, tune parents first assuming best values for ancestors and current values for descendants

A singleton is tuned in isolation

Select values using Simplex algorithm and value sets

# experiments increases with connectivity, value set size

Summary

Find important parameters

Find dependencies

Tune parameters

Normal execution

Evolution

Outline Motivation Approach and Contributions Automatic Configuration Infrastructure

(ACI) Tuning Parameters Evaluation Conclusions

Evaluation: Performance Tuning 3-tier auction (Apache, Tomcat, MySQL) on 9

nodes; 24 performance parameters

Management complexity and mistake-elimination results in the paper

Efficiency: Number of throughput-tuning experiments after DB machine upgrade, 2 application server removal

Online Auction Dependencies

Performance Tuning Approaches Exhaustive: O(RN) runs; R = parameter range and

N = # parameters = 24; infeasible

Dependency graph + exhaustive: N = 9; infeasible

Simplex

ACI = Dependency graph + Simplex

Performance Tuning Efficiency

ACI needs 2 service changes to amortize the graph overhead

EvolutionTuning Approach

Throughput (reqs/sec)

Number of Experiments

DB node upgrade

Best prior values

343 -

Simplex search

372 299

ACI 377 821 + 73

Removal of 2 application

servers

Best prior values

91 -

Simplex search

114 220

ACI 110 0 + 35

Amortizing the Graph Overhead Does dependency graph change when service

evolves?

DB node upgrade• Graph does not change

Removal of 2 application server nodes• Graph does not change

Upgrade MySQL server version• Graph is a subset of original one

Conclusions Proposed automatic configuration infrastructure

Proposed notion of parameter dependency graph

Quantified benefits experimentally for realistic service

ACI reduces config complexity, eliminates misconfigs, and produces high-performance services efficiently

Questions?

http://vivo.cs.rutgers.edu/