Surviving Your Peak Database Load

Ben DeBowCEO

FORTIFIED DATA

Surviving Your

Peak Database Load

www.devconnections.com

SURVIVING YOUR PEAK DATABASE LOAD

CEO Fortified Data

Working with SQL Server for 20yrs

Author, speaker, and consultant focusing

on Performance, Scalability, and Availability

E-mail: [email protected]

Twitter: @BBQSQL

Web/Blog: http://www. fortifieddb.com

ABOUT ME

mailto:[email protected]

http://www.sqlha.com/

mailto:[email protected]

http://www.sqlha.com/



DEFINING THE PEAK LOAD

We need to scale, but what does that mean?

Do you know the applications and servers involved?

What are the business transactions?

# of sales ordered in 1hour

1 sales order = 50 or 10,000 database calls?

What are the service level agreements (SLA)?

When is the peak load on the system?

Daily, Monthly, Seasonal

Before any tuning takes place, answer these

questions.



APPLICATION ASSESSMENT Configuration

Operation System

SQL Server

What is installed and being used?

OLTP versus OLAP

Supporting Infrastructure

Disk Subsystem

Server Hardware

Network Topology

Need to know current state!!

Leverage existing CMDB’s



SLAS AND TRANSACTIONS

Number of transactions at the peak hour?

What % of transactions exceed the SLA?

What are the core transactions?

Most important to the business!

What is the busiest day of the week? Hour?



“

”

I THINK THE SYSTEM CAN HANDLE THE

LOAD, WE PROCESS 8 MILLION

RETURNS?

6

Yes, BUT the avg. response

time is 5 seconds and not < 3.



BASELINE THE SYSTEM

Create a Performance Profile

Collect Performance Data

Leverage DMVs

Check SQL uptime before collecting

Know what the configuration is

DBs size and properties

Always look at current system activity!!

Get to know the typical activity



DEMO

8



PERFORMANCE DATA

Disk

Reads/sec

0

20

40

60

80

100

120

140

160

Apr

22

(17)

Apr

29

(18)

May

06

(19)

May

13

(20)

May

20

(21)

May

27

(22)

Jun

03

(23)

Jun

10

(24)

Jun

17

(25)

Jun

24

(26)

Jul 01

(27)

Jul 08

(28)

Jul 15

(29)

Host Latency - Node A

Average (ms)

Batch

Request/sec



APPLICATION STACK



APPLICATION LATENCY August

The average request took 414 ms

5% of requests took over 1.4 seconds


Weekly request peak of 7,900,000

September

The average request took 512.7 ms




October

The average request took 499 ms






THE ANALYSIS

Start at the OS and end at the code

Changing code increases the risk and timeline

Look for misconfigurations

OS, SQL, NICs, SAN, DBs, drivers and firmware

Clean up the room first

Align configurations to “Best Practices”

Changes system behavior

Identify the real issues

Collect more data



THE ANALYSIS - PERFMON Time Slices

Sample intervals

Seconds versus minutes – GOAL?

**Correlation of object counters

Thresholds

Every system is different

Baseline to define your ranges

Start high and drill down

Chart versus report



PERFMON ARTWORK



DEMO

15



ARCHITECTURES Scale up

Scale out

Reporting Tier with Availability Groups

Re-direct read-only requests with Availability Groups

Sharding

The CLOUD

Caching Servers

Active/Active

Replication – Bi-directional Transactional



ACTION PLAN 2 Scenarios

More juice out of the lemon

Have money, will spend

Affects recommendations

Let’s squeeze more juice out of the lemon

Identify system constraints

What is limiting factor(s) for scaling?

How do the findings play into the peak goals?

How do you measure the system?

Stress test if possible



PROVEN APPROACH

1. Infrastructure and OS configuration

2. SQL Server configuration

3. Database configuration

4. Scheduling and non-essential processes

5. Reduce contention

6. Performance tuning

1. Indexes

2. Statements

3. Process re-design

7. Re-assess performance



SUPPORTING THE SYSTEM Monitoring

Proactive and reactive

Fast identification and resolution

Ex. Blocking chains will impact the SLA’s

Communication

Contact numbers

On-call schedules

Perfmon Maintenance before peak load

Turn off during peak window



SUMMARY Understand system architecture

Capture configuration and performance

data to determine the limiting factors

Iterative approach helps reduce risk and

gauge impact

Plan for 20% above peak

Monitoring is key

Fast identification and resolution



QUESTIONS

21

Surviving Your Peak Database Load

Data & Analytics