Sql Server 2005 Memory Internals

"Technology is dominated by two types of

people: those who understand what they do

not manage, and those who manage what

they do not understand.“

“Putt's Law and the Successful Technocrat

How to Win in the Information Age”

Archibald Putt

DAT322: SQL Server

2005 Memory Internal

Geyzerskiy Dmitriy

IT Architect, Senior Technologies Consultant

[email protected]

Session Objectives and Agenda

• Windows Memory Management

• NUMA Architecture

• SQL Server Memory Management

• AWE vs. 64-bit

Myths

Windows Memory Overview • Virtual Memory

• 32 bits: 4GB

• 64 bits: 17,179,869,184 GB

• VM states: Committed, Reserved, Free

• Page size

• 4KB on x86, EM64T, AMD64

• 8KB on Itanium

• Large pages: 4MB on x64, 16MB on Itanium

• Paging

• Page Faults

• Working Sets

• Kernel Memory And User Memory

Retrieving system information using

Win32 API and Windbg

3GB Process Space Option

Application Memory

2GB

System Memory

2GB

Application Memory

3GB

System Memory

1GB

• /3GB switch in BOOT.INI

• /USERVA (between 2048 and 3072, in 128MB increments)

• .EXE must be linked with LARGEADDRESSAWARE flag

64 bit Address Space

8TB

User

6TB

System

• Map more data into the address space

• The application “speed” is the same on 32 bit and 64 bit

• OS needs 2GB of memory to hold pointers to 16GB or

more physical memory

x64

7TB

User

6TB

System

IA64

Paging Dynamics

Working

Sets

Standby

List

Modified

List

Free

List

Zero

List

Exploring Page Lists using Windows

Kernel Debugger

Sizing the Page File

• More RAM should mean smaller page file!

• Crash dump settings affect

• Full: size of RAM

• Kernel dump: much smaller

• To size correctly, review what goes there

• Minimum should = commit charge peak

• Maximum could be a multiple of this

Measuring memory performance

Agenda

• Windows Memory Management

• NUMA Architecture

• SQL Server Memory Management

• AWE vs. 64-bit

NUMA

What is SMP

• SMP – Symmetric Multi-Processing

• Front-bus point of contention

• Difficult to scale beyond 32 CPU

Memory

CPU 0 CPU 1 . . . CPU n

Front-bus

What is NUMA

Memory

CPU 0 CPU 1 CPU 2 CPU 3

Memory

CPU … CPU … CPU … CPU n

NUMA (Non-Uniformed Memory Access)

Local Memory Access

Local Memory Access

What is Interleaved-NUMA

• Enable NUMA hardware to behave as SMP

• Memory are used by all CPUs

• Each CPU’s cache line access slice of memory from all nodes

• SQL Server 2000 should use interleaved-NUMA

Local Memory Access Foreign Memory Access

Memory

CPU 0 CPU 1 CPU 2 CPU 3

Memory

CPU … CPU … CPU … CPU n

What is Soft-NUMA

• Activates custom NUMA configuration on top of

any of hardware

• Provides greater performance, scalability,

and manageability on SMP as well as on real

NUMA hardware.

• Provides one I/O thread and one lazy writer

thread per NUMA node.

Soft-NUMA Configuration Example

We have:

NUMA system with 2 nodes and 4 CPU per

Node

We need:

2 CPUs for loading application and the rest

of CPUs for queries.

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL

Server\90\NodeConfiguration\Node0]

"CPUMask"=dword:0000000F

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL

Server\90\NodeConfiguration\Node1]

"CPUMask"=dword:00000030

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL

Server\90\NodeConfiguration\Node2]

"CPUMask"=dword:000000C0

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL

Server\MSSQL.1\MSSQLServer\SuperSocketNetLib\Tcp\IPAll]

"TcpPort"="1433[0x3],2000[0x4]"

"TcpDynamicPorts"=""

"DisplayName"="Any IP Address"

Soft-NUMA Configuration Example

But don’t try to configure Soft-NUMA on the

system with 4 or less CPUs and at least 1 node

has exactly 1 CPU

Agenda

• Windows Memory Management

• NUMA Architecture

• SQL Server Memory Management

• AWE vs. 64-bit

SQL Server 2005 Architecture

Deadlock

Monitor

SQ

LO

S H

os

ting

AP

I

Scheduling

Memory Manager

Buffer

Pool

Lock

Manager

Synchronization

Resource

Monitor

Lazy

Writer

SQLOS API

I/O

SQ

LO

S A

PI

Exte

rnal C

om

po

nen

ts (C

LR

/MD

AC

)

SQLOS

Scheduler

Monitor

= thread

Parser Optimizer SQL Manager Database

Manager

Query

Executor

Query Processor

Transaction Services

Lock Manager

File Manager

Buffer Manager

Utilities:

Bulk Load DBCC

Backup/Restore

Access Methods

Managers for:

Row Operations

Indexes

Pages

Allocations

Versions

Storage Engine

Protocols

SQL Server Memory Management

Resource

Monitor

Generic

Memory Clerk

Cache Memory

Clerk

Buffer Pool

Memory Clerk

CLR Memory

Clerk

Memory Node

High Physical Internal / External

Low VAS

Low Physical Internal / External

Memory Pressure

CP CacheStore

(@1 numeric(2,1), @2 numeric(3,2))

insert [t] values(@1 , @2)

insert t values (2.1, 3.12) insert t values (3.33, 4.1)

(@1 numeric(3,2), @2 numeric(2,1))

insert [t] values(@1 , @2)

Auto-parameterization

Bucket 0 Bucket 1 Bucket 2

SQL Server 2005 RTM SQL Server 2005 SP2

Monitoring Memory with SQL Server DMVs

Agenda

• Windows Memory Management

• NUMA Architecture

• SQL Server Memory Management

• AWE vs. 64-bit

Address Windowing Extensions (AWE)

• Access more than 4GB of physical memory.

• Is ignored on systems with less than 3GB of physical memory.

• Is never swapped to disk.

Allocate the physical memory (Lock Pages in Memory) 1

Create a region in the process address space to serve as

a window for mapping views of this physical memory 2

Map a view of the physical memory into the virtual

memory window 3

SQL Server Process Address Space with AWE

0xFFFFFFFF

0x00000000

0x80000000

0xC0000000

Operating System

SQL Server or OS (/3GB switch)

SQ

L S

erv

er

MemToLeave area

Thread stacks

Oth

er

Lo

ck

s

Qu

ery

Wo

rks

pa

ce

Pla

n C

ac

he

DB

Pa

ge

Ca

ch

e

(Bu

ffer

Po

ol)

AWE

Memory

SQL Server 2005 32-bit AWE Memory

• Right OS version

• Windows Server 2003 Standard and up

• /PAE in boot.ini enables 32-bit OS

to address more than 4GB memory

• SQL Server Edition

• Enterprise Edition

• Developer Edition

• sp_configure ‘awe enabled’

Lock Pages In Memory Option

• Entry in the SQLERROR log

• 64 bit: Using locked pages for buffer pool

• 32 bit: Address Windowing Extensions is enabled

• Discarded in Standard Edition

• The Local System account has the 'lock pages in

memory' privilege by default

A significant part of sql server process memory has been paged out.

This may result in a performance degradation. Duration: 0 seconds.

Working set (KB): 1086400, committed (KB): 2160928, memory

utilization: 50%.

SQL Server 2005 64 bit vs. 32 bit

• The only way to get virtual memory > 3GB

• What is different from 32-bit?

• All pointers are 64-bit

• SQL Server commits ‘min server memory’ memory at startup

• Some internal memory-related data-structure constants larger

• 64-bit alignment of data structures

SQL Server 2005 64 bit vs. 32 bit

• What is the same?

• No on-disk database format changes

• No differences in buffer pool policy / algorithms from 32-bit

• All uses of memory can use additional 64-bit memory

• DB Page Cache, Query Workspace Memory, Plan Cache,

Locks, External uses, Utilities, …

32-Bit and 64-Bit System Limitations

Architectural

component

64-bit Windows 32-bit Windows

Virtual memory 16 TB 4 GB

Paging file size 512 TB 16 TB

Hyperspace 8 GB 4 MB

Paged pool 128 GB 470 MB

Non-paged pool 128 GB 256 MB

System cache 1 TB 1 GB

System PTEs 128 GB 660 MB

Resources

• Blogs • Slava Oks’ blog: http://blogs.msdn.com/slavao

• SQL Programmability & API Development Team Blog: http://blogs.msdn.com/sqlprogrammability/

• External Links: • NUMA FAQ: http://lse.sourceforge.net/numa/faq

• Books: • Eldad Eilam: Reversing: Secrets of Reverse Engineering

• Ken Henderson: SQL Server 2005 Practical Troubleshooting The Database Engine

• Kalen Delaney: Inside Microsoft SQL Server 2005 The Storage Engine

!כדאי למלא משוב

?איך ממלאים ,בעקבות מייל שישלח בסיום כל יום•

, HPבמתחם Business Center-ב•

בעמדות האינטרנט במלונות הילטון ודן•

!Live Itמגיעה לך חולצת ? מילאת משוב

?מלאת משוב בשלושת ימי הכנס

מתנת סוכנות כרטיס טיסה לתאילנד יש לך הזדמנות לזכות ב

BTC , מכשיר , מתנת סמסונגק 'ג בלאקמכשירHTC מתנת

...ועוד DataSafeמתנת מדיה סנטר , ניופאן

Summary

• It pays to understand SQL Server memory management

• A number of performance issues either originate or manifest as

memory issues

• Memory-based performance tuning is a very useful technique

• Significant internal and external changes in SQL Server 2005

• Consider NUMA for your next large-scale project

• Upgrade your system to 64 bit

!המסיבה.... כמה מילים על

יוצאות הסעות מבתי המלון למסיבה בנמל 20:30החל משעה •

.אילת

.ארוחת הערב תוגש במסיבה עצמה•

בשל בעיות אבטחה ! שימו לב•

כבדות בנמל אין אפשרות להגיע

לא . עם רכבים פרטיים או מוניות

נוכל להבטיח כניסה למי שיבחר

.להגיע שלא באמצעות ההסעות

© 2008 Microsoft Corporation. All rights reserved.

This presentation is for informational purposes only.

MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.