Top Banner
66

Mysteries of Windows Memory Management Revealed

Feb 23, 2016

Download

Documents

Derick Derick

WCL405. Mysteries of Windows Memory Management Revealed. Mark Russinovich Technical Fellow Windows Azure (created jointly with Dave Solomon). Goals. Deep dive on: Process virtual and physical memory usage Operating system virtual and physical memory usage - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Mysteries of Windows Memory Management Revealed
Page 2: Mysteries of Windows Memory Management Revealed

Mysteries of Windows Memory Management Revealed

Mark RussinovichTechnical FellowWindows Azure(created jointly with Dave Solomon)

WCL405

Page 3: Mysteries of Windows Memory Management Revealed

Goals

Deep dive on:Process virtual and physical memory usageOperating system virtual and physical memory usage

Crisply define memory-related terminologyHighlight tools that reveal memory usageDescribe ‘dark spots’ in memory analysis counters and tools

Page 4: Mysteries of Windows Memory Management Revealed

Agenda

Virtual MemoryAddress Space UsageProcess CommitSystem Commit

Physical MemoryWorking SetsPaging Lists

Hard to Track Memory

Page 5: Mysteries of Windows Memory Management Revealed

Tools We’ll Use

Task ManagerSysinternals Process ExplorerSysinternals Vmmap

Process virtual and physical memory usageSysinternals Rammap

System physical memory usageSysinternals Testlimit

Test program to leak different kinds of memory

Sysinternals tools are free at www.sysinternals.com

Page 6: Mysteries of Windows Memory Management Revealed

Virtual Memory

Page 7: Mysteries of Windows Memory Management Revealed

Memory Management Fundamentals

Windows has demand-paged memory managementProcesses “demand” memory as neededThere is no swappingA page is 4 KB (8 KB on Itanium)Allocations must align on 64 KB boundaries

Large pages are available for improved TLB usagex86: 4 MBX64 and x86 PAE: 2 MBItanium: 16 MB

There is NO (will, almost no) connection between virtual memory and physical memory

Page 8: Mysteries of Windows Memory Management Revealed

32-bit x86 Address Space32-bits = 2^32 = 4 GB

/3GB and /USERVA can extend process address up to 3 GBProcess must be marked “large address space aware” to use memory above 2 GB

Default 3 GB user space

2 GBPer-Process

Space

2 GBSystemSpace

3 GBPer-Process

Space

1 GBSystemSpace

Page 9: Mysteries of Windows Memory Management Revealed

64-bit Address Spaces64-bits = 2^64 = 17,179,869,184 GB

x64 today supports 48 bits virtual = 262,144 GB = 256 TBIA-64 today support 50 bits virtual= 1,048,576 GB = 1024 TB64-bit Windows supports 44 bits = 16,384 GB = 16 TB

x64 32-bit processon x64

8 TBSystemSpace

4 GBPer-Process

Space

8 TBSystemSpace

8 TBPer-Process

Space

Page 10: Mysteries of Windows Memory Management Revealed

Virtual Address Space Components

Committed: in-useReserved: reserved for future useAddress space breakdown

Private (e.g. process heap)Reserved or committed

Shareable (e.g. EXE, DLL, shared memory, other memory mapped files)

Reserved or committedFree (not yet defined)

Page 11: Mysteries of Windows Memory Management Revealed

Why Reserve Memory?

Reserved memory lets an application lazily commit contiguous memory Used for stack and heap expansion

Before Expansion

Committed

Reserved

After Expansion

Committed

Reserved

ThreadStack

StackGrowsDown

GuardGuard

Page 12: Mysteries of Windows Memory Management Revealed

Viewing Address Space Breakdown

Task Manager only lets you see private bytes Before Vista: column called “VM Size”Vista and later: column called “Commit Size”

Process Explorer shows both virtual size and private bytesAdd 2 columns to process list

Virtual SizePrivate Bytes

Run Testlimit twice Testlimit -rTestlimit -m

Note: if on 64-bit Windows, 32-bit Testlimit can grow to 4GB

Page 13: Mysteries of Windows Memory Management Revealed

Understanding Process Address Space Usage

Most virtual memory problems are due to a process leaking private committed memory

Heap, GC heap, language heaps (CRT)Private Bytes only tells part of the story

Doesn’t account for shareable memory that’s not shared (e.g. DLLs loaded only by this process)Fragmentation can be an issue

Address space can effectively be exhausted prematurely

Basic performance counters don’t provide enough information to troubleshoot

FragmentedAddressSpace

Page 14: Mysteries of Windows Memory Management Revealed

Viewing Processes with VMMap

VMMap shows detailed breakdown of process address space:Private process memory

Copy-on-write Private (VirtualAlloc)Heap and GC HeapStack

Shareable process memoryImage - executablesShareable – shareable memoryMapped File – memory mapped files

Page table – page table pagesUnusable – gap between allocation and next allocation boundary

Note that “shareable” types can have private commitmentRead/write pages in shared memoryCopy-on-write pages

Page 15: Mysteries of Windows Memory Management Revealed

Viewing FragmentationFragmentation is visible by selecting Options->Show Free Regions, selecting the Free type, and sorting by size

Largest free block is largest allocation possibleClickable fragmentation map in View->Fragmentation ViewRun testlimit -t on 64-bit Windows

Threads need 256 KB 64-bit stack and 1 MB 32-bit stack

Page 16: Mysteries of Windows Memory Management Revealed

File MappingsFile mapping enables an application to read and write file data through memory operationsFile mappings are used for

Image (.EXE and .DLL) loading: “Image” in VMMapData files access (e.g. NLS files): “Mapped File” in VMMap“Pagefile-backed” shared memory: “Shareable” in VMMap

Entire file doesn’t have to be mappedAllows for “windows” into the file

AddressSpace

Database.db

Page 17: Mysteries of Windows Memory Management Revealed

Tracing File Mapping with Process Monitor

Procmon can trace image loader activity

Page 18: Mysteries of Windows Memory Management Revealed

VMMap Differencing

Press F5 to refresh the viewVMMap keeps all snapshots

Use the timeline to select snapshots to compare

Page 19: Mysteries of Windows Memory Management Revealed

Tracing with VMMap

You can launch a process with profilingDetours tracks virtual and heap activity

Page 20: Mysteries of Windows Memory Management Revealed

The System Commit Limit

System committed virtual memory must be backed either by physical memory or stored in the paging file

Sum of (most of) physical memory and current paging filesAllocations charged against the system commit limit:

Process private bytes Pagefile-backed shared memory Copy-on-write pagesRead/write file pagesSystem paged and nonpaged code and data

When limit is reached, virtual memory allocations failProcesses may crash (or corrupt data)

Page 21: Mysteries of Windows Memory Management Revealed

Changing the System Commit Limit

You can increase the system commit limit by adding RAM or increasing the pagefile sizeThe system commit limit can grow if paging files configured to expand

So the system commit limit might be the current limit, not the maximumDefault configuration (“System Managed”):

Minimum: 1.5x RAM if RAM < 1 GB; RAM otherwiseMaximum: 3x RAM or 4 GB, whichever is larger

Maximum system commit limit should be based on system commit peak for extreme workload

Page 22: Mysteries of Windows Memory Management Revealed

Viewing System Commit UsagePerformance Counters:

Committed BytesCommit Limit

Task Manager XP: commit charge labeled “PF Usage”Vista: commit charge labeled “Page File”Win7: commit charge labeled “Commit”Vista and Win7 show commit limit after slash

Page 23: Mysteries of Windows Memory Management Revealed

Viewing the System Commit LimitProcess Explorer shows commit charge (with history), commit limit, and commit peak

No built-in tool shows peak any more

Page 24: Mysteries of Windows Memory Management Revealed

Exhausting the System Commit Limit

On 32-bit system, run “Testlimit –m” multiple times until system commit limit exhaustedOn 64-bits, “Testlimit64 –m” will exhaust the system commit limit before its address space:

Page 25: Mysteries of Windows Memory Management Revealed

Sizing the Paging File

If you enough RAM to support your commit needs, why even have one?System can page out unused, modified private pages vs keeping them in RAMMore RAM available for useful stuff

Many recommendations use a formula based on RAM (1.5x, 2x, etc.)Actually, the more RAM, the smaller the paging file neededShould be based on workload usage of committed virtual memory

Look at commit peak after workload has runPre-Vista: Task ManagerVista+: Process ExplorerApply a formula to that to give buffer (1.5x or 2x)Make sure it’s big enough to hold a kernel crash dump

Page 26: Mysteries of Windows Memory Management Revealed

Physical Memory

Page 27: Mysteries of Windows Memory Management Revealed

Working Set List

All the physical pages “owned” by a processE.g. the pages the process can reference without incurring a page fault

A process always starts with an empty working setIt then incurs page faults when referencing a page that isn’t in its working set

Hard fault: resolved from file on disk (paging file, mapped file)Soft fault: resolved from memory

Working Set

newer pages older pages

Page 28: Mysteries of Windows Memory Management Revealed

Working Set

Each process has a default working set minimum and maximum

Can change with SetProcessWorkingSetWorking set minimum controls maximum number of locked pages (VirtualLock)Minimum is also reserved from RAM as a guarantee to the processWorking set maximum is ignored

If there’s ample memory, process working set represents all the memory it has referenced (but not freed)

If memory is tight, working sets get trimmed

Page 29: Mysteries of Windows Memory Management Revealed

Working Set Replacement

When memory manager decides the process is large enough, it give up pages to make room for new pagesLocal page replacement policy

Means that a single process cannot take over all of physical memory unless other processes aren’t using itPage replacement algorithm is least recently accessed (pages are aged when available memory is low)

Working Set

To standby or modified

page list

Page 30: Mysteries of Windows Memory Management Revealed

Working Set Breakdown

Consists of 2 types of pages:Shareable (of which some may be shared)Private

Four performance counters available:Working Set Shareable

Working Set Shared (subset of shareable that are currently shared)Working Set PrivateWorking Set Size (total of WS Shareable+Private)

Note: adding this up for each process overcounts shared pages

Caveats:Working set does not include trimmed memory that is still cachedShareable working set should be viewed as “private” if it’s not shared

Page 31: Mysteries of Windows Memory Management Revealed

Viewing Working Set with Task Manager

Displays private working set sizeCalls it “Memory (Private Working Set)”

Page 32: Mysteries of Windows Memory Management Revealed

Viewing Working Set with Process Explorer

Process Explorer shows all the performance counters

Virtual BytesPrivate BytesWS Shareable BytesWS Shared BytesWS Private Bytes

Run Testlimit three times:Testlimit -r 1024 -c 1Testlimit -m 1024 -c 1Testlimit -d 1024 -c 1

Note how working set numbers don’t at all represent the process virtual memory usage

Page 33: Mysteries of Windows Memory Management Revealed

Viewing the Working Set with VMMap

Vmmap shows working set size of each component of address spaceAlso shows locked pagesCopy-on-write pages will show up as Private WS in shareable regions

Page 34: Mysteries of Windows Memory Management Revealed

Physicalmemory

Page 3

Page 1

How Copy-On-Write WorksBefore

ProcessAddressSpace

Orig. Data

ProcessAddressSpace

Orig. Data

Page 2

Page 35: Mysteries of Windows Memory Management Revealed

ProcessAddressSpace

Physicalmemory

How Copy-On-Write WorksAfter

ProcessAddressSpace

Orig. Data

Page 3

Page 1

Page 2

Copy of page 2

Mod’d. Data

Page 36: Mysteries of Windows Memory Management Revealed

Managing Physical Memory

System keeps unassigned physical pages on one of several lists

Free page listModified page listStandby page lists (8 as of Vista & later)Zero page listROM page listBad page list - pages that failed memory test at system startup

Lists are implemented by entries in the “PFN database”Maintained as FIFO lists or queues

Page 37: Mysteries of Windows Memory Management Revealed

Paging Dynamics

New pages are allocated to working sets from the top of the free or zero page listPages released from the working set due to working set replacement go to the bottom of:

The modified page list (if they were modified while in the working set)The standby page list (if not modified)

Decision made based on “D” (dirty = modified) bit in page table entryAssociation between the process and the physical page is still maintained while the page is on either of these lists

Page 38: Mysteries of Windows Memory Management Revealed

Standby and Modified Page Lists

Modified pages go to modified (dirty) listAvoids writing pages back to disk too soon

Unmodified pages go to standby (clean) listsThey form a system-wide cache of “pages likely to be needed again”

Pages can be faulted back into a process from the standby and modified page listThese are counted as page faults, but not page reads

Page 39: Mysteries of Windows Memory Management Revealed

Modified Page Writer

When modified list reaches certain size, modified page writer system thread is awoken to write pages out

Also triggered when memory is overcommitted (too few free pages)Does not flush entire modified page list

Two system threadsOne for mapped files, one for the paging file

Pages move from the modified list to the standby listE.g. can still be soft faulted into a working set

Page 40: Mysteries of Windows Memory Management Revealed

Free and Zero Page Lists

Free Page ListUsed for page readsPrivate modified pages go here on process exitPages contain junk in them (e.g. not zeroed)On most busy systems, this is empty

Zero Page ListUsed to satisfy demand zero page faults

References to private pages that have not been created yetWhen free page list has 8 or more pages, a priority zero thread is awoken to zero themOn most busy systems, this is empty too

Page 41: Mysteries of Windows Memory Management Revealed

Paging Dynamics

StandbyPageLists

ZeroPageList

FreePageList

WorkingSets

page read from disk or kernel allocations

(“hard” page faults)

demand zero page faults

working set replacement

ModifiedPageList

modifiedpagewriter

zeropage

thread

“soft”pagefaults

BadPageList

Private pages at process exit

“global valid” faults

Page 42: Mysteries of Windows Memory Management Revealed

Viewing the Paging Lists with Task Manager

XP/2003:Available = Standby + Zero + FreeSystem Cache = Standby + Modified + System Working Set

Vista/Server 2008:Replaced Available with Free

Free + Zero listSystem Cache relabeled Cached

Windows 7/Server 2008 R2Available put back

Page 43: Mysteries of Windows Memory Management Revealed

Viewing the Paging Lists with Process Explorer

Process Explorer shows each paging listClick View->System Information

Page 44: Mysteries of Windows Memory Management Revealed

Total Process Private Memory Usage

Working Set size does not include:

Private memory on standby or modified listsPage tables

Rammap shows this on Processes tab

Page 45: Mysteries of Windows Memory Management Revealed

Viewing Memory Usage with Rammap

In addition to showing size of paging lists, shows usage breakdown:Process privateMapped fileShared memoryPage tablesPaged poolNonpaged poolSystem PTESession privateMetafileAWEDriver lockedKernel stack

Page 46: Mysteries of Windows Memory Management Revealed

Prioritized Standby Lists

In Vista & later, there are 8 prioritized standby listsPages are removed from lowest priority list first

Low memory priority process will keep re-using low priority pagesHigher priority information remains cached

0

1

2

3

4

5

6

7

Prioritized Standby Lists

Pages removed

Pages added

Page 47: Mysteries of Windows Memory Management Revealed

SuperFetch™

Superfetch proactively repopulates RAM with the most useful dataSets priority of pages to optimal value, based the page history and other analysis that it performsTakes into account frequency of page usage, usage of page in context of other pages in memoryAdapts application launch patterns, in chunks of 8 hours (times a day) and weekend vs weekday

Scenarios SuperFetch improves includeResume from hibernate and suspendFast user switchingPerformance after infrequent or low priority tasks executeApplication launch

Windows 7: Disabled if the OS is booted of an SSD

Page 48: Mysteries of Windows Memory Management Revealed

Memory Priority

Each thread has its own memory priority5: normal1: low

This determines which standby list is used for the page (when/if it arrives on the standby list)Thread priority comes from process memory priority

Can be changed for process or individual threadSetPriorityClass or SetThreadPriority “background mode”

Page 49: Mysteries of Windows Memory Management Revealed

Standby List Population

Priority 7 come from a static set (pre-trained at Microsoft) Pre-populated at each bootIncludes pages related to user input that requires fast responsiveness (right-click, desktop properties, control panel, start menu, etc.)

Priority 6 are pages that SuperFetch considers important, or useful (will rarely get repurposed)Priority 5 are standard user pages (memory priority 5)Priority 1 are low priority user pages (memory priority 1)Priority 0-4 may be Superfetch decayed, cache manager read-ahead and pagefault clustering

Page 50: Mysteries of Windows Memory Management Revealed

How Much of the Standby List has Been Consumed?

RAMMap shows the amount of memory repurposed off each standby list since boot:

Page 51: Mysteries of Windows Memory Management Revealed

What File Data is In the Standby Lists? Viewing Cached Files with Rammap

Page 52: Mysteries of Windows Memory Management Revealed

Do You Have Enough Memory?

There’s no sure-fire rule or counter to tell if you if you have enough memoryThe general rule: available memory remains generally low

Use Perfmon to monitor available memory over timeUse Process Explorer, or on Vista and later, Task Manager, to monitor physical memory usageUse Process Explorer, or Task Manager to see instantaneous value of available memoryWatch in Process Monitor for excessive reads from paging file

Page 53: Mysteries of Windows Memory Management Revealed

Tracing Paging with Procmon

Procmon distinguishes paging I/Os in the details columnCan set filter for “detail contains paging”

I/O to Pagefile.sys excluded by defaultEnable advanced output or remove exclude filterExcessive reads from paging file indicates need more RAM

Page 54: Mysteries of Windows Memory Management Revealed

Hard to Track Memory

Page 55: Mysteries of Windows Memory Management Revealed

Hidden Cost of Reserved Memory

Memory Manager charges for page tables for reserved address space not yet committed

Charged against process private bytes (and therefore system commit limit)Cannot track this down

Experiment:Testlimit64 –r 100000 –c 10

Testlimit64 –rReserves ~8192GBPrivate bytes grows to >16GB!

Page 56: Mysteries of Windows Memory Management Revealed

Cost of Reserved Memory

Virtual Address Space (VAD) descriptors come from nonpaged poolExample:

Testlimit64 –r results in 640mb of nonpaged pool usage for VADsPoolmon shows this:

Page 57: Mysteries of Windows Memory Management Revealed

Shared Memory

Shared memory is backed by virtual memoryEither paging file (if there is one), else physical memory

However, amount created not charged to process commit limit

Therefore, a shared memory VM leak is hard to track down

Page 58: Mysteries of Windows Memory Management Revealed

Demo: Shared Memory Leak

Testlimit –s 1000 –c 3This creates a 3 GB shared memory section

Note that process virtual and private bytes do not include this value

And only virtual bytes rise when process maps the section into its address space

Page 59: Mysteries of Windows Memory Management Revealed

VirtualLock Locked Pages

No special privilege is required to VirtualLock pages (as of Vista) Allows process to allocate non-paged memory

Locked memory has a major impact on the system:Overrides memory management policiesPrevents contiguous physical memory allocationCan prevent hibernation

Testlimit -l 1024 –c 1DWM locks memory:

Page 60: Mysteries of Windows Memory Management Revealed

Driver Locked Pages and AWE pages

VPC and Hyper-V use driver locked pagesSQL Server uses AWE memoryCounts against system commit, but not otherwise detectable

No way to track back to owner

Page 61: Mysteries of Windows Memory Management Revealed

More Information

More information in Windows Internals, 5th EditionMemory Management chapter

MSDN memory management API documentationThe best way to gain an understanding of memory manager behavior is to experiment and observeCome to my other sessions tomorrow!

Inside Windows Azure: 10:15Case of the Unexplained: 2:45

Book signing at the TechEd bookstore: 12:30-1:30

Page 62: Mysteries of Windows Memory Management Revealed

Track Resources

Don’t forget to visit the Cloud Power area within the TLC (Blue Section) to see product demos and speak with experts about the Server & Cloud Platform solutions that help drive your business forward.You can also find the latest information about our products at the following links:

Windows Azure - http://www.microsoft.com/windowsazure/

Microsoft System Center - http://www.microsoft.com/systemcenter/

Microsoft Forefront - http://www.microsoft.com/forefront/

Windows Server - http://www.microsoft.com/windowsserver/

Cloud Power - http://www.microsoft.com/cloud/

Private Cloud - http://www.microsoft.com/privatecloud/

Page 63: Mysteries of Windows Memory Management Revealed

Resources

www.microsoft.com/teched

Sessions On-Demand & Community Microsoft Certification & Training Resources

Resources for IT Professionals Resources for Developers

www.microsoft.com/learning

http://microsoft.com/technet http://microsoft.com/msdn

Learning

http://northamerica.msteched.com

Connect. Share. Discuss.

Page 64: Mysteries of Windows Memory Management Revealed

Complete an evaluation on CommNet and enter to win!

Page 65: Mysteries of Windows Memory Management Revealed

Scan the Tag to evaluate this session now on myTech•Ed Mobile

Page 66: Mysteries of Windows Memory Management Revealed