Memory Usage and .NET
Module Overview - Troubleshooting .NET Memory Issues
Memory Management: General Review
GC Heap
Generations: Structure & Benefits
GC Sequence
GC Modes
Maximizing GC Performance
Roots
Finalization
Review
2
Native Memory Usage
Memory Management: General Review
Virtual Memory
32-bit system
- 4 GB (2GB-2GB for user and kernel mode)
- Can be altered with /3GB switch or USERVA in boot.ini
- Important: /LARGEADDRESSAWARE flag
64-bit system
- 16 TB – huge address space
Memory states
- Free, Reserved, Committed
- Perfmon: Process\Virtual Bytes, Process\Private Bytes
4
Virtual Memory Map (2GB-32Bit)
No access region
Private Process Space
Stacks
0x00000000
0x0000FFFF
0x7FFE1000
0x7FFFFFFF
0x7FFE0000
Module images
Heaps
Virtual Allocations
Unused
TEBs
PEB
Shared User Data (Date,Time,Win32 Version,..)
No access region
NT (Windows 32) Heaps
Other HEAPs .NET Heap
Virtual Memory Layout
Virtual Memory
Process 1 Physical Memory
(RAM)
Virtual Memory
Process 2
NTDLL.DLL
Process 2 Data
Proc2.exe
Process 2
Data
Process 1
Data
NTDLL.DLL
Proc2.exe
Process 2 Data
Process 2
Data
Process1 Data
Proc1.exe
NTDLL.DLL
Process1 Data
Proc1.exe
Process 1
Data
0x00000000
0xFFFFFFFF
Use
r M
od
eK
ern
el M
od
e
Memory Limits 32 <-> 64
Memory Limits 32-bit 64-bit
Win 7/2008
64Bit
Win 8.1/2012 R2
Total Virtual Address Space 4 GB 16 TB 256 TB
Virtual Address Space per
32-bit Process
2 GB 2 GB 2 GB
Virtual Address Space per
32-bit Process compiled
with
/LARGEADDRESSAWARE
3 GB if
system is
booted with
/3gb switch
and
4GB 4GB
Virtual Address Space per
64-bit Process
x64
Not
applicable
=16 TB-64k
no access
region
0-7ffffff0000
=128 TB-64k no
access region
Paged Pool 470 MB 128 GB 384GB
Non-paged Pool 256 MB 128 GB 15.5TB
System Cache 1 GB 1 TB 16TB
Memory Management: General Review #3
Memory Managers
- NT Heap
- GC Heap (mscorwks.dll, clr.dll)
Acquiring and Committing Memory
- VirtualAlloc
- HeapAlloc
- New/malloc
8
.NET Heap
Layered Memory Management in Win32
Win32 Application
Local, Global
Memory API
VC6 <
malloc
NT Heap
Virtual Memory Allocator
Memory Mapped
Files
NT Virtual Memory Manager
Physical
Memory Hard Disk
Win32
SubSystem
.NET Application
Kernel
Memory Management: General Review #2
VirtualAlloc API
- lpAddress – where to allocate
- dwSize – how much
- flAllocationType (MEM_COMMIT, MEM_RESERVE, MEM_RESET
)
- flProtect - protection level
Smallest chunk it can reserve: 64KB
base memory allocator in Windows.
enables a process to reserve a range of its virtual address space
without consuming physical storage until it is needed
Memory managers are taking the role of reserving the memory
10
Memory Management: General Review #3
Memory Managers
- NT Heap
- GC Heap (mscorwks.dll, clr.dll)
Acquiring and Committing Memory
- VirtualAlloc
- HeapAlloc
- New/malloc
11
NT Heap (HeapAlloc,..)
Used from the „most“ Win32 Applications
Users can create their own process heaps using HeapCreate()
or use the default process heap by using the GetProcessHeap() API.
Allocation does not have to be aligned on a page boundary
In Windows 2000 and 2003, applications that frequently use allocations of varying sizes or lifetimes can use the low fragmentation heap (LFH) )(via HeapSetInformation())
Design changed to use LFH up from Vista
X64: Required to return memory on a 16-byte boundary
Process Environment Block
(PEB)Describes a user-mode process.
Created, maintained, and destroyed by the operating system.
Contains information needed by the image loader, heap manager, and other Win32 subsystems.
One and only one PEB for each user process.
This is the user-mode parallel of the kernel’s KPROCESS block.
Accessible using the !PEB debugger within Windbg debugger
!PEB0:010> !peb
PEB at 7ffd7000
InheritedAddressSpace: No
ReadImageFileExecOptions: No
BeingDebugged: Yes
ImageBaseAddress: 00440000
Ldr 00241e90
Ldr.Initialized: Yes
Ldr.InInitializationOrderModuleList: 00241f28 . 002443f0
Ldr.InLoadOrderModuleList: 00241ec0 . 002443e0
Ldr.InMemoryOrderModuleList: 00241ec8 . 002443e8
Base TimeStamp Module
440000 40f6450f Jul 15 10:49:19 2004
\\?\C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\aspnet_wp.exe
7c900000 411096b4 Aug 04 09:56:36 2004
C:\WINDOWS\system32\ntdll.dll
7c800000 411096b4 Aug 04 09:56:36 2004
Thread Environment Block (TEB)
Per-thread structure that describes a running thread within a process.
The operating system maintains a region of virtual address space that stores TEBs for running threads.
Similar to the kernel’s ETHREAD structure.
Accessible under the debugger using the !TEB command for the current thread context.
contains pointer to ExceptionList, Stack Base, Last Win32 Error, ...
!TEB
0:000> !teb
TEB at 000007fffffde000
ExceptionList: 0000000000000000
StackBase: 0000000000230000
StackLimit: 000000000021c000
SubSystemTib: 0000000000000000
FiberData: 0000000000001e00
ArbitraryUserPointer: 0000000000000000
Self: 000007fffffde000
EnvironmentPointer: 0000000000000000
ClientId: 000000000000139c . 0000000000001280
RpcHandle: 0000000000000000
Tls Storage: 000007fffffde058
PEB Address: 000007fffffd8000
LastErrorValue: 5
LastStatusValue: c0000034
Count Owned Locks: 0
HardErrorMode: 0
Understanding Managed Memory
Managed heaps
Loader heap
Native heaps
Threads
Images (.dll files)
Other (virtual allocations)
Free
0x00000000 0x7FFFFFFF
17
.NET Memory Management
Memory Management
Native
malloc, free, new, delete
You can choose by yourself when to call free / ~destruktor
No „automatic“ way to free memory
Memory Management
.NET
new, a = null
Leaks are not leaks in the classical sense of meaning
Automatic collection of not referenced objects with the
Garbage Collector
time of free / calling ~Destructor is NOT predictable
MyFontObj.Dispose() - C#: using (MyFontObj)
{ .. }
GC Heap
Segments
- Reserved “chunks” of virtual memory
Small Object Heap (SOH) segment(s)
- Objects categorized in generations
- Gen 0, 1 & 2
Large Object Heap (LOH) segments
- “House” large objects
New object created at increasing addresses of the heap
Allocation is very fast
21
Generations: Structure and Benefits
One heap for generations 0, 1, 2 and large objects
- Large Object is > 85,000 bytes
Efficient memory usage
- Locality of reference
Objects close together on heap
Allocated consecutively and stored contiguously
Created at same time and therefore related
- Speed of allocations
Locality of reference
Allocations located by adding value to a pointer
22
GC Heap
The oldest objects are in the oldest heap segment, new objects are created in the youngest heap segment.
The Heap will expand/contract through adding/removing the youngest heap segments
Ephemeral generations (gen0 and gen1) live in the same heap segment
Gen2 lives in its own segments or share the ephemeral (youngest) segment.
oldest youngest
Gen 2 Gen 1 Gen 0 freeGen 2
What can trigger a garbage collection?
Allocation request that cannot be satisfied
Explicit call to GC.Collect()
Under high system memory pressure
24
Display the heap with sos..
!EEHeap
...
Number of GC Heaps: 1
generation 0 starts at 0x0a788fc0
generation 1 starts at 0x0a251000
generation 2 starts at 0x00ab1000
ephemeral segment allocation context: (0x0a7b0c64, 0x0a7b0fcc)
segment begin allocated size
0x00ab0000 0x00ab1000 0x01a64ca0 0x00fb3ca0(16,465,056)
0x0b250000 0x0b251000 0x0bddd068 0x00b8c068(12,107,880)
0x03b40000 0x03b41000 0x04b3fbf4 0x00ffebf4(16,772,084)
0x04b40000 0x04b41000 0x05b3fbdc 0x00ffebdc(16,772,060)
0x05b40000 0x05b41000 0x06b3fbf4 0x00ffebf4(16,772,084)
0x06b40000 0x06b41000 0x07b3fbdc 0x00ffebdc(16,772,060)
0x07b40000 0x07b41000 0x08b3fbdc 0x00ffebdc(16,772,060)
0x09250000 0x09251000 0x09d17c28 0x00ac6c28(11,299,880)
0x0a250000 0x0a251000 0x0a7b0fcc 0x0055ffcc(5,636,044)
Large object heap starts at 0x01ab1000
segment begin allocated size
0x01ab0000 0x01ab1000 0x01af3830 0x00042830(272,432)
Total Size 0x7ba2ca8(129,641,640)
------------------------------
GC Heap Size 0x7ba2ca8(129,641,640)
0x0a7b0fcc
-0x0a251000
=
0x0055ffcc
size
Syncblk Index
Ptr to MethodTable
Instance Data
Object A
Objects in Managed Memory SyncBlock Table
Syncblk Index
Ptr to MethodTable
Instance data
Object B
MethodTable for ClassA
MethodTable for ClassB
gcinfo
Ptr to EEClass
Virtual pointers
gcinfo
Ptr to EEClass
Virtual pointers
Opt Padding 64Bit
Opt Padding 64Bit
• Locality of reference
• Allocated consecutively and stored contiguously
• Allocations located by adding value to a pointer
Object Layout in Memory
The Gargabe Collection Process
Gen 0Gen 1
MarkPlanRelocate+Compactt
Gen 2
31 2 5 6 7 98 104 11 12 13 14 15 16
GC Sequence in Motion
28
GC Sequence Non-Concurrent
Markgc_heap::mark_phase
Sweepmake_free_lists
Plangc_heap::plan_phase
Relocategc_heap::relocate_phase
Compactgc_heap::compact_phase
Compact?
SuspendEE
RestartEE
No
Yes
29
Non-Concurrent Workstation GC
30
What’s new in 4.0?
Background GC vs Ephemeral GC
Full GC is running in the Background
If required ephemeral collection is possible at the same time
- Background GC is suspended while the allocating thread is doing the
collection
For Concurrent Workstation mode only < .NET 4.5
31
Concurrent Workstation GC
Background GC in action!
32
Server GC
33
Server GC Notification within .NET 4.0
public static void Main(string[] args)
{
try
{
// Register for a set of notifications.
// Parameters require tuning. First is
// for Gen2, second, Large Object Heap
GC.RegisterForFullGCNotification(10, 10);
// Start a thread using WaitForFullGCProc
Thread thWaitForFullGC = new Thread(new
ThreadStart(WaitForFullGCProc));
thWaitForFullGC.Start();
}
catch (InvalidOperationException invalidOp)
{
Console.WriteLine("GC Notifications are
not supported while concurrent GC is
enabled.\n” + invalidOp.Message);
}
}
public static void WaitForFullGCProc()
{
while (true)
{
// Wait for a notification
GCNotificationStatus s =
GC.WaitForFullGCApproach();
if (s == GCNotificationStatus.Succeeded)
{
// This call will direct new traffic
// away from machine; wait for old
// traffic to finish; then call
// GC.Collect()
OnFullGCApproachNotify();
}
// Wait for a notification of completion
s = GC.WaitForFullGCComplete();
if (s == GCNotificationStatus.Succeeded)
{
OnFullGCCompleteEndNotify();
}
}
}
Concurrent workstation GC –(mscorwks::WKS)
Non-concurrent workstation GC (mscorwks::WKS)
Server GC (mscorwks::SVR)
Design goal Balance throughput and responsiveness for client application with GUI
Maximize throughput on single-processor computer
Maximize throughput on MP machine for server apps which creates multiple threads to handle same type of requests
# of heaps 1 1 1 per core
Threads which performs GC
Dedicated thread withtimeout
The thread that does the allocation
Dedicated thread/corewith no timeout
EE Suspension
EE is suspended much shorter, but several times during GC
EE is suspended during GC
EE is suspended during GC (.NET < 4.5)
Config file setting
<gcConcurrentenabled = true/>
<gcConcurrentenabled = false/>
<gcServer enabled = true/>
API GCLatencyMode.Interactive
GCLatencyMode.Batch
GC Modes .NET 4.x
35
.NET 4.5: Background GC For Server
Mode (On, Off)
<configuration>
<runtime>
//default is client/workstation<gcServer enabled="true"/>
//default is background, this disables it<gcConcurrent enabled="false"/>
</runtime>
</configuration>
.NET 4.5: Background GC For Server
Mode Gen0/Gen1 collections can proceed during Gen2 GC
.NET 4.5.1CompactOnce - next time a full GC occurs the LOH will be compacted.
Once done, it reverts back to the default behavior which is non compacting.
GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
// This will cause the LOH to be compacted (once).
GC.Collect();
Maximizing GC Performance
Consider problems caused by the following :
Allocating too many temporary objects
- E.g. “Chunky” ASP.NET page request
Pre-allocating objects
Too many roots (keeping objects alive)
Pointer/Reference rich objects
- E.g. “Kitchen Sink” static/GCHandle cache such as Hashtable
Too many almost-long-lived objects (generational churn)
Pinning (GCHandleType.Pinned, Interop, fixed location)
Too many Weak references to really small objects
Extensive using of the LOH results in fragmentation
39
Roots
Primary Roots:
- Objects on threads still in use
- Objects allocated on GC Handle
Tree of object references (Object Graph) leading to roots not
collectable
GC Handle
- Purpose: Prevent objects from GC’ed
- Type: Strong, Weak & Pinned
!GCRoot to locate roots in Debugger
40
Finalization: Problems
Releases resources before GC
Problems
- Resource-intensive
- Objects live longer
- At least two GCs required
- Only one Finalizer thread
• May get backlogged or blocked
41
Finalization and Garbage Collection - Start
1. GC is unable to fulfil the memory request.
2. GC marks all objects which are „reachable“
3. All objects which are not reachable and have an entry in theFinalisation list move to the „far reachable“ Freachable list
A B C D E FG H I J
C E F I J
Finalization List
Managed Heap
Freachable List
ROOTS
(strong
references)
Global,
Static, Local
Variables+
CPU Registers
Finalization: Programming
Clean up managed or unmanaged resources before Finalize
method is called
Don’t use Finalize method to clean up managed resources
Use Dispose method also in finalization code
Use the Dispose pattern:
Dispose(true) called from user’s code
Dispose(false) called by CLR
GC.SuppressFinalize()
43
Finalization: Tips
Use finalizers sparingly
Use with objects that have few object pointers
Use short finalization code path
C# using statement is a best practice
Monitor performance
- Perfmon counters for finalization
- Dumps and !Finalizequeue SOS command
44
Demo: Finalizers
The instructor will demonstrate how
application behavior differs when
calling GC.SuppressFinalize.
Memory Leaks
Unmanaged
- Symptom: Private Bytes increase while #Bytes In All Heaps
remains flat
- Troubleshooting path: Various tools (DebugDiag, LeakDiag, etc.)
Managed
- Symptom: Private Bytes and #Bytes In All Heaps both increase
- Troubleshooting path: For most cases, one or more dump files
46
Symptoms in ASP.NET
Out Of Memory Exceptions
Memory Recycling
• memoryLimit
• PeriodicRestartMemory
• PeriodicRestartPrivateMemory
System event log: worker process recycles if memory limit set
Log Name: System
Source: Microsoft-Windows-WAS
Date: 7/16/2010 1:15:27 AM
Event ID: 5117
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: <COMPUTER NAME>
Description:
A worker process with process id of '7420' serving application pool
'DefaultAppPool' has requested a recycle because it reached its private
bytes memory limit.
47
Perfmon Counters
General
.NET CLR Loading
• Current AppDomains
• Current Assemblies
• Bytes In LoaderHeap
.NET CLR Memory
• # Bytes In All Heaps
Process
• Handle Count
• Private Bytes
• Virtual Bytes
ASP.NET-specifc
ASP.NET Application
• Cache Total Entries
• Cache API Entries
• OutputCache Total Entries
48
Memory – Troubleshooting
Performance Counter can be used to indentify the
kind of leak
Native and managed leaks
- set complus_GCBreakOnOOM=1
can used to get DebugBreak() on OOM
- Can be use with the adplus –CTCFG option to get a dump
Memory – Troubleshooting sequence
Yes
.NET CLR Memory: Bytes in all Heaps go upProcess: Private Bytes go up
Obtain Perfmon log (started preferably
during process startup) and 2
process dumps 5-7 minutes apart during problem occurrence
Use DebugDiag or Xperf, LeakDiag
Obtain Perfmon log (started preferably
during process startup) and process
dump during problem occurrence
at least 2 hang dumps during
problem occurrence -> with some time gap between to see the memory growthAnalyse with Windbg
or Perfview
memory leak
net issue?
No
.NET CLR Memory: Bytes in all Heaps stay flatProcess: Private Bytes are going up
Tools for Memory Leakage
Native
Applocation Tracing using DebugDiag, PerfView, Windows Performance Toolkit (Wprui/Xperf)
Managed
Dump Analysis with Windbg or Visual Studio 2013 Ultimate or PerfViewAllocation Tracing with PerfView using the Collect Feature with
• .NET Sample Alloc (fast) • .NET Alloc
Demo: managed Memory with Windbg and PerfView
Troubleshooting Memory ASP.NET Issues
Historically for x86, if we get OOM exceptions and private bytes are
>= 800 MB we are either leaking memory or just have high memory
consumption
- Use VMMAP, Perfmon, or a debugger to narrow down where
memory is used
- Use DebugDiag for native leaks or PerfView or WinDbg for
managed memory leaks
53
Troubleshooting Memory ASP.NET Issues
(Cont.)
If private bytes are < 700 MB and we get OOM exceptions we have
likely fragmented the memory space somehow and need to figure
out why our memory is fragmented
- Use VMMAP in conjunction with WinDbg to determine what is
fragmenting the memory space.
54
useful commands for Windbg
Displays the size and locations of the managed heaps!sos.eeheap [-gc|-loader]
Displays statistics of all managed objects still in memory!sos.dumpheap –stat
Displays statistics of all managed objects of a specific type still in memory!sos.dumpheap –mt <MethodTable> or!sos.dumpheap –type <type>
Displays the size of a specific object and all it’s children!sos.objsize <addr>
Displays who is holding on to a specific object!sos.gcroot <addr>
Displays all appdomains and the assemblies that are loaded in each domain!sos.dumpdomain
Write a log to get a for the CLR Profiler
!TraverseHeap
References
Garbage Collection Performance Hints:
- http://msdn.microsoft.com/en-us/library/ms973837.aspx
Large Object Heap:
- http://msdn.microsoft.com/en-us/magazine/cc534993.aspx
VM Hoarding:
- http://blogs.msdn.com/b/maoni/archive/2005/10/03/476750.aspx
56