1
Improving Direct-Mapped Cache Performance by the Addition
of a Small Fully-Associative Cache and Prefetch Buffers
BySreemukha KandlakuntaPhani Shashank Nagari
2
Outline
Base Line Design
Reducing Conflict Misses
Miss Caching
Victim Caching
Reducing Capacity and Compulsory Misses
Stream Buffers
Multi-way Stream Buffers
3
Base Line Design
4
Base Line Design Contd..
• Size of on-chip caches usually varies
• High speed technologies result in smaller on chip caches
• L1 caches are assumed to be direct mapped
• L1 cache line sizes – 16 - 32 B
• L2 cache line sizes – 128-256B
5
Parameters assumed
• Processor Speed: 1000 MIPS
• L1 Inst and Data Cache
Size : 4Kb
Line Size : 16B
• L2 Inst and Data Cache
Size : 1MB
Line Size : 128B
6
Parameters assumed Contd..
Miss Penalty
L1- 24 Inst times
L2- 320 Inst times
7
Test Program Characteristics
8
Base Line system L1 Cache Miss Rates
9
Base Line DesignPerformance
10
Inferences
• Potential performance loss in memory hierarchy
• Improving performance of memory hierarchy rather than CPU performance
• H/w Techniques are used for improving the performance of the baseline M-H
11
How Direct Mapped Cache works
Main Memory
Tag Data Block Number
01110101100011010
100011010
001
000111
010
000111110101
000111110101100011
000
010011
101100
0000000000000000
0101010101010101
1010101010101010
111111111111
011010001000
111110101100
Direct Mapped Cache with 8 Blocks
How to identify?
•Match the Tag
•Tag 01 in block 001 means address 01001 is there
How to search
•00101, 01101, 10101, 11101 maps to block 101
001
001
001
001001
001
12
How Fully-associative Cache works
Main Memory
Tag Data Block Number
110101100011010001
100011010001000111
010001000111110101
000111110101100011
000
001010011
101100
0000000000000000
0101010101010101
1010101010101010
111111111111
011010001000
111110101100
Fully Associative Cache with 8 Blocks
Where to search?
•Every Block in Cache
•Very Expensive
13
Cache Misses
• Three Kinds
- Instruction read miss: Causes most delay, CPU has to wait until the instruction Is fetched from the DRAM
- Data read miss: Causes less delay, Inst not dependent on cache miss can continue execution until data is returned from DRAM
- Data write miss: causes least delay, write can be queued & CPU can continue until queue is full
14
Types of Misses
• Conflict Misses
Reduced by caching : Miss and Victim
• Compulsory Misses
• Capacity Misses
Both are reduced by prefecthing:
Stream Buffers
Multi-way Buffers
15
Conflict Miss
• Conflict Misses are the misses which would not occur if the cache was Fully associative and had LRU
• If an item has been evicted from the cache and the next miss corresponds to that item then that kind of miss is called the conflict miss
16
Conflict Misses Contd..
• Conflict Misses account to
– 20-40% of overall D-M misses
– 39% of L1-D$ misses
– 29% of L1-I$ misses
17
Conflict Misses,4Kb I&D
18
Outline
Base Line Design
Reducing Conflict Misses
Miss Caching
Victim Caching
Reducing Capacity and Compulsory Misses
Stream Buffers
Multi-way Stream Buffers
19
Miss Caching
• Small, Fully associative on-chip cache
• On Miss
Data is returned to
-Direct mapped cache
-Small Miss cache ( Where it replaces LRU item)
• Processor probes both D-M and Miss cache
20
Miss cache Organization
21
Observations
• Eliminates long off-chip miss penalty • More data conflicts misses are removed than
Instruction conflict misses- Instructions within a procedure do not
conflict as long as the procedure size is < cache size
- If an instruction within the program calls another program which may be mapped else where, a conflict arises- instruction conflict
22
Miss Cache Performance
• For 4 KB D$ size - Miss cache of 2 entries can remove 25% of D$ conflict misses i.e. 13% of overall D$ misses- Miss cache of 4 entries can remove 36% of D$ conflict misses i.e. 18% of overall D$ misses
• After 4 entries the improvement is minor
23
Conflict Misses removed by Miss caching
24
Overall Cache Misses removed by Miss Caching
25
Outline
Base Line Design
Reducing Conflict Misses
Miss Caching
Victim Caching
Reducing Capacity and Compulsory Misses
Stream Buffers
Multi-way Stream Buffers
26
Victim Caching
• Duplication of the data wastes storage space in miss cache
• Loads F-A cache with victim line from the
D-M cache
• When data misses in the D-M cache but hits in the Victim cache, contents are swapped
27
Victim Cache Organization
28
Victim Cache Performance
• Victim cache consisting of just one line is better than miss cache consisting of 2 lines
• Significant improvement in the performance of all the benchmark programs
29
Conflict Misses removed by Victim Caching
30
Overall Cache Misses removed by Victim Caching
31
Comparison of Miss cache and Victim cache performances
32
Effect of D-M cache size on Victim cache performance
• Smaller D-M caches – Most benefited due to addition of victim cache
• As D-M cache size increases, likelihood of conflict misses removed by victim cache decreases
• As the percentage of conflict misses decreases, the percentage of these misses removed by victim cache decreases
33
Victim cache: vary direct-map cache size
34
Effect of Line Size on Victim CachePerformance
• As line size increases the number of conflict misses increases
• As a result percentage of misses removed by victim cache increases
35
Victim cache: vary data cache line size
36
Victim caches and L2 Caches
• Victim caches are also useful for L2 caches due to large line sizes
• Using L1 victim cache can also reduce the number of L2 conflict misses
37
Outline
Base Line Design
Reducing Conflict Misses
Miss Caching
Victim Caching
Reducing Capacity and Compulsory Misses
Stream Buffers
Multi-way Stream Buffers
38
Reducing Capacity and Compulsory Misses
• Compulsory Misses
First reference to a piece of data
• Capacity MissesDue to insufficient cache size
39
Prefetching Algorithms
• Prefetch Always : Access to line “i” implies to prefetch access for “i+1”
• Prefetch on miss : Reference to block “i”
causes prefetch to block “i+1” Iff the block was a miss
• Tagged Prefetch : Tag bit is set to `0` when a block is prefetched and to set 1 when block is used
40
Limited Time For Prefetch
41
Outline
Base Line Design
Reducing Conflict Misses
Miss Caching
Victim Caching
Reducing Capacity and Compulsory Misses
Stream Buffers
Multi-way Stream Buffers
42
Stream Buffers
• Prefetched lines are placed in buffer in order to avoid polluting
• Each entry consists of tag ,an available bit and data line
• If a reference misses in the cache but hits in the buffer , the cache can be reloaded
• When a line is moved from the SB , entries in the SB shift up and new successive data is fetched
43
Stream Buffer Mechanism
44
Stream Buffer Mechanism Contd..
• On Miss Prefetch successive lines Enter tag for address in to the SB Set available bit to false
• On return of the prefetched data Place data in entry with its tag Set available bit to true
45
Stream Buffer Performance
• Most instruction references break the purely sequential access pattern by the time the 6th successive line is fetched
• Data references end even sooner
• As a result , Stream buffers show better performance at removing I$ misses
46
Sequential SB performance
Limitations of Stream Buffers
• Stream buffers considered are FIFO queues
• Head of the queue has tag comparator
• Elements must be removed strictly in sequence
• Works only for sequential line misses
• Fails for a non-sequential line miss
47
48
Outline
Base Line Design
Reducing Conflict Misses
Miss Caching
Victim Caching
Reducing Capacity and Compulsory Misses
Stream Buffers
Multi-way Stream Buffers
49
Multi-way stream buffers
• Single data SB`s could remove 72% of I$ misses and 25% of D$ misses
• Multi-way SB was simulated- to improve performance of SB`s for data references
• Consists of 4 SB in parallel
• On Miss the least recently Hit SB is cleared and data is started fetching from the miss address
50
Multi-way stream Buffer Design
51
Observations
• Performance of the instruction stream remains virtually unchanged
• Significant improvement in the performance of the data stream
• Removes 43% of the misses for the test programs i.e. almost twice the performance of single SB
52
Four-way SB performance
53
SB Performance Vs Cache size
54
SB Performance Vs Line size
55
Performance Evaluation
• Over the set of 6 benchmarks on an average 2.5% of 4KB D-M D$ misses that hit in a 4 entry victim cache also hit in a 4 way SB
• The combination of buffers and victim caches reduces the L1 miss rate to less than half of that of the base line system
• Resulting in an average of 143% improvement in system performance for the 6 benchmarks
56
Improved System Performance
57
Future Enhancements
• The study has concentrated on applying these H/W techniques to L1 caches
• Application of these techniques to L2 caches forms an interesting area of future work
• Performance of victim caching and stream buffers can be investigated for OS design and for multi-programming work loads
58
Conclusions
• Miss caches remove tight conflicts where several addresses map to the same cache line
• Victim caches are an improvement to miss caching that save the victim of the cache miss
• Stream buffers prefetch cache lines after missed cache line
• Multi-way stream buffers are a set of stream buffers that can do concurrent prefetches
59