Top Banner
2003 Exchange PROGRESS   Some Surprising Benefits of Using Storage Areas - and - Managing A Parallel Dump And Load In A Tight Window T om Bascom [email protected]
20

Progress surprises

Apr 03, 2018

Download

Documents

casorohi
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Progress surprises

7/28/2019 Progress surprises

http://slidepdf.com/reader/full/progress-surprises 1/20

Page 2: Progress surprises

7/28/2019 Progress surprises

http://slidepdf.com/reader/full/progress-surprises 2/20

© 2003 Progress Software CorporationBP0430: Some Surprising Bebefits Of Using Storage Areas - 2   S   i  m

  p   l   i   f  y  y  o  u  r   b  u  s   i  n  e  s  s

2003 Exchange 

PROGRESS  

Opening Remarks… 

Storage Areas

Dumping & Loading

Surprising Results

Page 3: Progress surprises

7/28/2019 Progress surprises

http://slidepdf.com/reader/full/progress-surprises 3/20

© 2003 Progress Software CorporationBP0430: Some Surprising Bebefits Of Using Storage Areas - 3   S   i  m

  p   l   i   f  y  y  o  u  r   b  u  s   i  n  e  s  s

2003 Exchange 

PROGRESS  

Storage Areas

Structure for Grouping Tables and Indexes Administrative Benefits

 – Index Rebuilds

 – Dump & Load

 – Dropping Areas

 – DB Analysis

Drawbacks

 –More complex structure

 – No single “high water” mark 

 – More “overhead” (free space in areas) 

Page 4: Progress surprises

7/28/2019 Progress surprises

http://slidepdf.com/reader/full/progress-surprises 4/20

© 2003 Progress Software CorporationBP0430: Some Surprising Bebefits Of Using Storage Areas - 4   S   i  m

  p   l   i   f  y  y  o  u  r   b  u  s   i  n  e  s  s

2003 Exchange 

PROGRESS  

Dump & Load Choices

“Classic” Dictionary Dump & Load  Bulk Loader 

Binary Dump & Load

Parallelization

 – Many Benefits

 – Difficult to Effectively Balance

Automating the process Index Rebuild – inline vs single thread

Exotic techniques – buffer copy…, -RO, TableFilters

Page 5: Progress surprises

7/28/2019 Progress surprises

http://slidepdf.com/reader/full/progress-surprises 5/20

© 2003 Progress Software CorporationBP0430: Some Surprising Bebefits Of Using Storage Areas - 5   S   i  m

  p   l   i   f  y  y  o  u  r   b  u  s   i  n  e  s  s

2003 Exchange 

PROGRESS  

Classic Dictionary D&L

Order-lineCust order hist

cust Order-line order 

1st the dump… 

… and then the load. 

Page 6: Progress surprises

7/28/2019 Progress surprises

http://slidepdf.com/reader/full/progress-surprises 6/20

© 2003 Progress Software CorporationBP0430: Some Surprising Bebefits Of Using Storage Areas - 6   S   i  m

  p   l   i   f  y  y  o  u  r   b  u  s   i  n  e  s  s

2003 Exchange 

PROGRESS  

Multi-threaded D&L

Order-line

cust

order hist

cust

Order-line order 

1st the dump… 

… and then the load. 

Page 7: Progress surprises

7/28/2019 Progress surprises

http://slidepdf.com/reader/full/progress-surprises 7/20© 2003 Progress Software CorporationBP0430: Some Surprising Bebefits Of Using Storage Areas - 7   S   i  m

  p   l   i   f  y  y  o  u  r   b  u  s   i  n  e  s  s

2003 Exchange 

PROGRESS  

Load Balanced D&L

Order-line

cust

order hist

cust

Order-line

order 

Both the dump… 

… and the load – in

parallel, multi-

threaded and load-

balanced.

Page 8: Progress surprises

7/28/2019 Progress surprises

http://slidepdf.com/reader/full/progress-surprises 8/20© 2003 Progress Software CorporationBP0430: Some Surprising Bebefits Of Using Storage Areas - 8

   S   i  m

  p   l   i   f  y  y  o  u  r   b  u  s   i  n  e  s  s

2003 Exchange 

PROGRESS  

Drivers

What is the window?

What are the goals?

What resources are available?

Page 9: Progress surprises

7/28/2019 Progress surprises

http://slidepdf.com/reader/full/progress-surprises 9/20

Page 10: Progress surprises

7/28/2019 Progress surprises

http://slidepdf.com/reader/full/progress-surprises 10/20© 2003 Progress Software CorporationBP0430: Some Surprising Bebefits Of Using Storage Areas - 10

   S   i  m

  p   l   i   f  y  y  o  u  r   b  u  s   i  n  e  s  s

2003 Exchange 

PROGRESS  

The Goals

Reduce Downtime for Maintenance

 – Index Rebuilds

 – Dump & Load

 –Application Maintenance

Reduce Disk Footprint

Improve Reporting Performance

Page 11: Progress surprises

7/28/2019 Progress surprises

http://slidepdf.com/reader/full/progress-surprises 11/20© 2003 Progress Software CorporationBP0430: Some Surprising Bebefits Of Using Storage Areas - 11

   S   i  m

  p   l   i   f  y  y  o  u  r   b  u  s   i  n  e  s  s

2003 Exchange 

PROGRESS  

Resources

Production Server (HP rp7410) – 8 x 750mhz

 – 16GB RAM

 – HPUX 11i

Failover Server (HP V2250)

 –16 x 240mhz

 – 8GB RAM

 – HPUX 11

Old EMC Disk Array Symmetrix 8100 –

Database striped (and mirrored) across 16 disks New EMC Disk Array Symmetrix 8830

 – 4 Meta volumes of 4 LUNs

Page 12: Progress surprises

7/28/2019 Progress surprises

http://slidepdf.com/reader/full/progress-surprises 12/20© 2003 Progress Software CorporationBP0430: Some Surprising Bebefits Of Using Storage Areas - 12

   S   i  m

  p   l   i   f  y  y  o  u  r   b  u  s   i  n  e  s  s

2003 Exchange 

PROGRESS  

The SA Design

Separate Tables from Indexes

Isolate “large” Tables (80/20 rule) 

 – Any table over 1 million rows

 –Any table over 1GB

Isolate Transaction Data from Overhead Data

 – Easily identifiable in dictionary

256 Records per Block

Page 13: Progress surprises

7/28/2019 Progress surprises

http://slidepdf.com/reader/full/progress-surprises 13/20

© 2003 Progress Software CorporationBP0430: Some Surprising Bebefits Of Using Storage Areas - 13   S   i  m

  p   l   i   f  y  y  o  u  r   b  u  s   i  n  e  s  s

2003 Exchange 

PROGRESS  

The SA Design -- continued

80 Storage Areas 40 Data, 40 Index

 – 1 “miscellaneous” transaction data 

 – 1 “miscellaneous” transaction indexes 

 – 1 “miscellaneous” overhead data 

 – 1 “miscellaneous” overhead indexes 

 – 38 large table data areas

 –38 index areas for large tables

 – Plus schema, bi & ai areas

Page 14: Progress surprises

7/28/2019 Progress surprises

http://slidepdf.com/reader/full/progress-surprises 14/20

© 2003 Progress Software CorporationBP0430: Some Surprising Bebefits Of Using Storage Areas - 14   S   i  m  p   l   i   f  y  y  o  u  r   b  u  s   i  n  e  s  s

2003 Exchange 

PROGRESS  

Load Balanced Binary D&L

Page 15: Progress surprises

7/28/2019 Progress surprises

http://slidepdf.com/reader/full/progress-surprises 15/20

© 2003 Progress Software CorporationBP0430: Some Surprising Bebefits Of Using Storage Areas - 15   S   i  m  p   l   i   f  y  y  o  u  r   b  u  s   i  n  e  s  s

2003 Exchange 

PROGRESS  

Parallel Binary Dump & Load

24 Dump Threads – Effect of adding threads

47 Load Threads – Resource contention issues

Tabanalys

Expected D&L Time is Longest Combined D&L of asingle given table – Longest dump may not be longest load

 – And vice versa

 – Some variability is possible

 – Other d&l threads can overlap – reducing contention

Page 16: Progress surprises

7/28/2019 Progress surprises

http://slidepdf.com/reader/full/progress-surprises 16/20

© 2003 Progress Software CorporationBP0430: Some Surprising Bebefits Of Using Storage Areas - 16   S   i  m  p   l   i   f  y  y  o  u  r   b  u  s   i  n  e  s  s

2003 Exchange 

PROGRESS  

(Expected) Results

Old New

DB Size 110GB 64GB

Tabanalys 10.0h 1.0h

Warehouse Extract 90m 15m

Page 17: Progress surprises

7/28/2019 Progress surprises

http://slidepdf.com/reader/full/progress-surprises 17/20

© 2003 Progress Software CorporationBP0430: Some Surprising Bebefits Of Using Storage Areas - 17   S   i  m  p   l   i   f  y  y  o  u  r   b  u  s   i  n  e  s  s

2003 Exchange 

PROGRESS  

Application Load

Graphs show before &after application load

(unchanged)

 – Logical Reads

 –Commits/sec

Page 18: Progress surprises

7/28/2019 Progress surprises

http://slidepdf.com/reader/full/progress-surprises 18/20

© 2003 Progress Software CorporationBP0430: Some Surprising Bebefits Of Using Storage Areas - 18   S   i  m  p   l   i   f  y  y  o  u  r   b  u  s   i  n  e  s  s

2003 Exchange 

PROGRESS  

Surprise!

Dramatic Reduction in: – OS Reads

 – Latch Contention

Page 19: Progress surprises

7/28/2019 Progress surprises

http://slidepdf.com/reader/full/progress-surprises 19/20

© 2003 Progress Software CorporationBP0430: Some Surprising Bebefits Of Using Storage Areas - 19   S   i  m  p   l   i   f  y  y  o  u  r   b  u  s   i  n  e  s  s

2003 Exchange 

PROGRESS  

Why?

Homogenous Data Blocks Take Better Advantage of “Locality of Reference” 

 – LOR says that most data access is “close” toprevious access

 –Mixed data blocks scatter data widely Prior to d&l a block with 64 slots might have 3

records from a table; best case hit ratio for atable scan is about 85%

Post d&l it has 256; best case hit ratio becomes99.98%

Large number of SAs enables more tables toshare in benefit

Page 20: Progress surprises

7/28/2019 Progress surprises

http://slidepdf.com/reader/full/progress-surprises 20/20

© 2003 Progress Software CorporationBP0430: Some Surprising Bebefits Of Using Storage Areas - 20   S   i  m  p   l   i   f  y  y  o  u  r   b  u  s   i  n  e  s  s

2003 Exchange 

PROGRESS  

Questions 

http://www.greenfieldtech.com/downloads.shtml