TURKCELL DBAs Metin YILMAZ & Ferhat ŞENGÖNÜL TURKCELLs EXADATA Journey September 2010 ORACLE RACK Pack Team Sanjay SINGH S316103 : Life with the Sun Oracle.
Post on 29-Mar-2015
216 Views
Preview:
Transcript
TURKCELL DBAsMetin YILMAZ & Ferhat ŞENGÖNÜL
TURKCELL’s EXADATA Journey
September 2010
ORACLE RACK Pack TeamSanjay SINGH
S316103 : Life with the Sun Oracle Database Machine v2
Turkcell Bio
• Leading Telco in Turkey• 33.9 million subscribers in Turkey as of June 30, 2010• Third biggest GSM operator in Europe in terms of subscribers• Turkcell started its operations in February 1994• Turkcell co-operates in 8 other countries and has over 60
million subscribers total• First and only Turkish company listed on NYSE
• Over 500 Oracle Databases, 150+ Production• DB Machine hosts our biggest Database from DWH Domain
Overview
• Turkcell’s largest 100 TB (~250 TB uncompressed) DB was migrated to DBM v2, now only 25 TB with the help of HCC on Full SAS Rack
• There are over 50K reports run every month on this DB, Performance improvement up-to 400x, average 10x
• Ability to scale linearly: add one more Rack and forget
Business Drivers - Why Exadata ?
EMC DMX-470 TB
HITACHI USP-V50 TB
Sun M9000 Sparc 7176 Threads
ORACLE Exadata V2
OLD SYSTEM NEW SYSTEMServer Model Sun M9000 Oracle Exadata V2Cpu Type Sun Sparc 7 2.52 GHz Xeon® E5540 Processors 2.53 GHzNumber Of Cpu Threads 176 128Total Main Memory 512 GB 576 GBTotal Storage Capacity 120 TB 30 TBStorage Connection Technology Fiber Channel (32 x 4 Gtbit/s) Infiniband ( 8 x 40 GbitStorage Maximum IO Throughput Capacity 5 GB/s 21 GB/sServer + Storage units Total Power 57 KVA 20 KVAServer + Storage units Total Form Factor 11 Rack 1 RackApproximate Data Backup Duration 44 Hours 14 Hours Number of Backup Tape Cartridges Per Backup 159 57
Business Drivers - Why Exadata ?
Simplified Architecture• Full Rack replaces Sun M9000 & ten(10) Storage Cabinets• Single Vendor Strategy
• It took Turkcell a few years to perfect current environment – Exadata up & running in 5 days
• We need to address/deal with one party if a problem occurs
Effortless Scalability: one to Multi-Rack• Data volume explodes – data size doubles every year (45TB to 100TB)• Old storage environment was maxed out – no chance to scale out• Old env 600+ (EMC DMX4 & Hitatchi USP-V) disks with max I/O
throughput of 5 GB/sec. vs 168 SAS disks with a max I/O throughput of 21 GB/sec.
Business Drivers - Why Exadata ?0-
5 dk
5-10
dk
10-3
0 dk
30-4
5 dk
45-6
0 dk
1-1.
5 sa
at1.
5-2
saat
2-2.
5 sa
at2.
5-3
saat
3-3.
5 sa
at3.
5- 4
saa
t4
saat
uze
ri0-
5 dk
5-10
dk
10-3
0 dk
30-4
5 dk
45-6
0 dk
1-1.
5 sa
at1.
5-2
saat
2-2.
5 sa
at2.
5-3
saat
3-3.
5 sa
at3.
5- 4
saa
t4
saat
uze
ri0-
5 dk
5-10
dk
10-3
0 dk
30-4
5 dk
45-6
0 dk
1-1.
5 sa
at1.
5-2
saat
2-2.
5 sa
at2.
5-3
saat
3-3.
5 sa
at3.
5- 4
saa
t4
saat
uze
ri0-
5 dk
5-10
dk
10-3
0 dk
30-4
5 dk
45-6
0 dk
1-1.
5 sa
at1.
5-2
saat
2-2.
5 sa
at2.
5-3
saat
3-3.
5 sa
at3.
5- 4
saa
t4
saat
uze
ri0-
5 dk
5-10
dk
10-3
0 dk
30-4
5 dk
45-6
0 dk
1-1.
5 sa
at1.
5-2
saat
2-2.
5 sa
at2.
5-3
saat
3-3.
5 sa
at0-
5 dk
5-10
dk
10-3
0 dk
30-4
5 dk
45-6
0 dk
1-1.
5 sa
at1.
5-2
saat
2-2.
5 sa
at2.
5-3
saat
3.5-
4 s
aat
Temmuz (05-11) Temmuz (12-18) Temmuz (19-25) Temmuz (26-01) Agustos(02-08) Agustos(09-15)
0
1000
2000
3000
4000
5000
6000
2486
6481161
3141931771394049332987
2944
665
1403
4892712601125245351775
2227
5411023
3611791921075734282588
3284
454346883328 9 5 2 4 0 6
4827
376199342214 3 2 1 3
4750
4192523519 4 2 1 1 1
Weekly Report Count & Run-time
25 min 27 min 7 min 3 min 3 min25 min
Avg Time
Exadata Migration
Performance Needs• There over 50K reports run every month on this DB• Average report run time is reduced from 27 minutes to 3 minutes !!!• Reports completed less than 5 mins went from %45 to %90• Reports running more than 4 hours went from 87 to 1
The Project Overview
Planned as simple migration • No Application version or interface changes • Migrate to new Exadata V2 machine • Upgrade to new Oracle Version (from 10.2.0.4 to 11.2.0.1)• Move from Single Instance to RAC
4 Months (Apr-July) testing of all the components• Crash Tests (working closely with support)• Network Transfer Speed Tests (10G vs IB)• Backup/Restore Tests• Performance Tests (RAT, Report run times, data load times)
Turkcell’s BI Environment
ReportingMSTR
ETLAB Initio
Oracle ODISource
DBs
Exadata
Other DWH DBs
Amount Of Data • 1.5 Billion CDR per Day• 600 -1000 GB raw data extracted from 20+ source database• 5 TB Data on file system processed• 2-3 TB loaded into databases, mostly into Exadata• Approximately 500 TB Customer data stored in multiple Databases
• 250 TB (25 TB compressed) on Exadata, 250 TB (100 TB compressed) on non-Exadata
Turkcell’s BI Environment – Application Footprint
ETL Environment• AB Initio – runs on Sun E25K and 8 Node Linux Cluster• Data loaded daily between 19:00 - 08:00 (SLA) • Sql*Loader used to load tables with 16 parallel threads
Reporting Environment• MSTR (Microstrategy) is used mostly• 300+ Users • 3000 distinct reports with 50k run per month• Runs between 08:00 – 20:00 (SLA)
Project Challenges
• Will we fit into 30 TB ? (~100 TB 10g compressed)• How to move that much data in 2 days ?
• 100TB 10g compressed, how much of it can be moved before/after• How much data needs to be moved during mig window• What kind of Network infrastructure is needed to support such xfr rate
• 8-node RAC ? (earlier attempts to use RAC did not go through)• Training needs for RAC and exadata storage architecture
• Rollback plan: Parallel run (load data to both DB) Environment
Migration Facts
• Insert/append over DB Links • Platform and version change forced us to use insert over db-link• None of the other methods like TTS, ASM Rebalance was applicable
• Used in-house pl/sql utility to perform migration• Metadata (Tablespaces were re-organized)• Insert/Append with 128 parallel sessions partition-wise
• 40TB data was transferred during the migration window• Transfer rate 300mB/sec, around 1 TB per hour approximately• Completed in 36 hours. • High CPU load on comp nodes because of HCC (expected)• SQL*Net compressing the data 2-3x (this was a nice finding)
• 20 TB readonly data was transferred before, ~30TB after• Network Infrastructure
• 10Gb-to-IB (Voltaire 4036E) gateway used to connect old system
Migration Facts
• After the test migration we ended-up staying on Exadata• End users were so happy, they did not let us go back • 3 Weeks before the go-live date we were live
• Parallel runs continued for few weeks until we completely feel comfortable on Exadata• Stability of the system under real load was proved for various load
patterns• Backup/Restore tests were completed
• Questions on Migration ?
Compression in Action
Old System 10gR2 Compression• ~2-3 times ~250TB raw data to 100TB
Exadata V2 with EHCC• Raw Data 250TB to 25TB (Data)+ 5TB (Temp) = 30TB • EHCC - Compress ratio ~7-10x • Archive compression is efficient but high CPU consumption
http://ferhatsengonul.wordpress.com/2010/08/09/getting-the-most-from-hybrid-columnar-compression/
SORT COMPRESS SIZE (GB) RATIONOSORT NOCOMP 137,59 1NOSORT Q_HIGH 21,21 6,48SORT_A Q_HIGH 12,18 11,29SORT_B Q_HIGH 15,37 8,95SORT_A_B Q_HIGH 11,64 11,80
Performance GainsReport Name Old System Exadata by…X
CRC Control Report 0:15:48.73 0:05:06.07 X2
prepaid bireysel toplam harcanan kontor tl 8:02:10.59 1:51:33.20 X4.3eom_equipment_utilization report 0:38:17.77 0:00:23.34 X163Eom equipment arpu report 0:09:46.25 0:02:08.00 X4.5Son 10 gun icinde yapilan gonderimler 0:17:57.95 0:00:37.61 X45.7Rapor 01. Satış Kanalı Kırılımında Blackberry 0:03:22.24 0:00:00.66 X487
İnternet Paketleri Postpaid paketiçi ve paket 0:05:41.34 0:01:00.34
X4.3
Connectcard Fatura tutarları 0:31:32.38 0:00:46.51 X66.3
Connectcard Aktivasyon 0:25:21.00 0:00:54.88 X44.9
• There over 50K reports run every month• Performance improvement is up-to 400x for some reports and
on average it is 10x
Life with Exadata
No need to be scared about RAC• We use scan-IP, rest is assured: loads are distributed over the nodes• We use heavily PQ; so far no problems running slaves over multiple nodes.• Tons of log files could be a little overwhelming for troubleshooting• Temporary Tablespace should be at least twice as more
DB Machine is indeed redundant• We lost entire cell or individual hard disk or flash disk but database stayed
up, no interruptions for the service
Enterprise Manager is must• Grid Control • SQL Monitor
Implementation Challenges
Hardware Failure
Sun: issues with failed IB switch - all node hung.• Faulty switch replaced • Hanging due to configuration setup of switch
Disk Failure – no down time , everything continuing working• Disk replaced online – just replaced all taken care automaticaly
Issues work in progress• Cannot explain why cell encountered “root filesystem readonly”
• Bug: 9752748 • Not all the Instance’s are registered with the scan-vip after re-cycle listeners
• Registers after a few hours.• No impact when partly registered.
Implementation Challenges
SQL Loader session hangs on enq ST contention• Bug9772225 – “ST enqueue acquisition issue” - Complex fix in review• Bug9326041 - “Buffer pre-warm issue” W/A: _db_cache_pre_warm=FALSE
Exadata problem ORA-07445: core dump [kdzdcol_skip_slots()]• Bug 9655552 – EHCC optimization issues – W/A : _arch_comp_dbg_scan"=16
RS-7445 [Serv MS absent]• Bug 9252694 MS FREQUENTLY REPORTS LOW VIRTUAL HEAP MEMORY AND
SHUTS DOWN - Fixed 11.2.1.3.X• bug 9217088 : using memcpy (on cell nodes)• Disable chained row optimization in Exadata• bug.9338087 is fixed in 11.2.1.2.4
User Feedbacks
• We heard before that infrastructure changes would give us performance gains but this time we were surprised with it, it was well over our expectations. Now we can take faster actions in this competitive environment. Director of Marketing Department
• XDDS is fantastic in a single word, none of the reports take more than 10 minutes, It was taking 3-4 hours before now it completes in 3 minutes. It sounds like un-real but it is real. Power end-user from Finance Department
• It was a never ending race to match the business' performance and capacity needs. With the Database Machine V2, we have outperformed our user's expectations and we are prepared for the future growth.Veteran System Admin
• “You started to scare me MSTR” updated her status on facebook. End-user from Marketing Department
What is waiting for the future
• Adding second Rack• We’ll see if it will scale :) • Partition nodes for test/dev purposes
• Several more DWH Databases could be consolidated into another full Rack
• Evaluate Exadata for OLTP Environments• Real-Time DWH
I would like to Thank here to • Turkcell’s DB and OS Admins for completing migration before it’s
planned date and keeping it running successfully• Oracle RAC Pack Team: Sanjay, Su, Martina• Oracle DWH Global Leaders Team: Cemil, Reiner• Oracle Turkey’s local Sales Support and Support Organization
Thank YOU
Questions
www.turkcell.com.tr
Ferhat ŞENGÖNÜLferhat.sengonul@turkcell.com.tr
top related