Running A Megasite Running A Megasite On Microsoft On Microsoft Technologies Technologies Casey Jacobs Casey Jacobs Aber Whitcomb Aber Whitcomb Director of Engineering Director of Engineering CTO CTO Microsoft.com Microsoft.com MySpace.com MySpace.com Chris St.Amand Chris St.Amand Jim Benedetto Jim Benedetto Sr. System Engineer Sr. System Engineer VP of VP of Technology Technology Microsoft.com Microsoft.com MySpace.com MySpace.com NGW046 NGW046
MySpace and Microsoft.com are two of the most-visited Web sites on the planet. Come to this session to hear about lessons learned using Microsoft technologies to run Web applications on a massive scale. Representatives from Microsoft.com talk about lessons learned using an all-Microsoft datacenter. Representatives from MySpace talk about the realities of using Microsoft technologies in a scalable, federated environment using SQL Server 2005, .NET 2.0 and IIS 6 on Windows Server 2003 64-bit editions. This session features an open Q&A with a panel of technical managers and engineers from MySpace and Microsoft.com.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Running A Megasite On Running A Megasite On Microsoft TechnologiesMicrosoft Technologies
Casey JacobsCasey Jacobs Aber WhitcombAber WhitcombDirector of EngineeringDirector of Engineering CTOCTOMicrosoft.comMicrosoft.com MySpace.comMySpace.com
Chris St.AmandChris St.Amand Jim BenedettoJim Benedetto Sr. System EngineerSr. System Engineer VP of TechnologyVP of TechnologyMicrosoft.comMicrosoft.com MySpace.comMySpace.com
MySpace Company OverviewMySpace Company OverviewLaunched Sept, 2003Launched Sept, 2003Latest as of February 2006Latest as of February 2006
64+ MM Registered Users64+ MM Registered Users38 MM UUsers & 2.3M 38 MM UUsers & 2.3M ConcurrentConcurrent260K New Registered 260K New Registered Users/DayUsers/Day23 Billion Page* Views/Month23 Billion Page* Views/Month
DemographicsDemographics50.2% Female / 49.8% Male50.2% Female / 49.8% MalePrimary Age Demo: 14-34Primary Age Demo: 14-34
Site TrendsSite Trends260K New Users/Day260K New Users/Day430M Total Images 430M Total Images Millions of Songs Streamed/DayMillions of Songs Streamed/Day1000’s of New MP3’s/Day1000’s of New MP3’s/Day20 Million Comments Posted20 Million Comments Posted
Media Metrix February 2006 Audience RankingsMedia Metrix February 2006 Audience Rankings
Source comScore Media Metrix February - 2006
Internet RankInternet Rank Pageviews in ‘000sPageviews in ‘000s
YahooYahoo #1#1 29,50829,508
MySpaceMySpace #2#2 23,56623,566
MSNMSN #3#3 14,69514,695
EbayEbay #4#4 9,6329,632
GoogleGoogle #5#5 7,3297,329
HotmailHotmail #6#6 6,8126,812
MySpace.com MySpace.com Quick FactsQuick Facts
Infrastructure and Application FootprintInfrastructure and Application Footprint3 Internet Data Centers3 Internet Data CentersServer BreakdownServer Breakdown
2682 Web and 650 Database Servers2682 Web and 650 Database Servers90 Cache Servers 16gb RAM90 Cache Servers 16gb RAM650 Dart servers650 Dart servers60 DB Servers60 DB Servers150 Media servers150 Media servers
3000 disks in SAN architecture3000 disks in SAN architectureEgress ManagementEgress Management
17,000 mb/s bandwidth17,000 mb/s bandwidth15,000 mb/s on CDN15,000 mb/s on CDN
Transactional replication doesn’t work Transactional replication doesn’t work for all workloads and data typesfor all workloads and data types
Use a combination of Vertical Use a combination of Vertical Partitioning and replicationPartitioning and replication
1M Users
2 Million2 MillionSANSAN
Start to reconsider SCSI arrays for the Start to reconsider SCSI arrays for the long-termlong-termSCSI arrays have good performance SCSI arrays have good performance but reliability issuesbut reliability issuesSANS provide better performance, SANS provide better performance, uptime, and redundancyuptime, and redundancyMove to a clarion and enjoy better Move to a clarion and enjoy better these benefitsthese benefits
Vertical Partitions see Vertical Partitions see performance problemsperformance problems
Decide we need to re-architect the Decide we need to re-architect the databasedatabase
Horizontal partitioning is the Horizontal partitioning is the answer but is difficult to do while answer but is difficult to do while in productionin production
3M Users
Horizontal PartitioningHorizontal Partitioning
All features reside on All features reside on a single database servera single database server
Data is partitioned by user IDData is partitioned by user ID
Some data cannot be partitioned Some data cannot be partitioned especially on a social networking siteespecially on a social networking site
Trying to partition storage on the Trying to partition storage on the backend is time consuming and backend is time consuming and inefficientinefficient
Maxing out SANs is very costlyMaxing out SANs is very costly
We realize scalable storage is keyWe realize scalable storage is key
10M Users
15 Million15 MillionDB’s versus CachingDB’s versus Caching
Databases still having perf issuesDatabases still having perf issuesDatabases are expensiveDatabases are expensive
Have a lot of transactional overheadHave a lot of transactional overhead
Caching tierCaching tierHigh speed cache is perfect for readsHigh speed cache is perfect for reads
LRU algorithm is self managingLRU algorithm is self managing
Scale Up at Lower CostsScale Up at Lower Costs Data Center A
Database Mirroring
Web Cluster 1 Web Cluster 2
Principle Mirror
Sync / Async
Transactions
Data Center BData Center A
ICPSQL.PHX.GBL
Peer-To-Peer Replication
SQL A SQL B SQL C SQL D
NLB VIP 1 NLB VIP 2
MySpaceMySpaceScaling SQL ServerScaling SQL Server
V1: Single Instance – < 1 Million UsersV1: Single Instance – < 1 Million UsersSingle SQL Server Instance Supports All Users and FeaturesSingle SQL Server Instance Supports All Users and Features
V2: Single Instance Replicating to Read Only V2: Single Instance Replicating to Read Only Full Copies < 2 Million UsersFull Copies < 2 Million Users
Single server handles all write transactions, read Single server handles all write transactions, read transactions spread across multiple transactional transactions spread across multiple transactional replication copiesreplication copies
V3: Vertical Partitioning - < 4 Million UsersV3: Vertical Partitioning - < 4 Million UsersEach Feature/Page of the site on its own SQL ServerEach Feature/Page of the site on its own SQL Server
MySpaceMySpaceScaling SQL ServerScaling SQL Server
V4: Horizontal Partitioning - < 8 Million UsersV4: Horizontal Partitioning - < 8 Million UsersAll features/pages brought back to single database schemaAll features/pages brought back to single database schema
Standard schema across all databasesStandard schema across all databases
User ranges partitioned across databasesUser ranges partitioned across databases
V5: Horizontally Partitioned Core with Replicated V5: Horizontally Partitioned Core with Replicated Content, Vertically Partitioned Features Databases, Content, Vertically Partitioned Features Databases, “Shared Content” Databases - > 8 Million Users“Shared Content” Databases - > 8 Million Users
Primary Myspace schema exists across large farm of servers Primary Myspace schema exists across large farm of servers
Small amounts of content replicated to all horizontally partitioned Small amounts of content replicated to all horizontally partitioned servers to allow for features spanning all servers to allow for features spanning all user rangesuser ranges
V6: Migration to SQL Server 2005 - >26 Million UsersV6: Migration to SQL Server 2005 - >26 Million Users
SQL Server 2005SQL Server 200564 bit64 bit
Memory Pressure under 4GB 32 Limit Memory Pressure under 4GB 32 Limit Servers loaded with 32Gigs of RAMServers loaded with 32Gigs of RAM
<4 Gig Addressable to the memory pools we were <4 Gig Addressable to the memory pools we were stressingstressing
Problem Eliminated on 64bit ArchProblem Eliminated on 64bit Arch Connection/Sort memory pools now able to Connection/Sort memory pools now able to address all 32Gigs of RAMaddress all 32Gigs of RAM
Virtualizing StorageVirtualizing Storage
What is it?What is it?Software layer between your disks & hostsSoftware layer between your disks & hosts
AdvantagesAdvantagesProvisioning is very simple, makes capacity Provisioning is very simple, makes capacity planning more predictableplanning more predictable
Much better performanceMuch better performance
Can easily add more capacity to a LUNCan easily add more capacity to a LUN
What do we use?What do we use?3par3par
14 week bake off14 week bake off
Longhorn And IIS 7.0Longhorn And IIS 7.0Product technology enablersProduct technology enablers