A Front-Row Seat to Ticketmaster’s use of MongoDB! Ed Presz – VP, Database Services Linda Xu – Director Database Development/Architecture Tuesday, June 24 1:30-2:30PM
A Front-Row Seat to Ticketmaster’s Use of MongoDB
Aug 29, 2014
Ticketmaster is the world leader in selling tickets. After more than a decade of developing applications extensively on Oracle and MySQL, Ticketmaster made the move to MongoDB. The reasons for the move are generally in line with those of other companies – increased flexibility and performance, and decreased costs and time-to-market. In this session we’ll discuss how the conversion to MongoDB went at Ticketmaster and we’ll take a deeper dive into some of the successes and set-backs that we faced. We’ll give an overview of the MongoDB topology at Ticketmaster, discuss exactly what data we’re writing to MongoDB and comment on the MongoDB support model that we’re using. We’ll also touch on the transition from relational DBA to NoSQL DBA at Ticketmaster.
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
A Front-Row Seat to Ticketmaster’s use of MongoDB!
Ed Presz – VP, Database ServicesLinda Xu – Director Database Development/Architectur e
Tuesday, June 24 1:30 -2:30PM
Live Nation EntertainmentThe world's leading live entertainment and ecommerce company, comprised of
four market leaders: Ticketmaster.com, Live Nation Concerts, Front Line Management Group and Live Nation Network
Ticketmaster.comOne of the world's top five ecommerce sites
Database with 350 million customers/fans worldwide
27 million monthly unique visitors to www.ticketmaster.com and www.livenation.com
Overview – Live Nation/Ticketmaster
Ticketmaster and Live Nation
merged in 2010
www.livenation.com
> 15,000 tickets/minute sold during large on-sales
Live Nation Concerts produces over 22,000 shows annually for more than 2,300 artists
Ticketmaster was official ticket company for 2012 London Olympics
2
MongoDB at Ticketmaster
3
WriteConcern
Replica Sets JSON/BSON
getLastError collections documents
election
oplog
The Move to MongoDB
Ticketmaster Database Services team� 26 DBAs located globally� Strong Oracle/MySQL/MS SQL Server background� Consists of Operational DBAs, Architects and Database
Engineers� Database performance/scale is a huge part of what we
do
Transition to MongoDB � Flexible, development friendly� Ease of deployment/speed-to-market� Cost savings� Re-training relational DB experts (new challenge)� Performance trade-offs
MongoDB at Ticketmaster
February, 2012 �Ticketmaster starts evaluating MongoDB
June, 2012 �Begin development on MongoDB
May, 2013 �Enterprise Subscription support for MongoDB
July, 2013 July, 2013 �Go live with MongoDB 2.2.4 for Ticketmaster
ResaleMarch, 2014 �Upgrade to MongoDB 2.4 (currently using 2.4.9)
April, 2014 �Reviewing MongoDB 2.6
Concurrency improvements in MongoDB 2.8, including document-level locking.
MongoDB at Ticketmaster
6"Humongo us“ – huge , monstrous
Ticketmaster MongoDB use-cases
MongoDB is used as a compliment to relational databases (MySQL and Oracle)
Not used for highly transactional applications and/or sensitive customer/financial data
Currently for TM+ (resale)�Order details/history (secondary market)�Order details/history (secondary market)�Shopping cart and messages�Configuration management�Release tools
Concerts-related services in the future�Venue view (View from section)�Event Content Service�B2B (session recovery)�Client Reports
Operational challenge 1 – MongoDB SecurityNoSQL security concernsEarlier versions of MongoDB (prior to 2.0),
auth mode was not available (run in trusted environment).
Big advancements in security with later versions of MongoDBversions of MongoDB
Ticketmaster has enabled authentication� By default authentication is disabled� Edit your MongoDB configuration file to enable auth (auth = true) or
specify a shared key file for replica sets� MongoDB supports simple role-based authentication at the database
level � Only two permissions are available: read-only and read/write.
Read/write is the default.� Regular user setup is local to each database
Requirements: Implement MongoDB replica set that is highly redundant, fault tolerant, and highly available in 2 datacenters. Need to provide a solution that allows datacenter #1 to be brought off-line for routine maintenance.
Operational challenge 2 – Geographically Distributed Replica Sets
MongoDB Nodes: Red Hat Enterprise Linux Server release 5.8 (64-bit)XenSever VMIntel Xeon dual quad-core processors w/32 GB RAM
Operational challenge 2 – Geographically Distributed Replica Sets
Step #1:Add Arbiter
Step #2:Remove Arbiter
Step #4: ShutdownShift in primary
New Primary
Step #3:Shutdown
Moved Arbiter to third datacenter
Operational challenge 2 – Geographically Distributed Replica Sets
Operational challenge 3 – Performance on NFS
Performance on NFS � 2 of 3 MongoDB outages with data rollback in
production were cased by NFS mount issues� Journal write performance is common bottleneck on
NFS� Journal file has direct disk mapping every 100ms
(default) or more frequently � MongoDB is sensitive to IO contention � MongoDB is sensitive to IO contention � Compete for resource on back-end filer with other
services on Live Nation system� Use NFS to take snapshot/backups. Application is
running 24*7, we can’t quiet writes to take consistent backup
Operational challenge 3 - Performance on NFS MongoDB writing to NETAPP filer.
Operational challenge 3 - Performance on NFS
Solution�Moved journal and data files to block device/local Storage�Build extra secondary replica on each data center and run
data and journal files on filer for (for snapshot/backup purposes)
Notes�We tried moving journal only to block devices with limited �We tried moving journal only to block devices with limited
performance gains.�Block device has limited quota size. Extending storage is not
easy. We added monitoring script to check space usage < 250GB
�Non-standard setup using symlinks to point to data files and journal files
�Extra nodes have priority set to 1 to have less chance of being promoted to primary
Operational challenge 3 - Performance on NFS MongoDB writing to block device/local storage
Operational challenge 3 - Performance on NFS Benefits
• Internal testing showed 20x speedup over NFS v3 when we moved to local drive/block devices
• Besides performance gains, also less page faults, less queuing and % locks
• Development also confirmed improved performance
Other Operational challenges
Understanding the write behavior ( WriteConcern vs performance)
� Write concern describes the guarantee on a write operation (i.e. the strength of the write)
� The weaker the WriteConcern, the faster the write operationsBackups (mongodump vs NETAPP filer snapshots vs
MMS)� Delay replicate slaves� Delay replicate slaves
Adding indexes in MongoDB� Create index for replica set on one member at a time� Take nodes out of replica set to add indexes (bring up Mongo
instance with different port#)Resizing the oplog
� Shrinking the oplog was tricky (documentation geared towards making oplog bigger)
� We built database with default (5% of available space) with default size for oplog. 130GB oplog at go-live.
MongoDB Support
Enterprise Support42 JIRA tickets open for MongoDB support
with company = “Ticketmaster”Overall, good turnaroundSample JIRA tickets….
� Setting up Arbiter documentation - still initializing� Setting up Arbiter documentation - still initializing� Resizing oplog does not seem to shrink the size of the local data files� Very high CPU/iowait and long read/write locks on primary Master yesterday
evening� MongoDB query review
MongoDB Monitoring (MMS 1.4)
Let’s have some fun!
• O2 Arena in London, U.K. • Phones 4u Arena in Manchester, U.K. • Barclays Center in Brooklyn, NY
• Bon Jovi• Michael Jackson The Immortal World Tour• P!nk
Top Tours of 2013
A Front-Row Seat to Ticketmaster’s use of MongoDB!
Ed Presz – VP, Database Services, [email protected] Xu – Director Database Development and Archite cture,
Question and Answer!
Related Documents
