Copyright © 2014 Criteo Real time advertising at NoSQL speed Nicolas Helleringer Site Reliability Engineering Manager March 23 rd 2015
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Copyright © 2014 Criteo
Real time advertising at NoSQL speed
Nicolas Helleringer
Site Reliability Engineering Manager
March 23rd 2015
Copyright © 2014 Criteo
Criteo
• French start up since 2005• Keeping the spirit
• Machine Learning and Big Data oriented
• Algorithms as root
• Ad Tech business
• 210 in R&D 1400 worldwide• R&D teams in Paris
Copyright © 2014 Criteo
Criteo – Personalized performance advertising at scale
• Personalized• Per user per view
• Real time product recommendation
• Performance• Cost of sales model
• Cross device
• Scale• 994 millions unique users reach globally
• 741 billions ads in 2014
Copyright © 2014 Criteo
Site Reliability Engineering at Criteo
50 DevOps engineers and growing with the company
Software infrastructure
Systems (CentOS, Windows, monitoring)
Automation (Chef)
Big Data infrastructure (Hadoop, Storm, Kafka, HBase)
NoSQL infrastructure (MemCache, Couchbase, Graphite,ElasticSearch, MongoDB)
Escalation
Predictive monitoring
Build & continuous integration
Copyright © 2014 Criteo
Hardware Infrastructure
• 6 Datacenters
• 10k+ server (50/50 windows/linux)
• Private worldwide dedicated network up to 10Gb/s
• One of the biggest Hadoop clusters in Europe(1k+ servers, 37 PB)
Copyright © 2014 Criteo
From SQL to some NoSQL
• Criteo started with• User data in client side cookies
• Client and campaign configurations in MS SQL server
• First issues• RAM cache in IIS servers
• Size of the cookies
• Added some MemCache …. big instances
Copyright © 2014 Criteo
Next steps
• Criteo connected to RTBs
• Real Time Bidding networks
• Server to servers calls => bye bye client side cookies
• Additional challenge : multi datacenter => sync !
• Moved to automated by Chef scale out MemCache/Couchbase clusters
• Persistency !
• C# driver with double write
Copyright © 2014 Criteo
Grown big !
• 550 Couchbase servers (from1.8 to 3)
• 24 clusters
• 107 TB Total RAM and SSD space
• 10 millions hits / sec on average
Copyright © 2014 Criteo
Lessons learned / quick wins
• Do not mix RAM and persisted usages• This voids the flexibility of persisted only clusters
• Before 3.0 metadata memory usage is the limiting factor on large clusters
• Couchbase admin console is fine to begin with. Extracted raw stats and put them in Graphite for more flexibility
• Invest the same in Dev than in Ops
Copyright © 2014 Criteo
Copyright © 2014 Criteo
Copyright © 2014 Criteo
Next
• Replication and Metadata dynamic memory in 3.0
• Cross datacenter replication via Couchbase
• C# driver with failover on replicasmove from CP to AP
• Set/Get with metadata
Copyright © 2014 Criteo
Thanks !
Questions ?
Related Documents
