20130714 php matsuri - highly available php

High availability PHPBuilding scalable PHP systems

Graham Weldon

Platform as a Service Development & Operations GroupArchitecture Committee OfficeRakuten, Inc.

About me

• PHP Developer (15 years)

• Game Developer

• Dev Ops

• CakePHP Contributor

• Public Speaker for PHP

• Public Speaker for Open Source

• Used to live in Australia

• Moved to Tokyo, Japan to work for Rakuten, Inc.

• I love Tokyo (and Sapporo!)

(CakeMatsuri) PHP Matsuri

(CakeMatsuri) PHP Matsuri

2009

(CakeMatsuri) PHP Matsuri

2009 2010

(CakeMatsuri) PHP Matsuri

2009 2010 2011

(CakeMatsuri) PHP Matsuri

2009 20122010 2011

(CakeMatsuri) PHP Matsuri

2009 201320122010 2011

What I do at work

Rakuten’s PaaS

Rakuten’s PaaS

• We build on top of Cloud Foundry

• Build new services

• Build new runtimes

• Support application developers

• Ensure a stable, forward-thinking platform is available

• Internal use only

• Highly available

• Easily scalable

By the way ...

• We’re hiring!

• If you’re interested in:

• large systems

• scalable architecture

• solving challenging problems

• Ask me for information!

• @predominant

• graham.weldon@mail.rakuten.com

We’re Hiring!

• RakutenPlatform as a Service

• http://bit.ly/rakutenpaas

• RakutenPublic Cloud Section

• http://bit.ly/rakutenpcs

What does “Highly available” mean?

• No “single point of failure”

• Redundancy

• Failover systems

What does “Highly available” mean?

• In the event of a system failure there are backup / alternative systems available to take over

• No system failure should result in applications being “down” or offline

• The system is designed to continue operating in the event of failure

What does “scalable” mean?

• Easy to add more servers

• Distribute the load of users amongst many servers

• Transparently add to the compute pool to provide faster response times

+

+

+

System design / architecture

Simple Web Setup

Internet

Simple Web Setup

Internet Limited Limited BandwidthBandwidth

Simple Web Setup

Internet Limited Limited BandwidthBandwidth

Single Single WebserverWebserver

Simple Web Setup

Internet Limited Limited BandwidthBandwidth

Single Single WebserverWebserver

Single Single DatabaseDatabase

Simple Web Setup

Internet Limited Limited BandwidthBandwidth

Single Single WebserverWebserver

Single Single DatabaseDatabase

No Failover No Failover solutionsolution

Simple Web Setup

Internet Limited Limited BandwidthBandwidth

Single Single WebserverWebserver

Single Single DatabaseDatabase

No Failover No Failover solutionsolution

No scalingNo scaling

Simple Setup

• We have all done this before

• It works well for very small sites

• You will encounter problems when the number of users grows

• There is no way to recover from system failure

• Any failure will produce extensive downtime

A better solution

Internet

A better solution

Internet

PrimaryPrimary

Backup / Backup / FailoverFailover

Traffic goes to the primary server

A better solution

Internet

PrimaryPrimary

Backup / Backup / FailoverFailover

When a failure occurs on theprimary server...

A better solution

Internet

PrimaryPrimary

Backup / Backup / FailoverFailover

Traffic is redirected to thebackup server

A better solution

Internet

PrimaryPrimary

Backup / Backup / FailoverFailover

FailoverFailoveravailableavailable

A better solution

Internet

PrimaryPrimary

Backup / Backup / FailoverFailover

FailoverFailoveravailableavailableFailover Failover

relies on relies on DNSDNS

A better solution

Internet

PrimaryPrimary

Backup / Backup / FailoverFailover

FailoverFailoveravailableavailableFailover Failover

relies on relies on DNSDNS

NotNotscalablescalable

A better solution

Internet

PrimaryPrimary

Backup / Backup / FailoverFailover

FailoverFailoveravailableavailableFailover Failover

relies on relies on DNSDNS

NotNotscalablescalable

Database Database Sync Sync

problems?problems?

A better solution

• A backup is now available

• We can switch traffic to the backup server in case of failure

• Highly available?

• A little bit

• Scalable?

• No

• Its a better solution than the previous example

• But its not ideal

An even better solution

Internet

Load Load BalancersBalancers

Load balancers distribute the load

An even better solution

Internet

Load Load BalancersBalancers

WebWebServersServers

A pool of webservers are availableto handle the incoming traffic

An even better solution

Internet

Load Load BalancersBalancers

WebWebServersServers

DBDBServersServers

Webservers connect to a database clusterto distribute load

An even better solution

Internet

Load Load BalancersBalancers

WebWebServersServers

DBDBServersServers

LoadLoadbalancedbalanced

An even better solution

Internet

Load Load BalancersBalancers

WebWebServersServers

DBDBServersServers

ScalableScalableLoadLoadbalancedbalanced

An even better solution

Internet

Load Load BalancersBalancers

WebWebServersServers

DBDBServersServers

FailoverFailoveravailableavailable

ScalableScalableLoadLoadbalancedbalanced

A perfect solution?

• There are good solutions available

• No solution is perfect

• Consider the parts of the system that can fail

• Build it in a highly available way

• Allow additional servers / services to be added through configuration

• Or.. through auto detection

Scaling PHP

A Case Study of the Rakuten Platform

Web Server

• Nginx

• Why?

• Fast

• Reliable

• Stable

• Low resource usage

• Designed for high speed

• Easy configuration

Optimization

• PageSpeed

• Optimization tool from Google

• Open Source

• Easy to install

• Lots of options

• Optimizes your pages

• Image processing

• JS/CSS compression

PHP

• PHP-FPM

• “Fast Process Manager”

• Easy to configure

• Fast when configured right

• Process pooling

• Simple to setup

Databases

• A range of options

• Redis

• MongoDB

• Clusterix (MySQL Compatible)

• Each in a clustered setup

• Highly available

• Scalable

Database considerations

• Optimize queries

• Use fast disks!

• Large memory

• Lots of CPU

• Database is often a bottleneck

• Make sure you have enough resources

Content Delivery

• Use a Content Delivery Network for static content

• Or.. use a separate server for delivering static content to ensure PHP has the processing power it needs

• Speeds up browser requests for static files

• Allows concurrent requests with less work / effort

Varnish

• HTTP accelerator

• In-memory caching

• Huge performance increases

• 300-1000x speed increase

Special considerations

Session Storage

• You need a common place for all servers to access for session storage

• We recommend Redis

• Common storage ensures that sessions will be maintained

• Without common storage, users will lose their session data between page requests.

Session Storage

Web Servers Session Store

Session Storage

• php.ini

• session.save_handler = ‘redis’

• session.save_path = ‘tcp://host:port, ...’

Opcode Caching

• OpCache (Extension)

• Bundled with PHP 5.5+

• Available as an extension for earlier versions

• Faster than XCache

• Simple to setup

• OpCache will cache per-server

• You might want to pre-warm the cache for all servers

• I don’t have a better solution for this yet

Thankyou!

Any questions?

• We’re looking for talented engineers!

• Interested in DevOps?

• Interested in highly available systems?

• Contact me!

• @predominant

• RakutenPlatform as a Servicehttp://bit.ly/rakutenpaas

• RakutenPublic Cloud Sectionhttp://bit.ly/rakutenpcs

• graham.weldon@mail.rakuten.com