1401 Avenue du Mondial 98 Immeuble Oxygène 34000 Montpellier
04 99 13 68 60
1 KALIOP.COM
How to deploy & optimize eZ Publish high-traffic / bigdata context
Le 09/10/2014
High traffic ? Huge volume ?
2
High-traffic ?
Exemples > 5 M pages / vues / mois
Peak loads ?
Big Data ?
Transactions ?(import – export - REST)
User profiling ?(auth – settings – peruser
UI)
500K + user/day500K + user/day
+200%
1M+ datas
100k transact./day
50%+ authenticated(portal features)
+
++
+
++
+++
Critical issues : Critical mass : Complexity to solve :
Exemples > 5 M pages / vues / mois
Replicated ?(PRA – Worldwide cloud)
2+ ++++
Le 09/10/2014
Keys of success
3
Exemples > 5 M pages / vues / mois
Project management approach
Technical expertise Tools
Do not focus solely on technical expertise !
Web Architect / SCRUM Master
Web Architect / SCRUM Master
Le 09/10/2014
Keys of success / PM approach
4
Exemples > 5 M pages / vues / mois
Project management approach
Role of the Web Architect : Never stop to (re)design. Arbitrate & find workarounds for all the contradictory issues : UX, front-end, performances, dev, hosting, delay, cost, stability, SEO, marketing operations...
Role of the project manager (or SCRUM Master) : Protect the team. Understand, explain & make the POC possible , code refactoring, continuous integration & devOps
Role of the project owner : Build the team & be clear on the KPIs
Exemple : possible to challenge the UX ? (VS performances)Exemple : ready to work without a Gantt ? (VS kanban)
Le 09/10/2014
Keys of success / PM approach
5
Tools
Monitoring : complexity to monitor all the stack(in most cases, only on low level services, not on PHP / slow queries)
Metrics : complexity to have metrics on all the the stack(in most cases, only on low level services, or end-users trafic)
Deployment tools : the easier it is, the more consuming it is(in most cases, copy the files & purge the cache...)
Testing : complexity to automate unit testing / user testing(in most cases... nothing)
Le 09/10/2014
Keys of success / Technical expertise
6
Technical expertise(not only eZ)
FPM
FastCGI
APC
And more...
Le 09/10/2014
Keys of success / Technical expertise
7
eZ FS2 StaleCache
INITemplatesMédiasViewcacheTemplate-block
Médias (copy)
HTTP CachesESI
++++
Easy to hostSave ressourcesEasy to setup/switchStaleCacheSimplify the deployment process
----
High NFS activityRequire top-level ($$$) NFSRequire an high eZ quality devRequire frequent purges
APC APC APC
Stale Cache( no flock )
Le 09/10/2014
Keys of success / Technical expertise
8
eZ DFS Cluster
INITemplatesCache (copy)
INITemplatesCache (copy)
INITemplatesCache (copy)
Médias (copy)
HTTP CachesESI
APC APC APC
EZ DFS
MédiasViewcacheTemplate-block...
META DATA
++++
Compatible with most of platformsLow NFS activityEasy to setup/switchStaleCache
----
Waste of ressourcesClusterDB bottleneckComplex deployment processRequire an high eZ quality devRequire frequent purges
Le 09/10/2014
Keys of success / Technical expertise
9
DevOps mode
Do not expect the hosting architecture (sizing) to make up for the application issues...
Do not expect the development (hack, trick) to make up for the hosting architecture issues...
Le 09/10/2014
Keys of success / Technical expertise
10
DevOps mode(expertise in the middle)
APCapc.filters = "-cache.*\.php"(exlude compiled templates)
MySqltmpdir → tmpfs (avoid « on disk » storage)Add missing indexes (remote_id)
SolrSplit read/write operations
VarnishSynchronize with eZ (x-node pruge)
APC
Le 09/10/2014
Keys of success / Technical expertise
11
eZ PublishOptimize your code(what you can deduce)
Reduce theCache-block activity
& volume
Optimize the viewcache
activity
RemoveUseless SQL
Use small range keysSet an expiry valueUse ezpagedata() insteadReplace with ESINo more than 1K files !
Avoid too many view parametersOptimize smartviewcacheSynchronize with Varnish
Replace with SolrUse PHP cacheAvoid datamap loadOptimize
Solrsplit read / write activities
Delay « commit » (not only index)
Schedule your background tasks
Le 09/10/2014
Keys of success / Technical expertise
12
eZ PublishOptimize your code(read tutos)
Le 09/10/2014
Keys of success / Technical expertise
13
eZ PublishOptimize your code(Learn how it works)
Basic optimization :DelayedIndexing = true (→ will not delay hard commits for all actions ! )OptimizeOnCommit = disabledCRON // ezfoptimizeindex (3 minutes)CRON // ezfindexcontent (1 day)
Medium optimization :OptimizeOnCommit = disabledDisableDeleteCommits = true (+ CRON to commit)DisableDirectCommits=true (+ CRON to commit)CRON // bin/php/ezsolrcommit.php (3 minutes)CRON // ezfoptimizeindex (1 day)
Other great optimizations : - Replicate Solr, and split the Read (search..) / Write (commit, optimize), to avoid concurrencies - Avoid hard commits (soft commits to avoid fsync ops, commitWithin to delay pending docs within Solr)
addDoc Hard Commit (rebuild indexes + fsync)
Soft Commit (rebuild indexes)
CommitWithin (commit all pending docs within X secs)
Le 09/10/2014
Keys of success / Technical expertise
14
Optimize your code(Get metrics)
(what you can not guess)
APC
EZ DFS
META DATA
GET METRICS
Le 09/10/2014
Keys of success / Technical expertise
15
Optimize your code(Get metrics)
(what you can not guess)
Why session_start lock ???
eZ Bug on the checkValidity=false setting...
will always create a session_start on the index.php
Le 09/10/2014
Keys of success / Technical expertise
16
Optimize your code(Get metrics)
(what you can not guess)
Why a unique table insert needs 9,6 sec ???
Server side tuning requested (memory, table optimize, host the clusterDB on another server...)
Le 09/10/2014
Keys of success / Technical expertise
17
Optimize your code(Get metrics)
(what you can not guess)
Why my code provides 1988 cURL calls ???
Mmhh...How to say...
NEXT
Le 09/10/2014
Enough to resolve the following context ?
18
High-traffic ?
Exemples > 5 M pages / vues / mois
Peak loads ?
Big Data ?
Transactions ?(import – export - REST)
User profiling ?(auth – settings – peruser
UI)
500K + user/day500K + user/day
+200%
1M+ datas
100k transact./day
50%+ authentifié(portal features)
+
++
+
++
+++
Critical issues : Critical mass : Complexity to solve :
Exemples > 5 M pages / vues / mois
Replicated ?(PRA – Worldwide cloud)
2+ ++++
Le 09/10/2014
Enough to resolve the following context ?
19
Externalize the user database (if too big)Build a full ESI front-end (pure HTTP cache logical)Replace the DB cluster with Redis or memcachedExternalize medias on CDNUse a custom PHP Cache (exemple)Use messaging toolsBuild POC with dockerBuild custom metrics tools
...
NO ! Be creative !
Le 09/10/2014
Enough to resolve the following context ?
20
Be creative !
Object.datamap → PHP arrayPHP cache file (eZPHPCreator)
On publish
Store all the attributesPre-generate related values
On read
Read object datamap
Fetch, filter, facet, sort..
HTML
Le 09/10/2014
Enough to resolve the following context ?
21
Be creative !
Custom metrics tools
Le 09/10/2014
Ready for eZ5 ?
22
Ready for eZP5...
Native HTTP CacheNative Storage abstractionNative pure MVCNative ESI support…Full Symfony stack
Le 09/10/2014
Ready for eZ5 ? HTTP Cache
23
Twig compiled templatesRoutingservice containertranslations...
Persistence cache
User
GET /
200 OKAge : 100
Cache-control: Maxage=300
X-Location-Id:2
X-Location-Id:{nodeID}Vary: X-User-Hash
expired
eZ
PROD
DEV
HTTP Cache
Clear onViewcacheevent
Le 09/10/2014
Ready for eZ5 ? ESI logical
24
HEADER
FOOTER
SLIDER
LASTNEWS
LASTCOMMENTS
USERSELECTION
FAVORITESTOPICS
ESI key : Shared contentExpiry ? 1 day TTL
ESI key : Per user contentExpiry ? Long TTL + Dynamic, on change
ESI key : Shared contentExpiry ? Long TTL + Dynamic, on change
ESI key : Shared contentExpiry ? 1 day TTL
ESI key : Per topicsExpiry ? Long TTL + Dynamic, on change
ESI key : Shared contentExpiry ? 5 min TTL
LASTTweets
AJAX
1401 Avenue du Mondial 98 Immeuble Oxygène 34000 Montpellier
04 99 13 68 60
25
KALIOP.COM
How to deploy & optimize eZ Publish high-traffic / bigdata context
Question ?