1 Plone TuneUp challenges in search of enterprise intranet solution Andrew Mleczko (redturtle.net) European Plone Symposium 2009 Sorrento, Italy
1
Plone TuneUp challenges
in search of enterprise intranet solution
Andrew Mleczko (redturtle.net)
European Plone Symposium 2009
Sorrento, Italy
2
PoznanPoznan FerraraFerrara
3
● Co-founder of ACC Poland Ltd., the first Polish
Plone company (past)
● Plone consultant at RedTurtle Technology -
based in Ferrara (present)
● Plone-based and Lotus-based web applications
4
RedTurtle Technology
“Sharing knowledge
about knowledge sharing”
5
So what can you expect...
● Plone is just a CMS?
● Case study – Intranet Provincia di Ferrara
● Loadtests environment
● Plone TuneUp challenges
6
How we can use
Plone?
7
Plone variety
● Community portals
● Document management system (DMS)
● Groupware - Collaboration
● Subsite management system
● Enterprise intranets
● ...
● Combination of all mantioned above
8
Plone as CMS
vs.
Plone as framework
9
Why?
“Plone is NOT a general web development framework.“
/Carlos de la Guardia/
10
Plone can be SLOW*
*when using the wrong way
11
Framework comparision
● Used hardware:
– CPU: AMD OpteronT Processor 146 (2 GHz)
– Memory: 2 GB
● Testing with Siege 2.68
● Read scenario
● Results for 50, 200, 300 Concurent Users (CU)
● More details: www.alrond.com/en/2007/jan/25/performance-test-of-6-leading-frameworks
12
Comparision
Django RubyOnRails TurboGears0
0,5
1
1,5
2
2,5
50 CU200 CU300 CU
Response time, secs.
13
Comparision
Django RubyOnRails TurboGears0
5
10
15
20
25
30
35
50 CU200 CU300 CU
Longest transaction, secs.
14
So where is Plone?
15
Far, far away
Django RubyOnRails TurboGears Plone0
2
4
6
8
10
12
14
16
18
50 CU200 CU300 CU
Response time, secs.
Django RubyOnRails TurboGears Plone0
10
20
30
40
50
60
50 CU200 CU300 CU
Longest transaction, secs.
16
But comparing with Zope2
Django + Psyco RubyOnRails 1.2.1 TurboGears Plone Zope20
2
4
6
8
10
12
14
16
18
50 CU200 CU300 CU
Response time, secs.
17
What can we do...
● Do not treat Plone as a framework
● For some use cases:
use ContentMirror + fast framework (django,
pylons, repoze)
● But what about enterprise intranet solutions?
18
Plone for enterprise intranet
● Large number of users (more than 500) and
documents (more than 100 000)
● A lot of simultaneous write requests (>20 CU/s)
● Using of caching limited
● Slowest acceptable resp. time <10 seconds
the limit for keeping the user's attention focused [Miller 1968; Card et al. 1991]
19
Intranet Provincia di Ferrara
Case study
20
Prerequisites
● 800 users stored in Domino Server (by PAS)
● 170.000 documents
● Yearly growth: ca 25.000 documents
● Peak traffic: 20 CU/s - write transactions
(based on Domino logs)
21
Objective
● User response time (mixed read/write):
– Slowest (5-6 seconds)
– Recommended (2-4 seconds) 90% of all request
– Fast (<2 seconds)
22
Test environment
● Hardware (current):
– 4 dual-core Intel Xeon 2.8 Ghz
– 32 GB RAM
● Amazon EC2 (future)
23
Test environment
● collective.loadtesting
– funkload
– collective.funkload
● PageTemplate Profiler
● mr.bent
● Munin + redturtle.munin
24
class Readonly(testcase.FLTestCase): def setUp(self): self.server_url = 'http://localhost:8080/plone'
def test_ReadOnly(self): server_url = self.server_url self.get(server_url + "/plone/login_form", description="Get /plone/login_form")
self.post(server_url + "/plone/login_form", params=[ ['came_from', server_url + '/plone'], ['form.submitted', '1'], ['cookies_enabled', ''], ['login_name', ''], ['pwd_empty', '0'], ['__ac_name', 'admin'], ['__ac_password', 'admin'], ['submit', 'Accedi']], description="Post /plone/login_form")
collective.loadtesting
25
mr.bent
>>> from mr.bent.wrapper import mkwrapper>>> from mr.bent.tests.test_bent import callcounter>>> from Products.CMFPlone.CatalogTool import \ CatalogTool
>>> mkwrapper(CatalogTool.searchResults, callcounter, "catalogsearches")
26
Test scenario
● Log in
● View random folder
● Create document (2 requests)
● View newly created document
● Publish it
● View home page
● Logout
27
Log in Folder view Create document Save document View Publish Home page Logout
0
10
20
30
40
50
60
5CU20CU60CU
Default clean Plone
Response time, secs.
28
Default clean Plone
Log in Folder view Create document Save document View Publish Home page Logout
0
10
20
30
40
50
60
70
80
90
100
5CU20CU60CU
% Errors (ConflictErrors)
29
Let's start TuneUp and install
the basics:
Varnish + Pound (with sticky session) +
+ 6 ZEO Clients (18 threads) + ZEO Server
30
● Apache – proxy/rewriting urls
● Varnish – cache resources, add http headers
● Pound – balancing with sticky sessison
● 6 ZEO clients
● ZEO Server
31
+ Varnish + Pound + ZEO
Log in Folder view Create document Save document View Publish Home page Logout
0
5
10
15
20
25
30
35
40
5CU20CU60CU
Response time, secs.
32
+ Varnish + Pound + ZEO
Log in Folder view Create document Save document View Publish Home page Logout
0
10
20
30
40
50
60
70
80
90
100
5CU20CU60CU
% Errors (ConflictErrors)
33
Comparing to default Plone
34
Log in Folder view Create document Save document View Publish Home page Logout
0
5
10
15
20
25
30
35
40
Default PloneImprovment
Comparision
Response time, secs., 20 CU
35
Where are the conflict errors
comming from?mainly portal_catalog
36
Let's try:mount portal_catalog to different ZODB
and increase zodb cache on it
37
● Install redturtle.catalogmount● Add zope-conf-additional:
<zodb_db catalog> mount-point /plone/portal_catalog container-class Products.CMFPlone.CatalogTool.CatalogTool cache-size 300000 <zeoclient> server ${zeoserver:zeo-address} storage 2 name catalogstorage var ${buildout:parts-directory}/instance1/var cache-size 400MB </zeoclient> </zodb_db>
38
● And add zeo-conf-additional:
<filestorage 2> path ${buildout:directory}/var/filestorage/CatalogData.fs</filestorage>
39
+ redturtle.catalogmount
Log in Folder view Create document Save document View Publish Home page Logout
0
5
10
15
20
25
5CU20CU60CU
Response time, secs.
40
+ redturtle.catalogmount
Log in Folder view Create document Save document View Publish Home page Logout
0
10
20
30
40
50
60
70
80
90
100
5CU20CU60CU
% Errors (ConflictErrors)
41
Maybe there is a more
stable solution:collective.indexing
experimental.catalogqueryplan
archetypes.schematuning
42
Log in Folder view Create document Save document View Publish Home page Logout
0
2
4
6
8
10
12
14
16
5CU20CU60CU
+ collective.indexing + archetypes.schematuning
Response time, secs.
43
+ collective.indexing + archetypes.schematuning
Log in Folder view Create document Save document View Publish Home page Logout
0
10
20
30
40
50
60
70
80
90
100
5CU20CU60CU
% Errors (ConflictErrors)
44
So what is the improvment?
45
Improvment
Log in Folder view Create document Save document View Publish Home page Logout
0
5
10
15
20
25
30
35
40
fresh Ploneafter TuneUp
Response time, secs., 20 CU
46
Let's try to scale it...
47
ZEO scalability problem
Log in Folder view Create document Save document View Publish Home page Logout
0
5
10
15
20
25
30
35
40
45
1:104:206:1810:30
Response time, secs. 20 CU
48
ZEO scalability problem
Log in Folder view Create document Save document View Publish Home page Logout
0
10
20
30
40
50
60
1:104:206:1810:30
% Errors (ConflictErrors), 20 CU
49
Plone intranet limits
● ZODB is slow when handling concurrent writes
transactions
● Scalability problem using flat ZEO
● Missing 'out of the box' performance
(comparing to other framworks, CMS)
50
How we can solve the problem?
51
Ideas
● RelStorage for scalability problem
● collective.solr
● others ?
● share knowledge at plone-enterprise mailing list