Zimbra Service Provider Benchmark

8/2/2019 Zimbra Service Provider Benchmark

1/23

Copyright Zimbra, Inc. 2006, www.zimbra.com. All Rights Reserved Page 1

Zimbra Collaboration Suite

Scalability & Performance Benchmarks for Service Providers

Zimbra Collaborat ion Suit e (ZCS) is open source server and clientsoftware for messaging and collaboration (email, calendaring, contacts,web document aut horing, etc.). Zimbra is being used by serviceproviders, businesses, educational inst it ut ions, governments, and non-profits.

This document describes performance and load t est ing of the ZimbraCollaborat ion Suit e for consumer-oriented service provider deployments,as performed by Zimbra and HP working together at the HP SolutionsCenter lab in Houston, Texas, in mid-2006.

The net result of t his work was t hat 50,000 simult aneously act iveconsumer users were successfull y simulat ed wit h no degradat ion inperf ormance on a single dual-core, dual-CPU HP server backed by HPEVA SAN st orage. Since for t ypi cal service provider deployment s(ISPs, Telco s) perhaps 10-15% of t he overal l users are act ive at anygiven t ime, t hese result s demonst rate t hat for t his part icular profi le(8 non-spam receives, 8 reads, 8 sends al l of 32K-sized messages peruser session), ZCS can manage mult iple 100,000s of consumermailboxes per PC-class server.

These results indicate that overall Zimbra scalability and performance isat l east on par wit h the capabil it ies of other leading providers ofmessaging systems for the service provider market. Given Zimbra is arelatively young technology, we believe these results (1) prove out thearchit ectural approach that the Zimbra team has taken, (2) validat e thematurit y of t he open source infrast ructure upon which Zimbra is built ,and (3) bode well for t he future as Zimbra is tuned and enhanced overt ime (syst ems soft ware tends to take years to reach it s ult imate peak in

performance and scalability).

Zimbra is a trademark of Zimbra, Inc. All other trademarks used herein belong to their respective owners.


2/23


CONTENTS

Sect ion 1: Int roduct ionSect ion 2: CaveatsSection 3: Objectives of Test ProgramSect ion 4: Test Methodology

Section 5: Perf ormance Profi leSection 6: Performance Graphs

Sect ion 1: Int roduct ion

Using an assumed user-profi le and load-prof i le (specif ied later in thisdocument), the following load levels were achieved on a single HPProliant dual-core, dual-CPU Opteron server with 8Gb of RAM runningRed Hat Enterpr ise Linux (32-bit ), backed wit h an HP EVA SAN

(specifications below): 50,000 concurrent consumer-profile users atpeak hours were successfully simulated wi th no degradation inperformance/ reponse t imes. The server was provisioned wi th 200,000total users, while 50,000 of those users were simult aneously activesending and receiving email via the Zimbra web interface.

In general, benchmark results are intended for proving out scalability andperformance, and for comparing alt ernat ive t echnologies. Benchmarksrepresent high water marks for soft ware and/ or hardwarecapturingwhat can be achieved when you pull out the stops by st ressing thetechnologies closer to their limits. These results are generally not, then,int ended to be used direct ly for capacity planningfor example, thesizing of a part icular ZCS deployment.

We are actually unsure whether we pushed this configuration to thelimit. As can be seen in the summary graphs below, we averaged about50%idle t ime on the Proliant CPUs during the 50,000 user load test . Ofcourse, messaging benchmarks are typically dominated by disk

input / output (I/ O) costs. While iostat was at least indicati ve of t herebeing addit ional I/ O headroom as well , i t is bett er to judge I/ Out ilization based on the capacit y of t he backing SAN (also see the graphsat t he end). Wit h further analysis and tuning, we are confident ZCS couldaccommodate more t han 50,000 act ive consumer users for t he purposesof this benchmark, but at the same time, we would not recommend suchloading for product ion deployment.


3/23


Although ZCS is designed for multi-server deployment, the benchmarkresult s herein were obtained on a single server. The only int eractionsindividual ZCS server inst ances have with external software components

are (1) responding to the load from the Soapgen client test fixture, and(2) access to the shared LDAP repository (user provisioning and log-in).Since LDAP itself support s replicat ion and part it ioning (via federation),there are no points of contention that would prevent the results of this

benchmark from being replicated across a large number of servers solong as the network and network-attached st orage did not becomebottlenecks.

The balance of t his document describes methodology, t ools and results of

the testing. The Soapgen test ing tool described herein i s not current lypart of t he Zimbra Collaborat ion Suit e release, but can be madeavailable to Zimbra partners and service provider customers interested inreproducing these results or creat ing their own benchmarks.

Our sincere thanks to the HP team, whose support and equipment werecrucial to achieving these results.

Sect ion 2: Caveats

This benchmark exercised the Zimbra mailbox server processestheserver t hat hosts the end-user s mailbox (and responds to web request sas wel l as IMAP, POP, etc.), the message store (layered on top of t heLinux Ext3 file system), and the meta-data store (layered on top of the

MySQL relational database).

This benchmark was not intended to measure the Mail Transfer Agent(MTA) t ier of Zimbra that includes mail rout ing to and from t he Internetand security processinganti-spam (AS) and anti-vi rus (AV). The ZimbraMTA was run on a separat e set of servers that provided adequate CPUand I/ O that MTA processing would not be a limit ing resource tobenchmarking the mailbox server ( in addit ion, AS/ AV processing wasturned off for these tests). Our rationale was that some service providershave existing t echnology they want t o leverage for t he MTA and AS/ AVt ier and use that in place of t he technology bundled with Zimbra.However, for service providers that seek to use Zimbras native MTA(based on Post f ix) and AS/ AV (based on SpamAssassin and Clam/ AV), MTAsecurity processing would almost be certainly be done in a separateserver f arm t hat was independent ly conf igured and sized, and hencewould not impact the mailbox server benchmarks and capacit y planning.Zimbras MTA ult imately needs to be the subject of independent


4/23


benchmarking analogous to that herein, but those result s would notimpact the opt imal configurat ion and scalabil it y of the Zimbra mailboxserver t ier.

Zimbra advanced features. This benchmark also did not exercise someof the more advanced features of Zimbra, such as calendaring, contact

management, web document aut horing and management, and so on. Ourrat ionale was two-fold: (1) we wanted to achieve an apples to applescomparison with exist ing alternat ives to Zimbra that do not provide suchfeatures for service providers; and (2) under typical use, t he largemaj ority of t he ZCS server workload is due to email processing (ratherthan calendaring, contacts, and documents). Since all of t hese featurescan be easil y turned off by the service provider, the decision t o providesuch addit ional value-add to the customer can be made independent ly ofchoosing ZCS (and indeed is considered an upsell by some of Zimbrasservice provider customers).

Most importantly, however, the most advanced features of Zimbragenerally (under t ypical usage prof il es) have negligible aggregate impacton mailbox server scalability and performance:

Aj ax mash-ups/ Zimlets,

Advertising via banners, portlets, and automated content linking,

HTML document and spreadsheet authoring, including Aj ax Linking &Embedding (ALE),

Calendaring (including Internet and friends & family sharing),

Contact management,

Tagging, Sharing and delegati on (of calendars, contacts, documents, etc.),

RSS int egrat ion (the load can be modeled with email ), and

VoIP/ SIPsimple integrat ion.

Indexing & search. Indexing (and search) is an exception to this rule inthat it would clearly impact benchmarks at t his level. Zimbra includesthe abil it y t o pre-index all message bodies and most every att achmenttype in order to provide nearly instantaneous search within (and, for

administrators, across) mailboxes. Zimbra, like web search engines, doesall of the necessary indexing a priorion message arr ival so that evencomplex ad hocsearch results can be returned in milliseconds.

Indexing and search was also explicitly excluded from these benchmarkresult s (as well as Zimbras nat ive abil it y to render complex at tachmentsin HTML). In this case, the cost of indexing all messages and attachments


5/23


is not insignificant (it would certainly have helped us use up some of thatidle CPU), but again we felt it was import ant to fi rst have benchmarksshowing how ZCS compares to existing technologies that do not providenat ive search. So again, the decision whether to provide very fast searchacross an end user s ent ire mailbox is an independent business decision.(In the future, we expect to provide capacit y planning documentationthat covers the overhead of indexing and search, but presumably it will

not make sense to benchmark these features until there are competingalternatives.)

IMAP, POP, and other persistent clients. Similarly, t his benchmark didnot measure the impact of non-web clients that manage their ownpersistent message stores, including

Exist ing IMAP/ POP email clients (Apple Mail , Out look, Evolut ion,Thunderbird, Eudora, etc.);

Microsoft Out look via the ZCS Connector for Outlook (using MAPI);

Apple desktop via the ZCS Connector for Apple (using iSync);

Novell Evolution via the ZCS Connector for Evolution; and

Mobile devices via ZCS Mobile, including Nokia, Motorola, Samsung,Palm/ Treo, and other ZCS compat ible smart phones/ PDAs.

For consumer-oriented benchmarks such as this, mixing in some fractionof IMAP/ POP and mobile device l oad seems the most appropriate, butsuch a mix will inherently differ between service providers and overtime.

Zimbra is nevert heless l ikely to fair well for such benchmarks because

POP, IMAP, and the above synchronizat ion prot ocols are all handledwithin the same mailbox server process: Other solutions whose protocolhandlers are packaged as separate processes are unable t o delivercaching at t he edge (i. e., wit hin the protocol handling process), whichresult s in addit ional computation and latency.

Data dependent rout ing. Finally, this benchmark presumes the exist enceof smart request rout ing to the appropriate ZCS mailbox server. EachZCS server will automat ically forward a network request dest ined for a

mailbox that is actuall y located on another server, but the addit ionaloverhead for such proxy forwarding was not f actored int o our result s.The rationale was that we expect most scaled deployments of ZCS willf ind it more cost -eff ect ive to use (1) browser redirects on log-in, or (2) asmart rout ing t ier such as via web server proxy plug-ins or smart webload-balancing switches, and smart IMAP/ POP routing (support for whichis packaged with ZCS). Such approaches have already been proven out for


6/23


large-scale messaging systems as well as for web infrastructure ingeneral.

Section 3: Objectives of Test Program

The goal of performance and load testing is to simulate the software as it

wil l run in a service provider environment so it s capabil it ies in thatenvironment can be understood. Therefore, t he test program wasdesigned to load standard HP Proliant servers with a large number oftotal users, and a relat ively high percentage of active users(approximately 25%), and t hen test a common usage prof ile against thatuser base. (The test ing was also intended to prove that non-active usersdo not degrade performance on the production systems, so servers canbe provisioned for maximum active rather than total users.) Theexpected outcome is a measurement of the load, performance, andoverall behavior of the servers throughout the duration of the testing,

and at peak periods. (Note: In this test , other t han init ial cache warmup,the ent ire test duration simulated peak hours of processing.)

Sect ion 4: Test Methodology

Envir onment and Variables

The most accurate testing occurs when all key variables in the customerenvironment are accurately modeled, including user profile, userbehavior and deployment hardware.

The following variables are most import ant t o assess the ant icipatedperformance of Zimbra Mail:

Average message size

Percent of messages with attachments, and at tachment t ypes

Average number of recipients per message

Average number of messages received per user per day

Average number of messages sent per user per day

Quota and retent ion policies

Length of a user email session (all day, few minutes twice daily,etc.)

Number of user logins per day

Mail deletion rates

Percent of user messages that are read per day

Number of searches executed per day

Percent of activity that occurs in the busiest hour


7/23


Anti-spam and anti-virus processing

Storage (SAN, internal RAID, etc.)

Pref erred CPU architecture (1 CPU, 2 CPU, etc.)

Failover and reliabil it y plans

The more of this informat ion available f or a specif ic customer profi le,the more accurate the load test ing can be. When any of this informati on

is absent, we st rive to use our j udgment and experience to makereasonable assumptions about the environment. Obviously, it is alwayspreferable to have complete information.

Tools

Zimbra has developed a test fixture called Soapgen that simulates fromone user t o tens-of-t housands of concurrent users of the Zimbra Webclient. This Java-based client mimics the AJAX model of web browserclients and uses XML/ SOAP requests to communicate with the server. To

simulat e users, Soapgen relies on the mapping from user actions to theresult ing SOAP calls that are made to the server. From t he server s

perspective, Soapgen s automated requests are the equivalent of having

the simulated number of simult aneously active users. Soapgen takes asinputs

the number of active users,

the server to test against,

the number of t hreads to use, which is independent of (and muchless than) the number of active users, and

a profile, which is an XML file describing the key variables listed

above.Soapgen should be run on a separate machine(s) f rom those that arebeing benchmarked.

At start -up, Soapgen reads the prof il e fi le and determines all the act ionsthat will be executed. These act ions would take the form of, say, user

#555 sending an e-mail to user #777 and user #999. Soapgen spawnsthe number of t hreads prescribed by the command line argument togenerate requests for such act ions on the server in parallel.

Soapgen operates in one of two modes: real-time mode and ASAP mode.In real-t ime mode, Soapgen will spread out the required actions roughlyevenly over the test period specif ied as number of hours. Usually, weare most interested in t he systems behavior in t he busiest (or peak)hour. To implement such behavior, define a profi le fi le t hat describes

the peak hour s load and start Soapgen. Soapgen can be stopped aft er


8/23


an hour if desired, or in order to provide a longer-term stress test, canbe run for more hours.

In ASAP mode, Soapgen wil l at tempt to complete t he requestedworkload as soon as possible. This is the same amount of work t hat

would be done in real-t ime mode. However, Soapgen wil l use allavailable threads all the time to ensure that the server is kept as busy as

Soapgen is able t o keep it unt il the workload is completed. By cont rast ,wit h real-t ime mode, Soapgen uses only the number of threads that arerequired to support the amount of work at any given moment.

Monit oring and Statist ics

Soapgen has built -in monit oring and stat ist ics report ing. Soapgen willraise an alert

if it receives an error f rom the server, or

if it falls behind in i t s scheduled tasks (in real-t ime mode only).

Soapgen s stati st ics include

average and maximum latencies for each operat ion t ype,

an overall operations-per-second measurement,

operation counts, and

average t ime (in mill iseconds) t hat tasks are fall ing behindschedule (when the generated load is sufficiently high)

The Profile File

The profi le fi le is read by Soapgen at start -up. It includes a defini t ionof the message sizes that will be used, the number of recipients for themessages, and the definit ion of a user session.

The message sizes are det ermined in the element. One

directory on a file system accessible to Soapgen is specified as the body

directory. Soapgen wil l select a fi le at random from this directory tocreate the text body of a composed message. The size of t he text partcan then be controlled by setting the size distribution of files in thisdirectory.

Message at tachments are det ermined by a set of elements,

another directory in the file system accessible to Soapgen. In thatdirect ory are f i les that will be added to composed messages. Each


9/23


attach element def ines a directory and a frequency. Once a directory ischosen, the file to be used is chosen from within that directory atrandom. If a directory has no fi les wit hin it , then the message wil l haveno attachments added to it. That directory is chosen based on thefrequency at t ribute of t he att ach element. The frequency specif ies apercentage of t ime that t hat di rectory wi ll be used. For example, adirectory wit h a frequency of 30 wil l have one of it s f il es att ached to a

composed message 30%of the t ime.

The frequency distribution allows one to control attachment size andtype by split t ing the attachments up as desired. For example, all GIFimages in one directory, all Word docs less than 50K in another, and soon.

Today, Soapgen only attaches one file per composed message, butmultiple attachments can be successfully simulated (from theperspective of the ZCS server) with an att achment that is the size of t he

sum of the individual attachments.

The number of recipients is determined by the element. Itmaps a number of recipients (via the at t ribute num ) t o a frequency.The meaning of the frequency attribute is identical to that for themessage dist ribution. For example, wi th the frequency att ribute set t o40 for t he number 2, that means that 40%of the messages that arecomposed will go to 2 people.

The profile file also defines a user session and the number of such

sessions that occur during the specif ied test durat ion. A user session is asequence of user actions.

The profi le fi le allows for a wide range of user actions, including

login,

read a conversation,

read a message in a conversat ion,

compose,

reply,

move a message,

empty trash, sleep, and

search.Moving a message emulates delet ion (move t o Trash).


10/23


Replies are a special case of compose where the number of recipients isexact ly one (the sender of the message being replied t o) and the body ofthe original message excluding attachments is quoted.

The sleep action simulates elapsed t ime between user actions. Forexample, to model a case where a user reads a message for a minute and

then replies, a element is inserted between t he

read message and reply actions.

Several profi le fi les have been defined to simulate dif ferent condit ions.They are st raightforward to create. To creat e a profi le fi le one needs to

keep in mind that Soapgen maintains much of the same state about theuser s session as the client would.


11/23


Limitations

Soapgen does have some limit at ions:

Soapgen doesn t do client-side caching, so repeated message read forinstance can exercise the server more than in real life, since theactual browser client would satisfy the second and subsequent

requests out of it s cache without server t rips. The author of Soapgenprofi le f ile should be mindful of t his and include only the actions thatresult in actual server request s.

Soapgen simulates AJAX clients, whereas a separate test client isavailable for IMAP. (We don t have a client simulat ion for POP yet.)

Soapgen only sends messages via the SOAP/ XML interface. Thissimulates users on the same domain (or virtual domains) sendingmessages to each other. It doesn t simulate receiving messages from

an external domain via SMTP. A separate tool, LmtpInject, iscapable of sending large amounts of email via direct LMTP connectionto the server, bypassing the MTA. (The MTA is bypassed so it cannotbe a limit ing factor.)

Soapgen also runs against only one server at a t ime. To test mult iple

servers concurrently, multiple copies of Soapgen can be run.

Similarly, Soapgen can load only one profile file at a time, and aprof ile can describe only one t ype of a user. To run a test with

several dif ferent types of users, define a profi le f ile for each type ofuser and then launch several Soapgen clients, one for each user type.


12/23


Sample Profile File

Here is a sample profile f ile t hat il lust rates most of the actions that

Soapgen supports.


13/23


Hardware

The test ing discussed in t his document was perf ormed at the HP Solut ionsCenter lab in Houston, Texas, using several dual -core, dual-CPU AMDOpt eron servers wi t h 8 GB RAM and Fiber Host Bus Adapters (HBAs),connected to an HP EVA SAN with 168 available disks. Whil e mult ipleservers were used for t his benchmark configurat ion, the goal was to test

the capacity of a single dual-core, dual-CPU server backed by fast SAN insuch a way that the results could be scaled linearly via additionalmailbox servers.

Addit ionally, for ongoing test ing in t he Zimbra lab, a variet y of hardwarehas been purchased to emulate customer environments. Dual-CPUmachines appear t o be the best value at this t ime, so Zimbra hasinvested in that type of hardware. The hardware includes:

Dual AMD Opt eron servers with 4 GB RAM and Fiber HBAs

HP MSA 1500 with 2 trays fully populated (28 Ult ra-SCSI drivers) Dual AMD Opt eron servers wi th int ernal SCSI RAID arrays

Dual AMD Opteron servers wit h int ernal SATA RAID array

Dual Intel Xeon servers wi th 2 GB RAM

Dual Int el Xeon servers wi th 2 GB RAM and internal SATA RAIDarrays

Wide select ion of single CPU machines

This variety of hardware allows t he running of single or mult i-hostconfigurations t o emulate a wide variety of deployment architectures.

Section 5: Perf ormance Profi le

Summary

As a reminder, with the user profile and load profile that was assumed,the following load levels were simulated: On a dual-Opteron serverrunning Red Hat Enterpr ise Linux, backed with a SAN (per specs below),50,000 concurrent consumer users were simulated at t he peak hour

comfortably with no degradation in performance with the load profile asdiscussed below. The server was provisioned with 200,000 users with50,000 of them active. The profile of the user and the access patternsare described below.

Soft ware Configurati on


14/23


The version tested was Zimbra Coll aboration Suite 3.1.3 wit h Class ofService sett ings t o match an apples to apples feature set and loadcomparison with legacy messaging systems. Conversat ions, t ags andsearch were turned off to compare against vanil la mail productoffering from our competitors.

HW Configuration - LDAP, MTA and Client Loader

LDAP with 1 million accounts provisioned on local disk (t his wasnot a bottleneck)

MTA and load client (Soapgen) - ut il it y servers

Message Store Server

Dual-CPU Dual-core AMD Opt eron 280 2.4GHz (4 logical CPUs)

8GB RAM 32-bi t Red Hat Enterpr ise Linux 4 AS Update 4 beta

HP EVA6000 SAN, 168-disk RAID0

Diagram

Zimbra

Mailbox

Server

HP SAN

Soapgen

Client Test

Fixture Server

XML/

SOAP LDAP

Server

Zimbra

MTA

Server

SMTP LMTP

Benchmark

Focus


15/23


Methodology

At the st art of the test , Zimbra processes are restart ed on the messagestore server and MTA server . The MTA s delivery queue is cleared.Monitoring processes are st arted on each server, including the load cl ientmachine, to monitor CPU, memory, disk and application-specificvariables such as the size of the MTA delivery queue, number of busy

client threads, latency, throughput, and more.

The Soapgen load generator sends login requests for all the activeaccounts to be used in the test . This warms up the cache on the ZCS

message store server. Soapgen then begins the main test phase byscheduling the requests over t he test duration, in this case 2 hours. Thestart of each user session is staggered to spread the load across theent ire test durat ion, avoiding unrealist ic burst y loads. The requestsare then sent on schedule. If requests cannot be sent on schedule due toserver unresponsiveness, this is measured and report ed.

At the end of the test duration all the monitoring output is collected bythe load generator and chart s are generated. Examinat ion of t he chart sindicates if the test succeeded, and if not , why and how it failed.

User and Load Profi le

The user profile used was based on a typical consumer using their emailsystem hosted by their service provider. (The specific profile was derivedvia discussions with a number of service providers that are either using or

considering Zimbra.) A typical session was simulated where it wasassumed that a user receives and reads 8 messages, sends a total of 8messages wi th 5-second pauses bet ween sends and reads.

The precise user behavior simulated was as follows:

Login and view t he Inbox (in message view)

Read 4 messages wi th 5 second pauses bet ween each

Send 4 messages after a 5 second pause

Read 4 more messages wi th 5 second pauses bet ween each

Send 4 more messages after a 5 second pause

The size of each email sent and received is assumed t o be 32KBtext/ plain message wit h no at tachments. This is a generous est imatesince most consumer emails sent/ received are much small er t han 32KB.


16/23


The dual-Opteron Linux server described above was provisioned with200,000 users, wit h 50,000 of t hem active. With 50,000 active userssimulat ed, the above session was executed once during a peak hour andrun for 2 hours for a total of 100k sessions over 2 hours. Based on thisrun and the chart s generated, for t his user profil e, wi th t his hardwarethe archit ecture can support 50,000 concurrent users at peak hour.


17/23


Section 6: Performance Graphs

The following perf ormance graphs were generated by the Zimbra

Soapgen program. The graphs show the results of t est ing during arepresentative peri od of 2+ hours, during which t he standard usageprof ile was run continuously across the 50,000 act ive users. The unevenload at the beginning until around 22:35 is the cache warmup. Then the

main test load is run for 2 hours.

Tomcat represents the Zimbra mailbox server process (which is runs within theApache Tomcat application server). The MySQL process is collocated with theZCS mailbox server, while t he MTA and LDAP servers were located on separate

hardware (and were not the focus of this benchmark study). Convertdrepresents the cost of the separate and potentially distributed process forattachment indexing (text extraction) and HTML conversion process, which isunused in t his benchmark.


18/23


Tomcat represents the Zimbra mailbox server process (which is runs within theApache Tomcat application server):

hda represents the system disk;

hdb represents the swap part it ion;

sdg represents the ZCS MySQL data f il e;

sdh represents the ZCS Apache Lucene-based index (not used for t hisbenchmark study); and

sdi represents the ZCS message (a.k.a. blob ) store.


19/23



20/23



21/23


Sect ion 7: Conclusions

Significance of the Test Results

To date, Zimbra has been most ly known for it s innovat ionsrich Aj axclient, mash-ups (which provide service provider and aff il iate market ingopport unit ies), advanced search, tagging, open XML/ SOAP prot ocols for

all operations, web document authoring, Aj ax Linking and Embedding(ALE), and so on. Zimbra has also received accolades for being an opensource solut ion, and hence providing long-t erm invest ment protection toits customers.

However, the load and perf ormance test ing described in this documentalso prove that the three-year old Zimbra Coll aborat ion Suit e running onHP hardware is at least as per formant and scalableas the solut ionsprovided by the current marketshare leaders of pre-Web service providermessaging solut ions.

How did ZCS manage to scale so quickly?

First, Zimbra makes heavier use of open source software than anymessaging solution to date: Linux file system (message store), ApacheTomcat (server container), MySQL (meta-data), Lucene (search), Postfix(MTA), OpenLDAP (provisioning and conf igurat ion dat a), SpamAssassin(ant i-spam), ClamAV (ant i-virus), and so on. (Click here for acomprehensive l isting.) Zimbra thereby benefits from all the investmentsthat have gone into hardening and opt imizing these technologies.

Second, the Zimbra architecture emphasizes dist ributed syst ems designprinciples that are the basis for t he most scalable systems on theInternet. Of paramount importance to scali ng is part it ioning. Part it ioningleverages "localit y of reference" for bot h processing and dataif certainservers can be specialized to solve some subset of the bigger problem,then the essential code and data are more likely already to be in memoryor close at hand on fast disk. Partitioning techniques include the"vertical" partitioning of functional tasks and the "horizontal" partitioningof data and the associated processing (more below). Partitioning isaugmented by other well-honed dist ributed systems t echniques likeautomated replicat ion, dat a dependent routing, load balancing, andfailover. Overall, these techniques have proven (e.g., Google, Yahoo!,Amazon, etc.) to scale well beyond the reach of more centralizedarchitectures t hat, say, rely on stateless processing and a single very-large database.


22/23


Vert ical part it ioning allows complex processing tasks to be dividedint o subtasks that can be more independent ly opt imized, managed,and debugged. Vert ical part it ioning within Zimbra primari lyconsists of off-loading the computationally expensive securit y t ier ,which interfaces between Zimbra and the greater Internet , fromthe mailbox servers, which manage user datamessages,appointments, contacts, etc. This securit y t ier includes Post f ix (t he

Mail Transfer Agent/ MTA included wit hin Zimbra for mail rout ing,policy, etc.) as well as any on-premises anti-spam and anti-virus(Zimbra includes the leading open source technologies---SpamAssassin and ClamAV, but is also compat ible with alt ernativecommercial AS/ AV). What is more, Zimbra's securit y t ier is"effectively stateless" (the SMTP protocol provides for t heautomatic redelivery of unacknowledged messages, and Zimbradoesn't ackuntil the message is t ransact ionally st ored within theuser s mailbox). This allows independently sizing the ZCS MTAserver farm based on aggregate securit y workload, while let t ing

ZCS automaticall y manage all the dist ributed subcomponents aswell as the rout ing of communications to and from the ZCS mailboxservers (via SMTP & LMTP).

Horizontal part it ioning is far more crit ical f or very large-scaledeployments, since there is generally a lot more data than tasks.Large Zimbra deployments are horizontally part it ioned acrossservers (and the att ached storage) by end user mailboxes. An enduser's mailbox includes his or her messages, calendar, contacts,notes, and so on, which are all collocated for ef f icient user contextswitching. So ZCS servers are inherent ly statefuleach serves as

the "primary" location for a subset of the aggregate mailboxes. Thisrequires that each ZCS server have the smarts to reroute a protocolrequest (via XML/ SOAP, IMAP, POP, .. . ) t o the appropriate primaryserver in the event that an in-bound load balancer makes thewrong decision.

Automated repli cat ion and failover is also essential. For example,LDAP configurat ion data (which includes end user mailboxlocations) is fully replicated to as many replica hosts as arerequired to meet performance and availability requirements. LDAPreplica hosts may be collocated with other ZCS servers or

"vert ically part it ioned" to dedicated servers. Mailbox data, on theother hand, can be transparent ly replicat ed within storage system(such as via RAID or mirroring) for availabil it y only. (It does notmake sense to replicate mail boxes for scalabil it y, given howfrequently state data is updated.) In the future, Zimbra will alsosupport mailbox replication on "vanilla" storage by replaying theZCS t ransact ion or change log (used to guarantee consistency


23/23

between the message and meta-data stores) on a secondary server.In either case, clustering technology is used to automaticallyfailover f rom the primary server t o a preconfigured secondary (ortert iary) server that assumes the role of primary for that mailbox(and assures that the former primary no longer has "write" accessto the mailbox in order to avoid "split-brained syndrome").

Meta-data optimization/partitioning is one of the less frequently

discussed scaling techniques employed within the Zimbraarchitecture. Meta-datafor a mailbox is all of t he data requiredfor "navigat ing" to the appropriate message or meeting. Zimbrameta-data includes ZCS's very efficient, Lucene-based index intoall the text contained in every message, meeting, contact,attached document, and so on. Zimbra meta-data also includes thest ructured meta-dat a that captures folders, t ags, dates, savedsearches, etc. Zimbra uses an off-the-shelf SQL relational databaseto opt imize st ructured meta-data queries and updates, but meta-data is, of course, horizontally partitioned by user. The key insight

is that t his meta-data should also be part it ioned from the targetdata (message, meeting, etc.) to ensure very efficient processing.This allows sophist icated navigation to be nearl y instantaneouseven across very large mailboxes (2 and 3Gb are more typicalnow). Latency in access to the message body itself (which could,for example, reside in an HSM system or ultimately even be splitout in another t ier) is not nearly so problemat ic to the userexperience as latency or inefficiency in accessing the meta-data.Partitioned meta-data also allows potentially expensive operationssuch as compliance-related cross-mailbox discovery to be handled

eff icient ly (via simply composing the appropriate horizontally-partitioned search results).

Thank you for reading. We hope you found this content helpful. TheZimbra and HP teams invite your feedback.

Zimbra Service Provider Benchmark

Documents