www.jboss.com 2 IN SEARCH OF HIGH-PERFORMANCE APPLICATIONS 2 PERFORMANCE TUNING PRINCIPLES 2 Why tune for performance? 2 Performance priorities 3 A word about performance benchmarks 5 JBOSS EAP 5.0 TUNING 5 Connection pooling 6 Thread pooling 8 Object and component pools 9 Logging 9 Caching 11 JBoss EAP performance tuning summary 11 Linux-specific tuning: Large memory pages 12 Linux-specific tuning: Tuning the virtual memory manager 12 Database tuning update 12 Case study 17 APPLICATION PERFORMANCE TUNING: A CONTINUOUS PROCESS 18 APPENDIX: USING LARGE-PAGE MEMORY (LINUX-SPECIFIC INSTRUCTIONS) BEST PRACTICES FOR PERFORMANCE TUNING JBOSS ENTERPRISE APPLICATION PLATFORM 5 TIPS AND TRICKS FOR OPTIMIZING YOUR APPLICATION’S PERFORMANCE
20
Embed
Best practices for performance tuning JBoss enterprIse ... · PDF fileBest practices for performance tuning ... recent developments in Java. But aside from that, every application
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
17 applIcatIon performance tunIng: a contInuous process
18 appendIx: usIng large-page memory (lInux-specIfIc InstructIons)
Best practices for performance tuning JBoss enterprIse applIcatIon platform 5 tIps and trIcks for optImIzIng your applIcatIon’s performance
2 www.jboss.com
in search of high-performance applications
Overall,performancetuningisaveryimportantpartofcreating,maintaining,anddeployingasuccessfulbusinessapplication.Whetheryouarebuildingcustomapplicationsordeployingcommercial,off-the-shelfsolutions,youwilllikelyneedtotunetheapplication,thedatabase,themiddleware,orallthree.Infact,75%of performance issues originate with the application itself.
Whenorganizationsselectapplicationmiddleware,performanceisalwaysoneoftheirmostimportantselec-tioncriteria,ifnotthemostimportant.InmanycasesourcustomerstellustheychoseJBossEnterpriseApplication Platform (EAP) because of its superior performance. They know that many users of JBoss EAP areachievingsuperiorapplicationperformancedayafterday.
IfyouarenewtoJBossEAPorperformancetuning,thispaperwillintroduceyoutobestpracticesthatcanhelpyouavoidcommonperformancepitfallsasyouprepareyourapplicationforproduction.Ifyou’reanoldhandatapplicationperformanceissues,youknowthattechnologyisconstantlychanging.Youmaybenefitfrom an update on best practices in JBoss EAP performance tuning.
performance tuning principles
Why tune for performance?
Performance was once considered just another feature of an application. Today it is frequently consid-eredthemostimportantcharacteristicoftheapplication—onethatcanhaveasignificantimpactonyourbusinessandyourproductivity.Consideryourreactiontoaslowwebsite.Ifyou’relikemostpeople,youbecomefrustrated,losepatience,andgoelsewhere.IfthatcompanyiscountingonrevenuefromWebsales,ithasnotonlylostyourattention—ithaslostbusiness.Evenforinternalapplications,poorperformancecanaffectproductivityifusershavetowaitordealwithunpredictablebehavior.Atbest,theymightfeelannoyed,losealittletime,orformanegativeopinionoftheirITdepartments.Atworst,businesstrans-actionsmaybelost,orcustomersmaygowithoutimportantserviceifusersmustworkaroundapoorlyperforming application.
But user experience is not the only reason to tune for performance. A well-performing application will generallyusefewerhardwareandsoftwareresources.Acompanycanoptimizeitsinvestmentinhardwarewhenapplicationsaretunedappropriately,whetherthatmeansusingoldersystemslonger,purchasingnewsystemsthataremoremodestlysized,orusingfewersystemsoverall.Onthesoftwareside,awell-performingapplicationwillgenerallyneedtousefewerCPUcountsorsoftwarelicenses,nomatterwhattypeofsoftwareisinvolved.Reducingsoftwarecostscansavethecompanysignificantmoneyovertime.
timetuningtheircustom-builtapplicationsanddatabasesthantheirunderlyingapplicationservers.Sokeepinmindthatasuperiorlyperformingapplicationservermayhaveonlyaminorimpactontheoverallperfor-mance of your applications.
a Word aBout performance Benchmarks
Some organizations rely on industry-standard performance benchmarks when selecting middleware. While benchmarkscanhelpvendors,theycandeceiveyouasanevaluatorbecausebenchmarkapplicationsareusuallyverydifferentfromtheapplicationsyouwillruninproduction.Thecurrentbenchmarkforapplica-tionservers,forexample,SPECjAppServer2004,isa2004applicationthatdoesn’ttakeadvantageofmanyrecentdevelopmentsinJava.Butasidefromthat,everyapplicationisdifferentandthesystemsonwhichbenchmarkapplicationsarerunmaybeverydifferentfromyours.
performance tunIng prIncIple #2: plan for peaks, not averages
Asyouexamineperformancerequirements,oneofyourgoalsshouldbetodevelopaprofileofyourapplica-tion’sworkloadwithspecialattentiontothepeaks.Forexample,manybusinessapplicationsexperiencedailypeaksinthemorningandafternoonwithavalleyduringthemiddleofthedaywhenemployeesareeatinglunch.Someapplicationsexperiencepeaksattheendofthemonth,quarter,orseason.Yourapplication’sworkloadprofilewilldependonthespecificsofyourbusiness,butyoushouldalwayspayparticularattentiontoperiodsofpeakworkload.Oneofthebiggestmistakesdevelopersmakeistorelyonanaverage(averagedailyworkload,forexample).Averagesarenotsufficienttoensurethatyourapplicationwillperformduringperiods of peak load.
performance tunIng prIncIple #3: alWays Instrument your applIcatIon
Allapplicationsshouldbeinstrumentedtoprovideinformationforperformanceanalysis.Businesscondi-tions,includingcustomerbehaviorandworkloadcurves,canchangedramaticallyovertime,soevenifanapplicationonceperformedwell,itmaynotperformwellundertoday’sconditions.Ifyourunintotrouble,andyourapplicationhasnotbeeninstrumentedforperformance,thenyouhavenoeasywaytoknowwhereyourproblemsare.Butifyourapplicationisappropriatelyinstrumented,thenyou’llbeabletomonitorchanges in business conditions easily and tune your applications to match before problems occur.
Finally,forperformance-criticalsituationswhereoneoftheothertoolsdoesn’tprovidetheinformationyouneed,youcanwriteyourowninstrumentationusingtheJBossAOPframework.Thisframeworkoffersquitea few features that enable you to see exactly what is happening at runtime. Whether you choose to write yourowninstrumentationwilldependontheapplication,yourskillsandavailabletime,andtheimportanceoftheapplication’sperformancecharacteristicstoitsoverallsuccess.
performance tunIng prIncIple #4: understand Where your applIcatIon spends Its tIme
One important reason to instrument your application is to understand where time is being spent. While you wanttoknowabouttimeacrosseachlayerofyourapplicationstack,thetoolswithinJBossEAPwillhelp youunderstandonlypartofthisequation.Ifyourapplicationisspendingtoomuchtimeinthedatabase, forexample,thenyoumayneedtofocusonyourdatabasestatisticstopindowntheproblem.
Byunderstandingwhereyourapplicationspendsitstime,youwillbeabletoavoidthe“shotgunmethod”of performance tuning — trying multiple solutions to common problems without knowing whether any of themarerelevanttoyourproblem.Youmightsolveyourproblemthisway,buttheprobabilityislow.Manydeveloperswhoassumetheyknowwhatiscausingtheirperformanceproblemsaremistaken,resultinginproblemsthatpersistformonthsorsometimesyears.Thatiswhyhavinganobjectivewaytoknowwhereyourapplicationisspendingtimewillhelpyouavoidspendingtoomuchtimesolvingperformanceproblems.
do not assume a linear relationship.• Performancedoesn’toftenscalelinearly.Whilevendors maytellyoutheyexperiencelinearresults,thisisrarelythecaseforrealproductionapplications.
Be conservative in creating your model, and use actual historical data wherever possible.• Ifyou canbaseyourmodelondatafrompastexperiencewithotherapplications,youwillbeabletorefine itovertimeandincreaseyourconfidenceinitsaccuracy.
Connection pooling and thread pooling are the most important areas to consider when you want to maximize throughputonmodernhardware.Intermsofsystemresources,databaseconnectionsareexpensivetosetupandteardown.Butinspiteofthis,someapplicationscreateanewconnectiontothedatabasewitheveryqueryortransactionandthenclosethatconnectionimmediately.Thispracticeaddsagreatdealofoverheadtotransactionprocessingandcanleadtopoorperformance.
Thread pooling is the next most important area to consider as you tune your application for performance. JBossEAPhasrobustthreadpooling,butbeforeyoucansizethethreadpoolsappropriately,youneedtoknow how they are used and which ones might be affecting your application’s performance. The character-isticsofyourspecificapplicationwilldeterminewhichthreadpoolsareusedandwhichonesmightbecomebottlenecks.Thiscanvarysignificantlyfromapplicationtoapplication.Thetablebelowprovidesasummaryof how each thread pool is used.
Ifyouarecertainthataparticularconnectorwillnotbeusedinyourapplication,considerremovingit. Forexample,manyapplicationsuseeitherthehttpdthreadpoolorthemod_jkthreadpoolbutnotboth. Soyoucanremovetheoneyoudon’tneed.monItorIng and tunIng thread pools
MonitoryourthreadpoolsviatheEAP5AdminConsole,whichdisplaysnotonlythenumberofactivethreadsforeachpoolbutalsothequeuesize.InboththeEAP5AdminConsoleandJBossON,youcanadjustanddefinethreadpoolsettingsthatpersist.SettingsadjustedviatheJMXconsolewillnotsurvive areboot.IfyouusetheJMXconsole,remembertogobackandeditthefileifyouwanttheadjustments to be permanent.
thread pool Where is it defined? hoW is it used?
System thread pool Injboss-service.xmlintheconfdirectory ForJNDInaming—thedefaultsettingisfineformost cases
<!-- Define an AJP 1.3 Connector on port 8009 --> <Connector port=”8009” address=”${jboss.bind.address}” protocol=”AJP/1.3” emptySessionPath=”true” enableLookups=”false” redirectPort=”8443” maxThreads=”200” />
example: Jca thread pool
<!-- THREAD POOL --> <bean name=”WorkManagerThreadPool” class=”org.jboss.util. threadpool.BasicThreadPool”> <!-- Expose via JMX --> <annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name=”jboss.jca: service=WorkManagerThreadPool”, exposedInterface=org.jboss.util.threadpool.BasicThreadPoolMBean.class)</annotation> <!-- The name that appears in thread names --> <property name=”name”>WorkManager</property> <!-- The maximum amount of work in the queue --> <property name=”maximumQueueSize”>1024</property> <!-- The maximum number of active threads --> <property name=”maximumPoolSize”>100</property> <!-- How long to keep threads alive after their last work (default one minute) --> <property name=”keepAliveTime”>60000</property> </bean>
oBJect and component pools
Object pools and component pools are essentially the same thing. Their settings represent the number of objectinstances.ForEJB3,twotypesofpoolsaredefinedin<server>/deploy/ejb3-interceptors-aop.xml.ThesearetheThreadLocalPoolandtheStrictMaxPool.Bydefault,StatelessSessionandStatefulSessionBeansusetheThreadLocalPool,whichisbackedbyanInfinitePoolwithnomaximumsize.Therefore,itgrowsaccordingtovolumeinyourapplication.Thishasthedistinctadvantageofnotneedingtobetuned.Bydefault,MessageDrivenBeans(MDBs)usetheStrictMaxPool.Thispoolactuallyobeysamaximum,willqueueuprequestswhenthatmaximumhasbeenreached,andwilltimeoutanythinginthequeueifthereisnotanavailablereferencefromthepool.Inthiscase,thesystemwillthrowanexceptionandiftheproblemoccurredinmid-transaction,youwillexperienceatransactionrollback.Giventheimpactfailedtransactionscanhaveonyourbusiness,youshouldmonitortheStrictMaxPoolcloselyviatheJMXconsole.
Developersshouldtakefulladvantageoflogginginthedevelopmentandtestingphasesoftheapplicationlifecycle.Inproduction,however,loggingcancausebottlenecks.Youwanttobesurethatloggingprovidesyou with useful information without hurting application throughput. Consider making the following changes asyoupromoteyourapplicationintoproduction:
turn off console logging in production.• InJBossEAP’sdefaultconfiguration,consoleloggingisenabled,whichmeansyouhavetheopportunitytoseeallthelogsfromyourIDE.Inproduction,thisisanexpen-siveprocesswithunbufferedI/O.Whilesomeapplicationsmaybefinewithconsolelogging,high-volumeapplicationsbenefitfromturningitoff.JBossEAP5providesaconfigurationset,namedProduction,whichgivesdevelopersabetterstartingpointforcreatingaproductionenvironment.IntheProductionconfiguration,consoleloggingisturnedoff.
turn down logging verbosity.• Thelessyoulog,thelessI/Owilloccur,andthebetteryouroverallappli-cationthroughputwillbe.Loggingisalwaysatradeoff,sothinkcarefullyabouthowmuchloggingyoureally need in production.
use asynchronous logging.• This can make a big difference for high-throughput applications. With asynchronouslogging,logmessageswillgointoaqueueandcontrolreturnstotheapplicationasif the logging had been completed. Then a separate thread executes the log operations from the queue.
Wrap debug log statements with• If(debugEnabled()). This simple practice can make a huge differ-enceifyourapplicationcontainsalotofdebuglogstatements.Withoutthisconditionset,yourapplica-tioncreatesallofthestringobjectsforeachofthelogstatements,andLog4jcreatestheLoggingEventobjectforeachlogstatementregardlessoftheloglevelthatissetbecausetheloglevelischeckedonlyafteralloftheseobjectshavebeencreated.InsomecasesthiscanleadtocreationofthousandsandthousandsoftemporaryStringandLoggingEventobjects,resultinginmemoryandgarbagecollectionissues and reducing throughput dramatically. By placing a conditional wrapper around your debug log statements,youcanensurethatunnecessarylogprocessingdoesnotaffectyourthroughput.
OneoftheeasiestpotentialperformanceenhancementsyoucanmakeistocacheEJB3entities.Todefinewhichentitiesyouwantcached,modifythefilepersistence.xml that you deploy with your EJB 3 applica-tion(anexampleisshownbelow)[email protected]@Cachedannotation,youspecifytheusageasbeingoneofthefollowing:
CacheConncurrencyStrategy.READ_ONLY, READ_WRITE, NONSTRICT_READ_WRITE, or TRANSACTIONAL.
Asalways,itisrecommendedthatanalysisbeconductedwithinyourenvironmenttodeterminehowlarge tomakethepoolsizeasthiswilldependonthepreparedstatementsthatarespecifictoyourapplicationand data source.
Ifyouareusingmessage-drivenbeans,monitortheStrictMaxPoolcloselytoensurethatthemaximum•object pool size is not a bottleneck.
Employdifferentloggingstrategiesfordevelopmentandproduction.Especiallyforhigh-volumeappli-•cations,turnconsoleloggingoff,reduceloggingverbosity,useasynchronouslogging,andalwayswrapdebug log statements with If (debugEnabled ()).
Largepagesupportbeginsat2MBandcanrunashighas256MBonsomehardwarearchitectures.Thesenumberswillvary,andyouwillneedtofindoutthevaluesforyourspecificserver.AllthemajorJVMsystemssupportlargememorypagesonLinux.Becauseitcanbetrickytosetup,weprovidesomespecificinstructions in the appendix.
In/etc/sysctl.conf you can set vm.swappinessto1topreventapplicationsfrombeingswappedto disk when there is memory pressure.
dataBase tunIng update
Moderndatabases,especially64-bit,areextremelyefficientatcachingdata.Intheearlydaysof64-bit databases,largebuffersizeswouldslowperformanceduetoelongatedsearchtimes,butthisisnolongertrue.RedHat'stesting,hasresultedinOLTPapplicationswithverylargebuffercachesthatshowvery good results.
YoushoulduseDirectI/Oifyourdatabasesupportsit.Especiallywithalargecacheorawrite-intensiveworkload,youshouldavoiddoublebufferingwiththefilesystembuffercache.NotethatMySQL5documen-tationindicatesthatqueriesmayslowdownbyuptoafactorofthreewhenusingDIRECTI/O,butRedHathasnotexperiencedthisincaseswhereaproperlysizedbufferpoolisused.IfyouareusingDIRECTI/O,youshouldbesuretousethevirtualmemorysettingmentionedearlier.ConsiderusingasynchronousI/Oifyourdatabase supports it.
case study
TodemonstratethedramaticresultsthatcanbeachievedwithJBossEAP5performancetuning, an experiment was performed using a sample application.
Performanceresultsforbothscenariosareshownbelowinthefollowingsetofgraphs.Thefirstsetofresults measures scalability using mean transactions per second (TPS). The initial test run was conducted withdefault,non-optimizedconfigurations(Figure1A).Thesecondtestrunusedthesameapplication,butleveragedtheperformanceoptimizationshighlightedpreviously(Figure1B).
application performance tuning: a continuous process
Tuninganapplicationforoptimalperformancecanensureapositiveuserexperience,promotebusinessproductivity,andhelptooptimizeuseofhardwareandsoftwareresources.Performancetuningisnotaone-timetask,butanongoingprocessthatensuresawell-performingapplicationasbusinessconditionsandsystemtechnologychangeovertime.Applicationdevelopersandarchitectsshouldalwaysbepreparedtotunetheirapplicationsforperformancebothbeforeandaftertheyareputintoproduction.Asalways,themorebusiness-criticalanapplicationisandthehigherthevolumeoftransactionsitmustsupport,themoreimportant performance tuning will be to your business.
Whentunedinaccordancewiththecharacteristicsofyourapplication,JBossEAPcanprovidesuperiorapplicationperformance.Thispaperhasgivenyouanoverviewofbasicperformanceprinciplesaswellasan introduction to performance tuning best practices for JBoss EAP. Keep in mind that these techniques applytoanyoftheRedHatplatformsthatJBossEAPsupports:JBossEnterprisePortalPlatform,JBossEnterpriseSOAPlatform,andJBossEnterpriseBRMS.
TheSuninstructionsleaveitatthat,butifyoudonothingelseyouwillmostlikelygetthefollowingerror:Failed to reserve shared memory (error-no=12)
The following sections describe additional steps you should complete.
2. set kernel parameters.
Set three kernel parameters in /etc/sysctl.confasfollows:
kernel.shmmax = n • Where n is equal to the number of bytes of the maximum shared memory segment allowed on the system. YoushouldsetittoatleastthesizeofthelargestheapsizeyouwanttousefortheJVM.Alternatively,youcansetittothetotalamountofmemoryinthesystem,andyouwillneverhave torevisitit.
vm.huge_tlb_shm_group = gid• WheregidistheIDofasharedgroupIDfortheusersyouwanttohaveaccesstothelargepages. This setting enables you to limit access to the large memory segment.
Whenlargepagesareallocated,/proc/meminfo will display a non-zero number for HugePages_Total. Ifyoudonotseeanon-zeronumber,thenyouarenotusingthelargepages,andsomethingisconfiguredincorrectly.
WehavesometimesseenaproblemwithMySQLwheretheSELinuxpolicywaspreventingitfromaccessingthe large pages. Check /var/log/messages for avc_denied messages (error-no=13 “permission denied”) in the mysqld.log.
jboss soft memlock 6291456 jboss hard memlock 6291456 mysql soft memlock 6291456 mysql hard memlock 6291456 root soft memlock 6291456 root hard memlock 6291456
Calculations:
3072largepages*2048KBpagesize–3072*2048=6291456
Configuration:/etc/group
AddJBossandMySQLuserstothe501(• hugetlb) group in /etc/grouptogiveuserspermission to attach to the shared memory segment.