Automated performance testing using Maven & JMeter George Barnett, Atlassian Software Systems @georgebarnett
Nov 03, 2014
Automated performance testing
using Maven & JMeter
George Barnett, Atlassian Software Systems@georgebarnett
• Create controllable JMeter tests
• Configure Maven to create a repeatable cycle
• Run this “build” in your CI server every 8 hours
• Find performance regressions faster
WHY?
• Catch regressions quickly
• Stop hunting for bugs
• Code with confidence
• Know the performance cost of features
• Fearless external dependency upgrades
• Go home on time
ATLASSIAN
• 15,000+ customers
• 100+ developers
• 8 software streams
• 1 performance engineer
ATLASSIAN
• JIRA - Issue management
• Confluence - Enterprise Wiki
• Bamboo - Continuous Integration
• FishEye, Crucible & Clover - Dev Tools
• Crowd - Single Sign-On.
BEFORE WE BEGIN
• Continuous Integration is critical
• Keep code working with Unit and Functional tests
• Broken or irregular builds means no code to performance test
• Performance testing is NOT a replacement for other testing (but it helps)
PROCESS
• Write code (1-6 months)
• Compile code
• Send it to QA (or Production?!)
• Performance Regression
• PANIC!
No performance testing!
A BETTER WAY
• Write code
• CI compiles code and runs tests
• Send it to QA (or Production?!)
• Performance Regression
• PANIC!
No performance testing!
A BETTER WAY 2.0
• Performance test artifacts as soon as they pass Unit and Functional testing
• Fast feedback for developers
• No Surprises
TOOLS
• JMeter
• Maven & plugins
• Bamboo & plugins
• A Profiler
JMETER
• Apache Project
• Java
• Supports HTTP, JDBC, SOAP, XML-RPC, FTP, SMTP, LDAP, JUnit & more.
MAVEN
• Apache Project
• Java
• Build Tool
• Dependency management
• Documentation
MAVEN CONCEPTS
• Project Object Model (POM)
• Standardised directory structure
• Build Lifecycle
• Reporting Lifecycle
• Plugins
MAVEN POM<project> <groupid>com.atlassian.com.performance.jira</groupid> <artifactid>automated-performance</artifactid>
<properties> <test.dataset>/opt/dataset/dataset1.zip</test.dataset> </properties>
<build> <!--Main Lifecycle plugins configured here--> </build>
<reporting> <!--Reporting Lifecycle here--> </reporting></project>
MAVEN PLUGINS
• Chronos
• Ant-run
• Cargo
• Maven Assembly plugin
• Maven Dependency plugin
BAMBOO
• Continuous Integration
• Can run builds on Amazon’s EC2
• JMeter Aggregator Plugin
A PROFILER
• Java Profiler
• CPU Snapshots
• GC & Memory information
• Thread usage and contention
• Triggers & offline sampling
STEP 1
• Maven Assembly plugin
• Create a .zip from project directories
• `mvn deploy` to repository as a SNAPSHOT
Package the JMeter test
• Package application state into zip files
• Ant-run - application pre-setup
• Cargo - deploy container
• Chronos - configure the JMeter tests
STEP 2Configure the Maven automated build
MAVEN PROFILES<project> .. <properties> <test.dataset>/opt/dataset/dataset1.zip</test.dataset> </properties>
<profile> <id>dataset1</id> <properties> <test.dataset>/opt/dataset/dataset1.zip</test.dataset> ..
<profile> <id>dataset2</id> <properties> <test.dataset>/opt/dataset/dataset2.zip</test.dataset> ..
MAVEN PROFILES• Invoked on command line
• eg: mvn verify -Pmysql, tomcat5, dataset1
• Configuration in profile is added to build
• Use for a wider variation in test environments
• Use profiles for individual “work units”, eg JMeter
ADDING A PROFILER
<global.jvm.args>-Xms2000m -Xloggc:${project.build.directory}/gc.log</global.jvm.args><jvm.args>${global.jvm.args}</jvm.args>
<profile> <id>jprofiler</id> <properties>
<jvm.args>-agentlib:jprofilerti=offline,id=${jprofiler.profileid},config=${jprofiler.configfile} -Xbootclasspath/a:${jprofiler.home}/bin/agent.jar ${global.jvm.args}</jvm.args>
</properties></profile>
$ export LD_LIBRARY_PATH=”$JPHOME/bin/linux-x64/
REPORTING
• Available in maven
• eg: mvn site / mvn chronos:report
• Generates graphs and a numerical report
• Can compare to historical data
• Static :(
CHRONOS
REPORTINGCHRONOS
REPORTINGCHRONOS
REPORTINGCHRONOS
REPORTINGCHRONOS
REPORTINGCHRONOS - HISTORY
REPORTINGCHRONOS - HISTORY
REPORTING
• Open source Bamboo plugin (supported!)
• James Roper (Atlassian) 20% time project
• Can graph JMeter .JTL and CSV files
• Dynamic
BAMBOO JMETER AGGREGATOR
REPORTINGBAMBOO JMETER AGGREGATOR
TIPS
• Allocate time for fixing tests
• Your tests WILL break
• Work out a process for fixing bugs
• Gather as many artifacts from each test run as possible
• React sooner rather than later
Q&Agbarnett @ atlassian . com
@georgebarnett
Code samples online soon!blogs.atlassian.com/developer
Meet Atlassian @ The Summit:May 31 - June 2 in SF
www.atlassian.com/summit