Page 1
Understanding your system
Andreas Bjärlestam 2016-03-30
_____ .__.__ / _ \ ____ |__| | ____ / /_\ \ / ___\| | | _/ __ \ / | \/ /_/ > | |_\ ___/ \____|__ /\___ /|__|____/\___ > \//_____/ \/ _____ ______ ____________/ ____\___________ _____ _____ ____ ____ ____ \____ \_/ __ \_ __ \ __\/ _ \_ __ \/ \\__ \ / \_/ ___\/ __ \ | |_> > ___/| | \/| | ( <_> ) | \/ Y Y \/ __ \| | \ \__\ ___/ | __/ \___ >__| |__| \____/|__| |__|_| (____ /___| /\___ >___ > |__| \/ \/ \/ \/ \/ \/ __ __ .__ _/ |_ ____ _______/ |_|__| ____ ____ \ __\/ __ \ / ___/\ __\ |/ \ / ___\ | | \ ___/ \___ \ | | | | | \/ /_/ > |__| \___ >____ > |__| |__|___| /\___ / \/ \/ \//_____/
Page 2
Do you know on top of your head:• How many req/s your system can handle?
• What response time it has?
• How it scales?
• What the bottlenecks are?
• How stable it is over time?
Page 3
Many still do big bang performance testing
Page 4
Stop seeing performance tests as verification
It should be an integrated part of your development cycle
Continuously analyze your system
Page 5
You should do it all the time!
Page 6
I’m lazy, so performance testing must be quick and simple
Page 8
Your test client should do nothing but loadtesting, no interference
Page 9
Amazon AWS + tmux=
win!
Page 10
You can leave it on and come back to check every now and then
Page 11
Combine with monitoring:NewrelicKibana
Graphite
Page 12
> sudo yum install siege
Page 13
> siege -c10 -t30s -d1 -i -f urls.txt
Page 14
siege
• Quick and simple• Instant visual feedback• Good summary of most important metrics• Good enough for most scenarios
= You can work in quick iterations
Page 15
Create a Traffic Model
Page 16
Your best guess of how the system will be used
Page 17
When replacing a system
Get access logs from the old system
Analyze which parts of the system that generate the most load
Plot them over time to get a feeling for peaks and average load
Page 18
If you replay access logs against your system with siege you can
gain a lot of insights and find problems like unhandled urls,
unnecessary redirects etc
Page 19
Build a urls.txt file
Based on traffic model
Fill a file with urls that represent your expected traffic, one url per line
Think about the proportions of different types of urls
Page 20
Build a urls.txt file
Expecting broad traffic -> many urls
Expecting narrow traffic -> not so many
Page 21
Build a urls.txt file
http://example.comhttp://example.com/user/stinahttp://example.com/user/olofhttp://example.com/user/svenhttp://example.com/user/sivhttp://example.com/user/ellenhttp://example.com/country/swedenhttp://example.com/country/norway
Page 22
Build a urls.txt file
curl cut jq grep etc are your friends
Page 23
Build a urls.txt file
You can send POST as well
http://example.com/user/stina POST age=23http://example.com/user/stina POST a=1&b=2http://example.com/user/stina POST <./stina.txt
Page 24
Finding the system limits
Page 25
10 req/s OK100 req/s OK500 req/s Slooow300 req/s OK
Page 26
Scale up with more CPU or processors and try again
Page 27
Does it scale linearly?
Page 28
This is a good time to think about the bottlenecks of your system
I/OCPU
Sync / Async
Page 29
Adjust and try again
Page 30
Keep an eye on system metrics
Response timeCPU load
Memory usageError rates
etc
Page 31
Stability testing
Page 32
Start a linux machine on AWSSet up a session with tmuxStart siegeLeave it running
Page 33
> siege -c10 -t24h -i -d1 -f urls.txt
Page 34
Put your system under continuous load from your first commit
Keep an eye on system metrics every now and then
Page 35
Good to know: siege counts error responses and will stop if it
encounters more than 1024 errors
Page 36
TLDR
Stop doing performance tests like its 1999Put your system under load from day 1
Run tests interactively, be creativeGain understanding
AWS + tmux + siege is awesome!
Page 37
tmux cheat sheet
> tmux new -s loadtest starts a new session named loadtest
ctrl-b + d exits session
> tmux list-sessions lists all current sessions
> tmux attach -t loadtest attaches to the session named loadtest
Page 38
If siege does not fit your use case
You could try- gatling- locust- wrk
They are bigger (more complex) hammers