Load Testing
Post on 23-Feb-2016
41 Views
Preview:
DESCRIPTION
Transcript
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
Load Testing
Dennis Waldron, CERN IT/DM/DACASTOR Face-to-Face Meeting, Feb 19th 2009
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Outline
• Certification (functionality) setup• Stress testing setup• Current stress tests
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Certification Setup (CERT2)
DLF NS & CUPV Stager
Database Frontend
HeadnodeNS & CUPV
Nam
e se
rver
Syn
chro
nisa
tion
Mover callbacks
Hostnames: lxc2disk[21-24]4 diskservers in total10 filesystems, 15GB each, 150GB in total per diskserverNo RAIDSingle core, 1.8GB of RAMSLC4, x86_64 ONLY!
Hostnames: lxcastorsrv101(NS & CUPV) lxcastordev10 (aliased: castorcert2)Dual core, 3.7GB of RAMNo RAIDSLC4, x86_64 ONLY!Running secure and unsecure services
Note: NS & CUPV frontend is shared across all castor development machines!
Hostname: castordev64
ORACLE 10.2.0.4Non RAC based.RHES4, x86_644GB of RAM500GB disk spaceNot dedicated, also contains the schemas for:
All certification stager and DLF databasesThree development setupsThe development NS and CUPVDevelopment SRM, Repack and VDQM
NO TAPE CERTIFICATION!!DISKCACHE ONLY
VIR
TUA
L M
AC
HIN
ES
1 Physical machine, 8 Cores, 16GB of RAM“CASTOR in a BOX”
1
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Certification Setup - Notes
• Key principle: “Everything on Virtual Machines”• Why?
– Hardware is expensive! (currently: 7 Dom0’s = 40 DomU’s)– Minimise on idle resources.– Reduce power consumption.
• All CASTOR releases are functionality tested on:– CASTORCERT1 2.1.7 (-24)– CASTORCERT2 2.1.8 (-6) + XROOT (1.0.4-2)
• SLC4, 64 bit guests ONLY!!!• Fully quattor’ised (CERT reinstall < 2 hours)• Test suites are evolving all the time. ~ 250 test cases (mainly
functionality)
• Future plans:– Development setup for tape– 2 additional certification setups for SLC5 (CERT3 and CERT4)– 4 SRM endpoints (2.7 and 2.8 series)– Nightly CVS builds, installations and tests
2
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Stress Test Setup - ITDC42 DiskserversOut of warranty hardware (i.e > 3 years old)All SLC4, x86_64Configured in 5 services classes: (default, itdc, diskonly, replicate, repack)Mixed hardware types (non homogeneous). Therefore diskservers have:
Different capacities Different number of filesystems Different performance results
Two databases:
c2itdcstgdb (stager):ORACLE 10.2.0.4Identical to production VO databases but non-RAC based (i.e. Single node)
c2itdcdlfdb (DLF):ORACLE 10.2.0.4RHES4, x86_644GB of RAM500GB disk space
Headnode 1:jobManager, rmMasterDaemon (master), stager, nsdaemon, repackserver, LSF (master), rhserver (internal/private)
Headnode 2:jobManager, rmMasterDaemon (slave), stager, LSF (slave), rhserver (public), expertd, mighunter, rechandler, rtcpclientd, dlfserver
Hardware:8 Core, 16GBSLC4, x86_64Production
Production
Headnode 1 Headnode 2
c2itdcstgdb DLFStager c2itdcdlfdb
Tape
Sto
rage
Cen
tral
Ser
vice
s
Pro
duct
ion
300 clients nodes
OUTDATED!!!
3
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Stress Test Setup - Notes
• Diskservers with hardware problems are retired (Good testing for admin tools )
• ORACLE database for the stager is:– Less powerful than production– Runs on a single node (Why? Provides more detailed deadlock
trace files)• Runs the lemon-sensor-castormon.
– Provides detailed profiling of memory, file descriptor usage, thread activity, nb core dumps and key performance indicators.
• Access to 300 dedicated lxbatch nodes, 4-8 cores each.– Heavily dependant on resource allocation plans.– On average the stress test runs with 100 nodes.
• Future plans:– Installation of an additional 20 diskservers for SLC5 tests.– Split of resources into 2 stress test setups (SLC4, SLC5, 2.1.[8|9])
4
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Current tests
• Several categories of tests:– Throughput tests, maximum IO rates.– Database loading tests, max ops/sec.– Scheduling throughput.– Dedicated activity tests.
• Throughput tests:– Require enough clients to saturation diskserver connectivity.– File writing, 100M, 2G, 10G in a loop.– Depending on the test case, files maybe deleted or read (0..n)
times after creation, from the same service class or different service classes.
– Tape is involved when appropriate.
• Database load tests:– Designed to stress the stager database.– Exposes row lock contention and potential deadlocks.
5
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Current tests II
– Requires very few physical clients, but many threads!!– Typical test, mass file queries on a list of files.
• Scheduling throughput:– Designed to reach LSF’s maximum submission rate of 20 jobs per
second.– Very small files, 3k but 1000’s of clients. x2 the number of
available job slots.– Includes replicationOnClose functionality tests D2T0.
• Dedicated activity tests:– Repack.– Tape both migration and recall.– Race condition tests (small no of files, random commands)– Diskserver draining tools.– The killer!
• As many tests as possible running simultaneously, all service classes used, all job slots occupied.
6
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Conclusion
• Many test scripts exist (/afs/cern.ch/project/castor/stresstest)– Heavily customized to the CERN environment.– Each test should run for 24-48 hours.
• Requires expert knowledge. There is no green light at the end!• Not all elements of CASTOR are tested!• Tests are customized for each CASTOR version.
• Requires effort to find bugs!– Looking at database states.– Reviewing errors in log files.– Hunting for inconsistencies.– Reviewing monitoring information.
7
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Questions?
8
top related