Testing and Validation of the NPACKage

University of Virginia

NATIONAL PARTNERSHIP FOR ADVANCED COMPUTATIONAL INFRASTRUCTURE

Testing and Validation of the NPACKage

NPACI All-Hands MeetingMarch 19, 2003

Marty HumphreyAssistant Professor

Computer Science DepartmentUniversity of Virginia



NPACKage Project Structure• Participation across the partnership

• UCSB, SDSC, UVA, ISI• Dedicated staff at each site

• Software contributors• Activities focused under current NPACI allocation

• NPACKage specific activities• Packaging team• Builds team• Testing team

• Resource leads• Responsible for local deployment

• User Services• Executive Committee

• Frequent review of progress



What’s So Tough about Software Test?

• Software has lousy quality historically• Simple program can have huge space of

execution• “Testing is not innovation but rather

verification”



Why Does Software Have Bugs?

• miscommunication • software complexity• programming errors• changing requirements• time pressures• poorly documented code• bugs in software development tools



Basics of Software Testing

• Unsatisfied goals • Find cases where the program does not do what it is

supposed to do

• Unwanted side effects• Find cases where the program does things it is not

supposed to do



Methodology

1. Define the expected output or result. 2. Don't test your own programs. 3. Inspect the results of each test completely. 4. Include test cases for invalid or unexpected conditions.5. Test the program to see if it does what it is not

supposed to do as well as what it is supposed to do.6. Avoid disposable test cases unless the program itself is

disposable. 7. Do not plan tests assuming that no errors will be found. 8. The probability of locating more errors in any one

module is directly proportional to the number of errors already found in that module.



Terminology

• White box testing• Based on knowledge of internal logic/source code

• Black box testing• Not white box testing :^)

• Unit testing• To test particular functions or code modules

• Integration testing• Testing of combined parts



Terminology (cont.)

• Performance / Stress testing• Good for testing scalability

• Regression testing• Re-testing after modifications

• Acceptance testing• Performed by end-users

• Completion criteria• How do you know when you’re done?



Don’t the individual projects test their software?

• Yes, of course, but we can do more• Value of centralized testing

• Complex interaction tests can be better achieved organizationally, not in isolation

• Incomplete testing of any component reflects poorly on the organization (NPACI)

• Independent testing can greatly improve software quality



Value Added at UVa

• Work closely with other NPACKagers• NPACKage packagers (Larry Miller, UCSB; Bill Link, SDSC)• NPACKage “Builders” and “Testbed Deployers” (Mats Rynge,

USC/ISI)• UVa: RH Linux 7.3 (8.0?), AIX 4.3, [Solaris]• Combination of “black box” and “white box”• Use Drivers and Stubs provided by NPACKage

contributors• Provide feedback to NPACKage contributors

• Goal: Testing Deployment• “Given a Deployment, does it work?”• NPACI Testbed and NPACI Production

• Goal: Testing Deployability• Broader impact in community



Software Contributor

PackageL.Miller, B.Link, Contributor

Testing: packaging/build/testbed-deploy; Platform; Integration; Component; ApplicationM.Humphrey, Contributor, L.Miller, B.Link

Deploy toTestbed

M.Rynge

BuildM.Rynge

Team certifies that distribution is ready for Resource Site ReviewM.Humphrey, J.McGee, L.Miller, B.Link, M.Rynge, R.Wolski

Process Flow

Feedback Flow

Distribution Reviewed by Resource Site(s)N.Wilkins-Diehr, other RS people?, M.Humphrey

Distribution Deployed Across Resource SitesRS-Administrators, N.Wilkins-Diehr

A

B

B A Pacman software cache has been heavily tested and made available to the resource sites

A A Pacman software cache has been prepared, and is ready for cross platform build, deploy, and test



Software for the NPACKage

• NMI• Globus, Condor-G, NWS, GSI-OpenSSH, KX.509, GridConfig,

(MyProxy, MPICH-G2)• DataCutter• Storage Resource Broker – SRB • GridPort• Ganglia• APST• GridSolve• LAPACK For Clusters – LFC



Testing of NMI: SURA

• UVa: part of SURA NMI Testbed• Globus• Condor-G• NWS• GSI-OpenSSH

• UVa co-PI on MyProxy NMI grant



Testing of DataCutter

• Last extracted from CVS on March 11, 2003• Tests from developers

a. Make sure the daemons start g. Placement testb. Buffer test h. Layout testc. Instance test i. Cluster testd. Returns test j. Directory teste. Crash test k. Grid array averagerf. Endian test

• Thanks to Shannon Hastings



Testing of SRB

• Just received SRB 2.0.0/2.0.1 early this week• Plan

• Exercise unit tests in SRB (awaiting descriptions)• Interactions with other NPACKage components

• DataCutter• GridPort• GridFTP• GSI

• Many clients, server

• Thanks to Reagan Moore, Wayne Schroeder



Testing of GridPort

• Last extracted (v 2.2) from CVS on March 14, 2003

• Run test Perl scripts from distribution• We use in our Alpha Project

• “Protein Folding on the Grid” (with C. Brooks, M. Crowley of TSRI)

• Thanks to Maytal Dahan, Mary Thomas



Testing of Ganglia

• Last extracted from CVS on March 11, 2003• Currently executing on 9 nodes on UVa’s Centurion• Tests

• Start a ‘gmond’ and pull its XML tree (“telnet localhost 8649”); verify with “xmllint --verify out.xml“

• Planned• The ganglia-python package from SDSC's Rocks group includes an

MDS provider that obtains data from ganglia. • Is the information correct?

• Thanks to Federico Sacerdoti, Phil Papadopoulos



NPackage Testbed

• Machine contributions from resource sites • Many thanks for the access to the systems for development and

testing, especially:• U.Mich

– 6 AIX4.3, 1 ia64 RH7.1• SDSC

– Cluster: 4 RH7.3, 2 AIX5.1• UCSB

– 1 SUSE 8.1• USC/ISI

– 1 RH7.2

• Point your favorite LDAP browser at:• giis.npaci.edu:2135



Testing NPACKage Stage-1

• Resource discovery and monitoring infrastructure• Hierarchical discovery and monitoring cache/index (MDS)• Host resource information (MDS)• Cluster resource inform (Ganglia)

• Binaries• ia64-redhat-7.1 • power3-aix-4.3 (testing...)• power3-aix-5L • x86-redhat-7.2 • x86-redhat-7.3 (tested)• x86-suse-8.1

• Is the information correct?



Automation

• Some tools exist• Dart• Tinderbox / TreeFire (Mats)• “Inca Build System” (TeraGrid)

• Issue: Determining Cause/Blame• Builder• Deployer• Packager• Resource Provider• Software Developer



Bottom Line

• NPACI software infrastructure is transitioning to production

• Evolving plan for testing and validationProviders’ tests tests at UVa tests on NPACI testbed deployment to NPACI sites

• Integrate and influence NMI testing and TeraGrid testing• Next: APST v2.0, GridSolve, LFC• NPACKage: infrastructure (and soon) integration with

Applications (e.g., CHARMM Portal)

Testing and Validation of the NPACKage

Documents

disposable test cases

software qualityvalue

testing deploymentgiven

isolationincomplete

npaci testbed

effectsfind cases

number of errors

npaci production goal