National Alliance for Medical Image Computing Testing with CTest and CDash Julien Jomier Kitware Inc.

National Alliance for Medical Image Computing http://na-mic.org

Testing with CTest and CDash

Julien JomierKitware Inc.


Testing Overview

CVS/SVN

CDash

Web Browser

CTest

Developers


Testing Terminology

Sites

dash5.kitware

iris.elmtech

Linux-g++

Builds

Linux-gcc3

Linux-gcc4

Dashboard


Why Test Daily?

• Large code base too large for any single developer to understand

• Developers distributed around the world

• Identify problems as they occur

• Insure that object API remains unchanged

• Provide feedback to developers as they experiment with new implementations


“If it’s not tested it doesn’t work”

• Dasboards and Software Process– Roots in GE Six Sigma initiative

• Cross platform testing

• Nightly regression testing Memory (Purify/ValGrind) testing

• Coverage testing

• Style checking

• Documentation evaluation


How to write a good test?

• Use empirical values (5/2=2.5)

• Test most of the use cases (even the boring/most difficult ones)

• Make your tests as efficient as possible

• Have someone else write the test for your algorithm


Regression Testing

• Compare generated image against standard “correct” image– pixel-by-pixel comparison– can use a threshold metric– adjusted for effects like dithering


CTest

• Testing client

• Distributed with CMake

• Submit test results to the dashboard

• XML submission

• Independent of CMake


CTest with CMakeCMakeLists.txtINCLUDE(CTest)ENABLE_TESTING()# Add a simple testADD_TEST(CDashTest1 CDashTest CDashTest) ADD_EXECUTABLE(CDashTest main.cxx)

CTestConfig.txtset(CTEST_PROJECT_NAME "MyProject")set(CTEST_NIGHTLY_START_TIME "21:00:00 EDT")

set(CTEST_DROP_METHOD "http")set(CTEST_DROP_SITE "localhost")set(CTEST_DROP_LOCATION "/CDash/submit.php?

project=MyProject")set(CTEST_DROP_SITE_CDASH TRUE)


Testing Submission

XML Submission

XML Parsing

HTTP PUT

CTest

CDash

Backup XML File

Fill Database

Send Email

Update RSS Feed

Update.xml

Configure.xml

Build.xml

Test.xml

Coverage.xml

DynamicAnalysis.xml

Compiler/Tester


CDash

• New generation of Dart• Written in Php/Javascript• MySQL• Open Source• Run on top of a web server• XSL implementation• Web 2.0


CDash Features

• Ajax requests• Graphs: build, test times• Builds group• User management• Email notifications• Timing defects


Coverage Support

• Email Low Coverage• Bullseye support


Build Notes

• Share with other developers• Keep it simple


My CDash

• Personal Page• Subscribe to project• View sites• Email preferences


Project Roles

• Normal user• Site maintainer• Project administrator


Claiming Sites

• Site information• Automatic (CTest 2.6)• Information history• Manual editing• Personal view/email


CDash Misc.

• Automatic Geolocation from IP addresses

• Support for iPhone


Installing CDash

• (L/W)AMP : Apache, MySQL, PHP- XAMPP, EasyPHP

• Download CDash (www.cdash.org) in your Apache web directory

• Edit config.php• Point Web Browser to CDash

- Setup the database- Create first project


Style Checking

• KWStyle• C++ Style checker• Configuration via XML• HTML Generator (TBI in CDash) • Open Source• Written in C++


Contribute• Website

- http://www.cdash.org- http://public.kitware.com/KWStyle

• Bug Tracker- http://www.cmake.org/Bug

• Mailing List- http://public.kitware.com/cgi-bin/mailman/listinfo/cdash


Happy Testing!

National Alliance for Medical Image Computing Testing with CTest and CDash Julien Jomier Kitware Inc.

Documents