SmarTest: Proposal for accelerating the detection of faults in Drupal Ana Belén Sánchez #DrupalcampSpain2015
SmarTest: Proposal for accelerating the detection of faults in Drupal
Ana Belén Sánchez #DrupalcampSpain2015
Ana B. Sánchez #DrupalcampSpain2015
¿Who am I? Where do I come from?
Ana Belén Sánchez [email protected]
Research Group of Applied Software Engineering
(www.isa.us.es)
Escuela Técnica Superior de Ingeniería Informática
University of Seville
I am a researcher and a Teacher
Ana B. Sánchez #DrupalcampSpain2015
¿Why am I here, in a Drupal Conference?
Ph.D. in automated testing on variability-intensive systems
Ana B. Sánchez #DrupalcampSpain2015
Variability-intensive systems
Ana B. Sánchez #DrupalcampSpain2015
Testing problems
Too many configura-ons!
Time-‐consuming
Ana B. Sánchez #DrupalcampSpain2015
Testing variability-intensive systems
Test case selection techniques reduce the test space by selecting a portion of the configurations to be tested.
Ana B. Sánchez #DrupalcampSpain2015
Testing variability-intensive systems
Test case prioritization techniques schedule test cases for execution in an order that attempts to increase their effectiveness at meeting some performance goal.
Ana B. Sánchez #DrupalcampSpain2015
Testing variability-intensive systems
Test case selection techniques reduce the test space by selecting a portion of the configurations to be tested.
Test case prioritization techniques schedule test cases for execution in an order that attempts to increase their effectiveness at meeting some performance goal.
Ana B. Sánchez #DrupalcampSpain2015
Testing variability-intensive systems
Artificial models to test
How to evaluate these testing techniques?
Ana B. Sánchez #DrupalcampSpain2015
Towards real highly-configurable tools
Studied Tools
Ana B. Sánchez #DrupalcampSpain2015
Testing of real highly-configurable tools
1. Available code
2. Detailed fault reports
3. Automated test cases
4. Extensive documentation
5. > 630,000 users and developers
6. > 30,000 modules
Ana B. Sánchez #DrupalcampSpain2015
Our approach
Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing in the Wild: The Drupal Case Study Published in Software and Systems Modeling Journal, 2015
(http://www.isa.us.es/publications/type/article-journal/9998/variability-testing-wild-drupal-case-study)
Ana B. Sánchez #DrupalcampSpain2015
Our approach: The Drupal case study
Module .info files
Drupal documentation
Module JIT
Ana B. Sánchez #DrupalcampSpain2015
The Drupal variability model
48 modules 21 dependencies
Ana B. Sánchez #DrupalcampSpain2015
Drupal information
LoC
Module Size
Changes Commits
Tests 352 test cases 24152 assertions
Ana B. Sánchez #DrupalcampSpain2015
Drupal information
CC
Cyclomatic Complexity
Drupal Installations
Drupal Contributors
Ana B. Sánchez #DrupalcampSpain2015
Drupal faults
Bug tracking System
3.232 Faults in v7.23 160 integration faults:
• 132 caused by 2 modules
• 25 caused by 3 modules
• 3 caused by 4 modules
Ana B. Sánchez #DrupalcampSpain2015
Statistical study
Module Size Faults Module
Changes Faults
Faults v7.22
Faults v7.23
Module CC Faults
Ana B. Sánchez #DrupalcampSpain2015
Statistical study
Module Developers Faults
Tests ? Faults ?
Optional Modules Faults
91 %
Core Mandatory Modules Faults
41 %
Ana B. Sánchez #DrupalcampSpain2015
Statistical study
Module Size
Faults Module Changes
Faults Faults v7.22
Faults v7.23
CC Faults Tests ? Faults ?
Developers Faults
Ana B. Sánchez #DrupalcampSpain2015
Statistical study
Optional Modules Faults
Core Modules Faults
91 % 41 %
Ana B. Sánchez #DrupalcampSpain2015
Statistical study
Optional Modules Faults
Core Modules Faults
91 % 41 %
72 % normal faults
16 % major faults
7 % minor faults
5 % critical faults
No apparent correlation
Ana B. Sánchez #DrupalcampSpain2015
Test case prioritization criteria
Size–driven Criterion
Fault–driven Criterion
CC-driven Criterion
Change–driven Criterion
Priori-za-on criteria based on previous sta-s-cal study
Ana B. Sánchez #DrupalcampSpain2015
Evaluation
94.2 % Test case selection
Fault-driven Prioritization
2.
Could we accelerate the detection of faults in Drupal using previous information?
86.2 % Test case selection
1.
Ana B. Sánchez #DrupalcampSpain2015
Applying our proposal to Drupal
SimpleTest
Ana B. Sánchez #DrupalcampSpain2015
Applying our proposal to Drupal
Analyzing the SimpleTest module
Main view SimpleTest
Ana B. Sánchez #DrupalcampSpain2015
Our proposal: SmarTest
Main view SmarTest
Customizable dashboard
Ana B. Sánchez #DrupalcampSpain2015
Our proposal: SmarTest
Ana B. Sánchez #DrupalcampSpain2015
Our proposal: SmarTest
Ana B. Sánchez #DrupalcampSpain2015
Our proposal: SmarTest
Test Prioritization SmarTest
Ana B. Sánchez #DrupalcampSpain2015
Our proposal: SmarTest
Continuos feedback in real time SmartTest
Ana B. Sánchez #DrupalcampSpain2015
Our proposal: SmarTest
¿What improvements do we add to SimpleTest?
SimpleTest
ü Designed to make testing easier
ü A powerful automated testing
ü Selector to select tests to execute
ü Detailed inform. about test execution
ü Possibility to follow the trace of errors
SmarTest
ü Customizable Dashboard with run-time extracted data to guide the testing
ü Test prioritization fo detect faults faster
ü Automated testing with feedback in real time
ü Time-out option to automatically stop the test execution
Ana B. Sánchez #DrupalcampSpain2015
Our proposal: SmarTest
Things that we are working on…
• Improve the dashboard with more information and new widgets
• Include new test prioritization criteria and multi-prioritization criteria
• Study the integration of SmarTest with other test systems: Behat, phpUnit…
• Improve the SmartTest UI
• Add the SmarTest to Drupal sandbox
• ¿Any suggestions?
Ana B. Sánchez #DrupalcampSpain2015
Acknowledgments
This work was partially supported by the European Commission (FEDER), the Spanish and the Andalusian R&D&I programmes (Grants IPT-2012-0890-390000 (SAAS FIRE- 46 WALL), TIN2012-32273 (TAPAS), TIC-5906 (THEOS), TIC-1867 (COPAS)).
Ana B. Sánchez #DrupalcampSpain2015