The Impact of Continuous Integration on Other Software Development Practices: A Large-Scale Empirical Study Yangyang Alexander Yuming Vladimir Bogdan Zhao Serebrenik Zhou Filkov Vasilescu Nanjing U TU Eindhoven Nanjing U DECAL at UC Davis STRUDEL at CMU @aserebrenik @vfilkov @b_vasilescu
38
Embed
The Impact of Continuous Integration on Other Software ... · Continuous Integration on Other Software Development Practices: A Large-Scale Empirical Study Yangyang Alexander Yuming
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
The Impact of Continuous Integration on
Other Software Development Practices: A Large-Scale Empirical Study
Yangyang Alexander Yuming Vladimir BogdanZhao Serebrenik Zhou Filkov Vasilescu
Nanjing U TU Eindhoven Nanjing U DECALat UC Davis
STRUDEL at CMU
@aserebrenik @vfilkov @b_vasilescu
https://imgur.com/0TBJ9OW
Happy Halloween!
Interventions are common in software engineering
• SVN —> git
• push —> pull request
• ? —> continuous integration
• …
Interventions are common in software engineering
• SVN —> git
• push —> pull request
• ? —> continuous integration
• …
How to measure effects using trace data?
Evaluating the effects of an intervention: before vs. after
change in slope
t-test no difference
change in slope
Evaluating the effects of an intervention: before vs. after
change in level
Evaluating the effects of an intervention: before vs. after
change in level
t-test no difference
Evaluating the effects of an intervention: before vs. after
Today
Methodology to empirically study the
effects of an intervention(continuous integration)
†Department of Computer Science ‡College of ComputerUniversity of California, Davis National University of Defense Technology
Davis, CA 95616, USA Changsha, 410073, China{vasilescu, ptdevanbu, vfilkov}@ucdavis.edu {yuyue, hmwang}@nudt.edu.cn
ABSTRACTSoftware processes comprise many steps; coding is followedby building, integration testing, system testing, deployment,operations, among others. Software process integration andautomation have been areas of key concern in software engi-neering, ever since the pioneering work of Osterweil; marketpressures for Agility, and open, decentralized, software de-velopment have provided additional pressures for progress inthis area. But do these innovations actually help projects?Given the numerous confounding factors that can influenceproject performance, it can be a challenge to discern the ef-fects of process integration and automation. Software projectecosystems such as GitHub provide a new opportunity inthis regard: one can readily find large numbers of projectsin various stages of process integration and automation, andgather data on various influencing factors as well as produc-tivity and quality outcomes. In this paper we use large,historical data on process metrics and outcomes in GitHubprojects to discern the e↵ects of one specific innovation inprocess automation: continuous integration. Our main find-ing is that continuous integration improves the productivityof project teams, who can integrate more outside contribu-tions, without an observable diminishment in code quality.
Categories and Subject DescriptorsD.2.5 [Software Engineering]: Testing and Debugging—Testing tools
⇤Bogdan Vasilescu and Yue Yu are both first authors, andcontributed equally to the work.
1. INTRODUCTIONInnovations in software technology are central to economic
growth. People place ever-increasing demands on software,in terms of features, security, reliability, cost, and ubiquity;and these demands come at an increasingly faster rate. Asthe appetites grow for ever more powerful software, the hu-man teams working on them have to grow, and work moree�ciently together.
Modern games, for example, require very large bodies ofcode, matched by teams in the tens and hundreds of devel-opers, and development time in years. Meanwhile, teamsare globally distributed, and sometimes (e.g., with opensource software development) even have no centralized con-trol. Keeping up with market demands in an agile, orga-nized, repeatable fashion, with little or no centralized con-trol, requires a variety of approaches, including the adop-tion of technology to enable process automation. ProcessAutomation per se is an old idea, going back to the pio-neering work of Osterweil [32]; but recent trends such asopen-source, distributed development, cloud computing, andsoftware-as-a-service, have increased demands for this tech-nology, and led to many innovations. Examples of such in-novations are distributed collaborative technologies like gitrepositories, forking, pull requests, continuous integration,and the DEVOPS movement [36]. Despite rapid changes, itis di�cult to know how much these innovations are helpingimprove project outcomes such as productivity and quality.A great many factors such as code size, age, team size, anduser interest can influence outcomes; therefore, teasing outthe e↵ect of any kind of technological or process innovationcan be a challenge.
The GitHub ecosystem provides a very timely opportu-nity for study of this specific issue. It is very popular (in-creasingly so) and hosts a tremendous diversity of projects.GitHub also comprises a variety of technologies for dis-tributed, decentralized, social software development, com-prising version control, social networking features, and pro-cess automation. The development process on GitHub ismore democratic than most open-source projects: anyonecan submit contributions in the form of pull requests. A pullrequest is a candidate, proposed code change, sometimesresponsive to a previously submitted modification request(or issue). These pull requests are reviewed by project in-siders (aka core developers, or integrators), and accepted ifdeemed of su�cient quality and utility. Projects that aremore popular and widely used can be expected to attractmore interest, and more pull requests; these will have to be
ESEC/FSE ‘15
Closed PRsAmong others:
• On average, more PRs are being closed per unit time after adopting Travis CI