Top Banner
Systematic Literature Review Prof. Dr. Tiago Silva da Silva (ICT/UNIFESP) [email protected]
110

Systematic Literature Review

Jan 17, 2017

Download

Education

Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Systematic Literature Review

Systematic Literature Review

Prof. Dr. Tiago Silva da Silva (ICT/UNIFESP)

[email protected]

Page 2: Systematic Literature Review

Literature Review

Page 3: Systematic Literature Review

Literature Review

• Experts can be wrong.

Page 4: Systematic Literature Review

Literature Review

• Experts can be wrong.

• Researcher’s choice of “related studies” can be biased.

Page 5: Systematic Literature Review

Literature Review

• Experts can be wrong.

• Researcher’s choice of “related studies” can be biased.

• Informal reviews can miss important studies.

Page 6: Systematic Literature Review

Systematic Literature Review

• Systematic Review

• Medical domain

• Evidence-based software engineering (Kitchenham)

• “We cannot have evidence-based software engineering without a sound methodology for aggregating evidence from different empirical studies. SRs provide that methodology.”

Page 7: Systematic Literature Review

Systematic Literature Review

• The major advantage of a SR is that it is based on a well-defined methodology.

• SRs aim to find, assess, and aggregate all relevant evidence about some topic of interest in a fair, repeatable, and auditable manner.

• A researcher conducting a SR selects empirical studies that are relevant to the particular research question, assesses the validity of each one, and then determines the trend shown by those studies.

Page 8: Systematic Literature Review

Systematic Literature Review

• To investigate all available evidence that supports or refutes a particular “topic of interest”.

• To summarize the existing evidence concerning a treatment or technology, e.g. to summarize the empirical evidence of the benefits and limitations of a specific agile method.

• To identify any gaps in current research in order to suggest areas for further investigation.

• To provide a framework/background in order to appropriately position new research activities.

• Sometimes a single researcher, such as a PhD student, needs to do this activity alone, but that limitation raises the risk of missing relevant papers.

Page 9: Systematic Literature Review

Advantages

Page 10: Systematic Literature Review

Advantages

• The well-defined methodology makes it less likely that the results of the literature are biased, although it does not protect against publication bias in the primary studies.

Page 11: Systematic Literature Review

Advantages• The well-defined methodology makes it less likely that

the results of the literature are biased, although it does not protect against publication bias in the primary studies.

• They can provide information about the effects of some phenomenon across a wide range of settings and empirical methods. If studies give consistent results, systematic reviews provide evidence that the phenomenon is robust and transferable. If the studies give inconsistent results, sources of variation can be studied.

Page 12: Systematic Literature Review

Disadvantage

Page 13: Systematic Literature Review

Disadvantage

• Requires considerably more effort than traditional literature reviews.

Page 14: Systematic Literature Review

Systematic Literature Review

• Planning the review

• Conducting the review

• Reporting the review

Page 15: Systematic Literature Review

Systematic Literature Review

Systematic Review in Software Engineering Technical Report ES 679/05

COPPE/UFRJ/PESC Biolchini, Mian, Natali, Travassos 10

articles are extracted and synthesized during the result analysis phase. Meanwhile which one of these phases is executed, their results must be stored. Therefore, systematic review packaging is performed through the whole process. There are two checkpoints in the proposed systematic review process. Before executing the systematic review, it is necessary to guarantee that the planning is suitable. The protocol must be evaluated and if problems are found, the researcher must return to the planning stage to review the protocol. Similarly, if problems regarding web search engines are found during the execution phase, the systematic review may be re-executed.

Figure 2. Systematic Review Process.

The stages listed above may appear to be sequential, but it is important to recognize that many of the stages involve iteration. In particular, many activities are initiated during the protocol development stage, and refined when the review proper takes place. For example [Kitchenham et al., 2002]: - The selection of primary studies is governed by inclusion and exclusion criteria.

These criteria are initially specified when the protocol is defined but may be refined after quality criteria are defined.

- Data extraction forms initially prepared during construction of the protocol will be amended when quality criteria are agreed. Data synthesis methods defined in the protocol may be amended once data has been

collected.

4. The Developed Template for Systematic Reviews in Software Engineering

Despite it importance, conducting systematic reviews is not a simple task. A systematic review uses specific concepts and terms that may be unknown to researches used to conduct informal reviews. Besides, systematic reviews require an additional conduction effort. The review must be planned before execution and the whole process must be documented, including, intermediary results.

To facilitate systematic reviews planning and the execution, a review protocol template has been developed. This template was based on systematic review protocols developed in the medical area, on the guidelines proposed by [Kitchenham, 2004] and the protocol example found in [Mendes and Kitchenham, 2004]. This template can be seen in Appendix 1.

The objective of this template is to serve as a guideline to Software Engineering researchers when conducting the systematic review. Therefore, the template lead researchers through each step of the systematic review process presented previously, defining clearly the content of each protocol section.

Planning Execution Result Analisys

Packaging

[ protocol plan approved ]

[ protocol plan disapproved ]

[ execution approved ]

[ execution disapproved ]

Planning Conducting Reporting

Page 16: Systematic Literature Review

Systematic Literature Review

of applying the systematic literature review process to thesoftware engineering domain for the purposes of identifyingthose aspects of the process that transfer ‘as is’ to softwareengineering, those that need to be adapted to the particularcharacteristics of the domain and those areas whereimprovements to current software engineering infrastruc-ture and practices are needed in order to gain greatest advan-tage from the process.

Section 2 provides an introduction to systematic litera-ture review highlighting the main phases and stages ofthe process. This is followed, in Sections 3–5, respectively,by overviews of three systematic reviews relating to servicebased systems, the technology acceptance model and guide-lines for conducting systematic literature reviews. Lessonslearned from our experiences of undertaking these reviewsare then described and discussed, and finally some conclu-sions are drawn.

2. Systematic literature review

The wider context for this study is that of investigatingthe use of the evidence-based paradigm in software engi-neering. The possibility of applying the evidence-based par-adigm to the software engineering field was raised,discussed and enthusiastically supported at ICSE 2004(Kitchenham et al., 2004). The goal of evidence-based soft-ware engineering (EBSE) is summarised by Kitchenhamet al., as being: ‘‘to provide the means by which currentbest evidence from research can be integrated with practi-cal experience and human values in the decision makingprocess regarding the development and maintenance ofsoftware’’.

Within the medical domain, the steps that need to be fol-lowed to practise evidence-based medicine have been iden-tified and documented (Sackett et al., 2000), and these havesubsequently been re-formulated to address evidence-basedsoftware engineering (Dyba et al., 2005; Kitchenham et al.,2004). The steps are:

1. convert the need for information (about a technique,procedure, etc.) into an answerable question;

2. find the best evidence with which to answer the question;3. critically appraise the evidence for its validity (closeness

to the truth), impact (size of the effect), and applicability(usefulness);

4. integrate the critical appraisal with software engineeringexpertise and with stakeholders’ values andcircumstances;

5. evaluate the effectiveness and efficiency in executingsteps 1–4 and seek ways to improve them.

The first three of these steps essentially constitute a sys-tematic review of the literature, conducted in order to pro-vide a balanced and objective summary that is relevant tomeeting a particular need for information. A systematic(literature) review is ‘‘a means of evaluating and interpret-ing all available research relevant to a particular research

question or topic area or phenomenon of interest’’ (Kitch-enham, 2004). The research papers summarised in thereview are referred to as primary studies, while the reviewitself is a secondary study. The accumulation of evidencethrough secondary studies can be very valuable in offeringnew insights or in identifying where an issue might beclarified by additional primary studies. For example, astudy of software cost overruns showed that the resultsreported in a highly influential study carried out in theearly 1990s (The CHAOS report) were significantly outof step with those reported in other studies (Jørgensenand Moløkken-Østvold, 2006). A critical evaluation ofthe CHAOS report by Jørgensen and Moløkken-Østvoldidentified several methodological problems. Anotherexample, where new insights have emerged, is a systematicreview of statistical power in software engineering experi-ments (Dyba et al., 2006). Here, the results show ‘‘that thestatistical power of software engineering experiments fallssubstantially below accepted norms as well as the levelsfound in related discipline of information systemsresearch’’. The authors go on to make recommendationsabout how empirical software engineering researchersmight address the reported shortcomings.

Performing a systematic review involves several discreteactivities, which can be grouped into three main phases:planning; conducting the review; and reporting the review.Fig. 1 illustrates the overall 10-stage review process.

Systematic literature reviews are primarily concernedwith the problem of aggregating empirical evidence whichmay have been obtained using a variety of techniques,and in (potentially) widely differing contexts—which iscommonly the case for software engineering. While theyare used in information systems research (Webster andWatson, 2002), they are less common in software engineer-ing (however, see (Glass et al., 2002) as an example of a sec-ondary study that samples literature within the softwareengineering domain). Indeed, at the present time, outsideof information systems research, reviews in any form, aswell as review journals are really not part of the computing

9. Write Review Report

10. Validate Report

2. Develop Review Protocol

3. Validate Review Protocol

Phase 1: Plan Review

Phase 2: Conduct Review

Phase 3: Document Review

8. Synthesise Data

4. Identify Relevant Research

5. Select Primary Studies

7. Extract Required Data

6. Assess Study Quality

1. Specify Research Questions

Fig. 1. Systematic literature review process.

572 P. Brereton et al. / The Journal of Systems and Software 80 (2007) 571–583

Page 17: Systematic Literature Review

Planning the Review

1. Identification of the need for a review

2. Commissioning a review*

3. Specifying the research question(s)

4. Developing a review protocol

5. Evaluating the review protocol*

Page 18: Systematic Literature Review

Conducting the Review

1. Identification of research

2. Selection of primary studies

3. Study quality assessment

4. Data extraction and monitoring

5. Data synthesis

Page 19: Systematic Literature Review

Reporting the Review

1. Specifying dissemination mechanisms

2. Formatting the main report

3. Evaluating the report*

Page 20: Systematic Literature Review

Planning

Page 21: Systematic Literature Review

The need for a systematic review

• What are the review’s objectives?

• What sources were searched to identify primary studies? Were there any restrictions?

• What were the inclusion/exclusion criteria and how were they applied?

• What criteria were used to assess the quality of primary studies?

Page 22: Systematic Literature Review

The need for a systematic review

• How were quality criteria applied?

• How were the data extracted from the primary studies?

• How were the data synthesized?

• How were differences between studies investigated?

• How were the data combined?

• Was it reasonable to combine the studies?

• Do the conclusions flow from the evidence?

Page 23: Systematic Literature Review

The Research Question(s)

• The most important part of any systematic review. The question(s) drive(s) the entire methodology:

• The search process must identify primary studies that address the research questions.

• The data extraction process must extract the data items needed to answer the questions.

• The data analysis process must synthesize the data in such a way that the questions can be answered.

Page 24: Systematic Literature Review

The Research Question(s)

• Assessing the effect of a technology.

• Assessing the frequency or rate of a project development factor such as the adoption of a technology, or the frequency or rate of project success or failure.

• Identifying cost and risk factors associated with a technology.

• Identifying the impact of technologies on reliability, performance and cost models.

• Cost benefit analysis of employing specific software development technologies or software applications.

Page 25: Systematic Literature Review

The Research Question(s) example

• Question 1: What evidence is there that cross-company estimation models are not significantly different from within-company estimation models for predicting effort for software/Web projects?

• Question 2: What characteristics of the study data sets and the data analysis methods used in the study affect the outcome of within- and cross-company effort estimation accuracy studies?

• Question 3: Which experimental procedure is most appropriate for studies comparing within- and cross-company estimation models?

Page 26: Systematic Literature Review

The Research Question(s) structure• Population: software or web-project.

• Intervention: cross-company project effort estimation model.

• Comparison: single-company project effort estimation model.

• Outcomes: prediction or estimate accuracy.

Page 27: Systematic Literature Review

Developing a Review Protocol

• A review protocol specifies the methods that will be used to undertake a specific systematic review

• Background: the rationale for the survey.

• The research questions that the review is intended to answer.

• The strategy that will be used to search for primary studies including search terms and resources to be searched: digital libraries, specific journals and proceedings.

Page 28: Systematic Literature Review

Developing a Review Protocol

• Study selection criteria: used to determine which studies are included in, or excluded from.

• Study selection procedures: how the selection criteria will be applied e.g. how many assessors will evaluate each prospective primary study, and how disagreements among assessors will be resolved.

Page 29: Systematic Literature Review

Developing a Review Protocol

• Study quality assessment checklists and procedures: quality checklists to assess the individual studies.

• Data extraction strategy: how the information required from each primary study will be obtained.

• Synthesis of the extracted data: defines the synthesis strategy.

• Dissemination strategy

Page 30: Systematic Literature Review

Conducting

Page 31: Systematic Literature Review

Identification of Research

• Generating a search strategy

• Preliminary searches aimed at both identifying existing systematic reviews and assessing the volume of potentially relevant studies.

• Trial searches using various combinations of search terms derived from the research question.

• Checking trial research strings against lists of already known primary studies.

• Consultations with experts in the field.

Page 32: Systematic Literature Review

Identification of Research

• Publication bias

• The problem that positive results are more likely to be published than negative results. The concept of positive or negative results sometimes depends on the viewpoint of the researcher.

• Scanning the grey literature

• Scanning conference proceedings

Page 33: Systematic Literature Review

• Bibliography Management and Document Retrieval

• Reference Manager (http://www.refman.com/)

• Endnote (http://endnote.com/)

• Mendeley (http://www.mendeley.com/)

• Papers (http://papersapp.com/mac/)

• Documenting the Search (The process of performing a systematic literature review must be transparent and replicable (as far as possible)

Identification of Research

Page 34: Systematic Literature Review

Identification of Research

x Contacting experts and researchers working in the area and asking them if they know of any unpublished results.

In addition, statistical analysis techniques can be used to identify the potential significance of publication bias (see Section 6.5.7).

6.1.3 Bibliography Management and Document Retrieval Bibliographic packages such as Reference Manager or Endnote may be useful for managing the large number of references that can be obtained from a thorough literature search. Once reference lists have been finalised the full articles of potentially useful studies will need to be obtained. A logging system is needed to make sure all relevant studies are obtained.

6.1.4 Documenting the Search The process of performing a systematic literature review must be transparent and replicable (as far as possible): x The review must be documented in sufficient detail for readers to be able to

assess the thoroughness of the search. x The search should be documented as it occurs and changes noted and justified. x The unfiltered search results should be saved and retained for possible reanalysis. Procedures for documenting the search process are given in Table 2.

Table 2 Search process documentation Data Source Documentation Digital Library Name of database

Search strategy for the database Date of search Years covered by search

Journal Hand Searches Name of journal Years searched Any issues not searched

Conference proceedings Title of proceedings Name of conference (if different) Title translation (if necessary) Journal name (if published as part of a journal)

Efforts to identify unpublished studies

Research groups and researchers contacted (Names and contact details) Research web sites searched (Date and URL)

Other sources Date Searched/Contacted URL Any specific conditions pertaining to the search

Researchers should specify their rationale for:

x The digital libraries to be searched. x The journal and conference proceedings to be searched. x The use of electronic or manual searches or a combination of both. Although

most text books emphasise the use of electronic search procedures, they are not usually sufficient by themselves, and some researchers strongly advocate the use of manual searches (e.g. Jørgensen, [18]).

16

• ACM Digital Library (http://dl.acm.org/)

• IEEE Xplore (http://ieeexplore.ieee.org/)

• Science Direct (http://www.sciencedirect.com/)

• ISI Web of Science (http://isiknowledge.com/)

• Scopus (http://www.scopus.com/)

• Google Scholar*

Page 35: Systematic Literature Review

Study Selection• Study selection criteria: to identify those primary studies that provide

direct evidence about the research question. Inclusion and exclusion criteria should be based on the research question.

• Inclusion:

• any study that compared predictions of cross-company models with within-company models based on analysis of single company project data.

• Exclusion:

• studies where projects were only collected from a small number of different sources (e.g. 2 or 3 companies),

• studies where models derived from a within-company data set were compared with predictions from a general cost estimation model.

Page 36: Systematic Literature Review

Study Selection• Study selection process:

• Language

• Journal

• Authors

• Setting

• Participants or subjects

• Research Design

• Date of publication

• Reliability of inclusion decisions:

• When two or more researchers assess each paper, agreement between researchers can be measured using the Cohen Kappa statistic.

Page 37: Systematic Literature Review

Identify relevant studies - automated and manual

search

Exclude studies on the basis of Title

Exclude studies on the basis of Abstract

Obtain primary papers and critically appraise studies

n papers

n papers

n papers

n papers

Page 38: Systematic Literature Review

Study Quality Assessment

• To provide still more detailed inclusion/exclusion criteria.

• To investigate whether quality differences provide an explanation for differences in study results.

• As a means of weighting the importance of individual studies when results are being synthesized.

• To guide the interpretation of findings and determine the strength of inferences.

• To guide recommendations for further research.

Page 39: Systematic Literature Review

• Development of Quality Instruments

• Design

• Conduct

• Analysis

• Conclusions

• Example:

1.Is the data analysis process appropriate?

1.1.Was the data investigated to identify outliers and to assess distributional properties before analysis?

1.2.Was the result of the investigation used appropriately to transform the data and select appropriate data points?

Study Quality Assessment

Page 40: Systematic Literature Review

• Less than 10 projects: Poor quality (score=0)

• Between 10 and 20 projects: Fair quality (score=0.33)

• Between 21 and 40 projects: Good quality (score=0.67)

• More than 40 projects: Excellent quality (score=1)

Study Quality Assessment example

Page 41: Systematic Literature Review

Data Extraction

• Design of Data Extraction Forms

• Content of Data Collection Forms

of a set of primary studies and are a prerequisite for meta-analysis (i.e. statistical techniques aimed at integrating the results of the primary studies). Data extraction forms need to be piloted on a sample of primary studies. If several researchers will use the forms, they should all take part in the pilot. The pilot studies are intended to assess both technical issues such as the completeness of the forms and usability issues such as the clarity of user instructions and the ordering of questions. Electronic forms are useful and can facilitate subsequent analysis.

6.4.2 Contents of Data Collection Forms In addition to including all the questions needed to answer the review question and quality evaluation criteria, data collection forms should provide standard information including: x Name of Reviewer x Date of Data extraction x Title, authors, journal, publication details x Space for additional notes Examples Kitchenham et al. [21] used the extraction form shown in Table 7 (note the actual form also included the quality questions).

Table 7 Data Collection form completed for Maxwell et al., 1998 Data item Value Additional notes Data Extractor Data Checker Study Identifier S1 Application domain Space, military and industrial Name of database European Space Agency (ESA) Number of projects in database (including within-company projects)

108

Number of cross-company projects

60

Number of projects in within-company data set

29

Size metric(s): FP (Yes/No) Version used: LOC (Yes/No) Version used: Others (Yes/No) Number:

FP: No LOC: Yes (KLOC) Others: No

Number of companies 37 Number of countries represented

8 European only

Were quality controls applied to data collection?

No

If quality control, please describe

How was accuracy measured?

Measures: R2 (for model construction only) MMRE Pred(25) r (Correlation between estimate and actual)

30

Page 42: Systematic Literature Review

Data Extraction• Data extraction procedures

• Whenever feasible, data extraction should be performed independently by two or more researchers. Data from the researchers must be compared and disagreements resolved either by consensus among researchers or arbitration by an additional independent researcher.

• Multiple publications of the same data

• It is important not to include multiple publications of the same data in a systematic review synthesis because duplicate reports would seriously bias any results. When there are duplicate publications, the most complete should be used.

Page 43: Systematic Literature Review

Data Synthesis• Unpublished data, missing data and data requiring

manipulation

• Collating and summarizing the results of the included primary studies. The data synthesis activities should be specified in the review protocol.

• Descriptive (Narrative) Synthesis

• Extracted information about the studies (i.e. intervention, population, context, sample sizes, outcomes, study quality) should be tabulated in a manner consistent with the review question.

Page 44: Systematic Literature Review

Data Synthesis• Quantitative Synthesis: quantitative data should also

be presented in tabular form including:

• Sample size for each intervention.

• Estimates effect size for each intervention with standard errors for each effect.

• Difference between the mean values for each intervention, and the confidence interval for the difference.

• Units used for measuring the effect.

Page 45: Systematic Literature Review

Data Synthesis• Presentation of Quantitative Results

• Charts x Tables

• Qualitative Synthesis

• Tries to integrate studies comprising natural language results and conclusions, where different researchers may have used terms and concepts with subtly (or grossly) different meanings

• Synthesis of Qualitative and Quantitative Synthesis

• Synthesize the quantitative and qualitative studies separately.

• Then attempt to integrate the qualitative and quantitative results by investigating whether the qualitative results can help explain the quantitative results. For example qualitative studies can suggest reasons why a treatment does or does not work in specific circumstances.

Page 46: Systematic Literature Review

Data Synthesis• Sensitivity Analysis

• High quality primary studies only.

• Primary studies of particular types.

• Primary studies for which data extraction presented no difficulties (i.e. excluding any studies where there was some residual disagreement about the data extracted).

• The experimental method used by the primary studies.

Page 47: Systematic Literature Review

Reporting

Page 48: Systematic Literature Review

Reporting• Specifying the Dissemination Strategy

• Journals

• Conferences

• Formatting the Main Systematic Review Report

• Technical Report

• Section of a thesis

• Journal or conference format

• Evaluating Systematic Review Reports

Page 49: Systematic Literature Review

Problems associated with conducting a SLR

Page 50: Systematic Literature Review

Problems associated with conducting a SLR• The protocol may not have identified all the variations in the

designing and reporting of relevant primary studies.

• Selection of the digital libraries to be searched and whether the search is automated or manual.

• A manual search involves looking at the back issues of a set of journals (either paper or online) and deciding from the title and abstract which individual papers are candidates for inclusion in the review.

• An automated search uses strings, usually based on complex Boolean formulas, to turn up papers in online catalogs.

Page 51: Systematic Literature Review

(software OR application OR product OR Web OR WWW OR Internet OR World-Wide Web OR project OR development) AND (method OR process OR system OR technique OR methodology OR procedure) AND (cross company OR cross organisation OR cross organization OR cross organizational OR cross organisational OR cross- company OR cross-organisation OR cross-organization OR cross-organizational OR cross-organisational OR multi company OR multi organisation OR multi organization OR multi organizational OR multi organisational OR multi- company OR multi-organisation OR multi-organization OR multi-organizational OR multi-organisational OR multiple company OR multiple organisation OR multiple organization OR multiple organizational OR multiple organisational OR multiple-company OR multiple-organisation OR multiple-organization OR multiple-organizational OR multiple- organisational OR within company OR within organisation OR within organization OR within organizational OR within organisational OR within-company OR within-organisation OR within-organization OR within-organizational OR within-organisational OR single company OR single organisation OR single organization OR single organizational OR single organisational OR single-company OR single-organisation OR single-organization OR single-organizational OR single-organisational OR company-specific) AND (model OR modeling OR modelling) AND (effort OR cost OR resource) AND (estimation OR prediction OR assessment)

Page 52: Systematic Literature Review

Problems associated with conducting a SLR• Digital libraries differ in subtle ways in their implementations of

searches:

• Digital libraries have different interfaces and different tolerances for the complex Boolean formulas typically used by SR researchers to turn up relevant papers.

• Digital libraries also have different procedures for searching the body of the paper, or only the title, abstract, and keywords. In addition, of course, indexing systems only search titles, keywords, and abstracts.

• Automated searches from different sources may overlap (i.e., the same paper may be found in different digital libraries), and every search includes a large number of irrelevant papers .

Page 53: Systematic Literature Review

Problems associated with conducting a SLR

• Digital libraries differ in subtle ways in their implementations of searches:

• (TITLE-ABS-KEY(usability) OR TITLE-ABS-KEY("human-computer interaction") OR TITLE-ABS-KEY("computer-human interaction") OR TITLE-ABS-KEY("human factor") OR TITLE-ABS-KEY("user experience") OR TITLE-ABS-KEY("user-centered design")) AND (TITLE-ABS-KEY(agile) OR TITLE-ABS-KEY("agile method") OR TITLE-ABS-KEY("agile development") OR TITLE-ABS-KEY("agile practice") OR TITLE-ABS-KEY("agile project") OR TITLE-ABS-KEY("agile lifecycle") OR TITLE-ABS-KEY(scrum) OR TITLE-ABS-KEY("extreme programming") OR TITLE-ABS-KEY("lean development") OR TITLE-ABS-KEY("feature driven development") OR TITLE-ABS-KEY("dynamic system development method") OR TITLE-ABS-KEY("agile unified process")) AND PUBYEAR AFT 2000 AND (LIMIT-TO(SUBJAREA, "COMP") OR LIMIT-TO(SUBJAREA, "MULT"))

• (usability OR "human-computer interaction" OR "computer-human interaction" OR "human factor" OR "user experience" OR "user-centered design") AND (agile OR "agile method" OR "agile development" OR "agile practice" OR "agile project" OR "agile lifecycle" OR scrum OR "extreme programming" OR "lean development" OR "feature driven development" OR "dynamic system development method" OR "agile unified process")

Page 54: Systematic Literature Review

Problems associated with conducting a SLR• For manual searches:

• Select the journals and conference proceedings you intend to search.

• Justify your selection of sources. However, rather surprisingly, in one case study we found that a targeted manual search was much quicker than a broad automated search.

• In practice, need for a mixed strategy.

• If you do a manual search of some sources (including specialist conference proceedings), this should give you a baseline set of candidate primary studies against which you can validate the efficacy of your automated search strings.

• Alternatively, a domain expert can identify a baseline set of papers.

Page 55: Systematic Literature Review

Other types of reviews

• Systematic Mapping Study: a broad review of primary studies in a specific topic area that aims to identify what evidence is available on the topic.

• Tertiary Study: a review of secondary studies related to the same research question.

Page 56: Systematic Literature Review

Lessons Learned

of applying the systematic literature review process to thesoftware engineering domain for the purposes of identifyingthose aspects of the process that transfer ‘as is’ to softwareengineering, those that need to be adapted to the particularcharacteristics of the domain and those areas whereimprovements to current software engineering infrastruc-ture and practices are needed in order to gain greatest advan-tage from the process.

Section 2 provides an introduction to systematic litera-ture review highlighting the main phases and stages ofthe process. This is followed, in Sections 3–5, respectively,by overviews of three systematic reviews relating to servicebased systems, the technology acceptance model and guide-lines for conducting systematic literature reviews. Lessonslearned from our experiences of undertaking these reviewsare then described and discussed, and finally some conclu-sions are drawn.

2. Systematic literature review

The wider context for this study is that of investigatingthe use of the evidence-based paradigm in software engi-neering. The possibility of applying the evidence-based par-adigm to the software engineering field was raised,discussed and enthusiastically supported at ICSE 2004(Kitchenham et al., 2004). The goal of evidence-based soft-ware engineering (EBSE) is summarised by Kitchenhamet al., as being: ‘‘to provide the means by which currentbest evidence from research can be integrated with practi-cal experience and human values in the decision makingprocess regarding the development and maintenance ofsoftware’’.

Within the medical domain, the steps that need to be fol-lowed to practise evidence-based medicine have been iden-tified and documented (Sackett et al., 2000), and these havesubsequently been re-formulated to address evidence-basedsoftware engineering (Dyba et al., 2005; Kitchenham et al.,2004). The steps are:

1. convert the need for information (about a technique,procedure, etc.) into an answerable question;

2. find the best evidence with which to answer the question;3. critically appraise the evidence for its validity (closeness

to the truth), impact (size of the effect), and applicability(usefulness);

4. integrate the critical appraisal with software engineeringexpertise and with stakeholders’ values andcircumstances;

5. evaluate the effectiveness and efficiency in executingsteps 1–4 and seek ways to improve them.

The first three of these steps essentially constitute a sys-tematic review of the literature, conducted in order to pro-vide a balanced and objective summary that is relevant tomeeting a particular need for information. A systematic(literature) review is ‘‘a means of evaluating and interpret-ing all available research relevant to a particular research

question or topic area or phenomenon of interest’’ (Kitch-enham, 2004). The research papers summarised in thereview are referred to as primary studies, while the reviewitself is a secondary study. The accumulation of evidencethrough secondary studies can be very valuable in offeringnew insights or in identifying where an issue might beclarified by additional primary studies. For example, astudy of software cost overruns showed that the resultsreported in a highly influential study carried out in theearly 1990s (The CHAOS report) were significantly outof step with those reported in other studies (Jørgensenand Moløkken-Østvold, 2006). A critical evaluation ofthe CHAOS report by Jørgensen and Moløkken-Østvoldidentified several methodological problems. Anotherexample, where new insights have emerged, is a systematicreview of statistical power in software engineering experi-ments (Dyba et al., 2006). Here, the results show ‘‘that thestatistical power of software engineering experiments fallssubstantially below accepted norms as well as the levelsfound in related discipline of information systemsresearch’’. The authors go on to make recommendationsabout how empirical software engineering researchersmight address the reported shortcomings.

Performing a systematic review involves several discreteactivities, which can be grouped into three main phases:planning; conducting the review; and reporting the review.Fig. 1 illustrates the overall 10-stage review process.

Systematic literature reviews are primarily concernedwith the problem of aggregating empirical evidence whichmay have been obtained using a variety of techniques,and in (potentially) widely differing contexts—which iscommonly the case for software engineering. While theyare used in information systems research (Webster andWatson, 2002), they are less common in software engineer-ing (however, see (Glass et al., 2002) as an example of a sec-ondary study that samples literature within the softwareengineering domain). Indeed, at the present time, outsideof information systems research, reviews in any form, aswell as review journals are really not part of the computing

9. Write Review Report

10. Validate Report

2. Develop Review Protocol

3. Validate Review Protocol

Phase 1: Plan Review

Phase 2: Conduct Review

Phase 3: Document Review

8. Synthesise Data

4. Identify Relevant Research

5. Select Primary Studies

7. Extract Required Data

6. Assess Study Quality

1. Specify Research Questions

Fig. 1. Systematic literature review process.

572 P. Brereton et al. / The Journal of Systems and Software 80 (2007) 571–583

Page 57: Systematic Literature Review

Lessons Learned

• Stage 1: Specify Research Questions

• L1: Expect to revise your questions during protocol development, as your understanding of the problem increases.

• L2: A pre-review mapping study may help in scoping research questions.

Page 58: Systematic Literature Review

Lessons Learned

• Stage 2: Develop Review Protocol

• L3: All systematic review team members need to take an active part in developing the review protocol.

• L4: Piloting the research protocol is essential. It will find mistakes in your data collection and aggregation procedures. It may also indicate that you need to change the methodology you intend to use to address the research questions.

Page 59: Systematic Literature Review

Lessons Learned

• Stage 3: Validate Review Protocol

• L5: Data extraction is assisted by having data definitions and data extraction guidelines from the protocol recorded in a separate short document.

• L6: There needs to be an agreed validation process separate from the protocol piloting activity. Ideally, external reviewers should undertake this validation process.

Page 60: Systematic Literature Review

Lessons Learned• Stage 4: Identify Relevant Research

• L7: There are alternative search strategies that enable you to achieve different sort of search completion criteria. You must select and justify a search strategy that is appropriate for your research question.

• L8: We need to search many different electronic sources; no single source finds all of the primary studies.

• L9: Current software engineering search engines are not designed to support systematic literature reviews. Unlike medical researchers, software engineering researchers need to perform resource-dependent searches.

Page 61: Systematic Literature Review

Lessons Learned

• Stage 5: Select Primary Studies

• L10: The standard of IT and software engineering abstracts is too poor to rely on when selecting primary studies. You should also review the conclusions.

Page 62: Systematic Literature Review

Lessons Learned

• Stage 6: Assess Study Quality

• L11: All the medical standards emphasise that it is necessary to assess the quality of primary studies. However, it depends on the type of systematic literature review you are undertaking.

• L12: It is important to be sure how the quality assessment will be used in the subsequent data aggregation and analysis.

Page 63: Systematic Literature Review

Lessons Learned

• Stage 7: Extract Required Data

• L13:  Having one reader act as data extractor and one act as data checker may be helpful when there are a large number of papers to review.

• L14:  Review team members must make sure they understand the protocol and the data extraction process.

Page 64: Systematic Literature Review

Lessons Learned• Stage 8: Synthesise Data

• L15:  Software engineering systematic reviews are likely to be qualitative in nature.

• L16:  Even when collecting quantitative information it may not be possible to perform meta-analysis of software engineering studies because the reporting protocols vary so much from study to study.

• L17:  Tabulating the data is a useful means of aggregation but it is necessary to explain how the aggregated data actually answers the research questions.

Page 65: Systematic Literature Review

Lessons Learned• Stage 9: Write Review Report

• L18:  Review teams need to keep a detailed record of decisions made throughout the review process.

• L19:  The software engineering community needs to establish mechanisms for publishing systematic literature reviews which may result in papers that are longer than those traditionally accepted by many software engineering outlets or that have appendices stored in electronic repositories.

Page 66: Systematic Literature Review

Lessons Learned

• Stage 10: Validate Report

Page 67: Systematic Literature Review

References• Kitchenham, B.. (2004). Procedures for Performing Systematic Reviews. Joint Technical Report.

• Biolchini, J.; Mian, P. G.; Natali, A. C. C.; Travassos, G. H. T.. (2005) .Systematic Review in Software Engineering. Technical Report.

• Kitchneham, B.. (2007). Guidelines for performing Systematic Literature Reviews in Software Engineering. EBSE Technical Report.

• Brereton, P.; Kitchenham, B.; Budgen, D.; Turner, M.; Khalil, M.. (2008). Lessons from applying the systematic literature review process within the software engineering domain. The Journal of Systems and Software.

• Dyba , T.; Dingsøyr, T.. (2008). Empirical studies of agile software development: A systematic review. Information and Software Technology.

• Kitchenham, B.; Brereton, P.; Budgen, D.; Turner, M.; Bailey, J.; Linkman, S.. (2009). Systematic literature reviews in software engineering – A systematic literature review. Information and Software Technology.

• Kitchenham, B.; Pretorius, R.; Budgen, D.; Brereton, P.; Turner, M.; Niazi, M.; Linkman, S.. (2010). Systematic literature reviews in software engineering – A tertiary study. Information and Software Technology.

• Kitchenham, B.; Mendes, E.; Travassos, G. H. T..(2007) A Systematic Review of Cross- vs. Within-Company Cost Estimation Studies, IEEE Trans on SE.

Page 68: Systematic Literature Review

Systematic Literature Review

Prof. Dr. Tiago Silva da Silva (ICT/UNIFESP)

[email protected]

Page 69: Systematic Literature Review
Page 70: Systematic Literature Review

User-Centered Design and Agile Methods:

A Systematic ReviewTiago Silva da Silva, Angela Martin,

Frank Maurer, Milene Silveira

To contact me after my presentation, text BDL to

INTRO (46876)

Page 71: Systematic Literature Review

Defining the terms...

Page 72: Systematic Literature Review

Category Keyword

UCD

Usability

Human-Computer Interatcion

Computer-Human Interaction

Human Factor

User Experience

User-Centered Design

User Interface

Agile

Agile

Agile Method

Agile Development

Agile Practice

Agile Project

Agile Lifecycle

Scrum

Extreme Programming

Lean Development

Feature Driven Development

Dynamic System Development

Agile Unified Process

Page 73: Systematic Literature Review

Search Stringusability OR "human-computer interaction" OR "computer-human interaction" OR "human factor" OR "user experience" OR "user-

centered design" OR "user interface"

AND

agile OR "agile method" OR "agile development" OR "agile practice" OR "agile project" OR "agile lifecycle" OR scrum OR "extreme

programming" OR "lean development" OR "feature driven development" OR "dynamic system development" OR "agile unified

process"

Page 74: Systematic Literature Review

Inclusion

Exclusion

Page 75: Systematic Literature Review

Digital Library Amount of papers

First Classification

Inclusion Exclusion

IEEExplore 59 24 35

Science Direct 4 0 4

CiteSeerX 59 0 59

Scopus 154 24 130

Agile 21 21 0

XP 12 12 0

Total 309 81 228

Percentage 100% 26% 74%

Page 76: Systematic Literature Review

Digital LibraryAmount

of papers

Second ClassificationTotal

SelectedRepeated

Not Relevant

Total 81 16 7 58

Percentage 100% 20% 8% 72%

Page 77: Systematic Literature Review

309

Page 78: Systematic Literature Review

30981

Page 79: Systematic Literature Review

3098158

Page 80: Systematic Literature Review
Page 81: Systematic Literature Review
Page 82: Systematic Literature Review

Descriptive Information

Content-related Information

Page 83: Systematic Literature Review

Descriptive Information

Page 84: Systematic Literature Review
Page 85: Systematic Literature Review

Theoretical Experience Report Empirical Experimental

Page 86: Systematic Literature Review

Focus

Page 87: Systematic Literature Review
Page 88: Systematic Literature Review
Page 89: Systematic Literature Review

Approach

Page 90: Systematic Literature Review
Page 91: Systematic Literature Review
Page 92: Systematic Literature Review

Results

Page 93: Systematic Literature Review
Page 94: Systematic Literature Review
Page 95: Systematic Literature Review

Artefacts, Practices, Needs...

Page 96: Systematic Literature Review
Page 97: Systematic Literature Review
Page 98: Systematic Literature Review
Page 99: Systematic Literature Review
Page 100: Systematic Literature Review

Exploring principles of user-centered agile software development: Aliterature review

Manuel Brhel a, Hendrik Meth b, Alexander Maedche a,b,⇑, Karl Werder a

a Chair of Information Systems IV, University of Mannheim, L 15, 1-6, 68131 Mannheim, Germanyb Institute for Enterprise Systems, University of Mannheim, L 15, 1-6, 68131 Mannheim, Germany

a r t i c l e i n f o

Article history:Received 5 September 2013Received in revised form 7 January 2015Accepted 7 January 2015Available online 17 January 2015

Keywords:Agile software developmentUser-centered designSystematic literature review

a b s t r a c t

Context: In the last decade, software development has been characterized by two major approaches: agilesoftware development, which aims to achieve increased velocity and flexibility during the developmentprocess, and user-centered design, which places the goals and needs of the system’s end-users at the cen-ter of software development in order to deliver software with appropriate usability. Hybrid developmentmodels, referred to as user-centered agile software development (UCASD) in this article, propose to com-bine the merits of both approaches in order to design software that is both useful and usable.Objective: This paper aims to capture the current state of the art in UCASD approaches and to derive gen-eric principles from these approaches. More specifically, we investigate the following research question:Which principles constitute a user-centered agile software development approach?Method: We conduct a systematic review of the literature on UCASD. Identified works are analyzed usinga coding scheme that differentiates four levels of UCASD: the process, practices, people/social and tech-nology dimensions. Through subsequent synthesis, we derive generic principles of UCASD.Results: We identified and analyzed 83 relevant publications. The analysis resulted in a comprehensivecoding system and five principles for UCASD: (1) separate product discovery and product creation, (2)iterative and incremental design and development, (3) parallel interwoven creation tracks, (4) continuousstakeholder involvement, and (5) artifact-mediated communication.Conclusion: Our paper contributes to the software development body of knowledge by (1) providing abroad overview of existing works in the area of UCASD, (2) deriving an analysis framework (in form a cod-ing system) for works in this area, going beyond former classifications, and (3) identifying generic prin-ciples of UCASD and associating them with specific practices and processes.

! 2015 Published by Elsevier B.V.

Contents

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1642. Foundations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164

2.1. Summary of existing literature reviews. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1652.2. Gap analysis of existing literature reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

3. Research method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1653.1. Data sources and search strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1663.2. Paper selection and pre-assessment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1663.3. Paper analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167

3.3.1. Identification of dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1673.3.2. Identification of codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1673.3.3. Identification of candidate principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168

4. Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1684.1. Number and distribution of publications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168

http://dx.doi.org/10.1016/j.infsof.2015.01.0040950-5849/! 2015 Published by Elsevier B.V.

⇑ Corresponding author at: Institute for Enterprise Systems, University of Mannheim, L 15, 1-6, 68131 Mannheim, Germany. Tel.: +46 621 181 3606; fax: +49 621 181 3627.E-mail addresses: [email protected] (M. Brhel), [email protected] (H. Meth), [email protected] (A. Maedche), [email protected].

de (K. Werder).

Information and Software Technology 61 (2015) 163–181

Contents lists available at ScienceDirect

Information and Software Technology

journal homepage: www.elsevier .com/locate / infsof

Page 101: Systematic Literature Review

(4) A data extraction and synthesis method, representing the sys-tematic approach to consolidate and integrate existingknowledge.

The review protocol was developed in cooperation with the firstand second authors, and validated by the third author prior to con-ducting the review. In the following, we describe the implementa-tion of the review protocol.

3.1. Data sources and search strategy

Contributions to the research topic at hand may be found in dif-ferent domains (i.e. information systems [IS], and computer sci-ence [CS]), and in different sub-domains within these domains(e.g. HCI). For each of these domains, the most relevant databaseswere selected for the search. The IS literature has focused on threeof these databases, namely ProQuest, Elsevier ScienceDirect, andEBSCO Host. Additionally, we queried three databases, namely IEEEXplore, ACM Digital Library, and Springer Link, which mainly focusCS topics. The search string was composed to cover both fields ofinterest, i.e. UCD and ASD. Key words relevant to the first field werederived from an exploratory literature review, while key words forthe second field were adopted from Dybå and Dingsøyr [31], whoperformed a systematic literature review of empirical studies onASD. Additionally, the terms ‘‘software development’’ and ‘‘sys-tems development’’ were added to reflect the activity of interestand restrict the search space. Table 1 lists the key words and theircorresponding logical operators.

3.2. Paper selection and pre-assessment

The selection process involved four stages. In each stage, thesample size was reduced based on the inclusion criteria applicable

in this stage as depicted in Fig. 1. In the first stage, relevant publi-cations were retrieved by querying the databases mentioned abovewith the search string depicted in Table 1. All database querieswere made in the first two weeks of October 2012. This yielded atotal of 1152 initial results, and 1034 publications after theremoval of duplicates, which were included in the second stage.

In the second stage, publications were excluded based on theirtitles and abstract. Criteria for inclusion in the following stage werethat the article focuses on the integration of UCD and ASD. Specialformats, such as editorials, prefaces, article summaries, interviews,news, correspondence, discussions, and readers’ letters, wereexcluded. Moreover, we excluded articles summarizing tutorials,workshops, panels, or poster sessions. As Brereton et al. [30] note,titles and abstracts of IS, CS, and SE publications are often of lowquality. Thus, articles were included in the next stage only if theevaluation of titles and abstract did not lead to a conclusive deci-sion as Dybå and Dingsøyr [31] demonstrate. After the analysisof titles and abstracts, 148 publications were included in the fol-lowing stage.

The third stage involved a more detailed analysis of the articles’content. Thus, the full text of each publication in this stage wasretrieved for local storage. Each article in the sample’s introduc-tion, conclusion, and, in case of doubt, content was skimmed toassess its relevance. In this stage, stricter criteria were applied todetermine the relevance of articles, as studies or practitionerreports should give recommendations or requirements for soft-ware development integrating UCD and ASD. These recommenda-tions or requirements may be given explicitly, for example: ‘‘UCDpractitioners must be given ample time in order to discover thebasic needs of their users before any code gets released into theshared coding environment’’ [35]. On the other hand, they mightemerge as a result of the study, for example, if a positive impactof applying a certain principle or practice has been observed. Based

Table 1Composition of the search string.

AND OR Ergonomics, human–computer interaction, computer–human interaction, interaction design, usability, user experience, user-centered design, ui design,interface design

OR Agile, scrum, extreme programming, lean, crystal clear, feature driven development, dynamic software developmentOR Software development, systems development

Fig. 1. Selection process (based on [31]).

166 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181

(4) A data extraction and synthesis method, representing the sys-tematic approach to consolidate and integrate existingknowledge.

The review protocol was developed in cooperation with the firstand second authors, and validated by the third author prior to con-ducting the review. In the following, we describe the implementa-tion of the review protocol.

3.1. Data sources and search strategy

Contributions to the research topic at hand may be found in dif-ferent domains (i.e. information systems [IS], and computer sci-ence [CS]), and in different sub-domains within these domains(e.g. HCI). For each of these domains, the most relevant databaseswere selected for the search. The IS literature has focused on threeof these databases, namely ProQuest, Elsevier ScienceDirect, andEBSCO Host. Additionally, we queried three databases, namely IEEEXplore, ACM Digital Library, and Springer Link, which mainly focusCS topics. The search string was composed to cover both fields ofinterest, i.e. UCD and ASD. Key words relevant to the first field werederived from an exploratory literature review, while key words forthe second field were adopted from Dybå and Dingsøyr [31], whoperformed a systematic literature review of empirical studies onASD. Additionally, the terms ‘‘software development’’ and ‘‘sys-tems development’’ were added to reflect the activity of interestand restrict the search space. Table 1 lists the key words and theircorresponding logical operators.

3.2. Paper selection and pre-assessment

The selection process involved four stages. In each stage, thesample size was reduced based on the inclusion criteria applicable

in this stage as depicted in Fig. 1. In the first stage, relevant publi-cations were retrieved by querying the databases mentioned abovewith the search string depicted in Table 1. All database querieswere made in the first two weeks of October 2012. This yielded atotal of 1152 initial results, and 1034 publications after theremoval of duplicates, which were included in the second stage.

In the second stage, publications were excluded based on theirtitles and abstract. Criteria for inclusion in the following stage werethat the article focuses on the integration of UCD and ASD. Specialformats, such as editorials, prefaces, article summaries, interviews,news, correspondence, discussions, and readers’ letters, wereexcluded. Moreover, we excluded articles summarizing tutorials,workshops, panels, or poster sessions. As Brereton et al. [30] note,titles and abstracts of IS, CS, and SE publications are often of lowquality. Thus, articles were included in the next stage only if theevaluation of titles and abstract did not lead to a conclusive deci-sion as Dybå and Dingsøyr [31] demonstrate. After the analysisof titles and abstracts, 148 publications were included in the fol-lowing stage.

The third stage involved a more detailed analysis of the articles’content. Thus, the full text of each publication in this stage wasretrieved for local storage. Each article in the sample’s introduc-tion, conclusion, and, in case of doubt, content was skimmed toassess its relevance. In this stage, stricter criteria were applied todetermine the relevance of articles, as studies or practitionerreports should give recommendations or requirements for soft-ware development integrating UCD and ASD. These recommenda-tions or requirements may be given explicitly, for example: ‘‘UCDpractitioners must be given ample time in order to discover thebasic needs of their users before any code gets released into theshared coding environment’’ [35]. On the other hand, they mightemerge as a result of the study, for example, if a positive impactof applying a certain principle or practice has been observed. Based

Table 1Composition of the search string.

AND OR Ergonomics, human–computer interaction, computer–human interaction, interaction design, usability, user experience, user-centered design, ui design,interface design

OR Agile, scrum, extreme programming, lean, crystal clear, feature driven development, dynamic software developmentOR Software development, systems development

Fig. 1. Selection process (based on [31]).

166 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181

Page 102: Systematic Literature Review

on these criteria, 74 articles were considered relevant for inclusionin the next stage. Following the strategy suggested by Webster andWatson [36], we conducted a forward and backward search basedon these key publications. The retrieved new and potentially rele-vant articles were fed to the second stage for further processing,resulting in an iterative extension of the sample. This resulted inan additional set of nine articles for inclusion in stage four. It hasto be noted that both da Silva et al.’s [20] and Sohaib and Khan’s[23] reviews were included in the stage three sample. However,they were excluded from the detailed analysis in stage four as theydo not present the results of primary research. In total, 83 paperswere selected for the final sample.1

In the fourth stage, the final sample passed a first categorizationand a quality assessment. During the categorization, each paperwas assigned to one of the four research foci, which will be intro-duced in the next section. A quality assessment of each paper wassubsequently conducted to obtain additional information support-ing the interpretation of the paper’s recommendations. This assess-ment was based on four questions, which are given in Table 2. Eachof the four criteria was graded on a dichotomous scale as ‘‘yes’’ or‘‘no,’’ while question 3b was answered by means of the respectivemethod, which was either explicitly stated in the paper or deter-mined by the researcher based on the available information.

3.3. Paper analysis

3.3.1. Identification of dimensionsTo analyze prior research, a comprehensive, hierarchical coding

system was established. UCASD is essentially about the integrationof UCD and ASD. Therefore, to identify the basic dimensions of thecoding system, it is helpful to investigate on which levels integra-tion may be accomplished. Barksdale and McCrickard [22] presentan approach, in which the existing literature is classified into fivetypes of integration for UCASD: Process integration, practices inte-gration, people integration, social integration, and technology inte-gration. Process integration is understood as the merging andsynchronizing of independent UCD and ASD processes, providinga unified process incorporating both perspectives. Practices integra-tion represents the incorporation of UCD practices into ASD andvice versa. People integration is understood as changes in the teamcomposition to bring experts of the two different disciplinestogether (e.g. adding a designer to a team of developers). Socialintegration reflects social interaction and the joint creation ofknowledge. Finally, technology integration entails the use of techno-logical means to support and coordinate activities.

The different levels of integration presented by Barksdale andMcCrickard [22] allow for a comprehensive and differentiated clas-sification of existing works on UCASD. In our literature analysis, wetherefore used these integration levels as a starting point to deter-mine the basic dimensions of our coding system. In contrast to

Barksdale and McCrickard [22], we did not differentiate betweenpeople integration and social integration, as we consider the twoaspects almost inseparable.

3.3.2. Identification of codesWe used the qualitative data analysis software MAXQDA2 to

code the papers. As a universal tool for qualitative data analysis,MAXQDA is usually employed to analyze textual data, such as inter-view transcripts, and supports a variety of qualitative research meth-ods. Various authors (e.g. [31,37,38]) have reported positiveexperiences concerning the use of similar software (e.g. NVivo3) tosynthesize data from a corpus of texts emerging from a systematicliterature review. Motivated by these experiences, we adopted thisapproach to process a large amount of textual data rigorously andtransparently. This seems especially useful as evidence found inextant literature, which may be used to derive principles, can beretrieved easily in later stages of the research process. Using MAX-QDA, codes can be assigned to text segments or images in docu-ments. Besides descriptive statistics on the occurrence of codes indocuments, MAXQDA provides a convenient way to extract codedtext passages, allowing for easy data synthesis. Moreover, MAXQDAallowed us to conduct a quality analysis of the included papers. Tobe able to retrieve papers within a certain category in a convenientmanner, the categorization was done by assigning a code reflectingthe appropriate category to the title and abstract of each paper. Fur-thermore, if a criterion given in Table 2 was assessed to be fulfilled,the text segment providing the information was coded accordingly.The analysis of the document corpus was aimed at identifyingaspects that have to be taken into account when integrating UCDand ASD, with the overarching aim of deriving principles. Duringthe analysis of the document corpus, observations made concerningdifferent aspects of integration were assigned a meaningful code,with the aim of consolidating the perspective different authors takeon the same issue later on.

The dimensions introduced above formed the basis of a first setof high-level codes. For simplification reasons, we use the terms‘‘process,’’ ‘‘practices,’’ ‘‘people/social,’’ and ‘‘technology’’ as high-level codes in the following. Based on our analysis and comparisonof the existing studies, as presented in Section 2, we assigned ini-tial sub-codes to each high-level code (see Table 3). Each of thepapers in the final sample was carefully read, analyzed, and codedaccording to this system. While the initial set of codes proved to beexhaustive, we added additional lower level codes and hierarchylevels during the analysis of the literature to document the insightsreported in these studies. This process becomes omnipresent whencomparing the two coding systems. The initial coding systemreflects 21 codes with two hierarchy levels, whereas the final cod-ing system contains 73 codes in four hierarchy levels. Various rea-sons, such as the addition of specific practices as codes, drove theextension of the coding system. During the coding process, textpassages were related to one of the codes. While the group assign-ment according to the main research focus was mutually exclusive,this does not mean that only aspects concerning this stream ofresearch were discussed in the paper. For example, when theauthors of a paper with a clear process focus recommend the useof personas, the corresponding text passage was coded as ‘‘perso-nas,’’ which was applied to every document in the corpus discuss-ing personas independently of its research focus. Moreover,throughout the coding process, potential new codes were chal-lenged against existing codes and vice versa. Initially, this led tomany changes in the coding system. However, the codes becamemore stable as they developed organically.

Table 2Criteria for quality assessment.

1. Is there a clear statement of the research goals (e.g. in an explicitlyverbalized research question)?

2. Is there an adequate description of the context in which the research wascarried out?

3. (Only applicable to empirical research papers):a. Is the research method explicitly stated?b. Which research method was chosen?

4. Are there explicit recommendations, which could be aggregated asprinciples?

1 The full list of papers and their corresponding classifications can be accessed athttps://madata.bib.uni-mannheim.de/80/.

2 http://www.maxqda.de/.3 http://www.qsrinternational.com/products_nvivo.aspx.

M. Brhel et al. / Information and Software Technology 61 (2015) 163–181 167

3.3.3. Identification of candidate principlesIn order to identify candidate principles, each code was investi-

gated concerning two characteristics. First, the frequency of occur-rence was assessed to gain an impression of the relativeimportance of each aspect. As an example, the code ‘‘prototyping’’was assigned in 40 (48.2%) articles in the literature review, indicat-ing that the use of prototypes was discussed in this document. Incontrast, the idea of ‘‘remote usability testing’’ was only found inone (1.2%) article, leading to the conclusion that, while the formeraspect has a high significance for the scientific community, the lat-ter is a marginal topic. For this assessment, we also considered theinsights reported in existing literature reviews (cf. Section 2.1).

Second, we collected guidance and recommendations for the con-ceptualization of an integrated UCASD approach. While 17 (20.5%)of the articles included in the review contain explicit advice onhow to integrate UCD and ASD, implicit recommendations can bederived too if applying a certain practice has been observed as hav-ing a positive impact. Thus, the content of the coded text passageswas analyzed for either explicit or implicit guidance concerningthe integration of UCD and ASD. Finally, to obtain an overview ofthe recommendations in the extant literature, the coded text pas-sages were reviewed once again and organized along emerging

patterns. These patterns represent related ideas or concepts pre-sented in different publications along the four dimensions, leadingto the merger of independent aspects in the coding system. Theidentification of such patterns was based on a qualitative analysisand the expertise of the researchers in this domain. In this step,also the findings of the three assessed reviews were drawn uponto identify themes for potential principles [33].

4. Results

4.1. Number and distribution of publications

When analyzing the result set, it became apparent that, whilethe integration of UCD and ASD was first discussed a decade ago,the topic gained momentum in 2007 and, since then, a constantlyhigh number of relevant articles have been published every year, asFig. 3 illustrates. This reflects that, while the idea of integratingUCD and ASD, has been around for some time, many integrationissues are still unresolved and research is ongoing. In particular,at least 17 relevant articles have been published since da Silvaet al.’s [20] systematic literature was conducted in late 2010, justi-fying an update of their findings.

Moreover, we identified the research type of each paper as pre-sented in Fig. 2. We classified each paper according to one of fourresearch types, which are non-exclusive of each other. The typesidentified are qualitative research (N = 30), (i.e. action research(N = 2), case study (N = 17), ethnography (N = 4), grounded theory(N = 4) and interview (N = 3)), quantitative research (N = 4), theo-retical work (N = 23), and none (N = 27) of the aforementioned(e.g. practice reports). Industrial experience reports refer to workthat present practical experience. This is supported by existing evi-dence that industrial experience strongly influences agile research[20,46].

Table 4 presents the results of categorizing the 83 publicationsincluded in the final sample according to their main research focusbased on the dimension of our coding system. The analysis

Table 3Initial coding system.

Level 1 Process Practices People/social Technology

Level 2 Little design up front Prototyping Close collaboration Data exchangeUser testing⁄ User testing⁄ Cross-functionality IDE integrationOne sprint ahead User stories Knowledge transferCohesive overall design ScenariosParallel tracks PersonasIterative design/developmentIncremental design/development

Fig. 2. Number of publications by research type.

Fig. 3. Number of publications by year.

168 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181

Page 103: Systematic Literature Review

3.3.3. Identification of candidate principlesIn order to identify candidate principles, each code was investi-

gated concerning two characteristics. First, the frequency of occur-rence was assessed to gain an impression of the relativeimportance of each aspect. As an example, the code ‘‘prototyping’’was assigned in 40 (48.2%) articles in the literature review, indicat-ing that the use of prototypes was discussed in this document. Incontrast, the idea of ‘‘remote usability testing’’ was only found inone (1.2%) article, leading to the conclusion that, while the formeraspect has a high significance for the scientific community, the lat-ter is a marginal topic. For this assessment, we also considered theinsights reported in existing literature reviews (cf. Section 2.1).

Second, we collected guidance and recommendations for the con-ceptualization of an integrated UCASD approach. While 17 (20.5%)of the articles included in the review contain explicit advice onhow to integrate UCD and ASD, implicit recommendations can bederived too if applying a certain practice has been observed as hav-ing a positive impact. Thus, the content of the coded text passageswas analyzed for either explicit or implicit guidance concerningthe integration of UCD and ASD. Finally, to obtain an overview ofthe recommendations in the extant literature, the coded text pas-sages were reviewed once again and organized along emerging

patterns. These patterns represent related ideas or concepts pre-sented in different publications along the four dimensions, leadingto the merger of independent aspects in the coding system. Theidentification of such patterns was based on a qualitative analysisand the expertise of the researchers in this domain. In this step,also the findings of the three assessed reviews were drawn uponto identify themes for potential principles [33].

4. Results

4.1. Number and distribution of publications

When analyzing the result set, it became apparent that, whilethe integration of UCD and ASD was first discussed a decade ago,the topic gained momentum in 2007 and, since then, a constantlyhigh number of relevant articles have been published every year, asFig. 3 illustrates. This reflects that, while the idea of integratingUCD and ASD, has been around for some time, many integrationissues are still unresolved and research is ongoing. In particular,at least 17 relevant articles have been published since da Silvaet al.’s [20] systematic literature was conducted in late 2010, justi-fying an update of their findings.

Moreover, we identified the research type of each paper as pre-sented in Fig. 2. We classified each paper according to one of fourresearch types, which are non-exclusive of each other. The typesidentified are qualitative research (N = 30), (i.e. action research(N = 2), case study (N = 17), ethnography (N = 4), grounded theory(N = 4) and interview (N = 3)), quantitative research (N = 4), theo-retical work (N = 23), and none (N = 27) of the aforementioned(e.g. practice reports). Industrial experience reports refer to workthat present practical experience. This is supported by existing evi-dence that industrial experience strongly influences agile research[20,46].

Table 4 presents the results of categorizing the 83 publicationsincluded in the final sample according to their main research focusbased on the dimension of our coding system. The analysis

Table 3Initial coding system.

Level 1 Process Practices People/social Technology

Level 2 Little design up front Prototyping Close collaboration Data exchangeUser testing⁄ User testing⁄ Cross-functionality IDE integrationOne sprint ahead User stories Knowledge transferCohesive overall design ScenariosParallel tracks PersonasIterative design/developmentIncremental design/development

Fig. 2. Number of publications by research type.

Fig. 3. Number of publications by year.

168 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181

3.3.3. Identification of candidate principlesIn order to identify candidate principles, each code was investi-

gated concerning two characteristics. First, the frequency of occur-rence was assessed to gain an impression of the relativeimportance of each aspect. As an example, the code ‘‘prototyping’’was assigned in 40 (48.2%) articles in the literature review, indicat-ing that the use of prototypes was discussed in this document. Incontrast, the idea of ‘‘remote usability testing’’ was only found inone (1.2%) article, leading to the conclusion that, while the formeraspect has a high significance for the scientific community, the lat-ter is a marginal topic. For this assessment, we also considered theinsights reported in existing literature reviews (cf. Section 2.1).

Second, we collected guidance and recommendations for the con-ceptualization of an integrated UCASD approach. While 17 (20.5%)of the articles included in the review contain explicit advice onhow to integrate UCD and ASD, implicit recommendations can bederived too if applying a certain practice has been observed as hav-ing a positive impact. Thus, the content of the coded text passageswas analyzed for either explicit or implicit guidance concerningthe integration of UCD and ASD. Finally, to obtain an overview ofthe recommendations in the extant literature, the coded text pas-sages were reviewed once again and organized along emerging

patterns. These patterns represent related ideas or concepts pre-sented in different publications along the four dimensions, leadingto the merger of independent aspects in the coding system. Theidentification of such patterns was based on a qualitative analysisand the expertise of the researchers in this domain. In this step,also the findings of the three assessed reviews were drawn uponto identify themes for potential principles [33].

4. Results

4.1. Number and distribution of publications

When analyzing the result set, it became apparent that, whilethe integration of UCD and ASD was first discussed a decade ago,the topic gained momentum in 2007 and, since then, a constantlyhigh number of relevant articles have been published every year, asFig. 3 illustrates. This reflects that, while the idea of integratingUCD and ASD, has been around for some time, many integrationissues are still unresolved and research is ongoing. In particular,at least 17 relevant articles have been published since da Silvaet al.’s [20] systematic literature was conducted in late 2010, justi-fying an update of their findings.

Moreover, we identified the research type of each paper as pre-sented in Fig. 2. We classified each paper according to one of fourresearch types, which are non-exclusive of each other. The typesidentified are qualitative research (N = 30), (i.e. action research(N = 2), case study (N = 17), ethnography (N = 4), grounded theory(N = 4) and interview (N = 3)), quantitative research (N = 4), theo-retical work (N = 23), and none (N = 27) of the aforementioned(e.g. practice reports). Industrial experience reports refer to workthat present practical experience. This is supported by existing evi-dence that industrial experience strongly influences agile research[20,46].

Table 4 presents the results of categorizing the 83 publicationsincluded in the final sample according to their main research focusbased on the dimension of our coding system. The analysis

Table 3Initial coding system.

Level 1 Process Practices People/social Technology

Level 2 Little design up front Prototyping Close collaboration Data exchangeUser testing⁄ User testing⁄ Cross-functionality IDE integrationOne sprint ahead User stories Knowledge transferCohesive overall design ScenariosParallel tracks PersonasIterative design/developmentIncremental design/development

Fig. 2. Number of publications by research type.

Fig. 3. Number of publications by year.

168 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181

Page 104: Systematic Literature Review

revealed that both process and practice integration aspects arestrongly represented in the final sample. While the former are dis-cussed in 24 (28.9%) publications, the latter form the biggest groupwith 38 (45.8%) of the included articles. However, the high numberof publications discussing process integration aspects does notmean that a variety of mature integrated approaches exists. Amongthe 24 publications that focus on process integration, only 15 pres-ent novel approaches, while the remaining articles discuss onlyapproaches that were published earlier by the same authors. Ingeneral, it is difficult to distinguishing between a focus on processor practice integration aspects, as authors frequently describe theirwork as integrated approaches, but only discuss the integration ofUCD into ASD while neglecting the developer’s perspective, whichdid not meet the definition of the process focus taken here. Peopleand social integration aspects were discussed in 14 (16.9%) papers,while technological integration received the least attention withseven (8.4%) of the included publications looking into it.

4.2. Synthesized view

A key objective of the literature review was the synthesis of evi-dence found in extant literature in order to derive principles byapplying a coding system to each article’s content. Fig. 4 depictsthe final version of the coding system, which evolved for the ‘‘pro-cess,’’ ‘‘people/social,’’ ‘‘technology,’’ and ‘‘practices’’ dimensions.Some codes were omitted to improve readability (see completecoding system in Appendix A).

The coding system shows that, while some codes explicitlyemerged from the integration of ASD and UCD approaches (e.g.‘‘synchronization’’ and ‘‘collaboration’’), other codes representaspects of the individual, original approaches (e.g. ‘‘iterativedesign’’ or ‘‘incremental design’’). This result illustrates that inte-grated approaches strive to keep the best aspects of the originalapproaches on the one hand, while trying to enable a smoothmerge of the two worlds on the other hand. Interestingly for theprocess dimension, many more codes have been identified forthe other two dimensions. Moreover, the codes in the processdimension are mainly specific to software development, whilethe majority of the codes in the other dimensions are rather gen-eric (e.g. codes like ‘‘collaboration’’ or ‘‘data exchange’’). This mightbe due to the overall number of papers assigned to the ‘‘process’’dimension being larger than in the other dimensions. Apart fromthat, it might also be an indicator of the depth and maturity of dis-course in each of the dimensions. While aspects of process integra-tion have been intensively discussed and refined, the discussion in

the people/social and technology dimension seems to be ratherlimited in UCASD research.

Owing to the large variety of practices discussed in the review,we sub-categorized the practices dimension based on the processphases. While strict process phases are uncommon in ASD, UCDproceeds along pre-defined (but not necessarily sequential) phases.Our classification follows the four phases defined in the ISO stan-dard for human-centered design: research, specification, design,and evaluation [117]. The four aforementioned phases are consid-ered generic enough for identifying sub-codes within the practicesdimension, One example is the inclusion of user research, which isabsent in other development methods. Abstracting these phasesfrom the general activities to be done (investigate the context,specify requirements, design the software, and, finally, test its con-gruence with the requirements), this classification also allows formapping typical agile practices (such as user stories). Fig. 4 dis-plays all codes for the practices dimension and their assignmentto sub-dimensions.

The results confirm that, especially in UCD, there is a plethora ofdifferent methods as others (cf. [118]) have shown. Recommenda-tions for individual practices differ across the studies, resulting in aheterogeneous picture of practices with an overlapping, if not iden-tical, scope. For example, in the specification phase, scenarios andUI stories, as well as mock-ups and wireframes could be used alter-natively. As suggested earlier, it is therefore helpful to abstractphases from specific practices.

The final coding system contained in Fig. 4 provides a compre-hensive overview of aspects relevant to UCASD. Consequently,these codes, respectively the underlying text passages, were thekey takeaway from the literature review, whose overarching aimis to derive principles to guide the integrated UCASD approach.Throughout the data gathering process, we identified a lack of con-tent for the technology dimensions. While we found aspects con-cerning technological means to support user-centered agileprocesses (N = 10), the text passages were of a general natureand did not provide guidance on technology support. Therefore,no principles concerning technology integration were derived. Adifferent challenge emerged for the people/social dimension.Although we found more sources, the recommendations were con-tradictory. While ASD suggests the collective accountability of theteam [119], UCD methodologies usually suggest individual respon-sibility, for example, allocating the responsibility of developing ausable product to the UCD expert [122–124]. Furthermore, theteam setup led to contradictory evidence. On the one hand, twodedicated teams, which handle design and development activities

Table 4Overview of publications in the final sample.

Dimension Included publications Number ofpublications

Process Benigni et al. [39], Budwig et al. [40], Felker et al. [41], Ferreira et al. [14], Ferreira et al. [42], Fox et al. [21], Holzinger et al. [43], Hussainet al. [44], Hussain et al. [45], Kuusinen et al. [46], Lee & McCrickard [47], Lee et al. [48], Lee et al. [49], Losada et al. [50], Losada et al.[51], Losada et al. [52], Memmel et al. [53], Memmel et al. [54], Miller [55], Najafi & Toyoshiba [56], Paelke & Nebe [57], Paelke & Sester[58], Wolkerstorfer et al. [59], Zhang et al. [60]

24 (28.9%)

Practices Adikari et al. [61], Beyer et al. [62], Broschinsky & Baker [63], Carvalho [64], Chamberlain et al. [35], Cho [65], Constantine [13],Constantine & Lockwood [66], da Silva et al. [67], Detweiler [68], Düchting et al. [69], Evnin & Pries [70], Ferré et al. [71], Fisher &Bankston [72], Haikara [73], Hansson et al. [74], Hellmann et al. [75], Hellmann et al. [76], Hennings [77], Hodgetts [78], Humayounet al. [79], Hussain et al. [80], Illmensee & Muff [81], Isomursu et al. [82], Kane [83], Lárusdóttir [84], Lárusdóttir et al. [85], Lee et al.[86], Medina-Medina et al. [87], Memmel et al. [88], Meszaros & Aston [89], Obendorf & Finck [90], Obendorf et al. [91], Patton [92],Patton [93], Petrovic & Siegmann [94], Rafla et al. [95], Rittenbruch et al. [96]

38 (45.8%)

People &Social

Barksdale & McCrickard [22], Barksdale & McCrickard [97], Barksdale et al. [98], Brown et al. [99], Brown et al. [100], Ferreira et al.[101], Ferreira et al. [102], Kollmann et al. [103], Leszek & Courage [104], Lievesley & Yee [105], Singh [106], Ungar [107], Ungar & White[108], Williams & Ferguson [109]

14 (16.9%)

Technology Feiner & Andrews [110], Gonçalves & Santos [111], Hosseini-Khayat et al. [112], Humayoun et al. [113], Lee [114], Peixoto [115], Peixoto[116]

7 (8.4%)

Total 83 (100.0%)

M. Brhel et al. / Information and Software Technology 61 (2015) 163–181 169

Page 105: Systematic Literature Review

respectively, can be formed. On the other hand, a cross-functionalteam, handling both tasks, might be established [101,102]. There-fore, we derived no principles for the people/social dimension.The paper’s conclusion discusses how future work might derivecorresponding principles for the technology and people/socialdimensions.

5. Principles of UCASD

In the following, we discuss the identified principles for theintegration of UCD and ASD organized along the two centraldimensions processes and practices.

5.1. Process principles

First, a process perspective is necessary to decide how activitiesinvolved in the software development process are organized on afundamental level. Various aspects focusing on the process per-spective were frequently mentioned in the identified publicationsand appeared to have a high level of maturity as a consensuswas identified for most of them. In Fig. 5, the codes assigned tothe process perspective are listed along with their occurrence inthe identified publications. The items have been consolidated intothree key principles forming a UCASD approach.

5.1.1. Separate product discovery and product creationFocus Area. The valuation of up-front analysis and design was

recognized as one of the main tension points between UCD andASD early on. While the former promotes the extensive up-front

analysis of user requirements and the design of the users’ interac-tion with the system, the latter focuses on delivering a workingcode at the expense of an exhaustive planning and design phase.Moreover, there is no common definition of what constitutes suffi-cient preparation, and the effort put into initial analysis and designactivities ranges from days (e.g. [13]) to weeks (e.g. [40]).

Evidence. The results of our literature review confirm theimportance of an up-front analysis and design: 42 (50.6%) of theincluded publications discuss the necessity of up-front designefforts. Seven of these present empirical evidence. A total of 39articles argue that, while an extensive up-front design is againstagile principles, little design up front (LDUF) is needed for successfulUCD efforts in an agile environment. As an example, Miller [55] andSy [24] suggest that interaction designers use an up-front cycle forplanning and collecting customer input. Fox et al. [21] are morespecific with regard to methods and reserve an up-front phasefor contextual inquiry and iterative low-fidelity prototyping,resulting in a preliminary design and a list of requirements thatare handed over for development. This is in line with the resultsof da Silva et al. [20], who find that 31 of the 58 papers theyreviewed recommend a strictly limited up-front design effort.

In particular, 10 (12.0%) of the included publications implementthe LDUF concept by reserving a cycle zero, or sprint zero, for anal-ysis and design activities before actual agile development itera-tions start. The idea of time-boxing initial analysis and design forthe duration of a development cycle was first suggested in Miller’s[55] integrated process model and has since been adopted andmodified by various authors, such as Sy [24], Fox et al. [21], andda Silva et al. [20].

Fig. 4. Final coding system for the process, people/social, technology, and practices dimensions.

170 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181

Page 106: Systematic Literature Review

There is evidence that a limited up-front design effort isparticularly necessary to provide a consistent and cohesive userinterface and navigation structure as it supports designers in find-ing a suitable design concept from the very beginning [42,61]. Pat-ton [93] states that user needs and goals have to be clearly definedbefore conducting design and development efforts in order toensure the usability and usefulness of the software in ASD. Variousauthors have confirmed this view, presenting user research as anessential aspect of up-front design efforts in agile environments[20,21,24,35,49,55]. In total, 22 (26.5%) of the papers in the reviewconfirm a relationship between LDUF and a cohesive overall designof the final product, which is challenging to achieve in an agileenvironment. Meszaros and Aston [89] assert that evolutionarydesign is not suitable for user interfaces when insufficient concep-tual guidance is given. Moreover, Constantine [13] argues that laterevisions of the user interface are unacceptable as such revisionswould be highly disruptive for users already familiar with theapplication.

While there is no broad agreement in the literature on theextent and outcome of LDUF activities, it is evident that what iscommonly called up-front ‘‘design’’ also has the character of a con-stituting analysis involving extensive stakeholder interactions.Other than the actual UI design itself, user research does not yetinvolve any commitment, thus maintaining flexibility. Conse-quently, up-front design activities should first and foremost beunderstood as a preliminary exploration phase, harnessing thepotential of UCD to generate new ideas. Twenty-two (26.5%) ofthe publications included in the review address this issue, andsee UCD as a means to establish and communicate a product visionin order to ‘‘improve the line of sight between strategy and team-level execution’’ [72]. While UCD provides suitable concepts forapproaching ill-defined problems and delivering a product with ahigh degree of innovation, exploratory activities are insufficientlyaddressed in ASD [125]. Kettunen [119] points out that, while agilemethodologies are useful means to iteratively select and refineproduct features, large-scale product innovations are beyond theirscope [119]. They answer the question of how to build a productcorrectly, i.e. delivering a valuable product to the market as soonas possible, while maintaining flexibility along the way. However,product scoping or ideation in general, which would address thequestion of building the right product, is not part of agile method-ologies. This shortfall can be countervailed by drawing on UCD, inwhich collecting stakeholder needs, expectations, and ideas arecore functions. We consequently conclude that the usefulnessfocus of SE has to be complemented by usability concerns not onlyduring product development, but also during product planning. Inorder to deliver a consistent and cohesive design throughout soft-ware product development, such concerns should be included inthe product vision [40,42,46,72,89,103,105,106].

Suggestion. A shift from an up-front design to up-front analysisconcept is identified. In order to deliver a product that is valuableto the customer and end-user in that it is both usable and useful,product discovery and product creation should be separated inUCASD. Overall, as listed in Table 5, four codes from our coding sys-tem support our suggestion to include an additional principle inthis specific context.

Besides making room for sufficient up-front design activities asa prerequisite for delivering a usable product with a consistentuser interaction concept, it allows for mitigating agile shortcom-ings with regard to product discovery, fostering the delivery of auseful product. Thus, we formulate the first principle as:

Principle 1 (Separate product discovery and product crea-tion). User-centered agile software development should be basedon separated product discovery and product creation phases.

While the literature clearly evidences the importance of prod-uct discovery in general, there is currently a knowledge gap withregard to the extent of product discovery activities. This is also vis-ible within the ten listed items identified in the publications, inwhich a pair of antipodes can be found, namely the contrastbetween little or extensive up-front design efforts. Thus, furtherempirical research is required to investigate the extent of productdiscovery activities in respect of the different contingency factorson the team, product, and organizational levels. For example, itwould be of interest to understand how different software productcharacteristics, such as the maturity, complexity, and degree ofinteraction, influence the extent and outcome of product discoveryactivities.

5.1.2. Iterative and Incremental Design and DevelopmentFocus Area. Both ASD and UCD promote an iterative and incre-

mental approach to software development. This allows for usingfeedback collected in previous iterations to enhance the emergingproduct in future iterations. Thus, intermediate solutions to thedevelopment problem are used as a pathway to finding a completeand desirable solution for the customer or end-user.

Evidence. The main commonalities of ASD and UCD have beenidentified in earlier literature reviews by da Silva et al. [20], and

Fig. 5. Codes and articles related to the process dimension.

Table 5Codes related to the separate product discovery and product creation principle.

Included codes Resulting principle

Little design up front Separate product discovery and product creationCycle zeroCohesive overall designProduct vision/innovation

M. Brhel et al. / Information and Software Technology 61 (2015) 163–181 171

Page 107: Systematic Literature Review

There is evidence that a limited up-front design effort isparticularly necessary to provide a consistent and cohesive userinterface and navigation structure as it supports designers in find-ing a suitable design concept from the very beginning [42,61]. Pat-ton [93] states that user needs and goals have to be clearly definedbefore conducting design and development efforts in order toensure the usability and usefulness of the software in ASD. Variousauthors have confirmed this view, presenting user research as anessential aspect of up-front design efforts in agile environments[20,21,24,35,49,55]. In total, 22 (26.5%) of the papers in the reviewconfirm a relationship between LDUF and a cohesive overall designof the final product, which is challenging to achieve in an agileenvironment. Meszaros and Aston [89] assert that evolutionarydesign is not suitable for user interfaces when insufficient concep-tual guidance is given. Moreover, Constantine [13] argues that laterevisions of the user interface are unacceptable as such revisionswould be highly disruptive for users already familiar with theapplication.

While there is no broad agreement in the literature on theextent and outcome of LDUF activities, it is evident that what iscommonly called up-front ‘‘design’’ also has the character of a con-stituting analysis involving extensive stakeholder interactions.Other than the actual UI design itself, user research does not yetinvolve any commitment, thus maintaining flexibility. Conse-quently, up-front design activities should first and foremost beunderstood as a preliminary exploration phase, harnessing thepotential of UCD to generate new ideas. Twenty-two (26.5%) ofthe publications included in the review address this issue, andsee UCD as a means to establish and communicate a product visionin order to ‘‘improve the line of sight between strategy and team-level execution’’ [72]. While UCD provides suitable concepts forapproaching ill-defined problems and delivering a product with ahigh degree of innovation, exploratory activities are insufficientlyaddressed in ASD [125]. Kettunen [119] points out that, while agilemethodologies are useful means to iteratively select and refineproduct features, large-scale product innovations are beyond theirscope [119]. They answer the question of how to build a productcorrectly, i.e. delivering a valuable product to the market as soonas possible, while maintaining flexibility along the way. However,product scoping or ideation in general, which would address thequestion of building the right product, is not part of agile method-ologies. This shortfall can be countervailed by drawing on UCD, inwhich collecting stakeholder needs, expectations, and ideas arecore functions. We consequently conclude that the usefulnessfocus of SE has to be complemented by usability concerns not onlyduring product development, but also during product planning. Inorder to deliver a consistent and cohesive design throughout soft-ware product development, such concerns should be included inthe product vision [40,42,46,72,89,103,105,106].

Suggestion. A shift from an up-front design to up-front analysisconcept is identified. In order to deliver a product that is valuableto the customer and end-user in that it is both usable and useful,product discovery and product creation should be separated inUCASD. Overall, as listed in Table 5, four codes from our coding sys-tem support our suggestion to include an additional principle inthis specific context.

Besides making room for sufficient up-front design activities asa prerequisite for delivering a usable product with a consistentuser interaction concept, it allows for mitigating agile shortcom-ings with regard to product discovery, fostering the delivery of auseful product. Thus, we formulate the first principle as:

Principle 1 (Separate product discovery and product crea-tion). User-centered agile software development should be basedon separated product discovery and product creation phases.

While the literature clearly evidences the importance of prod-uct discovery in general, there is currently a knowledge gap withregard to the extent of product discovery activities. This is also vis-ible within the ten listed items identified in the publications, inwhich a pair of antipodes can be found, namely the contrastbetween little or extensive up-front design efforts. Thus, furtherempirical research is required to investigate the extent of productdiscovery activities in respect of the different contingency factorson the team, product, and organizational levels. For example, itwould be of interest to understand how different software productcharacteristics, such as the maturity, complexity, and degree ofinteraction, influence the extent and outcome of product discoveryactivities.

5.1.2. Iterative and Incremental Design and DevelopmentFocus Area. Both ASD and UCD promote an iterative and incre-

mental approach to software development. This allows for usingfeedback collected in previous iterations to enhance the emergingproduct in future iterations. Thus, intermediate solutions to thedevelopment problem are used as a pathway to finding a completeand desirable solution for the customer or end-user.

Evidence. The main commonalities of ASD and UCD have beenidentified in earlier literature reviews by da Silva et al. [20], and

Fig. 5. Codes and articles related to the process dimension.

Table 5Codes related to the separate product discovery and product creation principle.

Included codes Resulting principle

Little design up front Separate product discovery and product creationCycle zeroCohesive overall designProduct vision/innovation

M. Brhel et al. / Information and Software Technology 61 (2015) 163–181 171

Sohaib and Khan [23]. Specifically, their reviews confirm the signif-icance of iterative and incremental design and development. Thereis general agreement among UCD researchers that an iterativeapproach is essential for developing a product with high usability[123,126]. According to Mao et al. [118], design iterations are themost commonly used UCD paradigm; however, feedback is alsoan essential aspect in ASD. In ASD, the overall project is typicallycomposed of subsequent self-contained cycles, each of which com-prises analysis, design, programming, and test activities [127].Among the included papers, 33 (39.8%) explicitly discuss designor development iterations. In accordance with UCD approaches,empirical feedback is used to revise designs in the next iteration.The iterative refinement and evaluation is continued until theuser’s needs are met, thus ensuring that an adequate level ofusability is achieved [124,128]. In particular, all publications pre-sented in the process dimension (see Table 3) promote an iterativeprocess setup.

An iterative approach allows for the product under develop-ment to be refined based on feedback collected at an earlier stage.In contrast, an incremental strategy means that system functional-ity is partitioned into thin vertical slices from the user interface tothe database layer. Each slice is functionally coherent and demon-strable. Instead of delivering the required functionality all at once,the scope of the system grows progressively with the addition ofconsecutive slices [129,130]. Solutions to software developmentproblems are often complex and exhibit many degrees of freedom.Thus, eliciting requirements from users often results in whatBoehm [131] calls the IKIWISI symptom: ‘‘I can’t tell you inadvance, but I’ll know it when I see it’’ [131]. Especially for interac-tion features, users might not be able to specify their needs unlessconfronted with a tangible representation of the system, such as aworking product increment that would allow them to clarify andrefine their vision of the software [9,127,131]. Drawing on thisinsight, the aspect of incremental development was mentioned in11 (13.3%) articles, typically with reference to functional tests orusability evaluations.

The associated papers identified in the literature provide noempirical evidence for the outcome of iterative and incrementaldesign and development. This may be because the empirical stud-ies not only consider the user-centered design or agile develop-ment perspective independently from each other, but as acombination of the two [21,35,42,46,49,103,105].

Suggestion. In summary, developing software in an iterativemanner creates short feedback cycles, while an incremental devel-opment in vertical slices allows for evaluating working productincrements. Additionally, adapting the product scope after eachiteration embraces change and is a prerequisite to deliver valuablesoftware (see Table 6).

Beyond the frequent occurrence of the two aspects in the liter-ature, a general acceptance of this strategy can be inferred from

references to iterative and incremental concepts for ASD (e.g.Scrum) or UCD (e.g. ISO 9241-210). Even though the reviewedpapers do not explicitly discuss an iterative and incrementalapproach, none of them challenge this paradigm. Thus, the secondprinciple is articulated as follows:

Principle 2 (Iterative and incremental design and develop-ment). User-centered agile approaches should support softwaredesign and development in short iterations and in an incrementalmanner.

5.1.3. Parallel Interwoven Creation TracksFocus Area. Extending the separate product discovery and

product creation principle, the third principle shapes the subse-quent course of action after the start of regular design and develop-ment activities. As a restricted up-front phase of analysis anddesign allows for specifying the user interaction only for the mostimportant features of a system, features with a lower priority haveto be considered in later iterations in parallel to development.Thus, it is necessary to conduct user research and prepare designsfor the upcoming development cycle at least one iteration (orsprint) ahead of the development team.

Evidence. In total, 18 (21.7%) of the publications included in theliterature review support this principles, whereas five are empiri-cally grounded. On the one hand, the literature explicitly combinesthe LDUF concept with the organization of continuous analysis anddesign activities. On the other hand, development activities need tobe organized in parallel tracks. The literature offers no alternativeapproach for the latter. The incremental nature of ASD enablesUCD experts to prepare the interaction concept for envisioned sys-tem features consecutively according to their priority. Thus, an ini-tial interaction concept for the most important system features canbe prepared in an upstream iteration prior to development.Enhancing this concept in parallel to development allows for main-taining a suitable balance between necessary up-front design andflexibility, while designs are prepared for implementation just intime. Additionally, engaging in continuous analysis and designactivities in parallel to development allows for incorporatingchanging user and customer needs, thus leading to an increaseddegree of software usability and usefulness.

The concept of designing one sprint ahead is commonly found inexisting UCASD approaches, and was mentioned in 17 (20.5%) pub-lications in the review. However, only two publications supportthis argument with empirical evidence [49,56]. As examples, Mill-er’s [55] and Sy’s [24] proposals schedule activities for the designteam to gather customer data and develop user interface specifica-tions for the next cycle, while the development team simulta-neously implements specifications prepared in the previouscycle. Fox et al. [21] moreover propose that the development teamimplements the design concept prepared in the preceding itera-tion, while the UCD staff conducts a contextual inquiry and pre-pares a design prototype for the next iteration.

Despite the need to work in parallel, seven (11.86%) of the pub-lications in the review point out that a deferred developmentnecessitates mechanisms for the synchronization and integrationof design and development work. However, the scheduling ofresources in practice is a non-trivial task and can be identified asone constraint. Researchers investigating the topic of cross-func-tional integration have discussed the question of how to integratethe development and design functions since the inception ofUCASD more than a decade ago. The potential of cross-functionalintegration has also been intensively investigated from an empiri-cal point of view in product development research [e.g., 142,143].Two specific dimensions of cross-functional integration in the con-text of UCASD are identifiable in the literature. On the one hand,

Table 6Codes related to the iterative and incremental design and development principle.

Included codes Resulting principle

Iterative design/development Iterative and incremental design anddevelopmentIncremental design/

development

Table 7Codes related to the parallel interwoven creation tracks principle.

Included codes Resulting principle

Parallel tracks Parallel interwoven creation tracksDesign one sprint aheadSynchronization/integration

172 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181

Sohaib and Khan [23]. Specifically, their reviews confirm the signif-icance of iterative and incremental design and development. Thereis general agreement among UCD researchers that an iterativeapproach is essential for developing a product with high usability[123,126]. According to Mao et al. [118], design iterations are themost commonly used UCD paradigm; however, feedback is alsoan essential aspect in ASD. In ASD, the overall project is typicallycomposed of subsequent self-contained cycles, each of which com-prises analysis, design, programming, and test activities [127].Among the included papers, 33 (39.8%) explicitly discuss designor development iterations. In accordance with UCD approaches,empirical feedback is used to revise designs in the next iteration.The iterative refinement and evaluation is continued until theuser’s needs are met, thus ensuring that an adequate level ofusability is achieved [124,128]. In particular, all publications pre-sented in the process dimension (see Table 3) promote an iterativeprocess setup.

An iterative approach allows for the product under develop-ment to be refined based on feedback collected at an earlier stage.In contrast, an incremental strategy means that system functional-ity is partitioned into thin vertical slices from the user interface tothe database layer. Each slice is functionally coherent and demon-strable. Instead of delivering the required functionality all at once,the scope of the system grows progressively with the addition ofconsecutive slices [129,130]. Solutions to software developmentproblems are often complex and exhibit many degrees of freedom.Thus, eliciting requirements from users often results in whatBoehm [131] calls the IKIWISI symptom: ‘‘I can’t tell you inadvance, but I’ll know it when I see it’’ [131]. Especially for interac-tion features, users might not be able to specify their needs unlessconfronted with a tangible representation of the system, such as aworking product increment that would allow them to clarify andrefine their vision of the software [9,127,131]. Drawing on thisinsight, the aspect of incremental development was mentioned in11 (13.3%) articles, typically with reference to functional tests orusability evaluations.

The associated papers identified in the literature provide noempirical evidence for the outcome of iterative and incrementaldesign and development. This may be because the empirical stud-ies not only consider the user-centered design or agile develop-ment perspective independently from each other, but as acombination of the two [21,35,42,46,49,103,105].

Suggestion. In summary, developing software in an iterativemanner creates short feedback cycles, while an incremental devel-opment in vertical slices allows for evaluating working productincrements. Additionally, adapting the product scope after eachiteration embraces change and is a prerequisite to deliver valuablesoftware (see Table 6).

Beyond the frequent occurrence of the two aspects in the liter-ature, a general acceptance of this strategy can be inferred from

references to iterative and incremental concepts for ASD (e.g.Scrum) or UCD (e.g. ISO 9241-210). Even though the reviewedpapers do not explicitly discuss an iterative and incrementalapproach, none of them challenge this paradigm. Thus, the secondprinciple is articulated as follows:

Principle 2 (Iterative and incremental design and develop-ment). User-centered agile approaches should support softwaredesign and development in short iterations and in an incrementalmanner.

5.1.3. Parallel Interwoven Creation TracksFocus Area. Extending the separate product discovery and

product creation principle, the third principle shapes the subse-quent course of action after the start of regular design and develop-ment activities. As a restricted up-front phase of analysis anddesign allows for specifying the user interaction only for the mostimportant features of a system, features with a lower priority haveto be considered in later iterations in parallel to development.Thus, it is necessary to conduct user research and prepare designsfor the upcoming development cycle at least one iteration (orsprint) ahead of the development team.

Evidence. In total, 18 (21.7%) of the publications included in theliterature review support this principles, whereas five are empiri-cally grounded. On the one hand, the literature explicitly combinesthe LDUF concept with the organization of continuous analysis anddesign activities. On the other hand, development activities need tobe organized in parallel tracks. The literature offers no alternativeapproach for the latter. The incremental nature of ASD enablesUCD experts to prepare the interaction concept for envisioned sys-tem features consecutively according to their priority. Thus, an ini-tial interaction concept for the most important system features canbe prepared in an upstream iteration prior to development.Enhancing this concept in parallel to development allows for main-taining a suitable balance between necessary up-front design andflexibility, while designs are prepared for implementation just intime. Additionally, engaging in continuous analysis and designactivities in parallel to development allows for incorporatingchanging user and customer needs, thus leading to an increaseddegree of software usability and usefulness.

The concept of designing one sprint ahead is commonly found inexisting UCASD approaches, and was mentioned in 17 (20.5%) pub-lications in the review. However, only two publications supportthis argument with empirical evidence [49,56]. As examples, Mill-er’s [55] and Sy’s [24] proposals schedule activities for the designteam to gather customer data and develop user interface specifica-tions for the next cycle, while the development team simulta-neously implements specifications prepared in the previouscycle. Fox et al. [21] moreover propose that the development teamimplements the design concept prepared in the preceding itera-tion, while the UCD staff conducts a contextual inquiry and pre-pares a design prototype for the next iteration.

Despite the need to work in parallel, seven (11.86%) of the pub-lications in the review point out that a deferred developmentnecessitates mechanisms for the synchronization and integrationof design and development work. However, the scheduling ofresources in practice is a non-trivial task and can be identified asone constraint. Researchers investigating the topic of cross-func-tional integration have discussed the question of how to integratethe development and design functions since the inception ofUCASD more than a decade ago. The potential of cross-functionalintegration has also been intensively investigated from an empiri-cal point of view in product development research [e.g., 142,143].Two specific dimensions of cross-functional integration in the con-text of UCASD are identifiable in the literature. On the one hand,

Table 6Codes related to the iterative and incremental design and development principle.

Included codes Resulting principle

Iterative design/development Iterative and incremental design anddevelopmentIncremental design/

development

Table 7Codes related to the parallel interwoven creation tracks principle.

Included codes Resulting principle

Parallel tracks Parallel interwoven creation tracksDesign one sprint aheadSynchronization/integration

172 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181

Page 108: Systematic Literature Review

the necessity of functional diversity on the team level, i.e. thedegree to which team members differ with regard to their func-tional background and experiences [132], results in an organiza-tional challenge to integrate design and development experts. Onthe other hand, team members with multi-knowledge, individualknowledge, and experience across different functional areas [133]can be important enablers of cross-functional work in softwaredevelopment projects. Concerning the former aspect, the prefera-ble organizational setup for UCASD is yet to be determined. Theoptions are to appoint two dedicated teams to handle design anddevelopment activities, respectively, or to establish a cross-func-tional team that handles both tasks [101,102]. The latter is in linewith ASD, which promotes the idea of a team including ‘‘peoplewith all the skills and perspectives necessary for the project to suc-ceed’’ [5]. Schwaber and Sutherland [121] emphasize that an agileteam should have ‘‘all competencies needed to accomplish thework without depending on others not part of the team’’ [121].Similarly, 34 (41.0%) of the papers included in the literature reviewdescribe a cross-functional setup, in which UCD expertise is avail-able within an agile development team. According to Kuusinenet al. [46], this is also practitioners’ most frequent proposal toimprove the cooperation between UCD experts and developers.As it is frequently mentioned, a cross-functional team allows fora direct and unmediated exchange of knowledge and informationbetween design and development experts [22,100,102]. In particu-lar, it allows for maximizing collaborative activities in order to cre-ate possibilities for knowledge transfers. As an example, Ungar andWhite [108] suggest workshops in which designers, developers,and stakeholders collaborate and explore design alternatives as aforum for ad-hoc knowledge exchanges between the involvedparties.

Suggestion. Design and development activities as part of theproduct creation phase are often described as parallel tracks. Ithas to be kept in mind that an interaction concept destined forimplementation in an upcoming iteration has to be finished priorto the start of its development. Table 7 lists the codes that builtthe foundation for suggesting one further principle, i.e. parallelinterwoven creation tracks, from a process perspective.

To summarize, design activities need to start one sprint aheadof the development activities. Given the parallelization,mechanisms need to be adopted to assure the synchronizationand integration of such tracks.

Principle 3 (Parallel interwoven creation tracks). In user-centeredagile approaches, design and development should proceed inparallel interwoven tracks.

Further empirical research is required to investigate differentcross-functional UCASD creation track setups under considerationof contingency factors. Existing literature providing empiricalinsights into cross-functional integration in product developmentmight serve as a useful starting point. Such insights would requirefurther refinement to focus on UCASD and its outcomes on a teamand product level.

5.2. Practices for principles

The practices perspective considers concrete methods that areexecuted within the coding system. The literature included inour review discusses a large variety of practices (see Fig. 8 inAppendix A). In an effort to consolidate this diversity of practices,principles for the practices dimension of UCASD should capturethe rationale behind these practices rather than pay attention tothe practices themselves. Consequently, practices conducted inan UCASD approach have been classified along two holisticprinciples.

5.2.1. Continuous Stakeholder InvolvementFocus Area. While ASD is inherently people-centric and the

active involvement of stakeholders is one of its essential elements[3,135], it lacks a clear distinction between the customer and theend-user of the software [35]. It is not unusual for one person,for example a domain expert or a product manager from the clientorganization, to fill both the customer and user roles [136]. Suchroles are not able to represent all stakeholders in the system[62]. However, UCD approaches require direct and unmediatedcontact with the end-user of the software [123,126].

Evidence. Various publications in the literature review discussthe decision between direct or mediated stakeholder relationsfrom a general perspective without reference to specific concepts.Thereby, 17 (20.5%) of the included publications promote the directinvolvement of stakeholders, i.e. direct and unmediated contactbetween stakeholders and design or development experts. In theirethnographic study focusing on the commonalities of agile meth-ods and user-centered development, Chamberlain et al. [35] con-firm the importance of continuous stakeholder integration. In

Fig. 6. Codes and number of articles related to the continuous stakeholder involvement principle.

M. Brhel et al. / Information and Software Technology 61 (2015) 163–181 173

Page 109: Systematic Literature Review

contrast, eight (9.6%) of the articles recognize different motivationsto involve stakeholder representatives, which might be preferableconsidering resource constraints or difficulties to gain direct accessto prospective users. Beyond the specific scope of UCASD, empiricalliterature has intensively researched the importance of user inte-gration in software development [e.g. 144,145].

Going beyond the general perspective, two focal areas of stake-holder involvement in the context of UCASD became apparent inthe review. The evaluation of the system can be recognized asthe most important motivation to establish direct contact withcustomers, i.e. end-users. In particular, 37 (44.6%) of the reviewedpublications discuss usability testing with end-users. Nevertheless,practices used in the early phases of software development, inwhich interactions with stakeholders are necessary to establishsystem requirements, are equally important. While 12 (14.5%) ofthe articles only take a generic perspective on user research byhighlighting the necessity to conduct user research or by discuss-ing the inclusion of user research in an agile development process,10 (12.0%) publications give explicit attention to contextual inquiryin an agile environment. Other practices for stakeholder involve-ment, including task analysis (6 publications, 7.2%), focus groups(4 publications, 4.8%), interviews (2 publications, 2.4%), and surveys(2 publications, 2.4%) are mentioned, but receive less attention.

Suggestion. Stakeholders’ continuous involvement in thedesign and development process is one of the main commonalitiesof UCD and ASD. Both emphasize the human aspect of softwaredevelopment (see Fig. 6).

The value of individuals and interactions is documented in theagile manifesto [134]. According to UCD, understanding users andtheir tasks should be the focus of software development [126].Moreover, UCD and ASD both encourage customer or user partici-pation in the development of new systems or software. Thus, wesuggest the following principle:

Principle 4 (Continuous stakeholder involvement). Stakeholdersshould be actively involved in user-centered agile approachesearly on and should remain involved throughout the entiredevelopment process to collect input and feedback.

In the context of UCASD, there is a lack of empirical studies sys-tematically investigating the continuous stakeholder involvementprinciple. Building on literature on user involvement [144,145],the extent and outcome of continuous stakeholder integration inUCASD requires further empirical research. From a practical pointof view, empirical insights could be leveraged to give context-specific recommendations for applying different stakeholder inte-gration strategies, using associated practices. A deeper understand-ing of contingency factors and their influence is required to derivecorresponding recommendations for successfully applying thisprinciple.

5.2.2. Artifact-Mediated CommunicationFocus Area. In contrast to traditional plan-based approaches,

ASD discourages the use of formal documentation, as it values‘‘[w]orking software over comprehensive documentation’’ [134],and knowledge about products and processes is supposed tobecome tacit and should be exchanged through interactionsbetween team members [137]. In order to reduce ‘‘waste,’’ docu-mentation should be reduced to a minimum, with the team onlydocumenting what is absolutely necessary [3,138]. As Kuusinenet al. [46] state, there are as yet no commonly agreed upon con-cepts for communicating design or documenting user require-ments in UCASD. Nevertheless, while analyzing the documentcorpus in the literature review, it became apparent that, besides

Fig. 7. Codes and number of articles related to the artifact-mediated communication principle.

Table 8Principles for UCASD.

Principle Description

Principle 1 Separate product discovery and product creationPrinciple 2 Iterative and incremental design and developmentPrinciple 3 Parallel interwoven creation tracksPrinciple 4 Continuous stakeholder involvementPrinciple 5 Artifact-mediated communication

Table 9Overview of recently published papers in 2013/2014 along the four dimensions.

Dimension Included publications

Process Salvador et al. [147], Butt et al. [150], da Silvia et al. [152], Plonka et al. [162], Kuusinen [166], Ahmad et al. [167], Maurer & Hellmann [169], Humayounet al. [171]

Practices Salvador et al. [147], Arnowitz [148], da Silvia et al. [152], Peres et al. [153], Ardito et al. [155], Inayat [156], Salah et al. [157], Cajander et al. [159],Bertholdo et al. [160], Caballero et al. [161], Plonka et al. [162], Lizano et al. [165], Kuusinen [166], Maurer & Hellmann [169], Häger et al. [170]

People &social

Arnowitz [148], Kropp & Koischwitz [149], Jurca et al. [151], Kuusinen & Mikkonen [154], Raison & Schmidt [158], Cajander et al. [159], Bertholdo et al.[160], Plonka et al. [162], da Silva et al. [163], Heimgärtner & Solanki [164], Lizano et al. [165], Kuusinen [166], Wale-Kolade et al. [168], Häger et al.[170]

Technology Salvador et al. [147], Humayoun et al. [171]

174 M. Brhel et al. / Information and Software Technology 61 (2015) 163–181

Page 110: Systematic Literature Review

stakeholder involvement, the use of various artifacts related to thedesign and development process is most prominently discussed inresearch publications. Artifacts are defined as an ‘‘. . . aspect of thematerial world that has been modified over the history of its incor-poration into goal-directed human action.’’ [139]. This definitionincorporates both the tangible and mutable nature of artifactsand their purposeful use. Artifacts, such as those described below,have a long tradition of being used for documentation purposes indesign and development activities [99]. Besides supportingcreative processes, they are an important means for organizingcommunication and collaboration among internal and externalparties.

Evidence. Discussed in 40 (48.2%) of the publications includedin the literature review, prototypes, i.e. visual instantiations ofthe design concept [140], is the most frequently occurring artifacttype. However, only three papers provide empirical evidence sup-porting this principle [46,48,99]. Remaining very general, Lee et al.propose a new approach that combines user-centered and agilesoftware development using central design records (CDRs) as arti-facts [48]. Kuusinnen et al. focus on the interaction of the UX teamand the development team in a large software development orga-nization, identifying the creation and communication of designs asan important task of the UX team [46]. Going into more detail, anethnographic study by Brown et al. [99] analyzes the role ofartifacts during the interaction between design and development.

While Brown et al. [99] mention prototypes with different lev-els of fidelity, they usually recommend transitioning from low-fidelity to high fidelity prototypes in design iterations. Addition-ally, 17 (20.5%) of the included publications consider the use ofsketches or mock-ups as artifacts to support the early ideationstage of the design process [140], and nine (10.8%) articles mentionthe use of wireframes to depict the envisioned layout of a userinterface. Kuusinen et al. [46] describe challenges during the inter-action of design and development efforts based on the example of amulti-continental development team. Two studies (2.4%) furtherdiscuss the use of multiple wireframes in a user interface flow,which is a set of wireframes that visualize which elements of a userinterface are used in an interaction path through the system.

A second group of artifacts is inherently linked to stakeholderinvolvement, as the documentation of stakeholder properties andneeds are also of interest to the user-centered agile community.The employment of user stories to describe features providing busi-ness value to the customer is popular, and mentioned in 21 (25.3%)of the publications in the review. While user stories stem from theagile world, the idea of modeling the system’s value proposition inscenarios providing a step-by-step narrative on how a prospectiveuser will benefit from the envisioned product is a user-centereddevelopment concept, and is discussed in 15 (19.3%) articles. Fur-thermore, the use of personas as concrete representations of userarchetypes is highly popular in UCASD, and discussed in 13

Fig. 8. Codes and number of articles related to dimension ‘‘Practices’’.

M. Brhel et al. / Information and Software Technology 61 (2015) 163–181 175