Understanding software quality assurance in Open Source Software

1

Understanding Software Quality Assurance In OSS

Behutiye Woubshet Nema

Adeyemi Bolaji

Shikur Henok 2013

2

3

Table of Contents Abstract .................................................................................................................................... 4

1. Introduction ............................................................................................................................ 5

2. Overview of Software quality assurance ................................................................................ 7

3. Quality Assurance in Open Source Software .......................................................................... 9

3.1 Quality assurance involvements in OSS ..........................................................................12

3.1.1 Developer perspective ..............................................................................................12

3.1.2 User perspective .......................................................................................................12

3.1.3 Organization perspective .........................................................................................13

4.0 Discussion...........................................................................................................................14

5. Conclusion ............................................................................................................................17

References ...............................................................................................................................18

4

Abstract

There are lots of challenges confronting the open source software development, and one of the

problems is quality assurance. And with regards to this fact, the paper investigates the

responsible actors in assuring quality in open source software development. We meticulously

explore, evaluate and interpret the existing literature as regards quality in Open Source Software

(OSS) context.

The principal findings of the study show that sustainable community participation, development

processes, governance, systems and tools used to support development affect the quality

assurance of open source software. The paper reflects the activities which principal stakeholders

of OSS community should execute in order to achieve quality. The need to investigate on how to

encourage developers’ adherence towards strict development processes in OSS was also

proposed as future work. The rationale behind this is that, it will help address the discrepancy in

development process which is a key source of quality issues in OSS. The study shows the key

stakeholders that are responsible in ensuring quality in different phases of software development

in OSS and the respective activities.

5

1. Introduction

Open source communities have been rapidly attracting attentions across different fields. The

underlying reason for this unprecedented development is still not completely known, although,

many Researchers concluded that, the reason is due to low cost of software development, others

(De, 2009; Nakakoji, Yamamoto, Nishinaka, Kishida, & Ye, 2002) argued that, because it is free,

so it attracts lot of users. However, there are lots of challenges confronting Open Source

Software(OSS) projects, in general. According to (A. Khanjani & Sulaiman, 2011a), identified

one of the challenges to be in the area of quality of the software, and they also mentioned that,

one of the biggest task of OSS is assuring quality, same conclusion is reflected in literature, for

example (Nakakoji et al., 2002). Subsequently, OSS is yet to attain its full potential, as lot of

critical projects do not consider using OSS mainly because there is no strict process in assuring

quality, in other words, “it seems no one is responsible for assuring quality”. Further, Hedberg et

al. (2007) predicted that, with the increase in commercial use of OSS, there will more people

involved in OSS communities, particularly, the end-users with cravings for quality products

(Hedberg, Iivari, Rajanen, & Harjumaa, 2007).

According to ISO 8402 quality assurance is define as “All those planned and systematic actions

necessary to provide adequate confidence that a product or services will satisfy given

requirement of quality” (ISO 8402). With regards to the aforementioned definition, it is logical to

say that before quality can be attained there must be a strict process to be followed, and in the

context of software engineering this process is called software testing process. Regrettably,

Gacek and Arief in (2004) stated that OSS developers are less interested in both documentation

and testing (Gacek & Arief, 2004) processes vital in assuring quality. Herberg et al. (2007) also

supported this claim using these words “usually there is no testing plan, the test coverage is not

addressed” (Hedberg et al., 2007). Nonetheless, (Opensource, 2013) mentioned that OSS model

is peculiar both in organizational structure and the development process. The OSS development

process is not meticulously planned at the beginning of the project, it evolves in response to its

users demands, as contrast to the proprietary software model. OSS software development

involves distributed contributions of codes (A. Khanjani & Sulaiman, 2011a; Nakakoji et al.,

2002). Thus, the software product of OSS is highly prone to defects or bugs, because the

developers (contributors) may be following different development processes (modules).

Although, some Researchers over the years has concluded that open source development has a

greater chance of increasing the quality of product or reducing the bug, because the underlying

model in open source is that, the more people that have access to the codes, the more chances of

detecting bugs, contrast to the traditional closed source.

In addition, OSS is unique in a distinctive way, because of its structure- developer, co-developer

(who are regarded as contributor) and the end-users, these are the stakeholders in any OSS

project and they cluster around an OSS project as a community. However, without the presence

of the aforementioned principal actors, there is high possibility that a given OSS project may fail

(Nakakoji et al., 2002). In order to maintain a well-focused literature review, we have drafted

6

research questions as a guide to our review. Thus, the current study aim to investigate assuring

quality in OSS by examining different stakeholders(participant) in the development stage, with a

very focused research question of- “who is responsible for quality assurance in OSS projects?”.

We believe that, answering this question will help uncover the areas that need improvement in

terms of strictly following development processes (developers) or encouragement- for example

increase user’s participation.

We intend to conduct this study, solely by analyzing literatures with the anticipation of bringing

to light the gaps in literature as well as identifying, whose role it is to assure quality in the

different phases of software life cycle in OSS. In addition, we assumed that, the result of the

current literature review will give a broader and concise definition over who is responsible for

software quality assurance in open source software development. As result, inform both research

communities and practitioners.

The primary search process was done using online databases (the ones Oulu University

subscribed to), search engines, electronic journals, and conference proceedings. The common

online resources includes; Google scholars, Web of science, Scopus, ProQuest Databases, ACM

Digital library, IEEE xplore, SpringerLink, EBSCO, ScienceDirect, etc.

Further, papers that discussed software quality, software defect management, and software

quality assurance are the studies taken as the primary studies for the review. Additionally, the

study classified the papers based on the year of publication, the research data, research method,

and most importantly the findings. Furthermore, papers or scientific studies that are not free or in

the database research support of university of Oulu were excluded, regardless the year of

publication or the content of the paper. The papers with the above selection criteria were written

in English language, this is identified as one of the potential setbacks of the current study.

The rest of this paper is organized as follows. Section 2 provides an overview of quality

assurance in software engineering in general, as well as from open source software community

perspective. Section 3 explains in detail the factors affecting quality in OSS as describe in the

literature. Section 4 presents the discussion of the study. And finally, Section 5 presents the

conclusion of the study.

7

2. Overview of Software quality assurance

Many studies have been conducted in the context of assuring software quality (Gacek & Arief,

2004; Hedberg et al., 2007). Amidst these studies are definition of QA in software engineering,

the best practices in assuring quality, and the factors that directly impact quality- development

process, process conformance, and improvement of processes as well as validating processes.

Quality assurance is a way to assess whether the product has reached or satisfied customer

expectation. In quality assurance process there are steps that should be followed such as: design,

development or evaluation of the product by testing the result against the requirements.

Although, these processes might change depending on the project and specific companies

requirement. The whole theme of quality assurance is to provide good quality products and

services for customers. (A. Khanjani & Sulaiman, 2011b)(A. Khanjani & Sulaiman, 2011a)

Furthermore, quality assurance is a substantial part of software development as it can be related

to correctness, usability, completeness and maintainability of software which affect economic

and technical decisions made during software development. Software quality impacts a number

of important factors in our daily lives such as economy, personal and national security, health

and safety (Naik & Tripathy, 2011). Moreover, advancements in technologies and complexity of

software systems also make it necessary to understand and achieve quality assurance in our

products and development processes. This is also supported by (Otte, Moreton, & Knoell,

2008a), “The execution of SQA activities requires the introduction of a quality model to obtain

process, transparency, customer satisfaction, repeatable processes and methods.”

In proprietary software development process, quality assurance process includes different steps

performed by developers and testers along the development process. During the process different

standards have been applied to assure compliances with set of requirements. Performing quality

assurance has been a problem in open source software development. The method used in

traditional software development process is not pertinent to OSS development since OSS

development life cycle is unstructured and disordered. (Ardagna, Banzi, Damiani, & Frati, 2009).

Moreover, Software QA needs an integral collaboration of different elements (stakeholders) in

the software development ecosystem and a follow up of systematic procedures to attain quality.

It is highly important to ensure that processes and tools are utilized in efficient manner to achieve

quality.

Tuteja and Dubey (2012), argue that quality cannot be improved by assessing an already

completed product and highlight the importance of quality assurance measures targeted to

improve software development procedures, standards and tools used in the development. In

addition, arguing software testing techniques as measurements of software quality the authors

depict that applying software testing methods in different lifecycle stages of software

development contribute to ensuring software quality. (Tuteja & Dubey, 2012)

Further, to achieve quality product, the conformance to the development process must be strict

and followed also in distributed teams (Ardagna et al., 2009; Tuteja & Dubey, 2012). Sivert

8

(1997) defines process conformance in the context of software development, informally as “the

degree of agreement between the software development process that is really carried out and the

process that is believed to be carried out” (Sørumgård, 1997). The study aimed to uncover the

importance of process conformance in software engineering field. As a result the study helped to

propose four key components to measure process conformance in software development. The

components are -model for definition, measurement based on the model, an alternative

measurement and guidelines for modification of a process, so as to enable conformance

measurement (Sørumgård, 1997). Rigidly, conforming to the processes of development,

subsequently assuring quality of the software product. In addition, we can infer that, if software

products have low quality- the reason maybe that the process of development is not conformed to

or the process of development requires some improvement. Bhandari et al. (1993) selected a

narrowed aspect of software process improvement, which is defect-based oriented. They were

motivated by the inconsistency in implementation of processes in the context of software

development. However, the study responded to a practical question by a developer; “I know what

I must produce; I know the process I must follow; and I have started development. How do I

detect if things are going wrong, and what do I do to get back on track?” Bhandari et al., (1993),

also to create process improvement metric during development, further, the research method is a

case study of a developing team of 25 people and the main conclusion of the study was that

process improvement works (Bhandari et al., 1993). This implies that during development

process, developing team (developers) could continually improve their processes, thus assuring

quality.

In terms of development processes, it is highly important to validate the process. A relevant

study in this regard was conducted by Cook and Jonathan (1999), the study provides deeper

insight in the relationship between a formal process model and process execution. As taken from

the study; “When process models and process executions diverge, something significant is

happening”. They investigated amidst the metrics involved in the processes, the human or

behavioral aspects. Further, they proposed three metrics for process validation; Recognition

Metric (REC), Simple String Distance Metric (SSD) and Non-linear String Distance Metric.

These metrics were demonstrated using the Test Unit task from (Kellner et al., 1990). This is

unique in the sense that it uses the data in process validation activity (Cook & Wolf, 1994).

In addition, a recent study by Dimech et al. (2013) aimed on improving process conformance as

regards to the requirements in the context of design-driven development. They support the idea

that initial requirement can be traced from high-level design forms through the final product.

Furthermore, the authors proposed an insight of handling issues relating the problem space

among which is conformance using DiaSuite methodology (Dimech & Balasubramaniam, 2013)

9

3. Quality Assurance in Open Source Software

Unlike proprietary software development model which bases on systematic and procedural

testing, the open source software development highly utilizes post-delivery bug reports in order

to establish Quality assurance of software. Developing open source software products requires

specific methods and techniques for quality assurance process to control the quality of the

product (A. Khanjani & Sulaiman, 2011a). In addition, quick evolution of the software

development process which is mainly driven by various actors including developers, users, firms

and other participants is also another unique characteristic of the OSS model contributing to

Quality assurance. As stated by Bonaccorsi and Rossi (2003), increased participation brings

project momentum and frequent releases lead to bugs correction and generate feedback flows

that improve the code and the understanding of the underlying problem, a practice in contrast to

the conventional way of assuring quality in proprietary software (Bonaccorsi & Rossi, 2003).

A survey research on applied quality assurance methods of OSS (Otte, Moreton, & Knoell,

2008b) found out that most successful OSS projects apply structured and organized development

processes. The study also showed that engaging experienced project management, skilled

developers and users in the OSS community is substantial for quality assurance. Moreover, it

was also shown that it is essential to implement guidelines, documentations, standards and

processes to improve quality. The findings support the indispensable inclusion of a wide and

experienced OSS community together with adoption of systematic tools and development

processes to achieve software quality assurance.(Otte, Moreton, & Knoell, 2008b)

Further, high Quality Open source software development depends on fundamental sector of

sustainable community, code modularity, project management and test management. In order to

attain software quality, OSS practitioners should fully understand the dependency between these

sectors. Many researchers concluded that having a large sustainable community should be the

main objective in OSS. A research on OSS project by Mark Aberdour (2007) concluded that a

large contributor presence was one of the main factors for high OSS quality (Aberdour, 2007).

Also, Code modularity is still one of the main factors for OSS quality which programmers can

extend the program by working on separate modules, without the need to know the full

understanding of the system. This minimizes the risk of new bugs being introduced in other

modules. In addition, peer review and people management play a big role for OSS quality. Peer

review mainly focuses on assessing whether the contribution quality compliance with the code-

base. Moreover, people management plays a great role on establishing an effective environment

and culture; it is seen as important as system design. (Aberdour, 2007)

Furthermore, QA in open source software depends on two processes: code review and data

testing. The accessibility of open source software allows anyone to analyze and make thorough

assessment and get more bugs. Thus, having large community for knowledge sharing, paid and

unpaid contributors, expert developers for controlling and managing bugs, etc. For example,

open source software like GNU/Linux, Apache, Firefox, etc., receive more bugs due to the fact

10

that different contributors share their idea that leads to the possibility of increased bugs to be

released. (A. Khanjani & Sulaiman, 2011a)

Referring to the study of Nakakoji, Kumiyo, et al. (2002), “ a full understanding of the evolution

of an OSS system cannot be complete without understanding the evolution of the OSS

community and its roles in driving the evolution of the system ” (Nakakoji et al., 2002). Hence,

we aim to examine the role of each stakeholder as regard to quality of product in OSS project.

The perceptions covered in this study includes; developers- core and contributors, users and

organization. Also, this classification is informed by the popular onion model of OSS pattern.

Figure 1, adopted originally from the onion model, shows that sustainable community is

constructed from small number of core developers and increasing number of contributing

developers, bug reporters, users and organizations. Sustainable community is like multi layered

onion with users in the outer layer and move towards core through bug reporters and contributing

developers. A user increases their involvement on the basis of their knowledge through the

process. As the users move towards the core, user becomes bug reporters, contributing

developers, and core developers consecutively.

Stakeholder involvements in different stages of open source software development are explained

by drawing a metaphor based on the onion model as shown below. In order to achieve high

quality Open Source Software, the OSS community should organize processes from various

perspectives: organization perspective, user perspective, developers’ perspective and take

measures to assure quality (Aberdour, 2007).

Figure 1. A metaphor based on the onion model (Aberdour, 2007), shows sustainable community

in quality assurance from different stakeholder perspectives.

11

A number of models and methods have been introduced to ensure software quality. Some of the

recommended practices Hedberg et. al (2007), suggests organization and companies should

follow stricter quality assurance methods, since the commercial aspect of OSS is increasing

abundantly (Iivari, Hedberg, Rajanen, & Harjumaa, )(Hedberg, Iivari, Rajanen, & Harjumaa,

2007). In addition, it is stated that planning and documenting quality assurance process eases the

communication problem between the increasing developers and users in OSS. It is also suggested

that more emphasis should be given to testing and Test Driven Method (TDD) method should be

used in the development process. Others (Wahyudin, Schatten, Winkler, & Biffl, 2007), also

concluded that software testing and peer reviews help to reduce software defects hence software

quality is ensured.

User, developer and organizational participations are seen as the main stakeholders essential to

maintain QA of OSS. The user participation and feedback constitutes one of the backbones of the

open source model (Raymond, 2008) Different studies have showed that OSS projects that have

the support of firms and a large scale of user community benefit from participation of these

actors in assuring Software quality. Therefore, it is important to understand different aspects of

stakeholder involvements and how each stakeholder’s participation affects quality of the

software. Hence, with a focused research question- who is responsible for quality assurance in

OSS?

Sunindyo et al. (2013) study the business process in connections to OSS and proposed a

framework that could help OSS project managers to efficiently manage the business process.

Owing to the fact that, it is the responsibility of the OSS project managers to manage the

development process and delivering of product to the End-users. Although, this role could be

demanding, considering the collection and analysis of various data collected from different

stakeholders. Further, the proposed framework consists of four levels; data source, data

collection, data integration and the validation level. The result of the study shows that, Open

source Model framework (OSMF) can assist managers handle business process more efficiently,

as well as enhancing judgment on project quality. (Sunindyo & Ekaputra, 2013)

Abdou et al. (2012) studied popular OSS projects; Apache HTTP server, Mozilla Web browser,

and NetBeans IDE. Their focus was to analyse the test processes undertaken in those projects.

The results revealed that, three activities were in compliance to the Test Process Standard

ISO/IEC, although, significant dissimilarities were noticed in regards to the task associated with

the test process practices. They proposed a framework for software test process in OSS called

(OSS-TPF), which was compared with ISO/IEC Test process to reveal the areas that requires

improvement in achieving quality product from OSS software development. (Abdou, Grogono,

& Kamthan, 2012)

In Rashid et al. (2012), studied perception of quality of OSS product with the focus of

investigating the process practices and problems associated with the development process from

the perception of the developers. The result of the study revealed that, there is no coordination

among the developers, because of the distributed nature of OSS developmental model. However,

they highlighted the relevance of coordination between developers, thus, encourage developers

12

to coordinate, since it makes the release process systematic. As a result assist to obtain the

objective of software release process. (Rashid, 2012)

3.1 Quality assurance involvements in OSS

Different sectors of the open source software community play roles in quality assurance process.

Along community participation, governance, methodologies (development processes), systems

and tools contribute to ensuring quality in the open source software platform. However, after

reviewing processes and participations in this literature review only three key stakeholder

involvements will be emphasized to describe quality assurance participation and roles.

3.1.1 Developer perspective

Roberts et al., (2006) studied the motivations behind developer communities of the Open Source

software platform and showed that there is a significant interest for ensuring quality besides

regular contributions. The authors argue that developers’ interest in reputation in a specific

knowledge domain as a driving force for contributions of quality work. In other words,

developers’ intention to seek a status and reputation for specific skill can be related to achieving

Quality in OSS. (Roberts, Adviser-Hann, & Adviser-Slaughter, 2006)

Luyin and Sebastian (2002) studied OSS quality assurance techniques and perception of QA by

OSS developers. The study shows that, in addition to high level of user participation, developers’

contribution is indispensable element in ensuring quality assurance under the OSS development

model. Discussing practices of developers, it was also shown that often OSS developers act as

users to validate functionality of software rather than following formal procedures of traditional

software testing. (Zhao & Elbaum, 2003)

3.1.2 User perspective

User participation in OSS software development is seen as crucial element that is needed to

acquire quality and improved product. OSS projects follow models that encourage user

involvement in order to facilitate development processes and assure quality. This is also

supported by different literatures and theories. Raymond E. on the Cathedral and the bazar

argues the importance of having users, “Treating your users as co-developers is your least hassle

route to rapid code improvement and effective debugging”. (Raymond, 2008)

Furthermore, the level of participation of users can also be related to the success and failure of

OSS projects. High level of user participation is seen as a common pattern and driving force of

QA behind most OSS projects.

Tobias et al. (2008) proposed a quality assurance process framework for ensuring quality in OSS.

The key factors emphasized in the design of the framework included, balanced interaction of

13

human factors, management skills and operating environments of organization and development

measures etc. These factors are shown to contribute to quality assurance of OSS products.

Moreover, the authors argue that “QA is highly influenced due to user testing and debugging,

therefore projects need to concentrate on enabling tasks, such as structured organization, good

documentation and effective communication”.

User’s skill in a specific knowledge domain together with the scale of an open source project can

be used to determine the value of user participation in OSS software quality assurance.

(Jullien & Zimmermann, 2009) show how users’ contribution can be affecting the quality of

OSS by introducing the concept of dominant user skill. Emphasizing on user base perspective

,users’ skill level are used to show how product quality can be improved with bug reports,

patches( fixes) and new features from experienced users.

3.1.3 Organization perspective

Currently, different organizations are on the process of changing their critical systems and

applications to open source application, although, this decision is still highly restricted by the

quality of software in OSS. However, the underlying motivation in the adoption of OSS can be

justified by reduced costs and improved quality of software that exploits experiences of bigger

community. Sarrab et al. 2013, demands for improved and additional features along maintaining

product quality is also additional reason for organizational participation in Open source

software.(Sarrab & Rehman, 2013)

Organizations involved in development and usage of complex and mission critical software

applications find security pitfalls of the OSS as the main problem in integrating (adopting) OSS

into their platforms. And as a result, companies and enterprises are interested in pragmatic

solutions that enable assurance of OSS software. Hence, organizations will be involved in OSS

projects by initiating and making contributions to assure quality aspects of a project. Claudio et

al. (2009b) show how assurance plays a key part as requirements for telecom companies

adopting OSS for their critical systems. In addition, the authors state that “The criticality and

complexity of Telco systems (e.g., for OSS/BSS systems) require the analysis and control of

software quality during the whole development process (ensured by the quality or assessment

levels of the factory or of the partner companies or vendors)”(Ardagna et al., 2009b). This shows

organizational participation and contribution have higher importance in ensuring quality of OSS

projects.

14

4.0 Discussion

In this literature review, a research question for determining principal actors of quality assurance

behind open source software was established as a basis of the study. And hence, areas related to

quality assurance, development processes and community interest of participation have been

covered in the study to determine roles and participation in ensuring quality in OSS.

The principal findings of the study show that sustainable community participation, development

processes, governance, systems and tools used to support development affect the quality

assurance of open source software. By emphasizing on interests behind participation and

processes, three stakeholder involvements have been used to determine OSS quality assurance

roles and responsibilities. However, the findings also showed that governance models, level of

user and developer participation and scale of project affect the quality assurance of open source

software as shown in Figure 2. These findings also align with Aberdour (2007), which argue

“that High-quality OSS relies on having a large, sustainable community results in rapid code

development, effective debugging, and new features” (Aberdour, 2007).

15

Figure2.Factors influencing software quality in OSS development

The participation of different communities: users, developers and organizations together with

governance, systems and tools used in OSS development play an important role in assuring

quality process.

In order to support the idea of sustainable community participation behind quality assurance in

OSS, we can further analyze the relationship between different actors and activities involved in

OSS projects. As it can be shown from Table 1, it is essential to have participation of

organizations and developers in development and post -release phases. In these stages, the actors

(quality assurers) can enhance quality of software by creating coordination, utilizing efficient

systems and tools (test processes, quality code, patches.), following governance models that

encourage contributions etc. User participation (end user) is also shown in activities related to

bug reports, new feature suggestions, detecting bugs which mainly occur in post release phases

of OSS projects. Moreover, the table depicts interdependence among the (quality assurers) actors

in ensuring quality of OSS.

QA Assurer Activities Involvement

Developers (core-developer, contributor)

Govern Code review Peer review Develop test process Strict development

process Coordination Bug report Patch fixes

Development & Post -release phase

Users (end users)

Detect defect Report bugs Suggests features

Post release phase

Organizations Govern Coordination Develop process Test Sponsor(Fund)

Development, post release

Table1. Quality assurance activities in OSS and stakeholder involvements

16

Table1 above shows the recommended activities that each stakeholder should perform to ensure

software quality in OSS. The core developers and contributors have more similarities in terms of

activities recommended, however, the contributor is required to follow strictly the development

process, as well report bug. Similarly, the End- user is expected to also report bugs and suggest

additional features. Organizations should also govern, coordinate and develop processes.

As reflected in Table 1. There is a need to investigate how to encourage developers to follow

development process strictly in OSS. This will address the discrepancy in development process

which is a key source of quality issues in OSS.

Even though the study findings show that mainly organizational, user participation and developer

roles of involvement contribute to quality assurance of OSS, it lacks to support a detailed

analysis of different community participations and organizational structures of OSS models. In

addition, categorizations of the OSS community (stakeholder) involvements made in the study

concentrate on developer, user and organizational participation. However, this comes as a

drawback since there are OSS projects with defined QA teams, project management roles,

documentation roles, software testing team and support roles which are responsible to quality

assurance to different degrees.

17

5. Conclusion

In this review, we aim to answer the question “who is responsible for assuring quality in OSS

project”? Toward that, we meticulously explore, evaluate and interpret the existing literature as

regards quality in OSS context. Consequently, the existing knowledge revealed key factors that

have direct influence in assuring quality in OSS, these factors include: developers, contributors,

organization, and the End-users. It is equally important to emphasize that, OSS project is

different from traditional software development because it is situated within a community, and as

such a holistic approach should be employed when investigating OSS project

The major hindrance in assuring quality in OSS is the fact that there is no formal planning, along

with developers’ lack of conformance to development processes, consequently, making

assurance of quality uncertain. To this end, the existing literature suggested important activities

that should be performed by the aforementioned stakeholders in the preceding paragraph.

The findings are in accordance with the literatures explored in this review, that developers

should provide a development plan, as well as encourage the contributor to strictly adhere to the

process. Similarly, contributors in the other hand are advised to follow development process (if

any), when developing software, they should also report bugs, fix patches. Finally, the End-users

are expected to report defects and suggest features.

The contribution of this literature can be observed from two perspectives- first, for practitioners

this review enhances decision making process in terms of quality by identifying different

activities that should be performed by the key stakeholder, as well as it uncover areas for

improvement. Secondly, for academia, it provides deeper understanding of how quality

assurance is practiced in reality, and also it suggests future work.

Future research should focus on raising quality assurance awareness in the open source

community and also encouraging uniformity of development processes among developers.

18

References

Abdou, T., Grogono, P., & Kamthan, P. (2012). A conceptual framework for open source

software test process. Computer Software and Applications Conference Workshops

(COMPSACW), 2012 IEEE 36th Annual, 458-463.

Aberdour, M. (2007). Achieving quality in open-source software. Software, IEEE, 24(1), 58-

64.

Ardagna, C. A., Banzi, M., Damiani, E., & Frati, F. (2009). Assurance process for large

open source code bases. Computational Science and Engineering, 2009. CSE'09.

International Conference on, , 3 412-417.

Ardagna, C. A., Banzi, M., Damiani, E., Frati, F., & El Ioini, N. (2009). An assurance model

for OSS adoption in next-generation telco environments. Digital Ecosystems and

Technologies, 2009. DEST'09. 3rd IEEE International Conference on, 619-624.

Bhandari, I., Halliday, M., Tarver, E., Brown, D., Chaar, J., & Chillarege, R. (1993). A case

study of software process improvement during development. Software Engineering, IEEE

Transactions on, 19(12), 1157-1170.

Bonaccorsi, A., & Rossi, C. (2003). Why open source software can succeed. Research Policy,

32(7), 1243-1258.

Cook, J. E., & Wolf, A. L. (1994). Toward metrics for process validation. Software Process,

1994.'Applying the Software Process', Proceedings., Third International Conference on the,

33-44.

De, R. (2009). Economic impact of free and open source software: A study in india.

Dimech, C., & Balasubramaniam, D. (2013). Maintaining architectural conformance during

software development: A practical approach. Software architecture (pp. 208-223) Springer.

Gacek, C., & Arief, B. (2004). The many meanings of open source. Software, IEEE, 21(1),

34-40.

19

Hedberg, H., Iivari, N., Rajanen, M., & Harjumaa, L. (2007). Assuring quality and usability

in open soruce software development. Emerging Trends in FLOSS Research and

Development, , 2-2.

Iivari, N., Hedberg, H., Rajanen, M., & Harjumaa, L.Assuring quality and usability in open

source software development.

Jullien, N., & Zimmermann, J. (2009). Firms' contribution to open‐ source software and the

dominant user's skill. European Management Review, 6(2), 130-139.

Kellner, M. I., Feiler, P. H., Finkelstein, A., Katayama, T., Osterweil, L. J., Penedo, M. H., &

Rombach, H. D. (1990). Software process modeling example problem. Software Process

Workshop, 1990.'Support for the Software Process'., Proceedings of the 6th International,

19-29.

Khanjani, A., & Sulaiman, R. (2011a). The process of quality assurance under open source

software development. Computers & Informatics (ISCI), 2011 IEEE Symposium on, 548-552.

Khanjani, A., & Sulaiman, R. (2011b). The process of quality assurance under open source

software development. ISCI 2011 - 2011 IEEE Symposium on Computers and Informatics,

548-552.

Naik, S., & Tripathy, P. (2011). Software testing and quality assurance: Theory and practice

John Wiley & Sons.

Nakakoji, K., Yamamoto, Y., Nishinaka, Y., Kishida, K., & Ye, Y. (2002). Evolution patterns

of open-source software systems and communities. Proceedings of the International

Workshop on Principles of Software Evolution, 76-85.

Opensource, i. (2013). Open source. Retrieved 15.10, 2013, from opensource.org

Otte, T., Moreton, R., & Knoell, H. D. (2008a). Applied quality assurance methods under the

open source development model. Computer Software and Applications, 2008. COMPSAC'08.

32nd Annual IEEE International, 1247-1252.

Otte, T., Moreton, R., & Knoell, H. D. (2008b). Development of a quality assurance

framework for the open source development model. Software Engineering Advances, 2008.

ICSEA'08. the Third International Conference on, 123-131.

Rashid, M. M. (2012). Release process on quality improvement in open source software

project management. Journal of Computer Science, 8

Roberts, J. A., Adviser-Hann, I., & Adviser-Slaughter, S. A. (2006). Open source software

participation: Developer motivations and performance.

20

Sarrab, M., & Rehman, O. M. H. (2013). Selection criteria of open source software: First

stage for adoption. International Journal of Information Processing and Management, 4(4),

51-58.

Sørumgård, S. (1997). Verification of process conformance in empirical studies of software

development. Department of Computer and Information Science, the Norwegian University

of Science and Technology,

Sunindyo, W. D., & Ekaputra, F. J. (2013). OSMF: A framework for OSS process

measurement. Information and communicatiaon technology (pp. 71-80) Springer.

Tuteja, M., & Dubey, G. (2012). A research study on importance of testing and quality

assurance in software development life cycle (SDLC) models. International Journal of Soft

Computing, 2

Wahyudin, D., Schatten, A., Winkler, D., & Biffl, S. (2007). Aspects of software quality

assurance in open source software projects: Two case studies from apache project. Software

Engineering and Advanced Applications, 2007. 33rd EUROMICRO Conference on, 229-236.

Zhao, L., & Elbaum, S. (2003). Quality assurance under the open source development model.

Journal of Systems and Software, 66(1), 65-75.