8/6/2019 Quality Assurance and Testing - Group 3
1/38
Quality assurance andTesting
Group 39th May 2011
Andre Bruintjies
Desir Minnaar
Neil Jonkers
Adolf Lamprecht
8/6/2019 Quality Assurance and Testing - Group 3
2/38
Agenda
Quality and the History
Software Quality Models
Neils Topics
Adolfs Topics
8/6/2019 Quality Assurance and Testing - Group 3
3/38
8/6/2019 Quality Assurance and Testing - Group 3
4/38
What is Quality and Quality Assurance?
Quality
The totality offeatures and characteristics of a product orservice thatbear on its ability to satisfy stated orimplied needs
Computer Dictionary Online .(1995). Quality . Available: http://www.computer-dictionary-online.org/?q=quality [24 April 2011]
The standard of something as measured against other things of asimilar kind; the degree of excellence of something
Oxford dictionaries. (2011). Quality. Available: http://oxforddictionaries.com/view/entry/m_en_gb0678350#m_en_gb0678350 [24 April 2011]
QualityAssurance
Aplanned and systematic pattern of all actions necessary to provideadequate confidence that the product optimally fulfils customersexpectations, i.e. that it is problem-free and will able to perform the task itwas designed for.
Computer Dictionary Online .(1995). Quality Assurance . Available: http://www.computer-dictionary-online.org/?q=quality%20assurance .[24 April 2011]
8/6/2019 Quality Assurance and Testing - Group 3
5/38
The history of quality assurance
History (Evolution) of Quality Control .(2009).
Available: http://www.rajeshtimane.com/56/academics/history-of-quality-control.html . [24April 2011]
Inspection
Total Quality
8/6/2019 Quality Assurance and Testing - Group 3
6/38
Engineering Approaches evolved to IT.
First definition of software engineering (Bauer)
Set of principles seeking a solution to a common problem
Systematic planning
Hoares (1975) four characteristics of software engineering
Wassermann (1996) eight characteristics of software engineering
Yingxu Wang and Dilip Patel. (2000). Comparative Software Engineering. Available:http://www.google.co.za/url?sa=t&source=web&cd=3&ved=0CCgQFjAC&url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.90.9432%26rep%3Drep1%26type%3Dpdf&rct=j&q=Annals%20of%20Software%20Engineering%20Volume%2010&ei=_3bCTY3gOY3qOcO77J0I&usg=AFQjCNGBj0m2lrlRXWC8sAacoKxfDhKR5g&sig2=ZFf3XupDB8rEIqVpRZKbng&cad=rja[24 April 2011]
8/6/2019 Quality Assurance and Testing - Group 3
7/38
Why is quality assurance required in IT.
Conformance to specification
Meeting customers needs
Change (Industry)
Increased confidence (customers will be kept interested)
8/6/2019 Quality Assurance and Testing - Group 3
8/38
Software development life cycle
Planning Phase Analyze Phase Design Phase Test PhaseImplementation
Phase
Satzinger, et. al. 2004. Systems Analysis & Design in a changing world. Thomson, United states of America.
8/6/2019 Quality Assurance and Testing - Group 3
9/38
Desire Minnaar
8/6/2019 Quality Assurance and Testing - Group 3
10/38
What is Quality .?
IEEE [1990] defines quality as the degree to which a system,component or process meets specified requirements and customer(user) needs (expectations).
conformance to explicitly stated functional and performance
requirements, explicitly documented development standards, andimplicit characteristics that are expected of all professionallydeveloped software.
Petrasch [1999] defines it as the existence ofcharacteristics of aproduct which can be assigned to requirements.
[ISO, 1999] as the totality of characteristics of an entity that bear onits ability to satisfy stated and implied needs,
2 Important key points:
Conformance to specification
Meeting customer needs:
Nasib S. G. ,(2005), Factors affecting effective software quality management revisited., SIGSOFT Softw. Eng. Notes 30, 2 (March 2005), pp.1-4
8/6/2019 Quality Assurance and Testing - Group 3
11/38
McCalls Quality Model (1977)
By Jim McCall, also known asGeneral Electrics Model of 1977
Originates in the US military
Bridges the gap between usersand developers
3 major perspectives for definingand identifying quality of softwareproduct
Product Revision
Product Transition
Product operations
11 Quality Factors (To specify)
23 Quality Criteria (To build)
Metrics (To Control)
Milicic , D. 2005 , Software Quality Attributes and Trade-Offs, Chapter 1 - Software Quality Models and Philosophies, viewed 25 April 2100,http://www.bth.se/tek/besq.nsf/(WebFiles)/5A52350A52726F51C12570A8004CB613/$FILE/Software_quality_attributes.pdf
Rafa E. Al-Qutaish, PhD ,(2001), Quality Models in Software Engineering Literature : An Analytical and Comparative Study, Journal ofAmerican Science ,Vol 6 Iss. 3, pp. 166-175
8/6/2019 Quality Assurance and Testing - Group 3
12/38
McCalls Quality Model (1977)
Rafa E. Al-Qutaish, PhD ,(2001), Quality Models in Software Engineering Literature: An Analytical and Comparative Study, Journal ofAmerican Science , Vol 6Iss. 3, pp. 166-175
8/6/2019 Quality Assurance and Testing - Group 3
13/38
Boehms Quality Model (1978) By Boehm to automatically and
quantitatively evaluate the qualityof software
Similar to McCall Quality Model:hierarchical quality model structure
Consists of
3 High-level characteristicsAddress the main questions of thesoftware buyer
7 Intermediate-level
Represent the qualities expected
15 Lowest-level (primitive)
Primitive characteristics providefoundation for defining qualitiesmetrics
Milicic , D. 2005 , Software Quality Attributes and Trade-Offs, Chapter 1 - Software Quality Models and Philosophies,viewed 25 April 2100,http://www.bth.se/tek/besq.nsf/(WebFiles)/5A52350A52726F51C12570A8004CB613/$FILE/Software_quality_attributes.pdf
Rafa E. Al-Qutaish, PhD ,(2001), Quality Models in Software Engineering Literature: An Analytical and Comparative Study,Journal ofAmerican Science , Vol 6 Iss. 3, pp. 166-175
8/6/2019 Quality Assurance and Testing - Group 3
14/38
ISO 9126
Established 1991 by the International Organization forStandardization
Software Product Evaluation: Quality characteristics & Guidelines forTheir Use-Standard
Based on McCall and Boehm models
Currentversion has 1 International Standard, 3 Technical Reports
ISO IS 9126 -1:Quality Model [ISO,1991]
ISO TR 9126-2: External Metrics [ISO,2003]
ISO TR 9126-3: Internal Metrics [ISO,2003]
ISO TR 9126-4:Quality in Use Metrics [ISO,2004]
Quality Model contains 2 parts
1. Internal and External quality model :
6 characteristics & 27 sub-characteristics
2. Quality in use model 4 Quality in use characteristics
Rafa E. Al-Qutaish, PhD ,(2001), Quality Models in Software Engineering Literature: An Analytical and Comparative Study, Journal ofAmerican Science, Vol 6 Iss. 3, pp. 166-175
8/6/2019 Quality Assurance and Testing - Group 3
15/38
ISO 9126: Internal External Quality Model &Quality in Use Characteristics
Rafa E. Al-Qutaish, PhD ,(2001), Quality Models in Software Engineering Literature: An Analytical and Comparative Study, Journal ofAmerican Science , Vol 6Iss. 3, pp. 166-175
8/6/2019 Quality Assurance and Testing - Group 3
16/38
ISO 9126: Quality in the Lifecycle [ISO, 2001]
Rafa E. Al-Qutaish, PhD ,(2001), Quality Models in Software Engineering Literature: An Analytical and Comparative Study, Journal ofAmerican Science , Vol 6Iss. 3, pp. 166-175
8/6/2019 Quality Assurance and Testing - Group 3
17/38
Comparisons
Rafa E. Al-Qutaish, PhD ,(2001), Quality Models in Software Engineering Literature: An Analytical and Comparative Study, Journal ofAmerican Science , Vol 6Iss. 3, pp. 166-175
8/6/2019 Quality Assurance and Testing - Group 3
18/38
ISO 9000 family of Standards Quality Management Process
Resource Management Process
Regulatory Research Process Market Research Process
Product Design Process
Purchasing Process
Production Process
Ser vice Provision Process
Product Protection Process Customer NeedsAssessment Process
Customer Communications Process
Internal Communications Process
Document Control Process
Record Keeping Process
Planning Process
Training Process
InternalAudit Process
Management Review Process
Monitoring and Measuring Process
Non conformance Management Proces
Continual Improvement Process
A series of international standards dealing withquality systems that can be used for externalquality assurances purposes
Established 1987
Refers to 3 standards
ISO 9000 : definitions & terminology
ISO 9001 : requirements for certification
ISO 9004 : guidelines for sustainedsuccess
ISO 9001 consists 3 main parts:
Framework, describes organization aspect of a
quality system for software production
Life cycle activities, defines required actions
Supporting activities, describes the minimalrequirements to support production, supply maintenance
of software
The requirements specified are aimed primarily
atachieving customer satisfaction
bypreventing non-conformityat all stages
from design through servicing
Bazzana, G.; Andersen, O.; Jokela, T. (1993), ISO 9126 and ISO 9000: friends or foes?, Software Engineering Standards Symposium, 1993. Proceedings., 1993 , 30 Aug-3 Sep 1993, pp 79 - 88
Milicic , D. 2005 , Software Quality Attributes and Trade-Offs, Chapter 1 - Software Quality Models and Philosophies, viewed 25 April 2100,http://www.bth.se/tek/besq.nsf/(WebFiles)/5A52350A52726F51C12570A8004CB613/$FILE/Software_quality_attributes.pdf
8/6/2019 Quality Assurance and Testing - Group 3
19/38
8 Steps to implement ISO 9001 requirements
Helio Yang, Y. (2001) "Software quality management and ISO 9000 implementation", Industrial Management & Data Systems, Vol. 101 Iss: 7, pp.329 - 338
1. Establish Commitment ( 4-12 weeks)
2. Define program structure ( 4 weeks)
3. Establish foundation elements (24 weeks)
4. Define development methodologies ( 32 weeks)
5. Define major supporting elements
6. Define remaining supporting elements
7. PreliminaryAudit
Audit (3 days)
Correction ( up to 12 weeks)
8. RegistrationAudit
Audit (3 days)
Correction ( up to 8 weeks)
International Organization for standardisation, , ISO 9001 - Quality management systems Requirements,viewed 2 May 2011,http://www.iso.org/iso/survey2009.pdf
8/6/2019 Quality Assurance and Testing - Group 3
20/38
ISO Challenges
The amount ofmoney, time and paperwork required for registration
ISO 9001 promotes specification, control, and procedures ratherthan understanding and improvement
The added cost to certify and then maintain certification may not bejustified if product end users do not require ISO 9000
The standard is seen as especially prone to failure when a companyis interested in certification before quality. The standard is seenas especially prone to failure when a company is interested in
certification before quality Another problem reported is the competition among the numerous
certifying bodies, leading to a softer approach to the defects noticedin the operation of the Quality System of a firm.
Barnes, F , 2000, Good Business Sense Is the Key to Confronting ISO 9000" , Review of Business, 22 March, p1-10, viewed : 03 May 2011,http://www.allbusiness.com/specialty-businesses/713376-1.htmlSeddon, J. ,(1997), "Ten arguments against ISO 9000", Managing Service Quality, Vol. 7 Iss: 4, pp.162 - 16
8/6/2019 Quality Assurance and Testing - Group 3
21/38
Other Software Quality Models & Frameworks
Milicic , D. 2005 , Software Quality Attributes and Trade-Offs, Chapter 1 - Software Quality Models and Philosophies, viewed 25 April 2100,http://www.bth.se/tek/besq.nsf/(WebFiles)/5A52350A52726F51C12570A8004CB613/$FILE/Software_quality_attributes.pdf
Rafa E. Al-Qutaish, PhD ,(2001), Quality Models in Software Engineering Literature: An Analytical and Comparative Study, Journal ofAmerican Science , Vol 6 Iss. 3, pp. 166-175
8/6/2019 Quality Assurance and Testing - Group 3
22/38
Adolf Lamprecht
8/6/2019 Quality Assurance and Testing - Group 3
23/38
Types of testingManual testing
According to Right Hand Technologies: Software testing isperformed to verify that the completed software packagefunctions according to the expectations defined by therequirements/ specifications. The overall objective to not to find everysoftware bug that exists, but to uncover situations that could
negatively impact the customer, usability and/or maintainability.
According to Myers "Software Testing is the process of executing aprogram or system with the intent of finding errors."
Automated testing
Definition (ApTest, the software testing specialists):
Testing using software tools to execute tests without manualintervention. Can be applied in GUI, performance,API, etc. testing.
The use of software to control the execution of tests, the comparisonof actual outcomes to predicted outcomes, the setting up of testpreconditions, and other test control and test reporting functions.
http://www.aptest.com/resources.html#app-func http:www.righthandtech.com/software-testing.php
8/6/2019 Quality Assurance and Testing - Group 3
24/38
When does manual testing make sense?
High per-test or maintenance costs are an indicator that a test shouldbe done manually.
The need for human judgment to assess the correctness of the resultor extensive, ongoing human intervention to keep the test running
Installation, setup, operations, and maintenance like loading ofCDs etc.
Configuration and compatibility.
Error handling and recovery
Localisation
Documentation and help
Usability
Black, R., (2002), Investing in Software Testing: Manual orAutomated? White Paper
8/6/2019 Quality Assurance and Testing - Group 3
25/38
Wild cards (auto & manual testing)
Functional testing can either be done manually or auto
Use cases (user scenarios)
User interface
Date and time handling
Higher per-test case costs and needs for human skills, judgment, andinteraction lean towards manual testing. If you need to repeat testsmany times or reduce the cycle time for test execution, lean towardsautomated testing
Black, R., (2002), Investing in Software Testing: Manual orAutomated? White Paper
8/6/2019 Quality Assurance and Testing - Group 3
26/38
Automated testing
According to Fewster & Graham (2000) Automated Testing is not aseasy as buying a popular test execution tool, recording the manual testsand playing them back whenever you want to.
Fewster, M., (1999), Software Test Automation. Addison-Wesley, 1999
8/6/2019 Quality Assurance and Testing - Group 3
27/38
When does auto testing make sense?
According to Edith et.al (2010) they have between 50000 and 100000test cases at Ericsson per release with a time limit for testing.
Regression and confirmation testing
Monkey (or random). Load, volume, and capacity
Performance and reliability
Structural, especiallyAPI-based unit, component, integration
Edith, S., (2010), Towards fully automated test management for large complex systems, 2010 3rd International Conference on Software Testing,Verification and Validation, IEEE Computer societyBlack, R., (2002), Investing in Software Testing: Manual orAutomated? White Paper
8/6/2019 Quality Assurance and Testing - Group 3
28/38
Reasons why test automation fails
QA people are not programmers smart tests Not enough time and resources allocated for development of
tests inadequate planning
Tests are fragile
T
ests are too complicated and too interdependent Expensive and cost of maintenance too high
Expectations
Lack of usefulness
Learning curve too steep
Poor choices of which parts to automate
Lack of training
http://agilesoftwaretesting.com/?p=56 and http://productdevelop.blogspot.com/2010/07/some-reasons-why-test-automation.html
8/6/2019 Quality Assurance and Testing - Group 3
29/38
Types of auto testing tools
Source testing tools (24) Functional testing tools (32)
Performance testing tools (15)
Java testing tools (27)
Embedded testing tools (7)
Database testing tools (13)
Web testing tools (109)
Bug tracking tools (66)
Test management tools (9)
Communication testing tools (109)
Other (+- 25) thus in total > 300
http://www.aptest.com/resources.html#app-func
8/6/2019 Quality Assurance and Testing - Group 3
30/38
Choosing the right tool according to Fewster(1999)
The tool selection project
The tool selection team
Investigate which commercial tools are suitable for you- this order isv
ery important Start by evaluating your requirements
Identify your constraints
Build or buy?
Identify what is available on the market
Evaluate the shortlisted candidate tools
Make the decision
Fewster, M., (1999), Software Test Automation. Addison-Wesley, 1999
8/6/2019 Quality Assurance and Testing - Group 3
31/38
Major testing tools
Tool Vendor
Loadrunner Mercury / HP
Winnrunner 10 Mercury / HP
Quick Test Professional (QTP) Mercury / HPSilk Test / Silk Performer etc Borland formerly Segue (www.borland.com)
QARun 4.9 Compuware (http://www.compuware.com/qarun.htm)
QAWizard Pro 2011 Seapine Software (www.seapine.com/qawizard.html)
Visual Test Rational Software (developed by MS)
8/6/2019 Quality Assurance and Testing - Group 3
32/38
Lessons learned according to Dustin (1999)
The various tools used throughout the development lifecycle didnot easily integrate
Duplicate information was kept in multiple repositories
The automated testing tool drove the testing effort.
Everyone on the testing team was trying to automate scripts. Elaborate test scripts were developed, duplicating the
development effort.
Automated test script creation was cumbersome.
Training was too late in the process, so test engineers lackedtool knowledge.
Testers resisted the tool
There were expectations of early payback
Dustin, E., (1999), Software Testing & Quality Engineering, Lessons in Test automation, Sept/Oct 1999 (Vol. 1, issue 5).
8/6/2019 Quality Assurance and Testing - Group 3
33/38
Case studies by Dr Dieter Kreuer et.al.(1999)
Ericsson and the cellular network provider Mannesmann Mobilfunk
Both companies seek to improve testing efficiency by means of testautomation
Main focus was on load testing Test platform created by Ericssons and enhanced by Mannesmann
High reuse rate of test cases make auto testing even more attractive
Kreuer, D. (1999) Applying Test Automation to Type Acceptance Testing ofTelecom Networks: A Case Study with CustomerParticipation , 14th IEEE International Conference on Automated Software Engineering (ASE'99)pp. 216-224
8/6/2019 Quality Assurance and Testing - Group 3
34/38
Forecasted development of Mannesmann CSS TAcc effort
Kreuer, D. (1999) Applying Test Automation to Type Acceptance Testing ofTelecom Networks: A Case Study with CustomerParticipation , 14th IEEE International Conference on Automated Software Engineering (ASE'99)pp. 216-224.
8/6/2019 Quality Assurance and Testing - Group 3
35/38
Why is it still so difficult to get quality right inIT?
Testers are ill equipped
Ever more complex software solutions
Testers have to cover as many aspects as possible
Minimum criteria that testers use to judge the readiness of testedsoftware
Any code specific fix can
(a) fix only the problem that was reported
(b) fail to fix the problem
(c) fix the problem but break something that was previouslyworking,
(d) fail to fix the problem and break somethingelse
Whittaker, J.A., What Is Software Testing? And Why Is It So Hard? IEEE Software, January/February 2000, pp. 70-79
8/6/2019 Quality Assurance and Testing - Group 3
36/38
References
Barnes, F , 2000, Good Business Sense Is the Key to Confronting ISO 9000" , Review of Business, 22 March, p1-10, viewed : 03 May 2011, http://www.allbusiness.com/specialty-businesses/713376-1.html
Bazzana, G.; Andersen, O.; Jokela, T. (1993), ISO 9126 and ISO 9000: friends or foes?, Software EngineeringStandards Symposium, 1993. Proceedings., 1993 , 30 Aug-3 Sep 1993, pp 79 88
Black, R., (2002), Investing in Software Testing: Manual orAutomated? White Paper
Clifford S, (2005), So many standards to follow, so little payoff,Inc. Magazine, 1 May,p.1 viewed 03 May 2011http://www.inc.com/magazine/20050501/management.html
Dustin, E., (1999), Software Testing & Quality Engineering, Lessons in Test automation, Sept/Oct 1999 (Vol. 1, issue5).
Edith, S., (2010), Towards fully automated test management for large complex systems, 2010 3rd InternationalConference on Software Testing, Verification and Validation, IEEE Computer society
Fewster, M., (1999), Software Test Automation. Addison-Wesley, 1999
Helio Yang, Y. (2001) "Software quality management and ISO 9000 implementation", Industrial Management & DataSystems, Vol. 101 Iss: 7, pp.329 - 338
http://agilesoftwaretesting.com/?p=56
http://www.aptest.com/resources.html#app-func
http://www.ece.cmu.edu/~koopman/des_s99/sw_testing/
http:www.righthandtech.com/software-testing.php
8/6/2019 Quality Assurance and Testing - Group 3
37/38
Referencehttp:www.stqemagazine.com
International Organization for standardisation, , ISO 9001 - Quality management systems Requirements,viewed 2 may 2011,http://www.iso.org/iso/survey2009.pdf
Milicic , D. 2005 , Software Quality Attributes and Trade-Offs, Chapter 1 - Software Quality Models andPhilosophies, viewed 25 April 2100,http://www.bth.se/tek/besq.nsf/(WebFiles)/5A52350A52726F51C12570A8004CB613/$FILE/Software_quality_attributes.pdf
Naik, S., Tripathy ,P.,(2008), Software Testing and Quality Assurance: Theory and Practice , John Wiley &Sons, INC., Hoboken, New Jersey
Nasib S. G. ,(2005), Factors affecting effective software quality management revisited., SIGSOFT Softw.Eng. Notes 30, 2 (March 2005), pp.1-4
Rafa E. Al-Qutaish, PhD ,(2001), Quality Models in Software Engineering Literature: An Analytical andComparative Study , Journal ofAmerican Science , Vol 6 Iss. 3, pp. 166-175
Seddon, J. ,(1997), "Ten arguments against ISO 9000", Managing Service Quality, Vol. 7 Iss: 4, pp.162 - 16
Kreuer, D. (1999) Applying Test Automation to Type Acceptance Testing ofTelecom Networks: A Case
Study with Customer Participation, 14th IEEE International Conference on Automated Software
Engineering (ASE'99)pp. 216-224.
Myers, G.J., The Art of Software Testing, Second Edition (2004), John Wiley & Sons, Inc., Hoboken,New Jersey
Whittaker, J.A., What Is Software Testing? And Why Is It So Hard? IEEE Software, January/February 2000,pp. 70-79
8/6/2019 Quality Assurance and Testing - Group 3
38/38
References
Barnes, F , 2000, Good Business Sense Is the Key to Confronting ISO 9000" , Review of Business, 22 March, p1-10, viewed : 03 May 2011, http://www.allbusiness.com/specialty-businesses/713376-1.html
Bazzana, G.; Andersen, O.; Jokela, T. (1993), ISO 9126 and ISO 9000: friends or foes?, Software EngineeringStandards Symposium, 1993. Proceedings., 1993 , 30 Aug-3 Sep 1993, pp 79 88
Clifford S, (2005), So many standards to follow, so little payoff,Inc. Magazine, 1 May,p.1 viewed 03 May 2011http://www.inc.com/magazine/20050501/management.html
Helio Yang, Y. (2001) "Software quality management and ISO 9000 implementation", Industrial Management & Data
Systems, Vol. 101 Iss: 7, pp.329 - 338
International Organization for standardisation, , ISO 9001 - Quality management systems Requirements, viewed 2may 2011,http://www.iso.org/iso/survey2009.pdf
Milicic , D. 2005 , Software Quality Attributes and Trade-Offs, Chapter 1 - Software Quality Models andPhilosophies, viewed 25 April 2100,http://www.bth.se/tek/besq.nsf/(WebFiles)/5A52350A52726F51C12570A8004CB613/$FILE/Software_quality_attributes.pdf
Naik, S., Tripathy ,P.,(2008), Software Testing and Quality Assurance: Theory and Practice , John Wiley & Sons,
INC., Hoboken, New Jersey
Nasib S. G. ,(2005), Factors affecting effective software quality management revisited., SIGSOFT Softw. Eng.Notes 30, 2 (March 2005), pp.1-4
Rafa E. Al-Qutaish, PhD ,(2001), Quality Models in Software Engineering Literature: An Analytical and ComparativeStudy, Journal ofAmerican Science , Vol 6 Iss. 3, pp. 166-175
Seddon, J. ,(1997), "Ten arguments against ISO 9000", Managing Service Quality, Vol. 7 Iss: 4, pp.162 - 16