Ch14

Byoungju Choi, EWHA Womans University 2

TMM_SETP_E v2.0

Your Instructor

Rex BlackPresident RBCS, CTO RBCSNZ, CTO PureTesting:

worldwide testing services companies

President ASTQB and ISTQB: non-profit professional organizations

Quarter century of software engineering experience

Author of 5 books and dozens of articles

Married, father of two


TMM_SETP_E v2.0

The Materials

The slides follow the ISTQB Foundation Syllabus 2007

The terms come from the latest ISTQB Glossary

Two mock exams are included to assess your readiness for the ISTQB Foundation exam

Each section includes sample exam questions to check your knowledge as you proceed

Exercises are included for main techniques


TMM_SETP_E v2.0

ISTQB Foundation Syllabus 2007

Developed by a team of nine authors spanning seven countries: Thomas Müller (chair), Rex Black, Sigrid Eldh, Debra Friedenberg, Dorothy Graham, Kalus Olsen, Maaret Pyhäjärvi, Geoff Thompson and Erik van Veenendaal

Forty primary reviewers spanning nine countries

Final review and approval by 33 National Boards

Distills over 1,000 person-years of experience

The ISTQB and the authors are the source of the syllabus (copyright © 2005 and 2007 by the authors and ISTQB) which is used by permission as the basis for this course


TMM_SETP_E v2.0

The Main Topics Covered

1. Fundamentals of testing

2. Testing throughout the software lifecycle

3. Static techniques

4. Test design techniques

5. Test management

6. Tool support for testing


TMM_SETP_E v2.0

What You’ll Gain from This Course

Understand the foundational best practices of software testing

Adapt testing efforts for the context in which they will occur

Design tests using black-box, white-box, and experienced-based techniques

Build confidence, reduce risk, and find defects in the software you test

Explain software testing objectives and results to non-testers in terms they’ll understand

Identify and successfully introduce appropriate testing tools and processes


TMM_SETP_E v2.0

Preparing for the ISTQB Foundation Exam

Download the Foundation Syllabus and Glossary from www.astqb.org

Read the Syllabus and study Glossary definitions

Take the first mock exam (end of this section), noting carefully areas you need to give special attention

Work through each chapter and section of this course, trying all sample exam questions and exercises, reviewing topics as needed

Take the second mock exam at the end of the courseIf you pass, you should be ready for the real exam

If you fail, go back and review the materials you missed

Contact your National Board about exam options (e.g., www.astqb.org in the United States)


TMM_SETP_E v2.0

Test Engineering FoundationEssential Knowledge for Test Professionals

Mock Exam


TMM_SETP_E v2.0

Let’s See Where You Stand

Starting on the next slide, you’ll find the first mock exam for this course

This exam will measure where you stand on the material covered in the syllabus

Don’t worry if you don’t pass this exam, just pay attention to the questions you miss

You’ll want to pay special attention to these topics


TMM_SETP_E v2.0

Articulate Quizmaker Quiz

Place holder - Mock Exam1

Chapter 1 : Fundamentals of Testing


TMM_SETP_E v2.0


Chapter 1:

Foundation of Testing


TMM_SETP_E v2.0

1.Fundamentals of Testing

1. Why is testing necessary?

2. What is testing?

3. General testing principles

4. Fundamental test process

5. The psychology of testing


TMM_SETP_E v2.0


Chapter 1:

Fundamentals of Testing

Section 1:

Why is testing necessary?

Part 1


TMM_SETP_E v2.0

Why is Testing Necessary?

Key conceptsHow bugs can cause harm

Bugs and their effects

The necessity of testing

The role of testing in quality assurance

Terms to remember


TMM_SETP_E v2.0

The Threat of Bugs

CompanyDamaged reputation for quality

High or unpredictable maintenance costs

Unexpected delays in release cycles

Lack of confidence in system

Lawsuits

EnvironmentPollution

Waste

People, societies, and statesLost jobs

Lost lives

Lost rights

Lost missions

Lost wars


TMM_SETP_E v2.0

Where Bugs Come From and What Bugs Do

People make errors which put bugs (defects) into the systemRequirements and design specifications

Code (business logic and user interface)

Documentation (electronic and hard copy)

When defective code is executed, failures occur

If there failures are visible to customers, users, or other stakeholders, dissatisfaction with system quality may result


TMM_SETP_E v2.0


Chapter 1:


Section 1:


Part 1: Sample exam questions


TMM_SETP_E v2.0


Placeholder – Ch1S1P1


TMM_SETP_E v2.0


Chapter 1:


Section 1:


Part 2


TMM_SETP_E v2.0

What Does “Quality” Mean to You?

“Fitness for use” vs. “Conformance to requirements”

Testing and qualityTests give confidence where they find few bugs

Passing tests reduce the level of quality risk

Failing tests provide a chance to improve quality

The test set gives an assessment of quality

What are the important quality characteristics for your system? Are you testing them (enough)?

Testing, quality assurance, and quality improvementIdeally, testing is part of a larger quality assurance strategy for a project

For future projects, analyze the root causes of defects found on current projects and take steps to reduce their incidence


TMM_SETP_E v2.0


Chapter 1:


Section 1:




TMM_SETP_E v2.0




TMM_SETP_E v2.0


Chapter 1:


Section 2:

What is testing?

Part 1


TMM_SETP_E v2.0

What is Testing?

Key conceptsCommon objectives of testing

The purpose of testing…• in software development, maintenance, and operations…

• …to find defects, provide confidence and information, and prevent defects

Terms to remember


TMM_SETP_E v2.0

Testing Objectives

Typical general testing objectivesFind bugs and provide programmers with the information they need to fix important bugs

Gain confidence about the level of quality of the system

Prevent defects (through early involvement in reviews and advanced test design)

Provide information about the most important aspects of the quality of the system under test

Help management understand system quality

Can you think of others for your projects?

Do you tent to align your plans and actions with the testing objectives set by management?


TMM_SETP_E v2.0

Test Phases and Objectives

Unit/Component Test: Find bugs in the individual pieces of the system under test before the pieces are fully integrated into the system

Integration/String Test: Find bugs in the relationships and interfaces between pairs and groups of components in the system under test as the pieces come together

System Test: Find bugs in the overall and particular behaviors, functions, and responses of the system under test as a whole

Acceptance/Pilot Test: Demonstrate that the product is ready for deployment/release or to assess quality and give information on the risk of deployment/release

Maintenance Test: Check for defects introduced during development of the changes

Operational Test: Assess non-functional system characteristics such as reliability or availability usually in the operational environment


TMM_SETP_E v2.0

Effectiveness and Efficiency

Effective: Producing a decided, decisive, or desired result; impressive

To be effective testers, we must select the appropriate objective and desired results

Efficient: Productive of desired effects;

especially productive without waste

To be efficient testers, we must allocate resources (time and money)

These terms are most meaningful in the context of the entire development or maintenance process, not just the testing process


TMM_SETP_E v2.0


Chapter 1:


Section 2:

What is testing?



TMM_SETP_E v2.0




TMM_SETP_E v2.0


Chapter 1:


Section 2:

What is testing?

Part 2


TMM_SETP_E v2.0

Testing vs. Debugging

Testing finds failures that are caused by bugs

Debugging……identifies the root cause of a bug…

…repairs the code…

…and checks that the defect is fixed correctly

Confirmation testing ensures the fix resolves the observed failure

Different responsibilities:Testers test

Programmers debug


TMM_SETP_E v2.0

Find-Debug-Confirm

Testing Debugging

Developer

Tester

Tester

Test Team

Bug Report

Clear handoffs and

Division of Labor

Through Bug Reporting

And Test Release Process

Bug Fix

Development Team

7.Is the problem fixed? Does the system now pass the same test it failed before? Does the test of the system still behave properly?

1. Can I reproduce the failure? 2. Does the failure indicate a

test bug or a system bug?3. What factors influence the failure?

4. What is the root cause of the failure?5. How can I repair the defect without introducing new problems?6. Is my fix properly debugged?


TMM_SETP_E v2.0

Beyond Test Execution

Testing is not just running tests against a running system

Other test activities, before and after execution, include…Planning and control

Choosing test conditions

Design test cases

Checking test results

Evaluating exit criteria

Test result reporting

Closure/ end-of-test tasks

We will revisit this later


TMM_SETP_E v2.0


Chapter 1:


Section 2:

What is testing?



TMM_SETP_E v2.0




TMM_SETP_E v2.0


Chapter 1:


Section 3:

General testing principles

Part 1


TMM_SETP_E v2.0

General Testing Principles

Key conceptsTesting reveals the presence of bugs

Impossibility of exhaustive testing

Benefits of early testing

Lumpiness of bugs: defect clustering

Pesticide paradox

Testing should adapt to specific needs

Absence-of-errors fallacy

Terms to remember


TMM_SETP_E v2.0

Testing Reveals Presence of Bugs: A Parable

You have a beautiful vegetable garden, but one day you see eaten leaves on the tomatoes

“Oh no,” you think, “I have hornworms!”

You know you have bugs in your garden

If you had not seen the symptoms, could you be sure you had no bugs?

Some bugs are easy to spot, others aren’t

Testing can reveal the presence of bugs, but cannot prove their absence


TMM_SETP_E v2.0

Mission Impossible: Exhaustive Testing

“Just make sure the software works before we ship it…”

This charter is demonstrably impossibleThe execution paths in non-trivial software are almost infinite

Large dataflows separated across space (features) and time (static data)

Slight changes can cause regressions which are not linear to the size of the change

Myriad usage profiles and field configurations, some unknown and some unknowable

Bottom line: Exhaustive testing (all combinations of inputs and preconditions) is not possible


TMM_SETP_E v2.0

Defusing Exhaustive Testing Expectations

Exhaustive testing as a way to prove the software works is a common (mis)expectation

Bad expectations create problems for test professionals and test teams

Unachievable high demands on test group

Perception of incompetence when these demands aren’t met

Testers must be ready to communicate (in words the project stakeholders will understand) how testing can contribute


TMM_SETP_E v2.0


Chapter 1:


Section 3:




TMM_SETP_E v2.0




TMM_SETP_E v2.0


Chapter 1:


Section 3:


Part 2


TMM_SETP_E v2.0

Defect Clustering

Studies have long shown and continue to show the unequal distribution of bugs

MVS: 38% of field bugs in 4% of modules

IMS: 57% of field bugs in 7% of modules

Capers Jones reports that the excessive presence of error-prone modules causes a 50% reduction of productivity in software maintenance


TMM_SETP_E v2.0

Clustering of Bugs by Component

586

454

413

204

109 97 94

14028%

50%

69%

79%84%

89%93%

100%

OSS Browser E-Mail WMAL Address Book CCOM UI Design Other

Bug

Cou

nt

Cum

ulat

ive

%

Five components with bug counts of less that 60 each.


TMM_SETP_E v2.0

Benefits of Early QA and Testing

The cost of a bug tends to increase as the project continues

Most of the costs associated with pre-release bugs tend to be associated with the effort required to remove them, so the higher cost means longer schedules

The more bugs enter a quality assurance or test activity, the more bugs will escape from that activity


TMM_SETP_E v2.0

그림


TMM_SETP_E v2.0


Chapter 1:


Section 3:


Part 2: Cost of quality questions


TMM_SETP_E v2.0




TMM_SETP_E v2.0


Chapter 1:


Section 3:


Part 3


TMM_SETP_E v2.0

Pesticide Paradox

Return to your vegetable garden

Your spray pesticide on your garden, and the hornworms die, but the pesticide is not effective against all bugs

Just as pesticides become less effective, so do tests

Functional tests can’t find performance bugs

Try new test techniques―if the objective is to find bugs


TMM_SETP_E v2.02.50

0.65 0.6 0.60 0.580.52 0.51

0.43

0.380.28

0.11

0.25

Weig

hte

d B

ugs/

Hour

2.00

0.44

0.30 0.27 0.24 0.22 0.21 0.19 0.17 0.15

Weig

hte

d B

ugs/

Hour

These two pictures compare efficiencies of the top 11 bug-finding test suites (out of 27). The top graph shows the first time the set of test suites was run. The bottom graph shows the results for the fifth time that same set of test suites was run. The average test suite efficiency in the first pass, 0.4, is used as the axis crossing point in both graphs.

With one exception, the same test suites are in the top 11 test suites, which shows bug clustering again. However, all test suites are less effective after five executions.


TMM_SETP_E v2.0

Absence-of-Errors Fallacy

Finding and fixing many bugs does not guarantee user, customer, and/or stakeholder satisfaction

Many low-defect products have failed in the market place

Successful projects balance competing forces in terms of features, schedule, budget, and qualtiy


TMM_SETP_E v2.0

Testing Should Adapt to Needs

Different projects, organizations, and products have different testing needs

Best testing practices exist (and are discussed in this course) but you need to tailor them to your project

Failure to adapt the test team and its methods to these needs is a common result of dissolution of test teams


TMM_SETP_E v2.0


Chapter 1:


Section 3:




TMM_SETP_E v2.0




TMM_SETP_E v2.0


Chapter 1:


Section 4:

Fundamental test process

Part 1


TMM_SETP_E v2.0

Fundamental Test Process

Key conceptsPlanning and control

Analysis and design

Implementation and execution

Evaluating test exit criteria and reporting

Test closure activities

Terms to remember


TMM_SETP_E v2.0

ISTQB Fundamental Test Process

You can think of the test process in terms of the following steps

Planning and control

Analysis and design

Implementation and execution

Evaluating test exit criteria and reporting

Test closure activities

These steps may overlap, be concurrent, and/or iterate


TMM_SETP_E v2.0

Planning and Control

Planning

Determine test scope, risks, objectives, strategies

Determine required test resources

Implement the test strategies

Schedule test analysis and design

Schedule implementation, execution and evaluation of tests

Determine the test exit criteria

Control

Measure and analyze results

Monitor and document progress, coverage and test exit criteria

Initiate corrective actions

Make decisions

Many planning and control activities involve obtaining project team and project management agreement, support, and consensus.


TMM_SETP_E v2.0


Chapter 1:


Section 4:




TMM_SETP_E v2.0




TMM_SETP_E v2.0


Chapter 1:


Section 4:


Part 2


TMM_SETP_E v2.0

Analysis and Design

Analysis

Review the test basis (e.g., requirements or design specifications, network/system architecture, quality risks)

Identify and prioritize test conditions, test requirements, or test objectives and required test data based on anlysis of test items (e.g., their behavior, specification, and structure)

Evaluate testability of the requirements and system

Design

Design and prioritize combinations of test data, actions, and expected results

Identify the test data needed for test conditions and cases

Design the test environment

Identify infrastructure, tools


TMM_SETP_E v2.0

Implementation and Execution

Implementation

Develop, implement, and prioritize test cases, create data, write procedures

Create test harnesses, scripts

Organize test suites and sequences of test procedures

Verify the test environment

Execution

Execute test cases (manual or automated)

Log test results, and the versions of the software under test, test tools and the testware

Compare actual and expected results

Report and analyze incidents

Repeat corrected and/or updated tests

Run confirmation and/or regression tests


TMM_SETP_E v2.0

Exit Criteria, Reporting, and Closure

Exit and reporting

Check test logs against the exit criteria in the test plan

Assess if more tests are needed or if the exit criteria specified should be changed

Write a test summary report for stakeholders

Closure

Confirm test deliverables, final resolution or deferral of bug reports, and the acceptance of the system

Finalize and archive testware, test environment and test infrastructure

Deliver testware to the maintenance organization

Perform a retrospective to capture improvements for future releases, projects, and test processes


TMM_SETP_E v2.0


Chapter 1:


Section 4:




TMM_SETP_E v2.0




TMM_SETP_E v2.0


Chapter 1:


Section 5:

The psychology of testing

Part 1


TMM_SETP_E v2.0

Psychology of Testing

Key conceptsPsychological factors for testing success

Clear objectives

Self-testing and independent testing

Respectful communication

Programmer and tester outlooks for success

Terms to remember


TMM_SETP_E v2.0

Good Tester Attributes

The attributes of a good tester include:Curiosity

Professional pessimism

A critical eye

Attention to detail

Good communication skills

The proper mix of skills is needed, too


TMM_SETP_E v2.0

Defining Tester Skills

ReadingSpecifications, e-mail, test cases, etc.

WritingTest cases, bug reports, test documentation, etc.

Technology, project, testing skillsTechnology: Programming languages and more, like operating systems, networking, HTML/Web, etc.

Application domain: banking, human factors, office applications, etc.

Testing: scripting, exploring and attacking the system, automation, performance modeling, etc.


TMM_SETP_E v2.0

Differing Mindsets

Testing and reviewing are different activities than developing, and the mindsets are different, too

Developers can test their code, but often have the wrong mindset to do it effectively

Separation of duties (via independent testing) helps focus testing

Professional testers are more effective at testing activities, especially finding failures, as they are more objective and don’t have the author’s bias


TMM_SETP_E v2.0

Degrees of Independence

Though developers can (and should) test their own code, independent testers are typically more effective at finding failures

Levels of test independenceBy test item’s author (low level of independence)

By another person(s) within the same team

By a person(s) from a different team or test specialists

By a person(s) from a different organization or company (high level of independence)


TMM_SETP_E v2.0


Chapter 1:


Section 5:




TMM_SETP_E v2.0




TMM_SETP_E v2.0


Chapter 1:


Section 5:


Part 2


TMM_SETP_E v2.0

Clear Objectives

People and projects are driven by objectives

People usually try to align themselves with objectives set by management and other stakeholders, but often confusion exists about the objectives of testing

For example should testing find defects or confirm that software works?

If it is to find defects, what percentage?

If it is to build confidence, to what level?

A test policy can help to clearly state the objectives of testing


TMM_SETP_E v2.0

Focus

Focus problemsPursuing issues narrow-mindedly, losing sight of more important priorities

Getting distracted from key tasks

Balance and re-evaluate priorities every so often

Stay focused on the goals and objectives of the test project


TMM_SETP_E v2.0

Constructor or Destructor?

Some testers who find failures are perceived as criticizing the product or the author

Sometimes, testing is seen as a destructive activity

Done properly, testing is essential to managing product risks

Project teams must recognize the difference between making a mess, pointing out a mess, and cleaning up a mess…and know who is responsible for each

Testers must communicate about the messes they point out in a constructive way


TMM_SETP_E v2.0

Bad News or Bad Guy?

Testers are sometimes on receiving end of emotions brought on by news of project problems

Communication skills can helpCollaborate for better quality

Communicate neutrally, about facts, without criticism

Understand your colleagues and how they’ll react to your findings

Confirm your colleagues understood what you said

Confirm you understand your colleagues

Schedule pressure can exacerbate this problem


TMM_SETP_E v2.0


Chapter 1:


Section 5:




TMM_SETP_E v2.0




TMM_SETP_E v2.0


Chapter 1:


Cross-section

Sample exam questions


TMM_SETP_E v2.0


Placeholder – Ch1XS


TMM_SETP_E v2.0

Ch14

Business

design of organizations

design people

organization design

organizations product

entire organization

employees job

designing organizations

organizations ability