ppt

Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 1

Black Box Software Testing(Professional Seminar)

Cem Kaner, J.D., Ph.D.Professor of Computer SciencesFlorida Institute of Technology

Section:33Project Planning

Summer, 2002Contact Information:

[email protected] (testing website)

www.badsoftware.com (legal website)

I grant permission to make digital or hard copies of this work for personal or classroom use, with or without fee, provided that (a) copies are not made or distributed for profit or commercial advantage, (b) copies bear this notice and full citation on the first page, and if you distribute the work in portions, the notice and citation must appear on the first page of each portion, (c) each page bear the notice "Copyright (c) Cem Kaner" or if you changed the page, "Adapted from Notes Provided by Cem Kaner". Abstracting with credit is permitted. The proper citation for this work is Cem Kaner, A Course in Black Box Software Testing (Professional Version), Summer-2002, www.testing-education.org. To copy otherwise, to republish or post on servers, or to distribute to lists requires prior specific permission and a fee. Request permission to republish from [email protected].


Planning and Negotiating the Testing Project

• The notes here focus on your process of negotiating resources and quality level.

• Please see Testing Computer Software, Chapter 13 for a detailed discussion of planning and testing tasks across the time line of the project.


Planning the Testing Project

Things you can do very early in the project:• Analyze any requirements documents for testability, ambiguity.• Facilitate inspection and walkthrough meetings.• Prepare a preliminary list of hardware configurations and start

arranging for loaners.• Ask for testing support code, such as debug monitors, memory

meters, support for testpoints.• Ask for a clear error handling message structure.• Discuss the possibility of code coverage measurement (which will

require programmer support).


Planning the Testing Project

Things you can do very early in the project:• Prepare for test automation. This involves reaching agreements in

principle on breadth of automation and automation support staffing level.

• Order automation support equipment.• Order external test suites.• Learn about the product’s market and competition.• Evaluate coding tools that facilitate automation (e.g. test 3rd party

custom controls against MS-Test)


Planning the Testing Project:First Principles

• You can’t nail everything down.• You will face difficult prioritization choices, and many

constraints will be out of your direct control.• You can influence several constraints by opening up your

judgments to other stakeholders in the company. This is more than getting a sign-off.

• Reality is far more important than your ability to cast blame.

• Your task is to manage project-level risks. This includes risks to the project’s cost, schedule, and interpersonal dynamics, as well as to the product’s feature set and reliability.


Planning the Testing Project:Deciding Your Depth of Testing

You have three key objectives:

1 Achieve a uniform and well-understood minimum level of testing.

2 Be explicit about the level of testing for each area:» mainstream» guerrilla» formally planned» structured regression

3 Reach corporate agreement on the level of testing for each area. Just like bug deferrals, depth-of-testing restrictions must rest on sound business decisions.This should not be your decision to make.


Planning the Testing Project:Identify the Tasks

Develop a project task list with your staff. Remember, you are looking for realism in estimates of complexity, difficulty, and time.

• Make the listing and estimation process public, and allow public comment.

• Identify all potential sources of information.• List all main functional areas, and all other cross-functional

approaches that you will take in testing the program.• List every task that appears to need more than 1/2 day.

(Probably you will do this by a group brainstorming session on flipcharts, listing sources on one chart. Gather the sources. List areas and approaches on a few charts. Make one chart for each area of work, and list tasks for each chart. Tentatively assign times to each task, possibly by a museum tour.)


Planning the Testing Project:Estimate the Time

1 Assign time estimates to each task. Invite programmers, marketers, and project managers to walk through the charts and provide their own estimates. Explore the bases of differences. You might have underestimated a task’s complexity. Or you might be seeing a priority disagreement. Or wishful thinking.

2 Make your best-estimate task list. Provide the time needed to achieve formal, planned testing for each area. Include estimates for structured regression for key areas. This number is probably absurdly huge.

3 Add to the list your suggestions for time-cuts to guerrila-level and mainstream-level testing for selected areas.

4 Circulate your lists to all stakeholders, call one or more planning meetings, and reach agreements on the level of testing for each area. Keep cutting tasks and/or adding staff until you reach an achievable result.


Planning the Testing Project:Estimate the Time

Don’t forget:• Budget for meetings and administrative overhead (10-30%).• Budget for bug regression (5-15%).• Budget for down time.• Budget for holidays and vacations.• Never develop a budget that relies on overtime. Leave the overtime free for

discretionary additional work by the individual tester.• Don’t let yourself be bullied or embarrassed, and don’t bully or embarrass

your staff, into making underestimates. To achieve a result, insist on cutting tasks, adding staff, or finding realistic ways to improve productivity.


Planning the Testing Project:Identify Dependencies & Milestones

You can’t start reviewing the manual until you receive it, right? List these dependencies and point out, in every project meeting, what is due to you that is not yet delivered and holding you up.

Try to reach verifiable, realistic definitions for each milestone. For example, if “gamma” is 6 weeks before release, the product can’t have more than 6 weeks of schedule risk at gamma. Negotiate a definition.


Planning the Testing Project:Monitor Progress

• Put the tasks and agreed-to durations on a timeline and review progress of all staff every week.

• When prerequisite materials aren’t provided, find other tasks to do instead. When you can’t meet the schedule, due to absent materials, publish new estimates.

• When design changes invalidate your estimates, publish new estimates.

• If you’re falling consistently behind (e.g. due to underestimated overhead), publish the problem and ask for fewer tasks, more staff, or some other realistic solution.

• If one of your staff has trouble with an area, recognize it and deal

with it.


Planning the Testing Project:Control Late-Stage Issues

• Watch out for late changes. Encourage people to be cautious about late bug fixes, and super-cautious about late design changes.

• Provide interim and final deferred-bug reviews.• Take a final inventory of your testing coverage.• Carry out final integrity tests.• You may have to assess and reporting the reliability of the tested

product.• Plan for post-release processes. Develop a release kit for product

support.• Don’t forget the party.


Notes

______________________________________________________________________________________________________________________ ___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________


Project Planning

Testing Impossible Software Under Impossible Deadlines

Presented at Software Testing Analysis East, 1999


Introduction

I didn’t come up with the idea for this talk--the STAR folk suggested it as something that I might be able to provide a few insights into. After all, this is a chronic complaint among testers. . . .

Well, here goes. How should you deal with a situation in which the software comes to you undocumented on a tight deadline?

I think that the answer depends on the causes of the situation that you’re in. Your best solution might be technical or political or resume-ical. It depends on how you and the company got into this mess. If it is a mess.


What Caused This Situation?

So, let’s start with some questions about causation:• Why is the software undocumented?• Why do you have so little time?• What are the quality issues for this customer?


Why Do You Have So Little Time?

A Few Possibilities

• Time-to-release drives competition• Cash flow drives release date• Key fiscal milestone drives release date• Executive belief that you’ll never be satisfied, so your

schedule input is irrelevant• Executive belief that testing group is out of control and

needs to be controlled by tight budgets• Executive belief that you have the wrong priorities (e.g.

paperwork rather than bugs)• Executive belief that testing is irrelevant• Structural lack of concern about the end customer, or• Maybe you have an appropriate amount of time.


Quality Issues For This Customer?

• In-house, “In-house” outsourced• External, custom• External, large package, packaged• External, mass-market, packaged• What is quality in this market? What are their costs of

failure? » User groups, Software stores, Sales calls, Support calls

Over the long term, a good understanding of quality in the market, and as it affects your company’s costs, will buy you credibility and authority.


Political Approaches: Buying Time

Many of the ideas in this section will help you if you’re dealing with a single project that is out of control.

If your problem is structural (reflects policy or standard practice of the company), then some of these ideas will be counter-productive.


Buying Time: Delaying a Premature Release -2

Take the high road• “I want to see knives in the chests, not in the backs.”

» Trip Hawkins, founding President, Electronic Arts

• Communicate honestly.• Avoid springing surprises on people.• Never sabotage the project.• Don’t become The Adversary: If you are nasty and

personal, you will become a tool, to be used by someone else. And you will be disposable.



• Search for show-stoppers. If you can, dedicate a specialist within your group to this task.

• Circulate deferred bug lists broadly.• Consider writing mid-project and late-in-project

assessments of:» extent of testing (by area)» extent of defects (by area, with predictions about

level of bugs left)» deferred bugs» likely out of box experience or reviewers’

experience



Do regular, effective status reporting. List:• your group’s issues (deliverables due to you, things

slowing or blocking your progress, etc., areas in which you are behind or ahead of plan)

• project issues• bug statistics

Find allies (think in terms of quality-related costs)


Signing Off on the Release?

• Don’t sign off.• Don’t accept the authority to sign off.• Don’t pretend to be “QA” when you (obviously) are not.• Position yourself as a technical information provider.

» Publish pre-release reports that provide data on the stability of the product and the extent of completed testing

» Publish a report that lays out the likely issues that will be raised by magazine reviewers (or other third parities) when they see the product

» Let the people who want to ship prematurely own their responsibility. Your responsibility is to provide them with the information they need to make that decision.


Technical Approaches

• Find reference docs (you can get lots of data, even if you can’t get specs.) See the notes on this in the section on Spec-Driven testing

• Re-use materials across projects.• Plan for exploratory testing.• Do cost-effective automation.• Use tools to find bugs faster

» web page syntax checkers, spiders, etc.

• Facilitate inspections » (Get those bugs out before they reach you.)

• Cover the obvious legal issues.


Reusable Test Matrix


Group Management Approaches

Can you add head count?• Will they let you?• Can you absorb them?

» Do a work flow analysis to figure out what parts of each task could be split off and delegated to a newcomer.

Stay organized• Use functional outlines• Or use a detailed project plan• Create and publish explicit coverage reports

Get critical processes under control• Example: release management


Notes

______________________________________________________________________________________________________________________ ___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

___________________________________________________________

ppt

Documents

executive

structured

dont forget

premature

cem kaner

testing project

cutting tasks

buying time