Measurements &milestones for monitoring and controlling

• All process should have metrics.

• Metrics in testing domain – track test progress ,evaluate quality , test effectiveness & when to stop testing .

Meaningful metrics begin with questions –

Which measure to collect? Purpose ? Who collects them ? Forms & Tools used ? Who analyses the data ? Who has the access to report ?

Begin with a test plan . Work done is compared with the work

planned To support monitoring , Test plans

should contain Testing milestones .

Milestones are tangible events that are expected to occur at a certain time in the project’s lifetime.

Gives the status of the project .

• Test Milestones can be used to monitor the progress of the testing efforts associated with a software project .

• Guideposts or Goals that needs to be met .

• Test Manager uses current testing effort data to find how close the milestone achievement is

• Milestone usually appear in the Scheduling Component of the test plan.

• Different levels of testing have different

milestones .

Completion of the master test plan ?

Completion of Branch Coverage ? ( Unit test )

Execution of all planned system tests ?

Completion of Test summary Report ?

Every milestone has to be scheduled for completion during a certain period of time

A group of test members are assigned for achieving the milestone on time and allotted budget .

Test planners need to be sure that milestone are plausible & meaningful .

Milestones should be descripted in simple terms and ambiguities should be avoided.

Monitoring Data are collected that relates to the current status of the testing cycle .

• Graphs Can be used to show the trends or the phases of the monitoring process over a limited period of time .

• The graphs can be in form of bar charts and can be represented for days , weeks or months based on the scalability of the test to be done .

Project & test leaders present up-to-date measurements , graphs and plots showing the testing efforts.

Testing milestones achieved and those which are not .

Test logs , test incident reports & other test-related documents

Managers question the team .

Testing Status

Tester Productivity

Testing costs

Errors , Faults & Failures .

Identifying the current state of the testing process .

Manager needs to determine if the testing tasks are being completed on time and within budget .

• Which tasks are on time ?

• Which are behind Schedule , and by how much ?

• Have the scheduled milestones for this date been met ?

Test Mangers measures how much coverage has been actually achieved and how much is left .

Compare it with the planned coverage.

• White box testing – Degree of statement , branch , data flow , basis path ( planned , actual )

• Actual / planned degree of coverage

Black Box testing – Number of requirements to be tested , equivalence classes identified , equivalence classes actually covered

No of features actually covered/ total no of features to be covered .

To monitor the progress of test case development , and can be applied to all levels of testing .

Estimated Test cases- No of planned test cases

Test cases complete – No of available test cases .

Change in requirements – Unplanned test cases .

• Test manager will want to determine if the execution process is going according to plan .

• Appropriate measures No of available test cases executed No of available test cases exe & passed No of unplanned test cases No of unplanned test cases exe and

passed .

For new release where there can be regression testing

No of planned regression tests exe No of planned regression test exe and

planned

Imp that Test managers To monitor progress of the development of the test harness code needed for unit and integration test – Timely manner

Size – applied by manager to help estimate the amt of effort needed to develop a s/w pgm .

• Size can be measured in different ways Lines of code Function points Feature points

• Commonly used is Loc

Monitor the test harness development effort over time – Available LOC / planned LOC .

Productivity of the staff and how it changes the Project progresses.

Measuring productivity in s/w domain is difficult task .

Since developers are involved in many activities , many are complex

The existing method of measuring productivity of developers and testers are not credible .

Productivity of tester can be measured using a combination of

No of test cases developed /unit time No of test cases exe No of Loc Test Harness developed /unit

time No of defects detected in unit time.

Good Technique – budget and resource monitoring is called earned value tracking .

To calculate Planned earned value Total estimated time of budget for

overall testing effort Estimated time or budget for each

testing task

• Earned value can be calculated separately for each level of testing

This will help in monitoring the budget and resource usage for individual testing phase .

Actual time for each testing task . Earned value for testing task to date .

Useful for Evaluating product quality

Evaluating testing effectiveness

Making stop-test decisions .

Defect casual analysis

Defect prevention

Test process improvement

Development process improvement .

• Total number of incident reports.

• Number of incident reports resolved / unresolved.

• Number of defects found of each given type.

• Number of failures .

• Allows managers to determine if test resources have been used wisely and productivity to remove defects and evaluate product quality .

• Evaluation is made in several ways ,• Before release – no of defects found

during testing / number of defects estimated .

• After release – no of defects found / 1000 lines of new and changed code

• Defect removal ratio Pre-ship fault density / post- ship fault

density .

• Measurement suggested by Graham

No of defects detected in a given test phase/ total number of defects found in testing.

Detect removal leverage = no of defects found integration test / no of defects found in unit test .