But Did You Test It

But-- Did You Test It?

Ruth BlakelyQuality Assurance Coordinator

Athabasca University Canada

Today we will talk testing!

Discussion of what ‘Quality Assurance’ is versus software testing

Talk about planning Suggest how to build a test matrix Suggest some resources Answer questions

What is Quality Assurance

Testing is part of Quality Assurance Wikipedia describes Quality Assurance (QA)

as covering all activities from design..., development, production, installation, servicing and documentation

QA in every part of PDCA (Plan-Do-Check-Act)

QA versus Testing

Software testing is the process used to help identify the correctness, completeness, security, and quality of developed computer software.

The word testing is connoted to mean the dynamic analysis of the product—putting the product through its paces

‘Testing is for wimps’

Some developers will assume their code works every time (many developers are not trained in test design or techniques)

NOT testing is risky behaviour Confidential student and university information is put

at risk without adequate testing Need to justify testing? Reductions in the cost of

software by using quality range from a whopping seventy percent of the total production cost to a minimum of twenty to thirty percent*

*R. Black, Managing the Testing Process, Second Edition. Wiley, NY.

Building a Test Plan

Whether your organization does full fledged QA or just provides software testing, a plan is good practice

A QA plan is more than just matrix containing your test scenarios

It evaluates need and assesses risk

Test Plan Prep

Determine the objectives for testing Determine the types of tests that will be

executed. Determine the level of automation. What, if

anything, will be automated? Determine the test definition list

Build a Matrix

A test matrix (at AU) is a collection of use cases (test cases, user actions) that will be executed to verify correctness.

In the case of Moodle the wide variety of roles can multiply the required tests into an unreasonably large matrix.

We use simple software tools to help focus testing and minimize the number of tests

Take the output and assign priority (with the user community).

Using Tools to Focus Testing

Using combinatorial testing tools such as PICT or Allpairs, you can limit the required number of tests

Combinatorial tools use orthogonal arrays (aka pair-wise) to identify the ‘must tests’

With 10 roles you would have to execute each of the 11 actions illustrated in the example 10 different times – do the math

And still more prep…

Determine the priority of each item in the test definition list

Determine the tools and resources needed for testing.

Determine the schedule for testing.

How to Set Priority

User Impact (low to high)1 – user can work without this function 2 - user has a workaround but it would be noticeable3 - user would be severely impacted by loss of this function

Frequency of use (low to high)1 – executed relatively infrequently2 - executed moderately frequently3 - executed nearly all of the time

Development risk (low to high)1 - known to be relatively stable 2 - average stability from development perspective

3 - suspected or known to be unstable- error prone in past - based on changing or new technology – logic is complicated

Regression

The purpose of a regression test is to verify that: Failure fixes have been made since last build Related functionality that could be affected by

fixes or modifications still exists Critical functionality of software is covered This may be the most appropriate place for

automation (i.e. jMeter, Selenium)

Tools, Resources and Set Up

What tools are required? Servers, pc’s, operating systems, special

software

What resources?

Developers, primary tester, user acceptance team, documentation specialist

Bug Tracking

Use your test matrix or a bug tracking tool to follow reports, developer updates and retest results

AU uses Bugzilla as our test tracking tool

Determine bug flow before the start of the project

Back to The Plan

Clearly state the objectives of your testing Each project will have a different set of

requirements as to what is an acceptable level of testing.

Articulate the scope of testing types of tests level of automation if applicable

List the process you will follow for testing

Sometimes Forgotten

Determine ahead of time who will record failures

Set guidelines for reporting failures Train your testers to write good bug reports or

set up a system so they can fill in the blanks Determine the procedures once failures are

logged. (i.e. will it be fixed in this release, the next release, not at all)

Plan to retest

What Programmers say to testers

"That's weird..." "It worked yesterday." "It must be a hardware

problem." "Somebody must have

changed my code." "It works on my

machine"

Testing is just PART of QA

Remember from earlier--PDCA Plan a pilot to test the improvement. Do the improvement. Check that the process actually improved. Act to adopt, adjust or abandon the change.

Keys to Success

Good Planning Adequate Resources Realistic Goals Follow your processes Foster teamwork

Places to go for help…

www.sqatesters.com has white papers, online forums, strategy documents

http://www.sqe.com (Software Quality Engineering) provides QA certification training

http://www.softwareqatest.com/ has excellent FAQ All Pairs (which is open source Perl) is available at

www.satisfice.com PICT is freeware

http://msdn.microsoft.com/en-us/testing/bb980925.aspx www.bugzilla.com if you are interested in the tracking

software that we use

Come for a virtual visit…

www.athabascau.ca

Ruth Blakely [email protected]