1 Security Architecture and Analysis Software Inspections and Verification Software Testing and Certification.

1

Security Architecture and Analysis

• Software Inspections and Verification

• Software Testing and Certification

2

The Software System Development Process

Customerrequirements

Usagespecification

Architecture Defn(COTS, legacy), Increment Plng

Design andVerification

Testing and Certification

Functionspecification

Deployment and operation

Incremental Development

Highly Incremental Project team

3

Software Inspections

• Organization– Team-based activity– Specific roles and steps– Training required

• Management view– Proven path to quality improvement– A clear “exit criteria”– Inspect all work products: specifications, architecture,

designs, implementations, ...– Measure and manage results

4

Correctness Verification

• The next step beyond ordinary inspections

– Based on theory, but has practical application

– Can do verbal proofs in “verification inspections”

– Can do written proofs for more rigor

5

Programming Fundamentals

• All sequential programs, no matter what language or subject matter, can be represented in just three basic structures:– Sequence: do-end– Alternation: if-then-else-end – Iteration: while-do-end

• We will explore verification “look and feel” for sequence structures only, using the Trace Table method

6

An Example Sequence Structure

• In text form:

do

x := x + 1

y := y - x

x := x + y

end

• In flowchart form:

x := x - 1 y := y - x x := x + y

7

Verifying Long Division

• Division:

16

232 3712

232

1392

1392

0• Verification:

16 x 232 = 3712

Verification is by a different process than the one that produced the original answer

8

Software Correctness Verification Basics

• Verification is accomplished by comparing:

– What a program is supposed to do to variables -- this is its specification (called its intended function)

– What a program actually does to variables -- this is its net effect from beginning to end (called its program function)

• If the intended function and the program function are identical, the program is correct

9

The Verification Process

Write the IntendedFunction

Write the Program

ProgramFunction

Step 2:Compare forequivalence

Development Verification

(What the programactually does)

(What the programis supposed to do;its specification)

Step 1:Derive

10

Defining Intended Functions

• Concurrent assignment statements can be used to define intended functions

Values of expressions on the right are simultaneously assigned to corresponding variables on the left (old values on the right, new values on the left)

• Example:

[a, b, c := a + 1, a + b, b - c] means:

a := a + 1

b := a + b Assignments are simultaneous

c := b - c

11

Concurrent Assignments

• [x, y, z := x + a, y - x, a + y] means?

12

Correctness Verification Steps

• Given a program and its intended function:

• Step 1: Derive the program function

– It is the net effect of the program on the variables to which it assigns new values

– “Net effect” mean the change from initial values at the start of the program to final values at the end of the program

• Step 2: Compare the program function to the intended function for equivalence (or not)

13

A Miniature Program -- Is it Correct?

• Intended function (all variables are integers):

[x, y, t := y, x, x]

• Program:

do

t := x

x := y

y := t

end

14

Trace Table Derivation of Program Functions

• Trace Table construction

– Define a row for each assignment, and a column for each variable whose value is changed

– Mechanically fill in the variable changes in each row

• Append initial values in the first row with a “0” suffix

• Increment the suffix with each successive row

• Trace Table Derivation

– Start with last row and mechanically substitute values until you get to the “0” suffixes from first row

– Final expression is the program function

15

Verifying the Miniature Program

Program t x y

1 t := x t1 := x0 x1 = x0 y1 := y0 2 x := y t2 := t1 x2 = y1 y2 := y1 3 y := t t3 := t2 x3 = x2 y3 := t2

Derivations:

t3 = t2 x3 = x2 y3 = t2 = t1 = y1 = t1 = x0 = y0 = x0 Derived program function:

[x3, y3, t3 := y0, x0, x0], or simply [x, y, t := y, x, x]

Therefore, the program computes the intended function’s exchange of the values of x and y, and the setting of t to the value of x.

Trace Table to derive the program function:

16

Is This Program Correct?

• Intended function:

[x, y := y, x]

• Program:

do

x := x + y

y := x - y

x := x - y

end

17

The Trace Table Proof

Program x y

1 x := x + y 2 y := x - y 3 x := x - y

Trace Table:

Derivation:

x3 = y3 =

Derived program function:

18

Is This Program Correct?

• Intended function:

[x, y := y, y + x - 1]

• Program:

do

x := x + 1

y := y - x

x := x + y

end

19

The Trace Table Proof

20

Verifying Branching Logic

[(x > y --> x, y := y, x) | (x <= y --> x, y := x + 1, y - 1)]

if

x > y

then

do [x, y := y, x]

x := x + y

y := x - y

x := x - y

end

else

do [x, y := x + 1, y - 1]

x := x + 1

y := x - 1

end

endif

21

Verification Perspective

• Other sources of errors– Human fallibility in verification – Compiler errors– Support software errors– Hardware errors

• Verification, combined with other rigorous software engineering practices, can produce software of extremely high quality

22

Testing and Certification

• You are the director of quality control for a light bulb factory

• 100,000 light bulbs come off the assembly line every hour:

– 50,000 60w bulbs

– 30,000 75w bulbs

– 15,000 100w bulbs

– 5,000 250w bulbs

• (This is also the distribution of customer purchases)

23

Define a Testing Process for Quality Control

• How will you predict customer quality experience with the light bulbs?

24

A Light Bulb Testing Process

• Sample the population of bulbs according to customer usage, then the quality of the sample will predict customer quality experience with the bulbs in the population that were not tested

• The sample composition should be:– 60W: 50%– 75w: 30%– 100w: 15%– 250w: 5%

• Test the sample to determine MTTF• Compare results to quality objectives

25

Software Testing Realities

• A software system is capable of a virtually infinite number of possible executions (each is a potential test case)

• No testing process, no matter how thorough, can exercise more than a minute fraction of these possible executions

• All testing is sampling; the only question is how to define the sample

26

Software Testing Methods -- 1

• Test to debug the code (40 years of widespread use)

– Objective of most testing today, but can result in buggy systems

• Testing can show the presence of bugs, but never their absence

– Often called coverage testing

– Test cases are often invented based on knowledge of code internals (this is the sample)

– Produces only anecdotal evidence of quality (“The code ran these 100 tests ok”)

– Not useful for predictions about all the executions not tested

27

Software Testing Methods -- 2

• Test to certify the code (10 years of minimal use)– Objective is not to debug, but to certify quality– Called statistical, usage-based testing – Test cases are based on the external usage that

actual users engage in– If sample the population of executions according to

usage, the results can predict user experience with all the possible executions that were not tested.

– Produces scientific measures of code quality

28

Statistical Usage-Based Testing

Develop test cases for thesample

Define a statistical sample of the population based on how the systemwill be used

Execute thetest cases, compute MTTF

Population ofpossibleexecutions

Compare toquality goals

Improve the production process as necessary

29

Testing According to Usage

• Example: Usage distribution for a banking system defines the test case distribution

– Account updates: 60% of usage and tests

– Fund transfers: 25%

– Statement generation: 10%

– Audit checks: 5%

30

Lessons

• To obtain valid estimates of software quality– Test the system based on anticipated usage by

actual users

• Usage scenarios play a major role in modern testing methods

• If a system’s usage environment changes, it will require usage-based testing for the new environment