Algorithm Analysis and Design

Algorithm Analysis and Design (R 606) 1

Department of Computer Science & Engineering SJCET, Palai



ALGORITHM ANALYSIS AND DESIGN

R606 3+1+0

Module 1 Introduction and Complexity

What is an algorithm – Properties of an Algorithm, Difference between Algorithm,

Computational Procedure and Program, Study of Algorithms; Pseudo-code

Conventions; Recursive Algorithms –Space and Time Complexity –Asymptotic

Notations – ‗Oh‘, ‗Omega‘, ‗Theta‘, Common Complexity Functions; Recurrence

Relations and Recurrence Trees for Complexity Calculations; Profiling. –

Deterministic and non - deterministic algorithms.

Module 2 Divide and Conquer

Control Abstraction, Finding Maximum and Minimum, Binary Search, Divide and

Conquer Matrix Multiplication, Stressen‘s Matrix Multiplication, Merge Sort, Quick

Sort.

Module 3 Greedy Strategy

Control Abstraction, General Knapsack Problem, Optimal Storage on Tapes,

Minimum Cost Spanning Trees – Prim‘s Algorithm, Kruskal‘s Algorithm – Job

sequencing with deadlines.

Module 4 Dynamic Programming

Principle of Optimality, Multi-stage Graph, All-Pairs Shortest Paths, Travelling

Salesman Problem.

Lower Bound Theory - Comparison Trees for Searching and Sorting, Oracles and

Adversary Arguments – Merging, Insertion & Selection Sort; Selection of ‗k‘th

Smallest Element.

Module 5 Backtracking

Control Abstraction - Bounding Functions, Control Abstraction, N-Queens Problem,

Sum of Subsets, Knapsack problem.

Branch and Bound Techniques – FIFO, LIFO, and LC Control Abstractions, 15-

puzzle, Travelling Salesman Problem.



Text Book

1. Fundamentals of Computer Algorithms - Horowitz and Sahni, Galgotia

References

1. Computer Algorithms – Introduction to Design and Analysis - Sara Baase & Allen Van

Gelder, Pearson Education

2. Data Structures algorithms and applications - Sahni, Tata McGrHill

3. Foundations of Algorithms - Richard Neapolitan, Kumarss N., DC Hearth & Company

4. Introduction to algorithm- Thomas Coremen, Charles, Ronald Rivest -PHI



TABLE OF CONTENTS

Module 1

What is an algorithm ……………………………………………………… 7

Properties of an Algorithm ……………………………………................ .. 7

Difference between Algorithm Computational

Procedure and Program …………………………………...................... 8

Study of Algorithms........................................................................ 10

Pseudocode convention………………………………………………….. 10

Recursive Algorithms …………………………………........................ 12

Space Complexity ………………………………………….................. 19

Time Complexity............................................................................. 20

Asymptotic Notations ………………………………………………....... 25

Oh……………………………………………...................................... 26

Omega........................................................................................... ......... 28

Theta............................................................................................ .......... 30

Common Complexity Functions……………………………………….... 33

Recurrence Relations....................................................................... 35

Recurrence Trees for Complexity Calculations…................................ 39

Profiling……………………………………………............................... 41

Deterministic and non - deterministic algorithms………………………… 41



Module 2

Control Abstraction ………………………………………………………. 44

Finding Maximum and Minimum ………………………………………... 45

Binary Search ………………………………………………………........ 51

Divide and Conquer Matrix Multiplication …………………................. 54

Stressen‘s Matrix Multiplication ……………………………………….. 56

Merge Sort ……………………………………………………………... 60

Quick Sort …………………………………………………………….... 69

Module 3

Greedy Strategy

Control Abstraction ……………………………………………………… 87

General Knapsack Problem ……………………………………………… 88

Optimal Storage on Tapes ……………………………………………… 90

Minimum Cost Spanning Trees ………………………………………... 91

Prim‘s Algorithm …………………………………………….............. 93

Kruskal‘s Algorithm …………………………………………………... 99

Job sequencing with deadlines……………………………………. 100

Module 4

Dynamic Programming

Principle of Optimality ……………………………………………… 105

Multi-stage Graph ………………………………………………....... 106



All-Pairs Shortest Paths …………………………………………….. 118

Lower Bound Theory

Comparison Trees for Searching and Sorting .................................. 127

Oracles and Adversary Arguments.................................................. 134

Merging…………………............................................................... 135

Insertion………………................................................................... 136

Selection Sort……………............................................................... 142

Selection of ‗k‘th Smallest Element……………………....................... 144

Module 5

Backtracking

Control Abstraction ………………………………………………….... 149

Bounding Functions ………………………………………………....... 153

Control Abstraction …………………………………………………….. 149

N-Queens Problem.......................................................................... 153

Sum of Subsets ……....................................................................... 158

Knapsack problem……………......................................................... 164

Branch and Bound Techniques

FIFO …........................................................................................... 172

LIFO…........................................................................................... ............ 177

LC Control Abstractions..................................................................... 181

15-puzzle….................................................................................... .......... 182

Travelling Salesman Problem…........................................................ 184



MODULE 1



WHAT IS AN ALGORITHM

Definition: An algorithm is a finite set of instructions that, if followed, accomplishes a

particular task.

PROPERTIES OF AN ALGORITHM

All algorithms must satisfy the following criteria:

1. Input. Zero or more quantities are externally supplied.

2. Output. At least one quantity is produced.

3. Definiteness. Each instruction is clear and unambiguous.

4. Finiteness. If we trace out the instructions of an algorithm, then for all cases, the

algorithm terminates after a finite number of steps.

5. Effectiveness. Every instruction must be very basic so that it can be carried out, in

principle, by a person using only pencil and paper. It is not enough that each

operation be definite as in criterion 3; it also must be feasible.

An algorithm is composed of a finite set of steps, each of which may require one or more

operations. The possibility of a computer carrying out these operations necessitates that

certain constraints be placed on the type of operations an algorithm can include.

Criteria 1 and 2 require that an algorithm produce one or more outputs and have zero or more

inputs that are externally supplied. According to criteria 3, each operation must be definite,

meaning that it must be perfectly clear what should be done.

The fourth criterion for algorithms is that they terminate after a finite number of operations.

A related consideration is that the time for termination should be reasonably short.

Criteria 5 requires that each operation be effective; each step must be such that it can, at least

in principle, be done by a person using pencil and paper in a finite amount of time.

Performing arithmetic on integers is an example of an effective operation, but arithmetic with

real numbers is not, since some values may be expressible only by infinitely long decimal

expansion.



DIFFERENCE BETWEEN ALGORITHM, COMPUTATIONAL

PROCEDURE AND PROGRAM

COMPUTATIONAL PROCEDURE

Algorithms that are definite and effective are also called computational procedures. One

important example of computational procedures is the operating system of a digital computer.

This procedure is designed to control the execution of jobs, in such a way that when no jobs

are available, it does not terminate but continues in a waiting state until a new job is entered.

PROGRAM

To help us achieve the criterion of definiteness, algorithms are written in a

programming language. Such languages are designed so that each legitimate sentence has a

unique meaning. A program is the expression of an algorithm in a programming language.

Sometimes words such as procedure, function and subroutine are used synonymously for

program.

The study of algorithms includes many important and active areas of research. There

are four distinct areas of study:

1. How to devise algorithms: - Creating an algorithm is an art which may never

be fully automated. There are several techniques with which you can devise

new and useful algorithms. Dynamic programming is one such technique.

Some of the techniques are especially useful in fields other than computer

science such as operations research and electrical engineering.

2. How to validate algorithms: - Once an algorithm is devised, it is necessary to

show that it computes the correct answer for all possible legal inputs. This

process is referred to as algorithm validation. It is sufficient to state the

algorithm in any precise way and need not be expressed as a program. The

purpose of validation is to assure us that this algorithm will work correctly

independently of the issues concerning the programming language it will

eventually be written in. once the validity of the method has been shown, a

program can be written and a second phase begins. This phase is referred to as

program proving or program verification. A proof of correctness requires that



the solution be stated in two forms. One form is usually as a program which is

annotated by a set of assertions about the input and output variables of the

program. These assertions are often expressed in predicate calculus. The

second form is called a specification, and this may also be expressed in the

predicate calculus. A proof consists of showing that these two forms are

equivalent in that for every given legal input, they describe the same output. A

complete proof of program correctness requires that each statement of the

programming language be precisely defined and all basic operations be proved

correct.

3. How to analyze algorithms: - this field of study is called analysis of

algorithms. As an algorithm is executed, it uses the computer‘s central

processing unit (CPU) to perform operations and its memory (both immediate

and auxiliary) to hold the program and data. Analysis of algorithms or

performance analysis refers to the task of determining how much computing

time and storage an algorithm requires. An important result of this study is that

it allows you to make quantitative judgments about the value of one algorithm

over another. Another result is that it allows you to predict whether the

software will meet any efficiency constraints that exist. Questions such as how

well an algorithm performs in the best case, in the worst case, or on the

average are typical.

4. How to test a program: - testing a program consists of two phases: debugging

and profiling (or performance measurement). Debugging is the process of

executing programs on sample data sets to determine whether faulty results

occur and, if so to correct them. In cases in which we cannot verify the

correctness of output on sample data, the following strategy can be employed:

let more than one programmer develop programs for the same problem, and

compare the outputs produced by these programs. If the outputs match, then

there is a good chance that they are correct. A proof of correctness is much

more valuable than a thousand tests, since it guarantees that the program will

work correctly for all possible inputs. Profiling or performance measurement

is the process of executing a correct program on data sets and measuring the

time and space it takes to compute the results. These timing figures are useful

in that they may confirm a previously done analysis and point out logical

places to perform useful optimization.



STUDY OF ALGORITHM

An algorithm is a finite set of instructions that , if followed , accomplishes a particular task.

The study of algorithms includes many important and active areas of research. There are four

distinct areas of study

1. How to devise algorithms: Creating an algorithm is an art which may never be

fully automated.To study various design techniques that have proven to be useful in

that they have often yielded good algorithms.Dynamic programming is one such

technique.some important design techniques are linear,nonlinear and integer

programming .

2. How to validate algorithms: Once an algorithm is devised , it is necessary to show

that it computes the correct answer for all possible legal inputs.This process is

algorithm validation.The purpose of the validation is to assure us this algorithm will

work correctly independent of the issues concerning the programming language it will

eventually be written in.

3. How to analyze algorithms: This field of study is called analysis of algorithms. As

an algorithm is executed , it uses the computer‘s central processing unit to hold the

program and data.Analysis of algorithms refers to the task of determining how much

computing time and storage an algorithm requires.It allows you to predict whether

the software will meet any efficiency constraints that exits.

4. How to test a program: Testing a program consists of two phases, debugging and

profiling.Debugging is the process of executing programs on sample data sets to

determine whether faulty results occur and , if so, to correct them. Profiling is the

process of executing a correct program on data sets and measuring the time and space

it takes to compute the results.

PSEUDOCODE CONVENTIONS

We can describe an algorithm in many ways.We can use a natural language like

English, although I we select this option,we must make sure that the resulting instructions are

definite.

We can present most of our algorithms using a pseudocode that resembles c

1. Comments begin with // and continueuntill the end of line

Eg: count :=count+1;//count is global ;It is initially zero.



2. Blocks are indicated with matching braces: and .A compound statement

can be represent as a block.The body of a procedure also forms a block.Statements

are delimited by ;

Eg: for j:= 1 to n do

Count:=count+1;

C[i,j]:=a[i,j]+b[i,j];

Count:=count +1;

3. An identifier begins with a letter. The data types of variables are not explicitly

declared. The types will be clear from the context .Whether a variable is global or

local to a procedure will also be evident from the context. Compound data types

can be formed with records.

Eg: node=record

datatype_1 data_1;

:

datatype_n data_n;

node *link;

4. Assignment of values to variables is done using the assignment statement

<variable> := <expression>;

Eg: count:= count+1;

5. There are two Boolean values true and false. In order to produce these values, the

logical operators and , or , and not and the relational relational operators

<,<=,=,!=,>= and > are provided.

Eg: if (j>1) then k:=i-1; else k:=n-1;

6. Elements of multidimentional arrays are accessed using [ and ].

For eg: if A is a two dimentional array , the (I,j) th element of the array is denoted as A[I,j].

Array indicates start at zero.

7. The following looping statements are employed: for,while and repeat until. The

while loop takes the following form.

While (condition) do

<statement 1>



:

:

<statement n>

8. A conditional statement has the following forms:

If < condition > then <statement>

If<condition> then <statement 1> else <statement 2>

Here < condition > is a Boolean expression and <statement>,<statement 1>, and

< statement 2> are arbitrary statements.

9. Input and output are done using the instructions read and write. No format is used

to specify the size of input or output quantities.

Eg: write (―n is even‖);

10. There is only one type of procedure: Algorithm. An algorithm consists of a

heading and a body. The heading takes the form

Algorithm Nmae (<parameter list>)

RECURSIVE ALGORITHMS

A function that calls itself repeatedly, satisfying some condition is called a Recursive

Function. The algorithm that does this is called a recursive algorithm. Using recursion, we

split a complex problem into its single simplest case. The recursive function only knows how

to solve that simplest case.

TYPES OF RECURSION:

Linear Recursion

A linear recursive function is a function that only makes a single call to itself each time the

function runs (as opposed to one that would call itself multiple times during its execution).

The factorial function is a good example of linear recursion. Another example of a linear

recursive function would be one to compute the square root of a number using Newton's

method (assume EPSILON to be a very small number close to 0):

double my_sqrt(double x, double a)



double difference = a*x-x;

if (difference < 0.0) difference = -difference;

if (difference < EPSILON) return(a);

else return(my_sqrt(x,(a+x/a)/2.0));

Tail recursive

Tail recursion is a form of linear recursion. In tail recursion, the recursive call is the last thing

the function does. Often, the value of the recursive call is returned. As such, tail recursive

functions can often be easily implemented in an iterative manner; by taking out the recursive

call and replacing it with a loop, the same effect can generally be achieved. In fact, a good

compiler can recognize tail recursion and convert it to iteration in order to optimize the

performance of the code.

A good example of a tail recursive function is a function to compute the GCD, or Greatest

Common Denominator, of two numbers:

int gcd(int m, int n)

int r;

if (m < n) return gcd(n,m);

r = m%n;

if (r == 0) return(n);

else return(gcd(n,r));

Binary Recursive

Some recursive functions don't just have one call to themself, they have two (or more).

Functions with two recursive calls are referred to as binary recursive functions. The

mathematical combinations operation is a good example of a function that can quickly be



implemented as a binary recursive function. The number of combinations, often represented

as nCk where we are choosing n elements out of a set of k elements, can be implemented as

follows:

int choose(int n, int k)

if (k == 0 || n == k) return(1);

else return(choose(n-1,k) + choose(n-1,k-1));

Exponential recursion

An exponential recursive function is one that, if you were to draw out a representation of all

the function calls, would have an exponential number of calls in relation to the size of the

data set (exponential meaning if there were n elements, there would be O(an) function calls

where a is a positive number). A good example an exponentially recursive function is a

function to compute all the permutations of a data set. Let's write a function to take an array

of n integers and print out every permutation of it.

void print_array(int arr[], int n)

int i;

for(i=0; i<n; i) printf("%d ", arr[i]);

printf("\n");

void print_permutations(int arr[], int n, int i)

int j, swap;

print_array(arr, n);

for(j=i+1; j<n; j)

swap = arr[i]; arr[i] = arr[j]; arr[j] = swap;

print_permutations(arr, n, i+1);

swap = arr[i]; arr[i] = arr[j]; arr[j] = swap;



To run this function on an array arr of length n, we'd do print_permutations(arr, n, 0) where

the 0 tells it to start at the beginning of the array.

Nested Recursion

In nested recursion, one of the arguments to the recursive function is the recursive function

itself! These functions tend to grow extremely fast. A good example is the classic

mathematical function, "Ackerman's function. It grows very quickly (even for small values of

x and y, Ackermann(x,y) is extremely large) and it cannot be computed with only definite

iteration (a completely defined for() loop for example); it requires indefinite iteration

(recursion, for example).

Ackerman's function

int ackerman(int m, int n)

if (m == 0) return(n+1);

else if (n == 0) return(ackerman(m-1,1));

else return(ackerman(m-1,ackerman(m,n-1)));

Mutual Recursion

A recursive function doesn't necessarily need to call itself. Some recursive functions work in

pairs or even larger groups. For example, function A calls function B which calls function C

which in turn calls function A. A simple example of mutual recursion is a set of function to

determine whether an integer is even or odd.

int is_even(unsigned int n)

if (n==0) return 1;

else return(is_odd(n-1));

int is_odd(unsigned int n)

return (!iseven(n));



EXAMPLES OF RECURSIVE ALGORITHMS:

The Towers of Hanoi

The Towers of Hanoi puzzle (TOH) was first posed by a French professor, ´Edouard Lucas,

in 1883. Although commonly sold today as a children‘s toy, it is often discussed in discrete

mathematics or computer science books because it provides a simple example of recursion. In

addition, its analysis is straightforward and it has many variations of varying difficulty.

The object of the Towers of Hanoi problem is to specify the steps required to move the disks

or, as we will sometimes call them, rings) from pole r (r = 1, 2, or 3) to pole s (s = 1, 2, or 3;

s _= r), observing the following rules:

i) Only one disk at a time may be moved.

ii) At no time may a larger disk be on top of a smaller one.

The most common form of the problem has r = 1 and s = 3.

FIGURE

.

The Towers of Hanoi problem

Solution The algorithm to solve this problem exemplifies the recursive paradigm. We

imagine that we know a solution for n − 1 disks (―reduce to a previous case‖), and then we

use this solution to solve the problem for n disks. Thus to move n disks from pole 1 to pole 3,

we would:

1. Move n − 1 disks (the imagined known solution) from pole 1 to pole 2.

However we do this, the nth disk on pole 1 will never be in our way because any valid

sequence of moves with only n −1 disks will still be valid if there is an nth (larger) disk

always sitting at the bottom of pole 1 (why?).

2. Move disk n from pole 1 to pole 3.

3. Use the same method as in Step 1 to move the n −1 disks now on pole 2 to



pole 3.

Algorithm:

Input num [Number of disks]

Pinit [Initial pole; 1 ≤ Pinit ≤ 3]

Pfin [Final pole; 1 ≤ Pfin ≤ 3, Pinit _= Pfin]

Output The sequence of commands to the robot to move the disks from

pole Pinit to pole Pfin

Algorithm Hanoi

procedure H(in n, r, s) [Move n disks from pole r to pole s]

if n = 1 then robot(r → s)

else H(n−1, r, 6−r−s)

robot(r → s)

H(n−1, 6−r−s, s)

endif

endpro

H(num, Pinit, Pfin) [Main algorithm]

ALGORITHM DESIGN TECHNIQUES

For a given problem, there are many ways to solve them. The different methods are listed

below.

1. Divide and Conquer.

2. Greedy Algorithm.

3. Dynamic Programming.

4. Branch and Bound.

5. Backtracking Algorithms.

6. Randomized Algorithm.

Now let us discuss each method briefly.

1. Divide and Conquer



Divide and conquer method consists of three steps.

a. Divide the original problem into a set of sub-problems.

b. Solve every sub-problem individually, recursively.

c. Combine the solutions of the sub-problems into a solution of the whole original

problem.

2. Greedy Approach

Greedy algorithms seek to optimize a function by making choices which

are the best locally but do not look at the global problem. The result is a good solution but not

necessarily the best one. Greedy Algorithm does not always guarantee the optimal solution

however it generally produces solutions that are very close in value to the optimal solution.

3.Dynamic Programming.

Dynamic Programming is a technique for efficient solution. It is a

method of solving problems exhibiting the properties of overlapping sub problems and

optimal sub-structure that takes much less time than other methods.

4. Branch and Bound Algorithm.

In Branch and Bound Algorithm a given Algorithm

which cannot be bounded has to be divided into at least two new restricted sub-problems.

Branch and Bound Algorithm can be slow, however in the worst case they require efforts that

grows exponentially with problem size. But in some cases the methods converge with much

less effort. Branch and Bound Algorithms are methods for global optimization in non-convex

problems.

5.Backtracking Algorithm.

They try each possibility until they find the right one. It is a depth

first search of a set of possible solution. During the search if an alternative doesn‘t work, the

search backtrack to the choice point, the place which presented different alternatives and tries

the next alternative. If there are no more choice points the search fails.

6.Randomized Algorithm.

A Randomized Algorithm is defined as an algorithm that is

allowed to access a source of independent, unbiased, random bits, and it is then allowed to

use these random bits to influence its computation.



ALGORITHMIC COMPLEXITY

The time complexity of an algorithm is given by the number of steps taken by the

algorithm to compute the function it was written for. The number of steps is itself a function

of the instance characteristics. Althogh any specific instance may have several characteristics,

the number of steps computed as a function of some subset of these. Usually, we might wish

to know how the computing time increases as the number of inputs increase. In this case the

number of steps will be computed as a function of the number of inputs alone. For a different

algorithm , we might be interested in determining how the computing time increases as the

magnitude of one of the input increases. In this case the number of steps will be computed as

a function of the magnitude of this input alone. Thus before the step count of an algorithm

can be determined, we need to know which characteristics of the problem instance are to be

used. These define the variables in the expression for the step count. In the case of sum, we

chose to measure the time complexity as a function of the number n of elements being added.

For algorithm Add, the choice of characteristics was the number m of rows and the number n

of columns in the matrices being added.

SPACE COMPLEXITY

Algorithm abc computes a+b+b*c+(a+b-c)/(a+b)+4.0;

Algorithm Sum computes Ei to n = 1 a[i] iteratively, where the a[i]‘s are real numbers; and

RSum is a recursive algorithm that computes Ei n =1 a [i]

The space needed by each of these algorithms is seen to be the sum of the following

components:

1. A fixed part that is independent of the characteristics of the inputs and ouputs . This

part typically includes the instruction space, space for simple variables and fixed-size

component variables, space for constants.

2. A variable part that consists of the space needed by component variables whose size is

dependent on the particular problem instance being solved, the space needed by

reference variables and he recursion stack space.

The space requirement s(p) of any algorithm p may thereore be written as s(p) =c+Sp,

Where c is a constant.



TIME COMPLEXITY

COMPLEXITY OF SIMPLE ALGORITHMS

Time complexity of an algorithm is given by the number of steps taken

by the algorithm to compute the function it was written for. The time T(P) taken by a

program P is the sum of compile time and run time. The compile time does not depends the

instance characteristics. A compiled program will run several times without recompilation.

So we have to concern with the run time of the program which is denoted by tp (instance

characteristics).

If we know the characteristics of the compiler to be used ,we could proceed to determine the

number of additions, subtractions, multiplications, divisions, compares, loads, stores and so

on,that would be made the code for P. So we could obtain an expression for tp (n) of the

form

tp(n)=ca ADD(n) + cs SUB(n) + cm MUL(n) + cd DIV(n)+…….

Where n denotes the instance characteristics, and ca, cs , cm, cd, and so on, respectively, denote

the time needed for an addition, subtraction, multiplication, division ,and so on, and

ADD,SUB,MUL, DIV, and so on, are functions whose values are the numbers of additions,

subtractions, multiplications, divisions, and so on ,that are performed when the code for P is

used on an instance with characteristic n. The value of tp(n) for any n can be obtained only

experimentally.The program is typed,compiled and run on a particular machine.The

execution time is physically blocked,and tp(n) obtained.In a multi user system ,execution time

depends on factors such as system load and number of other programs running on the

computer at the time P is running,the characteristics of these other programs and so on.

A propgram step is loosely defined as a syntactically or semantically meaningful segment of

a program that has an execution time independent of the instance characteristics. For

example,consider the entire stataement

return a+b+b*c+(a+b-c)/(a+b)+4.0 of the program given below



Algorihm abc(a,b,c)

return a+b+b*c+(a+b-c)/(a+b)+4.0

The above line could be considered as a step since its execution time is independent of the the

instance characteristics.The number of steps any program statement is assigned depends on

the kind of statement.For example comments count as zero steps;an assignment statement

which does not involve any calls to other algorithms is counted as one step;in an iterative

statement such as the for ,while,and repeat until statements ,we consider the step count only

for the control part of the the statement.The control parts for for and while statements have

the following forms:

for i=(expr) to (expr1) do

while(expr) do

Each execution of the control part of a while statement is given by a step count equal to the

number of step counts assignable to(expr).The step count for each execution of the

controlpart of a for statement is one,unless the counts attributable to (expr) and (expr1) are

the functions of the instance characteristics. In the latter case,the first execution of the

control part the for has a step count equal to the sum of counts for (expr)

and (expr1).Remaining executions of the for statement have a step count of one;and o on.

We can determine the number of steps needed by a program to solve a

particular program instance in one of the two ways.In the first method we introduce a new

variable,count into the problem.This is a global variable with initial value equals

zero.Statement to increment count by appropriate amount are introduced into the

program.This is done so that each time a statement in the original program is executed,count

is incremented by the step count of that statement.



EXAMPLE FOR TIIME COMPLEXITY CALCULATION

Fibonacci series of numbers starts as 0,1,1,2,3,5,8,13,21,34,55…………..

Each new term is obtained by taking the sum of two previous terms.If we call the first term

of the sequence f0,then f0=0,f1=1,and in general

fn=fn-1+fn-2, n>=2

Algorithm Fibonacci(n)

// Compute the nth

Fibonacci number

if (n<=1) then

Write(n);

else

fnm2:=0;fnm=1;

For I := 1 to n do

fn = fnm1+fnm2;

fnm2 =fnm1;fnm1=fn;

Write(fn)

To analyze the complexity of this algorithm we need to consider two cases (1)

n=0 or 1 and (2) n>1.When n=0 or 1 lines 4and 5 get executed once each.Since each line has

an s/e of 1,total step count for this case is 2.When n>1,lines 4, 8 and 14 are each executed

once.Line 9 gets executed n times ,and lines 11 and 12 get executed n-1 times each.Line 8 has

an s/e of 2,line 12 has an s/e of 2 and line 13 has s/e of 0.The remaining lines that get

executed have s/e‘s of 1 The total steps for the case n>1 is therefore 4n+1.



Example 1 Sum of n numbers

Algorithm with count statements added

Algorithm sum(a,n)

S:=0.0;

Count := count+1;//count is global; it is initially zero.

For i:=1 to n do

Count :=count + 1;//For for

S:= s+a[i];count:=count+1;//For assignment

Count :=count+1;//For last time of for

Count := count+1;//For the return

Return s;

Simplified version of algorithm

Algorithm Sum(a,n)

For i:=1 to n do count:= count+2;

Count:= count+3;

Complexity calculation

tRSum(n)=2+tRSum(n-1) =2+2+tRSum(n-2)

=2(2)+tRSum(n-2)

:

=n(2)+tRSum(0) =2n+2, n>=0

Example 2 :Complexity of Fibonacci series



Algorithm Fibonacci(n)

//compute the nth Fibonacci number

If(n<=1) then

Write (n);

else

Fnm2:=0; fnm1:= 1;

for i:=2 to n do

fn:= fnm1+fnm2;

fnm2:=fnm1;fnm1:=fn;

Write (fn);

To analyse the time complexity of this algorithm, we need to consider the two cases (1) n=0

or 1 and(2) n>1.When n=0 or 1 , lines 4 and 5 get executed once each. Since each line has an

s/e of 1, the total step count for this case is 2.When n>1, lines 4,8 and 14 are each executed

once. Line 9 gets executed n times and lines 11 and 12 get executed n-1 times each.Line 8 has

an s/e of 2, line 12 has an s/e of 2,and line 13 has an s/e of 0. The remaning lines that get

executed have s/e of 1. The total steps for the case n>1 is therefore 4n+1.



ASYMPTOTIC NOTATION

Introduction

A problem may have numerous algorithmic solutions. In order to choose the best algorithm

for a particular task, you need to be able to judge how long a particular solution will take to

run. Or, more accurately, you need to be able to judge how long two solutions will take to

run, and choose the better of the two. You don't need to know how many minutes and

seconds they will take, but you do need some way to compare algorithms against one another.

Asymptotic complexity is a way of expressing the main component of the cost of an algorithm,

using idealized units of computational work. Consider, for example, the algorithm for sorting

a deck of cards, which proceeds by repeatedly searching through the deck for the lowest card.

The asymptotic complexity of this algorithm is the square of the number of cards in the deck.

This quadratic behavior is the main term in the complexity formula, it says, e.g., if you

double the size of the deck, then the work is roughly quadrupled.

The exact formula for the cost is more complex, and contains more details than are needed to

understand the essential complexity of the algorithm. With our deck of cards, in the worst

case, the deck would start out reverse-sorted, so our scans would have to go all the way to the

end. The first scan would involve scanning 52 cards, the next would take 51, etc. So the cost

formula is 52 + 51 + ... + 1. generally, letting N be the number of cards, the formula is 1 + 2

+ ... + N, which equals (N + 1) * (N / 2) = (N2 + N) / 2 = (1 / 2)N

2 + N / 2. But the N^2 term

dominates the expression, and this is what is key for comparing algorithm costs. (This is in

fact an expensive algorithm; the best sorting algorithms run in sub-quadratic time.)

Asymptotically speaking, in the limit as N tends towards infinity, 1 + 2 + ... + N gets closer

and closer to the pure quadratic function (1/2) N^2. And what difference does the constant

factor of 1/2 make, at this level of abstraction. So the behavior is said to be O(n2).

Now let us consider how we would go about comparing the complexity of two algorithms.

Let f(n) be the cost, in the worst case, of one algorithm, expressed as a function of the input

size n, and g(n) be the cost function for the other algorithm. E.g., for sorting algorithms, f(10)

and g(10) would be the maximum number of steps that the algorithms would take on a list of

10 items. If, for all values of n >= 0, f(n) is less than or equal to g(n), then the algorithm with



complexity function f is strictly faster. But, generally speaking, our concern for

computational cost is for the cases with large inputs; so the comparison of f(n) and g(n) for

small values of n is less significant than the "long term" comparison of f(n) and g(n), for n

larger than some threshold.

Note that we have been speaking about bounds on the performance of algorithms, rather than

giving exact speeds. The actual number of steps required to sort our deck of cards (with our

naive quadratic algorithm) will depend upon the order in which the cards begin. The actual

time to perform each of our steps will depend upon our processor speed, the condition of our

processor cache, etc., etc. It's all very complicated in the concrete details, and moreover not

relevant to the essence of the algorithm.

BIG-O NOTATION

Definition

Big-O is the formal method of expressing the upper bound of an algorithm's running time. It's

a measure of the longest amount of time it could possibly take for the algorithm to complete.

More formally, for non-negative functions, f(n) and g(n), if there exists an integer n0 and a

constant c > 0 such that for all integers n > n0, f(n) ≤ cg(n), then f(n) is Big O of g(n). This is

denoted as "f(n) = O(g(n))". If graphed, g(n) serves as an upper bound to the curve you are

analyzing, f(n).

O-Notation (Upper Bound)

This notation gives an upper bound for a function to within a constant factor. We

write f(n) = O(g(n)) if there are positive constants n0 and c such that to the right of n0,

the value of f(n) always lies on or below cg(n).



Theory Examples

So, let's take an example of Big-O. Say that f(n) = 2n + 8, and g(n) = n2. Can we find a

constant c, so that 2n + 8 <= n2? The number 4 works here, giving us 16 <= 16. For any

number c greater than 4, this will still work. Since we're trying to generalize this for large

values of n, and small values (1, 2, 3) aren't that important, we can say that f(n) is generally

faster than g(n); that is, f(n) is bound by g(n), and will always be less than it.

It could then be said that f(n) runs in O(n2) time: "f-of-n runs in Big-O of n-squared time".

To find the upper bound - the Big-O time - assuming we know that f(n) is equal to (exactly)

2n + 8, we can take a few shortcuts. For example, we can remove all constants from the

runtime; eventually, at some value of c, they become irrelevant. This makes f(n) = 2n. Also,

for convenience of comparison, we remove constant multipliers; in this case, the 2. This

makes f(n) = n. It could also be said that f(n) runs in O(n) time; that lets us put a tighter

(closer) upper bound onto the estimate.

Practical Examples

O(n): printing a list of n items to the screen, looking at each item once. O(ln n): also "log n",

taking a list of items, cutting it in half repeatedly until there's only one item left. O(n2): taking

a list of n items, and comparing every item to every other item.



BIG-OMEGA NOTATION

For non-negative functions, f(n) and g(n), if there exists an integer n0 and a constant c > 0

such that for all integers n > n0, f(n) ≥ cg(n), then f(n) is omega of g(n). This is denoted as

"f(n) = Ω(g(n))".

This is almost the same definition as Big Oh, except that "f(n) ≥ cg(n)", this makes g(n) a

lower bound function, instead of an upper bound function. It describes the best that can

happen for a given data size.

Ω-Notation (Lower Bound)

This notation gives a lower bound for a function to within a constant factor. We write

f(n) = Ω(g(n)) if there are positive constants n0 and c such that to the right of n0, the

value of f(n) always lies on or above cg(n).

How asymptotic notation relates to analyzing complexity

Temporal comparison is not the only issue in algorithms. There are space issues as well.

Generally, a trade off between time and space is noticed in algorithms. Asymptotic notation

empowers you to make that trade off. If you think of the amount of time and space your

algorithm uses as a function of your data over time or space (time and space are usually

analyzed separately), you can analyze how the time and space is handled when you introduce

more data to your program.



This is important in data structures because you want a structure that behaves efficiently as

you increase the amount of data it handles. Keep in mind though that algorithms that are

efficient with large amounts of data are not always simple and efficient for small amounts of

data. So if you know you are working with only a small amount of data and you have

concerns for speed and code space, a trade off can be made for a function that does not

behave well for large amounts of data.

A few examples of asymptotic notation

Generally, we use asymptotic notation as a convenient way to examine what can happen in a

function in the worst case or in the best case. For example, if you want to write a function

that searches through an array of numbers and returns the smallest one:

function find-min(array a[1..n])

let j :=

for i := 1 to n:

j := min(j, a[i])

repeat

return j

end

Regardless of how big or small the array is, every time we run find-min, we have to initialize

the i and j integer variables and return j at the end. Therefore, we can just think of those parts

of the function as constant and ignore them.

So, how can we use asymptotic notation to discuss the find-min function? If we search

through an array with 87 elements, then the for loop iterates 87 times, even if the very first

element we hit turns out to be the minimum. Likewise, for n elements, the for loop iterates n

times. Therefore we say the function runs in time O(n).

What about this function:

function find-min-plus-max(array a[1..n])

// First, find the smallest element in the array

let j := ;

for i := 1 to n:

j := min(j, a[i])

repeat



let minim := j

// Now, find the biggest element, add it to the smallest and

j := ;

for i := 1 to n:

j := max(j, a[i])

repeat

let maxim := j

// return the sum of the two

return minim + maxim;

end

What's the running time for find-min-plus-max? There are two for loops, that each iterate n

times, so the running time is clearly O(2n). Because 2 is a constant, we throw it away and

write the running time as O(n). Why can you do this? If you recall the definition of Big-O

notation, the function whose bound you're testing can be multiplied by some constant. If

f(x) = 2x, we can see that if g(x) = x, then the Big-O condition holds. Thus O(2n) = O(n). This

rule is general for the various asymptotic notations.

THETA

Definition:

The function f(n)=theta(g(n))(read as ―f of n is theta of g of n‖) iff there exist

positive constants c1,c2, and n0 such that c1g(n) <=f(n)<=c2g(n) for all n, n>=n0.

Example:

The function 3n+2=theta(n) as 3n+2>=3n for all n>=2 and 3n+2<=4n for all

n>=2, so c1=3,c2=4 and n0=2. 3n+3= theta(n), 10n2+4n+2=theta(n2),6*2n+n2=theta(2n),

and 10* log n+4= theta (log n). 3n+2 !=theta(1), 3n+3 != theta(n2), 10n2+4n+2 !=theta(n),

10n2+4n+2 !=theta(1), 6*2n+n2 !=theta(n2), 6*2n +n2 !=theta(n100), and 6 * 2n +n2

!=theta(1).

The theta notation is more precise than both the big oh and big omega

notations. The function f(n)=theta(g(n)) iff g(n) is both lower and upper bound of f(n).



Little oh

Definition:

The function f(n)=0(g(n)) (read as ―f of n is little oh of g of n‖) iff

Lim f(n)/g(n)=0

n->infinity

Example:

The function 3n+2=o(n2) since lim n->infinity (3n+2)/n2=0. 3n+2=o(n log

n). 3n+2= o(n log log n). 6*2n+n2=o(3n). 6*2n+n2=o(2n log n). 3n+2 !=o(n). 6*2n+n2

!=o(2n).



Little omega

Definition:

The function f(n)=w(g(n)) (read as ―f of n is little omega of g of n‖) iff

Lim g(n)/f(n)=0

n->infinity

Example: Algorithm Sum(a,n)

S:=0.0;

for i=1 to n do

s:=s+a[i];

return s;

Alg 1. Iterative function for sum

For this algorithm Sum wec determined that tSum(n)=

2n+3. So, tSum(n)=theta(n)

Asymptotic Notation Properties

Let f(n) and g(n) be asymptotically positive functions. Prove or disprove each of the

following conjectures.



a. f(n)=O(g(n)) implies g(n)=O(f(n)).

b. f(n)+g(n)= (min(f(n),g(n))).

c. f(n)=O(g(n)) implies lg(f(n))=O(lg(g(n))),where lg(g(n))>=1 and f(n)>=1 for all

sufficiently large n.

d. f(n)=O(g(n)) implies 2^f(n)=O(2^g(n)).

e. f(n)=O((f(n))^2).

f. f(n)=O(g(n)) implies g(n) and Ω(f(n)).

g. f(n)= (f(n/2)).

h. f(n)+o(f(n))= (f(n)).

COMMON COMPLEXITY FUNCTIONS

Comparison of functions

Many of the relational properties of real numbers apply to asymptotic comparisons as well.

For the following, assume that f(n) and g(n) are asymptotically positive.

Transitivity:

f(n)= (g(n)) and g(n)= (h(n)) imply f(n)= (h(n)),

f(n)=O(g(n)) and g(n)=O(h(n)) imply f(n)=O(h(n)),

f(n)=Ω(g(n)) and g(n)=Ω(h(n)) imply f(n)=Ω(h(n)),

f(n)=o(g(n)) and g(n)=o(h(n)) imply f(n)=o(h(n)),

f(n)=ω(g(n)) and g(n)=ω(h(n)) imply f(n)=ω(h(n)).

Reflexivity:

f(n)= (f(n)),

f(n)=O(f(n)),

f(n)=Ω(f(n)).

Symmetry:

f(n)= (g(n)) if and only if g(n)= (f(n)).



Transpose symmetry

f(n)=O(g(n)) if and only if g(n)=Ω(f(n)),

f(n)=O(g(n)) if and only if g(n)=Ω(f(n)),

Because these properties hold for asymptotic notations, one can draw an analogy between the

asymptotic comparison of two functions f and g and the comparison of two real numbers a

and b:

f(n)=O(g(n)) similar to a<=b,

f(n)=Ω(g(n)) similar to a>=b,

f(n)= (g(n)) similar to a=b,

f(n)=o(g(n)) similar to a<b,

f(n)=ω(g(n)) similar to a>b.

We say that f(n) is asymptotically smaller than g(n) if f(n)=o(g(n)), and f(n) is asymptotically

larger than g(n) if f(n)=ω(g(n)).

One property of real numbers, however, does not carry over to asymptotic notations:

Trichotomy: For any two real numbers a and b, exactly one of the following must hold:

a<b, a=b, or a>b.

Although any two real numbers can be compared, not all functions are asymptotically

comparable. That is, for two functions f(n) and g(n), it may be the case that neither

f(n)=O(g(n)) nor f(n)=Ω(g(n)) holds. For example, the functions n and n^(1+sin n) cannot be

compared using asymptotic notation, since the value of the exponent in n^(1+sin n) oscillates

between 0 and 2, taking on all values in between.



RECURRENCE RELATIONS

The recurrence relation is an equation or inequality that describes a function in terms of its

values of smaller inputs. The main tool for analyzing the time efficiency of a recurrence

algorithm is to setup a sum expressing the number executions of its basic operation and

ascertain the solution‘s order of growth

To solve a recurrence relation means ,to obtain a function defined on natural numbers that

satisfies the recurrence.

Examble: tn = 2tn-1

T(n)= 3T(n/4)+(θn²)

Different methods for solving recurrence relation are:-

Substitution method

Itration method

Changing variables method

Recurrence tree

Characteristic equation method

n^α method

Master theorem

Solving recurrences by Substitution method

Idea: Make a guess for the form of the solution and prove by induction.

Can be used to prove both upper bounds O() and lower bounds Ω().

Let‘s solve T(n) = 2T(n/2) + n using substitution

– Guess T(n) ≤ cn log n for some constant c (that is, T(n) = O(n log n))

– Proof:

Base case: we need to show that our guess holds for some base case (not

necessarily

n = 1, some small n is ok). Ok, since function constant for small constant n.

Assume holds for n/2: T(n/2) ≤ c n∕2 log n∕2

Prove that holds for n: T(n) ≤cn log n

T(n)= 2T(n/2)+n



≤ 2(c n/2 logn/2) + n

= cn log n/2 +n

=cn log n-cn log 2+ n

=cn log n-cn + n

So ok if c ≥ 1

Similarly it can be shown that T(n) =Ω (n log n)

Similarly it can be shown that T(n) = T([n/2]) + T([n/2]) + n is θ(n lg n).

The hard part of the substitution method is often to make a good guess.

Solving Recurrences with the Iteration

In the iteration method we iteratively ―unfold‖ the recurrence until we ―see the

pattern‖.

The iteration method does not require making a good guess like the substitution

method (but

it is often more involved than using induction).

Example: Solve T(n) = 8T(n/2) + n² (T(1) = 1)

T(n) = n² + 8T(n/2)

= n² + 8(8T( n/2² ) + (n/2)²)

= n² + 8²T( n/2 ²) + 8(n²/4))

= n²+ 2n² + 8²T( n/2² )

= n² + 2n² + 8²(8T( n/2³ ) + ( n/2² )²)

= n² + 2n² + 8³T( n/2³ ) + 8²(n²/4² ))

= n² + 2n² + 2²n²+ 8³T( n/2³ )

= . . .

= n² + 2n² + 2²n²+ 2²n³ + . . .

Masters theorem

The master method is used for solving the following type of recurrence

T(n)=aT(n/b)+f(n),a 1 and b>1.In above recurrence the problem is divided in to ‗a‘

subproblems each of size atmost ‗n/b‘.The subproblems are solved recursively each in T(n/b)



time.The cost of split the problem or combine the solutions of subproblems is given by

function f(n).It should be note that the number of leaves in the recursion tree is where

E=

THEOREM

Let T(n) be defined on the non-negative integers by the recurrence

T(n)=aT(n/b)+f(n) where a 1,b>1 are constants and f(n) be a function.Then T(n) can be

asymtotically as,

CASE 1:

If f(n) O( for some e>0,then T(n)€ ( )

CASE 2:

If f(n) € ( ) then T(n) € (f(n).log n)

CASE 3:

If f(n) € Ω( for some e>0 and f(n) € O( for some e then T(n) € (f(n))



RECURRENCE PROBLEMS

Case 1 problem:

1. T(n)=4T(n/2)+n

E=log 4/log2=2

O(

O( =n

T(n) € ( )

Case 2 problem:

2.T(n)=4T(n/2)+

a=4, b=2

f(n)=

E=log 4/log 2=2

( )

T(n) € ( .log n)

Case 3 problem:

3.T(n)=4T(n/2)+

a=4, b=2

f(n)=

E=log 4/log 2=2

Ω( )

= Ω( )

= Ω( )

O( )

= O( )

= O( ), T(n) € ( )



RECURSION TREES FOR COMPLEXITY CALCULATIONS

A different way to look at the iteration method: is the recursion-tree

we draw out the recursion tree with cost of single call in each node—running time is

sum of

costs in all nodes

if you are careful drawing the recursion tree and summing up the costs, the recursion

tree is

a direct proof for the solution of the recurrence, just like iteration and substitution

Example: T(n) = 8T(n/2) + n² (T(1) = 1)

T(n)=n²+ 2n²+ 2²n²+ 2³n²+ .....+2^(log n-1) n²+8^(log n)



Changing variables

Sometimes reucurrences can be reduced to simpler ones by changing variables

Example: Solve T(n) = 2T(√n) + log n

Let m = log 2^m = n √n = 2^m/2

T(n) = 2T(√n) + log n ) T(2^m) = 2T(2^m/2) + m

Let S(m) = T(2^m)

T(2^m) = 2T(2^m/2) + m ) S(m) = 2S(m/2) + m)

S(m) = O(mlogm))

T(n) = T(2^m) = S(m) = O(mlogm) = O(log n log log n)

Other recurrences

Some important/typical bounds on recurrences not covered by master method:

Logarithmic: θ(log n)

– Recurrence: T(n) = 1 + T(n/2)

– Typical example: Recurse on half the input (and throw half away)

– Variations: T(n) = 1 + T(99n/100)

Linear: θ(N)

– Recurrence: T(n) = 1 + T(n − 1)

– Typical example: Single loop

– Variations: T(n) = 1+2T(n/2), T(n) = n+T(n/2), T(n) = T(n/5)+T(7n/10+6)+n

Quadratic: θ(n2)

– Recurrence: T(n) = n + T(n − 1)

– Typical example: Nested loops

Exponential: θ(2^n)

– Recurrence: T(n) = 2T(n − 1)



PROFILING

Profiling or performance measurement is the process of executing a correct program on data

set and measuring the time and space it takes to compute the result.

NONDETERMINISTIC ALGORITHMS

The notation of algorithm that we have been using has the property that the result of every

operation is uniquely defined.Algoithm with this property are termed deterministic

algorithms.

Such algorithms agree with the programs are executed on computer. In a theoretical

framework we can remove this rustication on the outcomes of every operation. We can allow

algorithms to be containing operations whose outcomes of every operation. We can allow

algorithm to contain operations whose outcomes are not uniquely defined but are limited to

specified set of possibilities.

The machine executing such operations is allowed to choose any one of these outcomes

subject to be a defined later. This leads to the concept of nondeterministic algorithms, we

introduce three functions.

1. Choice (S) arbitrarily chooses one of the elements of element of set S.

2. Failure () signals an unsuccessful completion.

3. Success () signals a successful completion.

The assignment statement x=Choice (1, n) could result in x being assigned any

one of the integers in the range [1,n].there is the rule of specifying how this choice is to be

made. The Failure () and Success () signals are used to define a computation of the algorithm.

These statements cannot be used to effect a return. Whenever there is a set of choices that

leads to a successful completion, then one such set of choices is always made and the

algorithm terminates successfully. A nondeterministic algorithm terminates unsuccessfully if

and only if there exists no set choices leading to a success signal. The computing times for.

Choice, Failure, and Success are taken to be O(1).A machine capable of executing a non



nondeterministic algorithm in this way is called a nondeterministic machine. Although

nondeterministic machines do not exist in practice, we see that they provide strong intuitive

reasons to conclude that certain problems cannot be solved by fast deterministic algorithms.

Example

Consider the problem of searching for an element x in a given set of elements A[1:n],n>1.We

are required to determine an index j such that A[j]=x or j=0 if x is not in A.

1. J:=Choice(1.n);

2. If A[j]=x then write (j);Success();

3. Write (0); Failure();

From the way a nondeterministic computation is defined, it follows that no is 0 can be the

output if and only if there is no j such that A[j]=x.

Complexity of nondeterministic search algorithms =O(1).

Since A is not ordered, every deterministic algorithm is of complexity Ω (n).

DETERMINISTIC ALGOITHM for n>p

The deterministic algorithm for a selection whose run time is O(n/p log log p+ log

n)The basic idea of this algorithm is same to sequential algorithm.

The sequential algorithm partitions the input into groups(of size,says,5),finds the median of

each group, and output recursively the median(call it M)of these group meadians.Then the

rank

rM of M in the input is computed, and as a result, all element from the input that are either

≤ M or >Mare dropped, depending on whether i >rM or i ≤ rM,

Respectively. Finally an appropriate selection is performed from the remaining keys

recursively.

We showed that run time of this algorithm was O(n).



MODULE 2



DIVIDE AND CONQUER

Given a function to compute on n inputs the divide and conquer strategy suggests splitting the

inputs into k distinct subsets,1<k<n,yielding k subproblems. These subproblems must be

solved,and then a method must be found to combine subsolutions into a solution of whole. If

the subproblems are still relatively large,then the divide and conquer strategy can possibly

reapplied.The principle is natrually expressed using recursive algorithm.that is smaller and

smaller subproblems of the same kind are generated untill eventually subproblems that are

small enough to be solved without splitting are produced.

CONTROL ABSTRACTION

Control abstraction means a procedures whose flowof control is clear but whose

primary operations are specified by other procedures whose precise meanings are left

defined.Below algorithmDAndC is initially invoked as DandC(p),where p is the problem to

be solved.

Small(p) is a boolean-valued function that determines whether the input size is small enough

that the answer can be computed without splitting.If this is so,the function S is

invoked.Otherwise the problem p is divided into smaller subproblems.These subproblems

P1,P2,----,Pkare solved by recursive applications of DandC.Combine is a function that

determines the solutions to P using the solutions to the k subproblems.

ALGORITHM

Algorithm DAndC(p)

if small(p) then return s(p);

else

divide p into smaller instances p1,p2,p3-----p(k),k>=1;

apply DandC to each of these subproblems;

return combine(DandC(p1),DandC(p2),-----,DandC(pk));



FINDING THE MAXIMUM AND MINIMUM

This is a simple problem that can be solved by the divide and-conquer

technique. The problem is to find the maximum and minimum items in a set of ‗n ‗elements.

ALGORITHM-1

Algorithm StraightMaxMin (a, n, max, min)

//Set ‗max’ to the maximum and ‗min‘ to the minimum of a[1:n];

max:=min:=a[1];

for i:=2 to n do

if (a[i]>max) then max:= a[i];

if a[i]<min) then min:= a[i];

This is a straight forward algorithm to accomplish the above problem. In

analyzing the time complexity of this algorithm, we concentrate on the number of element

comparisons. The frequency count for other operations in this algorithm is of the same order

as that for element comparisons. StraightMaxMin requires 2(n-1) element comparisons in

the best, average, and worst cases.

The best case occurs when the elements are in increasing order. The number

of element comparisons is (n-1). The worst case occurs when the elements are in the

decreasing order. In this case, the number of element comparisons is 2(n-1).

The average number of element comparisons is less than 2(n-1). On the

average, a[i] is greater than max, half the time, and so the average number of comparisons is

3n/2-1. The maximum and minimum are a[i] if n=1. If n=2, the problem can be solved by

making one comparison.



ALGORITHM-2

Algorithm MaxMin ( i, j, max, min )

//a[1:n] is a global array. Parameters I and j are integers, 1<=i<=j<=n. The effect is to set

//max and min to the largest and smallest values in a [i : j] ,respectively.

if (i=j) then max :=min :=a[i]; // Small(p)

else if (i=j-1) then //Another case of Small(p)

if( a[i] < a[j]) then

max:=a[j];

min:=a[i];

else

max:=a[i];

min:=a[j];

else

// If P is not small, divide P into subproblems. Find where to split

// mid := [(i+j)/2];

// Solve the subproblems.

MaxMin ( i, mid, max, min )

MaxMin ( mid+1, j, max, min)

if (max<max1)then max:=max1;

if (min>min1)then min:=min1;

MaxMin is a recursive algorithm that finds the maximum and minimum of

the set of elements a(i),a(i+1),…………….a(j).The situation of set sizes one (i=j) and two

(i=j-1) are handled separately. For sets containing more than two elements, the midpoint is

determined (just as in binary search) and two new subproblems are generated. When the

maxima and minima of these subproblems are generated, the two maxima are compared and

the two minima are compared to achieve the solution for the entire set.



Eg:- Suppose we simulate MaxMin on the following nine elements:

a: [1] [2] [3] [4] [5] [6] [7] [8] [9]

22 13 -5 -8 15 60 17 31 47

A good way of keeping track of recursive calls is to build a tree by adding a

node each time anew call is made. For this algorithm each node has four items of

information: i, j, max, and min. On the array a[] above, the tree is produced.

Trees of recursive calls of MaxMin

We see that the root node contains 1 and 9 as the values of i and j corresponding to

the initial call to MaxMin. This execution produces new call to MaxMin where i and j have

the values 1 ,5 and 6,9, respectively, and thus split the set into two subsets of approximately

the same size. From the tree, we can immediately see that the maximum depth of recursion is

four(including the first call).

The order in which max and min are assigned values are follows: [1,2,22,13] [3,3,-5,-5]

[1,3,22,-5] [4,5,15,-8] [1,5,22,-8] [6,7,60,17] [8,9,47,31] [6,9,60,17] [1,9,60,-8].

Number of element comparisons needed for MaxMin:

If T(n) represents the number of element comparisons needed, then the

resulting recurrence relation is:

T ([n/2]) + T ([n/2]) +2 n>2

T (n) = 1 n=2

0 n=1

1, 9,60,-8

6,9,60,17

1, 3, 22, -5 4, 5, 15,-8 6,7,60,17

8, 9, 47, 31

1,2,22,13 3,3,-5,-5

1, 5, 22, -8



When n is a power of 2, n=2k we can solve this equation by successive substitutions:

T(n) = 2T(n/2)+2

= 2(2T(n/4)+2) +2

= 4T(n/4)+ 4 + 2

.

.

= 2k-1

T(2) +∑ 1< = i < = k-1 2i

= 2k-1

+ 2k

-2

= 3n/2 - 2

When n is a power of 2, the number of comparisons in the best , average and worst case is

3n/2-2.

COMPLEXITY ANALYSIS OF FINDING MAXIMUM&MINIMUM

Consider the following nine elements to simulate MaxMin

22 13 -5 -8 15 60 17 31 47

Fig below shows the tree constructed for the algorithm. Each node has four items of

information: i, j, max and min.

9

5 8

3 4 6 7

1 2

Consider the total number of element comparison needed for MaxMin? If T(n) represents this

number, then the resulting recurrence relation is

T ( n/2 ) + T( n/2 ) + 2 n>2

T (n)= 1 n=2

0 n=1

1, 9, 60,-8

1, 5, 22,-8

1, 3, 22,-5

6, 9, 60,17

4, 5, 15,-8 6, 7, 60,17 1, 9, 60,-8

1, 2, 22, 13 3, 3, -5,-5



When n is a power of 2, n=2k

for some positive integer k, then

T(n) = 2T( n/2 ) + 2

= 2(2T (n/4) +2) +2

= 4T (n/4) +4+2

:

:

= 2k-1

T (2) +∑1<=i<=k-1 2i

= 2k-1

+2k-2

=3n/2-2

Therefore 3n/2 is the best, average and worst case number of comparisons when n is a power

of two.

Consider the count when element comparisons have the same cost as comparisons between i

and j. Let C (n) be this number.

Assuming n=2k for some positive integer k, we get

C(n) = 2C(n/2) + 3 n>2

2 n=2

C (n) = 2C (n/2) + 3

=4 C (n/4) + 6 + 3

:

:

=2k-1

C (2)+3∑0k-2

2i

=2k+3*2

k-1-3

=5n/2-3

if comparisons between array elements are costlier than comparisons of integer variable, then

divide and conquer method is more efficient

In both cases mentioned above the best case, average case and worst case complexity is θ(n)



RECURSION TREE

DEFINITION:

A recursion tree is a tree that depicts the entire recursion process.

Consider the following recurrence relation.

T (n) =3T (n/4)+ θ(n2)

= 3T (n/4) + cn2

cn2

c(n/4)2 c(n/4)

2 c(n/4)

2

:

T(n/16) T(n/16) T(n/16) T(n/16) T(n/16) T(n/16) :

: : : :

c(n/16)2

c(n/16)2

c(n/16)2

: : : :

: : : : : :

T(n/16) T(n/16) T(n/16) : : : : : :

: : : : : : : : :

T(1) T(1) T(1) T(1) T(1) T(1) T(1) T(1) T(1)

Sub problem size at depth i=n/4i

Boundary condition (n=1) :=(n/4i)=1 ie i=log4n

Depth of the tree=log4n+1 (0, 1, 2,………logn)

Cost of each node at level i=c(n/4)i

No: of node at level log4n=3log4n

Cost of each node=3log4n.T(1)=θ(3log4n)=θ(nlog43)

Total cost=cn2+3/16cn

2+ (3/16)

2cn

2+ (3/16)

log4n-1cn

2

= (3/16)I cn

2+ θ(nlog43)+ θ(nlog43)



= (3/16)I cn

2+ θ(nlog43)

= 1/(1-3/16)cn2+ θ(nlog43)

= 16/13cn2+ θ (nlog43)

= O (n2)

BINARY SEARCH ALGORITHMS

The Binary search requires an ordered list.

Iterative Algorithm

int find (const list, int target)

// pre: list is sorted in ascending order

//post: ITERATIVE binary search will return the index of the target element, else -1

int mid;

int first = 0;

int last = list.length( ) -1;

while ( first <= last )

mid = (first + last) / 2;

if ( list[mid] == target )

return mid;

if ( list[mid] > target )

last = mid - 1;

else first = mid + 1;

return -1;

Recursive Algorithm

int find (const list, int target, int first, int last)

// pre: list is sorted in ascending order

//post: RECURSIVE binary search will return the index of the target element, else -1



if (first > last)

return -1;

int mid = (first + last) / 2;

if (list[mid] == target)

return mid;

if (list[mid] < target)

return find(list, target, mid+1, last);

return find(list, target, first, mid-1);

Complexity Analysis of Binary Search

For binary search, storage is required for the n elements of the array plus the variables

low, high, mid and x.

A binary search tree can be created to explain the binary search where the mid value

of each node is the value of the node. For example, if n=14, the resultant tree will be like the

one shown below.

3

7

11

1 5 9 13

2 4 6 8 10 12 14



The first comparison is s with a[7].if x<a[7], then the next comparison is with a[3];

similarly, if x>a[7], then the next comparison is with a[11]. If x is present, then the algorithm

will end at one of the circular nodes tat lists the index into the array where x was found. If x

is not present the algorithm will terminate at one of the square nodes.

Theorem:

If n is in the range [2k-1

,2k), then binary search makes at most k element comparison

for a successful search and either k-1 or k comparison for an unsuccessful search. In other

words the time for a successful search is O (log n) and for an unsuccessful search is ϴ (log n).

E=I+2n

Where E=External path length (sum of the distance of all external nodes from the root)

I=Internal path length (sum of the distance of all internal nodes from the root).

Let As (n) is the average number of comparison in a successful search and Au (n) is

the average number of comparison in an unsuccessful search.

The number of comparison to find an element represented by an internal node is one

more than the distance of this node from the root.

As (n) =1+I/n

The number of comparisons on any path from the root to an external node is equal to

the distance between the root and the external node. Since every binary tree with n internal

nodes has n+1 external nodes

Au (n) =E/ (n+1)

As (n) =I/n+1

= (E-2n/n) + 1

=E/n-1

= (Au (n). (n+1))/n -1

= (n. Au (n) + Au (n))/n -1

= Au (n) + Au (n)/n -1

As (n) = Au (n) (1 +1/n) -1



Successful searches

Best-ϴ (1) (only one element is compared)

Average- ϴ (log n)

Worst- ϴ (log n)

Unsuccessful searches

Best, Average and Worst- ϴ (log n)

DIVIDE-AND-CONQUER MATRIX MULTIPLICATION ALGORITHM

The product of two n x n matrices X and Y is a third , n x n matrix Z=XY , with (i,j)th

entry

n

Zij=∑Xik Ykj

k=1

In general XY, not the same as YX; matrix multiplication is not commutative.

The formula above implies an O (n3)

algorithm for matrix multiplication: there are n

2 entries

to be computed, and each takes linear time. For quite a while, this was widely believed

to be the best running time possible, and it was even proved that no algorithm which used

just additions and multiplications could do better. It was therefore a source of great

excitement when in 1969, Strassen announced a signi_cantly more ef_cient algorithm, based

upon divide-and-conquer.

Matrix multiplication is particularly easy to break into subproblems, because it can be

performed blockwise. To see what this means, carve X into four n/2 x n/2 blocks, and also Y:

X = A B

Y = E F

C D ,

G H

Then their product can be expressed in terms of these blocks, and is exactly as if the blocks

were single elements.

XY= A B E F

= AE+BG AF+BH



C D G H CE+DG CF+DH

We now have a divide-and-conquer strategy: to compute the size-n, product XY,

recursively compute eight size-n/2 products AE,BG,AF,BH,CE,DG,CF,DH and then do a

few O(n2) time additions. The total running time is described by the recurrence relation

T(n)= 8T(n/2) + O(n2),

which comes out to O(n3), the same as for the default algorithm. However, an improvement

in the time bound is possible, and as with integer multiplication, it relies upon algebraic

tricks. It turns out that XY can be computed from just seven sub problems.

P1 =

A(F – H)

P2 =

(A + B)H

P3 =

(C + D)E

P4 =

D(G – E)

P5 =

(A + D) (E + H)

P6 =

(B - D) (G + H)

P7 =

(A - C) (E + F)

XY = P

5+P

4 – P

2 +P

6 P

1+P

2

P

3+P

4 P

1+P

5 – P

3 – P

7

This translates into a running time of

T(n)= 7T(n/2) + O(n2),

which by the result is O(nlog

2 7

)≈ O(n2.81

).



STRESSEN’S MATRIX MULTIPLICATION

Strassen showed that 2x2 matrix multiplication can be accomplished in 7 multiplication and

18 additions or subtractions. This reduce can be done by Divide and Conquer

Approach.Divide the input data S in two or more disjoint subsets S1, S2. Solve the

subproblems recursively. Combine the solutions for S1, S2, …, into a solution for S. The base

case for the recursion are subproblems of constant size.Analysis can be done using recurrence

equations.Divide matrices in sub-matrices and recursively multiply sub-matrices

Let A, B be two square matrices over a ring R. We want to calculate the matrix product C as

If the matrices A, B are not of type 2n x 2

n we fill the missing rows and columns with zeros.

We partition A, B and C into equally sized block matrices

with

then

With this construction we have not reduced the number of multiplications. We still need 8

multiplications to calculate the Ci,j matrices, the same number of multiplications we need

when using standard matrix multiplication.

Now comes the important part. We define new matrices

http://en.wikipedia.org/wiki/Square_matrix

http://en.wikipedia.org/wiki/Ring_%28mathematics%29

http://en.wikipedia.org/wiki/Block_matrix



which are then used to express the Ci,j in terms of Mk. Because of our definition of the Mk we

can eliminate one matrix multiplication and reduce the number of multiplications to 7 (one

multiplication for each Mk) and express the Ci,j as

We iterate this division process n-times until the submatrices degenerate into numbers (group

elements).

Practical implementations of Strassen's algorithm switch to standard methods of matrix

multiplication for small enough submatrices, for which they are more efficient. The particular

crossover point for which Strassen's algorithm is more efficient depends on the specific

implementation and hardware. It has been estimated that Strassen's algorithm is faster for

matrices with widths from 32 to 128 for optimized implementations and 60,000 or more for

basic implementations.

Numerical analysis

The standard matrix multiplications takes approximately 2n3 arithmetic operations (additions

and multiplications); the asymptotic complexity is O(n3).

http://en.wikipedia.org/wiki/Submatrices



The number of additions and multiplications required in the Strassen algorithm can be

calculated as follows: let f(k) be the number of operations for a matrix. Then by

recursive application of the Strassen algorithm, we see that f(k) = 7f(k − 1) + l4k, for some

constant l that depends on the number of additions performed at each application of the

algorithm. Hence f(k) = (7 + o(1))k, i.e., the asymptotic complexity for multiplying matrices

of size n = 2k using the Strassen algorithm is

.

The reduction in the number of arithmetic operations however comes at the price of a

somewhat reduced numerical stability.

Algorithm

void matmul(int *A, int *B, int *R, int n)

if (n == 1)

(*R) += (*A) * (*B);

else

matmul(A, B, R, n/4);

)()( Thus 3

1 1

3

1

,

1

,,

NOcNcNT

baC

N

i

N

j

N

k

jk

N

k

kiji

http://en.wikipedia.org/wiki/Numerical_stability



matmul(A, B+(n/4), R+(n/4), n/4);

matmul(A+2*(n/4), B, R+2*(n/4), n/4);

matmul(A+2*(n/4), B+(n/4), R+3*(n/4), n/4);

matmul(A+(n/4), B+2*(n/4), R, n/4);

matmul(A+(n/4), B+3*(n/4), R+(n/4), n/4);

matmul(A+3*(n/4), B+2*(n/4), R+2*(n/4), n/4);

matmul(A+3*(n/4), B+3*(n/4), R+3*(n/4), n/4);



MERGE SORT

Merge sort is an example of divide and conquer algorithm. Its worst case complexity is

O(nlog n). To understand merge sort assume that elements are arranged in the non decreasing

order. Assume a sequence of n elements a[1],a[2],...a[n]. Split them into two sets like

a[1],a[2],..a[n/2] and a[(n/2)+1],...a[n]. Each set is individually sorted, and the resulting

sorted sequences are merged to produce a single sorted sequence of n elements. Merge-sort is

based on the divide-and-conquer paradigm.

The Merge-sort algorithm can be described in general terms as consisting of the following

three steps:

1. Divide Step: If given array A has zero or one element,it is already sorted. Otherwise,

divide A into two arrays, each containing about half of the elements of A.

2. Recursion Step: Recursively sort array A1 and A2.

3. Conquer Step: Combine the elements back in A by merging the sorted arrays A1 and A2

into a sorted sequence.

Consider the following example:

Consider the array of ten elements a [1:10] = (310,285,179,652,351,423,861,254,450,520)

The merge sort algorithm first divides the array as follows

a[1:5] and a[6:10] and then

a[1:3] and a[4:5] and a[6:8] and a[9:10] and then into

a[1:2] and a[3:3] and a[6:7] and a[8:8] and finally it will look like

a[1:1],a[2:2],a[3:3],a[4:4],a[5:5],a[6:6],a[7:7],a[8:8],a[9:9],a[10:10]

Pictorially the file can now be viewed as

(310|285|179|652,351|423,861,254,450,520)

Elements a[1] and a[2] are merged as

(285,310|179|652,351|423,861,254,450,520)



And then a[3] is merged with a[1:2]

(179,285, 310 |652,351|423,861,254,450,520) next a[4] and a[5]

(179,285, 310 |351,652|423,861,254,450,520)and then a[1:3] and a[4:5]

(179,285, 310,351,652|423,861,254,450,520)

Repeated recursive calls are invoked producing the following array

(179,285, 310 ,351,652|254,423, 450,520,861) and finally

(179,254,285,310, 351,423, 450,520,652,861)

At this point there are two sorted subarrays and final merge produces the fully sorted result

If the time for the merging operation is proportional to n, then the computing time for merge

sort is described by the recurrence relation

a n= 1a a constant

1, 10

1, 5

1, 3 4, 5

6, 10

6, 8 9, 10

1, 1

3, 3 1, 2 4, 4 6, 7 5, 5

2, 2

8, 8 9, 9 10, 10

7, 7 6, 6



T(n)=

2T(n/2)+cn n>1, c a constant

When n is a power of 2, n=2k we can solve this equation by successive substitutions:

T(n) = 2(2T(n4)+cn/2)+cn

= 4T(n/4)+2cn

= 4(2T(n/8)+cn/4) + 2cn

.

.

= 2kT(1) + kcn

= an+cn log n

It is easy to see that if 2k < n <2

k+1, then T(n)<=T(2

k+1). Therefore T(n)=O (nlogn).

24

MergeSort describes the process very succinctly using recursion and a function Merge which

merges two sorted sets. Before executing MergeSort , the n elements should be placed in

a[1:n]. Then MergeSort (1,n) causes the keys to be rearranged into nondecreasing order in a.

Algorithm MergeSort(low,high)

//a[low:high] is a global array to be sorted.

//Small(P) is true if there is only one element to sort.

//In this case the list is already sorted.

if(low<high) then // If there are more than one element.

//Divide P into subproblems.



//Find where to split the set .

mid:=[(low+high)/2];

//Solve the subproblems.

MergeSort(low,high);

MergeSort(mid+1,high);

//Combine the solutions.

Merge(low,mid,high);

Algorithm Merge(low,mid,high)

//a[low:high] is a global array containing two sorted subsets in

//a[low:mid] and in a[mid+1:high].The goal is to merge these two

// sets into single set residing in a[low:high].b[] is an auxiliary global

//array.

h:=low; i:=low; j:=mid+1;

while ((h ≤ mid) and (j ≥ high)) do

if (a[h] ≤ a[j]) then

b[i]:=a[h]; h:=h+1;

else



b[i]:=a[j]; j:=j+1;

i:=i+1;

if(h>mid) then

for k:=j to high do

b[i]:=a[k]; i:=i+1;

else

for k:=h to mid do

b[i]:=a[k]; i:=i+1;

for k:=low to high do a[k]:=b[k];



COMPLEXITY ANALYSIS OF MERGE SORT

Merge sort is an ideal example of the divide and conquer strategy in which the given set of

elements is split into two equal-sized sets and the combining operation is the merging of two

sorted sets into one.

Given a sequence of n elements a[1],.....,a[n], the general idea is to imagine them to split into

two sets a[1],...a[n/2] and a[n/2+1],...a[n].Each set is individually sorted and the resulting

sorted sequences are merged to produce a single sorted sequence of n elements.

MergeSort describes this process using recursion and a function Merge which merges two

sorted sets.

Example:

Consider an array of ten elements a[1:10] = (310,285,179,652,351,423,861,254,450,520).

Algorithm MergeSort begins by splitting a[] into two subarrays each of size five (a[1:5] and

a[6:10]). The elements in a[1:5] are then split into two subarrays of size three (a[1:3]) and

two (a[4:5]). Then the items in a[1:3] are split into subarrays of size two (a[1:2]) and one

(a[3:3]). The two values in a[1:2] are split final time into one-element subarrays , and now the

merging begins. A record of the subarrays is implicitly maintained by the recursive

mechanism.

Pictorially the file can be viewed as

( 310 | 285 | 179 | 652 , 351 | 423 , 861 , 254 , 450 , 520 ) where vertical bars indicate the

boundaries of subarrays. Elements a[1] and a[2] are merged to yield

( 285 , 310 | 179 | 652 , 351 | 423 , 861 , 254 , 450 , 520 )

Then a[3] is merged with a[1:2] and

( 179 , 285 , 310 | 652 , 351 | 423 , 861 , 254 , 450 , 520 ) is produced. Next elements a[4]

and a[5] are merged:

( 179 , 285 , 310 | 351 , 652 | 423 , 861 , 254 , 450 , 520 )

And then a[1:3] and a[4:5]:

( 179 , 285 , 310 , 351 , 652 | 423 , 861 , 254 , 450 , 520 )



At this point the algorithm has returned to the first invocation of MergeSort and is about to

process the second recursive call. Repeated recursive calls are invoked producing the

following subarrays:

( 179 , 285 , 310 , 351 , 652 | 423 | 861 | 254 | 450 , 520 )

Elements a[6] and a[7] are merged. Then a[8] is merged with a[6:7]:

( 179 , 285 , 310 , 351 , 652 | 254 , 423 , 861 | 450 , 520 )

Next a[9] and a[10] are merged and then a[6:8] and a[9:10]:

( 179 , 285 , 310 , 351 , 652 | 254 , 423 , 450 , 520 , 861 )

At this point there are two sorted subarrays and the final merge produces the fully sorted

result

( 179 , 254 , 285 , 310 , 351 , 423 , 450 , 520 , 652 , 861 )

Figure (1.1) is a tree that represents the sequence of recursive calls that are produced by

MergeSort when it is applied to ten elements. The pair of values in each node are the values

of parameters low and high. The splitting continues until the sets containing a single element

are produced.

Figure (1.2) is a tree representing the calls to produce Merge by MergeSort. For example,

the node containing 1 , 2 and 3 represents the merging of a[1:2] with a[3].



Tree of calls of Merge

( Fig 1.1 )

Tree of calls of MergeSort(1,10)

( Fig 1.2 )

1,10

1,5 6,10

1,3 4,5 6,8 9,10

1,2 3,3 9,9 10,10 8,8 6,7 4,4 5,5

1,1 2,2 6,6 7,7

1,1,2 6,6,7

1,2,3 6,7,8 9,9,10 4,4,5

1,3,5 6,8,10

1,5,10



If the time for the merging operation is proportional to n, then the computing time for merge

sort is described by the recurrence relation

T(n) = a n=1 , a a constant

2T(n/2) + cn n>1 , c a constant

When n is a power of 2, n = 2^k, we can solve this equation by successive substitutions:

T(n) = 2(2T(n/4)+ cn/2) +cn

= 4T(n/4) + 2cn

= 4(2T(n/8) + cn/4) + 2cn

.

.

.

= 2^kT(1) + kcn

= an + cn logn

It is easy to see that if 2^k < n < 2^(k+1) , then T(n) < T(2^k+1).

T(n) = O(n logn)



QUICK SORT

Quicksort is one of the fastest and simplest sorting algorithms . It works recursively by a

divide-and-conquer strategy.

Idea

First, the sequence to be sorted a is partitioned into two parts, such that all elements of the

first part b are less than or equal to all elements of the second part c (divide). Then the two

parts are sorted separately by recursive application of the same procedure (conquer).

Recombination of the two parts yields the sorted sequence (combine). Figure 1 illustrates this

approach.

Figure 1: Quicksort(n)

The first step of the partition procedure is choosing a comparison element x. All elements of

the sequence that are less than x are placed in the first part, all elements greater than x are

placed in the second part. For elements equal to x it does not matter into which part they

come. In the following algorithm it may also happen that an element equal to x remains

between the two parts.



After partitioning the sequence, quicksort treats the two parts recursively by the same

procedure. The recursion ends whenever a part consists of one element only.

Program

The following Java program implements quicksort.

void quicksort (int[] a, int lo, int hi)

// lo is the lower index, hi is the upper index

// of the region of array a that is to be sorted

int i=lo, j=hi, h;

int x=a[(lo+hi)/2];

// partition

do

while (a[i]<x) i++;

while (a[j]>x) j--;

if (i<=j)

h=a[i]; a[i]=a[j]; a[j]=h;

i++; j--;

while (i<=j);

// recursion

if (lo<j) quicksort(a, lo, j);

if (i<hi) quicksort(a, i, hi);

Analysis

The best-case behavior of the quicksort algorithm occurs when in each recursion step the

partitioning produces two parts of equal length. In order to sort n elements, in this case the



running time is in Θ(n log(n)). This is because the recursion depth is log(n) and on each level

there are n elements to be treated (Figure 2 a).

The worst case occurs when in each recursion step an unbalanced partitioning is produced,

namely that one part consists of only one element and the other part consists of the rest of the

elements (Figure 2 c). Then the recursion depth is n-1 and quicksort runs in time Θ(n2).

In the average case a partitioning as shown in Figure 2 b is to be expected.

The choice of the comparison element x determines which partition is achieved. Suppose that

the first element of the sequence is chosen as comparison element. This would lead to the

worst

case behavior of the algorithm when the sequence is initially sorted. Therefore, it is better to

choose the element in the middle of the sequence as comparison element.

Even better would it be to take the n/2-th greatest element of the sequence (the median). Then

the optimal partition is achieved. Actually, it is possible to compute the median in linear time

[AHU 74]. This variant of quicksort would run in time O(n log(n)) even in the worst case.

However, the beauty of quicksort lies in its simplicity. And it turns out that even in its simple

form quicksort runs in O(n log(n)) on the average. Moreover, the constant hidden in the O-

notation is small. Therefore, we trade this for the (rare) worst case behavior of Θ(n2).

Proposition: The time complexity of quicksort is in

Θ(n log(n)) In the average case and in

Θ(n2) In the worst case

Conclusions

Quicksort turns out to be the fastest sorting algorithm in practice. It has a time complexity of

Θ(n log(n)) on the average. However, in the (very rare) worst case quicksort is as slow as

Bubblesort, namely in Θ(n2). There are sorting algorithms with a time complexity of O(n

log(n)) even in the worst case, e.g. Heapsort and Mergesort. But on the average, these

algorithms are by a constant factor slower than quicksort.

It is possible to obtain a worst case complexity of O(n log(n)) with a variant of quicksort (by

choosing the median as comparison element). But this algorithm is on the average and in the

http://www.iti.fh-flensburg.de/lang/algorithmen/asympen.htm

http://www.iti.fh-flensburg.de/lang/algorithmen/sortieren/networks/sortieren.htm#section3

http://www.iti.fh-flensburg.de/lang/algorithmen/sortieren/heap/heapen.htm

http://www.iti.fh-flensburg.de/lang/algorithmen/sortieren/merge/mergen.htm



worst case by a constant factor slower than Heapsort or Mergesort; therefore, it is not

interesting in practice.

The quicksort algorithm was invented by C.A.R. Hoare in 1962. I learned about it from "The

C Programming Language, Second Edition" by Brian W. Kernighan and Dennis M. Ritchie

(page 87). I found the K&R code a bit hard to follow; I believe I've come upon a better way

of explaining the algorithm, using a few figures of speech.

The point of any sort routine is to take an array of objects and arrange the objects in some

kind of sequence. We might, for example, want to take this collection of numbers:

06 34 69 33 75 64 04 74 25 95 15 58 78 36 51 73 13 27

and put them into numerical order, with the smallest number first, like this:

04 06 13 15 25 27 33 34 36 51 58 64 69 73 74 75 78 95

The general approach of the quick sort is to select an element from the middle (or from close

to the middle) of an array and then put all other elements which are less than or equal to the

selected element to its left and all elements which are greater than the selected element to its

right. The selected element is called the "pivot element" because the other elements,

figuratively speaking, turn around it. This crude "sorting" around the pivot element yields

two sub-arrays: a left one and a right one. If, for example, we start with the original array

above and use "25" as the pivot element, our first sorting yields:

[ 04 06 15 13 ] 25 [ 64 34 74 69 95 33 58 78 36 51 73 75 27 ]

The next step is for quicksort to call itself to have the left and right sub-arrays sorted. Of

course, a "sub array" with zero elements or only one element is already in proper order and

does not need to be sorted. The interesting part of the quick sort is how it comes up with the

sub arrays. I will illustrate.

From our original array, take out the element in (or close to) the middle of the array for a

pivot element. This, figuratively speaking, leaves a hole in the array:



06 34 69 33 75 64 04 74 __ 95 15 58 78 36 51 73 13 27

p.e. = 25

Now "move the hole" to the first position in the array by moving the current occupant of that

position into the current hole's position:

__ 34 69 33 75 64 04 74 06 95 15 58 78 36 51 73 13 27

p.e. = 25

Now we get to work. We start with the first element to the right of the hole. If it is greater

than the pivot element, we do nothing. If it is less than or equal to the pivot element, then we

want to put the element in the hole and then re-establish the hole immediately to the right of

the moved element. The first element of our sample array that needs to be moved is "04":

+-----------------+

| |

V |

04 34 69 33 75 64 __ 74 06 95 15 58 78 36 51 73 13 27

+--------------+

| |

| V

04 _ 69 33 75 64 34 74 06 95 15 58 78 36 51 73 13 27

p.e. = 25

After we similarly process "06" we will have:

04 06 __ 33 75 64 34 74 69 95 15 58 78 36 51 73 13 27

p.e. = 25

At the end of the process, we have:



04 06 15 13 __ 64 34 74 69 95 33 58 78 36 51 73 75 27

p.e. = 25

As you can see, all elements to the left of the hole are less than the pivot element; all

elements to the right of the hole are greater than the pivot element. To finish up, we sort the

left sub-array then the right sub-array, then we stuff the pivot element back into the hole and

we are done.

Algorithm partition (a,m,p)

//Within a[m], a[m+1],…,a[p-1]the elements are rearranged in such

//a manner that if initially t=a[m], then after completion a[q] =t for

//some q between m and p-1,a[k]<=t form<=k<q, and a[k]>=t for

//q<k<p. q is returned. Set a[p]=.

v :=a[m]; I :=m; j :=p;

repeat

repeat

I:=i+1;

until ( a[i]>=v);

repeat

j:=j-1;

until(a[j]<=v);



if(i<j) then interchange (a,i,j);

until(i>=j);

a[m]:=a[j]; a[j]:=v; return j;

Algorithm interchange (a,i,j)

// exchange a[i] with a[j].

p :=a[i];

a[i] :=a[j]; a[j] :=p;

In quick sort, the division into sub arrays is made so that the

sorted sub arrays do not need to be merged later.

This is accomplished by rearranging the elements in a[1:n] such that a[i]<=a[j] for all I

between 1 and m and all j between m+1 and n

for some m,1<=m<=n. Thus the elements in a[1:m] and a[m+1:n] can be independently

sorted. No merge is needed. The rearrangement of

the element is accomplished by picking some element of a[],say t =a[s],and then reordering

the other elements so that all

elements appearing before t in a[1:n] are less than or equal to t and all elements appearing

after t are greater than or equal to t. This is rearranging is referred to as partitioning.

Function Partition of Algorithm accomplishes an in-place

partitioning of the elements of a[m:p-1].It is assumed that a[p]>=a[m] and that a[m] is the

partitioning elements. If m=1 and p-1=n, then a[n+1] must be defined and must be greater

than or equal to all elements in a[1:n]. The assumption that a[m] is the partition element is

merely for convenience; other choices for the partitioning element than the first item in the

set are better in practice. The function Interchange (a, i, j) exchanges a[i] with a[j].



Algorithm Quick Sort (p,q)

//Sorts the elements a[p],……,a[q] which reside in the global array //a[1:n] into ascending

order; a[n+1] is considered to be defined and //must be >= all the elements in a[1:n].

if (p<q) then //If there are more than one element

// divide P into two subproblems.

j:=Partition(a,p,q+1);

//j is the position of the partitioning element.

// Solve the subproblems.

QuickSort (p,j-1);

Quick Sort(j+1,q);

// There is no need for combining solutions.

Using Hoare‘s clever method of partitioning a set of elements about a chosen element, we

can directly devise a divide-and-conquer method for completely sorting n elements.

Following a call to the function Partition, two sets S1 and S2 are produced. All elements in

S1 are less than or equal to the element in S2. Hence S1 and S2 can be sorted independently.

Each set is sorted by reusing the function Partition



Quicksort

Quicksort

Quicksort in action on a list of numbers. The horizontal lines are pivot values.

Class Sorting algorithm

Data structure Varies

Worst case performance Θ(n2)

Best case performance Θ(nlogn)

Average case performance Θ(nlogn) comparisons

Worst case space complexity Varies by implementation

Optimal Sometimes

Quicksort is a well-known sorting algorithm developed by C. A. R. Hoare that, on average,

makes Θ(nlogn) (big O notation) comparisons to sort n items. However, in the worst case, it

makes Θ(n2) comparisons. Typically, quicksort is significantly faster in practice than other

Θ(nlogn) algorithms, because its inner loop can be efficiently implemented on most

architectures, and in most real-world data, it is possible to make design choices which

minimize the probability of requiring quadratic time.

http://en.wikipedia.org/wiki/File:Sorting_quicksort_anim.gif

http://en.wikipedia.org/wiki/Sorting_algorithm

http://en.wikipedia.org/wiki/Sorting_algorithm

http://en.wikipedia.org/wiki/C._A._R._Hoare

http://en.wikipedia.org/wiki/Average_performance

http://en.wikipedia.org/wiki/Big_O_notation

http://en.wikipedia.org/wiki/Best,_worst_and_average_case

http://en.wikipedia.org/wiki/Quadratic_time



Quicksort is a comparison sort and, in efficient implementations, is not a stable sort.

Algorithm

Quicksort sorts by employing a divide and conquer strategy to divide a list into two sub-lists.

The steps are:

1. Pick an element, called a pivot, from the list.

2. Reorder the list so that all elements which are less than the pivot come before the

pivot and so that all elements greater than the pivot come after it (equal values can go

either way). After this partitioning, the pivot is in its final position. This is called the

partition operation.

3. Recursively sort the sub-list of lesser elements and the sub-list of greater elements.

The base case of the recursion are lists of size zero or one, which are always sorted.

In simple pseudocode, the algorithm might be expressed as this:

function quicksort(array)

var list less, greater

if length(array) ≤ 1

return array

select and remove a pivot value pivot from array

for each x in array

if x ≤ pivot then append x to less

else append x to greater

return concatenate(quicksort(less), pivot, quicksort(greater))

Notice that we only examine elements by comparing them to other elements. This makes

quicksort a comparison sort. This version is also a stable sort (assuming that the "for each"

method retrieves elements in original order, and the pivot selected is the last among those of

equal value).

The correctness of the partition algorithm is based on the following two arguments:

http://en.wikipedia.org/wiki/Comparison_sort

http://en.wikipedia.org/wiki/Sorting_algorithm#Classification

http://en.wikipedia.org/wiki/Divide_and_conquer_algorithm

http://en.wikipedia.org/wiki/List_(computing)

http://en.wikipedia.org/wiki/Pivot_element

http://en.wikipedia.org/wiki/Recursion_(computer_science)

http://en.wikipedia.org/wiki/Base_case#Recursive_programming

http://en.wikipedia.org/wiki/Pseudocode

http://en.wikipedia.org/wiki/Comparison_sort



At each iteration, all the elements processed so far are in the desired position: before

the pivot if less than or equal to the pivot's value, after the pivot otherwise (loop

invariant).

Each iteration leaves one fewer element to be processed (loop variant).

The correctness of the overall algorithm follows from inductive reasoning: for zero or one

element, the algorithm leaves the data unchanged; for a larger data set it produces the

concatenation of two parts, elements less than or equal to the pivot and elements greater than

it, themselves sorted by the recursive hypothesis.

The disadvantage of the simple version above is that it requires Ω(n) extra storage space,

which is as bad as merge sort. The additional memory allocations required can also

drastically impact speed and cache performance in practical implementations. There is a more

complex version which uses an in-place partition algorithm and can achieve the complete sort

using O(nlogn) space use on average (for the call stack):

function partition(array, left, right, pivotIndex)

pivotValue := array[pivotIndex]

swap array[pivotIndex] and array[right] // Move pivot to end

storeIndex := left

for i from left to right − 1

if array[i] ≤ pivotValue

swap array[i] and array[storeIndex]

storeIndex := storeIndex + 1

swap array[storeIndex] and array[right] // Move pivot to its final place

return storeIndex

In-place partition in action on a small list. The boxed element is the pivot element, blue

elements are less or equal, and red elements are larger.

This is the in-place partition algorithm. It partitions the portion of the array between indexes

left and right, inclusively, by moving all elements less than or equal to a[pivotIndex] to the

beginning of the subarray, leaving all the greater elements following them. In the process it

also finds the final position for the pivot element, which it returns. It temporarily moves the

pivot element to the end of the subarray, so that it doesn't get in the way. Because it only uses

exchanges, the final list has the same elements as the original list. Notice that an element may

be exchanged multiple times before reaching its final place.

http://en.wikipedia.org/wiki/Merge_sort

http://en.wikipedia.org/wiki/In-place

http://en.wikipedia.org/wiki/Call_stack



This form of the partition algorithm is not the original form; multiple variations can be found

in various textbooks, such as versions not having the storeIndex. However, this form is

probably the easiest to understand.

Once we have this, writing quicksort itself is easy:

procedure quicksort(array, left, right)

if right > left

select a pivot index (e.g. pivotIndex := left)

pivotNewIndex := partition(array, left, right, pivotIndex)

quicksort(array, left, pivotNewIndex - 1)

quicksort(array, pivotNewIndex + 1, right)

However, since partition reorders elements within a partition, this version of quicksort is not

a stable sort.

Formal analysis

From the initial description it's not obvious that quicksort takes Θ(nlogn) time on average. It's

not hard to see that the partition operation, which simply loops over the elements of the array

once, uses Θ(n) time. In versions that perform concatenation, this operation is also Θ(n).

In the best case, each time we perform a partition we divide the list into two nearly equal

pieces. This means each recursive call processes a list of half the size. Consequently, we can

make only logn nested calls before we reach a list of size 1. This means that the depth of the

call tree is Θ(logn). But no two calls at the same level of the call tree process the same part of

the original list; thus, each level of calls needs only Θ(n) time all together (each call has some

constant overhead, but since there are only Θ(n) calls at each level, this is subsumed in the

Θ(n) factor). The result is that the algorithm uses only Θ(nlogn) time.

An alternate approach is to set up a recurrence relation for the T(n) factor, the time needed to

sort a list of size n. Because a single quicksort call involves Θ(n) factor work plus two

recursive calls on lists of size n / 2 in the best case, the relation would be:

The master theorem tells us that T(n) = Θ(nlogn).

http://en.wikipedia.org/wiki/Call_stack

http://en.wikipedia.org/wiki/Recurrence_relation

http://en.wikipedia.org/wiki/Master_theorem



In fact, it's not necessary to divide the list this precisely; even if each pivot splits the elements

with 99% on one side and 1% on the other (or any other fixed fraction), the call depth is still

limited to 100logn, so the total running time is still Θ(nlogn).

In the worst case, however, the two sublists have size 1 and n − 1 (for example, if the array

consists of the same element by value), and the call tree becomes a linear chain of n nested

calls. The ith call does Θ(n − i) work, and . The recurrence relation is:

T(n) = Θ(n) + T(0) + T(n − 1) = O(n) + T(n − 1)

This is the same relation as for insertion sort and selection sort, and it solves to T(n) = Θ(n2).

Given knowledge of which comparisons are performed by the sort, there are adaptive

algorithms that are effective at generating worst-case input for quicksort on-the-fly,

regardless of the pivot selection strategy.

Randomized quicksort expected complexity

Randomized quicksort has the desirable property that it requires only Θ(nlogn) expected

time, regardless of the input.

Suppose we sort the list and then divide it into four parts. The two parts in the middle will

contain the best pivots; each of them is larger than at least 25% of the elements and smaller

than at least 25% of the elements. If we could consistently choose an element from these two

middle parts, we would only have to split the list at most 2log2n times before reaching lists of

size 1, yielding an Θ(nlogn) algorithm.

A random choice will only choose from these middle parts half the time. However, this is

good enough. Imagine that you are flipping a coin over and over until you get k heads.

Although this could take a long time, on average only 2k flips are required, and the chance

that you won't get k heads after 100k flips is highly improbable. By the same argument,

quicksort's recursion will terminate on average at a call depth of only 2(2log2n). But if its

average call depth is Θ(logn), and each level of the call tree processes at most n elements, the

total amount of work done on average is the product, Θ(nlogn). Note that the algorithm does

not have to verify that the pivot is in the middle half - if we hit it any constant fraction of the

times, that is enough for the desired complexity.

http://en.wikipedia.org/wiki/Insertion_sort

http://en.wikipedia.org/wiki/Selection_sort

http://en.wikipedia.org/wiki/Expected_value



The outline of a formal proof of the O(nlogn) expected time complexity follows. Assume that

there are no duplicates as duplicates could be handled with linear time pre- and post-

processing, or considered cases easier than the analyzed. Choosing a pivot, uniformly at

random from 0 to n − 1, is then equivalent to choosing the size of one particular partition,

uniformly at random from 0 to n − 1. With this observation, the continuation of the proof is

analogous to the one given in the average complexity section.

http://en.wikipedia.org/wiki/Quicksort#Average_complexity



Average complexity

Even if pivots aren't chosen randomly, quicksort still requires only Θ(nlogn) time over all

possible permutations of its input. Because this average is simply the sum of the times over

all permutations of the input divided by n factorial, it's equivalent to choosing a random

permutation of the input. When we do this, the pivot choices are essentially random, leading

to an algorithm with the same running time as randomized quicksort.

More precisely, the average number of comparisons over all permutations of the input

sequence can be estimated accurately by solving the recurrence relation:

Here, n − 1 is the number of comparisons the partition uses. Since the pivot is equally likely

to fall anywhere in the sorted list order, the sum is averaging over all possible splits.

This means that, on average, quicksort performs only about 39% worse than the ideal number

of comparisons, which is its best case. In this sense it is closer to the best case than the worst

case. This fast average runtime is another reason for quicksort's practical dominance over

other sorting algorithms.

Space complexity

The space used by quicksort depends on the version used.

Quicksort has a space complexity of Θ(logn), even in the worst case, when it is carefully

implemented such that

in-place partitioning is used. This requires Θ(1).

After partitioning, the partition with the fewest elements is (recursively) sorted first,

requiring at most Θ(logn) space. Then the other partition is sorted using tail recursion

or iteration.

The version of quicksort with in-place partitioning uses only constant additional space before

making any recursive call. However, if it has made Θ(logn) nested recursive calls, it needs to

store a constant amount of information from each of them. Since the best case makes at most

Θ(logn) nested recursive calls, it uses Θ(logn) space. The worst case makes Θ(n) nested

recursive calls, and so needs Θ(n) space; Sedgewick's improved version using tail recursion

requires Θ(logn) space in the worst case.

http://en.wikipedia.org/wiki/Tail_recursion



We are eliding a small detail here, however. If we consider sorting arbitrarily large lists, we

have to keep in mind that our variables like left and right can no longer be considered to

occupy constant space; it takes Θ(logn) bits to index into a list of n items. Because we have

variables like this in every stack frame, in reality quicksort requires Θ((logn)2) bits of space

in the best and average case and Θ(nlogn) space in the worst case. This isn't too terrible,

though, since if the list contains mostly distinct elements, the list itself will also occupy

Θ(nlogn) bits of space.

The not-in-place version of quicksort uses Θ(n) space before it even makes any recursive

calls. In the best case its space is still limited to Θ(n), because each level of the recursion uses

half as much space as the last, and

Its worst case is dismal, requiring

space, far more than the list itself. If the list elements are not themselves constant size, the

problem grows even larger; for example, if most of the list elements are distinct, each would

require about ΘO(logn) bits, leading to a best-case Θ(nlogn) and worst-case Θ(n2logn) space

requirement.

Variants

There are three variants of quicksort that are worth mentioning:

Balanced quicksort: choose a pivot likely to represent the middle of the values to be

sorted, and then follow the regular quicksort algorithm.

External quicksort: The same as regular quicksort except the pivot is replaced by a

buffer. First, read the M/2 first and last elements into the buffer and sort them. Read

the next element from the beginning or end to balance writing. If the next element is

less than the least of the buffer, write it to available space at the beginning. If greater

than the greatest, write it to the end. Otherwise write the greatest or least of the buffer,

and put the next element in the buffer. Keep the maximum lower and minimum upper

keys written to avoid resorting middle elements that are in order. When done, write

the buffer. Recursively sort the smaller partition, and loop to sort the remaining

partition.

http://en.wikipedia.org/w/index.php?title=Balanced_quicksort&action=edit&redlink=1

http://en.wikipedia.org/w/index.php?title=External_quicksort&action=edit&redlink=1



Three-way radix quicksort (also called multikey quicksort): is a combination of radix

sort and quicksort. Pick an element from the array (the pivot) and consider the first

character (key) of the string (multikey). Partition the remaining elements into three

sets: those whose corresponding character is less than, equal to, and greater than the

pivot's character. Recursively sort the "less than" and "greater than" partitions on the

same character. Recursively sort the "equal to" partition by the next character (key).

Comparison with other sorting algorithms

Quicksort is a space-optimized version of the binary tree sort. Instead of inserting items

sequentially into an explicit tree, quicksort organizes them concurrently into a tree that is

implied by the recursive calls. The algorithms make exactly the same comparisons, but in a

different order.

The most direct competitor of quicksort is heapsort. Heapsort is typically somewhat slower

than quicksort, but the worst-case running time is always Θ(nlogn). Quicksort is usually

faster, though there remains the chance of worst case performance except in the introsort

variant. If it's known in advance that heapsort is going to be necessary, using it directly will

be faster than waiting for introsort to switch to it.

Quicksort also competes with mergesort, another recursive sort algorithm but with the benefit

of worst-case Θ(nlogn) running time. Mergesort is a stable sort, unlike quicksort and

heapsort, and can be easily adapted to operate on linked lists and very large lists stored on

slow-to-access media such as disk storage or network attached storage. Although quicksort

can be written to operate on linked lists, it will often suffer from poor pivot choices without

random access. The main disadvantage of mergesort is that, when operating on arrays, it

requires Θ(n) auxiliary space in the best case, whereas the variant of quicksort with in-place

partitioning and tail recursion uses only Θ(logn) space. (Note that when operating on linked

lists, mergesort only requires a small, constant amount of auxiliary storage.)

http://en.wikipedia.org/w/index.php?title=Three-way_radix_quicksort&action=edit&redlink=1

http://en.wikipedia.org/wiki/Radix_sort

http://en.wikipedia.org/wiki/Radix_sort

http://en.wikipedia.org/wiki/Binary_tree_sort

http://en.wikipedia.org/wiki/Heapsort

http://en.wikipedia.org/wiki/Introsort

http://en.wikipedia.org/wiki/Mergesort

http://en.wikipedia.org/wiki/Stable_sort

http://en.wikipedia.org/wiki/Linked_list

http://en.wikipedia.org/wiki/Disk_storage

http://en.wikipedia.org/wiki/Network_attached_storage



MODULE 3



THE GREEDY METHOD – CONTROL ABSTRACTION

The greedy method suggests that one can devise an algorithm that works in stages,

considering one input at a time. At each stage, a decision is made regarding whether a

particular input is in an optimal solution. This is done by considering the inputs in an order

determined by some selection procedure. If the inclusion of the next input into the partially

constructed optimal solution will result in an infeasible solution, then this input is not assed to

the partial solution. Otherwise, it is added. The selection procedure itself is based on some

optimization measure. The measure may be the objective function. In fact, several different

optimization measures may be plausible for a given problem. Most of these, however, will

result in algorithms that generate suboptimal solutions. This version of the greedy technique

is called the subset paradigm.

The function Select selects an input from a[ ] and removes it. The selected input‘s value is

assigned to x. Feasible is a Boolean-valued function that determines whether x can be

included into the solution vector. The function Union combines x with the solution and

updates the objective function. The function Greedy describes the essential way that a greedy

algorithm will look, once a particular problem is chosen and the functions Select, Feasible,

and Union are properly implemented.

Algorithm Greedy (a,n)

//a[1 : n] contains the n inputs.

Solution :=ø;// Initialize the solution.

for i:=1 to n do

x:=Select(a);

if Fesible(solution,x) then

solution := Union(solution,x);

return solution;



GENERAL KAPSACK PROBLEM

In the knapsack problem, we have n objects and a knapsack or a bag. Object

i has a weight wi and the knapsack has a capacity m. If a fraction xi , 0 ≤ xi ≤ 1, of object i

is placed into the knapsack, then a profit of pixi is earned. The objective is to obtain a filling

of the knapsack that maximizes the total profit earned. Since the knapsack capacity is

m, we require the total weight of all chosen objects to be at most

m. Formally, the problem can be stated as

maximize ∑ pixi

1≤ i≤n

Subject to ∑ wixi ≤ m

1≤i≤ n

And 0 ≤ xi ≤ 1, 1 ≤ i ≤ n

The profit and weight are positive numbers. This is done by using greedy method.

ALGORITHM

ALGORITHM greedyknapsack(m, n )

/ / p[ 1.. n] , w[ 1.. n]

/ / pi/ wi ≥ pi+1/ wi+1

/ / x[1 …. n] solution

for i=1 to n do

x[1] =0.0 ;

u=m ;

for i= 1 to n do

If ( w[i] > u) then break ;

x[i] = 1.0 ;

u = u – w[i] ;

If ( i ≤ n ) then x[i] = u/ w[i] ;



If p1 / w1 ≥ p2 /w2 ≥ ……………. ≥ pn / wn , then greedy knapsack generate an

Optimum solution to the given instant of knapsack problem. All optimum solutions

will fill the knapsack exactly complexity of knapsack algorithm is 0( n).

Greedy algorithms

Similarly to dynamic programming, greedy algorithms are used to solve optimization

problems. In contrast to dynamic programming, no global optimal solution is computed, but

rather locally optimal decisions are taken. That is why the approach is called greedy.

Example: Knapsack Problem I

2 possible greedy-strategies: select the most valuable element in each step that still fits

into the knapsack: Knapsack capacity: 15

Object Size Value

g1 3 3

g2 4 5

g3 6 8

g4 7 9

choose: g4; value: 9;

remaining capacity: 8

choose: g4; value: 9;

remaining capacity: 1

y not optimal

Example: Knapsack Problem II

select the relatively most valuable element (max( v(gi )s(gi ) ) )

This will lead to the optimal solution in the case described above. However, this need not

always be the case!

Knapsack capacity: 50

Object Size Value vs

g1 30 90 3

g2 40 100 2,5

g3 50 110 2,2

The strategy will choose

g1 (value 90).



However, the optimal would be g3 (value 110) Greedy algorithms do not always provide the

optimal solution. In exchange, they are far less expensive than dynamic

programming. Complexity of the greedy-knapsack: O(C)

Optimal greedy algorithms

There are problems where greedy algorithms produce optimal solutions. Example: resource

planning problem. Given: set of activities S = 1, . . . , n that want to use some resource, for

example a lecture hall. The resource can only be used by one activity at a time. Each activity i

has a starting time si and a completion time fi with si < fi . The activity takes place in the time

interval [si , fi ), i.e., the next activity can start at time fi . Compatibility of activities:

compatible(i , j) :, si _ fj _ sj _ fi . Problem: compute the largest possible set of compatible

Activitie

OPTIMAL STORAGE ON TAPES

Optimal storage on tapes I

Given: n programs P1, . . . , Pn with lengths l1, . . . , ln.

The programs are stored on a tape in order Pi1 , . . . , Pin .

Time to access Program Pij :

T(i1, . . . , in, j) = li1 + · · · + lij .

Average access time:

T(i1, . . . , in) = 1n

Xn

j=1

T(i1, . . . , in, j)

= 1n

Xn

j=1

Xj

i=1

lij

= 1n

Xn

j=1

(n − j + 1)lij

Optimal storage on tapes II

How to store the programs on the tape so that the average access time becomes minimal?



Let n be fixed; hence 1n can be ignored.

Example: Let P1, P2, P3 have lengths 17, 5, 10.

order acess time

P1P2P3 17 + (17 + 5) + (17 + 5 + 10) = 71

P1P3P2 17 + (17 + 10) + (17 + 10 + 5) = 76

P2P1P3 59

P3P1P2 69

P2P3P1 52

P3P2P1 52

Optimal storage on tapes III

Fact:

If for the lenghts l1, . . . , ln the relation l1 _ l2 _ · · · _ ln holds, then the average access time is

minimized by the order P1 . . . Pn. Hence, a greedy strategy that always selects the shortest

program leads to the minimal average access time.

What have we learnt on greedy algorithms?

Greedy algorithms treat optimization problems by taking locally optimal decisions. Hence,

they may fail to determine the global optimum. Greedy algorithms need less time than, e.g.,

dynamic programming. There are problems where a greedy strategy leads to an optimal

solution.

SPANNING TREES-MINIMUM COST SPANNING TREES

Definition: Let G= (V, E) be an undirected connected graph. A graph t= (V, E‘) of G is a

spanning tree of G if T is a tree.

Spanning trees have many applications. For example, they can be used to obtain

independent set of circuit equation for an electric network.

Another application of spanning trees arises from the property that a spanning tree is

a minimal subgraph G‘ of G such that V(G‘) =V(G) and G‘ is connected. A minimal

subgraph is one with fewest numbers of edges. Any connected graph with n vertices must

have least n-1 edges and all connected graphs with n-1 edges are trees. If the nodes of G

represent cities and the edges represent possible communication links connecting two cities,



then the minimum number of links needed to connect the cities is n-1. The spanning trees of

G represent all feasible choices.

In practical situations, the edges have weights assigned to them. These weights may

represent the cost of construction, the length of the link, and so on. Given such weighted

graph, one would then wish to select cities to have minimum total cost or minimum total

length. In either case the links selected have to form a tree. If this is not so, then selection of

links contain cycle. Removal of any of the links on this cycle results in a link selection of less

cost connecting all the cities. So that the minimum cost spanning tree of G can be obtained.

The cost of the spanning tree is the sum of all the edges in that tree.

6

1

3

2

7

5

4

28

8

16

10

12

18

14

25

22

24

(a)



Figure (a) represents a graph and (b) represents the minimum cost spanning tree.

PRIM’S ALGORITHM

At first a peak is chosen in random order ,which for simplicity we accept it as V(1).This way

two sets of pointers are initialized ,the 0=1 and P=2...n.

The O set (the O is taken from the Greek word Oristiko which means Terminal), will always

contain the pointers of those peaks which are terminally attached in the T tree. The V(1) peak

has already been attached in the T tree. The P set( P is taken from the Greek word Prosorino

which means Temporary) contains the rest of the pointers for the peaks, P=1...n-O which

are those pointers who have not been terminally connected with a node of T, that means they

are not attached in the tree.

In every execution of the Prim Algorithm a new peak will be connected to the T tree, not

always with their numbering order, for example the V(4) peak can be connected to the tree

before the V(2) peak. The corresponding pointer of the newly connected peak will be deleted

6

1

3

2

7

5

4

16

10

12 25

22

14

(b)



from P set and will be inserted to the O set. When all peaks are connected there will be

O=1,...n and P=0.This of course means the end of the algorithm.

The new peak every time will be chosen by using greedy method ,among all sides of G which

connect peaks already inserted in the T (pointers in the O set ) tree with the rest of the peaks

(pointers in the P set ),we choose one with minimum cost. If the chosen one is e(ij) then i

belongs in the O set , V(i) peak is already in the T tree, j belongs in the P set , and V(j) peak

has not been attached in the T tree yet. We put V(j) in the T tree, we change the O set by

putting the j pointer, and we also change the P set by removing the j pointer.

This may seem to you extremely complicated but it is easily understood by a set of examples.

Pseudocode For The Prim Algorithm.

INPUT :n,c[e(ij)],i,j belonging to 1,...,n.

OUTPUT :p(j) j=2,...,n (pointer of peaks j father in the T tree).

STEPS

1. :(initializations).

O=1 (V(1) root of the T tree).

P=2,...,n

For every j belonging to P :e(j):=c[e(j1)] , p(j)=1

( all peaks connected to the root. By definition of the cost function :e(j)=infinite

when V(j) does not connect to V(1).).

2. Choose a k for which e(k)<=e(j) for every j belonging to P In case of tight choose the

smaller one.

Exchange the O set with the set produced by the union of the O set and k . Exchange

the P set with the set produced by the difference of the P set and k .(P<-P-k) If

P=0 then stop.

3. For every j belonging to P compare e(j) with c[e(kj)].

If e(j) >c[e(kj)] exchange e(j) <-c(e(kj)).Go back to Step 1.



An example for Prim‘s algorithm

shortest paths from v0 to all destinations



(a)

6

1

3

2

7

5

4

6

1

3

2

7

5

4

10



(c)

6

1

3

2

7

5

4

10

12

(d)

6

1

3

2

7

5

4

10

12

14



(e)

6

1

3

2

7

5

4

16

10

12

14

(f)

6

1

3

2

7

5

4

16

10

12

14

22



Figure (a) shows the current graph with no edges selected. Edge (1, 6) is the first edge considered. It

is included in the spanning tree being built. This yields the graph of fig (b). Next edge (3, 4) is

selected and included in the tree (fig(c)). The next edge to be considered is (2, 7). Its inclusion in tree

is being built does not create a cycle, so we get the graph of fig (d). Edge (2, 3) is considered next and

included in the home fig (e). Of the edges not yet considered, (7, 4) has the least cost. It is considered

next. Its inclusion in the tree results in a cycle, so this edge is discarded. Edge (5,4) is next edge to be

added to the tree being built. This results in the configuration of fig (f). The next edge to be

considered is (7, 5). It is discarded as it creates a cycle. Finally, edge (6, 5) is considered and included

in the tree being built. This completes the spanning tree. The resulting tree has the cost 99.

KRUSKAL’S ALGORITHM

ALGORITHM

ALGORITHM Kruskal ( E, cost, n, t)

/ / E is the set of edges in G. G has n vertices. Cost [u, v] is the cost of

/ / edge (u, v). t is the set of edges in the minimum – cost spanning tree.

/ / the final cost is returned.

Construct a heap out of the edge costs using Heapify ;

for i := n do parent [i] := -1 ;

/ / Each vertex is in a different set.

i := 0 ; mincost := 0.0 ;

while ( i < n – 1 ) and ( heap not empty )) do

Delete a minimum cost edge (u, v) from the heap and

reheapify using Adjust ;

j := Find (u) ;

k := Find (v) ;

if (j ≠ k) then

i := i+1 ;

t[ i, 1] := u ; t [i, 2 ] := v ;

mincost := mincost + cost [u, v] ;



Union (j, k) ;

If ( i ≠ n – 1 ) then write (― no spanning tree‖) ;

else return mincost;

COMPLEXITY

The set t is the set of edges to be included in the minimum – cost spanning tree

and i is the number of edges in t. The set t can be represented as a sequential list using

a two – dimensional array t [ 1: n – 1, 1 : 2]. Edge (u, v) can be added to t by the

assignments t[ i, 1] := u ; and y[ i, 2] := v; . In the while loop of line 10, edges are

removed from the heap one by one in nondecreasing order of cost. Line 14 determines

the sets containing u and v. If j ≠ k then vertices u and v are in different sets and edge

(u, v) is included into t. The sets containing u and v are combined in line 20. If u = v

the edge (u, v) is discarded as its inclusions into t would create a cycle. Line 23

determines whether a spanning tree was found. It follows that i ≠ n – 1 iff the graph G

is not connected. The computing time is O( | E | log |E|), where e is the edge set of G.

JOB SEQUENCING WITH DEADLINES

If we are given a set of n jobs. Associated with job i is integer deadline di ≥0 and a profit

pi>0. For any job i the profit pi is earned if the job is completed by its deadline. To complete a

job one has to process the job on the machine for one unit of time. Only one machine is

available for processing jobs. A feasible solution for this problem is a subset J of such that

each job in this can be completed by its deadline. The value of a feasible solution J is the sum

of the profits of the jobs in J, or ∑i€J pi an optimal solution is a feasible solution with

maximum value.



Example:- Let n=4,(p1,p2,p3,p4)=(100,10,15,27) and (d1,d2,d3,d4)=(2,1,2,1). The feasible

solutions and their values are:

No Feasible solution Processing sequence Value

1. (1,2) 2,1 110

2. (1,3) 1,3 or 3,1 115

3. (1,4) 4,1 127

4. (2,3) 2,3 25

5. (3,4) 4,3 42

6. (1) 1 100

7. (2) 2 10

8. (3) 3 15

9. (4) 4 27

Solution 3 is optimal. In this solution only jobs 1 and 4 are processed and the value is 127.

These jobs must be processed in the order job 4 followed by job 1. Thus the processing of job

begins at time zero and that of job 1 is completed at time 2.

High level description of job sequencing algorithm

Algorithm Greedy job(d,J,n)

//J is a set of jobs that can be completed by their deadlines.

J:=1;

for i:=2 to n do

if(all jobs in J U i can be completed by their headlines) then J:=J U i;



Greedy algorithm for sequencing unit time jobs with deadlines and profits

Algorithm JS(d,j,n)

//d[i]≥1, 1≤i≤n are the deadlines,n≥1. The jobs are ordered such that p[1]≥p[2]≥...≥p[n].

//J[i] is the ith

job in the optimal solution,1≤i≤k.

//Also at termination d[J][i]]≤d[J[i+1]],1≤i≤k.

d[0] :=J[0]:=0;//initialize.

J[1]:=1;// Include job 1.

k:=1;

for i:=2 to n do

//consider jobs in nonincreasing order of p[i].

// Find position for i and check feasibility of insertion.

r:=k;

while((d[J[r]]>d[i]) and (d[J][r]]≠r)) do r:=r-1;

if((d[J[r]]≤d[i]) and (d[i]>r)) then

//Insert i into J[ ].

for q:=k to (r+1) step-1 do J[q+1]:=J[q];

J[r+1]:=i; k:=k+1;

return k;

The fuction JS is a correct implemention of the greedy-based method. Since d[i]≥1,the job

with largest pi will always be in the greedy solution. As the jobs are in the nonincreasing



order of pi‘s,line 6 in the above algorithm includes the job with largest pi. The for loop of the

line 8 considers the remaining job in the jobs in the order required by the greedy algorithm

method. At all times, the set of jobs already included in the solution is maintained in J. If

J[i],1≤i≤k, is the set already included,then J is such that d[J][i]]≤d[J[i+1]],1≤i≤k. When job i

is being considered, the while loop determines where in J this job has to be inserted.

The worst case computing time of Algorithm JS is θ(n2). The computing time of JS can be

reduced from θ(n2) to nearby O(n

2) by using disjoint set union and find algorithms and a

different method to determine the feasibility of a partial solution.



MODULE 4



DYNAMIC PROGRAMMING

Dynamic programming is an algorithm design method that can be used when the solution to a

problem can be viewed as the result of a sequence of decisions.Below are some of the

examples of problems that can be viewed this way.

1)KNAPSACK-The solution to the knapsack problem can be viewed as the result of a

sequence of decisions.We have to decide the values of xi,1 ≤ i ≤ n.An optimal sequence of

decisions maximizes the objective function ∑pixi.It also satisfies the constraints ∑wixi≤m and

0 ≤ xi ≤1.

2)OPTIMAL MERGE PATTERNS-An optimal merge pattern tells us which pair of files

should be merged at each step.An optimal sequence of decisions is a least cost sequence.

3)SHORTEST PATH-One way to find a shortest path from vertex i to vertex j in a directed

graph G is to decide which vertex should be the second vertex,which the third and so on.An

optimal sequence of decisions is one that results in a path of least length.

For some of the problems that may be viewed in this way,an optimal sequence

of decisions can be found by making the decisions one at a time and never making an

erroneous decision.This is true for all problems solvable by the greedy method.For many

problems it is not possible to make stepwise decisions in such a manner that the sequence of

decisions made is optimal.

One way to solve problems for which it is not possible to make a sequence of

stepwise decisions leading to an optimal decision sequence is to try all possible decision

sequences.We could enumerate all decision sequences and then pick out the best.But the time

and space requirements may be prohibitive.Dynamic programming often drastically reduces

the amount of enumeration by avoiding the enumeration of some decision sequences that

cannot possibly be optimal.In dynamic programming an optimal sequence of decisions is

obtained by making explicit appeal to the principle of optimality.

PRINCIPLE OF OPTIMALITY

The principle of optimality states that an optimal sequence of decisions has the property that

whatever the initial state and decisions are,the remaining decisions must constitute an optimal

decision sequence with regard to the state resulting from the first decision.



Thus, the essential difference between the greedy method and dynamic

programming is that in the greedy method only one decision sequence is ever generated.In

dynamic programming ,many decision sequences may be generated.However,sequences

containing suboptimal subsequences cannot be optimal(if the principle of optimality

holds)and so will not be generated as far as possible.

Example:SHORTEST PATH

Consider the shortest path problem.Assume that i,i1,i2,.........ik,j is a shortest

path from i to j.Starting with the initial vertex i,a decision has been made to go to vertex

i1.Following this decision the problem state is defined by vertex i1 and we need to find a path

from i1 to j.It is clear that the sequence i1,i2,.......ik,j must constitute a shortest i1 to j path.If not

,let i1,r1,r2,........rq,j be a shortest i1 to j path.Then i,i1,r1,.......,rq,j is an i to j path that is shorter

than the path i,i1,i2,.......,ik,j.Therefore the principle of optimality applies for this problem.

Because of the use of principle of optimality , decision sequences containing

subsequences that are suboptimal are not considered. Although the total number of different

decision sequences is exponential in the number of decisions(if there are d choices for each of

the n decisions to be made then there are dn possible sequences),dynamic programming

algorithms often have a polynomial complexity.

Another important feature of dynamic programming approach is that optimal

solutions to subproblems are retained so as to avoid recomputing their values.The use of

these tabulated values make it natural to recast the recursive equations into an iterative

algorithm.

MULTISTAGE GRAPHS

EXPLANATION

Theory

A multistage graph G = (V, E) is a directed graph in which the vertices

are partitioned into k>=2 disjoint sets Vi, 1<=i<=k. If (u, v) is an edge in E, then u ε Vi and

v ε Vi+1 for some i, 1<=i<k. The sets V1 and Vk are such that | V1|=|Vk|=1. Let s and t be

the vertex in V1 and Vk. The vertex s is the source, and t the sink. Let c (i,j) be the cost of

edge (i, j).The cost of a path from s to t is the sum of the costs of the edges on the path.



The multistage graph problem is to find a minimum-cost path from s to t. Each set Vi defines

a stage in the graph. Because of the constraints on E, every path from s to t starts

in stage 1, goes to stage 2, then to stage 3, then stage 4, etc., and eventually terminates in

stage k.

A dynamic programming formulation for a k-stage graph problem is obtained

by using the principle of optimality. According to this principle a path (i, k) is said to be

optimal if and only if the intermediate paths (i, j) and (j, k) are also optimal.

The multistage graph problem has two approaches namely, forward and

backward . In forward approach the resulting path is obtained by moving in the direction of

destination from source, even though it appears in reverse. In the latter case it from

destination to source tracing the lowest cost path.

Equation

The cost of a path (i, j) is given by the equation

cost (i, j) = min c (j, l) + cost (i+1, l)

l ε Vi+1

(j, l) ε E

where j is a vertex in Vi.

Here cost (i, j) means i th level and the cost of edge from j th node to the last node.

EXAMPLE

Consider the following figure. The graph is a 5 stage graph, with 12 vertices and

the cost of each node is given. The aim is to find the minimum cost path from the first node

to the last node.

Figure

V1 V2 V3 V4 V5



Cost Calculation

cost (4, 9) =4

cost (4,10) =2

cost (4,11) =5

cost (3,6) =cost from 6 th to 12 th node = min 5+ cost (4, 9),6+ cost (4, 10) =7

cost (3,7) = min 4+ cost (4, 9),3+ cost (4, 10) =5

cost (3,8) = min 5+ cost (4, 10),6+ cost (4, 11) =7

cost (2,2) = min 4+ cost (3, 6),2+ cost (3, 7), 1+ cost (3, 8) =7

cost (2,3) = min 2+ cost (3, 6),7+ cost (3, 7)=9

cost (2,4) = 11+cost (3, 8)=18

cost (2,5) = min 11+ cost (3, 7),8+ cost (3, 8)=15

cost (1,1) = min 9+ cost (2, 2),7+ cost (2,3), 11+ cost (2, 4) ,2+ cost (2, 5)=16

By selecting the least cost edges from each group we can determine which all edges

will be involved in the minimum cost path.

Thus the minimum cost path in forward approach is (1, 2, 7, 10, 12) and in

backward approach it is (1, 3, 6, 10, 12).Both have a cost of 16 each.

FORWARD APPROACH

EXPLANATION:

1

2

3

4

5

6

7

7

7

8

9

10

00

0

11

12

9

7

3

2

4

2

2

7

1

11

11

8

6

5

4

3

5

6

5

2

4



A multistage graph G = (V, E) is a directed graph in which the vertices are partitioned into

k>=2 disjoint sets Vi, 1<=i<=k. If (u, v) is an edge in E, then u ε Vi and v ε Vi+1 for some i,

1<=i<k. The sets V1 and Vk are such that | V1|=|Vk|=1. Let s and t be the vertex in V1 and

Vk. The vertex s is the source, and t the sink. Let c (i,j) be the cost of edge (i, j).The cost of a

path from s to t is the sum of the costs of the edges on the path.

The multistage graph problem is to find a minimum-cost path from s to t. Each set Vi defines

a stage in the graph. Because of the constraints on E, every path from s to t starts

in stage 1, goes to stage 2, then to stage 3, then stage 4, etc., and eventually terminates in

stage k.

A dynamic programming formulation for a k-stage graph problem is obtained by using the

principle of optimality. According to this principle a path (i, k) is said to be optimal if and

only if the intermediate paths (i, j) and (j, k) are also optimal.

The multistage graph problem has two approaches namely, forward and backward . In

forward approach the resulting path is obtained by moving in the direction of destination

from source, even though it appears in reverse. In the latter case it from destination to source

tracing the lowest cost path.

By using the forward approach,we obtain the cost of a path (i, j) which is given by the

following equation:

cost (i, j) = min c (j, l) + cost (i+1, l)

l ε Vi+1

(j, l) ε E , where j is a vertex in Vi.

Here cost (i, j) means i th level and the cost of edge from j th node to the last node.

Consider the example given below. This is a 5 stage graph, with 12 vertices and the cost of

each node is given. The aim is to find the minimum cost path from the first node to the last

node. This can be done by using the forward approach as shown below:

EXAMPLE:



V1 V2 V3 V4 V5

CALCULATING THE MINIMUM COST:

cost (4, 9) = cost from 4th node to the 9th node = 4

cost (4,10) = 2

cost (4,11) =5

cost (3,6) = min 6+ cost (4, 9), 6+ cost (4, 10) = 7



cost (2,2) = min 4+ cost (3, 6), 2+ cost (3, 7), 1+ cost (3, 8) = 7

While calculating the value of cost (2,2) the values of cost (3,6), cost (3,7) and cost (3,8) have

been reused so as to avoid their re-computation.

cost (2,3) = min 2+ cost (3, 6), 7+ cost (3, 7)= 9

cost (2,4) = min11+cost (3, 8)= 18

1

2

3

4

5

6

7

7

7

8

9

10

00

0

11

12

2

9

7

3

2

4

2

2

7

1

11

11

8

6

5

4

3

5

6

5

2

4



cost (2,5) = min 11+ cost (3, 7), 8+ cost (3, 8)= 15

cost (1,1) = min 9+ cost (2, 2), 7+ cost (2,3), 11+ cost (2, 4) , 2+ cost (2, 5)= 16

A minimum cost s to t path has a cost of 16.

Thus, the cost of forward approach is= 16.

By selecting the least cost edges from each group we can determine which all edges will be

involved in the minimum cost path.

Thus the minimum cost path in forward approach is (1, 2, 7, 10, 12) which has the cost of 16.

It is represented using dotted lines.

This is the multistage graph forward approach problem.

BACKWARD APPROACH :

The Multistage graph problem can be solved using backward approach in the

following manner. Let bp(i,j) be a minimum-cost path from vertex s to a vertex j in Vi . let

bcost(i,j) be the cost of bp(i,j). From the backward approach we obtain

bcost(i,j) = min bcost(i-1,l) + c(l,j)

l Є Vi-1

<l,j> Є E

Algorithm in psedocode corresponding to obtain a minimum-cost s-t graph is

BGraph.(Algorithm 1.1) The first subscript on bcost, p and d are omitted for the same reasons

as before. This algorithm has the same complexity as FGraph provided G is now represented

by itsinverse adjacency lists (i.e., for each vertex v we have a list of vertices w such that

<w,v> Є E



ALGORITHM

1. Algorithm BGraph(G,k,n,p)

2. // Same function as FGraph

3.

4. bcost[1] :=0.0; <r,j>

5. for j :=2 to n do

6. / / Compute bcost[j].

7. Let r be such that is an edge of G is an edge of and bcost is an edge of

8. G and bcost[r] + c[r,j] is minimum;

9. bcost[j] :=bcost[r] + c[r,j];

10. d[j] := r;

11.

12. / / Find a minimum-cost path.

13. p[1] := 1; p[k] :=n;

14. for j := k – 1 to 2 do p[j] := d[p[j+1]];

15.

It should be easy to see that both FGraph and BGraph work correctly even on a more

generalized version of multistage graphs. In this generalization, the graph is permitted to have

edges <u,v> such that u Є Vi , v Є Vj and i< j.

In the pseudocodes FGraph and BGraph, bcost(i,j) is set to ∞ for any <i,j> Є E. When

programming these pseudocodes, one could use the maximum allowable floating point

number for ∞. If the weight of any such edge is added to some other costs, a floatin point

overflow might occur. Care should be taken to avoid such overflows.

Eg:

d(B, T) = min9+d(D, T), 5+d(E, T), 16+d(F, T)

= min9+18, 5+13, 16+2 = 18.

d(C, T) = min 2+d(F, T) = 2+2 = 4



d(S, T) = min1+d(A, T), 2+d(B, T), 5+d(C, T)

= min1+22, 2+18, 5+4 = 9.

The above way of reasoning is called

backward reasoning.

Backward approach

(forward reasoning)

d(S, A) = 1

d(S, B) = 2

d(S, C) = 5

d(S,D)=mind(S, A)+d(A, D),d(S, B)+d(B, D)

= min 1+4, 2+9 = 5

d(S,E)=mind(S, A)+d(A, E),d(S, B)+d(B, E)

= min 1+11, 2+5 = 7

d(S,F)=mind(S, A)+d(A, F),d(S, B)+d(B, F)

= min 2+16, 5+2 = 7

d(S,T) = mind(S, D)+d(D, T),d(S,E)+d(E,T), d(S, F)+d(F, T)

= min 5+18, 7+13, 7+2

= 9

B T5

E

D

F

9

16d(F, T)

d(E, T)

d(D, T)



Forward approach and backward approach:

Note that if the recurrence relations are formulated using the forward approach

then the relations are solved backwards . i.e., beginning with the last decision

On the other hand if the relations are formulated using the backward approach,

they are solved forwards.

Sample Graph and Simulation of the Algorithm

SIMULATION OF THE SOLUTION USING BACKWARD COSTS

Format: COST(stage, node) = minimum cost of traveling to the node in stage from the

source(node 1).

STEP I

COST(I,1) = 0

STEP II

COST(II,2) = COST(I,1) + cost(1,2) = 0 + 10 = 10

Stage

I

Stage

II

Stage

III

Stage

IV

Stage

V

Stage

VI

11

2

4

3

7

6

5

1

10

00

0

9

8

14

13

12

15





STEP III

COST(III,5) = minCOST(II,2) + cost(2,5),

COST(II,3) + cost(3,5),

COST(II,4) + cost(4,5)

=min10 +10, 20+40, 30+∞

= 20---via the path 1-2-5




=min10 +20, 20+∞, 30+40





=min10 +30, 20+50, 30+30


STEP IV

COST(IV,8)=minCOST(III,5) + cost(5,8),

COST(III,6) + cost(6,8),

COST(III,7) + cost(7,8)

=min20+10, 20+∞ ,40+∞



=30---via the path 1-2-5-8




=min20+20, 20+20 ,40+∞

=40---via the path 1-2-5-9 or via the path 1-3-6-9




=min20+10, 20+∞ ,40+∞

=30---via the path 1-2-5-10




=min20+30, 20+30 ,40+30

=50---via the path 1-2-5-11 or via the path 1-3-6-11

STEP V

COST(V,12)=minCOST(IV,8) + cost(8,12),

COST(IV,9) + cost(9,12),

COST(IV,10) + cost(10,12),

COST(IV,11) + cost(11,12)

=min30+10 ,40+∞ ,30+∞,50+∞

= 40—via the path 1-2-5-8-12





COST(IV,10) + cost(10,13),


=min30+20 ,40+20 ,30+10,50+10

= 40—via the path 1-2-5-8-13



COST(IV,10) + cost(10,14),


=min30+30 ,40+10 ,30+20,50+30

=50---via the path 1-2-5-10-14 or via 1-2-5-9-14 or 1-3-6-9-14

STEP VI

COST(VI,15)=minCOST(V,12)+cost(12,15),

COST(V,13)+cost(13,15),

COST(V,14)+cost(14,15)

=min40+20 ,40+10 ,50+30

=50---via the path 1-2-5-8-13-15



ALL PAIRS SHORTEST PATH

Finding all pairs shortest path consists of finding the shortest distance between every pair of

nodes in a possibly directed graph. Various means of doing so are known, and the following

list gives a few of the common methods:

Floyd-Warshall algorithm is an elegant, quickly implementable O(n3) algorithm

(Assumes absence of negatively-weighed cycles).

Johnson's algorithm is harder to implement, but might perform better for sparse

graphs.

Algorithms for single source problem might also be repeatedly used, although they

often perform worse or are harder to optimize

FLOYD-WARSHELL ALGORITHM

In computer science, the Floyd–Warshall algorithm (sometimes known as the WFI

Algorithm or Roy–Floyd algorithm, since Bernard Roy described this algorithm in 1959) is

a graph analysis algorithm for finding shortest paths in a weighted, directed graph. A single

execution of the algorithm will find the shortest paths between all pairs of vertices. The

Floyd–Warshall algorithm is an example of dynamic programming

Algorithm

The Floyd-Warshall algorithm compares all possible paths through the graph between each

pair of vertices. It is able to do this with only V3 comparisons. This is remarkable considering

that there may be up to V2 edges in the graph, and every combination of edges is tested. It

does so by incrementally improving an estimate on the shortest path between two vertices,

until the estimate is known to be optimal.

Consider a graph G with vertices V, each numbered 1 through N. Further consider a function

shortestPath(i,j,k) that returns the shortest possible path from i to j using only vertices 1

through k as intermediate points along the way. Now, given this function, our goal is to find

the shortest path from each i to each j using only nodes 1 through k + 1.

There are two candidates for this path: either the true shortest path only uses nodes in the set

(1...k); or there exists some path that goes from i to k + 1, then from k + 1 to j that is better.

http://en.wikipedia.org/wiki/Node_(computer_science)

http://en.wikipedia.org/wiki/Graph_(mathematics)

http://en.wikipedia.org/wiki/Floyd-Warshall_algorithm

http://en.wikipedia.org/wiki/Johnson%27s_algorithm

http://en.wikipedia.org/wiki/Dense_graph

http://en.wikipedia.org/wiki/Dense_graph

http://en.wikipedia.org/wiki/Shortest_path_problem

http://en.wikipedia.org/wiki/Computer_science

http://en.wikipedia.org/w/index.php?title=Bernard_Roy&action=edit&redlink=1

http://en.wikipedia.org/wiki/Graph_(mathematics)

http://en.wikipedia.org/wiki/Algorithm

http://en.wikipedia.org/wiki/Shortest_path_problem

http://en.wikipedia.org/wiki/Dynamic_programming



We know that the best path from i to j that only uses nodes 1 through k is defined by

shortestPath(i,j,k), and it is clear that if there were a better path from i to k + 1 to j, then the

length of this path would be the concatenation of the shortest path from i to k + 1 (using

vertices in (1...k)) and the shortest path from k + 1 to j (also using vertices in (1...k)).

Therefore, we can define shortestPath(i,j,k) in terms of the following recursive formula

This formula is the heart of Floyd Warshall. The algorithm works by first computing

shortestPath(i,j,1) for all (i,j) pairs, then using that to find shortestPath(i,j,2) for all (i,j) pairs,

etc. This process continues until k=n, and we have found the shortest path for all (i,j) pairs

using any intermediate vertices.

Pseudocode

Conveniently, when calculating the kth case, one can overwrite the information saved from

the computation of k − 1. This means the algorithm uses quadratic memory. Be careful to

note the initialization conditions:

1 /* Assume a function edgeCost(i,j) which returns the cost of the edge from i to j

2 (infinity if there is none).

3 Also assume that n is the number of vertices and edgeCost(i,i)=0

4 */

5

6 int path[][];

7 /* A 2-dimensional matrix. At each step in the algorithm, path[i][j] is the shortest path

8 from i to j using intermediate vertices (1..k-1). Each path[i][j] is initialized to

9 edgeCost(i,j) or infinity if there is no edge between i and j.

10 */

11

12 procedure FloydWarshall ()

13 for k: = 1 to n

http://en.wikipedia.org/wiki/Recursive



14 for each (i,j) in 1,..,n2

15 path[i][j] = min ( path[i][j], path[i][k]+path[k][j] );

Analysis

To find all n2 of from those of requires 2n

2 bit operations. Since we begin with

and compute the sequence of n zero-one matrices , , ..., ,

the total number of bit operations used is . Therefore, the complexity of

the algorithm is Θ(n3) and can be solved by a deterministic machine in polynomial time

Applications and generalizations

The Floyd–Warshall algorithm can be used to solve the following problems, among others:

Shortest paths in directed graphs (Floyd's algorithm).

Transitive closure of directed graphs (Warshall's algorithm). In Warshall's original

formulation of the algorithm, the graph is unweighted and represented by a Boolean

adjacency matrix. Then the addition operation is replaced by logical conjunction

(AND) and the minimum operation by logical disjunction (OR).

Finding a regular expression denoting the regular language accepted by a finite

automaton (Kleene's algorithm)

Inversion of real matrices (Gauss-Jordan algorithm).

Optimal routing. In this application one is interested in finding the path with the

maximum flow between two vertices. This means that, rather than taking minima as

in the pseudocode above, one instead takes maxima. The edge weights represent fixed

constraints on flow. Path weights represent bottlenecks; so the addition operation

above is replaced by the minimum operation.

Testing whether an undirected graph is bipartite.

Fast computation of Pathfinder Networks.

Maximum Bandwidth Paths in Flow Networks

http://en.wikipedia.org/wiki/Computational_complexity_theory

http://en.wikipedia.org/wiki/Big_theta

http://en.wikipedia.org/wiki/Deterministic_Turing_machine

http://en.wikipedia.org/wiki/Polynomial_time

http://en.wikipedia.org/wiki/Transitive_closure

http://en.wikipedia.org/wiki/Logical_conjunction

http://en.wikipedia.org/wiki/Logical_disjunction

http://en.wikipedia.org/wiki/Regular_expression

http://en.wikipedia.org/wiki/Regular_language

http://en.wikipedia.org/wiki/Finite_automaton

http://en.wikipedia.org/wiki/Finite_automaton

http://en.wikipedia.org/wiki/Invertible_matrix

http://en.wikipedia.org/wiki/Real_number

http://en.wikipedia.org/wiki/Matrix_(mathematics)

http://en.wikipedia.org/wiki/Gauss-Jordan_elimination

http://en.wikipedia.org/wiki/Undirected_graph

http://en.wikipedia.org/wiki/Bipartite_graph

http://en.wikipedia.org/wiki/Pathfinder_Networks



JOHNSON’S ALGORITHM

Johnson's algorithm is a way to find shortest paths between all pairs of vertices in a sparse

directed graph. It allows some of the edge weights to be negative numbers, but no negative-

weight cycles may exist.

Johnson's algorithm consists of the following steps:

1. First, a new node q is added to the graph, connected by zero-weight edge to each other

node.

2. Second, the Bellman-Ford algorithm is used, starting from the new vertex q, to find

for each vertex v the least weight h(v) of a path from q to v. If this step detects a

negative cycle, the algorithm is terminated.

3. Next the edges of the original graph are reweighted using the values computed by the

Bellman-Ford algorithm: an edge from u to v, having length w(u,v), is given the new

length w(u,v) + h(u) −h(v).

4. Finally, for each node s, Dijkstra's algorithm is used to find the shortest paths from s

to each other vertex in the reweighted graph.

In the reweighted graph, all paths between a pair s and t of nodes have the same quantity

h(s) -h(t) added to them, so a path that is shortest in the original graph remains shortest in the

modified graph and vice versa. However, due to the way the values h(v) were computed, all

modified edge lengths are non-negative, ensuring the optimality of the paths found by

Dijkstra's algorithm. The distances in the original graph may be calculated from the distances

calculated by Dijkstra's algorithm in the reweighted graph by reversing the reweighting

transformation.

Analysis

The time complexity of this algorithm, using Fibonacci heaps in the implementation of

Dijkstra's algorithm, is O(V2log V + VE): the algorithm uses O(VE) time for the Bellman-Ford

stage of the algorithm, and O(V log V + E) for each of V instantiations of Dijkstra's algorithm.

http://en.wikipedia.org/wiki/Shortest_path

http://en.wikipedia.org/wiki/All-pairs_shortest_path_problem

http://en.wikipedia.org/wiki/Sparse_graph

http://en.wikipedia.org/wiki/Directed_graph

http://en.wikipedia.org/wiki/Weighted_graph

http://en.wikipedia.org/wiki/Negative_number

http://en.wikipedia.org/wiki/Cycle_(graph_theory)

http://en.wikipedia.org/wiki/Vertex_(graph_theory)

http://en.wikipedia.org/wiki/Edge_(graph_theory)

http://en.wikipedia.org/wiki/Bellman-Ford_algorithm

http://en.wikipedia.org/wiki/Dijkstra%27s_algorithm

http://en.wikipedia.org/wiki/Time_complexity

http://en.wikipedia.org/wiki/Fibonacci_heap



Thus, when the graph is sparse, the total time can be faster than the Floyd-Warshall

algorithm, which solves the same problem in time O(V3).

Example

The first three stages of Johnson's algorithm are depicted in the illustration below.

The graph on the left of the illustration has two negative edges, but no negative cycles. At the

center is shown the new vertex q, a shortest path tree as computed by the Bellman-Ford

algorithm with q as starting vertex, and the values h(v) computed at each other node as the

length of the shortest path from q to that node. Note that these values are all non-positive,

because q has a length-zero edge to each vertex and the shortest path can be no longer than

that edge. On the right is shown the reweighted graph, formed by replacing each edge weight

w(u,v) by w(u,v) + h(u) −h(v). In this reweighted graph, all edge weights are non-negative,

but the shortest path between any two nodes uses the same sequence of edges as the shortest

path between the same two nodes in the original graph. The algorithm concludes by applying

Dijkstra's algorithm to each of the four starting nodes in the reweighted graph.



http://en.wikipedia.org/wiki/File:Johnson's_algorithm.svg



SHORTEST PATH PROBLEM

A graph with 6 vertices and 7 edges

In graph theory, the shortest path problem is the problem of finding a path between two

vertices (or nodes) such that the sum of the weights of its constituent edges is minimized. An

example is finding the quickest way to get from one location to another on a road map; in this

case, the vertices represent locations and the edges represent segments of road and are

weighted by the time needed to travel that segment.

Formally, given a weighted graph (that is, a set V of vertices, a set E of edges, and a real-

valued weight function f : E → R), and one element v of V, find a path P from v to each v' of

V so that

is minimal among all paths connecting v to v' .

The problem is also sometimes called the single-pair shortest path problem, to distinguish

it from the following generalizations:

The single-source shortest path problem, in which we have to find shortest paths

from a source vertex v to all other vertices in the graph.

The single-destination shortest path problem, in which we have to find shortest

paths from all vertices in the graph to a single destination vertex v. This can be

reduced to the single-source shortest path problem by reversing the edges in the

graph.

http://en.wikipedia.org/wiki/File:6n-graf.svg

http://en.wikipedia.org/wiki/File:6n-graf.svg

http://en.wikipedia.org/wiki/Graph_theory

http://en.wikipedia.org/wiki/Path_(graph_theory)

http://en.wikipedia.org/wiki/Vertex_(graph_theory)

http://en.wikipedia.org/wiki/Glossary_of_graph_theory#Weighted_graphs_and_networks

http://en.wikipedia.org/wiki/Real_number



The all-pairs shortest path problem, in which we have to find shortest paths

between every pair of vertices v, v' in the graph.

These generalizations have significantly more efficient algorithms than the simplistic

approach of running a single-pair shortest path algorithm on all relevant pairs of vertices.

Algorithms

The most important algorithms for solving this problem are:

Dijkstra's algorithm solves the single-pair, single-source, and single-destination

shortest path problems.

Bellman-Ford algorithm solves single source problem if edge weights may be

negative.

A* search algorithm solves for single pair shortest path using heuristics to try to speed

up the search.

Floyd-Warshall algorithm solves all pairs shortest paths.

Johnson's algorithm solves all pairs shortest paths, and may be faster than Floyd-

Warshall on sparse graphs.

Perturbation theory finds (at worst) the locally shortest path.

Applications

Shortest path algorithms are applied to automatically find directions between physical

locations, such as driving directions on web mapping websites like Mapquest or Google

Maps.

If one represents a nondeterministic abstract machine as a graph where vertices describe

states and edges describe possible transitions, shortest path algorithms can be used to find an

optimal sequence of choices to reach a certain goal state, or to establish lower bounds on the

time needed to reach a given state. For example, if vertices represents the states of a puzzle

like a Rubik's Cube and each directed edge corresponds to a single move or turn, shortest

path algorithms can be used to find a solution that uses the minimum possible number of

moves.

http://en.wikipedia.org/wiki/Dijkstra%27s_algorithm

http://en.wikipedia.org/wiki/Bellman-Ford_algorithm

http://en.wikipedia.org/wiki/A*_search_algorithm


http://en.wikipedia.org/wiki/Johnson%27s_algorithm

http://en.wikipedia.org/wiki/Sparse_graph

http://en.wikipedia.org/wiki/Perturbation_theory

http://en.wikipedia.org/wiki/Web_mapping

http://en.wikipedia.org/wiki/Mapquest

http://en.wikipedia.org/wiki/Google_Maps

http://en.wikipedia.org/wiki/Google_Maps

http://en.wikipedia.org/wiki/Abstract_machine

http://en.wikipedia.org/wiki/Rubik%27s_Cube



In a networking or telecommunications mindset, this shortest path problem is sometimes

called the min-delay path problem and usually tied with a widest path problem. For example,

the algorithm may seek the shortest (min-delay) widest path, or widest shortest (min-delay)

path.

A more lighthearted application is the games of "six degrees of separation" that try to find the

shortest path in graphs like movie stars appearing in the same film.

Other applications include "operations research, plant and facility layout, robotics,

transportation, and VLSI design".

Related problems

For shortest path problems in computational geometry, see Euclidean shortest path.

The traveling salesman problem is the problem of finding the shortest path that goes through

every vertex exactly once, and returns to the start. Unlike the shortest path problem, this

problem is NP-complete and, as such, is believed not to be efficiently solvable (see P = NP

problem) . The problem of finding the longest path in a graph is also NP-complete.

The Canadian traveller problem and the stochastic shortest path problem are generalizations

where either the graph isn't completely known to the mover, changes over time, or where

actions (traversals) are probabilistic.

The problems of recalculation of shortest paths arises if some graph transformations (e.g.,

shrinkage of nodes) are made with a graph

http://en.wikipedia.org/w/index.php?title=Widest_path_problem&action=edit&redlink=1

http://en.wikipedia.org/wiki/Six_degrees_of_separation

http://en.wikipedia.org/wiki/Operations_research

http://en.wikipedia.org/wiki/Robotics

http://en.wikipedia.org/wiki/Transportation

http://en.wikipedia.org/wiki/Very-large-scale_integration

http://en.wikipedia.org/wiki/Computational_geometry

http://en.wikipedia.org/wiki/Euclidean_shortest_path

http://en.wikipedia.org/wiki/Traveling_salesman_problem

http://en.wikipedia.org/wiki/NP-complete

http://en.wikipedia.org/wiki/P_%3D_NP_problem

http://en.wikipedia.org/wiki/P_%3D_NP_problem

http://en.wikipedia.org/wiki/Longest_path_problem

http://en.wikipedia.org/wiki/Canadian_traveller_problem



LOWER BOUND THEORY

Our main task for each problem is to obtain a correct and efficient solution. If two algorithms

solving the same problem were discovered and their times differed by an order of magnitude,

then the one with the smaller order was generally regarded as superior. To establish a given

algorithm in the most efficient way, a function g(n) , that is a lower bound on the time, is

discovered. If we have an algorithm whose computing time is the same order as g(n), then we

know that asymptotically we can do no better.

There is a mathematical notation for expressing lower bounds. If f(n) is the time for some

algorithm, then we write f(n)=Ω(g(n)) to mean that g(n) is a lower bound for f(n). Formally

this equation can be written if there exist positive constants c and n0 such that |f(n)| ≥ c|g(n)|

for all n > n0. In addition to developing lower bounds to within a constant factor, we are also

concerned with determining more exact bounds.

Deriving good lower bounds is often more difficult than devising efficient algorithms.

Perhaps this is because a lower bound states a fact about all possible algorithms for solving a

problem. Usually we cannot enumerate and analyze all these algorithms, so lower bound

proofs are often hard to obtain.

However , for many problems it is possible to easily observe that a lower bound identical to n

exists, where n is the number of inputs( or possibly outputs) to the problem. For example,

consider all algorithms that find the maximum of an unordered set of n integers. Clearly

every integer must be examined at least once, so Ω(n) is a lower bound for any algorithm that

solves this problem. Or, suppose we wish to find an algorithm that efficiently multiplies two

n×n matrices. Then Ω(n2) is a lower bound on any such algorithm since there are 2n

2 inputs

that must be examined and n2 outputs that must be computed. Bounds such as these are often

reffered to as trivial lower bounds because they are so easy to obtain. We know how to find

the maximum of n elements by an algorithm that uses only n-1 comparisons so there is no

gap between the upper and lower bounds for this problem. But for matrix multiplication the

best known algorithm requires O(n2+€

) operations (€ > 0) , and so there is no reason to believe

that a better method cannot be found.



COMPARISON TREES FOR SEARCHING AND SORTING

A comparison tree for binary search on an array is a binary tree that depicts all possible

search paths.i.e it shows all different sequences of comparisons undertaken by binary search

when searching for keys that may or may not be present in the array.

Suppose that we are given a set S of distinct values on which an ordering relation < holds.

The sorting problem calls for determining a permutation of the integers 1 to n, say p(1) to

p(n), such that the n distinct values from S stored in A[1:n] satisfy A[p(1)] < A[p(2)] <. .. .<

A[p(n)]. The ordered searching problem asks whether a given element x E S occurs within

the elements in A[1:n], then we are to determine an I between 1 and n such that A[i] = x. The

merging problem assumes that two ordered sets of distinct inputs from S are given in A[1:m]

and B[1:n] such that A[1] < . . . < A[m] and B[1] < . . . B[n]; these m+n values are to be

rearranged into an array C[1:m+n] so that C[1] < . . . < C[m+n]. For all these problems we

restrict the class of algorithms we are considering to those which work solely by making

comparisons between elements. No arithmetic involving elements is permitted, though it is

possible for the algorithm to move elements around.Thes algorithms are referred to as

X:A(1)

X: A(2)

X:A(n)

Failure

Failure

Failure

X:A(1)

X: A(2) Failure

Failure

X:A(1)

X: A(2) Failure

Failure X:A(n)

X:A(1)

X: A(2) Failure

Failure X:A(n)

X:A(1)

X: A(2) Failure

Failure

Failure Failure

X:A(n)

X:A(1)

X: A(2) Failure

Failure



comparison based algorithms. We rule out algorithms such as radix sort that decompose the

values into subparts.

a) Ordered Searching

In obtaining the lower bound for the ordered searching problem, we consider

only those comparison based algorithms in which every comparison between two elements of

S is of the type ―compare x and A[i]‖. Each internal node in the binary tree represents a

comparison between x and an A[i], or x > A[i].If x=A[i],the algorithm terminates. The left

branch is taken if x < A[i], and the right is taken if x > A[i].If the algorithm terminates

following a left or right branch , then no I has been found such that x = A[i] and the algorithm

must declare the search successful.

b) Sorting

We can describe any sorting algorithm that satisfies the restrictions of the

comparison tree model by a binary tree. Consider the case in which the n numbers A[1:n] to

be sorted are distinct.Now,any comparison between A[i] and A[j] must result in one of two

possibilities: either A[i] < A[j] or A[i] > A[j].So, the comparison tree is a binary tree in which

each internal node is labeled by the pair i:j which represents the comparison of A[i] with

A[j].If A[i] < A[j] , then the algorithm proceeds down the left branch of the tree; otherwise

right branch. The external nodes represent termination of the algorithm. Associate with every

path from the root to an external node is a unique permutation.

c) Selection

Any comparison tree that models comparison-based algorithms for finding

the maximum of n elements has at least 2^ n-1 external nodes since each path from the root to

an external node must contain at least n-1 internal nodes. This implies at least n-1

comparisons for otherwise at least two of the input items never lose a comparison and the

largest is not yet found.

Comparison trees

comparison tree

n! leaves : every permutation must be a leaf

-case # comparisons = height of tree



a1a2a3 : a1< a2?

a1a2a3 : a2<a3 ? a2a1a3 : a1< a3?

a1a2a3 a1a3a2 : a1< a3? a2a1a3 a2a3a1 : a2< a3?

a1a3a2 a3a1a2 a2a3a1 a3a2a1

Comparison tree for insertion sort of three items

LOWER BOUND FOR COMPARISON-BASED SORTING

It is generally much more difficult to show that an algorithm is ‗‗best possible‘‘ for a certain

task, than to come up with a

‗‗good‘‘ algorithm for that task. In the latter case, one need only argue about one algorithm—

the one that is supposedly

‗‗good‘‘; in the former case, one must argue about all possible algorithms — even those

which one knows nothing about!

To answer the question ‗‗what‘s the best way to do a job‘‘ we must start by fixing the

set of tools that may be used to do the job. For the problem at hand, sorting, we will take

pair wise comparisons of keys as the available tools. Of course, this leaves out of

consideration

Bin sorting (which is not based on key comparisons). But, in a sense, we can justify

this on the ground that Bin sorting is not a ‗‗general‘‘ sorting method: it is applicable only

if keys have a particular form. At any rate, it‘s important to keep in mind that the result



we are about to present only applies to sorting algorithms based on comparisons.

Suppose we want to sort n keys K1 , K2 , . . . , Kn . Let‘s assume that all keys are

distinct, so that for any Ki , Kj where i j , either Ki < Kj or Ki > Kj. The main theoretical

device we‘ll use to analyze our problem is a decision (or comparison) tree. This is

a useful way of representing any comparison-based algorithm that sorts n keys (for any

given n). Before introducing the formal definition, let‘s develop some intuition about

decision trees by studying an example.

Example 1: Below is a decision tree that corresponds to one possible algorithm for sorting

3 keys, K1 , K2 , K3 .

1

<

>

< >

<

>

< >

>

<

1:2

2:3

2:3

1:3

1:

3

1 2 3

3 2 1

2 3 1

3 1 2

1 3 2

2 1 3



The internal nodes of this tree correspond to comparisons the algorithm makes. The labels in

those nodes, specify which keys are to be compared. For example the label "1:2" of the root

indicates that K1 is to be compared with K2. Depending on the outcome of the comparison in

a node, the left or the right branch out of that node is taken. The label of each leaf is the

permutation specifying how to rearrange the keys to get them sorted. For example, consider

the node with label "2,3,1". This means that K2<K3<K1. Note that the fact this is the order of

the three keys is implied by the outcomes of the comparisons made in the path from the root

to that leaf: To get to that leaf we must go from the root to the right (signifying that K1>K2);

then from node "2:3" to the left (signifying that K2<K3) and from node "1:3" to the right

(signifying that K1>K3). These three comparisons imply that K2<K3<K1. This is the case for

the path from the root to any -2- leaf.

Thus the decision tree specifies the sequence of comparisons that the algorithm will perform

to sort 3 keys. An execution of the algorithm for a specific input (of 3 keys) corresponds to a

path from root to a leaf. For example, suppose that the input keys are K c, K a, K3=b

(where a < b < c). The execution starts at the root of the tree. The node there specifies that K1

and K2 are to be compared. Since K1>K2, we take the right branch from the root and arrive at

the "2:3" node. This indicates we must compare K2 to K3 and since K2<K3, we take the left

branch and arrive at the "1:3" node. By comparing K1 and K3 we discover that K1>K3 and

therefore we take the right branch which leads us to the leaf labeled "2,3,1". This indicates

that the keys in sorted order are K2<K3<K1, as indeed is the case. A more programming

language-like description of the algorithm corresponding to the above decision tree is:

if K1<K2 then

if K2<K3 then sorted order of keys is K1 , K2 , K3

else K2>K3


else K1>K3 sorted order of keys is K3 , K1 , K2

else K1>K2

if K2<K3 then




Using the intuition we have gained by studying this example, let us now give the formal



definition of a decision tree.

Definition 1: A decision tree of order n is a binary tree so that

(1) It has n! leaves, each labeled by a different permutation of 1, 2, . . . , n.

(2) Internal nodes are labeled by pairs of indices of the form ‗‗i : j i j n.

(3) In the path from the root to a leaf p (1) p (2) . . . p (n) (p is a permutation of

1, 2, . . . , n) there is either a node ‗‗p (i) : p (i — in which case the path goes

from that node to its left child — or ‗‗p (i i)‘‘ — in which case the path goes

from that node to its right child — i < n. (The path may contain other

nodes too but must contain at least these.)

The idea is that leaves correspond to the possible outcomes of sorting n distinct

keys. Internal node ‗‗i : j‘‘ corresponds to the comparison of Ki and Kj. If the outcome is

Ki < Kj then the left subtree of the node ‗‗i : j‘‘ contains the subsequent comparisons

made until the order of the keys is determined. Symmetrically, if the outcome is Ki > Kj ,

the right subtree of ‗‗i : j‘‘ contains the subsequent comparisons. Part (3) of the definition

essentially requires that every relationship determined by the algorithm must have been

established by actual comparisons: the algorithm cannot ‗‗guess‘‘.

Any comparison-based algorithm for sorting n keys corresponds to a decision tree of

order n. The execution of the algorithm on input sequence of keys K1, K2, . . . , Kn follows

the path from the root to the leaf labeled by the permutation p such that

Kp (1) < Kp (2) < . . . < Kp (n). Note that for a given sorting algorithm we need a different

-3-

decision tree for each n to represent all possible executions of that algorithm when the

input consists of n keys.

Example 2: Consider the algorithm for insertion sort:

INSERTION -SORT(A)

begin

for i := 2 to n do

for j := i down to 2 do

if A[ j] < A[ j 1] then A[ j A[ j 1]

else break

end



WORST CASE LOWER BOUND ON THE NUMBER OF

COMPARISONS

Let Cn denote the minimum number of comparisons required to sort n keys. For any decision

tree, the worst case number of comparisons required by the algorithm represented by that tree

is precisely the height of the tree.

Lemma: Any binary tree of height h has at most 2h leaves.

Proof: Trivial induction on h.

By definition, any decision tree of order n is a binary tree with n! leaves. Thus, by the

lemma, it must have height at least n Cn n

approximation

says that n n (n/e)n. Thus,

Cn n (n/e)n n/e)n = n log n n log e

from which it follows that Cn n log n).

AVERAGE CASE LOWER BOUND ON THE NUMBER OF

COMPARISONS

Consider a decision tree of order n representing some sorting algorithm. If we assume that all

initial arrangements of the keys to be sorted are equally likely, the average case number of

comparisons performed by that algorithm is equal to the external path length of the decision

tree divided by the number of leaves in the tree.† The following fact can be easily proved:

Fact 2: The tree that minimizes the external path length has all leaves in at most two

depths d and d 1, for some d.

By Facts 1 and 2 we may assume that in the decision tree that minimizes the average number

of comparisons, all leaves have depth d or d 1 for some d. Let Cn be the minimum average

case number of comparisons needed to sort n keys. Also, let Nd and Nd−1 be the number of

leaves in the corresponding decision tree at depth d and d 1 respectively.

Recall that the number of leaves in T is n!. Thus we have

Cn = [(d 1) Nd− d Nd ]/n! (1)

But we have:

Nd Nd−1 = n! (2)

and



Nd Nd−1 =2d (3)

Equation (2) says that the total number of leaves in T is n!. For equation (3) note that if

we gave 2 children to each leaf of depth d 1, we would get the maximum possible number

of nodes at depth d, which is 2d .

† Recall that the external path length of the tree is the sum of the depths of all leaves.

-5-

Solving (2) and (3) for Nd and Nd−1, we get:

Nd =2 n d (4)

and

Nd−1 =2d n! (5)

Substituting (4) and (5) in (1) we obtain:

Cn = [(d 1) Nd− d Nd ]/n!

= (d n n d )/n!

But d n = log n

Cn = (n! log n n n n! 2e )/n!

= log n

-

Cn n n(n/e)n n/e)n = n log n n log e.

Hence, Cn n log n).

Therefore, any comparison- n log n) comparisons

both in the worst and in the average case.

ORACLES AND ADVERSARY ARGUMENTS

One of the proof techniques that is useful for obtaining lower bounds consists of making use

of an oracle. The most famous oracle in the history was called the Delphic oracle, located in

Delphi in Greece. This oracle can still be found, situated in the side of a hill embedded in

some rocks. In olden times people would approach the oracle and ask it a question. After

some period of time elapsed, the oracle would reply and a caretaker would interpret the

oracle‘s answer.



A similar phenomenon takes place when we use an oracle to establish a lower bound. Given

some model of computation such as comparison trees, the oracle tells us the outcome of each

comparison. To derive a good lower bound, the oracle tries its best to cause the algorithm to

work as hard as it can. It does this by choosing as the outcome of the next test, the result that

causes the most work to be required to determine the final answer. And by keeping tracks of

the work that is done, a worst-case lower bound for the problem can be derived.

MERGING

Now we consider the merging problem. Given the sets A[l:m] and B[l:n], Where the item in

A and the item in B are sorted, we investigate lower bounds for algorithms that merge these

two sets to give a single sorted set. As was the case of sorting, we assume that all the m+n

elements are distinct and that A[1]<A[2]……..<A[m] and B[1]<B[2]…….<B[n]. It is

possible that after these two sets are merged, the n elements of B can be inter leaved within A

in every possible way. Elementary combinatorics tells us that there are ( ) ways that the

A‘s and B‘s can merge together while preserving the ordering within A and B. For example,

if m=3, n=2, A[i]=x,A[2]=y,A[3]=z,B[1]=u,B[2]=v, there are ( )=10 ways in which A and

B merge: u,v,x,y,z;u,x,v,y,z;u,x,y,v,z;u,x,y,z,u;x,u,v,y,z;x,u,y,v,z;x,u,y,z,v;x,y,u,v,z;x,y,u,z,v;

and x,y,u,v.

Thus if we use comparison trees as our model for merging algorithm, then there will be

external nodes, and therefore at least [log ] comparisons are required by any comparison-

based merging algorithm. The conventional merging algorithm takes m+n-1 comparisons. If

we let MERGE(m,n) be the minimum number of comarisons needed to merge m items, then

we have the inequality [log ] MERGE(m,n)-1

The exercises show that these upper and lower bounds can get arbitrarily far apart as m gets

much smaller than n. This should not be a surprise becoz the conventional algorithm is

designed to work best when m and n binary insertion would required the fewest number of

comparisons needed to merge A[1] into B[1],…..,B[n].

When m and n are equal, the lower bound given by the comparison tree model is too and the

number of comparisons for the conventional merging algorithm can be shown to be optimal.



Theorm: MERGE(m,m)=2m-1, for m≥1.

Proof: Consider any algorithm that merges the two sets A[1] <……..< A[m] and

B[1]<…….<B[m]. We already have an algorithm that requires 2m-1 comparisons. If we can

show that MERGE(m,m)≥2m-1, then the theorm follows. Consider any comparison-based

algorithm for solving A[1]<B[2]<A[2]<………<B[m]<A[m], that is, for which the B‘s and

A‘s alternate. Any merging algorithm must make each of the 2m-1 comparisons

B[1]:A[1],A[1]:B[2],B[2]:A[2],…..,B[m]:A[m] while merging the given input. To see this,

supose that a comparison of type B[i]:A[i] is not made for some i. Then the algorithm cannot

distinguish b/w the previous ordering and the one in which

B[1]<A[1]<……<A[i-1]<A[i]<B[i]<B[i+1]<…….<B[m]<A[m].

So the algorithm will not neccessarly merge the A‘s and B‘s properly. If a comparison of type

A[i]:B[i+1] is not made, then the algorithm will not be able to distinguish b/w the cases in

which B[1]<A[1]<B[2]<……<B[m]<A[m] and in which B[1]<A[1]<B[2]<A[2]……<A[i-

1]<B[i]<B[i+1]<A[i]<a[i+1]<…….<B[m]<A[m]. So any algorithm must make all 2m-1

comparisons to produce this final result. The theorm follows.

INSERTION SORT

If the first few objects are already sorted, an unsorted object can be inserted in the sorted set

in proper place. This is called insertion sort. An algorithm consider the elements one at a

time, inserting each in its suitable place among those already considered (keeping them

sorted).

Insertion sort is an example of an incremental algorithm; it builds the sorted sequence one

number at a time.

INSERTION_SORT (A)

1. For j = 2 to length [A] do

2. key = A[j]

3. Put A[j] into the sorted sequence A[1 . . j-1]

4. i ← j -1

5. while i > 0 and A[i] > key do

6. A[i+1] = A[i]



7. i = i-1

8. A[i+1] = key

Analysis

Best-Case

The while-loop in line 5 executed only once for each j. This happens if given array A is

already sorted.

T(n) = an + b = O(n)

It is a linear function of n.

Worst-Case

The worst-case occurs, when line 5 executed j times for each j. This can happens if

array A starts out in reverse order

T(n) = an2 + bc + c = O(n

2)

It is a quadratic function of n.



The graph shows the n2 complexity of the insertion sort.

Stability

Since multiple keys with the same value are placed in the sorted array in the same order that

they appear in the input array, Insertion sort is stable.

Extra Memory

This algorithm does not require extra memory.

For Insertion sort we say the worst-case running time is θ(n2), and the best-case

running time is θ(n).

Insertion sort use no extra memory it sort in place.

The time of Insertion sort is depends on the original order of a input. It takes a time in

Ω(n2) in the worst-case, despite the fact that a time in order of n is sufficient to solve

large instances in which the items are already sorted.



Implementation

void insertionSort(int numbers[], int array_size)

int i, j, index;

for (i=1; i < array_size; i++)

index = numbers[i];

j = i;

while ((j > 0) && (numbers[j-1] > index))

numbers[j] = numbers[j-1];

j = j - 1;

numbers[j] = index;



Adversary argument

• Suppose we have an algorithm we think efficient.

• Image an adversary tries to prove otherwise.

• At each point in the algorithm, whenever an decision (i.e., key comparison) is made, the

adversary tells us the result of the decision.

• The adversary chooses the answer which tries to force the algorithm work hard (i.e., do a lot

of decision, or to say, the answer releases as less new information as possible).

• You can think the adversary is constructing a ―bad‖ input while it is answering the

questions. The only requirement on the answers is that they must be internally consistent.

• If the adversary can force the algorithm to perform f(n) steps, then f(n) is the lower bound,

i.e, at least how many steps in the worst case.

Simply Put:

• Playing a guessing game between you and your friend.

– You are to pick up a date, and the friend will try to guess the date by asking YES/NO

questions.

– Your purpose is forcing your friend to ask as many questions as possible.

• To question ―is it in winter‖, your answer should be NO.

• To question ―is the first letter of the month‘s name in the first half of the alphabet‖? Your

answer should be YES.

• Idea:

– You did not pick up a date in advance at all, but

– Construct a date according to the questions. The

– The requirement is that the finally constructed date should be

consistent to all your answers to the questions.

– Looks like cheating, but it is a good way to find the lower bound.



Lower Bound for Algorithms that remove at most One Inversion per

Comparison

Insertion Sort, either does nothing or moves the key in the jth slot to the (j+1)st slot. By

moving the key in the jth slot up one slot, we have remedied the fact that ‗x‘ (number being

compared with) should come before the key. However, this is all we have accomplished. We

show that all sorting algorithms that sort only by comparisons of keys, and accomplish such a

limited amount of rearranging after each time, require at least quadratic time. We obtain our

results under the assumption that the keys to be sorted are distinct. Clearly, the worst case

bound still holds true with this restriction removed because a lower bound on the worst-case

performance from some inputs of some subsets is also a lower bound when all inputs are

considered.

In general, we are concerned with sorting n distinct keys that come from any ordered

set. However, without loss of generality, we can assume that the keys to be sorted are simply

the positive integers 1,2,…….,n, because we can substitute 1 for the smallest key, 2 for the

second smallest, and so on. For example, suppose we have alpha input, [Ralph, Clyde, Dave].

We can associate 1 with Clyde, 2 with Dave and 3 with Ralph. To obtain the equivalent input,

[3, 1, 2]. Any algorithm that sorts these integers only by comparison of keys would have to

do the same number of comparisons to sort the three names.

A permutation of the first n positive integers can be thought of as an ordering of those

integers. Because there are n! Permutations, of the first n positive integers. There are n!

different orderings of those integers. For example the following six permutations are all the

ordering of the first three positive integers:

[1,2,3] [1,3,2] [2,1,3] [2,3,1] [3,1,2] [3,2,1]

This means that n! Different inputs (to a sorting algorithm) containing n different

keys. These six permutations are the different inputs of size three (3).

We denote a permutation by [k1,k2,…………,kn]. That is ki is the integer at the ith position.

For the permutation [3, 1, 2], for example,

K1=3, k2=1, k3=2

R and s must be integers between 1 and n such that s > r.



Given a permutation, the pair (s, r) is an inversion in either the permutation or its transpose

but not both. Showing that there are n ( n - 1 ) / 2 such pairs of integers between 1 and n is

left as an exercise. This means that a permutation and its transpose have exactly, n ( n - 1 ) / 2

inversions between them. So the average no of inversions in a permutation and its transpose

is,

4

)1(

2

)1(*

2

1 nnnn

Therefore, we consider all permutations equally probable for the input, the average number of

inversions in the input is also n (n-1) / 4. Because we assume that the algorithm removes at

most one inversion after each comparison, on an average it must do at least this many

comparisons to remove all inversion sand thereby sort the input.

Insertion Sort removes at most the inversion consisting of S[j] and x after each comparison,

and therefore this algorithm is in the class of algorithms addressed by Theorem.

Because Insertion Sort‘s worst case time complexity is n ( n – 1 ) / 2 and its average – case

time complexity is about n2/4, its about as good as we can hope to do (as far as comparisons

of keys are concerned) with algorithms that sort only by comparisons of keys and remove at

least one inversion after each comparison.

SELECTION SORT

This type of sorting is called "Selection Sort" because it works by repeatedly element. It

works as follows: first find the smallest in the array and exchange it with the element in the

first position, then find the second smallest element and exchange it with the element in the

second position, and continue in this way until the entire array is sorted.

SELECTION_SORT (A)

for i ← 1 to n-1 do

min j ← i;

min x ← A[i]

for j ← i + 1 to n do

If A[j] < min x then

min j ← j



min x ← A[j]

A[min j] ← A [i]

A[i] ← min x

Selection sort is among the simplest of sorting techniques and it work very well for

small files. Furthermore, despite its evident "naïve approach "Selection sort has a quite

important application because each item is actually moved at most once, Section sort is a

method of choice for sorting files with very large objects (records) and small keys.

The worst case occurs if the array is already sorted in descending order. Nonetheless, the time

require by selection sort algorithm is not very sensitive to the original order of the array to be

sorted: the test "if A[j] < min x" is executed exactly the same number of times in every case.

The variation in time is only due to the number of times the "then" part (i.e., min j ← j; min x

← A[j] of this test are executed.

The Selection sort spends most of its time trying to find the minimum element in the

"unsorted" part of the array. It clearly shows the similarity between Selection sort and Bubble

sort. Bubble sort "selects" the maximum remaining elements at each stage, but wastes some

effort imparting some order to "unsorted" part of the array. Selection sort is quadratic in both

the worst and the average case, and requires no extra memory.

For each i from 1 to n - 1, there is one exchange and n - i comparisons, so there is a total

of n -1 exchanges and (n -1) + (n -2) + . . . + 2 + 1 = n(n -1)/2 comparisons. These

observations hold no matter what the input data is. In the worst case, this could be quadratic,

but in the average case, this quantity is O(n log n). It implies that the running time of

Selection sort is quite insensitive to the input.

Implementation

void selectionSort(int numbers[], int array_size)

int i, j;

int min, temp;

for (i = 0; i < array_size-1; i++)



min = i;

for (j = i+1; j < array_size; j++)

if (numbers[j] < numbers[min])

min = j;

temp = numbers[i];

numbers[i] = numbers[min];

numbers[min] = temp;

SELECTION OF KTH

SMALLEST ELEMENT

In computer science, a selection algorithm is an algorithm for finding the k-th smallest

number in a list (such a number is called the kth order statistic.) This includes the cases of

finding the minimum, maximum, and median elements. There are worst-case linear time

selection algorithms. Selection is a subproblem of more complex problems like the nearest

neighbor problem and shortest path problems.

A worst-case linear algorithm for the general case of selecting the kth largest element

was published by Blum, Floyd, Pratt, Rivest, and Tarjan in their 1973 paper Time bounds for

selection, sometimes called BFPRT after the last names of the authors. The algorithm that it

is based on was conceived by the inventor of quicksort,C.A.R.Hoare, and is known as Hoare's

selection algorithm or quickselect.

In quicksort, there is a subprocedure called partition that can, in linear time, group a list

(ranging from indices left to right) into two parts, those less than a certain element, and those

greater than or equal to the element. Here is pseudocode that performs a partition about the

element list[pivotIndex]:

Algorithm function partition(list, left, right, pivotIndex)

pivotValue := list[pivotIndex]

swap list[pivotIndex] and list[right] // Move pivot to end



storeIndex := left

for i from left to right-1

if list[i] < pivotValue

swap list[storeIndex] and list[i]

storeIndex := storeIndex + 1

swap list[right] and list[storeIndex] // Move pivot to its final place

return storeIndex

In quicksort, we recursively sort both branches, leading to best-case Ω(n log n) time.

However, when doing selection, we already know which partition our desired element lies in,

since the pivot is in its final sorted position, with all those preceding it in sorted order

preceding it and all those following it in sorted order following it. Thus a single recursive call

locates the desired element in the correct partition:

Algorithm function select(list, left, right, k)

select pivotIndex between left and right

pivotNewIndex := partition(list, left, right, pivotIndex)

if k = pivotNewIndex

return list[k]

else if k < pivotNewIndex

return select(list, left, pivotNewIndex-1, k)

else

return select(list, pivotNewIndex+1, right, k)

Note the resemblance to quicksort; indeed, just as the minimum-based selection algorithm is

a partial selection sort, this is a partial quicksort, generating and partitioning only O(log n) of

its O(n) partitions. This simple procedure has expected linear performance, and, like

quicksort, has quite good performance in practice. It is also an in –place algorithm, requiring

only constant memory overhead, since the tail recursion can be eliminated with a loop like

this:

Algorithm function select(list, left, right, k)

loop

select pivotIndex between left and right

pivotNewIndex := partition(list, left, right, pivotIndex)



if k = pivotNewIndex

return list[k]

else if k < pivotNewIndex

right := pivotNewIndex-1

else

left := pivotNewIndex+1

Like quicksort, the performance of the algorithm is sensitive to the pivot that is chosen. If bad

pivots are consistently chosen, this degrades to the minimum-based selection described

previously, and so can require as much as O(n2) time. David Musser describes a "median-of-3

killer" sequence that can force the well-known median-of-three pivot selection algorithm to

fail with worst-case behavior .

The algorithm works as follows:

1. Find the minimum value in the list

2. Swap it with the value in the first position

3. Repeat the steps above for remainder of the list (starting at the second position)

Effectively, we divide the list into two parts: the sublist of items already sorted, which we

build up from left to right and is found at the beginning, and the sublist of items remaining to

be sorted, occupying the remainder of the array.

Here is an example of this sort algorithm sorting five elements:

64 25 12 22 11

11 25 12 22 64

11 12 25 22 64

11 12 22 25 64



Selection sort can also be used on list structures that make add and remove efficient, such as a

linked list. In this case it's more common to remove the minimum element from the

remainder of the list, and then insert it at the end of the values sorted so far. For example:

64 25 12 22 11

11 64 25 12 22

11 12 64 25 22

11 12 22 64 25

11 12 22 25 64



MODULE 5



BACKTRACKING

Many problems with searching a set of solutions or which ask for an optimal solution

satisfying some constraints can be solved using the backtracking formulation. In many

applications of the backtrack method, the desired solution is expressible as an

n-tuple(x1,…..,xn), where the xi are chosen from some from finite set Si. Often the problem to

be solved calls for finding one vector that maximizes (or minimizes or satisfies) a criterion

function P(x1,…..,xn). Sometimes it seeks all vectors that satisfy P. The criterion function P is

the inequality a[xi]≤a[xi+1] for 1 ≤ i < n. The set Si is finite and includes the integers 1 through

n.

Many of the problems we solve using backtracking require that all the solutions satisfy a

complex set of constraints. For any problem these constraints can be classified into two

categories: explicit and implicit. The explicit constraints depend on the particular instance I

of the problem being solved. All tuples that satisfy the explicit constraints define a possible

solution space for I. the implicit constraints are the rules that determine which of the tuples in

the solution space of I satisfy the criterion function. Thus implicit constraints describe the

way in which xi must relate to each other.

CONTROL ABSTRACTION

(i) Recursive backtracking algorithm

Algorithm Backtrack(k)

//This schema describes the backtracking process using recursion. On entering, the first //k-1

values x[1], x[2],…..,x[k-1] of the solution vector x[1:n] have been assigned. x[ ] and //n are

global.

for (each x[k] Є T(x[1],…..,x[k-1]) do

if (Bk(x[1],x[2],…..,x[k])!=0) then



if (x[1],x[2],…..,x[k] is a path to an answer node)

then write (x[1:k]);

if (k<n) then backtrack(k+1);

Explanation

Let (x1,x2,…..xi) be a path from the root to a node in a state space tree. Let

T(x1,x2,…..xi) be the set of all possible values for xi+1 such that (x1,x2,…..xi+1) is also a path to

a problem state. T(x1,x2,…..xn) =Ф. We assume the existence of bounding function Bi+1

(expressed as predicates) such that if Bi+1(x1,x2,…..xi+1) is false for a path (x1,x2,…..xi+1) from

the root node to a problem state, then the path cannot be extended to reach an answer node.

Thus the candidates for position i+1 of the solution vector (x1,x2,…..xn) are those values

which are generated by T and satisfy Bi+1. The above recursive algorithm is initially invoked

by Backtrack(1);.

The solution vector (x1,x2,…..xn), is treated as a global array x[1:n]. All the possible

elements for the kth

position of the tuple that satisfy Bk are generated, one by one, and

adjoined to the current vector (x1,x2,…..xk-1). Each time xk is attached, a check is made to

determine whether a solution has been found. Then the algorithm is recursively invoked.

When the ‗for‘ loop is exited, no more values for xk exist and the current copy of Backtrack

ends. The last unresolved call now resumes.

(ii)General iterative backtracking method

Algorithm IBacktrack(n)

//This schema describes the backtracking process.

//All solutions are generated in x[1:n] and printed as soon as they are determined.



k:= 1;

while(k≠0) do

if (there remains an untried x[k] Є T(x[1],x[2],…..,x[k-1]) and Bk (x[1],…..,x[k]) is true) then

if (x[1],….x[k] is a path to answer node)

then write (x[1:k]);

k:=k+1;//Consider the next set.

else k:=k-1; //Backtrack to the previous set.

Explanation

T() will yield the set of possible values that can be placed as the first component x[1]

of the solution vector. The component x[1] will take on those for which the bounding

function B1x(1) is true. The variable k is continually incremented and a solution vector is

grown until either a solution in found or no untried value of xk remains. When k is

decremented, the algorithm must resume the generation of possible elements for the kth

position that have not yet been tried.

Conclusion

The efficiency of both algorithms depends very much on four factors:

1. The time to generate next xk

2. The number of xk satisfying the explicit constraints

3. The time for the bounding functions Bk

4. The number of xk satisfying the Bk



Constraints

Solutions must satisfy a set of constraints

Explicit vs. implicit

Definition 1: Explicit constraints are rules that restrict each xi to take on values

only from a given set.

eg)

xi 0 or Si = all nonnegative real numbers

xi = 0 or 1 or Si = 0, 1

li xi ui or Si = a : li a ui

All tuples satisfying the explicit constraints define a possible solution space for I

(I=problem instance)

Definition 2: The implicit constraints are rules that determine which of the tuples

in the solution space of I satisfy the criterion function. Thus implicit constrains

describe the way in which the xi must relate to each other.

Classic combinatorial problem

Place eight queens on an 8*8 chessboard so that no two attack

Without loss of generality, assume that queen i is placed on row i

All solutions represented as 8-tuples (x1,x2,…,x8) where xi is the column on which

queen i is placed

Constraints

• Explicit constraints

– Si=1,2,…,8

– So, solution space consists of 88 8-tuples

• Implicit constraints

– No two xi‘s can be the same (By this, solution space reduced

from 88 to 8!)

– No two queens can be on the same diagonal



BOUNDING FUNCTIONS

The backtrack algorithm as its virtue the ability to yield the same answer with far

fewer than m trials. Its basic idea is to build solution vector, one competent at a time and to

use modified criterion functions Pi(x1,……,xn) called bounding functions, to test whether

the vector being formed has any chance of success. The major advantage of this method is

this: if it is realized that the partial vector (x1, x2,….., xi) can in no way lead to an optimal

solution, then mi+1……mn possible test vectors can be ignored entirely.

N-QUEENS PROBLEM

The n-queens problem is a generalization of the 8-queens problem. Here n queens are to be

placed on an nxn chessboard so that no two attack, that is no two queens are on the same row,

column, or diagonal. The solution space consists of all n! permutations of the n-tuple

(1,2,…,n). The tree is called permutation tree. The edges are labeled by possible values of xi.

Edges from level 1 to level 2 nodes specify the values for x1. Edges from level i to i+1 are

labeled with values of xi. The solution space is defined by all the paths from the root node to

a leaf node. For eg. If n=4, then there will be 4! =24 leaf nodes in the tree.

Eg: 8-queens problem

A classic problem is to place eight queens on an 8x8 chessboard so that no two

―attack,‖ that is, so that no two of them are on the same row, column, or diagonal.



• Bounding function

– No two queens placed on the same column xi’s are distinct

– No two queens placed on the same diagonal how to test?

• The same value of ―row-column‖ or ―row+column‖

• Supposing two queens on (i,j) and (k,l)

– i-j=k-l (i.e., j-l=i-k) or i+j=k+l (i.e., j-l=k-i)

• So, two queens placed on the same diagonal iff

|j-l| = |i-k|



This example shows how the backtracking works(4-Queens)



TREE STRUCTURE

Tree organization of the 4-queens solution space. Nodes are numbered as in depth first

search.

The n-queens problem is a generalization of the 8-queens problem. Now n queens are to be

placed on an n×n chess board so that no two attack; that is, no two queens are on the same

row, column or diagonal.Generalising our discussions, the solution space consists of all n!

permutations of n-tuple (1, 2.., n).The above figure shows a possible tree organization for the

case n=4. A tree such as this is called a permutation tree. The edges are labeled by possible

values of xi. Edges from level 1 to level 2 nodes specify the value for x1.Thus the leftmost

sub tree contains all solutions with x1=1 and x2=2,and so on. Edges from level to level

i+1are labeled with the value of xi.the solution space is defined by all paths from the root

node to a leaf node. There are 4! =24 nodes in the tree.

ALGORITHM

Algorithm Place (k, ί)

// Returns true if a queen can be placed in the kth row and

//ith column. Otherwise it returns false. x[] is a

//global array whose first (k –1) values have been set.

//Abs (p) returns the absolute value of p.



for j: =1 to k-1 do

if ((x [j] = ί) // Two in the same column

or (Abs (x [j] – ί ) = Abs (j-k)) )

// or in the same diagonal

then return false;

return true;

Algorithm Nqueens (k, n)

// Using backtracking, this procedure prints all

// possible placements of n queens on an n x n

// chessboard so that they are nonattacking.

for ί: =1 to n do

if Place (k, ί) then

x [k] : = ί;

if (k=n) then write (x[1:n]);

else Nqueens (k+1,n);



SUM OF SUBSETS

Given positive numbers wi, 1≤i≤n, and m, find all subsets of the wi whose sum is

m

eg) (w1,w2,w3,w4) = (11,13,24,7) and m=31

Solutions are (11,13,7) and (24,7)

Representation of solution vector

Variable-sized

By giving indices

In the above example, (1,2,4) and (3,4)

Explicit constraints: xi ∈ j | j is integer and 1≤j≤n

Implicit constraints: no two be the same, the sums are m

Fixed-sized

n-tuple (x1,x2,…,xn) where xi∈0,1, 1≤i≤n

In the above example, (1,1,0,1) and (0,0,1,1)

Variable tuple size

Suppose we are given n distinct positive numbers(usually called weights) and we desire to

find all combinations of these numbers whose sums are m.This is called sum of subsets

problem.

We could formulate this problem using either fixed-or variable-sized tuple

Given positive numbers wi,1<=i<=n, and m, this problem calls for finding all subsets of the

wi whose sums are m.For example if n=4,( w1,w2,w3,w4)=(11,13,24,7), and m=31,then the

desired subsets are (11,13,7) and (24,7).Rather than represent the solution vector by the wi

which sum to m we could represent the solution vector by giving the indices of these wi.Now

the two solutions are described by the vectors (1,2,4) and (3,4).In general all solutions are k-

tuples (x1,x2,.....xk), 1<=k<=n and different solutions may have different sized tuples.



• Solution space defined by all paths from the root to any node

Fixed tuple size:

• Solution space defined by all paths from the root to a leaf node • Bounding function

• Bk(x1,..,xk)=true iff

– Assuming wi’s in nondecreasing order, (x1,..,xk) cannot lead to an answer node if



– So, the bounding function is

• Backtracking algorithm for the sum of sunsets

– Invoked by SumOfSubsets(0,1,∑i=1,nwi)



Example:n=6, w[1:6]=5,10,12,13,15,18, m=30

• Terminology

– A node which has been generated and all of whose children have not yet been

generated is called a live node

– The live node whose children are currently being generated is called E-node

– A dead node is a generated node which is not to be expanded further or all of

whose children have been generated

• Two different ways of tree generation

• Backtracking

• Depth first node generation with bounding functions

– Branch-and-bound

– E-node remains E-node until it is dead

Four factors for efficiency of the backtracking algorithms

Time to generate the next xk

Number of xk satisfying the explicit constraints

Time for the bounding function Bk

Number of xk satisfying Bk



A solution space organization for the sum of subsets problem.Nodes are numbered in

BFS

* The first three relatively independent of the problem instance, and the last dependent

The tree of fig 1 represents corresponds to variable tuple size formulation.The edges are

labelled such that an edge from a level i node to a level i+1 node represents a value for xi.At

each node, the solution space is partitioned into subsolution spaces.The solution space is

defined by all paths from the root node to any node in the tree,since any such path

corresponds to a subset satisfying the explicit constraints.Thr possible paths are

(1),(1,2),(1,2,3),(1,2,3,4),(1,2,4),(1,3,4),(2),(2,3) and so on.Thus the leftmost subtree defines

all subsets containing w1,the next subtree defines all subsets containing w2 but not w1 and so

on.

X1=1

X1=2

X1=4

10

1

1

2

2

6

6

7

7

8

9

99

11

11

1

12

13

14

6

15

6

16

3

3

4

4

5

5

X2=2

X

X2=3 X2=4 X2=4

X3=3 X3=4 X3=4 X3=4

X4=4

Fig 1



Another organization for the sum of subsets problem.Nodes are numbered in D-search

The tree of fig 2 corresponds to the fixed tuple size formation.Edges from level i nodes level

i+l nodes are labelled with the value of xi w hich is either zero or one.All paths from root to a

leaf node defining the solution space .The left subtree of the root defines all subsets

containing w1,the right subtree defines all subsets not containing w1, and so on.Now there are

24 leaf nodes.

1

1

2

12

1

3

1

18

11

81

19

19

1

4

1

5

11

26

22

6

1

27

12

1

20

20

1

21

11

12

11

2

1

13

11

6

11

7

1

30

11

31

11

28

22

8

1

29

11

24

24

1

25

1

22

22

1

23

1

1

16

61

1

17

1

14

14

1

15

1

10

11

0

1

11

1

8

11

9

19

1

FIG 2

X1=1 X1=0

X2=1 X2=0 X2=1 X2=0

X3=1

X3=0 X3=1

X3=0 X3=1

1

X3=0 X3=1

X3=

0

X4=1 X4=1 X4=1 X4=

1

X4=1 X4=

1

X4

=1

X4=0 X4=0 X4=0

0000

==

X4=

0



KNAPSACK PROBLEM

Explanation:

Given n positive weights wi , n positive profits pi, and a positive number m that is the

knapsack capacity, this problem calls for choosing a subset of the weights such that

∑ i≤i≤n wixi ≤ m and ∑ i≤i≤n pixi is maximized.

The xi‘s constitute a zero-one-valued vector. The solution space for this problem consists of

the 2n distinct ways to assign zero or one values to the xi‘s. Thus the solution space is the

same as that for the sum of subsets proble. Two possible tree organizations are possible. One

corresponds to the fixed tuple size formulation and the other to the variable tuple size

formulation. Backtracking algorithms for the knapsack problem can arrived by using either of

these two state space trees. Regardless of which is used, bounding functions are needed to

help kill some live nodes without expanding them. A bounding function for this problem is

obtained by using an upper bound on the value of the best feasible solution obtainable by

expanding the given live node and any of its descendants. If this upper bound is not higher

than the value of the best solution determined so far, then that live node can be killed.

Now by using the fixed tuple size formulation, if at node Z the values of xi, 1≤i≤k, have

already been determined, then an upper bound for Z can be obtained by relaxing the

requirement xi=0 or 1 to 0≤xi ≤1 for k+1≤i≤n and using the greedy algorithm to solve the

relaxed problem. Function Bound(cp,cw,k) determines an upper bound on the best solution

obtainable by expanding any node Z at level k+1 of the state space tree. The object weights

and profits are w[i] and p[i]. It is assumed that p[i]/w[i]≥p[i+1]/w[i+1], 1≤i<n.

From Bound it follows that the bound for a feasible left child of a node Z is the same as that

for Z. Hence, the bounding function need not be used whenever the backtracking algorithm

makes a move to the left child of a node. The resulting algorithm is BKnap. It was obtained

form the recursive backtracking schema. Initially set fp;=-1;. This algorithm is invoked as

BKnap(1,0,0);

When fp ≠ -1, x[i]. 1≤ i≤ n, is such that ∑ni=1 p[i]x[i] = fp. In lines 8 to 18 children are

generated. In line 20, Bound is used to test whether a right child should be generated. The



path y[i], 1≤ i ≤ k, is the path to the current node. The current weight cw = ∑k-1

i=1 w[i]y[i] and

cp= ∑i=1k-1

p[i]y[i]. In lines 13 to 17 and 23 to 27 the solution vector is updated if need

be.

So far, all the backtracking algorithms have worked on a static state space tree.

Algorithm: Bounding Function

Algorithm Bound(cp,cw,k)

//cp is the current profit total, cw is the current

//weight total; k is the index of the last removed

//item; and m is the knapsack size.

b:= cp; c:=cw;

for i:=k+1 to n do

c:=c+w[i];

if(c<m) then b:=b+p[i];

else return b+(1-(c-m)/w[i])*p[i];

return b;

Algorithm: Backtracking solution to the 0/1 knapsack problem.

Algorithm Bknap(k,cp,cw)

//m is the size of the knapsack; n is the number of weights.

//and profits. w[] and p[] are the weights and profits.

//p[i]/w[i] ≥ p[i+1]/w[i+1]. fw is the final weight of

//knapsack; fp is the final maximum profit. x[k] = 0 if w[k]

//is not in the knapsack; else x[k]=1.

// Generate left child.

if (cw + w[k] ≤ m) then



y[k]:=1;

if(k<n) then BKnap(k+1,cp+p[k], cw +w[k]);

if ((cp+p[k] > fp) and (k=n)) then

fp:=cp +p[k]; fw:=cw+w[k];

for j:=1 to k do x[j]:=y[j];

// Generate right child.

if (Bound(cp,cw,k)≥fp) then

y[k] :=0; if (k<n) then BKnap(k+1,cp,cw);

if((cp>fp) and (k=n)) then

fp:=cp; fw:=cw;

for j:=1 to k do x[j]:=y[j];

ALGORITHM

// CP-Current Profit;

//K-index of last removed access

b=cp;c=cw;

for P=K+1 to n do

c=c+w[i];

if(c<n)the b=b+p[i];



else

return b+(1-(c-m)/w[i])+p[i];

Return(b);

Algorithm B knap(K,cp,cw)

If(w+w[K]<=m) then

Y[K]=1;

If(K<n)then

B knap(K+1,cp+p[K],cw+w[K]);

If((cp=p[k])>fp)and(K=n)then

fp=cp+p[K];fw=cw+w[K];

for j=1 to K do

x[j]=y[j];

If(Bound(cp,w,K)>=fp) then

Y[K]=0;

If(k<n)then



B knap(K+1,cp,cw);

If((cp>fp)and(k=n)) then

fp=cp;

fw=cw;

for j=1 to k do

x[j]=y[j];

BRANCH AND BOUND ALGORITHM TECHNIQUE

Introduction

Branch and bound is another algorithm technique that we are going to present in our multi-

part article series covering algorithm design patterns and techniques. B&B, as it is often

abbreviated, is one of the most complex techniques and surely cannot be discussed in its

entirety in a single article. Thus, we are going to focus on the so-called A* algorithm that is

the most distinctive B&B graph search algorithm.

If you have followed this article series then you know that we have already covered the most

important techniques such as backtracking, the greedy strategy, divide and conquer, dynamic

programming, and even genetic programming. As a result, in this part we will compare

branch and bound with the previously mentioned techniques as well. It is really useful to

understand the differences.

Branch and bound is an algorithm technique that is often implemented for finding the optimal

solutions in case of optimization problems; it is mainly used for combinatorial and discrete

global optimizations of problems. In a nutshell, we opt for this technique when the domain of



possible candidates is way too large and all of the other algorithms fail. This technique is

based on the en masse elimination of the candidates.

You should already be familiar with the tree structure of algorithms. Out of the techniques

that we have learned, both the backtracking and divide and conquer traverse the tree in its

depth, though they take opposite routes. The greedy strategy picks a single route and forgets

about the rest. Dynamic programming approaches this in a sort of breadth-first search

variation (BFS).

Now if the decision tree of the problem that we are planning to solve has practically

unlimited depth, then, by definition, the backtracking and divide and conquer algorithms are

out. We shouldn't rely on greedy because that is problem-dependent and never promises to

deliver a global optimum, unless proven otherwise mathematically.

As our last resort we may even think about dynamic programming. The truth is that maybe

the problem can indeed be solved with dynamic programming, but the implementation

wouldn't be an efficient approach; additionally, it would be very hard to implement. You see,

if we have a complex problem where we would need lots of parameters to describe the

solutions of sub-problems, DP becomes inefficient.

Branch and bound is a systematic method for solving optimization problems B&B is a rather

general optimization technique that applies where the greedy method and dynamic

programming fail. However, it is much slower. Indeed, it often leads to exponential time

complexities in the worst case. On the other hand, if applied carefully, it can lead to

algorithms that run reasonably fast on average. The general idea of B&B is a BFS-like search

for the optimal solution, but not all nodes get expanded (i.e., their children generated).

Rather, a carefully selected criterion determines which node to expand and when, and another

criterion tells the algorithm when an optimal solution has been found. The basic concept

underlying the branch-and-bound technique is to divide and conquer. Since the original

―large‖ problem is hard to solve directly,it is divided into smaller and smaller subproblems

until these subproblems can be conquered. The dividing (branching) is done by partitioning

the entire set of feasible solutions into smaller and smaller subsets.The conquering

(fathoming) is done partially by (i) giving a bound for the best solution in the subset;(ii)

discarding the subset if the bound indicates that it can‘t contain an optimal solution.These



three basic steps – branching, bounding, and fathoming – are illustrated on the following

example.

Branch-and-Bound Algorithms

A counter-part of the backtracking search algorithm which, in the absence of a cost criteria,

the algorithm traverses a spanning tree of the solution space using the breadth-first approach.

That is, a queue is used, and the nodes are processed in first-in-first-out order

Procedure Expand(E)

begin

/* Generate all the children of E; */

I := E->I;

X,p: nodepointer;

S[1:n]: Boolean;

/* S is a bitmap set initialized to 0*/

/* S will contain all the jobs that have been

assigned by the partial path from the root

to E */

p := E;

while (p is not the root) do

S[p->J] := 1;

p := p-> Parent;

endwhile

for job=1 to n do

if S[job] = 0 then

X := new(node);

X->I := I + 1;

X->J := job;

X->Parent := E;

X->CC := E->CC + AX->I,X->J-mX->I;

Insert(X,H);

endif

endfor

http://www.cse.ohio-state.edu/~gurari/course/cis680/cis6802.html#QQ2-52-135



end

.

If a cost criteria is available, the node to be expanded next (i.e., the branch) is the one with

the best cost within the queue. In such a case, the cost function may also be used to discard

(i.e., the bound) from the queue nodes that can be determined to be expensive. A priority

queue is needed here.

Cost-Based Tree Traversal of branch and bound

A function can be considered to be a tree generator, if when given a node X and index i it

produces the i‘th child of the node.

The following function produces a complete binary tree of 11 nodes.

The recursive function provided for deriving permutations is another example of a function

that may be used to generate trees.

Besides for a tree generator function, we also need a cost function to decide in what order to

traverse the nodes when searching for a solution. The algorithm proceeds in the following

manner.

1. Initialization: The root of the of the tree is declared to be alive.

2. Visit: The cost criteria decides which of the live nodes is to process next.

3. Replacement: The chosen node is removed from the set of live nodes, and its children

are inserted into the set. The children are determined by the tree generator function.

4. Iteration: The visitation and replacement steps are repeated until no alive nodes are

left.




In the case of backtracking the cost criteria assumes a last-in-first-out (LIFO) function, which

can be realized with a stack memory. A first-in-first-out cost criteria implies the FIFO

branch-and-bound algorithm, and it can be realized with queue memory. A generalization to

arbitrary cost criteria is the basis for the priority branch-and-bound algorithm, and a priority

queue memory can be employed to realize the function.

FIFO BRANCH AND BOUND ALGORITHM

FIFO branch and bound algorithm for the job sequencing problem can begin with upper=

infinity. While implementing FIFO B&B algorithm it is not economical to kill live nodes

with c®> upper each time upper is updated. Each solution is assumed to be expressible as an

array X[1:n] (as was seen in Backtracking).

A predictor, called an approximate cost function CC, is assumed to have been defined.

A live node is a node that has not been expanded.A dead node is a node that has been

expanded. The expanded node (or E-node for short) is the live node with the best CC value.

Branch and Bound is a general search method. Starting by considering the root problem (the

original problem with the complete feasible region), the lower-bounding and upper-bounding

procedures are applied to the root problem.If the bounds match, then an optimal solution has

been found and the procedure terminates. Otherwise, the feasible region is divided into two or

more regions, these subproblems partition the feasible region. The algorithm is applied

recursively to the subproblems. If an optimal solution is found to a subproblem, it is a

feasible solution to the full problem, but not necessarily globally optimal. If the lower bound

for a node exceeds the best known feasible solution, no globally optimal solution can exist in

the subspace of the feasible region represented by the node. Therefore, the node can be

removed from consideration. The search proceeds until all nodes have been solved or pruned,

or until some specified threshold is met between the best solution found and the lower bounds

on all unsolved subproblems. A counter-part of the backtracking search algorithm which, in

the absence of a cost criteria, the algorithm traverses a spanning tree of the solution space

using the breadth-first approach. That is, a queue is used, and the nodes are processed in first-

in-first-out order. If a cost criteria is available, the node to be expanded next (i.e., the branch)

is the one with the best cost within the queue. In such a case, the cost function may also be

used to discard (i.e., the bound) from the queue nodes that can be determined to be expensive.

A priority queue is needed here.



The general FIFO B&B algorithm follows:

Procedure B&B()

begin

E: nodepointer;

E := new(node); -- this is the root node which

-- is the dummy start node

H: heap; -- A heap for all the live nodes

-- H is a min-heap for minimization problems,

-- and a max-heap for maximization problems.

while (true) do

if (E is a final leaf) then

-- E is an optimal solution

print out the path from E to the root;

return;

endif

Expand(E);

if (H is empty) then

report that there is no solution;

return;

endif

E := delete-top(H);

endwhile

end

Procedure Expand(E)

begin

- Generate all the children of E;

- Compute the approximate cost value CC of each child;

- Insert each child into the heap H;

end



We need to define the full record of a node .We need to fully implement the Expand

procedure

Every node corresponds to something like X[i]=j, which signifies that the job X[i] assigned to

person i is j. Every node must store its CC value. Each node must point to its parent so that

when an optimal leaf is generated, the path from that leaf to the root can be traced and printed

out as the optimal solution.Therefore, a node record structure should look like:

Record node

Begin

Parent: nodepointer;

I: integer; -- person I

J: integer; -- Job J is assigned to person I

CC: real;

End

Take the 2nd CC formula:

CC(X at level k) = cost so far + sumn

i=k+1mi

where mi is the minimum of row i.

observe that if X is a pointer to a node, then

X->CC = X->Parent->CC + AX->I,A->J - mX->I

Write a piece of code that computes the mis for i=1,2,...,n

Code for Expand(E):

Procedure Expand(E)

begin

/* Generate all the children of E; */

I := E->I;

X,p: nodepointer;

S[1:n]: Boolean;

/* S is a bitmap set initialized to 0*/

/* S will contain all the jobs that have been

assigned by the partial path from the root

to E */

p := E;

while (p is not the root) do



S[p->J] := 1;

p := p-> Parent;

endwhile

for job=1 to n do

if S[job] = 0 then

X := new(node);

X->I := I + 1;

X->J := job;

X->Parent := E;

X->CC := E->CC + AX->I,X->J-mX->I;

Insert(X,H);

endif

endfor

end.

Although a number of algorithms have been proposed for the integer linear programming

problem, the FIFO branch-and-bound technique has proven to be reasonably efficient on

practical problems, and it has the added advantage that it solves continuous linear programs

as sub problems.The technique is also used in a lot of software in global optimization.

Fifo B&B Cost-Based Tree Traversal


produces the i‘th child of the node.The following function produces a complete binary tree of

11 nodes.








manner.



3. Replacement: The chosen node is removed from the set of live nodes, and its

children are inserted into the set. The children are determined by the tree generator

function.


left.

In the case of backtracking the cost criteria assumes a last-in-first-out (LIFO) function,

which can be realized with a stack memory. A first-in-first-out cost criteria implies the

FIFO branch-and-bound algorithm, and it can be realized with queue memory. A

generalization to arbitrary cost criteria is the basis for the priority branch-and-bound

algorithm, and a priority queue memory can be employed to realize the function.

• Search the tree using a breadth-first search (FIFO branch and bound).

• Search the tree as in a bfs, but replace the FIFO queue with a stack (LIFO branch and

bound).

• Replace the FIFO queue with a priority queue (least-cost (or max priority) branch and

bound). The priority of a node p in the queue is based on an estimate of the likelihood

that the answer node is in the subtree whose root is p. FIFO branch and bound finds

solution closest to root.Backtracking may never find a solution because tree depth is

infinite (unless repeating configurations are eliminated).

• Least-cost branch and bound directs the search to parts of the space most likely to

contain the answer. So it could perform better than backtracking.



LIFO BRANCH AND BOUND ALGORITHM

LIFO branch and bound algorithm for the job sequencing problem can begin with upper=

infinity. While implementing LIFO B&B algorithm it is not economical to kill live nodes

with c®> upper each time upper is updated. Each solution is assumed to be expressible as an

array X[1:n] (as was seen in Backtracking).

A predictor, called an approximate cost function CC, is assumed to have been defined.

A live node is a node that has not been expanded.A dead node is a node that has been

expanded. The expanded node (or E-node for short) is the live node with the best CC value.

Branch and Bound is a general search method. Starting by considering the root problem (the

original problem with the complete feasible region), the lower-bounding and upper-bounding

procedures are applied to the root problem.If the bounds match, then an optimal solution has

been found and the procedure terminates. Otherwise, the feasible region is divided into two or

more regions, these subproblems partition the feasible region. The algorithm is applied

recursively to the subproblems. If an optimal solution is found to a subproblem, it is a

feasible solution to the full problem, but not necessarily globally optimal. If the lower bound

for a node exceeds the best known feasible solution, no globally optimal solution can exist in

the subspace of the feasible region represented by the node. Therefore, the node can be

removed from consideration. The search proceeds until all nodes have been solved or pruned,

or until some specified threshold is met between the best solution found and the lower bounds

on all unsolved subproblems. A counter-part of the backtracking search algorithm which, in

the absence of a cost criteria, the algorithm traverses a spanning tree of the solution space

using the breadth-first approach. That is, a queue is used, and the nodes are processed in first-

in-first-out order. If a cost criteria is available, the node to be expanded next (i.e., the branch)

is the one with the best cost within the queue. In such a case, the cost function may also be

used to discard (i.e., the bound) from the queue nodes that can be determined to be expensive.

A priority queue is needed here.

A general LIFO B&B algorithm :

Procedure B&B()

begin

E: nodepointer;

E := new(node); -- this is the root node which

-- is the dummy start node



H: heap; -- A heap for all the live nodes

-- H is a min-heap for minimization problems,

-- and a max-heap for maximization problems.

while (true) do

if (E is a final leaf) then

-- E is an optimal solution

print out the path from E to the root;

return;

endif

Expand(E);

if (H is empty) then

report that there is no solution;

return;

endif

E := delete-top(H);

endwhile

end

Procedure Expand(E)

begin

- Generate all the children of E;

- Compute the approximate cost value CC of each child;

- Insert each child into the heap H;

end

We need to define the full record of a node .We need to fully implement the Expand

procedure

Every node corresponds to something like X[i]=j, which signifies that the job X[i] assigned to

person i is j. Every node must store its CC value. Each node must point to its parent so that

when an optimal leaf is generated, the path from that leaf to the root can be traced and printed

out as the optimal solution.Therefore, a node record structure should look like:

Record node

Begin



Parent: nodepointer;

I: integer; -- person I

J: integer; -- Job J is assigned to person I

CC: real;

End

Take the 2nd CC formula:

CC(X at level k) = cost so far + sumn

i=k+1mi

where mi is the minimum of row i.

observe that if X is a pointer to a node, then

X->CC = X->Parent->CC + AX->I,A->J - mX->I

Write a piece of code that computes the mis for i=1,2,...,n

Although a number of algorithms have been proposed for the integer linear programming

problem, the LIFO branch-and-bound technique has proven to be reasonably efficient on

practical problems, and it has the added advantage that it solves continuous linear programs

as sub problems.The technique is also used in a lot of software in global optimization.

Lifo B&B Cost-Based Tree Traversal


produces the i‘th child of the node.The following function produces a complete binary tree of

11 nodes.








manner.



7. Replacement: The chosen node is removed from the set of live nodes, and its

children are inserted into the set. The children are determined by the tree generator

function.


left.

In the case of backtracking the cost criteria assumes a last-in-first-out (LIFO) function,

which can be realized with a stack memory. A last-in-first-out cost criteria implies the

LIFO branch-and-bound algorithm, and it can be realized with queue memory. A

generalization to arbitrary cost criteria is the basis for the priority branch-and-bound

algorithm, and a priority queue memory can be employed to realize the function.

• Search the tree using a breadth-first search (FIFO branch and bound).

• Search the tree as in a bfs, but replace the FIFO queue with a stack (LIFO branch and

bound).

• Replace the FIFO queue with a priority queue (least-cost (or max priority) branch and

bound). The priority of a node p in the queue is based on an estimate of the likelihood

that the answer node is in the subtree whose root is p. LIFO branch and bound finds

solution closest to root.Backtracking may never find a solution because tree depth is

infinite (unless repeating configurations are eliminated).

• Least-cost branch and bound directs the search to parts of the space most likely to

contain the answer. So it could perform better than backtracking.



LC CONTROL ABSTRACTION

listnode= record

Listnode *next,*parent;

Float cost;

Algorithm LCSearch(t)

//Search t for an answer node.

If *t is an answer node then output t and return

E:=t; //E-node

Initalize the list of live nodes to be empty;

Repeat

for each child x of E do

If x is an answer node then output the path from x to t and return;

Add(x);//x is a new live node.

(x->parent):=E;//Pointer for path to root

If there are no more live nodes then

Write(―No answer node‖);return;

E:=Least();

until(false);



15- PUZZLE

The 15 puzzle consists of 15 squares numbered from 1 to 15 that are placed in a box

leaving one position out of the 16 empty. The goal is to reposition the squares from a given

arbitrary starting arrangement by sliding them one at a time into the configuration shown

above. For some initial arrangements, this rearrangement is possible, but for others, it is not.

The n-puzzle is known in various versions, including the 8 puzzle, the 15 puzzle, and with

various names. It is a sliding puzzle that consists of a frame of numbered square tiles in

random order with one tile missing. If the size is 3×3, the puzzle is called the 8-puzzle or 9-

puzzle, and if 4×4, the puzzle is called the 15-puzzle or 16-puzzle. The object of the puzzle is

to place the tiles in order (see diagram) by making sliding moves that use the empty space.

Theorem:

The goal state is reachavle from intial state iff ∑16

i=1less(i)+x is even.

For any state let less(i) be the number of tiles j such that j<I and position(j)>position(i).Let

position(i) be the position number in the intial state of the tile numbered i.

http://en.wikipedia.org/wiki/Sliding_puzzle



First ten steps in depth first search



TRAVELLING SALESMAN PROBLEM

The Travelling Salesman problem (TSP) is a problem in combinatorial optimization studied

in operations research and theoretical computer science. Given a list of cities and their

pairwise distances, the task is to find a shortest possible tour that visits each city exactly once.

It is used as a benchmark for many optimization methods.

The TSP has several applications even in its purest formulation, such as planning, logistics,

and the manufacture of microchips. Slightly modified, it appears as a sub-problem in many

areas, such as genome sequencing. In these applications, the concept city represents, for

example, customers, soldering points, or DNA fragments, and the concept distance represents

travelling times or cost, or a similarity measure between DNA fragments.

Asymmetric and symmetric

In the symmetric TSP, the distance between two cities is the same in each direction. Thus, the

underlying structure is an undirected graph between; especially, each tour has the same length

in both directions. In the asymmetric TSP, the distance from one city to the other need not

equal the distance in the other direction, in general, there may not even be a connection in the

other direction.

Problem:

You are given a list of n cities along with the distances between each pair of cities. The goal

is to find a tour which starts at the first city, visits each city exactly once and returns to the

first city, such that the distance traveled is as small as possible. This problem is known to be

NP-complete , i.e. no serial algorithm exists that runs in time polynomial in n, only in time

exponential in n, and it is widely believed that no polynomial time algorithm exists. In

practice, we want to compute an approximate solution, i.e. a single tour whose length is as

short as possible, in a given amount of time.

More formally, we are given a graph G=(N,V,W) consisting of a set N of n nodes (or cities),

a set of edges V = (i,j) connecting cities, and a set of nonnegative weights W = w(i,j)

giving the length of edge (i,j) (distance from city i to city j). The graph is directed, so that an



edge (i,j) may only be traversed in the direction from i to j, and edge (j,i) may or may not

exist. Similarly, w(i,j) does not necessarily equal w(j,i), if both edges exist.

Branch-and-Bound for TSP

A simple improvement on the algorithm prunes the search tree by observing that if a partial

tour is already longer than the best solution found so far, there is no reason to continue

searching that path.

Naive Branch-and-Bound Solution of TSP

w = w(1,2) + w(2,3) + w(3,4) + ... + w(n-1,n) + w(n,1)

Best_S_so_far = ( n, [ 1, 2, 3, ... , n-1, n ], w )

S = ( 1, [ 1 ], 0 )

Search( S, Best_S_so_far )

print Best_S_so_far

procedure Search( S, Best_S_so_far )

let ( k, [ i1, i2, ... , ik ], w ) = S

let ( n, [ i1B, i2B, ... , inB ], wB ) = Best_S_so_far

if k = n then

new_w = w + w(ik,i1)

if new_w < wB then

Best_S_so_far = ( k, [ i1, i2, ... , ik ], new_w )

end if

else

for all j not in [ i1, i2, ... , ik ]

new_w = w + w(ik,j)

if new_w < wB then

New_S = ( k+1, [ i1, i2, ... , ik, j ], new_w )

Search( New_S, Best_S_so_far )

end if

end for

endif

return

end

Better Branch and Bound Algorithm for TSP



Another strategy for searching the solution space is to repeatedly divide it into two parts:

those with a given edge and those without the edge. The search tree would unfold as follows:

-----------

| all solns |

-----------

/ \

---------------- -----------------

| solns with e_i | | solns w/out e_i |

---------------- -----------------

/ \ / \

----------- ---------- ---------- -----------

| with e_j | | w/out e_j | | with e_k | | w/out e_k |

----------- ---------- ---------- -----------

Bounding the solutions: Assume the input is given as a dense adjacency matrix. We will put

infinities, rather than zeros on the diagonal to avoid traversing these self-edges.

i\j 1 2 3 4 5 6 7

\ ________________________________________

1 |Inf 3 93 13 33 9 57

2 | 4 Inf 77 42 21 16 34

3 | 45 17 Inf 36 16 28 25

4 | 39 90 80 Inf 56 7 91

5 | 28 46 88 33 Inf 25 57

6 | 3 88 18 46 92 Inf 7

7 | 44 26 33 27 84 39 Inf

We can subtract a constant from a given row or column, as long as the values remain non-

negative. This changes the weight of each tour, but not the set of legal tours or their relative

weights. We therefore normalize the matrix by subtracting the minimum value in each row

from the row and the minimum value of each column from the column. This results in a

matrix with at least one zero in every row and column.

i\j 1 2 3 4 5 6 7

\ ________________________________________

1 |Inf 0 83 9 30 6 50

2 | 0 Inf 66 37 17 12 26

3 | 29 1 Inf 19 0 12 5



4 | 32 83 66 Inf 49 0 80

5 | 3 21 56 7 Inf 0 28

6 | 0 85 8 42 89 Inf 0

7 | 18 0 0 0 58 13 Inf

Any solution must use one entry from every row and every column, so the sum of the values

we just subtracted is a lower bound on the weight of solution. In this case, we subtracted [3 4

16 7 25 3 26] from the rows and then [0 0 7 1 0 0 4] from the columns, so the lower bound on

the weight of any solution is 96.

Representing the set of solutions: The adjacency matrix can be used to represent the set of

solutions. When an edge is chosen for the solution, the row and column containing that edge

is deleted. When an edge is eliminated from the solution, its entry is changed to infinity so

that it will never be chosen. In the above example, assume we choose to split the search space

on the edge from 4 to 6. For the right subtree, which represents all solutions not containing

(4,6), we replace the (4,6) entry by infinity. The minimum value in row 4 is now 32, so we

can renormalize the matrix and improve the lower bound to 96+32 = 128. Column 6 has

another 0 entry, so it remains unchanged. The matrix for the right subtree is:

i\j 1 2 3 4 5 6 7

\ ________________________________________

1 |Inf 0 83 9 30 6 50

2 | 0 Inf 66 37 17 12 26

3 | 29 1 Inf 19 0 12 5

4 | 0 51 34 Inf 17 Inf 48

5 | 3 21 56 7 Inf 0 28

6 | 0 85 8 42 89 Inf 0

7 | 18 0 0 0 58 13 Inf

The left subtree represents all solutions containing (4,6). We therefore delete row 4 and

column 6 from the matrix and renormalize. In addition, since we have used edge (4,6), edge

(6,4) is no longer usable, so we replace the (6,4) with infinity.The matrix can now be

renormalized, in this case by subtracting 3 from the row with i=5 (now the 4th row in the

smaller matrix), yielding a lower bound of 96+3=99 and the matrix:

i\j 1 2 3 4 5 7

\ __________________________________

1 |Inf 0 83 9 30 50



2 | 0 Inf 66 37 17 26

3 | 29 1 Inf 19 0 5

5 | 0 18 53 4 Inf 25

6 | 0 85 8 Inf 89 0

7 | 18 0 0 0 58 Inf

The process outlined so far can be used by repeatedly following these steps. Following the

left branches we are guaranteed to reach some solution in n levels. Following the right

branch, we will eventually have a matrix full of infinities, at which point the lower bound

Choice of the splitting edge: In general, the right branches represent a larger set of solutions

than the left branches. Therefore, in choosing the splitting edge, we look for something that

will raise the lower bound of the right-hand subtree as much as possible. In the example, edge

(4,6) was chosen because its value was zero and the next larger value in row 4 was 32. There

are other zero entries in the matrix, for example, (3,5), but the improvement in the lower

bound for that case would have been only 1 (for row 3) plus 17 (for column 5). Therefore, the

general rule is to search for the zero entry at (i,j) that maximizes the increase in the lower

bound (largest sum of minimum in row i and minimum in column j, not counting the zero at

(i,j)).

Insertion of infinities in left-branches: When expanding a left branch in the search tree for

edge (i,j), we noted that edge (j,i) should be made infinity. In general, we are trying to avoid

the creation of non-covering cycles in the tour, i.e., cycles that traverse only a subset of the

nodes. For the example above, assume that the left-most search path unfolds with the

following edge splits: (4,6), (3,5), and (2,1). At this point the partial solution contains three

disconnected paths; all cycles were prevented by marking (6,4), (5,3), and (1,2) as infinity.



-----------

| all solns |

-----------

/ \

------------ -------------

| with (4,6)| | w/out (4,6)|

------------ -------------

/ \

------------ -------------

| with (3,5)| | w/out (3,5)|

------------ -------------

/ \

------------ ------------

| with (2,1)| | w/out (2,1)|

------------ ------------

The next choice of splitting edge is (1,4), so according to our above rule (4,1) should be

changed to infinity. However, at this point the adjacency matrix does not contain a row for 4,

since it was removed with the (4,6) step. Moreover, the partial solution is now: 2-1-4-6 &3-5.

Algorithm Analysis and Design

Documents

pairs shortest

stressens

computer science

kth smallest

conquer matrix

travelling

lower bound

lc control