CS 615: Design & Analysis of Algorithms Chapter 2: Efficiency of Algorithms
CS 615: Design & Analysis of
Algorithms
Chapter 2: Efficiency of Algorithms
April 21, 2023 CS 615 Design & Analysis of Algorithms 2
Course Content
1. Introduction, Algorithmic Notation and Flowcharts (Brassard & Bratley, Chap. Chapter 3)
2.2. Efficiency of Algorithms (Brassard & Efficiency of Algorithms (Brassard & BratleyBratley,, Chap. 2) Chap. 2)
3. Basic Data Structures (Brassard & Bratley, Chap. 5)
4. Sorting (Weiss, Chap. 7)5. Searching (Brassard & Bratley Chap.: 9)6. Graph Algorithms (Weiss, Chap.: 9)7. Randomized Algorithms (Weiss, Chap.: 10)8. String Searching (Sedgewick, Chap. 19)9. NP Completeness (Sedgewick, Chap. 40)
April 21, 2023 CS 615 Design & Analysis of Algorithms 3
DefinitionsProblem:
A situation to be solved by an algorithmExample:
Multiply two integers
InstanceA special case of the problemExample:
Multiply(981, 1234)
An algorithm must work correctly On every instance of the problem it claims to solve
To prove an algorithm is not correctFind an instance which the algorithm cannot solve correctly
April 21, 2023 CS 615 Design & Analysis of Algorithms 4
Efficiency of Algorithms
To decide which algorithm to choose:
Empirical ApproachProgram the competing algorithmsTry them on different instances
with the help of the computer(s)
Resources:Computing timeStorage spaceNumber of processes (for parallel algorithms)
April 21, 2023 CS 615 Design & Analysis of Algorithms 5
Efficiency of Algorithms
To decide which algorithm to choose:
Theoritical ApproachUsing formal methods to anlyze the efficiencyDoes not depend on the computerNo need to make “programming”
April 21, 2023 CS 615 Design & Analysis of Algorithms 6
Efficiency of Algorithms
To decide which algorithm to choose
Hybrid ApproachDescribe algorithm’s efficiency function theoritically
Empirically determine numerical parameters for a particular machine
Predict the time an actual implementation will take to solve an instance
April 21, 2023 CS 615 Design & Analysis of Algorithms 7
Principle of Invariance
Two different implementations of an algorithm
Will not differ in efficiency by more than some multiplicative constant
ExampleIf the constant is 5:if the first implementation
takes 1 second to solve an instance
then a second implementation (possible on a different machine)
Will not take more than 5 seconds
April 21, 2023 CS 615 Design & Analysis of Algorithms 8
Principle of Invariance
For an instance of size nImplementation 1:
Takes time of t1(n)
Implementation 2:Takes time of t2(n)
There always exist positive constants c and d such that
t1(n)c*t2(n) and
t2(n)d *t1(n)
where n is sufficiently large
April 21, 2023 CS 615 Design & Analysis of Algorithms 9
Results of Principle of Invarience1. A change on the implementation of the
same algorithm 1. Can only cause a constant of change in
efficiency
2. The principle does not depend onThe computer we useThe compiler we implementThe abilities of the person making the coding
3. If we want a radical change in efficiencyWe need to change the algorithm itself
April 21, 2023 CS 615 Design & Analysis of Algorithms 10
Theoritical EfficiencyFor a given function t
an algorithm for some problem takes a time in the order of t(n),
if there exist a positive constant c the algorithm is capable of
solving every instance of size
thenn is not more than
c*t(n) seconds/hours/years.
For numerical problems n may sometimes be the value rather than the size of the instance
April 21, 2023 CS 615 Design & Analysis of Algorithms 11
Algorithm TypesTime takes to solve an instance of a
Linear Algorithm isNever greater than c*n
Quadratic Algorithm isNever greater than c*n2
Cubic Algorithm isNever greater than c*n3
Polynomial Algorithm isNever greater than nk
Exponential Algorithm isNever greater than cn
where c & k are appropriate constants
April 21, 2023 CS 615 Design & Analysis of Algorithms 12
Worst Case AnalysisWhen to analyse an algorithm
Considering the cases only take maximum amount of time
If the algorithm is capable of solving cases in t(n)
then the worst case should not be greater thanc*t(n)
Useful if the algorithm is to be applied to casesthe upper bound of an algorithm must be known
Example: Response time for a nuclear power plant.
April 21, 2023 CS 615 Design & Analysis of Algorithms 13
Average Time Analysis
If the algorithm is going to be used many times
it is useful to know the average execution time
on instances of size n
It is harder to analyse the average casethe distribution of data should be known
Insertion sorting average time isin the order of n2
April 21, 2023 CS 615 Design & Analysis of Algorithms 14
Elementary OperationIs one whose execution time is
bounded above by a constantThe constant does not depend on
The sizeor other parameters of the instance considered
Examplex=y+w*z is it elementary operation?Suppose
ta : time to execute an addition (constant)tm : time to execute a multiplication (constant)ts : time to execute an assignment (constant)
t: time required to execute an addition, multiplication, & asignment:
ta ta + m tm + ts s where a,m,s are constantst max(ta , tm , ts ) x (a+m+s)
April 21, 2023 CS 615 Design & Analysis of Algorithms 15
Elementary Operation
A single line of of programmay correspond to a variable number of elemenary operations
x=min{T[i] | 1 i n}Time required to compute min
increases with n
min() is not an elementary operation !
April 21, 2023 CS 615 Design & Analysis of Algorithms 16
Elementary Operation
addition, multiplication:Normally
Time required to computeaddition, multiplication
Depends on the length of the operands
Butit is reasonable to assume
addition and multiplication are elementary operations
when the operands are in fixed length
April 21, 2023 CS 615 Design & Analysis of Algorithms 17
Some Algorithm Examples
Calculating DeterminantsSortingMultiplication of Large IntegersCalculating the Greatest Common DivisorCalculating Fibonacci SequencesFourier Transforms
April 21, 2023 CS 615 Design & Analysis of Algorithms 18
Calculating DeterminantsRecursive definition of Algorithm
To compute a determinant of n x n matrixTakes time proportional to n!Worse than taking exponential timeExperiments:
5 x 5 matrix 20 sec.10 x 10 matrix 10 min.
Estimation20 x 20 matrix 10 million years.
Gauss-Jordan EliminationTo compute a determinant of n x n matrixTakes time proportional to n3
Experiments:10 x 10 matrix 0.01 sec.20 x 20 matrix 0.05 sec.100 x 100 matrix 5.5 sec.
April 21, 2023 CS 615 Design & Analysis of Algorithms 19
SortingArranging n objects based on
the “ordering function” defined for these objects
No sorting algorithm is faster than order of nlogn
Insertion sortingTakes time proportional to n2
Experiment: Sorting 1000 elements takes 3 sec.
Estimation: Sorting 100 000 elements would take 9.5 hrs.
Selection sortingTakes time proportional to n2
April 21, 2023 CS 615 Design & Analysis of Algorithms 20
SortingHeapsort
Takes time proportional to nlogneven in worst cases
MergesortTakes time proportional to nlogn
even in worst cases
QuicksortTakes time proportional to nlognExperiment:
Sorting 1000 elements takes 0.2 sec.Sorting 100 000 elements takes 30 sec.
April 21, 2023 CS 615 Design & Analysis of Algorithms 21
Multiplication of Large IntegersWhen multiplying large integers
Operands might become too large to hold in a single word
Assume two large integers of sizes m and n are to be multipliedMultiply each digit of one integer
by the digit of the second digit
Takes time proportional to m x n
More efficient algorithms:Divide-and-conquer:
Takes time proportional to n x mlg(3/2)
=n x m0.59
m is the size of the smaller integer
April 21, 2023 CS 615 Design & Analysis of Algorithms 22
Calculating the Greatest Common Divisor
Denoted by gcd(m,n)Finding the largest integer
divides both m and n exactly
gcd(6,15)=3gcd(10,21)=1
gcd algorithm takes time of order nEuclid’s algorithm takes order of logn
function gcd(m,n)i=min(m,n)+1repeat
i=i-1until i divides both m and n exactlyreturn i
function Euclid(m,n)while m>0 do
t=mm=n mod mn=t
return n
April 21, 2023 CS 615 Design & Analysis of Algorithms 23
Calculating Fibonacci Sequences
Fibonacci Sequence:f0=0;
f1=1;
fn= fn-1 + fn-2
Order of Fibrec is fn
Order of Fibiter is n
function Fibrec(n)if n<2 then return nelse return Fibrec(n-1)+Fibrec(n-2)
function Fibiter(n)i=1j=0for k=1 to n do
j=i+ji=j-i
return j
n 10 20 30 50 100
Fibrec 8ms 1sec 2min 21days
109years
Fibiter 0.17ms
0.33ms 0.5ms 0.75ms
1.5ms
April 21, 2023 CS 615 Design & Analysis of Algorithms 24
Fourier TransformsOne of the most useful algorithm in historyUsed in
OpticsAcousticsQuantum physicsTelecommunicationsSystem theorySignal processingSpeech processing
ExampleUsed to analyze data from
earthquake in Alaska 1964Classic algorithm takes 26 minutes of computationA new algorithm need less than 2.5 seconds
April 21, 2023 CS 615 Design & Analysis of Algorithms 25
End of Chapter 3Efficiency of Algorithms