YOU ARE DOWNLOADING DOCUMENT

Please tick the box to continue:

Transcript
Page 1: Aad introduction

Algorithmic Analysis & Design

Introduction

Page 2: Aad introduction

Introduction

Abu Ja'far Muhammad ibn Musa Al-Khwarizmi [Born: about 780 in Baghdad (now in Iraq). Died: about 850]

[al-jabr means "restoring", referring to the process of moving a subtracted quantity to the other side of an equation; al-muqabala is "comparing" and refers to subtracting equal quantities from both sides of an equation.]

Page 3: Aad introduction

Introduction

An algorithm, named after the ninth century scholar Abu Jafar Muhammad Ibn Musu Al-Khowarizmi, is defined as follows: Roughly speaking:

An algorithm is a set of rules for carrying out calculation either by hand or on a machine.

An algorithm is a finite step-by-step procedure to achieve a required result.

An algorithm is a sequence of computational steps that transform the input into the output.

An algorithm is a sequence of operations performed on data that have to be organized in data structures.

An algorithm is an abstraction of a program to be executed on a physical machine (model of Computation).

Page 4: Aad introduction

What is an Algorithm?

An algorithm is a sequence of unambiguous instructions for solving a problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time.

Page 5: Aad introduction
Page 6: Aad introduction

The most famous algorithm in history dates well before the time of the ancient Greeks: this is Euclid's algorithm for calculating the greatest common divisor of two integers.

Page 7: Aad introduction

Example-1: The Classic Multiplication Algorithm

1. Multiplication, the American way:

Multiply the multiplicand one after another by each digit of the multiplier taken from right to left.

Page 8: Aad introduction

2. Multiplication, the English way:

Multiply the multiplicand one after another by each digit of the multiplier taken from left to right.

Page 9: Aad introduction

Example-2: Chain Matrix Multiplication

Want: ABCD =? Method 1: (AB)(CD) Method 2: A((BC)D) Method 1 is much more efficient than Method

2.

Page 10: Aad introduction

What is this course about?

There are usually more than one algorithm for solving a problem.

Some algorithms are more efficient than others.

We want the most efficient algorithm.

Page 11: Aad introduction

What is this course about?

If we have two algorithms, we need to find out which one is more efficient.

To do so, we need to analyze each of them to determine their efficiency.

Of course, we must also make sure the algorithm are correct.

Page 12: Aad introduction

What is this course about?

In this course, we will discuss fundamental techniques for:1. Proving the correctness of algorithms,2. Analyzing the running times of algorithms,3. Designing efficient algorithms,4. Showing efficient algorithms do not exist for some problems

Notes: Analysis and design go hand in hand:

By analyzing the running times of algorithms, we will know how to design fast algorithms.

In CS207, you have learned techniques for 1, 2, 3 w.r.t sorting and searching. In CS401, we will discuss them in the contexts of other problems.

Page 13: Aad introduction

Differences between CS-207 & CS-401

Page 14: Aad introduction

Analysis and Design

Algorithmic is a branch of computer science that consists of designing and analyzing computer algorithms

1. The “design” pertain to The description of algorithm at an abstract level by

means of a pseudo language, and Proof of correctness that is, the algorithm solves the

given problem in all cases.

2. The “analysis” deals with performance evaluation (complexity analysis).

Page 15: Aad introduction

Analysis and Design

We start with defining the model of computation, which is usually the Random Access Machine (RAM) model, but other models of computations can be use such as PRAM. Once the model of computation has been defined, an algorithm can be describe using a simple language (or pseudo language) whose syntax is close to programming language such as C# or java.

Page 16: Aad introduction

Analysis of algorithms

The theoretical study of computer-program performance and resource usage.

What’s more important than performance?• modularity• correctness• maintainability• functionality• robustness

• user-friendliness• programmer time• simplicity• extensibility• reliability

Page 17: Aad introduction

Why study algorithms and performance?

Algorithms help us to understand scalability. Performance often draws the line between

what is feasible and what is impossible. Algorithmic mathematics provides a

language for talking about program behavior. Performance is the currency of computing. The lessons of program performance

generalize to other computing resources. Speed is fun!

Page 18: Aad introduction

Algorithm's Performance

Two important ways to characterize the effectiveness of an algorithm are its:

Time Complexity: how much time will the program take?

Space Complexity: how much storage will the program need?

Page 19: Aad introduction

Complexity Complexity refers to the rate at which the storage or time

grows as a function of the problem size. The absolute growth depends on the machine used to execute the program, the compiler used to construct the program, and many other factors. We would like to have a way of describing the inherent complexity of a program (or piece of a program), independent of machine/compiler considerations. This means that we must not try to describe the absolute time or storage needed. We must instead concentrate on a "proportionality" approach, expressing the complexity in terms of its relationship to some known function. This type of analysis is known as asymptotic analysis.

Page 20: Aad introduction

Complexity

Time complexity of an algorithm concerns determining an expression of the number of steps needed as a function of the problem size. Since the step count measure is somewhat coarse, one does not aim at obtaining an exact step count.

Instead, one attempts only to get asymptotic bounds on the step count. Asymptotic analysis makes use of the O (Big Oh) notation. Two other notational constructs used by computer scientists in the analysis of algorithms are Θ (Big Theta) notation and Ω (Big Omega) notation.

Page 21: Aad introduction

O-Notation (Upper Bound)

This notation gives an upper bound for a function to within a constant factor. We write f(n) = O(g(n)) if there are positive constants n0 and c such that to the right of n0, the value of f(n) always lies on or below cg(n).

Page 22: Aad introduction

Θ-Notation (Same order)

This notation bounds a function to within constant factors. We say f(n) = Θ(g(n)) if there exist positive constants n0, c1 and c2 such that to the right of n0 the value of f(n) always lies between c1g(n) and c2g(n) inclusive.

Page 23: Aad introduction

Ω-Notation (Lower Bound)

This notation gives a lower bound for a function to within a constant factor. We write f(n) = Ω(g(n)) if there are positive constants n0 and c such that to the right of n0, the value of f(n) always lies on or above cg(n).

Page 24: Aad introduction

Asymptotic Analysis Asymptotic analysis is based on the idea that as the

problem size grows, the complexity can be described as a simple proportionality to some known function. This idea is incorporated in the "Big Oh" notation for asymptotic performance.

Definition: T(n) = O(f(n)) if and only if there are constants c0 and n0 such that T(n) <= c0 f(n) for all n >= n0. The expression "T(n) = O(f(n))" is read as "T of n is in Big Oh of f of n." Big Oh is sometimes said to describe an "upper-bound" on the complexity. Other forms of asymptotic analysis ("Big Omega", "Little Oh", "Theta") are similar in spirit to Big Oh.

Page 25: Aad introduction

Example 1 Suppose we have a program that takes some constant amount

of time to set up, then grows linearly with the problem size n. The constant time might be used to prompt the user for a filename and open the file. Neither of these operations are dependent on the amount of data in the file. After these setup operations, we read the data from the file and do something with it (say print it). The amount of time required to read the file is certainly proportional to the amount of data in the file. We let n be the amount of data. This program has time complexity of O(n). To see this, let's assume that the setup time is really long, say 500 time units. Let's also assume that the time taken to read the data is 10n, 10 time units for each data point read. The following graph shows the function 500 + 10n plotted against n, the problem size. Also shown are the functions n and 20 n.

Page 26: Aad introduction
Page 27: Aad introduction

Note that the function n will never be larger than the function 500 + 10 n, no matter how large n gets. However, there are constants c0 and n0 such that 500 + 10n <= c0 n when n >= n0. One choice for these constants is c0 = 20 and n0 = 50. Therefore, 500 + 10n = O(n). There are, of course, other choices for c0 and n0. For example, any value of c0 > 10 will work for n0 = 50.

Page 28: Aad introduction

Big Oh Does Not Tell the Whole Story Suppose you have a choice of two approaches to writing

a program. Both approaches have the same asymptotic performance (for example, both are O(n lg(n)). Why select one over the other, they're both the same, right? They may not be the same. There is this small matter of the constant of proportionality. Suppose algorithms A and B have the same asymptotic performance, TA(n) = TB(n) = O(g(n)). Now suppose that A does ten operations for each data item, but algorithm B only does three. It is reasonable to expect B to be faster than A even though both have the same asymptotic performance. The reason is that asymptotic analysis ignores constants of proportionality.

Page 29: Aad introduction

As a specific example, let's say that algorithm A is: { set up the algorithm, taking 50 time units; read in n elements into array A; /* 3 units per element */ for (i = 0; i < n; i++) { do operation1 on A[i]; /* takes 10 units */ do operation2 on A[i]; /* takes 5 units */ do operation3 on A[i]; /* takes 15 units */ }}

Page 30: Aad introduction

Let's now say that algorithm B is {

set up the algorithm, taking 200 time units;

read in n elements into array A; /* 3 units per element */

for (i = 0; i < n; i++)

{

do operation1 on A[i]; /* takes 10 units */

do operation2 on A[i]; /* takes 5 units */

}

}

Page 31: Aad introduction

Algorithm A sets up faster than B, but does more operations on the data. The execution time of A and B will be TA(n) = 50 + 3*n + (10 + 5 + 15)*n = 50 + 33*n

and TB(n) =200 + 3*n + (10 + 5)*n = 200 + 18*n respectively.

Page 32: Aad introduction

The following graph shows the execution time for the two algorithms as a function of n. Algorithm A is the better choice for small values of n. For values of n > 10, algorithm B is the better choice. Remember that both algorithms have time complexity O(n).

Page 33: Aad introduction
Page 34: Aad introduction

Analyzing Algorithms

Predict resource utilization1- Running time (time complexity) — focus of this course2- Memory (space complexity)

Running time: the number of primitive operations (e.g., addition,multiplication, comparisons) used to solve the problemDepends on problem instance: often we find an upper bound:T(input size) and use asymptotic notations (big-Oh)

To make your life easier! Growth rate much more important than constants in big-

Oh.Input size n: rigorous definition given later

sorting: number of items to be sorted graphs: number of vertices and edges

Page 35: Aad introduction

The running time depends on the input: an already sorted sequence is easier to sort.

Parameterize the running time by the size of the input, since short sequences are easier to sort than long ones.

Generally, we seek upper bounds on the running time, because everybody likes a guarantee.

Page 36: Aad introduction

Three Cases of Analysis: I Best Case: An instance for a given size n that

results in the fastest possible running time.

Page 37: Aad introduction

Three Cases of Analysis: II

Worst Case: An instance for a given size n that results in the slowest possible running time.

Page 38: Aad introduction

Three Cases of Analysis: III

Average Case: Average running time over every possible instances for the given size, assuming some probability distribution of the instances.

Page 39: Aad introduction

Three Cases of Analysis Best case: Clearly the worst Worst case: Commonly used, will also be used in this

course Gives a running time guarantee no matter what the input

is Fair comparison among different algorithms

Average case: Used sometimes Need to assume some distribution: real-world inputs are

seldom uniformly random! Analysis is complicated Will not use in this course

Page 40: Aad introduction

Because, it is quite difficult to estimate the statistical behavior of the input, most of the time we content ourselves to a worst case behavior. Most of the time, the complexity of g(n) is approximated by its family o(f(n)) where f(n) is one of the following functions. n (linear complexity), log n (logarithmic complexity), na where a≥2 (polynomial complexity), an (exponential complexity).

Page 41: Aad introduction

Optimality

Once the complexity of an algorithm has been estimated, the question arises whether this algorithm is optimal. An algorithm for a given problem is optimal if its complexity reaches the lower bound over all the algorithms solving this problem. For example, any algorithm solving “the intersection of n segments” problem will execute at least n2 operations in the worst case even if it does nothing but print the output. This is abbreviated by saying that the problem has Ω(n2) complexity. If one finds an O(n2) algorithm that solve this problem, it will be optimal and of complexity Θ(n2).

Page 42: Aad introduction

Reduction Another technique for estimating the complexity of a

problem is the transformation of problems, also called problem reduction. As an example, suppose we know a lower bound for a problem A, and that we would like to estimate a lower bound for a problem B. If we can transform A into B by a transformation step whose cost is less than that for solving A, then B has the same bound as A.

The Convex hull problem nicely illustrates "reduction" technique.  A lower bound of Convex-hull problem established by reducing the sorting problem (complexity: Θ(nlogn)) to the Convex hull problem. 

Page 43: Aad introduction

Algorithm

An algorithm is a well defined computational procedure that transforms inputs into outputs, achieving the desired input-output relationship

A correct algorithm halts with the correct output for every input instance. We can then say that the algorithm solves the problem

Page 44: Aad introduction

Definition: An algorithm is a finite set of instructions which, if followed, will accomplish a particular task. In addition every algorithm must satisfy the following criteria:i) input: there are zero or more quantities which are externally

supplied;ii) output: at least one quantity is produced;

iii) definiteness: each instruction must be clear and unambiguous;

iv) finiteness: if we trace out the instructions of the algorithm, then for all valid cases the algorithm will terminate after a finite number of steps;v) effectiveness: every instruction must be sufficiently basic that it can in principle be carried out by a person

using only a pencil and paper. It is not enough that each operation be definite as in (iii), but it must be feasible. [Hor76]

Page 45: Aad introduction

Algorithmic Effectiveness

Note the above requirement that an algorithm “in principle be carried out by a person using only a pencil and paper”.

This limits the operations that are allowable in a “pure algorithm”.The basic list is: addition and subtraction; multiplication and division; and the operations immediately derived from

these.

Page 46: Aad introduction

One might argue for expansion of this list to include a few other operations that can be done by pencil and paper. I know an easy algorithm for taking the square root of integers, and can do that with a pencil on paper.

However, this algorithm will produce an exact answer for only those integers that are the square of other integers; their square root must be an integer.

Operations, such as the trigonometric functions, are not basic operations. These can appear in algorithms only to the extent that they are understood to represent a sequence of basic operations that can be well defined.

Page 47: Aad introduction

More on Effectiveness Valid Examples:

Sort an array in non–increasing order. Note: The step does not need to correspond to a single

instruction. Find the square root of a given real number to a specific

precision. Invalid Examples:

Find four integers A 0, B 0, C 0, and N 3, such that AN + BN = CN. This is generally thought to be impossible.

Find the exact square root of an arbitrary real number. The square root of 4.0 is exactly 2.0. The square root of 7.0 is some number between 2.645 751 311

064 590 and 2.645 751 311 064 591. Later we shall consider methods to terminate an algorithm based

on a known bound on the precision of the answer.

Page 48: Aad introduction

Heuristics vs. Algorithms Computer Science Definition

A heuristic is a procedure, similar to an algorithm, that often produces a correct answer, but is not guaranteed to produce anything.

Operations Research DefinitionProblems are usually optimization problems;

either maximize profits and minimize costs.An algorithm is defined in the sense of computer sciencewith the additional requirement that it always produces an optimal answer.

A heuristic is a procedure that often produces an optimal answer,but cannot be proven to do so.

NOTE: Both algorithms and heuristics can contain high–level statements that cannot be implemented directly in common programming languages; e.g. “sort the array low–to–high”.

Page 49: Aad introduction

Euclid’s Algorithm Compute the greatest common divisor of two non–negative

integers. For two integers K 0 and N 0, we say that “K divides N”,

equivalently “N is a multiple of K”, if N is an integral multiple of K: N =

KL. This property is denoted “K | N”, which is read as “K divides N”. The greatest common divisor of two integers M 0 and N 0,

denoted gcd(M, N) is defined as follows.

1) gcd(M, 0) = gcd(0, M) = M2) gcd(M, N) = K, such that

a) K | Mb) K | Nc) K is the largest integer with this property.

Euclid’s Algorithm: gcd(M, N) = gcd (N, M mod N)

Page 50: Aad introduction

Example of Euclid’s Algorithm

gcd(M, N) = gcd (N, M mod N)gcd(225, 63) = gcd(63, 36) as 225 = 63 3 + 36, so 225 mod

63 = 36= gcd(36, 27) as 63 = 36 1 + 27, so 63 mod 36

= 27

= gcd(27, 9) as 36 = 27 1 + 9, so 36 mod 27 = 9= gcd(9, 0) as 27 = 9 3, so 27 mod 9 = 0= 9 as gcd(N, 0) = N for any N ≥ 0.

gcd(63, 255) = gcd(255, 63) as 63 = 0 255 + 63, so 63 mod 255 = 63


Related Documents