Algorithmic Analysis & Design Introduction
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
- 1. Algorithmic Analysis & Design Introduction
- 2. Introduction Abu Jafar Muhammad ibn Musa Al- Khwarizmi [Born: about 780 in Baghdad (now in Iraq). Died: about 850] [al-jabr means "restoring", referring to the process of moving a subtracted quantity to the other side of an equation; al- muqabala is "comparing" and refers to subtracting equal quantities from both sides of an equation.]
- 3. Introduction An algorithm, named after the ninth century scholar Abu Jafar Muhammad Ibn Musu Al-Khowarizmi, is defined as follows: Roughly speaking: An algorithm is a set of rules for carrying out calculation either by hand or on a machine. An algorithm is a finite step-by-step procedure to achieve a required result. An algorithm is a sequence of computational steps that transform the input into the output. An algorithm is a sequence of operations performed on data that have to be organized in data structures. An algorithm is an abstraction of a program to be executed on a physical machine (model of Computation).
- 4. What is an Algorithm? An algorithm is a sequence of unambiguous instructions for solving a problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time.
- 5. The most famous algorithm in history dates well before the time of the ancient Greeks: this is Euclids algorithm for calculating the greatest common divisor of two integers.
- 6. Example-1: The ClassicMultiplication Algorithm1. Multiplication, the American way: Multiply the multiplicand one after another by each digit of the multiplier taken from right to left.
- 7. 2. Multiplication, the English way: Multiply the multiplicand one after another by each digit of the multiplier taken from left to right.
- 8. Example-2: Chain MatrixMultiplication Want: ABCD =? Method 1: (AB)(CD) Method 2: A((BC)D) Method 1 is much more efficient than Method 2.
- 9. What is this course about? There are usually more than one algorithm for solving a problem. Some algorithms are more efficient than others. We want the most efficient algorithm.
- 10. What is this course about? If we have two algorithms, we need to find out which one is more efficient. To do so, we need to analyze each of them to determine their efficiency. Of course, we must also make sure the algorithm are correct.
- 11. What is this course about?In this course, we will discuss fundamental techniques for:1. Proving the correctness of algorithms,2. Analyzing the running times of algorithms,3. Designing efficient algorithms,4. Showing efficient algorithms do not exist for some problemsNotes: Analysis and design go hand in hand: By analyzing the running times of algorithms, we will know how to design fast algorithms. In CS207, you have learned techniques for 1, 2, 3 w.r.t sorting and searching. In CS401, we will discuss them in the contexts of other problems.
- 12. Differences between CS-207 &CS-401
- 13. Analysis and Design Algorithmic is a branch of computer science that consists of designing and analyzing computer algorithms1. The design pertain to The description of algorithm at an abstract level by means of a pseudo language, and Proof of correctness that is, the algorithm solves the given problem in all cases.2. The analysis deals with performance evaluation (complexity analysis).
- 14. Analysis and Design We start with defining the model of computation, which is usually the Random Access Machine (RAM) model, but other models of computations can be use such as PRAM. Once the model of computation has been defined, an algorithm can be describe using a simple language (or pseudo language) whose syntax is close to programming language such as C# or java.
- 15. Analysis of algorithms The theoretical study of computer- program performance and resource usage. Whats more important than performance? modularity user-friendliness correctness programmer time maintainability simplicity functionality extensibility robustness reliability
- 16. Why study algorithms andperformance? Algorithms help us to understand scalability. Performance often draws the line between what is feasible and what is impossible. Algorithmic mathematics provides a language for talking about program behavior. Performance is the currency of computing. The lessons of program performance generalize to other computing resources. Speed is fun!
- 17. Algorithms Performance Two important ways to characterize the effectiveness of an algorithm are its: Time Complexity: how much time will the program take? Space Complexity: how much storage will the program need?
- 18. Complexity Complexity refers to the rate at which the storage or time grows as a function of the problem size. The absolute growth depends on the machine used to execute the program, the compiler used to construct the program, and many other factors. We would like to have a way of describing the inherent complexity of a program (or piece of a program), independent of machine/compiler considerations. This means that we must not try to describe the absolute time or storage needed. We must instead concentrate on a "proportionality" approach, expressing the complexity in terms of its relationship to some known function. This type of analysis is known as asymptotic analysis.
- 19. Complexity Time complexity of an algorithm concerns determining an expression of the number of steps needed as a function of the problem size. Since the step count measure is somewhat coarse, one does not aim at obtaining an exact step count. Instead, one attempts only to get asymptotic bounds on the step count. Asymptotic analysis makes use of the O (Big Oh) notation. Two other notational constructs used by computer scientists in the analysis of algorithms are (Big Theta) notation and (Big Omega) notation.
- 20. O-Notation (Upper Bound) This notation gives an upper bound for a function to within a constant factor. We write f(n) = O(g(n)) if there are positive constants n0 and c such that to the right of n0, the value of f(n) always lies on or below cg(n).
- 21. -Notation (Same order) This notation bounds a function to within constant factors. We say f(n) = (g(n)) if there exist positive constants n0, c1 and c2 such that to the right of n0 the value of f(n) always lies between c1g(n) and c2g(n) inclusive.
- 22. -Notation (Lower Bound) This notation gives a lower bound for a function to within a constant factor. We write f(n) = (g(n)) if there are positive constants n0 and c such that to the right of n0, the value of f(n) always lies on or above cg(n).
- 23. Asymptotic Analysis Asymptotic analysis is based on the idea that as the problem size grows, the complexity can be described as a simple proportionality to some known function. This idea is incorporated in the "Big Oh" notation for asymptotic performance. Definition: T(n) = O(f(n)) if and only if there are constants c0 and n0 such that T(n) = n0. The expression "T(n) = O(f(n))" is read as "T of n is in Big Oh of f of n." Big Oh is sometimes said to describe an "upper-bound" on the complexity. Other forms of asymptotic analysis ("Big Omega", "Little Oh", "Theta") are similar in spirit to Big Oh.
- 24. Example 1 Suppose we have a program that takes some constant amount of time to set up, then grows linearly with the problem size n. The constant time might be used to prompt the user for a filename and open the file. Neither of these operations are dependent on the amount of data in the file. After these setup operations, we read the data from the file and do something with it (say print it). The amount of time required to read the file is certainly proportional to the amount of data in the file. We let n be the amount of data. This program has time complexity of O(n). To see this, lets assume that the setup time is really long, say 500 time units. Lets also assume that the time taken to read the data is 10n, 10 time units for each data point read. The following graph shows the function 500 + 10n plotted against n, the problem size. Also shown are the functions n and 20 n.
- 25. Note that the function n will never be larger than the function 500 + 10 n, no matter how large n gets. However, there are constants c0 and n0 such that 500 + 10n = n0. One choice for these constants is c0 = 20 and n0 = 50. Therefore, 500 + 10n = O(n). There are, of course, other choices for c0 and n0. For example, any value of c0 > 10 will work for n0 = 50.
- 26. Big Oh Does Not Tell theWhole Story Suppose you have a choice of two approaches to writing a program. Both approaches have the same asymptotic performance (for example, both are O(n lg(n)). Why select one over the other, theyre both the same, right? They may not be the same. There is this small matter of the constant of proportionality. Suppose algorithms A and B have the same asymptotic performance, TA(n) = TB(n) = O(g(n)). Now suppose that A does ten operations for each data item, but algorithm B only does three. It is reasonable to expect B to be faster than A even though both have the same asymptotic performance. The reason is that asymptotic analysis ignores constants of proportionality.
- 27. As a specific example, lets say that algorithm A is: { set up the algorithm, taking 50 time units; read in n elements into array A; /* 3 units per element */ for (i = 0; i < n; i++) { do operation1 on A[i]; /* takes 10 units */ do operation2 on A[i]; /* takes 5 units */ do operation3 on A[i]; /* takes 15 units */ } }
- 28. Lets now say that algorithm B is { set up the algorithm, taking 200 time units; read in n elements into array A; /* 3 units per element */ for (i = 0; i < n; i++) { do operation1 on A[i]; /* takes 10 units */ do operation2 on A[i]; /* takes 5 units */ } }
- 29. Algorithm A sets up faster than B, but does more operations on the data. The execution time of A and B will be TA(n) = 50 + 3*n + (10 + 5 + 15)*n = 50 + 33*n and TB(n) =200 + 3*n + (10 + 5)*n = 200 + 18*n respectively.
- 30. The following graph shows the execution time for the two algorithms as a function of n. Algorithm A is the better choice for small values of n. For values of n > 10, algorithm B is the better choice. Remember that both algorithms have time complexity O(n).
- 31. Analyzing AlgorithmsPredict resource utilization 1- Running time (time complexity) focus of this course 2- Memory (space complexity)Running time: the number of primitive operations (e.g., addition,multiplication, comparisons) used to solve the problemDepends on problem instance: often we find an upper bound:T(input size) and use asymptotic notations (big-Oh) To make your life easier! Growth rate much more important than constants in big- Oh.Input size n: rigorous definition given later sorting: number of items to be sorted graphs: number of vertices and edges
- 32. The running time depends on the input: an already sorted sequence is easier to sort. Parameterize the running time by the size of the input, since short sequences are easier to sort than long ones. Generally, we seek upper bounds on the running time, because everybody likes a guarantee.
- 33. Three Cases of Analysis: I Best Case: An instance for a given size n that results in the fastest possible running time.
- 34. Three Cases of Analysis: II Worst Case: An instance for a given size n that results in the slowest possible running time.
- 35. Three Cases of Analysis: III Average Case: Average running time over every possible instances for the given size, assuming some probability distribution of the instances.
- 36. Three Cases of Analysis Best case: Clearly the worst Worst case: Commonly used, will also be used in this course Gives a running time guarantee no matter what the input is Fair comparison among different algorithms Average case: Used sometimes Need to assume some distribution: real-world inputs are seldom uniformly random! Analysis is complicated Will not use in this course
- 37. Because, it is quite difficult to estimate the statistical behavior of the input, most of the time we content ourselves to a worst case behavior. Most of the time, the complexity of g(n) is approximated by its family o(f(n)) where f(n) is one of the following functions. n (linear complexity), log n (logarithmic complexity), na where a2 (polynomial complexity), an (exponential complexity).
- 38. Optimality Once the complexity of an algorithm has been estimated, the question arises whether this algorithm is optimal. An algorithm for a given problem is optimal if its complexity reaches the lower bound over all the algorithms solving this problem. For example, any algorithm solving the intersection of n segments problem will execute at least n2 operations in the worst case even if it does nothing but print the output. This is abbreviated by saying that the problem has (n2) complexity. If one finds an O(n2) algorithm that solve this problem, it will be optimal and of complexity (n2).
- 39. Reduction Another technique for estimating the complexity of a problem is the transformation of problems, also called problem reduction. As an example, suppose we know a lower bound for a problem A, and that we would like to estimate a lower bound for a problem B. If we can transform A into B by a transformation step whose cost is less than that for solving A, then B has the same bound as A. The Convex hull problem nicely illustrates "reduction" technique. A lower bound of Convex- hull problem established by reducing the sorting problem (complexity: (nlogn)) to the Convex hull problem.
- 40. Algorithm An algorithm is a well defined computational procedure that transforms inputs into outputs, achieving the desired input-output relationship A correct algorithm halts with the correct output for every input instance. We can then say that the algorithm solves the problem
- 41. Definition: An algorithm is a finite set of instructions which, if followed, will accomplish a particular task. In addition every algorithm must satisfy the following criteria: i) input: there are zero or more quantities which are externally supplied; ii) output: at least one quantity is produced; iii) definiteness: each instruction must be clear and unambiguous; iv) finiteness: if we trace out the instructions of the algorithm, then for all valid cases the algorithm will terminate after a finite number of steps; v) effectiveness: every instruction must be sufficiently basic that it can in principle be carried out by a person using only a pencil and paper. It is not enough that each operation be definite as in (iii), but it must be feasible. [Hor76]
- 42. Algorithmic Effectiveness Note the above requirement that an algorithm in principle be carried out by a person using only a pencil and paper. This limits the operations that are allowable in a pure algorithm. The basic list is: addition and subtraction; multiplication and division; and the operations immediately derived from these.
- 43. One might argue for expansion of this list to include a few other operations that can be done by pencil and paper. I know an easy algorithm for taking the square root of integers, and can do that with a pencil on paper. However, this algorithm will produce an exact answer for only those integers that are the square of other integers; their square root must be an integer. Operations, such as the trigonometric functions, are not basic operations. These can appear in algorithms only to the extent that they are understood to represent a sequence of basic operations that can be well defined.
- 44. More on Effectiveness Valid Examples: Sort an array in nonincreasing order. Note: The step does not need to correspond to a single instruction. Find the square root of a given real number to a specific precision. Invalid Examples: Find four integers A 0, B 0, C 0, and N 3, such that AN + BN = CN. This is generally thought to be impossible. Find the exact square root of an arbitrary real number. The square root of 4.0 is exactly 2.0. The square root of 7.0 is some number between 2.645 751 311 064 590 and 2.645 751 311 064 591. Later we shall consider methods to terminate an algorithm based on a known bound on the precision of the answer.
- 45. Heuristics vs. Algorithms Computer Science Definition A heuristic is a procedure, similar to an algorithm, that often produces a correct answer, but is not guaranteed to produce anything. Operations Research Definition Problems are usually optimization problems; either maximize profits and minimize costs. An algorithm is defined in the sense of computer science with the additional requirement that it always produces an optimal answer. A heuristic is a procedure that often produces an optimal answer, but cannot be proven to do so. NOTE: Both algorithms and heuristics can contain highlevel statements that cannot be implemented directly in common programming languages; e.g. sort the array lowtohigh.
- 46. Euclids Algorithm Compute the greatest common divisor of two nonnegative integers. For two integers K 0 and N 0, we say that K divides N, equivalently N is a multiple of K, if N is an integral multiple of K: N = KL. This property is denoted K | N, which is read as K divides N. The greatest common divisor of two integers M 0 and N 0, denoted gcd(M, N) is defined as follows. 1) gcd(M, 0) = gcd(0, M) = M 2) gcd(M, N) = K, such that a) K|M b) K|N c) K is the largest integer with this property. Euclids Algorithm: gcd(M, N) = gcd (N, M mod N)
- 47. Example of Euclids Algorithmgcd(M, N) = gcd (N, M mod N)gcd(225, 63) = gcd(63, 36) as 225 = 63 3 + 36, so 225 mod 63 = 36 = gcd(36, 27) as 63 = 36 1 + 27, so 63 mod 36 = 27 = gcd(27, 9) as 36 = 27 1 + 9, so 36 mod 27 = 9 = gcd(9, 0) as 27 = 9 3, so 27 mod 9 = 0 = 9 as gcd(N, 0) = N for any N 0. gcd(63, 255) = gcd(255, 63)as 63 = 0 255 + 63, so 63 mod 255 = 63
Related Documents
