Algorithms Chapter 1 Oct. 5 2011

Design and Analysis of Algorithm By Dr. Ahmad Awwad

Petra UniversityFaculty of Information Technology

Design and Analysis of

Algorithms

Chapter 1

1 | P a g e


Introduction

Q: what is the importance of studying algorithms?

1- To know the standard set of important algorithms from different areas of computing.

2- To be able to design new algorithms and analyze their efficiency3- Compare programs that would not exist without algorithms.4- The usefulness and importance of algorithms in delivering analytical skills.

Algorithm is the core of computer science and it is relevant to most of science, business, technology, etc….

Q: what is an algorithm?

Informally, an algorithm is any well-defined computational procedure that takes some value, or set of values, as input and produces some value, or set of values, as output. An algorithm is thus a sequence of computational steps that transform the input into the output.

Furthermore: The algorithm could be defined as a sequence of unambiguous instructions for solving a problem to obtain a required output.

2 | P a g e

Algorithm

Input Output

Problem

Processing Computer


- Very important points in solving algorithms.1- The range of input for which an algorithm works has to be specified

carefully.2- The same algorithm can be represented in several different ways.3- Several algorithms for solving the same problem may exist.4- Algorithms for the same problem can be based on very different ideas and

may solve problems with different speeds.

Algorithms can be classified according to:

1- The problem type (sorting, searching, graphs, string matching)

2- Algorithm design techniques.

Example 1:

Three algorithms that will solve the GCD of 2 integers:

The greatest common divisor (GCD) of 2 non-negative & non-both-zero, denoted by GCD (m, n), Defined as the largest integer that divides both m & n with a remainder zero.

Solution:

A. Euclid’s" proposed an algorithm for solving this problem [Euclids Alg.] GCD (m, n) = GCD (n, m % n). { "(%) mod" is the reminder of division "m" by "n" until n = 0}.

GCD (60, 24): GCD (24, 60%24) → GCD (24,12) → GCD (12, 24%12) → GCD (12,0) = 12

The Euclid’s Algorithm for computing GCD(m, n) is:Step1: If n=0 return the value of m as the answer and stop, otherwise proceed to step2.Step2: Divide “mod” m by n and assign the reminder to r.Step3: Assign the value of n to m & the value of r to n.Step4: Go to step1.

We may write the above algorithm in pseudo-code:

//Compute GCD (m, n) using Euclid's algorithm

//Input: two nun-negative, non-both-zeros integers’ m & n.

3 | P a g e


//Output: GCD of m & n.

While n ≠ 0 do

r ← m mod n

m ← n // assign n to m

n ← r

return m

B. Consecutive integer checking algorithm computing GCD(m, n):

Step1: Assign the value of min (m, n) to t.Step2: Divide m by t. If the reminder of this division is 0, go to step 3; otherwise go to step 4.Step3: Divide n by t. If the reminder of this division is 0, return the value of t as the answer and stop; otherwise, proceed to step 4.

Step 4: Decrease the value of t by 1. Go to step 2.

C. Middle school procedure for computing GCD(m, n):

Step1: Find the primes of m.Step2: Find the primes of n.Step3: find the common factor in the two primes found in step 1 & 2.[If P is a common factors occurring Pn & Pm times in m & n respectively it should be repeated min {Pn, Pm} times].Step4: compute the product of all common factors and return it as the GCD of given numbers.

The Algorithm for generating consecutive primes not exceeding any given integer "Sieve of Eratosthense" is introduced below:

Algorithm Sieve(n)// Input: An integer n >= 2// Output: Array L of all prime numbers less than or equal to nFor p 2 to n do A[p] p For p 2 to sqtr(n) do

If A[p] 0 // p hasn’t been eliminated on previous passesJ p*pWhile j n do

4 | P a g e


A[ j ] 0 // mark element as eliminatedj j + p

// copy the remaining elements of A to array L of the primesi 0

For p 2 to n do

If A[p] 0 L[ i ] A[ p ] i i + 1

Return L

Homework # 1:

1. Do some research on al-Khorezmi, the man from whose name the word “algorithm” is derived? In particular, you should learn what the origins of the words “algorithm” and “algebra” have in common.

2. Euclid’s algorithm, as presented in Euclid’s treatise, uses subtractions rather than integer divisions. Write a pseudocode for this version of Euclid’s algorithm

3. Implement the 3 algorithm that finds the greatest common divisor (GCD) of 2 non-negative & non-both-zero GCD of to programs in any language.

5 | P a g e


Fundamentals of algorithm problem solving

Below a brief list of steps one typically goes through in designing and analyzing of algorithms

1- Understand the problem:- Before design an algorithm, you should understand the problem

completely. - It's helpful to understand how such algorithm works to know its strength

& weakness. Especially if you have to choose among several available algorithms.

*This figure lights on algorithms design and analysis process.

2- Make sure of the computational device abilities.- The architecture called (RAM) Random Access Machine, its assumption is

that instructions are executed one after another.

6 | P a g e

Understand the problem

Decide on:

Computational means.

Exact against approximately solving.

Data structure.

Algorithm design techniques.

Design an algorithm

Prove correctness

Analyze the algorithm

Code the algorithm


- Algorithms designed to be executed on such machines are called sequential algorithms.

- Note that the above assumption of RAM does not hold for newer computers that execute operations concurrently in parallel. "Parallel algorithms"

3- Choosing between exact & approximate problem solving:- An algorithm is called an exact algorithm when it solves the problem

exactly.- An algorithm is called approximate when it solves the algorithms

approximately.

Approximate: ←

Exact: ←

4- Deciding on approximate data structure:- In the new world of object oriented programing, data structure are very

important for both design & analysis algorithms.- The basic data structures will be discussed later. - Algorithms + data structure = program

5- Algorithm design techniques, what it is:It is a general method for mixture of problems from different of computing areas.There are many design techniques including:

Brute force "It is a straight forward technique of solving any problem" – usually directly based on the problem statement and definitions of the concepts involved.

Divide and conquer " this technique is probably the best known algorithm design technique and it is achieved according to the following steps:

A. A problem instance is divided into several smaller instances of the same problem (ideally of the same size).

B. The smaller instances are solved typically recursively.C. Then the solution is obtained from the smaller instances to

get a solution to the original instances.

7 | P a g e


Decrease and conquer "This technique is based on exploiting the relationship a solution to given instance and a solution to a smaller instance of the same problem".

Note – once such relationship is established, it can be exploited either top down (recursively) or bottom up (without a recursion).Recursion is an approach to problem solving in which the solution to a particular problem depends on solutions to smaller instances of the same problem

Greedy algorithms – This technique suggests constructing a solution to an optimization problem through a sequence of steps, each expanding a partially constructed solution obtained so far, until a complete solution to the problem is obtained and reached. Note that on each step, the choice made must be feasible, locally and optimally.A greedy algorithm is a mathematical process that recursively constructs a set of objects from the smallest possible constituent parts.Greedy algorithms look for simple, easy-to-implement solutions to complex, multi-step problems by deciding which next step will provide the most obvious benefit. Such algorithms are called greedy because while the optimal solution to each smaller instance will provide an immediate output, the algorithm doesn’t consider the larger problem as a whole. Once a decision has been made, it is never reconsidered.

Dynamic programming – Is an algorithm design technique is like divide and conquer method it solves problems by combining the solutions to subproblems.Note that divide and conquer algorithms partition the problem into independent subproblems, solve the subproblems recursively, and then combine their solutions to solve the original problem. On the other hand Dynamic programming applicable when the subproblems are not independent, that is when subproblems share subproblems.

Transform and conquer – This design method is based on the idea of transformation – These methods work as two stage procedures. First in the transformation stage, the problem's instance is modified to, for one reason or another, more amenable (willing) to solution. Then in the second of conquering stage, it is solved.

8 | P a g e

http://searchcio-midmarket.techtarget.com/definition/recursion

http://searchsecurity.techtarget.com/definition/set


Example: Solve system of linear equations in n unknowns; we can solve such problem by generalizing the substitution method for solving system of 2-linear equations.

6- Methods of specifying an algorithms:

- Note that once you have designed an algorithm, you need to specify it in some fashion.

- Using natural algorithm has an obvious demand.- Also an algorithm in natural language can be written in pseuducode

which is mixture of natural language & programming language.- Pseuducode is more exact than natural language.- Flowchart is a method of expressing an algorithm by collection of

representation geometric shapes containing description of an algorithm steps.

7- Proving an algorithm correctness:

- Note that once an algorithm has been written we have to prove its correctness.

- It means that the algorithm yields a required result for all the possible input in a finite amount of time [ex. Euclid's Alg.]. “Definition”

- Prove of the correctness of algorithm might be very easy or complex.- Common techniques for proving correctness is to use the mathematical

induction.

8- Analyzing algorithm:

- There are two kinds of algorithms efficiency time & space efficiency.1) Time efficiency indicates how fast the algorithm runs.2) Space efficiency indicates how much memory the algorithm needs.

More characteristic of an algorithm are:

Simplicity: (how simple is the algorithm). Generality: how possible that an algorithm can be used on other

machines (sometimes called portability). Portability: Scalability:

9 | P a g e


Readability:

9- Coding the algorithm:

- Algorithms are designed to be implemented on different computers in any language.

Q: write an algorithm to find the minimum distance between two elements in an array of elements [using pseuducode].

Solution:

// input: array A[0,1, 2, 3……., n-1].

// output: minimum distance between two of its elements.

Dimn ← ∞ //"can have the value of".

for i = 0 to n-2 do

for j = 1 to n-1 do

If i ≠ j & | A[ i ] – A[ j ] | < dimn

dimn ← | A[ i ] – A [ j ] |

return dimn

10 | P a g e


Q: What is the task of the following algorithm ?

Call Algorithm MaxIndex (A[0 … n − 1]) where algorithm MaxIndex (A[l..r])

// Input: A portion of array A[0 ... n − 1] between indices l and r (l ≤ r)

// Output: The index of the largest element in A[l..r]

if l = r return l // there is only one element in the array

else temp1 ← MaxIndex (A[l.. (l+r)/2]) //Greatest element in the

//1st half of the portion of A

temp2 ← MaxIndex (A[ (l + r)/2 + 1..r])

if A[temp1] ≥ A[temp2]

return temp1

else return temp2

Homework # 2:

1. Write a pseudocode for an algorithm for finding real roots of equation ax2 + bx + c = 0 for arbitrary real coefficients a, b, and c. (You may assume the availability of the square root function sqrt(x).)

2. Describe the standard algorithm for finding the binary representation ofa positive decimal integera. in English.b. in a pseudocode.

11 | P a g e


Important problems types

The most important problem types:

1) Sorting.2) Searching.3) String processing.4) Graph problems.5) Combinational problems.6) Geometric problem.7) Numerical problem.

1- Sorting:

Sorting problems means to rearrange the items of given list in an increasing or decreasing order.

- Importance of sorting: The list becomes easier to search.- In case of records "contains more than one item of information". We need to

choose a piece of information to guide sorting. “Key”

Example: When we want to sort students records in alphabetical order of names or by student number or by student grads points' average.→ the chosen piece of information called a KEY.

2- Searching problems:

- The searching problems deals with finding a given value called a searching key in the list.

- There are many searching algorithms to choose from, the simplest is sequential search algorithm.

- Note that some algorithms work faster than others, but requires more memory.

- Some algorithms are very fast but applicable only to sorted arrays.

3- String processing:

- String is a sequence of character from an alphabet.- Text strings comprise letters, numbers, and special characters.

12 | P a g e


- Bit strings comprise zeros and ones.- String matching: a problem of searching a given word in a text.

4- Graph problems:

- One of the oldest and most interesting areas in algorithms is graph’s algorithms.

- Graphs can be defined as a collection of points called vertices, some of which are connected by line segments called edges (links).

- Graphs can be used for modeling wide kinds of real life problems and applications including transportation and common networks, projects scheduling and games.

- One of the most important criteria in graphs is diameter, (which is the maximum number of links one need to follow to search one web-page from another by the most direct route between them). “Example S3”

- Graph traversal algorithm: how can one visit the points in a network? "Shortest path algorithm” what is the best route between two cities."

- Traveling sales man problem (TSP) is the problem of finding the shortest tour through cities that visits every city exactly once.

- Graph-coloring problem: assign the smallest number of colors to vertices of a graph and no two adjacent vertices are the same color.

Example: in the real life problems is event scheduling.

5- Combinatorial problem:

- These problems asks (explicitly or implicitly) to find combinatorial objects such as a permutations, a combinations or a subnet that satisfies a certain constraints such as maximize a value or minimize a cost.

- In more abstract perspective the (TSP) and graph coloring problem are example of combinatorial problem.

6- Geometric problems:

- A geometric algorithm deals with geometric objects such as points and lines.- Closest-pair problem: given n points in the plane, find the closest pair among

them.

7- Numerical problems:

13 | P a g e


- Numerical problems are application problems that involve mathematical objects of continuous nature: solving equations and system of equations, computing definite integrals, evaluation functions and so on.

Think about it:An algorithm for the sorting problem that sorts an array by counting for each of its elements, the number of smaller elements, then uses this information to put the elements in its appropriate position in the sorted array.

Homework:

1. Consider the algorithm for the sorting problem that sorts an array by counting, for each of its elements, the number of smaller elements and then uses this information to put the element in its appropriate position in the sorted array:

Algorithm comparison counting sort (A [0, 1, 2, ……, n-1])

// Sort an array by comparison counting

// Input: array A [0, 1, ….,n-1] of ordered values

// Output: array S [0, 1, …., n-1] of A elements sorted in a

// non-decreasing order

For i ← 0 to n-1 do

Count [i] ← 0


For j ← = i+1 to n-1 do

If A [i] < A [j]

Count [j] ←count [j] + 1

Else count [i] ←count[i] + 1


S [count [i]] ← A[i]

Return S

Homework # 3:a. Apply the above algorithm to sorting the list 60, 35, 81, 98, 14, 47.

14 | P a g e


* Hamiltonian Circuit (HC): A path of visits all graphs vertices exactly once before returning to the starting vertex.

Question:

Find and draw the HC for the graph bellow.

Consider the following map:a. Explain how we can use the graph-coloring problem to color the map so that no two neighboring regions are colored the same.

(R: Red, Y: Yellow, G: Green, B: Blue)

15 | P a g e

RG

R

G

BY

1

2

345

6

7 8

9

10

11

12

13

14

15

16

17

18

19

20


Fundamental Data Structure

- Data structure can be defined as particular scheme of organizing related data items.

- Data structure can range elementary data types (integers, characters) to data structure (one-dimension-array).

- Linear data structure: the most elementary data structure is the array & linked list.

- One-dimension-array is sequence of items of the same data type by specifying value of array index.

- Usually one-dimensional-arrays are used for implementing matrices

Item [0] Item [1] Item [2] ……………… Item [n-1]

- Note that in most cases, the index can be integer between of (0 & n-1) or (1 & n).

- Each array element of an array occurrence one time [this feature distinguishes the array from list regardless where the element is located in the array].

Linear Data Structures:

The most important elementary data structures are the array and linked list.

Array: Is a sequence of n items of the same data type that are stored contiguously in computer memory and made accessible by specifying its index 0index n-1. "n # of elements"

Notes:

Each element of the array can be accessed in the same constant amount of time.

Each element in the array occupies the same amount of computer memory.

Arrays are used for implementing a variety data structure as strings [sequence of characters] binary or bit strings which composed of zeros and ones.

Linked List: Is a sequence of zero or more elements called nodes each containing 2 kinds of information: some data and one ore more links called pointers to other nodes of the linked list.

16 | P a g e


Notes:

Special pointer: Called null used to indicate the absence of a node's successor.

Single linked list: Each node expects the last one contains a single pointer to the next element.

The time needed to access an element on singly linked list, we start with the first node & Travers the pointer chain until the particular node is reached [disadvantage].

Linked list do not require any pre reservation of computer memory also insertion & deletion can be made efficiently in a linked list by reconnecting a few appropriate pointers We start a linked list with a special node called the header, it contains information such as: the length of the linked list; pointer to the first element and pointer to the linked list's last element.

Double linked list:

Every node except 1st & last contain pointers to both its successor & predecessor nodes.

……

Notes: Note that the array and linked list are two main choices in represent APS

(Abstract Data Structure) called liner list or list. List is finite sequence of data items which arranged by certain liner order Two special types of lists : stack & queue. The basic operations performed on List are searching, inserting and

deleting an element. Stack: is a list in which insertion & deletion can be done only at the end

(top). - LIFO Adding elements = pushed onto Deleting elements = popped off

17 | P a g e

Item1 Item2 Item3 Itemn-1 null

Item1 Item2 Item3 Item n-1 null


Queue: is a list from which elements are deleted from one end of structure (front), this operation called dequeue.

In queue new elements are added to the other end (rear-this operation called enqueue [FIFO]).

Priority queue is a collection of data items from a totally order universe (integers or real numbers). The most operations done on priority queue is finding its largest element, deleting largest element.

18 | P a g e


Graphs

Definition: A graph is a collection of points in the plane called vertices or nodes; some of them connected by line segment called edges or arcs.

→ Formally

Graph G (V, E) is a defined by pair of two sets: a finite set V of items called vertices & set E of pairs of these vertices called edges.

A pair of vertices (u, v) is the same pair (v, u) → the vertices u & v are adjacent to each other & they are connected by undirected edge (u, v).

A graph G is called undirected if every edge in it is undirected. Note that if a pair of vertices (u, v) is not the same as the pair (v, u), we

say that the edge (u, v) is directed from u to the vertex v. Directed graphs may also called digraphs.

Example:

Figure 'A' has 6 vertices & 7 edges “originally”V = {a, c, b, f, e, d}.E = {(a, c), (c, b), (c, e), (b, f), (e, f), (e, d), (a, d)}.

The following inequality for the number of edges |E| in undirected graph 0 ≤ |E| ≤ (|V| (|V|-1)) /2 → this inequality give us the largest number of edges in the graph if there is an edge connecting each of its vertices with all |V|-1 other vertices [complete graph or fully connected graph].

19 | P a g e

a c b

fed

a c b

fed

)A( (B)


Graph representation

Graphs for computer algorithms can be represented in two main ways:

Adjacency matrix of graph with n vertices in n-by-n Boolean matrix with one row & one column for each of the graph vertices in which the elements, in which the elements in the row & the column is =1, if there is an edge the ith vertex to the vertex and =0 otherwise.

Example: The adjacency matrix of the previous figure.

a b c d e f

a 0 0 1 1 0 0

b 0 0 1 0 0 1

c 1 1 0 0 1 0

d 1 0 0 0 1 0

e 0 0 1 1 0 1

f 0 1 0 0 1 0

Adjacency list of a graph with N vertices is n-by-n representation as:a → c → db → c → fc → a → b → ed → a → ee → c → d → ff → b → e

Weighted graph or diagraph

Defined as a graph with numbers assigned to its edges, these numbers called weights or costs.

20 | P a g e


The following figure shows: 1) weighted graph 2) Its weight matrix 3) Its adjacency lists.

a b c da ∞ 5 1 ∞b 5 ∞ 7 4c 1 7 ∞ 2d ∞ 4 2 ∞

a → b,5 → c,1

b → a,5 → c,7 →d,4

c → a,1 → b,7 → d,2

d→ b,4 → c,2

- Note that the special symbol ∞ is used when no connection.

Paths and cycles

- A path from vertex u to vertex v of a graph G can be defined as a sequence of adjacent vertices that starts at u and ends at v.

- A directed path is a sequence of vertices in which every consecutive pair of the vertices is connected by an edge directed from the vertex listed first to the vertex listed next.

- When all vertices of graph are distinct, the path is simple.- The length of a path is total of vertices in a vertex sequence defining the

path minus one.

Example: a, c, e, f is a directed graph from "a to f in a previous graph.

- Graph is said to be connected if for every pair of its vertices u & v there is a path from u to v.

Example: the following figures A & B has 2 connected components with vertices {a, b, c, d, e} & {f, j, h, i}

- A cycle is a path of positive length that starts & ends at the same vertex and doesn't traverse the same edge more than once.

21 | P a g e

e

c

d

a

1

5

4

2

7

a

e

cb d

a

e

b d)A( (B)


- A graph with no cycle is said to be acyclic.- Sub graph: subgraph of a given graph G = (V,E) is a graph G’ (V’, E’) such

that V’ V & E’ E.

Trees:- Tree (free tree) is a connected cyclic graph (Figure A - below).- Graph that has no cycle, but is not necessary connected is called a forest

(Figure B); each of its connected components is a tree.

- Number of edges in a tree is always one less than the number of its vertices. E = V -1

Rotated tree

- On very important property of a tree is that for every 2 vertices in a tree there is always found exactly one path from one of these vertices to the other one.

- From the above property it is possible to select an arbitrary vertex in a free tree and consider it as the root of the so-called root tree. (figure below)

- The rooted tree is achieved by placing its root on the top.- Obvious applications of trees are hierarchies, from file directories to

organizational charts of enterprises.

22 | P a g e

)A(

a

f

c

b

d

g

a

f

c

b

d

g

e

h

j

i(B)

i

h

c b

g

a

h

f

e

(A)

(B)

a

db e

gc f

hi

Level (0)

Level (1)

Level (2)

Level (3)


- If (u, v) is the last edge of the sample path from the root to vertex the u & v ≠ v, u is said to be the parent of v and v is called the child of u.

- Vertices that have the same parent are said to be siblings.- Vertex with no children is called leaf.- Vertex with at least one child called parental.- The depth of the vortex v is the length of the length of the simple path

form the root to v.- Length of the tree is the longest simple path from the root to the leaf.

Ordered Trees:

- An ordered tree is a rooted tree in which all the children of each vertex are ordered from left to right.

- Binary tree is an ordered tree in which every vertex has no more than two children & each child is either a left child or right child of its parents. (Fig. A)

- The sub tree with its root left/right child of a vertex is called left/right sub tree of the vertices.

- Note that in figure B some numbers are assigned to return of binary tree in A. All numbers assigned to each parent at vertex is larger than all numbers in the left sub tree & smaller than all numbers in the right sub tree, called binary search trees.

- Binary trees are binary search trees have many applications in CS.- A binary tree usually implemented for computing purpose by collection of nodes

corresponding to vertices of the tree, each node contains some information associated with the vertex and two pointers to the nodes representing the left and right child of the vertex.

23 | P a g e

(A)

99

3 12

71 10

4 (B)


Standard Implementation of the binary search tree

First child-next sibling representation of the graph A p. 25 "Top"

24 | P a g e

9

12 null

ull5

1null 10null 7null null

4null null

a null

null f null

b

null g null c

null i null

null d

h

e null


Sets & Dictionaries

Set: is an unordered collection (possible empty) of distinct items called elements of the set.

Example: S = {2, 3, 5, 7} or S = {n: n is a prime number & n<10}.

- The most important operations are checking membership of a given item in a given set, finding the union of the two sets, [set comprise all elements that belong to either of two sets].

- Universal set (U): the largest set.- If set U has n elements then any subset S of U can be represented by a bit

string of size n, called a bit vector, in which the ith element is 1 if and only if the ith element of U is included in set S.

Example: U = {1, 2, 3, 4, 5, 6, 7, 8, 9}

S = {2, 3, 5, 7}

→ will be represented by a string "011010100"

- This way used to represent a set for computing purposes.- Other common way used to represent set for computing purpose is to use

the list structure to indicate the set's element.

The two principle distinction between sets and list:A. Set can not contain identical items, a list canB. Set is unordered collection of items; we may change its

element's order. List is an ordered collection of items (we may not change the order of items)

- List: an ordered collection of objects with several operations that can be performed on them is called an abstract data type (ADT).

- The List, Stack, Queue the Priority Queue and the dictionary are important example of ADT.

- Note that modern object-oriented-language support implementations of ADT by mean of classes.

25 | P a g e


Homework:

1. Describe how one can implement each of the following operations on an array so that the time it takes does not depend on the array’s size n.a. Delete the ith element of an array (1 ≤ i ≤ n).b. Delete the ith element of a sorted array (the remaining array has to stay sorted, of course).

2. a. Show the stack after each operation of the following sequence that starts with the empty stack:push(a), push(b), pop, push(c), push(d), pop

b. Show the queue after each operation of the following sequence that starts with the empty queue:enqueue(a), enqueue(b), dequeue, enqueue(c), enqueue(d), dequeue

3.a. Let A be the adjacency matrix of an undirected graph. Explain what property of the matrix indicates that:

i. The graph is complete.ii. The graph has a loop, i.e., an edge connecting a vertex to itselfiii. The graph has an isolated vertex, i.e., a vertex with no edges incident to it.

b. Answer the same questions for the adjacency list representation.

26 | P a g e

Algorithms Chapter 1 Oct. 5 2011

Documents

greatest common

real life

algorithm

algorithm

data structure

data type

numerical

decreasing