Algorithms and their Applications CS2004 (2012-2013) Dr Stephen Swift 4.1 Time Complexity and Asymptotic Notation.

Algorithms and their Applications

CS2004 (2012-2013)

Dr Stephen Swift

4.1 Time Complexity and Asymptotic Notation

The Laboratories – Part 1The list of laboratories and what grade they cover is on BlackboardNote we are using threshold based gradingWith the Mathematics Test:

You only need to take one test (both parts) to get an E gradeYou get up to five attempts however…The purpose of the test is for you to assess your own Mathematics abilitySign up to ASK workshops if there is anything that you need to refresh yourself on

Time Complexity and Asymptotic Notation Slide 2

The Laboratories – Part 2Most of the worksheets go towards a specific gradeThe worksheets must be completed in orderWhen you feel you have completed a worksheet, ask a GTA to assess itKeep a log book of any feedback and comments

If the worksheet is deemed to have been completed successfully the GTA will “sign off” the worksheetWe will keep a note and you must get the GTA to date and sign a print out of the worksheet coversheet


Previously On CS2004…

So far we have looked at:Concepts of Computation and AlgorithmsComparing algorithmsSome mathematical foundation


Time Complexity and Asymptotic Notation

Within this lecture we will discuss:

The Selection sort algorithmThe Binary search algorithmBig-Oh notationBig-Omega and Big-Theta notationsThis builds upon the core topic of counting Primitive Operations we covered two weeks ago...


Why Sorting?Sorting is one of the most common tasks in data analysisExamples:

Print out a collection of employees sorted by salaryPrint out a list of names in alphabetical order

There are many sorting algorithmsThe Selection Sort algorithm repeatedly finds the smallest element in the unsorted tail region of the array and moves it to the frontQuestion: How do you sort 11,9,17,5,12?Time Complexity and Asymptotic Notation Slide 6

Selection Sort algorithm – Part 1Sorting an Array of IntegersArray in original order

Find the smallest and swap it with the first element

11 9 17 5 12

5 9 17 11 12


Find the next smallest It is already in the correct place

Find the next smallest and swap it with the first element of unsorted portion

Selection Sort algorithm – Part 2

5 9 17 11 12

5 9 11 17 12


Repeat

When the unsorted portion is of length 1, we are done

Selection Sort algorithm – Part 3

5 9 11 12 17

5 9 11 12 17


How Fast is the Algorithm?In an array of size n, count how many primitive operations are neededTo find the smallest, visit n elements + 2 visits for the swap To find the next smallest, visit (n-1) elements + 2 visits for the swap The last term is 2 elements visited to find the smallest + 2 visits for the swap

The number of visits:

(n + 2) + [(n-1) + 2] +[(n-2) +2] + . .. +(1+ 2) + 2 This can be simplified to

n2/2 + 5n/2 + 2 5n/2 +2 is small compared to n2/2 - so let's ignore it Also ignore the 1/2 - use the simplest expression of the classThe number of visits is of the order n2


The Big-Oh notationThe number of visits is of the order n2 Using Big-Oh notation:

The number of visits is O(n2) If you multiply the number of elements in an array by 2, the processing time will be multiplied by 4!So what is the formal definition of Big-Oh notation?


Big-Oh Notation – Part 1Given functions f(n) and g(n), we say that f(n) is O(g(n)) if there are positive constants c and n0 such that:

f(n) cg(n) for n n0

1

10

100

1,000

10,000

1 10 100 1,000n

3n

2n+10

n


Example: Is 2n + 10 O(n)? Can we find c and n0? 2n + 10 cn(c-2)n 10n 10/(c-2)

Pick c = 3 and n0 = 10

f(n) is O(g(n)) iff f(n) cg(n) for n n0


Big-Oh Notation – Part 2Example: the function n2 is not O(n)? Why?

n2 cnn c

The above inequality cannot be satisfied since c must be a constant

1

10

100

1,000

10,000

100,000

1,000,000

1 10 100 1,000n

n 2̂

100n

10n

n


Big-Oh and Growth RateThe Big-Oh notation gives an upper bound on the growth rate of a functionThe statement “f(n) is O(g(n))” means that the growth rate of f(n) is no more than the growth rate of g(n)f(n) grows no faster than g(n)We can use the Big-Oh notation to rank functions according to their growth rateTime Complexity and Asymptotic Notation Slide 15

Example: Binary SearchTask: search for a key in a sorted listFirst check the middle list elementIf the key matches the middle element, we are doneIf the key is less than the middle element, the key must be in the first halfIf the key is larger than the middle element, the key must be in the second halfExamples?

Spell checkers, phone books, internet browser cache…



Binary Search Example

1

2

3

4

5

6

7

1

2

3

4

5

6

7

3

1

2

3

4

5

6

7

3

3

Binary search is an O( log2(n) ) algorithm

The Binary search algorithm is much faster that the linear search algorithm of order O(n)

But it only works on ordered data....

QuestionsT(n) will be used to denote the time an algorithm takes to execute

Is T(n) = 9n4 + 876n O(n4)?

Is T(n) = 9n4 + 876n O(n3)?

Is T(n) = 9n4 + 876n O(n27)?

T(n) = n2 + 100n O(?)

T(n) = 3n + 32n3 + 767249999.3n2 O(?)


Big-Oh RulesIf f(n) is a polynomial of degree d, then f(n) is O(nd), i.e.,

Drop lower-order termsDrop constant factors

Use the smallest possible class of functions

Say “2n is O(n)” instead of “2n is O(n2)”Use the simplest expression of the class

Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)”


Asymptotic Algorithm AnalysisThe asymptotic analysis of an algorithm determines the running time in Big-Oh notation

The word asymptotic refers to long term (large input) algorithmic trendsTo perform the asymptotic analysis

We find the worst-case number of primitive operations executed as a function of the input size, T(n)We express this function with Big-Oh notation

Example:We have already known that the algorithm ArrayMax executes at most T(n) = 8n 5 primitive operationsWe say that algorithm ArrayMax “runs in O(n) time”

Since constant factors and lower-order terms are eventually dropped, we can disregard them when counting primitive operations

This is why we do not have to be 100% accurate as long as we get the powers of n correctI.e. we do not miscount n3 as n2 etc…Confusing 7n for 5n will not matter……


Rank From Fast to Slow…

Before…T(n) = n4

T(n) = n log n

T(n) = n2

T(n) = n2 log n

T(n) = n

T(n) = 2n

T(n) = log n

T(n) = n + 2n


After…1. T(n) =log n O(log n)

2. T(n) = n O(n)

2. T(n) = n + 2n O(n)

4. T(n) = n log n O(n log n)

5. T(n) = n2 O(n2)

6. T(n) = n2 log n O(n2 log n)

7. T(n) = n4 O(n4)

8. T(n) = 2n O(2n)

Relatives of Big-Oh

W(g): functions that grow at least as fast as g

Q(g): functions that grow at the same rate as gO(g): functions that grow no faster than g

Big-Omega

Big-Theta

Big-Oh


Asymptotic Notation

W(g): functions that grow at least as fast as g

Q(g): functions that grow at the same rate as g

O(g): functions that grow no faster than g

y=g(x)


Big-Oh is the interesting oneBecause we are interested in efficiency, W(g) will not be of much interest to us because W(n2) includes all functions that grow faster than n2, for example, n3 and 2n

For a similar reason, we are not much interested in Q(g), the class of functions that grow at the same rate as the function g

Big-Oh is the class of functions that will be of the greatest interest to us

Considering two algorithms, we will want to know if the first is in Big-Oh of the second

If yes, we know that the second algorithm does not do better than the first in solving the problem


Slide 25

But Can We Do Better? There are algorithms that have a

computational complexity of O(2n) (this is very poor: 2501015)

But ......

How do we know there isn’t a better algorithm which only takes polynomial time anyway?


Slide 26

Some Classes of Problems

P

NP

Problems which canbe solved in polynomial

time using a “magic box”(Non-deterministic)

Problems which can be solved in polynomial time


Slide 27

Problems in NP but not in P

NPP

NPP?=

is a subset of

but

Most problems forwhich the best knownalgorithm is believed to be O(2n) are in NP

P

NP


28

The Hardest NP Problems

If a polynomial time algorithm is

found for any problem in NP-Complete then every problem

in NP can be solved in

polynomial time

i,e, P = NP

PNP

NP-Complete


The “hardest”problems in NP

Slide 29

Examples of NP-Complete Problems


The travelling salesman problemFinding the shortest common superstring

Checking whether two finite automata accept the same language

Given three positive integers a, b, and c, do there exist positive integers (x and y) such that ax2 + by2 = c

Slide 30

How to be Very Famous!

Find a polynomial time algorithm for this problem:

Given n objects of different weights,

split them into two equally heavy piles

(or say if this isn’t possible)


Slide 31

Consequences of P=NP


If this was found to be true, the news would make world headlines!

All passwords could be cracked in polynomial time (very, very fast)

The End of the World??!!!! You would not be able to secure any computer

or device on the internet, wireless or mobile networks.....

Algorithms that currently take years might take minutes!

Luckily in the last year a researcher (Vinay Deolalikar at HP Labs, Palo Altois) has claimed that they have proved PNP...... (phew!)

This Weeks Laboratory


This laboratory contributes towards the D gradeYou will be implementing and comparing two algorithms

Computing T(n) Computing O(n) Running some experiments

Next Lecture


We will look at data structures and their applications

Algorithms and their Applications CS2004 (2012-2013) Dr Stephen Swift 4.1 Time Complexity and Asymptotic Notation.

Documents

asymptotic notationslide

order n2 time complexity

asymptotic notationwithin

smallest element

n elements

array of size n

classthe number of visits

list of laboratories