Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Computer Science: An Overview Tenth Edition by J. Glenn Brookshear Chapter 12: Theory of Computation Presentation files modified by Farn Wang
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Computer Science: An Overview
Tenth Edition
by
J. Glenn Brookshear
Chapter 12:
Theory of Computation
Presentation files modified by Farn Wang
1-2
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Incomputability
The issue:
There are some questions with an answer
that we, human being, cannot compute.
• Also called unsolvability
• called undecidability for Boolean problems.
– with answer yes or no.
– also called decision problem
12-2
1-3
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-3
Chapter 12: Theory of Computation
• 12.1 Functions and Their Computation
• 12.2 Turing Machines
• 12.3 Universal Programming Languages
• 12.4 A incomputable Function
• 12.5 Complexity of Problems
• 12.6 Public-Key Cryptography
1-4
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
The computatbility issue
• functions: what we have in mind.
– can be specifications.
– that is what we want to compute.
• programs:
– can be algorithms.
• must always stop with correct answers.
• finitely many simple and executable statements.
– that is what we have at hands.
Can all functions be computed with algorithms ?
12-4
1-5
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Functions
Given an input, yields the correct output.
• max(a,b) : ranges of a, b ?
• sorting: input list to output list
• sum(x1x2…xn): input list to a number
• sumcheck(x1x2…xn,y): to a Boolean value.
• prime(b1b2…bn): Are the bits for a prime ?
Note that if the range of input is finite, the
function is always computable.
12-5
1-6
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
A simple argument for
incomputability
Some Boolean functions of finite bit
streams is incomputable.
Why ?
• There are more such functions than
programs (or algorithms).
• Programs (algorithms) are countably many.
– as many as integers, rationals, …
• Boolean functions are uncountably many.
– as many as reals. 12-6
1-7
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
A simple argument for
incomputability
Programs countably many ?
Why ?
• It is just a finite number of bits or bytes.
12-7
1-8
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
A simple argument for
incomputability
Boolean functions are uncountably many.
Why ?
• We prove by contradiction.
• The contradiction is established by the
famous argument of diagonalization.
12-8
1-9
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-9
Uncountability of Boolean functions
- Proof by diagonalization
input bit streams interpreted as integers.
input 0 1 2 3 4 5 6 7 …
f0 0 1 0 1 0 1 0 1 …
f1 0 0 1 0 0 1 1 0 …
f2 0 1 1 0 0 1 1 1 …
f3 1 1 0 1 0 1 1 0 …
… … …
2(7) is
true.
3(7) is
false.
Which Boolean function is missing ?
countable
sequence
of
functions.
1-10
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-10
Functions
- our research targets of computability
A correspondence between
• a collection of possible input values and
– e.g., N, Z, R, texts, tuples, trees, graphs, ….
• a collection of possible output values
– e.g, {0,1}, N, Z, R, ….
so that each possible input is assigned a
unique output value.
We assume that all functions are total.
• defined output for all input values.
1-11
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-11
Functions (continued)
• Computing a function: Determining the
output value associated with a given set of
input values
• Noncomputable function: A function
that cannot be computed by any algorithm
– Are there any ?
Note: an algorithm
• always halts with correct output values.
• consists of finite number of effective steps.
1-12
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-12
Function
- example
converting
measurements
• in yards
• into meters
1-13
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Functions
- comments on the restrictions
• Not all software systems are meant to halt.
– An OS is not supposed to halt its service.
• However, we can also use similar
techniques to show the following.
– Some system calls can be implemented with
algorithms.
– Some properties of OS cannot be checked
with algorithms.
12-13
1-14
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-14
Effective calculation
- Clarification
What is a reasonable computation step ?
• Oracle ?
• Turing machines
• recursive functions
• -calculus
• Algebrae
• Logics
1-15
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-15
Turing machine
- the components finite-state
One-way infinite infinite memory
1 1 1 0 * 0 1 1 *
1-16
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-16
Turing Machine
- Operation
• Inputs at each step
– State
– Value at current tape position
• Actions at each step
– Write a value at current tape position
– Move read/write head
– Change state
1-17
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-17
Turing machine
- an example for incrementing a
value on a tape
1-18
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-18
Church-Turing Thesis
The functions that are computable by a Turing
machine are exactly the functions that can be
computed by any algorithmic means.
• History remark: Turing used “algorithmic
means” for “function” nowadays.
1-19
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Church–Turing thesis
- Church’s version
Church's thesis, Church's conjecture
Every effectively calculable function (effectively
decidable predicate) is general recursive.
(Kleene 1952:300)
12-19
1-20
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Church–Turing thesis
- Turing’s version
Turing's thesis
Turing's thesis that every function which would
naturally be regarded as computable is
computable under his definition, i.e. by one of his
machines, is equivalent to Church's thesis by
Theorem XXX.
(Kleene 1952:376)
12-20
1-21
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Church–Turing thesis
The proof of equivalence of the following three
computational processes
• recursion,
• λ-calculus, and
• Turing machine
by Alonzo Church, Stephen Kleene and J.B.
Rosser (1934–6) and
by Alan Turing (1936–7).
12-21
1-22
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Church–Turing thesis
- The fundamental premise The notion of
• "effectively computable" or
• "effectively calculable"
is a vague intuitive one.
12-22
1-23
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Church–Turing thesis
Neither thesis can be proven.
(i) "heuristic [observational, experiential]
evidence",
(ii) the "equivalence of 'diverse formulations'"
(e.g. the three computational processes) and
(iii) on an observational analysis of a human
with a pencil and paper following a set of
rules (Turing's analysis) and of a "worker" in
boxes (Emil Post's analysis 1936).
12-23
1-24
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Turing machine
- as an effective computation model Based on the Church-Turing thesis,
we can use
• Turing machines, or
• any programming system that is equivalent
to Turing machine in computation power
as an effective computation model.
12-24
1-25
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-25
Universal Programming Language
A language
• with which a solution to any computable
function can be expressed
• that is equivalent in computation power to
Turing machines.
Examples:
• “Bare Bones” and
• most popular programming languages
1-26
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-26
The Bare Bones Language
• Bare Bones is a simple, yet universal
language.
– Actually, a counter machine
– a counter is an integer variable that can only
be decremented, incremented, and tested
against zero.
• 4 types of statements
– x=0;
– x++;
– x--;
– while (x!=0) {… };
1-27
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-27
The Halting Function
- Some assumptions on the encoding
• Given the encoded version of any program
M and input value v, return
– 1 if M(v) halts, or
– 0 if M(v) does not.
• We assume
– a program can be encoded as a (huge) integer
– the input parameter values can be encoded as
an integer.
1-28
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Turing machine
- Halting problem and halting function • Given
– a Bare Bone program (or a Turing machine) M and
– an input values v,
does M(v) eventually halt ?
• Conceptually, we can define a halting function,
halt(M#v)
that outputs
– 1 if M(v) halts and
– 0 otherwise.
12-28
an encoding of M
followed by v.
1-29
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-29
Diagonalization arguments
- Georg Cantor (1845-1918)
• Originally for the uncountability of reals.
• First, we need to encode all reals as binary numbers. – the first bit: the sign
– the remaining even bits for positions behind the point.
– the odd bits for positions before the point.
Example:
• 6.75 = 0 0 1 1 1 1 000000000000…
• -1.33 = 1100100010001…..
+ 1 0.5 2 0.25 4
1-30
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-30
Proof by diagonalization
- Georg Cantor (1845-1918)
Diagonalization
• Proof plan: Proof by contradiction.
• Assume reals in [0,1) are countable.
reals p1 p2 p3 p4 p5 p6 p7 p8 …
r0 1 1 0 1 0 1 0 1 …
r1 0 1 1 0 0 1 1 0 …
r2 1 1 0 1 1 1 0 1 …
r3 1 1 0 1 0 1 1 0 …
… … …
Which real is missing ?
Enumeration of
reals in binary
encoding.
1-31
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Continuum Hypothesis
- Georg Cantor (1845-1918)
Since R is larger than N, what is the size
(Cardinality) of R (|R|) ?
|R|=2|N|
12-31
reals p1 p2 p3 p4 p5 p6 p7 p8 … Integer set representations
r0 1 1 0 1 0 1 0 1 … {1,2,4,6,8,…}
r1 0 1 1 0 0 1 1 0 … {2,3,6,7,…}
r2 1 1 0 1 1 1 0 1 … {1,2,4,5,6,8,…}
r3 1 1 0 1 0 1 1 0 … {1,2,4,6,7,…}
… … … …
1-32
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Continuum Hypothesis
- Georg Cantor (1845-1918)
• There are different cardinalities of infinite
sets.
• Transfinite numbers
– 0, the number of integers
– i+1 =2i
• 1 is the number of reals
The hypothesis (The 1st Hilbert’s problem).
• 0 < 1 < … < k < k+1 < ……….
• There is no set S with 0 < |S| < 1.
12-32
1-33
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Continuum Hypothesis
- Georg Cantor (1845-1918)
There is no set S with 0 < |S| < 1
• The first Hilbert’s open problem.
• Cantor tried to prove it for many years in
vain.
• With the standard axioms of set theory,
– Gödel proved that it cannot be disproved
(1940).
– Cohen proved that it cannot be proved (1963).
• Still an open problem! 12-33
1-34
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Intuitive argument for
incomputability.
Why some functions are incomputable as
programs ?
• the number of programs is 0
– program files are bits in the storage.
• The number of functions is 1
– f(1)=1, f(2)=0, f(3)=1, f(4)=1, ……
• So it is not possible to define a one-to-one
relation between programs and functions.
12-34
1-35
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Some incomputable problems (1/2)
• The validity of 1st-order logic formula
(Hilbert’s 2nd problem)
xyz(single(x) (parent(y,x)worried(y))
• Mortal matrix problem
Given 15 33 matrices, M1, …, M15, are
there Mi1 … Mim = 0 ?
• CFL ambiguity problem
12-35
1-36
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Some incomputable problems (2/2)
• Multivariable polynomial equations
(Hilbert’s 10th problem)
Incomputable for degree 4.
Incomputable for 36 real variables.
Incomputable for 11 integer variables.
12-36
1-37
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-37
Complexity of Problems
• Time Complexity: The number of instruction executions required
– Unless otherwise noted, “complexity” means “time complexity.”
• Complexity upper-bound:
– A problem is in class O(f(n)) if it can be solved by an algorithm in Q(f(n)).
• Complexity lower-bound:
– A problem is in class Q(f(n)) if the best algorithm to solve it is in class Q(f(n)).
1-38
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-38
A procedure MergeLists for merging
two lists
1-39
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
MergeList(h, len)
Given A[1..n],
• assume A[h,h+len-1] and
A[h+len, h+2len-1] are sorted
• sort A[h,h+len-1] and A[h+len, h+2len-1]
into A[h,h+2len-1].
12-39
1-40
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
MergeList(h, len)
let a = h; b = h+len-1;
for (; a< h+len && b < h+2len; ) {
if (A[a] < A[b]) { output A[a]; a++; }
else { output A[b]; b++; }
}
if (a < h+len) for (; a<h+len; a++) output A[a];
else for (; b<h+2len;b++) output A[b];
12-40
1-41
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
MergeList
- in the running
12-41
4 9 2 3
7 8 1 5
8 9 5 7 3 4 1 2
1-42
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-42
The merge sort algorithm
implemented as a procedure
MergeSort
merg
1-43
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
MergeSort(A, n)
for (i=1; i <= log(n); i++) {
k = 2^i;
for (h = 1; h <= n; h=h+2k)
A[h,h+2k-1] = MergeList(h,k);
}
complexity O(nlog n)
12-43
1-44
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
MergeSort
12-44
9
4
2
3
1
5
7
8
4 9
5 7
1 8
2 3
4 9 2 3
7 8 1 5
8 9 5 7 3 4 1 2
1-45
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-45
The hierarchy of problems generated
by the merge sort algorithm
1-46
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-46
Graphs of the mathematical
expressions n, lg n, n lg n, and n2
1-47
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Complexities
Given input size of n bits
• of algorithms
– how many steps O(f(n)) needed to compute
f(k) with log k = n.
• of problems
– lower bound f(n): for any input of n bits, you
can not design an algorithm that runs in lower
than O(f(n)).
– upper-bound f(n): you know that there is an
algorithm that runs in O(f(n)).
12-47
1-48
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-48
A graphic summation of the problem
classification
PSPACE
EXPTIME
EXSPACE
Double EXPTIME
non-elementary
1-49
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
common complexity heirarchy
• O(log(n)): searching
• O(n), O(p(n)): p(n) is a polynomial of n.
– also called P, or PTIME, deterministic
• NP, nondeterministically polynomial time
• PSPACE, space(p(n))
• EXPTIME, time O(2^n)
• NEXPTIME
• EXPSPACE
• DEXPSPACE, time O(2^{2^n})
• …
• non-elementary complexity O(2^{2^{2^{…..}}})
• incomputable
12-49
1-50
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
A non-deterministic finite-state
machine
12-50
buck
ginger ale
coffee
buck
one event, many
choices
1-51
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
A non-deterministic Turing machine
12-51
1dollar
ginger ale
coffee
1dollar
1-52
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-52
Nondeterministic algorithms
• An “algorithm” whose steps may not be
uniquely and completely determined by the
process state
• Intuitively, you can guess in
nondeterministic algorithm.
• The problem is considered solved if one of
the guess solves the problem.
1-53
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
NP-complete problems
Given input of size n, the solution of the
problem can be guessed in f(n) steps (by a
Turing machine) where f() is a polynomial
function.
12-53
In f(n) steps to halt.
1-54
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
NP-complete problems
• The easiest hard problems
• including many planning problems.
– Tour planning problem with a constraint.
– Bin packing problem.
– ……
12-54
1-55
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Boolean formula satisfiability problem.
- The first NP-complete problem.
Satisfiability problem of Boolean formula.
f(x,y,z) = (x y z) (x y z)
Is there valuation that satisfies f(x,y,z) ?
Why it is NP-complete ?
1. By guessing the values of x,y,z, we may
solve the problem in |f()| steps.
2. By reduction from Turing machine NP
problems.
12-55
1-56
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
SAT problem - Reduction from Turing machine NP problems.
Construct a formula of f(n) * f(n) * C * p
variables that encodes the f(n) tape
configurations of the computation of |f(n)|
steps.
12-56
In f(n) steps to halt.
states of control head positions
1-57
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-57
Nondeterministic algorithms
- Example
Traveling saleman: Given a map of cities,
find a tour that visit all cities exactly once
with shortest distance.
• polynomial time with nondeterministic
algorithms
• not known to be in PTIME
• an NP-complete problem!
1-58
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-58
Nondeterministic algorithms
- Example Bin Packing: Given
• a backpack of capacity B and
• n items with sizes s1, …, sn,
What is the minimum number of times that
we need to carry the backpack in order to
convey the n items ?
• polynomial time with nondeterministic
algorithms
• not known to be in PTIME
• an NP-complete problem.
1-59
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-59
P versus NP • Class P: All problems in any class Q(f(n)), where
f(n) is a polynomial
• Class NP: All problems that can be solved by a
nondeterministic algorithm in polynomial time
Nondeterministic algorithm
• an “algorithm” whose steps may not be uniquely
and completely determined by the process state
• Intuitively, you can guess in nondeterministic
algorithm.
• The problem is considered solved if one of the
guess solves the problem.
• Whether the class NP is bigger than class P is
currently unknown.
1-60
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Reduction
- Establishing complexity lower-bound
To show a problem A is at least of a certain
complexity K,
• we may use another problem B of
complexity K,
• propose a construction procedure () such
that for every problem instance b of B,
– (b) is a problem instance of A, and
– (b) can be constructed in lower than K
compleixty.
12-60
1-61
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Example of a reduction
From a Boolean satisfiability problem
to a vertex cover problem.
• A vertex cover of a graph is a set of vertices
such that each edge of the graph is incident to at
least one vertex of the set.
12-61
1-62
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Example of a reduction
From a Boolean satisfiability problem
to a vertex cover problem.
• The problem of finding a minimum vertex cover
is a classical optimization problem.
• A typical example of an NP-hard
12-62
1-63
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley
Example of a reduction From a Boolean satisfiability problem
to a vertex cover problem.
Blue vertices form a vertex cover which corresponds to truth values.
12-63
A
A B
Theorem [Karp’72]:
There is a truth
assignment
if and only if there the
minimum vertex cover
is L+2C.
• L: # of variables
• C: # of clauses
1-64
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-64
Application of NP problems
- Public-Key Cryptography
• Key: A value used to encrypt or decrypt a
message
– Public key: Used to encrypt messages
– Private key: Used to decrypt messages
• RSA: A popular public key cryptographic
algorithm
– Relies on the (presumed) intractability of the
problem of factoring large numbers
1-65
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-65
Encrypting the Message 10111
• Encrypting keys: n = 91 and e = 5
• 10111two = 23ten
• 23e = 235 = 6,436,343
• 6,436,343 ÷ 91 has a remainder of 4
• 4ten = 100two
• Therefore, encrypted version of 10111 is
100.
1-66
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-66
Decrypting the Message 100
• Decrypting keys: d = 29, n = 91
• 100two = 4ten
• 4d = 429 = 288,230,376,151,711,744
• 288,230,376,151,711,744 ÷ 91 has a remainder of 23
• 23ten = 10111two
• Therefore, decrypted version of 100 is 10111.
1-67
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-67
Public key cryptography
1-68
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley 12-68
Establishing an RSA public key
encryption system