Jeff Edmonds York University COSC 4111 Lecture 1 Church's ThesisChurch's ThesisChurch's ThesisChurch's Thesis Primitive RecursivePrimitive RecursivePrimitive.

Jeff EdmondsYork University

COSC 4111Lecture 1

• Church's Thesis• Primitive Recursive• TM to Primitive Recursive?• Primitive Recursive << TM• TM to Recursive• Register Machine• And/Or/Not Circuits• Computer Circuits• Circuits Depth• Arithmetic Circuits

Models of Computability• Neural Nets

• Poly-Time• Non-Deterministic Machine• Quantum Machine• Context Free Grammars• JAVA to Machine Code

(Compiling/Parsing)• Context Sensitive Grammars• TM to Grammar• Decide vs Accept• Humans• Complexity Classes

Please feel free to ask questions!

Please give me feedbackso that I can better serve

you.

Thanks for the feedback that you have given me already.

Church’s ThesisA computational problem is computable• by a Java Program

• by a Turing Machine

• by a (simple) Recursive Program

• by a (simple) Register Machine

• by a Circuit: And/Or/Not, Computer, Arithmetic, & Neural Nets.

• by a Non-Deterministic Machine

• by a Quantum Machine

• by a Context Sensitive Grammar

NonUniform unreasonable Uniform reasonableComputes the same,

but faster.

Decide vs Accept

Computes the same,but faster.

Church’s Thesis

A Java program can easily simulate each of them.

A computational problem is computable• by a Java Program








Proof

Church’s Thesis Proof

A Java program can be simulated by

a TM. A TM can be simulated by

a Recursive Program. A Recursive Program can be simulated by a

Register Machine. A TMcan be simulated by a

Circuit.

A TMcan be simulated by a

Context Sensitive Grammar.

Non-Deterministic Machine

can be simulated by a Deterministic one.









A Quantum Machine can be simulated by a

Deterministic one.










Note you can get anywhere from

anywhere.



• by a (simple) Recursive Program Define

X = 33Y = 12ac = 3bd = 6

(a+b)(c+d) = 18XY = 396

RecursionFriends & Strong Induction

• Consider your input instance• Allocate work

• Construct one or more sub-instances• It must be smaller

• and meet the precondition• Assume by magic your friends give you the

answer for these.• Don’t micro-manage them by tracing

out what they and their friends friends do.

• Use this help to solve your own instance.• Do not worry about anything else.

• Who your boss is…• How your friends solve their instance…X = 3

Y = 1XY = 3

X = 3Y = 2

XY = 6

X = 6Y = 3

XY = 18

MULT(X,Y):

If |X| = |Y| = 1 then RETURN XY

Break X into a,b and Y into c,d

e = MULT(a,c) and f =MULT(b,d)

RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f

We will now define two models of computation:• Primitive Recursion Functions and • μ-Recursive Functions• They were developed about the same time as

Turing Machines• Like TMs, the allowed set of allowed operations

is very restricted.• It was a big historical deal when it was prove that

The Class of μ-Recursive Problems = The Class of Turing Computable Problems

Primitive Recursive

Primitive Recursivef is defined from g & h by primitive recursion if• f(x,y) = h(x,y-1,f(x,y-1))• f(x,0) = g(x)

Does not change. Could be x1,x2,..,xn.

Smaller instanceHelp from friend

(very specific)

Base caseMy solution


f(x,y) could be computed iteratively:yˈ = 0f = g(x)loop Loop Invariant: exit when yˈ=y

yˈ = yˈ+1fˈ = h(x,yˈ-1,f’)end loopreturn(fˈ)

fˈ= f(x,yˈ)


f is defined from g & h by composition if• f(x,y) = g(h1(x,y),…,hn(x,y))

Initial functions:• Zero(x) = 0• Successor(x) = x+1• In,i(x1,x2,..,xn ) = xi

Ex• If you have defined g(y,x)

and want f(x,y) = g(y,x)• f(x,y) = g(I2,2(x,y),I2,1(x,y))



No really dude.If it is not on this page,

you can’t do it.

And you want me to be able to compute everything?

Initial functions:• Zero(x) = 0• Successor(x) = x+1• In,i(x1,x2,..,xn ) = xi



There is a beauty to the simplicity of these programs.Each routine f() is defined

by specifyingInitial functions:• Zero(x) = 0• Successor(x) = x+1• In,i(x1,x2,..,xn ) = xi

• not lots of fancy code, but only• the number of arguments

f(x1,x2,..,xn,y) • and which functions

g & h to call.


Examples:

x+y = [x+(y-1)] Get help from your friend.My solution from his.

+1x+0 = x

This is not formal enough.

Ok.


Examples:

Sum(x,y) = h( x,y-1,Sum(x,y-1) )

Sum(x,0) = g(x)• We define a new primitive recursive function Sum• from a previously define h & g.• h has arguments:

• All the other variables x left unchanged.• y one smaller.• Recursive solution on input with y one smaller.


Examples:

Sum(x,y) = h( x,y-1,Sum(x,y-1) )

Sum(x,0) = g(x)• We define a new primitive recursive function Sum• from a previously define ones h & g.• When y is zero, Sum is defined from

• g on all the other variables x left unchanged.


Examples:

Sum(x,y) = h( x,y-1,Sum(x,y-1) ) = Successor( I3,3( x,y-1,Sum(x,y-1)) ) = Sum(x,y-1) + 1

• We want to use the recursive solution• But it must be extracted from the tuple• using the I function.• Then we add one to this recursive solution.


Examples:

Sum(x,y) = h( x,y-1,Sum(x,y-1) ) = Successor( I3,3( x,y-1,Sum(x,y-1)) ) = Sum(x,y-1) + 1Sum(x,0) = g(x) = I1,1(x)


Examples:

x×y = [x×(y-1)] Get help from your friend.My solution from his.

+xx×0 = 0


Examples:

Prod(x,y) = h( x,y-1,Prod(x,y-1) ) = Sum( I3,3( x,y-1,Prod(x,y-1)), I3,1( x,y-1,Prod(x,y-1)) ) = Sum( Prod(x,y-1), x )

• Now we define Prod from previously defined Sum.• Sum has two arguments

• Prod(x,y-1) • x

Extracted from tupleExtracted from tuple


Examples:

xy = [xy-1] Get help from your friend.My solution from his.

×xx0 = 1


Examples:

log(y/2) =

log(16) = 24 = 164 because

log(20) = first power of 2 below it is 16.4 because

20/2=10, 10/2=5, 5/2=2, 2/2=1

log(20)=4

log(y) = # of time to divide by 2 until you get 1

one less


Examples:

log(y) = log(y/2) Get help from your friend.My solution from his.

+1

This is true. But is it formally Primitive

Recursive?

Sure!We showed y/2 is prim. rec.So we use it to show log is.

= log( div(y,2) ) +1


Examples:

log(y) = log(y/2) Get help from your friend.My solution from his.

+1

Oops

= log( div(y,2) ) +1

You must usef(y) = h(y-1,f(y-1))You must give your

friend <y-1>.


Is this legal?f(x,y) = h(x/2,y+5,f(x,y-1))

Not officially.But you can legally defineh’(x,y,c) = h(x/2,y+5,c)

and then definef(x,y) = h’(x,y,f(x,y-1))


Examples:

Predecessor(y) = Predecessor(y-1) Get help from your friend.

My solution from his.+1

Now a hard one.How do you make something smaller?

This looks fine.


Examples:

Predecessor(y) Get help from your friend.My solution from his.

+1

This does not make anything smaller, but defers to its friend to makes

something else smaller.What is the first thing to get smaller?

= Predecessor(y-1)


Examples:


+1

But what is the base case?

Predecessor(0) = -1

Bug: Negative values are not defined!

= Predecessor(y-1)


Examples:


+1

Ok, as a special case, we will set it to zero.

Predecessor(0) = -10

Predecessor(2) = 1,Predecessor(1) = 0, Predecessor(0) =

= Predecessor(y-1)

0.

But what is the base case?


Examples:


+1

What about y=1?

We need more cases.

Predecessor(1) = Predecessor(0)+1 = 0 +1 0

= Predecessor(y-1)

Predecessor(0) = -10


Examples:

Predecessor(y)

Oh dear! I don’t know.It seems impossible.

Predecessor(0) = 0

Pred+1 if y > 1= h(y-1,Pred(y-1)) =

0 if y = 1

Bug: “if “ and “equality” are not defined.


Examples:

h(x,y-1,f(x,y-1))

That is what we want.

h(x,y-1,f) = I3,2(x,y-1,f)

Lets start back at the formal definition

Predecessor(y) =

Predecessor(0) = g(x) = 0

Yay! That does it.


Examples:

h(x,y-1,f(x,y-1))

That is what we want.

h(x,y-1,f) = I3,2(x,y-1,f)

Lets start back at the formal definition

Predecessor(y) =

Predecessor(0) = g(x) = 0Actually, to avoid negative values,the actual definition is• f(x, Successor(y)) = h(x,y,f(x,y))I changed it because I thought it looked better.


Examples:

x-y = [x-(y-1)] Get help from your friend.My solution from his.

-1

x-0 = x

3-5 = Pred(3-4) = Pred(Pred(Pred(Pred(Pred(3-0)))))= Pred(Pred(Pred(Pred(Pred(3)))))= Pred(Pred(0))= 0 Fair enough.


max(x,y) = (x-y)+y

Examples:

(8-5)+5 = (3)+5 = 8

(5-8)+8 = (0)+8 = 8

min(x,y) = x+y-max(x,y)


Examples:

f(x,0) = g(x) = 0 f(x,y) = h(x,y-1,f(x,y-1)) = 1

Truth(y) = 0 if y=01 else

Here true≥1 and false=0.


Examples:

Truth(y) = 0 if y=01 else

Here true≥1 and false=0.

Neg(R) = 1-Truth(R)

(R and S) = min(R,S)

(R or S) = max(R,S)

(x>y) = x-y(x=y) = Neg(x>y) and Neg(y>x)


Examples:

Select(x) = 5 if x = 03 if x = 17 if x = 2 …4 if x = r

= 5×(x=0) + 3×(x=1) + 7×(x=2) + … + 4×(x=r)

3 = Successor(Successor(Successor(Zero)))

mod(y,x) = modx(y) = Remainder(y/x)eg mod(12,5) = 2

This is a hugely important in math & computer science

In field mod 5, the universe has 5 “numbers” {0,1,2,3,4}Think of them as equivalence classes … -8 =mod 5 -3 =mod 5 2 =mod 5 7 …Two operations + and ×

3+4 = 7 =mod 5 2 3×4 = 12 =mod 5 2Additive inverse

-3 = 2 because -3+2 =mod 5 0Multiplicative Inverse

½ = 3 because 2×3=mod 5 1

Primitive Recursive

mod(y,x) = modx(y) = Remainder(y/x)eg mod(8,3) = 2

Counting Mod 3

1 2 0 1 2 0 1 2

SuccModx(r) = 1 if r = 02 if r = 23 if r = 3 …x-1 if r= x-20 if r= x-1

= (r+1)×(r=x)

Primitive Recursive


Examples:

modx(y) =

Get help from your friend.

My solution from his.modx(y-1)SuccModx( )

SuccModx(r) = 1 if r = 02 if r = 23 if r = 3 …x-1 if r= x-20 if r= x-1

modx(0) = 0


Examples:

y/x = [(y-1)/x] Get help from your friend.My solution from his.

13/5 = 2 12/5 = 210/5 = 2 9/5 = 1

[(y-1)/x] +1

When does each occur?


Examples:

y/x = [(y-1)/x]

13/5 = 2 12/5 = 210/5 = 2 9/5 = 1

[(y-1)/x] +1

When does each occur?

0/x = 0

elseif modx(y)=0

mod(y,x) = modx(y) = Remainder(y/x)eg mod(8,3) = 2 8/3 = 2

Counting Mod 3

# people/3 = The number of time go

back to zero.

1 2 0 1 2 0 1 2

1 2

Primitive Recursive

How do I check if 11 is prime?

Make sure r = 2,3,4,…,10 don’t divide into it,

Primitive Recursive

2 3 4 5 6 7 8 9 10

I can do each withmod(11,r) ≠ 0

Lets call such an r good.

How do I know all these n guys are good?

The first n-1 are good.

Primitive Recursive

Get help from your friend.

My solution from his.

2 3 4 5 6 7 8 9 10

I’m good

Ok they must all be good.


Examples:

AllGood(x,y) = r ≤ y Good(x,r)

= AllGood(x,y-1)

Get help from your friend.My solution from his.

and Good(x,y)

AllGood(x,0) = Good(x,0)

Prime(x) = r ≤ x-1 mod(x,r) ≠0 = AllGood(x,x-1) where Good(x,r) = [mod(x,r) ≠0 or r{0,1}]

or r{0,1}


Examples:

ExistsGood(x,y) = r ≤ y Good(x,r)

= ExitsGood(x,y-1)


or Good(x,y)

ExistsGood(x,0) = Good(x,0)

PowerOf2(x) = yes for 1,2,4,8,16,32,… = r ≤ x 2r=x = ExistsGood(x,x) where Good(x,r) = [2r=x]


Examples:

NumGood(x,y) = |{ r≤ y | Good(x,r) }|

= NumGood(x,y-1)


+ Good(x,y)

NumGood(x,0) = Good(x,0)

log(16) = 24 = 164 because

log(20) = first power of 2 below it is 16.4 because

How many powers of 2 are below? |{1,2,4,8,16}=5one more than log(20)

=log(8)+1or becauserecursively

log(2) =1 log(1) =0


Examples:

NumGood(x,y) = |{ r≤ y | Good(x,r) }|

= NumGood(x,y-1)


+ Good(x,y)

NumGood(x,0) = Good(x,0)

log(x) = = NumGood(x,x) where Good(x,r) = PowerOf2(r) and r≠1

|{r ≤ x | PowerOf2(r) and r≠1 }|

Home workto do starting now

4.


A computational problem is computable



Primitive Recursive Programs seem to be able to compute a lot.

But can they compute everything a TM can?

The input to a TM M is a binary string Istring.The input to a PR P is an integer Iinteger.Istring is the binary representation of Iinteger.





To prove this, we must do a simulation of a TM computation

by a primitive recursive program.2)

TM to Primitive Recursive

• We want to encode this as the configuration of• A recursive program at time t,• i.e. as a tuple <q,tape,head> of whole numbers.

• Consider some computational problem P(I)that is computable by some TM M.• We must construct a recursive program computing P(I).• Consider some input I.• Consider some time step t.• Consider the configuration of M on I at time t.

101 0 0 1 0q

• Let tape =


10100102

be the binary number with the bits on the tape.

Grows this way.

Grows this way.

• Flip around the tape.

Consider some configuration of a TM

101 0 0 1 0q


• Let tape =

• Let head =

• <q,tape,head> specifies the current configuration of the TM.

1 0 10010q

1 0 00000

= tape

= head


01001012

be the binary number with the bits on the tape. 00001002

be the binary number with a one only where the head is.

Consider some configuration <q,tape,head> of a TM

• Let NextQ(q,tape,head) = q’ NextTape(q,tape,head) = tape’ NextHead(q,tape,head) = head’ giving that <q’,tape’,head’> is the next configuration that the TM will be in.

• We need to show that these functions are primitive recursive.

1 0 10010q

1 0 00000

= tape

= head



• rightTape = Contents 01001 of tape starting at head =

• Char c under head =

1 0 10010q

1 0 00000

tape/head

= tape

= head

Remainder( rightTape/2 )

c



• TM specifies the transition function δ(q,c) = <q’,c’,direction>

• Note that q and c both are from a finite range.• Hence δ can be computed using a primitive recursive select.

1 0 10010q

1 0 00000

= tape

= head

c

Select(x) = 5 if x = 03 if x = 17 if x = 2 …4 if x = r

= 5×(x=0) + 3×(x=1) + 7×(x=2) + … + 4×(x=r)



• TM specifies the transition function δ(q,c) = <q’,c’,direction>

• Note that q and c both are from a finite range.• Hence δ can be computed using a primitive recursive select.

1 0 10010q

1 0 00000

= tape

= head

c

• Hence NextQ(q,tape,head) = q’ NextC(q,tape,head) = c’ NextDirection(q,tape,head) = direction are primitive recursive.



1 0 10010q

1 0 00000

= tape

= head


• NextTape(q,tape,head) =

tape + headtape - headtape

if c=0 & c’=1if c=1 & c’=0if c=c’

• NextHead(q,tape,head) =head×2

head/2if direction = rightif direction = left

are also primitive recursive


• Let Config(x,y) = <q,tape,head> be the configuration that the TM will be in on input x after y time steps.

1 0 10010q

1 0 00000

= tape

= head


Config(x,y) = Get help from your friend.My solution from his.

Config(x,y-1)Next( )

Config(x,0) = <qstart,x,1> is the configuration at time zero.

Note a TM is designed to stay in the same configuration once it reaches a halting state qhalt.


• Let Output(x,t) be the output of the TM on input x if it halts within t time steps.

1 0 10010q

1 0 00000

= tape

= head


Note a TM is designed to have the output on the tape when it halts.

Output(x,t) = Tape(Config(x,t)) if TM has halted by time t“Has not halted” else

• We are done because everything is primitive recursive!Careful!


1 0 10010q

1 0 00000

= tape

= head


• We need this to be primitive recursive!Is it?

• Let Output(x) = the output of the TM if it halts.∞ else


1 0 10010q

1 0 00000

= tape

= head


• Suppose the TM is known to halt in time at most 2n

t ≤ 2x = 2 where n = log2 x = size(x)

• Then Output(x) = Output(x,2x)which is primitive recursive.




• by a (simple) Recursive Program Primitive

in double exponential time

• Comparing these models:• Is there a danger a primitive recursive program

will run forever on an input?• Is there a danger a TM

will run forever on an input?• Is it possible that they

have the same computing power?

No!

Definitely

No!





• Comparing these models:• Having a TM run forever is a bad thing.

But being able to compute as long as it needs to is a good thing.

• There are computational problems computable

by TMs but not by primitive recursive programs.





• Let us bound the running time of a given primitive recursive program.

For this we need Ackermann’s function.

• Comparing these models:

Ackermann’s Function

) )(nA( A(n) A

)( A) (A

kk-k

kk

1

:Step Inductive

00

:Case Base

1

) )))))(A(A(A(A(A( A(n) A

k

kk-k-k-k-k-k 0

that,on induction by Proof

11111

n applications

.each for function different a i.e.

Define

kA

A(k,n)(n) A

k

k

How big is A(5,5)?

n (n) A 2 0


) )))))(A(A(A(A(A( A(n) A

k

kk-k-k-k-k-k 0


11111

n applicationsn (n) A 2 0

(n) A 1 )(T 02222 1

n applications

n 2


) )))))(A(A(A(A(A( A(n) A

k

kk-k-k-k-k-k 0


11111


n (n) A 2 1

(n) A 2 )(A 02222 1

n applications

n2


n (n) A 2 2

(n) A 3

(n) A 4

) )))))(A(A(A(A(A( A(n) A

k

kk-k-k-k-k-k 0


11111


n (n) A 2 1


A3(n)

A4(0)

A4(1)


A4(1)

A3(n)

A4(1)





y (y) A 2 0

y (y) A 2 1

R0(y) = succ(succ(y))

= 2+2×(n-1)

For which k can Ackermann’s Ak(y) be computed?

R1(y) = R1(y-1)R0( )



y (y) A 2 0

y (y) A 2 1

y (y) A 2 2


= 2+2×(n-1)


= 2×2n-1 R1(y) = R1(y-1)R0( )


R2(y) = R2(y-1)R1( )


y (y) A 2 0

y (y) A 2 1

y (y) A 2 2

(y) A 3


= 2+2×(n-1)


= 2×2n-1

= 2 -1

R1(y) = R1(y-1)R0( )


R2(y) = R2(y-1)R1( )

R3(y) = R3(y-1)R2( )


) )))))(A(A(A(A(A( A(y) A

k

kk-k-k-k-k-k 0


11111

y applications


][ 011111 ) ))))(A(A(A(A(A A kk-k-k-k-k-

y-1 applications


Rk(y) = Rk(y-1)Rk-1( )


R1(y) = R1(y-1)R0( )R2(y) = R2(y-1)R1( )

R3(y) = R3(y-1)R2( )

Ackermann’s FunctionFor which k can Ackermann’s Ak(y) be computed?


R1(y) = R1(y-1)R0( )R2(y) = R2(y-1)R1( )

Hence (by induction) For each k, Ackermann’s Ak(y) is computed by the primitive recursive program Rk(y)

k, PR Rk, y Rk (y) = Ak(y)

This is a different program for each k!Is there one PR program R(k,y) that works for all k?

PR R, y,k R (k,y) = A(k,y)

Rk(y) = Rk(y-1)Rk-1( )

R3(y) = R3(y-1)R2( )

Sorry no. How does the complexity of Rk increase with k in a waythat cant grow with input size?Applications of Prim Recur

Constant vs FiniteConstant Resources

Fixed at Compile TimeResources

Grow with Input

Java:

TM:

PR:

• # Lines of code• # & Range of Variables • Instruction Set

(Computed in One Step)

• Allocated memory• Running time (Might not Halt)

• # of States (# Bits on Black Board)

• Tape Alphabet• State Transitions

(Computed in One Step)

• # Cells used• Running time (Might not Halt)

• # Lines of code• # of Variables • Function Names

Mult(x,y), Sum(x,y), Succ(y)• Applications of Prim Recur

• Range of Variables• Running time

x+yx+(y-1)

x+(y-2)….

Depth of Rec Tree

x+0

Will HaltTime is

Bounded

Model of Computation: Turing Machine First Order Logic

q q’

k, TM M, x,yk M multiplies xy in one time step.

Yes. We showed you can.The number of states needed grows as a

function of k.

Model of Computation: Turing Machine First Order Logic

q q’

No, it cannot remember arbitraryintegers in its states.

The number of states needed grows as a function of x & y.

TM M, x,y M multiplies xy in one time step.

k, PR Rk, n Rk (n) = Ackk(n)

Yes. We showed you can.It needs k applications of primitive recursion.

First Order Logicf is defined from g & h by primitive recursion if• f(x,y) = h(x,y-1,f(x,y-1))• f(x,0) = g(x)

No, it cannot have a growing number of applications of primitive recursion.

PR R, k,n R(k,n) = Ack(k,n)


Ackermann’s Function• Claim: For each k, Ak(y) can be computed

with primitive recursive Rk(y) with k applications of primitive recursion.

• Proof: • True for k = 0, because R0(y) = y+2• Assume by way of induction that it is true for k-1.


Rk(y) = Rk(y-1)Rk-1( ).

Ackermann’s Function• Claim: No primitive recursive

with k applications of primitive recursion grows as fast as than Ak+1(y).

• Proof: By seeing that this is tight.

To design the fastest growing Fastk(y) with k applications of primitive recursion:• First let Fastk-1(y) be the fastest growing

with k-1 applications of primitive recursion:• By induction assume Fastk-1(y)≈Ak-1(y).

• Make Fastk(y) as big as you can.


Fastk(y) = Fastk(y-1)Fastk-1( Fastk-1( Fastk-1( Gives Ak(y).Gives Ak(3y) << Ak+1(y)


• Claim: A(k,n) is not primitive recursive.• Proof: Suppose it was by program R(k,n).• Let kR denote the number of applications of

primitive recursion in R(k,n).• For k>kR, A(k,n) grows much faster than R(k,n).• Contradiction.

k, PR Rk, n Rk (n) = Ackk(n)

Yes. We showed you can.It needs k applications of primitive recursion.


No, it cannot have a growing number of applications of primitive recursion.

PR R, k,n R(k,n) = Ack(k,n)






• There are computational problems computable by TMs but not by primitive recursive programs.

• We need to give primitive recursive programs more power. • For one, it needs to be given the possibility of

running as long as it wants to. (at the danger of running forever.)

Recursive

• Any primitive recursive function f is recursive. But we need more.

RecursiveLet μ denote the least number operator. If g is recursive then so is f(x) = μy [g(x,y)] computed as follows.

procedure f(x) for y = 0..∞

if g(x,y) = 0 return(y)

Note if • "y g(x,y) ≠ 0 • or first a y is tried for which g(x,y) does not halt.

(denoted g(x,y) = ∞)Then this code does not halt (denoted f(x) = ∞)

RecursiveLet μ denote the least number operator. If g is recursive then so is f(x) = μy [g(x,y)] computed as follows.

procedure f(x) for y = 0..∞

if g(x,y) = 0 return(y)

=

the least number y such that • g(x,y) = 0• "y’<y g(x,y’) ≠ ∞

∞ if no such y exists


• Recall Config(x,t) = <q,tape,head> is the configuration that the TM will be in on input x after t time steps.

• Let HaltTime(x) = t be the time at which the TM will halt.

1 0 10010q

1 0 00000

= tape

= head

TM to Recursive

HaltTime(x) = HasHalted( )

0 iff this is a halting configuration.HasHalted(<q,tape,head>) =

μt Config(x,t)


• Recall Config(x,t) = <q,tape,head> is the configuration that the TM will be in on input x after t time steps.

• Let HaltTime(x) = t be the time at which the TM will halt.• Recall Output(x,t) is the output of the TM on input x

if it halts after t time steps.and Output(x) is the output of the TM on input x.

1 0 10010q

1 0 00000

= tape

= head

TM to Recursive

Output(x) = Output(x,HaltTime(x))

Proving that this is Recursive!






Done

Define and prove this now.

Hilbert’s 10th problem was is there an algorithm to find an integer solution to a set (or even single) polynomial equation.Register Machines were designed to perhaps be better than Turing Machines at this task.

Register Machine Model

Contents of Memory Cell

Number of Memory Cells

Turing Machines constant size(0/1)

arbitrarily many

Register Machines arbitrarilylarge integer

constant number(10)

Fixed size as input grows

$ c, " I,

Size grows with input" I, $ c,


Model of Computation: Register Machine

• The model has a constant number of registers that can take on any integer ≥ 0.

• A program is specified by code made from the following commands:• Ri = 0• Ri = Ri+1• if Ri=Rj goto line k

Surely this is not enough!How do you decrement?

Copy: Ri = Rj

Ri

1. Ri = 02. if Ri=Rj goto line 53. Ri = Ri+14. if Ri=Ri goto line 25. done

Rj


Add: Ri = Rj + Rk

Ri

1. Ri = Rj

2. Rk’ = 03. if Rk’=Rk goto line 74. Ri = Ri+15. Rk’ = Rk’+16. goto line 37. done

RkRj Rk’


Subtract: Ri = Rj - Rk

Ri

1. Ri = 02. Rj’ = Rk

3. if Rj’=Rj goto line 74. Ri = Ri+15. Rj’ = Rj’+16. goto line 37. done

RjRkRj’


Subroutine Call: Ri = subtract(x,y)

1. Ri = 02. Rj’ = Rk

3. if Rj’=Rj goto line 74. Ri = Ri+15. Rj’ = Rj’+16. goto line 37. done

Register Machine Modelsubtract(Rj,Rk)

Solutions:1. Copy the code in every

where you use it.2. Ri = 0• Rj = 0• if Ri=Rj goto line k

3. But how do you where to return to?•Before calling save a pointer where to return to 10. Rreturn = 5 11. goto line 1

8. Rreturn = 5 goto line 12

Recursive to Register Machine

• Zero(x) = 0• Successor(x) = x+1• In,i(x1,x2,..,xn ) = xi

• f(x,y) = g(h1(x,y),…,hn(x,y))

Recursive Register Machine

Ri = 0Ri = Ri+1Rk = Ri

Use results as inputs

Primitive recursion• f(x,y) = h(x,y-1,f(x,y-1))• f(x,0) = g(x)

f = g(x)for y’=1..y

f = h(x,y’-1,f)return(f)

Least Number Operator f(x) = μy [g(x,y)]

for y = 0..∞if g(x,y) = 0 return(y)





• by a (simple) Register Machine Done






• by an And/Or/Not Circuit

x3x2x1

OR

ORANDAND

OR

NOT

• A circuit is a directed acyclic graph of and/or/not gates

0 1 0

• An input X assigns a bit to each incoming wire.

0 0

0

0 0

1

And/Or/Not Circuits

The bits percolate down to the output wires.

• Key differences in this model.A given TM • can take an input with

an arbitrarily large number of bits.

And/Or/Not Circuits

A given circuit• has a fixed number n of

input wires an hence can only take inputs with this number of bits.

• Key differences in this model.A given TM • has a fixed finite

description

And/Or/Not Circuits

A given circuit• For each n, the circuit Cn

has a finite description, but in order handle inputs of all size, we need a different one for each n, C1, C2, C3, ….

• Key differences in this model.A given TM • has a fixed finite

description

And/Or/Not Circuits

A given circuit• Allowing it to have

an infinite description C1, C2, C3, …. is not fair.

• To make it fair, each Cn

should be constructed in a Uniform way.

• Key differences in this model.A given TM • the contents of the tape

and location of the head depend on the input (and on the time step) and hence are not known at “compile time”.

And/Or/Not Circuits

A given circuit• the 0/1 value on

each wire (input & internal) depend on the input (but not on time)

• Key differences in this model.A given TM • the contents of the tape


And/Or/Not Circuits

A given circuit• the gates and

their connections must be fixed at “compile time”

• Clearly circuits compute.• Any function f(X) of n bits can be computed

with a nonuniform circuit of size O(2n).

And/Or/Not Circuits

X f(X)

000000 0

000001 1

000010 0

000011 0

000100 1

2n

f(x)

n

= Cn

And/Or/Not Circuits

X f(X)

000000 0

000001 1

000010 0

000011 0

000100 1

2n

n

Outputs 1iff

X = 000001

¬x1¬x2 ¬x3 …xn

AND

¬x1 ¬x2 ¬x3 x4 … ¬xn

AND

Outputs 1iff

X = 000100

x1 x2 ¬x3 … ¬xn

AND

Outputs 1iff

X = 110..0

…

Repeat this for every value of X

for which f(X)=1.

And/Or/Not Circuits¬x1¬x2 ¬x3 …xn

AND

¬x1 ¬x2 ¬x3 x4 … ¬xn

AND

x1 x2 ¬x3 … ¬xn

AND…

Outputs 1 iff f(X)=1.

OR

Computable

History of Computability

• Which computational problemsare computable by such circuits?

• Unreasonable because for each n,a separate circuit C1, C2, C3, …. of size 2n needs to be defined.

• This is big enough to list all the answers. This “algorithm” does not have finite size.

• An algorithm that needs something different specified for each n is called “nonuniform”.





NonUniform & arbitrarily large input unreasonableNonUniform& fixed input size reasonableUniform & arbitrarily large input reasonable

If a TM can compute it in time T(n), then it can be computed by

a uniform circuit of size O(T2(n)).

TM to CircuitsGiven a TM M,

Need a separate circuit C1, C2, C3, ….

for each input size n.

f(x)

whose running time on inputs of size n is at most T(n)..

T(n)

T(n)

n

A given circuit• has a fixed number n of

input wires an hence can only take inputs with this number of bits.

TM to CircuitsGiven a TM M,

All dependent on the input.f(x)

whose running time on inputs of size n is at most T(n)..

T(n)

T(n)

n

• the 0/1 value on each wire at the tth level encodes the tape and head position of the TM at time t.

t

TM to CircuitsASCII-celled TM at time t

OLLEH

Encoded by wires at layer t+1 of the circuits.O

00110100H

01001000E

01000011A

01000000L

01001101q’

Encoded by wires at layer t of the circuits.

O00110100

H01001000

E01000011

L01001101

L01001101

q

ASCII-celled TM at time t+1

OLAEH

TM to Circuits

Put gates between these layers to compute t+1 from t.

O00110100

H01001000

E01000011

A01000000

L01001101

q’

O00110100

H01001000

E01000011

L01001101

L01001101

q

TM to Circuits

O00110100

H01001000

E01000011

A01000000

L01001101

q’

O00110100

H01001000

E01000011

L01001101

L01001101

q

These bits(i.e. if head is here and contents of this cell)

influence which bits?

TM to Circuits

O00110100

H01001000

E01000011

A01000000

L01001101

q’

O00110100

H01001000

E01000011

L01001101

L01001101

q

These bits

Move head rightand change state

Write Character

or move leftand change state

leave putand change state

TM to Circuits

O00110100

H01001000

E01000011

A01000000

L01001101

q’

O00110100

H01001000

E01000011

L01001101

L01001101

q

Conversely, these output bits depend on which input bits?

TM to Circuits

O00110100

H01001000

E01000011

A01000000

L01001101

q’

O00110100

H01001000

E01000011

L01001101

L01001101

q

Some smallcircuit

Computes the entiretransition function

δ(q,c) = <q’,c’,direction>

TM to Circuits

O00110100

H01001000

E01000011

A01000000

L01001101

q’

O00110100

H01001000

E01000011

L01001101

L01001101

q

Some smallcircuit

Hence, the same small circuitgets copied all along.

A given TM • the contents of the tape


A given circuit• the gates and

their connections must be fixed at “compile time”

TM to Circuits

O00110100

H01001000

E01000011

A01000000

L01001101

q’

O00110100

H01001000

E01000011

L01001101

L01001101

q

Some smallcircuit

Some smallcircuit

Some smallcircuit

Some smallcircuit

Some smallcircuit

TM to Circuits

O00110100

H01001000

E01000011

L01001101

L01001101

qstart

Some smallcircuit

Some smallcircuit

Some smallcircuit

Some smallcircuit

Some smallcircuit

qstart x1 x2 x3 x4 x5 …. Top layer is specified by input X.

TM to Circuits

O00110100

H01001000

E01000011

A01000000

L01001101

qhalt

Some smallcircuit

Some smallcircuit

Some smallcircuit

Some smallcircuit

Some smallcircuit

y1 y2 y3 y4 y5 ….

The bottom layer specifies the output.

O00110100

H01001000

E01000011

L01001101

L01001101

qstart

Some smallcircuit

Some smallcircuit

Some smallcircuit

Some smallcircuit

Some smallcircuit

qstart x1 x2 x3 x4 x5 ….

TM to Circuits• If a TM can compute it in time T(n), • then it uses at most T(n) cells of tape,• then the size of the circuit is O(T(n)×T(n)).• The circuit is constructed in a uniform way

from many copies of the same small circuit.





If a TM can compute it in time T(n), then it can be computed by

a uniform circuit of size O(T2(n)).

Done

A Java program• on an input of size n• can produce this

uniform T(n)×T(n) circuit• and evaluate it on the input.

Uniform & arbitrarily large input reasonable



• by an And/Or/Not CircuitComputer Circuit

• A computer circuit differs from a circuit in two ways:• Memory• Cycles in the graphs of gates

Computer Circuits• Time is kept by a clock.• Value xt may change through time.• But when clock is high, current value

is stored in memory.• When clock is low, it remembers

previous value.• Output of memory is this value stored.

Memory

xt

xt

Clockxtxt-1

Computer Circuits• Time is kept by a clock.• Value xt may change through time.• But when clock is high, current value

is stored in memory.• When clock is low, it remembers

previous value.• Output of memory is this value stored.

Memory

xt

zt

Clockyt

zt = yt

xt if clock=1if clock=0

¬c

AND

OR

c

AND

xt+1

yt

Computer CircuitsASCII-celled TM at time t

OLLEHq

ASCII-celled TM at time t+1

OLAEHq’

Encoded by wires at layer t+1 of the circuits.O

00110100H

01001000E

01000011A

01000000L

01001101q’

Encoded by wires at layer t of the circuits.

O00110100

H01001000

E01000011

L01001101

L01001101

q

Computer Circuits

O00110100

H01001000

E01000011

A01000000

L01001101

q’

O00110100

H01001000

E01000011

L01001101

L01001101

qmemory

Some smallcircuit

Some smallcircuit

Some smallcircuit

Some smallcircuit

Some smallcircuit

Computer Circuits

O00110100

H01001000

E01000011

L01001101

L01001101

qmemory

Some smallcircuit

Some smallcircuit

Some smallcircuit

Some smallcircuit

Some smallcircuit



• by an And/Or/Not CircuitComputer Circuit Done

Circuit Depth

Circuits Depth• The depth of a circuit:

• is the length of the longest path from an input to an output.

• It indicates evaluation time.• It relates to parallel

computation time.

x3x2x1

OR

ORANDAND

OR

NOT

0 1 0

0

0 0

0 0 1• The Size of a circuit:• is the number of gates.• It relates to sequential

computation time.

x1 ¬x2 x3 …¬xn

AND

Any function f(X) of n bits can be computed with a circuit of size 2n.

¬x1 x2 ¬x3 … xn

AND

x1 x2 ¬x3 … ¬xn

AND


…

OR

1

If you allow arbitrary fan.But we assume 2 inputs.

Circuits Depth

x1 ¬x2 x3 …¬xn

AND

Any function f(X) of n bits can be computed with a circuit of size 2n.

¬x1 x2 ¬x3 … xn

AND

x1 x2 ¬x3 … ¬xn

AND


…

OR

O(log(n))

O(n)

Circuits Depth

O00110100

H01001000

E01000011

L01001101

L01001101

qstart

Some smallcircuit

Some smallcircuit

Some smallcircuit

Some smallcircuit

Some smallcircuit

qstart x1 x2 x3 x4 x5 ….

• If a TM can compute it in time T(n), • then it uses at most T(n) cells of tape,• then the size of the circuit is O(T(n)×T(n)).• The circuit is constructed in a uniform way

from many copies of the same small circuit. O(T(n))

Circuits Depth

How to add 2 n-bit numbers.

**

**

**

**

**

**

**

**

**

**

**

+

**

*

**

**

**

**

**

* **

**

**

**

**

+


**

*

**

**

**

**

* **

* **

*

**

**

**

**

+


**

*

**

**

**

* **

* **

*

* **

*

**

**

**

**

+


**

*

**

**

* **

* **

*

* **

*

* **

*

**

**

**

**

+


**

*

* **

*

* **

*

* **

*

* **

*

* **

*

* **

*

* **

*

* **

*

* **

*

***

*

+*

*


**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

+*

*

*

**

*

Takes O(n) time.Even with parallel help.

O(n) circuit depth.


**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

+*

*

*

**

*

xi = ith input bit of X. yi = ith input bit of Y.

zi = ith output bit of Z.

ci = ith carry bit.

zi = lowBit( ci+xi+yi ) +

xi yi ci

ci+1 zi ci+1 = highBit( ci+xi+yi )


c1 x1 y1

+z1

x0 y0 c0

+z0

c2 x2 y2

+z2

c3 x3 y3

+z3

c4 x4 y4

+z4

c5 x5 y5

+z5

c5 x5 y5

+z6 z7

O(n) circuit depth.

We need to compute the carries ci sooner.

Is O(n) depth intrinsic to adding or can it be done with less depth?


**

**

**

**

**

**

**

**

**

**

**

+ **

Previous alg I was taught Little extra thought.

Size (# of gates)

Depth (path leng)

O(n)

O(n)

O(n2)

O(logn)

O(n)

O(logn)


Trade off between size and depth! Win-Win

**

**

**

**

**

**

**

**

**

**

**

+ **

At level l= 1,2,3,…,logn from the circuit inputs.Break the input bits into n/d blocks of size d = 2l.

n/d = 3 blocks d = 4 bits per blocks


Xd0

Yd0

Xd1

Yd1

Xd2

Yd2


Note 0 ≤ Xli ≤ 2d-1

Value of 2d gives a carry out of these d bits.Xd

i + Ydi ≥ 2d, then there is definitely a carry

to the next block. We say the block “Generates” a carry. Wire gd

i = true.Xd

i + Ydi ≥ 2d-1, then there is a carry out

only if there is a carry in. We say the block “Propagates” a carry. Wire pd

i = true.

1


**

**

**

**

**

**

**

**

**

**

+

AND

yixi

g1i

1n/1 = 12 blocks d = 1 bits per blocks

Wire g1i = block “Generates” a carry (ie. one out for sure)

xi+yi ≥ 21 = 2

g1i = And( xi, yi )

Wire pdi = block “Propagates” a carry (ie. one out if one in)

xi+yi ≥ 21-1 = 1 pd

i = Or( xi, yi )

1

*1

OR

yixi

p1i

10



Wire gdi = block “Generates” a carry (ie. one out for sure)

iff first half generates a carry and second half propagates it or second half generates it = (gd/2

2i and pd/22i+1) or (gd/2

2i+1)

Wire pdi = block “Propogates” a carry (ie. one out if one in)

iff first half propagates a carry and second half propagates a carry = (pd/2

2i and pd/22i+1)

1

How to add 2 n-bit numbers. 1

Xd/22i+1

Yd/22i+1

Xd/22i

Yd/22i

1 1

n/d = 3 blocks d = 4 bits per blocks How to add 2 n-bit numbers.

Xd/22i+1

Yd/22i+1

Xd/22i

Yd/22i

gdi pd

i

gd/22i+1 pd/2

2i+1 gd/22i pd/2

2i

gdi+1 p

di+1

gd/22i+3 pd/2

2i+3 gd/22i+2 pd/2

2i+2

Xd/22i+1

Yd/22i+1

Xd/22i

Yd/22i

g2di/2 p

2di/2

Little extra thought.

Size

Depth

n+n/2+n/4+…1 = O(n)

O(logn)


Suppose you know the carries between blocks of size d.Then there is a carry between these block of size d/2, iff carry at beginning and propagate or generate.

How to add 2 n-bit numbers. *

Xd/22i+1

Yd/22i+1

Xd/22i

Yd/22i

* 1 1

Compute ALL carries between bits.


*


Size

Depth

n+n/2+n/4+…1 = O(n)

O(logn)

* * * * * * * * * * * * * * * *

Compute ALL carries between bits.

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

+*

*

*

**

*

xi = ith input bit of X. yi = ith input bit of Y.

zi = ith output bit of Z.

ci = ith carry bit.

zi = lowBit( ci+xi+yi )

+

xi yi ci

zi



Size

Depth

O(n)O(1)

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

+*

*

*

**

*


Previous alg I was taught Little extra thought.

Size (# of gates)

Depth (path leng)

O(n)

O(n)

O(n2)

O(logn)

O(n)

O(logn)

How to multiply 2 n-bit numbers.

* * * * * * * * * * * * * * * *

* * * * * * * * * * * * * * * *

* * * * * * * * * * * * * * * *

* * * * * * * * * * * * * * * *

* * * * * * * * * * * * * * * *

* * * * * * * * * * * * * * * *

x1 x2 x3

xn

Add n 2n-bit

numbers

O(n logn) circuit depth.Can we multiply in O(log2n) depth?

How to multiply 2 n-bit numbers.

x3

+

x1 x2

+

x4

+x5

+x6

+x7

+

Can we multiply in O(logn) depth?

O(logn) circuit depth.


6.




DoneCircuit Depth

Arithmetic Circuits

x3x2x1

×

+ × +

/

-

• An arithmetic circuit has +, -, ×, & / gates.

3 5 7

• An input X assigns a real number to each incoming wire.

8 21

-14

168 -12

12

Arithmetic Circuits

The real numbers percolate down to the output wires.




Done

Circuit Depth

Arithmetic CircuitsNeural Nets

Neural Nets

• Inputs x1, x2 , x3 , …, xn and output y are binary.• Weights w1 , w2 , w3 , …, wn are real numbers

(possibly negative).• y = 1 iff Σi wi×xi ≥ T• The neural net learns by adjusting weights wi.

y

Threshold

T

x1 x2 x3 … xn w1 w2 w3 … wn

Neural Nets

• You can build an AND, OR, and NOT gate from these.• Hence, they are as powerful as circuits.

y

Threshold

T

x1 x2 x3 … xn w1 w2 w3 … wn

Neural Nets




Done

Circuit Depth

Arithmetic CircuitsNeural Nets






Done

Turing 1936

Java ComputableHistory of Computability

TM ComputableAre these definitionsequivalent?

Church says “Yes”All reasonable models

of computation are equivalentas far as what they can compute.

What about time?

Computable

Exp

Poly

Halting

Jack Edmonds 1965

Time: input size running timeFor large n, n100 << 2n


Computable

Exp

Poly

Jack Edmonds 1965


Revised Church says:All reasonable models

of computation are equivalentwithin a polynomial in time

Halting


Revised Church says:All reasonable models

of computation are equivalentwithin a polynomial in time• For any two models of computation M1 and M2,

• there is a constant c,• for any computation problem P,• if P can be solved in time T(n) in M1,• then it can be solved in at most time (nT(n))c in M2.

Note c depends on the models M1 and M2, but not on the problem P.


7.






• by a Non-Deterministic Machine?

Proof

Reasonable?

Reasonable?

Reasonable?

Reasonable

Yes, polynomially slower

See assignment

See assignmentReasonable

Non-Deterministic MachinesA Non-Deterministic TM/Java/… is • the same as a deterministic one,

except that its moves are not deterministic. • It has a “choice” as to which step to take next.• Deterministic: (qi,c) = <qj,c',right>

• Non-Deterministic: <qi,c; qj,c',right> ϵ • A non-deterministic machine M on input I

is said to accept its input • if there exists an accepting computation.

• Note problems computed need yes/no answers.

Jack Edmonds Steve Cook

Non-Deterministic Machines• If I is a “yes” instance:• A non-deterministic (fairy god mother) could prove

it to you by telling you which moves to make. • This information is said to be a witness.• Given an instance I and a witness/solution S,

there is a poly-time deterministic alg Valid(I,S) to test whether or not S is a valid.

• With this you can convince your non-believing boss that I is a “yes” instance:

• If I is a “no” instance:• There is no witness she could tell you. • You cannot convince your boss.

• An equivalent definition:• Problem P can be computed non-deterministically

if P(I) = S Valid(I,S)

• Example: Satisfiablity:• Instance: A circuit I.• Solution: A satisfying assignment S. • I is a yes instance if there is such an assignment . • Given an instance I and a solution S,

there is a poly-time alg Valid(I,S) to test whether or not S is a valid solution of I.

Non-Deterministic Machines

Circuit I:

0 1 1 0 1 0 1 Assignment S:

1

P(I) = S Valid(I,S)

• Suppose you and your boss only have polynomial time i.e. Time(Valid(I,S)) n1000, where n = |I|. Note |S| n1000. Then the problem P is said to be in Non-Deterministic Polynomial Time (NP).


P(I) = S Valid(I,S)

NPIf P NP Deterministic Time is? Given an input I, try all witnesses S.n1000

2

Exp

Poly

SAT

• Suppose you and your boss only have polynomial time i.e. Time(Valid(I,S)) n1000, where n = |I|. Note |S| n1000. Then the problem P is said to be in Non-Deterministic Polynomial Time (NP).


P(I) = S Valid(I,S)

NP

Exp

Poly

GCD

SAT

If P Polynomial Time, You don’t even need the fairy good mother. P NP

• Suppose you and your boss have TM computable time i.e. Valid(I,S)) is computed by a TM. Note |S| can be arbitrarily long. Then the problem P is said to be in Acceptable


NP

If I don’t have Fairy God mother,why don’t I just try every possible witness S?

Exp

Poly

Computable

Acceptable

GCD

SAT• If I is a “yes” instance

there is a witness alg halts and answers “yes”

• If I is a “no” instance there is no witness alg runs forever

A Problem P is said to be computable/decidable/recursive

• if on every input the machine halts and give the correct answer.

Acceptable

Computable


Acceptable

Computable

Three equivalent definitions for a problem PAcceptable: TM M• On every yes input, the machine halts

and gives the correct answer.• On every no input, the machine

could halt and gives the correct answer or could run for ever.

• Witnesses: computable Valid such that P(I) = S Valid(I,S)• Enumerable: TM M• Every yes input I, is eventually printed.• No no input is ever printed.


Acceptable

ComputableIs there a languagethat can be acceptedbut not computed?

Turing proves uncomputable.Alg: Run M on I. If it halts halt and say “yes” If it does not halt run forever.

Halting(M,I) = Does TM M halt on input I.

Halting


The Post Correspondence Problem (PCP)• The input is a finite collection of dominoes

• A solution is a finite sequence of the dominoes bca

aab

caa

abcc

aab

• So that the combined string on the top is the same as that on the bottom

a b c a a a b ca b c a a a b c

bca ,

aab ,

caa ,

abccI =


I can give a solution as a

witness.

bca

aab

caa

abcc

aab


I can give a solution as a

witness.

But how big is this witness?

NP

Exp

Poly

Computable

Acceptable

• If dominos can’t be repeated, |S| |I|

• If dominos can be repeated, |S| can be arbitrarily long.

PCP?

PCP?

Without Repeats

WithRepeats

CoNPnot SAT

Computable

Exp

Poly

NP

GCD

SAT

HaltingAcceptable CoAcceptable

Not Halting

Non-DeterministicYes instances I

• have accepting computations

• have witness/solutions.No instances I

• Do not.

Co-Non-DeterministicNo instances I

• have accepting computations

• have witness/solutions.Yes instances I

• Do not.


Computable

Exp

Poly

Known

GCD

NP

complete

NP-Complete Problems

Problem Pnew is NP-Complete

• Pnew not too hard.

• Pnew NP

Pnew

Test in poly-time if a given solution

is valid

Computable

Exp

Poly

Known

GCD

NP

complete

NP-Complete Problems

• Pnew sufficiently hard.

Pnew

Sat

Problem Pnew is NP-Complete

• Pnew not too hard.

• Pnew NP

If being able to solve this problem fast means that you can solve every problem in the class fast.







• by a Quantum Machine?

Proof

Reasonable?

Reasonable?

Reasonable?

Reasonable


See assignment


Reasonable wrt computing power

Unreasonably fast.

Quantum MachinesWhat about Quantum Machines?• Based on quantum mechanics,

at each point in time a quantum TM is in the super-position of any number of configurations of normal TMs. • In time T, it can only flip T quantum coins so can only be

in the super-position of 2T configurations.• Hence, can be simulated by a normal TM in time 2T×T

time.• Hence, Quantum Machines can’t compute more,

just sometimes faster.• Factoring can be done in poly-time, ie 6=2×3.• It is believed that NP-complete problems still take

exponential time.








Proof

Reasonable?

Reasonable?

Reasonable?

Reasonable


See assignment


Reasonable wrt computing powerUnreasonablely fast.

Reasonable wrt computing powerUnreasonablely fast.








Proof


These are all Automata Machines

computing to accept or reject inputs.

What about Grammars that

generate strings?

Generates Strings:

GrammarsContext Free Grammar:

Generates Language/Computational Problem:Grammar generates string

iff string is in the language iff Computational Problem says Yes.

GrammarsContext Free Grammar:• Terminals: a,b,c,…

characters in the final string being generated.• NonTerminals: A,B,C,…

characters that generate more substrings.• Rules: A aAb

cBccBcc At any time, you can replace A with aAb• Start Symbol: S• Stop when no more nonterminals in your string.• Examples:• English, JAVA, …

Grammars• All a CFG can do is • union, • concatenate, • and spew & plop.

GrammarsGrammar for union L1 L2:

• S S1

S2

SS1

L1

SS2

L2

or

GrammarsGrammar for concatenation L1 L2:

• S S1 S2 S

L1 L2

S1 S2

GrammarsGrammar for spew & plop an # bn:• Spew: SaSb• Plop: S # S

aaSbbaaaSbbb

aaaaSbbbbaaaaaSbbbbb

aSb

aaaaa#bbbbb

Grammars

Grammar for S {0,1}* an bm c2m+3 dn {0,1}*

Linked because must be the same size n.

Linked because must be

the double the size.

• Step 1: Look for links and concatenations

Not Linked because *

means anything.

Grammars

Extra stuck in

Extra concatenated on

Extra concatenated on



Grammars


A Q A

• Step 1: Look for links and concatenations• Step 2: Give names to the parts.• Step 3: Build Grammar rules.• Step 4: Recurse.

• Concatenate: S AQA

Grammars

Grammar for A {0,1}i


Spew

(empty string)

C

• Spew: A CA• Plop: A ε

A

CCACCCA

CCCCACCCCCA

CA

CCCCC

Grammars

Grammar for C {0,1}

• Step 1: Look for links and concatenations• Step 2: Give names to the parts.• Step 3: Build Grammar rules.

Union

• Union: C 0 | 1

CCCCC1 0 0 1 0

Grammars

Grammar for Q an bm c2m+3 dn

• Which to spew first a&d or b&c

• Step 1: Look for links and concatenations.

Linked because must be the same size n.


the double the size.

Grammars

Grammar for Q an bm c2m+3 dn


T

• Spew: Q aQd• Plop: Q Tccc

Q

aaQddaaaQddd

aaaaQddddaaaaaQddddd

aQd

aaaaaTcccddddd

Grammars

Grammar for T bm c2m


Q

aaQddaaaQddd

aaaaQddddaaaaaQddddd

aQd

aaaaaTcccdddddaaaaabTcccccddddd

aaaaabbTcccccccdddddaaaaabbbTcccccccccdddddaaaaabbbcccccccccddddd

• Spew: T bTcc• Plop: T ε

Grammars


(concatenate)S AQA

AQ

AT


(spew & plop)A CA | ε(union)C 0 | 1(spew & plop)Q aQd | Tccc(spew & plop)T bTcc | ε

Q an bm c2m+3 dn

A {0,1}*

T bm c2m

C {0,1}

S

Grammars

Grammar for S an bm cn dm



the same size.

Grammars



Spew

Plop

Not linked

Grammars



Spew

Plop

Not linked

Links over lap and hencecan’t be done by a CFG!

Pumping Lemma (ugly)

Grammars

Grammar for S an bn cn



the same size.


Grammars

Grammar for S α1 α2 … αn # α1 α2 … αn


Linked because must be the same string.

…


Grammars

Grammar for S α1 α2 … αn # αn αn-1 … α1


Linked because must be the same string.

…

Links do not over lap and hencecan be done by a CFG!

S 0S0 | 1S1 | ε

Grammars

S • begin Alg• x = 5• begin loop• x = ( [ { 3 } ] )

• exit loop• return x• end Alg

Linked

This is what CFG is really good at!



7.8.

Machine Code vs JavaMachine Model:• one line of code

Java Model:• fancy data structures• loops• recursions• object oriented

Machine Code vs JavaMachine Model:• one line of code• one line of memory cells.


Machine Code vs JavaMachine Model:• one line of code• one line of memory cells.


Proof:• This is what a compiler does.

by Java Programby a Machine Code

If a computational problem is computable

Input:Output:

s=6*8+((2+42)*(5+12)+987*7*123+15*54)

Parsing/Compiling

Input: Java Code

Parsing/Compiling

Output: MARIE Machine Codesimulating the Java code.

Input: Java Code

Parsing/Compiling

Output: MARIE Machine Codesimulating the Java code.

Challenge: Keep track of three algorithms simultaneously • The compiler• The Java code being compiled• The MARIE code being produced.

Parsing/Compiling

Parsing/Compiling

Friends - Strong Induction View of Recursion:• The sub-instance given must be

• smaller and • must be an instance to the same problem.

• Combine solution given by friend to construct your own solution for your instance.

• Focus on one step. • Do not talk of their friends friends friends.

• Solve small instances on your own.

Parsing/Compiling

Parsing/Compiling

Algorithm: GetExp( s, i )Input: s is a string of tokens i is a start indexOutput: p is a parsing of the longest valid expression j is the end index

s=6*8+((2+42)*(5+12)+987*7*123+15*54)

Parsing/Compiling

Algorithm: GetTerm( s, i )Input: s is a string of tokens i is a start indexOutput: p is a parsing of the longest valid term j is the end index

s=6*8+((2+42)*(5+12)+987*7*123+15*54)

Parsing/Compiling

Algorithm: GetFact( s, i )Input: s is a string of tokens i is a start indexOutput: p is a parsing of the longest valid factor j is the end index

s=6*8+((2+42)*(5+12)+987*7*123+15*54)

Algorithm: GetExp( s, i )

s=6*8+((2+42)*(5+12)+987*7*123+15*54)

p<term,1> p<term,2> p<term,3>

Algorithm: GetExp( s, i )

Exp

…p<term,1> p<term,2> p<term,k>* * *

Algorithm: GetExp( m )

MARIE Machine Code thatevaluates an expression and stores its value in memory cell indexed by m.

Output:

Algorithm: GetTerm( s, i )

s=6*8+((2+42)*(5+12)+987*7*123+15*54)

p<fact,1> p<fact,2>

Algorithm: GetTerm( s, i )

Term

…p<fact,1> p<fact,2> p<fact,k>+ + +

Algorithm: GetTerm( m )

MARIE Machine Code thatevaluates a term and stores its value in memory cell indexed by m.

Output:

Parsing

Algorithm: GetFact( s, i )

s=6*8+((2+42)*(5+12)+987*7*123+15*54)

Parsing


Fact

42


s=6*8+((2+42)*(5+12)+987*7*123+15*54)

p<exp>


Fact

p<exp>( )

Algorithm: GetFact( m )MARIE Machine Code thatevaluates a factor and stores its value in memory cell indexed by m.

Next token determines which case the factor is.

Output:

Algorithm: GetFactArray( m )MARIE Machine Code thatevaluates a factor and stores its value in memory cell indexed by m.

Output:

Algorithm: GetIfStatement()MARIE Machine Code thatexecutes an IfStatement

Output:


9.

Grammars

A language (a set if strings)• is accepted by a TM

if the TM accepts every string in the language and no other. • is generated by a Grammar

if the grammar generates every string in the language and no other.

The language anbn is generated by the grammar• SaSb• S

The language anbncn is generated by no context free grammars

GrammarsS

B

• Start with the start symbol, S.• You can think of choosing which rule to apply

Non-Deterministically. • If all characters in the string become terminal symbols,

then this string is generated • If the process ends because no rule can be applied

then this process fails to generate a string.• or if it continues forever.

D A

Which strings the does the following grammar generate? • S A

D B

• A a• B bB a bbB

bbbBbbbbB

bbbbbB

bB

This grammar only generates the one string a.

GrammarsContext Free Grammar:• C aBc• Context does not matter.

Context Sensitive Grammar:• dCc aBcc• Context does matter.

acBadCcdaBAbCaBc acBad aBc cdaBAb

acBadCcdaBAbdCa

acBa aBcc daBAb

GrammarsGrammar for anbncn :• Spew equal numbers: SABCS• Allow sorting: BAAB

CAAC CBBC• Turn Cc: S Tc

CTc Tcc

• Turn Bb: Tc Tb

BTb Tbb

• Turn Aa: Tb Ta

ATa Taa

• End: Ta ε• Nothing else can be generated

because, no other way to get rid of all the nonterminals.

SABCABCABCS

AAABBBCCCTc

ABCABCACBS

AAABBBCCCS

AAABBBCCTcc

AAABBBTcccc

AAATbbbbccc

AAABBBTbccc

Taaaabbbccc

aaabbbccc


10.





TM to Grammar

• Encode configuration of the TM as a string of terminals and nontermnials.

• Encode tape as string of terminals: 10110100• Mark the beginning and the end of the tape• Encode state as • Encode head location as


101 1 0 1 0q

10Qi11010a nonterminal Qi for each state qi.

nonterms 1011010

TM to Grammar

Qstart1011010Beginning

TM:

TM starting configuration on input 10110110:

101 1 0 1 0qstart

Qstart1011010Grammar:

TM to Grammar

Qstart1011010Beginning

TM:

QacceptGrammar:

We assume that if the input is in the language, then the TM halts in the state qaccept

with the tape empty.

TM accepts with configuration:

qaccept

QacceptEnding

TM to Grammar

Qstart1011010 QacceptBeginning Computation Ending

TM: Once input is given,computation

is deterministic.

TM to Grammar

Qstart1011010

Grammar:

Rule: A 0AA 1A

QacceptBeginning Computation Ending

TM: Deterministic.

S 1011010The grammar mustbe able to generate

each string in the language.

Hence, some of the steps of the derivation must be nondeterministic.

TM to Grammar

Qstart1011010

Grammar:


TM: Deterministic.

S 1011010Nondeterministic

The TM and the Grammar seem to go in opposite directions.

The standard solution is to have the Grammar simulate

the TM backwards.

TM to Grammar

Qstart1011010 QacceptBeginning Computation Ending

TM: Deterministic.

The TM deterministically overwrites the 0/1 with c.

The backwards TMnondeterministically guesses

whether the overwritten value was 0 or 1.

Backwards TM:

Transitions: (qi,0) = <qj,c,right> (qi,1) = <qj,c,right>

We, however, will stick with forward moving TMs.

TM to Grammar

Qstart1011010

Grammar:


TM: Deterministic.


Remembers the input at the end, so that it can be “outputted”.

Our Grammar:

1011010 Qaccept

Needs to have two copies of the input at the beginning.

1011010 Qstart1011010

TM to Grammar

Qstart1011010

Grammar:


TM: Deterministic.


Nondeterministically guesses the inputand then simulates the TM

deterministically.

Our Grammar:

1011010 Qaccept1011010 Qstart1011010

TM to Grammar

S• Start:Generates:Our Grammar:

1011010 Qstart1011010Start config

of TMCopy of

input

Nondeterministically

guesses the input.

TM to Grammar

S S’ S’ 0S’0 1S’1 Q’start

S• Start:

Our Grammar:

1011010 Qstart1011010Two copies of the input

1011010 Q’start0101101Reverse input

Generates:

Q’start See assignment.

TM to Grammar

S• Start:

Our Grammar:

1011010 Qstart1011010

Generates:See assignment.

b

• Blanks: Add trailing blanks.

1011010 Qstart1011010bbbb

• Accepting TM Computation:

1011010 Qacceptbbbb

1011010 Qaccept

• Blanks: Remove trailing blanks.b

1011010

• Accepting: Qaccept ε

TM to Grammar

S• Start:

Our Grammar:

1011010 Qstart1011010


b



1011010 ???Q’????

• Rejecting or not halting TM computation:

1011010

The derivation never removes all nonterminals and

hence no string is generated!

TM to Grammar

Transition (qi,1) = <qj,c,right>

TM

101 1 0 1 0qi

10Qi11010Grammar:

TM is in state qi and sees a 1.

c01 1 0 1 0

qj

Transformations

10Qj1101010Qjc101010cQj1010

Rule: Qi1 cQj

TM to Grammar

Transition (qi,1) = <qj,c,left>

TM

101 1 0 1 0qi

10Qi11010Grammar:

TM is in state qi and sees a 1.

c01 1 0 1 0

qj

Transformations

10Qj1101010Qjc10101Qj0c1010

Rule: 0Qi1 Qj0c1Qi1 Qj1c

TM to Grammar

S• Start:

Our Grammar:

1011010 Qstart1011010


b



1011010 Qacceptbbbb

1011010 Qaccept

• Blanks: Remove trailing blanks.b

1011010

• Accepting: Qaccept ε

• Transitions: Qi1 cQj0Qi1 Qj0c1Qi1 Qj1c




• by a Context Sensitive Grammardone

A Java program can easily determine if a

string can be generated by a

grammar.

Or can it?

Grammar to Java• Given a string, a JAVA program would have to determine

if it can be parsed by the context sensitive grammar.• It wont know which rules to choose.• But it can try all combinations of rules, ie. all derivations.

Grammar to Java• But how many possible derivations are there

and how long is each?• With a context free grammar, • each rule makes the string longer.• Hence, no derivation can be longer then the length n of

the string.• Hence, there can’t be more than (#rules)n derivations.

• With a context sensitive grammar, • some rules make the string longer and others make it

shorter.• Hence, derivation can be arbitrarily long.• Hence, there are an infinite number of derivations.

Grammar to Java• Given that there an infinite number of derivations,

how can a Java program try them all?• If it follows one path of rules too long it will never get to

try other paths.• It should try all paths of length one, then two, then three, …• Each derivation will eventually be reached.• If there is a derivation of the string, then

the Java program will eventually halt and accept the string.• If there is a not derivation of the string, then

the Java program willrun for ever!

Computable

AcceptableThe set of languages generated by

context sensitive grammarsis equal to the class of languages

accepted by a TM.

Acceptable• if on every yes input • the TM halts and says yes.• the grammar generates the input

• on every no input• the TM runs for ever or says no.• the grammar fails to generates the input

• gets to a nonterminal Qno with no rule• or never stops derivation.

Grammar to Java

Computable

Acceptable

HaltingIs there a

context sensitive grammarthat generates all <P,I>for which P halts on I?

Yes!• Let MU be a “universal” TM

which halts on input <P,I> iff P halts on I.• Let GHalt be the grammar that generates <P,I>

iff MU halts on <P,I>.

Grammar to Java




• by a Context Sensitive Grammardone

acceptable








NonUniform unreasonable Uniform reasonableComputes the same,

but faster.

Decide vs Accept







• by a Context Sensitive Grammar • by a Human?

Human

What about the human brain? Can it compute more than a TM?• Science: • The brain is just an elaborate machine (neural net).• Hence can’t do any more than a TM.

• New Age:• Quantum mechanics is magical.• The brain is based on quantum mechanics.• Hence, the brain can do much more than a machine.• But we already showed QM can’t compute more.

• Religion:• The human has a soul.• Hence, the brain can do much more than a machine.

Computable

Exp

Poly

A complexity class is a set of computational problems that have a similar difficulty in computing them.

Design a new class:1. Choose some model

Java or Circuits2. Deterministic or

Nondeterministic3. Limit some resource

Time or Spaceto Log, Poly, Exp, ….

Complexity Classes

NP Co-NP

Computable

Exp

Poly


Proving C1 C2 is not too hard.Prove C2 can simulate C1.

Proving C1 C2 is hard.Prove P ϵ C2 and P ϵ C1.

Complexity Classes

NP Co-NP

Computable

Exp

Poly


A problem is complete for a class if being able to solve this problem fast means that you can solve every problem in the class fast.

There are complete problems for every class.

Complexity Classes

complete

NP Co-NP

Computable

Exp

Poly

NP

complete

GCD

SATCoNP

completenot SAT

Halting

Games

CoAcceptableAcceptableNon-Deterministic Machines

Jeff EdmondsYork University COSC 4111

Lecture 2

ComputableHalting

Acceptable, Witness, EnumerableComputableAckermann's TimeDouble Exponential TimeE: Exponential TimeExp TimePSpacePH: Poly-Time HierarchyNP & Co-NP

Complexity ClassesPoly-TimeNC: Poly-log depth CircuitsNC2NL: Non-Det Log SpaceL: Log-SpaceAC0, Thres0, Arith0

: Constant DepthNC0: Constant Fan-out

End

Jeff Edmonds York University COSC 4111 Lecture 1 Church's ThesisChurch's ThesisChurch's ThesisChurch's Thesis Primitive RecursivePrimitive RecursivePrimitive.

Documents

primitive recursion

string iff string

string of terminals

ones h g

context sensitive grammar

language anbncn

human brain

grammars context free