Class 28: Entropy
cs1120 Fall 2011David Evans28 October 2011
A giant python swallows an alligator in Everglades National Park, Fla..Paul Zenk/PBS Nature “Invasion of the Giant Pythons” (yesterday’s New York Times)
2
Plan
Story so Far: Liberal Arts RecapReversing in PythonShannon’s Information Theory: EntropyReturning Quiz 3, PS5, etc.
Course RoadmapSynthesis
Analysis
Ch 2
: Lan
guag
e
Ch 3
: Pro
gram
min
g
Ch 4
: Pro
cedu
res
Ch 5
: Dat
aCh
6: M
achi
nes
Ch 7
: Cos
t
Ch 8
: Sor
ting
and
Sear
chin
g
PS5,
Ch
9: S
tate
PS6,
Ch
10: O
bjec
ts
Intr
acta
bilit
y
PS7,
Ch
11: I
nter
pret
ers
Ch 1
2: C
ompu
tabi
lity PS
8, 9
You
are
here
Ch 1
: Com
putin
g
cs1120 Main Topics
Language: How to describe information processes by defining procedures (Chapters 3, 4, 5, 9, 10)Programming with procedures, lists, recursion (PS1-4)Programming with state (PS5), objects (PS6), languages (PS7)
Logic: How to predict properties about information processes (Chapter 6, 7, 8)Predicting how running time grows with input sizeAre there problems which can’t be solved by algorithms? (Ch 12)
Machines: How to efficiently implement information processes (Chapter 6) (not much on this)Machines, Digital LogicHow to implement a Scheme interpreter (Ch 11, PS7)
From Lecture 1:
• Grammar: study of meaning in written expression
• Rhetoric: comprehension of verbal and written discourse
• Logic: argumentative discourse for discovering truth
• Arithmetic: understanding numbers
• Geometry: quantification of space
• Music: number in time
• Astronomy
BNF, RTN, rules of evaluation for meaning
Interfaces between components (PS6),
evaluation rules
Rules of evaluation, if, recursive definitions
adding with logic,representing numbers
Curves as procedures, fractals (PS3)
Ada, GEB Chapter
Nothing!
Triv
ium
Qua
driv
ium
But…read the Neil DeGrasse Tyson essay!
7
Lists (Tuples) in Python
Scheme (list)> (define p (list 1 2 3))> (car p)1> (cdr p)(2 3)> (cdr (cdr (cdr p)))null
Python (tuple)>>> p = (1, 2, 3)>>> p[0]1>>> p[1:](2, 3)>>> p[1:][1:][1:]()
Python Shortcuts
>>> p = (1, 2, 3)>>> p[2]3>>> p[3:]()>>> p[:2](1, 2)
list[i] Evaluates to ith (counting from 0) element of tuple/list
Runs in (approximately) constant time!
9
Mutable Lists (Lists) in Python
Scheme (mutable list)> (define p (mlist 1 2 3))> (mcar p)1> (mcdr p)(2 3)> (mcdr (mcdr (mcdr p)))null
Python [List]>>> p = [1, 2, 3]>>> p[0]1>>> p[1:](2, 3)>>> p[1:][1:][1:]()
Using Mutable Lists
>>> p = [1, 2, 3]>>> p[0]1>>> p[0] = 2>>> p[2, 2, 3]>>> p.append(4)>>> p[2, 2, 3, 4]
Note: this is different from mappend!It take a single element, not a list.
Scheme Application:(<verb> <operands>)Python procedure call:<verb>(<operand>)Method call:<object>.<verb>(<parameters>)
Trick Question
>>> p = (1, 2, 3)>>> p[0] = 2Traceback (most recent call last): File "<pyshell#23>", line 1, in <module> p[0] = 2TypeError: 'tuple' object does not support item assignment
Reversing a Mutable List
1
p:
2 3
Scheme:
1 2 3
p:
Python:
mlist-reverse!
(define (mlist-reverse! p) (if (null? (mcdr p)) (void) (let ((rest (mcdr p)) (carp (mcar p))) (mlist-reverse! rest) (set-mcar! p (mcar rest)) (set-mcdr! p (mcdr rest)) (mlist-append! p (mcons carp null)))))
Literal Translation(define (mlist-reverse! p) (if (null? (mcdr p)) (void) (let ((rest (mcdr p)) (carp (mcar p))) (mlist-reverse! rest) (set-mcar! p (mcar rest)) (set-mcdr! p (mcdr rest)) (mlist-append! p (mcons carp null)))))
Literal Translation
>>> p=[1,2,3]>>> sreverse(p)>>> p[3, 2, 1]
def sreverse(p): if len(p) == 1: return else: rest = p[1:] first = p[0] sreverse(rest) p[0] = rest[0] p[1:] = rest[1:] p.append(first)
(define (mlist-reverse! p) (if (null? (mcdr p)) (void) (let ((rest (mcdr p)) (carp (mcar p))) (mlist-reverse! rest) (set-mcar! p (mcar rest)) (set-mcdr! p (mcdr rest)) (mlist-append! p (mcons carp null)))))
This is correct, but no sane Python programmer would do it this way!
Python for
Statement ::= for Varible in Expression: Block def gaussSum (n): sum = 0 for i in range(1, n+1): sum = sum + i return sum
(define (loop index result test update proc) (if (test index) (loop (update index) (proc index result) test update proc) result)) (define (gauss-sum n) (loop 1 0 (lambda (i) (<= i n)) (lambda (i) (+ i 1)) (lambda (i sum) (+ i sum))))
Reverse in Python
1 2 3
p:
Note: there is a built-in list method, p.reverse().
def reverse(p): for i in range(0, len(p) >> 1): p[i], p[len(p) – 1 - i] = p[len(p) – 1 - i], p[i]
int(len(p) / 2) len(p) >> 1
19
Information Theory
20
Entropy
H = -pi log2 pi
21
(Ideal) Coin Toss
H = -pi log2 pi
flickr cc:Daniel Dale
22
Nucleotide
N {A, C, G, T}Intuition: need two bits to distinguish four possible values
00 = A, 01 = C, 10 = G, 11 = T
H = -pi log2 pi
23
24
Claude Shannon’s Juggling W. C. Fields
Real Coin Toss?
25
Real Coin TossAccording to Susan Holmes, 51% of the time, a coin will land same way it started. [link]
H = -pi log2 pi
26
27
Are Randall Munroe’s entropy estimates correct?Left for course blog discussion.
28
Class Sorting using “Odette”
Assuming alphabet randomly distributed: O is 15th out of 26 letters in alphabet
0.9880135796356116381290864023063602427958811193759308857790... 0.9880135796356116381290864023063602427958811193759308857790...
= 0.988…
29
Class Sorting using “Odette”
The real class: 41 out of 49 students have names before “Odette”!
0.9880135796356116381290864023063602427958811193759308857790... 0.9880135796356116381290864023063602427958811193759308857790...
-((41/49) * log_2 (41/49) + (7/49) * log_2 (7/49)) = 0.6162235892609252
30
Better Class SortingFront of Class
Hannah – Li-Chang
Anthony - Casey
crm Caroline - Gregory
mdc Michael - Yuan
Charge
Office hours:Peter – right nowSunday, 1-6pmEveryone should be making progress on PS6
Monday: Trick-or-Treat Protocols!Inheritance