of 24 11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 1 (Deterministic) Communication amid Uncertainty Madhu Sudan Microsoft, New England Based on joint works with: (1) Adam Kalai (MSR), Sanjeev Khanna (U.Penn), Brendan Juba (Harvard) and (2) Elad Haramaty (Technion)
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
of 2411/20/2012 TIFR: Deterministic Communication Amid Uncertainty 1
(Deterministic) Communication amid Uncertainty
Madhu SudanMicrosoft, New England
Based on joint works with:(1) Adam Kalai (MSR), Sanjeev Khanna (U.Penn), Brendan Juba
(Harvard)and (2) Elad Haramaty (Technion)
of 24
Classical Communication
The Shannon setting Alice gets chosen from distribution Sends some compression to Bob. Bob computes
(with knowledge of ). Hope .
Classical Uncertainty:
Today’s talk: Bob knows .
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 2
of 24
Outline
Part 1: Motivation Part 2: Formalism Part 3: Randomized Solution Part 4: Issues with Randomized Solution Part 5: Deterministic Issues.
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 3
of 24
Motivation: Human Communication
Human communication (dictated by languages, grammars) very different from Shannon setting. Grammar: Rules, often violated. Dictionary: Often multiple meanings to a word. Redundant: But not as in any predefined way
(not an error-correcting code).
Theory? Information theory? Linguistics? (Universal grammars etc.)?
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 4
of 24
Behavioral aspects of natural communication
(Vast) Implicit context. Sender sends increasingly long messages to
receiver till receiver “gets” (the meaning of) the message. Where do the options come from?
Sender may use feedback from receiver if available; or estimates receiver’s knowledge if not. How does estimation influence message.
Language provides sequence of (increasingly) long ways to represent a message. How? Why? What features are good/bad.
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 5
of 24
Model:
Reason to choose short messages: Compression. Channel is still a scarce resource; still want to
use optimally.
Reason to choose long messages (when short ones are available): Reducing ambiguity. Sender unsure of receiver’s prior (context). Sender wishes to ensure receiver gets the
message, no matter what its prior (within reason).
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 6
of 24
Back to Problem
Design encoding/decoding schemes () so that Sender has distribution on Receiver has distribution on Sender gets Sends to receiver. Receiver receives Decodes to
Want: (provided close), While minimizing
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 7
of 24
Contrast with some previous models
Universal compression? Doesn’t apply: P,Q are not finitely specified. Don’t have a sequence of samples from P; just
one! K-L divergence?
Measures inefficiency of compressing for Q if real distribution is P.
But assumes encoding/decoding according to same distribution Q.
Semantic Communication: Uncertainty of sender/receiver; but no special
goal.
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 8
of 24
Closeness of distributions:
is -close to if for all , -close to (symmetrized, “worst-case” KL-divergence)
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 9
of 24
Dictionary = Shared Randomness?
Modelling the dictionary: What should it be?
Simplifying assumption – it is shared randomness, so …
Assume sender and receiver have some shared randomness and are independent of .
Want
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 10
of 24
Solution (variant of Arith. Coding)
Use to define sequences
…
where chosen s.t. Either Or
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 11
of 24
Performance
Obviously decoding always correct.
Easy exercise:
( “binary entropy”)
Limits: No scheme can achieve Can reduce randomness needed.
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 12
of 24
Implications
Reflects the tension between ambiguity resolution and compression. Larger the ((estimated) gap in context), larger
the encoding length. Coding scheme reflects the nature of human
process (extend messages till they feel unambiguous).
The “shared randomness’’ is a convenient starting point for discussion Dictionaries do have more structure. But have plenty of entropy too. Still … should try to do without it.
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 13
of 24
Deterministic Compression?
Randomness fundamental to solution. Needs independent of to work.
Can there be a deterministic solution? Technically: Hard to come up with single
scheme that compresses consistently for all (). Conceptually: Nicer to know “dictionary” and
context can be interdependent.
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 14
of 24
Challenging special case
Alice has permutation on i.e., 1-1 function mapping
Bob has permutation Know both are close:
(say ) Alice and Bob know (say ).
Alice wishes to communicate to Bob. Can we do this with few bits?
Say bits if .
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 15
of 24
Consider family of graphs Vertices = permutations on Edges = close permutations with distinct
messages. (two potential Alices).
Central question: What is ?
Model as a graph coloring problem
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 16