CS336: Intelligent Information Retrieval

CS336: Intelligent Information Retrieval

Lecture 6: Compression

Compression

• Change the representation of a file so that– takes less space to store– less time to transmit

• Text compression: original file can be reconstructed exactly

• Data Compression: can tolerate small changes or noise– typically sound or images

• already a digital approximation of an analog waveform

Text Compression: Two classes

• Symbol-wise Methods– estimate the probability of occurrence of symbols

– More accurate estimates => Greater compression

• Dictionary Methods– represent several symbols as one codeword

• compression performance based on length of codewords

– replace words and other fragments of text with an index to an entry in a “dictionary”

Compression: Based on text model

• Determine output code based on pr distribution of model– # bits should use to encode a symbol equivalent

to information content (H(s) = - log Pr(s))– Recall Information Theory and Entropy (H)

• avg amount of information per symbol over an alphabet• H = (Pr(s) * I(s))

– The more probable a symbol is, the less information it provides (easier to predict)

– The less probable, the more information it provides

textcompressed text

encoder

model

decoder

model

Compression: Based on text model

• Symbol-wise methods (statistical)– Can assume independence assumption– Considering context achieves best compression rates

• e.g. probability of seeing ‘u’ next is higher if we’ve just seen ‘q’

– Must yield probability distribution such that the probability of the symbol actually occurring is very high

• Dictionary methods– string representations typically based upon text seen so far

• most chars coded as part of a string that has already occurred

• Space is saved if the reference or pointer takes fewer bits than the string it replaces

textcompressed text

encoder

model

decoder

model

Stastic Statistical Models• Static: same model regardless of text processed

– But, won’t be good for all inputs

• Recall entropy: measures information content over collection of symbols

– high pr(s) => low entropy; low pr(s) => high entropy– pr(s) = 1 => only s is possible, no encoding needed– pr(s) = 0 =>s won’t occur, so s can not be encoded

– What is implication for models in which some pr(s) = 0, but s does in fact occur?

ii

i ppE 21

log

s can not be encoded!Therefore in practice, all symbols must be assigned pr(s) > 0

Semi-static Statistical Models

• Generate model on fly for file being processed

– first pass to estimate probabilities

– probabilities transmitted to decoder before encoded transmission

– disadvantage is having to transmit model first

Adaptive Statistical Models

• Model constructed from the text just encoded

– begin with general model, modify gradually as more text is seen

• best models take into account context

– decoder can generate same model at each time step since it has seen all characters up to that point

– advantages: effective w/out fine-tuning to particular text and only 1 pass needed

Adaptive Statistical Models• What about pr(0) problem?

– allow 1 extra count divided evenly amongst unseen symbols

• recall probabilities estimated via occurrence counts• assume 72,300 total occurrences, 5 unseen characters:• each unseen char, u, gets pr(u) = pr(char is one of the unseen) * pr(all unseen

char) = 1/5 * 1/72301

• Not good for full-text retrieval– must be decoded from beginning

• therefore not good for random access to files

Symbol-wise Methods

• Morse Code– common symbols assigned short codes (few bits)– rare symbols have longer codes

• Huffman coding (prefix-free code)– no codeword is a prefix of another symbol’s

codewordSymbol Codeword Probability

a 0000 0.05 b 0001 0.05 c 001 0.1 d 01 0.2 e 10 0.3 f 110 0.2 g 111 0.1

Huffman codingSymbol Codeword Probability

a 0000 0.05 b 0001 0.05 c 001 0.1 d 01 0.2 e 10 0.3 f 110 0.2 g 111 0.1

• What is the string for 1010110111111110?

• Decoder identifies 10 as first codeword => e• Decoding proceeds l to rt on remainder of string• Good when prob. distribution is static• Best when model is word-based• Typically used for compressing text in IR

eefggf

Symbol Codeword Probability a 0000 0.05 b 0001 0.05 c 001 0.1 d 01 0.2 e 10 0.3 f 110 0.2 g 111 0.1

Create a decoding tree bottom-up1. Each symbol and pr are leafs2. 2 nodes w/ smallest p are joined

under same parent (p = p(s1)+p(s2)3. Repeat, ignoring children, until all

nodes are connected4. Label nodes• Left branch gets 0• Right branch 1

e 0.3

a 0.05

b 0.05

c 0.1

d 0.2

f 0.2

g 0.1

0.1

0.2

0.4

0.3

0.6

1.00

0

0

0 1 1 1

1

1

10 0

• Requires 2 passes over the document

1. gather statistics and build the coding table 2. encode the document

• Must explicitly store coding table with document

– eats into your space savings on short documents

• Exploit only non-uniformity in symbol distribution

– adaptive algorithms can recognize the higher-order redundancy in strings e.g. 0101010101....

Huffman coding

Dictionary Methods

• Braille– special codes are used to represent whole words

• Dictionary: list of sub-strings with corresponding code-words

– we do this naturally e.g) replace “december” w/ “12”

– codebook for ascii might contain • 128 characters and 128 common letter pairs• At best each char encoded in 4 bits rather than 7

– can be static, semi-static, or adaptive (best)

Dictionary Methods• Ziv-Lempel

– Built on the fly– Replace strings w/ reference to a previous

occurrence – codebook

• all words seen prior to current position• codewords represented by “pointers”

– Compression: pointer stored in fewer bits than string– Especially useful when resources used must be

minimized (e.g. 1 machine distributes data to many)• Relatively easy to implement• Decoding is fast• Requires only small amount of memory

Ziv-Lempel• Coded output: series of triples <x,y,z>

• x: how far back to look in previous decoded text to find next string

• y: how long the string is• z: next character from the input

– only necessary if char to be coded did not occur previously

ptr intotext

<0,0,a> <0,0,b> <2,1,a> <3,2,b> <5,3,b> <1,10,a>

a b a a bb a aa b b 10 b’s followed by a

Accessing Compressed Text

• Store information identifying a document’s location relative to the beginning of the file– Store bit offset

• variable length codes mean docs may not begin/end on byte boundaries

– insist that docs begin on byte boundaries• some docs will have waste bits at the end• need a method to explicitly identify end of coded

text

Text Compression• Key difference b/w symbol-wise & dictionary

– symbol-wise base coding of symbol on context– dictionary groups symbols together creating an implicit context

• Present techniques give compression of ~2 bits/character for general English text

• In order to do better, techniques must leverage– semantic content– external world knowledge

• Rule of thumb: The greater the compression,– the slower the program runs or – the more memory it uses.

• Inverted lists contain skewed data, so text compression methods are not appropriate.

Inverted File Compression

• Note: each inverted list can be stored as an ascending sequence of integers– <8; 2, 4, 9, 45, 57, 76, 78, 80>

• Replace with initial position w/ d-gaps– <8; 2,2,5,36,12,19,2,2>– on average gaps < largest doc num

• Compression models describe probability distribution of d-gap sizes

Fixed-Length Compression

• Typically, integer stored in 4 bytes

• To compress the d-gaps for : <5; 1, 3, 7, 70, 250 > <5; 1, 2, 4, 63, 180 >

• Use fewer bytes:– Two leftmost bits store the number of bytes

(1-4)– The d-gap is stored in the next 6, 14, 22, 30

bitsLength Bytes0x<64 164x<16,384 216,384x<4,194,304 34,194,304x<1,073,741,824 4

Value Compressed1 00 0000012 00 0000104 00 00010063 00 111111180 01 000000 10110100

Example

The inverted list is: 1, 3, 7, 70, 250

After computing gaps: 1, 2, 4, 63, 180

Number bytes reduced from 5*4 to 6

Code• This method is superior to a binary encoding• Represent the number x in two parts:

1. Unary code: specifies bits needed to code x– 1 + log x followed by

2. binary code: codes x in that many bits log x bits that represent x – 2 log x

Let x = 9: log x = 3 ( 8 = 23 < 9 < 24 = 16)

part 1: 1 + 3 = 4 => 1110 (unary)

part 2: 9 – 8 = 1

code: 1110 001

Unary code: (n-1) 1-bits followed by a 0, therefore 1 represented by 0, 2 represented by 10, 3 by 110, etc.

Value Compressed1 02 10 04 110 0063 111110 11111180 11111110 0110100

Example

The inverted list is: 1, 3, 7, 70, 250After computing gaps: 1, 2, 4, 63, 180

Number bits reduced from 5*32 to 36.Decode: let u = unary, b = binary

x = 2u-1 +b

1 + log x

x – 2 log x

0

Comparison of the text Compression techniques

Arithmetic

Huffman(charactr)

Huffman(word)

Ziv-Lempel

Compression ratio

Very good

Poor Very good Good

Compression speed

Slow Fast Fast Very fast

Decompression speed

Slow Fast Very fast Very fast

Memory space

Low Low High Moderate

Compressed pat. matching

No Yes Yes Yes

Random access

No Yes yes No