Why We Need Corpora and the Sketch Engine Adam Kilgarriff Lexical Computing Ltd, UK Universities of Leeds and Sussex.

Why We Need Corpora and the Sketch Engine

Adam KilgarriffLexical Computing Ltd, UKUniversities of Leeds and Sussex

Madrid April 2010 Kilgarriff: Why corpora and how 2

Corpora show us the facts of the language


Exercise

planet Think about the word What could you say about it if you

were writing a dictionary entry Write down three (or more) things


The Sketch Engine: demo

http://www.sketchengine.co.uk


Dictionaries

How to decide what to say about the word?


Dictionaries

How to decide what to say about the word? What the native speaker knows

(introspection)


Dictionaries


(introspection) What other dictionaries say


Dictionaries


(introspection) What other dictionaries say corpus


Four ages of corpus lexicography


Age 1:

Pre-computer

Oxford English Dictionary:• 20 million index cards


Age 2: KWIC Concordances

From 1980 Computerised Overhauled lexicography


Age 2: limitations

as corpora get bigger:too much data

• 50 lines for a word: :read all • 500 lines: could read all, takes a long

time, slow • 5000 lines: no


Age 3: Collocation statistics

Problem:too much data - how to summarise?

Solution:list of words occurring in neighbourhood of headword, with frequencies

Sorted by salience


Collocation listing

For collocates of save (>5 hits), to right of nodeword

word word

forests life

$1.2 dollars

lives costs

enormous thousands

annually face

jobs estimated

money your


Age-3 collocation statistics: limitations

Lists contain junk unsorted for type

mixes together adverbs, subjects, objects, prepositions

What we really want: noise-free lists one list for each grammatical relation


Age 4: The word sketch

Large well-balanced corpus Parse to find

subjects, objects, heads, modifiers etc

One list for each grammatical relation Statistics to sort each list, as before


Macmillan English DictionaryFor Advanced Learners

Ed: Rundell, 2002, 2007


Demo part 2


Fruit task

Choose fruit Concordance

Lemma, noun, lower case Frequency: node forms Write down

Plural freq (pl) Singular freq (sing)

Compute proportion: pl/(pl+sing)


What is a corpus?

A collection of texts (as used for linguistic study)

Which texts? How many?


Which texts?

Written Spoken


Written Books

Fiction Non-fiction Textbooks

Newspapers Letters, unpublished Web pages Academic journals Student essays …


Spoken

Must be transcribed, for text corpora Conversation

Who? Region, class, age-group, situation… Lectures TV and Radio Film transcripts Meetings, seminars …


Which texts?

Different purposes, different text types

Making dictionaries: Cover the whole language Some of everything


How much?

Most words are rare Zipf’s Law To get enough data for most words,

we need very big corpora


Zipf’s Law

Word (pos) r f r x f

the (det) 1 6187267 6187267 to (prep) 10 917579 9175790as (adv) 100 91583 9158300playing (vb) 1000 9738 9738000paint (vb) 2000 4539 9078000amateur (adj) 10,000 741 7410000


Zipf’s Law the: 6%

100 most frequent: 45% 7500 most frequent: 90% all others: rare


Zipf’s Law

0102030405060708090

100

'the' 100 mostfrequent

3500most

frequent

7500most

frequent

% of all texts


Leading English Corpora: Size

109

108

107

106

Size of

Corpora

(in words)

1960s 1970s 1980s 1990s 2000s

Brown/LOB COBUILD BNC OEC


Good news

The web


Thank you

http://www.sketchengine.co.uk

Why We Need Corpora and the Sketch Engine Adam Kilgarriff Lexical Computing Ltd, UK Universities of Leeds and Sussex.

Documents

rare word

cobuild english

list of words

native speaker

common words

cobuild project

grammatical relation

seminarswhich texts