Empirical Research Methods in Computer Science Lecture 1, Part 1 October 12, 2005 Noah Smith nasmith/erm.

Empirical Research Methods in Computer Science

Lecture 1, Part 1October 12, 2005Noah Smithhttp://nlp.cs.jhu.edu/~nasmith/erm

Empiricism

empeiros: experienced (peira = trial or test)

cf. rationalism

Exploration & Experiment

Exploratory Data Analysis (lecture ≈5)

Hypothesis Testing (lectures 1,2)

explorevisualize

summarizemodel

experimentconfirmyes/no?

Computer What?

Theory Algorithms, Computation

Practice Software Engineering,

Application Areas Systems

OS, Architecture

Who cares?

1. anyone who wants to do research2. anyone who wants to follow research

(i.e., read papers)

3. anyone who wants to be able to make smart decisions / draw conclusions

4. anyone who likes thinking critically

Basic Research Questions

Basic Research Questions

int foo() { ...}

Why bother?

int foo() { ...}

int foo() { ...}

int foo() { ...}

int foo() { ...}

int foo() { ...}

int foo() { ...}

Variation → Statistics

int foo() { ...}

determinism isn’t good enough any more!

Statistics, in this Course

Nonparametric tests Sampling

Later: Parametric tests (when and why)

Warning

Theory (complexity analysis, etc.) is important, too!

Many phenomena aren’t surprising if you know your math.

Goals

Know how to look for the interesting experiments

Know how to construct experiments Know how to analyze the results Be critical of all claims

Develop an aesthetic for good empirical work!

Empiricism is FUN!

Especially in computer science!

Basic Course Information

instructors: Noah and David{n,d}[email protected]

Wednesdays 4-5:15 pm no class Thanksgiving week homeworks (65%); final exam

(30%)

About Us

Combined 19 years of experience in CS; 36 years programming

Autodidact empiricists Research interests in statistical

modeling and machine learning (Eisner/Yarowsky lab)

NEB 332

Plan

Hypothesis testing, statistics (2) Case study: runtime (2) Exploratory data analysis (1) Parametric testing, modeling (1-2) Statistical analysis of computer

programs (1)

MO

Come to class. Send us feedback anytime.

What do you want to know? Bring us papers.

Empirical Research Methods in Computer Science

Lecture 1, Part 2October 12, 2005David Smith

Terminological Prelude

Populations Population distributions “All possible files”. How big?

Samples Sampling distributions “Files on my system”

Statistics Functions of data “Size of my files”

Models Parameters

And now for some data

Abnormality

Abnormality

The Bootstrap

Simulates the sampling distribution

Proposed by Efron in 1979 Anticipated by permutation tests,

jackknife, cross-validation From original sample of size n,

draw B samples of size n with replacement and calculate the statistic on each

Sampling Distributions

μ

μ

μ

μμ

Bootstrapping the Mean

What’s Going On?

Why is bootstrap distribution normal?

Remember, this is a mean Linearity of Expectation Central Limit Theorem Closed form standard error for

means

More Heavy Tails

Sampling Still Normal

Bivariate Data

Compression Performance

Bootstrapping Correlation

Error, Confidence, Testing

Standard error from sampling distribution

Confidence intervals: bounding error probability (e.g. to 5%)

Hypothesis testing: how likely is a particular statistic under our assumptions?

Hypothesis Testing

One-sample “Are these data normal/Poisson/…?”

Two-sample “Are these two samples from the

same distribution?” Paired-sample

“Is this technique better than that?”

Your First Assignment

Data compression Three-way tradeoff

Compression Speed Loss

Degenerate cases (cat, echo ‘’, …) Unknown distribution of input