CS3245 Information Retrieval - NUS Computingzhaojin/cs3245_2019/w9.pdf · 2019-03-22 · CS3245 – Information Retrieval Should we use accuracy for evaluation instead? Information

Introduction to Information Retrieval

CS3245

Information Retrieval

Lecture 9: IR Evaluation 9

CS3245 – Information Retrieval

Last TimeThe VSM Reloaded

… optimized for your pleasure!

Improvements to the computation and selection process

Use of heuristics to avoid unnecessary / time consuming computations1. Index elimination 2. Tiered lists3. Early termination 4. Cluster pruning

Information Retrieval 2

Ch. 7


Today: Evaluation How do we know if our results are any good? Benchmarks Precision and Recall; Composite measures Test collection

A/B Testing


Ch. 8


EVALUATING SEARCH ENGINES



Measures for a search engine How fast does it index? Number of documents/hour (Average document size)

How fast does it search? Latency as a function of index size

Expressiveness of query language? Ability to express complex information needs Speed on complex queries

How much does it cost? Fee required to use the search engine


Sec. 8.6


Measures for a search engine All of the preceding criteria are measurable: we can

quantify speed/size we can make expressiveness precise

The key measure: user happiness Speed of response/size of index are factors But blindingly fast, useless answers won’t make a user

happy

Need a way of quantifying user happiness


Sec. 8.6


Measuring user happiness Question: who is the user we are trying to make happy? Depends on the setting

Web engine: Users find what they want and return to the engine next time

Can measure rate of returning users User completes their task – search as a means, not end

eCommerce site: Users find what they want and buy

Is it the end-user, or the eCommerce site, whose happiness we measure? Measure time to purchase, or fraction of searchers who become buyers?


Sec. 8.6.2


Measuring user happiness Enterprise (company/govt/academic): Care about “user productivity”

How much time do my users save when looking for information? Many other criteria having to do with breadth of access, secure

access, etc.


Sec. 8.6.2


Happiness: elusive to measure Most common proxy: relevance of search results But how do you measure relevance?

We’ll examine one method and the issues around it Relevance measurement requires 3 elements:

1. A set document collection2. A set suite of queries3. A usually binary assessment of either Relevant or

Non-relevant for each query and each document Some work on graded relevance, but not the standard


Sec. 8.1


Evaluating an IR system Note: the information need is translated into a

query Relevance is assessed relative to the information

need not the query

E.g., Information need: I'm looking for information on whether drinking red wine is more effective at reducing your risk of heart attacks than white wine.

Query: wine red white heart attack effective

i.e., we evaluate whether the doc addresses the information need, not whether it has these words


Sec. 8.1


Why it’s important:Example Think-Aloud Session


Slide courtesy Google Inc.


Unranked retrieval evaluation:Precision and Recall


Relevant Non-relevant

Retrieved true positive false positive

Not Retrieved false negative true negative

Sec. 8.3

Precision: fraction of retrieved docs that are relevant = P(relevant | retrieved)

Recall: fraction of relevant docs that are retrieved = P(retrieved | relevant)

Precision P = tp / (tp + fp)Recall R = tp / (tp + fn)


Should we use accuracy forevaluation instead?


Sec. 8.3

Given a query, a Boolean engine classifies each doc as Relevant or Non-Relevant

The accuracy of an engine: the fraction of these classifications that are correct Accuracy = (tp + tn) / (tp + fp + tn + fn)

Accuracy is a commonly used evaluation measure in classification (e.g., HW1)

Quick Question: Why is this not a very useful evaluation measure in IR?


Precision/Recall You can get high recall (but low precision) by

retrieving all docs for all queries! Recall is a non-decreasing function of the number

of docs retrieved

In a good system, precision decreases as either the number of docs retrieved or recall increases This is not a theorem, but a result with strong empirical

confirmation


Sec. 8.3


Difficulties in using precision/recall Should average over large document

collection/query ensembles Need human relevance assessments But people are subjective; they aren’t reliable assessors

Assessments have to be binary Can we give graded assessments?

Heavily skewed by collection/queries pairing Results may not translate from one collection to another


Sec. 8.3

We’ll return to this point later


A combined measure: F


Sec. 8.3

Combined measure that assesses precision / recall tradeoff is F measure (weighted harmonic mean):

𝐹𝐹 = 1𝛼𝛼1𝑃𝑃+(1−𝛼𝛼)

1𝑅𝑅

= 𝛽𝛽2+1 𝑃𝑃𝑃𝑃𝛽𝛽2𝑃𝑃+𝑃𝑃

People usually use balanced F1 measure i.e., with β = 1 or α = 1

2

Harmonic mean is a conservative average

2𝑃𝑃𝑃𝑃𝑃𝑃 + 𝑃𝑃



Sec. 8.3

Combined Measures

0

20

40

60

80

100

0 20 40 60 80 100

Precision (Recall fixed at 70%)

MinimumMaximumArithmeticGeometricHarmonic

Co

mb

ined

Avera

ge

Blanks on slides, you may want to fill in

Chart1

00000

11111

22222

33333

44444

55555

66666

77777

88888

99999

1010101010

1111111111

1212121212

1313131313

1414141414

1515151515

1616161616

1717171717

1818181818

1919191919

2020202020

2121212121

2222222222

2323232323

2424242424

2525252525

2626262626

2727272727

2828282828

2929292929

3030303030

3131313131

3232323232

3333333333

3434343434

3535353535

3636363636

3737373737

3838383838

3939393939

4040404040

4141414141

4242424242

4343434343

4444444444

4545454545

4646464646

4747474747

4848484848

4949494949

5050505050

5151515151

5252525252

5353535353

5454545454

5555555555

5656565656

5757575757

5858585858

5959595959

6060606060

6161616161

6262626262

6363636363

6464646464

6565656565

6666666666

6767676767

6868686868

6969696969

7070707070

7171717171

7272727272

7373737373

7474747474

7575757575

7676767676

7777777777

7878787878

7979797979

8080808080

8181818181

8282828282

8383838383

8484848484

8585858585

8686868686

8787878787

8888888888

8989898989

9090909090

9191919191

9292929292

9393939393

9494949494

9595959595

9696969696

9797979797

9898989898

9999999999

100100100100100

Minimum

Maximum

Arithmetic

Geometric

Harmonic


Combined Measures

0

70

35

0

0

1

70

35.5

8.3666002653

1.9718309859

2

70

36

11.8321595662

3.8888888889

3

70

36.5

14.4913767462

5.7534246575

4

70

37

16.7332005307

7.5675675676

5

70

37.5

18.7082869339

9.3333333333

6

70

38

20.4939015319

11.0526315789

7

70

38.5

22.1359436212

12.7272727273

8

70

39

23.6643191324

14.358974359

9

70

39.5

25.099800796

15.9493670886

10

70

40

26.4575131106

17.5

11

70

40.5

27.748873851

19.012345679

12

70

41

28.9827534924

20.487804878

13

70

41.5

30.166206258

21.9277108434

14

70

42

31.304951685

23.3333333333

15

70

42.5

32.403703492

24.7058823529

16

70

43

33.4664010614

26.0465116279

17

70

43.5

34.4963766213

27.3563218391

18

70

44

35.4964786986

28.6363636364

19

70

44.5

36.4691650576

29.8876404494

20

70

45

37.4165738677

31.1111111111

21

70

45.5

38.3405790254

32.3076923077

22

70

46

39.2428337407

33.4782608696

23

70

46.5

40.1248052955

34.623655914

24

70

47

40.9878030638

35.7446808511

25

70

47.5

41.8330013267

36.8421052632

26

70

48

42.6614580154

37.9166666667

27

70

48.5

43.4741302386

38.9690721649

28

70

49

44.2718872424

40

29

70

49.5

45.0555213043

41.0101010101

30

70

50

45.8257569496

42

31

70

50.5

46.5832587954

42.9702970297

32

70

51

47.3286382648

43.9215686275

33

70

51.5

48.0624593628

44.854368932

34

70

52

48.7852436706

45.7692307692

35

70

52.5

49.4974746831

46.6666666667

36

70

53

50.199601592

47.5471698113

37

70

53.5

50.8920426

48.4112149533

38

70

54

51.5751878329

49.2592592593

39

70

54.5

52.2494019105

50.0917431193

40

70

55

52.9150262213

50.9090909091

41

70

55.5

53.5723809439

51.7117117117

42

70

56

54.2217668469

52.5

43

70

56.5

54.8634668974

53.2743362832

44

70

57

55.497747702

54.0350877193

45

70

57.5

56.1248608016

54.7826086957

46

70

58

56.7450438364

55.5172413793

47

70

58.5

57.3585215988

56.2393162393

48

70

59

57.9655069848

56.9491525424

49

70

59.5

58.5662018574

57.6470588235

50

70

60

59.160797831

58.3333333333

51

70

60.5

59.7494769852

59.0082644628

52

70

61

60.332412516

59.6721311475

53

70

61.5

60.9097693314

60.325203252

54

70

62

61.4817045958

60.9677419355

55

70

62.5

62.04836823

61.6

56

70

63

62.60990337

62.2222222222

57

70

63.5

63.1664467894

62.8346456693

58

70

64

63.7181292883

63.4375

59

70

64.5

64.2650760522

64.0310077519

60

70

65

64.8074069841

64.6153846154

61

70

65.5

65.3452370108

65.1908396947

62

70

66

65.878676368

65.7575757576

63

70

66.5

66.4078308635

66.3157894737

64

70

67

66.9328021227

66.8656716418

65

70

67.5

67.4536878162

67.4074074074

66

70

68

67.9705818719

67.9411764706

67

70

68.5

68.4835746731

68.4671532847

68

70

69

68.9927532426

68.9855072464

69

70

69.5

69.4982014156

69.4964028777

70

70

70

70

70

70

71

70.5

70.4982269281

70.4964539007

70

72

71

70.9929573972

70.985915493

70

73

71.5

71.4842640026

71.4685314685

70

74

72

71.9722168618

71.9444444444

70

75

72.5

72.4568837309

72.4137931034

70

76

73

72.9383301152

72.8767123288

70

77

73.5

73.4166193719

73.3333333333

70

78

74

73.8918128076

73.7837837838

70

79

74.5

74.3639697703

74.2281879195

70

80

75

74.8331477355

74.6666666667

70

81

75.5

75.2994023881

75.0993377483

70

82

76

75.7627876995

75.5263157895

70

83

76.5

76.2233560006

75.9477124183

70

84

77

76.6811580507

76.3636363636

70

85

77.5

77.1362431027

76.7741935484

70

86

78

77.5886589651

77.1794871795

70

87

78.5

78.0384520605

77.5796178344

70

88

79

78.4856674814

77.9746835443

70

89

79.5

78.9303490427

78.3647798742

70

90

80

79.3725393319

78.75

70

91

80.5

79.8122797569

79.1304347826

70

92

81

80.249610591

79.5061728395

70

93

81.5

80.6845710158

79.8773006135

70

94

82

81.1171991627

80.243902439

70

95

82.5

81.5475321515

80.6060606061

70

96

83

81.9756061277

80.9638554217

70

97

83.5

82.4014562978

81.3173652695

70

98

84

82.8251169634

81.6666666667

70

99

84.5

83.2466215531

82.0118343195

70

100

85

83.6660026534

82.3529411765

Sheet1

PrecisionRecallMinimumMaximumArithmeticGeometricHarmonic

0700703500

14,000,000.0017017035.58.36660026531.9718309859

2702703611.83215956623.8888888889

37037036.514.49137674625.7534246575

4704703716.73320053077.5675675676

57057037.518.70828693399.3333333333

6706703820.493901531911.0526315789

77077038.522.135943621212.7272727273

8708703923.664319132414.358974359

97097039.525.09980079615.9493670886

107010704026.457513110617.5

1170117040.527.74887385119.012345679

127012704128.982753492420.487804878

1370137041.530.16620625821.9277108434

147014704231.30495168523.3333333333

1570157042.532.40370349224.7058823529

167016704333.466401061426.0465116279

1770177043.534.496376621327.3563218391

187018704435.496478698628.6363636364

1970197044.536.469165057629.8876404494

207020704537.416573867731.1111111111

2170217045.538.340579025432.3076923077

227022704639.242833740733.4782608696

2370237046.540.124805295534.623655914

247024704740.987803063835.7446808511

2570257047.541.833001326736.8421052632

267026704842.661458015437.9166666667

2770277048.543.474130238638.9690721649

287028704944.271887242440

2970297049.545.055521304341.0101010101

307030705045.825756949642

3170317050.546.583258795442.9702970297

327032705147.328638264843.9215686275

3370337051.548.062459362844.854368932

347034705248.785243670645.7692307692

3570357052.549.497474683146.6666666667

367036705350.19960159247.5471698113

3770377053.550.892042648.4112149533

387038705451.575187832949.2592592593

3970397054.552.249401910550.0917431193

407040705552.915026221350.9090909091

4170417055.553.572380943951.7117117117

427042705654.221766846952.5

4370437056.554.863466897453.2743362832

447044705755.49774770254.0350877193

4570457057.556.124860801654.7826086957

467046705856.745043836455.5172413793

4770477058.557.358521598856.2393162393

487048705957.965506984856.9491525424

4970497059.558.566201857457.6470588235

507050706059.16079783158.3333333333

5170517060.559.749476985259.0082644628

527052706160.33241251659.6721311475

5370537061.560.909769331460.325203252

547054706261.481704595860.9677419355

5570557062.562.0483682361.6

567056706362.6099033762.2222222222

5770577063.563.166446789462.8346456693

587058706463.718129288363.4375

5970597064.564.265076052264.0310077519

607060706564.807406984164.6153846154

6170617065.565.345237010865.1908396947

627062706665.87867636865.7575757576

6370637066.566.407830863566.3157894737

647064706766.932802122766.8656716418

6570657067.567.453687816267.4074074074

667066706867.970581871967.9411764706

6770677068.568.483574673168.4671532847

687068706968.992753242668.9855072464

6970697069.569.498201415669.4964028777

70707070707070

7170707170.570.498226928170.4964539007

727070727170.992957397270.985915493

7370707371.571.484264002671.4685314685

747070747271.972216861871.9444444444

7570707572.572.456883730972.4137931034

767070767372.938330115272.8767123288

7770707773.573.416619371973.3333333333

787070787473.891812807673.7837837838

7970707974.574.363969770374.2281879195

807070807574.833147735574.6666666667

8170708175.575.299402388175.0993377483

827070827675.762787699575.5263157895

8370708376.576.223356000675.9477124183

847070847776.681158050776.3636363636

8570708577.577.136243102776.7741935484

867070867877.588658965177.1794871795

8770708778.578.038452060577.5796178344

887070887978.485667481477.9746835443

8970708979.578.930349042778.3647798742

907070908079.372539331978.75

9170709180.579.812279756979.1304347826

927070928180.24961059179.5061728395

9370709381.580.684571015879.8773006135

947070948281.117199162780.243902439

9570709582.581.547532151580.6060606061

967070968381.975606127780.9638554217

9770709783.582.401456297881.3173652695

987070988482.825116963481.6666666667

9970709984.583.246621553182.0118343195

10070701008583.666002653482.3529411765

Sheet1

Minimum

Maximum

Arithmetic

Geometric

Harmonic


Combined Measures

Sheet2

Sheet3


Evaluating ranked results Evaluation of ranked results: The relevant documents should be ranked higher than the

non relevant documents By taking various numbers of the top returned documents

(levels of recall), we can produce a precision-recall curve


Sec. 8.4


A precision-recall curve

0.0

0.2

0.4

0.6

0.8

1.0

0.0 0.2 0.4 0.6 0.8 1.0Recall

Pre

cisi

on


Sec. 8.4

Chart1

0.02500.575

0.0250.025

0.050.025

0.0750.05

0.0750.075

0.10.075

0.10.1

0.10.1

0.10.1

0.1250.1

0.150.125

0.150.15

0.1750.15

0.20.175

0.2250.2

0.250.225

0.250.25

0.250.25

0.2750.25

0.2750.275

0.2750.275

0.30.275

0.30.3

0.30.3

0.3250.3

0.3250.325

0.3250.325

0.3250.325

0.3250.325

0.350.325

0.350.35

0.350.35

0.3750.35

0.3750.375

0.3750.375

0.3750.375

0.40.375

0.4250.4

0.4250.425

0.4250.425

0.4250.425

0.450.425

0.450.45

0.450.45

0.450.45

0.4750.45

0.4750.475

0.4750.475

0.50.475

0.50.5

0.50.5

0.50.5

0.50.5

0.5250.5

0.5250.525

0.5250.525

0.5250.525

0.550.525

0.550.55

0.550.55

0.550.55

0.5750.55

0.5750.575

0.5750.575

0.5750.575

0.5750.575

0.60.575

0.60.6

0.60.6

0.60.6

0.60.6

0.60.6

0.60.6

0.60.6

0.6250.6

0.6250.625

0.6250.625

0.6250.625

0.6250.625

0.6250.625

0.6250.625

0.6250.625

0.650.625

0.650.65

0.650.65

0.6750.65

0.6750.675

0.6750.675

0.6750.675

0.6750.675

0.6750.675

0.6750.675

0.6750.675

0.6750.675

0.6750.675

0.6750.675

0.6750.675

0.70.675

0.70.7

0.70.7

0.70.7

0.70.7

0.70.7

0.70.7

0.70.7

0.70.7

0.70.7

0.70.7

0.70.7

0.7250.7

0.7250.725

0.7250.725

0.7250.725

0.7250.725

0.7250.725

0.7250.725

0.7250.725

0.7250.725

0.7250.725

0.7250.725

0.7250.725

0.7250.725

0.7250.725

0.7250.725

0.7250.725

0.7250.725

0.7250.725

0.750.725

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.750.75

0.7750.75

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.7750.775

0.80.775

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.80.8

0.8250.8

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.8250.825

0.850.825

0.850.85

0.850.85

0.850.85

0.850.85

0.850.85

0.850.85

0.850.85

0.850.85

0.850.85

0.850.85

0.850.85

0.850.85

0.850.85

0.850.85

0.850.85

0.850.85

0.850.85

0.850.85

0.850.85

0.850.85

0.850.85

0.850.85

0.850.85

0.850.85

0.8750.85

0.8750.875

0.90.875

0.90.9

0.90.9

0.90.9

0.90.9

0.90.9

0.90.9

0.90.9

0.90.9

0.90.9

0.90.9

0.90.9

0.90.9

0.90.9

0.90.9

0.90.9

0.9250.9

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.9250.925

0.950.925

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.950.95

0.9750.95

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

0.9750.975

10.975

11

11

11

11

11

11

11

11

11

11

11

11

11

1

Precison

interp prec

F1

Recall

Precision

1

1

0.3709677419

0.5

1

0.6666666667

0.75

0.75

0.75

0.6

0.75

0.6666666667

0.6666666667

0.5714285714

0.6666666667

0.5

0.625

0.4444444444

0.625

0.5

0.625

0.5454545455

0.625

0.5

0.625

0.5384615385

0.625

0.5714285714

0.625

0.6

0.625

0.625

0.625

0.5882352941

0.625

0.5555555556

0.5882352941

0.5789473684

0.5789473684

0.55

0.5789473684

0.5238095238

0.55

0.5454545455

0.5454545455

0.5217391304

0.5454545455

0.5

0.5217391304

0.52

0.52

0.5

0.52

0.4814814815

0.5

0.4642857143

0.4814814815

0.4482758621

0.4666666667

0.4666666667

0.4666666667

0.4516129032

0.4666666667

0.4375

0.4545454545

0.4545454545

0.4545454545

0.4411764706

0.4545454545

0.4285714286

0.4473684211

0.4166666667

0.4473684211

0.4324324324

0.4473684211

0.4473684211

0.4473684211

0.4358974359

0.4473684211

0.425

0.4358974359

0.4146341463

0.4285714286

0.4285714286

0.4285714286

0.4186046512

0.4285714286

0.4090909091

0.4186046512

0.4

0.4130434783

0.4130434783

0.4130434783

0.4042553191

0.4130434783

0.3958333333

0.4081632653

0.4081632653

0.4081632653

0.4

0.4081632653

0.3921568627

0.4

0.3846153846

0.3921568627

0.3773584906

0.3888888889

0.3888888889

0.3888888889

0.3818181818

0.3888888889

0.375

0.3818181818

0.3684210526

0.3793103448

0.3793103448

0.3793103448

0.3728813559

0.3793103448

0.3666666667

0.3728813559

0.3606557377

0.3709677419

0.3709677419

0.3709677419

0.3650793651

0.3709677419

0.359375

0.3650793651

0.3538461538

0.359375

0.3484848485

0.3582089552

0.3582089552

0.3582089552

0.3529411765

0.3582089552

0.347826087

0.3529411765

0.3428571429

0.347826087

0.338028169

0.3428571429

0.3333333333

0.338028169

0.3287671233

0.3333333333

0.3243243243

0.3333333333

0.3333333333

0.3333333333

0.3289473684

0.3333333333

0.3246753247

0.3289473684

0.3205128205

0.3246753247

0.3164556962

0.3205128205

0.3125

0.3164556962

0.3086419753

0.3139534884

0.3048780488

0.3139534884

0.313253012

0.3139534884

0.3095238095

0.3139534884

0.3058823529

0.3139534884

0.3139534884

0.3139534884

0.3103448276

0.3139534884

0.3068181818

0.3103448276

0.3033707865

0.3068181818

0.3

0.3033707865

0.2967032967

0.3

0.2934782609

0.2967032967

0.2903225806

0.2934782609

0.2872340426

0.2903225806

0.2842105263

0.2872340426

0.28125

0.2857142857

0.2783505155

0.2857142857

0.2857142857

0.2857142857

0.2828282828

0.2857142857

0.28

0.2828282828

0.2772277228

0.28

0.2745098039

0.2772277228

0.2718446602

0.2745098039

0.2692307692

0.2718446602

0.2666666667

0.2692307692

0.2641509434

0.2666666667

0.261682243

0.2641509434

0.2592592593

0.2636363636

0.2568807339

0.2636363636

0.2636363636

0.2636363636

0.2612612613

0.2636363636

0.2589285714

0.2612612613

0.2566371681

0.2589285714

0.2543859649

0.2566371681

0.252173913

0.2543859649

0.25

0.252173913

0.2478632479

0.25

0.2457627119

0.2478632479

0.243697479

0.2457627119

0.2416666667

0.243697479

0.2396694215

0.2416666667

0.237704918

0.2396694215

0.2357723577

0.237704918

0.2338709677

0.2357723577

0.232

0.234375

0.2301587302

0.234375

0.2283464567

0.234375

0.234375

0.234375

0.2325581395

0.234375

0.2307692308

0.2325581395

0.2290076336

0.2307692308

0.2272727273

0.2290076336

0.2255639098

0.2272727273

0.223880597

0.2255639098

0.2222222222

0.223880597

0.2205882353

0.2222222222

0.2189781022

0.2205882353

0.2173913043

0.2189781022

0.2158273381

0.2173913043

0.2142857143

0.2158273381

0.2127659574

0.2142857143

0.2112676056

0.2127659574

0.2097902098

0.2112676056

0.2083333333

0.2097902098

0.2068965517

0.2083333333

0.2054794521

0.2068965517

0.2040816327

0.2054794521

0.2027027027

0.2040816327

0.2013422819

0.2027027027

0.2

0.2013422819

0.1986754967

0.2

0.1973684211

0.1986754967

0.1960784314

0.1973684211

0.1948051948

0.1960784314

0.1935483871

0.1948051948

0.1923076923

0.1935483871

0.1910828025

0.1923076923

0.1898734177

0.1910828025

0.1886792453

0.1898734177

0.1875

0.1886792453

0.1863354037

0.1875

0.1851851852

0.1863354037

0.1840490798

0.1851851852

0.1829268293

0.1840490798

0.1818181818

0.1829268293

0.1807228916

0.1818181818

0.1796407186

0.1807228916

0.1785714286

0.1802325581

0.1775147929

0.1802325581

0.1764705882

0.1802325581

0.1754385965

0.1802325581

0.1802325581

0.1802325581

0.1791907514

0.1802325581

0.1781609195

0.1791907514

0.1771428571

0.1781609195

0.1761363636

0.1771428571

0.1751412429

0.1761363636

0.1741573034

0.1751412429

0.1731843575

0.1741573034

0.1722222222

0.1731843575

0.1712707182

0.1722222222

0.1703296703

0.1712707182

0.1693989071

0.1703296703

0.1684782609

0.1693989071

0.1675675676

0.1684782609

0.1666666667

0.1675675676

0.1657754011

0.1666666667

0.164893617

0.1657754011

0.164021164

0.164893617

0.1631578947

0.164021164

0.1623036649

0.1631578947

0.1614583333

0.1623036649

0.1606217617

0.1614583333

0.1597938144

0.1606217617

0.158974359

0.1597938144

0.1581632653

0.158974359

0.1573604061

0.1581632653

0.1565656566

0.1573604061

0.1557788945

0.1565656566

0.155

0.1557788945

0.1542288557

0.155

0.1534653465

0.1542288557

0.1527093596

0.1534653465

0.1519607843

0.1527093596

0.1512195122

0.1519607843

0.1504854369

0.1512195122

0.1497584541

0.1504854369

0.1490384615

0.1497584541

0.1483253589

0.1490384615

0.1476190476

0.1483253589

0.1469194313

0.1476190476

0.1462264151

0.1469194313

0.1455399061

0.1462264151

0.1448598131

0.1455399061

0.1441860465

0.1448598131

0.1435185185

0.1441860465

0.1428571429

0.1435185185

0.1422018349

0.1428571429

0.1415525114

0.1422018349

0.1409090909

0.1415525114

0.1402714932

0.1409090909

0.1396396396

0.1402714932

0.1390134529

0.1396396396

0.1383928571

0.1390134529

0.1377777778

0.1383928571

0.1371681416

0.1377777778

0.1365638767

0.1371681416

0.1359649123

0.1365638767

0.135371179

0.1359649123

0.1347826087

0.135371179

0.1341991342

0.1347826087

0.1336206897

0.1341991342

0.1330472103

0.1338912134

0.1324786325

0.1338912134

0.1319148936

0.1338912134

0.1313559322

0.1338912134

0.1308016878

0.1338912134

0.1302521008

0.1338912134

0.1338912134

0.1338912134

0.1333333333

0.1338912134

0.132780083

0.1333333333

0.132231405

0.132780083

0.1316872428

0.132231405

0.131147541

0.1316872428

0.1306122449

0.131147541

0.1300813008

0.1306122449

0.1295546559

0.1300813008

0.1290322581

0.1295546559

0.1285140562

0.1290322581

0.128

0.1285140562

0.1274900398

0.128

0.126984127

0.1274900398

0.1264822134

0.126984127

0.125984252

0.1264822134

0.1254901961

0.125984252

0.125

0.1254901961

0.1245136187

0.125

0.1240310078

0.1245136187

0.1235521236

0.1240310078

0.1230769231

0.1235521236

0.122605364

0.1230769231

0.1221374046

0.122605364

0.1216730038

0.1221374046

0.1212121212

0.1216730038

0.120754717

0.1212121212

0.1203007519

0.120754717

0.1198501873

0.1203007519

0.1194029851

0.1198501873

0.1189591078

0.1194029851

0.1185185185

0.1189591078

0.1180811808

0.118705036

0.1176470588

0.118705036

0.1172161172

0.118705036

0.1167883212

0.118705036

0.1163636364

0.118705036

0.115942029

0.118705036

0.1155234657

0.118705036

0.118705036

0.118705036

0.1182795699

0.118705036

0.1178571429

0.1182795699

0.1174377224

0.1178571429

0.1170212766

0.1174377224

0.1166077739

0.1170212766

0.1161971831

0.1166077739

0.1157894737

0.1161971831

0.1153846154

0.1157894737

0.1149825784

0.1153846154

0.1145833333

0.1149825784

0.1141868512

0.1145833333

0.1137931034

0.1141868512

0.1134020619

0.1137931034

0.1130136986

0.1134020619

0.1126279863

0.1130136986

0.112244898

0.1126279863

0.1118644068

0.112244898

0.1114864865

0.1118644068

0.1111111111

0.1114864865

0.110738255

0.1111111111

0.110367893

0.110738255

0.11

0.110367893

0.1096345515

0.11

0.1092715232

0.1096345515

0.1089108911

0.1092715232

0.1085526316

0.1089108911

0.1081967213

0.1085526316

0.1078431373

0.1081967213

0.1074918567

0.1078431373

0.1071428571

0.1074918567

0.1067961165

0.1071428571

0.1064516129

0.1067961165

0.1061093248

0.1064516129

0.1057692308

0.1061093248

0.1054313099

0.1057692308

0.1050955414

0.1055900621

0.1047619048

0.1055900621

0.1044303797

0.1055900621

0.1041009464

0.1055900621

0.1037735849

0.1055900621

0.1034482759

0.1055900621

0.103125

0.1055900621

0.1028037383

0.1055900621

0.1055900621

0.1055900621

0.1052631579

0.1055900621

0.1049382716

0.1052631579

0.1046153846

0.1049382716

0.1042944785

0.1046153846

0.1039755352

0.1042944785

0.1036585366

0.1039755352

0.103343465

0.1036585366

0.103030303

0.103343465

0.1027190332

0.1031518625

0.1024096386

0.1031518625

0.1021021021

0.1031518625

0.1017964072

0.1031518625

0.1014925373

0.1031518625

0.1011904762

0.1031518625

0.1008902077

0.1031518625

0.100591716

0.1031518625

0.1002949853

0.1031518625

0.1

0.1031518625

0.0997067449

0.1031518625

0.0994152047

0.1031518625

0.0991253644

0.1031518625

0.0988372093

0.1031518625

0.0985507246

0.1031518625

0.098265896

0.1031518625

0.1008645533

0.1031518625

0.1005747126

0.1031518625

0.1031518625

0.1031518625

0.1028571429

0.1031518625

0.1025641026

0.1028571429

0.1022727273

0.1025641026

0.1019830028

0.1022727273

0.1016949153

0.1019830028

0.1014084507

0.1016949153

0.1011235955

0.1014084507

0.1008403361

0.101369863

0.1005586592

0.101369863

0.1002785515

0.101369863

0.1

0.101369863

0.0997229917

0.101369863

0.0994475138

0.101369863

0.0991735537

0.101369863

0.0989010989

0.101369863

0.101369863

0.101369863

0.1010928962

0.101369863

0.1008174387

0.1010928962

0.1005434783

0.1008174387

0.1002710027

0.1005434783

0.1

0.1002710027

0.0997304582

0.1

0.0994623656

0.0997304582

0.0991957105

0.0994623656

0.0989304813

0.0991957105

0.0986666667

0.0989304813

0.0984042553

0.0986666667

0.0981432361

0.0984042553

0.0978835979

0.0981432361

0.0976253298

0.0978835979

0.0973684211

0.0976253298

0.0971128609

0.0973684211

0.0968586387

0.0971128609

0.0966057441

0.0968586387

0.0963541667

0.0966057441

0.0961038961

0.0963541667

0.0958549223

0.0961038961

0.0956072351

0.0958549223

0.0953608247

0.0956072351

0.0951156812

0.0953608247

0.0948717949

0.0951156812

0.094629156

0.0948717949

0.0943877551

0.094629156

0.0941475827

0.0943877551

0.0939086294

0.0941475827

0.0936708861

0.0939086294

0.0934343434

0.0936708861

0.0931989924

0.0934343434

0.0929648241

0.0931989924

0.0927318296

0.0929648241

0.0925

0.0927318296

0.0922693267

0.0925

0.092039801

0.0922693267

0.0918114144

0.092039801

0.0915841584

0.0918114144

0.0913580247

0.0915841584

0.0911330049

0.0913580247

0.0909090909

0.0911330049

0.0906862745

0.0909090909

0.0904645477

0.0906862745

0.0902439024

0.0904645477

0.0900243309

0.0902439024

0.0898058252

0.0900243309

0.0895883777

0.0898058252

0.0893719807

0.0896226415

0.0891566265

0.0896226415

0.0889423077

0.0896226415

0.0887290168

0.0896226415

0.0885167464

0.0896226415

0.0883054893

0.0896226415

0.0880952381

0.0896226415

0.0878859857

0.0896226415

0.0876777251

0.0896226415

0.0874704492

0.0896226415

0.0896226415

0.0896226415

0.0894117647

0.0896226415

0.0892018779

0.0894117647

0.0889929742

0.0892018779

0.0887850467

0.0889929742

0.0885780886

0.0887850467

0.088372093

0.0885780886

0.0881670534

0.088372093

0.087962963

0.0881670534

0.0877598152

0.087962963

0.0875576037

0.0877598152

0.0873563218

0.0875576037

0.0871559633

0.0873563218

0.0869565217

0.0871559633

0.0867579909

0.0869565217

0.0865603645

0.0867579909

0.0863636364

0.0865603645

0.0861678005

0.0863636364

0.0859728507

0.0861678005

0.085778781

0.0859728507

0.0855855856

0.085778781

0.0853932584

0.0855855856

0.0852017937

0.0853932584

0.0850111857

0.0852017937

0.0848214286

0.0850111857

0.0846325167

0.0848214286

0.0844444444

0.0846325167

0.0842572062

0.0845986985

0.0840707965

0.0845986985

0.0838852097

0.0845986985

0.0837004405

0.0845986985

0.0835164835

0.0845986985

0.0833333333

0.0845986985

0.0831509847

0.0845986985

0.0829694323

0.0845986985

0.082788671

0.0845986985

0.0826086957

0.0845986985

0.0845986985

0.0845986985

0.0844155844

0.0845986985

0.0842332613

0.0844155844

0.0840517241

0.0842332613

0.0838709677

0.0840517241

0.0836909871

0.0838709677

0.0835117773

0.0836909871

0.0833333333

0.0835117773

0.0831556503

0.0833333333

0.0829787234

0.0831556503

0.0828025478

0.0829787234

0.0826271186

0.0828025478

0.0824524313

0.0826271186

0.082278481

0.0824524313

0.0821052632

0.082278481

0.0819327731

0.0821355236

0.0817610063

0.0821355236

0.0815899582

0.0821355236

0.0814196242

0.0821355236

0.08125

0.0821355236

0.0810810811

0.0821355236

0.0809128631

0.0821355236

0.0807453416

0.0821355236

0.0805785124

0.0821355236

0.0804123711

0.0821355236

0.0802469136

0.0821355236

0.0821355236

0.0821355236

0.0819672131

0.0821355236

0.081799591

0.0819672131

0.0816326531

0.081799591

0.0814663951

0.0816326531

0.081300813

0.0814663951

0.0811359026

0.081300813

0.0809716599

0.0811359026

0.0808080808

0.0809716599

0.0806451613

0.0808080808

0.0804828974

0.0806451613

0.0803212851

0.0804828974

0.0801603206

0.0803212851

0.08

0.0801603206

0.08

Sheet1

RelevantPrecisonRecallF1Fb=0.33Fb=3interp precSpecificity1-SpecRecInterp prec

001.0001.00000.01.00

111.001.00000.02500.04880.20410.02771.0000100.10.67

201.000.50000.02500.04760.17240.02760.75000.9979959920.0020040080.20.63

312.000.66670.05000.09300.29850.05510.75000.99799196790.00200803210.30.55

413.000.75000.07500.13640.39470.08240.75000.99798792760.00201207240.40.45

503.000.60000.07500.13330.35290.08220.66670.99597585510.00402414490.50.41

614.000.66670.10000.17390.42550.10930.66670.99596774190.00403225810.60.36

704.000.57140.10000.17020.38830.10900.62500.99395161290.00604838710.70.29

804.000.50000.10000.16670.35710.10870.62500.99193548390.00806451610.80.13

904.000.44440.10000.16330.33060.10840.62500.98991935480.01008064520.90.10

1015.000.50000.12500.20000.38460.13510.62500.98989898990.01010101011.00.08

1116.000.54550.15000.23530.43170.16170.62500.98987854250.0101214575

1206.000.50000.15000.23080.40540.16130.62500.9878542510.012145749

1317.000.53850.17500.26420.44590.18770.62500.98782961460.0121703854

1418.000.57140.20000.29630.48190.21390.62500.9878048780.012195122

1519.000.60000.22500.32730.51430.24000.62500.98778004070.0122199593

16110.000.62500.25000.35710.54350.26600.62500.9877551020.012244898

17010.000.58820.25000.35090.51810.26530.58820.98571428570.0142857143

18010.000.55560.25000.34480.49500.26460.57890.98367346940.0163265306

19111.000.57890.27500.37290.52130.29020.57890.98364008180.0163599182

20011.000.55000.27500.36670.50000.28950.55000.9815950920.018404908

21011.000.52380.27500.36070.48030.28870.54550.97955010220.0204498978

22112.000.54550.30000.38710.50420.31410.54550.97950819670.0204918033

23012.000.52170.30000.38100.48580.31330.52170.97745901640.0225409836

24012.000.50000.30000.37500.46880.31250.52000.97540983610.0245901639

25113.000.52000.32500.40000.49060.33770.52000.97535934290.0246406571

26013.000.50000.32500.39390.47450.33680.50000.97330595480.0266940452

27013.000.48150.32500.38810.45940.33590.48150.97125256670.0287474333

28013.000.46430.32500.38240.44520.33510.46670.96919917860.0308008214

29013.000.44830.32500.37680.43190.33420.46670.96714579060.0328542094

30114.000.46670.35000.40000.45160.35900.46670.96707818930.0329218107

31014.000.45160.35000.39440.43890.35810.45450.96502057610.0349794239

32014.000.43750.35000.38890.42680.35710.45450.9629629630.037037037

33115.000.45450.37500.41100.44510.38170.45450.96288659790.0371134021

34015.000.44120.37500.40540.43350.38070.44740.96082474230.0391752577

35015.000.42860.37500.40000.42250.37970.44740.95876288660.0412371134

36015.000.41670.37500.39470.41210.37880.44740.95670103090.0432989691

37116.000.43240.40000.41560.42900.40300.44740.95661157020.0433884298

38117.000.44740.42500.43590.44500.42710.44740.95652173910.0434782609

39017.000.43590.42500.43040.43480.42610.43590.95445134580.0455486542

40017.000.42500.42500.42500.42500.42500.42860.95238095240.0476190476

41017.000.41460.42500.41980.41560.42390.42860.9503105590.049689441

42118.000.42860.45000.43900.43060.44780.42860.95020746890.0497925311

43018.000.41860.45000.43370.42150.44670.41860.94813278010.0518672199

44018.000.40910.45000.42860.41280.44550.41300.94605809130.0539419087

45018.000.40000.45000.42350.40450.44440.41300.94398340250.0560165975

46119.000.41300.47500.44190.41850.46800.41300.94386694390.0561330561

47019.000.40430.47500.43680.41040.46680.40820.94178794180.0582120582

48019.000.39580.47500.43180.40250.46570.40820.93970893970.0602910603

49120.000.40820.50000.44940.41580.48900.40820.93958333330.0604166667

50020.000.40000.50000.44440.40820.48780.40000.93750.0625

51020.000.39220.50000.43960.40080.48660.39220.93541666670.0645833333

52020.000.38460.50000.43480.39370.48540.38890.93333333330.0666666667

53020.000.37740.50000.43010.38680.48430.38890.931250.06875

54121.000.38890.52500.44680.39920.50720.38890.93110647180.0688935282

55021.000.38180.52500.44210.39250.50600.38180.92901878910.0709812109

56021.000.37500.52500.43750.38600.50480.37930.92693110650.0730688935

57021.000.36840.52500.43300.37970.50360.37930.92484342380.0751565762

58122.000.37930.55000.44900.39150.52630.37930.92468619250.0753138075

59022.000.37290.55000.44440.38530.52510.37290.92259414230.0774058577

60022.000.36670.55000.44000.37930.52380.37100.92050209210.0794979079

61022.000.36070.55000.43560.37350.52260.37100.91841004180.0815899582

62123.000.37100.57500.45100.38460.54500.37100.91823899370.0817610063

63023.000.36510.57500.44660.37890.54370.36510.91614255770.0838574423

64023.000.35940.57500.44230.37340.54250.35940.91404612160.0859538784

65023.000.35380.57500.43810.36800.54120.35820.91194968550.0880503145

66023.000.34850.57500.43400.36280.53990.35820.90985324950.0901467505

67124.000.35820.60000.44860.37330.56210.35820.90966386550.0903361345

68024.000.35290.60000.44440.36810.56070.35290.90756302520.0924369748

69024.000.34780.60000.44040.36310.55940.34780.90546218490.0945378151

70024.000.34290.60000.43640.35820.55810.34290.90336134450.0966386555

71024.000.33800.60000.43240.35350.55680.33800.90126050420.0987394958

72024.000.33330.60000.42860.34880.55560.33330.89915966390.1008403361

73024.000.32880.60000.42480.34430.55430.33330.89705882350.1029411765Example 11pt precision (SabIR/Cornell 8A1) from TREC 8 (1999)

74024.000.32430.60000.42110.33990.55300.33330.89495798320.1050420168Recall LevelAve. Precision

75125.000.33330.62500.43480.34970.57470.33330.89473684210.105263157900.736

76025.000.32890.62500.43100.34530.57340.32890.89263157890.10736842110.10.5107

77025.000.32470.62500.42740.34110.57210.32470.89052631580.10947368420.20.4059

78025.000.32050.62500.42370.33690.57080.32050.88842105260.11157894740.30.3424

79025.000.31650.62500.42020.33290.56950.31650.88631578950.11368421050.40.2931

80025.000.31250.62500.41670.32890.56820.31400.88421052630.11578947370.50.2457

81025.000.30860.62500.41320.32510.56690.31400.88210526320.11789473680.60.1873

82025.000.30490.62500.40980.32130.56560.31400.880.120.70.1391

83126.000.31330.65000.42280.33040.58690.31400.87974683540.12025316460.80.0881

84026.000.30950.65000.41940.32660.58560.31400.87763713080.12236286920.90.0545

85026.000.30590.65000.41600.32300.58430.31400.87552742620.124472573810.0197

86127.000.31400.67500.42860.33170.60540.31400.87526427060.1247357294Average precision:0.2553

87027.000.31030.67500.42520.32810.60400.31030.87315010570.1268498943

88027.000.30680.67500.42190.32450.60270.30680.87103594080.1289640592

89027.000.30340.67500.41860.32100.60130.30340.86892177590.1310782241

90027.000.30000.67500.41540.31760.60000.30000.8668076110.133192389

91027.000.29670.67500.41220.31430.59870.29670.86469344610.1353065539

92027.000.29350.67500.40910.31110.59730.29350.86257928120.1374207188

93027.000.29030.67500.40600.30790.59600.29030.86046511630.1395348837

94027.000.28720.67500.40300.30470.59470.28720.85835095140.1416490486

95027.000.28420.67500.40000.30170.59340.28570.85623678650.1437632135

96027.000.28130.67500.39710.29870.59210.28570.85412262160.1458773784

97027.000.27840.67500.39420.29570.59080.28570.85200845670.1479915433

98128.000.28570.70000.40580.30370.61140.28570.85169491530.1483050847

99028.000.28280.70000.40290.30080.61000.28280.84957627120.1504237288

100028.000.28000.70000.40000.29790.60870.28000.84745762710.1525423729

101028.000.27720.70000.39720.29500.60740.27720.84533898310.1546610169

102028.000.27450.70000.39440.29230.60610.27450.8432203390.156779661

103028.000.27180.70000.39160.28960.60480.27180.84110169490.1588983051

104028.000.26920.70000.38890.28690.60340.26920.83898305080.1610169492

105028.000.26670.70000.38620.28430.60220.26670.83686440680.1631355932

106028.000.26420.70000.38360.28170.60090.26420.83474576270.1652542373

107028.000.26170.70000.38100.27920.59960.26360.83262711860.1673728814

108028.000.25930.70000.37840.27670.59830.26360.83050847460.1694915254

109028.000.25690.70000.37580.27420.59700.26360.82838983050.1716101695

110129.000.26360.72500.38670.28160.61700.26360.82802547770.1719745223

111029.000.26130.72500.38410.27910.61570.26130.82590233550.1740976645

112029.000.25890.72500.38160.27670.61440.25890.82377919320.1762208068

113029.000.25660.72500.37910.27440.61310.25660.8216560510.178343949

114029.000.25440.72500.37660.27200.61180.25440.81953290870.1804670913

115029.000.25220.72500.37420.26980.61050.25220.81740976650.1825902335

116029.000.25000.72500.37180.26750.60920.25000.81528662420.1847133758

117029.000.24790.72500.36940.26530.60800.24790.8131634820.186836518

118029.000.24580.72500.36710.26320.60670.24580.81104033970.1889596603

119029.000.24370.72500.36480.26100.60540.24370.80891719750.1910828025

120029.000.24170.72500.36250.25890.60420.24170.80679405520.1932059448

121029.000.23970.72500.36020.25690.60290.23970.8046709130.195329087

122029.000.23770.72500.35800.25480.60170.23770.80254777070.1974522293

123029.000.23580.72500.35580.25280.60040.23580.80042462850.1995753715

124029.000.23390.72500.35370.25090.59920.23440.79830148620.2016985138

125029.000.23200.72500.35150.24890.59790.23440.79617834390.2038216561

126029.000.23020.72500.34940.24700.59670.23440.79405520170.2059447983

127029.000.22830.72500.34730.24510.59550.23440.79193205940.2080679406

128130.000.23440.75000.35710.25170.61480.23440.79148936170.2085106383

129030.000.23260.75000.35500.24980.61350.23260.78936170210.2106382979

130030.000.23080.75000.35290.24790.61220.23080.78723404260.2127659574

131030.000.22900.75000.35090.24610.61100.22900.7851063830.214893617

132030.000.22730.75000.34880.24430.60980.22730.78297872340.2170212766

133030.000.22560.75000.34680.24250.60850.22560.78085106380.2191489362

134030.000.22390.75000.34480.24080.60730.22390.77872340430.2212765957

135030.000.22220.75000.34290.23900.60610.22220.77659574470.2234042553

136030.000.22060.75000.34090.23730.60480.22060.77446808510.2255319149

137030.000.21900.75000.33900.23570.60360.21900.77234042550.2276595745

138030.000.21740.75000.33710.23400.60240.21740.7702127660.229787234

139030.000.21580.75000.33520.23240.60120.21580.76808510640.2319148936

140030.000.21430.75000.33330.23080.60000.21430.76595744680.2340425532

141030.000.21280.75000.33150.22920.59880.21280.76382978720.2361702128

142030.000.21130.75000.32970.22760.59760.21130.76170212770.2382978723

143030.000.20980.75000.32790.22610.59640.20980.75957446810.2404255319

144030.000.20830.75000.32610.22460.59520.20830.75744680850.2425531915

145030.000.20690.75000.32430.22300.59410.20690.75531914890.2446808511

146030.000.20550.75000.32260.22160.59290.20550.75319148940.2468085106

147030.000.20410.75000.32090.22010.59170.20410.75106382980.2489361702

148030.000.20270.75000.31910.21870.59060.20270.74893617020.2510638298

149030.000.20130.75000.31750.21720.58940.20130.74680851060.2531914894

150030.000.20000.75000.31580.21580.58820.20000.74468085110.2553191489

151030.000.19870.75000.31410.21440.58710.19870.74255319150.2574468085

152030.000.19740.75000.31250.21310.58590.19740.74042553190.2595744681

153030.000.19610.75000.31090.21170.58480.19610.73829787230.2617021277

154030.000.19480.75000.30930.21040.58370.19480.73617021280.2638297872

155030.000.19350.75000.30770.20910.58250.19350.73404255320.2659574468

156030.000.19230.75000.30610.20780.58140.19230.73191489360.2680851064

157030.000.19110.75000.30460.20650.58030.19110.7297872340.270212766

158030.000.18990.75000.30300.20520.57920.18990.72765957450.2723404255

159030.000.18870.75000.30150.20390.57800.18870.72553191490.2744680851

160030.000.18750.75000.30000.20270.57690.18750.72340425530.2765957447

161030.000.18630.75000.29850.20150.57580.18630.72127659570.2787234043

162030.000.18520.75000.29700.20030.57470.18520.71914893620.2808510638

163030.000.18400.75000.29560.19910.57360.18400.71702127660.2829787234

164030.000.18290.75000.29410.19790.57250.18290.7148936170.285106383

165030.000.18180.75000.29270.19670.57140.18180.71276595740.2872340426

166030.000.18070.75000.29130.19560.57030.18070.71063829790.2893617021

167030.000.17960.75000.28990.19440.56930.18020.70851063830.2914893617

168030.000.17860.75000.28850.19330.56820.18020.70638297870.2936170213

169030.000.17750.75000.28710.19220.56710.18020.70425531910.2957446809

170030.000.17650.75000.28570.19110.56600.18020.70212765960.2978723404

171030.000.17540.75000.28440.19000.56500.18020.70.3

172131.000.18020.77500.29250.19520.58270.18020.69936034120.3006396588

173031.000.17920.77500.29110.19410.58160.17920.6972281450.302771855

174031.000.17820.77500.28970.19300.58050.17820.69509594880.3049040512

175031.000.17710.77500.28840.19200.57940.17710.69296375270.3070362473

176031.000.17610.77500.28700.19090.57840.17610.69083155650.3091684435

177031.000.17510.77500.28570.18980.57730.17510.68869936030.3113006397

178031.000.17420.77500.28440.18880.57620.17420.68656716420.3134328358

179031.000.17320.77500.28310.18780.57510.17320.6844349680.315565032

180031.000.17220.77500.28180.18670.57410.17220.68230277190.3176972281

181031.000.17130.77500.28050.18570.57300.17130.68017057570.3198294243

182031.000.17030.77500.27930.18470.57200.17030.67803837950.3219616205

183031.000.16940.77500.27800.18380.57090.16940.67590618340.3240938166

184031.000.16850.77500.27680.18280.56990.16850.67377398720.3262260128

185031.000.16760.77500.27560.18180.56880.16760.6716417910.328358209

186031.000.16670.77500.27430.18090.56780.16670.66950959490.3304904051

187031.000.16580.77500.27310.17990.56670.16580.66737739870.3326226013

188031.000.16490.77500.27190.17900.56570.16490.66524520260.3347547974

189031.000.16400.77500.27070.17810.56470.16400.66311300640.3368869936

190031.000.16320.77500.26960.17710.56360.16320.66098081020.3390191898

191031.000.16230.77500.26840.17620.56260.16230.65884861410.3411513859

192031.000.16150.77500.26720.17530.56160.16150.65671641790.3432835821

193031.000.16060.77500.26610.17450.56060.16060.65458422170.3454157783

194031.000.15980.77500.26500.17360.55960.15980.65245202560.3475479744

195031.000.15900.77500.26380.17270.55860.15900.65031982940.3496801706

196031.000.15820.77500.26270.17180.55760.15820.64818763330.3518123667

197031.000.15740.77500.26160.17100.55660.15740.64605543710.3539445629

198031.000.15660.77500.26050.17010.55560.15660.64392324090.3560767591

199031.000.15580.77500.25940.16930.55460.15580.64179104480.3582089552

200031.000.15500.77500.25830.16850.55360.15500.63965884860.3603411514

201031.000.15420.77500.25730.16770.55260.15420.63752665250.3624733475

202031.000.15350.77500.25620.16680.55160.15350.63539445630.3646055437

203031.000.15270.77500.25510.16600.55060.15270.63326226010.3667377399

204031.000.15200.77500.25410.16520.54960.15200.6311300640.368869936

205031.000.15120.77500.25310.16450.54870.15120.62899786780.3710021322

206031.000.15050.77500.25200.16370.54770.15050.62686567160.3731343284

207031.000.14980.77500.25100.16290.54670.14980.62473347550.3752665245

208031.000.14900.77500.25000.16210.54580.14900.62260127930.3773987207

209031.000.14830.77500.24900.16140.54480.14830.62046908320.3795309168

210031.000.14760.77500.24800.16060.54390.14760.6183368870.381663113

211031.000.14690.77500.24700.15990.54290.14690.61620469080.3837953092

212031.000.14620.77500.24600.15910.54200.14620.61407249470.3859275053

213031.000.14550.77500.24510.15840.54100.14550.61194029850.3880597015

214031.000.14490.77500.24410.15770.54010.14490.60980810230.3901918977

215031.000.14420.77500.24310.15700.53910.14420.60767590620.3923240938

216031.000.14350.77500.24220.15630.53820.14350.605543710.39445629

217031.000.14290.77500.24120.15550.53730.14290.60341151390.3965884861

218031.000.14220.77500.24030.15480.53630.14220.60127931770.3987206823

219031.000.14160.77500.23940.15420.53540.14160.59914712150.4008528785

220031.000.14090.77500.23850.15350.53450.14090.59701492540.4029850746

221031.000.14030.77500.23750.15280.53360.14030.59488272920.4051172708

222031.000.13960.77500.23660.15210.53260.13960.5927505330.407249467

223031.000.13900.77500.23570.15140.53170.13900.59061833690.4093816631

224031.000.13840.77500.23480.15080.53080.13840.58848614070.4115138593

225031.000.13780.77500.23400.15010.52990.13780.58635394460.4136460554

226031.000.13720.77500.23310.14950.52900.13720.58422174840.4157782516

227031.000.13660.77500.23220.14880.52810.13660.58208955220.4179104478

228031.000.13600.77500.23130.14820.52720.13600.57995735610.4200426439

229031.000.13540.77500.23050.14750.52630.13540.57782515990.4221748401

230031.000.13480.77500.22960.14690.52540.13480.57569296380.4243070362

231031.000.13420.77500.22880.14630.52450.13420.57356076760.4264392324

232031.000.13360.77500.22790.14570.52360.13390.57142857140.4285714286

233031.000.13300.77500.22710.14510.52280.13390.56929637530.4307036247

234031.000.13250.77500.22630.14450.52190.13390.56716417910.4328358209

235031.000.13190.77500.22550.14390.52100.13390.56503198290.4349680171

236031.000.13140.77500.22460.14330.52010.13390.56289978680.4371002132

237031.000.13080.77500.22380.14270.51930.13390.56076759060.4392324094

238031.000.13030.77500.22300.14210.51840.13390.55863539450.4413646055

239132.000.13390.80000.22940.14610.53420.13390.55769230770.4423076923

240032.000.13330.80000.22860.14550.53330.13330.55555555560.4444444444

241032.000.13280.80000.22780.14490.53240.13280.55341880340.4465811966

242032.000.13220.80000.22700.14430.53160.13220.55128205130.4487179487

243032.000.13170.80000.22610.14370.53070.13170.54914529910.4508547009

244032.000.13110.80000.22540.14310.52980.13110.5470085470.452991453

245032.000.13060.80000.22460.14250.52890.13060.54487179490.4551282051

246032.000.13010.80000.22380.14200.52810.13010.54273504270.4572649573

247032.000.12960.80000.22300.14140.52720.12960.54059829060.4594017094

248032.000.12900.80000.22220.14080.52630.12900.53846153850.4615384615

249032.000.12850.80000.22150.14030.52550.12850.53632478630.4636752137

250032.000.12800.80000.22070.13970.52460.12800.53418803420.4658119658

251032.000.12750.80000.21990.13920.52370.12750.53205128210.4679487179

252032.000.12700.80000.21920.13860.52290.12700.52991452990.4700854701

253032.000.12650.80000.21840.13810.52200.12650.52777777780.4722222222

254032.000.12600.80000.21770.13760.52120.12600.52564102560.4743589744

255032.000.12550.80000.21690.13700.52030.12550.52350427350.4764957265

256032.000.12500.80000.21620.13650.51950.12500.52136752140.4786324786

257032.000.12450.80000.21550.13600.51860.12450.51923076920.4807692308

258032.000.12400.80000.21480.13550.51780.12400.51709401710.4829059829

259032.000.12360.80000.21400.13500.51700.12360.5149572650.485042735

260032.000.12310.80000.21330.13450.51610.12310.51282051280.4871794872

261032.000.12260.80000.21260.13390.51530.12260.51068376070.4893162393

262032.000.12210.80000.21190.13340.51450.12210.50854700850.4914529915

263032.000.12170.80000.21120.13290.51360.12170.50641025640.4935897436

264032.000.12120.80000.21050.13250.51280.12120.50427350430.4957264957

265032.000.12080.80000.20980.13200.51200.12080.50213675210.4978632479

266032.000.12030.80000.20920.13150.51120.12030.50.5

267032.000.11990.80000.20850.13100.51040.11990.49786324790.5021367521

268032.000.11940.80000.20780.13050.50960.11940.49572649570.5042735043

269032.000.11900.80000.20710.13000.50870.11900.49358974360.5064102564

270032.000.11850.80000.20650.12960.50790.11870.49145299150.5085470085

271032.000.11810.80000.20580.12910.50710.11870.48931623930.5106837607

272032.000.11760.80000.20510.12860.50630.11870.48717948720.5128205128

273032.000.11720.80000.20450.12820.50550.11870.4850427350.514957265

274032.000.11680.80000.20380.12770.50470.11870.48290598290.5170940171

275032.000.11640.80000.20320.12720.50390.11870.48076923080.5192307692

276032.000.11590.80000.20250.12680.50310.11870.47863247860.5213675214

277032.000.11550.80000.20190.12630.50240.11870.47649572650.5235042735

278133.000.11870.82500.20750.12980.51720.11870.47537473230.5246252677

279033.000.11830.82500.20690.12940.51640.11830.47323340470.5267665953

280033.000.11790.82500.20630.12890.51560.11790.47109207710.5289079229

281033.000.11740.82500.20560.12850.51480.11740.46895074950.5310492505

282033.000.11700.82500.20500.12800.51400.11700.46680942180.5331905782

283033.000.11660.82500.20430.12760.51320.11660.46466809420.5353319058

284033.000.11620.82500.20370.12710.51240.11620.46252676660.5374732334

285033.000.11580.82500.20310.12670.51160.11580.4603854390.539614561

286033.000.11540.82500.20250.12620.51080.11540.45824411130.5417558887

287033.000.11500.82500.20180.12580.51000.11500.45610278370.5438972163

288033.000.11460.82500.20120.12540.50930.11460.45396145610.5460385439

289033.000.11420.82500.20060.12500.50850.11420.45182012850.5481798715

290033.000.11380.82500.20000.12450.50770.11380.44967880090.5503211991

291033.000.11340.82500.19940.12410.50690.11340.44753747320.5524625268

292033.000.11300.82500.19880.12370.50610.11300.44539614560.5546038544

293033.000.11260.82500.19820.12330.50540.11260.4432548180.556745182

294033.000.11220.82500.19760.12290.50460.11220.44111349040.5588865096

295033.000.11190.82500.19700.12240.50380.11190.43897216270.5610278373

296033.000.11150.82500.19640.12200.50300.11150.43683083510.5631691649

297033.000.11110.82500.19580.12160.50230.11110.43468950750.5653104925

298033.000.11070.82500.19530.12120.50150.11070.43254817990.5674518201

299033.000.11040.82500.19470.12080.50080.11040.43040685220.5695931478

300033.000.11000.82500.19410.12040.50000.11000.42826552460.5717344754

301033.000.10960.82500.19350.12000.49920.10960.4261241970.573875803

302033.000.10930.82500.19300.11970.49850.10930.42398286940.5760171306

303033.000.10890.82500.19240.11930.49770.10890.42184154180.5781584582

304033.000.10860.82500.19190.11890.49700.10860.41970021410.5802997859

305033.000.10820.82500.19130.11850.49620.10820.41755888650.5824411135

306033.000.10780.82500.19080.11810.49550.10780.41541755890.5845824411

307033.000.10750.82500.19020.11770.49480.10750.41327623130.5867237687

308033.000.10710.82500.18970.11740.49400.10710.41113490360.5888650964

309033.000.10680.82500.18910.11700.49330.10680.4089935760.591006424

310033.000.10650.82500.18860.11660.49250.10650.40685224840.5931477516

311033.000.10610.82500.18800.11620.49180.10610.40471092080.5952890792

312033.000.10580.82500.18750.11590.49110.10580.40256959310.5974304069

313033.000.10540.82500.18700.11550.49030.10560.40042826550.5995717345

314033.000.10510.82500.18640.11510.48960.10560.39828693790.6017130621

315033.000.10480.82500.18590.11480.48890.10560.39614561030.6038543897

316033.000.10440.82500.18540.11440.48820.10560.39400428270.6059957173

317033.000.10410.82500.18490.11410.48740.10560.3918629550.608137045

318033.000.10380.82500.18440.11370.48670.10560.38972162740.6102783726

319033.000.10340.82500.18380.11340.48600.10560.38758029980.6124197002

320033.000.10310.82500.18330.11300.48530.10560.38543897220.6145610278

321033.000.10280.82500.18280.11270.48460.10560.38329764450.6167023555

322134.000.10560.85000.18780.11570.49850.10560.38197424890.6180257511

323034.000.10530.85000.18730.11540.49780.10530.37982832620.6201716738

324034.000.10490.85000.18680.11500.49710.10490.37768240340.6223175966

325034.000.10460.85000.18630.11470.49640.10460.37553648070.6244635193

326034.000.10430.85000.18580.11430.49560.10430.37339055790.6266094421

327034.000.10400.85000.18530.11400.49490.10400.37124463520.6287553648

328034.000.10370.85000.18480.11360.49420.10370.36909871240.6309012876

329034.000.10330.85000.18430.11330.49350.10330.36695278970.6330472103

330034.000.10300.85000.18380.11300.49280.10320.3648068670.635193133

331034.000.10270.85000.18330.11260.49200.10320.36266094420.6373390558

332034.000.10240.85000.18280.11230.49130.10320.36051502150.6394849785

333034.000.10210.85000.18230.11200.49060.10320.35836909870.6416309013

334034.000.10180.85000.18180.11160.48990.10320.3562231760.643776824

335034.000.10150.85000.18130.11130.48920.10320.35407725320.6459227468

336034.000.10120.85000.18090.11100.48850.10320.35193133050.6480686695

337034.000.10090.85000.18040.11060.48780.10320.34978540770.6502145923

338034.000.10060.85000.17990.11030.48710.10320.3476394850.652360515

339034.000.10030.85000.17940.11000.48640.10320.34549356220.6545064378

340034.000.10000.85000.17890.10970.48570.10320.34334763950.6566523605

341034.000.09970.85000.17850.10940.48500.10320.34120171670.6587982833

342034.000.09940.85000.17800.10900.48430.10320.3390557940.660944206

343034.000.09910.85000.17750.10870.48360.10320.33690987120.6630901288

344034.000.09880.85000.17710.10840.48300.10320.33476394850.6652360515

345034.000.09860.85000.17660.10810.48230.10320.33261802580.6673819742

346034.000.09830.85000.17620.10780.48160.10320.3304721030.669527897

347135.000.10090.87500.18090.11070.49500.10320.32903225810.6709677419

348035.000.10060.87500.18040.11030.49440.10320.32688172040.6731182796

349136.000.10320.90000.18510.11320.50780.10320.32543103450.6745689655

350036.000.10290.90000.18460.11290.50700.10290.32327586210.6767241379

351036.000.10260.90000.18410.11250.50630.10260.32112068970.6788793103

352036.000.10230.90000.18370.11220.50560.10230.31896551720.6810344828

353036.000.10200.90000.18320.11190.50490.10200.31681034480.6831896552

354036.000.10170.90000.18270.11160.50420.10170.31465517240.6853448276

355036.000.10140.90000.18230.11130.50350.10140.31250.6875

356036.000.10110.90000.18180.11100.50280.10140.31034482760.6896551724

357036.000.10080.90000.18140.11070.50210.10140.30818965520.6918103448

358036.000.10060.90000.18090.11040.50140.10140.30603448280.6939655172

359036.000.10030.90000.18050.11010.50070.10140.30387931030.6961206897

360036.000.10000.90000.18000.10980.50000.10140.30172413790.6982758621

361036.000.09970.90000.17960.10950.49930.10140.29956896550.7004310345

362036.000.09940.90000.17910.10920.49860.10140.29741379310.7025862069

363036.000.09920.90000.17870.10890.49790.10140.29525862070.7047413793

364036.000.09890.90000.17820.10860.49720.10140.29310344830.7068965517

365137.000.10140.92500.18270.11130.51030.10140.29157667390.7084233261

366037.000.10110.92500.18230.11100.50960.10110.28941684670.7105831533

367037.000.10080.92500.18180.11070.50890.10080.28725701940.7127429806

368037.000.10050.92500.18140.11040.50820.10050.28509719220.7149028078

369037.000.10030.92500.18090.11010.50750.10030.2829373650.717062635

370037.000.10000.92500.18050.10980.50680.10000.28077753780.7192224622

371037.000.09970.92500.18000.10950.50620.09970.27861771060.7213822894

372037.000.09950.92500.17960.10920.50550.09950.27645788340.7235421166

373037.000.09920.92500.17920.10890.50480.09920.27429805620.7257019438

374037.000.09890.92500.17870.10860.50410.09890.27213822890.7278617711

375037.000.09870.92500.17830.10830.50340.09870.26997840170.7300215983

376037.000.09840.92500.17790.10810.50270.09840.26781857450.7321814255

377037.000.09810.92500.17750.10780.50200.09810.26565874730.7343412527

378037.000.09790.92500.17700.10750.50140.09790.26349892010.7365010799

379037.000.09760.92500.17660.10720.50070.09760.26133909290.7386609071

380037.000.09740.92500.17620.10690.50000.09740.25917926570.7408207343

381037.000.09710.92500.17580.10670.49930.09710.25701943840.7429805616

382037.000.09690.92500.17540.10640.49870.09690.25485961120.7451403888

383037.000.09660.92500.17490.10610.49800.09660.2526997840.747300216

384037.000.09640.92500.17450.10580.49730.09640.25053995680.7494600432

385037.000.09610.92500.17410.10560.49660.09610.24838012960.7516198704

386037.000.09590.92500.17370.10530.49600.09590.24622030240.7537796976

387037.000.09560.92500.17330.10500.49530.09560.24406047520.7559395248

388037.000.09540.92500.17290.10480.49470.09540.24190064790.7580993521

389037.000.09510.92500.17250.10450.49400.09510.23974082070.7602591793

390037.000.09490.92500.17210.10420.49330.09490.23758099350.7624190065

391037.000.09460.92500.17170.10400.49270.09460.23542116630.7645788337

392037.000.09440.92500.17130.10370.49200.09440.23326133910.7667386609

393037.000.09410.92500.17090.10340.49140.09410.23110151190.7688984881

394037.000.09390.92500.17050.10320.49070.09390.22894168470.7710583153

395037.000.09370.92500.17010.10290.49010.09370.22678185750.7732181425

396037.000.09340.92500.16970.10270.48940.09340.22462203020.7753779698

397037.000.09320.92500.16930.10240.48880.09320.2224622030.777537797

398037.000.09300.92500.16890.10220.48810.09300.22030237580.7796976242

399037.000.09270.92500.16860.10190.48750.09270.21814254860.7818574514

400037.000.09250.92500.16820.10160.48680.09250.21598272140.7840172786

401037.000.09230.92500.16780.10140.48620.09230.21382289420.7861771058

402037.000.09200.92500.16740.10110.48560.09200.2116630670.788336933

403037.000.09180.92500.16700.10090.48490.09180.20950323970.7904967603

404037.000.09160.92500.16670.10070.48430.09160.20734341250.7926565875

405037.000.09140.92500.16630.10040.48370.09140.20518358530.7948164147

406037.000.09110.92500.16590.10020.48300.09110.20302375810.7969762419

407037.000.09090.92500.16550.09990.48240.09090.20086393090.7991360691

408037.000.09070.92500.16520.09970.48180.09070.19870410370.8012958963

409037.000.09050.92500.16480.09940.48110.09050.19654427650.8034557235

410037.000.09020.92500.16440.09920.48050.09020.19438444920.8056155508

411037.000.09000.92500.16410.09900.47990.09000.1922246220.807775378

412037.000.08980.92500.16370.09870.47930.08980.19006479480.8099352052

413037.000.08960.92500.16340.09850.47870.08960.18790496760.8120950324

414037.000.08940.92500.16300.09820.47800.08960.18574514040.8142548596

415037.000.08920.92500.16260.09800.47740.08960.18358531320.8164146868

416037.000.08890.92500.16230.09780.47680.08960.1814254860.818574514

417037.000.08870.92500.16190.09750.47620.08960.17926565870.8207343413

418037.000.08850.92500.16160.09730.47560.08960.17710583150.8228941685

419037.000.08830.92500.16120.09710.47500.08960.17494600430.8250539957

420037.000.08810.92500.16090.09690.47440.08960.17278617710.8272138229

421037.000.08790.92500.16050.09660.47380.08960.17062634990.8293736501

422037.000.08770.92500.16020.09640.47310.08960.16846652270.8315334773

423037.000.08750.92500.15980.09620.47250.08960.16630669550.8336933045

424138.000.08960.95000.16380.09850.48470.08960.16450216450.8354978355

425038.000.08940.95000.16340.09830.48410.08940.16233766230.8376623377

426038.000.08920.95000.16310.09810.48350.08920.16017316020.8398268398

427038.000.08900.95000.16270.09790.48280.08900.1580086580.841991342

428038.000.08880.95000.16240.09760.48220.08880.15584415580.8441558442

429038.000.08860.95000.16200.09740.48160.08860.15367965370.8463203463

430038.000.08840.95000.16170.09720.48100.08840.15151515150.8484848485

431038.000.08820.95000.16140.09700.48040.08820.14935064940.8506493506

432038.000.08800.95000.16100.09670.47980.08800.14718614720.8528138528

433038.000.08780.95000.16070.09650.47920.08780.1450216450.854978355

434038.000.08760.95000.16030.09630.47860.08760.14285714290.8571428571

435038.000.08740.95000.16000.09610.47800.08740.14069264070.8593073593

436038.000.08720.95000.15970.09590.47740.08720.13852813850.8614718615

437038.000.08700.95000.15930.09560.47680.08700.13636363640.8636363636

438038.000.08680.95000.15900.09540.47620.08680.13419913420.8658008658

439038.000.08660.95000.15870.09520.47560.08660.1320346320.867965368

440038.000.08640.95000.15830.09500.47500.08640.12987012990.8701298701

441038.000.08620.95000.15800.09480.47440.08620.12770562770.8722943723

442038.000.08600.95000.15770.09460.47380.08600.12554112550.8744588745

443038.000.08580.95000.15730.09440.47320.08580.12337662340.8766233766

444038.000.08560.95000.15700.09420.47260.08560.12121212120.8787878788

445038.000.08540.95000.15670.09390.47200.08540.1190476190.880952381

446038.000.08520.95000.15640.09370.47150.08520.11688311690.8831168831

447038.000.08500.95000.15610.09350.47090.08500.11471861470.8852813853

448038.000.08480.95000.15570.09330.47030.08480.11255411260.8874458874

449038.000.08460.95000.15540.09310.46970.08460.11038961040.8896103896

450038.000.08440.95000.15510.09290.46910.08460.10822510820.8917748918

451038.000.08430.95000.15480.09270.46860.08460.10606060610.8939393939

452038.000.08410.95000.15450.09250.46800.08460.10389610390.8961038961

453038.000.08390.95000.15420.09230.46740.08460.10173160170.8982683983

454038.000.08370.95000.15380.09210.46680.08460.09956709960.9004329004

455038.000.08350.95000.15350.09190.46630.08460.09740259740.9025974026

456038.000.08330.95000.15320.09170.46570.08460.09523809520.9047619048

457038.000.08320.95000.15290.09150.46510.08460.09307359310.9069264069

458038.000.08300.95000.15260.09130.46450.08460.09090909090.9090909091

459038.000.08280.95000.15230.09110.46400.08460.08874458870.9112554113

460038.000.08260.95000.15200.09090.46340.08460.08658008660.9134199134

461139.000.08460.97500.15570.09310.47500.08460.08459869850.9154013015

462039.000.08440.97500.15540.09290.47450.08440.08242950110.9175704989

463039.000.08420.97500.15510.09270.47390.08420.08026030370.9197396963

464039.000.08410.97500.15480.09250.47330.08410.07809110630.9219088937

465039.000.08390.97500.15450.09230.47270.08390.07592190890.9240780911

466039.000.08370.97500.15420.09210.47220.08370.07375271150.9262472885

467039.000.08350.97500.15380.09190.47160.08350.07158351410.9284164859

468039.000.08330.97500.15350.09170.47100.08330.06941431670.9305856833

469039.000.08320.97500.15320.09150.47040.08320.06724511930.9327548807

470039.000.08300.97500.15290.09130.46990.08300.06507592190.9349240781

471039.000.08280.97500.15260.09110.46930.08280.06290672450.9370932755

472039.000.08260.97500.15230.09100.46880.08260.06073752710.9392624729

473039.000.08250.97500.15200.09080.46820.08250.05856832970.9414316703

474039.000.08230.97500.15180.09060.46760.08230.05639913230.9436008677

475039.000.08210.97500.15150.09040.46710.08210.05422993490.9457700651

476039.000.08190.97500.15120.09020.46650.08210.05206073750.9479392625

477039.000.08180.97500.15090.09000.46590.08210.04989154010.9501084599

478039.000.08160.97500.15060.08980.46540.08210.04772234270.9522776573

479039.000.08140.97500.15030.08960.46480.08210.04555314530.9544468547

480039.000.08130.97500.15000.08940.46430.08210.04338394790.9566160521

481039.000.08110.97500.14970.08930.46370.08210.04121475050.9587852495

482039.000.08090.97500.14940.08910.46320.08210.03904555310.9609544469

483039.000.08070.97500.14910.08890.46260.08210.03687635570.9631236443

484039.000.08060.97500.14890.08870.46210.08210.03470715840.9652928416

485039.000.08040.97500.14860.08850.46150.08210.0325379610.967462039

486039.000.08020.97500.14830.08840.46100.08210.03036876360.9696312364

487140.000.08211.00000.15180.09040.47230.08210.02826086960.9717391304

488040.000.08201.00000.15150.09030.47170.08200.02608695650.9739130435

489040.000.08181.00000.15120.09010.47110.08180.02391304350.9760869565

490040.000.08161.00000.15090.08990.47060.08160.02173913040.9782608696

491040.000.08151.00000.15070.08970.47000.08150.01956521740.9804347826

492040.000.08131.00000.15040.08950.46950.08130.01739130430.9826086957

493040.000.08111.00000.15010.08930.46890.08110.01521739130.9847826087

494040.000.08101.00000.14980.08920.46840.08100.01304347830.9869565217

495040.000.08081.00000.14950.08900.46780.08080.01086956520.9891304348

496040.000.08061.00000.14930.08880.46730.08060.00869565220.9913043478

497040.000.08051.00000.14900.08860.46670.08050.00652173910.9934782609

498040.000.08031.00000.14870.08850.46620.08030.00434782610.9956521739

499040.000.08021.00000.14840.08830.46570.08020.0021739130.997826087

500040.000.08001.00000.14810.08810.46510.080001

Total relevant:400.45100.54350.6170

Sheet1

Precison

interp prec

F1

Recall

Precision

Sheet2

ROC curve

1 − specificity

sensitivity

Sheet3

Ave. Precision

Recall

Precision


Interpolated precision Idea: If locally precision increases with increasing

recall, then you should get to count that… So you take the max of precisions to the right of the

value


Sec. 8.4


Evaluation


Sec. 8.4

Graphs are good, but often we want a summary measure! Precision at fixed retrieval level

Prevision-at-k: Precision of top k results Perhaps appropriate for most of web search: all people want are

good matches on the first one or two result pages But: averages badly and has an arbitrary parameters of k

11-point interpolated average precisionThe standard measure in the early TREC competitions: you take the precision at 11 levels of recall varying from 0 to 1 by tenths of the documents, using interpolation (the value for 0 is always interpolated!), and average them Evaluates performance at all recall levels


Yet more evaluation measures… Mean average precision (MAP) Average of the precision value obtained for the top k

documents, each time a relevant doc is retrieved Avoids interpolation, use of fixed recall levels MAP for query collection is arithmetic ave.

Macro-averaging: each query counts equally

R-precision If have known (though perhaps incomplete) set of relevant

documents of size Rel, then calculate precision of top Reldocs returned

Perfect system could score 1.0.


Sec. 8.4


Variance For a test collection, it is usual that a system does

poorly on some information needs (e.g., MAP = 0.1) and excellent on others (e.g., MAP = 0.7)

Indeed, it is usually the case that the variance in performance of the same system across queries is much greater than the variance of different systems on the same query.

That is, there are easy information needs and hard ones!


Sec. 8.4


CREATING TEST COLLECTIONS FOR EVALUATION



Test Collections


Sec. 8.5

Scientific papers

Medical

Medical

Scientific papers

News

News


From document collections to test collectionsStill need the other 2 things

1.Test queries Must be relevant to docs available Best designed by domain experts Random query terms generally not a good idea

2.Relevance assessments Human judges, time-consuming Are human panels perfect?


Sec. 8.5


Kappa measure for inter-judge (dis)agreement


Sec. 8.5

Kappa measure Agreement measure among judges Designed for categorical judgments Corrects for chance agreement

Kappa (K) = [P(A)-P(E)][1-P(E)] P(A) – proportion of time judges agree P(E) – what agreement would be by chance

Gives 0 for chance agreement, 1 for total agreement



Sec. 8.5

Kappa Measure: Example

# of docs matching judgment type

Judge 1 Judge 2

300 Relevant Relevant70 Non-relevant Non-relevant20 Relevant Non-relevant10 Non-relevant Relevant

What is P(A)?How about P(E)?


Kappa Measure: Example


Sec. 8.5

P(a) = 370 / 400 = 0.925

Judge 1: P(relevant) = 320/400 = 0.8, P(non-relevant) = 1-0.8 = 0.2Judge 2: P(relevant) = 310/400 = 0.775, P(non-relevant) = 1-0.775= 0.225

P(E) = 0.8*0.775 + 0.2*0.225 = 0.62+0.045 = 0.665Kappa = K = (0.925 - 0.665) / (1-0.665) = 0.776

Kappa > 0.8 Good agreement 0.67 < Kappa < 0.8 Tentative conclusions

Depend on purpose of study For >2 judges: average pairwise kappas (or ANOVA)


TREC TREC's Ad Hoc task from first 8 TRECs was the standard IR task

50 detailed information needs a year Human evaluation of pooled results returned More recently other related things: Web, Hard, QA, interactive track

A query from TREC 5 (1996)

225

What is the main function of the Federal Emergency Management Age

CS3245 Information Retrieval - NUS Computingzhaojin/cs3245_2019/w9.pdf · 2019-03-22 · CS3245 – Information Retrieval Should we use accuracy for evaluation instead? Information

Documents