Tutorial on StatisticalTests

Statistical Tests for Computational Intelligence Research and

Human Subjective Tests

Hideyuki TAKAGIKyushu University, Japan

http://www.design.kyushu-u.ac.jp/~takagi/ver. March 26, 2015ver. July 15, 2013ver. July 11, 2013ver. April 23, 2013

Slides are downloadable fromhttp://www.design.kyushu-u.ac.jp/~takagi

Contents2 groups n groups (n > 2)

datadistribution

・unpaired ｔ -test

・sign test・Wilcoxon signed-ranks test ・Friedman test

・Kruskal-Wallis test

・ one-way ANOVA

・ two-way ANOVA

(no

norm

ality

)

one-waydata

two-waydata

Para

met

ric T

est

Non

-par

amet

ric T

est

(nor

mal

ity)

unpa

ired

(inde

pend

ent)

paire

d(re

late

d)un

paire

d(in

depe

nden

t)pa

ired

(rela

ted)

・paired ｔ -test

・Mann-Whitney U-test

ANO

VA(A

naly

sis

of V

aria

nce)

＋Scheffé's method of paired comparison for Human Subjective Tests

fitne

ss

generations

fitne

ss

generations

conventionalEC

proposed EC1 proposed EC2

How to Show Significance?

conventionalEC

Just compare averages visually?It is not scientific.

Fig. XX Average convergence curves of n times of trial runs.


sound made byconventional IEC

sound made byproposed IEC1

sound made byproposed IEC2

Sound design concept: exiting

Which method is good to make exiting sound?How to show it?

You cannot show the superiority of your method without statistical tests.

Papers without statistics tests may be rejected.

statistical test

My method is significantly better!

Which Test Should We Use?2 groups n groups (n > 2)

datadistribution




・ one-way ANOVA

・ two-way ANOVA

(no

norm

ality

)

one-waydata

two-waydata

Para

met

ric T

est

Non

-par

amet

ric T

est

(nor

mal

ity)

unpa

ired

(inde

pend

ent)

paire

d(re

late

d)un

paire

d(in

depe

nden

t)pa

ired

(rela

ted)

・paired ｔ -test


ANO

VA(A

naly

sis

of V

aria

nce)

Which Test Should We Use?2 groups n groups (n > 2)

datadistribution




・ one-way ANOVA

・ two-way ANOVA

(no

norm

ality

)

one-waydata

two-waydata

Para

met

ric T

est

Non

-par

amet

ric T

est

(nor

mal

ity)

unpa

ired

(inde

pend

ent)

paire

d(re

late

d)un

paire

d(in

depe

nden

t)pa

ired

(rela

ted)

・paired ｔ -test


ANO

VA(A

naly

sis

of V

aria

nce)

n-th generation n-th generation

2 groups n groups (n > 2)

datadistribution




・ one-way ANOVA

・ two-way ANOVA

one-waydata

two-waydata

unpa

ired

(inde

pend

ent)

paire

d(re

late

d)un

paire

d(in

depe

nden

t)pa

ired

(rela

ted)

・paired ｔ -test


ANO

VA(A

naly

sis

of V

aria

nce)

(no

norm

ality

)Pa

ram

etric

Tes

tN

on-p

aram

etric

Tes

t(n

orm

ality

)

Which Test Should we Use?

• Anderson-Darling test• D'Agostino-Pearson test• Kolmogorov-Smirnov test• Shapiro-Wilk test• Jarque–Bera test

・・・・

Normality Test


datadistribution




・ one-way ANOVA

・ two-way ANOVA

one-waydata

two-waydata

・paired ｔ -test


ANO

VA(A

naly

sis

of V

aria

nce)

(no

norm

ality

)Pa

ram

etric

Tes

tN

on-p

aram

etric

Tes

t(n

orm

ality

)

Which Test Should We Use?

unpa

ired

(inde

pend

ent)

paire

d(re

late

d)un

paire

d(in

depe

nden

t)pa

ired

(rela

ted)

initialdata #

conventional proposed

1 4.23 2.51

2 3.21 3.30

3 3.63 3.75

4 4.42 3.22

5 4.08 3.99

6 3.98 3.65

unpaired data(independent)

paired data(related)

group A group B

4.23 2.51

3.21 3.3

3.63 3.75

4.42 3.22

4.08 3.99

3.98 3.65


datadistribution




・ one-way ANOVA

・ two-way ANOVA

one-waydata

two-waydata

・paired ｔ -test


ANO

VA(A

naly

sis

of V

aria

nce)

(no

norm

ality

)Pa

ram

etric

Tes

tN

on-p

aram

etric

Tes

t(n

orm

ality

)


unpa

ired

(inde

pend

ent)

paire

d(re

late

d)un

paire

d(in

depe

nden

t)pa

ired

(rela

ted)



A group data B group data initialdata # GA proposed


datadistribution




・ one-way ANOVA

・ two-way ANOVA

one-waydata

two-waydata

・paired ｔ -test


ANO

VA(A

naly

sis

of V

aria

nce)

(no

norm

ality

)Pa

ram

etric

Tes

tN

on-p

aram

etric

Tes

t(n

orm

ality

)


unpa

ired

(inde

pend

ent)

paire

d(re

late

d)un

paire

d(in

depe

nden

t)pa

ired

(rela

ted)



A group data B group data initialdata # GA proposed

Q1: Which tests are more sensitive,those for unpaired data or paired data?

A1: Statistical tests for paired data because of more data information.


datadistribution




・ one-way ANOVA

・ two-way ANOVA

one-waydata

two-waydata

・paired ｔ -test


ANO

VA(A

naly

sis

of V

aria

nce)

(no

norm

ality

)Pa

ram

etric

Tes

tN

on-p

aram

etric

Tes

t(n

orm

ality

)


unpa

ired

(inde

pend

ent)

paire

d(re

late

d)un

paire

d(in

depe

nden

t)pa

ired

(rela

ted)

Q2: How should you design your experimental conditions to use statistical tests for paired data and reduce the # of trial runs?

A2: Use the same initialized data for the set of (method A, method B) at each trial run.

n-th generation

significant?


datadistribution




・ one-way ANOVA

・ two-way ANOVA

one-waydata

two-waydata

unpa

ired

(inde

pend

ent)

paire

d(re

late

d)un

paire

d(in

depe

nden

t)pa

ired

(rela

ted)

・paired ｔ -test


ANO

VA(A

naly

sis

of V

aria

nce)

(no

norm

ality

)Pa

ram

etric

Tes

tN

on-p

aram

etric

Tes

t(n

orm

ality

)

Which Test Should we Use?Q3: Which statistical tests are sensitive,

parametric tests or non-parametric onesand why?

A3: Parametric tests which can use information of assumed data distribution.

t -Test2 groups n groups (n > 2)

datadistribution




・ one-way ANOVA

・ two-way ANOVA

(no

norm

ality

)

one-waydata

two-waydata

Para

met

ric T

est

Non

-par

amet

ric T

est

(nor

mal

ity)

unpa

ired

(inde

pend

ent)

paire

d(re

late

d)un

paire

d(in

depe

nden

t)pa

ired

(rela

ted)

・paired ｔ -test


ANO

VA(A

naly

sis

of V

aria

nce)

ｔ -test

gt-Test


n-th generation

significant? A B12 1014 914 711 1516 1119 10

significantdifference?

Conditions to use t-tests:(1) normality(2) equal variances (not essential though )

Test this difference with assuming no difference.（null hypothesis）

t-Testt-Test

Conditions to use t-tests:(1) normality(2) equal variances (not essential though )

A B12 1014 914 711 1516 1119 10

significantdifference?

Test this difference with assuming no difference.（null hypothesis）

Normality Test• Anderson-Darling test• D'Agostino-Pearson test• Kolmogorov-Smirnov test• Shapiro-Wilk test• Jarque–Bera test

・・・・

F-Test

When (p > 0.05), we assume that there is no significant difference between σ2

A and σ2B .

t-Testt-Test t-Testt-TestExcel (32 bits version only?) has t-tests and ANOVA in Data Analysis Tools. You must install its add-in. (File -> option -> add-in, and set its add-in.)

t-Test(1) t-Test: Pairs two sample for means

(2) t-Test: Two-sample assuming equal variances

(3) t-Test: Two-sample assuming unequal variances: Welch's t-test

n-th generation

significant? This is a case when each pair of two methods with the same initial condition.

t-Test

A B 4.23 2.51 3.21 3.31 3.63 3.75 4.42 3.22 4.08 3.99 3.98 3.65 3.68 3.35 4.18 3.93 3.85 3.91 3.71 3.82

Variable 1 Variable 2Mean 3.897 3.544Variance 0.125823333 0.208693333Observations 10 10Pearson Correlation -0.161190073Hypothesized Mean Difference 0

df 9t Stat 1.794964241P(T<=t) one-tail 0.053116886t Critical one-tail 1.833112933P(T<=t) two-tail 0.106233772t Critical two-tail 2.262157163

sample data t-Test: Paired Two Sample for Means

t-Test

A B 4.23 2.51 3.21 3.31 3.63 3.75 4.42 3.22 4.08 3.99 3.98 3.65 3.68 3.35 4.18 3.93 3.85 3.91 3.71 3.82

Variable 1 Variable 2Mean 3.897 3.544Variance 0.125823333 0.208693333Observations 10 10Pearson Correlation -0.161190073Hypothesized Mean Difference 0

df 9t Stat 1.794964241P(T<=t) one-tail 0.053116886t Critical one-tail 1.833112933P(T<=t) two-tail 0.106233772t Critical two-tail 2.262157163

sample data t-Test: Paired Two Sample for Means

2.5% 2.5% 5%

When p-value is less than 0.01 or 0.05, we assume that there is significant difference with the level of significance of (p < 0.01) or (p < 0.05).

A > B A < BA ≈ B When A>B never happens, you may use a one-tail test.

t-Test

(1) t-Test: Pairs two sample for means (2) t-Test: Two-sample assuming equal variances

Difference between two groups is significant (p < 0.01).

We cannot say that there is a significant difference between two group.

ANOVA: Analysis of Variance2 groups n groups (n > 2)

datadistribution




・ one-way ANOVA

・ two-way ANOVA

(no

norm

ality

)

one-waydata

two-waydata

Para

met

ric T

est

Non

-par

amet

ric T

est

(nor

mal

ity)

unpa

ired

(inde

pend

ent)

paire

d(re

late

d)un

paire

d(in

depe

nden

t)pa

ired

(rela

ted)

・paired ｔ -test


ANO

VA(A

naly

sis

of V

aria

nce)

ANOVA

ANOVA: Analysis of Variance

n-th generation

significant?

A B C11.0 12.8 9.4 9.3 11.3 12.4

11.5 9.5 16.8 16.4 14.0 14.3 16.0 15.2 17.0 15.0 13.0 14.6 12.8 12.4 17.0 13.6 15.0 14.3 13.0 12.4 15.6 12.0 17.8 15.0 13.4 12.6 18.6 10.0 13.4 12.4 10.8 16.8 15.4

1. Analysis of more than two data groups.2. Normality and equal variance are required.

C A B

Excel has ANOVA in Data Analysis Tools.


A B C11.0 12.8 9.4 9.3 11.3 12.4

11.5 9.5 16.8 16.4 14.0 14.3 16.0 15.2 17.0 15.0 13.0 14.6 12.8 12.4 17.0 13.6 15.0 14.3 13.0 12.4 15.6 12.0 17.8 15.0 13.4 12.6 18.6 10.0 13.4 12.4 10.8 16.8 15.4

1. Analysis of more than two data groups.2. Normality and equal variance are required.

C A B

three t-tests one ANOVA=1-(1-0.05)3 = 0.14

Three times of t-test with (p<0.05) equivalent one ANOVA (p<0.14).

Excel has ANOVA in Data Analysis Tools.


Check it using the Bartlett test.


n-th generation

When data are independent, use one-way ANOVA (single factor ANOVA).

When data correspond each other, use two-way ANOVA (two-factor ANOVA).


When data are independent, use one-way ANOVA (single factor ANOVA).

When data correspond each other, use two-way ANOVA (two-factor ANOVA).

Q1: What are "single factor" and "two factors"?

A1: A column factor (e.g. three groups) and a sample factor (e.g. initialized condition).

column factor

sam

ple

fact

or

column factor


column factor

sam

ple

fact

or

column factor

one-factor (one-way) ANOVA two-factor (two-way) ANOVA

group A group B group C4.23 2.51 3.043.21 3.3 2.893.63 3.75 3.554.42 3.22 4.394.08 3.99 3.863.98 3.65 3.53.75 2.62 3.63.22 2.93 3.21

initial condition group A group B group C

#1 4.23 2.51 3.04#2 3.21 3.3 2.89#3 3.63 3.75 3.55#4 4.42 3.22 4.39#5 4.08 3.99 3.86#6 3.98 3.65 3.5#7 3.75 2.62 3.6#8 3.22 2.93 3.21

We cannot say that three groups are significantly different. (p=0.089)

There are significant difference somewhere among three groups. (p<0.05)


Output of the one-way ANOVASource of Variation SS df MS F P-value F crit

Between Groups 6.11342 2 3.05671 15.30677 3.6E-05 3.354131Within Groups 5.39181 27 0.199697

Total 11.50523 29

Output of the two-way ANOVASource of Variation SS df MS F P-value F crit

Sample 0.755233 2 0.377617 2.755097 0.103596 3.885294Columns 3.582272 1 3.582272 26.13631 0.000256 4.747225Interaction 0.139411 2 0.069706 0.508573 0.613752 3.885294Within 1.644733 12 0.137061

Total 6.12165 17

When (p-value < 0.01 or 0.05), there is(are) significant differencesomewhere among data groups.

• Significant difference among Sample (e.g. initial conditions) cannot be found (p > 0.05).

• Significant difference can be found somewhere among Columns (e.g. three methods) (p < 0.01).

• We need not care an interaction effect between two factors (e.g. initial condition vs. methods) (p > 0.05).

A B C11.0 12.8 9.4 9.3 11.3 12.4

11.5 9.5 16.8 16.4 14.0 14.3 16.0 15.2 17.0 15.0 13.0 14.6 12.8 12.4 17.0 13.6 15.0 14.3 13.0 12.4 15.6 12.0 17.8 15.0 13.4 12.6 18.6 10.0 13.4 12.4 10.8 16.8 15.4

Sam

ple

fact

or

Columnfactor


Source of Variation SS df MS F P-value F critSample 0.755233 2 0.377617 2.755097 0.103596 3.885294Columns 3.582272 1 3.582272 26.13631 0.000256 4.747225Interaction 0.139411 2 0.069706 0.508573 0.613752 3.885294Within 1.644733 12 0.137061

Total 6.12165 17

A B C11.0 12.8 9.4 9.3 11.3 12.4

11.5 9.5 16.8 16.4 14.0 14.3 16.0 15.2 17.0 15.0 13.0 14.6 12.8 12.4 17.0 13.6 15.0 14.3 13.0 12.4 15.6 12.0 17.8 15.0 13.4 12.6 18.6 10.0 13.4 12.4 10.8 16.8 15.4

Sam

ple

fact

or

Columnfactor

Q1: Where is significant among A, B, and C?A1: Apply multiple comparisons between all pairs

among columns.(Fisher's PLSD method, Scheffé method, Bonferroni-Dunn test, Dunnett method, Williams method, Tukey method, Nemenyi test, Tukey-Kramer method, Games/Howell method, Duncan's new multiple range test, Student-Newman-Keuls method, etc. Each has different characteristics.)

significant?

Non-Parametric Tests2 groups n groups (n > 2)

datadistribution




・ one-way ANOVA

・ two-way ANOVA(n

o no

rmal

ity)

one-waydata

two-waydata

Para

met

ric T

est

Non

-par

amet

ric T

est

(nor

mal

ity)

unpa

ired

(inde

pend

ent)

paire

d(re

late

d)un

paire

d(in

depe

nden

t)pa

ired

(rela

ted)

・paired ｔ -test


ANO

VA(A

naly

sis

of V

aria

nce)

If normality and equal variances are not guaranteed, use non-parametric tests.

Mann-Whitney U-test2 groups n groups (n > 2)

datadistribution




・ one-way ANOVA

・ two-way ANOVA

(no

norm

ality

)

one-waydata

two-waydata

Para

met

ric T

est

Non

-par

amet

ric T

est

(nor

mal

ity)

unpa

ired

(inde

pend

ent)

paire

d(re

late

d)un

paire

d(in

depe

nden

t)pa

ired

(rela

ted)

・paired ｔ -test


ANO

VA(A

naly

sis

of V

aria

nce)

Mann-Whitney U-test(Wilcoxon-Mann-Whitney test, two sample Wilcoxon test)

n-th generation

??no normality

1. Comparison of two groups.2. Data have no normality.3. There are no data corresponding

between two groups (independent).

Mann-Whitney U-test(Wilcoxon-Mann-Whitney test, two sample Wilcoxon test)

1. Calculate a U value. 0

2

34

U = 0 + 2 + 3 + 4 = 9U' = 11 (U + U' = n1n2)

when two values are the same, count as 0.5.( )

2. See a U-test table.

Mann-Whitney U-test (cont.)(Wilcoxon-Mann-Whitney test, two sample Wilcoxon test)

• Use the smaller value of U or U'.• When n1 ≤ 20 and n2 ≤ 20 , see a Mann-Whitney test table.

(where n1 and n2 are the # of data of two groups.)• Otherwise, since U follows the below normal distribution roughly,

normalize U as and check a standard normal distribution table

with the , where and .

12

)1(,2

, 2121212 nnnnnnNN UU

U

UUz

221nn

U 12

)1( 2121

nnnnUz

Use an Excel function to calculate the p-value for the z-value:p-value = 1 - NORM.S.DIST( z )

Examples: Mann-Whitney U-test(Wilcoxon-Mann-Whitney test, two sample Wilcoxon test)

U = 9U' = 11

0

2

34

00.5

2.54

5

3.555

5

U = 23.5U' = 1.5

U = 12U' = 13

Ex.1 Ex.2 Ex.3

4 5 6 ・・・

・・・ー・・・・・・・・・

4 0 1 2 ・・・

5 2 3 ・・・

・・・・・・・・・

n1n2

(p < 0.05)4 5 6 ・・・

・・・ー・・・・・・・・・

4 ーー 0 ・・・

5 1 1 ・・・

・・・・・・・・・

n1n2

(p < 0.01)

5

Exercise: Mann-Whitney U-test(Wilcoxon-Mann-Whitney test, two sample Wilcoxon test)

2.5

45

6

66

U = 29.5U' = 6.5

4 5 6 7

3 ー 0 1 1

4 0 1 2 3

5 2 3 5

6 5 6

n1n2

(p < 0.05)4 5 6 7

3 ーーーー

4 ーー 0 0

5 1 1 1

6 2 3

n1n2

(p < 0.01)

Since U' > 5, (p > 0.05): significance is not found( )

Sign Test2 groups n groups (n > 2)

datadistribution




・ one-way ANOVA

・ two-way ANOVA

(no

norm

ality

)

one-waydata

two-waydata

Para

met

ric T

est

Non

-par

amet

ric T

est

(nor

mal

ity)

unpa

ired

(inde

pend

ent)

paire

d(re

late

d)un

paire

d(in

depe

nden

t)pa

ired

(rela

ted)

・paired ｔ -test


ANO

VA(A

naly

sis

of V

aria

nce)

（1）Sign Test

（2）Wilcoxon's Signed Ranks Test

significance test between the # of winnings and losses

173 174

143 137

158 151

156 143

176 180

165 162

- ++ -+ -+ -- ++ -

-1+6+7+13-4+3

data of 2 groups

# of winnings and losses

the level of winnings/losses

Sign Test

significance test using both the # of winnings and losses and the level of winnings/losses

n-th generation

Sign Test

1. Calculate the # of winnings and losses by comparing runs with the same initial data.

2. Check a sign test table to show significance of two methods.

Sign Test Generations 0 10 20 30 40 50 |__________|__________|__________|__________|__________|F1: DE_N vs. DE_LR +++++++++++++++++++++++++++++++++++++++++++++++++F1: DE_N vs. DE_LS +++++++++++++++++++++++++++++++++++++++++++++++++F1: DE_N vs. DE_FR_GLB_nD + +++++++++++++++++++++++++++++++++++++F1: DE_N vs. DE_FR_LOC_nD ++ ++++++++++++++++++++++++++++++++++++F1: DE_N vs. DE_FR_GLB_1D + +++++++++++++++++++++++++++++++++++++F1: DE_N vs. DE_FR_LOC_1D +++ ++++++++++++++++++++++++++++++++++++F2: DE_N vs. DE_LR + ++++++++++++++++++++++++++++++++++++++++++F2: DE_N vs. DE_LS ++++++++++++++++++++++++++++++++++++++++++++F2: DE_N vs. DE_FR_GLB_nDF2: DE_N vs. DE_FR_LOC_nDF2: DE_N vs. DE_FR_GLB_1DF2: DE_N vs. DE_FR_LOC_1DF3: DE_N vs. DE_LRF3: DE_N vs. DE_LSF3: DE_N vs. DE_FR_GLB_nD ++++++++++++++++++++++++F3: DE_N vs. DE_FR_LOC_nDF3: DE_N vs. DE_FR_GLB_1D F3: DE_N vs. DE_FR_LOC_1D + +F4: DE_N vs. DE_LR +++++++++++++++++++++++++++++++++++++++++++++++++F4: DE_N vs. DE_LS ++++++++++++++++++++++++++++++++++++++++++++++++++F4: DE_N vs. DE_FR_GLB_nD ++++++++++++++++++++++++++++++++++++++++++++++F4: DE_N vs. DE_FR_LOC_nDF4: DE_N vs. DE_FR_GLB_1D + ++++++++++++++++++++++++++++++++++++++++++++++F4: DE_N vs. DE_FR_LOC_1D ++++++++++++++++++++++++++++++++++++++++++++++F5: DE_N vs. DE_LR + ++ +F5: DE_N vs. DE_LSF5: DE_N vs. DE_FR_GLB_nD +++++++++++++++++++ F5: DE_N vs. DE_FR_LOC_nD ++F5: DE_N vs. DE_FR_GLB_1D +++ F5: DE_N vs. DE_FR_LOC_1D +F6: DE_N vs. DE_LR ++++++++++++ + +++++++F6: DE_N vs. DE_LS +++++++++++++++++++++++++++++++++++ +++++F6: DE_N vs. DE_FR_GLB_nD + + +++++++++++++++++++++++++++++++++++F6: DE_N vs. DE_FR_LOC_nD +++++++++++++++++++++++++++++++++F6: DE_N vs. DE_FR_GLB_1D + + +++++++++++++++++++++++++++++++++++F6: DE_N vs. DE_FR_LOC_1D +++++++++ ++++++++++++++++++++++ +++F7: DE_N vs. DE_LR +F7: DE_N vs. DE_LSF7: DE_N vs. DE_FR_GLB_nD ++++++ + ++F7: DE_N vs. DE_FR_LOC_nD +F7: DE_N vs. DE_FR_GLB_1D +F7: DE_N vs. DE_FR_LOC_1DF8: DE_N vs. DE_LR +++++++++++++++++++++++++++++++++++F8: DE_N vs. DE_LS ++++++++++++++++++++++++++++++++++++++++++++++++F8: DE_N vs. DE_FR_GLB_nD +++++++++++++++++++++++++++++++++++++++++++++++F8: DE_N vs. DE_FR_LOC_nD +F8: DE_N vs. DE_FR_GLB_1D +++++++++++++++++++++++++++++++++++F8: DE_N vs. DE_FR_LOC_1D +++ +++++++++++++++++++++++++++++++++++++++++++++

Fig.3 in Y. Pei and H. Takagi, "Fourier analysis of the fitness landscape for evolutionary search acceleration," IEEE Congress on Evolutionary Computation (CEC), pp.1-7, Brisbane, Australia (June 10-15, 2012).The (+,-) marks show whether our proposed methods converge significantly better or poorer than normal DE, respectively, (p ≤0.05).

Fig.2 in the same paper.

Task ExampleWhether performances of pattern recognition methods A and B are significantly different?

n1 cases: Both methods succeeded.n2 cases: Method A succeeded, and method B failed.n3 cases: Method A failed, and method B succeeded.n4 cases: Both methods failed.

1. Set N = n2 + n3.2. Check the right table with the N.3. If min(n2, n3) is smaller than the number for the N,

we can say that there is significant difference with the significant risk level of XX.

How to check?

Whether there is significant difference for n2 = 12 and n3 = 28?

Exercise

level of significanceSign Test %%

ANSWER:Check the right table with N = 40.As n2 is bigger than 11 and smaller than 13, we can say that there is a significant difference between two with (p < 0.05) but cannot say so with (p < 0.01).

% %

level of significance level of significance

%%Sign Test

Let's think about the case of N = 17.

To say that n1 and n2 are significantly different,(n1 vs. n2) = (17 vs. 0), (16 vs. 1), or (15 vs. 2) (p < 0.01)or(n1 vs. n2) = (14 vs. 3) or (13 vs. 4) (p < 0.05)

Check the significance of:

Exercise: Sign Test level of significance%%

16 vs. 4

14 vs. 1

9 vs. 3

18 vs. 5

Wilcoxon Signed-Ranks Test2 groups n groups (n > 2)

datadistribution




・ one-way ANOVA

・ two-way ANOVA

(no

norm

ality

)

one-waydata

two-waydata

Para

met

ric T

est

Non

-par

amet

ric T

est

(nor

mal

ity)

unpa

ired

(inde

pend

ent)

paire

d(re

late

d)un

paire

d(in

depe

nden

t)pa

ired

(rela

ted)

・paired ｔ -test


ANO

VA(A

naly

sis

of V

aria

nce)

n-th generation

Wilcoxon Signed-Ranks Test

Q: When a sign test could not show significance,how to do?

A: Try the Wilcoxon signed-ranks test. It is more sensitive than a simple sign test due to more information use.

（1）Sign Test

（2）Wilcoxon's Signed Ranks Test

significance test between the # of winnings and losses

173 174

143 137

158 151

156 143

176 180

165 162

- ++ -+ -+ -- ++ -

-1+6+7+13-4+3

data of 2 groups

# of winnings and losses

the level of winnings/losses

significance test using both the # of winnings and losses and the level of winnings/losses



v (system A) v (system B) difference d rank of |d| add sign to the ranks

rank of fewer # of signs

182 163 19 7 7169 142 27 8 8172 173 －1 1 －1 1143 137 6 4 4158 151 7 5 5156 143 13 6 6176 172 4 3 3165 168 －3 2 －2 2

(step 1) (step 2) (step 3) (step 4)

(step 5) )4(# StepofT3

8n

(step 6) Wilcoxon test table

Example:

(step 6) n = 8T = 3

Wilcoxon Test Table: significance point of T

T=3 ≤ 3 (n=8, p<0.05), then difference between systems A and B is significant.

T=3 > 0 (n=8, p<0.01), then we cannot say there is a significant difference.

one-tail p < 0.025 p < 0.005two-tail p < 0.05 p < 0.01

n = 6789

10111213141516171819202122232425

02358

101317212529344046525865738189

013579

121519232732374248546168

When n > 25As T follows the below normal distribution roughly,

normalize T as the below and check a standard normal distribution table with the z; see andin the above equation.

24

)12)(1(,4

)1(, 2 nnnnnNN TT

T

TTz

T T




176 163 13 7 → 6.5 6.5142 142 0172 173 －1 1 －1 1143 137 6 4 4158 151 7 5 5156 143 13 6 → 6.5 6.5176 172 4 3 3165 168 －3 2 －2 2


Tips:1. When d = 0, ignore the data.2. When there are the same ranks of |d|,

give average ranks.

10

1 2

3

4 5

6

7 89

Give the average rank 6.5 = (5+6+7+8)/4.

Tip #1Tip #2

Tip #2

Exercise 1: Wilcoxon Signed-Ranks Test(step 1) (step 2) (step 3) (step 4)



182 163169 142173 172143 137158 151156 143176 172165 168

n = (step 5) )4(# StepofT


T =

n = 8

Exercise 1: Wilcoxon Signed-Ranks Test(step 1) (step 2) (step 3) (step 4)

(step 5) )4(# StepofT




182 163 19 7 7169 142 27 8 8173 172 1 1 1143 137 6 4 4158 151 7 5 5156 143 13 6 6176 172 4 3 3165 168 -3 2 -2 2

T = 2

As T(=2) < 3, there is a significant difference between A and B (p<0.05).But, as 0 < T(=2), we cannot say so with the significance level of (p<0.01).






27 3120 2534 3325 2731 3123 2926 2724 3035 34

n =

Exercise 2: Wilcoxon Signed-Ranks Test

n = 8 (no count for d = 0.)






27 31 -4 5 -520 25 -5 6 -634 33 1 2 2 225 27 -2 4 -431 31 023 29 -6 7.5 -7.526 27 -1 2 -224 30 -6 7.5 -7.535 34 1 2 2 2

(No need to care the case of d = 0.)

T = 4

As T > 3, we cannot say that there is a significant difference between A and B.


n-th generation

Explain how to apply this test to test whether two groups are significantly different at the below generation?


Kruskal-Wallis Test2 groups n groups (n > 2)

datadistribution


・sign test

・Wilcoxon signed-ranks test ・Friedman test


・ one-way ANOVA

・ two-way ANOVA

(no

norm

ality

)

two-waydata

Para

met

ric T

est

Non

-par

amet

ric T

est

(nor

mal

ity)

unpa

ired

(inde

pend

ent)

paire

d(re

late

d)un

paire

d(in

depe

nden

t)pa

ired

(rela

ted)

・paired ｔ -test


ANO

VA(A

naly

sis

of V

aria

nce)

Kruskal-Wallis Test

n-th generation

1. Comparison of more than two groups.2. Data have no normality.3. There are no data corresponding

among groups (independent). ??? no

nor

mal

ity

Kruskal-Wallis TestLet's use ranks of data.

1

23

4 56 7

8 91011 1213

14 1516

17

Kruskal-Wallis TestN: total # of datak: # of groupsni: # of data of group iRi : sum of ranks of group i

R1 = 38 R2 = 69 R3 = 46

1. Rank all data.2. Calculate N, k, ni and Ri .3. Calculate statistical value H.

4. If k = 3 and N ≤ 17, compare the H with a significant point in a Kruskal-Wallis test table.Otherwise, assume that Hfollows the χ2 distribution and test the H using a χ2

distribution table of (k-1) degrees of freedom

)1(3)1(

121

2

NnR

NNH

k

i i

i

1

2 34 56 7

8 91011 121314 15

16

17

How to Test

Example: Kruskal-Wallis Testn1 n2 n3 p < 0.05 p < 0.01 n1 n2 n3 p < 0.05 p < 0.012 2 2 - - 3 3 3 5.606 7.200 2 2 3 4.714 - 3 3 4 5.791 6.746 2 2 4 5.333 - 3 3 5 6.649 7.079 2 2 5 5.160 6.533 3 3 6 5.615 7.410 2 2 6 5.346 6.655 3 3 7 5.620 7.228 2 2 7 5.143 7.000 3 3 8 5.617 7.350 2 2 8 5.356 6.664 3 3 9 5.589 7.422 2 2 9 5.260 6.897 3 3 10 5.588 7.372 2 2 10 5.120 6.537 3 3 11 5.583 7.418 2 2 11 5.164 6.766 3 4 4 5.599 7.144 2 2 12 5.173 6.761 3 4 5 5.656 7.445 2 2 13 5.199 6.792 3 4 6 5.610 7.500 2 3 3 5.361 - 3 4 7 5.623 7.550 2 3 4 5.444 6.444 3 4 8 5.623 7.585 2 3 5 5.251 6.909 3 4 9 5.652 7.614 2 3 6 5.349 6.970 3 4 10 5.661 7.617 2 3 7 5.357 6.839 3 5 5 5.706 7.578 2 3 8 5.316 7.022 3 5 6 5.602 7.591 2 3 9 5.340 7.006 3 5 7 5.607 7.697 2 3 10 5.362 7.042 3 5 8 5.614 7.706 2 3 11 5.374 7.094 3 5 9 5.670 7.733 2 3 12 5.350 7.134 3 6 6 5.625 7.725 2 4 4 5.455 7.036 3 6 7 5.689 7.756 2 4 5 5.273 7.205 3 6 8 5.678 7.796 2 4 6 5.340 7.340 3 7 7 5.688 7.810 2 4 7 5.376 7.321 4 4 4 5.692 7.654 2 4 8 5.393 7.350 4 4 5 5.657 7.760 2 4 9 5.400 7.364 4 4 6 6.681 7.795 2 4 10 5.345 7.357 4 4 7 5.650 7.814 2 4 11 5.365 7.396 4 4 8 5.779 7.853 2 5 5 5.339 7.339 4 4 9 5.704 7.910 2 5 6 5.339 7.376 4 5 5 5.666 7.823 2 5 7 5.393 7.450 4 5 6 5.661 7.936 2 5 8 5.415 7.440 4 5 7 5.733 7.931 2 5 9 5.396 7.447 4 5 8 5.718 7.992 2 5 10 5.420 7.514 4 6 6 5.724 8.000 2 6 6 5.410 7.467 4 6 7 5.706 8.039 2 6 7 5.357 7.491 5 5 5 5.780 8.000 2 6 8 5.404 7.522 5 5 6 5.729 8.028 2 6 9 5.392 7.566 5 5 7 5.708 8.108 2 7 7 5.398 7.491 5 6 6 5.765 8.124 2 7 8 5.403 7.571

Kruskal-Wallis Test Table(for k = 3 and N ≤17)

N = n1+n2+n3 = 17 datak = 3 groups(n1, n2, n3) = (6, 5, 6)(R1, R2, R3) = (38, 69, 46)

)1(3)1(

121

2

NnR

NNH

k

i i

i

)117(36

46*465

69*696

38*38)117(17

12

= 6.609

Since significant points of (p<0.05) and (p<0.01) for (n1, n2, n3) = (6, 5, 6) are 5.765 and 8.124, respectively, there are significant difference(s) somewhere among three groups (p<0.05).

6.6098.1245.765

significance point of (p<0.05)


Example: Kruskal-Wallis Testn1 n2 n3 p < 0.05 p < 0.01 n1 n2 n3 p < 0.05 p < 0.012 2 2 - - 3 3 3 5.606 7.200 2 2 3 4.714 - 3 3 4 5.791 6.746 2 2 4 5.333 - 3 3 5 6.649 7.079 2 2 5 5.160 6.533 3 3 6 5.615 7.410 2 2 6 5.346 6.655 3 3 7 5.620 7.228 2 2 7 5.143 7.000 3 3 8 5.617 7.350 2 2 8 5.356 6.664 3 3 9 5.589 7.422 2 2 9 5.260 6.897 3 3 10 5.588 7.372 2 2 10 5.120 6.537 3 3 11 5.583 7.418 2 2 11 5.164 6.766 3 4 4 5.599 7.144 2 2 12 5.173 6.761 3 4 5 5.656 7.445 2 2 13 5.199 6.792 3 4 6 5.610 7.500 2 3 3 5.361 - 3 4 7 5.623 7.550 2 3 4 5.444 6.444 3 4 8 5.623 7.585 2 3 5 5.251 6.909 3 4 9 5.652 7.614 2 3 6 5.349 6.970 3 4 10 5.661 7.617 2 3 7 5.357 6.839 3 5 5 5.706 7.578 2 3 8 5.316 7.022 3 5 6 5.602 7.591 2 3 9 5.340 7.006 3 5 7 5.607 7.697 2 3 10 5.362 7.042 3 5 8 5.614 7.706 2 3 11 5.374 7.094 3 5 9 5.670 7.733 2 3 12 5.350 7.134 3 6 6 5.625 7.725 2 4 4 5.455 7.036 3 6 7 5.689 7.756 2 4 5 5.273 7.205 3 6 8 5.678 7.796 2 4 6 5.340 7.340 3 7 7 5.688 7.810 2 4 7 5.376 7.321 4 4 4 5.692 7.654 2 4 8 5.393 7.350 4 4 5 5.657 7.760 2 4 9 5.400 7.364 4 4 6 6.681 7.795 2 4 10 5.345 7.357 4 4 7 5.650 7.814 2 4 11 5.365 7.396 4 4 8 5.779 7.853 2 5 5 5.339 7.339 4 4 9 5.704 7.910 2 5 6 5.339 7.376 4 5 5 5.666 7.823 2 5 7 5.393 7.450 4 5 6 5.661 7.936 2 5 8 5.415 7.440 4 5 7 5.733 7.931 2 5 9 5.396 7.447 4 5 8 5.718 7.992 2 5 10 5.420 7.514 4 6 6 5.724 8.000 2 6 6 5.410 7.467 4 6 7 5.706 8.039 2 6 7 5.357 7.491 5 5 5 5.780 8.000 2 6 8 5.404 7.522 5 5 6 5.729 8.028 2 6 9 5.392 7.566 5 5 7 5.708 8.108 2 7 7 5.398 7.491 5 6 6 5.765 8.124 2 7 8 5.403 7.571

Kruskal-Wallis Test Table(for k = 3 and N ≤17)

N = n1+n2+n3 = 17 datak = 3 groups(n1, n2, n3) = (6, 5, 6)(R1, R2, R3) = (38, 69, 46)

)1(3)1(

121

2

NnR

NNH

k

i i

i

)117(36

46*465

69*696

38*38)117(17

12 3

1

2

i i

i

nR

= 6.609

Since significant points of (p<0.05) and (p<0.01) for (n1, n2, n3) = (6, 5, 6) are 5.765 and 8.124, respectively, there are significant difference(s) somewhere among three groups (p<0.05).

6.6098.1245.765



Q1: Where is significant among A, B, and C?A1: Apply multiple comparisons between all pairs


R1 = R2 = R3 =

1

2

4

7

10

8

111213

3

5

6

9

N = n1+n2+n3 =k =(n1, n2, n3) =(R1, R2, R3) =

)1(3)1(

121

2

NnR

NNH

k

i i

i

=

Exercise: Kruskal-Wallis Test

24 44 23

13 samples3 groups( 5, 4, 4)(24, 44, 23)

6.227

7.7605.657



6.227

There is/are significant difference(s) somewhere among three groups (p<0.05).

Friedman Test2 groups n groups (n > 2)

datadistribution


・sign test

・Wilcoxon signed-ranks test ・Friedman test


・ one-way ANOVA

・ two-way ANOVA(n

o no

rmal

ity) one-way

data

Para

met

ric T

est

Non

-par

amet

ric T

est

(nor

mal

ity)

unpa

ired

(inde

pend

ent)

paire

d(re

late

d)un

paire

d(in

depe

nden

t)pa

ired

(rela

ted)

・paired ｔ -test


ANO

VA(A

naly

sis

of V

aria

nce)

Friedman TestWhen

(1) more than two groups, (2) data have correspondence (not independent), but(3) the conditions of two-way ANOVA are not satisfied,

Let' use ranks of data and Friedman test.

benchmarktasks

methods

a b c d

A 0.92 0.75 0.65 0.81

B 0.48 0.45 0.41 0.52

C 0.56 0.41 0.47 0.50

D 0.61 0.50 0.56 0.54

(ex.) Comparison of recognition rates. a b c dmethods

12

3

4

1

2 34 1 234

1

234

Friedman TestStep 1: Make a ranking table.Step 2: Sum ranks of the factor that you want to test.

Step 3: Calculate the Friedman test value, χ2r .

Step 4: If k =3 or 4, compare χ2r with a significant point

in a Friedman test table. Otherwise, use a χ2 table of (k-1) degrees of freedom.

methods

12

34

1

2 34 1 234

123

4

a b c d

)1(3)1(

121

22

knRknk

k

iir

where (k, n) are the # of levels of factors 1 and 2.

# of

dat

a(n

= 4)

# of methods (k = 4)

benchmarktasks

methoda b c d

A 4 2 1 3 B 3 2 1 4 C 4 1 2 3 D 4 1 3 2

Σ 15 6 7 12

ranking among methods

Example: Friedman TestStep 1: Make a ranking table.Step 2: Sum ranks of the factor that you want to test.


Step 4: Since significant point for (k,n) = (4,4) is7.80,there is/are significant difference(s) somewhereamong four methods, a, b, c, and d (p<0.05).

1.8

5*4*31276155*4*4

12

)1(3)1(

12

2222

1

22

knRknk

k

iir

# of

dat

a (n

= 4)


benchmarktasks

methoda b c d

A 4 2 1 3 B 3 2 1 4 C 4 1 2 3 D 4 1 3 2

Σ 15 6 7 12

k n p<0.05 p<0.01

3

3 6.00 －

4 6.50 8.00 5 6.40 8.40 6 7.00 9.00 7 7.14 8.86 8 6.25 9.00 9 6.22 9.56 ∞ 5.99 9.21

4

3 7.40 9.00 4 7.80 9.60 5 7.80 9.96 ∞ 7.81 11.34

Friedman test table.

8.19.67.8



Example: Friedman TestStep 1: Make a ranking table.Step 2: Sum ranks of the factor that you want to test.


Step 4: Since significant point for (k,n) = (4,4) is7.80,there is/are significant difference(s) somewhereamong four methods, a, b, c, and d (p<0.05).

1.8

5*4*31276155*4*4

12

)1(3)1(

12

2222

1

22

knRknk

k

iir

# of

dat

a (n

= 4)


benchmarktasks

methoda b c d

A 4 2 1 3 B 3 2 1 4 C 4 1 2 3 D 4 1 3 2

Σ 15 6 7 12

k n p<0.05 p<0.01

3

3 6.00 －

4 6.50 8.00 5 6.40 8.40 6 7.00 9.00 7 7.14 8.86 8 6.25 9.00 9 6.22 9.56 ∞ 5.99 9.21

4

3 7.40 9.00 4 7.80 9.60 5 7.80 9.96 ∞ 7.81 11.34

Friedman test table.

8.19.67.8



Q1: Where is significant among a, b, c, or d?

A1: Apply multiple comparisons between all pairsamong columns.

(Fisher's PLSD method, Scheffé method, Bonferroni-Dunn test, Dunnett method, Williams method, Tukey method, Nemenyi test, Tukey-Kramer method, Games/Howell method, Duncan's new multiple range test, Student-Newman-Keuls method, etc. Each has different characteristics.)

Multiple Comparisons

When there is significant difference among groups, multiple comparison is used to know which group is significantly difference from others.

4C2 = 6 times of pair comparisons with (p < 0.05)

1 - (1 - 0.05)6 = significance level 26.5%!

Example

Multiple Comparisons

When there is significant difference among groups, multiple comparison is used to know which group is significantly difference from others.

4C2 = 6 times of pair comparisons with (p < 0.05)

1 - (1 - 0.05)6 = significance level 26.5%!

Solution is to apply multiple pair comparisons with more strict significance level.

Example

Multiple Comparisons-- Bobferroni method --

When pair comparisons are applied m times,let's use a significance level of p / m.

4C2 = 6 times of pair comparisons with (p < )0.056

Features:(1) Simple.(2) Rather strict, i.e. showing significances is rather hard.

Multiple Comparisons-- Holm method --

Corrected Bonferroni method to detect significances easily.

Example pair comparisons p-value corrected

p-value eqn.corrected p-value

0.0076 = p-value* 6 0.0456

0.0095 = p-value* 5 0.0475

0.0280 = p-value* 4 0.1120

0.0320 = p-value* 3 0.0960

0.0380 = p-value* 2 0.0760

0.0410 = p-value* 1 0.0410

vs.

vs.

vs.

vs.

vs.

vs.

normality(parametric)


datadistribution

ｔ -test

・sign test

・Wilcoxon Signed-Ranks Test

ANOVA(Analysis of

Variance)

・Friedman test

・kruskal-wallis test

one-way ANOVA

two-way ANOVA

no normality(non-parametric)

one-waydata

two-waydata

Scheffé's Method of Paired Comparison


IEC

lighting design of 3-D CG

measuring mental scale

geological simulation

hearing-aid fitting

CorridorW

K

Wall

LB

Verenda

MEMS design

EvolutionaryComputation

Target System

subjectiveevaluations

Interactive Evolutionary Computation

imag

e en

hanc

emen

t pro

cess

ing

room layoutplanning design

room lighting design byoptimizing LED assignments

Can you hear me ?

??


ANOVA based on nC2 paired comparisons for n objects.

ANOVAevenbetter betterslightly better

slightly better

evenbetter betterslightly better

slightly better


slightly better

significance checkusing a yardstick


Original method and three modified methodsAll subjects must evaluate all pairs.

no yes

yesoriginal

(原法, 1952)Ura's variation（浦の変法, 1956)

no Haga's variation（芳賀の変法）

Nakaya's variation（中屋の変法, 1970)or

der e

ffect

Order Effect

(1) and then

(2) and then

may result different evaluation.


Scheffé's Method of Paired Comparison1. Ask N human subjects to evaluate t objects in 3, 5 or 7 grades. 2. Assign [-1, +1], [-2, +2] or [-3, +3] for these grades.3. Then, start calculation (see other material).


slightly better


slightly better


slightly better

O1 O2 O3 O4 O5 O6

A1 - A2 2 1 1 2 1 2

A1 - A3 2 2 1 1 1 1

A2 - A3 1 0 1 1 -1 0

Six subjects (N = 6)

Paire

d co

mpa

rison

sfo

r t=3

obj

ects

.

Questionnaire Total row data

・・・

strap for a mobile phone

invitation to a dinner

tea /coffee stuffed animal fountain pen

Ex. Q.

Application Example:

What is the best present to be her/his boy/girl friend?

[SITUATION] He/he is my longing. I want to be her/his boy/girl friend before we graduate from our university. To get over my one-way love, I decided to present something of about 3,000 JPY and express my heart.

I show you 5C2 pairs of presents. Please compare each pair and mark your relative evaluation in five levels.

・・・・

present from a male(significant difference)

Results of Scheffé's Method of Paired Comparison (Nakaya's variation)

What is the best present to be her/his boy/girl friend?

present from a female

I thi

nk e

ffect

ive.

Rea

lity

is ..

.

-1 -0.5 0 0.5 1

I will catch her heart by dinner.

mor

eef

fect

ive

less

effe

ctiv

e

-1 -0.5 0 0.5 1

How about tea leave or a stuffed anima?

mor

eef

fect

ive

less

effe

ctiv

e

-1 -0.5 0 0.5 1

Eat! Eat! Eat!

mor

eef

fect

ive

less

effe

ctiv

e

-1 -0.5 0 0.5 1

mor

eef

fect

ive

less

effe

ctiv

e

I hesitate to accept it as we have not gone about with him.


no yes

yes original(原法, 1952)

Ura's variation（浦の変法, 1956)



der e

ffect

y yScheffé's Method of Paired Comparison

Modified methods by Ura and Nakaya

yScheffé's Method of Paired Comparison

Modified method by UraPairwise comparisons for objects which are effected by display order (order effect).


slightly better


slightly better


slightly better


slightly better


slightly better


slightly better

-2 -1 0 1 2

-2 -1 0 1 2

-2 -1 0 1 2

-2 -1 0 1 2

-2 -1 0 1 2

-2 -1 0 1 2


slightly better


slightly better


slightly better


slightly better


slightly better


slightly better

-2 -1 0 1 2

-2 -1 0 1 2

-2 -1 0 1 2

-2 -1 0 1 2

-2 -1 0 1 2

-2 -1 0 1 2


Modified method by UraAsk N human subjects to evaluate 2×tC2 pairs for tobjects in 3, 5 or 7 grades and assign [-1, +1], [-2, +2] or [-3, +3], respectively.


Modified method by UraStep 1: Make paired comparison table of each human subject.


slightly better

-2 -1 0 1 2

-2 -1 0 1 2

-2 -1 0 1 2

-2 -1 0 1 2

-2 -1 0 1 2

-2 -1 0 1 2

A1

A2

A1

A1

A2

A4

A4

A3

A3

A2

A4

A3

・・・

・・・

・・・

A1 A2 A3 A4

A1 0 -1 -1

A2 3 0 0

A3 3 1 -1

A4 3 3 1

ijlx : evaluation value when the l-th human subject compares the i-th object with the j-th object.

SubjectO1

Subject O3

SubjectO2


Modified method by UraStep 1: Make paired comparison table of each human subject.

)(2

1ˆ iii xxtN

-1.1667 0.5417-0.5000 1.1250

A4 A3 A2 A1

1̂2̂3̂4̂

Average of four objects

where t: # of object (4) N: # of human subjects (3)


Modified method by UraStep 2: Make a table summing all subjects' data and

calculate the average evaluations for all objects.

27 13 -12 -28

SxxN

S

Sxxt

S

xxtN

S

i ijjiij

l iilliB

iii

2

2)(

2

)(21

)(21

)(2

1

l i ijijlT

BBT

ilB

xS

SSSSSSS

Sxtt

S

xtNt

S

2

)()(

2)(

2

)1(1

)1(1

freedom. of degree and,,,,,, where )()(

fSSSSSSSS TBB


Modified method by UraStep 3: Make a ANOVA table.

unbiased variance = S/f

F = unbiased varianceunbiased variance of S

for F tests.

SxxN

S

Sxxt

S

xxtN

S

i ijjiij

l iilliB

iii

2

2)(

2

)(21

)(21

)(2

1

l i ijijlT

BBT

ilB

xS

SSSSSSS

Sxtt

S

xtNt

S

2

)()(

2)(

2

)1(1

)1(1


Modified method by Ura

ANOVA table.

-1.1667 0.5417-0.5000 1.1250

A4 A3 A2 A1

1̂2̂3̂4̂There are significant difference among A1 - A4



ANOVA table.


Modified method by UraStep 4: Apply multiple comparisons.

Q1: Where is significant among A1, A2, and A3?A1: Apply multiple comparisons between all pairs.(Fisher's PLSD method, Scheffé method, Bonferroni-Dunn test, Dunnett method, Williams method, Tukey method, Nemenyi test, Tukey-Kramer method, Games/Howell method, Duncan's new multiple range test, Student-Newman-Keuls method, etc. Each has different characteristics.)

Step 4: Apply multiple comparisons between all pairs and find which distance is significant.

Example of a simple multiple comparison.• Calculate a studentized yardstick • When a difference of average > a studentized yardstick,

the distance is significant.



-1.1667 0.5417-0.5000 1.1250

A4 A3 A2 A1

1̂2̂3̂4̂


(See in the next slide.)



tNftqY 2/ˆ),( 2

Step 4: Example of a simple multiple comparisons.

(studentized yardstick)

When (t, f) = (4,21), studentized yardsticks for significance levels of 5% and 1% are:

),,ˆ( 2 Ntwhere are an unbiased variance of Sε, the # of objects, and the #ofhuman subjects; is a studentized range obtained is a statistical test table for t, the degree of freedom of Sε ( ), and the significant level of φ; see these variables in an ANOVA table.

),( ftqf

)21,4(05.0q

Studentized yardstick ),(05.0 ftq2 3 4 5 6 7 8 9 10 12 15 20

1 18.0 27.0 32.8 37.1 40.4 43.1 45.4 47.4 49.1 52.0 55.4 59.6 2 6.09 8.30 9.80 10.9 11.7 12.4 13.0 13.5 14.0 14.7 15.7 16.8 3 4.50 5.91 6.82 7.50 8.04 8.48 8.85 9.18 9.46 9.95 10.5 11.2 4 3.93 5.04 5.76 6.29 6.71 7.05 7.35 7.60 7.83 8.21 8.66 9.23 5 3.64 4.60 5.22 5.67 6.03 6.33 6.58 6.80 6.99 7.32 7.72 8.21 6 3.46 4.34 4.90 5.31 5.63 5.89 6.12 6.32 6.49 6.79 7.14 7.59 7 3.34 4.16 4.68 5.06 5.36 5.61 5.82 6.00 6.16 6.43 6.76 7.17 8 3.26 4.04 4.53 4.89 5.17 5.40 5.60 5.77 5.92 6.18 6.48 6.87 9 3.20 3.95 4.42 4.76 5.02 5.24 5.43 5.60 5.74 5.98 6.28 6.64 10 3.15 3.88 4.33 4.65 4.91 5.12 5.30 5.46 5.60 5.83 6.11 6.47 11 3.11 3.82 4.26 4.57 4.82 5.03 5.20 5.35 5.49 5.71 5.99 6.33 12 3.08 3.77 4.20 4.51 4.75 4.95 5.12 5.27 5.40 5.62 5.88 6.21 13 3.06 3.73 4.15 4.45 4.69 4.88 5.05 5.19 5.32 5.53 5.79 6.11 14 3.03 3.70 4.11 4.41 4.67 4.83 4.99 5.10 5.25 5.46 5.72 6.03 15 3.01 3.67 4.08 4.37 4.60 4.78 4.94 5.08 5.20 5.40 5.66 5.96 16 3.00 3.65 4.05 4.33 4.56 4.74 4.90 5.03 5.15 5.35 5.59 5.90 17 2.98 3.63 4.02 4.30 4.52 4.71 4.86 4.99 5.11 5.31 5.55 5.84 18 2.97 3.61 4.00 4.28 4.49 4.67 4.82 4.96 5.07 5.27 5.50 5.79 19 2.96 3.59 3.98 4.25 4.47 4.65 4.79 4.92 5.04 5.23 5.46 5.75 20 2.95 3.58 3.96 4.23 4.45 4.62 4.77 4.90 5.01 5.20 5.43 5.71 24 2.92 3.53 3.90 4.17 4.37 4.54 4.68 4.81 4.92 5.10 5.32 5.59 30 2.89 3.49 3.84 4.10 4.30 4.46 4.60 4.72 4.83 5.00 5.21 5.48 40 2.86 3.44 3.79 4.04 4.23 4.39 4.52 4.63 4.74 4.91 5.11 5.36 60 2.83 3.40 3.74 3.98 4.16 4.31 4.44 4.55 4.65 4.81 5.00 5.24

120 2.80 3.36 3.69 3.92 4.10 4.24 4.36 4.48 4.56 4.72 4.90 5.13 ∞ 2.77 3.31 3.63 3.86 4.03 4.17 4.29 4.39 4.47 4.62 4.80 5.01

f t


Modified method by UraStep 4: Example of a simple multiple comparisons.


no yes

yes original(原法, 1952)

Ura's variation（浦の変法, 1956)



der e

ffect


Modified methods by Ura and Nakaya


slightly better


slightly better


slightly better

-2 -1 0 1 2

-2 -1 0 1 2

-2 -1 0 1 2


Modified method by NakayaPairwise comparisons for objects that can be compared without order effect.


Modified method by Nakaya1. Ask N human subjects to evaluate t objects in 3, 5 or 7 grades. 2. Assign [-1, +1], [-2, +2] or [-3, +3] for these grades, respectively.3. Then, start calculation (see other material).

O1 O2 O3 O4 O5 O6

A1 - A2 2 3 3 2 0 1

A1 - A3 2 0 0 1 1 0

A2 - A3 -3 -2 -1 -1 -3 -2

Six human subjects (N = 6)

Paire

d co

mpa

rison

sfo

r t=3

obj

ects

.

Questionnaire


slightly better


slightly better


slightly better

-2 -1 0 1 2

-2 -1 0 1 2

-2 -1 0 1 2


Modified method by NakayaStep 1: Make paired comparison table of each human subject.

: evaluation value when the l-th human subject compares the i-th object with the j-th object.

ijlx

ii xtN1̂

Average of four objects

where t: # of object (3)N: # of human subjects (6)


Modified method by NakayaStep 2: Make a table summing all subjects' data and

calculate the average evaluations for all objects.

ii

l iliB

ii

xtN

S

Sxt

S

xtN

S

2

2

..

2.)(

..

1

1

1

SF

SSSSS

Sxt

S

BT

l iliB

of varianceUnbariased varianceUnbiased

1

)(

2.)(

There are significant difference among A1 - A3


Modified method by NakayaStep 3: Make a ANOVA table.

ANOVA table.


Modified method by NakayaStep 4: Apply multiple comparisons.

Q1: Where is significant among A1, A2, and A3?A1: Apply multiple comparisons between all pairs


ANOVA table.


Modified method by NakayaStep 4: Apply multiple comparisons between all pairs and

find which distance is significant.

Example of a simple multiple comparison.• Calculate a studentized yardstick • When a difference of average > a studentized yardstick,

the distance is significant.


1980.263/79.197.6

4506.163/79.160.4

01.0

05.0

Y

Y


Modified method by NakayaStep 4: Example of a simple multiple comparisons.

tNftqY /ˆ),( 2 (studentized yardstick)

),,ˆ( 2 Ntwhere are an unbiased variance of Sε, the # of objects, and the #ofhuman subjects; is a studentized range obtained is a statistical test table for t, the degree of freedom of Sε ( ), and the significant level of φ; see these variables in an ANOVA table.

),( ftqf

(See in the next slide.))5,3(05.0q

Studentized yardstick ),(05.0 ftq2 3 4 5 6 7 8 9 10 12 15 20

1 18.0 27.0 32.8 37.1 40.4 43.1 45.4 47.4 49.1 52.0 55.4 59.6 2 6.09 8.30 9.80 10.9 11.7 12.4 13.0 13.5 14.0 14.7 15.7 16.8 3 4.50 5.91 6.82 7.50 8.04 8.48 8.85 9.18 9.46 9.95 10.5 11.2 4 3.93 5.04 5.76 6.29 6.71 7.05 7.35 7.60 7.83 8.21 8.66 9.23 5 3.64 4.60 5.22 5.67 6.03 6.33 6.58 6.80 6.99 7.32 7.72 8.21 6 3.46 4.34 4.90 5.31 5.63 5.89 6.12 6.32 6.49 6.79 7.14 7.59 7 3.34 4.16 4.68 5.06 5.36 5.61 5.82 6.00 6.16 6.43 6.76 7.17 8 3.26 4.04 4.53 4.89 5.17 5.40 5.60 5.77 5.92 6.18 6.48 6.87 9 3.20 3.95 4.42 4.76 5.02 5.24 5.43 5.60 5.74 5.98 6.28 6.64 10 3.15 3.88 4.33 4.65 4.91 5.12 5.30 5.46 5.60 5.83 6.11 6.47 11 3.11 3.82 4.26 4.57 4.82 5.03 5.20 5.35 5.49 5.71 5.99 6.33 12 3.08 3.77 4.20 4.51 4.75 4.95 5.12 5.27 5.40 5.62 5.88 6.21 13 3.06 3.73 4.15 4.45 4.69 4.88 5.05 5.19 5.32 5.53 5.79 6.11 14 3.03 3.70 4.11 4.41 4.67 4.83 4.99 5.10 5.25 5.46 5.72 6.03 15 3.01 3.67 4.08 4.37 4.60 4.78 4.94 5.08 5.20 5.40 5.66 5.96 16 3.00 3.65 4.05 4.33 4.56 4.74 4.90 5.03 5.15 5.35 5.59 5.90 17 2.98 3.63 4.02 4.30 4.52 4.71 4.86 4.99 5.11 5.31 5.55 5.84 18 2.97 3.61 4.00 4.28 4.49 4.67 4.82 4.96 5.07 5.27 5.50 5.79 19 2.96 3.59 3.98 4.25 4.47 4.65 4.79 4.92 5.04 5.23 5.46 5.75 20 2.95 3.58 3.96 4.23 4.45 4.62 4.77 4.90 5.01 5.20 5.43 5.71 24 2.92 3.53 3.90 4.17 4.37 4.54 4.68 4.81 4.92 5.10 5.32 5.59 30 2.89 3.49 3.84 4.10 4.30 4.46 4.60 4.72 4.83 5.00 5.21 5.48 40 2.86 3.44 3.79 4.04 4.23 4.39 4.52 4.63 4.74 4.91 5.11 5.36 60 2.83 3.40 3.74 3.98 4.16 4.31 4.44 4.55 4.65 4.81 5.00 5.24

120 2.80 3.36 3.69 3.92 4.10 4.24 4.36 4.48 4.56 4.72 4.90 5.13 ∞ 2.77 3.31 3.63 3.86 4.03 4.17 4.29 4.39 4.47 4.62 4.80 5.01

f t SUMMARY


datadistribution




・ one-way ANOVA

・ two-way ANOVA

(no

norm

ality

) one-waydata

two-waydata

Para

met

ric T

est

Non

-par

amet

ric T

est

(nor

mal

ity)

unpa

ired

(inde

pend

ent)

paire

d(re

late

d)un

paire

d(in

depe

nden

t)pa

ired

(rela

ted)

・paired ｔ -test


ANO

VA(A

naly

sis

of V

aria

nce)


1. We overview which statistical test we should use for which case.

2. We can appeal the effectiveness of our experiments with correct use of statistical tests.

Tutorial on StatisticalTests

Documents