DAY1 Seminar A

8/10/2019 DAY1 Seminar A

1/23

Day 1

Prof Edwin P. Dinauto


2/23

Lecture 1 objectives develop basic understanding on classical test

theory concepts identify some limitations of using traditional test

data review current research findings on the limitations

of classical testing and of teacher judgment

infer from given test results using traditional testprocedure.

differentiate briefly the nature of Australian andPhilippine curriculum and educational system.


3/23

Nature of education (Australia) Australian schools are independent (observe competency-

based standards but no national curriculum, no state orboard examinations, no prescribed subjects by the State)

Almost all assessment papers are research based and issubmitted once. (no monthly exams, no quizzes, noentrance exams, rule on plagiarism applies, ).

Independent learning is totally observed (debates inclassrooms not necessary, its okay to disagree)

Streaming is done by specialization (i.e. Medical courseswith superior requirements)

Use of the internet is part of the schooling process and notjust a source for information.


4/23

Classical/Traditional Test Theory Development and Related Models...

Classical Test Theory is a theory about test scores thatintroduces three concepts often called observedscore, true score, and error score.

The theory behind it is that the observed score is equalto the true score plus the measurement error (X =T+E).For example, I know 80% of the materials but my scoreis 85% because of lucky guessing. In this case, myobserved score is 85 while my true score is 80. Theadditional five points are due to the measurement

error. It means that there are two unknowns in the equation

and that the equation is not solvable unless somesimplifying assumptions are made.


5/23


The assumptions in classical test theory:

True scores and error scores are uncorrelated. The average error score in the population of examinee is

zero.

Error scores on parallel tests are uncorrelated.

In this formulation, where error scores are defined,true score is the difference between test score and errorscore (T =X E). True score is easily shown to be theexpected test score across parallel forms.

In other formulations of this model (for example Lordand Novick, 1968), true score is defined as the expectedtest score over parallel forms, and then the resultingproperties of error are derived.


6/23


To derive other models within the frameworkof classical test theory, researchers havemoved in many directions including dropping

or revising one or more of the basicassumptions, or adding distributionalassumptions about error and true scores.


7/23


EXCEL DEMO on BASIC Statisticalimplications on TEST RESULTS on theconcept of error and standard deviation in

testing.

Basic concepts on Mean, Standard Deviation,

and Variance (word file)


8/23


The Main issue in CTT: RELIABILITY

The concept on Attenuation Paradox.

(Karl Pearson vs Charles Spearman and the series ofdebates)

What followed next was a move to align item difficulty andprocedures on item analysis.

i.e. John Stocklein (1947); Chung Teh Fan (1952)


9/23


To derive other models within the

framework of classical test theory,researchers have moved in many directionsincluding dropping or revising one or moreof the basic assumptions, or adding

distributional assumptions about error andtrue scores.


10/23


11/23


Still other researchers have extended classical testmodels to specify considerable detail the errorscore by identifying components of error (such aserrors resulting from the scorer, item format, and

the particular test administration) and thendesigning studies to assess these components andtheir impact on test score variance and testreliability (Shavelson & Webb, 1991). In sum, the

field of classical test theory is served by a variety ofmodels.


12/23


Most of the work in classical test theory hasfocused on models of the test-score level (incontrast to item response theory). That is, themodels have linked test scores to true scores rather

than item scores to true scores. However, itemstatistics that represent item difficulty (oftendenotedp) and item discriminating power (oftendenoted r) have been developed, and their

connection to test statistics such as test scores,mean, standard deviation, and reliability are well-known and are used in the test development.


13/23


14/23


Suffice it to say here that these item statistics (and

their variations)have proven valuable in testdevelopment process.

Still, one main shortcoming is that they are sampledependent, and this dependency reduces similar tothe examinee population.

In actual testing, sampling problems may be resolvedbut in relationships that are typically nonlinear,analysis may even become more complicated. (i.e.anxiety and performance)


15/23


4 Common Methods of Testing Reliability (Calmorin, 1997)

Test Retest Method (measure of stability)

Parallel-form method (measure of equivalence)

Split-half method (measure of internal consistency)

Internal consistency method (homogeneity of the research instrument)

(Psychological tests with dichotomously scored items)


16/23


3 Types of Validity (Adapted Youtube Video Presentation)

Content validity (mostly applicable in classroom settings)

Criterion Related validity

Construct validity


17/23

Resulting total scores

ProProvide cess

Person Total

David J 15

Tina M 14

Barry Y 13

Sharon Y 12

Mason B 11

Katrina P 10

Paul R 9

Melissa P 8

Bethany D 7

Heather G 7

Gabrielle D 6

Brian N 5Garry A 4

Felicity T 3

Joshua K 2

Nathan R 1

Peta B 0

Provide the following: (Short task...)

A. One test objective

B. Any statistical interpretation, etc.


18/23

Presentations.........

from volunteer groups


19/23

Criticisms of Large-scale testing (based on recent researches)

Critics on the use of classical approach to measurement on large-scale testing have grown considerably especially among superiorfirst-world countries. Test critics observations and findingscurrently affect the shift from traditional approach to applyingmodern concepts and practices in assessment.

Some inconsistencies from traditional testing include: Traditional tests only provide normative information but not

necessarily diagnostic information (Sloane and Kelly, 2003)

High achievement in one test does not guarantee a similar outcome

in another test given a different preparation (Shepard, 2003) The test disregard higher-order concepts (Gunzerhauser, 2003)

Tests lead to narrowing curricular program and reducing ofinstructional time (Groves, 2002)


20/23

Criticisms of Large-scale testing (based on recent researches)

inconsistencies from traditional testing:

Even a recent US literature confirms that while several lowperforming Schools, Districts, and States asserted test scoregains for at-risk or low-performing students, but those gains

were not confirmed in reliable, national sample testing (Lukeand Woods, 2007).

All these concerns among critics in general may imply thattheir unified concern was not simply on the procedural

conduct of the test but mainly on the less significantimplication of the results and the trivial effects it bringstowards improving student achievement in a global context.


21/23

Criticisms of Teacher Judgment

Some arguments from academics and teachers who express

their concern about the need for refocusing on teachers valuejudgment are:

Teachers assessment have greater impact on learnerperformance than externally imposed tests (Black &

William, 1998) Teachers are the primary change agents (Wyatt-Smith,

2000)

Most information students, parents, and teachers acquireabout learning comes from the classroom (Brookhart,2003)

These ideas from experts are even more realistic andacceptable as they seem to challenge the efforts made bytesting authorities and the amount spent by governments

for the purpose.


22/23

Criticisms of Teacher Judgment

On the call for financing teacher assessment trainings:

Wyatt-Smith recommends, there is an urgent need to invest inteacher judgment, training up through professionaldevelopment programs focusing sharply on assessment, andthrough system support mechanisms including those providedtrough internal and external moderation networks (2000, p. 125)

In US educational setting, Stiggins (2001, p.6) asserts, we stillcannot guarantee the accuracy of the assessments developed andused by teachers in the US and millions of teachers around theworld.

Some authorities even suspect teachers ability to makejudgments reliable enough to measure student growth inlearning (Bond and Caust, 2005; McGaw, cited in Clarke &Gipps, 2000).


23/23

No more criticism......

End of lecture 1.....

DAY1 Seminar A

Documents