Item analysis

ITEMANALYSIS

What is an item?

“a single task or question that usually cannot be broken downinto any smaller units.”An arithmetical problem may be an item, a manipulative taskmay be an item, a mechanical puzzle may be an item.

What are the characteristics of

item?

Do you suffer from insomnia? Yes/NoDo you feel shy in talking with members of the opposite sex? Yes/No

An item should be phrased in such a manner that there is no ambiguity regarding its meaning for both the item writer as well as the examinees who take the test.

Ahmed said that the sun rose in the east. Right/WrongHe said that honesty is the best policy. Right/Wrong

The item should not be too easy or too difficult.

Pakistan came into being on 14th August……… 1946,1947,1948The capital of Pakistan is………… Lahore, Karachi, Islamabad

It should have discriminating power, that is, it must clearlydistinguish between those who possess a trait and those who do not.

Write the outcomes of psychological testing on the society?Do you think our examination system is producing the right manpower?

It should not be concerned with the trivial aspects of the subject-matter,that is, it must only measure the significant aspects of knowledge orunderstanding.

The main organ of our respiratory system is……… brain, heart, lungDefine the respiratory system?

As far as possible, it should not encourage guesswork by the subject.

Why to analyze the items of a test paper?

Item analysis demonstrates how effectively a given test item functions within the total test.

What is item analysis?

Item analysis is a set of procedures, that is applied to know the indicesfor truthfulness(or validly) of items.

Item analysis is a technique through which those items whichare valid and suited to the purpose are selected and the restare either eliminated or modified to suit the purpose.

What are the objectives of item analysis?

Difficulty Value

Item analysis indicates which items are difficult, easy, moderately difficult or moderately easy. In other words, it provide an index of thedifficulty value to each item.

Discrimination Value

It also provide indices of the ability of the item to discriminatebetween inferior and superior. In other words , item analysisindicates the discrimination value of each item. This is knownas item validity.

Distractor Analysis

It indicates the effectiveness of the distractors in multiple-choice items. Since multiple-choice items are one of the most powerful and flexible objective items and many of the standardized tests utilize this form of item. A thorough item analysis is done to indicate the extent to which the distractors or foils are effective in each item.

Function effectiveness

It sometimes also indicates why a particular item in the test has not functioned effectively and how this might be modifiedso that its functional significance can be increased.

Power Tests

A power test is one where the examinee is allowed sufficient time for answering all items of the test. Thus, here emphasis is upon measurement of the ability (or power) ofthe examinee and not upon his speed. Ideally, in such a test, items are of different difficulty values and they are arranged in increasing order of difficulty.

Item Difficulty

In item analysis, the first step is to find out the difficulty value of the itemor the index of difficulty of an item.

The difficulty value of an item is the proportion or percentage of the examinees or individuals who answer the item correctly. This proportionor percentage is known as the index of difficulty of an item.

Item answered by 90% of the examinees is very easywhile the item answered by only 3% of examinees is very difficult anddiscriminating only genius. Also, an item which is answered correctlyby 100% or 0% examinees has no differentiating significance.

50 x 50 = 2500

When an item is answered correctly by 50% of the examinees and answered wrongly by 50%. But this does not mean that the tests composed of items of only 50% difficulty level are always to be preferredbecause such tests would simply divide the total examinees into two groups; those above the 50% difficulty level and those below the 50%level of difficulty.

If a test is composed only to of items of higher indices of difficultyor only of lower indices of difficulty,it will never be a good measuringinstrument.Any test to be called good measuringinstrument must have some itemsof higher indices of difficulty whichshould usually be placed at thebeginning of the test, some items,of moderate indices of difficulty(ranging from 40% through 60%)which should preferable appear afterthat, and some items of lower indices of difficulty which should appear last ofall.

At the beginningof the test can be placed those items which arepassed by 100%,while at the end, those items can be placed which are passed byno one or zeropercent.

Method of Judgment

In this method the difficulty value of an item is determined on the basis of the judgment of experts. Items are given to a group of experts with the instruction to rankthe item in their increasing order of difficulty. Subsequently, the test constructor takesa final decision, keeping in view the commonality of ranks assigned to each item by the different judges or experts. Sometimes, an estimate regarding the index of difficulty ismade by the test constructor himself on the basis of the total time taken by the examinees in its solution.

Empirical Method

The empirical method, also called the statistical method, is the basic and scientificmethod of determining the index of difficulty of an item. There are two common statistical methods through which the index of the difficulty can be estimated. Onemethod determines the index on the basis of the response of all the examinees whereasanother method determines the index on the basis of the responses of only a portion ofthe examinees. Usually, the correct answer in the test is given a score of 1 and the incorrect answer a score of 0.

Optimal Difficulty Value for a Reliable Test

Generally, the test constructor is faced with the question; What should the optimallevel of difficulty of the items be? The question can be answered by referring to what ismeant by optimal level of difficulty. Obviously, most test constructors mean that optimal level of difficulty is one which produces maximum item intercorrelations, and therebymaximum item-total correlation and also a higher reliability of test scores. So, the optimal level has a direct link with the reliability of the test scores. Difficulty levels producing higher reliability are the optimal levels.

Index ofDiscrimination

The determination of the index of discrimination also known as the item validity index is another important aspect in item analysis.

Index of discrimination is that ability of the item on the basis of whichthe discrimination or distinction is made between superiors and inferiors“The degree to which the single item separates the superior from the inferior individuals in the trait or group of traits being measured.”

The discriminatory power or validity of the item may be defined as theextent to which success and failure on that item indicate the possessionof the trait or achievement being measured”From the point of view of discriminatory power, all test item can be divided into items that are either ;i. positively discriminating ii. negatively discriminatingor iii. non-discriminating

A positively discriminating item may be defined as one in which theproportion or percentage of correct answers is higher in the upper group A negatively discriminating item may be defined as one in which proportion or percentage of correct answers is lower in the upper group. Likewise, a non-discriminating item may be defined as one in whichthe percentage or proportion of correct answer is equal or approximately equal in both the upper and lower groups.

There are two common ways of determining the index of discrimination:1. By applying a test of significance of the difference between two proportions or percentage.2. By applying correlational techniques.

A Test of Significance of the Difference between Two Proportions or Percentage

In this method of determining the validity index of an item, the examinees are divided preferably into two equal groups on the basis of the total score.(For this purpose we can take top and bottom 40% or 50% and comparethe proportions of passing in each of these two extreme groups. But common observation has been that when the dichotomy is made near the median, it has little influence upon the discrimination index)

Two Extreme Groups

For the purpose of making two extreme groups, the total scores are arranged from highest to lowest and then the top 27% and bottom 27%are selected. Thus the total score is the criterion for making these two extreme groups.

Correlational Techniques

Correlational techniques have also been frequently employed as the measures of theindex of item discrimination. In such situations each item is correlated against the internal criterion of the total score, that is, each item is validated against the internalcriterion of total score.

Item Analysis Worksheet

Distractor Analysis

Item analysis especially in multiple-choice items aims at determining the effectivenessof the distractors. Indices of item difficulty and discriminative power simply indicate whether an item is doing its intended job, not why or why not. If a particular item is notfunctioning well or is found to be defective, we have to examine its possible reasonsand one obvious method is to examine the distractibility of the incorrect options.

Speed Test

Some standardized tests emphasize the speed of the response of the examinee. They provide a limited time within which an examinee is expected to reach the last item in the test. In other words, they place primary importance upon the speed with which anexaminee can answer the item. Ideally, in a speed test all items should be of uniform degree of difficulty. Often item analysis of such speed tests provides a special problemto the test constructor. The index of difficulty and the index of discrimination cannotbe determined in exactly the same way as is done in the case of a power test.

Index of Difficulty

In a speed test many examinees never reach the items at the end of the test not becauseof their intrinsic difficulty but because of their position in the test. Hence, the usual way to determine the index of difficulty by dividing the number of correct answers by thenumber of the examinees who take the test, would be a misleading index. Hence, in thespeed test the index of difficulty of an item is determined by the number of correctanswers to an item divided by the number of examinees who answer or reach that item(and not the total number of examinees)

Index of Discrimination

The index of discrimination in a speed test cannot be determined through the correlation between the individual item score. Why? The answer is that in a speed testthe items are arranged in an arbitrary order and not from easier to difficult order as inmost power tests. Besides, the time in the speed test does not allow all examinees to reach the last item in the test. Consequently, items placed in the beginning of the test will have a higher difficulty value whereas items placed towards the end of the test willhave a lower difficulty value.

In the case of the speed tests where the reliability is dependent upon the reasonabletime limit allowed for the completion of items. Therefore, in the speed test items areselected according to the principles that will establish a reasonable time limit, which may yield a most reliable distribution of total scores.

The ICC summarizes much of the information conveyed by item analysis. It is a graphicrepresentation of the probability of giving the correct answer to an item as a function of the level of the attribute assessed by the test. In fact ICC serves as a foundation of oneof the most important theories, the item response theory for latent trait theory. This theory was specially developed for the purpose of understanding how individual differences in attributes of the examinees affect his behavior when confronted with a specific item.