Objectives At the end of the lecture students will be able to 1.Identify instruments used for quantitative data collection. 2.Explain the standards.

EDRS6208Lecture ThreeInstruments

andInstrumentationData Collection

Objectives

At the end of the lecture students will be able to1. Identify instruments used for quantitative

data collection.2. Explain the standards used for judging the

quality of data collection.3. Define Survey Research4. Explain the factors to consider when

designing a survey research.

Data Collection

• The purpose of data collection is to learn something about other people or things. The focus is on the particular attribute or quality of the person or setting( Mertens, 1998, p. 286).

Planning data Collection

• Identify the attributes of interest• Decide how to collect data about these

attributes.• Consider the target population• Establish quality of the data collected

Composition of Instrument

• Constructs need to be defined ( underlying ability measured)

• Target population must be taken into consideration

• Validity of instrument• Reliability of instrument

Standards for judging quality of data collection in quantitative research

Reliability

the extent to which an instrument would yieldconsistent results when the characteristic beingmeasured has not changed ( Leedy & Ormrod,

2005, p.93).

Forms of Reliability

• Reliability with observers– Interrater reliability– Intrarater reliability

• Internal consistency• Repeated measures– Test-retest reliability– Parallel forms

Reliability with observers

• Interrater reliability: Data collected through observations, the researcher ascertains reliability between two independent observers or raters.

• Intrarater reliability: comparisons made between two data collection efforts by the same observer.

Internal Consistency

• Estimates reliability by grouping questions in a questionnaire that measure the same concept.

Example: Write two sets of three questions that measure

the same concept after collecting the responses run a correlation between those two groups of three questions to determine if the instrument is reliable.

Repeated Measures Reliability

• Test-retest: An instrument is administered to a group of individuals twice. The second administration can occur immediately or after a time delay. Scores are compared.

• Parallel forms: similar to test-retest instead an equivalent form of the test is administered the second time.

Validity

• Validity the extent to which the instrument and or design measure what they actually intend to measure.

• Instrument Validity in measurement deals with the degree to which the concepts studied are accurately represented by items on the instrument.

• Research Design Validity the degree to which the research design adequately represents the concepts being studied.

Types of Validity

• Content• Criterion– Concurrent– predictive

• Internal• External

Content Validity

• The extent to which the a measurement instrument is a representative sample of the content area being measured.

• Important in studies that purport to compare two or more different curricular, teaching strategies.

Criterion validity

• The extent to which the results of an assessment instrument correlate with another ( it is theory based)

• Predictive: the extent to which the instrument predicts the outcomes that are expected.

• Concurrent: the extent to which the scores in the instrument agree with scores on other factors that are expected to relate to them.

Internal validity

Establishes whether there is a relationship between the predictors and response:

Example:Did the attendance policy cause class

participation to increase? Predictor: attendance policyResponse: class participation

External Validity

• The ability to generalise the of the study to other settings.– From one sample to another– From one setting to another

Threats to internal validity

1. History2. Maturation3. Testing4. Instrumentation5. Mortality6. Regression

History

• This happens when a historical event affects the data collection process. In the earlier example, it would not mean that a stricter attendance policy that caused an increase in participation. If a number of students were expelled earlier because they did not participate in school activities this would impact on the results.

Maturation

• When changes in the participant over time impact on the outcome, for example, if the students in the sample “grew up” and realised the importance of participation in class. That may increase their participation and not the stricter attendance policy.

Testing

• The impact of pre-test on subsequent tests. For example, if you measured students participation prior to implementing the attendance policy and students were forewarned that there was an emphasis on participation, that may increase their participation. The outcome may be as a result of a testing threat but not the treatment.

Instrumentation

• The instrument is not precise and does not measure what it intends to measure.

Mortality

• Subjects drop out of the research and it leads to an inflated measure of the effect. For example, if as a result of a stricter attendance policy, most students dropout of a class, leaving only the more serious ones this would mean that the effect is overestimated.

Regression

• There is a tendency for the participants to score close to the mean from the pre-test to the post-test.

Other threats to internal validity Multiple group threats

• Group differences• Groups mature at different rates• The treatment is communicated between

groups• Control group over performs the treatment

group.• The control group gets discouraged when the

treatment is withheld.

Threats to External validity

• Hawthorn effect: impact on the response when participants know they are subjects of an experiment.

• Novelty and disruptions effect: the experimenter contributes to the effectiveness of the instrument.

• Explicit description of the treatment: the researcher needs to explain the treatment in detail so others can replicate the study.

• Multiple treatment influence: when using more than one treatment one treatment may influence the effectiveness of another due to sequencing.

• Pre-test sensitisation: the pre-test interacts with the experimental treatment and influences the response.

• Interaction of history and treatment effects: history may interact with the treatment effects which makes replication difficult.

• Measurement of the response: how you measure the dependent variable has to be consistent with the definition.

• Interaction of time measurement and treatment: measuring the dependent variable at different times can produce different results.

Difference between Reliability and Validity

Reliability estimates the consistency of the measurement and validity involves the degree to which you are measuring what you are supposed to measure.

Question:If an instrument does not accurately measurewhat it is supposed to measure, will can it

measure consistency?

Some ways to improve validity

• Ensure goals and objectives are clearly defined.• Match assessment measures with goals and

objectives.• Have instruments reviewed using a variety of

methods.• Compare measures with other measures if

possible.• Control for confounding variables

Data Collection Instruments in Quantitative research

• Archival Database• Surveys/questionnaires• Tests• Attitudinal scales• Interviews• Observations

Survey Design

What is Survey Research?

A method of acquiring information about one or more groups of people – perhaps about their characteristics, opinions, attitudes, or previous experiences – by asking them questions and tabulating their answers ( Leedy and Ormrod, 2005, p. 147).

Aim: to learn about a large population by surveying a sample of the population

Steps conducting a survey

• Establish the goals of the research• Determine the sample• Create the questionnaire• Pre-test the questionnaire• Analyse the data

Goals of Research

• What do you want to learn?• This determines whom you will survey and

what you will ask them. If the goals are unclear then the results will not be clear.

Select your sample

• What is your target population?• What is your sample size?• How will the representative sample be drawn?• How will you avoid biased sample?• What is your sampling strategy?

Survey Type

• Establish the type of survey – Telephone survey– Web-based surveys– Personal interviews– Mail surveys– Email surveys

Factors to consider when choosing a survey method

Speed Email and web-based surveys are fastest methods, followed by telephone interviewing, mail surveys are the slowest.

Cost Personal interviews are the most expensive followed by telephone then mail. Email and web-based are the least expensive for large samples,.

Internet usage

Webpage and email surveys offer significant advantages, but you may not be able to generalise their results to the population

Literacy levels

Illiterate and less educated people early respond to surveys.

Sensitive questions

People are more likely to answer sensitive questions when interviewed directly by a computer in one form or another.

Video, sound, graphics

A need to get reactions to video, music, or a picture limits your options. You can play a video on a web page, in a computer directed interview, or in person. You can play music when using these methods or over a phone. you can show pictures on a web page, computer directed interview, in person and in a mail survey.

Questionnaire Design

• KISS: keep it short and simple.• Start with an introduction or a welcome

message.• Allow a “don’t know” or “ not applicable”

response to all questions except those in which you are certain that the respondents will have a clear answer.

• Include “other” or “none” whenever they are logically possible answers.

Question Types

• Multiple choice• Numeric open end• Text open end• Rating scales• Agreement scales

Question and answer choice order

• Early questions should be easy and pleasant to answer.

• Difficult and sensitive questions near the end of survey when possible.

• Whenever there is a logical or natural order to answer choices use it. ( agree – disagree in that order)

• Randomise the order of related questions.

Pilot the Questionnaire

The last step in questionnaire design is to test the questionnaire with a small sample before you conduct the main research. This will reveal unanticipated problems with question wording.

Class Discussion

1. Do you think a more reliable test is automatically more valid?

2. Think of examples of surveys that it would be best to conduct (a) by mail, (b) face to face (c) on the telephone. What are the advantages and disadvantages of each method in terms of efficiency.

Assignment two

DUE OCTOBER 8, 2010

• PAGE 198. NUMBER 10.• PAGE 347. NUMBERS 1, 2, AND 4.