Survey Research Chapter 17: How To Design And Evaluate Research In Education

Survey ResearchSurvey ResearchChapter 17: How To Design And Evaluate Chapter 17: How To Design And Evaluate

Research In EducationResearch In Education

James BlackwoodJames BlackwoodAED 615 – Fall Semester 2006AED 615 – Fall Semester 2006

Survey Research TopicsSurvey Research Topics

1.1. What Is A Survey?What Is A Survey?2.2. Why Are Surveys Conducted?Why Are Surveys Conducted?3.3. Types of SurveysTypes of Surveys4.4. Correlational ResearchCorrelational Research5.5. Steps In Survey ResearchSteps In Survey Research6.6. NonresponseNonresponse7.7. Problems With The InstrumentProblems With The Instrument8.8. Evaluating Threats To Internal ValidityEvaluating Threats To Internal Validity9.9. Data AnalysisData Analysis10.10. Journal Article Employing Survey Journal Article Employing Survey

ResearchResearch

What Is A SurveyWhat Is A Survey

1.1. Gathering of a sample of Gathering of a sample of data or opinions data or opinions considered to be considered to be representative of a whole representative of a whole group or population.group or population.

2.2. Information is collected Information is collected primarily by asking primarily by asking questions.questions.

3.3. Information is collected Information is collected from a sample (or portion) from a sample (or portion) of a population. of a population.

Why Are Surveys Why Are Surveys Conducted?Conducted?

1.1. Surveys are primarily used to Surveys are primarily used to describe characteristics of a describe characteristics of a population.population.

2.2. Researchers use surveys to Researchers use surveys to examine the distribution of examine the distribution of the identified characteristics the identified characteristics within the population.within the population.

3.3. The description of the The description of the population as a whole is population as a whole is inferred by the results inferred by the results obtained from the sample.obtained from the sample.

Useful Surveys (Arlene Fink, Useful Surveys (Arlene Fink, 1995)1995)

1.1. Specific, measurable objectivesSpecific, measurable objectives

2.2. Sound research designSound research design

3.3. Sound choice of population or Sound choice of population or samplesample

4.4. Reliable and validReliable and valid

5.5. Appropriate analysisAppropriate analysis

6.6. Accurate reporting of resultsAccurate reporting of results

Types of SurveysTypes of Surveys

There are two major There are two major types of surveys..types of surveys..

Cross-Sectional Cross-Sectional SurveysSurveys

&&Longitudinal Longitudinal

SurveysSurveys

Cross-Sectional SurveysCross-Sectional Surveys Information collected from Information collected from

a sample of a a sample of a predetermined populationpredetermined population

Information is collected at Information is collected at approximately one point in approximately one point in timetime

If an entire population is If an entire population is surveyed, it is described as surveyed, it is described as being a census (Like the being a census (Like the US Census that is US Census that is conducted every 10 years). conducted every 10 years).

Longitudinal SurveysLongitudinal SurveysInformation is collected at different points in time to Information is collected at different points in time to

study changes that occur over a time periodstudy changes that occur over a time period

Trend Study – samples a population whose Trend Study – samples a population whose members may change over the course of members may change over the course of the studythe study

Cohort Study - samples a population Cohort Study - samples a population whose members do not change over the whose members do not change over the course of the studycourse of the study

Panel Study – survey the same sample of a Panel Study – survey the same sample of a population at different times during the population at different times during the course of the studycourse of the study

Correlational ResearchCorrelational Research

Correlation of the results of two separate surveys to Correlation of the results of two separate surveys to determine if a relationship existsdetermine if a relationship exists

Scores or results of surveys are examined in this Scores or results of surveys are examined in this method by calculating correlation coefficients or method by calculating correlation coefficients or by preparing contingency tablesby preparing contingency tables

Steps In Survey ResearchSteps In Survey Research

1.1. Define the problemDefine the problem2.2. Identify the populationIdentify the population3.3. Choose the type of instrument to collect Choose the type of instrument to collect

the datathe data4.4. Design, construct, pilot and refine the Design, construct, pilot and refine the

instrumentinstrument5.5. Select a representative sampleSelect a representative sample6.6. Administer the surveyAdminister the survey7.7. Analyze, interpret, and communicate Analyze, interpret, and communicate

your findingsyour findings

NonresponseNonresponse

Nonresponse refers to members of a Nonresponse refers to members of a sample who do not respond to the sample who do not respond to the

instrumentinstrument

Total Nonresponse – not responding to Total Nonresponse – not responding to the entire survey/instrumentthe entire survey/instrument

Item Nonresponse – not responding to Item Nonresponse – not responding to some of the questions within a some of the questions within a

survey/instrumentsurvey/instrument

Reducing NonresponseReducing Nonresponse

Examples of methods to Examples of methods to reduce nonresponse…reduce nonresponse…

1.1. Pretesting an instrumentPretesting an instrument

2.2. Training interviewers Training interviewers

3.3. Providing incentives for Providing incentives for participationparticipation

Contacting non-respondents Contacting non-respondents using with an alternate using with an alternate form of the instrument form of the instrument used for testing for used for testing for nonresponse errornonresponse error

Problems With The Problems With The InstrumentInstrument

Many issues can affect Many issues can affect the validity of an the validity of an instrumentinstrument

1.1. Venue/Time (location or Venue/Time (location or time that the instrument time that the instrument is presented)is presented)

2.2. Question preparation Question preparation (leading questions)(leading questions)

3.3. External issuesExternal issues4.4. Characteristics of the Characteristics of the

data collectordata collector

Evaluating Threats To Internal Evaluating Threats To Internal ValidityValidity

Four main threats to internal validityFour main threats to internal validity1.1. Mortailty – removing data from lost Mortailty – removing data from lost

members of a studymembers of a study2.2. Location threat – data collection in Location threat – data collection in

locations that would affect responseslocations that would affect responses3.3. Instrumentation – defects in the Instrumentation – defects in the

instrument itselfinstrument itself4.4. Instrument decay – interviewers get Instrument decay – interviewers get

tired or are rushedtired or are rushed

Data AnalysisData Analysis1.1. Summarize responses to Summarize responses to

draw conclusions from draw conclusions from the resultsthe results

2.2. Size of sample & Size of sample & percentage of returns percentage of returns reportedreported

3.3. Percentage of total Percentage of total sample responding for sample responding for each item reportedeach item reported

4.4. Percentage of Percentage of respondents who chose respondents who chose each alternative for each each alternative for each question reportedquestion reported

Journal ArticleJournal ArticleExample of Survey ResearchExample of Survey Research

Public Perception of Extension (1996) – JOE 34(4)Paul D. Warner , James A. Christenson, Don A. Dillman, & Priscilla Salant

This article addressed the issues of how people's perception/awareness and use of Extension changed over a 13-year period. Telephone surveys of the U.S. population in 1982 and again in 1995 were utilized as the instrument in this study.

Thanks For Listening!Thanks For Listening!