Survey research for elt

SURVEY RESEARCH for ELT

Parlindungan PardedeChristian University of Indonesia

Defining Surveys• Survey research designs are procedures in quantitative research in

which investigators collect quantitative, numbered data using questionnaires (e.g., mailed questionnaires) or interviews (e.g., one-on-one interviews) and statistically analyze the data to describe the attitudes, opinions, behaviors, or characteristics of the surveyed population and to test research questions or hypotheses. (Creswell, 2008).

• Survey Research is the systematic gathering of information from respondents for the purpose of understanding and/or predicting some aspects of the behavior of the population of interest.

• Language surveys are any studies that gather data on the characteristics and views of informants about the nature of language or language learning through the use of oral interviews or written questionnaires (Brown, 2001),

Goals of Surveys

• To describe trends, such as high school students’ preferences in having native speaker teachers or non-native ones.

• To determine individual opinions about policy issues, such as whether English should be taught in primary schools.

• To identify important beliefs and attitudes of individuals, such as college students’ perceptions on the use of blogs to develop writing skills.

• To get information necessary to evaluate programs in schools, such as the success of using multimedia laboratory in English teaching.

Three Types of Information Surveys Can Provide

• Factual information, such as the characteristics of individual teachers and learners (e.g., students’ age, gender, ethnicity, language background, proficiency level, etc.).

• Behavioral information, i.e. the one that describe what students or teachers have done or regularly do in terms of their language teaching and learning. Such data are frequently collected on language learning strategy questionnaires in which students are asked, for example, to report how often they look up unfamiliar words in a dictionary or make an outline before they write an essay.

• Attitudinal information, which depict the opinions, beliefs, or interests of teachers or learners. These questions are often used in needs analysis research when researchers want to gather information on such topics as what learning goals students have or what skill areas they are most interested in.

KEY CHARACTERISTICS OF SURVEY RESEARCH

• Sampling from a population• Collecting data through questionnaires or

interviews • Designing instruments for data collection• Obtaining a high response rate

Types of Survey: CROSS-SECTIONAL

A cross-sectional study is one that produces a ‘snapshot’ of a population at a particular point in time. The researcher collects data at one point in time in order to:(1) to examine current attitudes, beliefs, opinions, or

practices;(2) to compare two or more educational groups (students

with students, students with teachers, students with parents) in terms of attitudes, beliefs, opinions, or practices;

(3) to measure community needs of educational services as they relate to programs, courses, or school facilities projects;

(4) to evaluate a program, such as a survey that provides useful information to decision makers.

Types of Survey: LONGITUDINAL

• The researcher collects data to study individuals over time. This design is differentiated into: (1) Trend study, involves identifying a population and examining

changes within that population over time, e.g., Gallup Poll, which is used during elections to monitor trends in the population of voters from the primary to the final election;

(2) cohort study, in which a researcher identifies a subpopulation based on some specific characteristic and then studies that subpopulation over time. For instance, a group of 18-year-old students is studied in the year 2001. Five years later (in 2006), a group of 23-year-olds is studied. (They may or may not be the same individuals studied in 2001.) Five years after that (in 2011), a group of 28-year-olds is studied.

CROSS-SECTIONAL vs. LONGITUDINAL SURVEY

Designing Surveys

Determine purpose

Overall study design

Selecting the sample

Revise questionnaire (if necessary)

Conduct pilot test

Designing the questionnaire

Conduct Research Analyze Data Writing Report

Questionnaire DesignTips for designing questionnaire:1) Consider whether a survey instrument is available to

measure your variables2) Consider modifying an existing instrument3) design your own instrument by following 3 steps:

Write different types of questions, including personal, attitudinal, and behavioral questions; sensitive questions; and closed-and open-ended questions.

Use strategies for good question construction, i.e. using clear language, making sure the answer options do not overlap, and posing questions that are applicable to all participants.

Perform a pilot test of the questions, and make revision based on obtained feedback.

Guidelines for Designing Questionnaire1. Keep the questionnaire sufficiently short (30 minutes maximum).

2. Avoid jargon. Seek simplicity but avoid being condescending.

3. Keep questions short, as long and complex questions are difficult to understand.

4. Split double-barrelled questions, e.g. How long have you studied English and been in receipt of Government support? into two questions.

5. Avoid leading questions which encourage a particular answer, e.g. Do you agree that your supervisor is supportive?

6. Avoid negatively framed questions or statements which are difficult to understand, particularly when asked to agree or disagree.

7. Make sure respondents have the knowledge, otherwise you may get false answers.

8. Check terms are suitable for the context in which they are used, as meanings may vary for different age groups, religions, cultures etc.

9. Ensure the frame of reference for each question is clear e.g. when asking for the frequency of an event, specify the time period.

10. Avoid creating opinions. Respondents do not necessarily hold opinions on topics. Allow a no opinion alternative.

11. Use personal wording if you want the respondents to express their feelings.

A good questionnaire …• is complete, i.e. gets all the data you need;• is short, i.e. doesn't abuse the respondents’ time or concentration;• asks only relevant questions;• gives clear instructions;• has precise, unambiguous and understandable questions;• has objective questions, i.e. doesn't suggest answers;• starts with general questions;• has appropriate questions;• puts sensitive questions at the end; is complete, i.e. gets all the data you need;• is short, i.e. doesn't abuse the respondents7 time or concentration;• asks only relevant questions;• 0 gives clear instructions;• has precise, unambiguous and understandable questions;• has objective questions, i.e. doesn't suggest answers;• starts with general questions;• has appropriate questions;• puts sensitive questions at the end;

Types of Sampling MethodsSampling Techniques

Non-probabilitySampling Techniques

ProbabilitySampling Techniques

ConvenienceSampling

JudgmentalSampling

QuotaSampling

SnowballSampling

SystematicSampling

StratifiedSampling

ClusterSampling

Other SamplingTechniques

Simple RandomSampling

DATA COLLECTING TECHNIQUES

Surveys

Questionnaire

Telephone

Mail

Web/Email

Face-to-Face Interviews

Data Analysis Techniques• DESCRIPTIVE. Deals with the question of ‘what’ things

are like, not ‘why’ they are that way, and includes means, standard deviations, frequency counts, graphs, and charts.

• ANALYTICAL. Seeks to explain relationships, causes or consequences, and include bivariate and multivariate analyses such as correlations, cross-tabulations and regressions

• CONTEXTUAL. Narrows down the context by reinterpreting the data for subgroups. E.g. EFL students vs. ESL students, academic vs non-academic employees.

Instrument Validity

• Construct validity: Does the questionnaire really measure the construct being examined?

• Criterion-related validity. Does the instrument accurately predicts (predictive validity) or diagnoses (concurrent validity) some particular variable (criterion).

• Content validity. Does the contents of the questionnaire really measure the variable being measured ? To achieve this, compare your questionnaire to existing related instruments. If none exists, gather expert opinion on each question on the instrument to determine whether or not it actually tests what it is supposed to.

Measures for Assuring a Survey’s Reliability

• Giving the same survey on two occasions to the same individuals and checking the consistency of the same response to the same item.

• Having the same individuals taking two forms of a survey.

• check the internal consistency of responses in a survey, i.e. seeing how consistently the same respondents answer similar questions formulated in different forms

References

McKay, S. L. (2006). Researching second language classrooms. New Jersey: Lawrence Erlbaum Associates, Inc., Publishers

Burns. A. (2010). Doing action research in english language teaching: A guide for practitioners. New York: Routledge:

Creswell, J. W. 2008. Educational research: Planning, conducting, and evaluating quantitative and qualitative research. New Jersey: Pearson

Denscombe, M. (2010). The good research guide for small-scale social research projects. New York: McGraw-Hill

Goddard, W & Melville, S. (2006). Research methodology: An introduction. Lansdowne: Juta & Co, Ltd.

Ross, Kenneth N. (ed.). (2005). Educational research: Some basic concepts and terminology. Paris: International Institute for Educational Planning/ UNESCO.