1 Chapter 4 Conceptualization and measurement. 2 Every time you begin to review or design a research study, you will have to answer two questions: (1)What.

1

Chapter 4

Conceptualization and measurement

2

Every time you begin to review or design a research study, you will have to answer two questions:

(1)(1) What do the main concepts mean in this research?What do the main concepts mean in this research?

(2) How are the main concepts measured?(2) How are the main concepts measured?

Both questions must be answered to evaluate the validity of any research.

3

ConceptualizationConceptualization

•A continuing challenge for social scientists rests on the fact that many of our important topics of study (social health, for instance) are not clearly defined things or objects (like trees or rocks), but are abstract concepts or ideas.

•A concept concept is an image or idea, not a simple object.

•It is especially important to define clearly concepts that are abstract or unfamiliar.

4

ConceptualizationConceptualization is the process of specifying what we mean by a term.

•In deductive research, conceptualization helps to translate portions of an abstract theory into specific variables that can be used in testable hypotheses.

•In inductive research, conceptualization is an important part of the process used to make sense of related observations.

5

Variables and constantsVariables and constants

•After we define the concepts for a study, we must identify variables that correspond to those concepts.

•The term variablevariable is used to refer to some specific aspect of a concept that varies, and for which we then have to select even more concrete indicators.

•Not every concept in a particular study is represented by a variable.

•Some are called a constantconstant (it’s always the same), not a variable.

6

1. Examine the theories that are relevant to our research question to identify those concepts that would be expected to have some bearing on the phenomena we are investigating.

2. Review the relevant research literature and assess the utility of variables used in prior research.

How do we know what concepts to consider and thenHow do we know what concepts to consider and then which variables to include in a study?which variables to include in a study?

7

3. Consider the constraints and opportunities for measurement that are associated with the specific setting(s) we will study. Distinguish constants from variables in this setting.

4. Look ahead to our analysis of the data. What role will each variable play in our analyses?

8

How Will We Know When We’ve Found It?How Will We Know When We’ve Found It?

•OperationalizationOperationalization is the process of connecting concepts to observations. It is the process of specifying the operations that will indicate the value of cases on a variable.

•You can think of it as the empirical counterpart of the process of conceptualization.

•The goal is to devise operations that actually measure the concepts we intend to measure—in other words, to achieve measurement validity.

9

Insert exhibit 4.1

10

Using available data

•Government reports are rich, accessible sources of social science data.

•The data collected in many social science surveys are archived and made available for researchers who were not involved in the original survey project.

•We also cannot assume that available data are accurate, even when they appear to measure the concept.

•Another rich source of already-collected data is survey datasets archived and made available to university researchers by the Inter-University Consortium for Political and Social Research.

11

Constructing questionsConstructing questions

•Asking people questions is the most common and probably the most versatile operation for measuring social variables.

•Most concepts about individuals can be defined in such a way that measurement with one or more questions becomes an option.

•Of course, in spite of the fact that questions are, in principle, a straightforward and efficient means to measure individual characteristics, facts about events, level of knowledge, and opinions of any sort, can easily result in misleading or inappropriate answers.

12

Different types of questions used in Different types of questions used in social researchsocial research

•Measuring variables with single questions is very popular.

•Public opinion polls based on answers to single questions are reported frequently in newspaper articles and TV newscasts.

•Single questions can be designed with or without explicit response choices.

13

•Closed-ended (fixed-choice) questionClosed-ended (fixed-choice) question: : a survey question that provides preformatted response choices for the respondent to circle or check.

•Open-ended question:Open-ended question: a survey question to which the respondent replies in his or her own words, either by writing or by talking.

14

Indexes and scalesIndexes and scales

•When several questions are used to measure one concept, the responses may be combined by taking the sum or average of responses.

•A composite measure based on this type of sum or average is termed an index.

•Because of the popularity of survey research, indexes already have been developed to measure many concepts.

•In a scale, we give different weights to the responses to different questions before summing or averaging the responses.

15

Insert exhibit 4.2

16

Making observationsMaking observations

•Observations can be used to measure characteristics of individuals, events, and places.

•The observations may be the primary form of measurement in a study, or they may supplement measures obtained through questioning.

•Direct observations can be used as indicators of some concepts.

•Observations may also supplement data collected in an interview study.

17

Content analysisContent analysis

•Content analysisContent analysis is a research method for systematically analyzing and making inferences from text.

•You can think of a content analysis as a “survey” of documents ranging from newspapers, books, or TV shows to persons referred to in other communications, themes expressed in government documents, or propositions made in tape-recorded debates.

•Computer programs for content analysis can be used to enhance reliability.

18

Collecting unobtrusive measuresCollecting unobtrusive measures

•Unobtrusive measuresUnobtrusive measures allow us to collect data about individuals or groups without their directknowledge or participation.

•Eugene Webb and his colleagues (2000) identified fourtypes of unobtrusive measures: physical trace evidence,archives (available data), simple observation, and contrivedobservation (using hidden recording hardware or manipulation to elicit a response).

19

•Unobtrusive measures can also be created from such diverse forms of media as newspaper archives or magazine articles, TV or radio talk shows, legal opinions, historical documents, personal letters, or e-mail messages.

20

Combining measurement operationsCombining measurement operations

•Using available data, asking questions, making observations, and using unobtrusive indicators are interrelated measurementtools, each of which may include or be supplemented bythe others.

•Researchers may use insights gleaned from questioning participants to make sense of the social interaction they have observed.

•Unobtrusive indicators can be used to evaluate the honesty of survey responses.

21

TriangulationTriangulation

•Triangulation—the use of two or more different measures of the same variable—can strengthen measurement considerably.

•When we achieve similar results with different measures of the same variable, particularly when they are based onsuch different methods as survey questions and field-basedobservations, we can be more confident in the validity of each measure.

22

Levels of MeasurementLevels of Measurement

•Level of measurement:Level of measurement: The mathematical precision with which the values of a variable can be expressed.

•When we know a variable’s level of measurement, we can better understand how cases vary on that variableand so understand more fully what we have measured.

•The nominal level of measurement, which is qualitative,has no mathematical interpretation; the quantitative levels of measurement— ordinal, interval, and ratio—areprogressively more precise mathematically.

23

Nominal level of measurementNominal level of measurement

•The nominal level of measurementnominal level of measurement identifies variables whose values have no mathematical interpretation; they vary in kind or quality but not in amount (they may also be called categorical or qualitative variables).

•“State” (referring to the United States) is one example.

•Nationality, occupation, religious affiliation, and region of the country are also measured at the nominal level.

24

Insert exhibit 4.3

25

•Although the attributes of categorical variables do not have a mathematical meaning, they must be assigned to cases with great care. The attributes we use to measure, or categorize, cases must be mutually exclusive and exhaustive:

A variable’s attributes or values are mutually mutually exclusiveexclusive if every case can have only one attribute.

•A variable’s attributes or values are exhaustiveexhaustive when every case can be classified into one of the categories.

26

Ordinal level of measurementOrdinal level of measurement

•The first of the three quantitative levels is the ordinalordinal level of measurementlevel of measurement..

•At this level, the numbers assigned to cases specify only the order of the cases, permitting “greater than” and “less than” distinctions.

• If the response choices to a question range from “very wrong” to “not wrong at all”, for example.

•The different values of a variable measured at the ordinal level must be mutually exclusive and exhaustive.

27

Interval level of measurementInterval level of measurement

•The numbers indicating the values of a variable at the interval level of measurementinterval level of measurement represent fixed measurement units but have no absolute, or fixed, zero point.

•The numbers can therefore be added and subtracted, but ratios are not meaningful.

• Again, the values must be mutually exclusive and exhaustive.

•This level of measurement is represented by the difference between two Fahrenheit temperatures.

28

Insert exhibit 4.4

29

Ratio level of measurementRatio level of measurement

•The numbers indicating the values of a variable at the ratio level of measurementratio level of measurement represent fixed measuring units and an absolute zero point (zero means absolutely no amount of whatever the variable indicates).

•For most statistical analyses in social science research, the interval and ratio levels of measurement can be treatedas equivalent.

•On a ratio scale, 10 is 2 points higher than 8 and is also 2 times greater than 5—the numbers can be compared in a ratio.

30

Comparison of levels of measurementComparison of levels of measurement

•Researchers choose levels of measurement in the process of operationalizing variables; the level of measurement is not inherent in the variable itself. Many variables can be measured at different levels, with different procedures.

•It is usually is a good idea to try to measure variables at the highest level of measurement possible. The more information available, the more ways we have to compare cases.

31

Insert exhibit 4.5

32

Did we measure what we wanted to measure?Did we measure what we wanted to measure?

•Do the operations developed to measure our variables actually do so—are they valid?

• If we have weighed our measurement options, carefully constructed our questions and observational procedures, and selected sensibly from the availabledata indicators, we should be on the right track.

•But we cannot have much confidence in a measure until we have empirically evaluated its validity.

33

Measurement validityMeasurement validity

•Measurement validity refers to the extent to which measures indicate what they are intended to measure.

•When a measure “misses the mark”—when it is not valid—our measurement procedure has been affected by measurement error.

34

Face validityFace validity

•Researchers apply the term face validityface validity to the confidence gained from careful inspection of a concept to see if it is appropriate “on its face.”

•More precisely, we can say that a measure is face valid if it obviously pertains to the meaning of the concept being measured more than to other concepts (Brewer & Hunter 1989:131).

•Although every measure should be inspected in this way, face validation in itself does not provide convincing evidence of measurement validity.

35

Content validityContent validity

•Content validityContent validity establishes that the measure covers the full range of the concept’s meaning.

•To determine that range of meaning, the researcher may solicit the opinions of experts and review literature that identifies the different aspects, or dimensions, of the concept.

36

Criterion validityCriterion validity

•Criterion validityCriterion validity is established when the scores obtained on one measure can be accurately compared to those obtainedwith a more direct or already validated measures of the same phenomenon (the criterion).

•Inconsistent findings about the validity of a measure can occur because of differences in the adequacy of a measure across settings and populations.

•We cannot simply assume that a measure that was validated in one study is also valid in another setting or with a different population.

37

•The criterion that researchers select can be measured either at the same time as the variable to be validated or after that time.

•Concurrent validityConcurrent validity exists when a measure yields scores that are closely related to scores on a criterion measured at the same time.

•Predictive validityPredictive validity is the ability of a measure to predict scores on a criterion measured in the future.

38

Construct validityConstruct validity

•Measurement validity can also be established by showing that a measure is related to a variety of other measures as specified in a theory.

•This validation approached, known as construct validityconstruct validity, is commonly used in social research when no clear criterion exists for validation purposes.

39

ReliabilityReliability

•ReliabilityReliability means that a measurement procedure yields consistent scores when the phenomenon being measured is not changing (or that the measured scores change in direct correspondence to actual changes in the phenomenon).

•If a measure is reliable, it is affected less by random error, or chance variation, than if it is unreliable.

•Reliability is a prerequisite for measurement validity: We cannot really measure a phenomenon if the measure we are using gives inconsistent results.

40

Four possible indications of unreliabilityFour possible indications of unreliability

1. test-retest reliability 2. interim reliability 3. alternate-forms reliability4. Inter-observer reliability

41

Test-retest reliabilityTest-retest reliability

•When researchers measure a phenomenon that does not change between two point separated by an interval of time,the degree to which the two measurements are related to each other is the test-retest reliabilitytest-retest reliability of the measure.

42

Interitem reliabilityInteritem reliability

•When researchers use multiple items to measure a single concept, they must be concerned with interim reliabilityinterim reliability(or interim consistency).

•The stronger the association is among the individual items and the more items there are that are included, the higher the reliability of the index will be.

43

Alternate-forms reliabilityAlternate-forms reliability

•Researchers are testing alternate-forms reliabilityalternate-forms reliabilitywhen they compare subjects’ answers to slightly different versions of survey questions (Litwin 1995:13–21).

•A related test of reliability is the split-half reliabilitysplit-half reliability approach. Reliability is achieved when responses to the same questions by two randomly selected halves of a sample are about the same.

44

Interobserver reliabilityInterobserver reliability

•When researchers use more than one observer to rate the same people, events, or places, interobserver reliabilityinterobserver reliability is their goal.

•Assessing interobserver reliability is most important when the rating task is complex.

45

Ways to improve reliability and validityWays to improve reliability and validity

•Whatever the concept measured or the validation method used, no measure is without some error, nor can we expectit to be valid for all times and places.

•Remember that a reliable measure is not necessarily a valid measure.

•Unfortunately, many measures are judged to be worthwhile on the basis of only a reliability test.

46

•Remember always that measurement validity is a necessary foundation for social research.

• Gathering data without careful conceptualization or conscientious efforts to operationalize key concepts often is a wasted effort.

•The difficulties of achieving valid measurement vary with the concept being operationalized and the circumstances of the particular study.

47

•Planning ahead is the key to achieving valid measurementin your own research; careful evaluation is the key to sound decisions about the validity of measures in others’ research.

•Statistical tests can help to determine whether a givenmeasure is valid after data have been collected, but if it appears after the fact that a measure is invalid, little canbe done to correct the situation.

•If you cannot tell how key concepts were operationalized when you read a research report, don’t trust the findings.

•And if a researcher does not indicate the results of tests used to establish the reliability and validity of key measures, remain skeptical.