Other Basic Considerations in Designing Measures “Not everything that counts can be measured, and not everything that can be measured counts”, Albert Einstein.

Other Basic Considerations in Designing MeasuresOther Basic Considerations in Designing Measures

““NNot everything that counts can be ot everything that counts can be measured, and not everything that can be measured, and not everything that can be measured countsmeasured counts””,, Albert Einstein. Albert Einstein. Levels of MeasurementLevels of Measurement

Continuous vs. discrete variables Continuous vs. discrete variables (implications for analysis of findings)(implications for analysis of findings)

Scales and IndicesScales and Indices

Levels of MeasurementLevels of Measurement

1-Nominal (ex. Mother tongue)1-Nominal (ex. Mother tongue) different categories (names, labels, images) different categories (names, labels, images) not rankednot ranked

2-Ordinal (county fair prizewinners ranked by first, second & third prize)2-Ordinal (county fair prizewinners ranked by first, second & third prize) different categoriesdifferent categories rank-orderedrank-ordered attributes indicate relatively more or less of that variableattributes indicate relatively more or less of that variable distance between the attributes of a variable is imprecisedistance between the attributes of a variable is imprecise

Levels of Measurement (cont’d)Levels of Measurement (cont’d) 3- Interval Measures (ex. age measured by 5 year age groups)3- Interval Measures (ex. age measured by 5 year age groups)

different categoriesdifferent categories ranked in orderranked in order Can tell amount of difference between categoriesCan tell amount of difference between categories Usually no true zeroUsually no true zero

4- Ratio Measures (age measured by date of birth)4- Ratio Measures (age measured by date of birth) different categoriesdifferent categories ranked in order ranked in order amount of difference between categories amount of difference between categories also possible to state proportion (have a true zero)also possible to state proportion (have a true zero)

Relations between levels --Relations between levels --can collapse from higher can collapse from higher into lower, not vice versainto lower, not vice versa

Scales & IndicesScales & Indices

Often used interchangeablyOften used interchangeably Scale Scale

Intensity or degree of variable or constructIntensity or degree of variable or construct• often nominal or ordinaloften nominal or ordinal

Different types of common scaling techniquesDifferent types of common scaling techniques

IndexIndex Combines multiple indicators into single scoreCombines multiple indicators into single score Often interval or ratio levelOften interval or ratio level

Measurement principles for scales and indicesMeasurement principles for scales and indices Mutually exclusive and exhaustive Mutually exclusive and exhaustive

attributesattributes Unidimensionality (measure a single Unidimensionality (measure a single

construct)construct)

What are composite measures?What are composite measures?

Composite measures are instruments that use Composite measures are instruments that use several questions to measure a given variable several questions to measure a given variable (construct).(construct).

A composite measure can be either A composite measure can be either unidimensionalunidimensional or or multidimensional.multidimensional.

Indexes and scales are two types of composite Indexes and scales are two types of composite measures.measures.

Logic of Index Constructionactions combined in single measure, usually ordinal level of measurement

Logic of Index Constructionactions combined in single measure, usually ordinal level of measurement

Logic of Scalesactions rankedLogic of Scalesactions ranked

Technical Issues in Index ConstructionTechnical Issues in Index Construction

weightingweighting e.g. Quality of life indexe.g. Quality of life index

treatment of missing datatreatment of missing data standardizationstandardization

Standardization exampleactually about weightingStandardization exampleactually about weighting

Treatment of Missing DataTreatment of Missing Data

eliminate cases with missing eliminate cases with missing data?data?

substitute average score ?substitute average score ? Guess ?Guess ? insert random value ?insert random value ?

Rates and StandardizationRates and Standardization

makes comparison possible makes comparison possible e.g. rate of mobile phone owners per e.g. rate of mobile phone owners per

class class # of owners# of owners/size of class/size of class

Issues: deciding what measure to use Issues: deciding what measure to use for reference populationsfor reference populations e.g. employment ratese.g. employment rates

Scales: ex. Likert ScaleScales: ex. Likert Scale

ordinal level measure ordinal level measure Debates about inclusion of neutral category (!?)Debates about inclusion of neutral category (!?) switching directions of some questions (to avoid switching directions of some questions (to avoid

response setsresponse sets)) use to create index (scoring issues)use to create index (scoring issues)

Ex. E-mail has positively transformed Ex. E-mail has positively transformed communication between people (p. 179)communication between people (p. 179)

examplesexamples

Thurston ScaleThurston Scale

uses judges to pick statements uses judges to pick statements tests agreement/disagreementtests agreement/disagreement comparative measurement, not often used comparative measurement, not often used

todaytoday

Semantic Differential ScaleSemantic Differential Scale

mark spot on mark spot on continuum continuum between paired between paired oppositesopposites

difficult to analyze difficult to analyze resultsresults

Fig. 1. Modern Japanese version of the Semantic Differential. The Kanji characters in background stand for "God" and "Wind" respectively, with the compound reading "Kamikaze".

Guttman ScalingGuttman Scaling

hierarchical relationship hierarchical relationship established by response to established by response to indicatorsindicators

e.g. Attitudes to abortion, e.g. Attitudes to abortion, attitudes to alcohol, cigarettes attitudes to alcohol, cigarettes and drugsand drugs

Link to on-line descriptionLink to on-line description http://www.socialresearchmethhttp://www.socialresearchmeth

ods.net/kb/scalgutt.htmods.net/kb/scalgutt.htm See also Buddenbaum & See also Buddenbaum &

Novak Novak Applied Applied Communication ResearchCommunication Research

http://books.http://books.googlegoogle.com/books?id=s93sL7l81XMC&pg=PA98&.com/books?id=s93sL7l81XMC&pg=PA98&lpg=PA98lpg=PA98&&dq=guttmandq=guttman+scale+calculate+score&source=web&+scale+calculate+score&source=web&ots=dJfvTHPJDZots=dJfvTHPJDZ&&sig=XIydBYJOmsxptjCmVbhV__Elsb8sig=XIydBYJOmsxptjCmVbhV__Elsb8&hl=en&&hl=en&sa=Xsa=X&&oi=book_resultoi=book_result&&resnum=1resnum=1&ct=result#PPA100,M1&ct=result#PPA100,M1

Develop the Cumulative Scale. The key to Guttman scaling is in the analysis. We construct a matrix or table that shows the responses of all the respondents on all of the items. We then sort this matrix so that respondents who agree with more statements are listed at the top and those agreeing with fewer

Perfect Guttman ScalePerfect Guttman Scale

Bogardus Social Distance Scale (type of Guttman scale)

Bogardus Social Distance Scale (type of Guttman scale)

ordered statements expressing closeness to ordered statements expressing closeness to respondentrespondent

must be refer to specific contextmust be refer to specific context problems comparing behaviour and feelingsproblems comparing behaviour and feelings

exampleexample

Indexes (Indices)Indexes (Indices)

combine responses into single scorecombine responses into single score e.g. quality of lifee.g. quality of life

Weighting Index ItemsWeighting Index Items importance given to components of indeximportance given to components of index unweighted indexunweighted index

The Macleans IndexThe Macleans Index

STUDENT BODYSTUDENT BODY 5 items 5 items CLASSESCLASSES 3 items 3 items FACULTYFACULTY 4 items 4 items FINANCESFINANCES 3 items 3 items LIBRARYLIBRARY 3 items 3 items REPUTATIONREPUTATION 2 items 2 items

Based on twenty discrete rankings of six attributes of universities:

Operational DefinitionStudent Body (21%)Operational DefinitionStudent Body (21%)

average grades of incoming students 12

proportion of incoming students with 75% or more 3

proportion of incoming students from out of province 1

graduation rates (% full-time undergrads in second year who graduate)

national academic awards won by students

2

3

Operational DefinitionClasses (18%)Operational DefinitionClasses (18%)

7.5

class sizes at 3rd and 4th year levels 7.5

class sizes at 1st and 2nd year levels (1-25, 26-50, 51-100, 101-250, 251-500, 501 plus)

% 1st year courses taught by tenured or tenure-track professors

3

Operational DefinitionFaculty (17%)Operational DefinitionFaculty (17%)

3

success of eligible faculty in winning federal grants(arts and social sciences) 5.5

% with Ph.D’s

success of eligible faculty in winning federal grants(medicine and sciences) 5.5

national academic awards won by profs 3

Weighting of Macleans IndexWeighting of Macleans Index

STUDENT BODYSTUDENT BODY 21%21%

CLASSESCLASSES 18%18% FACULTYFACULTY 17%17% FINANCESFINANCES 12%12% LIBRARYLIBRARY 12%12% REPUTATIONREPUTATION 20%20%

Critique of Maclean’s Survey as Measurement toolCritique of Maclean’s Survey as Measurement tool What are the operational measures?What are the operational measures? Which measures (or parts of the “composite Which measures (or parts of the “composite

measure”) are more important?measure”) are more important? What is the concept being measured?What is the concept being measured?

Advantage of Scales & IndicesAdvantage of Scales & Indices

Easy to use & administerEasy to use & administer Inexpensive Inexpensive Time-efficient (fast for respondent & Time-efficient (fast for respondent &

researcher)researcher) Easy to calculate scoresEasy to calculate scores

Disadvantages of ScalesDisadvantages of Scales

Assumption that we know what is importantAssumption that we know what is important Tend to be very broadTend to be very broad Predetermined questions & problems may not be Predetermined questions & problems may not be

suited or cover full meaningssuited or cover full meanings Response set effect (answering the same way for Response set effect (answering the same way for

all questions)all questions) But there are techniques for counteracting response set But there are techniques for counteracting response set

biasbias

Specialized techniques for Index constructionSpecialized techniques for Index construction

factor analysisfactor analysis q-sortq-sort cluster analysiscluster analysis