Quantitative Approaches to Dissertation Research Dr Meredith Rolfe February 20 & 2, 2013.

Quantitative Approaches to Dissertation Research

Dr Meredith Rolfe

February 20 & 2, 2013

Agenda

Scientific framework for answering questions Measurement Case Selection Analysis: Quantitative Methods & Experiments

Data Sources

Examples

MG420 Feb 2013 2

Why did the pot of water boil?

• The water reached 100°C• Phase transition and percolation• Someone turned the hob on• Someone was hungry• Time for breakfast• Porridge is a traditional

breakfast food in this country• Owned a pot• Had water supply• The gas bill had been paid• The hob was not broken• Elevation• Pot had a lid• Copper vs. Stainless steel

MG420 Feb 2013 3

Theoretical System

MG420 Feb 2013 4

So, there are 3 (4) types of questions you might ask…

1. Measurement projects:• How do we measure X or Y (e.g., corporate

reputation or innovation)?• How do we identify successful X or Y (e.g.,

policies or practices)?

2. External validity projects:• Does a previous finding generalize to a new

setting? Does an experiment work with different subjects?

• Meta-analysis projects

5MG420 Feb 2013

A. Causal Questions:• Does a change in X produce a change in Y?

• Does cloud computing improve govt. efficiency?• Does mentoring influence managerial career

advancement?

B. Case Analysis questions:• Ex-post: Is X having expected effect on Y?• Ex-ante: Will X have an effect on Y?

6

Third type of question (cont.)

MG420 Feb 2013

Measurement

7MG420 Feb 2013

Science = Measurement?

In physical science the first essential step in the direction of learning any subject is to find

principles of numerical reckoning and practicable methods for measuring some quality connected with it. I often say that when you can measure what you are speaking about, and express it in

numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the

beginning of knowledge, but you have scarcely in your thoughts advanced to the state of Science,

whatever the matter may be.Baron William Thomson Kelvin, Popular Lectures and Addresses 1:73 (originally: Lecture to the Institution of Civil Engineers, 3 May 1883)

MG420 Feb 2013 8

Measurement: Another Perspective

9MG420 Feb 2013

Steps in a Measurement Process

1. Identify theoretical concepts

2. Choose one or more indicators/variablesa. Brainstorm possible variables

b. Choose operational rule or definition

c. Level of measurement

d. Measurement technique

3. Assess Measurement Reliability & Validity

MG420 Feb 2013 10

Step 1) Identify theoretical constructs or concepts

• States• Regime type• Democracy?• Economic power• State capacity• Free press

• Individuals• Income• Skills• Work experience• Education or training• Authoritarianism• Political interest• Job success• Compliance

• Corporations• Performance• Financial success• Size• Positional advantage• Leadership• HR policy• Corporate culture• Hierarchy

• Industry• Level of uncertainty• Profitability

MG420 Feb 2013 11

Step 2) Variables: operational rules to turn observations into data

• State: Democracy• fair elections• GNP/capita

• State: Capacity• size of military• Total expenditures

• Individual: Job success• Income• # Promotions• Job title• Compliance

• Behavioural• Attitudes

• Corporation: Performance• Growth• Profits• Ranking• ROI

• Corporation: Positional Adv.• Network Centrality• Categorical

• Corporate Culture/Hierarchy• Work practices • Work policies• Emp/Mgmt attitudes• Network Analysis

MG420 Feb 2013 12

Variables:Levels of measurement

• Nominal • Algeria, Russia (country)• gender or ethnicity of respondent• Revolution, civil war (0/1)

• Ordinal• democracy in 7 categories• war• Corporate rank• Education (degree)

• Interval• agree/disagree 5 point scale

• Ratio (Continuous)• income• length of war• corporate profits

MG420 Feb 2013 13

Discrete “Variables”

SuccessfulUnsuccessful

Net ProfitsMG420 Feb 2013 14

Measurement techniques

• Ordering and counting observed phenomena (words, elections, dollars, number of ads placed, and so on)• Sources: government statistics, corporate statistics,

corporate reports, newspapers, archives• Content analysis (text, verbal)• Categorical measures: Is there a work from home policy?

Are there more than two major parties?• Observed behaviours: How many safety violations are

reported by employees? How many patents are registered by companies? How many polluters are identified and prosecuted by the govt? How many emails are sent by employees? How many times does a person smile (gesture) during a conversation?

MG420 Feb 2013 15

Examples of existing data sources

Opportunity: Old data, new questions

Data Library LSE http://www2.lse.ac.uk/library/eresources/data/Home.aspx

Microdata - Government surveys, longitudinal data and opinion polls

Aggregated Data - OECD, EUROSTAT, IMF, World Bank. Includes Census data

Financial Databases - Information on markets, companies, exchange rates

Geographic Information Resources, GIS - Covering EU admin boundaries and Ordnance Survey data

International Data Centres and Statistical Institutes - Government statistics

MG420 Feb 2013 16

Measurement Techniques 2

• Asking people (perceptions or self-reported behaviours)• Survey questions

• Fixed format• Open-response (requires coding)

• Interviews (elite or random sample)• Sorting tasks (measurement projects)• Implicit attitudes (psychology, often experimental)

• Experimental Manipulation (Independent variable)• Media message (newspaper or video clip)• Assignment to a role• Priming task (think about X)• Information provided or framing

17MG420 Feb 2013

Small survey: Example

• Finding the first job in China (previous dissertation)

• Started with theories about how people find jobs in general, and about the advantages/ disadvantages of using various channels: ads, firms websites, informal channels (strong vs. weak ties)

• Formulated hypotheses

• Applied a questionnaire to 80 young Chinese

• Analysis (in this case it was only frequency tables)

• Discuss to what extent the analysis confirms/ disconfirms the hypotheses and why

MG420 Feb 2013 18

Political Discussion Name Generator

MG420 Feb 2013 19

Self-Monitoring

MG420 Feb 2013 20

Corporate Ranking: Admiration

MG420 Feb 2013 21

Corporate Ranking: Trust

MG420 Feb 2013 22

Measurement techniques 3

• Statistical: Some measurement techniques extract underlying constructs from existing quantitative data• Principle Components• Factor Analysis• Network Analysis

23MG420 Feb 2013

Source: Krebs, orgnet.com

Partisanship & book purchases

MG420 Feb 2013 24

Source: Ronald Inglehart and Christian Welzel, M

odernization, Cultural Change and D

emocracy N

ew York: Cam

bridg University

Press, 2005: page 63.

MG420 Feb 2013 25

Measuring social

meaning &

mental m

aps

MG420 Feb 2013 26

Step 3) Assess Measurement Reliability & Validity

Reliable measures are…

• consistent (repeated measurement gives similar results for the same case)• inter-coder reliability (qualitative data)

• unbiased (right answer, on average)

• precise/low error (close to the right answer)

MG420 Feb 2013 27

Reliable Measures

Consistent

MG420 Feb 2013 28

Step 3) Assess Measurement Reliability & Validity

Reliable measures are…

• consistent (repeated measurement gives similar results for the same case)• inter-coder reliability (qualitative data)

• unbiased (right answer, on average)

• precise/low error (close to the right answer)

MG420 Feb 2013 29

Reliable Measures

ConsistentConsistentUnbiased

MG420 Feb 2013 30

Step 3) Assess Measurement Reliability & Validity

Reliable measures are…

• consistent (repeated measurement gives similar results for the same case)• inter-coder reliability (qualitative data)

• unbiased (right answer, on average)

• precise/low error (close to the right answer)

MG420 Feb 2013 31

Reliable Measures

ConsistentConsistentUnbiased

ConsistentUnbiased

Low VarianceMG420 Feb 2013 32

Valid measures are…

• Content or substance of measure captures construct• face validity – ask an expert

• “what are you thinking about” probes

• manipulation checks (experiments)

• Criterion-related validity• Predictive: Correct relationship with some

predicted outcome variable

• Decision validity: Useful or pragmatic applications

MG420 Feb 2013 33

Valid Measures are… (cont.)

• Construct validity• Convergent validity: Related to other

theoretically related variables/indicators (correlations, alpha scores, factor analysis)

• Example: GNP/capita, tax revenue/capita

• Discriminate validity: empirically discriminates between related constructs (correlations, factor analysis)

• Example: poverty, education

MG420 Feb 2013 34

Measurement: What can go wrong?

Example 1: Do female managers have more empathy than male managers?

== What is empathy? How will it be measured? Behavioural empathy or self-reported expressions of concern? Could these two measures be tapping into different constructs? (Previous studies: YES!)

Example 2: Exposure to ethical discourse will change discriminatory practices in HR.

== Change is what? decrease, increase, both (contingencies)

== What is ethical discourse? What is exposure to ethical discourse (reading an article, participating in a conversation, seeing a poster, overhearing colleagues)

MG420 Feb 2013 35

Questions?

• What constructs might you want to measure?

• How would you would measure them?

36MG420 Feb 2013

Case Selection

37MG420 Feb 2013

Theoretical System

MG420 Feb 2013 38

Universe of Cases orPopulation of Interest

The universe of cases or population of interest includes all of the entities we wish to make generalizations about

• All large (>10k employees) companies• All tech start ups• Employees (in general)• IT professionals• “the public”• Public reform episodes (country-year-policy)

39MG420 Feb 2013

Learning about the population of interest

• We want to learn about the population of interest or universe of cases

• BUT we can’t study all cases in the population (e.g., employees, the public)

• BUT we don’t believe that what we observe represents population of interest (all large companies that have existed or might exist)

• THEREFORE we choose a sample of cases from the larger population or universe of interest

40MG420 Feb 2013

Case Selection Process

1. Define a population of interest

2. Specify a sampling frame

3. Choose a sampling method

4. Choose the sample size* (expert advice)

5. Choose the actual sample of cases

6. Collect data!!

41MG420 Feb 2013

Specify a sampling frame

A sampling frame is a list or enumeration of the population or universe of interest•Phone book•Voter register•Employee records•UN nation list•NYSE or FTSE listing•Professional Assoc. member or licensee

42MG420 Feb 2013

Case or Sample Selection Methods

• Entire universe (e.g., all EU countries, all companies listed on NYSE larger than XX)• Pros: ability to generalize, network analysis• Cons: Time consuming, small-n problem

• Random selection from “universe”• Pros: Best ability to generalize to universe• Cons: Time-consuming, requires data on 20

or more cases (100s-1000s)

43MG420 Feb 2013

Case selection methods

• Convenience samples (LSE students, employees at nearby company)• Pros: Low-cost, easy to gather additional data• Cons: Bias, limited generalizability

• Snowball sample (select initial entities who provide links to additional entities)• Pros: Good when sample frame is not available,

network studies• Cons: Bias, potentially limited generalizability,

hard to identify missing cases

44MG420 Feb 2013

Small-nCase selection methods

These methods are used most often when you study only a few (2-10) cases•Most similar cases (Choose cases with largely similar i.v.s but different outcome)•Least similar cases (Choose cases with largely different i.v.s but same outcome)•Anomalous case* (single case)•Pros: Best for intensive case studies, rare events (e.g., revolutions)•Cons: Sensitive to omitted i.v.s

45MG420 Feb 2013

Sampling: What can go wrong?

• Observations are not independent• Example: using snowball method to test how my

friends and their friends are finding jobs• Selecting on the dependent variable

• Example: Choosing to study three successful companies in order to learn more about how companies become successful (need a few failures too)

MG420 Feb 2013 46

Analysis: Quantitative Methods & Experiments

47MG420 Feb 2013

Theoretical System

MG420 Feb 2013 48

Formulating and testing hypotheses

• Good hypotheses are:• Testable (e.g. identify the dependent and independent variable, and the

relationship between them)• Have a clear null (you test HA against H0)

Example: I believe that employees with artistic training are more innovative.

• Testability: define artistic training, think about the Y (dummy variable; years of training; thresholds); define innovation (e.g. finding a new solution to an old problem)

• Null? Innovativeness is randomly distributed in the population, i.e. the probability to find an innovative employee, p, is not conditionally dependent on an employee’s artistic training

HA: Employees who have more than xxx-years of training in art are more likely to find an innovative solution.

MG420 Feb 2013 49

Lab Experiments: Example

Hypothesis: Competitive environments decrease idea sharing

Construct an experimental design that represents all the elements and conditions specified in the hypotheses

- Select sample of subjects (see previous)- Identify and control non experimental factors (see what literature

says about idea sharing by gender, culture, age etc)- Experimental factors – 2-3 conditions, high/ medium/ low

competition- Validate instruments to measure outcomes (how do you measure

idea sharing)- Design a manipulation check (e.g., ask the subjects how competitive

they believe the situation is)- Pilot study - Run actual experimental trials

Data Analysis: ANOVA

MG420 Feb 2013 50

Survey Experiment on Immigration & Citizenship

Asylum seekers should have the same access to jobs and benefits as everyone else.

Strongly agreeAgreeNeither agree nor disagreeDisagreeStrongly Disagree

Asylum seekers who have been granted citizenship should have the same access to jobs and benefits as everyone else.Strongly agreeAgreeNeither agree nor disagreeDisagreeStrongly Disagree

4%11%

17%

39%

30%

12%30%20%20%18%

MG420 Feb 2013 51

Field EXPERIMENTS

Note: Not all controls factors can be dealt with, but it is expected that large sample size will alleviate some problems

Example: “Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination”. M. BERTRAND & S. MULLAINATHAN, American Economic Review, 2004

Hypothesis: There is race discrimination in the US labor market; Whites are favored over African-Americans

Design: - sending fictitious CVs to job ads in Boston and Chicago newspapers- Identical CVs, the only difference is the name (White vs. African-

American)- about 5,000 CVs sentOutcome: White names receive 50 percent more call-backs for

interviews.

MG420 Feb 2013 52

Experiments: What can go wrong?

Example 1: Compare the impact of two treatments (information about product performance and appeals to consumer identity) on consumer intention to buy?

== Ensure that there are no other relevant differences between the two treatments (e.g., use of emotional language) that might affect intention to buy

Example 2: Compare the impact of assignment to organizational role (Manager vs. Employee) on endorsement of ethical principles.

== Include manipulation checks to ensure that the experimental manipulation is working as expected (e.g., through changes in sense of personal power.)

MG420 Feb 2013 53

Observational DataAnalysis methods

• Descriptive statistics (mean, median, sd, tables)• Basic conditional dependence (correlation, Chi-

squared)• Regression (OLS, Probit, Logit, etc)• Time series or event history• Network analysis • Meta-analysis

MG420 Feb 2013 54

Analysis of existing observational data

• Cedric Herring: Does Diversity Pay?: Race, Gender, and the Business Case for Diversity, American Sociological Review, 2009

• Motivation: a wide literature on group diversity, but few studies linking workforce diversity and profitability

• Uses National Organizations Survey, a national sample of for-profit business organizations

• Literature review on the advantages and disadvantages of diverse work force – diversity: (1) gender diversity (2) racial diversity

• 8 hypothesis e.g. Hypothesis: As racial (gender) workforce diversity increases, a business

organization’s sales revenues will increase.

Test: OLS estimation

Sales ($) = f(diversity, controls)

MG420 Feb 2013 55

Event history • When you want to determine how likely is an event, what are the

causes of an event• Example:

• Who is more likely to display unethical behavior, big reputable brands or less known brands (Event = instance of unethical behavior)

• what is the likelihood of unemployment for high skilled professionals (Event = became unemployed)

• Case selection:• Large observation window (min 5+ years --- 10-20)• Relatively large samples (100+)• A combination of firms which had and did not have the event yet

• Censoring issues (you do not observe the entire history of each case)

• Warning! Do not employ event history unless you know that you data satisfies the criteria specified above!

MG420 Feb 2013 56

Meta-Analysis

• When you study a question which has been widely studied, but on which there is little agreement

• “quant literature review”

• Example: Branches (of multinational firms) led by foreign managers perform better/ worse that branches led by nationals.

• What do you do?

• A pure lit review• A meta-analysis

MG420 Feb 2013 57

Meta-Analysis: Steps

- Collect (almost) all relevant quant studies on the topic- If there are too many (hundreds and hundreds) you may want to

sample – e.g. most cited ones, restricted to an area (Asia-Europe)- Snowball samples also work

- Organize all the data in tables - What is the likely direction of the correlation, if any (e.g. is there

any statistically significant evidence, across studies, that manager’s nationality and performance are correlated)

• What is the magnitude of the effect

MG420 Feb 2013 58

What can go wrong?

Despite perfect execution, your hypotheses are not confirmed

What do you do?• Check again that everything is all right with the sample & that the analysis is

correct and complete.

• Do not be afraid to write a paper with “negative results”! Hardly publishable, but of great value to a business!

- revisit the theory to understand nuances- look at the data used in previous studies (what’s the difference between your data and theirs? Old data? Different culture? Technology? etc)- in depth interviews with some of your subjects (see qualitative research class)

• Empirical irregularities allow you to make a contribution to the theory!!!

MG420 Feb 2013 59

Additional ReadingsKilduff, M. & W. Tsai (2003) Social Networks and Organizations. Sage, London

Lewis-Beck, M. S. (1995) Data Analysis: An introduction, Sage (Series: Quantitative Applications in the Social Sciences 103)

Wolf, F. M. (1986) Meta-Analysis, Sage (Series: Quantitative Applications in the Social Sciences 59)

Yamaguchi K. (1991) Event history analysis, Sage (Series: Applied social research Methods 28)

MG420 Feb 2013