1 Universitat de Girona Measurement Quality in Social Networks LLuís Coromina Quantitative Methods, University of Girona Department of Economics, Campus Montilivi, 17071 Girona, Spain E-mail: [email protected]
1
Universitat de Girona
Measurement Quality in Social Networks
LLuís Coromina
Quantitative Methods, University of Girona Department of Economics,
Campus Montilivi, 17071 Girona, Spain E-mail: [email protected]
2Universitat de Girona
Outline
• Studies on Social Network Data Quality
• Survey Design o Data collection o Questionnaire Design
• Types of hypothesis tested in other Quality studies
• Quality Measurement (Validity and Reliability)
• Measurement procedure o Multitrait-Multimethod Model (MTMM) o Multilevel MTMM o Interpretation
3Universitat de Girona
Outline
• INSOC PROJECT: o Social Network Data collection Design o Multitrait-Multimethod Model results o Meta-analysis Design o Meta-analysis Results
• Results and comparison from others studies.
Intro
4
Objective
Universitat de Girona
4
Main purpose of scientific research is the interpretation and/or prediction
of phenomena
Quality measurement instruments (empirical data) is of crucial importance
It is important to know how well we can measure with certain
measurement instruments and how much a researcher can rely on findings
based on data collected.
Intro
5
Objective
Universitat de Girona
5
Egocentered network data are especially important.
Network characteristics (network size, structure and composition)
Characteristics of network members (feeling of closeness or importance;
frequency of advice, collaboration, social support) collected from the
respondent (ego).
Intro
6Universitat de Girona
Quality in Social Network Survey Data
6
Studies on Social Network Data Quality
Killworth & Bernard, 1976, 1980; Bernard et al., 1979,1980, 1982;
Hammer, 1984; Sudman, 1985, 1988; Freeman and Romney, 1987;
Freeman, Romney & Freeman, 1987; Corman and Bradford, 1993;
Hlebec, 1993; Brewer and Webster, 1997,…
Intro
7Universitat de Girona
Quality in Social Network Survey Data
7
People are generally very inaccurate in reporting on their past
interactions with other people.
People remember long-term or typical patterns of interaction with other
people rather well.
Accuracy of reporting about interactions is also influenced by the
frequency of interaction (more frequent contact with group members,
more accurate reports) and by the reliability of an individual
respondent on about actual interactions (Romney and Faust, 1982; Romney
and Weller, 1984).
Intro
8Universitat de Girona
Quality in Social Network Survey Data
8
Quality of measurement, specially reliability, can be affected by the type of
social support (Ferligoj & Hlebec, 1999).
+ Emotional and informational support dimensions
Reliability
- Social companionship and material support dimensions
Reasons for ‘higher quality’
More “intimate” types of support � more important to the respondents.
� Relatively small number of very
important people.
Less effortful for the respondent. Network
members are important and close
Intro
9Universitat de Girona
Data Collection Methods
9
DATA COLLECTION METHODS
In Social Sciences, the most frequent measurement instrument is a SURVEY
Intro
10Universitat de Girona
Data Collection Methods
10
Intro
11Universitat de Girona
Data Collection Methods
11
With interviewer Self-administration
Paper •••• Personal (Face-to-face)
•••• telephone
•••• “paper self-
administered”
Electronic •••• computer assisted personal
interview (CAPI)
•••• computer assisted telephone
interview (CATI)
•••• Web surveys
•••• Computer Assisted Self-
Interview (CASI)
•••• ACASI
•••• Mobile phone surveys
NO ONE COLLECTION METHOD IS BEST FOR ALL CIRCUMSTANCES.
THE CHOICE MUST BE MADE WITHIN THE CONTEXT OF THE PARTICULAR
OBJECTIVES OF THE SURVEY AND THE RESOURCES AVAILABLE.
Intro
12Universitat de Girona
Data Collection Methods
12
Characteristics of the Population
+
Characteristics of The Sample
+
Types of Questions
+
Question Topic
+
Response Rate
+
€€ Cost $$
+
Time
Intro
13Universitat de Girona
Data Collection Methods
13
PERSONAL INTERVIEWING
ADVANTAGES: DISADVANTAGES:
Generally: highest cooperation and lowest
refusal rates
Most costly mode of administration
Allows for longer, more complex interviews Longer data collection period
High response quality Interviewer concerns
Takes advantage of interviewer presence
Multi-method data collection
TELEPHONE INTERVIEWING
ADVANTAGES: DISADVANTAGES:
Less expensive than personal interviews Samples of general population Non-response
Shorter data collection period than personal Questionnaire constraints
Interviewer administration (vs. mail) Difficult to administer questionnaires on
sensitive or complex topics
Better control and supervision of interviewers
(vs. personal)
Biased against households without
telephones, unlisted numbers
Better response rate than mail for list samples
Intro
14Universitat de Girona
Data Collection Methods
14
MAIL SURVEYS
ADVANTAGES: DISADVANTAGES:
Generally lowest cost Most difficult to obtain cooperation
Can be administered by smaller team of
people (no field staff)
No interviewer involved in collection of data
Access to otherwise difficult to locate, busy
populations
More likely to need an incentive for
respondents
Respondents can look up information or
consult with others
Need good sample
Slower data collection period than telephone
Intro
15Universitat de Girona
Data Collection Methods
15
WEB SURVEYS
ADVANTAGES: DISADVANTAGES
Lower cost (no paper, postage, mailing,
data entry costs)
% of homes own a computer;
% have home e-mail.
Can reach international populations Representative samples difficult - cannot
generate random samples of general
population
Time required for implementation reduced Differences in capabilities of people's
computers and software for accessing
Web surveys
Complex skip patterns can be programmed Different ISPs/line speeds limits extent of
graphics that can be used
Sample size can be greater
Intro
16Universitat de Girona
Data Collection Methods
16
MEASUREMENT QUALITY IMPLICATIONS OF MODE SELECTION
Social Desirability
� Under-represent (illegalities) or Over-represent (voting) different population groups.
� Sensitive information from respondents
� Self-administered methods produce fewer social desirability effects.
Response effects (wording, answer categories order, or questions order)
Response order (Primacy-visually- and recency –telephone)
Data completeness (with interviewers more questions answered)
Completion times
Intro
17Universitat de Girona
Data Collection Methods
17
Mean completion times(in seconds)
by mode (web vs Telephone) and age group
Intro
18Universitat de Girona
Data Collection Methods
18
Differences in data quality for different methods????
Data collection method for ego-centered network data.
Most used face-to-face data collection mode.
Face to face vs. Telephone (Kogovšek et al., 2002)
Validity and Reliability of web surveys
(Couper, 2000; Dillman, 2000; Couper et al., 2001; Vehovar et al., 2002)
Quality of Egocentered Network Data Collected via Web
Marin (2002) ; Lozar Manfreda et al. (2004).
Intro
19Universitat de Girona
19
Questionnaire Design
QUESTIONNAIRE DESIGN
Problems in the response process Survey ERRORS
Krosnick (1999); Groves (2004),
Tourangeau, Rips & Rasinski (2000), Dillman, (2007)
Intro
20Universitat de Girona
20
Questionnaire Design
1) Failure to encode the information required
Problem of asking people about what topics that they do not pay much
attention OR are not important.
Improvement: Pre-test; Filter questions (or multiple name generator)
2) Misinterpretation of the questions
Hospital study: During the past 12 months, since _______, how many times have you
seen or talked to doctor [NAME i ] about your health? Do not count any time you might
have seen a doctor while you were a patient in a hospital, but count all other times your
actually saw or talked to a medical doctor of any kind.
Intro
21Universitat de Girona
21
Questionnaire Design
3) Forgetting and other memory problems
Example:
Low options High responses
Responses % Responses %
<1/2 hr 7.4 < 2 ½ hr 62.5%
½ to 1hr 17.7 2 ½ to 3 hr 23.4%
1 to 1 ½ hr 26.5 3 to 3 ½ hr 7.8%
1 ½ hr to 2 hr 14.7 3 ½ hr to 4 hr 4.7%
2 to 2 ½ hr 17.7 4 to 4 ½ hr 1.6%
>2 ½ hr 16.2 >4 ½ hr 0%
Question about television watching can go from 16% to 37% saying that they watched
more than 2.5 hours/day, depending on the set categories.
Intro
22Universitat de Girona
22
Questionnaire Design
4) Sensitive (social desirability) questions (recommendations)
• More time to respondents to answer in order to not misreport information (Krosnick 1999)
• Question example for strategy for over-reported behaviors:
In talking about elections, we often find that a lot of people were not able to vote because they are not registered, they were sick, or they just didn’t have the time. How about you – did you vote in the elections this November?
• Use self-administration, computer-administered questionnaire or similar methods to improve reporting.
Intro
23Universitat de Girona
23
Questionnaire Design
5) Failure to follow instructions (mainly for self-administered respondents)
If first page is half-page explaining instructions, respondent use to skip that part.
Intro
24Universitat de Girona
24
Questionnaire Design
Instructions must be exactly where is needed and not in a separate section
Intro
25Universitat de Girona
25
Questionnaire Design
Format of response scale (Network Questionnaires)
• Binary judgments (least difficult for respondents) about whether
respondents have a specified relationship with each actor on the roster.
• Ordinal ratings of tie strength
• Rankings: More demanding for respondents.
Majority of respondents preferred binary over ranking or rating…
Intro
26Universitat de Girona
26
Questionnaire Design
…. BUT reliability for ratings > binary judgments (Ferligoj & Hlebec, 1999).
Recommendations (not only network questionnaires)
• Use closed questions for measuring attitudes
• Use 5 point response scale and labeled every scale point. With 9 or 11
point response scale the beginning and the ending.
• Use ranking (preferences order) only if the respondents can see all the
alternatives: otherwise use paired comparisons.
Intro
27Universitat de Girona
27
Questionnaire Design
Name Generator Design
Studies suggest using minimal modules of name generators:
Van Der Poel (1993):
3-item: Discussing a major life change, aid with household tasks, and monthly
visiting.
5-items, add: Borrowing household money and going out socially.
Bernard et al. (1990):
5 items: Social activities, hobbies, personal problems, advice about important
decisions, closeness
Burt (1997):
3-item: GSS important matters issue, socializing, and discussion of a job
change
Intro
28Universitat de Girona
28
Questionnaire Design
Clusters of persons named by relationships (families, workplaces,…) than by
similarity of individual features -gender, race or age- (Fiske ,1995)
Use Recognition rather than Recall (when possible) - Level of ‘forgetting’-
(Brewer, 2000)
Multiple name generators may limit forgetting because persons forgotten for one
generator are often name in response to others.
Single name generator maybe sufficient for ‘core networks’
Intro
29Universitat de Girona
Quality in Social Network Data
29
Measurement Comparisons
Questionnaire components
Network ties
Intro
30Universitat de Girona
Quality in Social Network Data
30
In summary,
Validity and Reliability of survey data, social networks, can be affected by
many characteristics of the measurement instrument ( Name Interpreters).
a) Response scale (binary; 5 ordinal ending; 5 ordinal all; line drawing scale; 11
ordinal)
b) Response category labels( all categories or end points).
c) Name Generator method (Recognition vs free recall).
c) Data collection method (face-to-face interview, telephone, web survey,…)
e) Question wording (after we obtain the list of alters with name generators,
we can ask name interpreter questions “by alters” or “by questions”).
f) Layout of the questionnaire: Plain vs Graphical
Intro
31Universitat de Girona
Quality in Social Network Data
31
Types of networks analyzed:
Social support (Weiss, 1974; Hirsch, 1980; Wills, 1985:, vaux, 1988)
1) instrumental support (aid, material support, or tangible support) is the
provision of financial aid, material support, and necessary services),
2) informational support (advice support, appraisal support, or cognitive guidance)
involves help in defining, understanding, and coping with problematic events)
3) emotional support (or close support),
4) social companionship (diffuse support and belongingness) relates to time spent
with others in leisure activities).
Intro
32Universitat de Girona
Quality in Social Network Data
32
I) Frequency of Scientific advice II) Frequency of Collaboration III)Asking for crucial Information IV) Social activities outside the work
Tie Characteristics: Demographic variables
a) Measures of tie strength Education Frequency of contact Gender Feeling of closeness Age
b) Feeling of importance
c) Frequency of the alter
upsetting the ego
Intro
33Universitat de Girona
Hypothesis and Research Questions from Studies
33
Hypothesis and Research Questions from Studies
Intro
34Universitat de Girona
Hypothesis and Research Questions from Studies
34
• To test whether the recognition data collection technique is more stable than that of free recall (Ferligoj & Hlebec,1999)
• The size of the social network and age of the respondent should not
affect the similarity of free-recall and recognition methods.
• Reliability and validity of the measurement of tie characteristics for emotional and informational support alters would be higher compared to that for social companionship and material support alters. Kogovsek & Ferligoj (2004)
• The stability of social support : Emotional support (close and important ties) more stable than Material support (provided by specialized sources).
Closeness is not required for providers of material support and informational support. Hlebec & Ferligoj (2002)
Intro
35Universitat de Girona
Hypothesis and Research Questions from Studies
35
Hypothesis Predictors Kogovsek & Ferligoj (2005)
Measurement instrument Higher quality of measurement with by alters than by questions data collection
technique Higher quality of measurement of cognitively demanding questions with the
personal interview/by alters Higher quality of measurement of sensitive questions with the telephone
interview/by alters Network size Higher quality of measurement in small networks Personal characteristics Higher quality of measurement among female respondents Higher quality of measurement among younger respondents Higher quality of measurement among more educated respondents Personality characteristics Higher quality of measurement among extraverts
Intro
36Universitat de Girona
Hypothesis and Research Questions from Studies
36
• We might expect that every tie or alter characteristic would be more reliably and more validly measured when the question is posed by alters than by questions ( Kogovšek , Ferligoj , Coenders , Saris, 2002)
We therefore expect that cognitively more demanding questions would be more prone to measurement errors in the telephone than in the face-to-face mode, therefore:
• It is expected that demanding questions would be more reliably and more validly measured in face-to-face interviews than by telephone
Intro
37
Quality of measurement
Universitat de Girona
37
How to Measure
Intro
38
Quality of measurement
Universitat de Girona
38
Reliability: The extent to which any questionnaire, test or measure
produces the same results on repeated experiments. Random error.
Validity: The extent to which any measure measures what is intended to
measure (Carmines & Zeller, 1979:12). Systematic error, bias.
Convergent validity refers to common trait variance and is inferred
from large and statistically significant correlations between measures of the
same trait using different methods.
Discriminant validity refers to the distinctiveness of the different traits;
it is inferred when correlations among different traits are less than one.
Intro
39
Quality of measurement
Universitat de Girona
39
Test–retest stability of networks comparing data collection techniques of network data (Example: Free recall vs Recognition method) The test is performed twice. Giving a group of participants the same questionnaire on two different occasions, the test-retest reliability is the correlation between separate administrations of the test is high.
Intro
40
Quality of measurement
Universitat de Girona
40
Q
U
A
L
I
T
Y
Multilevel
Multitrait
Multimethod
Data collection
Questionnaire Design
Reliability and validity estimates by country
Meta-analysis design (MCA)
Meta Analysis results
Intro
41
Multitrait Multimethod model
Universitat de Girona
41
The MTMM model has rarely been used for measurement quality assessment in social network analysis. Hlebec (1999), Ferligoj and Hlebec (1999), Kogovšek et al. (2002), egocentered networks (TS model).
For model identification purposes the MTMM approach usually requires at least 3
repeated measurements of the same variable, using three different methods
(Kenny, 1976) � Burden on the respondent and increases the cost of data
collection.
To reduce these problems:
Split ballot MTMM experimental design (Saris, 1999; Saris & Coenders,
2000) combinations of just two repetitions (methods).
Intro
42
Multitrait Multimethod model
Universitat de Girona
42
Reliability based on the classical test theory (Lord & Novick, 1968):
Yij = Sij + eij e Y S
• Yij is the response of variable i measured by method j.
• Sij is called the True Score (Saris & Andrews, 1991)
• eij is the random error, related to lack of reliability.
True Score is the result of Trait and Method:
Sij = mij Mj + tij Ti
Intro
43
Multitrait Multimethod model
Universitat de Girona
43
T1= Trait / variable of interest M = reaction to S= True Score the method Y = observed response e = random error
• The strength of the relationship between T and S is called Validity
Validity = 1- var (M)
Intro
44
Multitrait Multimethod model
Universitat de Girona
44
Confirmatory Factor Analysis (CFA) specification of the MTMM model:
Traits (Coromina, Coenders, Ferligoj, 2006): Yij = mij Mj + tij Ti + eij
Trait 1: Ask for scientific advice
Trait 2: Collaboration with your colleagues concerning research
Trait 3: Ask for information /data/software
Trait 4: Social activities outside the work
Intro
45
Multitrait Multimethod model
Universitat de Girona
45
Assumptions (Andrews, 1984):
Cov(Ti,eij)=0 ∀ij
Cov(Mj,eij)=0 ∀ij
Cov(Ti ,Mj)=0 ∀ij
It makes possible to decompose the variance of Yij
Trait variance tij2Var(Ti)
method variance mij2Var(Mj)
random error variance Var(eij)
Intro
46
Multitrait Multimethod model
Universitat de Girona
46
Y11 Y21 Y31 Y41 Y22 Y12 Y32 Y42
M1 M2
T1 T2 T3 T4
e11
e21 e31 e41 e12 e22 e32
e42
Intro
47
Multitrait Multimethod model
Universitat de Girona
47
Reliability increases not only with true or trait variance, but also with method variance, which also belongs to the stable or repeatable part of the measurements.
)(
)(
ij
ij
YVar
SVar
= )(
)()( 22
ij
iijjij
YVar
TVartMVarm +
Validity, % of variance of the True Score explained by the Trait:
)()(
)(
)(
)(22
22
iijjij
iij
ij
iij
TVartMVarm
TVart
SVar
TVart
+=
QUALITY, strength of the relationship between Y and T
= RELIABILITY * VALIDITY
Intro
48Universitat de Girona
Multilevel MTMM
48
Egocentered network data Two-level MTMM models.
The mean centred individual scores for group “g” and individual “k” can be decomposed into:
Ego (g) Between group component (g)
Alter1 Alter
2 … Alter
n Within group component (k)
YYY gkgkT −=
YYY ggB−=
ggkWgk YYY −=
Intro
49Universitat de Girona
Multilevel MTMM
49
WITHIN BETWEEN
Between level: Units of measurement are egocentered networks as a whole,
average across all egocentric networks � Quality of averages studied.
Within level: Individual ego-alter
Intro
50Universitat de Girona
Multilevel MTMM
50
Decomposition of the sample covariance matrix.
Total population covariance matrix ΣΣΣΣT
ΣΣΣΣT = ΣΣΣΣB + ΣΣΣΣW
Analysis of each component separately:
Yij = mij Mj + tij Ti + eij
Yij = mBij MBj + tBij TBi + eBij + mwij Mwj + twij Twi + ewij
YBij YWij
Intro
51Universitat de Girona
Interpretation
51
We obtain two reliabilities and two validities for each trait-method combination:
Between and within groups
Groups are respondents Different interpretation
Individuals are stimuli evaluated by respondents from standard way
The between-group reliabilities and validities
Quality of the measurement of the egocentered network as a whole (average).
If the ego is the focus of interest and these averages are used as data instead of
the raw responses regarding individual alters � Between-group measurement
quality is the relevant one to look at.
Intro
52Universitat de Girona
Interpretation
52
The within-group reliabilities and validities
Each subject is a separate unit of analysis and thus variance is defined across
stimuli presented to the same subject, not across subjects.
If the relationship is the focus of interest� within group measurement quality.
We know the between and within scores add to a total score:
Yij = mBij MBj + tBij TBi + eBij + mwij Mwj + twij Twi + ewij
YBij YWij
Then…
Intro
53Universitat de Girona
Interpretation
53
It is possible to compute percentages of variance.
Decomposition of variance:
Var(Yij) = mwij2Var(MWj) + mBij
2Var (MBj) +
twij2Var(TWi) + tBij
2Var(TBi)+
Var(ewij) + Var(eBij)
From the decomposition one can:
• compute overall reliabilities and validities:
By aggregating all trait, method, and error components.
• compute overall percentages of within and between variance:
By aggregating all within components and all between components.
Intro
54Universitat de Girona
Interpretation
54
• compute the percentage of between and within trait variance over the
total trait variance. A higher within percentage shows a higher alter variability
in the tie characteristics within a network and a lower variability of average
values of the tie characteristics between networks.
• compute a percentage of pure random error variance (i.e., within
error variance) over the total variance of the observed variables (grand total,
i.e., including all 6 components). The percentage of total variance explained by
any of the other 5 components can be computed in a similar way.
Intro
55
INSOC Project
Universitat de Girona
55
INSOC project
International Network on Social Capital and Performance
Intro
56
INSOC Project
Universitat de Girona
56
• Explain the academic PhD students success based on social capital.
• Develop and refine data collection methods.
• Develop social network analysis methods.
Intro
57
INSOC Project
Universitat de Girona
57
Web Survey design
• Verbal (words and numbers) + rich visual.
• Special navigational features (progress indicators), animations…
• Lower costs
• Faster data collection and data analysis process
• Easy and fast questionnaire modification.
• Response when convenient
• Piping: Assigning questionnaire items based on earlier answers from the
respondent
Intro
58
INSOC Project
Universitat de Girona
58
Coverage error !!!
Population with nearly universal internet access (PhD students)
Intro
59
INSOC Project
Universitat de Girona
59
INSOC Questionnaire Design
Focus groups
Comparable versions (Catalan, Slovenian and Dutch)
Two independent translations
Pre-test
Intro
60Universitat de Girona
Population, sample and data collection
60
PhD students at the Universities of Girona, Ljubljana and Ghent
April 2003: To know PhD students’ promoters
Personal interview to promoters � name generator questions
1. Name all the teaching assistants (or doctoral assistants) whose research is mainly under your supervision.
2. Name all the researchers of whom you are formally the mentor and who work on or participate in a
research project.
3. Name your colleague professors, senior researchers, junior researchers or people working in the private
sector with whom you substantially work together on those research projects in which PhD student X [name
PhD student] is involved.
Intro
61Universitat de Girona
Population, sample and data collection
61
November 2003: e-mail invitations (86 in Girona, 191 in Ljubljana and 233
in Ghent).
Follow-up: after 1, 4 and 8, weeks
PhD responses Method 1
2weeks PhD responses Method 2
Girona 67 78% 61 91% Ljubljana 118 62% 81 69% Ghent 198 85% 55* 60%
Method 2: Different question order, different style of response category
labels and different graphical display and lay-out of the questions.
Intro
62Universitat de Girona
Population, sample and data collection
62
0%10%20%30%40%50%60%70%80%
srv
Res
pons
es
Supervisors
PhD Students
1st reminder(2/12/03)
2nd reminder(23/12/03)
3rd reminder(23/01/04
to 26/01/04)
Intro
63Universitat de Girona
Traits & Methods
63
Frequency Traits
• Trait 1: Ask for scientific advice
• Trait 2: Collaboration with your colleagues concerning research
• Trait 3: Ask for information/data/software
• Trait 4: Social activities outside the work
sample repetition Factor 1 Factor 2 Factor 3 Girona Only one Main questionnaire by questions all labels plain Girona Only one Follow-up by alters end labels plain
Kogovsek & Ferligoj (2004, 2005)
Group N First interview Second interview
1 320 Face-to-face/by alters Telephone/by alters
2 311 Face-to-face/by alters Telephone/by questions
3 402 Telephone/by alters Telephone/by questions
Intro
64Universitat de Girona
Traits & Methods
64
Intro
65Universitat de Girona
Traits & Methods
65
Intro
66Universitat de Girona
Traits & Methods
66
Example of Method 2 in Girona
Intro
67
MTMM Results -- University of Girona
Universitat de Girona
67
Advice Collaboration Info Social act
within trait variance M1 67.5% 65.3% 55.6% 67.0%M2 66.1% 59.3% 47.7% 42.4%within method variance M1 0.0% 0.0% 0.0% 0.0%M2 9.6% 9.2% 12.3% 13.1%within error variance M1 13.3% 17.2% 17.3% 14.5%M2 7.0% 12.3% 16.3% 25.2%between trait variance M1 12.2% 13.1% 18.5% 11.4%M2 12.0% 11.9% 15.9% 7.2%between method variance M1 0.4% 0.4% 0.6% 0.9%M2 5.2% 5.0% 6.7% 7.1%between error variance M1 6.6% 3.9% 8.0% 6.3%M2 0.0% 2.3% 1.2% 4.9%
Goodness of fit: Yuan-Bentler χ2= 103.1 with 33df. TLI: .945
Intro
68
MTMM Results -- University of Girona
Universitat de Girona
68
% of within trait variance over all trait variance. The results show that most of the error free variance corresponds to the within level. This means that egos really discriminate between the different alters.
T1 T2 T3 T4 twij
2Var(TWi)/ [ twij2Var(Twi) + tBij
2Var(TBi)] 84.6% 83.3% 75.0% 85.4%
Within level Between level Overall level T1 T2 T3 T4 T1 T2 T3 T4 T1 T2 T3 T4 Reliability coef. M1 .91 .89 .87 .91 .81 .88 .84 .81 .90 .89 .86 .89 M2 .96 .92 .89 .83 1.00 .94 .98 .86 .96 .92 .91 .84 Validity coef. M1 1.00 1.00 1.00 1.00 .98 .98 .98 .96 1.001.001.00.99 M2 .93 .93 .89 .87 .83 .84 .84 .71 .92 .91 .88 .84 Quality coef. M1 .91 .89 .87 .91 .79 .86 .82 .78 .90 .89 .86 .88 M2 .89 .86 .79 .72 .83 .79 .82 .61 .88 .83 .80 .71
Intro
69
Meta analysis Design
Universitat de Girona
69
Meta-Analysis Design and Results
Intro
70
Meta analysis Design
Universitat de Girona
70
Summarizing the results of studies carried out by the universities of Girona, Ljubljana and Gent.
University sample repetition Factor 1 Factor 2 Factor 3 Girona (Spain) Only one Main questionnaire by questions all labels plain Girona (Spain) Only one Follow-up by alters end labels plain Ljubljana (Slovenia) 1 and 2 Main questionnaire by questions all labels plain Ljubljana (Slovenia) 1 Follow-up by questions end labels Graphical Ljubljana (Slovenia) 2 Follow-up by alters all labels Graphical Ghent (Belgium) Only one Main questionnaire by questions all labels plain Ghent (Belgium) Only one Follow-up by alters end labels plain
Categorical predictors (country, trait and factor 1 to factor 3)
Multiple classification analysis (MCA) was used
Estimation of the contribution of each factor on reliability and validity.
Intro
71
Meta analysis Design
Universitat de Girona
71
Question order: by alters or by questions.
Intro
72
Meta analysis Design
Universitat de Girona
72
Formulation by alters, with end labels and with a plain lay-out
Kogovšek et al. (2002) “by alters” seems to be more reliable for telephone
interviews.
Intro
73
Meta analysis Design
Universitat de Girona
73
Response category labels: for all categories or for the end points of the
response scale.
Revilla & Saris (2011) find more quality in all labeled.
Költringer (1995) reports no effect of the way of labeling on either reliability or
validity. Personal and telephone interviews and mostly related to attitudinal (i.e.,
not network) variables using vague category labels of the type “rather satisfied”,
“completely agree”
Our study: Category labels are not vague quantifiers. Precise actual frequencies of
behavior and thus additional labels may help respondents give precise answers
about the frequency of contact with their social network.
Intro
74
Meta analysis Design
Universitat de Girona
74
Lay-out of the questions and the web page: plain or graphical display.
Dillman et al. (1998b), who suggest that using a plain questionnaire provided
better results (response rate, less time…) than a graphical display version.
Deutskens et al. (2004) found that visual effects actually increase response quality.
Intro
75Universitat de Girona
Meta-analysis results
75
Means of each level of all factors, corrected by the levels of other factors
country trait Girona Ljubl. Ghent trait 1 trait 2 trait 3 trait 4 % of between trait variance .133 .088 .117 .079 .125 .118 .111
% of within trait variance .610 .612 .555 .674 .654 .552 .508
% of between method variance .024 .025 .002 .017 .015 .022 .030
% of within method variance .049 .074 .077 .052 .047 .072 .114
% of between error variance .049 .064 .057 .053 .039 .063 .078
% of within error variance .152 .166 .189 .137 .132 .186 .219
Between reliability .874 .735 .839 .794 .875 .804 .698
Between validity .921 .907 .943 .916 .945 .919 .876
Within reliability .899 .895 .875 .916 .917 .877 .856
Within validity .974 .932 .931 .963 .965 .936 .887
Overall reliability .885 .882 .869 .892 .913 .867 .849
Overall validity .963 .923 .935 .955 .960 .932 .867
% of trait variance at within level .823 .864 .826 .895 .840 .823 .804
Intro
76Universitat de Girona
Meta-analysis results
76
factor 1 factor 2 factor 3
by questions by alters all labels end labels plain lay-out
graph. lay-out
% of between trait variance .125 .082 .103 .116 .105 .117
% of within trait variance .650 .508 .595 .600 .593 .608
% of between method variance .013 .031 .006 .042 .026 .008
% of within method variance .035 .120 .070 .074 .082 .045
% of between error variance .065 .046 .061 .054 .059 .058
% of within error variance .136 .215 .181 .147 .163 .183
Between reliability .785 .809 .777 .825 .789 .806
Between validity .948 .880 .961 .856 .902 .958
Within reliability .912 .857 .884 .903 .894 .883
Within validity .975 .888 .942 .932 .927 .965
Overall reliability .895 .852 .873 .893 .884 .868
Overall validity .967 .893 .947 .914 .922 .962
% of trait variance at within level .835 .855 .850 .830 .844 .841
Intro
77Universitat de Girona
Meta-analysis results
77
Overall reliabilities and validities are acceptable: around or above 0.85
By traits (pattern is consistent at different levels):
Collaboration (T2)
Advice (T1)
Asking for crucial information (T3)
Socializing (T4)
Overall reliability and validity is higher when:
Social network questions in a survey are organized by questions
All labeled categories
Graphical display questionnaire design (not within validities)
Intro
78Universitat de Girona
Meta-analysis results
78
Country: No significant effect on reliability or validity.
% of trait variance at the within level:
Advice � Collaboration � Ask for information � Socializing
Scientific advice tends to be asked rather often to some group
members and not often to other group members by the same ego, but all
egos have a rather similar frequency average, while this occurs to a lesser
extent for socializing.
Intro
79Universitat de Girona
Meta-analysis results from Other Studies
79
Meta-Analysis Results from other Studies
Intro
80Universitat de Girona
Meta-analysis results from Other Studies
80
Kogovšek , Ferligoj , Coenders , Saris (2002) Coromina, Coenders, Kogovsek (2004) By type of networks: +: Frequency of contact is measured with the highest reliability Feeling of importance of the alter Feeling of closeness between ego and the alter - : Frequency of being upset.
Frequency of contact is less reliably measured over the telephone (speed of
telephone communication) compared with ‘the degree of closeness between the
ego and the alter’ or ‘importance of the alter’)
Intro
81Universitat de Girona
Meta-analysis results from Other Studies
81
Answering the question about frequency of contact is probably more
cognitively demanding than answering about the degree of closeness for which the
answer may be more readily available
Cognitively more demanding name interpreter questions would be more reliably
measured by face-to-face than by telephone mode.
Kogovsek & Ferligoj (2005) Reliability Results : Method +reliable: Telephone/by alters (except contact) Personal/by alters -reliable: Telephone/by questions
Intro
82Universitat de Girona
Meta-analysis results from Other Studies
82
When Telephone mode only � by alters (higher quality).
Telephone � speed of the mode � smaller elicited networks (would reduce
respondent burden and therefore lead to data of better quality)
BUT mean network size was actually smaller in the face-to-face (6.84) than in the
telephone mode (7.24).
Method: +reliable: face to face +valid: telephone (anonymity of phone) -reliable: telephone -valid: face to face Cognitively more demanding questions would be measured more reliably by face-to-face mode, and that questions that are potentially more sensitive would be measured more reliably in the telephone by alters condition.
Intro
83Universitat de Girona
Meta-analysis results from Other Studies
83
It is quite possible that context effects are stronger when the data is collected by questions, since the respondent answers the same question for all alters first and not vice versa.
If we consider a question relating to feelings of closeness, it is more likely that the respondent would compare the alters while answering the question. In that case, it would not really be the actual feelings of closeness towards each individual alter that would be measured, but the feelings of closeness relative to previous alters on the list.
Coromina et al. (2004): Not so clear that telephone method is better than face-to
-face method at the between level.
Telephone produces better quality data at the within level, and an analysis of the
SB such as the one done by Kogovšek et al (2002) is inevitably contaminated by
the within level structure according to E(SB)= ΣW + cΣB
Intro
84Universitat de Girona
Meta-analysis results from Other Studies
84
Social support (reliability) Ferligoj & Hlebec (1999)
+: Emotional support (discussion of important matters) +: Informational support (Illness ) Social Companionship (birthday party) -: Instrumental support (material) (importance of the topic to the respondents).
Response scale (the most important predictor of reliability estimates) Hlebec & Ferligoj (2002) +: 5 point category all labels. +: 11 point category scale ending
5 point category scale ending labels Line drawing scale
- : Binary scale
Intro
85Universitat de Girona
Meta-analysis results from Other Studies
85
Response scale +reliable: all labelled -reliable: ending labels Name Generator: Recognition data collection enhances the reporting of both strong and weak ties, especially when the measurement scales employed can also measure the strength of ties.
Ferligoj & Hlebec (1999) ( Recognition / Free recall ) no differences Interview design (2nd most powerful predictor for reliability estimates) +reliable : Two measures (when two measures of the same trait are presented in the same interview) -reliable: One measure (a question is presented alone in a questionnaire)
Intro
86Universitat de Girona
Meta-analysis results from Other Studies
86
Network size (not significant) 1–5 +valid: 1-5 6+ -valid: 6+ (larger proportion of weak ties and
measurement error more pronounced)
With smaller networks (important alters, consistent answers) the data collection technique (by alters/by questions) is more important than the data collection mode (telephone/face-to-face).
In larger networks: (seems that context effects are no longer so prominent). +valid: telephone -valid: face-to-face
Intro
87Universitat de Girona
Meta-analysis results from Other Studies
87
Type of question +reliable: Behavior +valid: behaviour .975 -reliable Emotional, but not very different -valid: emotional .958
Education Up to compl. second. (no effect) College or more
Age +reliable: 40 years or less +valid: 40 years or less -reliable: 41+ -valid: +41 (memory or hearing problems;
more weak ties...)
Intro
88Universitat de Girona
Meta-analysis results from Other Studies
88
Gender +reliable: female +valid: male -reliable: male -validity: female (maybe because women have
larger networks and more weak ties and might increase method effect)
Interaction Age / Gender -valid: older women +valid: other categories
Interaction Method / Gender Female/method Male/method +valid: face-to-face (.957) +valid: telephone (.999) - valid: telephone (.930) -valid: face-to-face (.928) Being interviewed about personal relationships by telephone suits both genders better, but face-to-face interviews suit women better than men.
Intro
89Universitat de Girona
Project Conclusions
89
Considering egocentered network data as hierarchical, Multilevel recommended.
Qualities at Within, Between and Overall level.
Combination of % from decomposed variance.
Egocentered Network data quality collected by Websurvey:
ordered by Questions
All categories of the scale labeled Overall and Within
Graphical display validity