The Unrecognized Interviewer Studying Respondent Behavior in an Establishment Survey of U.S. Academic Institutions Presented at the Third International.
Post on 24-Dec-2015
218 Views
Preview:
Transcript
The Unrecognized Interviewer
Studying Respondent Behavior in an
Establishment Survey of U.S. Academic
Institutions
Presented at the Third International
Conference on Establishment
Surveys (ICES-III)
June 20, 2007
Scott D. Crawford Survey Sciences
Group, LLC
Emilda B. RiversNational Science
Foundation
2
Our Establishment Survey
• Survey of Graduate Students and Postdoctorates in Science and Engineering (GSS)
• Surveys U.S. academic institutions granting graduate degrees in science, engineering (S&E), and selected health-related fields
• Jointly sponsored by NSF, the National Institutes of Health and the Department of Energy
3
Brief History of the GSS
4
What is collected in the GSS?
• GSS produces national estimates for S&E and selected health-related fields on: fall graduate enrollment counts
by demographic categories by main mechanisms and sources of financial support
postdoctoral (postdoc) appointment counts by demographic categories by main sources of support by first professional degrees in medical and related fields
non faculty research staff counts by demographic categories by first professional degrees in medical and related fields
5
How is the GSS data collected?
6
What potential errors exist in this effort?
• Errors of NonobservationSamplingCoverageNonresponse
• Observational ErrorsRespondent InstrumentMode
Very little is known!
7
The GSS Respondent: Very Little is Known
• Are they knowledgeable about both postdocs and graduate students? If so, are they the best person to respond
to these requests?• What resources do the respondents rely upon
to provide a response?• Do they have direct access to the source of
these data?• How are the data organized?• How do respondents translate definitional
issues in their response?
8
The Establishment Response Process
9
The Establishment Response Process
Response Behavior Survey
10
The Response Behavior Survey (RBS)
• Unique characteristics that make this approach feasibleHigh response rates to GSS
Demonstrates strong motivation to complete
Institutional nature of the data collection Many respondents have it as part of their job
responsibilities to respond
Survey design unchanged in years may have provided a culture where it is easy for respondents to critique
Comfort with the web in the GSS Allows for a rapid (web) follow-up
11
RBS Sample
• Frame GSS department contact person list
• Things considered Consistency with GSS design Large enough to identify trends and comparison
cells in specific response characteristics (quantitative baseline study)
Small enough to minimize burden on respondents in an ongoing study
Some GSS departments do not have postdocs Given the respondent-department link, we needed
to consider the fact that some respondents had multiple departments
12
RBS Questionnaire Content
• Respondent characteristics• Involvement in the GSS• Institutional characteristics• Identifying resources for the GSS
response• Data sources• Barriers to access of data sources• Assessment of data quality• Survey topic definitions – their use and
understanding
13
GSS / RBS Survey Flow
14
RBS Contact Strategy
• Data and mode of each contact effort for the early and late sample• Approximately 45 days of data collection required, most of it was
during the primary study (GSS) data collection period.
Contact Mode Days After Prev. Contact
1st Contact Heads up email from NSF NA
2nd Contact Invitation letter mailed 14 days
3rd Contact Invitation email sent 7 days
4th Contact 1st email reminder sent 6 days
5th Contact Letter reminder mailed 3 days
6th Contact 2nd email reminder sent 5-7 days
7th Contact 3rd email reminder sent 4 days
8th Contact Telephone calls All conducted in late Sept. and early Oct.
15
RBS Response & Completion Rates
Contact Mode Cumulative Response Rate (RR2)
Completion Rates
1st Contact Heads up email from NSF
0.0% NA
2nd Contact Invitation letter 1.8% NA
3rd Contact Invitation email 24.9% NA
4th Contact 1st email reminder 38.8% NA
5th Contact Letter reminder 45.9% NA
6th Contact 2nd email reminder 54.6% NA
7th Contact 3rd email reminder 59.4% NA
8th Contact Telephone calls 72.1% NA
OVERALL 72.1% 84.5%
EARLY SAMPLE 74.7% 85.3%
LATE SAMPLE 61.6% 80.7%
16
The “It’s Not Me” Phenomenon
• Right from the start the RBS uncovered something amiss in the response behavior of the departmental person “of record” 6.5% of the sample reported that they were not
involved in the 2005 GSS
• Potential causes (unknown) Sample frame problem – ineligible department Institutional responder using departmental responder
information for something other than originally intended – potentially indicates a need for a 2nd layer RBS
17
Who are our GSS respondents?
• DemographicsGender, 26.2% are maleEducation
29.0% have less than a BA/BS degree 25.4% only a BA/BS
• Position (top 4 categories)27.8% administrative24.2% administrative support10.9% department chair9.5% faculty
18
Who are our GSS respondents?
• Involvement in maintaining records56.6% maintain graduate school records18.5% maintain postdoc records Percent for maintaining postdoc records
only increases to 37.8% when looking at those in postdoc only departments
19
Who are our GSS respondents?
• Are professional survey respondentsHave been involved in the GSS for a mean
of 6.5 years29.5% respond to other NSF surveys, 52.3%
respond to other surveys in generalHowever, only 7.1% have been involved
with IPEDS• May not be the most knowledgeable
Only 58.9% believe they are the most knowledgeable person to answer questions about postdocs
• 19.9% are users of GSS data
20
What do respondent characteristics tell us?
Education• While the survey is meant to collect
data on grad students and postdocs, the responder is not likely to be of same educational status
Assumptions on definitions may be incorrect
Interpretations that are effecting counts may not be appropriate
21
What do respondent characteristics tell us?
Professional respondents • These are likely to be people with job
responsibilities to complete surveys
• With 52.3% of the GSS responders also responding to other general surveys, the potential exists for a more streamlined format that may rely on these professionals’ experience and resources
22
What do respondent characteristics tell us?
Most knowledgeable • With nearly half not the most
knowledgeable respondent, this raises questions about potential problems in:
The departmental respondent selection process
How the survey is structured
23
What did the GSS respondents report on their own?
• When asked about reporting all of the data for specific data types
The majority reported all of the data on graduate students
However, most were not able to report all of the data on postdocs
Relied on assistance and other resources to report
They reported that they had less knowledge of the computer systems that manage such data
24
RBS Measures of Data Quality
• Actual measures (break-off rates, item missing data in the GSS) were not sufficient to be used as a measure of data quality (few beak-offs and few item missing data)
• Self-reported estimate of measure quality was used – respondent provided their thoughts on the quality of the data they were providing.
Some internal validity was found as our expectations of correlations with self-assessed data quality panned out through further analyses.
25
Response Behavior Effects on Data Quality
Impact on Perception of Data Quality
Positive No Impact NegativeWhen there is a person who always responds to requests for student data (Postdoc counts andfinancial support data only)
Whether the respondent is usually selected by position or on an individual basis.
Whether the respondent is one of the people always responsible for responding to student data requests or the GSS
Whether respondents have been involved in providing data for other surveys
26
Response Behavior Effects on Data Quality
Impact on Perception of Data Quality
Positive No Impact NegativeData sources is easily accessible by respondents
With how the primary data source is maintained or stored
When GSS respondents do not know whether the department has an official postdoc definition
Data source is considered accurate by the respondent
The format of the primary data source (being aggregate or individual)
Respondents who consider their data source as “complex”
Respondent is familiar with how the data is entered in the primary data source
27
Response Behavior Effects on Data Quality
• It is clear that the current GSS departmental contact is not the right one to provide postdoc data in most cases. Future evaluations of postdoc data collection should focus on the development of protocols for the identification of the correct person.
Postdoc offices appear to be a key starting point for this exploration
Knowledge of postdoc policies, as well as knowledge and accessibility of data sources, should be explored as a way to quickly identify the correct postdoc reporting individual
28
Instrument Interaction in the Response Behavior Context
• A little more than 1/2 of GSS respondents indicated that they had read the instructions for the criteria of what constitutes a postdoc
Their perceived data quality was much higher than for those who had not read the criteria.
29
Instrument Interaction in the Response Behavior Context
• Less than half of departments’ postdoc definitions are consistent with the GSS’ definition.
• However, there is no significant difference in self-assessed data quality between departments whose definition is consistent with the GSS and those whose is not.
• While departments/institutions have different definitions of postdocs, if the respondent read the GSS definition, they were able to provide good quality data (perceived). Quality problems emerged when the definition was not used.
30
Instrument Interaction in the Response Behavior Context
RECOMMENDATIONS
• Definitions and their usability must be improved
• Efforts to improve postdoc data should examine why foreign counts are the easiest data
element to provide – something about that data makes it easier to report
31
The Practice of Responding
• Does time of response (early, middle or late) correlate with perceived postdoc data quality?
• What specific recommendations does the respondent have about improving data quality?
32
9 out 10 Reported that Current Data Collection Timing was Adequate
Those who didn’t, reported that these months would be better
0
5
10
15
20
25
30
35
40
Better Month 2.4 1.4 2.2 33.6 39.3 35.7 36.9 19.4 8.7 6.6 3.7 4.5 3.3 2.7 3.1 28.6
Jul-04
Aug-04
Sep-04
Oct-04
Nov-04
Dec-04
Jan-05
Feb-05
Mar-05
Apr-05
May-05
Jun-05
Jul-05
Aug-05
Sep-05
Don't
Kno
33
How did timing of response play a role in data quality?
• Middle responders provided lower perceived quality postdoc data than early and late responders.
• There was no difference between self-assessed data quality between early responders and late responders.
34
Postdoc Counts Varied by Response Timing to the GSS
• Does lower perceived data quality (middle responders) translate into meaningful or significant survey response differences?
Mean Count of Postdocs Reported per Department
3.01
4.03
4.74
00.5
11.5
22.5
33.5
44.5
5
EarlyResponder
MiddleResponder
LateResponder
35
Responses:The Web Mode
• Most GSS respondents used the web survey as their primary mode of providing data
• The majority of respondents reported a preference for the web mode
36
Responses:The Paper Mode
• The paper survey was used more as a worksheet to collect responses prior to submission.
Approx. one-third of the respondents rely on the paper survey to use as a worksheet for completing the Web later.
37
Top Responses for Improving Response Time & Data Quality
Regarding GSS submission Providing more time
– Earlier distribution of the GSS– Extending the deadline
Improving data contacts (frequency and the right person)
Regarding their own institutions Improving their databases Improving support, reducing workload, expanding
personnel
Regarding the format of the GSS Make the format more user-friendly, simplify the form Clarify wording and definitions Designate a contact person for support
38
Study Recommendations
• Consider additional efforts to support institutions and responders in their process of providing GSS and postdoc data
• Given differences in response patterns uncovered in the RBS, a more extensive nonresponse analysis of GSS data is recommended.
• Research the design of paper worksheet versions of the GSS rather than a form intended to be submitted.
39
Is the RBS an effective tool?
• Potential strengthsAssisted in the identification of potential
frame issues (“it’s not me”)Effectively described the demographic
characteristics of the individual responderProvided some understanding of how the
respondent – establishment relationship can effect data quality
Identified at least one area (timing of GSS) where conflicting results show that the answer for a redesign is not so simple
40
Is the RBS an effective tool?
• Potential weaknessesDid not capture much information about
other “people” who were involved in the response process
Self reported measure of quality has little hard evidence so far – should be validated
41
Is the RBS an effective tool?
• Overall, the answer is yes.
• Further research is required to further develop models of the response process when an individual is responding for and within the context of an establishment.
42
Thank you.
Questions / Comments:Scott D. Crawford
scott@surveysciences.com
top related