European Health and Social Integration Survey...List of countries 6 EU Comparative Quality and Technical Report: European Health and Social Integration Survey List of countries AT

European Health and Social Integration Survey EU comparative quality report June 2015

Table of contents

3 EU Comparative Quality and Technical Report: European Health and Social Integration Survey

List of abbreviations .................................................................................................................. 5 List of countries ......................................................................................................................... 6

1. INTRODUCTION ..................................................................................................................................... 7 1.1 Survey Coverage and Timing ............................................................................................. 7 1.2 Assessing Cross-National Survey Quality .......................................................................... 8

2. BACKGROUND AND METHODOLOGY .................................................................................................. 9 2.1 Cross-national Harmonisation ............................................................................................ 9 2.2 Basic Concepts ................................................................................................................. 10

2.2.1 Measuring Disability in EHSIS ................................................................................... 10 2.2.2 Other Measures of Disability ...................................................................................... 11 2.2.3 Measuring the Severity of Disability in EHSIS ........................................................... 11

3. QUALITY CONTROL ............................................................................................................................. 12 3.1 Contractors’ Overall Quality Assessment ......................................................................... 12

3.1.1 EU Core Social Variables .......................................................................................... 13 3.1.2 Lessons Learned and Contractors’ Suggested Improvements ................................. 13

4. PARTICIPATION ................................................................................................................................... 15 4.1 Countries Taking Part ....................................................................................................... 15 4.2 Survey Population ............................................................................................................. 16 4.3 Data Collection Period ...................................................................................................... 16 4.4 Survey Reference Period .................................................................................................. 17 4.5 Voluntary Nature of the Survey ......................................................................................... 17

5. ACCURACY ........................................................................................................................................... 18 5.1 Sample Design .................................................................................................................. 18

5.1.1 Sample Size ............................................................................................................... 19 5.1.2 Substitution ................................................................................................................ 20

5.2 Sampling Errors ................................................................................................................ 20 5.3 Nonsampling Errors .......................................................................................................... 24

5.3.1 Coverage Error .......................................................................................................... 24 5.3.2 Nonresponse .............................................................................................................. 24

5.3.2.1 Imputation ........................................................................................................... 28 5.3.3 Measurement Error .................................................................................................... 28

5.3.3.1 Data Collection Mode ......................................................................................... 28 5.3.3.2 The Questionnaire .............................................................................................. 30 5.3.3.3 The Interviewer ................................................................................................... 34 5.3.3.4 The Respondent ................................................................................................. 38

5.3.4 Data Processing Error ............................................................................................... 39 5.3.4.1 Data Entry, Coding and Cleaning ....................................................................... 39 5.3.4.2 Weighting Adjustments ....................................................................................... 42

6. REFERENCES ...................................................................................................................................... 46 7. ANNEXES .............................................................................................................................................. 48

Annex A .................................................................................................................................. 48 Annex B .................................................................................................................................. 52 Annex C .................................................................................................................................. 72

Table of contents


Figures Figure 5.1: Response Rates by Mode and Country ........................................................................... 25

Tables Table 1.1: Contractual Wave by Country ............................................................................................... 7

Table 1.1: Contractual Wave by Country (continued) .......................................................................... 8

Table 1.2: Common Dimensions of a Survey Quality Framework ..................................................... 8

Table 2.1: Overall Measure of Disability .............................................................................................. 10

Table 4.1: Countries Administering the Survey ................................................................................... 15

Table 4.2: Data Collection Period ......................................................................................................... 16

Table 5.1: Sampling Frame, Unit and Design by Country and Mode of Data Collection .............. 19

Table 5.2: Minimum Required and Achieved Sample Size by Country ........................................... 21

Table 5.3: Proportions, Standard Errors and Confidence Intervals for Six Key Disability Measures by Country .............................................................................................................................. 22

Table 5.4: Percent of Nonrespondents in Each Country by Reason for Nonresponse ................. 26

Table 5.5: Average, Minimum and Maximum Item Nonresponse Rates by Country ..................... 27

Table 5.6: Mode of Data Collection by Country .................................................................................. 29

Table 5.7: Languages Used to Conduct the Survey by Country ...................................................... 30

Table 5.8: Interview Duration by Mode of Administration .................................................................. 33

Table 5.9: Respondent/Interviewer Ratio by Country and Mode of Administration ....................... 36

Table 5.10: Mean Number of Contact Attempts by Country and Mode of Administration ............ 37

Table 5.11: Percent Proxy Interviews by Country ............................................................................... 38

Table 5.12: Data Processing Errors/Issues Detected by Country and Mode of Administration .. 41

Table 5.13: Weighting Procedure by Country ..................................................................................... 43

Table A-1: Annex A: Questionnaire Guide ........................................................................................... 48

Table B-1: Comparison of Population Percentages to Weighted and Unweighted Sample Estimates of Demographic Characteristics by Country ...................................................................... 52

Table C-1: Question Modifications and Problems by Country .......................................................... 72

List of abbreviations


List of abbreviations CAPI Computer-Assisted Personal Interviewing

CATI Computer-Assisted Telephone Interviewing

CAWI Computer-Assisted Web Interviewing

EHIS European Health Interview Survey

EHSIS European Health and Social Integration Survey

ESS European Statistical System

EU European Union

ICF International Classification of Functioning, Disability and Health

ITT Invitation To Tender

ISCED International Standard Classification of Education

ISCO International Standard Classification of Occupations

LFS Labour Force Survey

MEHM Mini-European Health Module

MS Member State

NR Nonresponse

PAPI Paper-And-Pencil Interviewing

TSE Total Survey Error

TSQ Total Survey Quality

RDD Random Digit Dialing

List of countries


List of countries AT Austria

BE Belgium

BG Bulgaria

CZ Czech Republic

DK Denmark

DE Germany

EE Estonia

EL Greece

ES Spain

FR France

IT Italy

CY Cyprus

LV Latvia

LT Lithuania

LU Luxembourg

HU Hungary

IS Iceland

MT Malta

NL Netherlands

NO Norway

PL Poland

PT Portugal

RO Romania

SI Slovenia

SK Slovakia

FI Finland

SE Sweden

UK United Kingdom

Introduction


1

Acknowledgements This publication is the outcome of coordinated efforts by Pascal Wolff and Lucian Agafitei.

SOGETI Luxembourg SA supported the work, the team being composed of Laurent Jacquet (project manager), Julien Arcade (statisticians and SAS programmers). Furthermore, methodological support was provided by Amanda Wilmot and Howard Meltzer (University of Leicester).

1. INTRODUCTION This report provides a summary of the quality and technical aspects of the European Health and Social Integration Survey (EHSIS). The EHSIS is a cross-national survey carried out for the first time in 28 European countries in 2012/13 with the goal of providing comparable cross-national data on disability.

The design of EHSIS was different from most other disability surveys. Taking account of the International Classification of Functioning Disability and Health (ICF) conceptual approach to disability, questions were designed to measure the biopsychosocial model of disability, to take account of the person, their impairments, activity limitations and the environment and the interrelationships between them. The concepts used to measure disability in this way are described further in section 2.

1.1 Survey Coverage and Timing The survey was tendered in 2011 by Eurostat (2011/S 122-201426 and 2011/S 249-404645). The invitation was extended to the 27 European Member States1 (MS) plus Iceland and Norway.

The contract for Ireland was cancelled and, therefore, the survey in this country did not take place. Survey preparation, administration and delivery of final tables along with each country’s quality and technical reports took place between December 2011 and April 2014. The contracts were let in two waves. Table 1.1 shows which countries conducted the survey according to each wave. The first wave was tendered to begin December 2011 for a 21-month duration; the second wave was tendered to begin April 2012 for a 17-month duration. In the second wave, the field work period was extended to maximum 6 months. Spain run the survey without participating to the above-mentioned calls for tenders.

Table 1.1: Contractual Wave by Country

Wave Country Wave Country

1 Bulgaria 2 Austria

Denmark Belgium

Estonia Cyprus

France Czech Republic

Germany Finland

Greece Iceland

Hungary Italy

(continued)

1 Croatia was not a Member State at the moment of launching the call for tender

1 Introduction


Table 1.1: Contractual Wave by Country (continued)

Wave Country Wave Country

Latvia Luxembourg

Lithuania Netherlands

Malta Norway

Poland Portugal

Romania Sweden

Slovak Republic

Slovenia

United Kingdom

1.2 Assessing Cross-National Survey Quality The aim of this quality report is to provide information on how well the survey satisfies the goals for each quality component. In order to facilitate the reporting of survey quality across countries, guidelines are provided by the European Statistical System (ESS) based on a standardized Quality Framework. In this report, output quality is assessed in line with the ESS Quality Framework Guidelines and is reported in terms of the components in Table 1.2.

Table 1.2: Common Dimensions of a Survey Quality Framework

Accuracy Total survey error is minimized.

Credibility Data are considered trustworthy by the survey community.

Comparability Demographic, spatial and temporal comparisons are valid.

Usability/Interpretability Documentation is clear and metadata are well managed.

Relevance Data satisfy user needs.

Timeliness/Punctuality Data deliveries adhere to schedules.

Completeness Data are rich enough to satisfy the analysis objectives without undue burden on respondents.

Coherence Estimates from different sources can be reliably combined.

Source: Biemer, P. (2010). Total survey error. In M. Lovric (Ed.), International encyclopedia of statistical science. New York: Springer.

It is worth noting that, when assessing total survey quality, a fitness-for-use concept is incorporated that includes, relevance, usability and timeliness of the data production. The report is based on the ESS guidelines. However, it is reliant on the information provided by each individual country conducting the survey. Each in-country contractor was required to provide a quality report based on a quality reporting template provided by Eurostat. All of the described quality components are covered. This report also documents the methodology as well as the administration of the survey, thus fulfilling the function of a technical report as well as a quality report. This report will describe the process of conducting the survey and comment on the survey quality components therein with the aim of highlighting any shortfall in quality that may have affected survey estimates.

National quality and technical reports delivered to Eurostat constitute the source for the present report

Background and methodology


2

2. BACKGROUND AND METHODOLOGY 2.1 Cross-national Harmonisation In order to facilitate cross-national comparison using one data collection instrument, a decision was taken to provide harmonised inputs for EHSIS by providing a model questionnaire. The questionnaire provided specified question wording that had been previously tested across a number of European countries, both through quantitative pilot study and qualitative cognitive pretesting. The development of the survey questionnaire was iterative and comprised an initial wave of testing and reviews with three Member States: United Kingdom, Italy and Lithuania (Meltzer 2008). A second wave of testing and review was carried out in 10 Member States involving: Bulgaria, the Czech Republic, Estonia, Finland, Greece, Hungary, Latvia, Malta, Slovakia and Spain. (Wilmot and Meltzer 2010).

In addition to attempting to harmonise the questionnaire, and in order to further control for other variants when administering a survey cross-nationally, the survey specification required certain elements to be functionally harmonised whilst being adaptive to specific country situations. Eurostat provided additional technical specifications to be followed. These included: the timing of the data collection; specified target population; sample selection including minimum sample size; mode of administration (2) ; the data collection instrument and interviewer instructions (3), translation protocol and pretesting requirements; the technique and organisation of the interviews; data processing; validation and weighting rules, and the way in which the data were to be delivered. Each is covered in this report. The introduction to each section of the report will describe the relevant specification requirement.

Survey errors can arise throughout the statistical value chain. Although the aim of the survey administration is, of course, to maximize the precision of measurement, there will undoubtedly be sources of survey error that affect that precision. While some sources of error can be statistically controlled for, others cannot be quantified or controlled and therefore, in such instances, a qualitative assessment of those errors, such as translation issues or adaptions made to the questionnaire, is made or simply described.

(2) It should be noted that although the EHSIS questionnaire was originally designed to be interviewer-administered face-to-face, the tender specifications, although strongly recommending this mode of data collection, also allowed for telephone or web-based data collection.

(3) The classificatory questions were also prespecified and included the 2011 EU core social variables, although it should be noted that these questions are specified according to harmonized outputs, rather than inputs. CORE VARIABLES: UPDATED 2011 GUIDELINES Doc. Eurostat/F/11/DSS/01/2.3EN, MEETING OF THE EUROPEAN DIRECTORS OF SOCIAL STATISTICSLUXEMBOURG, 21-22 SEPTEMBER 2011

2 Background and methodology


2.2 Basic Concepts 2.2.1 Measuring Disability in EHSIS EHSIS was designed to measure the biopsychosocial model of disability. This is not the same as measuring disability according to the more traditional medical model, which locates the problem within the individual. According to the biopsychosocial model applied to the survey, people with disabilities are those who face barriers to participation in any of 10 domain activities from the ICF identified as being of most relevance, associated inter alia with a health problem or basic activity limitation.

Each section of the questionnaire corresponds to one of the 10 life domains identified: mobility; transport; accessibility to buildings; education and training; employment; the internet; social contact and support; leisure pursuits; economic life, and the attitudes and behaviour of others. Each section of the questionnaire examines different facets of life that enable an individual to be a fully functional and integrated member of society. Each section covers disadvantage or restrictions to social participation that people, with and without a health problem, face in their everyday lives. Hence, information is collected on the experience of barriers in each of the 10 life domains.

Owing to the constraints of producing a questionnaire of reasonable length, plus the need for questions to apply to everyone across Europe, the decision was made to sacrifice level of detail in order to cover the variety of social circumstances. Each section starts with a question aimed at identifying any barriers to social integration identified by survey respondents. The section continues by looking at the contribution of health conditions or basic activity limitation to that experience. The reasons people have problems in social integration can be a mixture of personal factors, environmental factors, as well as health conditions or difficulties with basic activities; and in many cases they operate simultaneously. To try to reflect real life, the questions on barriers list these factors, and respondents could rate as many items they feel contribute to their lack of social integration in each domain. A person would be categorised as disabled if he or she selected a health condition or activity limitation at any of the barrier questions in each of the 10 life domains included in the survey (Meltzer 2011) —see Table 2.1. The questions focused on barriers to participation, as opposed to facilitators that may already have been put in place and that people may not be aware of or have taken for granted.

Table 2.1: Overall Measure of Disability

Section Variable description

Mobility Barriers to mobility

Transport Barriers to using private vehicle

Transport Barriers to using other forms of transport

Accessibility to buildings Barriers to accessing buildings

Education and Training Barriers to education and training

Employment Barriers to employment

Internet use Barriers to using the internet

Social contact and support Barriers to speaking with people feel close to

Leisure pursuits Barriers to pursuing hobbies or interests

Leisure pursuits Barriers to attending cultural events

Economic life Barriers to paying for the essential things in life

Attitudes and behaviour Reasons for feeling treated unfairly

Background and methodology


2

2.2.2 Other Measures of Disability In addition to the EHSIS measure of disability described above, the survey questionnaire is supplemented with other measures of disability used in cross-national European surveys namely: the three MEHM (4) questions on self-perceived health, chronic conditions and activity limitation, used in surveys such as the European Health Interview Survey (EHIS) and the European Union Statistics on Income and Living Conditions (EU-SILC); questions from the Labour Force Survey 2011 Ad Hoc Module on the employment of disabled people that presented a list of health problems and basic activity limitation questions: seeing, hearing, walking etc.; as well as questions used in the EHIS on activities of daily living (ADL): feeding oneself, dressing, bathing etc.; and Instrumental Activities of Daily Living (IADL): preparing meals, shopping, light housework etc. The consequence of adding these extra questions is that alternative definitions of disability can be derived and cross-tabulations made with the EHSIS definition (see Meltzer et al. 2010).

2.2.3 Measuring the Severity of Disability in EHSIS Several measures of severity can be derived from the survey. For the biopsychosocial model, severity of disability would have to take account of whether a health problem or a basic activity limitation was a contributory factor in the lack of participation. So only participation restrictions linked to health problems or impairments are included in the measure.

A further refinement can be added by looking at the supplementary questions asked of those with a participation restriction related to a health problem who also have a lack of special equipment and/or personal assistance. This has a great advantage over questions which just ask respondents to report a degree of difficulty as such distinctions can be interpreted in different ways.

Therefore, one can have four levels of severity based on these questions.

- Lacks no specialised equipment or personal assistance - Lacks specialised special aids or equipment - Lacks personal help or assistance - Lacks specialised equipment and personal assistance

Another way for measuring severity is by adding up the number of life domains where a person encounters a barrier associated with a health problem or basic activity limitation.

(4) Mini European Health Module

3 Quality control


3. QUALITY CONTROL 3.1 Contractors’ Overall Quality Assessment One section of the quality reporting form asked each country contractor to provide an overall assessment of the strengths and weaknesses of the survey; these assessments are discussed in this section. (Note that some quotes are provided in italics to illustrate and support the points discussed.)

Overall contractors reported positively, both on the aims of the survey and on the overall quality of the survey and its administration.

“The strongest point about the EHSIS is its set-up and existence: it deserves a positive assessment that the EHSIS attempts to close the obvious data and information gaps…in context of the international monitoring requirements of the UN Convention on the Rights of Persons with Disabilities and the European Disability Strategy. “ [Germany]

“The strengths of the survey are: the purpose and value of the project.” [Lithuania]

“Overall, the survey was of high quality, and very few issues were experienced.” [Iceland]

“The EHSIS project was prepared and conducted very precisely with big attention on the details. The general organisation of the project was at a very high level.” [Bulgaria]

A number of countries commented that both interviewers and respondents recognised the importance of the subject matter.

“The subject of the survey is generally perceived as important and worthwhile for both interviewers and respondents.” [Norway]

“The survey was clearly important to respondents; those who agreed to participate were keen to do so.” [Finland]

“The respondents found the content of the survey as very interesting and they did not report on difficulties answering the questions.” [Slovenia]

The questionnaire and interviewer instructions were said to be clear and well structured. In general, survey respondents were able to comprehend and respond to the questions asked.

“Questionnaire handled a potentially sensitive issues—disability—in a very appropriate manner.” [Netherlands]

“We were impressed by the quality of the questionnaire.” [Poland]

“Strong points—questions were easy to understand and answer.” [Cyprus]

One of the main weaknesses of the survey was said to be the length of the questionnaire, which also seemed repetitive to those respondents who did not report any health problems or activity limitations.

“Weakness points—length of interview.” [Cyprus]

“The respondents found the questionnaire very long indeed.” [Denmark]

“Some respondents especially those without any physical disabilities, found the questions to be rather repetitive. However, we understand the necessity of this questioning technique, and so would not recommend any changes.” [Sweden]

The repetitive nature of the questionnaire may have been more apparent for those taking part in a telephone interview. The questionnaire was originally designed to be interviewer-administered face-to-face, either CAPI or PAPI. It is recommended that a questionnaire designed for telephone or web administration should be prepared for any future rounds of the survey in order to help ensure comparability across modes. See Table 5.7 for mode of data collection.

Quality control


3

“The quality of the survey is good for personal interviews. When it comes to telephone and web interviews we are uncertain of the effect on quality.” [Denmark]

“This questionnaire is designed better for CAPI or CAWI interviews that for CATI interviews…the interviewer has to read all of the options of the ‘life domain question’ that are very repetitive and it causes fatigues for the interviewed people.” [Spain]

3.1.1 EU Core Social Variables The requirement for the use of the EU core social variables, which are harmonised on outputs rather than inputs, caused some problems at the development stage of the survey. Contractors were unsure of question wording or had difficulties mapping outputs to the international classifications used, although a list of in-country coordinators/advisers for the EU core social variables, located within the respective National Statistical Institutes, was provided by Eurostat. Contractors asked for the actual question wording for the core social variables for any future rounds of the survey.

3.1.2 Lessons Learned and Contractors’ Suggested Improvements The main issue was conducting the survey in modes other than face-to-face. Because each response option was read aloud by telephone interviewers (that would otherwise have been shown using a card), contractors administering the survey by telephone reported that the interviews were long and repetitive, which could have led to respondents terminating the interview.

“The questionnaire was thoroughly tested in advance, but it was not tested for other methods than face-to-face interviewing.” [Denmark]

“Hard to keep respondents attention due to repetition of questions, Respondents get a feeling of the “the same over the same” and get tired.” [Lithuania]

“From our point of view, the questionnaire structure was too reiterative and tedious.” [Luxembourg]

Some of the contractors said that they would have preferred a longer field period in order to accommodate the complications of collecting data during the winter. In the Czech Republic, for example, performing data collection during a long and difficult winter season made face-to-face survey administration more difficult. Another contractor stated:

“The cold winter (with short days) made fieldwork even more difficult; it would be better to carry out this type of study in late Spring/early Summer or very early in the Autumn.” [Italy]

Overall, contractors recommended a longer field period.

“If possible the field period should be longer than 6 months.” [Finland]

“Improvement could have been done if the field work would be longer.” [France]

The length of the data collection period was reported as having affected survey response rates. It was suggested that Eurostat could have provided further guidance on maintaining response rates and avoiding refusals from respondents. Italy suggested including the Eurostat letter head in the advance letter and other respondent materials to improve the response to the survey. Eurostat was mentioned in the suggested text for the advance letter. However, Eurostat had no evidence that using its logo helps with response.

It was also suggested that substitutions should be allowed where a field period is restricted. Indeed, Bulgaria and Lithuania reported that sample substitution had been permitted—but with no further

3 Quality control


explanation. Cyprus selected additional Primary Sampling Units “due to the fact that some roads had no houses or high nonresponse rate in some areas (i.e., high class areas) and areas with foreigners.” See section 5.1 sample design.

Another issue related to the section on employment and education was that additional explanation was required when the question was asked of older retired people; they felt these sections did not apply to them.

The United Kingdom, a CATI survey, expressed concern over the length of the introduction and therefore allowed interviewers to shorten the introduction by explaining who the survey was for at the end and not at the start, as stipulated.

One of the issues that caused problems for contractors was in applying the correct weights.

Participation


4

4. PARTICIPATION 4.1 Countries Taking Part Twenty-eight countries took part in the survey as shown in Table 4.1: 26 Member States of the European Union (5), plus Iceland and Norway.

Table 4.1: Countries Administering the Survey Country Country Abbreviation

Austria AT

Belgium BE

Bulgaria BG

Cyprus CY

Czech Republic CZ

Denmark DK

Estonia EE

Finland FI

France FR

Germany DE

Greece EL

Hungary HU

Iceland IS

Italy IT

Latvia LV

Lithuania LT

Luxembourg LU

Malta MT

Netherlands NL

Norway NO

Poland PL

Portugal PT

Romania RO

Slovak Republic SK

Slovenia SI

Spain ES

Sweden SE

United Kingdom UK

(5) The survey was not conducted in Croatia and Ireland.

4 Participation


4.2 Survey Population In each participating country the target population was all those aged 15 or over living in private households. In France the target population was described as individuals aged 15 or over living in private households and resident in metropolitan France. In Malta the territory covered included Malta and Gozo, while in Portugal the target population was described as “individuals aged 15 or over living in private households in localities with 10 or more dwellings, living in Mainland Portugal.”

In each country the survey was administered to only one person in a household.

4.3 Data Collection Period The length of the field period can affect response rates, as commented on in some national reports. For most countries, field work took place during autumn 2012 and beginning of 2013. Table 4.2 shows the data collection periods for each country taking part in the survey. Data collection lasted from 1.5 months (Hungary) to 8 months (Portugal).

Table 4.2: Data Collection Period

Country Data Collection Period

AT 8 October 2012–28 February 2013

BE 18 September 2012–15 January 2013

BG 1 September 2012–30 November 2012

CY 15 October 2012–10 January 2013

CZ 16 November 2012–12 April 2013

DK 6 August 2012–10 October 2012

EE 14 September 2012—3 December 2012

FI 14 September 2012–26 March 2013

FR 7 September 2012 -20 December 2012

DE 15 August 2012 -3 0 November 2012

EL 7 September 2012-13 December 2012

HU 15 September 2012-30 October 2012

IS 18 September2012–28 November 2012

IT 1 February 2013–3 June 2013

LV 1 September 2012 to 9 December 2012

LT 5 September 2012—16 December 2012

LU 23 October 2012–28 February 2012

MT 2 September 2012 to 30 November 2012

NL 20 September 2012–11 January 2013

(continued)

Participation


4

Table 4.2: Data Collection Period (continued)

Country Data Collection Period

NO 5 November 2012—18 February 2013

PL 6 September 2012–12 December 2012

PT 5 November 2012—9 July 2013

RO 15 October 2012–20 December 2012

SK 1 September 2012—30 November 2012

SI 24 August 2012–15 December 2012

ES 24 September 2012 to 13 March 2013

SE 21 September 2012–9 January 2013

UK 13 September 2012–3 February 2013

4.4 Survey Reference Period There were no specific reference periods for the survey. The survey questions asked about the respondent’s current situation at the time of interview.

4.5 Voluntary Nature of the Survey In all countries, respondent participation in the survey was voluntary.

5 Accuracy


5. ACCURACY Surveys collect information about a specific population and report estimates of unknown population characteristics. Such estimates may or may not match exactly the unknown true population values. The accuracy of such estimates may be affected by various errors of different magnitude and from different sources that can potentially bias the estimates or contribute to greater variability from the true values. We divide this section into two major subsections, based on error sources—sampling errors and nonsampling errors. We summarize the methodological challenges across countries in dealing with these sources of error and what has been done to minimize and adjust for them.

5.1 Sample Design The tender specifications stated that the sampling design should be based on a probability sampling method that ensures accurate and representative results for the whole population within that country. Based on information provided in the technical and quality reports, all countries used probability-based samples. The target population for all countries was defined as individuals 15 years of age or older, living in private households. One eligible member per household was selected in each country. When the sampling frame was at the household, rather than person level, the Kish or last birthday methods were used for selecting the respondent.

Even though the recommended mode for the EHSIS was face-to-face, many countries employed different modes; thus, a variety of sampling designs were used. A majority of countries employed some kind of multi-stage stratified design. Table 5.1 summarizes the sampling design used in each country by mode of data collection. The most common design for face-to-face modes was stratified cluster sampling, implemented in eight countries; while the most common sampling design for the telephone mode was stratified random sample, used in all but one of the 14 countries conducting the survey via telephone.

Seven countries (CY, FI, HU, LV, NO, PL and PT) oversampled the youngest members of the population (15-24 years old) under the premises that the disability prevalence in such subpopulations is lower and overrepresentation is needed for having reliable data (6).

A variety of sampling frames were employed across countries. In all countries the frames were as exhaustive as possible with respect to the target population and in majority of the countries, all sampling units had a known, non-zero probability of selection (7). Population registers were used in 11 countries, random digit dialing (RDD) and list-assisted RDD was used in 10 countries, and area probability samples were drawn in the remaining seven countries. In 18 countries, the sampling frame was updated during the year of the survey data collection (2012); in six countries (CY, CZ, IT, LT, MT, and ES) the frame was a year older. An older frame (8) (often supplemented by 2011 census data) was used in three countries (EL, PT, RO). Only EE indicated ‘no applicable’ for the year of the sampling frame.

6 Oversampling of young population was recommended in the ITT 7 Several countries (Bulgaria, Cyprus, Greece, Portugal and Lithuania) used the random walk approach within each PSU. In Bulgaria,

probabilities of selection were not calculated even at the within-household level. In Cyprus, Greece and Lithuania, within household probabilities of selection were calculated. Portugal’s sampling weight included probabilities of selection at the PSU, housing unit and within-household level.

8 Frames dated between 2001-2006

Accuracy


5

Table 5.1: Sampling Frame, Unit and Design by Country and Mode of Data Collection

Mode Sampling Frame Sampling Unit Sampling Design Country

Face-to-face

Area probability Address Stratified cluster sample BG, CY, CZ, EL, IT, LT, PT

Population register Address Stratified random sample PL

Population register Address Stratified systematic sample

RO

Population register Individual Stratified random sample FI, HU, SI

Population register Individual Stratified cluster sample LV (a)

Population register Individual Stratified systematic sample

MT

Telephone List (landline)

List-assisted RDD (mobile)

Household (landline)

Individual (mobile)

Stratified random sample BE, NL, SE

List (landline)

RDD (mobile)


Individual (mobile)

Stratified random sample SK

List-assisted RDD (landline)

RDD (mobile)


Individual (mobile)

Stratified random sample (landline)

Systematic sample (mobile)

UK

RDD Household (landline)

Individual (mobile)

Stratified random sample AT, DE

RDD Household (landline)

Individual (mobile)

Simple random sample EE, FR

RDD Household Simple random sample LU

Population register Address Stratified random sample ES (b)

Population register Household (landline)

Individual (mobile)

Stratified random sample NO

Population register Individual Systematic sample DK (c)

Population register Individual Stratified random sample IS

(a) Data collection methods also included CATI (b) Web option available

(c) Web option for individuals with unknown phone numbers

5.1.1 Sample Size

5 Accuracy


Three definitions of sample size are of interest—the minimum required sample size, the achieved sample size (number of completed interviews), and the effective sample size (the achieved sample size, divided by the design effect (9)). The tender specifications included a minimum required sample size for each country. Table 5.2 presents the required and achieved sample size by country. The effective sample size is not included because most countries did not report design effects.

Assuming a design effect (10) of 1, all countries achieved or exceeded the required minimum sample size. However, this information should not be taken for granted, as most countries employed complex sample designs and weighting; thus, the design effect is likely to exceed 1. This means that if effective sample size is considered, the match between achieved and effective may not be perfect.

5.1.2 Substitution Only Bulgaria, Cyprus, and Lithuania used substitution. Bulgaria did not provide reasons for substitutions. In Cyprus, after three unsuccessful attempts, a new unit was selected. In Lithuania, substitution was allowed for unsafe area, non-residential addresses, or if the addresses could not be accessed for any reason (buildings with security codes, uninhabited villages, etc). Substitution sample points were taken by continuing the random route from the previous visited apartment or house.

5.2 Sampling Errors Sampling errors result from the fact that not all units in the target population are surveyed. That is, instead of taking a census of the population, we take a sample; thus, a replication of the survey will result in a different sample and different estimates. Sampling error can be controlled through appropriate sampling design and sufficient sample size. Probability based samples can quantify the amount of sampling error present in the estimates—this is typically expressed as a confidence interval around the estimate.

Table 5.3 presents the estimates for six disability measures and the 95% confidence interval around these estimates.

(9) The loss in efficiency due to the use of a complex sampling design

(10

) Loss in efficiency due to complex survey design and weighting

Accuracy


5

Table 5.2: Minimum Required and Achieved Sample Size by Country

Country Minimum Required Sample Size Achieved Sample Size

AT 5,500 5,522

BE 5,800 5,804

BG 6,586 6,741

CY 3,000 3,002

CZ 5,800 5,807

DK 5,784 5,934

EE 4,253 4,253

FI 4,600 4,615

FR 15,727 15,768

DE 18,452 18,512

EL 7,564 7,567

HU 7,228 7,745

IS 3,000 3,007

IT 12,000 12,041

LV 4,675 7,515

LT 5,096 6,038

LU 3,000 3,004

MT 3,795 3,815

NL 7,000 7,011

NO 4,500 4,511

PL 12,713 12,938

PT 6,000 6,397

RO 9,854 9,962 (a)

SK 5,816 5,822

SI 4,561 4,641

ES 13,835 14,614

SE 5,500 5,500

UK 15,679 16,469

(a) Includes partial interviews

5 Accuracy


Table 5.3: Proportions, Standard Errors and Confidence Intervals for Six Key Disability Measures by Country

Country

Longstanding Illness or Health Problem

Severely Limited in Activities due to Health Problems for at Least the Past 6 Months

Prevented Buildings Use due to a Longstanding Health

Condition, Illness or Disease

Prevented Buildings Use due to Longstanding Difficulties

with Basic Activities

Prevented Paid Work due to a Longstanding Health


Prevented Paid Work due to Longstanding Difficulties


Propor-tion SE 95% CI






Austria 0.29 0.01 0.27–0.31 0.09 0.01 0.08–0.10 0.04 0.00 0.03–0.05 0.03 0.00 0.02–0.03 0.05 0.01 0.05–0.07 0.03 0.00 0.03–0.04

Belgium 0.26 0.01 0.25–0.27 0.07 0.01 0.06–0.07 0.04 0.00 0.04–0.05 0.03 0.00 0.03–0.03 0.07 0.00 0.06–0.08 0.04 0.00 0.03–0.04

Bulgaria 0.40 0.74 Not provided 0.06 0.29 Not provided 0.07 0.30 Not provided 0.05 0.28 Not provided

0.06 0.27 Not provided 0.02 0.17 Not provided

Cyprus 0.35 0.01 0.33–0.36 0.05 0.00 0.04–0.06 0.04 0.00 0.03–0.04 0.03 0.00 0.03–0.04 0.05 0.01 0.04–0.06 0.02 0.00 0.01–0.03

Czech Republic

0.26 0.01 0.25–0.27 0.05 0.00 0.04–0.05 0.05 0.00 0.04–0.05 0.02 0.00 0.02–0.03 0.05 0.00 0.05–0.06 0.01 0.00 0.01–0.01

Denmark 0.31 0.01 0.30–0.32 0.07 0.00 0.06–0.08 0.06 0.00 0.05–0.06 0.04 0.00 0.04–0.05 0.09 0.00 0.08–0.10 0.06 0.00 0.05–0.07

Estonia 0.43 0.01 0.41–0.44 0.09 0.00 0.08–0.10 0.05 0.00 0.05–0.06 0.02 0.00 0.01–0.02 0.05 0.00 0.04–0.05 0.01 0.00 0.01–0.01

Finland 0.70 0.01 0.69–0.72 0.06 0.00 0.05–0.07 0.03 0.00 0.03–0.04 0.03 0.00 0.03–0.04 0.02 0.00 0.02–0.03 0.02 0.00 0.01–0.02

France 0.32 0.01 0.31 –0.33 0.07 0.02 0.06–0.09 0.03 0.02 0.02–0.05 0.03 0.02 0.01–0.04 0.04 0.02 0.03–0.06 0.02 0.02 0a–0.03

Germany 0.41 0.01 0.40–0.43 0.12 0.00 0.11–0.13 0.06 0.00 0.05–0.07 0.04 0.00 0.04–0.05 0.08 0.00 0.07–0.08 0.03 0.00 0.03–0.03

Greece 0.34 0.01 0.33–0.35 0.07 0.00 0.06–0.07 0.06 0.00 0.06–0.07 0.02 0.00 0.02–0.03 0.03 0.00 0.03–0.04 0.01 0.00 0.01–0.01

Hungary 0.54 0.00 0.53–0.55 0.12 0.00 0.11–0.12 0.10 0.00 0.09–0.10 0.08 0.00 0.08–0.09 0.10 0.00 0.09–0.11 0.06 0.00 0.05–0.06

Iceland 0.27 0.01 0.25–0.29 0.06 0.01 0.05–0.06 0.02 0.00 0.01–0.02 0.01 0.00 0.01–0.01 0.06 0.00 0.05–0.07 0.01 0.00 0.01–0.02

Italy 0.22 0.01 0.21–0.23 0.05 0.00 0.04–0.05 0.04 0.00 0.04–0.05 0.02 0.00 0.02–0.02 0.02 0.00 0.02–0.03 0.01 0.00 0.01–0.01

Latvia 0.45 0.01 0.43–0.46 0.11 0.00 0.10–0.12 0.05 0.00 0.04–0.06 0.02 0.00 0.02–0.03 0.08 0.00 0.07–0.09 0.03 0.00 0.03–0.04

Lithuania 0.41 0.01 0.40-0.42 0.11 0.01 0.10-0.12 0.07 0.00 0.06-0.07 0.03 0.00 0.03-0.04 0.09 0.00 0.08-0.10 0.02 0.00 0.02-0.02

Luxembourg 0.27 0.01 0.25–0.29 0.07 0.01 0.06–0.09 0.04 0.00 0.03–0.05 0.02 0.00 0.01–0.03 0.05 0.00 0.04–0.06 0.02 0.00 0.02–0.03

Malta 0.51 0.01 0.49–0.53 0.05 0.00 0.04–0.06 0.03 0.00 0.03–0.04 0.02 0.00 0.01–0.02 0.01 0.00 0.01–0.01 0.00 NA NA

Netherlands 0.31 0.01 0.30–0.32 0.07 0.01 0.06–0.07(b) 0.04 0.00 0.04–0.05(b) 0.02 0.00 0.02–0.02 0.08 0.00 0.08–0.09 0.03 0.00 0.02–0.03

Norway (11) 0.33 Not provided 0.14 Not provided 0.04 Not provided 0.03 Not provided 0.11 Not provided 0.06 Not provided

(11) Estimates were updated based on frequency tables provided. Standard errors or Confidence Interval information was not available

Accuracy 1

1

5

EU Comparative Quality and Technical Report: European Health and Social Integration Survey 23

Country



Prevented Buildings Use due to a Longstanding Health














Poland 0.37 0.01 0.35–0.38 0.10 0.00 0.09–0.10 0.06 0.00 0.05–0.06 0.03 0.00 0.03–0.04 0.07 0.00 0.06–0.08 0.03 0.00 0.02–0.03

Portugal 0.33 0.01 0.32–0.34 0.05 0.00 0.05–0.06 0.05 0.00 0.04–0.05 0.02 0.00 0.02–0.02 0.05 0.00 0.05–0.06 0.01 0.00 0.01–0.02

Romania 0.24 0.01 0.23–0.25 0.05 0.00 0.04–0.05 0.08 0.00 0.08–0.09 0.05 0.00 0.04–0.05 0.05 0.00 0.05–0.06 0.02 0.00 0.02–0.02

Table 5.3: Proportions, Standard Errors and Confidence Intervals for Six Key Disability Measures by Country (continued)

Country



Prevented Buildings Access due to a Longstanding

Health Condition, Illness or Disease













Slovak Republic

0.43 0.01 0.42–0.44 0.10 0.01 0.08–0.12 0.03 0.00 0.03–0.04 0.01 0.00 0.01–0.02 0.03 0.00 0.03–0.04 0.01 0.00 0–0.01

Slovenia 0.34 0.01 0.32–0.35 0.07 0.00 0.06–0.08 0.07 0.00 0.06–0.08 0.04 0.00 0.04–0.05 0.04 0.00 0.03–0.05 0.02 0.00 0.02–0.03

Spain 0.39 0.01 0.36–0.42 0.05 0.04 0 (a)–0.13 0.05 0.04 0 (a)–0.13 0.04 0.05 0 (a)–0.13 0.06 0.04 0 (a)–0.14 0.04 0.05 0 (a)–0.14

Sweden 0.32 0.01 0.31–0.33 0.09 0.01 0.08–0.10 0.02 0.00 0.02–0.03 0.01 0.00 0.01–0.01 0.06 0.00 0.06–0.07 0.01 0.00 0.01–0.02

United Kingdom

0.63 0.01 0.62–0.65 0.39 0.01 0.37–0.40 0.04 0.00 0.03–0.04 0.01 0.00 0.01–0.02 0.05 0.00 0.04–0.05 0.01 0.00 0.01–0.01

(a) The lower bound of the confidence interval was set to 0 as the countries provided negative lower bound. (b) Mismatch between the reported standard error and confidence intervals.

5 Accuracy


5.3 Nonsampling Errors 5.3.1 Coverage Error Failure to include all members from the target population in the sampling frame yields coverage error. There are four major types of coverage problems—missing elements, foreign elements, duplication, and clustering. Missing sampling elements on the frame lead to undercoverage, which often requires a supplemental frame or some type of linking procedure. Sampling units that are not part of the target population (i.e., foreign elements), but are included on the frame, lead to overcoverage. The best way to avoid overcoverage is to clean the sampling frame before the sample selection occurs; however, this procedure is often costly. Duplication occurs when the same sampling unit is represented more than once on the sampling frame; thus, some units have a higher probability of getting selected relative to others. Duplication problems are usually resolved via elimination of duplicates or weighting each unit by the inverse of its number of listing and using selection probabilities proportionate to that number. Finally, clustering occurs when more than one sampling elements appear together under the same listing. Depending on the frequency and size of clustering, possible solutions are to include in the sample all elements within the cluster, select at random one element from the cluster, or relist the elements.

One way to evaluate the extent to which the selected sample is representative of the target population is to compare the unweighted respondent estimates to known estimates for the target population (usually, demographic characteristics). Table B-1 annex B presents such estimates for household composition, age, gender, education, region and urbanization by country. For most countries, the respondent pool is largely unbalanced on these characteristics. Not surprisingly, after post-stratification, the weighted estimates match or come very close to the population percentages (12). Several countries are excluded from Table B-1 - Denmark, (13) Finland, Poland and Slovakia provided population and respondent totals and we do not have information on the level of item missingness for these variables to be able to calculate percentages.

Certain data are missing for some countries either because the information was not included in the quality or technical reports provided, or it is not available in these countries. Additionally, no data on level of urbanization was included for Sweden and Netherlands.

5.3.2 Nonresponse Response rates are often used as a measure of survey quality, based on the fact that the probability sampling inference requires 100% response rate. When the latter cannot be achieved, the inferential paradigm does not pertain and often models of the impact of nonresponse on data quality are needed. Given the absence of strong correlation between response rates and bias (Groves and Peytcheva, 2008), response rates should be viewed as only one of many data quality indicators. Traditionally, face-to-face surveys yield the highest response rates, followed by telephone survey, mail and web. Figure 5.1 presents the response rates (14) by country and mode, as reported in the quality reports by each country.

One caveat in the calculation of the presented response rates is that they do not account for cases with unknown eligibility. Thus, in telephone surveys for example, ring-no-answer cases are likely included in the denominator (eligible units), even though it is unknown whether these are business lines, or otherwise nonworking numbers. This possibility might explain the drastically low response rates for telephone surveys observed in Figure 5.1. Countries taking part in the EHSIS were required to report sample size, number of eligibles, final disposition codes and unit response rate (the ratio of completed interviews to

(12) Most countries employed household composition, age, gender, education, region and urbanization in their final weights.

(13) The subcategories for each demographic variable were also not labeled; thus, rendering the data unusable.

(14) Calculated as ratio of completed interviews to the number of eligible elements.

Accuracy


5

eligible sample elements). In addition, contractors were asked to report AAPOR RR1 and RR3 response rates, but barely half reported one or the other. When we further examined the countries with very low response rate (under 15%), we noted that the reported RR1 and RR3 were the same as the unit response rate for Austria and Germany, but surprisingly, even lower (by about 1%) for Belgium, Luxemburg, Netherlands and Sweden. Slovakia, Estonia, Norway and the United Kingdom did not provide the AAPOR response rates.

Figure 5.1: Response Rates by Mode and Country

This suggests that in addition to possibly having house effects in the assignment of final disposition codes (even their definition), contractors were not consistent in the way response rates were calculated. Thus, a cross-country response rate comparison should be done with caution. We recommend future rounds of the survey to require standardize final disposition codes and outline a procedure for assigning them, along with detailed instructions how the response rates of interest should be calculated.

In most countries, the main reason for nonreponse was refusal to take part in the interview. Table 5.4 presents the distribution of nonresponse outcomes by country. In only 10 countries (BE, DK, EE, FI, LV, LT, MT, PT, RO and SE), the main reason for nonresponse was failure to contact the selected sampled person. The extent to which refusals and noncontacts are systematically different from those interviewed in the EHSIS on disability-related measures may introduce bias in the estimates reported by country. Most countries used post-stratification to adjust for nonresponse (see Adjustment Error section).

0

10

20

30

40

50

60

70

80

90

100

HUROES LV MTBG PT DK CZ IT EL CY SI PL FR LT IS FI EE LU SK DENO NL AT BE SE UK

Face to Face

Telephone

Country

5 Accuracy


Table 5.4: Percent of Nonrespondents in Each Country by Reason for Nonresponse

Country Percent Noncontacts

Percent Refusals

Percent Inability to be Interviewed

Percent Rejected Interviews due

to Interview Quality (a)

Percent Other

Nonresponse Total

AT 40.5 55.1 1.4 0.1 2.9 100

BE 61.2 36.5 1.6 0.7 0.0 100

BG 39.5 59.6 0.8 0.0 0.0 100

CY 36.9 57.5 0.4 0.1 5.2 100

CZ 8.1 83.1 1.3 0.0 7.6 100

DK 50.6 37.6 2.2 0.1 9.5 100

EE 83.6 16.0 0.0 0.0 0.4 100

FI 64.0 36.0 0.0 0.0 0.0 100

FR 36.8 41.4 9.7 12.2 0.0 100

DE 25.1 66.1 2.2 0.8 5.8 100

EL 0.0 70.7 4.8 0.0 24.5 100

HU 0.0 97.8 2.2 0.0 0.0 100

IS 45.6 47.1 5.7 1.7 0.0 100

IT 1.1 90.3 2.7 2.5 3.4 100

LV 57.0 40.8 1.7 0.0 0.5 100

LT 58.0 41.4 0.0 0.1 0.4 100

LU 19.7 63.1 5.7 1.1 10.3 100

MT 50.5 49.5 n/a 0.0 0.0 100

NL 41.9 54.5 0.9 2.7 0.0 100

NO 33.1 64.0 0.8 1.1 1.0 100

PL 35.1 64.9 0.0 0.1 0.0 100

PT 50.9 40.4 1.6 5.6 1.5 100

RO 76.0 24.0 0.0 0.0 0.0 100

SK 37.0 56.5 0.9 5.6 0.0 100

SI 28.4 71.6 0.0 0.0 0.0 100

ES 43.8 49.0 7.2 0.0 0.0 100

SE 53.5 28.4 1.0 17.1 0.0 100

UK 23.7 26.8 0.0 0.9 48.6 100

(a) The tender specifies that any questionnaire containing more than 50% item nonresponse must be rejected.

All countries were asked to provide an average rate for item missingness, minimum and maximum values. The reported averages ranged from 0% to 15%; minimum values ranged from 0% to 0.1%, and maximum values ranged from 0% to 50% across countries. The highest percent item missingness across countries was for household income, which is common in surveys due to the sensitivity of the information requested. Table 5.6 presents the average, minimum and maximum item nonresponse by country. France

Accuracy


5

did not provide any information on the percent item missingness in their data. Poland and Slovakia reported the highest average percent item nonresponse (8.4% and 8.9% respectively), but the reports do not provide further detail what is driving these rates.

Table 5.5 should be interpreted with caution—we discovered, that countries differed greatly in the way they reported item missingness in two aspects: 1) some countries reported item nonresponse rates only for the substantive variables, excluding all demographics, and 2) some countries excluded “Refused” and “Don’t Know” responses from the item missing count.

Table 5.5: Average, Minimum and Maximum Item Nonresponse Rates by Country Country Average Percent Minimum Percent Maximum Percent

AT 0.3 0.0 28.4

BE 0.0 0.0 0.0

BG 0.5 0.0 12.1

CY 0.1 0.1 15.0

CZ 0.6 (a) 0.0 50.3

DK 0.0 0.0 0.0

FI 0.1 0.0 16.0

DE 0.2 0.0 10.3

EE 0.6 0.0 17.6

EL 0.9 0.1 9.2

HU 0.2 0.0 22.7

IS 0.0 0.0 0.0

IT 0.4 (b) 0.0 41.2

LV 0.3 0.1 16.3

LT 0.4 0.0 8.8

LU 0.2 0.0 8.5

MT (c) 0.5 0.0 0.5

NL 0.0 0.0 0.0

NO 1.0 0.0 12.4

PL 8.4 0.0 54.0

PT 1.6 0.0 45.8

RO 3.1 0.0 35.1

SK 8.9 0.0 29.7

SI 0.2 0.0 30.2

ES 2.0 0.0 10.7

SE 0.0 0.0 0.0

UK 0.0 0.0 0.0

(a) Based on substantive questions only. (b) Based on substantive questions only.

(c) Reported “less than 1%” for average and maximum percent.

5 Accuracy


5.3.2.1 Imputation Only five countries employed imputation methods to correct for item nonresponse. Finland, Hungary and Poland used multiple imputation; Latvia and Portugal used single imputation—interval regression and hot deck, respectively. Denmark imputed for only one variable (JOBSTAT), as it was not collected in the survey and the register data they used were incomplete.

5.3.3 Measurement Error Measurement error can lead to bias in survey estimates. Quantifying measurement error can be difficult and, therefore, more emphasis is placed on attempting to control for measurement error through the survey implementation process.

Four sources of measurement error are usually identified:

1. The method of data collection can affect responses to survey questions depending on whether the survey is interviewer-administered in-person, over the telephone, or self-administered by the respondent without the presence of an interviewer.

2. The questionnaire itself may affect responses due to the subject matter or poorly designed questions and question wording, including the order in which the questions are asked.

3. The effect of the interviewers themselves can also lead to error in the estimates from a survey if the interviewer does not read questions correctly or confuses the respondent.

4. The respondents themselves may not provide accurate or adequate answers to all survey items if they misunderstand the questioning, may not provide a truthful response or may not respond at all.

All of these areas of potential measurement error as they relate to the EHSIS are discussed and described in this section.

5.3.3.1 Data Collection Mode The survey was designed for face-to-face administration, but in some countries the survey was administered using the telephone or via a web-based application. The decision as to which mode to use was based on cost and the appropriateness of a particular mode in a given country. For example, in some countries such as Denmark, internet coverage is greater than in some other countries and respondents are typically more comfortable with internet use. In addition, a suitable sampling frame is available for a web-based design. This differs from Germany where, in general, respondents are more used to surveys being carried out over the telephone.

Only two-fifths of the countries taking part (11 of 28) used computer assisted or paper and pencil face-to-face interviewing exclusively, while in 12 countries the survey was administered over the telephone. In Latvia both telephone and face-to-face modes of data collection were used. In Denmark respondents younger than 65 completed the survey themselves online while those respondents aged 65 or older were interviewed using the telephone because those in the older age group were more likely to respond by phone than via the web. Spain also used a mixed mode approach combining telephone and web-based interviews (see Table 5.6).

The use of different modes of data collection can lead to mode effects, where the same questions asked in different modes produce different results. Dillman 2007 considers these distinctions between modes and suggests that the way information is communicated to respondents can influence the thought process involved in interpreting questions and response options. Furthermore, category order effects, which can cause variations in the selection process of response categories, can occur as a result of these differences. The most common of these are primacy and recency effects. The primacy effect is the tendency to choose answer categories towards the beginning of a list of options. This effect is more common in self-

Accuracy


5

administered surveys or surveys where respondents are asked to read from a list shown on a card. In contrast, the recency effect is the tendency for categories towards the end of a list to be selected. This effect is more common in telephone-administered surveys where response options are read out by interviewers. Answer categories towards the end of a list tend to be selected more frequently because these are the ones respondents are more likely to remember, although sometimes not reflecting the respondents’ true circumstance or opinion (Dillman, 2007; Krosnick and Alwin, 1987). For telephone administration in EHSIS contractors should have individually prompted each response category (Yes/No) to ensure that all response options were noted by the respondent rather than simply reading out the list in its entirety. It is not entirely clear which method was followed for each of the contractors using telephone administration. However, over the telephone, both methods can cause respondent fatigue. As mentioned earlier, direction and testing and adaption for CATI mode should be included for any future rounds of the survey.

Countries conducting the survey using CATI reported that reading long lists over the telephone was cognitively burdensome for respondents making it harder for them to process the information received, and may have impacted on data quality. Contractors commented that the survey questions seemed repetitive to respondents and they became tired or lost interest.

Table 5.6: Mode of Data Collection by Country Country Mode of Data Collection

AT CATI

BE CATI

BG PAPI

CY PAPI

CZ CAPI

DK CAWI / CATI

EE CATI

FI CAPI

FR CATI

DE CATI

EL CAPI

HU CAPI / PAPI

IS CATI

IT CAPI

LV (15) CAPI/ CATI

LT CAPI

LU CATI

MT PAPI

NL CATI

NO CATI

PL PAPI

PT CAPI

RO PAPI

SK CATI

(15) PAPI was used as an alternative to the CAPI mode where interviewers experienced technical problems with the CAPI interview or were

concerned about using their laptops for safety reasons. The PAPI questionnaires were subsequently entered into the CAPI program.

5 Accuracy


5.3.3.2 The Questionnaire Translation A source questionnaire and interviewer instructions were provided in English . A guide to the variables used in the questionnaire is provided in annex 1 of this report.

Each contractor was required to translate the questionnaire into their target languages in order to carry out the survey in their respective country. Table 5.7 shows which languages were used during the survey administration for each country.

Table 5.7: Languages Used to Conduct the Survey by Country

Country Language(s) Country Language(s)

AT German LV Latvian; Russian

BE French; Dutch LT Lithuanian

BG Bulgarian LU French; Luxembourgish; German

CY Greek MT Maltese; English

CZ Czech NL Dutch

DK Danish NO Norwegian

EE Estonian; Russian PL Polish

FI Finish; Swedish PT Portuguese

FR French RO Romanian

DE German SK Slovak; Hungarian

EL Greek SI Slovenian

HU Hungarian ES Spanish; regional official languages (Catalan, Valenciano, Euskera, and Gallego); English

IS Icelandic SE Swedish

IT Italian UK English

A standard three-step translation protocol was prescribed for use, by Eurostat, which contractors were required to follow. According to this protocol, each contractor had to:

Step 1: Translate the source questionnaire into the target language(s). The translator should work in the health/social statistics field, understand the health/disability. concepts used and use the target language as mother tongue and English as a working language.

Step 2: After the initial translation, a checker with the same characteristics as the original translator checks the translation while making use of the interviewer instructions to inform their conceptual understanding.

Step 3: The checker’s views and those of the initial translator are then bought together in a final translation. If they do not agree, a third expert is solicited to make the final decisions.

Countries sharing the same language could organise a coordinating translation process in order to obtain the same questionnaire or a version of it taking into account the linguistic and cultural differences. This occurred in the case of Austria. The Austrian questionnaire was based on the German questionnaire whilst making adaptions to include the Austrian demographic standards to the EU core social variables. In Slovakia the questionnaire was also translated into Hungarian. The Hungarian contractor provided the

Accuracy


5

translated questionnaire and minor adaptions were made to suit the Slovakian context.

Each country was required to complete a translation report form detailing where problems in translation occurred and their proposed solution. The forms were vetted as the survey development progressed and, where necessary, the country coordinator contacted. Only Greece and Lithuania appear to have deviated from the translation protocol, using a back translation process from English to their national language and then back to English.

No deviation from the original questionnaire was allowed except those resulting from cultural differences. The question wording, meaning and order in which the questions were asked had to remain consistent cross-nationally. Additional questions or modules could not be added to the questionnaire since it was considered that this would alter the contextual mindset of the respondent, add to the length of the interview and increase respondent burden.

It is difficult to know whether or not any major deviations from the original source questionnaire exist if not explicitly reported, since national translations were not checked. In general, only minor changes were reported when it was necessary to ensure that the questions were conceptually equivalent. For example, the introduction to the employment section was cited by Greece and France in this regard. Finland reported that one of the answer options to the barriers to education question (EdPrv) was ambiguous when translated directly into Finnish and, therefore, additional instructions were added to the existing interviewer instructions.

“Difficulties in getting to learning facility.” Some have wondered if this meant physical accessing to a learning facility, or difficulties being admitted to a school. This was somewhat ambiguous in Finnish translation. The former was the correct, this was explained in the interviewer instructions; thus, no corrections to the questionnaire were needed. “[FINLAND]

However, a translation error was identified in the German questionnaire following the completion of fieldwork that did change the conceptual meaning of the last of the MEHM (16) questions (MEHMH3). Respondents were asked to assess the severity of their longstanding health problem in terms of how it impacted on their ability to carry out basic activities for more than 6 months by asking: ‘for at least the past 6 months’. The German translation asked only about the period ‘over the past 6 months’. This translation error would also have affected the Austrian data collection because the correction was not made in the Austrian questionnaire until 1 month prior to the end of fieldwork.

Pretesting All of the countries conducting the survey carried out some form of pretesting prior to main stage field work in the form of a pilot study and interviewer debrief. Pretesting was carried out to ensure that the data collection instrument was programmed correctly and that supporting systems were correctly operationalized. Length of interview and respondent engagement was also monitored.

All but five countries (Austria, Germany, Luxembourg, Norway and Slovakia) also carried out pre-field testing in the form of cognitive interviewing. Since the previous version of the EHSIS questionnaire had been extensively cognitively tested and piloted (see section 2.1) the questions themselves were fixed and their meaning described in the interviewer instructions. Therefore, the purpose of any cognitive testing at this stage was to test and support the questionnaire translation to ensure the best translation for the underlying concepts. Hence, literal translations may have needed adaptation if they did not reflect the meaning of the original questions.

(16) Mini-European Health Module

5 Accuracy


Changes Made to Questionnaire Question sequence. All of the survey contractors followed the sequence of the questionnaire except for the United Kingdom—see table C-1 in annex C. In the United Kingdom respondents were asked for their postcode in order to derive the information required for the ‘region of residence’ and ‘degree of urbanisation’ core social variables. Respondents were also asked household grid questions in order to obtain the information needed for the ‘household composition’ core variable. The UK contractor felt that collecting this information near the start of the interview affected response rates because respondents terminated the interview at that point. As a result, the postcode question and most of the household grid questions (apart from the question asking for household size) were moved closer to the end of the questionnaire, between the ‘country of birth of the mother’ question and the ‘net monthly income of household’ question.

Question modifications. Modifications made to the questionnaire and a summary of questions that proved problematic during field work are described in the contractors own words in table C-1 annex C.

In general, minor adaptions were made to the question wording to account for the mode of administration. As Austria pointed out, in a CATI survey where telephone numbers are generated, it is not possible to send out an advance letter; therefore, the interviewer introductions were adapted accordingly. Another example of the kind of adaptions that were made because of mode, relates to the show card instruction “Please use this card as a guide and choose all that apply,” which could be changed to “I am going to read out some reasons, please tell me those that apply to you.” Furthermore, in the barrier questions, if the survey is CATI, each response option should have been individually prompted in order to try and provide a common stimulus across modes and avoid potential mode effects such as primacy and recency effects see section 3.3.3.1. Where the response options were individually prompted some contractors chose not to read out the last category ‘No, nothing prevents me from .....’ since if the respondent had not answered ‘yes’ to any of the barriers during the individual prompt, including the ‘other category’, then it could be assumed that the option ‘No, nothing prevents me…’ applied.

In general, adaptions made were designed to minimize mode effect and maintain comparability cross-nationally. However, some of the adaptions may have caused variations. Denmark, for example, appeared to have added a filter question in the employment section asking ‘Do you want a paid job?’. It is unclear how this was administered and if respondents had the opportunity to hear all of the barrier reasons that might apply to them before answering. The Slovenian contractor added a category to the list of barriers provided to the question about access to private vehicles ‘I do not have anyone to take me’. In this regard, the Slovenia data may not be comparable to other countries.

With regard to the EU core social variables, the Spanish contractor indicated that they simplified the questioning relating to household composition, but it is not clear how the data would be affected by this modification. The Norwegian contractor did not collect Net Monthly Income according to the specifications of the EU Core Social variables, but instead collected Gross Annual Income. The income data from Norway will not be comparable with other countries. Italy reported a reluctance by respondents to report income at all and suggested a shorter list of income bands. Italy also reported that because it was no longer compulsory to report marital status on official documents such as identity cards, respondents were more reluctant to provide marital status information in surveys.

Interview duration. The EHSIS interview was expected to take around 40 minutes in total when administered by an interviewer in a face-to-face mode. Across all countries the average time taken to conduct the interview was just over 25 minutes. However, it is worth noting that some of the minimum duration times reported were extremely low and were qualified with explanations about technical issues in reporting the calculations. This may have bought down the average duration time.

Table 5.8 shows that Estonia and Iceland had the lowest averages (14 minutes and 15 minutes, respectively), while Cyprus and Lithuania had the highest averages (42 and 40 minutes, respectively). The longest interviews took place in Hungary where a maximum length of 3 hours and 48 minutes was reported, and Spain where a maximum length of 3 hours and 15 minutes was reported. Some further investigation as to why some interviews took so long would be useful to inform any future rounds of the survey.

Accuracy


5

The face-to-face surveys took just over 7 minutes longer to administer on average than the telephone and web surveys.

Table 5.8: Interview Duration by Mode of Administration

Country Mode Minimum Length (mins)

Maximum Length (mins)

Average Length (mins)

AT CATI 10

105 22

BE CATI 1 88 20

BG PAPI 15 100 34

CY PAPI 20 60 42

CZ CAPI 15 135 28.16

DK CATI CAWI

5 5.7

94.2 90.5

15.45 16.62

EE CATI 5 50 14.2

FI CAPI 4 145 17

FR CATI 13 164 29

DE CATI 10 145 29

EL CAPI 30 65 33

HU CAPI/PAPI 7 228 22

IS CATI 2 67 15

IT CAPI 14 90 19.5

LV PAPI/CAPI/ CATI

2 1

76 68

17.6 16.4

LT CAPI 29 60 40

LU CATI 15 96 32

MT PAPI 20 85 38

NL CAPI 0 73 23

NO CATI 8.33 79.10 20.4

PL PAPI 16 90 38

PT CAPI 9.41 169.4 23.59

RO PAPI 10 150 36.47

SK CATI 4.32 118 16.33

SI CAPI 10 150 22

ES CATI/ CAWI

12

Not available

195

Not available

32.14

Not available

SE CATI 0 76 20

UK CATI 8 129 25

5 Accuracy


5.3.3.3 The Interviewer The role of the interviewer is important in household surveys that necessitate face-to-face data collection or those that require interaction with a respondent in order to obtain their participation, such as phone surveys. Often acting as the sole representative of a survey organization, interviewers are tasked with gaining a participant’s trust, accurately recording their responses, and transmitting those data securely.

Interviewer effects have been well-studied and can impact the quality of the data collected. Several quality indicators were reported to ensure high performance during fieldwork. Quality assurances related to interviewer recruitment and training included the duration of training, level of interviewer experience, and type of training administered. Other quality indicators included interviewer education level and prior interviewing experience, the ratio of interviewers to supervisors, the ratio of respondents to interviewers, and the mean number of contact attempts made by interviewers before marking a case nonparticipation.

Interviewer Recruitment One requirement for this survey was that only interviewers who had finished lower secondary education (ISCED 2) and those with previous experience in population surveys should be used during fieldwork. A majority of contractors recruited interviewers with at least an education level of ISCED 3-4 or higher.

In addition to education, countries were asked to provide details on the age range and economic activity status of their interviewers. A majority of contractors used interviewers that primarily fell into two age groups: 15-29 (reported by 10 countries) and 45-59 (reported by 9). Six contractors reported that the highest percent of their interviewers were between the ages of 30 and 44. Interviewers recruited also tended to be primarily employed, reported by 15 countries. Five countries reported using a majority of students as their interviewers. The majority of contractors used experienced interviewers employed directly by the organization that fielded the survey and those with experience working on social surveys.

Interviewer Training Training interviewers and their supervisors is integral to ensuring that an instrument is administered correctly, high participation rates are achieved, and data are collected accurately. Effective training can minimize interviewer effects and should provide a background to the project and include training in the mode(s) of data collection. Good training should also allow time for practice interviewing and discussions on how to handle difficult cases (e.g., refusals).

Specific survey-related training sessions were required for both the interviewers and their supervising staff. The type of training varied; however, all contractors reported providing training to their interviewers prior to data collection. Finland conducted interviewer training via telephone, while Poland used e-training. Other contractors reported training sessions that incorporated role play and practice sessions. Some countries that used CAPI data collection also trained interviewers on computer-assisted data entry and other CAPI exercises. Other methods of interviewer training included monitoring the first live interviews and holding follow-up debriefings to discuss areas where more guidance was needed. Other contractors required their interviewers to perform test interviews before being permitted to start fieldwork.

It was recommended that a 15% subsample of interviews be back checked to ensure data quality. Other quality checks implemented on the EHSIS included monitoring a random sample of live and previously recorded interviews, cross-checks of all questionnaires submitted, daily performance and progress monitoring, including collecting refusal and response rates, call times, and number of attempts. Re-interviews were also conducted. Ongoing feedback was given and additional training was provided as needed.

Length of interviewer training varied from 2 hours (Belgium, Czech Republic, Iceland, Netherlands, and Sweden) to 3 days (Spain). Poland’s interviewers had access to online training materials for up to 2 weeks. The average amount of time spent on interviewer training was a little over half a day, or 5.6

Accuracy


5

hours (17)

Interviewer/Supervisor Ratio The ratio of interviewers to supervisors provides a good measure of interviewer performance as it may be indicative of the level of quality monitoring by each organization.

There is likely little difference between interviewer/supervisor ratios observed in face-to-face surveys (PAPI or CAPI); however, one would expect a higher ratio on telephone surveys where a single supervisor could monitor a large group of interviewers located in a single location. It is more difficult to supervise the same number of interviewers in a field setting and when daily travel is required. Therefore, we might expect to see a lower ratio of interviewers to supervisors in CAPI and PAPI-administered surveys.

The country that reported the highest interviewer/supervisor ratio was Italy with 139 interviewers to every 1 supervisor and collected data via CAPI. The lowest ratio of interviewers to supervisors was reported by Denmark, approximately 4:1. Denmark mainly utilized CATI, but also used a web mode for respondents older than 65. The average interviewer to supervisor ratio for telephone surveys was 22.6, and 27.9 (18) for face-to-face interviews. It is unknown why these data trend in the opposite direction as would be expected. We can only hypothesize that they are the result of varying national standards including differences in supervisor titles or the organizational structure of survey operations and possibly interviewer availability.

Respondent/Interviewer Ratio Mode of data collection is likely to affect the ratio of respondents to interviewers. One might expect CATI survey interviewers to be able to survey more participants than modes that require face-to-face data collection such as PAPI and CAPI. This trend is reflected in the average ratios reported. The average ratio for CATI interviews was 60 respondents to every interviewer; the average ratio for the PAPI and CAPI modes was 46 respondents for every interviewer.

Table 5.9 displays the respondent/interviewer ratio as the number of respondents to one interviewer by mode of administration.

(17) Poland provided e-training materials for up to 2 weeks and therefore are excluded from this mean analysis.

(18) Countries that collected data in multiple modes and only reported one ratio are not included in this analysis: Denmark, Latvia, and Spain.

5 Accuracy


Table 5.9: Respondent/Interviewer Ratio by Country and Mode of Administration

Country Mode Ratio

BG Face-to-Face 33.5

CY Face- to-Face 83

CZ Face-to-Face 17.44

FI Face-to-Face 71

EL Face-to-Face 68

HU Face-to-Face 20.4

IT Face-to-Face 43.5

LT Face-to-Face 67

LV Face-to-Face and telephone 117

MT Face-to-Face 54.5

PL Face-to-Face 31.3

PT Face-to-Face 67.3

RO Face-to-Face 12.77

SI Face-to-Face 27

AT Telephone 19

BE Telephone 72

DK Telephone 76.8

EE Telephone 70

FR Telephone 148

DE Telephone 32

IS Telephone 97

LU Telephone 19

NL Telephone 50

NO Telephone 46

SK Telephone 44

ES Telephone 365.5

SE Telephone 95

UK Telephone 31

Spain reported the highest respondent/interviewer ratio, where data were collected via CATI and CAWI, and reported 365.5 respondents to every interviewer. The lowest ratio was observed in Romania, who utilized PAPI data collection and reported 13 respondents to every interviewer.

The survey required that the respondent/interviewer ratio should not exceed 70, as EHSIS was designed for face-to-face data collection. In seven countries this limit was exceeded. Two countries (Cyprus and

Accuracy


5

Finland) administered the survey face-to-face; while the remaining five (Belgium, Denmark, France, Iceland and Sweden) administered the EHSIS via telephone.(19)

Contact attempts made before declaring nonparticipation. It is equally important to encourage selected respondents to take part in the survey. As such, the mean number of contact attempts before declaring nonparticipation was also reported. However, it should be noted that it is difficult to make direct comparisons cross-nationally as respondent behaviours and the ease with which contact can be made will vary by country and survey mode.

In general, countries administering the survey face-to-face reported lower mean number of contact attempts before declaring non-participation than did those via telephone. This is common due to higher costs of in-person attempts and their greater effectiveness of resulting in an interview. Table 5.10 displays the mean number of contact attempts before declaring non-participation by mode of administration.

Table 5.10: Mean Number of Contact Attempts by Country and Mode of Administration

Country Mode Mean Number of Contact Attempts

BG Face-to-Face 2.14

CY Face-to-Face 3

CZ Face-to-Face 1.4

FI Face-to-Face 1.8

IT Face-to-Face 1.4

LT Face-to-Face 1

MT Face-to-Face 3

PL Face-to-Face 1.68

PT Face-to-Face 3

RO Face-to-Face 3.5

SI Face-to-Face 3

BE Telephone 8

DK Telephone 6.9

EE Telephone 1.3

FR Telephone 18

NO Telephone 5.4

SK Telephone 4.94

ES Telephone 24.2

UK Telephone 6.1

The average number of contact attempts made in the face-to-face surveys was 2.3. The average number of contact attempts was not recorded for Greece, Hungary or Latvia. The average number of contact attempts made for the telephone surveys was 9. Six countries that administered EHSIS by telephone did not report the average number of contact attempts.(20) Denmark and Spain used a mixed-mode approach to data collection (phone and web) and are included in the analysis of mean contact attempts for telephone.

(19) Latvia reported 117 respondents to every interviewer, but only provided a single ratio for both modes of data collection.

(20) AT, DE, IS, LU, NL, SE.

5 Accuracy


Spain reported the highest mean number of contact attempts at 24.2. The lowest average number of contact attempts was reported by the Lithuanian contractor at 1. This survey was administered face-to-face and collected data for approximately 4 months.

5.3.3.4 The Respondent Partials Partial interviews are interviews where the whole survey is not completed due to break off or other reasons. The survey required that partials with less than 50% responses were not counted as interviews. Only one country, Italy, reported partials, but it is unclear if they were included in the total number of interviews, as was the case with Romania.

Proxy Interviews Proxy interviews typically induce measurement error in the survey estimates as proxies are asked to report on someone other than themselves, for whom they may not have the most accurate information. This is especially true for private behaviours, subjective perceptions, or sensitive topics. Most countries allowed for proxy interviews, but employed stringent rules as to when proxies should be allowed. Since many of the questions in the EHSIS interview are subjective and rely on respondent’s own assessment of their situation, proxy interviews were permissible only when the sampled person was severely impaired. Table 5.11 presents the percentage of proxy interviews out of all interviews by country, among the countries that allowed proxy responding. Four countries (Cyprus, Hungary, Poland and Spain) reported higher than 5% proxy interviews. It appears that this was because these countries allowed a broader definition of a proxy interview, for example allowing proxy interviews when the selected sample person was absent from the household for education purposes or absent from the household for ‘other’ reasons.

Table 5.11: Percent Proxy Interviews by Country

Country Percent Proxy Interviews

BE 0.5

BG 1.0

CY 7

CZ 1.4

FI 0

FR 2

EL 1.5

HU 14.3

IS 0.1

IT 5.0

LV 1.7

LT 1.0

MT 3.5

Accuracy


5

Table 5.11: Percent Proxy Interviews by Country (continued)

Country Percent Proxy Interviews

NL 0.6

NO 0.4

PL 6.3

PT 3.8

RO 4.0

SK 1.2

ES 11.1

SE 0.1

UK 0.2

5.3.4 Data Processing Error Another stage of the survey process where error can be introduced is the data processing stage. The data processing stage comprises data entry, coding and data cleaning, and weighting adjustments.

5.3.4.1 Data Entry, Coding and Cleaning Errors may be introduced at this stage when data are miss keyed by the interviewer or the data entry specialist or coding is not accurate and coder bias may be apparent, range and consistency checks are not in place, computations are incorrect and/or data are not checked and validated thoroughly. This section of the report describes the data entry, coding and cleaning process as it was carried out cross-nationally and attempts to identify any issues that could affect the final data outputs where they are described by the contractors.

It should be noted that the mode of data collection will affect the way in which the data processing is carried out. For countries using computer-assisted interviewing (CAI) methods, routing, range and consistency checks were integrated within the computer program and carried out automatically, to a greater or lesser extent, depending on whether the survey was interviewer-administered (CAPI/CATI) or self-administered (CAWI). This limited the amount of data processing and editing needed at the end of fieldwork. Derived variables were computed within the interviewing programme or following data collection. For example, the number of persons in the household could be computed from the household grid information; the region and degree of urbanization from the sample frame information. Coding of open-ended questions was carried out after data collection was completed.

“Interviews conducted using CAPI…so most of the validation rules were already implemented in the script used to collect the data. Open questions were coded by two people with experience and then 100% of the coding was revised.” [Portugal]

The data processing stage can also be used to monitor the outputs and improve data quality as fieldwork progresses. Some countries stated that they were able to minimize item nonresponse for example, by identifying where proportions were high and retraining or replacing interviewers reporting a high percentage of items missing.

As shown earlier in table 5.6, five countries used PAPI administration, namely: Bulgaria; Cyprus; Malta; Poland; Romania. When PAPI administration was used, a separate data editing and coding programme was required. These countries provided specific EHSIS training for their coders and editors. It should also be noted that quality checks on coding and editing were also carried out by the supervisory staff. This is important for reducing errors and also avoiding coder bias. The countries following this process checked back directly with respondents for clarification where errors were found.

5 Accuracy


“The data processing … specialist responsible for EHSIS project made additional cross-check over the entered data. Each appeared mistake was first checked in the hard copy questionnaire. If the problem was not due to the entry activity but due to interviewing, a phone call to the respective respondent was made to clarify the information.” [Bulgaria]

“Data was entered using a data entry program which was specifically designed for this survey. This program had a number of in-built validations which helped minimise biases during the data entry stage. Data checking/editing was conducted in three main phases: i) Physical checking of the questionnaire when delivered to the office; ii) Checking during data entry; iii) Data was then filtered during the data analysis stage using numerous validations and logical tests.” [Malta]

“During the data entry process a screen level validation process and control and editing of errors was done by the data entry operators. There was a range of values checked for each variable. About 20% of the entered questionnaires were checked randomly for any other data entry errors (possible errors that did not conflict with the value range checks implemented in the IT application).” [Romania]

All contractors were asked to report where errors were detected and what had been done to minimize these errors. In general this process did not appear too problematic, although some reported that it was time-consuming. The errors that were detected were mainly resolved. Table 5.12 presents comments made by contractors in this regard. These quotes were selected because they were considered important for reference in future rounds of this survey.

Accuracy


5

Table 5.12: Data Processing Errors/Issues Detected by Country and Mode of Administration

Country Mode Main Errors Detected / Issues

AT/DE/LU CATI ▪ One consistency check was missing which could have been installed in the course of programming, that was comparison between the number of persons in the household (HH-Anz) and the number of persons taken up in the household matrix question (HHMX, HHMXage). For some other generated variables referring to the number of household members, the household matrix data was taken as reference. Only if the household matrix data was incomplete, it was referred to the information out of the variables HH_Anz & HH_Alter.

FI CAPI ▪ There were 141 cases where the respondent either didn’t know the number of persons in the household, their employment status, or the answer was some other way inconsistent. These may have been also punching errors by the interviewer.

We were able to eliminate these inconsistencies; we didn’t have to eliminate answers from the database. We were able to determine from the questions the right answer.

EL CAPI ▪ MARSTDEFACTO: incorrect coding in some cases

▪ JOBISCO-HATLEVEL: inconsistencies between the two variables

▪ HHINCOME: unreasonable and unusual numbers

▪ HHTYPE: wrong coding of “other household”

All variables were checked for errors and whenever necessary call backs have been made.

HU CAPI / PAPI

▪ Income data entry was not correct in 175 cases.

▪ Refusal codes were not correctly entered in 56 cases.

MT PAPI ▪ The largest share of errors was found in the core variables module (mainly discrepancies between household type and household size / number of workers).

▪ In some cases, inconsistencies were noted between different modules of the questionnaire (e.g., general health good or very good; however suffers from many health problems / disabilities)

In these cases, the respondents were re-contacted in order to verify these inconsistencies

PL PAPI ▪ Punching errors—correction in database.

▪ Questions that should be omitted because of filters—answers deleted from database.

▪ We also discovered 77 cases when respondent didn’t know the age of household member. In 71 cases we were able to determine age range, in 6 questionnaires it was impossible and they were deleted from the database.

RO PAPI ▪ There were errors regarding the date of the interview (wrong year or month entered).

▪ Also the start time and finish time (the interview started at a later time than the when it ended-the start time and end time were reversed when entering the data). There were about 30 such cases.

▪ The relationships between the members of the household (regarding the household questionnaire) were in some cases (about 40-50 cases) incorrectly entered.

5 Accuracy


Table 5.12: Data Processing Errors/Issues Detected by Country and Mode of Administration (continued)

Country Mode Main Errors Detected / Issues

SK CATI ▪ Because of the programmed questionnaire there were minimum mistakes. Due to the missing control in the questionnaire (HHLABOUR_EMP). We had to code “-1” because some respondents did not mentioned (count) some family members. It could be influenced by the fact that they did not count students or mothers on maternity leave.

UK CATI ▪ For 8 interviews, the interview length was not recorded accurately and so interview lengths for them are not available.

▪ In 18 instances, invalid dates of birth were recorded for the respondent. In 26 cases this also happened for other household members. In such instances, the CATI script calculated age from the year of birth alone and this age has been used in the data.

▪ 17 respondents were not routed to the questions used to derive JOBISCO and LOCNACE.

▪ 1 respondent did not code 11 or 12 at the question used to derive MAINSTAT but was routed to, and gave responses at, the questions used to derive JOBSTAT, JOBISCO and LOCNACE.

▪ 1 respondent was coded as null at all of the sub-variables for LHCD and LDBA.

5.3.4.2 Weighting Adjustments Almost all countries created selection weights and post-stratification weights. Twenty countries (21) reported post-stratifying their data by various auxiliary variables (mostly demographic) to correct for coverage and nonresponse error. Table 5.13 describes briefly the weighting procedure employed by each country.

21

It was unclear from the reports of the rest whether post-stratification was employed or not.

Accuracy


5

Table 5.13: Weighting Procedure by Country

Country Weighting Procedure

AT Probability weight applied (Horvitz-Thompson estimator) to account for the dual-frame approach; calibration using generalized regression estimator input (starting weight for the calibration is the probability weight). Auxiliary variables: joined distribution of age group x, sex, highest level of education, employment status, degree of urbanization, household size, federal state.

BE For the landline sample, within household probabilities of selection were calculated (the selection probability weight for the mobile sample was set to 1). The overall sample was then weighted in order to reflect the national population by region, gender, age, and highest level of education.

BG A post-stratification weighting procedure by sex, age and size of residence place, considering the latest available data on Bulgarian population. (The probabilities of selection for each housing unit were unknown due to the random walk approach).

CY Two sets of weights —household level probability of selection (inverse of the number of eligible persons in the household) with a maximum size of 4, and post-stratification. The probabilities of selection for each housing unit were unknown due to the random walk approach. The post-stratification weighting was carried out using Raking Adjustment for Nonresponse (the selection probability weights were used as base). Weights were developed by making the marginal distributions of the auxiliary variables (Age by Sex combination and degree of urbanisation) in the sample conform to the population marginals.

CZ A respondent’s design weight was calculated as the inverse of the product of three inclusion probabilities—the inclusion probability of a specific municipality (stage 1); the inclusion probability of a specific household in that municipality (stage 2) and the inclusion probability of one adult/respondent in this household (stage 3). At the end, the design weight was rescaled by number of 15+ population, by strata, in order that the sum of the design weight in all 32 stratum could be equal to the total 15+ population number from the Czech Republic.

A post-stratification weighting was carried out, in order to adjust the sample to selected control

Totals. The raking procedure included age by NUTS2, activity by NUTS2, urbanization, and highest level of education. The final weights were trimmed (a maximum for the weights was set at 3.0).

DK Clan-tool. Following variables used: Ancestry, education and family type. Sex, age and income have been part of the weighting, each by itself.

EE Each respondent unit was assigned a sampling weight to make the respondent’s contribution larger or smaller to compensate for the fact that persons with certain characteristics are not as reachable and likely to respond to the study, and for the data to be representative of the Estonian population. The census data of Statistics Estonia on distribution of the universe according to the set of demographic characteristics was used for weighting: gender, age, settlement type and main language according to five regions (NUTS 3).

FI Inclusion probabilities where corrected in order to match marginal distributions of several demographic variables like age, gender, marital status, educational level. Correction was performed by means of quadratic programming with objective function proportional to squared sum of individual sampling weight change.

FR Data weighted by sex, age, region, HHNPERS, education and DEGUR using Rim weighting. Weights were trimmed at 3.

(continued)

5 Accuracy


Table 5.13: Weighting Procedure by Country (continued)


DE Weighting Step 1: Design weight (Probability weight), based on the Horvitz-Thompson estimator (inverse of the sampling probability) accounting for the dual-frame approach. Weighting Step 2: Calibration (GREG estimator) to account for selective Nonresponse. Auxiliary Variables: joined distribution of age group x sex, highest level of education, employment status, degree of urbanization, household size, Federal, state.

EL Two sets of weights —household level probability of selection (inverse of the number of eligible persons in the household) with a maximum size of 4, and post-stratification. The probabilities of selection for each housing unit were unknown due to the random walk approach. Post-stratification weighting used a combination of sex and age with 8 levels, region (NUTS 2 level) and degree of urbanization; the last two were also included in the sample stratification.

HU Design weight is the reciprocal of the inclusion probability for each selected individual. For respondents the design weights were multiplied with a nonresponse adjustment factor at locality by age group level, where the factor is the ratio of the number of selected individuals in scope and the number of respondents. Calibration was done at regional level. In each region we had 48 population totals the sample was to match with. Population by gender by age group, population by household size, population size group-of-locality by gender by broader age group.

IS Probability of selection weights were not calculated as the sampling frame was a population register at the individual level. The data were weighted by region, gender, age, and education.

IT A respondent’s design weight was calculated as the inverse of the product of three inclusion probabilities—the inclusion probability of a specific municipality (stage 1); the inclusion probability of a specific household in that municipality (stage 2) and the inclusion probability of one adult/respondent in this household (stage 3). At the end, the design weight was rescaled by number of 15+ population, by strata, in order that the sum of the design weight in all 48 strata could equal to the total 15+ population in Italy.

A post-stratification weighting was carried out, in order to adjust the sample to selected control totals.

The raking procedure included NUTS2, urbanization, age by gender, activity and highest level of education. The final weights were trimmed (a maximum for the weights was set at 3.0).

LV The design weights were computed as inverse of the sample inclusion probabilities. The design weights were adjusted by constant in each stratum to match the population size of each stratum. The adjusted weights were calibrated to the known population totals by the linear calibration using demographic and disability data (from population registers).

LT Post-stratification weighting was carried out using Raking Adjustment for Nonresponse (raking). Weights were developed by making the marginal distributions of the auxiliary variables (Age x Sex combination and degree of urbanisation) in the sample conform to the population marginals. Selection probability weights were incorporated by using them as starting weights in the procedure.

LU Calibration using generalized regression estimator (GREG). Input (starting weight) for the calibration is the design weight (probability weight). Auxiliary variables: joined distribution of age group x, sex, highest level of education, employment status, degree of urbanization, household size

MT Post-stratification weighting, basing on age, sex and district (LAU 1)

(continued)

Accuracy


5

Table 5.13: Weighting Procedure by Country (continued)


NL For the landline sample, within household probabilities of selection were calculated (the selection probability weight for the mobile sample was set to 1). The overall sample was then weighted in order to reflect the national population by region, gender, age, and highest level of education.

NO The final weights for the survey have been calculated as the product of nonresponse weights and design weights. The basis for the calculation of nonresponse weights is population data given by Statistics Norway in 2013. The population data splits the Norwegian population into subgroups based on gender (2 subgroups), geographic region (7 subgroups), age (6 subgroups) and education (2 subgroups). A rim weighing algorithm is applied to the survey data, giving the nonresponse weights.

PL Data weighted by marital status, gender, voivodship, urbanization density and education.

PT Two sets of weights were used —design and calibration weights. The design weights accounted for probabilities of selection at the PSU, housing unit and household level. The calibration weight post-stratified the data by age, gender, marital status, degree of urbanization, education, number of people in the household, working status.

RO Probability of selection weights, adjustment for nonresponse by county and urbanicity (response homogenous group method), and post-stratification by age, gender, area of residence and region.

SK Post-stratification using Iterative Proportional Fitting procedure to adjust weights in order to fit population marginal distributions of age, gender, education, region.

SI Survey weights were obtained with an automatic iterative weighting (“‘raking”)’ procedure, using population margin for sex, age groups, NUTS-3 regions, highest level of education completed and degree of urbanisation.

ES Calibration by age, sex, urbanicity, nationality, and household size.

SE For the landline sample, within household probabilities of selection were calculated (the selection probability weight for the mobile sample was set to 1). The overall sample was then weighted in order to reflect the national population by region, gender, age, and highest level of education.

UK The landline sample was weighted for noncontact, refusal and selection of the individual; the mobile sample was weighted by number of mobile phones. Final weights were a combination of mobile and landline weights, correcting for age, gender and telephone status.

6 Preferences


6. REFERENCES Agafitei, L. (2012). Recent developments on disability statistics in the European Union. Presented at the 10th meeting of the Washington Group on Disability Statistics http://www.cdc.gov/nchs/ppt/citygroup/meeting10/WG10_Session1_3_Agafitei.pdf

Biemer, P. Groves, R. M., Lyberg, L. E., Mathiowetz, N. A. and Sudman, S. (1991). Measurement errors in surveys. New York: Wiley & Sons.

Biemer, P. (2010). Total survey error: Design, implementation, and evaluation. Public Opinion Quarterly, 74 (5): 817-848.

Biemer, P. (2010). Total survey error. In M. Lovric (Ed.), International encyclopedia of statistical science. New York: Springer.

Biemer, P. and Lyberg, L. (2003). Introduction to survey quality. New York: Wiley & Sons.

De Leeuw, E. D., Hox, J. J., and Dillman D. A. (2008). International handbook of survey methodology. New Jersey: Lawrence Erlbaum Associates.

Dillman, D. (2007). Mail and internet surveys: The tailored design method. New York: Wiley & Sons.

European Statistics Code of Practice—revised edition 2011 http://epp.eurostat.ec.europa.eu/portal/page/portal/product_details/publication?p_product_code=KS-32-11-955

ESS Standard for Quality Report 2009 Edition Eurostat methodologies and working papers.

Eurostat: Directorate B: Quality, Methodology and Information Systems. Editing building block 2011 user’s manual. August 2012.

Eurostat Core variables updated 2011 guidelines. Doc. Eurostat/F/11/DSS/01/2.3EN

Eurostat Tender specification (2011/S 122-201426 and 2011/S 249-404645). http://epp.eurostat.ec.europa.eu/portal/page/portal/calls_for_tenders/archives

Ehling, M. and Körner, T. (eds) (2007). Handbook on data quality assessment methods and Tools. Prepared for Eurostat. http://epp.eurostat.ec.europa.eu/portal/page/portal/quality/documents/HANDBOOK%20ON%20DATA%20QUALITY%20ASSESSMENT%20METHODS%20AND%20TOOLS%20%20I.pdf

Groves, R. (1989). Survey errors and survey costs. New York: Wiley & Sons.

Groves, R. and Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse error: A meta-analysis. Public Opinion Quarterly, 72 (2): 167-189.

Groves, R. and Couper M. (1998). Nonresponse in household interview surveys. New York: Wiley & Sons.

Krosnick, J. A., and Alwin, D. F., (1987). An evaluation of a cognitive theory of response order effects in survey measurement. Public Opinion Quarterly, 51, 201-219.

Meltzer, H. (2008). Report on the development of a survey module on disability and social integration. https://circabc.europa.eu/w/browse/450b8367-d620-4398-844e-07bc5bdf8e6a

Meltzer, H. (2011). Opportunities for statistical indicator development relevant to measuring disability equality in Europe. PowerPoint presentation. ANED Conference, Brussels. http://www.disability-europe.net/content/aned/media/Powerpoint%20Meltzer_Howard_Presentation.pdf

Meltzer, H., Wilmot, A., and Demarest, S. (June 2010) Complementarity of definitions and methods of measuring disability across European surveys. https://circabc.europa.eu/w/browse/268c3432-83ad-4967-9a5e-ea3bb16f5710 (restricted access)

Preferences


6

Wilmot, A. and Meltzer, H. (September 2010) Report on the development of the European Disability and Social Integration Survey. https://circabc.europa.eu/w/browse/f420b9de-972b-42ca-b929-a585df7c7b43

World Health Organization (WHO). (2001) International classification of functioning, disability and health. http://www.disabilitaincifre.it/documenti/ICF_18.pdf

7 Annexes


7. ANNEXES Annex A Table A-1: Annex A: Questionnaire Guide

No. Question Description Purpose

Eurostat core social variables

1 Country of Residence Respondent’s country of usual residence (NUTS) Classificatory

2 Region of residence Region where the individual is living (place of usual residence).

Classificatory

3 Degree of urbanisation The type of locality individual is living in, namely whether an urban or a rural area (or a borderline case).

Classificatory

4 Sex Biological sex of person Classificatory

5 Age Age last birthday in completed years Classificatory

6 Legal marital status Legal status of individual in relation to the marriage laws of the country.

Classificatory

7 De facto marital status Marital status of individual in terms of his/her actual living arrangements within the household. Consensual union is defined as the union between non-married partners.

Classificatory

8 Household composition Size and composition of the private household derived from information on relationship between household members and the economic activity status.

Classificatory

9 Education level successfully completed

Highest education level successfully completed. Classificatory

10 Self-declared labour status

Normal or current ‘main’ labour status as perceived by the respondent.

Classificatory

11 Employment status Professional status of employed person (Self-employed/employee)

Classificatory

12 Occupation Occupation in main job Classificatory

13 Economic sector in employment

Economic activity of the local unit where the respondent is employed (incl. Self-employed).

Classificatory

Minimum European Health Module

14 MEHMHS1 General Health Descriptive

15 MEHMHS2 Longstanding illness or health problem Descriptive

16 MEHMHS3 Severity level of longstanding health problem Descriptive

Longstanding health conditions and difficulties with basic activities

17 LHCD Type of health condition or disease Descriptive

18 LDBA Difficulties with basic activities Descriptive

(continued)

Annexes


7

Table A-1: Annex A: Questionnaire Guide (continued)


Mobility

19 MobPrv Barriers to mobility: leaving home Key variable

20 MobPrv1 Lack of special aids or equipment a barrier to mobility

Key variable

21 MobPrv2 Lack of personal help or assistance a barrier to mobility

Key variable

Transport

22 UseVcl Continuous use of household motorised vehicle Filter

23 DrPass Go out in household vehicle as driver or passenger

Filter

24 VclOft Go out in household vehicle less than want to Filter

25 VclPrv Barriers to mobility: household vehicle. Key variable

26 VclPrv1 Lack of special aids or equipment a barrier to mobility

K Key variable

27 VclPrv2 Lack of personal help or assistance a barrier to mobility

Key variable

28 OthTrn Use of other forms of motorised transport Filter

29 OthTOft Use other forms of transport less than want to Filter

30 OthTPrv Barriers to mobility: other forms of transport Key variable

31 OthTPrv1 Lack of special aids or equipment a barrier to mobility

Key variable

32 OthTPrv2 Lack of personal help or assistance a barrier to mobility

Key variable

Accessibility to buildings

33 BldPrv Barriers to accessing buildings Key variable

34 BldPrv1 Lack of special aids or equipment a barrier to accessing buildings

Key variable

35 BldPrv2 Lack of personal help or assistance a barrier to accessing buildings

Key variable

Education and Training

36 Educ Currently studying for a qualification Filter

37 EdPrv Barriers to education and training Key variable

38 EdPrv1 Lack of special aids or equipment a barrier to education and training

Key variable

39 EdPrv2 Lack of personal help or assistance a barrier to education and training

Key variable

(continued)

7 Annexes




Employment

40 EmpPrv Barriers to employment Key variable

41 EmpPrv1 Lack of special aids or equipment a barrier to employment

Key variable

42 EmpPrv2 Lack of personal help or assistance a barrier to employment

Key variable

Internet use

43 IntUse Uses Internet Filter

44 IntFor Internet purpose Descriptive

45 IntMor Would like to use Internet (more) Filter

46 IntPrv Barriers to using Internet Key variable

47 IntPrv1 Lack of special aids or equipment a barrier to using Internet

Key variable

48 IntPrv2 Lack of personal help or assistance a barrier to using Internet

Key variable

Social contact

49 CloseNum Number of people feel close to Key variable

50 SpkOft Spoke with people feel close to as often as wanted to

Filter

51 SpkPrv Barriers to speaking with people feel close to Key variable

52 SpkPrv1 Lack of special aids or equipment a barrier to speaking with people feel close to

Key variable

53 SpkPrv2 Lack of personal help of assistance a barrier to speaking with people feel close to

Key variable

Leisure pursuits

54 HobPrv Barriers to pursuing hobbies or interests Key variable

55 HobPrv1 Lack of special aids or equipment a barrier to hobbies or interests

Key variable

56 HobPrv2 Lack of personal help of assistance a barrier to hobbies or interests

Key variable

57 EntPrv Barriers to attending cultural events Key variable

58 EntPrv1 Lack of special aids or equipment a barrier to attending cultural events

Key variable

59 EntPrv2 Lack of personal help of assistance a barrier to attending cultural events

Key variable

(continued)

Annexes


7



Economic life

60 EcPay Ability to pay for the essential things in life Filter

61 EcPrv Barriers to paying for the essential things in life

Key variable

Attitudes and behaviour

62 Discrim Reasons for feeling treated unfairly Key variable

63 DisWho Types of people who treat respondent unfairly Descriptive

Eurostat core social variables (cont.)

64 Country of Citizenship Country of citizenship at time of data collection Classificatory

65 Country of birth Country where respondent was born Classificatory

66 Income Net monthly income of household Classificatory

7 Annexes


Annex B Table B-1: Comparison of Population Percentages to Weighted and Unweighted Sample Estimates of Demographic Characteristics by Country

Country Characteristics Percent in the

Population

Unweighted Sample

Estimate (%)

Weighted Sample

Estimate (%)

AT Age group

15-19 7 3 7

20-24 7 3 7

25-29 8 3 8

30-34 8 4 8

35-39 8 5 8

40-44 9 8 9

45-49 10 10 10

50-54 9 11 9

55-59 7 10 7

60-64 7 10 7

65-69 6 11 6

70-74 6 11 6

75 years and older 9 10 9

Sex

Male 48 41 48

Female 52 59 52

Degree of urbanization

Densely populated area 30 30 28

Intermediate populated area 29 30 28

Thinly-populated area 41 34 39

Unknown/Missing 6 5

Household composition

1 person 14 24 14

2 persons 24 40 24

3 persons 21 14 21

4 persons 24 14 24

5 persons and more 18 9 18

Highest level of education completed

Primary and lower secondary education (ISCED 1-2)

25 16 25

Upper secondary and post-secondary nontertiary education (ISCED 3-4)

59 62 59

Tertiary education (ISCED 5-6) 16 21 16

(continued)

Annexes


7

Table B-1: Comparison of Population Percentages to Weighted and Unweighted Sample Estimates of Demographic Characteristics by Country (con’t)


Population

Unweighted Sample

Estimate (%)

Weighted Sample

Estimate (%)

Federal state

Burgenland 3 3 3

Carinthia 19 19 18

Lower Austria 21 20 19

Upper Austria 7 5 6

Salzburg 14 15 14

Styria 17 17 16

Tyrol 6 5 6

Vorarlberg 9 7 8

Vienna 4 4 4

Missing 7 5

BE Region (EQLS 18+)

Région de Bruxelles-Capitale / Brussels Hoofdstedelijk Gewest

10 14 10

Prov. Antwerpen 16 12 16

Prov. Limburg (BE) 8 12 8

Prov. Oost-Vlaanderen 13 10 13

Prov. Vlaams-Brabant 10 10 10

Prov. West-Vlaanderen 11 9 11

Prov. Brabant Wallon 4 4 4

Prov. Hainaut 12 12 12

Prov. Liège 10 10 10

Prov. Luxembourg (BE) 2 3 2

Prov. Namur 4 4 4

Gender

Male 47 46 47

Female 53 54 53

Age

15-17 4 4 4

18-29 18 19 18

30-39 16 15 16

40-49 18 18 18

50-59 16 16 16

60-69 12 13 12

70+ 15 15 15

Level of Education

ISCED 0-2 34 27 35

(continued)

7 Annexes




Population

Unweighted Sample

Estimate (%)

Weighted Sample

Estimate (%)

ISCED 3-4 36 35 36

ISCED 5-6 30 38 30

BG Age

15-24 14 10 14

25-59 57 53 57

60-74 20 26 20

75+ 9 12 9

Sex

Male 48 48 48

Female 52 52 52

Region of residence

North-West 12 12 12

North-Central 12 11 11

North-East 13 13 13

South-East 14 15 15

South-West 29 28 29

South-Central 20 21 20

Degree of urbanisation

Densely-populated area 43 43 44



Household type

One-person households n/a 22 21

Lone parent with child(ren) aged less than 25

n/a 3 4

Couple without child(ren) aged less than 25

n/a 27 24

Couple with child(ren) aged less than 25

n/a 22 25

Couple or lone parent with child(ren) aged less than 25 and

n/a 8 9

Other type of household n/a 18 18 Highest level of education

completed

No formal education or below ISCED 1

2 1 1

Primary education 5 4 4

Lower secondary education 24 20 20

Upper secondary education 47 54 55

Tertiary education 21 20 20

(continued)

Annexes


7



Population

Unweighted Sample

Estimate (%)

Weighted Sample

Estimate (%)

CY Age

15-24 18 24 30

25-34 25 18 15

35-44 21 16 14

45-54 20 19 20

55-64 16 23 20 Sex

Male 47 41 43

Female 53 59 57



Intermediate urbanised area 19 23 23


Household Composition

Single adult without children 8 15 6

More than one persons 92 85 94

Education level (15-74)

ED0-2 30 33 31

ED3_4 37 39 41

ED5_6 33 29 27

CZ Sex, GP15+

Male 49 46 49

Female 51 54 51

Age Categories, GP15+

15-24 14 12 15

25-34 17 16 14

35-44 18 19 20

45-54 15 14 18

55-64 17 16 15

65+ 19 23 19

Region, GP15+

CZ01 12 12 12

CZ02 12 12 12

CZ03 12 11 12

CZ04 11 11 11

CZ05 14 14 14

CZ06 16 16 16

CZ07 12 12 12

(continued)

7 Annexes




Population

Unweighted Sample

Estimate (%)

Weighted Sample

Estimate (%)

CZ08 12 12 12

Degree of urbanization, GP15+ (preliminary census data)




Nr of persons in private households, GP total

1 person 14 23 13

More than 1 person 86 77 87 Education, GP15+

Low 54 46 54

Medium 32 40 32

High 15 14 15

EE Age

15-24 15 13 15

25-39 26 26 26

40-59 32 32 32

60-74 18 16 18

75 and older 10 10 10

Gender

Female 55 57 55 Male 45 44 45

Region NUTS 1 = Estonia, 100% Degree of urbanization





One-person household 36 21 25

Multi-person household 64 79 75 Highest level of educationa

completed

ISEC 1—primary 18 3 2

ISEC 2—lower secondary 10 10

ISEC 3—upper secondary 51 32 34

ISEC 4—post secondary 25 25

ISEC 5—tertiary, first stage 31 30 29

ISEC 6—tertiary, second stage 1 1

(continued)

Annexes


7



Population

Unweighted Sample

Estimate (%)

Weighted Sample

Estimate (%)

FR Sex

Male 48 43 48

Female 52 57 52

Age

15-24 15 11 15

25-34 16 14 14

35-44 18 17 18

45-54 17 16 18

55-64 15 19 16

65+ 20 23 20

Region

FR1 19 15 18

FR2 17 17 17

FR3 6 6 6

FR4 9 9 9

FR5 14 15 13

FR6 11 12 11

FR7 12 13 12

FR8 13 11 13

Degree of Urbanisation




Number of persons in the household

Only 1 person in the household 18 28 18

More than one person 82 72 82

Education

ISCED 0-2 37 26 37

ISCED 3-4 40 41 40

ISCED 5-6 23 32 23

Refusal/Don’t Know 0.6 0.6

DE Age

15-19 6 5 6

20-24 7 5 7

25-29 7 6 7

30-34 7 6 7

35-39 7 7 7

(continued)

7 Annexes




Population

Unweighted Sample

Estimate (%)

Weighted Sample

Estimate (%) 40-44 9 9 9

45-49 10 11 10

50-54 9 11 9

55-59 8 10 8

60-64 7 9 7

65-69 6 8 6

70-74 7 7 7

75 years and older 11 8 11

Sex

Male 49 45 49

Female 51 55 51

Federal state

Schleswig-Holstein (Nuts-1: DEF)

4 3 3

Hamburg (Nuts-1: DE6) 2 2 2

Niedersachsen (Nuts-1: DE9) 10 9 9

Bremen (Nuts-1: DE5) 1 1 1

Nordrhein-Westfalen (Nuts-1: DEA)

22 18 21

Hessen (Nuts-1: DE7) 7 7 7

Rheinland-Pfalz (Nuts-1: DEB) 5 4 5

Baden-Württemberg (Nuts-1: DE1)

13 11 13

Bayern (Nuts-1: DE2) 15 16 15

Saarland (Nuts-1: DEC) 1 1 1

Berlin (Nuts-1: DE3) 4 5 4

Brandenburg (Nuts-1: DE4) 3 4 3

Federal state (continued)

Mecklenburg-Vorpommern (Nuts-1: DE8)

2 3 2

Sachsen (Nuts-1: DED) 5 6 5

Sachsen-Anhalt (Nuts-1: DEE) 3 3 3

Thüringen (Nuts-1: DEG) 3 3 3

unknown/Missing 3 4





Unknown/Missing 4 5

HH composition

1 person 22 26 22

(continued)

Annexes


7



Population

Unweighted Sample

Estimate (%)

Weighted Sample

Estimate (%)

2 persons 38 37 38

3 persons 18 17 18

4 persons 16 14 16

5 persons and more 7 6 7


Hauptschulabschluss, POS 8 44 20 44

Realschulabschluss, POS 10 22 35 22

Fachhochschulreife, Hochschulreife

27 44 27

kein Schulabschluss, noch in Ausbildung

8 2 8

EL Age

15-24 15 18 18

25-34 21 19 19

35-44 24 24 23

45-54 22 22 22

55-64 19 18 18

Sex

Male 49 42 43

Female 51 58 57

Degree of Urbanization


Intermediate urbanised area 28 26 26




More than one person 87 79 91

Education Level

ED0-2 41 36 42

ED3-4 38 49 44

ED5-6 21 16 14

Region

1 32 32 32

2 22 22 22

3 36 35 36

4 10 10 10

HU Age

15-24 14 16 14

25-34 17 17 17

(continued)

7 Annexes




Population

Unweighted Sample

Estimate (%)

Weighted Sample

Estimate (%) 35-44 17 18 17

45-54 15 15 15

55-64 16 15 16

65-74 11 11 11

75+ 9 8 9

Sex

Male 47 47 47

Female 53 53 53

NUTS1

HU1 30 28 30

HU2 30 30 30

HU3 39 43 39

Size of household

1 11 15 11

2 28 28 28

3 23 23 23

4-x 39 34 39

IS Region (Eurostat 15+) Höfudborgarsvædi 64 65 64

Landsbyggd 36 35 36

Gender

Male 49 49 49

Female 51 51 51

Age

15-17 6 6 6

18-29 22 22 22

30-39 18 17 18

40-49 17 17 17

50-59 16 16 16

60-69 11 11 11

70+ 11 11 11

Level of Education

ISCED 0-2 37 29 37

ISCED 3-4 35 37 35

ISCED 5-6 27 34 27

(continued)

Annexes


7



Population

Unweighted Sample

Estimate (%)

Weighted Sample

Estimate (%)

IT Age Categories, GP15+

15-24 12 11 12

25-34 14 14 13

35-44 18 19 19

45-54 17 15 17 55-64 15 13 14

65+ 24 28 24

Sex, GP15+

Male 48 45 48

Female 52 55 52

Region, GP15+

ITC1 Piemonte 7 7 7

ITC2 Valle d’Aosta 0 0 0

ITC3 Liguria 3 3 3

ITC4 Lombardia 16 17 16

ITF1 Abruzzo 2 2 2 Region, GP15+ (continued)

ITF2 Molise 1 1 1

ITF3 Campania 9 10 10

ITF4 Puglia 7 7 7

ITF5 Basilicata 1 1 1

ITF6 Calabria 3 3 3

ITG1 Sicilia 8 8 8

ITG2 Sardegna 3 3 3

ITH1 Provincia Autonoma di Bolzano

1 1 1

ITH2 Provincia Autonoma di Trento

1 1 1

ITH3 Veneto 8 8 8

ITH4 Friuli-Venezia Giulia 2 2 2

ITH5 Emilia-Romagna 7 7 7 ITI1 Toscana 6 6 6

ITI2 Umbria 2 1 2

ITI3 Marche 3 3 3

ITI4 Lazio 9 9 9





(continued)

7 Annexes




Population

Unweighted Sample

Estimate (%)

Weighted Sample

Estimate (%)

Nr of persons by size of household

1 person 10 28 22

More than 1 person 90 72 78

Education, GP15+

Low 53 51 53

Middle 35 38 35

High 12 11 12

LV Age

15-19 6 13 6

20-24 8 26 8

25-29 9 4 9

30-34 8 4 8

35-39 8 5 8

40-44 8 6 8

45-49 8 7 8

50-54 9 9 9

55-59 8 9 7

60-64 7 7 7

65-69 6 2 6

70-74 6 3 6

75+ 10 5 10

Sex

Male 45 44 44

Female 55 56 56

Region (NUTS 1 level)

Latvia 100 100 100


DEGURBA 1 n/a 36 42

DEGURBA 2 n/a 24 20

DEGURBA 3 n/a 40 38


1 person 18 14 15



Low 23 14 22

Medium 53 54 55

High 23 32 23

(continued)

Annexes


7



Population

Unweighted Sample

Estimate (%)

Weighted Sample

Estimate (%)

LT Age

15-24 21 19 21

25-34 19 20 19

35-44 20 19 20

45-54 23 27 27 55-64 18 15 13

Sex

Male 45 45 48

Female 55 55 52



Intermediate urbanized area 10 11 10




More than one persons 82 80 90 Level of Education

ED0-2 18 16 20

ED3_4 55 58 56

ED5_6 27 25 24

LU Age

15-19 7 5 7

20-24 7 3 7

25-29 9 5 9

30-34 9 7 9

35-39 9 9 9

40-44 10 12 10

45-49 10 11 10 50-54 9 11 9

55-59 7 10 7

60-64 6 8 6

65-69 6 7 6

70-74 4 5 4

75 and older 6 6 6

Sex

Male 50 43 50

Female 50 57 50

(continued)

7 Annexes




Population

Unweighted Sample

Estimate (%)

Weighted Sample

Estimate (%)





Unknown/Missing 8 8


1 Person 12 24 12

2 Persons 23 28 23

3 Persons 20 18 20

4 Persons 26 19 26

5 Persons and more 19 11 19


Primary and lower secondary education (ISCED 1-2)

30 24 30

Upper secondary and postsecondary nontertiary education (ISCED 3-4)

41 41 40

Tertiary education (ISCED 5-6) 29 35 29

Unknown/Missing 1 1

MT Age

15-19 7 7 7

20-29 17 16 17

30-39 17 14 17

40-49 15 15 15

50-59 17 17 17

60-69 16 19 16

70+ 12 13 12

Sex

Male 50 49 50

Female 50 51 50


2 100 100 100

3 0 0 0

Type of household

1 7 8 7

2 3 3 3

3 18 20 18

4 38 36 38

5 2 2 2

6 31 31 31

-1 0 0 0

(continued)

Annexes


7



Population

Unweighted Sample

Estimate (%)

Weighted Sample

Estimate (%)

Education

0 2 3 2

1 20 22 20

2 36 36 36

3 19 18 19

4 9 9 9

5 11 11 11

6 1 1 1

-1 0 0 0

NL Region (EQLS 18+)

Groningen 3 4 3

Friesland (NL) 4 3 4

Drenthe 5 3 5

Overijssel 7 6 7

Gelderland 2 2 2

Flevoland 12 13 12

Utrecht 7 8 7

Noord-Holland 11 12 11

Zuid-Holland 14 15 14

Zeeland 2 2 2

Noord-Brabant 15 14 15

Limburg (NL) 7 7 7

A’dam, R’dam en Den Haag 11 12 11

Gender

Male 49 47 49

Female 51 53 51

Age

15-17 4 2 4

18-29 18 16 18

30-39 16 17 16

40-49 19 20 19

50-59 17 18 17

60-69 14 14 14

70+ 12 13 13

Level of Education

ISCED 0-2 33 26 33

ISCED 3-4 39 29 39

ISCEd 5-6 28 45 28

(continued)

7 Annexes




Population

Unweighted Sample

Estimate (%)

Weighted Sample

Estimate (%)

Household Size

1 person 36 28 34

2 people 33 33 33

3 people 12 13 13

4 or more people 19 25 20

NO Sex

Male 50 50 50

Female 50 51 50

Age

15-24 16 13 16

25-34 16 9 16

35-44 18 16 18

45-54 17 22 17

55-64 15 20 15

65+ 19 20 19

Household Size

1 person 18 21 19

2 or more 82 79 81

Education

Lower 28 14 17

Medium 41 33 45

Higher 31 53 38

Urbanization

Densely 27 27 27

Intermediate 34 34 33

Thinly populated 39 39 39

PT Age

15-24 13 14 18

25-34 16 11 16

35-44 18 14 16

45-54 17 14 19

55-64 15 17 14

65+ 23 30 18

Sex

Male 47 43 49

Female 53 57 51

Region of residence

Norte 37 33 31

(continued)

Annexes


7



Population

Unweighted Sample

Estimate (%)

Weighted Sample

Estimate (%) Centro 23 27 19

Lisboa 28 25 36

Alentejo 7 10 8

Algarve 4 5 5





Number of person(s) in the household

1 person 8 19 3



Low 69 81 68

Medium 17 11 19

High 14 8 14

RO Age 15-24 15 7 11

25-34 19 9 12

35-44 20 14 17

45-54 14 18 24

55-64 15 20 18

Sex

Female 52 52 49

Male 48 48 51

Region (NUTS 1 level)

Macroregion RO1 24 25 24





Density populated area No Data 34 35

Intermediate area No Data 21 23

Thinly populated area No Data 45 42


1 person No Data 29 9

More than 1 person No Data 71 91

(continued)

7 Annexes




Population

Unweighted Sample

Estimate (%)

Weighted Sample

Estimate (%)


Low No Data 36 30

Medium No Data 53 58

High No Data 11 12

SI Age

15-24 No data 13 13

25-34 No data 15 17

35-44 No data 16 17

45-54 No data 17 18

55-64 No data 17 16

65-74 No data 13 11

75-84 No data 8 7

85+ No data 2 2

Sex

Male No data 46 49

Female No data 54 51

Statistical Regions (NUTS-3 Level)

Gorenjska No data 11 10

Goriska No data 7 6

Jugovzhodna Slovenija No data 6 7

Koroska No data 6 4

Notranjsko-kraska No data 2 3

Obalno-kraska No data 5 6

Osrednjeslovenska No data 20 26

Podravska No data 16 16

Pomurska No data 7 6

Savinjska No data 14 13

Spodnjeposavska No data 4 3

Zasavska No data 2 2


Densely populated area No data 12 19

Intermediate density area No data 33 31

Thinly-populated area No data 56 50


Below upper secondary education

No data 23 29

Upper secondary education No data 56 53

Tertiary education No data 21 18

(continued)

Annexes


7



Population

Unweighted Sample

Estimate (%)

Weighted Sample

Estimate (%)

ES Age

15-24 No Data 8 12

25-34 No Data 12 17

35-44 No Data 21 20

45-54 No Data 19 18

55-64 No Data 15 14

65+ No Data 25 20

Sex

Female No Data 54 51

Male No Data 46 49

Region

1 No Data 14 10

2 No Data 17 10

3 No Data 10 14

4 No Data 16 12

5 No Data 22 29

6 No Data 18 21

7 No Data 4 5


Densely No Data 48 49

Intermediate No Data 24 25

Thinly No Data 29 26


One person No Data 21 9

Lone parent with ch. No Data 5 5

Couple without ch. No Data 22 19

Couple with child No Data 31 37

Lone/coup + other No Data 5 9

Other No Data 17 21

Education

Low No Data 55 54

Medium No Data 19 20

High No Data 26 26

SE Region (EQLS 18+)

Stockholm 22 21 22

Ostra Mellansverige 17 17 17

Småland med öarna 9 9 9

Sydsverige 15 15 15

(continued)

7 Annexes




Population

Unweighted Sample

Estimate (%)

Weighted Sample

Estimate (%)

Västsverige 20 20 20

Norra Mellansverige 9 9 9

Mellersta Norrland 4 4 4

Övre Norrland 5 5 5

Gender

Male 48 48 48

Female 52 52 52

Age 15-17 5 5 5

18-29 19 19 19

30-39 16 16 16

40-49 17 17 17

50-59 15 15 15

60-69 15 15 15

70+ 15 15 15

Level of Education

ISCED 0-2 25 18 24

ISCED 3-4 50 41 50

ISCED 5-6 26 41 26

UK Age

15-24 16 6 16

25-34 16 10 16

34-44 17 15 17

45-54 17 19 17

55-64 15 20 14

Sex

Female 51 57 51

Male 49 43 49

Region (NUTS1 level) UKC 4 4 3

UKD 11 9 9

UKE 8 7 7

UKF 7 5 6

UKG 9 7 7

UKH 9 7 7

UKI 13 8 13

UKJ 14 11 10

(continued)

Annexes


7



Population

Unweighted Sample

Estimate (%)

Weighted Sample

Estimate (%)

UKK 9 7 6

Region (NUTS1 level) (continued)

UKL 5 4 4

UKM 9 10 8

UKN 3 3 3



Intermediate area 29 24 22

Thinly populated area 13 13 10


One-person household No Data 35 24

Lone parent with child(ren) aged less than 25

No Data 4 4

Couple without child(ren) aged less than 25

No Data 27 23

Couple with child(ren) aged less than 25

No Data 14 18

Couple or lone parent with child(ren) aged less than 25 and other persons living in household

No Data 2 3

Other type of household No Data 17 25

Refusal or Don’t Know No Data 2 3


No formal education or below ISCED 1

No Data 2 1

ISCED 1—Primary education No Data 1 1

ISCED 2—Lower secondary education

No Data 13 11

ISCED 3—Upper secondary education

No Data 33 36

ISCED 4—Postsecondary education but not in tertiary

No Data 2 2

ISCED 5—Tertiary education, first stage

No Data 43 43

ISCED 6—Tertiary education, second stage

No Data 2 2

Refusal or Don’t Know No Data 4 4 a Population values based on 15-74 year olds

7 Annexes

72

Annex C Table C-1: Question Modifications and Problems by Country

Name of Country Sequence Followed Yes/No

Questions Modified Yes/No

Modifications Made Problematic Questions

Austria Y N N/A None reported

Belgium Y N ▪ When the selected respondent was 15 to 17 years old, an additional question was included, asking for their parent’s permission for their participation in the research.

▪ None—there were no comprehension problems with the questionnaire.

Bulgaria Y Y ▪ Household type (sector Household composition) was modified in the Bulgarian version as in the predominant part of homes (houses or flats/apartments) live two or three generations together and the given in the questionnaire break down/split covers only part of family members combinations. At about 18% of all respondents are in “Other” answer.

▪ Classificatory variables could cause a problem with the quality of the received data as this part of the questionnaire was provided only as indicators. Thanks to the coordinator for Bulgaria, we learned for availability of the Core variables manual. Instructions in this manual helped us to prepare the proper questions and instructions for them.

▪ In some towns people were not inclined to give information about their date of birth. To identify the respondent, the household members were asked to tell at least the month of birth and the age in completed years.

Cyprus Y N N/A None reported

Czech Republic Y Y ▪ Age of children of less than one year was recorded as 1 (household roster)

▪ LHCD: “NONE” was added as a spontaneous response category (i.e., not read out by interviewers)

▪ Most respondents had issues understanding the concept of “basic activities”; translated into Czech, this concept does not realty make sense. It would have been better to read out the question without phrasing “basic activities,” and so just read out “for example, seeing, hearing, concentration and moving around.”

(continued)

7 Annexes

73EU Comparative Quality and Technical Report: European Health and Social Integration Survey

Table C-1: Question Modifications and Problems by Country (continued)

Name of Country Sequence

Followed Yes/No



Denmark Y Y ▪ Some questions were slightly modified due to the method used in collection of data (telephone and web instead of face-to-face interviews). We skipped the last sentence from the question LHCD (skipped=bold): May I just check, do you have any of the following longstanding health conditions, illnesses or diseases? Please use this card as a guide and indicate all that apply. As no interviews were conducted face-to-face, it made no sense to the respondent to read or hear this last sentence.

▪ The following questions from the original questionnaire were not suited for telephone interviews: EdPrv12, EmpPrv12, HobPrv11, EntPrv11. E.g., EmpPrv12: Don’t want paid work. This question does not work well in a telephone interview and therefore it was modified to a question: Do you want a paid job? The question was asked in the same order as in the original questionnaire. It should be noticed that the original filter was kept and both employed and unemployed were asked the question.

▪ We added an extra text before question LHCD: MEHMHS2: Do you have any longstanding illness or longstanding health problem? By longstanding I mean illnesses or health problems which have lasted, or are expected to last, for 6 months or more. Followed by: MEHMHS3: For at least the last 6 months, to what extent have you been limited because of a health problem, in activities people usually do? Would you say you have been. In the original questionnaire LHCD we now asked: May I just check, do you have any of the following longstanding health conditions, illnesses or diseases? With 18 areas for the respondent to be asked regardless that some respondents already had answered ‘no’ to any longstanding illness or health problems in question MEHMHS2. The text we added came before LHCD and goes: Even though you have answered that you do not have any longstanding illness or health problems we would like to ask you some thoroughly questions as some illnesses or health problems are often neglected.

See previous column

(continued)

7 Annexes






Estonia Y N N/A ▪ Some respondents found the question LHCD (asking about having different longstanding health conditions or diseases) to be too personal and irritating. Some resistance was also met in relation to the questions on household composition.

▪ Respondents found the questions on issues that they did not have problems with not concerning them; for example some respondents found the questions on mobility to be humorous in relation to their lives. In those cases it was possible to explain why there is a need to ask different kinds of questions from all respondents.

Finland Y N N/A ▪ EdPrv, option 7, “Difficulties in getting to learning facility.” Some have wondered if this meant physical accessing to a learning facility, or difficulties being admitted to a school. This was somewhat ambiguous in Finnish translation. The former was the correct, this was explained in the interviewer instructions, thus no corrections to the questionnaire were needed.

France Y N N/A ▪ To ensure concept understood correctly. Issues with IntoEmp being difficult to translate into French.

Germany Y N N/A None

(continued)

7 Annexes



Name of Country Sequence Followed

Yes/No



Greece Y Y ▪ In HATLEVEL we have used code 0 for those who have not completed a level of education.

▪ In JOBISCO we have insert the categories that we have found at internet for Greece(ISCO 08).

▪ We have improved the translation for EHISHA1 code4,6;

▪ EmpPrv; IntroSoc and CloseNum; IntroEc and EcPay; IntroDis.

▪ In HHINCOME the addition of a scale was deemed necessary and has been added.

▪ Overall there weren’t major problems with the questionnaire, since the usage of CAPI enabled us to avoid routing and other errors.

▪ Nevertheless one thing that attracted our attention was the percentage of “other” option in some questions (e.g., intfor,ecprv). Maybe in a future wave the “other” should be open-ended (better checking, greater knowledge).

Hungary Y N N/A None reported

Iceland Y N N/A None reported

Italy Y Y ▪ Age of children of less than 1 year old was recorded as 1 (household roster)

▪ LHCD: “NONE” was added as a spontaneous response category (i.e., not read out by interviewers).

▪ Household roster: marital status for all household members aged 15 and above; solution for future rounds: use a simplified household roster.

▪ Level of education: older respondents had difficulties “translating” their level of education into the pre-coded list of educational categories

▪ ISCO-code (profession) and NACE-code (sector of activity): respondents found it difficult to provide enough detailed information that would allow to select the correct classification code.

Latvia Y N N/A ▪ Mainly translation issues.

(continued)

7 Annexes






Lithuania Y N N/A ▪ EHISPC2 EHISHA3—Wider explanation of “lack of special aids or equipment” (in Lithuanian—specialių priemonių arba įrangos trūkumas) was needed.

▪ MobPrv—Question on “leaving your home.” Is it leaving your home to go to work or on vacation? Solution to the problem—Explanation of “Leaving” meaning

▪ EcPrv:The ambiguity of the word “essential things.” Wider explanation was needed.Solution to the problem—Explanation of “Essential things” meaning (in Lithuanian—”pagrindinės reikmės”).

Luxembourg Y N N/A None reported

Malta Y N N/A None reported

Netherlands Y N N/A None—there were no comprehension problems with the questionnaire.

Norway Y Y ▪ De facto marital status: In an earlier version of this question, we included an answer code saying ‘SPONTANEOUS ONLY Same sex couple (but not in a formal registered Civil Partnership).’ We have removed this though as we do not need it in order to provide you with the answer codes you have asked for in the transmission guidelines.

▪ HHLABOUR_EMP and HHLABOUR_NEMP are set as one question in our questionnaire (HHLABOUR_EMP)—this will be redefined to original variables in the data file.

▪ HATLEVEL is changed so that it is suitable for the Norwegian standards.

▪ HHINCOME is changed so that it is suitable for Norwegians from Net.

▪ Monthly Income to Gross Annual Income. The Gross Annual Income is included in our datafiles.

▪ All the xx_PRV questions were causing fatigue both for the respondent and the interviewer. It was quite a difficult task to get the respondents to not just answer what they remembered/ assumed, but get the respondent to listen to all the statements per PRV. The questions were not regarded as “sensitive,” so it was the magnitude of statements, not the content that was perceived as challenging.

(continued)

7 Annexes



Name of Country Sequence Followed

Yes/No



Poland Y N N/A EmpPrv, when asked retired or pensioners. There should be answer option “retired / pensioned” because the choice “don’t want paid job” although formally accurate can be misleading or perceived as insulting by that persons.

Portugal Y N N/A Income and Household composition caused some problems with respondents.

Romania Y Y ▪ The Question coded MAINSTAT (self-declared labor status) was modified: we removed the answer in compulsory military service as it no longer applies to Romania.

▪ The answers to the ISCO (Occupation in employment) and NACE (Economic sector in employment) questions were required in 4 digits codes and we later recoded them.

▪ Questions 43 and 44 (variable EmpPrv) from the adult questionnaire should have had separate responses and show cards too, and not a shared one (even though the response categories were identical).

▪ Some persons were apprehensive about answering the income section (especially people with higher incomes).

▪ Some questions in the sections referring to Education and Training and Employment were inadequate for older persons (retired persons) and should have not applied to them.

Slovak Republic Y N N/A ▪ There were small problems with wording in some questions but these problems were solved after pilot interviews were done: for example in the household Composition‟ part, the respondents had the tendency to quote the number of household members while excluding themselves. Therefore the question wording also included the phrase ”..members including yourself...‟. Slovak words: “vrátane vás” (which means “including you”).

(continued)

7 Annexes



Name of Country

Sequence Followed Yes/No



Slovenia Y Y ▪ One statement was added in the VCLPRV question: statement nr. 7 (in questionnaire in national language) “I do not have anyone to take me.”

▪ In several questions respondents had the possibility to report that “Other reasons” prevent them from different activities. In the Slovene version of questionnaire we asked respondents to specify the reasons (open-ended question) if they answered “Yes” at statements “Other reasons.”

Spain Y N N/A ▪ The elderly have trouble the following sections: employment; education. Instructions and materials were given to the interviewers during the training in order to solve this kind these of problems during data collection. They were also given instructions to encourage the interviewed to answer in case of refusal.

▪ It is not clear for some respondents the meaning of ‘special aids’

▪ The respondents reluctantly answer the following topics: Household income; Occupation in their job; The discrimination section. One idea to get information about sources of income and total amount of income for the next round is to use administrative sources. During the preparation of the questionnaire we modified the question household composition to simplify the data collection of members of the household.

Sweden Y N N/A ▪ None—there were no comprehension problems with the questionnaire.

(continued)





United Kingdom N Y ▪ Following the cognitive testing of the questionnaire, the following changes were made to the wording of questions in the main body of the questionnaire, as detailed in the Cognitive Interview Report and agreed with the European Health and Social Integration Survey (EHSIS) coordinator.

▪ The wording, sentence structure and grammar of some of the answer codes was tweaked so that they flowed more easily when being read out to respondents. This was necessary because, as the survey was conducted over the telephone rather than face-to-face, the answer codes were read to respondents by interviewers rather than presented on a show card for respondents to read themselves. It was ensured that the meaning of the questions was not changed. Some specific examples are as follows:

▪ LHCD

– ‘Problems with arms or hands (include arthritis or rheumatism)’ was changed to ‘Problems with your arms or hands (including arthritis or rheumatism)’.

▪ LDBA

– ‘Using hands and fingers, such as picking up small objects, for example, a button or pencil, or opening or closing containers or bottles’ was changed to ‘Using hands or fingers, such as picking up small objects like a button or a pencil, or opening or closing containers or bottles’.

▪ LDBA, EHISPC1, EHISHA1

– The words ‘Unable to do’ were removed from ‘Cannot do at all / Unable to do’.

▪ EHISPC1

– ‘Getting in and out of bed or chair’ was changed to ‘Getting in and out of your bed or a chair’.

▪ EmpPrv

– ‘Affects receipt of benefits or services’ was changed to ‘It would affect the benefits or services you receive’.

▪ Discrim

– Interviewers were asked not to read out the answer code ‘None of these’. This is because, if a respondent did not answer yes to any of the other answer codes at this question, the interviewer would know to code the respondent as ‘None of these’ without reading it out.

▪ As described above, some respondents found the questionnaire long and repetitive, which irritated them at times. Examples of questions that respondents found particularly repetitive are provided below:

– HobPrv, HobEquip and HobPhelp were perceived as very similar to EntPrv, EntEquip and EntPhelp. In a future survey, it might be worth including some sentences before EntPrv for interviewers to read out which explain what the next three questions are about and how they differ from the previous three questions.

– At EdPrv, EmpPrv, SpkPrv, HobPrv and EntPrv respondents are asked about difficulties accessing or using buildings, when this issue has already been covered in detail at BldPrv, BldEquip and BldPhelp.

– If respondents said ‘No’ at MEHMHS2 and ‘Not limited at all’ at MEHMHS3, they were sometimes irritated to be asked LHCD. A similar point was raised in our Cognitive Interview Report, following which it was understood that these questions had to be asked. However, in any future surveys, it might be worth thinking about different ways to ask these questions.

– The xxxEquip and xxxPhelp were seen as particularly repetitive.

– The long lists of answer codes at the xxPrv questions allowed respondents the opportunity to lose concentration and stop listening.

7 Annexes


European Health and Social Integration Survey...List of countries 6 EU Comparative Quality and Technical Report: European Health and Social Integration Survey List of countries AT

Documents