How to interpret results of public opinion surveys? How to respond? EU Accession Communications Network Ohrid – 21-22 September 2006 Leendert de Voogd.

How to interpret results of public opinion surveys? How to respond?EU Accession Communications Network

Ohrid – 21-22 September 2006

Leendert de Voogd

2

How to interpret results of public opinion surveys? How to respond?

CONTENT

Back to basics

How to interpret the results?

Some hints

Case studies

What strategy?

Back to basics

4

A survey, what for?It is not necessary to interview everyone or we simply cannot.

There could be many reasons for conducting a survey. Some examples:

To identify attitudes, needs, concerns and/or expectations about specific topics

To measure opinions (agreement or opposition) to certain ideas

To test specific messages and communication plans

A survey will never:

Replace a census

Replace an election

It is a picture at a given time, in a given context, of the diversity of

opinions

5

Types of research

Two types of research: quantitative and qualitative

Quantitative: the objective is to ‘quantify’ opinions related to a specific issue. This can be done for different targets (general public, specific target groups, opinion leaders…)

Qualitative: to explore in depth motivations, opinions, attitudes and behaviours of individuals facing a given problematic. This is can be done through focus groups or in-depth interviews.

Both types are/could be complementary

6

Data collection modes

PAPI, CAPI, CATI,…CAWI

CATI stands for Computer Assisted Telephone Interviewing

Face-to-face, telephone or Internet?

FACE TO FACE

(PAPI/CAPI)

TELEPHONE

(CATI)

INTERNET

(CAWI)

For long questionnaire

Material to show

For difficult questions

For fast results

Targets difficult to reach

For difficult areas to get in

Self administration

Specific targets

Cheaper

7

Representativeness of a sampleFirst step: definition of the universe to represent

Second step: definition of a sample

The first quality of a sample is to be representative of the surveyed universe. This means without BIAS.- Sampling methods mostly used: random or quota sampling

The law of large numbers: The law says that with a large enough sample many odd coincidences are likely to happen.

The weight of the universe to survey has almost no incidence on the size of the

sample

8

Margin of error

Survey results remain approximations, the accuracy of which, everything being equal, rests upon the sample size and upon the observed percentage. Depending on sample sizes, the real percentages vary within precise confidence limits*. The formula:

±2√(p*q)/n

For 1000 interviews

Observed % 10% or 90% 20% or 80% 30% or 70% 40% or 60% 50%

Confidence limits

± 1.9 points ± 2.5 points ± 2.7 points ± 3.0 points ± 3.1 points

For 10 000 interviews

Observed % 10% or 90% 20% or 80% 30% or 70% 40% or 60% 50%

Confidence limits

± 0.6 point ± 0.8 point ± 0.9 point ± 1 point ± 1 point

* at 95% level of confidence

9

Margin of error (cont’d)

Survey results

(n=1000)Hyp.: Whole population

Yes 55% 51.85 < X < 58.15

No 45% 41.85 < Y < 48.15

How to interpret results?

11

Interpretation of the resultsInterpretation of the results is the step following the tabulation of the responses.

Rely on own basic knowledge:

Theoretical knowledge in sociology, economy, psychology and political science

Marketing models

Analyse the influence of social factors:

Gender: could refer to own perceptions of role differences between men & women

Age: could indicate an evolution in time but could also reveal generation’s effect

Education: cultural indicator but also social status indicator

Profession: social status indicator but also economical indicator

Constructed variables: opinion leaders (e.g. people who talk politics on a regular basis at home and who try to convince other people)

12

Interpretation of the results (cont’d)While interpreting results we must keep extra attention to the following:

At an individual level:

Age or ageing effect : public opinion vary with age, whatever the generation considered is

Generation effect : public opinion is typical from one generation; the youngest, while ageing, do not have the same response pattern as the oldest

Period effect: public opinion evolve in time, whatever the age.

At a collective level:

Structural changes models: major social or economic evolutions, technical innovations…

Political or economical cycles

13

Interpretation of the results (cont’d)

Behaviours Social and

demographic variables

Attitudes

Some hints

15

Context of the study

In an information society, public opinion is reacting almost immediately to particular events. This has a clear impact on survey results.

Some examples:

Bombings of Sarajevo’s markets obliged us to conduct new fieldwork in some countries

Shifts in American public opinion after 9/11: support for Bush increased

Better image of NGO’s after Tsunami in South-East Asia

Level of information on the Euro after the changeover took place (+20 points between November 2001 and January 2002)

16

Basis of respondents

Before drawing any conclusion on one particular result, one must look at the basis of respondents (i.e. how many people answer the question?)

A general rule that could be followed:

A minimum of 50 respondents per variable considered seems necessary for a sample of 1000 interviews

A minimum of 100 respondents per region considered seems necessary to speak of some regional trends

A minimum of 1000 respondents per variable

considered seems necessary for a sample

of 25000 interviews (e.g. in the EU)

17

Going beyond the obvious

Cross tabulations could be very helpful but sometimes are not really necessary.

Vertical %

(DK not shown)

Total

EU 25

Has benefited from EU

membership

Has not benefited from EU

membership

EU membership: a good thing

56% 82% 24%

EU membership: a bad thing

13% 2% 32%

EU membership: neither a good nor a bad thing 28% 14% 42%

18

Going beyond the obvious (cont’d)

E.G.: Level of information with the support to further enlargement

Vertical %

Total

EU 25

Feel well informed about

enlargement

Feel badly informed about

enlargement

FOR further enlargement

45% 53% 41%

AGAINST further enlargement

42% 41% 43%

DK 13% 6% 16%

2 hyp.: either good information produces support to enlargement; or people who already support enlargement are more interested in this topic and tend to seek more information.

19

Going beyond the obvious (cont’d)

Be careful to redundancy. The same persons are present in other sub-groups.

Examples given:

- Managers have similar answers to those who finished their education after the age of 20. Of course… they are often the same persons.

- Less educated persons have similar patterns of responses than those aged 65+.

It does not bring added value

20

Significant differencesA difference between two percentages has to be significant if one wants to comment it.

21

Correlation does not imply causality

There is a positive correlation between two variables if they are moving in the same direction. This does not mean that one variable is the cause (or the consequence) of the other.

Example given:

- People have a good image of the EU and feel happy.

On the contrary :

- People acknowledge to have received better information on one specific issue and objectively know more things about the topic

Assumptions can still be made

22

ScalesCommentators tend to forget what is behind a sub-total. It is essential to have a closer look at the distribution of responses.

65% is not equal to 65%

20

45

28

25

0

10

20

30

40

50

A new enlargement of the EU

Totally agree Tend to agree Tend to disagree Totally disagree DK

45

20

2

28

5

0

10

20

30

40

50

A new enlargement of the EU

Totally agree Tend to agree Tend to disagree Totally disagree DK

65

30

5

0

20

40

60

A new enlargement of the EU

Total agree Total disagree DK

23

Multi-answers vs. single answer possible

The same written percentage can have a different weight depending on how many answers the respondent could give.

When analysing multi-answers questions, it is possible to process the data on the basis of responses and not respondents.

50

40

30

20

5

0

10

20

30

40

50

3 answers were possible

A B C D DK

50

30

105 5

0

10

20

30

40

50

Only 1 answer was possible

A B C D DK

Case studies

25

EU Constitution

Surveys prior to referenda in France and the Netherlands showed that:

Reasons for supporting the ‘NO’ were clearly more concrete than those mentioned by supporters of the ‘YES’.

Other public opinion indicators (like public opinion mood, support for government in place…) were not helping the ‘YES’

Historical indicators should have been taken into account:

- Citizens in the former Member States were not consulted for the various wavesof enlargement. Especially not for the last one that took place in 2004.

- Very close win of the ‘YES’ in France in 1992

Clear signs of fragility

26

Threats to peace in the world

One of our survey was clearly misinterpreted by the media.

Original question: ‘For each of the following countries, tell me if in your opinion, it represents or not a threat to peace in the world’

Interpretation made: ‘EU poll: Israel 'biggest threat' to world peace’

‘Over half of Europeans think Israel poses the "biggest threat to world peace," according to a controversial poll commissioned by the European Commission.’ Source: WorldNetDaily.com

YES NO DK

Israel 59% 37% 4%

Iran 53% 41% 6%

North Korea 53% 40% 7%

What strategy?

28

Changing communication plans

Surveys can identified new themes, new issues, new messages and new targets to work on.

Results showed recently that European citizens made a difference between information sources legitimate to talk about a certain topic and sources they are using to get informed.

Potential actions:

Mobilize these legitimate

sources and make them intervene in

traditional media.

Focus on some particular publics,

geographical areas.

Use new messages, simplify them

29

Going forward

EU Constitution:

Recent surveys about the future of Europe showed a significant support for a EU Constitution (even in France and the Netherlands).

Results also clearly indicate that amongst supporters of a Constitution, there is a need to renegotiate and adapt the content of the current draft to include more political and social aspects.

Further enlargements:

EU citizens call for more information and communication about benefits and challenges of the process (especially in economical and social terms)

National Governments and media have a crucial role to play

grazie mille!

thank you!vielen Dank!

tack!merci!

bedankt!muchas gracias!

kiitos!