Usability Test – UNESCO.org Full Report June 8 2006 Laetitia Giannettini[[email protected]]Usability Specialist Cyril Alegre [[email protected]]Usability.

Usability Test – UNESCO.org

Full Report June 8 2006

Laetitia Giannettini [[email protected]] Usability SpecialistCyril Alegre [[email protected]] Usability Specialist

19/04/23 Axance User Experience Design 5, rue Régnault 93500 Pantin France

2Table of Contents

1. Study Background and Objectives

2. Methodology

3. Executive Summary

4. Detail of Results

5. Main Recommendations

1. Study Background and Objectives


4Background of the Study and Objectives

The United Nations Educational, Scientific and Cultural Organization (UNESCO) is a specialized United Nations agency that promotes peace through Education, Social and Natural Science, Culture and Communication programmes across the world.

UNESCO’s Bureau of Public Information (BPI) is the service in charge, among other functions, of public relations and providing audiovisual media. Its also in charge of the administration and editorial coordination of the UNESCO web portal: www.unesco.org.

The web portal aggregates a great amount of distinct UNESCO websites with different contents and look and feel. BPI wishes to create a standard setting and a capacity building process to harmonize standardize these websites.

A framework has been developed to migrate all the websites’ pages into a new look and feel essentially based on usability and accessibility criteria and previous user tests. The home page and the Worldwide and Organisation sections have been migrated so far.

In addition, the UNESCO portal now provides content in 6 languages: Arabic, Chinese, Russian and Spanish (the four other official languages) were added to French and English (the Organization’s working languages languages).



BPI wished to assess the perception and understanding of the portal by the representatives of the UNESCO Executive Board

The study’s objectives were to assess:

• Perception and understanding of the home page and its main items (access, tools, navigation…)

• Perception and understanding of pages in the main sections (Themes, Worldwide, Organisation, Services)

• Opinions toward the online version of the UNESCO Courier

• Opinions toward the availability of multilingual contents

• Opinions toward the legitimacy of BPI in coordinating the portal

The applied methodology was user testing (see section 2. Methodology)



This actual user testing follows a previous one, which took place in February 2004

The main objectives of the first study were to assess the utility (usefulness) and usability of the portal by UNESCO’s target public.

18 users divided into 6 profiles (3 participants per profile) were consulted for the first study:

• Researchers/Scientists/Academics

• Media

• Teachers/Educators/Trainers

• Decision-makers/Civil Servants in Supervising Ministry/Governing bodies

• Development agents (NGOs, IGOs, UN)

• Students/Young people

2. Methodology



User testing is mostly a qualitative method based on observational and oral data

It assesses users’ experience through two major criteria:

• usability (usefulness) of the website: is the website easily and efficiently usable?

• utility of the website: does the website meets users’ needs and expectations (contents, tools, etc.)?


9Methodology > Usability Testing

Individual sessions in three steps:

• Step 1: Pre-test questionnaire

– User’s profile and their use of the UNESCO web portal

• Step 2: Website browsing

– perception and understanding of the home page

– perception and understanding of pages in the main sections

– opinions toward the online version of the UNESCO Courier

• Step 3: Post-test questionnaire

– General opinion of the website

– Performance and satisfaction rating (questionnaire for performance indicators)

Duration of Sessions : variable – between 10 and 45 minutes according to the participants availability


10Methodology > User Profiles

User profiles (cf. Excel spreadsheet for details) :

• 55 participants to sessions of the UNESCO’s Executive Board

• 22 female and 33 male

• From 39 Permanent Delegations (cf. list on next slide)

• Age 22 to 66

• 30 users saw the portal in English and 25 users saw the portal in French


11Methodology > User Profiles

Participants to this test are from the following Countries:

Algeria (2) Argentina Austria Bahrain (2) Bangladesh Canada Cambodia China

Czech Republic (2)

Egypt Ethiopia European Commission (2)

Greece Guatemala Holy See (2) India (2)

Indonesia Italy (2) Kazakhstan Kuwait Luxembourg (4)

Madagascar Netherlands Nigeria

Norway Palestine (2) Poland Portugal (3) Romania Russian Federation

Slovenia (2) Sri Lanka (2)

Switzerland Thailand (2) Uganda United Kingdom

Uruguay Yemen Zimbabwe

3. Executive Summary


13Strong Points

Plenty of information available – the portal is a useful working tool

Some information is available in several languages

The home page layout is clear

The work done in order to add some consistency across the portal (the Framework) was noticed and appreciated

The “Worldwide” section is found useful (the possibility to find information about members states, its programs)

The online UNESCO Courier was appreciated


14Weak Points

Guidance needs some improvement

The portal is rather expert-oriented – outsiders and new comers need some time to understand it

Some contents are outdated or missing

Users have difficulty finding certain information and documents and the search engines are not efficient enough

There is still a lack of consistency between the different sections – this hurts understanding and learnability

Users expect more multilingual content

4. Detail of Results


16Results Details – Severity ratings

The results are classified according to their severity. This classification should also be seen as an emergency rating – the higher the number the sooner rework should be done:

Graphical issue

Minor usability issue

Medium usability issue

Major usability issue

Good point

2

4

3

1


17Results Details – Table of contents

A. Pre-test questionnaire – Users’ profiles and web habits

B. Browsing observation – Perception and understanding of the home page

C. Browsing observation – Perception and understanding of the “Worldwide” section

D. Browsing observation – Perception and understanding of the UNESCO Courier

E. Browsing observation – Transversal issues

F. Post-test questionnaire – General opinions

G. Post-test questionnaire – Performance and satisfaction rating


18A. Pre-test questionnaire

The age spread was as shown in the graphic opposite

The average age was around 41

Half of the participants had spent more than 2 years working for UNESCO

The average seniority was around 3.5 years

Seniority

5+16%

2-537%

1-225%

0-122%

Age spread

20-2920%

30-3917%

40-4937%

50-5917%

60+9%



Most participants were web experts: 83% of them have been using for more than 5 years – on average for 10 years and for 16 hours per week nowadays

Spreads as shown in the graphic opposite

Internet expertise (years)

10+23%

0-10%

1-24% 2-5

13%

5-1060%

Hours spent using Internet every week

20+16%

5-1022%

10-2054%

0-58%



Almost all participants had already browsed UNESCO.org.

Most often, participants started using the portal as they started working at UNESCO. On average, they have been using it pretty often (on a daily basis for most respondents) for 4.5 years

Spread is as shown in the graphic opposite

UNESCO portal expertise (years)5+9%

0-115%

1-228%

2-548%



Mostly, the participants browse the portal to find information and documents. The most visited sections are:

• Themes (mentioned by 22 participants) to get information about UNESCO’s programs and activities

• Organization (22) to collect information and documents from the Executive Board, the General Conference (meetings, resolutions, speeches…)

• Services (20) to find some documents and publications (conventions, resolutions…) and information about legal instruments

• News (14) and Events (11) to check what’s happening (UNESCO’s activities, exhibitions, conferences…)

• Communities (7) to find contacts



The main strong points reported by the participants according to their own experience were:

• The great amount of information availableThe portal is seen as a useful source of information to keep up to date about UNESCO news, about “what’s currently going on at UNESCO”. Furthermore, the UNESCO portal is often seen and used as a regular working tool: some users, for example, use it to collect information in order to prepare Executive Board meetings

• The overall ease of use and easy access to the information

• The contents available in 6 languages

Strong points



The main weak points reported by the participants were:

• Overall complexity, especially for outsiders or newcomers – the portal was often said to take some understanding of UNESCO structure to be efficiently used

• Difficulty in finding certain information and documents: all the participants said they could find most of what they were looking for but many also reported some problems in finding some information

• Outdated contents: many users deplored the slowness – or lack, at worst – of updates

• Lack of multilingual contents

Weak points



Most participants were aware that BPI was in charge of the UNESCO website

Most of them also had some general knowledge of BPI activities:

“Press information, Public Relations” (Holy See)

“Publishing, Communications” (China)

“Public information about UNESCO’s activities” (Italy)

“Promotion of UNESCO’s image” (Sri Lanka)

Other main activities reported: website administration, content feed, updates and coordination











26B. Perception and understanding of the home page

Most users found the home page layout clear and user friendly

They understood the 3-column layout and the left and right menus

The sections’ organisation and the wordings were well understood

“It’s clear, one can find almost everything he needs” (China)

“It’s practical, I can find what I want” (Indonesia)

The home page is clear



The languages available on the home page were well perceived and the work done to provide multilingual contents highly appreciated

“They are very useful” (Italy)

The tools are well perceived – the search engine is the most attractive

“Practical, I find what I want” (Indonesia)

The home page is clear



The home page is seen as being kept up to date (with changing pictures, themes, news and events), which gives the website a dynamic look that was appreciated

The home page is regularly kept up to date



Users found the home page had too much information and links on it, especially in the right menu

“Too many things – I can’t get any information” (Holy See)

“It’s compact, there are too many things on it, which distracts one’s attention – even more for someone browsing it for the first time” (India)

“The right menu offers too many things – nothing shows up” (Switzerland)

High informational density3



A few users did not understand the wording “Communities” nor anticipate the content of this section

The right menu is too heavy for non-UNESCO users – it takes some good knowledge of the UNESCO organisation to understand the menu

The high density in the right menu restrains the visibility of the links of interest – they do not attract attention (e.g. “General Conference” and “Executive Board”)




Hide the submenus (under “Secretariat”) – and make it available by a click on “Secretariat”?

By default, the Organisation menu should only offer “About UNESCO”, “General Conference”, “Executive Board”, “Director-General” and “Secretariat”

“The home page should focus: it should only give access to UNESCO’s main sectors and organs” (India)

“The ‘General Conference’ and ‘Executive Board’ links should show up more” (Switzerland)

This 3 column layout is more adapted for 1024 resolution.




A few users deplored the fact that the home page like the whole website did not convey efficiently UNESCO’s mission, values and messages

“I would prefer a global picture in the central area, a more universal look to it” (Kazakhstan)

“The home page fails at showing what UNESCO is” (Canada)

The home page does not communicate UNESCO’s spirit enough

3



The home page content is rather insider-oriented: non-UNESCO users may not get a good insight of what UNESCO is and does

A baseline on top of the page could provide a good introductory overview of UNESCO’s definition missions:

“UNESCO - the United Nations Educational, Scientific and Cultural Organization (UNESCO) was founded on 16 November 1945. For this specialized United Nations agency, Education, Social and Natural Science, Culture and Communication are the means to a far more ambitious goal : to build peace in the minds of men”

The home page does not communicate UNESCO’s spirit enough

3



The central area does not give an overview of UNESCO’s core activities (i.e. key international programs)

The central area is not well perceived, it doesn’t catch UNESCO people’s attention a lot – insiders are more attracted to the menus

“I don’t browse the central area” (Holy See)

“It doesn’t catch my attention” (Kazakhstan)

3The home page does not communicate UNESCO’s spirit enough



The central area layout is not clear:

• The articles in the “News” and “Events” boxes are separate with white lines and look like they’re discrete boxes – therefore, the second article, for example, does not seem to be part of the news

The boxes should be united

The central boxes’ layout is confusing3



The themes tabs on top of the page were not well perceived by some users – the lack of consistency across the portal restrains learnability of the navigation:

• Different graphical aspect

• Different tabs – some sites offer an “The Organisation” tab or a “Site map” tab, which could be extended to the whole portal

Some accesses are not perceived2



Some users deplored the fact that the home page did not mention the ongoing Executive Board (Palestine, European Commission)

The home page does not mention some current events2



The tab’s icon and label are so far from each other that they don’t seem to be linked together

The site map icon needs some graphical readjustment

The French “Plan” tab is not well designed2











40C. Perception and understanding of the “Worldwide” section

The “Worldwide” access was well perceived by most participants

It was also well understood: most participants had a good insight of what the section offers – local information about UNESCO organisation and programmes

The Worldwide section available from the left menu was generally appreciated as a useful source of local information

The “Worldwide” access was well perceived and appreciated



The “Worldwide” access was easily used by most users – they had no difficulties accessing the page dedicated to their country

The “Worldwide” access is mostly clear



A few users got confused by the “Worldwide” access, which follows UNESCO’s own logic in aggregating regions of the world, which does not match the common logic about world’s continents and therefore may confuse non-UNESCO people – e.g. Europe and North America form a single region and the Russian Federation and Turkey are listed twice, both in the “Europe and North America” and the “Asia and the Pacific” regions.

The “Worldwide” access is also confusing2



Several interpretations were given as for UNESCO’s own logic in aggregating regions of the world:

• Regions are voting groups

• Regions follow a “political” logic

The “Worldwide” access is also confusing2



The availability of country-specific information (themes) was mostly appreciated

The cross-referencing of information principle was appreciated



Some users did not notice that some more content was available in the lower area of the page

The page layout should allow users to notice the content below – the browser should “cut” the page where some content is partly visible in order to suggest some more content

3The themes were not easily perceived by users



A few users deplored some difficulties anticipating the contents “behind” the themes links on the country-specific pages

The links are not clear enough, not understood as being contextual, country-specific

There is no clear introduction

Explanatory sentences should introduce the themes’ content (what about the type of links is)

The themes’ content was not easily anticipated by users 2



Many users did not see the “Organisation” menu on the country-specific pages

• The background colour and the search engine and “UNESCO at 60” picture above the menu make it “secondary” (Austria)

• The “organisation” menu lacks consistency

• “The Organization” label is not country-specific and the “UNESCO in this Country” label is not well located – it applies only to the 2 links underneath it

The “Organisation” menu lacks visibility3



The menu should be moved upward in the place of the search engine and picture and gain consistency from one page/section to another

The menu label should be more specific

The “Organisation” menu lacks visibility3



According to the users, some contents are poor or information is missing or not enough emphasised :

• The countries’ main sectors of activity and the key programmes (Luxembourg, Greece, Holy See, Poland, Slovenia, Ethiopia, etc.)

• The statistics should come along with interpretations (Indonesia)

• The National Commission programme (Netherlands)

• A link to the country website from the National Commission page (Argentina, Kazakhstan)

Content was sometimes assessed not to be full-accomplished

2



Some contents are not up to date – and some pages are still empty (Slovenia)

Some users (Slovenia, Luxembourg) suggested the National Commissions should be involved and provide BPI with information – it could be through a form –, given the National Commissions are in charge of defining their status, composition principles and main sectors of activities

Some country-specific pages content is outdated2



Some users found the pictures could be more attractive (India, Luxembourg) and linked to the content

Some country-specific pages pictures are disliked2



The text is compact – its format should be improved in order to ease the reading:

The main words and sentences should be written in bold font

Text should be divided into different paragraphs

There should be more pictures on both side of the central area

Text format is not adapted2











54D. Perception and understanding of the UNESCO Courier

Most users would likely read the UNESCO Courier online

Most of them would like to print it out – be it in PDF format or directly from their Internet browser

The UNESCO Courier was appreciated



All the users would likely participate in the diffusion of the UNESCO Courier, mostly via email – as a link, as a PDF document or using a “send this article to a friend” feature

Users would diffuse the UNESCO Courier



The website offers several accesses to the UNESCO Courier

Several ways to access the UNESCO Courier



The UNESCO Courier box was not perceived: many users missed it although they had scrolled the page down to the bottom

The UNESCO Courier access should be easier: it should show up more, either visually (a specific colour, layout) and/or by its location (to be moved upward, in a menu?)

The title “The UNESCO Courier” should be clickable

The UNESCO Courier central access is not well perceived3



The first paragraph is a general introductory mention to the online version of the UNESCO Courier

As such, it should be located on top of the page, above the first picture and first article

The “Free subscription” link should be located right under the first paragraph

The right menu is not clear enough: the first item is not clearly related to the Courier’s main theme – water

Put an introductory mention like “Read more (about this theme)” instead of “Features”

The UNESCO Courier layout needs improvement2











60B. Transversal issues

The standardization work done so far and to be continued was appreciated for adding consistency to the portal and bringing better ease of use to the user experience: layout and navigational principles

The Framework was appreciated



Many users reported difficulties finding what they want (mostly documents):

• The website was often reported as requiring some learning – the content is seen as huge and the structure as complex

• Searches by keywords and themes were reported to give poor results

• The search engines don’t give acceptable results often enough – some participants use Google to find documents in the UNESCO portal

• A French-speaking participant (Algeria) deplored the fact that the advanced search feature is only available in English

Search issues3


62F. Perception and understanding of other main sections

The “Executive Board” search engine suggests that only searches by document codes can be run here – some users weren’t aware that a search by keywords was also possible here

A few users wished they could run searches by themes

Document-finding difficulties3



Some information on the website is outdated – especially in the “Worldwide” and “Themes” sections

Some documents are uploaded and available on the website too late vis-à-vis UNESCO users’ needs during the Executive Board – these late updates add some difficulties in finding the desired information or documents

“The information is brought out to late” (Algeria)

“The meetings’ outcomes are not published right after the meeting is over” (Portugal)

Late updates3



The lack of consistency across the portal (different layouts) restrains learnability and understanding from one section to another

Lack of consistency across the portal3



Page titles and layout fail to provide efficient guidance due to a lack of a logical and consistent principles in building titles:

The page title building principle should be the same through the whole portal

The page title building principle should always refer to the relevant menu building principle – i.e. the items’ organisation and order

The title of a page should always refer to the link that brought to the page and vice versa

Lack of consistent and efficient guidance principles3



On the first page shown on the left the subtitle doesn’t refer to any item in the menu on the right

On the second page, “About UNESCO” should remain visible as the page title while “The Organisation’s History” should be the subtitle

The titles and subtitles should be the same format – titles either bigger or smaller than the subtitles – through the whole portal




The last page of the breadcrumbs does not always refers to the last clicked link and the current page title

The last page of the breadcrumbs should always match the last clicked link and the current page title

Therefore the last page of the breadcrumbs should not be clickable




The breadcrumbs are usually located on top of the central area rather than above the left menu

Moreover, the breadcrumbs format (font type and size, background colour) is the same as the menu below, which reduces its visibility

Breadcrumbs should start reading above the central area in order to offer the best visibility possible

The background should be white




Some UNESCO sites do not permit going back to the portal home page

• e.g. the World Heritage web site has the same UNESCO logo as the portal although this one leads to the World Heritage home page

• The UIS web site offers no link to the portal




The “Organisation” menu changes (content, format, location) from one page or section to another, which is confusing




The home page links – top tab, breadcrumbs and top left logo – were not well perceived

Most participants used the browser’s “Back” button

Home page links not well perceived3



A few users deplored the lack of explanations about the acronyms used on the website

Acronyms are not clear2



A few users wished the themes landing pages would provide users with a baseline mentioning UNESCO’s key goals and programmes in the relevant field

Expectations – Themes landing page



Some users wished the website offered:

• Yearly and monthly meetings schedules

• Daily meetings summary

Expectations – Some extra content











76G. General opinions

The main strong points reported by the participants were:

• A great amount of useful information available

• Numerous accesses on the home page

• The standardization process

• Country-specific information

• 6 languages

Strong points



The main weak points were:

• Overall complexity, especially for outsiders or newcomers

• Problems to find certain information and documents

• Outdated contents

• Heavy home page

Weak points


78H. Post-test questionnaire – Performance and satisfaction rating

Performance indicators: 5 criteria rated from 1 to 5

• Attractiveness / Website utility

• Browsing/Locating throughout the site (guidance)

• Efficiency / Ease of access to information

• Content structure (Logic in content organization)

• Understanding / Learnability

Criteria assessed through a questionnaire (inspired from the Wammi questionnaire): participants are asked to tell whether they agree or not (from 0 – not at all – to 5 – totally) with 20 statements (4 statements per criterion)

• E.g.: “This website has much that is of interest to me”

Fully disagree • • • • • Fully agree



Utility has a very good score (4.2) – it might refer to the rich content (compared to 3.5 for the first study)

Browsing and Understanding/Learnability could be better (resp. 3.3 and 3.2) – it might refer to the difficulties in finding certain contents (compared to 2.6 for the first study)

Efficiency/Ease of access has a good score (3.8) – maybe due to the expertise of most of the participants (compared to 3.3 for the first study)

Content structure has also a good score (3.6) (compared to 2.9 for the first study)

Scores for the performance indicators were pretty good

Performance indicators

1,0

1,5

2,0

2,5

3,0

3,5

4,0

4,5

5,0 Attractiveness/ Websiteutility

Browsing/locatingthroughout the site

Efficiency/Ease ofaccess

Content structure

Understanding/Learnability

5. Main recommendations


81Main recommendations

The standardization work must be continued to offer a better overall user experience

• Strong and efficient guidance principles should be applied across the portal

The portal targets need to be clearly defined and the website be adapted to them: is it a portal for everyone or a working tool for the insiders?

• The right column offers too many links, specially for the general public

The home page needs more clarity, especially the layout of the central area

The work done to offer more multilingual contents should also be continued as it was very appreciated

The updates need to be done more regularly – especially for a working tool

The “Worldwide” section should offer more contents and be updated more regularly

The search engines need improvement


82

Axance « User Experience Design »

5, rue Régnault93 697 Pantin Cedex FranceTel : 01 48 91 99 93

Axance© Juin 2006

Usability Test – UNESCO.org Full Report June 8 2006 Laetitia Giannettini[[email protected]]Usability Specialist Cyril Alegre [[email protected]]Usability.

Documents

axance user experience

unesco portal

rue rgnault

pantin france

unesco web portal

languages languages

previous user tests

distinct unesco websites