Strategic Approach to Web Evaluation

Strategic Approach to Web Evaluationby Fred B. WoodOffice of Health Information Programs DevelopmentNational Library of Medicine

NIH MEDLINEplus Advisory Group MeetingJune 22, 2001

M+AdvGrp1.ppt

Overview--Web Evaluation Methods TestedOnline random survey of MEDLINEplus users (in collaboration with CyberDialogue)Nationwide telephone survey of online health information users (CyberDialogue, syndicated)Online random survey of non-NLM health information users (PCData online panel)Comparisons with other surveys and NLM web log dataCompetitive analysis of MEDLINEplus (CyberDialogue online panel)Internet audience measurement (PCData, netScore panels)Internet connectivity performance tests

MEDLINEplus Online SurveyNLM developed 22-question survey in collaboration with Cyber Dialogue, Inc.First time visitors were asked only background and demographic questions, repeat visitors also were asked evaluative and impact questionsSurvey instrument approved under OMB blanket clearance procedureSite visitors were randomly intercepted and given opportunity to respond

MEDLINEplus Online Survey (contd)93,852 visitors intercepted between February 14-27, 20012,969 completed the survey (3% overall response rate--typical of online web surveys)Transient cookies used to preclude repeat intercepts, expired at end of survey, and no behavioral tracking (consistent with OMB cookie policy)1,307 (44% of total respondents) expressed interest in participating in future MEDLINEplus surveys and provided an e-mail address

*

Methodology

2) Visitors are invited to take part in the survey.

3) Visitors who agree to take part in the survey are temporarily redirected to a secure Cyber Dialogue server and take the survey in a separate browser window.

1) MEDLINEplus visitors are randomly intercepted on the site.

4) After completing the survey, visitors continue surfing.

Intercept/Survey Process:

*

Respondent Profile

Gender, Age, Education, Race & Ethnicity

*Data Source: CCH 2000, Cyber Dialogue**Repercentaged after removing Refused/No Answers

On average, MEDLINEplus visitors are more likely to be female, college graduates and older than health site visitors or U.S. online adults in general.

*

Respondent Profile

Country of Residence & Reading Language

Almost three-quarters of respondents access the site from the U.S, and English is the primary reading language for 88% of respondents.

Access Location:

Primary Reading Language:

*

Respondent Profile

MEDLINEplus Access Location

Visitors most likely access the site from home. However, half of repeat visitors access the site from outside the home (including work, school, libraries, etc.)

*

Learned of Site

Search engines and links from another site are the primary means that MEDLINEplus visitors first learned of the site. Libraries, health organizations and links from other NIH sites appear to be effective in attracting repeat visitors.

*

Visitor Roles

*

Reactions to MEDLINEplus

Overall Satisfaction with MEDLINEplus

*Less than .5%** Based on normative data from 110 sites

81% of MEDLINEplus repeat visitors are extremely or very satisfied with the site, exceeding the Cyber Dialogue norm of 59%**.

*

Reactions to MEDLINEplus

Likelihood to Return

- No respondents*Less than .5%** Based on Cyber Dialogue Norms from 110 Sites

92% of repeat visitors are extremely or very likely to return to the site, exceeding the Cyber Dialogue norm of 57%**.

*

Reactions to MEDLINEplus

Frequency of Finding Information on MEDLINEplus

86% of repeat visitors cite they always or frequently find the information they are looking for on the site.

(Base: Repeat Visitors, n=1455)

*

Results of Obtaining Information

Most repeat visitors cited an improved understanding of a condition (64%) as a result of using information obtained from MEDLINEplus.

(Base: Repeat Visitors, n=1455)

* Disease, condition, diagnosis, or treatment

Cross-Comparison with Web Log Data--MEDLINEplusTime-of-day/day-of-week variability--97% correlation between survey data and web log data.US/non-US user split--73% US/27% non-US per survey, 65% US/35% non-US per web log data (but majority of log data cannot be resolved).Repeat visitors (2+ times per month)--36% per survey, 21% per web log data (but log data subject to error factors both ways)Survey non-response bias may be relatively minimal.

MEDLINEplus Competitive AnalysisCompared MEDLINEplus, WebMD, Mayo Health, InteliHealth, and MedscapeUsed CyberDialogues virtual online panel (120K)Panelists randomly sampled to identify 950 online health site usersEach panelist asked to compare two of the five sites and complete an online survey520 completed the survey with 482 valid responses (51% net response rate)

*

MEDLINEplus vs. Competitors

MEDLINEplus outperforms each competitor and Cyber Dialogues norm in terms of overall site satisfaction. More than three-quarters of respondents (78%) are extremely or very satisfied with the MEDLINEplus site.

Overall Satisfaction

*

MEDLINEplus vs. Competitors

Six out of 10 respondents are extremely/very likely to return to MEDLINEplus within the next three months. This is on par with MayoHealth, WebMD (both at 62%) and the Cyber Dialogue norm (57%).

Likelihood to Return within 3 Months

*

MEDLINEplus vs. WebMD

Highly Important

Moderately Important

Less Important

Note: Underlined attributes have a 10 point or greater difference between site ratings.

Internet Audience Measurement Online PanelsNLM contracted with PCDataOnline, Inc. for external usage data based on PCDatas online panel of 120K home users who agreed to monitoring of web usageResults extrapolated to US home Internet marketPCData ceased operations in March 2001, client base bought by comScorecomScore offers netScore with a panel of 1.5M users covering home, work, and schoolCan cross-compare PCData, netScore, & log data

Sheet4

M+Chart

1349387151

1549458158

1227398127

1529498164

1762548187

1969617218

2219650235

2324686248

2233668238

2313720265

2476760298

2468782324

3036915387

2947915389

2445757321

33111015434

36651023421

43771172455

53801117438

Pages

Total Visits

Unique Visitors

Pages, Visits, Unique Visitors (x000)

MEDLINEplus Web Log Usage DataOctober 1999-April 2001 (courtesy T. Luedtke, OCCS)

M+Chart2

1349387151

1549458158

1227398127

1529498164

1762548187

1969617218

2219650235

2324686248

2233668238

2313720265

2476760298

2468782324

4591915387

4364915389

3901757321

49171015434

50661023421

63461172455

60831117438

60461314518

Pages

Total Visits

Unique Visitors

Pages, Visits, Unique Visitors (x000)

MEDLINEplus Web Log Usage DataOctober 1999-May 2001

M+data

DatePagesTotal VisitsUnique Visitors

Oct-991349387151

Nov-991549458158

Dec-991227398127

Jan-001529498164

Feb-001762548187

Mar-001969617218

Apr-002219650235

May-002324686248

Jun-002233668238

Jul-002313720265

Aug-002476760298

Sep-002468782324

Oct-004591915387

Nov-004364915389

Dec-003901757321

Jan-0149171015434

1-Feb50661023421

1-Mar63461172455

1-Apr60831117438

1-May60461314518

Sheet2

Sheet3

HealthChartUS

36192361921062369485433542

362203622012671891280345794

36251362519472018290166588

163362817192200319365549

53136312746875320170446

512363426371474300267401

12073479763474399514766

76559610975539415675681

275471510594654434585513

251478911193639510687668

24266907974153393439453

3794103612382383476793604

304695311233327522699677

40099921487484566810841054

51781382193924287665631031

67603247176722129229061079

54602978129420046861321690

644135921587234447617271125

637631281675233384121831608

79731912176621267021854707

830823112217165197112001367

10006870232713827666841114

10410101121801070689682825

106223689226481057734462257

10465369232604952745431212

10757369512955878792459229

909422953229900834468192

673117632969934894453214

WebMD

AllHealth

NIH.gov

DrKoop

Medscape

InteliHealth

MayoHealth

Unique Visitors (x000)

US Users of Health Web Sites--Unique Visitors Per Month, 1999-2001 (Sources: PCData through Nov 2000, netScore thereafter)

HealthDataUS

WebMDAllHealthNIH.govDrKoopMedscapeInteliHealthMayoHealth

Feb-991062369485433542

Mar-9912671891280345794

Apr-999472018290166588

May-991637192200319365549

Jun-99531746875320170446

Jul-995126371474300267401

Aug-9912073479763474399514766

Sep-9976559610975539415675681

Oct-99275471510594654434585513

Nov-99251478911193639510687668

Dec-9924266907974153393439453

Jan-003794103612382383476793604

Feb-00304695311233327522699677

Mar-0040099921487484566810841054

Apr-0051781382193924287665631031

May-0067603247176722129229061079

Jun-0054602978129420046861321690

Jul-00644135921587234447617271125

Aug-00637631281675233384121831608

Sep-0079731912176621267021854707

Oct-00830823112217165197112001367

Nov-0010006870232713827666841114

Dec-0010410101121801070689682825

1-Jan1062226481057734462257

1-Feb104652604952745431212

1-Mar107572955878792459229

1-Apr909422953229900834468192

1-May673117632969934894453214

GovtChartUS

36192361921062122015181251

362204041267104014931016

362514459477921958904

362814447193051042790

36312412746248854603

363424016372781065609

3637358497636817291023

3640475110975311494868

3643474210594331653902

3646570111194371803747

3649511087974481884645

365268851238175019331305

363021821123223517451675

208421281487137518131482

292427021939215921931623

5542759176743118391410

4942477129442115151342

7193019158739715941713

6103203167562918411317

7553516176650820841186

6334103221765023091195

6653838232758523121628

9994159218021127691380

4147438026483689224722518

5835343826043692326312888

43273630295535830782989

55634000322941831663033

14573951296926828282676

IRS

USPS

NIH.gov

FedWorld

NASA

DoEd

Unique Visitors (x000)

US Users of US Government Web Sites--Unique Visitors Per Month, 1999-2001 (Sources: PCData through Nov 2000, netScore thereafter)

GovtDataUS

DateIRSUSPSNIH.govFedWorldNASADoEd

Feb-991062122015181251

Mar-994041267104014931016

Apr-994459477921958904

May-994447193051042790

Jun-99412746248854603

Jul-994016372781065609

Aug-9958497636817291023

Sep-9975110975311494868

Oct-9974210594331653902

Nov-9970111194371803747

Dec-9911087974481884645

Jan-008851238175019331305

Feb-00363021821123223517451675

Mar-00208421281487137518131482

Apr-00292427021939215921931623

May-005542759176743118391410

Jun-004942477129442115151342

Jul-007193019158739715941713

Aug-006103203167562918411317

Sep-007553516176650820841186

Oct-006334103221765023091195

Nov-006653838232758523121628

Dec-009994159218021127691380

1-Jan41474380264824722518

1-Feb58353438260426312888

1-Mar43273630295535830782989

1-Apr55634000322941831663033

1-May14573951296926828282676

CombinedIntl

DateWebMD-USWebMD-IntlNIH.gov-USNIH.gov-IntlNASA-USNASA-IntlMedscape-USMedscape-Intl

Dec-00104101490721803788276952596891235

1-Jan106221577326484638247251467341317

1-Feb104651489426044593263152487451365

1-Mar107571539729555133307865117921531

1-Apr90941246632295314316660748341517

1-May6731914929694971282855948941527

HealthChartUS

36192361921062369485433542

362203622012671891280345794

36251362519472018290166588

163362817192200319365549

53136312746875320170446

512363426371474300267401

12073479763474399514766

76559610975539415675681

275471510594654434585513

251478911193639510687668

24266907974153393439453

3794103612382383476793604

304695311233327522699677

40099921487484566810841054

51781382193924287665631031

67603247176722129229061079

54602978129420046861321690

644135921587234447617271125

637631281675233384121831608

79731912176621267021854707

830823112217165197112001367

10006870232713827666841114

10410101121801070689682825

106223689226481057734462257

10465369232604952745431212

10757369512955878792459229

909422953229900834468192

673117632969934894453214

WebMD

AllHealth

NIH.gov

DrKoop

Medscape

InteliHealth

MayoHealth

Unique Visitors (x000)

US Users of Health Web Sites--Unique Visitors Per Month, 1999-2001 (Sources: PCData through Nov 2000, netScore thereafter)

HealthDataUS

WebMDAllHealthNIH.govDrKoopMedscapeInteliHealthMayoHealth

Feb-991062369485433542

Mar-9912671891280345794

Apr-999472018290166588

May-991637192200319365549

Jun-99531746875320170446

Jul-995126371474300267401

Aug-9912073479763474399514766

Sep-9976559610975539415675681

Oct-99275471510594654434585513

Nov-99251478911193639510687668

Dec-9924266907974153393439453

Jan-003794103612382383476793604

Feb-00304695311233327522699677

Mar-0040099921487484566810841054

Apr-0051781382193924287665631031

May-0067603247176722129229061079

Jun-0054602978129420046861321690

Jul-00644135921587234447617271125

Aug-00637631281675233384121831608

Sep-0079731912176621267021854707

Oct-00830823112217165197112001367

Nov-0010006870232713827666841114

Dec-0010410101121801070689682825

1-Jan1062226481057734462257

1-Feb104652604952745431212

1-Mar107572955878792459229

1-Apr909422953229900834468192

1-May673117632969934894453214

GovtChartUS

36192361921062122015181251

362204041267104014931016

362514459477921958904

362814447193051042790

36312412746248854603

363424016372781065609

3637358497636817291023

3640475110975311494868

3643474210594331653902

3646570111194371803747

3649511087974481884645

365268851238175019331305

363021821123223517451675

208421281487137518131482

292427021939215921931623

5542759176743118391410

4942477129442115151342

7193019158739715941713

6103203167562918411317

7553516176650820841186

6334103221765023091195

6653838232758523121628

9994159218021127691380

4147438026483689224722518

5835343826043692326312888

43273630295535830782989

55634000322941831663033

14573951296926828282676

IRS

USPS

NIH.gov

FedWorld

NASA

DoEd

Unique Visitors (x000)

US Users of US Government Web Sites--Unique Visitors Per Month, 1999-2001 (Sources: PCData through Nov 2000, netScore thereafter)

GovtDataUS

DateIRSUSPSNIH.govFedWorldNASADoEd

Feb-991062122015181251

Mar-994041267104014931016

Apr-994459477921958904

May-994447193051042790

Jun-99412746248854603

Jul-994016372781065609

Aug-9958497636817291023

Sep-9975110975311494868

Oct-9974210594331653902

Nov-9970111194371803747

Dec-9911087974481884645

Jan-008851238175019331305

Feb-00363021821123223517451675

Mar-00208421281487137518131482

Apr-00292427021939215921931623

May-005542759176743118391410

Jun-004942477129442115151342

Jul-007193019158739715941713

Aug-006103203167562918411317

Sep-007553516176650820841186

Oct-006334103221765023091195

Nov-006653838232758523121628

Dec-009994159218021127691380

1-Jan41474380264824722518

1-Feb58353438260426312888

1-Mar43273630295535830782989

1-Apr55634000322941831663033

1-May14573951296926828282676

CombinedIntl

DateWebMD-USWebMD-IntlNIH.gov-USNIH.gov-IntlNASA-USNASA-IntlMedscape-USMedscape-Intl

Dec-00104101490721803788276952596891235

1-Jan106221577326484638247251467341317

1-Feb104651489426044593263152487451365

1-Mar107571539729555133307865117921531

1-Apr90941246632295314316660748341517

1-May6731914929694971282855948941527

HealthChartUS

36192361921062369485433542

362203622012671891280345794

36251362519472018290166588

163362817192200319365549

53136312746875320170446

512363426371474300267401

12073479763474399514766

76559610975539415675681

275471510594654434585513

251478911193639510687668

24266907974153393439453

3794103612382383476793604

304695311233327522699677

40099921487484566810841054

51781382193924287665631031

67603247176722129229061079

54602978129420046861321690

644135921587234447617271125

637631281675233384121831608

79731912176621267021854707

830823112217165197112001367

10006870232713827666841114

10410101121801070689682825

106223689226481057734462257

10465369232604952745431212

10757369512955878792459229

909422953229900834468192

673117632969934894453214

WebMD

AllHealth

NIH.gov

DrKoop

Medscape

InteliHealth

MayoHealth

Unique Visitors (x000)

US Users of Health Web Sites--Unique Visitors Per Month, 1999-2001 (Sources: PCData through Nov 2000, netScore thereafter)

HealthDataUS

WebMDAllHealthNIH.govDrKoopMedscapeInteliHealthMayoHealth

Feb-991062369485433542

Mar-9912671891280345794

Apr-999472018290166588

May-991637192200319365549

Jun-99531746875320170446

Jul-995126371474300267401

Aug-9912073479763474399514766

Sep-9976559610975539415675681

Oct-99275471510594654434585513

Nov-99251478911193639510687668

Dec-9924266907974153393439453

Jan-003794103612382383476793604

Feb-00304695311233327522699677

Mar-0040099921487484566810841054

Apr-0051781382193924287665631031

May-0067603247176722129229061079

Jun-0054602978129420046861321690

Jul-00644135921587234447617271125

Aug-00637631281675233384121831608

Sep-0079731912176621267021854707

Oct-00830823112217165197112001367

Nov-0010006870232713827666841114

Dec-0010410101121801070689682825

1-Jan1062226481057734462257

1-Feb104652604952745431212

1-Mar107572955878792459229

1-Apr909422953229900834468192

1-May673117632969934894453214

GovtChartUS

36192361921062122015181251

362204041267104014931016

362514459477921958904

362814447193051042790

36312412746248854603

363424016372781065609

3637358497636817291023

3640475110975311494868

3643474210594331653902

3646570111194371803747

3649511087974481884645

365268851238175019331305

363021821123223517451675

208421281487137518131482

292427021939215921931623

5542759176743118391410

4942477129442115151342

7193019158739715941713

6103203167562918411317

7553516176650820841186

6334103221765023091195

6653838232758523121628

9994159218021127691380

4147438026483689224722518

5835343826043692326312888

43273630295535830782989

55634000322941831663033

14573951296926828282676

IRS

USPS

NIH.gov

FedWorld

NASA

DoEd

Unique Visitors (x000)

US Users of US Government Web Sites--Unique Visitors Per Month, 1999-2001 (Sources: PCData through Nov 2000, netScore thereafter)

GovtDataUS

DateIRSUSPSNIH.govFedWorldNASADoEd

Feb-991062122015181251

Mar-994041267104014931016

Apr-994459477921958904

May-994447193051042790

Jun-99412746248854603

Jul-994016372781065609

Aug-9958497636817291023

Sep-9975110975311494868

Oct-9974210594331653902

Nov-9970111194371803747

Dec-9911087974481884645

Jan-008851238175019331305

Feb-00363021821123223517451675

Mar-00208421281487137518131482

Apr-00292427021939215921931623

May-005542759176743118391410

Jun-004942477129442115151342

Jul-007193019158739715941713

Aug-006103203167562918411317

Sep-007553516176650820841186

Oct-006334103221765023091195

Nov-006653838232758523121628

Dec-009994159218021127691380

1-Jan41474380264824722518

1-Feb58353438260426312888

1-Mar43273630295535830782989

1-Apr55634000322941831663033

1-May14573951296926828282676

CombinedChart

104101490721803788276952596891235

106221577326484638247251467341317

104651489426044593263152487451365

107571539729555133307865117921531

90941246632295314316660748341517

6731914929694971282855948941527

WebMD-US

WebMD-Global

NIH.gov-US

NIH.gov-Global

NASA-US

NASA-Global

Medscape-US

Medscape-Global

Unique Visitors (x000)

US and Global Users of Select Web Sites--Unique Visitors Per MonthDec 2000-May 2001 (Source: netScore)

CombinedIntl

DateWebMD-USWebMD-GlobalNIH.gov-USNIH.gov-GlobalNASA-USNASA-GlobalMedscape-USMedscape-Global

Dec-00104101490721803788276952596891235

1-Jan106221577326484638247251467341317

1-Feb104651489426044593263152487451365

1-Mar107571539729555133307865117921531

1-Apr90941246632295314316660748341517

1-May6731914929694971282855948941527

Illustrative Audience Measurement DataUS Home Internet Market--NIH Drill DownUS Unique Home Users (% of NIH Total) [source: PCData]Dec 2000Jan 2001Feb 2001Domainsnlm.nih.gov42.6%45.3%50.4%niddk.nih.gov11.2 8.7 7.9nci.nih.gov 7.1 7.4 8.7nhlbi.nih.gov 4.4 3.8 6.0Web sitesMedlinePlus14.2%18.0%17.5%PubMed13.418.320.1www.nih.gov13.611.117.0www.niddk.gov11.2 8.7 7.9

How Accurate Are These Data?Different companies use different methodologiesPanel composition differs--home vs home/work/schoolRaw data are extrapolated based on demographicsOf US Government web domains, NIH.gov ranked between #5 and #7 by PCData and netScore in Feb 2001.NIH.gov ranked #3 (after USPS & NASA) by both companies, before tax and student aid seasons began, in Dec 2000Of health info domains, NIH.gov ranked #3 or #4 (after webMD, allhealth [iVillage], and ediets.com) by both companies in Feb 2001

How Accurate Are These Data (contd)?Relative rankings may be accurate, and absolute numbers may be in the ball parkBased on Feb 2001 NLM web log data: PubMed (~2M) + MEDLINEplus (~0.45M) + www.nih.gov (~0.45M) = 2.9M unique visitorsCompares with 2.5M for PCData and 2.6M for netScore, for the NIH.gov domainnetScore also measures international traffic, 2M unique visitors to NIH.gov in Feb 2001 or ~ 40% of total 4.6MAssuming these three web sites ~ 55% of total NIH.gov visitors, extrapolated total from web log data = 5.3M4.6M vs 5.3M is within 20%--NOT BAD!

CENDI Symposium--Co-Sponsored by NLMPresentations by government, private sector, and non-profit organizationsPresentations available on the CENDI web site at http://www.dtic.mil/cendi/ click on whats newOr go to www.dtic.mil/cendi/activities/ 04_17_01_eval_pres_agenda.html

EVALUATING OUR WEB PRESENCE:

CHALLENGES, METRICS, RESULTS