Rashmi Xerox Parc

1

MindCanvas: Reimagining online research

Rashmi Sinha

Uzantowww.uzanto.com

www.themindcanvas.com

2

Duct-tape research!

(in the trenches - Information Architecture Redesign of eBay.com)

3

Background to IA redesign

• eBay.com has grown in organic fashion – inconsistent, fractured user experience– Separate design / usability efforts for parts of eBay

• first effort to focus on user experience for entire site

• Basic eBay facts– World’s largest online marketplace (79 million in US)– Types of users

• Buyers & sellers• New & experienced• Individuals & corporations• Different countries & cultures

4

Challenges

• New IA would need to– Serve diverse user group

• varying goals / mental models

– Be future-oriented, allow for change & growth– Changes needed to be evolutionary not revolutionary

• accommodate users familiar with current site

• Senior management at eBay needed proof– Is there need for such drastic restructuring?

• Research to be completed in 6 weeks!

5

Goals of research

• Research has multiple goals– Need successful design, not just good design

• Serves user• Satisfies business goals • Is accepted by management

• 3-stage research method• Stages: Exploration Understanding Verification• Stages complement & build upon each other

• Mix of qualitative & quantitative methods • Qualitative: rich, open-ended understanding of users / domain• Quantitative: recommendations grounded in data (help make case

to management)

6

3-stage user research

Stage 1: ExploreInterviews, free-listing

Stage 3: Verify / Refine

Large-sample closed card-sorting

3 stages

Stakeholders

User thinking / categorization

Preliminary IA

Stage 2: UnderstandOpen card-sorting

ebay.com

Community

Buy SellSub cat1

SubCat2

SubCat3

xx

SubCat1

Subcat4

Subcat3

Subcat2

xx

xx

xxx

SubCat3

SubCat5

SubCat4

SubCat6

SubCat4

xx

xxx

xx

xxx

xxx

xxx

SubCat4

SubCat5

SubCat1

SubCat2

xxx

xx

xx

Verify categorization

Top-level

cate

gories

Scope of content & functionality

7

Stage 1: Explore domain with Freelisting

• Goals– Explore domain in open-ended fashion, map scope.– Generate list of current & “Horizon Tasks” (planned for

future)

• Method– Interviews & Freelisting– With eBay users, stakeholders & designers

• Freelisting tasks (eBay is about tasks!)– What are all the tasks users can do on eBay.com

• List an item, Pay for it– What else will users be able to do on eBay.com 2-5

years that they can’t do today?

• Results: List of 100 representative tasks (25 Horizon)

8

Stage 2: Explore user categorizations

• Goals: – Gain insight into user thinking about eBay– Identify top-level categories for ALL site

content and functionality

• Method: Interviews & Open card sorting – List of 100 tasks (including 20 Horizon

Tasks).– 35 participants (including sellers & buyers;

new and advanced users).

9

Identify top levels of hierarchy

• Hierarchical Cluster analysis to generate aggregate user categorizations.

• Separate analysis for 4 user groups – Identify inconsistencies in

categorizations– Reconcile to create one scheme.

• Result: Preliminary IA– 5 top-level categories accounting for all

site content and functionality.

0.000

0.200

0.400

0.600

0.800

1.000

See eBay time

Fn eBay help

Fn cusTrk SupportReq. tech. support

Read eBay guidance

See eBay overview

Get eBay appsRefer friend

Ask experts

Ask other members

Fn user groupsRead eBay msgs

View eBay news

Read member msgs

See aucTrk graphs

List item

Cancel auction

Add image

Add res./BIN priceRevise listings

Create listing

Answer question

Send invoiceManage inventory

List regionally

List internationally

List item to donateCross-sell items

Manage eBay store

Trk shipments

IntLrn customsEsTrk shipping

Reg. as buyerReg. as seller

Report bad itemsReport accTrk theft

Report NPB/fraudRsrch. promotions

Rsrch. items sold

Rsrch. for listing

Rsrch. categoriesLrn manage bus.

Lrn seller benefits

Get selling tools

Be Power SellerFn sales by member

Fn items by bidderFn info. on eBay-OK code

Rsrch. partneringRsrch. eBay services

Be Trad. AssTrk / consultSee aucTrk cancels.

See bid retractions

View id history

Fn eBay memberMember contact info.

Lrn manage rep.Read feedback

Post feedbackBIN

Bid

Buy digital

Bid promo. option

Reverse auction

Negotiate price

Offer for unsold item

EsTrk car payment

Get buying tools

Contact seller

Contact buyer

Fn accessoriesFn items by seller

Search & browse

Fn BIN items

Fn items nearby

Fn specific item

Fn types of items

Browse items

Fn proFn servicesView item desc.

Translate listing

Fn eBay internatLrn

Save itemsTask Y

Task U

Task OTask F

Task R

Task S

Task D

Task L

Task M

Task A

Task K

Task GeTask P

Task X

Task B

Task ZTask N

Dendrogram- User group !

0.000

0.200

0.400

0.600

0.800

1.000

See eBay time

Fn eBay help

Fn cusTrk SupportReq. tech. support

Read eBay guidance

See eBay overview

Get eBay appsRefer friend

Ask experts

Ask other members

Fn user groupsRead eBay msgs

View eBay news

Read member msgs

See aucTrk graphs

List item

Cancel auction

Add image

Add res./BIN priceRevise listings

Create listing

Answer question

Send invoiceManage inventory

List regionally

List internationally

List item to donateCross-sell items

Manage eBay store

Trk shipments

IntLrn customsEsTrk shipping

Reg. as buyerReg. as seller

Report bad itemsReport accTrk theft

Report NPB/fraudRsrch. promotions

Rsrch. items sold

Rsrch. for listing

Rsrch. categoriesLrn manage bus.

Lrn seller benefits

Get selling tools

Be Power SellerFn sales by member

Fn items by bidderFn info. on eBay-OK code

Rsrch. partneringRsrch. eBay services

Be Trad. AssTrk / consultSee aucTrk cancels.

See bid retractions

View id history

Fn eBay memberMember contact info.

Lrn manage rep.Read feedback

Post feedbackBIN

Bid

Buy digital

Bid promo. option

Reverse auction

Negotiate price

Offer for unsold item

EsTrk car payment

Get buying tools

Contact seller

Contact buyer

Fn accessoriesFn items by seller

Search & browse

Fn BIN items

Fn items nearby

Fn specific item

Fn types of items

Browse items

Fn proFn servicesView item desc.

Translate listing

Fn eBay internatLrn

Save itemsTask Y

Task U

Task OTask F

Task R

Task S

Task D

Task L

Task M

Task A

Task K

Task GeTask P

Task X

Task B

Task ZTask N

0.000

0.200

0.400

0.600

0.800

1.000

0.000

0.200

0.400

0.600

0.800

1.000

0.000

0.200

0.400

0.600

0.800

1.000

See eBay time

Fn eBay help

Fn cusTrk SupportReq. tech. support

Read eBay guidance

See eBay overview

Get eBay appsRefer friend

Ask experts

Ask other members

Fn user groupsRead eBay msgs

View eBay news

Read member msgs

See aucTrk graphs

List item

Cancel auction

Add image

Add res./BIN priceRevise listings

Create listing

Answer question

Send invoiceManage inventory

List regionally

List internationally

List item to donateCross-sell items

Manage eBay store

Trk shipments

IntLrn customsEsTrk shipping

Reg. as buyerReg. as seller

Report bad itemsReport accTrk theft

Report NPB/fraudRsrch. promotions

Rsrch. items sold

Rsrch. for listing

Rsrch. categoriesLrn manage bus.

Lrn seller benefits

Get selling tools

Be Power SellerFn sales by member

Fn items by bidderFn info. on eBay-OK code

Rsrch. partneringRsrch. eBay services

Be Trad. AssTrk / consultSee aucTrk cancels.

See bid retractions

View id history

Fn eBay memberMember contact info.

Lrn manage rep.Read feedback

Post feedbackBIN

Bid

Buy digital

Bid promo. option

Reverse auction

Negotiate price

Offer for unsold item

EsTrk car payment

Get buying tools

Contact seller

Contact buyer

Fn accessoriesFn items by seller

Search & browse

Fn BIN items

Fn items nearby

Fn specific item

Fn types of items

Browse items

Fn proFn servicesView item desc.

Translate listing

Fn eBay internatLrn

Save itemsTask Y

Task U

Task OTask F

Task R

Task S

Task D

Task L

Task M

Task A

Task K

Task GeTask P

Task X

Task B

Task ZTask N

See eBay time

Fn eBay help

Fn cusTrk SupportReq. tech. support

Read eBay guidance

See eBay overview

Get eBay appsRefer friend

Ask experts

Ask other members

Fn user groupsRead eBay msgs

View eBay news

Read member msgs

See aucTrk graphs

List item

Cancel auction

Add image

Add res./BIN priceRevise listings

Create listing

Answer question

Send invoiceManage inventory

List regionally

List internationally

List item to donateCross-sell items

Manage eBay store

Trk shipments

IntLrn customsEsTrk shipping

Reg. as buyerReg. as seller

Report bad itemsReport accTrk theft

Report NPB/fraudRsrch. promotions

Rsrch. items sold

Rsrch. for listing

Rsrch. categoriesLrn manage bus.

Lrn seller benefits

Get selling tools

Be Power SellerFn sales by member

Fn items by bidderFn info. on eBay-OK code

Rsrch. partneringRsrch. eBay services

Be Trad. AssTrk / consultSee aucTrk cancels.

See bid retractions

View id history

Fn eBay memberMember contact info.

Lrn manage rep.Read feedback

Post feedbackBIN

Bid

Buy digital

Bid promo. option

Reverse auction

Negotiate price

Offer for unsold item

EsTrk car payment

Get buying tools

Contact seller

Contact buyer

Fn accessoriesFn items by seller

Search & browse

Fn BIN items

Fn items nearby

Fn specific item

Fn types of items

Browse items

Fn proFn servicesView item desc.

Translate listing

Fn eBay internatLrn

Save itemsTask Y

Task U

Task OTask F

Task R

Task S

Task D

Task L

Task M

Task A

Task K

Task GeTask P

Task X

Task B

Task ZTask N

Dendrogram- User group !

Dendogram

10

Stage 3: Verify & refine scheme

• Goals:– Verify 5-category scheme covers all of eBay.

• What tasks/concepts don’t fit? Where do users expect this info?– Input from larger, more diverse set of users.– Establish method for future verification of IA.

• and to generalize scheme to international eBays.

• Method: closed card sorting – Large sample (~ 1,000) representing all eBay user types.– Study conducted using online survey.

• Participants categorized tasks into one of categories (or selected Other if it didn’t belong)

• Result– Site-map showing the structure for all main parts of the

eBay.com

11

Sell - Before

12

Sell Hub - After

13

eBay redesigned successfully!

• Three-stage methodology– Understand business context – Delineate user categorizations– Create IA blueprint

• Deliverable: site-map with structure for main parts of eBay.com. – map used as blueprint of IA redesign

(took 2 years).– Generalized to international eBays.– Positive business implications (ROI)!– Users did not protest!

site.com

TopCategory 3

TopCategory1

TopCategory 2

TopCategory 4

SubCat1Sub cat1

SubCat2

xx

SubCat3

xx

xx

SubCat2

SubCat3

SubCat5

xx

xx

xx

xx

SubCat1

Subcat4

Subcat3

Subcat2

xx

xx

xxx

SubCat3

SubCat5

SubCat4

SubCat6

SubCat4

xxx

xx

xx

xxx

xx

xxx

xxx

xxx

SubCat7

SubCat4

SubCat5

SubCat6

SubCat1

SubCat2

xx

xx

xxx

xx

xxx

xx

xx

SubCat4

xx

Preliminary IA

14

Duct-tape research!

There has to be a better way - fast, cheap (relatively) & online…

15

Why we built MindCanvas

• Qualitative research is great, takes a lot of time

• Online tools suck, get used anyway (users not engaged)

• Business stakeholders respond to quantitative analysis & large sample sizes

• Deliverables do not cater to designers

• Research findings remain locked up with analyst

16

Inspiration

• Luis Von Ahn’s ESP Game– Rely on games to label the web

• PopCap Games– Quick, short, engaging games

• Visualization research– bringing statistics to designers

17

18

back to the future!

MindCanvas interpretation

19

What is MindCanvas?

Remote research methods

Statistical Analysis & Data Mining

Interactive Visualizations

loosely coupled, use as qualitative or quantitative

20

Game-like elicitation methods

• Surveys do not engage people. Games are fun!– People engage in complex tasks willingly

• Do game-like methods work better?

– MindCanvas has game-like elements (its not a game)• Interactive, visual, screen build up, fun

• For both qualitative & quantitative– Remote moderated for qualitative– Remote unmoderated for quantitative

21

Rich interactive visualizations

• Large datasets need statistical analysis & data mining

• Visualizations-to-go• Self-contained files - embedded into PowerPoint, emailed, shared

• Interactive visualizations with slides & knobs!

• Easy to share with team

• Visualization posters– Designers want printouts to hang on walls, and scribble on

22

MindCanvas research focuses on the mind

• Inspiration from cognitive anthropology, psychology, market research

• Semi-structured methods

• Quantitative aggregation & visualization

Categorizations

PreferencesLanguage

23

Who we built for: designer persona

Needs convenient

methods to do research

Needs convenient

methods to do research

Does not want to spend time

learning statistics

Does not want to spend time

learning statistics

The busy designer / researcher who wants to look at patterns visually

Want to directly interact with users, does not have time

Want to directly interact with users, does not have time

24

Who we built for: participant persona

“This was the most fun I

have had in a while!”

“This was the most fun I

have had in a while!”

“It is very easy to complete the task. In fact, I had fun doing

this.”

“It is very easy to complete the task. In fact, I had fun doing

this.”

The not-tech savvy person who occasionally takes surveys

“I think there are too many cards. But it was kind of

fun.”

“I think there are too many cards. But it was kind of

fun.”

25

Where MindCanvas could help

• Product Development: Finding the right feature-set for the right users (or personas)

• Information Architecture: Understanding how people think for information design.

• Early visual design validation: Understanding what people think & feel about a design when its just an image.

26

MindCanvas in product development

• Initial understanding of domain– What’s important to people? What words do they

use?• Method: Freelisting• Deliverable: ListMap

• Prioritizing features / outcomes users care about– Method: Divide-the-dollar– Deliverable:

• WeightMap• ClusterBrowser

27

MindCanvas in Information Architecture

• How do people think about a domain?– Method: OpenSort– Deliverable

• Dendogram• SimilarityBrowser• VocabularyBrowser1, 2

• Is your information architecture effective? Is one scheme more effective than another?– Method: TreeSort– Deliverable:

• ClusterBrowser

28

29

Questions?

MindCanvas team is…Delhi

Kapil MohanKamalmeet SinghDebasish Biswas

Sunil GuptaAmit Ranjan

Mountain ViewJon Boutelle

Rashmi Sinha

[email protected]

30

Methods & Deliverables

• Current Methods– OpenSort: Dendogram, VocabBrowser, SimilarityBrowser

– TreeSort: ClusterBrowser, SimilarityBrowser

– Divide-the-Dollar: WeightMap, ClusterBrowser

– Free Listing: ListMap

– Utility Questions: MultipleChoice, EssayType

• Upcoming Methods: MenuTrace, MindMontage, Likert Sort

• Upcoming Designables: OnionMap

31

Collaborative Research Process

• High quality research needs expertise– research design + statistics

• Service should enable high quality research– Structured research process

• Templates for study design• Help with data collection

32

What is MindCanvas

• An innovative research platform for high quality research– Game-like elicitation methods– Interactive visualizations (designables-to-go)– Research expertise