PERSPECTIVES ON ADAPTIVITY IN INFORMATION RETRIEVAL INTERACTION an ASIS&T 2010 annual meeting panel PAIRI
PERSPECTIVES ON
ADAPTIVITY IN INFORMATION RETRIEVAL INTERACTION
an ASIS&T 2010 annual meeting panel
PAIRI
PAIRI
Agenda/Outline
Adaptivity in IR Interaction
Four Dimensions
Pecha-Kucha Presentation and Timeline
Audience Interaction!
Wrapping Up
PAIRI
PRESENTERS
Birger Larsen, Royal School of LIS, Copenhagen, Denmark
Marianne Lykke, Aalborg University, Denmark
Diane Kelly, University of North Carolina at Chapel Hill, USA
Peiling Wang, University of Tennessee, Knoxville, USA
ModeratorPeter Ingwersen, Royal Schoolof LIS, Copenhagen, Denmark
PAIRI
ADAPTIVITY in IR INTERACTION
… requires that IR systems adapt to users’ situations, and
… the users adapt to the systems.
PAIRI
SYSTEM ADAPTION to USERS
… entails dynamic user modeling;
effective information architecture founded in practice, and
enhanced search features, such as
search integration
relevance feedback
PAIRI
SEARCHER ADAPTION to SYSTEMs through INTERACTION
… entails mental model building of systems and modification; through
Learning – leading to
Knowledge change, such as from an anomalous state of knowledge (ASK) towards
a coherent state of knowledge (CSK);
PAIRI
Adaption - Central DIMENSIONS:
throughintegration of
information objects
of the Information Retrieval system to the current searcher
to context and practice through
information architecture
of searchers to the Information Retrieval systems
PAIRI
PRESENTATION – PECHA-KUCHA
Present Introduction – 7 minutes
First Dimension: 20 slides x 20 seconds (7 min) Questions – interaction (10 minutes)
Second Dimension: 20 slides x 20 sec. (7 min)
Questions – interaction (10 minutes)
Third Dimension: 20 slides x 20 sec. (7 min) Questions – interaction (10 minutes)
Fourth Dimension: 20 slides x 20 sec. (7 min) Questions – interaction (10 minutes)
10 minutes vivid interaction on additional audience questions
Wrapping up (5 min)
PAIRI
QUESTIONS ON PAPER SLIPS
During presentations you may write questions to the panel (aside from oral questions!!)
’Collectors’ will collect the slips and hand over to Moderator
Moderator selects which questions to be posed during last 10 minutes of interaction
Moderator knows which questions will be dealt with by ensuing panellists.
PAIRI
Adaption through…
…integration of information objects
PAIRI
Adaptivity example: me!
Wrap it up –this is not
your topic…
PAIRI
Me – again!
Work web page
Images
Bibliography
PAIRI
Different ‘verticals’
PAIRI
How do we handle ‘verticals’ in libraries?
As separate silos!User
Catalogue .
Repositories Databases eJournals
JSTOR
Publisher 1
Publisher 2
eBooks
PAIRI
How to access the silos?
Information literacy! = educate users how to use each system:
Content, fields, indexing, search operators, interfaces…
But…
‘Why do we want to teach our users to be librarians?’
(Dave Pattern, Library Systems Manager)United Kingdom Serials Group 2009 Annual conference
PAIRI
Users don’t want silos!
‘Why is Google so easy and the library so hard?’
(Claire Duddy - student)
United Kingdom Serials Group 2009 Annual conference
PAIRI
Challenge
Each silo has its own set of metadata and standards developed for different purposes Very low common denominator…
PAIRI
Solution: Federated search?
(William H. Mischo, 2005)
PAIRI
Solution: Federated search?
“The jury is still out on federated search systems, even though more libraries now have them. There are murmurings that federated search has lower-than-expected use and may not be the magic search bullet we were led to believe” (Tenopir, 2007)
PAIRI
Solution: Integrated search?
Harvest all of the relevant data sources, normalize them into a single metadata schema, and index all of them together in one large union index
Add Google-like search box and ranking
(…a kind of federated search 2.0)
PAIRI
Solution: Integrated search?
Is being implemented in university and research libraries as we speak University of Huddersfield , UK Queens University, Canada State and Uni library, Denmark …
Several commercial products Summon by Serial Solutions EBSCO Discovery Service WorldCat Local ExLibris' Primo Central III Encore Synergy …
PAIRI
Solution: Integrated search?
But…
Development handed over to large commercial vendors
or giant web search services (Google; Amazon)
Where does that leave us as a field, as practitioners and researchers?
PAIRI
Research opportunities
There is sooo much we don’t know: How to optimise each type of information
object?
How to integrate information objects?
How to do relevance feedback in integrated search?
How to best involve users in testing?
etc etc…
PAIRI
Research opportunities
So, we can study various elements and details…
But how to generate real knowledge of these colossal integrated systems? Larger, more complex than ever
How to evaluate these scientifically, and not just rely on vendors?
PAIRI
Information Retrieval (IR) test collection
Purpose: to facilitate studies of integrating different document types
iSearch test collection
PAIRI
Three types of documents from physics
18,841 book records
291,244 metadata records, incl. abstract
143,569 full text articles
iSearch test collection
PAIRI
65 thoroughly described Information tasks1. Information need - What are you looking for?
2. Work task context - Why are you looking for this?
3. Knowledge state - Background knowledge of topic?
4. Ideal information - To solve your problem or task?
5. Search terms - Which search term would you use?
+ graded relevance assessments
iSearch test collection
PAIRI
What can we do with this? Study these tasks
Optimise each document type
Optimise the integration
Simulate relevance feedback and interfaces?
What we can’t do
Study another domain
Study real user interaction
iSearch test collection
PAIRI
Adaption through integrationof information objects
Pertinent issues How to design and evaluate systems that
successfully integrate genres, media and document types?
Even with shallow data?
How to generate real knowledge and evaluate these scientifically?
Without relying on what vendors offer?
PAIRI
Adaptation of …
information retrieval (IR) system to the user
PAIRI
TECHNIQUE EXPLICIT-O-SAURUS
PAIRI
USER-MODEL-O-SAUR
PAIRI
TECHNIQUE IMPLICIT-O-SAURUS
PAIRI
PAIRI
Schultz, C. K. (1968). H.P. Luhn: Pioneer of Information Science (p.32). London, UK: American Documentation Institute.
1950s: Luhn’s Selective Dissemination of Information
PAIRI
1960s: Salton, Lesk, Rocchio, Ide and Others
Ide, E. (1967). User interaction with an automated information retrieval system. In G. Salton (Ed.) Information Storage and Retrieval: Scientific Report No. ISR-12.
PAIRI
Oddy, R. N. (1977). Information retrieval through man-machine dialogue. Journal of Documentation, 33(1), 1-14.
1970s: Oddy’s Thomas
PAIRI
1980s: USER-MODEL-O-SAUR
Allen, R. B. (1990). User models: Theory, method, and practice. International Journal of Man-Machine Studies, 32, 511-543.
Rich, E. (1983). Users are individuals: Individualizing user models. International Journal of Human-Computer Studies, 51, 323-338.
“While the term ‘user model’ emphasizes the information about the person, it is obvious that a great deal of situational, task, or environmental information may be encoded in the model.”
PAIRI
Croft, W. B., & Thompson, R. H. (1987). I3R: A new approach to the design of document retrieval systems. Journal of the American Society for Information Science, 38, 389-404.
1980s: Intelligent IR
PAIRI
Maes, P. (1994). Agents that reduce work and information overload. CACM, 37(7), 30-40.
1990s: Agents
PAIRI
Maes, P. (1994). Agents that reduce work and information overload. CACM, 37(7), 30-40.
SYSTEM
USER
PAIRI
Belkin, N. J., Cool, C., Kelly, D., Lin, S.-J., Park, S.Y., Perez-Carballo, J., & Sikora, C. (2001). Iterative exploration, design and evaluation of support for query reformulation in interactive information retrieval. Information Processing & Management 37(3), 404-434.
And more relevance feedback
PAIRI
What Caused Explicit-O-Saurus’ Demise?
Users are unwilling to put forth the effort required to provide feedback
Users don’t have the additional cognitive resources to engage in explicit feedback
PAIRI
2000s: Implicit-O-Saurus
Click-through Dwell time Scrolling Query Behavior
PAIRI
Users are unwilling to put forth the effort required to provide feedback.
http://www.pewinternet.org/http://www.visualeconomics.com/how-the-world-spends-its-time-online_2010-06-16/
REALLY?
PAIRI
http://www.movielens.org/rateMorehttp://www.grouplens.org/
2010
In My Own Words
Interests
PAIRI
Users don’t have the additional cognitive resources to engage in explicit feedback.
Well, maybe back then …
PAIRI
We need to figure out better ways of eliciting feedback from users:
Better questions
Better measures
More creative
More engaging
More adaptive
Saving Explicit-O-Saurus
PAIRI
http://hunch.com/
PAIRI
Saving Explicit-O-Saurus
Questions
How can we create better questions and measures?
How can we make the process creative, engaging and adaptive?
Objection
Users are creatures of habit. Let’s reintroduce TECHNIQUE
EXPLICIT-O-SAURUS into the information ecology
Users will adapt …
PAIRI
Users’ Adaptability in IT
Environment
Adaptivity is a much needed system functionality, yes but
System alone cannot solve all the interaction problems
Humans are evolutionary learners
IT users must / can adapt to the system
PAIRI
Biological View
PAIRI
Psychological View
Mooers’ Law?
Satisficing principle
Today, majority IT users
Do not really have a choice at one level
Do have choices at another level
PAIRI
Cognitive View
Individual differences
Context/situation-based
Knowledge
Skills
Mental models
Tacit
PAIRI
Constructivist View
Problem-based learning
Social learning
• Observational learning specially relevant to building IT skills
• Knowledge sharing
• Collective intelligence
PAIRI
Adaptability — What is it?
AdaptAbility is a key
metacompetency
which enables
individuals quickly,
effectively respond
to unexpected
environmental
changes
PAIRI
Adaptability in Contexts
In workplace: job adaptability
In multicultural society: cross-cultural adaptability
In information environment: ITadaptability
PAIRI
Measuring Adaptability
Job Adaptability Inventory (JAI)
Cross-cultural Adaptability Inventory(CCAI)
Boyatzis-Kolb Adaptive Style Inventory
IT Adaptability – we need instruments
Adaptability – contributing factors?
PAIRI
Job Adaptability
Creative problem solving (new …)
Dealing Uncertainty/unpredictable work situations
Learning new tasks, tech, procedures …
Adapting culturally
Adapting interpersonally
Physical factors
PAIRI
Cross-Cultural Adaptability Inventory (CCAI)
A survey of 50 questions to uncover individuals’ current strengths and weaknesses within four critical skill areas proven necessary for effective cross-cultural communication:
Emotional Resilience
Flexibility/Openness
Perceptual Acuity
Personal Autonomy
PAIRI
Adaptive Style Inventory
• Acting situation• new• on time
• Deciding Situation• two alternatives• an opportunity
• Thinking• an idea• analyzing something
• Valuing• my feelings• see the world as the another
person sees it
--Kolb Learning Style Inventory?
PAIRI
IT Adaptive Behaviour
We need systematic research to understand factors underlying the IT adaptive behavior
We need instruments to measure IT adaptability (either adopt existing or develop new relevant one)
PAIRI
Where do we start?
Adaptive situations
IT system being updated
Need a function never used
New IT system
“I have used it before, but how do I get to it now”?
“The output doesn’t look right”!
“Where are my files I saw all of them just a moment ago?! Or, no, I cannot lose them ...”
PAIRI
Adaptive Performance
User’s context
IT Environment
User’s IT competency
User’s mental model
The task situation
PAIRI
AdaptAbility vs. LearnAbility
User adaptability calls for the system’s learnAbility
IT system learnability support user’s adaptAbility
learnability is an aspect of usability –rarely studied
PAIRI
ALPHA-IT
A (Adaptability)with
L (Learnability)for
P (Personalizing)&
H (Humanizing)
A (Adaptivity)in
IT (Information Technology)
PAIRI
Personalizing & Humanizing
Personalizing: Beyond user profiling
IT knowledge state
IT competency
Adaptive style
Humanizing : Beyond affordances
Demonstrate/encourage empathy
Affective responses
PAIRI
IT Affordance
Specifies the range of possible actions about an object (physical or digital)
Must be visible to the users to be usable
Individual differences in perceived affordances
When affordances fail users: an action does not result in an expected result --?
PAIRI
Building IT Adaptability as Social Learning
Sharing learning experience
Transferring knowledge
Collecting user problems
Banking user strategies
PAIRI
Study of User Adaptability & IT Design
How do we study users adaptive behavior
in adopting a new system, new applications, new functions, new …?
How can personal adaptive behaviors help
system design?
PAIRI
Adaption to context and practice
Focus on adapting information architecture (IA)
Focus on models and methods for developing tailored information architecture (IA)
PAIRI
Quick definition of IA (1)
ORGANISATION SYSTEM
LABELSYSTEM
Structural design
Combination of organisation, labelling,search and navigation
PAIRI
Quick definition of IA (2)
NAVIGATION SYSTEM
SEARCHSYSTEM
PAIRI
Adapting IA
Organization system, e.g.
Categories– tailored to user needs
Organization– organized to user perspective
Labels, e.g.
Terminology– jargon, language of youth, basic level
PAIRI
Adapting IA
Navigation system, e.g.
Tailored short cuts
Tailored links
Tailored recommending,
Search system, e.g.
Tailored filters
Tailored ranking
PAIRI
Adapting IA
PAIRI
System development process
(Rosenfeld & Morville, 2007, 232)
PAIRI
Research process
Context
UsersContent
Methods for data collection: e.g. interview, workshops, questionnaires, content analysis, log files
Mission, information behavior, policies, culture, technology, resources (time, economy, competences)
Content types, number, genre, usage, relation to work tasks
Target groups, information behavior, work tasks, search tasks, discourses, terminology
(Rosenfeld & Morville, 2007, 233)
PAIRI
Strategy process
(Rosenfeld & Morville, 2007, 269)
PAIRI
System development models
Plenty of models and approaches:
User-oriented
Domain-oriented
Work-task oriented
Interaction-oriented
Participation-oriented
…
All believe that we can grasp the context
PAIRI
System development models
Can we grasp, understand, model the context and practice – and adapt?
Are we“heroes” between
“victims” and “tyrants”?
(Spinuzzi, 2004)
PAIRI
Borger.dk – an example
Development of organisation system for Danish Government portal Borger.dk
PAIRI
Borger.dk – an example
Timeline
2008 Analysis, development, implementation
2009 Evaluation: usability tests, online survey and search log analysis
2009 Redesign and implementation of updated version and new features
PAIRI
Borger.dk – design methodology
Development phrase Survey of domain of public digital
communication
Personas – user needs and behaviour
Bench marking – related portals and organisations systems
Content analysis
Expert evaluation
Usability tests - 12 citizens completing 4 tasks
PAIRI
Borger.dk – design methodology
Evaluation phrase
Usability tests - 40 citizens completing 4 tasks
Online survey - 453 self-selected respondents
Search log analysis - Google Analytics over 11 month
PAIRI
Borger.dk - redesign
Information needs• 75% of needs met by categories
• Miss personal and factual information
Categories, organization, navigation• Unclear labels
• Short cuts - to frequent categories and forms
• Cross references
• Prioritized ordering
• Search • Synonym rings
• Best bets
Marianne Lykke & Brian Kirkegaard Lunn
PAIRI
Explanation of difficult concepts
PAIRI
More and tailored cross references
PAIRI
Best bets for important topics
PAIRI
??? Adaption to context ???
Do “heroes” exist within system design and adaption?
What models are feasible to adaption to context and practice?
What tools should be developed?
PAIRI
Last round of questions & Interaction!!
PAPER SLIP QUESTIONS FROM
YOU
PAIRI
WRAPPING UP the PANEL
Interesting discussions on:
Design and evaluation of integrated systems?
How to avoid re-engineering of vendor offerings?
Feasibility of explicit relevance feedback for systems adaptivity?
PAIRI
WRAPPING UP the PANEL - 2
Interesting discussions on:
How does personal adaption behaviorhelp systems (design)?
Research methdology and tools studying adaptive behavior to new ...?
How to adapt to context and work practice?
PAIRI
THANK
YOU