Biased Information Retrieval in Pharmaceutical Drug Development

Biased Information Retrieval in Pharmaceutical Drug Development

ICIC - 20th October 2015

Kasper Højby Nielsen

[email protected]

Information scientist, team leader

Novo Nordisk A/S, Denmark

mailto:[email protected]

Background: Medicine, human biology 1993

Profession: Academic librarian 1995

-> information scientist 2006

-> information consultant 2015

I have performed hundreds if not thousands of searches!

Who am I?!

20-10-2015ICIC 2015: Biased Information Retrieval 2

Library

ICIC 2015: Biased Information Retrieval 20-10-2015 3

Knowledge Centre


An information scientist combines

• Field knowledge (scientific field)

• Business understanding

• Library field insight and experience

and acts as an internal information consultant:

Global Information & Analysis in Novo Nordisk


• I am not an IT specialist

• I am an advanced user of IT systems/databases

• I have performed a study of vendor search deliveries

Why am I invited to ICIC?


• Information searching

• Bias in information retrieval

• Investigation objectives

• Methods

• Results

• Conclusions and recommendations

Agenda


Oxford Dictionary:

Search (for somebody/something):

1. An attempt to find somebody/something, especially by looking carefully for them/it

2. An act or the activity of looking for information in a computer database or network

What is a search?


• IT perspective

• Information scientist perspective: Searching for data in bibliographiesA combination of queries leading to a filtered result

What is a “search”?


What is a “search”?


1_: ((SAFETY OR (ADVERSE ADJ EVENT$1)) OR 1386359 docs

(ADVERSE ADJ REACTION$1)) OR (ADVERSE

ADJ EFFECT$1)

2_: ((SUSAR$1 OR (SIDE ADJ EFFECT$1)) OR 969051 docs

(DRUG ADJ REACTION$1)) OR (DRUG ADJ

EFFECT$1)

3_: COMPLICATION$1 1417042 docs

4_: NOVOMIX$1 OR NOVOLOG$1 OR NOVORAPID$1 9939 docs

OR LEVEMIR$1 OR LANTUS$1 OR ASPART$1 OR

GLARGINE$1

5_: (DETEMIR$1 OR ((BIPHASIC ADJ INSULIN) 6216 docs

ADJ (LISPRO$1 OR ASPART$1))) OR

(INSULIN ADJ ANALOG$4)

6_: HUMALOG$1 512 docs

7_: 1 OR 3 OR 2 3210368 docs

8_: 6 OR 5 OR 4 12349 docs

9_: 8 AND 7 6151 docs

10_: "2014".PY. 4729501 docs

11_: 9 AND 10 1143 docs

((SAFETY OR (ADVERSE ADJ EVENT$1)) OR (ADVERSE ADJ REACTION$1)) OR (ADVERSE ADJ EFFECT$1) 1386359 docs

What is a search?


What is a good search?


It all depends!



Standardisation? Building the search profile



SafetyAdverseeventAdverse reactionAdverse effectAESAE…

DrugInsulinanalogueModern insulinProductLevemirDetemirLantus Glargine…

ChildrenChildPaediatricTeenagerTeen…

What is a good search? Intersection


“Please give me everything about diabetes”

Results: 498.147 (PubMed as of 11. October 2015)

“Please everything about diabetes mellitus”

Results: 380.259 (PubMed as of 11. October 2015)

What is a good search query?


Biased information retrieval


Sources consulted Search methodology

Many ways to perform a search

Are search results from different vendors alike – how much do they differ?

What is the impact of strengthening the interaction between customer and vendor?

Had something like this been done before? Not to my knowledge

Study to compare vendor results


It is hypothesised that a potential difference in response from third party providers to identical literature search requests can be avoided or at least substantially reduced by strengthening the communication and feedback between requester and providers.

Hypothesis


Methods - highlights

• Preparations

• Field study

• Processing of data

• Analysis

Slide no 2020-10-2015ICIC 2015: Biased Information Retrieval

Methods - preparations

• Preparations of research questions

• Contact to a set of information providers – 6 in total

• Blinded (did not know about the project)

• The providers were paid for their services

• Novo Nordisk could use the data

• International

• Public/Private

• Same communication to all vendors



Literature search request no. 1

Email:

I would like a comprehensive literature search performed on the following subject – preferably within 5-10 working days. Would this be feasible?

In Systemic Lupus Erythematosus (SLE): What is the frequency of different comorbid diseases (mortality related) and do you find the occurrence related to disease severity? Gender differences? Please attach the reference list (including abstracts) as well as the search criteria used including databases searched.



Literature search request no. 2:

Email:

Please provide a comprehensive literature search covering the following questions within the next 5-10 working days:

What is the frequency of comorbid diseases in Rheumatoid Arthritis (RA) regarding cardiovascular events and cancer? Is there a difference in occurrence of the above between the following subpopulations: MTX-naïve patients? MTX-IR RA patients? TNF-IR patients? (IR=inadequate response).

Please attach the reference list (including abstracts) for the recent 5-10 years as well as the search criteria used including databases searched.

Please contact me via email with any questions you may have for this query.


Methods - Field study

• Interaction with the information providers

• Search no. 1: Minimal interaction

• Search no. 2: Attempts to increase communication

• Email correspondence

• Providers kept blinded

• Reception of responses/results

ICIC 2015: Biased Information RetrievalSlide no

2420-10-2015

Methods - Processing of data

• Re-retrieval of references (intra-provider duplicate exclusion)

• Transfer to Reference Manager (citation handling system)

• Duplicate determination (overlap)

• Relevance review

• Vendor search methodology review


No or very little response/interaction

Vendor behaviour


Results

Responses from providers (search no. 1)

Slide no 27

Provider no.

No. of referencesPY = 2009-2013

Total no. of references

1 64 123 (no PY limitation)

2 94 94 (2009-2013)

3 37 (2012-2013) 37 (2012-2013)

4 75 77 (2008-2013)

5 67 109 (2004-2013)

6 133 252 (no PY limitation)

Total 472 692

20-10-2015ICIC 2015: Biased Information Retrieval

Results – Overlap search no. 1


Results – Overlap search no. 2

ICIC 2015: Biased Information RetrievalSlide no

2920-10-2015

Most important findings:

Overlap: Search no. 1: 35% (provider 4 and 5)

2% (provider 1 and 2)

Search no. 2: 24% (provider 5 and 6)

5% (provider 2 and 6)

None of the references were identified by all providers

Results


Core article


• The hypothesis was rejected: Attempts to increase interaction between customer and providers did not lead to an increase of overlap between provider results

• There is a risk of introducing information retrieval bias

Conclusions


• Decision makers´ perception of information research

• Outsourcing?

1. Lack of understanding (business understanding/query understanding)

2. Lack of communication initiative

3. No use of advanced search methodology

Concerns


1. Understand the information search request in detail

2. Establish a search strategy

3. Generate a search profile

4. Initiate the search as an iterative process

5. Combine various search methods

6. Accept large sets of search results

7. Allow for time and allocate resources to be able to filter down and analyse the search results for relevance

8. Consider if it is possible to apply an advanced analytical tool like e.g. text mining in order to analyse large sets of data

Recommendations


Summary


Thank you!

Questions?


• Who were the vendors?

• Are the results published?

• Case: Ph.d. literature search course

Back up slides


Provider number

Number of retrieved articles

n

Provider contacted requester with clarifying questions

Yes/No

Number of relevant articles

n

Ratio of relevant articles

%

Ratio of potential relevant articles

%

Ratio of irrelevant articles

%

1 123 N 69 56 24 20

2 94 N 46 48 29 23

3 37 Y 26 68 21 11

4 77 N 39 51 27 22

5 109 N 52 48 20 32

6 252 N 89 35 23 42

Relevance ranking for search 1 and 2



Provider no.

Used database keywords (subject headings)

Yes/No

S1 S2

Used database major subject headings

Yes/No

S1 S2

Used database subject subheadings

Yes/No

S1 S2

Used truncation

Yes/No

S1 S2

Usedarticle titlesearching

Yes/No

S1 S2

Used proximity operators

Yes/No

S1 S2

1 N N N N N N Y Y N N N N

2 N N N N N N Y Y Y Y Y Y

3 Y Y Y N N N Y Y Y Y Y Y

4 Y Y Y Y Y Y N N Y Y N N

5 N N N N N N Y Y N N Y Y

6 Y Y N N Y Y Y Y Y Y N N

Search methodology


Biased Information Retrieval in Pharmaceutical Drug Development

Internet