A Survey of REB Systems in Canada (CAREB AGM Montreal 2014) Michael Wilson (Research Ethics Officer) & Dr. Brian Detlor (Chair) McMaster Research Ethics.

A Survey of REB Systems in Canada(CAREB AGM Montreal 2014)

Michael Wilson (Research Ethics Officer) &Dr. Brian Detlor (Chair)

McMaster Research Ethics Board (MREB)

The Access Database

Year 2000 Year 2004

MREB Active – Master Paper Record

Born to Scan

Sharp 2700N copier / scanner

Close to my office

Archiving and emailing PDFs

Program or Be Programmed!

Coldfusion – Life is but a Dreamweaver

On your mark…choose a vendor…

UK – National Research Ethics Service

So Many REB Systems?

One software system for UK research ethics review – developed by Infonetica

I developed, designed & programmed my own REB system

RFPs for larger systems i.e, > $100,000

IT projects have a high 40% - 70% failure rate

In-house system – or what happens if your developer leaves?

Our REB systems survey – what didn’t we know?

REB Systems in Canada Survey

Purpose of the Research Project

To obtain a descriptive snapshot of the human-participant REB systems landscape in Canada.

REB systems in Canada implemented? Key characteristics of successful REB systems? Satisfaction of stakeholders with REB systems? What works well & what could be improved? Context of systems & influences how fared? Recommendations on implementing REB systems

Motivation

Our own “home-grown” REB system is at its maximum capacity We need a more robust system

Handle increases number of protocols Automate many tedious, monotonous tasks Better reporting capabilities Expedite the review of applications Eliminate the “master” paper record Better security / more reliable servers/ backups &

redundancy Better track revisions Provide researchers with a better experience

Creating protocols; obtaining status updates on the review process; submitting “change requests”

Motivation

For a new and improved system we want to keep Design our forms and workflow in a way that works for our

Board Control changes system and make them quickly Low cost

DeLone & McLean Model of Information System Success

Methods

Web survey sent CAREB members via CAREB listserv before & after Calgary AGM May 2013

Recruitment in person in Calgary

Survey instrument (Limesurvey on secure McMaster server) Quantitative and qualitative questions on REB background,

characteristics of REB system, user satisfaction, important REB system features Probing questions about the respondent’s REB system A very rich data set was collected – 64 complete

responses Many detailed qualitative comments

Respondent Profile

Number of REBs per Institution

Type of Institution

One REB: 42%

> One REB: 58% 7%

67%

10%

2%

5%

9%Hospital basedUniversity basedUniversity and hospi-tal basedGovernmentPrivate or Inde-pendentIndustry basedCollege based

Respondent Profile

Predominant Role of Respondent

14%

64%

9%

14%

REB ChairREB Administrator/Staff MemberREB MemberResearch Director or Administrator

Type of REB System Currently Used

Current REB System

3%17%

28%

11%

45%

2%

34%

2% A stand alone ASP/Cloud sys-temA system integrated with my institution's Research Services systemsAn in-house developed systemAn outsourced system devel-oped by commercial (outside) vendorsA hybrid paper and electronic systemA regional REB system within a larger shared ASP/Cloud system A mostly paper-based system, with a desktop databaseA paper only system

Current REB System

Amount of Time Using the Current REB System

1 - 3 years 3 - 5 years 5 - 8 years More than 8 years

Don't know

No answer 0%

5%

10%

15%

20%

25%

30%

35%

What are the key characteristics of a successful REB system? Answers in rank order by frequency

1. Useful functionality (45 comments) E.g., generates reports; generates reminders; generates

statistics; creates meetings and agendas; facilitates robust and easy searching; tracks versions of documents; check status of a protocol

2. Efficient and meaningful processes and procedures (25 comments)

E.g., workflow meets the needs of all users; processes are streamlined; facilitates quick processing of protocols

3. Easy to use / User-friendly (24 comments) Intuitive forms; smart forms; minimal form navigation

What are the key characteristics of a successful REB system? Answers in rank order by frequency (cont’d)

4. Sufficient support (18 comments) E.g., adequate number of admin staff; competent admin

staff; knowledgeable REB members; adequate IT support in terms of staff and infrastructure

5. Captures all information needed (9 comments)

6. Automates tasks wherever possible (9 comments)

7. Flexible (8 comments)

8. Integrates with other institutional systems (7 comments)

9. Provides online access and electronic submissions (4 comments)

10. Follows good project management practice (2 comments)

How well do these characteristics compare with what’s in place? Not as well as one would like...

Functionality offered by the current REB system Permits researchers to submit an electronic application (protocol)

44% Handles annual renewals and amendments electronically 36% Handles serious adverse events reporting 25% Has an online reviewer system 30% Tracks applications through the review process 30% Serves as the master record 45% Permits data extraction for internal report purposes 41% Creates online statistics 20% Creates meetings, agendas, reports, reminders 28% Permits electronic signatures or variations of such 17%

How satisfied are respondents with their current REB systems? Most are satisfied

Not Very Satisfied Satisfied Very Satisfied0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

How satisfied do you think REB staff are with your current REB system? (n = 54)How satisfied do you think re-searchers are with your current REB system? (n = 48)

How satisfied are respondents with their current REB systems? Twice as many negative comments given than positive

ones

Negative comments (in rank order) concerned Usefulness (23 comments) Flexibility (10 comments) Support / Infrastructure (8 comments) Ease of Use (5 comments)

Positive comments (in rank order) concerned Usefulness (9 comments) Support / Infrastructure (9 comments) Flexibility (1 comment)

Observations

“Satisfied” or “Very Satisfied” respondents utilized a wide variety of REB systems No single type of REB system determined

satisfaction Satisfied respondents utilized the full gambit of REB

systems expensive solutions & in-expensive solutions, vendor-based solutions & custom in-house solutions, paper-based solutions, desktop database solutions, &

full electronic solutions Why? What does this imply for practice?

Observations

Why does no one type of REB system determine satisfaction? Satisfaction with an REB system seems to depend on the

context in which they are situated Critical factors include:

The number of protocols processed by the REB The greater the # of protocols, the more likely a full robust

REB system was needed The expectations of researchers and staff (e.g., lower

expectations lead to greater satisfaction) The amount of support available (both REB staff and IT support)

The more support available to handle administrative work and IT concerns, the greater the satisfaction

The willingness of an institution to supply this support is critical

Observations

Certain factors can undermine satisfaction with implementing an expensive vendor-based REB solution Factors that inhibit satisfaction include:

high cost loss of control making timely changes to the

system loss of prior functionality, absence of needed

functionality, inclusion of unwanted functionality using processes and workflows that do not match

the way the institution works REBs should ensure that these factors are addressed prior

to implementation However, this is not always possible to do

Other thoughts

Though electronic REB systems offer many benefits, they are, at the end of day, still just administrative systems They tend to offer more efficiency benefits (e.g.,

automated processes; online submissions; quicker tracking; easier reporting) than effectiveness benefits (e.g., better quality reviews)

“Researchers, reviewers and ethics staff are spared clerical, paper-heavy administriva in favour of more time and energy to focus on content. However, online systems will not compensate for poorly advised researchers, reviewer shortages or poorly informed ethics staff.”

“REBs and administrators focus a lot on improving efficiency, but I’d like to explore a system that improves effectiveness of REB processes.”

Other thoughts

There is a desire for standardized software and harmonization of software

“There should be a coordinated effort to have all REBs using the same software.”

“Canada needs to develop one system only for multi-university research projects so that researchers are not being asked to fill in one for each university they have as participants.”

Change management is critical when implementing a new REB system

“There is an incredible amount of time involved from REB administrators to set up an efficient system (consulting, adapting system and policies and procedures, training etc.) that I think is often underestimated by senior administration. Managing change alone can be an enormous task and plans have to be in place to support implementation.”

Recommendations?

Get your REB systems / IT project in writing – have a Project Charter signed by key stakeholders at the highest level

Look in-house first – your organization may have the talent and resources

Canadian REB system vendors and developers know the Canadian landscape

A web-based REB system solution reduces data entry, paper usage, improves efficiency & maximizes effectiveness

Integration with other internal systems is nice – but first have a system that first works for your REB, not some other office

Questions?

Detailed Survey results online and in Handout here

https://reo.mcmaster.ca/REBsurvey

Later: E-mail us

Journal publication forthcoming ...

Michael [email protected]

Brian [email protected]