Unified Vocabulary Review Process

Unified Vocabulary Review Process

Aug 8, 2007

Martin Ringwald

The Jackson Laboratory

[email protected]

Jim Cimino

Columbia University

[email protected]

Agenda

• Background

• Group Charter

• The Process

• Vocabulary Criteria checklist

• Discussion

Team Members

• Jim Cimino M.D. Columbia University

• Martin Ringwald, Ph. D. The Jackson Laboratory

• Terry Hayamizu, M.D., Ph. D. The Jackson Laboratory

• Grace Stafford, Ph. D. The Jackson Laboratory

Vocabulary Standards

What does it mean to be a “caBIG Designated Standard Vocabulary”?

• In mentoring development projects, developers will be encouraged to use these vocabularies for concepts and value domains

• In silver level reviews, use of these vocabularies for concepts and value domains will ease reviews

• caBIG-wide, re-use of the same standard vocabularies will increase interoperability

Background

Milestone Events• 2005-2006: VCDE WS participants work to outline important

aspects to consider for Vocabulary Evaluation Criteria• Summer 2006: VCDE WS establishes many general and some

specific criteria in:• Understandability, Reproducabiliy, and Usability (URU)• Documentation• Maintenance and Extensions (Change management)• Accessibility and Distribution• Intellectual Property Considerations• QC and QA• Concept Definitions• Community Acceptance• Reporting Requirements

• Fall 2006: VCDE WS decides to let experts try and use criteria to evaluate three Vocabularies: GO, NCI Thesaurus and CTCAE.

Background (cont)

Milestone Events

•Sept-Dec 2006: Terry Hayamizu and Martin Ringwald Evaluate GO

• https://gforge.nci.nih.gov/projects/vocabcriteria/•Sept-Dec 2006: Olivier Bodenreider Evaluates NCI Thesaurus

• https://gforge.nci.nih.gov/projects/evalncit/•Nov2006-Feb 2007: Jim Cimino Evaluates CTCAE

• https://gforge.nci.nih.gov/projects/evalctcae/•April 5 and April 19th: Presentation of 3 reviewed vocabularies and Vote as Pass/Fail as caBIG “standard Vocabularies.

Group Charter

• https://gforge.nci.nih.gov/projects/univocab/

• July-Sept 2007 : Jim Cimino, Martin Ringwald, Terry Hayamizu, Grace Stafford: “Unify” vocabulary review process (based on GO, CTCAE and NCI Thesaurus reviews)

• Modify existing Criteria list

• Suggested and Required Criteria

• Develop Standard Operating Procedure (SOP) for group review of vocabularies

• Suggest List of Vocabularies for review

• Sept – December 2007 : review one more vocabulary with small group in VCDE WS

• VCDE WS participants to serve as reviewers

• Serves as a “Training the trainers”

• January 2008: Use lessons Learned to Re-evaluate Criteria List and SOP

Standard Vocabulary Submission and Review Process

• preliminary proposal

• adapted from a draft developed by the VCDE WS

• modified based on lessons learned during the EVRC projects (GO, CTCAA)

Standard Vocabulary Submission and Review Process

Key Recommendations

• Where possible, coding systems should be converted to terminologies prior to evaluation

• Terminology evaluator(s) should be provided with a definitive, electronic version of the terminology

• Vocabulary developer / submitter should provide detailed information and documentation about the terminology to expedite the vocabulary review process

• Vocabulary Review Criteria (VRC) checklist should detail the required information and be used

• by the vocabulary submitter as a template in order to provide the necessary

information

• by the reviewers to manage additional information collected during the review process and to record their assessments

Standard Vocabulary Submission and Review Process

A. Formal Submission of Vocabulary

1. Submitter of Vocabulary announces intention to submit vocabulary to the VCDE WS Lead, and schedules presentation at Bi-Weekly VCDE WS Teleconference.

2. Submitter of Vocabulary submits artifacts for review.Vocabulary Standard Submission Package will include:

a) Electronic version of vocabulary or pointer to site for download

b) Pointer(s) to tools available to use/browse the vocabulary

c) Documentation that describes the terminology

d) Pointers to relevant publications including peer-reviewed publications that evaluate the terminology

e) Completed VRC checklist.

f) Justification of why the vocabulary is important as a caBIG standard.

3. Submitter of Vocabulary makes a formal presentation at Bi-Weekly VCDE WS

Teleconference

Standard Vocabulary Submission and Review Process

B. Appointment of Small Review Group

1. VCDE WS Lead assembles group to be composed of:a. Authorities in field of biomedical vocabularies, including appropriate NCI experts.

b. Biomedical domain experts familiar with the vocabulary in question and/or the requirements of the community in which the vocabulary will be used.

2. Leader of Review Group is designated.a. Group leader reviews submission package.

b. If package is determined to be inadequate, group leader contacts vocabulary submitter to get additional artifacts.

c. Group leader organizes and convenes meeting of review group.

Standard Vocabulary Submission and Review Process

C Review by Small Review Group

1. Group members receive all artifacts, including a completed VRC Checklist.a. Members review checklist and accompanying artifacts.

b. If additional information or analysis is thought to be required, members initiate steps to obtain the necessary data.

c. Members use the VRC Checklist to store and organize information collected.

2. Group members evaluate vocabulary.a. Small group review proceeds for 2 weeks.

b. Group presents preliminary results at Bi-Weekly VCDE WS Teleconference.

3. Small group review proceeds for an additional 2 weeks.a. Group augments and refines the vocabulary assessment, analyzes the results and makes a

determination regarding validation and approval of the vocabulary.

b. Group presents final results at next Bi-Weekly VCDE WS Teleconference.

c. Review should include a narrative report and the completed VRC Checklist

Standard Vocabulary Submission and Review Process

Additional notes regarding the Evaluation Process:

For some criteria, such as those regarding the documentation itself, reliance on information provided by the vocabulary submitter is probably sufficient to demonstrate whether the terminology meets the criterion.

For other criteria, the information provided by the submitter is not necessarily sufficient to prove that a given criterion is fulfilled by the vocabulary, and this may need to be evaluated further.

For some criteria, assessment of the vocabulary might be amenable to quantitative and / or automated evaluation. This approach should be pursued if possible, and if it augments and speeds up the evaluation process.

In most cases, methodical spot-checking of the terminology, in conjunction with thorough evaluation of the documentation, will probably be sufficient to determine whether or not the vocabulary fulfills a given criterion.

Standard Vocabulary Submission and Review Process

D. Review by the VCDE WS

1. Following presentation of preliminary results by the Small Review Group, VCDE WS discusses group’s work so far.

2. Following subsequent presentation by Small Review Group, VCDE WS discusses group’s final assessment.

3. VCDE WS makes decision whether to:a. Validate and approve the vocabulary, orb. Send the vocabulary back to the submitter for more work.

Vocabulary Checklist

• Criteria Categories

• Criteria Requirement Levels

• Overview of Specific Recommendations

Vocabulary Checklist: Criteria Categories

A. Structure – criteria related to the data model of the terminology

B. Content – criteria related to the information contained in the terminology

C. Documentation — criteria related to information available about the terminology

D. Editorial Process - criteria related to the activities involved in designing, creating, distributing and maintaining the terminology

Vocabulary Checklist: Requirement Levels

• Fulfillment should be required

• Fulfillment is recommended

• May not apply to all terminologies

• Fulfillment may be difficult to assess

Vocabulary Checklist: Specific Criteria

Vocabulary Checklist: Structure Criteria (1)

A.1. Concept orientation –Is terminologic information organized around meaning of terms?

A.2. Concept permanence - Is the meaning of a concept, once created, inviolate and does the data model accommodate name changes and retirement?

A.3. Nonsemantic concept identifiers - Does each concept have a unique identifier that is free of hierarchical or other implicit meaning and are not re-used?

A.4. Polyhierarchical organization - Is it allowed? Is it appropriate?

A.5. Graceful evolution - How are updates applied to the content?

A.6 Explicitness of relations – Are the meanings of inter-term relations explicit?

Vocabulary Checklist: Structure Criteria (2)

A.7. Multiple granularities - If the terminology is intended to serve multiple purposes, does it provide different levels of granularity appropriate for the different purposes?

A.8. Multiple consistent views - If the terminology is intended to serve multiple purposes, does it provide multiple views suitable for the different purposes?

A.9. Formal definitions - Does term representation provide a definitive set of relationships to other concepts that, taken together, are both individually necessary and collectively sufficient to distinguish the concept from all other concepts?

A.10. Recognition of redundancy - Is the structure sufficiently rich to support detection of redundant meaning?

A.11 Extensibility - Does the structure avoid imposing limits on the ability of the terminology to cover the domain? (e.g the decimal hierarchical codes of ICD9-CM)

Vocabulary Checklist: Content Criteria (1)

B.1. Content coverage - Does the terminology provide comprehensive or explicit in-depth coverage of the domain of interest it claims to address as stated in purpose and scope of the terminology segment?

B.2. Polyhierarchy - If it is allowed and appropriate, is it used? – That is, is every term in all the classes to which it should belong?

B.3. Rejection of NEC terms - Are "not elsewhere classified" (NEC) and "other" terms avoided? Does the terminology provide a way to represent information not explicitly covered in the terminology?

B.4. Context representation - Does the terminology provide formal, explicit information about how concepts are used?

Vocabulary Checklist: Content Criteria (2)

B.5. Textual Definitions - Does the terminology provide a clear textual definition of each term in the terminology and are the textual definitions sufficient to distinguish the meaning of each concept from other concepts in the terminology?

B.6 Formal Definitions - Does each term in fact have a definitive set of relationships to other concepts that, taken together, are both individually necessary and collectively sufficient to distinguish the concept from all other concepts?

Vocabulary Checklist: Documentation Criteria

C.1. Purpose and scope - Is the purpose and scope of the terminology clearly stated in operational terms so that its fitness for particular purpose can be assessed and evaluated?

C.2. Statement of indended use - Is there a statement of the terminology's intended use, intended users and scope?

C.3. Documentation descriptions - Does the available documentation describe terminology structure and organizing principles, use of concept codes/identifiers, use of semantic relationships, output format(s)

C4. Version documentation - Are new versions accompanied by adequate documentation that describes how the new version differs from the one it replaces?

C.5 Tool documentation - Is there a description of methods or tools for acquisition and application of the terminology?

Vocabulary Checklist: Editorial Criteria (1)

D.1. Process for maintenance and extensions - Does the editorial process enable changes for 'good' reasons and discourage change for 'bad' reasons, and does it maintain concept permanence while correcting recognized redundancy?

D.2. Quality Assurance and Quality Control - Are there internal checks to detect and eliminate errors in modeling and/or editing, is there a process for review by independent experts from the field in which the terminology will be used, and is there a process in which the terminology developer can improve the terminology in response to the findings and recommendations of the review?

D.3. Methods for extending the terminology - Is the terminology evolving to maintain domain coverage?

Vocabulary Checklist: Editorial Criteria

D.3. Methods for extending the terminology - Is the terminology evolving to maintain domain coverage?

D.4. Organization criteria - Is maintenance of the terminology a core part of the organization’s business?

D.5. Extensions to other terminologies - If the terminology extends or overlays other terminologies, do they have a formal methodology for expanding content?

D.6. Availability of lists of concepts, terms and definitions - Is the terminology included in an EVS-type terminology server? If this is not possible, then flat files (such as used by the UMLS) should be available.  

D.7. Methods and tools for acquisition and application - Is the terminology freely available for download in a format(s) (e.g. RRF, OWL, XML, OBO) that can be readily used by the community? Has an effective user interface been built? Is there support for computer interface and system implementers?

Vocabulary Checklist: Editorial Criteria

D.8. Intellectual Property Considerations - Is the terminology available to all classifications of users (e.g. government agencies, for-profit and not-for-profit institutions, academia, private citizens, etc.), without fee, permission requirement, or restrictions?

D.9. Community Acceptance - Has a scientific community accepted the terminology as a de facto standard?

D.10. Reporting Requirements - Has a health regulatory body required this terminology for reporting? If so, which one(s)?

D.11. Editorial Process - Is there evidence of a thoughtful editorial process, carried out by experts in the domain of interest and terminology representation, ideally with input from potential users of the terminology?

D.12. Mechanisms for accepting and incorporating external contributions - these include error reporting and requests for additional content

Discussion Points

• Does Bronze, Silver and Gold apply to Vocabulary Standards?

• Which vocabularies to evaluate?

• Other reactions to critera or processes?

Next Steps

• Develop Standard Operating Procedure (SOP) for group review of vocabularies

• Suggest List of Vocabularies for review

• Sept – December 2007 : review one more vocabulary with small group of VCDE WS participants to serve as reviewers; serves to “training the trainers”

• January 2008: Use lessons Learned to Re-evaluate Criteria List and SOP