Transcript
04/12/23
Automated Retrieval and Generation of Automated Retrieval and Generation of Brain CT Radiology ReportsBrain CT Radiology Reports
Gong Tianxia
SOC NUS
04/12/23
Outline
Background Motivation Research Work Conclusion
04/12/23
Background
Computer Tomography (CT) has been used to examine the abnormality of human brain due to various causes
The result of each brain CT examination consists of: A set of CT scan image A report written by a radiologist
04/12/23
Abnormalities
Head traumas epidural hemorrhage(EDH) acute subdural hemorrhage (SDH_Acute) chronic subdural hemorrhage (SDH_Chronic) intracerebral hemorrhage (ICH) intraventricular hemorrhage (IVH) subarachnoid hemorrhage (SAH)
Fractures Edemas Others
Midline shift Etc.
04/12/23
Background
Brain CT Scans Samples
Normal EDH
04/12/23
Background
Brain CT Scans Samples
ICHSDH_Acute, SDH_Chronic,
Midline Shift
04/12/23
Background
Report
Unenhanced axial CT head was obtained. No previous study is available for comparison.
There is acute subdural haemorrhage overlying the left convexity & midline falx, which
measures up to a maximum of 1.4 cm in thickness. Subarachnoid haemorrhage is seen in
the sulci at the left fronto-temporal lobe, bilateral Sylvian fissure & cistern and the
basal cistern. Intraventricular extension of haemorrhage with blood seen in all four
ventricles is noted. There is intraparenchymal haemorrhage in the bilateral frontal lobes
raising the suspicion of haemorrhagic contusion. There is considerable mass effect with
midline shift to the right, generalised effacement of cerebral sulci and compression of
the left lateral ventricle. Prominence of the right temporal horn is suspicious for a
hydrocephalus. No skull vault fracture is seen in the CT scan.
04/12/23
Background
Comments
Acute left fronto-temporal-parietal subdural haematoma with bifrontal parenchymal
haematoma and bilateral subarachnoid haemorrhage with intraventricular extension.
Associated mass effect with midline shift to the right, compression of the left lateral
ventricle and generalised effacement of cerebral sulci. Hydrocephalus with right
ventricle dilated.
04/12/23
Motivation
Radiology reports contain rich information which is not used in many medical database systems
The proposed system is aimed to: Provide convenient search functions for radiology reports
and images Help doctors, radiologists, and medical informaticians to
gather needed information for their research Give references to radiologists to compare results Facilitate education systems for researchers, junior
doctors, and medical students Integrate medical records from various sources Provide platform for medical community to exchange
information and knowledge
04/12/23
Automated Retrieval and Generation of Brain CT Radiology Reports
Content-based Retrieval of CT Scan Brain Images
Two Research Directions
04/12/23
Information Extraction from Radiology Reports
Automatic Generation of Medical Reports Free Text Assisted Medical Image Retrieval
Related Work
04/12/23
Information Extraction from Radiology Reports
Automatic Generation of Medical Reports Free Text Assisted Medical Image Retrieval
Research Work
04/12/23
MedLEE: Medical Language Extraction and Encoding System
RADA: RADiology Analysis Tool Statistical Natural Language Processor for
Medical Reports
Related Work
04/12/23
MedLEE: Medical Language Extraction and Encoding System
04/12/23
RADA: Radiology Analysis Tool
04/12/23
Statistical Natural Language Processor for Medical Reports
04/12/23
An example of structured representation output
Statistical Natural Language Processor for Medical Reports
04/12/23
Negations Insufficient understanding of the text Ungrammatical writing styles Large vocabulary Assumed knowledge between writer
and reader
Challenges
04/12/23
Information Extraction from Radiology Reports
Automatic Generation of Medical Reports Free Text Assisted Medical Image Retrieval
Related Work
04/12/23
Most existing medical report automatic generation systems use the following “template filling” approaches:
Structured Data Entry Mail Merge Canned Text
Automatic Generation of Medical Reports
04/12/23
NLG: NLG is still premature application of medical
document generation There is still no system based on NLG principles in
routine use generates medical reports with fluent, concise and readable text
Challenges of NLG in general domain also exist in medical domain
Systems that automatically generate medical report from medical images are still lacking.
Challenges
04/12/23
Information Extraction from Radiology Reports
Automatic Generation of Medical Reports Free Text Assisted Medical Image Retrieval
Related Work
04/12/23
NeuRadIR: Web-Based Neuroradiological Information Retrieval System
Information Retrieval on MR Brain Images and Radiology Reports
Free Text Assisted Medical Image Retrieval
04/12/23
NeuRadIR
04/12/23
MRI Brain Image and Report Retrieval
04/12/23
Complexity of the system, as the system Consists of many functional components Needs knowledge from various research
areas
Challenges
04/12/23
Information Extraction from Brain CT Radiology Reports
Automatic Generation of Brain CT Radiology Reports
Radiology Reports Assisted Brain CT Images Retrieval
Research Areas
04/12/23
Information Extraction from Brain CT Radiology Reports
Automatic Generation of Brain CT Radiology Reports
Radiology Reports Assisted Brain CT Images Retrieval
Research Areas
04/12/23
Information Extraction from Brain CT Radiology Information Extraction from Brain CT Radiology ReportsReports
Our major task in this research area is to extract structured medical findings from the free text brain CT radiology reports
04/12/23
Input & OutputInput & Output
Input example:
An extra-dural haematoma overlying the right frontal lobe is seen measuring 1.2 cm in thickness.
Finding haematoma
type extradural
location overlying
brain_part lobe
orientation right
orientation frontal
thickness 1.2 cm
Output example:
04/12/23
System ArchitectureSystem Architecture
The system will have these componentsDocument ChunkerParserTerm MapperFinding ExtractorReport Constructor
04/12/23
Document ChunkerDocument Chunker
Decompose the radiology report into three sections Reasons for examination Detailed description of observations and
findings Comments or conclusion
We will focus on second and third sections, as they contain medical findings
04/12/23
ParserParser
Parse each sentence of a report and outputs a typed dependence tree
Parser output example:null:seen
nsubjpass:hematomadet:Anamod:extra-duralpartmod:overlying
dobj:lobedet:theamod:rightamod:frontal
auxpass:ispartmod:measuring
dobj:cmnum:1.2prep-in:thickness
Grammatical relation to parent word
04/12/23
Term MapperTerm Mapper
Maps words to standard forms specified in our medical knowledge source (Unified Medical Language System UMLS and other radiology thesaurus)
Reduces spelling variations
04/12/23
Finding ExtractorFinding Extractor
Apply semantic rules that are derived from semantic features of the words to translate the typed dependency relationship to logical relationship between findings and modifiers (finding’s attributes)
Merge the same finding from different sentences into one finding
Remove the redundant finding
04/12/23
Report ConstructorReport Constructor
Construct structured report according to findings, modifiers, and their logical relationship extracted from the finding extractor
04/12/23
Research AreasResearch Areas
Information Extraction from Brain CT Radiology Reports
Automatic Generation of Brain CT Radiology Reports
Radiology Reports Assisted Brain CT Images Retrieval
04/12/23
Automatic Generation of Brain CT Automatic Generation of Brain CT Radiology ReportsRadiology Reports
A traditional approach based on typical NLG system Content determination Discourse planning Sentence aggregation Lexicalization Referring expression
generation Linguistic realization
04/12/23
Content DeterminationContent Determination
Creates a set of messages from the features extracted from the new brain CT Images
Doctors use size, shape and location of the potential hemorrhage region to determine head trauma types
The system uses similar features for content determination: area, major axis length, minor axis length, eccentricity, solidity, extent, adjacency to skull, adjacency to background
04/12/23
Content DeterminationContent Determination
Image Segmentation
Features Extraction
04/12/23
Discourse PlanningDiscourse Planning
Uses Rhetorical Structure Theory (RST) to organize the text based on relationships that hold between parts of the text
04/12/23
Sentence AggregationSentence Aggregation
Groups messages together into sentences and paragraphs
04/12/23
Sentence AggregationSentence Aggregation
Groups messages together into sentences and paragraphs
04/12/23
LexicalizationLexicalization
• Decides which specific words and phrases should be chosen to express the domain concepts and relations which appear in the messages
• Uses hardcoded specific word and phrases to standardize the output language radiology reporting
• Uses NLG system to generate radiology reports of various writing styles to cater different user groups (at later stage of our project)
04/12/23
Final StepsFinal Steps
Referring Expression Generation Linguistic Realization
04/12/23
A Machine Learning ApproachA Machine Learning Approach
Based on the concept of statistical machine translation
Image and report are two representations of the same medical condition
In a sense, image and text are two different languages
04/12/23
Statistical Machine TranslationStatistical Machine Translation
Foreign/English
parallel text
Englis
h text
Statistical analysis Statistical analysis
Translation Model Language Model
Decoding Algorithm
04/12/23
Syntax Tree Based SMTSyntax Tree Based SMT
IP
VP
BA
NN VP
W PN
把 钢笔 给 我
Give the pen to me
DT NN TO PRP
NP PPVB
VP
04/12/23
Report Generation based on Report Generation based on SMT conceptsSMT concepts
04/12/23
Research AreasResearch Areas
Information Extraction from Brain CT Radiology Reports
Automatic Generation of Brain CT Radiology Reports
Radiology Reports Assisted Brain CT Images Retrieval
04/12/23
Radiology Reports Assisted Brain CT Radiology Reports Assisted Brain CT Images RetrievalImages Retrieval
04/12/23
Radiology Reports Assisted Brain CT Radiology Reports Assisted Brain CT Images RetrievalImages Retrieval
04/12/23
Project StatusProject Status
Project Funding Sources University Research Grant Ministry of Education Academic Research Grant
Project Collaborators School of Computing, NUS National Neuroscience Institute Institute for Infocomm Research
Project Phases Phase I: Pilot Study (Feb 2007 – April 2008) Phase II: R&D (April 2008 – Mar 2011)
top related