Semantic Natural Language Understanding with Machine Learned Annotators and Deep Learned Ontologies at Scale: Spark Summit East talk by David Talby

NATURAL LANGUAGE UNDERSTANDING WITH MACHINE LEARNED ANNOTATORS & DEEP LEARNED ONTOLOGIES AT SCALE

David TalbyPh.D., MBA, CTO @ Atigeo

The problem

Whoneedstobevaccinated?

Whofitsthisclinicaltrial?

Whoisatriskforsepsis?

Whoisgettingmedsthey’reallergicto?

Whoonthisprotocoldidnothavethisside

effect?

At the beginning, there was search

Scalable&robustIndexingpipelineTokenizers&analyzersSynonyms,spellers&Auto-suggestFileformats&headerboostingRankers,link&reputationboosting

Then there was semantic search“cheap red prom dresses”“laptops under $500”“italian restaurants near me that deliver”“captain america civil war tonight”“nba scores”

DictionaryBasedAttributeExtraction

Dell - XPS 15.6 4K Ultra HD Touch-Screen Laptop - Intel Core i5 - 8GBMemory - 256GB Solid State Drive -Silver

MachineLearnedAttributeExtraction

If you go for the ambience, you'll bedisappointed. If you go for good,inexpensive and authentic Mexicanfood, then you're in the right place.

Then, you need to understand languagePrescribing sick days due to diagnosis of influenza. Positive

Jane complains about flu-like symptoms. Speculative

Jane may be experiencing some sort of flu episode. Possible

Jane’s RIDT came back negative for influenza. Negative

Jane is at high risk for flu if she’s not vaccinated. Conditional

Jane’s older brother had the flu last month. Familyhistory

Jane had a severe case of flu last year. Patienthistory

1.

Language gets complex

and domain specific

Human language is wonderfully nuancedJoe expressed concerns about the risks of bird flu. Nothing

Joe shows no signs of stroke, except for numbness. DoubleNegative

Nausea, vomiting and ankle swelling negative. Compound

(itgetsworse– inrealityalotoftextisn’tvalidEnglish)

Patient denies alcohol abuse. Speculative

Allergies: Penicillin, Dust, Sneezing. Compound

Let’s build this!

Theinput(patientrecords)

Theprocessingframework

Theoutput Thequeryengines

SENTENCEDETECTION

SECTIONDETECTION

TOKENIZER LEMMATIZER

STOPWORDREMOVAL

NEGATIONDETECTION

CONDITIONALSCOPE

SPECULATIVESCOPE

DATE NUMBER UNIT QUANITITY

CONCEPTEXTRACTION

2.

you’ll need

machine learning early

Machine learned annotators

GrammaticalPatterns

If…then…

DirectInferences

Age<18==>Child

Lookups

RIDT(labtest)

Under-diagnosedconditions

FluDepression

ImpliedbyContext

relevantlabsnormal

Sometimes,it’seasiertojustcodeanannotation’sbusinesslogic

Butsometimesit’seasiertolearnitfromexamples:

3.

bootstrap and then expand

your vocabulary

Expanding & updating ontologies

Word2Vec

Let’s build this too!

Summary: How Summary: Why

1. Languagegetscomplexanddomainspecific

2. You’llneedmachinelearningearly

3. Bootstrap&thenexpandyourvocabulary

Whoneedstobevaccinated?

Whofitsthisclinicaltrial?

Whoisatriskforsepsis?

Thank You!github.com/atigeo/nlp_demo

@davidtalby