In this presentation, O'Reilly author and Digital Reasoning CTO Matthew Russell along with Dr. Steve Kramer, founder and chief scientist at Paragon Science, discuss how Digital Reasoning processed the Enron corpus with its advanced Natural Language Processing (NLP) technology - effectively transforming it into building blocks that are viable for data science. Then, Paragon Science used dynamic graph analysis inspired from particle physics to tease out insights from the data in order to better understand whether an enterprise fiasco such as the Enron scandal could have been thwarted.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Got Chaos? Extracting Business Intelligence from Email with Natural Language Processing and Dynamic Graph Analysis
Steve Kramer, Ph.D. President & Chief Scientist Paragon Science, Inc.
1. READ: Forge natural language building blocks & build an information graph 2. RESOLVE: Introduce hierarchy & compress the information graph into knowledge 3. REASON: Exploit the knowledge to solve business problems
Extracting Critical Business Intelligence from Data Streams
• Digital Reasoning’s Synthesys platform for natural language processing
• Paragon Science’s dynamic anomaly detection software • Business goals:
• Help seize opportunities • Mitigate risks
• Case study: Enron email data set • Identify key individuals by position in network • Highlight unusual changes in sentiments • Drill down to relevant source messages
• For the central individuals in the Enron network, we construct dynamic networks using the phrases extracted from the emails by Digital Reasoning.
• Use the LIWC (Linguistic Inquiry and Word Count) tool to classify the sentiment scores of the phrases. • Prof. James Pennebaker from UT Austin (http://
Message-ID: <8261377.1075848143751.JavaMail.evans@thyme>Date: Wed, 28 Feb 2001 06:19:00 -0800 (PST)From: [email protected] To: [email protected] Subject: Energy Issues … Forwarded by Steven J Kean/NA/Enron on 02/28/2001 02:19 PM ----- To: Ann M Schmidt/Corp/Enron@ENRON, Bryan Seyfried/LON/ECT@ECT, [email protected], Riverside Press 2/23: "Power line plan has new foes"Contra Costa Times, 2/23: "GOP in a fix over power crisis"Sac Bee, Fri 2/23: "Davis says State has tentative deal for Edison grid"SF Chron 2/23: "Utilities Search for Long-Term Fix During Breather... GOP in a fix over power crisis Leaders seek solutions that won't undercut past support for deregulation By Daniel Borenstein TIMES POLITICAL EDITOR California Republican Party leaders are trying desperately to politicize the state's electricity crisis by blaming Gray Davis, but it seems that the harder they try the more popular the Democratic governor becomes. The latest GOP push will come today when party leaders, meeting in Sacramento for the start of their three-day state convention, hold a hastily organized workshop on energy. "We should be explaining to California that Gray Davis was asleep at the switch last year," said one of the scheduled speakers, Republican strategist Dan Schnur. "But that message can only work if it's coupled with a proposal for an alternative solution."
• Similarly, characteristic vectors derived from the flow map’s Jacobian can describe the generalized directions of the local stretching or contraction.
• Possible derivation approaches: • Weight-based column sampling • Singular value decomposition (SVD) • Principal component analysis (PCA)
the framework with modern credit risk measurement tools leads to a liquidity risk VAR measure 2000-‐08-‐08 02:34:00 0.9
This event has rapidly establishing itself as the risk management industry ,s most important meeDng point 2000-‐08-‐17 04:22:00 0.7
This event has rapidly establishing itself as the risk management industry's most important meeDng point 2000-‐08-‐17 07:01:00 0.7
system providers this is THE financial risk management event of the year 2000-‐08-‐17 04:22:00 0.6 system providers this is THE financial risk management event of the year 2000-‐08-‐17 07:01:00 0.6 bank liquidity risk can be viewed as a variaDon of credit risk analysis 2000-‐08-‐08 02:34:00 0.6
the New Products Designed to Stabilise VolaDlity Energy Credit Risk Management GARP 2000-‐08-‐17 04:22:00 0.5 this ConvenDon will consider include Market Risk (stress tesDng, liquidity, jump diffusion 2000-‐08-‐17 04:22:00 0.5 validaDon, EVT) Advanced Asset & Liability Management, Corporate/Energy Risk Management and the Insurance & Capital Markets
2000-‐08-‐17 04:22:00 0.5
VolaDlity Energy Credit Risk Management GARP is a not-‐for-‐profit, independent organisaDon of risk management pracDDoner
the New Products Designed to Stabilise VolaDlity Energy Credit Risk Management GARP 2000-‐08-‐17 07:01:00 0.5 this ConvenDon will consider include Market Risk (stress tesDng, liquidity, jump diffusion 2000-‐08-‐17 07:01:00 0.5 validaDon, EVT) Advanced Asset & Liability Management, Corporate/Energy Risk Management and the Insurance & Capital Markets
2000-‐08-‐17 07:01:00 0.5
VolaDlity Energy Credit Risk Management GARP is a not-‐for-‐profit, independent organisaDon of risk management pracDDoner
Sentence Message Date Anxiety Score any quesDons please do not hesitate to contact me 2000-‐08-‐17 04:22:00 0.4 INVITATION TO SPEAK GARP 2001 The 2nd Annual Risk Management ConvenDon 13th & 14th February, 2001 ) MarrioH World
2000-‐08-‐17 04:22:00 0.4
Three simultaneous streams address tradiDonal pricing and risk management techniques 2000-‐08-‐17 04:22:00 0.4 any quesDons, please do not hesitate to call me at ext 2000-‐08-‐07 01:27:00 0.4 your risk factor will be reduced by a factor equal the sqrt(5 2000-‐07-‐28 10:05:00 0.4 I have referred him to the "Managing Energy Price Risk" book 2000-‐08-‐08 02:29:00 0.4 any quesDons please do not hesitate to contact me 2000-‐08-‐17 07:01:00 0.4 Three simultaneous streams address tradiDonal pricing and risk management techniques 2000-‐08-‐17 07:01:00 0.4 conflicts of schedule, please do not hesitate to contact me 2000-‐07-‐31 08:46:00 0.4 modelling approaches to credit risk. 2000-‐08-‐02 01:11:00 0.4 market risks, has received liHle aHenDon in professional or academic journals 2000-‐08-‐08 02:34:00 0.4 the Global AssociaDon of Risk Professionals, I have great pleasure 2000-‐08-‐17 04:22:00 0.3 you to speak at our 2nd Annual Risk Management ConvenDon ) GARP 2001 2000-‐08-‐17 04:22:00 0.3 we lack desperately technical talent 2000-‐08-‐07 04:13:00 0.3 we lack desperately technical talent 2000-‐08-‐08 02:39:00 0.3 Energy Risk -‐ Tackling Price VolaDlity, AdapDng VaR, Scenario Modelling and Regulatory Requirements Vince Kaminski
2000-‐08-‐17 07:01:00 0.3
GARP 2000 reflected the key concerns of risk management experts 2000-‐08-‐17 07:01:00 0.3 INVITATION TO SPEAK GARP 2001 The 2nd Annual Risk Management ConvenDon 13(superscript 2000-‐08-‐17 07:01:00 0.3 the Global AssociaDon of Risk Professionals, I have great pleasure 2000-‐08-‐17 07:01:00 0.3 you to speak at our 2(superscript: nd) Annual Risk Management ConvenDon 2000-‐08-‐17 07:01:00 0.3 approaches for measuring these risks for EES deals 2000-‐07-‐26 02:08:00 0.3 EES deals given the uncertainDes in energy consumpDon 2000-‐07-‐26 02:08:00 0.3 we lack desperately technical talent 2000-‐08-‐11 00:42:00 0.3 a framework allowing to measure a bank's structural liquidity risk 2000-‐08-‐08 02:34:00 0.3 I taking over responsibility for semng-‐up EES Europe's Risk Management acDvity 2000-‐08-‐03 08:51:00 0.3
Message-‐ID: <2250375.1075856554294.JavaMail.evans@thyme> Date: Thu, 17 Aug 2000 08:55:00 -‐0700 (PDT) From: [email protected] To: [email protected] Subject: Re: Cairn Gas Purchase Bid Douglas S Parsons@ENRON_DEVELOPMENT 08/15/2000 09:30 AM To: Doug Leach/HOU/ECT@ECT cc: Marc De La Roche/HOU/ECT@ECT Subject: Re: Cairn Gas Purchase Bid I can appreciate and share your objec`ve. Earlier today I sent a separate note to Vince forwarding your concerns and asking again for his assistance. … To: Douglas S Parsons/ENRON_DEVELOPMENT@ENRON_DEVELOPMENT Subject: Re: Cairn Gas Purchase Bid I gave you several clear alterna`ves such as contac`ng Vince's structuring group, Michael Popkin's Southern Cone structuring group and a long discussion regarding the pricing and suggested "collar." I also asked if you had spoken to your customer about what they were willing to pay, but that was a non starter. Trust me, I have seen almost every bad deal Enron has entered into or afempted to enter into and I am trying to get Metgas to objec`vely relook at their offer to Cairn become it becomes another bad deal.
Sentence Message Date Anger Score we have a fighDng chance 2000-‐10-‐21 06:27:00 0.4 a criDcal point and needs to understand the Nigeria tax implicaDons for the venture 2000-‐09-‐01 05:05:00 0.3
the commercial team is now at a criDcal point 2000-‐09-‐01 05:05:00 0.3 he had a heart aHack 2000-‐10-‐10 02:26:00 0.3 5 really bothers her 2000-‐09-‐18 07:36:00 0.3 The Midwest is dominated by verDcally integrated Transmission Providers ("TPs 2000-‐11-‐10 08:56:00 0.3
the Aquarium closed -‐ killed by bike lanes 2000-‐09-‐14 02:54:00 0.3 Assignment language -‐ new language being tortured within the Enron ranks, to GE 2000-‐11-‐28 05:20:00 0.3
my gas won't be cut off this COLD November 2000-‐11-‐22 02:55:00 0.3 such case, you should destroy this message 2000-‐09-‐01 05:05:00 0.2 you tonight to trick or treat 2000-‐10-‐30 07:45:00 0.2 michael cut is toe, I 2000-‐10-‐16 06:46:00 0.2 the criDcal juncture, we should enlist external support 2000-‐09-‐15 05:38:00 0.2
Enron should protest or intervene 2000-‐11-‐09 01:06:00 0.2 Calif., who represents San Diego 2000-‐09-‐12 02:13:00 0.2 State Mkt Structure Blamed For Crisis The hearing 2000-‐09-‐12 02:13:00 0.2 the staff probe, appeared to place much of the blame 2000-‐09-‐12 02:13:00 0.2 I will be leaving around 2:00 for suburban trick or treaDng 2000-‐11-‐09 01:45:00 0.2 Enron should protest or intervene 2000-‐11-‐06 07:34:00 0.2 it screwed up the parking 2000-‐09-‐14 06:21:00 0.2 our criDcal final zoning vote is next Thursday, I 2000-‐11-‐01 06:17:00 0.2 She's not a dumb dog 2000-‐10-‐20 05:32:00 0.2 it just hasn't done the trick 2000-‐10-‐27 01:02:00 0.2 FP&L revealed violaDons 2000-‐11-‐10 08:56:00 0.2
Date: Sat, 21 Oct 2000 06:27:00 -‐0700 (PDT) From: [email protected] To: [email protected], [email protected] Subject: Re: Power Plant Development Powerpoint PresentaDon I don't know if you got this, so here 'Ds. I was out of town and missed the presentaDon, although I've worked with Herman, Roger and David on these issues. I'm sure they would be glad to give the presentaDon again for others with a need to understand the complexiDes of off balance sheet treatment of power plants. Kay … To: Suzanne Adams/HOU/ECT@ECT cc: Karen E Jones/HOU/ECT@ECT, Bob Carter/HOU/ECT@ECT, Barton Clark/HOU/ECT@ECT, Dale Rasmussen/HOU/ECT@ECT, Ed B Hearn III/HOU/ECT@ECT, [email protected], [email protected], Stuart Zisman/HOU/ECT@ECT, Peggy Banczak/HOU/ECT@ECT, David Leboe/HOU/ECT@ECT Subject: Re: Power Plant Development Powerpoint PresentaDon Could you check with David and see if he could fax copies of the PowerPoint plates to me in Portland so we have a fighDng chance of following along by tying in by phone? If that is possible, try to send them by Tuesday. Thank you very much. Al Larsen
Date: Fri, 1 Sep 2000 05:05:00 -‐0700 (PDT) From: [email protected] To: [email protected] Subject: Nigeria Tax Issues Seyi, James MacCallon has now ler for Mexico. I am just returning from India and must step in where he ler off. I see James has sent a few emails and faxes to you but I can find no reply from AA although there is a 14-‐July email from you indicaDng a reply would be sent by 16-‐July. Have you sent a reply and I have missed it? Given that the commercial team is now at a criDcal point and needs to understand the Nigeria tax implicaDons for the venture and bid, perhaps we should short-‐cut this Q&A process and have a conference call. I would suggest early AM Tuesday 20th, what works for you?