An Approach to Understanding Frank Greco 1/28/2016 IBM & Cognitive Computing.

An Approach to Understanding Frank Greco1/28/2016 IBM & Cognitive Computing 2015 IBM Corporation In Memory of Marvin Minsky 2015 IBM Corporation My Background BA, Mathematics, U Chicago MS, Computer Science and Mathematics, U Minnesota, with help from U Chicago, Todd Dupont & IBM IBM; work with multiple industries in multiple roles Developer, Data Center Technical Coordinator, Consultant - I/T Strategy & Application Architecture, SW Application Architect -Technical Advisor & Facilitator Current Activities with the University IBM Client teams Center for Research Informatics UC Medical Center Research Computation Center U Chicago Cognitive Computing Challenge : https://goo.gl/yrg1gXhttps://goo.gl/yrg1gX Academic Initiative Watson & Bluemix 2015 IBM Corporation Agenda Perspective On IBM Cognitive Computing Motivation and Definition Watson as an Example Vision, Challenges, and Considerations Ways to Engage 2015 IBM Corporation 2 Notable IBM - U Chicago Connections John Opel, Former IBM CEO U Chicago MBA Irving Wladawsky-Berger I retired from IBM on May 31, 2007 after 37 years with the company, where I was responsible for identifying emerging technologies and marketplace developments that are critical to the future of the IT industry. I was also responsible for our university relations office and for the IBM Academy of Technology where I served as Chairman of the Board of Governors. I led a number of IBMs company wide initiatives including the Internet and e-business, supercomputing and Linux. former member of University of Chicago Board of Governors for Argonne National Laboratories, the Board of Overseers for Fermilab, and BP's Technology Advisory Council. I am a Fellow of the American Academy of Arts and Sciences as well as a Fellow of Londons Royal Society of Arts. Having been born in Cuba and come to the US at the age of 15, I was named 2001 Hispanic Engineer of the Year. I have an M.S. and Ph. D. in physics from the University of Chicago. 2015 IBM Corporation Result from running the comment on the previous page through the Personality Insights API You are inner-directed, restrained and strict. You are philosophical: you are open to and intrigued by new ideas and love to explore them. You are authority-challenging: you prefer to challenge authority and traditional values to help bring about positive changes. And you are driven: you have high goals for yourself and work hard to achieve them. Your choices are driven by a desire for self-expression. You consider achieving success to guide a large part of what you do: you seek out opportunities to improve yourself and demonstrate that you are a capable person. You are relatively unconcerned with taking pleasure in life: you prefer activities with a purpose greater than just personal enjoyment. https://developer.ibm.com/watson/blog/2015/03/23/ibm-watson-personality-insights-science-behind-service/ 2015 IBM Corporation IBM Vital Stats (Wikipedia) TypePublic Traded as NYSE: IBM Dow Jones Industrial Average Component S&P 500 Component Industry IT consulting IT services Computer software Computer hardware Founded June 16, 1911; 104 years ago Endicott, New York, U.S. [1] FounderCharles Ranlett Flint HeadquartersArmonk, New York, U.S. Area served170 countries Key people Ginni Rometty (Chairman, President and CEO) ProductsSee IBM products RevenueUS$ billion (2014) Operating incomeUS$ billion (2014) Net incomeUS$ billion (2014) Total assetsUS$ billion (2014) Total equityUS$ billion (2014) Number of employees379,592 (2014) DivisionsHardware, Services, Software WebsiteIBM.com 2015 IBM Corporation Perspective on IBM History: Transformative Research 2015 IBM Corporation The Present Does Not Occur Without the PastEvolution to Cognitive Computing 10 2015 International Business Machines Corporation Cognitive EraProgrammatic Era Tabulation Era 2015 IBM Corporation Thomas Watson Sr. Fired from NCR ; THINK 1911 C-T-R (Computing - Tabulating Recording) 1928 Ben Wood, professor Columbia University Pioneered standardized tests; led to the problem of scoring 10s of 1,000s of tests Inspired Tom Watson Sr how IBM machines could be used to measure intellect and psychology Professors pushed IBM into scientific computing Thomas J Watson Astronomical Computing Bureau at Columbia 1940s WWII ; Los Alamos Project; Enrico Fermi 1945 Watson Sr. Estabkished the first corporate pure science research laboratory connected to a university 1950s Tom Watson Jr. Sets up a true research division --> Place where one could go deep and invent something Information age Van Neumann holds court Arthur Samuel Artificial intelligence checkers (1952) Marvin Minsky, in 1959, co-founded the M.I.T. Artificial Intelligence with his colleague John McCarthy, who is credited with coining the term artificial intelligence. Perceptron invented at Cornel by Rosenblatt 1960s IBM 360 Development Fred Brooks Mythical Man Month US Space Program... J.C.R. Licklider writes Man-Computer Symbiosis Anti-trust 1970s Ralph Gomory, head of IBM Research tied research with (product) development - joint programs Expanded collaboration with Universities (today 6,000 Us and 30,000 faculty members) Personal Computer 1990s Innovation by acquistion Deep Blue plays chess 2000s Open Innovation Collaboration with clients, amonsgt internal organizations, coopetition 2011 Watson Wins Jeopardy! Today On Collaboration across global systems (Systems of systems) Cognitive ecosystem bult atop a hybrid Cloud platform A Selective Thread Through IBMs History Cognitive Era Programmatic Era Tabulation Era 2015 IBM Corporation Progress Does Not Occur Without Collaboration and Innovation 12 Primary & Applied Research Product Development Application Business & Government AcademiaIBM The only way you survive is you continuously transform into something else. It's this idea of continuous transformation that makes you an innovation company. Ginni Rometty, IBM CEO 2015 IBM Corporation Impact on Traditional Research-Developer-User Roles End Use Application Development System Development Research End Use (Training Cycles) R&D (More tightly coupled) Continuous Delivery Programmatic Cognitive Exploratory I/T disciplines LOB & Cognitive specialist collaboration Involvement Skill Mix 2015 IBM Corporation 12 labs. 6 continents. ~3,000 Researchers $6B R&D budget Brazil (2010) Almaden Austin Watson1945/1961 IBM Research-Africa (2012) Dublin (2011) Tokyo (1982) China (1995) Melbourne (2010), India (1998) Haifa (1972) Zurich (1956) IBM's global research labs are focused on four key areas: Industries and Solutions: This organization will focus on the transformation of business through data. Computing as a Service: This organization will advance all areas surrounding the transformation of IT through Cloud. Cognitive Computing: This organization will be comprised of the teams focused on next generation IBM Watson capabilities, data and information management, human-computer interface and computer science. Science & Technology: IBM will continue to invest in fundamental science to advance the core technologies that will create the future of computing and enable this new model. IBM Research - The worlds largest private research institution 2015 IBM Corporation China Watson Almaden Austin Tokyo Haifa Zurich India Dublin Melbourne Brazil IBM Research labs Labs added since 2010 Keny a Accolades Nobel Laureates Georg Bednorz and Alex Mueller, also of Zurich, in 1987, for research in superconductivity; Georg Bednorz and Alex Mueller Gerd Bining and Heinrich Rohrer, of the Zurich Research Center, in 1986, for the scanning tunneling microscope; Gerd Bining and Heinrich Rohrer Leo Esaki, of the Thomas J. Watson Research Center in Yorktown Heights, N.Y., in 1973, for work in semiconductors. Leo Esaki Turing Award Recipients Francis Elizabeth Allen (2006) - Optimizing compiler techniques and automatic parallel execution Francis Elizabeth Allen Fred Brooks (1999) - Contributions to computer architecture, operating systems, and software engineering. Fred Brooks James Nicholas Gray (1998) - Transactions / ACID properties James Nicholas Gray John Cocke (1987) - RISC, CYK algorithm John Cocke Edgar Codd (1981) - Fundamental and continuing contributions to the theory and practice of database management systems Edgar Codd John Backus (1977) - FORTRAN, BNF, Functional programming John Backus Five Nobel Laureates 10 U.S. National Medals of Technology 5 U.S. National Medals of Science 6 Turing Awards 19 inductees in the National Academy of Sciences 14 inductees into the U.S. National Inventors Hall of Fame the most of any company 23 inductees in the U.S. National Academy of Engineering. Skills in mathematics, computer science, chemistry, physics, operations research, economics, anthropology and many more 2015 IBM Corporation Accomplishments th Consecutive Year of Patent Leadership 2011 Watson System 2009 Nanoscale Magnetic Resonance Imaging (MRI) 2008 Worlds First Petaflop Supercomputer 2007 Web-scale Mining 2006 Core Extensible Markup Language (XML) Standards 2006 Services Science, Management, Engineering (SSME) 2005 Cell Processor 2004 Blue Gene/L 2003 Carbon Nanotubes 2000 Java Performance 1997 Copper Interconnect Wiring 1997 Secure Internet Communication (HMAC, IPsec) 1997 Deep Blue 1994 Design Patterns 1994 Silicon Germanium (SiGe) 1990 Statistical Machine Translation 1987 High-Temperature Superconductivity* 1986 Scanning Tunneling Microscope* 1980 Reduced Instruction Set Computing (RISC) 1971 Speech Recognition 1970 Relational Database 1967 Fractals 1966 One-Device Memory Cell 1957 FORTRAN 1956 Random Access Memory Accounting Machine (RAMAC) Floppy disk Hard disk drive Electronic keypunch Automated Teller Machine (ATM) Magnetic strip card Virtual machine Universal Product Code (UPC) Financial swap SABRE airline reservation system Dynamic random access memory (DRAM) US Space Program Twitter Partnership Apple Partnership The Weather Company Human Genome project National Geographic Genographic project Green Horizons project 2015 IBM Corporation Examples U.S. PATENT #8,547,214: System for preventing handheld device use while operating a vehicle THINK OF IT ASA mobile phone forcefield for textaholics. This innovation can analyze your fingerprints, voice patterns, retina images, heartbeat and moreto prevent you from using a handheld device while operating a vehicle. So less attention on your phone and more on that moose that just stepped out in front of you U.S. PATENT #8,509,526: Object finder within digital images THINK OF IT AS Ctrl+F for finding images instead of words. This patented tool learns what an object looks like, then recognizes it, working more like the brain than a simple search tool. 2015 IBM Corporation Fun Facts Won an Emmy! https://www.research.ibm.com/index.shtml Cognitive Computing An Introduction 2015 IBM Corporation Jeopardy Recap 2015 IBM Corporation Cognitive Computing - Definition Cognitive systems form a category of technologies that use natural language processing, [speech and image recognition, and computer vision] and machine learning to enable people and machines to interact more naturally to extend and magnify human expertise and cognition. These systems will learn and interact to provide expert assistance to scientists, engineers, lawyers, and other professionals in a fraction of the time it now takes [at scale]. In addition to human like interaction and machine learning, cognitive systems generally are based around a domain specific semantic model. Cognitive systems are trained; and probabilistic in their behavior. The allure of understanding the brain, what it means to know, to create, to reason is not a new topic. AI has been around for 50+ yearsSo, why now? 2015 IBM Corporation Data Grows Exponentially Demands New Technology and Strategy You are here 44 zettabytes unstructured data structured data Source: IBM GTO 2015 2015 IBM Corporation The cost of not knowing is of critical concern 2015 IBM Corporation Data at the Edge Changing How We Look at Data 90% By 2017 Of data created over the last 10 years was never captured or analyzed The collective computing and storage capacity of smartphones will surpass all worldwide servers 60% 2X Of valuable sensory data loses value in milliseconds Rate of data creation compared to the expansion of bandwidth over the past decade Source: IBM GTO 2015 2015 IBM Corporation Convergence of Need and Feasibility (and interest) Exponential Growth of Predominately Unstructured (dark) Data Human Constraints: Its not humanly possible to keep up with all the data The reasoning experts bring to bear is not always conducive to deterministic programmatic approaches Even if it were, domain experts are typically not programmers Economic Considerations: The cost of not knowing is of critical concern Increased Feasibility: Key enabling technologies are more readily available Biologically inspired systems research Parallel processing / grid systems* On going research in NLP, machine learning, scientific/statistical computing e.g., named entity recognition and relation extraction Advances in processor, storage, and network technologies ** Semantic web / linked data Open systems * Computing, Cognition and the Future of Knowing; Dr. John E. Kelly III, Sr. VP, IBM Research and Solutions Portfolio ** More will be needed IBM GTO 20150 2015 IBM Corporation TD Gammon, 1992 (Gerald Tesauro) a self-teaching neural network that learned to play backgammon at human world championship level Chess, Deep Blue, 1997 A finite, mathematically well-defined search space Limited number of moves and states Grounded in explicit, unambiguous mathematical rules We learned something important: H+C > H or C alone Human Language, Watson, 2011 Ambiguous, contextual and implicit Grounded only in human cognition Seemingly infinite number of ways to express the same meaning Challenges: Past & Present Watson Example A Humanities Approach 2015 IBM Corporation Given Rich Natural Language Questions Over a Broad Domain of Knowledge Deliver Precise Answers: Determine what is being asked & give precise response Accurate Confidences: Determine likelihood answer is correct Consumable Justifications: Explain why the answer is right Fast Response Time: Precision & Confidence in leverage Linked Open Data for increased coverage and finer grained type information AI Planning-based solution to interpreting and populating frames. Automatically induce typical concepts and relations in a frame from a large corpus using data mining. 2015 IBM Corporation Type Coercion, a departure from prior approaches that limit answers searches with type constraints. Late binding of answer to LAT. Logical Framework MultipleTyCor components Used throughout the pipeline Form of evidence scoring Uses both structured and unstructured sources e.g., YAGO, WordNet, PRISMATIC Answer Scoring: Type Coercion (TYCOR) EDM Type Retrieval PDM Type Alignment Candidate LAT TyCor EDM: Entity Disambiguation and Matching PDM: Predicate-Argument Disambiguation and Matching YAGO Gender Closed LAT Lexical NED WordNet Wiki-Category Wiki-List Wiki-Intro Identify Passage PRISMATIC 2015 IBM Corporation TyCor Example: Consider a question asking for an emporer and the candidate answer is Napoleon Step 1: EDM TyCor uses Wikipedia or derived source e.g., YAGO, Wiki-List) to find dbbedia:Napolean Also finds dbpedia:Napoleon%28card_game_%2 9, scores this low WordNet TyCor finds 3 difference word senses Passage TyCor finds Napoleon in supporting passages NED & Identify TyCor declare Napoleon and entity Step 2:PDM Basically a similar process looks at the LAT focusing on emporer viz., In this case NED TyCor marks emporer as a NOMINAL reference to a Political leader As with TR, some PDM strategies return a string and defer to TA 2015 IBM Corporation TyCor Example Continued Step 3: TR (Text Retrieval) TyCor strategies that identify a formal entity during EDM e.g. DBPedia URL or WordNet synset are generally able to look up 1 or more formal types for that entity in a structured source e.g. WordNet TyCor labels Napoleon and instance of emporer NED TyCor labels Napoleon as A NAME reference to a Political leader The TyCor strategies that product text string defer to Type Alignment Step 4: TA (Text Alignment) For the TyCors, such as WordNet, YAGO, NED that produce formal types in a Type hierarchy from TR and PDM,the alignment process matches the graphs ie., looking for subsumption, disjointness, etc. e.g, is the entity type (from TR) a hyponym (more specific instance) of the questions type (from the PDM)? Where text strings are produced from TR and PDM viz, Wiki-List TyCor, the strategy is to conclude that a French monarch is an emperor by parsing both (e.g., finding that monarch is the headword of French monarch) and matching terms using resources such as WordNet and Wikipedia redirects. 2015 IBM Corporation 4A: Answer Scoring 2015 IBM Corporation 4B: Passage Scoring 2015 IBM Corporation Step 6: Final Confidence Merging and Ranking 2015 IBM Corporation Final Confidence Merging and Ranking Regularized logistic regression is applied as follows: QA i : Question-Answer Pair is an instance. Each instance QA i has a feature vector X i consisting of ranking scores Each instance is labeled Y i as +1 or -1. i=1,,M Regularized* Objective function: Where g is the logistic function.Gradient descent optimization: *regularized to avoid overfitting 2015 IBM Corporation Phase Based Machine Learning Framework Hitlist Normalization Rank and retain top 100 Base Partition into question classes Transfer Learning For Uncommon question Answer Learning Merge evidence between equivalent answers and select canonical forms Elite Successive refinement top 5 Multi-Answers Join answers candidates for Question requiring multiple answers Evidence Merging Post-processing Evidence Diffusion Diffuse Evidence between related answers Each phase has these 3 steps: Classification Evidence Merging Post-processing Classification Evidence Merging Post-processing Classification Evidence Merging Post-processing Classification Evidence Merging Post-processing Classification Evidence Merging Post-processing Classification Evidence Merging Post-processing Classification 2015 IBM Corporation Successive classification cycles of training and application Training using Manual Answer key Application to Testing data Training using Manual Answer key Application to Testing data Phase n Phase n+1 Evidence Merging Post- processing Removes/deri vces fetures Classifier: Training Mode Produce model Classifier: Application Mode Rank/estimate confidence Evidence Merging Post- processing Removes/deri vces fetures Contribution to feature set Model 2015 IBM Corporation Minimal Deep QA Pipeline (Complete) Vision, Challenges and Considerations 2015 IBM Corporation Watson Oncology https://www.mskcc.org/about/innovative- collaborations/watson-oncology 2015 IBM Corporation Vision: Building a Society of Cognitive Agents Watso n Cognitive Agent to Agent Outage Model Consequence Table Smart Swaps Lighting Critica l Sites Objective Identification Sensitivity Analysis Sentiment Analysis Systems of cognitive agents that collaborate effectively with one another Cognitive agents that collaborate effectively with people through natural user interfaces A nucleus from which an internet-scale cognitive computing cloud can be built Personal Avatar Deep Thunder Crew Scheduler News Human to Human Cognitive Agent to Human 2015 IBM Corporation The Challenges Curation and the challenges of handling ever increasing volume of data Integration of new forms of input e.g., visual, audio, which will require Specialized and scalable infrastructure Closed domain adaptation Cross domain understanding Unsupervised learning Relation detection in text Integration of cognitive capabilities into existing processes and applications All requiring collaboration across a variety of scientific, technological, and academic disciplines. 2015 IBM Corporation Factorization of Cognitive components https://console.ng.bluemix.net/catalog/?org=6cd214ad ef-bfad- cb9b6b8fa44d&space=539c b5-4c6f-8534-bbe2d5ddd32a&context=services 2015 IBM Corporation Ethical Consideration Technology creates possibilities and potential, but ultimately, the future we get will depend on the choices we make. Technology is not destiny. We shape our destiny. Erik Brynjolfsson, MIT 2015 IBM Corporation Ways to Engage UChicago Cognitive Computing Challenge ( Sign up for the "IBM Watson Cognitive Computing Practicum" being offered this Spring! You'll get to work with me and Cornelia! Got an idea for a cognitive project of your own? Submit it here: https://goo.gl/yrg1gX and we'll evaluate it.https://goo.gl/yrg1gX UChicago Departments of Computer Science Statistics Linguistics Psychology Neurology IBM Cloud http://www.ibm.com/smarterplanet/us/en/ibmwatson/developercloud/gallery.html?lnk=ushpv18 ce3http://www.ibm.com/smarterplanet/us/en/ibmwatson/developercloud/gallery.html?lnk=ushpv18 ce3 http://www.ibm.com/smarterplanet/us/en/ibmwatson/developercloud/services-catalog.htmlhttp://www.ibm.com/smarterplanet/us/en/ibmwatson/developercloud/services-catalog.html https://console.ng.bluemix.net/https://console.ng.bluemix.net/ IBM & IBM Research http://www-03.ibm.com/employment/http://www-03.ibm.com/employment/ http://www.research.ibm.com/careers/http://www.research.ibm.com/careers/ References https://www.research.ibm.com/labs/watson/https://www.research.ibm.com/labs/watson/ https://www.research.ibm.com/cognitive-computing/index.shtmlhttps://www.research.ibm.com/cognitive-computing/index.shtml And there is the world at large! 2015 IBM Corporation Appendix 2015 IBM Corporation Systems of Insight Architecture: Portfolio Integration Existing Systems of Record Cognitive Component Corpus Data Engagement Layer End Users References 2015 IBM Corporation References 1.Dr. John E. Kelly III,Senior VP, IBM Research and Solutions Portfolio; Computing, Cognition and the Future of Knowing, Whitepaper. 2.D. A. Ferrucci; Introduction to This is Watson; IBM J. Res. & Dev., vol 56, no. , May/Jul A. Lally, J. M. Prager, M. C. McCord, B. K. Boguraev, S. Patwardhan, J. Fan, P. Fodor,and J. Chu-Carroll; Question analysis: How Watson reads a clue; IBM J. Res. & Dev., vol 56, no. , May/Jul M. C. McCord, J. W. Murdock, and B. K. Boguraev; Deep parsing in Watson; IBM J. Res. & Dev., vol 56, no. , May/Jul J. Chu-Carroll, J. Fan, N. Schlaefer, and W. Zadrozny; Textual resource acquisition and engineering; IBM J. Res. & Dev., vol 56, no. , May/Jul J. Fan, A. Kalyanpur, D. C. Gondek, and D. A. Ferrucci; Automatic knowledge extraction from documents; IBM J. Res. & Dev., vol 56, no. , May/Jul J. Chu-Carroll, J. Fan, B. K. Boguraev, D. Carmel, D. Sheinwald, and C. Welty; Finding needles in the haystack: Search and candidate generation; IBM J. Res. & Dev., vol 56, no. , May/Jul 2012. 2015 IBM Corporation References Continued 8.J. W. Murdock, A. Kalyanpur, C. Welty, J. Fan, D. A. Ferrucci, D. C. Gondek, L. Zhang, and H. Kanayama; Typing candidate answers using type coercion; IBM J. Res. & Dev., vol 56, no. , May/Jul J. W. Murdock, J. Fan, A. Lally, H. Shima, and B. K. Boguraev; Textual evidence gathering and analysis; IBM J. Res. & Dev., vol 56, no. , May/Jul C. Wang, A. Kalyanpur, J. Fan, B. K. Boguraev, and D. C. Gondek; Relation extraction and scoring in DeepQA; IBM J. Res. & Dev., vol 56, no. , May/Jul A. Kalyanpur, B. K. Boguraev, S. Patwardhan, J. W. Murdock, A. Lally, C. Welty, J. M. Prager,B. Coppola, A. Fokoue-Nkoutche, L. Zhang, Y. Pan, and Z. M. Qiu; Structured data and inference in DeepQA; IBM J. Res. & Dev., vol 56, no. , May/Jul J. M. Prager, E. W. Brown, and J. Chu-Carroll; Special Questions and techniques; IBM J. Res. & Dev., vol 56, no. , May/Jul J. Chu-Carroll, E. W. Brown, A. Lally, and J. W. Murdock; Identifying implicit relationships; IBM J. Res. & Dev., vol 56, no. , May/Jul A. Kalyanpur, S. Patwardhan, B. K. Boguraev, A. Lally, and J. Chu-Carroll; Fact-based question decomposition in DeepQA; IBM J. Res. & Dev., vol 56, no. , May/Jul 2012. 2015 IBM Corporation References Continued 15.D. C. Gondek, A. Lally, A. Kalyanpur, J. W. Murdock, P. A. Duboue, L. Zhang, Y. Pan, Z. M. Qiu,and C. Welty; A framework for merging and ranking of answers in DeepQA ; IBM J. Res. & Dev., vol 56, no. , May/Jul E. A. Epstein, M. I. Schor, B. S. Iyer, A. Lally, E. W. Brown, and J. Cwiklik; Making Watson fast; IBM J. Res. & Dev., vol 56, no. , May/Jul G. Tesauro, D. C. Gondek, J. Lenchner, J. Fan, and J. M. Prager; Simulation, learning, and optimization techniques in Watsons game strategies; IBM J. Res. & Dev., vol 56, no. , May/Jul B. L. Lewis; In the game: The interface between Watson and Jeopardy!; IBM J. Res. & Dev., vol 56, no. , May/Jul IBM Global Technology Outlook K. Maney, S. Hamm, J.M. OBrien; Making the World Work Better The Ideas That Shaped A Century and a Company; NY, IBM Press, Dr. John Goldsmith, Computational Linguistics (class notes), University of Chicago,January 2, 2013. 2015 IBM Corporation References Continued 22.J. Hurwitz, M. Kaufman, A. Bowles; Cognitive Computing and Big Data Analysis; Indianapolis, Wiley & Sons, D. Ferrucci, E. Brown, J. Chu-Carroll, J. Fan, D. Gondek, A.A. Kalyanpur, A. Lally, J. William Murdock, E. Nyberg, J.Prager, N. Schlaeger, C. Welty; The AI Behind Watson The Technical Article; AI Magazine Fall, Alfio Massimiliano Gliozzo; Cognitive computing Deep QA in IBMs Watson Jeopardy! system and beyond (Classroom slides) ; 2014.