T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India) E : [email protected]W : www.rittmanmead.com What is Big Data Discovery, and how it complements traditional Business Analytics Mark Rittman, CTO, Rittman Mead November 2015
60
Embed
What is Big Data Discovery, and how it complements traditional business analytics
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
•Mark Rittman, Co-Founder of Rittman Mead •Oracle ACE Director, specialising in Oracle BI&DW •14 Years Experience with Oracle Technology •Regular columnist for Oracle Magazine •Author of two Oracle Press Oracle BI books •Oracle Business Intelligence Developers Guide •Oracle Exalytics Revealed •Writer for Rittman Mead Blog :http://www.rittmanmead.com/blog
•Oracle BI and DW Gold partner •Winner of five UKOUG Partner of the Year awards in 2013 - including BI •World leading specialist partner for technical excellence, solutions delivery and innovation in Oracle BI
•Approximately 80 consultants worldwide •All expert in Oracle BI and DW •Offices in US (Atlanta), Europe, Australia and India •Skills in broad range of supporting Oracle tools: ‣OBIEE, OBIA, ODIEE ‣Big Data, Hadoop, NoSQL & Big Data Discovery ‣Essbase, Oracle OLAP ‣GoldenGate ‣Endeca
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
Many Organisations are Running Big Data Initiatives
•Many customers and organisations are running initiatives around “big data” •Some are IT-led and are looking for cost-savings around data warehouse storage + ETL •Others are “skunkworks” projects in the marketing department that are now scaling-up •Projects now emerging from pilot exercises •And design patterns starting to emerge
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
•Typical implementation of Hadoop and big data in an analytic context is the “data lake” •Additional data storage platform with cheap storage, flexible schema support + compute •Data lands in the data lake or reservoir in raw form, then minimally processed •Data then accessed directly by “data scientists”, or processed further into DW
MartsData Warehouse
Σ Σ
BusinessIntelligence
• Online• Scalable• Flexible• Cost
Effective
Hadoop
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
Key Differentiator from DW Storage : Flexible Cheap Storage
•HDFS storage and “schema-on-read” technology gives you ability to store data in “raw” form ‣Cheap expandable storage makes it feasible to retain all incoming data at detail level ‣Schema-on-read technology gives you ability to apply formats and schema on-demand
•A useful complement to the curated, structured and focused datasets in the DW
•Underlying Hadoop platform supports compute and analysis on these raw datasets
•But how do you make sense of what’s in there?
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
Data Integration and BI Tools can Access Data Reservoir
•Data integration tools such as Oracle Data Integrator can now load and process Hadoop data •BI tools such as Oracle Business Intelligence 12c can report on Hadoop data •Great for answering questions you know about •Fantastic for navigating hierarchies and drilling •Superb when you understand the dataand the value that’s within it ‣But what if you don’t yet?
Access direct Hive or extract using ODI12c for structured OBIEE dashboard analysis
What pages are people visiting? Who is referring to us on Twitter? What content has the most reach?
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
But … Getting Value and Information Out of Hadoop is Hard
•Specialist skills typically needed to ingest and understand data coming into Hadoop •Data loaded into the reservoir needs preparation and curation before presenting to users •But we’ve heard a similar story before, from the world of BI…
•Part of the acquisition of Endeca back in 2012 by Oracle Corporation
•Based on search technology and concept of “faceted search”
•Data stored in flexible NoSQL-style in-memory database called “Endeca Server”
•Added aggregation, text analytics and text enrichment features for “data discovery” ‣Explore data in raw form, loose connections, navigate via search rather than hierarchies ‣Useful to find out what is relevant and valuable in a dataset before formal modeling
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
Endeca Server Technology Combined Search + Analytics
•Proprietary database engine focused on search and analytics •Data organized as records, made up of attributes stored as key/value pairs •No over-arching schema, no tables, self-describing attributes
•Endeca Server hallmarks: ‣Minimal upfront design ‣Support for “jagged” data ‣Administered via web service calls ‣“No data left behind” ‣“Load and Go”
•But … limited in scale (>1m records) ‣… what if it could be rebuilt on Hadoop?
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
•A visual front-end to the Hadoop data reservoir, providing end-user access to datasets •Catalog, profile, analyse and combine schema-on-read datasets across the Hadoop cluster •Visualize and search datasets to gain insights, potentially load in summary form into DW
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
•Provide a visual catalog and search function across data in the data reservoir
•Profile and understand data, relationships, data quality issues
•Apply simple changes, enrichment to incoming data •Visualize datasets including combinations (joins) ‣Bring in additional data from JDBC + file sources
•Prepare more structured Hadoop datasets for use with OBIEE
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
BDD for Reporting and Cataloging the Data Reservoir
•Very good tool for analysing and cataloging the data in the production data reservoir •Basic BI-type reporting against datasets, joined together, filtered, transformed etc •Cross-functional and cross-subject area analysis and visualization •Great complement to OBIEE ‣Able to easily access semi-structured data ‣Familiar BI-type graph components ‣Ideal way to give analysts access to semi-structured customer datasets
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
•Data in the data reservoir typically is raw, hasn’t been organised into facts, dimensions yet • In this initial phase, you don’t want to it to be - too much up-front work with unknown data •Later on though, users will benefit from structure and hierarchies being added to data •But this takes work, and you need to understand cost/benefit of doing it now vs. later
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
Supporting the Transition From Data Lab to Data Factory
•Most data science and big data projects start small, using R and single data scientist •Big Data Discovery help opens up data science and the data reservoir to the enterprise
6
D a t aL a b
Invent Commercialize
D a t aF a c t o r y
Research Develop
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
Enabling Hadoop Datasets for Access by Big Data Discovery
•Relies on datasets in Hadoop being registered with Hive Catalog •Presents semi-structured and other datasets as tables, columns •Hive SerDe and Storage Handler technologies allow Hive to run over most datasets •Hive tables need to be defined before dataset can be used by BDD
Ingesting & Sampling Datasets for the DGraph Engine
•Datasets in Hive have to be ingested into DGraph engine before analysis, transformation •Can either define an automatic Hive table detector process, or manually upload •Typically ingests 1m row random sample ‣1m row sample provides > 99% confidence that answer is within 2% of value shownno matter how big the full dataset (1m, 1b, 1q+) ‣Makes interactivity cheap - representative dataset
Amount'of'data'queried
The'100%'premium
CostAccuracy
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
•Rittman Mead want to understand drivers and audience for their website ‣What is our most popular content? Who are the most in-demand blog authors? ‣Who are the influencers? What do they read?
•Enrich and transform incoming schema-on-read social media data to add value •Do some initial profiling, analysis to understand datasets •Provide a user interface for the Big Data team •Provide a means to identify candidate dimensions and facts for a more structured (managed) data discovery tool like Visual Analyzer
Combine with site content, semantics, text enrichment Catalog and explore using Oracle Big Data Discovery
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
Ingesting Site Activity and Tweet Data into DGraph
•Tweets and Website Log Activity stored already in data reservoir as Hive tables •Upload triggered by manual call to BDD Data Processing CLI ‣Runs Oozie job in the background to profile,enrich and then ingest data into DGraph
Ingesting Site Activity and Tweet Data into DGraph
•Two output datasets from ODI process have to be ingested into DGraph engine •Upload triggered by manual call to BDD Data Processing CLI ‣Runs Oozie job in the background to profile,enrich and then ingest data into DGraph
• Ingestion process has automatically geo-coded host IP addresses •Other automatic enrichments run after initial discovery step, based on datatypes, content
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
Initial Data Exploration On Uploaded Dataset Attributes
•For the ACCESS_PER_POST_CAT_AUTHORS dataset, 18 attributes now available •Combination of original attributes, and derived attributes added by enrichment process
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
•Select refinement (filter) values from refinement pane •Visualization in scratchpad now filtered by that attribute ‣Repeat to filter by multiple attribute values
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
•Group and bin attribute values; filter on attribute values, etc •Use Transformation Editor for custom transformations (Groovy, incl. enrichment functions)
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
Datatype Conversion Example : String to Date / Time
•Datatypes can be converted into other datatypes, with data transformed if required •Example : convert Apache Combined Format Log date/time to Java date/time
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
•Users can upload their own datasets into BDD, from MS Excel or CSV file •Uploaded data is first loaded into Hive table, then sampled/ingested as normal
12
3
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
•Used to create a dataset based on the intersection (typically) of two datasets •Not required to just view two or more datasets together - think of this as a JOIN and SELECT
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
Join Example : Add Post + Author Details to Tweet URL
•Tweets ingested into data reservoir can reference a page URL •Site Content dataset contains title, content, keywords etc for RM website pages •We would like to add these details to the tweets where an RM web page was mentioned ‣And also add page author details missing from the site contents upload
Main “driving” dataset Contains tweet user details,tweet text, hashtags, URL referenced, location of tweeter etc
Contains full details of each site page, including URL, title, content, category
Join on URL referenced in tweet
Contains the post author details missing from the Site Content dataset
Join on internal Page ID
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
Multi-Dataset Join Step 1 : Join Site Contents to Posts
•Site contents dataset needs to gain access to the page author attribute only found in Posts •Create join in the Dataset Relationships panel, using Post ID as the common attribute •Join from Site contents to Posts, to create left-outer join from first to second table
1
2
3
Previews rows from the join, based onpost_id = a (post_id column)
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
•URLs in Twitter dataset have trailing ‘/‘, whereas URLs in RM site data do not •Use the Transformation feature in Studio to add trailing ‘/‘ to RM site URLs •Select option to replace the current URL values and overwrite within project dataset
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
Multi-Dataset Join Step 3 : Join Tweets to Site Content
•Join on the standardised-format URL attributes in the two datasets •Data view will now contain the page content and author for each tweet mentioning RM
1
2
3
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
Key BDD Studio Differentiator : Faceted Search Across Hadoop
•BDD Studio dashboards support faceted search across all attributes, refinements •Auto-filter dashboard contents on selected attribute values - for data discovery •Fast analysis and summarisation through Endeca Server technology
Further refinement on“OBIEE” in post keywords
3Results now filteredon two refinements
4
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
Key BDD Studio Differentiator : Faceted Search Across Hadoop
•BDD Studio dashboards support faceted search across all attributes, refinements •Auto-filter dashboard contents on selected attribute values •Fast analysis and summarisation through Endeca Server technology
“Mark Rittman” selected from Post Authors Results filtered on
selected refinement
1 2
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
Export Prepared Datasets Back to Hive, for OBIEE + VA
•Transformations within BDD Studio can then be used to create curated fact + dim Hive tables •Can be used then as a more suitable dataset for use with OBIEE RPD + Visual Analyzer •Or exported then in to Exadata or Exalytics to combine with main DW datasets
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
•Articles on the Rittman Mead Blog ‣http://www.rittmanmead.com/category/oracle-big-data-appliance/ ‣http://www.rittmanmead.com/category/big-data/ ‣http://www.rittmanmead.com/category/oracle-big-data-discovery/
•Slides will be on the BI Forum USB sticks •Rittman Mead offer consulting, training and managed services for Oracle Big Data ‣Oracle & Cloudera partners ‣http://www.rittmanmead.com/bigdata