A Data Project - How does it different from a traditional IT project?

Data ProjectJOSEPH MAN

Data is never Simple

It is not just about storing dataIt is not just about presenting reports

It is not just about gathering information

Source: Keith Gordon, Principles of Data Management: Facilitating Information Sharing

Data is not just about Technology

Source: Jerrard B. Gaertner, Identifying and Overcoming Project Management Challenges in Big Data and Predictive Analytics;Mark Troester , Analytics Infrastructure: 15 Considerations

http://blogs.sas.com/content/datamanagement/2012/05/09/analytics-infrastructure-15-considerations/



Multiple Disciplines are Involved

Source: Jerrard B. Gaertner, Identifying and Overcoming Project Management Challenges in Big Data and Predictive Analytics

Multiple Data Sources are needed

ETL Process

• Knowledge on multiple disciplines are required• Processes should be understand for each source of data collection• Gaps of meta data should be well studied and addressed• Data modelling and data cleansing are essential• Require good understanding of Data & Process

Multiple Parties are servedReports

ForecastingPrediction

• Demands on reporting could be huge• Visualization facilitates decision making• Interpretation and intervention drives business

improvement• Machine Learning for Prediction Marketing

Data Project is like a Spiral

Source: MIT Online Course, Big Data and Social Analytics

Data projects are different

from Waterfall,

Agile, Iterative.

3V’s and 5R’s need to be Considered

Relevancy The data are those needed?Recency Are the data out-dated?Range Are there enough coverage and granularity?Robustness Does the noise grows faster than the signal?Reliability Does the data collection accurate and reliable?


Security and Privacy should always be Addressed

Source: Jerrard B. Gaertner, Identifying and Overcoming Project Management Challenges in Big Data and Predictive Analytics;MIT Online Course, Big Data and Social Analytics

Project Risks could be Challenging

Concentration, Conversion

(ETL) and Data Quality Risks

Few Security and Privacy

Tools

Staff Lack Familiarity and

Training

Architectural Complexity

Lack of Proven Reliability and

3rd Party Certification

Unrealistic Expectations and Pressure to Produce

Difficult to Test in

Conventional Ways


Despite, the Company can be hugely Benefit from Big Data

Data marts, data bases,

BI, forecasting

Ad hoc reporting, standardized reporting,

fixed reporting

Prescriptive & Predictive Analytics

Forecasting

Optimization & Prediction

Extrapolation

What, When,

How, WhyWhat,When,HowWhat

,When


Executive Summary Data is never simple

Data is not just about technology Multiple disciples are involved Multiple data sources are needed Multiple parties are served

Data Project is like a spiral 3V’s and 5R’s need to be considered Security and Privacy should always be addressed Project risks could be challenging

Despite, the company can be hugely benefit from Big Data

Q&A

Appendix - Analytics Infrastructure Considerations Vision & Strategy Drive analytic success via common vision

shared by all constituents Support multiple forms of analytics for

maximum value Analytics = Art + Science. IT should focus

on the science to enable analytic art

People It’s not just “Business” and “IT” Get the right people: Don’t skip on training Leverage Center of Excellence (COE)

principles

Process Understand the analytics lifecycle Ultimately it’s about improving the business

process Strike the right balance between control

and user flexibility

Technology Leverage Enterprise Architecture principles

to ensure proper design Think big! Big data & big analytics: High

Performance Analytics is key Integrate BI Platform into the overall IT

infrastructure Data / Information Design data strategy that results in information as a strategic asset Leverage comprehensive Information Management approach Step up to data preparation: Free up the scarce analytic resources

Source: Mark Troester , Analytics Infrastructure: 15 Considerations




Appendix – An Example of Andorra


Appendix – Big Data Project


Appendix – User Defined Reports, Pros and Cons

Pros IT can provide platforms while not involved in operations Business and operations can have reports in a timely manner

Cons Massive amount of reports would be created but not maintained

and controlled Reports invalid after database change or system enhancement in

future projects

Appendix – Authentication and Entitlement Authentication could link up to the enterprise infrastructure, e.g. Active

Directory Entitlement can be controlled at report level to allow access of certain data by

particular user group only. Further access control can be explore on database tables or columns for user

defined reports

Appendix – Data volume growth exponentially


Appendix – Pillars of Analytics Success

TANGIBLES1. Quality, relevant data sources (internal and external)2. Sufficient hardware/processing capacity (on site or cloud)3. Appropriate software/analytic tools (local, SAAS, hybrid)

OPERATIONAL BUSINESS PROCESSES4. Appropriate research & operational processes5. Effective data and data quality management6. Defining and implementing proper metrics

MEASUREMENT AND CONTROL7. Proper risk management8. Security & privacy best practices


Appendix – Pillars of Analytics Success

HUMAN RESOURCES AND SKILLS9. Adequate business process knowledge10. Properly trained and skilled staff11. Access to third party expertise and resources, if needed

ORGANIZATION AND CULTURE12. Suitable (adaptable) organization structure and exceptional change management13. Effective reporting structures and internal communications (includes cross functional)14. Nurturing creative inquiry and celebrating insight

MOST IMPORTANT OF ALL15. Asking the Right Question!Source: Jerrard B. Gaertner, Identifying and Overcoming Project Management Challenges in Big Data and Predictive Analytics