ENTERPRISE DATA INTEGRATION APPLICATION ARCHITECTURE COMMITTEE OCTOBER 8, 2012 3-5 Year Strategic Initiatives.

ENTERPRISE DATA INTEGRATION

APPLICATION ARCHITECTURE COMMITTEE

OCTOBER 8, 2012

3-5 Year Strategic Initiatives

TABLE OF CONTENTS

•Committee Membership•Background and Context•List of Current Projects•Gaps and Needs•Vision for the Future•Proposed New Projects

• Overview of proposed initiative• Benefits and Impact• Key accompanying recommended changes to process,

people or organization• Order of magnitude implementation costs

•Draft Timeline•Other Opportunities for Consideration

2

APPLICATION ARCHITECTURE COMMITTEE

•Add committee membership here

3

BACKGROUND AND CONTEXT Data integration is essential to the day-to-day operation of UCSF systems. Some data is critical

across the enterprise, such as person data, financial data, and departmental data. Other data is generated and maintained within specific departments. Increasingly, it is important to share data across the enterprise to provide the most efficient IT solutions to UCSF business needs.

The exposure of Enterprise Data involves some risk to UCSF, therefore ensuring that data release policies are honored is essential to mitigating risk of data exposure

UCSF currently lacks central coordination around data integration and data release

Historically, most data has been shared at UCSF via nightly “batch” files, which integrate one system to another (point-to-point integrations). That means a campus department which needs data from multiple systems needs to manage multiple point-to-point integrations. Increasingly, campus departments are seeking solutions which bundle data from multiple systems and expose the data via a single web services interface. This reduces cost for the department as well as the need to store data locally.

Other applications and systems have a need to push updates to one systems to other systems in real time, so that data is up-to-date regardless of where it is accessed.

Finally, regardless of the data integration technology (batch, web services, message-based, etc), UCSF lacks a central organizational home to manage data integration requests. As a result, the process for accessing data is not clear, new systems take longer to bring online, and data release policies are not consistently enforced.

This proposal calls for two initiatives, one to develop organizational oversight for data integration requests and another to build a new data integration platform which supports Service Oriented Architecture.

4

CURRENT & PLANNED TECHNOLOGY PROJECTS

5

Research Administration – Systems integration (priority 3)

GAPS AND NEEDS• No Day-to-Day Oversight and Coordination of Data Integration

• At present, many departmental, vendor, and cloud-based applications need access to Enterprise data, and there is no clear process on how to access such data and ensure that data release is granted according to policy.

• Many campus departments are not sure where to get the data they need, and they often end up with data that has not been approved for release to their application or data which is out of date.

• At a bare minimum, UCSF needs better coordination around definition of Enterprise Data, and coordination of data integration between Enterprise Data and those campus, vendor and cloud-based applications.

• No Core Technology to Support Service Oriented Architecture

• Enterprise data is currently integrated through batch feeds (either via ETL tools like Cloverleaf and Informatica) or direct flat files generated from the mainframe, or via materialized views.

• Some data is exposed via web services, but not in a standard, re-usable way• UCSF does not currently support an operational data integration service that

supports event-driven, realtime integration• In the current environment, there is no easy way to aggregate data from multiple

systems into a single web services interface. Data must be consumed via multiple point-to-point feeds and aggregated at the consuming system itself.

6

VISION FOR THE FUTURE• Clear Organizational Contact for Data Integration at UCSF

• Campus departments which need Enterprise Data will know exactly where to start to get the data they need

• Procedures for requesting data release will be consistent and clearly documented. • Specific staff will be charged with managing data integration requests from start to

end, including any architectural review and data release processes.• Data Integration Service to Support SOA

• Many campus data integration needs are served by existing point-to-point solutions. Others demand a more service oriented approach.

• This project will stand up and support an ESB-based data integration service which will serve those immediate use cases where such an approach would yield the most value for the institution.

• The intent would not be to roll out SOA across UCSF immediately, but to establish a base infrastructure, use it to serve immediate use cases as appropriate, and to grow SOA over time at UCSF.

7

PROPOSED NEW PROJECTS

PriorityCriteria Score

Project Name Summary Stage

1 Enterprise Data Integration Coordination

Proposal

2 SOA Data Integration Service Proposal

8

#1 Enterprise Data Integration Coordination• Overview

• Many application owners note that they don’t currently know where to go when they need to access Enterprise Data for their applications and systems (e.g. person data, financial data, departmental data)

• Some historical primary sources of Enterprise Data, like the AdHoc database, grew organically are not necessarily the best source of Enterprise Data. AdHoc will likely be replaced in coming years by a new Operational Data Store, but decisions have not yet been made about what data will live in the ODS and the best ways to make that data available to campus.

• In many cases, departmental IT staff end up getting data feeds or extracts from data sources which they have heard about through word of mouth, but which do not enforce data release policies consistently, resulting in potential data exposure for UCSF.

• This project calls for the addition of 1-2 FTE to oversee the distribution of Enterprise Data – set standards for authoritative stores of Enterprise Data, set standards for how that data is shared with downstream customers, oversee and intake process for data integration requests.

• If funded, the FTE requested here could be combined with data integration requests from other committees to form a data integration team comprised of a manager, business analysts, and developers to oversee and implement data integration services at UCSF.

• Benefits and Impact

• Speed time of bringing up new applications and systems• Reduce risk of data exposure (ensure data release policies are enforced• Eliminate redundant data stores• Reduce reliance on fragile point-to-point integrations and replace them with re-usable interfaces

• Key process, people or organization changes

• Begin by creating new data integration coordination role at UCSF• Ultimately, establish an SOA framework for UCSF, likely under the guidance of an Enterprise Architect

• Estimated Project Length; 12 -18 months

• Order of Magnitude Implementation Costs: $100-500K (could include some approved funding for data integration from other initiative requests)

9

#2 SOA Data Integration Service

• Overview

• A number of campus departments are requesting that central IT provide a Service Oriented data integration service. • Key drivers include:

• Desire to aggregate data from multiple sources via a single web services interface• Need for realtime data integration• Desire to eliminate local data stores and fragile point-to-point data interfaces and instead, consume data directly from Enterprise

Data stores via standard service APIs

• This project calls for the creation of an ESB to support immediate use-cases for realtime and service-oriented data integration, and over time, as a generic platform to support the adoption of SOA architecture across UCSF.

• Benefits and Impact

• Would support data integration requests from other strategic initiatives and other specific use-cases at UCSF. See:• Department of Medicine - SOA Use Case• CTSI-Advance - SOA Use Case• Committee on Business Technology - SOA Use Case - Need content• Committee on Educational Technology - SOA Use Case - Need content• Committee on Research Technology - SOA Use Case - Need to confirm content• Identity and Access Management Steering Committee - SOA Use Case

• Salesforce - SOA Use Case - Need content

• Key process, people or organization changes

• Choose ESB platform• Fund ongoing operating expense for infrastructure and base staff support (1 FTE)• Establish process for determine priority for data integrations• Build service incrementally over time, starting where use case is clear (see links above)

• Estimated Project Length; 24 months

• Order of Magnitude Implementation Costs: $1,000,000 (mostly in UCSF in-house labor)10

TIMELINE FOR PROPOSED NEW PROJECTS

11

Data Integration Strategic Initatives

Project Names

Q4 FY1

2 Jan-Jun

FY 13July-Dec

FY 14Jan-Jun

FY 14July-Dec

FY 15Jan-Jun

FY 15July-Dec

FY 16Jan-Jun

FY 161 Enterprise Data Integration Coordination Scoping and Design Implementation 2 SOA Data Integration Service Scoping and Design Implementation

APPENDIX

12