Top Banner
37
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.
Page 2: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Connectivity Options for Data Integration with Microsoft SQL Server Integration Services (SSIS)

Cedric LabuschagneSolutions ArchitectBI-LateralBIN301

Page 3: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Agenda

What is new in SSIS 2008?Connectivity componentsConnectivity CategoriesNew SSIS connectors

Page 4: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

ERP

Thinking Bigger About Business Intelligence

Majority of the content for decision making are in multiple heterogeneous sources (e.g. ERP systems, databases, flat files)

Need to integrate & consolidate data from all these sources, and make quality data available to users

Connectivity is the key to address these challenges

FlatFile

ERP Systems

Excel

XML

Databases

Page 5: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

What is new in SSIS 2008?

Improved and new task componentsNew Lookup Capabilities (caching enhanced)Data Profiling Task (data quality checking)C# Scripting. VSA replaced by VSTAScalable Data Flow (thread pooling)

Enhanced and new connectivity supportADO.NET Source and DestinationNew connectors for Oracle, Teradata and SAP BI

Page 6: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

SSIS Connectivity ComponentsSSIS Package

Data Flow Task

FTP FLATFILE ADO.NETOLEDB

FTP ExecuteSQL Task FlatFileLookup ADO.NET

ConnectionManagerTask Data Source/

DestinationDataTransformation

Note: Execute SQL Task can also make use of ADO.Net, ODBC Connection Managers

Page 7: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Component InteractionSSIS

Data Storage

AcquireConnection()

Connection Object

Write() Read()

Taskor

Component

Connection Manager

Page 8: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Data Flow Connectivity OptionsData Flow Task

OLEDBConnector

ADO.NETConnector

CustomConnector

OLEDBProvider

NativeADO.NETAdapter

ADO.NETODBC Data Source

SpecificInterface/Protocol

Data Source Specific Interface/ Protocol

ODBC

Data Storage

Page 9: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Data Access APIs and SSIS

"When developing in BIDS, 32-bit data providers are always used at design time, if 64-bit providers will be used in production packages, the equivalent 32-bit provider must be available at design time."

API Pros Cons

OLEDB • Comprehensive support with SSIS • 64-bit OLE DB providers less prevalent.

ADO.NET• Good support with SSIS• Direct support with ADO.Net Source & ADO.Net Destination

• Somewhat limited availability of out of the box ADO.NET providers

ODBC • ODBC drivers for possible data sources widely available

• Requires the use of a bridge - introduces performance and reliability issues

Custom-API • Designed for performance of native utilities

•Limited availability for most databases • Inconsistent support with SSIS - varies according to architecture

Page 10: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

SSIS Enterprise Data Connectivity Versions Suggested Connector SSIS Source &

DestinationSSIS Connection Manager

SQL Server All SQL Data Access Components ADO.NETOLEDB

ADO.NETOLEDB

Oracle 9iR2, 10, 11 Microsoft Connector for Oracle Oracle MSORA

Teradata V2R6.0/1/2, 12 Microsoft Connector for Teradata Teradata MSTERA

SAP BI 7 Microsoft Connector for SAP BI SAP BI SAPBI

DB2 8.1, 9.x Microsoft OLEDB Provider for DB2 OLEDB OLEDB

SAP R/3 4.6c, 4.7, 5, 6 BizTalk Adapter 3.0 for mySAP ADO.NET ADO.NET

Siebel 7.5, 7.7, 7.8, 8 BizTalk Adapter 3.0 for Siebel ADO.NET ADO.NET

Page 11: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Connectivity CategoriesDatabase Systems (RDBMS)• Microsoft SQL Server • Oracle • Sybase • IBM DB2 • Teradata • MS Access

Queue Systems & Protocols• MSMQ (Message Queue Task) • (s)FTP (FTP Task) ** • HTTP/HTTPs (Web Service Task) • IBM MQ Series* • Tibco Rendezveus*

Unstructured ,Semi-structured Data• Microsoft Excel, CSV • Text • XML • EDI (Electronic Data Interchange)

Application Systems • Siebel • Microsoft Excel • Hyperion (Essbase, Planning,

Financial Management)• SAP R/3 & SAPBI

* not supported out-of-the-box, however, custom components can do this.** sFTP is not supported out-of-the-box

Page 12: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Microsoft Connector for SAP BI

Page 13: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Microsoft Connector for SAP BI

Provider Microsoft

Version SAP NetWeaver BI Version 7

Pre-requisites SQL Server Enterprise LicenseSAP librfc32 librarySAP OpenHub license

Platform x86, x64, ia64

Page 14: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Microsoft Connector for SAP BIData Flow Task

SAP BIConnection Mgr

Remote Function Calls (RFC’s) *

SAP BI

SAP BIData Source

SAP BIData Destination

OpenHub BAPI

*standard programming interface used by SAP

Page 15: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

SQL Server

Application Scenario 1Extracting data from SAP BI into SQL Server

SAP BI

OHSDestination

SSIS

SSIS Package

OLE dBDestination

SAP BISource

Ope

n H

ub

Info Cube

ProcessChainTriggered

SAP BIConnection

Manager

Page 16: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

SAPBI Connectordemo

Page 17: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Microsoft Connector for SAP BI Extracting data from SAP BI to SQL Server

1. SAP BI Configuration1. Set up the RFC Destination2. Configure and create the Open Hub Destination3. Create the Data Transfer Process (DTP) and transformation4. Configure the process chain

2. SSIS Package1. Setup the SAP BI Source2. Setup the connection manager for SAP BI3. Setup the Destination (OLE DB, SQL Destination)4. Construct Workflow

3. Run the SSIS package

Page 18: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

SAP BI

Stag

ing

BAPI

Application Scenario 2Loading non-SAP data into SAP BI

SSIS

SSIS Package

SAP BIConnection

Manager

OLE DBSource

SAP BIDestinationSQL

Server XML

FlatFile

InfoSource/InfoPackage

Page 19: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Microsoft Connector for Oracle (by Attunity)

Page 20: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Microsoft Connector for Oracle

Provider AttunitySupported by Microsoft

Version Oracle 9iR2, 10, and 11

Pre-requisites SQL Server Enterprise License*Oracle client components

Platform SSIS: x86, x64, ia64

* 32 bit @design time, 64 bit @ runtime)

Page 21: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Microsoft Connector for OracleData Flow Task

OracleConnection Mgr

OCI

Oracle dB

OracleData Source

OCIArray Binding

OracleData Destination

OCIDirect Path

OCI

Page 22: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Performance of Oracle Connectors

Load Extract

27680

37362

1646

29756

Microsoft Connector for Oracle (by Attunity)Microsoft OLE DB Provider for Oracle

Tota

l Thr

ough

put (

Row

s/Se

c)

x16 x 1.25

Page 23: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Performance of Microsoft Connector for Oracle

Load Load - 10K Extract Extract - 10K

2901933747 36722 39466

26413

50158

8425491895

Mixed Types String Only

Tota

l Thr

ough

put (

Row

s/Se

c)

Page 24: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Microsoft Connector for OracleCustom Properties

Component Access Mode Property (Default)Oracle Source Table Name Batch Size (100)

Prefetch Count (0)SQLOracle Destination Table Name Batch Size (100)

Transaction SizeNo Logging (False)

Table Name – Fast Load Parallel (False)No Logging (False)Buffer Size (64KB)

Page 25: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Oracle Connectordemo

Page 26: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Microsoft Connector for Teradata (by Attunity)

Page 27: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Microsoft Connector for Teradata

Provider AttunitySupported by Microsoft

Version Teradata V2R6.0/1/2, 12

Pre-requisites SQL Server Enterprise LicenseTeradata ODBC driverTPT 12 for high-performance

Platform x86, x64

Page 28: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Microsoft Connector for TeradataData Flow Task

TeradataConnection Mgr

TPT12

Teradata DW

TeradataData Source

TeradataData Destination

TPTFastLoad

TPTTPUMP

TPTFast

Export

Page 29: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Microsoft Connector for TeradataCustom Properties

Component Access Mode Property (Default)Teradata Source Table Name – TPT Export BlockSize (64330)

DataEncryption (False)Max Sessions (1)Min Sessions (1)TenacityHours(4)TenacitySleep(6)

SQL Command – TPT Export

Teradata Destination Table Name – TPT Stream BlockSize (64330)DataEncryption(False)MaxErrors (0)MaxSessions (1)MinSessions(1)TenacityHours(4)TenacitySleep(6)

Table Name – TPT Load

Page 30: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

TeraData Connectordemo

Page 31: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Recap

SSIS connectivity Provides a robust connectivity architectureProvides an extensible connectivity architectureProvides rich connectivity options to enterprise data via :

Built-in connectorsAdditional connectors (available from Microsoft downloads)*Third party connectors

Rich ecosystem for SSIS connectors

*[codeplex or custom dev]

Page 32: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

question & answer

Page 33: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

www.microsoft.com/teched

International Content & Community

http://microsoft.com/technet

Resources for IT Professionals

http://microsoft.com/msdn

Resources for Developers

www.microsoft.com/learning

Microsoft Certification & Training Resources

Resources Tech·Ed Africa 2009 sessions will be made available for download the week after the event from: www.tech-ed.co.za

Page 34: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

SSIS Community ResourcesSSIS Community wiki

http://wiki.sqlis.com/default.aspxSSIS Codeplex

http://sqlsrvintegrationsrv.codeplex.com/SSIS Junkies :

http://blogs.conchango.com/jamiethomson/Attunity Support Forums :

http://www.attunity.com/forums/microsoft-ssis-oracle-connectorhttp://www.attunity.com/forums/micorosft-ssis-teradata-connector

Additional Community ResourcesTechNet Community for IT Professionalshttp://technet.microsoft.com/en-us/sqlserver/bb671048.aspxDeveloper Center http://msdn.microsoft.com/en-us/sqlserver/bb671064.aspxSQL Server 2008 Learning Portalhttp://www.microsoft.com/learning/sql/2008/default.mspx

Page 35: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Additional Resources

External ResourcesSSIS ETL Record – Loaded 1TB in 30 Minutes! http://msdn.microsoft.com/en-us/library/dd537533.aspxConnectivity Options for Microsoft SQL Server 2008 Integration Serviceshttp://msdn.microsoft.com/en-us/library/dd299429.aspxMicrosoft Connectors for Oracle and Teradata by Attunity http://www.microsoft.com/downloads/details.aspx?FamilyId=D9CB21FE-32E9-4D34-A381-6F9231D84F1EUsing SQL Server 2008 Integration Services with SAP BI 7.0

http://msdn.microsoft.com/en-us/library/dd299430.aspx

SQL Server 2008 Business Value Calculator: www.moresqlserver.com

• Team Forum: http://social.msdn.microsoft.com/forums/en-US/sqlintegrationservices/threads/

Page 36: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

Complete a session evaluation and enter to win!

10 pairs of MP3 sunglasses to be won

Page 37: Cedric Labuschagne Solutions Architect BI-Lateral BIN301.

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS,

IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.