DTP

Data Transfer Process

© SAP 2008 / Page 2

1. Data Transfer Process2. Monitoring3. Delta4. Error-Handling5. Manual Update6. Roadmap and Useful Transactions

Agenda

© SAP 2008 / Page 3

Data Transfer Process: Simple Example 1

DataTransfer Process

InfoPackage

TRANSFORMATION

DataSource (PSA)

SourceSystem 1

Process Chain

BW

© SAP 2008 / Page 4

Data Transfer Process: Simple Example 2

Process Chain

DataTransfer Process

InfoPackage

DataSource (PSA)

SourceSystem 1

BW

InfoSource (7.0)

TRANSFORMATION 3

DataTransfer Process

TRANSFORMATION 1

TRANSFORMATION 2

© SAP 2008 / Page 5

Data Transfer Process: Complex Example

DataStore Object 1

DTP

DataStore Object 2

DataStore Object 3

InfoSource (7.0)

DTPDTP

DTPDTP

DTP

IP IP

TR

TR TR

TR

TRTR

DataSource (PSA) DataSource (PSA)

SourceSystem 1 SourceSystem 2

TRBW

© SAP 2008 / Page 6

Benefits of DTP

Data Transfer Process (DTP): data ‘distribution’ within SAP BI

Improved transparency of staging processes across data warehouse layers (PSA, DWH layer, ODS layer, Architected Data Marts)

Separation of delta mechanism for different data targets Improved performance: Optimized parallelization Enhanced filtering in dataflow Enhanced error handling via error stack and temporary data storage is

possible.

© SAP 2008 / Page 7

Creating DTP

Once the transformation is defined, it is possible to create the Data Transfer Process.

Header-Table for DTP: RSBKDTP

© SAP 2008 / Page 8

Filter in DTP

Different data selections are possible for one DataSource via different DTPs

For the same DataTargetFor different DataTargets.

Extraction mode:Delta or Full

© SAP 2008 / Page 9

DTP for Open Hub Destination

Open Hub Destination also possible as DTP Data Target InfoSpokes are obsolete

© SAP 2008 / Page 10

1. Data Transfer Process2. Monitoring3. Delta4. Error-Handling5. Manual Update6. Roadmap and Useful Transactions

Agenda

© SAP 2008 / Page 11

Monitor in general

To run the Monitor Transaction RSMO or use the Button You can find both InfoPackage- and DTP-Monitor here

Or use Transaction RSRQ to enter a DTP-ID or DTP-SID directly

© SAP 2008 / Page 12

The different tabs – Header

Shows Technical status source and target of DTP

Possibility to jump to Job overview Error stack Data Target Administration DataSource Administration Different analysis tools

which use the time frame of the given request

Request-ID

© SAP 2008 / Page 13

The different tabs – Details

Important for runtime analysis

© SAP 2008 / Page 14

Temporary Storage for DTP

The Details-Tab also shows the temporary storage for the different sub-steps of the DTP.

The DTP can be set up from the temporary storage in case of a problem (see „manual update“).

You can view and verify the data in the temporary storage in case of problems.

© SAP 2008 / Page 15

Temporary Storage for DTP

Identifying the detail levels of temporary storage

Identifying the valid time for temporary storage

Here you can switch on / off the temporary storage for each substep of the DTP process.

© SAP 2008 / Page 16

Debugging possibilities

Switch Processing Mode to „Serially in the Dialog Process (for Debugging)“ allows to simulate the DTP execution

Always the next set of data is simulated Various possibilities for breakpoints Even DataPackage number can be specified for single breakpoints

© SAP 2008 / Page 17

Request Simulation along the time line

You can „rerun“ the protocol creation of a DTP request Shows only the monitor-steps which already happened at that time, allows

scrolling back and forward. Helpful for analysis of lock situations or parallel processing with different data

packages.

Detail-screen – Fixed – Scroll back and forward with the arrows

© SAP 2008 / Page 18

Customizing

Most customizing in SPRO is not used for DTP: DTP is executed via batch processes No dialog timeouts anymore DTP is scheduled via process chain Monitoring via CCMS, no monitor

assistant. Traffic light for „no data available“ is always green in DTP monitor.

Customizing setting for warning-messages is the only one which is used in DTP monitor.

© SAP 2008 / Page 19

1. Data Transfer Process2. Monitoring3. Delta4. Error-Handling5. Manual Update6. Roadmap and Useful Transactions

Agenda

© SAP 2008 / Page 20

DTP Delta

Delta is determined by the updating DTP Delta is NOT determined anymore by the combination DataSource /

Sourcesystem Separation of delta mechanism for different data targets

OLD Datamart-Handling:One Delta-Handling for each DataSource / Sourcesystem combination

NEW DTP-Delta:Separate Delta-Handling for each DTP

DataSource (PSA) Delta DataSource (PSA)

DTP 1 DTP 2Delta Delta

© SAP 2008 / Page 21

Delta Determination

Delta Determination: Check table RSBKREQUEST for all green requests of a DTP

Table RSBKREQUEST: Shows all DTP-RequestsUSTATE (User-set status) = “2” means “processed successfully”

Check table RSSTATMANREQMAP to find which of the Source-Requests were already updated

Table RSSTATMANREQMAP: Mapping Target-Request – Source-Request

Next delta-request updates all Source-Requests which were not yet updated before

© SAP 2008 / Page 22

Reset Delta-Status

Reset Delta-Status: Delete Request(s) from DataTarget Deleted requests disappear from mapping-table

RSSTATMANREQMAP and therefore are picked up with next delta-upload

No other status has to be reset (like “DM-Status” in BW 3.x)

No „repeat“-request necessary, just normal delta-request

Table RSSTATMANREQMAP

DataTarget

© SAP 2008 / Page 23

Multiple DTPs for one transformation (I)

Multiple DTPs can be defined for one transformation: Full-Mode: No constraints Delta-Mode: Disjunctive selection criteria Up to now, separate Delta-Handling for each DTP

Even if source, target and transformation are the same!

In future, it will be possible to “merge” delta-DTPs with disjunctive selection criteria, so that only one DTP is necessary for delta-processing of both DTPs.

DataSource (PSA)

Delta-DTP 1Selection: Customer A

Delta-DTP 2Selection:Customer B

Delta-DTP 1Selection: Customer A, customer B

DataSource (PSA)

Merged

© SAP 2008 / Page 24

Multiple DTPs for one transformation (II)

You can upload mixed full- / delta-requests to any DataTarget Even to DataStore-Objects “Full-Repair”-Request is not necessary anymore User has more responsibility which data is uploaded in which order!

No “Reconstruct” possible anymore with DTP “Reconstruct”-tab does not show DTP-requests Not necessary due to separate delta-handling for each DataTarget

© SAP 2008 / Page 25

Where-Used List

Available with SP07, the Source-Object offers a Where-Used List about the requests updated to the connected DataTargets.

Source-Object:

The Where-Used List shows: To which DataTargets the

request was updated. Which DataMart-requests have

to be deleted before this request can be deleted.

Attention: deletes the request itself, not just a status.

© SAP 2008 / Page 26

Data-Consistency and Delta Invalidation (I)

To guarantee Data-Consistency and prevent Delta Invalidation: It is not possible to delete a source-request without deleting its DataMart-

requests before. PSA-requests (which often are the “source-request” of a DTP) can of course

be deleted. This is guaranteed with table RSMDATASTATE:

When a Delta-DTP is started, pointer DMEXIST is increased to prevent the deletion of the source-request.

InfoCube DMEXIST DMALL

Cube A 2 1

Cube ARequest 1Request 2

Cube BRequest 1Request 2

Cube CRequest 1

RSMDATASTATE:

© SAP 2008 / Page 27

Data-Consistency and Delta Invalidation (II)

To guarantee Data-Consistency and prevent Delta Invalidation: When an InfoCube is source for a DTP, you can only compress requests which

are already updated in all connected DataMarts. This is guaranteed with table RSMDATASTATE:

As long as there is still a Delta-DTP or Export-DataSource that has NOT yet picked up the request, DMALL is not updated and the request cannot be compressed.

When ALL Delta-DTPs AND Export-DataSources have picked up the source-request, the pointer DMALL is increased.

InfoCube DMEXIST DMALL

Cube A 2 1

Cube ARequest 1Request 2

Cube BRequest 1Request 2

Cube CRequest 1

RSMDATASTATE:

© SAP 2008 / Page 28

Delta without Data Transfer

The delta-status for a DTP can be set to „already picked up“ for all records in the Source-Object without transferring data.

© SAP 2008 / Page 29

How to switch Delta from InfoPackage to DTP?

The already-existing Delta data flow using InfoPackages can be switched easily to the new Delta data flow using DTP without data loss.

© SAP 2008 / Page 30

1. Data Transfer Process2. Monitoring3. Delta4. Error-Handling5. Manual Update6. Roadmap and Useful Transactions

Agenda

© SAP 2008 / Page 31

Error Handling Overview

DTP Scheduler

Error StackError Stack

DataSource (PSA)IP DTP

No error handling for InfoPackages. In case of invalid records, data needs to be reloaded from the sourcesystem.

Invalid records can be corrected in the error stack and updated into the data target

Source System

Error DTP

© SAP 2008 / Page 32

Captured errors

Un-allowed characteristic values Lower case letters Arithmetic and conversion errors User built routine with return-code <> 0 Master data read unsuccessful Currency translation or time conversion error Checks during Master data and Text update No SID for navigational attribute No language in text upload Double records concerning the key Overlapping or invalid time intervals Data does not map with the scheduler selection “Do not update, when no master data exists” Errors in Hierarchy structure Overlapping time intervals No SID for characteristic values

© SAP 2008 / Page 33

Error Handling

Error Handling – Possibility to choose in the scheduler: 1. option: Once errors occur, the whole data package is

terminated. The request is not released for reporting. 2. option: Valid records are updated. After manual release of

the request, data is valid for reporting. 3. option: Valid records are updated and available for

reporting.

© SAP 2008 / Page 34

Error Handling

You can define the Number of wrong records which lead to a wrong request.

If you select the indicator ‘No Aggregation Allowed’, the request is regarded as incorrect if the number of records received in BW does not match the number of updated records.

© SAP 2008 / Page 35

Data Transfer Process Monitor

Data display in temporary storage

Data display in Error-Stack

© SAP 2008 / Page 36

Temporary Storage and Error Stack

© SAP 2008 / Page 37

How to reload from Error Stack

After the correction of the data in the Error Stack, the data can be reloaded using the “Error DTP”.

The “Error DTP” has to be created once in the original DTP, can be used like a normal DTP afterwards.

© SAP 2008 / Page 38

Error Handling with DataStore Objects

Error-Handling is now also possible for DataStore Objects: In BW 3.x, this was not possible for ODS-Objects. Due to the

„Overwrite“-functionality, data consistency could not be guaranteed. Now new records with the same key as the erroneous ones are also

filtered out, when you define a key for the error stack.

© SAP 2008 / Page 39

The Key of Error Stack

Choose key of error stack as detailed as possible Maximum number of key fields: 16 A detailed key leads to a small amount of entries in the error stack.

Only the key fields of the DataStore Object can be defined as key for the error stack (all key fields or only some of them).

© SAP 2008 / Page 40

Example: Error Stack (I)

The following data records will be written into Error Stack: Incorrect data records Records with the same key of error stack within the same data package and

same request, that come after the error record.

Request number

Data Rec.

Order number

Calendar Day Quantity

1 01 1000 01.07.2005 %&§

1 02 1000 03.07.2005 10

1 03 1000 02.07.2005 70

1 04 1001 04.07.2005 20

1 05 1002 01.07.2005 30

Error StackKey: Order Number

Correct records

Records with

same key as

incorrect ones

Incorrect record

© SAP 2008 / Page 41

Example: Error Stack (Ia) – One request

Assumption One record is marked red in a custom-developed transformation routine The key of the error stack is order number

Result Records 2 and 3 will be written to the error stack

Request number

Data Rec.

Order number

Calendar Day Qty

109882 01 1000 01.07.2005 30

109882 02 1000 03.07.2005 50

109882 03 1000 02.07.2005 70

109882 02 20050703 1000 50

109882 03 20050702 1000 70

Error Stack

© SAP 2008 / Page 42

Example: Error Stack (II)

Once the request in DataStore Object is deleted, the related data records in error stack are automatically deleted.

Request number

Data Rec.

Order number

Calendar Day

Quantity

2 01 1000 03.07.2005 50

2 02 1001 01.08.2005 20

2 03 1011 02.08.2005 10

The following data records will be written into Error Stack: Incorrect data records Records with the same key of error stack within the same data package and same

request, that come after the error record. Records with the same key of error stack within different requests

Request number

Data Rec.

Order number

Calendar Day

Quantity

1 01 1000 01.07.2005 %&§ Error StackKey: Order Number

Record with same

key within different

requests

Incorrect record

Correct records

© SAP 2008 / Page 43

Example: Error Stack (IIb) – More requests

Result As the first request for order number 1000 contains an error, all records with the same

key of the same request and all records with the same key in subsequent requests will be written to the error stack.

Request number

Data Rec.

Order number

Calendar Day Qty

109882 01 1000 01.07.2005 30

109882 02 1000 03.07.2005 50

Request number

Data Rec.

Order number

Calendar Day Qty

109883 01 1000 02.07.2005 70

109882 02 20050703 1000 50

109883 01 20050702 1000 70

Error Stack

© SAP 2008 / Page 44

Error Handling

Monitor entry

Abort of update

Upd. valid records

Marked in tem. storage

Update into Error Stack

Color of Request

No Error handling X X redError handling

No update, no reporting X X X red

Update valid records, no reporting

X X X Xred

Update valid records,

Reporting possible

X X X Xgreen

Differences between error-handling variants:

© SAP 2008 / Page 45

Error Handling - Summary

Possibility to choose in the scheduler: To abort process when errors occur To process the correct records but do not allow reporting on them To process the correct records and allow reporting on them The Number of wrong records which lead to a wrong request

Error-Handling can be used for DataStore Object and Delta now!

Invalid records can be written into an error stack.

Keys should be defined for error stack to enable the error handling of DataStore Object.

Temporary data storage can be switched on/off for each sub-step of the loading process.

Invalid records can be updated into data targets after their correction.

© SAP 2008 / Page 46

1. Data Transfer Process2. Monitoring3. Delta4. Error-Handling5. Manual Update6. Roadmap and Useful Transactions

Agenda

© SAP 2008 / Page 47

Manual Posting from Monitor

Possibility to reload a red request from the monitor without deleting the request in the DataTarget

Reloads always the complete request

© SAP 2008 / Page 48

Manual Posting from Monitor

The processing mode „Serially in Dialog Process“ indicates that the request was posted manually.

© SAP 2008 / Page 49

When can a request be posted manually?

Two scenarios where a request can be posted manually:

Scenario 1: The Source is a PSA-request, which is still available. The DataPackages are not changed by the DTP, so that the system knows before

the extraction which DTP-package contains which source-package. Then the PSA-packages can be read selectively by the manual posting.

This does NOT work with semantic groups (error stack, master data upload).

Scenario 2: The Source is an InfoCube or DataStore Object. The main process has finished and created a temporary storage of each

package. Then the manual posting is restarted from the temporary storage.

© SAP 2008 / Page 50

Request Reverse Posting

Advantage Disadvantage

Delete Request More flexible (also possible for ODS, no PSA necessary)

Either deactivate compression of aggregate after rollup or deletion of aggregates necessary

Request Reverse Posting

Possible for already compressed requests in

aggregates

Not possible for DTP-Requests

Restricted to InfoCubes

Just for upload via PSA

No further development on this feature

© SAP 2008 / Page 51

1. Data Transfer Process2. Monitoring3. Delta4. Error-Handling5. Manual Update6. Roadmap and Useful Transactions

Agenda

© SAP 2008 / Page 52

Steps in Loading Process

SAP BW

SAP-Syste

mExtraction job

InfopackageScheduling

ALE

ALE ALE

ALE

SAPISAPI

Transfer RulesTransfer Rules

Update RulesUpdate Rules

Communication structure

Transfer Structure

Transfer Structure

tRFC

ALE

PSA

1

23

4 5 67

8tRFC

10

9

© SAP 2008 / Page 53

Steps to analyze Loading Problems (1)

1. Load only to PSA / check whether data arrived in PSA

YES NO

Conclusion:

Cause of problem is in BW-system

Conclusion:

Problem in source system

or communication between systems

Analyze monitormessages, transferrules, update rules(see simulation slides )

Analyze source system (next slides)

78

1

32

4

56

109

© SAP 2008 / Page 54

Steps to analyze loading problems (2)

Monitor in BW-system SAP source system

No data arrived in PSA : Check extraction job in SM37 (in case of SAP source system) Naming convention of job:

BI + <Requestid> or BI_PROCESS_LOADING

© SAP 2008 / Page 55

Extraction job

Possibilities There is no job:

The job cancelled:

The job ran successful, but no data in PSA:

1 2 3

6

5

4

87

© SAP 2008 / Page 56

Request error analysis (1)

Check for short-dumps in BW (ST22) Check selection routines / selection in

InfoPackage

Check tRFC queue (SM58) in BW system for entries and errors

1

2

© SAP 2008 / Page 57

Request error analysis (2)

Check source system connection in RSA1

Check that the right destinations are entered in both systems (SM59)

Check for not processed IDocs (BD87) in both systems (SAP Note 555229)

Check that remote-user in R/3 has enough authorities for background job processing and extraction.

3 4

© SAP 2008 / Page 58

Request error analysis (3)

Check extraction job log (SM37) for error messages ( shortdump, syslog)

Extractor checker (RSA3) Check entries in Delta Queue (RSA7),

see also SAP Note 380078 Check source system connection in

RSA1 Check that the right destinations are

entered in both systems (SM59) Check for not processed IDocs (BD87)

in both systems (SAP Note 555229)

5

6

7

© SAP 2008 / Page 59

Request error analysis (4)

Check tRFC queue (SM58) in source system for entries and errors

Check in BW for dumps and syslog entries

Analyze BW Monitor messages Use simulation

8

9 10

© SAP 2008 / Page 60

Thank you!

© SAP 2008 / Page 61

Copyright 2008 SAP AGAll rights reserved

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice.

Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.

SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, Duet, Business ByDesign, ByDesign, PartnerEdge and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned and associated logos displayed are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.

The information in this document is proprietary to SAP. This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This document contains only intended strategies, developments, and functionalities of the SAP® product and is not intended to be binding upon SAP to any particular course of business, product strategy, and/or development. SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, or other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, or non-infringement.

SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation shall not apply in cases of intent or gross negligence.

The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages

Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durch SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden.

Einige von der SAP AG und deren Vertriebspartnern vertriebene Softwareprodukte können Softwarekomponenten umfassen, die Eigentum anderer Softwarehersteller sind.

SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, Duet, Business ByDesign, ByDesign, PartnerEdge und andere in diesem Dokument erwähnte SAP-Produkte und Services sowie die dazugehörigen Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und in mehreren anderen Ländern weltweit. Alle anderen in diesem Dokument erwähnten Namen von Produkten und Services sowie die damit verbundenen Firmenlogos sind Marken der jeweiligen Unternehmen. Die Angaben im Text sind unverbindlich und dienen lediglich zu Informationszwecken. Produkte können länderspezifische Unterschiede aufweisen.

Die in diesem Dokument enthaltenen Informationen sind Eigentum von SAP. Dieses Dokument ist eine Vorabversion und unterliegt nicht Ihrer Lizenzvereinbarung oder einer anderen Vereinbarung mit SAP. Dieses Dokument enthält nur vorgesehene Strategien, Entwicklungen und Funktionen des SAP®-Produkts und ist für SAP nicht bindend, einen bestimmten Geschäftsweg, eine Produktstrategie bzw. -entwicklung einzuschlagen. SAP übernimmt keine Verantwortung für Fehler oder Auslassungen in diesen Materialien. SAP garantiert nicht die Richtigkeit oder Vollständigkeit der Informationen, Texte, Grafiken, Links oder anderer in diesen Materialien enthaltenen Elemente. Diese Publikation wird ohne jegliche Gewähr, weder ausdrücklich noch stillschweigend, bereitgestellt. Dies gilt u. a., aber nicht ausschließlich, hinsichtlich der Gewährleistung der Marktgängigkeit und der Eignung für einen bestimmten Zweck sowie für die Gewährleistung der Nichtverletzung geltenden Rechts.

SAP übernimmt keine Haftung für Schäden jeglicher Art, einschließlich und ohne Einschränkung für direkte, spezielle, indirekte oder Folgeschäden im Zusammenhang mit der Verwendung dieser Unterlagen. Diese Einschränkung gilt nicht bei Vorsatz oder grober Fahrlässigkeit.

Die gesetzliche Haftung bei Personenschäden oder die Produkthaftung bleibt unberührt. Die Informationen, auf die Sie möglicherweise über die in diesem Material enthaltenen Hotlinks zugreifen, unterliegen nicht dem Einfluss von SAP, und SAP unterstützt nicht die Nutzung von Internetseiten Dritter durch Sie und gibt keinerlei Gewährleistungen oder Zusagen über Internetseiten Dritter ab.

Alle Rechte vorbehalten.