SAP NetWeaver How-To Guide MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry Applicable Releases: SAP NetWeaver Master Data Management 5.5 SAP NetWeaver Master Data Management 7.1 SAP Business Objects Data Services XI 3.1 and above Topic Area: Enterprise Information Management Capability: Master Data Management Version 1.1 June 2010
33
Embed
MDM and Data Services Integration Scenarios - Part II
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
SAP NetWeaver
How-To Guide
MDM and Data Services Integration
Scenarios - Part II: Consolidation On
Data Entry
Applicable Releases:
SAP NetWeaver Master Data Management 5.5
SAP NetWeaver Master Data Management 7.1
SAP Business Objects Data Services XI 3.1 and above
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 1
1. Introduction
SAP NetWeaver Master Data Management (MDM) is a powerful solution to import, maintain and
syndicate master data of all types across different systems. Its in-memory technology allows searching
for master data within the MDM repository with very short response times. You can identify potential
duplicates across systems and manage key mappings as a prerequisite for master data distribution.
For new master data to be stored in the MDM repository – either when creating manually or when
connecting new Client systems to an MDM landscape – it is best practice to perform a data cleansing
and also a matching in order to find duplicate records. MDM itself supports cleansing and matching,
however SAP Business Objects Data Services (DS) provides more comprehensive cleansing and
matching capabilities, e.g., standardizing and correcting names, connecting to international address
data directories such as US Postal directory, supporting different matching algorithms (heuristic, rules-
based, probabilistic, deterministic, empirical, template), etc. Besides this, it provides full ETL
functionality to extract data from any kind of source, transform it, and load it to any destination.
We have identified the following scenarios where both solutions can be combined in order to
complement each other:
Advanced cleansing and consolidation (data migration scenario):
You would like to connect a new client system to your MDM landscape, hence initially load the master data into MDM. Use the full flexibility of Data Services to extract, transform, and load the data into the MDM repository. Run cleansing, validation, normalization, and de-duplication in Data Services prior to loading the data into MDM. In order to match the new data against the existing in the MDM repository, read data via DB views. After having loaded the data in MDM,
syndicate the same to your clients.
Real-time cleansing and enrichment on data entry:
You have chosen MDM as the central repository for your master data. You would like to create new master data in the MDM Data Manager. You can ensure data governance via MDM workflow. On data entry, the data needs to be enriched by calling data quality and enrichment services provided by Data Services. Integration to Data Services is done via the Enrichment
Architecture.
Periodic cleansing, and consolidation of data in MDM repository:
You have chosen MDM as the central repository for your master data. You have noticed that some of your data is incomplete. Furthermore, it might be that some data is outdated, e.g., data must be updated with the most recent address information. So, on a regular basis you need to run cleansing, and consolidation jobs to ensure data integrity. The data needs to be exported, processed and re-imported into the MDM repository which is handled via the Enrichment Architecture.
Central master data creation with data quality check on data entry:
You have chosen MDM as the central repository for your master data. Maintenance of your master data is done from within the Enterprise Portal, either via iViews or Web Dynpro components. You can control data governance via SAP NetWeaver Business Process Management (BPM). Integration to MDM is done via Web Services or MDM Java API. In order to prevent duplicate entries from entering the MDM repository, you call data quality services on data entry such as duplicate check, address cleansing, name cleansing, and normalization, etc.
Integration to Data Services is done via Web Services.
Extracting data for all kinds of sources and data transfer into MDM:
Use native ETL functionality of Data Services to extract, transform, and load your master data into MDM, see also above.
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 2
Master data joined with analytics:
Run data profiling reports provided by Data Services to get an insight of the quality of your master data in MDM, and visualize the results in dashboards.
This document is the second part of a how-to guide series dealing with the integration between SAP
NetWeaver MDM and SAP Business Object Data Services. The goal of the how-to guides is to provide
you with guidelines about how to implement the scenarios above. We will stick to the data migration,
the cleansing on data entry (including periodic cleansing), and the central master data management
scenarios. Implementation teams can use those findings as a jump start for their own integration
projects.
In the first part of the how-to guide series, we cover the data migration scenario. In the third part, we
focus on the central master data creation scenario. You can download those how-to guides from the
SAP NetWeaver how-to guides area on SAP Developer Network (SDN)
https://www.sdn.sap.com/howtoguides.
In this document, we explain how to setup the on entry and periodic cleansing scenarios which are
actually similar from an integration and data flow perspective, and hence dealt with together. The
scenario is described in more detail in the following chapter.
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 6
Select button Add.
Select the respective real-time job, specify a name for the web service, select the server where the jobs should run, and apply your settings.
Note
The service name has to be identical to the port code of the MDM inbound and outbound
ports.
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 7
3. Go to Web Services, and switch to tab Web Services Configuration to configure your web
service, e.g., enable or disable security settings.
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 8
4.2 Setup of Enrichment Architecture ...
1. Download the latest Enrichment Controller release from the SAP Service Marketplace, area
Software Distribution Center http://service.sap.com/swdc (note: logon required), Support
Packages and Patches Entry by Application Group SAP NetWeaver SAP MDM SAP
NETWEAVER MDM 7.1 Entry by Component Enrichment Controller 7.0.
The Enrichment Controller package comes with an SCA file, a sample configuration file, a sample adapter, and simulator files. Latter can be used to test your enrichment environment
once it has been setup.
2. Deploy the SCA on the SAP NetWeaver Application Server Java. As a prerequisite you have
already deployed the MDM Java API.
Note
As of SAP NetWeaver MDM 7.1 SP04, the SAP Business Objects Enrichment Adapter is delivered together with the SAP MDM Enrichment Controller. Before, this had to be deployed separately. For more details, please refer to the SAP NetWeaver MDM 7.1 SP04 release notes http://help.sap.com/saphelp_nwmdm71/helpdata/en/4b/58416df2ec4cb9e10000000a421
89b/frameset.htm.
3. Copy the folder MDM_EnrichmentController to any location on your server. The folder contains
the configuration file MDM_EnrichmentControllerConfiguration.xml and the respective xsd
scheme.
Maintain the following parameters in the configuration file (see also complete configuration file in
the appendix):
a. Host names of MDM server, MDIS, and MDSS
b. MDM Repository name, here MDMBOBJ_BP
c. Enrichment user, here Enrich
d. Main table, here MDM_BUSINESS_PARTNERS
e. Enrichment provider system, here BOBJ. This has to be identical to the remote system
code as defined in the MDM Console
f. Enrichment service port code as defined in the MDM Console, here
DQAddressCleansing. The enrichment service port represents the data quality
service. The parameter must be identical to the name of the respective web service.
g. Workflow name, here DQAddressCleansingWF
h. Name of the workflow step that triggers the syndication, here SendToEC
i. Enrichment Adapter:
i. vendorName="sap.com"
ii. applicationName="tc~mdm~enrichment~bobj~adapter"
iii. ejbName="DQMDMAdapterBean"
Note
The sample configuration file already contains a section for the DQ MDM Adapter for the simulator repository. You can use this as a template, and adapt accordingly.
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 9
j. Time-out parameters. You have the possibility to specify three different time-out
parameters: for the MDSS, the Enrichment Adapter, and the MDIS.
Note
Since the adapter supports synchronous communication only, you may easily run into a time-out when dealing with a larger number of records. You may have to increase the several time-out parameters to suit the number of records that shall be processed within one workflow job. Even though you can adapt the time-out parameters, it is not feasible to send the complete set of records in one run, e.g., for the periodic cleansing use case,
you may have to split the overall number of records in consumable chunks of data.
4. Start the Visual Administrator, and select the Configuration Adapter service to change the
property sheet of the Enrichment Controller. Navigate to Configurations apps sap.com
tc~mdm~enrichment~controller appcfg Property Sheet application.global.properties, and
switch to edit mode.
Maintain the following parameters:
a. Change the property ApplicationRootFolderPath. It has to point to the folder that contains
the MDM_EnrichmentController folder.
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 10
b. For each user maintained in the configuration file, a parameter is added to the property
sheet which securely stores its password. The format of the parameter is as follows:
<User>@<MDM Repository>@<MDM Server>
Maintain the password of the enrichment user.
Note
As of SAP NetWeaver MDM 7.1 SP04, the password is no longer maintained in the configuration file. For more details, please refer to the SAP NetWeaver MDM 7.1 SP04 release notes http://help.sap.com/saphelp_nwmdm71/helpdata/en/4b/58416df2ec4cb9e10000000a421
89b/frameset.htm.
5. Maintain the HTTP destination URL for the DQ Adapter's Web Service proxy and the
Enrichment Controller Web Service proxy.
In the Visual Administrator, select the Destinations service, and switch to tab Runtime.
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 12
4.3 Enhancements in MDM ...
...
1. Create lookup table DQStatuses. This table stores the status codes and status descriptions of
the DQ job such as run was successful, address conflicts occurred, country was missing, etc.
Define both fields Status Code and Status Message as display fields.
The result of the DQ job only provides the status code. However, you may like to get the code and a meaningful status text displayed, i.e., you need to map from one source field to two display fields. This can be achieved by enabling Key Mapping, and using the remote-key during mapping. For more details, please refer to the ‘Good to Know Topics for a Smooth SAP Netweaver MDM 7.1 Implementation’ how-to guide, chapter 6.8 importing single source field to multiple display fields http://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/70b0caa1-92e5-
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 13
To initialize the lookup table DQStatuses, get a text file from DQ containing all status codes and
texts, and import the same. Map the status code to the remote key.
2. In order to log status information and error messages occurring during the enrichment process,
you have to enhance the MDM repository schema.
In MDM Console, you have to create an Enrichment Statuses lookup table, and add an Enrichment Status field of type qualified lookup to your main table.
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 14
Note
For more details about the structure of the lookup table, please refer to the SAP Help Portal http://help.sap.com/saphelp_nwmdm71/helpdata/en/44/ab794565d37455e10000000a11
466f/frameset.htm.
Optionally, the repository schema can be automatically enhanced in the MDM Enrichment Controller Administration Tool. Launch the administration tool via URL http://<EC Host Name>:<EC Port>/webdynpro/dispatcher/sap.com/tc~mdm~enrichment~ui/MDM_EC.
Prerequisite is that you have assigned role MDM EC Administrator to your user.
Switch to tab Administration, and select link Update Schema.
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 15
Choose the respective repository, and select button Update Schema. If the Enrichment Status
structures are already available, the button is deactivated.
Note
You need to authenticate at the repository. Furthermore, you need the respective authorization to enhance the repository schema.
During the update, the repository is unloaded, so ensure that no other users are logged
on.
Once the schema has been created, add field DQ Status to the Enrichment Statuses table as a lookup to the DQStatuses table.
3. In MDM Console, add a new user which is used for the communication with the Enrichment
Controller, here Enrich.
4. In MDM Console, create a new remote system representing your service provider, here BOBJ
(note, this corresponds to settings in the Enrichment Controller Configuration file).
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 16
5. Import XML schemas for the web service request and response messages.
6. For the beforehand created remote system, create inbound and outbound ports. Once import
and syndication mappings have been created (see below), assign them to the ports.
The processing type of the outbound port has to be manual since syndication is triggered by the Enrichment Controller.
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 17
The processing type of the inbound port has to be automatic to ensure that the response of the
Web Service is automatically imported to MDM.
Note
The codes of the ports have to be equals the name of the Web Service.
7. In MDM Syndicator, create a syndication map for the request message.
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 18
8. In MDM Import Manager, create an import map for the response message.
a. Add EC_Port field to the source of the import map, and map it to the Enrichment Status
field which is the non-qualifier of the Enrichment Statuses table. The mapped value has
to be the port code representing the enrichment process, here DQAddressCleansing.
Set the qualified update options for the selected qualified lookup destination field Enrichment Status as follows:
Select Set Qualified Update Update from the context menu.
For new links choose Skip, for existing links choose Update (All Mapped Qualifiers).
Note
The qualified link is added by the Enrichment Controller at the beginning of a new enrichment job, the import updates this link. So, you do not have to create new links. If you do not specify any matching qualifier, matching is done by the non-
qualifier only, here the port code.
If you do not like to keep the history status information, do not select any matching qualifiers. A new enrichment run will overwrite the status of an old and already finished
run whereas the status entry will be updated for the same enrichment run.
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 19
If you like to keep the status information of finished jobs (history), i.e., you rather like to add status entries, add the GUID or the Job ID to the Matching Qualifiers. The rest of the settings remain unchanged. This however requires that GUID or Job ID is syndicated and routed through the enrichment job to be available during import of the response.
b. Add Completed field to the source of the import map, and map it to the Enrichment
Status Completed field. Map the value to TRUE to indicate that the enrichment process
was successfully completed.
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 20
c. Map the STATUS field to the Enrichment Status DQ Status destination field. The
STATUS field holds the status information of the DQ job. Since we enabled Key
Mapping, auto-mapping is based on the remote-keys.
9. In MDM Data Manager, create a workflow, here DQAddressCleansingWF.
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 21
Create at least two process steps in a row. The first process step triggers the enrichment
process, here SendToEC. As mentioned above, the name of the workflow and the process step
has to correspond with the name maintained in the Enrichment Controller Configuration file.
Both steps have to be assigned to the enrichment user. See also SAP note 1341061.
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 22
In case that you like to use check-out and check-in functionality, e.g., you first like to run through an approval step before checking-in the records, check-out the records at the beginning of the
workflow, and check-in at the end.
In the MDM Console, change the user of the MDIS which is used to communicate with the MDS to the enrichment user. Otherwise, you run into an error during import.
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 23
4.4 Runtime and Monitoring ...
1. Once you have done all settings in the Enrichment Controller and SAP MDM, go to the System
Monitor of the MDM Enrichment Controller Administration Tool. It shows you the status of any
single configuration step. Furthermore, you have the possibility to test the enrichment service
before running the end-to-end process.
2. Create a new record in Data Manager. This automatically launches a new workflow job,
assuming that you have configured the workflow settings accordingly, see above. Once the
trigger step in the workflow is reached, the Enrichment Controller triggers the syndication of the
data, and writes the current status in the Enrichment status field.
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 24
During import of the result of the DQ enrichment run, the Enrichment Status is updated: the time
stamp is updated, the Completed field is set to TRUE, and the DQ Status is added.
In case of a time-out, a respective error message is logged in the status field. The Completed field is set to TRUE, so that the workflow job carries on. You may like to add a branch and a
validation step in the workflow to control the proceeding.
3. You can monitor the DQ real-time job in the Data Services Management Console. Navigate to
Administrator Real-Time <host> Real-Time Services, and select your service.
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry
June 2010 25
This provides you statistical information about your service. Select the process ID to get more
details.
From here, you can also access trace log and error log to analyze errors.
MDM and Data Services Integration Scenarios - Part II: Consolidation On Data Entry