Siebel Data Loading Best Practices on SQL Server Frank Earl McBath [email protected].

Siebel Data Loading Best Siebel Data Loading Best Practices on SQL ServerPractices on SQL Server

Frank Earl McBathFrank Earl McBath

[email protected]@microsoft.com

How many of you have used SQL Server and EIM?

Objectives

What is EIM?

What makes SQL Server different than other platforms?

What can we optimize?

How do we optimize? What tools can I use? What techniques can I try?

What is EIM?

EIM is the process that Siebel uses to load data into the database.

EIM is used on every platform.

It validates the data (PK/FK relationships) and generates unique ROWIDs.

Every customer uses it.

It tends to be the first problem that every customer hits with Siebel.

You cannot bypass EIM.

Symptoms of EIM?

The first several thousand will always go fast.

Performance deteriorates over time in a logarithmic pattern. Why? Because the b-trees grow to more levels. Ex. The first 5K rows go in at a rate of 2 minutes. After 2 weeks of

loading that same 5K rows takes 1 hour.

What makes EIM query different? (vs. a normal Siebel query)

EIM is batch oriented

Have better logging

Typically more complicated

Reads system catalog on each run.

You can alter the indexes on the EIM tables.

Typically less configurable than a business rule.

Common Siebel Performance Problems

Docking replication turned on Too many indexes Wrong indexes Clustered index issues Table spools (non-EIM) Implicit cursor conversion (non-EIM) Too much meta data being loaded Batch size not optimum Blocking/Deadlocks while loading in parallel Stats are stale (EIM only) Maybe hints in the Siebel EIM job

The pre-EIM Loading Issue

Siebel provides EIM, but no mechanism to populate the EIM_ tables.

Every customer has to invent the wheel.

Some customers do it good… others not. Don’t distribute over network Don’t validate/scrub while loading

What the Alliance is working on… A better wheel.

What are the typical problems with EIM? (I)

Too many indexes Both on EIM_* tables… and… Base tables

Incorrect indexes Poor selectivity

What are the typical problems with EIM? (II)

You are allowed to modify/add/drop to the EIM_* tables IMPORTANT: You are not allowed to change base table indexes

without ES approval. Build a case and present it to them. They use “expert” mode for Siebel Tools and alter the meta

data.

Poor statistics & Bad plans

Tools to Use (I)

SQL Server Profiler Gives you a SQL Server kernel view of the query.

Level 8 EIM Trace Will show you network times, etc… Will not always show you hints Will not show you plans

Tools to Use (II)

Index Wizard Will never suggest a better index. Why? Because Siebel indexes

just about every possible configuration! What it will show you is what indexes ARE being used. Hence, you

can deduce what indexes ARE NOT being used.

DBCC SHOWSTATISTICS Will show you what indexes are 100% of one value. Probably not being used.

Tools to Use (III)

ISQLW Use ODBC wrapper

Need to use proper “wrapper” Will get incorrect plan if you do not

SET STATISTICS PROFILE Will show plans with costing

SET STATISTICS IO Shows the IO only

Only use one at a time. DBCC DBREINDEX UPDATE STATISTICS Long Running Query

SQL Profiler

Powerful tool, but can burn up a lot of CPU if you are not careful.

Not a “Black Box” flight recorder. Use it sparingly.

Filter on the SPID or CLIENT PROCESS

Capture on the following Events: Stored Procedures -> RPC:Completed and TSQL -> SQL:BatchCompleted

Save off to a file, then to a table and analyze

How To Optimize EIM (I):

Turn on “Level 8” EIM logging. Very verbose

Turn on SQL Server Profiler for the run.

Run an EIM batch.

How to Optimize EIM (II):

Look at the EIM log. What is the longest execution time? Tip: Load this log into Excel and sort. Then search the original log.

Look at the SQL Profiler trace. Sort on longest duration and then on reads. These queries should match those in the EIM log. Look at the execution plans. Why is it taking a long time? Use Excel to help sort.

Once the problem has been isolated, figure out the fix.

Hints (I) SQL Server:

Siebel uses hints by default. This helps rule based optimizers, but hurts cost based ones.

Big performance gains can be made by just taking the hints out.

Why you just can’t drop the index: the query will not compile

DBCC trace flag will cause optimizer to bypass all hints DBCC TRACEON (-1,8602) Query will compile even if index not there.

Hints (II) Siebel:

From the IFB file configuration

Test with / without Index Hints USEINDEXHINTS, FALSE USEESSENTIALHINTS, FALSE

More IFB File Configuration

Consider using the parameters as appropriate: ONLY BASE TABLES, IGNORE BASE TABLES, ONLY BASE COLUMNS, IGNORE BASE COLUMNS

This will reduce the amount of processing performed by EIM. ONLY and IGNORE are mutually exclusive.

Index Tuning Wizard

Tends not to help much because Siebel has just about every index on there already.

Will not suggest a clustered index.

Just takes a long time to run…

… but… it can help you determine which indexes to drop. By showing you what indexes are used, you can infer which ones

are not.

Which Indexes Are Used?

Look at the Profiler trace from the EIM run.

Within the trace, look at the execution plan. Will tell you which indexes used. Remember, you have to capture with the following:

Event: Performance -> Show Plan Statistics Data Columns: BinaryData Output Example:

Clustered Index Scan(OBJECT:([master].[dbo].[sysobjects].[sysobjects]))

Index Removal (I)

Why remove indexes? Penalties on INSERTS/DELETES which is what the bulk of EIM

Many indexes are not used

Only keep the EIM indexes that are used.

Fewer indexes mean that there are fewer choices and fewer chances of a “wrong plan” being generated.

Less information to the optimizer can be good sometimes.

Index Removal (II)

Feel free to remove indexes on the EIM_* tables

Work with Siebel ES to remove un-needed indexes on the base tables.

Build a case. For example, show that a column is 100% NULL, etc… Optimizer probably will never use single value indexes.

Only in a “COUNT” on that column would it use it.

Index Removal (III)

DBCC SHOWSTATISTICS Look at “Density” and “Steps”

Does the Table have a Clustered Index?

You’d be surprised… Many tables don’t

Run the script in the notes to find which ones don’t

Compare the results to your mappings and ER diagram

Ask Siebel why they don’t have a clustered index…

Clearly Think Out Your Clustered Index

Smart design can help in covering the index

Non clustered indexes are built on top of clustered indexes Point to the cluster key Every Siebel query has ROWID in it, that’s why it’s the clustered

index on all base tables.

Look at DBCC SHOW STATISTICS

Index Strategies (I)

A successful strategy used has been: Analyze what columns are being used Put them in the clustered index

Use “BOE” method for selectivity Drop all non-clustered indexes from the EIM table.

The premise is that EIM is operating on every row in the batch anyways, so why not just scan them all.

Is it worth the extra overhead of more indexes for what amounts to be a scan?

Index Strategies (II)

This strategy works because the BATCH_NUM “firewalls” the scan from a full table scan. The BATCH_NUM is the first column in the clustered index. Thus,

bounding the scan.

Performance Testing

After query is isolated…

Make a baseline Use ISQLW Use SET STATISTICS IO ON Use SET STATISTICS PROFILE ON

(One at a time. Turn off between.) DBCC DROPCLEANBUFFERS DBCC FREEPROCCACHE

Change/Add index, rerun the query with IO and PROFILE.

The McBath Oil Argument

The Oil Argument

Do you notice when you change the oil in your car? Your engine does.

Looking at something that is massively iterated on and saving 1 IO.

Reindex, fill factor

Resequencing columns in an index Check IO with SET STATISTICS IO

Fragmentation Happens…

During big data loads

Run DBCC REINDEX to correct

Think about: Fill Factor Pad Index

Look at sample script to defrag

Defrag & update stats between runs

DBCC INDEXDEFRAG, REINDEX, and drop recreate Samples and run times Pro’s and con’s

Stale Statistics (I)

What happens when stats get old… Bad plans. A query that normally runs in a few seconds can take 30

minutes.

How do the stats get stale? EIM updates every row in the EIM_* table. The thread that auto updates stats doesn’t “wake up” in time

between runs. Small tables will never be updated.

Stale Statistics (II)

Correct this by running UPDATE STATISTICS between runs or a SQL AGENT job that wakes up and runs.

Consider turning off auto update stats for the data load.

It’s all about getting predictable performance.

Multiple EIM Batches

Number of batches directly related to your clustered index design

Good index will keep deadlocks from happening

You can run multiple batches against the same _IF table!

EIM Partioned tables

Use SP_INDEXOPTION to enable row level locks only. Cuts down on blocking issues.

Efficiently Loading Data (I)

Load into pre-staging tables Scrub in tempdb

Minimal logging

Scrub in SQL Server: Efficiencies of cache Use set wise processing, not cursors If have to use cursors, use Fast Forward/Read Only

Run all local on the database server. Not distributed over the network.

Efficiently Loading Data (II)

BULK INSERT vs BCP BULK INSERT is “in memory” BCP is more configurable

Both are single threaded Only run on one CPU

Run multiple BULK INSERTS at once across multiple CPUs.

If the order of the data is not a concern, or you'd rather take the hit when creating the index, it's best to run BULK INSERT into the EIM tables in parallel by deploying a separate thread per CPU. You can use the TABLOCK hint to allow multiple non-blocking inserts.

Efficiently Loading Data (III)

Rules of thumb for previous: Use only 2 - 3 threads at max (only if you have the processors) Limit the batch Commit Size (batch size) to about 2,000 rows per batch. Adjust up or down based on your testing. Remember, if loading in clustered index sequence, only use one thread.

Bulk operations are very high performance. They do log. Conditions in BOL (ex. TABLOCK)

Efficiently Loading Data (IV)

Disk Slices Even with a SAN, break out the following on separate slices if

possible: EIM_* Tables TEMPDB Base (DATA) Tables INDEXES

Do this by dropping the table and recreating on a different file group

Cuts down on fragmentation and contention

Efficiently Loading Data (V)

RAID and EIM Due to the constant UPDATE, INSERT and DELETE, try and use

RAID 0+1 if possible. Parity bit calculation penalty can be significant.

See “Balanced System Design” for more information on slicing

Size the Siebel database (siebeldb or whatever it is named in production) appropriately, and ensure that it will not have to autogrow during the process; that will hurt disk I/O and performance.

Efficiently Loading Data (VI)

When running EIM itself, run processes in parallel. Set different batch numbers, but they can be executed against the same interface tables.

Try and run from “opposite” ends of the batch range. Can help cut down on the blocking. Ex. Run 1 & 5 at the same time, not 1 & 2.

Test to see how many threads can be run on your system. Start with two and add as appropriate.

If you are blocking and getting lock escalations, use sp_indexoption and set the clustered index to no page locks. See BOL for more information.

Efficiently Loading Data (VII)

Disable any triggers on the databases (such as workflow triggers) and then re-apply/enable them after the process is done.

If this is not for an initial data load, this means that Workflow Manager or Assignment Manager will not function during the load for the new or updated data.

Efficiently Loading Data (VIII)

Load multiple base tables from one interface table.

In the IFB table, set the parameter USING SYNONYMS to FALSE only if you are not associating multiple addresses with accounts.

If you have a 1:1 ratio, you are telling the EIM process that account synonyms do not require processing during the import, reducing the amount of work EIM needs to do (as well as load on the system).

Recovery Models During EIM

Use SIMPLE or BULK LOGGED if possible.

Run a full backup freqeuntly during the day.

Weigh the issues of recovery vs. lost data.

Note: Switching from FULL to SIMPLE will break the log chaing and have recovery consequeneces. Always make a full backup after switching to SIMPLE.

Bringing It All Together (I)

Optimize your EIM Batch Size Hint Removal: Siebel and SQL Server Turn off docking replication Get rid of workflow triggers Only load up the tables needed from the Siebel meta data. Loading

the whole catalog can represent 25% the time of your whole batch run.

Run batches in parallel Exclude validation of non-used data. If you know something is

never NULL (ex. EmpID), then don’t check for it.

Bringing It All Together (II)

Update Stats Stale stats can cause problems Investigate turning autostats off

Defrag between large runs Defrag both EIM_ and base tables On large initial loads, put fill factor and pad index to 50%. Cut down on page splits.

Default is 100% full.

Use minimally logged operations to load and scrub. Bulk Insert, SELECT/INTO Recovery Models Run all data loading locally Scrub data inside SQL Server. No cursors.

Make the right indexes Try monster clustered index only Get rid of unused indexes Add them back after runs Work with ES to resolve support issues.

Bringing It Together (III)

Slice your disk right More spindles = More performance. Don’t believe the vendor when

they say cache will solve your problems. It helps hide them. No RAID 5 if possible Separate Data, Log, Indexes

Questions?

Frank McBath

[email protected]