© 2012 Wellesley Information Services. All rights reserved. Preventing, Diagnosing, and Resolving the 20 Most Common Dashboard Performance Problems Dr. Bjarne Berg Comerit
Feb 23, 2016
© 2012 Wellesley Information Services. All rights reserved.
Preventing, Diagnosing,
and Resolving the 20 Most Common Dashboard Performance ProblemsDr. Bjarne BergComerit
Deployment,Testing & Change
Management
Best-in-Class Dashboards
Performance & Security
Options & Prototyping
• Scoping vs. requirements gathering • KPI definitions• Required skills and resources• Data connectivity deep dive• Key criteria to retrieve data sets
• Dashboards vs. reports• Answers to dashboard FAQs• SAP BusinessObjects Dashboards
4.0 overview• Product updates and
implementation criteria• Recent changes to dashboard
terms
• Key dashboard roll-out decisions• Mobilizing your dashboard• Support organization• Volume, stress, and UAT• Training and change management
Day 1
Day
3
Seminar Roadmap
Landscape, Connectivity &
Sizing• Hands-on lab: Build a dashboard
with BOBJ Dashboards 4.0• Sizing and scaling recommendations• User management and access
control• SAP NetWeaver® BW
Accelerator and SAP HANA
• Hands-on lab: Advanced techniques• Web service integration and Adobe
Flex Builder• Panel discussion: Dashboard Projects
•Ownership and branding•Post-production changes
4
678
9
12 2
Customization, Branding & Governance
Day 2
• Common causes of poor dashboard performance
• Effective performance testing• Performance-enhancing design
techniques• Preventing unauthorized access
to dashboards• Password protection
and SSO
14
13
11
5
3
We are here10
1
3
Background
• We already covered hardware sizing, compatibility, and server options in an earlier session; so now we will look at the application, design, and interfaces
• We will specifically look at dashboard design, query design, connectivity impacts, in-memory processing options, as well as dashboard performance monitoring options
4
Functionality vs. Performance: What Wins?
5
What We’ll Cover …
• Choosing the right connectivity and back end • Exploring query performance• Thinking about the dashboard design• Increasing query performance with infrastructure and in-memory
processing• Leveraging pre-caching capabilities and aggregates• Obtaining strategies for performance testing: Load and stress• Looking at EarlyWatch Reports and the performance checklist• Wrap-up
Problems #1 and #2: Connectivity and Performance
• As we covered in the earlier session, the type of connectivity matters for the performance
Always pick the fastest interface available for the data source you are building dashboard onSource: SAP AG, 2012
• BICS connectors perform well
• Avoid the MDX interface(it is slow)
• Avoid direct access to theInfoProviderssince thisbypassesthe BI analyticalengine in SAP NetWeaver® BW
6
Problem #3: Data Connectivity — SAP Crystal Reports and SAP BusinessObjects Live Office
• You can use transient providers to create real-time dashboards on top of SAP ERP data
• You can also use SAP Crystal Reports for detailed drill-down analysis
• If you always use the “refresh on load” option for Live Office connections, your users will experience periodic slow performance
By leveraging the aggregation in SAP Crystal Reports, you can also get faster SAP Dashboards (formerly Xcelsius®) response time 7
Problem #4: Back End — Build on a Solid Performance Foundation
Modularize the data and create sub-sets of data for really fast dashboarding
Generic “metrics” data tables can be created for summarized KPI and scorecard dashboards
The summary, or snapshot, data can be accessed much faster than underlying data tables with millions of records
Real example
9
Problem #5: Back End — Dashboard Performance Architecture
• In this example, the company uses snapshots for performance reasons Dashboards for
executive users Pre-delivered SAP
BusinessObjects Web Intelligence reports for casual users
Ad hoc SAP BusinessObjects Web Intelligence reports for power users
The dashboards are only built on the low-volume daily snapshot cube (this is also placed in SAP NetWeaver BW Accelerator for very high performance)
Real example
10
What We’ll Cover …
• Choosing the right connectivity and back end • Exploring query performance• Thinking about the dashboard design• Increasing query performance with infrastructure and in-memory
processing• Leveraging pre-caching capabilities and aggregates• Obtaining strategies for performance testing: Load and stress• Looking at EarlyWatch Reports and the performance checklist• Wrap-up
11
Problem #6: Query Read Modes
• There are three query read modes that determine the amount of data to be fetched from a database and sent to the application server1. Read all data
All data is read from a database and stored in user memory space
2. Read data during navigation Data is read from a database only on demand during
navigation3. Read data during navigation and when expanding the hierarchy
Data is read when requested by users in navigation
Reading data during navigation minimizes the impact on the application server resources because only data that the user requires will be retrieved
12
Problem #7: Recommendation — Query Read Mode for Large Hierarchies• For queries involving large hierarchies, it is smart to select “Read
data during navigation” and when expanding this option to avoid reading data for the hierarchy nodes that are not expanded
• Reserve the Read all data mode for special queries I.e., when a majority of the users need a given query to slice and
dice against all dimensions, or data mining This places heavy demand on database and memory resources,
and may impact other SAP NetWeaver BW processes A query read mode can be defined on an individual query or as
a default for new queries (transaction RSRT)
Recommendations for OLAP universes and SAP BusinessObjects Web Intelligence analysis
Use of hierarchy variable is recommended Hierarchy support in SAP BusinessObjects Web Intelligence for SAP NetWeaver BW
is limited The Use Query Drill option significantly improves drill-down performance Look at the Query Stripping option for power users
Problem #8: Reduce the Use of Conditions and Exceptions Reporting• Conditions and exceptions are usually processed by the
application server This generates additional data transfer between database and
application servers• If conditions and exceptions have to be used, the amount of data
to be processed should be minimized with filters When multiple drilldowns are required, separate the drill-down
steps by using free characteristics, rather than rows and columns
• BENEFIT: This results in a smaller initial result set and, therefore, faster query processing and data transport, as compared to a query where all characteristics are in rows
This approach separates the drill-down steps. In addition to accelerating query processing, it provides the user more manageable portions of data.
13
14
Performance Settings for Query Execution
This decides how many records are read during navigation
Examine the request status when reading the InfoProvider
In SAP NetWeaver BW 7.x, the BI Analytical engine can read deltas into the cache. Does not invalidate existing query cache.
Displays the level of statistics collected
Turn off/on parallel processing
When will the query program be regenerated based on database statistics?
15
Problem #9: Filters in Queries Used in Dashboards
• Using filters contributes to reducing the number of database reads and the size of the result set Thereby significantly improving query runtimes
Filters are especially valuable when associated with large dimensionswhere there are a large number of characteristics, such as customers and document numbers
Problem #10: The RSRT Transaction to Examine Slow Queries
16
P1 of 3
The RSRT transaction is one of the most beneficial transactions to examine the query performance and to conduct “diagnostics” on slow queries from the SAP NetWeaver BW system
Do You Need an Aggregate: Some Hints
17
This suggests that an Aggregate would have been beneficial
P2 of 3
18
Get Database Info
In this example, the Basis team should be involved to research why the Oracle settings are not per SAP’s recommendation
The RSRT and RSRV codes are key for debugging and analyzing slow queries
P3 of 3
HINT: Track front-end data transfers and OLAP performance by using RSTT in SAP NetWeaver BW 7.3 (RSRTRACE in SAP BW 3.5)
19
Problem #11: Debug Queries Using the RSRT Transaction
Using RSRT you can execute the query and see each breakpoint, thereby debugging the query and seeing where the execution is slow
Try running slow queries in debug mode with parallel processing deactivated to see if they run faster
• After SAP BusinessObjects Enterprise XI 3.1 FP 1.1, the impact of large numbers of key figures was somewhat reduced by retrieving metadata information only when the unit/currency metadata info is selected
• However, this is still best practice
Recommendation for Key Figures in OLAP Universes
• A large number of key figures (KFs) in the BEx query will incur a significant performance penalty when running queries, regardless of whether the key figures are included in the universe
• Only include key figures used for the dashboard in the BEx query (keep it small)
• This performance impact is due to time spent loading metadata for units, executed for all measures in the query
20
21
Problem #12: The Performance Killers — Restrictive Key Figures • When Restrictive Key Figures (RKF) are included in a query,
conditioning is done for each of them during query execution This is very time consuming, and a high number of RKFs can
seriously hurt query performance
• My Recommendation: Reduce RKFs in the query to as few as possible Also, define calculated key figures and RKFs on the
InfoProvider level instead of locally within the query. Why?
Benefit: Formulas within an InfoProvider are returned at runtime and held in cache
Drawback: Local formulas and selections are calculated with each navigation step
22
Dashboard Performance Killers: Calculated Key Figures• Calculated Key Figures (CKF) are computed during
runtime, and many CKFs can slow down the query performance
• How to fix this Many of the CKFs can be done during data loads and physically
stored in the InfoProvider This reduces the number of computations, and the query can use
simple table reads instead Do not use total rows when not required (this requires additional
processing on the OLAP side)
Recommendation for OLAP universes• RKF and CKF should be built as part of the
underlying BEx query to use the SAP NetWeaver BW back-end processing for better performance
• Queries with a larger set of such KFs should use the “Use Selection of Structure Members” option in the Query Monitor (RSRT) to leverage the OLAP engine
23
What We’ll Cover …
• Choosing the right connectivity and back end • Exploring query performance• Thinking about the dashboard design• Increasing query performance with infrastructure and in-memory
processing• Leveraging pre-caching capabilities and aggregates• Obtaining strategies for performance testing: Load and stress• Looking at EarlyWatch Reports and the performance checklist• Wrap-up
24
Problem #13: Dashboard Performance Hint — The Number of Rows in the Result Set
Limit the numberof rows in your result set to between 100-500
Returning query result sets with few records of a numeric type or with keys and indicators provides for the best dashboard performance
The Length of each record (# of columns) and the data type also impacts performance
In exceptional cases, when you have leveraged other performance-tuning methods, you may extend this to up to 1,000 rows
Divide and Get PerformanceDrill-down options
Link to Details Dashboard
• Split your dashboards into logical units and get new data when drilldowns are executed• This keeps the result set for each query small, and also decreases the load time for each
dashboard 25
26
Problem #14: Excel Performance Considerations — What to Avoid• The logic you build into your Excel spreadsheet is also compiled
into the Flash file when you export it• Since some “daisy-chain” functions are very time consuming, you
should be careful not to add too many conditions in the data Lookup functions and conditioning that should be avoided
include: Lookups
Mid strings (MID) Right and left strings (RIGHT/LEFT) Horizontal Lookups (HLOOKUP) Vertical Lookups (VLOOKUP)
Condition General conditioning (IF) Count if a condition is true (COUNTIF) Sum if a condition is true (SUMIF)
Complex logic and nested logic create large SWF files and take a long time to open. Try to keep as much of the calculation and logic in the query instead of the spreadsheet.
Hint: Reducing the text in the query will also speed up the query processing time
User Sorts themselves
Problem #15: The BI Analytical Engine and Sorting
• Sorting is done by the BI Analytical Engine Like all computer systems, sorting data in a
report with large result sets can be time consuming• Reduce the number of sorts in the “default view”
This will provide the users with data faster. They can then choose to sort the data themselves.
27
These are dashboard objects that you need to carefully consider before employing
Problem #16: Dashboard Objects That Can Cause Slow Performance
28
29
What We’ll Cover …
• Choosing the right connectivity and back end • Exploring query performance• Thinking about the dashboard design• Increasing query performance with infrastructure and in-memory
processing• Leveraging pre-caching capabilities and aggregates• Obtaining strategies for performance testing: Load and stress• Looking at EarlyWatch Reports and the performance checklist• Wrap-up
• It is hard to build a fast dashboard with many queries and panels without SAP NetWeaver BW Accelerator This provides in-memory processing of queries
that is 10-100 faster
Problem #17: It Is All About Performance, Performance, Performance
• What we simply do is place the data in-memory and retrieve itmuch faster There is also some limited OLAP functionality that can be built
into SAP NetWeaver BW Accelerator 7.3, but most data processing still occurs in the BI Analytical engine
You can also place non-SAP data in-memory using SAP BusinessObjects Data Services 30
The major improvement is to make query executions more predictable and faster overall
Seconds
Num
ber o
f Que
ries
Num
ber o
f Que
ries
Seconds
SAP NetWeaver BW Accelerator Performance Increases: Real Example
31
32
BI Analytical Engine’s Query Executing Priorities
Query ExecutionWithout SAP NetWeaver
BW Accelerator
Query Executionwith SAP NetWeaver
BW Accelerator
Information Broadcasting/Pre-Calculation
Query Cache
Aggregates
InfoProvider
Information Broadcasting/Pre-Calculation
Query Cache
SAP NetWeaver BW Accelerator
Aggregates can be replaced with SAP NetWeaver BW Accelerator while the memory cache is still useful
Source: SAP AG
Persistence Layer
Looking Inside SAP HANA: In-Memory Computing Engine (IMCE)
Disk Storage
Data
VolumesPage Mgmt.
BusinessObjects Data Services
Log
Volumes
Logger
Metadata
Manager
Authorization
Manager
Transaction
Manager
Relational Engine
-Row Store -Column Store
Load
Controller
SQL Script
Calculation
Engine
Replication Server
SQL Parser
MDX
Session Manager
You can also move data to SAP HANA and access the data in-memory; this creates a much faster response time for all your dashboards
33
SAP HANA: Sources and Target Interfaces
ERP
Database
HANA Appliance
In-Memory
Computing
Engine
Sybase
Replication
Server
SAP BW Third Party
Semantic Layer
SAP BusinessObjects 4.0Dashboards
Crystal
Web Intelligence
Analysis – Office
Crystal
Explorer
Sybase UnwiredOthers
Third-Party Applications
Custom Web Development
Third-Party Applications
Microsoft Excel (certified)
SQL (JDBC/ODBC)
DBSQL
BICS
SQL (JDBC/ODBC)
MDX (ODBO)SAP BusinessObjects Data Services
Real-time
A great benefit is the real-time loading of SAP HANA from ERP; this can provide real-time analytics to end users
34
35
What We’ll Cover …
• Choosing the right connectivity and back end • Exploring query performance• Thinking about the dashboard design• Increasing query performance with infrastructure and in-memory
processing• Leveraging pre-caching capabilities and aggregates• Obtaining strategies for performance testing: Load and stress• Looking at EarlyWatch Reports and the performance checklist• Wrap-up
36
Problem #18: Different Uses of the MDX and OLAP Cache
• The OLAP Cache is used by SAP NetWeaver BW as the core in-memory data set It retrieves the data from the server if the data set is available
• The cache is based on first in last out This means that the query result set that was accessed by one
user at 8:00 am may no longer be available in-memory when another user is accessing it at 1:00 pm Therefore, queries may appear to run slower sometimes
The MDX cache is used by MDX-based interfaces, including the OLAP universe
Use the BEx Broadcaster to Pre-Fill the Cache
37
Distribution Types
• You can increase query speed by broadcasting the query result of commonly used queries to the cache
• Users do not need to execute the query from the database Instead, the result is already in the system
memory (much faster)
The Memory Cache Size
• The OLAP Cache is, by default, 100MB for local and 200MB for global use This may be too low ...
• Look at available hardware and work with your Basis team to see if you can increase this
• If you decide to increase the cache, use the transaction code RSCUSTV14
The OLAP Cache is not used when a query contains a Virtual Key Figure or virtual characteristics or when the query is accessing
a transactional DSO or a virtual InfoProvider38
Monitor Application Servers and Adjust Cache Size
To monitor the usage of the cache on each of the application servers, use transaction code RSRCACHE, and also periodically review the analysis of load distribution using ST03N – Expert Mode
39
P.S.! The size of the OLAP Cache is physically limited by the amount of memory set in system parameter rsdb/esm/buffersize_kbThe settings are available in RSPFPAR and RZ11
The Four Options for OLAP Cache Persistence Settings
CACHE OLAP Persistence SettingsNote When What T-code
Default Flat file
Change the logical file BW_OLAP_CACHE when installing the system (not valid name) FILE
Optional Cluster table Medium and small result setsRSR_CACHE_DBS_IX RSR_CACHE_DB_IX
OptionalBinary Large Objects (blob) Best for large result sets
RSR_CACHE_DBS_BLRSR_CACHE_DB_BL
Available since SAP NetWeaver BW 7.0 SP 14
Blob/Cluster Enhanced
No central cache directory or lock concept (enqueue). The mode is not available by default.
Set RSR_CACHE_ACTIVATE_NEW RSADMIN VALUE=x
40
41
Problem #19: Correct Aggregates Are Easy to Build
We can create proposals from the query, last navigation by users, or by BW statistics
• Create aggregate proposals based on BW statistics. For example: Select the runtime of queries to
be analyzed Select the time period to be
analyzed Only those queries executed in
this time period will be reviewed to create the proposal
Create aggregate proposals based on queries that are performing poorly
Activate the Aggregate
The process of turning 'on' the aggregates is simple
1. Click on Jobs to see how the program is progressing
42
Fill the Aggregate with Summary Data
44
What We’ll Cover …
• Choosing the right connectivity and back end • Exploring query performance• Thinking about the dashboard design• Increasing query performance with infrastructure and in-memory
processing• Leveraging pre-caching capabilities and aggregates• Obtaining strategies for performance testing: Load and stress• Looking at EarlyWatch Reports and the performance checklist• Wrap-up
45
Problem #20: Performance Testing — Load and Stress
• Load testing is done to 20% of the named user base Turn off the cache (we assume all hits “new data”) Execute the dashboard URLs using a tool or a simple
JavaScript Monitor database, portal, and BI system load Log response time and have multiple browsers and PCs hitting
the data from multiple locations (network testing)• Stress testing is done to 40% of named user base
The test is done the same way as on the load testing, just with more “users”
The system may not be able to pass at this level, but the break-points are identified
All dashboard systems should be load testedto 20% of user base prior to Go-Live
46
Bonus Problem #1: Server Locations and Network Capacity
• Having a central global install of SAP BusinessObjects BI 4.x with many users can cause significant network load and performance issues
Consider the network topology, capacity, and the user locations before implementing global dashboards
47
What We’ll Cover …
• Choosing the right connectivity and back end • Exploring query performance• Thinking about the dashboard design• Increasing query performance with infrastructure and in-memory
processing• Leveraging pre-caching capabilities and aggregates• Obtaining strategies for performance testing: Load and stress• Looking at EarlyWatch Reports and the performance checklist• Wrap-up
48
Bonus Problem #2: EarlyWatch Reports in SAP Solution Manager• EarlyWatch reports provide a simple way to confirm how your
system is running and to catch problems A “goldmine” for system recommendations
• EarlyWatch Reports have been available since SAP Solution Manager version 3.2 SP8
• The more statistics cubes you have activated in SAP NetWeaver BW, the better usage information you will get Depending on your version of SAP NetWeaver BW, you can
activate 11-13 InfoCubes Also, make sure you capture statistics at the query level (set
it to “all”)
System issues can be hard to pin-down without access to EarlyWatch Reports. Monitoring reports allows you to tune the system before a user complains.
49
Information About a Pending “Disaster”
This system is about to crash
The system is growing by 400+ GB per month, the app server is 100% utilized, and the DB server is at 92%
This customer needed to improve the hardware to get the query performance to an acceptable level
50
Bonus Problem #3: The Dashboard Performance Checklist
1. The hardware servers — Check sizing2. The server locations and networks — Check loads3. Query review — Look at database and calculation time
and design4. Interface review — Make sure you are using the best for the data source5. Dashboard review — Look at Excel logic, container usage, number of
Flash objects, sorts, size of result set, and simplification opportunities6. In-memory review — Look at cache usage, hit rations, and SAP
NetWeaver BW Accelerator usage7. Review data sources — Examine if snapshots can be leveraged, and
look for possibilities to create aggregates8. Examine compatibilities between browsers, Flash, and Microsoft office
versions9. Review PC performance issues — Memory, disk, and processors
Performance is complex, look at more than one area (e.g., Web portal bottlenecks and LDAP servers)
51
What We’ll Cover …
• Choosing the right connectivity and back end • Exploring query performance• Thinking about the dashboard design• Increasing query performance with infrastructure and in-memory
processing• Leveraging pre-caching capabilities and aggregates• Obtaining strategies for performance testing: Load and stress• Looking at EarlyWatch Reports and the performance checklist• Wrap-up
52
Where to Find More Information
• Chris Dinkel, “Tuning SAP BusinessObjects Solutions for Optimal Performance: Tips from the Trenches” (The SAP BusinessObjects Seminar, 2011).
• Steve Bickerton, “SAP BusinessObjects Tuning” (SAP AG, April 2011). www.scribd.com/doc/89894543/BOCX-Speaker-Performance-
Tuning-Steve-Bickerton• SAP Service Marketplace for sizing guidelines
https://websmp104.sap-ag.de/sizing and follow the Sizing Guidelines menu path SBO_BI_4_0_Dashboard_designer.pdf
Need to be a customer to access this Requires login credentials to the SAP Service Marketplace
53
7 Key Points to Take Home
• Dashboards are all about performance, performance, and performance
• You have to spend time on the back end performance tuning• Avoid direct querying of high data volumes; create summaries
instead• Consider in-memory processing for all critical dashboards• Your interface to the data will impact the performance —
Avoid MDX• Size your hardware one size “too big” — It is hard to make a
second first impression• Use a gradual rollout of your dashboards, monitor the
performance, and conduct load and stress tests before any major go-lives
DisclaimerSAP, R/3, mySAP, mySAP.com, SAP NetWeaver®, Duet®, PartnerEdge, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are the trademarks of their respective companies. Wellesley Information Services is neither owned nor controlled by SAP.
55