Daily Operations and Administration of Your HANA environmentscsc-studentweb.lr.edu/swp/Berg/articles/BIexpert/BIEXpertMagazine... · Daily Operations and Administration of Your HANA

1

Daily Operations and Administration of Your HANA environments By Dr. Bjarne Berg, VP SAP Business Intelligence, Comerit and

Professor at SAP University Alliance at Lenoir Rhyne University BI Expert Magazine December 2015

There are many ways to install and operate your HANA environments on premise. However, a creating an daily, weekly and

monthly Standard Operating Procedure (SOP) is a good way to make sure that the system stays well-tuned, and that

potential issues are avoided. This is also known as the daily operations handbook. In this article we look at what the

different landscape options are and how you can start creating your own SOP for your data center.

The first decision you have to make when setting up your data center for your HANA environments is to decide if you are

going to place it on-premise, as part of an outsourcing agreement, or on the cloud. The on premise approach is currently

most common and it basically means that you will have to integrate the hardware into you current data center and possibly

an off-site data center if you are implementing a high-availability (HA) solution.

A major consideration for the on-premise approach would be to make sure that your hardware fits into your existing

chassis, racks, power outlets, cooling plan and the outlay of your data center. For example, many are not aware that some

of the larger HANA systems like for example Lenovo’s x3850 x6 requires a 4U height in a data center, but if you are using

the Lenovo’s x3950 x6, you will need to double that size requirement (since that is basically two stacked 3850s). While

other vendors such as Cisco’s C880 M4 Server requires 10U height. So, it is very important to decide what hardware

deployment options you are going with. As of September 2015, the most common forms of certified hardware can be

summarized as:

Figure 1: Common HANA Hardware Platforms for On-Premise Deployment

2

Table 1: Certified HANA Hardware options as of September 2015.

Customer HANA Admin and Support Responsibilities

As you start with your plan to write an SOP, it is important that you are aware of the normal support, install, monitoring

roles and the responsibilities of SAP, your hardware vendor and your own team. The normal support responsibilities can be

summarized as in Table 2 below.

Scale-

out

Scale-

up

Business

Suite

bullion S2 512-1536 GB 2 x 8890v3 x 2.5 Ghz x x

bullion S4 1024-3072 GB 4 x 8890v3 x 2.5 Ghz x x

bullion S8 2048-6144 GB 8 x 8890v3 x 2.5 Ghz x x

UCS B260 M4 512-1536 GB 2 x 4890v2 x x x x

UCS C460 M4 128-3072 GB 4 x 8890v3 x 2.5 Ghz x x x

UCS C880 M4 2048-6144GB 8 x 8890v2 x 2.8 Ghz x x x

Dell PowerEdge R930 128-3072 GB 4 x 8890v3 x 2.5 Ghz x x

PQ 2400 E/S/L 128-1024 GB 4 x 8890v3 x 2.5 Ghz x x

PQ 2800B2/E2 128-6144 GB 8 x 8890v3 x 2.5 Ghz x x

RX4770M2 128-3072 GB 4 x 8880v3 x 2.3 Ghz x x

CS-500 128-3072 GB 4 x 8880v3 x 2.3 Ghz x x x

CS-900 1024-12288 GB 8 x 2890v2 x 2.8 GHz x x x

Hitachi CB520X B2 256-6144 GB 8 x 8880v3 x 2.3 Ghz x x x

FusionCube E9000 512-1024 GB 4 x 4890v2 x 2.8 GHz x

RH5885H V3 128-3072 GB 4 x 8880v3 x 2.3 Ghz x x x

RH8100 V3 128-6144 GB 8 x 8880v2 x 2.5 Ghz x x x

Flex x880 X6 128-6144 GB 8 x 8890v2 x 2.8 Ghz x x x

x3850 X6 128-3072 GB 4 x 8880v3 x 2.3 Ghz x x x

x3950 X6 512-6144 GB 8 x 8880v3 x 2.3 Ghz x x x

NEC Exp. 5800/A2040b 128-2048 GB 4 x 4890v2 x 2.8 GHz x x

Silicon Graphics UV 300H 256-6144 GB 8 x 8890v2 x 2.8 Ghz x x

Unisys Forward! 4150-B 128-3072 GB 4 x 4880v2 x 2.5 GHz x x

VCE UCS B460 M4 1024 GB 4 x 4890v2 x 2.8 GHz x

Target usage

HP

Huawei

Lenovo

Vendor SystemMemory

(RAM)

Bull SAS

Cisco

Fujitsu

CPUIntel Ivy Bridge

EX E7 15 Cores

(2014)

Max Number

and Type

CPU

Speed

Intel Haswell

EX E7 18 Cores

(2015)

3

Table 2: Summary of Key Responsibilities

It is important to note that the responsibilities as outlined in Table 3, is based on an on-premise installation of SAP HANA

and that no other support agreements are made with the hardware vendor, a cloud vendor, or an outsourcing partners.

Depending on how you write your support agreement with these vendors, some or all of the customer responsibilities may

be assumed by these partners. The trick to make sure on what you are responsible for is to specify these activities in a

Service Level Agreement (SLA) if you are using other vendors to support your systems and landscapes.

There are also different cloud options that some companies might consider. For each or these options the responsibilities of

the customer are significantly different. First, you can have your HANA system and applications delivered as a “Software as

a Service” (SaaS). Under this offering you can get software applications such as SAP Business Suite, SAP Business Warehouse

(BW), and SAP Rapid Deployment solutions (RDS) as SaaS HANA cloud solutions from several vendors who then take over all

customer responsibilities for daily monitoring, support and maintenance.

Another option is the “Platform as a Service” (PaaS). This is normally provided as a solution where the database, operating

system, connectivity and hardware is supported by a cloud vendor, but where daily operations and monitoring of the

application is the customer’s responsibilities. Finally, the lowest level of cloud offerings is known as “Infrastructure as a

Service: (IaaS). As the name implies, you are normally responsible for all tasks as illustrated in Table 3, except the hardware

maintenance which is then hosted in the cloud.

However, in this article we are going to assume that the support is for an on-premise implementation and that the

customer is assuming the normal support, maintenance and monitoring roles and that a cloud solution is not in place.

System vs. Landscape Administration

There are several different tools and procedures that should be developed that are different based on a system or

landscape administration perspective. For example, for system administration you should leverage SAP’s HANA

Administration guide that can be downloaded on help.sap.com. This guide is maintained and updated by SAP on a release

Area TaskHardware

vendorCustomer SAP

Hardware installation and health check x

Linux OS installation x

HANA platform installation x

Data source connectivity x

Adding DB instances (MCOS) x

SMD agent installation x

HANA DB admin x

Third party software installations x

HANA system monitoring x

HANA DB monitoring x

Backup and recovery x

"Bare metal" recovery x

Firmware patching (x)* x

Linux OS upgrades and patching (x)* x

Peripheral components patching x

HANA platform components updates & patching x

Support Issue resolution process (x)* x x

* depending on support contract

Initial Setup

Operations

Maintenance

4

basis. It shows you how to use the HANA cockpit (a Fiori LaunchPad application) and HANA studio for the main system

administration, the core functions of high-availability and disaster recovery, scalability (up and out), security administration,

as well as how to manage and monitor applications for data provisioning and custom applications built in the extended

services (XS) framework.

We can also monitor the system through the DBA Cockpit in Solution Manager and leverage the Trouble Shooting and

Performance Tuning guide from SAP when issues arise. However, from a landscape administration perspective we leverage

the Technical Operational manual from SAP and the DB control center, as well as any respective application support for the

systems you might be running. So, when you start developing your SOP/Daily Operating handbook, you should start by

familiarize yourself with these very important documents and tools and think about system admin as different from

landscape administration.

Table 3: Key SAP resources for HANA System and Landscape Administration

HANA System Monitoring tools and Education

You can also choose one or more ways to perform you system monitoring. For example, you can monitor system databases

and also tenant data bases (in MCOD/MCOS) by directly connecting to a database using the HANA cockpit, the DBA Cockpit

in Solution Manager, or through regular SAP HANA studio.

In HANA Studio in the administration perspective, you get access to most database and system information. There are

several tabs that displays landscape, alerts (automatic scheduled monitoring jobs), performance statistics, disk volume

information, configuration settings, overall system information, diagnostic files and configuration of traces and trace files.

Area Tool Purpose Web Resource

HANA Administration guide

- How to use the HANA cockpit and HANA studio for system admin.

- Core functions of high-availability, disaster recovery & scalability

- Security administration

- How to manage and monitor applications for data provisioning and

custom applications built in the extended services (XS) framework.

tinyurl.com/AdminHana

DBA Cockpit for HANATool to manage system landscape connections and central

management of DB configurationstinyurl.com/DBACockpit

HANA Troubleshooting and

Performance Analysis Guide

How to trouble shoot and fix DB performance issues and guidance

on general optimization.tinyurl.com/TroubleGuide

Multitenant DB GuideHow to monitor, setup and manage systems that have HANA

multitenant DBstinyurl.com/HanaDBs

Technical Operations Manual How to operate and administrate a HANA landscape. tinyurl.com/TechOperations

SAP DB Control Center (DCC) Guide on how to use DCC to monitor HANA and other databases tinyurl.com/databaseCC

Landscape

Admin

System

Admin

5

Figure 2: The SAP HANA Studio Administration Console Perspective

Also, since SPS9 of HANA in late 2014, the enhanced HANA Cockpit is a now very interesting way to get access to a simple

web based monitoring application that shows you key statuses of your HANA systems and databases. As mentioned before,

the HANA Cockpit is basically a Fiori LaunchPad site that you can also customize to show only the items you are interested

in for daily operation monitoring.

6

Figure 3: Administration with HANA Cockpit in Fiori

The customization of this application is a simple click-and drag of the tiles (much like on you cell phone). You can also

choose the refresh rate of the information in HANA Cockpit and the application can run on a web browser and is therefore

mobile and simple to deploy. The HANA Cockpit also have a ‘Manage Databases’ app that allows you to monitor single and

multi-tenant databases in HANA.

As you click on each of these tiles, a vast array of detail information is provided for your in-depth analysis and system

monitoring. However, it is important to note that while the HANA Cockpit supports core administration of tenant databases

(i.e. MCOS), HANA Studio and some command-line tools may still be required for key tasks for tenant databases. Frankly,

the only minor drawback with the HANA Cockpit is that it may require additional licenses depending on what you bought

with the initial license package.

At a higher level the SAP Database Control Center (DCC) is also a Fiori application that allows you to monitor both HANA

and other type databases from a central application. As you get more familiar with these tools, you probably will find it

useful to start with one or two of these as choose the others as alternatives when you get stuck on a certain task. Most

system administrators include HANA studio and either the DBA or the HANA cockpit for daily monitoring.

To get started to learn about these tools, first download and study the guides outlined in Table 1 and then take the 5-day

SAP Course called HA-200 “SAP HANA - Installation & Operations”. This course is strongly recommended for experienced

support staff as well as for beginners and should be taken before you start writing your own SOP.

Solution Manager and LVM Tools

Many of the tools used for system monitoring is also used for database monitoring. First, you can conduct many of the

individual database admin functions through HANA studio and the HANA cockpit from a Web browser. From here you can

make changes to the database system settings and also add users and privileges and most standard database admin tasks.

Also, like all SAP software, you can use Solution Manager (SolMan) for core monitoring and admin of multiple systems in

7

your landscape and as the backbone for CTS+ integration of transports between the systems in your landscape. Solution

Manager can also be used to generate EarlyWatch reports on a periodical basis that shows growth, usage trends and

technical support information. You will also find the DBA Cockpit in Solution Manager. This tool allows you to monitor the

HANA database and exposes almost all of the technical information you would otherwise find in the administrator console

perspective in HANA Studio.

Figure 4: HANA Admin and monitoring with the DBA Cockpit in Solution Manager

Solution Manager and the DBA Cockpit also support trace analysis, workload analysis and exception analysis of HANA

databases. Most customers therefore find this tool invaluable when monitoring and managing SAP landscapes with both

HANA and other types of databases.

In addition to these tools the Landscape and Virtualization Manager (LVM) from SAP is also supported for HANA. This allows

you to conduct core operation of complex landscapes that may have HANA and non-HANA based servers. There is a

standard edition of LVM that can be downloaded from SAP for free, and there is an Enterprise Edition that has more

features that require a license before you can use it.

8

Figure 5: SAP Landscape and Virtualization Manager (LVM)

When any of these administration and management tools is deployed, it is important that your support staff, that is

monitoring, maintaining and operating a HANA landscape, has a good understanding of the capabilities of each of these.

This can be obtained through regular SAP training classes. If you are new to these tools, you might start with one or two,

and integrate them into your support landscape and add more as your experience level increase.

Daily Operations HANA Checklist

Once you have decided on you monitoring tool, downloaded and studied the available support documents in Table 1 and

completed the 5-day HA-200 HANA - Installation & Operations class from SAP, you are ready to start writing your Standard

Operating Procedure. The SOP should consist of daily operations, weekly jobs and periodic upgrades and patches as

supplied by SAP. In this section we take a look at the most common daily operations tasks that you will be doing.

While many prefer to have active or ‘passive’ monitoring of systems, best-practices are to have a combination of these.

Passive monitoring usually means activating and scheduling some of the alerts available in HANA Studio. You can place

thresholds on the alerts (i.e. when memory consumed exceeds a certain number of GB), and you can schedule how often

the checks are performed on the database. These alerts when triggered show up in the HANA cockpit, DBA Cockpit and in

HANA studio in both detail and overview pages. Today there are 74 standard alerts that come with the HANA system.

9

Figure 6: SAP HANA Alerts in HANA studio

You can also setup email alerts if you have the system privilege CATALOG READ as well as SELECT privilege on the

_SYS_STATISTICS schema, and also the system privilege INIFILE ADMIN. The first of these privileges is included in the

standard SAP HANA role called MONITORING. This role can be assigned to non-system admin users and allow other

technical resources access to see what is happening inside the HANA system without the ability to change anything.

There is also a list of historically executed alerts in HANA studio, but be aware that this list is restricted to the last 1,000

occurrences from the last 30 days. Also, when an alert is triggered, a priority is assigned by the system. In general, there

are 4 different priorities with different timing when action is recommended:

Table 4: Alert Priorities in SAP HANA

There are also 10 different categories of alerts relating to Availability, Backup, Configuration, CPU, Diagnosis Files, Disk,

Memory, Security, Sessions and System. So, when deciding when to schedule these alerts and when to monitor them is a

critical task of the HANA administrator. Activating and monitoring the recommended daily, and intra-day, alerts through

any of the tools outlined previously will allow you to early detect any performance issues.

To get started, take a look at table 5 as the first step of your own tailored daily HANA admin SOP. In this table you will find

all of the available HANA automated alerts, alert IDs (so that you can find them in HANA Studio), the suggested frequency

when these alerts should be activated/monitored, descriptions and SAP’s official recommendation on how to resolve any

issues.

Alert

PriorityAction Reccomended

HighAn Immediate action is required to reduce

risk of corrupt or lost data and outages

MediumAn action is required in the next hours/days

to reduce downtime risk

LowAn action required in the next days/weeks

to reduce downtime risks

InformationAn action can be taken to improve stability

and performance of the system

10

Table 5: Admin Monitoring Frequency, Alert IDs and SAP recommended actions

Check Type

ID Time Description SAP Recommended Admin Action

Availa-bility

0 Intra-day Identifies internal statistics server problem. Resolve the problem. For more information, see the trace files. You may need to activate tracing first.

3 Intra-day Identifies inactive services. Investigate why the service is inactive, for example, by checking the service's trace files.

4 Intra-day Restarted Services- services that have restarted since the last time of the check.

Investigate why the service had to restart or be restarted, for example, by checking service's trace files.

21 Daily Identifies internal DB events. Resolve the event and then mark it as resolved by executing the SQL statement ALTER SYSTEM SET EVENT HANDLED '<host>:<port>' <id>.

22 Intra-day Notification of all alerts- if any alerts since the last check is triggered

These alerts can trigger email blasts to specified recipients. Investigate the alerts.

23 Intra-day Notification of medium and high priority alerts- since the last check is triggered

24 Intra-day Notification of high priority alerts- since the last check is triggered

31 Daily

License expiry-If the disks to which data and log files are written are full. A disk-full event causes DB to stop

Obtain a valid license and install it. For the expiration date, see the monitoring view M_LICENSE.

41 Daily

In-memory DataStore activation- If a problem with the activation of an in-memory DataStore object exists

For more information, see the table _SYS_STATISTICS.GLOBAL_DEC_EXTRACTOR_STATUS and SAP Note 1665553.

70 Periodic Consistency of internal system components after system upgrade

Contact SAP support.

78 Daily

Connection between systems in system replication setup- closed connections between primary/ secondary system.

If connections are closed, the primary system is no longer being replicated. Investigate why connections are closed (i.e., network problem) and resolve the issue.

80 As

needed

Availability of asynchronous table replication- Monitors error messages related to asynch table replication.

Determine which tables encountered the table replication error using system view M_ASYNCHRONOUS_TABLE_REPLICAS, and check the corresponding indexserver alert traces.

Back-up

28 Periodic

Most recent savepoint operation- How long ago the last savepoint was defined, that is, how long ago a complete, consistent image of the DB was persisted to disk.

Investigate why there was a delay defining the last savepoint and consider triggering the operation manually by executing the SQL statement ALTER SYSTEM SAVEPOINT.

32 Periodic

Log mode LEGACY- If the DB is running in log mode "legacy". Log mode "legacy" does not support point-in-recovery and is not recommended for productive systems.

If you need point-in-time recovery, reconfigure the log mode of your system to "normal". In the "persistence" section of the global.ini configuration file, set the parameter "log_mode" to "normal" for the System layer. When you change the log mode, you must restart the DB system to activate the changes. It is also recommended that you perform a full data backup.

33 Periodic

Log mode OVERWRITE- If the DB is running in log mode "overwrite". Log mode "overwrite" does not support point-in-recovery (only recovery to data backup) and is not recommended for prod systems.

35 Daily Existence of data backup Perform a data backup as soon as possible.

36 Daily Status of most recent data backup Investigate why failed, resolve the problem, and perform a new data backup as soon as possible.

37 Daily Age of most recent successful data backup Perform a data backup as soon as possible.

11

38 Daily

Status of most recent log backups- If the most recent log backups for services and volumes were successful.

Investigate why the log backup failed and resolve the problem.

54 Periodic Savepoint duration- Identifies long-running savepoint operations.

Check disk I/O performance.

65 As

needed

Runtime of the log backups currently running- If the most recent log backup terminates in the given time.

Investigate why the log backup runs for too long, and resolve the issue.

66 As

needed

Storage snapshot is prepared- if the period, during the DB is prepared for a storage snapshot, exceeds threshold.

Investigate why the storage snapshot was not confirmed or abandoned, and resolve the issue.

69 Periodic Enablement of automatic log backup- if automatic log backup is enabled.

Enable automatic log backup. For more details please see SAP HANA Administration Guide.

72 Daily

Number of log segments- segments in the log volume of each service Check for number of log segments. Make sure that log backups are being auto created and that there is enough space

Check whether the system has been frequently and unusually restarting services. If it has, then resolve the root cause of this issue and create log backups as soon as possible.

Configuration

3 As

needed

Discrepancy between host server times- discrepancies in a scale-out system.

Check operating system time settings.

10 Periodic

Delta merge (mergdog) configuration- If the 'active' parameter in the 'mergedog' section of system configuration file(s) is 'yes'.

mergedog is the system process that periodically checks column tables to determine if a delta merge operation needs to be executed. Change in SYSTEM layer the parameter active in section(s) mergedog to yes

16 Periodic

Lock wait timeout configuration- if 'lock_waittimeout' parameter in 'transaction' section of indexserver.ini file is between 100,000 and 7,200,000.

In the 'transaction' section of the indexserver.ini file, set the 'lock_wait_timeout' parameter to a value between 100,000 and 7,200,000 for the System layer.

26 Periodic Unassigned volumes- Identifies volumes that are not assigned a service.

Investigate why the volume is not assigned a service. I.e.., assigned service is not active, the removal of a host failed, or the service removal was performed incorrectly.

34 Daily If all volumes are available. Investigate why the volume is not available.

79 Periodic

Configuration consistency of systems in system replication setup- Identifies configuration parameters that do not have the same value on the primary system and a secondary system.

The identified configuration parameter(s) should have the same value in both systems, adjust the configuration. If different values are acceptable, add the parameter(s) as an exception in global.ini/[inifile_checker].

CPU 5 Intra-day

Host CPU Usage- Determines the % CPU idle time on the host and therefore if CPU resources are running low.

Investigate CPU usage

Diag-nosis

Files

46 As

needed

RTEdump files- Identifies new runtime dump files (*rtedump*) have been generated in the trace directory.

These files These contain information about, for example, build, loaded modules, running threads, CPU, etc..Check contents of the dump files.

50 Periodic Number of diagnosis files- written by the system (excluding zip-files).

A large number of files can indicate a problem with the DB (i.e., problem with trace file rotation or a high number of crashes). Investigate the diagnosis files.

51 Daily Size of diagnosis files- very large file sizes can indicate a problem with DB.

Check the diagnosis files in the SAP HANA studio for details.

52 Daily Crashdump files- new files that have been generated in the trace directory

Check the contents of the dump files.

53 Daily Pagedump files- new files that have been generated in the trace directory

12

56 Periodic Python trace activity- If trace is active and for how long. Trace affects performance.

If no longer required, deactivate the python trace in the relevant configuration file.

Disk

2 Intra-day

Disk Usage- Determines what % of each disk containing data, log, and trace files is used. This includes space used by non-SAP HANA files.

Investigate disk usage of processes. Increase disk space, for example by shrinking volumes, deleting diagnosis files, or adding additional storage.

30 Intra-day

Check internal disk full event- If the disks to which data and log files are written are full. A disk-full event causes your DB to stop and must be resolved.

Resolve the disk-full event: In the Admin Editor on the Overview tab, choose the \"Disk Full Events\" link and mark the event as handled. Alternatively, execute the SQL statements ALTER SYSTEM SET EVENT ACKNOWLEDGED '<host>:<port>' <id> and ALTER SYSTEM SET

EVENT HANDLED '<host>:<port>'<id>.

60 Periodic Sync/Async read ratio- Identifies a bad trigger asynchronous read ratio.

This means that asynchronous reads are blocking and behave almost like synchronous reads. This might have negative impact on SAP HANA I/O performance in certain scenarios. Note 1930979. 61 Periodic

Sync/Async write ratio- Identifies a bad trigger asynchronous write ratio.

77 Intra-day

DB disk usage- The total used disk space of the DB. All data, logs, traces and backups are considered.

Investigate the disk usage of the DB. See system view M_DISK_USAGE for more details.

Mem- ory

1 Intra-day Host physical memory usage- The % of total physical memory available on the host

All processes consuming memory are considered, including non-SAP HANA processes. Investigate memory usage of processes.

3 Periodic Row store fragmentation Implement SAP Note 1813245.

12 Intra-day

Memory usage of name server- Determines what % of allocated shared memory is being used by the name server on a host.

Increase the shared memory size of the name server. In the 'topology' section of the nameserver.ini file, increase the value of the 'size' parameter.

17 Periodic

Record count of non-partitioned column-store tables- Current table size is not critical.

Partitioning need only be considered if tables are expected to grow rapidly. A non-partitioned table cannot contain more than 2,000,000,000 (2 billion) rows). Consider partitioning the table only if you expect it to grow rapidly.

20 Periodic Table growth rate of non-partitioned column-store table

27 Periodic Record count of column-store table partitions

29 Periodic Size of delta storage of column-store tables Investigate the delta merge history in the monitoring view M_DELTA_MERGE_STATISTICS. Consider merging the table delta manually.

40 Daily

Total memory usage of column-store tables- The % of the effective alloc limit being consumed by individual column-store tables as a whole

This is the cumulative size of all of a table's columns and internal structures. Consider partitioning or repartitioning the table.

43 Daily Memory usage of services- % of effective alloc limit a service is using.

Check for services that consume a lot of memory.

44 Periodic Licensed memory usage- % used. Increase licensed amount of main memory. See the peak memory allocation since installation in the system view M_LICENSE, column PRODUCT_USAGE

45 Periodic

Memory usage of main storage of column-store tables- % of effective alloc limit consumed by column-store tables.

Consider partitioning or repartitioning the table.

55 Periodic Columnstore unloads- # of columns that have been unloaded from memory.

Can indicate performance issues. Check sizing with respect to data distribution.

58 As

needed

Plan cache size- if the plan cache is too small.

Increase the size of the plan cache. In the 'sql' section of the indexserver.ini file, increase the value of the 'plan_cache_size' parameter.

13

67 Periodic Table growth of rowstore tables Reduce the size by removing unused data

68 Periodic Total memory usage of row store used by a service

Investigate memory usage by row store tables and consider cleanup of unused data

73 Periodic Overflow ratio of rowstore version space. Identify the connection or transaction that is blocking version garbage collection. You can do this in the SAP HANA studio by executing the "MVCC Blocker Connection" and "MVCC Blocker Transaction" statements available on the System Information tab of the Administration editor. If possible, kill the blocking connection or transaction.

74 Periodic Overflow ratio of metadata version space.

75 Periodic Rowstore version space skew- if rowstore version chain is too long.

81 Periodic Cached view size- how much memory is occupied by cached view

Increase size of the cached view. In the "view_cache" section of the indexserver.ini file, increase the value of the "total_size" parameter.

Secur-ity

57 Daily Secure store file system (SSFS) consistency regarding the DB

Check and make sure that the secure storage file system (SSFS) is accessible and consistent regarding the DB.

62 Daily

User passwords- Identifies DB users whose password is due to expire with the PW policy. If it expires, the user will be locked. This may impact application availability.

Change password of the DB user. It is recommended that you disable the password lifetime check of technical users so that their password never expires (ALTER USER

<username> DISABLE PASSWORD LIFETIME).

63 Daily

Granting of SAP_INTERNAL_HANA_SUPPORT role- if the internal support role is currently granted to any DB users.

Check if the corresponding users still need the role. If not, revoke the role from them.

64 Periodic

Total memory usage of table-based audit log- % of the effective allocation limit is being consumed by the DB table used for table-based audit logging.

Consider exporting the content of the table and then truncating the table.

Sess-ions

25 Daily Open connections- % of the max number of permitted SQL connections open.

The max number of permitted connections is configured in the "session" section of the indexserver.ini file.Investigate why max number is being approached.

Sess-ion &

Transa-ctions

39 Daily Long-running SQL statements Investigate the statement. For more info, see table _SYS_STATISTICS.HOST_LONG_RUNNING_STATEMENTS.

42 As

needed Long-idling cursors Close cursor, uncommitted transaction, or the

serializable transaction in the application, kill connection, or by executing the SQL statement ALTER

SYSTEM DISCONNECT SESSION <LOGICAL_CONNECTION_ID>. For more information, see the tables HOST_LONG_IDLE_CURSOR, HOST_LONG_SERIALIZABLE_TRANSACTION and HOST_UNCOMMITTED_WRITE_TRANSACTION (_SYS_STATISTICS).

47 Periodic Long-running serializable transactions

48 Periodic Long-running uncommitted write transactions

49 Periodic Long-running blocking situations Investigate the blocking and blocked transactions and if appropriate cancel one of them. 59 Daily Percentage of blocked transactions

System 83 Daily Table consistency- the number of table consistency errors and affected tables

Contact SAP support

Periodic and Active monitoring

In addition to these passive alerts, you can also add active monitoring by the system administrator to starts to predict when

performance issues might arise. This include tracking weekly and/or monthly increases in data files, CPU utilization, planned

14

projects, memory consumption and user activity. This will allow you to plan and budget resources for system upgrades,

data archiving, NLS implementations and hardware changes.

In addition, it is imperative that you also monitor new security patches and software fixes available from SAP and determine

if this is something that you should consider for rapid implementation, or can bundle them into periodic upgrades, service

packs and patches on a monthly or quarterly basis. The list of tasks in Table 5 should get you started with the automatic

checks, but you can also build your own additional monitoring process by accessing the system information in SAP HANA.

This is available in the administrator perspective in HANA studio under the ‘System Information’ tab page and is also mostly

exposed in the HANA cockpit and in the DBA Cockpit.

Figure 5: The System Information tab in HANA Studio

15

Periodic SOP Tasks for Keeping a HANA System Smaller

In addition to the daily and periodic standard monitoring tasks, you might want to do an active monitoring of a HANA

system during a go-live of a new project, as well as during a ‘hyper care’ period shortly after new functionality has been

added to the system. In this section we look at some useful tips that helps you do active monitoring using available

transactions in SAP. Since most HANA systems are currently using SAP BW as an application (this will most likely change

over time), we are focusing on keeping BW 7.3 and 7.4 on HANA systems as small as possible. However, some of the tasks

in this section also apply to a BusinesSuite on HANA system as well,

First, if you will want to quickly get access to see the largest HANA tables (and monitor their growth). This can be done by

using the transaction code DB02. Here you will find the largest tables in memory and also their respective record counts.

Figure 6: View Large HANA Tables using DB02

In a SAP BW system there are also a set of tables that are likely to grow faster than others. These include the application log

tables BALHDR, BALHDRP, BALM, BALMP, BALDAT, BALC, and BAL_INDX. The tables for linking IDocs (IDOCREL, SRRELROLES)

and the Short dump table SNAP.

Other request administration data in BW that you also want to monitor include data in the RSBMLOGPAR,

RSBMLOGPAR_DTP, RSBMNODES, RSBMONMESS, RSBMONMESS_DTP, RSBMREQ_DTP, RSCRTDONE, RSDELDONE,

RSHIEDONE, RSLDTDONE, RSMONFACT, RSMONICTAB, RSMONMESS, RSMONRQTAB, RSREQDONE, RSRULEDONE,

RSSELDONE, RSTCPDONE, RSUICDONE tables, as well as the Dictionary logs found in DDPRS. In this section we will take a

look on how to best manage these tables to keep the HANA system as small as possible.

In addition to these tables you should also monitor the size of the BW workbook table RSRWBSTORE and the BW statistics

data found in RSDDSTATAGGR, RSDDSTATAGGRDEF, RSDDSTATCOND, RSDDSTATDELE, RSDDSTATDM, RSDDSTATEVDATA,

RSDDSTATHEADER, RSDDSTATINFO, RSDDSTATLOGGING, and RSDDSTATDTP. All of these tables tend to grow rapidly in

16

active systems, but keeping them small is rather simple. For example, to clean up the stats entries in these tables, you can

run the program RSDDSTAT_DATA_DELETE in SE38 and then schedule the job to run periodically in SM36. In general, it may

be useful to keep 12 months of stats data, so you don’t want to remove it all.

Figure 7: Deleting Statistics Entries over 365 days old using the RSDDSTAT_DATA_DELETE program

You should also monitoring the size of the DTP error log in RSBERRORLOG, the Process Chain logs (RSPCINSTANCE), and the

BW batch runtime data (RSBATCHDATA). These logs contains numerous warnings and errors in transformation recordings.

You can delete these on a periodic basis using RSBM_ERRORLOG_DELETE and schedule it to run to remove entries over

60/90 days old.

Furthermore, you can also keep your system small by periodically remove data in the RSPCLOGCHAIN, RSPCPROCESSLOG,

and the RSPCINSTANCET table. This is done by using the RSPC_LOG_DELETE report in SE38.

Figure 8: RSPC_LOG_DELETE of Process Chain Data in BW.

17

In addition to the tips and tricks mentioned above, to keep the application log tables small, you should use SM36 to

schedule a job to run periodically. In the screen, click on the ‘Step’ button to define which program will be run by the job

(pick SBAL_DELETE). After that, you can also choose a variant to decide how much data you want to keep. For example, in

Figure 9 we delete case logs older than 60 days.

Figure 9: Cleaning BW Application Log tables in HANA

18

After this setup, you can then schedule the job to run on periodic basis. This job will proactively help keep the memory

utilization of the HANA system smaller than it would be if all these application logs are not periodically cleaned.

Figure 10: Scheduling Archiving Job to run Periodically

You should also periodically archive IDocs to keep the HANA system smaller. IDOCs are used for communication between

BW and source system. When BW executes an InfoPackage for data extraction, a request IDocs, RSREQUEST, is sent to the

source system’s Application Link Enabler (ALE) inbox. The source system acknowledges the receipt of this IDoc by sending

an info IDoc (RSINFO) back to the BW system.

In addition the source system sends an IDoc with all the requested data using the message type (RSSEND). Therefore, these

tables can grow fast over time. You should therefore consider archiving IDocs older than 3-6 month. You can do this using

the transaction “SARA” and then click on “write” as illustrated in Figure 11.

Figure 11: Keeping HANA Small by Archiving IDocs using the SARA Transaction

19

Figure 12: Selecting tables for IDocs archiving Using SARA

Figure 13: Maintaining “Created On” Filter Dynamically to Archive all Entries Older than 30, 60 or 90 Days

Then you simply give the job a description and schedule the job to run periodically. However, it is important to note that

your Basis team will need to periodically backup and delete the archiving file in directory folder. This is typically done on an

annual basis. The SARA transaction is also available in the BusinesSuite on HANA system as well.

In addition to the jobs and programs outlined above, you should also consider cleaning up obsolete IDocs links from on a

periodic basis. Links are written in the ALE and IDoc environment, resulting in entries in the IDOCREL and SRRELROLES

tables. The links are required for IDoc document trace and ALE audit monitoring. You can delete links in IDC8 and IDCA

regularly (they are not required after the IDocs are posted). To do delete the links, you can go to SE38 and the report

RSRLDREL. Under the "Selection mode", pick “Select using relationship type" and create a variant for the link types IDC8 and

IDCA. Then, under the "Deletion Criterion", select “Without existence check" and finally, as we did in Figure 11 (the regular

20

IDoc removal), you can now create a dynamic end date variable, and use SM36 to schedule the job to run on a periodic

basis. This will help you maintain a small HANA system and keep the BW system much cleaner.

Furthermore, you can make the Request Administration data in HANA smaller. While you should never delete entries in

theses tables, you can archive them and re-load old entries if necessary. Before you start this, you should make sure that

the reports RSSTATMAN_CHECK_CONVERT_DTA and RSSTATMAN_CHECK_CONVERT_PSA have been executed in SE38 at

least once for all objects. You start by using SARA and select the archiving object called BWREQARCH and then schedule

this to run to archive entries that are more than 90 days old (see figure 14).

Figure 14: Periodic Maintenance by Archiving Request Administration Data in BW

In addition to these archiving jobs, you can also remove data in the Dictionary Logs. Basically, the DDPRS dictionary log

table contains all activities that do any change on the data dictionary objects in SAP BW. This can grow quite large over

time, and you might want to remove some of these log entries on an annual basis. To clean up these log entries go to SE38

and run the RADPROTA report. Just make sure you execute the job as a background job (see Figure 15).

Figure 15: Removing Dictionary Logs in SAP BW to reduce HANA Size

21

You also want to make sure that non reportable tables, that have no daily loads going into them, have their memory

cleansed. You do that with transaction RSHDBMON using the load/unload the data options, or flag it for ‘Early Unload’. The

last option allows HANA to decide when the data should be unloaded (i.e. when the data is not accessed, or data is not

loaded). This will allow you to keep your HANA memory usage smaller, while still have access to the data when needed.

Figure 16: Unload Non-Active Data from HANA Memory with RSHDBMON

Also, occasionally you need to clean up short dump table “SNAP” in BW. To do this simply go to transaction ST22 and select

“reorg” (see Figure 17).

Figure 17: Periodic Cleaning up of the Short-Dump table in SAP BW

22

And delete short dumps older than 90 days

Finally, from a HANA monitoring standpoint you may want to ensure that the delta merge is performed regularly. You can

monitor this in the HANA Admin perspective of Eclipse in HANA Studio.

Figure 17: Monitoring the Delta Merge Processing in HANA using HANA Studio

Summary

Administrating HANA on a daily basis is a very interesting challenge. There are many technical aspects from hardware

health, system connectivity, database performance, security, file management, backup, table management, memory

consumption, disk utilization and much more. It is therefore very important that you plan a very structured support

approach to SAP HANA and that tasks are scheduled as automated as possible. Using checklists can assures that you are not

surprised by activities that could easily be addressed if only monitored in a very organized fashion.

Writing a daily tasks list, or SOP, and also create periodic tasks lists for other activities can assure that you have a very high

degree of business continuity. HANA is a highly scalable platform with a substantial amount of built-in fault tolerance at the

hardware and software level. Frankly, system failures are rather unusual. However, like all databases, no amount of

‘cleverness’ can totally prevent issues if the utilization is not monitored on a regular basis, and preventive measures are

taken well in advance.

23

In this article we explored some of the new, and existing admin tools for SAP HANA and provided an informed opinion on

how to best leverage the available system and admin alerts available. We also looked at some of the most common system

admin tasks you should periodically perform on a SAP BW system and how to automate them as much as possible,

HANA is now almost 5 years old and best practices are starting to get solidified. So, as you start writing your own tailored

SOP, I recommend you leverage access to the SAP community, SAP employees and skilled HANA consultant resources with

real hands-on experience to guide you through the process. However, the items in this article should at least get you

started.

Dr. Berg