How Simple Finance Removes Redundancy
Posted byJens KruegeronSeptember 30, 2014More by this author11At
SAP, we set out to change how finance is done. Traditional systems
relied on inflexible models, precomputed data, and slow
computational systems. With SAP HANA and SAP Simple Finance we
changed that, to give you a more agile approach to financial
management and planning. By eliminating things such as needleless
duplication and precomputation of data which clogs other systems,
we have significantly lowered your cost of managing Financials. And
because our experience at building robust accounting systems, we
can do this without disruption to your business. By freeing your
system of the constraints of precomputed data models, we also open
up your system to a more agile way of computing, allowing financial
analysts to quickly experiment with what-if models while speeding
quarterly closes.This blog post is the first of a series that dives
deeper into various aspects of SAP Simple Finance and their
technical underpinnings.Recently,Hasso Plattner wrote about the
impact of aggregatesand explored the negative impact of
materialized aggregates and why relying on them for performance
severely restricts flexibility. Generally speaking, the generic
termmaterialized viewand the special casematerialized
aggregaterefer to the physical storage of derived and redundant
data in a database. In this deep dive, we look closer into the
technical foundations and highlight the positive impacts that can
be realized when removing materialized views. We do this by looking
atSAP Simple Finance SAPs next-generation Financials solution that
for the first time does accounting without materialized views
powered by SAP HANA. We are going to explore why materialized views
and materialized aggregates are no longer necessary and how
removing redundant data storage in a non-disruptive manner improves
transactional throughput and lowers storage costs without
compromising analysis performance.In this blog post, we focus on
the non-disruptive changes to the data model that removed
redundancy from SAP Simple Finance and why switching to Simple
Finance is possible in an entirely non-disruptive manner. In the
next two blogs we will investigate the concepts of materialized
views and materialized aggregates, respectively, and demonstrate
that it is indeed feasible with in-memory database systems to get
rid of these redundant constructs. Future parts of the deep dive
series will also highlight additional improvements and paradigms of
Simple Finance that are possible thanks to SAP HANA and focus on,
for example, the business value associated with Simple Finance,
non-disruptive innovation, and how Simple Finance enables decision
makers to overcome aggregate information loss.In this first part of
the series, we begin withexploring the concept of redundancy in
general. Afterwards, we look deeper into thechanges of the data
modelbrought with Simple Finance and highlight how
thisnon-disruptive innovationshas been possible, allowing an almost
seamless switch-over to SAP Simple Finance. We demonstrate the
positive impact of the new data model ondatabase
footprintandtransactional throughput.While we focus on the
Financial Accounting component, other components such as Management
Accounting (Controlling) have similarly been changed, so our
comments apply to Simple Finance as a whole. The following
explanations focus on the classical data model in order to foster
easier understanding for all readers, even those not familiar with
New General Ledger Accounting (New G/L). Please note that SAP
Simple Finance is using New G/L.SAP HANA Makes Redundant Data
ObsoleteRedundant datarefers to data that is physically stored in a
database but could be derived by calculating it from other data in
the database (Codd: A Relational Model of Data for Large Shared
Data Banks; Communications of the ACM 13(6); 1970). Redundancy is a
frequent source of inconsistency and anomalies. As redundant data
needs to be kept in sync on updates, redundancy in a data model
leads to slower updates. To reduce redundancy, database
normalization techniques such as normal forms have been introduced
(Codd: Further Normalization of the Data Base Relational Model; IBM
Research Report RJ909; 1971). Historically, redundant data has
nevertheless often been stored explicitly only to improve read
performance, as the calculations to derive it in the absence of
materialization required too much additional effort.For years,
enterprise applications have employed redundant data storage in
order to provide sufficient performance to users in transactional
and analytical applications alike. The limited performance of
traditional, disk-based database systems required redundantly kept
data that could be accessed quickly, but needed to be updated and
kept in sync with transactional changes. SAP ERP Financials has
historically not been an exception. In view of millions or billions
of accounting documents (headers, stored in tableBKPF) and their
line items (tableBSEG), materialized views and aggregates as
mechanisms to maintain redundant data provided fast access to items
with specific properties. For example, SAP ERP Financials contained
a materialized view for all open Accounts Receivable line items
(tableBSID) and materialized aggregates of the debit, credit, and
balance amounts for each customer per fiscal period
(tableKNC1).Building on SAP HANAs in-memory technology, it has now
been possible to non-disruptively transform the Financials system
into a purely line-item-based Simple Finance that gets rid of all
redundant financials data. Thus, SAP Simple Finance overcomes the
associated costs such as reduced transactional throughput and
increased database footprint. At the same time, SAP Simple Finance
is a non-disruptive innovation of the classical ERP Financials
because it replaces the materialized views with non-materialized
compatibility views. All applications that have read from the
materialized views, be it SAP standard reports or customer
modifications, continue to access the views as before without
requiring any changes.Although these are already big breakthroughs,
the most important advantage of SAP Simple Finance is the
dramatically improved flexibility that, for example, encourages
exploring the data according to various analysis needs and
contributes to the success of Simple Finance as an information
system. Researchers have since long stressed the value of
information flexibility see, for example, Searching and Scanning:
How Executives Obtain Information from Executive Information
Systems (Vandenbosch and Huff, 1997; MIS Quarterly 21(1)),
Exploring the perceived business value of the flexibility enabled
by information technology infrastructure (Fink and Neumann, 2009;
Information & Management 46), or An empirical investigation of
the factors affecting data warehousing success (Wixom and Watson,
2001; MIS Quarterly 25(1)). A follow-up blog post will elaborate on
the business value that can be realized thanks to the gain in
flexibility and the simplification.A New, Non-disruptive Data Model
with Zero RedundancyThe fundamental and essential data tuples of
every financial accounting system are the accounting documents and
line items, in this case stored in the tablesBKPFandBSEG,
respectively. The traditional data model of SAP ERP Financials
which is running on SAP HANA as part of SAP Business Suite powered
by SAP HANA (in the following: Suite on HANA) additionally
contained materialized views that allowed fast access on disk-based
database systems to open and cleared line items, separated by
accounts receivable, accounts payable, and general ledger accounts
(tablesBSID,BSAD,BSIK, ). As calculations and observations in
future installments of this deep dive series will show, SAP HANAs
in-memory database makes materialization obsolete thanks to the
improved performance. Hence, the decision for SAP Simple Finance
was clear: the previously existing materialized views have been
removed and replaced by compatibility views. These compatibility
views transparently provide access to the same information
calculated on the fly to ensure that the changes are
non-disruptive.Materialized aggregates in the traditional
Financials data model included totals per account receivable
(KNC1), account payable (LFC1), and G/L account (GLT0/FAGLFLEXT).
In addition to removing the materialized views, SAP Simple Finance
also evolves the traditional Financials data model by removing the
materialized aggregates from the system, again replacing them by
compatibility views.In the end, the transactional data model of
core Financial Accounting besides master data consists only of the
essential accounting document header (BKPF) and accounting document
line item tables (BSEG). All the other tables mentioned on the left
side of the following Figure 1 are then obsolete. Instead,
transparent access to exactly the same information is provided
on-the-fly by equally named compatibility views, as outlined on the
right side of Figure 1.
Figure 1: Key elements of Financials data model before and after
simplificationNon-disruptive InnovationSAP ERP Financials is being
used all over the world by many thousand customers. It offers rich
modification options to customer to adapt the system to their
needs, including custom modifications. Changes to the core data
model need to take this into account. They have to be
non-disruptive with regard to the large amount of code (by SAP and
its customers) that builds on top of the data model. To ensure
adoption of the benefits of SAP Simple Finance, all innovations
have been implemented in a non-disruptive manner. Switching to SAP
Simple Finance does not require updating any custom coding for
reading from the database.So-called compatibility views have been
the means to achieve this non-disruptiveness: acompatibility viewis
a non-materialized view essentially, a named query that seamlessly
replaces a materialized view (or materialized aggregate). As it has
the same name and is transparently accessed via the data
dictionary, existing applications that so far relied on the
materialized view for accessing items or aggregates do not
experience any difference. They continue to work as before without
any changes to the code. For example, a query accessingBSIDhas
previously read the value from the materialized view. Now, the same
access toBSIDis resolved into a query using the view definition and
calculated on-the-fly. The result is the same in both cases. As the
calculations have demonstrated, this is feasible without
compromising performance.As a consequence of the non-disruptive
innovation, all of the benefits associated with Simple Finance such
as increased throughput, flexibility, and reduced database size are
available at no adaptation costs to customers a migration of the
database (mostly removing the redundant tables) is sufficient.A
future deep dive will look into non-disruptive innovation and the
technology of compatibility views in more detail.As a benefit of
the non-disruptive nature of the above-mentioned changes, code
changes in the Financials component (done by SAP) were only
necessary to remove the now unnecessary writing operations to
former materialized views and aggregates when posting accounting
documents. In addition, code has been adapted to further leverage
the possibilities of SAP HANA. This includes code pushdown of
data-intensive logic to the database and backend decoupling for new
Web-based user interfaces (SAP Fiori).Reduction of Database
FootprintThe removal of both materialized views and materialized
aggregates significantly reduces the database footprint of
Financials that has been occupied by redundant data. In an
exemplary system from an SAP customer, the total size occupied by
the set of core tables for financial accounting is reduced by a
factor of 6.5 with SAP Simple Finance. For details, please refer to
the following table that includes a non-exhaustive list of tables
that have been replaced by compatibility views.Table sizesof
customer system
TableDescriptionERP Financials in Suite on HANASimple Finance on
HANA
BKPFAccounting documents1.2 GB1.2 GB
BSEGAccounting document line items4.5 GB** 5.0 GB
BSIDOpen items Accounts Receivable0.1 GB
BSIKOpen items Accounts Payable0.0 GB
BSISOpen items General Ledger* 23.2 GB
BSADCleared items Accounts Receivable3.3 GB
BSAKCleared items Accounts Payable2.1 GB
BSASCleared items General Ledger* 5.5 GB
KNC1Totals Accounts Receivable0.1 GB
LFC1Totals Accounts Payable0.0 GB
GLT0Totals General Ledger0.0 GB
Total (of above tables)40.0 GB6.2 GB
* Includes partially archived items (not considered here)**
Includes additional fields for New G/L
The numbers shown above are in both cases already for a system
running on SAP HANA as the database. Factoring in the additional
savings in database size enabled by HANAs storage architecture and
compression, the potential savings are even more impressive.
Looking at SAPs main internal ERP system, the following Figure 2
demonstrates the end-to-end possibilities in database size
reduction. The combination of SAP HANAs inherent compression with
the removal of redundancy allows running a company such as SAP with
revenue of close to 17 billion Euro in 2013, over 250,000
customers, and over 65,000 employees completely in-memory using
commodity enterprise hardware.
Figure 2: Database sizes of SAPs main internal ERPIncrease of
Transactional ThroughputThe removal of materialized views and
materialized aggregates immediately has a positive impact on the
transactional throughput of the system. Posting an accounting
document no longer requires inserting redundant duplicates into
materialized views nor updates of redundant total figures.
Beforehand, especially the concurrent aggregate updates could lead
to contention as materialized aggregates had to be locked for
updating. In case of, for example, heavily used G/L accounts, the
database system had to handle the otherwise parallel postings
sequentially. In Simple Finance, posting now only requires
inserting the actual document and its line items. Experimental
measurements based on customer data (see Figure 3) have shown that
the end-to-end transaction time required for posting 500 documents
with 6 line items each has been reduced by a factor of more than 2,
doubling the throughput. The number of tuples affected by the
database operations for posting the documents has been reduced by a
factor of almost 2.5 thanks to the no longer required
synchronization of redundant data.
Figure 3: Increased transactional throughputBenefits of SAP
Simple Finance in SummaryThe data model of SAP Simple Finance is
now exclusively consisting of line items. It is simply recording
all business transactions as they happen. Everything else is being
calculated on-the-fly by algorithms on the data. This implies that
the previously needed modifying operations to maintain the
materialized views are no longer necessary, thereby simplifying the
program code and increasing the transactional throughput.The fact
that it is feasible to remove the redundancy from the Financials
data model demonstrates that in-memory database systems enable new
levels of flexibility. Instead of pre-defining materialized views
and materialized aggregates, it is possible to aggregate flexibly
directly on the line items. Instead of being restricted to specific
analysis questions supported by the underlying model of
materialization, all questions are feasible that can be answered
based on the actual items themselves via any aggregation on top of
them. An improved and user-friendly user experience adds to that
and allows unprecedented user productivity and novel insights.This
first blog post of the Simple Finance deep dive series looked at
the non-disruptive changes to the data model. In particular, we
outlined the practical benefits associated with removing redundancy
of materialized views and materialized aggregates; increased
throughput and reduced database footprint. In the upcoming blog
posts of the series, we will demonstrate how SAP HANA as an
in-memory database makes these changes feasible using performance
calculations and technical considerations.Average User
RatingRating: 5.0/5(4 votes cast)2685ViewsPreviousNextComments1.
Vaidhya on September 30, 2014Excellent article! keenly awaiting the
next posts Log in to Reply Like(0) 2. mkr on October 13,
2014Outstanding. I get it! Log in to Reply Like(0) 3. mkr on
October 14, 2014Can you write about central journal concept. Log in
to Reply Like(0) Jens Kruegeron October 22, 2014mkr, thank you for
your interest. Yes, a future blog post will discuss the Central
Finance concept. Log in to Reply Like(0)4. Evgenii Tikhomirovon
October 21, 2014What happens if BSEG and BKPF themselves will be
migrated into many fully normalized tables and replaced by views?
Clearly these are hugely redundant tables with > 300 fields. Log
in to Reply Like(0) Jens Kruegeron October 22, 2014Evgenii, the
tables for accounting document header (BKPF) and line items (BSEG)
remain the basis for all financial accounting in Simple Finance (as
shown in figure 1). They continue to record the fundamental
transactions and are more than ever the single source of truth, as
aggregates are derived on-the-fly from them. The data in these
tables is thus not redundant. Customers use the existing fields of
these tables according to their needs. They willnotbe migrated to
many separate tables. Log in to Reply Like(0) Evgenii Tikhomirovon
October 22, 2014Technically, from the point of relational
normalization theory, the tables BKPF and BSEG are redundant,
because theyre not in 6NF.But youre right, Ive completely forgot
that these tables should be updated. And HANA (likely) doesnt
support view updating. Log in to Reply Like(0)5. GBWis on October
28, 2014I am new to SAP and trying to comprehend Simple Finance.
Are the Ledgers obsolete too or the transactions still would have
to post to a ledger table ? It would be nice if Ledger updates and
Average Daily Balances dont have to be calculated each time you
post something. Log in to Reply Like(0) Gregory Misiorekon January
19, 2015GBWis,Simple Finance is very simple, so to speak. the idea
of posting an accounting entry is to actually update at least 2
balances, which in turn net to 0. aggregation has been needed for
generating statements, but with calculation on-the-fly its no
longer necessary as each insert results in a new set of subtotals
that dont exist anywhere but in memory. Log in to Reply Like(0)6.
Mike on October 30, 2014Hello Jens,I am sure that the fact that
materialized views are not needed any longer will enable many new
use cases that have not been possible before (because of the
described overhead).However, can you quantify the number of
materialized aggregates that already have be removed and the
potential (number) of materialized tables that remain to be removed
in future? (for sFI, ERP, )BR, Mike Log in to Reply Like(0) Jens
Kruegeron January 19, 2015Mike,for your reference, these
materialized views have been removed: BSIS, BSAS, BSID, BSAD, BSIK,
BSAK, FAGLBSIS, FAGLBSAS; and these materialized aggregates: GLT0,
GLT3, FAGLFLEXT, KNC1, LFC1, KNC3, LFC3, COSS, COSP.Other areas are
now also replacing materialized tables with on-the-fly calculation
based on SAP HANA. Log in to Reply Like(0)7. Justin Mutindaon
January 16, 2015Thanks Jens,The non-materialized views have the
same name as the replaced materialized views. Do the
non-materialized views have the same structures as the replaced
materialized views as well? Log in to Reply Like(0) Jens Kruegeron
January 19, 2015Justin,yes, the compatibility views (as we call the
non-materialized replacement) have the same structure as the former
materialized views they replace. This ensures that all code reading
from, for example, BSID seamlessly works as before and returns the
same values, but calculated on-the-fly. Log in to Reply Like(0)