An Automated approach towards Dev-Ops based continuous Regression Testing for SAP Landscapes Navaneetha Kowdle – Associate Director Akshay Mullick - Consultant Cognizant
An Automated approach towards Dev-Ops based
continuous Regression Testing for SAP Landscapes
Navaneetha Kowdle – Associate Director
Akshay Mullick - Consultant
Cognizant
Abstract
Executive Summary
Wedged between an increasingly convoluted digital landscape and the obligation to expedite
releases, the world around today’s QA Organization has been turbulent. QA Managers face a
daunting task of assuring complex business applications yet maintaining a pre-meditated
balance between quality and speed. This solution approach manuscript provides an
introduction to Business Process Change Analyser (BPCA) as an effective tool that increases
Test Coverage and Test Case Optimization for large and medium-sized SAP releases. The
document explores a two-phased approach towards a robust regression suite to perform BPCA
based impact analysis and analyses the benefits realized by one our major Customer.
Correlating Impact analysis to Business Process Change Analyser (BPCA)
The surge in organizations ‘going’ digital has augmented the complexity of the ERP landscape.
With a mission to deploy features at a shorter period, it made sense for SAP deployment
managers to consider a viable impact analysis tool during design and before the testing phase
(especially regression testing) to optimize test effort in addition to unearthing possible gaps in
coverage. Initially limited to increasing Test Scope Optimization (TSO) for SAP, the impact
analysis has now become a viable tool of choice for many SAP consultants and deployment
managers. Applying impact analysis during the design phase enables easy effort estimation for
change while a post development impact analysis aids in the identification of test cases that
were impacted by change.
Its primary focus is to identify affected modules in the system that were introduced through
new features or upgrades. By quickly identifying and focusing on regression testing of
impacted areas, impact analysis reduces regression cycle time, decreases cost and at the same
time ensures a focused approach to testing.
Business Process Change Analyser (BPCA) is an in-house tool provided by SAP that exposes
modules impacted due to modifications made through new features or change requests. It
helps compare the changes brought in through the transports with the current system and
identifies all the impacted areas (Objects). With the right configuration and a positive link
between the test management solution and the system, BPCA guides test scoping and reduces
the cost to testing by enabling businesses in identifying scenarios impacted by change. It
supports a wide range of areas of SAP modules including ECC, CRM, APO, SCM etc. BPCA also
has the advantage of being able to map impact changes to specific test scenarios. This is
necessary as sometimes it may be required to identify process and data variants to be tested
due to changes.
The Business Situation
The need for impact analysis began when one of our customers began investigating the list of
Urgent incidents that had an impact on their business. The analysis was conducted to
recognize Regression Testing Gaps that required attention despite the thoroughness and rigor
in the customer’s regression testing capability.
The analysis considered all the Urgent Incidents that were raised throughout the year 2014
against its IT Services. Based on these findings the customer wanted to:
1. Improve the quality of regression testing by increasing the granularity of transaction
coverage i.e. Identify & Test scenarios that can test object coverage instead of
transaction code
2. Increase Test Coverage by identifying gaps within the existing regression test suite and
designing new scenarios to plug the gaps
Considering the above, the customer worked collaboratively with cognizant to come up with a
viable solution. Post an exercise that compared multiple commercially available tools, it was
decided that BPCA will be the tool of choice to enable impact based regression testing.
Execution Approach
The deployment of BPCA was conducted in 2 distinct phases
Phase 1 – Set-up and Pre-requisites
BPCA relies on 2 key inputs for its analysis. A comprehensive Business Blueprint (BBP) –
solution snapshot and a Technical Bill of Materials (TBOMS) which is a list of objects assigned
to various Business Process in SAP. The quality of the analysis conducted and the ability of
BPCA to analyse is dependent on the breadth of the Business Blueprint and depth of the
TBOMS.
SAP automatically provides a means to build business blueprints and compile static or semi-
dynamic TBOMs. However, the maximum potential towards test scope optimization or test
coverage gap identification could only be realized if TBOMs generated from existing test assets
(Dynamic TBOMs) and the BBP is constructed as per the structure in the test management
tool.
Since the customer was pre-equipped with a robust set of artifacts and SAP TAO automated
scripts were readily made available, creating Dynamic TBOMs and structuring the BBP was
easier than starting from scratch. The Dynamic TBOM build went through a two stage process
-
a) Manually building a business blueprint that mirrored the E2E test scenario folder
structure in the test management tool (e.g. QC-ALM). BBP entries and test cases were
then mapped together which further allowed for impacted test scenarios to be
categorized on the basis of time and effort before the regression test cycle
b) Enable the Solution Manager to generate TBOMs while the test cases were being run
from the test management tool. This generated corresponding dynamic TBOMs for
each TCodes /Test scenario in the regression suite.
Phase 2 – Deployment and usage
To cater to the monthly release cadence that comprised of medium and major releases, the
BPCA tool was deployed to perform impact analysis. The tool empowered QA Managers to
focus on testing areas that were impacted by the changes, thereby reducing effort and cost.
Improving Test Scope Optimization by implementing BPCA
Conventionally, regression testing requires test scenarios in a particular landscape to be re-
tested – manually or through automation. However, the implementation of BPCA and the
results obtained through Impact Analysis empowered the customer to sort and rank test
scenarios as per preference.
Immediate effects of BPCA are illustrated in the diagram below:
BPCA analysis conducted over multiple monthly medium releases have indicated that the
effort and elapsed time savings in regression testing can be around 30 - 80%, depending how
BPCA becoming parameterized and on the risk-appetite of decision-makers.
Increasing Test Coverage for SAP Testing through BPCA
Although BPCA does not provide information on test coverage but provides data for TSO, the
analysis and data points that are made available supports the Testing team in recognizing the
adequacy/inadequacy in coverage for critical changes.
BPCA is executed on a set of transports that are involved in code or configuration changes that
could have impacted objects.
Dynamic TBOMs created by using the regression suite will have complete knowledge of objects
covered by the test scenarios – one might call it the “Coverage DNA” of a scenario. Therefore,
to increase the Test Coverage, the tangible way forward was to identify Objects/Scenarios that
weren’t covered by the TBOMs list but are present in the set of transports being analysed for
impact analysis. By comparing the list of known Objects with new Objects discovered in the
transports, new scenarios that covered the new objects were added to the regression suite and
this resulted in additional test coverage.
A comparison between the lists of objects changed by transports as against those provided by
TBOM analysis facilitates in the discovery of objects that aren’t covered by the regression test
suite. As Standard BPCA does not provide gap identification in the granularity level as
required by the client, we requested a SAP partner to develop an Add-On that fully utilizes the
granularity level as provided by TBOMs and also identify gaps in a more granular approach
than what BPCA provides in SAP standard.
Implementation of the above has shown an increase of an additional 5-10% unique test
scenarios qualifying as opposed to former baseline approach.
Phase 3 – Moving towards Dev-Ops based continuous regression testing
Implementation of BPCA on the SAP landscape and consistent usage opened up a host of
opportunities to take it to the next level. Traditionally SAP landscapes are complex and are not
a first choice candidate to consider for continuous testing. But the advantages BPCA provided
to implement focused regression testing pushed the team to consider a method to establish
just that. The result was a home grown automation tool that performs an End to End
automated regression testing on all changed transports with zero human intervention. The
tool integrates ChaRm, SolMan and HP-ALM to provide an end-to-end mechanism to identify
changed transports, perform impact analysis, identify regression scope and complete
automated regression testing daily. Not only has this helped the team identify defects earlier in
the lifecycle but the Regression Testing window that typically used to happen right at the end
of the release cycle over 1 week has been reduced to 2 days.
Conclusion
The implementation of BPCA and continuous testing has provided the customer with early
benefits such as reduced cycle time and increased test coverage by identifying optional
scenarios and new scenarios. And although the project continues to proceed as usual, it is
estimated that the customer stands to significantly gain through savings in efforts and cost.
Client’s comment:
After more than four years of executing Regression Testing in a so-called Baseline approach,
i.e. executing per release a fixed set of test scenarios mirroring our most critical business
processes, we started investigating into a more intelligent approach helping us to focus our test
execution on the change-impacted parts of the system and more importantly to strengthen the
RT capability regarding coverage.
Cognizant as our Testing Services provider was supporting us in market screening, in tool
evaluation, identifying our key demands and finally also in planning and implementing the
chosen toolset.
By end 2016, we have been successfully implementing BPCA in the European region whilst we
are still working on rollout to two more regions (AAR and Americas).
The benefits, especially the strengthening of our RT capability has turned out as successful for
Europe. Second stage in 2017 will be to leverage the focused RT capability and move into
continuous Regression Testing rather than it manifesting into a dedicated phase per release
cycle. This will become one key enabler for moving into a more agile Continuous Deployment
mode even for ERP Releases.
Note: Second stage as described the customer has already happened and is covered in the
paper above as “Phase 3 - Moving towards Dev-Ops
Client Testimonial from Dec 2016
Author Biography
Navneetha Kowdle is an Associate Director with the Process &
Quality consulting division (a part of Cognizant Business Consulting
group). In this capacity, his role entails overall management of PQC
practice in the Communication, Media and Technology space, in
addition to personally getting engaged with customers in providing
consultancy services in niche areas of Quality Assurance such as
DevOps, Business Process Testing, QA transformation strategy etc
Akshay Mullick is working as a Consultant with the Process &
Quality consulting division (a part of Cognizant Business Consulting
group). My role is to act as a bridge between business and
technology for client and propose solutions which address their
problem statement. My area of expertise is Agile, Dev-Ops, Change
Management, and Tools & Automation around Impact based testing.
Main Author
Co - Author