SAP Solution Manager Solution Documentation and Test ......• Scope: Upgrade all SAP systems (3xECC, BW, CRM, SCM, Portal, TREX, ... • Evaluate the available Focused Build solutions

Test SuiteKnowledge sharing - Solution Manager 7.2

By: Endre Rasmussen

At: Coop Norge

Date: 07.06.18

Expectations“Experience Sharing and Demonstration of Test Suite,

Solution Manager 7.2 by Coop”

Coop will share their experience of using Test Suite in Solution Manager 7.2. Learn and

understand how Solution Documentation is the foundation to successful use of Test

Suite. Gain insight and build experience in test planning, reporting, execution and defect

handling with integration to ChaRM.

Agenda

• Facts about Coop

• SolMan usage in Coop

– High level concept

• Solution Documentation

– Experience and learning points

• Test Suite

– Experience and learning points

• Demo

Facts about Coop

SAP landscape

• ECC Retail, ECC Industry, ECC slag, SCM, CRM, BW (HANA/ABAP), CAR (HANA/ABAP), SLT, Enterprise Portal, NWDI

• Three tier system landscape + sandbox

• >4000 users

• Solution Manager 7.2 SP6

Data from March 2017

28.000employees

1,5 million members

45 billionin turnover

1350stores

29,4 % marked share in

grocery

Custom Code Management

• Code Inspector (integration to ChaRM)

• CCLM – Strategic roadmap in progress

High-level concept in CoopOne Solution with integration to other modules

Business Process Improvement - PILOT

• Measure ROI based on usages statistic

in the production system

• Identify critical business processes

based on most used objects

• Identify improvement areas based on

not used objects (lack of training etc.)

• Create Solution reports for audit and

new projects

Change Control Management

• Integration to ChaRM documents

• Process structure and document

change control

Business Process Operation

• Business Process Monitoring (BPMon)

• Job Monitoring

• Data Volume Management

Test Suite

• Link to both manual and automated (CBTA)

test scripts

• Business Process Change Analyzer (BPCA)

• Scope and Effort Analyzer (SEA) - pilot

Solution Documentation

Solution DocumentationOur «Single Source of Truth»

• Solution: 1 Solution, 75 logical component groups, 2 process groups

• E2E Processes: 5 Scenarios, 9 Processes, 159 Process steps, 7 diagrams

• Modular Processes: 144 Scenarios, 458 Processes, 2395 process steps, 785

executables, 214 diagrams, 4385 test cases (only 1500 assigned to processes)

Statement: Process modelling is the foundation for control and measure of our solution, and

a common visual language between IT and Business

Solution DocumentationCoop Experience and learning points

General recommendations

- Recommend to upgrade to at least SP05 with possibility to define business process variants

- Identify the processes responsible to ensure ownership and continues maintenance

- Recommend numbering the defined processes as part of naming standard, for easier

reporting and search

- Define needed customer attributes from start to simplify the search and reporting (via

SOLMAN_SETUP)

- Use collaboration diagram in order to allow modelling of interfaces to third parties

- Define how the process can be measured and controlled (input to BPMon)

General experience

- Very good search and filtering functionality

- Experience that IE browser is slow to work with, Chrome is better but does not have the edit

online function for document handling

- User training is needed in order to navigate in a good and efficient way

Solution DocumentationCoop Experience and learning points

Relevant for Test Suite

- Decide whether to create and assign test cases on step or process level

- Create a manual test case template for consistency (expected result, actual result etc)

- Recommend to ensure test cases reflects the modelled processes, including interfaces

- Recommend to assign custom attributes to test cases, for easy test planning (example:

test phase relevance, criticality, relevant for test automation)

- Note that test cases not possible to be created for interfaces in interfaces library. This

makes it hard to report and sign off testing of critical interfaces

Modell process Create test case based on templateTest Case reflects

the process modell

Test Suite

Test SuiteExperience taken from SAP upgrade project in Coop

• Scope: Upgrade all SAP systems (3xECC, BW, CRM, SCM, Portal, TREX, Java Hub, NDWI)• Timeline: 6 months (April – October 2017)• Facts: 38 systems upgraded, 130 involved people, 1766 manual test cases executed, 159

errors detected and corrected

TEST PHASE PACKAGES TEST CASES NO RESULT

IN PROGRESS ERRORS OK DEFECTS COMPLETION (%)

SANDBOX (ST) 124 503 16 4 16 467 36 93

DEVELOPMENT (DT) 9 89 21 5 2 61 2 69

USER ACCEPTANCE TESTING (UAT)

251 931 0 0 0 931 119 100

PRODUCTION VERIFICATION (PRODT)

65 308 0 0 1 307 2 100

TOTAL OF ALL TEST PHASES 449 1831 37 9 19 1766 159 100

One of the key success factors was the preparation work, identify and secure test coverage of critical business processes and related interfaces

Test SuiteHigh level Test flow

• Run BPCA to scope test

• Creation of Test Plan, Test

Package and Test Case and

Tester assignment

• Creation of Test Reports

• Test defect handling in ITSM

• Using Defect Correction in

ChaRM (close link to test defect)

• Test Execution via Tester

Worklist

• Test Note creation

• Test Defect creation

Tester DeveloperTest

Manager

Planning Execute Defect handling

Test Suite - Test PlanningExperience and Learning points

Experience

• Nice integration with Solution Documentation, structure is maintained and well known

• Note that changes in test case (SolDoc) will not be visible in existing test packages, new test

package needs to be created and assigned to tester

• Missing mass update function of tester in test packages for Test Manager, comes with SP05

• Good reporting possibilities (drill down in all levels)

• BPCA analyse is complex and requires detailed investigation in order to understand the result

• No experience in Test Case sequence, possible but not used

Learning points

• Decide on which level in structure the test plan and test packages should be created for easy

reporting and follow up

Test

Manager

Example of report:

Test Suite - Test ExecutionExperience and Learning points

Experience

• Tester recognize the structure in tester worklist from the process structure, which is good

• Too many pop ups/different windows when working with manual execution. Solved with Focus Build

• Test Cases opens in a separate window, not embedded as in 7.1. Solved with Focus Build

• Many refresh buttons under Tester Worklist. Needs to be refreshed manually

• Start Test Execution button makes sense if you have test case on process step level (Coops

decision to have test case on process level). Solved with Focus Build

• No experience in TBOM recording, decided not to use refer to statement above

Learning points

• Decide the use of Test Note and Test Result, and method (check in check out or upload new

version)

• Evaluate the available Focused Build solutions offered by SAP related to Test Suite (Simplified test

execution etc)

NEWS: Focused Build for 7.2 - New manual test solution with grid approach fully

embedded in Test Suite etc – link (slide 27-45)

Tester

Test Suite - Defect HandlingExperience and Learning points

Experience

• Nice follow up document integration between Test Defect (Incident) and Defect Correction

(ChaRM), text and field entries can be copied over

• (SP05) No interlink when processing Test Defect and Defect Correction. Developer needed to

work on both Test Defect and Defect Correction document when updating status etc

• No direct link to test notes from Test Defect, only link to test case in SOLDOC (where actual

result is blank) Our method was to attach test note to Test Defect

• Note that Testers must go into ITSM in order to reply and confirm reported Test Defects (direct

link from tester worklist or mail notification)

Learning points

• Analyse the standard Defect Handling workflow and evaluate if customizing is needed

• Upgrade to SP06 in order to get the status interlinked between Test Defect and Defect

Correction which can cause manual work and continual progress*

*Shipped with SP06: Status profiles for Test Defect and Defect Correction interlinked – link (slide 24)

Developer

Demo

SolMan 7.2 SP6