This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
For Attention Of BSC Parties and other interested parties
Overview or Purpose of Document:
Purpose
The purpose of this document is to define the testing to be undertaken for the changes included in the June 09 Release. These changes are defined in the relevant Business Requirements Solution documents (Reference 5, Reference 6, Reference 7).
This document is produced in accordance with the Change Delivery Guidelines for Testing Changes to BSC Systems (Reference 1).
3 Phases of Testing Details....................................................................................11
4 Test Management ...............................................................................................16
5 Assumptions, Risks and Issues...........................................................................18
6 Document Control ...............................................................................................19
A Appendix A – Terms used in this Document .......................................................20
B Appendix B – Test Specifications & Scripts that will be run...............................21
Intellectual Property Rights, Copyright and Disclaimer The copyright and other intellectual property rights in this document are vested in ELEXON or appear with the consent of the copyright owner. These materials are made available for you for the purposes of your participation in the electricity industry. If you have an interest in the electricity industry, you may view, download, copy, distribute, modify, transmit, publish, sell or create derivative works (in whatever format) from this document or in other cases use for personal academic or other non-commercial purposes. All copyright and other proprietary notices contained in the document must be retained on any copy you make.
All other rights of the copyright owner not expressly dealt with above are reserved.
No representation, warranty or guarantee is made that the information in this document is accurate or complete. While care is taken in the collection and provision of this information, ELEXON Limited shall not be liable for any errors, omissions, misstatements or mistakes in any information or damages resulting from the use of this information or action taken in reliance on it.
The June 2009 Release (“June 09 Release”) involves a number of changes to the Central Systems, including the BMRA, CRA, CDCA and ECVAA Applications. These changes will:
• Introduce new Credit Cover arrangements for BM Units, including a new value to be included in the calculation of Energy Indebtedness (EI), Metered Energy Indebtedness (MEI), to be calculated in ECVAA from Metered Volumes provided by the CDCA, where applicable.
• For Production BM Units who do not qualify for the MEI value, Final Physical Notifications (FPNs) will be used instead of the Generations Capacity (GC) and Credit Assessment Load Factor (CALF) value to calculate CAQCE.
• This calculation will apply only to ‘Credit Qualifying BM Units1’, which will be added as a category in the CRA.
• There are also minor impacts on TOMAS and Gatekeeper software.
• Introduce pages to the BMRS for the publication of market-critical Large Combustion Plant Directive (LCPD) data, as submitted to BSCCo by BSC Parties.
1.1 Objective
The objectives of this test strategy for the June 09 Release are as follows:
• to describe the overall testing framework;
• to define the testing to be undertaken;
• to define the responsibilities of the various parties involved in testing;
• to ensure that all areas of testing requirements are addressed; and
• to ensure that the coverage of the testing is adequate to meet those acceptance criteria that are subject to testing.
This test strategy covers the testing required to provide ELEXON with the assurance that:
• the changes made to the trading arrangements support the requirements and solutions defined in the Business Requirements Solution documents relevant to the June 09 Release (Reference 5, Reference 6, Reference 7);
• the changes made have not adversely impacted unchanged software systems and documents that support the trading arrangements; and
• the changes made conform to the BSC and Code Subsidiary Documents.
1.2 Scope
The scope of this document is the testing of the changes to the BMRA, CRA, CDCA and ECVAA software Applications to deliver the Business Requirement Solutions for P215 and P226 to be
1 A BM Unit shall be determined a ‘Credit Qualifying BM Unit’ if its Production/Consumption flag is Production, or it is an export exempt BM Unit, or the BSC Panel has determined it to be.
implemented as part of the June 09 Release, to ensure that all new functionality has been implemented correctly and that unchanged functionality is not adversely impacted.
The changes to the NHHDA software for P222 were implemented in the February 09 Release. The Modification itself will be implemented as part of the June 09 Release.
The June 09 Release currently comprises 3 Approved Modifications and 5 Approved Change Proposals further details of which are listed below.
The following 3 Approved Modifications and 5 Change Proposals will be included in the June 09 Release:
Change Reference
Title & Description Key Impacts
P215 Revised Credit Cover Methodology
Software: CRA, CDCA, ECVAA, TOMAS and Gatekeeper.
System and Design Specs: CRA, CDCA, ECVAA
URS: CRA, CDCA, ECVAA
Service Descriptions: CRA, CDCA, ECVAA
BSCPs: BSCP15
Business Definition Documents: IDD Part 1, IDD Part 2, Reporting Catalogue
BSCCo Systems: TOMAS Data Catalogue, TOMAS System Design, TOMAS User Guide, ELEXON Obligations Register, CALF LWI, Credit Default LWI, BM Unit Registrations LWI.
P222 EAC Data to Distributors Report BSCPs: BSCP505, BSCP515
Business Definition Documents: SVA Data Catalogue Pt 1, SVA Data Catalogue Pt 2
P226 Improving Large Combustion Plant Directive Information Disclosure
Software: BMRA
BSCPs: BSCP33 (new)
System and Design Specs: BMRA
Service Descriptions: BMRA Service Description
Business Definition Documents: IDD Part 1, IDD Part 2
BSCP504, BSCP505, SVA Data Catalogue Volumes 1 and 2
CP1256 Action on Backdated D0052 flows
BSCPs: BSCP504, BSCP520
CP1257 Calculation of EAC for Temporary Supplies
BSCPs: BSCP515, BSCP520
CP1259 Distributor-Supplier Notification where a Site is capable
Business Definition Documents: SVA Data Catalogue Pt 1
CP1264 Clarification of Password Requirements in the Codes of Practice
BSCP601, Code of Practices 1, 2, 3, 5, 6 and 7.
The detailed requirements and solutions for each change can be found in the P215, P222 and P226 Business Requirement Solution documents (Reference 5, Reference 6, Reference 7) and the relevant Change Proposals.
1.3 Testing Required
Such changes to NHHDA software as required under P222 have been developed as part of the February 09 Release and are out of scope of the June 09 Release. Included in the June 09 Release are the developments to the Code Subsidiary Documents, as listed in the table above in section 1.2. Please refer to the P222 Business Requirements Solution (Reference 6) for more information.
Changes to BSC Party and Party Agent systems that are impacted by the June 09 Release are outside the scope of ELEXON. However, ELEXON will communicate with BSC Parties and Party Agents to ensure they understand the changes required and will offer them the opportunity to take part in testing, which they will be able to do as part of the Participant Testing phase. The BSC Party and Party Agent systems impacted by the June 09 Release are specified in the Business Requirements Solution documents (Reference 5, Reference 6, Reference 7).
The full details of the scope, approach and deliverables, including timescales and dates for the individual test phases are defined in the PID and Release Plan for the June 09 Release (Reference 2).
2 Testing Requirements
The ELEXON Testing Guidelines (Reference 1) have been used to identify the testing that needs to be carried out for the June 09 Release. The testing requirements are summarised in the tables and sections below.
Logica and ELEXON will run a Test Scenario workshop at which Logica will present their testing approach based on the Test Strategy outlined in this document.
Logica will be required to log all defects raised during all Formal test phases using the BSC Services Helpdesk, this excludes any defects raised during the development lifecycle before the start of the
Formal system testing (i.e. Unit, Module and System2 Testing defects will not be logged), in a way which includes information such as the following to clearly identify:
• the test phases to which the defect relates
• the application
• severity
• date or time when defect found
• Logica defect reference id
• further details of the defect
• proposed solution reference id
• details of proposed solution
• date solution will be delivered.
A Logica as the Application Management and Developer (AM/Dev)
B Logica as the Business Process Operator (BPO)
E ELEXON
I Industry
2 Logica refer to System Testing as ‘Integration Testing’ of the changed modules, and use System Testing to refer to change Specific, Regression, and Deployment Testing. For avoidance of doubt Logica shall interpret ELEXON’s use of System Testing as Integration Testing.
Change Delivery’s Testing Guidelines (Reference 1) have been used to identify the testing that needs to be carried out for the Release and this is summarised as follows:
Ch
ange
Ref
eren
ce
Bu
sin
ess
Ris
k
Scop
e
Doc
um
enta
tion
R
evie
w
Wal
k-th
roug
h/P
age-
Turn
ing
Un
it, M
odu
le, S
yste
m
Test
Ch
ange
Spe
cifi
c
Reg
ress
ion
Dep
loym
ent
Ope
rati
onal
A
ccep
tan
ce (
OA
T)3
Par
tici
pan
t Te
stin
g
P215 – system High High A, E n/a A A A A B B,E,I
P215 – Documentation High High A,E,I E n/a n/a n/a n/a n/a n/a
P222 – system Low Med n/a n/a n/a n/a n/a n/a n/a n/a
Participant Testing Participant Testing Logica, ELEXON, Industry
2.4 Entry/Exit Criteria
A test phase will be deemed to have been completed successfully:
• when all high- and medium-severity defects have been fixed and re-tested; and
• when low-severity defects have been fixed or a plan in place to address them has been agreed by ELEXON.
Test Phase Entry Criteria Exit Criteria
Unit and Module Testing
Logica design for the CVA changes has been agreed by ELEXON, the ISG and the Industry.
Unit, Module and System2 Testing has been completed successfully.
When the system works in accordance with the Module Specifications.
Unit and Module test evidence has been compiled for internal Logica review and a plan to resolve outstanding defects before the start of Formal System Testing has been communicated to ELEXON.
Formal System Testing
Development phase has been completed No Severity 1 or Severity 2 Defects are
TEST STRATEGY for the June 09 BSC Systems Release Page 8 of 21
Test Phase Entry Criteria Exit Criteria
successfully.
Module Testing has completed successfully for the previously existing defect fixes.
A test environment has been fully prepared.
All System Test Scripts have been prepared (including expected results).
outstanding.
All System Tests in Formal System Testing have been completed successfully.
When the system works in accordance with the relevant URS, SS and DS.
All defects have been logged on the BSC Services Helpdesk and a plan is agreed and Accepted by ELEXON to resolve any outstanding defects.
A Test Report has been produced by Logica and reviewed by ELEXON.
The results of Witness Testing have been recorded by ELEXON in a Witness Test Report and this report has been reviewed by Logica
BPO Testing The BPO has received the developed software and documentation from the AM/Dev.
A test environment has been fully prepared.
No Severity 1 or Severity 2 Defects are outstanding.
All System Tests in BPO have been completed successfully.
All defects have been logged on the BSC Services Helpdesk and a plan is agreed and Accepted by ELEXON to resolve any outstanding defects.
The results of Witness Testing have been recorded by ELEXON in a Witness Test Report and this report has been reviewed by Logica
A Test Report has been produced by Logica and reviewed by ELEXON.
Participant Testing Test Participants have been selected and organised by ELEXON.
Test Participants, ELEXON and Logica are all aware of the times and content of the test flows that will be sent.
All BPO Testing has been completed.
A Test Report has been produced by ELEXON and reviewed by Logica.
The test flows conform fully to the IDDs, the test flow is received by the Participant and it is in accordance with what was sent.
All defects have been logged on the BSC Services Helpdesk and a plan is agreed and Accepted by ELEXON to resolve any outstanding defects.
TEST STRATEGY for the June 09 BSC Systems Release Page 9 of 21
2.5 Logica approach to Testing
Logica will develop a Release Test Strategy describing their overall approach to the testing for the June 09 Release. ELEXON will approve the Test Strategy, subject to a satisfactory review.
They will also develop changes to the Regression Test Pack resulting from the Change Specific Testing phase of the June 09 Release. This is required to achieve acceptance of the software.
TEST STRATEGY for the June 09 BSC Systems Release Page 10 of 21
3 Phases of Testing Details
Note that this test strategy should be read in conjunction with the Logica June 09 Release Test Strategy (Reference 8).
3.1 Documentation Review
3.1.1 Purpose
All documentation identified as being impacted by the June 09 Release will be subject to a Product Review, which will be carried out in accordance with the Quality Manual (Reference 4).
The ELEXON Release Team will review the evidence and results from System Testing, Operational Testing and Participant Testing, and witness further phases of the testing conducted by Logica, to gain assurance that the tests proposed in the Service Provider’s Test Strategies, Specifications and Scripts are carried out fully and rigorously.
3.1.2 Test Scope
All documents indicated as being impacted by P215, P222 and P226 in the June 09 Release Impact Matrix (Reference 9) will be drafted, and subsequently reviewed, prior to the commencement of development work. All documentation amended by ELEXON and Logica will be given an interim version number, and will be reviewed by the ELEXON Release Team on an ongoing basis.
Documents amended by Logica, the Design and System Specifications, the URSs and the IDDs, will be provided for review in extract form, as Document Change Records (DCRs). Logica will make full versions of such documents available prior to the go-live date of the June 09 Release.
Documents amended by ELEXON, the BSCPs, the SVA Data Catalogues, the Reporting Catalogue and the Service Descriptions, will be provided to Logica and the Industry in full, with changes highlighted using tracked changes.
Logica will provide test evidence and test results from the following test phases for review by the ELEXON Release Team:
• Change Specific Testing;
• Regression Testing (P215 only);
• Deployment Testing;
• Operational Acceptance Testing; and
• Interface/Participant Testing (P215 only).
Logica will provide the necessary facilities for ELEXON to witness regression testing, change specific testing, and interface/participant testing (all P215 only).
Two Page-Turning sessions are scheduled: one for the new P215 processes and one for the new P226 processes.
The purpose of the P215 Page-Turn is to ensure that the new BSCP15 manual business processes, 3.16 ‘Application Process for Credit Qualifying BM Unit Status’ and 3.17 ‘Monitoring Process for Credit Qualifying BM Unit Status’ are workable and compliant. It will be conducted with Logica at the Test Workshop to gain understanding of their role.
The purpose of the P226 Page-Turn is to ensure that the new LCPD data submission and publication process introduced in the new BSCP33 ‘Large Combustion Plant Directive Data Submission’ are workable and compliant, and will be conducted with Logica at ELEXON premises.
The ELEXON Release Team has determined that the processes introduced under P215 and P226 are of low risk and therefore do not justify a Walkthrough.
3.2.2 Test Scope
The P215 Page-Turning will cover the following:
• BSCP processes as described in 3.2.1
• The method of communication to be used for ELEXON sending data to Logica.
• The ability and timeliness of ELEXON to produce the required evidence for presentation to the Panel.
The P226 Page-Turning will cover the BSCP processes as described in 3.2.1.
3.2.3 Procedure and Responsibilities
The Lead Analyst from ELEXON for the June Release will be responsible for organising the page-turning sessions and for completing all necessary preparation.
3.3 Unit, Module and System Testing
3.3.1 Purpose
Unit and module tests will be carried out to ensure that the changes to individual modules of the CRA, CDCA and ECVAA Applications for P215 and the BMRA Application for P226 are consistent with the changes made to the logical and physical designs.
System testing will be carried out at application level to ensure that the BMRA, CRA, CDCA and ECVAA Applications are consistent with the requirements specified in the logical and physical designs.
3.3.2 Test Scope
The Logica Development Team will carry out Unit, Module and System2 tests of the changes to the BMRA, CRA, CDCA and ECVAA Applications as part of the development phase.
There is no requirement for the Logica Development Team to provide a formal test report to ELEXON for this phase of testing as this forms part of the normal development process. However, there is a requirement for Logica to confirm that this phase has been successfully completed in line with the Project Plan and that any defects identified during this phase of testing have been, or will be, addressed.
3.4 Regression Testing
3.4.1 Purpose
Regression testing will be performed on the CRA, CDCA and ECVAA Applications in order to verify that unchanged functionality is not affected by changes implemented for P215 in the June 09 Release.
No regression testing is necessary for the P226 changes.
3.4.2 Test Scope
The Logica Test Team will perform a comprehensive regression testing exercise during System Testing for the CRA, CDCA and ECVAA Applications. The regression testing will use existing scenarios from Logica’s test pack that are relevant to P215. Logica shall verify system calculated results against independently or manually derived values. Flows between systems that are not already covered in participant testing will need to be tested by Logica.
The ELEXON Release Team may witness the regression testing.
3.5 Change Specific Testing
3.5.1 Purpose
The purpose of change specific testing is to focus testing effort on only those areas of the BMRA, CRA, CDCA and ECVAA Applications functionality that have changed.
3.5.2 Test Scope
Logica will define the scope and details of the change specific tests planned for the BMRA, CRA, CDCA and ECVAA Applications in their Test Strategy (Reference 8). This document will be reviewed by the ELEXON Release Team in accordance with the Quality Manual (Reference 4). Logica shall verify system calculated results against independently or manually derived values. Flows between systems that are not already covered in participant testing will need to be tested by Logica to ensure that the changed interfaces are functioning correctly.
The testing will include independent verification of results.
3.5.3 Additional Information
Following the successful implementation of P215 and P226, any suitable change specific tests will be incorporated into Logica’s regression test pack for the testing of future software changes.
As the changes to the CRA, CDCA and ECVAA Applications for P215 are considered to be high risk, the ELEXON Release Team will witness this phase of testing, as well as review the results of testing.
No witnessing will take place for the testing of the specific changes to the BMRA Application for P226, as they are considered to be of low risk.
3.6 Operational Acceptance Testing (OAT)
3.6.1 Purpose
Operational Acceptance Testing (OAT) will be conducted by Logica as the BPO to ensure that the CRA/CDCA/SAA and ECVAA software, for P215, and the BMRA software for P226, is operational and that changes do not impact participants’ operational requirements. OAT will focus on testing the software in the context of simulated real-world business scenarios.
3.6.2 Test Scope
Logica will carry out OAT encompassing elements of baseline/regression testing and change specific testing that focuses on the BPO’s revised operational procedures. The testing will be carried out in a full width test environment to ensure the system is operationally viable. OAT will incorporate elements of volume testing, to demonstrate that the applications continue to function and operate as expected using current production volumes without unjustifiable performance degradation. The Logica June 09 Release Test Strategy (Reference 8) will contain further details of this phase of testing.
ELEXON will conduct a dry run of the new P226 process during Logica’s OAT phase.
3.7 Deployment Testing
3.7.1 Purpose
The purpose of deployment testing is to test that the updated CRA, CDCA and ECVAA Applications for P215, and BMRA Application for P226, can be deployed to test systems and, if necessary, backed out. Deployment Testing will be carried out as by Logica during their System test phases.
3.7.2 Test Scope
As part of Logica change specific testing and Logica OAT, test environments will need to be created and the BMRA, CRA, CDCA and ECVAA Applications deployed to those environments.
3.7.3 Additional Information
For the BMRA, CRA, CDCA and ECVAA Applications, deployment testing will be performed in two phases, in parallel with change specific testing as well as following the installation of the OAT test environments. Deployment testing will only be ruled to have passed if both change specific testing and OAT have been completed successfully.
3.8 Interface/Participant Testing
3.8.1 Purpose
Participant testing demonstrates that new and amended electronic interfaces are correct and consistent with the approved IDD changes and CRA, CDCA and ECVAA Design Specifications (Reference 12, Reference 13, Reference 14), for P215. Interface testing is intended to provide
additional assurance with particular attention focused on the correction of the interfaces between systems.
Participant testing will not be conducted for the P226 change although ELEXON will conduct a dry run of the new process during the OAT test phase.
3.8.2 Test Scope
For the P215 Changes, interface and participant testing will encompass the sending and receiving of the following flows/reports:
• CRA-I014 sub-flow 5, sent to BSC Parties, containing the new Boolean fields Credit Qualifying Flag and Credit Qualifying Status;
• CRA-I020, sent to BSCCo, containing the new Boolean fields Credit Qualifying Flag and Credit Qualifying Status;
• ECVAA-I014, sent to BSC Parties, containing the new fields Metered Energy Indebtedness (MEI, in MWh), and the From Settlement Date and To Settlement Date for the MEI value.
• CRA-I009, sent from BSCCo to the CRA, containing a BM Unit Id, Credit Qualifying Flag, Effective From Date and (if required) Effective To Date.
This phase will also ensure that the data sent to users is in accordance with the P215 Business Requirement Solution (Reference 5).
3.8.3 Additional Information
ELEXON will need to liaise with the industry to identify suitable participants who are willing to partake in participant testing.
In order to gain early assurance of the P215 changes, ELEXON and Logica will carry out a limited form of interface testing during Logica’s system testing phase. Once Logica has completed its system testing phases, a more comprehensive interface testing phase will take place. During this phase, Logica will need to set up a suitable test environment for participant testing. The environment should allow for the sending, receipt and processing of the impacted and/or new flows that will be sent to Industry as part of P215.
Defects identified by ELEXON during product reviews (testing and documentation reviews) will be notified to the product developer in accordance with the procedures specified in the Quality Manual (Reference 4), and in accordance with the contracts in place with Logica.
The test procedures to be used by ELEXON will be based on procedures used in previous ELEXON projects, such as Test Witnessing Responsibilities and Procedures (Reference 10), and will be used and tailored where appropriate for the June 09 Release.
Defects raised during testing of the BMRA, CRA, CDCA and ECVAA Applications will be recorded in the Logica Test Report, and shall at a minimum include:
• defect reference
• a description of the defect
• fix date
• detail of the solution/fix
It is the responsibility of the Logica Test Team to ensure that test phases carried out and managed solely by them will be carried out consistent with Good Industry Practice and in accordance with their own test procedures.
In general, software defects raised during one phase of testing should be cleared by the start of the next phase, although defects of low significance and low business impact may be left outstanding, rather than delay the start of the next phase, subject to agreement from ELEXON and to there being a plan of action in place for them to be addressed.
Logica will be required to log all defects during all Formal System Test phases using the BSC Services Helpdesk.
Code Subsidiary Documents will have been authorised by the relevant Panel Committee prior to implementation. Any defects identified against these products after the implementation date will be managed by issuing a derogation or notice to the users of the product and by raising a Change Proposal to effect the change. These products will have gone through an extensive review procedure prior to Committee approval, so the risk of issues is believed to be small.
At the end of the project, ELEXON will raise any defects that have not been resolved as Trading Arrangement Issues for further investigation.
4.2 Test Report
Test Reports will be produced to summarise the test results from each of the types of test carried out, to describe any defects found during testing and the steps taken to resolve them, and to specify any deviations from the relevant test strategies and test specifications.
• The ELEXON Release team will write a Witness Test Report and a Participant Test Report for the June 2009 Release; and
• The Logica Test Team will write a Test Report covering the outcome of testing to the CRA, CDCA and ECVAA, encompassing all test phases (including Participant Testing).
4.3 Test Phase and Software Acceptance
The Logica Test Team shall ensure that the CRA, CDCA and ECVAA changes meet the baseline requirements, defined in the reviewed and agreed changes to the CRA, CDCA and ECVAA System Specifications, and that they are consistent with the P215 Impact Assessments.
ELEXON will ensure overall that the requirements defined in the Logica documentation (Impact Assessments, Design Specifications, System Specifications) are captured and consistent with the Business Requirements Solution for P215 (Reference 5).
ELEXON will ensure overall that the scope of the testing supports the BRS document, and that the acceptance criteria that are subject to testing have been met.
The scope of the audit of testing carried out by Corporate Assurance will be comparable with previous audits including the review of Logica’s test evidence and internal reviews.
ELEXON Corporate Assurance will review the generic acceptance criteria specified by the Change Delivery Release Acceptance Criteria (Reference 11) and prepare a statement of compliance for the June 09 Release for approval by the BSC Systems Programme Board.
4.4 Timescales
The timescales and dates for the individual test phases will be defined in the PID and Release Plan for the June 09 Release (Reference 2).
There is an assumption for the implementation of P222 ‘EAC Data to Distributors Report’ all NHHDAs will have installed the version of software as required by the go-live date; Logica will release the NHHDA software changes to Participants by the end of February 2009 and thus will not be expected to deliver any software changes for the industry planned go live for P222 via the June 09 Release.
5.1.2 Risks
The following risk has been identified during the development of the Test Strategy. The risk below will be added to the project risk register and managed through that mechanism:
• There is a risk of an impact on resourcing from the BPO Transition on both ELEXON and Logica causing delays or compromising quality.
Mitigation: ELEXON and Logica working closely together to support each other to ensure quality and timely delivery of testing for the June 2009 Release.
• There is a risk for the P222 implementation that one or more NHHDAs will not have installed the software before 25 June 2009.
Mitigation: Closely monitor the situation via the Software Technical Assurance Group (STAG), the question should be raised with the STAG members at every meeting.
It should also be noted this is a high risk change, as credit cover is an important part of settlement. Testing needs to be thorough to mitigate any risk of a Defect that has an impact upon industry.
5.1.3 Issues
No issues have been identified during the development of this Test Strategy. Any issues identified subsequent to this will be added to the project issue register and managed through that mechanism.