Integration Update 1 External Interfaces & Integration July 24 2007 Stephen Kerr.

1 Integration Update

External Interfaces & Integration

July 24 2007

Stephen Kerr

2 Integration Update

Agenda

• Updates to External Interface Specification– New Services– Sandbox & EDS deployment– Notifications & Listeners & Security

• Web Service testing

• Integration Design Process– Artifacts– Process– Exceptions

3 Integration Update

Updates to External Interface Specification

• Certain elements of the External Interface Specification were approved on May 21st on condition that regular updates were provided to the specification

• Revision 1.02 was posted on July 20th for review– Review feedback required by August 3rd – Updated document will be republished by August 9th – TPTF vote on August 14th

• The changes in version 1.02 are– Removed deleteOutputSchedules from ThreePartOffer– Added statement related to ordering guarantees in section 2.6.– Added note regarding the handling of timestamps that are not interval ‘aligned’ in

section 2.3.2– Corrected figure 10 to reflect use of ‘INC’ or ‘DEC’ type in mRID for IncDecOffers– Corrected Figure 9 to not have expiration column– Modified Figure 10 to add column to map protocol terms to XML tags– Modified sections 3.3.1, 3.3.3, 3.3.4, 3.3.5 and 3.3.11 to not use expirationTime

as a key value– Corrected references to ‘IrregularIntervalSchedule’ to be ‘TmSchedule’– Revised section 5.1 and Figure 109 to reflect revised notification scheme– Revised ASOffer structure, uses a variation of PriceCurve that provides price

points for each specific AS type. This affects sections 3.3.4, 2.3.1, 2.3.3 and 2.3.4

4 Integration Update

New services being built

• The services we are building for Aug 30th release as loopback services are listed

• There will be no back-end functionality associated with these services in the Sandbox

• The back-end systems have not implemented these services yet and are subject to change

• The Market Information types will be simulated but we have not yet decided what the response will be– Simple document or XML response

• The services will be deployed to the EDS environment on the transition plan schedule

• New Market Transaction Services– Self Arranged Ancillary Services– Ancillary Services Offer– Ancillary Services Trade– DAM Energy Bid– DAM Energy Only Offer– Congestion Revenue Rights Offer– PTP Obligation Bid– Self Schedule

• New Market Information Services– Proxy Curves– Startup and Shutdown Instructions– Total Regulation– Load Ratio Share– Unit Availability– Forecasted Load– Real-Time System Load– Market LMPs and SPPs– Mitigated Curves

• New Notifications– Outage Notifications– Notices and Alerts– CRR Awards

• New Outage Scheduling Services– Outage Creation

5 Integration Update

Sandbox & EDS deployment

• Following services are available– Three Part Offer– Incremental/Decremental Offers– Current Operating Plan– Output Schedule– Bid Set Acceptance– Bid Set Errors– Get mRID– System Status

• Defects are being posted here:– http://nodal.ercot.com/readiness/sandbox/documents/docs/Sandbox_Defects.xls– Problem with optional fields being made mandatory

• Notifications system is functional but network configuration has to be reworked to get notifications through the DMZ

• Back-end integration with MMS enters test in early August for early EDS3– Three Part Offer– Incremental/ Decremental Offers– Current Operating Plan– Output Schedule

• Pending sent through the general Notification service NOT– Bid Set Acceptance– Bid Set Errors

6 Integration Update

Notifications & Listeners

Proxy Server

Request Broker

Notification Service

Event Source

Target Application

MP Application

HTTPS - Notify

HTTPS - Acknowledge

MP Service

HTTPS - Request

HTTPS - Reply

HTTPS

HTTPS

Event

Back-end systems will be available to stimulate notifications toward end

07/early 08.Until then notifications will be

simulated

Network design is in progress around the

outbound and inbound traffic through DMZ

7 Integration Update

Solution Use Case

The following sequence of steps would be used to send a notification message to an MP system:

1. ERCOT Notification Service establishes an HTTPS connection to target MP system at port 443 of URL pre-specified by MP

2. Notify message is signed by ERCOT using the ERCOT server certificate (the same one used for external web services)

3. MP web service is invoked by ERCOT Notification Service, sending the Notify message

4. MP system validates the signature, using the ERCOT public key5. Notify message is consumed by the MP system6. MP system replies with a simple, unsigned Acknowledge

message using the same HTTPS connection7. ERCOT Notification Service receives Acknowledge message and

marks the transmission as complete8. Process is repeated for next notification or if a retry is needed

according to established rules

8 Integration Update 8

Solution Approach

• MPs can use any server certificate of their choosing for HTTPS connections to their web server

• ERCOT will not use a client certificate when establishing an HTTPS connection to a MP notification listener service

• There is no need to manage additional client certificates over those needed for the external web services, greatly simplifying ERCOT implementation and administration

• MPs must be able to read ERCOT’s signature to verify that ERCOT sent the Notification

• MPs can not use Certificate Revocation Lists (CRLs), as ERCOT would not have a client certificate for each MP system

• Implementation is easier for MPs, as they don’t need to sign the Acknowledge messages (there is only non-sensitive data on the Acknowledge)

9 Integration Update 9

Steps to test Notifications & Listeners

• MP’s must get a new Nodal Test Certificate from ERCOT. The detailed instructions for this request will be communicated through the EDS meetings

• MP’s must get ERCOT’s public certificate from their Client Services representative in order to read ERCOT signatures

• MP’s must get the WSDL & XSD from ERCOT, published on http://nodal.ercot.com/readiness/sandbox/websvc/index.html

• MP’s are responsible for constructing the Notification listener service, using whatever technologies suits

• MP’s must notify ERCOT of the listener addresses (primary & secondary) and email details (via [email protected])

• MP’s must set up their infrastructure to accept an https connection from ERCOT

• MP’s can schedule time with ERCOT to test out the service – we will provide 1:1 time with developers (via [email protected])

• ERCOT verifies the service by sending a test Notification to the MP’s service

10 Integration Update

Testing for EWS Loopback release

• Testing was done a three levels

– Development• Developers executed 19 formal unit tests, along with a large number of informal tests

• Issue defects were fixed

– iFAT• iFAT test team executed 133 formal test scripts

– Common services (audit and error handling) 5 tests

– External clients 11 tests

– Functional 30 tests

– Notifications 10 tests

– Security 5 tests

– XML validation 72 tests

• Identified 36 issues or defects

• All issues were resolved

– Smoke test of Sandbox• iFAT test team executed 14 formal test scripts

• Found that notifications fail because of the setup of the sandbox environment

11 Integration Update

Sample test case

Step Name

Description Expected

Step 1 Pick the attached sample 3PO valid xml Sample xml is successfully picked.

Step 2 set the following Changes are reflected.

verb = change

Set FipFop value to 850 or make any other changes to payload/bidset

Step 3 Place this xml in the folder used by test harness client Xml is placed in the folder.

Step 4 A request is to be sent and a descriptive message is expected along with Ok Ok is expected.

Step 5 Do a "Get" request depending on the mrID from the step 4. A response reflecting the changes should be displayed

Step 6 Validate that change is reflected in the response message. Change should be visible.

Step 7 Once the test is complete set the xml back to its sample state. Xml is reset.

12 Integration Update

Interface Design Process

• Identify Integrations– System of Systems Architecture (SoSA) describes the enterprise level integration points

needed for the overall Nodal solution.– Integration Design Authority (Technical & Business Architects), project teams and

integration vendors work together to produce artifacts

• Project Teams are accountable for determining what information they need – Project teams (CRR, MMS, CMM etc) create CSDs & Use Cases that indicate dependencies

on information from other systems– Project teams determine what information is necessary and whether they can be supplied– Integration picks up from that point

• Interface Specifications & Design– Integration artifacts are produced that contain:

• Data Element Name, Element Attributes, Volumetrics, Data Frequency etc.• Business Triggering Events that drive the movement of information

– The integration teams use these artifacts to analyze the interface and develop Use Cases• Identify gaps, integration characteristics, mismatches and transformations of data that is necessary between systems

– Integrations specifications and Use Cases are the input to design and implementation of the integration between systems.

• Exceptions are handled through IDA and Design ClearingHouse– Cases where projects have made conflicting assumptions – Cases where integration discovers missing or incorrect information elements that do not

match

13 Integration Update

The integration factory constructs interfaces

Interface Build Plan

Interface Specification

Interface Design

Interface Build

FAT & ITESTInterface

teamassigned

Interface specs agreed

Interface design agreed

Interface conforms with spec

Inputs

Activities

EDS ScheduleEDS DefinitionsProject SchedulesInterface Priorities

Seek project commitments

Ramp up design and build teams

Produce iteration plan for the interface

Outline the information flows in the interface

Information flows in the interface are enumerated

Interface team resolves discrepancies in interface definition between source, target & system

Define behavior and information contained in the interface

SoSA – use cases & domainProject CSDs & UCsInterface docs from source and target systems

Integration design patternsExisting designsSource & target designs (CSD & Detailed Design)

Numerous build inputs incl. Test Cases, Environment config, makefiles, …

Integration pattern selected

Impacts on source and target assessed and incorporated

Collaborate with Source & target systems teams

Spec stubs and data

Interface tests defined

Updated design documentation

Updated test and dev environments

Updated source and libraries

Stubs built

14 Integration Update

Artifacts from the integration

Registration Interface Spec

Sample artifacts that are in progress

S&B - CMM Spec

Designs are based on our pattern library:

15 Integration Update

Artifacts from the integration

Designs are based on our pattern library:

JMS Consumer Pattern