Parallel Ops / Go-Live Checkpoint Weekly Update – Friday Meeting January 17, 2014
Jan 18, 2016
Parallel Ops / Go-Live Checkpoint
Weekly Update – Friday MeetingJanuary 17, 2014
Agenda• IDT Debrief 9:00A – 9:20A
• Updated IDT Schedule and Conditions• IDT Report Review: IDT 1/7 and IDT 1/10
• Analysis and Announcements Updates 9:20A – 9:40A• Multi-Day Tags • Preliminary Scenario Results
• Look-Ahead 9:40A – 10:00A• Upcoming IDTs and Scenarios • Weekly Updates: Release Scorecard, Migration Report, Known Issues Log
• Settlements Update 10:00A – 10:15A• Readiness Update 10:15A – 10:45A
2
IDT Schedule and Conditions
3
IDT Date IDT Times Conditions Notes
1/21/14 Originally: 3:30P-8:30P2hr IDT begins at 5:30P
Now: 7:30A-4:30P7hr IDT begins at 8:30A
Regulation Pre-selection
Tieline Drop
SPP loses select tie-line measurements. SPP will simulate the loss of real-time data with significant tie-line data providers. MPs will not need to take any action; this is an internal process for our operators.
EDR Testing with specific MPs will occur during this IDT
Turnaround Ramp Rate set to 1
1/27/14 Originally: 4:30P-10:30P4hr IDT begins at 5:30P
Now: 7:30A-4:30P7hr IDT begins at 8:30A
Tieline CheckoutADIRegulation Pre-selectionTurn Around Ramp Rate to Regular Submission
2/25/14 8:30A-12:30P2hr IDT begins at 9:30A
Regulation Pre-selection
2/28/14 7:00P4hr IDT begins at 8:00P
Cutover for March 1 Go-Live
Ginny Watson
IDT 01/07 ReportIDT reports posted to spp.org here 7 calendar days after each IDT
spp.org > Integrated Marketplace > Market Trials > Integrated Deployment Test
IDT 01/10 ReportIDT reports posted to spp.org here 7 calendar days after each IDT
spp.org > Integrated Marketplace > Market Trials > Integrated Deployment Test
4Bryan Wood
Announcements
5
January Friday Meetings:Friday January 24: No Regular Friday Meeting
IDT 1/27 Prep Call at 3P
Friday January 31: Regular Parallel Ops / Go Live Checkpoint Meeting at 9A
SPP will retest Mitigation Scenarios (16.1 - DA, 16.2 - RUC, and 16.3 - RTBM) on Jan 28
•Any MP who wants to see results of Mitigation needs to fail the behavior test with at least one unit. Units will be picked up in the re-run.
•If you are interested in participating in Scenario 16.3 – RTBM Mitigation, please submit the Resource(s) you would like to see Mitigated via RMS (Parallel Ops Quick Pick) no later than Monday, Jan 27 at 12P CPT.
Ginny Watson
Multi-Day Tags for Day Ahead Market • An MP requested that SPP allow Monthly/Multi Day Tags for Day
Ahead Market Bids/Offers
• The Market Protocols only allow Day Ahead bids/offers up to 7 days prior to OD
• MISO has a Multi-day Process
• SPP will request an impact assessment for system changes necessary to allow Multi-Day Tags up to 7 days prior
• Depending upon cost and member approval this change would be a post Go-live enhancement
6Shari Brown
Shortage & Excess Capacity Scenarios
7Jarrett Friddle
SPP retested Shortage & Excess Capacity Scenarios (5.1, 5.2, 6.1, 6.2) in lower environments
•Defects described on the following slide caused Scenarios to fail
•Fixes are due in R1.9.3– Presuming patch correctly fixes reported issues on time, SPP will
retest Scenarios in lower environments by January 24– Detailed results will be presented to MPs.
8
• Summary – Scenarios in Parallel Ops:– 4 Scenarios, 2 Partial Scenarios, Dispatch Simulator Test successful:
Scenario 7 Must Offer Scenario 28a Scenario 15 Commitment Mismatch Part 2 Scenario 24 Scenario 13 DDR w/ Settlements Scenario 29 RSG Scenario 31 BDR w/ Settlements – success pending Settlements Scenario 36 GFA
Dispatch Simulator Test– Note: Dispatch Simulator Test in process again from Jan 15-17
• Summary – Retests Pending:– 2 entire Scenario retests pending
Scenario 11 scheduled for Jan 21; Non-Functional Testing Scenario under analysis
– 8 individual retest requests pending
Scenario Tracker as of Jan 8
Ginny Watson
Scenario 31 BDR w/ Settlements
9
Results• On the Markets side, this Scenario passed• Market Participant data was correct.• Resource behaved as designed in RTBM and EMS.• Dispatch instructions were observed via XML and Markets UI.
• Settlements • KIL #899, which will go into Production later this month, directly impacts the
charge types associated with BDRs and DRRs• Settlements is currently testing the resolution for KIL #899 and intend to have it
in Production for the Final Settlement
Jarrett Friddle
Non-Functional Testing
10Casey Cathey
Results page 1 of 2• Planned January 14th 10:00P – 12:00A• Extended 30 minutes to 00:30 so SPP could research RTBM solutions that were produced• Goal of Exercise was to test systems and identify areas that might need additional
attention in a failover event. Follow-up analysis and collaboration with vendors are underway to identify, test, and validate solutions. SPP’s planning team will recommend and communicate the next steps including timing and scope to MPs following analysis.
• Summary of Expected v. Actual Results • Other than exceptions noted, all systems functioned as expected.
Expected Result: EMS automatically fails over is fully functioning with RTGEN for CBA Actual Result: EMS failed over after manual intervention; once the
failover was completed, EMS functioned as expected
Expected Result: ICCP is fully functioning and sending setpointsActual Result: ICCP failed over correctly, functioned as expected
Expect Result: Marketplace Portal User Interface and API available for data entry and retrievalActual Result: Marketplace Portal and associated system UIs/APIs failed over correctly. Two external users experienced connectivity issues with the Markets UI. Users were able to reboot and connect. SPP is researching reported errors for additional information to provide to vendors as needed.
Non-Functional Testing
11Casey Cathey
Results page 1 of 2• Summary of Expected v. Actual Results • Other than exceptions noted, all systems functioned as expected.
Expect Result: Web Services (RTO_SS) availableActual Result: Remained available, as expected
Expected Result: Market System is fully functioning and solving RTBM for approx. 1 hrActual Result: RTBM missed intervals between 22:20, 22:25 and 22:30 while the failover was occurring. The first study solved at SPP’s backup facility at 22:35. SPP resources were onsite and continued to research the Market Systems during the exercise. Additional studies missed during the exercise were 23:10, 23:30, 23:35, and 23:50. SPP is working with the vendor to research and resolve this issue.
Expected Result: Email is available for sending and receiving communicationActual Result: Email was unavailable from the backup facility servers and was reverted to SPP’s primary facility while Markets trouble shooting occurred. Microsoft is on-site to assist in identifying solutions.
After the exercise was terminated, all systems and applications restored successfully to Chenal operations.
36.0 – GFA Carve Out SchedulesResults
•Preliminary Pass– MPs submitted BSS headers for all but one of the GFA Carve Outs
– No Import Interchange Transactions for GFA Carve Outs were submitted Only applies to 4 GFAs
MPs did not submit hourly BSS profiles for these import transactions
– SPP has enough data to test the GFA Carve Out process, and appropriate data to successfully test the settlements uplift distribution
– MPs should start seeing GFA Carve Out uplift charges on or around 1/22
12Jarrett Friddle
Look-Ahead: Jan 19 – Jan 31
13
Parallel Ops Calendar on spp.orgspp.org > Integrated Marketplace > Market Trials > Parallel Operations Test> Integrated Marketplace Parallel Operations Project Documents
• Settlements details on the Settlements Calendar marketplace.spp.org > Public > Settlements > CalendarsJan 19 – Jan 25 Schedule:Ramp Sharing Off Mon Jan 20: IDT Prep Call at 3PTues Jan 21: IDT Call 7:30A – 4:30P, 7hr IDT begins at 8:30A
Run Scenario #11 EDRWed Jan 22: IDT Debrief Call at 3P
CWGThurs Jan 23: CWGFri Jan 24: No regular P Ops Meeting. IDT Jan 27 Prep Call at 3P
Jan 26 – Jan 31 Schedule:Ramp Sharing OnMon Jan 27: IDT Call7:30A – 4:30P, 7hr IDT begins at 8:30ATues Jan 28: IDT Debrief at 3P and 16.1 / 16.2 ScenarioWed Jan 29: 16.3 Mitigation ScenarioFri Jan 31: Regular Friday Meeting at 9A, Parallel Ops Ends
Ginny Watson
Look Ahead Reminders• 1/21 IDT Call starts at 7:30 AM, 7 hour IDT begins at 8:30 AM
– Tieline Drop– Regulation Pre-Selection– EDR Scenario: Applicable MPs are in contact with SPP
This scenario will demonstrate the clearing and dispatch of an EDR resource for all qualified products.
• Please submit Spin/Supp as ‘Market’
• 1/27 IDT Call starts at 7:30 AM, 7 hour IDT begins at 8:30 AM– Tieline Checkout– Regulation Pre-Selection– ADI
• Please submit Spin/Supp as ‘Market’
14Bryan Wood / Jarrett Friddle
Migration and Maintenance Report
15
Migration and Maintenance Report posted to spp.orgspp.org > Integrated Marketplace > Market Trials > Parallel Operations Test> Integrated Marketplace Parallel Operations Project Documents
Wendy Reynolds
16
Release Scorecard posted to spp.org:spp.org > Integrated Marketplace > Market Trials > Parallel Operations Test> Integrated Marketplace Parallel Operations Project Documents
Release Scorecard
Ginny Watson
Known Issues Log
17
• Please review the Market Trials Known Issues Log on spp.orgspp.org > Integrated Marketplace > Market Trials > Structured/Unstructured Test > Integrated Marketplace Market Trials Project Documentation Folder
Shari Brown
Market Trials Useful ResourcesIntegrated Marketplace Parallel Operations Project Documents spp.org > Integrated Marketplace > Market Trials > Parallel Operations Test> Integrated Marketplace Parallel Operations Project Documents
•Parallel Ops Presentations, Statistics, MP Meetings Folder•Parallel Ops Reference Documents Folder•Scenarios in Parallel Ops Folder > Scenarios Schedule•Parallel Ops Communication Plan•Parallel Ops MP Guide•Parallel Ops Calendar•Known Issues Log•Release Scorecard•Migration Report
Integrated Deployment Test (as part of Parallel Ops)spp.org > Integrated Marketplace > Market Trials > Integrated Deployment Test
•IDT MP Guide•Parallel Ops Calendar
Additional documents:•Settlements Calendar marketplace.spp.org > Public > Settlements > Calendars•Change Tracker spp.org > Integrated Marketplace > Marketplace Documents•Integrated Marketplace Structured/Unstructured Test Project Documents spp.org > Integrated Marketplace > Market Trials > Structured / Unstructured Test> Integrated Marketplace Structured / Unstructured Test Project Documents
18
SETTLEMENTS UPDATEBrent Wilcox
Brent Wilcox 19
Settlements Update - January • The December release was moved into production on Jan 8th as
scheduled. – Some days in early January were posted late due to the timing of the
migration of this release.
– A couple of defects were in the December release. A hot fix from Alstom is scheduled to be applied on 1/16.
• All January Operating Days will be processed using the December release. – Some Jan. initials were processed prior to the hot fix being installed
but all Jan. finals processed with the hot fix in place.
20Brent Wilcox
Settlements Update - February • Last statements (1/31 I) will be posted Feb 3rd.
– Please see highlighted section of February calendar for details
• Statements will remain on portal until noon on Feb 5th
• Settlement system cutover begins the afternoon of Feb 5th.
21Brent Wilcox
Settlements Update – February Cont’d • Market Participants can start to submit their Bilateral Settlement
schedules on Feb 22nd. – Thirty day window will not be followed in this instance.
– Any Bilateral Settlement schedules submitted prior to February 22nd will be deleted as a part of the system cutover
• Meter windows will open on OD+1 (3/2/14).
22Brent Wilcox
OFFER DATA ACKNOWLEDGEMENT
23
• Markets Test Data vs. Production Level Data– Market Offer data will NOT be purged from the Markets Systems prior to 3/1
Includes existing offer data submitted since June 2013 Market Trials start -- even MP offer data that was seeded by SPP for testing purposes
– Market Participant test offer data MAY become financially binding on 3/1 and beyond if preventative measures aren’t taken before 2/22
Update Market Offer Data
24Erin Jester
Update Market Offer Data• MP Required Action for Updating Market Offer Data1. Complete Acknowledgement Letter by 2/14
Market Trials Liaisons received RMS ticket on 1/17 requesting acknowledgement of responsibility to update Market offer data and understanding of risks associated with having non-production ready offer data in the Market Systems after 2/22
Acknowledgement to be completed via RMS no later than 2/14
Escalation will occur for MPs not completing acknowledgement by 2/14
– Update all Market Offer Data in the Markets System by 2/22 To avoid test data becoming financially binding, update offer and parameter
data for DA Resource Offers, RTBM Resource Offers and Mitigated Offers
Market offer data MUST be updated prior to 2/22 to ensure good, production-ready data for Go-Live
25Erin Jester
MP GO-LIVE VALIDATION CHECKLIST
26
• Go-Live Validation Checklist posted to Readiness workstream folder on 1/17
• Tool to identify applicable tests and perform internal, independent validations of systematic and functional readiness to cutover to Go-Live
• Tool should be used now and after the February Maintenance Window (2/22), prior to Go-Live (3/1)
• Go-Live Checklist is not to be returned to SPP. Only for MP use in confirming internal readiness.
• Any issues discovered while validating readiness should be addressed immediately via RMS
MP Go-Live Validation Checklist
27Erin Jester
• Populate the following table based on MP & Asset Types
• Based on your selections, applicable tests will be identified that you will want to verify internally prior to Go-Live.
MP Go-Live Validation Checklist
28Erin Jester
READINESS UPDATEErin Jester
29Erin Jester
General Readiness Updates
• Go-Live Approach (similar to TCR) will be posted the week of January 20 to the Readiness folder– Will include the MP Go-Live Validation Checklist
• iDashboard will be updated today (January 17) per the established schedule– Total Number of Readiness Metrics: 50
Number Complete (Blue): 36
Number on Track (Green): 7
Number at Risk (Yellow/Red): 6
Number Not Started (Gray): 1
30Erin Jester
Parallel Ops Exit Criteria Status• POX-01 (Scenario Completion) changed from yellow to green for the first
two weeks of January– Scenario completion will continue to be tracked
• There are three “at risk” Parallel Ops criteria:
31
ID Metric Name Status Variance
POX-20 OPS-15 CROW Outage Submission Yellow
12/28-1/11: 15/15 of resource-owning MPs who have reported outages have not reported them through SCADA and CROW within 30 minutes.
POX-21 OPS-17 ICCP Model Validation Yellow 1/41 of MPs ICCP models are not validated, but is expected
to be in service shortly.
POX-22 OPS-18 ICCP Control Status Tracking Yellow
12/15-1/4: 26% of MPs submitted control modes correctly 100% of the time.
SPP is adjusting how OPS-18 is reported, and metrics are expected to improve once these changes are made. See ICCP Control Status slides)
Erin Jester
OPS- 15: At Risk Readiness Metric
32
Metric ID Metric Title Entity Status
OPS-15 CROW Outage Submission MP Red• 4.23% of the total outages reported by resource-owning MPs
were reported within 30 minutes.
• 1.41% of the total outages reported by resource-owning MPs were reported between 30 minutes to 1 hour.
• 94.37% of the total outages reported by resource-owning MPs were reported beyond 1 hour.
Data as of January 11, 2014*
*Based on data from January 5-11, 2014
Erin Jester
CROW Topics for Discussion at CWG
• CROW Outage Submission Benefits– Avoid penalties
– Avoid commitments that are not physically possible
• CROW Outage Submission Impacts– Examples of what happens within the IM when incorrect
outages exist.
• CROW Outage Submission Penalties– Discuss the RUC Make-Whole-Payment Distribution Amount
penalties for DA and RUC
33Erin Jester
CROW Hot Topic Calls
• The intent is to minimize SPP’s potential for incorrect commitments.
• Help the members meet the Criteria requires.
(Entry of an outage to CROW within 30 minutes of the actual outage)
• CROW Hot Topic Net Conferences are scheduled for9:30 AM – 10:30 AM CST on 1/28/2014
3:00 PM – 4:00 PM CST on 1/30/2014
Register via the SPP Learning Center
34Erin Jester
CROW Recommendation
• Resource owning Market Participants should attempt to meet the OPS-15 metric prior to the end of Parallel Operations on 1/31. • Provides confidence that starting the Marketplace 3/1 with
outage communication coordination will not adversely impact economics and reliability due to outage mismatches in CROW versus Real-time situation.
• Resource-owning MPs may continue and are encouraged to practice meeting OPS-15 throughout the February Extended Maintenance Window
35Erin Jester
Update on Attachment AH
• Received all but 3, and a couple additional require updates:
36
RMS MP Status
6147 APX No response after initial contact
6173 Calpine Contact unavailable this week; will follow-up
6201 ETEC Expected this week
6227 Golden Spread Update required
6246 TEA Update required
Erin Jester