Secure Remote Substation Access Interest Group kickoff meetingsmartgrid.epri.com/doc/Remote-Substation-Access-Interest-Group... · Secure Remote Substation Access Interest Group ...

Scott Sternfeld, Project Manager Smart Grid Substation & Cyber Security Research Labs

[email protected]

Utility co-chair: John Stewart, PM Grid ICT

[email protected]

Secure Remote Substation Access Interest Group Part 2: Development of Test Scenarios

July 17, 2013

mailto:[email protected]


2 © 2013 Electric Power Research Institute, Inc. All rights reserved.

EPRI Cyber Security and Privacy Team

Galen Rasche Technical Executive P183 Program Lead

650-353-0336 [email protected]

Glen Chason Project Manager


Scott Sternfeld Project Manager 843-619-0050

[email protected]

John McGuire Project Manager 865-218-8018

[email protected]

Annabelle Lee Senior Technical

Executive 202-293-6345 [email protected]

Eric Cardwell Sr. Project Manager











EPRI Cyber Security Collaboration

Trade Organizations

Vendors Policy / Regulators

Research Organizations

Standards Bodies

EPRI in collaboration

with utilities

Representing Utilities Through Coordination and Collaboration


Remote Access Interest Group Agenda

Remote Access System Overview

Development of Test Scenarios NERC CIP topics Schedule of next meetings


What is a Remote Substation (IED) Access System?


Assessment of Remote Access Solutions: 2012 Remote Substation Access Scenarios

1

2, 3

5 4


Development of new Remote Access Scenarios


Layout of Remote Access Scenarios

1. Problem statement (with background) 2. Ideal end state 3. Stated Assumptions 4. Possible limitations to solution 5. Related implementation issues

Prioritize scenarios for 2014 effort • Vendors to discuss, develop, or demonstrate


List of Remote Access Scenarios / Challenges

1. Integration with a “Password Vault” (Nathaniel)

2. Local (Substation) Access Issues: – Relay functional testing while maintaining CIP

compliance (Tim)

3. Vendor-IED interoperability/compatibility tests (Scott)

4. Password management: (John) A) IED password in IED configuration B) Password complexity tracking based on firmware

Output: Prioritize scenarios for 2014 effort • Vendors to discuss, develop, or demonstrate scenario

using a performance spec


List of Remote Access Scenarios / Challenges (future topics) Future: 1. Coordination with operations – coordination for alarms or safety issues

2. Alarm/Log generation (Incident Management)

– Standardization of syslog messages – Translate IED responses and RA user interactions into alarms.

3. Integration of “Smarter” IEDs with Remote Access system - multiple authentication points

4. Management of devices with web-based interfaces

5. Various architecture scenarios • Network connectivity to Corporate vs. SCADA • Local substation access – front panel access


Scenario 1: Integration with a “Password Vault” What is a “Password Vault”?

Benefits of a password management solution for shared/service/privileged account passwords: - Regular password changes improve security and compliance - Control which users have access to passwords - Allow detailed auditing of each use of these passwords

Some solutions offer advanced functionality including: - Password Changes - Password Verification - Password Reconciliation

Automatic password management of service accounts either through push or pull


Scenario 1: Integration with a “Password Vault”

1) Problem statement (with background): - Multiple password vaults, different security or logging levels

for various systems, makes difficult for auditing/compliance - Current IT Enterprise password vault products do not interact

with substation IEDs

2) Ideal end state - Single integrated password vault to manage all

shared/service account passwords. - Remote Access solution would connect to IEDs using

passwords stored in Password Vault solution 3) Stated Assumptions - Utility has or is considering both products - Remote Access and Password Vault solutions have network

connectivity between them


Scenario 1: Integration with a “Password Vault”

4) Possible limitations to solution - IEDs that have no connectivity to the remote access

system would have different process for password management.

- Password vault solution is offline (no redundancy) 5) Related implementation issues - Documentation, migration, testing and training for end

user. 6) Potential solutions - Integrate products via API


Scenario 2: Relay functional testing while maintaining CIP compliance 1) Problem statement: - Relay technicians use Doble or Manta test sets plugged into

technician’s laptop and require a simultaneous connection to the relay. Desire to log/record the actions by the relay tech to the remote access system. Remote access system is not currently included in this process.

2) Ideal end state - All interactions with relay are captured and correlated into

remote access system. - “Front panel access” is restricted 3) Stated Assumptions - Remote Access system is available for the intended relays


Scenario 2: Relay functional testing while maintaining CIP compliance

4) Possible limitations to solution - Remote Access system may not be conveniently located (Control house adjacent to location of nearest network

access point or remote access client) 5) Related implementation issues - May require extension of network presence to other

physical locations at substation 6) Potential solutions - 1000’ patch cable - Additional distributed remote access clients


Scenario 3: Remote Access Vendor IED interoperability / compatibility tests

RA system / IEDs

SEL 4xx relay

GE UR relay

ABB 670 Relay

Seimens 7SJxxx Relay

PQ meter

DFR Other

Crossbow ESNET Cooper SEL-3620 TDI Vendor X (example)

• Tests would be broken down into specific tasks such as: • Intelligent vs. Passive proxy • Ability to manage passwords • File/Event retrieval

• Could include gateway products (RTAC, SMP, 2020, D20, SSNET)

http://www.webdesign.org/photoshop/drawing-techniques/make-a-professional-logo.10543.html


Scenario 4A: Password Embedded in IED Config

1) Problem statement (with background): - Historically passwords have been included in configuration files

allowing for password changes through configuration downloads - Password must be inserted into files prior to download to prevent

password sync issues 2) Ideal end state - Password management system dynamically integrates correct

password into config files on the fly during download - Remove passwords from configs entirely and manage separately or

through trust mgmt infrastructure (IT) 3) Stated Assumptions - IED passwords aren’t known in advance to configuration engineer - Assumes utility has a variety of IED FW combinations that preclude

manual handling with unique processes


Scenario 4A: Password Embedded in IED Config

4) Possible limitations to solution - Config files are typically proprietary with limited toolsets

for parsing and modifying on the fly - Legacy systems will be managed with passwords

integrated in configs for a long period of time. 5) Related implementation issues - Integration issues between proprietary systems are the

most pronounced. 6) Potential solutions - Develop open interoperable config handling interface


Scenario 5B: Password Complexity Tracking

1) Problem statement (with background): - IED password complexity can change with firmware revisions

(even for same model)

2) Ideal end state - Automated solutions must correctly identify and incorporate

these changes within script variables - All vendors understand nuances of models and firmware of

vendor IEDs

3) Stated Assumptions - Existing Sources for this information exist - Vendor release notes which vary significantly in detail - Vendors would be willing to share this information


Scenario 5B: Password Complexity Tracking

4) Possible limitations to solution - Who takes ownership of this information? - Where would this master list reside? - This information has to be kept up to date - Funding: vendors or database upkeep

5) Related implementation issues

- Would require manual work by each utility to produce new scripts for firmware changes

- Changes in menu options, page layouts, command replies, etc can affect existing scripts.


IED Password Management Through remote substation access systems

Approach: • Identify the requirement, benefits and challenges associated with

implementing IED password management

Value: – Support CIP compliance and documentation for password change

requirements – Reduce risk of unauthorized access attempts by obscuring the password

• No more default or ‘utility standard’ passwords – Reduce the frequency of password updates – Reduce inefficiencies and costs through automated password changes.


Password Management – Poll!

Is your utility implementing randomized passwords for IEDs? 1) Already implementing 2) Pilot stage or planning stage 3) No current plans


CIP requirements:

“How do we be compliant without being intrusive to the

other maintenance activities??”


Security requirements relating to CIP

Selected Remote Access requirements including CIP references Speaker: Utility Business Analyst


Next steps relating to CIP?


EPRI’s Smart Grid Substation Lab Knoxville, TN

Product testing and demonstration site: Common environment for all vendors


Schedule of next meetings – Remote Access

Discussion topics. 1 hr session each. 2-3pm EDT. Each session should review and add test scenarios if appropriate.

• July 18th – Unique IEDs / Development of test scenarios (w/NERC CIP)

==============================================

• September 27th (1-2pm) [proposed] – Password Management

• October 31st [proposed] – NERC CIP requirements matrix

• November 21st [proposed] – System ownership / Review of 2014 plans


Key Take-Aways:

• Remote Substation Access products are still in their infancy.

• You are not the first ones to implement these products. – Some of your peers may be “super users”.

• We need to build and strengthen a community of users now!

– Users span multiple organizations

• Work together to share technical approaches to NERC CIP

• Developing unified utility requirements and test scenarios can improve the market offerings.


Together…Shaping the Future of Electricity


Legal Notices

Please observe these Antitrust Compliance Guidelines:

– Do not discuss pricing, production capacity, or cost information which is not publicly available; confidential market strategies or business plans; or other competitively sensitive information

– Be accurate, objective, and factual in any discussion of goods and services offered in the market by others.

– Do not agree with others to discriminate against or refuse to deal with a supplier; or to do business only on certain terms and conditions; or to divide markets, or allocate customers

– Do not try to influence or advise others on their business decisions and do not discuss yours except to the extent that they are already public


Deployment considerations


System Ownership • Who will assume overall ownership of the system?

– Single POC or group needed (champion) – Provides coordination between other users

• Many, many groups involved that can be considered “owners”: – Group that specifies and procures equipment? – Group that provides initial configuration? – Group that maintains equipment in the field? – IT: System upgrades, patching, disaster recovery, deployment to

users

• Other considerations to determine ownership: – Frequency of use: Who manages the configurations of the devices? – Volume: What is the quantity of IEDs to be managed? – Criticality: What is the criticality of these devices? – Availability: Is there 24/7 support from any of the organizations?


Architectures




Engineering Access and File Extraction


Remote Substation Access System

• What is it? – Provides for remote “engineering” (manual) access to all

substation devices (IEDs) in a secure fashion. – Optional: Integrated with (automated) file extraction as

part of an overall data integration solution. – Can be used as a replacement for a Windows Terminal

Server (jump host). – Can be used as a tool to aid in NERC CIP compliance. – May also include:

• Password management • Configuration (change) management for IEDs • Asset management


Cyber Security and Privacy 2012 Project: Assessment of Remote Access Solutions Purpose: Work with vendors and utilities to assess several products providing Interactive Remote Substation Access.

Approach: – Develop comprehensive list of requirements – Develop use cases/scenarios – Vendor deployment/development in Smart Grid

Substation Lab – Improved vendor products – Vendor final demonstrations

Presenting utility requirements with a ‘unified voice’


2012 project: Assessment of Remote Access Solutions

• Requirements workshops: – May 23rd and June 13th, 2012

• Product demonstration: – Oct 24-25th, 2012 Knoxville, TN – Wide range of audience – Technical update: Document ID: 1024424 - Dec 2012 “Substation Security and Remote Access Implementation Strategies”

Utility Value: – Awareness of available products – Common demonstration platform – Vendor products improved

Vendors and utility collaboration for accelerated technology transfer

http://membercenter.epri.com/abstracts/Pages/ProductAbstract.aspx?ProductId=000000000001024424

http://membercenter.epri.com/abstracts/Pages/ProductAbstract.aspx?ProductId=000000000001024424


EPRI’s Cyber Security Research Lab Knoxville, TN

Five vendors installed in the lab: • EnterpriseSERVER.NET by Subnet

Solutions

• CrossBow by Ruggedcom, a Siemens Business

• SEL-3620 by Schweitzer Engineering Labs

• ConsoleWorks by TDi Technologies

• IED Manager Suite (IMS) by Cooper Power Systems

Installation in a Common Demonstration Environment

Secure Remote Substation Access Interest Group kickoff meetingsmartgrid.epri.com/doc/Remote-Substation-Access-Interest-Group... · Secure Remote Substation Access Interest Group ...

Documents