Enabling a Cyber-Resilient and Secure Energy ...

EnablingaCyber-ResilientandSecureEnergyInfrastructurewithSoftware-DefinedNetworking

Dong(Kevin)JinDepartmentofComputerScienceIllinoisInstituteofTechnology

SoS Lablet/R2MonthlyMeeting,Jan20171

PartoftheSoS Lablet with

2

• DavidNicol

• BillSanders

• MatthewCaesar

• BrightenGodfrey

ProjectProgress

3

Publicationsinthecurrentquarter(Oct– Dec2016)• Jiaqi YanandDongJin.“ALightweightContainer-based

VirtualTimeSystemforSoftware-definedNetworkEmulation,” JournalofSimulation,November2016

• XinLiuandDongJin.“ConVenus:CongestionVerificationofNetworkUpdatesinSoftware-definedNetworks.”WinterSimulationConference (WSC),December 2016

• Ning Liu,AdnanHaider,DongJinandXian-HeSun.“AModelingandSimulationofExtreme-ScaleFat-TreeNetworksforHPCSystemsandDataCenters,”ACMTransactionsonModelingandComputerSimulation(TOMACS),December2016

ProjectProgress

4

Papersubmittedinthecurrentquarter(Oct– Dec2016)• DongJin,Zhiyi Li,ChristopherHannon,ChenChen,Jianhui

Wang,MohammadShahidehpour, Cheol WonLeeandJongCheol Moon.“TowardsaResilientandSecureMicrogridUsingSoftware-DefinedNetworking,” IEEETransactionsonSmartGrid,SpecialsectiononSmartGridCyber-PhysicalSecurity(Secondroundreview)

• ChristopherHannon, Jiaqi Yan,DongJin,ChenChen,andJianhui Wang.“CombiningSimulationandEmulationSystemsforSmartGridPlanningandEvaluation,” ACMTransactionsonModelingandComputerSimulation(TOMACS)

• ChristopherHannon,DongJin,ChenChen,andJianhui Wang,“UltimateForwardingResilienceinOpenFlow Networks,”ACMSIGCOMMSymposiumonSDNResearch2016

IndustrialControlSystems(ICS)

5

• Controlmanycriticalinfrastructures– e.g.,powergrids,gasandoildistributionnetworks,wastewatertreatment, transportationsystems…

• ModernICSesincreasinglyadoptInternettechnologytoboostcontrolefficiency,e.g.,smartgrid

NextGenerationofPowerGrid

LOADS SITESDISTRIBUTIONTRANSFORMER

DISTRIBUTIONSUBSTATION TRANSMISSION GENERATION

MoreEfficientorMoreVulnerable?

6 Picturesource:NISTFrameworkandRoadmapforSmartGridInteroperabilityStandards

DistributionOpsTransmission

Ops

Operations ServiceProviders

BulkGeneration Distribution Customer

MarketsRTO/ISOOps

DMS AssetMgmt

Enterprise

Bus

EMS

RTOSCADA

EMSWAMS

MDMSDemandResponse

Retailer/Wholesaler

Transmission

ISO/RTOParticipation

Aggregator

EnergyMarketClearing hosue

MarketServicesInterface

PlantControlSystem

Generators SubstationDevice

FieldDevice

DistributedGeneration

Utility

Provider

Third-Party

Provider

CIS

Billing

Home/BuildingManager

Aggregator

ElectricVehicle

DistributedGeneration

ElectricStorage

Appliances

ThermostatCustomerEMS

CustomerEquipment

Meter

Others

CIS

Billing

RetailEnergyProvider

Premises

Networks

EnergyServicesInterface

MeteringSystem

DistributionSCADA

Enterprise

Bus

TransmissionSCADA

Enterprise

Bus

WideArea

Network

Substation

LANs

Internet/

e-business

FieldArea

NetworksDataCollector

SubstationController

ElectricStorage

Internet/

e-business

Communication Path Network

CyberThreatsinPowerGrids

7

Picturesource: 1.NationalCybersecurityandCommunicationsIntegrationCenter(NCCIC).ICS-CERTMonitorSep2014– Feb20152.http://dailysignal.com/2016/01/13/ukraine-goes-dark-russia-attributed-hackers-take-down-power-grid/

• 245 incidents,reportedbyICS-CERT

• 32% inenergysector

UkrainePowerGridCyberAttack• 80,000 residentsin

westernUkraine• 6 hours,134MW

powerlostinDec2015

ProtectionofIndustrialControlSystems

8

• Commercialof-the-shelfproducts– e.g.,firewalls,antivirussoftware– fine-grainedprotectionatsingledeviceonly

• Howtochecksystem-wide requirements– Securitypolicy(e.g.,accesscontrol)– Performancerequirement(e.g.,end-to-enddelay)

• Howtosafelyincorporateexistingnetworkingtechnologiesincontrolsysteminfrastructures?

ProblemStatement

9

• MinimizethegapswithanSDN-enabledcommunicationarchitectureforICS

• CreateinnovativeSDN-awareapplicationsforICSsecurityandresiliency– Real-timenetworkverification– Self-healingnetworkmanagement– Context-awareintrusiondetection–Manymore...

ICS– industrialcontrolsystemSDN– software-definednetworking

SDNArchitecture

Net3

Net4

Net5

Net6

Net1 Net2

OpenFlow ProtocolDataPlane

ControlPlane

Applications QoS AccessControl VPN

OpenFlow Controller

OpenFlowSwitches

10

CyberResources

SCADAServers

FieldDevices

CommunicationNetworks Routing

PowerControlApplications

DemandResponse

FrequencyControl

StateEstimation

TopologyControl

…

…

• Instability• LossofLoad• SynchronizationFailure• Contingency• LossofEconomics

Impact

DenialofService

FalseDataInjection Malware Insider

Attack…

CyberAttacks

CurrentPowerGrid:PotentialCyberAttacksandTheirImplications

FutureSDN-enabledPowerGrid:ACyber-Attack-Resilient Platform

11

AnSDN-EnabledPowerGrid

PowerGridComponentLayer

PowerNetworkLayer

CommunicationNetworkLayer

SDNControlLayer

ApplicationLayer GridApplication

ControlManagemen

tMonitoring

Commun

icatio

nSystems

PowergridSy

stem

s

SDNApplication

IDSVerification

Self-healingNetwork

SolarPV

Gas Generator

Charging Station

Wind Turbine

ComEd

ComEdPershing Substation

(12.47 kV)FiskSubstation(12.47 kV)

TransitiontoanSDN-EnabledIITMicrogrid• Real-timereconfigurationofpowerdistributionassets• Real-timeislandingofcriticalloads• Real-timeoptimizationofpowersupplyresources

12

ControlCenter

ExistingMasterController

SDNMasterController

SDNApplications

GridApplicationsLocalSDNController1

PMU

LocalSDNController2BuildingControl

LocalSDNControllern

…

CommunicationNetworks

SolarPV

Gas Generator

Charging Station

Wind Turbine

ComEd

ComEdPershing Substation

(12.47 kV)FiskSubstation(12.47 kV)

13

TransitiontoanSDN-EnabledMicrogrid

• SDN-basedApplications– Real-timeVerification– Self-healingPMU

• HybridTestbed– SDNemulation+PowerDistributionSystemSimulation

14

Application1:NetworkVerification– Motivation

15

• Unauthorizedaccess• Unavailablecriticalservices• Systemperformancedrop• Instability• Lossofload• SynchronizationFailure

• …

89% ofoperatorsneversurethatconfig changesarebug-free1

82% concernedthatchangeswouldcauseproblemswithexistingfunctionality1

1. Surveyofnetworkoperators:[Kim,Reich,Gupta,Shahbaz,Feamster,Clark,USENIXNSDI2015]2. PicturesborrowedfromVeriFlowslides[Khurshid,Zou,Zhou,Caesar,GodfreyNSDI2013]

VerificationSystemDesign

16

ICSApplicationModels

NetworkModels

PolicyEngine

topologynetwork-layer states

(e.g.,forwarding tables)

Diagnosis

• Vulnerabilities• Errors

System Framework

DynamicModelUpdate Verification

DynamicNetworkData(topology,forwardingtables…)DynamicApplicationData(controlupdates…)User-specified Policy(security,performance…)

VerifiedSystemUpdates

VeriFlow

New rules

VeriFlow Operation

4/3/2013 Department of Computer Science, UIUC 11

Network Controller

Generate equivalence

classes

Generate forwarding

graphsRun queries

Diagnosis report• Type of invariant

violation• Affected set of

packets

Rules violating network invariant(s)

Good rules

Network-LayerVerification

17

PriorWork• FlowChecker

[Al-Shaer etal.,SafeConfig2010]• HeaderSpaceAnalysis

[Kazemian etal.,NSDI2012]• Anteater

[Maietal.,SIGCOMM2011]• VeriFlow

[Khurshid etal.,NSDI2012]

PicturesborrowedfromVeriFlowslides[Khurshid,Zou,Zhou,Caesar,GodfreyNSDI2013]

18

Switch'A' Switch'B'

Controller'

rule%1%

Challenges— TimingUncertaintyOldconfig:SwitchA=>SwitchBNewconfig:SwitchB=>SwitchA

19

Switch'A' Switch'B'

Controller'

Remove&rule&1& Install'rule'2'

rule%2%

Challenges— TimingUncertaintyOldconfig:SwitchA=>SwitchBNewconfig:SwitchB=>SwitchA

(1) (2)

Packet'

Challenges— TimingUncertainty

20

Switch'A' Switch'B'

Controller'

Install'rule'2'

rule%1%

rule%2%

Remove&rule&1&(delayed)&

Loop-freedomViolation

Uncertainty-awareModeling• Naively,representeverypossiblenetworkstateO(2n)• Uncertaingraph:representallpossiblecombinations

21

Updatesynthesisviaverification

Enforcingdynamiccorrectnesswithheuristicallymaximizedparallelism

22

AshouldreachB

2 1 3 4

OK,but…

23

Canthesystem“deadlock”?• Provedclassesofnetworksthatneverdeadlock• Experimentallyrareinpractice!• Lastresort:heavyweight“fallback”likeconsistentupdates[Reitblatt etal,SIGCOMM2012]

Isitfast?

0

5000

10000

15000

20000

25000

0 2 4 6 8 10 12 14 16

25000$

20000$

15000$

10000$

5000$

0$7/22/2014$22:00:00$

7/22/2014$23:00:00$

7/23/2014$0:00:00$

7/23/2014$1:00:00$

//$

//$

//$

//$

//$

//$

Time$

Num

ber$o

f$Rules$

in$th

e$Network$

7/22/2014$22:00:02$

7/22/2014$23:00:02$

7/23/2014$0:00:02$

7/23/2014$1:00:02$

0

5000

10000

15000

20000

25000

0 2 4 6 8 10 12 14 16

Immediate UpdateGCC

Consistent Updates 0

5000

10000

15000

20000

25000

0 2 4 6 8 10 12 14 16

Immediate UpdateGCC

Consistent UpdatesEndEndEnd

Comple?on$Time$} CCG

0

5000

10000

15000

20000

25000

0 2 4 6 8 10 12 14 16

Immediate UpdateGCC

Consistent UpdatesEndEndEnd

0

5000

10000

15000

20000

25000

0 2 4 6 8 10 12 14 16

Immediate UpdateGCC

Consistent UpdatesEndEndEnd

SlideborrowedfromBrightenGodfrey,TSSSeminar,Sep2015

Application2:Self-HealingPhasor MeasurementUnit(PMU)Networks

24

AffectedPMUs

NewpathsforaffectedPMUs

• Isolatecompromiseddevices• “Self-heal”thenetworkbyquicklyre-establishingroutes

– Torestorepowersystemobservability– Usinganintegerlinearprogrammodel

Self-HealingPhasorMeasurementUnit(PMU)Networks

25Self-healingSchemeonPMUNetworkforIEEE30-busSystem

VideoDemo

AHybridTestingPlatform

26

PowerDistributionSystemSimulation+SDN-basedNetworkEmulation

AHybridTestingPlatform

27

• Challenges– Temporalfidelityinnetworkemulation– Synchronizationbetweentwosub-systems• Emulation– executing“native”softwaretoproducebehaviorinwall-clocktime• Simulation– executingmodelsoftwaretoproducebehaviorinvirtualtime

IntegrationEmulation&Simulation

Issue:TemporalFidelityinemulationordinaryemulatorsembeddedinreal-time,butsimulatorsspeakinvirtualtime

VM VM VM VM VM

SystemTime

t=100

PhysicalMachine

VM- VirtualMachineTimeSlice– SystemExecutionUnit

e.g.,TimeSlice=100μs

EmulationSystem

30

t=200 t=300 t=400 t=500

Simulator

IntegrationEmulation&Simulation

Time

Supposethemediumissharedaccess…Supposethepacketsalljointhesamequeue….

Wrongbehaviorsduetotheemulator’sserializationofthetime

35

Ourapproach:VirtualTimeinEmulation

36

Whentheemulatorisembeddedinvirtualtime,timestampsonmessagesareclosertoreality

VM VM VM VM VM

SystemTime

t=100

vt =100

PhysicalMachineEmulationSystem

VM- VirtualMachineTimeSlice– SystemExecutionUnit

e.g.,TimeSlice=100μst=200

vt =100

t=300

vt =100

t=400

vt =100

t=500

vt =100

VirtualTimeSystemArchitectureforaContainer-basedNetworkEmulator

Sourcecode:https://github.com/littlepretty/VirtualTimeForMininet

37

VirtualTimetoEmulationFidelityEnhancement

38

VirtualTimeforSimulation/EmulationSynchronization

38

DSSNet UseCase

38

FutureWork

35

• MoreSDN-awareapplicationstoenableacyber-resilientandsecureenergyInfrastructure– e.g.,Specification-basedIntrusionDetection

• Networklayerà Applicationlayerà Cross-layerverification

• In-houseresearchideaà Realsystemdeployment– IITMicrogrid– FirstClusterofMicrogridsinUS(12MWIIT+10MWBronzeville)

36