Research, Education, and
Cybersecurity via Open/Remote
Access Test-beds
Kishan Baheti
National Science Foundation
Examples of CYber Attacks
Superyacht attack, demonstrated July 20132
UAV attack, demonstrated June 20131
Phasor measurement unit attack,
demonstrated December 20113
Illustration: John MacNeill
Automotive radar also appears vulnerable
1
A.J. Kerns et al. Journal of Field Robotics, 31(4): 617–636, 2014.
J.A. Bhatti and T.E. Humphreys. 2014. In preparation.
3 D.P. Shepard, T.E. Humphreys, A.A. Fansler. International Journal of Critical Infrastructure Protection, 2012.
2
Distributed laboratory space with
open/Remote access
• Develop and use
open/Remote access, shared
engineering laboratory spaces
• Potential laboratories:
robotics, power systems,
drones, medical systems, etc.
• Shared cost and leveraging
expertise
Images: Vehicle: I. Lee, KOCSEA Workshop 2014; Microgrid: https://der.lbl.gov/
IMU
GPS
Left
Encoder
Camera
Right
Encoder
Classroom and research access to
remote laboratory experience
• “Bare bones” multi-robot laboratory ~ $250k
• Price is a major stumbling block for many
universities and research groups
• Research is done in isolation
• Resource competition
Image: M. Egerstadt; Ga Tech. White Paper. 2015.
Combination of research and
education
• CPS security is a key requirement of these
laboratories
• Real-time code verification needs to be
developed
• Multiple groups of researchers and students
can simultaneously work on the systems
• Competitive educational games can be played
to test security of the systems
Energy, Power, Control & Networks
(EPCN)
• Design and analysis of complex dynamic
systems integrating sensing, imaging, control,
networking, and computational technologies
• Emphasis on electric power systems
– generation, transmission, distribution
• High power electronics and drives
• Regulatory and economic structures
Energy, Power, Control & Networks
(EPCN)
• Energy Harvesting, Storage Devices and Systems
• Solar and Wind Energy and Integration of
renewables with Grid
• Monitoring, Protection and Cyber Security of
Power Grid
• Advanced Power Electronics and Electrical
Machines
• Electric and Hybrid Vehicles; Integration with Grid
• Policy, Economics, Consumer Behavior and the
Power Grid
TRUSTWORTHINESS OF SMART GRIDS:
U.S. DEPARMENT OF ENERGY SUPPORT FOR THE GRID OF THE FUTURE
IEEE Conference on Innovative Smart Grid Technologies
Washington, DC
February 18, 2015
Gil Bindewald
Evolution in the Power Grid
Availability of Data is Growing
“Smart Grid” data sources enable real-time precision in
operations and control to dynamically optimize grid operations
to adapt to changing conditions
• Real-time data from distribution automation and smart
meter systems will significantly advance real-time
operations of distribution systems and enable customer
engagement through demand response, efficiency etc.
• Time-synchronized phasor data, linked with advanced
computation and visualization, enable advances in state
estimation, real-time contingency analysis, and real-time
monitoring of dynamic (oscillatory) behaviors in the
system.
Vast Quantities of Data
Energy Meets Big Data
With so much grid state and customer data available in nearreal time, how can operators make sense of it all?
• Tools must be developed to leverage the data
to assist system operators in effectively
predicting, preventing, and responding to
emerging system events
• Processing real time grid state and customer
data is critical in enabling operators to react to
grid disturbances
• Data architecture needs to address
heterogeneity, scale, timeliness, complexity
and privacy
The DOE Advanced Grid Modeling Program
enables predictive capabilities
5
Future Grid Architecture
Today's grid is generation that follows load.
Tomorrow’s grid is loads that follow generation.
•
•
•
•
Grid architecture must
accommodate more
distributed systems
Lines between
transmission &
distribution will be blurred
Linkage of EMS and
DMS will be important
Next Generation EMCS
(Energy Management &
Control System)
Architecture being
developed at CURENT
Complementary Program Activities:
Measurements and Model-Based Analytics
To turn real-time system data into actionable information requires an
understanding of not only “what is happening” but also “what could happen.”
Cyber-Physical ICT
Turning Massive Data into
Actionable Information
• Analytics initiatives are needed to understand and optimize
the electricity system
Dynamic States
• Furthermore, addressing emerging dynamics and
uncertainty will require merging rigorous data analysis and
modeling techniques for application to power system
problems in “real-time”
Data
cycle
Simulation
“What
could be”
Stability
Dynamic contingency analysis
Look-ahead dynamic simulation
Contingency Analysis
Dynamics state estimation
1/30 sec
2/30 sec
3/30 sec
Optimization
1 min
Time
State Estimation
Topology
“What is”
Promoting Strong Stakeholder
Collaboration
“Sustaining aggressive and proactive energy
delivery systems security improvements over
the long term requires a strong and enduring
commitment of resources, clear incentives, and
close collaboration among stakeholders. Energy
sector collaboration provides the resources and
incentives required for facilitating and
increasing sector resilience.”
1
Cyber-Physical ICT and Power:
Issues and Opportunities
Application Context Setting
Erich W. Gunther, P.E., IEEE Fellow
Chairman and CTO - EnerNex
[email protected]
2
Smart Grid Domains, Interfaces and Applications
Markets
Operations
Retailer /
Wholesaler
RTO/ISO
Ops
Transmission
Ops
Service Providers
Distribution Ops
DMS
WAMS
Service
Integration
Energy
Market
Clearinghouse
Demand
Response
Service
Integration
CIS
Billing
Service
Integration
Transmission
SCADA
Metering
System
Distribution
SCADA
Wide Area
Networks
Field Area
Networks
Energy
Services
Interface
Data
Collector
Substation
Integration
Substation
Controller
Substation
Device
Domain
Customer
Equipment
Meter
Field
Device
Distributed
Generation
Electric
Storage
Network
Roles and Actors
Gateway Role
Comms Path
Comms Path Changes Owner / Domain
Transmission
Others
Customer
EMS
Premises
Networks
Generators
Generation
Billing
Home / Building
Manager
Internet /
e-Business
Market
Services
Interface
CIS
Retail
Energy
Provider
MDMS
Internet /
e-Business
Plant Control
System
Third-Party
Provider
Aggregator
RTO
SCADA
ISO/RTO
Participant
Asset
Mgmt
EMS
EMS
Aggregator
Utility
Provider
Electric
Storage
Electric
Storage
Appliances
Electric
Vehicle
Distributed
Generation
Distribution
Distributed Energy Resources
Thermostat
Customer
Smart Grid Command and Control
Enterprise Strategy & Operations
• Asset Forecasting
• Workforce Forecasting
• Generation Planning
Monitor
•
•
•
•
•
•
•
•
•
Situation Awareness
Comm Status
Environmental
Smart Grid Devices
Sensors
Security
Market conditions
Maintenance Status
GIS
• Energy Efficiencies
• Total Ownership Cost
• Carbon Trading
Assess
•
•
•
•
•
•
•
•
•
Outages
Maintenance Data
Demand
Generation Capacity
Demand Response
Capacity
Intrusions
Trending
Actual vs. Planned
Data Mining
• Customer Trending
• Maintenance Forecasting
• Automated Outage Mgmt
Predict & Plan
•
•
•
•
•
•
•
•
•
•
Modeling
Maintenance
Predicted Demand
Weather
Ecological
Renewable Resources
Consumer Patterns
Market Pricing
T&D Constraints
Optimization
Control
• Classic SCADA &
System Control
• IT System Control
• Dist Resource & Load
Control
• Intelligent Device Mgmt
• Authorize Sales
• Crew Dispatching
• Regulatory Reports
• Back office
• Purchase Requests
Interfaces: Smart Grid Devices, Field Devices & Systems, Sensors, Market Pricing,
Weather, RTO Demand Response, Back Office Systems
Technology: SOA, C2 Apps, Modeling, Simulation, Data Mining/Correlation/Fusion,
Visualization, Business Intelligence, Large-scale Systems Mgmt & Optimization
SGCC Architecture
Predicted vs.
Energy Market
Environmental
Recommended
(Current, Predicted) (e.g., Wx, pollution)
Course of Actions
Presentation
(Operator Stations, Web Displays, Data Walls)
Enterprise Status
and Diagnostics
Demand vs.
Generation
Enterprise Strategy & Operations
Monitor
Assess
Net
Predict
& Plan
Control
Enterprise
System
Mgmt
Information Bus
Federated
Command
Federated
Center
Command
Center
Security
Grid
Comms
Networks
SOA Infrastructure
(Scalable, Open, Standards-Based)
Adaptation
Training
Data
Legacy
Data
COOP
Data
Enterprise Operations
Data
Alternate Site
Layered Enterprise Integration Platform
Operational Decision & Response Times
Increased Variability Requiring More Dynamic Operations
on Shorter Time Cycles
Background - Execution Cycles & Temporal Coordination
Slower cycles – 2 sec & over
faster cycles - under 2 sec
Hour-Ahead
Cycle
5 min
cycle
1 min
cycle
Power System
Including control,
measurement and
protection devices
2 sec
cycle
1 sec
cycle
100
m-sec
cycle
10
m-sec
cycle
Forward plans, schedules,
guidelines controls, messages
Feedback data, violations,
alerts, messages
Edge
Function Fn
Function F2
Function F1
(e.g. Voltage Stability)
Intelligent
Functional
Agent for F1
Intelligent
Functional
Agent for F1
Intelligent
Functional
Agent for Fn
Intelligent
Functional
Agent for F1
Intelligent
Functional
Agent for Fn
Intelligent
Functional
Agent for F1
Intelligent
Functional
Agent for F1
Actuator
Actuator
Actuator
Intelligent
Functional
Agent for Fn
Intelligent
Functional
Agent for F2
Actuator
Actuator
Actuator
Intelligent
Functional
Agent Fn
Actuator
Actuator
Actuator
Integrated Messaging/Data
Feeder Substation
Region
Central
Distributed Hierarchical Control System
Impact of Latency on Control
No Latency
Slower cycles – 2 sec & over
faster cycles - under 2 sec
Hour-Ahead
Cycle
Fixed Latency
Random Latency
5 min
cycle
1 min
cycle
2 sec
cycle
1 sec
cycle
100
m-sec
cycle
10
m-sec
cycle
Power System
Including control,
measurement and protection
devices
Forward plans, schedules,
guidelines controls, messages
Feedback data, violations,
alerts, messages
It’s All About the Requirements!
Developed under contract for EPRI by EnerNex and GE IECSA Team
11
New Technology Adoption is not Easy
•
•
•
•
•
•
•
•
•
•
•
•
Clear Business Objective
Technology Selection
Impact on Existing Infrastructure
Ability of Organization to Adapt
Method of Implementation
Reliability and Security Impacts
Testing and Certification
Metrics to Evaluate Implementation
Effectiveness
Cost Recovery and Other Regulatory Issues
Business Risk Assessment and Overall
Governance
Life Cycle Management
End of Life
13
Questions ????
• Erich W. Gunther [email protected]
GE Digital Energy
ICT and Power: Role of ICT in SG, Data
Management, Communications, IoT
and Analytics, R&D Opportunities
John D. McDonald, P.E.
Director
Technical Strategy & Policy Development
IEEE Fellow
IEEE PES Substations Committee Chair 2001-2002
IEEE PES President 2006-2007
IEEE Division VII Director 2008-2009
IEEE-SA Board of Governors 2010-2011
IEEE PES ISGT ToSG Workshop
Washington, DC
February 18, 2015
Role of ICT in Smart Grid
A “Smarter” Grid
Enabled
Utility Managers
‘New Applications enabled by Additional Infrastructure’
Management
“ Applications”
Economic
Dispatch
Energy
Optimization
Asset
Optimization
Demand
Delivery
Optimization Optimization
Enabled
Consumers
Control
“How Power Flows”
Gen & Trans
Mgt.
Transmission
Automation
Dist.
Mgt.
Sensors
Dist.
Automation
Heavy Metal
“ Generate & Deliver Power”
Thermal
Generation
Lines
Sub
Stations
Dist
Equipment
Old Grid
Voltage
Control
Renewable
Generation
Adv.Metering
System
Old Grid
Smart Grid Adds
Smart Grid
•
You call when the power goes out.
•
Utility knows power is out and usually restores it automatically.
•
Utility pays whatever it takes to meet peak demand.
•
Utility suppresses demand at peak. Lowers cost. Reduces CAPEX.
•
Difficult to manage high Wind and Solar penetration
•
No problem with higher wind and solar penetration.
•
Cannot manage distributed generation safely.
•
Can manage distributed generation safely.
•
~10% power loss in T&D
•
Power Loss reduced by 2+%… lowers emissions & customer bills.
Smart Grid Technology Roadmap
IT – Data Validation and
Management
Smart Meters/AMI Integration with
GIS, OMS and DMS
Smart Meters/AMI
• Meter Readings
• Voltage => DMS
• Last Gasp Communication => OMS
GIS
• Network Model Information => OMS, DMS
DMS
• Status Changes => OMS
Customers
• Phone Calls => OMS
• Social Media => OMS
Enterprise Data Management - Concept
Enterprise Data Management – Architecture
Distribution State Estimation (SE)
What are “States” – State Variables ?
States are the minimum set of variables that uniquely determine the grid
operation condition, e.g., Node Voltage Magnitudes and Phase Angles,
With node Vs and , all other quantities like Branch/Node Ps, Qs, Is can be
uniquely calculated
Can other sets of variables be state variables?
Why do we need State Estimation?
There are more known variables than the minimum number of knowns to
uniquely find out the unknowns (an over-determined problem)
The knowns are conflict and cannot consistently determine the unknowns
Different levels of trust to the accuracies of the knowns
What are the differences of SE for Distribution Grid vs. Transmission Grid?
Distribution SE - Data Validation
Usecases of Distribution SE
Bad Data Detection for real time and estimated measurements
Accurate Voltages and Power Factors for VVC Control in real time
Accurate awareness of grid operation condition in real time
Differences of SE for Dist. Grid vs. Trans. Grid
Unbalanced Gird – Single Phase, 2 – 3 Phases, Grounded, Non-grounded
Unbalanced Loads, Unbalanced DG Generations
Mainly Radial Grid Configuration
With Limited or lack of sufficient measurement redundancy
Taking advantage of radial network configuration
Available for decoupled SE solution: Sub. By Sub. Or even Feeder by Feeder
Communications – Foundational
Technology
Reliable, Empowered, Clean, Efficient, Productive
Complex Event Processing, Analytics, and Visualization
Smart
Meters
Volt/VAR
Control
Smart
Substation
Design
Tools
Microgrid
Applications
Energy
Efficiency
DERM
Modular
Design
Monitor &
Asset
Diagnostics
Management
Energy
Demand
EV Charging
Storage
Response
Automation
Renewable
Workforce
Modular
Generation
Management
Substations
FDIR/FLSIR
Smart
Appliances
Physical
Enterprise
Adapters
Adapters
Business Process
Management
Domain Services Data Services
Electrical
Network
Model
Federated
Data Model
System Security
Services
Head End Systems
Backhaul
Plant
LAN
Generation
System
WAN
Transmission
Substation
LAN
Substations
Field
LAN
Customer Area
Networks
Distribution
Customer
Premise
Distributed Generation and Storage
Economical, Societal and Environmental Sustainability
Security and Privacy
Comms
Data Services
Multiple Applications Platforms
NIST- Recognized
Standards Release 1.0
Following the April 28-29
Smart Grid Interoperability
workshop, NIST deemed that
sufficient consensus has
been achieved on 16 initial
standards
On May 8, NIST announced
intention to recognize these
standards following 30 day
comment period
NIST’s announcement
recognized that some of
these standards will require
further development and
many additional standards
will be needed.
NIST will recognize
additional standards as
consensus is achieved
Standard
Application
AMI-SEC System Security
Requirements
Advanced metering infrastructure (AMI) and Smart Grid end-to-end
security
ANSI C12.19/MC1219
Revenue metering information model
BACnet ANSI ASHRAE 135-2008/ISO
16484-5
Building automation
DNP3
Substation and feeder device automation
IEC 60870-6 / TASE.2
Inter-control center communications
IEC 61850
Substation automation and protection
IEC 61968/61970
Application level energy management system interfaces
IEC 62351 Parts 1-8
Information security for power system control operations
IEEE C37.118
Phasor measurement unit (PMU) communications
IEEE 1547
Physical and electrical interconnections between utility and
distributed generation (DG)
IEEE 1686-2007
Security for intelligent electronic devices (IEDs)
NERC CIP 002-009
Cyber security standards for the bulk power system
NIST Special Publication (SP) 80053, NIST SP 800-82
Cyber security standards and guidelines for federal information
systems, including those for the bulk power system
Open Automated Demand
Response (Open ADR)
Price responsive and direct load control
OpenHAN
Home Area Network device communication, measurement, and
control
ZigBee/HomePlug Smart Energy
Profile
Home Area Network (HAN) Device Communications and Information
Model
Communication Protocols
Control Center to Control Center
• IEC 60870-6/TASE.2 – Inter-control Center Communications
Protocol (ICCP)
Control Center to Field Equipment
• IEEE 1815 (DNP3) – North American Suppliers
• IEC 60870-5 – European Suppliers
• 101 – serial communications
• 103 – protection devices
• 104 – TCP/IP (network communications)
Field Equipment
• IEC 61850 – substation automation and protection
• IEEE 1815 (DNP3) – substation and feeder device automation
IoT and Analytics
Internet of Things (IoT)
Drive the next productivity revolution by connecting
intelligent machines with people at work
The “II” Connects…
1.
Intelligent
Machines
Leverage technology &
communication to costeffectively connect
machines
+
2.
Big Data
& Analytics
Combine the power of
big data, big analytics,
and industry physics
+
3.
People
at Work
Connecting people any
place, any way, and any
time for intelligent
operations
= A world that works better, faster, safer, cleaner and cheaper
Energy
Value:
Global Energy Capex
$1.9T/year
The first 1% annual savings
equals $300B over 15 years
Analytics
Meter Insight
Outage Insight
Reliability Insight
Renewables Insight
Consumer Insight
(in development)
(in development)
(in development)
(in design)
(in design)
• Automated KPI
data validation
• Dynamic KPI
dashboards
• Outage Event
Recorder
• Planned outage
optimization
• Predictive
Outage
Analytics
• Accurate ETR
• Predictive
vegetation
management
• Asset health
analysis
• System health
analysis
• Lifecycle
analysis and
portfolio
optimization
• Revenue
Protection
• Power Quality
and Reliability
• Load
Forecasting and
Research
• PV load
(dis)aggregatio
n/ hotspot
analysis
• Wind load
(dis)aggregatio
n and hotspot
analysis
• EV penetration/
impact analysis
• DER load
orchestration
• Social media
integration
• Customer
Segmentation
• Customer
Engagement
• Sentiment
Analysis
Research Opportunities
Impact of High Penetration of Rooftop
Solar PV on the Distribution System
New Applications of Power Electronics (my Power Electronics
magazine article – August 22, 2013 issue)
• Substation Transformer On-line Tap Changer
• Low Voltage Network Dynamic Grid Edge Controllers
• Increased capability from Inverters
The Death Spiral (Intelligent Utility magazine article – November
/December 2013 issue)
• Impact of High Penetration of Rooftop Solar PV in the State of
Queensland, Australia
Thank You!
The agile fractal grid
RED
Substation
BLUE
Substation
RED
Substation
BLUE
Substation
RED
Substation
BLUE
Substation
RED
Substation
BLUE
Substation
But what if simple back feeding is not enough?
Resiliency from
Advanced
Sensor
Technology
+
Advanced
Forecasting
+
Advanced
Analytics
+
Advanced
Control
Apply all “smart grid technologies” in a coordinated way
– Smart feeder switching
– Advanced sectionalization
– Rolling disconnects (down to meter level)
– Dispatchable backup generators
– Distributed energy
– Advanced Volt/VAR control
– Storage
…….
That’s how the agile grid can work
on a feeder. How would it look at
a system level
On a nice day, the
utility is running smoothly.
Everything is “Hot”
A bad storm comes through
and some areas lose power
But backup power,
distributed generation,
and stored energy allow
some areas to island
and continue operation…
Smart control systems allow
the islands to network
This may allow some
power to be restored in
additional areas as the
utility works to bring dark
areas on line
Soon, everyone has power
The smaller grids
reconnect with the utility
OK, it is an interesting
concept, but how does it
work?
All grid applications have the same basic structure
Implement Action
Implement Action
Derive Decision
Derive Decision
Perform Analysis
Perform Analysis
Transform and Organize Data
Transform and Organize Data
Collect Data
Collect Data
Abstraction Model
Action
Decision
Analysis
Information
Data