Hafslund AMS - Drinking from the fire hose at a large IoT project

Hafslund AMS

Drinking from the fire hose at a large IOT project

Jon Andreas Pretorius, Hafslund Nett

Axel Borge,Sesam Simen Sommerfeldt, Bouvet

to NDC 2015

Hafslund s.2

s.3

Operational structure

Heat Network Markets Power

Hydro&power& District&hea1ng&produc1on&and&distribu1on&Bio7energy&

Network&System&control&centre&

Power&sales&Billing&Customer&services&

•  Hafslund Nett owns and operates Norway's largest electricity grid and has long had one of the lowest net rents

•  Hafslund Nett owns and operates the regional grid in Oslo, Akershus county and Østfold county

•  Hafslund Nett owns and operates the distribution network in Oslo and most of Akershus and Østfold counties

•  Number of distribution network customers are 675,000 •  The network consists of 37,000 km overhead lines and

underground cables •  Hafslund Driftssentral is one of Europe's most advanced

operating centers, that controls, monitors and optimizes power to 1.4 million people, Hafslund Varme's district heating plants in the Oslo area and Hafslund Produksjon's power plants in Glomma

Business Area Network

s.4

Hafslund was founded more than 100 years ago - in 1898

s.5

There will be more changes to the power grid operation the next five years than the 100 last years

1899 1911 2011 2020

?&

Changes in regulations will increase complexity significant and increase demand for automation

s.7

35 000 enkle fjernavleste målere

Årlig driftskost pr måler: ca 950

700 000 komplekse fjernavleste målere

Har i dag buffer for feilretting. Stort sett bare ifm måler-avlesning at programmet er tett

Årlig driftskost pr måler: 150

Alt online og ingen buffer eller servicevindu. Alt må alltid være tilgjengelig. Kan illustreres ved å tenke at det var måleravlesning hver dag hele døgnet

IT er i liten grad en trussel for omdømme

Hacking, virus mm vil utgjøre en mye større trussel generelt og målere vil kunne hackes

Kompleksitet høy, men vi reddes av rolige perioder

Kompleksitet vil være betydelig høyere og konstant

I dag 2020

AMS elHUB

Kundens forventning som i 1990 Hvordan vil kundens forventning endres?

AMS:&

•  The&new&smart&meters&detect&power&consump1on&on&an&hourly&basis&and&automa1cally&sends&informa1on&about&consump1on&to&the&grid&company&

•  The&new&smart&meters&records&also&includes&data&related&to&voltage&quality,&ground&faults&etc&

•  The&new&meters&have&two7way&communica1on&between&the&meter&and&grid&companies,&and&will&give&customers&current&informa1on&about&their&own&consump1on&and&instant&prices&for&electricity&and&grid&transport.&

•  Grid&companies&are&responsible&for&the&installa1on&and&opera1on&of&the&AMS&system,&but&they&can&not&deny&other&service&providers&to&offer&customers&management&and&informa1on&services.&

&

Advanced&Metering&and&Control&Systems&(AMS)&are&introduced&within&2019.01.01&

AMS$%$the$founda/on$of$the$future$of$digital$power$grid

s.9

Elhub and the supplier centric model

ElHub Statnett has been commissioned by NVE to establish Elhub. Elhub shall collect all metering values for Norway and makie these values available for power suppliers and their end customers. Furthermore Elhub will support processes for customers moving or switching suppliers, and compile data for clearing between participants in the electricity market For Hafslund Nett this means that collected and verified hour ly values from all AMS meters shall be transferred to ElHub once a day When vthe supplier centric model is established , customers will only deal with the electricity company (example service and infrastructure provider of mobile telephony) The supplier centric model creates major changes in business processes and data exchange in the industry

Drawing from elhub.no

s.10

System&D& System&…&

System&C&System&B&System&A&

System&N&

Hafslund investigated two alternative solutions for integration architecture that will support the demands of new AMS solution; -  ServiceBus -  Data hub(Semantic/RDF) Hafslund has experience with both solutions, but the project consider a Data Hub based solution most appropriate in this context; -  Increased stability

(asynchronous data exchange)

-  Fewer integration points -  Similarly architecture chosen

for central El Hub

DataNAV&

Choice$of$integra/on$solu/on

s.11

IFS$ERP$

Warehousing&&&Logis1cs& Project&module& WO7module& 360º&Scheduling&

New&field&system&

Economy&installa1on&registry&Documenta1on&

HR/resource&

Rollout$AMS$

Opera/ons$AMS$

Stage&Planning&and&monitoring& Assign& Start&& Perform& Report&Project,)Opera-ons,)

Maintanance)excis-ng)Recep1on&7&

withdrawals&goods&

Data&Hub&

GeoNIS&#installa1on&

Quant&#AMS&&

Generis&#old&meters&

CAB&#Customer&&

Datawarehouse&/&archive&

Consolidated&customer&and&installa1on&data&from&Data&hub&

Data&sources&for&rollout&

Data&Recipients&rollout&

Historical&data&archive&and&analysis&

Integration engine

All masterdata is consolidated in Data Hub Data Hub is the only source for all business applications In the semantic data base all data are connected Data Hub provides great potential for management of the information model and analysis

Established$applica/on$solu/on$design

www.hafslund.no www.hafslund.no

Advisor and CTO, Bouvet Oslo Lær Kidsa Koding! Oslo IoT meetup

@sisomm

Master data: The Elephant

in the room

System System

System

System

System

System

System

System System

System

System

I have given many talks and written several articles about IoT

Hypothesis: Internet of Things projects always involve integration

Archive

HR

CMS

Finance

Payroll

Classic Point-to-

Point integration

Archive

HR

CMS

Finance

Payroll

ESB ESB

Classic integration with ESB

The systems keep changing their need for information from other systems

Archive

HR

CMS

Finance

Payroll

Common data model?

Traditional SOA with a canonical datamodel

PREVENTS flexibility

DOA – Data Oriented Architecture.

Enter

27

Archive

HR

CMS

Finance

Payroll

Keep the models, keep the data

We copy the domain

context in the systems into

SESAM

Convert data to triplets - RDF ID Name Position Born E-mail Manager

101 Tim Berners-Lee

Programmer 08061955 [email protected] 958

958 Vint Cerf Inventor 23061940 [email protected] 999

765 Pål Spilling Professor 04111940 [email protected] 765

Subject Predicate Object

101

101

101

101

Type Person

Name

Position

Born

E-mail

Manager 101

[email protected]

08061955 958 Programmer Tim Berners-Lee 101

Universally unique identifiers

Subject Predicate Object

www.org.no/data/system/person/1 Type Person

www.org.no/data/system/person/1 Name Tim Berners-Lee

www.org.no/data/system/person/1 Position Programmer

www.org.no/data/system/person/1 Manager www.org.no/data/system/person/2

www.org.no/data/system/person/2 Name Vint Cerf

www.org.no = Unique organisation on the internet www.org.no/data/system/person/1 = unique id of the information element

Archive

HR

CMS

Finance

Payroll

Link data from different GRAPHS

Duplicates Contact=Contact

Location= Depnr Empnr=ID1

Org=Depnr

With DUKE we can link elements with no common identifiers – using statistical algorithms and training on data sets

Archive

HR

CMS

Finance

Payroll

Data sources can enrich eachother

HUB

Doc

Meta1 Meta2

Doc

Archive

HR

Turnus

Finance

Payroll

Keep data from ”dead” systems

Nav

HR

Payroll

Search

Archive

HR

CMS

Finance

Payroll

Hub

How things are connected

HR Dest

SDShare

Source

HUB

•  Based on Atom: Pull data, don’t push •  Asynchronous •  Subscribers ask for data that has changed

since the last time •  Update frequencies are adjustable •  You can ask for changes or the whole dataset •  Data formats changed in transfer.

www.sdshare.org

Sesam uses a Hashtracker to keep track of the latest changes, or an explicit “latest changed” field

Read using SQL/File acces, Write using API’s

A push receiver is a http server receiving RDF fragments. It calls the API of the receiving system. If the call fails, Sesam can try again

DOA

Database'

Content'Mngmt'

File'System'

Enterprise'Search'

Reporting'

Analytics'

Data'Hub'

System'X'

Public'Open'Data'

Content'Mngmt'

Enhance'and'

Connect'

COLLECT

CONNECT

SHARE

Tran

sform'

Data objects flow NOT messages

Kafka for extra throughput

SDShare'Server'KaDa'

Provider'

RDF'Store'

KaDa'Queue'

The Kafka Provider Pulls Information off from the Queue and can add extra data from the RDF store before exposing it out via SDShare. It can also apply filters based on data in the hub or the item on the queue.

•  SQL Databases via jdbc •  CSV files •  RDF triple stores •  Sharepoint •  Kafka •  XML files •  LDAP providers •  Excel files •  MS Exchange server (mail and calendar) •  SDSHARE – anything! (MongoDB, etc)

Data sources and sinks

45

Data Analytics & Enhancement Existing Systems

Processes Search and Reporting

All Data Indexed

Contribute data

Drive process through state change

Models in data, Constraints in data

Act on all data Analytics results are

just more data

Complete views of all

systems and processes

Use Data

All people can ask all questions

Uniformly Structured data

from heterogeneous

sources

System Improved

Other systems can keep running even if one is down. And you can upgrade a system or install a new with fewer impacts

The customer controls the information model and

becomes more independent from vendors

Customers can prototype and combine data for new

solutions rapidly

1.  Thou shall only get data from other domains through Sesam

2.  SOA is dead, long live DOA. Processes advance through state changes

3.  There can never be a common data model in the company

4.  Thou shall never query Sesam directly, but through SDSHARE

5.  Thou shall be comfortable with eventual consistency

6.  Thou will always get the same answer when you ask Sesam the same

question. And Sesam can say the same things many times

7.  The world is asynchronus, as is Sesam. Don’t try to shoehorn synchronicity

8.  Thou shall embrace that data can have different sources/master and values

9.  The systems need not know about Sesam

10.  Sesam is not a backup.

The Commandments of Sesam

•  Runs in Docker containers •  Github and Saltstack are used to keep all

installations up-to-date •  At the core: Virtuoso Triple Store •  Includes a data browser •  Indexed with SOLR to provide universal search •  All communication happens with SDSHARE •  Configuration over coding

Sesam tech

A paradigm shift for developers •  Eventual consistency •  “Pilfering” of data •  RDF and SDShare •  Sparql is not SQL •  Idempotence: Sesam

can send duplicates •  No RPC calls or

message passing •  You need an information

architect in the project •  Don´t add more queues.

Pedal to the metal! Now implement AMS with SESAM!

A recap of the requirements

•  Massive amounts of data •  Many systems must be coordinated •  Many stages in the deployment, with changing

needs •  Systems will be upgraded and changed •  The systems were not designed to cooperate with

each other •  Bugs and errors happen – in systems and human

actions.

670.000 x 4 x 24 =

64.320.000 daily readings

YAY! Need for american-scale technology

Yay! Need for

AMERICAN scale

technology!

The first challenge is to sync information across systems before the measurements can begin

Construct the SQL for the output feeds, implement push receivers for input feeds.

All the arrows are

feeds in/out of Sesam

Slide source: Ståle Heitmann, Computas

Slide source: Ståle Heitmann, Computas

When to use Sesam

•  When all else is tried – you are f***ked •  If you have many domains in the company •  If your integration work involves a lot of

data transformation, lookup and conversion

•  If the logic in the ESB rivals that of the systems

•  For Internet of things projects •  As a collector for big data projects

Micro services need to get information from several places

Want to know more?

•  contact us at [email protected] and we will help you get started

•  www.sesam.no •  www.sdshare.org

•  Anders Volle •  Ståle Heitmann •  Steinar Rudsar •  Axel Borge •  Øystein Isaksen •  Graham Moore •  Lars Marius Garshol •  Steinar Rune Eriksen

Thanks to...

Questions?

Thanks!

[email protected]

@sisomm/995 07 733

[email protected]

@axelborge/905 92 955