Aalborg Universitet Automation of Smart Grid operations ... · intelligence to current power systems, helping with automatic anomaly de-tection and data accuracy diagnosis. Eventually,

Aalborg Universitet

Automation of Smart Grid operations through spatio-temporal data-driven systems

Stefan, Maria

Publication date:2019

Document VersionPublisher's PDF, also known as Version of record

Link to publication from Aalborg University

Citation for published version (APA):Stefan, M. (2019). Automation of Smart Grid operations through spatio-temporal data-driven systems. AalborgUniversitetsforlag. Ph.d.-serien for Det Tekniske Fakultet for IT og Design, Aalborg Universitet

General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright ownersand it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.

? Users may download and print one copy of any publication from the public portal for the purpose of private study or research. ? You may not further distribute the material or use it for any profit-making activity or commercial gain ? You may freely distribute the URL identifying the publication in the public portal ?

Take down policyIf you believe that this document breaches copyright please contact us at [email protected] providing details, and we will remove access tothe work immediately and investigate your claim.

Downloaded from vbn.aau.dk on: November 12, 2020

https://vbn.aau.dk/en/publications/41b41683-6d7a-4cb6-9048-7e5dd9fcda7e

Ma

ria Stefa

na

uto

Matio

n o

f SMa

rt G

rid

oper

ation

S thr

ou

Gh

Spatio-teM

por

al d

ata-d

riven

SySteMS

autoMation of SMartGrid operationS

throuGh Spatio-teMporaldata-driven SySteMS

byMaria Stefan

Dissertation submitteD 2019

Automation of SmartGrid operations

through spatio-temporaldata-driven systems

Ph.D. DissertationMaria Stefan

Dissertation submitted: May 29, 2019

Dissertation submitted: May 29, 2019

PhD supervisor: Assoc. Prof. Rasmus Løvenstein Olsen Department of Electronic Systems Aalborg University

Assistant PhD supervisor: Assoc. Prof. Jose Manuel Gutierrez Lopez Department of Electronic Systems Aalborg University

PhD committee: Associate Professor Tatiana Kozlova Madsen (chair.) Aalborg University

Professor Josep Solé-Pareta Universitat Politècnica de Catalunya

Dr. Piotr Kiedrowski University of Science and Technology in Bydgoszcz

PhD Series: Technical Faculty of IT and Design, Aalborg University

Department: Department of Electronic Systems

ISSN (online): 2446-1628 ISBN (online): 978-87-7210-445-4

Published by:Aalborg University PressLangagervej 2DK – 9220 Aalborg ØPhone: +45 [email protected]

© Copyright: Maria Stefan

Printed in Denmark by Rosendahls, 2019

Curriculum Vitae

Maria Stefan

Maria Stefan received her B.Sc. E.E with a specialization in Electrical, Elec-tronics and Communications Engineering from the Polytechnic University ofBucharest in 2013. In 2015 she received her M.Sc. E.E. in Wireless Commu-nication Systems from Aalborg University. Her M.Sc. thesis was the resultof research conducted with Nokia Solutions and Networks Denmark. Shewas employed for a few months after completing the M. Sc. studies as re-search assistant at Aalborg University. Since 2016 she has been employed as aPhD Fellow in the Wireless Communication Networks section (WCN) in theDepartment of Electronic Systems at Aalborg University. In 2018-2019 shevisited Universitat Politècnica de Catalunya in Barcelona, Spain as internshiptrainee, collaborating on the topic of data analysis and machine learning. Thefocus of her current research is on low-voltage electrical grids data analysis,processing and visualization, based on user experience studies.

iii

Curriculum Vitae

iv

Abstract

Traditional electricity grids are currently undergoing a transformation to-wards distributed generation, changing the state of the art operational pro-cesses for grid monitoring and maintenance. As Danish incentives for greenenergy production are being laid out, planning to have 100% renewable en-ergy production by year 2050, consumers have begun to install renewableenergy resources (RES) in the form of PVs, small wind turbines, heat pumpsand electrical vehicles. The typical consumers become small producers (so-called prosumers), producing a bi-directional power flow. The face of thelow-voltage electrical grid is therefore changing at a rapid pace, which posesoperational challenges to the Distributed System Operators (DSOs) in termsof grid monitoring and maintenance.

Electrical grid operation is furthermore influenced by the deployment ofAdvanced Metering Infrastructures (AMI), consisting of a large amount ofinterconnected sensors/consumers. Modern AMI are capable of deliveringmany more measured parameters compared with traditional metering infras-tructures, where the data is currently used only for billing. AMI opens thepossibility to utilize the available information for more efficient grid mon-itoring, planning and can even be used for prediction and event-detectionpurposes. Novel data-driven analysis techniques are therefore required toexplore the new AMI parameters, bringing the electrical grids research fieldtowards digitalization.

The large and varied amount of data conveyed by the AMI has recentlybeen referred to as Big Data, both in industry and in the research fields. Thisdefinition is furthermore enhanced by the modern communication infrastruc-tures, which make it possible to stream the data from the AMI with a muchfiner granularity, known as real-time data.

The aim of this PhD study is to investigate how AMI data can contributeto a more efficient grid operation for the DSOs, by means of processing, an-alytics and visualization techniques. The conducted research has been basedon a real electrical grid case scenario in collaboration with a Danish DSO fromThisted, located in the north-western part of the country. The focus has beenon designing and implementing a visualization system based on the DSOs’

v

Abstract

needs and requirements. At the same time, the available geographic andtime-series data was used to perform data accuracy studies and to propose apotential analytics platform for the DSOs.

User experience studies (UX) have been an important part of the work,especially for designing a simple and effective visual overview over the low-voltage electrical grid. In these studies, the users (DSOs) took part in on-siteinterviews and therefore helped shaping the user interface for the visualiza-tion prototype, which can be utilized for monitoring, planning and predic-tions. The contribution consists of enhancing the automation of the consumerlevel grid operations, by designing and implementing a decision support in-formation system. Furthermore, the use of geographic information systems(GIS) contributed to spatial and situation awareness, especially relevant in ahuman-dependent operational environment.

Additionally, the available time-series measurements and GIS grid topol-ogy have been part of a study concerning the validity and integrity of dataexchanged in the electrical grid. It has been found that due to the lack ofa fully data-integrated system there are often inaccuracies in the data ex-changed between the different parties, leading to erroneous use of informa-tion for the different operations. To provide the DSOs with smart functional-ities, consumer behavior studies have been conducted. Based on their results,a classification of the low-voltage grid consumers has been proposed accord-ing to their energy consumption. It was shown that the created clusters areuseful for grid planning even in the case of missing information, as well asfor predicting how a certain customer might behave based on its profile.

Finally, the outcome of this work involves the optimization of the DSOsdaily workflows by system redesign and minimizing the operating expenses(OpEx) by integrating smart analytical methods. The conducted researchproves that even simple statistics and machine learning methods can bringintelligence to current power systems, helping with automatic anomaly de-tection and data accuracy diagnosis. Eventually, this together with other cur-rent as well as future research will contribute to the development of so-calledSmart Grids.

vi

Resumé

Det danske elnet undergår i denne tid en forandring mod en større gradaf distribueret energigenerering, hvilket betyder nye procedurer for blandtandet overvågning og planlægning af nettet. Da der fra politsk side er etønske om at have 100% produktion af vedvarende energi i 2050, er forbrugereallerede nu i gang med at installere forskellige vedvarende energiressourcerså som solceller, små vindmøller, varmepumper og elektriske køretøjer. Tra-ditionelle forbrugere bliver dermed til småskala producenter ("prosumers"),der introducerer et tovejs flow af strøm. Lavspændingsnetværkets strukturændrer sig derfor hurtigt og resultatet afspejles i operationelle udfordringerhos de energi operatører (DSO) der kontrollerer netværkets overvågning ogvedligeholdelse.

Driften af elnettet er desuden påvirket af udbredelsen af Advanced Meter-ing Infrastructures (AMI), der består af en stor mængde af sammenkobledesensorer/forbrugere. Moderne AMI er i stand til af levere flere og forskel-lige parametre sammenholdt med traditionelle måleinfrastrukturer, hvor datakun bruges til fakturering. AMI åbner muligheden for at udnytte de tilgæn-gelige informationer til mere effektiv netovervågning, planlægning og kanendda bruges til forudsigelses- og hændelsesdetekteringsformål. Nye data-drevne analysetekniker er derfor nødvendige for at udforske de nye parame-tre fra AMI, der bringer elforskingsområdet mod digitalisering.

De store og varierede mængder data, der er fremsendt fra AMI, er fornylig blevet omtalt som Big Data inden for både industri og forskingsom-råder. Denne definition er endvidere forstærket af den moderne kommu-nikationsinfrastruktur, som gør det muligt at streame data fra AMI med enmeget finere granularitet, kendt som realtidsdata.

Formålet med dette ph.d. studie er at undersøge hvordan AMI data kanbidrage til en mere effektiv netdrift for DSO’er ved hjælp af behandling, anal-yse og visualiseringsteknikker. Den gennemførte forsking er baseret på etrealt elnet scenario i samarbejde med en dansk DSO fra Thisted. Der harværet fokus på design og implementering af et visualiseringssystem baseretpå DSO specifikke behov og krav. Samtidig blev de tilgængelige geografiskeog tidsseriedata brugt til at udføre data-nøjagtighedsundersøgelser og til at

vii

Resumé

foreslå en potentiel analyseplatform for DSO’erne.Brugervenligheds studier (UX) har været en vigtig del af arbejdet, især for

at designe et simpelt og effektivt visuelt overblik over lavspændingsnettet. Idisse undersøgelser deltog brugerne (DSO’er) i på-stedet interviews og hjalppå den måde med at forme brugergrænsefladen til den prototype på visu-alisering, som kan anvendes til overvågning, planlægning og forudsigelser.Bidraget består i at højne automatikken for netdrift-operationer på forbruger-niveau ved at designe og implementere et beslutningsstøttesystem. End-videre bidrog brugen af geografiske informationssystemer (GIS) til rumligog situationsbevidsthed, især væsentligt i et driftsmiljø afhængigt af men-neskelig indgriben.

Derudover har de tilgængelige tidsseriemålinger og GIS-nettopologi væreten del af en undersøgelse vedrørende præcisionen af data udvekslet i elnet-tet. Det har vist sig, at på grund af manglen på et fuldt integreret datasys-tem er der ofte unøjagtigheder i de data, der udvekles mellem de forskelligeparter, hvilet udmønter sig i fejlagtig brug af oplysninger til de forskelligenet-operationer. For at bibringe DSO’erne en højnet funktionalitet er for-brugeradfærdsstudier blevet gennemført. På baggrund af deres resultater eren klassificering af lavspændingsnetforbrugerne blevet forslået efter deres en-ergiforbrug. Det blev vist, at de oprettede clusters er nyttige til netplanlægn-ing, selv i tilfælde af manglede oplysninger, samt til at forudsige, hvordan enbestemt kunde måtte opføre sig på baggrund af sin profil.

Endelig indebærer resultatet af dette arbejde optimering af DSO’ernesdaglige arbejdsgange ved systemredesign og minimering af OPEX omkost-ningerne ved at integrere intelligente analysemetoder. Den udførte forsk-ing viser, at selv enkle statistik- og maskinindlæringsmetoder kan bringe in-telligens til det nuværende elsystem, som hjælper med automatisk anoma-litetsdetektering og med data-nøjagtighedsdiagnoser. I fremtiden vil dennesamt anden nuværende samt fremtidig forsking bidrage til udviklingen af desåkaldte Smart Grids.

viii

Contents

Curriculum Vitae iii

Abstract v

Resumé vii

Thesis Details xi

Preface xiii

I Introductory Chapters 1

1 Introduction 31.1 Electrical grids in Denmark . . . . . . . . . . . . . . . . . . . . . 31.2 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.2.1 Hypothesis . . . . . . . . . . . . . . . . . . . . . . . . . . 61.2.2 Case study - test area at Thy-Mors Energi . . . . . . . . 6

1.3 Research challenges and contributions . . . . . . . . . . . . . . . 7

2 Theoretical Background 92.1 Geographic Information Systems - GIS . . . . . . . . . . . . . . 92.2 Database Management Systems . . . . . . . . . . . . . . . . . . . 11

2.2.1 Database Privacy Features . . . . . . . . . . . . . . . . . . 122.3 Data processing and analytics techniques . . . . . . . . . . . . . 14

2.3.1 Data processing . . . . . . . . . . . . . . . . . . . . . . . . 142.3.1.1 Batch Processing . . . . . . . . . . . . . . . . . . 142.3.1.2 Stream Processing . . . . . . . . . . . . . . . . . 14

2.3.2 Data Analytics . . . . . . . . . . . . . . . . . . . . . . . . 162.4 Decisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

ix

Contents

3 Research tracks 193.1 User experience studies (UX) - Distributed System Operators

(DSOs) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.1.1 ’Day-in-the-Life’ Model . . . . . . . . . . . . . . . . . . . 203.1.2 User profiles . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.2 Information system for the low-voltage electrical grid . . . . . . 223.2.1 Main visualization themes as identified by the DSOs . . 23

4 Contributions 274.1 Scientific contributions . . . . . . . . . . . . . . . . . . . . . . . . 27

4.1.1 Visualization systems . . . . . . . . . . . . . . . . . . . . 274.1.2 Data analytics methods . . . . . . . . . . . . . . . . . . . 294.1.3 Overall scientific contribution . . . . . . . . . . . . . . . . 30

4.2 Practical contributions . . . . . . . . . . . . . . . . . . . . . . . . 304.2.1 Design of the visualization prototype . . . . . . . . . . . 31

4.3 Baseline for future development . . . . . . . . . . . . . . . . . . 32

5 Conclusion 35

II Papers 45

A Visualization Techniques for Electrical Grid Smart Metering Data:A Survey 47

B Data Analytics for Low Voltage Electrical Grids 65

C Exploring the Potential of Modern Advanced Metering Infrastruc-ture in Low-Voltage Grid Monitoring Systems 81

D Automation of smart grid operation tasks via spatio-temporal ex-ploratory visualization 97

E (Position paper) Characterizing the Behavior of Small Producers inSmart GridsA data sanity analysis 119

x

Thesis Details

Thesis Title: Automation of smart grid operations through spatio-temporal data-driven systems

PhD Candidate: Maria StefanSupervisors: Assoc. Prof. Rasmus Løvenstein Olsen - Aalborg Univer-

sityAssoc. Prof. Jose Manuel Gutierrez Lopez - Aalborg Uni-versity

This thesis is submitted as partial fulfilment of the requirements for the de-gree of Doctor of Philosophy (PhD) from Aalborg University, Denmark. Thethesis is compiled as a collection of papers resulting in the main part of thethesis being scientific papers published in, or submitted to, peer-reviewedjournals and conferences. The work presented in the thesis is the result ofthree years of research, in the period June 2016 – May 2019, as a PhD fellowin the Section of Wireless Communication Networks (WCN), Department ofElectronic Systems, Aalborg University.

The PhD stipend (nr. 8-16026) has been funded as a part of the Remote-GRID project. The ForskEL program under Energinet.dk have together withAalborg University and industry partners; Thy-Mors Energi and Kamstrupfinanced this project.

The main body of this thesis consist of the following papers:

A. Maria Stefan, Jose G. Lopez, Morten H. Andreasen and Rasmus L.Olsen, "Visualization Techniques for Electrical Grid Smart MeteringData: A Survey", IEEE Third International Conference on Big Data Com-puting Service and Applications (BigDataService), 2017

B. Maria Stefan, Jose G. Lopez, Morten H. Andreasen, Ruben Sanchez andRasmus L. Olsen, "Data Analytics for Low Voltage Electrical Grids",Proceedings of the 3rd International Conference on Internet of Things, BigData and Security - Volume 1: IoTBDS, pp.221-228, 2018

xi

Thesis Details

C. Maria Stefan, Jose G. Lopez and Rasmus L. Olsen, "Exploring the Poten-tial of Modern Advanced Metering Infrastructure in Low-Voltage GridMonitoring Systems", IEEE International Conference on Big Data, 2019

D. Maria Stefan, Morten H. Andreasen, Jose G. Lopez, Michael Lyhne andRasmus L. Olsen, "Automation of smart grid operation tasks via spatio-temporal exploratory visualization", The journal of Environment and Plan-ning B: Urban Analytics and City Science, SUBMITTED 2019

E. Maria Stefan, Jose Gutierrez, Pere Barlet, Oriol Gomis and Rasmus L.Olsen, "(Position paper) Characterizing the Behavior of Small Produc-ers in Smart Grids. A data sanity analysis", Journal of Applied Energy,SUBMITTED 2019

According to the Ministerial Order no. 1039 of August 27, 2013, regardingthe PhD Degree § 12, article 4, statements from each co-author have beenprovided to the PhD school for approval prior to the submission of this the-sis, regarding the PhD student’s contribution to the above-listed papers. Theco-author statements are also presented to the PhD committee and includedas a part of the assessment.

In addition to the listed papers as the main content of this thesis, thefollowing paper is co-authored during the PhD studies. As this paper is nota part of the main body of this thesis it has not been included in print. Thereader is therefore kindly asked to refer to the respective publishing channel.

1. Ruben Sanchez, Florin Iov, Mohammed Kemal, Maria Stefan and Ras-mus Olsen, "Observability of low voltage grids: Actual DSOs challengesand research questions", 52nd International Universities Power Engineer-ing Conference (UPEC), 2017

xii

Preface

I enjoy a challenge and I always make every effort to finish what I havestarted. However, when I came to Aalborg University in 2013 for my masterstudies in Wireless Communication Systems I did not expect that I wouldbe pursuing my career towards a PhD researcher. As an engineer with abackground in Telecommunications, I always thought that I would keep onshaping my career - either as researcher or as pure engineer, in the field of 5GNew Radio communications. However, it so happened that the opportunityarises for me to continue my studies with a PhD in data analysis and visual-ization in the domain of Smart Grids. The journey was both challenging andexciting, having to refresh my memory about the different computer scienceand electrical engineering topics that I have covered throughout my previousyears of study, as well as becoming up to date to the field of Smart Grids.

Thanks to this opportunity, I got the chance of collaborating closely withmy co-supervisor, Jose Gutierrez, who has always been my support bothmorally and working-wise. Therefore, I would like to extend all my gratitudeand respect to Jose Gutierrez, who had a great contribution to the overallwork done in the PhD, as well as in my personal development as researcherand as an individual. By the same token, I would like to acknowledge thehelp and friendship of our colleague, Morten Henius, whose positive attitudealways helped me move forward with my work, even in the most difficulttimes.

Another person who deserves my utmost gratitude is Prof. Josep SoléPareta from Universitat Politècnica de Catalunya, Spain, who was my ad-viser during my stay-abroad period. His professional advice and kindnesscontributed to a significant part of my PhD research, at the same time mak-ing me feel like Barcelona is my other home.

I would also like to extend my appreciation towards my closest colleagues,Kaspar Hageman and Thomas Kobber Panum, for the fruitful discussionsand for their willingness to listen to my complaints. I do hope that we willget the chance to work together again in the future.

This PhD would not have been possible without the support of my mainsupervisor, Rasmus Løvenstein Olsen, who introduced me to the field of

xiii

Preface

Smart Grids and opened up new research possibilities for me. I am grate-ful to him for all his support and for helping me get through the challengesof managing the PhD studies. Similarly, I give thanks to Michael Lyhne fromThy-Mors Energi for his patience and contribution to this research.

Lastly, I would like to acknowledge the unconditional support of my fam-ilies - from both the Romanian and the Danish side. I have received a greatdeal of support from my parents - Radu and Florentina, as well as from mypartner, Troels Jessen, who were always there for me even when I have livedfar away from home for a very long time. Without their encouragement andpositiveness, I would not have found out how far I can get away from mycomfort zone, which leads me to think of a quote that boosts my motivation:

There’s a better way to do it - find it. - Thomas Edison

Maria StefanAalborg University, May 29, 2019

xiv

Part I

Introductory Chapters

1

Chapter 1 - Introduction

The purpose of this introductory chapter is to bring out the main topics ofthe PhD thesis. The first two sections present the overall problem definitionand motivation for the Danish power system automation. This is followed bythe corresponding research challenges and contributions, which aim to givea short overview over this work.

1.1 Electrical grids in Denmark

When H.C. Andersen wrote his adventure stories, few people had knowledge of thevalue of oil, coal and natural gas. The world was on the dawn of industrial revolutionand, not least, oil was its drive. Years after the writer’s death, fossil fuels continueto bring welfare for millions of people, however this development has its cost. Theunpleasant consequences of a warmer global climate are due to coil-based power sta-tions and oil-based transport. — Jesper Tornbjerg, 2014 [91]

A safe energy supply is the core task of electricity companies all overthe world. Danish electricity regulations state that environmentally friendlyelectricity takes over coal-based power [90]. This means, for example, that thecurrent from wind turbines must be used before the one from power stations.In 2002, 63% of Denmark’s electricity was produced by large central powerstations, 14% by outlying stations and 23% by wind turbines [53]. The vastmajority of plants are combined heating and power (CHP), producing elec-tricity and supply district heating simultaneously. A cold, windy day meansmore electricity produced and consumed - wind turbines will be generatingmore, homes will turn up the heat, causing CHP stations to generate moreelectricity [36] [13]. Such situations can be problematic, especially at nightwhen industrial production and power consumption is lower, as too muchcurrent will cause the grid to break down [55].

A Danish electricity customer had power in average 99.9% of the time in2013 [23]. However, two storms with the wind speed of a hurricane strokeon October 28th and December 5th in 2013, challenging the capabilities of

3

Chapter 1. Introduction

the energy system, as windmills shut down at wind speeds higher that 25m/s [91] [72]. Automation and monitoring of the electricity grid can ensurethat there is current flow in the cables, while the data from the modern smartmeters can be utilized to find out what is happening in the electrical network.Data availability opens the possibility for optimized grid planning, such asreplacing old installations and fixing errors [15] [96].

Fig. 1.1: Representation of the modern Danish electrical grid, from transmission to consumers.

The sources of data in the modern Danish low-voltage electrical grid varyfrom heat pumps, windmills, electrical vehicles to solar panels, which can bedepicted in Figure 1.1 [24]. It is expected that the energy consumption will in-crease with the introduction of more electrical vehicles, CHP and other typesof green energy sources. Therefore, applications are required for maximiz-ing the value incoming from distributed energy resources (DER) and efficientenergy consumption management [4] [38]. Such an application can be smartcontrol of households, apartment buildings or corporations’ energy systems,by obtaining information about energy pricing.

In the next section the background for this PhD research is introduced,given the aforementioned presentation of the low-voltage Danish electricalgrids.

1.2 Problem statement

Electricity grid operators prepare for the future as state-of-the-art technolo-gies emerge and as they are implemented to enhance efficiency and businessopportunities. The subsequent electricity grid evolution is focused towardsthe development of smart grids, capable of utilizing complex data analyticscorrelating different high volume and mixed data sources, also known as theBig Data concept. Daily workflows for grid management can be improvedvia decision support systems to ensure an affordable, reliable, secure and

4

1.2. Problem statement

sustainable electricity supply [21] [92].The process is further motivated by the national Danish regulations as

well as international political climates [58]. As current and future legisla-tion demand not only efficiency via impending requirements, but they arealso very much focused on the inclusion of renewable energy sources (RES)as part of a climate centered strategy [99]. New technology systems requireutilizing and supporting the enormous influx of smart devices and sensors.The corresponding exponential growth in data originating from these de-vices reveal anomalies, among which the most common are cable faults orvoltage magnitude threshold reached [103]. Also, importantly, the electricitygrid was originally designed solely for central operator-controlled electric-ity production with a one-way flow model. However, driven by commercialand residential energy generation via the integration of RES, primarily pho-tovoltaic and wind turbine generators, electricity grids are shifting from aunidirectional flow topology towards distributed energy generation [70] [74].

Due to the volume and variety of data [69], the Danish Distributed SystemOperators (DSOs) face operational challenges, since the current system op-erations rely solely on customers’ input to manually report common issues,such as residential power surges and outright power outages [73]. The futureproliferation of RES is expected to induce increased instability as a byprod-uct of the adaptation process towards a decentralized power generation gridarchitecture [41]. As a consequence, increased stability and reliability in thelow-voltage grid for effective grid monitoring and advanced operation be-comes harder to maintain for DSOs, in order to allow for preemptive actionsas opposed to current reactive workflow patterns.

Future grid management and operations are promising due to utiliz-ing advanced metering infrastructures (AMI) data. AMI units are installedthroughout the low-voltage grid, either as natural replacement is requiredor as direct upgrades [59]. These AMI smart meters are capable of loggingand transmitting various detailed information and in much higher resolutionthan traditional electricity meters [62], with data ranging from electricity con-sumption to specific phase voltages. Present-day AMI data is utilized for con-ventional billing purposes [3], without putting to use the full potential of themass available information and the highly increased data granularity. Also,capabilities that facilitate near-real time monitoring and automated daily op-eration with instantaneous anomaly detection can be provided through mod-ern AMI [88].

This opens up a new spectrum of possibilities for investigating means todeploy automatic monitoring and planning solutions for the Danish electricalgrids, which have been inaccessible up until now.

5


1.2.1 Hypothesis

In reference to the problem statement presented in Section 1.2, the followinghypothesis is formulated in relation to this PhD research:

It is hypothesized that efficient data processing, analysis and visualization of smartmetering data can help the DSOs in making useful decisions for future grid planning,event predictions and for automatically detecting anomalies in the grid.

Proceeding from the hypothesis, the main methods and information sys-tems to be investigated are:

• Mapping and visualizing spatio-temporal data using Geographic Infor-mation Systems (GIS);

• Use of database architectures that support large amounts of data;• Data processing and analytics for extracting relevant parameters and

knowledge out of the available data;• Design and implementation of components and interfaces for automatic

decision support systems.

1.2.2 Case study - test area at Thy-Mors Energi

This research has been carried out in relation to a real-life test area locatedin the north-west part of Jutland, Denmark, which is shown in Figure 1.2.The information has been made available by the distribution company in thearea, as part of the project - Thy-Mors Energi [6]. The area is relevant forthis study due to the presence of renewable resources at the residential level,mostly small wind turbines and PV systems.

Some anomalies have been previously detected in this part of the grid,such as over and undervoltages or imbalances in households’ power. Whilethe distribution company is responsible for deriving offline procedures tocounteract these issues, an increasing number of reported problems will re-quire more man power and time spent on error debugging procedures, thusimplying economical repercussions. Currently, the DSOs from Thy-Mors En-ergi use various software programs for investigating historical events in thepower grid, in the form of visualization and/or parameter calculation tools.For the high and medium voltage parts of the grid, the SCADA system (Su-pervisory Control and Data Acquisition) is actively used for visualization andanomaly identification. However, the low-voltage information is currentlynot fully integrated into SCADA, making the various consumer-related datamanagement procedures challenging, as the DSOs have to manually handledifferent software tools to address errors or other significant events.

With the evolution of AMI and Big Data conceptualization, more effortwill be needed towards grid planning and event predictions, rather than the

6

1.3. Research challenges and contributions

Fig. 1.2: Representation of the test area polygon, including: the primary substation, secondarysubstations (red triangles), customers (green dots) and their interconnections. The red dotsrepresent the masts in the medium-voltage grid, while the different colors depict how eachsecondary substation feeds a certain group of users.

current time-consuming manual error debugging. As a consequence, an au-tomatic decision support data-driven system is considered adequate for theDSOs’ daily operations.

1.3 Research challenges and contributions

Based on the case study of the Danish DSO presented in Section 1.2.2, anintegrated analytical and visual information platform is expected to ease thelow-voltage grid interoperability and to increase the DSOs’ efficiency in theoverall business structure, by minimizing redundant procedures. The workin this PhD study is focused on the following research challenges:

• The choice of database environment to be used and identifying theevents involved in the data processing;

• Investigating how to process and convert data to optimize the interac-tion with the end visualization system;

• Providing the users (DSOs) with adequate information in order to makeuseful decisions;

• Obtaining an automatic information-based operational system via ana-lytical methods.

The corresponding contributions are made towards resolving the defined re-

7


search challenges, by designing and implementing a data-driven system suit-able for the DSOs’ daily operations. This was achieved both from a researchand from an enterprise point of view, using the theoretical background toestablish the most suitable tools for carrying out the study in both cases. Asthere is more decision freedom from a research perspective, an enterprise-oriented solution involves more specific knowledge about the DSOs in theirworking environment, thus adapting the proposed information system ac-cordingly.

To sum up, this PhD study aims to show how a combination of differenttools and theoretical knowledge can contribute to developing an automaticdecision support system for the Danish DSOs. Particularly, it is shown thatthe results from research can be applied by distribution companies to opti-mize the usage of the distribution network resources and to minimize themanual work.

8

Chapter 2 - Theoretical Background

Traditional electricity grid monitoring and decision support are based on amultitude of different systems, demonstrating a natural additive approach totechnology adoption over time [78] [14]. As new capabilities are deemed nec-essary or advantageous, different systems are introduced , aiming to createa dedicated data-driven technology platform. As mentioned in the introduc-tion, visualizing the low-voltage electrical grid data has the potential to eval-uate and to anticipate grid anomalies, and to speed up other correspondingactions regarding grid maintenance and monitoring. Therefore, this chapterwill cover the basis of the methods utilized for achieving the proper datapresentation in this research, by covering three main topics:

• Geographic Information Systems• Database Management Systems• Data processing and analytics techniques

2.1 Geographic Information Systems - GIS

Geographic Information Systems (GIS) are, as the name indicates, a com-bination of two different disciplines: geography and information systems[61] [18] [17]. Geography is the science dealing with the physical, biologi-cal and cultural features of the Earth, in other words, data associated to alocation. Information systems are, generally defined, as a set of componentsthat work together to achieve a common goal, by utilizing the data charac-teristics/attributes attached to the location. The components involve: data,hardware and software equipment, humans, operational procedures and sub-systems for data management, with the main goal of transforming the datainto valuable information, knowledge and wisdom [7] [33].

The DIKW diagram illustrated in Figure 2.1 shows the structural andfunctional relationships between data, information, knowledge and wisdom.By undergoing the transition from raw - meaning - context, data brings valueto the human interpretation by helping decrease the computational complex-ity at more advanced stages in the process [26] [66].

9

Chapter 2. Theoretical Background

Fig. 2.1: DIKW data transformation diagram [10]

Considering that data is the starting point in GIS, there are three maindata types [30]:

• Spatial (vector) data: features represented as points, lines and polygons,as previously shown in Figure 1.2;

• Attribute (tabular) data: qualitative and quantitative characteristics ofthe spatial entities;

• Raster data: landscape represented as a rectangular matrix of squarecells, useful for elevation, terrain, slope and risk analysis, etc.

This PhD research was based on the electrical grid spatial and attributedata in GIS [95], particularly for extracting labels for time series measure-ments, as it has been done in Paper E. Moreover, the benefits of the spatialfeatures with respect to improving the DSOs’ operational procedures wereevaluated in the studies performed in Papers C and D.

GIS usually have an integrated database management system [89] [77],where the data model is represented by the different objects in the spatialdatabase and the relationships among them. Each feature on the map can becharacterized by attribute data, which is typically manipulated in relationaldatabases by means of queries [48]. Three of the most common types ofdatabase systems will be subsequently presented in the following section.

10

2.2. Database Management Systems

2.2 Database Management Systems

This survey concentrates on the three primary Database Management Sys-tem (DBMS) categories and will present a high level introduction to thesespecific database types and highlight the technological characteristics, advan-tages and inherent shortcomings. Lastly, generic database evaluation criteriaare highlighted as a foundation for requirement specification. The survey isbased on the following sources: [94] [51] [20] [22] [87] [16].

A database schema is the design blue print of how the DBMS is con-structed. The schema defines the basic structures on both the logical andphysical level, providing a descriptive detail of the how data is organized,the corresponding relational structures including how every constraint is ap-plied, as well as storage definitions for all database elements. Thus, theschema plays a paramount role in determining application suitability andflexibility as well as both data and transaction integrity parameters. DBMSscalability identifies the abilities of the system to be upgraded and expanded,and hereby determining both present and future performance and capacitycapabilities.

The primary DBMS categories chosen for this study are:

1. Relational DBMS: uniquely based on proven mathematical foundation,specifically by Georg Cantor’s Set Theory [51] and Relational Theory byEdgar Codd [94], it guarantees a high level of stability and robustness.With a more than 30 years proven track record, the RDBMS is the in-dustry standard, commonly utilized as a comparison baseline, and it ischaracterized by offering sufficient data storage, protection and accesscapabilities. Also, its performance is reliable, making it adaptable forbusiness intelligence use cases;

2. NoSQL (Not Only SQL) DBMS: broad descriptor for next generationdatabase systems, typically characterized by being open source, non-relational, distributed and horizontal scaling. Polyglot Persistence [20]is the NoSQL jargon for selection between the different data modelsof the NoSQL DBMS categories matching use case and application re-quirements. Therefore, NoSQL databases provide freedom of choice tomatch a custom architecture specifically to the application and problemset [22], by reducing the DBMS complexity via Polyglot Persistence;

3. In-memory DBMS: An in-memory database (IMDB) is generally a RDBMSusing RAM instead of traditional disk storage [22] [87], typically pro-viding full SQL support. It can substitute existing RDBMS with onlyminor adaptations if not seamlessly. The vast majority of IMDBs arebased on a vertically scalable Symmetrical Processing (SMP) architec-ture, with limiting large-scale scalability and throughput, due to the

11


inherent inability of SQL join features to operate efficiently in a dis-tributed environment [16]. To take advantage of new middleware in-memory capabilities, IMDBs require keeping current access applica-tions functional via continued SQL processing operations but with sig-nificant changes to the existing database. The cost model for IMDBs ishence primarily dependent on single server architecture pricing.

The current Thy-Mors data model is implemented as a relational DBMS.Considering the current implementation is based on Microsoft SQL Server2014, which introduced Memory-Optimized Tables and Natively-CompiledStored Procedures In-memory features [5] [97], a continuation with relationalDBMS was considered favorable. Enhancement contrary to a complete tech-nology switch allows integration with the current data model implementa-tions as well as encourages taking advantage of the already existing in-houseexpertise and knowledge base. The In-memory Online Transaction Process-ing (OLTP) [16] [5] support of MS SQL Server enables adequate performanceenhancements and legacy capabilities for both near-real time (dynamic) andhistorical (static) data types. The workload areas which benefit the mostfrom In-memory OLTP technology [22] include high data rate insert ratewith smart metering as the primary example, read performance and scale,computer heavy data processing and low latency workload categories.

At the reseach level, an implementation of the relational DBMS based onMS SQL Server meets the main requirement for managing large historicaldata sets and is considered suitable for this work. Additionally, given theaim of visualizing spatio-temporal data, the PostgreSQL DBMS is the mostsuitable choice. This is integrated with the PostGIS spatial database exten-der which provides support for different spatial capabilities to the existingdatabase, such as geometrical data types and geocoding in-built functions.

For experimental purposes, MSSQL’s in-memory data storage was usedin Paper C to simulate data streams. Due to the requirements for spatio-temporal data visualization, the research conducted with real-life measure-ments and grid topology in Papers D and E was carried out using Post-greSQL.

2.2.1 Database Privacy Features

Electricity meters track energy use which can violate the right to privacy andprotection of personal data [44]. The customers’ energy consumption can re-veal information about the number of people in a household, daily routineand usage of appliances. Sometimes, the data can reveal particularly sen-sitive information, such as criminal offenses. Protection of personal data istherefore regulated in detail by the General Data Privacy Regulation (GDPR)[9]. This project is carried out taking into consideration the privacy regula-tions regarding personal data. In this context, there are two types of datasets

12

2.2. Database Management Systems

related to individuals: GIS data (location of the meters - addresses, as X andY coordinates) and metering data (value of any relevant parameter measuredby the meters).

For the research purposes, it is essential to keep these two datasets sep-arated and uncorrelated in order to preserve the privacy rights of the Thy-Mors customers in the test area. The concept is illustrated in Figure 2.2. The

Fig. 2.2: Separation of data tables and elements to secure privacy and anonymity of datacollected

data is anonymized by removing all references to physical meters such as IDand giving each data point a key, each of the meters within the test area hav-ing their own unique key. The keys can only be decoded using the PrivateKeys table, necessary to relate a key to a specific address. Two copies of thetable are at:

• The meter distributor (Kamstrup) for data anonymization;• Thy-Mors to obtain the GIS_ID, which is useful for the visualization in

the GIS environment.

By legal authorities, this table is only used when strictly necessary, inaccordance with applicable data protection and privacy regulations:

• for preventing or respond to cyberthreats or cybersecurity incidents[67];

• for preventing other criminal actions or for preventing actions by theconsumers which may cause a risk to the functionality of the grid.

In reference to the requirements of data processing, the next section willfocus on different processing and analytical techniques that build upon theprerequisite of obtaining valuable knowledge from the data.

13


2.3 Data processing and analytics techniques

Data processing is commonly categorized as either batch or stream process-ing. Where batch processing constitutes classic periodic data processing, andby inverting the paradigm, stream processing implements persistent dataflows, queries, analytics and application logic. Batch data sets and workloadsare characterized by having a finite data source, representing static at-rest in-formation. Inversely, stream data sets and workloads have a theoreticallyinfinite data structure, frequently described as event time series in-motioninformation.

2.3.1 Data processing

The following sections present the main characteristics of the two commonprocessing paradigms - batch and stream.

2.3.1.1 Batch Processing

This section is based on the following sources: [100] [101] [32] [1] [68].Batch processing (Figure 2.3a) represents the common case processing,

with management and computations over all or most of a data set. Theprocessing is run off-line on persistent data blocks frequently according topredefined periodic schedules. Traditionally, batch processing is focused onthroughput and complexity performance, designed to manage large data vol-umes while executing computational intensive algorithms. Latency is consid-ered a secondary objective, typically measured in minutes to hours.

The main characteristics of batch processing are:

• Static finite data at-rest data sets[32];• Volume-centric processing [32];• High throughput performance [101];• Scheduled offline processing;• Periodic recalculation over all/most data;• Data persistence [101];• Data scope in the range of minutes to years.

2.3.1.2 Stream Processing

The technical content of this section is based on the following sources: [54][86] [100] [19] [25].

The stream computing data processing paradigm (Figure 2.3b) enablesmanagement of continuously generated data, falling in the category of Com-plex Event Processing (CEP) as an element of Big Data technology. The pri-mary purpose of stream processing is to provide exceptionally low latency

14

2.3. Data processing and analytics techniques

(a) Representation of batch processing [19]

(b) Representation of stream processing [19]

Fig. 2.3: Data flow in batch and stream processing techniques.

velocities independent of high volume mass storage. Stream processing is atechnology introducing real-time or near-real-time, which, depending on theenvironment, is defined as microseconds to several days. Processing perfor-mance ensures uninterrupted information streams as well as direct interac-tion with non persistent data before any potential storage procedures. Streamprocessing unifies analytics and applications in a single common architecture,introducing direct analysis result integration into applications for automaticand instantaneous action. Providing continuous query capability is essen-tial for sensor applications, web events, machine and application logs, socialdata.

The main characteristics of stream processing are:

• Low latency processing [86];• Instantaneous application and analytics reaction to input events;• Management of individual records or micro batches;• Continuous and unbound event driven data sets;• Periodic recalculation over all/most data [100];• Decentralization and decoupling of infrastructure [100].

15


2.3.2 Data Analytics

In this section the main data analytics concepts are introduced. These con-cepts serve the purpose of converting processed data into relevant informa-tion, which subsequently is prepared for either additional analysis or directlyorganized for presentation and visualization objectives [12] [35].

Data analytics are organized into two main categories:

1. Historical: analysis based on the past; data-at-rest corresponds to batchdata processing [42]. Historical data analytics provide insight by uncov-ering data patterns and trends, allowing for a concise presentation oflarge data sets. By utilizing different algorithms for reducing complexdata sets [56] [8], event forecasting is also possible [76]. The drawbacksof historical analytics are related to their limited reactions to past eventsand update intervals resulting from the batch processing.

2. Real-time/streaming: analysis based on the present; data-in-motionequals stream data processing [65] [46]. Real-time data analytics makeit possible to enhance the reaction time for decision makers via clarityon current unfolding events. As a result, correlations between multipleand diverse data sources can be detected [98] [10], at the same timeopening the possibility for predicting imminent events or failures (i.e.fraud detection). Despite the explicit advantages, real-time analytics arevery much platform and hardware dependent [34], causing potential in-correct analysis and decisions. At organizational level, adapting to newwork patterns to take advantage of the continuous flow of informationis also challenging.

Thus, the choice of suitable analysis techniques is based both on the applica-tion requirements and on the implementation flexibility.

In the process of knowledge and information discovery, different typesof analytics can provide various levels of in-depth knowledge of the data,depending on the available data set and the application requirements. This isachieved through the four traditional types of analytics - descriptive, diagnostic,predictive and prescriptive [35] [52] [31], as shown in Figure 2.4. The trade-offbetween the obtained value and the implementation difficulty increases forpredictive and prescriptive analytics, as they open the possibility for processoptimization, as well as for better understanding and exploring the valueextracted from the data.

In this PhD research the focus in mainly on characterizing the behaviorof low-voltage grid consumers using basic statistical studies (descriptive anddiagnostic). Furthermore, the study in Paper E is solely based on the availablestatic data, in the form of GIS grid topology and historical time-series energyconsumption measurements. A step forward is taken in the analysis with

16

2.4. Decisions

Fig. 2.4: Representation of descriptive, diagnostic, predictive and prescriptive analytics typesand the corresponding questions to which they answer [35].

respect to predictive analysis by applying forecasting models to the availabledata set and by prescribing recommendations for human assessment.

2.4 Decisions

The three topics covered in this chapter - GIS, DBMS, data processing andanalysis, were meant to give an overview of the main tools used in this PhDresearch. The study was conducted both from a practical and from a scientificpoint of view, leading to two main research tracks:

• User experience (UX) studiesThe DSOs were the central part of the UX studies. The purpose wasto evaluate the DSOs’ daily working procedures and to identify somescenarios where they are constrained from operating the grid efficientlydue to manual error debugging. The DSOs’ feedback was useful fordeciding which analytical techniques are most suitable for designingand implementing an automated information system for monitoring,planning and prediction (Paper D).

• Developing an information system for the low-voltage electrical gridThis track was oriented towards the information system development,based on the previous UX studies. The system comprises of relationalDBMS for data storage and feature extraction (pre-processing) tech-niques, such as filtering and selection. The features can be used forstatistical analysis, data mining and forecasting, as it was done in Pa-per E. In terms of visualization tools, WebGIS was used for data pre-sentation in Papers C and D, while QGIS was employed for both datapresentation and analysis (Paper E).

17


18

Chapter 3 - Research tracks

The topic of this chapter is related to this work’s research tracks, which aredepicted on the roadmap in Figure 3.1. Considering that the end user of adata visualization and analysis system are the DSOs, their requirements andneeds come first when designing a dedicated application. User experiencestudies focused on the DSOs workflows are first presented. Secondly, theprior UX knowledge contributes to deriving a data-driven solution for thecurrent electrical grid system.

Fig. 3.1: Roadmap of the PhD research.

19

Chapter 3. Research tracks

3.1 User experience studies (UX) - Distributed Sys-tem Operators (DSOs)

In this section, the approach for performing UX studies is covered, wherethe users are the DSOs from Thy-Mors Energi (case study in Section 1.2.2).The UX study was performed based on on-site interviews. The users’ dailywork routine is analyzed via the ’Day-in-the-Life’ model and user profiles arecreated for the involved DSOs.

3.1.1 ’Day-in-the-Life’ Model

The purpose of utilizing the Day-in-the-Life Model [40] [39] in this work isfor identifying where DSOs operations can be improved time-wise. A generaluser profile, as represented in Figure 3.2, can help in understanding a DSO’sroutine in a normal working day. The Day-in-the-Life model brings togetherthe overall structure of how work fits into the user’s day and how this issupported by different mobile and stationary devices. The focus of this modelis on the different places, timings and platforms that together contribute toactivities getting done.

Fig. 3.2: Day-in-the-Life Model applied to the Danish DSOs, showing scenarios for an operatoron a shift at home, during transportation, at the TME headquarters and smaller offices.

Three main activities and spatial contexts are identified during the DSOs’day in their weekly shift: at the work place, at home doing everyday activi-ties (on call) and at home during an ongoing event. An on call DSO has tolive inside the distribution area and be able to act upon an event within 15

20

3.1. User experience studies (UX) - Distributed System Operators (DSOs)

minutes from its signaling. This limitation in the DSO’s daily life also impliesinterrupting everyday activities at any time of the day. The DSO might alsoneed to call another colleague for knowledge sharing and advice, meaningthat this limitation is general among the DSOs. At the work place, the DSOneeds to be able to interrupt a routine activity and prioritize an importantevent. The DSOs communicate with both internal and external actors - cus-tomers (private households and companies), technicians, contractors, otherDSOs and departments, through an error messaging system. One artifact hereis an SMS message to the customer informing them about a possible poweroutage. Having smaller sub-offices at several transformer stations in the dis-tribution area makes it possible for the DSOs to drive to the nearest office incase they are more than 15 minutes away from the official work place.

The user analysis during a working day can help establishing which ofthe work processes are most time consuming, so that they can be automatedfor an optimum operation and planning of the electrical grid.

For example, acting on a calling customer can involve less time if theexact event and location (address) of the customer are signaled through visualalarms. The demand of being able to work on the error within 15 minutes, aswell as to being alert at all times, impact the timeliness of these alarms.

In order to achieve the automatic fault detection in the electrical grid, userexperience studies have been made in order to establish what data and howthe end user (DSO) wants to visualize it. The steps of these studies are:

• Establishing different user profiles from the control center – electrician,electrical engineer;

• Establishing different scenarios where errors are reported;• Defining sequence models for the chosen scenarios and the current pro-

cedures to address these errors;• Identifying the main themes to be addressed as part of the final visual-

ization prototype;• Designing a prototype that would bring an automated solution to over-

come some of the challenges reported by the evaluated users;• Validate the solution by involving the different actors in the value chain

(vendors and DSOs).

3.1.2 User profiles

User profiles are seen as part of the consolidation in relation to contextualdesign models, in which the focus is directed on the DSOs in their workingenvironment, therefore they are the central part of the design process. Thepurpose of these profiles is to ensure that the system design will benefit theusers’ workflow and, as a consequence, it will be more directed to the DSOs.For this study, the following user profiles have been identified in Table 3.1.

21


Table 3.1: Distributed System Operator profiles based on on-site interviews.

Education Job title Timein thecompany

Area of responsibility Competencies

1 Electrician Systemoperator

3.5 years Planning, developing,building and maintain-ing transformer stations

Knowledge aboutbuilding transformerstations

2 Electricalengineer

Systemoperator

6 months Reviewing the 10/0.4 kVdistribution stations andreporting of errors

Engineering back-ground

3 Electrician Systemoperator

32 years Filing reports to theenergy consumptionagency, when the elec-tricians have solved anerror;Measuring the electricitygrid

Experienced in thefield and knowledge-able of the companyworking structure

3.2 Information system for the low-voltage electri-cal grid

The second research track is focused on designing a strategic data-driven in-formation system for low-voltage electrical grids, aiming to combine knowl-edge from both the UX studies and the processing of measurement data.

Measurement data from AMI in Denmark is nowadays typically done ev-ery 15 minutes and only for billing purposes. The aim of this research isto provide ways to process and analyze the incoming smart meter measure-ments with various types of readings containing different electrical parame-ters. The end goal is to provide a meaningful data display to the end users(DSOs), in order to obtain an overview over the current and the historicalprocesses that take place in the power grid.

An overview of the proposed data system is shown in Figure 3.3. The tworesearch tracks are illustrated as main inputs to the structure of the wholeinformation system. The UX input is used as starting point for designingthe data visualization platform. The incoming metering data from the AMInetwork is passed on to the DBMS for storage and processing, while the pro-cessing output can be utilized both for statistical analysis and prediction, andfor feature extraction in relation to the visualization. Furthermore, depend-ing on the result, feature extraction has an impact on what is most relevantfor the application design.

Historical data is relevant for understanding different consumers’ behav-ior by evaluating their consumption patterns, through classification/clustering

22

3.2. Information system for the low-voltage electrical grid

Fig. 3.3: Data flow for the proposed information system for low-voltage electrical grids.

methods, for example, consumers with or without installed RES. Consump-tion patterns also depend on environmental factors such as time of day(morning, afternoon, evening) or season. Consequently, the DSOs can usethis information for optimizing or planning ahead future updates in thesmart grid infrastructure, as well as for fraud and outage detection.

A meaningful information display is based on the DSOs requirementsand needs, thus impacting which type of processing techniques should beutilized to extract certain features of the data. The aim is to discover whichinformation is helpful for the DSOs to monitor the grid status and how topresent this in an integrated data system.

3.2.1 Main visualization themes as identified by the DSOs

The following visualization themes are meant to contribute to the design ofthe visualization system, according to the user analysis and the proposeddata flow.

1. Geographical low-voltage grid map

The purpose is to be able to see the values from a smart meter directlyon the map, when clicking on a particular data point of interest. Even-tually, the system should lead its user throughout the troubleshoot-ing process by identifying all the interconnections between the high,medium and low parts of the power grid. Therefore, map interactivityand sorting mechanisms are important in the visual design of the map,as overloading of measurement data points becomes an issue with the

23


low-voltage nodes density. In this case, feature extraction refers to fil-tering and/or selecting of the significant areas.

2. Historical data visualization

Historical data display comes as an extension of the map interactiv-ity function and it is mostly useful for planning grid reinforcements inthe areas which are prone to problems. Currently the ID of the me-ters which issued anomalies is manually found by the DSOs [88] andtheir corresponding nodal measurements are also manually extractedfrom the DBMS. This activity is redundant, time consuming and caneasily result in errors, such as incorrect or incomplete measured values.Due to this, data labeling can contribute to easily match nodal mea-surements with their corresponding GIS location and to easily extracthistorical measurements belonging to a certain consumer.

3. Alarm visualization

Alarm display is a subcase of the raw metering data visualization. Anykind of anomaly in the grid (over/undervoltage, flickering) should bealso visible among the incoming raw data. Map layer filtering andthe interactivity function make it possible to detect the extent of theanomalies in the grid, whether they are related to a specific substationor dispersed across a large area.

The design of the new software solution has to fit the users’ life and theirdifferent activities throughout the day, as shown in Figure 3.2. The pur-pose is to facilitate the DSOs’ job in solving different tasks and debuggingerrors when the required functionalities are integrated in one system. Dif-ferent types of visualization techniques are supposed to be alternative waysto the current manual searches of customer information and to allow for thepossibility of cross-integrating multiple software tools, which are currentlyutilized at the enterprise level.

The system assessment is done in terms of Capital and Operating Expen-ditures - CapEx and OpEx, as depicted in Figure 3.4. Although no market re-search was done in relation to this study, the plot conceptually illustrates theeconomic feasibility of developing and implementing such an information-based system. The research done in this thesis indicates that there is a clearbenefit (OpEx) reduction when implementing this model, by reducing thetime and human resources required to solve incidents in the electrical grid.In each specific concrete case (different DSOs), the relation between OpExreduction and the cost of developing and implementing the proposed sys-tem would indicate the economic feasibility of such migration in terms ofReturn of Investment (RoI). Normally, the acceptable RoI for software-basedsolutions is between one and three years [64] [45]. However, considering that

24

3.2. Information system for the low-voltage electrical grid

Fig. 3.4: Accumulated OpEx analysis for automation system integration at enterprise level.

electricity meters have a lifetime of up to 20 years [47] and that the proposedsolution is fully cloud-based, the DSOs may consider longer a RoI as accept-able.

The graph in Figure 3.4 is related to the automatic decision-based visual-ization system presented in Paper D.

The two research tracks depicted in this chapter serve as a link towardsthe contributions of the PhD study, which is the topic of the next chapter. Asthe contributions are in the form of a collection of papers, the scientific andpractical contributions will be detailed by referring to the specific articles.

25


26

Chapter 4 - Contributions

The combination of scientific and practical contributions is the strong pointof this PhD research, presented in this chapter. The main body of the thesiscomprises the papers A to E, which follow mainly the techniques for de-signing and implementing a strategic information system for the domain ofelectrical grids.

Following the description of the research tracks in Chapter 3, the paperswill be categorized according to their scientific (research-oriented) and prac-tical (enterprise-oriented) nature.

4.1 Scientific contributions

The main scientific contribution is to investigate and experiment on the pos-sibilities for designing and implementing a data-driven solution (informa-tion system) for the current electrical grid infrastructure. By introducingintelligence in the current power system where planning and monitoring op-erations are broadly manual, the contribution is made towards the develop-ment of the so-called "smart grids".

4.1.1 Visualization systems

Paper A is an initial survey of visualization techniques for electrical gridsmart metering data. The study focuses on the different techniques that con-tribute to obtaining a visualization system for monitoring and planning pur-poses, taking into consideration the real-time and historical data types asmain use cases. As a part of the system design, the uses of the CommonInformation Model (CIM) in both research and industry fields are presented,given that CIM is currently utilized for defining the standard data model inelectrical grids (ENTSO-E, Statnett). Due to the modern advances in the AMInetworks and the variety in data issued by the small producers in the low-voltage grids, the paradigm of Big Data is introduced to further emphasizethe need for efficient data analytics and visualization. Lastly, a survey of dif-ferent visualization desktop tools justifies the choice of using Quantum GIS

27

Chapter 4. Contributions

as data display tool.The survey was made based on the initial project requirements for real-

time and large volume (Big Data) historical data visualization. One conclud-ing remark is related to the benefits and drawbacks of the different visualiza-tion software tools in the study, which is presented in Table 4.1. It has been

Table 4.1: Comparison of different desktop GIS tools.

Desktop GISSoftware

Advantages Disadvantages

ArcGIS Receive real-time data from awide variety of sources (GeoEventProcessor extension)

Proprietary (expensive license)

Quantum GIS(QGIS)

Open source, integration withother open source tools (GRASS,gvSIG), fast processing speed

Difficult to export files and toinsert map elements

MapInfo Track frequently updated datausing (animation layer add in)

MapBasic implementation (not anaccessible language)

GRASS Modules for data management,spatial modelling and visualiza-tion

Inconvenient user interface, slowprocessing speed

gvSIG User-friendly GUI, fast loading oflarge data volumes

Limited compatibility with opensource tools (GRASS)

Maptitude Good for basic GIS mapping pur-poses

Little support for advanced GISprocessing

decided that QGIS fits best with the requirement for historical data man-agement, which is why it has been utilized for modeling and evaluating theelectrical grid topology, as well as for extracting labels related to the con-sumption measurements.

A Web-based GIS solution was chosen as proof of concept, which is pre-sented in Paper C. This paper proposes an implementation of a near-real-timemonitoring system for the DSOs. The study was partly done in collaborationwith the DSOs from Thy-Mors Energi who helped identifying the case ofhousehold power outage. The working procedures for debugging this casehave been defined based on interviews with the involved DSOs, who alsodemonstrated the use of different software tools within the company. Theresearch-oriented solution in Paper C is done by emulating a real-life AMI(Virtual AMI) and by creating real-time voltage measurements through a re-play functionality. Thus, some of the theoretical knowledge about stream pro-cessing was included as part of the scientific research. The implementation

28

4.1. Scientific contributions

of a visualization-based monitoring system clearly reveals the advantages ofspatial and situation awareness, leading to faster recovery and improved ser-vice to end consumers/prosumers. Also, the time-consuming operations arereduced, thus eliminating the risk of incorrect decisions during debugging.

Fig. 4.1: Stages of development for the visualization system.

Figure 4.1 depicts the stages of development for the proposed visualiza-tion system. The aim for the final solution is to integrate the visual elementswith their corresponding data analytics. Therefore, the appropriate choice fora data-driven operational system is QGIS, which was used for the analyticalpurpose in Paper E.

4.1.2 Data analytics methods

A more in-depth study of analytical methods is presented in Paper B, depart-ing from the theoretical basis described in Section 2.3.2. The main motivationfor this study was to identify the potential processes in an information systemfor low-voltage electrical grids and their suitable analytic techniques. There-fore, the use case of low-voltage grid observability is chosen. In this case,observability refers to making the knowledge available for the DSOs, eitherthrough explicit visualization or through mathematical or statistical models.

At the time Paper B was published, the design of the information sys-tem was still under definition, therefore the aim of this work is to propose asystem design for the grid observability case. The proposal takes into consid-eration both streaming and historical data types, by evaluating the pros andcons offered by the different analytics. The concept of distributed systemstate estimation (DSSE [60]) is introduced as an analytic method for predict-ing values of missing or incorrect data. The flow of data in automatic andinteractive events aims to justify the timing uncertainty in such a system,thus introducing the notion of near-real-time data, and the potential need forparallel in-memory and disk processing techniques.

Significant influence in the data analysis domain was brought by havingaccess to real-life electrical grid data, in the form of GIS grid topology andtime-series measurements, as explained in Section 3.2. The analysis of thisdata is presented in Paper E. The motivation for the study is closely relatedto the work done in Papers C and D, from which it was concluded thatthe behavior of small produces has a large impact on the DSOs operational

29


workflows.Based on this knowledge, Paper E depicts the study of the electrical grid

small producers’ behavior, with the focus on data accuracy analysis. Therelevance of this study is strongly related to the real-life data flow, by de-termining unaccounted for information links. As the purpose is to evaluatewhat can be extracted from the available data, clustering methods have beenapplied to classify the different consumers by their energy consumption. Theclusters were then used to test the accuracy of an ARIMA predictor.

The study in Paper E aims to evaluate the different contributions broughtto the DSOs by the data analysis with respect to anomaly detection, powerbalancing, planning and monitoring operations. The future potential con-cerns for data inaccuracy originate from data scalability with the increasingdevelopment of AMIs, electrical vehicle mobility and the requirements formaintaining customers’ privacy. Therefore, the work can be seen as a base-line for future development of information systems for power grids, as it willbe further explained in Section 4.3.

4.1.3 Overall scientific contribution

The overall scientific contribution of the PhD study was brought in terms ofvisualization and data analytics systems for low-voltage electrical grids. Con-currently, these two support the design and implementation of a decision-based information system, which aims to take over the redundant grid op-eration tasks by the DSOs. In this way, more time can be dedicated towardsplanning, reinforcing and forecasting potential events that may induce insta-bility in the power grid, thereby affecting the power quality at the customerend.

An initial version of such an information system is presented in the nextsection as part of the practical/on-site contributions towards the involveddistribution company.

4.2 Practical contributions

Departing from the study made in Paper C, Paper D is an extension of thatstudy. The focus of Paper D has been to perform a more in-depth analysisof the DSOs in their working environment through UX studies, as previouslypresented in Section 3.1. Besides the power outage case in Paper C, twomore sequence models have been defined for the cases: light flickering athousehold level and customer notification of potential errors in the mediumand high voltage levels.

Therefore, the implications of the study in Paper D are practical, orientedtowards the needs and requirements of the distribution company.

30

4.2. Practical contributions

4.2.1 Design of the visualization prototype

Fig. 4.2: Web-application map tab, including filters, customers, cable boxes and their intercon-nections

The design process is based on the user stories/sequence models iden-tified by the DSOs and presented in Papers C and D. Spatial awareness isthe key in designing this prototype, as the main visualization themes iden-tified in Section 3.2.1 are all interlinked via a GIS map. Figure 4.3 showsthe main items contained by the designed prototype: displaying all meteringdata points, statistics about households, cable boxes and substations, listingof the current reported errors, warnings and other kinds of alerts, and thelow-voltage grid overview.

Fig. 4.3: Focus areas in the overall structure of the visualization system.

31


The challenge lies in achieving the most relevant knowledge from thevisual perception, thus the focus is on the front-end design. It is expectedthat the large number of elements to be displayed leads to an overcrowdingof the GIS map, therefore a visual event detection will not be trivial.

The contribution comes from the advantages of GIS: organizing data intolayers and having the possibility to select/deselect a layer of interest. Thisfeature was implemented as a filtering function, as well as the errors aresorted out and displayed based on their severity. The interactive map al-lows to choose the different components (household, cable box, transformerstation, cable, renewable energy or customer type), as depicted by Figure 4.2.

In terms of workflow analysis, similarly to Paper C, it is shown that thedifferent debugging processes are time-wise minimized and optimized sothat more knowledge can be acquired from the existing system, due to the in-tegration of a data-driven approach. Moreover, prediction methods are madepossible through data integration, bringing the traditional data processingand analysis methods towards a prescriptive information system.

At the business level, the advantages materialize in the form of CapitalExpenditures (CapEx) and Operating Expenses (OpEx), as it was previouslyshown in Figure 3.4.

4.3 Baseline for future development

This PhD’s contributions in terms of data-driven analysis and visualizationsystems for low-voltage electrical grids opens up possibilities for future de-velopment. Particularly scalable solutions are of interest in the domain ofsmart grids, since it is expected that the volume and variety will increasewith the advancements in the different utilized technologies. Some of thepotential development tracks are identified as follows:

1. Hybrid processing techniques

The data processing domain is evolving towards an intermix of streamand batch paradigms, purposely designing frameworks fully integrat-ing both forms of processing introducing hybrid architectures and gen-uine hybrid processing engines [2] [63]. The development of hybridprocessing engines is primarily fueled by stream technology maturingto a level capable of high performance as well as computational com-plexity [29] [79] [28]. As stream processors continue to evolve capa-bilities such as complete fault tolerance, fault recovery and producingaccurate results, while delivering high performance computing, there isno longer an incentive to make a choice between fast or accurate results[75].

32

4.3. Baseline for future development

As the data system described in Section 3.2 is expected to evolve to-wards both stream and batch data types, hybrid processing techniquesare architectures should be considered. Apache Spark and ApacheFlink are two hybrid data processing frameworks [27] [84] [11]. ApacheSpark [85] is the next generation framework for batch processing thatalso includes streaming capabilities. Speed (due to in-memory compu-tation) and versatility (standalone cluster) are the pros of Apache Spark,while its limitations are due to high latency when processing large datastreams and the high cost of running it in RAM. The Apache Flinkopen-source framework [37] is oriented towards distributed stream pro-cessing. It has advantages in terms of accuracy (delayed data), statefuland high throughput performance when scaling to thousands of nodes[80] [43]. However, its drawback is that Flink is not so widely deployedyet. An enterprise-oriented deployment of Flink for large scale net-works would definitely bring a contribution towards stream processingin smart grids.

2. Focus on system optimization - predict and prescribe

With the evolution of the data processing methods, it is expected thatthere will be higher requirements in terms of their corresponding real-time and historical analytics. The current power system is mostly basedon the information extraction. However, due to scalability, data gran-ularity and velocity challenges, the future analytical techniques will bemore focused towards system optimization, previously referred to aspredictions and prescriptions in Figure 2.4.

Overall processing and analysis performance is a question of data avail-ability and velocity, typically ranging from near instantly available toonce or twice a day. The question is all about when data is processedand analyzed, which is determined by data and service time-degradationdependencies as real-time data expires continuously. Two of the mostfrequently referenced architecture frameworks implemented to supporta combination of both batch and streaming workloads are Lambda andKappa. Lambda [57] [71] [68] [83] is the target recommendation for thereal-time data processing architecture, where batching is used as theprimary processing method and streams are used to supplement earlybut unrefined results. Kappa [102] [81] may be considered for experi-mental and research purposes, in line with the Apache Flink framework[93]. The Kappa architecture contains only one stream processing layermaking it easier to maintain due to its lower implementation complex-ity, as opposed to Lambda having separate processing layers for streamand batch.

33


3. Topology analysis using Graph Neural Networks

Graph Neural Networks [82] is a concept which is currently used mostlyin the domains of biology, chemistry and computer vision [104] [49][50]. One possible research direction is to investigate their potential inthe domain of electrical grids. By making use of the available mediumand low-voltage grid topology, graph neural networks can help identi-fying how different topologies affect the losses in the power grid lines- the difference between measured powers at medium and low-voltagelevels gives an indication of losses in the lines. Therefore, the start-ing point would be to first analyze the different topologies at medium-voltage and then extend the research to the low-voltage level.

Case studies can be done by evaluating the secondary substations withthe highest number of connected consumers and meshed connections.The contribution of this study consists of proposing different ways oforganizing the power flow in the system and thus performing an anal-ysis of the power system reliability.

34

Chapter 5 - Conclusion

The overall theme of this PhD study was automation of Smart Grid opera-tions through spatio-temporal data-driven systems. As the Danish climateregulations aim to introduce 100% green energy by year 2050, the increasingnumber of small producers in the low-voltage electrical grid challenges theDSOs daily operations for delivering a reliable electricity supply. Therefore,the research in this PhD was focused on investigating how to design and de-velop an automatic decision-support system for the DSOs via efficient dataprocessing, analysis and visualization of smart metering data. To achievethis, the scope of this work was focused on exploring means to develop asuitable information system for the low-voltage electrical grid, based on userexperience studies. The main contribution was realized with the design andimplementation of a data-driven system for the DSOs’ daily tasks, usingthe existing operational system as baseline for the research.

The outcome has been evaluated both from a scientific and from a prac-tical implementation point of view. On the scientific level, the performedexperiments regarding system interfacing and and data analysis were exam-ined from both near-real-time and historical data perspective. These demon-strate how efficient data analytics as part of the integrated system opens upfor a wider spectrum of opportunities for the DSOs, than with the existingsystem. Analytical techniques such as statistics, forecasting and visualizationhave shown to bring deep insight into the consumption behavior of the smallproducers, enabling the DSOs to make documented decisions about the fu-ture grid changes. In other words, smartness in the current electrical gridoperation is brought by evolving towards prescriptive types of actions.

An important part in this research has been data visualization, designedas the final front-end solution. Thanks to input provided by the DSOs fromThy-Mors Energi regarding their daily operations, it was possible to per-form user experience studies. These studies have shown that manual errortroubleshooting restraints the DSOs from committing to more advanced op-erational tasks. Thereby, the user studies have had a big impact in the designand implementation of the visualization solution, which proved to minimizethe current error debugging time. At the same time, the advantages of inte-

35

Chapter 5. Conclusion

grating data analytics into one system aid to improve grid monitoring, plan-ning and prediction of events. Detecting and predicting errors automaticallybeforehand thereby upgrades the current manual debugging process.

The proof-of-concept analysis the scientific level provided means to ex-periment interfacing between the different chosen tools - GIS, DBMS andanalytics, in order to develop the desired data-driven system. A step forwardwas taken with respect to the practical contribution of the work, by creating avisualization prototype dedicated for the DSOs involved in the study. Fromthis, it was concluded that the spatial and situation awareness provided bythe GIS capabilities can help the DSOs cut down on some of their repetitiveworking procedures when addressing different kinds of errors.

Considering the current climate regulations and the impact of renewableson the energy supply’s reliability, this PhD work has overall contributedto demonstrating the advantages of automatic spatio-temporal informationsystem integration for the low-voltage grid operation. It is shown that eventhough the initial cost of operating is higher than operating the current sys-tem, more functionality help the DSOs to prepare for the future advances inthe electrical grid. In the long run, the return of investment will be collec-tively acquired from time saving operations, increased operation efficiencydue to the new system’s intelligence and a flatter trend in the OpEx.

By focusing on the future smart grid capabilities, there will be less invest-ments spent on adapting the already-existing operational system to futurechallenging use cases. Simultaneously, the DSOs can take over exciting taskswhich involve new business areas, potentially increasing productivity andbringing value towards general grid operation, management and to the soci-ety.

36

References

[1] Benchmarking big data architectures for social networks data processing usingpublic cloud platforms. Future Generation Computer Systems, 89:98 – 109, 2018.

[2] Zaia Alimazigui Abdeslam Mehenni1 and Mohamed Ahmed-Nacer. Decreasingdata analytics time: Hybrid architecture mapreduce-massive parallel processingfor a smart grid. Journal of Electrical Systems, pages 74 – 85, 2017.

[3] Fadi Al-Turjman and Mohammad Abujubbeh. Iot-enabled smart grid via sm:An overview. Future Generation Computer Systems, 2019.

[4] Dieter Gantenbein Anders Holm Foosnæs, Claus Amtrup Andersen;Linda Christensen and Ningling Rao. EDISON – Study on optimal grid integrationof electric vehicles. DTU Library, 2011.

[5] Artemakis Artemiou. Introducing sql server in-memory oltp,https://www.red-gate.com/simple-talk/sql/learn-sql-server/

introducing-sql-server-in-memory-oltp/, 2015.

[6] Thy-Mors Energi Elnet A/S. Beskrivelse af forskningsprojekter i elnettet,https://www.tme-elnet.dk/om-os/forskningsprojekter.

[7] Shivanand Balram, Balram Shivanand, Suzana Dragiâceviâc, and Suzana Drag-icevic. Collaborative Geographic Information Systems. 2006.

[8] Mininath Bendre and Ramchandra Manthalkar. Time series decomposition andpredictive analytics using mapreduce framework. Expert Systems with Applica-tions, 116:108 – 120, 2019.

[9] Paul De Hert; Vagelis Papakonstantinou; Gianclaudio Malgieri; Laurent Beslayand Ignacio Sanchez. The right to data portability in the gdpr: Towards user-centric interoperability of digital services. Computer Law and Security Review,34(2):193 – 203, 2018.

[10] Simon Elias Bibri. Smart Sustainable Cities of the Future. 2018.

[11] Ying Zhou Bilal Akil and Uwe Röhm. Technical report: On the usability ofhadoop mapreduce, apache spark and apache flink for data science. Technicalreport, Mar 28 2018.

[12] M. Boehm, A. Kumar, J. Yang, and H. V. Jagadish. Data Management in MachineLearning Systems. Morgan and Claypool, 2019.

[13] Henrik Breum. The Danish Energy Model. Innovative, efficient and sustainable.Technical report, The Danish Energy Agency.

37

https://www.red-gate.com/simple-talk/sql/learn-sql-server/introducing-sql-server-in-memory-oltp/

https://www.red-gate.com/simple-talk/sql/learn-sql-server/introducing-sql-server-in-memory-oltp/

https://www.tme-elnet.dk/om-os/forskningsprojekter

References

[14] Wayne Cascio and Ramiro Montealegre. How technology is changing workand organizations. Annual Review of Organizational Psychology and OrganizationalBehavior, 3:349–375, 03 2016.

[15] Richard Charnah. What exactly is a smart grid? a smart electrical grid has thepotential to play a key role in the effort to lower energy costs for consumers,reduce energy dependency and cut greenhouse gas emissions. how can all ofthis be realized and what can governments do to ensure such a grid can bemode a reality? dr. richard charnah, technology director of areva t&d uk, ex-plains.(smart grids). 17:60, 2009.

[16] René J. Chevance. 8 - dbms and server architectures. In René J. Chevance, editor,Server Architectures, pages 549 – 584. Digital Press, Burlington, 2005.

[17] Timothy C Coburn. Geographic information systems: an introduction: 2nd ed.tor bernhardsen. Computers and Geosciences, 26(7):853 – 854, 2000.

[18] Peter Dale. Information technology in geography and planning: Principles ofgeographical information systems: Ian bracken and christopher webster rout-ledge, london, uk, 1990, 444 pp. Cities, 8(1):73, 1991. Special Feature: GIS andurban management.

[19] dataArtisans. da platform stream processing for real-time business powered byapache flink. Technical report, 2018.

[20] Ganesh Chandra Deka. Chapter nine - nosql polyglot persistence. In PethuruRaj and Ganesh Chandra Deka, editors, A Deep Dive into NoSQL Databases: TheUse Cases and Applications, volume 109 of Advances in Computers, pages 357 –390. Elsevier, 2018.

[21] Werner Dubitzky. Data mining in grid computing environments. 2008.

[22] Siddhartha Duggirala. Chapter two - newsql databases and scalable in-memoryanalytics. In Pethuru Raj and Ganesh Chandra Deka, editors, A Deep Dive intoNoSQL Databases: The Use Cases and Applications, volume 109 of Advances inComputers, pages 49 – 76. Elsevier, 2018.

[23] Energinet. Electricity Security of Supply Report. Technical report, 2017.

[24] Energinet. Energinet’s analysis assumptions. Technical report, 2017.

[25] Hueske Fabian and Ewen Stephan. Stateful stream processing withapache flink, https://www.infoworld.com/article/3267966/big-data/

stateful-stream-processing-with-apache-flink.html, 2018.

[26] Zbigniew J. Gackowski. The helix of human cognition: knowledge managementaccording to dikw, e2e, and the proposed view. 15:93–, 2012.

[27] García-Gil, Diego García Gil, Sergio Ramírez Gallego, Salvador García, andFrancisco Herrera. A comparison on scalability for batch big data processingon apache spark and apache flink. 2:1–11, 2017.

[28] Jana Giceva and Mohammad Sadoghi. Hybrid OLTP and OLAP, pages 1–8. 012018.

[29] Corinna Giebler, Christoph Stach, Holger Schwarz, and Bernhard Mitschang.Braid: A hybrid processing architecture for big data. In Proceedings of the 7th

38

https://www.infoworld.com/article/3267966/big-data/stateful-stream-processing-with-apache-flink.html

https://www.infoworld.com/article/3267966/big-data/stateful-stream-processing-with-apache-flink.html

References

International Conference on Data Science, Technology and Applications, DATA’18,pages 294–301. SCITEPRESS, July 2018.

[30] Daniel Goldberg, Miriam Olivares, Zhongxia Li, and Andrew G. Klein.Maps&gis data libraries in the era of big data and cloud computing. 10:100–122,2014.

[31] Venkat N. Gudivada. Chapter 2 - data analytics: Fundamentals. In MashrurChowdhury, Amy Apon, and Kakan Dey, editors, Data Analytics for IntelligentTransportation Systems, pages 31 – 67. Elsevier, 2017.

[32] Iris Reinhartz-Berger; Jens Gulden; Selmin Nurcan; Wided Guédria and PalashBera. Enterprise, Business-Process and Information Systems Modeling. Springer,Cham, 2017.

[33] Angappa Gunasekaran, Gunasekaran, and Maqsood Sandhu. Handbook OnBusiness Information Systems. 2010.

[34] Gurusamy, Vairaprakash Gurusamy, S. Kannan, and K. Nandhini. The real timebig data processing framework advantages and limitations. 5:305–312, 2017.

[35] John Hagerty. 2017 planning guide for data and analytics. Technical report,Gartner, Inc., 2016.

[36] Benoît Bizet Hans Peter Slente, Peter Alexandersen and Torsten Hasforth. WindEnergy Moving Ahead. How wind energy has changed the Danish energy sys-tem. Technical report, Think Denmark, July 2017.

[37] Paris Carbone; Asterios Katsifodimos; Stephan Ewen; Volker Markl; Seif Haridiand Kostas Tzoumas. Apache flink™: Stream and batch processing in a singleengine. IEEE Data Eng. Bull., 38:28–38, 2015.

[38] Richard Heinberg and David Fridley. Our Renewable Future. Laying the Path forOne Hundred Percent Clean Energy. June 2016.

[39] Karen Holtzblatt and Hugh Beyer. 10 - the bridge from data to design: The wallwalk. In Karen Holtzblatt and Hugh Beyer, editors, Contextual Design (SecondEdition), Interactive Technologies, pages 253 – 275. Morgan Kaufmann, Boston,second edition edition, 2017.

[40] Karen Holtzblatt and Hugh Beyer. 7 - building experience models. In KarenHoltzblatt and Hugh Beyer, editors, Contextual Design (Second Edition), Interac-tive Technologies, pages 147 – 206. Morgan Kaufmann, Boston, second editionedition, 2017.

[41] M.S. Hossain, N.A. Madlool, N.A. Rahim, J. Selvaraj, A.K. Pandey, and Ab-dul Faheem Khan. Role of smart grid in renewable energy: An overview. Re-newable and Sustainable Energy Reviews, 60:1168 – 1184, 2016.

[42] H. Hu, Y. Wen, T. Chua, and X. Li. Toward scalable systems for big data analyt-ics: A technology tutorial. IEEE Access, 2:652–687, 2014.

[43] Fabian Hueske. How to build stateful streaming applications with apache flink.2018.

[44] Abdulaziz Almehmadi James Curzon and Khalil El-Khatib. A survey of privacyenhancing technologies for smart cities. Pervasive and Mobile Computing, 55:76 –95, 2019.

39

References

[45] Mark Jeffery. Return on investment analysis for e-business projects. 2003.

[46] Shibily Joseph, Jasmin E.A., and Soumya Chandran. Stream computing: Op-portunities and challenges in smart grid. Procedia Technology, 21:49 – 53, 2015.SMART GRID TECHNOLOGIES.

[47] Kamstrup. Electricity meter production, https://www.kamstrup.

com/en-en/electricity-solutions/smart-electricity-meters/

electricity-meter-production.

[48] Michael D. Kennedy. Introducing Geographic Information Systems with ArcGIS : AWorkbook Approach to Learning GIS. 2013.

[49] Jure Leskovec Keyulu Xu, Weihua Hu and Stefanie Jegelka. How powerful aregraph neural networks? CoRR, abs/1810.00826, 2018.

[50] Nicholas Choma; Federico Monti; Lisa Gerhardt; Tomasz Palczewski; ZahraRonaghi; Prabhat; Wahid Bhimji; Michael M. Bronstein; Spencer R. Klein andJoan Bruna. Graph neural networks for icecube signal classification. CoRR,abs/1809.06166, 2018.

[51] Dariusz Król, Ngoc Thanh Nguyen, and Kiyoaki Shirai. Advanced Topics inIntelligent Information and Database Systems, volume 710. 2017.

[52] Katerina Lepenioti, Alexandros Bousdekis, Dimitris Apostolou, and GregorisMentzas. Prescriptive analytics: Literature review and research challenges. In-ternational Journal of Information Management, 50:57 – 70, 2020.

[53] Clinton J. Levitt and Anders Sørensen. The Cost of Producing Electricity in Den-mark. The Rockwool Foundation, November 2014.

[54] X. Liu, A.V. Dastjerdi, and R. Buyya. Chapter 8 - stream processing in iot:foundations, state-of-the-art, and future directions. In Rajkumar Buyya andAmir Vahid Dastjerdi, editors, Internet of Things, pages 145 – 161. Morgan Kauf-mann, 2016.

[55] Zhenya Liu. Chapter 6 - innovation in global energy interconnection technolo-gies. In Zhenya Liu, editor, Global Energy Interconnection, pages 239 – 272. Aca-demic Press, Boston, 2015.

[56] Zhihui Lu, Nini Wang, Jie Wu, and Meikang Qiu. Iotdem: An iot big data-oriented mapreduce performance prediction extended model in multiple edgeclouds. Journal of Parallel and Distributed Computing, 118:316 – 327, 2018. Spe-cial Issue on Advanced Algorithms and Applications for IoT Cloud ComputingConvergence.

[57] M. Iacono M. Gribaudo and M. Kiran. A performance modeling frameworkfor lambda architecture based applications. Future Generation Computer Systems,86:1032 – 1041, 2018.

[58] Z. Ma, S. Sommer, and B. N. Jørgensen. The smart grid impact on the dan-ish dsos’ business model. In 2016 IEEE Electrical Power and Energy Conference(EPEC), pages 1–5, Oct 2016.

[59] S. T. Mak and E. So. Integration of pmu, scada, ami to accomplish expandedfunctional capabilities of smart grid. In 29th Conference on Precision Electromag-netic Measurements (CPEM 2014), pages 68–69, Aug 2014.

40

https://www.kamstrup.com/en-en/electricity-solutions/smart-electricity-meters/electricity-meter-production



References

[60] Ruben Sánchez Martin-Loeches, Florin Iov, Mohammed Seifu Kemal, Maria Ste-fan, and Rasmus Løvenstein Olsen. Observability of low voltage grids: actualdsos challenges and research questions. In Proceedings of the 2017 52nd Interna-tional Universities’ Power Engineering Conference (UPEC). IEEE Press, 8 2017.

[61] Patrick McGlamery. Library and Information Science Research, 27(3):418 – 419,2005.

[62] S. E. McHann. Grid analytics: How much data do you really need? In 2013IEEE Rural Electric Power Conference (REPC), pages C3–1–C3–4, April 2013.

[63] Abdeslam Mehenni, Zaia Alimazighi, Tarek Bouktir, and Mohamed Ahmed-Nacer. An optimal big data processing for smart grid based on hybrid mdm/rarchitecture to strengthening re integration and ee in datacenter. 2018.

[64] Express Metrix. Quantifying ROI: Building the Business Case for IT and Soft-ware Asset Management. Technical report.

[65] Z. Milosevic, W. Chen, A. Berry, and F.A. Rabhi. Chapter 2 - real-time analytics.In Rajkumar Buyya, Rodrigo N. Calheiros, and Amir Vahid Dastjerdi, editors,Big Data, pages 39 – 61. Morgan Kaufmann, 2016.

[66] Amit Kumar Mishra. A dikw architecture for cognitive engineering. ProcediaComputer Science, 123:285 – 289, 2018. 8th Annual International Conferenceon Biologically Inspired Cognitive Architectures, BICA 2017 (Eighth AnnualMeeting of the BICA Society), held August 1-6, 2017 in Moscow, Russia.

[67] Thomas Brandstetter Morta Vitunskaite, Ying He and Helge Janicke. Smartcities and cyber security: Are we there yet?a comparative study on the role ofstandards, third party risk management and security ownership. Computers andSecurity, 83:313 – 331, 2019.

[68] A. A. Munshi and Y. A. I. Mohamed. Data lake lambda architecture for smartgrids big data analytics. IEEE Access, 6:40463–40471, 2018.

[69] Amr A. Munshi and Yasser A.-R. I. Mohamed. Big data framework for analyticsin smart grids. Electric Power Systems Research, 151:369 – 380, 2017.

[70] A. Mutule, A. Obushevs, A. Lvov, H. Segerberg, N. Shao, S. You, R. Bacher,H. Sauvain, J. Remund, U. Schröder, and A. J. Engen. Efficient identificationof opportunities for distributed generation based on smart grid technology. InIEEE PES ISGT Europe 2013, pages 1–5, Oct 2013.

[71] Thanisa Numnonda. A real-time recommendation engine using lambda archi-tecture. Artificial Life and Robotics, 23(2):249–254, Jun 2018.

[72] CPH Post Online. Hurricane-force storm crossing denmark, http://cphpost.dk/news/national/hurricane-force-storm-crossing-denmark.html.

[73] M. Ourahou, W. Ayrir, B. EL Hassouni, and A. Haddi. Review on smartgrid control and reliability in presence of renewable energies: Challenges andprospects. Mathematics and Computers in Simulation, 2018.

[74] Trine Pallesen and Peter Holm Jacobsen. Solving infrastructural concernsthrough a market reorganization: A case study of a danish smart grid demon-stration. Energy Research and Social Science, 41:80 – 88, 2018. Energy Infrastruc-ture and the Fate of the Nation.

41

http://cphpost.dk/news/national/hurricane-force-storm-crossing-denmark.html

http://cphpost.dk/news/national/hurricane-force-storm-crossing-denmark.html

References

[75] Mrutyunjaya Panda. Intelligent data analysis for sustainable smart grids us-ing hybrid classification by genetic algorithm based discretization. IntelligentDecision Technologies, 11, 02 2017.

[76] S. Perera. Large scale data processing in real world: From analytics to pre-dictions. In 2014 14th International Conference on Advances in ICT for EmergingRegions (ICTer), pages 8–8, Dec 2014.

[77] Pries, Kim H. Pries, and Robert Dunnigan. Big Data Analytics. 2015.

[78] Alexander Maria Prostejovsky. Increased Observability in Electric DistributionGrids. Emerging Applications Promoting the Active Role of Distribution System Op-erators. PhD thesis, Technical University of Denmark, 2017.

[79] Xiangnan Ren and Olivier Curé. Strider: A hybrid adaptive distributed rdfstream processing engine, May 16 2017.

[80] Paris Carbone; Stephan Ewen; Gyula Fora; Seif Haridi; Stefan Richter andKostas Tzoumas. State management in apache flink? 10:1718–1729, 2017.

[81] Iman Samizadeh. A brief introduction to two data processing architectures -lambda and kappa for big data, March https://towardsdatascience.com/,2018.

[82] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini. Thegraph neural network model. IEEE Transactions on Neural Networks, 20(1):61–80,Jan 2009.

[83] Amazon Web Services. Lambda architecture for batch and stream processing.Technical report, October 2018.

[84] Ashansa Perera Shelan Perera and Kamal Hakimzadeh. Reproducible experi-ments for comparing apache flink and apache spark on public clouds. CoRR,abs/1610.04493, 2016.

[85] Matei Zaharia; Reynold S. Xin; Patrick Wendell; Tathagata Das; Michael Arm-brust; Ankur Dave; Xiangrui Meng; Josh Rosen; Shivaram Venkataraman;Michael Franklin; Ali Ghodsi; Joseph Gonzalez; Scott Shenker and Ion Stoica.Apache spark. 59:56–65, 2016.

[86] Anshu Shukla, Shilpa Chaturvedi, and Yogesh Simmhan. Riotbench: An iotbenchmark for distributed stream processing systems. 29:e4257–, 2017.

[87] Krishnarajanagar G. Srinivasa and Srinidhi Hiriyannaiah. Chapter five - com-parative study of different in-memory (no/new) sql databases. In Pethuru Rajand Ganesh Chandra Deka, editors, A Deep Dive into NoSQL Databases: The UseCases and Applications, volume 109 of Advances in Computers, pages 133 – 156.Elsevier, 2018.

[88] Maria Stefan, Jose G. Lopez, and Rasmus L. Olsen. Exploring the potentialof modern advanced metering infrastructure in low-voltage grid monitoringsystems. 2018 IEEE International Conference on Big Data (Big Data), pages 3543–3548, 2018.

[89] Sheila Lakshmi Steinberg and Steven J. Steinberg, Sheila Lakshmi; Steinberg.GIS Research Methods. 2015.

42

https://towardsdatascience.com/

References

[90] Ida Strand. 2050. Something’s Green in the State of Denmark. Scenarios for asustainable economy. Technical report, 2012.

[91] Jesper Tornbjerg. Green Energy for All. KLS Grafisk Hus A/S, November 2014.

[92] Lamberto Tronchin, Massimiliano Manfren, and Benedetto Nastasi. Energy ef-ficiency, demand side management and energy storage technologies – a criticalanalysis of possible paths of integration in the built environment. Renewable andSustainable Energy Reviews, 95:341 – 353, 2018.

[93] Kostas Tzoumas and Stephan Ewen. Real-time stream processing:The next step for apache flink, May https://www.confluent.io/blog/

real-time-stream-processing-the-next-step-for-apache-flink/, 2015.

[94] K. R. Ishwarya V. Vidhya, G. Jeyaram. Database management systems. 2016.

[95] Shaohua Wang, Yang Zhong, and Erqi Wang. An integrated gis platform archi-tecture for spatiotemporal big data. Future Generation Computer Systems, 94:160– 172, 2019.

[96] Chris Webb. Will the european grid smarten up? 15:20–21,23–24, 2007.

[97] The Microsoft website. Sql server 2016, https://www.microsoft.com/en-us/sql-server/sql-server-2016/.

[98] Wingerath, Wolfram Wingerath, Norbert Ritter, and Felix Gessert. Real-Time &Stream Data Management. 2019.

[99] Q. Wu, E. Larsen, K. Heussen, H. Binder, and P. Douglass. Remote off-gridsolutions for greenland and denmark: Using smart-grid technologies to ensuresecure, reliable energy for island power systems. IEEE Electrification Magazine,5(2):64–73, June 2017.

[100] Fatos Xhafa. Modeling and Processing for Next-Generation Big-Data Technologies,volume 4. 2015.

[101] Vinit Yadav. Processing Big Data with Azure HDInsight. 2017.

[102] Jared Zagelbaum. A different way to process data: Kappaarchitecture, January https://www.blue-granite.com/blog/

a-different-way-to-process-data-kappa-architecture, 2018.

[103] Kenneth K. Zame, Christopher A. Brehm, Alex T. Nitica, Christopher L.Richard, and Gordon D. Schweitzer III. Smart grid and energy storage: Pol-icy recommendations. Renewable and Sustainable Energy Reviews, 82:1646 – 1654,2018.

[104] Jie Zhou, Ganqu Cui, Zhengyan Zhang, Yang Cheng, Zhiyuan Liu, LifengWang, Changcheng Li, and Maosong Sun. Graph neural networks: A review ofmethods and applications, Mar 07 2019.

43

https://www.confluent.io/blog/real-time-stream-processing-the-next-step-for-apache-flink/

https://www.confluent.io/blog/real-time-stream-processing-the-next-step-for-apache-flink/

https://www.microsoft.com/en-us/sql-server/sql-server-2016/

https://www.microsoft.com/en-us/sql-server/sql-server-2016/

https://www.blue-granite.com/blog/a-different-way-to-process-data-kappa-architecture

https://www.blue-granite.com/blog/a-different-way-to-process-data-kappa-architecture

References

44

Part II

Papers

45

Paper A

Visualization Techniques for Electrical Grid SmartMetering Data: A Survey

Maria Stefan, Jose G. Lopez, Morten H. Andreasenand Rasmus L. Olsen

The paper has been published at the:2017 IEEE Third International Conference on Big Data Computing Service and

Applications (BigDataService)

© 2017 IEEEThe layout has been revised and reprinted with permission.

I. Introduction

Abstract

One of the considerable initiatives towards creating a smart society could be theguarantee of a smart, resilient and reliable power grid. As an attempt to improve theelectricity supply service, it would be meaningful for the distributed system operators(DSOs) to be able to monitor the current status of the grid. The prediction of futurepossible critical situations would then be feasible using the available information,whereas, based on historical data, further grid expansion and reinforcement may beplanned. A proper presentation and visualization of the near-real time metering datamay constitute the baseline for bringing improvements to the power grid. This paperpresents an approach to build an efficient visualization system so that the extractedsmart meters information can be used in a meaningful manner. An overview of theuse cases related to the visualization features is first presented, as a motivation for thechoice of the relevant state of the art research. In relation to the knowledge providedby the metering data, a definition of the big data concept will be further introduced,according to the requirements established by the project definition. Geographic In-formation System (GIS) tools are useful to help visualize the collected big data innear-real time. For this reason, a survey of existing GIS software will be made sothat the choice of the most suitable tool can be justified. Also, the integration ofGIS technologies into the Common Information Model (CIM) aims to improve thevisualization efficiency. As a consequence, investigating methods for adapting CIMstandards to the GIS platform are also important.

I Introduction

In the process of the development of a more efficient electrical grid, the con-cept of the so-called "smart grids" has emerged. The purpose is to create anaffordable, reliable and sustainable electricity supply. As a consequence ofthe development of smart grids, the distributed system operators (DSOs) inthe Danish electricity distribution grid are facing operational challenges dueto a large number of new smart electronic devices. These devices load theproducer utilities with a high amount of data, reporting issues such as cableand converter faults, voltage magnitude outside standard limits and networkcongestion [13]. In order to address these challenges, intelligent featuresare required so that the DSOs can obtain an overview of their low voltagegrid. This would allow the execution of near real-time daily operations in thegrid, as well as long term grid management and planning. The efficiency ofthe electrical grid could be improved through the collection, processing andanalysis of data and the outcome would have people as the main beneficiary.Eventually, the grid’s efficiency would be characterized by user satisfaction,economic implications, population reach etc. In the long run, the pursuitof progress in public and private sectors constitutes an initiative to create a

49

Paper A.

smart society. In the current paper different methods are investigated towardsbuilding such a visualization system (GIS tools, CIM modeling, implementa-tion languages). The uniqueness of this research consists of experiments thatcorrelate different methods to achieve the desired visualization.

Current electricity grids are the baseline for future grids, which couldaccount for the changes in innovative technologies, customer needs, envi-ronmental issues and increasing network congestion [10] [20]. The futuredistribution grids’ architecture is evolving from a one-direction power flowtowards a bi-directional flow between suppliers and consumers, according tothe European Commission’s view on electricity systems. The aim is to createa customer-oriented electricity system that will be flexible, accessible, reliableand economic [10].

Traditional power systems are developing towards digitalization, with theemerging ICT (Information and Communication Technology). In [27] digital-ization is defined as a key element in the further development of the powersystem, being able to provide an efficient time-critical monitoring of the gridstate, where issues would be signaled through the display of alarms.

In the energy sector, large amounts of data are accumulated daily. Themain source of data in a smart grid is the adaptive metering infrastructure(AMI), where a large number of smart meters are deployed at the user endside [32]. Mechanisms for collecting data from the smart meters are ad-dressed in [18] for the real-time state estimation of the grid. They are ad-vantageous because they can facilitate the power flow control and identifyexceeding limits of current and voltages. In other words, data collectionmechanisms offer some degree of certainty of data quality. Correctness, com-pleteness and timely data are some of the main attributes that can describethe data quality and it is important that they are ensured prior to its process-ing and viewing.

Three main steps need to be accomplished in order to obtain the data vi-sualization platform, as shown in Figure A.1: from storing data in a databaseto the "human eyes". Establishing efficient ways of transforming smart me-ters raw data into meaningful information can contribute to the operation,management and planning of the power grid.

Database

structure and

management

Data

processing VisualizationData

Fig. A.1: Data flow perspective.

A systematic storage of smart grid data is possible using database struc-tures. As it is presented in [19], database architectures and methods alsoallow the processing and analysis of large amounts of data, continuously

50

I. Introduction

aggregated by time flow.A more accurate representation of real-time data is closely related to the

delays involved in data transmission from the smart meters to the visual-ization platform. The delays involved in the delimitation of the “real-time”definition can be defined as the transmission and the processing delays, rep-resented in Figure A.2. The transmission delay (smart meter - database) isbound to the adaptive data collection mechanisms presented in [18]. The dataprocessing delay (database - visualization) is the aspect to be analyzed in thisresearch and it will be used for defining the use case related to the real-timedisplay of information.

Smart meter DatabaseVisualization

(GIS)

Transmission

delay

Processing

delay

Fig. A.2: Delays corresponding to grid state estimation and data processing.

The dynamics of information impacts the visualization in the sense that in-formation that changes rarely over time can be treated differently. More pro-cessing time can be spend on such information compared to very volatileinformation elements that often change values, meaning that it is highly de-pendent on the information granularity.

The outlined challenges lead therefore to investigating scalable and secureIT infrastructures for real-time events management in the smart grid. Extract-ing significant information is important for the decision-making process andthis is intended to be made through big data analytics. The challenge of thebig data research is to include a set of technologies that would ensure users’privacy, but still extract the valuable information needed for real-time gridoperations and long term scale planning.

A possible implementation of data analytics can be done via the openstandard Common Information Model (CIM). This model facilitates exchangeof power system network data between companies and allows data exchangebetween applications within a company; thus, easier implementation of dataanalytics. Its purpose is to increase reliability and reduce expenses in smartgrid infrastructures, as shown in [8].

The paper is organized as follows: Section II introduces the use cases con-cerning the real-time and historical features of the visualization platform. InSection III the motivation for choosing the CIM standard is presented, alongwith its newest uses and challenges both in research and industry. Section IVdescribes the related work regarding: big data use cases, visualization usingGIS software and their relation to big data analytics. Furthermore, a defini-tion of big data will be explained here in relation to the previously-mentioneduse cases. In Section V different GIS tools are evaluated to motivate the choice

51

Paper A.

of the most suitable one for carrying out the project.

II Data Visualization Use Cases

Data collected from the smart meters is a raw set of facts, without any partic-ular meaning or usability. The process of how knowledge is obtained out ofraw data is presented in the DIKW diagram shown in Figure A.3.

Data

Information

Knowledge

Fig. A.3: DIKW Diagram.

In order to obtain valuable information out of raw data, some manage-ment tools have to be established. First of all, database systems are a fun-damental tool for storing any kind of data, for later processing and analysis.The processing of data aims to provide the proper and timely input to thefuture visualization platform, moving the geospatial data to near-real-timeaccess [12].

Information is the outcome of data processing, whose patterns and factscan be analysed to determine what is needed in the development of the vi-sualization system. It becomes knowledge when it is applied in a particularsituation to answer questions such as "why" and "how".

In order to establish the relevant state of the art survey, some use caseshave to be defined. The visualization features can be divided into: time-critical (grid monitoring) and non-time critical (grid history).

A. Grid Monitoring

The real-time visualization features of the system are highly dependent onhow often data is received and stored in the database and the events in-volved in the real-time data processing. Hence, one has to define what data

52

II. Data Visualization Use Cases

needs to be processed for each visualization feature, to achieve the minimumprocessing delays in this context. At the same time, the visualization featuresneed to be scalable according to the grid size and the volume of data storedin the database.

The delay in data processing is time-critical for the monitoring system.It has to be minimized taking into account: the amount of data requiredto make qualified decisions, performance cost of data monitoring and thetriggered events, and the amount of time required for the necessary data tobe received.

Example: Voltage drop/rise

With the current technology it is possible to display the voltage magnitudeof the system. The voltage drop is an issue that has to be adjusted and it canbe done by monitoring the voltage stability margin, as explained in [20].

The overall quality of the distribution power grid can also be affected byvoltage unbalance (phase inbalance) [7]. Furthermore, voltage rise (due tolarge amount of distributed generation power output) is presented in [3] asone of the main challenges for the DSOs to regulate the voltage levels in thegrid.

Another useful feature that enables operators to quickly identify the faultlocation is be the ability to display real-time alarms in the visualization sys-tem [20]. This could be achieved using dedicated visualization features ofGIS tools, as presented in Table A.1. For example, the ArcGIS GeoEventProcessor makes it possible to track dynamic data, which changes locationfrequently. Likewise, MapInfo’s Animation Layer add in is utilized for appli-cations where data features update constantly.

The difference between the time stamp where the first suspicious event isreceived and how big is the delay to the actual alarm visualization should beinvestigated as well.

B. Grid History

The non-time-critical features are related to the electrical grid planning, tocreate models for future state estimates. Therefore, it should be establishedwhat parameters to keep track of, which data should be passed on to thevisualization system and how to process data so that the database visualiza-tion system can be optimized. Data analytics and data mining techniquescan help in discovering data patterns, that can be later useful in future gridplanning in a quicker and more efficient manner.

53

Paper A.

Example: Power balancing

An example of how tracking historical data helps providing a better under-standing of the grid state is presented in [1], for forecasting electrical loads.Statistical analysis can be created and further used to evaluate grid operatingconditions and thus, failure assessment.

The collected historical data contains information about grid events, forexample frequent oscillations in the electrical load power. Important infor-mation related to power failures can be extracted using historical data, whiledata mining techniques can provide an in-depth explanation for the failurecause. Parameters such as temperature, weather conditions and electricalload may be considered in the data analysis. Making use of historical datacreates the foundation for a more efficient future grid planning, creating theopportunity to avoid power failures.

III CIM Modeling

The initial motivation for the development of the CIM was driven by theincreasing requirements for Energy Management Systems (EMS) to enhanceupgradability, scalability and interoperability. Ambitious European energyand climate goals dictate the future and will change the very nature of thepower system, promoting cooperation between European utility enterprises(DSOs) in order to support the implementation of the EU energy policy. Spe-cial focus is on the integration of Renewable Energy Sources (RES), whichexpands distributed power generation, and the Internal Energy Market (IEM)to meet the EU’s energy policy objectives of affordability, sustainability andsecurity of supply. This will invariably increase the data exchange necessi-ties both internal and between the European utility enterprises and hence therelevance and need of a common semantic framework like CIM [31].

The use of ICT and advances in the design of power systems result in anincrease and variation of the data. Its analysis and recognition require ad-vanced data mining techniques, which are however heavily restricted by in-sufficient description of the data models. These issues lead the developmentfrom a vendor hard- and software specific API to a focus on common se-mantic and syntax models for data exchange between the EMS database andapplications, which advanced into the CIM architecture comprising severalstandards under the International Electrotechnical Commission (IEC) Techni-cal Committee 57 (TC57) and associated workgroups (WG) [16].

The integration of CIM has been adopted by Statnett, Norway’s nationalmain grid owner and operator. The future Statnett vision for CIM integrationis centered on the perceived benefits in conjunction with predicted necessitiesfor next generation smart grid development. CIM is envisioned to provide

54

III. CIM Modeling

EMS

IECn61970

DMS

IECn61850

DMS

IECn61850

Name/ServiceRegistry

BigndataAnalyticalnTools

SharednDatanandnServices

GIS

IECn61968p13

DMS

IECn61850

AMI

IECn61968p9

ENTERPRISEnSERVCEnBUS

CIMnstandard

UtilitynInformationSystems

Middleware

BignDataApplications

Fig. A.4: CIM-based utility big data integration [33].

One common Power System Model (PSM), delivering a complete enterpriseservice oriented integration that is adaptable to future requirements; intro-ducing a standard for data exchange via a common Enterprise Service Bus(ESB) (CIM EAI message bus - Figure A.4), capable of handling all data gov-ernance and data management services. As part of the development of thefuture power system, Statnett is undertaking pilot projects analyzing the po-tential and performance of smart grid technologies and communication sys-tems. One such project includes Demand Side Response (DSR) load controlvia AMI, where CIM was utilized as the standard for data exchange betweenthe distribution management system (DMS) and the AMI front end, utilizingCIM XML messaging (DSO - AMI front end; IEC 61968-9) [30].

The European Network of Transmission System Operators (ENTSO-E) is acollaboration between 42 European TSO’s representing 35 countries, includ-ing Norwegian Statnett and Energinet.dk of Denmark. By the same token,smart meter penetration of European consumers, i.e. households and Smalland Medium Enterprises (SMIs), is currently on the rise and is expected asa continuous trend for the near future. ENTSO-E utilize CIM IEC standardsto provide common data exchange formats to ensure compatibility for thevarious information sharing between transmission system operators (TSOs),third parties and service providers alike. Direct cooperation is made with theIEC workgroups responsible for CIM for transmission (IEC TC57 / WG13)and CIM for energy markets (IEC TC57 / WG16) securing TSO influence andcompliance, and supporting the continuous development of CIM [14].

The uses and challenges of CIM in the research area are addressed in [16],which deals with data driven interactive visualization of power systems. CIMbased model visualization is done via data manipulation algorithms based onempirical or mathematical derived utility data. The main benefit is to providesmart grid operation and analytics decision support tools, enabling electricsystem operators and analysists to perpetually monitor big data information

55

Paper A.

and events by images, diagrams, animations promoting communication andinterpretation with enhanced pattern recognition. Furthermore, emergingbusiness requirements of the electric power industry gain from the powersystems data visualization.

Complete GIS and Supervisory Control and Data Acquisition (SCADA)integrated for monitoring of the power distribution network are presentedin [29], including a common graphical user interface and shared networkmodel. CIM is utilized for modeling the distribution network and its stan-dard compliance facilitates data exchange. This allows for the aggregation ofpower equipment information and spatial GIS data with real time operationalstatus information, enhancing decision support and abnormal alert response.Application development costs are reduced with multi-platform support su-perior to commercial GIS, avoiding duplication of data while enhancing datavalidity and reducing human error.

The integration of smart substations in smart grid architecture supportingintelligence aggregation in utility operation and management is addressedin [8]. Complying with IEC CIM standards supports substation analyticsand system integration with enhanced value adding information exchange.As a result, smart connectivity is provided for Intelligent Electronic Devices(IEDs), promoting interoperability at all utility system levels and enablingoperating and functional information exchange suited for individual IEDs,utility and decision support systems.

In [33], an electric utility company utilizes big data via analytics and theproposed software framework is based on CIM IEC standards, in order toconvert utility big data into operational decision support, promote efficiencyand save costs. Figure A.4 illustrates how utility big data applications caninteract with each other using a CIM-based integration architecture, througha common enterprise service bus.

Given the above-mentioned examples of how CIM is applied both in in-dustry and in research, it can be concluded that its implementation is benefi-cial for an effective visualization of the electrical smart metering data due tothe following reasons:

- Shared common services through a message bus interface: mapping ofthe CIM class structure to an application’s external interface;

- Facilitation of a real time environment for dynamic data messaging;

- Enhanced data analytics;

- Identifying and resolving issues in order to ensure the quality of data;

- Platform independence.

56

IV. Big Data and Data Visualization

CIM modelling is feasible via multiple formats [23]:

- UML-CIM: standard defined in UML using classes, attributes and rela-tionships;

- XML-CIM: class structure mapping and data encapsulation format. En-coding of plain text enables human-machine interaction;

- RDF (Resource Document Framework): an XML schema that definesrelationships between XML nodes (outside parent/child class relation-ships). Nodes are assigned unique RFD IDs and resource attributes. Inaddition, an RFD schema is needed to provide the vocabulary for de-scribing an object oriented type of system for RDF. The combination ofRDF and RDF schema supports a class hierarchy structure XML schemathrough inter-class properties;

- CIM, XML and RDF: their combination is used to model the entire CIMpower system. Provides a readable format for both humans and ma-chines, due to the platform independent plain text format;

- XML Messaging: data exchange is provided through the XML datastructure (CIM messages), associated with an XML schema that definesclasses and attributes interpretation.

Therefore, competitive, privacy or security concerns prohibiting open ex-change of complete model data can be alleviated by layered data exchangevia CIM with restrictions ensuring only the required data is shared.

IV Big Data and Data Visualization

In the energy sector, the progressive penetration of Distributed Generation(multiple sources of small scale power generation) brings deep changes inthe design of the grid [24]. At the same time, the penetration of a largenumber of power electronic devices (PVs, heat pumps, smart meters) hasbrought major changes in the volume of collected data.

Big data analytics can bring new opportunities in the management of asmart grid, in terms of data storage, analysis and mining, as mentioned in[32]. The work also states the importance of GIS as a traditional and complexsource of big data, characterized by spatial attributes.

In various parts of the existing literature big data is often referred to asdata of very high volume, variety and massive continuous flow. Variety,volume and velocity, also known as the "3Vs" are some of the most commonbig data characteristics. Thus, the notion of real-time is closely related to thespeed required for processing and analyzing the data [32] [33] [21].

57

Paper A.

But these parameters cannot provide meaningful decision support to en-sure a smart, resilient and reliable power grid, unless valuable knowledgeis extracted from big data. In [32] the core of smart grid big data man-agement is presented: data mining techniques and knowledge representa-tion/visualization.

Another study [33] proposes additional solutions to enhance utility bigdata, apart from the "3Vs". Utilization and analytics are recommended forimplementing a user-centered application framework, while a presentationof the collected big data is useful for the visualization framework.

The design of a data platform supporting enterprise level Big Data Inte-gration (BDI) is addressed in [21]. The methodology proposes data integra-tion and big data analytics schemes into a common data repository platformfeaturing scalability, real time data and security. The objective is to overcomeconventional solution challenges and to create user friendly and powerfuldata query visualization and analytics tools.

Some examples of big data use cases are presented in [26]. The big datareference architectures of social networking services, such as Facebook, Twit-ter, LinkedIn and Netflix, are shown. The implementation of such use casesrequires a great variety of technologies regarding data integration, storage,analysis and visualization, with the purpose of a better understanding ofconsumers’ needs.

The necessity of being able to display in near real-time the acquired infor-mation from different sources has led many researchers to develop GIS-basedsystems. [22] and [4] address the issue of real time data integration by us-ing open source desktop GIS software, such as QGIS integrated with Grass.Other studies approach the combination of different types of GIS tools todisplay information: ArcGIS and QGIS [5], QGIS and Pmapper [22], QGIS,GRASS and MapServer [4]. An overview of GIS software tools utilized forvisualization purposes is presented in [2] and [17], the latter concluding thatQGIS is a better choice for data visualization and spatial analysis.

Given the aforementioned big data related work and the use cases pre-sented in Section II, it can be concluded that the "3Vs" may be a relevant bigdata definition. However, the knowledge acquired from big data has a heavyimpact in a decision making process. Therefore, a more accurate definitionshould include techniques on how and what big data can be actually usedfor, in the direction of developing a reliable, secure and effective power grid:data mining, visualization and data analytics.

58

IV. Big Data and Data Visualization

Tabl

eA

.1:S

urve

yof

GIS

Tool

s

Des

ktop

GIS

Soft

war

eLi

cens

eIm

plem

enta

tion

Lan-

guag

ePl

atfo

rms

Vis

uali

zati

onfe

atur

es(p

lug-

ins/

exte

nsio

ns/a

ddin

s)

Use

rIn

terf

ace

and

App

lica

tion

s

Arc

GIS

Prop

riet

ary

Pyth

on,C

++,R

Win

dow

sO

SG

eoEv

ent

Proc

esso

r,Tr

acki

ngA

naly

st(e

x-te

nsio

n)

Arc

Map

,Arc

Cat

alog

QG

ISFr

eePy

thon

,C++

Linu

x,U

nix,

Mac

OSX

,W

indo

ws

OS,

And

roid

Ope

nLa

yer

(plu

gin)

QG

ISBr

owse

r

Map

Info

Prop

riet

ary

Map

Basi

cW

indo

ws

OS

Ani

mat

ion

Laye

r(a

ddin

)C

olor

stre

tch

tool

GR

ASS

Free

Pyth

on,C

++Li

nux,

Uni

x,M

acO

SX,

Win

dow

sO

STe

mpo

ralF

ram

ewor

k(e

xten

sion

)Ve

ctor

Dig

itiz

er,

2.5D

/3D

visu

aliz

a-ti

onw

ith

wxN

viz

gvSI

GFr

eeJy

thon

(Pyt

hon

impl

e-m

enta

tion

inJa

va)

Linu

x,M

acO

SX,W

in-

dow

sO

S3D

Ani

mat

ion

Man

ager

(ext

ensi

on)

3DIn

terf

ace

Map

titu

dePr

opri

etar

yG

ISD

K(G

eogr

aphi

cIn

form

atio

nSy

stem

Dev

elop

er’s

Kit

)

Linu

x,U

nix,

Mac

OSX

,W

indo

ws

OS

Dri

ve-T

ime

Rin

gs,

Dri

ve-T

ime

Terr

itor

ies

(geo

grap

hic

anal

ysis

tool

s)

Map

Wiz

ard

(the

mat

icm

aps)

59

Paper A.

V Visualization Survey Analysis

Based on the studies in Section IV, some of the most popular desktop GIStools are summarized in Table A.1, according to a few key characteristics thatexplain the motivation for choosing the appropriate tool for the purpose ofthis research: license type, supported platforms and plugins/extensions/addins that could make possible a near-real time implementation.

Various tasks carried out by different kinds of GIS software are presentedin Table 1.1 in [34]. Due to privacy concerns and the possibility for localmanipulation of data, desktop GIS type has been chosen for the developmentof this project.

ArcGIS proprietary software is popular for its analytical functions, script-ing tools and the possibility for user developed functions in multiple pro-gramming languages. The Tracking Analyst extension makes it possible toreveal and analyse data patterns [25], while GeoEvent Processor is able toprocess time critical events [15] [25].

In MapInfo, the first desktop GIS product, additional tools can be imple-mented through its dedicated MapBasic programming language, such as theanimation layer add in, which is used for tracking frequently updated data,as in the case of real-time applications [28] [11].

Many of the common functionalities of a desktop GIS can also be foundwithin the Maptitude commercial software. It does not provide any real-timerelated analytic capabilities, but these can be customized using the GISDKapplication development platform [9].

GRASS and gvSIG are open source GIS software that come in handy forstoring and managing spatio-temporal data and solving planning issues [2].3D visualization of data and animations are achieved through their user in-terfaces and the customized extensions.

Open source QGIS software runs on various operating systems and sup-ports data formats from both ArcGIS and MapInfo [34]. Its browser interfacemakes it possible to access, organize, and visualize data within the supportedspatial layers [6]. Similar to MapInfo and Maptitude, its functionalities maybe extended by creating additional plugins using Python or C++. Therefore,it is possible to integrate features for real-time display of the data, as it hasbeen done in [19].

VI Conclusion and Future Work

This paper addressed the motivation and challenges for building an acces-sible and effective system to display real time and historical visualizationsbased on data acquired from smart meters. Due to the advantages of CIM,

60

it will be a part of the future work to model the components of an electricalgrid, in order to manage smart metering data and to represent GIS data. Itis expected that the implementation of CIM will result in enhanced possibili-ties for data analysis techniques, which constitutes one of the main steps thathave to be defined towards building the visualization platform. The choiceof a suitable GIS tool was done according to the requirements in the defineduse cases, especially the challenge of integrating real-time data visualization.QGIS is considered to be a suitable choice due to the fact that it is opensource, supported by multiple platforms, and its big variety of plugins, thatcan be implemented using commonly known programming languages.

The next step is to establish the requirements and specifications for datastorage using database architectures. It includes the choice of a suitable im-plementation language, database structure and the description of the electri-cal network structure, aiming to create a platform scalable with the integra-tion of CIM and GIS.

Acknowledgment

This work is financially supported by the Danish project RemoteGRID whichis a ForsKEL program under Energinet.dk with grant agreement no. 2016-1-12399.

References related to paper A

[1] Use of real-time/historical database in Smart Grid, 2011.

[2] Software Tools Required to Develop GIS Applications: An Overview, 2012.

[3] Voltage stability improvement through centralized reactive power management on theSmart Grid, 2012.

[4] UAV/UGV cooperation for surveying operations in humanitarian demining, 2013.

[5] GIS as a tool for enhancing the optimization of demand side management in residentialmicrogrid, 2015.

[6] Packt Books. Adding real-time weather data from openweathermap: Qgispython programming cookbook, May https://www.gislounge.com/, 2015.

[7] Stuart Borlase. Smart Grids. 2012.

[8] Jose L. P. Brittes, Paula S. D. Kayano Luiz C. Magrini, Osvaldo Rein Junior Ferdi-nando Crispino, Wagner S. Hokama Jose A. Jardini, and Luis G. Fernandez Silva.Substation smartizing: An iec based approach for utility smart analytics devel-opment. International Journal of Electronics and Electrical Engineering, 4:284–289,2016.

https://www.gislounge.com/

[9] The Caliper website, http://www.caliper.com/.

[10] The European Commission. Electricity network codes and guidelines,november https://ec.europa.eu/energy/en/topics/wholesale-market/

electricity-network-codes/, 2016.

[11] Larry Daniel, Paula Loree, and Angela Whitener. Inside MapInfo Professional.OnWord Press, New York; NY, 3. edition, 2002.

[12] Arup Dasgupta. The continuum: Big data, cloud and inter-net of things, august https://www.geospatialworld.net/article/

the-continuum-big-data-cloud-and-internet-of-things/, 2016.

[13] The Energinet.dk website, https://www.energinet.dk/.

[14] The European Network of Transmission System Operators for Electricity website,https://www.entsoe.eu/.

[15] ESRI. Arcgis enables real-time gis, http://www.esri.com/esri-news/arcnews/spring13articles/arcgis-enables-real-time-gis.

[16] E. Jun Zhu, Jun Zhu, Eric Zhuang, Chavdar Ivanov, and Ziwen Yao. A data-driven approach to interactive visualization of power systems. 26:2539–2546,2011.

[17] R. R. N. Kanapaka and R. K. Neelisetti. A survey of tools for visualizing geospatial data. In 2015 International Conference on Control, Instrumentation, Commu-nication and Computational Technologies (ICCICCT), pages 22–27, Dec 2015.

[18] Mohammed S. Kemal and Rasmus L. Olsen. Adaptive data collection mecha-nisms for smart monitoring of distribution grids. CoRR, abs/1608.06510, 2016.

[19] Sung Ah Kim, Dongyoun Shin, Yoon Choe, Thomas Seibert, and Steffen P. Walz.Integrated energy monitoring and visualization system for smart green city de-velopment. Elsevier, 22:51–59, 2012.

[20] Fangxing Li, Wei Qiao, Hongbin Sun, Hui Wan, Jianhui Wang, Yan Xia, ZhaoXu, and Pei Zhang. Smart transmission grid: Vision and framework. 1:168–177,2010.

[21] Hesen Liu, Jiahui Guo, Wenpeng Yu, Lin Zhu, Yilu Liu, Tao Xia, Rui Sun, andR. M. Gardner. The design and implementation of the enterprise level data plat-form and big data driven applications and analytics. In 2016 IEEE/PES Transmis-sion and Distribution Conference and Exposition (T D), pages 1–5, May 2016.

[22] M. Mangiameli and G. Mussumeci. Real time integration of field data into a gisplatform for the management of hydrological emergencies. XL-5/W3:153–158,2013.

[23] Dr Alan W. McMorran. An introduction to iec 61970-301 and 61968-11: The com-mon information model. Technical report, Institute for Energy and Environment,Department of Electronic and Electrical Engineering, University of Strathclyde,january 2007.

[24] Rosario Miceli, Salvatore Favuzza, and Fabio Genduso. A perspective on thefuture of distribution: Smart grids, state of the art, benefits and research plans.5:36–42, 2013.

62

http://www.caliper.com/

https://ec.europa.eu/energy/en/topics/wholesale-market/electricity-network-codes/

https://ec.europa.eu/energy/en/topics/wholesale-market/electricity-network-codes/

https://www.geospatialworld.net/article/the-continuum-big-data-cloud-and-internet-of-things/

https://www.geospatialworld.net/article/the-continuum-big-data-cloud-and-internet-of-things/

https://www.energinet.dk/

https://www.entsoe.eu/

http://www.esri.com/esri-news/arcnews/spring13articles/arcgis-enables-real-time-gis

http://www.esri.com/esri-news/arcnews/spring13articles/arcgis-enables-real-time-gis

[25] ESRI Developer Network. What is arcgis desktop?, http://edndoc.esri.com/arcobjects/.

[26] Pekka Pääkkönen and Daniel Pakkala. Reference architecture and classificationof technologies, products and services for big data systems. Elsevier, 2:166–186,2015.

[27] Maher Chebbo Pieter Vingerhoets and Nikos Hatziargyriou. The digital energysystem 4.0. Technical report, Smart Grids European Technology Platform, 2016.

[28] Pitney Bowes Software Inc. MapInfo Pro Version 12.5.1 User Guide, 2014.

[29] L. Qi, C. Wang, W. Zhou, and Z. Yang. Design of distribution scada systembased on open source gis. In Electric Utility Deregulation and Restructuring andPower Technologies (DRPT), 2011 4th International Conference on, pages 523–526,July 2011.

[30] The Statnett website, http://www.statnett.no/en/.

[31] Bruce Wollenberg, Jay Britton, Ed Dobrowolski, Robin Podmore, Jim Resek,John Scheidt, Jerry Russell, Terry Saxton, and Chavdar Ivanov. A brief history:The common information model, february http://sites.ieee.org/pes-enews/

2015/12/10/a-brief-history-the-common-information-model/, 2016.

[32] Kaile Zhou, Chao Fu, and Shanlin Yang. Big data driven smart energy manage-ment: From big data to big insights. 56:215–225, 2016.

[33] J. Zhu, E. Zhuang, J. Fu, J. Baranowski, A. Ford, and J. Shen. A framework-based approach to utility big data analytics. IEEE Transactions on Power Systems,31(3):2455–2462, May 2016.

[34] Xuan Zhu. GIS for Environmental Applications: A practical approach. Routledge,New York; NY, 2016.

63

http://edndoc.esri.com/arcobjects/

http://edndoc.esri.com/arcobjects/

http://www.statnett.no/en/

http://sites.ieee.org/pes-enews/2015/12/10/a-brief-history-the-common-information-model/

http://sites.ieee.org/pes-enews/2015/12/10/a-brief-history-the-common-information-model/

64

Paper B

Data Analytics for Low Voltage Electrical Grids

Maria Stefan, Jose G. Lopez, Morten H. Andreasen,Ruben Sanchez and Rasmus L. Olsen

The paper has been published in the:Proceedings of the 3rd International Conference on Internet of Things, Big Data

and Security - Volume 1: IoTBDS, pp.221-228, 2018.

© 2018 SCITEPRESS Digital LibraryThe layout has been revised and reprinted with permission.

I. Introduction

Abstract

At the consumer level in the electrical grid, the increase in distributed power gener-ation from renewable energy resources creates operational challenges for the DSOs.Nowadays, grid data is only used for billing purposes. Intelligent management toolscan facilitate enhanced control of the power system, where the first step is the abilityto monitor the grid state in near-real-time. Therefore, the concepts of smart gridsand Internet of Things can enable future enhancements via the application of smartanalytics. This paper introduces a use case for low voltage grid observability. Theproposal involves a state estimation algorithm (DSSE) that aims to eliminate errorsin the received meter data and provide an estimate of the actual grid state by replac-ing missing or insufficient data for the DSSE by pseudo-measurements acquired fromhistorical data. A state of the art of historical and near-real-time analytics techniquesis further presented. Based on the proposed study model and the survey, the teamnear-real-time is defined. The proposal concludes with an evaluation of the differ-ent analytical methods and a subsequent set of recommendations best suited for lowvoltage grid observability.

I Introduction

At the beginning of the 21st century, a massive improvement of Informationand Communications Technology (ICT) gave an opportunity for solving someexisting limitations of the electrical grid, while also reducing the operationalcosts [23]. This sparked people involved in the development of the futureenergy market to think of new concepts. Of these ideas, smart meters andsmart grid were the most popular, by adding ICT intelligence to the system,wherever useful.

These ideas led many countries to support various research programs inthe smart grid domain. Denmark, already having a long tradition in thegreen electricity market, published a set of recommendations for implement-ing these concepts in the report called Smart Grid in Denmark. One Danish fi-nanced research program is ForskEL [12], meant to support the developmentand integration of environmentally friendly power generation technologiesand grid connection.

One goal of ForskEL is to help the Distributed System Operators (DSOs)in making sensible decisions regarding future power grid planning and faultdiagnosis in near-real-time. This calls for the utilization of intelligent meth-ods for grid data visualization, as presented in [27].

The new challenge for the Danish DSOs arises as more distributed powergeneration is introduced at the low voltage grid level. This affects their abil-ity to monitor the state of the power grid without encountering operationalconstraints. One of the DSOs’ primary tools are to obtain full observability

67

Paper B.

of the low voltage grid, by making use of scalable data analytics, as intendedwith the Danish RemoteGRID project [21]. Hence, high-performance dataprocessing and analytical methods are fundamental for efficiently managingdistribution grid data.

Two relevant data types are considered in relation to the power grid:

• Geographic data: electrical network structure (cables, transformers, sub-stations, meters) and their geographical coordinates;

• Measurement data: three-phased generic grid measurements from eachload or connection point containing multiple loads (voltage, current,consumption).

This paper introduces a study of analytical methods suitable for obtain-ing low voltage grid observability. The paper is organized as follows: Sec-tion II presents the flow of data in a smart grid application. The proposedstudy is presented in Section III and it underlines the advantages of pseudo-measurements and state estimator for the smart grid scenario. In Section IV,both generic and state of the art analytic methods will be presented. Giventhe chosen case study and background, the most suitable analytics will beemphasized in Section V, along with the definitions of bulk and stream datatypes. Section 5 will summarize the aforementioned study requirements withfuture action plans for testing the above concepts.

II Study background

The underlying application structure is defined based on the requirementsimposed by the analytical methods suitable for the state estimation algorithmintroduced in Section III. In this study, the application structure is proposedas a client-server application, based on the IEC 61868-100 standard [8]. Thedata flow is depicted in Figure B.1. The IEC standard is meant to provideguidelines regarding message exchange and interface specifications for utilityenterprise distribution systems. Consequently, the key terms are clarified asfollows:

• Advanced Metering Infrastructure (AMI) [32]: main data source in asmart grid, characterized by a large number of nodes (meters) locatedat customer premises;

• Meter Data Management (MDM) [17]: software entity that involves thestorage and management of the AMI data. This includes the DatabaseManagement System (DBMS);

• Enterprise Service Bus (ESB) [24]: software-based integration layer spec-ifying a standardized communication interface facilitating services (rout-

68

II. Study background

Client

UserAnalyticsDBMS

Server

AMI

Fig. B.1: Data flow and exchange of automatic events according to IEC 61968-100.

ing, mediation, recording of data etc.) via standard event-driven mes-saging. The ESB middelware works as an adapter between differentdata formats and protocols in a Service Oriented Architecture (SOA).

Data is generated at the AMI (server entity) as an encoded packet, which isthen decoded at the MDM level and sent to a database management system(DBMS) for storage via XML messaging [22]. In this back-end architecture[31], the DBMS is defined as an integration feature, which provides the ESBmiddleware with raw data to be sent to the analytics module for processing.A processing unit in the analytics module extracts the desired information tobe displayed for the user (client entity). In the smart grid context, the useris usually located in the DSO control center. The event progression of datais storing - information extraction - information display. These events takeplace in a cyclic manner and thus, they are referred to as automatic events.

69

Paper B.

User

Analytics

DBMS

Fig. B.2: Data exchange of interactive events based on client request.

The data exchange sequences are not solely one-way. In case the client(user) detects unusual patterns or missing information in a certain geograph-ical area, additional data from that specific area (or specific meters) can berequested for enhanced monitoring purposes. If so, the data flow is basedon so-called interactive events, as shown in Figure B.2. The client’s request formore detailed information is transmitted to the DBMS via the ESB, to searchif there is a match for the requested data in the database. If a match is found,a reply is sent to the client for display and visualization. If not, the data re-quest may be forwarded to the AMI, which will configure the meters to sendthe required data. Timing is crucial in the DSOs decision making process andis notably affected by delays in the transmissions from data collection to datadisplay. Requesting certain information all the way from the AMI will resultin additional delays due to the increased number of messaging sequencesbetween entities.

As a part of the analytics module, the next section will introduce theDistributed System State Estimator (DSSE).

70

III. Study outline: Low voltage grid observability

III Study outline: Low voltage grid observability

Low voltage grids are undergoing a transformation from a passive to a moreactive role in the electrical network. Traditionally, conventional large gasor coal power plants, among others, are the source of electrical power gen-eration [30]. After being transmitted at a high voltage level, the energy isdistributed to supply the loads in the system. Lately, the penetration of dis-tributed generation, especially from Renewable Energy Resources (RES), atthe low voltage level has increased. It creates operational challenges for theDSOs since the low voltage grid was not designed to operate under such con-ditions. For example, generation peaks from RES do not necessarily matchpeaks of consumption, introducing power flows from the low to the highvoltage level.

In order to address operational concerns, the DSOs require advancedmanagement tools. Grid monitoring is the first step towards a more reliableoperational approach [3]. In fact, nowadays, the low voltage grid electricalparameters are not monitored in the DSOs control centers. Monitoring thesystem allows DSOs to determine whether or not the system is operatingunder normal conditions. A system is considered to operate under normalconditions if all the loads can be supplied without violating any operationalconstrains [3].

A. Low voltage grid state estimation - LV DSSE

Grid observability depends on where the measurement points are placedalong the electrical grid. In the case of low voltage grids, these measure-ments are provided by the smart meters. However, the information extractedfrom the meter’s data contains errors due to various factors, such as commu-nication issues or measurement deviations in the devices. Thus, as a first stepin control centers, efficient data analytics are required to properly determinethe state of the electrical grid. The state is defined as ”known” if the volt-ages and phase angles with respect to a certain voltage and angle referenceare known at every node (point where two or more circuit elements meet)[3].The process in charge of eliminating errors and providing the best esti-mate of the system state in distribution systems is the so-called distributionsystem state estimation (DSSE) [5].

Figure B.3 shows the block diagram of the observability analysis per-formed based on the raw measured data. This analysis determines if thesystem state can be estimated based on the set of acquired near-real-timereadings. For example, few or non-existing measurements are sometimesprovided from a specific geographical area of the system. This implies thatthe available data is insufficient to successfully estimate the state of the sys-tem. In that case, other data analytics methods are needed, where the unavail-

71

Paper B.

Fig. B.3: Evaluating observability based on the field near-real-time measurements.

able near-real-time measurements are substituted by the so-called pseudo-measurements obtained from historical data [18].

B. Pseudo-measurements

Traditionally, pseudo-measurements have been obtained from standardizeddaily load and generation profiles (DLP-DGP). Those are created for differentcustomer classes based on socio-demographic factors [19]. However, otherapproaches seeking more precise accuracy have been developed in the litera-ture. Artificial Neuronal Networks (ANN) are used in [11]. Besides, differentclustering techniques were utilized as it is the case of k-means [6], princi-ple component analysis [1], spectral clustering method [4] or finite mixturemodel [28], among others.

New solutions are to be studied in order to provide robust pseudo-measurementsfor low voltage grid applications based on the utilization of AMI data. Un-predictable behavior from RES is a challenge where efficiency in terms ofthe amount of stored data needs to be considered given the large number ofnodes at the low voltage level.

IV Analytic Methods

AMI data is by definition part of the Internet of Things (IoT) umbrella, in thesense that smart meters act as sensors in the electrical grid infrastructure. IoTdata analytics is characterized by autonomous or semi-autonomous examina-tion of data, employing sophisticated techniques and tools, typically beyond

72

IV. Analytic Methods

those of traditional Business Intelligence (BI). These techniques help to re-duce complex data sets into actionable insights, enhance and empower BIdecision support systems. By this token, some traditional analytics and algo-rithms include data mining, machine learning, pattern matching, forecasting,visualization, semantic analysis, sentiment analysis [13].

Analytics are classified by two main categories: historical and near-real-time analytics.

• Historical analytics: based on the past data values. Data-at-rest corre-sponds to batch data processing;

• Near-real-time analytics: based on the present. Data-in-motion equalsstream data processing.

A. Historical Analytics

Four traditional types of historical analysis are presented in the followingsubsections. They are a trade-off between the provided information valueand the implementation difficulty. This is illustrated in [7].

A..1 Descriptive

This type of data analysis is used to provide insight into past events, byidentifying overall themes and patterns. Descriptive analytics is commonlyclassified as BI and is the de facto standard analytics methodology. Typi-cal outputs include dashboards, reports and status emails stating historicalobservations by summarizing raw data for human interpretation. These aremainly obtained through methods such as data mining and data aggregation.

An example of descriptive analytics can be found in [20], where dailyprofile of consumption trends are obtained by means of data aggregation.This helps to understand daily habits of consumers and, at the same time, toinsure the privacy of end users through data anonymization [10].

A..2 Diagnostic

Diagnostic analysis helps answering questions like "Why was a certain eventtriggered", by providing a deep understanding of a limited problem spacevia in-depth data analysis, discovering the root causes and characteristics ofan event. Advancing from aggregate and summary information to detaileddata, based on specific focus attribute(s), is done via selection and queryingof data sets. Data granularity defines the limit for the analytic level of detail.The resulting output is typically an analytic dashboard.

Correlation methods are part of obtaining a diagnosis analysis. The re-view in [25] proposes a method for characterizing power system loads by thecorrelation between load demand and weather variables.

73

Paper B.

Collect

Advanced

Metering

Infrastructure

Diagnose &

Predict

Querying

Correlating

Statistical modelling

Forecasting

Drill-down/through

Prescribe

Business rules

Optimization

Visualization

Big Data

Describe &

Pre-process

Filtering

Aggregation

Mining

Storage - Database Management System

Fig. B.4: Proposal of streaming analytics architecture for low-voltage electrical grids [33].

A..3 Predictive

Predictive analytics is about foreseeing the future based on historical datapatterns. Future predictions and scenarios come from data mining, machinelearning and statistical modeling of raw data. Thus, actionable insights areobtained via plausible estimates of future outcomes. Typical deliverablesare in the form of predictive forecasts based on probabilistic and correlationanalysis.

Load forecasting is a common use case of predictive analysis in smartgrids [10]. The study made in [2] approaches a forecast method which isbased on a combination of ANNs and time series data models. Load fore-casting can be achieved using not only correlations, but also through ma-chine learning solutions, such as the MapReduce processing model [26] [14].MapReduce allows for massive scalability across a cluster of computers, forlarge data sets (in the range of Terabytes), which is a suitable solution in caseof AMI infrastructures.

A..4 Prescriptive

The primary focus of prescriptive analytics is to provide real-world recom-mendations. Datasets are evaluated via analytical models and the preferredcause of action for each specific event is selected. Then the result, in the formof explicit actionable information, is presented for human interaction, typi-cally making the final decision on acceptance or rejection. Hence, prescriptiveanalytics takes a step further than predictive analytics by reducing complexdata and algorithms to non-technical descriptors for immediately recogniz-able advice on predicted future outcomes. The analysis aids the decision-making process, having the potential to both maximize positive outcomes aswell as prevent undesirable events [16].

Simultaneous utilization of multi-source datasets includes historical andreal-time data, transactional and big data analytics, that affect marketing

74

V. Main findings and discussion

strategies [9]. For example, one significant tool to help utility companiesnavigate towards a smart grid platform is the Vitria IoT Analytics Platform,reported in [33]. This white paper states that a combination of prescriptiveanalytics and smart decisions provide the highest throughput in the analyticsvalue chain.

B. Near-Real-Time Analytics

The resilience of the power grid is part of the future requirements for evolv-ing towards intelligent grids. The main motivation for near-real-time analyt-ics lies in the lack of limited grid functionality to timely detect and preventfailures. This extends to the discovery of natural disasters or criminal actionsthat might have caused the failures. Therefore, these can be prevented bymaking use of real-time intelligence [33].

B..1 Streaming Analytics

Near-real-time decision support can be provided via data-in-motion pre-database processing, inspection, correlation and analysis. It enables instanta-neous management, monitoring, and continuous statistical analysis of data.Introducing real-time KPI overview, immediate access to metrics, and report-ing, improves reaction time and accelerates decision-making.

Streaming analytics provide value from the data in a similar manner astraditional historical analytics. The value of streaming data decreases non-linearly over time, meaning that events should be reacted upon quickly, innear-real-time. The progression from historical methods comes as analyticsare no longer performed ”at-rest”. Instead, data is processed before it isstored and therefore the decision-making process becomes timely and moreefficient [15] [29]. A summary of the modules involved in the data streamingbased on the surveyed analytics types is shown in Figure B.4. This figureshows that the same principles as in historical analytics can be applied tostreaming data.

V Main findings and discussion

The study presented above emphasized the importance of introducing ana-lytical methods to monitor the status of low voltage electrical grids and toplan future grid reinforcements. Historical data is used to create pseudo-measurements, aiming to fill in missing or erroneous data received from asmart grid infrastructure.

Given the back-end client-server architecture presented in Section II, theautomatic ingestion of data can be defined as a ”stream of data”:

75

Paper B.

Table B.1: Advantages and disadvantages of using historical and near-real-time analytics forproviding data to the DSSE.

Analytics Pros Cons

Historical(contextawareness)

• provide insight by uncoveringdata patterns and trends

• accuracy and realiability depen-dent on time

• quickly accessible and detailed(available and verified data)

• most machine learning algo-rithms do not deal with temporaleffects

• clarity by presentation of re-duced complex data sets - thoroughpresentation of large data sets

• reliance on batch processingand consequently limited by theresulting update intervals

Near-real-time(situationawareness)

• detects gross errors - accuracy • highly dependent on the delaysin the communication network

• avoid latency from filtering diskdata

• difficult to adapt to platform andhardware requirements

• detect emerging correlationsbetween multiple data sets

• risk of incorrect analysis viaimplementation dependency

• immediate pre-database dataavailability

Near-real-time measurements are characterized as a continuous, fast changing andvoluminous data flow, commonly known as stream.

To support the above-mentioned definition, the notion of near-real-timedata can be given in the context of the data flow architecture in Section II andthe use case presented in Section III:Assuming that the data packets sent from the low voltage grid arrive consec-utively with a fixed period of time, then a near-real-time data stream can bedefined as: a data packet characterized by the arrival granularity and received in atimely manner at the user side. Timing is then relative to the types of events involvedin the data flow: automatic or interactive.

The analytical methods involved in the DSSE algorithm are based on bothhistorical and near-real-time data. Due to their timely nature, the near-real-time measurements are more reliable and accurate than the historicalones. Therefore, the DSSE needs near-real-time data, that should be pre-processed in order for the estimator to ”understand” it, equivalently to thestreaming analytics procedures shown in Figure B.4. There are typically notenough near-real-time measurements available to successfully perform theDSSE. Therefore, there is not enough data to provide full grid observabil-ity. In order to fill in the gaps of missing information, pseudo-measurementscan be created by requesting raw data that has been previously stored in a

76

database. The requested information can therefore be extracted by means offiltering, mining or querying, making it comprehensible for the DSSE. In thiscase, the most suitable analytics are descriptive.

A summary of pros and cons of the aforementioned analytics for the DSSEis presented in Table B.1. The novelty of this study is based on the integrationof traditional analytics into the energy-related field, which consists of theDSSE algorithm. As historical based analytics are useful to build periodicreports for strategic and long-term decisions, they are also limited by thetemporal effects. Historical data may not give a true pattern of a data trend, ifthis has changed with time. While near-real-time analytical tools can addressthe temporal dependency, they are also platform sensitive.

VI Conclusion

This study addresses the challenges for choosing suitable data analytics meth-ods in the domain of low voltage smart grids. DSSE is an analytical methodfor providing a reliable source of information related to the state of the grid,by filtering the raw data and detecting gross errors. Ideally, DSSE makesuse of near-real-time data to provide a successful estimation. In many cases,this data is insufficient or non-available, so pseudo-measurements generatedfrom historical data will fill in for the lack of information. Traditional his-toric analytics can build predictive outputs useful for the DSSE, but there isa higher error probability in the pseudo-measurements.

By this token, the data analytics module should be built on a platformthat can accommodate for both historical and near-real-time analysis. Thenext step in this research is to test the functionality of a DSSE algorithm andanalyze the capabilites of processing large amounts of historical batch data.At the same time, the test aims to characterize the performance and bottle-necks of parallel processing of both stream and batch data types, taking intoaccount parameters such as memory usage, processing time and in-memoryprocessing behavior.

Acknowledgment

This work is financially supported by the Danish project RemoteGRID, whichis a ForskEL program under Energinet.dk with grant agreement no. 2016-1-12399.

References related to paper B

[1] Joana M Abreu, Francisco Câmara Pereira, and Paulo Ferrão. Using patternrecognition to identify habitual behavior in residential electricity consumption.Energy and buildings, 49:479–487, 2012.

[2] M. A. Abu-El-Magd and R. D. Findlay. A new approach using artifi-cial neural network and time series models for short term load forecasting.3:1723,1724,1725,1726, 2003.

[3] Ali Abur and Antonio Gomez Exposito. Power system state estimation: theory andimplementation. CRC press, 2004.

[4] Adrian Albert and Ram Rajagopal. Smart meter driven segmentation: What yourconsumption says about you. IEEE Transactions on Power Systems, 28(4):4019–4030, 2013.

[5] Arash Alimardani, Francis Therrien, Djordje Atanackovic, Juri Jatskevich, andEbrahim Vaahedi. Distribution system state estimation based on nonsynchro-nized smart meters. IEEE Transactions on Smart Grid, 6(6):2919–2928, 2015.

[6] Ignacio Benítez, Alfredo Quijano, José-Luis Díez, and Ignacio Delgado. Dy-namic clustering segmentation applied to load profiles of energy consumptionfrom spanish customers. International Journal of Electrical Power & Energy Systems,55:437–448, 2014.

[7] The four types of analytics, http://www.ciandt.com/card/

four-types-of-analytics-and-cognition, 2014.

[8] International Electrotechnical Commission. Application Integration at ElectricUtilities - System Interfaces for Distribution Management - Part 100: Implemen-tation Profiles. Technical Report 61968-100, International Electrotechnical Com-mission, July 2013.

[9] Houda Daki, Asmaa El Hannani, Abdelhak Aqqal, Abdelfattah Haidine, andAziz Dahbi. Big data management in smart grid: concepts, requirements andimplementation. Journal of Big Data, 4(1):13, Apr 2017.

[10] Panagiotis D. Diamantoulakis, Vasileios M. Kapinas, and George K. Karagianni-dis. Big data analytics for dynamic energy management in smart grids. 2:94–101,2015.

[11] MB Do Coutto Filho, JCS Souza, RSG Matos, and M Th Schilling. Preservingdata redundancy in state estimation through a predictive database. In ElectricPower Engineering, 1999. PowerTech Budapest 99. International Conference on, page271. IEEE, 1999.

[12] The Energinet.dk website, https://www.energinet.dk/.

[13] Gartner it glossary - advanced analytics, https://www.gartner.com/

it-glossary/advanced-analytics, 2017.

[14] GJSissons. Adaptive mapreduce: Part 1, https://developer.ibm.com/hadoop/2014/09/28/adaptive-mapreduce-part-i/, 2014.

78

http://www.ciandt.com/card/four-types-of-analytics-and-cognition

http://www.ciandt.com/card/four-types-of-analytics-and-cognition

https://www.energinet.dk/

https://www.gartner.com/it-glossary/advanced-analytics

https://www.gartner.com/it-glossary/advanced-analytics

https://developer.ibm.com/hadoop/2014/09/28/adaptive-mapreduce-part-i/

https://developer.ibm.com/hadoop/2014/09/28/adaptive-mapreduce-part-i/

[15] Daniel D. Gutierrez. InsideBIGDATA - Guide to Streaming Analytics. Technicalreport, Impetus Stream Analytix, 2016.

[16] Descriptive, predictive, and prescriptive analytics explained, https://halobi.com/blog/descriptive-predictive-and-prescriptive-analytics-explained/,2017.

[17] IEC/TC. Application Integration at Electric Utilities - System Interfaces for Dis-tribution Management - Part 9: Interfaces for meter reading and control. Techni-cal Report 61968-9, International Electrotechnical Commission, October 2013.

[18] Amin Khodabakhshian, Rahmatollah Hooshmand, and Yaser Raisee-Gahrooyi.A new pseudo load profile determination approach in low voltage distributionnetworks. IEEE Transactions on Power Systems, 2017.

[19] Vladan Krsman, Bojan Tesanovic, and Jovan Dojic. Pre-processing of pseudomeasurements based on ami data for distribution system state estimation. 2016.

[20] Xiufeng Liu, Lukasz Golab, Wojciech Golab, Ihab F. Ilyas, and Shichao Jin. Smartmeter data analytics. 42:1–39, 2016.


[22] Dr Alan W. McMorran. An introduction to iec 61970-301 and 61968-11: Thecommon information model. 2007.

[23] Rosario Miceli, Salvatore Favuzza, and Fabio Genduso. A perspective on thefuture of distribution: Smart grids, state of the art, benefits and research plans.5:36–42, 2013.

[24] Scott A. Neumann and Terry D. Nielsen. Cim interoperability challenges. pages1–5, 2010.

[25] Muhammad Qamar Raza and Abbas Khosravi. A review on artificial intelligencebased load demand forecasting techniques for smart grid and buildings. 50:1352–1372, 2015.

[26] Nikzad Babaii Rizvandi, Javid Taheri, and Albert Y. Zomaya. A study on usinguncertain time series matching algorithms in map-reduce applications. CoRR,abs/1112.5505, 2011.

[27] M. Stefan, J. G. Lopez, M. H. Andreasen, and R. L. Olsen. Visualization tech-niques for electrical grid smart metering data: A survey. In 2017 IEEE Third Inter-national Conference on Big Data Computing Service and Applications (BigDataService),pages 165–171, April 2017.

[28] Bruce Stephen, Antti J Mutanen, Stuart Galloway, Graeme Burt, and Pertti Jär-ventausta. Enhanced load profiling for residential network customers. IEEETransactions on Power Delivery, 29(1):88–96, 2014.

[29] Weighing the pros and cons of real-time big data analytics,https://www.techopedia.com/2/31245/technology-trends/big-data/

weighing-the-pros-and-cons-of-real-time-big-data-analytics, 2017.

79

https://halobi.com/blog/descriptive-predictive-and-prescriptive-analytics-explained/

https://halobi.com/blog/descriptive-predictive-and-prescriptive-analytics-explained/

https://www.techopedia.com/2/31245/technology-trends/big-data/weighing-the-pros-and-cons-of-real-time-big-data-analytics

https://www.techopedia.com/2/31245/technology-trends/big-data/weighing-the-pros-and-cons-of-real-time-big-data-analytics

[30] David Trebolle, Per Hallberg, Gunnar Lorenz, Pavla Mandatova, and Jorge TelloGuijarro. Active distribution system management. In Electricity Distribution(CIRED 2013), 22nd International Conference and Exhibition on, pages 1–4. IET, 2013.

[31] A beginner’s guide to back-end development, https://www.upwork.com/

hiring/development/a-beginners-guide-to-back-end-development/, 2017.

[32] Noelia Uribe-Pérez, Luis Hernandez, David de la Vega, and Itziar Angulo. Stateof the art and trends review of smart metering in electricity grids. 6:68, 2016.

[33] Vitria. Advanced Analytics for Energy Utilities: The Fast Path to a Smart Grid.Technical report, Vitria Technology, 2015.

80

https://www.upwork.com/hiring/development/a-beginners-guide-to-back-end-development/

https://www.upwork.com/hiring/development/a-beginners-guide-to-back-end-development/

Paper C

Exploring the Potential of Modern Advanced Meter-ing Infrastructure in Low-Voltage Grid Monitoring

Systems

Maria Stefan, Jose G. Lopez and Rasmus L. Olsen

The paper has been published in the:IEEE International Conference on Big Data, 2019.

© 2019 IEEEThe layout has been revised and reprinted with permission.

I. Introduction

Abstract

Energy systems are evolving towards 100% green energy production. The share ofgreen energy in electrical distribution systems is progressively increasing, implyingalso an increment on the number of renewable energy units in the low-voltage grid.Following this trend, thousands of consumers connected to the power grid in a de-centralized manner become small producers, changing the traditional paradigm ofenergy distribution from top to bottom. Currently, the modern Advanced MeteringInfrastructure (AMI) enables the possibility of collecting several types of status datafrom the end-users which Distribution System Operators’ (DSOs) can use to theiradvantage to optimize management and planning operations. As a part of this opti-mization, having a spatial overview over the low-voltage grid can speed up the moni-toring processes and allows to obtain a real-time insight on what is happening in thegrid, compared to the traditionally used analysis methods. Many business structuresfor smart grid cyber physical systems are looking into how to integrate advanced datamanagement models. Such models should provide the means for obtaining meaning-ful data visualization where only the relevant data is timely processed, filtered andvisualized for the operators to efficiently react to grid anomalies in real-time.

The purpose of this paper is to investigate how to efficiently design a monitor-ing/visualization system for low-voltage electrical grids based on the DSOs’ needsand feedback. The proposed system implementation stands on emulating an existinggeographic scenario by a virtual AMI integration. The efficiency of the prototype isevaluated versus the traditional monitoring operations derived from user experiencestudies, such as a reduction in time to perform a specific anomaly detection operation.Furthermore, the advantages of spatial awareness are meant to further strengthenthe motivation for integrating measurements into a Geographic Information System(GIS) environment.

I Introduction

The gradual replacement of traditional energy sources with green/renewableenergy, especially in countries with a long tradition in the green electricitymarket, such as Denmark, has sparked people to think about new concepts.Many business models are nowadays exploring integration methods that canaccommodate for the interaction between wide topics of interest, such as:intelligent power grids, big data, data analytics and visualization. At the en-terprise level, it is challenging to scale up with these rapidly advancing tech-nologies, without modifying too much the already working business struc-ture. The research community can help in this sense with designing andtesting modular systems that can be easily integrated within companies.

In Denmark, the electricity transmission network is built around an alter-nating current network (AC) of 132 kV, 150 kV, 220 kV and 400 kV. The West-

83

Paper C.

ern Danish Power System is a regular transit area with large interconnectionsto Norway (HVDC), Sweden (HVDC) and Germany (AC) [2]. The systemcan already operate with 20% wind and 50% CHP [3]. However, the energyplan in Denmark is to introduce 100% renewable energy by 2050, as a partof the smart grid strategy [4]. As new renewable energy units are connected(local CHP plants, wind turbines), the grid becomes more active because theconsumers can also act as small energy suppliers (prosumers) and the gridevolves towards a decentralized architecture [7]. This will influence the waythe traditional power grid operates.

Currently, the incoming measurement data from the advanced meteringinfrastructure (AMI) is used only for billing purposes. However, the alreadydeployed metering infrastructure allows collecting many more grid parame-ters. The availability of these parameters offers the possibility of implement-ing automated monitoring solutions for real-time anomaly detection in thegrid, which have been out of reach in the past. Nowadays, the most commonissues reported by the consumers to DSOs are: power outage at the consumerend and flickering of lights.

It is expected that with the high penetration of renewables there will bean increase in reported issues by the consumers. Measurement data can beutilized to characterize the consumption patterns and understand the behav-ior of the users. In this case, the need for a more efficient monitoring andoperation of the grid becomes essential.

The Danish RemoteGRID project [5] [18] is a research initiative that aimsto support the Danish DSOs in gaining a visual overview over their low-voltage electrical grid, using GIS-based systems. As a starting point, it isimportant to identify some of the DSOs’ challenges in a typical work day,based on some concrete examples. In this article, a near-real-time monitoringsystem is proposed and tested in comparison with the examples from userexperience studies. The aim is to show that a GIS-based monitoring solu-tion helps the human operators in identifying measurement anomalies fasterand more effectively than this is currently done, thereby improving the gridmonitoring procedures.

II Related Work

Given the case of small producers described in Section 1 and the DSOs’ needfor adequate grid monitoring, there is a clear motivation for investigatingdata management and visualization techniques in power systems.

Monitoring systems. Electrical networks are getting increasingly more com-plex with the implementation of Information and Communication Technolo-gies (ICT). These so-called Smart Grids collect data about power consump-

84

II. Related Work

tion, which is used to diagnose problems in the network and prevent out-ages. The resulting large amount of data makes it more difficult to monitorthe electrical grid.

Some available monitoring systems of real-time operational data use GIS[6] for data modeling (ArcGIS) and visualization (Web GIS). Apart from get-ting an overview of the grid status, the role of monitoring may also implyfault location. The article in [10] proposes such a system that is able to lo-calize faults and their cause in a medium-voltage power grid, by analyzingonly the electric current. The alarms are presented for human interaction bymeans of a SCADA (Supervisory Control and Data Acquisition) system [9].

Common challenges in the existing monitoring systems developed forsmart grids relate to the cost of the required hardware and software com-ponents, as well as scalability issues that arise due to finer data granularityand the amount of smart meters deployed.

Real-time data visualization and GIS. Traditionally, real-time data visualiza-tion and geographical information systems (GIS) in smart grids are decou-pled [22]. Visualization is usually achieved through dashboards, while GISis the tool to represent the electrical grid topology. The AMI generates dif-ferent types of measured values: three-phased voltages, currents, active andreactive powers. The key benefit consists in assigning these values to theircorresponding measuring points on a GIS map and thus, providing an inte-grated solution for data monitoring.

A cloud-based framework for visualizing time-series energy readings froman electrical grid infrastructure is proposed in [20]. The application was fo-cused on creating status dashboards using Tableau software in connectionwith the Apache Hadoop platform. Using timestamped measurements forfault/event detection in the electrical power system for a frequency monitor-ing network (FNET) is described in [11].

The paper [8] highlights the importance of spatial awareness on a visu-alization platform by demonstrating the spatio-temporal features of Quan-tumGIS (QGIS). An animation that used Time Manager plug-in made it pos-sible to display real-time values in QGIS.

Big data management in smart grids. The variety in measured data valuescomes from deploying renewable energy resources (RES) in the low-voltagegrid, interfering with its usual operation. This translates into the need forunderstanding the extent and behavior of the small producers in order tokeep track of them, by processing and analyzing the available measurementdata. Stream computing [12] is one of the main approaches for data analysisin real-time, given the amount of data generated per day in a smart grid(approximately 1 terabyte according to [24]). This is done through somededicated distributed stream computing platforms, such as Apache Storm

85

Paper C.

[24] [13], StreamBase [25], Apache Spark [23], etc.Other research initiatives have involved cloud computing frameworks.

Nephele has been used in [17] to simulate data incoming from 1000000 smartmeters in order to provide pricing updates every 10 seconds. Pricing strate-gies are also proposed in [14], using multiple cloud compatibility for streamdata computing, and in [17] through Infrastructure-as-a-Service (IaaS) cloudsfor parallel stream processing. These solutions point out the importance ofscalability in a big data management system.

Different data storage possibilities are supported within distributed com-puting, which are meant to ease the real-time data analysis process. A smartmeter data analysis system was developed in [15] by means of PostgreSQLdatabase and MADLib in-database machine learning library. This work isfollowed by a similar article presented in [16], where the processing of datais done using in-memory tables.

III Model and Simulation

Based on the aforementioned literature review, this paper will focus on de-signing and testing the efficiency of a monitoring system based on humanneeds and feedback. Two sequence models were identified to represent theintents and steps of activities regarding the low-voltage grid. The sequencesdescribe the troubleshooting process when a consumer calls with no power,or how to identify the geographical area affected by an error.

The designed system is currently costly and challenging to deploy in reallife, where it is not certain that the clocks (and thereby measurements) areproperly synchronized. This makes simulation the proper tool for assess-ment of the near-real-time functionality, where results can be compared in asynchronized setting. Evaluating and comparing the sequence models beforeand after the simulation will characterize the impact of GIS on the monitoringsystem.

A. Case description - household power outage

The most frequent reports received by the DSOs are customers calling into report the loss of power in their households. This user story has beenidentified by means of interviews with one of the Danish DSOs.

In this scenario, the DSO’s task is to figure out whether they are respon-sible for the outage, by checking the three-phased voltage values in the cor-responding meter assigned to a specific address. The DSO is able to demandinformation regarding 45 different values recorded by the meter. For this sce-nario, the DSO is merely interested in the three-phased voltages. Selecting

86

III. Model and Simulation

Search consumer's address

Fault at DSO end

Fault at consumer end

Select meter ID

Check 3-phased voltages

Find meter on

Geocortex map

Find neighboring meters

connected to the same

cable box

Check their 3-phased voltages

NO

Fig. C.1: Sequence model for identifying if the outage at the consumer level is the DSO’sresponsability or the consumer’s.

these values and sending a demand to the smart meter takes around 45 -60 seconds due to a slow radio frequency connection between the meters. Ifthe customer is found without power, the DSO tries to establish whether hisneighbors have the same problem.

The meter is located on a Geocortex map, together with the neighboringmeters connected to the same physical cable box. The same procedure ofchecking the three-phased voltage values is applied to the identified neigh-bors. If any of the neighboring addresses is also without electricity, the outageis the responsibility of the DSO.

The sequence model for evaluating the power outage in a household is

87

Paper C.

DSO

Fig. C.2: Data flow and information exchange between the meter provider and the DSO.

presented in Figure C.1. The lightnings represent alerts for the operationsthat are the most time consuming, due to the risk of choosing a wrong ad-dress or finding the relevant voltage measurements from the meters.

Throughout the interview, it was found that different software programsare used by the DSOs to assess different aspects of the low voltage grid,depending on specific problems. The two most commonly used programsare MidtVest Teknikerportal and Geocortex. Together these two were usedto locate the errors when customers called, claiming they have no power.MidtVest Teknikerportal is a technical program used to explore whether thereare any fluctuations in the three phases on the given address. Geocortex isthen used to see whether any of the neighboring houses have any fluctuationsas well and determine if there are any problems with the cable box.

The effects of added renewable energy such as personal windmills or solarpanels are also calculated using SonWin, to determine whether an outage ata consumer is the DSO’s responsibility or the consumer’s.

The next step is to simulate a near-real-time system, which can monitorthe received voltage measurements by means of a GIS map.

B. Data flow

Figure C.2 shows the flow of data in the power grid. The readings from thesmart meters are transferred from the AMI provider to the DSO over a radionetwork (it can be mesh, power line communication, fiber). All types of radionetworks have the common characteristics they were developed to supportonly the billing aspect, meaning that in most cases a low capacity network has

88


Fig. C.3: VAMI interface overview diagram.

been used. The AMI provider has information about the grid topology andparameters, customers (with or without RES), type of customers, historicalenergy readings and other value types. This data is filtered and analyzedaccording to the needs and requirements of the DSO: billing, planning ormonitoring.

An efficient monitoring system usually requires additional measurements,such as three-phased voltages in the nodes and cable currents. The chal-lenge lies in sending a large amount of varied information over the radionetwork, which overloads the network and slows down the monitoring pro-cess in near-real-time. At the same time, acquiring a large amount of mixedmeasurements overloads the map intended for visualizing this data.

It is important to analyze the work flow of the DSOs’ routine operations inorder to establish how much information should be displayed in the monitor-ing system and which of the steps in the work flow are most time consuming.It is hypothesized that spatial awareness leads to an effective monitoring byminimizing subsidiary procedures.

C. VAMI

The performance of a real life infrastructure of smart meters (as illustratedin Figure C.2) is emulated by a generalized software component called Vir-tual Advanced Metering Infrastructure (VAMI). The main functionality of VAMIis to intake and delay messages/signals from an output block in the usedSimulator (Simulink, in this case), as shown in Figure C.3.

The working principle of VAMI is based on its ability to handle a largeamount of incoming data on many different ports. In order to satisfy thisrequirement, Java Socket Selectors from Java NIO are used [1] [21]. Theinterfaces between the different components are listed in Table C.1.

In order to evaluate the performance of the monitoring data system, thenear-real-time functionality has been represented as a playback of alreadyrecorded historical data measurements provided by this virtual provider. In-coming data is outputted to a SQL table, where the playback functionality

89

Paper C.

Table C.1: Main interfaces in the VAMI architecture

Interface Type Comment

A - Smart meter datainput

UDP - any vector of data el-ements without separationcharacters, limited to one packetper vector element

Configurable port ranges

B - Simulator timesynch signal

UDP - double value of time insimulator at sending time

Configurable port - optional touse this synchronization methodvia VAMI configuration file

C - CDF for randomnumbers

Time series separated viacomma

Currently no implemented limitto the time series - limited byJava’s internal capacity to storedata in a vector

D - Configuration file XML - proprietary Set via first parameter whenstarting VAMI

E - Adapted output Output signals adapted to agiven sink - interfaces depen-dent on output sink. Defaultis UDP, raw: unmodified datareceived on Interface A

Configurable in VAMI configu-ration file

G - Real time synch System clock Alternatively to using simula-tion synchronization, VAMI canbe time synchronized via thesystem’s real/time clock if sim-ulator or any other componentis also time synchronized thisway

accesses the stored historical data to create a dynamic display on the WebGISmap. The data set is gradually overwritten with the runtime rate of VAMIand, as a result, it is assumed that the raw incoming data is near-real-time.Timing plays a vital role in a monitoring system and it is relative to the in-terfaces between different entities and the processes that take place in eachentity module. This can be seen in a real life scenario, as shown in FigureC.2.

D. WebGIS

Smart grid data for approximately 1000 households/smart meters is simu-lated and displayed in a GIS environment. The purpose of the displayinginformation on a WebGIS map is to achieve a visual perspective over thelow-voltage electrical grid. This means that a DSO is able to understand theconsumer’s behavior, as well as detect if a certain area of the grid presents

90


(a) Initial state inactive meters.

(b) Near-real-time GIS map update.

(c) Pop-up example containing three-phased voltagevalues and customer address.

Fig. C.4: Update and application of the WebGIS map in the monitoring system

anomalies. For example, supply voltage variations in households should notexceed ±10 % of the nominal voltage value, which represents the normaloperating conditions in the electrical grid [19].

Initially, the data set contains only 0 voltage values, meaning that all the

91

Paper C.

Filter "high meters" on GIS map

Check 3-phased

voltages

Fault at DSO end

Grouped

Fig. C.5: Sequence model for identifying if the outage at the consumer level is the DSO’sresponsability or the consumer’s using the GIS map.

meters are in a state called inactive (Figure C.4a). As the data is graduallyrecorded, the voltage values will randomly be updated and displayed on themap. Three subsets are defined based on the minimum and the maximumvoltage values registered: low, medium and high voltage. This classificationgives the visual overview over the measured values by placing them in oneof the three categories.

From a user experience point of view, having this viewpoint over the elec-trical grid can help to quickly establish the areas that require most attention.For example, the high voltage measurement points denoted "high meters"are represented in red in Figure C.4b. This layer may be of particular interestfor the DSOs, while a pop-up is displayed if the user would want to check aparticular point on the map. This pop-up in Figure C.4c shows the voltagevalues on the three phases and the location of the smart meter.

The advantage of integrating nodal measurements with GIS lies in thepossibility for data management from a visual point of view. Filtering thelayer of interest on the map decreases the amount of data displayed consider-ably and enhances the general overview when the map is zoomed out. Someof the sub-areas also constitute an interest since they are rich in RES. GISgives the possibility of zooming in and selecting only the meters from thatspecific area in order to analyze the behavior of those customers.

92

IV. Results and Discussion

IV Results and Discussion

The visualization system for the low-voltage grid monitoring has been achievedthrough a theoretical approach. The focus is on the human perception in re-lation to different ways of visualizing a map, such as noticeable differencesregarding the number of data points or map refreshing time. This approachdoes not require testing on system operators, but focuses on a broader rangeof people targeting the general human perception.

According to the designed GIS-based monitoring system, a new sequencemodel was identified in Figure C.5. From the process efficiency point of viewfor the specific case described in Subsection A., it can be noticed that thenumber of operations has been reduced from 6 to 2 due to GIS integration.

Compared to the initial sequence model from Section III, Figure C.1, thenew model does not contain alerts regarding time consuming operations. Themonitoring performance is improved due to linking measurement data andthe GIS location of the consumers, which makes it easier for the DSOs to spotanomalies, as well as their geographical location and address. Filtering onlythe points of interest on the map minimizes the amount of steps required todetermine whether a power outage report is due to an anomaly in the grid.Thus, the time to act on a certain report is also shortened.

The voltage data is only pushed to the visualization system whenever itfluctuates more than 10% of the display value and therefore, in normal con-ditions, there are no great requirements in terms of memory or data trans-actions. Moreover, it does not make sense to display a huge area at once fora human user, and therefore, the data is only displayed when zooming in tosubstation level. The maximum number of meters in a substation is around2000, needing an initial data pull for 2000 entities. The rest of the time a sub-station is being displayed, the requirements would be even lower than thatinitial case.

V Conclusion

From the data flow perspective, taking advantage of the unused new featuresof the current AMI in combination with intelligent data filtering and GISvisualization, allows the operators to understand the consumption patternsof the customers and to identify issues or anomalies that may arise from thelarge number of RES.

On that account, the monitoring system helps to speed up the decisionmaking process under these special conditions. With the support of spatialawareness, the efficiency of operating the power grid improves, and therebythe overall human working experience and competences.

The next step is to improve the design of the presented GIS monitoring

93

system by focusing on the system operators in their working environment.Taking this contextual approach would therefore include the DSO as a centralpart of the design process and would be more directly useful for electricitycompanies.

Acknowledgment

This work is financially supported by the Danish project RemoteGRID, whichis a ForskEL program under Energinet.dk with grant agreement no. 2016-1-12399. The authors would like to thank the two student groups at ProductDesign Psychology studies at Aalborg University who conducted the inter-views with the DSOs in the spring of 2018.

References related to paper C

[1] The rox java nio tutorial, http://rox-xmlrpc.sourceforge.net/niotut/.

[2] Kriegers flak - combined grid solution, https://en.energinet.dk/

Infrastructure-Projects/Projektliste/KriegersFlakCGS, 2017.

[3] Danish energy agency, February https://ens.dk/service/

statistik-data-noegletal-og-kort, 2018.

[4] Smart grid strategy. the intelligent energy system of the future, Mayhttps://ens.dk/sites/ens.dk/files/Globalcooperation/smart_grid_

strategy_eng.pdf, 2013.

[5] Projektbeskrivelse, http://www.remotegrid.dk/projektbeskrivelse/.

[6] Atefeh Dehghani Ashkezari, Nasser Hosseinzadeh, Ayoub Chebli, and Ma-hammed Albadi. Development of an enterprise geographic information system(gis) integrated with smart grid. Sustainable Energy, Grids and Networks, 14:25 –34, 2018.

[7] Danish Energy Association. Smart grid in denmark 2.0, https://www.usef.

energy/app/uploads/2016/12/Smart-Grid-in-Denmark-2.0-2.pdf, 2016.

[8] C. Xuen D. Nga O. See, D. Quang and L. Chee. Visualization techniques in smartgrid. Smart Grid and Renewable Energy, 3:175–185, 2012.

[9] Houda Daki, Asmaa El Hannani, Abdelhak Aqqal, Abdelfattah Haidine, andAziz Dahbi. Big data management in smart grid: concepts, requirements andimplementation. Journal of Big Data, 4(1):13, Apr 2017.

[10] Eduardo F. Ferreira and J. Dionísio Barros. Faults monitoring system in theelectric power grid of medium voltage. Procedia Computer Science, 130:696 – 703,2018. The 9th International Conference on Ambient Systems, Networks and Tech-nologies (ANT 2018) / The 8th International Conference on Sustainable EnergyInformation Technology (SEIT-2018) / Affiliated Workshops.

http://rox-xmlrpc.sourceforge.net/niotut/

https://en.energinet.dk/Infrastructure-Projects/Projektliste/KriegersFlakCGS

https://en.energinet.dk/Infrastructure-Projects/Projektliste/KriegersFlakCGS

https://ens.dk/service/statistik-data-noegletal-og-kort


https://ens.dk/sites/ens.dk/files/Globalcooperation/smart_grid_strategy_eng.pdf

https://ens.dk/sites/ens.dk/files/Globalcooperation/smart_grid_strategy_eng.pdf

http://www.remotegrid.dk/projektbeskrivelse/

https://www.usef.energy/app/uploads/2016/12/Smart-Grid-in-Denmark-2.0-2.pdf


[11] H.A. Gabbar, A. Zidan, and M. Xiaoli. Chapter 16 - data centers for smart energygrids. In Hossam A. Gabbar, editor, Smart Energy Grid Engineering, pages 433 –452. Academic Press, 2017.

[12] Joseph, Shibily Joseph, E. A. Jasmin, and Soumya Chandran. Stream computing:Opportunities and challenges in smart grid. 21:49–53, 2015.

[13] S. Joseph and E. A. Jasmin. Stream computing framework for outage detectionin smart grid. In 2015 International Conference on Power, Instrumentation, Controland Computing (PICC), pages 1–5, Dec 2015.

[14] H. Li, M. Dong, K. Ota, and M. Guo. Pricing and repurchasing for big dataprocessing in multi-clouds. IEEE Transactions on Emerging Topics in Computing,4(2):266–277, April 2016.

[15] X. Liu, L. Golab, and I. F. Ilyas. Smas: A smart meter data analytics system.In 2015 IEEE 31st International Conference on Data Engineering, pages 1476–1479,April 2015.

[16] Xiufeng Liu and Per Sieverts Nielsen. Streamlining Smart Meter Data Analytics.International Centre for Sustainable Development of Energy, Water and Environ-ment Systems, 2015.

[17] B. Lohrmann and O. Kao. Processing smart meter data streams in the cloud. In2011 2nd IEEE PES International Conference and Exhibition on Innovative Smart GridTechnologies, pages 1–8, Dec 2011.


[19] D. Mezera. Voltage quality in the low voltage distribution grids with the highpenetration of distributed energy sources. In 2015 16th International ScientificConference on Electric Power Engineering (EPE), pages 292–295, May 2015.

[20] A. A. Munshi and Y. A. I. Mohamed. Cloud-based visual analytics for smart gridsbig data. In 2016 IEEE Power Energy Society Innovative Smart Grid TechnologiesConference (ISGT), pages 1–5, Sept 2016.

[21] Nonblocking system architecture, http://www.onjava.com/pub/a/onjava/

2002/09/04/nio.html?page=2, 2017.

[22] P. A. Parikh and T. D. Nielsen. Transforming traditional geographic informationsystem to support smart distribution systems. In 2009 IEEE/PES Power SystemsConference and Exposition, pages 1–4, March 2009.

[23] R. Shyam, Bharathi Ganesh H B, Sachin Kumar S, Prabaharan Poornachandran,and K. P. Soman. Apache spark a big data analytics platform for smart grid.21:171–178, 2015.

[24] The right big data technology for smart grid – distributed stream computing,https://www.accenture.com/us-en/, 2014.

[25] Wenlu Yang, A. Da Silva, and M. L. Picard. Computing data quality indicatorson big data streams using a cep. In 2015 International Workshop on ComputationalIntelligence for Multimedia Understanding (IWCIM), pages 1–5, Oct 2015.

95

http://www.onjava.com/pub/a/onjava/2002/09/04/nio.html?page=2

http://www.onjava.com/pub/a/onjava/2002/09/04/nio.html?page=2

https://www.accenture.com/us-en/

96

Paper D

Automation of smart grid operation tasks via spatio-temporal exploratory visualization

Maria Stefan, Morten H. Andreasen, Jose G. Lopez,Michael Lyhne and Rasmus L. Olsen

The paper has been submitted to:The Journal of Environment and Planning B: Urban Analytics and City Science,

2019

© 2019 SAGE Publications LtdThe layout has been revised and reprinted with permission.

I. Introduction

Abstract

The evolution of electricity production is moving towards the use of 100% green en-ergy resources, influenced to a large extent, by worldwide policies supporting the cli-mate mitigation agenda. Consumers are evolving into energy producing prosumers,shaping the future of low-voltage grids into a distributed architecture.

State-of-the-art Advanced Metering Infrastructure (AMI) empower DistributionSystem Operators (DSOs) to collect various end-point status and condition data,in order to improve the grid operation, management and planning. In addition, theincrease in data availability and granularity opens up the possibility for investigatingthe new possibilities and benefits of integrating data-driven models into the operationand management processes for electrical grids. The purpose of this article is to exploreopportunities for designing a competent state-of-the-art monitoring and visualizationsystem for low-voltage electricity grids. Primarily, the objective is to highlight thebenefits of moving from traditional operation support systems in the field, to modernsmart grid cyber-physical systems, allowing among other features, immediate gridmonitoring awareness and preemptive actions.

The research and proof-of-concept in this work has been developed in close collab-oration with a DSO in Denmark, stating the real-world requirements such a systemmust fulfill, in order to provide some benefits compared to the already existing system.These benefits are evaluated and quantified by migrating the traditional monitoringoperations and workflows (extracted from on-site surveys) to the proposed referencemodels, possible due to the massive deployment of AMI the DSO has done in thepast few years. The results indicate that a high level of automation is possible whencombining the already existing AMI infrastructure with data-driven processes andvisualization in the evaluated cases. This automation implies a significant reductionof required time and human resources to resolve the investigated conflicts, decreasingthe total operational costs (OpEx) of the grid.

I Introduction

In consideration of the rapid evolution of electricity grids, this study pro-poses a software solution for electrical grid monitoring and planning, bydata analysis and feature presentation [15]. The solution aims to provide theDistributed System Operators (DSO) with an efficient decision support sys-tem which can accommodate for a large number of prosumers (low-voltagecustomers with both demand and generation), variety in the incoming meter-ing data (Advanced Metering Infrastructure - AMI readings) and automaticerror detection. The achieved solution comes in the form of a visualizationprototype, dedicated for users (DSOs) with different education and profes-sional backgrounds [16]. Data visualization is important as it enhances thedecision making processes due to the large implication of the human factor

99

Paper D.

in the current grid management operations [7].The work has been carried out departing from on-site interviews with

employees from an energy company in Denmark. Their feedback is valuablefor shaping different user profiles and for understanding their needs. Cur-rent daily workflows in error debugging are analyzed to identify the mosttime consuming operations regarding the low-voltage electrical grid. Thepreliminary user studies are then used as input to sketch the design andimplementation of an appropriate visualization prototype. Findings fromthe assessment of the implemented visualization underline the importanceof data presentation in practical applications and the benefits offered by geo-graphic information systems (GIS) in terms of spatial awareness.

II Related work

A. Smart grids

Electricity grids undergoing the evolution towards Smart Grids [1] and thesubsequent myriad data generators, naturally brings grid monitoring andcontrol systems into the Big Data era. The significant increase in data varietyand volume unlocks the possibility to exploit this data resource, to developnew functionalities and to improve performance via integrated systems, busi-ness intelligence, geographic information systems, big data processing andanalytics technologies, etc [23] [13].

The inclusion of consumer power generators, primarily photovoltaic andwind generators, impose challenges for an electricity grid conceptually de-signed for one-way electricity flow. Thus, modern grid interoperability neces-sitates both enhanced monitoring, tighter control and regulation capabilities[9]. The smartness in grid monitoring operations is introduced in this workby integrating the low-voltage electrical grid topology with time-series me-tering measurements, both historical and near-real-time [17], and with visualinformation provided by Earth API.

B. Data management and system design

Continuous data generation requires that it is presented into comprehensibleand perceptible formats by humans.

Originating from a data generator, such as an AMI customer, the datagoes through a chain of procedures before it is either presented and or stored.Common data manipulation procedures include: sorting, feature extraction,processing, analytics and visualization [20] [19] [10] [5].

Where traditionally there is a tradeoff between processing speed and in-formation precision, current data processing architectures theoretically dis-play performance and capabilities where no compromise is necessary [3] [14].

100

III. Sequence models and user stories

However the actual performance of an application is highly implementationdependable and it is taken into consideration in this study, by evaluating theimprovements in the DSOs’ workflows, brought by the proposed visualiza-tion system.

C. Data visualization and GIS

Typically, visualization of selective grid parameters is presented in dashboardstyle or by individual system interfaces with a disperse representation by thedifferent subsystem components [15] [12] [22], requiring switching and ex-changing data between systems for either continued work flow or compari-son.

Geographic Information Systems (GIS) make it possible for utilities tosmartly operate their electricity grid, by interlocking data management andanalysis, situation awareness, grid planning and workforce automation [6].Situation and spatial awareness in particular bring smartness to the existinggrid, enabling the utilities to detect and solve a power outage before cus-tomers call in [2] [4]. Aside from the data analytics platform, GIS also offersthe possibility to outline possible places for renewable energy resources, thusit is also a tool for green energy planning.

Moreover, workforce automation can be obtained through WebGIS devel-opment and sharing of information via maps from office to field, due to itscompatibility with both desktop and mobile devices [21], which is why it hasbeen utilized for implementation in this work.

III Sequence models and user stories

Thee following section presents two of the most common scenarios for fail-ure debugging. The two cases are evaluated both from a mathematical andsequence modeling point of view, with the purpose of extracting wokflowpatterns for the DSOs.

A. Flickering of lights

One of the most frequent issues reported by the consumers to the DSOs isthe flickering of lights. This use case is interesting for studying due to theDSOs different steps and approaches for evaluating and eventually solvingthe error. A general sequence model diagram for the case of lights flickeringis presented in Figure D.1.

The scenario depicts the case of a customer calling in with an undervolt-age error, claiming that their lights are flickering. In this scenario, the DSOfollows the procedure of looking up the phase voltages in a simplified user ex-perience (UX) AMI interface. Aside from this, two other programs are used: a

101

Paper D.

billing management software and an Excel spreadsheet as calculation software. Toexamine the cause of the undervoltage, the DSO uses a GIS tool to find otherconsumers connected to the same cable box as the calling consumer. At thecable box level it is possible to evaluate if this is a multiple issue across thelow-voltage grid consumers, in which case the problem is labeled as inter-nal and has to be passed on to the network planning department for furtherinvestigation.

If the reported issue is local with respect to the single cable box found onthe GIS map, the DSO will proceed to the next step: calculating the grid dis-tance from the consumer who reported the error to the nearest transformerstation, in order to determine the theoretical values which should have beenmeasured instead of the erroneous ones. To find these, the DSO changesback and forth between the AMI interface and the map to evaluate distances,as well as values from the individual smart meters. This calculation is re-peated until the point of reading the next transformer station (action labeledas “Loop” in Figure D.1). Furthermore, the billing management software pro-vides the calculation of the amount of power generated and consumed frompersonal renewable energy sources.

The next step of the procedure is to set up a measuring box in the cablebox related to the calling consumer. This is done in order to collect three-phased voltage measurement for one week period. This data is analyzed andthe DSOs are then able to see whether they do or do not meet the require-ments of normal voltage boundaries: ± 10 % of the nominal voltage value(230 V) for 95% of the time. If these requirements are not met, the previ-ously made calculations are useful in deciding what kind of electrical gridreinforcements should be applied: installing bigger cables or transformers.These actions are also depending on the type of user (household or com-pany) and operator. If the voltage is within boundaries, it is the customer’sresponsibility to contact their own electricity company.

It can be seen from the DSOs approach and work process that the above-mentioned case of light flickering at the consumer end is not a trivial one.The first step of checking the three-phased voltage measurements of the call-ing consumer is based on the latest identified values and does not take intoaccount any historical recordings of data. Even though the calculation loopseems time consuming, in the rural areas there are typically 1-2 connectedcustomers per cable box, for which these calculations can be done fairlyquick.

102


Fig. D.1: Sequence model representing the DSOs’ actions in case a household reports constantlight flickering.

103

Paper D.

However, the complexity increases with the number of connected cus-tomers like in the case of dense areas (i.e. cities), where the number of cablebox connected areas multiply.

However, depending on the severity of the issue, a continuous flickeringof lights could potentially turn into a power outage case, which was intro-duced in [18]. The DSOs would have to identify the affected area and informall the corresponding customers about it, which leads to examining the se-quence model of this use case.

The total time spent on debugging the case of light flickering (Tf lick) canbe modeled as in Equation D.1:

Tf lick = ts +m

∑im=1

tim (tcall + tGIS)+ td×(

m× dm

m

∑im=1

(m− 1)tim (tcall + tGIS) + tmap + tdcab

)+ tSW + trec × tinstall (D.1)

Which can be expressed as Equation D.2:

Tf lick = ts +(tcall + tGIS)m

∑im=1

tim + td×(

m(tmap + tdcab

)dm + (m− 1) (tcall + tGIS)

m

∑im=1

tim

)+ tSW + trec × tinstall (D.2)

Where the variables denote:

• m: number of meters;• im: meter IDs;• ts: time to search a consumer’s address;• tim : time to select the meter ID and to check the three-phased voltage

values;• tcall : time to call the customer;• tGIS: time to find the meter on the GIS map;• td: time to measure the distance to the nearest transformer station;• dm: time to calculate the theoretical distance on the map;• tmap: time to insert meter values in GIS;• tdcab

: time to measure the distance to the nearest cable box on the map;• tSW : time to find the meter in the selected transformer station and to

extract the values in the billing management software;• trec: time to install, record and analyze one week worth of measure-

ments in the specific cable box.• tinstall : time to perform grid reinforcement measures.

The human factor impacts all the aforementioned timings and it is thereforenot possible to assign exact numbers to them. However, it can be drawn from

104


Equation D.1 that the complexity of the working flow is highly dependent ofthe number of meters (m) and on the meter IDs (tim ). A large m results insearching across multiple tim for possible anomalies and thus creating redun-dancy between the two loops.

B. Identifying the area in the electrical grid that was affectedby an error and inform the corresponding customers

Indentify trasformer station

supplying the cable boxes

on the GIS map

Read values of found cable boxes

Locate the error

Send technician to corresponding

addresses with the error

An error report is sent to the technician

Select and message the affected customers

using the error messaging system

Contractor digs up affected cables

NO

Fig. D.2: Sequence model representing the DSOs’ actions for informing customers about anaffected area in the electrical grid.

This is the procedure followed by the DSOs after they have identified thatthey are responsible for an error, as presented in Figure D.2. This is doneeither by detecting the error through a similar sequence as in Figure D.1 orby a private technician informing them. Determining the exact area that hasbeen affected by the error is mainly done using the AMI interface and themap. When the area has been identified, a technician is sent to address the

105

Paper D.

issue and the affected consumers are informed through the error messagingsystem.

This use case is mainly presented to strengthen the motivation for whythere is a need for creating inter dependencies between different softwaretools utilized by the DSOs, for example between the map and the error mes-saging system. There is still the risk of incorrectly selecting and messagingthe affected consumers, due to the independence among these tools.

The total time spent on identifying a grid area with failures (Tmsg) can bemodeled as in Equation D.3:

Tmsg = tGIS + tread + terr + ttech + tm + tdig (D.3)


• tGIS: time to identify a transformer station on the map;• tread: time to read cable box values;• terr: time to locate the error;• ttech: time in which the technician reaches the address which issues

errors, while receiving the error report;• tm: time to message affected consumers via the error messaging system;• tdig: time to dig the affected cables.

The total Tmsg in Equation D.3 depends on the number of meters/customers(denoted as m) which are affected by an error and it is as well influenced bythe human impact in the debugging process.

The system operators use four different programs to assess different as-pects of the low voltage grid: the AMI interface, the map, the billing manage-ment tool and the calculation software. This may seem as a time-consumingway of monitoring the grid, due to the lack of efficient integration amongthese programs. Furthermore, it is observed that in the case of the Excelcalculation, the DSOs need to keep track of different parameters, such ascustomer and cable types, which the DSOs have to manually look through.An advantage would be to have a visual overview over different stages inthe debugging process, in connection with speeding up the error identifica-tion. Due to the subjective nature of these sequence models, the followingimplementation aims to:

• Minimize the risk of incorrect actions during the DSOs working routine;• Propose an data-driven integrated visualization solution as alternative

to the current manual procedures.

106

IV. Implementation

IV Implementation

This section contains a description of a visualization prototype implementa-tion, which aims to bring more automation to the procedures identified inSection III.

The test area in the electrical grid is located Thisted region, in Denmark[8]. The corresponding data is managed in a PostgreSQL database whichcontains medium and low-voltage grid topology (geographical information),attributes of the different entities and nodal measurements. The implementa-tion is based on ASP.NET developing stack to create the WebGIS visualizationapplication.

An overview of the WebGIS menu application is presented in Figure D.3for the test area. The map menu layer contains:

• customers - households and companies from the low-voltage layer;• secondary substations - belonging to the medium-voltage layer;• cable boxes - entity containing the electronic equipment which links the

medium to the low-voltage levels;• low-voltage (LV) wires - cables showing the interconnections between

customers and secondary substations;• medium-voltage (MV) wires - cables showing the interconnections among

substations;• high-voltage (HV) wires - cables showing the interconnections between

primary and secondary substation. No such information was availablein the data, but the implementation is ready for including this as well.

Fig. D.3: Overview of the WebGIS application and its features. The medium-voltage part of thevisualization includes secondary substations and their interconnections

107

Paper D.

Fig. D.4: Interconnections between the different kinds of nodes - from MV to LV. The group ofconnected cable boxes is useful for rerouting algorithms.

An error listing functionality is also included in the left-hand side of themap. Possible errors are prioritized and represented according to their sever-ity as: errors, warnings and other informative alarms. A description abouttheir nature can be obtained by clicking on the down arrow to inspect thepossible causes of the errors (Figure D.4).

The purpose of this application is manifold, providing the means for gridplanning, monitoring and prediction.

A. Planning

As desired, the WebGIS page allows for selecting a certain layer of the elec-trical grid, just as it is done in Figure D.3 depicting the MV grid (secondarysubstations) and their interconnections. Different grid operations require var-ious possible ways of displaying the data. In this case, having only the graphgives the opportunity to evaluate the current medium-voltage topology andassess the possibility of extending or changing the current topology by ap-plying graph networks algorithms. This view is mostly usual for grid rein-forcements and planning, for example in the case of light flickering (FigureD.1) where optimization procedures might be needed for the medium volt-age part of the grid. The same procedures can be applied to the low-voltagegrid if they are considered necessary.

108

IV. Implementation

B. Monitoring

An alternative to the graph view is adding an Open Street Map (OSM) layerto the WebGIS. This is done by selecting the view on the view button inthe upper right corner of the map, obtaining an Earth outlook as shown inFigure D.5. This representation is mostly useful for grid monitoring, as thelandscape information can be mined with the existing topology and measure-ments.

Fig. D.5: Earth view of a customer with a solar panel.

For example, it can be seen that the household shown in Figure D.5 has asolar panel installed on the roof. This is valuable information for the DSOs,especially in case they register inexplicable changes in the customers’ gener-ation and consumption patterns. It is possible that the some of the topologydata does not contain information about whether a certain customer has re-newable energy resources (RES) and, like in this case, the DSOs can visuallyevaluate it on the map.

More information can be obtained by clicking on the node associated tothat customer. A window like in Figure D.6 reveals details about the selectednode. In this case, the field denoted "Prosumer Generator Type" confirmsthat this specific household has a solar panel, meaning that this informationwas already updated in the initial data set.

109

Paper D.

Fig. D.6: Window pop-up showing the last measured values and historical time-series plots ofa certain node.

The pop-up window comprises both the latest measurement readings andhistorical time-series plots for values measured in the past 30 milliseconds.This granularity can be adjusted according to what is relevant for the DSOsin each application.

The ID of the selected node is denoted in the left upper corner with "#",after the address ("19 Skellet 7700"), and it can also be found in the "ExternalID" field. The date and time of the latest registered measurement are alsodisplayed. As it was required by the DSOs to visualize the three-phasedvoltage values in every node, these are as well recorded and laid out as bothlatest value and historical data. Each phase measurement can be individuallyanalyzed at any point in time by moving the mouse cursor on the plot (FigureD.6).

Current and frequency values are also available and can be analyzed ifrelevant, in the same manner as it is done with the voltage.

The "Voltage level" field denotes the layer of the grid to which the node be-longs. The value of 400 V indicates that the node is placed in the low-voltagelevel, while values of 10 and 60 kV would refer to nodes in the medium-voltage level.

Information about certain events is also available in the pop-up in "In-cident Reports". It contains the numbers of the incidents that have beenreported for a selected node and it is related to the error tab available onthe front page. A recorded event may refer to an error in the measurements(i.e. negative values of consumption energy), an alarm triggered by a voltagemeasurement outside the limit of ±10 % of the nominal voltage value of 230V (Figure D.1); or alarms and warnings generated by meters recording 0 Vmeasurements for a long period of time (i.e. cable faults which can be causedby someone digging in the ground).

Other nodes on the map are represented by cable boxes and secondarysubstations. Their detailed information can also be accessed through a similar

110

V. Assessment

window pane as in Figure D.7. "CP3" is the identifier for a cable box and"Holtab 630" for a secondary substation.

Fig. D.7: Detailed information about a cable box and a secondary substation.

C. Prediction

The linear representation of the grid topology may be utilized for predictionpurposes, provided the knowledge extracted from the planning and monitor-ing. Figure D.4 depicts the connectivity among a group of cable boxes. Suchinformation is useful for manipulating the existing tabular information of thegrid layout and exploring the possibilities and benefits given by new topolo-gies. Therefore, load analysis can be obtained by rerouting the information inthe electrical grid network based on the different possible layouts. From this,power redistribution algorithms can be designed to predict, for example, themost vulnerable areas in the power grid with respect to power failures.

V Assessment

As the purpose of this study is to explore how the DSOs working routinecan be optimized, this section contains the assessment of the WebGIS appli-cation presented in Section IV, by evaluating how the different implementedfeatures have improved the debugging process. This application was alreadypresented in a conference workshop [11], where both vendors for smart me-tering solutions and DSOs were present.

The diagram in Figure D.8 shows the new work flow process of the DSOsusing the WebGIS application. It can be noticed that this process model linksthe individual procedures presented in Figures D.1 and D.2. As a result of amore efficient software integration, events and errors are identified automat-ically, making the operators aware of them before customers calling in. Byintegrating the different computational models (i.e. calculating the distance

111

Paper D.

Open error message

Check 3-phased voltages

Fig. D.8: Updated sequence model for procedures of resolving two error cases using theWebGIS prototype.

from a node to the nearest transformer station) with the spatial awarenessachieved via GIS, clear improvements have been achieved in the process flowin Figure D.8, where they are denoted with a green check mark:

• Fast error identification, particularly in case of collective issues (fromMV to LV) - most relevant for grid monitoring;

• Instant visual correlation of a meter to its corresponding transformerstation (and vice-versa), due to the connectivity information in the gridtopology - most relevant for grid monitoring, as well as planning;

• Integrated calculation of the theoretical metering values and compari-

112

V. Assessment

son with the actual measurements (both near-real-time and historical),where mismatches are shown as incident reports - most relevant forprediction of potential incidents;

• Efficient use of data structures to quickly identify a desired node IDwithout any manual searching - relevant for data integrity and gridstructure maintenance.

By eliminating the loops in the new process model, the number of debuggingactions performed is also reduced in each type of error, thus minimizing thetotal debugging time. This can be modeled by Equation D.4.

Tdebug =

{tim + ttran + thist + tinstall , if V /∈ (±10% · 230)tm + ttech + tdig, if V = 0

(D.4)


• tim : time to check the voltage values;• ttran: time to find a transformer station on the map. Corresponds to td

in Equation D.1;• thist: time to check historical measurements and incident reports. Cor-

responds to trec in Equation D.1;• tinstall : same as in Equation D.1;• tm: time to locate a meter and message the corresponding customer.

Corresponds to tm in Equation D.3;• ttech: same as in Equation D.3;• tdig: same as in Equation D.3.

The influence of the human factor is minimized due to integrating the com-putational models into the same application and thus reducing the total timedependency per work flow to four variables in the over/under voltage case.A cutback in the number of operations can also be seen in case of recording0 voltage measurements for a long period of time.

Visual spatial awareness eliminates the risk of selecting and messaging theirrelevant customers and helps the operators keep track of the current stateof the grid, while focusing on the most vulnerable areas. The spatial bene-fit was also acknowledged by both the smart meter vendors and the DSOspresent in the SmartGridComm workshop [11], as it gives the possibility tofurther exploit the potential of the existing metering infrastructures, whichare currently used only for billing.

This implications of the visualization prototype are summarized in TableD.1.

113

Paper D.

Table D.1: Benefits and drawbacks of the proposed visualization application.

Benefits Drawbacks

• improved accuracy and process redesign • migration requirements to data-drivensystems

• system redesign for optimization of theDSOs’ workflows

• design standardization of the model

• improved OpEx by integrating data-driven business processes and analytics

• CapEx investment required for data stor-age and processing power (either physicalor cloud-based)

• unified software solution for grid opera-tion

VI Conclusion and Future work

Visualization and interpretability in practical applications must take into ac-count the human cognitive factor that any knowledge extraction process en-tails, according to the specific requirements of the application area. A keyidea in this study is that the main reason for a continuous integration (CI)-based system is the manual and time consuming error debugging process.For the Danish DSOs, interpretation is a tool to optimize their current work-flow procedures and to acquire new knowledge about the low-voltage grid,which can be used for monitoring, planning and event prediction.

Many of the present-day big data management solutions based on cloudapplications can accommodate for varied and voluminous data to be stored,processed and analyzed, but also offer system interoperability between ven-dors and DSOs. At the same time, WebGIS applications are flexible to beimplemented according to the users’ specific requirements, different profilesand working procedures and they are suited for mobile-based solutions.

Additionally, migrating towards a fully integrated system takes advantageof feature synergies and opens possibilities for development and adaption ofenhanced functionalities, and as a result simplifying daily operations andworkflows. This is done by integrating a grid model, continuous smart metermonitoring, geographic information and visualization techniques, thus offer-ing spatial and situation awareness in power grid management. Data-drivensystem ingration results in eliminating the need for an operator to manuallyswitch, transfer or compare data between different systems, resulting in re-duced error rates via automatic data validation as well as improved workflowefficiency by minimization of operator processes.

The WebGIS application presented in this work brings benefits for low-voltage grid data management, by decreasing anomaly detection time and,

114

as a consequence, by reducing the power grid OpEx.A feasible approach in the future work may imply more iterations in the

user analysis study, using the current prototype as basis for new on-site in-terviews with the DSOs and upgrading the application design accordingly.Moreover, the transition to mobile-based WebGIS should be taken into con-sideration, particularly for the on-call DSOs.

VII Copyright statement

A. Copyright

Copyright © 2019 SAGE Publications Ltd, 1 Oliver’s Yard, 55 City Road,London, EC1Y 1SP, UK. All rights reserved.

Acknowledgment

This work is financially supported by the Danish project RemoteGRID, whichis a ForskEL program under Energinet.dk with grant agreement no. 2016-1-12399.

References related to paper D

[1] S. S. Bhalshankar and C. S. Thorat. Integration of smart grid with renewableenergy for energy demand management: Puducherry case study. In 2016 Interna-tional Conference on Signal Processing, Communication, Power and Embedded System(SCOPES), pages 1–5, Oct 2016.

[2] Deborah Byrd. Bill meehan on using gis to help cre-ate a smart grid, https://earthsky.org/human-world/

bill-meehan-on-using-gis-to-help-create-a-smart-grid-2, 2011.

[3] V. Dehalwar, A. Kalam, and A. Zayegh. Infrastructure for real-time communica-tion in smart grid. In 2014 Saudi Arabia Smart Grid Conference (SASG), pages 1–4,Dec 2014.

[4] Charles H. Drinnan. Smart grid 2.0: The role of gis, https:

//electricenergyonline.com/energy/magazine/621/article/

Smart-Grid-2-0-The-Role-of-GIS.htm, 2012.

[5] A. El Khaouat and L. Benhlima. Big data based management for smart grids.In 2016 International Renewable and Sustainable Energy Conference (IRSEC), pages1044–1047, Nov 2016.

[6] Esri. Gis for smart grid, https://www.esri.com/library/brochures/pdfs/

gis-for-smart-grid.pdf, 2012.

https://earthsky.org/human-world/bill-meehan-on-using-gis-to-help-create-a-smart-grid-2

https://earthsky.org/human-world/bill-meehan-on-using-gis-to-help-create-a-smart-grid-2

https://electricenergyonline.com/energy/magazine/621/article/Smart-Grid-2-0-The-Role-of-GIS.htm



https://www.esri.com/library/brochures/pdfs/gis-for-smart-grid.pdf

https://www.esri.com/library/brochures/pdfs/gis-for-smart-grid.pdf

[7] Meiken Hansen and Bettina Hauge. Scripting, control, and privacy in domesticsmart grid technologies: Insights from a danish pilot study. Energy Research andSocial Science, 25:112 – 123, 2017.


[9] Patrick Matschoss, Benjamin Bayer, Heiko Thomas, and Adela Marian. The ger-man incentive regulation and its practical impact on the grid integration of re-newable energy systems. Renewable Energy, 134:727 – 738, 2019.

[10] Amr A. Munshi and Yasser A.-R. I. Mohamed. Big data framework for analyticsin smart grids. Electric Power Systems Research, 151:369 – 380, 2017.

[11] Rasmus Løvenstein Olsen. Challenges and solutions in future distributiongrids. In IEEE International Conference on Communications, Control, and Com-puting Technologies for Smart Grids, https://sgc2018.ieee-smartgridcomm.org/challenges-and-solutions-future-distribution-grids, 2018.

[12] Thomas J. Overbye and James Weber. Smart grid wide-area transmission systemvisualization. Engineering, 1(4):466 – 474, 2015.

[13] Miloš Radenkovic, Jelena Lukic, Marijana Despotovic-Zrakic, Aleksandra Labus,and Zorica Bogdanovic. Harnessing business intelligence in smart grids: A caseof the electricity market. Computers in Industry, 96:40 – 53, 2018.

[14] S. Roy, B. Bedanta, and S. Dawnee. Advanced metering infrastructure for realtime load management in a smart grid. In 2015 International Conference on Powerand Advanced Control Engineering (ICPACE), pages 104–108, Aug 2015.

[15] Maria-Angeles Sanchez-Hidalgo and Maria-Dolores Cano. A survey on visualdata representation for smart grids control and monitoring. Sustainable Energy,Grids and Networks, 16:351 – 369, 2018.

[16] Lea Schick and Christopher Gad. Flexible and inflexible energy engagements—astudy of the danish smart grid strategy. Energy Research and Social Science, 9:51 –59, 2015. Special Issue on Smart Grids and the Social Sciences.

[17] Maria Stefan, Jose G. Lopez, Morten H. Andreasen, Ruben Sanchez, and RasmusOlsen. Data analytics for low voltage electrical grids. pages 221–228, 01 2018.

[18] Maria Stefan, Jose G. Lopez, and Rasmus L. Olsen. Exploring the potential ofmodern advanced metering infrastructure in low-voltage grid monitoring sys-tems. 2018 IEEE International Conference on Big Data (Big Data), pages 3543–3548,2018.

[19] A. Vaccaro, I. Pisica, L.L. Lai, and A.F. Zobaa. A review of enabling methodolo-gies for information processing in smart grids. International Journal of ElectricalPower and Energy Systems, 107:516 – 522, 2019.

[20] Tom Wilcox, Nanlin Jin, Peter Flach, and Joshua Thumim. A big data platformfor smart meter data analytics. Computers in Industry, 105:250 – 259, 2019.

116

https://sgc2018.ieee-smartgridcomm.org/challenges-and-solutions-future-distribution-grids

https://sgc2018.ieee-smartgridcomm.org/challenges-and-solutions-future-distribution-grids

[21] Kehe Wu and Zhibin Zhang. Research and implementation of smart transmissiongrids based on webgis. In 2010 Second International Conference on CommunicationSystems, Networks and Applications, volume 1, pages 302–307, June 2010.

[22] Lin Xiqiao and Yang Zhou. Analysis of large-scale electric vehicles chargingbehavior using data visualization. In 2017 IEEE 14th International Conference onNetworking, Sensing and Control (ICNSC), pages 384–388, May 2017.

[23] Soo Wan Yen, Stella Morris, Morris A.G. Ezra, and Tang Jun Huat. Effect of smartmeter data collection frequency in an early detection of shorter-duration voltageanomalies in smart grids. International Journal of Electrical Power and Energy Sys-tems, 109:1 – 8, 2019.

117

118

Paper E

(Position paper) Characterizing the Behavior ofSmall Producers in Smart Grids

A data sanity analysis

Maria Stefan, Jose Gutierrez, Pere Barlet, Oriol Gomisand Rasmus L. Olsen

The paper has been submitted to:The Journal of Applied Energy, 2019

© 2019 Elsevier Ltd.The layout has been revised and reprinted with permission.

I. Introduction

Abstract

Renewable energy production throughout low-voltage grids has gradually increasedin electrical distribution systems, therefore introducing small energy producers - pro-sumers. This paradigm challenges the traditional unidirectional energy distributionflow to include disperse power production from renewables. To understand how en-ergy usage can be optimized in the dynamic electrical grid, it is important to under-stand the behavior of prosumers and their impact on the grid’s operational procedures.

The main focus of this study is to investigate how grid operators can obtainan automatic data-driven system for the low-voltage electrical grid management, byanalyzing the available grid topology and time-series consumption data from a real-life test area. The aim is to argue for how different consumer profiles, clustering andprediction methods contribute to the grid-related operations. Ultimately, this work isintended for future research directions that can contribute to improving the trade-offbetween systematic and scalable data models and software computational challenges.

I Introduction

The installation of diverse industrial and domestic renewable and green en-ergy generators and the subsequent decentralized architecture guide the powergrid progress towards a smart grid strategy. This is a direct result of thecommitment to 100% renewable energy production in Denmark by 2050 aspart of a national climate change mitigation plan, which is influenced byinternational interests (i.e. EU’s clean energy package and regulations [3]).Currently, the power grid supports up to 43% power generation from windturbines [9] [7] and can operate with up to 50% power supplied by com-bined heat and power (CHP) plants [1]. Consequently, as the penetration rateof renewable energy sources (RES) intensifies, the low-voltage power gridbecomes active as consumers evolve into prosumers, emphasizing decentral-ization, influencing daily grid operation and management for the distributedsystem operators (DSOs).

Currently, the incoming metering data is used only for billing purposes[13]. The increasing complexity of the grid will require an automated solutionin which different anomalies can be detected with minimum delay time. It isexpected that with the high penetration of RES there will be an increase in re-ported issues by the consumers [16] . Measurement data from the AdvancedMetering Infrastructures (AMI) can be utilized in this case to characterize theconsumption patterns and predict future possible issues. At the same time,the aim is to make use of the available data in order to prepare for a scenariowith 100% renewable energy.

This research topic aims to utilize the available billing data measurementsin order to understand the consumers’/prosumers’ behavior. Moreover, the

121

Paper E.

objective is to propose solutions to some of the foreseeable problems, by inte-grating electrical engineering knowledge into a computer software solution.In this way, the contribution comes from analyzing the quality of the avail-able electrical grid data. It is shown that it would be useful for the DSOs toconsider this data for analysis, in order to automate their most frequent gridmanagement procedures.

II Machine learning for electrical grid data - Re-lated work

Various machine learning techniques have been previously used in the powergrid domain to provide the DSOs with the right tools for grid planning, mon-itoring and forecasting. In general, understanding the energy behavior at thelow-voltage grid can be done by clustering households by specific attributes,which are defined by some analytic techniques:

• Extracting the electricity demands according to different times of theday, season and weekdays;

• Classification according to the chosen attributes (from low to large vari-ability);

• Reliability testing: sample robustness assessed using a bootstrappingmethod as in [10].

For forecasting purposes

Electricity short term load forecasting (STLF) applied to historical customerdata is addressed in [5], by means of data cleaning (smooth out irregularelectricity consumption patterns, such as holidays), error correction methodsand ANN (Artificial Neural Networks) with historical weather data.

Demand is very random over short periods of time, day-to-day profiles,therefore a demand forecast model is needed in the management controlsystem, as explained in [4]. The model was obtained through data pre-processing, correlation clustering and discrete classification NN (Neural Net-work). The total energy demand was adjusted according to the total energyforecast.

For monitoring purposes

The study in [15] is used to obtain forecast density estimation by searchingfor analogs in the historical data. It can be utilized for in-memory computingin distributed systems, by saving computational time for a high number ofsmart meters and by providing scalability.

122

III. A data sanity study for low-voltage electrical grids

For planning purposes

Short term state forecasting and operational planning is addressed in [11],resulting in the following services:

• Optimized distributed energy resources allocation, based on the loca-tion of energy resources in the distribution network. The amount ofrequired load adjustment is minimized to match with the network con-straints. This service can be utilized for energy balancing;

• Voltage estimation using historical smart meter data and estimates ofthe net demand. Probabilistic estimates of low-voltage profiles areobtained, assuming that the smart meters cannot measure voltage orpower quality.

The clustering methods for analyzing time-series data streams also pro-vide insight into the customers’ privacy, by identifying specific behaviors [8].The gained knowledge gives useful information regarding energy fraud de-tection, home invasion or children behavior. However, this method does notaddress the privacy issues that arise from the communication network infras-tructure or from storing household-related data.

III A data sanity study for low-voltage electricalgrids

A. Data system and data flow

The information exchange in the electrical grid corresponding to the Remote-GRID project [13] [14] is depicted in Figure E.1. The three actors defined

Fig. E.1: Information exchange among the meter provider, IT distribution and the DSO.

123

Paper E.

in Figure E.1 show the relationships between the DSO, AMI provider andIT distribution. The AMI provider [12] is in charge of the Meter Data Man-agement (MDM) module for data storage and processing and of the AMIcommunication infrastructure. It provides the DSO with smart meter (SM)readings in the form of time-series data. The DSO [2] uses these historicalreadings to check for potential anomalies (monitoring) and for grid planning.Simultaneously, the DSO is provided with grid topology information which ismanaged via a SCADA system (Supervisory Control and Data Acquisition).The IT distribution company [6] is in charge of data integration between GISand SCADA, as defined by the CIM standard. The available GIS data has tobe regularly modeled and converted to CIM to be correctly imported to theSCADA system, according to updates in the topology (i.e. new customerswith PVs and wind turbines).

The flow of the present-day data system lies in the lack of interactionbetween the IT distribution and the AMI provider. This missing link mayresult in data inaccuracy, posing operational challenges to the DSOs. Forexample, the quality of the GIS-modeled low-voltage network may not besufficient due to missing customer-related data. In this case, the DSO relieson knowledge of the number of customers connected to the transformersin the specific secondary substations (medium-voltage), instead of the low-voltage grid topology information.

B. Data types

The two data types received by the DSO from the AMI provider and fromthe IT distributor are described as follows:

• GIS dataThe grid topology comes in the form of geographic information, con-taining the connectivity information between the medium and the low-voltage part of the grid. This includes nodes (secondary substations,cable boxes and consumers) and their interconnecting cables. An ex-ample of the topology information is provided by Figure E.2, wheresubstations are represented by the red triangles, cable boxes by the bluesquares and consumers by green dots. The red dotted lines show theAC connections among the secondary substations. The low-voltage gridconnections are marked by the different colored lines, each color depict-ing the different groups of consumers fed by each of the substations.

• Time series dataActive and reactive energy measurements are provided for a period ofone year (from April 2017 to April 2018 inclusively), with a granular-ity of 15 minutes, which is defined by the current metering infrastruc-

124


Fig. E.2: Medium and low-voltage grid topology sample.

ture. The data model for the time-series measurements is shown inFigure E.3, comprising of three descriptive tables. The measurements ta-ble contains the meter ID ("meter_no"), measurement timestamp andconsumption values (active positive energy). The meter ID is used asforeign key element for the Meter_info table, which contains general in-formation about the individual metering points: address, generatingunit kind (solar cells, windmills, other) and customer category name(household, company, school, other). The meter_no field is used as for-eign key for the Cluster table, which is used to store information aboutconsumption classification. This topic will be covered in Subsection D..

Fig. E.3: Data model for the time-series measurements.

The meter ID field (obtained from the distribution company involved in thestudy [2]) was used to link the two data types via address geocoding, makingit possible to perform statistical analysis on the time-series measurements,based on the meters’ geographic information.

125

Paper E.

C. User profile analysis - labeled data

This subsection illustrates the customer profiles obtained from data related tosubstation 3011, which is shown in Figure E.4. This substation is chosen dueto the presence of PVs, as well as for comprising of both apartment buildingsand stand-alone houses. The blue lines represent the connections betweenthe substation and its consumers.Two types of labeled customers have been identified - households and com-

Fig. E.4: Low-voltage grid topology information for secondary substation 3011.

panies, which will be used as starting point in the analysis. Some of thecompanies are labeled with PVs, but there is no information about RES athousehold level.

• Household profilesThe plot is Figure E.5 shows the household consumption statistics (inkWh). Subfigure E.5a depicts the average consumption of all house-holds for the whole year per hour of the day, for each day of the week.All data is taken into account for all seasons of the year, without filter-ing out holidays. This is done in order to obtain a general overview overthe households’ consumption trends. It can be seen that the trend is asexpected, with the highest consumption peaks in the morning (around6-7 AM in the weekdays and later in the weekends) and in the after-noon (5-7 PM).Subfigure E.5b shows the individual household consumption per hour

of the day. The data is again averaged for the whole year and for all thedays of the week. As it can be noticed, there are two households whoseconsumption trends are different than the average. This informationcan also be evaluated statistically in Table E.1.

126


0 5 10 15 20hour

0.06

0.08

0.10

0.12

0.14

0.16

kW

Positive active energy for households per hour of the dayMondayTuesdayWednesdayThursdayFridaySaturdaySunday

(a) Households' average consumption patterns per hour.

0 5 10 15 20hour

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

kW

Individual meters consumption per hour of the day

(b) Individual households' consumption patterns per hour.

Fig. E.5: Positive active energy plots for labeled households

Table E.1: Statistics related to households and companies.

Label Mean(kWh)

Min(kWh)

Max(kWh)

Variance Standarddeviation

House 4 0.127 0.01 3.41 0.028 0.166

House 7 0.156 0.00 5.13 0.198 0.445

House 15 0.787 0.11 8.84 1.029 1.014

Comp 9 (solar) 0.521 0.00 6.41 0.342 0.585

Comp 58 (not solar) 4.211 2.34 9.20 0.688 0.830

Comp 87 (not solar) 7.011 0.45 39.82 74.867 8.653

127

Paper E.

• Company profilesSimilarly, the company consumption profiles (kWh) are presented inFigure E.6. The average consumption per year for every week day isshown in Subfigure E.6a. The trend represents a typical working weekin Denmark, starting early in the morning (6-8 AM) and ending at about4 PM in the weekdays and earlier on Friday. Also, the lowest consump-tion is registered in the weekends and after working hours. Individualcompany consumption per day is depicted in Subfigure E.6b, averagedover one year. Two companies seem to issue different trends in theirpatterns other than the rest, which can also be extracted from the sta-tistical values (Table E.1).

0 5 10 15 20hour

1.0

1.5

2.0

2.5

3.0

3.5

4.0

kW

Positive active energy for companies per hour of the dayMondayTuesdayWednesdayThursdayFridaySaturdaySunday

(a) Companies' average consumption patterns per hour.

0 5 10 15 20hour

0

2

4

6

8

10

12

14

16

kW

Individual meters consumption per hour of the day

(b) Individual companies' consumption patterns per hour.

Fig. E.6: Positive active energy plots for labeled companies

128


D. Customer classification

It can be concluded from the previous statistical analysis that there is somevariation in the individual consumption patterns for households and compa-nies. In order to help anticipating trends for the different metering points,with the purpose of detecting whether data is missing or erroneous, an auto-matic clustering method is applied for the two labeled data sets. This is doneusing the positive active energy values, obtaining three clusters per data set.The results are presented in Figure E.7, for households (Subfigure E.7a) andcompanies (Subfigure E.7b).

mean std min 25% 50% 75% maxStatistics factor

0

2

4

6

8

Activ

e En

ergy

Value

Cluster differentiation

(a) Cluster di�erentiation for households.

mean std min 25% 50% 75% maxStatistics factor

0

5

10

15

20

25

30

35

40

Activ

e En

ergy

Value

Cluster differentiation

(b) Cluster di�erentiation for companies.

Fig. E.7: Clustering method based on active energy consumption values for labeled householdsand companies

The results still show some variance in the data, particularly in the caseof companies. As a result of automatic clustering, all companies with PVs

129

Paper E.

have been assigned to same cluster, which is as desired. However, the im-plications of individual consumer behavior can be depicted from the largevariance values obtained in Subfigure E.7b. Therefore, in a data-driven soft-ware solution, an automatic anomaly detection would not be possible in sucha case, meaning that other data analytics methods should be applied for thisdata set.

E. Customer behavior prediction

The above-mentioned clustering technique was utilized in order to classifythe low-voltage grid customers into categories defined by their energy con-sumption patterns. Based on this classification, a step forward is taken inthe analysis towards forecasting models. A basic ARIMA model is used toillustrate the predicted consumption patterns for one of the clusters obtainedfrom the household labels and for the cluster containing the companies withPVs.

The plots in Figure E.8 represent the consumption values in kWh pernumber of samples (96 samples correspond to one day) for the two chosenclusters. It can be noticed from the plots that the predictions (red curves)follow the household and company profiles, resulting in MSE values of 0.009and 0.141, respectively. The low error values add up to the potential of usingprediction models based on clustering, however, in Subfigures E.8a and E.8bthe prediction curve is shifted from the actual measurements (blue curve), asa result of the ARIMA model fitting.

130

IV. Discussion

0 20 40 60 80

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

(a) ARIMA prediction for a cluster containing households;MSE = 0.009.

0 20 40 60 80

0.5

1.0

1.5

2.0

2.5

3.0

(b) ARIMA prediction for the cluster containing the companieswith PVs; MSE = 0.141.

Fig. E.8: ARIMA predictions based on clustering for households and companies

IV Discussion

The data analysis presented in Section III had the purpose of exploring thepotential uses of the available consumption data from the low-voltage grid.The analysis of the time-series data was based on statistical results, making itpossible to perform consumer classification/clustering out of the available ac-tive power measurements. From this, it can be concluded that the automaticclustering can be utilized as data pre-preocessing method, due to its abil-

131

Paper E.

Table E.2: Contributions to grid operations brought by the analytic methods.

DSO Operation Data accuracy concerns Analytic methods

Anomaly detection • missing/inaccurate data due to model in-consistency between the time-series and theGIS information

Profiling andPrediction

• customers suspected of fraud (i.e. stealingenergy)

• faults in the grid, related to possible cablefaults or power outages

Power balancing • unexpected change of pattern for a groupof customers not necessarily belonging to thesame substation

Clustering

Planning • necessary grid reinforcements due to cer-tain detected anomalies

Clustering andPrediction

• re-routing of information in the grid aspart of future grid planning and optimization

Monitoring • keeping track of the specific consumerswho are more prone to report anomalies

Profiling andClustering

ity to correctly place the six different individual user consumption patternsinto six corresponding clusters - three in each labeled category, household orcompany.

The results obtained from clustering still depict variance in the data, dueto the subjective behavior of the small producers (consumers with PVs). Fur-ther analysis was performed by using a simple ARIMA model for behaviorprediction. It can be noticed than even if the prediction follows the consump-tion patterns, ARIMA is not an accurate model in this case, as the predictedvalues are just a shifted version of the actual measurements. The modelcould be improved by taking into account seasonality in the data (week-days/weekends, holidays and seasons) and/or by introducing weather de-pendencies in the model. For example, in the case of consumers with PVs, apotential parameter of influence is the solar irradiation.

The data analysis study brings out other possibilities for the DSOs forusing the available data, than only for billing calculations. Understandingthe data is essential for understanding the behavior of the grid’s residentialconsumers, which is subjective to a large extent. This study is important forthe DSOs when taking into consideration future electrical grids consisting of100% renewable and distributed energy resources, due to some challenges

132

IV. Discussion

unaccounted for in the traditional low-voltage electrical grids:

• Mobile prosumers - it is expected that with the increasing proliferation ofelectrical vehicles (EVs), the amount of mobile users will also increase.Monitoring the users will then become even more challenging due totheir mobility and their subjective behavior, with more inconsistencyin the data. The resulting distribution grid is anticipated to develop arecurrent number of anomalies and imbalance in the distributed power,which can be addressed by some of the analytical methods presentedin Table E.2.

• Scalability - the amount and diversity in the data incoming from thedifferent distribution energy resources (DER) calls for a scalable andflexible data analysis solution. At the same time, the scalability mayalso refer to a collective group of operators (heat, water, transmissionsystem operators) who need to use their data for similar purposes asthe DSOs.

• Prosumers’ privacy - with a more accurate insight into the users’ electric-ity consumption and generation, the privacy issue evolves into beingmore sensitive. The trade-off lies between how much knowledge isneeded to provide the required and stable electricity supply and thebarrier towards accessing sensitive user data. One exception for break-ing the privacy rule is the case of customers who are suspected of fraud.

Given these challenges, the aforementioned study can bring contributionsto some of the DSOs daily operations by customer profiling, clustering andpredictions. The different concerns regarding accuracy in the data are pre-sented in Table E.2, along with the corresponding analytical methods that canhelp overcome them.

These methods are useful as a data sanity checkup in the different sit-uations where the available data is not labeled, missing or inaccurate. Theparticular lifestyle of the low-voltage grid consumers can nonetheless be de-ducted even after profiling and clustering, due to the high variance in thedata. This issue can be eliminated by performing a more refined classifica-tion, taking into account data seasonality.

The data sanity study was performed using only active energy measure-ments (consumption), though the developing AMI networks are capable ofcollecting more varied types of parameters, such as voltage and current traces.These values, combined with knowledge of the users’ consumption behavior,open up for the possibility of performing more accurate data analysis, forexample for anomaly detection.

The requirements for the future smart grids imply scalable computa-tional solutions for automatic anomaly detection, real-time grid monitoring,power balancing and planning. The computational power in an automatic

133

Paper E.

data-driven management system is challenged by the data variety, volumeand granularity, particularly when trying to adapt and optimize the existingDSOs’ operational system to real-time conditions.

V Conclusion

This study underlines the need for efficient data-driven solutions in the low-voltage electrical grid operation, as the traditional grids evolve into smartgrids. The data analysis presented in this work is meant to demonstrate howbasic statistical analysis can bring a contribution towards the challenges im-posed by new grid operating conditions and use cases which arise with theproliferation of smart grids. In this sense, predictions can be used for scenar-ios with new areas and entities in the low-voltage grid, in order to anticipateany operational constraints. The study also shows that low-voltage grid con-sumers can be characterized and classified by their consumption patterns inorder to facilitate some of the basic grid operations, such as anomaly detec-tion, power balancing, planning and monitoring.

It was found that due to the diversity in the users’ consumption patterns,the active energy alone is not enough for designing an automatic information-based management system. Additionally, scalable solutions depending onthe amount and variety of data require more information in the form ofvaried AMI parameters, weather-related variables or other machine learningtechniques.

Future research directions should test and take into consideration a morescalable solution for real-time data management operations in electricity grids,all the while accommodating for the imminent computational issues thatcome with the scalability.

Acknowledgment

This work is financially supported by the Danish project RemoteGRID, whichis a ForskEL program under Energinet.dk with grant agreement no. 2016-1-12399. The authors would like to thank Eduardo Prieto, Assistant Professor atUniversitat Politecnica de Catalunya, who helped defining the overall prob-lem statement and interpreting the results.

134

References related to paper E[1] Danish Energy Agency. Statistik, data, nøgletal og kort, February https://ens.

dk/service/statistik-data-noegletal-og-kort, 2018.

[2] Thy-Mors Energi Elnet A/S. Beskrivelse af forskningsprojekter i elnettet, https://www.tme-elnet.dk/om-os/forskningsprojekter.

[3] Danish Energy Association. Smart grid in denmark 2.0, https://www.usef.

energy/app/uploads/2016/12/Smart-Grid-in-Denmark-2.0-2.pdf, 2016.

[4] Christopher Bennett, Rodney Stewart, and Junwei Lu. Forecasting low voltagedistribution network demand profiles using a pattern recognition based expertsystem. 2014.

[5] Tobias Boßmann, Joachim Schleich, and Robert Schurk. Unravelling load pat-terns of residential end-uses from smart meter data. 06 2015.

[6] DAX. Network model management (nmm), https://dax.dk/

nmm-dansk-introduktion/.

[7] Energistyrelsen. Månedlig og årlig energistatistik, https:

//ens.dk/service/statistik-data-noegletal-og-kort/

maanedlig-og-aarlig-energistatistik.

[8] V. Ford and A. Siraj. Clustering of smart meter data for disaggregation. In 2013IEEE Global Conference on Signal and Information Processing, pages 507–510, Dec2013.

[9] Magnus Hornø Gottlieb. Danmark sætter ny rekord i vind, https://www.

danskenergi.dk/nyheder/danmark-saetter-ny-rekord-vind, 2018.

[10] S. Haben, C. Singleton, and P. Grindrod. Analysis and clustering of residentialcustomers energy behavioral demand using smart meter data. IEEE Transactionson Smart Grid, 7(1):136–144, Jan 2016.

[11] B. P. Hayes and M. Prodanovic. State forecasting and operational planning fordistribution network energy management systems. IEEE Transactions on SmartGrid, 7(2):1002–1011, March 2016.

[12] Kamstrup. Intelligent solutions for electricity utilities, https://www.kamstrup.com/en-en/electricity-solutions.


[14] Rasmus Løvenstein Olsen. Remotegrid - project description, https://www.en.remotegrid.dk/projectdescription/.

[15] M. Reis, A. Garcia, and R. J. Bessa. A scalable load forecasting system for lowvoltage grids. In 2017 IEEE Manchester PowerTech, pages 1–6, June 2017.

[16] Maria Stefan, Jose G. Lopez, and Rasmus L. Olsen. Exploring the potential ofmodern advanced metering infrastructure in low-voltage grid monitoring sys-tems. 2018 IEEE International Conference on Big Data (Big Data), pages 3543–3548,2018.







https://dax.dk/nmm-dansk-introduktion/

https://dax.dk/nmm-dansk-introduktion/

https://ens.dk/service/statistik-data-noegletal-og-kort/maanedlig-og-aarlig-energistatistik



https://www.danskenergi.dk/nyheder/danmark-saetter-ny-rekord-vind

https://www.danskenergi.dk/nyheder/danmark-saetter-ny-rekord-vind

https://www.kamstrup.com/en-en/electricity-solutions

https://www.kamstrup.com/en-en/electricity-solutions

https://www.en.remotegrid.dk/projectdescription/

https://www.en.remotegrid.dk/projectdescription/

Ma

ria Stefa

na

uto

Matio

n o

f SMa

rt G

rid

oper

ation

S thr

ou

Gh

Spatio-teM

por

al d

ata-d

riven

SySteMS

ISSN (online): 2446-1628 ISBN (online): 978-87-7210-445-4

Aalborg Universitet Automation of Smart Grid operations ... · intelligence to current power systems, helping with automatic anomaly de-tection and data accuracy diagnosis. Eventually,

Documents