Top Banner
March 2008 Overcoming Poor Data Quality Amidst a Crisis v Ulf Månsson SWECO
24

Overcoming Poor Data Quality amidst a Crisis

Nov 16, 2014

Download

Documents

Nostrad

 
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Overcoming Poor Data Quality amidst a Crisis

March 2008

Overcoming Poor Data Quality Amidst a Crisisv

Ulf MånssonSWECO

Page 2: Overcoming Poor Data Quality amidst a Crisis

Combined expertise in consulting engineering, environmental technology and architecture

So where does ULF constantlyinform (nag) about FME?

Page 3: Overcoming Poor Data Quality amidst a Crisis

SWECO

4

The Nordic region’s leading provider of consulting services in the fields of engineering, environmental technology and architecture

4,900 employees in 10 countries

3.9 billion SEK- annual net sales

24,000 assignments each year

7,000 clients

Page 4: Overcoming Poor Data Quality amidst a Crisis

Present Establishments

Sweden:2.560

Norway: 600

Denmark: 40

Finland:1.000

Estonia: 225

Lithuania: 175

Employees 20074.900

Russia: 110

Bulgaria

Czech rep: 270Slovakia

Page 5: Overcoming Poor Data Quality amidst a Crisis

Assignments Worldwide

Assignments in more than 100 countries since 1903

Increased international presence

Ongoing assignments in 75 countries

20% –

15% –

15% –

14% –

12% –

9% –

8% –

5% –

2% –

Industrial Engineering

Water and Environment

Transport and Civil Eng.

Energy Systems

Building Service Systems

Architecture

Structural Engineering

Project Management

Geographic Positioning

Page 6: Overcoming Poor Data Quality amidst a Crisis

Ulf Månsson

Modern organizations depend more on data quality than ever before.

Only a few spare parts in stock

Large centralized districs, no local offices

No extra staff

Modern Organizations

Page 7: Overcoming Poor Data Quality amidst a Crisis

Ulf Månsson

Modern Organizations

Organizations rely on systems and their data. Geographic data are crucial:

No local offices Poor knowledge of local geography

Where are my supplies/spare parts?

Where are my people? How can they find me?

Page 8: Overcoming Poor Data Quality amidst a Crisis

Ulf Månsson

C R I S I S

The importance of data quality becomes strikingly clear in a crisis management process.

FME is an important tool for managing ”ad-hoc” situations at a major crisis.

The hurricane Gudrun is the best Swedish example.

”Spontaneous ETL”

Instantly connect data from many systems to improve quality and derive the needed information output.

Page 9: Overcoming Poor Data Quality amidst a Crisis

Hurricane Gudrun 2005

Wind speed 10-46 m/s 20 000 km electrical cables damaged 2 500 people in workforce Out of supplies, gas, equipment,etc.

The – first map illustrating the situation. (2D-Pointreplacer)

Power outages lasted for weeks We used FME for weeks

Page 10: Overcoming Poor Data Quality amidst a Crisis

Services Provided for Our Client

Information to the crisis management teams Rescue workers and municipalities Daily delivery of maps and lists of persons/addresses

Data quality

Prognoses for re-connections

Providing maps for field-work

Maps for reconstruction

Page 11: Overcoming Poor Data Quality amidst a Crisis

”Spontaneous ETL”

Large power company – many IT systems Outage management systems Network schema systems Network Information Systems (NIS) Customer Information Systems (CIS)

Oracle Text files Excel Shape files Etc…

FME El Dorado !!!

Page 12: Overcoming Poor Data Quality amidst a Crisis

Challenge: Making a MapShowing Areas with Outages

Page 13: Overcoming Poor Data Quality amidst a Crisis

Connecting the Parts

OutageManagement

System (Oracle)

Customer IDOutage data

NetworkInformation

System(Oracle Spatial)

Customer IDX,Y

Customer Data(SAP->Txt files)

Customer IDAddresses, Zip-

Codes etc.

XLSLists of addresses

Bufferedareas with

outage data

Shp filesGIF files

FMEWorkbenches

Page 14: Overcoming Poor Data Quality amidst a Crisis

Key FME Features

The diversity of readers and writers GIF, XLS, Shape, etc… (today we would have used pdf)

Joiner-functionality

Buffer and dissolver

Scripting Today, we would have used the fanout-functionality

Page 15: Overcoming Poor Data Quality amidst a Crisis

It all worked gloriously (in theory) but bad data…

The mercyless ConvexHullAccumulator exposes bad quality.

Defining municipality by zip-code from CIS…

Solution Use municipality-

borders to clip data

Page 16: Overcoming Poor Data Quality amidst a Crisis

ConvexHullAccumulator - Good Data

Page 17: Overcoming Poor Data Quality amidst a Crisis

Ugly Data

”You should at least have used the

2DPointReplacer before deployment!”

Page 18: Overcoming Poor Data Quality amidst a Crisis

Connecting the Parts

OutageManagement

System (Oracle)

Customer IDOutage data

NetworkInformation

System(Oracle Spatial)

Customer IDX,Y

Customer Data(SAP->Txt files)

Customer IDAddresses, Zip-

Codes etc.

XLSLists of addresses

Bufferedareas with

outage data

Shp filesGIF files

FMEWorkbenches

Page 19: Overcoming Poor Data Quality amidst a Crisis

”To know what you not know”

Polygons representing reconnection forecasts A simple geometry manageable by each municipality The buffer radius correspond to the data quality Polygons marked as ”We simply don’t know…”

Page 20: Overcoming Poor Data Quality amidst a Crisis

Conclusions

Keep it simple GIF and PDF files are easy to distribute and print.

Spontaneous ETL Know your FME and it’s a great problem-solver in

unexpected situations.

FME Prototyping A great way to define and describe processes. These can

also be implemented in other systems.

Page 21: Overcoming Poor Data Quality amidst a Crisis

SWECO Vision

To be Europe’s most respected knowledge company in consulting engineering, environmental technology and architecture

Page 22: Overcoming Poor Data Quality amidst a Crisis

Recent artwork(or ”unexpected

results”)

Page 23: Overcoming Poor Data Quality amidst a Crisis

More artwork

Page 24: Overcoming Poor Data Quality amidst a Crisis

25

QUESTIONS?

More information:[email protected]

Thank You!