Roessler CHap One

7/30/2019 Roessler CHap One

1/40


2/40


3/40

Control SystemMigrations: A Practical

Project Management

Handbook

DANIEL ROESSLER

MOMENTUM PRESS, LLC, NEW YORK


4/40

Control System Migrations: A Practical Project Management Handbook

Copyright Momentum Press, LLC, 2013.

All rights reserved. No part o this publication may be reproduced, stored in a retrieval system,

or transmitted in any orm or by any meanselectronic, mechanical, photocopy, recording orany other except or brie quotations, not to exceed 400 words, without the prior permission

o the publisher.

First published by Momentum Press, LLC

222 East 46th Street, New York, NY 10017

www.momentumpress.net

ISBN-13: 978-1-60650-443-7 (hard cover, case bound)

ISBN-10: 1-60650-443-6 (hard cover, case bound)

ISBN-13: 978-1-60650-445-1 (e-book)

ISBN-10: 1-60650-445-2 (e-book)

DOI: 10.5643/9781606504452

Cover Design by Jonathan Pennell

Interior design by Exeter Premedia Services Private Ltd.,

Chennai, India

10 9 8 7 6 5 4 3 2 1

Printed in the United States o America.


5/40

To my Mom and Dad who established the strong oundation o values onwhich I rely every day. You have given me the gits o independent thinking,perseverance, and a strong work ethic. Thank you or your support, love, andencouragement throughout my lie.


6/40


7/40


8/40

Hardware and Sotware Requirements Speciications 41Control System Bid Instructions 41

Decision Criteria Matrix 43Selecting a Control System Vendor 48

Engineering, Procurement, and Construction Services 49Requirements Deinition 50

A Complete EPC Bid Request Package 53Bid Evaluation and Project Award 57

Summary 57

4. Scope, Schedule, and Budget 59

Scope 62Overall Organization and Approach 62Instrumentation 64Electrical 66Controls 68Civil-Mechanical-Building 70Communications and Integration 72Testing 73Training and Documentation 75Cutover 77

Budget 78Schedule 81Summary 83

5. Project Staffing 85Deining Project Resource Requirements 86

Project Organizational Chart 87Roles and Responsibilities Matrix 88Project Schedule Resourcing 90

Extending the Project Team 90

Establishing Team Communication 93Building an Eective Team 96Summary 98

6. Training 99Engineering 103Maintenance 105Operations 106Others 109

Summary 110

viii Contents


9/40

Contents ix

7. Progress Monitoring, Change Orders, and Reporting 113Monitoring 115

Scope Monitoring 117Schedule Tracking 118Budget Evaluation 120Overall Progress Calculations 122

Adjusting Plans 124Change Order Management 125Project Reporting 128Summary 135

8. High-Risk Areas 137

Graphics 139Third-Party Systems or Application Communications 143Staing Changes 144Poor Teamwork 146Unoreseen Logic Complexity 147Field Construction Obstacles 148Cutover Details 150Summary 152

9. Cutovers 153

Correct Methodology Decision 154Thorough Design Details 157Comprehensive Plan 161Prepared Field Team 163Control Room Leadership 164Strong Operations Coordination 166Complete Loop Packages 167Eicient Checkout Process 168Summary 170

10. Project Closeout and Lifecycle Management 173Documented Completion Scope 175Phased Financial Closing 177Remaining Milestones Schedule 178Project Delivery and Acceptance 178Final Project Review Meeting 180Liecycle Management 181Summary 184

Supplemental Resource List 187

Index 189


10/40


11/40

xi

List of Tables and Figures

Table 1.1. Common control system migration projectROI considerations 5

Table 2.1. Prioritization o key knowledge characteristics 18

Table 2.2. Summary o options or FEL execution 20

Table 2.3. Common FEL scope deliverables 21

Table 3.1. General control system unctional speciication

table o contents 40Table 3.2. General control system selection decision criteria matrix 44

Table 3.3. Example vendor analysis 49

Table 3.4. Key areas to deine in bid documents 50

Table 4.1. Common approaches to organizing scope o work documents 63

Table 4.2. Instrumentation scope checklist 66

Table 4.3. Electrical scope checklist 67

Table 4.4. Controls scope checklist 69

Table 4.5. Civil-mechanical-building scope checklist 71

Table 4.6. List o common third-party connections to control systems 72

Table 4.7. Communications and integration scope checklist 73

Table 4.8. Testing scope checklist 75

Table 4.9. Training and documentation scope checklist 76

Table 4.10. Cutover scope checklist 78

Table 5.1. RACI method deinitions 89

Table 6.1. Summary o training location options 102

Table 6.2. Engineering training recommendations 104

Table 6.3. Maintenance training recommendations 106


12/40

Table 6.4. Operations training recommendations 109

Table 8.1. Summary o high-risk areas 138

Table 8.2. Example logic complexity table 148

Table 9.1. Cutover methodology comparison 156

Table 9.2. Suggested I/O cutover list ields 158

Table 10.1. Project closeout scope punch list 175

Table 10.2. Final project documentation checklist 179

Table 10.3. Initial liecycle management plan elements 182

Figure 1.1. Typical control system liecycle reliability curve 6

Figure 1.2. Common third-party solutions requiringcontrol system integration 10

Figure 3.1. Sample EPC services bid tab spreadsheet 55

Figure 4.1. Basic project management triangle 60

Figure 4.2. Example budget headers or services 80

Figure 4.3. Example budget headers or materials 80

Figure 4.4. Schedule resource category options 82

Figure 5.1. Sample project organizational chart 88

Figure 5.2. Example partial roles and responsibilities matrix 89

Figure 5.3. Sample control system migrationproject team meeting agenda 95

Figure 6.1. Basic learning pyramid 103

Figure 7.1. Project monitoring process 117

Figure 7.2. Basic percentage o budget spent spreadsheet 121

Figure 7.3. Common worklow or handling potential change orders 126

Figure 7.4. Basic project management progress report 130

Figure 7.5. Earned value summary inormation 132

Figure 7.6. Monthly spending and earned value versus budget chart 133Figure 7.7. Overall earned value versus cumulative spending curves 133

Figure 9.1. Key elements o cutover success 154

Figure 9.2. Cutover planning spreadsheet 162

Figure 9.3. Sample daily cutover progress report 165

Figure 9.4. Example analog input checkout orm 169

Figure 10.1. Essential elements o the project closeout process 174

Figure 10.2. Example closeout budget 177

Figure 10.3. Simple example project acceptance orm 180Figure 10.4. Sample inal review meeting agenda 181

xii List of Tables and Figures


13/40

xiii

Acknowledgments

The author acknowledges the ollowing individuals, who have enriched thisbook by sharing their knowledge and encouragement through reviews, sug-gestions, and/or inormative discussions:

Dr. Vassilios TzouanasFor your review, insightul comments, and eedback.You are a respected colleague and valued riend.

Nigel JamesFor your enthusiasm, excitement, and encouragement about thisproject. Thank you or discussing your ideas with me, or writing a thoughtuloreword, and or your treasured riendship.

Billy AdneyFor reviewing the book, providing valuable eedback, and will-ingly sharing your vast process control knowledge and experience with methrough the years. It is a pleasure to call you my riend.

Joel SteinFor your eagerness to work with me on this project and contribut-ing your extensive technical publishing expertise.

Cindy DurandFor your review o my drats and your swit responsiveness.

Millicent TreloarFor sharing your eedback and oering guidance toimprove this book.


14/40


15/40

xv

Foreword

The release o this book provides the automation industry a comprehensiveguide through the complexities o control system migration projects. It shouldbe a required reerence or all project and program managers, engineers, andother resources involved in migrations. The book is insightul, methodical,

well-written, and based on actual experiences with meaningul anecdotes andrealistic examples. Whether you are part o an end user, engineering company,system integrator, or vendor organization, this book will be a valuable tool.

I have spent my entire career o almost 30 years in the controls and auto-mation arena working directly or major reneries as well as managing systemintegration and automation service companies. I have learned that controlsystem migration projects are among the most complex that most controlengineers ace in their career. From the blended resource teams required, toaddressing customization and interace details, the challenges are numerous.My personal involvement in countless migration projects has taught me thateach one is unique, but there are certain methodologies and best practices thatare applicable across the board.

I worked as Dans manager at a system integration company or a period otime and have known him or over 15 years. He is the perect author or thisbook on managing control system migration projects. I can vividly remembera project in the pulp and paper industry where developing and using theseconcepts helped save the project ater some early issues. I also remember arenery operations manager once stating o Dans work, This is the best con-trol system migration FEL I have ever read.

I know rsthand that this book is the ruit o numerous nights in thecontrol room. It evolved over a number o years while Dan was experiencingthe challenges o managing control system migration projects. The wealth o

experience shared in this book is ormed as much rom the ailures and deeats


16/40

xvi Foreword

as rom the successes. I am proud to see his journey result in a resource thatwill simpliy the migration experience or others.

We are entering another golden age o engineering, innovation, and proj-ect work. Our work orce is aging, and the speed to market o new technologyis at its astest pace in history. How do we ll the knowledge gap and handdown to our younger brethren the value o our experiences? That is exactly

what this book does. I believe reading it will establish a rm oundation orcontrol system migration success. I plan to make this book required readingor all my sta.

Theodore Roosevelt once stated, The credit belongs to the man whois actually in the arena. Dan is one o those men who dares greatly and has

spent himsel on a worthy cause. Nigel JamesPresident-Burrow Global Automation


17/40

xvii

Preface

In 1992, I had just earned my degree in electrical engineering and was begin-ning ull-time employment at a chemical (polymer) plant that I had internedat the previous two summers. The plant was in the midst o a control systemmigration and I was given an opportunity to help congure the new systemunder the watchul guidance o a senior controls engineer. It was my rstmigration project and a great learning experience. That project taught me alot about the process o converting points, graphics, and control logic rom an

older system to a newer system including some o the unique challenges thatmigration projects present.

The world o process automation has changed signicantly in the 20 yearssince my rst migration project. The most visible changes are in technology.

We have shited rom proprietary, largely independent control systems, some-what isolated rom other plant systems to PC-based solutions using standardoperating systems. Todays control systems now have more open connectivityand are commonly integrated with numerous third-party systems and applica-tions. They are also much easier to congure with extensive standard eaturesand unctions, minimizing the need or customization.

The way that control systems are used has also evolved. As a result, endusers now expect much more rom their control systems than they once didin most areas, such as reliability, fexibility, speed, and unctionality. And,importantly, the process o making control systems purchasing decisions haschanged. Now that the dierences in control systems oered by various ven-dors are more subtle, the system selection process requires end-users to moreclearly understand and prioritize their unique control system needs and deci-sion criteria.

As legacy control systems move toward the end o their liecycles and

in some cases reach obsolescence, migrations to newer technologies are


18/40

xviii Preface

becoming more prevalent. Even those companies which might preer to con-tinue extending the lie o their existing control system are oten compelled

to migrate because o system limitations related to critical issues like cybersecurity. As a result, companies across process industries are currently mak-ing large nancial investments to replace older control system inrastructure.This trend is expected to continue or the oreseeable uture. The pace otechnology advancements means that even some newer control systems willreach the end o their liecycle aster and require replacement or upgradesooner.

There are many variations on the denition o control system migrationslargely driven by the motivation o the individual tasked with dening it. For

the purposes o this book, I have included what some reer to as modern-izations because I believe the same considerations and project managementchallenges apply whether you are converting an older control system to a di-erent vendors system or upgrading to your existing vendors latest solution.Control system migrations update older systems to newer technology by sub-stantially changing both the hardware and sotware o a control system. Minorsotware version upgrades are not considered migrations.

Note that the term control system generally reers to the combination ohardware and sotware technology that provides the ability to operate andcontrol eld devices rom a centralized location. This includes Distributed

Control Systems (DCSs) as well as Programmable Logic Controllers (PLCs),and Human Machine Interace (HMI) systems. There are also other com-monly used control systems such as Saety Instrumented Systems (SISs) andSupervisory Control and Data Acquisition (SCADA) systems. For the pur-poses o this book, I group all o these systems and reer to them using thecommon term o control systems. While there are some dierences in thedetailed unctionality, purposes, applications, and architectures o these sys-tems, the process o migrating them to newer technologies requires the samebasic process which is the ocus o this book.

Control system migration projects have many obstacles that are not always

obvious. This handbook discusses some o these unique challenges and recom-mends various tools and approaches or handling them. Control system migra-tions require you to understand and document all aspects o the conguration

within the old system as well as develop a transition plan to the new system,oten times in phases, while minimizing impact to manuacturing production.The success o your migration project will be greatly infuenced by the choicesthat you make about when and how you move rom the planning phase to theexecution phase and nally to the operational phase o your project.

Fundamental project management principles apply to control systemmigration projects just as they apply to other disciplines, such as civil or


19/40

Preface xix

mechanical projects. In addition, good project management requires under-standing subtleties like:

When a scope is the right granularity?

When a schedule is realistic and accounts or hidden gotchas?

How to smoothly transition through the various phases o a project?

The project team is another key component o any projects success. Teammembers must understand the overall project and be clear on their specicresponsibilities. This handbook is intended to educate all parties on key areaso migration projects and provide insight into how to best plan and executethe project. I hope that you will reer back to this handbook oten as yourpreerred resource in guiding you to the successul completion o your controlsystem migration project.

Eectively managing control system migration projects is about a processand successul projects are a result o many actors. There are many ways toapproach a control system migration project and no two control system migra-tion projects are identical. The chances o success are directly related to goodplanning and the approaches used to manage and execute the project. This istrue o most projects and certainly applies to control system migration proj-ects. The project manager must be amiliar enough with the project to know

what is reasonable and be able to proactively identiy and manage high riskareas.I you are new to control system migrations, this book should advance

your understanding o the migration projects steps, help you identiy key areason which to ocus time and eort, and provide you with tools to better planand execute your project. I you are a control system migration veteran hope-ully you will nd many amiliar concepts and approaches along with a ewunique perspectives and valuable suggestions. It is my hope that this bookdelivers insight into control system migration projects to a broad audience oend users, system integrators, EPC companies, vendors, and control system

engineering students.Whether you are an operator helping design graphics as part o the con-

trol system migration team or the instrument and electrical (I&E) managerresponsible or the eld installation, this book will provide an overview o thesteps and stages o the migration process, tips and suggestions or success,and a better appreciation o your role in the project. Managers in operations,maintenance, and engineering departments should also nd this book helpulin better understanding the value and benets o a control system migration.Improving the understanding o all parties involved in a control system migra-tion project will help each understand how they infuence the projects suc-

cess. When the project is successul, the entire team wins.


20/40

xx Preface

My goal with this handbook is to capture migration best practices andensure successul transitions rom one control system to the next. The book is

organized in a logical project workfow and begins by examining how to buildan eective justication that will help initiate your control system migrationproject. We then cover the details o perorming a comprehensive Front EndLoading (FEL) study which orms the critical oundation o a successul con-trol system migration project. Our next ocus is on how to build complete andeective scope, schedule, and budget documents that are consistent relative toone another, enhancing your chances or a successul migration. There is also

valuable inormation included to help guide users through a logical vendorselection process that reduces the emotional component o decision-making

which is commonly a source o rustration on migration projects.The book includes a chapter on how to plan and execute training whichis oten an overlooked aspect o migration projects essential to success. Wereview common migration project challenges related to graphics, third-partyapplication integration, and numerous other high risk areas detailing specicissues and discussing ways to avoid them. We also examine the nuances o suc-cessully planning, managing, and executing a system cutover which is gener-ally the area o highest risk on a migration project.

Many processes are outlined, templates provided, and topics discussed spe-cic to project management activities in this handbook. We cover key elements

o project stang and how to build an eective team. We also examine howto handle project monitoring, change order management, and project report-ing throughout the project detailing how to make these project managementresponsibilities an extension o normal project activities reducing the amounto time and eort required. Finally, we address the project closeout processand how to transition rom the project to an eective liecycle managementprogram or your new control system.

The content o this book includes methods, approaches, and tools thathave worked or me on specic migration projects throughout my career. You

will want to selectively use and adapt this inormation so that it works best

or your particular project. It is my hope that by identiying and outlining keyconsiderations o a control system migration project and giving context tothese elements, you will be better prepared or migration success.

This handbook should help you, whether or not you are amiliar with con-trol systems, to approach the migration project in a methodical manner. Thisbook contains boxed item eatures o some o my experiences in anecdotes andexamples. I also outline some ways to establish common expectations amongall parties early in the project process so that everyone is aligned and workingtoward the same goals.

Many o the tables in this book can be used as checklist by your projectteam. Project team members cannot know or remember everything so this is


21/40

Preface xxi

written as a handbook that will enable you to quickly reerence specic areaswhen needed. Each chapter o the book begins with an overview o the topics

and ends with a summary o key takeaways, similar to the ollowing takeawaysrom this Preace:

Three Key Takeaways

This handbook is intended or a broad audience o people with diverse roles either

directly or indirectly involved in control system migration projects.

Topics covered in this book are comprehensive and include everything rom the

migration project justications to vendor selection processes to project manage-

ment reporting best practices.

The processes, tools, and tips shared in this book are a result o my experiences with

migration projects or more than 20 years as an end user, system integrator, and

control system vendor.

KEY WORDS

Control System Migration, DCS Migration, Control Room Modernization, SCADA Upgrade,

Cutover, Front End Loading, DCS System, Control System Projects, Project Management

Methodologies, Project Management Tools and Techniques, Project Management Process, ProjectManagement Books, Project Management Best Practices, Project Management Resources, Control

System, Control System Design, Control Systems Engineering, Distributed Control System,

Process Control Systems, Control System Upgrades


22/40


23/40

xxiii

About the Author

My early career was as a controls engi-neer and project manager or a chemi-cal company. I worked in two plantlocations over roughly nine years. Ourcontrol systems included a combina-tion o DCS and PLC systems that

were operated rom multiple con-

trol rooms. During my time at theplants, I managed several migrationrelated projects including DCS con-troller upgrades, a control room con-solidation, and operator workstationmigrations. Our team also had respon-

sibilities or the conguration and maintenance o these control systems aswell as third-party systems and applications interaced with the DCS, such asthe process historian and advanced process control solutions.

The next six years o my career were at a system integration company. Itwas during this time that I was exposed to migration projects that involvedmultiple industries, numerous control systems, and diverse scopes. My respon-sibilities included leading migration Front End Loading (FEL) studies andmanaging migration projects. There are noticeable dierences among indus-tries and systems, but I also learned that there are many more commonali-ties which contribute to control system migration project success. I was alsoresponsible or proposal development and as a result understand the details othe bid process. This experience has helped me understand how to positionbid documentation to get the most complete and comparable proposals rom

control system vendors and services providers.


24/40

xxiv About the Author

Finally, I have spent a number o years working or process automationvendors. This includes several years at a major control system vendor as well as

time with sotware vendors providing applications that integrate with controlsystems. I learned a tremendous amount about challenges that control systemmigration projects present to the vendors themselves. I also gained insightinto the strategies and approaches that vendors use in responding to bids.

As a result, I can oer tips on avoiding common misunderstandings during thebid process and streamlining control system vendor selection.

I have learned through my diverse work experiences that all parties,whether the control system vendor, a system integrator, or the end user, wanta successul control system migration. Unortunately, not everyone has the

same denition o success, hence the need or a handbook to help establishalignment among all team members. My experiences are unique to me and myperspectives are a direct result. Ater over 20 years involved in some way withcontrol system migration related projects, I am condent that the strategies,approaches, and processes that I provide in the ollowing pages are a roadmapto successully managing your migration project.


25/40

1

Migration Project

Justification

It is a testament to the automation vendors o the 1970s, 1980s, and 1990sthat so many legacy control systems rom these periods continue to operateindustrial acilities today. However, many process control engineers will tell

you this is both a blessing and a curse. Because these control systems continueto operate reasonably well, one o the biggest challenges controls engineers

ace is getting support or control system migrations. Short o ailures by thecontrol system that lead to signicant downtime, the need or migration proj-ects is oten scrutinized and questioned.

When compared with capital expenditures on more tangible and easilyunderstood return-on-investment (ROI) projects, control system migrationsare requently considered lower priorities, which oten results in repeateddelays to unding them. Understanding the ROI o a project like an equip-ment debottleneck is straightorward because additional throughput capacityis easily converted to dollars. The ROI on control system migration projectstends to be much less tangible and more dicult to convert to nancial ben-ets that are easily agreed on by key decision-makers.

It is important that someone take ownership o building the case or justi-ying a control system migration while also establishing realistic expectations

within the organization regarding the benets o replacing the control systemwith newer technology. This person is oten the controls engineer who may ormay not be the project manager on the eventual control system migration proj-ect. For example, some organizations have a separate project group to manageall capital projects. The project manager, i someone other than the controlsengineer, should be identied and involved in the justication process i at all

possible. This is the individual who will ultimately be responsible or ensuring

1


26/40

2 Control System Migrations: A Practical Project Management Handbook

that the project delivers on the economics dened in the Authorization orExpenditure (AFE) process.

In this chapter, we examine how to build an eective justication so thatyour control system migration project will be appealing. We begin by discuss-ing dierent unding request strategies as well as how to identiy and involvekey stakeholders that can help support justication eorts. Determining theappropriate timing or your control system migration is also examined. Wereview how to capture your control system migration projects ROI anddevelop supporting business cases and examples. Finally, detailed sections areincluded, which highlight common areas to consider in your justication pro-cess such as parts availability issues and limitations in integrating with third-

party systems and applications.

DETERMINING YOUR APPROACH

One o the rst steps in the justication process is determining what und-ing to initially request. There are two options, either request unding or theull project or request unding or a Front End Loading (FEL) study, to bet-ter dene the scope, schedule, budget, and other project details. RequestingFEL unding is generally a better approach. It requires substantially less initial

unds so is oten easier to gain approval. Also, a good FEL study details thescope, identies areas o risk and uncertainty, and generates a tighter estimate

with reduced variability. When the subsequent ull migration project undingis requested, there is typically much greater certainty in the estimated proj-ect cost increasing the condence o decision-makers and approvers. Even ithe migration project is not immediately approved, the FEL documentationis usually largely applicable when the project does move orward requiringminor revisions to account or any updates.

When requesting ull unding without having completed a thorough FEL,the budget is much more at risk and typically not more detailed than 25%.

For some organizations, this is an acceptable approach. I this approach isused, be sure to include money or upront FEL work as part o the overallscope and build this upront engineering eort into the schedule. This stepis crucial to help identiy and resolve potential problem areas beore projectexecution begins.

It is essential to include as many key stakeholders as possible in developingthe business case or a control system migration project. Identiying parties

with a vested interest early in the process and understanding their controlsystem needs not only helps you build a compelling business case but alsostrengthens support or the project. For instance, visit with the maintenance


27/40

Migration Project Justifcation 3

manager, instrument and electrical (I&E) supervisor, and I&E technicians tounderstand what challenges they have in maintaining the system. Maybe the

team is having trouble with common parts availability on input/output (I/O)cards, controllers, termination panels, or operator consoles? Maybe they havebeen purchasing updated eld instrumentation with expanded diagnosticcapability and cant take ull advantage o these newer diagnostics in the oldercontrol system? The team will appreciate you seeking their input and will bemore supportive o the project i they believe it will help alleviate some o theirspecic work challenges.

Operations, maintenance, and engineering are the common organiza-tions represented in the utilization and care o most control systems. While

not as obvious, there are other groups to include as well. InormationTechnology (IT) is oten involved in getting data out o the control systemor other applications. Increasingly, the separation o company IT and con-trol system networks are blurred. While most controls engineers will arguethat the process control network (PCN) is a separate entity, at the very least,critical inormation is transerred across networks daily to support busi-ness applications. Talking with the owners and users o various businessapplications that utilize control system data is also important. They mayhave diculty getting the inormation they need rom the control system,may need dierent ormatting or granularity, or have other challenges that

can also contribute to the business justication or migrating the controlsystem.

Another actor in the justication process is identiying the proper timingto propose the migration. The timing o when a control system migrationshould occur is not an exact science. Control system viability is based on bal-ancing a combination o actors such as reliability, total cost o ownership, andsystem perormance to meet dened business objectives. Whenever a givenelement is out o balance, it can signal that it is time to evaluate whether amigration might be necessary.

Obviously, there are numerous actors in the migration timing consider-

ation process. Each company must make the decision based on the availablecapital and relative prioritization with other projects. Some companies willtake a pro-active approach and continually evaluate the long-term viabilityo their control systems. Other companies hold on to their existing controlsystems well past the optimal point and are hesitant to migrate until a specicissue, such as security vulnerability, system ailures, etc. orces them to takeaction. It is important that the decision to migrate your control system is aneducated one that is based on a ull understanding o not only the nancialROI o a migration project but also those benets that may be dicult toquantiy. Every company should have a control system liecycle plan that is


28/40


reviewed and updated at a regular requency, which will help them evaluatethe liecycle status o their system.

DEFINING ROI

So how do you calculate ROI or a migration project? There is no univer-sally agreed upon answer. Companies use a variety o ways to calculate ROI,rate-o-return analysis, and other nancial metrics to determine whethera proposed project meets dened payback thresholds. I wont tackle ROIcalculation methodology details other than to note that it is seldom a sin-

gle actor that should be used to justiy a control system migration proj-ect. Instead a combination o additive actors is generally used to build astrong ROI case. There are o course projects that are approved based onactors other than ROI, such as saety and maintenance reliability projects.In these situations, many o the justication cases outlined below will stillapply.

For many industrial acilities today, control system migrations are part oa larger vision. For instance, acility siting requently identies problematiccontrol room locations. As a result, companies are building new control roomsin alternate locations within the acility and using this opportunity to upgrade

their control system. Many companies have also reduced stang and subse-quently consolidate control rooms as part o streamlined operations. In thesecases, control rooms are oten re-designed and control systems are updated aspart o the consolidation process as well.

When there are larger projects, such as these driving control systemreplacements or modernizations, it can reduce the challenges o the justi-cation process but comes with other pitalls. In these situations, the controlsystem migration is not the ocal point o the project and the scope as well asthe budget can get minimized to balance other parts o the project. I scopeor budget reductions to the control system migration occur and are substan-

tial they can impact the long-term benets o the migration. It is also notuncommon to see critical control related design strategies or control systemselections being made or nancial rather than technical reasons with prioriti-zation given to how it impacts the overall project rather than what is ideal orlong-term operations.

What are some eective justications or standalone control system migra-tion projects? The answer is complex because it is largely dependent on agiven acilities specic situation. Some common considerations that contributeto control system migration project justications are identied in Table 1.1below.


29/40


Table 1.1. Common control system migration project ROI considerations

Issue Result ROI Considerations

System ailures Decreased reliability Increased downtime

Lost production Unplanned outages Product schedule and

shipment disruptions

Parts availabilityor obsolescenceissues

Extended outages(oten unplanned)

Lost systemunctionality

Increased maintenancerequirements

Increased maintenancecosts

Lost production Impact to production

quality

Dicultyintegratingwith newerapplications andsystems

Cant realize ullpotential o newapplications

Key data not easilyavailable to decision-makers

Less optimizedoperational perormance

Slower businessdecisions

Higher costsassociated with projectimplementations andongoing support

Reducedavailability osupport services

Dicultytroubleshootingmaintenance issues

Extended schedulesor projects requiringengineering

Increased maintenancecosts

Increased engineeringcosts

Delays in realizingbenets o projectsinvolving control systemconguration

Operationalineciency

Inability to takeadvantage o currentbest practices

Operator mistakescontributing to productquality issues anddowntime

Reduced product quality Increased downtime Operator stress

As you begin to put together your justication, consider how many o theconsiderations in Table 1.1 are applicable to your control system migrationproject. The ROI or your specic project will be unique and may includenumerous other issues related to your particular migration. We examine thesecommon issues that drive control system migration projects in more detailthroughout the remaining sections o this chapter.


30/40


SYSTEM FAILURES

The most straightorward justication or a control system migration is whenthe system is ailing. When your plant shuts down multiple times in a yeardue to control system ailures, the justication process becomes immediatelyeasier. However, no controls engineer wants the rustrations or visibility thatcomes with these system ailures. I this is the situation at your company, thenit is possible that you are well past the best time to migrate your control sys-tem. Ideally, your migration project takes place prior to reaching a stage inthe liecycle where an unacceptable ailure rate is occurring. As indicated inFigure 1.1 below, ailure rates increase as control systems move toward theend o their liecycle.

Figure 1.1. Typical control system lifecyclereliability curve.

Decreasing

failure

rate

Increasing

failure

rate

Constant

failure

rate

Observed failure

rate

Constant (random)

failures

Early

infant

mortality

failure

Time

Failure

rate

Wear out

failures

Control system migration projects take a substantial amount o denitionand planning. I reliability has already become a signicant issue, the projectschedule is likely to be compressed. This oten causes deviations rom method-

ical and proven approaches or selecting the right vendor, perorming thor-ough upront planning, taking ull advantage o eatures and benets o thenew system and developing a comprehensive operational transition strategy.

I control system ailures create an urgent project need, project costs alsotend to increase. The higher costs are associated with compressed schedules,paying a premium or engineering or vendor resources, and poor project plan-ning. Without proper planning another area that is impacted is the abilityto take advantage o improved unctionality within the new control system.For instance, alarms in the control system are congured like or like withalarms in the old control system with no thought given to their applicability

and eectiveness. This can oten reduce the eectiveness o the new control


31/40


system and translate into a higher ongoing cost o ownership or the system.I your acility is in this situation and there is a push to get the control system

migration done quickly to eliminate shutdowns, take time to understand andcommunicate the potential negative impact on the project.

I reliability issues exist but are not to the point o orcing schedule com-pression, then you are in a strong position to get approval or the migrationproject while also ollowing best practices to execute the project. The eco-nomics associated with control system ailures are more tangible and clear tomanagement when they are experiencing lost production. It moves the argu-ment or the need to migrate a control system rom the theoretical to the real.The example below outlines the signicant nancial impact control system

ailures can have on a companys operations.

Example

A polymers plant with a continuous process has the control system ail three times in

the course o a year. Each ailure costs at least one day or 24 hours o production at

a normal production rate o 40,000 pounds per hour (PPH). The average price per

pound o product is 70 cents. Three days o downtime result in a nancial impact due

to lost production alone o $2 million.

This does not account or any cost associated with getting the unit back up

and running, such as maintenance and operations personnel overtime. This is also

unscheduled downtime, which means that the plant is oten not able to take advantageo these outages or other maintenance activities that could prevent or delay a scheduled

outage at a later time.

The impact o the example above is not isolated to lost production. Intodays environment o on-demand scheduling, the entire product wheel canbe thrown o when events like the example above occur. I this lost produc-tion occurs at a critical point in the schedule when you are making a specialtypolymer that is only produced on that specic production line it may result

in a missed shipment. This damages your brand. I it causes your customer tomiss production targets and shipments to their customers, it may even resultin your company losing the customer account.

What is the nancial impact o these additional areas? That is dicult toanswer and I do not suggest that you try to dene any quantitative nancialimpact or this in your ROI calculation unless it has actually occurred and

you have real numbers. However, I would suggest that you build this what-iscenario into your supporting justication documentation to raise awarenessthat when you start having system ailures, there is a domino eect on yourbusiness. This message will resonate with management teams who are verycustomer ocused.


32/40


PARTS AVAILABILITY OR OBSOLESCENCE ISSUES

As legacy systems get older, vendors reduce and eventually eliminate the avail-ability o certain components whether I/O cards, controllers, consoles, orany variety o subcomponents. Most vendors use a staged approach, but thesystem cost o ownership immediately begins escalating as parts phase-outsbegin. There are a ew dierent approaches that companies typically use toaddress potential issues with parts availability:

1. Pre-stock additional spares in-house

2. Coordinate with a distributor or third-party vendor to stock additional spares

3. Find a reurbished parts dealer as an alternate supplier.

Unortunately, parts cost a premium with any o these options. Maintainingexcess in-house inventory is inecient, has tax implications, and is generallynot desirable or most companies. Most distributors will charge stocking eesin addition to the elevated parts costs rom the vendor. And while there aremany reputable reurbished parts dealers the reliability o re-built parts isalways a concern.

In addition to the higher system costs, a second issue when parts becomeobsolete is the potential elimination o expansion capability. Control system

capacity limitations can create signicant challenges to a companys ability toexecute projects needing integration with the control system as highlighted inthe example below.

Example

Your plant site is going to build a new unit, which will add 500 points to your existing

control system. A last-time buy oering on controllers or your version o the control

system was two years ago. Your current controller is near recommended maximum

load and will not handle the additional unit. What are your options? There are creative

ways to use serial communications rom programmable logic controllers and only bring

absolutely required points into the controller. There is also an option to eliminate

nonessential points in the existing system, but there are likely not many that will

be identied. Both o these options are workarounds that just avoid the inevitable

conclusion that you need to replace the control system. Maybe the appropriate choice

is to use the new unit as a compelling reason to migrate?

Most vendors provide upgrade paths that can help buy additional timeoperating an older system without requiring complete transitions to a newersystem. These may be partial upgrade paths or designed interconnectivity o


33/40


older and newer control system components. This can be a viable option undercertain circumstances. However, when considering these options keep in mind

that the vendor is motivated to lock you in with a nancial commitment sothat when a complete upgrade does occur it will be more dicult to justiymigrating to another vendors system because o the additional investment

you have made in the current vendor. I you have no desire to consider othervendors and are planning on moving to your existing vendors latest platormin the uture, then this is an excellent option. One urther caution is that once

you start mixing components o various generations o products, it generallyincreases your maintenance costs and makes overall system maintenance moredicult.

I limited parts availability or obsolescence is a part o the issue that moti-vates the need or a control system migration, there are a ew quantitative val-ues that can be included in the ROI. The rst recommendation is to capturethe dierential cost between historical and anticipated escalated parts cost. I aparts price has increased by 25% in a year and the plant purchases roughly veannually, then capture that increase as additional cost associated with keepingthe existing control system minus any normal annual escalation pricing.

A second nancial indicator to track and capture is the increased cost oadding points to the control system. Cost per I/O is an average dollar amountthat represents the cost o hardware, sotware conguration, and any other

I&E and engineering services related to adding an I/O point to the controlsystem. Most controls engineers have a good idea o what this number is ortheir system. Consider areas where pricing has increased and calculate a newcost per I/O as the system nears end o lie. Use your historical cost per I/Oor the system as a baseline to do a comparison and capture the dieren-tial. This cost per I/O will increase dramatically as parts availability and moreimportantly, parts obsolescence becomes an issue.

In some cases, the severity o the parts availability issue may be limited todelays in parts delivery. For instance, instead o a standard our-week deliv-ery, a controller that is being phased out may require eight weeks to deliver.

While this may seem like little more than an inconvenience, it can have anancial impact in certain situations. I the part is needed or a project lon-ger lead times can usually be planned into the schedule, but or maintenanceactivities that is oten not the case. The only ways around this issue are topay expediting ees when available as an option or to stock extra compo-nents. Both o these approaches again increase the maintenance costs asso-ciated with the system. Isolated instances o parts availability issues can beovercome, but when numerous parts within a system become dicult toobtain in a timely manner the practical longevity o the system should beevaluated.


34/40


DIFFICULTY INTEGRATING WITH NEWER APPLICATIONS

AND SYSTEMS

The early generations o control systems were largely intended to work inisolation as standalone entities. Over the years this philosophy has changed.To varying degrees, modern control systems are interaced to and integrated

with many third-party systems and applications such as those refected inFigure 1.2.

Figure 1.2. Common third-partysolutions requiring control systemintegration.

Lab InformationManagement

Systems (LIMS)

Manufacturing

Execution Systems(MES) or data

historians

Advanced ProcessControl (APC)applications

Real-timeoptimization

solutions

Supply chain and

schedulingsoftware

Asset management

and maintenancesystems

The architecture and proprietary nature o historical control systems otendoes not lend itsel to seamless communication with these other products.

With many older generation control systems the interaces and/or integrationto these other systems is dicult and in a ew cases impossible. Brute orcemethods or customized solutions are oten used to enable communications or

transer data. These solutions are both labor intensive and dicult to main-tain. Even when data can be successully passed rom the control system, it isoten in a less than ideal ormat. This can reduce the eectiveness o third-party applications and systems decreasing their business benets.

Dening the ROI associated with interacing and integration issues is notstraightorward. First, document any customization that has been done toacilitate existing communications. I there are maintenance issues or chal-lenges with these customizations, then apply a maintenance cost to them,especially i there is recurring work involved. Second, note any manual activi-ties that take place as a result o system deciencies, such as the case outlined

in the example below.


35/40


Example

An inerential model was just developed, which resides on a third-party server. Themodel predicts several product quality parameters that will be used in an improved

control algorithm within the distributed control system (DCS). The expected cost

savings resulting rom reduced o-quality production losses is $3 million annually.

Unortunately, the model cannot pass predicted value data directly to the DCS because

there is no standard communications driver between the inerential model application

and the older control system.

Two options are identied as viable solutions. One option is to build a custom

driver, which introduces another maintenance point into the system and is expensive.

A second option is to have the inormation sent to the data historian and require that

the operator look up the value at some regular requency. The operator would thenhave to manually enter that inormation on a graphic at the DCS console so that it can

be used by the control algorithm.

Neither option is ideal, but to quickly get the improved control algorithm in place,

you select the option o having the operator manually enter the data at regular intervals.

This option requires operator time, distracts rom other operator responsibilities, and

runs the risk o a manual entry error that can aect plant perormance.

In this example scenario above, you would want to capture the costs aswell as the risk associated with this manual process in building your justica-tion. I newer control systems oer standard drivers that can securely pass datarom the inerential model to the control system, you would also be able tonote that as an additional benet to a migration.

It is equally important to identiy any deciencies in integrated third-party applications and systems that are related to the control systems infex-ible architecture and limited communication capabilities. Validate that newercontrol systems can use standard drivers or other communication methods toexchange inormation with the third-party system to eliminate the decien-cies. In most cases you will nd that they can. I in your situation they do,

you have the option o adding these to the benets o a migration either assupporting inormation or by including quantitative nancial benets in yourROI calculation. I would caution that unless you can determine a reasonablenancial benet value to claim, which will not be controversial, you may bebetter o just documenting this as an additional benet and citing supportingexamples.

REDUCED AVAILABILITY OF SUPPORT SERVICES

An oten overlooked pitall o older control systems is the challenge in nd-ing resources to support the system. This is true o both engineering and


36/40


maintenance support including vendor, third-party system integrator, and in-house resources. The retirement o many o the original contributors to the

design, installation, engineering, maintenance, and operation o these oldersystems results in the loss o signicant knowledge, which is not easily recap-tured. In the case o vendors and system integrators, they eventually reach abreak-over point where it is no longer reasonable to support these systemsbecause there is simply not enough need. Training or maintaining systemexperts much beyond the end o the product lie may not be cost eective.

When knowledgeable resources can be ound to support aging systems, theyare oten expensive and demand a premium knowing you have ew options.

You will want to capture any service price increases related to your system in

your ROI as a burden on the total cost o ownership. The example belowillustrates this point.

Example

Last year your maintenance contract with the control system vendor was $50,000. The

system has gone into the nal years o support and the vendor has increased the annual

support contract to $80,000 or the same scope o services citing ewer resources

available to support the older system as a primary reason. It is not uncommon to see

these kinds o increases or services at the end o a control system lie.

Lets assume your vendor normally includes a 5% annual escalation in the support

contract. In this circumstance beyond that 5% there is an additional increase o$27,500 that is directly related to the late stage liecycle o the control system. These

support costs will continue to increase until the system support services are no longer

available. As you build justication or a migration project, you capture the dierence

between the normal escalation and the current support contract pricing as added cost

to continue operating with your existing control system.

To be air to vendors, their support costs increase as systems near the endo their liecycle as explained in the example above. However, many vendors

also use this as an opportunity to help motivate end user companies to moveto their latest system.

When using external resources such as vendor or system integrator engi-neers and technicians to support near end-o-lie control systems, the avail-ability o these resources usually becomes more challenging, meaning you mayhave to wait longer or their services. When your control system was in themiddle o its liecycle, you may have been able to call and schedule someone to

work on your control system within a ew days to a week. An older system withewer resources to provide support may require scheduling a month or morein advance. When the services are needed or projects, you may be able to planor this, but on maintenance activities you oten cannot wait. For example, i


37/40


you have a workstation ail, you might need a quick response. In this situation,maintenance work will either be delayed or you may have the option o paying

extra ees or a premium or an expedited service call.To account or the increased cost o third-party services, review the pric-

ing or previous projects or maintenance activities. Request a quotation or thesame scope asking the service provider to update pricing with current rate. Besure to explain the purpose so that the service provider does not think it is anactive project. Use the dierential cost minus any normal escalations to showany increased cost and include this in your ROI calculation. You might also

want to ask or an estimate o a similar work scope on a newer control systemand include it or comparison purposes.

While issues nding third-party service providers or your control systemcan be challenging late in the lie o a control system, losing knowledgeablein-house resources over time can cause even more diculties or end usercompanies. In many older systems, a tremendous amount o coding, graphicsscripting, and other customizations were required to achieve the desired fex-ibility and unctionality rom the control system. In many companies, thesecustomizations are not well-documented and in-depth knowledge o the con-trol system is isolated to a ew individuals. When the resources responsible orprogramming, maintaining, and operating these systems are no longer avail-able due to retirement, job changes, etc., it reduces the eciency and increases

the cost o most control system related activities. For example, troubleshoot-ing activities can take substantially longer, leading to maintenance ineciencyand longer periods o downtime. When system changes are required, engi-neering and conguration eorts can also be both time-consuming and costly.

A percent eciency actor can be used to account or some o these eectsin an ROI calculation. Assuming an eciency actor o 1.0 or the expertresource, the eciency actor or a less experienced resource may be 0.70,

which means a 30% reduction in eciency, which can be applied to both costand schedule or control system-related activities. It is an estimated actor, butas long as you make logical assumptions and document them it is a reasonable

approach. This eciency actor can be applied to engineering and mainte-nance activities using in-house resources and included in the calculation ototal cost o ownership.

OPERATIONAL INEFFICIENCY

Understanding the capabilities o newer control systems is essential or iden-tiying operational improvement opportunities that modern technologiescan provide. Justications should not only be based on the costs associated

with the weaknesses o the existing control system but also on capturing the


38/40


operational benets o newer control systems. My thoughts on migrationprojects and operational ineciency are captured very well in the ollowing

article excerpt:

The objective should not be to simply replace and replicate, but rather to innovate.

DCS migration is an opportunity to design, implement, and maintain a 21st-century

control system that will enable the manuacturer to operate more eciently and

saely; position it or market growth; and rm up its stability or a quarter-century

or more. Further, it is an occasion or process improvement and expansion, as well

as a much needed opportunity to address the gap between how the plant is currently

operated and controlled and how it should be.1

The new unctions and eatures available in control systems today along

with the emergence o standards and best practices can enable major improve-ments in operational eciency compared to older control systems. For exam-ple, the automation industry has made tremendous progress in establishingbest practices in situational awareness areas over the past twenty years. Asour knowledge has grown on how to improve situational awareness, controlsystems have evolved to incorporate tools to align with those best practices inmodern systems.

These new system capabilities combined with the best practices can sig-nicantly improve operational eciency and perormance. Better situationalawareness helps operators to more quickly recognize and act on abnormal

events, which can be saety, quality, or equipment related. Better designsreduce operator distractions so that the operator can ocus more time on con-trol o the product quality and throughput. Improved contextualization oinormation also results in aster and better inormed decision-making. Thespecic impact on ROI includes incident avoidance, production quality andrate improvements, and reduced operational downtime.

Are your companys control room operators as ecient and eective aspossible? In most cases, the answer is unequivocally no and it is well-knownand generally accepted. Consider dening the costs o operator ineectiveness

where you can identiy and claim a realistic portion in your ROI calculation asbenets or a modernized control system.To capture operator ineectiveness, rst review the operational incidents

including saety, quality, and lost production over the past ve years. Capturethose that were likely due to or contributed to by deciencies in situationalawareness. Quantiy these incidents in terms o production losses, downtime,or o-quality product. It is unreasonable to expect that a new control system

will avoid every operational issue, but you can claim a reasonable percentage

1 Matt Sigmon, DCS Migration: Failure is Not an Option And Doing Nothing is Not a Solution,ControlXXV, no. 12 (December): 4142.


39/40


rom enhanced situational awareness tools such as improved graphics and bet-ter alarm handling and management. Be sure to state the basis o your assump-

tion as part o your ROI calculation explanation. The example below illustrateshow an operational incident can be used to capture ineciencies and operatorineectiveness with an older control system.

Example

An operator is responsible or running the distillation area within a chemical plant.

The HMI graphics were designed 25 years ago when the control system was installed.

They do not use current recommended best practices or color, layout, background,

or alarming. During a thunderstorm, a food o alarms occurs as temperature

deviation alarms are activated. The operator acknowledges the alarms but in theprocess acknowledged an alarm that one o the column eed pumps has shutdown.

Eventually the entire unit is shutdown. The enhanced alarm management capabilities

and streamlined HMI graphics available in a modern control system built using current

best practices would likely have avoided the alarm food and made the pump eed alarm

more easily identiable.

Another aspect to consider regarding operational eciency is reliabilityand percent uptime. Newer instruments have the capability to send a lot more

diagnostic inormation to the control system helping to quickly identiy theroot cause o maintenance issues. This enables a preventative approach tomaintenance oten avoiding or minimizing the operational impact o instru-ment and equipment problems. Many o the older control systems do not ullysupport the diagnostic capabilities o newer instruments and equipment. Inaddition, the internal diagnostics built into modern control systems can helpavoid and certainly minimizing troubleshooting time associated with controlsystem issues. In older control systems identiying root cause issues can be di-cult resulting in less ecient troubleshooting eorts.

In legacy control systems, the programming languages were oten cus-

tomized and vendor specic, while with newer systems they are standardized,common languages. This enables companies to bring much more consistencyto how things are programmed and reduces customization. The result is a sim-plication o troubleshooting or both maintenance technicians and engineers,

which can reduce downtime.These last two aspects o operational eciency are not something I would

recommend trying to quantiy. However, I think they are important pointsthat should be included in the business case justication dening the benetso control system migration. I would suggest including realistic case examplesapplicable to your operations as part o your supporting documentation.


40/40


SUMMARY

Each organization has unique decision processes when it comes to undingprojects. There is no replacement or your understanding and knowledge o

your individual organization. When key decision-makers must decide on themerits o a control system migration, they requently overlook the bene-cial aspects and ocus on the main points. They consider that the migrationmay include potential operational downtime, create chaos in the control roomduring the transition, require re-training o personnel, and cost a signicantamount o money.

Common misperceptions and biases regarding control system migrationsrequire controls engineers and project managers do an extremely eective job

dening the benets and ROI o the project to gain support and approval. Inthis chapter, we have examined some o the common motivations or migra-tions. As stated earlier, it is seldom a single actor but an additive combinationo several o these areas that ultimately builds the successul case or a controlsystem migration. Many o the benets o a modernization are intangible anddicult to quantiy in an ROI calculation. It is important to document thesebenets even i not numerically, to build understanding within the organiza-tion o your migration projects business value.

Many times, justication eorts are ocused on reliability, obsolescence,and other problems with the existing control system. These are valid and arecertainly key components o the case or migration. However, as you develop

your justication case, also document how improvements in technology willgive you an opportunity to improve operational perormance. Outline ways totake advantage o new unctionality and better tools within modern controlsystems as they relate to improving your process operations.

Gaining the support o others with a vested interest in seeing a migrationoccur is a critical starting point. Do not be araid to incorporate multiple sce-narios and justication points into your business case. Ultimately, the decisionto approve a control system migration will largely be based on your ability to

sell the value to your management team.

Three Key Takeaways

Include as many key stakeholders as possible early in the process to help build the

business case or your control system migration.

Justication is likely a combination o multiple actors and the ROI is an additive

result o these components.

Do your homework and identiy the operational perormance improvements that

you can take advantage o as a result o new tools and unctionality within a mod-

Roessler CHap One

Documents