Operated by Los Alamos National Security, LLC for DOE/NNSA Contractor Assurance Office (CAO) U N C L A S S I F I E D Learning About Learning Progress in.

Operated by Los Alamos National Security, LLC for DOE/NNSAContractor Assurance Office (CAO)

U N C L A S S I F I E D

Learning About LearningProgress in Lessons Learned at Los Alamos National Laboratory

presented to the

DOE Operating Experience Coordinators WorkshopMay 5, 2010

Learning About LearningProgress in Lessons Learned at Los Alamos National Laboratory

presented to the

DOE Operating Experience Coordinators WorkshopMay 5, 2010

Bob Stuewe

Operating Experience Coordinator

Contractor Assurance Office – Performance Feedback

Los Alamos National Laboratory

Bob Stuewe

Operating Experience Coordinator

Contractor Assurance Office – Performance Feedback

Los Alamos National Laboratory

Operated by Los Alamos National Security, LLC for NNSA


Operated by Los Alamos National Security, LLC for NNSAContractor Assurance Office


Bottom LineResults Take About Three Years to Manifest

End of CY 2006 163 processed per year

• Pass preliminary screening• Nearly all DOE/HSS• Safety• 10 internal per year

200 web users/month

250 in searchable archive

?? document accesses per month

?? searches/month

Trend Occurrences

Track Communications

2 FTEs• 1 FTE on the Trending

FY 2010 (Now) 800 processed per year

• Pass preliminary screening• Multiple pre-planned sources• All program/function areas• 150-200 internal per year

1500-3000 web users/month

>2100 in searchable archive

>30,000 document accesses per month

100-225 searches/month

Analyze/Trend Occurrences, Causes

Track Actions (risk-based)

2.5 FTEs• 1.2 FTE on Analysis

Slide 2





Learning About Learning Lessons About Effective Lessons Learned Programs

Champion Learning, Not Lessons

Reliable Processes Build Trust and Use

Clarify Roles to Eliminate Redundancy

Support Work Processes

Enable Risk Management

More Focused Equals More Effective

Build and Use Indicators of Effectiveness

Encourage Innovation

Slide 3





Champion Learning Not LessonsOperating Experience Programs are Key to Knowledge Management

Slide 4

Operating Experience Program

World of Possible Knowledge

Learning happens in the work unit in many, many ways(Required reading, training, requirements updates, work team meetings, plan of the day, procedure updates,

learning teams, process improvement teams, corrective action teams etc.)





Champion Learning Not LessonsFrom the Criteria for Performance Excellence (MBNQA)

Learning is one of the three pillars of organizational performance.

• Learning is referenced 103 times in 88 pages

• “The term “performance excellence” refers to an integrated approach to organizational performance management that results in (1) delivery of ever-improving value to customers and stakeholders, contributing to organizational sustainability; (2) improvement of overall organizational effectiveness and capabilities; and (3) organizational and personal learning.”

• “To be effective, learning should be embedded in the way an organization operates.”

Slide 5





Champion Learning Not LessonsKnow How Your Organization Learns and Support the Methods

Nuclear and High Hazard Facilities• Conduct of Operations: required reading, procedures, FLM weekly meeting,

critiques, facility lessons learned coordinators, etc.

Subject Areas• Electrical Safety Committee, Chemical Safety Committee, Glovebox Safety

Committee, Security Incident Analysis, etc.

Lower Hazard Organizations• Worker Safety and Security Teams, Learning Teams, Management Meetings,

mentoring, on-boarding, etc.

Institutional• Management review boards, requirements updates, training updates, institutional

web pages, etc.

Slide 6




U N C L A S S I F I E D Slide 7

Reliable Processes Build Trust and UseTrust Requires Relationships – With Products/Services Used

Operating Experience and Lessons Learned Process– Collects, screens, risk-prioritizes, and distributes operating experiences and lessons

learned from key external and internal sources using timely communications.– Three risk-priority levels with different disposition/communication pathways

Lessons Learned and Operating Experience Archive– Captures relevant lessons and best practices reported by the Laboratory managers

and workers, contractors, subcontractors, and from other DOE sites and external sources.

– Knowledge base

MirrorMirror: Translating Events into Actionable Information

– Communicates high-leverage actions, lessons learned and best practices associated with operating experience patterns and trends.





Reliable Processes Build Trust and UseProcess Management is Proven To Create Reliable Products/Services

1. Establish Process Management• Lean Six Sigma - Champion and Yellow Belt• Process Team (Weekly - Operations)• Integration Team (at least Annually - Integration and Strategy)

2. Benchmark (inside and outside DOE)

3. Define Product/Service Quality Specifications• Purpose -> Products/Services-> Customers-> Requirements -> Controlling characteristics• Defined processes (steps and roles) for each product (mapping)

4. Set up Process Controls and Improvement• Program Description, Procedures, Checklists, Tools• Use Contractor Assurance System: Goals, Measures, Assessments, Issues Mgt, Process

Improvement• Establish Task Roles (daily, weekly, monthly, quarterly, annual)• Annual Improvement Plan (effectiveness and efficiency)

5. Execute and Innovate

Slide 8





Reliable Processes Build Trust and UseReliable Processes Requires Reliable Operations

Slide 9

Monday

Process Team

Updates

Upcoming

Communications Plan

Operations

Improvement Plan

Wednesday

MIRROR Team

Organized by sections of the publication

Quality review and improve key messages and actions

Design year-end and next issue

Thursday

Screening Team

Process Team and occasionally the site office

Finalize screening

Determine dispositions

Write headlines

Publish Weekly Summary Email

Daily intake and preliminary screening Work on products and services (MIRROR, lessons, support customers) Work on Communications Plan and Improvements Plan

Tuesday

Integration

CAS Management Team

Institutional Management Review Board

As Needed

Friday

Week Close Out

In case extra time is needed to complete the weeks deliverables

Backlogs are deadly





Clarify Roles to Eliminate RedundancyFunctional Approaches Increased Costs, Decreased Effectiveness

LL requirements embedded in many directives, guides

Alignment and Integration can save taxpayer money

LANL approach• OPEX program supports multiple functions – use an integration team to plan annual

improvement (Safety, Security, Env Mgt, Project Mgt, Facilities, HPI, Training)

• Issue & Corrective Action Management - handles extent of condition for institutional/local, OPEX program helps with what’s in between

• QA/SCI – handles work-related suspect, counterfeit and defective items, OPEX serves as knowledge base, get directly involved at Urgent level

• Planned integration with internal security LL and Security Lessons Learned Center• Organizational and Subject Area LL Knowledge Bases and Process align with and link to

Institutional• Contractor Assurance System: Measures, Assessment Planning, Issue Effectiveness

Evaluation• Site office uses LANL screening and disposition process, oversight of higher risk

Slide 10





Support the Work ProcessesEasy to Think About the “Push”

Lessons Learned Integration with Work Performance

LL P

roce

ss T

eam

*M

ana

ger

sW

orke

rs

Monitor Internal & External Sources

of Lessons Learned and

Operating Experiences

1. Screen: Relevance Applicability Priority

2. Capture lessons learned in LL Archive

Communicate key Lessons Learned to

LANL managers , workers , and DOE

sites

Communicate relevant Lessons Learned to workers

Incorporate into Work Processes

Perform Work

Evaluate work to identify lessons

learned

Identify improvement opportunities

Share Lesson Learned with the

Laboratory

Lessons Learned Archive

Slide 12

Support the Work ProcessesWork is Always Underway – Don’t Try to Change Cars While Moving

Operated by Los Alamos National Security, LLC for NNSAContractor Assurance Offoce (CAO)


Support the Work ProcessesBeware of the Adverse Unintended Consequences of the “Push”

Common Lesson from OPEX: Not Thinking Through a Change

Nuclear Facilities• Procedures control the quality• Modifying procedures include many steps, for good reason

Conservation of Attention Law• Only so much time in a day• Rushing to implement a lessons learned can create rework

Resource Allocation is Not an OPEX Role• Acting on a lesson competes for resources• OPEX Coordinators do not know the daily priorities on the work floor

13





Support the Work ProcessesHard to Think about the “Pull” – But Equally Important

Lessons Learned Integration with Work PlanningL

L P

roce

ss T

eam

*M

ana

gers

Wor

kers

Access & search LL Archive for lessons relevant to new or modified work scope, training requirements ,

and/or procedures

Ensure that changes associated with lessons are effectively communicated to

workers

Use lessons to plan work:Project PlansDesign InputsIntegrated Work Documents (IWD)Laboratory proceduresLocal instructions and procedures Training plans and materialsOther work planning

Keep Lessons Learned Archive up to date

Define new or modified work scope, training requirements , and /or

procedures

Lessons Learned Archive



Support the Work ProcessesSupporting Work Processes Requires a Knowledge Base

Slide 15

MS Sharepoint

Reduced Cost• IT Costs

• Process Costs

Aligns with Work Processes• Topical Libraries

(key systems)

• Alerts (individual and topical groups)

• How-To Tools

• Full PDF search capability





Hazard inventory is pretty stable at the sites• Sharing known or low hazards provides little value to management• “New” and “Significant” Hazards important and valued

Risk Management• About reducing consequence or probability• Risk Management

— What is the risk? (to the mission)— How large are the potential adverse consequences?— What are you doing to handle it?— How likely are you to succeed?— Should you do more?

• Requires knowledge of organizational processes— Russian Hydro as an Example

Enable Risk Management To Engage Senior Management – Help Reduce Risk

Slide 16

Where Lessons Learned can add the most value to

management.



Work Planning Processes (Line)

TrainingDevelopment (Support)

Requirements Development (Support)

Enable Risk ManagementRisk-based Prioritization Matrix derived from O210.2 with Security from IMI levelsUrgent and Caution matrix essentially identical to Order – will add Best Practice un future

Communicate

URGENT

Caution

Informational

ClarifyScreenMonitorSources

Urgent = Immediate action is needed to ensure that the Laboratory is in a safe, secure, and environmentally sound mode relative to the information identified in the alert. Active engagement of relevant manager(s). Formal issues and corrective action tracking.

Caution = Timely action may be needed. Routine notification through Lessons Learned and Operating Experience Weekly Summary emails and LANL Home Page. LANL managers determine relevance and take needed action.

Informational = Future action may be needed. Posted to LANL Home Page if broadly applicable and actions clear. Important to knowledge base.





Enable Risk Management Risk Levels Differ from Site to Site

Slide 18

Pre-Screening Post Screening #HSS - Green Informational 21

HSS - BlueCaution 4Informational 44

HSS - RedCaution 4Informational 6

HSS - YellowCaution 38Informational 65

Urgent Lessons most commonly associated with DOE Safety Alerts and Advisories, defective

safety items, and other sources





Enable Risk ManagementSelect Sources to Support Risk Management – Might Change over Time

Primary Routine Sources MonitoringFrequency

DOE HSS Server/Web (LL, Alerts, OE Summaries, Security Lessons Learned Center (SLLC))

Daily

Partner Alerts (Bechtel, BWXT, UC, etc.) As Issued

ORPS (from Other Sites) Daily

Suspect Counterfeit/Defective Items Daily

U.S. Chemical Safety Board Monthly

Enforcement Actions and GAO Reports Quarterly

Consumer Products (CSPC, FDA) As Issued





More Focused Equals More Effective Tailor Communication Methods to Audience and Media Capability

Slide 20

1%

10%

12%

28%

49%

Dispositions of Last 400 Lessons Processed

Custom

LANL Homepage Only

Weekly Summary Email Only

Weekly Summary Email and LANL Homepage

Archive for Future Use



More Focused Equals More Effective Weekly Summary Email – Management and LL Roles with Prioritization

Recipients• Division Leaders and above• NNSA Site Office• LL Roles and Upon Request• Augments daily ORPS

Slide 21

Specific = Relevant1. Define Most Specific Customer

2. Select Words in Headline

3. Integrity Check

4. Order (First and Last Most Read)

Scan-able (6 most important/actionable; > 6 has lower read rate)



More Focused Equals More Effective LANL Homepage to Reach Workers Directly

Some Communication Overlap Helps Effectiveness

Main Users• Workers

Broadest lessons• Includes worker home

safety

Specific = Relevant• Even shorter headlines• Pre-job brief is major

customers

Slide 22



More Focused Equals More Effective Internal Lessons Learned Submission – Content over Format

Adobe Fill-able Form• Just the Facts

— Word Docs too slow, costly, hard to share with DOE

• Field Validation – reduce errors• Fields support processing

— Screening and prioritizing— Interface to DOE/HSS

• Includes picture objects• Submit Button linked to LL

Archive and Process• Becoming standard at facility

level

Slide 23



More Focused Equals More Effective LANL Mirror (Quarterly Publication)

Translating Events into Actionable Information• Events binned by systems

and processes• Patterns and trends

(incidents, occurrences, causes)

• Year-End and Two-Year Running Reflection

• Use statistics to identify changes/trends

• High-leverage lessons/actions aligned with greatest performance gaps

• Recognition for developing and sharing lessons learned

Slide 24

0 5 10 15 20 25 30

Packaging and Transportation

Suspect Counterfeit Items

Authorization Basis (DSA)

RCRA Compliance

Conduct of Operations

Largest Changes in Number of Occurrences by ProcessFY09 FY08

0

5

10

15

20

25

FY08Q1 FY08Q2 FY08Q3 FY08Q4 FY09Q1 FY09Q2 FY09Q3 FY09Q4

Num

ber

Fiscal Year Quarter

Baseline of Radiation Protection Occurrences

Mean = 4.1/qtr 95% Mean Confidence Level = 1.9/qtr



More Focused Equals More Effective LANL Mirror: Events Causes Checklist for Improvements

Slide 25

Do managers and workers understand what work practices are to be used?

Do managers and workers understand why prescribed work practices are the right ones and how to perform them well? Are managers and workers sufficiently engaged and motivated to do the work right, rather than making optimistic assumptions and avoiding work paths that require extra effort? Are enough of the right resources (skills, equipment, tools, support services) allocated to tasks to enable timely completion of work? Are the potential adverse impacts of changes to activities, processes, and systems effectively evaluated and mitigated? When activities cross organizational boundaries, are the communication and coordination methods as timely and effective as the work flow requires?



Build and Use Indicators of LL Effectiveness Fact-based Improvement is the Objective

Use CAS Dashboard• LL Indicators

— Processing time performance• Urgent within 24 hours

• Caution within 7 days

— Document Access Rate— Search Rate— Knowledge Inventory— Internal LL Generation Rate

• LL Program Execution— Routine Process Milestones— Progress on Improvement Plan— Corrective Actions

Slide 26



Build and Use Indicators of LL EffectivenessNeed measures than can help understand Cause and Effect

Slide 27



Build and Use Indicators of LL Effectiveness Need measures than can help measure use for Work Planning

Slide 28



Encourage InnovationLANL Performance Communication Center

Modernizing Communications

• Chat with an Expert— Builds Knowledge Base

• Blog

• Role-based information— Relevant Tools

— Relevant Requirements

• Best Practices (coming soon)

— Custom version of LL Form

— Recognition

Slide 29



Encourage InnovationSub-Unit Approaches Evolve as their Learning Processes Mature

R&D key lessons included in “On-Boarding” materials for new hires (might become a best practice)

Local LL Archives Linked to Institutional Archive

• Project Management

• Engineering

• D&D

• Maintenance

• Facilities

Learning Teams (linked to HPI implementation)

Slide 30



Learning About LearningProgress in Lessons Learned System at LANL

Questions?

Slide 31

Operated by Los Alamos National Security, LLC for DOE/NNSA Contractor Assurance Office (CAO) U N C L A S S I F I E D Learning About Learning Progress in.

Documents

s s i f i e d learning

s s i f i e d operated

s s i f i e d slide

los alamos national

learning lessons

nnsa u n c

security teams

learning teams