IAEAInternational Atomic Energy Agency
TOOLS AND METHODS FOR ENHANCING IRRS EFFECTIVENESS
AND EFFICIENCY
Presentation to the Workshop on Lessons Learned from IRRS Missions
Moscow, Russian Federation
9-11 December 2014
IAEA IRRS enhancements 2
Outline
• On the IRRS missions in general
• Continuous improvement of the IRRS programme
• Conclusions
IAEA IRRS enhancements 3
IRRS BASICS
IAEA IRRS enhancements 4
IRRS Process
• Prep-meeting about 6 months before
• Self-assessment by host• First mission: 12 days• Report finalization about
3 months• Follow-up missions after
2-3 years, about 7 days
IAEA IRRS enhancements 5
IRRS Guidelines
• Published in May 2013
• A series of Targeted Consultancies was held for collecting experience and advice for a thorough revision
• Novel edition will be compiled based on the consultancies
IAEA IRRS enhancements 6
IAEA Safety Standards used in IRRS
IAEA IRRS enhancements 7
Basis for findings: GSR Part 1 Requirements
IRRS Modules
+ other IAEA Safety Standards
IAEA IRRS enhancements 8
Missions in 2006-2014
Korea (f)
Niger UAE Zimbabwe
Mexico Germany Canada (f) France
Mauritius Ukraine Russia Switzerland Netherlands
Cameroon Sierra Leone UK (part 2) Australia (f) Belgium Cameroon
Kenya Namibia Vietnam Slovenia Czech Rep. Vietnam (f)
Uganda Madagascar Lebanon Ukraine (f) Germany (f) Finland Russia (f) Slovenia (f)
France Gabon Botswana Canada USA Korea Greece UK (f) Jordan
UK Australia Spain Peru China Spain (f) Slovakia Poland Pakistan
Romania Japan Cote d'Ivoire France (f) Iran Romania Sweden Bulgaria USA (f)
2006 2007 2008 2009 2010 2011 2012 2013 2014
3 9 8 7 4 9 4 6 10
IAEA IRRS enhancements 9
Participation in the missions (2006-2014)*
• No. of experts (including IAEA): 373• No. of participations: 707• From 59 countries
010
2030
4050
ALB
ALG
ARG
AUS
AUT
BEL
BIH
BKF
BUL
BRA
CAN
CHI
CPR
CUB
CZR
DEN
EGY
ETH
FIN
FRA
GFR
GRE
HUN
ICE
IND
IRE
ISR
ITA
JOR
JPNLIT
MEX
MO
RN
ETNIR
NO
RN
ZEPA
KPO
RRO
KRO
MRU
SSA
FSL
OSL
RSP
ASU
DSY
RSW
ESW
ITA
NTU
NTU
RU
AEUK
UKR
USA
Expe
rts
in IR
RS m
issi
ons
Num
ber
of p
artic
ipati
ons
Num
ber
of e
xper
ts
* by October 2014
IAEA IRRS enhancements 10
CONTINUOUS IMPROVEMENT OF THE IRRS PROGRAMME
IAEA IRRS enhancements 11
IAEA Nuclear Safety Action Plan
IAEA is to “Strengthen IAEA peer reviews in order to maximize the benefits to Member States”.
In specific:
2.4.1 IAEA Secretariat to review the effectiveness of their peer reviews: IRRS, and therein
2.4.1.2 Define effectiveness criteria, apply the criteria to a representative number of selected IRRS missions and prepare a report
IAEA IRRS enhancements 12
Ways to improve
• IRRS Training• IRRS Mission Standards• Feedbacks• Measuring effectiveness and
efficiency• Prompt Evaluation Report
IAEA IRRS enhancements 13
First Basic IRRS Training Course
• First held in October 2013• 62 participants from 38 MSs + EC• In the framework of the EC-IAEA
cooperation in the field of Nuclear Safety
• Covered all aspects of the IRRS programme, 16 Training Lectures, 3 Information Lectures
• Test Questionnaires for all Lectures
IAEA IRRS enhancements 14
Feedback on the course
Evaluation of the BIT Course by participantsQuestion Mark
4.64
How do you rate the training course presentations from the point of view of
the information therein and their completeness 4.45
their clarity 4.30
their format 4.29
How do you rate the training course lecturers from the pointof view of
their competence in the field 4.69
the way they held the presentations 4.26
What is your opinion on the course-related organization and arrangements by IAEA? 4.69
Average: 4.47
Number of answers: 56
How useful you consider the training in your possible future IRRS activity?
IAEA IRRS enhancements 15
Feedback from the Test Questionnaires
0.500.550.600.650.700.750.800.850.900.951.00
1 2 3 4 5 6 7 8 9 10 11
Solution ratios in Test Questions No.1
0.500.550.600.650.700.750.800.850.900.951.00
1 2 3 4 5 6 7 8 9
Solution ratios in Test Questions No.6
IAEA IRRS enhancements 16
2nd Edition
• Held in October 2014• 63 participants from 39 MSs + EC• Even more favourable feedback
Evaluation of the BIT2 Course by participantsQuestion Mark
4.41
How do you rate the training course presentations from the point of view of
the information therein and their completeness 4.42
their clarity 4.18
their format 4.13
How do you rate the training course lecturers from the pointof view of
their competence in the field 4.73
the way they held the presentations 4.17
What is your opinion on the course-related organization and arrangements by IAEA? 4.49
Average: 4.36
Number of answers: 62
How useful you consider the training in your possible future IRRS activity?
IAEA IRRS enhancements 17
On-site training
• By invitation from MSs• Minimum 10 competent
participants• First held at ONR (Bootle, UK)• 15 participants• Excellent acceptance
Evaluation of the BIT2-ONR Course by participantsQuestion Mark
4.21
How do you rate the training course presentations from the point of view of
the information therein and their completeness 4.14
their clarity 4.21
their format 3.86
How do you rate the training course lecturers from the pointof view of
their competence in the field 4.54
the way they held the presentations 3.79
What is your opinion on the course-related organization and arrangements by IAEA? 4.64
Average: 4.20
Number of answers: 14
How useful you consider the training in your possible future IRRS activity?
IAEA IRRS enhancements 18
Refresher Training
• On the 0th day of the missions• Covers details important for the given
missions:• Overview of the IRRS Process• Methodology of the review• Evaluation and documentation of Observations• Report writing• IRRS Modules to review• IRRS Mission Schedule
• Held in 2014 in the Netherlands and France
IAEA IRRS enhancements 19
Feedback
Evaluation of the Refresher IRRS Course (NET) by participantsQuestion Mark
4,70
How do you rate the training course presentations from the point of view of
the information therein and their completeness 4,45
their clarity 4,40
their format 4,47
How do you rate the training course lecturers from the pointof view of
their competence in the field 4,90
the way they held the presentations 4,76
What is your opinion on the course-related organization and arrangements by IAEA? 4,60
Average: 4,61
Number of answers: 20
How useful you consider the training in your possible future IRRS activity?
Evaluation of the Refresher IRRS Course (FRA) by participantsQuestion Mark
4.39
How do you rate the training course presentations from the point of view of
the information therein and their completeness 4.32
their clarity 4.23
their format 4.18
How do you rate the training course lecturers from the pointof view of
their competence in the field 4.55
the way they held the presentations 4.23
What is your opinion on the course-related organization and arrangements by IAEA? 4.23
Average: 4.30
Number of answers: 23
How useful you consider the training in your possible future IRRS activity?
IAEA IRRS enhancements 20
Ways to improve
• IRRS Training• IRRS Mission Standards• Feedbacks• Measuring effectiveness and
efficiency• Prompt Evaluation Report
IAEA IRRS enhancements 21
Mission Standards – ARM Report Template
Under development / discussion
IAEA IRRS enhancements 22
Mission Standards – Team Assignment
IAEA IRRS enhancements 23
Mission Standards – Report Template
• Cross-contribution table• Instructions for reviewers
• Issues to cover• Format of observations
IAEA IRRS enhancements 24
Mission Standards – Schedule
IAEA IRRS enhancements 25
Ways to improve
• IRRS Training• IRRS Mission Standards• Feedbacks• Measuring effectiveness and
efficiency• Prompt Evaluation Report
IAEA IRRS enhancements 26
Feedbacks on the IRRS process
• Questionnaire on the quality of the ARM – reviewers at mission start
• Questionnaire on effectiveness of the mission – reviewers at end of mission
• Round-table conclusions and suggestions – reviewers at end of mission
• Questionnaire on effectiveness of the mission – host after mission
• Questionnaire on effectiveness of the initial mission – host before follow-up mission
• Workshop for team leaders (January 2013)• Workshop for experienced reviewers (April
2015)
IAEA IRRS enhancements 27
Ways to improve
• IRRS Training• IRRS Mission Standards• Feedbacks• Measuring effectiveness and
efficiency• Prompt Evaluation Report
IAEA IRRS enhancements 28
Effectiveness and efficiency of IRRS
• NSAP requires establishing criteria for efficiency of IRRS missions
• 16 Performance Indicators introduced to characterize the E&E of a mission, such as:• Team size; team experience;• Time available to prepare for the mission;• Feedback from team members and from
hosts;• Coverage of the Action Plan by findings;• Balance of findings;• Conciseness of mission report; etc.
http://gnssn.iaea.org/regnet/irrs/Pages/IRRS_pub_docs.aspx
IAEA IRRS enhancements 29
Examples of Performance Indicators (1)
3
3.5
4
4.5
5
Miss1 Miss2 Miss3 Miss4 Miss5 Miss6
mar
k
mission
Host Feedback
0.000.100.200.300.400.500.600.70
Fu-1 Fu-2 Fu-3 Fu-4 Fu-5 Fu-6 Fu-7
Ratio of open issues in follow-upsRecommendations Suggestions All
0.000.100.200.300.400.500.600.700.800.901.00
Fu-1 Fu-2 Fu-3 Fu-4 Ini-1 Ini-2 Ini-3 Ini-4 Ini-5 Ini-6 Ini-7 Ini-8 Ini-9
ratio
of m
embe
rs w
ith e
xper
ienc
e
missions
Team Experience
0306090
120150180210240270300330360390
FRA
UK
JPN
MEX
SPA
UKR GFR
FRA
-fu
CAN
UK-
fuR
US
CPR US
RO
MSP
A-f
uR
OK
GFR
-fu
SLO
SWI
CAN
-fu
SWE
SLR
FIN
BUL
UK-
fu2
RU
S-fu
CZR
BEL
com
pleti
on ti
me
in d
ays
Mission
Report Completion Time
Optimum Acceptable Need attention
0 0.2 0.4
IAEA IRRS enhancements 30
Examples of Performance Indicators (2)
Balance of findings: R*GP, R/S const.
Expectations: a) R and GP are negatively correlatedb) R and S are positively correlated
1) The expected correlations are not present in all missions
2) The reasons in case of R and GP are:• reviewers feel obliged to compensate by GP • some topics (with possible issues) are more
frequently considered than others (with GP)
3) The reasons in case of R and S are:• R -> S: compromise between the team & host• S -> R: the team tends to be stricter
or the specificities of the given mission
0
5
10
15
20
25
30
35
O E S R J G T F H U P I N Q L V A M C D K B
Num
ber o
f obs
erva
tions
Mission
Correlation of Recommendations and Good PracticesR GP
0
5
10
15
20
25
30
35
O E S R J G T F H U P I N Q L V A M C D K B
Num
ber o
f find
ings
Mission
Correlation of Recommendations and SuggestionsR S
IAEA IRRS enhancements 31
E & E of past IRRS missions
Note: The Performance Indicator values presented here characterize the effectiveness and efficiency of the IRRS process and represent no indication whatsoever on the safety performance of the host countries of the missions
0.000
0.020
0.040
0.060
0.080
0.100
0.120
0.140
0.160
0.180
0.200
0.220
ROK GFR-fu SLO SWI CAN-fu SWE SLR FIN BUL UK-fu2 RUS-fu CZR BEL US-fu SLO-fu
Dist
ance
from
opti
mum
Mission
Efficiency and effectiveness of IRRS missions
to analyse
acceptable
effective
optimum
The distance from optimum tends to decrease, i.e. the E&E tends to increase
IAEA IRRS enhancements 32
Ways to improve
• IRRS Training• IRRS Mission Standards• Feedbacks• Measuring effectiveness and
efficiency• Prompt Evaluation Report
IAEA IRRS enhancements 33
Prompt evaluation report
CONTENTS• Basic mission data• Feedback from the team• Feedback from the hosts• Overall evaluation of effectiveness
and efficiency• Discussion, conclusions• Summary
IAEA IRRS enhancements 34
Evaluation of a mission via PI’s
IAEA IRRS enhancements 35
Reports on nuclear missions
IAEA IRRS enhancements 36
Reports on nuclear missions
IAEA IRRS enhancements 37
Reports on nuclear missions
IAEA IRRS enhancements 38
IN SUMMARY
IAEA IRRS enhancements 39
Conclusions
• IRRS is an effective means of reviewing compliance of RB status and activities with IAEA safety standards
• Regular invitation of IRRS missions by all IAEA MS is highly recommended
• The IRRS process is mature and effective – yet it is under continuous development by using novel standardized tools and feedback from participants
• Lessons learned from past missions as well as mission effectiveness and efficiency analysed
• Prompt evaluations of IRRS missions to nuclear countries were performed in a pilot phase
IAEA IRRS enhancements 40
THANK YOU FOR YOUR ATTENTION
Photo: V. Friedrich
This activity is conducted by the IAEA, with funding by the European Union. The views expressed in this presentation do not necessarily reflect the views of the European Commission