V&V Assessment for CASL/VERA - Department of … Introduction • Scope of Assessment – Challenge Problems – Capability vs Credibility – PCMM and Evidence • Challenge Problem

V&V Assessment for CASL/VERA

Milestone Number Key Staff CP Support

L2:VVI.P15.01 Jones, Hetzler,Seiger, Dinh, Athe All

Codes Involved Nature StatusMPACT, CTF,BISON, Star

CCM+, TiamatAssessment Complete, w’

Follow-on

2

Introduction

• Scope of Assessment– Challenge Problems– Capability vs Credibility– PCMM and Evidence

• Challenge Problem Assessments (3)– Capability Gaps– PCMM Scores

• Conclusion• Future Work

3

CASL Codes and Challenge Problems

• Present assessment focuses on:

• CIPS• PCI• DNB

• Challenge Problem scope helps to define requirements

• Assess only the capability related to CPs

4

Capability vs Credibility

• Capability refers to the presence of models (physics, numerics, etc.) used to solve the problem– Assessed using Phenomena Identification and Ranking Tables

developed with expert elicitation• Credibility is loosely defined as how trustworthy/mature

the predictions are – We assess prediction credibility with the Prediction Capability

Maturity Model1 (SNL VV/UQ and Credibility Processes Department does this primarily for ASC)

– Capability fidelity is included as part of credibility

1 Oberkampf, Pilch, Trucano, Predictive Capability Maturity Model for Computational Modeling and Simulation, SAND2007-5948 Albuquerque, NM, 2007.

5

Challenge Problem PIRT / Capability Assessment • For each CP, and abbreviated

PIRT was conducted to define requirements

• Future assessments should include sensitivity studies to quantify phenomenological importance

• VERA Capability and gaps are qualitative opinions of the authors

Physics PhenomenaImportance

for CIPSVERA

capability Gap Gap Description

Sub channel thermal hydraulics

Steaming Rate 3.0 3.0Subcooled Boiling on a clean metal surface 3.0 3.0

Subcooled Boiling In CRUD 3.0 1.0 2.0 Lack of SET data under reactor prototypic CRUD

Bulk Coolant Temperature 3.0 3.0Heat Flux 3.0 3.0

Wall Roughness 2.0 1.0 1.0 Lack of SET data under reactor prototypic CRUD

Lack of SET Data was the most common capability gap

6

PCMM and Evidence

• Six Elements of PCMM– Code Verification (Includes SQA/SQE)– Geometry Fidelity– Physical and Material Model Fidelity– Solution Verification– Validation (SET/IET)– Uncertainty Quantification

• Mutually Exclusive & Collectively Exhaustive• Each element is assessed using evidence (documentation)

– We recognize that there may be:• relevant work that may not be documented, or • may have been overlooked during this assessment

– We need help from the other Focus Areas and Code Teams ensuring we know what’s been done

7

PCMM Assessment for CP

CIPSPCMM attribute MPACT CTF MAMBA TIAMAT

Representation and Geometric Fidelity 3 2 1 N/A

Physics and Material Model Fidelity 3 2 1 N/A

Software Quality Assurance 2 2 0 1

Code Verification 1 1 0 1Solution Verification 1 1 1 N/A

Separate Effects Validation 2 1 0 N/A

Integral Effects Validation 2 2 1 N/A

Uncertainty Quantification 0 0 0 N/A

• Varied maturity between codes

• Overall CP predictive maturity heavily influenced by “weakest links”

• Validation is generally well developed across all codes

• Verification and UQ are weak

• UQ method and tool investments have been made, but little application

NOTE: The PCMM scores are “quantitative” yet are subjective. Furthermore, focusing on achieving a specific score or using PCMM scores for goal setting can be problematic. PCMM provides a systematic framework for discussing and assessing the multiple attributes that contribute to maturity

9

CIPS Capability Gaps

Physics PhenomenaImportance

for CIPSVERA

capability Gap

Sub channel thermal

hydraulics

Subcooled Boiling In CRUD 3.0 1.0 2.0Wall Roughness 2.0 1.0 1.0Mass Balance of Nickel and Iron 3.0 1.0 2.0CRUD Erosion 2.2 1.0 1.2Initial CRUD Thickness (Mass) 2.5 1.0 1.5Initial Coolant Nickel and Boron Concentration 2.7 1.0 1.7CRUD Source Term from SGs and other Surfaces 3.0 0.0 3.0CRUD Induced Change in Boiling Efficiency: 2.7 1.0 1.7Heat Flux Distribution (new phenomenon) 3.0 2.0 1.0Changes in Eff, CRUD Cond. due to Int. Fluid Flow & Boiling 2.0 1.0 1.0

Coolant chemistry

Local changes (near the rod) in the equation of state 2.4 1.0 1.4Chemical reaction rates are based on lower T and P 2.0 1.0 1.0CRUD Porosity 2.8 1.0 1.8CRUD Permeability 2.0 1.0 1.0CRUD Chimney Density 2.6 1.0 1.6

10

CIPS PCMM Assessment

CRUD Induced Power ShiftPCMM attribute MPACT CTF MAMBA TIAMAT

Representation and Geometric Fidelity 3 2 1 N/A

Physics and Material Model Fidelity 3 2 1 N/A

Software Quality Assurance 2 2 0 1Code Verification 1 1 0 1

Solution Verification 1 1 1 N/ASeparate Effects Validation 2 1 0 N/A

Integral Effects Validation 2 2 1 N/AUncertainty Quantification 0 0 0 N/A

11

PCI Capability GapsPhysics Phenomena Importance

for PCIVERA

capability Gap

Fuel Modeling

Cladding Creep 2.8 1.0 1.8

Pellet Cracking 2.8 1.0 1.8

Pellet Swelling 2.8 1.0 1.8

Pellet Densification 2.4 1.0 1.4

Gap Model 2.5 2.0 0.5

Thermal Creep In the Pellet and Clad 2.6 1.0 1.6

Friction Between Pellet and Clad 2.6 1.0 1.6Microstructure Impacts on Stress Driven Cracking 2.4 1.0 1.4

Material Properties for Time Varying Heterogeneous Fuel Pellet 2.4 1.0 1.4

12

PCI PCMM Assessment

Pellet Clad InteractionPCMM attribute MPACT CTF BISON TIAMAT

Representation and Geometric Fidelity 3 2 2 N/A

Physics and Material Model Fidelity 3 2 2 N/A

Software Quality Assurance 2 2 2 1Code Verification 1 1 1 1

Solution Verification 1 1 1 N/ASeparate Effects Validation 2 1 1 N/A

Integral Effects Validation 2 2 2 N/AUncertainty Quantification 0 0 0 N/A

13

DNB Capability GapsPhysics Phenomena Importance

for DNB CPVERA

Capability Gap

Fuel ModellingCladding surface heat transfer 2.5 2 0.5Fuel rod growth or densification 2 1 1Fuel rod bowing 2 1 1

Sub channel thermal hydraulics

Turbulent mixing single phase flow two-phase flow

32

21

11

Cross flow 3 2 1Nucleate boiling 3 2 1Two-phase flow 3 1 2Critical Heat Flux 3 2 1Flow regime 3 2 1

CFD (CMFD)

Bubble break-up and coalescence 2.5 2 0.5Nucleation site density 3 2 1Nucleation site interaction 3 2 1Wall Heat Transfer 3 2 1Surface effects 3 1 2Microlayer dynamics 3 1 2Spacer grid, MV effect 3 2 2Bubble departure frequency 3 2 1Average dry area 3 1 2

14

DNB PCMM Assessment

Departure From Nucleate BoilingPCMM attribute MPACT CTF Star CCM+

Representation and Geometric Fidelity 3 2 3

Physics and Material Model Fidelity 3 2 2

Software Quality Assurance 2 2 3Code Verification 1 1 2

Solution Verification 1 1 1Separate Effects Validation 2 1 1

Integral Effects Validation 2 2 1Uncertainty Quantification 0 0 0`

15

Conclusions

• Initial Credibility Assessment Completed• Question: how to respond to the initial assessment

– Low hanging fruit in multiple areas, heavier lifts in other areas– We propose value driven prioritization of FY18 Activities (e.g. PICK)

• V&V, SQA Scope for Code Teams• UQ/Sensitivity Scope for VVI/AMA

16

Future Work

• Initial CASL V&V Assessment conducted in 2017– Capability and credibility gaps assessed and documented– Commitment to periodic assessment to track maturity change

• Future funding decisions can be driven by gaps– Value driven assessment of investment based on required

resources and predicted impact• Improved documentation of V&V activities will naturally

increase credibility• Clear shortcomings in UQ and verification for all codes

and CPs• Improvements in the assessment process will be

implemented– QPIRT, Formalization of assessment feedback

17

www.casl.gov

18

Backup Slides

19

CIPS (1/2)

20

CIPS (2/2)

21

PCI

22

DNB (1/2)

23

DNB (2/2)

24

PICK Chart

Pay-Off

MPACT-P1

TIAMAT-P1

MAMBA-P1

MPACT-P3BISON-P1

MAMBA-P3

MAMBA-P2

TIAMAT-P2CTF-P1

VERA-P6

MPACT-P2

VERA-P1

Impl

emen

tatio

n

ImplementPossible

Keep for Later

Challenge

VERA-P4

VERA-P2