Modeling Quality Assurance Project Plans (QAPPs) Sandra Arismendez TCEQ Laboratory and Quality Assurance Section July 8, 2015
Modeling Quality Assurance Project Plans (QAPPs)
Sandra Arismendez
TCEQ Laboratory and Quality Assurance Section
July 8, 2015
QAPPs: Why Write Them?
• Because they are required – EPA Order CIO 2105.0 (formerly 5360.1 A2) sets forth
requirement for all organizations conducting environmental data operations on EPA’s behalf
– State agencies (including TCEQ and TSSWCB) and
TCEQ contractors follow suit
• So that projects can be planned and implemented effectively
Importance of Modeling QAPPs
• Helps ensure that the model(s) (and underlying data) are sufficient for achieving project objectives
• Provides documentation of specifications and activities
QAPP Requirements
• Graded Approach – Level of detail commensurate with nature of the work and intended use of data
Tools for QAPP Development
• EPA Requirements for Quality Assurance Project Plans (EPA QA/R-5) is governing requirement (http://www.epa.gov/quality/qs-docs/r5-final.pdf)
• EPA Guidance for Modeling QAPPs (EPA QA/G-5M) (http://www.epa.gov/QUALITY/qa_docs.html)
• QAPP Program Shells
• Other QAPPs
QAPP Content
• Section A – Project Management
• Section B – Measurement and Data Acquisition
• Section C – Assessment and Oversight
• Section D – Data Validation and Usability
Section A – Project Management
• A1, A2, A3 – Title Page, Signature Page, TOC, Distribution List
• A4 – Project/Task Organization • A5 – Problem Definition/Background • A6 – Project/Task Description and Schedule • A7 – Quality Objectives and Criteria • A8 – Special Training/Certification • A9 – Documents and Records
A5 – Problem Definition/Background
• Define problem, place modeling activities into overall context with the project, and explain why modeling is being conducted
• Provide justification for model(s) to be used • Summarize what questions will be answered
and/or what decisions will be made
A6 – Project/Task Description and Schedule
• Summarize work to be done
• Detail project tasks, schedule, and milestones
– Development, verification, and validation of code or software is a task (if applicable)
• Provide enough detail so it is clear as to what versions and
implementations of packaged models will be used • If multiple models are being used, any linkages should be
discussed (preferably diagrammed)
A7 – Quality Objectives and Criteria for Model Inputs/Outputs
• Inputs: Quantity, quality, and data type are relevant
– Quantity and quality: Precision, Bias, Accuracy, Sensitivity, Representativeness, Comparability, Completeness
– Type: data collection technique • Outputs: Qualitative and quantitative criteria and
measures – Quantitative: “Goodness of fit,” error and other forms of
uncertainty, sensitivity, graphical analysis – Qualitative: Reasonableness (what is reasonable should be
defined)
Sensitivity and Uncertainty Analyses
• Sensitivity analysis evaluates the effect of changes in input values or assumptions on a model’s results
• Uncertainty analysis investigates the effects of a lack
of knowledge or other potential sources of error in the model
• When used in combination, sensitivity and uncertainty analysis allow model users to be more informed about the confidence that can be placed in model results
* Text above copied directly from Guidance on the Development, Evaluation, and Application of Environmental Models, Council for Regulatory Environmental Modeling (EPA/100/K-09), USEPA, 2009.
A9 – Documents and Records
• Specify records, their contents, storage location, retention period, formats, and back-up protocols
• Examples: QAPPs, User’s Manual(s), various
reports, raw files, code, calibration records, verification/validation records
Section B – Measurement and Data Acquisition
• B7 – Calibration
• B9 – Non-direct Measurements (Data Acquisition Requirements)
• B10 – Data Management and Hardware/Software Configuration
B7 – Calibration • Identify calibration parameters (including literature
values)
• If calibration process is iterative and/or stepwise, spell out calibration steps
• Specify criteria/guidelines used to determine whether calibration is successful – Also, state what will be done if minimum requirements are
not met (e.g., recalibration with additional data, consideration of different modeling approach, etc.)
Section B9 – Non-Direct Measurements (Secondary Data)
• This section is critical as primary data collection is rarely described in modeling QAPPs
• Tables can be extremely helpful • Don’t forget the graded approach!
• If some data will be acquired that are not yet known,
this may be built into the QAPP – In this case, it is especially important to specify criteria for
acceptance and use of those data
B10 – Data Management • Details about data storage, production,
management, storage, and archival should be provided – If SOPs already exist, these may be referenced instead – If data coding or software development is taking place,
some reference to these procedures should be provided
• Hardware and software requirements/specifications should be included
• Flow diagrams can be very instructive
Section C – Assessment and Oversight
• C1– Assessments and Response Actions
• C2 – Reports to Management
C1 – Assessments and Response Actions
• Describe surveillance activities, audits, and the process for identifying, documenting, and correcting problems
• Also, describe assessments of the model performance, and the schedule for these activities
Section D – Data Validation and Usability
• D1 – Departures from Validation Criteria
• D2 –Validation Methods
• D3 – Reconciliation with User Requirements
Section D
• D1: Describe how results are evaluated and departures from procedures presented
• D2: Describe steps in the validation process
– Performance criteria from A7 should be referenced
• D3: Describe uses of data and how limitations
will be reported
Questions?