WG5 P02 Proposal 2014 Qualification of Standard Scripts

WG5 P02 Proposal 2014Qualification of Standard Scripts

Proposal through CSS 2014http://www.phusewiki.org/wiki/index.php?title=File:FDA_Scrips.ppt

• Anyone should be able to submit a script, according to a check list

• Categorize scripts according to complexity– Complexity:

low, medium, high, software– Output:

tabulated data, analysis data, table, figure, listing

• Metadata for script should indicate– Type of output:

tabulated data, analysis data, table, figure, listing– Study design:

parallel, crossover, etc– State of qualification:

?

Proposal through CSS 2104

• Test data– Overall project should have minimum test data (SDTM &

ADaM)– Scripts can propose new test data, must pass (Data fit?

Open CDISC?)– Share program to produce test data, never binary test data

• 2 levels of qualification to match script complexity/output– Light vs. Heavy qualification– Common elements include

• header• good programming practices• clearly declared scope of script (e.g., study design(s))• test data matches scope & passes "FDA Data Fit"

assessment (?)• documentation inputs/outputs/dependencies/usage

Proposal through CSS 2104

• Heavy qualification– Beta package includes

minimal elements for contribution• Specification & Documentation (could be in pgm

header)• Test data (Data Fit? or Open CDISC or other, as

appropriate)• Tests & Expected results defined• Peer Review: GPP, Specs & Docn reviewed, Tests

reproduced– Draft

• Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?)

– Test• Peer Review: Write qualification report, incl. log/output

from tests– Final

Proposal through CSS 2104

• Light qualification– Beta package includes

skip if >1 yr production use without ERROR– Draft

minimal elements for contribution• Specification & Documentation (could be in pgm

header)• Test data (Data Fit? or Open CDISC or other, as

appropriate)• Tests & Expected results defined• Peer Review: GPP, Specs & Docn reviewed, Tests

reproduced• Write qualification plan, Review tests for

completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?)

– Test• Peer Review: Write qualification report, incl. log/output

from tests– Final

Proposal through CSS 2104

Peer Review Checklist Heavy

Light

Requirement specification X ?

Documented or perhaps only documented in header X

User Guide X X

SDTM/ADaM used in input/output X X

Open CDISC validator or Data Fit used to check input/output

X X

GPP in source X X

Run according to Requirement specification X ?

Tested by qualification plan, tests & results all Peer reviewed

X ?

Tested by End users X ?

Robust without red errors in contributor's production environment

X X

Robust and used in FDA (other) scripts repository, ranked ******

X

Proposal meaningful terms in blueQualificationhttp://www.phusewiki.org/wiki/index.php?title=File:WG5_P02_Proposal_-_2014.pptx

• Qualification Instructions (see embedded template ð) – Certification phase of Qualification applies to new scripts

and tests– Confirmation phase applies to updates of existing scripts

• States:

Contributed, Development, Testing, Qualified

• Roles– Contributor: Anyone with appropriate skills & interests– Developer: CSS Working Group 5 volunteer familiar with

objectives**– Tester: CSS WG 05 volunteer familiar with objectives**– Environment Tester: Anyone in industry community able to

set up automatic test replication in their work environment– Reviewer: Author of white papers, designers of script

targets

** suggests a quick-start onboarding page in CSS Phusewiki

WG05-P02 Qualification plan (template).do

ProposalQualification

• Metadata for script should indicate– Whitepaper & output ID– Programming language & version (e.g., R 3.1.1)– Type of output:

tabulated data, analysis data, table, figure, listing– Study design:

parallel, crossover, etc– State of qualification:

Contributed, Development, Testing, Qualified

• Test Data requirements– available as a program or script (text, not binary)– based on expected standards (SDTM, ADaM)– data requirements clearly & accurately specified for each

script– include expected results from these data for defined

tests/checks

ProposalQualification

• End-user Objectives– Clear overview of purpose and resources– Inspire confidence from first sight– Ease of use, clear messaging from first run– Reproducible results– Consistency of scripts, learning first one makes remaining

familiar– Ease of converting users to contributors

• Contributor Objectives– Standardize routine steps– Modularize routine components– Automate testing, issue identification– Centralize & consolidate information & results

ProposalQualification

• Transitions

Contributed is the original State of all scripts– to Development, checklist includes

by Developer & Reviewer• R & D confer on suitability of contribution. Suitable

starting point?[ may require analysis details, specs, from contributor ]

• D reviews components• D works with Contributor to complete minimum

components[ including Test Data and Coverage of defined tests ]

• D adds standard parameter, dependency checking• D writes Qualification instructions .docx (see template,

above)– to Testing, checklist includes

by Tester• Review Qualification instructions, consider coverage of

tests• Execute Qualification instructions• Work with Developer to complete execution

successfully

ProposalQualification

• Transitions

continued– to Qualified

by Tester & Environment Tester & Reviewer• T updates posted test outputs from

certification/confirmation• E updates local tests and executes (posting PASS/FAIL

results)• R confirms script output matches intention &

qualification process covers important elements and considerations. Also provides user (rather than technical) feedback?

• Achieve "Qualified" state when all tests in all test environments PASS (i.e., match outputs that T has certified and/or confirmed) and that R agrees that target is achieved

ProposalQualification

• Efforts Required– Finalize Qualification states, roles, workflow, checklists, and

templates – Design test structure in google code– Develop scripts that will allow Environment Testing– Develop general components (e.g. parameter, dependency

checking)– Identify Environment Testers based on

• Host environment• SAS or R version

– Identify opportunities to automate qualification. E.g.,• Docx format for Qualification instructions is not easily

machine readable• Environment Testers to post results back as machine

readable• Script green-light/red-light qualification matrix on Phusewiki

Doing now what patients need next