John Troxell, Accenture - CDISCportal.cdisc.org/CDISC User Networks/North America/Atlantic... · Historically, SDTM validation checks were specified by a software tool developer and

John Troxell, Accenture

PharmaSUG 2016

Paper DS12

The CDISC Analysis Data Model (ADaM) Team has specified computer checks for ADaM compliance

Software tools implementing the checks produce error messages that conflict with ADaM Implementation Guide (ADaMIG)

Why does this happen, and how can we fix it?

2

Historically, SDTM validation checks were specified by a software tool developer and the FDA

In my opinion, CDISC should define validation checks for CDISC standards compliance

ADaM Validation Checks v1.0 published 2010◦ updated since then

3

1. The problem◦ Incorrect validation tool error messages

2. Understanding the problem◦ The space and vocabulary about it

◦ Six things that have to go right

◦ Why they don’t go right

3. Fixing the problem◦ Ideas

◦ Status

4

Check specifier:◦ The author of the specifications for a computer

compliance check of CDISC compliance

◦ Ideally, CDISC Team that developed the standard, e.g., the CDISC ADaM Team for the CDISC ADaMIG

Programmer or software developer:◦ The developer of a computer validation tool that

implements a computer compliance check

5

6

A computer validation tool is like a teacher

Correct error messages are instructive

Incorrect error messages are destructive:◦ teach the wrong thing

◦ waste time

to review and address

to explain in Analysis Data Reviewer’s Guide

◦ in the worst case, may cause someone to take the wrong “corrective” actions

7

8

“conformance”, “compliance”, and “validation” ◦ Words with many meanings

◦ Experience has shown that it is difficult to discuss the problem because of the vagueness

◦ Need better vocabulary

But before vocabulary, we need an understanding of what it is we are talking about◦ For discussion, an unofficial, suggested map of this

space is presented next

9

10

Regulatory Agency Technical Conformance

Aspects NOT Checkable by a

computer

Data Quality Checks

Accompanying Documentation SDRG, ADRG, ...

Version Acceptability

Tool-Readiness Checks

Data Standards Plan

Other Checks?

Agency-DefinedBusiness Rules

Non-CDISC Rules about CDISC

Agency Technical Conformance Computer Checks

CDISC-DefinedStandards Compliance

CDISC-Published Computer Checks

Defined in agency Technical Conformance Guide

Encompasses all items in the map

11

12



computer

Data Quality Checks




Data Standards Plan

Other Checks?






Compliance with CDISC standards as defined by CDISC in published CDISC standards documents

Some aspects are checkable by computer

Some aspects ARE NOT checkable by computer◦ maybe someday, with perfect metadata?

13

14



computer

Data Quality Checks




Data Standards Plan

Other Checks?






Computer checks of regulatory agency technical conformance defined by the agency

Does not include CDISC-published computer checks

15

16



computer

Data Quality Checks




Data Standards Plan

Other Checks?






Data Quality Checks◦ Checks having to do with the quality of collected

data

◦ Not about CDISC standards

Tool-Readiness Checks◦ Checks having to do with whether the data support

review tools

17

Non-CDISC Rules about CDISC◦ Checks about compliance to CDISC standards

invented by others than CDISC and that are used by the agency

◦ Example: The traditional SDTM checks, that were defined by a tool vendor and FDA

◦ May or may not be consistent with the CDISC standards

18

19



computer

Data Quality Checks




Data Standards Plan

Other Checks?






In regard to what a regulatory agency needs, “Conformance” seems like a good word◦ Technical Conformance Guide

In regard to adherence to a CDISC standard◦ “Conformance” is confusing because it is already

used in the broader context of agency needs

◦ “Compliance” seems like a good word, but it is a loaded word in some regulatory contexts

◦ “Adherence”?

◦ For now, I prefer “Compliance”

20

Some aspects of ADaM (or SDTM) compliance are checkable by computer

Some are NOT checkable by computer

If there are no error messages, it does not prove that the datasets are valid

“validation checks” could really be called “invalidation checks”◦ they can demonstrate non-compliance

◦ they cannot prove compliance

21

There is no consensus yet on better vocabulary for this space

We may never get agreement on one-word terms that eliminate confusion

Until we do, it is best to use as many adjectives as necessary so that both the speaker and the listener understand

22

23

A chain of six steps that all have to go right for correct validation tool error messages about ADaM compliance

Causes of breakdowns at each of the steps and discussion

Improvement ideas, activities, and status

24

There is a chain of steps that have to go right in order to prevent incorrect validation tool error messages about CDISC compliance

Failure at any step may cause an incorrect error message

25

1. CDISC IG is clear and consistent

2. Computer check specification does not conflict with IG

3. Computer check specification is clear and complete enough for a pure programmer

4. Computer check specification is understood by programmer

5. Check is implemented correctly by programmer

6. Content of error message is correct

26

The following slides identify some of the causes of breakdowns at each of the steps in the chain

Note: ADaM may be more challenging from a computer checks perspective than SDTM

But the chain and the causes apply in principle to computer checks for any CDISC data standard

27







28

We try hard to be clear and consistent but are not always 100% successful◦ it is a constant challenge and hard work

A major objective of ADaMIG v1.1 was to clarify v1.0◦ E.g., added scope to many statements (within a

study? dataset? parameter? ...?)

Clarification can mean:◦ expressing something clearly

◦ debate and resolution within the ADaM Team about the concept itself that is being clarified

29







30

Misunderstanding of the IG

Not reading other pertinent sections of the IG

Overly-aggressive hunts for rules, e.g.,◦ Misinterpreting an example as a general rule

◦ Misinterpreting statements made in the context of a particular variable to apply to all variables with certain variable name fragments

Likelier when computer check specifier is not the CDISC Team that developed the standard

31







32

Pure programmers were not the main audience implicitly assumed by authors of IG

Necessary information from other areas in IG not explicit in a particular check specification◦ For example, in ADaM, even for required variables,

nulls are allowed, unless otherwise specified

Computer check specification may ignore practical data scenarios, such as missing values

No examples provided of data passing and failing check

33







34

Likelier when the check is specified incompletely and when the programmer does not have an assumed background

Programmer does not read the whole IG◦ E.g., information about permissibility of missing

values might be found somewhere else in the documents than in a compliance check specification, or in the section of the document to which the check specification points as its source

35







36

Bugs

Programmers naturally want to generalize and extend similar checks – may not be appropriate

37







38

Content of the message not specified by the check specifier

Specified content not provided in message

Message does not identify the check specifier

Opinion: some room for value-added additional explanation on the part of the software developer, but only if clearly identified as not originating from the specifier of the check

39

Some ideas for improvement contributed by many sources

CDISC Winter Intrachange 2016 and elsewhere

Not all of them are agreed upon

Some are currently underway

There is consensus that we need to improve

40

Clarify CDISC Standards; Ongoing

Define CDISC compliance computer checks; ADaM published; SDTM drafted

Do not be overly aggressive in extracting rules from standards

Specify compliance checks completely enough for a pure, CDISC-naïve programmer

Specify a standard error message

41

ADaM Leadership Team review ADaM checks for consistency with IG; Started

Improve ADaM Team process for tracking and managing issues and suggestions about checks; Started

Respond expeditiously and reliably to issues and suggestions about ADaM checks provided by software developers; Agreed

Consider NOTE/WARNING/ERROR structure instead of only ERROR for ADaM checks

42

Consider adding a Define-XML dataset metadata SUBCLASS attribute

Consider creating test data for computer checks (both pass and fail examples)

Consider a common format across standards for CDISC-specified computer checks

43

Maybe someday every computer check specified by CDISC will be complete & clear

Until then◦ Learn the standard, not just the check

specifications

◦ Read the whole IG, because important information is scattered

Submit ideas about computer checks of CDISC compliance to CDISC Team; Started

Do not release invented checks about ADaM compliance unilaterally; Started

44

Identify the specifier of the computer check that resulted in the error message ◦ E.g., CDISC check specification document name,

agency name, software company

When the check specification defines a standard error message, clearly differentiate the standard error message from additional developer explanation or advice

45

Clarify which agency business rule checks are about CDISC compliance, and which are not

Submit ideas about computer checks of CDISC compliance to CDISC Team

Do not release invented checks about ADaM compliance unilaterally

When requesting that software developers implement checks about CDISC compliance, involve the relevant CDISC team◦ This may prevent agency-originated checks that

conflict with the CDISC standards

46

Refer to specific software and version in issue reports and discussions◦ Some issues are fixed and then become obsolete

Do not rely exclusively on software tools◦ Passing the computer checks does not prove

compliance

◦ Manual review is essential

If in any doubt, seek expert advice

47

Understand the space

Use clear descriptions and enough adjectives to define what you are talking about

We will all benefit by better communication

48

John Troxell

Senior ADaM Consultant

Accenture Accelerated R&D Services

+1 908-612-1991

[email protected]

49

John Troxell, Accenture - CDISCportal.cdisc.org/CDISC User Networks/North America/Atlantic... · Historically, SDTM validation checks were specified by a software tool developer and

Documents