The most common factors leading to wrongful conviction · error, the threat of bias, or the absence of sound ... ‘The obvious risk in a criminal trial when expert evidence is led

Professor Gary EdmondDirector, Program in Expertise, Evidence & LawChair, Evidence-based forensicsinitiative School of Law

Challenging scientific evidence

Overview1.Background – how we’re travelling2.A ‘little’ science 3.Admissibility (s79) and the form of the evidence4.Mandatory and exclusionary discretions (after XY)5.Prosecutorial obligations6.Codes of conduct for expert witnesses7.Other trial safeguards: Cross-examination, rebuttal expert, directions and warnings8.System issues and principle9.Strategic action10.Legal versus non-legal approaches to expertise

Kaye Ballantyne, Senior Research & Development Officer Victoria PoliceEmma Cunliffe Associate Professor, Allard School of Law, UBCGary Edmond, Professor & ARC Future Fellow, Faculty of Law UNSWBryan Found, Professor & Chief Forensic Scientist, Victoria PoliceDavid Hamer, Associate Professor, Law School, University of SydneyBrynn Hibbert, Emeritus Professor, School of Chemistry, UNSWRichard Kemp, Associate Professor, School of Psychology, UNSW

Andrew Ligertwood, Emeritus Fellow, Adelaide Law SchoolKristy Martire, ARC DECRA Fellow, Science, UNSWGianni Ribeiro, PhD Candidate, Science, UQAndrew Robert, Senior Lecturer, Law School, University of MelbourneMehera San Roque, Senior Lecturer Faculty of Law, UNSWRachel Searston, PhD Candidate, Science, UQJason Tangen, Senior Lecturer School of Psychology, UQMatthew Thompson, Post-Doctoral Fellow, Science, UQDavid White, Post-Doctoral Fellow, Science, UNSWRachel Dioso-Villa, Lecturer, School of Criminology & Criminal Justice, Griffith University

Evidence-Based

Forensics Initiative

1. Background – how we’re travelling

Trial safeguards and legal awareness?‘The report finds that the existing legal regime—including the rules governing the admissibility of forensic evidence, the applicable standards governing appellate review of trial court decisions, the limitations of the adversary process, and judges and lawyers who often lack the scientific expertise necessary to comprehend and evaluate forensic evidence—is inadequate to the task of curing the documented ills of the forensic science disciplines. This matters a great deal, because “forensic science is but the handmaiden of the legal system.” … there are serious issues regarding the capacity and quality of the current forensic science system; yet, the courts continue to rely on forensic evidence without fully understanding and addressing the limitations of different forensic science disciplines.

National Academy of Sciences, Forensic Science in the United States (2009) 85, 12, 53, 96, 109, 110.

2. A ‘little’ science

What do independent scientists, engineers and biomedical researchers have to say about latent fingerprint, ballistic, tool mark, bite mark, hair, document, voice and image comparisons and so on?

‘What lawyers should know about the forensic “sciences”’ (2015) 36 Adelaide Law Review 33.

Authoritative scientific reports and reviews

NAS, Strengthening forensic science in the United States: A path forward (2009)

• Background: fingerprint errors and DNA exonerations (US Innocence Projects).

• National Academy of Sciences (NAS) –National Research Council.

• Congressionally funded.• Multidisciplinary committee:

statisticians, a senior federal judge (Edwards), a chemist, senior forensic scientists, an engineer, biologists, computer scientists, a medical examiner and law professors.

• Two year inquiry (2007-2009): submissions and hearings.

• 13 Recommendations.

Using forensic science: Guiding principles

‘Two very important questions should underlie the law’s admission of and reliance upon forensic evidence in criminal trials: (1) the extent to which a particular forensic discipline is founded on a reliable scientific methodology that gives it the capacity to accurately analyze evidence and report findings and (2) the extent to which practitioners in a particular forensic discipline rely on human interpretation that could be tainted by error, the threat of bias, or the absence of sound operational procedures and robust performance standards. These questions are significant.’National Academy of Sciences, Strengthening Forensic Science in the United States (2009) 9.

Validation of techniques

‘The simple reality is that the interpretation of forensic evidence is not always based on scientific studies to determine its validity. This is a serious problem. … there is a notable dearth of peer-reviewed, published studies establishing the scientific bases and validity of many forensic methods.’

National Academy of Sciences, Strengthening Forensic Science in the United States (2009) 8.

Absence of validation – implications

‘With the exception of nuclear DNA analysis … no forensic method has been rigorously shown to have the capacity to consistently, and with a high degree of certainty, demonstrate a connection between evidence and a specific individual or source.’

(i.e. latent fingerprints*, shoe, foot, ear, tyre, bite mark, hair, fibre, soil, ballistics, handwriting, voice and image comparisons and so on).National Academy of Sciences, Strengthening Forensic Science in the United States (2009) 7.

Accuracy, uncertainty and errors

‘Few forensic science methods have developed adequate measures of the accuracy of inferences made by forensic scientists. All results for every forensic science method should indicate the uncertainty in the measurements that are made, and studies must be conducted that enable the estimation of those values.’

National Academy of Sciences, Strengthening Forensic Science in the United States (2009).

Terminology and expressions

‘many terms are used by forensic examiners in reports and in court testimony … Such terms include … “match,” “consistent with,” “identical,” “similar in all respects tested,” and “cannot be excluded as the source of.” … the forensic science disciplines have not reached agreement or consensus on the precise meaning of any of these terms. This imprecision in vocabulary stems in part from the paucity of research’.

National Academy of Sciences, Strengthening Forensic Science in the United States (2009).

Human factors‘Some initial and striking research has uncovered the effects of some biases in forensic science procedures ... The forensic science disciplines are just beginning to become aware of contextual bias and the dangers it poses. The traps created by such biases can be very subtle, and typically one is not aware that his or her judgment is being affected.’

National Academy of Sciences, Strengthening Forensic Science in the United States (2009).See also Expert Working Group, Latent Print Examination and Human Factors: Improving the Practice through a Systems Approach (US National Institute of Standards and Technology & National Institute of Justice, 2012).

Standards

‘Often there are no standard protocols governing forensic practice in a given discipline. And, even when protocols are in place, they often are vague and not enforced in any meaningful way. … These shortcomings obviously pose a continuing and serious threat to the quality and credibility of forensic science practice.’

National Academy of Sciences, Strengthening Forensic Science in the United States (2009)

3. Admissibility (s79) and the form of the evidence

‘The admissibility of forensic science and medicine evidence under the Uniform Evidence Law’ (2014) 38 Criminal Law Journal 136.

Admissibility under the Evidence Act 1995 (NSW)

76 Opinion Rule: opinion evidence is presumptively inadmissible

79 Exception: opinions based on specialised knowledge(1) If a person has specialised knowledge based on the person’s training, study or experience, the opinion rule does not apply to evidence of an opinion of that person that is wholly or substantially based on that knowledge.

Dasreef v Hawchar [2011] HCA 21

‘A failure to demonstrate that an opinion expressed by a witness is based on the witness’s specialised knowledge based on training, study or experience is a matter that goes to the admissibility of the evidence, not its weight.’

High Court s79 jurisprudence: two ‘criteria’

‘To be admissible under s 79(1) the evidence that is tendered must satisfy two criteria. The first is that the witness who gives the evidence “has specialised knowledge based on the person’s training, study or experience”; the second is that the opinion expressed in evidence by the witness “is wholly or substantially based on that knowledge”.’

Dasreef Pty Ltd v Hawchar (2011) 243 CLR 588 [32]. See also Honeysett v The Queen [2014] HCA 29, [23].

Section 79(1) requires the following:

1. specialised knowledge has to be based on training, study or experience

2. opinion has to be wholly or substantially based on specialised knowledge

Opinion specialised knowledge training study or experience

NOT opinion based on experience

‘Specialised knowledge’ is not ‘experience’

R v Tang [2006] NSWCCA 167 at [137]: ‘The focus must be on the words “specialised knowledge”, not on the introduction

of an extraneous idea such as “reliability”.’

‘one and the same’ and

‘unique identifiers’

Honeysett v The Queen [2014] HCA 29numerous similarities and no differences

Honeysett v The Queen [2014] HCA 29, [23]

‘“Specialised knowledge” is to be distinguished from matters of "common knowledge”. … It may be of matters that are not of a scientific or technical kind and a person without any formal qualifications may acquire specialised knowledge by experience. However, the person’s training, study or experience must result in the acquisition of knowledge. The Macquarie Dictionary defines “knowledge” as “acquaintance with facts, truths, or principles, as from study or investigation” (emphasis added) and it is in this sense that it is used in s 79(1). The concept is captured in Blackmun J’s formulation in Daubert: “the word ‘knowledge’ connotes more than subjective belief or unsupported speculation. … [It] applies to any body of known facts or to any body of ideas inferred from such facts or accepted as truths on good grounds”.’

4. Mandatory and exclusionary discretions (after Shamouil and XY)

‘Christie, section 137 and forensic science evidence after Dupas v The Queen and R v XY’ (2014) 40 Monash Law Review 389.

Mandatory exclusion (UEL)

137 Exclusion of prejudicial evidence in criminal proceedings In a criminal proceeding, the court must refuse to admit evidence adduced by the prosecutor if its probative value is outweighed by the danger of unfair prejudice to the defendant.

See also section 135

(Christie discretion at common law in SA)

Tuite v The Queen [2015] VSCA 148 (Victoria)A challenge to STRmix – used for mixed DNA profiles – via UEL s137.

Maxwell P, Redlich and Weinberg JJA:‘The obvious risk in a criminal trial when expert evidence is led from a forensic scientist is that a jury will give the evidence more weight than it deserves. To prevent unfair prejudice of that kind, it is essential that the reliability of expert evidence be established to the court’s satisfaction (under s 137) before it is led. We have concluded that the touchstone of reliability for this purpose is proof of appropriate validation, both of the underlying science (where necessary) and of the particular methodology being employed.’

danger of unfair prejudice to the defendant (NSW)The danger of unfair prejudice is the risk that the fact-finder may use the evidence on an improper basis. It also includes “the danger that on hearing the evidence the fact-finder would be satisfied with a lower degree of probability than would otherwise be required” (ALRC, 1985 at [644]; ALRC, 2006 at 16.23-16.26). The risk arises when:•The fact-finder is not placed in a position to rationally evaluate the forensic science.•Inappropriately deferring to the forensic practitioner.•Misunderstanding or over-valuing forensic science evidence.•Failing to appreciate the significance of formal evaluation or its absence. •Relying on general acceptance, longstanding use and previous admission. •Relying on, or substituting, experience, confidence and demeanour. •Failing to appreciate the corrosive potential of contextual information. •Procedural disadvantage to the defendant.

5. Prosecutorial obligations

‘(Ad)Ministering justice: Expert evidence and the professional responsibilities of prosecutors’ (2013) 36 UNSW Law Journal 921

Prosecutors have serious obligations (that apply to expert evidence)

‘It cannot be over-emphasised that the purpose of a criminal prosecution is not to obtain a conviction, it is to lay before a jury what the Crown considers to be credible evidence relevant to what is alleged to be a crime. Counsel have a duty to see that all available legal proof of the facts is presented: it should be done firmly and pressed to its legitimate strength but it must also be done fairly. The role of prosecutor excludes any notion of winning or losing …’

Boucher v The Queen (1954) 110 CCC 263, 270.

‘A prosecutor is a ‘minister of justice’. The prosecutor’s principal role is to assist the court to arrive at the truth and to do justice between the community and the accused according to law and the dictates of fairness.’

Office of the Director of Public Prosecutions (NSW), ‘Prosecution Guidelines of the Office of the Director of Public Prosecutions for New South Wales’ (1 June 2007) Guideline 2.

The New South Wales Barristers’ Rules include:

Rule 82. A prosecutor must fairly assist the court to arrive at the truth, must seek impartially to have the whole of the relevant evidence placed intelligibly before the court, and must seek to assist the court with adequate submissions of law to enable the law properly to be applied to the facts. Rule 83. A prosecutor must not press the prosecution's case for a conviction beyond a full and firm presentation of that case. Rule 84. A prosecutor must not, by language or other conduct, seek to inflame or bias the court against the accused.Rule 85. A prosecutor must not argue any proposition of fact or law which the prosecutor does not believe on reasonable grounds to be capable of contributing to a finding of guilt and also to carry weight.

6. Codes of conduct for expert witnesses

‘Model forensic science’ (2016) 48 Australian Journal of Forensic Sciences (forthcoming).

Practice Note: Expert Evidence in Criminal Trials (Victoria)

Purpose To enhance the quality and reliability of expert evidence relied on by the prosecution and the accused in criminal trials …

Expert’s duty to the Court 2.1 An expert has an overriding duty to assist the Court impartially, by giving objective, unbiased opinion on matters within the expert’s specialised knowledge. 2.2 This duty overrides any obligation to the commissioning party or to the person by whom the expert is paid.

Practice Directions: Content of all expert reports 4.1 All expert reports to which this Practice Direction applies (including primary expert reports and responding expert reports) shall state the opinion or opinions of the expert and shall state, specify or provide— …

(c) whether and to what extent the opinion(s) in the report are based on the expert’s specialised knowledge, and the training, study experience on which that specialised knowledge is based;

(d) the material, observed facts, reported facts, assumed facts and other assumptions on which each opinion expressed in the report is based (a letter of instructions may be annexed); …

(f) (if applicable) that a particular question, issue or matter falls outside the expert's specialised knowledge;

(g) any examinations, tests or other investigations on which the expert has relied, identifying the responsible laboratory by which, and the relevant accreditation standard under which, the examination, test or other investigation was performed; …

Practice Directions: Content of all expert reports 4.1 All expert reports to which this Practice Direction applies (including primary expert reports and responding expert reports) shall state the opinion or opinions of the expert and shall state, specify or provide— …

(i) any qualification of an opinion expressed in the report, without which the report would or might be incomplete or misleading;

(j) any limitation or uncertainty affecting the reliability of(i) the methods or techniques used; or (ii) the data relied on, to arrive at the opinion(s) in the report; and

(k) any limitation or uncertainty affecting the reliability of the opinion(s) in the report as a result of—(i) insufficient research; or (ii) insufficient data.

Practice Directions: Content of all expert reports

4.2 Where an expert is aware of any significant and recognised disagreement or controversy within the relevant field of specialised knowledge, which is directly relevant to the expert’s ability, technique or opinion, the expert must disclose the existence of that disagreement or controversy.

7. Other trial safeguards: Cross-examination, expressions, rebuttal experts, directions and warnings

‘How to cross-examine forensic scientists: A guide for lawyers’ (2014) 39 Australian Bar Review 174.

Other safeguards: Trials and appeals•Admissibility standards and exclusionary rules•Prosecutorial restraint (Wood v R)• Prosecutorial obligations (as a ‘minister of justice’)• Expert witness obligations and duties (and oath)• Restrictions on expressions (e.g. similarity evidence)• Cross-examination• Defence (‘rebuttal’) experts• Judicial directions and warnings• Jury (and trial judge)• Standard (i.e. BRD) and burden of proof• Appellate review

Trial safeguards?

‘Cross-examination, the adduction of contrary expert evidence and judicial guidance at the end of the trial are currently assumed to provide sufficient safeguards in relation to expert evidence, by revealing to the jury factors adversely affecting reliability and weight. However, … it is doubtful whether these are valid assumptions.’

Law Commission of England and Wales, Expert evidence in criminal proceedings in England and Wales (2011) [1.20]-[1.21], [3.4]. See also The Rt Hon Sir Brian Leveson, Review of the Effectiveness of Criminal Justice (London, 2015).

Expression of opinion(most reliable declining)

•DNA profiles: probabilistic (usually frequentist). (Validated – derived from mainstream scientific research)•Latent fingerprints: positive ID (i.e. individualisation).•Ballistics: individualisation to a specific weapon. (NV)•Incriminating images (e.g. CCTV): No individualisation, just similarities (and, in theory, differences). (NV)•Incriminating voice recordings: individualisation, even across languages. (NV)•Microscopic hair comparison: historically, very similar or the same and cannot exclude (Invalid) •Bite marks: historically individualisation. (Invalid)

NV – not validated

Recommendation 3.7: [E]mpirical evidence and

statistical reasoning do not support a source attribution

… latent [finger]print examiners should not report

or testify, directly or by implication, to a source

attribution to the exclusion of all others.

NIST & NIJ, Latent Print Examination (2012)

McQuiston & Saks, (2008) 59 Hastings Law Journal 1159

Lay interpretations (of expressions)American Board of Forensic Odontology

8. System issues – can the evidence be rationally evaluated?

‘Forensic science evidence and the conditions for rational (jury) evaluation’ (2015) 39 Melbourne University Law Review 75-123.

Makita from Hillstead v R [2005] WASCA 116

‘It is a primary duty imposed on experts in giving opinion evidence to furnish the trier of fact with the criteria to enable the evaluation of the expert conclusion: Makita (Australia) Pty Ltd v Sprowles. … The ‘bare ipse dixit’ of a scientist upon an issue in controversy should carry little weight: Davie v Magistrates of Edinburgh (1953).’

9. Strategic action

10. Legal versus non-legal approaches to expertise

‘Legal versus non-legal approaches to forensic science evidence’ (2016) 20 International Journal of Evidence & Proof 3

Demonstrable reliability is the most important issue for scientific (and non-scientific) expert evidence, especially forensic science.

For forensic science evidence (esp. comparison techniques), obtain clear answers to:

• Can the witness do the specific thing claimed and how do we know? How accurate are they and how do we know? What are the limitations with this technique?

• In the absence of this information, it is far from obvious that opinions are actually expert or susceptible to rational evaluation.

(in)Appropriate heuristics?According to courts: ‘field’, formal qualifications, experience, science/non-science, novelty, partisanship, previous admission (in Australia or elsewhere), jury can’t do it themselves, no identified error.

According to scientists: validation and reliability, uncertainty and error rates, empirically-based standards, elimination of contextual bias, publication, standardise expressions, independence from law enforcement.

GFMT scores for Australian passport officers

Normative (students): M = 81.3%; SD = 9.7Passport Officers: M = 79.2%; SD = 10.4

White et al. (2014). Passport Officers Errors in face Matching. PLOS ONE 9(8)

Legal performance•No court in Australia has produced anything remotely resembling the kinds of systematic criticisms and concerns identified in the NAS and other reports. Trial safeguards have not consistently identified, let alone conveyed, the most serious evidentiary limitations with forensic science evidence. •Legal rules and concerns are frequently displaced from the best independent scientific advice.•Trial safeguards cannot overcome evidentiary deficiencies – such as inadequate research.•Awareness of serious evidentiary and legal system limitations does not seem to be part of legal consciousness (or practice).

NSW is ‘out of step’ with comparator jurisdictions

• US (federal and most states) – valid and reliable

Daubert, Kumho, Federal Rules of Evidence r 702

• Canada – ‘threshold reliability’Trochym, JLJ, Abbey, DD and the Goudge

Report

• England – ‘sufficiently reliable’Law Commission report (20110, Criminal

Procedural Rules (2014), Part 33; Leveson review (2015).

Issues from overarching trial principles•What value does ‘expert’ opinion of unknown value have in a rational system of justice?•How is the jury supposed to gauge the value of opinions in the absence of information about whether the technique works, its limitations, an indicative error rate and insight into the proficiency of the ‘expert’? •Is the performance of an ‘expert’ under cross-examination, or her credibility and demeanour, or confidence, meant to operate as a surrogate to formal evaluation of techniques and procedures?•Why should the accused by obliged to identify, unpack and explain limitations and oversights with untested techniques (in routine use)? •Should legal practice be so divergent from mainstream science? Is there a plausible rationale?

[email protected]