Using TAR 2.0 to Reduce Review Costs

877.557.4273

catalystsecure.com

Using TAR 2.0 to Reduce Review Costs Moderator: Michael Arkfeld Speakers:

David Stanton

John Tredennick

Thomas C. Gricks

A primer for legal professionals

Webinar

v

John is a former trial lawyer and litigation

partner with a large national law firm, and has

written or edited five books and countless articles on litigation and

technology issues. He was recently named one of the top six e-

discovery trailblazers by The American Lawyer. He was also named

one of the “Top 100 Global Technology Leaders” by London's

CityTech magazine. John served as chair of the ABA Law Practice

Management Section and editor-in-chief of its flagship magazine.

Speakers John Tredennick

A prominent e-discovery lawyer and one of the

nation's leading authorities on the use of TAR in

litigation, Tom joined Catalyst in June. He advises corporations and law

firms on best practices for applying Catalyst's TAR technology, Insight

Predict, to reduce the time and cost of discovery. He has more than 25

years’ experience as a trial lawyer and in-house counsel, most recently

with the law firm Schnader Harrison Segal & Lewis, where he was a

partner and chair of the e-Discovery Practice Group.

Thomas C. Gricks III

David Stanton is a partner in the law firm’s

Litigation practice. He leads the firm’s nationally

recognized Information Law and Electronic Discovery group, oversees

the firm’s nationwide Litigation Support department, and he is a

member of Pillsbury’s Privacy, Data Security & Information Use

group. Mr. Stanton is also a member of the firm's Professional

Responsibility Committee and he serves as Pillsbury’s Executive

Partner for Anti-Bribery/Anti-Corruption Compliance.

David Stanton

Michael Arkfeld is a consultant, litigator, educator and

author. Michael is the Founder and Director of

Education for eDiscovery Education Center and Director of the Arkfeld

eDiscovery and Digital Program at the Sandra Day O’Connor College of

Law at Arizona State University. This program hosts the annual ASU-

Arkfeld eDiscovery and Digital Evidence Conference held in March of

each year.

Michael Arkfeld

CEO & Founder

Partner, Pillsbury Winthrop Shaw Pittman

Managing Director

Moderator

Founder, eDiscovery Education Center

Setting the Stage A TAR 2.0 Primer

1

Case Study: Bank Production Review

Collection: 2.1 million documents

Initial relevance (richness): 1%

Review team used CAL

Tagged relevance increased to 25 to 35%

Then relevance dropped toward zero

Result: 98% Recall

Review: 6.4%

What is Technology Assisted Review?

1. A process through which humans work with a

computer to teach it to identify relevant

documents.

2. Ordering documents by relevance for more

efficient review.

3. Stopping the review after you have reviewed a

high percentage of relevant documents.

We Already Use It

How Does it Work?

Support Vector Machines

Naïve Bayes

K-Nearest Neighbor

Geospatial Predictive Modeling

Latent Semantic

How Does it Work?

Support Vector Machines

Naïve Bayes

K-Nearest Neighbor

Geospatial Predictive Modeling

Latent Semantic

"I may be less interested in the science

behind the "black box” than in whether it

produced responsive documents.“

Judge Andrew Peck (SDNY)

What Are the Savings?

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Percen

tageofR

elevan

tDocum

entsFou

nd(R

ecall)

PercentageofDocumentsReviewed

YieldCurve

Linear Review

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

YieldCurve

%ofDocuments

%Relevan

t

What Are the Savings?

Linear Review

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

YieldCurve

%ofDocuments

%Relevan

t

What Are the Savings?

Review 12% and get 80% relevant documents

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

YieldCurve

%ofDocuments

%Relevan

t

What Are the Savings?

Review 24% and get 90% relevant documents

TAR 1.0: One-Time Training

Expert required for training

TAR 2:0 Continuous Active Learning Review team does the training

What is TAR 2.0?

1. Continuous Active Learning is key

(one bite at the apple is not enough).

2. Reviewers train while they review.

No SME required.

3. Ranking is against all the documents.

No control set involved.

4. Properly tokenizes foreign language

documents.

5. Contextual diversity helps find what you

don’t know. Random sampling not required.

Practical Issues: TAR 2.0 vs. 1.0

2

Vulnerabilities of TAR 1.0

Continuous Active Learning is key

(one bite at the apple is not enough).

Reviewers train while they review.

No SME required.

Ranking is against all the documents.

No control set involved.

Properly tokenizes foreign language

documents.

Contextual diversity helps find what you

don’t know. Random sampling not required.

TAR 2.0 – Harnessing the Wisdom of the Crowd

A large group's aggregated answers to

estimation questions have been found to be

as good as, or better than, the answer given

by any of the individuals.

TAR 2.0 operates as a mechanism for

aggregating the wisdom of the crowd.

A crowd's individual judgments can be

modeled as a probability distribution of

responses with the mean centered near the

true mean of the quantity to be estimated.

TAR 2.0 - Defining Responsiveness

Crowds tend to work best when there is a correct answer to the

question being posed, such as a question about geography or

mathematics.

Critical to define and train/drill team on meaning of responsiveness.

Provide context

Present examples

Pose hypotheticals

Dialogue and debate

Issue tags as “categories of responsiveness”

Inclusive

Complete

Quality Control – TAR 1.0 v. TAR 2.0

Lawyers have an ethical obligation to

appropriately supervise document reviews

and productions.

In TAR workflows, supervisory responsibility

requires a mechanism to obtain insight into

the quality of the assessments being made

by the algorithm, and to respond to them.

Sampling

TAR 2.0 ranking/categorization details.

TAR 2.0 Workflow From start to f inish

3

Continuous Active Learning – Contextual Diversity – Algorithmic QC

An Overview of TAR 2.0

Step 1 – Import Any Collection

About the Collection

1. Predict operates on text

2. Any collection, any time

Rolling collections are seamless

3. Incorporated into Ranked Collection

Initial collection is random

Rolling collections rank by existing coding

Certain documents are unranked

Step 2 – Seed/Train the Algorithm

Training is Immediate, Ubiquitous & Constant

1. You can use any documents to seed

Random or judgmental seeds

Synthetic Seeds

2. You can use any number of documents

3. EVERY attorney decision is used

Review

Search

“One-touch” coding

Step 3 – The Algorithm Ranks the Collection

Ranking Occurs Every Several Minutes

Reviewer decisions between rankings:

1. Rank every 10 minutes

10 reviewer decisions per ranking

2. Rank once per day

480 reviewer decisions per ranking

Waiting until the end of the day loses the

benefit of 48x the reviewers decisions…

Step 4 – Review Top-Ranked Documents

Background – Contextual Diversity & Algorithmic QC

Effectively Finding the Unknown

Contextual Diversity

X

- True positive

- True negative

- Coded positive

- Coded negative

Typical Quality Control

Separate QC for positive calls and negative calls

Catalyst Algorithmic Quality Control

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

1. Model the NEGATIVE documents

2. Rank only positive-coded documents

by their likelihood of being negative

1. Check the Positive Calls 2. Check the Negative Calls

1. Model the POSITIVE documents

2. Rank only negative-coded documents

by their likelihood of being positive

X

X

X

X

X

X

X

X

X

X

Overturn

Overturn

Overturn

Overturn

Overturn

Overturn

Step 5 – Review… Rank… Repeat

Typical Endpoint: Near Depletion

The Law of TAR A quick look at the

state of the law

4

“In evaluating whether search terms or search methods

employed to carry out the search were appropriate, the

court applies a reasonableness test to determine the

adequacy of search methodology. An adequate search is

one that could . . . have been expected to produce the

information requested.”

Treppel v. Biovail Corp., 233 F.R.D. 363, 373-374 (S.D.N.Y. 2006)(Francis, J.)

Eurand, Inc. v. Mylan Pharms., Inc., 266 F.R.D. 79, 85 (D. Del. 2010)

Rule 26(g)(1) - “certifies that to the best of the person’s knowledge, information, and belief formed after a reasonable inquiry . . .”

The Law of Search – “Reasonably Comprehensive”

“This judicial opinion now recognizes that

computer-assisted review is an

acceptable way to search for relevant ESI

in appropriate cases.”

Da Silva Moore v. Publicis Groupe

(S.D.N.Y. 2012)

Go Ahead, Dive In!

Unanimous Approval

Nat’l Day Laborer Org. Network v. U.S. Immigration & Customs Enforcement

Agency, No. 10 Civ. 2488 (SAS), 2012 WL 2878130 (S.D.N.Y. July 13, 2012).

EORHB, Inc. v. HOA Holdings, LLC, No. 7409-VCL (Del. Ch. Oct. 15, 2012).

Global Aerospace, Inc. v. Landow Aviation, L.P., No. CL 61040 (Vir. Cir. Ct. Apr.

23, 2012).

In re Actos (Pioglitazone) Prods. Liab. Litig., MDL No. 6:11-MD-2299 (W.D. La.

July 27, 2012).

2012

Unanimous Approval

Gordon v. Kaleida Health, No. 08-CV-378S(F), 2013 WL 2250579 (W.D.N.Y.

May 21, 2013).

In re Biomet M2a Magnum Hip Implant Prods. Liab. Litig., 2013 U.S. Dist.

LEXIS 84440 (N.D. Ind. Apr. 18, 2013).

In re Biomet M2a Magnum Hip Implant Prods. Liab. Litig., 2013 U.S. Dist.

LEXIS 172570 (N.D. Ind. Aug. 21, 2013).

EORHB, Inc. v. HOA Holdings, LLC, No. 7409-VCL, 2013 WL 1960621 (Del.

Ch. May 6, 2013).

Gabriel Techs., Corp. v. Qualcomm, Inc., No. 08CV1992 AJB (MDD), 2013

WL 410103 (S.D. Cal. Feb. 1, 2013).

2013

Unanimous Approval

Bridgestone Americas, Inc. v. Int. Bus. Machs. Corp., No. 3:13-1196 (M.D.

Tenn. July 22, 2014).

Dynamo Holdings Ltd. P’ship v. Comm’r of Internal Revenue, Nos. 2685-11,

8393-12 (T.C. Sept. 17, 2014).

FDIC v. Bowden, No. CV413-245, 2014 WL 2548137 (S.D. Ga. June 6, 2014).

In re Bridgepoint Educ., Inc., No. 12cv1737 JM (JLB), 2014 WL 3867495 (S.D.

Cal. Aug. 6, 2014).

Progressive Cas. Ins. Co. v. Delaney, No. 2:11-cv-00678-LRH-PAL, 2014 WL

2112927 (D. Nev. May 20, 2014).

2014

“The case law has developed

to the point where it is black

letter law that if a party wants

to utilize TAR for document

review, courts will permit it.”

The Law

Different Uses Other ways to use

TAR 2.0 Engines

5

Outbound productions

Reduce review costs and time dramatically by

establishing a cutoff point and stopping review.

Save up to 80-90% or more on review costs!

Run a privilege QC before production.

Sleep better at night!

How Can I Use it?

In-bound productions

In-bound productions allow your team to find

hot documents for depositions and trial.

Find relevant documents in a fraction of the time!

How Can I Use it?

Early case assessment

The SEC and DOJ actively use TAR to get a quick

handle on documents produced in their

investigations.

Greg Buckles, eDJ Analyst

How Can I Use it?

Focus team on multiple issues during review to

save time and effort

Issue/custodian review

How Can I Use it?

Non-English documents

TAR works effectively on any language so long

as you properly process the language.

How Can I Use it?

v

John is a former trial lawyer and litigation

partner with a large national law firm, and has

written or edited five books and countless articles on litigation and

technology issues. He was recently named one of the top six e-

discovery trailblazers by The American Lawyer. He was also named

one of the “Top 100 Global Technology Leaders” by London's

CityTech magazine. John served as chair of the ABA Law Practice

Management Section and editor-in-chief of its flagship magazine.

Speakers John Tredennick

A prominent e-discovery lawyer and one of the

nation's leading authorities on the use of TAR in

litigation, Tom joined Catalyst in June. He advises corporations and law

firms on best practices for applying Catalyst's TAR technology, Insight

Predict, to reduce the time and cost of discovery. He has more than 25

years’ experience as a trial lawyer and in-house counsel, most recently

with the law firm Schnader Harrison Segal & Lewis, where he was a

partner and chair of the e-Discovery Practice Group.

Thomas C. Gricks III

David Stanton is a partner in the law firm’s

Litigation practice. He leads the firm’s nationally

recognized Information Law and Electronic Discovery group, oversees

the firm’s nationwide Litigation Support department, and he is a

member of Pillsbury’s Privacy, Data Security & Information Use

group. Mr. Stanton is also a member of the firm's Professional

Responsibility Committee and he serves as Pillsbury’s Executive

Partner for Anti-Bribery/Anti-Corruption Compliance.

David Stanton

Michael Arkfeld is a consultant, litigator, educator and

author. Michael is the Founder and Director of

Education for eDiscovery Education Center and Director of the Arkfeld

eDiscovery and Digital Program at the Sandra Day O’Connor College of

Law at Arizona State University. This program hosts the annual ASU-

Arkfeld eDiscovery and Digital Evidence Conference held in March of

each year.

Michael Arkfeld

CEO & Founder

Partner, Pillsbury Winthrop Shaw Pittman

Managing Director

Moderator

Founder, eDiscovery Education Center

Catalyst designs, builds and hosts the world’s fastest

and most powerful document repositories for large-

scale discovery and regulatory compliance.

We back our award-winning technology with a highly skilled Professional Services team

and a global partner network to ensure the best e-discovery experience possible.

To view more the recording of this – and other Catalyst webinars – please visit:

http://catalystsecure.com/resources/events-and-webinars/on-demand-webinars