The Difference between Track and Testing Performance

AV Testing Workshop, Reykjavik, 16 May 2007

The difference between track and testing performance

Roel Schouwenberg, Senior Anti-Virus ResearcherKaspersky Lab [email protected]

AV Testing Workshop, Reykjavik, 16 May 2007

About:Roel

Malware analysis AV research Incident response

AV Testing Workshop, Reykjavik, 16 May 2007

Overview

Testing AV engine Testing AVendor’s response time Product technologies Conclusions

AV Testing Workshop, Reykjavik, 16 May 2007

Current testing

On-demand WildList (won’t go there) Large (zoo) test bed

Retrospective using x month old product

On-access (not so common or detailed)

AV Testing Workshop, Reykjavik, 16 May 2007

On-demand: obvious flaws

Trash files Age of samples Lack of transparency Response time is not a factor Lack of resources to perfect testing Etc.

AV Testing Workshop, Reykjavik, 16 May 2007

Infectors / Trojanizers

Trojanizers (PE, script) Real infectors Check response time for detection and disinfection

Creating trojanizer test bed can take a long time

AV Testing Workshop, Reykjavik, 16 May 2007

Online scan services

JottiScan, VirusTotal (and others) Much trash and ‘trash’ False positive issues Additional checks needed

SFX archives and so on

AV Testing Workshop, Reykjavik, 16 May 2007

Testing vs track performance

Detection on/of packer/crypter Compare results with and without packer detection Differentiate between packers

Regular vs custom packer/crypter Generic vs detecting specific family

Age of samples 1/2/3/6/12 months old

AV Testing Workshop, Reykjavik, 16 May 2007

Differentiate between malware

Regional malware Malware coming from a region Payload (Banker vs GameThief trojan)

Automagically fabricated samples How many Zlobs do you want in the equation?

AV Testing Workshop, Reykjavik, 16 May 2007

Response time

Global outbreak Localized outbreak Low priority malware Infectors/trojanizers

AV Testing Workshop, Reykjavik, 16 May 2007

Retrospective testing

1 second is enough Modified ‘droppers’ Type of samples

AV Testing Workshop, Reykjavik, 16 May 2007

Product technologies

HIPS-like module Components working together – AV vs IS (Memory scanner)

Not so relevant (in this case): Malware removal Registry cleanup Malware detection on infected system

AV Testing Workshop, Reykjavik, 16 May 2007

Conclusions

Other/nicer ways to check out the competition Product technologies make testing-life harder Testing will always be flawed

AV Testing Workshop, Reykjavik, 16 May 2007

The end

Thank you for your attention!

Questions or comments?

[email protected]