Active Testing

The Importance of Re-creating In-the-Wild Infection Conditions for Testing Multi-Layered Security ProductsMark Kennedy

May 15th, 2007

The Importance of Re-creating In-the-wild Infection Conditions for Testing Multi-Layered Security ProductsThe Importance of Re-creating In-the-wild Infection Conditions for Testing Multi-Layered Security Products

2

Overview

Current Trends11

Traditional Static Analysis22

Proactive Static Analysis33

Dynamic Analysis44

Lab Bias55

The Importance of Re-creating In-the-wild Infection Conditions for Testing Multi-Layered Security Products

Problem Statement

• Current testing methods only exercise a portion of security suites

– Heavily geared toward static file scanning• Signatures

• Packers

• Emulators

• New types of Security Suites require new types of testing

– Multiple layers protection

– Existing testing methods test only a portion of these solutions

3

The Importance of Re-creating In-the-wild Infection Conditions for Testing Multi-Layered Security ProductsThe Importance of Re-creating In-the-wild Infection Conditions for Testing Multi-Layered Security Products

Current Trends

Types and Techniques• Obfuscation Techniques

– Polymorphism

– Metamorphism

– Packed Variant

– In Memory only Threats (no on disk footprint)

• Yesterday’s Threats

– File Infectors

– Mass Mailing Worms

• VB Script

• SMTP Mass Mailers

• Current Threats– Non Self Replicating

– Targeted Attacks

• Threats created for a specific target

– File Infectors and Worms decline

Motivations and Payloads• Yesterday’s Threats

– Spreading

– Fame (infamy)• Making the news

– Vandalism

• Current Threats– Monetary gain

• Bancos

• Identity theft

– Long lasting control of the machine

– High value assets of specific machines

4


Traditional Testing Method

• Primarily Static Analysis

• Large directory of Zoo and ITW samples

• Extensions modified to prevent accidental execution

• Names changed to indicate threat or family

5


Traditional Testing Method

• Pros for Traditional Static Analysis

– Fast• Helps meet tight deadlines

– Well understood

– Large existing collections

• Cons for Traditional Static Analysis

– Highly dependent on signatures

– Limited heuristics due to threat not actually executing on a live system

– Vulnerable to obfuscation

– Limited effectiveness to truly new threats

6


Proactive Static Analysis

• Tested using Traditional Testing Method

– Freeze Virus signatures

– Rollback Virus signatures

• Windows emulators

– NOD32

• Sand Box Emulators

– BitDefender

– Norman Sandbox

7


Proactive Static Analysis

• Pros

– Detect threats prior to execution

– Detect threats without signatures

– Can bypass some obfuscation techniques

• Cons

– Performance intensive

– Vulnerable to sophisticated obfuscation techniques• Obfuscators which make use of obscure APIs cannot be emulated

• Obfuscators which make use of obscure instructions can fool them

• Malcode can detect the emulator and change its behavior

• Threat could require a minimum number of executions or time prior to becoming active

8


Results:

• Current testing methods are becoming less meaningful

– Only testing a portion of the Security Suite

– Individual results are accurate, but do not fully reflect the true customer experience

• Reliability

– Static testing has become unreliable due to the increased dynamic nature of malware

• Bottom line: Current tests are not producing as Customer-relevant results as they could

9


Multi-Layered Security Products

• Defense in Depth

– Firewall

– Host based Intrusion Prevention

– Buffer Overflow Protection / Browser Exploit Protection

– Real-time file scanning

– Shields

– Behavior Blocking

10


Page 11

Symantec Client Layered Protection Architecture

OS & Application Vulnerabilities Targeted Attacks & Insider Threats

Network Filtering“Block threats before they impact the client”

Behavior Blocking“Police execution activity”

Storage Filtering“Don’t let threats persist!”

Malware & SpywareZero Day Threats

My Only Marketing Slide (I promise)


A Word about Success

• Correct Decision making

– Blocks threat at earliest possible point

– Low False Positive rate

• Automatic decision making

– No prompting/asking for permission

– Most users are not qualified to answer correctly

– May become fatigued

– Turn solution off

12


You All Remember This

13


Defense in Depth: Firewall

• First line of defense

• Inbound

– Prevents threats from getting onto the machine by:• Blocking known C&C ports

• Blocking ports used by non-essential services e.g. RPC

• Outbound:

– If threats cannot communicate their damage can be limited

– Application control. Only allow known, authorized applications.

14


Defense in Depth: Host Based IPS

• Analysis of network blocks

– Blocks malicious behavior

– Lets good behavior through

• Detect and block known Command and Control sequences

– Outbound

– Inbound

• Detect incoming vulnerability exploit attacks

– Known signatures

– Generic exploit signatures• A generic signature can block an entire family

15


Defense in Depth: Buffer Overflow Protection / Browser Exploit Protection

• Protect against Drive-by Downloads

– One of the most popular vectors for malware to get on the machine.

– Any website is vulnerable, even trusted ones! Therefore any user can be infected, even if they only visit trusted websites.

• Prevents exploits in malicious HTML, VML etc.

• Detect buffer overflows in Browser script

• Detect abuse of Browser ActiveX objects

• BID 22680 (http://www.securityfocus.com/bid/22680)

– Microsoft Internet Explorer OnUnload Javascript Browser Entrapment Vulnerability

16


Defense in Depth: Real-time File Scanning

• Scans files when created or accessed

• Known signature detection

• Static Heuristic analysis

• Can analyze file prior to any access

17


Defense in Depth: Shields

• Monitor known hook points in OS

– Can look for suspicious hook points

– Can detect “over” hooking

• Monitor interactions with other processes on the system

– Detect injection, both direct and through Windows Hooks

– Detect attempts to terminate security processes

• Monitor tampering with security settings

– Attempts to disable firewall

– Attempts to add self to firewall exceptions

• Monitor tampering with HOSTS file

18


Defense in Depth: Behavior Blocking

• Closely related to Shields

• Can monitor how executables arrive on system

• Can correlate actions across numerous shield points

• Can detect collaboration between multiple processes

• Have a holistic view of system and interactions

• Has the context necessary to make correct decisions

19


An Analogy: Automobile Safety

• Past

– Safety was defined by seat belts

– Tests checked seat belts in isolation

• Current

– Auto safety is a system• Anti-lock brakes (ABS)

• Steering stabilization

• Crumple zones

• Airbags (driver, passenger, side)

• Seat belts

20

Is it fair to say one car is safer than another based only on seat belts?


Scoring Gradient: File Based Threat

21

Never executes

Executes but cannot

communicate

Communicates but is automatically

removed

Communicates but is removed by

definitions

Communicates and is never detected / cannot be removed

Content never reaches box

Never impact

Impact, but no damage (bumper)

Impact, but no injuries

Minor injuries, victims walk away

Major injuries, but survive

Some Fatalities

Fatalities, car explodes, kills

bystanders


Detractions

• Blocks which require user interaction should score lower

– Asking the user to make decisions is problematic

• Blocks which require updates should score lower

– Effectiveness subject to delays

• False positives should score lower

– User will lose confidence

– May impact productivity

22


This All Leads To…

Dynamic Testing: Testing real threats on real machines

Other Industries have adopted

• Auto industry stages real crashes with real cars

• Airline industry stages real crashes with real airplanes

23


Dynamic Testing

• Running real threats on real machines

– This is the acid test

– This is what matters to customers

• Running on real internet

– Many new threats need to phone home, or make contact in some way

– Many of today’s threats are primarily a threat to the machine they are running on, not to others (at least initially)

• Retrieving information off the test machine does no harm

• Only threats like spam bots which become active would be an issue, and that can be mitigated

– Some threats are dangerous, so you must know

24


Dynamic Testing (continued)

• Introduction vector and mode of execution important

– If a threat arrives from email and expects to be launched as an attachment, launching it another way may change its behavioral profile

– If a threat arrives via a browser exploit, then it should be created and launched by the browser

– The firewall must be configured just like the customer would for their environment

• In a home network environment, most customers put machines on their home network into the trusted zone.

• This would automatically open up ports that are normally closed by the firewall.

• Any machine that is infected on that network could infect this machine.

25


Discreet Dynamic Testing

• Isolate proactive portions of a product

• Prevent signature update

– Side effect: This may prevent product update

• Detections likely to have generic names

– Bloodhound

– Variant

– Exploit

– Newmalware

– Unknown

26


Dynamic Testing: Benefits

• Lab results better match real world

– Understand Lab Bias

– Take steps to limit it

• Greater Credibility

– Static testing is not as accurate a reflection of user experience

• Customer relevant results

• System testing methodology

– Legacy testing methods have inherent bias towards signatures that leads to skewed results

– As the threat landscape has evolved, and the security suites have evolved, so too must the testing methodology

27


Lab Biases

• Platform

• Method of introduction

• Method of invocation

• Internet connectivity

• Definition Rollback or freeze

28


Lab Biases: Platform

• VMWare and Virtual PC

– Threats may detect that they are executing in a virtual environment

– Once detected, they may modify their behavior

– Sufficient Resources required to run• If threat cannot perform escalation, or exceeds resources then the threat may

not function

• OS Revision and Patch Level

– Some threats may rely on unpatched vulnerabilities to operate

– Threat may not run, or may not exhibit malicious behavior under certain circumstances

• Open ports

• Installed components

29


Lab Bias: Method of Introduction

• Circumstances by which a threat is introduced to a system may be important

• Some Portals may be more trusted than others

– A Portal is way to introduce software• Email

• Browser

• CD

• USB key

– Some are more trusted• CD

– Than others• Email

• Browser

30


Lab Bias: Method of Invocation

• Automatic vs. manual vs. very manual

– Automatic• Drive-by download

• Downloader

– Manual• Email attachment

• Double-click

– Very manual• Command prompt, navigate, run

• These influence the behavioral score

31


Lab Bias: Internet Connectivity

• Many threats need to phone home

• Establish connection for Command and Control

• Establish connection for content delivery

32


Lab Bias: Definition Rollback or Freeze

• Tests some aspect of heuristic/behavior detection

• Artificial state that does not match customer experience

• Can inadvertently roll back heuristic/behavioral componentry

• Can create mismatch errors should components presume minimum version of definitions

33


Dynamic Testing: “Do”s

• Configure machines to natural conditions

– Test with unpatched OS

– Test with default security features of suite enabled

• Pay attention to threat injection vector

– Email borne threats should be tested from email

– Browser borne threats should be tested using the browser• If arrive from exploit, construct an exploit

• Pay attention to invocation

– If a threat needs to run twice, once to “install” and once to act, test it that way

• Use as much “real” internet as is safe

– If a threat does not affect other machines, give it freer reign

34


Dynamic Testing: “Don’t”s

• Just scan the file and conclude effectiveness

– Many other layers may provide detection

• Launch the threats manually

– Particularly from the desktop

• Publish tests without publishing criteria

– Important to understand what the data means

• Publish tests without publishing methodology

– Important to understand how the data was calculated

35


Summary

• Threats have changed

• Testing methodology must also change

– Better simulate real world conditions

– Actively execute threats

• Need objective method for comparing

• Not an easy problem to solve

– However, it is an important problem that must be solved

36


Questions?

37

The Importance of Re-creating In-the-wild Infection Conditions for Testing Multi-Layered Security ProductsThe Importance of Re-creating In-the-wild Infection Conditions for Testing Multi-Layered Security ProductsP

resentation Identifier Goes Here

38

Copyright © 2007 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners.This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.

Thank You!

Mark Kennedy

[email protected]

310-449-4263

Active Testing

Technology

new threats

traditional static analysis

day threats

proactive static analysis

dynamic analysis

payloads yesterdays

behavior threat

line of defense