A Review of Damballa Failsafe Advanced Threat Protection ...

Finding Advanced Threats Before They Strike:

A Review of Damballa Failsafe

Advanced Threat Protection and Containment

March 2014

A SANS Analyst Review

Written by Jerry Shenk

Damballa Failsafe Overview Page 2

Damballa Failsafe Installation Page 4

The Dashboard Page 7

Reporting Page 8

How It Works Page 10

Forensic Detail Page 14

Sponsored by Damballa

©2014 SANS™ Institute

“No mechanical contrivance will take the place of honest watchfulness, though it may greatly aid the conscientious guardian.”1

That statement about locks and keys, written 140 years ago, applies to today’s digital treasures. Damballa Failsafe will help the conscientious guardians in an IT department detect, identify and remediate hostile attacks with a minimum of false positives. No matter how well we protect the front door, we must be vigilant and aware of advanced threats that evade preventive measures.

Over the last several years, the industry has gravitated toward sandboxing as a new way of protecting the front door. While sandboxing has its place in the prevention controls of the security stack, it shouldn’t be considered a silver bullet. Because attackers always have the first move, they can outmaneuver sandboxing if they are determined to do so.

In this paper, we review how Damballa Failsafe can be the last line of defense to alert you when devices have been compromised and can provide a forensic trail to show when the attack happened, what evidence there is that the attack was successful and, often, where the attack came from. It can also cross-reference the executable among monitored devices and identify when the attacks arrived on the network. Damballa Failsafe can use file copies, DNS monitoring, peer-to-peer activity, automated traffic and other suspicious activity to identify compromised hosts.

SANS Analyst Program 1 Finding Advanced Threats Before They Strike

Executive Summary

1 “Locks and Keys,” The Manhattan and de la Salle Monthly, October 1873, page 175.

Damballa Failsafe Overview


Damballa Failsafe uses a combination of sandboxing, threat intelligence and behavioral analysis to determine if threats are active. Sandboxing alone isn’t enough to detect attacks because some attacks may be hidden. Relying on a single technology can give a false sense of security. The corroborative evidence provided by Damballa Failsafe’s eight profilers (think detection engines) enables the appliance to detect threats over multiple tests and make a verdict on the level of infection of any computer on the network.

Damballa Failsafe is a network security system that helps organizations identify security threats on the network and respond to individual threats relative to their danger to the network. The appliance is administered using a web browser. The Assets tab, shown in Figure 1, demonstrates how Damballa Failsafe identifies systems and sorts them by risk, making it possible to identify infected systems with the highest risk rating quickly.

Figure 1. Damballa Failsafe Assets Tab Showing Risk Order

Damballa Failsafe Overview (CONTINUED)


False positives are virtually eliminated because Damballa Failsafe uses eight different profilers to identify malicious traffic. The appliance doesn’t just look at a file and say it’s bad; it identifies the malicious file or other activity and then watches for signs that the file has actually executed or undertaken additional activity to strengthen the case for considering the endpoint to be infected. Eliminating false positives can be a big time-saver for IT staff. If antivirus deleted the malicious file, there would be no execution at the computer and IT wouldn’t need to get involved. Damballa Failsafe also prioritizes each infection so staff can deal with higher priority infections first.

Along with detecting infection, Damballa Failsafe also provides definitive evidence that an endpoint is infected. Damballa Failsafe creates an extensive forensic record of each infection that helps security staff prove what happened and when it happened. This forensic trail includes identifying downloaded files, executed files, DNS queries, sites connected to, relevant packet captures and other collected evidence.

In our testing, Damballa Failsafe never flagged a computer as infected when it was not, and it did catch infected hosts using a variety of methods and provide ample evidence to support that conclusion.

Installation is simple. The appliance is rack mountable and has connections for a VGA monitor and a keyboard and mouse. Most organizations will choose to use them for initial setup, but there are also options for setup using an iLO3 connection or a serial port. During the initial setup, we were walked through a questionnaire asking for basic network information, for example, IP addresses, domain names, DNS servers and other information commonly needed for any server setup.

Sensor Placement

The most important installation step is sensor placement. The sensor has four Gigabit Ethernet ports, and there are expansion slots to hold more ports of various speeds and physical configurations (e.g., 10 Gbps copper and optical configurations are available). One port is used for management. This is the port that administrators will connect to. The other three ports are monitor ports and are used to collect traffic for analysis.

The monitor ports needs to be connected where they can see all necessary traffic. The simplest connection would be where workstations all use Internet DNS servers and connect directly to the Internet. In that case, the switch port that feeds the Internet firewall would need to be mirrored to a switch port to which one of the monitor ports on the appliance is connected. That would allow the appliance to see all traffic coming to and from the Internet as well as all DNS traffic (see Figure 2).

Figure 2. Basic Failsafe Sensor Placement

Damballa Failsafe Installation


Damballa Failsafe Installation (CONTINUED)


Life typically isn’t quite that simple, however. Nearly all larger networks should have some type of egress filtering, local DNS servers and perhaps a proxy server. In this case, we would want to have a monitor port that can see the Internet traffic and one each to monitor the DNS server(s) and the incoming port on the proxy server. The reason for monitoring the DNS and proxy servers is so that the Damballa Failsafe appliance can see which internal clients are making DNS and proxy requests. Figure 3 illustrates this setup.

Figure 3. Damballa Failsafe Setup for More Complex Systems

In most cases, it would also be good to have a monitor port on each service network or DMZ. Each network is different, so we can’t guess every possible option, but it would be wise to think through the deployment so that you see all interesting traffic (Internet, DNS and web traffic are key). Tip: Be sure to avoid getting duplicate packets from the same host with different IP addresses; for example, you don’t want to be collecting traffic on both sides of a NAT device.

Damballa Failsafe Installation (CONTINUED)


Asset Categories

One other thing to consider before installation is the categorization of devices by criticality. The categorization feature can be useful for prioritizing which asset gets attention first. Damballa Failsafe can categorize assets (computers) and give those categories a rank from 1 to 5, based on criticality of the device. Set these categories up before assets are discovered as being suspected of having a problem because the categorization is applied when the asset is discovered. If the workstation’s IP address’s attempts to resolve to a name using either NMB or Reverse DNS lookups fails, Damballa Failsafe will track activity to the endpoint’s IP address. Be sure to configure names in the categories or turn off name resolution in the Systems tab and the Settings options to just use IP addresses.

In the lab, we set up asset categories after we had done most of the testing, so we didn’t get much value from it. With that said, however, categorization would be a useful feature for a large network. Security staff could tell at a glance how many computers had performed actions that would, at least, cause them to show up as a tracked computer, which is called an asset. Figure 4 illustrates the asset categories we set up.

Figure 4. System Asset Categories

Most organizations want to have a quick visual reference for the health of the network. Damballa Failsafe’s dashboard fills that need. This dashboard is customizable with a number of useful widgets that can be manip-ulated as it suits the organization. The view in Figure 5 shows three important metrics: the number of infected assets (computer systems), the rate of infections over time and the assets that have the highest risk rating.

Figure 5. Widgets Available on the Damballa Failsafe Dashboard

Things are not looking good on this network—seven systems are infected, and that number has been growing over the past month. Of course, this is data from a test lab, so we are intentionally infecting systems, but this type of information is valuable for an IT manager. This information also suggests some action items: The system at 10.1.3.102 needs to be checked out right away, and 10.1.3.3 isn’t far behind.

The dashboard is the kind of display that should be running on a big screen in the security operations center or other highly visible location in the IT department so people are constantly aware of assets that need to be taken care of before they become a larger problem. Once an asset has been fixed, it can be marked as remediated from the Assets tab by clicking on the IP address or name of the asset that has been fixed.

The Dashboard


Reporting is another necessary feature for any network management device. Damballa Failsafe has a number of useful reports. The Executive Report, a portion of which is shown in Figure 6, provides a numeric status summary for the past month and a graph showing the percentage of infected assets and the percentage of assets that have been remediated.

Figure 6. The Executive Report

There are five prebuilt reports that the system can distribute via email on an hourly, daily, weekly and monthly basis. The reports are sent out at the beginning of the time period, GMT. You can set up email recipients to receive the reports most useful to them. The Executive Report and the System Health Report (see Figure 7) would be a good choice for a CSO or CISO to get on a daily or weekly basis.

Reporting


Reporting (CONTINUED)


Figure 7. The System Health Report

The system can also send alerts when infections are detected, when a sensor is down or when a network interface controller (NIC) is down. This type of appliance tends to get ignored when things are working well, so staffers might not notice if one of the sensor NICs was unplugged. People tend to assume that no alerts means the devices are doing their jobs well. This is an important notification that is often missed.

Damballa Failsafe currently has eight profilers that it uses to identify suspicious events. Profilers assess the behavior associated with an asset and assign a weight to each activity. Combinations of different activities can act as an accelerator to the overall scoring when an infection is present or, conversely, negate activity when the event is likely just a user surfing a suspicious site. When the score associated with an asset is high enough, that asset is determined to be infected. We’ll look at some of the profilers in more detail.

As an overview, the File Profiler detects when files are downloaded across the wire. The Domain Fluxing profiler watches for DNS requests that statistically fit the profile of domain generation behaviors (an advanced evasion technique used by malware authors). The Connection Profiler captures the raw network packets; the Request Profiler analyzes network HTTP requests for statistically similar behavior of known malicious traffic. The Connection and Request profilers are covered in the “Forensic Detail” section of this paper.

There are other profiles included now, and more are constantly in the works. Damballa Failsafe is designed to add profilers as they are needed to increase the detection and continue to keep false positives almost nonexistent.

File Profiler

Damballa Failsafe monitors traffic on the network using its various sensor ports to look for suspicious traffic. One key indicator of a network compromise is when a computer (asset) downloads a file. This is not the only way the appliance detects an infected asset, but it is the unique way Damballa Failsafe performs file analysis and seems to work well. The appliance does more than just catch the download; it also monitors communication to see if the downloaded file actually runs. Many times, a computer will download a malicious file, and antivirus will detect and delete the file before it runs. In that instance, there is no reason to have IT check on that computer—the antivirus software already took care of it. This is one way that Damballa helps eliminate false positives. Damballa will examine the file to determine whether it is a known bad file or a known benign file or whether the file needs further examination. Notice in the Assets tab shown in Figure 8 that three different assets, or computers, have done something suspicious and are listed as suspicious assets.

Figure 8. Damballa Failsafe Assets Tab Showing Suspicious Assets

How It Works


How It Works (CONTINUED)


The system has given two of the assets a verdict of suspected and a verdict of infected to one asset. To determine the risk and verdict of the assets, the local appliance examines the file and, if necessary, uploads it to Damballa labs for further analysis. Known files can be identified as a known bad file (malware) or as a benign file quickly by their MD5 hash. If there is no hash for the file, then the appliance can send it to Damballa for further analysis.

This analysis is similar to sandboxing, which can happen locally, but Damballa Failsafe doesn’t rely on sandboxing alone. Sandboxing alone leaves detection gaps—so Damballa examines the network characteristics of downloaded files. When those characteristics are known, the asset can then be monitored for matching network activity. If matching activity is detected, then Damballa Failsafe knows that the file was, in fact, executed. On the Assets tab, when we click on an IP address, it opens a detail window (see Figure 9).

Figure 9. The Assets Detail Window

In the figure, you can see that the computer with the IP address of 10.1.3.8 downloaded five files and executed two of them. You can also see that four of the files are malicious and one of them is suspicious. After a file is downloaded, the Damballa Failsafe appliance buffers network activity for 72 hours. This gives an advantage to investigators working on infections that haven’t been identified yet.



DNS Fluxing Profiler

DNS fluxing is a relatively new method that malware authors have developed to enable compromised computers to connect to a “master” computer. Threat actors need DNS names because IP addresses get blocked as soon as malicious activity is detected. But DNS names get blocked, too—so malware developers use domain fluxing so they can delete a DNS name after it has been detected—sometimes after only an hour or less of use. The malware has a range of DNS names that it will work through until it finds a host. In Figure 10, 10.1.3.4 is infected with a Zeus variant, which the Domain Fluxing Profiler caught.

Figure 10. Malware Caught with the Domain Fluxing Profiler



In this example, you can see under Event Count that there were 6390 events. In this case, that means that 6390 domain lookups were performed. The sheer quantity of lookups seems suspicious, but you can get even more information by clicking the number of events to display more detail, as shown in Figure 11.

Figure 11. Domain Lookup Details

The names that were looked up also support the idea that this is not normal workstation activity. The top one looks like a bunch of random letters followed by .ru, a Russian domain. In this case, the infected endpoint generated domains to match the threat actor, who has the same algorithm, and was searching for a domain that would resolve to an IP address, thus enabling it to connect to the threat actor.

Missing Profilers

One thing that I was surprised about was that outbound spam did not trip any of the profilers. Spam is a serious problem on the Internet, and it is a component of many types of malware. Most enterprise networks block outbound port 25 traffic—or at least the organizations that are investing in serious security hardware do—so perhaps this is not that important. Even if the traffic is blocked by egress rules on the firewall, those repeated, consistent attempts by devices that are not mail servers could be useful in identifying malware infections. Such instances of malware in our test lab were detected by other means.

One other piece of malware connected to an IP address over the Internet using IRC-like commands. Damballa Failsafe did not pick up this behavior itself, but it did pick up malware on this machine using the File Profiler.

There is a lot of malware out there, and it is nearly impossible to keep up with the threats. These missed indicators were minimal in our testing. Overall, the profilers do a good job of finding, identifying and reporting malicious software.

One problem that often confronts incident investigators is the lack of evidence. We often get an alert to an event, and whatever alerted us expects us to just trust them. Well, often we don’t—and we shouldn’t. We should have enough information to verify the alerts we are given. The Damballa Failsafe appliance provides that information in an organized, logical, drill-down style that makes it easy to find what you need. Nearly every step along the way has an option for saving the information to a CSV file. During testing, we used a utility called Snagit to capture screenshots. With the combination of screenshots and the CSV files, we could build a very strong case about an infection.

Earlier in this paper, we looked at examples of the File Profiler—which documented file downloads and whether they were executed or not—and the Domain Fluxing Profiler. Profilers could be called detection engines. They look at network communication to see whether it matches any existing profile of known malicious activity. Figure 12 is an example of the Request Profiler.

Figure 12. Request Profiler

Here we have an example of a web-based GET request—a major part of web traffic. This looks like a normal page request over port 80. Damballa Failsafe also captured the raw network communication as additional evidence.

The Request Profiler says that this traffic matched something that made it think that it was malicious, but it also shows us exactly what it doesn’t like. We love this concept of answering the question, “Prove it.” In this case, the structure of the GET request is the same, but the variables are different. Damballa Failsafe then takes this analysis a step farther and shows us the malware that created this “similar request”—that is the MD5 code of the malware that created the sample that our traffic is compared with. We reviewed the other information that we have about this host, and it never downloaded this file. This was a host that we infected directly using malware that was transported using a USB drive.

Forensic Detail


Forensic Detail (CONTINUED)


The goal of this exercise was to see if Damballa Failsafe would detect an infection even if the initial infection vector was hidden. Some examples of how this might happen in real life include somebody finding a USB drive or other removable media that had malware on it or taking a laptop to a coffee shop or hotel, where it can get infected while it is outside the control and protection of the corporate environment. Another hidden infection vector is downloading an executable file over an encrypted network connection, such as an SSL-protected website. Often assets are compromised while the initial compromise is hidden, so it is imperative to pick up signs of the compromise.

Damballa Failsafe also stores the raw network packets. We used Wireshark, a widely used packet sniffer, to look at the raw packets that went across the wire. Figure 13 is an image of Wireshark’s packet-level view of the same communication.

Figure 13. Wireshark Packet-Level Inspection

The connection information was downloaded as a pcap file and loaded into Wireshark for display. Damballa Failsafe’s Connection Profiler captured this same information and also makes it available as a CSV file containing the summary of the connection (IP addresses, ports, time, and so on). Such information would be useful for a longer-duration issue in which there were a number of connections.

IT organizations everywhere are stretched thin trying to get a handle on the growing malware problem. Resources are often wasted on false alarms—and after enough false alarms, the devices sounding the alarms get ignored. Damballa Failsafe virtually eliminates false alarms by comparing evidence collected by its eight profilers and information from Damballa threat intelligence. By assigning a graded verdict to infections, IT departments are able to allocate their limited resources where they are needed most.

Damballa Failsafe does not rely on sandboxing or any other single method for detecting malware. By using multiple detection and analysis profilers, Damballa is able to do well at detecting infections and determining which ones are actionable. Managers will appreciate how Damballa Failsafe not only monitors network traffic to discover infected devices and rank them, but also maintains statistics that can be distributed via email on a consistent, automated schedule.

IT administrators and forensic analysts will appreciate the detailed information about what infected hosts did and when they did it—and they’ll like having the raw network captures to prove it. IT administrators will also appreciate not wasting time checking up on workstations that are not actually infected.

In our testing, we found the Damballa Failsafe to do well at its job of discovering malware, avoiding false positives and providing detailed analysis for remediation teams. In addition to the SANS testing, Damballa has done some comparisons between Damballa Failsafe and sandboxing solutions. The results of that testing are available upon request from Damballa.

Summary


Jerry Shenk currently serves as a senior analyst for the SANS Institute and is senior security analyst for Windstream Communications, working out of the company’s Ephrata, Pennsylvania location. Since 1984, he has consulted with companies and financial and educational institutions on issues of network design, security, forensic analysis and penetration testing. His experience spans networks of all sizes, from small home-office systems to global networks. Along with some vendor-specific certifications, Jerry holds six Global Information Assurance Certifications (GIACs), all completed with honors: GIAC-Certified Intrusion Analyst (GCIA), GIAC-Certified Incident Handler (GCIH), GIAC-Certified Firewall Analyst (GCFW), GIAC Systems and Network Auditor (GSNA), GIAC Penetration Tester (GPEN) and GIAC-Certified Forensic Analyst (GCFA). Five of his certifications are Gold certifications. He also holds the CISSP certification.

About the Author


SANS would like to thank this paper’s sponsor:

A Review of Damballa Failsafe Advanced Threat Protection ...

Documents