NETWORK SECURITY ATTACKS - Routledge

NETWORKSECURITY ATTACKS

A CRCPRESS FREEBOOK

CLICI< HERE

TO BROWSE FULL RANGE OF SECURITY TITLES

SAVE 20% AND FREE STANDARD SHIPPING WITH DISCOUNT CODE

JML20

https://www.routledge.com/search?pg=1&sb=ITC10A&so=pub&pp=12&view=list&pd=published&publishedFilter=alltitles

https://www.routledge.com/Security-for-Software-Engineers/Helfrich/p/book/9781138583825



22 | Chapter 02: Classification of Attacks | Unit 1: Attack Vectors | Security for Software Engineers

Chapter 02: Classification of Attacks

The essential skill of this chapter is to recognize the many different ways an attack can be manifested. Only by understanding all of these attack vectors can we take steps to prevent them.

The first step in writing secure code is to understand how black hats exploit vulnerabilities. For this to be done, a few terms need to be defined.

Asset An asset is something of value that a defender wishes to protect and the attacker wishes to possess. Obvious assets include credit-card numbers or passwords. Other assets include network bandwidth, processing power, or privileges. A user’s reputation can even be considered an asset.

Threat A threat is a potential event causing the asset to devalue for the defender or come into the possession of the attacker. Common threats to an asset include transfer of ownership, destruction of the asset, disclosure, or corruption.

Vulnerability A threat to an asset cannot come to pass unless there exists a weakness in the system protecting it. This weakness is called a vulnerability. It is the role of the software engineer to minimize vulnerabilities by creating software that is free of defects and uses the most reliable asset protection mechanisms available.

Risk A risk is a vulnerability paired with a threat. If the means to compromise an asset exists (threat) and insufficient protection mechanisms exist to prevent this from occurring (vulnerability), then the possibility exists that an attack may happen (risk).

Attack An attack is a risk realized. This occurs when an attacker has the knowledge, will, and means to exploit a risk. Of course not all risks result in an attack, but all attacks are the result of a risk being exploited.

Mitigation Mitigation is the process of the defender reducing the risk of an attack. Attacks are not mitigated; instead risks are mitigated. There are two fundamental ways this can be accomplished: by reducing vulnerabilities or by devaluing assets.

An attack vector is the path an attacker follows to reach an asset. This may include more than one vulnerability or more than one asset. To see how these concepts are related, consider the following scenario: a malicious student wishes to change his physics grade on the school’s server. First we start with the asset: the grade. Next we consider the threat: damage the integrity of the grade by altering the data on the school’s server. The intent of the system is for this threat to be impossible. If a vulnerability in the system exists, then the threat is possible. This may happen if one of the administrators uses unsafe password handling procedures. Consider the case where one of the employees in the registrar’s office keeps his password written on a post-it note under his keyboard. Now we have a risk: a malicious student could obtain that password and impersonate the administrator. This leads us to the attack. Our malicious student learns of the post-it note and, when no one is looking, writes down the password. The next day

Security for Software Engineers | Unit 1: Attack Vectors | Chapter 02: Classification of Attacks | 23

he logs in as the administrator, navigates to the change-grade form, and alters the grade. Fortunately, an alert teacher notices that the failing physics grade was changed to an ‘A’. With some work, the source of the problem was identified. Mitigation of this attack vector is then to create a university policy where no passwords are to be written down by employees.

Classification of Attacks Software engineers are concerned about attack vectors because they illustrate the types of vulnerabilities that could yield an asset being compromised. An important part of this process is classification of possible attacks. There are three axes or dimensions of this classification process: the state of the asset, the type of assurance the asset offers, and the type of vulnerability necessary for an attack to be carried out. These three axes are collectively called the McCumber Cube (McCumber, 1991). Classification schemes are useful not only in precisely identifying a given attack that has transpired or is currently underway (an activity more in line with an I.T. professional than a software engineer), but also to brainstorm about different attack vectors that may be possible at a given point in the computing system. Perhaps it is best to explain these attributes by example. Consider a bank attempting to prevent a thief from misusing client assets (money in this case). The thief has a wide range of options available to him when contemplating theft of the assets.

Type of Asset The asset face of the McCumber Cube maps directly to the three security assurances. While we analyze these independently, it is important to realize that most user assets are a combination of the three.

Confidentiality

Confidentiality is the assurance that the software system will keep the user’s private data private. This could also be described as the assurance that only the owner of an asset can specify how the asset is viewed. Attacks of confidentiality are called disclosure attacks. If an individual views an asset contrary to the wish of the owner, then a confidentiality breech or a disclosure attack has occurred. Back to the aforementioned bank example, this will assure the client of a bank that his account balance will not be disclosed to the public. The attacker does not need to steal the client’s money to attack the money; he can simply post the client’s bank statement onto the front door of the bank.

Figure 02.1: McCumber Cube


Integrity

The assurance is that the software system will preserve the user’s data. It is a promise that the data is not destroyed, will not be corrupted accidentally, nor will it be altered maliciously. In other words, the user’s asset will remain in the same condition in which the user left it. In this digital age, it is difficult to make integrity guarantees. Instead, the best we can hope for is to detect unauthorized tampering. Attacks on integrity are called alteration attacks. If a change has been made to the user’s data contrary to his will, then integrity has been compromised or an alteration attack has occurred. In the banking example, there are many ways in which the attacker can launch an alteration attack. He could steal the contents of the client’s safety deposit box, he could alter the password to the client’s account, or he could deface the bank’s website. In each case, the state of one of the bank’s assets has changed in a way that was contrary to the bank’s will.

Availability

The availability assurance is that the user can have access to his informational, computational, or communication resources when he requires it. As with the integrity assurance, this includes resistance to availability problems stemming from software defects as well as denial attacks from individuals with malicious intent. Attacks on availability are called denial attacks, also known as denial of service (D.o.S.). If an attacker is able to disrupt the normal operation of a system in such a way that the availability of system resources is impacted, then a denial attack has occurred.

In the banking example, there are many ways that an attacker can launch a denial attack. He could put sleeping gas in the ventilation system temporarily incapacitating the employees, he could get a hundred of his friends to flash-mob the bank thereby consuming all the time and attention of the clerks, or he could detonate a bomb in the building destroying everything inside. In other words, denial attacks can be inconveniences, temporary outages, or permanent outages.

Attacks on C.I.A. assurances are collectively called D.A.D. for disclosure, alteration, and denial. All attacks fall into one or more of these categories. However, when a software engineer is analyzing a given system for vulnerabilities, it is often helpful to use a more detailed taxonomy. To this end, the S.T.R.I.D.E. system was developed.

S.T.R.I.D.E. The S.T.R.I.D.E. taxonomy was developed in 2002 by Microsoft Corp. to enable software engineers to more accurately and systematically identify defects in code they are evaluating. The S.T.R.I.D.E. model is an elaboration of the more familiar C.I.A. model but facilitates more accurate identification of security assets. There are six components of S.T.R.I.D.E. (Howard & LeBlanc, 2003).


Spoofing

Spoofing identity is pretending to be someone other than who you really are such as by getting access to someone else’s passwords and then using them to access data as if the attacker were that person. Spoofing attacks frequently lead to other types of attack. Examples include:

� Masking a real IP address so another can gain access to something that otherwise would have been restricted.

� Writing a program to mimic a login screen for the purpose of capturing authentication information.

Tampering

Tampering with data is possibly the easiest component of S.T.R.I.D.E. to understand: it involves changing data in some way. This could involve simply deleting critical data or it could involve modifying legitimate data to fit some other purposes. Examples include:

� Someone intercepting a transmission over a network and modifying the content before sending it on to the recipient.

� Using a virus to modify the program logic of a host so malicious code is executed every time the host program is loaded.

� Modifying the contents of a webpage without authorization.

Repudiation

Repudiation is the process of denying or disavowing an action. In other words, hiding your tracks. The final stages of an attack sometimes include modifying logs to hide the fact that the attacker accessed the system at all. Another example is a murderer wiping his fingerprints off of the murder weapon — he is trying to deny that he did anything. Repudiation typically occurs after another type of threat has been exploited. Note that repudiation is a special type of tampering attack. Examples include:

� Changing log files so actions cannot be traced.

� Signing a credit card with a name other than what is on the card and telling the credit card company that the purchase was not made by the card owner. This would allow the card owner to disavow the purchase and get the purchase amount refunded.

Information Disclosure

Information disclosure occurs when a user’s confidential data is exposed to individuals against the wishes of the owner of the information. Often times, these attacks receive a great deal of media attention. Organizations like TJ Maxx, Equifax, and the US Department of Veterans Affairs have been involved in the inappropriate disclosure of information such as credit card numbers and personal health records. These disclosures have been the results of both malicious attacks and simple human negligence. Examples include:

� Getting information from co-workers that is not supposed to be shared.

� Someone watching a network and viewing confidential information in plaintext.


Denial of Service

Denial of Service (D.o.S) is another common type of attack involving making service unavailable to legitimate users. D.o.S. attacks can target a wide variety of services, including computational resources, data, communication channels, time, or even the user’s attention. Many organizations, including national governments, have been victims of denial of service attacks. Examples include:

� Getting a large number of people to show up in a school building so that classes cannot be held.

� Interrupting the power supply to an electrical device so it cannot be used.

� Sending a web server an overwhelming number of requests, thereby consuming all the server’s CPU cycles. This makes it incapable of responding to legitimate user requests.

� Changing an authorized user’s account credentials so they no longer have access to the system.

Elevation of Privilege

Elevation of privilege can lead to almost any other type of attack, and involves finding a way to do things that are normally prohibited. In each case, the user is not pretending to be someone else. Instead, the user is able to achieve greater privilege than he normally would have under his current identity. Examples include:

� A buffer overrun attack, which allows an unprivileged application to execute arbitrary code, granting much greater access than was intended.

� A user with limited privileges modifies her account to add more privileges thereby allowing her to use an application that requires those privileges.

In most cases, a single attack can yield other attacks. For example, an attacker able to elevate his privilege to an administrator can normally go on to delete the logs associated with the attack. This yields the following threat tree (figure on the left). In this scenario, an elevation of privilege attack can also lead to a repudiation attack. Figure 02.2 is a graphical way to represent this relation. There are many reasons why threat trees are important and useful tools. First, we can easily see the root attack that is the source of the problem. Second, we can see all the attacks that are likely to follow if the root attack is successful. This is important because often the root attack is not the most severe attack in the tree. Finally, the threat tree gives us a good idea of the path the attacker will follow to get to the high value assets.

Figure 02.2: Simple attack tree


In many situations, the threat tree can involve several stages and be quite involved. Consider, for example, an attacker that notices that unintended information is made available on an e-commerce site. From this disclosure, the attacker is able to impersonate a normal customer. With this minimal amount of privilege, the attacker pokes around and finds a way to change the user’s role to administrator. Once in this state, the attacker can wipe the logs, create a new account for himself so he can re-enter the system at will, sell confidential information to the highest bidder, and shut down the site at will. The complete threat tree for this scenario is the figure on the left. One final thought about threat trees. It is tempting to simply address the root problem, believing that the entire attack will be mitigated if the first step is blocked. This approach is problematic. If the attacker is able to find another way to get to the head of the threat tree through a previously unknown attack vector, then the entire threat tree can be realized. It is far safer to attempt to address every step of the tree. This principle is called “defense in depth.”

Information States The second dimension of the McCumber Cube is the state in which information assets may reside. Though most problems focus on one or two states, it is not uncommon for a problem to span all three states.

Storage

This is also called “data at rest.” Examples include data on a hard disk, data in memory, or even data in a register. Any data that is not currently being used or moved is considered storage. Back to our banking example, this would include money in an account or valuables in a safety deposit box. The vast majority of the world’s data is in storage at any moment in time. Though storage data is certainly easiest to protect, it often holds an organization’s most valuable assets. Therefore, storage strategies must be carefully chosen.

Transmission

Data being moved from one location to another. Network security is primarily concerned about this data state. This may include data moving along a wire or transmitted through Wi-Fi. This brings up the question: is a CD sent through the post in storage or transmission state? In our banking example, this would include money in an armored car moving assets to another bank or even account information being transmitted to an ATM. When an attack involves transmission, it is usually necessary to provide more detail. There are many different parts or phases to transmitting a message from one location to another. The most widely accepted model we use to describe network traffic is the O.S.I. reference model.

Figure 02.3: Somewhat complex attack tree


Processing

The processing state of data occurs when data is currently being used. Processing of money in a bank might include a teller counting money during a deposit or an interest calculation function changing an account balance. Most vulnerabilities occur when data is being processed. This is when code is operating on data transforming it from one state to another. If an attacker can trick the system to perform unintended processing or to make a mistake in processing, then a processing attack may be possible.

Of the three information states (storage, transmission, and processing), transmission is the most complex to protect because it involves so many phases. It is instructive to look at each phase individually when searching for vulnerabilities or designing a system that needs to provide security assurances. These phases or layers are defined by the OSI model. The Open Systems Interconnection (OSI) model was first defined in 1977 in an effort to make it possible for different network technologies to work together (Zimmermann, 1980). These layers are physical, data link, network, transport, session, presentation, and application. Note that many network textbooks compress the OSI 7-layer model into only 5 layers (rolling session and presentation into the application layer). Since these layers perform distinct functions from a security standpoint, the full 7-layer model will be presented here. The OSI model defined a structured way to integrate new ideas and technologies. This is accomplished by defining the behavior of a network based on what layers of processing need to occur for network communication to take place. Each layer will be described and attacks against the information assurance (confidentiality, integrity, or availability) will be discussed.

OSI Layer 1 - Physical The physical layer is the means through which data travels on the network. This includes the medium itself and the mechanism by which a signal is placed on the medium. The most commonly used physical layers include:


Copper Wire Information is passed through electron transfer in the form of electrical current. This is typically accomplished through voltage or current variations. Data transfer is point-to-point: from the location where the electrons are placed on the wire, down the wire itself, to the location where electrons are measured. Copper wire has two properties making it vulnerable to attack: it is easy to splice (thereby making confidentiality assurances difficult) and to cut (making availability assurances difficult). Common uses are CAT-5 cable and coaxial cable.

Fiber Optic Cable Information is passed through photon transfer in the form of light pulses. The medium is glass and signals are passed through the glass with different frequencies of light. Data transfer is point-to-point; it is difficult to splice but easy to cut.

Electromagnetic (EM) waves

Information is passed through air using photon transfer in the form of EM waves at specific frequencies. Data transfer starts at the transmitter and many receivers may listen; it is easy to “splice” (because the signal can be viewed by many observers simultaneously) and “cutting” can occur through jamming. Common frequencies include: 450, 950, 1800, and 2100 MHz for cellular networks, 2400-2480 MHz for Bluetooth, 2.4, 3.6, and 5 GHz for Wi-Fi.

The physical layer is the lowest OSI layer. It provides a medium on which data link connections are made. Note that software engineers rarely concern themselves with the physical layer; electrical engineers work at this layer.

Confidentiality Disclosure attacks on the physical Layer occur when the attacker obtains access to a signal which the defender of the network did not anticipate. If the defender relied on physical security, then the entire network could be compromised when the attacker achieves physical access.

In the 1990’s the NSA discovered an undersea cable connecting a Russian submarine base with the mainland. Because the cable was so inaccessible, the NSA theorized, there would probably not be many other security mechanisms on the network. In conjunction with the U.S. Navy, the cable was found and a listening device was attached to it by a specially modified submarine. For several years, this submarine periodically re-visited the cable to retrieve data captured by the recording device. Eventually the cable was damaged in a storm and, when it was repaired, the listening device was discovered. Ownership of the device was easy to ascertain because a label was printed on the inside cover: “Property of the CIA.”


Another example occurred by utilizing the availability of wireless signals far outside their intended range:

In a Lowe’s home-improvement store in 2003, an employee of the store set up a small Wi-Fi network whose range did not extend beyond the perimeter of the store. Due to this presumably secure physical layer, minimum security was placed on the network. Adam Botbyl and Brian Salcedo, however, discovered the unprotected network and decided to launch an attack. Operating in a Pontiac Grand Prix in the store’s parking lot, they extended the physical extent of the network with a home-made antenna. From this entry point, they quickly defeated the minimal security measures and began stealing credit card data from the store’s customers.

Finally, some thieves are able to trick customers into revealing their credit card number:

A Skimmer is a device placed on an ATM machine or a credit card reader that records the swipe information from a victim’s card. If the victim does not notice the presence of the skimmer, the victim’s card data will be recorded while a valid purchase is made.

Integrity Alteration attacks on the physical layer are rare because the attacker needs to be able to selectively change the composition of a network signal. Typically, integrity attacks occur at the data link, transport, and presentation layers.

Availability There are three forms of denial attacks on the physical layer: blocking a signal, saturating a signal, and stealing access to a network.

Block The simplest example of a blocking denial attack on the physical layer is to cut a fiber optic, coaxial, or CAT-5 cable. This can occur any time the attacker has physical access to the wired component of a network. Other examples of blocking attacks include removing, damaging, or denying power to network hardware. Stealing a Wi-Fi router would be an example of a blocking attack.

Saturating Saturation network attacks occur when the attacker floods the physical layer with bogus signals. This is commonly known as jamming. An attacker can jam a Wi-Fi network by broadcasting large amounts of EM energy at the 2.4, 3.6, and 5 GHz frequencies. Another jamming attack occurs when a collection of rogue Wi-Fi access points is set up in the midst of an authentic wireless network. When all the frequencies are consumed by rogue signals, it becomes difficult for legitimate users to access the network.

Figure 02.4: ATM skimmer (Reproduced with permission

from Aurora Police Department)


Stealing Access A final denial attack of the physical layer occurs when the attacker attempts to get unauthorized access to network resources. For Wi-Fi networks, this process is commonly called wardriving. Wardriving is the process of driving through neighborhoods and local business districts looking for open wireless networks. This typically involves little more than a laptop, a hand-made antenna, and some readily available software.

OSI Layer 2 - Data Link Data link is the means by which data passes between adjacent entities on the network. It is defined as “[T]he functional and procedural means to establish, maintain, and release data links between network entities.” (Zimmerman, OSI Reference Model, 1980) Consider broadcast radio. FM radio (frequency modulation) transmits information through electromagnetic waves by altering the frequency of a carrier wave. AM radio (amplitude modulation) does the same by altering the amplitude or amount of energy transmitted through a given frequency of electromagnetic waves. DAB (digital audio broadcast) does the same by sending discrete pulses of information through electromagnetic waves. Each of these three data link technologies can operate on the same physical layer (frequency of electromagnetic energy) though the way the physical layer is used is different. Some of the services provided at the data link layer include multiplexing (allowing more than one message to travel over the physical layer at a time), error notification (detecting that information was not transmitted accurately), physical addresses (such as a MAC address), and reset functionality (signaling the network connection has returned to a known state). Data link layer attacks are relatively rare for wired networks. This is in part due to the fact that a single entity (typically the owner of the network) completely manages the data link connection. Since that entity both puts data onto the wire and takes it off, there is no opportunity for an attacker to put malware on the wire. However, if it is possible for an attacker to have direct access to a network, then attacks may follow. Each attack would therefore be specific to the exact type of network that is implemented on the data link layer. Note that data link technology is typically developed by computer engineers and electrical engineers, not software engineers.

Confidentiality Disclosure attack on the data link layer are very rare in part because few data link services make confidentiality assurances. For the most part, confidentiality assurances are made in higher layers.

Integrity Alteration attacks vary according to the specifics of the technology used in the point-to-point communication. One example would be a token ring, a network type where messages follow a circular path through a network riding on a carrier signal (token). An alteration attack on a token ring can be accomplished if a rogue network node corrupts the carrier token.


Availability A denial attack can occur on a network where a fixed number of ports or channels are available. A rogue access point could claim all ports or channels thereby denying legitimate traffic from getting on the network. This could occur even if there is plenty of bandwidth still available on the network.

OSI Layer 3 - Network Network layer services fundamentally work with routing. There are several types of routing: broadcast (sending a message to all nodes such as a TV or radio broadcast station), anycast (sending a message to any node, typically the closest), multicast (sending a message to a specific subset of nodes, like an e-mail sent to all currently registered students at the university), and unicast (sending a message to a specific node). For most types of routing to occur, network protocols must contain the destination address of the message and network agents must know how to interpret these addresses to make routing decisions. The most common network protocol is the Internet protocol (IP). The router reads the IP header of an incoming packet and determines which direction to send it so it gets closer to the destination. This requires the router to be familiar with the topology of the network. Note that most network services are off-the-shelf. Designing a network to withstand network attacks is typically the job of an information technology (IT) engineer.

Confidentiality Disclosure attacks at the network layer are the result of routing information of confidential communications being revealed to an attacker. This can occur if a rogue node on a network is able to read the network header. Since the header must be read by all routers in order to perform routing services, this is trivial; both the Source and the Destination fields of the IP header are easily read. One way to mitigate confidentiality attacks on network services is to use a virtual private network (VPN). A VPN is:

A VPN is a communications environment in which access is controlled to permit peer connections only within a defined community of interest, and is constructed through some form of partitioning of a common underlying communications medium, where this underlying communications medium provides services to the network on a non-exclusive basis.

(Braun et al., Virtual Private Network Architecture, 1999)

The idea is to encapsulate a private, encrypted network header in the body of a standard IP packet. While any observer will be able to tell where the packet leaves the Internet (for example, a large company), they will not be able to tell the final destination (for example, the president’s computer). Only those able to decrypt the encapsulated header will be able to tell the final destination. This process is commonly known as “tunneling.”


Integrity Alteration attacks on the network layer occur when an IP header is modified or when IP processes can be altered. Examples of alteration attacks include:

Food Fight The destination address of a packet is changed to match the source address. When the recipient replies to the packet, a message will be sent back to itself rather than to the originator of the packet.

Redirection A message is sent to an innocent bystander with the source address faked and the destination directed to the intended victim. The bystander will then send messages to the victim from the bystander without any evidence of the attacker’s role.

DNS Poisoning The Domain Name Server (DNS) is a network device translating URLs (e.g. www.byui.edu) with IP addresses (e.g. 157.201.130.3). If the DNS mappings between URLs and IP addresses can be altered, then message routing will not occur as the network designer intends.

Attempts are made to address these and other integrity attacks in IPv6 and IPsec. This is accomplished with signing and integrity checks on IP headers.

Availability Nearly all availability network attacks occur at the Domain Name System (DNS). The DNS is a network device that translates URLs to IP addresses. For example, it would convert www.byui.edu to 157.201.130.3. An analogy in the physical world would be the map in a post office indicating the city corresponding to a given zip code. This is an essential part of the Internet architecture: how else can a packet find its way to the destination? The most common denial attacks on the DNS include DNS flood, DNS DoS, and DNS crash. Other denial attacks on the network layer focus on the ability of network routers to correctly direct packets.

DNS Flood A DNS flood is an attack on the DNS servers by inundating (or flooding) them with pointless requests. As the DNS servers respond to these requests, they become too busy to respond to valid DNS requests. As a result, routers are unable to send packets in the direction of their destination.

DNS DoS A DNS DoS attack is an attack on the DoS server itself rendering it incapable of responding to valid DNS requests. Common DNS DoS attacks include attacking the server’s implementation of the TCP/IP stack forcing it to handle requests less efficiently or tricking the server into consuming some other vital system resource.


DNS Crash A DNS crash attack is an attack on the system hosting the DNS causing it or some vital service to shut down. This can be accomplished by exploiting vulnerabilities in the system hosting the DNS, exploiting vulnerabilities on the DNS resolution software itself, or physically attacking the DNS servers. DNS Crash attacks are rare because DNS servers have been rigorously analyzed and are considered hardened against such attacks.

Host Resolution This attack prevents a router from connecting to a DNS and thereby compromises its ability to route packets. This can be accomplished by removing DNS request packets from the network or by altering DNS results.

Smurf A Smurf attack is a network attack where two hosts are tricked into engaging in a pointless high-bandwidth conversation. This occurs when an attacker sends a ping request to a target with the return address being an innocent 3rd party. As the target responds to the 3rd party, the 3rd party responds with a ping of its own. This back-and-forth continues until the channels between the target and the 3rd party are saturated. The result of a Smurf attack is that all traffic routed through these connections is blocked

Modern routers and DNS servers are immune to these and other denial attacks.

OSI Layer 4 - Transport The transport layer ensures the messages traveling across the network arrive intact and using the minimum amount of resources. This is commonly done by subdividing and recombining messages and maximizing the efficiency of the network. This service is commonly called “packet switching.” The most common transport protocol is Transmission Control Protocol (TCP). As with the data link and network layer, most networks are built from off-the-shelf technology configured by I.T. professionals. Thus few software engineers work at the transport layer.

Confidentiality The transport layer does not make confidentiality assurances. Confidentiality of message routing is provided by the network layer. Confidentiality of the message composition is provided at the presentation layer. While the size of the message is revealed in the Window Size and the Sequence Number fields, that information is also available from other sources.

Integrity The TCP protocol provides integrity assurances through the checksum field and the re-send mechanism. If the checksum value does not match the header and the body, then the recipient is to request a retransmission of the message. Because the checksum mechanism is not confidential, it is easy for an attacker to alter a datagram and insert a valid checksum. This would preclude the recipient from knowing of the alteration. Another alteration attack on the transport layer is called the TCP sequence prediction attack. This can occur if an eavesdropper is able to predict the next sequence number in a message transmission. By inserting a new packet with the


forged sequence number, the imposter can insert his datagram into the message without being detected. This attack has been mitigated with proposal RCF 1948 describing sequence numbers difficult to predict.

Availability Denial attacks target the way that messages are created from individual packets. There are two broad categories of attacks: creating a sequence of packets in such a way that the receiver will have difficulty reconstructing them, or hijacking a conversation. The first category of attacks can be implemented in a wide variety of ways, including incomplete connections and reconstruction. The second category involves a rogue network node manipulating packets that pass between the source and destination points. Note that just removing these packets does not guarantee a denial attack; the packets will simply be re-sent on a different path. Denial attacks at the transport layer consist of altering the TCP component of the packets in such a way that the destination becomes satisfied that the entire message has arrived. Two examples of such attacks are the TCP Reset attack and the Stream Redirect attack.

Incomplete Connection Opening a TCP connection but not completing the 3-way handshake can leave the recipient with a partially opened connection. If this is done enough times, then the recipient will lose the capability to begin a normal connection. Another name for this is a “SYN Flood attack” because the 3-way handshake is initiated with a session setup packet with the SYN field set.

Reconstruction Sending packets with overlapping reference frames or with gaps, making it impossible for the recipient to reconstruct the message. This is commonly called a “teardrop” attack.

TCP Reset Attack A rogue network node modifies a packet from the source with the Reset command set. This command serves to terminate the message and denies the recipient from the intended message. This attack can also occur if the rogue network node is able to predict the next sequence number in a packet stream.

Stream Redirect The TCP connection between two points is corrupted by sending misleading SEQ/ACK signals. The attacker then mimics the source’s connection state and assume the source’s end of the conversation. Once this is accomplished, the attacker can terminate the conversation with the destination.

OSI Layer 5 - Session The session layer provides a connection between individual messages between parties so a conversation can exist. This requires both parties to identify themselves, initiate the conversation, associate exchanged messages as being part of the conversation, and terminate the conversation. None of these services is provided by the Hypertext Transfer Protocol (HTTP) standard. As a result, software engineers need to come up with a method to provide these services. There are many examples of session Layer services commonly used on networks today, the most common being SSH (Secure Shell, a tool allowing an individual to remotely connect via the command line to another computer) and HTTPS


(multiple interactions with the server governed by a single login). The latter is accomplished with the cookie mechanism. Cookies are a mechanism built into web browsers in 1994 by Mosaic Netscape for the purpose of enabling session information to pass between the client and the server. The term “cookie” originates from 1987 where a collection of session keys are kept in a “Cookie Jar.” A cookie is defined as:

A token or packet of data that is passed between computers or programs to allow access or to activate certain features; (in recent use spec.) a packet of data sent by an Internet server to a browser, which is returned by the browser each time it subsequently accesses the same server, thereby identifying the user or monitoring his or her access to the server.

(Oxford English Dictionary, 2012)

Cookies consist of three components: name, value, and domain. However it is not uncommon for other attributes to be attached. Servers can send a cookie to a client which the client may choose to accept. If the client accepts the cookie, then subsequent requests to the server will include the cookie information. This allows the server to be able to keep track of conversations involving multiple messages.

Confidentiality A disclosure attack on the session layer involves an eavesdropper being able to recognize that the individual messages sent between the client and the server are part of a larger conversation. This information can be ascertained by the eavesdropper if a cookie is detected. Note, however, that not all cookies are used to maintain session state.

Another example of a disclosure attack is called session hijacking. In this case, an eavesdropper is able to inject himself into the conversation and read private communications.

Integrity An integrity attack on the session layer involves an eavesdropper being able to modify a conversation between a client and a server. This is typically accomplished by the attacker injecting messages in the conversation without the recipient being aware of the forgery. This process is called Session Hijacking, Cookie Theft, Cookie Poisoning, or SideJacking. SideJacking is the process of an eavesdropper capturing or guessing a session cookie being passed between a client and a server. With this cookie, the eavesdropper can send a new request to the server and the server will believe the request came from the authenticated client. The eavesdropper can often guess a cookie if the server does not use a strong random number to represent session state. If, for example, the server uses a simple counter (every subsequent session has a value one greater than the preceding session), then the attacker can guess the cookie by obtaining a valid cookie and generating requests with higher values. Another example would be a cookie being a random 8-bit number. With 256 guesses, the attacker will be able to correctly guess any session cookie and thereby hijack the session. Both of these


attacks can be avoided if the client uses a sufficiently large and random number to represent session state.

Availability An availability attack on the session layer involves making it difficult for the client and the server to maintain a conversation. This can occur if the attacker is able to interrupt the cookie exchange process, destroy the user’s cookie, or make the server unable to recognize the client’s cookie.

OSI Layer 6 - Presentation The presentation layer governs how a message is encoded by the application before being sent on the network. There can be more than one option at the presentation layer for encoding a given type of data. For example, textual data can be encoded as ASCII (one-byte), Unicode (two-byte), or UTF-8 (multi-byte ranging from 1 to 8). Note that formats such as HTML and C++ are built on top of presentation layer formats. In other words, it is possible to encode HTML with ASCII, Unicode, or UTF-8. All encryption mechanisms are also presentation layer services. Since software engineers typically define file formats used to represent user data, the presentation layer is typically a software engineer decision. Because most presentation layer formats are designed to be readily generated and read, they are vulnerable to all three information assurance attacks.

Confidentiality Confidentiality assurances at the presentation layer typically come in the form of encryption algorithms. If a strong encryption algorithm is used correctly and a strong key is chosen, then it is unlikely that the user’s confidential information will be disclosed. For more information on this, please see Chapter 12: Encryption.

Integrity Integrity assurances at the presentation layer typically come in the form of digital signatures. These consist of a hash generated from a message that is encrypted using a private key known only by legitimate authors. If the key is disclosed to others or if it is possible to guess the key, then alteration attacks are possible. For more information on this, please see Chapter 12: Encryption.

Availability If the attacker can alter a message in such a way that the file format (such as PDF) is no longer valid, then a denial attack can be made.


OSI Layer 7 - Application The application layer is the originator of network messages and the ultimate consumer of messages. It is defined as:

[The] highest layer in the OSI Architecture. Protocols of this layer directly serve the end user by providing the distributed information service appropriate to an application, to its management, and to system management.

(Zimmerman, OSI Reference Model, 1980)

An application-layer attack occurs when an attacker targets the recipient of a network communication. This recipient could, of course, be either the client or the server. Examples of programs that may be targets to application-layer attacks are: web browsers, e-mail servers, networked video games, and mobile social applications. Virtually any program can be the target of an application-layer attack. Because of this diversity, discussion will be focused on areas common to most applications: denial attacks.

Application Crash An application crash DoS attack is any attack where a maliciously formed input can cause the recipient to terminate unexpectedly. There are two underlying sources of crash vulnerabilities: a previously unseen defect which the testing team should have found, or an incorrect assumption about the format of the incoming data. The former source is not a security issue per se, but rather a quality issue. Robust software engineering practices can mitigate this type of attack. This cannot be said about the second source: incorrect assumptions. Most software engineers design file and network interfaces with the assumption that data will come in the expected format. While error checking may be built into the process, import code is optimized for the common case (well-formed data) rather than the worst case. Attackers do not honor this assumption. They strive to crash the application by creating input designed to exercise the worst-case scenario.

CPU Starvation A CPU Starvation attack occurs when the attacker tricks a program into performing an expensive operation consuming many CPU cycles. Some algorithms, like sorting or parsing algorithms, can make an application vulnerable to this kind of DoS attack. CPU Starvation attacks typically occur when the developer optimizes on the common case rather than the worst case. Attackers leverage this mistake by creating pathologically complex input designed to exercise this worst case. As the server responds to this input, it becomes difficult to respond to legitimate input from the user. One CPU Starvation mitigation strategy is to avoid designing protocols that are cheap for the attacker to produce but expensive for the recipient to consume. Network protocols and file formats should be designed with size, ease of creation, and ease of consumption in mind. If the latter is not taken into account, opportunities exist for CPU Starvation attacks. One disclaimer: often software engineers are not given the luxury of specifying communication protocols.


Memory Starvation Memory starvation occurs when the demands on the memory allocator degrade performance or cause a program to malfunction. The attacker simply needs to trick the program into doing something that would degrade memory, such as successive new/delete statements or make the target application consume all available memory. Memory starvation attacks can be difficult to mitigate because it is difficult to tell exactly how memory usage patterns are tied to network input. However, the following guidelines are generally applicable: Avoid dynamic memory allocation. Stack memory is guaranteed to not fragment. Heap memory management is much more complex and difficult to predict. Fregmentation is a leading cause of memory starvation. Use the minimal amount of memory. Though it is often convenient to reserve the maximal amount of memory available, this can lead to memory starvation attacks. Consider writing a custom memory management system. Generic memory management systems built into the operating system and most compilers do not leverage insider knowledge about how a given application works. If, for example, it is known that all memory allocations are 4k, a custom memory manager can be made much more efficient than one designed to handle arbitrary sizes. See Appendix F: The Heap for details as to how this can be done.


Examples

1. Q Is it possible to have an attack without a threat?

A An attack is defined as a risk realized. A risk is defined as vulnerability paired with a threat. Therefore, without a risk, there can be no attack. This makes sense because you cannot have an attack if there does not exist a way for an asset to be devalued by an attacker.

2. Q Can there be a vulnerability without an asset?

A A vulnerability is a weakness in the system. The existence of this weakness is independent of the presence of an asset. For example, you can have a safe with a weak combination. This vulnerability exists regardless of whether there is anything in the safe.

3. Q What reasons might exist why there might be a risk but no attack?

A This happens whenever an attacker has not gotten around to launching an attack. Presumably the attacker has motivation to launch the attack (there is an asset, after all) and has the ability to launch the attack (there is a vulnerability after all). However, for some reason, a motivated and qualified attacker has not discovered the risk or has had more pressing things to do.

4. Q What are the similarities and differences between denial of service and information disclosure?

A From the C.I.A. triad, denial of service is a loss of availability while information disclosure is a loss of confidentiality. They represent completely different assurances.

5. Q Identify the threat category for the following: A virus deletes all the files on a computer.

A Denial of Service because the user of the files no longer has access to them.

6. Q Identify the threat category for the following: A student notices her Professor’s computer is not locked so she changes her grade to an 'A'.

A Tampering because the correct grade has been changed contrary to the user’s wish.

7. Q Identify the threat category for the following: The first day of class, a student walks into the classroom and impersonates the professor.

A Spoofing because the student pretends he is the teacher.


8. Q A certain user is conducting some online banking from her smart phone using an app provided by her bank. Classify the following attack according to the three dimensions of the McCumber cube: An attacker intercepts the password as it is passed over the Internet and then impersonates the user.

A � Type: Confidentiality; the asset is the password and it is meant to be private.

� State: Transmission; the password is being sent from the phone to the cellular tower.

� Protection: Technology; it is the software encrypting the password which is the weak link.

9. Q A certain user is conducting some online banking from her smart phone using an app provided by her bank. Classify the following attack according to the three dimensions of the McCumber cube: An attacker convinces the user to allow him to touch the phone. When he touches it, he breaks it.

A � Type: Availability; the phone can no longer be used by the intended user.

� State: Storage; at the time of the attack, all the data is at rest.

� Protection: Policy; the user should, by policy, not let strangers touch her phone.


10. Q A certain user is a spy trying to send top secret information back to headquarters. Can you list attacks on this scenario involving all the components of the McCumber cube?

A � Confidentiality: the attacker can learn the message and send it to the police.

� Integrity: the attacker can intercept the message and set in misleading intelligence.

� Availability: the attacker can block the message by destroying the spy’s communication equipment.

� Storage: the attacker can find the message in the spy’s notebook and disclose it to the police.

� Transmission: the attacker can intercept the message as it is being sent and alter it in some way.

� Processing: the attacker can disrupt the ability of the spy to encrypt the message by distracting him.

� Technology: the attacker can find the transmission machine and destroy it.

� Policy: the attacker can learn the protocol for meeting other spies and impersonate a friend.

� Training: the attacker can get the spy drunk thereby making him less able to defend himself.


11. Q Consider a waitress. She is serving several tables at the same time in a busy European restaurant. For each O.S.I. layer, 1) describe how the waitress uses the layer and 2) describe an attack against that layer.

A � Application: The application is getting the correct food to the correct customer in a timely manner. An attack would be to order a combination that the restaurant would be unable to fulfill. For example, a patron can order a hundred eggs knowing that they only have less than 50 in stock.

� Presentation: The presentation layer is the language in which the patron is conversing. Since this is a European restaurant, many languages are probably being spoken. An attack would be for the patron to recognize the dialect and make inferences about the background of the waitress which the waitress may choose to keep confidential.

� Session: The session layer is the conversation occurring over the course of the hour involving many interactions between the waitress and the patron. A session attack would be for one patron to impersonate another patron and thereby place an undesirable order on the victim’s bill.

� Transport: Transport is the process of providing throughput and integrity assurances on a single communication between the waitress and the patron. If the waitress did not understand what was said, she would say “what?” An attack on the transport layer would be for the attacker at a nearby table to keep yelling words, thereby making it difficult to verify what the victim was saying.

� Network: The network layer provides routing services. In this case, the waitress needs to route food orders to the cook and service concerns to the manager. A network attack would be to intercept the order sheet from the waitress before it reaches the cook, thereby making it impossible for the cook to know what food is being ordered.

� Data Link: The data link is the direct one-time communication between the waitress and the patron. In this case, they are using the spoken word. An attack on the data link layer might be to cover the waitress' ears with head phones. This will prevent her from hearing the words spoken by the patrons.

� Physical: The physical layer is the medium in which the communication is taking place. In this case, the medium is sound or, more specifically, vibrating air. An attack on the physical layer might be to broadcast a very large amount of white noise. This would make it impossible for anyone to hear anything in the restaurant.


12. Q Consider the following scenario: A boy texting a girl in an attempt to get a date. Identify the O.S.I. layer being targeted: A large transmitter antenna is placed near the boy’s phone broadcasting on the cellular frequency.

A Physical because the medium in which the cell phone communicates with the cellular tower is saturated.

13. Q Consider the following scenario: A boy texting a girl in an attempt to get a date. Identify the O.S.I. layer being targeted: The girl’s friend steals his phone and interjects comments into the conversation.

A Session because the conversation is hijacked by an attacker.

14. Q Consider the following scenario: A boy texting a girl in an attempt to get a date. Identify the O.S.I. layer being targeted: The boy sends an emoji (emotional icon such as the smiley face) to the girl but her phone cannot handle it. Her phone crashes.

A Application, specifically Application Crash.

15. Q Classify the following as Broadcast, Anycast, Multicast, or Unicast: BYU-Idaho’s radio station is 91.5 FM, playing “a pleasant mix of uplifting music, music, BYU-Idaho devotionals, LDS general conference addresses, and other inspirational content.”

A Broadcast because it is from one to many.

16. Q Classify the following as Broadcast, Anycast, Multicast, or Unicast: I want to spread a rumor that shorts will be allowed on campus. This rumor spreads like wildfire!

A Anycast. People randomly talk to their nearest neighbor who then passes it on.

17. Q Classify the following as Broadcast, Anycast, Multicast, or Unicast: I walk into class and would like a student to explain the solution to a homework assignment. I start by asking if anyone feels like explaining it. Then I walk over and have a short conversation with the volunteer.

A Multicast because this is a one-to-unique situation.

18. Q Classify the following as Broadcast, Anycast, Multicast, or Unicast: A boy walks into a party and notices a pretty girl. He walks directly to her and they talk throughout the night. They talk so much, in fact, that they do not notice anyone else.

A Unicast because they had a one-to-one conversation.


19. Q Consider the telephone system before there were phone numbers (1890-1920). Here you need to talk to an operator to make a call. In this scenario a grandchild is going to tell her grandma how her basketball game went. For each O.S.I. layer, describe how the telephone implements the layer.

A Physical: Twisted pair of insulated wires, electrical current, ringer, hookswitch, earphone (receiver), and microphones (transmitter). Data Link: Initiation of a call is made through an AC 75 volt signal. A currently active call is made using a DC signal of less than 300 ohms, and zero current indicates that the line is not being used. Network: Routing performed manually through operators following a policy. The routing information is spoken over the phone to the operator. Transport: Error correction/detection is performed manually (“Huh? Repeat that!”) through training. Bandwidth sharing is performed socially through training, namely by not interrupting another speaker and by not dominating the conversation through speaking too much. Session: Begins with first a “thumper” (predecessor to the ring), then “Hello, is Mary there?” and ends with “Good bye!” Both parties maintain the state of the conversation through policy (social etiquette). Presentation: The voice message is encoded using Amplitude Modulation (AM) plus the language of the speaker (English). Note that other possible presentations are Morse Code and digital signals through modems. Application: Grandma and grandchild talking about a basketball game.


Exercises

1 From memory, please define the following terms: asset, threat, vulnerability, risk, attack, mitigation.

2 What is the difference between a threat and a vulnerability?

3 What is the relationship between a risk and an attack?

4 From memory, list and define S.T.R.I.D.E.

5 What are the similarities and differences between tampering and repudiation?

6 What are the similarities and differences between spoofing and elevation of privilege?

7 Identify the threat category for each of the following. Do this according to the state of the asset (including the O.S.I. layer if the asset is in transmission), type of assurance the asset offers (using S.T.R.I.D.E.), and the type of vulnerability necessary for an attack to be carried out:

� A program wipes the logs of any evidence of its presence.

� SPAM botnet software consumes bandwidth and CPU cycles.

� Malware turns off your virus scanner software.

� A telemarketer calls you at dinner time.

� After having broken into my teacher’s office, I wiped my fingerprints from all the surfaces.

� I was unprepared for the in-lab test today so I disabled all the machines in the Linux lab with a hammer.

� I have obtained the grader’s password and have logged in as him.

� I changed the file permissions on the professor’s answer key so anyone can view the contents .

� I have intercepted the packets leaving my teacher’s computer and altered them to reflect the grade I wish I earned.

� The teacher left himself logged in on E-Learn so I changed my role from “student” to “grader”.

8 Define and explain C.I.A. in terms a non-technical person would understand. Explain why each assurance is essential for a system to provide.

9 What do you call an attack on confidentiality? on integrity? on availability?


10 A certain user keeps his financial data in an Intuit Quicken file on his desktop computer at home. For each of the following problems, classify them according to the three dimensions of the McCumber cube:

� An attacker standing outside the house shoots the computer in the hard drive.

� An attacker tricks the software to accept a patch which sends passwords to his server.

� An attacker breaks into the house and installs a camera in the corner of the office, capturing pictures of the balance sheet to be posted on the Internet.

� An attacker intercepts messages from the user’s bank and changes them so the resulting balance will not be accurate.

� An attacker convinces a member of the household to delete the file.

� An attacker places spyware on the computer which intercepts the file password as it is being typed.

11 Consider the following scenario: An executive uses a special mobile application to communicate with his secretary while he is on the road. Can you list attacks on this scenario involving all the components of the McCumber cube?

� Describe three attacks - one for each of the three types of assets (C.I.A.).

� Describe three attacks - one for each of the three information states (storage, transmission, and processing).

� Describe three attacks - one for each of the three protection mechanisms (technology, policy & practice, and training).

12 From memory, list and define each of the O.S.I. layers. What service does each layer provide?


13 Consider the following scenario: an author and a publisher are collaborating on a cookbook through the conventional mail. Unfortunately, an attacker is intent on preventing this from happening. For each of the following attacks, identify which O.S.I. layer is being targeted.

� Have more than one post office report as representing a given zip code.

� Introduce a rogue editor replacing the role of the real editor, yielding inappropriate or misleading instructions for the author.

� Remove the mailbox so the mailman cannot deliver a message.

� Change the sequence of messages so that the reconstructed book will appear differently than intended.

� Inject a rogue update into the conversation yielding an unintended addition to the book.

� Translate the message from English to French, a language the recipient cannot understand.

� Change or obscure the address on the side of the destination address.

� Immerse the mailbag in water to deny the recipients their data.

� Terminate the conversation before it is completed.

� Alter the instructions so the resulting meal is not tasty to eat.

� Adjust the address on an envelope while it is in route.

� Fill the mailbox so there is no room for incoming mail.

� Harm the user of the book by adding instructions to combine water and boiling oil.

� Remove the stamp on a letter so the mailman will return a message.

� Remove one or more messages, thereby making it impossible to reconstruct the completed work.

14 Consider a coach of a high school track team. Due to weather conditions, the coach needs to tell all the runners that the location of practice will be moved. He has several routing options at his disposal. For each routing option, describe how it would work in this scenario and describe an attack.

� Broadcast

� Anycast

� Multicast

� Unicast


Problems

1 Consider the telegraph (you might need to do some research to see how this works). In 1864, the Nevada Constitution was created and telegraphed to the United States Congress so Nevada could be made a state before the upcoming presidential election. For each O.S.I. layer:

1. Describe how the telegraph implements the layer.

2. Describe an attack against that layer. 3. Describe how one might defend against the above attack.

2 Consider the telephone system before there were phone numbers (you might need to do some research to see how this works). Here you need to talk to an operator to make a call. For each O.S.I. layer:

1. Describe how the telephone implements the layer.


3 Consider the SMS protocol for texting (you might need to do some research to see how this works). Here you are having a conversation with a classmate on how to complete a group homework assignment. For each O.S.I. layer:

1. Describe how texting implements the layer.


4 Consider an ATM machine in a vestibule of a bank. 1. Identify as many assets as you can according to Storage / Transmission

/ Processing. 2. Identify as many security measures as you can according to Policy /

Technology / and Training. 3. Identify as many attacks as you can according to S.T.R.I.D.E.

5 Schneier claims in the following article that finding security vulnerabilities is more an art than a science, requiring a mindset rather than an algorithm. Do you agree? Do you know anyone possessing this security mindset?

Schneier. (2008). Inside the Twisted Mind of a Security Professional.

6 What is the “Unexpected Attack Vector?” If we focus our security decisions on known attack vectors (such as those described by the McCumber model), are we at risk for missing the unexpected attacks? Granneman. (2005, February 10). Beware (of the) Unexpected Attack Vector. The Register.

7 Is a war-driver a white hat or a black hat?


8 While it is clearly illegal to steal a physical asset that resides inside another’s residence, it is also illegal to steal that asset if it causes the victim no harm (in fact, if he doesn't notice) and if that asset extends into public property. Clearly, there is nothing wrong with sitting outside a bread store and stealing the aroma of fresh bread. Is stealing wireless wrong?

https://www.routledge.com/Inside-the-Dark-Web/Ozkaya-Islam/p/book/9780367236229

https://www.routledge.com/Inside-the-Dark-Web/Ozkaya-Islam/p/book/9780367236229

3

Chapter 1

Introduction to Cybersecurity and Dark Web

IntroductionThe digital universe is huge, and the internet and World Wide Web (WWW) are much bigger than what we see through our regular browsing. The internet and its users are rapidly growing due to emerging applications of information technology (IT), and it is expected to continue to grow. However, the rapid growth of inter-net has left it susceptible to misuse and abuse which becomes significant threat and challenge in cyberspace around the globe. A big number of cybercriminals are trying to make illicit attempts every day to gain the access of unauthorized data through the internet. The majority of internet users are accessing the web through normal browsers such as Microsoft Internet Explorer, Mozilla Firefox, Google Chrome, and Apple Safari. The web accessed by a normal browser is called as sur-face web; however, a large part of the content remains hidden in the deep website. According to the literature, the modern search engines index only a very small part of the web, and a large amount of the web contents are concealed as it is in the deep website. The term dark web is part of the deep web which is targeted by the major-ity of cybercriminals, and they are doing criminal activities within the dark site of the web which is called dark net. This chapter will describe the following:

◾ Explanation of cybersecurity and cybercrime◾ Web and its categories◾ Dark web and its terminologies

4 ◾ Inside the Dark Web

◾ Origins of dark net◾ Dark web software.

Cybersecurity and CybercrimeThe expansion of internet has created a great opportunity among the users in different domains, for instance, academic, government, business, and industry sectors. However, this growing development has also created the opportunity to exploit vulnerabilities to attack the infrastructure and system, conduct espionage, and wage cyberwar. Therefore, the cyberspace needs to ensure the users are secure so that they can protect their privacy and safety in the cyber world. The term cybersecurity has been adopted by government and industry, and is understood as the process by which computer net-works and databases of national interest are protected, for example, large corporations and government agencies including civilian, military, and law enforcement.

CybersecurityThe term cybersecurity, also known as computer security and IT security, refers to the technologies, processes, and practices to safeguard from unwanted access of software, hardware, data, program, and intellectual property by cybercriminals over the internet. It is also considered to control the physical access of hardware and/or cyber physical infrastructure. Cybersecurity also refers to protect data from exfiltration, various code injection attacks (CIA) such as SQL injection and XSS, or any type of service disruption.

According to the International Telecommunications Union (ITU), cybersecu-rity is defined as

the collection of tools, policies, guidelines, risk management approaches, actions, training, best practices, assurance and technologies that can be used to protect the cyber environment and organization and user’s assets. Organization and user’s assets include connected computing devices, users, applications, services, telecommunications systems, and the totality of transmitted and/or stored information in the cyber envi-ronment. Cybersecurity ensures the attainment and maintenance of the security properties of the organization and user’s assets against relevant security risks in the cyber environment. The security properties include one or more of the following: availability, integrity, and confidentiality.

CIA (confidentiality, integrity, and availability) is the basic principles of cybersecu-rity. Confidentiality means that the information which is classified or sensitive must remain so and be shared only with appropriate users. Availability means that the information and systems must be available to those who need it. Integrity means that the information must retain its reliability and not be altered from its original state.

Introduction to Cybersecurity and Dark Web ◾ 5

Cybersecurity includes different elements such as application security, informa-tion security, network security, and disaster recovery. Cybersecurity includes differ-ent activities and operations aiming at the reduction and prevention of threat and vulnerabilities and having in place policies for protection, incident response, recov-ery, data assurance, law enforcement, and military and intelligence operations relat-ing to cyberspace security. It defends the systems from hacking and virus attacks.

Consequently, cybersecurity has had growing importance in the cyber world due to the increasing reliance on computer systems, smart devices, wireless net-works such as Bluetooth and Wi-Fi, and the growth of the internet. Cybersecurity involves protecting the information and systems we rely on every day—whether at home, office, or business.

Cybersecurity touches practically all activities and all citizens around the globe; it provides tremendous opportunities for enhancing human development as well as achieving better integration in the information society. It also supports wider access to knowledge and education, as well as to the development of policies and strategies.

In legal and regulatory institutions, the lack of cyberspace security undermines the realization of the full potential of the IT revolution. Consequently, special attention is needed to prevent cyberspace from turning into a source of danger for states and citizens, and to prevent the appearance of a cybercrime haven. The pre-vention of cybercrime is a key objective of cybersecurity.

CybercrimeAccording to Interpol definition, the cybercrime refers as follows:

Cybercrime is a fast‐growing area of crime. More and more criminals are exploiting the speed, convenience, and anonymity of the Internet to commit a diverse range of criminal activities that know no borders, either physical or virtual

– Interpol

Cybercrime is used most often by social scientists and is understood as the process by which criminals target computers or use computers as tools in the commission of a crime. The emphasis is on the offender and the victim. This focus on individual criminals and offenders means an expansive exploration of both crimes, including romance scams, online fraud schemes, cyberbullying, and online extremism.

Cybercrime or computer crimes are “offences against confidentiality, integrity and availability of computer data and systems” and “computer-related offences,” not lim-ited to computer-related forgery, intentional illegal computer system access, intentional illegal interception of computer data transmission, intentionally interfering with data without approval, systems interference, and misuses of electronic or computer devices.

Cybercrime costs billions of dollars to business during cyberattacks that cause direct damage and continue disrupting the business operations after the attack.


Due to the financial loss and business disruption, there are more targeted efforts to control the attacks. In addition, why the efforts are becoming more targeted is because our consumer lives are mostly online and a significant portion of attacks are difficult to detect.

In recent years, the cybercrime efforts are increasingly becoming more targeted by means of the time and cost of performing an attack versus the payback.

According to Australian cybercrime online report, the term cybercrime refers to “crimes which are directed at computers or other devices, and where computers or other devices… are integral to the offence.” This definition broadly defines the types of activities performed by cybercriminals. Their operations either target spe-cific computer networks by developing and deploying various forms of malicious software (such as viruses) or exploit these networks to further their own criminal agendas (phishing, identity theft, fraud, recruitment, etc.). The security experts are working hard to protect cyberspace from the growing cyberattacks including deliberate attempts. Therefore, cybersecurity is an important area that is needed to safeguard the details of internet users.

There has been a dramatic growth of malicious activities within the cyberspace. It is a major concern that thousands of new sophisticated malware and spam are released in an attempt to damage computer systems, or steal or destroy their data (Table 1.1).

Table 1.1 Differences between the Concepts of Cybersecurity and Cybercrime

Cybersecurity Cybercrime

1 Applied science-oriented coding and engineering strategies for making networks more secure

Pure science-oriented theoretical understandings of how and why crimes are committed

2 Science, technology, engineering, and mathematics (STEM) disciplines in particular computer science, computer engineering, IT

Social science disciplines: criminology, psychology, sociology

3 The primary law enforcement bodies are federal

The primary law enforcement bodies are state and local

4 The victims of interest are government and corporate networks

The victims of interest are individuals

5 The crimes are more focused on computer network; software and/or hardware is the target (malicious software, code injection, XSS, DoS attacks)

The crimes are more focused on computer as tool (identity theft, romance scams, cyberbullying, fraud)

6 Corporations and governments Families and individuals


Web and Its LevelsThe term dark web is part of the WWW and known as invisible/hidden web. The content on the dark web remains hidden and cannot be searched through con-ventional search engines. The content only exists on personal encrypted networks or peer-to-peer configurations, and it is not indexed by typical search engines. Therefore, the large part of the internet that is inaccessible to conventional search engines is known as deep web (invisible web). Everyone who uses the web virtually visits what could be reflected as deep websites on a daily basis without being aware.

The deep web is the anonymous internet where it is much difficult for hackers, spies, or government agencies to track internet users and have a look on which web-sites they are using and what they are doing there.

Web LevelsThere are various levels of deep web; for instance, the lower level (level 1) is gener-ally comprised of the “open to public” part of the web, and the upper level (called level 5) is known as dark web which is not accessible by normal web browser and needs to get The Onion Router (Tor) network or some other private network. The following table gives a brief understanding of the level of dark web:

Table 1.2 Dark Web Levels

Level 1 Common web

Level 2 Surface Web Reddit DiggTemp email services

Level 3 Bergie WebGoogle locked resultsHoney portsFreehive, Bunny Tube, etc.

Level 4 Charter Web Hacking GroupsShelling Networking AI theoristBanned videos, books, etc.

Level 5 Onion sitesHuman trafficking, bounty hunters, rare animal tradeQuestionable materialsExploits, black markets, drugs

You can learn more about dark web at: Parker A. et al., Introduction to Deep Web, IRJET, V 4, I 6, 2017.


Web Categories

This section describes the three different levels of web such as the public web, the deep web, and the dark web.

Public web: It typically refers to the unencrypted or non-dark net. This tradi-tional WWW has relatively low-base anonymity, with most websites rou-tinely identifying users by their IP address.

Deep web: It refers to internet content that is not part of the surface web. This means that instead of being able to search for places, you have to visit those directly. They’re waiting if you have an address, but there aren’t directions to get there. The internet is too large for search engines to cover completely; thus, deep web is largely present. The deep web generally mentions the web pages which are invisible by traditional search engine.

Dark web: It is part of the WWW and part of the deep web which can only be accessible by specific software, configurations, or authorization, often using nonstandard communication protocols and ports. The Onion Router is used to access the dark website which is called Tor network.

The following figure shows the differences between deep web, dark web, and internet.

Figure 1.1 Differences Between Deep Web, Dark Web, and Internet.


Dark NetThe term dark net is part of dark deep web, and it is a collection of networks and technologies used to share digital content. The dark net is hidden from the users who use to surf with normal or standard browser, and it is also hiding the web address and server locations. The following table shows the difference between sur-face web, deep web, dark web, and dark net.

The Implication of the Dark Web Crime

Security in the dark web is crucial for building confidence and security in the use of information technologies so as to ensure trust by the information society. Lack of security in cyberspace undermines confidence in the information society. This is especially the case with many intrusions around the globe resulting in the stealing of money; assets; and sensitive military, commercial, and economic information. With information flowing through boundaries of different legal systems connected to different networks around the globe, there is a growing need to protect personal

Table 1.3 Differences Between the Surface Web, Deep Web, Dark Web, and Darknet

Surface Web Deep Web Dark Web Dark Net

Description Content that search engine can find

Content that search engine cannot find

Content that is hidden intentionally

–

Known as Visible web, indexed web, indexable web, lightnet

Invisible web, hidden web, deep net

– Underbelly of internet

Constitutes Web Web Web Network

Contents Legal Legal + illegal Illegal Illegal

Information Found

4% 96% – –

Browser Google Chrome, Mozilla Firefox, Opera, etc.

– Tor Browser Freenet, Tor, GNUnet, I2P, OneSwarm, RetroShare


information, funds, and assets, as well as national security. As a result, cybersecu-rity is gaining interest by both the public as well as the private sectors.

With the emerging applications of computer and IT, cybercrime has become a significant challenge all over the world. Thousands of cybercriminals attempt every day to attack against computer systems to illegally access them through the inter-net. Hundreds of new computer viruses and spam are released every month in an attempt to damage computer systems, or steal or destroy their data. Such threats are expensive, not only in terms of quantity but also in terms of quality. In recent years, experts are becoming more concerned about protecting computer and communica-tion systems from growing cyberattacks including deliberate attempts to access the computer systems by unauthorized persons with the goal of stealing crucial data; to make illegal financial transfers; to disrupt, damage, or manipulate data; or execute any other unlawful actions.

As computer security has advanced, maintaining network persistence has grown harder. As per Australian Cyber Security Centre (ACSC) report, the culture has adapted to this environment, focusing on low-risk, high-reward targets to achieve their goals, with a focus on the development of social engineering methodologies to implement new attacks.

Further to this, the ubiquitous nature of the internet has allowed these nefari-ous individuals to gain increasingly detailed profiles of individuals through exploi-tation and analysis of their digital footprints. This has resulted in higher rates of spear-phishing attacks, identity theft and fraud, and the development of highly specialized malware tools.

There are many risks and pitfalls in cybersecurity incident that can seriously affect computer and network systems. It can be due to improper cybersecurity controls, man-made or natural disasters, or malicious users. The following section mentions some major incidents in the cyberspace.

Ransomware

The ACSC report indicates that the ransomware is a type of post-exploitation denial-of-service (DoS) attack that uses malware to stop legitimate network user(s) from accessing the content of their device(s) or the system. It is typically accom-panied by a request for payment (ransom demand) from the victim, to unlock compromised computer. Modern forms of ransomware use encryption to encrypt the data on compromised systems. This forces the victim to pay the ransom, in exchange for the decryption key.

Payment is typically demanded through Bitcoin or other cryptocurrencies due to the anonymity associated with its use. The instructions are released through a “ransom message” that explains how the payments are to take place. The adversary typically demands monetary payments according to what the targeted company or individual can afford. This information is gained through the scanning phase of the compromise. Some forms of ransomware also include payment increases to encourage


the prompt delivery of the funds into their respective accounts. It is important to note that “paying the ransom doesn’t always result in the data being unlocked” (Table 1.4).

Malware, Worms, and Trojan Horses

These spread by email, instant messaging, malicious websites, and infected non-malicious websites. Some websites will automatically download the malware with-out the user’s knowledge or intervention. Other methods will require the users to click on a link or button.

Table 1.4 Recent Ransomware Variants

Ransomware Type Summary

WannaCry It can self-propagate through a network through the exploitation of vulnerabilities within Windows operating systems (released by the Shadow Brokers). This type of ransomware encrypts “176 different file types and appends. WCRY to the end of the file name”. Once complete, it issues a ransom letter to the victim asking for payment via Bitcoin

Crysis Self-propagation properties allow for reinfection of victim’s machines if any compromised devices remain within the network. Interestingly, AV firm ESET has developed a decryption algorithm which will restore the data affected by the Crysis strain of ransomware

PETYA Utilizes the same vulnerability as WannaCry; however, the compromise occurs through an M.E Doc software update. This software is integrated into Ukrainian Government systems. PETYA overrode the master boot record on the system, allowing the adversary to gain command and control of the device

SAMSAM Ransomware targeted at hospitals and other healthcare systems. It exploits unpatched servers and uses these to travel through the network, compromising each machine. SAMSAM encrypts the data on the compromised system

Locky “Widely distributed by spam as a macro attachment”. The latest file extensions are renamed “.diablo6” or “.lukitis”. This type of ransomware is primarily distributed through spam emails. The emails contain “a malicious Microsoft Office file or a ZIP attachment”


Botnets and Zombies

A botnet, short for robot network, is an aggregation of compromised comput-ers that are connected to a central “controller.” The compromised computers are often referred to as “zombies.” These threats will continue to proliferate as the attack techniques evolve and become available to a broader audience, with less technical knowledge required to launch successful attacks. Botnets designed to steal data are improving their encryption capabilities and thus becoming more difficult to detect.

Distributed Denial-of-Service Attack

The distributed denial-of-service (DDoS) refers to an attack that successfully pre-vents or impairs the authorized functionality of networks, systems, or applications by exhausting resources.

Scareware

It’s a fake security software warning. This type of scam can be particularly profit-able for cybercriminals, as many users believe the pop-up warnings telling them their system is infected and are lured into downloading and paying for the special software to “protect” their system.

Social Network Attacks

Social network (SN) attacks are major sources of attacks because of the volume of users and the amount of personal information that is posted. Users’ inherent trust in their online friends is what makes these networks a prime target. For example, users may be prompted to follow a link on someone’s page, which could bring users to a malicious website.

Key Hitches

The Key Hitches are considered one of the most challenging threats in cyberspace which can be a rapidly and constantly evolving nature of security risks; therefore, the security experts can keep up with it. Furthermore, it imposes new types of commercial, professional, and social paradigms, giving rise to a number of legal and technical problems that must be addressed on the basis of respecting its special nature and needs. Hence, a different approach and different methodologies than what has been adopted are needed.

Cybersecurity has never been simple. And because attacks evolve every day as attackers become more inventive, it is critical to properly define cybersecurity and identify what constitutes a good cybersecurity.


Categories of Crime

This section gives some overview of various cybercrimes and their attack pattern. At the outset, the cybercrimes are broadly categorized into three broad categories, specifically crime against a(n):

◾ Individual level: This type of cybercrime can be in the form of cyberstalk-ing, distributing pornography, trafficking, “grooming,” etc.

◾ Property level: This type of cybercrime steals a person’s bank details and siphons off money, misuses credit card details to make numerous purchases online, runs a scam to get naive people to part with their money, uses malicious software to gain access to an organization’s website, or disrupts the systems of the organization.

◾ Government level: Although not as common as the other two categories, crimes against a government are referred to as cyber terrorism. If successful, this category can wreak havoc and cause panic among the civilian popula-tion. In this category, criminals hack government and military websites, or circulate propaganda. The perpetrators can be terrorist outfits or unfriendly governments of other nations.

◾ Cybercriminals through SN: Social media has become an integral part of daily life. Intruders can find this technology an attractive vehicle to com-mit cybercrimes. Cybercriminals use this platform and exploit it to steal user credentials, identity, and other classified information. There are various steps and methods used to carry out cyberattacks on SNs:– Target: Cybercriminal communicates with an individual via a social media

outlet. The message passing through SN contains a link to a fraudulent web-site or an attachment which initiates an installation file. The most frequently targeted social media networks are Facebook, Twitter, and LinkedIn.

– Infect: Cyberattackers use malware payloads to infect a user’s computer or network. Types of malware include Trojan horse, BotNet, or FakeAV. In the past, pop-up ads and attachments containing viruses were the primary methods of delivering malware. Sophisticated techniques are now used to compromise legitimate websites in order to spread malware through holes in a user’s OS.

– Attack: Armed with a network of “zombie” machines, perpetrators can attack an enormous scale or target specific individuals. The attacks can hypothetically be carried out on any individual, corporate office, govern-ment, or online retailer connected to the internet. The attacks can also persist for long periods of time, as proxy connections and IPs can be constantly changed by the attacker.

– Control: The infected machine, commonly defined as a “zombie,” con-tacts a public server that the attacker has set up as a control plane to issue commands. The infected machine will first be controlled to recruit other machines using the same process of scanning for vulnerabilities.


Malicious Activities in the Dark WebThe malicious activities in the dark web are considered one of the serious threats. Through this malicious activities, the intruder is either trying to disrupt the normal computer operation or gather sensitive information from private computer systems. Antivirus (AV) vendors are receiving a huge number of distinct malware samples per day.

This problem is very important as the proliferation and exponential increase of malware has continued to present a serious threat to the security of information systems. Furthermore, with the development of ever more sophisticated methods of evading detection, malware has posed serious challenges to combat it. Moreover, due to the continuous changes in malware design, anti-malware (AM) strategy that has been successful in a given time period will not work at a much later date.

Taxonomy of Malware

With the rapid development and popularity of the internet, malware has become more and more complicated and evolved into more and more types, from the very first appeared virus, worms, Trojan followed by the currently notorious rootkits. Here I attempt to clarify the meaning of each of these terms to help understand what they are and their potential dangers. The following figures show the taxonomy of malware.

Challenges of Malware in Cyberspace

The malware is now the biggest challenge among the cyber community due to their unpredictable and sophisticated attack patterns. For instance:

Enterprise networks: The enterprise networks are challenged by the growing sophistication and obfuscation technique of malware with unknown threats.

Figure 1.2 Taxonomy of Malware.


Financially motivated attacks: Cybercriminals commonly take advantage of exploits in web browsers, plug-ins, and document readers to compromise end users with Remote Access Trojans (RATs). RATs allow hackers to maintain access while evading detection and compromise sensitive user information such as bank account credentials. Ransomware attacks encrypt all user files. To obtain the key needed to decrypt the files, the victim must pay a ransom to recover their data. The success of ransomware attacks seems to indicate that they will increase over time.

Email: Email-based malware is used to deliver an executable code to the host and then replicate it out. Today, social engineering is going into email deliver-ies, such as embedding compromised documents in encrypted zip files while providing the password in the email, or disguising the file type, such as mak-ing an executable file look like a PDF file.

Mobile malware: Mobile malware attacks are general information-stealing malware, which tracks what you are doing, gets your credentials, or hijacks transactions, including financial ones. Mobile devices introduce security risks when they are used to access company resources; they easily connect with the internet and third-party cloud services, and with computers with security postures that are potentially unknown and outside of the enterprise’s control.

Cloud network: The term cloud network is actually considered as cloud-based network, is a combination of WAN (wide area networking) or internet-oriented access techniques through which one can access the resources of network tools and technologies from a centralized third-party provider. It is related to the idea of cloud computing, where computers and resources are interconnected and shared with all their stakeholders. Due to its hyper con-nectivity nature in the cloud system, there is a serious concern and growing probability of malicious activities.

1. Compact network topology—In cloud-computing environment, the packed network topology of clouds coupled with the likelihood of homo-geneous software exploitation could allow rapidly propagating malware to propagate even faster than the classical network. This makes detection of malware more challenging compared to the non-cloud atmosphere.

2. Insecure interfaces and APIs—Cloud-computing providers expose to a set of software interfaces or APIs that customers use to manage and interact with could services. The securities, as well as accessibility of cloud services, depending on the security of APIs, which initiates the complexity of new layered API’s. This complexity of new layer APIs makes malware detection more challenging as it allows malware run-ning on one virtual machine (VM) to execute code or access data on another VM.

3. Virtualization—One of the problems in the cloud is virtualization which introduces an increase in software complexity and hence increases the


opportunity for vulnerabilities and hardware sharing. There is always a risk of an improper implementation or configuration of complex software that makes more defenseless of detecting malicious software.

Malware AnalysisThe taxonomy of malware detection methods largely employs two features for extraction processes: (i) static features—extracted from executables and (ii) dynamic features—extracted from runtime behavior of executables. The following figure gives an overview of classical malware analysis, in particular malware feature analysis.

Static Analysis

Conventional malware detection and classification systems are based on static fea-tures extracted from executables by reverse engineering. The static feature extrac-tion process is based on four different types of features:

◾ Portable executable (PE) ◾ Byte-sequence n-grams ◾ Function length ◾ String features.

The following figure illustrates a high-level overview of static malware analysis method. In this figure, the WEKA (https://cs.waikato.ac.nz/ml/weka/) interface

Figure 1.3 Classical Malware Analysis.


is given as classification and evaluation as it is well-known classification algorithm developed by machine learning group at the University of Waikato.

Dynamic Analysis

Dynamic analysis is the behavioral analysis of malware which is time-consuming as each malware sample must be executed for a certain time period and its actions logged all within a controlled environment to ensure that it cannot infect an active platform. This controlled, virtual environment is quite different from a real run-time environment, and the malware may act in different ways in the two environ-ments resulting in an inaccurate picture of the malware in the logs. The following figure gives an overview of the dynamic analysis process.

Additionally, some malware behavior is triggered only under certain conditions (via a specific command or interaction with a human, for example), and this cannot be picked up in the virtual environment. On the other hand, it has been suggested

Figure 1.4 Overview of Static Malware Analysis.

Figure 1.5 Overview of Dynamic Analysis.


that dynamic extraction is a necessary complement to static techniques as it is sig-nificantly less vulnerable to code obfuscating transformations.

Defense against Malware

While a substantial number of various malware detection techniques based on static and dynamic analyses have been studied in the literature, the recent malware proliferation technique with dynamic nature has attracted attention among the AM researchers and vendors. These malware capabilities render an AV strategy which has been successful in a given time period not to work at a much later date. Moreover, because malware evolves with time and eventually becomes unrecogniz-able from the original form or because completely new malware is designed which is unlike any known malware and so would not be detected by AV software con-structed to detect known types of malware.

It is obvious from the current literature that present malware detection meth-ods will not easily detect future malware. To solve this problem, researchers also proposed cumulative timeline analysis (CTA) that retains high accuracy over an extended time period. This technique provides strong support for the argument that both static and dynamic features are needed in malware detection and also that these features can be chosen in such a way that they leverage better results when they act independently and so complement each other. By extracting both static and dynamic features from executables and by accumulating these features over inter-vals in the 10-year period, the technique provides a high-accuracy malware detec-tion method that retains very close to the same accuracy along the entire timeline.

The Dark Web in the Context of Emerging Crime ThreatsThe emerging crime and threat are now ongoing development in the dark site of the web. Search engines like google provide access to freely available content which is 0.3% of the internet, according to the study of Kristin (2017). However, deep web is of massive size which is accessible only through the use of anonymous browser such as TOR. Kristin also mentioned in the report that the size of deep web is 4,000–5,000 times larger than that of surface web.

Numerous criminal activities are going on in the deep web including drug deal-ings (selling or buying), contract for assassinations, pornography industry includ-ing child pornography, sales of human body organs, human trafficking and sex trafficking, transactions for illegal shipment of arms, sales of stolen goods, sales of hacked cyber identity information, terrorism activities, and many more. The easiest way to browse is hidden wiki and deep search engine. These sites provide link access to many other links in deep web. The following figure shows a snapshot of deep search interface (Tor access only).


Human Trafficking and Sex Trafficking

Human trafficking and sex trafficking are a large part of crimes (homeland) and have extremely increased due to online forums, chat services, and anonymousness of the deep web. Human traffickers do negotiate and make contracts to recruit victims for human trafficking and sex trafficking. They can easily avoid detection,

Figure 1.6 Dark Web activities.

(Source: Human trafficking)

Figure 1.7 Snapshot of Deep Search Interface.


censorship, and surveillance system which are employed by the government and anti-human trafficking organizations using deep web and Tor network.

Pornography Industry

Victims of human trafficking and sex trafficking, particularly women, are exploited by pornography industry. Peak mentioned in a report that once a male or female signs the agreement for accomplishing the acts, traffickers force them by fear of assassination for pornography production. The sex traffickers record video with-out the consent of victims. Then, pornography industry distributes those to inter-ested parties. Traffickers also publish those recordings and photos in their website, as reported in Covenant Eyes (2011). Many websites are hosted in the deep web. Similar to human and sex traffickers, pornography industry uses deep web, social media, and online forums to recruit or kidnap victims but hide their identification.

Assassinations and Its Marketing

According to the Daily mail report (2013), criminals are using deep web to sell their assassin skills. A number of websites including MailOnline, White Wolves, and C’thuthlu provide advertisement for criminals mentioning that they can be hired for $10,000 in the United States and $12,000 in Europe. This ranges from 40,000 to 15 million of hire price for a police officer to a high-ranking politician.

Figure 1.8 Silk Road.

(Source: Silk Road: http://silkroadvb5piz3r.onion/index.php)


Drug Transactions

Anonymity of deep web access has led significant increase in drug transaction from the deep web which has created digital black market for drug. Criminals are using deep web for drug dealings, because there is no need for face-to-face communica-tion for buying/selling drugs. One of the examples is Silk Road website that sold the drug for over a billion dollar.

Where drug is posted by the courier company DHL ((Dalsey, Hillblom and Lynn) International GmbH) or drop shipping. A statistics has been provided in following figure.

Child Pornography

Children are using social media and many applications for communicating with their peers, friends, and family members. Often children come in contact with many strangers without their parent’s knowledge. Many of these applications such as Omegle and Ask.fm hide the identity of their users (CyberSafetyCop).

Figure 1.9 Statistics of Drug Transactions.

(Source: Economist)


This facilitates pedophiles to interact with the children. Pedophiles and related criminals are using deep web massively for child pornography, sharing photos and porns. One of the known companies is freedom hosting which uses 550 servers all over the Europe offering space to anyone for hosting porn from children (the US mulls). The statistics show that the FBI (FBI report in 2017) arrested several hundreds of pedophiles from the United States and international territories, which involve 2,000,000 users, 23,000 explicit images, and 9,000 video files with explicit sexuality in operation pacifier.

Terrorist and ISIS Use the Dark Web

ISIS uses dark web as a weapon for terrorism, where they provide live streaming and recording of mass execution of prisoners. They use the dark web as broadcasting media where they upload small video clips. The report published in June 20, 2017 by Singer and Brooking indicated that they used the dark web to recruit the soldiers around the world.

Techniques to Locate Criminals in the Deep Web and Challenges

Social media and deep web are jointly used to identify criminal activities. Different media including YouTube and Facebook are used to identify suspects. The US law enforcement agencies are using Metasploit Decloaking Engine and Memex systems by indexing the deep websites in an intelligent way to identify the criminals in the deep web [darpa], particularly the human traffickers. Bitcoin is the virtual money in the deep web [news.vice]. Law and policing agencies are also using Bitcoin flow in the deep web to locate criminals. The criminal’s websites are monitored; pur-chases of contraband merchandise are also used by the agencies for this purpose. One of the successful examples of identifying criminals from the deep web is the Silk Road server. The FBI identified the location of the server in Iceland data cen-ter. Although TOR is anonymized, this was identified due to a misconfiguration in Silk Road’s login page which described the IP addresses and physical location of the server. The FBI has also been successfully able to hack into the underground discussion board dark mode where criminals join every day, share information, and discuss about their criminal activities.

Although there are some successes in identifying the criminals, still there are more challenges to overcome. One of the biggest challenges is international borders which hinder further investigation and make it time-consuming. Privacy law stops the surveillance and information-gathering operations. The anonymity of deep web is also a big challenge to follow the money trail and criminals. The IP addresses and names are frequently changed in the deep web by criminals, so it is difficult to track.


To identify criminals in the deep web in a better and efficient way, more research-ers are required such as cyber black market economics and logistics, technologies for counter anonymity, and building the cyber threat vectors that can characterize and asses the crime. Finally, the techniques of collecting digital evidence should be such that they are collected, retrieved, and analyzed by following the law of the country to avoid further problems in the court.

SummaryThis chapter provides the fundamentals of cybersecurity and the dark web. The chapter started with a high-level overview of cybersecurity and cybercrime, how cybercrime becomes more targeted, and what criminal activities are going to hap-pen in the deep dark web. The chapter further explores the WWW and its catego-ries and various web levels. Some basic terminologies of the dark web and dark net alongside their consequences are discussed. The chapter further explores the threat landscape in the dark web and how to access the deep dark web. The basic idea of malicious software (malware), the taxonomy of malware, and the analysis of mal-ware by both static and dynamic methods are discussed in this chapter.

A detailed analysis of the dark web in the context of emerging threats and their impact on various domains are also presented.

Questions 1. Describe with an example the main differences between cyberwar and

cybercrime. 2. Define cybercrime. Explain, why cybercrime efforts are becoming more

targeted. 3. What are the characteristics of malicious software? 4. Why are malicious insiders a focus of security experts? 5. List the terms that relate to the cybercriminal behavior involving computers. 6. What is extortion? How do criminals engage in online extortion? 7. Briefly explain the general categories of cybercriminals in modern society. 8. Distinguish between surface web and deep web in terms of threat intelligence. 9. Define with an example the traditional and contemporary techniques used by

organized crime groups. 10. Explain with an example the technique used in social engineering.

Further ReadingThe reader can explore the following materials and websites for further understanding:


Amores R., Paganini P., The Deep Dark Web: The Hidden World, Vol. 1. Seattle, WA: CreateSpace Independent Publishing Platform, 2014.

Bartlett J., The Dark Net: Inside the Digital Underworld. Brooklyn, NY: Melville House, 2016.

Bishop M., Introduction to Computer Security. Boston, MA: Addison Wesley Professional, 2004.

Chen H., Dark Web: Exploring the Data Mining the Dark Side of the Web. New York: Springer, 2012.

Eric A.F., Cybersecurity issues and challenges. In Brief, Congressional Research Service (CRS) Report- R43831, August 12, 2016. Available: http://crs.gov.

Henderson L., Darknet: A Beginner’s Guide to Staying Anonymous Online. Seattle, WA: CreateSpace Independent Publishing Platform, 2013.

https://turbofuture.com/internet/A-Beginners-Guide-to-Exploring-the- Darknet.https://theguardian.com/technology/ 2009/nov/26/dark-side-internet- freenet.https://deepweb-sites.com/.ITU Publication, Understanding cybercrime: phenomena, challenges and legal response,

September 2012. Available: www.itu.int/ITU- D/cyb/cybersecurity/legislation.html.Kim P., The Hacker Playbook: Practical Guide to Penetration Testing. Seattle, WA: CreateSpace

Independent Publishing Platform, 2014.Lee N., Counterterrorism and Cybersecurity: Total Information Awareness, 2nd edn.

New York: Springer, 2015.Nikola Z., Computer security and mobile security challenges, ResearchGate, 2015.

Available: https://researchgate.net/publication/298807979.Rogers D., Mobile Security: A Guide for Users. Copper Horse Solutions Limited, 2013.Singer P.W., Friedman A., Cybersecurity and Cyberwar: What Everyone Needs to Know.

New York: Oxford University Press, 2014.State of Alabama Information Services Division, Why cyber security is important. Available:

www.cybersecurity.alabama.gov, Retrieved on 20 November 2016.Wu C.H., Irwin J.D, Introduction to Computer Networks and Cybersecurity. Boca Raton,

MA: CRC Press, 2013.

https://www.routledge.com/Security-Analytics-for-the-Internet-of-Everything/Ahmed-Barkat-Ullah-Pathan/p/book/9781003010463

https://www.routledge.com/Security-Analytics-for-the-Internet-of-Everything/Ahmed-Barkat-Ullah-Pathan/p/book/9781003010463

215

Chapter 9

Cryptographic Attacks, Impacts and Countermeasures

Hazaa Al Fahdi and Mohiuddin AhmedEdith Cowan University

Contents9.1 Introduction 216 .............................................................................................

9.1.1 Chapter Roadmap 217 .........................................................................9.2 Cryptography Overview 217 ...........................................................................

9.2.1 Cryptography Algorithms 217 .............................................................9.2.1.1 Symmetric Key Cryptography 218........................................9.2.1.2 Asymmetric Key Cryptography 219 ......................................

9.2.2 The Purpose of Cryptography 219 .......................................................9.3 Cryptography Attacks 220 .............................................................................

9.3.1 Brute Force Attack 220.......................................................................9.3.2 Implementation Attack 222 ................................................................9.3.3 Side-Channel Attack 222 ....................................................................9.3.4 Fault Analysis 222 ..............................................................................9.3.5 Probing Attacks 223 ............................................................................9.3.6 Birthday Attack 223 ............................................................................9.3.7 Chosen-Ciphertext Attack 223 ............................................................9.3.8 Ciphertext-Only Attack (COA) 223....................................................

216 ◾ Security Analytics for the Internet of Everything

9.1 IntroductionCryptography is the use of mathematics to produce encrypted data and to decrypt it. Cryptography involves two main operations: encryption and decryption. Both of the operations ensure information security, which includes confidentiality, integ-rity, and availability. Moreover, most commercial companies, governments, and other business fields are using cryptography to ensure data security. For example, banks use an encryption algorithm to encrypt payments and money transactions of their clients. Thus, the user of the encryption algorithms will ensure the banks’ services security and will avoid any breaches of the data [1]. Furthermore, most encryption algorithms have been attacked by different cyber-attacks. For example, the data encryption standard (DES) is symmetric that is used to encrypt electronic data. DES has a key length of 56 bits, and due to its key length, it has been tracked several times by brute force attack [1].

Due to the increase in the usage of encryption, everyone is aiming to secure his/her data. This led to increasing cyber security attacks against cryptography algorithms. The attacks left many impacts and risks behind them on their users such as governments, business, and normal users. The risk is high to loss sensi-tive data, damage systems, or financial losses. Therefore, the users of encryption features are required to implement a set of countermeasures to ensure that the usage of the encryption algorithm is secure, and also all business data or secret operations data are safe while performing encryption processes and transferring of the data. Otherwise, hackers can breach their data due to the lack of security. Moreover, different types of software, hardware, and policies can be implemented as best practice solutions to prevent different types of attacks against cryptography algorithms [1].

9.3.9 Power Analysis Attack (PAA) 224........................................................9.3.10 Known Plain Text Attack (KPA) 224 ..................................................9.3.11 Chosen-Plain Text Attack (CPA) 224..................................................

9.4 Cryptography Attacks Impacts 224 .................................................................9.4.1 Brute Force Attacks 225 ......................................................................9.4.2 Side-Channel Attacks 226 ..................................................................9.4.3 Fault Analysis Attacks 226..................................................................9.4.4 Probing Attacks 226 ...........................................................................9.4.5 Birthday Attacks 226 ..........................................................................9.4.6 Ciphertext-Only Attack (COA) 227 ....................................................9.4.7 Chosen-Ciphertext Attacks 227 ...........................................................9.4.8 Known Plain Text Attack (KPA) 227 ..................................................

9.5 Countermeasures 227 ......................................................................................9.6 Conclusion 228 ..............................................................................................References 229 .........................................................................................................

Cryptography ◾ 217

9.1.1 Chapter Roadmap

This chapter is divided into four main sections, which are as follows: Section 9.1 is about introduction of the chapter. Section 9.2 is about an overview of cryptography concept, Section 9.3 is focusing on different types of cryptography attacks, and Section 9.4 is about the impacts of cryptography attacks, and the last section is about countermeasures of the cryptography attacks.

9.2 Cryptography OverviewCryptography is a method of altering and transferring sensitive data in an encryp-tion process so that only who has an authorization can reveal it and work on it. Cryptography is a Greek word in which “crypto” means secret or hidden and “graphy” means writing [1]. Thus, cryptography is the process of hiding the original data to produce another data that is hard to understand, which ensure confidentiality, non-repudiation, integrity, and authenticity for the transferred data [2]. The cryptogra-phy has different number of components used in both encryption and decryption processes. Basically, any cryptographic system will have the following components:

A. Plain text: It is the data which is to be secured while communication [2].B. Ciphertext: It is the changed plain text which cannot be read or under-

stand when looking at it. To read it, it is required to apply the decryption algorithm. The ciphertext might or might not be secure, which depends on the encryption algorithm [2].

C. Encryption algorithm: It is the way to decode the plain text to produce unreadable text. This can be used in specific algorithms such as advanced encryption standard (AES) or DES. Also, it is used at the sender’s side [2].

D. Decryption algorithm: It is the opposite process of the used encryption algorithm. Thus, it uses ciphertext and decryption key to reveal the original plain text [2].

F. Decryption key: This key is used within the decryption algorithm in order to obtain the plain text from the ciphertext. The key is either known only to the sender or the receiver usually, and it depends on the algorithm used [2].

9.2.1 Cryptography Algorithms

Modern cryptography is based on strong scientific methods. The new cryptography is divided into several types. In this book, this chapter will discuss symmetric key and asymmetric key cryptography algorithms. An encryption method is based on

E. Encryption key: This key is used with the encryption algorithm to produce the ciphertext of the plain text. The key is either known only to the sender or the receiver usually, and it depends on the algorithm used [2].


one key of encryption and decryption processes. Both the receiver and the sender use the same key. This method was the only one that was known for encryption until June 1961. The symmetric cryptography of encryption and decryption is based on the study of the block ciphers and stream ciphers [1].

A block cipher is a modern embodiment of Alberti’s polyalphabetic cipher. The block cipher takes a block of plain text as an input and a key; this block will go through an encryption process to produce an output of block of ciphertext in the same size [3]. On the other hand, a stream cipher is opposite to the block cipher. The stream cipher is taking the key and combined with the plain text as bit-by-bit or character-by-character. The output of stream cipher is based on altering the plain text orders to generate the stream cipher [3].

9.2.1.1 Symmetric Key Cryptography

Symmetric encryption is one kind of encryption method that uses one key for both encryption and decryption operations. The key which is used for encryp-tion and decryption is called secret key, and it is only accessible by a specific user. Moreover, the symmetric encryption uses two types of encryption algorithms. The first algorithm is called block algorithm, which has different lengths of bits that encrypted in different blocks by the secret key. The encrypted data will be held until all the blocks complete. The second is the stream algorithm in which all data are encrypted as a stream rather than reserved in the memory [4]. Furthermore, sym-metric method is fast for encryption and decryption operations compared to asym-metric method. Also, it has several types of algorithms such as AES, DES, RC4, RC5, and IDEA encryption algorithms. Moreover, symmetric is used to encrypt a large value of data such as database encryption of an organization, payment for banking, and online store [5].

Figure 9.1 shows the encryption and decryption processes where the plain text is encrypted and decrypted by using one key for the two parties.

The DES and the AES are block ciphers based on symmetric-key cryptography. Both the algorithms have different features, weakness, and strengths. The DES encrypts a block of 64 bits by using a key of 56 bits in length. The encryption process includes 16 rounds of confusion and diffusion. The confusion is an encryp-tion operation that makes the relation between the key and ciphertext obscured.

Plain text Cipher text Plain textEncryption

Secret Key

Decryption

Secret Key

Same key for both operations

Figure 9.1 Symmetric encryption.


The diffusion is also an encryption operation that gathers one of the plain text values over many ciphertext values to produce a strong ciphertext. In addition, it is not recommended to use the DES due to its key length as well as other aspects of the algorithm that have been kept secret by the US government [1].

Furthermore, the AES can encrypt a block of 128 bits. The AES has different sizes of keys of 128, 192, and 256 bits. Each key length uses a different number of subkeys. Moreover, the AES is more secured compared with DES, because it uses large size of keys, which makes it strong against some types of cyber security attacks such as brute force attack. Also, the AES is part of known standards such as IPsec and TLS, which make the users more confident to implement the AES in their software, hardware, communications, and other types of field that required to make the data secured and encrypted [1].

9.2.1.2 Asymmetric Key Cryptography

The asymmetric cryptosystem also called public-key cryptosystem has two different types of keys, namely, the public key and the private key. The public key is usu-ally used for encryption by the sender, where the plain text is combined with the key to produce the ciphertext (depends on the algorithm used); also, the public key can be distributed publicly. The private key is used by the receiver for decryption operation, and it must be kept secret [1]. In addition, several protocols used asym-metric cryptography, for example, SSH, SSL, OpenPGP, and digital signature. Also, Rivest–Shamir–Adleman (RSA) is using asymmetric method, which pro-vides security for different channels. For example, RSA is using SSL/TLS protocol to provide secure communication of users over a network. Moreover, asymmetric method provides digital signature, which ensures the identification of a sender or receiver of a document [1].

Figure 9.2 shows the encryption and decryption processes in asymmetric cryp-tography. The encryption process is using the public key, and the private key is used to decrypt the encrypted data.

9.2.2 T he Purpose of Cryptography

The usage of cryptography is to ensure confidentiality, integrity, non-repudiation, and authentication. These features are divided into cryptography algorithms. For example, the symmetric cryptography such as AES can ensure the confidentiality

Plain text Cipher text Plain textEncryption

Public Key

Decryption

Private Key

Figure 9.2 Asymmetric cryptography.


and authentication of a communication between two parties by key session. Also, the non-repudiation and authentication can be found while using the RSA and DSA of asymmetric cryptography algorithms [1]. In spite of this, the cryptography is a useful technique to secure communication and data transmission, but there are several breaches on cryptography methods that led to many impacts on the usage of the cryptography and its users [5].

9.3 Cryptography AttacksIn this day, the usage of information is not limited to technology devices, but to all human life aspects such as communication, for example, emails, fax, and social media applications. The share of critical information has become an impor-tant thing to ensure the confidentiality and integrity of transmitted data. Thus, most of the organizations, business, and governments give high attention to their sensitive information by paying a high amount to implementing policies, soft-ware, and hardware to make sure their data is safe. Unfortunately, some of the cryptography algorithms have failed to secure the encrypted data against several attacks [6,7].

The attackers can apply passive or active attacks on a target to get some informa-tion about it. The passive attack is playing a sensitive role to obtain information on a system or user device without authorization to access. Also, the passive attack does not affect the systems resources, but affects the data’s confidentiality, for example, eavesdropping on two parties’ communication channel. This is done without the two parties knowing which usually leads to stealing the information [7].

On the other hand, the active attack aims to modify or damage a system’s resources or operation in a way to obtain the wanted data. The active attack leaves high effects on the target systems and threats the integrity and availability of it [1].

Table 9.1 reflects several attacks that have been approved against the cryptog-raphy algorithms.

To discuss in depth about each of the above attacks, the following sections explain more about them.

9.3.1 B rute Force Attack

A brute force attack is one of the techniques used to reveal an important data for a specific target such as password of a system, encryption keys of different types of encryption algorithms, and SSH logins. According to researchers, the brute force is used against encryption algorithm. For example, the brute force attack has been approved against the DES due to its key length of 56 bits. Thus, the attacker can apply the attack to obtain encrypted data within a few hours [5]. Furthermore, the brute force attack depends on a specific wordlist or an intelligent database, which is used to detect the algorithm key or password of the target. In spite of brute force


useful attack, it is limited to short keys. For example, the AES has different key lengths of 128, 192, or 256 bits. Therefore, the brute force attack would take a long time to attack 128 bits of AES key [8].

Table 9.1 Attacks Key Features

Attack Name Key Feature(s)

Brute force attack Reveals sensitive information such as password, encryption, and decryption keys and SSH logins

Dictionary attack Reveals passwords, algorithms keys

Rainbow table Cracks passwords or hashes

John the Ripper Crack passwords through a wordlistRuns in different operation systems

Side-channel attack Gathers vulnerabilities of a system or cryptography algorithm

Gains access to encrypted data

Probing attacks Scans a system, networks, algorithms to discover vulnerabilities

Implementation attack Exploits the vulnerability of an application, software, or algorithms

Fault analysis Gathers a high amount of erroneous information to gain access to sensitive information

Chosen-ciphertext attack Based on selecting numbers of ciphertext to obtain decryption data by using a random or unknown key

COA Uses only a set of ciphertext to decrypt it

PAA Determines the secret key by gathering specific information about cryptography algorithms

Birthday attack Discloses a sensitive information

KPA Aims to use the known plain text to decrypt the set of ciphertext

CPA Determines the encryption key of cryptography algorithms through choosing a plain text


Furthermore, there are several tools or techniques used by brute force attacks such as dictionary attack, Rainbow table, and John the Ripper.

Dictionary attack: The attackers use a dictionary attack to obtain sensitive information of an encryption algorithm such as decryption key. The process is to build a dictionary of ciphertexts, keys, and identical plain text, and then to try all possible possibilities in a period of time. The dictionary attack can be a suc-cess due to a lack of security of users, for example, usage of poor password and sharing secret pair keys with other parties. Thus, the attackers can take advantage of users’ faults and build the dictionary to reveal the sensitive information of the target [9].

Rainbow table: This attack is usually used to crack passwords or hashes that have been created of plain text or password. The cracking method is to try each possible hash to obtain encrypted data [10].

John the Ripper: A software tool is used to crack passwords through a wordlist. Also, it can run in different operating systems such as Unix and Windows. Thus, by giving a limited key length and enough time, a brute force attack can be launched successfully on a ciphertext or plain text [11].

9.3.2 Implementation Attack

An implementation attack exploits the vulnerability of an application, software, or algorithms. Most of the encryption algorithms rely on outside resources of the algorithms. The implementation attack can breach some of the encrypted data due to weaknesses in the algorithms. There are three types of implementation attacks [1].

9.3.3 Side-Channel Attack

The attack is based on gathering vulnerabilities, a database of a specific target such as a company system, or a cryptography algorithm. Also, the side-channel attack can monitor all power’s usage during encryption and decryption operations. This can lead to gain access to encrypted data such as algorithm keys. The most com-mon cryptography algorithm that has been affected by the side-channel attack is AES [1].

9.3.4 Fault Analysis

The attack is based on forcing a target’s system into an error situation to get wrong results of it. Usually, the attackers repeat the process until they gather a high amount of erroneous information. Finally, the attackers compare the results from the attack with previously known information of the target system; then, they analyze the information to gain sensitive information about the systems, which includes cryp-tography’s algorithm data such as the secret key [12].


9.3.5 Probing Attacks

This attack is based on scanning a system, networks, and algorithms to discover vulnerabilities, which can be used to gain more privileges and steal sensitive information of a special target such as banking applications [13].

9.3.6 Birthday Attack

This is one of the most effective attacks, and it is used against the cryptographic hash. Birthday attack can lead to many potential effects such as disclosing sensitive information of a communication between two parties or more. For example, when students at a university asked about their date of birth, the answer of each student will be one of the 365 dates. Thus, if a student birthday is 3/8/199x, then we need 25 students to be asked. On the other hand, if a hash function has 64 bits of values, then the probability of hash values is 1.8 × 1019 [14].

In addition, if an attacker repeats the hash function for different values of inputs, a similar output will be after 5.1 × 109 of random values; so if an attacker finds two inputs having same hash value, then it will lead to break the hash func-tion. Therefore, the birthday attack can lead to launch a collision, breaking hash functions and cryptography algorithms [14].

9.3.7 Chosen-Ciphertext Attack

A chosen-ciphertext attack is one of the model attacks that is used for cryptanaly-sis. The cryptanalysts gather different pieces of information to promote the attack successfully. Thus, adversaries use this type of attack to gather secret and impor-tant information about cryptography algorithms. The attack technique is based on selecting numbers of ciphertext to obtain its decryption versions by using a random or unknown key. The adversaries will launch the attack based on a set of cipher-text until the decrypted data appear. There are different number of cryptography schemes which have been attacked by the chosen-ciphertext attack such as RSA and SSL protocol [15].

9.3.8 Ciphertext-Only Attack (COA)

COA is one of the cryptographic attacks which is used for cryptanalysis. The attack is based on accessing only a set of ciphertext. The attackers are aiming to obtain the encrypted messages, information, or the secret keys of them to gain more data or other information in cryptography algorithms. Moreover, it may be hard to attack modern ciphers through COA. Thus, there are two methods that can help to breach the modern ciphers. The first method is exploiting the two-time pad. Cryptography experts say that never use the same keystream more than one time because the cipher will be able to attack by COA. The second method is frequency analysis,


which is a common type of COA attack which focuses on analyzing the frequency of letters in a ciphertext. Each language or ciphertext will have its own letters that will make the frequency analysis to get successful results [1].

9.3.9 P ower Analysis Attack (PAA)

This attack is based on gathering and abating information about an algorithm con-sumption in a way to determine the secret key or other sensitive data that can be used to decrypt the ciphertext. Moreover, this attack is similar to the timing attack, which is based on gathering facts of a process. For example, if an encryption process takes a long time, then the encryption algorithm uses a long secret key [16].

9.3.10 Known Plain Text Attack (KPA)

This attack is based on plain text. The attackers exploit the known plain text of some set of ciphertext. The aim is to use the known plain text to decrypt the set of ciphertext. This might be done by identifying the key or through other tech-niques. The most common example of this attack is linear cryptanalysis against block ciphers [17].

9.3.11 Chosen-Plain Text Attack (CPA)

This attack is based on a chosen ciphertext-plain text pair of an attacker choice. This means that the attacker has the plain text, which makes the attack process easier and is successful. Thus, the attacker aims to determine the encryption key of the cryptography algorithm. The CPA has been approved against different fields of cryp-tography such as hash functions, public-key cryptosystem, and RSA algorithm [18].

9.4 Cryptography Attacks ImpactsIn general, any attack leads to leave an impact on the target due to many reasons. The cryptography algorithms have become vulnerable to numbers of cyber secu-rity attacks. Many aspects have been affected by the cyber security attacks against encryption algorithms, for example, information security breaches, financial losses, reputation, confidentially, integrity, and availability [1].

The information security is a significant aspect that should not be affected by any cyber-attacks. Each piece of information is playing an important role in the business field and different operations of sharing the information. Furthermore, losing a piece of information can lead to many issues, for example, if an organi-zation is aiming to make a sale of a new technology product that does not exist yet. The new product has several futures and a specific price. Thus, if an attacker attacked this organization data center, they can steal this sensitive information and use it in malicious activities, for example, selling the secret information for other


competitors in the market, which leads to many effects for the original organiza-tion. Therefore, the information should be secured to avoid similar impacts [19].

The financial loss is one of the most impacts that can affect a business situation directly. This means many organizations or governments will be affected economi-cally due to many attacks. Thus, they cannot continue producing products or ser-vices for their clients and their operations will hold for a period of time due to this issue. This will affect employees’ income, which can affect other aspects in their life such as health situation. In addition, if the financial issues continue, they may lead to bankruptcy, which is a huge impact on the business fields. This will decrease the operations of producing products or services, thus leading to loss of partnership with other companies [20].

Cryptography attacks such as brute force and man in the middle attack can leave a huge impact on an organization’s reputation. The reputation plays a sensitive role in the market as it can attract new staff or investors. This can be done only if an organization has a strong name in the market such as never attacked by cyber-attacks and has a professional staff in cyber security, operations, and recovery. On the other hand, if an organization has a poor name in the market, it will lose its revenue and name due to the cryptography breaches. The reputation of an organiza-tion can also be affected by other reasons such as economy issues or provides poor services, which make the customers to submit a complaint against the services. [21].

Confidentiality is a significant field that should be secured. It can be found in different areas such as business, health sector, immigration, and secret governments operations. Each of these sectors can be in a high risk and impacts if one of the cryp-tography attacks found on these sectors. For instance, the health sector contains a classified information of a client. The attacks can lead to reveal this information which should be protected and should have more privacy. Moreover, immigration sector also contains sensitive information such as passport numbers, visa numbers, and photo-graphs of the passport holders. Any leaking on this type of data can breach the con-fidentiality of its users. Also, this sensitive information can be sold in black market, which can also be used in malicious activities under the passport’s holders [22].

Furthermore, the following paragraphs explain more about the impacts of cryp-tography attacks:

9.4.1 Brute Force Attacks

The brute force attacks usually try to reveal hidden aspects of encrypted value or a plan password. Also, the attack can be used against numbers of encryption algorithms. Thus, the impacts which attack leave the following: Reveal sensitive information such as secret key value and plain text value which is usually a secret message between two parties and discolor of password values. These impacts actu-ally will lead to other impacts of ozonizations or governments. For instance, gov-ernment employees are sharing a piece of classified information regarding some business with other parties, but the lack of security in the used encryption algorithm


such as DES leads to the loss of financial, loss of clients, and loss of reputation in the market [1]. Furthermore, the financial losses may be a reason that leads to other impacts. For example, an organization will lose its staff, because they are not paid enough and other competitors give them better offers. In situation, the losses will increase reputation, which is a sensitive issue [21].

9.4.2 Side-Channel Attacks

A side-channel attack can put cryptography algorithms users in high risks and can lead to some serious impacts. In addition, the most common algorithm that has been affected by the side-channel attack is the AES. The attackers use this kind of attacks which they are aiming to access the algorithm data to gain more informa-tion such as the secret key. Thus, once they got the secret key, other information became easier to reveal. This includes the transferred message or plain text data. Therefore, the side-channel attack is one of the strongest attacks, which leads to disclose the sensitive information [1].

9.4.3 Fault Analysis Attacks

A fault analysis attack affects the implementation of cryptosystem and its mathe-matical functions. For example, the elliptic curve cryptosystem has been proved that it is vulnerable to many cyber-attacks such as fault analysis attack. Thus, this type of attack exploits any fault that can happen in the implantation of the cryptosystems. For instance, faults can be exploited in many methods or locations, and almost the whole of the system can be attacked, such as the system parameters, dummy opera-tions base point, intermediate results, dummy operations, and validation tests [13]. This leads to reveal and discover some secret information of the cryptosystem [12].

9.4.4 Probing Attacks

A probing attack is based on gathering sensitive information about a target. Usually, the gathered information will be vulnerabilities about the target by using specific tools and software. This kind of attack affects the target system by exploiting its weaknesses. For example, knowing the faults in a cryptography algorithm compro-mises it and steals all important data. The important data can be the secret key or the encrypted data. Thus, hackers exploit applications and system vulnerabilities to gain more access into it, and most organizations are vulnerable to probing attack if they are not updating their system or software [13].

9.4.5 Birthday Attacks

A birthday attack is one of the strong attacks in cyber security field. It is used to break cryptographic hash function such as MD5 or SHA hash function. This kind


of hash function may contain some sensitive information such as a secret key of a cryptography algorithm or a password of a communication channel. Thus, the birthday attack can reveal sensitive information of a communication between two parties or more by attacking the cryptography process or hashes, which leads to the loss of the confidentiality and authentication of the communication channel. Moreover, the birthday attack can lead to launch hash collision to find the hash values. Also, breaks cryptography algorithms through [14].

9.4.6 Ciphertext-Only Attack (COA)

A COA, also known as known ciphertext attack, is a model attack for cryptanalysis, that is, the attackers can access only to a set of ciphertext. The attack can reveal the encryption or decryption key if the ciphertext has been decoded. This can be a risk for users who use cryptography as security features. Therefore, the loss of privacy, integrity, and authentication is possible if COA occurred [15].

9.4.7 Chosen-Ciphertext Attacks

A chosen-ciphertext attack leads to gather a secret or general information of cryptog-raphy algorithm. The attack is based on selecting numbers of ciphertexts to obtain its decryption versions by using a random or unknown key. From the decrypted versions, the attackers can recover or read some hidden aspects, such as the secret key which is used for encryption. One of the common examples that have been affected by CPA is the El Gamal cryptosystem. In fact, the El Gamal cryptosystem is secured against plain text attack, but it can be exploited by chosen-ciphertext attack. Also, the early versions of RSA use SSL protocol that was vulnerable to the chosen-ciphertext attack [15].

9.4.8 Known Plain Text Attack (KPA)

A KPA aims to affect an encryption scheme in a way to reveal or decode secret infor-mation. In addition, the attack is based on a set of plain texts and the encrypted version of them. Thus, the attackers try to use them to reveal sensitive information. The effects behind this attack are similar to those of COA and CPA, which reveal secret information and encrypted messages about an encryption algorithm [15].

9.5 CountermeasuresNo one is immune in the cyber security field. Thus, the need for countermeasures is necessary to provide a high level of security to protect data of communication, online transactions, and e-commerce. All of these are using encryption processes to protect secret information against several breaches. The users of cryptography


algorithms are required to select a secure method of encryption to ensure the secu-rity of their data. This can be through numerous techniques, tools, and policies while using an encryption scheme. Security and countermeasures will provide con-fidentiality, integrity, availability, and authentication of systems or sensitive data. In this chapter, the countermeasures are discussed in several methods [23].

Cryptography attacks prevention is not different from other cyber-attacks. The need in the cryptography field is to secure the encrypted data. This can be done by avoiding old encryption algorithms such as the DES, which is vulnerable to brute force attacks due to the small key length. Therefore, it is recommended to use secure encryption algorithms with a long key. For example, AES has different keys of 128, 192, and 256 bits. Also, it has a number of different rounds of encryption and decryption, so the key of 128 bits will be in 10 rounds, the key of 192 bits will be in 12 rounds, and the key of 256 bits will be in 14 rounds. Thus, the brute force attack cannot affect AES, due to the long keys used in the algorithm. Therefore, the usage of a strong encryption algorithm such as AES will ensure that the encrypted data cannot be breached [1].

The digital signatures play a significant role in securing identification and data. Using digital signature such as RSA while transferring data between the two par-ties, it will ensure the authentication of the two parties. In addition, the RSA algo-rithms use 1,024-bit and 2,048-bit key long, which make it more secure. In terms of cryptography concept, encryption of the data is not enough to make sure it is secured. Therefore, adding the digital signatures will add a new challenge to attack-ers, which increases their attack process [24].

In terms of securing encrypted data, it is recommended to implement informa-tion technology security policy such as ISO27001. This kind of policy will ensure several areas of security and increase awareness of the users such as how to choose system password [25]. Moreover, the awareness programs help to increase the cyber security awareness of users in organizations, banks, and business companies. This will aid to avoid many attacks such as phishing attacks. Phishing attacks can steal sensitive data from a user’s device; this may include information of an encryption algorithm. Also, awareness programs aid to keep the organizations and others using cryptography futures in their environment in the latest update of security futures in cyber field such as encryption schemes. Thus, they will be aware of how to imple-ment an encryption scheme securely [26].

9.6 ConclusionCryptography concept has become an essential thing in business field and govern-ment operations. This is to secure their data against breaches. Moreover, symmet-ric cryptography and asymmetric cryptography are used for different purposes in the cryptography field. One of the essential purposes is to ensure the confiden-tiality, integrity, and availability of their data. The symmetric cryptography has


only one key to encrypt and decrypt a plain text. On the other hand, asymmetric has two keys used in encryption and decryption processes. Even though sym-metric cryptography and asymmetric cryptography provide good security futures such as secure sensitive data, they can be attacked by several cyber-attacks. For example, a brute force attack takes advantage of the key length to launch the attack. Also, the birthday attack aims to break a hash function to steal its data. Moreover, there are other attacks that can be used to break symmetric cryptogra-phy and asymmetric cryptography, such as dictionary attack and COA, chosen-plain text attack, PAA, and side-channel attack. These attacks can break different types of encryption algorithms, and lead to reveal sensitive information of the algorithms such as private keys. Thus, this will lead to disclose sensitive informa-tion from the encrypted information. Therefore, the need of countermeasures is an essential thing to secure the users of cryptography algorithms. This chapter discussed the countermeasures by dividing it into prevention and deletion. The prevention section was referring to use unbreakable encryption algorithms such as AES scheme.

References 1. Gupta, A. & Walia, N. K. Cryptography algorithms: A review. International Journal

of Engineering Development and Research, 2(2), 1667–1672, ISSN: 2321-9939.2. Peter, S. & Dawn, T. (2019). Symmetric Key Encryption - Why, Where and How It’s

Used in Banking [Blog]. Retrieved from www.cryptomathic.com/news-events/blog/symmetric-key-encryption-why-where-and-how-its-used-in-banking.

3. Al-Vahed, A., & Sahhavi, H. (2011). An overview of modern cryptography. World Applied Programming, 1(1), 55–61.

4. Shree, D., & Ahlawat, S. (2017). A review on cryptography, attacks and cyber security. International Journal of Advanced Research in Computer Science, 8(5), 1–4.

5. Paar, C., & Pelzl, J. (2010). Understanding Cryptography: A Textbook for Students and Practitioners. Heidelberg: Springer. Retrieved from https://ecu.on.worldcat.org/search?databaseList=&queryString=Understanding+cryptography%3A#/oclc/527339793.

6. Paliwal, S., & Gupta, R. (2012). Denial-of-service, probing & remote to user (R2L) attack detection using genetic algorithm. International Journal of Computer Applications, 60(19), 57–62.

7. Köpf, B., & Smith, G. (2010, July). Vulnerability bounds and leakage resilience of blinded cryptography under timing attacks. In 2010 23rd IEEE Computer Security Foundations Symposium (pp. 44–56). IEEE, Edinburgh, UK.

8. Conrad, E. (n.d.). Types of Cryptographic Attacks. Retrieved from www.academia.edu/4739047/Types_of_Cryptographic_Attacks.

9. Adams, C. (2011). Dictionary Attack (pp. 1–76). School Of Information Technology and Engineering (SITE), University Of Ottawa. doi:10.1007/978-1-4419-5906-5_74.

10. Kalenderi, M., Pnevmatikatos, D., Papaefstathiou, I., & Manifavas, C. (2012, August). Breaking the GSM A5/1 cryptography algorithm with rainbow tables and high-end FPGAS. In 22nd International conference on field programmable logic and applications (FPL) (pp. 747–753). IEEE, Oslo, Norway.


11. Weir, M., Aggarwal, S., De Medeiros, B., & Glodek, B. (2009, May). Password crack-ing using probabilistic context-free grammars. In 2009 30th IEEE Symposium on Security and Privacy (pp. 391–405). IEEE, Berkeley, CA.

12. Joye, M., & Tunstall, M. (Eds.). (2012). Fault Analysis in Cryptography (Vol. 147). Heidelberg: Springer.

13. Saraswat, S., Tanisha S., & Neetu F. (2017). Fault analysis in cryptography. International Journal of Latest Trends in Engineering And Technology. doi:10.21172/1.91.29.

14. Gupta, G. (2015). What is Birthday Attack?? (pp. 1–14). The Maharaja Sayajirao University of Baroda. Retrieved from www.researchgate.net/publication/271704029_What_is_Birthday_attack.

15. Sadkha, S. (2012). Methods of Cryptanalysis. Retrieved from www.uobabylon.edu.iq/eprints/paper_5_7264_649.pdf.

16. Ors, S. B., Gurkaynak, F., Oswald, E., & Preneel, B. (2004, April). Power-analysis attack on an ASIC AES implementation. In International Conference on Information Technology: Coding and Computing, 2004. Proceedings. ITCC 2004 (Vol. 2, pp. 546–552). IEEE, Washington, DC.

17. Van Oorschot, P. C., & Wiener, M. J. (1990, May). A known-plaintext attack on two-key triple encryption. In Workshop on the Theory and Application of of Cryptographic Techniques (pp. 318–325). Springer, Berlin, Heidelberg.

18. Kiltz, E., O’Neill, A., & Smith, A. (2010, August). Instantiability of RSA-OAEP under chosen-plaintext attack. In Annual Cryptology Conference (pp. 295–313). Springer, Berlin, Heidelberg.

19. Zhang, D. (2018, October). Big data security and privacy protection. In 8th International Conference on Management and Computer Science (ICMCS 2018). Atlantis Press, Shenyang, China.

20. Bouveret, A. (2018). Cyber Risk for the Financial Sector: A Framework for Quantitative Assessment. International Monetary Fund.

21. Mitic, P. (2018). Reputation risk: Measured. International Journal of Safety and Security Engineering, 8(1), 171–180.

22. Levenstein, M. C., Tyler, A. R., & Davidson Bleckman, J. (2018). The Researcher Passport: Improving Data Access and Confidentiality Protection.

23. Ghosh, S., Mukhopadhyay, D., & Chowdhury, D. R. (2011). Fault attack, counter-measures on pairing based cryptography. IJ Network Security, 12(1), 21–28

24. Merkle, R. C. (1989, August). A certified digital signature. In Conference on the Theory and Application of Cryptology (pp. 218–238). Springer, New York.

25. Disterer, G. (2013). ISO/IEC 27000, 27001 and 27002 for Information Security Management.

26. Abawajy, J. (2014). User preference of cyber security awareness delivery methods. Behaviour & Information Technology, 33(3), 237–248.

https://www.routledge.com/Why-Dont-We-Defend-Better-Data-Breaches-Risk-Management-and-Public/Sloan-Warner/p/book/9780815356622

https://www.routledge.com/Why-Dont-We-Defend-Better-Data-Breaches-Risk-Management-and-Public/Sloan-Warner/p/book/9780815356622

25

C H A P T E R 2

Software Vulnerabilities

We’ll use the term vulnerability in its technical, information security sense: a weakness that could lead

to a security breach. Software vulnerabilities are weaknesses of the software, in fact defects, that hackers can exploit to gain unauthorized access. Software programs currently contain an unacceptable number of vulnerabilities. As we will discuss, it’s possible to write software with many fewer, albeit still some, vulnerabilities.

Since today’s rate of software vulnerabilities is high, and since there will always be some software vulnerabilities, all organizations, and especially large organizations, ought to manage their software to defend against attacks. All too often, large corporations such as Wyndham Hotels or Equifax fail to take appropriate security measures to defend adequately. For example, they don’t have adequate anti-malware software in place or they don’t patch software in a timely manner. We will call these software management vulnerabilities. We discuss these vulnerabilities in Chapter 3.

26 ◾ Why Don’t We Defend Better?

Software engineers know how to minimize (though, alas, not how to totally eliminate) software defects, also known as bugs. The most common and familiar kind of bugs are errors in programming leading to software crashing or giving the wrong result for some expected use from authorized users. The defects we are most concerned about here are (security) vulnerabilities (also called security bugs), by which we mean a defect in software, which can be exploited to gain unauthorized access or privileges.

Software engineering basics, including both how to write individual small computer programs well, and how to combine them into large software systems, is a fairly well-developed subject. The basics of high-quality code construction and software engineering generally form a significant fraction of the required portion of the model computer science bachelor’s degree curriculum jointly published by the two main professional societies for computer science both in 20131 and in earlier versions.2 Avoiding security vulnerabilities in particular started getting significant space in the model curriculum in the 2008 version and is prominently featured in the 2013 version.

Years of studies dating back to the 1980s, confirm the common wisdom among experts in software development that proper attention to software development leads to lower defect rates.3 So, why do software developers not do what they know how to do? Why is software so full of bugs, including vulnerabilities?

DISTRIBUTION OF VULNERABILITIES OVER TYPES OF SOFTWAREBefore answering, we should clarify the question. What we really want to know is: Why is there a relatively small group of very widely installed and commonly used programs that have lots of security vulnerabilities? The security vulnerabilities database CVE Details lists thousands of products that have had at least one reported security vulnerability sometime in the past 20 years. However, a fairly small group of programs is responsible for a

Software Vulnerabilities ◾ 27

wildly disproportionate share of the reported vulnerabilities over the years. These include:4

• Consumer and end-user facing operating systems: Windows and Mac OS X and Android and iOS (iPhone/iPad operating system); in the 2018 list, Android (both in the Android operating system itself and in underlying Qualcomm firmware) and Microsoft Windows are especially prominent

• Adobe PDF readers and writers: In 2018, Adobe DC and Adobe Reader DC, as well as various versions in earlier years

• Web browsers: Chrome, Firefox, Internet Explorer, and Safari

• Operating systems used primarily by software professionals (Linux) and server operating systems that would be used by organizations for purposes such as serving their own web pages (various versions of Windows Server and Enterprise Linux server)

• Microsoft Office

• Adobe Flash

A simple count of vulnerabilities only provides an extremely crude idea of which ones will be exploited.5 Nevertheless, we can see that there are some extremely widely deployed pieces of end-user facing software that have large numbers of vulnerabilities. Indeed, a different methodology found that vulnerabilities exploited in practice are even more concentrated. In both 2015 and 2016, all the top 10 vulnerabilities used by criminal exploit kits were found in Adobe and Microsoft products.6

SOURCES OF SOFTWARE DEFECTSThe sources of vulnerabilities are generally the same as the sources of defects. Vulnerabilities, after all, are a particular type of software


defect, one a hacker can exploit to gain unauthorized access. As the software experts Capers Jones and Oliver Bonsignour note, “Software bugs or defects stem from many sources and have many causes.”7 Our concerns, however, are more limited. We are especially interested in why there are so many defects and what can be done about them. First, though, we want to discuss the reasons for the inevitability of defects, and for that purpose, it is sufficient to focus on four sources of defects.

Complexity

Defects are inevitable in sufficiently complex software. As far back as the 1980s, a panel convened to study the issues with software for President Reagan’s Strategic Defense Initiative noted, “Simply because of its inevitable large size, the software capable of performing the battle management task for strategic defense will contain errors. All systems of useful complexity contain software errors [emphasis added].”8 As Capers Jones notes, one goal of software engineering best practices is to increase the percentage of bugs removed prior to delivery from 85% to something that “approach[es] 99%,” not 100%.9 In contrast, design flaws are not inevitable in, for example, refrigerators, batteries, and bridges even when they exhibit considerable complexity. Software alone combines complexity and inevitable flaws. Thus, no matter how much one invests in development procedures designed to reduce programming flaws, flaws—likely including vulnerabilities—will remain.

Computing Progress Adds to Today’s Complexity

Consumers demand a variety of different types of software, and within those types, they demand software that performs an increasingly rich and varied number of tasks. Your word processor alone will create a variety of different file formats, proofread what you produce, autocorrect typos, allow you to insert graphics, let you select from a variety of fonts, and so on. Laptops costing only several hundred dollars can run games of remarkable complexity


with excellent graphics that can be displayed on large screens. None of this was possible 20 years ago.

One source of the complexity of software is that your computer is a general-purpose machine. It can in theory run whatever can be programmed, which is why back in the 1980s it could already be accurately said that, “All systems of useful complexity contain software errors.”

However, the rapid increase in computing power over the last 30-plus years ensures that computers can run ever more complex software. To handle the variety of complex programs, a contemporary computer is a very powerful, very complex general-purpose machine. A high-end CPU may have billions of transistors. Today’s software, ranging from specific applications, through all the infrastructure software on every computer (e.g., the operating system) to all the infrastructure software behind the web, is all vastly more complex than ever before.

Incidentally, there are also vastly more de facto computers running complex software than ever before, and their number is growing at an astonishing rate. This is IoT, which we will discuss in Chapter 6.

Discrete Mathematics: The Difficulty of Over-Engineering for Safety

Software is different from other engineered products in that sufficiently complex software inevitably has programming flaws. Flaws are not inevitable in, for example, the design and manufacture of refrigerators, batteries, or bridges even when they exhibit considerable complexity. Software alone combines complexity and inevitable flaws.

One reason is that most engineering is governed by continuous mathematics, whereas software is governed by discrete mathematics. Continuous mathematics includes the mathematics of the real numbers, which describes the physics of motion and electricity. Discrete mathematics includes the mathematics of the integers and of strings of letters. For our purposes, the heart of continuous mathematics is the notion of a continuous


function. The definition of a continuous function is typically given in calculus classes using Greek lambdas and epsilons, but what a continuous function means to an engineer is that if, in a continuous system, you make a very small error in one of your inputs, the error in the behavior of your system must also be small. The discrete mathematics that governs software offers no such guarantees. An error in a single line of a million-line program can cause arbitrarily large errors.

One consequence is that there is no way to “over-engineer” for safety in designing software, as one can in designing many physical systems. For example, to design a building to withstand 140 mph winds, the calculations about the necessary material strength, thickness, and so on, to withstand 150 mph winds need to be done, and then it should be built according to those calculations, thus creating an extra margin for safety. There are analogous things to do in many engineering situations but not in the construction of software.

The Consumer Demand for Insecure Software

Complexity and discrete mathematics account for some software defects. However, even within those limits, software engineers could create far more secure software than they typically do today. Why don’t they?

The fundamental reason is that reducing defects, including vulnerabilities, generally requires a longer and more costly development process. Consumers have been unwilling to pay for the added value of security through slightly higher retail prices, and companies dependent on consumer sales don’t offer what consumers don’t want. “Businesses are profit-making ventures, so they make decisions based on both short- and long-term profitability,”10 and the “market often rewards first-to-sell and lowest cost rather than extra time and cost in development.”11 The typical profit-maximizing strategy is to keep costs down and be the first to offer a particular type of software, even if it is imperfect in a variety of ways, including having vulnerabilities.12


This is the profit-maximizing strategy because the damage caused by software vulnerabilities is a negative externality for software manufacturers. As we discussed in Chapter 1, negative externalities warp market incentives. Since they do not bear the loss, profit-driven businesses do not invest time, effort, and money in eliminating negative externalities such as vulnerabilities. Such investment would just reduce profits.

Society as a whole would clearly be better off if software developers did make some investment in reducing vulnerabilities. Developing software with fewer vulnerabilities would consume less time, effort, and money than the billions we lose from unauthorized access. In short, society as a whole would make more efficient use of our economic resources if there were no negative externality.

We can end the externality in two ways: use legal regulation or wait for buyers to change their attitudes and demand secure software. However, buyers might not be able to demand secure software since they might be unable to tell if software is secure. We discuss this issue later in this chapter in the section on lemons markets. Legal regulation can end the externality by requiring software developers to compensate buyers for losses caused by vulnerabilities. The legal requirement of compensation is a way to make buyers’ losses impose corresponding losses on developers. When legal enforcement is sufficiently widespread and certain, profit-driven developers have an incentive to invest in reducing vulnerabilities.

THE “MAKE THEM LIABLE” REMEDY FOR SOFTWARE VULNERABILITIES AND ITS LIMITSMany endorse the following remedy to software vulnerabilities: make software manufacturers liable for the damage caused by hackers when they exploit software vulnerabilities to gain unauthorized access to computers and networks.13 The two core ideas are straightforward: (1) It is unreasonable for manufacturers to produce vulnerability-ridden software when they know how


to reduce vulnerabilities, and (2) give them an incentive to behave more reasonably by making them liable for producing vulnerability-ridden software. We consider three legal options for implementing this idea: strict liability, negligence liability, and liability for defective product design.

Strict Liability

Strict liability makes you responsible for losses you cause, even if the loss was not your fault. Strict liability would impose the entire cost of software vulnerabilities on software developers. The difficulty is that software vulnerabilities differ fundamentally from the type of case in which the law typically imposes strict liability.

Classic examples of strict liability are inherently dangerous activities. Keeping wild animals is an example. Keep wild animals and you are strictly liable for the harm they cause. To compare software, first note that when you keep wild animals, you can take precautions to prevent the harm, and the more precautions you take, the more the risk of harm decreases. Although you cannot completely eliminate the risk, you can make it as small as you like. You could keep your tigers in a cage within a cage, with top-of-the-line locks, 24-hour electronic monitoring, and as many round-the-clock human guards as you chose. You will eventually decide, at some point, that the time, effort, and money required for increased precautions are not worth the degree of risk reduction they offer, but the point is you have the choice.

Software developers don’t have that choice. They cannot drive risks arbitrarily close to zero by spending larger and larger sums. There will always be some security vulnerabilities in any system. No matter what they do, in sufficiently complex programs enough vulnerabilities will remain to create a significant risk of loss. Compare this to keeping wild animals. Suppose that the animals would escape once a month—no matter how careful you were. How many people would keep animals if they faced once-a-month liability for the harm they would cause? Zoos might, but visiting them would be very costly since their fees would have to generate


enough money to cover the amounts they would constantly pay out in compensation (or in insurance premiums). Strict liability could create this situation for software vulnerabilities. It would significantly discourage the sale of existing software and the development of new programs. Only where developers could charge enough to cover their inevitable liability would they offer software for sale. This is a highly undesirable outcome in a world now highly dependent on digital technology and on its continued development and refinement.

We can avoid the problems that confront strict liability by making software developers liable for only the losses caused by some vulnerabilities, not all of them. This is what negligence liability and liability for defective design do.

Negligence Liability

To avoid negligence liability, you must act reasonably. If you fail to do so, you are liable for the losses caused by the foreseeable harms of your actions. Thus, a software developer would be liable in negligence for losses resulting from a vulnerability only if the vulnerability was a foreseeable result of the developer’s failure to act as a reasonable developer would. Our main objection to negligence liability is that it only sufficiently reduces software vulnerabilities if it is combined with a solution to the lack of information problem we discussed in the previous chapter. We argue for this point in the next section.

Another difficulty with negligence liability is that determining what is reasonable behavior for creating low-defect software behavior can be tricky because of the inevitability of some software defects. That difficulty, though real, is almost certainly overstated by major software producers.

Liability for Defective Product Design

You are liable for defective product design when the use of the product involves a foreseeable and unreasonable risk of harm. Holding software developers liable for defective design would


create an incentive for developers to adopt vulnerability-reducing development techniques and procedures, as long as courts hold that not adopting them creates a foreseeable and unreasonable risk of harm. Given the references to foreseeability and unreasonableness, you may well wonder how liability for defective design differs from negligence liability. The answer lies in the application and interpretation of foreseeability and unreasonableness in the specific context of product design. For our purposes, we need not pursue that issue. Our objection to defective design liability is the same as our objection to negligence liability. It will only sufficiently reduce software vulnerabilities if it is combined with a solution to the lack of information problem.

LACK OF INFORMATION ABOUT COSTS AND PROBABILITIESTo see the problem, consider that the law cannot simply tell software developers to “invest more.” That would be like telling students they must write a paper of a certain number of pages to get a passing grade but not telling them how many pages. Some would write too little; some, too much. The same would happen with software. Some developers would invest too much; some, too little. Ideally, we want developers to offer buyers software options that, when embedded in the overall system, yield a combination of defensive measures that best minimizes the sum of the cost of the defensive measures and all the expected losses with those defensive measures. The problem is the lack of information needed to estimate the expected loss avoided for a given company from adopting a type of defense against a type of data breach. When courts cannot give businesses a sufficiently clear indication of what they should do instead of following prevailing industry practices, it is quite difficult to convince a court that a business that followed those practices acted unreasonably.14 Standard negligence liability will not create a sufficient incentive to adopt vulnerability-reducing development practices.

Essentially, the same arguments lead to the same conclusion for product liability for defective design. A product is defective in


design when its use involves a foreseeable and unreasonable risk of harm. Holding software developers liable for defective design would create an incentive for developers to adopt vulnerability-reducing development techniques and procedures, as long as the courts held that not adopting them would create an foreseeable and unreasonable risk of harm. However, holding developers to this standard would not provide an adequate incentive to reduce vulnerabilities. The reason is the same as in the negligence case: following existing custom and practice is evidence of reasonableness, and, as a practical matter, it is difficult to overcome a business’s claim that followed the prevailing industry practices and hence acted reasonably.15 Extreme cases aside, it is unlikely that courts will hold that not adopting vulnerability-reducing development techniques and procedures creates an unreasonable risk of harm.

Our critique of negligence and products liability applies to any standard that incorporates a reasonableness requirement for software development where the courts will rely primarily on custom and prevailing industry practice to define what counts as “reasonable.” Relying on reasonableness requirements is nonetheless quite common. As information security law expert Tom Smedinghoff notes, “Laws and regulations rarely specify the security measures a business should implement to satisfy its legal obligations. Most simply obligate companies to establish and maintain ‘reasonable’ or ‘appropriate’ security measures, controls, safeguards, or procedures, but give no further direction or guidance.”16

To avoid misunderstanding, we should emphasize that we are by no means rejecting relying on legal regulation that imposes reasonableness requirements. Quite the contrary, we will propose such regulation in the chapters that follow. We address the lack of information problem in the next chapter. In the case of software vulnerabilities, however, there is an attractive alternative. Legal liability focuses on changing the behavior of businesses. There is another option: change the behavior of consumers.


CHANGING CONSUMER DEMANDIf enough consumers demanded more secure, less vulnerability-ridden software, profit-driven businesses would provide it. But how does one change what consumers currently demand? Our answer, which we have worked out elsewhere, argues for the creation of a market norm under which software developers offer adequately secure software. We show how a combination of legal regulation and education can make it the norm for consumers to demand sufficiently secure software.17 But, one might well ask, what about the lack of information problem?

After all, sufficiently secure software is software that adequately promotes the business and consumer risk management goals. We’ve argued that meeting those risk management goals requires information about costs and probabilities. So how could a process of legal regulation and education to change norms generate the necessary information? It can’t. We explain how to generate the necessary information in Chapter 4. We conclude this chapter by considering another problem that appears to confront our “consumer demand” proposal.

A LEMONS MARKET FOR SOFTWARE?Our proposal that we should rely on the norm that consumers demand sufficiently secure software requires that consumers be able to tell secure from insecure software. Can they? The stakes are high. If they cannot, the result is a “lemons market” in which only insecure software is available. We explain a lemons market, then consider whether consumers can tell secure from insecure software.

We explain a lemons market using a version of the “used car” example first employed by the economist George Akerlof in his seminal article, “The Market for Lemons.”18 Suppose a town has 300 used cars for sale: 100 good ones worth $6,000, 100 so-so ones worth $4,000, and 100 lemons worth $2,000. Buyers cannot tell the difference between a good and bad car; thus, buying a used


car means entering a lottery in which the buyer has a 1/3 chance of getting a good car, a 1/3 chance of getting a so-so car, and a 1/3 chance of getting a lemon. The expected value of that purchase is $4,000, so rational buyers will be willing to pay at most $4,000. Sellers who value their good cars at over $4,000 will not offer those good cars for sale, so the market will contain only lemons worth $2,000 and not-so-good cars worth $4,000. As a result, the expected value of a used car drops to $3,000, so rational buyers will not pay more than $3,000 for a car, and now sellers who value their cars above $3,000 do not offer them for sale, and thus only the lemons are left on the market. In general, a lemons market exists when four conditions are fulfilled: (1) the products on the market vary significantly in quality, that is, the extent to which they have certain properties that buyers are willing to pay more for; (2) buyers cannot discriminate among products with different degrees of quality, but sellers can at least partially distinguish them; (3) there is no reliable signal of quality (i.e., sellers with an excellent car have no way to reliably disclose this fact to buyers); and (4) buyers know there is a mix of products on the market.

Are these four conditions fulfilled for software vulnerabilities?In answering this question, it is important to distinguish between

two markets: the market for security software and systems (such as firewalls, antivirus software, or secure USB memory sticks), and the market for other sorts of mass-consumer software. Bruce Schneier has argued convincingly that the former market is a lemons market.19 Others have picked up on his claim and argued that it may also apply to software that is (relatively) secure—that is, software which is relatively free of vulnerabilities.20 We are not so sure. While there are strong arguments that security software is a lemons market, it is unclear whether secure software is a lemons market. Conditions (2) and (4) are arguably fulfilled, but (1) and (3) are problematic. We first briefly review the arguments in favor of regarding conditions (2) and (4) as being fulfilled. Condition (2): Typical consumers do not have the expertise to distinguish by inspecting the software between secure and insecure software,21 while the developers do


know something about what production practices they are using. Condition (4): Buyers—or at least a significant portion of buyers—do know that the market contains both vulnerability-ridden and not so vulnerability-ridden software.22

Condition (1) requires (in part) that buyers be willing to pay more for software with few vulnerabilities than for similar vulnerability-ridden software. At the moment, this is not true. Buyers are, on the whole, not willing to pay more for more secure software.

Condition (3) requires that there do not exist any reliable signals that differentiate vulnerability-ridden from similar software with significantly fewer vulnerabilities. Typical consumers do not have the expertise to distinguish by inspecting the software between vulnerability-ridden software and software with significantly fewer vulnerabilities. Inspection is not, however, the only way to determine the extent to which software suffers from vulnerabilities. The general quality of the software is a moderately reliable signal of the extent to which it contains vulnerabilities. Vulnerabilities are a kind of flaw, or defect, in the software, and it is reasonable to assume that their occurrence correlates with the occurrence of other flaws, such as a tendency to crash or give wrong answers. Indeed, it is routine not to distinguish sharply between defects and vulnerabilities. As security experts observed back in 2004, “Software defects are the single most critical weakness in computer systems …. [S]oftware defects lead directly to software exploit[ation].”23 The correlation between vulnerabilities and defects is sufficiently strong that at least some buyers will infer that improperly functioning software is likely to contain significant vulnerabilities. This signaling mechanism is far from perfect, but sufficient detection does not require that all or most buyers detect vulnerability-ridden software, just that enough do to impose losses on sellers who offer such software. Thus, there is very possibly a signaling mechanism that is strong enough to prevent a lemons market.

If this signaling mechanism is not enough, there are other ways to create the required signal. These include warranties (signaling


the manufacturer’s confidence that the software is secure), industry standards (compliance signals that the software is secure), consumer protection laws (compliance signals that the software is secure), and product certification (signals the certifier’s confidence that the software is secure).

ARTIFICIAL INTELLIGENCE: A FUTURE SOLUTION?If we could teach a machine learning system what a software defect looks like, the system could run through millions of lines of code looking for the patterns indicative of defects. We do not yet know how to teach a system what a defect looks like, but there is academic work on that topic.24 If sufficiently cheap and accurate machine learning detection became a standard part of software development, that could greatly reduce the number of software defects (and thus vulnerabilities). Bruce Schneier imagines a future in which we would say, “Remember those years when software vulnerabilities were a thing, before ML [machine learning] vulnerability finders were built into every compiler and fixed them before the software was ever released? Wow, those were crazy years.”25 For now, however, and for the immediate future, significant software defects are the reality.

CONCLUSIONThe next chapter turns to security management vulnerabilities. The “change consumer demand” approach we suggest for software vulnerabilities will not work for security management vulnerabilities. We suggest legal liability for mismanagement instead. The suggestion opens us to our own objection that we lack the information about costs and probabilities necessary for fully effective legal regulation. We address this problem in Chapter 4.

ENDNOTES

1. The Joint Task Force on Computing Curricula, Association for Computing Machinery (ACM), IEEE Computer Society, “Computer Science Curricula 2013: Curriculum Guidelines for


Undergraduate Degree Programs in Computer Science,” December 20, 2013, https://dl.acm.org/citation.cfm?id=2534860.

2. See, e.g., E. Roberts et al., Computing Curricula 2001: Computer Science (IEEE Computer Society Press, 2001).

3. For example, I. J. Hayes, “Applying Formal Specification to Software Development in Industry,” IEEE Transactions on Software Engineering SE-11, no. 2, 1985: 169–78; A. MacCormack et al., “Trade-Offs between Productivity and Quality in Selecting Software Development Practices,” IEEE Software 20, no. 5, 2003: 78–85.

4. “Top 50 Products Having Highest Number of CVE Security Vulnerabilities,” accessed December 30, 2018, https://www.cvedetails.com/top-50-products.php.

5. K. Nayak et al., “Some Vulnerabilities Are Different Than Others,” in International Workshop on Recent Advances in Intrusion Detection (Springer, 2014), 426–446.

6. “New Kit, Same Player: Top 10 Vulnerabilities Used by Exploit Kits in 2016,” accessed January 3, 2019, https://www.recordedfuture.com/top-vulnerabilities-2016/.

7. C. Jones and O. Bonsignour, The Economics of Software Quality (Upper Saddle River, NJ: Addison-Wesley Professional, 2011).

8. Strategic Def. Initiative Org., Dept of Def., 19980819-140. Eastport Study Group: Summer Study 1985. A Report to the Director, Strategic Initiative Organization 14, 1985, available at http://www.dtic.mil/dtic/tr/fulltext/u2/a169210.pdf.

9. C. Jones, Software Engineering Best Practices: Lessons from Successful Projects in the Top Companies (New York: McGraw-Hill, 2010), xxvi.

10. B. Schneier, “Information Security and Externalities,” Schneier on Security (blog), January 2007, https://www.schneier.com/essays/archives/2007/01/information_security_1.html.

11. E. H. Spafford, Remembrances of Things Pest, 53 Comm. ACM 35, no. 36, 2010.

12. See generally, C. Shapiro and H. R. Varian, “Information Rules: A Strategic Guide to the Network Economy,” 50–51, 1999. The economics and information security community has developed Shapiro and Varian’s initial insights. Much of this work has been reported in the annual Workshop on the Economics of Information Security since 2002, https://econinfosec.org/. For a good early survey, see R. Anderson and T. Moore, “Information Security: Where Computer Science, Economics and Psychology Meet,” 367 Phil. Transactions Royal Soc’y A 2717, 2009: 2721–22.


13. Bruce Schneier has been a prominent advocate of this view. See B. Schneier, “Liability Changes Everything,” Schneier on Security (blog), November 2003, http://www.schneier.com/essay-025.html, and B. Schneier, Click Here to Kill Everybody: Security and Survival in a Hyper-Connected World (New York: W. W. Norton & Company, 2018).

14. Egregiously bad programming practices are sometimes an exception.

15. Evidence of industry practices is relevant under both of the main tests used to determine defectiveness—the “risk/utility test” (a product is defective when its risk of harm exceeds its benefits), and the “consumer expectations” test (a product is defective when it fails to meet the reasonable expectations of consumers).

16. T. J. Smedinghoff, “Defining the Legal Standard for Information Security: What Does ‘Reasonable’ Security Really Mean?” in Securing Privacy in the Internet Age, eds. Chander, G., Gelman, L., and Radin M. J. (Stanford University Press, 2008), 19–40.

17. R. H. Sloan and R. Warner, Unauthorized Access: The Crisis in Online Privacy and Information Security (Boca Raton, FL: Chapman & Hall/CRC Press, 2013).

18. G. A. Akerlof, “The Market for ‘Lemons’: Quality Uncertainty and the Market Mechanism,” Quarterly Journal of Economics 84, no. 3, 1970: 488–500.

19. B. Schneier, “How Security Companies Sucker Us with Lemons,” Wired, April 19, 2007, http://www.wired.com/politics/security/commentary/securitymatters/2007/04/securitymatters_0419.

20. See D. Barnes, “Deworming the Internet,” Texas Law Review 83, no. 1, 2004.

21. B. Schneier, “How Security Companies Sucker Us with Lemons,” Wired, April 19, 2007, https://www.wired.com/2007/04/securitymatters-0419/.

22. R. W. Hahn and Anne Layne-Farrar, “The Law and Economics of Software Security,” Harvard Journal of Law & Public Policy 30, 2006: 302.

23. G. Hoglund and G. McGraw, Exploiting Software: How to Break Code (Addison-Wesley Professional, 2004). These lines come at the end of an introductory section of the book that moves from discussing famous software defects that had nothing to do with security and attackers to discussing defects that constitute security holes. Two examples of non-security defects the authors give are NASA’s 1999 Mars lander software failure, where a metric versus


English units error caused the loss of the $165 million system and the Denver International Airport automated baggage handling system fiasco.

24. J. J. Kronjee, “Discovering Vulnerabilities Using Data-Flow Analysis and Machine Learning” (Master’s Thesis, Open University, 2018), https://dspace.ou.nl/bitstream/1820/9725/1/Kronjee%20J%20IM9906%20AF%20scriptie.pdf.

25. B. Schneier, “Machine Learning to Detect Software Vulnerabilities,” Schneier on Security (blog), January 15, 2019, https://www.schneier.com/blog/archives/2019/01/machine_learnin.html.

DETECTING BOTNETS AND

UNl<NOWN NETWORI<

ATTACl<S IN BIG TRAFFIC

DATA

This chapter is excerpted from

Botnets Architectures, Countermeasures, and Challenges

by Georgios Kambourakis, Maries Anagnostopoulos, Weizhi Meng, Peng Zhou

© [2019] Taylor f:t Francis Group. All rights reserved.

0 Learn more

Q

https://www.routledge.com/Botnets-Architectures-Countermeasures-and-Challenges/Kambourakis-Anagnostopoulos-Meng-Zhou/p/book/9780367191542

https://www.routledge.com/Botnets-Architectures-Countermeasures-and-Challenges/Kambourakis-Anagnostopoulos-Meng-Zhou/p/book/9780367191542

Chapter 7

Detecting Botnets andUnknown NetworkAttacks in BigTraffic Data

Luis SacramentoINESC-ID, Instituto Superior Técnico, Universidade de Lisboa, Portugal

Ibéria MedeirosLasige, Faculdade de Ciências, Universidade de Lisboa, Portugal

João BotaVodafone Portugal, Portugal

Miguel CorreiaINESC-ID, Instituto Superior Técnico, Universidade de Lisboa, Portugal

Contents7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2387.2 Context and Related Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .241

7.2.1 Network Flows and Basic Flow Tools . . . . . . . . . . . . . . . . . .2417.2.2 Intrusion Detection based on Network Flows. . . . . . . . . . . . .242

7.3 The FlowHacker Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2437.4 Obtaining Vectors of Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .245

237

7.4.1 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2457.4.1.1 Layer 2, 3, and 4 Features . . . . . . . . . . . . . . . . . . .2467.4.1.2 Statistic Features . . . . . . . . . . . . . . . . . . . . . . . . . .2477.4.1.3 Threat Intelligence Feature . . . . . . . . . . . . . . . . . . .247

7.4.2 Flow Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . .2487.4.2.1 Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2487.4.2.2 MapReduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2487.4.2.3 Mapper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2497.4.2.4 Reducer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2497.4.2.5 Normalization . . . . . . . . . . . . . . . . . . . . . . . . . . .250

7.4.3 Obtaining Threat Intelligence . . . . . . . . . . . . . . . . . . . . . . .2517.5 Detecting Network Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .251

7.5.1 Similarity Aggregation . . . . . . . . . . . . . . . . . . . . . . . . . . . .2527.5.1.1 Choosing the Number of Clusters . . . . . . . . . . . . . .2527.5.1.2 Describing Clusters . . . . . . . . . . . . . . . . . . . . . . . .253

7.5.2 Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2547.5.3 Learning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .255

7.6 FlowHacker Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2557.7 Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .256

7.7.1 Evaluation with Synthetic Data . . . . . . . . . . . . . . . . . . . . . .2567.7.1.1 Cluster Analysis . . . . . . . . . . . . . . . . . . . . . . . . . .2577.7.1.2 Unsupervised Classification. . . . . . . . . . . . . . . . . . .2597.7.1.3 Result Validation . . . . . . . . . . . . . . . . . . . . . . . . .260

7.7.2 Evaluation with Real Data . . . . . . . . . . . . . . . . . . . . . . . . .2607.7.2.1 Source Aggregation Key Clustering . . . . . . . . . . . . .2617.7.2.2 Destination Aggregation Key Clustering . . . . . . . . . .2637.7.2.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .265

7.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265

7.1 IntroductionInternet service providers (ISPs) supply their customers communication servicesthat are easily abused by cyber-criminals. The major ISPs are interested inreducing malicious activity, and detecting it is often a first step toward thatgoal. However, these organizations now run backbones with bandwidths in theorder of 1 to 10 or even 100 Gbps [1], which support communication servicesthat were not possible in the past, but also make detection a difficult challenge.

A particular pernicious case of network abuse are botnets. These networks ofcompromised hosts (zombies or bots) are platforms for a large set of illegalactivities, e.g., for executing distributed denial of service (DDoS) attacks, dis-seminating malware, running phishing attacks, and supporting click fraud [2].Several severe botnet attacks occurred recently, affecting millions of Internet users

238 ■ Botnets

[3,4]. ISPs have necessarily a role to play in the mitigation of botnets, as they arekey traffic intermediaries [5]. Therefore, ISPs have to be able to detect them.

ISPs often use network intrusion detection systems (NIDSs) for identifyingmalicious activity. Classical NIDSs do deep packet inspection (DPI), i.e., theyanalyze the payload of the packets passing through specific points of the network(e.g., an edge or border router), looking for a certain signature or behavioralpattern. Classical NIDSs also fall in one of two categories: signature-based NIDSsand behavior-based NIDSs. These approaches require, respectively, knowledgeabout existing attacks (signatures) and traffic without attacks (normal behavior)for training purposes, neither of which is simple to obtain [6]. This kind ofanalysis is feasible in reasonably slow link connections but not in modern high-speed backbones. Furthermore, nowadays most of the traffic payload is encrypteddue to the adoption of secure protocols, such as TLS, HTTPS, SSH, and IPSec,which makes this kind of inspection even harder and less useful [7].

The difficulty of monitoring high-speed traffic has led Cisco to introduce theconcept of network flows in the context of the NetFlow router feature [8]. Theconcept was later adopted by all major router vendors and standardized by the IETF[9]. A flow can be defined as a sequence of packets with a common set of features,passing through an observation point, in a given interval of time. Flows are a way ofmonitoring communication in a summarized way, without inspecting the content ofthe packets, using instead high-level information about connections (source/destina-tion IP address, source/destination port, etc.) but not the data transferred itself.Analyzing this information is more efficient than doing DPI in terms of protection ofthe privacy of users and consumption of computational resources, once flows do notcarry payload content, requiring less processing to be analyzed.

Network flows, or simply flows, are an alternative to the previously mentionedapproaches for network intrusion detection. Network flow analysis allows detect-ing internal and external actions like network misconfiguration and policyviolation [10]. Flows allow detecting many network layer and transport layerattacks. They also allow detecting botnets, because bots perform identifiablenetwork activity such as contacting command and control (C&C) servers orDDoS attacks. Flows generically do not allow detecting application layer attackssuch as SQL injection, cross-site scripting, buffer overflows, races, etc., for thesimple reason that they do not contain the message payloads.

The use of machine learning (ML) in the context of network intrusion detection isfar from new. A major application of ML is behavior-based (network) intrusiondetection, also called anomaly detection [11,12]. Both traditional signature-basedNIDSs and NIDSs based on flows can also use ML techniques, but the precision andaccuracy of these systems depend on the completeness of the knowledge they haveabout the threats that they will detect, as they need to be fed and trained with thatknowledge. ML techniques aim to provide knowledge to such systems, allowingthem to discover hidden patterns in input data based on the knowledge they learned,

Detecting Botnets ■ 239

and classify that data. However, even when using ML, there are challenges inanalyzing network flow data, such as the huge amount of traffic flow becomingfrom larger and faster networks as, for instance, connection links of ISPs.

This chapter presents a new approach to detect malicious traffic, even if as partof new attacks, and to identify the malicious hosts involved by inspectingnetwork flows. The approach uses a combination of unsupervised ML techniques,without a priori knowledge, and threat intelligence information to achieve itsgoal. The approach is based on the following key insights:

■ There is much more normal traffic than malicious traffic.■ Malicious traffic is qualitatively different from normal traffic.■ Similar traffic within each category (normal, malicious) can be summarized

using unsupervised ML.

Our approach involves dividing traffic (flows) into clusters. The larger clusterstypically correspond to normal traffic, so the smaller clusters are the ones we haveto worry about. For the latter, we propose a classification method based onunsupervised ML to classify them as malicious or benign, so detecting malicioustraffic and identify malicious hosts. This classification allows reducing drasticallythe amount of time spent in analyzing the flows, reducing the size of the problemof processing the amount of traffic at the speed of ISP networks.

The approach works in a loop, iteratively and continuously detecting networkattacks and malicious hosts. Between iterations, clusters are classified and learned, sothat this knowledge can be used in the following iterations. This form of learningprovides increasing autonomy to the system and may significantly reduce thenetwork managers’ constant need for intervention, although not being completelyfree from human intervention, as no NIDS is. In fact, human intervention isunavoidable when the goal is to detect attacks without requiring either previousknowledge about attacks (signatures) or traffic without attacks (clean traffic for training).

The chapter also presents the FlowHacker NIDS that implements our approach.This tool uses the Hadoop MapReduce platform [13,14] to summarize networksflows and a set of ML algorithms to process these summaries, besides providing visualtools for human analysis. We evaluated the FlowHacker NIDS with two kinds oftraffic and it identified botnet C&C servers, SSH brute-force attacks, and denial ofservice events. For validating the system, we used a synthetic traffic flow data set [15].For testing the system, we used real data from a ISP, a large Portuguese telecommu-nications company with a few million customers, which provides Internet, TV-over-IP, phone-over-IP, and GSM/3G/4G cellular phone services. FlowHacker was ableto detect several cases of botnet activity.

The main contributions of the chapter are: (1) an approach for improving networksecurity based on the inspection of network flows by using a combination ofunsupervised ML techniques to detect intrusions; (2) an iterative learning process; (3)

240 ■ Botnets

a NIDS that implements the approach; (4) an experimental evaluation that showsthe ability of the system to detect intrusions in computers communication usingnetwork flows.

The remaining of the chapter is organized as follows. Section 7.2 presentsbackground on network flow field and discusses related work. Section 7.3 presentsthe approach, Sections 7.4 and 7.5 present more details about it, and Section 7.6 itsimplementation. Section 7.7 presents and discusses the evaluations results, andSection 7.8 concludes the chapter.

7.2 Context and Related WorkThis section provides an overview of intrusion detection using network flows.Section 7.2.1 provides some insight on some of the existing tools to perform flowanalysis. Section 7.2.2 gives an overview of some of the most addressed networkintrusions and respective works/tools that show how to detect them using a flow-level analysis rather than payload inspection.

7.2.1 Network Flows and Basic Flow Tools

The first network protocol to handle network flows was NetFlow, developed byCisco [8]. It consists of a built-in feature in the Cisco routers, and is used tocollect and export flow records. Since then, it has been evolved and its recentversion—NetFlow v9—already includes integration with other protocols, such asMultiprotocol Label Switching (MPLS).

Network flow technology is built-in in network devices, so it allows to select,from all the traffic passing through that device, the traffic that matches the set offeatures that were previously defined by the network administrator, in order toobtain what he wants to analyze. For example, by deploying this technology ina border router, all of the traffic going in and out of that network will be filteredby NetFlow. Upon the reception of an IP packet, the network device looks atthat packet’s fields in order to find any matching feature with those previouslydefined. In case the packet’s features do match, then an entry is created in a datastructure called flow cache, for that flow. Note that a flow may correspond toseveral packets, and many different flows can be collected.

Apart from NetFlow, many other vendors have their own implementation forflow collection and exporting. Examples of such implementations are NetFlow-lite, sFlow, and NetStream. However, due to the heterogeneous nature of thesetechnologies between different vendors, the Internet Engineering Task Force(IETF) created the IP Flow Information eXport (IPFIX) protocol [9] to standar-dize flow collection and exportation, allowing thus for the clients to easily deploytheir flow-based applications. As previously stated, packets that share common


properties are grouped in flows, and in the IPFIX terminology these propertiesare referred as flow keys. They can form, for example, the following tuple:(IP_source, IP_destination, port_source, port_destination, type_of_Service).

In order to simplify the collection and extraction of flows, some tools weredeveloped. The nfdump tool [16] is one of them. The tool is compatible withversions v5, v7, and v9 of NetFlow and supports data conversion to plain text (inform of txt files). nfdump reads the NetFlow data, stores it into binary files, andperforms some analysis on it, such as some statistics and aggregation.

The System for Internet-Level Knowledge (SiLK) tool [17] is a widelydeployed flow analysis tool developed by the CERT Network Situational Aware-ness Team. It is compatible with both IPFIX and NetFlow (versions v5 and v9).Like nfdump, it allows to convert NetFlow data to some specific format, and alsohas built-in tools to analyze these files, such as performing filtering on thegathered flows and retrieve statistical data.

7.2.2 Intrusion Detection Based on Network Flows

Several papers presented flow-based intrusion detection schemes for specificnetwork attacks, including botnets [2,18–22], and others such as port scans[23,24], worms [25–27], and denial of service [28–30]. Each of these approacheswas designed to detect only one of these attack types, but they are related to ourwork and useful to explain how flows can be used to detect a certain attack.

An increasing trend in intrusion detection systems is the use of ML techniques[10,31]. ML can be defined as a collection of methods that aim to attainknowledge by building an intelligent system through the observation of patternsin a given environment [10]. This knowledge may be refined and improved ateach iteration, by learning from previous experiences and observations. Suchmethods have been used in several and different applications, in many differentfields of science, such as natural language processing (NLP), speech recognition,bioinformatics, spam detection, network intrusion detection, among many others.

ML algorithms can be divided into two major types: supervised learning andunsupervised learning. The first one relies on a labeled training data set. The data setconsists in a set data labeled and categorized in classes by humans that aims to train thesystem, making correspondences between features and their meaning or interpretationthat is expected to the system. After the training phase, the system is ready to classifyinput data based on the learning that obtained during the training. Examples ofsupervised algorithms are Decision Trees, Naïve Bayes, and Support Vector Machine(SVM). On the other hand, unsupervised learning does not have trained labeled dataset. In contrast, it receives as input a unlabeled feature vector, and then it is processedfor discovering similar groups into it. Clustering is an example of this kind of learning,and K-Means is one of the best-known algorithms for clustering.

242 ■ Botnets

In the field of network intrusion detection, ML techniques have been able toclassify network traffic and identify both anomalous patterns and potentiallyharmful users [11,12,32]. When NIDS systems integrate this technique, theadopted strategy is usually behavior-based detection (or anomaly detection), inwhich normal traffic patterns are differentiated from anomalous ones. It focusesits attention on finding patterns that would not be expected from the user’sbehavior. Unlike what signature-based NIDSs detect, these patterns are unknownto the system, as they are trained with intrusion-free data. However, thisapproach requires clean traffic for training, i.e., traffic that does not containattacks, which is difficult to obtain in ISP networks in production.

Portnoy et al. presented a scheme to detect intrusions based on clustering that isneither behavior—nor signature-based [33]. However, that work does not considerthe iterative model we do and does not use flows. The Unsupervised NetworkIntrusion Detection System (UNIDS) was able to detect unknown attacks withoutrequiring any labelling, signatures or training [34]. UNIDS uses various clusteringtechniques such as sub-space clustering, density-based clustering and evidenceaccumulation. However, it does not consider the iteration process we do. Gonçalveset al. follow an approach closer to ours but inspect logs, not flows [35]. That workand a few others use open source threat intelligence in combination with othertechniques [35,36]. A short version of the present work appeared before, but did notpresent the approach in detail [37].

7.3 The FlowHacker ApproachThe FlowHacker approach does flow processing using unsupervised ML algorithmswith the assistance of threat intelligence to detect unknown network attacks. Asexplained in the introduction, the approach is based on the assumption that most ofthe traffic is legitimate, so malicious traffic is much less, as well as that malicioustraffic is qualitatively different from normal traffic. Taking this into account, theapplication of the unsupervised ML algorithm allows splitting the malicious trafficfrom the clean traffic, so that the biggest clusters are those containing normal traffic,whereas the smaller ones are those that may be malicious (although that is notmandatory; there may small clusters of legitimate traffic). These assumptions incombination with the use of flows allow to cover (1) the difficulty of reacting to anunknown pattern when real traffic is analyzed, and (2) the slow processing andanalysis of the traffic payload. The first drawback may be countered by using anunsupervised ML algorithm, and the second by performing the analysis at flow level.

The approach works in a loop, improving the knowledge that has been acquired inprevious iterations. This allows improving detection performance with time. For eachloop iteration all phases involved in the approach are executed. Therefore, for each setof collected flows, the tool gains insights from them, improves such insights with


threat intelligence information, applies an unsupervised algorithm on the improvedinsights for getting clusters, and then classifies the smaller clusters as being maliciousor benign hosts by using a classifier method based on a unsupervised algorithm.Lastly, these new classified clusters are added to the existing knowledge for theunsupervised data set to be used in the next loop iterations. This learning phasebetween loop iterations allows increasing knowledge gradually with every iteration.

The approach comprises six phases, as shown in Figure 7.1:

1. Flows collection: to collect flows from different hosts or routers belonging toa network infrastructure, corresponding to a certain period of time (e.g., a day).Each flow summarizes a set of packets collected from a host during a period oftime.

2. Features extraction: to extract data from the collected flows in order to createvectors of features that allow characterizing the flows. Flows are filtered toextracting relevant data, afterward this data goes through MapReduce forgetting statistics and summarizing their values, then they are normalized andthe vectors created (see Section 7.4.2).

3. Threat intelligence retrieving: to automatically retrieve information aboutthreats from online databases, namely blacklists of subnets and IP addresses.Afterward, this information is used to complement and complete the data ofvectors of features (see Section 7.4.3).

4. Similarity aggregation: to apply an unsupervised ML algorithm over thefeature vectors in order to aggregate similar vectors (vectors with similarfeature values), resulting in clusters that represent hosts having a similar

Infrastructurenetwork

Flowscollection

Features extraction

Filtering

Map & Reduce

Normalization

Online databases Threat

Intelligent retrieving

Vectors creation

Similiarityaggregation

LearningManual labeling

Automaticlabeling

Detection

Dataset

Malicioushosts

detected

Figure 7.1 Overview of the approach and its six phrases: flows collection,features extraction, threat intelligence retrieving, similarity aggregation, detection,and learning.

244 ■ Botnets

behavior. The biggest clusters are assumed to contain clean traffic and thesmaller clusters have to be further analyzed (see Section 7.5.1).

5. Detection: to detect unknown network attacking hosts. Based on the knowledgeacquired in the learning phase a classification method using unsupervised MLclassifies the small clusters as being malicious or clean traffic. In the first loopiteration, the classification is done manually, but automatically subsequentlywhen the system has already knowledge from the previous classifications (loopiterations). Clusters are labeled with its classification and the classified data set isupdated with them (see Section 7.5.2).

6. Learning: to learn the new cluster classifications resulting from the detectionphase. For each loop iteration, the clusters classified in the detection phaseare learned for later to be used as knowledge in the next loop iterations bythe classifier (see Section 7.5.3).

The following sections present the approach in detail.

7.4 Obtaining Vectors of FeaturesThis section presents the process of extracting data from the collected flows andretrieving threat intelligence information from online databases to create the vectorsof features with this data. This process constitutes the second and third phases of thepresented approach—extraction features and threat intelligence retrieving (see Figure 7.1).

As said, a flow summarizes a set of packets with similar characteristics that wereobserved during a period of time. However, a flow summarizes packets coming fromdifferent source IPs and going to different destination IPs. Therefore, to obtaininformation about individual hosts we must aggregate flows by source and destina-tion IPs. These aggregated flows summarize traffic sent or received by each host.

To use these aggregated flows in ML algorithms, we represent each of them bya vector of features, i.e., a vector of attributes that characterize these aggregates in a waythat is useful for our purposes (intrusion detection). The features can be extractedfrom the aggregated flows directly, or from external sources of threat intelligence data(e.g., the information if a certain IP address appears in a certain blacklist of not).

The next sections present the features we defined to compose vectors offeatures and these two phases in detail.

7.4.1 FeaturesThe features have to be carefully chosen because the accuracy and precision of theapproach depend on that choice. We characterize aggregated flows using 19features from 5 categories (Table 7.1): three to represent characteristics fromlayer 2, layer 3, and layer 4 of the TCP/IP stack, one for statistics about theaggregated flow, and one for threat intelligence data.


7.4.1.1 Layer 2, 3, and 4 Features

We defined 11 features to represent characteristics from layers 2, 3, and 4,respectively, 3, 4, and 4 features (lines 2 to 12 of Table 7.1).

Table 7.1 Set of features that describe an aggregated flow in the form ofa vector

Feature Description #

L2

AggregationKey IP address used as identifier (to which theother features relate to)

1NumSIPs/NumDIPs The number of IP addresses contacted

LocationCodeCode for the country associated with theaddress

L3

NumSportsThe number of different source portscontacted

2

NumDportThe number of different destination portscontacted

3

ICMPRateThe ratio of ICMP packets, and total numberof packets

13

SynRateThe ratio of packets with a SYN flag and thetotal number of packets 14

L4

NumHTTP The number of packets to/from port 80 (HTTP) 4,8

NumIRCThe number of packets to/from ports 194 or6667 (IRC) 5,9

NumSMTPThe number of packets to/from port 25(SMTP) 6

NumSSH The number of packets to/from port 22 (SSH) 7,10

Statistic

TotalNumPkts The total number of packets exchanged 11

TotalNumBytes The overall sum of bytes 15

PktRate The ratio of the number of packets sent andits duration 12

AvgPktSize The average packet size 16

TI

BadSubnetThis field expresses whether the IP addressbelongs to a blacklisted subnet

MaliciousIPThis field expresses whether the IP address isblacklisted

OpenVaultBlacklistedIPSimilar to the above but from another data-base [38]

MaliciousASNThis field signals if the IP address belongs toa blacklisted ASN

246 ■ Botnets

From layer 2 are extracted the IP addresses, both source and destination. However,it is necessary to identify a vector of features uniquely, therefore, packets comingfrom different source IPs and different destinations IPs constitute different vectors offeatures. We defined the AggregationKey feature to have this role of identifyinga feature vector uniquely, which receives the IP address (source or destination). Thischoice was inspired by [34] that used source and destination IP addresses todistinguish groups of 1-to-N and N-to-1 anomalies. The NumSIPs/NumDIPsfeatures represent the number of all different IP addresses contacted contained inthe aggregated flow, for that key, whereas the LocationCode feature contains thecountry code of the address IP that fills the AggregationKey feature.

Features from layer 3 are related to source and destination ports, the SYN flagfrom TCP, and ICMP protocol. Features NumSports and NumDports representthe first two characteristics, containing the number of all different source portsand destination ports contained in the aggregated flow, for that key. If the SYNflag is observed, the SYNRate feature will be filled by the number of times thata SYN flag is sent divided by the total number of packets in the aggregated flowfor that key. For the ICMPRate feature the same procedure is applied, that is, thenumber of times the ICMP protocol is used divided by the total number of bytesof the aggregated flow for that key.

Features from layer 4 are related to the application protocols that are used inthe aggregated flow for that key. Features NumHTTP, NumIRC, NumSMTP,and SSH represent the number of occurrences of contacting ports 80, 194 or6667, 25, and 22, respectively, for the HTTP, IRC, SMTP, and SSH protocols.

7.4.1.2 Statistic Features

Four statistic features (lines 13 to 16 of table) were defined to summarize anaggregated flow for a given key, regarding its number of packets and size. Total-NumPkts and TotalNumBytes features are defined to represent these two character-istics that are obtained by summing all the values of them, that is, the number ofpackets and the number of bytes, respectively. From these two features we obtainanother two: PktRate and AvgPktSize. The PktRate feature gives the aggregated flowpacket rate, which is obtained by dividing TotalNumPkts by the total duration of theaggregated flow for a given Key, and the AvgPktSize feature gives the aggregated flowpacket average, which is obtained by dividing TotalNumBytes by TotalNumPkts.

7.4.1.3 Threat Intelligence Features

The last four lines of Table 7.1 present the features related to evidence of threatsin the aggregated flow. The BadSubnet and MaliciousIP features indicate if theaggregation key (AggregationKey) belongs to a subnet or malicious IP blacklist,whereas the OpenVaultBlacklistedIP and MaliciousASN features have the same


mean than MaliciousIP but the threat intelligence source is different. All of thesefeatures are binary, meaning that their content is 0 or 1, i.e., the key does notbelong or belongs to such lists.

7.4.2 Flow Feature Extraction

The first 15 features are extracted from the aggregated flows in the feature extractionphase. This phase is composed by three tasks, namely, filtering, mapping & reducing,and normalization (see Figure 7.1), which are described as following.

7.4.2.1 Filtering

Upon the receiving of the gathered flows, a filtering is performed to get thecharacteristics related with 9 features referenced above (lines 3, and 5 to 12 of Table7.1). Since the data of these features are contained in fields of the header protocolsfrom layers 2, 3, and 4, the easiest way of filtering them consists in removing someunnecessary characteristics from the flows (e.g., its payload content and date). Eachpacket is unencapsulated for accessing to the protocol header fields, extracting datafrom these fields according to the defined features, creating a tuple with these data,and storing the tuple temporarily to later be processed. Specifically, nine character-istics are extracted to form a tuple, namely source and destination IP addresses, sourceand destination ports, number of sent packets, which protocol was used, TCP flag (ifany), number of exchanged bytes and its duration. A representation of a tuple is<srcIP, dstIP, srcPort, dstPort, #pkts, protocol, flag, #bytes, duration>.

7.4.2.2 MapReduce

This task processes the tuples, by first aggregating them in order to form aggregatedflows, secondly for each aggregated flow to be represented by a vector of 15 featurespresented in Table 7.1 (lines 2 to 16). Aggregating flows means to merge into a singlerepresentation all the tuples that have the same source or destination IP address(destination and source addresses for destination and source aggregations, respec-tively). Notice that the flows collected from a host are those that outgoing, and sorepresented by the source IP address, and those that incoming, which are representedby the destination IP address. To get the vector of features it is necessary mapping alldata inside a aggregated flow, and then reducing these data, merging it into one.

To achieve such actions, the MapReduce paradigm is used. MapReduce was firstintroduced by Google [13]. It allows processing big data in parallel in large serverclusters. For that purpose, it divides processing jobs in two phases that run,respectively, a mapper and a reducer function. The mapper phase involves applyingthe mapper function to each of the input files and obtain a set of pairs5key;value4;

248 ■ Botnets

the reducer phase runs one or more reducers that receive as input the pairs generatedby the mappers, and aggregate key pairs with the same keys, performing someoperation on their respective values.

7.4.2.3 Mapper

Following the algorithm, the mapper for every tuple parses it, gets the ninecharacteristics extracted in the filtering task and creates a 5key;value4 pair foreach one, being the key a string in the format “S/D,feature,IPaddress” and the valuethe characteristic value for the feature in question. The key is composed of threeelements that combined allow uniquely identify the origin of the feature, which isassociated to a source or a destination IP. The S/D element denotes if the tuplerepresents an outgoing (S) or an incoming (D) flow, being S and D representatives ofsource and destination IP address, respectively. The feature element is the featureparsed from the tuple, and the IPaddress element is the IP address of the sender host.

We defined 22 key pairs, i.e., 11 key pairs for each aggregation key. Figure 7.2 showsthe key pairs for the source IP aggregation key. The last four defined key pairs areregarding the counters for the number of times the protocol port was used, allowingthus calculate the layer 4 features (lines 9 to 12 of Table 7.1). The key pairs todestination IP as aggregation key are analogous, replacing in the key the S by D, andsrcIP by dstIP.

For example, the outgoing tuple <192.168.0.105, 10.10.5.2, 80, 80, 52,HTTP, 1050, 31> sent by the 192.168.0.105 source IP when received by theMapper, the following key pairs showed in Figure 7.3 are produced.

7.4.2.4 Reducer

The reducer receives the key pairs of every tuple, aggregates them by key, andcalculates counts and sums using them in order to obtain the final values needed tofill the vectors of features. When the reducer receives a pair with an IP address key

<"S,dstIP,srcIP";dstIP> <"S,srcPort,srcIP";srcPort> <"S,dstPort,srcIP";dstPort> <"S,pkts,srcIP";#packets> <"S,protocol,srcIP";protocol+flag> <"S,bytes,srcIP";#bytes> <"S,duration,srcIP";duration> <"S,HTTPPort,srcIP";yes/no> <"S,IRCPort,srcIP";yes/no> <"S,SMTPPort,srcIP";yes/no> <"S,SSHPort,srcIP";yes/no>

Figure 7.2 Key pairs defined for the source IP as aggregation key.


that it has not yet seen, it creates a new feature vector which the Aggregation Keyfeature filled with that IP address. On the other hand, when it receives entries withkey IP addresses that have already known it aggregates their values in the respectivevectors. For example, in order to summarize the number of bytes sent by the192.168.0.105 source IP, i.e., calculating the TotalNumBytes feature, the mapperproduces the 5“S,bytes,192.168.0.105”;value4 pair for each tuple, being value thenumber of bytes sent in the tuple. The reducer, on the other hand, receives theserecords, and sum the value of all the records that have the same 5“S,bytes,192.168.0.105”4 key, aggregating in this way such values.

After all flows are aggregated in a single one and represented by a vector, thereducer produces the four statistic features (lines 13 to 16 in Table 7.1) derivedfrom the other nine features. Finally, the LocationCode feature is fetched byusing the Aggregation Key value and an IP tracker that references geographic IPs.

At the end of the MapReduce algorithm execution, we have two sets of featurevectors: one representing the outgoing aggregated flows (identified by S) andanother representing the incoming aggregated flows (identified by D).

7.4.2.5 Normalization

Most features carry numeric data, but there is the need to keep every value in onecommon scale. Moreover, there are some features that are not expressed ina numerical manner, such as the IP addresses and the country. In these cases,these features are mapped to numerical values, which can be reversed to text. Inaddition, there are other features (e.g., NumDport) that are numeric but theirvalues must be normalized. On the other hand, there are features that do notneed to be normalized because they already are, such as threat intelligence featuresin which their values are binary, i.e., 0 or 1.

Normalizing a set of values means mapping these values to a specific range. Wewant normalize the feature values to the interval [0,1], where 0 is absolute

<"S,dstIP,192.168.0.105";10.10.5.2> <"S,srcPort,192.168.0.105";80> <"S,dstPort,192.168.0.105";80> <"S,pkts,192.168.0.105";52> <"S,protocol,192.168.0.105";HTTP> <"S,bytes,192.168.0.105";1050> <"S,duration,192.168.0.105";31> <"S,HTTPPort,192.168.0.105";yes> <"S,IRCPort,192.168.0.105";no> <"S,SMTPPort,192.168.0.105";no> <"S,SSHPort,192.168.0.105";no>

Figure 7.3 Key pairs of an outgoing tuple for the 192.168.0.105 source IP.

250 ■ Botnets

minimum, and 1 the absolute maximum. To achieve so, given the feature valuesfrom a feature vector in the form X ¼ ðx1;…; xnÞ, the correspondent normalizedY ¼ ðy1;…; ynÞ vector is obtained using:

yi ¼ xi � minðX ÞmaxðX Þ � minðX Þ ; yi 2 ½0; 1� ð7:1Þ

7.4.3 Obtaining Threat IntelligenceIn order to complement the information of aggregated flows, the feature vectorsare completed with threat intelligence about blacklisted subnets and maliciousIPs. This way, online threat intelligence repositories are accessed to retrieve theselists. After flows were processed and feature vector created and filled by MapRe-duce algorithm, the four threat intelligence features (last four lines of Table 7.1)are added to the vectors. Next, for each vector is checked if the Aggregation Keyvalue belongs to those lists, and in such case the correspondent features are set to1, otherwise they are set to 0.

7.5 Detecting Network AttacksWe propose an approach to detect unknown network attacks based on the assump-tion that the majority of the observed traffic is benign rather than malicious, as well asthat malicious traffic is qualitatively different to the regular, normal traffic. Toachieve so, the proposed approach uses unsupervised ML algorithms in two fashions:to separate both kinds of traffic, generating clusters, and to confirm if the smallerclusters generated are actually malicious, classifying them.

We propose a detection process using unsupervised techniques over sets ofvectors of features (see Section 7.4). An unsupervised clustering algorithm isapplied on the feature vectors, aggregating groups of vectors having similar valuesin their features, forming in this way various large groups of hosts, and someoutliers (small groups of hosts). Afterward, the resulting outliers are classified asbeing malicious or not by using a unsupervised algorithm, that we call a classifier.According to the mentioned assumption, the outliers may represent an attack,although this may not always be the case. Such outliers could also represent, forinstance, some application that are less frequently used by a host, or evena machine whose characteristics are not very common, therefore producing flowfeatures that are different from regular traffic that is found in bigger clusters. So,it is of utmost importance to analyze and classify them, in order to differentiatethe actual attacks from these benign outlier traffic patterns. This detection processregards to the similarity aggregation and detection phases of the approach presentedin Section 7.3, illustrated in Figure 7.1, and detailed in next sections.


7.5.1 Similarity Aggregation

The idea behind the unsupervised algorithms we consider—clustering—is togroup different instances of a dataset into k distinct groups, i.e., clusters,according to the similarity of their values or characteristics. For instance, applyinga clustering algorithm to a dataset of network traffic, it would generate k clusters,where one would be representative of regular DNS traffic, another one would besimple SMTP traffic, and so on. Therefore, we want to apply the same idea ofclustering to separate and represent normal and abnormal traffic.

Depending on the algorithm used, the value of k may or may not be chosenautomatically. For example, the DBSCAN algorithm [39] does not need a predefinedk value, but the K-Means and the Mini Batch K-Means algorithms [40] need it. Weopted by algorithms in which k must be specified, since we want to find out which isthe best k for dealing with diversity of data. We chosen K-Means and Mini BatchK-Means algorithms because the former is the most used algorithm for that task dueto its simplicity and efficiency, and the latter we want to investigate if it behaves aswell as the former, and can give us other insights not given by K-Means.

7.5.1.1 Choosing the Number of Clusters

Both K-Means and Mini Batch K-Means algorithms require the number k ofclusters to be specified. There is no obvious value for k. However, there are sometechniques that give us a hint of what the value of k should be. Such is the case ofthe Elbow Method. These clustering algorithms converge when the variation ofthe distance between the data points and the clusters centers start converging to0. With this in mind, the Elbow Method starts by computing the error functionthat is used as a stopping criterion in the algorithm, known as total within-clustersum of square (WSS), which is mathematically defined as follows:

WSS ¼Xk

i¼1

X

x2cidistðx; ciÞ2 ð7:2Þ

Equation (2) produces values for k in a specified range, which it is provided bythe user. For example, for a specified k ¼ 30 and k-means clustering algorithm,the method calculates the WSS for different values of k, by varying k from 1 to30 clusters. By plotting these values according to the number of clusters k, weobtain a curve that will be decreasing with the increase of k, as we can observe inthe plot graph on the top of Figure 7.4. Theoretically, the optimal value of theWSS would be 0, but this value is only obtained when the number of k clusters isequal to the number of entries in the dataset, which would mean that each datapoint would be in its own clusters, and this process would not provide interesting

252 ■ Botnets

information at all. Instead, the Elbow Method indicates that appropriate numberof clusters is when the slope of the WSS has a sudden break. Apart from the WSS,the Elbow Method also calculates the percentage of variance explained (PoVE)metric for each value of k (bottom graph of Figure 7.4) which reflects the ratio ofthe between-group variance (BSS) and total variance (TSS), and indicates anoptimal k when it suffers an abrupt change.

According to the method and analyzing the Figure 7.4 as example, the optimal kwould be 2 because the biggest slope is found for k ¼ 2 (x-axis), which is can not fit forour case. However, there is a rule of thumb often used, which consists in starting offwith k ¼ ffiffin

2

p, where n is the number of entries in the data set. Applying this rule we

found that k ¼ 10 is a number of clusters that successfully and coherently divides thedifferent datasets of various sizes. Observing the Figure 7.4, we can see that aroundk ¼ 10 the KSS and PoVE values start to stabilize, and its variation is close to 0.

7.5.1.2 Describing Clusters

The potential malicious aggregated flows, i.e., the aggregated flows that cancorrespond to intrusions, are assumed to be placed in the clusters with smallersize. In order to obtain a coarse grained overview of each cluster’s content and toclassify each cluster easily, each feature of each cluster is described by its meanvalue and standard deviation. In this way, it is possible to have an idea of eachcluster’s behavior and each cluster’s feature distribution. Also, this allows describ-ing a cluster’s content representing it by a single feature vector composed by themean values of features and the standard deviation of them.

Figure 7.4 Plot of the elbow method.


The resulting vector, we call it descriptor vector, and for the classification taskwe focus on those descriptor vectors that have higher feature values, i.e., highermean values and standard deviations.

7.5.2 Detection

In order to automate the detection of malicious hosts, a classification method basedon unsupervised ML is applied to the descriptor vectors resulting from the similarityaggregation phase. This method applies again an unsupervised algorithm to thedescriptor vectors, after they are joined to a temporary dataset comprising descriptorvectors provided from previous loop iterations and classified as being malicious, andthen verifies if some outlier results.

The idea behind this classification method relies in the fact that malicioushosts have higher feature values in more than one of their features, and ifa clustering technique is applied to a set of instances with these characteristics,there will be a resulting cluster that contains such instances. Therefore, givena set of malicious descriptor vectors and a descriptor vector that we want toclassify, if we add this vector to the set and then apply an unsupervisedalgorithm for k ¼ 2, we envisage two possible results: (1) all instances will beput in a single cluster (i.e., resulting an empty cluster), meaning that thedescriptor vector is classified as malicious; (2) both clusters will be populated,which one of them is considered an outlier and contains the descriptor vectorwe want to classify, meaning that such vector is an unusual normal traffic case,and then classified as normal. Figure 7.5 shows the data flow of this classifica-tion method and the process explained above.

Smallerclusters

Describingcluster

ClusteringK=2

outlier?

Dataset withdescriptor

vectors

Malicioushosts

detected

Unusualnormal traffic

Automaticlabeling

Y N

Figure 7.5 Detection phase data flow.

254 ■ Botnets

In this process of classification, we can say that the system runs a unsupervisedlearning algorithm that is trained with the malicious labeled data produced by theprevious classifications, and will proceed to classify the traffic that was perceivedas being outlier by the first clustering algorithm application (on similarityaggregation phase). Upon this classification, the system should be able to correctlyidentify the observed malicious traffic, thus allowing the detection of themalicious hosts. This process is to be performed on a daily basis. If the analysisperiod was smaller, some attacks would not be possible to detect—as some ofthem last for long periods of time; if it was longer than a day, the obtained valueswould become noisy, as the flows are aggregated, some IP addresses may bereused from one day to another, therefore achieving very high feature values,misleading us to think that it is indeed an attack.

7.5.3 Learning

In a first run of the system, the classification module has not yet any knowledgeat all, and so there is a need for a manual intervention that will classify and labelthe descriptor vectors of the outliers provided from the similarity aggregationphase. Therefore, upon the clustering of the data resulting from the similarityaggregation phase, a manual intervention is performed in order to analyze thecharacteristics of the traffic—summarized in the form of clusters—ultimatelyleading to the production of a labeled dataset that will serve as training for theclassification method of the detection phase.

This initial manual classification serves as input to the unsupervised algorithmdeployed in the detection phase. In the following runs of the system, the data setwill be increasing with the previous classifications made. So, over time, there maynot be a need for manual intervention since the classification method will comemore and more capable of classifying on its own, which is based on the previousclassifications made and learned by the method (see Section 7.5.2). However, werecall that only descriptor vectors classified as malicious are used to compose thedata set used on clustering task of the detection phase. Therefore, the vectorsclassified as normal, manually and automatically, are discarded.

7.6 FlowHacker ImplementationTo evaluate our approach we implemented it in the FlowHacker NIDS, whichwe developed in Python. The system is composed by two modules—similarityand classification—for aggregating the feature vectors in clusters and detectingmalicious hosts, respectively. In addition, FlowHacker interacts with the Hadoopframework for running MapReduce over the flows, obtaining aggregated flowsand vectors of features.


Before using FlowHacker, the first step is to gather the flow collection,obtained using NetFlow-enabled routers placed at the border routers betweenthe core network and the connection to the ISP. For the sake of analyzing andtreating these flows, all of this data is converted to the SiLK format. Next,a filtering to the flows is performed for getting the nine features needed tocompose the feature vectors (see as Section 7.4.2). Afterward, these filteredfeatures are processed by the Hadoop framework. Hadoop is an open-sourceframework that features both distributed storage and parallel processing of BigData, making it very scalable to very large amounts of data. To support theparallel data processing, Hadoop implements Google’s MapReduce algorithm[13]. This model operates on a virtual environment called HDFS, which hasboth Mapper and Reducer nodes. This model can be divided into two main steps,Mapping and Reducing, which realize the operations described in Section 7.4.2.

FlowHacker starts with the similarity module that allows the users managingthe clusters, such as change or calculate the number of clusters (k), generate theclusters, and visualize their contents, and then the classification module gets thesmaller clusters resulting of the first module, verifies if they are malicious or not,classifying them, and updates the data set with the malicious ones for furtherclassifications. Moreover, the tool has a terminal interface that was developed tofacilitate the similarity aggregation tasks and visualize the outcomes.

7.7 Experimental EvaluationIn order to validate our approach, we evaluated FlowHacker experimentally withtwo data sets: the ISCX synthetic data set1 (Section 7.7.1), and real data providedby the large Portuguese ISP (Section 7.7.2).

The objective of the experimental evaluation was to obtain answers for thefollowing questions:

1. Is FlowHacker able to detect attacks against synthetic data and real data?2. Is FlowHacker able to identify the type of attacks performed?3. How does it perform in terms of false positives and false negatives?4. Is FlowHacker able to detect botnet activity?

7.7.1 Evaluation with Synthetic DataThe ISCX data set consists in flows corresponding to one week, and aims toprovide a complete test data set for IDSs [15]. The data set contains attacks of

1 www.unb.ca/research/iscx/dataset/iscx-IDS-dataset.html

256 ■ Botnets

four classes (Table 7.2): probing, to gather information about the network; denialof service (DoS), to compromise system availability; user to root (U2R), in whichthe attacker leverages access to a normal account to gain root privileges; andremove to login (R2L), in which the attacker exploits some vulnerability to gainaccess. Moreover, there is a train data set and a test data set, whereas the second isa superset of the first, as shown in the Table 7.2.

All of the flows in the ISCX data set are labeled, therefore allowing fora validation of the accuracy of FlowHacker. Upon the cluster generation andrespective manual labeling of this data, the results were compared to the groundtruth provided by the original data set. After the data that was labeled, weproceeded to train our classifier, which is the classifier for further flows to beanalyzed.

7.7.1.1 Cluster Analysis

The data set is divided in six subsets, each one representing a weekday, fromSaturday to Thursday (no Friday). Attacks were detected in all of these days,except for Wednesday, that was found to be attack-free. For the data of each day,we did filtering, extraction, and normalization. Next, it was processed byHadoop, and then we used FlowHacker configured for 10 clusters. Next wepresent the clustering results.

Saturday: By analyzing the contents of the clusters corresponding to this day,we found one cluster that presented features that are rather alarming. In this one,the number of different source ports used and number of connections throughthe SSH port are highlighted, being the number of connections through the SSHport at its absolute maximum value. A study on Brute-Force SSH attacks [24] hasshown that these two features together are representative of a Brute-Force SSHattack. Given that the rest of the traffic presents feature values that are rather

Table 7.2 Attacks in the UNB ISCX train and test data sets (all attacks fromthe first exist also in the second)

Class Train data set attacks Test data set only attacks

Probing portsweep, ipsweep, satan, guesspasswd,spy, nmap

snmpguess, saint, mscan,xsnoop

DoS back, smurf, neptune, land, pod, teardrop,buffer overflow, warezclient, warezmaster

apache2, worm, udpstorm,xterm

R2L imap, phf, multihop snmpget, httptunnel, xlock,sendmail, ps

U2R loadmodule, ftp write, rootkit sqlattack, mailbomb, proces-stable, perl


normal (i.e., none of them is indicting the presence of an intrusion), weconsidered that the flow present in this cluster was performing such an attack,therefore highlighting it as an intrusion flow, as the remainder of the traffic wasconsidered to be normal.

Sunday: The results for Sunday have shown a very different pattern. Unlike theprevious day, almost every cluster presents very high feature values. Features such as theaverage packet size, the number of source ports and the number of HTTP connectionare high in the great majority of the clusters. Also, the number of SMTP connectionswas also found to be very high in two different clusters. This behavior shows us thatsomething is not right, as the SMTP connections are usually grouped together ina single cluster, and this analysis shows us that two different clusters have thesecharacteristics. Taking this into account we assume that these flows, although havingthis feature with very high values, were grouped into different clusters because they havea different behavior, and therefore showing us that these flows are not normal. As forthe remaining clusters, we found four that have very high values for the number ofHTTP connections, alongside with the number of different source ports and averagepacket sizes. These three features together may indicate that a large volume attack isbeing perpetrated, exploring the HTTP protocol, therefore also labeling these clusters asattacks.

Monday: On Monday, we found two clusters that immediately distinguishthemselves from the rest. The first one has a mean value of 0.998 for the ICMPRate, being it the cluster with the biggest dimension (it contains 375 different flows);the second one has the number of destination ports and number of SMTP and IRCconnections at its highest value possible. However, this is not considered an alarmingbehavior, because even though the value for ICMP Rate in indeed at a very highvalue, no other feature in that cluster was showing a high value; as for the secondcluster, throughout the whole evaluation of the system, there was always one clusterwith such characteristics, and we can infer that this cluster corresponds only toregular clients using email services. Apart from these two, other four clusters alsopresent an alarming pattern. All these clusters share high values for the number ofdifferent source ports, number of HTTP connection and also for the average packetsizes. Such pattern may be attributed to a DoS attack, as each host is send a greatamount of packets from many different ports, all direct to the port 80 (or port 8080,in some cases), with an high average packet size. This is the case of the DoS HTTPFlood attack. However, this is an attack that is easily identifiable by inspecting itspayload, and this flow-based approach does no allow us to perform such an analysis,being these features our only way to hint the presence of such an attack.

Tuesday: For Tuesday, we observed that there were multiple clusters witha very high value for the ICMP Rate. However, this feature appear alone, i.e., itis the only feature in these clusters that has a relevant high value, no otherfeatures show up, apart from one cluster that also has a high value for the averagepacket size, which also does not correspond to a recognizable pattern. From all

258 ■ Botnets

these clusters, the one that grabs our attention is the tenth, which features a highvalue for the number of source ports, HTTP connections and average packet size.Also, it has a high packet rate, average packet size and total number of bytes.From what we have seen so far, this can only correspond to an attack, andtherefore the content of this cluster was labeled as being an attack.

Wednesday: As mentioned, no attacks were identified for Wednesday.Thursday: For Thursday, just like for Saturday, there is one cluster that was

found to have an absolute maximum value for the number of SSH connectionalongside with a high value of number of different source ports, thus indicatingus the presence of a Brute-Force SSH attack. Also, three other clusters have highvalues for the number of different source ports, number of HTTP connectionsand also a high average packet size, also possibly indicating the presence of anattack. Therefore, these two clusters were also labeled as malicious.

7.7.1.2 Unsupervised Classification

In parallel with the daily analysis, FlowHacker may also autonomously identifymalicious activities using the classification method implemented, which classifiesthe smaller clusters resulting from the similarity module based on an unsupervisedalgorithm and a training data set. Before the FlowHacker is able to classify data itis needed, at least, malicious labeled data from the first day, which results fromthe manual intervention described in Section 7.5.3. From this day on, the classifieris able to produce results on its own, and these results may be refined with everyiteration of the system (for the purposes of this work, an iteration corresponds tothe period of one day), by training the system again, as new patterns are identifiedand manually labeled.

The classifier was trained for the first day with data from the analysis forSaturday, as seen in Section 7.7.1. As expected when the system processed theclusters for that day, it correctly identified the malicious flow. However, whentrying to classify the results for Sunday, the classifier did not found any sort ofmalicious activity. This was due to the fact that the system’s only knowledge aboutthe intrusions observed during Saturday, which does not give sufficient informationto the system to detect other attacks. After training the system once again with theanalysis done for Sunday, the classifier was now able to identify the maliciousactivities, although it could not identify them all. This same behavior was foundwhen classifying the remainder of data set throughout the rest of the days.

Table 7.3 shows the results with more detail. Along the system iterations,FlowHacker detected 7 attacks, which correspond to flows whose were perpe-trating the attack with greater intensity, i.e., producing large volumes of traffic,whereas the remainder of the attacks was not successfully identified. Althoughwe observed 24 false negatives, it is visible that they were decreased during theiterations, which is justified by increasing of knowledge that the system was


learned. Regarding the Wednesday, the system misclassified five flows (i.e., falsepositives) as being malicious, when the traffic relative to Wednesday is allnormal, intrusion-free traffic. This is due to the fact that Wednesday was oneof the days that had the largest amount of traffic, and therefore the flows thatbelong to it also produced higher features values, leading to it being perceivedas malicious. At the end the system misclassified 11 flows as being malicious.This means that it needs some refine in the classification method. However, weprefer to have a system giving false positives than unreported attacks, i.e., withfalse negatives.

7.7.1.3 Result Validation

The ISCX data set is a data set with known attacks put in a database containinginformation about them. The data set is used as the ground truth validation, sowe compared the FlowHacker results with this ground truth to validate them andfind out the accuracy and precision of our system. Each flow belonging to ISCXis identified by a unique ID, meaning that the flows contained in our clustershave this ID. Therefore, through this ID we were able to trace back it to its IPaddress (which is stored in the database), and this way we were able to identifythe malicious hosts.

We were able to verify that all clusters containing malicious flows identifiedby our system actually such flows are malicious, meaning thus that our systemis able to detect and classify correctly attacks under traffic analysis. Also, weobserve that correctly we identify the type of attack evolved in such flows.However, we also verified that our system generates some false positives andmisses some malicious flows, i.e., has some false negatives, such as is evidencedby results in Table 7.3.

These results suggest a positive answer to questions 1 to 3.

7.7.2 Evaluation with Real Data

The following results were obtained from data provided by the above-mentionedlarge Portuguese ISP. Our NIDS approach assumes that data is collected using

Table 7.3 FlowHacker results with the synthetic data set, showing howresults improve with the number of iterations

Saturday Sunday Monday Tuesday Wednesday Thursday

True Positive 1 3 2 0 0 1

False Positive 0 1 0 1 5 4

False Negative 0 17 4 3 0 0

260 ■ Botnets

NetFlow-enabled routers, e.g., placed at the border routers between the corenetwork of the Portuguese ISP and the connection to its own international ISP.All the data that reaches the Portuguese ISP network comes from that interna-tional ISP that is connected to the Portuguese company via routers that areplaced at the borders of the core network. These routers are NetFlow-enabled,and are protected by firewalls, in order to filter any wanted data, according totheir security policies, therefore ensuring that only supposedly clean data reachesits clients. However, not all bad traffic is filtered, which is why there are neededextra security measures, such as NIDSs. So, the data that reaches these routersfrom the outside (i.e., the data incoming from the ISP) is collected by NetFlow,for further analysis. For the purpose of this evaluation, a data sample wascollected for a few hours. The data set was obtained without a priori knowledgeabout the existence of attacks, so we had no data to validate our results, unlikewhat happened with the synthetic data set (Section 7.7.1).

After performing the filtering and MapReduce phases, we obtained twosubdata sets comprising the source flows and the destination flows, respectively.Therefore, the following analysis reports to each of these subdata sets. Theclustering was performed with the number of clusters set to 30 (i.e., k ¼ 30).

7.7.2.1 Source Aggregation Key Clustering

In the content of the source aggregation key clustering, we observe five clustersthat present high feature values, as shown in Table 7.4 (clusters 13, 15, 17, 21,and 30).

The first presents a high number of different source ports, as well as a highnumber of total bytes sent. However, such pattern was not found to besuspicious, as the number of source ports itself does not represent an alarmingnetwork threat, as opposed to the number of destination ports, and no addressfound in this cluster was found to be in any IP blacklist.

When analyzing the second one, we see that is presents a high connectivity tovarious users, under various ports, receiving communication on an IRC port, andcommunicating through HTTP, with a high number of packets sent, as well asa high number of bytes. This leads us to assume that this machine is eithera major spammer, or it could be a DoS attack, given its traffic pattern, and it wasthus labeled as being a malicious host.

Moving on the third cluster, it was found to have a high number of SSHcommunications alone, which could represent a Brute-Force SSH attack, in justlike had observed in the previous section, thus also being labeled as malicious hosts.

The fourth presented a high number of IRC (which is used as a portal forbotnet’s C&C communications) communications, alongside with a high averagepacket size. This feature distribution led us to consider that this could a botnetcommunicating, and thus labeling it as malicious hosts. Moreover, we decided to


Table7.4

Source

keyclustering

conten

tusing

K-M

eans

Clu

ster

s1

23

45

67

89

1011

1213

1415

1617

1819

2021

2223

2425

2627

2829

30

# Fl

ow

s

1605

51,773

6485

13,305

529

1730

1729

21,507

8523

8522

1498

4686

10

5653

1

824

5

4606

1864

1676

12

107

13

2233

2264

8091

10

23,897

16,843

35

Feat

ure

s

1-

--

--

--

--

--

--

-1,

00

1,00

1,00

1,00

--

--

--

--

--

--

--

-

2-

--

--

--

--

--

--

--

--

--

--

--

--

--

--

-

3-

--

--

--

--

--

-0.

37-

--

--

--

--

--

--

--

-

4-

--

--

--

--

--

--

--

--

--

--

--

--

--

--

-

5-

--

--

--

--

--

--

-0.

67-

--

--

--

0.38

;

0.18

--

--

--

-

6-

--

--

--

--

--

--

--

--

--

--

--

--

--

--

0.24

;

0.16

7-

--

--

--

--

--

--

--

--

--

--

--

--

--

--

-

8-

--

--

--

--

--

--

--

--

--

--

--

--

--

--

9-

--

--

--

--

--

--

--

--

--

-0.

54;

0.18

--

--

--

--

-

10-

--

--

--

--

--

--

--

-0.

63;

0.21

--

--

--

--

--

--

-

11-

--

--

--

--

--

--

--

--

--

--

--

--

--

--

12-

--

--

--

--

--

--

--

--

--

--

--

--

--

--

-

13-

--

--

--

--

--

--

--

--

--

--

--

--

--

--

-

14-

--

--

--

--

--

--

--

--

--

--

--

--

--

--

15-

--

--

--

--

--

-0.

61;

0.21

-0.

84-

--

--

--

--

--

--

--

16-

--

--

--

--

--

--

--

--

--

-0.

26-

--

--

--

--

17-

--

--

--

--

--

--

--

--

--

--

--

--

--

--

-

262 ■ Botnets

track the IP addresses, and check them against a number of public availableblacklists, which confirmed and categorized the IP addresses as sources ofBotnets/Spam. One of the IP addresses led us directly to an authentication pageof a C&C.

When analyzing the last alarming cluster, we observed that it presenteda high number of SMTP communications, but when analyzing its IP addresses,we found that these were only mail server’s communication, and we found noharm in it. Prior to this analysis, all of the IP addresses present in themalicious clusters, were found to be present in several blacklist, thus confirm-ing our suspicion.

7.7.2.2 Destination Aggregation Key Clustering

In relation to the destination aggregation key clustering, we observe that there arefive clusters (16, 20, 22, 25, 29) with alarming features (Table 7.5).

Analyzing cluster 16, we see that it has a feature distribution that is similar towhat we had understood as a DoS HTTP Flood attack when analyzing the ISCXdata, except that this cluster is missing a high value for the number of HTTPconnections. Therefore, this could also represent a DoS attack, but directed toother applications, e.g., DNS. We cannot be sure of this attack, because none ofthe monitored ports are presenting high values, and so we cannot infer anythingmore about it.

Cluster 20 presents a high number of different source IP addresses, destinationports and number of bytes. Because these flows do not have a high average packetsize, it could possibly indicate that this a network scan, as these flow contactedmany different port of many different IP addresses, resulting in a high value ofbytes sent throughout this process.

Cluster 22, on the other hand, presents a feature distribution that is similar towhat had previously perceived as a DoS attack: it has a high number of differentsource IP addresses, number of source ports, number of HTTP connections, anda number of bytes sent. However, it still lacks a high value for the average packetsize. Therefore, this may be, just like cluster 20, a network scan, but this timedirected to the HTTP application, i.e., it may be a probing of a website in orderto locate some vulnerability, for example.

Cluster 25 presents a high number of different IP addresses, average packetsizes and number of bytes sent. These features alone do not seem to correspondto a malicious behavior, as we interpreted them a simple burst of traffic.

At last, cluster 29 hold a have number of source IP addresses, number ofdestination ports, number of source HTTP connections, average packet sizes andnumber of bytes sent. This pattern very similar to what we have seen for the DDoSIRC botnet attacks, expect for the number of IRC connections. Therefore, this mayalso correspond to infected hosts that are being used as a third party for attacks, but


Table7.5

Destina

tion

keyclustering

conten

tusing

K-M

eans

Clu

ster

s1

23

45

67

89

1011

1213

1415

1617

1819

2021

2223

2425

2627

2829

30

# Fl

ow

s

34,565

1447

7734

888

312

17,180

53,767

497

7

2442

2177

3089

232

62,515

644

12

1242

1699

4987

84

93

23

10,883

754

36

22,543

16

503

3

8680

Feat

ure

s

1-

--

--

--

-0.

84;

0.11

--

--

--

0.41

;

0.07

--

-0.

16;

0.07

-0.

21;

0.05

--

0.15

;

0.09

--

-0.

16;

0.09

-

2-

--

--

--

--

--

--

--

--

--

0.22

;

0.01

--

--

--

--

0.24

;

0.09

-

3-

--

--

--

-0.

86;

0.16

--

--

--

0.59

;

0.18

--

--

-0.

52;

0.14

--

--

--

--

4-

--

--

--

--

--

--

--

--

--

--

--

--

--

-0.

84;

0.15

-

5-

--

--

--

--

--

--

--

--

--

--

--

--

-0.

40;

0.18

--

-

6-

--

--

--

--

--

--

--

--

--

--

--

--

--

--

-

7-

--

--

--

--

--

--

--

--

--

--

--

--

--

--

-

8-

--

--

--

-0.

26;

0.16

--

--

--

--

--

--

0.12

;

0.11

--

--

--

--

9-

--

--

--

--

--

--

--

--

--

--

--

--

--

--

-

10-

--

--

--

--

--

--

--

--

--

--

--

--

--

--

-

11-

--

--

--

-0.

85;

0.17

--

--

--

0.45

;

0.11

--

--

--

--

0.12

;

0.05

--

-0.

10;

0.01

-

12-

--

--

--

--

--

--

--

--

--

-0.

23;

0.01

--

--

--

--

-

13-

--

--

--

--

--

--

--

--

--

--

--

--

--

--

-

14-

--

--

--

--

--

--

--

--

--

--

--

--

--

--

-

15-

--

--

--

-0.

26;

0.07

--

--

--

--

--

0.15

-0.

21;

0.07

--

0.45

;

0.18

--

-0.

40;

0.05

-

16-

--

--

--

--

--

--

--

--

--

--

--

--

--

--

-

17-

--

--

--

--

--

--

--

--

--

--

--

--

--

--

-

264 ■ Botnets

contacting its bot master through a C&C server other than an IRC, or they could bevictims of an attacker who is using spoofed IP addresses to use them as a third party.

7.7.2.3 Discussion

Table 7.6 summarizes the information regarding the intrusions detectedthroughout this analysis, emphasizing those related to botnet activity. For thefeature numbers, refer to the last column of Table 7.1. The table shows thatFlowHacker was able to detect and identify attacks in real traffic of a company,both in incoming and outgoing traffic. Such results allow us to verify that realthreats are effective and a concern for companies. Also, they allow us to verifythat our system is effective in detection of such threats, including botnets, andcan be used by companies for avoiding them. This allow us to answer positivelyto questions 1 to 4.

7.8 ConclusionThis chapter presents an approach and a system to analyze traffic from fastnetworks, such as the fast connection links of ISPs, in which traditional IDSs arelimited due their incapacity to analyze big amounts of traffic that circulates inthese links and to analyze encrypted data.

The system is based on analysis of network flows, which makes it capable ofanalyzing such connection links. The approach behind of the system allowsdetecting malicious hosts without requiring previous knowledge about what wewere looking for or clean training data. A combination of data mining techniquesfor the feature extraction from netflows, and ML techniques for data analysisallows the detection of malicious behaviors without requiring specific training,except for the inevitable human intervention in a first run of the system.

Table 7.6 Real data analysis results with botnet cases emphasized

Cluster # Aggregation Key Highlighted Features Type of Attack

15 Source 1, 3, 5, 8, 11, 15 Spam/DoS

16 Destination 1, 3, 6 DoS

17 Source 10 Brute-force SSH

20 Destination 1, 2, 15 Network scan

21 Source 9, 16 Botnet communication

22 Destination 1, 3, 8, 15 Web application probing

27 Source 1, 2, 5, 8, 11, 15 DDoS IRC botnet

29 Destination 1, 2, 4, 11, 15 DDoS botnet


The FlowHacker NIDS implements the approach and it was evaluated withboth synthetic and real data, being the real data provided by a large PortugueseISP. The results of both analyses suggest that the system can be used in detectionof threats, such as DDoS command by botnet’s C&C communications detectedin the real data analysis.

AcknowledgmentsThis work was partially supported by the EC through project H2020-700692(DiSIEM), and by national funds through Fundação para a Ciência e a Tecno-logia (FCT) with references UID/CEC/00408/2019 (LASIGE) and UID/CEC/50021/2019 (INESC-ID). We warmly thank Henrique Santos for feedback ona previous version of this work.

References[1] Akamai. State of the internet, 2017. Q1 2017 report.[2] Maryam Feily, Alireza Shahrestani, and Sureswaran Ramadass. A survey of botnet

and botnet detection. In 3rd International Conference on Emerging Security Informa-tion, Systems and Technologies, pages 268–273, 2009.

[3] Constantinos Kolias, Georgios Kambourakis, Angelos Stavrou, and Jeffrey Voas.DDoS in the IoT: Mirai and other botnets. IEEE Computer, 50(7):80–84, 2017.

[4] Marios Anagnostopoulos, Georgios Kambourakis, and Stefanos Gritzalis. Newfacets of mobile botnet: Architecture and evaluation. International Journal ofInformation Security, 15(5):455–473, 2016.

[5] Michel Van Eeten, Johannes Bauer, Hadi Asghari, Shirin Tabatabaie, andDavid Rand. The role of internet service providers in botnet mitigation anempirical analysis based on spam data, 2010. OECD STI Working Paper2010/5.

[6] Guy Bruneau. The history and evolution of intrusion detection, SANS institute,2001. www.sans.org/reading-room/whitepapers/detection/history-evolution-intrusion-detection-344.

[7] Justine Sherry, Chang Lan, Raluca Ada Popa, and Sylvia Ratnasamy. Blindbox:Deep packet inspection over encrypted traffic. ACM SIGCOMM Computer Com-munication Review, 45(4):213–226, 2015.

[8] Benoit Claise. Cisco systems netflow services export version 9. RFC 3954, October2004.

[9] Benoit Claise, Brian Trammell, and Paul Aitken. Specification of the IP flowinformation export (IPFIX) protocol for the exchange of flow information. STD,77, September 2013.

[10] Bingdong Li, Jeff Springer, George Bebis, and Mehmet Hadi Gunes. A survey ofnetwork flow applications. Journal of Network and Computer Applications,36(2):567–581, 2013.

266 ■ Botnets

[11] S. Harold Javitz and Alfonso Valdes. The SRI IDES statistical anomaly detector. InProceedings IEEE Computer Society Symposium on Research in Security and Privacy,1991.

[12] Herve Debar, Marc Dacier, and Andreas Wespi. Towards a taxonomy of intrusiondetection systems. Computer Networks, 31(8):805–822, April 1999.

[13] Jeffrey Dean and Sanjay Ghemawat. Mapreduce: Simplified data processing on largeclusters. Communications of the ACM, 51(1):107–113, 2008.

[14] Konstantin Shvachko, Hairong Kuang, Sanjay Radia, and Robert Chansler. TheHadoop distributed file system. In IEEE 26th Symposium on Mass Storage Systemsand Technologies, pages 1–10, 2010.

[15] Ali Shiravi, Hadi Shiravi, Mahbod Tavallaee, and Ali A Ghorbani. Toward devel-oping a systematic approach to generate benchmark datasets for intrusion detection.Computers & Security, 31(3):357–374, 2012.

[16] NfDump. https://github.com/phaag/nfdump. Accessed: 2018-12-27.[17] SiLK. https://tools.netsa.cert.org/silk/. Accessed: 2018-12-27.[18] Carl Livadas, Bob Walsh, David Lapsley, and Tim Strayer. Using machine learning

technliques to identify botnet traffic. In Proceedings of the 31st IEEE Conference onLocal Computer Networks, 2006.

[19] Guofei Gu, Roberto Perdisci, Junjie Zhang, and Wenke Lee. Botminer: Clusteringanalysis of network traffic for protocol- and structure-independent botnet detection.In Proceedings of the 17th USENIX Security Symposium, 2008.

[20] Guofei Gu, Junjie Zhang, and Wenke Lee. BotSniffer: Detecting botnet commandand control channels in network traffic. In Proceedings of the 15th Annual Networkand Distributed System Security Symposium, 2008.

[21] W. Timothy Strayer, David Lapsely, Robert Walsh, and Carl Livadas. Botnetdetection based on network behavior. In Botnet detection, pages 1–24. Springer, 2008.

[22] Yuanyuan Zeng, Xin Hu, and Kang G Shin. Detection of botnets using combinedhost-and network-level information. In Proceedings of the 2010 IEEE/IFIP Interna-tional Conference on Dependable Systems and Networks, pages 291–300, 2010.

[23] Anna Sperotto and Aiko Pras. Flow-based intrusion detection. In 12th IFIP/IEEEInternational Symposium on Integrated Network Management and Workshops, pages958–963, 2011.

[24] Laurens Hellemons, Luuk Hendriks, Rick Hofstede, Anna Sperotto, Ramin Sadre,and Aiko Pras. SSHCure: A flow-based SSH intrusion detection system. In IFIPInternational Conference on Autonomous Infrastructure, Management and Security,pages 86–97, 2012.

[25] M. Patrick Collins and Michael K Reiter. Hit-list worm detection and botidentification in large networks using protocol graphs. In Recent Advances inIntrusion Detection, pages 276–295. Springer, 2007.

[26] Thomas Dubendorfer, Arno Wagner, and Bernhard Plattner. A framework forreal-time worm attack detection and backbone monitoring. In 1st IEEE Interna-tional Workshop on Critical Infrastructure Protection, 2005.

[27] Thomas Dubendorfer and Bernhard Plattner. Host behaviour based early detection ofworm outbreaks in internet backbones. In 14th IEEE International Workshop onEnabling Technologies: Infrastructure for Collaborative Enterprise, pages 166–171, 2005.


[28] Tao Peng, Christopher Leckie, and Kotagiri Ramamohanarao. Survey ofnetwork-based defense mechanisms countering the DoS and DDoS problems.ACM Computing Surveys, 39(1):3, 2007.

[29] Aiyung-Sup Kim, Hun-Jeong Kong, Seong-Cheol Hong, Seung-Hwa Chung, andJames W Hong. A flow-based method for abnormal network traffic detection. InIEEE/IFIP Network Operations and Management Symposium, pages 599–612, 2004.

[30] Yan Gao, Zhichun Li, and Yan Chen. A dos resilient flow-level intrusion detectionapproach for high-speed networks. In 26th IEEE International Conference onDistributed Computing Systems, 2006.

[31] Anna Sperotto, Gregor Schaffrath, Ramin Sadre, Cristian Morariu, Aiko Pras, andBurkhard Stiller. An overview of IP flow-based intrusion detection. CommunicationsSurveys & Tutorials, IEEE, 12(3):343–356, 2010.

[32] Gustavo Nascimento and Miguel Correia. Anomaly-based intrusion detection insoftware as a service. In 1st International Workshop on Dependability of Clouds, DataCenters and Virtual Computing Environments, pages 19–24, 2011.

[33] Leonid Portnoy, Eleazar Eskin, and Sal Stolfo. Intrusion detection with unlabeleddata using clustering. In In Proceedings of ACM CSS Workshop on Data MiningApplied to Security, 2001.

[34] Pedro Casas, Johan Mazel, and Philippe Owezarski. Unsupervised network intru-sion detection systems: Detecting the unknown without knowledge. ComputerCommunications, 35(7):772–783, 2012.

[35] Daniel Gonçalves, João Bota, and Miguel Correia. Big data analytics for detectinghost misbehavior in large logs. In Proceedings of IEEE Trustcom, 2015.

[36] Ivo Vacas, Ibéria Medeiros, and Nuno Neves. Detecting network threats usingOSINT knowledge-based IDS. In Proceedings of the 14th European DependableComputing Conference, pages 128–135, 2018.

[37] Luis Sacramento, Ibéria Medeiros, João Bota, and Miguel Correia. FlowHacker:Detecting unknown network attacks in big traffic data using network flows. InProceedings of IEEE Trustcom, 2018.

[38] OpenVault. Open vault. http://openvault.com/. Accessed: 2018-12-27.[39] Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, A density-based

algorithm for discovering clusters in large spatial databases with noise. DataMining and Knowledge Discovery, 96:226–231, 1996.

[40] Ian H. Witten, Eibe Frank, and Mark A. Hall. Data Mining: Practical MachineLearning Tools and Techniques. Morgan Kaufmann, 3rd edition, 2011 Burlington,MA, USA.

268 ■ Botnets

NETWORK SECURITY ATTACKS - Routledge

Documents