Orphaned Servers and Broken Processes 2007 Security Professionals Conference April 12, 2007.

Orphaned Servers and

Broken Processes2007 Security Professionals Conference

April 12, 2007

Agenda

IntroductionsBackgroundDo You Have Orphaned Servers?Vigilance and AwarenessLessons Applied @ CSUNReflections @ CSUNLessons Relearned @ CSUNSummaryQuestions / Discussion

Background

Moran Technology Consulting (MTC) conducted an Incident Audit for a large research university

Audit focused on two (2) systems that were thought to have been compromised:1. A server housed in central IT data center

containing alumni donor information, including > 100,000 social security numbers

2. A server outside of central IT data center containing campus health center data, including > 50,000 patient records

Health Center Server Incident Detection

Vigilant scanning of system following prior incident

Systems Management No monitoring or support

from central IT(in spite of HIPPA related risks)

Comprises detected: A Rootkit (Hacker Defender

Trojan) W32/pate.b virus Server had participated in

attack on another university server

Alumni Donor Server Incident Detection

External complaints of brute force attack emanating from server

Systems Management System was “decommissioned”

and powered-off and removed from the network

System was removed and returned to the network several times (as an orphaned server)

Following compromises detected: A Rootkit 4 GB of unauthorized music

files The Hideout virus

Background

The Compromises occurred 4 months and possibly 12 months prior to detection, respectively.

Background

Why did these compromises occur? Insufficient Operational practices and procedures within

central IT Lack of formalized standards for secure server configuration No clear roles and lines of responsibility resulted in lack of

accountability for systems Failure of change management procedures to ensure proper

system tracking and decommissioning While 75% of campus servers reside outside of central IT

data center little or no effort had gone into securely supporting or monitoring these systems A “silo” culture existed within central IT Staff reductions left central IT with too few resources Previous attempts to establish campus-wide polices failed

due to lack of executive support

Do You Have Orphaned Servers? Do you know who owns and is managing EVERY

server: In the central data center? Across the campus?

How much do you know about the servers on your campus? What types of data do they house or connect to? Who is accountable for these systems and data? Who owns and maintains these systems

Do you have servers on your network that are believed to have been powered OFF but are really powered ON? What are your current decommissioning policies and

procedures Have these policies and procedures been formalized? Do they apply and are they in practice campus-wide?

Vigilance and Awareness

What formal IT Policies and procedures guide security management roles and responsibilities?

Do the they cover EVERY server on the campus, including both Central Data Center servers AND department servers?

Executive Ownership and Participation is REQUIRED for Security Policies to be effective

Lessons Applied @ CSUN1. Decided to use the incident at another institution to

repose the salient questions and to review our security stance!

Have we identified EVERY server in the central data center? Have we identified EVERY server exposed to the Internet? Do we have sufficient information to protect these assets? What additional policies, procedures, and processes are needed?

2. Developed an approach to review our environment. Reviewed firewall rules for all openings Reviewed DNS for named servers Reviewed ARP tables for IP assignments Performed network scans to identified live servers Established primary and secondary system owners for each server

3. Formulated a plan to address deficiencies found.

Reflections @ CSUNWe thought we were in a very secure position … but we were wrong!

We had spent a lot of time protecting the systems and networks … everything you would expect of a professionally run IT organization to do.

We were very surprised by the number of orphans we found on our network.

What introduced our orphaned servers? Well meaning staff (“I’ll get to it.”) Shifts of responsibilities and new staff. (“I thought they/he

owned it”) Overloaded staff and lack of resources. (refer back to #1)

This is a President’s worst nightmare! The overall cost of an incident includes the damage done to the

reputation and credibility of the institution. The question is not whether but when… Hackers want two things:

Personal information they can sell and Access to powerful computing resources

Lessons Relearned @ CSUN1. Always presume that you are less secure than you are!

2. Use external and internal security events to Increase security awareness at both the staff and executive levels

Perform ad-hoc reviews of your environment

3. Establish periodic reviews of your environment, which includes: Scans of the network (don’t assume your change management process are fail-proof)

Review the ownership of servers and identify the administrators

Review the associated Firewall and DNS rules for all servers

4. Structure your IT organization/environment with Clearly defined role and responsibilities for every unit and individual

Predefined separation of duties, with effective working teams.

Established strong change-management procedures that include removing of services from production. (The cleanup process is the killer.)

The “one throat to choke” principal does not apply! Everyone must participate to ensure success.

Summary

1. Security Incidents happen to everyone. Incidents that occur at other institutions are excellent

opportunities to improve your own security awareness and practices

2. Preparedness and Vigilance are key to minimizing cost Presume that you are still vulnerable (You are!) Perform on-going reviews of your environment Develop a Incident Response Plan

3. Institute policies and procedures to identify, manage and properly decommission every server on campus Orphan servers provide a window for hackers to exploit

your entire IT environment

Questions / Discussion