Assessing Large Networks

Assessing Large NetworksAssessing Large Networks

George G. McBride, CISSP

RSA Conference 2004 San Francisco, CA

George G. McBride, CISSP

RSA Conference 2004 San Francisco, CA

The Goals This Morning:

To share with you some ideas and techniques to efficiently assess large data networks for security vulnerabilities.

These ideas may help for smaller networks, but are optimized for the larger ones.

These are just guidelines. You will have to do what is comfortable and best for you. There is a going to be a learning curve and will take some time to “get it right”.

Presentation Outline

Introduction

Before You Begin

Interviewing

Scanning

Correlating the results

Getting the results back to the System Administrators

Post-Mortem

Getting ready for the next assessment

Notes:

Be advised that all tools can potentially disrupt network operations. Run all tools at your own risk. Myself, Lucent, and RSA are not liable!

US Export and your country’s import laws may restrict the use of certain tools.

Double check any IP address that you scan to ensure that you are authorized to scan the addresses and that the addresses are accurate.

All IP addresses captured in this report were captured on the author’s home network or are screen shots from the product’s home page.

What is a large network?

More than one subnet?

More than the corporate office?

Too many machines that you can’t scan in one day?

More than one country?

More than one continent

All of the continents?

Does it Matter? Let’s assume that a “large” network is pretty big.

Everything is relative…

Vint Cerf’s presentation to RSA

Gives new meaning to “large”

Getting it right the first time:

Do as much as possible up front before you start to gather and collect data.

Set up questionnaires to ensure you get the required data from the appropriate people.

Scanning templates? What are we scanning for?

Repetitive tasks should be streamlined.

Plans, procedures, tasks, and databases should be reviewed and optimized if necessary.

Spend the time the first time to do things right.

Where Do You Start?

The hardest part of large assessments is agreeing on a valuable and useful scope. Will you assess by:— Business Unit (Widget Design)

— Location or Region (Palo Alto Manufacturing)

— System Administrator (George’s systems)

— Platform or Operating System (all Linux boxes)

— IP Address or Subnet (10.5.4.x)

— Vulnerabilities (Only RPC or blank Administrator passwords)

Every location, every company, and every situation will be significantly different from each other. Be flexible. What works today may not work tomorrow.

Get Ready, Get Set, And Hang On!

Start all long term processes early. War Dialing and Wireless sweeps can take a tremendous amount of time.— Don’t underestimate the contributing factors that can increase the time to

complete some of these tasks.

I like to work directly with each location and be able to distribute the collected data at each location.

For example, if I was reviewing the Tokyo office which may have three different business units, I may do one large effort and then divide the assessment results into three completely separate reports.

However, if all business units are managed by the same officers or the same IT group supports the businesses, one report may be better!

Things to Consider During Scope Layout:

Your assessment may involve the same personnel multiple times if working more than one business unit. Likewise, you may not meet everybody.— Ensure you’ve got all questions ready the first time.

With large networks and small IT shops, it may be difficult to obtain specific IP Addresses.— Scan subnets to find active machines and services and then work

with the customer to determine which machines should be scanned.

Nobody wants to take responsibility for a shared machine.— But you can still scan it with proper notification if your policy

permits it!

The Interview

What about questionnaires?— Electronic vs Paper-based

— Preliminary vs Complete

— Ensure that you are meeting with the proper people

I’d still recommend meeting with the personnel, not only for some face time, but to review responses and answer any new questions that come up from the questionnaire.

Make sure that you take the time to acknowledge and thank those that provide information to you.— Recognition and thanks go a long way!

It’s Good To Meet You!

Meeting with the system administrators can give you MORE information than you want to know.

You know you’re asking the proper questions when the responses set your expectations of the scan results.— I.E., you’ll know the scan results before you start scanning.

I like to have a living list of questions already prepared for each group (DBA, help desk, system administrator, etc.).

Make sure you schedule some time to collate and review the information.

Due to political issues, you may need to interview some non-essential persons!

Interviewing (Cont’d)

If you send off the interview questions to be completed prior to arrival, MAKE SURE you review them prior to meeting with the people.— Don’t ask questions for which you’ve already got answers.

— Questionnaires make follow-ups and more in depth questions significantly easier to follow

— Questionnaires help identify vulnerabilities where you might not normally look.

Let your questions be your guide, not a verbatim reading.

I like to schedule interviews on a one to one basis, for about one hour.

Understanding the Network

If the client doesn’t have network diagrams, generate your own.— It’s an awesome deliverable

A picture is worth a thousand words

Don’t get too caught up in the generation of the network diagrams

A great time-saver to understand the network topology

Network Mapping: The Big Picture

The previous page illustrated a Cheops-NG map, available at: http://cheops-ng.sourceforge.net/ is a great and free tool.

LuMeta, at http://www.lumeta.com offers some great mapping services as well as helping you find your perimeter.

HP’s OpenView and other network management tools may be useful to understand the architecture and topology.

Scanning

Before you start scanning, plan on how you will mine the data that you will collect with the scanners.

Getting the data out of the scanner continues to be the “hardest” part, but is a one-time effort.— Database Format

— Spreadsheet

Research and understand the reporting or database structure of your scanning tools to understand how the data will be collected.

Sometimes a simple “Microsoft Excel import” works wonders.

Only Scan For Data You Will Use

It sounds obvious, but don’t scan for data that will be discarded.

Watch DOS attacks.

Nessus, Newt, ISS, eEye, and all scanners allow you to select and de-select vulnerability checks as required.

Review every option.

Only Scan For Data You Will Use

Just for comparison, a Nessus screen.

Note the DOS option!

All scanners have the potential to have undocumented DOS tests!

Data-Output

ISS’ Internet Scanner has several different output options, some of which may be easy to import into a database.

A-Ha! The data is stored locally on the operators PC in a Microsoft Data Engine database format.

Nessus Output

Adobe PDF, Microsoft .DOC, and .TXT file format outputs.

Nessus Output: .NBE Format

Nessus .NBE file format makes it easy to convert into a database format.

Got your baseline?

If you are only interested in checking for changes since the last scan, try the delta or “differential” scan.

This will highlight changes since the last scan including new systems.

Is there ever a time you aren’t really concerned with the “old stuff”?— Only if the old stuff is noise

Extra Sensitive Systems?

Are you or your client concerned with generating traffic which may “negatively impact service” (take the network down)?

Consider exploring the use of Tenable Security’s Nevo tool which is a passive scanner.— Generates no traffic

— Fills the gap between active scans since it immediately detects any new systems once they generate traffic that pass through one of its sensors

By definition, it generates no traffic. If a vulnerable system does not generate enough traffic, all vulnerabilities may not be identified.

Nevo: Passive Vulnerability Detection

Screen shot of Nevo detecting traffic as it runs.

You can see DHCP server identified, WWW server (And version) and SSH running.

Sometimes it gets specific service versions, sometimes it doesn’t.

Leaves some more ambiguity than a passive scan which could conduct additional probes.

Nevo: Passive Vulnerability Detection

Nevo Output = Nessus Input

Based on identified systems and services as well as their version numbers; vulnerabilities can be identified.

Again, if the system wasn’t accessed or didn’t generate any traffic, Nevo won’t find it.

What Systems Should Be Scanned?

What’s on a typical network:— User’s desktops and workstations

— Servers such as file, print, WWW, database, and major applications

— Network equipment such as routers, firewalls, wireless access points, network management equipment

Workstation Risks (Windows, UNIX, Linux, Mac, etc.)— A lot of local data (mail, personal files, local working documents)

— Possibly exploit trust to access other machines

— Often managed by users who can change anything on the system

— A LOT of workstations to scan and A LOT of risks

What Systems Should Be Scanned?

Servers:— Lots of user’s data

— Sensitive data including source code libraries, print queues, restricted web documents

— Malicious users could change, add, delete, and data on the server

— In general, managed by IT Organization, should follow some security standards

— Fewer servers than workstations, but generally more sensitive

Network Nodes:— Continue to find default passwords installed

— Can be used to sniff traffic

— Can be used to disable network segments

Divide and Conquer

Avoid getting lost in the sea of numbers. A scan of a hundred machines properly analyzed is probably better than a scan of several thousand machines.

Reduce the number of machines to scan:— Multi-phased approach where only servers or critical / sensitive

machines

— Perhaps a scan of a cross-section of systems by:

• Operating system

• Administrator

• Purpose – Function

• Configuration

Saturating Your Network And Hosts

Won’t happen with a passive assessment

Hosts that are close to network and processor overload can be pushed to the “edge” and impact performance.

Most scanners err on the side of caution with the number of parallel system scans and service scans. But that is user changeable!

The only time that I consistently see any type of true problem introduced during a network scan is when the scan is across some low-speed WAN connection.— Consider distributed scanning!

Network Overload

Watch Your Network Boundaries

It’s not just your network anymore!— You’ve got connections to customer networks

— And connections to vendor networks

— And connection to business partners

— And joint ventures

— And dial-up users and remote administration / maintenance

If you are assessing ISP provide equipment, ensure that you have the required approval and notify their administration

Watch all local laws, procedures, regulations, etc. What you can do where you start your scan may not be the same laws where you scan terminates.

Speaking of Bad Things

Check and then double-check that you DO NOT HAVE denial of service checks turned on.

If you are doing a large scan, it may be wise to do a subnet or two first. If the machines are vulnerable to some particular test, it’s better to find it out now.

And always, make sure that you notify the system and network administrators when and what you are scanning.— Give everybody your pager or mobile number

Unfortunately, if you do enough scans, “stuff will happen”.

Looking for the needle in the haystack

When you can’t scan every machine for every vulnerability, consider scanning for the top ten threats.

Check out the SANS Top 20 resources at (http://www.sans.org/top20/)

This document is updated somewhat regularly and lists the top 10 UNIX/Linux vulnerabilities and the top 10 Microsoft Windows vulnerabilities

Another SANS Site, http://www.incidents.org, has a real-time “Top 10” list

Top 10 Methodologies…Why Scan?

Several Lucent and Bell Labs researchers performed a study in late 1999 to identify and understand the trends of network and host security vulnerabilities on the Lucent network.

That study is available at: http://www.lucent.com/minds/techjournal/common/arc_issues.html.

The study shows that the “top nine vulnerabilities account for 89 percent of all high risk vulnerabilities”.

The study also indicates that the high and medium risk vulnerabilities account for at 80 percent of all of the vulnerabilities.

When you don’t have time to scan or perhaps prior to your next “enterprise” scan, consider pushing out the fixes for the top ten or twenty vulnerabilities to the systems on your network.

Tackling the False Positive Problems

Given:— 100 Hosts

— 10 Vulnerabilities Per Host

— 95% Confidence Level

1000 Vulnerabilities means that fifty may be suspect. But which fifty and how do you find them?

What about the vulnerabilities that you didn’t detect?

Check all of the results? Don’t check at all?

Random spot check? Run the tool again? Run a different tool?

Verification of Data

Each false positive or undetected vulnerability counts against you in the “credibility” category!

I recommend a sanity check approach which requires a manual review of vulnerability findings with interview responses and configuration information.

Some vulnerabilities are prone to false positives— These should all be checked prior to report distribution

Review the results to make sure that the vulnerabilities match the machine— You can’t have a BIND vulnerability on an HP Printer.

Watch items in the reports that say “may be vulnerable if a file is present”. It should be up to you to clarify those findings.

Data Presentation

In a report where you’ve scanned a large number of systems transcending multiple locations, business units, or support staff, you should consider multiple reports, specific to each recipient.

Summarize the findings into higher levels to present trends and summaries.

In general, only the system administrators need to get a detailed report of vulnerabilities by IP address (with the required fix information).

Include the good things that you found.

The report should be distributed in a draft format immediately after completion.

Getting Ready for the Next One!

Consider a post-mortem:— At least after your first few assessments with all team members

that were involved.

— Even if things went “well”, I suspect that there was room for improvement or positive criticism.

— If you are comfortable, talk to the key contacts at the customer site and solicit their feedback.

Review the processes and steps that took the most time or those that are the most labor intensive. Can anything be done with these?

Continually keep your tools up to date and complete.

Before the next assessment…

Subscribe to (and read!) relevant mailing lists on Security Focus!

NT Bugtraq is an excellent resource

INCIDENTS.ORG and SANS.ORG are equally valuable

FRESHMEAT.NET, ISECOM.ORG, INSECURE.ORG, NESSUS.ORG, and even SNORT.ORG and their mailing lists are excellent resources!— Most have archives to search past messages and lists

Establish a stand-alone network to install and test new tools. As always, your customers network is not a test network.

Questions?

Contact me at [email protected] with any questions that you may have or any thoughts or comments on this talk.

Lucent TechnologiesBell Labs Innovations

Lucent Technologies Inc.Room 2N-611G101 Crawfords Corner RoadHolmdel, NJ 07733Phone: +1.732.949.3408E-mail: [email protected]

George McBrideSenior Manager

IT Risk Management