Carol Sabbar and Jim Walsh Carthage College Towards a Better September: Controlling Residence Hall Computing.

Carol Sabbar and Jim Walsh

Carthage College

Towards a Better September: Controlling Residence Hall

Computing

Surviving September?

Can it really get better?

Do we have any control at all?

Who we are: Lowly IT people from a liberal arts college with a pretty limited budget and about 1,200

resident students whose computers are all infected…

Fall 2003 – Blaster!

What happened?• Blaster and Welchia infected nearly every

student computer on campus• DoS attacks shut down the core switch• We shut down whole residence halls to

protect the core• We thought that students could help

themselves… or not…• We turned off ports for hundreds of rooms• We cleaned and patched hundreds of

student computers

We survived but…

• We resolved to never let that happen again

• We had to figure out something• Identify what happened and

why• Figure out how to prevent it• Figure out what we could afford

Determining the CausesVlan110 is up, line protocol is up Hardware is Cat5k Virtual Ethernet, address is 0008.7c6d.d800 (bia 0008.7c6d.d800) Description: Hedberg User VLAN Internet address is 10.7.0.1/16 MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters 2d00h Input queue: 1/75/1139/55 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue :0/40 (size/max) 5 minute input rate 27000 bits/sec, 37 packets/sec 5 minute output rate 149000 bits/sec, 25 packets/sec 13871279 packets input, 2992610801 bytes, 0 no buffer Received 2214895 broadcasts, 0 runts, 0 giants, 170 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 10781241 packets output, 3862081410 bytes, 0 underruns 0 output errors, 0 interface resets 0 output buffer failures, 0 output buffers swapped out

Identifying the Culprits

• Seeing the traffic• Understanding how worms work• Finding infected computers

– How many are patched– How many have anti-virus

• The patch is not the fix, and anti-virus won’t clean these

• Finding the secondary problems:– Spyware– Peer-to-peer

Looking forward to 2004

• Containing outbreaks– Can we isolate them to a building?– To a room?

• Preventing infections– Can we mandate patches?– SP2: good or bad?– Can we mandate anti-virus?

• Chasing down infected machines faster and easier– Can we identify infected machines?

What we couldn’t do…

Some proven solutions just wouldn’t work for us. These included:

• Perfigo or Bradford software – too expensive

• Anything requiring an agent on a student computer – too many installations to do

• A Packeteer for ResNet users only – too expensive

• Broadcast storm control – anomaly related to our wiring plant in res halls

Isolating Outbreaks

• Subnets– Already in place but only the base

• ACLs (details on next screen)

– Isolation to the building– Moving them out to the edge switches– Required new hardware– Required outside expertise

• Storm control on ports– Problematic in our environment– Maybe fall 2005

ACLs on Cisco Switches

Extended IP access list 180 deny icmp any any (92 matches) deny udp any any eq tftp deny udp any any eq 135 deny tcp any any eq 135 deny udp any any eq netbios-ss deny tcp any any eq 139 deny tcp any any eq 445 deny udp any any eq 445 deny tcp any any eq 4444 permit ip 10.12.128.0 0.0.127.255 host 10.2.3.4 deny ip 10.12.128.0 0.0.127.255 any (1410 matches) permit ip any any (23199006 matches)

Preventing infections – Part 1

• Distribution of Symantec anti-virus in summer 2004– Changing our Symantec licensing to

make it “free”– Mailing out the CDs, “Update before

you get here!”– Handing them out at check-in– We do not yet check for its existence

before network access

Preventing infections – Part 2

• Patching– Is SP2 really recommended?– In November, we decided “yes”– Working on PatchLink for on-campus

computers, but requires agent for student computers

– We do not yet require any specific patches for network access

NetReg – a Tool for the Hunt

• Required registration in fall 2004

• Decreases identifying infected machines by several steps

• Turned off rooms posted on our web site

• Need to have someone well-versed in Linux to set it up

Fall 2004 – Any Better?

• No dorms totally shut down in Sept • We cleaned less than 20 computers in

September and October• Infections seldom traveled from building

to building• Infected machines were identified and

ports unplugged the same day• A different story in November… started in

an administrative building with no ACLs

New Problems

With the elimination of the bulk of virus-related outages, we experienced other problems:

• Rogue wireless/wired routers• More spyware• Issues with Windows settings like

“connection bridging” and 802.1x• Some education issues related to

NetReg

Looking forward to fall 2005

• Do all the same as last year

• Increase functionality of NetReg?

• Use a product like Cisco Network Access Control?

• Deploy more switches that can discard DHCP response packets

• Deploy our own wireless in res halls

• We’re open for suggestions…