5/15/2015How to Build a NOC Quilt NOC Workshop Panel Discussion: Indiana University/Global NOC WiscNet Pacific Northwest GigaPop October 3, 2007 How to.

04/18/23 How to Build a NOC

Quilt NOC Workshop Panel Discussion: Indiana University/Global NOC

WiscNetPacific Northwest GigaPop

October 3, 2007

How to Build a NOC


Customers and Expectations

• Who are your customers and what are their expectations/SLA’s?

• Campus, University System, StateNet, GigaPoP, RON, National Backbone, International Connections

• 24x7, Business Hours, Best Effort• Problem Resolution, Triage, Problem

Identification, Service Desk


Supported Services

• In addition to networking, what other services does your organization support?

• Computer Operations

• Support Center

• Security Response

• Grid Computing

• Consulting


Monitoring and Troubleshooting

How large and complex is your network? Large and Simple, or Small and Complex Types of Networking Supported

Optical DWDM Layer 2 – Switch Network Layer 3 – Routed Network

How many hats being worn? Incestuous operational and support relationships?



What level troubleshooting and/or monitoring will your NOC do?

Troubleshooting is based on delegation of responsibility within a NOC organization and/or other related NOC’s

Monitoring based on SLA’s and health of the network & services provided



How will you communicate outages and planned work to customers?

Phone Email / Listserv Web page announcements RSS


Staffing

• Staffing requirements due to SLA's or after hour service response policies

• 24x7, Business Hours, Best Effort


Staffing

• Service Hours - Hours of coverage? Not all NOCs need to be 7x24x365, but what about holidays? Weekends? On-call?


Staffing

• What level of staff needs to be present, and when?

• Tier One: Service Desk (Call Center), Customer Service, Problem Assessment, Network Knowledgeable

• Tier Two: Engineering, Problem Resolution, Perform Maintenance

• Tier Three: Advanced Engineering, Complex Problem Resolution, Escalation Point, Network Planning


Staffing

• Means of responding to issues when NOC is not staffed 24x7?

• Other group within organization answering phone, email, and watches monitoring, contacting the on-call

• Monitoring sends message directly to the on-call pager

• Out-Source after-hours


Organizational Structure

• What staffing tiers/hierarchy will you have for support? Techs? Leads? NEs?



• Escalation practices and policies

• When to move a ticket to an escalation group or person within an organization

• When to inform key personnel within organization or network supported about outages/problems



• Writing/updating procedures, training manuals, etc.

• Who is charged with this? When is it accomplished?

• NOC personnel in conjunction with their other responsibilities

• Dedicated resources


NOC Location

• What is your facility like?

• Does your facility have any unique or particular advantages?

• How do you want to arrange your staff? Separate offices? "War room"?


NOC Funding

• How is your organization funded

• University funds

• State appropriations

• GigaPoP / RON revenue

• Contracts, grants


Tools

• How will you track trouble tickets?

• Enterprise wide systems shared used on university or state wide level

• Proprietary system supported by the NOC and/or other related support groups

• Commercial application or Homegrown


Tools

• How will you track customer information? (Database needs, CRM?)

• Ticketing system

• Database

• Web or Wiki information repository


Tools

• How will you monitor and troubleshoot? Tools, specifically.

• Network monitoring system like Nagios, WhatsUp Gold, HP OpenView

• Weather Maps

• MRTG


Tools

• Are you writing any of your own tools?

• Who will maintain your applications?


Reporting

• What are the key metrics for a NOC?

• How will you measure these?

• Uptime availability

• Nodes monitored

• Trouble tickets

• Phone calls

• Emails


NOC Evolution

• What factors have determined operational changes for your organization - new services, expanded hours, increased number of customers, new equipment types, deeper skill level


October 3, 2007

Building a NOCIndiana University/Global NOC

Case StudySteve Peck



• Who are your customers and what are their expectations/SLA’s

• Indiana University

• Indiana GigaPoP & IP Grid

• I-Light (state of Indiana Higher Ed)

• Internet2

• National LambdaRail

• CIC OmniPoP

IU/Global NOC




• TransPAC2

• AMPATH

• MAN LAN

• HOPI

• Connecticut Education Network (pending)

• OneNet (consulting)

IU/Global NOC


Supported Services

• In addition to networking, what other services does your organization support?

• REN-ISAC (Security service)

• Open Science Grid (Grid monitoring service)

IU/Global NOC



How large and complex is your network?

YES!!! Types of Networking Supported

Optical DWDM Layer 2 – Switch Network Layer 3 – Routed Network

How many hats being worn? Yes! Incestuous operational and support

relationships? Yes!

IU/Global NOC



What level troubleshooting and/or monitoring will your NOC do?

Within the division of work between our Service Desk and Network Engineering groups, our NOC is able to perform all levels of troubleshooting and monitoring. Ranges from simple layer 3 connections to complex DWDM systems.

IU/Global NOC




Email / Listserv Web page announcements RSS Feeds Web and iCalendar based Maintenance/Outage

Calendars Phone (in limited circumstances)

IU/Global NOC


Staffing

• Service Hours - Hours of coverage? Not all NOCs need to be 7x24x365, but what about holidays? Weekends? On-call?

• Service Desk: 24x7x365

• Engineering: Business Hours & On-Call

• Systems Engineering: Business Hours &

On-Call

IU/Global NOC


Staffing

• Staffing requirements due to SLA's or after hour service response policies




On-Call

IU/Global NOC


Staffing

• What level of staff needs to be present, and when?




On-Call

IU/Global NOC


Staffing

• Means of responding to issues when NOC is not staffed 24x7?

• We have on-call rotation for Engineering and System Engineering groups, as well as Service Desk Supervisors.

IU/Global NOC




Service Desk• 2 Shift Supervisors (Day & Night shifts)• 5 Senior Technicians (at least one on

every shift)• 13 Technicians (including hourlys)• 5 Off Front-Line support personnel

IU/Global NOC




Network Engineering• 17 Network Engineers

– Network Engineering Team– Network Planning Team

Systems Engineering• 7 Systems Engineers (+1 open position)

– Application Developers– System Administrators

IU/Global NOC



• Escalation practices and policies

• Service Desk has 15 minutes to assess problem or outage before escalating to Engineering.

• Standard “escalation” processes for outages and problems (immediate, 1 hour, 4 hours, 12 hours, etc.)

IU/Global NOC




• NOC personnel in conjunction with their other responsibilities (Service Desk & Engineering)

• Recently have hired dedicated resources to focus on internal documentation environment

IU/Global NOC


NOC Location• What is your facility like?• State of the art• Does your facility have any unique or particular

advantages?• Showpiece for tours, on the edge of downtown

Indianapolis, close to State Capitol• How do you want to arrange your staff? Separate

offices? "War room"?• War room (for most part), plus offices for

appropriate staff

IU/Global NOC


NOC Funding

• How is your organization funded

• Contracts, grants

• University funds

• State appropriations

• GigaPoP revenue

IU/Global NOC


Tools

• How will you track trouble tickets?

• Footprints ticketing system (manufactured by Numara) for all GRNOC networks & projects

• Peregrine for all IU campus related tickets

IU/Global NOC


Tools

• How will you track customer information? (Database needs, CRM?)

• Ticketing system

• Great Database (developed in-house)

• Web or Wiki information repository

IU/Global NOC


Tools• How will you monitor and troubleshoot?

Tools, specifically.• Monitoring: AlertMon homegrown web based

alert interface links Nagios to Footprints ticketing system

• Visualization: Weather Maps, Utilization Graphs (SNAPP)

• Management: GRNOC Database and linked systems (RADIUS, DNS, etc.)

• Other special-purpose tools (examples: Spanning Tree state map, Juniper Firewall Filter Grapher, Syslog Analysis Scripts, prefix list diff checker, etc.)

•IU/Global NOC


Tools

• Are you writing any of your own tools?

• Yes. Large deployment of custom developed and open source software with a sprinkle of commercial software.

• Who will maintain your applications?

• Systems Engineering Team: Software Developers and System Administrators.

IU/Global NOC


Tools: Monitoring• AlertMon: “Big Board” front-end to monitoring system• Nagios: open source network monitoring system with

custom-developed Plug-ins and monitoring agents. Monitor variety of services: BGP session status, Interface up/down, IS-IS Adjacency, Router/Switch CPU load, MSDP status, etc.

IU/Global NOC


Tools: MonitoringNagios Monitoring System

IU/Global NOC


Tools: Weathermaps

“Mini Maps” embedded in NOC web sites

IU/Global NOC


Tools: Weathermaps

IU/Global NOC


Tools: Management

• Network Management System including:– Contact Management– Device Management

• Inventory• RADIUS-based authentication• DNS record generation• Configuration archiving• Tied to utilization measurement system

– Circuit Management (Layer 0, Layer 1, Layer 2)– IP Address Management– Services Management

GRNOC Database

IU/Global NOC


Tools: GRNOC DB

IU/Global NOC


Tools: GRNOC DB

IU/Global NOC


Reporting

• What are the key metrics for a NOC?

• How will you measure these?

• Uptime availability

• Services monitored

• Trouble tickets

• Phone calls

• Emails

IU/Global NOC


NOC Evolution

• What factors have determined operational changes for your organization - new services, expanded hours, increased number of customers, new equipment types, deeper skill level

• Yes! Optical networking and dynamic circuit switching has made the most difference

IU/Global NOC


October 3, 2007

Building a NOCWiscNet

Case StudyKevin Schmidtke


Supported Services

• In addition to networking, what other services does your organization support?– Service & System support, providing support for all

servers, mainframes and applications serving UW-Madison as well as some UW- System wide applications.

– All NOC staff are cross trained to provide both network support as well as server and application support

WiscNet


NOC Funding

• How is your organization funded– Budget is completely revenue based

– Service support (monitoring, troubleshooting and incident management) and server hosting fees account for 64% of total budget/revenue.

– Network support accounts for 36% of budget/revenue• WiscNet (44%), UW Madison campus (41%) and BOREAS

Net (15%)

WiscNet




– UW Madison Campus– WiscNet– BOREAS Net

• No defined SLA’s for any of customers• Customers expect maximum up-time, efficient

incident management, effective change management at a reasonable price.

WiscNet




– UW Madison Campus• Cisco based network reaching 80 campus buildings• Managing/monitoring over 1,140 3750 Cisco switches• Managing/monitoring over 40,000 ports across

campus • Managing/monitoring 1,930 wireless access points

across campus

WiscNet




– WiscNet• State-wide backbone connecting 500 + members• 20+ local telcos provide circuits to members

throughout the state. • Members include all UW campuses and institutions

across the state, all state technical colleges, all private colleges in the state, the majority of K-12 school districts and some libraries across the state.

WiscNet




– BOREAS Net• Regional Optical Network• WiscNet is the operator• A collaboration of four major research institutions in

the upper Midwest: Iowa State University, the University of Iowa, the University of Minnesota, and the University of Wisconsin-Madison.

WiscNet



What level troubleshooting and/or monitoring will your NOC do? Node up/node down Network Closet environmental alarms UPS alarms Backbone error monitoring Access Switch port errors Bandwidth threshold monitoring

WiscNet




– Outage pages for incidents/unplanned outages– E-mail communications for changes/planned

outages• In house tools used to determine sites affected and

build custom e-mail list.

WiscNet


Staffing

• Service Hours– NOC level 1 staffed 24X7X365 including all

holidays– Level 2 engineering on-call during off-shift and

holidays– Remote ‘hand and eyes’ on call for UW Campus

and BOREAS Net

WiscNet


Staffing

• What level of staff needs to be present, and when?– A minimum of two NOC level 1 support

personnel staffed at all times except holidays; only one NOC level 1 support personnel staffed on holidays.

– Level 2 network support (Operational Engineers) and Network engineers available during the day.

WiscNet


Staffing

• After hour service response policies

– Varies for each of the networks depending on on-call or best effort support model

– Immediate phone support expected for on-call staff

– If on-call staff presence is necessary, one hour on-site response is expected

WiscNet



• Escalation practices and policies – Escalation policies and procedures defined in

an online knowledge database– Policies and procedures differ from network-

to-network– On-call and best effort support models in

place

WiscNet




– UW Campus• NOC Level 1 on site at all times• Operation Engineers/Level 2 on-site during the day,

on call after hours• Network Engineers on-site during the day, best

effort after hours

WiscNet




– WiscNet• NOC Level 1 on-site at all times• Network Engineers on site during the day, best

effort after hours• Tech Support (service support) on site during the

day, best effort after hours• Telco providers available at all times• Remote tech contact is best effort

WiscNet




– BOREAS Net• NOC Level 1 on-site at all times• Network Engineers on site during the day, best

effort after hours• Level 3 Communications available at all times

WiscNet




– Shared responsibility between NOC staff, Tech Support and Network Engineering

– All support/training documents stored in an online knowledge database

– Ability to share documents between instances of the knowledge database

– A defined lifecycle for each article; draft, team review, production, update/review, etc.

WiscNet


NOC Location

• What is your facility like?– Large control center facility. Network support

and service support duties performed in same area

– War room and the primary workstation in control center available to manage major incidents. Primary workstation in control center is preferred.

– Network engineers and tech support staff in adjacent campus buildings.

WiscNet


Tools

• How do you track customer information? (Database needs, CRM?)

– All customer information (support, configuration, contact, etc.) is maintained in database tools developed in-house.

– Different networks use different tools– Customer information can be searched by

site id, circuit id, vlan, IP address, etc.

WiscNet


Tools

• How will you monitor and troubleshoot? Tools, specifically.

– UW Madison campus – HP Network Node Manager, syslogs and custom monitoring tools.

– WiscNet – HP Network Node Manager, syslogs and custom monitoring tool.

– BOREAS Net – Infinera Digital Network Administrator

WiscNet


Tools

• Are you writing any of your own tools?– None of our support tools are written at the

NOC. The NOC identifies the need for new tools or modifications to existing tools and the network engineers do the development.

• Who will maintain your applications?– The tools are maintained by the Network

Engineering staff

WiscNet


Tools

• How do you track trouble tickets?

– Incidents for all supported networks are tracked through Clarify

– Each network has a unique product name– The subject line includes the site name

and/or device name – Weekly reports are generated for each of

the networks and distributed

WiscNet


Reporting

• What are the key metrics for a NOC?– Incident management

• Detection• Troubleshooting• Communication• Escalation

• How will you measure these? – Post incident review – Customer feedback– Problem management

WiscNet


NOC Evolution

• What factors have determined operational changes for your organization?– New technologies & network configurations– Changes in support models dictated by

campus– Newly configured networks– Incident tracking trends and analysis.

WiscNet


October 3, 2007

Building a NOCPacific Northwest GigaPop

Case StudyLinda Hornung



• Our customers are in the Pacific Northwest and Pacific Rim.

• We track customer information in a database designed and maintained locally.

• Time zones have driven our need for 7x24.

• We cover 15 time zones which also makes scheduling planned maintenance difficult.

PNWGP



• Escalation policies also drive our need for on-call schedules and on-site personnel.

• Our customers expect 48 hours notice prior to work, unless it is an emergency.

• Outages are communicated via an application we have built locally.

• We also want to know when our customers have planned work.

PNWGP



• Expectations are included in the contract and located on the PNWGP web page.

• We prefer to have the NOC as the primary customer contact point for our organization in order to maintain quality of service when a ticket moves between groups.

PNWGP


Supported Services

• In addition to network monitoring for PNWGP, our NOC monitors:

• UW campus connectivity, including two hospitals (both layer 2 and layer 3, and wired as well as wireless, many UPSs),

• Approximately 500 sites throughout WA state for the K-20 network,

• Pacific Wave, and Transit Rail.

PNWGP


Supported Services

• We do not do any host, server, or customer application monitoring.

• We do not do desktop support.

• We do not reset passwords or arrange services such as email and web.

• We do not monitor data center security alarms such as for fire or flood.

PNWGP



PNWGP: approx. 20 customers, 5 node sites, MPLS, BGP, VPN

Campus: 115,000 connected devices, ~100 remote locations, ~7,000 wireless APs, ~4,000 layer 2 switches, ~150 routers.

PNWGP



Over 450 Washington state K-20 sites monitored.

Transit Rail, approx. 5 node sites,BGP and ISIS status.

PNWGP



PNWGP troubleshoots with the customer on routing problems, latency, and loss of connectivity.

IS-IS and IGP status is monitored. We manage DWDM, Sonet, and MPLS

circuits. Complexity is increased with escalation

paths being different depending on what isn’t working.

PNWGP



Outages and planned events are sent via email announcement in a standard format.

We include the date/time of the work or when the outage began.

If the customer’s connectivity is entirely down, we also call or page them.

Updates are sent at predefined intervals for large events, or when we have a change in status.

PNWGP


Staffing

• We are 7x24 with full-time staff. • Weekends we only have one person

covering each day, so vacations and sick time are problems.

• Holidays are covered by on-site and on-call staff.

• On-call consists of a 7-day period and rotates among all NOC staff on a regular schedule.

PNWGP


Staffing

• Tier 1 are student staff, also called Network Analysts.

• Tier 2 are full-time staff, most are titled Network Specialists.

• Tier 3 are full-time staff, titled Network Engineers.

PNWGP


Staffing

• We advertise to all customers an on-site 7x24 staff for immediate response to outages.

• Our SLAs indicate when a problem should be escalated beyond the oncall staff to a manager, director, and higher at any time of the day or week.

PNWGP


Staffing

• Day shift M-F requires that we have engineers (Tier 3) staff in the building.

• Off-hours we have Tier 2 staff always present in the building. They will escalate as necessary, either due to outage severity or complexity, or at a customer’s request.

PNWGP


Staffing

• To maximize staffing efficiencies, 2nd and 3rd shift personnel report to Computer Operations managers, rather than the NOC manager.

• These staff provide all of the same support and have the same training/access as the daytime Tier 2 staff in the NOC.

PNWGP


Staffing

• We have on-call lists with a primary and secondary person for backup when NOC staff is not on-site.

• A separate call list exists for escalation to managers.

• Engineering and other service groups are also available on-call 7x24x365.

PNWGP



• PNWGP has recently adopted a tiered approach for staffing.

• NOC Network Engineers are available after 6pm M-F and weekends only via on-call.

• One manager, reporting to the Director of Operations.

PNWGP



• Our escalation practices and policies are based on length or severity of outage.

• At predetermined intervals additional management levels are notified of severe outages in order to help with escalation at other organizations (telcos), or to keep peers updated at the affected sites.

PNWGP



• Many people in the NOC work to provide and update training materials; SMEs write procedures.

• We use a wiki to maintain our documentation.

• We were able to have a tech writing expert give us training on writing effective procedural documents and re-organize our wiki.

PNWGP


NOC Location

• Currently, our NOC has some single-occupant offices and some multiple (2-3). We are all on the same floor.

• We find that being located near the Network Engineering team is quite helpful for urgent escalations.

• Next year we will be moving into a new location and we do not yet have details on what that will look like.

PNWGP


NOC Location

• Consider lighting and noise control with shared offices.

• How many monitors will each person need? Will you use a large central monitor for some things?

• Provide an impromptu meeting space for collaboration on big events.

PNWGP


NOC Location

• Conference bridges greatly enhance collaboration across geographic distances whether working on outages or events.

PNWGP


NOC Funding

• UW funding comes primarily from the state tax payers of Washington,and UW Medical Centers.

• Funding also comes from the PNWGP and customers and WA state K-20 network.

• If you recharge you will need a business and billing model.

• Or will you use time and materials?

PNWGP


Tools

• A very useful tool is live chat or IM for coordinating efforts no matter where your office is.

• Our customer information is tracked in a home-grown database which has grown and morphed over a dozen years.

• New needs such as SLAs and layer 1 info now require significant investment in upgrades.

PNWGP


Tools

• Our monitoring system – surprise – is also “homegrown”.

• We monitor interface state and IP reachability; performance and protocol state connectivity will soon be integrated into our “event system” (NMS)

• Automated tools can page the appropriate group to notify them of outages or threshold conditions.

PNWGP


Tools

• We have a separate Tools team (with 10 staff members) who design, write, implement, and maintain tools.

• This allows us to have full-featured and robust tools.

• One trade-off is fewer “one-off” tools for specific or isolated issues.

PNWGP


Tools

• PNWGP uses Request Tracker, RT, an open-source application to track trouble tickets.

• Weekly reports are generated for our Directors by sector, severity, and type.

• Monthly reports are generated by sector for billing purposes.

PNWGP


Reporting

• Key metrics we track include: 1. Ticket numbers by sector for billing

2. Phone call volumes

3. Duration of outages

4. Root Cause Analysis for high-impact events

• Outage time is measured by duration of the customer impact.

• After Action Review and Follow-up is conducted for serious events.

PNWGP


Reporting

• Monthly report is emailed to the customer for traffic sent to/from their site.

• Our internal reporting includes “operational impacts” to groups under our main organization.

• How do you measure your NOC’s success? Response times? Reduced calls?

PNWGP


NOC Evolution

• Factors that have determined operational changes for our organization have been increased size, complexity and number of networks monitored;

• Need to respond to outages 24 hours/day with on-site personnel rather than paging;

• Skill and responsibility levels have increased significantly, and continue to do so.

PNWGP


Panel Contact Information

Linda HornungPacific Northwest GigaPop

[email protected]

Steve PeckIndiana University/Global NOC

[email protected]

Kevin SchmidtkeWiscNet

[email protected]

5/15/2015How to Build a NOC Quilt NOC Workshop Panel Discussion: Indiana University/Global NOC WiscNet Pacific Northwest GigaPop October 3, 2007 How to.

Documents

noc slide

noc noc location

noc monitoring

noc customers

noc personnel

noc organization andor

outagesproblems slide

network planning slide