View Source

Table of content

Description

At the TF-NOC meeting during fall 2010 we agreed to collect information about what the TF-NOC participants thought would be interesting to share from their organisation, as well as information they would be interested in receiving from other organisations. The aim is to summarize that collected information on this page.

Summary of interesting topics

BELNET
What BELNET would like to present:

lessons learned from outsourced incident & change management
approach & tools: how the different departments & the NOC's work together
scripting & naming convention: making the network more consistent for the NOC
introduction of our customers to the outsourced NOC's: how to change their attitude
keeping the quality of the NOC's in check over time: means & experiences

What we would like to hear from other participants:

procedures: presentation of the NOC's current procedures & who is responsible for maintaining them
authentication: how do you verify the caller / e-mail sender
reponsibility of your NOC: how far does it go?
good & bad experiences with outsourcing
distributed NOC's (as opposed to a NOC on a central location): pro's & con's

DANTE
Other basic information

Organization (tiers, and which tiers are in-house or outsourced);
Responsibilities (e.g. monitoring, configuring/provisioning, direct or indirect repairs, DNS updates, RIPE updates, POP access and deliveries, reporting);
Hours of operation (office and on-call);
Network type (IP only, hybrid, switched/transmission only, LAN only);
Network services: IPv4/IPv6/multicast IPv4/multicast IPv6/VPNs/QoS/etc

Of most interest to learn from others

Main Monitoring systems and alarm thresholds;
Target repair times;
Number (approx) of ticket mail lists the NOC is on, and approx number of tickets received on these lists
Connected networks: Types of organization and approx numbers for subscribers, peers and upstream IP providers
Core equipment vendor/make, approx numbers;

CARNet
We would like to include in our presentation topics about:

CARNet network, users & Core/Access nodes - how to treat different users and different parts of network
CARNet NOC - organization, tiers, team operating hours, knowledge sharing
NMS - our tool functionalities / developement
prodecudes and documentations - what should be strictly put "on paper" (level of knowledge and procedures) from our experience ...

Topics about what we would like to learn from others are:

NOC key performance indexes - how to measure/to improve ?
Knowledge management and NOC staff training
NMS - tools (functionalities)
NOC organization and procedures: Tier1-N functions, # of persons in shifts and in total, work quality control, peak hours
How to work proactively in NOC (active/passive monitoring)

PIONIER
Five topics PSNC would like to present about NOC:

networks PIONIER (NREN) and POZMAN - technical structure (DWDM, switches, routers) and its influence to NOC structure, organizational structure: Layer 1,2 (NOC) and Layer 3 (IP NOC), employees and their functions
procedures used in our NOC
our TTS - public part, experience, missing functions
NOC tools - tools used in the NOC, missing functions, tools developed and maintained by PSNC
Experiences in joint management application design & deployment with commercial company

Five topics (or questions) that PSNC is interested in knowing how other NOCs handle:

how to minimize users calls (public available FAQ? tools? documentation?)
reports - what? how often?
data and tools integration - how? recommended tools?
serving users calls - how to automate it, distributing calls among different NOC administrators
(D-)DoS - how to detect network attacks

ARNES
ARNES could present:

Pros and contras for dedicated NOC personnel
NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)

ARNES would like to hear from others about:

NOC staff job description
NOC assesment and KPI – measuring NOC performance
NOC to user communication (communication with more or less experienced users, how to keep user contact info up-to-date, personalized web interface for NOC to user communication)
NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)
How many customers do you have? (depends on if you are counting persons, networks, hosts,..)
What kind of agreements do you have with service providers and customers (SLA,..)? Without financial info, but just roughly descriptions of responsibilities.

NORDUnet
Things NORDUnet could present:

Troubleticketing system, experiences from building our own.
Managing user interfaces (using people outside the NOC to aggregate user requests)
Using external community tools (letting the community answer questions instead of the NOC).
NOC organisation (staffing, internal escalations, handover and schedule)
Reaching your network from anywhere (out-of-band, in-band and VPN)

Things NORDUnet would like to hear about from others:

How is knowledge dissemination handled between groups and new NOC employees?
How is monitoring and alarm correlation handled between different systems?
How are new services and users provisioned?
What kind of routines are used in regards to change management (e.g. in planned maintenance)?
What other services than the network does your NOC operate and how do they differentiate from the network service?

GNOC at Indiana University
Five topics to introduce the Global Research NOC at Indiana University:

Global Research NOC Service Desk<http://globalnoc.iu.edu/tour.html> provides 24x7x365 technical call center support, trouble ticket management, and workflow support. The service desk is housed in a state-of-the-art call center in Indiana University's Informatics and Communications Technology Complex (ICTC) on Indiana University's Indianapolis campus. This call center features a fully customizable 30-foot screen used to monitor and troubleshoot the current health of the various networks supported. We provide NOC support for the most advanced research and education networks in the country. http://globalnoc.iu.edu/tour.html
Willingness to assist network customers with other services and requests outside of traditional front-line network operations, such as network redesign planning and utilizing our tool set.
A network knowledgeable tier-one Service Desk that can assess network issues intelligently and can provide engineering with helpful information as they begin troubleshooting.
Internally developed tools such our network database http://globalnoc.iu.edu/grnoc-tool-set.html
International networking efforts

Five topics the GlobalNOC at Indiana University would like to know about other NOC's:

What methodologies and tools do you use to understand complex cross-network relationships and dependencies?
How do you organize and update your internal documentation environment for easy access and search, as well as providing clear and accurate information?
How do you integrate the different groups within your NOC (Service Desk-Tier One, Engineering-Tier Two & Three, Systems Engineering, etc) so each has clear expectations of what each group does, with all working as a unified team?
With R&E networking moving towards having more regional connectedness between smaller networks, RON's, universities, etc, are you planning for a new phase in networking where specific operational differences between NOC's may negatively affect the reliability of the services provided? (for example: different response levels from hardware and fiber vendors, vastly different maintenance windows, NOC's not being 24x7, etc.)
What type of training methodologies do you use to bring new employees up-to-speed quickly so they can start making a valuable contribution to the operation?

SWITCH
The topics SWITCH could tell about:

network, equipment, services, customers
management of fibers - with segments from different providers
staff
outsourced services
work flows
tools
experiences and things we like to improve

The topics SWITCH is interested to know from other NOCs:
The same points like above with special focus on:

incident handling
tools and especially
tools to support the distributed handling of incidents

GRNET
GRNET is keen in presenting one of the topics trailed below:

Efficient administration of many Linux based servers,
Providing VPS service,
Network database (GRNnet-DBII),
Visualisation tools,

On the other hand, we would like to hear how our colleagues tackle the followings:

NOC key performance indexes,
Automated SLA monitoring tools,
Network monitoring tools (may be different than the ones above),
operation of CERT team and security policies

RedIRIS
Five topics to introduce RedIRIS NOC:

Type of network. Equipment and type of lines.
Out of band network.
Trouble ticket system.
SLAs with vendors and providers and type of contact.
Organisation of our NOC.

Five topics RedIRIS would like to know about other NOCs:

Organisation and roles of NOC members. Number of levels. Outsourced vs. staff-in NOC. Pros and cons. Functions. Experience with externalized NOC.
Monitoring tools.
Acquiring, management and operation of dark fibre. Procedures. Optical NOC differenciated from IP NOC?
Services managed by the NOC.
Network database and integration with other monitoring tools.

FCCN
Five topics to introduce FCCN's NOC:

Deployed network (Optical/Ethernet/IP)
Out-of-band systems (MRV using GSM/GPRS)
Service Desk & Automated reporting (per member)
Deployed probes network (QoSMetrics solution)
Past horror stories: What you can try to avoid & Murphy's Law.

Five topics FCCN would like to know about other NOCs:

How is the 24x7x365 NOC service guaranteed?
Which monitoring tools are you currently using. Which ones have you already used in the past and were abandoned. How do you keep network/topology maps updated?
Which services and incidents are covered by the NOC? How do you deal with requests/incidents that are outside your NOC's scope?
Do you have/use a fixed maintenance window?
How do you deal with performance issues/incidents and e2e issues regarding traffic that flows through your network.

CERN
CERN would be glad to present from the following list of topics:

Transforming 2 layer network support into 3 layer support with outsourcing
Integration of network and telecom support
Creation of a knowledge repository for network/telecom support
Structure and roles in a multi-layer NOC operating multi-type networks (campus network, international research network, internet exchange point)
The challenge of using multiple ticketing systems

CERN would be interested in hearing about:

Roles and responsibilities (structure, who does what)
Relation and communication between the network operations team and other internal entities (design, deployment)
How operation is done outside working hours including weekends
How common experience is handled: specialized vs. universal team members
How procedures are created and stored

RESTENA
Five topics to introduce RESTENA's NOC:

Infrastructure building and operations (dark fiber, net devices,server-rooms, POPs, servers)
An "all-in-one" NOC: structure/organization/tools
24x7x365 service support
Coverage of user support (helpdesk)
Services operated (from lambdas to large scale e-mail accounts)

Five topics RESTENA would like to know about other NOCs:

User (customer) information exchange: channels and content
NOC activities/coverage and staff (network, services, community details)
Organization of inventory (tools, DB, naming conventions,etc.) and change management (technically and administratively)
Staff organization, i.e. services support strategy (per person/team/NOC) with their respective pros/cons
NOC awareness and identification: internally and among user community

SURFnet
Items we from SURFnet can present our experiences

acquring and managing dark fiber
managing an outsourced NOC
procedures - the good and bad things

Things we would like to hear about from other NOC's:

how to inform/communicate with customers
branding of an outsourced NOC
ticketing systems
CMDB experiences

CESCA
And these are topics that we would like to introduce about our NOC:

Virtualization of the network.
NOC tools and how to adapt them to our needs (development).
Users management. How to approach our users, depending on the action (change, incident, request,..).
Survey for our services, what to ask and how?
Out-of-band access to our network.

These are the topics that we would want to know from other NOCs:

Ticketing Systems, how to optimize their use.
Integration of diferent tools.
Partially outsourced NOCs (24x7, weekends,...): what kind of information do you give them, how do they interact with you and your users?
Escalation of incidents. How many levels?
Documentation: what can and can not be documented.

HEAnet
5 important bullet points introducing our NOC(s)

Heuristic Failure analysis (how we use heuristics to determine SLA calculations and where they are and are not appropriate.) (Also interested in discovering if others are doing this.)
Ticketing, including automatic ticket generation for incidents.
Contract and supplier management - how HEAnet does it and how well it works (or doesn't work.)
Reporting to your clients. How do NRENs report to clients? Periodical and Real-time reporting. On demand reporting.
24 x 7 NOC cover. Do you provide it? What SLA? How do you hire staff and plan rotas, etc? Is there a real client demand for it?

5 questions what you want to know about other NOCs

How are staff hired for the NOC? Skillsets? Interview processes?
Standards compliance and process adoption. Have other NOCs adopted ISO9000? ITIL? PRINCE2? APM? ISO20000? Six Sigma? Other standards?
What scope of services does the NOC support? Networking? Servers? Value added services like DNS, web hosting, etc.? HPC?
How do NOCs source support and SLAs? What about penalty clauses? What's reasonable, what works and how are they enforced?
How do you assign value to what the NOC does? Do you cross charge for projects that will be handed over to the NOC? How to you prove to upper management that the NOC is good value for money?

Interesting topics mapped to the work items

Taxonomy of NOC organisations, internal processes, workflows

Topics participants are interested in sharing information about	Topics participants are interested in getting input on
Network	Network
Infrastructure building and operations (dark fiber, net devices, server-rooms, POPs, servers)	Core equipment vendor/make, approx numbers
Deployed network (Optical/Ethernet/IP)
Network type (IP only, hybrid, switched/transmission only, LAN only)
Acquiring and managing dark fiber	Acquiring, management and operation of dark fibre.
Management of fibers - with segments from different providers
Network services: IPv4/IPv6/multicast IPv4/multicast IPv6/VPNs/QoS/etc
Network, users & Core/Access nodes - how to treat different users and different parts of network
Reaching your network from anywhere
Out-of-band access to our network
Out-of-band systems (MRV using GSM/GPRS)
NOC functions	NOC functions
Survey for our services, what to ask and how?	Services managed by the NOC.
Services operated (from lambdas to large scale e-mail accounts)	What scope of services does the NOC support? Networking? Servers? Value added services like DNS, web hosting, etc.? HPC?
Providing VPS service
Responsibilities (e.g. monitoring, configuring/provisioning, direct or indirect repairs, DNS updates, RIPE updates, POP access and deliveries, reporting);	Responsibility of your NOC: how far does it go?
	What other services than the network does your NOC operate and how do they differentiate from the network service?
	Which services and incidents are covered by the NOC? How do you deal with requests/incidents that are outside your NOC's scope?
	Connected networks: Types of organization and approx numbers for subscribers, peers and upstream IP providers- Roles and responsibilities (structure, who does what)
	Operation of CERT team and security policies
	Number (approx) of ticket mail lists the NOC is on, and approx number of tickets received on these lists
NOC structure	NOC structure
Staffing, internal escalations, handover and schedule	NOC activities/coverage and staff (network, services, community details)
24x7x365 service support
24 x 7 NOC cover. Do you provide it? What SLA? How do you hire staff and plan rotas, etc? Is there a real client demand for it?	How is the 24x7x365 NOC service guaranteed?
Hours of operation (office and on-call)	How operation is done outside working hours including weekends
Pros and contras for dedicated NOC personnel	How common experience is handled: specialized vs. universal team members
A network knowledgeable tier-one Service Desk that can assess network issues intelligently and can provide engineering with helpful information as they begin troubleshooting.	NOC organization and procedures: Tier1-N functions, # of persons in shifts and in total, work quality control, peak hours
Structure and roles in a multi-layer NOC operating multi-type networks (campus network, international research network, internet exchange point)	Staff organization, i.e. services support strategy (per person/team/NOC) with their respective pros/cons
Global Research NOC Service Desk provides 24x7x365 technical call center support, trouble ticket management, and workflow support. The service desk is housed in a state-of-the-art call center in Indiana University's Informatics and Communications Technology Complex (ICTC) on Indiana University's Indianapolis campus. This call center features a fully customizable 30-foot screen used to monitor and troubleshoot the current health of the various networks supported. We provide NOC support for the most advanced research and education networks in the country.	With R&E networking moving towards having more regional connectedness between smaller networks, RON's, universities, etc, are you planning for a new phase in networking where specific operational differences between NOC's may negatively affect the reliability of the services provided? (for example: different response levels from hardware and fiber vendors, vastly different maintenance windows, NOC's not being 24x7, etc.)
Technical structure (DWDM, switches, routers) and its influence to NOC structure, organizational structure: Layer 1,2 (NOC) and Layer 3 (IP NOC), employees and their functions
	Distributed NOC's (as opposed to a NOC on a central location): pro's & con's
	Optical NOC differentiated from IP NOC?
Transforming 2 layer network support into 3 layer support with outsourcing	Organisation and roles of NOC members. Number of levels. Outsourced vs. staff-in NOC. Pros and cons. Functions. Experience with externalized NOC.
Tiers, and which tiers are in-house or outsourced	Partially outsourced NOCs (24x7, weekends,...): what kind of information do you give them, how do they interact with you and your users?
Managing an outsourced NOC	Branding of an outsourced NOC
Lessons learned from outsourced incident & change management	Good & bad experiences with outsourcing
Setting up and maintaining a NOC	Setting up and maintaining a NOC
Knowledge sharing	Knowledge management and NOC staff training
	NOC staff job description
	How are staff hired for the NOC? Skillsets? Interview processes?
NOC assessment	NOC assessment
	NOC key performance indexes - how to measure/to improve?
	How do you assign value to what the NOC does? Do you cross charge for projects that will be handed over to the NOC? How to you prove to upper management that the NOC is good value for money?
NOC workflows	NOC workflows
Procedures used in the NOC - the good and bad things	Presentation of the NOC's current procedures & who is responsible for maintaining them
Workflows
	How do you deal with performance issues/incidents and e2e issues regarding traffic that flows through your network.
	What kind of routines are used in regards to change management (e.g. in planned maintenance)?
	Do you have/use a fixed maintenance window?
	How are new services and users provisioned?
	How does your NOC take care of incident handling?
	(D-)DoS - how to detect network attacks
	What are your target repair times?
	How to work proactively in NOC (active/passive monitoring)
	How are new services and users provisioned?

NOC front-end development and improvement

NOC tools, interworking/interfacing issues, and automation

Topics participants are interested in sharing information about	Topics participants are interested in getting input on
Monitoring tools	Monitoring tools
CARNET: NMS - our tool functionalities / developement	DANTE: Main Monitoring systems and alarm thresholds
PIONIER: NOC tools - tools used in the NOC, missing functions, tools developed and maintained by PSNC	CARNET: NMS - tools (functionalities)
ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)	ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)
GNOC IU: Internally developed tools such our network database http://globalnoc.iu.edu/grnoc-tool-set.html	SWITCH: tools
SWITCH: tools	GRNET: Network monitoring tools (may be different than the ones above)
GRNET: Visualisation tools	RedIRIS: Monitoring tools.
CESCA: NOC tools and how to adapt them to our needs (development).	FCCN: Which monitoring tools are you currently using. Which ones have you already used in the past and were abandoned. How do you keep network/topology maps updated?
Multidomain tools: Distributed handling of incidents	Multidomain tools: Distributed handling of incidents
	GNOC IU: What methodologies and tools do you use to understand complex cross-network relationships and dependencies?
	SWITCH: tools to support the distributed handling of incidents
Ticketing systems	Ticketing systems
NORDUNET: Troubleticketing system, experiences from building our own.	SURFNET: ticketing systems
	CESCA: Ticketing Systems, how to optimize their use.
Knowledge management: Configuration Management Database/Network Database	Knowledge management: Configuration Management Database/Network Database
GNOC IU: Internally developed tools such our network database http://globalnoc.iu.edu/grnoc-tool-set.html	RedIRIS: Network database and integration with other monitoring tools.
	RESTENA: Organization of inventory (tools, DB, naming conventions,etc.) and change management (technically and administratively)
	SURFNET: CMDB experiences
SLA reporting and monitoring tools	SLA reporting and monitoring tools
ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)	ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)
	GRNET: Automated SLA monitoring tools
Chat/communication/coordination tools	Chat/communication/coordination tools
NORDUNET: Using external community tools (letting the community answer questions instead of the NOC).
Integration of tools	Integration of tools
BELNET: approach & tools: how the different departments & the NOC's work together	PIONIER: data and tools integration - how? recommended tools?
ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)	ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)
RESTENA: An ""all-in-one"" NOC: structure/organization/tools	RedIRIS: Network database and integration with other monitoring tools.
CESCA: NOC tools and how to adapt them to our needs (development).	CESCA: Integration of diferent tools.