You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Next »

Table of content

Description

At the TF-NOC meeting during fall 2010 we agreed to collect information about what the TF-NOC participants thought would be interesting to share from their organisation, as well as information they would be interested in receiving from other organisations. The aim is to summarize that collected information on this page.

Summary of interesting topics

BELNET
What BELNET would like to present:

  • lessons learned from outsourced incident & change management
  • approach & tools: how the different departments & the NOC's work together
  • scripting & naming convention: making the network more consistent for the NOC
  • introduction of our customers to the outsourced NOC's: how to change their attitude
  • keeping the quality of the NOC's in check over time: means & experiences

What we would like to hear from other participants:

  • procedures: presentation of the NOC's current procedures & who is responsible for maintaining them
  • authentication: how do you verify the caller / e-mail sender
  • reponsibility of your NOC: how far does it go?
  • good & bad experiences with outsourcing
  • distributed NOC's (as opposed to a NOC on a central location): pro's & con's

DANTE
Other basic information

  • Organization (tiers, and which tiers are in-house or outsourced);
  • Responsibilities (e.g. monitoring, configuring/provisioning, direct or indirect repairs, DNS updates, RIPE updates, POP access and deliveries, reporting);
  • Hours of operation (office and on-call);
  • Network type (IP only, hybrid, switched/transmission only, LAN only);
  • Network services: IPv4/IPv6/multicast IPv4/multicast IPv6/VPNs/QoS/etc

Of most interest to learn from others

  • Main Monitoring systems and alarm thresholds;
  • Target repair times;
  • Number (approx) of ticket mail lists the NOC is on, and approx number of tickets received on these lists
  • Connected networks: Types of organization and approx numbers for subscribers, peers and upstream IP providers
  • Core equipment vendor/make, approx numbers;

CARNet
We would like to include in our presentation topics about:

  • CARNet network, users & Core/Access nodes - how to treat different users and different parts of network
  • CARNet NOC - organization, tiers, team operating hours, knowledge sharing
  • NMS - our tool functionalities / developement
  • prodecudes and documentations - what should be strictly put "on paper" (level of knowledge and procedures) from our experience ...

Topics about what we would like to learn from others are:

  • NOC key performance indexes - how to measure/to improve ?
  • Knowledge management and NOC staff training
  • NMS - tools (functionalities)
  • NOC organization and procedures: Tier1-N functions, # of persons in shifts and in total, work quality control, peak hours
  • How to work proactively in NOC (active/passive monitoring)

PIONIER
Five topics PSNC would like to present about NOC:

  • networks PIONIER (NREN) and POZMAN - technical structure (DWDM, switches, routers) and its influence to NOC structure, organizational structure: Layer 1,2 (NOC) and Layer 3 (IP NOC), employees and their functions
  • procedures used in our NOC
  • our TTS - public part, experience, missing functions
  • NOC tools - tools used in the NOC, missing functions, tools developed and maintained by PSNC
  • Experiences in joint management application design & deployment with commercial company

Five topics (or questions) that PSNC is interested in knowing how other NOCs handle:

  • how to minimize users calls (public available FAQ? tools? documentation?)
  • reports - what? how often?
  • data and tools integration - how? recommended tools?
  • serving users calls - how to automate it, distributing calls among different NOC administrators
  • (D-)DoS - how to detect network attacks

ARNES
ARNES could present:

  • Pros and contras for dedicated NOC personnel
  • NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)

ARNES would like to hear from others about:

  • NOC staff job description
  • NOC assesment and KPI – measuring NOC performance
  • NOC to user communication (communication with more or less experienced users, how to keep user contact info up-to-date, personalized web interface for NOC to user communication)
  • NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)
  • How many customers do you have? (depends on if you are counting persons, networks, hosts,..)
  • What kind of agreements do you have with service providers and customers (SLA,..)? Without financial info, but just roughly descriptions of responsibilities.

NORDUnet
Things NORDUnet could present:

  • Troubleticketing system, experiences from building our own.
  • Managing user interfaces (using people outside the NOC to aggregate user requests)
  • Using external community tools (letting the community answer questions instead of the NOC).
  • NOC organisation (staffing, internal escalations, handover and schedule)
  • Reaching your network from anywhere (out-of-band, in-band and VPN)

Things NORDUnet would like to hear about from others:

  • How is knowledge dissemination handled between groups and new NOC employees?
  • How is monitoring and alarm correlation handled between different systems?
  • How are new services and users provisioned?
  • What kind of routines are used in regards to change management (e.g. in planned maintenance)?
  • What other services than the network does your NOC operate and how do they differentiate from the network service?

GNOC at Indiana University
Five topics to introduce the Global Research NOC at Indiana University:

  • Global Research NOC Service Desk<http://globalnoc.iu.edu/tour.html> provides 24x7x365 technical call center support, trouble ticket management, and workflow support. The service desk is housed in a state-of-the-art call center in Indiana University's Informatics and Communications Technology Complex (ICTC) on Indiana University's Indianapolis campus. This call center features a fully customizable 30-foot screen used to monitor and troubleshoot the current health of the various networks supported. We provide NOC support for the most advanced research and education networks in the country. http://globalnoc.iu.edu/tour.html
  • Willingness to assist network customers with other services and requests outside of traditional front-line network operations, such as network redesign planning and utilizing our tool set.
  • A network knowledgeable tier-one Service Desk that can assess network issues intelligently and can provide engineering with helpful information as they begin troubleshooting.
  • Internally developed tools such our network database http://globalnoc.iu.edu/grnoc-tool-set.html
  • International networking efforts

Five topics the GlobalNOC at Indiana University would like to know about other NOC's:

  • What methodologies and tools do you use to understand complex cross-network relationships and dependencies?
  • How do you organize and update your internal documentation environment for easy access and search, as well as providing clear and accurate information?
  • How do you integrate the different groups within your NOC (Service Desk-Tier One, Engineering-Tier Two & Three, Systems Engineering, etc) so each has clear expectations of what each group does, with all working as a unified team?
  • With R&E networking moving towards having more regional connectedness between smaller networks, RON's, universities, etc, are you planning for a new phase in networking where specific operational differences between NOC's may negatively affect the reliability of the services provided? (for example: different response levels from hardware and fiber vendors, vastly different maintenance windows, NOC's not being 24x7, etc.)
  • What type of training methodologies do you use to bring new employees up-to-speed quickly so they can start making a valuable contribution to the operation?

SWITCH
The topics SWITCH could tell about:

  • network, equipment, services, customers
  • management of fibers - with segments from different providers
  • staff
  • outsourced services
  • work flows
  • tools
  • experiences and things we like to improve

The topics SWITCH is interested to know from other NOCs:
The same points like above with special focus on:

  • incident handling
  • tools and especially
  • tools to support the distributed handling of incidents

GRNET
GRNET is keen in presenting one of the topics trailed below:

  • Efficient administration of many Linux based servers,
  • Providing VPS service,
  • Network database (GRNnet-DBII),
  • Visualisation tools,

On the other hand, we would like to hear how our colleagues tackle the followings:

  • NOC key performance indexes,
  • Automated SLA monitoring tools,
  • Network monitoring tools (may be different than the ones above),
  • operation of CERT team and security policies

RedIRIS
Five topics to introduce RedIRIS NOC:

  • Type of network. Equipment and type of lines.
  • Out of band network.
  • Trouble ticket system.
  • SLAs with vendors and providers and type of contact.
  • Organisation of our NOC.

Five topics RedIRIS would like to know about other NOCs:

  • Organisation and roles of NOC members. Number of levels. Outsourced vs. staff-in NOC. Pros and cons. Functions. Experience with externalized NOC.
  • Monitoring tools.
  • Acquiring, management and operation of dark fibre. Procedures. Optical NOC differenciated from IP NOC?
  • Services managed by the NOC.
  • Network database and integration with other monitoring tools.

FCCN
Five topics to introduce FCCN's NOC:

  • Deployed network (Optical/Ethernet/IP)
  • Out-of-band systems (MRV using GSM/GPRS)
  • Service Desk & Automated reporting (per member)
  • Deployed probes network (QoSMetrics solution)
  • Past horror stories: What you can try to avoid & Murphy's Law. (smile)

Five topics FCCN would like to know about other NOCs:

  • How is the 24x7x365 NOC service guaranteed?
  • Which monitoring tools are you currently using. Which ones have you already used in the past and were abandoned. How do you keep network/topology maps updated?
  • Which services and incidents are covered by the NOC? How do you deal with requests/incidents that are outside your NOC's scope?
  • Do you have/use a fixed maintenance window?
  • How do you deal with performance issues/incidents and e2e issues regarding traffic that flows through your network.

CERN
CERN would be glad to present from the following list of topics:

  • Transforming 2 layer network support into 3 layer support with outsourcing
  • Integration of network and telecom support
  • Creation of a knowledge repository for network/telecom support
  • Structure and roles in a multi-layer NOC operating multi-type networks (campus network, international research network, internet exchange point)
  • The challenge of using multiple ticketing systems

CERN would be interested in hearing about:

  • Roles and responsibilities (structure, who does what)
  • Relation and communication between the network operations team and other internal entities (design, deployment)
  • How operation is done outside working hours including weekends
  • How common experience is handled: specialized vs. universal team members
  • How procedures are created and stored

RESTENA
Five topics to introduce RESTENA's NOC:

  • Infrastructure building and operations (dark fiber, net devices,server-rooms, POPs, servers)
  • An "all-in-one" NOC: structure/organization/tools
  • 24x7x365 service support
  • Coverage of user support (helpdesk)
  • Services operated (from lambdas to large scale e-mail accounts)

Five topics RESTENA would like to know about other NOCs:

  • User (customer) information exchange: channels and content
  • NOC activities/coverage and staff (network, services, community details)
  • Organization of inventory (tools, DB, naming conventions,etc.) and change management (technically and administratively)
  • Staff organization, i.e. services support strategy (per person/team/NOC) with their respective pros/cons
  • NOC awareness and identification: internally and among user community

SURFnet
Items we from SURFnet can present our experiences

  • acquring and managing dark fiber
  • managing an outsourced NOC
  • procedures - the good and bad things

Things we would like to hear about from other NOC's:

  • how to inform/communicate with customers
  • branding of an outsourced NOC
  • ticketing systems
  • CMDB experiences

CESCA
And these are topics that we would like to introduce about our NOC:

  • Virtualization of the network.
  • NOC tools and how to adapt them to our needs (development).
  • Users management. How to approach our users, depending on the action (change, incident, request,..).
  • Survey for our services, what to ask and how?
  • Out-of-band access to our network.

These are the topics that we would want to know from other NOCs:

  • Ticketing Systems, how to optimize their use.
  • Integration of diferent tools.
  • Partially outsourced NOCs (24x7, weekends,...): what kind of information do you give them, how do they interact with you and your users?
  • Escalation of incidents. How many levels?
  • Documentation: what can and can not be documented.

HEAnet
5 important bullet points introducing our NOC(s)

  • Heuristic Failure analysis (how we use heuristics to determine SLA calculations and where they are and are not appropriate.) (Also interested in discovering if others are doing this.)
  • Ticketing, including automatic ticket generation for incidents.
  • Contract and supplier management - how HEAnet does it and how well it works (or doesn't work.)
  • Reporting to your clients. How do NRENs report to clients? Periodical and Real-time reporting. On demand reporting.
  • 24 x 7 NOC cover. Do you provide it? What SLA? How do you hire staff and plan rotas, etc? Is there a real client demand for it?

5 questions what you want to know about other NOCs

  • How are staff hired for the NOC? Skillsets? Interview processes?
  • Standards compliance and process adoption. Have other NOCs adopted ISO9000? ITIL? PRINCE2? APM? ISO20000? Six Sigma? Other standards?
  • What scope of services does the NOC support? Networking? Servers? Value added services like DNS, web hosting, etc.? HPC?
  • How do NOCs source support and SLAs? What about penalty clauses? What's reasonable, what works and how are they enforced?
  • How do you assign value to what the NOC does? Do you cross charge for projects that will be handed over to the NOC? How to you prove to upper management that the NOC is good value for money?

Interesting topics mapped to the work items

Taxonomy of NOC organisations, internal processes, workflows

Topics participants are interested in sharing information about

Topics participants are interested in getting input on

Network

Network

Infrastructure building and operations
(dark fiber, net devices, server-rooms, POPs, servers)

Core equipment vendor/make, approx numbers

Deployed network (Optical/Ethernet/IP)

 

Network type (IP only, hybrid, switched/transmission only, LAN only)

 

Acquiring and managing dark fiber

Acquiring, management and operation of dark fibre.

Management of fibers - with segments from different providers

 

Network services: IPv4/IPv6/multicast IPv4/multicast IPv6/VPNs/QoS/etc

 

Network, users & Core/Access nodes
- how to treat different users and different parts of network

 

Reaching your network from anywhere

 

Out-of-band access to our network

 

Out-of-band systems (MRV using GSM/GPRS)

 

NOC functions

NOC functions

Survey for our services, what to ask and how?

Services managed by the NOC.

Services operated (from lambdas to large scale e-mail accounts)

What scope of services does the NOC support? Networking?
Servers? Value added services like DNS, web hosting, etc.? HPC?

Providing VPS service

 

Responsibilities (e.g. monitoring, configuring/provisioning,
direct or indirect repairs, DNS updates, RIPE updates,
POP access and deliveries, reporting);

Responsibility of your NOC: how far does it go?

 

What other services than the network does your NOC operate
and how do they differentiate from the network service?

 

Which services and incidents are covered by the NOC?
How do you deal with requests/incidents that are outside your NOC's scope?

 

Connected networks: Types of organization and approx numbers for
subscribers, peers and upstream IP providers- Roles and responsibilities
(structure, who does what)

 

Operation of CERT team and security policies

 

Number (approx) of ticket mail lists the NOC is on,
and approx number of tickets received on these lists

NOC structure

NOC structure

Staffing, internal escalations, handover and schedule

NOC activities/coverage and staff (network, services, community details)

24x7x365 service support

 

24 x 7 NOC cover. Do you provide it? What SLA?
How do you hire staff and plan rotas, etc?
Is there a real client demand for it?

How is the 24x7x365 NOC service guaranteed?

Hours of operation (office and on-call)

How operation is done outside working hours including weekends

Pros and contras for dedicated NOC personnel

How common experience is handled: specialized vs. universal team members

A network knowledgeable tier-one Service Desk that can
assess network issues intelligently and can provide engineering
with helpful information as they begin troubleshooting.

NOC organization and procedures: Tier1-N functions,
# of persons in shifts and in total, work quality control, peak hours

Structure and roles in a multi-layer NOC operating multi-type networks
(campus network, international research network, internet exchange point)

Staff organization, i.e. services support strategy
(per person/team/NOC) with their respective pros/cons

Global Research NOC Service Desk provides 24x7x365 technical call center
support, trouble ticket management, and workflow support. The service desk
is housed in a state-of-the-art call center in Indiana University's Informatics
and Communications Technology Complex (ICTC) on Indiana University's
Indianapolis campus. This call center features a fully customizable 30-foot
screen used to monitor and troubleshoot the current health of the various
networks supported.  We provide NOC support for the most advanced
research and education networks in the country.

With R&E networking moving towards having more regional connectedness
between smaller networks, RON's, universities, etc, are you planning for a
new phase in networking where specific operational differences between
NOC's may negatively affect the reliability of the services provided?
(for example: different response levels from hardware and fiber vendors,
vastly different maintenance windows, NOC's not being 24x7, etc.)

Technical structure (DWDM, switches, routers) and its influence to
NOC structure, organizational structure: Layer 1,2 (NOC) and
Layer 3 (IP NOC), employees and their functions

 

 

Distributed NOC's (as opposed to a NOC on a central location): pro's & con's

 

Optical NOC differentiated from IP NOC?

Transforming 2 layer network support into 3 layer support with outsourcing

Organisation and roles of NOC members. Number of levels.
Outsourced vs. staff-in NOC. Pros and cons. Functions.
Experience with externalized NOC.

Tiers, and which tiers are in-house or outsourced

Partially outsourced NOCs (24x7, weekends,...): what kind of information
do you give them, how do they interact with you and your users?

Managing an outsourced NOC

Branding of an outsourced NOC

Lessons learned from outsourced incident & change management

Good & bad experiences with outsourcing

Setting up and maintaining a NOC

Setting up and maintaining a NOC

Knowledge sharing

Knowledge management and NOC staff training

 

NOC staff job description

 

How are staff hired for the NOC? Skillsets? Interview processes?

NOC assessment

NOC assessment


NOC key performance indexes
- how to measure/to improve?

 

How do you assign value to what the NOC does?
 Do you cross charge for projects that will be handed over to the NOC?
 How to you prove to upper management that the NOC is good value for money?

NOC workflows

NOC workflows

Procedures used in the NOC - the good and bad things

Presentation of the NOC's current procedures &
 who is responsible for maintaining them

Workflows

 

 

How do you deal with performance issues/incidents and e2e issues
regarding traffic that flows through your network.

 

What kind of routines are used in regards to change management
(e.g. in planned maintenance)?

 

Do you have/use a fixed maintenance window?

 

How are new services and users provisioned?

 

How does your NOC take care of incident handling?

 

(D-)DoS - how to detect network attacks

 

What are your target repair times?

 

How to work proactively in NOC (active/passive monitoring)

 

How are new services and users provisioned?

NOC front-end development and improvement

NOC tools, interworking/interfacing issues, and automation

Topics participants are interested in sharing information about

Topics participants are interested in getting input on

Monitoring tools

Monitoring tools

CARNET: NMS - our tool functionalities / developement

DANTE: Main Monitoring systems and alarm thresholds

PIONIER: NOC tools - tools used in the NOC, missing functions, tools developed and maintained by PSNC

CARNET: NMS - tools (functionalities)

ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)

ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)

GNOC IU: Internally developed tools such our network database http://globalnoc.iu.edu/grnoc-tool-set.html

SWITCH: tools

SWITCH: tools

GRNET: Network monitoring tools (may be different than the ones above)

GRNET: Visualisation tools

RedIRIS: Monitoring tools.

CESCA: NOC tools and how to adapt them to our needs (development).

FCCN: Which monitoring tools are you currently using. Which ones have you already used in the past and were abandoned. How do you keep network/topology maps updated?

Multidomain tools: Distributed handling of incidents

Multidomain tools: Distributed handling of incidents

 

GNOC IU: What methodologies and tools do you use to understand complex cross-network relationships and dependencies?

 

SWITCH: tools to support the distributed handling of incidents

Ticketing systems

Ticketing systems

NORDUNET: Troubleticketing system, experiences from building our own.

SURFNET: ticketing systems

 

CESCA: Ticketing Systems, how to optimize their use.

Knowledge management: Configuration Management Database/Network Database

Knowledge management: Configuration Management Database/Network Database

GNOC IU: Internally developed tools such our network database http://globalnoc.iu.edu/grnoc-tool-set.html

RedIRIS: Network database and integration with other monitoring tools.

 

RESTENA: Organization of inventory (tools, DB, naming conventions,etc.) and change management (technically and administratively)

 

SURFNET: CMDB experiences

SLA reporting and monitoring tools

SLA reporting and monitoring tools

ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)

ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)

 

GRNET: Automated SLA monitoring tools

Chat/communication/coordination tools

Chat/communication/coordination tools

NORDUNET: Using external community tools (letting the community answer questions instead of the NOC).

Integration of tools

Integration of tools

BELNET: approach & tools: how the different departments & the NOC's work together

PIONIER: data and tools integration - how? recommended tools?

ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)

ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management)

RESTENA: An ""all-in-one"" NOC: structure/organization/tools

RedIRIS: Network database and integration with other monitoring tools.

CESCA: NOC tools and how to adapt them to our needs (development).

CESCA: Integration of diferent tools.

Efficient communication/collaboration tools and practices

Facilitate the collection/creation of best practice documents

Investigate and liaise with other communities

Flash presentation template

(warning) Work in progress

Purpose

The purpose of the flash presentations going to be held at the TF-NOC meeting in February 2011 is the following:
To get an overview of how other NOCs participating in TF-NOC work.
To give some food for thoughts to what is interesting to include in the upcoming TF-NOC survey.
To give some answers to questions that some NOCs are interested to hear how others are handling.

Is the above purpose "correct" or should it be extended/changed?

Topics

Slide 1 - Network

  • Infrastructure deployed
    (e.g. fiber + equipment)
  • Services offered and maintained
  • Tools used to manage/monitor the network

Slide 2 - NOC

  • Staff roles, coverage and responsibility
    (e.g. dispatchers or experts, 24/7 or daytime)
  • Structure of NOC
    (e.g. centralized, distributed or out-sourced)
  • How the NOC work with different kinds of tools
    (e.g. tool integration)

Slide 3 - Front end

  • Types of users using the network and services
  • SLAs or agreements with customers
  • Tools used to communicate with and keep track of users

Slide 4 - Inter NOC communication

  • Communication between NOC and other internal groups in your organisation
  • Communication with NOCs outside your organization
    (e.g. provider and/or customer NOCs)
  • Tools used to facilitate inter-NOC communication

Slide 5 - Documentation

  • What is documented
  • Tools used to create and update documentation
  • No labels