Blog from July, 2020

In article #001 and #002 you essentially created a NMaaS domain and enabled connectivity between your dedicated and isolated domain to your out of band management equipment network.

Requirement

  • Completed #001
  • Completed #002
  • Basic knowledge related to configuration management 

Overview

We are going to deploy our first NMaaS service for your organisation: Oxidized

Article objective

This is done in 2 steps:

  1. Oxidized application deployment via the NMaaS portal
  2. Oxidized configuration specific to RARE domain  

Diagram

NMaaS portal: Oxidized

[#003] - Cookbook

  • Having completed #001
  • Having completed #002
  • Once your domain is created and associated to your account, log into https://nmaas.eu as in #001
  • select Oxidized application

  • select "Deploy"

  • choose a name for your service instance, in our case we chose: "p4-oxi-srv"

The name has a particular importance as it will dynamically create a FQDN for the NMaaS service in the form: <service_name>.<domain>.nmaas.eu 

In my example it is: oxidized.rare.nmaas.eu

  • fill in the mandatory basic configuration information

  • Oxidized access username

we chose: oxidized

  • Oxidized access password

we chose: oxidized

  • Device access username (login used by Oxidized to access the equipment via SSH)

we chose: rare

  • Device access password (password used to access the equipment via SSH)

we chose: rare

  • Device (IP address)

we chose: 172.16.26.103,172.16.26.105,172.16.26.108,172.16.26.109 

VPN Connectivity Warning

 It is important to note that you'll be connected inside a dedicated VPN so you'll be isolated from the outside world as if you were running your own Out of band management network. So we can assume that your domain is secured.

  • Congratulation. You should have completed Oxidized deployment

In the RARE domain we had a specific requirement that requires a specific profiles for the RARE network equipment.

We are using then NMaaS configuration feature (also refer to NMaaS configuration process), which actually will provide us the way to alter Oxidized configuration software.

  • From the NMaaS portal service instance page select "Update configuration" button

  • you should be provided a git command that will let you clone your Oxidized NMaaS configuration repository

From a terminal, clone oxidized configuration repository
git clone ssh://git@gitlab.nmaas.eu/groups-rare/rare-oxidized-210.git
cd rare-oxidized-210

ls -l
total 0
drwxr-xr-x  4 loui  staff  128 Jul 30 11:10 base
drwxr-xr-x  4 loui  staff  128 Jul 30 11:13 model
Oxidized base configuration
cd base 
ls -l
total 16
-rw-r--r--  1 loui  staff  734 Jul 30 11:12 config
-rw-r--r--  1 loui  staff  141 Jul 30 11:12 router.db
Oxidized config file sample
---
username: rare
password: rare
model: rare
interval: 600
use_syslog: false
debug: false
threads: 30
timeout: 20
retries: 3
prompt: !ruby/regexp /([\w.@-]+[#>]\s?)$/
rest: 0.0.0.0:8888
vars: {}
groups:
   wedge-bf100-32x:
     vars:
       ssh_port: 2001
pid: "/storage/pid"
input:
  default: ssh
  debug: false
  ssh:
    secure: false
output:
  default: git
  file:
    directory: "/storage/configs"
  git:
    single_repo: true
    user: oxidized
    email: oxidized@man.poznan.pl
    repo: "/storage/oxidized.git"
source:
  default: csv
  csv:
    file: "/root/.config/oxidized/router.db"
    delimiter: !ruby/regexp /:/
    map:
      name: 0
      model: 1
      group: 2
model_map:
  rare: rare
  cisco: ios
  juniper: junos
Oxidized rare.rb file sample
class RARE < Oxidized::Model
  prompt /([\w.@()-]+[#>]\s?)$/
  #prompt /^([\w.@()-]+[#>]\s?)$/
  comment '! '
  cmd :all do |cfg|
    # cfg.gsub! /\cH+\s{8}/, '' # example how to handle pager cfg.gsub! /\cH+/, '' # example how to handle pager get rid of errors for commands that don't work on some devices
    cfg.gsub! /^% Invalid input detected at '\^' marker\.$|^\s+\^$/, ''
    cfg.cut_both
  end
  cmd :secret do |cfg|
    cfg.gsub! /^(snmp-server community).*/, '\\1 <configuration removed>'
    cfg.gsub! /^(snmp-server host \S+( vrf \S+)?( version (1|2c|3))?)\s+\S+((\s+\S*)*)\s*/, '\\1 <secret hidden> \\5'
    cfg.gsub! /^(username .+ (password|secret) \d) .+/, '\\1 <secret hidden>'
    cfg.gsub! /^(enable (password|secret)( level \d+)? \d) .+/, '\\1 <secret hidden>'
    cfg.gsub! /^(\s+(?:password|secret)) (?:\d )?\S+/, '\\1 <secret hidden>'
    cfg.gsub! /^(.*wpa-psk ascii \d) (\S+)/, '\\1 <secret hidden>'
    cfg.gsub! /^(.*key 7) (\d.+)/, '\\1 <secret hidden>'
    cfg.gsub! /^(tacacs-server (.+ )?key) .+/, '\\1 <secret hidden>'
    cfg.gsub! /^(crypto isakmp key) (\S+) (.*)/, '\\1 <secret hidden> \\3'
    cfg.gsub! /^(\s+ip ospf message-digest-key \d+ md5) .+/, '\\1 <secret hidden>'
    cfg.gsub! /^(\s+ip ospf authentication-key) .+/, '\\1 <secret hidden>'
    cfg.gsub! /^(\s+neighbor \S+ password) .+/, '\\1 <secret hidden>'
    cfg.gsub! /^(\s+vrrp \d+ authentication text) .+/, '\\1 <secret hidden>'
    cfg.gsub! /^(\s+standby \d+ authentication) .{1,8}$/, '\\1 <secret hidden>'
    cfg.gsub! /^(\s+standby \d+ authentication md5 key-string) .+?( timeout \d+)?$/, '\\1 <secret hidden> \\2'
    cfg.gsub! /^(\s+key-string) .+/, '\\1 <secret hidden>'
    cfg.gsub! /^((tacacs|radius) server [^\n]+\n(\s+[^\n]+\n)*\s+key) [^\n]+$/m, '\1 <secret hidden>'
    cfg
  end
  cmd 'show platform' do |cfg|
    comment "TEST: show platform"
    comments = []
    comments << cfg.lines.first
    lines = cfg.lines
    lines.each_with_index do |line, i|
      if line !~ /^mem:|^uptime:/
        comments << line.strip!
      end
    end
    comments << "\n"
    comment comments.join "\n"
  end
  cmd 'show interfaces description' do |cfg|
    comment cfg
  end
  cmd 'show running-config' do |cfg|
    cfg = cfg.each_line.to_a[3..-1]
    cfg = cfg.reject { |line| line.match /^ntp clock-period / }.join
    cfg.gsub! /^Current configuration : [^\n]*\n/, ''
    cfg.gsub! /^ tunnel mpls traffic-eng bandwidth[^\n]*\n*(
                  (?: [^\n]*\n*)*
                  tunnel mpls traffic-eng auto-bw)/mx, '\1'
    cfg
  end
  cfg :telnet do
    username /^Username:/i
    password /^Password:/i
  end
  cfg :telnet, :ssh do
    # preferred way to handle additional passwords
    post_login do
      if vars(:enable) == true
        cmd "enable"
      elsif vars(:enable)
        cmd "enable", /^[pP]assword:/
        cmd vars(:enable)
      end
    end
    post_login 'terminal length 0'
    post_login 'terminal width 0'
    pre_logout 'exit'
  end 
end

Oxidized router.db file sample

172.16.26.103:rare:wedge-bf100-32x
172.16.26.105:rare:wedge-bf100-32x
172.16.26.108:rare:wedge-bf100-32x
172.16.26.109:rare:wedge-bf100-32x

Oxidized model files
cd model 
ls -l
total 16
-rw-r--r--  1 loui  staff  2977 Jul 30 11:13 rare.rb
-rw-r--r--  1 loui  staff    69 Jul 30 11:10 readme.txt

Oxidized model configuration

Oxidized has the property to associate a model file specific to your equipment. In RARE context we needed to define a specific profile specifying the prompt used and also the command of interest during configuration versioning process.

Verification

Congratulations! You have deployed your first NMaaS service specifically for your domain !

Conclusion

In this article you:

  • You have deployed a powerful CMDB software for your organisation
  • You have learned how to apply specific configurations to it in order to match your requirements


[ #003 ] NMaaS-101 - key take-away

  • Deploying a NMaaS service is as easy as deploying an application on your mobile phone, you just have to log into the NMaaS portal and of course have the sufficient privileges to deploy application for your domain
  • Deploying an application is a 2 steps process
    • deploy the application via the portal
    • configure the application via git tool
  • Even if Oxidized deployment by NMaaS is made easy, it is mandatory to have a strong knowledge of the tool implemented. In this case, it is of course essential to read documentation from Oxidized GitHub.



In article #001 your organisation has now a NMaaS domain and you are the domain manager for this domain. 

Requirement

  • Use existing or deploy out of band management VLAN dedicated to your equipment
  • Use existing or deploy an OpenVPN client on a separate server or VM
  • Knowledge to implement OpenVPN tunnels

Overview

As it is now NMaaS is pretty useless for your organisation even if you deploy myriads of network management applications, and this is for 2 reasons:

  1. There is no particular connectivity enabling communication between NMaaS and the equipment to be supervised in your network.
  2. There is no particular connectivity enabling communication between your laptop and NMaaS network management application GUI.

Article objective

In this article, we will expose what is needed in order to enable communication between the NMaaS service and your equipment in your network and what is the process to configure your VPN client in order to use NMaaS services.

Let's take the example of the RARE domain, described in article #001, the objective is to:

  • Check that you have an out of band management network enabling reachability to all your equipment
  • Provide information required by the NMaaS team (list of users and also the internal out of band management subnet)
  • Use existing or deploy a new OpenVPN client that has network reachability to the network above
  • Establish a site to site OpenVPN tunnel towards the NMaaS OpenVPN server using the site to site OpenVPN profile (coming from the NMaaS team based on the information you provide)
  • Configure a client to site OpenVPN tunnel towards the NMaaS OpenVPN server using the client to site OpenVPN profile (coming from the NMaaS team based on the information your provide)

Diagram

RARE lab

The picture above depicts the four p4 switches connected by 10G circuit on top of GÉANT backbone. Each switch has:

  • One console port (aka BMC port) connected to an equipment it slef connected to DSL (ISDN or even RTC) broadband network management network
  • Ethernet management port connected to the P4 Lab out of band management network. 

[#002] - Cookbook

  • Network Administration knowledge

If this management network does not exist beforehand, you should be able to implement or have it implemented by your network operation team. 

  • Network Management network isolation

This management network should be only used to convey network management traffic (i.e. no user traffic or user interaction). This is also called a Data Communication Network (DCN).

In the RARE network example, this network is a multipoint to multipoint L2 VPLS implemented on top of GÉANT backbone by GEANT OC team. All the switches have their management Ethernet ports connected to this VPLS MPLS VPN.

In this particular case, the P4 Lab network span multiple countries, hence the VPLS implementation, it could have been a full mesh of L2VPN point to point tunnels or a L2 EVPN. Most of the case, in your organisation, the OOBM network is a simple 802.1q VLAN that spans your internal L2 network.

  • Equipment IPv4 subnet

In the example, all switches can be reached via their management ports inside 172.16.66.6.0/24 network in the VPLS VPN. This is required so that the NMaaS team can configure the proper routing within the NMaaS environment.

  • List of users

This includes the names and email addresses of the users that should be granted access. The NMaaS team uses this information to provision the VPN connection and to generate the necessary site-to-site and client-access VPN profiles.

  • Equipment IPv4 subnet

This information will be configured at NMaaS VPN server level in order to enable routing between NMaaS service and the network used to manage your equipment.

  • List of users

This information will be used to create your OpenVPN profile. One profile specific to user.

In the RARE network example, the VPN client is a PfSense firewall using the built-in OpenVPN plugin to establish the site-to-site VPN connection between the management subnet and the NMaaS network.

Once deployed you'll have to use the site to site OpenVPN profile provided by the NMaaS team in order to setup up the DCN VPN tunnel towards the NMaaS VPN server.

Once setup, you should have a full connectivity between the equipment and all the NMaaS services deployed in your domain.

Your namespace is implemented inside a namespace that is specific to your domain, and your domain only. All domains are isolated between each other via this concept. This ensures that only people from your organization have access to your resources along with the deployed applications in the NMaaS environment.

Once setup, you should have a full connectivity between your laptop and all the NMaaS services deployed in your domain.

Your namespace is implemented inside a namespace that is specific to your domain, and your domain only. All domains are isolated between each other via this concept. This ensures that only people from your organization have access to your resources along with the deployed applications in the NMaaS environment.

Verification


In order to test your site-to-site VPN connectivity you can execute the following steps:

  1. Try to access your private reverse proxy that will be responsible for providing web access to network management services deployed inside your NMaaS domain. You can first test the access to this proxy from your VPN concentrator. The IP address will be provided to you by the NMaaS team during the on-boarding process.
    1. Ensure that the correct routing table entries have been pushed to your concentrator during the VPN connection phase.
  2. Try to access the same reverse proxy but this time from one of your client devices that you expect to be managed by NMaaS. In order for this test to work, you will have to configure the required routes on your devices so that traffic destined for NMaaS goes through your VPN concentrator. If you use the same device acting as a VPN concentrator as your default gateway in your network, then you are all set; if not, routing entries will have to be manually added or pushed to your client devices. Depending on the software being used on the VPN concentrator, the methods for configuring it as a router so that it will accept transit traffic will vary. The most common scenario, using a simple Linux VM would require enabling the ip forwarding option on your system and setting the necessary iptables FORWARDING rules.


Note on GUI-less devices

Since it is expected that most of your devices that you would like to manage are only providing console access, reachability of the reverse proxy can be tested with various tools, such as curl https://<PROVIDED_IP_ADDRESS> or even establishing a TCP session to port 443 using the built-in telnet client: telnet <PROVIDED_IP_ADDRESS> 443. 

The telnet approach is expected to be more widely used since the majority of the devices in use today have a telnet client available.

The same steps as above can be used to verify that you have access to your dedicated NMaaS domain while connected from your workstation using the client-to-site VPN. The reverse proxy IP address remains the same, and you can open your browser and navigate to the provided IP address where you should be greeted with a 404 HTTP page.

Congratulations! From this point on:

  • You should have enabled full connectivity between your equipment and NMaaS service
  • You should have access to NMaaS service user interface via an interactive client to site OpenVPN access.

VPN Connectivity Warning

While any software application that supports the OpenVPN protocol can be used to establish both the client-access and the site-to-site VPN connections, it is important that OpenVPN 2.4+ is supported. Furthermore, if you use a version of Linux on your workstation and try to connect using your built-in network manager, instead of the OpenVPN command line utility, then the connection might fail silently due to issues negotiating the supported compression schemes. This is a known bug affecting recent versions of the most popular Linux distributions and in such cases the OpenVPN command line client can be used.

  • apt install openvpn
  • openvpn --config <OVPN_PROFILE>

Conclusion

After performing all of the above steps you should be ready to deploy your first NMaaS application and start managing your network! We will see in the next article how to deploy our first NMaaS service and consider oxidized CMDB software.

In this article you:

  • Had a brief explanation regarding the mandatory connectivity required by NMaaS
  • One is a permanent connectivity between the OOBM network and NMaaS services in which only network management information is conveyed, also called a Data Communication Network (DCN).
  • The second one is an on demand connectivity enabled by an interactive VPN access.

[ #002 ] NMaaS-101 - key take-away

  • An existing OOBM network has to be implemented
  • PfSense/OpenVPN is used as OpenVPN server, you can also deploy a OpenVPN client
  • This manual step is a bit tedious but the good news is that it has to be configured only once. Once OpenVPN accesses are setup, they are valid for all NMaaS services available in the catalog.



The main objective [ NMaaS 101 ] series is to help you getting started with NMaaS service without any prior knowledge.

Requirement

  • Having an eduGain account
  • Identified as NMaaS Domain Manager on behalf of your organisation
  • If you are not an NREN, have your local NREN endorsement for using NMaaS 

Overview

Network Management as a Service (NMaaS) is a GÉANT service proposed to R&E institutions. It proposes the opportunity for any R&E organisation to subscribe and deploy Network Management tools in a cloud environment via an intuitive web portal. Most of GÉANT members have their own network management software suite. Some encompasses off the shelf tools, but others are using Open Source tools as per the last SIG-NOC tools report. But all the ~40 GÉANT community members are not equal in that perspective. Some have a fully integrated sophisticated network management system, usually sophistication implies also "home made" and some small NRENs have difficulties to maintain existing systems. 

NMaaS target customer audience is:

  • Small and Emerging NRENs

Smaller NRENs may have limited resources to develop their own NMS

By using a shared and supported platform, NRENs can focus on the monitoring and management of their service components

  • Campuses

NMaaS platform is ideally suited for Campus Network Management

  • Small Organisations

NMaaS supports the needs of institutional users, either on an NREN managed NMaaS platform or the centralised GÉANT platform.

  • Distributed research projects

It can be a global research project like LSST, SKA, JIVE etc. In our case I'll expose you the particular case of RARE, which is a GN4-3 research project.

RARE project brief description

RARE (Router for Academia, Research & Education) is an ongoing effort under the GÉANT 3rd programme which focus on determining if a routing software platform solution can fit R&E use cases. The project aims to integrate different pieces of software related to these building blocks:

  • control plane: RARE uses FreeRouter under the hood used as the control plane component
  • data plane: P4 is used to describe the behavioral model of RARE data plane
  • and communication interface between the control plane and data plane: Interface compliant to P4Runtime specification ensure this function

In order to validate the code produced by the RARE team we deployed a P4 Lab distributed aming various European countries:

  • NMaaS is used as the platform to supervise all the P4 switches deployed in this Testbed.
  • We will see in subsequent articles how NMaaS platform proposition value is helping the RARE team to deploy and use Network Management Application.
  • NMaaS flexibility concept, enabled us to add supervision tools following KISS approach.
  • The NMaaS platform is step by step is becoming a turn key solution for P4 switch network monitoring and supervision. 


Article objective

This article is meant to guide you and provide you the very first mandatory steps in order to create a NMaaS domain for your organization.

Let's take the example of the RARE domain, the objective is to:

  • Explain how to trigger a domain creation
  • How to access it

In the end you should have access the NMaaS portal specifically dedicated to your organization. i.e. your organization NMaaS domain.

Diagram

[ #001 ] - Cookbook

  • eduGain R&E IDP

Your organization should be part of the eduGain R&E federated Identity provider.

  • Designated as NMaaS domain manager internally by your organisation

Usually, this is CIO role, but at least you should have been granted the privileges to deploy applications on behalf of your institution.

  • Via your favorite browser, go to nmaas.eu, you should be granted by a welcome page:

  • Click on "Login / Register" button and then "Federated login"

  • You should be now familiar with eduGain authentication system

  • You should be now familiar with national eduGain authentication system

  • During first login you are asked to submit additional account information and login again

  • At this point you should now have access to NMaaS portal but with no domain

  • At that precise point your connection attempt is logged by the NMaaS team and your email contact associated to your eduGain account
  • The final step is to send a mail to nmaas@lists.geant.org

This mail should briefly present your organisation, eventually your project, mention the domain name (like RARE in my case) and have the endorsement of your local NREN.

With these information the NMaaS team should be able to:

  • create the NMaaS domain you specified in your request
  • associate your eduGain account as Domain manager for your organisation

Verification

Congratulations ! Your organisation has now a NMaaS domain and you are Domain manager for your organisation ! 

Conclusion

In this article you:

  • Had a brief introduction to the mandatory steps to complete in order to obtain a NMaaS domain
  • NMaaS is for organisation that would like to outsource their network management maintenance activity
  • As NRENs, we have the possibility to publish application via NMaaS portal
  • NMaaS presents a community aspect where sharing knowledge and experience is at the heart of the concept

[ #001 ] NMaaS-101 - key take-away

  • NMaaS is a way to subscribe and deploy easily Network Management application (for now) in the cloud
  • NMaaS customer targets are small, medium R&E institutions that would like to outsource their network management
  • Your organisation must have an eduGain Identity Provider