Blog

Starting from SOHO #007, everyone inside home network can:

  • reach the Internet thanks to NAT translation described in SOHO #004 
  • reach now their favorite Internet service by name thanks for SOHO #005
  • and get IPv4 (or IPv6) address from DHCP server (we took as a switch connected to sdn6)
  • this applies to all wired host but also all mobile host connected to the home wireless network

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

I'm very fond listening music at home and started my personal music collection. I was aware of this practice, and since then started 24/96 flac collection. As mechanical hard drives became more and more affordable, storage was not an issue for the NAS I deployed. I purchased then a refurbished but still decent DLNA network player (DMP) that would be able to stream music sources from DLNA server(DMS). All of this is orchestrated though my mobile phone acting as DLNA media controller (DMC). So far so good, but DLNA alliance expects that all DLNA protagonists are in the same and unique broadcast domain.

You might have guess already, but I'm not a huge FAN of having a single layer 2 domain at home and throw everything inside the same subnetwork. Granted the fact that not all home network are running OSPF/ ISIS MPLS or Segment Routing, everything is so simple and more predictable to manage at layer 3 whether you use IPv4 or IPv6.

Long story short, I can't listen music by structuring my home network using layer 3 as it is now ... (sad)

Article objective

In this article we will pursue the SOHO network appliance installation and enable THE UNIQUE freeRouter feature that you won't find in any router in the world: UPNP forwarder server.

Diagrams

source: DLNA German wikipedia 

[ SOHO #008 ] - "It's the end of L2 at my home... At last !"

 Step -1- Configure UPNP forwarder server

The first step is to configure an UPNP hub server

Identify WIFI module hardware
server upnphub upnp
 interface loopback0
 vrf inet
 exit
  • As any other server it is bound to loopback0
  • And operate inside VRF inet
 Step -2- Configure UPNP forwarder client

Now that you have an UPNP hub, we need to configure any upnp forwarder client where we expect to see any incoming UPNP/DLNA message

UPNP forwarder client @ integrated WIFI
server upnpfwd wifi4-appliance
 target 192.168.254.1
 interface hairpin12
 vrf inet
 exit

In this case, WIFI network we encompasses all mobile clients. My phone is acting as DMC here, but it could also be a DMS. The upnp forwarder:

  • has a target UPNP hub (we configured 102.168.254.1)
  • is bound to interface sdn2
  • operates within VRF inet
UPNP forwarder client @ integrated WIFI
server upnpfwd wired-media
 target 192.168.254.1
 interface sdn2
 vrf inet
 exit

In this precise case wired-media is another UPNP fowarder client that will forward all UPNP message to UPNP hub 192.168.254.1. As you might have guessed behind sdn2 is cascaded my NAS and DLNA network streamer (DMP or DMR)

Tip

In this example the key idea is:

  • Create a UPNP hub
  • Create a UPNP forwarder bound to any interface where you expect DLNA/UPNP communication
  • In this case DMS and DMR are behind sdn2
  • DMC is behind WIFI

Verification

 Connect mobile phone to WIFI

 Launch DLNA DMC App

In the screenshot below, the DLNA player is shutdown

 Power on DLNA network player

Check that the App is able to discover the DLNA server DMS in other L3 subnet

 Enjoy your music

Conclusion

In this article we enabled RARE/freeRouter DLNA/UPNP server/client:

  • You have now the possibility to position any DLNA devices on any L3 subnet of your home network
  • Bye bye flat single VLAN design
  • Not having a flat VLAN is desirable when you have lots of IPv4/IPv6 devices of different nature. You would want domotic/IoT devices not in the same subnet as your wifi client and have a separate subnet for you NAS or wired services


RARE validated design: [ SOHO #006 ] - key take-away

In this example the key take-away are:

  • Configure an UPNP hub
  • Configure UPNP forwarder on very interface you expect DLNA/UPNP communication
  • Of course this is IPv4/IPv6 compliant

This is a pretty unique freeRouter feature !


So everyone inside home network can:

  • reach the Internet thanks to NAT translation described in SOHO #004 
  • reach now their favorite Internet service by name thanks for SOHO #005
  • and get IPv4 (or IPv6) address from DHCP server (we took as a switch connected to sdn6)

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

In the previous articles we describe the hardware in SOHO #001 and initial staging steps in SOHO #002.

Note

You technically juts have to cut'n paste the config if you get the same hardware and operating system.

But if you pay attention, we did not cover one particular aspect of the hardware: integrated WIFI

Article objective

In this article we will pursue the SOHO network appliance installation and enable the integrated module so that all mobile host @ home can benefit from connectivity offered by SOHO router. In this basic example I'll show you basic WIFI implementation

  • WIFI is b/g/n module (therefore no 5Ghz wifi)
  • WIFI won't be directly under RARE/freeRouter control

Diagrams

[ SOHO #007 ] - "I'm not wired ... I need connectivity too !"

 Identify integrated WIFI module hardware

Log into linux appliance via management interface using previously configured veth1a IPv4 address: 192.168.128.1

Identify WIFI module hardware
lspci | grep -i wire
09:00.0 Network controller: Qualcomm Atheros AR928X Wireless Network Adapter (PCI-Express) (rev 01)
 Identify integrated WIFI module linux driver

My google-fu indicated me that the lunux driver should be ath9k

Check integrated WIFI hardware driver is loaded
lsmod | grep ath
ath9k                 139264  0
ath9k_common           20480  1 ath9k
ath9k_hw              487424  2 ath9k_common,ath9k
ath                    36864  3 ath9k_common,ath9k,ath9k_hw
mac80211              917504  1 ath9k
cfg80211              897024  4 ath9k_common,ath9k,ath,mac80211

So it seems that Debian kernel has detected and loaded the proper module

 Check WIFI interface availability
Linux Access point software installation
ifconfig | grep w
wlan0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500

wlan0 interface appeared ! 

Warning

  • When freshly configured, wlan0 is disabled. You can enable it:
Linux Access point software installation
ip link set wlan0 up
  • However, note that in SOHO #002, all is set during startup script (/rtr/hwdet-all.sh) via systemd
  • Last observation is that in SOHO #002 we also disabled Linux systemd networking
    • systemd use to name linux interface based on their pcie id: wlp9s0
    • after disabling it, WIFI interface name appears to become: wlan0
 Install Access Point software

Now that the hardware and corresponding linux driver is loaded we can proceed to Linux access point software installation

Linux Access point software installation
apt-get update
apt-get install hostapd
 hostapd configuration

hostapd configuration is Debian is in /etc/hostapd/hostapd.conf. But remember this is no more under systemd startup control as we disabled entirely systemd networking.

hostapd.conf
cat /etc/hostapd/hostapd.conf
#change wlan0 to your wireless device
interface=wlan0
# "g" simply means 2.4GHz band
hw_mode=g
# the channel to use
channel=acs_survey
# limit the frequencies used to those allowed in the country
ieee80211d=1
# the country code
country_code=FR
# 802.11n support
ieee80211n=1
# QoS support, also required for full speed on 802.11n/ac/ax
wmm_enabled=1
driver=nl80211
ssid=YOUR_HOME_WIFI_SSID
macaddr_acl=0
auth_algs=1
ignore_broadcast_ssid=0
wpa=3
wpa_passphrase=y0urverys1cr1tpassw0rd
wpa_key_mgmt=WPA-PSK FT-PSK WPA-PSK-SHA256 SAE FT-SAE
wpa_pairwise=CCMP
rsn_pairwise=CCMP

You can check that hostapd is correctly configured by launching it manually

hostapd manual launch
hostapd /etc/hostapd/hostapd.conf 
...

From that point from your mobile phone or laptop:

  • you should be able to see YOUR_HOME_WIFI_SSID appearing in the list of available wireless network
  • you should be able to connect to YOUR_HOME_WIFI_SSID
  • But your mobile device might say: "Internet might not be available or Cannot retrieve IP from YOUR_HOME_WIFI_SSID" 
 Assumptions
  • hostapd id triggered when SOHO router is booting up (described in rtr-hw.txt)  
  • Remember wlan0 is visible from the Linux kernel and managed by hostapd
  • we created a veth pair (veth2a@linux side and veth2b@sdn998 SOHO router side)
  • we bind wlan0 to veth2a (Cf. rtr-hw.txt → proc wlan /rtr/pcap2pcap.bin wlan0 veth2a)
  • and then veth2a is bound to sdn998  which is interface veth2b controlled by DPDK (Cf. SOHO #002 and SOHO #003)

So at that point all traffic coming from wlan0 will also make its way to sdn998

 bridge wireless traffic from wlan0 --> veth2a --> veth2b (into freeRouter sdn998 )

Now we need to make wireless traffic visible through VRF inet as we need to enable IPv4 (or IPv6) connectivity for mobile device.

So this is done in 2 steps.

Step - 1 - Create a bridge for wireless

Wireless bridge
bridge 1
 mac-learn
 mac-move
 exit

Note

Creating a bridge will also create interface bvi.

Step - 2 - Add veth2b to the wireless bridge

sdn998 addition into bridge 1
interface sdn998
 description SOHO@WLAN[veth2b-veth2a]
 mtu 1500
 bridge-group 1
 no shutdown
 no log-link-change
 exit

Warning

Make sure that bridge 1, sdn998, hairpin11 and hairpin12 are in declared into p4lang server

Linux Access point software installation
conf t
server p4lang p4 
 export-bridge 1
 export-port sdn998 7 1 0 0 0
 export-port hairpin11 11 0 0 0 0
 export-port hairpin12 12 0 0 0 0
 Inject bridge 1 traffic into VRF inet

The trick is to use hairpin interfaces. For experienced Junos user, this corresponds to Junos logical tunnel lt interface 

Step - 1 - Create a hairpin pair in order to redirect wireless traffic into VRF inet

Wireless bridge
conf t
hairpin 1

Note

Creating a hairpin 1 will also create interface hairpin11 and hairpin12.

Step - 2 - Add hairpin11 to the wireless bridge

hairpin11 addition into bridge 1
interface hairpin11
 no description
 bridge-group 1
 no shutdown
 no log-link-change
 exit

Step - 3 - Add hairpin12 into VRF inet

hairpin11 addition into bridge 1
interface hairpin12
 description SOHO@hairpin11
 mtu 1500
 vrf forwarding inet
 ipv4 address 192.168.129.1 255.255.255.0
 no shutdown
 no log-link-change
 exit

Warning

At that point all traffic from/to wireless will transit via hairpin12 and reach VRF inet

 Configure DHCP IPv4 server for Wireless

Now, DHCP request coming from mobile client can reach SOHO router VRF inet via hairpin12.

Step - 1 - Create DHCP server for Wireless client in VRF inet

DHCP configuration for Wireless client
server dhcp4 dh4-wlan
 pool 192.168.129.2 192.168.129.254
 gateway 192.168.129.1
 netmask 255.255.255.0
 dns-server 192.168.254.1
 domain-name local
 interface hairpin12
 vrf inet
 exit        

Note

Creating hairpin 1 will also create interface hairpin11 and hairpin12.

Verification

 DHCP verification

Connect to WIFI via laptop wifi

DHCP debug command
debug server dhcp?                                                     
  dhcp4      - ipv4 dynamic host config protocol
  dhcp6      - ipv6 dynamic host config protocol

mjolnir#debug server dhcp4 ?                                                   
  <cr>
mjolnir#debug server dhcp4                                                     
mjolnir#terminal monitor    
DHCP debug ouput when connecting laptop @ sdn6
mjolnir#debug server dhcp4                                                     
mjolnir#term mon                                                               
mjolnir#info ip.ipCor6.parseIPheader:ipCor6.java:95 got bad version from ::    
debug serv.servDhcp4worker.doer:servDhcp4.java:679 rx op=req sec=1 cia=0.0.0.0 yia=0.0.0.0 sia=0.0.0.0 gia=0.0.0.0 cha=88e9.fe76.7f9b srv= fil= op=discover dhcpsrv=null hstnm=MBP-de-Frederic dom=null lease=7776000 renew=0 mask=null gw=null dns1=null dns2=null req=null
debug serv.servDhcp4.sendPack:servDhcp4.java:482 tx 192.168.129.83 op=rep sec=1 cia=0.0.0.0 yia=192.168.129.83 sia=192.168.129.1 gia=0.0.0.0 cha=88e9.fe76.7f9b srv= fil= op=offer dhcpsrv=192.168.129.1 hstnm=null dom=lan lease=43200 renew=21600 mask=255.255.255.0 gw=192.168.129.1 dns1=192.168.254.1 dns2=null req=null
debug serv.servDhcp4worker.doer:servDhcp4.java:686 tx op=rep sec=1 cia=0.0.0.0 yia=192.168.129.83 sia=192.168.129.1 gia=0.0.0.0 cha=88e9.fe76.7f9b srv= fil= op=offer dhcpsrv=192.168.129.1 hstnm=null dom=lan lease=43200 renew=21600 mask=255.255.255.0 gw=192.168.129.1 dns1=192.168.254.1 dns2=null req=null
debug serv.servDhcp4worker.doer:servDhcp4.java:679 rx op=req sec=2 cia=0.0.0.0 yia=0.0.0.0 sia=0.0.0.0 gia=0.0.0.0 cha=88e9.fe76.7f9b srv= fil= op=request dhcpsrv=192.168.129.1 hstnm=MBP-de-Frederic dom=null lease=0 renew=0 mask=null gw=null dns1=null dns2=null req=192.168.129.83
debug serv.servDhcp4.sendPack:servDhcp4.java:482 tx 192.168.129.83 op=rep sec=2 cia=0.0.0.0 yia=192.168.129.83 sia=192.168.129.1 gia=0.0.0.0 cha=88e9.fe76.7f9b srv= fil= op=ack dhcpsrv=192.168.129.1 hstnm=null dom=lan lease=43200 renew=21600 mask=255.255.255.0 gw=192.168.129.1 dns1=192.168.254.1 dns2=null req=null
debug serv.servDhcp4worker.doer:servDhcp4.java:686 tx op=rep sec=2 cia=0.0.0.0 yia=192.168.129.83 sia=192.168.129.1 gia=0.0.0.0 cha=88e9.fe76.7f9b srv= fil= op=ack dhcpsrv=192.168.129.1 hstnm=null dom=lan lease=43200 renew=21600 mask=255.255.255.0 gw=192.168.129.1 dns1=192.168.254.1 dns2=null req=null
...
mjolnir#   

So based on the debug output:

  • DHCP allocated 192.168.129.83
  • Primary DNS (dns1) is 192.168.254.1
  • Network has /24 CIDR


ARP entries @ hairpin12
mjolnir#sh ipv4 arp hairpin12                                                  
mac             address          time      static
...
88e9.fe76.7f9b  192.168.129.83   00:00:13  false
...
mjolnir#   
 DHCP check @ wifi client laptop
DHCP debug command
╭─[11/3/20|3:17:21]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  ifconfig en0    
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=400<CHANNEL_IO>
        ether 88:e9:fe:76:7f:9b 
        inet6 fe80::1cf1:eacf:9ba9:43c3%en0 prefixlen 64 secured scopeid 0x5 
        inet 192.168.129.83 netmask 0xffffff00 broadcast 192.168.129.255
        inet6 2a01:e0a:159:2857:86:8cf9:a786:8f18 prefixlen 64 autoconf secured 
        inet6 2a01:e0a:159:2857:904b:4faa:5684:b7a0 prefixlen 64 autoconf temporary 
        nd6 options=201<PERFORMNUD,DAD>
        media: autoselect
        status: active
╭─[11/3/20|3:24:08]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  nslookup 212.27.48.10
Server:         fd00:2200::1
Address:        fd00:2200::1#53

Non-authoritative answer:
10.48.27.212.in-addr.arpa       name = www.free.fr.

Authoritative answers can be found from:

Pay attention to the server that answered ! At the present moment it should not be fd00:2200::1. It is simply because I had a setup with IPv6 fully enabled.

 IPv4 connectivity verification from laptop
DNS resolution from CLI (DNS query originated by router)
╭─[11/3/20|3:14:17]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  ping 212.27.48.10
PING www.free.fr (212.27.48.10): 56 data bytes
64 bytes from 212.27.48.10: icmp_seq=0 ttl=57 time=6.528 ms
64 bytes from 212.27.48.10: icmp_seq=1 ttl=57 time=4.666 ms
64 bytes from 212.27.48.10: icmp_seq=2 ttl=57 time=4.330 ms
64 bytes from 212.27.48.10: icmp_seq=3 ttl=57 time=4.446 ms
Check that NAT is occuring for 192.168.129.83
show ipv4 nat inet translations | i 192.168.129.83                     
1      192.168.129.83 -211222528  212.27.48.10 -211222528  192.168.0.90 -211222528  212.27.48.10 -211222528    00:00:17  00:00:17  00:05:00  1        84
1      212.27.48.10 -211222528    192.168.0.90 -211222528  212.27.48.10 -211222528  192.168.129.83 -211222528  00:00:17  00:00:17  00:05:00  1        84
1      192.168.129.83 -211222527  212.27.48.10 -211222527  192.168.0.90 -211222527  212.27.48.10 -211222527    00:00:16  00:00:16  00:05:00  1        84
1      212.27.48.10 -211222527    192.168.0.90 -211222527  212.27.48.10 -211222527  192.168.129.83 -211222527  00:00:16  00:00:16  00:05:00  1        84
1      192.168.129.83 -211222526  212.27.48.10 -211222526  192.168.0.90 -211222526  212.27.48.10 -211222526    00:00:15  00:00:15  00:05:00  1        84
1      212.27.48.10 -211222526    192.168.0.90 -211222526  212.27.48.10 -211222526  192.168.129.83 -211222526  00:00:15  00:00:15  00:05:00  1        84
 Internet connectivity Check
DNS resolution from CLI (DNS query originated by router)
╭─[11/2/20|4:36:18]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  ping www.free.fr -c 5 
PING www.free.fr (212.27.48.10): 56 data bytes
64 bytes from 212.27.48.10: icmp_seq=0 ttl=57 time=3.903 ms
64 bytes from 212.27.48.10: icmp_seq=1 ttl=57 time=5.883 ms
64 bytes from 212.27.48.10: icmp_seq=2 ttl=57 time=3.658 ms
64 bytes from 212.27.48.10: icmp_seq=3 ttl=57 time=4.872 ms
64 bytes from 212.27.48.10: icmp_seq=4 ttl=57 time=4.488 ms

--- www.free.fr ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.658/4.561/5.883/0.787 ms
Check that NAT is occuring for 192.168.136.126 and www.free.fr IPv4 address
show ipv4 nat inet translations | i 192.168.129.83                     
1      192.168.129.83 -211222528  212.27.48.10 -211222528  192.168.0.90 -211222528  212.27.48.10 -211222528    00:00:17  00:00:17  00:05:00  1        84
1      212.27.48.10 -211222528    192.168.0.90 -211222528  212.27.48.10 -211222528  192.168.129.83 -211222528  00:00:17  00:00:17  00:05:00  1        84
1      192.168.129.83 -211222527  212.27.48.10 -211222527  192.168.0.90 -211222527  212.27.48.10 -211222527    00:00:16  00:00:16  00:05:00  1        84
1      212.27.48.10 -211222527    192.168.0.90 -211222527  212.27.48.10 -211222527  192.168.129.83 -211222527  00:00:16  00:00:16  00:05:00  1        84
1      192.168.129.83 -211222526  212.27.48.10 -211222526  192.168.0.90 -211222526  212.27.48.10 -211222526    00:00:15  00:00:15  00:05:00  1        84
1      212.27.48.10 -211222526    192.168.0.90 -211222526  212.27.48.10 -211222526  192.168.129.83 -211222526  00:00:15  00:00:15  00:05:00  1        84

Conclusion

In this article we enabled the appliance WIFI integrated hardware:

  • We ensured that wifi hardware was detected by Linux kernel
  • We also ensured that Linux loaded the right wifi driver kernel module
  • hostapd would control wireless interface
  • we create a veth pair  (veth2a Linux side - veth2b / DPDK side)
  • wlan0 traffic is bound to veth2a using pcap2pcap utility (in freeRouter binaries bundle)
  • veth2b is bound to sdn998 (and declalred in p4lang server)
  • we create a hairpin 1 (hairpin11 , hairpin12) interface
  • bridge 1 has also been created
  • sdn998 and hairpin11 have been added to the bridge
  • hairpin12 is a routed interface declared in VRF inet and has an IP 192.168.129.1 inside wireless subnet 192.168.129.0/24


RARE validated design: [ SOHO #006 ] - key take-away

In this example the key take-away are:

  • The above action lead to bridge wireless traffic and pour it into VRF inet
  • From that point all defined previously will apply (NAT, DHCP) but now with wireless subnetwork

This example cover the case of a basic bridge in which we included a basic interface sdn998 and a hairpin interfaces pouring traffic from/to VRF inet. freeRouter is acting as a wireless controller local to the integrated WIFI. We will see in further article another typical WIFI implementation more flexible. Pleas note that we could have directly applied routing at sdn998 (so without the use of the bridge and hairpin). The usage of bridge and hairpin setup will be described in a next article describing alternate wifi implementation. In short we will add an OpenWRT Access point behind sdn6 and SOHO router will act as WIFI controller for both the integrated WIFI hotspot and the newly added OpenWRT.

So everyone inside home network can reach the Internet thanks to NAT translation described in SOHO #004 and reach now their favorite Internet service by name thanks for SOHO #005.

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

In the previous article during the verification we assume to have one host connected to a subnetwork cascaded behind SOHO router@sdn6. However, we did not described how the host could get connectivity.

Article objective

In this article we will pursue the SOHO network appliance installation and enable DHCP for IPv4 server. SOHO router can then answer to DHCPv4 request ingressing sdn6 interface.

  • Let's assume we have a switch connected to SOHO router@sdn6 for the future
  • sdn6 has ipv4 interface 192.168.136.1 manually configured
  • switch has IPv4 192.168.136.2
  • subnetwork behind sdn6 is 192.168.136.0/24
  • DHCPv4 will serve 192.168.136.3 → 192.168.136.254

Diagrams

[ SOHO #006 ] - "Do you need an IP ?"

 Configure interface sdn6, gateway for 192.168.136.0/24 subnet

First step, configure sdn6 and bind it to VRF inet. few considerations:

  • sdn6 has ipv4 192.168.136.1 manually configured
  • sdn6 will act as gateway for all connected host in subnetwork 192.168.136.0/24
  • In SOHO #004 ,  192.168.136.0/24 NAT is already taken into account


SOHO router DHCP server in VRF inet
conf t
interface sdn6
 mtu 1500
 vrf forwarding inet
 ipv4 address 192.168.136.1 255.255.255.0
 no shutdown
 no log-link-change
 exit
!
 Configure DHCPv4 server

Second step, configure a DHCPv4 server@ SOHO router

SOHO router DHCP server in VRF inet
conf t
server dhcp4 dh4-16
 pool 192.168.136.3 192.168.136.254
 gateway 192.168.136.1
 netmask 255.255.255.0
 dns-server 192.168.254.1
 domain-name local
 interface sdn6
 vrf inet
 exit
!

In this case DHCPv4 will allocate IPv4 address:

  • from a pool going from 192.168.136.3 → 192.168.136.254
  • All host will set their gateway to 192.168.136.1 (which is SOHO@sdn6)
  • All connected host to sdn6 subnet will get an IP from pool within 192.168.136.0/24 subnet
  • All hosts will also consider SOHO@192.168.254.1 as DNS server as we implemented in SOHO #005
  • And we bind this DHCPv4 to interface sdn6 (without binding DHCP request coming from all interface belonging to VRF inet will be processed)
  • lastly DHCPv4 server will act into VRF inet only 

Verification

 DHCP verification

Connect a host to the switch connected to sdn6 then issue DHCP debug command

DHCP debug command
debug server dhcp?                                                     
  dhcp4      - ipv4 dynamic host config protocol
  dhcp6      - ipv6 dynamic host config protocol

mjolnir#debug server dhcp4 ?                                                   
  <cr>
mjolnir#debug server dhcp4                                                     
mjolnir#terminal monitor    
DHCP debug ouput when connecting laptop @ sdn6
mjolnir#..                                                          debug serv.servDhcp4worker.doer:servDhcp4.java:679 rx op=req sec=1 cia=0.0.0.0 yia=0.0.0.0 sia=0.0.0.0 gia=0.0.0.0 cha=9ceb.e8d5.2c51 srv= fil= op=discover dhcpsrv=null hstnm=MBP-de-Frederic dom=null lease=7776000 renew=0 mask=null gw=null dns1=null dns2=null req=null
debug serv.servDhcp4.sendPack:servDhcp4.java:482 tx 192.168.136.123 op=rep sec=1 cia=0.0.0.0 yia=192.168.136.123 sia=192.168.136.1 gia=0.0.0.0 cha=9ceb.e8d5.2c51 srv= fil= op=offer dhcpsrv=192.168.136.1 hstnm=null dom=lan lease=43200 renew=21600 mask=255.255.255.0 gw=192.168.136.1 dns1=192.168.254.1 dns2=null req=null
debug serv.servDhcp4worker.doer:servDhcp4.java:686 tx op=rep sec=1 cia=0.0.0.0 yia=192.168.136.123 sia=192.168.136.1 gia=0.0.0.0 cha=9ceb.e8d5.2c51 srv= fil= op=offer dhcpsrv=192.168.136.1 hstnm=null dom=lan lease=43200 renew=21600 mask=255.255.255.0 gw=192.168.136.1 dns1=192.168.254.1 dns2=null req=null
debug serv.servDhcp4worker.doer:servDhcp4.java:679 rx op=req sec=2 cia=0.0.0.0 yia=0.0.0.0 sia=0.0.0.0 gia=0.0.0.0 cha=9ceb.e8d5.2c51 srv= fil= op=request dhcpsrv=192.168.136.1 hstnm=MBP-de-Frederic dom=null lease=0 renew=0 mask=null gw=null dns1=null dns2=null req=192.168.136.123
debug serv.servDhcp4.sendPack:servDhcp4.java:482 tx 192.168.136.123 op=rep sec=2 cia=0.0.0.0 yia=192.168.136.123 sia=192.168.136.1 gia=0.0.0.0 cha=9ceb.e8d5.2c51 srv= fil= op=ack dhcpsrv=192.168.136.1 hstnm=null dom=lan lease=43200 renew=21600 mask=255.255.255.0 gw=192.168.136.1 dns1=192.168.254.1 dns2=null req=null
debug serv.servDhcp4worker.doer:servDhcp4.java:686 tx op=rep sec=2 cia=0.0.0.0 yia=192.168.136.123 sia=192.168.136.1 gia=0.0.0.0 cha=9ceb.e8d5.2c51 srv= fil= op=ack dhcpsrv=192.168.136.1 hstnm=null dom=lan lease=43200 renew=21600 mask=255.255.255.0 gw=192.168.136.1 dns1=192.168.254.1 dns2=null req=null
info ip.ipCor6.parseIPheader:ipCor6.java:95 got bad version from ::
...
mjolnir#   

So based on the debug output:

  • DHCP allocated 192.168.136.123
  • Primary DNS (dns1) is 192.168.136.1
  • Network has /24 CIDR
 DHCP check @ laptop
DHCP debug command
╭─[11/2/20|4:30:09]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  ifconfig en8
en8: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=6407<RXCSUM,TXCSUM,VLAN_MTU,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
        ether 9c:eb:e8:d5:2c:51 
        inet6 fe80::c93:c3b1:dfb3:77c0%en8 prefixlen 64 secured scopeid 0x13 
        inet 192.168.136.123 netmask 0xffffff00 broadcast 192.168.136.255
        inet6 2a01:e0a:159:2856:832:82f5:8519:70 prefixlen 64 autoconf secured 
        inet6 2a01:e0a:159:2856:653c:d1b2:dca9:c9da prefixlen 64 autoconf temporary 
        nd6 options=201<PERFORMNUD,DAD>
        media: autoselect (1000baseT <full-duplex>)
        status: active

╭─[11/2/20|4:37:05]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  nslookup www.free.fr      
Server:         192.168.254.1
Address:        192.168.254.1#53

Non-authoritative answer:
Name:   www.free.fr
Address: 212.27.48.10

Pay attention to the DNS server that answered ! Sweet !

 IPv4 connectivity verification from laptop
DNS resolution from CLI (DNS query originated by router)
╭─[11/2/20|4:30:11]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  ping 8.8.8.8 -c 5
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=119 time=4.171 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=4.334 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=119 time=4.208 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=119 time=3.856 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=119 time=4.090 ms
Check that NAT is occuring for 192.168.136.126
mjolnir#show ipv4 nat inet translations | i 192.168.136.123                    
1      8.8.8.8 710148096           192.168.0.90 710148096   8.8.8.8 710148096        192.168.136.123 710148096   00:02:46  00:02:46  00:05:00  1       84
1      192.168.136.123 710148096   8.8.8.8 710148096        192.168.0.90 710148096   8.8.8.8 710148096           00:02:46  00:02:46  00:05:00  1       84
1      8.8.8.8 710148097           192.168.0.90 710148097   8.8.8.8 710148097        192.168.136.123 710148097   00:02:45  00:02:45  00:05:00  1       84
1      192.168.136.123 710148097   8.8.8.8 710148097        192.168.0.90 710148097   8.8.8.8 710148097           00:02:45  00:02:45  00:05:00  1       84
1      8.8.8.8 710148098           192.168.0.90 710148098   8.8.8.8 710148098        192.168.136.123 710148098   00:02:44  00:02:44  00:05:00  1       84
 Internet connectivity Check
DNS resolution from CLI (DNS query originated by router)
╭─[11/2/20|4:36:18]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  ping www.free.fr -c 5 
PING www.free.fr (212.27.48.10): 56 data bytes
64 bytes from 212.27.48.10: icmp_seq=0 ttl=57 time=3.903 ms
64 bytes from 212.27.48.10: icmp_seq=1 ttl=57 time=5.883 ms
64 bytes from 212.27.48.10: icmp_seq=2 ttl=57 time=3.658 ms
64 bytes from 212.27.48.10: icmp_seq=3 ttl=57 time=4.872 ms
64 bytes from 212.27.48.10: icmp_seq=4 ttl=57 time=4.488 ms

--- www.free.fr ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.658/4.561/5.883/0.787 ms
Check that NAT is occuring for 192.168.136.126 and www.free.fr IPv4 address
mjolnir#show ipv4 nat inet translations | i 212.27.48.10                       
1      192.168.136.123 1263796224  212.27.48.10 1263796224  192.168.0.90 1263796224  212.27.48.10 1263796224     00:03:47  00:03:47  00:05:00  1       84
1      212.27.48.10 1263796224     192.168.0.90 1263796224  212.27.48.10 1263796224  192.168.136.123 1263796224  00:03:47  00:03:47  00:05:00  1       84
1      192.168.136.123 1263796225  212.27.48.10 1263796225  192.168.0.90 1263796225  212.27.48.10 1263796225     00:03:46  00:03:46  00:05:00  1       84
1      212.27.48.10 1263796225     192.168.0.90 1263796225  212.27.48.10 1263796225  192.168.136.123 1263796225  00:03:46  00:03:46  00:05:00  1       84
1      192.168.136.123 1263796226  212.27.48.10 1263796226  192.168.0.90 1263796226  212.27.48.10 1263796226     00:03:45  00:03:45  00:05:00  1       84

Conclusion

In this article DHCP service has been enabled at:

  • SOHO router level 
  • All host getting an IPv4 via DHCP will get a DNS server set to SOHO@loopback0 (192.168.254.1)

RARE validated design: [ SOHO #005 ] - key take-away

In this example the key take-away are:

  • DHCP is available for IPv4 and IPv6
  • DHCP here is bound to an interface so rogue DHCP request coming from any router interface other than the speficied interface won't be honoured
  • Feel free to explore all DHCP options propose by RARE/freeRouter

DHCP for IPv6 is also available if you want to strictly reproduce IPv4 allocation scheme. In our case we will use IPv6 SLAC.

So everyone inside home network can reach the Internet thanks to NAT translation described in SOHO #004

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

In the previous article we enabled and checked IPv4 connectivity between all potential host within 192.168.128.0/17 and the outside Networks beyond ISP box. But, this is pretty useless as I can't imagine my kids typing IPv6 address (2001:8b0:0:30::666:102) in the browser in order to play a FUN puzzle. (Though for now we are suppose to have only IPv4 (smile)) So we definitely need to provide name service resolution at the SOHO router itself.

Article objective

In this article we will pursue the SOHO network appliance installation and enable name service to all host @ home.

Diagrams

[ SOHO #005 ] - "Got your Id number, but ... What's your name ?"

 Configure SOHO router as DNS client to an existing DNS server

First step, it is need to configure the router as a client name for an existing DNS server.

SOHO router in VRF inet
!
conf t
client name-server 8.8.8.8 1.1.1.1 
!

So this declare our SOHO router as DNS client for 8.8.8.8 as primary DNS server and 1.1.1.1 as backup DNS server.

 Set VRF for locally originated traffic from the SOHO router

This step is mandatory as it will bind traffic originated from SOHO router to a specific VRF (here: inet). So this can be also qualified as "VRF proxy-awareness". In this way all DNS traffic originated from the router will be bound to VRF inet. This is done in 2 steps. The first step is to create the proxy-profile and bind it to the main VRF inet. The second step is to declare the SOHO router as client of this proxy-profile service.

SOHO router in VRF inet
!
! step 1
!
proxy-profile pp-inet
 vrf inet
 exit
!
! step 2
!
client proxy pp-inet
!
 Setting UP DNS cache acting as DNS server for all host @ home network

Step -3-, configure DNS cache / server 

  • enable recursion (recursive query toward other DNS defined 8.8.8.8, 1.1.1.1)
  • bind it to a specific interface (so SOHO router will answer only DNS from this interface)
  • bind it to VRF inet
SOHO router DNS cache
!
server dns ns-inet
 recursion enable
 interface loopback0
 vrf inet
 exit
!

So this declare our SOHO router as DNS client for 8.8.8.8 as primary DNS server and 1.1.1.1 as backup DNS server

 Configure DNS into DHCP to propogate default "local" zone

Step -4-, configure DNS and DHCP to propagate default dummy zone local

  • Use local if you don't plan to propagate a domain name
  • create local as dummy zone


Propagate "local" zone
!
server dhcp4 dh4-inet
 dns-server 192.168.254.1
 domain-name local
 exit
!
server dns ns-inet
 zone local
 exit
!

Verification

 step -1- and step -2- outcomes

When -1- and -2- are realised the router can resolve name

DNS resolution from CLI (DNS query originated by router)
ping www.free.fr /vrf inet                                             
pinging 212.27.48.10, src=null, vrf=inet, cnt=5, len=64, tim=1000, ttl=255, tos=0, sweep=false
!!!!!
result=100%, recv/sent/lost=5/5/0, rtt min/avg/max/total=3/3/4/16
 step -3- and step -4- outcomes

This can be verified only using a host connected to SOHO router. Let's assume a laptop connected behind sdn6.

ping & ping6 hostname
...
╭─[10/31/20|3:01:19]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  ping www.free.fr
PING www.free.fr (212.27.48.10): 56 data bytes
64 bytes from 212.27.48.10: icmp_seq=0 ttl=57 time=3.670 ms
64 bytes from 212.27.48.10: icmp_seq=1 ttl=57 time=6.666 ms
64 bytes from 212.27.48.10: icmp_seq=2 ttl=57 time=6.163 ms
64 bytes from 212.27.48.10: icmp_seq=3 ttl=57 time=6.118 ms
^C
--- www.free.fr ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.670/5.654/6.666/1.166 ms
╭─[10/31/20|3:40:12]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  ping6 www.free.fr
PING6(56=40+8+8 bytes) 2a01:e0a:159:2857:b9d9:e9e0:ae30:88e5 --> 2a01:e0c:1::1
16 bytes from 2a01:e0c:1::1, icmp_seq=0 hlim=56 time=3.805 ms
16 bytes from 2a01:e0c:1::1, icmp_seq=1 hlim=56 time=6.898 ms
16 bytes from 2a01:e0c:1::1, icmp_seq=2 hlim=56 time=5.868 ms
16 bytes from 2a01:e0c:1::1, icmp_seq=3 hlim=56 time=5.729 ms
^C
--- www.free.fr ping6 statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 3.805/5.575/6.898/1.117 ms

...

IPv4 / IPv6 name resolution
...
╭─[10/31/20|3:42:11]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  dig www.free.fr                                                                                                                                                127 ↵

; <<>> DiG 9.10.6 <<>> www.free.fr
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25030
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.free.fr.                   IN      A

;; ANSWER SECTION:
www.free.fr.            20961   IN      A       212.27.48.10

;; Query time: 21 msec
;; SERVER: 192.168.254.1#53(192.168.254.1)
;; WHEN: Sat Oct 31 15:42:18 CET 2020
;; MSG SIZE  rcvd: 56

╭─[10/31/20|3:42:18]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  dig AAAA www.free.fr  

; <<>> DiG 9.10.6 <<>> AAAA www.free.fr
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21770
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.free.fr.                   IN      AAAA

;; ANSWER SECTION:
www.free.fr.            21075   IN      AAAA    2a01:e0c:1::1

;; Query time: 5 msec
;; SERVER: 192.168.254.1#53(192.168.254.1)
;; WHEN: Sat Oct 31 15:42:29 CET 2020
;; MSG SIZE  rcvd: 68
...

As said IPv6 verification are just FYI, as we are supposed to have deployed only IPv4 till now. The point to show off IPv6 verification is to verify DNS AAAA request are working properly.

Conclusion

In this article DNS service has been enabled at:

  • SOHO router level 
  • All host getting an IPv4 via DHCP will get a DNS server set to SOHO@loopback0 (192.168.254.1)

RARE validated design: [ SOHO #005 ] - key take-away

In this example the key take-away are:

  • proxy-profile usage in order to proxy DNS query into VRF inet
  • proxy-profile can be used to proxy other types of traffic
  • data/routed traffic is not affected by proxy-profile



If you have followed sequentially previous SOHO articles, you should have now a strong vanilla base in order to develop you home office network or you home network. 

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

In the previous article we enabled and checked IPv4 connectivity between RARE/freeRouter and ISP box using sdn1 interface within 192.168.0.0/24 network. But as stated in the previous post, I'd like:

  • all people connected within 192.168.128.0/17
  • to access the external world

Article objective

In this article we will pursue the SOHO network appliance installation and enable IPv4 connectivity for all host connected within your internal network to the external world.

Diagrams

[ #004 ] - Do you need translation ? 

 Create router loopback in VRF inet

First step, let's create an interface Loopback0 within 192.168.128.0/17, let's say 192.168.254.1/32

SOHO router in VRF inet
sh run loopback0                                                        
interface loopback0
 no description
 vrf forwarding inet
 ipv4 address 192.168.254.1 255.255.255.255
 no shutdown
 no log-link-change
 exit
!
 Problem observation

First step, let's try to ping 8.8.8.8

ping 8.8.8.8
ping 8.8.8.8 /vrf inet /interface lo0                                  
pinging 8.8.8.8, src=192.168.254.1, vrf=inet, cnt=5, len=64, tim=1000, ttl=255, tos=0, sweep=false
.....
result=0%, recv/sent/lost=0/5/5, rtt min/avg/max/total=10000/0/0/5003

Hey ! It seems that it's not working as expected ! At the present time, it seems that no one is able to reach outside world. Let's try to figure out why.

Note

  • It is a common best practice to dedicate a loopback to a router in a VRF
  • Loopback are mostly used to identify a service proposed by the router
  • Each services are bound to this Loopback in a specific VRF 
 First troubleshooting action
traceroute 8.8.8.8 using looback0 as source address
traceroute 8.8.8.8 /vrf inet /interface lo0                            
tracing 8.8.8.8, src=192.168.254.1, vrf=inet, prt=0/33440, tim=1000, tos=0, len=64
1 192.168.254.1 time=0
2 null time=1000
3 null time=1000
4 null time=1000
5 null time=1000
6 null time=1000
7 null time=1000
8 null time=1000
9 null time=1000
10 null time=1000

This confirms the ping failures we observed previously. The output above indicate the packet does not even egress our SOHO router.

What is the inet VRF says ?

routes inside VRF inet
show ipv4 route inet                                                   
typ  prefix            metric  iface      hop            time
C    192.168.0.0/24    0/0     sdn1       null           14:30:07
LOC  192.168.0.90/32   0/1     sdn1       null           14:30:07
C    192.168.128.0/24  0/0     sdn999     null           14:30:13
LOC  192.168.128.1/32  0/1     sdn999     null           14:30:13
C    192.168.254.1/32  0/0     loopback0  null           14:30:15

So we have no default routes . Let's configure one then pointing towards ISP BOX gateway:

Default route configuration
conf t
ipv4 route inet 0.0.0.0 0.0.0.0 192.168.0.254
routes inside VRF inet
show ipv4 route inet                                                   
typ  prefix            metric  iface      hop            time
S    0.0.0.0/0         1/0     sdn1       192.168.0.254  14:30:07
C    192.168.0.0/24    0/0     sdn1       null           14:30:07
LOC  192.168.0.90/32   0/1     sdn1       null           14:30:07
C    192.168.128.0/24  0/0     sdn999     null           14:30:13
LOC  192.168.128.1/32  0/1     sdn999     null           14:30:13
C    192.168.254.1/32  0/0     loopback0  null           14:30:15

So at that point, packet send to 8.8.8.8 are sent to nexthop 192.168.0.254 via sdn1.

ping 8.8.8.8
ping 8.8.8.8 /vrf inet /interface lo0                                  
pinging 8.8.8.8, src=192.168.254.1, vrf=inet, cnt=5, len=64, tim=1000, ttl=255, tos=0, sweep=false
.....
result=0%, recv/sent/lost=0/5/5, rtt min/avg/max/total=10000/0/0/5003

But ping is still not not working. Let's figure out what's going on here.

 Networking environment assumption

As depicted in previous article:

  • ISP box has a demarcation point set to 192.168.0.254
  • So ISP box at some point is configured to perform Network Address Translation from 192.168.0.0/24 → ISP public IPv4 interface
  • When ISP box receives a ICMP ping from 192.168.254.1 which does not match any ISP box NAT rules → Packet is discarded

Therefore in order to have a working seamless networking environment from the ISP box point of view, traffic coming from 192.168.128.0/17 might need to be NAT(ed) into 192.168.0.0/24 network. Let's see If our guess is right.

 NAT to the rescue

Let's configure IPv4 Network Address Translation specifically for Loopback0

ping 8.8.8.8
!
conf t
access-list ACL-NAT4   
 sequence 10 deny all 192.168.128.0 255.255.128.0 all 192.168.128.0 255.255.128.0 all
 sequence 20 deny all any all 224.0.0.0 240.0.0.0 all                                          
 permit all 192.168.254.1 255.255.255.255 all any all     
exit
!
ipv4 nat inet sequence 10 srclist ACL-NAT4 interface sdn1
!

 In the config stanza above we configure an access-list called ACL-NAT4 in VRF inet that would translate all incoming packet matching 192.168.254.1 going to anywhere to interface sdn1 IPv4. But we also don't NAT for inter-vlan traffic within 192.168.128.0/17 and also does not touch multicast traffic.

ping 8.8.8.8
ping 8.8.8.8 /vrf inet /interface sdn1                                 
pinging 8.8.8.8, src=192.168.0.90, vrf=inet, cnt=5, len=64, tim=1000, ttl=255, tos=0, sweep=false
!!!!!
result=100%, recv/sent/lost=5/5/0, rtt min/avg/max/total=3/3/4/16

Yahhoooo ! It works ... Our guess was good after all... (smile)

 Final NAT configuration and subnetting assumptions

For the future we assume that the home network is subnetted as follow and are all inside VRF inet:

  • appliance port#1 @ sdn1: 192.168.0.0/24 (ISP subnet)
  • appliance port#2 @ sdn2: 192.168.130.0/24
  • appliance port#3 @ sdn3: 192.168.133.0/24
  • appliance port#4 @ sdn4: 192.168.134.0/24 (unused)
  • appliance port#5 @ sdn5: 192.168.135.0/24
  • appliance port#6 @ sdn6: 192.168.136.0/24
  • appliance integrated wifi @ sdn998: 192.168.129.0/24
  • appliance loopback inet @ loopback0: 192.168.129.0/24


ping 8.8.8.8
!
conf t
access-list ACL-NAT4
 sequence 10 deny all 192.168.128.0 255.255.128.0 all 192.168.128.0 255.255.128.0 all
 sequence 20 deny all any all 224.0.0.0 240.0.0.0 all
 sequence 30 permit all 192.168.128.0 255.255.255.0 all any all
 sequence 40 permit all 192.168.129.0 255.255.255.0 all any all
 sequence 50 permit all 192.168.130.0 255.255.255.0 all any all
 sequence 60 permit all 192.168.133.0 255.255.255.0 all any all
 sequence 70 permit all 192.168.135.0 255.255.255.0 all any all
 sequence 80 permit all 192.168.136.0 255.255.255.0 all any all
 sequence 90 permit all 192.168.254.1 255.255.255.255 all any all
 exit
!
ipv4 nat inet sequence 10 srclist ACL-NAT4 interface sdn1
!

 In the config stanza above we configure an access-list called ACL-NAT4 in VRF inet that would translate all incoming packet matching 192.168.[128|129|130|133|135|136|254].0/24 going to anywhere.

Note

Each time you modify ACL-NAT4 you will need to re-apply the NAT calls:

ping 8.8.8.8
!
conf t
! modify access-list ACL-NAT4
!
no ipv4 nat inet sequence 10 srclist ACL-NAT4 interface sdn1
ipv4 nat inet sequence 10 srclist ACL-NAT4 interface sdn1
!

Also notice that we are NAT(ing), 192.168.128.0/24 which is the OOBM subnet of the Linux appliance itself. This is needed during Debian system update/upgrade operation and also needed during the auto-upgrade feature that I'll describe in a next article.

IPv6 does need NAT in my specific case as my ISP has allocated me public IPv6 prefixes. We will see IPv6 configuration in the next articles.

Verification

 Available NAT command verification
NAT show commands
!
show ipv4 nat ?                                                        
  <vrf> - name of routing table
!
show ipv4 nat inet ?                                                   
  statistics   - list of configuration entries
  translations - list of translation entries
!
 Trigger NAT traffic translation
ping 8.8.8.8 in order to trigger NAT translation
ping 8.8.8.8 /vrf inet /interface lo0                                 
pinging 8.8.8.8, src=192.168.254.1, vrf=inet, cnt=5, len=64, tim=1000, ttl=255, tos=0, sweep=false
!!!!!
result=100%, recv/sent/lost=5/5/0, rtt min/avg/max/total=2/3/4/16
 Check NAT translation
NAT translation verification
mjolnir#show ipv4 nat inet translations | i 192.168.254.1                      
1      8.8.8.8 4392          192.168.0.90 4392      8.8.8.8 4392          192.168.254.1 4392     00:00:01  00:00:01  00:05:00  1        64
1      192.168.254.1 4392    8.8.8.8 4392           192.168.0.90 4392     8.8.8.8 4392           00:00:01  00:00:01  00:05:00  1        64
1      8.8.8.8 4393          192.168.0.90 4393      8.8.8.8 4393          192.168.254.1 4393     00:00:01  00:00:01  00:05:00  1        64
1      192.168.254.1 4393    8.8.8.8 4393           192.168.0.90 4393     8.8.8.8 4393           00:00:01  00:00:01  00:05:00  1        64
1      8.8.8.8 4394          192.168.0.90 4394      8.8.8.8 4394          192.168.254.1 4394     00:00:01  00:00:01  00:05:00  1        64
1      192.168.254.1 4394    8.8.8.8 4394           192.168.0.90 4394     8.8.8.8 4394           00:00:01  00:00:01  00:05:00  1        64
1      8.8.8.8 4395          192.168.0.90 4395      8.8.8.8 4395          192.168.254.1 4395     00:00:01  00:00:01  00:05:00  1        64
1      192.168.254.1 4395    8.8.8.8 4395           192.168.0.90 4395     8.8.8.8 4395           00:00:01  00:00:01  00:05:00  1        64
1      8.8.8.8 4396          192.168.0.90 4396      8.8.8.8 4396          192.168.254.1 4396     00:00:01  00:00:01  00:05:00  1        64
1      192.168.254.1 4396    8.8.8.8 4396           192.168.0.90 4396     8.8.8.8 4396           00:00:01  00:00:01  00:05:00  1        64

Conclusion

In this article

  • We finally have a router that enables connectivity for all hosts inside the home network to the outside world
  • due to the ISP specific setup, our router had to translate inner home IP subnets to subnet that can be in turn NAT'ed by the ISP box.
  • We have a consistent IPv4 addressing plan 
  • We now can add very exciting feature from now on. (In next articles !)


RARE validated design: [ SOHO #004 ] - key take-away

In this example we are proposing a basic connectivity scenario. However, keep in mind that depending on your location the configuration might be drastically different. But do not fear ! RARE/freeRouter has all the features need to enable connectivity !

  • NAT64 is available. So in case you want to run a pure IPv6 network, freeRouter can NAT64 traffic for you.
  • NAT46 is also available. In case you are desperate and don't want to implement a pure IPv6 home network and have an ISP running only IPv6, freeRouter can NAT46 your traffic for you !
  • In the example described, we are lucky to have IPv6 public global IPv6 address. We will see IPv6 configuration in subsequent articles.

Now that you have a hardware for your SOHO appliance, installed an Operating System and prepared  systemd init script in order to resume freeRouter operation in case of unexpected outage (power cut, reset button etc.), let's proceed to RARE/freeRouter installation itself.

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

When installing RARE/freeRouter on x86, you have 2 choices:

  • installation with a software dataplane
  • installation with a DPDK dataplane


In this precise case, we will consider a DPDK dataplane installation as our hardware is compliant to the requirement listed below.

DPDK requirements

  • CPU with SSE4 support
  • DPDK compatible NIC 

Note that freeRouter is available where JVM is available

  • x86
  • ARM

Article objective

In this article we will pursue the SOHO network appliance installation based on the diagram below, and freeRouter installation using  DPDK dataplane. In this situation, the appliance is behind ISP FTTH box demarcation point. As it is typical to French FTTH domestic deployment. 

Deployment consideration

In this case, RARE/freeRouter is connected to a ISP box demarcation point that deliver copper connectivity. Nothing prevents you, following your context, to deploy a similar equipment with with SFP uplinks directly connected to your Provider Edge backbone routers if you own also the dark fiber paths local to the MAN. 

Diagrams

[ #003 ] - RARE/freeRouter DPDK SOHO installation 

 Requirements
  • Own a similar hardware described in SOHO #001
  • Having installed an Operating System with Java Runtime Environment
  • Configured systemd so that RARE/freeRouter can take over networking at each reboot as described in SOHO #002.
 IPv4 addressing plan

Let's consider the following assumptions:

  • ISP box comes with 192.168.0.0/24 subnet configured at RJ45 demarcation point
  • Home networkS will be within 192.168.128.0/17
  • 192.168.128.0/17 will be subnetted further into multiple /24 in order to accomodate home network requirement
  • RARE/freeRouter is connected to the FTTP ISP box via appliance DPDK port #0 (interface sdn1)
  • Home traffic going to outside world will be subject to port address translation (NAT/PAT) using an IPv4 within ISP subnet range
  • appliance port #1 will be connected to FTTH ISP box and will have an IP within 192.168.0.0/24

IPv6 addressing plan has not been forgotten. It is not mentioned here on purpose in order to not complicate explanations. IPv6 we be the object of further articles. It is not that IPv6 is a complex topic. It just that it deserves special attention. You might not realised it, but IPv6 is everywhere and is used by default between peers as soon as IPv6 is enable. So IMHO we need to get used to it as soon as possible especially if you are a network administrator.

 Create configuration files for RARE/freerouter

FreeRouter uses 2 configuration files in order to run, let's write these configuration files in /rtr

freeRouter hardware configuration file: rtr-hw.txt
hwid j1900-i211
! cpu_port
int eth0 eth - 127.0.0.1 20001 127.0.0.1 20002
! freerouter control port for message packet-in/out in P4 VRF _ONLY_
tcp2vrf 9080 p4 9080
! freeroouter local access in p4 VRF _ONLY_
tcp2vrf 2323 p4 23
! launch a process called "veth0" that actually link to veth0b
! cmd for control plane/dataplane communication unified messaging: ip link add veth0a type veth peer name veth0b
! cmd for appliance Linux access: ip link add veth1a type veth peer name veth1b
! cmd for integrated wifi: ip link add veth2a type veth peer name veth2b
! external wifi AP
proc hostapd /usr/sbin/hostapd /etc/hostapd/hostapd.conf
! integrated wifi AP
proc wlan /rtr/pcap2pcap.bin wlan0 veth2a
! DP/CP communication process
proc veth0 /rtr/pcapInt.bin veth0a 20002 127.0.0.1 20001 127.0.0.1
! DP DPDK process
proc p4emu /rtr/p4dpdk.bin --vdev=net_af_packet0,iface=veth0b --vdev=net_af_packet1,iface=veth2b --vdev=net_af_packet2,iface=veth1b 127.0.0.1 9080 6

Note:

Let's spend some times on this hardware configuration file, as you might have notice there are additional interesting lines worth to mention:

  • Exclamation mark "!" are comments
  • hwid is a text field that would just designate the hardware on which freeRouter is running. (output of : show platform)
  • proc <process-name>

It is possible within freeRouter startup to launch processes. We use here this feature to start control plane / dataplane communication via veth pair: veth0a and veth0b and also P4Emu/dpdk, p4dpdk.bin packet processing backend.

  • proc p4emu /rtr/p4dpdk.bin --vdev=net_af_packet0,iface=veth0b --vdev=net_af_packet1,iface=veth2b --vdev=net_af_packet2,iface=veth1b 127.0.0.1 9080 6

In dpdk, by default dpdk interfaces have port_ids that are sequentially allocated and in the order of appearance in dpdk-devbind --status output usually sorted by pci_id. In the below output interface enp0s1 has port_id #0 and in dpdk it would be pci_id:00:01.0

enp0s1 would be: #0 with pci_id: 00:01.0

enp0s2 would be: #1 with pci_id: 00:02.0

enp0s5 would be: #2 with pci_id: 00:05.0

enp0s6 would be: #3 with pci_id: 00:06.0

enp0s7 would be: #4 with pci_id: 00:07.0

enp0s8 would be: #5 with pci_id: 00:08.0

DPDK diagnosis
dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:01:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb
0000:02:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb
0000:05:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb
0000:06:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb
0000:07:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb
0000:08:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb

Network devices using kernel driver
===================================
0000:09:00.0 'AR928X Wireless Network Adapter (PCI-Express) 002a' if=wlan0 drv=ath9k unused=uio_pci_generic 

No 'Baseband' devices detected
==============================

Other Crypto devices
====================
0000:00:1a.0 'Atom Processor Z36xxx/Z37xxx Series Trusted Execution Engine 0f18' unused=uio_pci_generic

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================
  • DPDK --vdev addition. In this precise case we instruct DPDK to take into account additional veth endpoint we created respectively for
    • Control plane / data plane communication
    • Linux out of band management access via SSH we installed previously during Debian package installation
    • integrated hardware WIFI access point
  • in DPDK vdev interface will have in order of apparition in the command line:
    • DP/CP communication: 6 ↔ veth0b
    • integrated WIFI: 7 ↔ veth2b
    • Linux out of band management access: 8 ↔ veth1b

external WIFI access point will be bound directly to an interface of the appliance via DPDK. This will be describe in future articles.

freeRouter software configuration file: rtr-sw.txt
hostname mjolnir
buggy
!
!
vrf definition inet
 exit
!
vrf definition p4
 exit
!
interface ethernet0
 description freerouter@P4_CPU_PORT[veth0a]
 no shutdown
 no log-link-change
 exit
!
interface sdn1
 description freerouter@DPDK[port-1]
 mtu 1500
 vrf forwarding inet
 ipv4 address 192.168.0.90 255.255.255.0
 no shutdown
 no log-link-change
 exit
!
interface sdn2
 description freerouter@DPDK[port-2]
 mtu 1500
 shutdown
 no log-link-change
 exit
!
interface sdn3
 description freerouter@DPDK[port-3]
 mtu 1500
 shutdown
 no log-link-change
 exit
!
interface sdn4
 description freerouter@DPDK[port-4]
 mtu 1500
 shutdown
 no log-link-change
 exit
!
interface sdn5
 description freerouter@DPDK[port-5]
 mtu 1500
 shutdown
 no log-link-change
 exit
!
interface sdn6
 description freerouter@DPDK[port-6]
 mtu 1500
 shutdown
 no log-link-change
 exit
!
interface sdn998
 description freerouter@DPDK[port-7 --> veth2a] integrated wifi
 mtu 1500
 shutdown
 no log-link-change
 exit
!
interface sdn999
 description freerouter@OOBM[port-8 --> veth1a] Linux management
 mtu 1500
 vrf forwarding inet
 ipv4 address 192.168.128.1 255.255.255.0
 no shutdown
 no log-link-change
 exit

server telnet telnet
 security protocol telnet
 no exec authorization
 no login authentication
 vrf p4
 exit
!
server p4lang p4
 export-vrf inet 1
 export-port sdn1 0 1 0 0 0
 export-port sdn2 1 1 0 0 0
 export-port sdn3 2 1 0 0 0
 export-port sdn4 3 1 0 0 0
 export-port sdn5 4 1 0 0 0
 export-port sdn6 5 1 0 0 0
 export-port sdn998 7 1 0 0 0
 export-port sdn999 8 1 0 0 0
 interconnect ethernet0
 vrf p4
 exit
!
!
end
  • For now integrated wifi is shut. We will see in later article how to activate it
  • At Linux level, if you noticed in the previous article
    • management IP subnet is 192.168.128.0/24. OOBM appliance IP is then 192.168.128.254
appliance management IP@Linux level (check previous article)
ip addr flush dev veth1a
ip addr add 192.168.128.254/24 dev veth1a
  • management IP seen from freeRouter@sdn999 with IP 192.168.128.1 within 192.168.128.0/24
  • with configured a Linux static routes
Add default route to OOBM sdn999@Linux level (check previous article)
route add default gw 192.168.128.1

Security note

  • If you pay attention p4lang server in p4 VRF
    • This VRF has no bound interface
    • Is isolated then from the other VRF
  • This will allow only local Linux host control plane and dataplane communication 

Verification

 connectivity check from freeRouter to ISP IPv4 gateway
ping ISP demarcation point IP
ping 192.168.0.254 /vrf inet /interface sdn1                           
pinging 192.168.0.254, src=192.168.0.90, vrf=inet, cnt=5, len=64, tim=1000, ttl=255, tos=0, sweep=false
!!!!!
result=100%, recv/sent/lost=5/5/0, rtt min/avg/max/total=0/0/1/4
ARP discovery
mjolnir#sh ipv4 arp sdn1                                                       
mac             address        time      static
0024.d4a0.0cd3  192.168.0.254  00:00:20  false
 Check freeRouter interface configuration
Add default route to OOBM sdn999@Linux level (check previous article)
sh int sdn1                                                          
sdn1 is up (since 13:14:14, 2 changes)
 description: mjolnir@LAN1[01:00.0]
 type is sdn, hwaddr=003b.7671.764f, mtu=1500, bw=8000kbps, vrf=inet
 ip4 address=192.168.0.90/24, netmask=255.255.255.0, ifcid=10014
 received 64038 packets (17841459 bytes) dropped 4 packets (326 bytes)
 transmitted 250217 packets (38032822 bytes) promisc=false macsec=false
interface summary
show interfaces summary                                                
interface    state  tx        rx        drop
ethernet0    up     74690935  51798769  0
sdn1         up     37954707  17828649  326
sdn2         admin  0         0         0
sdn3         admin  0         0         0
sdn4         admin  0         0         0
sdn5         admin  0         0         0
sdn6         admin  0         0         0
sdn998       admin  0         0         0
sdn999       up     23646     17904     0
interface summary
interface   state  tx          rx          drop
sdn1        up     674397352   3883928390  948
sdn2        admin  0           0           0
sdn3        admin  0           0           0
sdn4        admin  0           0           0
sdn5        admin  0           0           0
sdn6        admin  0           0           0
sdn998      admin  0           0           0
sdn999      up     110520      85072       0
 Check freeRouter CLI access via localhost

Check Linux appliance local routes

From linux terminal
root@mjolnir:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.128.1   0.0.0.0         UG    0      0        0 veth1a

Test local telnet access from linux/localhost

Conclusion

In this article

  • we finally launched RARE/freeRouter with DPDK dataplane
  • configure RARE/freeRouter with a vanilla config that takes into account all the appliance physical interfaces
  • added veth pair in the config in order to take into account:
    • Control plane / Data plane communication 
    • linux OOBM
    • integrated WIFI
  • Enabled and checked IPv4 connectivity between freeRouter@sdn1 and ISP demarcation point
  • Check telnet access to freeRouter from localhost only

RARE validated design: [ SOHO #003 ] - key take-away

From this point you have a complete freeRouter connected to ISP box via SDN1 as uplink in 192.168.0.0/24 subnet. We will extend further this base configuration step by step in order to enrich user experience !

  • Now you would want to enable IPv4/IPv6  connectivity to all potential hosts@home whether they are connected via RJ45 or via built-in WIFI.
  • you would also want to distribute IPv4, IPv6 to all the of hosts@home
  • IPv4/IPv6 connectivity is not enough, you would like to provide Domain Name Service to them
  • Domain Name Service is not enough if they can't reach outside world. As we are using RFC1918 addressing plan we should figure out a way to ensure NAT/PAT address translation in order to enable egress traffic toward the Internet
  • Your home might have several floors and only one WIFI access point is not enough ? Let's see how we can add additional WIFI AP in the network
  • Maybe you have an outsourced network management service ? Let's see how connectivity can be enable via OpenVPN encrypted tunnel
  • Last but not least, let's see how we can connect DN42 parallel network using a Wireguard tunnel relying on an IPv6 underlay.

You've guessed it, all of these points will be elaborated in the futures articles. Therefore stay tuned !

This is a new article for the blog serie called "RARE Day One". Today we will explore one of freeRouter feature that is used a lot in Service Provider trusted environment" TFTP server

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

I'm not sure if this is still the case now, but back in 1999, I had the opportunity to managed multiple VPNs at a very huge French Service Provider. I'm saying huge as in this type of MPLS muti-service core network, you could have hundreds of VRF in the same PE router connecting a myriads of CPE via X25 (XOT), frame-relay and ATM PVC at best. In that context, some companies could have several thousands of routers in their VPNs and it was not common to follow a high pace deployment which was at ~10 CPEs per day for a new customer VPN implementation. So one of my favorite CLI command was:

staging a CPE with its final configuration
copy tftp run
Address or name of remote host []? <x.y.z.t>
Source filename []? <router-cpe-config-file-name>
Destination filename [running-config]?
...

That being said, I'm not sure if this has evolved since then as TFTP occurred inside a very protected out of band management network, it was very good and did a perfect job. Keep in mind that we could be hundreds of "VPN owner" deploying CPEs at the same time. This has to be highly available.

That was for the anecdote, but recently I attempted to upgrade my OpenWRT wifi router from 18.06.02 to the latest code train: 19.07.4. As a I'm lazy, I just sticked with OpenWRT web upgrade via LuCI. Not sure if I was right ... I don't know why and how but the upgrade failed and my wifi router got "bricked". (smile)

After a lot of googling and reading, i concluded that I had only one solution: restore from factory and re-install OpenWRT 19.07.04 installation by hand. You have guess the rest of the article, the factory-reset procedure requires a TFTP server. (smile) 

Note

But before that, I had to solder an USB - UART module as described here.

Article objective

As again i was lazy on installing a TFTP server on my MAC and disconnect my current LAN access in order to have a direct connectivity with the OpenWRT box, I had an idea (this is not often (smile)) off the top of my head: "Hey, maybe freeRouter has a TFTP server that I can activate in few lines ?"... Well, after a terminal connection to my home router let me introduce you to freeRouter/TFTP server:

 

[ #004 ] - Saving private OpenWRT", thanks freeRouter's TFTP server !

 Log into freeRouter

If you are familiar with Cisco operating system you will feel at home with this TFTP server. 

Log into freeRouter in config mode:
   __               ____             _
  / _|_ __ ___  ___|  _ \ ___  _   _| |_ ___ _ __
 | |_| '__/ _ \/ _ \ |_) / _ \| | | | __/ _ \ '__|
 |  _| | |  __/  __/  _ < (_) | |_| | ||  __/ |
 |_| |_|  \___|\___|_| \_\___/ \__,_|\__\___|_|
  _ __ ___   ___| | _____  | |
 | '__/ _ \ / __| |/ / __| | |
 | | | (_) | (__|   <\__ \ |_|
 |_|  \___/ \___|_|\_\___/ (_)

welcome
line ready
mjolnir#conf t                                                                 
mjolnir(cfg)#server tftp openwrt                                               
mjolnir(cfg-server)#?                                                          
  access-blackhole4 - propagate and check violating prefixes
  access-blackhole6 - propagate and check violating prefixes
  access-class      - set access list
  access-log        - log dropped attemps
  access-map        - set route map
  access-peer       - per client session limit
  access-policy     - set route policy
  access-prefix     - set prefix list
  access-rate       - access rate for this server
  access-startup    - initial downtime for this server
  access-subnet     - per subnet session limit
  access-total      - session limit for this server
  do                - execute one exec command
  end               - close this config session
  exit              - go back to previous mode
  interface         - interface to bind to
  no                - negate a command
  path              - set root folder
  port              - set port to listen on
  protocol          - set lower protocols to use
  readonly          - set write protection
  security          - set security parameters
  show              - running system information
  vrf               - set vrf to use

 TFTP server configuration

sdn6 is the port #6 connected from my SOHO router to OpenWRT router.

TFTP server configuration
sh run tftp                                                            
server tftp openwrt
 path /rtr/owrt/
 interface sdn6
 vrf inet
 exit
!
sh run sdn6                                                           
interface sdn6
 description mjolnir@LAN6[08:00.0]
 mtu 1500
 macaddr 004c.7307.0a77
 vrf forwarding inet
 ipv4 address 192.168.136.1 255.255.255.0
 ipv4 broadcast-multicast
 no shutdown
 no log-link-change
 exit
!
...

So the LAN port of my OpenWRT router is like this:

OpenWRT config (this can be done via Web GUI)
...
config interface 'lan'
        option type 'bridge'
        option proto 'static'
        option ipaddr '192.168.136.2'
        option netmask '255.255.255.0'
        option broadcast '192.168.136.255'
        option gateway '192.168.136.1'
        option ip6assign '60'
        list dns '192.168.254.1'
        option ifname 'eth0 eth0.1 eth0.2 wlan0 wlan1' 
...

Basic connectivity check (well technically you could not ping as it is part if TFTP restore to factory process. Remember our box crashed ! (smile) )

ping OpenWRT
ping 192.168.136.2 /vrf inet                                           
pinging 192.168.136.2, src=null, vrf=inet, cnt=5, len=64, tim=1000, ttl=255, tos=0, sweep=false
!!!!!
result=100%, recv/sent/lost=5/5/0, rtt min/avg/max/total=0/1/2/5
...
 Launch OpenWRT TFTP factory reset

So we are basically ready ...

Initiate OpenWRT factory restore process via TFTP
===================================================================
                MT7621   stage1 code 10:33:11 (ASIC)
                CPU=50000000 HZ BUS=12500000 HZ
==================================================================
Change MPLL source from XTAL to CR...
do MEMPLL setting..
MEMPLL Config : 0x11100000
3PLL mode + External loopback
=== XTAL-40Mhz === DDR-1200Mhz ===
PLL2 FB_DL: 0x9, 1/0 = 567/457 25000000
PLL3 FB_DL: 0xc, 1/0 = 596/428 31000000
PLL4 FB_DL: 0x11, 1/0 = 560/464 45000000
do DDR setting..[00320381]
Apply DDR3 Setting...(use customer AC)
          0    8   16   24   32   40   48   56   64   72   80   88   96  104  112  120
      --------------------------------------------------------------------------------
0000:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0001:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0002:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0003:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0004:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0005:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0006:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0007:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0008:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0009:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
000A:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
000B:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
000C:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
000D:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    1
000E:|    0    0    0    0    0    0    0    0    0    1    1    1    1    1    1    1
000F:|    0    0    0    0    1    1    1    1    1    1    1    1    1    1    0    0
0010:|    1    1    1    1    1    1    1    1    1    0    0    0    0    0    0    0
0011:|    1    1    1    0    0    0    0    0    0    0    0    0    0    0    0    0
0012:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0013:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0014:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0015:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0016:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0017:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0018:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0019:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
001A:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
001B:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
001C:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
001D:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
001E:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
001F:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
rank 0 coarse = 15
rank 0 fine = 72
B:|    0    0    0    0    0    0    0    0    0    0    1    1    1    0    0    0
opt_dle value:11
DRAMC_R0DELDLY[018]=00001F1F
==================================================================
                RX      DQS perbit delay software calibration 
==================================================================
1.0-15 bit dq delay value
==================================================================
bit|     0  1  2  3  4  5  6  7  8  9
--------------------------------------
0 |    10 7 9 9 7 7 8 7 3 6 
10 |    6 7 7 9 6 9 
--------------------------------------

==================================================================
2.dqs window
x=pass dqs delay value (min~max)center 
y=0-7bit DQ of every group
input delay:DQS0 =31 DQS1 = 31
==================================================================
bit     DQS0     bit      DQS1
0  (1~61)31  8  (1~56)28
1  (1~58)29  9  (1~61)31
2  (1~60)30  10  (1~59)30
3  (1~58)29  11  (1~57)29
4  (1~57)29  12  (1~60)30
5  (1~61)31  13  (1~60)30
6  (1~58)29  14  (1~61)31
7  (1~62)31  15  (1~61)31
==================================================================
3.dq delay value last
==================================================================
bit|    0  1  2  3  4  5  6  7  8   9
--------------------------------------
0 |    10 9 10 11 9 7 10 7 6 6 
10 |    7 9 8 10 6 9 
==================================================================
==================================================================
     TX  perbyte calibration 
==================================================================
DQS loop = 15, cmp_err_1 = ffff0000 
dqs_perbyte_dly.last_dqsdly_pass[0]=15,  finish count=1 
dqs_perbyte_dly.last_dqsdly_pass[1]=15,  finish count=2 
DQ loop=15, cmp_err_1 = ffff0080
dqs_perbyte_dly.last_dqdly_pass[1]=15,  finish count=1 
DQ loop=14, cmp_err_1 = ffff0000
dqs_perbyte_dly.last_dqdly_pass[0]=14,  finish count=2 
byte:0, (DQS,DQ)=(8,8)
byte:1, (DQS,DQ)=(8,8)
20,data:88
[EMI] DRAMC calibration passed

===================================================================
                MT7621   stage1 code done 
                CPU=50000000 HZ BUS=12500000 HZ
===================================================================


U-Boot 1.1.3 (Apr 17 2017 - 17:00:02)

Board: Ralink APSoC DRAM:  256 MB
Power on memory test. Memory size= 256 MB...OK!
relocate_code Pointer at: 8ffac000

Config XHCI 40M PLL 
******************************
Software System Reset Occurred
******************************
Allocate 16 byte aligned buffer: 8ffdffd0
Enable NFI Clock
# MTK NAND # : Use HW ECC
NAND ID [C8 D1 80 95 42]
Device not found, ID: c8d1
Not Support this Device! 
chip_mode=00000001
Support this Device in MTK table! c8d1 
select_chip
[NAND]select ecc bit:4, sparesize :64 spare_per_sector=16
Signature matched and data read!
load_fact_bbt success 1023
load fact bbt success
[mtk_nand] probe successfully!
mtd->writesize=2048 mtd->oobsize=64,    mtd->erasesize=131072  devinfo.iowidth=8
..============================================ 
Ralink UBoot Version: 5.0.0.0
-------------------------------------------- 
ASIC MT7621A DualCore (MAC to MT7530 Mode)
DRAM_CONF_FROM: Auto-Detection 
DRAM_TYPE: DDR3 
DRAM bus: 16 bit
Xtal Mode=5 OCP Ratio=1/4
Flash component: NAND Flash
Date:Apr 17 2017  Time:17:00:02
============================================ 
icache: sets:256, ways:4, linesz:32 ,total:32768
dcache: sets:256, ways:4, linesz:32 ,total:32768 

 ##### The CPU freq = 880 MHZ #### 
 estimate memory size =256 Mbytes
#Reset_MT7530
set LAN/WAN LWLLL

Please choose the operation: 
   1: Load system code to SDRAM via TFTP. 
   2: Load system code then write to Flash via TFTP. 
   3: Boot system code via Flash (default).
   4: Entr boot command line interface.
   7: Load Boot Loader code then write to Flash via Serial. 
   9: Load Boot Loader code then write to Flash via TFTP. 
 4 
You choosed 2

 0 

   
2: System Load Linux Kernel then write to Flash via TFTP. 
 Warning!! Erase Linux in Flash then burn new one. Are you sure?(Y/N)
 Please Input new ones /or Ctrl-C to discard
        Input device IP (192.168.31.1) ==:192.168.31.1
        Input server IP (192.168.31.100) ==:192.168.31.2  
        Input Linux Kernel filename () ==: <my_factory_router_image>


...

And ... Voilà !

Note

We won this factory reset battle but the war is over. After having restored the genuine official vendor image, we need to re-install OpenWRT with the latest 19.07.4 image and configure OpenWRT so that it can acts as a "dummy Wifi Access Point". DHCP, DNS will be served by the SOHO router.

Discussion

You can deploy freeRouter manually in a VM or container and bind it to a linux interface if you need a TFTP server in order to apply configuration to all your equipment. When final staging are done in a secure Out of Band management network context having a TFTP server is a blessing as it correspond to a gain of time in a production environment. Imaging hundreds of people working in a SP environment and working at the same time.

Conclusion

In this 4th article:

  • We presented freeRouter TFTP embedded server 
  • You can use it in order to undertake network equipment deployment requiring TFTP
  • This TFTP server is compatible with IPv4/IPv6

TFTP is a basic but a common tool in SP environment (or it was? If it is still used, yes please confirm !) In this example, I demonstrated the use of TFTP server in order to flash a wifi router to factory default. I have 802.11ac back up and running !

Final words

freeRouter can be perceived not only as a router but it is a networking Swiss army knife. in further articles we will shed some lights in various treasures hidden into freeRouter... And for free !

Last but not least, you can play with these different servers from this sandbox: (You'll be able to spot amazing server that will be the object of further article.)

type "telnet dl.nop.hu" in a terminal and choose "1"
Trying 193.224.23.5...
Connected to dl.nop.hu.
Escape character is '^]'.
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXX XXXXX XXX    XXX     XXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX  XXXX XX XXXX XX XXXX XX XX XX XXXX XXXXX/~~~~~~\XXXXXX
XXXX X XXX XX XXXX XX XXXX XX XX XX XXXX XXXX| player |XXXXX
XXXX XX XX XX XXXX XX     XXX    XX XXXX XXXXX\______/XXXXXX
XXXX XXX X XX XXXX XX XXXXXXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX XXXX  XX XXXX XX XXXXXXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX XXXXX XXX    XXX XXX XXX XX XXX    XXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
welcome
line ready
menu lab:
# - reboot router1
$ - reboot router2
% - reboot router3
1 - connect to router1
2 - connect to router2
3 - connect to router3
^ - rebuild routers
l - connect to lg.nop.dn42
x - exit
choose:1 - attach vdc lab1 

welcome
line ready
yourname#conf t                                                                
warning user.userLineHandler.doExec:userLine.java:606 <nobody> configuring from tty1
yourname(cfg)#server ?                                                         
  bmp2mrt      - configure an bmp to mrt server
  bstun        - configure a bstun server
  chargen      - configure a chargen server
  daytime      - configure a daytime server
  dcp          - configure a dcp server
  dhcp4        - configure a dhcp4 server
  dhcp6        - configure a dhcp6 server
  discard      - configure a discard server
  dns          - configure a dns server
  echo         - configure an echo server
  etherip      - configure a etherip server
  forwarder    - configure a forwarder server
  ftp          - configure a ftp server
  geneve       - configure a geneve server
  gopher       - configure a gopher server
  gre          - configure a gre server
  gtp          - configure a gtp server
  honeypot     - configure a honeypot server
  http         - configure a http server
  irc          - configure an irc server
  iscsi        - configure an iscsi server
  l2f          - configure a l2f server
  l2tp2        - configure a l2tp v2 server
  l2tp3        - configure a l2tp v3 server
  loadbalancer - configure a loadbalancer server
  lpd          - configure a lpd server
  modem        - configure a modem server
  mplsip       - configure a mplsip server
  mplsudp      - configure a mplsudp server
  multiplexer  - configure a multiplexer server
  netflow      - configure an netflow server
  nrpe         - configure a nrpe server
  ntp          - configure a ntp server
  openflow     - configure an openflow server
  p4lang       - configure an p4lang server
  pcep         - configure a pcep server
  pckodtls     - configure a pckodtls server
  pckotcp      - configure a pckotcp server
  pckotxt      - configure a pckotxt server
  pckoudp      - configure a pckoudp server
  pop3         - configure a pop3 server
  pptp         - configure a pptp server
  prometheus   - configure a prometheus server
  quote        - configure a quote server
  radius       - configure a radius server
  rfb          - configure a rfb server
  rpki         - configure a rpki server
  sip          - configure a sip server
  smtp         - configure a smtp server
  snmp         - configure a snmp server
  socks        - configure a socks server
  streamingmdt - configure a streaming telemetry server
  stun         - configure a stun server
  syslog       - configure a syslog server
  tacacs       - configure a tacacs server
  telnet       - configure a telnet server
  tftp         - configure a tftp server
  time         - configure a time server
  udpfwd       - configure an udp forwarder server
  udptn        - configure an udptn server
  upnpfwd      - configure an upnp forwarder server
  upnphub      - configure an upnp hub server
  voice        - configure a voice server
  vxlan        - configure a vxlan server

yourname(cfg)#server                              
...

In order to exit the sandbox session use the following escape sequence: Ctrl-c + Ctrl-x









This is a new category of article that falls under "RARE software architecture" special blog series. As its name implies, it deals with topics related to RARE/freeRouter software / Monitoring.

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

In Greek mythology, Prometheus is a Titan that is credited mankind creation by stealing Fire from Gods and by giving it to human. In the RARE context, Prometheus is a the software from prometheus.io project. It became very popular in the IT industry as it is very simple to implement/configure while providing a great number of metrics without impacting application performance. It is heavily used in microservices environment such as docker and Kubernetes. The mythological reference gives us an indication of how Prometheus is operating. At a constant rate, Prometheus metric collector or server is stealing metrics from Prometheus agent. All the stolen metrics are then consolidated in Time Series database ready to be poured to a queueing system for proper visualization. 

Before going further, allow me a brief digression by sharing with you a small anecdote that leds to this ongoing work related to network monitoring for RARE. As mentioned previously, our focus is to elaborate RARE/freeRouter solution the possibility to be monitored in an operational environment. In that context, we started with the implementation of a lightweight SNMP stack that provided relevant result via SNMP tools like LibreNMS. This is great for organisation that wouldn’t want invest time on anything but SNMP.

However, we felt a lack of flexibility due to SNMP inherent structure and we needed more versatile and instant monitoring capabilities.  More importantly the need to export infinite metric type from Control Plane in a more flexible way arise. How metrics such as: Number of IPv4/IPv6 routes, IPv4 BGP prefix, IPv6 BGP prefix platform JVM memory etc. could be shared without too much hassle ?

After some internal discussion, I just said: "I’m not a monitoring expert but we have tools like ELK and PROMETHEUS and GRAFANA in NMaaS catalog … Shouldn’t we consider use this ?"

The answer was: « Let’s give it a try and fire up a Prometheus and Grafana instance from NMaaS platform !»

Some hacking at the control plane code level were initiated, after few hours freeRouter lead developer came up with a solution and said: Let me introduce you "freeRouter prometheus agent »

And thanks to the great support of NMaaS team, in few minutes and some point and clicks (it took longer than expected as I’m not good with GUI) we were able to test this agent.

Why is it important you might say ? It is just that with prometheus simplicity and low resource overhead with have full control plane metrics visibility !

As a side note this is not a replacement for INT/Telemetry/Netflow/IPFIX that provide different type of data that are to at the same scale…
People with INT/TELEMETRY/NETFLOW/IPFIX are talking about a "data lake" or "data deluge". Which is correct, if you think about the complexity of resolving a  gigantic producer/consumer data problem. This needs the relevant IT infrastructure in order to process all of the data provided by these protocol at the NREN scale.

While in our case, we are just focusing on exposing CONTROL PLANE METRICS at the network element level. We simply monitor and ensure a router operation by using prometheus metrics

Note

While he above might be true, the number of metrics exported from a prometheus target can be very high. Fine tuning might be necessary in order to make sure that all metrics are really necessary for network monitoring purpose. This explosion of metrics exposure can add unnecessary workload at the control plane level. 

Again, kudos to NMaaS team that made this happen so that we could test this on the P4 LAB with — ZERO — effort.

Article objective

In this article, we will present freeRouter and Prometheus integration and as an example we will implement one of the 22 grafana dashboard that we developed and published here. In the rest of the article we will assume that you are a running one or more freeRouter nodes.

Diagram

[ #001 ] - Cookbook

 Configure a Prometheus server

The first step is to implement a prometheus server. Using NMaaS it is pretty instantaneous. However, if you plan to deploy prometheus in an other platform just follow the installation guide here.

Once deployed you can push the following prometheus.yaml config:

prometheus.yaml
global:
  scrape_interval: 15s
  evaluation_interval: 30s
alerting:
  alertmanagers:
    - static_configs:
      - targets:
rule_files:
scrape_configs:
  - job_name: 'router'
    metrics_path: /metrics
    scrape_interval: 15s
    static_configs:
    - targets: ['192.168.0.1:9001','192.168.0.2:9001']
      labels:

In this configuration we assume that we have 2 freeRouters that are configured as above (192.168.0.1:9001 and 192.168.0.2:9001) in prometheus worls these are called targets:

  • each target are interrogated or "scraped" very "scrap_interval" which is 15s here
  • the main job name is called; "router"
  • metrics_path is: "/metrics" so the scraped URL is: "http://192.168.0.1:9001/metrics

Note that this had to be deployed only once for all of your routers. However, each time you'd like to add a new router, you have to add a new target in the "targets" YAML list.

 Configure Prometheus FreeRouter control plane

In this example let's focus our interest interface metrics. Please note that this configuration should be deployed on each freeRouter and connectivity should be available between all targets and the prometheus server.

The objective is to tell freeRouter control plane to expose hardware and software counter interface metric. In order to do this just copy/paste the stanza here below via freeRouter CLI:

prometheus interface metric configuration
!
server prometheus <PROMETHEUS_SERVER_NAME>
 metric inthw command sho inter hwsumm
 metric inthw prepend iface_hw_byte_
 metric inthw name 0 ifc=
 metric inthw replace \. _
 metric inthw column 1 name st
 metric inthw column 1 replace admin -1
 metric inthw column 1 replace down 0
 metric inthw column 1 replace up 1
 metric inthw column 2 name tx
 metric inthw column 3 name rx
 metric inthw column 4 name dr

 metric intsw command sho inter summ
 metric intsw prepend iface_sw_byte_
 metric intsw name 0 ifc=
 metric intsw replace \. _
 metric intsw column 1 name st
 metric intsw column 1 replace admin -1
 metric intsw column 1 replace down 0
 metric intsw column 1 replace up 1
 metric intsw column 2 name tx
 metric intsw column 3 name rx
 metric intsw column 4 name dr
 vrf <VRF_NAME>
 exit
!

So this basically means:

  • From freeRouter CLI, issue the following command:
prometheus interface metric configuration
sho inter hwsumm
interface   state  tx          rx          drop
hairpin41   up     67404       0           0
hairpin42   up     153134      0           0
sdn1        up     412319805   1057514903  1152305
sdn2        up     1038840147  407307558   202
sdn3        admin  0           0           0
sdn4        admin  0           0           0
sdn5        admin  0           0           0
sdn6        admin  0           0           0
sdn998      up     9154        0           0
sdn999      up     199178      262939      0
tunnel1965  up     0           9122896     0 
  • prepend to the metric name: "iface_hw_byte_"
  • column 0 will have prometheus label ifc=
  • replace all dots "." by "_" . (so interface bundle1.123 will become bundle1_123)
  • column defines a metric name  "iface_hw_byte_" concatenated to "st" => "iface_hw_byte_st" which is essentially interface status
  • if column 1 "state" value is admin/down/up we associate value -1/0/1 
  • column defines a metric name  "iface_hw_byte_" concatenated to "tx" => "iface_hw_byte_tx" which is essentially interface bytes transmitted counter 
  • column defines a metric name  "iface_hw_byte_" concatenated to "rx" => "iface_hw_byte_rx" which is essentially interface bytes received counter 
  • column defines a metric name  "iface_hw_byte_" concatenated to "dr" => "iface_hw_byte_dr" which is essentially interface bytes dropped counter 

And if you followed this correctly, we are repeating these lines for software interface counter metric.

Tip

You can view Prometheus configuration for various Grafana dashboard here. Feel free to study these Prometheus configuration and activate them as you see fit depending on your requirements. The set of dashboard is not exhaustive and is by no means absolute. Feel free to submit additional dashboard ! We would gladly add them in the current list of freeRouter Dashboard.

Note

After this definition a freeRouter level you should have:

4 metrics related to hardware counters 

  • iface_hw_byte_st
  • iface_hw_byte_tx
  • iface_hw_byte_rx
  • iface_hw_byte_dr

4 metrics related to software counters

  • iface_sw_byte_st
  • iface_sw_byte_tx
  • iface_sw_byte_rx
  • iface_sw_byte_dr

Which is a total of 8 metrics

Tip

From that point you can check via prometheus console:

check the "Targets" menu drop down selection

From that point you should be able to use PromQL query filed in order to check that you can retrieve the metrics we defined above.

 Grafana configuration

For metric visualisation, we will use Grafana. Therefore:

  • install Grafana from official web site.
  • Once installed configure Prometheus as Grafana data source:

 

  • fill in all the prometheus server information

  • check the the data source is defined correctly by clicking the "Save & test" button

At that point your Grafana and Prometheus are correctly binded.

  • now you need to import "RARE/freeRouter interface bytes" dashboard

  • download freeRouter interface bytes dashboard here 


  • import the dashboard via ID or simply download JSON or use JSON panel

And Voila ! 

In order to immediately see the graph zoom in to 5m period with a refresh of 5s and you should see automagically the interface bytes TX/RX on all interface for each targets.

Discussion

This example related to interface metrics is universal, as the metrics at freeRouter level are yielded through a generic CLI command:

  • "show interface hwsummary"
  • or "show interface swsummary".

However some metrics cannot be retrieved by generic interface. Some metrics will be tied to specificities of your network. These can be the AS number, IGP process name, VRF name etc.

Let me give you a couple of examples:

 the metrics below assume that you have deployed a link state IGP called: "isis 1" 

But your network context you could have arbitrary deployed "isis 2200". (2200 is RENATER AS number) 

prometheus interface metric configuration
 metric lsigp4int command sho ipv4 isis 1 interface
 metric lsigp4int prepend lsigp4_iface_
 metric lsigp4int name 0 proto="isis1",ifc=
 metric lsigp4int replace \. _
 metric lsigp4int column 1 name neighbors
 metric lsigp4peer command sho ipv4 isis 1 topology 2
 metric lsigp4peer prepend lsigp4_peers_
 metric lsigp4peer name 0 proto="isis1",node=
 metric lsigp4peer replace \. _
 metric lsigp4peer column 1 name reachable
 metric lsigp4peer column 1 replace false 0
 metric lsigp4peer column 1 replace true 1
 metric lsigp4peer column 6 name neighbors
 metric lsigp4perf command sho ipv4 isis 1 spf 2 | inc reachable|fill|calc|run
 metric lsigp4perf prepend lsigp4_perf_
 metric lsigp4perf labels proto="isis1"
 metric lsigp4perf skip 0
 metric lsigp4perf column 1 name val

 metric lsigp6int command sho ipv6 isis 1 interface
 metric lsigp6int prepend lsigp6_iface_
 metric lsigp6int name 0 proto="isis1",ifc=
 metric lsigp6int replace \. _
 metric lsigp6int column 1 name neighbors
 metric lsigp6peer command sho ipv6 isis 1 topology 2
 metric lsigp6peer name 0 proto="isis1",node=
 metric lsigp6peer prepend lsigp6_peers_
 metric lsigp6peer replace \. _
 metric lsigp6peer column 1 name reachable
 metric lsigp6peer column 1 replace false 0
 metric lsigp6peer column 1 replace true 1
 metric lsigp6peer column 6 name neighbors
 metric lsigp6perf command sho ipv6 isis 1 spf 2 | inc reachable|fill|calc|run
 metric lsigp6perf prepend lsigp6_perf_
 metric lsigp6perf labels proto="isis1"
 metric lsigp6perf skip 0
 metric lsigp6perf column 1 name val
 in the metric below the variable is BGP AS number "65535": 


prometheus interface metric configuration
 metric bgp4peer command sho ipv4 bgp 65535 summ
 metric bgp4peer prepend bgp4_peer_
 metric bgp4peer name 4 peer=
 metric bgp4peer replace \. _
 metric bgp4peer column 1 name learn
 metric bgp4peer column 2 name advert
 metric bgp4peer column 3 name state
 metric bgp4peer column 3 replace false 0
 metric bgp4peer column 3 replace true 1
 metric bgp4perf command sho ipv4 bgp 65535 best | exc last
 metric bgp4perf prepend bgp4_perf_
 metric bgp4perf replace \s _
 metric bgp4perf column 1 name val

 metric bgp6peer command sho ipv6 bgp 65535 summ
 metric bgp6peer prepend bgp6_peer_
 metric bgp6peer name 4 peer=
 metric bgp6peer replace \: _
 metric bgp6peer column 1 name learn
 metric bgp6peer column 2 name advert
 metric bgp6peer column 3 name state
 metric bgp6peer column 3 replace false 0
 metric bgp6peer column 3 replace true 1
 metric bgp6perf command sho ipv6 bgp 65535 best | exc last
 metric bgp6perf prepend bgp6_perf_
 metric bgp6perf replace \s _
 metric bgp6perf column 1 name val 
 Last example with "LDP null" metrics, in this particular case the variable object is the VRF name: "inet"
prometheus interface metric configuration
 metric ldp4nul command sho ipv4 ldp inet nulled-summary
 metric ldp4nul prepend ldp4null_
 metric ldp4nul name 3 ip=
 metric ldp4nul skip 2
 metric ldp4nul replace \. _
 metric ldp4nul column 0 name prefix_learn
 metric ldp4nul column 1 name prefix_advert
 metric ldp4nul column 2 name prefix_nulled

 metric ldp6nul command sho ipv6 ldp inet nulled-summary
 metric ldp6nul prepend ldp6null_
 metric ldp6nul name 3 ip=
 metric ldp6nul skip 2
 metric ldp6nul replace \: _
 metric ldp6nul column 0 name prefix_learn
 metric ldp6nul column 1 name prefix_advert
 metric ldp6nul column 2 name prefix_nulled 


Conclusion

In this 1st article, you were presented :

  • freeRouter/Prometheus integration 
  • How to add a new router in the list of Prometheus target
  • How to integrate a RARE/freeRouter Grafana Dashboard. (Feel free to adapt the other available dashboard query to your context !)

Final words

In Prometheus philosophy, normally the user should do only the minimum of tweaking regarding configuration. Ultimately, he should be only be able to enable a metric or simply disable it if the scrape cost is too high. However in freeRouter/Prometheus integration process, you see that some metric are issued using specific $variable (VRF, BGP/IGP process number ...) Which makes impossible to maintain this universality. However, from the network operator point of view this should not be a showstopper. On the contrary, it is a powerful choice to be able to alter these command via $variables.

Remember in freeRouter philosophy you can have multiple VRF, multiple IGP and multiple BGP process number ! (Which is not the case for all routing platform)

Last but not least, this Prometheus agent was developed quickly because of one reason, all the objects  at the control plane level were already well structured in table form as previously described in this article. So implementing this table row/column logic in order to derive a prometheus metric was technically possible without too much hassle.





This is a new article for the blog serie called "RARE Day One". Today we will explore one of freeRouter features meant to fine tune terminal user environment and behaviour in order to best match your taste/preferences.

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

We all have our habits that are inherited from our past experience. Some people are used to IOS, IOS-XR, NX-OX, IOS-XE others prefer Junos etc. Using freeRouter provides a different user experience. Some feature such as show/view/watch/differ diagnosis commands are pretty unique to freeRouter. However, freeRouter have some cards in its sleeves in order to provide you a familiar experience.

Article objective

In this article, we will focus on these features:

  • monitor
  • length
  • width
  • spacetab
  • tablemode
  • timestamps
  • colorized

Basically these commands are accessed through freeRouter user mode. If you need to use them from config mode, please use the "do" keyword.

[ #003 ] - "monitor/length/width/spacetab/tablemode/timestamps/colorized"

 monitor

If you are familiar with Cisco operating system you will feel at home with "terminal monitor" mode. This mode is usually used in combination with "debug" diagnosis command and actually redirect console amd debug output also in your current terminal session. (VTY in Cisco language)

Let's assume we want to debug IPv4 BGP

show BGP configuration from running config
r1#debug proto bgp ?                                                      
  computation - computation events
  event       - table events
  full        - full events
  incremental - incremental events
  traffic     - interface packets

Let's activate BGP debug event

Check BGP IPv4 peers status in VRF dn42
r1#debug rtr.rtrBgpSpeak.packRecv:rtrBgpSpeak.java:1134 got update from 172.23.215.177
debug rtr.rtrBgp.routerCreateComputed:rtrBgp.java:1902 create table
debug rtr.rtrBgpSpeak.packSend:rtrBgpSpeak.java:1080 sending update to 172.23.215.177
r1#                                                                       
r1#debug rtr.rtrBgpSpeak.packRecv:rtrBgpSpeak.java:1134 got keepalive from 172.23.215.177
debug rtr.rtrBgpSpeak.packSend:rtrBgpSpeak.java:1080 sending keepalive to 172.23.215.177
debug rtr.rtrBgpSpeak.packRecv:rtrBgpSpeak.java:1134 got update from fd40:cc1e:c0de::151
debug rtr.rtrBgp.routerCreateComputed:rtrBgp.java:1902 create table
debug rtr.rtrBgpSpeak.packSend:rtrBgpSpeak.java:1080 sending update to fd40:cc1e:c0de::151
debug rtr.rtrBgpSpeak.packRecv:rtrBgpSpeak.java:1134 got update from fd40:cc1e:c0de::151
debug rtr.rtrBgp.routerCreateComputed:rtrBgp.java:1902 create table
debug rtr.rtrBgpSpeak.packRecv:rtrBgpSpeak.java:1134 got update from fd40:cc1e:c0de::151
debug rtr.rtrBgp.routerCreateComputed:rtrBgp.java:1902 create table
debug rtr.rtrBgpSpeak.packSend:rtrBgpSpeak.java:1080 sending update to fd40:cc1e:c0de::151
debug rtr.rtrBgpSpeak.packRecv:rtrBgpSpeak.java:1134 got update from fd40:cc1e:c0de::151
debug rtr.rtrBgp.routerCreateComputed:rtrBgp.java:1902 create table
...

in order to cancel debug:

undebug all (or specific debug)
r1#un all
...

in order to stop console output from your terminal session:

termonal no monitor
r1#term no mon
...

Note

Similar to Cisco gear, "debug" can be very chatty. Therefore, be ready to issue "term no mon" or better log debug into a file for further off-line forensics.

 terminal length

In my previous examples, the output of "show ipv4 bgp 42 unicast database" command could not fit my window as the output as tomany lines. "terminal length" can be used to alter the number of  lines of the terminal.

Check hardware traffic counters
r1##terminal length ?                                                      
  <num> - height in lines

r1#terminal length
...

Note

"terminal length" can have no effect if you are using a more sophisticated terminal. However, this will have a visible impact on view/display/differ buffer.

 terminal spacetab

"terminal spacetab" is specifically decidated to Junos user. It basically does the same effect as the <TAB> key, but it add also contextual completion with the <SPACE>

Check hardware traffic counters
r1#terminal spacetab ?                                                    
  <cr>

r1#terminal spacetab                                                      
...

Note

"terminal spacetab" does not remove <TAB> key behaviour.

 terminal tablemode

"terminal tablemode" provide pre-formatted table output. 

"terminal tablemode" available format
mjolnir#terminal tablemode ?                                                   
  csv    - select csv mode
  fancy  - select fancy mode
  html   - select html mode
  normal - select normal mode
  raw    - select raw mode
  table  - select table mode
r1#terminal spacetab ?                                                    
  <cr>

Let's select "fancy"

Fancy mode
r1#show ipv4 bgp 42 summary                                               
 |~~~~~~~~~~~~|~~~~~~~|~~~~~~|~~~~~~~|~~~~~~~~~~~~~~~~|~~~~~~~~~~|
 | as         | learn | done | ready | neighbor       | uptime   |
 |------------|-------|------|-------|----------------|----------|
 | 4242421955 | 516   | 517  | true  | 172.23.215.177 | 01:16:23 |
 |____________|_______|______|_______|________________|__________|

Note

Feel free to play all format proposed by "terminal tablemode". This is pretty useful when you have to prepare some report related to the VPN or network you are currently managing.

 terminal timestamps

"terminal timestamps" will simply prepend command timestamps.

Fancy mode
r1#show ipv4 bgp 42 summary                                               
2020-09-30 09:46:32
 |~~~~~~~~~~~~|~~~~~~~|~~~~~~|~~~~~~~|~~~~~~~~~~~~~~~~|~~~~~~~~~~|
 | as         | learn | done | ready | neighbor       | uptime   |
 |------------|-------|------|-------|----------------|----------|
 | 4242421955 | 516   | 517  | true  | 172.23.215.177 | 01:18:45 |
 |____________|_______|______|_______|________________|__________|

Note

As you can see, you can stack these modes. Here we activated "terminal tablemode + timestamps"

 terminal colorized

"terminal colorized" will simply  color  your prompt

 Make terminal <mode> permanent

"terminal <mode>" is specific to your current session. If you want a persistent behaviour you would need to activate these features from the  "server telnet" stanza. Which as its name wrongly implies, is not about configuring a telnet server only. From this stanza you'll able to configure any type of server dedicated to terminal connection. (SSH)

Fancy mode
r1#(cfg-server)#exec ?                                                     
  authorization - set authorization
  autocommand   - set automatic command
  autohangup    - disconnect user after autocommand
  bye           - set goodbye message
  colorized     - enable colorization
  height        - set height of terminal
  interface     - set interface to use for framing
  logging       - enable logging
  privilege     - set default privilege
  ready         - set ready message
  spacetab      - enable space as tab
  tablemode     - set table mode
  timeout       - set timeout value
  timestamp     - enable timestamps
  welcome       - set welcome message
  width         - number of columns

Note

This "server telnet" section will provide you lots of possibility to fine tune your terminal access !  Feel free to use them in order to feel at home !

Discussion

monitor/length/width/spacetab/tablemode/timestamps/colorized is a set of feature meant to ease your experience with freeRouter in mimic'ing well know behaviour and proposing you additional convenient features. One main behaviour is that all command issue from the CLI is instantly taken into account. 

Conclusion

In this 3rd article:

  • We presented freeRouter monitor/length/width/spacetab/tablemode/timestamps/colorized terminal customization command
  • These are very useful if you come from Cisco or Junos world as it mimic some termnal behaviour.

Final words

As said, these are terminal commands are not specific to freeRouter. Some behaviour are mimic'ed from IOS and Junos. Anyway, these have been developed for one purpose:

"Make network engineers feel at ease and provide then an enjoyable operation experience "

Feel free to try and use them according to your environment taste!

Last but not least, you can play with these different mode from this sandbox:

type "telnet dl.nop.hu" in a terminal and choose "1"
telnet dl.nop.hu
Trying 193.224.23.5...
Connected to dl.nop.hu.
Escape character is '^]'.
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXX XXXXX XXX    XXX     XXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX  XXXX XX XXXX XX XXXX XX XX XX XXXX XXXXX/~~~~~~\XXXXXX
XXXX X XXX XX XXXX XX XXXX XX XX XX XXXX XXXX| player |XXXXX
XXXX XX XX XX XXXX XX     XXX    XX XXXX XXXXX\______/XXXXXX
XXXX XXX X XX XXXX XX XXXXXXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX XXXX  XX XXXX XX XXXXXXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX XXXXX XXX    XXX XXX XXX XX XXX    XXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
welcome
line ready
menu lab:
# - reboot router1
$ - reboot router2
% - reboot router3
1 - connect to router1
2 - connect to router2
3 - connect to router3
^ - rebuild routers
l - connect to lg.nop.dn42
x - exit
choose:1 - attach vdc lab1 

welcome
line ready
yourname#terminal ?                                                            
  colorized  - sending to ansi terminal
  length     - set terminal length
  monitor    - log to this terminal
  no         - negate a parameter
  spacetab   - treat space as tabulator
  tablemode  - select table formatting mode
  timestamps - put time before each executed command
  width      - set terminal width

yourname#terminal                            
...

In order to exit the sandbox session use the following escape sequence: Ctrl-c + Ctrl-x








This is a new article for the blog serie called "RARE Day One". Today we will explore one of freeRouter killer feature that will make your life easier during your day to day operation: freeRouter assisted diagnosis command.

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

As previously mentioned in the precedent article, when you log into a network equipment such as a router, you tend to have some automatic reflex. You usually:

  • Check router configuration: show run or sh conf
  • Check ipv4 / ipv6 / or LFIB forwarding table
  • So you basically issue diagnosis, troubleshooting command
  • An then you want to configure the router

Article objective

In this article, we will focus on the 3rd bullet point and will present you freeRouter available diagnosis command. They are grouped into 5 categories:

  • show 
  • view
  • watch
  • display
  • differ

Basically these commands are accessed through freeRouter user mode. If you need to use them from config mode, please use the "do" keyword.

[ #002 ] - "show/view/watch/display/differ"

 show

You would mostly be familiar with the "show" command. It is very good and can basically be used to get output from control plane object. Most of the time this can be used against static object like config.

Let's assume that I would like to get BGP config from my home router:

show BGP configuration from running config
show running-config bgp4                                               
router bgp4 42                                                                 
 vrf dn42                                                                      
 local-as 4242421975                                                           
 router-id 172.22.105.65                                                       
 address-family unicast multicast other flowspec vpnuni vpnmlt vpnflw ovpnuni ovpnmlt ovpnflw vpls mspw evpn mdt srte mvpn omvpn
 neighbor 172.23.215.177 remote-as 4242421955                                  
 neighbor 172.23.215.177 description NOP.DN42                                  
 neighbor 172.23.215.177 local-as 4242421975                                   
 neighbor 172.23.215.177 address-family unicast multicast other flowspec vpnuni vpnmlt vpnflw ovpnuni ovpnmlt ovpnflw vpls mspw evpn mdt srte mvpn omvpn
 neighbor 172.23.215.177 distance 20                                           
 justadvert loopback42                                                         
 exit                      

But I can also check the status of BGP peering into VRF dn42

Check BGP IPv4 peers status in VRF dn42
show ipv4 bgp 42 summary                                               
as          learn  done  ready  neighbor        uptime
4242421955  517    518   true   172.23.215.177  00:38:18                      

Check the same BGP peering but now for IPv6

Check BGP IPv6 peers status in VRF dn42
r1#show ipv6 bgp 42 summary                                               
as          learn  done  ready  neighbor             uptime
4242421955  351    352   true   fd40:cc1e:c0de::151  00:40:40
show ipv4 bgp 42 summary                                               
as          learn  done  ready  neighbor        uptime
4242421955  517    518   true   172.23.215.177  00:38:18                      

Let's see some BGP prefix received in VRF dn42 bgp table:

so my screen is too small for all the IPv6 BGP prefix into DN42 VRF

As a last example, something we usually do as network operators is to check ongoing interface traffic level:

Check interface traffic level (received/transmitted) )
r1#sh int sdn1                                                            
sdn1 is up (since 09:41:21, 2 changes)
 description: mjolnir@LAN1[01:00.0]
 type is sdn, hwaddr=003b.7671.764f, mtu=1500, bw=8000kbps, vrf=inet
 ip4 address=192.168.0.90/24, netmask=255.255.255.0, ifcid=10013
 ip6 address=2a01:e0a:159:2850::666/64, netmask=ffff:ffff:ffff:ffff::, ifcid=10013
 received 52013 packets (17638316 bytes) dropped 5 packets (448 bytes)
 transmitted 80765 packets (15101696 bytes) promisc=false macsec=false

r1#sh int sdn1                                                            
sdn1 is up (since 09:41:22, 2 changes)
 description: mjolnir@LAN1[01:00.0]
 type is sdn, hwaddr=003b.7671.764f, mtu=1500, bw=8000kbps, vrf=inet
 ip4 address=192.168.0.90/24, netmask=255.255.255.0, ifcid=10013
 ip6 address=2a01:e0a:159:2850::666/64, netmask=ffff:ffff:ffff:ffff::, ifcid=10013
 received 52013 packets (17638316 bytes) dropped 5 packets (448 bytes)
 transmitted 80766 packets (15101778 bytes) promisc=false macsec=false

r1#sh int sdn1                                                            
sdn1 is up (since 09:41:24, 2 changes)
 description: mjolnir@LAN1[01:00.0]
 type is sdn, hwaddr=003b.7671.764f, mtu=1500, bw=8000kbps, vrf=inet
 ip4 address=192.168.0.90/24, netmask=255.255.255.0, ifcid=10013
 ip6 address=2a01:e0a:159:2850::666/64, netmask=ffff:ffff:ffff:ffff::, ifcid=10013
 received 52015 packets (17638418 bytes) dropped 5 packets (448 bytes)
 transmitted 80766 packets (15101778 bytes) promisc=false macsec=false
                        

In the last example we repeatedly issue the "sh int sdn1" command and try to see if TX/RX packets counters increment or not.

This command can be improved in order to be less chatty:

Check interface traffic level (received/transmitted) )
r1#sh int sdn1 | i received|transmitted                            
 received 52256 packets (17681204 bytes) dropped 5 packets (448 bytes)
 transmitted 81130 packets (15162642 bytes) promisc=false macsec=false

r1#sh int sdn1 | i received|transmitted                            
 received 52256 packets (17681204 bytes) dropped 5 packets (448 bytes)
 transmitted 81130 packets (15162642 bytes) promisc=false macsec=false

r1#sh int sdn1 | i received|transmitted                            
 received 52260 packets (17681496 bytes) dropped 5 packets (448 bytes)
 transmitted 81132 packets (15162790 bytes) promisc=false macsec=false

Same goes if want want interface traffic for all interface

Check interface traffic level (received/transmitted) )
show interfaces summary                                                
interface   state  tx        rx        drop
loopback0   up     65856     0         0
loopback42  up     65856     0         0
ethernet0   up     31071917  33183183  0
hairpin41   up     85806     85552     0
hairpin42   up     85806     85552     0
sdn1        up     15200591  17703953  448
sdn2        up     15563546  8000994   794
sdn3        admin  0         0         0
sdn4        admin  0         0         0
sdn5        admin  0         0         0
sdn6        admin  0         0         0
sdn998      up     5850      0         0
sdn999      up     23268     18666     0
tunnel1965  up     5222281   7124950   0

Above was to check interface status related to software switched packet. What if I want to check hardware switched packet counters switched by P4 or DPDK ?

Check interface traffic level (received/transmitted) )
show interfaces hwsummary                                              
interface   state  tx         rx         drop
hairpin41   up     0          0          0
hairpin42   up     0          0          0
sdn1        up     317902736  590402538  1162971
sdn2        up     574923844  310497399  203
sdn3        admin  0          0          0
sdn4        admin  0          0          0
sdn5        admin  0          0          0
sdn6        admin  0          0          0
sdn998      up     9062       0          0
sdn999      up     103804     64470      0
tunnel1965  up     0          1301312    0

Note

As a network operator, the "show" command is your best friend, your wingman. Just explore now from freeRouter CLI using "show ?" and you'll understand the amazing list of diagnosis command available.

 view

In my previous examples, the output of "show ipv4 bgp 42 unicast database" command could not fit my window. Say hello to "view" keyword then !  

Let's now try to get hardware counters as above:

Check hardware traffic counters
r1#view ipv4 bgp 42 unicast database
...

Then you'll be able to see READ-ONLY text buffer where you can navigate and check the output that are beyond boundaries of your screen !

Note

"view" is similar to "show" but it will let you deal with a fixed buffer. "view" buffer won't be refreshed.

 watch

As mentioned above, "show" gives you diagnosis instant photo of a control plane object. In order to see counter increment, you'd have to issue "show" repeatedly. In order to avoid that, let me introduce you the "watch" command

Let's now try to get hardware counters as above:

Check hardware traffic counters
r1#watch interfaces hwsummary
...

It will clear the terminal session and gives you the same outout as above but with counter updated in a regular basis

So in this example you'll see a live output with counter incrementing. In the screenshot it is not noticeable, but in real life this is bluffing. See watch interface pretty much like Junos "monitor" keyword.

So needless to say that "watch" is applicable to every control plane object such as BGP:

Amazing, don't you think ?

 display

In my previous examples, the output of "show ipv4 bgp 42 unicast database" command could not fit my window. Say hello to "display" keyword then !  

Display BGP prefix from dn42 VRF
r1#display ipv4 bgp 42 unicast database
...

Then you'll be able to see READ-ONLY text buffer where you can navigate and check the output that are beyond boundaries of your screen !

As a side note, you can benefit from online help by pressing <f1>

You can press Ctrl+q in order to exit the editor. As the viewer is a READ-ONLY buffer

Note

Use "display" for output that have output that does not fit into your screen. "display" shows a buffer that is auto-refreshed similar to "watch". But instead the output is thrown into a buffer where you can navigate. But display, very useful to diagnoses object such as huge:

  • ACL
  • prefix-list
  • route policy list
  • route-map

As opposed to "view", "display" proposes an auto-refresh version of the buffer ! 

 differ

Last but not list. "differ" , this will split the window in 2 buffers reflecting the same output but with different version and it it signal line lines that have changed. 

Check BGP best path computation for BGP process 42
r1#diff ipv4 bgp 42 bestpath
...

With this view you can easily spot the differences between 2 advertisements interval.

To be honest, when i used this feature for the first time I was totally stumbled and said: Waouw ...

Simply amazing ... 

Discussion

show/view/watch/display/differ is pretty unique to freeRouter, and is really meant to provide you the best user experience as a network operator ! These command have proven to be helpful, especially if you deal with huge feed. However, be careful when you are working with very big output such BGP full feed. This won't crash the router of course as we used to when we issued "debug ip packet" but it will for sure imply a high CPU usage due to regular refresh at the control plane level.

Conclusion

In this 2nd article:

  • We presented freeRouter show/watch/display/differ diagnisis command
  • These are very useful when you have to deal with huge command output buffer.

Final words

As said, these are diagnosis commands are specific to freeRouter. 2 decades of know how and network experience have been pushed into these feature codes. These have been developed for one purpose:

"Provide a unique operation experience to network engineers"

Feel free to try and use them according to your environment taste!

Last but not least, you can play with these different mode from this sandbox:

type "ssh dl.nop.hu" in a terminal (any user/pass will do) and choose "l"
ssh dl.nop.hu -l random_user                                                                                                                              
Warning: Permanently added 'dl.nop.hu,193.224.23.5' (RSA) to the list of known hosts.
random_user@dl.nop.hu's password: 
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXX XXXXX XXX    XXX     XXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX  XXXX XX XXXX XX XXXX XX XX XX XXXX XXXXX/~~~~~~\XXXXXX
XXXX X XXX XX XXXX XX XXXX XX XX XX XXXX XXXX| player |XXXXX
XXXX XX XX XX XXXX XX     XXX    XX XXXX XXXXX\______/XXXXXX
XXXX XXX X XX XXXX XX XXXXXXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX XXXX  XX XXXX XX XXXXXXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX XXXXX XXX    XXX XXX XXX XX XXX    XXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
welcome
line ready
menu lab:
# - reboot router1
$ - reboot router2
% - reboot router3
1 - connect to router1
2 - connect to router2
3 - connect to router3
^ - rebuild routers
l - connect to lg.nop.dn42
x - exit
choose:l - telnet 172.23.199.110 23 /telnet 
 - connecting to 172.23.199.110 23
 - securing connection

hi there!
try the following:
  show ipv4 route dn42
  show ipv6 route dn42
  show ipv4 bgp 65535 vpnuni summary
  show ipv6 bgp 65535 vpnuni summary
  show ipv4 bgp 65535 vpnuni database
  show ipv6 bgp 65535 vpnuni database
  show ipv4 bgp 65535 vpnuni allroute <prefix> 65535:42
  show ipv6 bgp 65535 vpnuni allroute <prefix> 65535:42
  show ipv4 logger 42 flapstat 10
  show ipv6 logger 42 flapstat 10
  show ipv4 bgp 65535 vpnuni flapstat 10
  show ipv6 bgp 65535 vpnuni flapstat 10
  show ipv4 bgp 65535 vpnuni flappath <prefix> 65535:42
  show ipv6 bgp 65535 vpnuni flappath <prefix> 65535:42
have fun!
mc36
welcome
line ready
player-dn42>                                                                   
player-dn42>                   
...

Then issue a "diff" command:

differ example with BGP command
player-dn42>diff ipv4 bgp 65535 vpnuni database 10.11.160.0/20 65535:42
...

You'll be rewarded by this diff out related to the command which means:

"show me the prefix status of 10.11.160.0/20 within BGP process 65535 with rd: 65535:42"

After a quick look at VRF definition indicates that rd 65535:42 is tied to VRF dn42:

Check vrf list on router
player-dn42>sh start vrf                                                       
vrf definition dn42
 rd 65535:42
 rt-import 65535:42
 rt-export 65535:42
 source4route all
 source6route all
 mdt4
 mdt6
 exit
vrf definition rtbh
 rd 65535:666
 rt-import 65535:666
 rt-export 65535:666
 exit
vrf definition vpn
 rd 65535:1
 rt-import 65535:1
 rt-export 65535:1
 mdt4
 mdt6
 exit
...

In order to exit the sandbox session use the following escape sequence: Ctrl-c + Ctrl-x







This is a special blog series called "RARE Day One". I've always been a huge Cisco and JUNIPER fans, Cisco has unparalleled documentation and I really like JUNIPER "Day One" or "This Week" booklets. Similar to JUNIPER approach RARE "Day One" articles are dealing with essential topics that you need to get familiar with and that will become handy during your "RARE-freeRouter"-FU practices !  

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

Even in the era of zero touch configuration where everything can be modelled by YANG and automated by Ansible, CLI configuration mode is essential and will take a special important place into network engineers' heart.

Any network engineer in the room who never issued this command ?

Mythical "configure terminal" command
conf t
...

Article objective

In this article, we will present you freeRouter available configuration mode. This is an essential article as it will help you in your potential daily operation task. 

Diagram

[ #001 ] - "configure <mode>"

 configure terminal

When you log into a network equipment such as a router, you tend to have some automatic reflex. You usually:

  • Check router configuration: show run or sh conf
  • Check ipv4 / ipv6 / or LFIB forwarding table
  • An then you want to configure the router

Let's assume you want to configure interface sdn3 description:

Mythical "configure terminal" command
r1#sh run sdn3                                                            
interface sdn3
 description r1@LAN3[05:00.0]
 mtu 1500
 macaddr 007b.0c15.1e0c
 shutdown
 no log-link-change
 exit

...

r1#conf t
r1(cfg)#                                                                  
r1(cfg)#int sdn3                                                          
r1(cfg-if)#                                                               
r1(cfg)#int sdn3                                                          
r1(cfg-if)#description Hello Workd SDN3                                
r1(cfg-if)#      

As you would notice, configuring these from "config terminal" prompt has an immediate effect. Please note you can issue "show" command from config mode using the "do" keyword :

issue "show" command in configuration mode using "do" keywork
r1#conf t
r1(cfg)#                                                                  
r1(cfg)#int sdn3                                                          
r1(cfg-if)#                                                               
r1(cfg)#int sdn3                                                          
r1(cfg-if)#description Hello Workd SDN3                       
r1(cfg-if)#   
r1(cfg-if)#do sh run sdn3                                                 
interface sdn3
 description Hello Workd SDN3
 mtu 1500
 macaddr 007b.0c15.1e0c
 shutdown
 no log-link-change
 exit
   

At that point you have a running-config in router memory and you have a startup-config written into the freeRouter flash. In order to see the difference:

issue "show" command in configuration mode
...

r1(cfg-if)#do sh config                                                   
interface sdn3
 no description old_descrption
 description Hello Workd SDN3
 exit
...
r1(cfg-if)# end                                                           
r1#show config-differences                                                
interface sdn3
 no description old_descrption
 description Hello Workd SDN3
 exit   

Notice the use of "end" primitive in order to end configuration mode and revert to user mode. In the example we used shortcut command name:

  • sh config
  • show config-differences

So basically this command will show you the difference between running-config and startup-config. This is similar to Junos: show | compare except that in this context this a comparison between running and startup config.

In this case it just delete the current description and replace it by the new one.

Once you are happy you can write the running-config into the startup-config:

issue "show" command in configuration mode
...
r1#wr                                                                     
% success
r1#sh conf                                                                

r1#           

You observe that show config-differences has no relevant output. running-config is aligned to startup-config !

Note

This is the most intuitive and recommended way to start learning freeRouter as from this interactive mode, you'll benefit from the contextual help that can be triggered by '?'. In this way you'll even be able to discover new freeRouter feature yourself ! This piece of software holds a tremendous amount of secret functionality. In the output below we just check which control plane can be activated ...

issue "show" command in configuration mode
...
r1(cfg)#router ?                                                          
  babel4     - babel routing protocol
  babel6     - babel routing protocol
  bgp4       - border gateway protocol
  bgp6       - border gateway protocol
  blackhole4 - blackhole collector
  blackhole6 - blackhole collector
  deaggr4    - deaggregate creator
  deaggr6    - deaggregate creator
  download4  - route download
  download6  - route download
  eigrp4     - enhanced interior gateway routing protocol
  eigrp6     - enhanced interior gateway routing protocol
  flowspec4  - flowspec to flowspec rewriter
  flowspec6  - flowspec to flowspec rewriter
  isis4      - intermediate system intermediate system
  isis6      - intermediate system intermediate system
  logger4    - route logger
  logger6    - route logger
  lsrp4      - link state routing protocol
  lsrp6      - link state routing protocol
  mobile4    - mobile route creator
  mobile6    - mobile route creator
  msdp4      - multicast source discovery protocol
  msdp6      - multicast source discovery protocol
  olsr4      - optimized link state routing protocol
  olsr6      - optimized link state routing protocol
  ospf4      - open shortest path first
  ospf6      - open shortest path first
  pvrp4      - path vector routing protocol
  pvrp6      - path vector routing protocol
  rip4       - routing information protocol
  rip6       - routing information protocol
  uni2flow4  - unicast to flowspec converter
  uni2flow6  - unicast to flowspec converter
  uni2multi4 - unicast to multicast converter
  uni2multi6 - unicast to multicast converter         
       
 configure viewer

"configure viewer" is a very interesting mode as it gives you the possibility to review the router configuration from a  viewer inspired from "mcedit" (Norton Midnight Commander) 

configure viewer
r1#configure viewer
...

Then you'll be able to read your configuration from a READ-ONLY text buffer:

As a side note, you can benefit from online help by pressing <f1>

But what if I just want to view a specific object ? Let's find out how to check ONLY BGP configuration @ home:

route addition via freeRouter
r1#configure viewer bgp4
...

So in this case It'll just throw my IPv4 bgp config snippet onto the viewer buffer

Same if I want to only view all interface sdn<x> from the router config:

config viewer <regexp>
r1#configure viewer sdn
...

This is so cool, isn't ?

Note

In big TELCO Service Provider environment, most of the time you have Technical Project Manager that just need to perform some checks related to specific customer VPN deployment. So some times, I received some calls: "Can you please that from customer the HUB site prefix 1.2.3.0/24 is configured and advertised into BGP for customer ABC in VRF YXZ ?" With "configure viewer <object>", the TPM can just check it for himself without bothering you at all ! And this without the fear to alter router configuration by accident.

PS: For that you'll need to create a aaa security config with:

  • proper router aaa security policy with privilege level 1
  • with or without TACACS/RADIUS authentication / authorisation and accounting
  • and apply it to a specific OOBM SSH/telnet server in a specific VRF,

but this is not in the scope of the the present article and it will be the object of further articles.

In SP environment, you should not be surprised to see router configuration that has 100k lines or even more. In these environment, I've seen config with countless amount of VRF, NAT, DLSW, GRE, IPSEC tunnels, BGP peers ...  "config viewer" is a great tools when you want to verify a specific stanza on a per customer or object basis and in bonus without any risk the Provider Edge router configuration.

 configure editor

"configure viewer" gives you the possibility to view the config or some parts of the config in read-only mode. "configure editor" gives you simply the possibility to edit also the specific running-config config stanza.

route addition via freeRouter
r1#configure editor
...

Then you'll be able to edit your configuration from a READ-WRITE text buffer:

As a side note, you can benefit from online help by pressing <f1>

You can press Ctrl+q in order to exit the editor. As you did not change anything it will exit the editor.

But what if I just want to edit a specific object ? Let's find out how to check ONLY BGP configuration @ home:

config editor <regexp>
r1#configure editor bgp4
...

So in this case I'll just throw my IPv4 bgp config onto the editor buffer

In this buffer let's just create a description for BGP neighbor 172.23.215.177.

Now just press Ctrl-q (as per the online help accessible using <f1>). However, freeRouter detect the buffer changed has we added BGP description configuration. Therefore it will ask you if you want to save the buffer change into the running-config and apply it.

At that moment you'll be displayed a small recap of what has been applied. 

Even more cool no ? 

Warning

Even if "config editor" is seducing and seems more appealing especially for beginners. This is absolutely not the case. "configure editor" mode is meant for advanced users who knows freeRouter CLI by heart. Why, you might say ? Just try to edit a gigantic BGP configuration without any contextual help just by writing a textual file and you'll understand the risk behind using "config editor". Therefore it is no recommend to use it against complex control plane object.

Please take note that "config editor" alter the running-configuration directly when you saved the editor buffer !

Note

So what's the point of having this cool feature ? This feature is powerful when it comes to simple control plane object or big repetitive object. This is very practical to use this feature against: ACL / Prefix-List / Route Policy Object / Route Map etc.

  • ACL
  • prefix-list
  • route policy list
  • route-map

but nothing to prevent you to edit BGP stanza if you feel that your freeRouter-fu needs to be challenged (wink)

 configure startup

Same as "config editor", but instead of working against the running-config you are editing the startup-config. Which is more safe ... till the next reload (wink)

config startup
r1#configure startup
...
 configure reload

"configure reload" as its name implies is not about reloading a router whatsoever (smile)

config reload
r1#configure reload ?                                                     
  <url> - source url

r1#configure reload    
...

This command take a <url> as argument. Basically it will fetch router configuration from the specified <url> and load it into the startup-config. It is an equivalent to Cisco "copy <url> start". From that point:

  • it is up to the network operator to check the startup configuration
  • and issue a reload warm in order to restart the router and test that connectivity is resuming as expected
  • Check the running-config is aligned to startup-config


Warning

(repetition is not harmful) As said before "configure reload" does not reload the router. It just load the config from specified <url> into the startup-configuration. And this steps precedes a reload that has to be triggered manually by the operator after having checked the config.

Note

in day to day operation, startup-config is usually not altered directly. In TELCO SP environment, IIRC, I used it mainly to retrieve configuration from a CMDB server during 2 situations:

  • Router first time installation after basic configuration staging enabling minimum connectivity
  • Router hardware replacement

Note that in SP environment, as VPN owner we could handle a portfolio of customer (~10). Each customer could have ~ 2000 CPEs. You can see why "config reload" can be very handy.

 configure network

"configure network" gives you the possibility to update/merge existing  running-config from config exposed from a web server.

route addition via freeRouter
r1#configure network ?                                                    
  <url> - source url

r1#configure network
...

This command take a <url> as argument. Basically it will fetch specified configuration from the specified <url> and merge it into the running-config. It is an equivalent to Cisco "copy <url> run". So, from that point:

Warning

  • only running-config is altered.
  • If not saved all changes will be lost in the next reload

Note

in day to day operation, In TELCO SP environment, "configure network" is very useful when you want to apply the same configuration stanza to several router at the same time.

 configure overwrite-network

Same as "configure network" gives you the possibility to replace running-config from config exposed from a web server.

route addition via freeRouter
r1#configure overwrite-network ?                                                    
  <url> - source url

r1#configure overwrite-network
...

This command take a <url> as argument. Basically it will fetch specified configuration from the specified <url> and replace the running-config. It is an equivalent to Cisco "copy <url> run". So, from that point:

Warning

  • only running-config is altered.
  • If not saved all changes will be lost in the next reload

Note

in day to day operation, In TELCO SP environment, "configure network" is very useful when you want to apply the same configuration stanza to several router at the same time from a clean slate state. (no merger)

 configure banner

"configure banner" is one of my favorite mode. It will display an editor allowing you to edit the banner of your router.

route addition via freeRouter
r1#configure banner                                                   
...

Press Ctrl-q and then y in order to save the banner.

Log in to you router again in order to check your new banner:


Note

in day to day operation, this banner can be written in configuration using banner encoded command

banner encoded
banner encoded ICAgX18gICAgICAgICAgICAgICBfX19fICAgICAgICAgICAgIF8NCiAgLyBffF8gX18gX19fICBfX198ICBfIFwgX19fICBfICAgX3wgfF8gX19fIF8gX18NCiB8IHxffCAnX18vIF8gXC8gXyBcIHxfKSAvIF8gXHwgfCB8IHwgX18vIF8gXCAnX198DQogfCAgX3wgfCB8ICBfXy8gIF9fLyAgXyA8IChfKSB8IHxffCB8IHx8ICBfXy8gfA0KIHxffCB8X3wgIFxfX198XF9fX3xffCBcX1xfX18vIFxfXyxffFxfX1xfX198X3wNCiAgXyBfXyBfX18gICBfX198IHwgX19fX18gIHwgfA0KIHwgJ19fLyBfIFwgLyBfX3wgfC8gLyBfX3wgfCB8DQogfCB8IHwgKF8pIHwgKF9ffCAgIDxcX18gXCB8X3wNCiB8X3wgIFxfX18vIFxfX198X3xcX1xfX18vIChfKQ0KDQo=

the command corresponds to the banner mentioned above.

 configure revert

"configure revert" revert the running-config to the startup config. For Junos fan it is equivalent to "rollback 0"

configure description
r1#sh run int sdn3                                                        
interface sdn3
 description r1@LAN3[05:00.0]
 mtu 1500
 macaddr 007b.0c15.1e0c
 shutdown
 no log-link-change
 exit
!
configure description
r1# conf t                                                                
r1(cfg)#int sdn3
r1(cfg-if)#description "This is the new description"

mjolnir(cfg-if)#do sh conf                                                     
interface sdn3
 no description r1@LAN3[05:00.0]
 description "This is the new description "
 exit
Let's diff between running and startup config
r1(cfg-if)#do sh conf                                                     
interface sdn3
 no description r1@LAN3[05:00.0]
 description "This is the new description "
 exit
sh run sdn3
mjolnir(cfg)#exit                                                              
mjolnir#sh run sdn3                                                            
interface sdn3
 description "This is the new description "
 mtu 1500
 macaddr 007b.0c15.1e0c
 shutdown
 no log-link-change
 exit
sh run sdn3
mjolnir#configure revert                                                       
     1: interface sdn3
     2:  no description "This is the new description "
     3:  description r1@LAN3[05:00.0]
     4:  exit

errors=0
sh run sdn3
mjolnir#sh run sdn3                                                            
interface sdn3
 description r1@LAN3[05:00.0]
 mtu 1500
 macaddr 007b.0c15.1e0c
 shutdown
 no log-link-change
 exit

Note

in day to day operation, In TELCO SP environment, "configure revert" should be used as "rollback 0" upon the running config. This means that you are about to abandon the current running config and re-apply the config that figures in the startup-config. In our case, it was changing a description, but in some case it can have more impact. (change route filtering, route advertising etc.)

 configure rollback

"configure rollback" is very useful when you are in an operational  situation that needs "trial and error" approach, and sometimes the error can lead to loss of connectivity on the router itself... Who never experienced that ?

First of all we have a saying a French saying: "Il n'y a que ceux qui ne font rien qui ne font pas de bêtise". So don't feel guilty about that... I remembered having isolated some sites just by accident ...

In this situation "configure rollback" is a combination of "configure revert" and a loss of CLI TCP session. What does this practically means ?

Imagine you are configuring a redistribution between IS-IS and OSPF and that you forgot that the network have 2 connections. This redistribution without careful route filtering will result in a routing loop and it happens that you lose connectivity upon that configuration. (never ending routing advertisement loop, high cpu load etc...)

Upon losing TCP connection, in "configure rollback" freeRouter will automatically revert to its startup config.

You will therefore auto-magically get back connection before it was the route redistribution that caused the havoc.

How cool is that !

Note

In IOS, i used to use  "reload in <x>" command, in JunOS of course you have "commit confirm" and same goes for IOS-XR. So this airbag is not only unique to freeRouter, but IT IS THERE !

 configure file

"configure file" gives you to the possibility to update/merge running configuration from a local file from the flash filesystem.

route addition via freeRouter
r1#configure file ?                                                       
  <file> - source file

r1#configure file
...

This command take a <file> as argument. Basically it will load specified configuration from the specified <file> and update/merge the running-config. It is an equivalent to Cisco "copy <flash:file> run". So, from that point:

show flash
mjolnir#show flash /rtr                                                        
date                 size     name
2009-12-31 23:00:00  18048    bundle.bin
2020-07-30 15:47:05  2477     c.sh
2009-12-31 23:00:00  22648    hdlcInt.bin
2020-08-26 07:35:35  2937     hwdet-all.sh
2020-07-31 13:31:28  203      hwdet-main.sh
2009-12-31 23:00:00  18616    mapInt.bin
2020-09-29 08:58:48  554856   mjolnir.log
2009-12-31 23:00:00  18088    modem.bin
2009-12-31 23:00:00  131432   p4dpdk.bin
2009-12-31 23:00:00  121896   p4emu.bin
2009-12-31 23:00:00  63144    p4pkt.bin
2009-12-31 23:00:00  18088    pcap2pcap.bin
2009-12-31 23:00:00  18608    pcapInt.bin
2009-12-31 23:00:00  18384    rawInt.bin
2020-09-28 11:54:12  598      rtr-hw.txt
2020-09-28 21:16:19  14607    rtr-sw.txt
2020-07-30 15:47:37  2022     rtr.err
2020-09-29 03:09:25  5587321  rtr.jar
2020-09-29 03:09:16  5585713  rtr.jar.bak
2020-09-29 03:09:26  24       rtr.rld
2020-09-23 03:06:12  529      rtr.scr
2020-09-23 03:06:11  483      rtr.scr.bak
2020-08-23 17:34:19  46       rtr.scr2
2020-08-23 17:34:18  0        rtr.scr2.bak
2020-09-23 03:06:11  542720   rtr.tar
2020-09-23 03:06:09  522240   rtr.tar.bak
2020-09-29 03:11:04  2330     rtr.ver
2020-09-29 03:11:03  3790694  rtr.zip
2020-09-29 03:10:57  3789659  rtr.zip.bak
2020-07-30 15:47:05  388      setup_dpdk.sh
2020-07-30 15:47:05  48       setup_route.sh
2020-07-30 15:47:05  2171     setup_veth.sh
2009-12-31 23:00:00  18048    stdLin.bin
2009-12-31 23:00:00  18440    tapInt.bin
2009-12-31 23:00:00  18224    ttyLin.bin
2009-12-31 23:00:00  18256    vlan.bin
 configure replace

"configure file" gives you to the possibility to replace running configuration from a local file from the flash filesystem.

route addition via freeRouter
r1#configure replace ?                                                       
  <file> - source file

r1#configure replace
...

This command take a <file> as argument. Basically it will load specified configuration from the specified <file> and replace the running-config. It is an equivalent to Cisco "copy <flash:file> run". So, from that point:

show flash
mjolnir#show flash /rtr                                                        
date                 size     name
2009-12-31 23:00:00  18048    bundle.bin
2020-07-30 15:47:05  2477     c.sh
2009-12-31 23:00:00  22648    hdlcInt.bin
2020-08-26 07:35:35  2937     hwdet-all.sh
2020-07-31 13:31:28  203      hwdet-main.sh
2009-12-31 23:00:00  18616    mapInt.bin
2020-09-29 08:58:48  554856   mjolnir.log
2009-12-31 23:00:00  18088    modem.bin
2009-12-31 23:00:00  131432   p4dpdk.bin
2009-12-31 23:00:00  121896   p4emu.bin
2009-12-31 23:00:00  63144    p4pkt.bin
2009-12-31 23:00:00  18088    pcap2pcap.bin
2009-12-31 23:00:00  18608    pcapInt.bin
2009-12-31 23:00:00  18384    rawInt.bin
2020-09-28 11:54:12  598      rtr-hw.txt
2020-09-28 21:16:19  14607    rtr-sw.txt
2020-07-30 15:47:37  2022     rtr.err
2020-09-29 03:09:25  5587321  rtr.jar
2020-09-29 03:09:16  5585713  rtr.jar.bak
2020-09-29 03:09:26  24       rtr.rld
2020-09-23 03:06:12  529      rtr.scr
2020-09-23 03:06:11  483      rtr.scr.bak
2020-08-23 17:34:19  46       rtr.scr2
2020-08-23 17:34:18  0        rtr.scr2.bak
2020-09-23 03:06:11  542720   rtr.tar
2020-09-23 03:06:09  522240   rtr.tar.bak
2020-09-29 03:11:04  2330     rtr.ver
2020-09-29 03:11:03  3790694  rtr.zip
2020-09-29 03:10:57  3789659  rtr.zip.bak
2020-07-30 15:47:05  388      setup_dpdk.sh
2020-07-30 15:47:05  48       setup_route.sh
2020-07-30 15:47:05  2171     setup_veth.sh
2009-12-31 23:00:00  18048    stdLin.bin
2009-12-31 23:00:00  18440    tapInt.bin
2009-12-31 23:00:00  18224    ttyLin.bin
2009-12-31 23:00:00  18256    vlan.bin

Discussion

Most of you will simply use the basic "conf t" mode, but keep in mind that depending on your context, all the other modes are proven to be very handy. The possibility to configure 1000 router with one single config file using "config network" is a savior. Having the possibility to trigger automatic definitive router staging using "conf reload" is tremendously useful when you have to deploy 10 routers a day. As said "config view" can give non operation staff to check if some configs are there or not ... "config editor" is very powerful when you want to edit a never ending access-list, but remember to avoid to use it for complex BGP config... You have been warned !

Conclusion

In this 1st article:

  • We presented freeRouter config mode
  • Most of these are useful in various different context

Final words

All these modes are not new. IOS, IOS-XR, IOX-XE, NW-OX, JUNOS have their own config mode that are somewhat similar. In any case freeRouter config mode implementation is meant to address  all needs from the network operators perspective. As you can observe, configure mode has an impressive list of mode. Feel free to try and use them according to your environment taste!

Last but not least, you can play with these different mode from this sandbox:

type "telnet dl.nop.hu" in a terminal and choose "1"
 telnet dl.nop.hu                                                                                                                                                 1 ↵
Trying 193.224.23.5...
Connected to dl.nop.hu.
Escape character is '^]'.
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXX XXXXX XXX    XXX     XXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX  XXXX XX XXXX XX XXXX XX XX XX XXXX XXXXX/~~~~~~\XXXXXX
XXXX X XXX XX XXXX XX XXXX XX XX XX XXXX XXXX| player |XXXXX
XXXX XX XX XX XXXX XX     XXX    XX XXXX XXXXX\______/XXXXXX
XXXX XXX X XX XXXX XX XXXXXXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX XXXX  XX XXXX XX XXXXXXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX XXXXX XXX    XXX XXX XXX XX XXX    XXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
welcome
line ready
menu lab:
...
type "ssh dl.nop.hu" in a terminal (any user/pass will do) and choose "1"
ssh dl.nop.hu -l random_user                                                                                                                                     1 ↵
Warning: Permanently added 'dl.nop.hu,193.224.23.5' (RSA) to the list of known hosts.
random_user@dl.nop.hu's password: 
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXX XXXXX XXX    XXX     XXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX  XXXX XX XXXX XX XXXX XX XX XX XXXX XXXXX/~~~~~~\XXXXXX
XXXX X XXX XX XXXX XX XXXX XX XX XX XXXX XXXX| player |XXXXX
XXXX XX XX XX XXXX XX     XXX    XX XXXX XXXXX\______/XXXXXX
XXXX XXX X XX XXXX XX XXXXXXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX XXXX  XX XXXX XX XXXXXXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX XXXXX XXX    XXX XXX XXX XX XXX    XXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
welcome
line ready
menu lab:
# - reboot router1
$ - reboot router2
% - reboot router3
1 - connect to router1
2 - connect to router2
3 - connect to router3
^ - rebuild routers
l - connect to lg.nop.dn42
x - exit
choose:1 - attach vdc lab1 

yourname#                                                                      
yourname#configure ?                                                           
  <cr>
  banner            - edit the banner
  editor            - configure from editor
  file              - append to running configuration
  network           - append to running configuration
  overwrite-network - overwrite the running configuration
  reapply           - !!!EXPERiMENTAL!!! try to reapply current configuration
  reload            - overwrite the startup configuration
  replace           - overwrite the running configuration
  revert            - revert to startup configuration
  rollback          - configure within auto-revert session
  startup           - edit the startup configuration
  terminal          - configure from this terminal
  viewer            - view current configuration

yourname#configure                                      
...

In order to exit the sandbox session use the following escape sequence: Ctrl-c + Ctrl-x

Another method to access the sandbox, by click here, this will open a terminal webapp into your browser:







This is a special blog series called "RARE software architecture". As its name implies, it deals with topics related to RARE/freeRouter software design choice.

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

RARE project objective is to provide a routing platform proposing various solutions addressing multiple use cases in the R&E landscape. In the picture below you see in purple the different use cases:

As you can notice, each use case will run on different hardware that potentially can have different dataplanes. As we were starting from a clean slate environment without much choice, especially with P4 programmability - the first dataplane or P4 target considered was BMv2. BMv2 is an excellent way to learn P4, it is also the first target we use in order to program and validate new features. After 6 months of practising our "P4-fu" we developed:

  • a P4lang repository for ubuntu bionic and focal
  • a debian 10 repository
  • had our first RARE/FreeRouter prototype powered by a P4 BMv2 dataplane !

Our initial work, considering FreeRouter's Java nature, was to write a Java P4Runtime GRPC client that would be able to program the entries in the tables exposed by BMv2 via the P4Info file. However, this would have intimately tied FreeRouter code to P4Runtime gRPC code. Even if it's more natural to choose this solution, going in that direction implied that dataplanes other than BMv2 would be compliant to P4Runtime. It turns out that this is not the case. We then opted for a simple message API via a bi-directional raw UNIX socket. We will see what this means later in this blog.

Motivated by the successful experience with BMv2, we then decided to move forward and started to study TOFINO as a target. We were greedy and eager to apply our P4 code against multi-terabits traffic. After a few P4 program compilations, the first impression from my personal perspective was ... mind blowing ! INTEL/BAREFOOT TOFINO effectively opened the door to multi-terabits packet processing... Just to have at the tip of your finger the possibility to process traffic at these traffic levels was exciting !

As a side note, the journey was not without suffering and pain... (smile) We had to port our BMv2 code - and to port to TOFINO was not "Une lettre à la poste"... It is not that TOFINO programming is gratuitously painful. It is just that it is p4c-tofino's job to make sure that our packets are processed at silicon lighting speed. Imagine you are asked to  convey parcels by driving from Paris to Amsterdam with a car that has an infinitely sized trunk, with an infinite gas tank and no particular speed constraint along the road. And then you are asked to do the same trip, but with an actual real car that has a trunk with a fixed size and with a 50 litre gas tank, and of course you'll have to follow speed signs along the road.

In the first case, you would put as many parcels as you would like and you even won't bother looking at your gas tank level and maybe you'd set the speed to 200 Km/h. The second case forces you to carefully think about how many parcels you can put in your trunk, check to see if one completely full tank can be sufficient for the trip and of course, you would have to follow the speed signs.

If you allow me this comparison, this is where BMv2 and TOFINO programming differs.  

But, this pain was not in vain, it was for the greater good... You can't imagine the inherent joy when you see the TOFINO compiler displaying the DONE word ! For the veterans who can remember, it is the same feeling when you manage to compile your first program in the ADA language. The compiler is not so strict that compiling an ADA program is in itself a feat. No wonder why this language is used in Spatial rocket (Ariane).

Back to our dataplane interface story, even TOFINO and BMv2 share some roots, while BMv2 had P4Runtime as a northnound interface, INTEL/BAREFOOT pushed into TOFINO platform with P4_16 their gRPC interface counterpart: BfRuntime.

Our best bet paid off as FreeRouter message API was unchanged and without much effort we could add a new dataplane "wingman" to the FreeRouter control plane.

To recap:

  • For BMv2: Our interface yields P4Runtime RPC calls. This program is called: forwarder.py
  • For TOFINO: Our interface yields BfRuntime RPC calls. This program is called witout too much originality: bf_forwader.py

At that point we were starting to have a decent LSR/LER router for CORE and Aggregation use cases.

But we still had nothing at the EDGE/AGGREGATION layer in terms of a solution proposal, deploying P4 hardware might be way too expensive in specific contexts such as small R&E institutions like primary schools or small R&E labs. To that purpose, we started to study new targets such as VMWARE XDP and a very promising project: T4P4S ELTE. While we could not use XDP without a lot of P4 code rewriting and compromise, T4P4S ELTE was from our perpective very promising. But due to a compilation issue, we could not move forward.

FPGA was also a solution that we considered but had no access to any FPGA hardware that was P4 compliant.

As a result, we were a little bit bitter and started to read the DPDK library. And we started to play with DPDK examples... These examples were tremendously useful as it sparked some DPDK development into the RARE team. Csaba, the FreeRouter lead developer, step by step came up with this GENIUS idea: why don't we just use emulate P4 RARE P4 dataplane program ? We can still revert to using T4P4S ELTE when it will be ready ?

P4emu/P4dpdk was then born ! 

To conclude this short story, RARE/FreeRouter has now 3 completely different dataplanes: (in order of appearance)

  • BMv2
  • TOFINO
  • DPDK


Unique RARE/FreeRouter feature

However, please note that FreeRouter message API is common to the three dataplanes listed above. You'll see further how this structure make the solution: an open modular, interchangeable solution.

Article objective

In this article, let's present RARE/FreeRouter platform structure and focus on the interface(S) between FreeRouter control plane and various dataplane.

Diagram

[ #001 ] - Modular design

 FreeRouter control plane

In this designs, FreeRouter is focusing on running control plane processes, such as routing protocols IGP(s), BGP(s). There are other control plane processes but let's just focus on these latter. At some point in time, all IGPs/EGP converge and will have to create an entry in a routing table. In case of IPv4 the entry will be created into an IPv4 forwarding table and similarly, an IPv6 route entry will be created into IPv6 forwarding table. From FreeRouter point of view these entry creation will be triggered by yielding one Java function twice that will generate these 2 API messages, one for IPv4 and the other one for IPv6.

 Common message API

Let's add an IPv4 route into freeRouter CLI

route addition via freeRouter
conf t
ipv4 route v1 1.2.3.0 255.255.255.0 4.4.4.4
...

Upon entering the ipv4 route and pressing <enter>, you'll see the following message appearing

message API: route4_add
...
rx: ['route4_add', '1.2.3.0/24', '13063', '4.4.4.4', '1', '\n']
...

Let's delete the route via FreeRouter CLI

route deletion via freeRouter
conf t
no ipv4 route v1 1.2.3.0 255.255.255.0 4.4.4.4
...
message API: route4_del
...
rx: ['route4_del', '1.2.3.0/24', '13063', '4.4.4.4', '1', '\n']
...

Important note

In short, the message API is simply a collection of message that would trigger an entry ADD/DELETE/MODIFY into the dataplane corresponding table.

The documentation of this message API will be documented and published soon, but for those who are curious and can't wait this documentation, you can read forwarder.py, bf_forwarder.py or p4dpdk.bin  source code

 Candidate dataplane platform

As said in the beginning of the article, freeRouter control plane would have to deal with dataplane of different nature. And we concluded in mentioning that for now, freeRouter has three dataplanes. Each of these dataplanes have their own northbound interface, whether this is P4Runtime for BMv2, BfRuntime for TOFINO or P4DPDK for system compatible with DPDK and having DPDK complinnt NIC.

For BMv2 we just had to write an interface that would translate freeRouter API message into P4Runtime GRPC calls. For BMv2 this interface is called forwarder.py:

For TOFINO we just had to write an interface that would translate freeRouter API message into BfRuntime GRPC calls. For TOFINO this interface is called bf_forwarder.py:

For DPDK we just had to write an interface that would translate freeRouter API message into DPDK primitives. This interface is included into DPDK dataplane bundled into freeRouter binaries: p4dpdk.bin

It is just as simple as that !

Discussion

 Dataplane addition made easy

This design is pretty unique because, if for any reason you would like to "hook" freeRouter control plane to an other dataplane such as:

  • FPGA
  • or dataplane powered by kernel bypass technique such as RDMA
  • Or other NPU based dataplane
  • etc.

This is possible !

You would "just" have to port your P4 code logic into the target dataplane and create an interface able to translate API messages from FreeRouter into understandable message from the target dataplane.

Be cautious with the word "just"

The "just" word can be misleading. Indeed, depending on the target dataplane, it can be a huge task. With DPDK, we were lucky in getting enough material in order to move forward and again p4dpdk.bin was a simple trial at the very beginning. But some other dataplane can just be simply be ignored if we don't get enough material/support from NPU vendors. 

 You can use your own control plane too !

One thing that we did not experience, but this can be maybe one day a reality.

What if you have your own control plane and that you absolutely want to keep it, but would like to re-use BMv2/TOFINO or DPDK RARE dataplane ?

Well this is possible !

Long time ago I met Thomas MANGIN (yet another cool and nice French guy (smile) ) which is the author of Exa-BGP, i did not talk to him about this and I don't want to give him bad idea, but what if he would like to hook a TOFINO P4 dataplane to Exa-BGP ?

Well, he actually would just have to teach exaBGP to handle entry ADD/DELETE/MODIFY message according to the message API above.

I also love the work DONE at the SoNIC project level and I know that SoNIC has already a P4 dataplane called switch.p4. I doubt it will be the case one day but, what if SoNIC project wanted to re-use RARE dataplane for especially for Service Provider capability ?

OK, this sounds crazy, but the modular design we proposed here is valid and can make the RARE dataplane available for other control plane.

Of course, we strongly suggest you to stick with FreeRouter as you will just realize IMHO that in the TELCO Service Provider space there is no match. You'll have the venerable IOS-XR and JUNOS, but these are not Open Source counterparts.


Conclusion

In this 1st article you:

  • had a 10K feet view description of RARE/FreeRouter modular design
  • This design allow rapid dataplane addtion without altering whatsoever FreeRouter code base
  • In case you would like to re-use BMv2/TOFINO/P4DPDK dataplane, this has been never implemented but this is possible !

Message API documentation

From the time being this API  message is not yet publicly documented. However, it is available and buried inside forwarder.py or bf_forwarder.py source code. This is work in progress but if you feel an urgent need to use it feel free to read the code.

PS: We will publish this document ASAP, but time plays against us ...




This is a special blog series called "RARE hardware platform". As its name implies it deals with certified and tested platform on which RARE/freeRouter can run out of the box.

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

We will deal with a series of article related to STORDIS BF2556X-1T P4 switch. The key highlight of this box is: 

  • It is a P4 TOFINO NPU based switch
  • TOFINO version has 2 cores (i.e. 2 pipes) and can manage up to 2 Tbps
  • It offers multiple connection types and rates:
    • 48x25GSFP28 and 8x100GQSFP28
      • SFP28 port [1 - 16] can configure into 1G/10G/25G
      • SFP28 port [17 - 48] can configure into 10G/25G
      • QSFP28 port [49 - 56] Each QSFP28 port can configure into 1x100G,2x50G,4x25G, 1x40G or 4x10G Mode.
  • SyncE and 1588 support

Article objective

In this article, we will just do a basic introduction of the BF2556X-1T

[ #001 ] - BF2556X-1T in a nutshell

 BF2556X-1T unboxing

Parcel

What's in the box

Included items

Quick Installation guide

Front panel

Back panel

STORDIS BF2556X-1T Racked

STORDIS BF2556X-1T alongside to his P4 brothers: Edgecore WEDGE100BF32X

 Hardware specification

BF2556X-1T specification

The system uses Barefoot BFN-T10-032D-020 (Tofino 2.0T) Switch Chip which can support 20 x 100GE ports.

Major features are:

  • 2.0 Tbps bandwidth
  • One Barefoot BFN-T10-032D-020(Tofino 2.0T) Switch ASIC
    • Ethernet support 80x25G SERDES ports
    • Management SERDES support four ports 10G-KR
    • PCIe Gen3 x 4lanes
  • Eight Marvell 98PX1024
    • Single chip support 4x25G SERDES
  • Network Interface
    • 48x25G SFP28 and 8x100G QSFP28
    • SFP28 port 1~16 can configure into1G/10G/25G.
    • SFP28 port17~48 can configure into 10G/25G.
    • Each QSFP28 port can configure into 1x100G,2x50G,4x25G, 1x40G or 4x10G Mode.
  • CPU Module: Optional Module design for flexibility
    • Intel® Xeon® Processors D1527 (BDXDE)
  • BMC: Base Board Management Controller
    • BMC is a specialized service processor that monitors the physical state of a system.
    • ASPEED AST2520
  • Management Port:
    • 3xRJ45 10/100/1000Mbps OOBM(Out Of Band Management) port
    • 1xConsoleRJ45
    • 1xUSB3.0
  • FAN Tray:
    • Four 40mmx56mm Fan-tray
    • Supporting 3+1 redundancy
    • Support front to back and back to front air direction.
  • PSU:
    • 1+1 redundant PSU
    • Each PSU will be supporting 850W power to system.
    • 12V standby power for system management chips.
    • Support DC power supply
CPU specification
lscpu
Architecture:        x86_64                                                         
CPU op-mode(s):      32-bit, 64-bit                                                 
Byte Order:          Little Endian                                                  
CPU(s):              16                                                             
On-line CPU(s) list: 0-15                                                           
Thread(s) per core:  2                                                              
Core(s) per socket:  8                                                              
Socket(s):           1                                                              
NUMA node(s):        1                                                              
Vendor ID:           GenuineIntel                                                   
CPU family:          6                                                              
Model:               86                                                             
Model name:          Intel(R) Xeon(R) CPU D-1548 @ 2.00GHz                          
Stepping:            3                                                              
CPU MHz:             799.832                                                        
CPU max MHz:         2600.0000                                                      
CPU min MHz:         800.0000                                                       
BogoMIPS:            4000.16                                                        
Virtualization:      VT-x                                                           
L1d cache:           32K                                                            
L1i cache:           32K                                                            
L2 cache:            256K                                                           
L3 cache:            12288K                                                         
NUMA node0 CPU(s):   0-15                                                           
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperf mperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_ad just bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsa veopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d                                                                      

Discussion

 Hardware consideration

The BF2556X-1T is a horse power:

  • the usage of 8 cores having each one 2 threads speeds up P4 program compilation. (BF2556X-1T  as 2x more core than the WEDGE100BF32X)
  • SyncE 1588 might be certainly important for you should your P4 application require precise time synchronisation capability 
  • Having 1G/10G/25G/40/50G/100G connectivity via SFP28 and QSFP28 will make the BF2556X-1T ready for multiple use case.
    • In a P/PE architecture MPLS PE proposing 1G/10G connectivity and having uplink toward the core
    • In a collapse core can be used a MPLS PE router
    • Can be used as a leaf or Tor switch/router
    • BRAS/BNG router

Conclusion

In this 1st article you:

  • had a brief description STORDIS BF2556X-1T hardware platform
  • The hardware provide p4 connectivity at 1GE capacity (16x1GE ports is available)
  • In addition to 1GE it also provide 10/25/40/50/100G connectivity

RARE hardware plarform: [ BF2556X-1T #001 ] - key take-away

  • From RARE/FreeRouter point of view, BF2556X-1T is very good candidate for PE (Provider Edge) router.

The 8x100G ports can make as a strong in a collapse core architecture (P function merge with PE functions), the box can also be used a a BGP route as it boast with 32 GB of RAM (~10 full BGP feeds), but you won't leverage the ports availability. It can be used to implement BRAS/BNG use case but would be also a good candidate as a ToR in Data Center envionment with BGP/MPLS capability and the possibility to provide 1GE connection to existing server purchased beforehand.

  • SyncE 1588 support is a key features if your application needs precision provided by PTP

As we will discover the box, we will explain in further articles how to benefit from this features. 

  • RARE/freeRouter @design can coexist with Virtualisation technology BF2556X-1T

We just started our experience with this box. You'll find further, a series of article dedicated to BF2556X-1T depicting:

  • How to proceed to initial OS installation
  • Proceed to STORDIS BF2556X-1T software installation (TOFINO SDE and Gearbox) installation
  • Port operations on TOFINO ports SFP28 port 16-47 and QSFP28 port 48-56  
  • Port operations on GearBox ports SFP28 port 1-16 (1G/10G/25G)
  • How to benefit from SyncE 1588 support
  • RARE/freeRouter effective installation

The installation will be implemented should be compliant to ISP TELECOM standard. (It should survives power outage, easy upgrade features, start automatically at boot time without any human intervention)



The 1st article presented you the hardware platform and the rationale behind the choices. Let's dive into the subject now!

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

Several choices were possible, we finally ended up in following the KISS method. The Operating system requirements are:

  • requirement #0: LTS operating system 
  • requirement #1: Benefit from LTS security patches
  • requirement #2: Must be able to run DPDK
  • requirement #3: (personal requirement) Must be familiar to me
  • requirement #4: Able to run Java software as freeRouter is written in Java
  • requirement #5: Small operating system software footprint
  • requirement #6: Support for IPv4/IPv6

The hardest path would be:

The objective is to have tight control of the software installed on the appliance. This guarantees the smallest footprint we hope to obtain. For those familiar with OpenWRT, we can reach a tiny image size. My OpenWRT image is 5Mb.

  • Use of NixOS or Nix package manager

This provides an incredible feature: commit/rollback functionality at the package management level!

Note

The features above are still under study into RARE group. We will introduce these technologies once we feel more confident on how to integrate these technologies into a streamlined deployment process.

Article objective

In this article we will go through the major steps in deploying Debian 10 stable aka Buster in order to prepare freeRouter installation.

Diagrams

[ #002 ] - Cookbook

 Operating system installation preparation
Get debian 10 minimal ISO
wget http://ftp.nl.debian.org/debian/dists/buster/main/installer-amd64/current/images/

On MACOSX, burn the iso using balenaEtcher

balenaEtcher can be downloaded here

Via the appliance BIOS settings:

  • activate console port redirection:

Option d'activation du port série

  • configure serial port settings

Now that you have activated console port:

  • plug the USB key on which you previously burnt Debian 10
  • make sure you  set boot option from USB in BIOS settings
  • reboot

You can now proceed to the next step: Debian 10 installation

 Operating system installation

We will assume that you have installed Debian 10 on the 256 Gb SSD.

Just as a side note during the installation process you'll be prompted the: "Software selection" window, in this steps we will:

  • unselect everything
  • select "SSH server"

Software selection

This will guarantee the tiniest Debian 10 operating system software footprint. We will on demand install the needed packages manually.

 packages installation needed by RARE/freeRouter

On minimal installation, sudo is not installed, so all the software will be done as root.

minimal Java installation
apt-get update
apt-get install default-jre-headless

The latest DPDK software is needed. We use the Debian 10 backport repository in orcer to get DPDK 19.11.2-1~bpo10+1

dpdk from debian 10 backports repository
echo "deb http://deb.debian.org/debian buster-backports main" | tee /etc/apt/sources.list.d/buster-backports.list
apt-get update
apt-get install dpdk dpdk-dev
Check DPDK version
dpkg -l | grep dpdk
ii  dpdk                                    19.11.2-1~bpo10+1            amd64        Data Plane Development Kit (runtime)
ii  dpdk-dev                                19.11.2-1~bpo10+1            amd64        Data Plane Development Kit (dev tools)
ii  libdpdk-dev:amd64                       19.11.2-1~bpo10+1            amd64        Data Plane Development Kit (basic development files)
additional 3rd party software used by freeRouter
apt-get update
apt-get install unzip net-tools libpcap-dev ethtool default-jre-headless psmisc tcpdump
 create freeRouter /rtr folder

In this setup we will create a freeRouter folder at the filesystem root directory

Create freeRouter folder at filesystem root directory
mkdir /rtr
get freeRouter control plane software
cd /rtr 
wget http://freerouter.nop.hu/rtr.jar
get freeRouter net-tools tarball
cd /rtr 
tar xvf rtr.tar -C /rtr
rm rtr.tar
 Disable host networking (One time installation)

As freeRouter is handling the networking task, we have to disable the appliance networking. Forgetting to do so will result in conflicts and unpredictable behaviour. 

Disable networking from systemd perspective
systemctl set-default multi-user.target
rm /usr/lib/systemd/network/*
SVC="network-manager NetworkManager ModemManager systemd-network-generator systemd-networkd systemd-networkd-wait-online systemd-resolved hostapd wpa_supplicant"
systemctl disable $SVC
systemctl mask $SVC
 freeRouter systemd startup script
freeRouter systemd startup script
cat /lib/systemd/system/rtr.service

[Unit]
Description=router processes
Wants=network.target
After=network-pre.target
Before=network.target

[Service]
Type=forking
ExecStart=/rtr/hwdet-all.sh

[Install]
WantedBy=multi-user.target
/rtr/hwdet-all.sh script
cat /rtr/hwdet-all.sh

#!/bin/sh

cd /rtr
echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6
echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6
echo 0 > /proc/sys/net/ipv6/conf/lo/disable_ipv6
ip link set lo up mtu 65535
ip addr add 127.0.0.1/8 dev lo
ip addr add ::1/128 dev lo

# DPDK
echo 96 > /proc/sys/vm/nr_hugepages
modprobe uio_pci_generic

dpdk-devbind.py -b uio_pci_generic 01:00.0 
dpdk-devbind.py -b uio_pci_generic 02:00.0 
dpdk-devbind.py -b uio_pci_generic 05:00.0 
dpdk-devbind.py -b uio_pci_generic 06:00.0 
dpdk-devbind.py -b uio_pci_generic 07:00.0 
dpdk-devbind.py -b uio_pci_generic 08:00.0 

#VETH for CPU_PORT and OOBM_PORT
ip link add veth0a type veth peer name veth0b

ip link set veth0a multicast on
ip link set veth0a allmulti on
ip link set veth0a promisc on
ip link set veth0a mtu 8192
ip link set veth0a up

ip link set veth0b multicast on
ip link set veth0b allmulti on
ip link set veth0b promisc on
ip link set veth0b mtu 8192
ip link set veth0b up

ethtool -K veth0a rx off
ethtool -K veth0a tx off
ethtool -K veth0a sg off
ethtool -K veth0a tso off
ethtool -K veth0a ufo off
ethtool -K veth0a gso off
ethtool -K veth0a gro off
ethtool -K veth0a lro off
ethtool -K veth0a rxvlan off
ethtool -K veth0a txvlan off
ethtool -K veth0a ntuple off
ethtool -K veth0a rxhash off
ethtool --set-eee veth0a eee off

ethtool -K veth0b rx off
ethtool -K veth0b tx off
ethtool -K veth0b sg off
ethtool -K veth0b tso off
ethtool -K veth0b ufo off
ethtool -K veth0b gso off
ethtool -K veth0b gro off
ethtool -K veth0b lro off
ethtool -K veth0b rxvlan off
ethtool -K veth0b txvlan off
ethtool -K veth0b ntuple off
ethtool -K veth0b rxhash off
ethtool --set-eee veth0b eee off

ip link add veth1a type veth peer name veth1b

ip link set veth1a multicast on
ip link set veth1a allmulti on
ip link set veth1a promisc on
ip link set veth1a mtu 1500
ip link set veth1a up

ip link set veth1b multicast on
ip link set veth1b allmulti on
ip link set veth1b promisc on
ip link set veth1b mtu 8192
ip link set veth1b up

ip link set wlan0 up

ethtool -K veth1a rx off
ethtool -K veth1a tx off
ethtool -K veth1a sg off
ethtool -K veth1a tso off
ethtool -K veth1a ufo off
ethtool -K veth1a gso off
ethtool -K veth1a gro off
ethtool -K veth1a lro off
ethtool -K veth1a rxvlan off
ethtool -K veth1a txvlan off
ethtool -K veth1a ntuple off
ethtool -K veth1a rxhash off
ethtool --set-eee veth1a eee off

ethtool -K veth1b rx off
ethtool -K veth1b tx off
ethtool -K veth1b sg off
ethtool -K veth1b tso off
ethtool -K veth1b ufo off
ethtool -K veth1b gso off
ethtool -K veth1b gro off
ethtool -K veth1b lro off
ethtool -K veth1b rxvlan off
ethtool -K veth1b txvlan off
ethtool -K veth1b ntuple off
ethtool -K veth1b rxhash off
ethtool --set-eee veth1b eee off

ip addr flush dev veth1a 
ip addr add 192.168.128.254/24 dev veth1a

#ADD DEFAULT ROUTE to OOBM SDN999
route add default gw 192.168.128.1

# START RTR !
start-stop-daemon -S -b -x /rtr/hwdet-main.sh
make hwdet-main.sh executable
chmod u+x /rtr/hwdet-main.sh

A bit of explanation

Disable IPv6
echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6
echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6
echo 0 > /proc/sys/net/ipv6/conf/lo/disable_ipv6
ip link set lo up mtu 65535

Note that IPv6 operation would occur on the host itself, IPv6 will be handled at freeRouter level

Disable IPv6
echo 96 > /proc/sys/vm/nr_hugepages
modprobe uio_pci_generic

dpdk-devbind.py -b uio_pci_generic 01:00.0 
dpdk-devbind.py -b uio_pci_generic 02:00.0 
dpdk-devbind.py -b uio_pci_generic 05:00.0 
dpdk-devbind.py -b uio_pci_generic 06:00.0 
dpdk-devbind.py -b uio_pci_generic 07:00.0 
dpdk-devbind.py -b uio_pci_generic 08:00.0 

In the stanza above, we configure DPDK (required)

  • Configure HugePages

In this case we use 96 hugepages, this value can be different if you are using a box with different characteristics (# of ports, memory etc.) The objective is to configure a value that is not too high (waste of resources) and not too small. otherwise p4dpdk won't run. In this case this leaves 10 Free HugePages.

HugesPages Verification
grep HugePages_ /proc/meminfo
HugePages_Total:      96
HugePages_Free:       10
HugePages_Rsvd:        0
HugePages_Surp:        0 
  • Activate UIO_PCI_GENERIC driver
  • Bind the interfaces to DPDK, DPDK will control them now. Keep in mind that now they will be invisible from the linux kernel.

This command use device PCI ID. In order to check device PCI ID just issue the below command:

List PCI device ID list ready to be use (or not by DPDK)
 dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:01:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb
0000:02:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb
0000:05:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb
0000:06:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb
0000:07:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb
0000:08:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb

Network devices using kernel driver
===================================
0000:09:00.0 'AR928X Wireless Network Adapter (PCI-Express) 002a' if=wlan0 drv=ath9k unused=uio_pci_generic 

No 'Baseband' devices detected
==============================

Other Crypto devices
====================
0000:00:1a.0 'Atom Processor Z36xxx/Z37xxx Series Trusted Execution Engine 0f18' unused=uio_pci_generic

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
=================================== 
  • Configure the appliance OOBM via veth pair (as all physical ports are handled by DPDK and will be invisible from the Linux kernel)
Disable IPv6
#VETH for CPU_PORT and OOBM_PORT
ip link add veth0a type veth peer name veth0b

ip link set veth0a multicast on
ip link set veth0a allmulti on
ip link set veth0a promisc on
ip link set veth0a mtu 8192
ip link set veth0a up

ip link set veth0b multicast on
ip link set veth0b allmulti on
ip link set veth0b promisc on
ip link set veth0b mtu 8192
ip link set veth0b up

ethtool -K veth0a rx off
ethtool -K veth0a tx off
ethtool -K veth0a sg off
ethtool -K veth0a tso off
ethtool -K veth0a ufo off
ethtool -K veth0a gso off
ethtool -K veth0a gro off
ethtool -K veth0a lro off
ethtool -K veth0a rxvlan off
ethtool -K veth0a txvlan off
ethtool -K veth0a ntuple off
ethtool -K veth0a rxhash off
ethtool --set-eee veth0a eee off

ethtool -K veth0b rx off
ethtool -K veth0b tx off
ethtool -K veth0b sg off
ethtool -K veth0b tso off
ethtool -K veth0b ufo off
ethtool -K veth0b gso off
ethtool -K veth0b gro off
ethtool -K veth0b lro off
ethtool -K veth0b rxvlan off
ethtool -K veth0b txvlan off
ethtool -K veth0b ntuple off
ethtool -K veth0b rxhash off
ethtool --set-eee veth0b eee off

So the above section is pretty straightforward:

  • It creates veth0a / veth0b pair. For those familiar with P4, this is similar to the channel between the control plane (freeRouter) and p4dpdk (dataplane) using CPU_PORT
  • It sets for veth0a/veth0b: multicast/allmulti/promisc flag + mtu=8192
  • It disables TCP offload for veth0a/veth0b

We do the same thing for the Out Of Band management (linux access)

veth1a/veth1b for OOB management
ip link add veth1a type veth peer name veth1b

ip link set veth1a multicast on
ip link set veth1a allmulti on
ip link set veth1a promisc on
ip link set veth1a mtu 1500
ip link set veth1a up

ip link set veth1b multicast on
ip link set veth1b allmulti on
ip link set veth1b promisc on
ip link set veth1b mtu 8192
ip link set veth1b up

ip link set wlan0 up

ethtool -K veth1a rx off
ethtool -K veth1a tx off
ethtool -K veth1a sg off
ethtool -K veth1a tso off
ethtool -K veth1a ufo off
ethtool -K veth1a gso off
ethtool -K veth1a gro off
ethtool -K veth1a lro off
ethtool -K veth1a rxvlan off
ethtool -K veth1a txvlan off
ethtool -K veth1a ntuple off
ethtool -K veth1a rxhash off
ethtool --set-eee veth1a eee off

ethtool -K veth1b rx off
ethtool -K veth1b tx off
ethtool -K veth1b sg off
ethtool -K veth1b tso off
ethtool -K veth1b ufo off
ethtool -K veth1b gso off
ethtool -K veth1b gro off
ethtool -K veth1b lro off
ethtool -K veth1b rxvlan off
ethtool -K veth1b txvlan off
ethtool -K veth1b ntuple off
ethtool -K veth1b rxhash off
ethtool --set-eee veth1b eee off

ip addr flush dev veth1a 
ip addr add 192.168.128.254/24 dev veth1a

Add default route to SDN999 for OOBM return traffic (192.168.128.1 is freeRouter sdn999: we will see the full config later)

#ADD DEFAULT ROUTE to OOBM SDN999
route add default gw 192.168.128.1

Effectively start freeRouter main loop

Start freeRouter inside main loop
start-stop-daemon -S -b -x /rtr/hwdet-main.sh

This main loop is triggered by the script hwdet-main.sh below:

/rtr/hwdet-all.sh script
cat /rtr/hwdet-main.sh 

#!/bin/sh

while (true); do
  cd /rtr/
  stty raw < /dev/tty
  java -Xmx4g -jar /rtr/rtr.jar router /rtr/rtr-
  if [ $? -eq 4 ] ; then
    sync
    reboot -f
  fi
  stty cooked < /dev/tty
  sleep 1
done  

A bit of explanation

Requirement considerations:

  • The box should run 24x7
  • It must survive a power cut, i.e the service should be restored each time the power is cut for any reasons
  • If no power cut but freeRouter has crashed for any reason, it should be restarted

Let me re-assure you, freeRouter usually don't crash, most often freeRouter has manual or better: auto-upgrades (smile) 

freeRouter infinite loop: freeRouter autoupgrade process restarts and self-restarts
while (true); do
  ...
done  
  • The appliance has 8Gb RAM which is enough for JVM running freeRouter. (Full routing IPv4/IPv6 at the control plane is possible at home!  ← ok this is useless but cool, no? :3 )
    • RAM allocation is for JVM and its tables
    • Additional RAM allocation is for p4dpdk and p4emu, as we have to store the table once for the native code too
    • Lastly the kernel also needs memory, so it's a good idea to leave some free RAM and not give everything to JVM.
Start freeRouter
java -Xmx4g -jar /rtr/rtr.jar router /rtr/rtr-
  • freeRouter "Cold reboot"  
Cold reboot
if [ $? -eq 4 ] ; then
  sync
  reboot -f
fi

Discussion

 Design choice considerations

All the choices have been made in order to make the appliance resilient as much as possible and provide an enjoyable user experience. We will see in a later article, a feature that I love: auto-upgrade. This will keep your appliance up to date over the network with the latest freeRouter train during low traffic period. Of course, for ISP P/PE core router we don't want this, but hey! why not? As soon as all customers are dual homed to 2 different PEs reachable via 2 direct core paths, this can be achieved during low traffic period after having set the metric to infinity on all the PE/P boxes to be upgraded. (use IS-IS overload bit or OSPF max-metric router-lsa)

Conclusion

In this article, we got our hands dirty and manually installed freeRouter with DPDK dataplane from a clean slate environment. This is done on purpose, as I'd like you to understand the whole installation process in detail. There is an automated installation alternative that will install freeRouter also. However this is will install freeRouter with software backend. If your hardware CPU+NIC is compatible you can just replace the software backend by DPDK backend. At that precise point we have a vanilla genuine installation of freeRouter with DPDK dataplane on an appliance that can survive physical wild environment and power cut. We have just now to create the 2 freeRouter configuration files:

freeRouter configuration files
ls -l rtr-*
-rw-r--r-- 1 root root  646 Jul 31 17:03 rtr-hw.txt
-rw-r--r-- 1 root root 9027 Aug 25 10:02 rtr-sw.txt


RARE validated design: [ SOHO #002 ] - key take-away

  • freeRouter installation is not complex. It just boils down to installing a basic supported Linux OS, install Java, some 3rd party software and the freeRouter jar and binaries itself
  • In the binary list you'll have a special one called p4dpdk that corresponds to freeRouter DPDK dataplane that emulate RARE P4 program on BMv2 (It does not emulate BMv2 !)
  • Though this installation is manual for pedagogic purpose, the installation can be fully automated, just fire up a VM with a bunch of interfaces and test it ! 
  • The installation proposed is highly resilient and will ease upgrade of the appliance (we will see in subsequent article what it means (wink) )

In the next article, we will configure the freeRouter appliance, start the router, and provide configuration in order to have effective basic ping reachability to the FTTH BROADBAND internal IP.

The "RARE/FreeRouter-101" series of articles is meant to help you quickly kickstart your very first RARE/freeRouter deployment and understand via a series of tutorials how it can be powered by various dataplanes. 101 article series also explained how RARE/freeRouter could be configured in order to be integrated into the external network environment. 101- [ #006 ] introduced an interesting solution for SOHO (small office/home office). You'll see in this "RARE validated design" series of articles,  an innovative implementation of a SOHO routing platform. These articles will draw your attention to an exceptional SOHO router with features usually implemented only by commercial solutions in service provider environments.

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

Back in 2004, I deployed a 8Mbps ATM circuit that connected an airline company hub site. Traffic growth increased amazingly since then! In 2020, what does SOHO (Small Office, Home Office) mean nowadays? In our use case we will consider a SOHO connected at 1GE link. This is for example:

  • Primary schools, Secondary schools
  • Small R&E institution spoke sites
  • Home office (especially considering the COVID context)
  • Small company spoke agencies

Article objective

In this article we will describe how to build a carrier grade SOHO router (aka CPE) from an actual real platform. In this example let me share with you my personal story and introduce you my SOHO hardware that I'm using at home. It is compliant with the requirements implied by the use cases listed above:

Requirements

  • requirement #0: n×1GE capable, ISP uplink is 1GE 
  • requirement #1: completely silent, the box can be moved to crowded room
  • requirement #2: small power consumption, as it is meant to run 24x7. (I'm paying the bill ! (smile) )
  • requirement #3: Run 64-bit linux 
  • requirement #4: native support of DPDK

Diagrams

[ #001 ] - Cookbook

 Hardware selection

Hardware specification

  • 6× Intel 211AT Gigabit Ethernet, support wake up on LAN
  • Support 1× mSATA SSD, 1x DDR3L 1.35V memory 1333/1600MHz, max to 8GB;
  • 1× VGA max resolution 1920x1080P
  • 1× COM RJ45 console
  • Support add WiFi module ( Mini PCI-E half height size )
  • Support automatically power on after power restore.
  • Ultra compact measured at 180×175×34mm;
  • Low power requirements save money and be more eco-friendly.
  • Fanless, passive cooling, noise-less

CPU specification

  • CPU identifier: J1900
  • of cores: 4

  • # of Threads: 4

  • Processor Base Frequency: 2.00 GHz

  • Burst Frequency: 2.42 GHz

  • Cache: 2 MB L2 Cache

  • TDP: 10 W

freeRouter is heavily multithreaded, so for 4 cores is appreciated, as a budget SOHO router, VPN hardware NIC assistance is not required. If VPN concentrator is needed, we can deploy in a SOHO environment a dedicated box that has a CPU with AES-NI support. freeRouter won't run as a VM, so VT-x nor VT-d and VT-c is not required.  

SOHO usage

  • home office work
  • regular 720p/1080p/4K (and more) on-line VC via RENATER RENDEZ-VOUS or ZOOM
  • (intensive grown up kids) online gaming (2–3 persons can play an online game at the same time)
  • these kids+wife can multitask and watch 480p/780p Youtube video at the same times (This is the digital natives ...)
  • streaming video from MyCanal (French Netflix competitor)
  • Operating system/school educational material  parallel downloads
  • Intensive social network usage via native mobile client having integrated video in the apps ...

Bandwidth check

So all the above usage require a high amount of connectivity as all of the action above can occur in parallel. This is Speedtest test result during crowded working hours:

So my ISP was not totally lying after all, though I could not reach the theoretical 1GE that the ISP advertisement boasts. (wink)

SOHO comments

Please note that this hardware has no optical/SFP port. There are indeed similar configuration with 1 optical uplink port in case you are also the service provider in your environment. This hardware is specific to FTTH environment currently deployed in France.

 Operating system selection

Operating system specification

  • Debian 10 (aka Buster) 
  • netinstall is used
  • minimal vanilla installation

Requirements

  • requirement #0: LTS operating system 
  • requirement #1: Benefit from LTS security patches
  • requirement #2: Must be able to run dpdk
  • requirement #3: (personal requirement) Must be familiar to me
  • requirement #4: Able to run java software as freeRouter is written in Java
  • requirement #5: small operating system software footprint
  • requirement #6: Support for IPv4/IPv6

Additional nice to have features (but not used here as we are not using VM nor require high VPN traffic load)

  • Virtualisation support: Check CPU support for VT-x (intel) AMD-V (AMD) 
  • I/O MMU virtualisation (Kernel bypass mechanism): Check CPU support for VT-d AMD-Vi (AMD) needed by dpdk with VFIO driver in order to ensure hardware NIC packet forwarding
  • Network virtualisation: Check CPU support for VT-c  (SR-IOV)
  • Hardware Encryption: Check CPU support for AES-NI (Tunnel mechanism using AES such as OpenVPN, however this is useless for other tunnel type such as Wireguard

Discussion

 Design choice considerations

Though the traffic distribution is totally different from a school or SOHO site traffic patterns, we can consider this hardware platform as a viable choice.

Platform considerations:

  • each 1GE port is wired to an Intel 211AT chipset. DPDK will take advantage of these chipset packet processing power burnt into the silicon in order to relieved the CPU load.
  • WIFI is not mandatory and the hardware included is not bleeding edge but considering the uplink bandwidth 802.11ax is not necessary. At least for Northbound traffic we are safe for the moment. At some points if East-West traffic such as NAS to wifi client require 10G traffic rate it will be the moment to buy a new appliance. If WIFI improvement is needed, 802.11ac card can be purchased with a 15€ budget. For WIFI client to WIFI client traffic 10GE traffic you can still purchase a 802.11ax mini PCIe card for around the same budget.

 freeRouter is supported on:

  • linux based system
  • android → yes, you can install freeRouter on your mobile phone and wander around your house, IPv4/IPv6 WIFI roaming will occur automagically!
  • freeRouter has a DPDK dataplane as well as a libpcap dataplane for older hardware
  • in this example I selected an appliance for convenient reasons but nothing prevents you from recycling an old laptop/desktop PC with multiple DPDK NICs. We can run a small PE (provider edge) router with multiple 1GE/10GE NICs. Note that the appliance can act as a 6x1GE provider edge router. This is the edge of the MPLS Seamless architecture.

Operating system future considerations:

  • In SP environment, the ideal situation is to have a custom Operating System (We are studying the Yocto project in order to create this custom OS)
  • This custom OS will encompasses the strict minimum software thus reducing the software footprint at its minimum
  • A very promising and unique features is also provided by: NixOS/Nix package manager : This will enable atomic commit/rollback at the package management level

The combination of Yocto + Nix can help develop your own specific DIY hardware (or for your company/organisation/institution) based on the popular concept that French ISPs love: "INTERNET BOX"

Conclusion

In this 1st article you:

  • had a brief description hardware platform suitable for SOHO
  • had a description of the SOHO use case in 2020
  • get a rationale on why this platform has been chosen
  • had a brief description of the selected Operating System
  • get a rationale on why this OS has been chosen

RARE validated design: [ SOHO #001 ] - key take-away

  • RARE/FreeRouter is a strong candidate for SOHO with multiple dataplane support solution.

If you are a company you run RARE/freeRouter with a versatile P4 switch such as STORDIS BF25561X-1T or WEDGE, but as a SOHO with a small budget you can run it with a DPDK dataplane and for older hardware you still have the possibility run it with a pure software dataplane

  • RARE/freeRouter is the first element at the very edge of the MPLS seamless architecture

End to end MPLS is now possible for the Service provider at an affordable price

  • RARE/freeRouter design can coexist with Virtualisation technology

CPU extension such as VT-x/AMD-V, VT-D/AMD-Vi, VT-c can provide coexistence between RARE/freeRouter and a small amount of storage and compute node. (Such as micro-K8/docker)

In the next article we will start our journey in creating a carrier grade CPE using the platform above.