Blog

This is additional post related to blog series called "RARE software architecture". As its name implies, it deals with topics related to RARE/freeRouter software design choice.

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

In [Modular design #001] article we described RARE/freeRtr main components:

image2020-9-21_12-19-56.png

  • a control plane
  • a data plane which can have different "flavor"
  • and a message API interface that is tying the 2 components above

Article objective

In most of our past articles, all the components listed above were running on the same host as it is mostly the case on traditional monolithic hardware. In this article we demonstrate that RARE/freeRtr inherently exposes a totally disaggregated model. Practically, what does this means ? We will show in subsequent section that each of the components above can run on different hosts.

Note

While the control plane and the message API interface can be run from any host such as bare metal server, VM or even container, the dataplane of your choice will have to run on specific platform. For example, if you plan to implement a core backbone MPLS router able to switch 6.4 Tbps of packet throughput, the dataplane must be a specific hardware.  (e.g. powered by INTEL TOFINO switch ASIC) 

Diagram

In this example we will consider 2 routers named DAV0001 and DAV0101 respectively. Both if these routers are establishing an OSPF adjacency between them.

  • DAV0001 is a P4 switch powered INTEL TOFINO ASIC
  • DAV0101 is a SOHO RARE/freeRtr DELL VEP1445

Let's assume the logical figure below:

[ #002 ] - Modular design

In reality, we will run DAV0001 components in a different host respectively. We assume of course that there is connectivity between each of them.


Let's first start the INTEL TOFINO dataplane.

pcapInt between WEDGE100BF32X and freeRtr
root@rare:~# /opt/freertr/bin/start_bfswd.sh
remote-dav0001-hw.txt
bf_switchd: no process found
Using SDE /opt/bf_switchd
Using SDE_INSTALL /opt/bf_switchd/install
Setting up DMA Memory Pool
Using TARGET_CONFIG_FILE /opt/bf_switchd/install/share/p4/targets/tofino/bf_router.conf
Using PATH /opt/bf_switchd/install/bin:/opt/freertr/bin:/opt/bf_switchd/install/bin:/opt/bf_switchd:/opt/bf_switchd/tools:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Using LD_LIBRARY_PATH /usr/local/lib:/opt/bf_switchd/install/lib:
bf_sysfs_fname /sys/class/bf/bf0/device/dev_add
kernel mode packet driver present, forcing kernel_pkt option!
Install dir: /opt/bf_switchd/install (0x55ac03623d20)
bf_switchd: system services initialized
bf_switchd: loading conf_file /opt/bf_switchd/install/share/p4/targets/tofino/bf_router.conf...
bf_switchd: processing device configuration...
Configuration for dev_id 0
  Family        : tofino
  pci_sysfs_str : /sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0
  pci_domain    : 0
  pci_bus       : 5
  pci_fn        : 0
  pci_dev       : 0
  pci_int_mode  : 1
  sbus_master_fw: /opt/bf_switchd/install/
  pcie_fw       : /opt/bf_switchd/install/
  serdes_fw     : /opt/bf_switchd/install/
  sds_fw_path   : /opt/bf_switchd/install/share/tofino_sds_fw/avago/firmware
  microp_fw_path: 
bf_switchd: processing P4 configuration...
P4 profile for dev_id 0
num P4 programs 1
  p4_name: bf_router
  p4_pipeline_name: pipe
    libpd: 
    libpdthrift: 
    context: /opt/bf_switchd/install/share/tofinopd/bf_router/pipe/context.json
    config: /opt/bf_switchd/install/share/tofinopd/bf_router/pipe/tofino.bin
  Pipes in scope [0 1 2 3 ]
  diag: 
  accton diag: 
  Agent[0]: /opt/bf_switchd/install/lib/libpltfm_mgr.so
  non_default_port_ppgs: 0
  SAI default initialize: 1 
bf_switchd: library /opt/bf_switchd/install/lib/libpltfm_mgr.so loaded
bf_switchd: agent[0] initialized
Health monitor started 
Operational mode set to ASIC
Initialized the device types using platforms infra API
ASIC detected at PCI /sys/class/bf/bf0/device
ASIC pci device id is 16
Skipped pkt-mgr init 
Starting PD-API RPC server on port 9090
bf_switchd: drivers initialized
bf_switchd: dev_id 0 initialized
bf_switchd: initialized 1 devices
Adding Thrift service for bf-platforms to server
bf_switchd: thrift initialized for agent : 0
bf_switchd: spawning cli server thread
bf_switchd: spawning driver shell
bf_switchd: server started - listening on port 9999
bfruntime gRPC server started on 0.0.0.0:50052

        ********************************************
        *      WARNING: Authorised Access Only     *
        ********************************************
    

bfshell> 

From that point the dataplane has activated its P4 CPU_PORT which is in our case ens1 Linux interface that appeared when you activate bf_kpkt modules.

Now the idea is to stitch ens1 CPU_PORT to freeRtr dataplane port and make sure that control plane / dataplane in band communication can occur. The secret sauce main ingredient is  pcapInt tool. In essence pcapInt is a freeRtr tool that will allow you to bind all the packet from an interface over UDP socket tunnel.

pcapInt between WEDGE100BF32X and freeRtr
root@rare:~# /opt/freertr/bin/pcapInt.bin ens1 20666 193.49.159.159 20666 193.49.159.237

So, the line above can be read as: tunnel all packet from ens1 interface over UDP tunnel whose source is 193.49.159.237:20666 (bf_switchd@p4 switch) and destination is 193.49.159.159:20666 (freeRtr@server)

pcapInt between WEDGE100BF32X and freeRtr output
binded to local port 193.49.159.237 20666.
will send to 193.49.159.159 20666.
pcap version: libpcap version 1.10.0 (with TPACKET_V3)
opening interface ens1
serving others
> d
iface counters:
                      packets                bytes
received                    0                    0
sent                        0                    0

>

DAV0001 hardware configuration file:

remote-dav0001-hw.txt
hwid remote-wedge100bf32x
port 22000 64000
tcp2vrf 2323 p4 23
tcp2vrf 2001 p4 22
tcp2vrf 9080 p4 9080
int eth0 eth 0000.0bad.c0de 193.49.159.159 20666 193.49.159.237 20666

DAV0001 software configuration file:

remote-dav0001-hw.txt
hostname DAV0001
buggy
!
logging buffered debug 10240
logging file debug /var/log/freertr.log
logging rotate 655360000 /var/log/freertr.old
!
prefix-list all4
 sequence 10 permit 0.0.0.0/0 ge 0 le 0
 exit
!
prefix-list all6
 sequence 10 permit ::/0 ge 0 le 0
 exit
!
vrf definition inet
 exit
!
vrf definition oob
 exit
!
vrf definition p4
 description P4 VRF _NEVER_EVER_ CONFIGURE IT
 exit
!
router ospf4 1
 vrf inet
 router-id 10.1.1.1
 traffeng-id 0.0.0.0
 area 0 enable
 exit
!
interface loopback0
 no description
 vrf forwarding inet
 ipv4 address 10.1.1.1 255.255.255.255
 no shutdown
 no log-link-change
 exit
!
interface ethernet0
 description CPU_PORT _NEVER_EVER_ CONFIGURE IT
 no shutdown
 no log-link-change
 exit
!
interface sdn10
 description frontpanel port 10/0
 mtu 1500
 macaddr 0031.755e.0363
 lldp enable
 vrf forwarding inet
 ipv4 address 10.1.12.1 255.255.255.0
 router ospf4 1 enable
 router ospf4 1 area 0
 router ospf4 1 cost 10000
 no shutdown
 log-link-change
 exit
!
interface sdn15
 description frontpanel port 15/0
 mtu 1500
 macaddr 0056.2158.6249
 lldp enable
 vrf forwarding inet
 ipv4 address 10.1.14.1 255.255.255.0
 router ospf4 1 enable
 router ospf4 1 area 0
 no shutdown
 log-link-change
 exit
!
interface sdn7
 description frontpanel port 7/0
 mtu 1500
 macaddr 0040.784a.0b38
 lldp enable
 vrf forwarding inet
 ipv4 address 10.1.13.1 255.255.255.0
 router ospf4 1 enable
 router ospf4 1 area 0
 no shutdown
 log-link-change
 exit
!
proxy-profile oob
 vrf oob
 exit
!
server telnet oob
 security protocol telnet
 exec logging
 no exec authorization
 no login authentication 
 login logging
 vrf oob
 exit
!
server telnet p4
 security protocol telnet
 exec logging
 no exec authorization
 no login authentication
 login logging
 vrf p4
 exit
!
server p4lang p4
 export-vrf inet 1
 export-port sdn15 12 10 0 0 0
 export-port sdn10 52 10 0 0 0
 export-port sdn7 176 10 0 0 0
 interconnect ethernet0
 vrf p4
 exit
!
client proxy oob
client name-server 1.1.1.1
client time-server europe.pool.ntp.org
client time-zone CET
!
end
remote-dav0001-hw.txt
root@vep1425:~# java -jar /rtr/rtr.jar routersc remote-dav0001-hw.txt remote-dav0001-sw.txt
remote-dav0001-hw.txt
root@vep1425:~# java -jar /rtr/rtr.jar routersc remote-dav0001-hw.txt remote-dav0001-sw.txt


#######                         ##################
 ##  ##                                 ##
 ##   # ## ###   #####   #####  ## ###  ## ## ###
 ## #    ### ## ##   ## ##   ##  ### ## ##  ### ##
 ####    ##  ## ####### #######  ##  ## ##  ##  ##
 ## #    ##     ##      ##       ##     ##  ##
 ##      ##     ##   ## ##   ##  ##     ##  ##
####     ##      #####   #####   ##     ##  ##

freeRouter v21.9.23-cur, done by cs@nop.

place on the web: http://www.freertr.net/
license: http://creativecommons.org/licenses/by-sa/4.0/
quote1: make the world better
quote2: if a machine can learn the value of human life, maybe we can too
quote3: be liberal in what you accept, and conservative in what you send
quote4: the beer-ware license for selected group of people:
cs@nop wrote these files. as long as you retain this notice you
can do whatever you want with this stuff. if we meet some day, and
you think this stuff is worth it, you can buy me a beer in return

info cfg.cfgInit.doInit:cfgInit.java:662 booting
info cfg.cfgInit.doInit:cfgInit.java:806 initializing hardware
info cfg.cfgInit.doInit:cfgInit.java:812 applying defaults
info cfg.cfgInit.doInit:cfgInit.java:819 applying configuration
warning ifc.ifcEthTyp.propagateState:ifcEthTyp.java:334 interface sdn15 change to up
warning ifc.ifcEthTyp.propagateState:ifcEthTyp.java:334 interface sdn10 change to up
warning ifc.ifcEthTyp.propagateState:ifcEthTyp.java:334 interface sdn7 change to up
info cfg.cfgInit.doInit:cfgInit.java:849 boot completed
welcome
line ready
DAV0001#                                              

Now it is the time to establish the interface between the control plane and the dataplane

remote-dav0001-hw.txt
root@rare:~# /opt/freertr/bin/bf_forwarder.py --freerouter-address 193.49.159.159 --freerouter-port 9080 --bfruntime-address 193.49.159.237:50052  

basically you can read this command as follow:

  • bf_forwarder.py is running 193.49.159.236
  • it is binding freeRtr control plane running at 193.49.159.159
  • and P4 dataplane  running at 193.49.159.237

The output below is showing you a successful binding operation and the entries creation during control plane and dataplane communication.

remote-dav0001-hw.txt
bf_forwarder.py running on: WEDGE100BF32X
GRPC_ADDRESS: 193.49.159.237:50052
P4_NAME: bf_router
CLIENT_ID: 0
Subscribe attempt #1
Subscribe response received 0
Received bf_router on GetForwarding on client 0, device 0
Binding with p4_name bf_router
Binding with p4_name bf_router successful!!
bf_switchd started with no SNMP export
Generic table clearing: (Order not matters)
  Clearing Table pipe.eg_ctl.eg_ctl_vlan_out.tbl_vlan_out
  Clearing Table pipe.ig_ctl.ig_ctl_outport.tbl_vlan_out
  Clearing Table pipe.ig_ctl.ig_ctl_mcast.tbl_mcast6
  Clearing Table pipe.ig_ctl.ig_ctl_mcast.tbl_mcast4
  Clearing Table pipe.ig_ctl.ig_ctl_mpls.tbl_mpls_fib_decap
  Clearing Table pipe.ig_ctl.ig_ctl_vrf.tbl_vrf
  Clearing Table pipe.ig_ctl.ig_ctl_mpls.tbl_mpls_fib
  Clearing Table pipe.eg_ctl.eg_ctl_mcast.tbl_mcast
  Clearing Table pipe.eg_ctl.eg_ctl_nexthop.tbl_nexthop
  Clearing Table pipe.ig_ctl.ig_ctl_outport.tbl_nexthop
  Clearing Table pipe.ig_ctl.ig_ctl_ipv6.tbl_ipv6_fib_lpm
  Clearing Table pipe.ig_ctl.ig_ctl_ipv4.tbl_ipv4_fib_host
  Clearing Table pipe.ig_ctl.ig_ctl_ipv6.tbl_ipv6_fib_host
  Clearing Table pipe.ig_ctl.ig_ctl_vlan_in.tbl_vlan_in
  Clearing Table pipe.eg_ctl.eg_ctl_hairpin.tbl_hairpin
  Clearing Table pipe.ig_ctl.ig_ctl_ipv4.tbl_ipv4_fib_lpm
Bundle specific clearing: (Order matters)
  Clearing Table pipe.ig_ctl.ig_ctl_bundle.tbl_nexthop_bundle
  Clearing Table pipe.ig_ctl.ig_ctl_bundle.ase_bundle
  Clearing Table pipe.ig_ctl.ig_ctl_bundle.apr_bundle
Multicast specific clearing: (Order matters)
  Clearing Table $pre.mgid
  Clearing Table $pre.node
BfForwarder - Main
BfForwarder - Entering message loop
BfIfStatus - main
rx: ['state', '12', '1', '10', '0', '0', '0', '\n']
BfSubIfCounter - main
BfSubIfCounter - No active ports
BfIfStatus - No active ports
rx: ['mtu', '12', '1500', '\n']
rx: ['portvrf_add', '12', '1', '\n']
rx: ['tcpmss4in_add', '12', '0', '\n']
rx: ['tcpmss4out_add', '12', '0', '\n']
rx: ['state', '52', '1', '10', '0', '0', '0', '\n']
rx: ['mtu', '52', '1500', '\n']
rx: ['portvrf_add', '52', '1', '\n']
rx: ['tcpmss4in_add', '52', '0', '\n']
rx: ['tcpmss4out_add', '52', '0', '\n']
rx: ['state', '176', '1', '10', '0', '0', '0', '\n']
rx: ['mtu', '176', '1500', '\n']
rx: ['portvrf_add', '176', '1', '\n']
rx: ['tcpmss4in_add', '176', '0', '\n']
rx: ['tcpmss4out_add', '176', '0', '\n']
rx: ['myaddr4_add', '224.0.0.0/4', '-1', '1', '\n']
rx: ['myaddr4_add', '255.255.255.255/32', '-1', '1', '\n']
rx: ['myaddr6_add', 'ff00::/8', '-1', '1', '\n']
rx: ['myaddr4_add', '10.1.1.1/32', '-1', '1', '\n']
rx: ['myaddr4_add', '10.1.12.0/24', '52', '1', '\n']
rx: ['myaddr4_add', '10.1.12.1/32', '52', '1', '\n']
rx: ['myaddr4_add', '10.1.13.0/24', '176', '1', '\n']
rx: ['myaddr4_add', '10.1.13.1/32', '176', '1', '\n']
rx: ['myaddr4_add', '10.1.14.0/24', '12', '1', '\n']
rx: ['myaddr4_add', '10.1.14.1/32', '12', '1', '\n']
rx: ['mylabel6_add', '126184', '1', '\n']
rx: ['mylabel4_add', '835737', '1', '\n']
BfIfStatus - PORTS_OPER_STATUS[176] does not exist, adding it ...
BfIfStatus - PORTS_OPER_STATUS[52] does not exist, adding it ...
BfIfStatus - PORTS_OPER_STATUS[12] does not exist, adding it ...
tx: ['state','176','1','\n']
rx: ['neigh4_add', '32360', '10.1.13.3', '00:5b:09:4d:1e:40', '1', '00:40:78:4a:0b:38', '176', '\n']
rx: ['nhop2port_add', '32360', '176', '176', '\n']
rx: ['route4_add', '10.1.3.3/32', '32360', '10.1.13.3', '1', '\n']
rx: ['keepalive', '\n']
...

Verification

Once bf_forwader.py established the connection with the control plane and the dataplane, freeRtr should expose  "session UP" message on console

remote control plane outout
welcome
line ready
DAV0001#warning serv.servP4lang.srvAccept:servP4lang.java:597 neighbor 193.49.159.236 up

Check that you are sending and receiving packet from freeRtr@eth0 and P4_switch@CPU_PORT:

remote control plane outout
DAV0001#sh int sum                                                                                                             
interface  state  tx       rx           drop
loopback0  up     0        0            0
ethernet0  up     34946    12232        0
sdn10      up     10928+0  0+0          0+0
sdn15      up     10928+0  0+0          0+0
sdn7       up     12214+0  11930+14346  0+0

LLDP is your friend

Physical connectivity verification
DAV0001#show lldp neighbor                                                                                                     
interface  hostname  iface  ipv4       ipv6
sdn7       DAV0101   sdn6   10.1.13.3  null

In the above configuration we have enabled OSPFv4 adjacency via sdn7 with DAV0101@sdn6

Control plane verification
DAV0001#sh ipv4 ospf 1 int                                                                                                     
interface  neighbors
sdn10      0
sdn15      0
sdn7       1

DAV0001#show ipv4 ospf 1 neighbor                                                                                              
interface  area  address    routerid  state  uptime
sdn7       0     10.1.13.3  10.1.3.3  4      00:05:18

DAV0001#sh ipv4 route inet                                                                                                     
typ  prefix        metric  iface      hop        time
C    10.1.1.1/32   0/0     loopback0  null       00:08:13
O    10.1.3.3/32   110/1   sdn7       10.1.13.3  00:07:32
C    10.1.12.0/24  0/0     sdn10      null       00:08:12
LOC  10.1.12.1/32  0/1     sdn10      null       00:08:12
C    10.1.13.0/24  0/0     sdn7       null       00:08:12
LOC  10.1.13.1/32  0/1     sdn7       null       00:08:12
C    10.1.14.0/24  0/0     sdn15      null       00:08:12
LOC  10.1.14.1/32  0/1     sdn15      null       00:08:12

The important thing to note is that we are pinging from a control plane that is running on a different host as the dataplane. Nothing special here except that this seemed to be totally invisible from the operator point of view !

mandatory ping
DAV0001#ping 10.1.3.3 /vrf inet                                                                                                
pinging 10.1.3.3, src=null, vrf=inet, cnt=5, len=64, tim=1000, gap=0, ttl=255, tos=0, fill=0, sweep=false, multi=false, detail=false
!!!!!
result=100%, recv/sent/lost/err=5/5/0/0, rtt min/avg/max/total=1/1/2/9
DAV0001#                                                                                                                       

Discussion

This modular design and its  property to run as fully disaggregated model is powerful as:

  • each component are inter-changeable, connecting freeRtr to another dataplane is a seamless operation
  • it can be the foundation of a decentralized architecture where the control plane is not attached to the dataplane
  • Resiliency has of course to be thouroughly strenghtened

Conclusion

In this 2nd article you:

  • had a showcase on how to implement a fully disaggregated RARE/freeRtr
  • even if the control plane and the interface can be run almost anywhere, the dataplane still needs to be specific and adapted to the use case you planned to deploy
  • pcapInt tool is a nitty gritty tool used to bind existing ports to a UDP socket tunnel.





Starting from SOHO #008, everyone inside home network can:

  • reach the Internet thanks to NAT translation described in SOHO #004 
  • reach now their favorite Internet service by name thanks for SOHO #005
  • and get IPv4 (or IPv6) address from DHCP server (we took as a switch connected to sdn6)
  • this applies to all wired host but also all mobile host connected to the home wireless network
  • And you get a Great UPNP server forwarder

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

Wireguard gained a lot popularity and attention when it went into Linux kernel in March 2020. And pure coincidence or not, 1 month later, Wireguard made its way through ubuntu 20.04 LTS. Csaba MATE, freeRtr maintainer took "the Taurus by his horns" (sorry for the pure French translation of the proverb (smile)) and in turn, he added Wireguard support into freeRtr few months later ...

In the current pandemic context, Wireguard is highly used to establish software VPN connectivity toward office resources. In my context, among other usages, I'm using it in order to reach DN42, an overlay network very good for learning protocol like BGP and explore routing technology.

Article objective

There are plenty of web resources debating Wireguard based VPN topics. The purpose of this article will simply put the focus on how to configure a Wireguard tunnel between:

  • ubuntu 20.04 host
  • RARE/freeRtr

Diagrams

In this example, I'll use my home router as Wireguard tunnel end point and a ubuntu 20.04 server in my home LAN.

The steps at RARE/freeRtr level are:

  • configure vrf v1234.
  • configure loopback1234 with IPv6 network 1234::666/64 at freeRtr level and bind it to vrf v1234
  • configure Wireguard crypto stanza
  • configure tunnel1234
    • with Wireguard crypto stanza defined above,
    • configure tunnel source, destination underlay in vrf inet
    • finally configure tunnel interface in vrf v1234
  • As RARE/freeRtr is oriented toward the future we will just forget about legacy IPv4, therefore the underlay tunnel will be established with IPv6 only. (←Joke (smile))

Tip

As we are running RARE/freeRtr with DPDK dataplane do not forget to update server p4lang p4 stanza:

  • add vrf v1234
  • add tunnel1234

[ SOHO #009 ] - "Network 1234::/64 ? Please follow tunnel1234 Wireguard tunnel !"

  • Install Wireguard
Update system and install Wireguard
sudo apt-get update
sudo apt-get upgrade
sudo apt install wireguard
  • generate for ubuntu private/public key pair
Generate key private/public key pair for ubuntu-20.04
umask 077
wg genkey > ubi20-private-key
wg pubkey < ubi20-private-key > ubi20-public-key
  • generate for freeRtr private/public key pair
Generate private/public key pair for freeRtr
umask 077
wg genkey > freertr-private-key
wg pubkey < freertr-private-key > freertr-public-key
  • edit ubuntu 20.04 Wireguard configuration
vi /etc/wireguard/wg0.conf
cat /etc/wireguard/wg0.conf 
[Interface]
Address = 6789::10/64 
PrivateKey = <ubuntu-private-key> 

ListenPort = 51820

[Peer]
PublicKey = <freertr-public-key> 
AllowedIPs = 1234::/64,6789::/64 
Endpoint = 2a01:e0a:159:2856::6:51820 
  • Activate wireguard tunnel
As root
wg-quick up wg0
  • First thing first configure vrf v1234
vrf v1234 configuration
!
vrf definition v1234
 exit
!
  • Configure loopback1234 and bind it to vrf v1234
Loopback1234 configuration
!
interface loopback1234
 description Wireguard allowed IPv6 network
 vrf forwarding v1234
 ipv6 address 1234::1 ffff:ffff:ffff:ffff::
 no shutdown
 no log-link-change
 exit
!
  • Configure Wireguard crypto stanza
vrf v1234 configuration
!
crypto ipsec wg-1234
 key <freertr-private-key><ubuntu-public-key>
 replay 0
 exit
!
  • Configure tunnel1234 Wireguard tunnel
Wireguard tunnel1234 configuration
!
interface tunnel1234
 description RARE/freeRtr Wireguard tunnel 
 tunnel key 51820
 tunnel vrf inet
 tunnel protection wg-1234
 tunnel source sdn6
 tunnel destination 2a01:e0a:159:2856:a00:27ff:fe5f:f085
 tunnel mode wireguard
 vrf forwarding v1234
 ipv6 address 6789::666 ffff:ffff:ffff:ffff::
 no shutdown
 no log-link-change
 exit
! 
  • if you are using a dataplane please add Wireguard tunnel and corresponding VRF in "server p4lang p4"
sh run p4lang
!
server p4lang p4
...
 export-vrf v1234 5
...
 export-port tunnel1234 45 0 0 0 0
...
 exit
!
! Where 45 is an ID not already used in server p4lang 

Verification

  • Check Wireguard  interface
ifconfig wg0
wg0: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 1420
        inet6 6789::10  prefixlen 64  scopeid 0x0<global>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 1000  (UNSPEC)
        RX packets 980  bytes 44240 (44.2 KB)
        RX errors 204  dropped 0  overruns 0  frame 204
        TX packets 191  bytes 15768 (15.7 KB)
        TX errors 0  dropped 64 overruns 0  carrier 0  collisions 0
  • Check ipv6 routing @ ubuntu 
ip -6 route
::1 dev lo proto kernel metric 256 pref medium
1234::/64 dev wg0 metric 1024 pref medium
2a01:e0a:159:2856::/64 dev enp0s3 proto ra metric 100 expires 2419185sec pref medium
6789::/64 dev wg0 proto kernel metric 256 pref medium
fe80::/64 dev enp0s3 proto kernel metric 256 pref medium
fe80::/64 dev enp0s8 proto kernel metric 256 pref medium
default via fe80::24c:73ff:fe07:a77 dev enp0s3 proto ra metric 100 expires 1785sec mtu 1500 pref medium
  • ping freeRtr tunnel endpoint
ping 6789::666
PING 6789::666(6789::666) 56 data bytes
64 bytes from 6789::666: icmp_seq=1 ttl=255 time=2.07 ms
64 bytes from 6789::666: icmp_seq=2 ttl=255 time=1.99 ms
64 bytes from 6789::666: icmp_seq=3 ttl=255 time=2.20 ms
64 bytes from 6789::666: icmp_seq=4 ttl=255 time=2.26 ms
  • ping cascaded loopback1234
ping 1234::1
PING 1234::1(1234::1) 56 data bytes
64 bytes from 1234::1: icmp_seq=1 ttl=255 time=2.04 ms
64 bytes from 1234::1: icmp_seq=2 ttl=255 time=1.92 ms
64 bytes from 1234::1: icmp_seq=3 ttl=255 time=2.10 ms
64 bytes from 1234::1: icmp_seq=4 ttl=255 time=1.96 ms

Congratulations ! 

Now let's proceed to verification at freeRtr level

  • Check Wireguard  interface
show interface tunnel1234
tunnel1234 is up (since 05:18:05, 1 changes)
 description: 
 type is wireguard, hwaddr=none, mtu=1400, bw=8000kbps, vrf=v1234
 ip4 address=7.8.9.254/24, netmask=255.255.255.0, ifcid=1052271466
 ip6 address=6789::666/64, netmask=ffff:ffff:ffff:ffff::, ifcid=823043043
 received 28 packets (2488 bytes) dropped 0 packets (0 bytes)
 transmitted 665 packets (51318 bytes) promisc=false macsec=false
  • Check ipv6 routing @ freeRtr
show ipv6 route v1234
typ  prefix         metric  iface         hop   time
C    1234::/64      0/0     loopback1234  null  02:20:16
LOC  1234::1/128    0/1     loopback1234  null  02:20:16
C    6789::/64      0/0     tunnel1234    null  01:25:49
LOC  6789::666/128  0/1     tunnel1234    null  01:25:49
  • Check IPv6 underlay reachability in vrf inet
show ipv6 route inet
typ  prefix                      metric  iface      hop                   time
S    ::/0                        1/0     sdn1       2a01:e0a:159:2850::1  05:21:27
...
C    2a01:e0a:159:2856::/64      0/0     sdn6       null                  05:21:27
LOC  2a01:e0a:159:2856::6/128    0/1     sdn6       null                  05:21:27
...
  • ping ubuntu 20.04 tunnel endpoint
ping 6789::10 /vrf v1234
pinging 6789::10, src=null, vrf=v1234, cnt=5, len=64, tim=1000, gap=0, ttl=255, tos=0, fill=0, sweep=false, multi=false, detail=false
!!!!!
result=100%, recv/sent/lost/err=5/5/0/0, rtt min/avg/max/total=2/2/2/10

Congratulations ! 

Conclusion

In this article we learned how to configure a Wireguard tunnel between ubuntu 20.04 and  RARE/freeRouter. Configuring Wireguard is a simple 2 steps operation:

  • Configure Wireguard crypto stanza
  • and bind it to a Wireguard tunnel


RARE validated design: [ SOHO #009 ] - key take-away

In this example the key take-away are:

  • Wireguard configuration is simple
  • We used ubuntu 20.04 but of course you can enabled Wireguard with any host supporting Wireguard protocol
  • IPv6 is used here. But of course IPv4 is also available (wink)
  • Last but not least do not forget to export VRF and Wireguard tunnel interface in "server p4lang p4" should you run RARE/freeRtr with a dedicated dataplane (p4emu, dpdk)

This is a pretty unique freeRouter feature ! Are you aware of any other router able to set up a Wireguard VPN ?

This is a new category of article that falls under "RARE software architecture" special blog series. As its name implies, it deals with topics related to Operating System is certified RARE/freeRouter software can runs.

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

 

This comes alongside "Winter" as a famous TV show would say. (smile). Mion is an interesting Operating System meant to run on network appliance. It is built on top of Yocto project and inherently share its root with Linux Open Embedded system. In order to learn more about the Operating system I invite you to read Mion project page.

Why consider such an OS you would say ? It actually boils down to:

  1. having a minimalistic Operating system with the smallest software footprint.
  2. predictable Operating System with the minimum software requirement
  3. A supported Operating System powered by a regular release cycle


Note

While trying to reach a minimalistic image in order to run the RARE software, for now we only managed to have a 5Gb image (with INTEL SDE installed). We are confident that we can still slimmed down the image but times is running against us. (smile) The very initial Mion image has a size of 45Mbytes !

Tip

Mion will support multiple network equipments ranging from APS bf2556x-1t, bf6064x, Edgecore wedge100bf32x and wedge100bf65x and many more (Delta, asgvolt64 etc.)

Article objective

In this article, we will present :

  • how to build Mion from scratch
  • how to install it on our favorites P4 switches (bf2556x-1t and wedge-100bf-32x)

[ #001 ] - Cookbook

Mion docs has an excellent getting started page. In this section we will emphasize how easy it is to build Mion. First, make sure you have a machine compliant to the "Build host" requirements. Install a pristine Debian 10 and you should be go to go ...

Install Yocto software requirement (5 minutes depending on your Internet connection)
sudo apt-get install gawk wget git-core diffstat unzip texinfo gcc-multilib \
     build-essential chrpath socat cpio python3 python3-pip python3-pexpect \
     xz-utils debianutils iputils-ping python3-git python3-jinja2 libegl1-mesa \
     libsdl1.2-dev pylint3 xterm
Get Mion softwares (1 minute)
git clone --recursive git@github.com:NetworkGradeLinux/mion.git
cd mion
# To obtain related mion layers:
git clone git@github.com:NetworkGradeLinux/meta-mion.git
# Obtain the mion hardware layer
git clone git@github.com:NetworkGradeLinux/meta-mion-bsp.git


Note

The above steps require you have a GitHub account and uploaded public key to your account. You'll also have to activate ssh-agent as the build script will fetch software from Git using SSH transport.

Source Yocto Build environment variables (5 seconds)
source openembedded-core/oe-init-build-env
Launch APS bf2556x-1t ONIE build (can be very long if you have a small build host ... It is time to do something else ...)
    time ../mc_build.sh -m stordis-bf2556x-1t -h host-onie:mion-onie-image-onlpv1 
(EDIT) Reward after a long build process ... (~151 minutes on a 4 vCPU VM)
Sstate summary: Wanted 1251 Found 0 Missed 1251 Current 0 (0% match, 0% complete)
NOTE: Executing Tasks
WARNING: mc:host-onie:expat-native-2.2.9-r0 do_fetch: Failed to fetch URL https://downloads.sourceforge.net/expat/expat-2.2.9.tar.bz2, attempting MIRRORS if available
NOTE: Tasks Summary: Attempted 3161 tasks of which 1 didn't need to be rerun and all succeeded.

Summary: There was 1 WARNING message shown.

real    150m43.758s
user    0m16.407s
sys     0m3.836s

Tip

Even if the build process can be long depending on your build host, build duration can reach 30 minutes with a 32 cores machines. Subsequent builds will take significantly less time as a cache mechanism will avoid to to redo the whole build process.

Believe me, you are done !

Just wait the end of the build process and grab the freshly baked image.

Let's relaunch the build again ... (~27 seconds on a 4 vCPU VM) Thanks sstate cache !
Sstate summary: Wanted 0 Found 0 Missed 0 Current 1251 (0% match, 100% complete)
NOTE: Executing Tasks
NOTE: Tasks Summary: Attempted 3161 tasks of which 3161 didn't need to be rerun and all succeeded.

real    0m27.430s
user    0m1.939s
sys     0m0.585s
Get the freshly baked Mion image !
p4@mion-build-host:~/mion/build$ cd tmp-glibc/deploy/images/stordis-bf2556x-1t/
p4@mion-build-host:~/mion/build/tmp-glibc/deploy/images/stordis-bf2556x-1t$ ls
bzImage
bzImage--5.4.49+git0+d626f9108d_ec485bd4af-r0-stordis-bf2556x-1t-20201216195930.bin
bzImage-stordis-bf2556x-1t.bin
grub-efi-bootx64.efi
mion-onie-image-onlpv1-stordis-bf2556x-1t-20201216195930.rootfs.ext4
mion-onie-image-onlpv1-stordis-bf2556x-1t-20201216195930.rootfs.manifest
mion-onie-image-onlpv1-stordis-bf2556x-1t-20201216195930.rootfs.tar.bz2
mion-onie-image-onlpv1-stordis-bf2556x-1t-20201216195930.rootfs.tar.xz
mion-onie-image-onlpv1-stordis-bf2556x-1t-20201216195930.rootfs.tar.xz.bmap
mion-onie-image-onlpv1-stordis-bf2556x-1t-20201216195930.testdata.json
mion-onie-image-onlpv1-stordis-bf2556x-1t.ext4
mion-onie-image-onlpv1-stordis-bf2556x-1t.manifest
mion-onie-image-onlpv1-stordis-bf2556x-1t.tar.bz2
mion-onie-image-onlpv1-stordis-bf2556x-1t.tar.xz
mion-onie-image-onlpv1-stordis-bf2556x-1t.tar.xz.bmap
mion-onie-image-onlpv1-stordis-bf2556x-1t.testdata.json
modules--5.4.49+git0+d626f9108d_ec485bd4af-r0-stordis-bf2556x-1t-20201216195930.tgz
modules-stordis-bf2556x-1t.tgz
onie-installer.bin
onie-installer-x86_64-stordis_bf2556x_1t.bin

Of course get onie-installer-x86_64-stordis_bf2556x_1t.bin image and you are good to go for next section (smile)

  • Boot bf2556x-1t via ONIE rescue session
  • put you Mion image at the root of a web server 


Kickstart Mion ONIE installation (1 minutes)
onie-nos-install http://<your_web_server>/onie-installer-x86_64-stordis_bf2556x_1t.bin

Once the installation has finished, your system reboot and you'll be rewarded by Mion Grub welcome page:

Tip

And in just 1 minute (warning) you have installed a minimalistic OS on your bf2556x-1t. In this post we will stop here. However, in order to finalise the installation you'll to install INTEL P4 Software Development Environment as of course you'll want to run your P4 program on the bf2556x-1t. We will cover this subsequent installation in a subsequent post.


Conclusion

In this short article we saw:

  • how to build Mion 
  • how to install Mion (process is the same on every ONIE based system: we tested bf2556x-1t, bf6064x wedge100bf-32x, wedge100bf-64x)

Please let us know if you want us to test your P4 platform with the RARE P4 software. We will then add it to the RARE Hardware Compatibility List (RARE HCL) !

(In that case you might have to arrange access to your P4 equipment or better ... Send us a sample !   (smile) )

Final words

Mion build is quite straightforward. As it share its root from Yocto, adding specificities related to your project is easy. Most probably a Bitbake recipe already exist and you'll just have to add a reference to the layer of your interest in Bitbake config file. First time build can be long and tedious if you have a small build host machine but subsequent build will be faster thanks to the sstate cache mechanism. 

Last but not least, we covered only the base OS installation, now that you have a minimal OS required to manage your machine, you need to add specific drivers and software needed to run P4 programs on APS or WEDGE platform.






Starting from SOHO #007, everyone inside home network can:

  • reach the Internet thanks to NAT translation described in SOHO #004 
  • reach now their favorite Internet service by name thanks for SOHO #005
  • and get IPv4 (or IPv6) address from DHCP server (we took as a switch connected to sdn6)
  • this applies to all wired host but also all mobile host connected to the home wireless network

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

I'm very fond listening music at home and started my personal music collection. I was aware of this practice, and since then started 24/96 flac collection. As mechanical hard drives became more and more affordable, storage was not an issue for the NAS I deployed. I purchased then a refurbished but still decent DLNA network player (DMP) that would be able to stream music sources from DLNA server(DMS). All of this is orchestrated though my mobile phone acting as DLNA media controller (DMC). So far so good, but DLNA alliance expects that all DLNA protagonists are in the same and unique broadcast domain.

You might have guess already, but I'm not a huge FAN of having a single layer 2 domain at home and throw everything inside the same subnetwork. Granted the fact that not all home network are running OSPF/ ISIS MPLS or Segment Routing, everything is so simple and more predictable to manage at layer 3 whether you use IPv4 or IPv6.

Long story short, I can't listen music by structuring my home network using layer 3 as it is now ... (sad)

Article objective

In this article we will pursue the SOHO network appliance installation and enable THE UNIQUE freeRouter feature that you won't find in any router in the world: UPNP forwarder server.

Diagrams

source: DLNA German wikipedia 

[ SOHO #008 ] - "It's the end of L2 at my home... At last !"

The first step is to configure an UPNP hub server

Identify WIFI module hardware
server upnphub upnp
 interface loopback0
 vrf inet
 exit
  • As any other server it is bound to loopback0
  • And operate inside VRF inet

Now that you have an UPNP hub, we need to configure any upnp forwarder client where we expect to see any incoming UPNP/DLNA message

UPNP forwarder client @ integrated WIFI
server upnpfwd wifi4-appliance
 target 192.168.254.1
 interface hairpin12
 vrf inet
 exit

In this case, WIFI network we encompasses all mobile clients. My phone is acting as DMC here, but it could also be a DMS. The upnp forwarder:

  • has a target UPNP hub (we configured 102.168.254.1)
  • is bound to interface sdn2
  • operates within VRF inet
UPNP forwarder client @ integrated WIFI
server upnpfwd wired-media
 target 192.168.254.1
 interface sdn2
 vrf inet
 exit

In this precise case wired-media is another UPNP fowarder client that will forward all UPNP message to UPNP hub 192.168.254.1. As you might have guessed behind sdn2 is cascaded my NAS and DLNA network streamer (DMP or DMR)

Tip

In this example the key idea is:

  • Create a UPNP hub
  • Create a UPNP forwarder bound to any interface where you expect DLNA/UPNP communication
  • In this case DMS and DMR are behind sdn2
  • DMC is behind WIFI

Verification

In the screenshot below, the DLNA player is shutdown

Check that the App is able to discover the DLNA server DMS in other L3 subnet

Conclusion

In this article we enabled RARE/freeRouter DLNA/UPNP server/client:

  • You have now the possibility to position any DLNA devices on any L3 subnet of your home network
  • Bye bye flat single VLAN design
  • Not having a flat VLAN is desirable when you have lots of IPv4/IPv6 devices of different nature. You would want domotic/IoT devices not in the same subnet as your wifi client and have a separate subnet for you NAS or wired services


RARE validated design: [ SOHO #006 ] - key take-away

In this example the key take-away are:

  • Configure an UPNP hub
  • Configure UPNP forwarder on very interface you expect DLNA/UPNP communication
  • Of course this is IPv4/IPv6 compliant

This is a pretty unique freeRouter feature !


So everyone inside home network can:

  • reach the Internet thanks to NAT translation described in SOHO #004 
  • reach now their favorite Internet service by name thanks for SOHO #005
  • and get IPv4 (or IPv6) address from DHCP server (we took as a switch connected to sdn6)

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

In the previous articles we describe the hardware in SOHO #001 and initial staging steps in SOHO #002.

Note

You technically juts have to cut'n paste the config if you get the same hardware and operating system.

But if you pay attention, we did not cover one particular aspect of the hardware: integrated WIFI

Article objective

In this article we will pursue the SOHO network appliance installation and enable the integrated module so that all mobile host @ home can benefit from connectivity offered by SOHO router. In this basic example I'll show you basic WIFI implementation

  • WIFI is b/g/n module (therefore no 5Ghz wifi)
  • WIFI won't be directly under RARE/freeRouter control

Diagrams

[ SOHO #007 ] - "I'm not wired ... I need connectivity too !"

Log into linux appliance via management interface using previously configured veth1a IPv4 address: 192.168.128.1

Identify WIFI module hardware
lspci | grep -i wire
09:00.0 Network controller: Qualcomm Atheros AR928X Wireless Network Adapter (PCI-Express) (rev 01)

My google-fu indicated me that the lunux driver should be ath9k

Check integrated WIFI hardware driver is loaded
lsmod | grep ath
ath9k                 139264  0
ath9k_common           20480  1 ath9k
ath9k_hw              487424  2 ath9k_common,ath9k
ath                    36864  3 ath9k_common,ath9k,ath9k_hw
mac80211              917504  1 ath9k
cfg80211              897024  4 ath9k_common,ath9k,ath,mac80211

So it seems that Debian kernel has detected and loaded the proper module

Linux Access point software installation
ifconfig | grep w
wlan0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500

wlan0 interface appeared ! 

Warning

  • When freshly configured, wlan0 is disabled. You can enable it:
Linux Access point software installation
ip link set wlan0 up
  • However, note that in SOHO #002, all is set during startup script (/rtr/hwdet-all.sh) via systemd
  • Last observation is that in SOHO #002 we also disabled Linux systemd networking
    • systemd use to name linux interface based on their pcie id: wlp9s0
    • after disabling it, WIFI interface name appears to become: wlan0

Now that the hardware and corresponding linux driver is loaded we can proceed to Linux access point software installation

Linux Access point software installation
apt-get update
apt-get install hostapd

hostapd configuration is Debian is in /etc/hostapd/hostapd.conf. But remember this is no more under systemd startup control as we disabled entirely systemd networking.

hostapd.conf
cat /etc/hostapd/hostapd.conf
#change wlan0 to your wireless device
interface=wlan0
# "g" simply means 2.4GHz band
hw_mode=g
# the channel to use
channel=acs_survey
# limit the frequencies used to those allowed in the country
ieee80211d=1
# the country code
country_code=FR
# 802.11n support
ieee80211n=1
# QoS support, also required for full speed on 802.11n/ac/ax
wmm_enabled=1
driver=nl80211
ssid=YOUR_HOME_WIFI_SSID
macaddr_acl=0
auth_algs=1
ignore_broadcast_ssid=0
wpa=3
wpa_passphrase=y0urverys1cr1tpassw0rd
wpa_key_mgmt=WPA-PSK FT-PSK WPA-PSK-SHA256 SAE FT-SAE
wpa_pairwise=CCMP
rsn_pairwise=CCMP

You can check that hostapd is correctly configured by launching it manually

hostapd manual launch
hostapd /etc/hostapd/hostapd.conf 
...

From that point from your mobile phone or laptop:

  • you should be able to see YOUR_HOME_WIFI_SSID appearing in the list of available wireless network
  • you should be able to connect to YOUR_HOME_WIFI_SSID
  • But your mobile device might say: "Internet might not be available or Cannot retrieve IP from YOUR_HOME_WIFI_SSID" 
  • hostapd id triggered when SOHO router is booting up (described in rtr-hw.txt)  
  • Remember wlan0 is visible from the Linux kernel and managed by hostapd
  • we created a veth pair (veth2a@linux side and veth2b@sdn998 SOHO router side)
  • we bind wlan0 to veth2a (Cf. rtr-hw.txt → proc wlan /rtr/pcap2pcap.bin wlan0 veth2a)
  • and then veth2a is bound to sdn998  which is interface veth2b controlled by DPDK (Cf. SOHO #002 and SOHO #003)

So at that point all traffic coming from wlan0 will also make its way to sdn998

Now we need to make wireless traffic visible through VRF inet as we need to enable IPv4 (or IPv6) connectivity for mobile device.

So this is done in 2 steps.

Step - 1 - Create a bridge for wireless

Wireless bridge
bridge 1
 mac-learn
 mac-move
 exit

Note

Creating a bridge will also create interface bvi.

Step - 2 - Add veth2b to the wireless bridge

sdn998 addition into bridge 1
interface sdn998
 description SOHO@WLAN[veth2b-veth2a]
 mtu 1500
 bridge-group 1
 no shutdown
 no log-link-change
 exit

Warning

Make sure that bridge 1, sdn998, hairpin11 and hairpin12 are in declared into p4lang server

Linux Access point software installation
conf t
server p4lang p4 
 export-bridge 1
 export-port sdn998 7 1 0 0 0
 export-port hairpin11 11 0 0 0 0
 export-port hairpin12 12 0 0 0 0

The trick is to use hairpin interfaces. For experienced Junos user, this corresponds to Junos logical tunnel lt interface 

Step - 1 - Create a hairpin pair in order to redirect wireless traffic into VRF inet

Wireless bridge
conf t
hairpin 1

Note

Creating a hairpin 1 will also create interface hairpin11 and hairpin12.

Step - 2 - Add hairpin11 to the wireless bridge

hairpin11 addition into bridge 1
interface hairpin11
 no description
 bridge-group 1
 no shutdown
 no log-link-change
 exit

Step - 3 - Add hairpin12 into VRF inet

hairpin11 addition into bridge 1
interface hairpin12
 description SOHO@hairpin11
 mtu 1500
 vrf forwarding inet
 ipv4 address 192.168.129.1 255.255.255.0
 no shutdown
 no log-link-change
 exit

Warning

At that point all traffic from/to wireless will transit via hairpin12 and reach VRF inet

Now, DHCP request coming from mobile client can reach SOHO router VRF inet via hairpin12.

Step - 1 - Create DHCP server for Wireless client in VRF inet

DHCP configuration for Wireless client
server dhcp4 dh4-wlan
 pool 192.168.129.2 192.168.129.254
 gateway 192.168.129.1
 netmask 255.255.255.0
 dns-server 192.168.254.1
 domain-name local
 interface hairpin12
 vrf inet
 exit        

Note

Creating hairpin 1 will also create interface hairpin11 and hairpin12.

Verification

Connect to WIFI via laptop wifi

DHCP debug command
debug server dhcp?                                                     
  dhcp4      - ipv4 dynamic host config protocol
  dhcp6      - ipv6 dynamic host config protocol

mjolnir#debug server dhcp4 ?                                                   
  <cr>
mjolnir#debug server dhcp4                                                     
mjolnir#terminal monitor    
DHCP debug ouput when connecting laptop @ sdn6
mjolnir#debug server dhcp4                                                     
mjolnir#term mon                                                               
mjolnir#info ip.ipCor6.parseIPheader:ipCor6.java:95 got bad version from ::    
debug serv.servDhcp4worker.doer:servDhcp4.java:679 rx op=req sec=1 cia=0.0.0.0 yia=0.0.0.0 sia=0.0.0.0 gia=0.0.0.0 cha=88e9.fe76.7f9b srv= fil= op=discover dhcpsrv=null hstnm=MBP-de-Frederic dom=null lease=7776000 renew=0 mask=null gw=null dns1=null dns2=null req=null
debug serv.servDhcp4.sendPack:servDhcp4.java:482 tx 192.168.129.83 op=rep sec=1 cia=0.0.0.0 yia=192.168.129.83 sia=192.168.129.1 gia=0.0.0.0 cha=88e9.fe76.7f9b srv= fil= op=offer dhcpsrv=192.168.129.1 hstnm=null dom=lan lease=43200 renew=21600 mask=255.255.255.0 gw=192.168.129.1 dns1=192.168.254.1 dns2=null req=null
debug serv.servDhcp4worker.doer:servDhcp4.java:686 tx op=rep sec=1 cia=0.0.0.0 yia=192.168.129.83 sia=192.168.129.1 gia=0.0.0.0 cha=88e9.fe76.7f9b srv= fil= op=offer dhcpsrv=192.168.129.1 hstnm=null dom=lan lease=43200 renew=21600 mask=255.255.255.0 gw=192.168.129.1 dns1=192.168.254.1 dns2=null req=null
debug serv.servDhcp4worker.doer:servDhcp4.java:679 rx op=req sec=2 cia=0.0.0.0 yia=0.0.0.0 sia=0.0.0.0 gia=0.0.0.0 cha=88e9.fe76.7f9b srv= fil= op=request dhcpsrv=192.168.129.1 hstnm=MBP-de-Frederic dom=null lease=0 renew=0 mask=null gw=null dns1=null dns2=null req=192.168.129.83
debug serv.servDhcp4.sendPack:servDhcp4.java:482 tx 192.168.129.83 op=rep sec=2 cia=0.0.0.0 yia=192.168.129.83 sia=192.168.129.1 gia=0.0.0.0 cha=88e9.fe76.7f9b srv= fil= op=ack dhcpsrv=192.168.129.1 hstnm=null dom=lan lease=43200 renew=21600 mask=255.255.255.0 gw=192.168.129.1 dns1=192.168.254.1 dns2=null req=null
debug serv.servDhcp4worker.doer:servDhcp4.java:686 tx op=rep sec=2 cia=0.0.0.0 yia=192.168.129.83 sia=192.168.129.1 gia=0.0.0.0 cha=88e9.fe76.7f9b srv= fil= op=ack dhcpsrv=192.168.129.1 hstnm=null dom=lan lease=43200 renew=21600 mask=255.255.255.0 gw=192.168.129.1 dns1=192.168.254.1 dns2=null req=null
...
mjolnir#   

So based on the debug output:

  • DHCP allocated 192.168.129.83
  • Primary DNS (dns1) is 192.168.254.1
  • Network has /24 CIDR


ARP entries @ hairpin12
mjolnir#sh ipv4 arp hairpin12                                                  
mac             address          time      static
...
88e9.fe76.7f9b  192.168.129.83   00:00:13  false
...
mjolnir#   
DHCP debug command
╭─[11/3/20|3:17:21]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  ifconfig en0    
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=400<CHANNEL_IO>
        ether 88:e9:fe:76:7f:9b 
        inet6 fe80::1cf1:eacf:9ba9:43c3%en0 prefixlen 64 secured scopeid 0x5 
        inet 192.168.129.83 netmask 0xffffff00 broadcast 192.168.129.255
        inet6 2a01:e0a:159:2857:86:8cf9:a786:8f18 prefixlen 64 autoconf secured 
        inet6 2a01:e0a:159:2857:904b:4faa:5684:b7a0 prefixlen 64 autoconf temporary 
        nd6 options=201<PERFORMNUD,DAD>
        media: autoselect
        status: active
╭─[11/3/20|3:24:08]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  nslookup 212.27.48.10
Server:         fd00:2200::1
Address:        fd00:2200::1#53

Non-authoritative answer:
10.48.27.212.in-addr.arpa       name = www.free.fr.

Authoritative answers can be found from:

Pay attention to the server that answered ! At the present moment it should not be fd00:2200::1. It is simply because I had a setup with IPv6 fully enabled.

DNS resolution from CLI (DNS query originated by router)
╭─[11/3/20|3:14:17]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  ping 212.27.48.10
PING www.free.fr (212.27.48.10): 56 data bytes
64 bytes from 212.27.48.10: icmp_seq=0 ttl=57 time=6.528 ms
64 bytes from 212.27.48.10: icmp_seq=1 ttl=57 time=4.666 ms
64 bytes from 212.27.48.10: icmp_seq=2 ttl=57 time=4.330 ms
64 bytes from 212.27.48.10: icmp_seq=3 ttl=57 time=4.446 ms
Check that NAT is occuring for 192.168.129.83
show ipv4 nat inet translations | i 192.168.129.83                     
1      192.168.129.83 -211222528  212.27.48.10 -211222528  192.168.0.90 -211222528  212.27.48.10 -211222528    00:00:17  00:00:17  00:05:00  1        84
1      212.27.48.10 -211222528    192.168.0.90 -211222528  212.27.48.10 -211222528  192.168.129.83 -211222528  00:00:17  00:00:17  00:05:00  1        84
1      192.168.129.83 -211222527  212.27.48.10 -211222527  192.168.0.90 -211222527  212.27.48.10 -211222527    00:00:16  00:00:16  00:05:00  1        84
1      212.27.48.10 -211222527    192.168.0.90 -211222527  212.27.48.10 -211222527  192.168.129.83 -211222527  00:00:16  00:00:16  00:05:00  1        84
1      192.168.129.83 -211222526  212.27.48.10 -211222526  192.168.0.90 -211222526  212.27.48.10 -211222526    00:00:15  00:00:15  00:05:00  1        84
1      212.27.48.10 -211222526    192.168.0.90 -211222526  212.27.48.10 -211222526  192.168.129.83 -211222526  00:00:15  00:00:15  00:05:00  1        84
DNS resolution from CLI (DNS query originated by router)
╭─[11/2/20|4:36:18]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  ping www.free.fr -c 5 
PING www.free.fr (212.27.48.10): 56 data bytes
64 bytes from 212.27.48.10: icmp_seq=0 ttl=57 time=3.903 ms
64 bytes from 212.27.48.10: icmp_seq=1 ttl=57 time=5.883 ms
64 bytes from 212.27.48.10: icmp_seq=2 ttl=57 time=3.658 ms
64 bytes from 212.27.48.10: icmp_seq=3 ttl=57 time=4.872 ms
64 bytes from 212.27.48.10: icmp_seq=4 ttl=57 time=4.488 ms

--- www.free.fr ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.658/4.561/5.883/0.787 ms
Check that NAT is occuring for 192.168.136.126 and www.free.fr IPv4 address
show ipv4 nat inet translations | i 192.168.129.83                     
1      192.168.129.83 -211222528  212.27.48.10 -211222528  192.168.0.90 -211222528  212.27.48.10 -211222528    00:00:17  00:00:17  00:05:00  1        84
1      212.27.48.10 -211222528    192.168.0.90 -211222528  212.27.48.10 -211222528  192.168.129.83 -211222528  00:00:17  00:00:17  00:05:00  1        84
1      192.168.129.83 -211222527  212.27.48.10 -211222527  192.168.0.90 -211222527  212.27.48.10 -211222527    00:00:16  00:00:16  00:05:00  1        84
1      212.27.48.10 -211222527    192.168.0.90 -211222527  212.27.48.10 -211222527  192.168.129.83 -211222527  00:00:16  00:00:16  00:05:00  1        84
1      192.168.129.83 -211222526  212.27.48.10 -211222526  192.168.0.90 -211222526  212.27.48.10 -211222526    00:00:15  00:00:15  00:05:00  1        84
1      212.27.48.10 -211222526    192.168.0.90 -211222526  212.27.48.10 -211222526  192.168.129.83 -211222526  00:00:15  00:00:15  00:05:00  1        84

Conclusion

In this article we enabled the appliance WIFI integrated hardware:

  • We ensured that wifi hardware was detected by Linux kernel
  • We also ensured that Linux loaded the right wifi driver kernel module
  • hostapd would control wireless interface
  • we create a veth pair  (veth2a Linux side - veth2b / DPDK side)
  • wlan0 traffic is bound to veth2a using pcap2pcap utility (in freeRouter binaries bundle)
  • veth2b is bound to sdn998 (and declalred in p4lang server)
  • we create a hairpin 1 (hairpin11 , hairpin12) interface
  • bridge 1 has also been created
  • sdn998 and hairpin11 have been added to the bridge
  • hairpin12 is a routed interface declared in VRF inet and has an IP 192.168.129.1 inside wireless subnet 192.168.129.0/24


RARE validated design: [ SOHO #006 ] - key take-away

In this example the key take-away are:

  • The above action lead to bridge wireless traffic and pour it into VRF inet
  • From that point all defined previously will apply (NAT, DHCP) but now with wireless subnetwork

This example cover the case of a basic bridge in which we included a basic interface sdn998 and a hairpin interfaces pouring traffic from/to VRF inet. freeRouter is acting as a wireless controller local to the integrated WIFI. We will see in further article another typical WIFI implementation more flexible. Pleas note that we could have directly applied routing at sdn998 (so without the use of the bridge and hairpin). The usage of bridge and hairpin setup will be described in a next article describing alternate wifi implementation. In short we will add an OpenWRT Access point behind sdn6 and SOHO router will act as WIFI controller for both the integrated WIFI hotspot and the newly added OpenWRT.

So everyone inside home network can reach the Internet thanks to NAT translation described in SOHO #004 and reach now their favorite Internet service by name thanks for SOHO #005.

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

In the previous article during the verification we assume to have one host connected to a subnetwork cascaded behind SOHO router@sdn6. However, we did not described how the host could get connectivity.

Article objective

In this article we will pursue the SOHO network appliance installation and enable DHCP for IPv4 server. SOHO router can then answer to DHCPv4 request ingressing sdn6 interface.

  • Let's assume we have a switch connected to SOHO router@sdn6 for the future
  • sdn6 has ipv4 interface 192.168.136.1 manually configured
  • switch has IPv4 192.168.136.2
  • subnetwork behind sdn6 is 192.168.136.0/24
  • DHCPv4 will serve 192.168.136.3 → 192.168.136.254

Diagrams

[ SOHO #006 ] - "Do you need an IP ?"

First step, configure sdn6 and bind it to VRF inet. few considerations:

  • sdn6 has ipv4 192.168.136.1 manually configured
  • sdn6 will act as gateway for all connected host in subnetwork 192.168.136.0/24
  • In SOHO #004 ,  192.168.136.0/24 NAT is already taken into account


SOHO router DHCP server in VRF inet
conf t
interface sdn6
 mtu 1500
 vrf forwarding inet
 ipv4 address 192.168.136.1 255.255.255.0
 no shutdown
 no log-link-change
 exit
!

Second step, configure a DHCPv4 server@ SOHO router

SOHO router DHCP server in VRF inet
conf t
server dhcp4 dh4-16
 pool 192.168.136.3 192.168.136.254
 gateway 192.168.136.1
 netmask 255.255.255.0
 dns-server 192.168.254.1
 domain-name local
 interface sdn6
 vrf inet
 exit
!

In this case DHCPv4 will allocate IPv4 address:

  • from a pool going from 192.168.136.3 → 192.168.136.254
  • All host will set their gateway to 192.168.136.1 (which is SOHO@sdn6)
  • All connected host to sdn6 subnet will get an IP from pool within 192.168.136.0/24 subnet
  • All hosts will also consider SOHO@192.168.254.1 as DNS server as we implemented in SOHO #005
  • And we bind this DHCPv4 to interface sdn6 (without binding DHCP request coming from all interface belonging to VRF inet will be processed)
  • lastly DHCPv4 server will act into VRF inet only 

Verification

Connect a host to the switch connected to sdn6 then issue DHCP debug command

DHCP debug command
debug server dhcp?                                                     
  dhcp4      - ipv4 dynamic host config protocol
  dhcp6      - ipv6 dynamic host config protocol

mjolnir#debug server dhcp4 ?                                                   
  <cr>
mjolnir#debug server dhcp4                                                     
mjolnir#terminal monitor    
DHCP debug ouput when connecting laptop @ sdn6
mjolnir#..                                                          debug serv.servDhcp4worker.doer:servDhcp4.java:679 rx op=req sec=1 cia=0.0.0.0 yia=0.0.0.0 sia=0.0.0.0 gia=0.0.0.0 cha=9ceb.e8d5.2c51 srv= fil= op=discover dhcpsrv=null hstnm=MBP-de-Frederic dom=null lease=7776000 renew=0 mask=null gw=null dns1=null dns2=null req=null
debug serv.servDhcp4.sendPack:servDhcp4.java:482 tx 192.168.136.123 op=rep sec=1 cia=0.0.0.0 yia=192.168.136.123 sia=192.168.136.1 gia=0.0.0.0 cha=9ceb.e8d5.2c51 srv= fil= op=offer dhcpsrv=192.168.136.1 hstnm=null dom=lan lease=43200 renew=21600 mask=255.255.255.0 gw=192.168.136.1 dns1=192.168.254.1 dns2=null req=null
debug serv.servDhcp4worker.doer:servDhcp4.java:686 tx op=rep sec=1 cia=0.0.0.0 yia=192.168.136.123 sia=192.168.136.1 gia=0.0.0.0 cha=9ceb.e8d5.2c51 srv= fil= op=offer dhcpsrv=192.168.136.1 hstnm=null dom=lan lease=43200 renew=21600 mask=255.255.255.0 gw=192.168.136.1 dns1=192.168.254.1 dns2=null req=null
debug serv.servDhcp4worker.doer:servDhcp4.java:679 rx op=req sec=2 cia=0.0.0.0 yia=0.0.0.0 sia=0.0.0.0 gia=0.0.0.0 cha=9ceb.e8d5.2c51 srv= fil= op=request dhcpsrv=192.168.136.1 hstnm=MBP-de-Frederic dom=null lease=0 renew=0 mask=null gw=null dns1=null dns2=null req=192.168.136.123
debug serv.servDhcp4.sendPack:servDhcp4.java:482 tx 192.168.136.123 op=rep sec=2 cia=0.0.0.0 yia=192.168.136.123 sia=192.168.136.1 gia=0.0.0.0 cha=9ceb.e8d5.2c51 srv= fil= op=ack dhcpsrv=192.168.136.1 hstnm=null dom=lan lease=43200 renew=21600 mask=255.255.255.0 gw=192.168.136.1 dns1=192.168.254.1 dns2=null req=null
debug serv.servDhcp4worker.doer:servDhcp4.java:686 tx op=rep sec=2 cia=0.0.0.0 yia=192.168.136.123 sia=192.168.136.1 gia=0.0.0.0 cha=9ceb.e8d5.2c51 srv= fil= op=ack dhcpsrv=192.168.136.1 hstnm=null dom=lan lease=43200 renew=21600 mask=255.255.255.0 gw=192.168.136.1 dns1=192.168.254.1 dns2=null req=null
info ip.ipCor6.parseIPheader:ipCor6.java:95 got bad version from ::
...
mjolnir#   

So based on the debug output:

  • DHCP allocated 192.168.136.123
  • Primary DNS (dns1) is 192.168.136.1
  • Network has /24 CIDR
DHCP debug command
╭─[11/2/20|4:30:09]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  ifconfig en8
en8: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=6407<RXCSUM,TXCSUM,VLAN_MTU,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
        ether 9c:eb:e8:d5:2c:51 
        inet6 fe80::c93:c3b1:dfb3:77c0%en8 prefixlen 64 secured scopeid 0x13 
        inet 192.168.136.123 netmask 0xffffff00 broadcast 192.168.136.255
        inet6 2a01:e0a:159:2856:832:82f5:8519:70 prefixlen 64 autoconf secured 
        inet6 2a01:e0a:159:2856:653c:d1b2:dca9:c9da prefixlen 64 autoconf temporary 
        nd6 options=201<PERFORMNUD,DAD>
        media: autoselect (1000baseT <full-duplex>)
        status: active

╭─[11/2/20|4:37:05]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  nslookup www.free.fr      
Server:         192.168.254.1
Address:        192.168.254.1#53

Non-authoritative answer:
Name:   www.free.fr
Address: 212.27.48.10

Pay attention to the DNS server that answered ! Sweet !

DNS resolution from CLI (DNS query originated by router)
╭─[11/2/20|4:30:11]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  ping 8.8.8.8 -c 5
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=119 time=4.171 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=4.334 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=119 time=4.208 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=119 time=3.856 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=119 time=4.090 ms
Check that NAT is occuring for 192.168.136.126
mjolnir#show ipv4 nat inet translations | i 192.168.136.123                    
1      8.8.8.8 710148096           192.168.0.90 710148096   8.8.8.8 710148096        192.168.136.123 710148096   00:02:46  00:02:46  00:05:00  1       84
1      192.168.136.123 710148096   8.8.8.8 710148096        192.168.0.90 710148096   8.8.8.8 710148096           00:02:46  00:02:46  00:05:00  1       84
1      8.8.8.8 710148097           192.168.0.90 710148097   8.8.8.8 710148097        192.168.136.123 710148097   00:02:45  00:02:45  00:05:00  1       84
1      192.168.136.123 710148097   8.8.8.8 710148097        192.168.0.90 710148097   8.8.8.8 710148097           00:02:45  00:02:45  00:05:00  1       84
1      8.8.8.8 710148098           192.168.0.90 710148098   8.8.8.8 710148098        192.168.136.123 710148098   00:02:44  00:02:44  00:05:00  1       84
DNS resolution from CLI (DNS query originated by router)
╭─[11/2/20|4:36:18]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  ping www.free.fr -c 5 
PING www.free.fr (212.27.48.10): 56 data bytes
64 bytes from 212.27.48.10: icmp_seq=0 ttl=57 time=3.903 ms
64 bytes from 212.27.48.10: icmp_seq=1 ttl=57 time=5.883 ms
64 bytes from 212.27.48.10: icmp_seq=2 ttl=57 time=3.658 ms
64 bytes from 212.27.48.10: icmp_seq=3 ttl=57 time=4.872 ms
64 bytes from 212.27.48.10: icmp_seq=4 ttl=57 time=4.488 ms

--- www.free.fr ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.658/4.561/5.883/0.787 ms
Check that NAT is occuring for 192.168.136.126 and www.free.fr IPv4 address
mjolnir#show ipv4 nat inet translations | i 212.27.48.10                       
1      192.168.136.123 1263796224  212.27.48.10 1263796224  192.168.0.90 1263796224  212.27.48.10 1263796224     00:03:47  00:03:47  00:05:00  1       84
1      212.27.48.10 1263796224     192.168.0.90 1263796224  212.27.48.10 1263796224  192.168.136.123 1263796224  00:03:47  00:03:47  00:05:00  1       84
1      192.168.136.123 1263796225  212.27.48.10 1263796225  192.168.0.90 1263796225  212.27.48.10 1263796225     00:03:46  00:03:46  00:05:00  1       84
1      212.27.48.10 1263796225     192.168.0.90 1263796225  212.27.48.10 1263796225  192.168.136.123 1263796225  00:03:46  00:03:46  00:05:00  1       84
1      192.168.136.123 1263796226  212.27.48.10 1263796226  192.168.0.90 1263796226  212.27.48.10 1263796226     00:03:45  00:03:45  00:05:00  1       84

Conclusion

In this article DHCP service has been enabled at:

  • SOHO router level 
  • All host getting an IPv4 via DHCP will get a DNS server set to SOHO@loopback0 (192.168.254.1)

RARE validated design: [ SOHO #005 ] - key take-away

In this example the key take-away are:

  • DHCP is available for IPv4 and IPv6
  • DHCP here is bound to an interface so rogue DHCP request coming from any router interface other than the speficied interface won't be honoured
  • Feel free to explore all DHCP options propose by RARE/freeRouter

DHCP for IPv6 is also available if you want to strictly reproduce IPv4 allocation scheme. In our case we will use IPv6 SLAC.

So everyone inside home network can reach the Internet thanks to NAT translation described in SOHO #004

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

In the previous article we enabled and checked IPv4 connectivity between all potential host within 192.168.128.0/17 and the outside Networks beyond ISP box. But, this is pretty useless as I can't imagine my kids typing IPv6 address (2001:8b0:0:30::666:102) in the browser in order to play a FUN puzzle. (Though for now we are suppose to have only IPv4 (smile)) So we definitely need to provide name service resolution at the SOHO router itself.

Article objective

In this article we will pursue the SOHO network appliance installation and enable name service to all host @ home.

Diagrams

[ SOHO #005 ] - "Got your Id number, but ... What's your name ?"

First step, it is need to configure the router as a client name for an existing DNS server.

SOHO router in VRF inet
!
conf t
client name-server 8.8.8.8 1.1.1.1 
!

So this declare our SOHO router as DNS client for 8.8.8.8 as primary DNS server and 1.1.1.1 as backup DNS server.

This step is mandatory as it will bind traffic originated from SOHO router to a specific VRF (here: inet). So this can be also qualified as "VRF proxy-awareness". In this way all DNS traffic originated from the router will be bound to VRF inet. This is done in 2 steps. The first step is to create the proxy-profile and bind it to the main VRF inet. The second step is to declare the SOHO router as client of this proxy-profile service.

SOHO router in VRF inet
!
! step 1
!
proxy-profile pp-inet
 vrf inet
 exit
!
! step 2
!
client proxy pp-inet
!

Step -3-, configure DNS cache / server 

  • enable recursion (recursive query toward other DNS defined 8.8.8.8, 1.1.1.1)
  • bind it to a specific interface (so SOHO router will answer only DNS from this interface)
  • bind it to VRF inet
SOHO router DNS cache
!
server dns ns-inet
 recursion enable
 interface loopback0
 vrf inet
 exit
!

So this declare our SOHO router as DNS client for 8.8.8.8 as primary DNS server and 1.1.1.1 as backup DNS server

Step -4-, configure DNS and DHCP to propagate default dummy zone local

  • Use local if you don't plan to propagate a domain name
  • create local as dummy zone


Propagate "local" zone
!
server dhcp4 dh4-inet
 dns-server 192.168.254.1
 domain-name local
 exit
!
server dns ns-inet
 zone local
 exit
!

Verification

When -1- and -2- are realised the router can resolve name

DNS resolution from CLI (DNS query originated by router)
ping www.free.fr /vrf inet                                             
pinging 212.27.48.10, src=null, vrf=inet, cnt=5, len=64, tim=1000, ttl=255, tos=0, sweep=false
!!!!!
result=100%, recv/sent/lost=5/5/0, rtt min/avg/max/total=3/3/4/16

This can be verified only using a host connected to SOHO router. Let's assume a laptop connected behind sdn6.

ping & ping6 hostname
...
╭─[10/31/20|3:01:19]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  ping www.free.fr
PING www.free.fr (212.27.48.10): 56 data bytes
64 bytes from 212.27.48.10: icmp_seq=0 ttl=57 time=3.670 ms
64 bytes from 212.27.48.10: icmp_seq=1 ttl=57 time=6.666 ms
64 bytes from 212.27.48.10: icmp_seq=2 ttl=57 time=6.163 ms
64 bytes from 212.27.48.10: icmp_seq=3 ttl=57 time=6.118 ms
^C
--- www.free.fr ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.670/5.654/6.666/1.166 ms
╭─[10/31/20|3:40:12]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  ping6 www.free.fr
PING6(56=40+8+8 bytes) 2a01:e0a:159:2857:b9d9:e9e0:ae30:88e5 --> 2a01:e0c:1::1
16 bytes from 2a01:e0c:1::1, icmp_seq=0 hlim=56 time=3.805 ms
16 bytes from 2a01:e0c:1::1, icmp_seq=1 hlim=56 time=6.898 ms
16 bytes from 2a01:e0c:1::1, icmp_seq=2 hlim=56 time=5.868 ms
16 bytes from 2a01:e0c:1::1, icmp_seq=3 hlim=56 time=5.729 ms
^C
--- www.free.fr ping6 statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 3.805/5.575/6.898/1.117 ms

...

IPv4 / IPv6 name resolution
...
╭─[10/31/20|3:42:11]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  dig www.free.fr                                                                                                                                                127 ↵

; <<>> DiG 9.10.6 <<>> www.free.fr
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25030
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.free.fr.                   IN      A

;; ANSWER SECTION:
www.free.fr.            20961   IN      A       212.27.48.10

;; Query time: 21 msec
;; SERVER: 192.168.254.1#53(192.168.254.1)
;; WHEN: Sat Oct 31 15:42:18 CET 2020
;; MSG SIZE  rcvd: 56

╭─[10/31/20|3:42:18]loui@MacBook-Pro-de-Frederic.local ~  
╰─➤  dig AAAA www.free.fr  

; <<>> DiG 9.10.6 <<>> AAAA www.free.fr
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21770
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.free.fr.                   IN      AAAA

;; ANSWER SECTION:
www.free.fr.            21075   IN      AAAA    2a01:e0c:1::1

;; Query time: 5 msec
;; SERVER: 192.168.254.1#53(192.168.254.1)
;; WHEN: Sat Oct 31 15:42:29 CET 2020
;; MSG SIZE  rcvd: 68
...

As said IPv6 verification are just FYI, as we are supposed to have deployed only IPv4 till now. The point to show off IPv6 verification is to verify DNS AAAA request are working properly.

Conclusion

In this article DNS service has been enabled at:

  • SOHO router level 
  • All host getting an IPv4 via DHCP will get a DNS server set to SOHO@loopback0 (192.168.254.1)

RARE validated design: [ SOHO #005 ] - key take-away

In this example the key take-away are:

  • proxy-profile usage in order to proxy DNS query into VRF inet
  • proxy-profile can be used to proxy other types of traffic
  • data/routed traffic is not affected by proxy-profile



If you have followed sequentially previous SOHO articles, you should have now a strong vanilla base in order to develop you home office network or you home network. 

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

In the previous article we enabled and checked IPv4 connectivity between RARE/freeRouter and ISP box using sdn1 interface within 192.168.0.0/24 network. But as stated in the previous post, I'd like:

  • all people connected within 192.168.128.0/17
  • to access the external world

Article objective

In this article we will pursue the SOHO network appliance installation and enable IPv4 connectivity for all host connected within your internal network to the external world.

Diagrams

[ #004 ] - Do you need translation ? 

First step, let's create an interface Loopback0 within 192.168.128.0/17, let's say 192.168.254.1/32

SOHO router in VRF inet
sh run loopback0                                                        
interface loopback0
 no description
 vrf forwarding inet
 ipv4 address 192.168.254.1 255.255.255.255
 no shutdown
 no log-link-change
 exit
!

First step, let's try to ping 8.8.8.8

ping 8.8.8.8
ping 8.8.8.8 /vrf inet /interface lo0                                  
pinging 8.8.8.8, src=192.168.254.1, vrf=inet, cnt=5, len=64, tim=1000, ttl=255, tos=0, sweep=false
.....
result=0%, recv/sent/lost=0/5/5, rtt min/avg/max/total=10000/0/0/5003

Hey ! It seems that it's not working as expected ! At the present time, it seems that no one is able to reach outside world. Let's try to figure out why.

Note

  • It is a common best practice to dedicate a loopback to a router in a VRF
  • Loopback are mostly used to identify a service proposed by the router
  • Each services are bound to this Loopback in a specific VRF 
traceroute 8.8.8.8 using looback0 as source address
traceroute 8.8.8.8 /vrf inet /interface lo0                            
tracing 8.8.8.8, src=192.168.254.1, vrf=inet, prt=0/33440, tim=1000, tos=0, len=64
1 192.168.254.1 time=0
2 null time=1000
3 null time=1000
4 null time=1000
5 null time=1000
6 null time=1000
7 null time=1000
8 null time=1000
9 null time=1000
10 null time=1000

This confirms the ping failures we observed previously. The output above indicate the packet does not even egress our SOHO router.

What is the inet VRF says ?

routes inside VRF inet
show ipv4 route inet                                                   
typ  prefix            metric  iface      hop            time
C    192.168.0.0/24    0/0     sdn1       null           14:30:07
LOC  192.168.0.90/32   0/1     sdn1       null           14:30:07
C    192.168.128.0/24  0/0     sdn999     null           14:30:13
LOC  192.168.128.1/32  0/1     sdn999     null           14:30:13
C    192.168.254.1/32  0/0     loopback0  null           14:30:15

So we have no default routes . Let's configure one then pointing towards ISP BOX gateway:

Default route configuration
conf t
ipv4 route inet 0.0.0.0 0.0.0.0 192.168.0.254
routes inside VRF inet
show ipv4 route inet                                                   
typ  prefix            metric  iface      hop            time
S    0.0.0.0/0         1/0     sdn1       192.168.0.254  14:30:07
C    192.168.0.0/24    0/0     sdn1       null           14:30:07
LOC  192.168.0.90/32   0/1     sdn1       null           14:30:07
C    192.168.128.0/24  0/0     sdn999     null           14:30:13
LOC  192.168.128.1/32  0/1     sdn999     null           14:30:13
C    192.168.254.1/32  0/0     loopback0  null           14:30:15

So at that point, packet send to 8.8.8.8 are sent to nexthop 192.168.0.254 via sdn1.

ping 8.8.8.8
ping 8.8.8.8 /vrf inet /interface lo0                                  
pinging 8.8.8.8, src=192.168.254.1, vrf=inet, cnt=5, len=64, tim=1000, ttl=255, tos=0, sweep=false
.....
result=0%, recv/sent/lost=0/5/5, rtt min/avg/max/total=10000/0/0/5003

But ping is still not not working. Let's figure out what's going on here.

As depicted in previous article:

  • ISP box has a demarcation point set to 192.168.0.254
  • So ISP box at some point is configured to perform Network Address Translation from 192.168.0.0/24 → ISP public IPv4 interface
  • When ISP box receives a ICMP ping from 192.168.254.1 which does not match any ISP box NAT rules → Packet is discarded

Therefore in order to have a working seamless networking environment from the ISP box point of view, traffic coming from 192.168.128.0/17 might need to be NAT(ed) into 192.168.0.0/24 network. Let's see If our guess is right.

Let's configure IPv4 Network Address Translation specifically for Loopback0

ping 8.8.8.8
!
conf t
access-list ACL-NAT4   
 sequence 10 deny all 192.168.128.0 255.255.128.0 all 192.168.128.0 255.255.128.0 all
 sequence 20 deny all any all 224.0.0.0 240.0.0.0 all                                          
 permit all 192.168.254.1 255.255.255.255 all any all     
exit
!
ipv4 nat inet sequence 10 srclist ACL-NAT4 interface sdn1
!

 In the config stanza above we configure an access-list called ACL-NAT4 in VRF inet that would translate all incoming packet matching 192.168.254.1 going to anywhere to interface sdn1 IPv4. But we also don't NAT for inter-vlan traffic within 192.168.128.0/17 and also does not touch multicast traffic.

ping 8.8.8.8
ping 8.8.8.8 /vrf inet /interface sdn1                                 
pinging 8.8.8.8, src=192.168.0.90, vrf=inet, cnt=5, len=64, tim=1000, ttl=255, tos=0, sweep=false
!!!!!
result=100%, recv/sent/lost=5/5/0, rtt min/avg/max/total=3/3/4/16

Yahhoooo ! It works ... Our guess was good after all... (smile)

For the future we assume that the home network is subnetted as follow and are all inside VRF inet:

  • appliance port#1 @ sdn1: 192.168.0.0/24 (ISP subnet)
  • appliance port#2 @ sdn2: 192.168.130.0/24
  • appliance port#3 @ sdn3: 192.168.133.0/24
  • appliance port#4 @ sdn4: 192.168.134.0/24 (unused)
  • appliance port#5 @ sdn5: 192.168.135.0/24
  • appliance port#6 @ sdn6: 192.168.136.0/24
  • appliance integrated wifi @ sdn998: 192.168.129.0/24
  • appliance loopback inet @ loopback0: 192.168.129.0/24


ping 8.8.8.8
!
conf t
access-list ACL-NAT4
 sequence 10 deny all 192.168.128.0 255.255.128.0 all 192.168.128.0 255.255.128.0 all
 sequence 20 deny all any all 224.0.0.0 240.0.0.0 all
 sequence 30 permit all 192.168.128.0 255.255.255.0 all any all
 sequence 40 permit all 192.168.129.0 255.255.255.0 all any all
 sequence 50 permit all 192.168.130.0 255.255.255.0 all any all
 sequence 60 permit all 192.168.133.0 255.255.255.0 all any all
 sequence 70 permit all 192.168.135.0 255.255.255.0 all any all
 sequence 80 permit all 192.168.136.0 255.255.255.0 all any all
 sequence 90 permit all 192.168.254.1 255.255.255.255 all any all
 exit
!
ipv4 nat inet sequence 10 srclist ACL-NAT4 interface sdn1
!

 In the config stanza above we configure an access-list called ACL-NAT4 in VRF inet that would translate all incoming packet matching 192.168.[128|129|130|133|135|136|254].0/24 going to anywhere.

Note

Each time you modify ACL-NAT4 you will need to re-apply the NAT calls:

ping 8.8.8.8
!
conf t
! modify access-list ACL-NAT4
!
no ipv4 nat inet sequence 10 srclist ACL-NAT4 interface sdn1
ipv4 nat inet sequence 10 srclist ACL-NAT4 interface sdn1
!

Also notice that we are NAT(ing), 192.168.128.0/24 which is the OOBM subnet of the Linux appliance itself. This is needed during Debian system update/upgrade operation and also needed during the auto-upgrade feature that I'll describe in a next article.

IPv6 does need NAT in my specific case as my ISP has allocated me public IPv6 prefixes. We will see IPv6 configuration in the next articles.

Verification

NAT show commands
!
show ipv4 nat ?                                                        
  <vrf> - name of routing table
!
show ipv4 nat inet ?                                                   
  statistics   - list of configuration entries
  translations - list of translation entries
!
ping 8.8.8.8 in order to trigger NAT translation
ping 8.8.8.8 /vrf inet /interface lo0                                 
pinging 8.8.8.8, src=192.168.254.1, vrf=inet, cnt=5, len=64, tim=1000, ttl=255, tos=0, sweep=false
!!!!!
result=100%, recv/sent/lost=5/5/0, rtt min/avg/max/total=2/3/4/16
NAT translation verification
mjolnir#show ipv4 nat inet translations | i 192.168.254.1                      
1      8.8.8.8 4392          192.168.0.90 4392      8.8.8.8 4392          192.168.254.1 4392     00:00:01  00:00:01  00:05:00  1        64
1      192.168.254.1 4392    8.8.8.8 4392           192.168.0.90 4392     8.8.8.8 4392           00:00:01  00:00:01  00:05:00  1        64
1      8.8.8.8 4393          192.168.0.90 4393      8.8.8.8 4393          192.168.254.1 4393     00:00:01  00:00:01  00:05:00  1        64
1      192.168.254.1 4393    8.8.8.8 4393           192.168.0.90 4393     8.8.8.8 4393           00:00:01  00:00:01  00:05:00  1        64
1      8.8.8.8 4394          192.168.0.90 4394      8.8.8.8 4394          192.168.254.1 4394     00:00:01  00:00:01  00:05:00  1        64
1      192.168.254.1 4394    8.8.8.8 4394           192.168.0.90 4394     8.8.8.8 4394           00:00:01  00:00:01  00:05:00  1        64
1      8.8.8.8 4395          192.168.0.90 4395      8.8.8.8 4395          192.168.254.1 4395     00:00:01  00:00:01  00:05:00  1        64
1      192.168.254.1 4395    8.8.8.8 4395           192.168.0.90 4395     8.8.8.8 4395           00:00:01  00:00:01  00:05:00  1        64
1      8.8.8.8 4396          192.168.0.90 4396      8.8.8.8 4396          192.168.254.1 4396     00:00:01  00:00:01  00:05:00  1        64
1      192.168.254.1 4396    8.8.8.8 4396           192.168.0.90 4396     8.8.8.8 4396           00:00:01  00:00:01  00:05:00  1        64

Conclusion

In this article

  • We finally have a router that enables connectivity for all hosts inside the home network to the outside world
  • due to the ISP specific setup, our router had to translate inner home IP subnets to subnet that can be in turn NAT'ed by the ISP box.
  • We have a consistent IPv4 addressing plan 
  • We now can add very exciting feature from now on. (In next articles !)


RARE validated design: [ SOHO #004 ] - key take-away

In this example we are proposing a basic connectivity scenario. However, keep in mind that depending on your location the configuration might be drastically different. But do not fear ! RARE/freeRouter has all the features need to enable connectivity !

  • NAT64 is available. So in case you want to run a pure IPv6 network, freeRouter can NAT64 traffic for you.
  • NAT46 is also available. In case you are desperate and don't want to implement a pure IPv6 home network and have an ISP running only IPv6, freeRouter can NAT46 your traffic for you !
  • In the example described, we are lucky to have IPv6 public global IPv6 address. We will see IPv6 configuration in subsequent articles.

Now that you have a hardware for your SOHO appliance, installed an Operating System and prepared  systemd init script in order to resume freeRouter operation in case of unexpected outage (power cut, reset button etc.), let's proceed to RARE/freeRouter installation itself.

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

When installing RARE/freeRouter on x86, you have 2 choices:

  • installation with a software dataplane
  • installation with a DPDK dataplane


In this precise case, we will consider a DPDK dataplane installation as our hardware is compliant to the requirement listed below.

DPDK requirements

  • CPU with SSE4 support
  • DPDK compatible NIC 

Note that freeRouter is available where JVM is available

  • x86
  • ARM

Article objective

In this article we will pursue the SOHO network appliance installation based on the diagram below, and freeRouter installation using  DPDK dataplane. In this situation, the appliance is behind ISP FTTH box demarcation point. As it is typical to French FTTH domestic deployment. 

Deployment consideration

In this case, RARE/freeRouter is connected to a ISP box demarcation point that deliver copper connectivity. Nothing prevents you, following your context, to deploy a similar equipment with with SFP uplinks directly connected to your Provider Edge backbone routers if you own also the dark fiber paths local to the MAN. 

Diagrams

[ #003 ] - RARE/freeRouter DPDK SOHO installation 

  • Own a similar hardware described in SOHO #001
  • Having installed an Operating System with Java Runtime Environment
  • Configured systemd so that RARE/freeRouter can take over networking at each reboot as described in SOHO #002.

Let's consider the following assumptions:

  • ISP box comes with 192.168.0.0/24 subnet configured at RJ45 demarcation point
  • Home networkS will be within 192.168.128.0/17
  • 192.168.128.0/17 will be subnetted further into multiple /24 in order to accomodate home network requirement
  • RARE/freeRouter is connected to the FTTP ISP box via appliance DPDK port #0 (interface sdn1)
  • Home traffic going to outside world will be subject to port address translation (NAT/PAT) using an IPv4 within ISP subnet range
  • appliance port #1 will be connected to FTTH ISP box and will have an IP within 192.168.0.0/24

IPv6 addressing plan has not been forgotten. It is not mentioned here on purpose in order to not complicate explanations. IPv6 we be the object of further articles. It is not that IPv6 is a complex topic. It just that it deserves special attention. You might not realised it, but IPv6 is everywhere and is used by default between peers as soon as IPv6 is enable. So IMHO we need to get used to it as soon as possible especially if you are a network administrator.

FreeRouter uses 2 configuration files in order to run, let's write these configuration files in /rtr

freeRouter hardware configuration file: rtr-hw.txt
hwid j1900-i211
! cpu_port
int eth0 eth - 127.0.0.1 20001 127.0.0.1 20002
! freerouter control port for message packet-in/out in P4 VRF _ONLY_
tcp2vrf 9080 p4 9080
! freeroouter local access in p4 VRF _ONLY_
tcp2vrf 2323 p4 23
! launch a process called "veth0" that actually link to veth0b
! cmd for control plane/dataplane communication unified messaging: ip link add veth0a type veth peer name veth0b
! cmd for appliance Linux access: ip link add veth1a type veth peer name veth1b
! cmd for integrated wifi: ip link add veth2a type veth peer name veth2b
! external wifi AP
proc hostapd /usr/sbin/hostapd /etc/hostapd/hostapd.conf
! integrated wifi AP
proc wlan /rtr/pcap2pcap.bin wlan0 veth2a
! DP/CP communication process
proc veth0 /rtr/pcapInt.bin veth0a 20002 127.0.0.1 20001 127.0.0.1
! DP DPDK process
proc p4emu /rtr/p4dpdk.bin --vdev=net_af_packet0,iface=veth0b --vdev=net_af_packet1,iface=veth2b --vdev=net_af_packet2,iface=veth1b 127.0.0.1 9080 6

Note:

Let's spend some times on this hardware configuration file, as you might have notice there are additional interesting lines worth to mention:

  • Exclamation mark "!" are comments
  • hwid is a text field that would just designate the hardware on which freeRouter is running. (output of : show platform)
  • proc <process-name>

It is possible within freeRouter startup to launch processes. We use here this feature to start control plane / dataplane communication via veth pair: veth0a and veth0b and also P4Emu/dpdk, p4dpdk.bin packet processing backend.

  • proc p4emu /rtr/p4dpdk.bin --vdev=net_af_packet0,iface=veth0b --vdev=net_af_packet1,iface=veth2b --vdev=net_af_packet2,iface=veth1b 127.0.0.1 9080 6

In dpdk, by default dpdk interfaces have port_ids that are sequentially allocated and in the order of appearance in dpdk-devbind --status output usually sorted by pci_id. In the below output interface enp0s1 has port_id #0 and in dpdk it would be pci_id:00:01.0

enp0s1 would be: #0 with pci_id: 00:01.0

enp0s2 would be: #1 with pci_id: 00:02.0

enp0s5 would be: #2 with pci_id: 00:05.0

enp0s6 would be: #3 with pci_id: 00:06.0

enp0s7 would be: #4 with pci_id: 00:07.0

enp0s8 would be: #5 with pci_id: 00:08.0

DPDK diagnosis
dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:01:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb
0000:02:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb
0000:05:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb
0000:06:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb
0000:07:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb
0000:08:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb

Network devices using kernel driver
===================================
0000:09:00.0 'AR928X Wireless Network Adapter (PCI-Express) 002a' if=wlan0 drv=ath9k unused=uio_pci_generic 

No 'Baseband' devices detected
==============================

Other Crypto devices
====================
0000:00:1a.0 'Atom Processor Z36xxx/Z37xxx Series Trusted Execution Engine 0f18' unused=uio_pci_generic

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================
  • DPDK --vdev addition. In this precise case we instruct DPDK to take into account additional veth endpoint we created respectively for
    • Control plane / data plane communication
    • Linux out of band management access via SSH we installed previously during Debian package installation
    • integrated hardware WIFI access point
  • in DPDK vdev interface will have in order of apparition in the command line:
    • DP/CP communication: 6 ↔ veth0b
    • integrated WIFI: 7 ↔ veth2b
    • Linux out of band management access: 8 ↔ veth1b

external WIFI access point will be bound directly to an interface of the appliance via DPDK. This will be describe in future articles.

freeRouter software configuration file: rtr-sw.txt
hostname mjolnir
buggy
!
!
vrf definition inet
 exit
!
vrf definition p4
 exit
!
interface ethernet0
 description freerouter@P4_CPU_PORT[veth0a]
 no shutdown
 no log-link-change
 exit
!
interface sdn1
 description freerouter@DPDK[port-1]
 mtu 1500
 vrf forwarding inet
 ipv4 address 192.168.0.90 255.255.255.0
 no shutdown
 no log-link-change
 exit
!
interface sdn2
 description freerouter@DPDK[port-2]
 mtu 1500
 shutdown
 no log-link-change
 exit
!
interface sdn3
 description freerouter@DPDK[port-3]
 mtu 1500
 shutdown
 no log-link-change
 exit
!
interface sdn4
 description freerouter@DPDK[port-4]
 mtu 1500
 shutdown
 no log-link-change
 exit
!
interface sdn5
 description freerouter@DPDK[port-5]
 mtu 1500
 shutdown
 no log-link-change
 exit
!
interface sdn6
 description freerouter@DPDK[port-6]
 mtu 1500
 shutdown
 no log-link-change
 exit
!
interface sdn998
 description freerouter@DPDK[port-7 --> veth2a] integrated wifi
 mtu 1500
 shutdown
 no log-link-change
 exit
!
interface sdn999
 description freerouter@OOBM[port-8 --> veth1a] Linux management
 mtu 1500
 vrf forwarding inet
 ipv4 address 192.168.128.1 255.255.255.0
 no shutdown
 no log-link-change
 exit

server telnet telnet
 security protocol telnet
 no exec authorization
 no login authentication
 vrf p4
 exit
!
server p4lang p4
 export-vrf inet 1
 export-port sdn1 0 1 0 0 0
 export-port sdn2 1 1 0 0 0
 export-port sdn3 2 1 0 0 0
 export-port sdn4 3 1 0 0 0
 export-port sdn5 4 1 0 0 0
 export-port sdn6 5 1 0 0 0
 export-port sdn998 7 1 0 0 0
 export-port sdn999 8 1 0 0 0
 interconnect ethernet0
 vrf p4
 exit
!
!
end
  • For now integrated wifi is shut. We will see in later article how to activate it
  • At Linux level, if you noticed in the previous article
    • management IP subnet is 192.168.128.0/24. OOBM appliance IP is then 192.168.128.254
appliance management IP@Linux level (check previous article)
ip addr flush dev veth1a
ip addr add 192.168.128.254/24 dev veth1a
  • management IP seen from freeRouter@sdn999 with IP 192.168.128.1 within 192.168.128.0/24
  • with configured a Linux static routes
Add default route to OOBM sdn999@Linux level (check previous article)
route add default gw 192.168.128.1

Security note

  • If you pay attention p4lang server in p4 VRF
    • This VRF has no bound interface
    • Is isolated then from the other VRF
  • This will allow only local Linux host control plane and dataplane communication 

Verification

ping ISP demarcation point IP
ping 192.168.0.254 /vrf inet /interface sdn1                           
pinging 192.168.0.254, src=192.168.0.90, vrf=inet, cnt=5, len=64, tim=1000, ttl=255, tos=0, sweep=false
!!!!!
result=100%, recv/sent/lost=5/5/0, rtt min/avg/max/total=0/0/1/4
ARP discovery
mjolnir#sh ipv4 arp sdn1                                                       
mac             address        time      static
0024.d4a0.0cd3  192.168.0.254  00:00:20  false
Add default route to OOBM sdn999@Linux level (check previous article)
sh int sdn1                                                          
sdn1 is up (since 13:14:14, 2 changes)
 description: mjolnir@LAN1[01:00.0]
 type is sdn, hwaddr=003b.7671.764f, mtu=1500, bw=8000kbps, vrf=inet
 ip4 address=192.168.0.90/24, netmask=255.255.255.0, ifcid=10014
 received 64038 packets (17841459 bytes) dropped 4 packets (326 bytes)
 transmitted 250217 packets (38032822 bytes) promisc=false macsec=false
interface summary
show interfaces summary                                                
interface    state  tx        rx        drop
ethernet0    up     74690935  51798769  0
sdn1         up     37954707  17828649  326
sdn2         admin  0         0         0
sdn3         admin  0         0         0
sdn4         admin  0         0         0
sdn5         admin  0         0         0
sdn6         admin  0         0         0
sdn998       admin  0         0         0
sdn999       up     23646     17904     0
interface summary
interface   state  tx          rx          drop
sdn1        up     674397352   3883928390  948
sdn2        admin  0           0           0
sdn3        admin  0           0           0
sdn4        admin  0           0           0
sdn5        admin  0           0           0
sdn6        admin  0           0           0
sdn998      admin  0           0           0
sdn999      up     110520      85072       0

Check Linux appliance local routes

From linux terminal
root@mjolnir:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.128.1   0.0.0.0         UG    0      0        0 veth1a

Test local telnet access from linux/localhost

Conclusion

In this article

  • we finally launched RARE/freeRouter with DPDK dataplane
  • configure RARE/freeRouter with a vanilla config that takes into account all the appliance physical interfaces
  • added veth pair in the config in order to take into account:
    • Control plane / Data plane communication 
    • linux OOBM
    • integrated WIFI
  • Enabled and checked IPv4 connectivity between freeRouter@sdn1 and ISP demarcation point
  • Check telnet access to freeRouter from localhost only

RARE validated design: [ SOHO #003 ] - key take-away

From this point you have a complete freeRouter connected to ISP box via SDN1 as uplink in 192.168.0.0/24 subnet. We will extend further this base configuration step by step in order to enrich user experience !

  • Now you would want to enable IPv4/IPv6  connectivity to all potential hosts@home whether they are connected via RJ45 or via built-in WIFI.
  • you would also want to distribute IPv4, IPv6 to all the of hosts@home
  • IPv4/IPv6 connectivity is not enough, you would like to provide Domain Name Service to them
  • Domain Name Service is not enough if they can't reach outside world. As we are using RFC1918 addressing plan we should figure out a way to ensure NAT/PAT address translation in order to enable egress traffic toward the Internet
  • Your home might have several floors and only one WIFI access point is not enough ? Let's see how we can add additional WIFI AP in the network
  • Maybe you have an outsourced network management service ? Let's see how connectivity can be enable via OpenVPN encrypted tunnel
  • Last but not least, let's see how we can connect DN42 parallel network using a Wireguard tunnel relying on an IPv6 underlay.

You've guessed it, all of these points will be elaborated in the futures articles. Therefore stay tuned !

This is a new article for the blog serie called "RARE Day One". Today we will explore one of freeRouter feature that is used a lot in Service Provider trusted environment" TFTP server

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

I'm not sure if this is still the case now, but back in 1999, I had the opportunity to managed multiple VPNs at a very huge French Service Provider. I'm saying huge as in this type of MPLS muti-service core network, you could have hundreds of VRF in the same PE router connecting a myriads of CPE via X25 (XOT), frame-relay and ATM PVC at best. In that context, some companies could have several thousands of routers in their VPNs and it was not common to follow a high pace deployment which was at ~10 CPEs per day for a new customer VPN implementation. So one of my favorite CLI command was:

staging a CPE with its final configuration
copy tftp run
Address or name of remote host []? <x.y.z.t>
Source filename []? <router-cpe-config-file-name>
Destination filename [running-config]?
...

That being said, I'm not sure if this has evolved since then as TFTP occurred inside a very protected out of band management network, it was very good and did a perfect job. Keep in mind that we could be hundreds of "VPN owner" deploying CPEs at the same time. This has to be highly available.

That was for the anecdote, but recently I attempted to upgrade my OpenWRT wifi router from 18.06.02 to the latest code train: 19.07.4. As a I'm lazy, I just sticked with OpenWRT web upgrade via LuCI. Not sure if I was right ... I don't know why and how but the upgrade failed and my wifi router got "bricked". (smile)

After a lot of googling and reading, i concluded that I had only one solution: restore from factory and re-install OpenWRT 19.07.04 installation by hand. You have guess the rest of the article, the factory-reset procedure requires a TFTP server. (smile) 

Note

But before that, I had to solder an USB - UART module as described here.

Article objective

As again i was lazy on installing a TFTP server on my MAC and disconnect my current LAN access in order to have a direct connectivity with the OpenWRT box, I had an idea (this is not often (smile)) off the top of my head: "Hey, maybe freeRouter has a TFTP server that I can activate in few lines ?"... Well, after a terminal connection to my home router let me introduce you to freeRouter/TFTP server:

 

[ #004 ] - Saving private OpenWRT", thanks freeRouter's TFTP server !

If you are familiar with Cisco operating system you will feel at home with this TFTP server. 

Log into freeRouter in config mode:
   __               ____             _
  / _|_ __ ___  ___|  _ \ ___  _   _| |_ ___ _ __
 | |_| '__/ _ \/ _ \ |_) / _ \| | | | __/ _ \ '__|
 |  _| | |  __/  __/  _ < (_) | |_| | ||  __/ |
 |_| |_|  \___|\___|_| \_\___/ \__,_|\__\___|_|
  _ __ ___   ___| | _____  | |
 | '__/ _ \ / __| |/ / __| | |
 | | | (_) | (__|   <\__ \ |_|
 |_|  \___/ \___|_|\_\___/ (_)

welcome
line ready
mjolnir#conf t                                                                 
mjolnir(cfg)#server tftp openwrt                                               
mjolnir(cfg-server)#?                                                          
  access-blackhole4 - propagate and check violating prefixes
  access-blackhole6 - propagate and check violating prefixes
  access-class      - set access list
  access-log        - log dropped attemps
  access-map        - set route map
  access-peer       - per client session limit
  access-policy     - set route policy
  access-prefix     - set prefix list
  access-rate       - access rate for this server
  access-startup    - initial downtime for this server
  access-subnet     - per subnet session limit
  access-total      - session limit for this server
  do                - execute one exec command
  end               - close this config session
  exit              - go back to previous mode
  interface         - interface to bind to
  no                - negate a command
  path              - set root folder
  port              - set port to listen on
  protocol          - set lower protocols to use
  readonly          - set write protection
  security          - set security parameters
  show              - running system information
  vrf               - set vrf to use

sdn6 is the port #6 connected from my SOHO router to OpenWRT router.

TFTP server configuration
sh run tftp                                                            
server tftp openwrt
 path /rtr/owrt/
 interface sdn6
 vrf inet
 exit
!
sh run sdn6                                                           
interface sdn6
 description mjolnir@LAN6[08:00.0]
 mtu 1500
 macaddr 004c.7307.0a77
 vrf forwarding inet
 ipv4 address 192.168.136.1 255.255.255.0
 ipv4 broadcast-multicast
 no shutdown
 no log-link-change
 exit
!
...

So the LAN port of my OpenWRT router is like this:

OpenWRT config (this can be done via Web GUI)
...
config interface 'lan'
        option type 'bridge'
        option proto 'static'
        option ipaddr '192.168.136.2'
        option netmask '255.255.255.0'
        option broadcast '192.168.136.255'
        option gateway '192.168.136.1'
        option ip6assign '60'
        list dns '192.168.254.1'
        option ifname 'eth0 eth0.1 eth0.2 wlan0 wlan1' 
...

Basic connectivity check (well technically you could not ping as it is part if TFTP restore to factory process. Remember our box crashed ! (smile) )

ping OpenWRT
ping 192.168.136.2 /vrf inet                                           
pinging 192.168.136.2, src=null, vrf=inet, cnt=5, len=64, tim=1000, ttl=255, tos=0, sweep=false
!!!!!
result=100%, recv/sent/lost=5/5/0, rtt min/avg/max/total=0/1/2/5
...

So we are basically ready ...

Initiate OpenWRT factory restore process via TFTP
===================================================================
                MT7621   stage1 code 10:33:11 (ASIC)
                CPU=50000000 HZ BUS=12500000 HZ
==================================================================
Change MPLL source from XTAL to CR...
do MEMPLL setting..
MEMPLL Config : 0x11100000
3PLL mode + External loopback
=== XTAL-40Mhz === DDR-1200Mhz ===
PLL2 FB_DL: 0x9, 1/0 = 567/457 25000000
PLL3 FB_DL: 0xc, 1/0 = 596/428 31000000
PLL4 FB_DL: 0x11, 1/0 = 560/464 45000000
do DDR setting..[00320381]
Apply DDR3 Setting...(use customer AC)
          0    8   16   24   32   40   48   56   64   72   80   88   96  104  112  120
      --------------------------------------------------------------------------------
0000:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0001:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0002:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0003:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0004:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0005:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0006:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0007:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0008:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0009:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
000A:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
000B:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
000C:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
000D:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    1
000E:|    0    0    0    0    0    0    0    0    0    1    1    1    1    1    1    1
000F:|    0    0    0    0    1    1    1    1    1    1    1    1    1    1    0    0
0010:|    1    1    1    1    1    1    1    1    1    0    0    0    0    0    0    0
0011:|    1    1    1    0    0    0    0    0    0    0    0    0    0    0    0    0
0012:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0013:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0014:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0015:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0016:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0017:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0018:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
0019:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
001A:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
001B:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
001C:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
001D:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
001E:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
001F:|    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
rank 0 coarse = 15
rank 0 fine = 72
B:|    0    0    0    0    0    0    0    0    0    0    1    1    1    0    0    0
opt_dle value:11
DRAMC_R0DELDLY[018]=00001F1F
==================================================================
                RX      DQS perbit delay software calibration 
==================================================================
1.0-15 bit dq delay value
==================================================================
bit|     0  1  2  3  4  5  6  7  8  9
--------------------------------------
0 |    10 7 9 9 7 7 8 7 3 6 
10 |    6 7 7 9 6 9 
--------------------------------------

==================================================================
2.dqs window
x=pass dqs delay value (min~max)center 
y=0-7bit DQ of every group
input delay:DQS0 =31 DQS1 = 31
==================================================================
bit     DQS0     bit      DQS1
0  (1~61)31  8  (1~56)28
1  (1~58)29  9  (1~61)31
2  (1~60)30  10  (1~59)30
3  (1~58)29  11  (1~57)29
4  (1~57)29  12  (1~60)30
5  (1~61)31  13  (1~60)30
6  (1~58)29  14  (1~61)31
7  (1~62)31  15  (1~61)31
==================================================================
3.dq delay value last
==================================================================
bit|    0  1  2  3  4  5  6  7  8   9
--------------------------------------
0 |    10 9 10 11 9 7 10 7 6 6 
10 |    7 9 8 10 6 9 
==================================================================
==================================================================
     TX  perbyte calibration 
==================================================================
DQS loop = 15, cmp_err_1 = ffff0000 
dqs_perbyte_dly.last_dqsdly_pass[0]=15,  finish count=1 
dqs_perbyte_dly.last_dqsdly_pass[1]=15,  finish count=2 
DQ loop=15, cmp_err_1 = ffff0080
dqs_perbyte_dly.last_dqdly_pass[1]=15,  finish count=1 
DQ loop=14, cmp_err_1 = ffff0000
dqs_perbyte_dly.last_dqdly_pass[0]=14,  finish count=2 
byte:0, (DQS,DQ)=(8,8)
byte:1, (DQS,DQ)=(8,8)
20,data:88
[EMI] DRAMC calibration passed

===================================================================
                MT7621   stage1 code done 
                CPU=50000000 HZ BUS=12500000 HZ
===================================================================


U-Boot 1.1.3 (Apr 17 2017 - 17:00:02)

Board: Ralink APSoC DRAM:  256 MB
Power on memory test. Memory size= 256 MB...OK!
relocate_code Pointer at: 8ffac000

Config XHCI 40M PLL 
******************************
Software System Reset Occurred
******************************
Allocate 16 byte aligned buffer: 8ffdffd0
Enable NFI Clock
# MTK NAND # : Use HW ECC
NAND ID [C8 D1 80 95 42]
Device not found, ID: c8d1
Not Support this Device! 
chip_mode=00000001
Support this Device in MTK table! c8d1 
select_chip
[NAND]select ecc bit:4, sparesize :64 spare_per_sector=16
Signature matched and data read!
load_fact_bbt success 1023
load fact bbt success
[mtk_nand] probe successfully!
mtd->writesize=2048 mtd->oobsize=64,    mtd->erasesize=131072  devinfo.iowidth=8
..============================================ 
Ralink UBoot Version: 5.0.0.0
-------------------------------------------- 
ASIC MT7621A DualCore (MAC to MT7530 Mode)
DRAM_CONF_FROM: Auto-Detection 
DRAM_TYPE: DDR3 
DRAM bus: 16 bit
Xtal Mode=5 OCP Ratio=1/4
Flash component: NAND Flash
Date:Apr 17 2017  Time:17:00:02
============================================ 
icache: sets:256, ways:4, linesz:32 ,total:32768
dcache: sets:256, ways:4, linesz:32 ,total:32768 

 ##### The CPU freq = 880 MHZ #### 
 estimate memory size =256 Mbytes
#Reset_MT7530
set LAN/WAN LWLLL

Please choose the operation: 
   1: Load system code to SDRAM via TFTP. 
   2: Load system code then write to Flash via TFTP. 
   3: Boot system code via Flash (default).
   4: Entr boot command line interface.
   7: Load Boot Loader code then write to Flash via Serial. 
   9: Load Boot Loader code then write to Flash via TFTP. 
 4 
You choosed 2

 0 

   
2: System Load Linux Kernel then write to Flash via TFTP. 
 Warning!! Erase Linux in Flash then burn new one. Are you sure?(Y/N)
 Please Input new ones /or Ctrl-C to discard
        Input device IP (192.168.31.1) ==:192.168.31.1
        Input server IP (192.168.31.100) ==:192.168.31.2  
        Input Linux Kernel filename () ==: <my_factory_router_image>


...

And ... Voilà !

Note

We won this factory reset battle but the war is over. After having restored the genuine official vendor image, we need to re-install OpenWRT with the latest 19.07.4 image and configure OpenWRT so that it can acts as a "dummy Wifi Access Point". DHCP, DNS will be served by the SOHO router.

Discussion

You can deploy freeRouter manually in a VM or container and bind it to a linux interface if you need a TFTP server in order to apply configuration to all your equipment. When final staging are done in a secure Out of Band management network context having a TFTP server is a blessing as it correspond to a gain of time in a production environment. Imaging hundreds of people working in a SP environment and working at the same time.

Conclusion

In this 4th article:

  • We presented freeRouter TFTP embedded server 
  • You can use it in order to undertake network equipment deployment requiring TFTP
  • This TFTP server is compatible with IPv4/IPv6

TFTP is a basic but a common tool in SP environment (or it was? If it is still used, yes please confirm !) In this example, I demonstrated the use of TFTP server in order to flash a wifi router to factory default. I have 802.11ac back up and running !

Final words

freeRouter can be perceived not only as a router but it is a networking Swiss army knife. in further articles we will shed some lights in various treasures hidden into freeRouter... And for free !

Last but not least, you can play with these different servers from this sandbox: (You'll be able to spot amazing server that will be the object of further article.)

type "telnet dl.nop.hu" in a terminal and choose "1"
Trying 193.224.23.5...
Connected to dl.nop.hu.
Escape character is '^]'.
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXX XXXXX XXX    XXX     XXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX  XXXX XX XXXX XX XXXX XX XX XX XXXX XXXXX/~~~~~~\XXXXXX
XXXX X XXX XX XXXX XX XXXX XX XX XX XXXX XXXX| player |XXXXX
XXXX XX XX XX XXXX XX     XXX    XX XXXX XXXXX\______/XXXXXX
XXXX XXX X XX XXXX XX XXXXXXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX XXXX  XX XXXX XX XXXXXXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX XXXXX XXX    XXX XXX XXX XX XXX    XXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
welcome
line ready
menu lab:
# - reboot router1
$ - reboot router2
% - reboot router3
1 - connect to router1
2 - connect to router2
3 - connect to router3
^ - rebuild routers
l - connect to lg.nop.dn42
x - exit
choose:1 - attach vdc lab1 

welcome
line ready
yourname#conf t                                                                
warning user.userLineHandler.doExec:userLine.java:606 <nobody> configuring from tty1
yourname(cfg)#server ?                                                         
  bmp2mrt      - configure an bmp to mrt server
  bstun        - configure a bstun server
  chargen      - configure a chargen server
  daytime      - configure a daytime server
  dcp          - configure a dcp server
  dhcp4        - configure a dhcp4 server
  dhcp6        - configure a dhcp6 server
  discard      - configure a discard server
  dns          - configure a dns server
  echo         - configure an echo server
  etherip      - configure a etherip server
  forwarder    - configure a forwarder server
  ftp          - configure a ftp server
  geneve       - configure a geneve server
  gopher       - configure a gopher server
  gre          - configure a gre server
  gtp          - configure a gtp server
  honeypot     - configure a honeypot server
  http         - configure a http server
  irc          - configure an irc server
  iscsi        - configure an iscsi server
  l2f          - configure a l2f server
  l2tp2        - configure a l2tp v2 server
  l2tp3        - configure a l2tp v3 server
  loadbalancer - configure a loadbalancer server
  lpd          - configure a lpd server
  modem        - configure a modem server
  mplsip       - configure a mplsip server
  mplsudp      - configure a mplsudp server
  multiplexer  - configure a multiplexer server
  netflow      - configure an netflow server
  nrpe         - configure a nrpe server
  ntp          - configure a ntp server
  openflow     - configure an openflow server
  p4lang       - configure an p4lang server
  pcep         - configure a pcep server
  pckodtls     - configure a pckodtls server
  pckotcp      - configure a pckotcp server
  pckotxt      - configure a pckotxt server
  pckoudp      - configure a pckoudp server
  pop3         - configure a pop3 server
  pptp         - configure a pptp server
  prometheus   - configure a prometheus server
  quote        - configure a quote server
  radius       - configure a radius server
  rfb          - configure a rfb server
  rpki         - configure a rpki server
  sip          - configure a sip server
  smtp         - configure a smtp server
  snmp         - configure a snmp server
  socks        - configure a socks server
  streamingmdt - configure a streaming telemetry server
  stun         - configure a stun server
  syslog       - configure a syslog server
  tacacs       - configure a tacacs server
  telnet       - configure a telnet server
  tftp         - configure a tftp server
  time         - configure a time server
  udpfwd       - configure an udp forwarder server
  udptn        - configure an udptn server
  upnpfwd      - configure an upnp forwarder server
  upnphub      - configure an upnp hub server
  voice        - configure a voice server
  vxlan        - configure a vxlan server

yourname(cfg)#server                              
...

In order to exit the sandbox session use the following escape sequence: Ctrl-c + Ctrl-x









This is a new category of article that falls under "RARE software architecture" special blog series. As its name implies, it deals with topics related to RARE/freeRouter software / Monitoring.

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

In Greek mythology, Prometheus is a Titan that is credited mankind creation by stealing Fire from Gods and by giving it to human. In the RARE context, Prometheus is a the software from prometheus.io project. It became very popular in the IT industry as it is very simple to implement/configure while providing a great number of metrics without impacting application performance. It is heavily used in microservices environment such as docker and Kubernetes. The mythological reference gives us an indication of how Prometheus is operating. At a constant rate, Prometheus metric collector or server is stealing metrics from Prometheus agent. All the stolen metrics are then consolidated in Time Series database ready to be poured to a queueing system for proper visualization. 

Before going further, allow me a brief digression by sharing with you a small anecdote that leds to this ongoing work related to network monitoring for RARE. As mentioned previously, our focus is to elaborate RARE/freeRouter solution the possibility to be monitored in an operational environment. In that context, we started with the implementation of a lightweight SNMP stack that provided relevant result via SNMP tools like LibreNMS. This is great for organisation that wouldn’t want invest time on anything but SNMP.

However, we felt a lack of flexibility due to SNMP inherent structure and we needed more versatile and instant monitoring capabilities.  More importantly the need to export infinite metric type from Control Plane in a more flexible way arise. How metrics such as: Number of IPv4/IPv6 routes, IPv4 BGP prefix, IPv6 BGP prefix platform JVM memory etc. could be shared without too much hassle ?

After some internal discussion, I just said: "I’m not a monitoring expert but we have tools like ELK and PROMETHEUS and GRAFANA in NMaaS catalog … Shouldn’t we consider use this ?"

The answer was: « Let’s give it a try and fire up a Prometheus and Grafana instance from NMaaS platform !»

Some hacking at the control plane code level were initiated, after few hours freeRouter lead developer came up with a solution and said: Let me introduce you "freeRouter prometheus agent »

And thanks to the great support of NMaaS team, in few minutes and some point and clicks (it took longer than expected as I’m not good with GUI) we were able to test this agent.

Why is it important you might say ? It is just that with prometheus simplicity and low resource overhead with have full control plane metrics visibility !

As a side note this is not a replacement for INT/Telemetry/Netflow/IPFIX that provide different type of data that are to at the same scale…
People with INT/TELEMETRY/NETFLOW/IPFIX are talking about a "data lake" or "data deluge". Which is correct, if you think about the complexity of resolving a  gigantic producer/consumer data problem. This needs the relevant IT infrastructure in order to process all of the data provided by these protocol at the NREN scale.

While in our case, we are just focusing on exposing CONTROL PLANE METRICS at the network element level. We simply monitor and ensure a router operation by using prometheus metrics

Note

While he above might be true, the number of metrics exported from a prometheus target can be very high. Fine tuning might be necessary in order to make sure that all metrics are really necessary for network monitoring purpose. This explosion of metrics exposure can add unnecessary workload at the control plane level. 

Again, kudos to NMaaS team that made this happen so that we could test this on the P4 LAB with — ZERO — effort.

Article objective

In this article, we will present freeRouter and Prometheus integration and as an example we will implement one of the 22 grafana dashboard that we developed and published here. In the rest of the article we will assume that you are a running one or more freeRouter nodes.

Diagram

[ #001 ] - Cookbook

The first step is to implement a prometheus server. Using NMaaS it is pretty instantaneous. However, if you plan to deploy prometheus in an other platform just follow the installation guide here.

Once deployed you can push the following prometheus.yaml config:

prometheus.yaml
global:
  scrape_interval: 15s
  evaluation_interval: 30s
alerting:
  alertmanagers:
    - static_configs:
      - targets:
rule_files:
scrape_configs:
  - job_name: 'router'
    metrics_path: /metrics
    scrape_interval: 15s
    static_configs:
    - targets: ['192.168.0.1:9001','192.168.0.2:9001']
      labels:

In this configuration we assume that we have 2 freeRouters that are configured as above (192.168.0.1:9001 and 192.168.0.2:9001) in prometheus worls these are called targets:

  • each target are interrogated or "scraped" very "scrap_interval" which is 15s here
  • the main job name is called; "router"
  • metrics_path is: "/metrics" so the scraped URL is: "http://192.168.0.1:9001/metrics

Note that this had to be deployed only once for all of your routers. However, each time you'd like to add a new router, you have to add a new target in the "targets" YAML list.

In this example let's focus our interest interface metrics. Please note that this configuration should be deployed on each freeRouter and connectivity should be available between all targets and the prometheus server.

The objective is to tell freeRouter control plane to expose hardware and software counter interface metric. In order to do this just copy/paste the stanza here below via freeRouter CLI:

prometheus interface metric configuration
!
server prometheus <PROMETHEUS_SERVER_NAME>
 metric inthw command sho inter hwsumm
 metric inthw prepend iface_hw_byte_
 metric inthw name 0 ifc=
 metric inthw replace \. _
 metric inthw column 1 name st
 metric inthw column 1 replace admin -1
 metric inthw column 1 replace down 0
 metric inthw column 1 replace up 1
 metric inthw column 2 name tx
 metric inthw column 3 name rx
 metric inthw column 4 name dr

 metric intsw command sho inter summ
 metric intsw prepend iface_sw_byte_
 metric intsw name 0 ifc=
 metric intsw replace \. _
 metric intsw column 1 name st
 metric intsw column 1 replace admin -1
 metric intsw column 1 replace down 0
 metric intsw column 1 replace up 1
 metric intsw column 2 name tx
 metric intsw column 3 name rx
 metric intsw column 4 name dr
 vrf <VRF_NAME>
 exit
!

So this basically means:

  • From freeRouter CLI, issue the following command:
prometheus interface metric configuration
sho inter hwsumm
interface   state  tx          rx          drop
hairpin41   up     67404       0           0
hairpin42   up     153134      0           0
sdn1        up     412319805   1057514903  1152305
sdn2        up     1038840147  407307558   202
sdn3        admin  0           0           0
sdn4        admin  0           0           0
sdn5        admin  0           0           0
sdn6        admin  0           0           0
sdn998      up     9154        0           0
sdn999      up     199178      262939      0
tunnel1965  up     0           9122896     0 
  • prepend to the metric name: "iface_hw_byte_"
  • column 0 will have prometheus label ifc=
  • replace all dots "." by "_" . (so interface bundle1.123 will become bundle1_123)
  • column defines a metric name  "iface_hw_byte_" concatenated to "st" => "iface_hw_byte_st" which is essentially interface status
  • if column 1 "state" value is admin/down/up we associate value -1/0/1 
  • column defines a metric name  "iface_hw_byte_" concatenated to "tx" => "iface_hw_byte_tx" which is essentially interface bytes transmitted counter 
  • column defines a metric name  "iface_hw_byte_" concatenated to "rx" => "iface_hw_byte_rx" which is essentially interface bytes received counter 
  • column defines a metric name  "iface_hw_byte_" concatenated to "dr" => "iface_hw_byte_dr" which is essentially interface bytes dropped counter 

And if you followed this correctly, we are repeating these lines for software interface counter metric.

Tip

You can view Prometheus configuration for various Grafana dashboard here. Feel free to study these Prometheus configuration and activate them as you see fit depending on your requirements. The set of dashboard is not exhaustive and is by no means absolute. Feel free to submit additional dashboard ! We would gladly add them in the current list of freeRouter Dashboard.

Note

After this definition a freeRouter level you should have:

4 metrics related to hardware counters 

  • iface_hw_byte_st
  • iface_hw_byte_tx
  • iface_hw_byte_rx
  • iface_hw_byte_dr

4 metrics related to software counters

  • iface_sw_byte_st
  • iface_sw_byte_tx
  • iface_sw_byte_rx
  • iface_sw_byte_dr

Which is a total of 8 metrics

Tip

From that point you can check via prometheus console:

check the "Targets" menu drop down selection

From that point you should be able to use PromQL query filed in order to check that you can retrieve the metrics we defined above.

For metric visualisation, we will use Grafana. Therefore:

  • install Grafana from official web site.
  • Once installed configure Prometheus as Grafana data source:

 

  • fill in all the prometheus server information

  • check the the data source is defined correctly by clicking the "Save & test" button

At that point your Grafana and Prometheus are correctly binded.

  • now you need to import "RARE/freeRouter interface bytes" dashboard

  • download freeRouter interface bytes dashboard here 


  • import the dashboard via ID or simply download JSON or use JSON panel

And Voila ! 

In order to immediately see the graph zoom in to 5m period with a refresh of 5s and you should see automagically the interface bytes TX/RX on all interface for each targets.

Discussion

This example related to interface metrics is universal, as the metrics at freeRouter level are yielded through a generic CLI command:

  • "show interface hwsummary"
  • or "show interface swsummary".

However some metrics cannot be retrieved by generic interface. Some metrics will be tied to specificities of your network. These can be the AS number, IGP process name, VRF name etc.

Let me give you a couple of examples:

But your network context you could have arbitrary deployed "isis 2200". (2200 is RENATER AS number) 

prometheus interface metric configuration
 metric lsigp4int command sho ipv4 isis 1 interface
 metric lsigp4int prepend lsigp4_iface_
 metric lsigp4int name 0 proto="isis1",ifc=
 metric lsigp4int replace \. _
 metric lsigp4int column 1 name neighbors
 metric lsigp4peer command sho ipv4 isis 1 topology 2
 metric lsigp4peer prepend lsigp4_peers_
 metric lsigp4peer name 0 proto="isis1",node=
 metric lsigp4peer replace \. _
 metric lsigp4peer column 1 name reachable
 metric lsigp4peer column 1 replace false 0
 metric lsigp4peer column 1 replace true 1
 metric lsigp4peer column 6 name neighbors
 metric lsigp4perf command sho ipv4 isis 1 spf 2 | inc reachable|fill|calc|run
 metric lsigp4perf prepend lsigp4_perf_
 metric lsigp4perf labels proto="isis1"
 metric lsigp4perf skip 0
 metric lsigp4perf column 1 name val

 metric lsigp6int command sho ipv6 isis 1 interface
 metric lsigp6int prepend lsigp6_iface_
 metric lsigp6int name 0 proto="isis1",ifc=
 metric lsigp6int replace \. _
 metric lsigp6int column 1 name neighbors
 metric lsigp6peer command sho ipv6 isis 1 topology 2
 metric lsigp6peer name 0 proto="isis1",node=
 metric lsigp6peer prepend lsigp6_peers_
 metric lsigp6peer replace \. _
 metric lsigp6peer column 1 name reachable
 metric lsigp6peer column 1 replace false 0
 metric lsigp6peer column 1 replace true 1
 metric lsigp6peer column 6 name neighbors
 metric lsigp6perf command sho ipv6 isis 1 spf 2 | inc reachable|fill|calc|run
 metric lsigp6perf prepend lsigp6_perf_
 metric lsigp6perf labels proto="isis1"
 metric lsigp6perf skip 0
 metric lsigp6perf column 1 name val


prometheus interface metric configuration
 metric bgp4peer command sho ipv4 bgp 65535 summ
 metric bgp4peer prepend bgp4_peer_
 metric bgp4peer name 4 peer=
 metric bgp4peer replace \. _
 metric bgp4peer column 1 name learn
 metric bgp4peer column 2 name advert
 metric bgp4peer column 3 name state
 metric bgp4peer column 3 replace false 0
 metric bgp4peer column 3 replace true 1
 metric bgp4perf command sho ipv4 bgp 65535 best | exc last
 metric bgp4perf prepend bgp4_perf_
 metric bgp4perf replace \s _
 metric bgp4perf column 1 name val

 metric bgp6peer command sho ipv6 bgp 65535 summ
 metric bgp6peer prepend bgp6_peer_
 metric bgp6peer name 4 peer=
 metric bgp6peer replace \: _
 metric bgp6peer column 1 name learn
 metric bgp6peer column 2 name advert
 metric bgp6peer column 3 name state
 metric bgp6peer column 3 replace false 0
 metric bgp6peer column 3 replace true 1
 metric bgp6perf command sho ipv6 bgp 65535 best | exc last
 metric bgp6perf prepend bgp6_perf_
 metric bgp6perf replace \s _
 metric bgp6perf column 1 name val 
prometheus interface metric configuration
 metric ldp4nul command sho ipv4 ldp inet nulled-summary
 metric ldp4nul prepend ldp4null_
 metric ldp4nul name 3 ip=
 metric ldp4nul skip 2
 metric ldp4nul replace \. _
 metric ldp4nul column 0 name prefix_learn
 metric ldp4nul column 1 name prefix_advert
 metric ldp4nul column 2 name prefix_nulled

 metric ldp6nul command sho ipv6 ldp inet nulled-summary
 metric ldp6nul prepend ldp6null_
 metric ldp6nul name 3 ip=
 metric ldp6nul skip 2
 metric ldp6nul replace \: _
 metric ldp6nul column 0 name prefix_learn
 metric ldp6nul column 1 name prefix_advert
 metric ldp6nul column 2 name prefix_nulled 


Conclusion

In this 1st article, you were presented :

  • freeRouter/Prometheus integration 
  • How to add a new router in the list of Prometheus target
  • How to integrate a RARE/freeRouter Grafana Dashboard. (Feel free to adapt the other available dashboard query to your context !)

Final words

In Prometheus philosophy, normally the user should do only the minimum of tweaking regarding configuration. Ultimately, he should be only be able to enable a metric or simply disable it if the scrape cost is too high. However in freeRouter/Prometheus integration process, you see that some metric are issued using specific $variable (VRF, BGP/IGP process number ...) Which makes impossible to maintain this universality. However, from the network operator point of view this should not be a showstopper. On the contrary, it is a powerful choice to be able to alter these command via $variables.

Remember in freeRouter philosophy you can have multiple VRF, multiple IGP and multiple BGP process number ! (Which is not the case for all routing platform)

Last but not least, this Prometheus agent was developed quickly because of one reason, all the objects  at the control plane level were already well structured in table form as previously described in this article. So implementing this table row/column logic in order to derive a prometheus metric was technically possible without too much hassle.





This is a new article for the blog serie called "RARE Day One". Today we will explore one of freeRouter features meant to fine tune terminal user environment and behaviour in order to best match your taste/preferences.

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

We all have our habits that are inherited from our past experience. Some people are used to IOS, IOS-XR, NX-OX, IOS-XE others prefer Junos etc. Using freeRouter provides a different user experience. Some feature such as show/view/watch/differ diagnosis commands are pretty unique to freeRouter. However, freeRouter have some cards in its sleeves in order to provide you a familiar experience.

Article objective

In this article, we will focus on these features:

  • monitor
  • length
  • width
  • spacetab
  • tablemode
  • timestamps
  • colorized

Basically these commands are accessed through freeRouter user mode. If you need to use them from config mode, please use the "do" keyword.

[ #003 ] - "monitor/length/width/spacetab/tablemode/timestamps/colorized"

If you are familiar with Cisco operating system you will feel at home with "terminal monitor" mode. This mode is usually used in combination with "debug" diagnosis command and actually redirect console amd debug output also in your current terminal session. (VTY in Cisco language)

Let's assume we want to debug IPv4 BGP

show BGP configuration from running config
r1#debug proto bgp ?                                                      
  computation - computation events
  event       - table events
  full        - full events
  incremental - incremental events
  traffic     - interface packets

Let's activate BGP debug event

Check BGP IPv4 peers status in VRF dn42
r1#debug rtr.rtrBgpSpeak.packRecv:rtrBgpSpeak.java:1134 got update from 172.23.215.177
debug rtr.rtrBgp.routerCreateComputed:rtrBgp.java:1902 create table
debug rtr.rtrBgpSpeak.packSend:rtrBgpSpeak.java:1080 sending update to 172.23.215.177
r1#                                                                       
r1#debug rtr.rtrBgpSpeak.packRecv:rtrBgpSpeak.java:1134 got keepalive from 172.23.215.177
debug rtr.rtrBgpSpeak.packSend:rtrBgpSpeak.java:1080 sending keepalive to 172.23.215.177
debug rtr.rtrBgpSpeak.packRecv:rtrBgpSpeak.java:1134 got update from fd40:cc1e:c0de::151
debug rtr.rtrBgp.routerCreateComputed:rtrBgp.java:1902 create table
debug rtr.rtrBgpSpeak.packSend:rtrBgpSpeak.java:1080 sending update to fd40:cc1e:c0de::151
debug rtr.rtrBgpSpeak.packRecv:rtrBgpSpeak.java:1134 got update from fd40:cc1e:c0de::151
debug rtr.rtrBgp.routerCreateComputed:rtrBgp.java:1902 create table
debug rtr.rtrBgpSpeak.packRecv:rtrBgpSpeak.java:1134 got update from fd40:cc1e:c0de::151
debug rtr.rtrBgp.routerCreateComputed:rtrBgp.java:1902 create table
debug rtr.rtrBgpSpeak.packSend:rtrBgpSpeak.java:1080 sending update to fd40:cc1e:c0de::151
debug rtr.rtrBgpSpeak.packRecv:rtrBgpSpeak.java:1134 got update from fd40:cc1e:c0de::151
debug rtr.rtrBgp.routerCreateComputed:rtrBgp.java:1902 create table
...

in order to cancel debug:

undebug all (or specific debug)
r1#un all
...

in order to stop console output from your terminal session:

termonal no monitor
r1#term no mon
...

Note

Similar to Cisco gear, "debug" can be very chatty. Therefore, be ready to issue "term no mon" or better log debug into a file for further off-line forensics.

In my previous examples, the output of "show ipv4 bgp 42 unicast database" command could not fit my window as the output as tomany lines. "terminal length" can be used to alter the number of  lines of the terminal.

Check hardware traffic counters
r1##terminal length ?                                                      
  <num> - height in lines

r1#terminal length
...

Note

"terminal length" can have no effect if you are using a more sophisticated terminal. However, this will have a visible impact on view/display/differ buffer.

"terminal spacetab" is specifically decidated to Junos user. It basically does the same effect as the <TAB> key, but it add also contextual completion with the <SPACE>

Check hardware traffic counters
r1#terminal spacetab ?                                                    
  <cr>

r1#terminal spacetab                                                      
...

Note

"terminal spacetab" does not remove <TAB> key behaviour.

"terminal tablemode" provide pre-formatted table output. 

"terminal tablemode" available format
mjolnir#terminal tablemode ?                                                   
  csv    - select csv mode
  fancy  - select fancy mode
  html   - select html mode
  normal - select normal mode
  raw    - select raw mode
  table  - select table mode
r1#terminal spacetab ?                                                    
  <cr>

Let's select "fancy"

Fancy mode
r1#show ipv4 bgp 42 summary                                               
 |~~~~~~~~~~~~|~~~~~~~|~~~~~~|~~~~~~~|~~~~~~~~~~~~~~~~|~~~~~~~~~~|
 | as         | learn | done | ready | neighbor       | uptime   |
 |------------|-------|------|-------|----------------|----------|
 | 4242421955 | 516   | 517  | true  | 172.23.215.177 | 01:16:23 |
 |____________|_______|______|_______|________________|__________|

Note

Feel free to play all format proposed by "terminal tablemode". This is pretty useful when you have to prepare some report related to the VPN or network you are currently managing.

"terminal timestamps" will simply prepend command timestamps.

Fancy mode
r1#show ipv4 bgp 42 summary                                               
2020-09-30 09:46:32
 |~~~~~~~~~~~~|~~~~~~~|~~~~~~|~~~~~~~|~~~~~~~~~~~~~~~~|~~~~~~~~~~|
 | as         | learn | done | ready | neighbor       | uptime   |
 |------------|-------|------|-------|----------------|----------|
 | 4242421955 | 516   | 517  | true  | 172.23.215.177 | 01:18:45 |
 |____________|_______|______|_______|________________|__________|

Note

As you can see, you can stack these modes. Here we activated "terminal tablemode + timestamps"

"terminal colorized" will simply  color  your prompt

"terminal <mode>" is specific to your current session. If you want a persistent behaviour you would need to activate these features from the  "server telnet" stanza. Which as its name wrongly implies, is not about configuring a telnet server only. From this stanza you'll able to configure any type of server dedicated to terminal connection. (SSH)

Fancy mode
r1#(cfg-server)#exec ?                                                     
  authorization - set authorization
  autocommand   - set automatic command
  autohangup    - disconnect user after autocommand
  bye           - set goodbye message
  colorized     - enable colorization
  height        - set height of terminal
  interface     - set interface to use for framing
  logging       - enable logging
  privilege     - set default privilege
  ready         - set ready message
  spacetab      - enable space as tab
  tablemode     - set table mode
  timeout       - set timeout value
  timestamp     - enable timestamps
  welcome       - set welcome message
  width         - number of columns

Note

This "server telnet" section will provide you lots of possibility to fine tune your terminal access !  Feel free to use them in order to feel at home !

Discussion

monitor/length/width/spacetab/tablemode/timestamps/colorized is a set of feature meant to ease your experience with freeRouter in mimic'ing well know behaviour and proposing you additional convenient features. One main behaviour is that all command issue from the CLI is instantly taken into account. 

Conclusion

In this 3rd article:

  • We presented freeRouter monitor/length/width/spacetab/tablemode/timestamps/colorized terminal customization command
  • These are very useful if you come from Cisco or Junos world as it mimic some termnal behaviour.

Final words

As said, these are terminal commands are not specific to freeRouter. Some behaviour are mimic'ed from IOS and Junos. Anyway, these have been developed for one purpose:

"Make network engineers feel at ease and provide then an enjoyable operation experience "

Feel free to try and use them according to your environment taste!

Last but not least, you can play with these different mode from this sandbox:

type "telnet dl.nop.hu" in a terminal and choose "1"
telnet dl.nop.hu
Trying 193.224.23.5...
Connected to dl.nop.hu.
Escape character is '^]'.
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXX XXXXX XXX    XXX     XXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX  XXXX XX XXXX XX XXXX XX XX XX XXXX XXXXX/~~~~~~\XXXXXX
XXXX X XXX XX XXXX XX XXXX XX XX XX XXXX XXXX| player |XXXXX
XXXX XX XX XX XXXX XX     XXX    XX XXXX XXXXX\______/XXXXXX
XXXX XXX X XX XXXX XX XXXXXXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX XXXX  XX XXXX XX XXXXXXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX XXXXX XXX    XXX XXX XXX XX XXX    XXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
welcome
line ready
menu lab:
# - reboot router1
$ - reboot router2
% - reboot router3
1 - connect to router1
2 - connect to router2
3 - connect to router3
^ - rebuild routers
l - connect to lg.nop.dn42
x - exit
choose:1 - attach vdc lab1 

welcome
line ready
yourname#terminal ?                                                            
  colorized  - sending to ansi terminal
  length     - set terminal length
  monitor    - log to this terminal
  no         - negate a parameter
  spacetab   - treat space as tabulator
  tablemode  - select table formatting mode
  timestamps - put time before each executed command
  width      - set terminal width

yourname#terminal                            
...

In order to exit the sandbox session use the following escape sequence: Ctrl-c + Ctrl-x








This is a new article for the blog serie called "RARE Day One". Today we will explore one of freeRouter killer feature that will make your life easier during your day to day operation: freeRouter assisted diagnosis command.

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

As previously mentioned in the precedent article, when you log into a network equipment such as a router, you tend to have some automatic reflex. You usually:

  • Check router configuration: show run or sh conf
  • Check ipv4 / ipv6 / or LFIB forwarding table
  • So you basically issue diagnosis, troubleshooting command
  • An then you want to configure the router

Article objective

In this article, we will focus on the 3rd bullet point and will present you freeRouter available diagnosis command. They are grouped into 5 categories:

  • show 
  • view
  • watch
  • display
  • differ

Basically these commands are accessed through freeRouter user mode. If you need to use them from config mode, please use the "do" keyword.

[ #002 ] - "show/view/watch/display/differ"

You would mostly be familiar with the "show" command. It is very good and can basically be used to get output from control plane object. Most of the time this can be used against static object like config.

Let's assume that I would like to get BGP config from my home router:

show BGP configuration from running config
show running-config bgp4                                               
router bgp4 42                                                                 
 vrf dn42                                                                      
 local-as 4242421975                                                           
 router-id 172.22.105.65                                                       
 address-family unicast multicast other flowspec vpnuni vpnmlt vpnflw ovpnuni ovpnmlt ovpnflw vpls mspw evpn mdt srte mvpn omvpn
 neighbor 172.23.215.177 remote-as 4242421955                                  
 neighbor 172.23.215.177 description NOP.DN42                                  
 neighbor 172.23.215.177 local-as 4242421975                                   
 neighbor 172.23.215.177 address-family unicast multicast other flowspec vpnuni vpnmlt vpnflw ovpnuni ovpnmlt ovpnflw vpls mspw evpn mdt srte mvpn omvpn
 neighbor 172.23.215.177 distance 20                                           
 justadvert loopback42                                                         
 exit                      

But I can also check the status of BGP peering into VRF dn42

Check BGP IPv4 peers status in VRF dn42
show ipv4 bgp 42 summary                                               
as          learn  done  ready  neighbor        uptime
4242421955  517    518   true   172.23.215.177  00:38:18                      

Check the same BGP peering but now for IPv6

Check BGP IPv6 peers status in VRF dn42
r1#show ipv6 bgp 42 summary                                               
as          learn  done  ready  neighbor             uptime
4242421955  351    352   true   fd40:cc1e:c0de::151  00:40:40
show ipv4 bgp 42 summary                                               
as          learn  done  ready  neighbor        uptime
4242421955  517    518   true   172.23.215.177  00:38:18                      

Let's see some BGP prefix received in VRF dn42 bgp table:

so my screen is too small for all the IPv6 BGP prefix into DN42 VRF

As a last example, something we usually do as network operators is to check ongoing interface traffic level:

Check interface traffic level (received/transmitted) )
r1#sh int sdn1                                                            
sdn1 is up (since 09:41:21, 2 changes)
 description: mjolnir@LAN1[01:00.0]
 type is sdn, hwaddr=003b.7671.764f, mtu=1500, bw=8000kbps, vrf=inet
 ip4 address=192.168.0.90/24, netmask=255.255.255.0, ifcid=10013
 ip6 address=2a01:e0a:159:2850::666/64, netmask=ffff:ffff:ffff:ffff::, ifcid=10013
 received 52013 packets (17638316 bytes) dropped 5 packets (448 bytes)
 transmitted 80765 packets (15101696 bytes) promisc=false macsec=false

r1#sh int sdn1                                                            
sdn1 is up (since 09:41:22, 2 changes)
 description: mjolnir@LAN1[01:00.0]
 type is sdn, hwaddr=003b.7671.764f, mtu=1500, bw=8000kbps, vrf=inet
 ip4 address=192.168.0.90/24, netmask=255.255.255.0, ifcid=10013
 ip6 address=2a01:e0a:159:2850::666/64, netmask=ffff:ffff:ffff:ffff::, ifcid=10013
 received 52013 packets (17638316 bytes) dropped 5 packets (448 bytes)
 transmitted 80766 packets (15101778 bytes) promisc=false macsec=false

r1#sh int sdn1                                                            
sdn1 is up (since 09:41:24, 2 changes)
 description: mjolnir@LAN1[01:00.0]
 type is sdn, hwaddr=003b.7671.764f, mtu=1500, bw=8000kbps, vrf=inet
 ip4 address=192.168.0.90/24, netmask=255.255.255.0, ifcid=10013
 ip6 address=2a01:e0a:159:2850::666/64, netmask=ffff:ffff:ffff:ffff::, ifcid=10013
 received 52015 packets (17638418 bytes) dropped 5 packets (448 bytes)
 transmitted 80766 packets (15101778 bytes) promisc=false macsec=false
                        

In the last example we repeatedly issue the "sh int sdn1" command and try to see if TX/RX packets counters increment or not.

This command can be improved in order to be less chatty:

Check interface traffic level (received/transmitted) )
r1#sh int sdn1 | i received|transmitted                            
 received 52256 packets (17681204 bytes) dropped 5 packets (448 bytes)
 transmitted 81130 packets (15162642 bytes) promisc=false macsec=false

r1#sh int sdn1 | i received|transmitted                            
 received 52256 packets (17681204 bytes) dropped 5 packets (448 bytes)
 transmitted 81130 packets (15162642 bytes) promisc=false macsec=false

r1#sh int sdn1 | i received|transmitted                            
 received 52260 packets (17681496 bytes) dropped 5 packets (448 bytes)
 transmitted 81132 packets (15162790 bytes) promisc=false macsec=false

Same goes if want want interface traffic for all interface

Check interface traffic level (received/transmitted) )
show interfaces summary                                                
interface   state  tx        rx        drop
loopback0   up     65856     0         0
loopback42  up     65856     0         0
ethernet0   up     31071917  33183183  0
hairpin41   up     85806     85552     0
hairpin42   up     85806     85552     0
sdn1        up     15200591  17703953  448
sdn2        up     15563546  8000994   794
sdn3        admin  0         0         0
sdn4        admin  0         0         0
sdn5        admin  0         0         0
sdn6        admin  0         0         0
sdn998      up     5850      0         0
sdn999      up     23268     18666     0
tunnel1965  up     5222281   7124950   0

Above was to check interface status related to software switched packet. What if I want to check hardware switched packet counters switched by P4 or DPDK ?

Check interface traffic level (received/transmitted) )
show interfaces hwsummary                                              
interface   state  tx         rx         drop
hairpin41   up     0          0          0
hairpin42   up     0          0          0
sdn1        up     317902736  590402538  1162971
sdn2        up     574923844  310497399  203
sdn3        admin  0          0          0
sdn4        admin  0          0          0
sdn5        admin  0          0          0
sdn6        admin  0          0          0
sdn998      up     9062       0          0
sdn999      up     103804     64470      0
tunnel1965  up     0          1301312    0

Note

As a network operator, the "show" command is your best friend, your wingman. Just explore now from freeRouter CLI using "show ?" and you'll understand the amazing list of diagnosis command available.

In my previous examples, the output of "show ipv4 bgp 42 unicast database" command could not fit my window. Say hello to "view" keyword then !  

Let's now try to get hardware counters as above:

Check hardware traffic counters
r1#view ipv4 bgp 42 unicast database
...

Then you'll be able to see READ-ONLY text buffer where you can navigate and check the output that are beyond boundaries of your screen !

Note

"view" is similar to "show" but it will let you deal with a fixed buffer. "view" buffer won't be refreshed.

As mentioned above, "show" gives you diagnosis instant photo of a control plane object. In order to see counter increment, you'd have to issue "show" repeatedly. In order to avoid that, let me introduce you the "watch" command

Let's now try to get hardware counters as above:

Check hardware traffic counters
r1#watch interfaces hwsummary
...

It will clear the terminal session and gives you the same outout as above but with counter updated in a regular basis

So in this example you'll see a live output with counter incrementing. In the screenshot it is not noticeable, but in real life this is bluffing. See watch interface pretty much like Junos "monitor" keyword.

So needless to say that "watch" is applicable to every control plane object such as BGP:

Amazing, don't you think ?

In my previous examples, the output of "show ipv4 bgp 42 unicast database" command could not fit my window. Say hello to "display" keyword then !  

Display BGP prefix from dn42 VRF
r1#display ipv4 bgp 42 unicast database
...

Then you'll be able to see READ-ONLY text buffer where you can navigate and check the output that are beyond boundaries of your screen !

As a side note, you can benefit from online help by pressing <f1>

You can press Ctrl+q in order to exit the editor. As the viewer is a READ-ONLY buffer

Note

Use "display" for output that have output that does not fit into your screen. "display" shows a buffer that is auto-refreshed similar to "watch". But instead the output is thrown into a buffer where you can navigate. But display, very useful to diagnoses object such as huge:

  • ACL
  • prefix-list
  • route policy list
  • route-map

As opposed to "view", "display" proposes an auto-refresh version of the buffer ! 

Last but not list. "differ" , this will split the window in 2 buffers reflecting the same output but with different version and it it signal line lines that have changed. 

Check BGP best path computation for BGP process 42
r1#diff ipv4 bgp 42 bestpath
...

With this view you can easily spot the differences between 2 advertisements interval.

To be honest, when i used this feature for the first time I was totally stumbled and said: Waouw ...

Simply amazing ... 

Discussion

show/view/watch/display/differ is pretty unique to freeRouter, and is really meant to provide you the best user experience as a network operator ! These command have proven to be helpful, especially if you deal with huge feed. However, be careful when you are working with very big output such BGP full feed. This won't crash the router of course as we used to when we issued "debug ip packet" but it will for sure imply a high CPU usage due to regular refresh at the control plane level.

Conclusion

In this 2nd article:

  • We presented freeRouter show/watch/display/differ diagnisis command
  • These are very useful when you have to deal with huge command output buffer.

Final words

As said, these are diagnosis commands are specific to freeRouter. 2 decades of know how and network experience have been pushed into these feature codes. These have been developed for one purpose:

"Provide a unique operation experience to network engineers"

Feel free to try and use them according to your environment taste!

Last but not least, you can play with these different mode from this sandbox:

type "ssh dl.nop.hu" in a terminal (any user/pass will do) and choose "l"
ssh dl.nop.hu -l random_user                                                                                                                              
Warning: Permanently added 'dl.nop.hu,193.224.23.5' (RSA) to the list of known hosts.
random_user@dl.nop.hu's password: 
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXX XXXXX XXX    XXX     XXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX  XXXX XX XXXX XX XXXX XX XX XX XXXX XXXXX/~~~~~~\XXXXXX
XXXX X XXX XX XXXX XX XXXX XX XX XX XXXX XXXX| player |XXXXX
XXXX XX XX XX XXXX XX     XXX    XX XXXX XXXXX\______/XXXXXX
XXXX XXX X XX XXXX XX XXXXXXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX XXXX  XX XXXX XX XXXXXXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX XXXXX XXX    XXX XXX XXX XX XXX    XXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
welcome
line ready
menu lab:
# - reboot router1
$ - reboot router2
% - reboot router3
1 - connect to router1
2 - connect to router2
3 - connect to router3
^ - rebuild routers
l - connect to lg.nop.dn42
x - exit
choose:l - telnet 172.23.199.110 23 /telnet 
 - connecting to 172.23.199.110 23
 - securing connection

hi there!
try the following:
  show ipv4 route dn42
  show ipv6 route dn42
  show ipv4 bgp 65535 vpnuni summary
  show ipv6 bgp 65535 vpnuni summary
  show ipv4 bgp 65535 vpnuni database
  show ipv6 bgp 65535 vpnuni database
  show ipv4 bgp 65535 vpnuni allroute <prefix> 65535:42
  show ipv6 bgp 65535 vpnuni allroute <prefix> 65535:42
  show ipv4 logger 42 flapstat 10
  show ipv6 logger 42 flapstat 10
  show ipv4 bgp 65535 vpnuni flapstat 10
  show ipv6 bgp 65535 vpnuni flapstat 10
  show ipv4 bgp 65535 vpnuni flappath <prefix> 65535:42
  show ipv6 bgp 65535 vpnuni flappath <prefix> 65535:42
have fun!
mc36
welcome
line ready
player-dn42>                                                                   
player-dn42>                   
...

Then issue a "diff" command:

differ example with BGP command
player-dn42>diff ipv4 bgp 65535 vpnuni database 10.11.160.0/20 65535:42
...

You'll be rewarded by this diff out related to the command which means:

"show me the prefix status of 10.11.160.0/20 within BGP process 65535 with rd: 65535:42"

After a quick look at VRF definition indicates that rd 65535:42 is tied to VRF dn42:

Check vrf list on router
player-dn42>sh start vrf                                                       
vrf definition dn42
 rd 65535:42
 rt-import 65535:42
 rt-export 65535:42
 source4route all
 source6route all
 mdt4
 mdt6
 exit
vrf definition rtbh
 rd 65535:666
 rt-import 65535:666
 rt-export 65535:666
 exit
vrf definition vpn
 rd 65535:1
 rt-import 65535:1
 rt-export 65535:1
 mdt4
 mdt6
 exit
...

In order to exit the sandbox session use the following escape sequence: Ctrl-c + Ctrl-x







This is a special blog series called "RARE Day One". I've always been a huge Cisco and JUNIPER fans, Cisco has unparalleled documentation and I really like JUNIPER "Day One" or "This Week" booklets. Similar to JUNIPER approach RARE "Day One" articles are dealing with essential topics that you need to get familiar with and that will become handy during your "RARE-freeRouter"-FU practices !  

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

Even in the era of zero touch configuration where everything can be modelled by YANG and automated by Ansible, CLI configuration mode is essential and will take a special important place into network engineers' heart.

Any network engineer in the room who never issued this command ?

Mythical "configure terminal" command
conf t
...

Article objective

In this article, we will present you freeRouter available configuration mode. This is an essential article as it will help you in your potential daily operation task. 

Diagram

[ #001 ] - "configure <mode>"

When you log into a network equipment such as a router, you tend to have some automatic reflex. You usually:

  • Check router configuration: show run or sh conf
  • Check ipv4 / ipv6 / or LFIB forwarding table
  • An then you want to configure the router

Let's assume you want to configure interface sdn3 description:

Mythical "configure terminal" command
r1#sh run sdn3                                                            
interface sdn3
 description r1@LAN3[05:00.0]
 mtu 1500
 macaddr 007b.0c15.1e0c
 shutdown
 no log-link-change
 exit

...

r1#conf t
r1(cfg)#                                                                  
r1(cfg)#int sdn3                                                          
r1(cfg-if)#                                                               
r1(cfg)#int sdn3                                                          
r1(cfg-if)#description Hello Workd SDN3                                
r1(cfg-if)#      

As you would notice, configuring these from "config terminal" prompt has an immediate effect. Please note you can issue "show" command from config mode using the "do" keyword :

issue "show" command in configuration mode using "do" keywork
r1#conf t
r1(cfg)#                                                                  
r1(cfg)#int sdn3                                                          
r1(cfg-if)#                                                               
r1(cfg)#int sdn3                                                          
r1(cfg-if)#description Hello Workd SDN3                       
r1(cfg-if)#   
r1(cfg-if)#do sh run sdn3                                                 
interface sdn3
 description Hello Workd SDN3
 mtu 1500
 macaddr 007b.0c15.1e0c
 shutdown
 no log-link-change
 exit
   

At that point you have a running-config in router memory and you have a startup-config written into the freeRouter flash. In order to see the difference:

issue "show" command in configuration mode
...

r1(cfg-if)#do sh config                                                   
interface sdn3
 no description old_descrption
 description Hello Workd SDN3
 exit
...
r1(cfg-if)# end                                                           
r1#show config-differences                                                
interface sdn3
 no description old_descrption
 description Hello Workd SDN3
 exit   

Notice the use of "end" primitive in order to end configuration mode and revert to user mode. In the example we used shortcut command name:

  • sh config
  • show config-differences

So basically this command will show you the difference between running-config and startup-config. This is similar to Junos: show | compare except that in this context this a comparison between running and startup config.

In this case it just delete the current description and replace it by the new one.

Once you are happy you can write the running-config into the startup-config:

issue "show" command in configuration mode
...
r1#wr                                                                     
% success
r1#sh conf                                                                

r1#           

You observe that show config-differences has no relevant output. running-config is aligned to startup-config !

Note

This is the most intuitive and recommended way to start learning freeRouter as from this interactive mode, you'll benefit from the contextual help that can be triggered by '?'. In this way you'll even be able to discover new freeRouter feature yourself ! This piece of software holds a tremendous amount of secret functionality. In the output below we just check which control plane can be activated ...

issue "show" command in configuration mode
...
r1(cfg)#router ?                                                          
  babel4     - babel routing protocol
  babel6     - babel routing protocol
  bgp4       - border gateway protocol
  bgp6       - border gateway protocol
  blackhole4 - blackhole collector
  blackhole6 - blackhole collector
  deaggr4    - deaggregate creator
  deaggr6    - deaggregate creator
  download4  - route download
  download6  - route download
  eigrp4     - enhanced interior gateway routing protocol
  eigrp6     - enhanced interior gateway routing protocol
  flowspec4  - flowspec to flowspec rewriter
  flowspec6  - flowspec to flowspec rewriter
  isis4      - intermediate system intermediate system
  isis6      - intermediate system intermediate system
  logger4    - route logger
  logger6    - route logger
  lsrp4      - link state routing protocol
  lsrp6      - link state routing protocol
  mobile4    - mobile route creator
  mobile6    - mobile route creator
  msdp4      - multicast source discovery protocol
  msdp6      - multicast source discovery protocol
  olsr4      - optimized link state routing protocol
  olsr6      - optimized link state routing protocol
  ospf4      - open shortest path first
  ospf6      - open shortest path first
  pvrp4      - path vector routing protocol
  pvrp6      - path vector routing protocol
  rip4       - routing information protocol
  rip6       - routing information protocol
  uni2flow4  - unicast to flowspec converter
  uni2flow6  - unicast to flowspec converter
  uni2multi4 - unicast to multicast converter
  uni2multi6 - unicast to multicast converter         
       

"configure viewer" is a very interesting mode as it gives you the possibility to review the router configuration from a  viewer inspired from "mcedit" (Norton Midnight Commander) 

configure viewer
r1#configure viewer
...

Then you'll be able to read your configuration from a READ-ONLY text buffer:

As a side note, you can benefit from online help by pressing <f1>

But what if I just want to view a specific object ? Let's find out how to check ONLY BGP configuration @ home:

route addition via freeRouter
r1#configure viewer bgp4
...

So in this case It'll just throw my IPv4 bgp config snippet onto the viewer buffer

Same if I want to only view all interface sdn<x> from the router config:

config viewer <regexp>
r1#configure viewer sdn
...

This is so cool, isn't ?

Note

In big TELCO Service Provider environment, most of the time you have Technical Project Manager that just need to perform some checks related to specific customer VPN deployment. So some times, I received some calls: "Can you please that from customer the HUB site prefix 1.2.3.0/24 is configured and advertised into BGP for customer ABC in VRF YXZ ?" With "configure viewer <object>", the TPM can just check it for himself without bothering you at all ! And this without the fear to alter router configuration by accident.

PS: For that you'll need to create a aaa security config with:

  • proper router aaa security policy with privilege level 1
  • with or without TACACS/RADIUS authentication / authorisation and accounting
  • and apply it to a specific OOBM SSH/telnet server in a specific VRF,

but this is not in the scope of the the present article and it will be the object of further articles.

In SP environment, you should not be surprised to see router configuration that has 100k lines or even more. In these environment, I've seen config with countless amount of VRF, NAT, DLSW, GRE, IPSEC tunnels, BGP peers ...  "config viewer" is a great tools when you want to verify a specific stanza on a per customer or object basis and in bonus without any risk the Provider Edge router configuration.

"configure viewer" gives you the possibility to view the config or some parts of the config in read-only mode. "configure editor" gives you simply the possibility to edit also the specific running-config config stanza.

route addition via freeRouter
r1#configure editor
...

Then you'll be able to edit your configuration from a READ-WRITE text buffer:

As a side note, you can benefit from online help by pressing <f1>

You can press Ctrl+q in order to exit the editor. As you did not change anything it will exit the editor.

But what if I just want to edit a specific object ? Let's find out how to check ONLY BGP configuration @ home:

config editor <regexp>
r1#configure editor bgp4
...

So in this case I'll just throw my IPv4 bgp config onto the editor buffer

In this buffer let's just create a description for BGP neighbor 172.23.215.177.

Now just press Ctrl-q (as per the online help accessible using <f1>). However, freeRouter detect the buffer changed has we added BGP description configuration. Therefore it will ask you if you want to save the buffer change into the running-config and apply it.

At that moment you'll be displayed a small recap of what has been applied. 

Even more cool no ? 

Warning

Even if "config editor" is seducing and seems more appealing especially for beginners. This is absolutely not the case. "configure editor" mode is meant for advanced users who knows freeRouter CLI by heart. Why, you might say ? Just try to edit a gigantic BGP configuration without any contextual help just by writing a textual file and you'll understand the risk behind using "config editor". Therefore it is no recommend to use it against complex control plane object.

Please take note that "config editor" alter the running-configuration directly when you saved the editor buffer !

Note

So what's the point of having this cool feature ? This feature is powerful when it comes to simple control plane object or big repetitive object. This is very practical to use this feature against: ACL / Prefix-List / Route Policy Object / Route Map etc.

  • ACL
  • prefix-list
  • route policy list
  • route-map

but nothing to prevent you to edit BGP stanza if you feel that your freeRouter-fu needs to be challenged (wink)

Same as "config editor", but instead of working against the running-config you are editing the startup-config. Which is more safe ... till the next reload (wink)

config startup
r1#configure startup
...

"configure reload" as its name implies is not about reloading a router whatsoever (smile)

config reload
r1#configure reload ?                                                     
  <url> - source url

r1#configure reload    
...

This command take a <url> as argument. Basically it will fetch router configuration from the specified <url> and load it into the startup-config. It is an equivalent to Cisco "copy <url> start". From that point:

  • it is up to the network operator to check the startup configuration
  • and issue a reload warm in order to restart the router and test that connectivity is resuming as expected
  • Check the running-config is aligned to startup-config


Warning

(repetition is not harmful) As said before "configure reload" does not reload the router. It just load the config from specified <url> into the startup-configuration. And this steps precedes a reload that has to be triggered manually by the operator after having checked the config.

Note

in day to day operation, startup-config is usually not altered directly. In TELCO SP environment, IIRC, I used it mainly to retrieve configuration from a CMDB server during 2 situations:

  • Router first time installation after basic configuration staging enabling minimum connectivity
  • Router hardware replacement

Note that in SP environment, as VPN owner we could handle a portfolio of customer (~10). Each customer could have ~ 2000 CPEs. You can see why "config reload" can be very handy.

"configure network" gives you the possibility to update/merge existing  running-config from config exposed from a web server.

route addition via freeRouter
r1#configure network ?                                                    
  <url> - source url

r1#configure network
...

This command take a <url> as argument. Basically it will fetch specified configuration from the specified <url> and merge it into the running-config. It is an equivalent to Cisco "copy <url> run". So, from that point:

Warning

  • only running-config is altered.
  • If not saved all changes will be lost in the next reload

Note

in day to day operation, In TELCO SP environment, "configure network" is very useful when you want to apply the same configuration stanza to several router at the same time.

Same as "configure network" gives you the possibility to replace running-config from config exposed from a web server.

route addition via freeRouter
r1#configure overwrite-network ?                                                    
  <url> - source url

r1#configure overwrite-network
...

This command take a <url> as argument. Basically it will fetch specified configuration from the specified <url> and replace the running-config. It is an equivalent to Cisco "copy <url> run". So, from that point:

Warning

  • only running-config is altered.
  • If not saved all changes will be lost in the next reload

Note

in day to day operation, In TELCO SP environment, "configure network" is very useful when you want to apply the same configuration stanza to several router at the same time from a clean slate state. (no merger)

"configure banner" is one of my favorite mode. It will display an editor allowing you to edit the banner of your router.

route addition via freeRouter
r1#configure banner                                                   
...

Press Ctrl-q and then y in order to save the banner.

Log in to you router again in order to check your new banner:


Note

in day to day operation, this banner can be written in configuration using banner encoded command

banner encoded
banner encoded ICAgX18gICAgICAgICAgICAgICBfX19fICAgICAgICAgICAgIF8NCiAgLyBffF8gX18gX19fICBfX198ICBfIFwgX19fICBfICAgX3wgfF8gX19fIF8gX18NCiB8IHxffCAnX18vIF8gXC8gXyBcIHxfKSAvIF8gXHwgfCB8IHwgX18vIF8gXCAnX198DQogfCAgX3wgfCB8ICBfXy8gIF9fLyAgXyA8IChfKSB8IHxffCB8IHx8ICBfXy8gfA0KIHxffCB8X3wgIFxfX198XF9fX3xffCBcX1xfX18vIFxfXyxffFxfX1xfX198X3wNCiAgXyBfXyBfX18gICBfX198IHwgX19fX18gIHwgfA0KIHwgJ19fLyBfIFwgLyBfX3wgfC8gLyBfX3wgfCB8DQogfCB8IHwgKF8pIHwgKF9ffCAgIDxcX18gXCB8X3wNCiB8X3wgIFxfX18vIFxfX198X3xcX1xfX18vIChfKQ0KDQo=

the command corresponds to the banner mentioned above.

"configure revert" revert the running-config to the startup config. For Junos fan it is equivalent to "rollback 0"

configure description
r1#sh run int sdn3                                                        
interface sdn3
 description r1@LAN3[05:00.0]
 mtu 1500
 macaddr 007b.0c15.1e0c
 shutdown
 no log-link-change
 exit
!
configure description
r1# conf t                                                                
r1(cfg)#int sdn3
r1(cfg-if)#description "This is the new description"

mjolnir(cfg-if)#do sh conf                                                     
interface sdn3
 no description r1@LAN3[05:00.0]
 description "This is the new description "
 exit
Let's diff between running and startup config
r1(cfg-if)#do sh conf                                                     
interface sdn3
 no description r1@LAN3[05:00.0]
 description "This is the new description "
 exit
sh run sdn3
mjolnir(cfg)#exit                                                              
mjolnir#sh run sdn3                                                            
interface sdn3
 description "This is the new description "
 mtu 1500
 macaddr 007b.0c15.1e0c
 shutdown
 no log-link-change
 exit
sh run sdn3
mjolnir#configure revert                                                       
     1: interface sdn3
     2:  no description "This is the new description "
     3:  description r1@LAN3[05:00.0]
     4:  exit

errors=0
sh run sdn3
mjolnir#sh run sdn3                                                            
interface sdn3
 description r1@LAN3[05:00.0]
 mtu 1500
 macaddr 007b.0c15.1e0c
 shutdown
 no log-link-change
 exit

Note

in day to day operation, In TELCO SP environment, "configure revert" should be used as "rollback 0" upon the running config. This means that you are about to abandon the current running config and re-apply the config that figures in the startup-config. In our case, it was changing a description, but in some case it can have more impact. (change route filtering, route advertising etc.)

"configure rollback" is very useful when you are in an operational  situation that needs "trial and error" approach, and sometimes the error can lead to loss of connectivity on the router itself... Who never experienced that ?

First of all we have a saying a French saying: "Il n'y a que ceux qui ne font rien qui ne font pas de bêtise". So don't feel guilty about that... I remembered having isolated some sites just by accident ...

In this situation "configure rollback" is a combination of "configure revert" and a loss of CLI TCP session. What does this practically means ?

Imagine you are configuring a redistribution between IS-IS and OSPF and that you forgot that the network have 2 connections. This redistribution without careful route filtering will result in a routing loop and it happens that you lose connectivity upon that configuration. (never ending routing advertisement loop, high cpu load etc...)

Upon losing TCP connection, in "configure rollback" freeRouter will automatically revert to its startup config.

You will therefore auto-magically get back connection before it was the route redistribution that caused the havoc.

How cool is that !

Note

In IOS, i used to use  "reload in <x>" command, in JunOS of course you have "commit confirm" and same goes for IOS-XR. So this airbag is not only unique to freeRouter, but IT IS THERE !

"configure file" gives you to the possibility to update/merge running configuration from a local file from the flash filesystem.

route addition via freeRouter
r1#configure file ?                                                       
  <file> - source file

r1#configure file
...

This command take a <file> as argument. Basically it will load specified configuration from the specified <file> and update/merge the running-config. It is an equivalent to Cisco "copy <flash:file> run". So, from that point:

show flash
mjolnir#show flash /rtr                                                        
date                 size     name
2009-12-31 23:00:00  18048    bundle.bin
2020-07-30 15:47:05  2477     c.sh
2009-12-31 23:00:00  22648    hdlcInt.bin
2020-08-26 07:35:35  2937     hwdet-all.sh
2020-07-31 13:31:28  203      hwdet-main.sh
2009-12-31 23:00:00  18616    mapInt.bin
2020-09-29 08:58:48  554856   mjolnir.log
2009-12-31 23:00:00  18088    modem.bin
2009-12-31 23:00:00  131432   p4dpdk.bin
2009-12-31 23:00:00  121896   p4emu.bin
2009-12-31 23:00:00  63144    p4pkt.bin
2009-12-31 23:00:00  18088    pcap2pcap.bin
2009-12-31 23:00:00  18608    pcapInt.bin
2009-12-31 23:00:00  18384    rawInt.bin
2020-09-28 11:54:12  598      rtr-hw.txt
2020-09-28 21:16:19  14607    rtr-sw.txt
2020-07-30 15:47:37  2022     rtr.err
2020-09-29 03:09:25  5587321  rtr.jar
2020-09-29 03:09:16  5585713  rtr.jar.bak
2020-09-29 03:09:26  24       rtr.rld
2020-09-23 03:06:12  529      rtr.scr
2020-09-23 03:06:11  483      rtr.scr.bak
2020-08-23 17:34:19  46       rtr.scr2
2020-08-23 17:34:18  0        rtr.scr2.bak
2020-09-23 03:06:11  542720   rtr.tar
2020-09-23 03:06:09  522240   rtr.tar.bak
2020-09-29 03:11:04  2330     rtr.ver
2020-09-29 03:11:03  3790694  rtr.zip
2020-09-29 03:10:57  3789659  rtr.zip.bak
2020-07-30 15:47:05  388      setup_dpdk.sh
2020-07-30 15:47:05  48       setup_route.sh
2020-07-30 15:47:05  2171     setup_veth.sh
2009-12-31 23:00:00  18048    stdLin.bin
2009-12-31 23:00:00  18440    tapInt.bin
2009-12-31 23:00:00  18224    ttyLin.bin
2009-12-31 23:00:00  18256    vlan.bin

"configure file" gives you to the possibility to replace running configuration from a local file from the flash filesystem.

route addition via freeRouter
r1#configure replace ?                                                       
  <file> - source file

r1#configure replace
...

This command take a <file> as argument. Basically it will load specified configuration from the specified <file> and replace the running-config. It is an equivalent to Cisco "copy <flash:file> run". So, from that point:

show flash
mjolnir#show flash /rtr                                                        
date                 size     name
2009-12-31 23:00:00  18048    bundle.bin
2020-07-30 15:47:05  2477     c.sh
2009-12-31 23:00:00  22648    hdlcInt.bin
2020-08-26 07:35:35  2937     hwdet-all.sh
2020-07-31 13:31:28  203      hwdet-main.sh
2009-12-31 23:00:00  18616    mapInt.bin
2020-09-29 08:58:48  554856   mjolnir.log
2009-12-31 23:00:00  18088    modem.bin
2009-12-31 23:00:00  131432   p4dpdk.bin
2009-12-31 23:00:00  121896   p4emu.bin
2009-12-31 23:00:00  63144    p4pkt.bin
2009-12-31 23:00:00  18088    pcap2pcap.bin
2009-12-31 23:00:00  18608    pcapInt.bin
2009-12-31 23:00:00  18384    rawInt.bin
2020-09-28 11:54:12  598      rtr-hw.txt
2020-09-28 21:16:19  14607    rtr-sw.txt
2020-07-30 15:47:37  2022     rtr.err
2020-09-29 03:09:25  5587321  rtr.jar
2020-09-29 03:09:16  5585713  rtr.jar.bak
2020-09-29 03:09:26  24       rtr.rld
2020-09-23 03:06:12  529      rtr.scr
2020-09-23 03:06:11  483      rtr.scr.bak
2020-08-23 17:34:19  46       rtr.scr2
2020-08-23 17:34:18  0        rtr.scr2.bak
2020-09-23 03:06:11  542720   rtr.tar
2020-09-23 03:06:09  522240   rtr.tar.bak
2020-09-29 03:11:04  2330     rtr.ver
2020-09-29 03:11:03  3790694  rtr.zip
2020-09-29 03:10:57  3789659  rtr.zip.bak
2020-07-30 15:47:05  388      setup_dpdk.sh
2020-07-30 15:47:05  48       setup_route.sh
2020-07-30 15:47:05  2171     setup_veth.sh
2009-12-31 23:00:00  18048    stdLin.bin
2009-12-31 23:00:00  18440    tapInt.bin
2009-12-31 23:00:00  18224    ttyLin.bin
2009-12-31 23:00:00  18256    vlan.bin

Discussion

Most of you will simply use the basic "conf t" mode, but keep in mind that depending on your context, all the other modes are proven to be very handy. The possibility to configure 1000 router with one single config file using "config network" is a savior. Having the possibility to trigger automatic definitive router staging using "conf reload" is tremendously useful when you have to deploy 10 routers a day. As said "config view" can give non operation staff to check if some configs are there or not ... "config editor" is very powerful when you want to edit a never ending access-list, but remember to avoid to use it for complex BGP config... You have been warned !

Conclusion

In this 1st article:

  • We presented freeRouter config mode
  • Most of these are useful in various different context

Final words

All these modes are not new. IOS, IOS-XR, IOX-XE, NW-OX, JUNOS have their own config mode that are somewhat similar. In any case freeRouter config mode implementation is meant to address  all needs from the network operators perspective. As you can observe, configure mode has an impressive list of mode. Feel free to try and use them according to your environment taste!

Last but not least, you can play with these different mode from this sandbox:

type "telnet dl.nop.hu" in a terminal and choose "1"
 telnet dl.nop.hu                                                                                                                                                 1 ↵
Trying 193.224.23.5...
Connected to dl.nop.hu.
Escape character is '^]'.
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXX XXXXX XXX    XXX     XXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX  XXXX XX XXXX XX XXXX XX XX XX XXXX XXXXX/~~~~~~\XXXXXX
XXXX X XXX XX XXXX XX XXXX XX XX XX XXXX XXXX| player |XXXXX
XXXX XX XX XX XXXX XX     XXX    XX XXXX XXXXX\______/XXXXXX
XXXX XXX X XX XXXX XX XXXXXXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX XXXX  XX XXXX XX XXXXXXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX XXXXX XXX    XXX XXX XXX XX XXX    XXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
welcome
line ready
menu lab:
...
type "ssh dl.nop.hu" in a terminal (any user/pass will do) and choose "1"
ssh dl.nop.hu -l random_user                                                                                                                                     1 ↵
Warning: Permanently added 'dl.nop.hu,193.224.23.5' (RSA) to the list of known hosts.
random_user@dl.nop.hu's password: 
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXX XXXXX XXX    XXX     XXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX  XXXX XX XXXX XX XXXX XX XX XX XXXX XXXXX/~~~~~~\XXXXXX
XXXX X XXX XX XXXX XX XXXX XX XX XX XXXX XXXX| player |XXXXX
XXXX XX XX XX XXXX XX     XXX    XX XXXX XXXXX\______/XXXXXX
XXXX XXX X XX XXXX XX XXXXXXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX XXXX  XX XXXX XX XXXXXXX XX XX XXXX XXXXXXXXXXXXXXXXXXX
XXXX XXXXX XXX    XXX XXX XXX XX XXX    XXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
welcome
line ready
menu lab:
# - reboot router1
$ - reboot router2
% - reboot router3
1 - connect to router1
2 - connect to router2
3 - connect to router3
^ - rebuild routers
l - connect to lg.nop.dn42
x - exit
choose:1 - attach vdc lab1 

yourname#                                                                      
yourname#configure ?                                                           
  <cr>
  banner            - edit the banner
  editor            - configure from editor
  file              - append to running configuration
  network           - append to running configuration
  overwrite-network - overwrite the running configuration
  reapply           - !!!EXPERiMENTAL!!! try to reapply current configuration
  reload            - overwrite the startup configuration
  replace           - overwrite the running configuration
  revert            - revert to startup configuration
  rollback          - configure within auto-revert session
  startup           - edit the startup configuration
  terminal          - configure from this terminal
  viewer            - view current configuration

yourname#configure                                      
...

In order to exit the sandbox session use the following escape sequence: Ctrl-c + Ctrl-x

Another method to access the sandbox, by click here, this will open a terminal webapp into your browser:







This is a special blog series called "RARE software architecture". As its name implies, it deals with topics related to RARE/freeRouter software design choice.

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

RARE project objective is to provide a routing platform proposing various solutions addressing multiple use cases in the R&E landscape. In the picture below you see in purple the different use cases:

As you can notice, each use case will run on different hardware that potentially can have different dataplanes. As we were starting from a clean slate environment without much choice, especially with P4 programmability - the first dataplane or P4 target considered was BMv2. BMv2 is an excellent way to learn P4, it is also the first target we use in order to program and validate new features. After 6 months of practising our "P4-fu" we developed:

  • a P4lang repository for ubuntu bionic and focal
  • a debian 10 repository
  • had our first RARE/FreeRouter prototype powered by a P4 BMv2 dataplane !

Our initial work, considering FreeRouter's Java nature, was to write a Java P4Runtime GRPC client that would be able to program the entries in the tables exposed by BMv2 via the P4Info file. However, this would have intimately tied FreeRouter code to P4Runtime gRPC code. Even if it's more natural to choose this solution, going in that direction implied that dataplanes other than BMv2 would be compliant to P4Runtime. It turns out that this is not the case. We then opted for a simple message API via a bi-directional raw UNIX socket. We will see what this means later in this blog.

Motivated by the successful experience with BMv2, we then decided to move forward and started to study TOFINO as a target. We were greedy and eager to apply our P4 code against multi-terabits traffic. After a few P4 program compilations, the first impression from my personal perspective was ... mind blowing ! INTEL/BAREFOOT TOFINO effectively opened the door to multi-terabits packet processing... Just to have at the tip of your finger the possibility to process traffic at these traffic levels was exciting !

As a side note, the journey was not without suffering and pain... (smile) We had to port our BMv2 code - and to port to TOFINO was not "Une lettre à la poste"... It is not that TOFINO programming is gratuitously painful. It is just that it is p4c-tofino's job to make sure that our packets are processed at silicon lighting speed. Imagine you are asked to  convey parcels by driving from Paris to Amsterdam with a car that has an infinitely sized trunk, with an infinite gas tank and no particular speed constraint along the road. And then you are asked to do the same trip, but with an actual real car that has a trunk with a fixed size and with a 50 litre gas tank, and of course you'll have to follow speed signs along the road.

In the first case, you would put as many parcels as you would like and you even won't bother looking at your gas tank level and maybe you'd set the speed to 200 Km/h. The second case forces you to carefully think about how many parcels you can put in your trunk, check to see if one completely full tank can be sufficient for the trip and of course, you would have to follow the speed signs.

If you allow me this comparison, this is where BMv2 and TOFINO programming differs.  

But, this pain was not in vain, it was for the greater good... You can't imagine the inherent joy when you see the TOFINO compiler displaying the DONE word ! For the veterans who can remember, it is the same feeling when you manage to compile your first program in the ADA language. The compiler is not so strict that compiling an ADA program is in itself a feat. No wonder why this language is used in Spatial rocket (Ariane).

Back to our dataplane interface story, even TOFINO and BMv2 share some roots, while BMv2 had P4Runtime as a northnound interface, INTEL/BAREFOOT pushed into TOFINO platform with P4_16 their gRPC interface counterpart: BfRuntime.

Our best bet paid off as FreeRouter message API was unchanged and without much effort we could add a new dataplane "wingman" to the FreeRouter control plane.

To recap:

  • For BMv2: Our interface yields P4Runtime RPC calls. This program is called: forwarder.py
  • For TOFINO: Our interface yields BfRuntime RPC calls. This program is called witout too much originality: bf_forwader.py

At that point we were starting to have a decent LSR/LER router for CORE and Aggregation use cases.

But we still had nothing at the EDGE/AGGREGATION layer in terms of a solution proposal, deploying P4 hardware might be way too expensive in specific contexts such as small R&E institutions like primary schools or small R&E labs. To that purpose, we started to study new targets such as VMWARE XDP and a very promising project: T4P4S ELTE. While we could not use XDP without a lot of P4 code rewriting and compromise, T4P4S ELTE was from our perpective very promising. But due to a compilation issue, we could not move forward.

FPGA was also a solution that we considered but had no access to any FPGA hardware that was P4 compliant.

As a result, we were a little bit bitter and started to read the DPDK library. And we started to play with DPDK examples... These examples were tremendously useful as it sparked some DPDK development into the RARE team. Csaba, the FreeRouter lead developer, step by step came up with this GENIUS idea: why don't we just use emulate P4 RARE P4 dataplane program ? We can still revert to using T4P4S ELTE when it will be ready ?

P4emu/P4dpdk was then born ! 

To conclude this short story, RARE/FreeRouter has now 3 completely different dataplanes: (in order of appearance)

  • BMv2
  • TOFINO
  • DPDK


Unique RARE/FreeRouter feature

However, please note that FreeRouter message API is common to the three dataplanes listed above. You'll see further how this structure make the solution: an open modular, interchangeable solution.

Article objective

In this article, let's present RARE/FreeRouter platform structure and focus on the interface(S) between FreeRouter control plane and various dataplane.

Diagram

[ #001 ] - Modular design

In this designs, FreeRouter is focusing on running control plane processes, such as routing protocols IGP(s), BGP(s). There are other control plane processes but let's just focus on these latter. At some point in time, all IGPs/EGP converge and will have to create an entry in a routing table. In case of IPv4 the entry will be created into an IPv4 forwarding table and similarly, an IPv6 route entry will be created into IPv6 forwarding table. From FreeRouter point of view these entry creation will be triggered by yielding one Java function twice that will generate these 2 API messages, one for IPv4 and the other one for IPv6.

Let's add an IPv4 route into freeRouter CLI

route addition via freeRouter
conf t
ipv4 route v1 1.2.3.0 255.255.255.0 4.4.4.4
...

Upon entering the ipv4 route and pressing <enter>, you'll see the following message appearing

message API: route4_add
...
rx: ['route4_add', '1.2.3.0/24', '13063', '4.4.4.4', '1', '\n']
...

Let's delete the route via FreeRouter CLI

route deletion via freeRouter
conf t
no ipv4 route v1 1.2.3.0 255.255.255.0 4.4.4.4
...
message API: route4_del
...
rx: ['route4_del', '1.2.3.0/24', '13063', '4.4.4.4', '1', '\n']
...

Important note

In short, the message API is simply a collection of message that would trigger an entry ADD/DELETE/MODIFY into the dataplane corresponding table.

The documentation of this message API will be documented and published soon, but for those who are curious and can't wait this documentation, you can read forwarder.py, bf_forwarder.py or p4dpdk.bin  source code

As said in the beginning of the article, freeRouter control plane would have to deal with dataplane of different nature. And we concluded in mentioning that for now, freeRouter has three dataplanes. Each of these dataplanes have their own northbound interface, whether this is P4Runtime for BMv2, BfRuntime for TOFINO or P4DPDK for system compatible with DPDK and having DPDK complinnt NIC.

For BMv2 we just had to write an interface that would translate freeRouter API message into P4Runtime GRPC calls. For BMv2 this interface is called forwarder.py:

For TOFINO we just had to write an interface that would translate freeRouter API message into BfRuntime GRPC calls. For TOFINO this interface is called bf_forwarder.py:

For DPDK we just had to write an interface that would translate freeRouter API message into DPDK primitives. This interface is included into DPDK dataplane bundled into freeRouter binaries: p4dpdk.bin

It is just as simple as that !

Discussion

This design is pretty unique because, if for any reason you would like to "hook" freeRouter control plane to an other dataplane such as:

  • FPGA
  • or dataplane powered by kernel bypass technique such as RDMA
  • Or other NPU based dataplane
  • etc.

This is possible !

You would "just" have to port your P4 code logic into the target dataplane and create an interface able to translate API messages from FreeRouter into understandable message from the target dataplane.

Be cautious with the word "just"

The "just" word can be misleading. Indeed, depending on the target dataplane, it can be a huge task. With DPDK, we were lucky in getting enough material in order to move forward and again p4dpdk.bin was a simple trial at the very beginning. But some other dataplane can just be simply be ignored if we don't get enough material/support from NPU vendors. 

One thing that we did not experience, but this can be maybe one day a reality.

What if you have your own control plane and that you absolutely want to keep it, but would like to re-use BMv2/TOFINO or DPDK RARE dataplane ?

Well this is possible !

Long time ago I met Thomas MANGIN (yet another cool and nice French guy (smile) ) which is the author of Exa-BGP, i did not talk to him about this and I don't want to give him bad idea, but what if he would like to hook a TOFINO P4 dataplane to Exa-BGP ?

Well, he actually would just have to teach exaBGP to handle entry ADD/DELETE/MODIFY message according to the message API above.

I also love the work DONE at the SoNIC project level and I know that SoNIC has already a P4 dataplane called switch.p4. I doubt it will be the case one day but, what if SoNIC project wanted to re-use RARE dataplane for especially for Service Provider capability ?

OK, this sounds crazy, but the modular design we proposed here is valid and can make the RARE dataplane available for other control plane.

Of course, we strongly suggest you to stick with FreeRouter as you will just realize IMHO that in the TELCO Service Provider space there is no match. You'll have the venerable IOS-XR and JUNOS, but these are not Open Source counterparts.


Conclusion

In this 1st article you:

  • had a 10K feet view description of RARE/FreeRouter modular design
  • This design allow rapid dataplane addtion without altering whatsoever FreeRouter code base
  • In case you would like to re-use BMv2/TOFINO/P4DPDK dataplane, this has been never implemented but this is possible !

Message API documentation

From the time being this API  message is not yet publicly documented. However, it is available and buried inside forwarder.py or bf_forwarder.py source code. This is work in progress but if you feel an urgent need to use it feel free to read the code.

PS: We will publish this document ASAP, but time plays against us ...