"RARE/FreeRouter-101" series of article are meant to help you quickly kickstart your RARE/freeRouter very first deployment and understand via a series of tutorial how it can be powered by various dataplane. 101 article series explained also how RARE/freeRouter could be configured in order to be integrated to the external network environment. However, even if 101- [ #006 ] is a robust and interesting solution for SOHO, you'll see in the "RARE validated design" series of articles,  a lot more interesting use case. This articles will draw your attention to mind blowing use cases that are usually implemented only by commercial solution in service provider environment.

Requirement

  • Basic Linux/Unix knowledge
  • Service provider networking knowledge

Overview

BGP is THE protocol of Internet, it is used to exchange routing information between other BGP systems between Internet domains. It comes in two flavours:

External BGP(eBGP): Network Layer Reachability Information (NLRI) is exchanged between network domain called Autonomous system usually administratively independant. We are speaking about BGP inter-domain routing. As an example, let's us assume a BGP speaker from AS2200 (RENATER) advertising NLRI information to AS20965 (GÉANT R&E). From that point AS20965 has the knowledge of how to reach any network advertised by AS2200 based on the NLRI information.

Internal BGP (iBGP): NLRI is propagated between BGP speakers inside the same domain. We are speaking about BGP intra-domain routing. As an example, assume border router AS2200 in Paris connected to GEANT network and get NLRI information from AS20965. I will then propagate this information internally and advertise GEANT NLRI information via iBGP session to other BGP speaker inside network domain for AS2200.

iBGP requires a full mesh network between all BGP speakers inside a domain because of an anti-AS loop avoidance. Thus requiring n*(n-1)/2 number of sessions to be implemented. BGP route reflection is a proposal that remove full mesh requirement. BGP Edge router has now only 1 BGP session toward the RR, thus reducing network equipment workload.

Article objective

In this article we will describe how to build a carrier grade route reflector cluster composed by RR1 and RR2. In order to reach Telecom Internet Service provider 99,999% of availability:

Let's consider the architecture network of a fictitious service provider below, router reflector RR1 and RR2 are dual homed to a core P routers.

Diagram

[ #001 ] - Cookbook


SR655 1 x EPYC 7302P, 64GB RAM, 2G CONTROLLER CACHE FLASH, 4x10G ports + SFP+ and 4x1G ports, 3 SSD 480GB MAINSTREAM, XCLARITY ENTERPRISE.

SR655 AMD EPYC 7302P (16C 2.8GHz 128MB Cache/155W) 32GB (2x32GB, 2Rx4 3200MHz RDIMM), No Backplane, SATA, 1x750W, Tooless Rails
ThinkSystem 2x32GB TruDDR4 3200MHz (2Rx4 1.2V) RDIMM-A
ThinkSystem SR655 2.5 SATA/SAS 8-Bay Backplane Kit
ThinkSystem RAID 930-8i 2GB Flash PCIe 12Gb Adapter
ThinkSystem 2.5 5300 480GB Mainstream SATA 6Gb Hot Swap SSD
ThinkSystem SR655 x16/x8/x8 PCIe Riser1 FH Kit
ThinkSystem SR635/SR655 x8 PCIe Internal Riser Kit
ThinkSystem Broadcom 57454 10/25GbE SFP28 4-port OCP Ethernet Adapter
ThinkSystem Broadcom 5720 1GbE RJ45 2-Port PCIe Ethernet Adapter
SFP+ SR Transceiver
ThinkSystem 750W(230/115V) Platinum Hot-Swap Power Supply
2.8m, 10A/100-250V, C13 to IEC 320-C14 Rack Power Cable
ThinkSystem Toolless Slide Rail Kit with 2U CMA
ThinkSystem SR655 Fan Option Kit
ThinkSystem SR635/SR655 Supercap Installation Kit


RR is a specific component inside a service provider environment:

  • The BGP RR is not in the data path inside the backbone, this can be adjusted by setting hight IGP metrics inside the code backbone. 
  • BGP traffic does not require a tremendous throughput so no need to have hardware NIC assisted forwarding mechanism such as dpdk.
  • A NREN route reflector with 2xIPv4 and 2xIPv6 full views coming from 2 upstream provider requires steady ~ 10 Mbps traffic rates, so we can assume that 10GE connection will be sufficient for the next decades all address-family included.
  • As of 2020/07/13, the Internet IPv4 routing table size is 839945 entries
  • As of 2020/07/13, the Internet IPv6 routing table size is 91062 entries

both cumulated with BGP other address families needs a constant usage of ~ 4GB of memory:

# show watchdog memory

  • So in the config above 64 Gbytes of RAM is sufficient in order to cache all the IPv4 and IPv6 routing table in memory (and also other BGP address family tables). It will be also largely enough in case of network instability, events that involves more CPU/memory usage related convergence computation.


  • We have no incentive in proposing a server with the above brand. It just happen that this server was already bought and its configuration is matching perfectly the use case requirement but again, this is pure coincidence
  • 10GE port connection might be overkill, but in a Service Provider context this is the norm. It will avoid adjacent core routers to implement 1GE connectivity
  • PCIe GEN4 is available, and thus provide a tremendous amount of bandwidth for disk R/W operation. Though useful for the OS application, BGP RR setup won't take a direct advantage from PCIE GEN4.
  • Indeed in this configuration considering the amount of RAM we have we will disable SWAP operations.




  • Connect the server with 2 NIC using optical  SFP ( Broadcom 57454 10/25GbE SFP28 4-port OCP Ethernet Adapter) to core backbone routers following distinct dark fiber path.
  • The link between C1 - C2 provides an additional level of redundancy


  • Connect the server with 1 NIC using RJ45 (Broadcom 5720 1GbE RJ45 2-Port PCIe Ethernet Adapter) to the KVM or Out fo band management network


One point overlooked is the environment. As said BGP is a central component in service provider network. It must be deployed considering the following recommendations:

  • Deploy an RR in carrier hotel
  • With sufficient cooling
  • With sufficient power. Make also sure to have redundant power and use dual PSU connected to different energy source
  • Rack properly the server and make sure it is installed without blocking airflow as per server vendor advice




  • Use only stable branch also called LTS operating system like Debian 10 or Ubuntu 18.04 and ubuntu 20.04
  • Apply your IT strip down security patch and make it enter your server maintenance process
  • In our case we will use Debian 10


It is important to note that now, BGP RR is subject to your company server hardware maintenance and that the software is not part of it.

  • Server hardware maintenance is now applied to a network equipment
  • The software is maintained by freeRouter project members




mkdir -p ~/freeRouter/bin ~/freeRouter/lib ~/freeRouter/etc ~/freeRouter/log
cd ~/freeRouter/lib
wget http://freerouter.nop.hu/rtr.jar


╭─[11:11:54]floui@debian ~ 
╰─➤ tree freeRouter
freeRouter
├── bin   # binary files      
├── etc   # configuration files      
├── lib   # library files      
└── log   # log files      





wget freerouter.nop.hu/rtr.tar



tar xvf rtr.tar -C ~/freeRouter/bin/


For those you would like to rebuild these binaries you can find the compilation shell script in freeRouter cloned git repository in: ~/freeRouter/src/native/c.sh



  • In this case simple pcapInt packet forwarding is recommended
  • In this setup all freeRouter functionalities are natively available
  • freeRouter heavily uses the concept of thread, hence 16 CPU cores will be fully exploited 


freeRouter upgrades involves 3 aspects:

  • It is pretty unusual, but as freeRouter is using Java, you have to follow Java software update recommandation 
  • freeRouter control plane software it self, it is essentiallaly a rtr.jar file that has to be replaced by the latest version
  • freeRouter dataplane software pcapInt upgrade. pcapInt upgrade are unusual but still has to be checked in freeRouter release notes

We are (at last) now ready to configure freeRouter as a BGP route reflector !


FreeRouter uses 2 configuration files in order to run, let's write these configuration files for R1 in ~/freeRouter/etc

int eth1 eth 0000.1111.0001 127.0.0.1 10011 127.0.0.1 10012
int eth2 eth 0000.2222.0002 127.0.0.1 10021 127.0.0.1 10022
tcp2vrf 2323 v1 23


  • eth1 is BGP port eth1, port 10011 is freeRouter port while 10012 is the port associated to pcapInt associated in linux interface in NIC #1 
  • eth2 is BGP port eth2,  port 10021 is freeRouter port while 10022 is the port associated to pcapInt associated in linux interface in NIC #2
  • For now freeRouter will be accessible only via telnet session on port 2323 


hostname rr1
buggy
!
!
access-list ACL-IPv4-RR-CLIENT
 sequence 10 permit all 1.1.1.1 255.255.255.255 all any all
 sequence 20 permit all 2.2.2.2 255.255.255.255 all any all
 sequence 30 permit all 3.3.3.3 255.255.255.255 all any all
 sequence 40 permit all 4.4.4.4 255.255.255.255 all any all
 sequence 50 permit all 5.5.5.5 255.255.255.255 all any all
 sequence 60 permit all 6.6.6.6 255.255.255.255 all any all
 sequence 70 permit all 7.7.7.7 255.255.255.255 all any all
 sequence 80 permit all 8.8.8.8 255.255.255.255 all any all
 exit
!
access-list ACL-IPv6-RR-CLIENT
 sequence 10 deny all fd00::a ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all any all
 sequence 20 deny all fd00::b ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff all any all
 sequence 30 permit all fd00:: ffff:: all any all
 exit
!
prefix-list PFX-IPv4-NHT
 sequence 10 permit 1.1.1.1/32 ge 32 le 32
 sequence 20 permit 2.2.2.2/32 ge 32 le 32
 sequence 30 permit 3.3.3.3/32 ge 32 le 32
 sequence 40 permit 4.4.4.4/32 ge 32 le 32
 sequence 50 permit 5.5.5.5/32 ge 32 le 32
 sequence 60 permit 6.6.6.6/32 ge 32 le 32
 sequence 70 permit 7.7.7.7/32 ge 32 le 32
 sequence 80 permit 8.8.8.8/32 ge 32 le 32
 sequence 100 permit 10.10.10.10/32 ge 32 le 32
 sequence 110 permit 11.11.11.11/32 ge 32 le 32
 exit
!
prefix-list PFX-IPv6-NHT
 sequence 10 permit fd00::/32 ge 128 le 128
 exit
!
route-policy NHT
 sequence 10 if distance 110
 sequence 20   pass
 sequence 30 else
 sequence 40   drop
 sequence 50 enif
 exit
!
vrf definition v1
 rd 1:1
 exit
!
router ospf4 1
 vrf v1
 router-id 4.4.4.10
 traffeng-id 0.0.0.0
 area 0 enable
 redistribute connected
 exit
!
router ospf6 1
 vrf v1
 router-id 6.6.6.10
 traffeng-id ::
 area 0 enable
 redistribute connected
 exit
!
interface loopback1
 no description
 vrf forwarding v1
 ipv4 address 10.10.10.10 255.255.255.255
 ipv6 address fd00::a ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff
 router ospf4 1 enable
 router ospf4 1 area 0
 router ospf4 1 passive
 router ospf6 1 enable
 router ospf6 1 area 0
 router ospf6 1 passive
 no shutdown
 no log-link-change
 exit
!
interface ethernet1
 no description
 vrf forwarding v1
 ipv4 address 10.1.10.10 255.255.255.0
 ipv6 address fd00:cafe::1:10:10 ffff:ffff:ffff:ffff:ffff:ffff:ffff::
 router ospf4 1 enable
 router ospf4 1 area 0
 router ospf4 1 cost 4444
 router ospf6 1 enable
 router ospf6 1 area 0
 router ospf6 1 cost 6666
 no shutdown
 no log-link-change
 exit
!
interface ethernet2
 no description
 vrf forwarding v1
 ipv4 address 10.4.10.10 255.255.255.0
 ipv6 address fd00:cafe::4:10:10 ffff:ffff:ffff:ffff:ffff:ffff:ffff::
 router ospf4 1 enable
 router ospf4 1 area 0
 router ospf4 1 cost 4444
 router ospf6 1 enable
 router ospf6 1 area 0
 router ospf6 1 cost 6666
 no shutdown
 no log-link-change
 exit
!
router bgp4 65535
 vrf v1
 local-as 65535
 router-id 10.10.10.10
 address-family unicast multicast other flowspec vpnuni vpnmlt vpnflw ovpnuni ovpnmlt ovpnflw vpls mspw evpn mdt srte mvpn omvpn
 nexthop route-policy NHT
 nexthop prefix-list PFX-IPv4-NHT
 template bgp4 remote-as 65535
 template bgp4 description rr clients
 template bgp4 local-as 65535
 template bgp4 address-family unicast multicast other flowspec vpnuni vpnmlt vpnflw ovpnuni ovpnmlt ovpnflw vpls mspw evpn mdt srte mvpn omvpn
 template bgp4 distance 255
 template bgp4 connection-mode active
 template bgp4 compression both
 template bgp4 update-source loopback1
 template bgp4 hostname
 template bgp4 aigp
 template bgp4 traffeng
 template bgp4 pmsitun
 template bgp4 tunenc
 template bgp4 attribset
 template bgp4 segrout
 template bgp4 bier
 template bgp4 route-reflector-client
 template bgp4 next-hop-unchanged
 template bgp4 send-community all
 listen ACL-IPv4-RR-CLIENT bgp4
 exit
!
router bgp6 65535
 vrf v1
 local-as 65535
 router-id 10.10.10.10
 address-family unicast multicast other flowspec vpnuni vpnmlt vpnflw ovpnuni ovpnmlt ovpnflw vpls mspw evpn mdt srte mvpn omvpn
 nexthop route-policy NHT
 nexthop prefix-list PFX-IPv6-NHT
 template bgp6 remote-as 65535
 template bgp6 description rr clients
 template bgp6 local-as 65535
 template bgp6 address-family unicast multicast other flowspec vpnuni vpnmlt vpnflw ovpnuni ovpnmlt ovpnflw vpls mspw evpn mdt srte mvpn omvpn
 template bgp6 distance 255
 template bgp6 connection-mode active
 template bgp6 compression both
 template bgp6 update-source loopback1
 template bgp6 hostname
 template bgp6 aigp
 template bgp6 traffeng
 template bgp6 pmsitun
 template bgp6 tunenc
 template bgp6 attribset
 template bgp6 segrout
 template bgp6 bier
 template bgp6 route-reflector-client
 template bgp6 next-hop-unchanged
 template bgp6 send-community all
 listen ACL-IPv6-RR-CLIENT bgp6
 exit
!
!
!
!
!
!
!
!
!
!
!
!
!
!
server telnet tel
 security protocol telnet
 no exec authorization
 no login authentication
 vrf v1
 exit
!
!
end





╭─[6:06:13]floui@debian ~/freeRouter  
╰─➤  java -jar lib/rtr.jar routersc etc/rr1-hw.txt etc/rr1-sw.txt                                                                                      
info cfg.cfgInit.doInit:cfgInit.java:556 booting
info cfg.cfgInit.doInit:cfgInit.java:680 initializing hardware
info cfg.cfgInit.doInit:cfgInit.java:687 applying defaults
info cfg.cfgInit.doInit:cfgInit.java:695 applying configuration
info cfg.cfgInit.doInit:cfgInit.java:721 done
welcome
line ready
rr1#                   


╭─[6:06:13]floui@debian[1]  ~/freeRouter/bin  
╰─➤  sudo ./pcapInt.bin enp0s9 10012 127.0.0.1 10011 127.0.0.1                                                                                                       
binded to local port 127.0.0.1 10012.
will send to 127.0.0.1 10011.
pcap version: libpcap version 1.8.1
opening interface enp0s9 with pcap1.x api
serving others
> 


╭─[6:06:13]floui@debian[1]  ~/freeRouter/bin  
╰─➤  sudo ./pcapInt.bin enp0s10 10022 127.0.0.1 10021 127.0.0.1                                                                                                      
binded to local port 127.0.0.1 10022.
will send to 127.0.0.1 10021.
pcap version: libpcap version 1.8.1
opening interface enp0s10 with pcap1.x api
serving others
>


Verification


╭─[1:09:28]floui@debian ~  
╰─➤  telnet localhost 10010
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
welcome
line ready
rr1#                   




rr1# sh ipv4 route v1                                                          
typ  prefix          metric    iface      hop        time
O    1.1.1.1/32      110/4444  ethernet1  10.1.10.1  00:05:05
O    2.2.2.2/32      110/4445  ethernet1  10.1.10.1  00:04:50
O    3.3.3.3/32      110/4445  ethernet2  10.4.10.4  00:04:32
O    4.4.4.4/32      110/4444  ethernet2  10.4.10.4  00:04:18
O    5.5.5.5/32      110/4445  ethernet1  10.1.10.1  00:04:00
O    6.6.6.6/32      110/4445  ethernet1  10.1.10.1  00:03:42
O    7.7.7.7/32      110/4446  ethernet1  10.1.10.1  00:03:28
O    8.8.8.8/32      110/4445  ethernet2  10.4.10.4  00:02:59
O    10.1.2.0/24     110/4444  ethernet1  10.1.10.1  00:22:47
O    10.1.4.0/24     110/4444  ethernet2  10.4.10.4  00:22:47
O    10.1.5.0/24     110/4444  ethernet1  10.1.10.1  00:22:47
O    10.1.6.0/24     110/4444  ethernet1  10.1.10.1  00:22:47
C    10.1.10.0/24    0/0       ethernet1  null       00:22:49
LOC  10.1.10.10/32   0/1       ethernet1  null       00:22:49
O    10.2.3.0/24     110/4445  ethernet2  10.4.10.4  00:22:35
O    10.2.6.0/24     110/4445  ethernet1  10.1.10.1  00:22:47
O    10.2.7.0/24     110/4445  ethernet1  10.1.10.1  00:22:38
O    10.2.11.0/24    110/4445  ethernet1  10.1.10.1  00:22:38
O    10.3.4.0/24     110/4444  ethernet2  10.4.10.4  00:22:47
O    10.3.7.0/24     110/4445  ethernet2  10.4.10.4  00:22:35
O    10.3.8.0/24     110/4445  ethernet2  10.4.10.4  00:22:32
O    10.3.11.0/24    110/4445  ethernet2  10.4.10.4  00:22:35
O    10.4.5.0/24     110/4444  ethernet2  10.4.10.4  00:22:47
O    10.4.8.0/24     110/4444  ethernet2  10.4.10.4  00:22:47
C    10.4.10.0/24    0/0       ethernet2  null       00:22:49
LOC  10.4.10.10/32   0/1       ethernet2  null       00:22:49
C    10.10.10.10/32  0/0       loopback1  null       00:22:49
O    11.11.11.11/32  110/8889  ethernet1  10.1.10.1  00:06:43

rr1# sh ipv4 ospf 1 topo 0                                                     
node      reach  via        ifc        met   hop  conn  sr  br  neighbors
4.4.4.1   true   10.1.10.1  ethernet1  4444  1    5     0   0   4.4.4.2=1=10.1.2.1 4.4.4.4=1=10.1.4.1 4.4.4.5=1=10.1.5.1 4.4.4.6=1=10.1.6.1 4.4.4.10=4444=10.1.10.1
4.4.4.2   true   10.1.10.1  ethernet1  4445  2    5     0   0   4.4.4.1=1=10.1.2.2 4.4.4.3=1=10.2.3.2 4.4.4.7=1=10.2.7.2 4.4.4.6=1=10.2.6.2 4.4.4.11=4444=10.2.11.2
4.4.4.3   true   10.4.10.4  ethernet2  4445  2    4     0   0   4.4.4.2=1=10.2.3.3 4.4.4.4=1=10.3.4.3 4.4.4.8=1=10.3.8.3 4.4.4.7=1=10.3.7.3
4.4.4.4   true   10.4.10.4  ethernet2  4444  1    5     0   0   4.4.4.3=1=10.3.4.4 4.4.4.8=1=10.4.8.4 4.4.4.5=1=10.4.5.4 4.4.4.1=1=10.1.4.4 4.4.4.10=4444=10.4.10.4
4.4.4.5   true   10.1.10.1  ethernet1  4445  2    2     0   0   4.4.4.1=1=10.1.5.5 4.4.4.4=1=10.4.5.5
4.4.4.6   true   10.1.10.1  ethernet1  4445  2    2     0   0   4.4.4.1=1=10.1.6.6 4.4.4.2=1=10.2.6.6
4.4.4.7   true   10.1.10.1  ethernet1  4446  3    2     0   0   4.4.4.2=1=10.2.7.7 4.4.4.3=1=10.3.7.7
4.4.4.8   true   10.4.10.4  ethernet2  4445  2    2     0   0   4.4.4.3=1=10.3.8.8 4.4.4.4=1=10.4.8.8
4.4.4.10  true   null       null       0     0    2     0   0   4.4.4.1=4444=10.1.10.10 4.4.4.4=4444=10.4.10.10
4.4.4.11  true   10.1.10.1  ethernet1  8889  3    1     0   0   4.4.4.2=4444=10.2.11.11

rr1# sh ipv6 route v1                                                          
typ  prefix                  metric     iface      hop                time
O    fd00::1/128             110/6666   ethernet1  fd00:cafe::1:10:1  00:06:01
O    fd00::2/128             110/6667   ethernet1  fd00:cafe::1:10:1  00:05:46
O    fd00::3/128             110/6667   ethernet2  fd00:cafe::4:10:4  00:05:28
O    fd00::4/128             110/6666   ethernet2  fd00:cafe::4:10:4  00:05:14
O    fd00::5/128             110/6667   ethernet1  fd00:cafe::1:10:1  00:04:56
O    fd00::6/128             110/6667   ethernet1  fd00:cafe::1:10:1  00:04:38
O    fd00::7/128             110/6668   ethernet1  fd00:cafe::1:10:1  00:04:24
O    fd00::8/128             110/6667   ethernet2  fd00:cafe::4:10:4  00:03:56
C    fd00::a/128             0/0        loopback1  null               00:23:45
O    fd00::b/128             110/13333  ethernet1  fd00:cafe::1:10:1  00:07:40
O    fd00:cafe::1:2:0/112    110/6666   ethernet1  fd00:cafe::1:10:1  00:23:43
O    fd00:cafe::1:4:0/112    110/6666   ethernet2  fd00:cafe::4:10:4  00:23:43
O    fd00:cafe::1:5:0/112    110/6666   ethernet1  fd00:cafe::1:10:1  00:23:43
O    fd00:cafe::1:6:0/112    110/6666   ethernet1  fd00:cafe::1:10:1  00:23:43
C    fd00:cafe::1:10:0/112   0/0        ethernet1  null               00:23:45
LOC  fd00:cafe::1:10:10/128  0/1        ethernet1  null               00:23:45
O    fd00:cafe::2:3:0/112    110/6667   ethernet1  fd00:cafe::1:10:1  00:23:32
O    fd00:cafe::2:6:0/112    110/6667   ethernet1  fd00:cafe::1:10:1  00:23:32
O    fd00:cafe::2:7:0/112    110/6667   ethernet1  fd00:cafe::1:10:1  00:23:32
O    fd00:cafe::2:11:0/112   110/6667   ethernet1  fd00:cafe::1:10:1  00:23:32
O    fd00:cafe::3:4:0/112    110/6666   ethernet2  fd00:cafe::4:10:4  00:23:43
O    fd00:cafe::3:7:0/112    110/6667   ethernet2  fd00:cafe::4:10:4  00:23:32
O    fd00:cafe::3:8:0/112    110/6667   ethernet2  fd00:cafe::4:10:4  00:23:32
O    fd00:cafe::3:11:0/112   110/6667   ethernet2  fd00:cafe::4:10:4  00:23:32
O    fd00:cafe::4:5:0/112    110/6666   ethernet2  fd00:cafe::4:10:4  00:23:43
O    fd00:cafe::4:8:0/112    110/6666   ethernet2  fd00:cafe::4:10:4  00:23:43
C    fd00:cafe::4:10:0/112   0/0        ethernet2  null               00:23:45
LOC  fd00:cafe::4:10:10/128  0/1        ethernet2  null               00:23:45

rr1# sh ipv6 ospf 1 topo 0                                                     
node               reach  via                ifc        met    hop  conn  sr  br  neighbors
6.6.6.1/00000000   true   fd00:cafe::1:10:1  ethernet1  6666   1    5     0   0   6.6.6.2/00000000=1=10012 6.6.6.4/00000000=1=10015 6.6.6.5/00000000=1=10012 6.6.6.6/00000000=1=10012 6.6.6.10/00000000=6666=10012
6.6.6.2/00000000   true   fd00:cafe::1:10:1  ethernet1  6667   2    5     0   0   6.6.6.1/00000000=1=10012 6.6.6.3/00000000=1=10012 6.6.6.7/00000000=1=10012 6.6.6.6/00000000=1=10013 6.6.6.11/00000000=6666=10012
6.6.6.3/00000000   true   fd00:cafe::4:10:4  ethernet2  6667   2    4     0   0   6.6.6.2/00000000=1=10013 6.6.6.4/00000000=1=10012 6.6.6.8/00000000=1=10012 6.6.6.7/00000000=1=10013
6.6.6.4/00000000   true   fd00:cafe::4:10:4  ethernet2  6666   1    5     0   0   6.6.6.3/00000000=1=10013 6.6.6.8/00000000=1=10013 6.6.6.5/00000000=1=10013 6.6.6.1/00000000=1=10013 6.6.6.10/00000000=6666=10013
6.6.6.5/00000000   true   fd00:cafe::1:10:1  ethernet1  6667   2    2     0   0   6.6.6.1/00000000=1=10014 6.6.6.4/00000000=1=10014
6.6.6.6/00000000   true   fd00:cafe::1:10:1  ethernet1  6667   2    2     0   0   6.6.6.1/00000000=1=10015 6.6.6.2/00000000=1=10015
6.6.6.7/00000000   true   fd00:cafe::1:10:1  ethernet1  6668   3    2     0   0   6.6.6.2/00000000=1=10014 6.6.6.3/00000000=1=10015
6.6.6.8/00000000   true   fd00:cafe::4:10:4  ethernet2  6667   2    2     0   0   6.6.6.3/00000000=1=10014 6.6.6.4/00000000=1=10013
6.6.6.10/00000000  true   null               null       0      0    2     0   0   6.6.6.1/00000000=6666=10016 6.6.6.4/00000000=6666=10016
6.6.6.11/00000000  true   fd00:cafe::1:10:1  ethernet1  13333  3    1     0   0   6.6.6.2/00000000=6666=10016





c5#sh ipv4 route v1                                                            
typ  prefix          metric    iface      hop       time
O    1.1.1.1/32      110/1     ethernet1  10.1.5.1  00:07:22
O    2.2.2.2/32      110/2     ethernet1  10.1.5.1  00:07:07
O    3.3.3.3/32      110/2     ethernet2  10.4.5.4  00:06:49
O    4.4.4.4/32      110/1     ethernet2  10.4.5.4  00:06:35
C    5.5.5.5/32      0/0       loopback1  null      00:25:07
O    6.6.6.6/32      110/2     ethernet1  10.1.5.1  00:06:00
O    7.7.7.7/32      110/3     ethernet1  10.1.5.1  00:05:46
O    8.8.8.8/32      110/2     ethernet2  10.4.5.4  00:05:17
O    10.1.2.0/24     110/1     ethernet1  10.1.5.1  00:25:06
O    10.1.4.0/24     110/1     ethernet2  10.4.5.4  00:25:05
C    10.1.5.0/24     0/0       ethernet1  null      00:25:07
LOC  10.1.5.5/32     0/1       ethernet1  null      00:25:07
O    10.1.6.0/24     110/1     ethernet1  10.1.5.1  00:25:06
O    10.1.10.0/24    110/1     ethernet1  10.1.5.1  00:25:06
O    10.2.3.0/24     110/2     ethernet2  10.4.5.4  00:24:53
O    10.2.6.0/24     110/2     ethernet1  10.1.5.1  00:25:05
O    10.2.7.0/24     110/2     ethernet1  10.1.5.1  00:24:56
O    10.2.11.0/24    110/2     ethernet1  10.1.5.1  00:24:56
O    10.3.4.0/24     110/1     ethernet2  10.4.5.4  00:25:05
O    10.3.7.0/24     110/2     ethernet2  10.4.5.4  00:24:53
O    10.3.8.0/24     110/2     ethernet2  10.4.5.4  00:24:50
O    10.3.11.0/24    110/2     ethernet2  10.4.5.4  00:24:53
C    10.4.5.0/24     0/0       ethernet2  null      00:25:07
LOC  10.4.5.5/32     0/1       ethernet2  null      00:25:07
O    10.4.8.0/24     110/1     ethernet2  10.4.5.4  00:25:05
O    10.4.10.0/24    110/1     ethernet2  10.4.5.4  00:25:05
O    10.10.10.10/32  110/4445  ethernet1  10.1.5.1  00:11:05
O    11.11.11.11/32  110/4446  ethernet1  10.1.5.1  00:09:01

c5#sh ipv4 ospf 1 topo 0                                                       
node      reach  via       ifc        met   hop  conn  sr  br  neighbors
4.4.4.1   true   10.1.5.1  ethernet1  1     1    5     0   0   4.4.4.2=1=10.1.2.1 4.4.4.4=1=10.1.4.1 4.4.4.5=1=10.1.5.1 4.4.4.6=1=10.1.6.1 4.4.4.10=4444=10.1.10.1
4.4.4.2   true   10.1.5.1  ethernet1  2     2    5     0   0   4.4.4.1=1=10.1.2.2 4.4.4.3=1=10.2.3.2 4.4.4.7=1=10.2.7.2 4.4.4.6=1=10.2.6.2 4.4.4.11=4444=10.2.11.2
4.4.4.3   true   10.4.5.4  ethernet2  2     2    4     0   0   4.4.4.2=1=10.2.3.3 4.4.4.4=1=10.3.4.3 4.4.4.8=1=10.3.8.3 4.4.4.7=1=10.3.7.3
4.4.4.4   true   10.4.5.4  ethernet2  1     1    5     0   0   4.4.4.3=1=10.3.4.4 4.4.4.8=1=10.4.8.4 4.4.4.5=1=10.4.5.4 4.4.4.1=1=10.1.4.4 4.4.4.10=4444=10.4.10.4
4.4.4.5   true   null      null       0     0    2     0   0   4.4.4.1=1=10.1.5.5 4.4.4.4=1=10.4.5.5
4.4.4.6   true   10.1.5.1  ethernet1  2     2    2     0   0   4.4.4.1=1=10.1.6.6 4.4.4.2=1=10.2.6.6
4.4.4.7   true   10.1.5.1  ethernet1  3     3    2     0   0   4.4.4.2=1=10.2.7.7 4.4.4.3=1=10.3.7.7
4.4.4.8   true   10.4.5.4  ethernet2  2     2    2     0   0   4.4.4.3=1=10.3.8.8 4.4.4.4=1=10.4.8.8
4.4.4.10  true   10.1.5.1  ethernet1  4445  2    2     0   0   4.4.4.1=4444=10.1.10.10 4.4.4.4=4444=10.4.10.10
4.4.4.11  true   10.1.5.1  ethernet1  4446  3    1     0   0   4.4.4.2=4444=10.2.11.11

c5#sh ipv6 route v1                                                            
typ  prefix                 metric    iface      hop               time
O    fd00::1/128            110/1     ethernet1  fd00:cafe::1:5:1  00:08:06
O    fd00::2/128            110/2     ethernet1  fd00:cafe::1:5:1  00:07:51
O    fd00::3/128            110/2     ethernet2  fd00:cafe::4:5:4  00:07:33
O    fd00::4/128            110/1     ethernet2  fd00:cafe::4:5:4  00:07:19
C    fd00::5/128            0/0       loopback1  null              00:25:51
O    fd00::6/128            110/2     ethernet1  fd00:cafe::1:5:1  00:06:43
O    fd00::7/128            110/3     ethernet1  fd00:cafe::1:5:1  00:06:29
O    fd00::8/128            110/2     ethernet2  fd00:cafe::4:5:4  00:06:01
O    fd00::a/128            110/6667  ethernet1  fd00:cafe::1:5:1  00:11:45
O    fd00::b/128            110/6668  ethernet1  fd00:cafe::1:5:1  00:09:45
O    fd00:cafe::1:2:0/112   110/1     ethernet1  fd00:cafe::1:5:1  00:25:49
O    fd00:cafe::1:4:0/112   110/1     ethernet2  fd00:cafe::4:5:4  00:25:49
C    fd00:cafe::1:5:0/112   0/0       ethernet1  null              00:25:51
LOC  fd00:cafe::1:5:5/128   0/1       ethernet1  null              00:25:51
O    fd00:cafe::1:6:0/112   110/1     ethernet1  fd00:cafe::1:5:1  00:25:49
O    fd00:cafe::1:10:0/112  110/1     ethernet1  fd00:cafe::1:5:1  00:25:49
O    fd00:cafe::2:3:0/112   110/2     ethernet1  fd00:cafe::1:5:1  00:25:37
O    fd00:cafe::2:6:0/112   110/2     ethernet1  fd00:cafe::1:5:1  00:25:37
O    fd00:cafe::2:7:0/112   110/2     ethernet1  fd00:cafe::1:5:1  00:25:37
O    fd00:cafe::2:11:0/112  110/2     ethernet1  fd00:cafe::1:5:1  00:25:37
O    fd00:cafe::3:4:0/112   110/1     ethernet2  fd00:cafe::4:5:4  00:25:49
O    fd00:cafe::3:7:0/112   110/2     ethernet2  fd00:cafe::4:5:4  00:25:37
O    fd00:cafe::3:8:0/112   110/2     ethernet2  fd00:cafe::4:5:4  00:25:37
O    fd00:cafe::3:11:0/112  110/2     ethernet2  fd00:cafe::4:5:4  00:25:37
C    fd00:cafe::4:5:0/112   0/0       ethernet2  null              00:25:51
LOC  fd00:cafe::4:5:5/128   0/1       ethernet2  null              00:25:51
O    fd00:cafe::4:8:0/112   110/1     ethernet2  fd00:cafe::4:5:4  00:25:49
O    fd00:cafe::4:10:0/112  110/1     ethernet2  fd00:cafe::4:5:4  00:25:49

c5#sh ipv6 ospf 1 topo 0                                                       
node               reach  via               ifc        met   hop  conn  sr  br  neighbors
6.6.6.1/00000000   true   fd00:cafe::1:5:1  ethernet1  1     1    5     0   0   6.6.6.2/00000000=1=10012 6.6.6.4/00000000=1=10015 6.6.6.5/00000000=1=10012 6.6.6.6/00000000=1=10012 6.6.6.10/00000000=6666=10012
6.6.6.2/00000000   true   fd00:cafe::1:5:1  ethernet1  2     2    5     0   0   6.6.6.1/00000000=1=10012 6.6.6.3/00000000=1=10012 6.6.6.7/00000000=1=10012 6.6.6.6/00000000=1=10013 6.6.6.11/00000000=6666=10012
6.6.6.3/00000000   true   fd00:cafe::4:5:4  ethernet2  2     2    4     0   0   6.6.6.2/00000000=1=10013 6.6.6.4/00000000=1=10012 6.6.6.8/00000000=1=10012 6.6.6.7/00000000=1=10013
6.6.6.4/00000000   true   fd00:cafe::4:5:4  ethernet2  1     1    5     0   0   6.6.6.3/00000000=1=10013 6.6.6.8/00000000=1=10013 6.6.6.5/00000000=1=10013 6.6.6.1/00000000=1=10013 6.6.6.10/00000000=6666=10013
6.6.6.5/00000000   true   null              null       0     0    2     0   0   6.6.6.1/00000000=1=10014 6.6.6.4/00000000=1=10014
6.6.6.6/00000000   true   fd00:cafe::1:5:1  ethernet1  2     2    2     0   0   6.6.6.1/00000000=1=10015 6.6.6.2/00000000=1=10015
6.6.6.7/00000000   true   fd00:cafe::1:5:1  ethernet1  3     3    2     0   0   6.6.6.2/00000000=1=10014 6.6.6.3/00000000=1=10015
6.6.6.8/00000000   true   fd00:cafe::4:5:4  ethernet2  2     2    2     0   0   6.6.6.3/00000000=1=10014 6.6.6.4/00000000=1=10013
6.6.6.10/00000000  true   fd00:cafe::1:5:1  ethernet1  6667  2    2     0   0   6.6.6.1/00000000=6666=10016 6.6.6.4/00000000=6666=10016
6.6.6.11/00000000  true   fd00:cafe::1:5:1  ethernet1  6668  3    1     0   0   6.6.6.2/00000000=6666=10016


c5#ping 10.10.10.10 /vrf v1                                                    
pinging 10.10.10.10, src=null, vrf=v1, cnt=5, len=64, tim=1000, ttl=255, tos=0, sweep=false
!!!!!
result=100%, recv/sent/lost=5/5/0, rtt min/avg/max/total=0/0/1/4
c5#ping fd00::a /vrf v1                                                        
pinging fd00::a, src=null, vrf=v1, cnt=5, len=64, tim=1000, ttl=255, tos=0, sweep=false
!!!!!
result=100%, recv/sent/lost=5/5/0, rtt min/avg/max/total=0/0/1/4
c5#                                                                                                                                                 




rr1#sh ipv4 bgp 65535 sum                                                      
as     learn  done  ready  neighbor  uptime
65535  0      0     true   1.1.1.1   16:22:28
65535  0      0     true   2.2.2.2   16:17:26
65535  0      0     true   3.3.3.3   16:16:44
65535  0      0     true   4.4.4.4   16:16:01
65535  0      0     true   5.5.5.5   16:15:32
65535  0      0     true   6.6.6.6   16:14:56
65535  0      0     true   7.7.7.7   16:14:30
65535  0      0     true   8.8.8.8   16:13:37

rr1#sh ipv6 bgp 65535 sum                                                      
as     learn  done  ready  neighbor  uptime
65535  0      0     true   fd00::1   16:20:41
65535  0      0     true   fd00::2   16:18:27
65535  0      0     true   fd00::3   16:17:32
65535  0      0     true   fd00::4   16:16:59
65535  0      0     true   fd00::5   16:16:22
65535  0      0     true   fd00::6   16:15:57
65535  0      0     true   fd00::7   16:15:15
65535  0      0     true   fd00::8   16:14:45




rr1#show ipv4 bgp 65535 neighbor 1.1.1.1 status                                
peer = 1.1.1.1
reachable state = true
reachable changed = 16:24:12
reachable changes = 1
fallover = null
update group = 0
type = routeReflectorClient
safi =  unicast multicast other flowspec vpnuni vpnmlt vpnflw ovpnuni ovpnmlt ovpnflw vpls mspw evpn mdt srte mvpn omvpn
local = 10.10.10.10
router id = 1.1.1.1
uptime = 16:24:12
hold time = 00:03:00
keepalive time = 00:01:00
32bit as = true
refresh = true, rx=0, tx=0
description = rr clients
hostname = null
compression = rx=true, tx=false
graceful = 
addpath rx = 
addpath tx = 
unicast advertised = 0 of 0, list = 0, accepted = 0 of 0
multicast advertised = 0 of 0, list = 0, accepted = 0 of 0
other advertised = 0 of 0, list = 0, accepted = 0 of 0
flowspec advertised = 0 of 0, list = 0, accepted = 0 of 0
vpnuni advertised = 0 of 0, list = 0, accepted = 0 of 0
vpnmlt advertised = 0 of 0, list = 0, accepted = 0 of 0
vpnflw advertised = 0 of 0, list = 0, accepted = 0 of 0
ovpnuni advertised = 0 of 0, list = 0, accepted = 0 of 0
ovpnmlt advertised = 0 of 0, list = 0, accepted = 0 of 0
ovpnflw advertised = 0 of 0, list = 0, accepted = 0 of 0
vpls advertised = 0 of 0, list = 0, accepted = 0 of 0
mspw advertised = 0 of 0, list = 0, accepted = 0 of 0
evpn advertised = 0 of 0, list = 0, accepted = 0 of 0
mdt advertised = 0 of 0, list = 0, accepted = 0 of 0
srte advertised = 0 of 0, list = 0, accepted = 0 of 0
mvpn advertised = 0 of 0, list = 0, accepted = 0 of 0
omvpn advertised = 0 of 0, list = 0, accepted = 0 of 0
version = 14 of 14, needfull=0, buffull=0
full = 9, 2020-07-27 16:32:29, 16:15:21 ago, 0 ms
incr = 2, 2020-07-28 08:13:10, 00:34:40 ago, 0 ms
connection = tx=173(987) rx=158(986) drp=0(0)
uncompressed = tx=0(0) rx=0(0) drp=0(0)
buffer = max=65536 rx=0 tx=65536

rr1#show ipv6 bgp 65535 neighbor fd00::1 status                                
peer = fd00::1
reachable state = true
reachable changed = 16:22:33
reachable changes = 1
fallover = null
update group = 0
type = routeReflectorClient
safi =  unicast multicast other flowspec vpnuni vpnmlt vpnflw ovpnuni ovpnmlt ovpnflw vpls mspw evpn mdt srte mvpn omvpn
local = fd00::a
router id = 1.1.1.1
uptime = 16:22:33
hold time = 00:03:00
keepalive time = 00:01:00
32bit as = true
refresh = true, rx=0, tx=0
description = rr clients
hostname = null
compression = rx=true, tx=false
graceful = 
addpath rx = 
addpath tx = 
unicast advertised = 0 of 0, list = 0, accepted = 0 of 0
multicast advertised = 0 of 0, list = 0, accepted = 0 of 0
other advertised = 0 of 0, list = 0, accepted = 0 of 0
flowspec advertised = 0 of 0, list = 0, accepted = 0 of 0
vpnuni advertised = 0 of 0, list = 0, accepted = 0 of 0
vpnmlt advertised = 0 of 0, list = 0, accepted = 0 of 0
vpnflw advertised = 0 of 0, list = 0, accepted = 0 of 0
ovpnuni advertised = 0 of 0, list = 0, accepted = 0 of 0
ovpnmlt advertised = 0 of 0, list = 0, accepted = 0 of 0
ovpnflw advertised = 0 of 0, list = 0, accepted = 0 of 0
vpls advertised = 0 of 0, list = 0, accepted = 0 of 0
mspw advertised = 0 of 0, list = 0, accepted = 0 of 0
evpn advertised = 0 of 0, list = 0, accepted = 0 of 0
mdt advertised = 0 of 0, list = 0, accepted = 0 of 0
srte advertised = 0 of 0, list = 0, accepted = 0 of 0
mvpn advertised = 0 of 0, list = 0, accepted = 0 of 0
omvpn advertised = 0 of 0, list = 0, accepted = 0 of 0
version = 14 of 14, needfull=0, buffull=0
full = 9, 2020-07-27 16:32:15, 16:16:37 ago, 0 ms
incr = 2, 2020-07-28 08:13:15, 00:35:38 ago, 0 ms
connection = tx=173(985) rx=158(984) drp=0(0)
uncompressed = tx=0(0) rx=0(0) drp=0(0)
buffer = max=65536 rx=0 tx=65536


Conclusion

In this article you:

  • BGP Router Reflector use case does not require a commercial vendor router, it can be handled perfectly by a sowftare solution running on a server with enoough RAM.

The example above an example of a high availability Route Reflector that is able to handle BGP signalling for a high carrier Service Provider for all address familay

  • Redundant BGP Router Reflection is ensured by deploying 2 RR (at minimum) belonging to the same BGP RR cluster 

In addition to have several RR for the whole domain, it is also common to see hierarchical RR design. SOme Service provider deploy dedicated RR for specific address family (L3VPN unicast for example)

  • RR in the same cluster run basic iBGP session

These RR also share the same cluster ID, in order to ensure route withdraw in case of routing advertisement

  • RR should not be in the traffic datapath

This is the reason why we are setting high cost (4444 and 6666) for IPv4 and IPv6 respectively on both direction on the RR(s) interconnections ports

  • RR design for a multi-service backbone

In the example, the RR client are running only IPv4/IPv6 but the RR design above can empower a Service provider backbone with additional service running on TOP of MPLS, L3VPN, 6VPE, VPLS EVPN etc.

  • In the next article we will dissect the rr1 configurations

This will demonstrate some nice features proposed by freeRouter such as BGP template and nexthop tracking among a list of other feature not mentioned here... (like BGP add-path)


You can test this design above in order to check RR and backbone router signalling.

  • Set up freeRouter environment as describe above
  • Get RARE code
 git clone https://github.com/frederic-loui/RARE.git


cd RARE/00-unit-labs/0101-rare-validated-design-bgp/
make


c1: telnet localhost 10001 
c2: telnet localhost 10002 
c3: telnet localhost 10003 
c4: telnet localhost 10004 
c4: telnet localhost 10005 
c6: telnet localhost 10006 
c7: telnet localhost 10007 
c8: telnet localhost 10008 
rr1: telnet localhost 10010 
rr2: telnet localhost 10011 


cd RARE/00-unit-labs/0101-rare-validated-design-bgp/
make clean