Requirement
|
|
Overview
Several choices were possible, we finally ended up in following the KISS method. The Operating system requirements are:
- requirement #0: LTS operating system
- requirement #1: Benefit from LTS security patches
- requirement #2: Must be able to run DPDK
- requirement #3: (personal requirement) Must be familiar to me
- requirement #4: Able to run Java software as freeRouter is written in Java
- requirement #5: Small operating system software footprint
- requirement #6: Support for IPv4/IPv6
The hardest path would be:
- Create a custom linux distribution using the Yocto project:
The objective is to have tight control of the software installed on the appliance. This guarantees the smallest footprint we hope to obtain. For those familiar with OpenWRT, we can reach a tiny image size. My OpenWRT image is 5Mb.
- Use of NixOS or Nix package manager
This provides an incredible feature: commit/rollback functionality at the package management level!
Note
The features above are still under study into RARE group. We will introduce these technologies once we feel more confident on how to integrate these technologies into a streamlined deployment process.
Article objective
In this article we will go through the major steps in deploying Debian 10 stable aka Buster in order to prepare freeRouter installation.
Diagrams
[ #002 ] - Cookbook
wget http://ftp.nl.debian.org/debian/dists/buster/main/installer-amd64/current/images/
On MACOSX, burn the iso using balenaEtcher
balenaEtcher can be downloaded here
Via the appliance BIOS settings:
- activate console port redirection:
- configure serial port settings
Now that you have activated console port:
- plug the USB key on which you previously burnt Debian 10
- make sure you set boot option from USB in BIOS settings
- reboot
You can now proceed to the next step: Debian 10 installation
We will assume that you have installed Debian 10 on the 256 Gb SSD.
Just as a side note during the installation process you'll be prompted the: "Software selection" window, in this steps we will:
- unselect everything
- select "SSH server"
This will guarantee the tiniest Debian 10 operating system software footprint. We will on demand install the needed packages manually.
cat /lib/systemd/system/rtr.service [Unit] Description=router processes Wants=network.target After=network-pre.target Before=network.target [Service] Type=forking ExecStart=/rtr/hwdet-all.sh [Install] WantedBy=multi-user.target
cat /rtr/hwdet-all.sh #!/bin/sh cd /rtr echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6 echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6 echo 0 > /proc/sys/net/ipv6/conf/lo/disable_ipv6 ip link set lo up mtu 65535 ip addr add 127.0.0.1/8 dev lo ip addr add ::1/128 dev lo # DPDK echo 96 > /proc/sys/vm/nr_hugepages modprobe uio_pci_generic dpdk-devbind.py -b uio_pci_generic 01:00.0 dpdk-devbind.py -b uio_pci_generic 02:00.0 dpdk-devbind.py -b uio_pci_generic 05:00.0 dpdk-devbind.py -b uio_pci_generic 06:00.0 dpdk-devbind.py -b uio_pci_generic 07:00.0 dpdk-devbind.py -b uio_pci_generic 08:00.0 #VETH for CPU_PORT and OOBM_PORT ip link add veth0a type veth peer name veth0b ip link set veth0a multicast on ip link set veth0a allmulti on ip link set veth0a promisc on ip link set veth0a mtu 8192 ip link set veth0a up ip link set veth0b multicast on ip link set veth0b allmulti on ip link set veth0b promisc on ip link set veth0b mtu 8192 ip link set veth0b up ethtool -K veth0a rx off ethtool -K veth0a tx off ethtool -K veth0a sg off ethtool -K veth0a tso off ethtool -K veth0a ufo off ethtool -K veth0a gso off ethtool -K veth0a gro off ethtool -K veth0a lro off ethtool -K veth0a rxvlan off ethtool -K veth0a txvlan off ethtool -K veth0a ntuple off ethtool -K veth0a rxhash off ethtool --set-eee veth0a eee off ethtool -K veth0b rx off ethtool -K veth0b tx off ethtool -K veth0b sg off ethtool -K veth0b tso off ethtool -K veth0b ufo off ethtool -K veth0b gso off ethtool -K veth0b gro off ethtool -K veth0b lro off ethtool -K veth0b rxvlan off ethtool -K veth0b txvlan off ethtool -K veth0b ntuple off ethtool -K veth0b rxhash off ethtool --set-eee veth0b eee off ip link add veth1a type veth peer name veth1b ip link set veth1a multicast on ip link set veth1a allmulti on ip link set veth1a promisc on ip link set veth1a mtu 1500 ip link set veth1a up ip link set veth1b multicast on ip link set veth1b allmulti on ip link set veth1b promisc on ip link set veth1b mtu 8192 ip link set veth1b up ip link set wlan0 up ethtool -K veth1a rx off ethtool -K veth1a tx off ethtool -K veth1a sg off ethtool -K veth1a tso off ethtool -K veth1a ufo off ethtool -K veth1a gso off ethtool -K veth1a gro off ethtool -K veth1a lro off ethtool -K veth1a rxvlan off ethtool -K veth1a txvlan off ethtool -K veth1a ntuple off ethtool -K veth1a rxhash off ethtool --set-eee veth1a eee off ethtool -K veth1b rx off ethtool -K veth1b tx off ethtool -K veth1b sg off ethtool -K veth1b tso off ethtool -K veth1b ufo off ethtool -K veth1b gso off ethtool -K veth1b gro off ethtool -K veth1b lro off ethtool -K veth1b rxvlan off ethtool -K veth1b txvlan off ethtool -K veth1b ntuple off ethtool -K veth1b rxhash off ethtool --set-eee veth1b eee off ip addr flush dev veth1a ip addr add 192.168.128.254/24 dev veth1a #ADD DEFAULT ROUTE to OOBM SDN999 route add default gw 192.168.128.1 # START RTR ! start-stop-daemon -S -b -x /rtr/hwdet-main.sh
chmod u+x /rtr/hwdet-main.sh
A bit of explanation
echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6 echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6 echo 0 > /proc/sys/net/ipv6/conf/lo/disable_ipv6 ip link set lo up mtu 65535
Note that IPv6 operation would occur on the host itself, IPv6 will be handled at freeRouter level
echo 96 > /proc/sys/vm/nr_hugepages modprobe uio_pci_generic dpdk-devbind.py -b uio_pci_generic 01:00.0 dpdk-devbind.py -b uio_pci_generic 02:00.0 dpdk-devbind.py -b uio_pci_generic 05:00.0 dpdk-devbind.py -b uio_pci_generic 06:00.0 dpdk-devbind.py -b uio_pci_generic 07:00.0 dpdk-devbind.py -b uio_pci_generic 08:00.0
In the stanza above, we configure DPDK (required)
- Configure HugePages
In this case we use 96 hugepages, this value can be different if you are using a box with different characteristics (# of ports, memory etc.) The objective is to configure a value that is not too high (waste of resources) and not too small. otherwise p4dpdk won't run. In this case this leaves 10 Free HugePages.
grep HugePages_ /proc/meminfo HugePages_Total: 96 HugePages_Free: 10 HugePages_Rsvd: 0 HugePages_Surp: 0
- Activate UIO_PCI_GENERIC driver
- Bind the interfaces to DPDK, DPDK will control them now. Keep in mind that now they will be invisible from the linux kernel.
This command use device PCI ID. In order to check device PCI ID just issue the below command:
dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:01:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb 0000:02:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb 0000:05:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb 0000:06:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb 0000:07:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb 0000:08:00.0 'I211 Gigabit Network Connection 1539' drv=uio_pci_generic unused=igb Network devices using kernel driver =================================== 0000:09:00.0 'AR928X Wireless Network Adapter (PCI-Express) 002a' if=wlan0 drv=ath9k unused=uio_pci_generic No 'Baseband' devices detected ============================== Other Crypto devices ==================== 0000:00:1a.0 'Atom Processor Z36xxx/Z37xxx Series Trusted Execution Engine 0f18' unused=uio_pci_generic No 'Eventdev' devices detected ============================== No 'Mempool' devices detected ============================= No 'Compress' devices detected ============================== No 'Misc (rawdev)' devices detected ===================================
- Configure the appliance OOBM via veth pair (as all physical ports are handled by DPDK and will be invisible from the Linux kernel)
#VETH for CPU_PORT and OOBM_PORT ip link add veth0a type veth peer name veth0b ip link set veth0a multicast on ip link set veth0a allmulti on ip link set veth0a promisc on ip link set veth0a mtu 8192 ip link set veth0a up ip link set veth0b multicast on ip link set veth0b allmulti on ip link set veth0b promisc on ip link set veth0b mtu 8192 ip link set veth0b up ethtool -K veth0a rx off ethtool -K veth0a tx off ethtool -K veth0a sg off ethtool -K veth0a tso off ethtool -K veth0a ufo off ethtool -K veth0a gso off ethtool -K veth0a gro off ethtool -K veth0a lro off ethtool -K veth0a rxvlan off ethtool -K veth0a txvlan off ethtool -K veth0a ntuple off ethtool -K veth0a rxhash off ethtool --set-eee veth0a eee off ethtool -K veth0b rx off ethtool -K veth0b tx off ethtool -K veth0b sg off ethtool -K veth0b tso off ethtool -K veth0b ufo off ethtool -K veth0b gso off ethtool -K veth0b gro off ethtool -K veth0b lro off ethtool -K veth0b rxvlan off ethtool -K veth0b txvlan off ethtool -K veth0b ntuple off ethtool -K veth0b rxhash off ethtool --set-eee veth0b eee off
So the above section is pretty straightforward:
- It creates veth0a / veth0b pair. For those familiar with P4, this is similar to the channel between the control plane (freeRouter) and p4dpdk (dataplane) using CPU_PORT
- It sets for veth0a/veth0b: multicast/allmulti/promisc flag + mtu=8192
- It disables TCP offload for veth0a/veth0b
We do the same thing for the wireless network
ip link add veth1a type veth peer name veth1b ip link set veth1a multicast on ip link set veth1a allmulti on ip link set veth1a promisc on ip link set veth1a mtu 1500 ip link set veth1a up ip link set veth1b multicast on ip link set veth1b allmulti on ip link set veth1b promisc on ip link set veth1b mtu 8192 ip link set veth1b up ip link set wlan0 up ethtool -K veth1a rx off ethtool -K veth1a tx off ethtool -K veth1a sg off ethtool -K veth1a tso off ethtool -K veth1a ufo off ethtool -K veth1a gso off ethtool -K veth1a gro off ethtool -K veth1a lro off ethtool -K veth1a rxvlan off ethtool -K veth1a txvlan off ethtool -K veth1a ntuple off ethtool -K veth1a rxhash off ethtool --set-eee veth1a eee off ethtool -K veth1b rx off ethtool -K veth1b tx off ethtool -K veth1b sg off ethtool -K veth1b tso off ethtool -K veth1b ufo off ethtool -K veth1b gso off ethtool -K veth1b gro off ethtool -K veth1b lro off ethtool -K veth1b rxvlan off ethtool -K veth1b txvlan off ethtool -K veth1b ntuple off ethtool -K veth1b rxhash off ethtool --set-eee veth1b eee off ip addr flush dev veth1a ip addr add 192.168.128.254/24 dev veth1a
Add default route to SDN999 for OOBM return traffic (192.168.128.1 is freeRouter sdn999: we will see the full config later)
route add default gw 192.168.128.1
Effectively start freeRouter main loop
start-stop-daemon -S -b -x /rtr/hwdet-main.sh
This main loop is triggered by the script hwdet-main.sh below:
cat /rtr/hwdet-main.sh
#!/bin/sh
while (true); do
cd /rtr/
stty raw < /dev/tty
java -Xmx4g -jar /rtr/rtr.jar router /rtr/rtr-
if [ $? -eq 4 ] ; then
sync
reboot -f
fi
stty cooked < /dev/tty
sleep 1
done
A bit of explanation
Requirement considerations:
- The box should run 24x7
- It must survive a power cut, i.e the service should be restored each time the power is cut for any reasons
- If no power cut but freeRouter has crashed for any reason, it should be restarted
Let me re-assure you, freeRouter usually don't crash, most often freeRouter has manual or better: auto-upgrades 
while (true); do ... done
- The appliance has 8Gb RAM which is enough for JVM running freeRouter. (Full routing IPv4/IPv6 at the control plane is possible at home! ← ok this is useless but cool, no? :3 )
- RAM allocation is for JVM and its tables
- Additional RAM allocation is for p4dpdk and p4emu, as we have to store the table once for the native code too
- Lastly the kernel also needs memory, so it's a good idea to leave some free RAM and not give everything to JVM.
java -Xmx4g -jar /rtr/rtr.jar router /rtr/rtr-
- freeRouter "Cold reboot"
if [ $? -eq 4 ] ; then sync reboot -f fi
Discussion
Conclusion
In this article, we got our hands dirty and manually installed freeRouter with DPDK dataplane from a clean slate environment. This is done on purpose, as I'd like you to understand the whole installation process in detail. There is an automated installation alternative that will install freeRouter also. However this is will install freeRouter with software backend. If your hardware CPU+NIC is compatible you can just replace the software backend by DPDK backend. At that precise point we have a vanilla genuine installation of freeRouter with DPDK dataplane on an appliance that can survive physical wild environment and power cut. We have just now to create the 2 freeRouter configuration files:
ls -l rtr-* -rw-r--r-- 1 root root 646 Jul 31 17:03 rtr-hw.txt -rw-r--r-- 1 root root 9027 Aug 25 10:02 rtr-sw.txt
RARE validated design: [ SOHO #002 ] - key take-away
- freeRouter installation is not complex. It just boils down to installing a basic supported Linux OS, install Java, some 3rd party software and the freeRouter jar and binaries itself
- In the binary list you'll have a special one called p4dpdk that corresponds to freeRouter DPDK dataplane that emulate RARE P4 program on BMv2 (It does not emulate BMv2 !)
- Though this installation is manual for pedagogic purpose, the installation can be fully automated, just fire up a VM with a bunch of interfaces and test it !
- The installation proposed is highly resilient and will ease upgrade of the appliance (we will see in subsequent article what it means
)
In the next article, we will configure the freeRouter appliance, start the router, and provide configuration in order to have effective basic ping reachability to the FTTH BROADBAND internal IP.