Though P4Emu/pcap can be used for SOHO and can handle nx1GE of traffic, this comes at a high CPU load cost and thus a higher power consumption.
"Why write yet another software dataplane as freeRouter has already a working native software dataplane ?"
The partial answer to the question raised in the previous article was:
"decoupling control plane from the dataplane"
We learned that P4Emu:
- is able to understand the VERY same strict control message from freeRouter as it occurs with a P4 dataplane
- is able to switch packet emulating router.p4 using libpcap packet forwarding backend.
However, even though libpcap is a performant packet processing library, the kernel is still heavily sollicited and the higher the traffic rate is, the higher CPU workload becomes.
Source Wikipedia: https://en.wikipedia.org/wiki/Data_Plane_Development_Kit
The Data Plane Development Kit (DPDK) is an Open source software project managed by the Linux Foundation. It provides a set of data plane libraries and network interface controller polling-mode drivers for offloading TCP packet processing from the operating system kernel to processes running in user space. This offloading achieves higher computing efficiency and higher packet throughput than is possible using the interrupt-driven processing provided in the kernel.
It is important to note that though its name implies, P4Emu/dpdk is not emulating V1Model. P4Emu is emulating router.p4 packet processing logic and uses a packet forwarding library to effectively transmit packets at specific ingress port to the right egress port defined by freeRouter control plane message. However, in this precise case, packet processing is offloaded from the kernel to user space. The consequence is the ability with dpdk compatible NIC and driver, to reach tremendous traffic rate. DPDK is not available on all hardware, please refer to DPDK HCL.
[ #006 ] - Cookbook
In our example we will use the ubuntu focal as we need dpdk 19.11.1 (latest current version is 20.05.0)
and we add a bridge network interface to or laptop RJ45 connection.
Add out of band management enp0s8 with Virtualbox
For those you would like to rebuild these binaries you can find the compilation shell script in freeRouter cloned git repository in: ~/freeRouter/src/native/c.sh
FreeRouter uses 2 configuration files in order to run, let's write these configuration files for R1 in ~/freeRouter/etc
Let's spend some times on this hardware configuration file, as you might have notice there are additional interesting lines worth to mention:
- proc <process-name>
It is possible within freeRouter startup to launch processes. We use here this feature to start control plane / dataplane communication via veth pair: veth0a and veth0b and also P4Emu/dpdk, p4dpdk.bin packet processing backend.
- proc p4emu /root/freertr/bin/p4dpdk.bin --vdev=net_af_packet0,iface=veth0b 127.0.0.1 9080 1
In dpdk, by default dpdk interfaces have port_ids that are sequentially allocated and in the order of appearance in dpdk-devbind --status output usually sorted by pci_id. In the previous output interface enp0s3 has port_id #0 and in dpdk veth0b (CPU_PORT has alwasy the last port_id beside dpdk data port_id, so here it is 1. If for exaplem we dedicate enp0s3, enp0s8, enp0s9, enp0s10 in virtualbox the command would have been:
proc p4emu /root/freertr/bin/p4dpdk.bin --vdev=net_af_packet0,iface=veth0b 127.0.0.1 9080 4
enp0s3 would be: #0 with pci_id: 00:03.0
enp0s8 would be: #1 with pci_id: 00:08.0
enp0s9 would be: #2 with pci_id: 00:09.0
enp0s10 would be: #3 with pci_id: 00:0a.0
Please observe the 1st ICMP packet loss that triggered ARP learning for respectively 192.168.0.254 and 192.168.0.62.
Please observe the 1st ICMP packet loss that triggered IPv6 neighbor discovery for respectively 2a01:e0a:159:2850::1 and 2a01:e0a:159:2850:e23f:49ff:fe6d:1899
In this article you:
- had a demonstration of how to integrate freeRouter into a local area network (Similar to article #002)
- However instead of using P4Emu/dpdk we used a P4Emu/dpdk dataplane
- communication between freeRouter control plane and P4Emu/dpdk is ensured by pcapInt via veth pair [ veth0a - veth0b ]
- In this example the freeRouter with P4Emu/dpdk has only 1 dataplane interface that is bound to enp0s3 VM interface exposed to the local network as a bridged interface
[ #006 ] RARE/FreeRouter-101 - key take-away
- FreeRouter is using UNIX socket in order to forward packet dedicated to control plane + dataplane communication.
This essential paradigm is used to ensure communication between freeRouter and P4Emu/dpdk dataplane. It is ensured by pcapInt binary from freeRouter net-tools that will bind freeRouter socket (veth0a@locathost:22001) to a virtual network interface (veth0b@localhost:22002) connected to CPU_PORT 1.
- freeRouter is the control plane for P4Emu/dpdk dataplane
freeRouter is doing all the control plane route computation and write/modify/remove message entry P4 entries are created/modified/removed accordingly from P4Emu/dpdk tables. Although the name is P4Emu, it does not emulate BMv2 V1Model.p4, but rather router.p4
- dpdk port_id allocation
dpkg port_id allocation follow pci_id port naming convention starting from id 0. p4dpdk.bin is invoked with the parameter: (number_of_dpdk_port - 1) + 1 <--- CPU_PORT
- In this setup the combination of freeRouter/P4Emu/dpdk delivers a solution for small campus network having 10GE links (100GE links to be validated)
dpkg removed the kernel intervention calls for each packet processed. In that configuration packet processing is now off loaded to user space. Reducing kernel intervention to ~ 0%. Congratulation you have a hardware NIC assisted forwarding is system !
In subsequent article we will see how this setup behaves with a DELL 640 server powered by Intel(R) Xeon(R) Gold 6138 CPU x 2 and equipped with a Mellanox ConnectX-5 EX Dual Port 100GbE QSFP28 PCIe Adapter Low Profile card. We will also see how to connect this server to a P4 switch, BF2556X-1T. So stay tuned !