In the previous article we enabled and checked IPv4 connectivity between all potential host within 192.168.128.0/17 and the outside Networks beyond ISP box. But, this is pretty useless as I can't imagine my kids typing IPv6 address (2001:8b0:0:30::666:102) in the browser in order to play a FUN puzzle. (Though for now we are suppose to have only IPv4 ) So we definitely need to provide name service resolution at the SOHO router itself.
In this article we will pursue the SOHO network appliance installation and enable name service to all host @ home.
[ SOHO #005 ] - "Got your Id number, but ... What's your name ?"
First step, it is need to configure the router as a client name for an existing DNS server.
So this declare our SOHO router as DNS client for 18.104.22.168 as primary DNS server and 22.214.171.124 as backup DNS server.
This step is mandatory as it will bind traffic originated from SOHO router to a specific VRF (here: inet). So this can be also qualified as "VRF proxy-awareness". In this way all DNS traffic originated from the router will be bound to VRF inet. This is done in 2 steps. The first step is to create the proxy-profile and bind it to the main VRF inet. The second step is to declare the SOHO router as client of this proxy-profile service.
Step -3-, configure DNS cache / server
- enable recursion (recursive query toward other DNS defined 126.96.36.199, 188.8.131.52)
- bind it to a specific interface (so SOHO router will answer only DNS from this interface)
- bind it to VRF inet
So this declare our SOHO router as DNS client for 184.108.40.206 as primary DNS server and 220.127.116.11 as backup DNS server
Step -4-, configure DNS and DHCP to propagate default dummy zone local
- Use local if you don't plan to propagate a domain name
- create local as dummy zone
When -1- and -2- are realised the router can resolve name
This can be verified only using a host connected to SOHO router. Let's assume a laptop connected behind sdn6.
As said IPv6 verification are just FYI, as we are supposed to have deployed only IPv4 till now. The point to show off IPv6 verification is to verify DNS AAAA request are working properly.
In this article DNS service has been enabled at:
- SOHO router level
- All host getting an IPv4 via DHCP will get a DNS server set to SOHO@loopback0 (192.168.254.1)
RARE validated design: [ SOHO #005 ] - key take-away
In this example the key take-away are:
- proxy-profile usage in order to proxy DNS query into VRF inet
- proxy-profile can be used to proxy other types of traffic
- data/routed traffic is not affected by proxy-profile
In the previous article we enabled and checked IPv4 connectivity between RARE/freeRouter and ISP box using sdn1 interface within 192.168.0.0/24 network. But as stated in the previous post, I'd like:
- all people connected within 192.168.128.0/17
- to access the external world
In this article we will pursue the SOHO network appliance installation and enable IPv4 connectivity for all host connected within your internal network to the external world.
[ #004 ] - Do you need translation ?
First step, let's create an interface Loopback0 within 192.168.128.0/17, let's say 192.168.254.1/32
First step, let's try to ping 18.104.22.168
Hey ! It seems that it's not working as expected ! At the present time, it seems that no one is able to reach outside world. Let's try to figure out why.
- It is a common best practice to dedicate a loopback to a router in a VRF
- Loopback are mostly used to identify a service proposed by the router
- Each services are bound to this Loopback in a specific VRF
This confirms the ping failures we observed previously. The output above indicate the packet does not even egress our SOHO router.
What is the inet VRF says ?
So we have no default routes . Let's configure one then pointing towards ISP BOX gateway:
So at that point, packet send to 22.214.171.124 are sent to nexthop 192.168.0.254 via sdn1.
But ping is still not not working. Let's figure out what's going on here.
As depicted in previous article:
- ISP box has a demarcation point set to 192.168.0.254
- So ISP box at some point is configured to perform Network Address Translation from 192.168.0.0/24 → ISP public IPv4 interface
- When ISP box receives a ICMP ping from 192.168.254.1 which does not match any ISP box NAT rules → Packet is discarded
Therefore in order to have a working seamless networking environment from the ISP box point of view, traffic coming from 192.168.128.0/17 might need to be NAT(ed) into 192.168.0.0/24 network. Let's see If our guess is right.
Let's configure IPv4 Network Address Translation specifically for Loopback0
In the config stanza above we configure an access-list called ACL-NAT4 in VRF inet that would translate all incoming packet matching 192.168.254.1 going to anywhere to interface sdn1 IPv4. But we also don't NAT for inter-vlan traffic within 192.168.128.0/17 and also does not touch multicast traffic.
Yahhoooo ! It works ... Our guess was good after all...
For the future we assume that the home network is subnetted as follow and are all inside VRF inet:
- appliance port#1 @ sdn1: 192.168.0.0/24 (ISP subnet)
- appliance port#2 @ sdn2: 192.168.130.0/24
- appliance port#3 @ sdn3: 192.168.133.0/24
- appliance port#4 @ sdn4: 192.168.134.0/24 (unused)
- appliance port#5 @ sdn5: 192.168.135.0/24
- appliance port#6 @ sdn6: 192.168.136.0/24
- appliance integrated wifi @ sdn998: 192.168.129.0/24
- appliance loopback inet @ loopback0: 192.168.129.0/24
In the config stanza above we configure an access-list called ACL-NAT4 in VRF inet that would translate all incoming packet matching 192.168.[128|129|130|133|135|136|254].0/24 going to anywhere.
Each time you modify ACL-NAT4 you will need to re-apply the NAT calls:
Also notice that we are NAT(ing), 192.168.128.0/24 which is the OOBM subnet of the Linux appliance itself. This is needed during Debian system update/upgrade operation and also needed during the auto-upgrade feature that I'll describe in a next article.
IPv6 does need NAT in my specific case as my ISP has allocated me public IPv6 prefixes. We will see IPv6 configuration in the next articles.
In this article
- We finally have a router that enables connectivity for all hosts inside the home network to the outside world
- due to the ISP specific setup, our router had to translate inner home IP subnets to subnet that can be in turn NAT'ed by the ISP box.
- We have a consistent IPv4 addressing plan
- We now can add very exciting feature from now on. (In next articles !)
RARE validated design: [ SOHO #004 ] - key take-away
In this example we are proposing a basic connectivity scenario. However, keep in mind that depending on your location the configuration might be drastically different. But do not fear ! RARE/freeRouter has all the features need to enable connectivity !
- NAT64 is available. So in case you want to run a pure IPv6 network, freeRouter can NAT64 traffic for you.
- NAT46 is also available. In case you are desperate and don't want to implement a pure IPv6 home network and have an ISP running only IPv6, freeRouter can NAT46 your traffic for you !
- In the example described, we are lucky to have IPv6 public global IPv6 address. We will see IPv6 configuration in subsequent articles.
When installing RARE/freeRouter on x86, you have 2 choices:
- installation with a software dataplane
- installation with a DPDK dataplane
In this precise case, we will consider a DPDK dataplane installation as our hardware is compliant to the requirement listed below.
- CPU with SSE4 support
- DPDK compatible NIC
Note that freeRouter is available where JVM is available
In this article we will pursue the SOHO network appliance installation based on the diagram below, and freeRouter installation using DPDK dataplane. In this situation, the appliance is behind ISP FTTH box demarcation point. As it is typical to French FTTH domestic deployment.
In this case, RARE/freeRouter is connected to a ISP box demarcation point that deliver copper connectivity. Nothing prevents you, following your context, to deploy a similar equipment with with SFP uplinks directly connected to your Provider Edge backbone routers if you own also the dark fiber paths local to the MAN.
[ #003 ] - RARE/freeRouter DPDK SOHO installation
Let's consider the following assumptions:
- ISP box comes with 192.168.0.0/24 subnet configured at RJ45 demarcation point
- Home networkS will be within 192.168.128.0/17
- 192.168.128.0/17 will be subnetted further into multiple /24 in order to accomodate home network requirement
- RARE/freeRouter is connected to the FTTP ISP box via appliance DPDK port #0 (interface sdn1)
- Home traffic going to outside world will be subject to port address translation (NAT/PAT) using an IPv4 within ISP subnet range
- appliance port #1 will be connected to FTTH ISP box and will have an IP within 192.168.0.0/24
IPv6 addressing plan has not been forgotten. It is not mentioned here on purpose in order to not complicate explanations. IPv6 we be the object of further articles. It is not that IPv6 is a complex topic. It just that it deserves special attention. You might not realised it, but IPv6 is everywhere and is used by default between peers as soon as IPv6 is enable. So IMHO we need to get used to it as soon as possible especially if you are a network administrator.
FreeRouter uses 2 configuration files in order to run, let's write these configuration files in /rtr
Let's spend some times on this hardware configuration file, as you might have notice there are additional interesting lines worth to mention:
- Exclamation mark "!" are comments
- hwid is a text field that would just designate the hardware on which freeRouter is running. (output of : show platform)
- proc <process-name>
It is possible within freeRouter startup to launch processes. We use here this feature to start control plane / dataplane communication via veth pair: veth0a and veth0b and also P4Emu/dpdk, p4dpdk.bin packet processing backend.
- proc p4emu /rtr/p4dpdk.bin --vdev=net_af_packet0,iface=veth0b --vdev=net_af_packet1,iface=veth2b --vdev=net_af_packet2,iface=veth1b 127.0.0.1 9080 6
In dpdk, by default dpdk interfaces have port_ids that are sequentially allocated and in the order of appearance in dpdk-devbind --status output usually sorted by pci_id. In the below output interface enp0s1 has port_id #0 and in dpdk it would be pci_id:00:01.0
enp0s1 would be: #0 with pci_id: 00:01.0
enp0s2 would be: #1 with pci_id: 00:02.0
enp0s5 would be: #2 with pci_id: 00:05.0
enp0s6 would be: #3 with pci_id: 00:06.0
enp0s7 would be: #4 with pci_id: 00:07.0
enp0s8 would be: #5 with pci_id: 00:08.0
- DPDK --vdev addition. In this precise case we instruct DPDK to take into account additional veth endpoint we created respectively for
- Control plane / data plane communication
- Linux out of band management access via SSH we installed previously during Debian package installation
- integrated hardware WIFI access point
- in DPDK vdev interface will have in order of apparition in the command line:
- DP/CP communication: 6 ↔ veth0b
- integrated WIFI: 7 ↔ veth2b
- Linux out of band management access: 8 ↔ veth1b
external WIFI access point will be bound directly to an interface of the appliance via DPDK. This will be describe in future articles.
- For now integrated wifi is shut. We will see in later article how to activate it
- At Linux level, if you noticed in the previous article
- management IP subnet is 192.168.128.0/24. OOBM appliance IP is then 192.168.128.254
- management IP seen from freeRouter@sdn999 with IP 192.168.128.1 within 192.168.128.0/24
- with configured a Linux static routes
- If you pay attention p4lang server in p4 VRF
- This VRF has no bound interface
- Is isolated then from the other VRF
- This will allow only local Linux host control plane and dataplane communication
Check Linux appliance local routes
Test local telnet access from linux/localhost
In this article
- we finally launched RARE/freeRouter with DPDK dataplane
- configure RARE/freeRouter with a vanilla config that takes into account all the appliance physical interfaces
- added veth pair in the config in order to take into account:
- Control plane / Data plane communication
- linux OOBM
- integrated WIFI
- Enabled and checked IPv4 connectivity between freeRouter@sdn1 and ISP demarcation point
- Check telnet access to freeRouter from localhost only
RARE validated design: [ SOHO #003 ] - key take-away
From this point you have a complete freeRouter connected to ISP box via SDN1 as uplink in 192.168.0.0/24 subnet. We will extend further this base configuration step by step in order to enrich user experience !
- Now you would want to enable IPv4/IPv6 connectivity to all potential hosts@home whether they are connected via RJ45 or via built-in WIFI.
- you would also want to distribute IPv4, IPv6 to all the of hosts@home
- IPv4/IPv6 connectivity is not enough, you would like to provide Domain Name Service to them
- Domain Name Service is not enough if they can't reach outside world. As we are using RFC1918 addressing plan we should figure out a way to ensure NAT/PAT address translation in order to enable egress traffic toward the Internet
- Your home might have several floors and only one WIFI access point is not enough ? Let's see how we can add additional WIFI AP in the network
- Maybe you have an outsourced network management service ? Let's see how connectivity can be enable via OpenVPN encrypted tunnel
- Last but not least, let's see how we can connect DN42 parallel network using a Wireguard tunnel relying on an IPv6 underlay.
You've guessed it, all of these points will be elaborated in the futures articles. Therefore stay tuned !
I'm not sure if this is still the case now, but back in 1999, I had the opportunity to managed multiple VPNs at a very huge French Service Provider. I'm saying huge as in this type of MPLS muti-service core network, you could have hundreds of VRF in the same PE router connecting a myriads of CPE via X25 (XOT), frame-relay and ATM PVC at best. In that context, some companies could have several thousands of routers in their VPNs and it was not common to follow a high pace deployment which was at ~10 CPEs per day for a new customer VPN implementation. So one of my favorite CLI command was:
That being said, I'm not sure if this has evolved since then as TFTP occurred inside a very protected out of band management network, it was very good and did a perfect job. Keep in mind that we could be hundreds of "VPN owner" deploying CPEs at the same time. This has to be highly available.
That was for the anecdote, but recently I attempted to upgrade my OpenWRT wifi router from 18.06.02 to the latest code train: 19.07.4. As a I'm lazy, I just sticked with OpenWRT web upgrade via LuCI. Not sure if I was right ... I don't know why and how but the upgrade failed and my wifi router got "bricked".
After a lot of googling and reading, i concluded that I had only one solution: restore from factory and re-install OpenWRT 19.07.04 installation by hand. You have guess the rest of the article, the factory-reset procedure requires a TFTP server.
But before that, I had to solder an USB - UART module as described here.
As again i was lazy on installing a TFTP server on my MAC and disconnect my current LAN access in order to have a direct connectivity with the OpenWRT box, I had an idea (this is not often ) off the top of my head: "Hey, maybe freeRouter has a TFTP server that I can activate in few lines ?"... Well, after a terminal connection to my home router let me introduce you to freeRouter/TFTP server:
[ #004 ] - Saving private OpenWRT", thanks freeRouter's TFTP server !
If you are familiar with Cisco operating system you will feel at home with this TFTP server.
sdn6 is the port #6 connected from my SOHO router to OpenWRT router.
So the LAN port of my OpenWRT router is like this:
Basic connectivity check (well technically you could not ping as it is part if TFTP restore to factory process. Remember our box crashed ! )
So we are basically ready ...
And ... Voilà !
You can deploy freeRouter manually in a VM or container and bind it to a linux interface if you need a TFTP server in order to apply configuration to all your equipment. When final staging are done in a secure Out of Band management network context having a TFTP server is a blessing as it correspond to a gain of time in a production environment. Imaging hundreds of people working in a SP environment and working at the same time.
In this 4th article:
- We presented freeRouter TFTP embedded server
- You can use it in order to undertake network equipment deployment requiring TFTP
- This TFTP server is compatible with IPv4/IPv6
TFTP is a basic but a common tool in SP environment (or it was? If it is still used, yes please confirm !) In this example, I demonstrated the use of TFTP server in order to flash a wifi router to factory default. I have 802.11ac back up and running !
freeRouter can be perceived not only as a router but it is a networking Swiss army knife. in further articles we will shed some lights in various treasures hidden into freeRouter... And for free !
Last but not least, you can play with these different servers from this sandbox: (You'll be able to spot amazing server that will be the object of further article.)
In order to exit the sandbox session use the following escape sequence: Ctrl-c + Ctrl-x
In Greek mythology, Prometheus is a Titan that is credited mankind creation by stealing Fire from Gods and by giving it to human. In the RARE context, Prometheus is a the software from prometheus.io project. It became very popular in the IT industry as it is very simple to implement/configure while providing a great number of metrics without impacting application performance. It is heavily used in microservices environment such as docker and Kubernetes. The mythological reference gives us an indication of how Prometheus is operating. At a constant rate, Prometheus metric collector or server is stealing metrics from Prometheus agent. All the stolen metrics are then consolidated in Time Series database ready to be poured to a queueing system for proper visualization.
Before going further, allow me a brief digression by sharing with you a small anecdote that leds to this ongoing work related to network monitoring for RARE. As mentioned previously, our focus is to elaborate RARE/freeRouter solution the possibility to be monitored in an operational environment. In that context, we started with the implementation of a lightweight SNMP stack that provided relevant result via SNMP tools like LibreNMS. This is great for organisation that wouldn’t want invest time on anything but SNMP.
However, we felt a lack of flexibility due to SNMP inherent structure and we needed more versatile and instant monitoring capabilities. More importantly the need to export infinite metric type from Control Plane in a more flexible way arise. How metrics such as: Number of IPv4/IPv6 routes, IPv4 BGP prefix, IPv6 BGP prefix platform JVM memory etc. could be shared without too much hassle ?
After some internal discussion, I just said: "I’m not a monitoring expert but we have tools like ELK and PROMETHEUS and GRAFANA in NMaaS catalog … Shouldn’t we consider use this ?"
The answer was: « Let’s give it a try and fire up a Prometheus and Grafana instance from NMaaS platform !»
Some hacking at the control plane code level were initiated, after few hours freeRouter lead developer came up with a solution and said: Let me introduce you "freeRouter prometheus agent »
And thanks to the great support of NMaaS team, in few minutes and some point and clicks (it took longer than expected as I’m not good with GUI) we were able to test this agent.
Why is it important you might say ? It is just that with prometheus simplicity and low resource overhead with have full control plane metrics visibility !
As a side note this is not a replacement for INT/Telemetry/Netflow/IPFIX that provide different type of data that are to at the same scale…
People with INT/TELEMETRY/NETFLOW/IPFIX are talking about a "data lake" or "data deluge". Which is correct, if you think about the complexity of resolving a gigantic producer/consumer data problem. This needs the relevant IT infrastructure in order to process all of the data provided by these protocol at the NREN scale.
While in our case, we are just focusing on exposing CONTROL PLANE METRICS at the network element level. We simply monitor and ensure a router operation by using prometheus metrics
While he above might be true, the number of metrics exported from a prometheus target can be very high. Fine tuning might be necessary in order to make sure that all metrics are really necessary for network monitoring purpose. This explosion of metrics exposure can add unnecessary workload at the control plane level.
Again, kudos to NMaaS team that made this happen so that we could test this on the P4 LAB with — ZERO — effort.
In this article, we will present freeRouter and Prometheus integration and as an example we will implement one of the 22 grafana dashboard that we developed and published here. In the rest of the article we will assume that you are a running one or more freeRouter nodes.
[ #001 ] - Cookbook
Once deployed you can push the following prometheus.yaml config:
In this configuration we assume that we have 2 freeRouters that are configured as above (192.168.0.1:9001 and 192.168.0.2:9001) in prometheus worls these are called targets:
- each target are interrogated or "scraped" very "scrap_interval" which is 15s here
- the main job name is called; "router"
- metrics_path is: "/metrics" so the scraped URL is: "http://192.168.0.1:9001/metrics"
Note that this had to be deployed only once for all of your routers. However, each time you'd like to add a new router, you have to add a new target in the "targets" YAML list.
In this example let's focus our interest interface metrics. Please note that this configuration should be deployed on each freeRouter and connectivity should be available between all targets and the prometheus server.
- The objective is to tell freeRouter control plane to expose hardware and software counter interface metric using the sensor object.
- You have 2 types of sensor:
- Universal sensor: Sensor definition that you can cut/paste anywhere
- User specific sensor: Sensor definition that you need to adjust depending freeRTr configuration implemented by user
So this basically means:
- From freeRouter CLI, issue the following command:
- prepend to the metric name: "iface_hw_byte_"
- column 0 will have prometheus label ifc=
- replace all dots "." by "_" . (so interface bundle1.123 will become bundle1_123)
- column 1 defines a metric name "iface_hw_byte_" concatenated to "st" => "iface_hw_byte_st" which is essentially interface status
- if column 1 "state" value is admin/down/up we associate value -1/0/1
- column 2 defines a metric name "iface_hw_byte_" concatenated to "tx" => "iface_hw_byte_tx" which is essentially interface bytes transmitted counter
- column 3 defines a metric name "iface_hw_byte_" concatenated to "rx" => "iface_hw_byte_rx" which is essentially interface bytes received counter
- column 4 defines a metric name "iface_hw_byte_" concatenated to "dr" => "iface_hw_byte_dr" which is essentially interface bytes dropped counter
- Then you need to bind the configured sensor to prometheus server:
And if you followed this correctly, we are repeating these lines for software interface counter metric.
You can view Prometheus configuration for various Grafana dashboard here. Feel free to study these Prometheus configuration and activate them as you see fit depending on your requirements. The set of dashboard is not exhaustive and is by no means absolute. Feel free to submit additional dashboard ! We would gladly add them in the current list of freeRouter Dashboard.
After this definition a freeRouter level you should have:
4 metrics related to hardware counters
4 metrics related to software counters
Which is a total of 8 metrics
From that point you can check via prometheus console:
check the "Targets" menu drop down selection
From that point you should be able to use PromQL query filed in order to check that you can retrieve the metrics we defined above.
For metric visualisation, we will use Grafana. Therefore:
- install Grafana from official web site.
- Once installed configure Prometheus as Grafana data source:
- fill in all the prometheus server information
- check the the data source is defined correctly by clicking the "Save & test" button
At that point your Grafana and Prometheus are correctly binded.
- now you need to import "RARE/freeRouter interface bytes" dashboard
- download freeRouter interface bytes dashboard here
- import the dashboard via ID or simply download JSON or use JSON panel
And Voila !
In order to immediately see the graph zoom in to 5m period with a refresh of 5s and you should see automagically the interface bytes TX/RX on all interface for each targets.
This example related to interface metrics is universal, as the metrics at freeRouter level are yielded through a generic CLI command:
- "show interface hwsummary"
- or "show interface swsummary".
However some metrics cannot be retrieved by generic interface. Some metrics will be tied to specificities of your network. These can be the AS number, IGP process name, VRF name etc.
Let me give you a couple of examples:
But your network context you could have arbitrary deployed "isis 2200". (2200 is RENATER AS number)
In this 1st article, you were presented :
- freeRouter/Prometheus integration
- How to add a new router in the list of Prometheus target
- How to integrate a RARE/freeRouter Grafana Dashboard. (Feel free to adapt the other available dashboard query to your context !)
In Prometheus philosophy, normally the user should do only the minimum of tweaking regarding configuration. Ultimately, he should be only be able to enable a metric or simply disable it if the scrape cost is too high. However in freeRouter/Prometheus integration process, you see that some metric are issued using specific $variable (VRF, BGP/IGP process number ...) Which makes impossible to maintain this universality. However, from the network operator point of view this should not be a showstopper. On the contrary, it is a powerful choice to be able to alter these command via $variables.
Remember in freeRouter philosophy you can have multiple VRF, multiple IGP and multiple BGP process number ! (Which is not the case for all routing platform)
Last but not least, this Prometheus agent was developed quickly because of one reason, all the objects at the control plane level were already well structured in table form as previously described in this article. So implementing this table row/column logic in order to derive a prometheus metric was technically possible without too much hassle.