Table of Contents
1. Introduction
To achieve its purpose, correlating user information with network performance data, WiFiMon needs RADIUS and/or DHCP logs to be streamed in an Elasticsearch cluster.
The sources generating log files are a FreeRadius and a DHCP server where Filebeat was installed as an agent. Therefore, the data flow starts with Filebeat collecting log events and forwarding them to Logstash. At Logstash, logs are filtered/enriched according to the needs of WiFiMon, before sending them towards Elasticsearch nodes in the cluster.
2. Package Installation
The filebeat package was installed in the DHCP and the FreeRadius server which implements the eduroam Service Provider. For more information see Repositories for APT and YUM.
All the packages implementing the cluster's components (Elasticsearch, Logstash, Kibana, Filebeat) must be of the same version. The version of the ELK cluster can be easily found from the "Cluster Management" option in Kibana. You should install the appropriate Filebeat package.
All of the following commands should be executed as "root".
3. Filebeat Monitoring
Filebeat monitors log files for new content, collects log events, and forwards them to Elasticsearch, either directly or via Logstash. In Filebeat terms one speaks about a) the input which looks in the configured log data locations, b) the harvester which reads a single log for new content and sends new log data to libbeat, and c) the output which aggregates and sends data to the configured output. For more information see Filebeat overview.
3.1. Filebeat Configuration
The configuration of Filebeat is done by editing the /etc/filebeat/filebeat.yml file. Filebeat will be configured to forward the data towards Logstash.
3.1.1. RADIUS Server
In the following, you are required to insert the FQDN to which the logs will be forwarded. This FQDN is in the form "WAS_HOSTNAME-elastic.WAS_SUFFIX". For example, if the FQDN of the WAS is "was.example.org", you will have to insert "was-elastic.example.org".
...
The important settings here are the multiline.* ones which manage multiline formatted logs. The .pattern matches lines starting with white-space. The .negate and .match work together, and combined as false and after make consecutive lines that match the pattern to be appended to the previous line that doesn't match it. This makes all the lines starting with white-space to be appended to the line that hold the date, actually the first line in the radius_sample_logs. For more information see Manage multiline messages.
3.1.2. DHCP Server
The following is the Filebeat configuration on the DHCP server that forwards data to Logstash:
/etc/filebeat/filebeat.yml
...
The above command loads the template from FQDN-elastic.example.org node where elasticsearch is installed. Detailed information is written in the Filebeat log file.
3.2. Log Format
Below are the sample log files used in tests. It's about a log event of a user interacting with the Eduroam Service Provider and another one interacting with the DHCP server.
/tmp/radius_sample_logs
|
...
|
4. References
The following links were very useful while writing this material and performing the tests mentioned in it.
...