...

The sources generating log files are a freeRadius FreeRadius and a DHCP server where Filebeat was installed as an agent. Thus the data flow starts with Filebeat collecting log events and forwarding them to Logstash. At Logstash, logs are filtered/enriched according to the needs of WiFiMon, before sending them toward Elasticsearch nodes in the cluster.

...

To implement this setup in your environment:

...

.

Package Installation

The filebeat package was installed in the dhcp DHCP and the freeRadius FreeRadius server which implements the Eduroam eduroam Service Provider. For more information see Repositories for APT and YUM.

All the packages implementing the cluster's components (Elasticsearch, Logstash, Kibana, Filebeat) must be of the same version. The version of the ELK cluster can be easily found from the "Cluster Management" option in Kibana. You should install the appropriate Filebeat package.

All of the following commands should be executed as "root".

Filebeat Configuration

Filebeat monitors log files for new content, collects log events, and forwards them to Elasticsearch, either directly or via Logstash. In Filebeat terms one speaks about a) the input which looks in the configured log data locations, b) the harvester which reads a single log for new content and sends new log data to libbeat, and c) the output which aggregates and sends data to the configured output. For more information see Filebeat overview.

...

filebeat.inputs:
- type: log
  enabled: true
  paths: /path/to/your/radius_logs
  multiline.pattern: '^[[:space:]]'
  multiline.negate: false
  multiline.match: after
output.logstash:
  hosts hosts: ["wifimonFQDN-elastic.unil.cloud.exampleswitch.orgch:5044"]
  ssl.certificate_authorities: ["/etc/filebeat/certs/ca-certificates.crt"] 
processors:
- add_fields:
    target: ''
    fields:
      logtype: radius
- drop_fields:
    fields: ['input', 'host', 'agent', 'acs', 'log', 'ecs']

...

filebeat.inputs:
- type: log
  enabled: true
  paths: /path/to/your/dhcp_logs
  include_lines: ['DHCPACK']
output.logstash:
  hosts: ["wifimonFQDN-elastic.example.org:5044"]
  ssl.certificate_authorities: ["/etc/filebeat/certs/ca-certificates.crt"] processors:
- add_fields:
    target: ''
    fields:
      logtype: dhcp
- drop_fields:
    fields: ['input', 'host', 'agent', 'acs', 'log', 'ecs']

...

set +o history
filebeat setup --index-management \
-E output.logstash.enabled=false \
-E 'output.elasticsearch.hosts=["wifimonFQDN-kibana.example.org:9200"]' \
-E output.elasticsearch.protocol=https \
-E output.elasticsearch.username=elastic \
-E output.elasticsearch.password=elastic-password-goes-here \
-E 'output.elasticsearch.ssl.certificate_authorities=["/etc/filebeat/certs/ca-certificates.crt"]'
set -o history

The above command loads the template from wifimonFQDN-kibana.example.org node where elasticsearch is installed. Detailed information is written in the Filebeat log file.

...